Just to add to what Sammy said, I try and protect the bacteria on the bioballs, but remove any noticable build up.
When doing a water change, you can fill a 5 gallon bucket about halfway with the used tank water. Before discarding it - I've grabbed some of the bioballs and tossed them in this bucket - swished them around a bit and get them back into the sump tower ASAP. You want to do this while the old tank water in the bucket is still warm as well.
The bioballs will still be slimey - indicating to me at least that I have not completely removed the bacteria.
Looking in the bottom of the bucket when I'm finished, you'll see bits of detritus or organic wastes that were on the balls. As Sammy said, if you're using a sponge prefilter, and cleaning it OFTEN, you will normally not see too much crap in the bottom of the bucket.