Yes you should stop. Some people will say you should vacuum off the dirt that is on top of the sand, called dietrus or something like that. However I am of the opinion that when you vacuum any of the sand you are sucking out the good bugs and bacteria that live in your sand. I personally have not vacuumed my sand for years now.