Silica sand isn't really toxic, it just usually has larger granules. If you have too large of sand, your sand-sifting critters wont be able to clean your sand.
FYI, if you have a glass aquarium, it will give off more silica than any amount of sand in your tank.