I think just about all sponges have toxic compounds as their natural defense. Protiens and ammino acids are converted into these compounds known as polytheonamides by bacteria that live in symbiosis with the sponge.
I wouldn't worry unless they get out of control. The fact that you can grow them is a good sign things are pretty stable.