The only reason the zooanthids are toxic is a deterrent to being eaten. They don't release it, they won't jump over and poison you, just don't try to eat them and keep your hands out of your mouth, nose, and eyes if you've been handling them.
I'd never be able to work on a tank in gloves, personally...too much trouble, too little tactile feel. We no longer have a reef with lots of zoos in them, altho we do have a few here and there. However, lots a folks ask if we wear gloves to work in our tanks because every one of them contains at least one venomous fish, but neither of us wear them. That being said, it's all about what gives you a warm fuzzy...