They are a great addition to your clean up crew. They can grow large, but just think what they must be eating to grow...
There is one particular species that preys upon corals, and a few species that can deliver a painful irritant if you touch them (fireworms).
All in all, great, quite-misunderstood scavengers.