You will be fine leaving the bristleworms alone as long as you don't overfeed. If you look at them closely at night you will actually see pieces of your sand inside of the worms. They are highly beneficial to keeping your tank clean.
Just do not handle them. They have little bristles called setae that will stick into you like cactus hairs and release an irritating chemical.