If you can pull the piece out it will be easier but it should peel off. Go slow and keep your finger or what ever tool you use peeling at the base. They come off relatively easy. Plus the sponge/tunicate doesn't like being out of water.
Take the rock out is a must when trying to get rid of xenia. It has a good chance to spread throughout your tank like wild fire if you keep ripping it out.
I got rid of mine by pulling the rock out and srubbing the rock with a bristle brush and rinsing it off.
I would slow down a bit on the feeding myself. While they will eat meaty foods they are also filter feeders and shouldn't get too much meaty food. This is my $.02