most sponges are harmless, but, if you like your mussel and want to keep it, scrape the edge of the sponge back (this wont hurt the sponge) and see if there is any deterioration of the shell, there are several types of borer sponges that prey on bi-valves and mollusks, if there is any noticable deterioration remove the rocks with the same sponges from the tank and let them air for 5 minutes. then place them back in the tank. this will not harm your mussel but will generally kill most sponge types. with about a 85% success rate. some species of sponge create gemmules that can survive impossible conditions so its not 100% successfull gauranteed.
but I agree its most likly a harmless filter feeder, but where it is on a mollusk its best to check and be sure. because if your tank does have boring sponges you will never succesfully keep any clam, mussel, barnacle and even snails.