Dead sponges REALLY stink.
They usually die pretty slow but if they die fast they can definately foul your water.
If you plan on removing it wear some gloves. Dead sponges can cause some really painful wounds that almost always get infected or worse... inject toxins.
