So I looked up sponges on the internet and on this forum and I don't think what I have looks like a sponge. Here are two more pictures.
The thing, whatever it is, is actually relatively smooth on the surface, with very very small pores. It looks rough in the first picture because the bicolor blennie chomps on it.
The second picture shows a tentacle like extension of the thing.