Sand gets lodged and clogs up the coral. Golden headed sleeper gobies are bad for traveling with the sand (what I have). I turn all my coral on the side, so sand can fall off when my goby dumps the sand on them. This works for me and may work for you.
Your yellow polyps may came out again, don't give up yet.