Take a clean, never used toothbrush and gently brush it off. (The toothbrush can kindof "grab" the algae right near the base, you should be able to do this with minimal to no contact with the polyps)
Is it green hair alge? That stuff is tuff to remove, I even tried a metal brush from HD (on the rock not coral) with no luck....hope it works better for you.