There are green, red, pink, and purple forms of coralline algae. There's also different types of coralline algae like encrusting coralline, plating coralline and even coralline digitata.
The yellow could be a sponge perhaps, or maybe something organic that is dead or dying. A pic would be needed to ID it all properly.
If something is red and looks like velvet and can be easily blown off the rocks with a turkey baster, then it could be cyanobacteria.