Typical phosphate concentrations in natural sea water are very low, sometimes as low as 0.005 ppm., and diatoms are still one of the most abundant phytoplankton in the sea. Your level of .1 could be considered more than 100x's that found in natural sea water. At concentrations below about 0.03 ppm, the growth rate of many species of phytoplankton is dependent on the phosphate concentration, while above this level their growth rate may be independent of phosphate concentration, meaning that above .03 they may grow very rapidly reguardless of what your phospate readings are. So if you want to deter algae growth by controlling phosphate, you need to keep phosphate levels quite low. With diatoms you also need very low silicates as well.
What brand test kit do you use? Many PO4 kits don't go lower than .1 which does not help much. How much flow do you have in your tank? Often the brown mat on the sand is dead diatom skeletons that can be picked up and removed by your skimmer and filter if they are aloud to stay in the water rather than float to the bottom. A very high amount of flow would help this. Also, how much and how often are you feeding your fish and do they eat all of the food? Fish food can often be made up of 1% phosphorus or more, which means that if you add 5 grams of flake food to a 100 gallon tank, there is the potential for the inorganic phosphate level to be raised by about 0.4 ppm in that single feeding.
