I have a 10g nano that gives me problems all the time with water quality. You have to stay on top of those water changes. I do a 50% every 2-3 weeks. I have a problem with the protein layer building up on the surface, so I skim it off with a cup and replace the water when it gets bad.
I have a whisper 10 on the back and a rio 180 moving the water in a circular pattern. I have been trying for some time now to find a HOB overflow small enough for this tank because I am going to set up a sump underneath. I have found the design for the CPR C-Siphon overflows on a website and would like to try and have a smaller version, 150-175gph, built. I also have been thikning about constructing a small closed-loop for the tank, like the ones I built for my bigger tanks.
As for lighting, I went with a 70w MH from Aqua Medic. I love it and so does my Kenya Tree, mushrooms, poylps, and the plating montipora I got on Sunday. I had a CSL retrofit 32w 50/50 pc on the tank before and all the mushrooms loved it. I would go with something at least 60w or more on the MH and the 96w quad pc is probably the best you can get for your little tank.