Hello! I had this EXACT same thing happen to me! I could keep inverts with absolutely no problem, but any fish I put in the tank died. They even died the exact same way that your did. First, they would show signs of labored breathing, then eventually would start doing the "dying fish dive" and soon totally died. I had no idea what was going on either, and everyone had all kinds of different ideas for me too.
What I recommend doing, and what worked for me, is let the tank sit....for a long long time with just your inverts. I let mine sit for 3 months. I know it doesn't sound like something that you want to do, but it worked for me. After 3 months, I went out and bought 2 pajama cardinals because they were only $14.99 each, and I didn't want to spend a whole lot of money when I was just testing to see whether or not they were going to survive. Well, that was about 6 months ago, and they're both still alive, plus I have a royal gramma, and just added a goby on Friday. I also have some yellow polyps and some climbing blue mushrooms.
The reason I decided to do this is because I figured that if I gave the tank enough time, it would correct itself biologically, naturally. I fed the tank every other day and I did weekly water changes, but other than that, I did nothing else except for watch my crabs, snails, and shrimp, and close my eyes so I could imagine there being fish in there! lol So, give it a shot!