If your puffer is anything like my baby, and it sounds like he has that potential if he is just being finicky and not ill, I put ghost shrimp in his tank too. And this was during his hypsalinity treatment, so the ghost shrimp had a lot of time to exist in his tank. He never ate them. In fact, the only time he got testy with them is when he was eating a piece of shrimp, human consumption shrimp, the ghost shrimp smelled the food and came walking over. Our Ipo looked at the ghost shrimp, gave him an evil eye an puffed at him. The ghost shrimp backed off and ip continued to eat his peice of shrimp. Not the ghost shrimp.
Don;t fight it - give him small bite sized pieces of raw medium sized shrimp. I have been there, done that.
Please let us know how you make out.