Not to get all gross, but color does matter here too. If it's the "general" red/brown color most fish waste is, then it might be nothing. If it's lighter, all the way to white, then definitely cause for concern.
Keep an eye on the fish's overall body weight, you probably won't notice a pitched stomach, as that would be where the parasites are, but look for a pitched area behind the eyes and gills, and along the lateral line to the top of the back, that's where most of the fish's fat is stored. If he's got worms, he'll burn through that quickly.