Yes, my scarlet, blue legged and left hand hermits have all grazed the gravel of diatoms but they are not effecient enough to make a big difference.
I find that doing a weekly small vacuuming works great.
BurnNSpy
Is this a new tank? Do you already have hermits in the tank? What size tank? Just curious if you are just getting these during or just after a cycle, I think that is a little normal to start and should go away. Otherwise, you may want to get a cleanup crew or as Burn stated vacuum the tank bottom a tad
--Andy