There is really no fixed limit, IMO, to the number of brittle/serpentstars you can keep. In a 180, unless you are a brittlestar freak like myself, you are likely to only put maybe 4 or 5 and be happy...when even more could fit. In the wild, they occur in high densities under "choice" rocks...perhaps a dozen or more of different species. I've kept 4 in an 8g and 5 in a 15. Just keep them fed.
The restrictions come in with other stars. For a blue Linckia, for example, I would say just one. Other Linckia, perhaps two. Those depend entirely on the amount of LR since they can not be spot fed. Predatory seastars like chocolate chips? Quite a few as they can be fed (but are not appropriate for a reef).
But brittlestars/serpentstars can be kept in relatively high numbers...just be warned that they may pose a threat to some small fish and shrimp. Not everyone has these issues, but it may be instinctive for some individuals.