It sound like I am - but I am not joking with the "wallowing in waste" scenario. If you wouldn't do it, don't make your fish do it.
Would you leave your dog in a yard and never clean up its waste? Nope. Same is true of fish. While some of the waste is "converted" by other animals, in the end, some accumulates.
Water changes, IMO, are absolutely necessary.