The waste buildup caused by running without a skimmer for a few days probably isn't going to be catastrophic (or even noticeably problematic). However, people tend to overlook the fact that a skimmer also does a lot to aerate a tank. If your tank isn't very heavily stocked, or you have other means of oxygen exchange, this shouldn't be an issue either. However, I have heard of people finding their fish dead or gasping after the skimmer went out. Adding a couple airstones will remedy that problem if it arises.