It depends on whom your speaking with. - From my experience, I wouldn't think it would harm the fish, or overly stress the animal out. However something to keep in mind; it is best to keep your hands out of your tank as much as possible.
If you have corals in your tank, you most defiantly do not want to move things around much as it will slow there growth. - Also the longer you leave your rocks in place the better, over time helpful sponges will grow between them and hold them together.