Yea, use a tripod, the auto macro mode will work well for most corals but I've noticed that some corals don't photograph well sometimes with the auto mode. Try using the manual focus to focus onto the hard to photo corals. Also try using the aperature mode to increase the depth of field.
Kip do you know if there's a way to take pics in 3:2 ratio or is 4:3 the only picture size on the Nikon 5700?