I bought a used 125G tank with a few fish... an 8 year old tomato clown, a 2 year old coral beauty angel, a 3 year old yellow tang, and two green chromis that I housed in a 24 gal tank for about a month while the 125G cycled. The tomato constantly harassed the coral beauty. Once I got them moved into their new home with a lot more room, things got a lot better. I have since added a domino and a yellowtail damsel. As long as they're apart, everything's fine. If the domino gets too close to the yellowtail, it will chase it off. Damsels are very territorial, so if you plan to have more than one, you need to introduce them at the same time. Adding any after the dominant one has set up it's territory is asking for trouble. Don't ask how I know...
As far as the clown and damsels getting along, there shouldn't be any problem. My clown is a grumpy old codger, but it never bothers the damsels. I also have two scarlet skunk cleaners and a pair of peppermint shrimp. They all coexist nicely, but it's because they're not overcrowded. Once you start stocking, it's hard to know when to stop. I believe it's two inches of fish per five gallons starting out, but unless you have uber filtration, I wouldn't go too far beyond that. Since you mentioned you're not "planning" on putting any other fish in besides them, you should be okay. Young damsels like hiding places, so provide plenty of cover. The more hiding places, the more likely fish are to stay out in the open...