I've had some pretty aggressive damsels in my tank. I had a three-striped, my first and favorite fish, kill a coral beauty angel within a matter of 20 minutes. The damsel was almost five inches long at that point, though, and the dwarf was a bit smaller.
From all the ones I've seen, yellow-tailed and orange-tailed blue damsels are the least aggressive as they mature.
Right now, I have a blue velvet (aka devil) damsel, and at about four inches long, he's mature but peaceful. I think a big part of it, though, is that most of my other fish are as big as or bigger than he is. Size definitely does matter!
If you want color in your tank, damsels are some of the prettiest and hardiest fish you can get. The only thing I'd worry about is size. Once your damsel gets larger, more often than not, he'll eat any fish that can fit into his mouth. Just make sure your fish are of similar sizes to level the playing field.
Good luck! :joy: