I also want to try a mixed tank, with aggressive and reef together. Or rather I am trying it now. I had bought a box fish for my aggressive tank, there was noting in it that would be aggressive towards it, and then I find out if it dies or is stress to the point it's on the verge of death it well release toxins killing everything so I moved it into my reef tank and move everything from there to my large tank. I desided to just load up one corner of the tank with rock and stuff of that nature so that the smaller fish can hide where the larger fish can't reach them. Will it work? We'll find out. So far it's been working great and I am really learning how to make it function correctly. I know the fish, like my stingray has absouluty no way to make its way through the rock work the way it's been set up but these small fish still might get too confedent and wander out to far where they can get picked off. But my royal gramma and watchman goby seems to be smarter than that so far and I figure if every now and again a three dollar damsel gets picked off it's not to much of a pain to replace one as long as it doesn't happen to often.
But I sugest if you try this that you should just give the reef a small section of the aquarium apart from the large fishes area so the smaller fish learn there place and stay there or else they may wander to much and can get caught unaware. If they know there place is just one corner the size of a small reef aquarium they learn to watch there back if they get close to the border of there territory just as in a real ocean. Also try stacking up the rock in a way that not only prevents the larger fish from penatrating it but that also alows for a small open area on the other side of the rock for these fish to swim freely and undisturbed for there own peace of mind and so that they will still be on display for you rather than dissapperin into the rock face all the time.
Try it! It will work, you just have to acept that some fish from time to time can and most likely will dissapper on you so unless you have the cash or trust the nature of the fish to avoid troble remember the price of the fish you through into the reef. Because it's like Vegas, don't put any money in you're not prepared to lose. No hundred dollar fish in otherwords.
Oh, and also, you have to remember how newly added fish act when first added to a new system. Some take a long time to find cover and in an agressive tank with fish looking to eat anything that hits the water acclimating can be a little difficalt, so just keep that in mind. Also you're going to want to spot feed your fish to keep them from having to come out in the open, or build up your rock work on one side high enough that the fish can reach the surface of the water by the rock work alown.