If yo can drill some some small holes in plastic and magnet it would give more "tooth" to grip , but it may work against you in drilling the magnet as some are pretty darn hard or drilling holes can alter their magnetism fields. How thatw ould affect it is anyones guess. I think what I would do is if yu can not find the Plastic Epoxie made by Devcon.......I would probably opt for MArine Goop, as I have had that stuff stick on all kinds of stuff, albeit not a super duper bond but stuck just the same. Clean surfaces good, and make sure its dry. Shoe goop is similar to marine goop....and what thew differences is I do not know, but MArine Goop has been holding various plastic items together in my setups for quite some time... Don't know how it will react with centrifugal forces applied and the magnets either, but what do you have to loose. I would give it a try and see what happens. Also make sure it is fuly cured before reinstalling it. Drilling holes in the plastic at an angle is a good idea, and would help, and epoxy should stick to most types of magnets. Look for the white expoxie as it does not contain any metallics in it like JB weld does, if you go with another type of epoxie.
All epoxy has a foul odor to them when curing, but once cured they basically are inert, and shold not pose any problems. Good luck and let us know how you make out.