In its crudest form, magic cannot co-exist with the world; without a Numinous Caul, it converts itself almost instantaneously into intense heat. For most people with the Gift, this means that they suddenly (and messily) spontaneously combust when their powers manifest.
Even for the lucky ones who are born with the Caul, magic still leaks into warmth. Magic users "naturally" have a high fever - which indirectly leads to madness, hallucinations, and an early death as their brain cooks itself internally. With training (and a habitual use of ice baths), this can be mitigated, to the point where some lucky mages manage to live into their 60s.
As a side effect of this generous internal heat, mages tend to dress as lightly as socially possible - it is not uncommon to see spellcasters walking around in swimming clothes in the middle of winter. Due to the fact that armor is a "wearable oven" - to quote one practitioner - spellcasters rarely wear armor that hasn't been mystically treated to disperse heat.
In other words, wizards run around in Conan-esque loincloths or bikinis while the warriors wear sensible armor. If you see a chick running around in a chainmail bikini, she's probably got some hefty magical defenses running... or she just wants you to believe that she does.
This is up there with my other magical justification, where magic is stored in adipose tissue. Since women "carry" fat better than men, most adventuring spellcasters would be women (though wizard academies would be full of people so morbidly obese that they could barely move on their own.)