First a bit of essential info for discussing this topic:
In my homebrew setting I've refluffed elves to be Neutral (instead of Chaotic) (Good) and both physically and psychically androgynous. They're not hermaphrodites or anything like that, it's just that elven males and females are very hard to pinpoint as a particular gender by non-elves, because their average heights and weights are the same and their body types, faces and voices (as well as personalities) are in-between that of typically male and female.
Elven society is very gender equal and their fashion is gender neutral (it's more about personal preference, practicality or position in the society rather than gender). All elves are attractive in appearance and although they can quite easily distinguish each other's genders they still look very much the same. Because of that elves base attractiveness on skills, abilities and/or personality of an individual. They still don't like ugliness, but don't have any particular feelings towards masculinity or femininity.
Elves that can trace any half-elves in their ancestry (no matter from how many generations ago), can (although very rarely) develop particularly feminine or masculine personalities or physical traits. For example a man might be larger than average or behave more macho. A woman might have bigger breasts or rounder hips than average or be more sensitive.
Also, a bit more common than the above, elves that live or have a lot of contact over a long time with other races might unwittingly pick up some of their personality quirks.
That finally brings us to the point of this topic:
I dunno how to treat those "out of the norm" elves in elven society. Should they be considered "freaks" or "exotic" for being different? Or maybe there's a third option?
There are two angles to this discussion - Personality and Appearance.
The later is pretty straightforward - the elf looks different.
The former is more problematic - all elves have more-or-less equal amounts of masculinity and femininity in them, so I'm not sure how their society would react to individuals who have more of one or the other. What I know is that it would most probably influence that individual's preferences and interests.