Do not take barbarian in any caster multiclass (except maybe paladin). It adds massive MAD (you need str, dex, con and your casting stat), you cannot cast or concentrate on spells while raging, and rage damage isn't really all too beneficial for what you get. Remember, you are still primarily a wizard that gets some survivability in melee and that gets a bit better options for melee than most of the other "melee wizard" builds. Your primary goal is still to cast spells for great explosions, not to take every hit in existence like a primary frontliner like the barbarian, paladin or fighter.
Well, I would still be dumping STR clearly (stats would be 8, 17, 15, 16, 8, 8 to start as elf with 27 pb). Yes Rage wouldn't be useful and I understand how weird it seems to just disregard the main draw of barbarian. However, this character wouldn't be causing great explosions, He would mostly use his spells for out of combat challenges, buffs like mage armor pre-combat and then cast a either a nice buff like Blur or Haste or a nice debuff like blindness or slow to start the combat before moving in to tank a bit and do melee damage with greenflame blade. As such he would need a good con save in case he takes damage, some decent hp in case he takes damage, and a good ac so he doesn't.
Really, if you are worried about durability then splash one or two levels of fighter. You get the con saves, you can pick up the defense fighting style to get +1 AC, you can just focus on dex, con and int, you get a d10 HD, you get second wind and, if you stick with it until level 2, you also pick up Action Surge. By level 20, with 20 dex, 20 int and standard issue studded leather armor, you hit 23 AC before casting Shield. Nothing crazy here, but pretty darn standard as far as optimization goes.
Well, if I compare this fighter at fig 1/wiz 4 to the barb 1/wiz 4 (after a 1 con, 1 dex stat increase): Barb has: 2 more HP, 3 more AC, doesn't have to buy or carry armor, and a somewhat useless rage feature used only for fun purposes out of combat (someone challenges the weakling wizard to an arm wrestling contest or some such). Fighter has second wind and an extra 1st level spell (thanks to barb needing to cast mage armor, so potentially 2-3 slots but my DM tends to have us rest in an inn and adventure for about 8 hours). 3 extra AC seems way better than second wind to me.