Designers definitely thought Strength was a big one if half-orcs are any example. +2 Str, -2 Int, -2 Cha... Yeah, that blows. And it's not because it's a physical ability, since dwarves get +2 Con but only a single -2 penalty with Cha. I'm not sure if I've seen a +Dex -Mental race, but I'd bet it's an even trade.
As far as size goes, Medium is meant to literally be the standard to set by. Everything else is compared to it after all. I personally wouldn't say it's the worst, but it's unlikely to be the best for the majority of situations.