Generally, the trick there is to avoid having the dice determine whether it works or not entirely, in any situation, theres:
Works great
Works
Fails
Fails badly
So, in combination with the dice, your skill determines your 'center'(the 50% point), do you roll to see how well you do(skill with investment) or do you roll to avoid getting screwed(skill without). The former is usually active(you chose to use the ability) in nature, while the latter is more reactive.
This is relatively easy to do, on a d20, since probability is straight, investment should put you at success from 6-20, with great success from 15-20. Extreme investment pushes you even closer to that, you'd succeed on everything above 1 and are basically rolling to determine degree of success. And vice versa.
So what you need to control is the extreme investment. The extant bonus types, while a fine concept, are broken by dint of sheer variety. What can be done towards this, I think, is a tighter division(only for roll combinations only).
-Innate bonuses. Your base chassis bonus of <class level> effect. BAB, skill points, CL, etc.
-Ability bonuses. Your stat bonus. Spell and item bonuses to stats go towards Internal bonuses below, not here.
-Internal bonuses. Bonuses from an internal source, such as buff spells, or temporary effects like Rage. Existing bonuses that fall here are Enhancement and Insight. If rebuilding the system, only long lasting bonuses that you can carry from encounter to encounter should go here.
-External bonuses. Bonuses from an external source, such as battlefield control, teamwork and circumstances. Existing bonuses that fall here are Luck, Sacred, Profane and Morale. If rebuilding the system, brief 'burst' bonuses go here.
By controlling magnitude availability(or just plain capping) of these bonuses, control is achieved over the potential rolls.
AC as an oddball, should be folded into a largely Innate bonus, with equipment counting as an Internal bonus.
Do note that the divisions are somewhat arbitrary and subject to change, its just a demo.
So this puts the kibosh on stacking lots of bonuses, and you can do probability range estimation. The actual numbers are, funny enough, irrelevant. What matters here is for active modifiers versus reactive modifiers.
Active modifiers would be player-initiated skills and all attacks. Here, you can assume the Innate and Ability bonuses to have low skew(they'd be only a couple of points different unless the player wants to be a unique snowflake), so the range of available variance lies in Internal and External bonuses. For uncontested active modifiers, you can then give challenges that they would succeed on 75% of the time, and with both bonuses, automatically succeed and roll for greater success.
-Contested active rolls are another matter, they're a headache to calculate for, since every modifier is present, and to make things worse, some(internal/external) are present twice. Innate difference+Ability difference+2*External difference+2*Internal difference.
The existing 3.5 method of handling this is that at low levels, Innate+Ability gap is small(Str 16 BAB +1 vs Str 8 BAB +0) while External/Internal effects are infrequent, reversing at higher levels with a large Innate+Ability gap, but with the Internal bonus consistent(due to the +X items)