A well-built spell-slaying TOB character might do it. You get close, have teleportation, and then have those Mage Slayer line that shuts down their ability to cast or get away. You'd want some way to do a ton of base damage on your weapon, too, perhaps just a high strength will do it. My unarmed Swordsage/Crusader/Mo9 build does a pretty good job at it incidentally. At least the Black Ethergaunts seemed to think so ...
Other ideas ...
I once built a strong Charisma-synergy debuffer going into Witch Hunter (ToM). His saves were insane and he had evasion and mettle, giving him some decent magical resistance. And, he had save debuffers. Combined with the Witch Hunter's momentary disjunction this could shut down a caster, and combined with poison it could put out their lights (their fort saves are crap, especially if you turn off all the magic they have). But, he'd need some items to teleport or the vest of freedom of movement to deal with some of their tricks. And, he's a full melee class, so able to hold his own against mundanes, too.
My gold dragon supermount character -- cleric/prestige paladin/arcane heirophant -- does a pretty good job at it. And he will qualify for spellcasting harrier soon, too. Having a ton of dragon hit dice means your saves are god awful, and with flight, etc. you can circumvent a lot of obstacles. You can also force casters to burn their defenses by laying down a fat maximized breath weapon.
That being said, "casters" is really really broad. It covers the most pages in D&D. So, it's hard to build a character to account for all of that.
