Off topic: All these discussion threads should get their own subforum or make it so other people can post in the Handbook forum but not create threads without being a Handbook writer. It's completely overwhelming everything else.
With that said, I'll share a few of the basics first.
Humanoid NPCs. 99% of the time, these guys are just free XP and loot. It's not hard to see why, with their low stats, low wealth relative to PCs and so forth. Still, it's possible to consistently hit that 1%. It just takes a lot of work in order to compensate for their innate suckiness. Both on an individual level and as a group. If you aren't using them as a synergistic group, you are wasting your time.
In addition to that, any boss creature, humanoid or not must be incredibly resilient. If it's not immune or near immune to anything the party can throw at it, the party will run right over it in half a round due to simple Action Economy at work.
The last main principle is that enemies will only get 1 or at the most 2 actions. Make them count. If they cannot make themselves memorable quickly, they will not make themselves memorable at all.
Here is a relatively simple example.
Wyrmlord Koth is a Bugbear Sorcerer with Abjurant Champion levels. On one side, it's not a very optimal race due to the Charisma hit. On the other, non associated class rules means he gets 8 levels of casting at CR 8. Clearly, he is well suited to being a gish.
With Greater Mage Armor, Shield, and Divine Companion his AC is 35 and his saves are 17/16/18, with a +2 to will saves from Bolstering Voice if the right troops are still alive. For the level 6 party he is meant to fight, he does what he is supposed to do. His touch AC is only 20, which is not so good. However he has Wings of Cover to cockblock any attack that he deems a problem such as debuffs, Dispels, and so forth. This allows Koth to stay healthy and able to sweep.
On the offensive side, he's no slouch either. His biggest move is Wings of Flurry, which will juggle parties nicely. However that isn't the only save or lose he has, as he also has Imperious Command + Never Outnumbered + Fearsome, a Sudden Stunning weapon, and Bladeweave, meaning that he can throw out two or even three save or loses a round every round, further helping to mitigate the Action Economy disadvantage, as well as preventing the party from mounting an effective counteroffensive.
He also has some useful support abilities, such as Invisibility and Fly to get into position while the party is being kept busy and start beating them Radiant Historia style by making them lose their turns, and the conventional typical save or lose abilities. And Wraithstrike, to make those melee attacks more likely to hit.
All in all, a very respectable boss creature.
Minion wise...
Karkilan: Minotaur Whirling Frenzy Barb 1/Fighter 2. Straightforward spiked chain beatdown and trip down. You don't want to deal with both of them at the same time. You'll have even less ability to fight back. Come to think of it, you don't want to be in his range at all. 30ish damage a swing * 3 = you will die very quickly if you get near him. He also has respectable defenses, AC 28 and all saves in the 10-12 range.
Uth-lar: Hobgoblin Warblade 5/Fighter 1. This one is heavily dependent on several other factors. As it is though he's a good physical sweeper, fairly resilient... and has WRT, so unless the party really wants to get Radiant Historiaed by chaining together multiple turns, he'll die quickly. Given that the reason he was there is for a grudge match of sorts, the party had plenty of reason to not like him so that wasn't a problem.
Wolf riders: These guys are entirely dependent on houserules that make throwing weapons actually worth something. Without those, they'd do about 0 damage and would be pointless to include as they'd be only marginally better than the mounted archers they were meant to replace. With them, they get dex to damage. And then get bard song to damage. So they can put out some decent damage while moving around the field quickly.
Guards: These guys lack offensive ability, plain and simple. 12-21 just isn't good output, though 14-28 is decent. Defensively wise they're rather sound. AC is lacking, being a mere 21, making them the lowest on the field not counting the mounts or warchanter. Special defenses though are respectable, and they have a save reroll which makes them remarkably hard to get a save or lose to stick to... and remarkably good about soaking all those up. In other words, they're the walls of the team. They have some support abilities, though some of them aren't worth remembering (I kept forgetting some of them have Iron Guard's Glare the entire battle) and some weak lockdown abilities to get enemies (as in the party) to hold still where they can be dealt with.
Warchanter: This guy does about nothing directly. (About his only direct action that meant anything was sniping at someone who was already at 12% health and KOing them towards the end) Indirectly though, he's boosting the to hit and damage of all enemies by 4 in the first round and by 5 in all subsequent rounds. Despite being the weakest unit on the field, he's perhaps the most important for this reason alone, especially considering how badly the wolf riders and Uth-lar need all the flat bonuses they can get to sweep. It matters less for the others, as they have bigger hits already but it still helps considerably.
As you can probably imagine, if you have to deal with all of this at once you're going to be swept in under a round. D&D is a fast and brutal game, and this encounter emphasizes it. Take out your enemies quickly or be overwhelmed by them. Due to the number and quality of the opponents, the standards for "quickly" are higher than the already high standards for a normal game.
I finished this battle not too long ago.
It ended with the party of mostly Tier 1s being out or almost out of spells and almost everyone at single digit HP or worse, as well as two deaths, one of which was soon reversed (the other didn't want to come back, so the last charge of Resurrection gets saved for another fight, and yes there already was a Resurrection item there).
Given the levels involved (the fight was worth 55% of a level by itself) that sounds about right.
