Dryads are nasty, their damage is puny and their fairly easy to kill, but their most nasty skill can lullaby all level 1-3 troops, this can really mess you up if the enemy army has multiple parties of dryads.
Most mage types can be nasty, though not the human ones.
I actually don't mind fighting dragons, or other high level troops as my own dragons take care of those, and take puny damage from getting hit back.
I love fighting slow units, my entire battle tactic revolves around taking care of ranged/mage units first, while using dragons to lure the enemy, without getting hit. Armies without ranged units or mages are doomed from the start.
For anyone that cares I am a level 20 knight with little over 15k leadership, I currently use:
Druids - I don't like em much, hoping to replace them with Hunters, Inquisitors, or even Ancient Ents.
Horsemen - Obviously for the Iron Fist skill, but also have 2 items which each give +5 def to horsemen, amongst other bonuses. With the right items equipped they could have 60+ def. I've outgrown them now and looking for something else.
Elves - Love em, in combination with precision or dragon arrows they can deal massive damage to any target. Possibly the best archers in the game.
Emerald Green Dragons - Nice troop, their Mana source skill can be extremely effective, even as knight I use my mana often. Good tankers and healable.
Black Dragons - My most fearsome ally, in combination with the Emerald dragons their really kicking ass. Wouldn't trade em for anything. They don't die.

I've changed my mind. I hate Demons just about more than anything. Domination and Furious are an absolute pain.

Black dragons and some Daemons. Always take some losses with them.