I've never played mage, but I'm beginning to understand now. I play paladin on impossible, and actually got got a pretty good save game start wherein the duplicate Bolo navi map in Scarlet always gives me Call Colossus. Unfortunately, the paladin does not have enough spell power (ie. intellect) and mana regeneration to employ such a tactic; and this is not even counting the inability to raise Distortion and Chaos high enough at the starting levels. Sure, I could still solo unit the first few islands without losses, but the best I could probably do is to get Order/Heal to level 2 and spend 50+ turns flying around my Red Dragon and throwing a breath attack from a distance every couple of turns.
I assume rage generation is not a problem for you given that you have "sacrificial lambs" to dish out pain; rage is critical for digging up chests for wanderer scrolls, specifically Ancient Knowledge. I think the problem encountered by other mage players is that they don't get lucky with spells and units. I believe the early game solo unit strategy for mages relies heavily on level 5 units being available, be it in shops or via Call Colossus. Even with a full complement of spells, is there a suitable level 1-4 unit that could be used to solo?