Impossible to take on Castle Karmag before Islands of Freedom, and taking him after means you can just take him anytime, even after taking Demonis.
The optimal time to take him is when you plan to return to the Islands of Freedom, so you could visit priest Olwen en route to finishing Enemen. By that time, he won't even put up a decent fight.
