On hard, I left the main castle with the key to Demonis for later; I tried it but concluded my losses would be just too great, so I returned there after I cleared out Haas' Labyrinth (so, basically, very close to the end).