Ice and Fire: How to pass guard which is blocking access to Prince Guilford
 

I already have the Heart of Ksaltotun and Staff of Starlight but I can not access the castle area where Prince Guilford is. There is a neutral guard is blocking the acees to the castle courtyard and I do not know how to pass him. As this is Ice and Fire I do not yet have flying. Is there something I have to do to remove the guard?