try some different Nvidia drivers like the 270.51 beta or the 270.61 WHQL which are reportedly much better/more efficient for 500 series GPU and work like a dream on my GTX275 unlike any of the 260 series drivers
see:
www.nvidia.com - Driver Download - search
remover one of the sides from your PC case to see if temperatures stay the same under load, if its then >5C cooler with the side off then you have a cooling problem perhaps with fan flows working agenised each other given if you have 2 fans blowing in you will actually have 3 fans sucking out (PSU, GPU, and 1 case fan) thus if the case fan & PSU fan are powerful then they will reduce flow from the GPU fan exhaust as the case will be in partial slight vacuum when it should at the very least match outside pressure or be a slight positive case pressure.
Likewise if you haven’t thought about air flow properly the intake for the GPU could be sheltered from the cool air flow in the case.
FYI: the Maximum GTX580 Temperature is 97 C vs. the 105C of the 200 & 400 series GPU