The real solution is the Oculus Rift with a Leap Motion device.
The Leap Motion is like an Xbox Connect with 1000x the resolution and almost zero latency. It can track multiple fingers, hands and objects in real time with minute detail. So you could have real 3D arms, hands and fingers operate switches, buttons and controls in your virtual 3D world. This is without gloves or any other aids, just using your hands... Im ordering a developer kit to muck around with.
https://leapmotion.com/