I don't think an expensive rotary encoder is the solution. I think the problems lays in the fact that a rotary encoder can generate pulses faster than the computer/simulation can keep up.
If this is your first build i would highly recommend you test out the electronics interface and how it reacts with sim before committing to build. Games tend to be built to accept keyboard/joystick inputs and sometimes don't behave as expected (and sometimes things are modelled in a peculiar way)
Dunno if this is of any use/interest to you but this is a little mission/script to put bombsight parameters in server bar. If nothing else it shows how to get instrument data from game (it's a Ju88 mission but it gives an idea)
http://www.mediafire.com/?6sch8633dpgsfkv