It matters enough for me not to ever subject myself to it, certainly. It is annoying when the plane moves in a direction opposite to control inputs, for example, especially during gunnery. Others do possibly not suffer from it either, which is a concern in multiplayer.
When flying hands off, in a plane that does has the required trims, it is a non-issue, since one can just trim it away when flying hands-off. In planes that don't have trims, it can be a real pain in the ass during cruising as the only option is either to fly at a much lower (or higher) RPM on the engine to counter the left-roll input, or to constantly hold the stick and apply tension on it to the side. I've tried it and could not deal with the strain on the wrists.
I just use external programs for it instead now, so for me the impact is very small: only a loss of some tweaking ability.
EDIT: Do tell more about how the systems work and should behave like, it is something I have not heard much about. I love learning about aircraft all across the past 25 years