The lower the sensor resolution (i.e. the more "fine grained"), the better. I forgot which value it is looking at, but it's the one affected by stealth - look at the stealth modules.

Side note on stealth:
The stealth modules work by lowering that value. So if that value is lower than a ship's radar resolution, it won't be able to see the stealthed ship. This is the basis behind the "4 Ghost modules ultra stealth" trick - each Ghost module reduces the ship's profile by 80% iirc, so 4 of them would reduce the profile to .2^4 = 0.0016 of the original value.

There's supposed to be a minimum limit actually (it's somewhere in one of the script files) and in practise you can get away with 3 Ghosts and still be able to fly right up into a hostile ship's face and remain undetectable. The broken thing about the game is that attacking doesn't invalidate stealth. Worst of all the AI does not respond to attacks made if it could not detect the attacking ship - hence the long range missile sniping trick and this invisible mode trick.

You'd think a ship being attacked by an unseen hostile would panic and start flying around trying to find the source, or failing that, flee - right? But no, it just continues as if nothing is amiss. Due to this behaviour I've given up on the whole radar / stealth thing.