Three questions

3. The longer the range of the scanner the further away you can detect ships. Appears to affect draw distance too.

1. I wished they were less distinct too. The appearance may probably be defined somewhere in one of the PFX subfolders but I don't think you can add conditions: modifying it will affect how it appears at any time. Unless the draw conditions are in a script somewhere - not impossible but unlikely. I'd love to be proven wrong though; the select boxes are especially jarring when trying to take screenshots.

Edit:
My bad, you meant the targeting brackets that appear around all ships regardless (allies=green, neutral=yellow, hostile=red). The boundary elipse around your selected pilot/mothership appears to be defined by \Data\TEXTURE\ship_selection.dds; however, that's just the texture. Replacing it with an empty one shows that the elipse is still drawn and highlighted in green. Might be able to find this with a little more digging. The targeting brackets may possibly be found similarly (same texture but 3 different RGB highlights).