AI breaking off is due to several factors in my experience:
fuel level- if low on fuel are more likely to break off
Bravery/discipline skill levels- low bravery (in combination with low fuel) increases the chances of breaking off. Being attacked/threatened seems to 'take a chunk' out of a 'pool' of bravery (as does low fuel). When the pool is empty AI go into a 'RTB dummy' mode where they will just try to get home at 500m/300km/h and will rarely do more than mild evasion when specifically attacked.
This behaviour is (slightly) better than on original release.
All this presumes AI have a landing point designated. If no landing point is spesified they will try to circle last waypoint
The AI sliders do have an effect, it seems as though they often act in conjunction with other sliders and some external influences.
To get a 'more mad furball' type fight push up the sliders- but put some variation in for individual aircraft
Basic flying- make full/very high
Advanced flying 3-5 (maybe a few planes higher)
Awareness 5-8
Gunnery 6-10
Tactics mostly high/full
Vision 7-10
Bravery- make variable but keep high if you want them to stick around
Discipline- make variable
would be where I'd start, but play around/watch on external/review tracks