Hi,
first of all, the GTX260 does support DX10, but only if the PC is running MS Vista or Windows 7. No, a DX11 GC won't improve the quality in COD, because COD doesn't support DX11 (at the moment).
I've noticed a big improvement in fps when I did move from WinXp to Windows 7 (+50%).
On my system, the GC is the limiting factor, none of the other components is running at the 100% limit.
For the VRAM I think the same can be said as for money: the more, the better
- I've heard a lot of good things about the GTX 560 Ti with 2GB