Benchmark analysis part 1: over water. VRAM bottleneck ?

[Skinny]

HI all,

Ive been doing some benchmarking with CoD and thought you might be interested in the results so far.

First of all, in this post, all results where obtained using my Turkey Shoot track, which is pretty light, and only over water, ~10 planes. You can watch it here:
http://www.youtube.com/watch?v=Or6pe3Yag60&feature=player_embedded

I intend to add benches over land later, but I might need a better videocard to make that useful.

My specs: Core2 Quad @ 3 GHz, 4870 512 MB, 4 GB RAM.
Reference settings I used, is the best balance between visuals and performance for my rig:

resolution: 1360x768
anti aliasing: 4x
Epilepsy: Off
Model: high
building: low
land:low
forest:low
effects:high
damage decals:medium
buildings:low
land shading: low
grass: off
shadows:off
roads:off

Lets check performance impact of some of the relevant IQ settings. First I wondered about damage decals:

http://img228.imageshack.us/img228/7690/damagedecals.png (http://img228.imageshack.us/i/damagedecals.png/)

There is barely any difference, but I should add I can also not see any difference in the track. Perhaps this only applies to the players own airplane?

Next, I quickly toggled a few other options:

http://img824.imageshack.us/img824/6877/20779050.png (http://img824.imageshack.us/i/20779050.png/)

Note that 'reference' refers to the settings I quoted above. Setting model detail to low (or even medium) seems like a pretty damn stupid thing to do. It only affects your own plane which then looks worse than a 1990's flightsim, and the difference in frame rate is pretty small. Do yourself a favor, set it to max.

"Visual Effects" does impact FPS when you would expect it (when bullets are fired and hit), but the performance penalty is modest and the IQ impact huge. Cant see much difference between medium and high here, but you dont want to set it to low I think.

Lastly, As you can see, shadows is a real FPS killer here. A shame, as it look pretty damn nice, but my rig cant cope with it.

Next, lets look how resolution and AA impact framerate on my venerable videocard:

http://img862.imageshack.us/img862/4336/resaaa.png

Reducing AA does help framerate, but only in those segments where they are already quite playable. In the segments where I struggle to maintain 30 FPS, AA settings dont matter much, if at all. In short, I better leave it on to compensate for the reduced res, since going to 1920x1080 native res of my monitor is not really playable, even with AA disabled.

One would think I need a faster videocard. But is it that simple?
To find out, I tried over/underclocking my videocard:

http://img217.imageshack.us/img217/967/gpuspeed.png (http://img217.imageshack.us/i/gpuspeed.png/)

Surprising, no? Running my videocard at 50% lower speeds, makes no difference whatsoever, not at 1360, not at 1920 res. My bottleneck is elsewhere, one would suspect the CPU.

Lets see what happens when I underclock CPU and RAM:

http://img97.imageshack.us/img97/8744/cpuram.png (http://img97.imageshack.us/i/cpuram.png/)

Not a whole lot! 50% lower cpu speed again makes almost no difference. RAM speed is even less of an issue.

So what is my main bottleck (again, over water) ? I can only imagine its the amount of VRAM.

Before I start testing over land, I would love to confirm this. Therefore, if anyone has a rig similar to mine, but with a 1GB 4870, please see this post:
http://forum.1cpublishing.eu/showpost.php?p=250584&postcount=15

[Flanker35M]

S!

Quite comprehensive test there, Skinny! So is the whole performance issue related to memory management as clock speeds do not seem to affect that much at all..

[Skinny]

S!

Quite comprehensive test there, Skinny! So is the whole performance issue related to memory management as clock speeds do not seem to affect that much at all..

On my rig, it certainly looks that way. I only have 512 MB VRAM, so its hard to say if people with modern videocards with >1 GB are in the same boat. But considering on the track I used there is no ground, no buildings, no nothing but water, I can only guess how bad the issue is over land. I doubt 1 Gig of vram would suddenly solve everything.

Does anyone know if there is a way to compress textures, like there was on il2?

[Space Communist]

Indeed it looks like right now the only people getting really good performance over land have 1.5-2 gb of vram. What this game really needs is a setup application that can check your settings against your vram and tell you if you are going over like in RoF and other games. I think if it had that a lot of people even before patching could get the game quite playable by manipulating the settings to stay under their vram limit.

Hopefully with patching they be able to get more effects on a smaller memory footprint.

[matsher]

Great info, thanks Skinny...

VRAM it is then... Thats a real bugger.
Do you reckon it is easy to fix??? For the !c team to spread the love,
so to speak, to all the other systems (CPU, RAM etc.)

[BadAim]

I'd tend to agree on the vram theory. I have a 1200mb GTX470 and I get good performance with the settings turned down somewhat. The buildings are still a problem though. We'll see in a couple of days how the building optimization turns out.

[Kankkis]

I'd tend to agree on the vram theory. I have a 1200mb GTX470 and I get good performance with the settings turned down somewhat. The buildings are still a problem though. We'll see in a couple of days how the building optimization turns out.

Or maybe today, Luthier put that information http://www.sukhoi.ru/forum/showthread.php?t=67866&page=16 saturday and saying testing takes day or two.

Kankkis

[335th_GRAthos]

Thank you Skinny for the great analysis as it helps cover many questions I had.

Indeed, VRAM sees to be the big challenge to solve and I found a nice tool to monitor it:

Finaly after three days of hard work testing the new setup under Win7 I noticed following interesting things
probably worthwhile sharing.

Yes, it is playable but your GPU must have lot's of memory!
Back in Christmas when I was buying my new GTX-570 with 1280Mb RAM I though the Nvidia guys had gone crazy.
I am happy now I kept that thought to myself.

Win7-64bit is a must, otherwise in WinXP you will be stuck to run the game at 640x480 kai 800x600 resolution (maybe also 1024x768 )

Sorry the photos are big because my desktop has 3840x1024 resolution (three monitors) and the game runs in the middle window at 1280x1024.
At the far right are the interesting stats on CPU and GPU workload:

#1 The game appears to be using nicely many CPUs, I have Process Affinity=6 (using cores 1+2 out of the four 0,1,2,3)
the system runs very smoothly and total CPU usage is 30% (whereas WinXP had only 13%).
So, thanks to Win7-64 the CPU doubled, this is good news! :-)
It is a bit surprising though as it conflicts with the statement that the game poorly support multiple-cores.
You will see that the CPU usage is very evenly spread around the four cores, I am also surprised myself.

#2 The less good news is the consumption of GPU memory. The freeware GPU-Z is an excellent way to monitor this in real time.
Here GPU-Z shows the results of the first GTX-570 (the second being idle).
The most important is in the middle of the GPU-Z window, the Memory Used
Above the sea, 1280x1024 Memory Used is 959Mb RAM :excl:
Above land, 1128Mb RAM :excl: :excl:
MY GPU has 1280Gb RAM so it can still run smoothly but if the GPU has less then problems will start and you will have
to lower the resolution or the quality (grass, roads, shadows, etc, etc).

Mission 1, bombers intercept with 12 airplanes in the air
http://www.stoimenos.com/temp/CoD/CoD_3840x1024_1280x1024.JPG

Quick mission over land with 6 airplanes
http://www.stoimenos.com/temp/CoD/CoD_3840x1024_1280x1024_b.JPG


resolution: 1280x1024
anti aliasing: 0x (I get very bad errors when I try to raise AA)
Epilepsy: Off
Model: high
building: very low
land:medium
forest: off
effects:high
damage decals:high
buildings:low
land shading: low
grass: on
shadows: on
roads: on

I must underline that I am more interesting in flying dogfight than enjoying eye-candy;
The new game has endless capabilities in graphics and it is definitively very heavy for most graphics cards available today
(at least until the next patch arrives) but I do not care that much; if I wanted nice graphics I could have been flying MS FSX four years earlier...
What I find very annoying on the other side though is the unconfirmed information that AI can see through thick clouds
(they had promised us they would fix this in the IL2FB days, it will be a major issue if they did not) -
I can not confirm as I have not tested it yet myself - too busy trimming my rig.

Using GPU-Z you can easily check whether you reached the limits of your system, watching Memory Used and Memory Controller load.
I hope this helps somebody, especialy those who start frustrated "my resolution is 1600x... the game is unplayable" types of threads

~S~

__________________________________________________ ___________
Look also at my post at
http://forum.1cpublishing.eu/showthread.php?t=19777&page=3
for the performance on 800x and 1024x under WinXP.

[15JG52_Brauer]

I've seen my GPUz memory load go to 1550mb over London. I did some tests myself with CPU clocks from 4.7 down to 2.66 Ghz - only saw 5fps difference (Using London Sightseeing as a test). Have not had time to do more in depth analysis) By the way, what are you using to graph the framerates - is there a tool to grab it from the in game framerate counter or are you using FRAPS benchmark and then manually creating the graphs?

[Skinny]

How could I have missed that. I actually had GPU-z running on my second monitor while benching, mostly to keep an eye on temps and load, I never noticed the "memory used".

Well, if I had any doubts, then I guess that settles it. Thanks for the tip!

As for threading, I wish I could say I agree, but I dont. First of all, you have 4 cores, but you also have hyperthreading, which means 8 cores are exposed to the OS. Each core has the ability to run 2 threads, so its presented as 8 logical cores.

Now by setting affinity mask to 6, you are doing the exact thing you should NOT do, that is pin the threads to 2 logical cores on the same physical core (logical core counting starts with zero). Thats precisely what you need to avoid.

Dont have time for elaborate explanation, but people should leave those affinity mask settings alone if you are using windows 7. You will not gain from it, you will only lose. Windows 7 scheduler is aware of hyperthreading, will not schedule one logical cores sharing physical resources unless all other cores are busy. The only reason to set a mask is if you need one or more cores to be entirely dedicated to another application.

XP is different as it doesnt know about hyperthreading and might therefore schedule 2 heavy threads on the same physical core even if 3 other cores are idle.

[Skinny]

By the way, what are you using to graph the framerates - is there a tool to grab it from the in game framerate counter or are you using FRAPS benchmark and then manually creating the graphs?

Manually is a big word, but yeah, just fraps+excel. I think charts are much more meaningful than average/high/low FPS especially with a game that lags randomly so often

[Skinny]

OK now I know why I missed the "memory used" entry in GPU-Z. It doesnt show with my videocard.

As point of reference, can someone run it while playing IL2 or some other game? Im actually wondering if it really is reporting texture memory and not just OS memory.

[335th_GRAthos]

How could I have missed that. I actually had GPU-z running on my second monitor while benching, mostly to keep an eye on temps and load, I never noticed the "memory used".

Dont have time for elaborate explanation, but people should leave those affinity mask settings alone if you are using windows 7. You will not gain from it, you will only lose. Windows 7 scheduler is aware of hyperthreading, will not schedule one logical cores sharing physical resources unless all other cores are busy.


You were probably using your GPU-Z on your second GPU (if you have SLI)

Thanks for the tip regarding process affinity under Win7 I will try.

Apart from that, IL2FB usage (processafinity=6 :-) )
GPU Memory used: 229Mb Mouahahahahahaha!
http://www.stoimenos.com/temp/CoD/IL2FB_Win7.JPG



~S~

[CharveL]

Great work guys but to be honest, besides offering a bit of insight, I'm afraid a lot of the conclusions that could be drawn from your test might not be anything to hang your hat on.

With a slew of patches coming shortly, a lot of memory sucking bugs will be squashed and rendering optimized - apparently to a large degree - which means that people might make some premature hardware decisions based on what you're seeing.

As an exercise though it will be great to compare these with post-patch optimizations to get a better idea what the improvements bring.

[666th_Lange]

MY GPU has 1280Gb RAM so it can still run smoothly but if the GPU has less then problems will start and you will have
to lower the resolution or the quality (grass, roads, shadows, etc, etc).

Just ran a little test myself after seeing your results. I have a GTX480 with 1536MB RAM and this is getting filled up to 1488 MB at lower game settings then yours.

[Skinny]

Great work guys but to be honest, besides offering a bit of insight, I'm afraid a lot of the conclusions that could be drawn from your test might not be anything to hang your hat on.

With a slew of patches coming shortly, a lot of memory sucking bugs will be squashed and rendering optimized - apparently to a large degree - which means that people might make some premature hardware decisions based on what you're seeing.

As an exercise though it will be great to compare these with post-patch optimizations to get a better idea what the improvements bring.

I certainly intend to rerun those tests after patches arrive, but for now they clearly indicate where the bottleneck is, at least on systems similar to mine.

I also think people are too optimistic about the result of those patches; I remember promises of patches to fix stuttering over cities in IL2 (especially with narrow FOV) , but ~10 years later the problem persists, even if hardware became almost 10x faster since.

Perhaps there is some low hanging fruit, like SLI support, that should be fairly easy to implement or fix, but they spent over 6 years developing this game. As Ilya posted elsewhere "Optimization has always been our top priority". Its unlikely in the next few weeks they will suddenly achieve miracle breakthroughs they couldnt achieve all these years.
I certainly hope to be wrong, but I wouldnt hold my breath.

[335th_GRAthos]

Just ran a little test myself after seeing your results. I have a GTX480 with 1536MB RAM and this is getting filled up to 1488 MB at lower game settings then yours.

Hi Lange,

That is interesting. Can you post some more details (resolution and your part of the conf.ini with the DX10 settings)?

[Flanker35M]

S!

I had MSI AfterBurner(latest beta) running while playing COD. Noticed that over water GPU usage was 98-100% and over land varied between 30-80% depending on what was visible(higher with water in sight).

Also overclocked my 6970HD from stock clocks 880/1375MHz to 900/1400MHz and the clock speeds did ramp up to that during gameplay. And also stayed there. When in splash screen or GUI the speeds dropped slightly, but in 3D they were constant. So works here.

My frame rate was around 45-80fps over water, even over 100fps when looking to sky. Over land I got 20-60fps average being around 35-40fps. Checked with game's own fps gauge. I run at 1650x1080 with low forest/buildings(+low building detail). Rest is medium except models High. Grass is disabled, roads/shadows on.

[335th_GRAthos]

Great work guys but to be honest, besides offering a bit of insight, I'm afraid a lot of the conclusions that could be drawn from your test might not be anything to hang your hat on.

With a slew of patches coming shortly, a lot of memory sucking bugs will be squashed and rendering optimized - apparently to a large degree - which means that people might make some premature hardware decisions based on what you're seeing.

As an exercise though it will be great to compare these with post-patch optimizations to get a better idea what the improvements bring.

What you are saying is very right CharveL.

But still, with my exeprience of 9 years flying this IL2 simulation, I bet that even after twenty patches, nothing will change in the big picture: Our PCs today have CPUs with power like hell, RAM in vast amounts, the only bottleneck is our graphic cards.
I have changed fifteen GPUs in the past ten years, it was always the same story (plus four sound cards - the sound chip was a bottleneck in the early years as well).

[666th_Lange]

Hi Lange,

That is interesting. Can you post some more details (resolution and your part of the conf.ini with the DX10 settings)?

Tried both 1680x1050 and 1920x1200 (those are the only Full Screen resolutions i have avialable in COD) at LOW and HIGH settings. Both give same results in filling up VRAM but 1680x1050 is a lot smoother at LOW then 1920x1200 at HIGH which is normal.
I also used an FPS limiter with FPS limit to 30 FPS. Same there, VRAM got filled up entirely so i guess those which have more VRAM will have it filled too.

Config.ini: only tried Afinitymask (15) but didn't noticed much difference in performance, just like turning on hypertreading again (turned that off for DCS: A-10C), ingame AA is set to 8, but it still looks horribly jagged. I which somethig could be set with the nVidia Control Panel but unfortunally...

[335th_GRAthos]

This is very interesting !

Can you run it on a lower resolution 800x and 1024x and tell us whether the same thing happens?
And also please try once with grass, shadow, OFF

In my pictures (the mission1 JU-88 intercept over the sea and the quick mission flying the Hurri above the ground, there is a difference on the amount of Memory used (shown at GPU-Z at the right side of the picture (you will need to scroll a lot))
I am querious to know what your card consumes.

~S~

PS. And by the way, I hope everybody has read this (second page, first post) ;-)
http://forum.1cpublishing.eu/showthread.php?p=252152&posted=1#post252152

[666th_Lange]

OK here we go:

Plane details always HIGH, AA @ 8x (rest see tests), FPS limited at 30 FPS


800x600 @ LOW
VRAM: 950MB, GPU Load 14% max
Overall smooth gameplay with a microstutter here and there.

800x600 @ MEDIUM
VRAM: 1117 MB, GPU Load 27% max, avg around 20%
Still quite smooth but stutters getting longer and FPS drops below 30

800x600 @ HIGH
VRAM: 1338 MB, GPU Load 68% max, avg around 38%
Stuttering becomes annoying, FPS over detail goes down to 15, Dynamic Shadow in the cockpit is not allways OK.

1024x768 @ LOW
VRAM: 1020 MB, GPU Load 27% max
Gameplay OK, microstutters a bit more then 800x600

1024x768 @ MEDIUM
VRAM: 1190 MB, GPU Load 40% max, avg around 30%
Stutter is frequently there when terrain or environment is detailed, becomes unpleasent to play.

1024x768 @ HIGH
VRAM: 1398 MB, GPU Load 85% max but avg 45-50%
Unplayable, way to much stuttering and the dynamic shadowing is not always OK, lot's of shimmering. Control should be possible through NVCP to cure this and get better Image Quality.

[335th_GRAthos]

Great to know, many thanks !

This also shows that the moment somebody increases the amount of "eye-candy" the memory usage is reaching the max available on the GPU.

From all posts read, on Friday they will begin testing the next patch and the trees and buildings are undergoing major tuning.

Also interesting to see that already at 800x you need to have 1Gb VRAM :-o

Trees is the first thing I swiched OFF at least until the next patch.

Thanks for sharing and RGDS,


@Skinny: Thanks for the tip regarding afinity in Win7, I removed the Affinity line and all four cores now are running equaly at about 60-70% each :-)

[Codex]

Skinny ... check your PM :)