Microstutter in latest-gen cards - examples included

[Arseface]

Yeah, for the heaven benchmark that's not too bad at all. I've been getting 20-25 for the most part, at 1920*1200 4xAA 16xAF. May still be fairly noticeable though, at a moderate framerate.

What drivers are you on? 258.96?



edit: This plot I posted earlier is a reasonable example of the framerate variation you can expect with a microstutter index of around 18-20:





edit: Just computed the microstutter index in excel for the (blue) dataset shown above, and it comes out as 18.6%. This excludes the first and last 3 data-points, as these are needed to form the local average for the adjacent points. So yeah, the above should give you a very good idea of what ~18% MS looks like. Although obviously it will vary over the course of the benchmark, some parts being worse and some parts being better.

[Arseface]

I once saw a nifty little program that you can test microstuttering with different variables. You should try it out if you can find. I think it's made by some german guy.

Sounds ace... I certainly wasn;t aware of any such program.

If you come across it then please post a link

[Asmola]

If you're running at settings which are largely GPU-limited, then you will see the microstutter (as you have found above).

In CPU-limited scenes you will see very low microstutter, as the GPU output syncs to the CPU, which it is waiting for after each frame.

Given that you have gained (virtually) nothing in this benchmark from overclocking the CPU, I would guess you're largely GPU-limited in this case. I suspect the scene that Zaxis benchmarked was largely CPU limited.


... Looks like I don't need the entire game in order to run the benchmark, so I'll grab it and do some comparisons using the GTX480s. Good to see more ATI results coming in now though - just what I'm after in order to compare the two technologies It would help if you could let me know the exact settings you used (resolution, AA/AF, detail levels etc), and when you started the FRAPS benchmark / how long you ran it for, so I can replicate as close as possible



Edit:

1920*1200, 4xAA, 16xAF, textures=very_high, SSAO=on, tessellation=on, shadow quality = high.

Ran the FRAPS bench for the entire run (stopped as the benchmark fades out)







Ouch... Pretty bad Although, as above, the sufficiently high framerate made it seem pretty smooth.

I used the same settings but my rez was 1080p as you did, and i had 1,2GHz overclock on that second run so it sure is gpu limited. I'll test what kind of result FC2 (games own benchmark-tool) with minimum DX10 settings gives @ 1080p, i think it's more CPU limited. Do same if you have that game.

Update: Far Cry 2 with minimum graphical settings (DX10 all off or lowes settings) there are almost non microstuttering as you can see:


Atleast that confirm's that microstuttering and gpu limiting are walking hand by hand..

[janolle]

Sounds ace... I certainly wasn;t aware of any such program.

If you come across it then please post a link

Found it. http://www.computerbase.de/downloads...er-single-gpu/
It's a simple program that shows moving circles. You can adjust the framerate and the delay of every other frame. Low delay value = big frametime difference. You can quickly change between "microstutter mode" and normal with the space button.

You'll really see the difference when moving your mouse...

EDIT:
Edit the settings file to set the right resolution.
A nice thing is that it also says what fps experience you're getting with the stutter. I think it's close to the % of stutter that we are measuring here.

[Arseface]

Atleast that confirm's that microstuttering and gpu limiting are walking hand by hand..

Yep, it makes logical sense that a CPU-limited scene would be virtually free from microstutter, and the results we have so far only work to confirm that

I'm afraid I don't have Far Cry 2, but I've done plenty of runs with crysis, incrementally dropping the resolution and looking at the GPU load in afterburner, and I find the same as you have shown with FC2; that as GPU load drops below around 90%, then microstutter rapidly disappears.

Found it. http://www.computerbase.de/downloads...er-single-gpu/
It's a simple program that shows moving circles. You can adjust the framerate and the delay of every other frame. Low delay value = big frametime difference. You can quickly change between "microstutter mode" and normal with the space button.

You'll really see the difference when moving your mouse...

EDIT:
Edit the settings file to set the right resolution.
A nice thing is that it also says what fps experience you're getting with the stutter. I think it's close to the % of stutter that we are measuring here.

What a fantastic little program - thanks!

Yeah, the "feels like" fps should use the same formula as I do - it's fairly logical once you know the true framerate and the maximum delay between frames. The only extra steps my program takes is to estimate the local average framerate based on a smoothing over surrounding frames (since FPS in-game is constantly changing, regardless of microstutter), and perform some basic statistics to average out the amount of microstutter (which will also be changing frame-by-frame).

It's worth noting, though, that this guy's program will give a "best case" scenario, where you have a uniform irregularity in frame output. In the real world, for a given "feels like" FPS there will be some patches where the performance is even worse (as well as patches where it is better).

[rge]

With my gtx 295, I dont see any significant difference between 1 and 2 gpus enabled in the variance in frame rendering (or microstutter as you are calling it). When benchmarking crysis, 3dmark6, cod, or any other games, I see varying framerates with both 1 and 2 gpus, and graphs look very similar with 1 vs 2 gpus enabled graphically and games look same visually, unless 1gpu gets framerates too low then it stutters.



Crysis with everything on highest level, varying frame rates are similar between single and multiple (magnitude roughly same, though sli appears more since x axis slightly different). These runs were from an earlier discussion where some were saying when you straffe instead of walk or run in crysis you see more microstutter, though I never visually saw microstutter in that game. Though I can cherry pick runs and show sli with less variance or vice versa.

Point is, I cant reconcile varying frame rates on graph with visually seeing microstutter, nor can I differentiate 1 vs 2 gpus varying frame rates. But I will give your program a whirl and see if it shows any difference between 1 vs 2 gpus on mine.

[demowhc]

Great little tool guys, here is 5850 Tri-CFX playing the BC2 level snowblind from start to finish, max settings 8xAA DX11 1080p

[demowhc]

And here is BC2 MP map valparaiso in rush mode, only played until the first 2 bombs were blown (~10mins gameplay)




And the Mafia2 benchmark

[blindbox]

So when you're CPU-limited you get less microstutter? Sooo... let's get to downclocking?

[demowhc]

3Dmark 06 GPU Test 1 Return to Proxycon

[Zaxis01]

The Stone Giant DX11 Benchmark.

[Arseface]

demowhc - We have established that there is very little microstutter in CPU-limited scenarios, as frame output is tied to the regular output from the CPU. Of more interest is the 'worst case' scenario, where the GPU is the limiting factor. I would love to see some tri-fire runs from crysis at 2560*1600 4xAA, or the heaven benchmark. Thanks for taking the time to do the runs you've posted already though


rge: - The plots you have are pretty interesting. I must say, in my testing I haven't come across any scenarios where the single-card stutter is that bad; the worst case being the heaven benchmark with extreme tessellation. I can certainly "cherry pick" a few consecutive frames from a framelog that will look bad, but nothing representative. Not suggesting that's what you have done - just pointing out that things seem to have improved as far as single cards setups go Perhaps a driver-based correction?

As for the multi-GPU results, certainly it's possible to get very consistent framerates with multi-GPU setups - any time that the GPUs are running at less than 100% load the output is very uniform. This is to logically to be expected, and we have plenty of runs to back this up. But again, I find a very consistent microstutter in GPU-limited game scenes. I have yet to find any with less than ~15% microstutter. Of course, I can only directly test the GTX480s, but results from other users seem to suggest a similar thing with ATI-based setups.

In terms of the frametime variation in the crysis plots you posted, taking "walking straight" as an example, the magnitude of variation with SLi is around 8ms, on an average of 12ms [so 4ms average deviation; implies around 33% MS index]. On single GPU, the variation is around 4ms against an average of 18ms [so ~2ms average deviation; ~ 11% MS index]. The absolute variation in frametime is not so different between the two setups, but the relative variation is much bigger. The relative variation is the important factor in quantifying the effect of microstutter. To see this fact more clearly, consider, for example, a consistent 10ms deviation: This is relatively insignificant at a 100ms average; i.e 10fps appears like 9fps, but at a 10ms average; i.e. 100fps, it represents an effective halving of the framerate, since the gap between frames is either 20ms or zero.

[rge]

just cause 2 here is sli with stutter index of 36 (1st pic), and single gpu enabled with 18, so sli was 2x in this game. Single card runs varied from 8 to 18, depending on higher or lower detail settings or just variability, didnt run enough times to check. SLI in thirties regardless of settings.

wish I still had crysis would be interesting to use your tool on, I always had to use excel, or just csv outputs that i ran, but dont feel like reinstalling just to run that benchmark.

[Cookiesowns]

I just had a thought, how about the Hard drives? SSD VS Mechanical, I mean if you think about it, it might be so the hardrives aren't keeping up too. Though highly unlikely, just a thought.

[Arseface]

rge: Thanks for the Just Cause 2 info mate I'm not at home right now, but I have JC2 so I can do some 480GTX comparisons tomorrow. I still find it a little surprising that the single cards show such a (relatively) high degree of microstutter, but clearly they do. That's twice as much MS as I've managed to find so far with a single GPU. Perhaps I haven't been testing the right things (I've focussed 90% on multi-GPU), or perhaps things have improved with this generation of cards. And yeah, I agree - no point reinstalling crysis just to do a few microstutter runs!


Cookiesowns: Unfortunately I don't think that HDD speed will have much to do with it. Once the game scene has been loaded into the system memory, the HDD doesn't have much bearing on the rest of the run. Besides, the computation of the MS index culls off the highest 1% of variations. This was done so as not to bias the results with "hitching", or other loading / paging type issues.

[demonkevy666]

Cookiesowns: Unfortunately I don't think that HDD speed will have much to do with it. Once the game scene has been loaded into the system memory, the HDD doesn't have much bearing on the rest of the run. Besides, the computation of the MS index culls off the highest 1% of variations. This was done so as not to bias the results with "hitching", or other loading / paging type issues.

On that note high mhz ram will increase my fps, along with tighter timings.
I think it comes down to how it renders the screen it would less noticed when it split in half by two card vs each frame rendered at once.

[Arseface]

On that note high mhz ram will increase my fps, along with tighter timings.
I think it comes down to how it renders the screen it would less noticed when it split in half by two card vs each frame rendered at once.

Certainly.

A split-frame rendering method (or "super-tiling" or any other method that has all GPUs working on the same frame at once) should do away with microstutter entirely. It should act like a single GPU.

[demowhc]

Stalker CoP and Metro 2033 were pretty bad with Tesselation enabled, about 18%, but with tesselation disabled they were back under 5%. With vsync though, how I play them, they are very steady under 2%.

Heaven 2.1 was a joke with tesselation set to normal, 4xAA 16xAF at a whopping 54% taking the frame rate from mid 80's down to an apparent 40 something fps, it ran like a mofo as you can imagin..

[demowhc]

Ran a loop of Crysis bench at 4xAA 16xAF DX10 VH 1080p

[damha]

demowhc you must be running very low res or that 4.4ghz oc is doing you a lot of good.

[Arseface]

demowhc you must be running very low res or that 4.4ghz oc is doing you a lot of good.

He's still largely CPU-limited.

In my first post I demonstrated that, using dual GTX480s, crysis is just beginning to become CPU bound at 1920*1200, 8xQ AA, 16xAF, very high. The MS index was around 8%. If I drop the settings slightly, the MS index drops also. Here, he is running at a lower resolution, with less AA, and a triple GPU setup. I would expect CPU-restriction here as well.

The heaven result is more in-line with that I would expect from a triple-GPU setup, being an almost-pure GPU benchmark. I suspect you will see similar results from crysis at 2560, 4xAA (you can run that on lower res monitors - it will downscale).

Metro2033 surprises me though - that should be pretty heavily GPU limited, and is still showing good results. I've been threatening to buy this game anyway, so perhaps I'll grab it and test it too. And yes - with double-buffer vsync enabled the microstutter should all-but disappear in any game.

[demowhc]

I use 1080p because I only play games with 60+ fps

I'll try Crysis with 24xEDSSAA in dx9 very high, should put some more strain on the GPU's.

Apart from that I dont have any demanding titles, except ArmA2 which has vsync forced on by default, but runs at a steady 60fps vsync at 3840x2160.

BTW, I actually got a higher frame rate in Heaven than the Crysis bench, so which is more GPU limited?

[demowhc]

Here is Crysis bench with 24xEDSSAA

[demowhc]

I'm fairly happy all my games run very smooth, the only problem I have is with Heaven benchmark, lucky I dont play Heaven I guess :P

oh wait I just saw you said you can run crysis in 2560 and downscale, will try it

[demowhc]

Must run out of Vram because Crysis bench craps itself at 2560x1440, even tried dropping texture res and although it was better it still died in the ass when overlooking the island