[PCPer] Frame Rating Part 3: First Results

[BababooeyHTJ]

PCPer has been looking into some new ways of testing smoothness. They just released an article on their process so far with some very interesting results.

If you haven't been following our sequence of stories that investigates a completely new testing methodology we are calling "frame rating", then you are really missing out. (Part 1 is here, part 2 is here.) The basic premise of Frame Rating is that the performance metrics that the industry is gathering using FRAPS are inaccurate in many cases and do not properly reflect the real-world gaming experience the user has.

Because of that, we are working on another method that uses high-end dual-link DVI capture equipment to directly record the raw output from the graphics card with an overlay technology that allows us to measure frame rates as they are presented on the screen, not as they are presented to the FRAPS software sub-system. With these tools we can measure average frame rates, frame times and stutter, all in a way that reflects exactly what the viewer sees from the game.

Link

[Mungri]

Interesting, I was doing some more reading on this issue regarding AMD crossfire, and apparently Radeon Pro can fix it:

http://www.tomshardware.com/reviews/...2,3329-11.html

[BababooeyHTJ]

Yeah, Toms just used a framerate cap. Sort of like when people use to use fps limiter in the 4870x2 days. It works great at eliminating stutter but when the framerate dips below that cap you'll still get microstutter.

Its a great workaround but not really a solution.

[STEvil]

Yeah, Toms just used a framerate cap. Sort of like when people use to use fps limiter in the 4870x2 days. It works great at eliminating stutter but when the framerate dips below that cap you'll still get microstutter.

Its a great workaround but not really a solution.

Tried dynamic vsync option?

[Particle]

This testing method is great for testing AFR, but it will be of limited usefulness for testing scissor or tiling. Personally, I'm a tiling fan even though it has the lowest scaling.

[BababooeyHTJ]

Tried dynamic vsync option?

The Tom's article was wrong. They were using dynamic framerate control, not dynamic vsync. Look at the Radeon pro screenshots that they were using. Dynamic vsync is similar to Nvidia's adaptive vsync. It just turns vsync off when it dips under your refresh rate. Very cool but doesn't help microstutter.

Dynamic framerate control afaik is just a hard cap like the old fpslimiter program. That I use all of the time but again its a workaround and not a real solution since if the framerate dips under that cap frame latency will become an issue again.


On that topic. I really do think that if AMD figures out a framerate smoothing option similar to what Nvidia uses it'll fix both the single card and afr stutter. If you look at techreport's old reviews with AMD's 6xxx series and especially 5xxx series they did very poorly with frametime consistency. What? The memory management system wasn't optimized for those cards either? I'm not buying into that excuse.

[bamtan2]

very damning. in bf3 at least, crossfire is garbage. the next time someone suggests crossfire, show them how it skips down to non-crossfire performance, making it pointless: http://www.pcper.com/image/view/2096...n=node%2F56621

[Mungri]

Only reason I was suggesting crossfire recently was because of the prices on 7950s / 7970s compared to Titans and even GK104s.

Seems like the only decent high end is dual GK104s or a Titan atm then.

[BababooeyHTJ]

Crossfire is still very useful. When you can buy thee 7950s for less than the price of on Titan and then pop them onto a mid range board like the UD5h you can get a lot of power for your money. As long as you cap the framerate performance is smooth. Even then microstutter isn't always consistent so you won't see it in every game or all of the time in the games that you do.

This is definitely an issue that AMD needs to address and I'm glad that some review sites are placing more emphasis on smoothness.

[boxleitnerb]

Sure, but at what cost? I've been using a frame cap in the past when my experience with SLI was still very new, but the drawbacks are just too severe.

Because you need to cap fps at or around your minimum fps, you will waste a lot of performance in scenes where those minimum fps are exceeded. So basically most of the time. For example when playing Skyrim with SGSSAA, my fps could dip to the low 40s in dense foliage and forests, which was stuttery. I capped my fps to 40 then, but that meant losing a full 20 fps (I play at 60 Hz vsync) during most of my gameplay. I found that quite unacceptable and it can utterly destroy scaling, calling going with multiple GPUs into question in the first place. Value doesn't look that good, then.
The only thing I really like about RadeonPro and its fps cap is that you can adjust the value via hotkey while gaming. Doesn't solve the underlying problem, but it's more comfortable than having to alt-tab out of the game, changing the value and hoping the game didn't crash due to the task switch.

[Mungri]

One of the main reasons I picked Nvidia this generation was adaptive Vsync and no longer having to have FPS capped to 30 when running Vsync.

I'm not sure how frame rate limiting works on AMD cards, but if it locks the FPS to anywhere below 45, then that's not good enough for me because I see stutter up to around 40 fps. I remember back when I had SLI I thought that the stutter I was seeing around 30-40 FPS was due to microstutter, but its still the same on a single 680 when I force my FPS to 35 for video capture..

And the prices on GTX 680s are still ridiculous, IMO if you need more than one then just get a Titan.

[SKYMTL]

very damning. in bf3 at least, crossfire is garbage. the next time someone suggests crossfire, show them how it skips down to non-crossfire performance, making it pointless: http://www.pcper.com/image/view/2096...n=node%2F56621

You are reading the chart completely wrong.

The variation you see on the Crossfire solution is ALL below 20ms, making it literally imperceptible to the end user.

Honestly, without perspective people can take these charts and have them completely misrepresent the information being provided. A card can do the bloody hokey pokey all over the place below 20ms and it wouldn't make one lick of difference since the actual REAL WORLD experience would still read as smooth.

[NEOAethyr]

20ms is slightly noticeable...
I know this from working with directsound buffers.
But forget about all that.

1000ms in a sec / 20ms of lag = 50fps max...
Does this 20ms include the monitor lag?, I don't know I should probably check the link lol.
I'm guessing it does.

Lets pretend for a sec here.
2x sli or cfx = 20ms of lag.
4x sli or cfx = 40ms of lag, or 25fps max per sec...
That's not good lol lmao...

I notice a little bit of "input" lag on a single monitor in farcry3 and dead island (dead island is way bad sometimes when driving, probably not the video card, probably my usb or something).

[boxleitnerb]

SKYMTL,
do you have actually tried SLI/CF for gaming, not just for benchmarking? These variations can and do affect playability depending on how perceptive you are. How can you make your absolute statements for all people?

[SKYMTL]

SKYMTL,
do you have actually tried SLI/CF for gaming, not just for benchmarking? These variations can and do affect playability depending on how perceptive you are. How can you make your absolute statements for all people?

Again, you are making a blanket statement.

Do rapid fluctuations affect gaming? YES. However, the microstutter you see is likely NOT below 20ms but rather higher than that. Typically when deltas of 10ms or more quickly fluctuate at 25ms and above, you'll see microstutter. Anything below 20ms and you'd either completely miss it or be VERY picky and go looking for it.

Let me reiterate:

- Below 20ms: unnoticeable
- Rapid, cyclical fluctuations above 25ms: you'll notice it
- Sudden "spikes" (non-cyclical) above 40ms: you'll notice it
- Grey area between 21ms and 25ms: debatable. Some will notice it while others won't.

[boxleitnerb]

I said "depending on how perceptive you are". So much for blanket statement...

The rest of your post is a blanket statement. You don't speak for everyone and you cannot possibly define what is generally noticeable and what is not.

[t800]

If enabling normal vsync, does it totally eliminate stuttering in CF/SLI regardless of fps/hz?

[Mungri]

I've tried plenty of Xfire / SLI setups before swapping to a single GK104, I noticed no difference in smoothness with the same FPS in any games.

Any game running at 30-40 FPS was still stuttering to me with both a 560 Ti SLI and single GK104.

Getting the FPS up to 45 made the games smooth in both setups. It terms of smoothness and stutter between SLI and single GPU setups, FPS was the only limiting factor to me.

The only issue I had with SLI / Xfire was that none of the MMOs I played supported it. DDO maxed out with 8x AA was dipping to around 30 FPS on SLI 560 tis, and the same with a single 560 ti. I had the exact same happening currently in Path of Exile, the 560 Ti config with one or two cards could not stay above 30 FPS because SLI wasnt working (I still have my 560 Tis and was using them when my 670 broke). On the other hand the 670 / 680 cards bring me up to 60-100+ FPS in those games at maxed settings, so for this reason I'm not bothering with SLI or Xfire again.

The way I noticed low FPS stuttering on my 680 was by using video capture locked to 35 FPS. I then tried 40 FPS and while a bit better, it was still stuttering. Locking it to 45 FPS made the stutter go away, but it was unrecordable without frame skipping too much over 35 FPS. Exactly the same results as my past SLI / Xfire setups.

[SKYMTL]

You don't speak for everyone and you cannot possibly define what is generally noticeable and what is not.

It isn't me who is defining "noticeable". Read any article or scientific paper about perceptive latency and you'll likely come to the same conclusion.

When I make a statement, don't think for a second that I haven't researched it to an extensive degree.

Links to some of said papers:

http://www.stuartcheshire.org/papers/LatencyQuest.html

http://www.ncbi.nlm.nih.gov/pubmed/16639613 (link to PDF @ source)

http://www.perceptionweb.com/abstract.cgi?id=p240749

If you have an account @ Sciverse / Science Direct, there are several other articles in their archives which discuss the same thing. Unfortunately, I can't post links to them due to copyright terms but you'll quickly get the idea after Googling a bit.

However, I will distill it down into plain speak: In extensive lab tests with and without human subjects, a time of 48 frames per second (ie: 20.83ms) was deemed to be the threshold at which the majority of people anbd test instruments saw completely smooth display images or refreshes.

One test actually went so far as to rapidly flash a lightbulb at variable speeds in order to demonstrate when people would see it "flicker". That threshold for 95% of the participants was at 50 flashes per second (ie: exactly 20ms).

Not sure how much more information anyone could be looking for here....

[Mungri]

That explains why anything below 45 FPS was stuttering to me.

[SKYMTL]

That explains why anything below 45 FPS was stuttering to me.

Precisely.

In addition, I think there has to be a distinction made between the continual stutter typically seen in multi GPU setups and individual frame stutter.

Continual stutter (also called micro stutter) is cyclical in nature and continues throughout a given sequence or section of a game, often repeating itself at regular intervals. For example, it would look something like this on a frame by frame basis: 10ms, 35ms, 10ms, 35ms, 10ms and so on. You would "see" the stutter every time a frame hit the 35ms mark. It would be seen like this:



Individual frame stutter meanwhile would point to instances where certain frames take an abnormally long time to render. It would look something like this: 10ms, 15ms, 12ms, 100ms, 10ms, 15ms, 11ms, 20ms, 18ms, 70ms. Here's an example of that:



Basically these many not be two completely separate phenomenons but they will read differently when viewed onscreen.

[BababooeyHTJ]

I don't agree with your claims SkyMTL. If microstutter below 20ms was unnoticeable as you claim how would people even benefit from a 120hz display? By those standards the difference between 60 and 90hz shouldn't be noticeable either.

[SKYMTL]

I don't agree with your claims SkyMTL. If microstutter below 20ms was unnoticeable as you claim how would people even benefit from a 120hz display? By those standards the difference between 60 and 90hz shouldn't be noticeable either.

You are confusing onscreen refresh rate and response times with processed frame times.

For example, the move from 60Hz to 120Hz isn't about eliminating flickering. Do you see less flicker on a 120Hz screen? Of course not. Rather it is about being able to drive a display ABOVE the 60 FPS mark with v-sync on. That leads to less input latency.

In addition, people don't buy 120Hz screens to eliminate microstutter. Indeed, a 120Hz screen won't do anything to eliminate microstutter if it is already being displayed due to above-20ms frame times.

Now, before you go and claim that's the same thing, it isn't.

First and foremost, quicker refresh rates all but eliminate the ghosting and input latency that occurs on lower-frequency LCD displays. Neither of those things is tied to GPU microstutter.

With that in mind, the difference between 90Hz and 60Hz will be noticeable on the INPUT side rather than on the overly VISUAL side (other than the ghosting of course).

Finally, they aren't claims. Perceptual latency detection above and below 48 FPS has a ton of science backing it up, much of which was sponsored by major production studios.

[boxleitnerb]

It isn't me who is defining "noticeable". Read any article or scientific paper about perceptive latency and you'll likely come to the same conclusion.

When I make a statement, don't think for a second that I haven't researched it to an extensive degree.

Links to some of said papers:

http://www.stuartcheshire.org/papers/LatencyQuest.html

http://www.ncbi.nlm.nih.gov/pubmed/16639613 (link to PDF @ source)

http://www.perceptionweb.com/abstract.cgi?id=p240749

If you have an account @ Sciverse / Science Direct, there are several other articles in their archives which discuss the same thing. Unfortunately, I can't post links to them due to copyright terms but you'll quickly get the idea after Googling a bit.

However, I will distill it down into plain speak: In extensive lab tests with and without human subjects, a time of 48 frames per second (ie: 20.83ms) was deemed to be the threshold at which the majority of people anbd test instruments saw completely smooth display images or refreshes.

One test actually went so far as to rapidly flash a lightbulb at variable speeds in order to demonstrate when people would see it "flicker". That threshold for 95% of the participants was at 50 flashes per second (ie: exactly 20ms).

Not sure how much more information anyone could be looking for here....

First, you absolutely cannot compare gameplay with just displayed sequences. 48 fps, is way too low for many games, think input lag. In some game engines, 100+ fps are required for truly direct control. Secondly, it is unimportant what the "majority" feels as long as there are still people who have higher standards.
So did you actually experience SLI/CF first hand or did you not? Theory has its place, but real life can be something different altogether. This smells like sugarcoating the problems that CF has. I'm always highly suspicious if some people - especially reviewers who should be objective and very careful with such stuff - tell me what I should or shouldn't be able to feel.

[Rollo]

Sheesh. I spent the day with my finger hovering over the "buy" button on a second 7970 so I could look into this.

I have my doubts that a 7970 CF truly feels no faster than a single, and I'd like to put it to the test.

Not to mention I want Crysis 3 and Bioshock Infinite more than any games out right now. I've been having a lot of fun playing Sleeping Dogs and Far Cry3 from the first bundle, AMD is really hitting on all cylinders with their games included.