GPU Benchmarking Methods Investigated: Fact vs. Fiction

Quote:

---

Benchmarks. Every website worth their marbles uses them to varying degrees of accuracy. Meanwhile, every reader wants to recreate them in some way, shape or form in order to do exactly what their favorite publications are doing: to evaluate the performance of their hardware choices and quantify their purchase. Benchmarks can also help diagnose a problem but more often than not websites like Hardware Canucks use these tools to determine how well a given product performs against the competition. As with all things, the number of programs we can attain results with is nearly infinite but it is the job of publications to choose the right set of tools which will accurately convey results to the masses. Unfortunately, as we will show you in this article choosing the right programs and sequences is extremely hard and most of the current methods are inaccurate.

The reason why we have chosen to focus on GPU benchmarking is because this really is the wild-west of the online review industry. A fortune in terms of traffic can be had if GPU reviews are published regularly but with potential traffic increases comes the risk of cutting corners in order to complete the time-consuming benchmarking portion as quickly as possible. Naturally, some time-cutting methods will still produce accurate results while others won’t.

In a general canvassing of over two dozen English-speaking tech websites we found a wide swath of benchmarks being used; from timedemos to stand-alone programs to in-game benchmarks to walkthroughs. What we also saw at times was a general lack of information beyond a game’s title regarding the actual type of benchmark used. For the most part it seemed many websites were using in-game benchmarking tools (mostly “rolling” demos) instead of actual gameplay and coming up with some interesting results. This along with comments in several forums got us wondering: is there a “right” way to benchmark a particular game? In addition, do these in-game or stand-alone benchmarking programs –like the recently released AvP DX11 test- represent in-game performance? If not, do they even provide an accurate enough analysis for a writer to formulate a conclusion about a given product? Well, we’re about to find out.

---

http://www.hardwarecanucks.com/forum...s-fiction.html

Good article! :clap::up:

I've always preferred a gaming run rather than a timedemo. The results are pretty consistent if the reviewer uses the same route and go gun blazing. An experience of mine is that despite the 100 fps+ source engine benchmarks, I'm still getting 20 fps min fps on some very heavy on the graphics card/CPU area :rofl:. Single threaded game, can't do much about that huh. Plus, we can hear the reviewer's touch of his gaming experience.

EDIT: http://www.hardwarecanucks.com/forum...fiction-7.html

L4d2 and FC2 is swapped.

Timedemoes are pretty accurate.

From QUAKE to Counter-Strike: Source to FEAR.

It depends on the game.

Quote:

---

DiRT 2

DiRT 2 is on of the only games we have in this article which uses a benchmarking sequence depecting actual gameplay.

---

I see a typo :p

nice article :)

Very cool article :up:

I've always liked hwc reviews, they have the nice clean straightforward layout when it comes to graphics and figures :)

We never run gaming "benchmarks" its more effective and realistic if you just run the actual game in a certain level and just take the statistics from that run rather than running a canned benchmark. The only reason we even run benchmarks that are usually considered synthetic is only to allow consumers to quantitatively compare against other synthetic benchmarks but in no way accurately reflects real world performance. My point of view is that real world performance results are more important than any 3DMark score when it comes to a product review.

Quote:

---

Originally Posted by Russian

My point of view is that real world performance results are more important than any 3DMark score when it comes to a product review.

---

:up:

i always think gpus had to have 2 reviews one for normal customers made by the normal reviewers and a overclocking review for oc community made by hardcore overclocklers

What's wrong with reviews that cover both aspects?

Quote:

---

Originally Posted by miahallen

What's wrong with reviews that cover both aspects?

---

there is nothing wrong but not enough for ocers. most read them but the final decision always giving after reading oc forums like here.

Quote:

---

Originally Posted by Russian

We never run gaming "benchmarks" its more effective and realistic if you just run the actual game in a certain level and just take the statistics from that run rather than running a canned benchmark. The only reason we even run benchmarks that are usually considered synthetic is only to allow consumers to quantitatively compare against other synthetic benchmarks but in no way accurately reflects real world performance. My point of view is that real world performance results are more important than any 3DMark score when it comes to a product review.

---

I agree. One of the main problems is consistency when it comes to actual gameplay versus some of the built-in / stand along rolling benchmarks. The vast majority of the time they do give reviewers a quick way to benchmark a game but for the most part they are inaccurate.

I have also seen talk (and I'm not sure where it stems from) that timedemos don't allow for accurate results. There weren't too many games to test this out with but from what I have seen, timedemos are actually an excellent way to benchmark.

It all comes down to knowledge of a game. Just loading up a built-in benchmark without know if it is accurate or not is a surefire way to come up with the wrong conclusion.

Whoa, doesn't that mean HardOCP have been doing it right all along????

Excellent article :)

Quote:

---

Originally Posted by SKYMTL

I agree. One of the main problems is consistency when it comes to actual gameplay versus some of the built-in / stand along rolling benchmarks. The vast majority of the time they do give reviewers a quick way to benchmark a game but for the most part they are inaccurate.

I have also seen talk (and I'm not sure where it stems from) that timedemos don't allow for accurate results. There weren't too many games to test this out with but from what I have seen, timedemos are actually an excellent way to benchmark.

It all comes down to knowledge of a game. Just loading up a built-in benchmark without know if it is accurate or not is a surefire way to come up with the wrong conclusion.

---

It comes from about a decade ago when nvidia and ati would optimize their drivers so that it performs better in a specific time demo. This was back when Quake 3 Arena was the standard bench for every review on every site. If you have a repeatable pattern then it will be exploited. This prompted tech-report to start recording their own time demos and [H] to make elaborate analysis.

Overall most reviews are pretty bad and lack true scientific analysis but most people don't really understand or care.

Quote:

---

Originally Posted by cegras

Whoa, doesn't that mean HardOCP have been doing it right all along????

---

I don't like how they prioritize resolution than AA. Not everyone here has a 2560x1600. HardOCP might have been using real gameplay, that's nice, but I still don't like their best settings methodology.

Excellent article.

Conclusion: the 5850 is still the king price/perf, perf/watt, heat & noise.

im pretty sure older cards are still better price/perf kings, considering a 4850 is 100$, nearly 1/3 the price, but i doubt its 3x slower

thank you SKYMTL for trying to introduce reason into this situation.

it is clear that some software doesn't represent interesting results. interesting results? results that reflect what people buy the hardware for. game performance, computation performance, ... but showing off with benchmarks doesn't really count. people only care about showing off benchmarks because they think they represent the games. they are devalued if they dont represent games.

here is a sample of the not-interesting results:
"While the “Dark Tower” benchmark does tend to almost come close to the overall results from our in-game testing, it still doesn’t represent real-world performance"
"In the game itself, there is a wide gap between the NVIDIA and ATI cards while the benchmarks result in very similar performance between the two solutions."

and of course just running through the game manually introduces error. but the timedemos were interesting, and you said this:
"One of the main issues we have with timedemos is that so few games actually support them."

so clearly it is a minefield of USELESS and USEFUL software. remember "just because you CAN, doesn't mean you SHOULD". that goes for benchmarks.

someone with a brain needs to filter out the useless software. so they can review hardware well.

SKYMTL to the rescue:

Quote:

---

However, there are currently a small number of games like DiRT 2 and HawX which incorporate benchmark sequences that accurately recreate in-game scenarios.

For the most part we have seen accurate results when timedemos are compared to in-game sequences

it is imperative publications state exactly what tools they are using for their benchmarks.

When we choose to add a game to our stable of benchmarkable titles, the first thing that’s done is a full play-through in order to determine what typical performance is like. With the help of FRAPS, a “worst case” scenario is derived from this playthrough which we will use for the basis of our benchmark run or to help determine if a built-in benchmark holds any foundation in reality.

you will find videos of our benchmark sequences (if applicable) as well as save game files, timedemos, graphics setting screenshots and even guides to show you how to set everything up.

---

thank you SKYMTL for trying to innovate and produce quality work. I wish you luck in the future as you continue to iterate through improvements of your benchmark methodology.

Thanks Mike ! Really good work :up:

Well done to Hardware Canucks for investing all this time and producing a very well written and interesting article. Cheers guys! Would rep you if we had such a system

Quote:

---

Originally Posted by kromosto

:up:

i always think gpus had to have 2 reviews one for normal customers made by the normal reviewers and a overclocking review for oc community made by hardcore overclocklers

---

Absolutely, same goes for games. I like the performance review Guru3D was doing for games (hint, hint do some more) as an alternative to actually reviewing a game.

This review of reviewing methods is important as there are regular instances of disagreements between some sites and other and it's only those who know what to look for who will see the difference.

I prefer time demo's because of their accuracy but looking at the results, on most of them the benchmark is lower than what you get in gameplay which could mislead people to thinking that a game might not play on a certain gfx card

In theory, I like Kyle's methods too, but these days few games can challenge the high end cards enough to drop down below 30" monitor resolutions. So they are mainly useful to owners of 30" monitors. And that's the problem with that method. It is too dependent on what size monitor you happen to have. My monitor is 1600x1200 and that res almost never makes it into benchmarks these days.

Quote:

---

Originally Posted by cegras

Whoa, doesn't that mean HardOCP have been doing it right all along????

---

No because they compare cards using different ingame settings.

The ironic thing here is that this article is perpetuating one of the most common benchmarking mistakes of today: providing minimum frame rates without qualifying them. Minimum FPS by itself is worthless, since for all you know it be for a single frame at the start of the level, or conversely that card might be hitting that minimum frame-rate all of that time. Another example, if one card hits a very low minimum frame rate once for a very short period, and another card hits a higher minimum frame rate but goes there more often, it's the first card with the lower min fps that is providing the better game play experience. If you want to provide minimum frame-rates, you MUST qualify them with a graph of fps over time, or at the very least a description of the gameplay. Unfortunately this poor methodology is very widespread.

Quote:

---

Originally Posted by cegras

Whoa, doesn't that mean HardOCP have been doing it right all along????

---

Kinda, except for with their reviews... they compare different cards with different settings enabled and usually include statements about the quality of the game difference with or without those settings enabled. In many cases, a faster card will have higher AA settings enabled and they usually try to peg the FPS as closely to each other. I'd rather just crank the settings identically on both cards and compare the performance deltas rather than to try to quantify the different AA settings.

Quote:

---

Originally Posted by hurleybird

The ironic thing here is that this article is perpetuating one of the most common benchmarking mistakes of today: providing minimum frame rates without qualifying them. Minimum FPS by itself is worthless, since for all you know it be for a single frame at the start of the level, or conversely that card might be hitting that minimum frame-rate all of that time. Another example, if one card hits a very low minimum frame rate once for a very short period, and another card hits a higher minimum frame rate but goes there more often, it's the first card with the lower min fps that is providing the better game play experience. If you want to provide minimum frame-rates, you MUST qualify them with a graph of fps over time, or at the very least a description of the gameplay. Unfortunately this poor methodology is very widespread.

---

Thats why you have the average frame rate ;)
the closer the min is to the average, the more often that card hits the low-end. I find min to be fairly useful in determining which cards will deliver smoother gameplay. Its not accurate all the time, but its good for when you want to breeze through the pages

Quote:

---

Originally Posted by Russian

Kinda, except for with their reviews... they compare different cards with different settings enabled and usually include statements about the quality of the game difference with or without those settings enabled. In many cases, a faster card will have higher AA settings enabled and they usually try to peg the FPS as closely to each other. I'd rather just crank the settings identically on both cards and compare the performance deltas rather than to try to quantify the different AA settings.

---

then go to every other site out there, let them be unique and have a niche.

there are many ways to do things, and there are also many ways that dont have a right/wrong method. putting cards to the best playable setting is really good since i dont care if 480 or 5870 get 200 to 250frps, since thats not the setting im going to use

but u have to be careful, if they put on super high res textures that kill 512MB cards, then when u look at once that does 0xaa and 0xaf, the other does 4xAA, and 16xAF, u think one is 10x stronger. but drop it down to just high res textures, and both can do 4/16, and its much closer in battle, and you only have to question if u want the utlra textures or not, instead of the impression of one being crap.

so what i like in a review:
especially for new architectures, test how a card is impacted with increase to AA, AF, textures, LOD, resolution, and maybe the few random effects like, water or shadows, but not primarlily.
then using that data u can assume how it handles each major performance to determine what becomes a quick bottleneck, and your best playable settings can be justified. for some people they must be able to game at 1080p, others dont have such a high res and must have 4xAA minimum, while there is always some give and take, the 2900 was a great example where resolution wasnt a problem, but AA was.
then do the standard tests for whats a best playable settings comparison,
do a decent overclocking comparison,
then the usual, power consumption, temps, overclocking

Quote:

---

Originally Posted by cegras

Whoa, doesn't that mean HardOCP have been doing it right all along????

---

Like I said in the conclusion: there is no "right" way to go about it. Kyle's team has a unique persepctive and it sets them (and sometimes their conlcusions) apart from the norm. They do use in-game sequences and research things properly it seems which is a huge step in the right direction. However, I am not actually sure if they include action sequences in their benchmarks or if they are just doing a run-through.

What the article is really meant to convey is that the vast majority of standard benchmarking methods (built in benchmarks & stand-alone) are dead wrong. This is why readers should push for a transparent benchmarking process where there is disclosure of exactly which methods were used.

Quote:

---

Originally Posted by hurleybird

The ironic thing here is that this article is perpetuating one of the most common benchmarking mistakes of today: providing minimum frame rates without qualifying them. Minimum FPS by itself is worthless, since for all you know it be for a single frame at the start of the level, or conversely that card might be hitting that minimum frame-rate all of that time. Another example, if one card hits a very low minimum frame rate once for a very short period, and another card hits a higher minimum frame rate but goes there more often, it's the first card with the lower min fps that is providing the better game play experience. If you want to provide minimum frame-rates, you MUST qualify them with a graph of fps over time, or at the very least a description of the gameplay. Unfortunately this poor methodology is very widespread.

---

I can't speak for other sites but the fact that we test every run three times and average out the results eliminates any "zingers" when it comes to minimum framerates. Average FPS are also there for a reason. :up:

Quote:

---

Originally Posted by ElSel10

No because they compare cards using different ingame settings.

---

This.

Quote:

---

Originally Posted by Aleki

Thats why you have the average frame rate ;)
the closer the min is to the average, the more often that card hits the low-end.

---

Not necessarily. If one card has a memory deficiency it could hit very low minimums far more often than another card that hits higher minimums less often. You can't tell for sure without looking a graph of FPS/time. Again, the minimum by itself is vague and misleading.

Quote:

---

Originally Posted by SKYMTL

I can't speak for other sites but the fact that we test every run three times and average out the results eliminates any "zingers" when it comes to minimum framerates. Average FPS are also there for a reason. :up:

---

Yes, obviously no one would read your reviews if you didn't have average FPS ;)

As far as averaging results, that may or may not make a difference depending on how that minimum is reached. If it is reached in a cut scene or during the loading of a scene or level, you can still have a very low min FPS that is reached only once and does not have any meaningful impact on game-play. On the other hand many people would interpret those numbers as if there was a meaningful impact in game-play experience. That's sloppy. Look at Heaven 2.0, as far as I can tell running on my 5870 it reaches it's min frame rate loading between two scenes for such a short time that its not really noticeable if you aren't looking at the FPS as it happens. However, for the most part frame-rate stays stable... I would say more so than in Heaven 1.0 -- yet how often do you see mention of Fermi cards "doubling" the min FPS of the Radeons in Heaven? It's obvious that there is a problem with methodology here. If you can't qualify a vague and easily misinterpreted number, you shouldn't be using it, period. Do it *right* or don't do it.

You can't really equate Heaven 2.0's shortcomings with potential in-game issues. In the end, I would rather have an idea of minimums rather than none at all.

Quote:

---

Originally Posted by SKYMTL

You can't really equate Heaven 2.0's shortcomings with potential in-game issues. In the end, I would rather have an idea of minimums rather than none at all.

---

i wouldnt, its way to misleading if its based only on the hard-drive, and theres no way to know that unless we have a frequency

what could help is standard deviations. basically if you get 60fps average, with 5fps as your standard deviation, then most frames are between 55-65, and a few 50-70, and extremely few above or below that.

or just a percentage within 10%, 20% and 50%

Quote:

---

Originally Posted by SKYMTL

You can't really equate Heaven 2.0's shortcomings with potential in-game issues. In the end, I would rather have an idea of minimums rather than none at all.

---

Here's the thing... making a graph of FPS over time might be difficult, I don't know... I know [H] is able to, although there are parts of their methodology which a lot of people including myself don't care for. However, the least that one can do if they want to use a metric like minimum FPS is to qualify it with a description of game-play experience, and the same goes for dual GPU options that might have micro-stuttering. Saying something like "we recorded a low minimum frame-rate, but we didn't experience any choppiness in-game," or "minimum frame rates were low, and it felt like we were hitting them often in scenes with a lot of action" would go a long way. As it is right now, a low minimum could mean everything or nothing, leaving that up to the user to interpret muddies the review, and in that case they *are* better left out. I think I sent you a PM earlier about this, but a far better metric, if it could be extracted, would be general frame stability via std deviation (EDIT: thanks poster above me!), or even minimums and maximums (and actually high maximums are a BAD thing because that means the frame-rate is less stable, but like minimums it can mean everything or nothing depending on how often it occurs) qualified by std deviation.

You make very good points and I'm actually quite happy this came up since some of your suggestions would add a new dimension to the reviews. This is especially true considering we are advocating using actual gameplay sequences which lends itself to a more accurate representaion of minimum framerates over time.

However, I do take issue to the statement that the article advocates an incorrect benchmarking process because I don't feel that it does. A follow-up article will be going into the uses of minimum framerates / averages but this one was really concentrating upon the methods to achieve results rather than how the results are shown and communicated in graph form. ;)

Quote:

---

Originally Posted by SKYMTL

You make very good points and I'm actually quite happy this came up since some of your suggestions would add a new dimension to the reviews. This is especially true considering we are advocating using actual gameplay sequences which lends itself to a more accurate representaion of minimum framerates over time.

However, I do take issue to the statement that the article advocates an incorrect benchmarking process because I don't feel that it does. A follow-up article will be going into the uses of minimum framerates / averages but this one was really concentrating upon the methods to achieve results rather than how the results are shown and communicated in graph form. ;)

---

:up:

Sounds good, can't wait for the next article.

there is room to talk here, but I want you to decide on your own how you want to express performance http://en.wikipedia.org/wiki/Three_sigma_rule

Quote:

---

Originally Posted by bamtan2

there is room to talk here, but I want you to decide on your own how you want to express performance http://en.wikipedia.org/wiki/Three_sigma_rule

---

Is this in reference to using multiple benchmark runs to create an average result?

Quote:

---

Originally Posted by STEvil

Is this in reference to using multiple benchmark runs to create an average result?

---

I think so. However, this early in the morning my math cap doesn't seem to sit properly on my head....

Actually, SKY, I felt that a lot of benchmarks never told the real story. Most glaringly was with games like TF2, that can be benched in a siiiimilar way through HL2:Ep 2 benches and L4D / L4D2 benches. I found that a 4850 really is inadequate, even at 1440 x 900.

I actually sat down and read through your article, it was quite good. Not much to say except I always felt that there was a lot of statistical analysis missing from reviews.

Quote:

---

Originally Posted by Manicdan

i wouldnt, its way to misleading if its based only on the hard-drive, and theres no way to know that unless we have a frequency

what could help is standard deviations. basically if you get 60fps average, with 5fps as your standard deviation, then most frames are between 55-65, and a few 50-70, and extremely few above or below that.

or just a percentage within 10%, 20% and 50%

---

I've been wondering why reviews haven't been using the standard deviation.

Quote:

---

Originally Posted by iMacmatician

I've been wondering why reviews haven't been using the standard deviation.

---

i think many programs are overly simplistic, they just take the last framerate, and add it to an average framerate. i dont care to get into the exact math, but the idea is that its able to give you an average, without actually having to save off any data other than current average, and number of frames. however due to the fact every game is probably not using all of your cpu, i dont think its hard to give good quality benchmarkes like the one in FC2, since i doubt its very taxing at all.

Quote:

---

Originally Posted by STEvil

Is this in reference to using multiple benchmark runs to create an average result?

---

na. standard deviation is used to express how far away from the mean a certain amount of the sample set is. you can change that certain amount by changing the number of standard deviations you count away from the mean.

so if you were concerned about the minimum framerate value expressing just some odd one-off circumstance, like some posters are, you could use the mean framerate minus three standard deviations instead. this would leave off the most extreme minimums while still capturing 99% of the other values.

you could even do this with only the values below the original mean, which would leave off all the high values you dont care about. standard deviation is normally used with "normal distributions" and using it with whatever a benchmark turns out to be may not produce results we could understand or predict right now

just think about the requirement (expressing useful information to potential video card buyers) and find a math tool from the massive library of math tools to manipulate your data set (ideally instantaneous frame rates like from fraps, because using pre-calculated fps is like cooking with flavored sardines) into what will help you

Quote:

---

Originally Posted by gojirasan

In theory, I like Kyle's methods too, but these days few games can challenge the high end cards enough to drop down below 30" monitor resolutions. So they are mainly useful to owners of 30" monitors. And that's the problem with that method. It is too dependent on what size monitor you happen to have. My monitor is 1600x1200 and that res almost never makes it into benchmarks these days.

---

Yeah, I play at 1080p or 1600x1200, but there are plenty of games that can slow down at these resolutions.

Quote:

---

Originally Posted by SKYMTL

Like I said in the conclusion: there is no "right" way to go about it. Kyle's team has a unique persepctive and it sets them (and sometimes their conlcusions) apart from the norm. They do use in-game sequences and research things properly it seems which is a huge step in the right direction. However, I am not actually sure if they include action sequences in their benchmarks or if they are just doing a run-through.

What the article is really meant to convey is that the vast majority of standard benchmarking methods (built in benchmarks & stand-alone) are dead wrong. This is why readers should push for a transparent benchmarking process where there is disclosure of exactly which methods were used.



I can't speak for other sites but the fact that we test every run three times and average out the results eliminates any "zingers" when it comes to minimum framerates. Average FPS are also there for a reason. :up:

---

Great article, thank you.

I agree with your post above, when reading reviews I personally weigh each review to peace an over-all picture of performance. But, I place more value on histograms, because visually that explains more information to me.

Max FPS has never (ever) been a criteria in which I judge, it has always been sustained minimal frames that I am most interested in avoiding.

Quote:

---

Originally Posted by bamtan2

na. standard deviation is used to express how far away from the mean a certain amount of the sample set is. you can change that certain amount by changing the number of standard deviations you count away from the mean.

so if you were concerned about the minimum framerate value expressing just some odd one-off circumstance, like some posters are, you could use the mean framerate minus three standard deviations instead. this would leave off the most extreme minimums while still capturing 99% of the other values.

you could even do this with only the values below the original mean, which would leave off all the high values you dont care about. standard deviation is normally used with "normal distributions" and using it with whatever a benchmark turns out to be may not produce results we could understand or predict right now

just think about the requirement (expressing useful information to potential video card buyers) and find a math tool from the massive library of math tools to manipulate your data set (ideally instantaneous frame rates like from fraps, because using pre-calculated fps is like cooking with flavored sardines) into what will help you

---

ah, makes more sense... also sounds like something somewhat would use to predict performance on a part before its mature for retail.... ;)

SKYMTL: once you do the leg work to see how the products from each company compare in each benchmark (which you've done), you can express the results with just a couple benchmarks. so for instance, if you want to compare some cards with the new drivers, you may only need the far cry 2 benchmark (which favors nvidia the most) and the dirt2 benchmark (which favors nvidia the least, if my guess is right). between the two benchmarks, the reader should have an idea of the worst case they'll be dealing with, whichever card they're looking at. the more benchmarks you add past that, the less useful each benchmark is, and the more time you've wasted.

there are two things the potential buyer should be looking for. (1) some general expression of performance to represent all games, present and future. (2) pretty specific performance information about the exact game(s) they play. you can hit #1 with probably two benchmarks. hitting #2 requires exhaustive coverage of benchmarks, resolutions, system configurations... so maybe 5x-10x the work for not much more usefulness.

Quote:

---

Originally Posted by bamtan2

SKYMTL: once you do the leg work to see how the products from each company compare in each benchmark (which you've done), you can express the results with just a couple benchmarks. so for instance, if you want to compare some cards with the new drivers, you may only need the far cry 2 benchmark (which favors nvidia the most) and the dirt2 benchmark (which favors nvidia the least, if my guess is right). between the two benchmarks, the reader should have an idea of the worst case they'll be dealing with, whichever card they're looking at. the more benchmarks you add past that, the less useful each benchmark is, and the more time you've wasted.

there are two things the potential buyer should be looking for. (1) some general expression of performance to represent all games, present and future. (2) pretty specific performance information about the exact game(s) they play. you can hit #1 with probably two benchmarks. hitting #2 requires exhaustive coverage of benchmarks, resolutions, system configurations... so maybe 5x-10x the work for not much more usefulness.

---

dirt2 likes nvidia cards too, cause of the extra tessellation power

bamtan: Yeah, but to get an accurate average you need at least 5 runs. And that's just for a making sure you have a good linear fit. The sample size for statistically relevant average + 3 SD rule would be LARGE.

Extrapolating from 3 runs gives you no useful information at all.

We do three runs and as it is, benching each card takes about 5 hours straight. Multiply that by five or six cards and things start getting a bit nuts when upping that to 6+ runs. I know I said it shouldn't come down to time but there comes a situation where so much time is invested redoing benchmarks when drivers are released that no reviews are actually posted...

Quote:

---

Originally Posted by iMacmatician

I've been wondering why reviews haven't been using the standard deviation.

---

Cos then we'd ask for confidence intervals and more hard stats.

Quote:

---

Originally Posted by cegras

bamtan: Yeah, but to get an accurate average you need at least 5 runs. And that's just for a making sure you have a good linear fit. The sample size for statistically relevant average + 3 SD rule would be LARGE.

Extrapolating from 3 runs gives you no useful information at all.

---

shrug, I don't think I mentioned how many runs should be done

but since you brought it up, I disagree with you because I don't think computer game benchmarks vary enough that "3 runs gives you no useful information at all"

I'm trying to help because every video card review on the web is at a grade 6 mathematics level. you are too, but you just landed a bridge too far

Quote:

---

Originally Posted by SKYMTL

Like I said in the conclusion: there is no "right" way to go about it. Kyle's team has a unique persepctive and it sets them (and sometimes their conlcusions) apart from the norm. They do use in-game sequences and research things properly it seems which is a huge step in the right direction. However, I am not actually sure if they include action sequences in their benchmarks or if they are just doing a run-through.

What the article is really meant to convey is that the vast majority of standard benchmarking methods (built in benchmarks & stand-alone) are dead wrong. This is why readers should push for a transparent benchmarking process where there is disclosure of exactly which methods were used.



I can't speak for other sites but the fact that we test every run three times and average out the results eliminates any "zingers" when it comes to minimum framerates. Average FPS are also there for a reason. :up:

---

I fully plan to include my testing methodology whenever i start up a new review... explaining the software and the environments used. Otherwise, i'm just putting the cloth over my readers' eyes. :p:

My point of view is that games are very finnicky and the most important thing is not scores or anything like that, its all about raw performance in games and the best bang for your bucks as well as the best performance money can buy. I don't care what company is better or worse, I just want to give my readers a clear cut and well informed as well as transparent review.

Quote:

---

Originally Posted by hurleybird

The ironic thing here is that this article is perpetuating one of the most common benchmarking mistakes of today: providing minimum frame rates without qualifying them. Minimum FPS by itself is worthless, since for all you know it be for a single frame at the start of the level, or conversely that card might be hitting that minimum frame-rate all of that time. Another example, if one card hits a very low minimum frame rate once for a very short period, and another card hits a higher minimum frame rate but goes there more often, it's the first card with the lower min fps that is providing the better game play experience. If you want to provide minimum frame-rates, you MUST qualify them with a graph of fps over time, or at the very least a description of the gameplay. Unfortunately this poor methodology is very widespread.

---

I think you are right, the solution by my estimate is to provide data on the MEDIAN of FPS, as this will tell you with average and minimum FPS where the range lies the most.

Quote:

---

Originally Posted by bamtan2

I'm trying to help because every video card review on the web is at a grade 6 mathematics level. you are too, but you just landed a bridge too far

---

just right for many of the readers then. i'm sure less than 1% of readers of a typical vga review understand what standard deviation is, people have difficulties "reading" linear x-y plots

Quote:

---

Originally Posted by W1zzard

just right for many of the readers then. i'm sure less than 1% of readers of a typical vga review understand what standard deviation is, people have difficulties "reading" linear x-y plots

---

+1.

People don't want to sit there for 2 minutes looking at linear graphs and charts. They want to see the result and see it quick. :yepp:

Quote:

---

Originally Posted by Stukov

I think you are right, the solution by my estimate is to provide data on the MEDIAN of FPS, as this will tell you with average and minimum FPS where the range lies the most.

---

You do know what median means right? Middle. Median of what? Median of FPS? What? The Mode is the most. :shrug:

Quote:

---

Originally Posted by BeepBeep2

+1.

People don't want to sit there for 2 minutes looking at linear graphs and charts. They want to see the result and see it quick. :yepp:



You do know what median means right? Middle. Median of what? Median of FPS? What? The Mode is the most. :shrug:

---

http://en.wikipedia.org/wiki/Median

Median is the "middle" of a series of numbers. So if you have 1, 1, 1, 2, 3, 4, 5...2 is the median number. Having the Median number would allow you to know on which side of the average you will get most of the time. Average mean can be heavily skewed by massive jumps in data (or FPS here) or massive drops.

Let's say its 1, 1, 1, 2, 3, 100 the average mean would be 18, however the mean would 1-2 (1.5). In terms of FPS, if you have a lot of an average of 60, a minimum of 20, and a median of 65 will tell you that getting 20 FPS is not very likely, but if the median was 30, it would tell you that you end up lower than 60 quite often, but there are some high peaks skewing it higher.

Quote:

---

Originally Posted by W1zzard

just right for many of the readers then. i'm sure less than 1% of readers of a typical vga review understand what standard deviation is, people have difficulties "reading" linear x-y plots

---

readers don't need to know what it is. it is just a tool to produce a result. like adding and dividing produces average, standard deviation can produce a useful minimum

Quote:

---

Originally Posted by Stukov

http://en.wikipedia.org/wiki/Median

Median is the "middle" of a series of numbers. So if you have 1, 1, 1, 2, 3, 4, 5...2 is the median number. Having the Median number would allow you to know on which side of the average you will get most of the time. Average mean can be heavily skewed by massive jumps in data (or FPS here) or massive drops.

Let's say its 1, 1, 1, 2, 3, 100 the average mean would be 18, however the mean would 1-2 (1.5). In terms of FPS, if you have a lot of an average of 60, a minimum of 20, and a median of 65 will tell you that getting 20 FPS is not very likely, but if the median was 30, it would tell you that you end up lower than 60 quite often, but there are some high peaks skewing it higher.

---

the biggest problem with median is the fluctuation of fps is very noticable. if your talking about mpg, you dont really care if one tank gets 20 and the next gets 30, and then back to 20, but in games thats called micro stuttering, lol, and people really hated that.

i think line graphs is where its at, 100%. anything else takes away from that. sure you can have summaries, but the raw data needs to be fully available for some readers who do take the time to read.

Quote:

---

Originally Posted by Manicdan

im pretty sure older cards are still better price/perf kings, considering a 4850 is 100$, nearly 1/3 the price, but i doubt its 3x slower

---

Yeah but it runs 3x hotter and it's not really faster than even, cheaper, smaller 8800GT.

5850 is the best price/performance card especially when clocked to 1000Mhz core.

Quote:

---

Originally Posted by QuadDamage

Yeah but it runs 3x hotter and it's not really faster than even, cheaper, smaller 8800GT.

5850 is the best price/performance card especially when clocked to 1000Mhz core.

---

when you speak in absolutes it takes only one counterexample to disprove you.

we are way off topic here, but you must be corrected. used cards have huge discounts, making it very difficult for any new card to compete for the price/performance crown.

example: gtx 260 goes for $120 on ebay. 5850 is about $300. 5850 is not near twice as fast as gtx 260.

http://completed.shop.ebay.com/i.htm...c0.m283&_rdc=1

http://www.guru3d.com/article/vga-ch...ecember-2009/4

you could display the spread of framerates like in the 1st attached pic, where the green GPU has higher average and max framerates but doesn't do as well at lower framerates

if you had more FPS ranges on the x axis (eg 30 instead of the 10 in my pic) it could look good as a line graph instead of a bar graph

if you had heaps of FPS ranges on the x axis (eg each fps range was an integer) and just used a colored dot for each value you'd get a cloud of scattered dots that would look good if you had enough data points.

but you'd often want to compare multiple GPUs and multiple resolutions and too many graphs can be a nuisance... but if you made the columns of a multiple GPU/resolution graph into representations of the FPS spread (eg the "red" GPU that i did for example's sake in the 2nd pic, where each horizontal line is perhaps a integer FPS value, and its width represents how much time the GPU spends at that FPS, i didn't bother mspainting the other bars to perfection) you could display the full story in a single JPG

Quote:

---

Originally Posted by W1zzard

just right for many of the readers then. i'm sure less than 1% of readers of a typical vga review understand what standard deviation is, people have difficulties "reading" linear x-y plots

---

implicitly they can understand it. kind of like how you understand physics and use it in every day life even though you dont solve PDE's in your head.

as shown above anyone can see the distribution of framerates and see that card X has more consistent framerates than card Y. if they dont know what to look for tell them or explain it.

btw im not a fan of typical reviews. sites should do something unique, something that gives them individuality and a reason to visit them. as a technical person i would like to see in depth statistical analysis of benchmarks.

http://www.xtremesystems.org/forums/...1&d=1277059023

a good point is made here, which is that just changing the presentation (what Y and X represent) of existing data can change how useful the picture is.

normal reviews have Y=framerates along the left, and X=time along the bottom. but wait a second, who cares about time? we don't even know what the benchmark was doing at any given time.

so rather than a graph presenting what speed things were going at a given time, why not use the exact same data to present how often things were going a given speed?

This here is the good :banana::banana::banana::banana:.

Here's what I do with vbscript based off of samples provided (la as a 2d array) to calculate "average" (median) fps. Comments/Criticism very welcome. :woot:

Code:

---

i="0"
Select Case UBound(la)
Case "1"
        avgfps=la(1)
        WScript.Echo "Average FPS=" & avgfps
Case "2"
        for i = 1 to ubound(la)
                sum=sum + la(i)
        Next
        avgfps=sum/ubound(la)
        WScript.Echo "Average FPS=" & avgfps
Case "3"
        for i = 1 to ubound(la)
                If sDebug=1 Then WScript.Echo "la(" & i & ")=" & la(i)
                If la(i)>la(high) Then high=i
                If la(i)<la(low) Then low=i
        Next
        for i = 1 to ubound(la)
                Select Case i
                Case high
                        If sDebug=1 Then WScript.Echo "la(" & i & ") is high!"        ' throw it away!
                Case low
                        If sDebug=1 Then WScript.Echo "la(" & i & ") is low!"        ' throw it away!
                Case Else
                        avgfps=la(i)
                End Select
        Next
        WScript.Echo "Average FPS=" & avgfps
Case "4"
        for i = 1 to ubound(la)
                If sDebug=1 Then WScript.Echo "la(" & i & ")=" & la(i)
                If la(i)>la(high) Then high=i
                If la(i)<la(low) Then low=i
        Next
        for i = 1 to ubound(la)
                Select Case i
                Case high
                        If sDebug=1 Then WScript.Echo "la(" & i & ") is high!"        ' throw it away!
                Case low
                        If sDebug=1 Then WScript.Echo "la(" & i & ") is low!"        ' throw it away!
                Case Else                       
                        avgfps=avgfps + la(i)
                        If sDebug=1 Then WScript.Echo "la(" & i & ") one of our sweet values!"        ' throw it away!
                End select
        Next
        If low=high Then
                avgfps=avgfps/3
                If sDebug=1 Then WScript.Echo "avgfps=(3)" & avgfps
        Else
                avgfps=avgfps/2
                If sDebug=1 Then WScript.Echo "avgfps(2)=" & avgfps
        End If
        WScript.Echo "Average FPS=" & avgfps
Case "5"
        for i = 1 to ubound(la)
                WScript.Echo "la(" & i & ")=" & la(i)
                If la(i)>la(high) Then high=i
                If la(i)<la(low) Then low=i
        Next
        for i = 1 to ubound(la)
                Select Case i
                Case high
                        WScript.Echo "la(" & i & ") is high!"        ' throw it away!
                Case low
                        WScript.Echo "la(" & i & ") is low!"        ' throw it away!
                Case Else                       
                        avgfps=avgfps + la(i)
                        WScript.Echo "la(" & i & ") one of our sweet values!"        ' throw it away!
                End select
        Next
        If low=high Then
                avgfps=avgfps/4
                WScript.Echo "avgfps=(3)" & avgfps        ' throw it away!
        Else
                avgfps=avgfps/3
                WScript.Echo "avgfps(2)=" & avgfps        ' throw it away!
        End If
        WScript.Echo "Average FPS=" & avgfps
Case Else
                for i = 1 to ubound(la)
                WScript.Echo "la(" & i & ")=" & la(i)
                If la(i)>la(high) Then high=i
                If la(i)<la(low) Then low=i
        Next
        for i = 1 to ubound(la)
                Select Case i
                Case high
                        WScript.Echo "la(" & i & ") is high!"        ' throw it away!
                Case low
                        WScript.Echo "la(" & i & ") is low!"        ' throw it away!
                Case Else                       
                        avgfps=avgfps + la(i)
                        WScript.Echo "la(" & i & ") one of our sweet values!"        ' throw it away!
                End select
        Next
        If low=high Then
                avgfps=avgfps/(ubound(la)-1)
                WScript.Echo "avgfps=(3)" & avgfps        ' throw it away!
        Else
                avgfps=avgfps/(ubound(la)-2)
                WScript.Echo "avgfps(2)=" & avgfps        ' throw it away!
        End If
        WScript.Echo "Average FPS=" & avgfps
End Select

avgfps=Round(avgfps,2)
logfile.WriteLine ("AVERAGE FPS: " & avgfps)

---

Interesting but how would you do that in a program like Excel in a quick and easy manner? I am guessing you could make a Custom List but that would take an insane amount of time. (directed @ Batman)

Also, NVIDIA's newest WHQL drivers pretty much bring my point about in-game benchmarks into pretty sharp focus. They are claiming a massive framerate jump in the Concrete Jungle benchmark. In game? Nadda.

Quote:

---

Originally Posted by W1zzard

just right for many of the readers then. i'm sure less than 1% of readers of a typical vga review understand what standard deviation is, people have difficulties "reading" linear x-y plots

---

Hey W1z; first of all, I'm a HUGE fan of your work. Thank you for all your contributions to the community. Just wanted to say that I think we're selling ourselves short. I would assert that the number is closer to 20%, but we look at it as a much smaller minority because that minority tends not to be very "vocal" in the first place. If someone gets it, they rarely complain; they just consume and move on.

To that end, I think there is an audience for more intelligent data, it's just an audience that is difficult to market to or convince to do much of anything through standard advertising means.

nice article... :toast:

Nice read :up:

One observation though - when it comes to actual performance in a game, there is a difference between stand-alone benchmarks and different portions of the game, thus sustaining your point. However, when it comes to the % diference of performance between 5850 and GTX 470 (your examples) I see almost not percentage difference between the two from one situation to the other.

The point is very simple, based on your tests and results - if real-life performance in a game is what you are after, then in-game testing, even if it is far less accurate then benchmarks due to a large number or variables, is the way to go. However, if accurate performance percentage difference between 1,2,3, etc products is what you are after, benchmarks are in no way less representative then in-game testing.

Quote:

---

Originally Posted by Monstru

The point is very simple, based on your tests and results - if real-life performance in a game is what you are after, then in-game testing, even if it is far less accurate then benchmarks due to a large number or variables, is the way to go. However, if accurate performance percentage difference between 1,2,3, etc products is what you are after, benchmarks are in no way less representative then in-game testing.

---

You can't forget that we're talking about playability as well. Take the AvP benchmark for example. If you were going by the stand-alone test, you would be under the wrong impression that none of the tested cards could actually play the game.

The same could be said about some in-game benchmarks such as the Desert sequence in Just Cause 2. If you went by that, you would once again get the wrong impression since most cards perform amazingly in it while the game itself is more challenging.

I agree with SKYMTL. Conclusion? We need more horsepower. I'm starting to feel the age of my 4890. Damn source games dipping me to below 30 fps sometimes.