How to benchmark the following games:

[ToTTenTranz]

For reviewing purposes, I'm looking for some help on how to benchmark the following games:


- Gears of War
- Timeshift
- Hellgate: London
- BlackSite: Area 51
- Clive Barker's Jericho
- Need for Speed: Pro Street
- Sega Rally: Revo
- The Witcher


Since none of these have a pre-scripted benchmark (that I know of), I've been thinking about creating timedemos for these games, but I just don't know how and google isn't helping me on this.. :(
Then I'd use FRAPS to indicate the average, max and min FPS on each game.

Any suggestions?

Thanks in advance.

[BenchZowner]

Only thing you can do for "real" tests is repetitive runs for the racing games.
E.g. same map, same car, no opponents, 3 laps ( to average things up ).

For the other games, if no "predifined" and pretty much "same" route is possible, then anything won't be that good...

[ToTTenTranz]

How about gears of war?

I saw a review (hardocp I think) where they recorded a timedemo.. I'm trying to figure out how to do that.

Any help?

[BenchZowner]

Gears Of War... while I haven't tested it ( benchmarked it ) something's telling me that it's using the timedemo command just like Unreal Tournament 3.
Check Epic's forums for the commands ( should be there ).

[gdement]

Benchmarkers always want exactly repeatable scripts, but there's another way. If you're familiar with some statistics methods, you can make a good (IMO actually better) analysis with a large number of random samples. It's time consuming though, and pretty nerdy.

Run FRAPS. Get the game into a mode which represents gameplay, but where you are NOT in control. An automated botmatch or whatever. Establish consistent settings and stick with them. Click whatever button transfers your view to another bot. At the same moment, immediately start logging framerates with fraps. Leave the view there until the logging ends (default is I think 30 seconds). Transfer the view and start logging again, etc. If the match ends while logging, or something else happens to invalidate the test, then stop the log early. Those truncated log entries are regarded as invalid and you will later delete them from the set. Restart another botmatch with the same settings and continue.

Always switch bot views when you start logging, because this way you will not know in advance what scene you're about to benchmark. This prevents any subconscious influence on when you choose to start logging. It's important to eliminate any possible source of bias or your data will lose meaning. If a bot stares at a wall, allow it. Don't give yourself the biased ability to throw that out, unless you can objectively define the criteria. Things like wall-staring will average out in the long run.

After you've done this a bunch of times, you will have a long list of framerates in your fraps log. Delete anything which was interrupted. Compile a list of the framerates into Excel and run some statistics.


The simplest stat is a mean framerate. But the most useful statistic IMO is a confidence interval. The CI can give you a range for what the actual average framerate should be, if you were to have run the test an infinite number of times. The precision of the CI improves as the sample size increases. This tells you how accurate your calculated FPS figure probably is.

I did this once to compare some hardware configurations with the demo version of Unreal 2004. It was accurate enough to show the subtle benefit of faster RAM (about 9-10% IIRC). The 95% confidence intervals had a radius of about 2% either direction from mean FPS, so they were pretty precise. I think I did about 30 samples of each configuration.
It was fun to try, but I'll probably not bother doing such a thing again.


***
[There's an underlying assumption that the set of all possible random framerates has a normal distribution. I think that's reasonable to assume. I used the kurtosis function KURT() on my data and it looked fine.]

[BenchZowner]

9-10% from faster RAM in gaming isn't for real...unless you were benching on pretty much CPU limited resolutions & settings gdement.

[gdement]

9-10% from faster RAM in gaming isn't for real...unless you were benching on pretty much CPU limited resolutions & settings gdement.

That was the result with a particular setup on Unreal 2004, I'm not trying to say it applies to all gaming. I was testing setups for my nephew's computer who I knew wanted to play that game. The RAM comparison was deliberately done on one of the "Assault" levels which are pretty slow on that machine. The machine was an AMD761 with an unlocked Thoroughbred, TI4200 video.
I would have liked to do a similar comparison on a fast level, like Rankin, but I didn't want to take the time and it didn't really matter, the slow stuff is what deserved attention.

I looked up the assault-convoy results. The 95% confidence interval with default 266cl2.5 timings was 29.148-30.243fps. With 266 at 2-2-6-2, the interval was 31.998-33.305fps. Comparing the median of the samples yielded 9.18%. Comparing the means yields 9.94%, and the possible range taking the extremes of the confidence intervals is 5.80-14.26%.