Crysis Benchmarking - Methodology and Mythbusting

[Nedjo]

For a long time I was trying to find out appropriate way to benchmark Crysis.

I wasn't satisfied with what mainstream media was doing with pointless flyby demo for GPU, and even worst demo for CPU testing.

I've done some digging and will present you in this thread what I've find and what I think is appropriate way to benchmark Crysis.

I hope that you'll find this useful, and will accept this as a default way for testing Crysis, so that we all can share and compare our results.

First of all, what's needed is a good front end app that will allow easy changes for settings. Luckely there is outstanding little app for this purpose!

Second what we need is appropriate benchmarking sequences. To my amazement (maybe some of you knew that) Crysis have benchmarking sequence for each single level!

When we mix up front end app, and do appropriate editing to have those sequences usable for benchmarking we get this archive that you need to download:

http://rapidshare.de/files/38479006/...rking.rar.html

Just unRAR this archive to the folder where you've installed Crysis (e.g" E:\Games\)

You can run benchmark for any level in Crysis, by simply choosing appropriate demo recording from level folders. You can even run demos for multiplayer levels.

In each folder there are two timedemo files. You need to select file with the same name as the level.

I've tried all timedemos, and it seems to me that demo from level "sphere" is the best one! There is a plenty of jungle, water, snow, ice, shaders… really the best representation of the majority of game.

You can see my results for "sphere" (Vista64, Phenom 9700, Radeon HD3870, Catalyst 8.1)


I usually run timedemo three times. You'll get average fps from all three runs at the end. But bear in mind that this nice app doesn't place coma at the right place in the resulting score. If you want score for each run, just scroll up slider…

You can setup bunch of different settings, so feel free to experiment!

[Nedjo]

OK, about "mythbusting" part of this thread!

You all know how Cevat was preaching around that this game will benefit more from lower clocked Quad CPU, then from higher clocked Dual CPU!

I want to prove to you that all his talk was only 'cos of Intel sponsorship money! This game is heavily GPU dependant, and it is NOT multicore optimized! It is however explicitly dual core optimized. In other words, game will run same with Dual Core, and Quad Core CPU's working on the same frequency! Of course higher clocked Dual Core will win over lower clocked Quad-Core!

Thanks to ASUS's M3A32-MVP DeLuxe, and Phenom 9700 CPU, I'll show you CPU benchmark scores with 2, 3, and of course 4 cores running timedemo!

I've setup options in a fashion that I'm finding relevant for CPU benchmarking. Also, I've used new, much more intensive and dynamic CPU2 benchmark that you can find in [i]"ice"[bi] level!

Once again don't be confused with the lack of comma that is separating 33 fps and decimal part of the score!

Dual-Core score: Triple-Core score: Quad-Core score:

What you'll notice if you look closely, is that Crysis CAN'T load more than two cores. Hell it's having trouble loading just two cores fully. Don’t pay attention to the peaks at the end of benchmarking session.

Today on market there are NO multicore optimized games! There are only dualcore optimized titles, and Cryris is one of them! Only use of multicore CPU's with this games is that you'll be able to run some parallel tasks (preferably on another HDD), during the gaming session!

I hope that you've find this thread useful and that will help you evaluate performance of your hardware in Crysis in proper way! Looking forward to your scores!

[Hosser]

Great post(s).

I've been using this tool for a while, but only with the standard timedemos - I wasn't aware of the fact that there were more, so thanks for that.

Also I was only getting an average of around 10 FPS with my HD 3870 crossfire setup, which seemed very low compared to your scores. I tried reinstalling the game and this time did not patch to 1.1 - now it runs far better, averaging between 20 and 30 on Very High settings, no AA @ 1024x768.

So if anyone's having Crossfire issues, try going back to unpatched retail version. The changelog states "improved Crossfire/SLI performance" but it actually reduced performance for me. Weird.

[purecain]

thanks for that Nedjo.... very interesting....

[Nedjo]

So if anyone's having Crossfire issues, try going back to unpatched retail version. The changelog states "improved Crossfire/SLI performance" but it actually reduced performance for me. Weird.

Latest beta drivers that are published for launch of HD3870 X2 are bringing great speedup for Crysis. And since 3870 X2 is essentially crossfire setup, I reckon that you would gain performance to.

Hopefully these optimizations will end up in February issue of WHQL Catalysts (8.2)

[Nedjo]

Also I was only getting an average of around 10 FPS with my HD 3870 crossfire setup, which seemed very low compared to your scores. I tried reinstalling the game and this time did not patch to 1.1 - now it runs far better, averaging between 20 and 30 on Very High settings, no AA @ 1024x768.

It’s just amazing how GPU heavy is Crysis. Here are results on HD3870 X2. And if you compare it with results from the first post you'll noticed respectable fps gain with this "all-in-one" Crossfire card! And the resolution is just 1024x768! Usually you can't see CF benefits below 1600x1200!!



In "very High" Settings the game is gorgeous, but you can't play it in rez higher then 1024x768! This might be a first title that truly asks for QuadFire to really shine!

[ColonelCain]

How do you get the app to show all of the FPS info in the "results" box? Mine only says the settings after each run, no more.

[gallardo]

Today on market there are NO multicore optimized games! There are only dualcore optimized titles, and Cryris is one of them! Only use of multicore CPU's with this games is that you'll be able to run some parallel tasks (preferably on another HDD), during the gaming session!

Supreme Commander.

[brandinb]

OK, about "mythbusting" part of this thread!

You all know how Cevat was preaching around that this game will benefit more from lower clocked Quad CPU, then from higher clocked Dual CPU!

I want to prove to you that all his talk was only 'cos of Intel sponsorship money! This game is heavily GPU dependant, and it is NOT multicore optimized! It is however explicitly dual core optimized. In other words, game will run same with Dual Core, and Quad Core CPU's working on the same frequency! Of course higher clocked Dual Core will win over lower clocked Quad-Core!

Thanks to ASUS's M3A32-MVP DeLuxe, and Phenom 9700 CPU, I'll show you CPU benchmark scores with 2, 3, and of course 4 cores running timedemo!

I've setup options in a fashion that I'm finding relevant for CPU benchmarking. Also, I've used new, much more intensive and dynamic CPU2 benchmark that you can find in [i]"ice"[bi] level!

Once again don't be confused with the lack of comma that is separating 33 fps and decimal part of the score!

Dual-Core score: Triple-Core score: Quad-Core score:

What you'll notice if you look closely, is that Crysis CAN'T load more than two cores. Hell it's having trouble loading just two cores fully. Don’t pay attention to the peaks at the end of benchmarking session.

Today on market there are NO multicore optimized games! There are only dualcore optimized titles, and Cryris is one of them! Only use of multicore CPU's with this games is that you'll be able to run some parallel tasks (preferably on another HDD), during the gaming session!

I hope that you've find this thread useful and that will help you evaluate performance of your hardware in Crysis in proper way! Looking forward to your scores!

Hmmm i think your wrong on this one.

why dont you open the editor and make a tower of barels very high and wide like a few hundred and then knock them over by bumping them.... i think you will see 4 core pin out 100% cpu usage

this game is just gpu limited for the majority of normal situations and the cpu CANNOT help the rendering faster so the cpu sits idle hence the 30% usage


oh here is some evidence it is ocho core optomized

http://enthusiast.hardocp.com/articl...50aHVzaWFzdA==

[Nedjo]

Hmmm i think your wrong on this one.

why dont you open the editor and make a tower of barels very high and wide like a few hundred and then knock them over by bumping them.... i think you will see 4 core pin out 100% cpu usage

There's no doubt that Crysis boast some nice Physics engine, but in game what you can see of it is maximally expressed in this specially tailored CPU demo. Your proposal with barrels is really interesting, but it's pointless 'cos it'll not give you real picture of what to expect from the game itself.

this game is just gpu limited for the majority of normal situations and the cpu CANNOT help the rendering faster so the cpu sits idle hence the 30% usage

That's why lousy CPU benchmarking tool it is.

oh here is some evidence it is ocho core optomized

http://enthusiast.hardocp.com/articl...50aHVzaWFzdA==

Really don't know what to think about this! We don't have any clue about used settings! I've provided exactly what and how I've tested, so I expect others to do same.

[Warboy]

2/5/2008 11:22:22 PM - Vista 64
Beginning Run #1 on Map-sphere, Demo-autotest
DX10 1024x768, AA=No AA, Vsync=Disabled, 64 bit test, FullScreen
Demo Loops=3, Time Of Day= 9
Global Game Quality: VeryHigh
================================================== ============
TimeDemo Play Started , (Total Frames: 2000, Recorded Time: 82.89s)
!TimeDemo Run 0 Finished.
Play Time: 67.04s, Average FPS: 29.83
Min FPS: 21.61 at frame 75, Max FPS: 43.92 at frame 791
Average Tri/Sec: -24918914, Tri/Frame: -835284
Recorded/Played Tris ratio: 0.79
!TimeDemo Run 1 Finished.
Play Time: 64.14s, Average FPS: 31.18
Min FPS: 17.44 at frame 1771, Max FPS: 43.92 at frame 791
Average Tri/Sec: -26048202, Tri/Frame: -835415
Recorded/Played Tris ratio: 0.79
!TimeDemo Run 2 Finished.
Play Time: 64.18s, Average FPS: 31.16
Min FPS: 17.44 at frame 1771, Max FPS: 43.92 at frame 791
Average Tri/Sec: -26035354, Tri/Frame: -835496
Recorded/Played Tris ratio: 0.79
TimeDemo Play Ended, (3 Runs Performed)
================================================== ============

Completed All Tests

<><><><><><><><><><><><><>>--SUMMARY--<<><><><><><><><><><><><><>

2/5/2008 11:22:22 PM - Vista 64

Run #1- DX10 1024x768 AA=No AA, 64 bit test, Quality: VeryHigh ~~ Overall Average FPS: 31.17
====================================


Specs below with 171.16 drivers

[Nedjo]

excellent Warboy!

But tell me did you run it on your "everyday system" or on your "Testbed system"??

[gallardo]

Love the name of the picture on imageshack. "crysissy"

[Warboy]

excellent Warboy!

But tell me did you run it on your "everyday system" or on your "Testbed system"??

Everyday

[WeStSiDePLaYa]

You wouldn't know why SLI gives me a performance drop in crysis?

All modes do it. I thought it was my old board and 6400+ bottlenecking it. But it still happens with my new setup, with a fresh install of XP.

[S1mon-]

I dunno whats the problem with my system... I get an average of about 16-17 FPS with my rig in signature... Dunno if it's because I'm using a graphic mod but performance is horrible...

[Warboy]

You wouldn't know why SLI gives me a performance drop in crysis?

All modes do it. I thought it was my old board and 6400+ bottlenecking it. But it still happens with my new setup, with a fresh install of XP.

Do you have the multi-Gpu command enabled in the auto cfg?

I dunno whats the problem with my system... I get an average of about 16-17 FPS with my rig in signature... Dunno if it's because I'm using a graphic mod but performance is horrible...

Its a Graphics mod, I removed mine and ran normal VeryHigh and it got good FPS, I put my Graphics mod back in and i got a avg of 20-26, so yea.

[S1mon-]

Thanks WarBoy, I redid the test and I'm getting 27 FPS average now on sphere

[WeStSiDePLaYa]

Do you have the multi-Gpu command enabled in the auto cfg?



Its a Graphics mod, I removed mine and ran normal VeryHigh and it got good FPS, I put my Graphics mod back in and i got a avg of 20-26, so yea.

What is the multi-gpu command?

Would that also affect the regular game, and not just bench?

[Warboy]

What is the multi-gpu command?

Would that also affect the regular game, and not just bench?

It will effect the game eng. So Map editor, Game and bench.

r_MultiGPU = 1

drop that into autoexec.cfg in

C:\*\Electronic Arts\Crytek\Crysis

Thanks WarBoy, I redid the test and I'm getting 27 FPS average now on sphere

So the GTX gets 4-7 FPS more then you cool.

[S1mon-]

Your GTX is highly overclocked though I guess

[Warboy]

Your GTX is highly overclocked though I guess

Nope, at Stock OC2 settings.

[S1mon-]

Nope, at Stock OC2 settings.

Ohh ok :P

[[XC] hipno650]

i just ran it on my system in my SIG.

All settings very high, 1024x768 noAA, 64bit, DX10, Sphere level

Overall average was 28.135FPS

i like this tool, i can now test ever level and tweak my settings for crysis

[jcool]

Nice it works
Running XP x64, 168.28 drivers. G80 GTS 640 @ 756/1890/2080
It doesn't run in 64bit mode however... anyone know why? When I start the game from the bin64 folder it works fine.

08.02.2008 01:09:08 -
Beginning Run #1 on Map-sphere, Demo-sphere
DX9 1900x1200, AA=No AA, Vsync=Disabled, 32 bit test, FullScreen
Demo Loops=2, Time Of Day= 9
Global Game Quality: High
================================================== ============
TimeDemo Play Started , (Total Frames: 2000, Recorded Time: 82.89s)
!TimeDemo Run 0 Finished.
Play Time: 84.35s, Average FPS: 23.71
Min FPS: 0.59 at frame 1759, Max FPS: 37.00 at frame 800
Average Tri/Sec: -24534384, Tri/Frame: -1034755
Recorded/Played Tris ratio: 0.64
!TimeDemo Run 1 Finished.
Play Time: 79.33s, Average FPS: 25.21
Min FPS: 0.59 at frame 1759, Max FPS: 37.00 at frame 800
Average Tri/Sec: -26038108, Tri/Frame: -1032771
Recorded/Played Tris ratio: 0.64
TimeDemo Play Ended, (2 Runs Performed)
================================================== ============

Completed All Tests

<><><><><><><><><><><><><>>--SUMMARY--<<><><><><><><><><><><><><>

08.02.2008 01:09:08 -

Run #1- DX9 1900x1200 AA=No AA, 32 bit test, Quality: High ~~ Last Average FPS: 25.21