Printable View
Wonder what showstopper bugs could creep in this late in the game, especially when it comes to consistency. I'm assuming the benchmarks have been run tens of thousands of times internally so inconsistent scoring is a strange thing to find at this point. Oh well.
Yeah, it is not unlikely that they haven't released the newest version of their useless bling bling benchmark because a client paid enough to make further adjustments.Quote:
Originally Posted by Borden
IHVs directly pay Futuremark for the service to "validate and optimize their software to the platform". Too bad that most people don't comprehend that Futuremark is significantly hurting the industry with that and they then even actively hype this miserable product that completely fails to mimic the workload of real (future) games.
And before some bling bling Benchmark fans go for my throat again, look - they even say this openly:http://www.futuremark.com/bdp/3dmark/Quote:
BDP Members get the following privileges and benefits
* Ability to influence Futuremark’s 3DMark roadmap and implementation
* Access to specification documents, alphas, betas and release candidate code drops of benchmarks
* Frequent communication and briefing
* Scheduled conference calls
* Troubleshooting for drivers and new technology
...
:mad:
So yeah, if you are smart enough to comprehend the issue: don't support them by using their useless benchmark and simply use real games instead.
Yeah but when hardware vendors sponsor games they do exactly the same thing in fact taking nvidia as a example they sometimes have more influence. AMD drivers aren't going to make much of a difference between the 580 and the 6870 anyway.
If AMD had enough influence to cause the launch to be in November for driver reasons then they'd have enough influence to force the launch to be delayed due to Cayman.
3Dmark.com = ORB (the new one)Quote:
Originally Posted by Manicdan
While I cannot go to exact details or give examples, you couldn't be more wrong.Quote:
Originally Posted by Katzenschleuder
The goal is to have a stable and fair benchmark. Has always been. BDP helps to ensure that - you have to understand that all major vendors are in BDP. They also watch very carefully what each other does (duh!) and get early builds for testing and comments - and trust me, they do care. They also bring a lot more eyeballs for verifying and testing the code. Who do you think is best at spotting stuff that could be deemed "unfair" to vendor X than, well, vendor X themselves?
And yes, BDP has been instrumental in helping to get that final polish done.
Wasn't really showstoppers, crashes or anything like that - just minor nagging things that presented us with an option - launch on time with known issues that would need a patch quickly or take a few extra days and fix them first. Since benchmarks are trickier to update after launch (especially if the issue affects score in any way), option 2 was, in our opinion, a better idea.Quote:
Originally Posted by trinibwoy
Any one tryed this yet with it coming out today!
From what i know isn't coming today, is delayed.
http://forum.lab501.ro/attachment.ph...1&d=1291046010
So you are saying yourself that the IHV not paying you will have a disadvantage. That you call this a process of "transparency and fairness", is just laughable!Quote:
Originally Posted by FM_Jarnis
To give you an example what your own customers have to say about your business model:http://www.opengl.org/discussion_boa...287#Post167287Quote:
Originally Posted by Matt Craighead, NVIDIA
So, in the end the IHVs pay Futuremark for useless benchmarks directly and they have to waste additional resources with costly API call wrappers.
You say that 3DMark isn't useless? The point of 3DMark Vantage has been to predict performance of Direct3D 10 games, right? Well look for yourself: http://img94.imageshack.us/img94/2428/3dmarkj.jpghttp://img829.imageshack.us/img829/739/games0.jpg Computerbase.de
6 out of 12 misplaced performance predictions - a perfect random mean! But what else can you expect from a benchmark that creates scenes that are totally atypical for games?!
So people: Don't support Futuremark by using or even buying their benchmarks. You are hurting the game and graphics hardware industry by doing that!
Tell me first which game is perfect ? All games have bugs, some games prefer red teams cards, some prefer the green teams cards... I don't like the way the game industry is heading with so many console games being recoded for PC use... makes them buggy, choppy and so annoying for keyboard inputs...
So what do you suggest then ? Can you create the ideal benchmark to predict similar gameplay ? It's impossible, you can just stress on new technology, it's new features..., every game code got it's own quirks, bugs, etc... If a benchmark doesn't favour their brand, then they will call it lame, if the other competitor performs better : they will accuse them of using optimised drivers etc... It's always something...
The game industry it's hurting itself by using sortlike apps as Steam. Not allowing to create dedicated servers for online/LAN party wars by users, bad patch behaviour,... etc, etc,... Not by Futuremark benchmarks... don't be silly... I don't even see how futuremark could hurt the graphics industry...
Mostly the GPU's that perform well in Vantage, will do so in games... Get your facts straight bud !
Your numbers suggest that the Xtreme preset is quite an accurate measurement, apart from the 4870X2 which is greatly underestimated. The other cards are in the right order and even the ratio is pretty much accurate.Quote:
Originally Posted by Katzenschleuder
The quality of your will to read and consider stated arguments is direct proportional to your use of sane punctuation.Quote:
Originally Posted by Leeghoofd
I have accidentally posted a unreleated benchmark that includes many D3D9 titles, which I fixed.Quote:
Originally Posted by massman
Can you also fix the chart so it compares similar resolutions? And as for the Vantage numbers: are these overall scores or GPU scores?Quote:
Originally Posted by Katzenschleuder
//edit: I saw on your site that you actually HAVE numbers with similar resolutions. Why not show that one? Why use a different graph? ...
The point is to have a GPU-limited scenario like in 3DMark and it doesn't make much difference anyway.Quote:
Originally Posted by massman
That's why I asked: "Are those points overall score or GPU score?"Quote:
Originally Posted by Katzenschleuder
And, yes, it actually makes a difference. At equal resolutions, the order and ratio are again quite accurate (slightly in favor of Nvidia cards) if you bear in mind that Vantage was released back in Apr'08 and has to measure the performance of a Nov'10 product.
http://hwbot.org/blog/wp-content//difference.png
I sincerly hope you don't spread this hate anywhere else.
I replied to your statement. What is the ideal benchmark to you ? Can it be done ? Don't think so... Take eg the Hawx 2 benchmark : performs brilliant on Nvidia and AMD performance is sub par... If that benchmark was taken as a reference I would partly agree with you...Quote:
Originally Posted by Katzenschleuder
The new numbers you posted show in the top that the 5970 dual GPU is maybe not correctly supported by the driver, the game doesn't benefit from it... the lower end cards, are hit and miss : maybe not enough ram on the GTX 460 (768 version) , drivers not optimised etc... I don't see that much discrepancies...
Take into consideration that Vantage is aging. Drivers, games & hardware have been optimised,... I just want to state that creating a benchmark ( maybe sometimes even before a decent game is released ( supporting the new technology ) is never gonna be ideal...
Are you saying FM is biasing their benchmark in favour of one team ? Being it the one that pays the most :p Then the beleive in the benchmark and company would go rapidly downhill... don't think they can make that man...
What the heck?! Most GPUs are still rated incorrectly: 5870/470; 6850/460; 5830/465; 4870/5770; 260/5770; 450/5750; 4870/260 ... They are all rated incorrectly.Quote:
Originally Posted by massman
The whole point here is that this is the only pretended intent of 3DMark!Quote:
if you bear in mind that Vantage was released back in Apr'08 and has to measure the performance of a Nov'10 product.
I am sure that you are going to continue to be happy to support Futuremark with their business model that is evidently hurting the industry.Quote:
I sincerly hope you don't spread this hate anywhere else.
I never said that it is realistic to think that one can create a benchmark that mimics the workload of upcoming games, early in the GPU generation.Quote:
Originally Posted by Leeghoofd
The whole point here is that Futuremark promises to deliver this and fails for good reason. Have you read the inofficial statement made by Nvidia which also makes this clear?
The even worse issue is that the business model of Futuremark makes their benchmark content directly dependent on the money paid by particular industry vendors, which is simply outrageous!
No, it's hurting itself by not taking the PC seriously enough and positioning it as a mainstream rival to PS3/XBOX360Quote:
Originally Posted by Leeghoofd
That statement Of Nvidia was that the one regarding 3Dmark2003 ? Think if the chosen option was in favour of Nvidia they wouldn't have nagged at all... Think Nvidia is one company that tends to opens a lot of cans of worms with bad expressions... There are even special created drivers by this company that allow "bugged" runs on FM benchmarks, so I wouldn't take it all too seriously... Seems you are not a bencher : we all have drivers ( beta drivers) that create "bugged" runs in sub tests of some Futuremark benchmarks and this results finally in a higher output score... in reply to the statement by the above mentioned nvidia employee, they themselves created the driver to "manipulate" the outcome...
Very good point there... hence why I don't like console converted games... usually need silly hardware requirements for choppy gameplay...Quote:
Originally Posted by initialised
Why in hell would that nonsense make a difference to the problem that Futuremark practices an outrageous business model?Quote:
Originally Posted by Leeghoofd
What the F are you talking about? ALL IHVs have to waste driver development resources for 3DMark API call wrappers.Quote:
There are even special created drivers by this company that allow "bugged" runs on FM benchmarks, so I wouldn't take it all too seriously...
No, they are rated like you'd expect. The combinations you mention are based on price/performance ratios, which is very sensitive to very small differences in both FPS and price swings.Quote:
Originally Posted by Katzenschleuder
The point of running 3DMark is that you have 1 number than estimates the rough positioning in the market. Eg: 5970 > GTX580, 480 > 5870 or 4890 >= 5750.
In a very recent interview, Futuremark has explicitly stated that end-users should run as many benchmarks as possible to get a good view on the factual performance of your product. That's quite different from your idea, which is apparently based on an interview from 2003.Quote:
Originally Posted by Katzenschleuder
Link to interview: http://hwbot.org/article/newsflash/9...mark_president
Evidently?Quote:
Originally Posted by Katzenschleuder
So far I've seen an outdated quote and a couple of charts that show 3DMark is fairly accurate in grouping similar performant products.
WTF?! The majority of the GPUs tested have an incorrect performance rating succession when comparing 3DMark with real games! All those combinations that I have listed are rated incorrectly by 3DMark.Quote:
Originally Posted by massman
Why are you making up the claim that this "based on price/performance ratios", when it clearly is not?
That even they are saying that 3DMark is not a good representation of game performance is in which way speaking for them?Quote:
In a very recent interview, Futuremark has explicitly stated that end-users should run as many benchmarks as possible to get a good view on the factual performance of your product.
Outdated? They continue with exactly the same business model that has been critisized by IHVs, which is even openly advertised by Futuremark today.Quote:
That's quite different from your idea, which is apparently based on an interview from 2003.
So far I've seen an outdated quote
Do you have to punch your head against the wall to make yourself believe that obvious untruth or do you just have to turn a switch for that?Quote:
and a couple of charts that show 3DMark is fairly accurate in grouping similar performant products.
I don't know where to even start, so I'll just state that FM stands behind it's products and states that they produce usable results within the margin of error that is reasonable to expect from a single benchmark.
Will you get more accurate results if you mix multiple benchmarks and games when compared to a single benchmark (even one that has multiple workloads)? Sure. That's simple math.
Does that invalidate any individual benchmark? Nope - a single benchmark has higher error margin than combined score from multiple benchmarks. Duh. That doesn't mean the result is invalid.
Are the differences between 3DMark and a pool of games material? No - they are well within the error margin, especially when you consider that many games run better on cards from specific vendor (accidentally or on purpose), creating bias. Spotty multi-GPU support also creates bias (while 3DMark is always offering full multi-GPU support). You also have to consider that benchmarking a pile of games and creating an average out of that is obviously (slightly) more accurate than any single result (from a benchmark or a game).
3DMark Vantage might not be the best or the prettiest product we've ever done, but one thing it does do is to produce a valid, unbiased score and seems to be doing it even with hardware that was years away from being released when it was shipped. With 3DMark 11, we have the exact same goal.
As for the business model side - I'm not the person to discuss about it. If you want to continue that discussion, I suggest you contact us at bdp [at] futuremark.com - I'm sure our guys are happy to do an interview or provide more details.