sorry for being away but i have to work to make my living
here is some new benches
http://img101.imageshack.us/img101/3...ptureya.th.jpg
Uploaded with ImageShack.us
Printable View
sorry for being away but i have to work to make my living
here is some new benches
http://img101.imageshack.us/img101/3...ptureya.th.jpg
Uploaded with ImageShack.us
http://img408.imageshack.us/img408/8...ture1na.th.jpg
Uploaded with ImageShack.us
http://img44.imageshack.us/img44/1467/newweb.th.jpg
Uploaded with ImageShack.us
http://img202.imageshack.us/img202/885/new311.th.jpg
Uploaded with ImageShack.us
[IMG=http://img140.imageshack.us/img140/6003/new31p.th.jpg][/IMG]
Uploaded with ImageShack.us
[IMG=http://img594.imageshack.us/img594/8962/new21.th.jpg][/IMG]
Uploaded with ImageShack.us
[IMG=http://img255.imageshack.us/img255/4913/new12.th.jpg][/IMG]
Uploaded with ImageShack.us
Welcome back Tiltevros,Quote:
Originally Posted by Tiltevros
How about a small tweakers guide for the 9211?
Some general advice would be great.
Welcome back.
Just a quick question: What drives, controller, and settings are you using to produce those numbers?
I see you get over 2000MB/s sequential read, and 1900MB/s+ 64KB (random?).
I'll also note, you get 100-120K IOPS both read and write in CDM, but around 180K IOPS in AS SSD. I will speculate you are CPU-bound in CDM since it's single-thread, but either controller-bound or drive-bound in AS SSD. 180K/8= 22,5K IOPS read pr drive if you are using 8R0. Considering 64/8 = QD 8 pr drive, that would suggest Intel x25 drives. The random write IOPS also supports the drives being x25-M, at about 10-15K each. But 500MB/s write from 8 drives seems a bit low, ca 60MB/s from each.
@ Gullars sequential reading test in IOmeter
LSI 9211 i8 with 8 intels @ R0 64k stripe size
Welcome back Tiltevros
+1 with Anvil - would love to see a 9211 tweakers guide from you!
im waiting some graphs from u to start writing but.........
Just to clarify, you are using integrated RAID, and still get 180K IOPS?
Would you mind running PCmark Vantage HDD test on the array?
Also, could you run this IOmeter config? It's the 512B-64KB random read QD 1-128 file i've used earlier to make some nice graphs.
It will take about 20-30 minutes to run.
I'd also love it if you could run the corresponding multi-threaded test (4 workers) to see if it makes a difference on IOPS. I custom made it for no cache so it will take 1/12 the time to run of the full test with 1 worker.
Since it's 4 workers, each will cycle QD 1-32 for QD 4-128.
Results should be fairly accurate for all controllers not using cache.
EDIT: if you run these 2 configs, i'll make you loads of nice graphs, and compare to other storage setups i already have the corresponding data for (the first config).
The second config will show if you are CPU-limited on one core on the first config, and comparing graphs will show how much compared to 4 cores.
I posted graphs wich can be seen as examples in the C300 thread.
EDIT2: link directly to the graphs:
http://www.xtremesystems.org/forums/...1&d=1272302732
http://www.xtremesystems.org/forums/...1&d=1272302732
here we go random read v4
great, and the second file with 4 workers?
and the 4 workers
Tilt - 1.7GB/s 64k random reads @ qdepth 64 - very nice!
@ 128QD i have 1,84GB/s :P
im still waiting for the graphs :P
welcome back Tiltevros!!! Let's see what you are showing off time....:eek::eek::eek::eek:
can u fix my cvsQuote:
Originally Posted by mike
wowowowow 2 days and still waiting lol
breaking the 2GB/s
I've kinda been flooded the last week, will get to it later today or tomorrow.
I'm under group preassure for some gaming tonight, so i'll see how long it goes on.
holy cow! awesome results tilt!
4 days and still waiting lol
Here's something you can look at while i finish up making more graphs. It's 3D diagrams mapping IOPS performance by block size and QD, bandwidth by block and QD, and average accesstime by block and qd.
EDIT: There is something wrong with the file you sent me with 0,5-64KB QD 1-128 4 workers. The file you sent me only had 1 worker, but the attached config file made 4 workers when I ran it.
New graph.
8R0 x25-M G2 80GB LSI 9211-8i, 4KB random read (4K alligned) 1 vs 8 workers, QD 1-128 (you somehow dropped the QD 32 step in the 8 workers config Tilt...)
Attachment 103724
EDIT: first upload was the wrong image :P
EDIT2: theoretical possible scaling will be 240-320K IOPS. It looks like you can pass 200K IOPS with higher queue depths, 256 would likely give around 210-220. 512 may give 230-240. (8 workers or more).
EDIT3: perhaps a more powerfull OC could help too?
~160MBps at QD of 8 and ~80MBps at QD of 4 random read is about the same it shows at the 9260 graphs, maybe the 9211 is even a bit faster.
can i have the fixed xls file ?
or if u want to put all the results in the same 2D graphs.
thnx.
Here's the xlsx file of your numbers and the graphs. Attachment 103736
Later, after i've done steve's numbers, i'll make some comparative graphs to the data i already have from other setups running the 0,5-64KB QD1-128 exp2 v4 config.
I may make a new thread for that comparative stuff.
GullLars
Do create a new thread!
Tiltevros,
Now, how about a small tweakers/getting started guide for the 9211 :)
the graphs over the C300 thread linked 2 posts ago show the about the same scaling for 4-8QD at 4KB and at 64KB, unless anyone care for another few tens of MBps or even a 100 when the difference between the 9260 and the 9211 is about 200MBps for the 9211 at irrelevant desktop usage QD of 64..Quote:
Now, how about a small tweakers/getting started guide for the 9211
that is if these results here are write through (seems they are).
the only deficit the 9260 has in compare to the 9211 is the ROC which might take some access time overhead and so deviate the bandwidth.
it would be very interesting though to see a softraid of 3 X25-M's on the ICHR or even the 790/850SB and 5 on the LSI card, or a strait compare to the 1231ML.
do u think that 3 x25-M's on ICHxR will be faster on softraid than 5 with LSI?Quote:
Originally Posted by onex
GullLars when u have time make icf file to run what ever u want with 16 threads.
ok, will do
no,Quote:
do u think that 3 x25-M's on ICHxR will be faster on softraid than 5 with LSI?
i'm saying, the ICH10R with 2 or three drives has showed some amazing results,
iirc, they were much better then 3 drives on a raid controller,
what was suggested, is merging the two together, i.e - take 3 X25-M's and place them on the ICH10R, then add another 5 on the RAID controller, and soft RAID them all together.
you might get the best of both worlds,
some kick from the onboard controller and the rest of the speed (which it can't give you due to it's ~700MBps limits),
get from the LSI.
i'm wondering of the benefit and the possibility of it working well,
steveRO seems to have done it with his ACARD's http://www.xtremesystems.org/forums/...13&postcount=8
;) so u want 3 on ICHxR and 3 on LSI and i have to break 98000+ IO/s @ 4k file random? :D hehe is that it? :P
:), lol, no, this is not a competition you mad bull :lol2:,
you seem already shaking your hands for a fight :cheer2:,
this is about testing the best solution (or the fastest atleast).
and it's about taking 3 on the ICHR and the rest 5 on the LSI..;).
hmmmmm i want some competition ;)
We should have some competition that has to do only with storage !!!!!
Now for ur request how many IO/s do u think that it can score in random 4k @ 64QD? any predictions?
Here you go, 2 new configs with 16 workers.
The first one is 4KB random read only.
The second is 0,5-64KB random read QD 16-256 (1-8 pr worker, exp2 step). I will compare results to your single worker results.
EDIT: forgot to attach it :P
BTW, the 0,5-64KB random read test is set up with 1GB testfile and 10sec runtime (1s ramp + 9s run), so it's for no cache only usage.
no idea really,Quote:
hmmmmm i want some competition
We should have some competition that has to do only with storage !!!!!
Now for ur request how many IO/s do u think that it can score in random 4k @ 64QD? any predictions?
the ICH10R should give some bust to the smaller random read and write patterns due to less operational overhead and so, should overall deliver better results for 3 SATA2 drives,
it might be happening due to a simpler design of the ICHR interface versus the PCI-e interface and might even due to the physical length the data has to travel, so eventually giving better access time and so higher IOPs count.
it seems to have never been tested before with such configuration, a 6 vs 6 drives for example are limited by the ICHR overall bandwidth and so would give a benefit to an 8 way controller card :).
E:
have a good look here, these are anvil's results gullars brought some months ago with 1-4 X25-V's.
as you can see, at QD of 64 and 4KB 100% read, your results on the 9260-4i with 4 X25-M's G1 are showing "mere" 45994 IOPs,
while anvil's 4 X25-V's on the ICH10R are showing 111,297 :),
so there should be some nice scaling :).
xmmm with 6 drives i could say round 800MB/s 200.000+ IOs ;) with 3+5 i would say round 310.000 IOs now with 9211+ICHxR+9260 3+3+2 loooooooooooooooool
no, seems you are going too far :),
maybe around 120,000 or 110,000 for 4 drives and another about 46,000 for another 4 so around 160,000 maybe 165,000,
the ICHR seems really good with the smaller random IOPs then the RAID controller cards,
the tests anvil has shown are for X25-V's which are slower then the X25-M's, so you might also get some nice bust over the 64KB :).
E:
this is done with 3 X25-M's on ICHR by Musho, some time ago:
100,000 IOPs 4KB 100% read QD of 128 :).
http://www.xtremesystems.org/forums/...1&d=1268613959
u wonna place ur bet? do not underestimate LSI with soft raid ;)
http://img169.imageshack.us/img169/2935/lolycr.jpg
Uploaded with ImageShack.us
http://img294.imageshack.us/img294/7510/lol2g.jpg
Uploaded with ImageShack.us
and its not even a soft raid ;)
WOOW,
hm, seen it somewhere, this is awesome,
so what is this all about :hm:?
http://www.xtremesystems.org/forums/...5&d=1264361402
anyway, this should be intriguing!
Onex, Tiltevros has posted results for 8R0 x25-M on LSI 9211-8i, and got about 106K 4KB random read @ QD 32 (with 1 worker).
With 8 workers, he got more than 125K @ QD 32, almost 185K @ QD 64, and just shy of 200K @ QD 128.
I've posted a couple of IOmeter configs for 16 workers, and i suspect we will see >200K IOPS @ 4KB random read (QD 64 or higher).
yes he will do very well...
i have hit over 200k on the 9260 :) (not random access though)
so the 9211 with his secret sauce should be able to get close.
if one of these cards, the 9260? got 211,421 IOPs up there, the softRAID should get probably around 240-260K.
soft raid is for girls:p:
canceled.
i like long walks by the beach, holding hands with my raid card, and candlelight dinners with my cpu. you?
my money says musho results the result of cache, soft raid of multiple raid arrays on ich....0+0!!!!!
:lol2:, though you will catch that :),
a girl with bristles :P.
anyway, no, really, seems like a nice ICH10R :).
E:
p.s - sorry, it wasn't very nice, just a stupid, silly remark :shrug:,
my bad...:(
u r not putting ur money on me???? :eek: :mad: :mad:Quote:
Originally Posted by Computurd
of course i put my money on you tilt! i was speaking of the intels on the ICH and the 100k iops above^^
onex-missed whatever was silly and stupid.
the girl part ;)...
LSI 9260 test 4 intels x25m's G2
4Mbyte random @ 1 QD (test file 4GB)
onex can u answer me one question plz???
how can 2 intels x25 E's score 110.000 PCmark HDD suite and me with 8 cant?
how can 2 C300 can score 100K+ and they r not on top10 or even that why 8 or less c300 cant even score that??
Why those scores are not used as Boot drives ;) ?
welcome to my world :D
it used to be a forum of changing knowledg and not barking around but if this what they want ...
just let it be..
the same question you asked, is the same i have asked at the C300 vs the LE thread, at last post (currently page 10),
the 6 acard setup which won the WR (with 12) has got only 63K at PCMV while the 2 C300 are getting 100K,
8 C300 should beat the 12 acards by a huge margin by this results, acually, wipe it out totaly,
yet it got much less comparatively.
i'm unsure, how these scores are being calculated, and what is happening inside the test itself,
to tell you another thing, which is kept secret,
one of the guys, managed to score higher or the same as the 12 acard setup with MUCH less hardware, MUCH less...
so to tell you the truth, i don't know how much these benches really emulate real time work load, as every setup seems to give a bit different results,
and it isn't for sure, that if you get the fastest system at any benchmarks, it would operate the fastest for the OS and work you do with it.
now, people bench the hardware, and everything shows one thing, then they will go to linux, and everything may look differently,
the only thing you are left with when you want a decent system to fast operate files, is just build one, and use it, and forget all the numbers,
if it's good and stable,
just use it.
thats why im stopping here there is nothing to learn or to discuss over here.
If i want to see numbers belive me everyone can.You can go to Cebit or to Computex and see the numbers but no knowlegd.
Im done with this this is not an IT section so no reason to exchange my knowlegd or knowhow.
look, tilt,
people are discussing here the hardware, i think mostly people are looking for numbers and not where these numbers are coming from..
optimizing your hardware for a benchmark or a test suit, doesn't make much difference, it's not about competition, but on finding out the causes for one device to operate differently then another.
most people are hard with it, as it isn't easy understanding the processes from the inside, on the superficial level, what you see is the out come, and when one is unfemiliar with the hardware, this is what he/she are generally looking at and for.
this gives him/her confidence, that his/her hardware is working well, at the beginning you look for speed, then you are looking for access time and IOP's, then after you are done with these outputs, you start looking where they are coming from...
you see, one device from different manufacturers, can give different result, and to test that, you need equipment from various manufacturers, and you need people that can take these results, and look inward into them, they have to have interest in it, they have to want to figure out what is happening inside, yet people mostly are fan-boying this product or the other.
you have a 12 acard system that showed 93K for the HDD suit at PCMV and 63 with 6 drives on a quad processor, now part of this test is also CPU included,
yet why is it, that these arrays, lose part of they're efficiency throughout the benches?
now this can happen due to the bench is not equally using the full array, this can be CPU bound which is coming from not decent enough programing, yet if what people are looking for is results, they might overlook such matters,
and then you find out that all the benches you have done, don't worth much.
there is also the matter that companies hide information from the users, as there is a lot of competition all around, and so it is hard to get accurate results when you are not getting every piece of information.
last thing, i don't think people dig enough, you can go over a benchmark, and it is showing one result and then at a later page it is showing another,
when you try to figure the difference you see that one benchmark was done with write back on and the other without, then one was with compression and the other without, one with overclock and the other without, one with more cache and the other without,
so you cannot work like that,
this is becoming even more confusing when you have to open multiple threads to compare a single result, search google for hints and find something which later comes out false, and people are running along.
i think, maybe the only way to be able to really do something, is having either a mutual effort, a lot of benching, and a lot of hardware you have by your own,
this is all very confusing, and i'm not sure it is going anywhere..
You just got right at the heart of what i've been doing and thinking the last 2,5 years :OQuote:
Originally Posted by onex
I will be making one or more threads aiming at a collaborative effort to chart and better understand high-performance storage and SSDs, taking a more scientiffic approach rather than boasting numbers an fanboi phrases.
I have already done a lot of this at the norwegian forum i've been mentioning in many discussions, where I, Anvil, Nizzen, and a couple of others have been doing "research" into SSDs for the past year and a half.
The most recent was my project with the IOmeter config "Random Read 0,5-64KB QD 1-128" i used to map IOPS performance of various setups, but that was just scratching the surface.
I've mentioned in the C300 vs LE thread that I'd like to make a thread dedicated to charting the difference overclocking has on storage performance. It seems units and RAIDs from ICH10R get a performance boost from increasing the PCIe frequency, and overclocking the CPU raises the performance roof of both single threads and multiple threads.
I've spoken brefly with Mike about moderating an "extreme storage" subforum, wich would be a great place to have such threads, and start educating ourselves on how to better set up our storage for our usage patterns (and what parts to buy), and for the IT enthusiasts, get a better look into the finer mechanics of SSDs and SSD RAIDs.
here is the problem.
you have some people who use "nefarious" means to gain IOPS etc.
these are referred to as 'unsustainable' arrays and configurations. they are also not bootable.
there are tricks and software to make cache on any solution, including ICH10r.
alot of the fantastic IOPS that you are referring to with several of these mentioned above are due to these methods.
what tilt is getting at is you have guys who are getting 110000 scores in hdd for vantage, yet they dont have a single result in vantage that is in the top 100 or more. they are using means that are not bootable. there are tons of these tricks, and most are holdovers from the pcmark05 glory days, the crap that goes on, out and out cheating and misrepresentation of arrays, is mostly from these types of tricks.
the guys who know their stuff and have been around awhile can spot these results a mile away. i have called several several individuals out on their results.
You cant build a database on people performing "tricks" with their devices and claiming they are normal setups.
you will be using erroneous data.
this is where the PCMark Vantage tests come in.
you can cheat any benchmark if you choose, especially stuff like iometer crystal diskmark as ssd etc etc etc. even pcmark05 is easily confused.
but what happened is that with the release of vantage they closed alot of these BS tricks out. it has gotten to where you cant even get points on pcmark05 without cheating.
they made it to where there is virtually no way you can run the PCMark suite on vantage with these tricks. however, you can run the HDD tests.
raid 0+0 etc etc that is not bootable is what is giving you these fantastic unbelievably good results. and vantage wont let it be run.
there are alot of software solutions that can be used to generate these types of setups, hybrid ram disks, etc. the only thing you can believe is regular standard benchmarks run from a BOOTABLE array, such as pcmark vantage standard runs. if someone shows me 115000 score in hdd suite, fine. but its bull:banana::banana::banana::banana: if it isn't backed up by a full run. you cant tell me that someone who doubles my hdd test on the hdd suite isnt in the top ten. period. or at least back it up with the HDD numbers from the PCMark suite.
its like the PCMark suite is your qualifier, if you cant run it and show results, well.....maybe you should be leery of the numbers you are being shown, or at least take them with a grain of salt.
you have guys who strip key files out of crysis levels, then load them and say WOW!! LOOK HOW GREAT I AM I AM LOADING CRYSIS IN FOUR SECONDS!!!! but upon closer inspection you can SEE that it is modified.
why do they do it? who knows? guess it gets their rocks off.
but if you call them out and say "hey dude i seen that" everyone ignores it anyways and drools and says "oh your so cool!"
LOFL...but these types of mistakes made by 'cheaters' are rare and hard to see if you dont know what your looking for.
and there are those same types of individuals who post obviously false results and flash them around, even though they know it is crap, and claim it is legit.
c'mon man its the internet, you cant believe :banana::banana::banana::banana:.
EDIT: one caveat to this might be some of the newer devices. however i find it funny that some people have posted results higher than some of the newer stuff for quite some time.
Calm down guys,Quote:
Originally Posted by Tiltevros
2 C300 does not score 100K+
2 C300 + 2 Vertex LEs did that (4R0) using the ICH, as I've said on several occations the drives are not the boot drives but they are bootable.
Tiltevros,
If you take 3-4 of your Intels and create an array on the ICH you'd be able to see for yourself. Sure the score will be a tad lower as long as they are used as OS drives but believe me the score would surprise you.
Why don't you give it a try?
The one thing you need to find out for yourself is what stripe size is optimal for your system. Sometimes it needs a high stripe size but generally the small stripe sizes are the best performers on the HDD test.
@Computurd,
The PCMV Suite is a different beast compared to the HDD subtest.
To be able to compete on the full Suite one needs more than just SSDs.
Since the 980X was released there is no way to compete without one, it scores like no other processor e.g. on the AES test.
edit:
@Onex,
Great post!
It takes a lot of work to figure out how things are working.
I've lost count on the number of benchmarks I've run during the last 12 months.
there is an easy way belive me if i wanted to cheat.. i can easyly put a 60K PCmV suite score to the board but it wont be true.. so i dont... everyone here knows that server board have more cache than everything on the market
You can easyly set 12GB of cache and run a benchmark and say "hey look i have 60k score" but i dont...
Now for researching ok u r right... BUT BUT sharing information can help ppl not discovering the wheel again and again... running benches only with WBC on or off is not something new. Everyone knows that we r gaining IO/s
with WBC so this is nothing new even for a rookie..
Everyone here is asking me and others to run test to see how in the hell do i/they score and say to them self WTF?!?!?! i have never say no even when i was under NDA even some ppl didnt belive it. Everyone was asking and i was answering at every question even sideways and not direct.
i wrote a guid about 9211 and 9260 in Greek but not in english cause none here was sharing information tips or guids or even tips that they work for all.
You are saying something about IT enthousiast. Yeah ok we will help them but how?? Just showing them that they can use differend stripe sizes?? or using WBC or not??? those are not even helping them.
Now about real life applications ok... Real time applications are how many explorers can be opened in a second?? or how many VMwares can be run with out a problem?? or how many clients can be run a databases without a problem??? did anyone here mention the problem that ITs have with Multipath or SQL setups?? or the only think that we r looking here is how to set a record in PCMV or in AS SSD?
im really worring about that i dont even know what the IT community is right now.
Im looking in GPUs forums and i see from users writing about good and bad things about it. but here the only thing that i see is numbers. i started a thread saying negative side of Raid controllers and i was the only one that said the problems that i see with my collection of them.. why??? buying something and speak about it is not bad even is a bad choose.
Good morning Tilt, I agree that we should point out the bad along with the good in the storage systems we test.
I think it is human nature to want to play up the positives and play down the negatives.
I agree, we do need to point out negatives - otherwise people may make uninformed decisions and be disappointed.
this is not only the problem there is no sharing informaton tips and solution of any kind.
I for one share anything i come across i feel worth mentioning. Anvil has also been kind enough to share a lot of info, and on the norwegian forum, i've also got quite a bit of info from Nizzen.
I hope you are up for sharing info, benchmark data, and tips when i make those new threads. I'm currently working on making graphs and diagrams from a huge dataset of random read performance (various block sizes and queue depths) of about 30-40 configurations with about 200-300 datapoints each. I'm trying to decide what setups to include in wich graphs to give meaningfull screenshots.
Tilt - I thought I was sharing all my settings in my pcmv runs.
When I bench I've been trying to give details.
im not intrested on hardware i can see them in orb :)
2,1GB/s 64k file @ 64 QD sequential
9211 64k stripe size 8 intels G1 & G2
Tilt, excellent - I have never been able to get anything close to this with the 1231.
i wish i could get the 9211 to play well with the acards - i will try to post a screenshot when I get home tonight.
LSI 9211-8i with 8xacard 9010 R0 - oc at 4.5 with pcie 105 - any idea what I'm doing wrong?
Incompatibility between 9211 and acards?
http://img52.imageshack.us/img52/168...d9010pcie1.png
Have you got the newest firmware for the LSI 9211? It could help.
Steve,
Why don't you create a new RAID card tuning/experience sharing thread?, that is if you've still got the 9211. (a new thread could be great anyways)
I'm expecting one of those, don't know exactly what day though, Monday-Tuesday?
@ GullLars - This is Paul's 9211 - he updated to the latest f/w before he sent it to me.
@ Anvil - good idea - I do have good info to share on my experiences with the 1231, I'll do that when i get the chance. I'm a total nub in tuning for the 9211 though.
try to set the spinning devices in bios ;) start playing there. Enable PCI bus mastering and set the packet size of 128 in PCI settings
A SSD technology/architecture/theory thread will be comming soon too. I just posted a 3600 word post in a thread on our norwegian forum that will be the basis of it togheter with my updated first post in our SSD thread from earlier this year with about 4200 words. I'll have to boil it down some. I'll get back to making the in-depth benchmarking thread first, and then write an article on the SSD lay of the land for Hardware.no and assist on an article looking deeper (and quantitatively) into SSD performance degrading over time.
Safe to say, i have enough on my plate the comming weeks :)
EDIT, in the mean time, i would suggest people read the following papers:
http://onfi.org/wp-content/uploads/2...bottleneck.pdf
http://www.haifa.ibm.com/conferences...pers/2_2_2.pdf
Tilt - what's an example of what I can set in the bios that would help aside from the obvious oc and pcie 119 (vs 100)?
UD7 mobo does not allow packet size adjustment just pcie speed.
when time comes :),Quote:
EDIT, in the mean time, i would suggest people read the following papers:
http://onfi.org/wp-content/uploads/2...bottleneck.pdf
http://www.haifa.ibm.com/conferences...pers/2_2_2.pdf
waiting for the english version :up:.Quote:
I just posted a 3600 word post in a thread on our norwegian forum that will be the basis of it togheter with my updated first post in our SSD thread from earlier this year with about 4200 words. I'll have to boil it down some. I'll get back to making the in-depth benchmarking thread first, and then write an article on the SSD lay of the land for Hardware.no and assist on an article looking deeper (and quantitatively) into SSD performance degrading over time.