first on the 9211, if someone could give me some simple instructions on upgrading the firmware I'd appreciate it..
http://i787.photobucket.com/albums/y...OKE/atto-1.png
Printable View
first on the 9211, if someone could give me some simple instructions on upgrading the firmware I'd appreciate it..
http://i787.photobucket.com/albums/y...OKE/atto-1.png
mbreslin,
Could you do a few benchmarks that includes random iops.
(ATTO is rather useless imho)
e.g
AS SSD, CrystalDiskMark 3.0 and some iometer benchmarks.
I prefer using the Snipping Tool for screenshots. (snipping tool is included in W7)
I will be doing lots atto was the first/easiest thing to grab, anyone have a good iometer pattern?
You'll find iometer random read, write and workstation config files at the end of post #1 on the following link.
Link (fairly basic tests but it's a good start)
Just change the file size and runtime to something appropriate.
(file size is 1GB and a runtime of 10seconds is default for those files)
edit:
It looks like a great setup :)
edit:
Could you do the winsat test?
It's a quick test
run "cmd" using elevated priveliges and then type
winsat disk -drive N
(subsitute N with the drive letter of your array)
The AS SSD and CDM tests are a bit low but I expect some tuning and upgrading the firmware should bring those numbers up a bit.
The winsat disk test is great, very nice scores all over the line :up:
I guess you'd easily get much better scores using the LSI 9260.
For testing purposes I'd use one of the drives for the OS and the others for testing. (but thats me :))
I think there is something wrong with the AS SSD and CDM results.
I'm sure there is lots of things I can do, I have no idea how to set stripe size when building the array, hardware raid controllers are new to me.
Edit: As for using all 8 for testing I just wanted to see the initial big number seqencials with all 8 drives, I plunked down the cash after all! :D
This PDF http://www.lsi.com/DistributionSyste...troller_UG.pdf might help you, not sure if you already read it, basically 8 x 500 = 4000, both lsi 9211 and 9260 give what it needs "x8 5GT/s PCI Express 2.0 host interface" http://www.lsi.com/channel/products/...18i/index.html
This setup should give you at least 800 ~ 1600 MB/s.
Edit: Well the logic would be sequential speeds of 350 x 8 = 2800 MB/s :D, but with the constraints involved this number could well be halved.
It does, I've gotten over 2GB/sec read/write on wei, and it seems great randoms though latency is kind of high but I knew that from reviews.
The scores generelly aren't that bad, well, the sequential scores are down a lot from what I would have expected. (AS SSD and CDM)
If you haven't upgraded the firmware I'd do that as soon as possible.
Tiltevros and Computurd should be able to help you on the 9211.
You've got a lot of nice toys to play with but you need to spend some time getting it right.
If you've got a spare HDD or SSD hook it up on the MB on the ICH or use one of the C300's, this way you can easily reconfigure the array without reinstalling the OS every time.
I'd say install the 9260 and play with that before you continue the 9211 trials.
Looking forward to your results :up:
Yes, great example.
@mbreslin, try first with one drive then with 2 drives and go up as you observe the results, also like Anvil said very important to update the firmware but before that take a look in the Crucial forum if there is any problem while updating firmwares.
http://www.forum.crucial.com/t5/Soli...s-SSD/bd-p/ssd
Edit: Take a look at this page http://www.forum.crucial.com/t5/Soli...p/10334/page/6 , the username zenfire used "Here is a bench of 8x120g drives (not crucial) setup on the LSI 9260-8i card in Raid 0. I wonder if I setup 8 of these 128gb drives if my numbers would be better then my sata-II drives. Would it be worth it?"
http://www.forum.crucial.com/t5/imag...v=mpbl-1&px=-1
ewww, those numbers don't look right at all.
If you are still playing around with the 9211 - you might want to try a soft raid (don't enable raid 0 in the LSI bios).
I also like the idea of starting with 1 drive and working your way up. :up:
The results are just fine for a 9211. The card starts throwing lower numbers once you get past 128kish. Dynamic raid should be much faster.
i would only say that let the superfetch enable with the 9211.
set the IOmeter to 64k file size at 32 IO/s and post the results... it will smach everything here belive me...
9260 asssd compared from above..
http://i787.photobucket.com/albums/y...WPOKE/9260.png
@ mbreslin
what settings r u using???
128k stripe read ahead
Did you update the firmware on the 9260?
No Read Ahead, Write Back, Direct IO works best for me. (using Intel's)
Stripe size 32KB-128KB depending on usage.
256k best stripe size No read ahead, Always write back (if u havent the BBU) and Direct IO
I will try Dynamic disk today and show some benchmarks
Installation:
=============
Use MegaCLI to flash the SAS controllers. The DOS version of MegaCLI is included
in the zip file. Windows and Linux MegaCLI can be downloaded from the download
section of www.lsi.com.
Command syntax: MegaCli -adpfwflash -f AF2108_FW_Image.rom -a0
BTW he has the latest firmware. 2.30 is the latest
My screenshot is on the right sorry to confuse, I don't have the latest, I guess I need to make a boot disk with the rom on it, Anvil?
mbreslin,
You can install the firmware using MSM. (MegaRaid Storage Manager)
I'll upload a screenshot shortly.
I guess this is cache :)
http://i787.photobucket.com/albums/y...OWPOKE/lol.png
lol !!!!!!!!!!
:) That's the cache. (a bit low on the 16KB random)
Attachment 101956
Log on to the MSM using your Windows username and password
Select the controller and click on Go To in the menu.
Select Controller and Update Controller Firmware.
You should end up with this.
Attachment 101957
Now just browse for the firmware.
4457 - 2800 = 1657. 50% more Sequential read speed performance. I guess the only thing missing is to upgrade the Crucial Marvel and LSI Megaraid controller firmware.
Firmware updated, Is there a way to make windows see the virtual disk besides using the w7 cd which takes forever to get to the partition creation app. I assume you do it with diskpart but it doesn't want to make a partition.
I'm not sure that I follow:)
Are you still using all 8 drives for the "OS" or are you using a separate drive for the OS?
I never use diskpart, I use Disk Management but then again I'm using a separate drive for the OS.
(a separate drive for the OS is a real timesaver for testing and on top of that you get better results on the array, for benchmarking)
I'm using a seperate drive for the os now, what I'm saying is after I build the array in the lsi bios windows doesn't see the disk, though it's listed under disk drives in device manager.
Edit: got it fine with Disk Management, thanks
OK, I get it.
Right click on the Computer icon, then Click on Manage, then click on Disk Management.
This is what you need.
Attachment 101964
Having defined the virtual drive you probably have to initialize the drive and then just right click create the partition.
I urge you to download the MSM, it's a lot easier to use than the bios.
edit:
Link to LSI downloads
You'll find the MSM under Miscellaneous. (way down)
Yeah I had it since you meantioned it I just didn't know about disk management, I don't think I've so much as added a hard drive in 8 years.. certainly never made any raid arrays.
It's never too late :)
Now, enjoy!
When you have the time, could you bench just one C300 on the LSI 9260, I've ordered one C300 and I'm curious to how it handles 6Gb/s drives.
http://i787.photobucket.com/albums/y...POKE/9260b.png
Results seem all over the place and seem to be degrading a bit logging out tonight for gc will hopefully help, I'll do your 1 disk shortly.
disable windows Search and windows Defender.
enable the cache policy and write back flusking.
The results are much better but the 4K random marks are still too low :(
I suppose the 4K random marks do not increase while working on raid. Probably there are exceptions.
I would highly recommend going to the 9260 for the begging of your test. the latency on it with the latest firmware is superb. i think you will be amazed what the cache and read ahead on that much more powerful IOP will do for you. the 9260 is leaps ahead of the 9211 in brute strength, and for an array of this speed believe me you will need it :) . I will link a configuration document for LSI that they changed recently to reflect the changed behavior of the 9260 with the new firmware. if you follow their setup recommendations for SSD you should be able to get some pretty amazing numbers. your latency with that 6gb/s interface should be killer. you need to use everest to measure your latency, because it is the best to measure latency with. nothing measures it closer or more accurately. you should focus on using iometer and everest to test with. also winsat runs. at least that is my recommendation. atto is trash. here is your link:
http://www.lsi.com/DistributionSyste..._2010_v2_0.pdf
I think he used the LSI 9260 for this comparison test.
http://www.xtremesystems.org/forums/...7&postcount=18
Thanks mbreslin,
A single run of CDM 3.0 at 1000MB or 2000MB is sufficient, it's just to establish that it treats 6Gb/s SATA drives like SAS 6Gb/s drives.
(meaning > 3GB/s sequential reads etc)
I've never seen the C300 benched in raid, degrading performance shouldn't be an issue but all drives degrade to some extent.
The C300 probably needs TRIM to perform at it's best, I'm not sure how GC is handled.
(idling overnight might do the trick)
When I'm creating the partition in windows should I match the stripe size with the unit size or leave unit size in windows as default (I think default is 4096)?
I suggest using the standard 4KB allocation unit size on SSD's.
http://i787.photobucket.com/albums/y...OKE/92601d.png
That's 1 drive with your usual 9260 r0 settings from earlier only thing I turned on cache, I bet that skewed it, I have to go out for an hour I'll fix it when I get home if you need something different. (64k stripe)
@computurd I somehow missed your post earlier, thanks! I had been testing between 64/128 stripes and now I see they recommend 256 so tomorrow should be fun, I've consistantly gotten better results with 64k though, so we'll see, also I'm dissapointed that 4k randoms don't seem to scale much at all.. here is 8x drives 64k stripe/no read ahead/write back/cache: http://i787.photobucket.com/albums/y...POKE/group.png
Thanks steve, I don't think I'll get too close to your acard scores though. I definitely like my ~2tb though.
you need to run PCmark vantage!
My vantage score was 7k or whatever off mike's wr, I don't remember and I'm too lazy to go look at the record, it's the nearest 10 thousandth to the wr, 60k or 70k one of those I don't recall. I will do a hardcore push for vantage whenever I get my loops finished and the 980 finally shows up, 60/70k was with 920@4ghz/5870/9260. I will have 980 at hopefully 4.5ghz or close/crossfire with 5970/5870. I should be able to put up quite a good number.
This is windows dynamic striping on the 9211..
http://i787.photobucket.com/albums/y...KE/dynamic.png
I'm pretty happy with this though as I whined about earlier 4k randoms don't scale at all.
I'm sure I read I can make this bootable but does anyone know of any way to winsat this configuration?
Edit: actually looked at vantage scores and I don't know why I thought 60/70k, 20k was my score, I'll throw it up again later.
[QUOTE=mbreslin;4285548]My vantage score was 7k or whatever off mike's wr, I don't remember and I'm too lazy to go look at the record, it's the nearest 10 thousandth to the wr, 60k or 70k one of those I don't recall. I will do a hardcore push for vantage whenever I get my loops finished and the 980 finally shows up, 60/70k was with 920@4ghz/5870/9260. I will have 980 at hopefully 4.5ghz or close/crossfire with 5970/5870. QUOTE]
yep me to on the 980 - hopefull for a 4.5.
To run winsat against a dynamic disk - you should be able to just format the d drive via windows disk mgt and identify the drive letter in the command line like any other disk.
jesus that is impressive. WOW !! that 9211 softraid is UBER
did you do the full vantage run, or just the HDD suite?
you should at least be able to be in the top twenty.
Oh I did full vantage, I'll have to buy it to do the hdd test, I will do this later this evening.
@steve, winsat fails and says this drive involves more than one physical.. :/
i was also curious if you had placed just one device on the controller and benched it to see if it is performing above 3g? does the controller see it as a 6gb/s device and treat it as such? your results are great keep up the good work~!
This is single drive 9260, I actually tested each one because I thought one was bad, seems fine now though. "only thing different is cache, I bet that skewed it" I said this then because until then I hadn't turned cache on with the 9260 (I didn't think you were supposed to with ssds, obviously I see now that the 9260's cache shines) The 9260 sees all of my drives as sata 6gb devices, (and it's obvious in this bench from well over sata ii speeds in reads).
Also, just to be clear, I'm not done with benches or anything and I'll be doing different array sizes to see scaling, maybe if steve or someone will give me a standard idea of what benches to do so he can easily arrange them I'll start them soon, I just only have so much time and I've tried to cram everything in.
My verdict so far is that the drives are pretty awesome..
4k random doesn't seem to scale at all, this is dissapointing.
On the 2 controllers I have and ich10r latency seems to be about the same, I knew this going in, I don't see what can be done possibly crucial updating firmware but this seems doubtful, I believe I read a mod post saying it was unlikely there would be a fw update anytime soon. Is my array worth what I paid in performance? Probably not, certainly not looking at steve's acards, but they sure are fast and it doesn't look like something much faster is coming anytime this year so that's good enough for me.
Minor funny thing about the drives, when you take them out of the bag within minutes those pretty crucial stickers on the drives peel up and don't stop peeling until they litterally become unstuck from the drives, it's hilarious, this is not a heat issue or anything like that as it's just ambient temps drives aren't plugged in yet or anything, just cheapo branding sticker, or cheap adhesive or whatever I just thought it was funny.
http://i787.photobucket.com/albums/y...E/dynamic2.png
windows dynamic striping on 9211, quite a good score I think.
Edit: dynamic disk is silly but according to tilt I can get these kind of iops without dynamic disk, I'll try in a little bit, lsi didn't lie, the 9211 can push 260k :) (this is 4k random read)
http://i787.photobucket.com/albums/y...WPOKE/lols.png
Vantage HDD scores go here..
9211: 64640
9211 w7 dynamic striping: 70857 (I haven't seen anyone else's HDD suite scores but I bet this score is uber, too bad this drive isn't bootable/wei-able)
Mr. mbreslin, great scores!
Maybe you can't do winsat on dynamic disk - Anvil?
Tomorrow I will try to post some iometer setups that I like - the others you already have - AS SSD, CDM, ATTO, winsat, ... - maybe Everest.
CompuT - what bench program did you say was best for latency test?
Personally - I like iometer - it seems to be the most trustworthy.
Iometer is so configurable; the trick is getting consistent test setups for comparisons - from one system to another.
That is why sharing iometer setup files makes sense for most of us - but you probably already knew all this.
@steve: I did and that's why I asked anvil for iometer config files, no you can't winsat dynamic disks, or boot from striped dynamic disks, so even though I get awesome results I will never use it for 24/7. My original goal was to spend a month testing/configuiring/building and then settle in to a 24/7 set and forget. The 980 will come and then I'll wait and see how the 1880 is then after that I'm done.
Here is the 3 IOMETER configs we use for testing in the norwegian forum :)
http://www.diskusjon.no/index.php?sh...ost&p=15357931 (IOMeter.zip)
Random read, random write and Workstation.
The benchresults are in the first post
http://www.diskusjon.no/index.php?sh...c=1149834&st=0
Exactly my plan also - 980 now and 1880 when it comes out.
However, I will never be done testing - testing is a favorite pasttime for me.
Also - I learn so much in the process - not only from the testing itself but also from the comparison and analysis when collaborating online as we do here.
Which 1880 will you wait for?
I'm holding out for the 24 port 4GB version this time.
Wouldn't it be great if MS could figure out how to make dynamic stripped disks bootable?
Dynamic disks behave differently in benchmarks, some work and some don't.
I did a few tests a while a go but I need to review the results to make sure what worked and not.
mbreslin,
According to AS SSD your array is not aligned but the scores are awesome :), whether a correct alignment would up the results I'm not sure but they probably would be.
I wanted to create a benchmark thread in cooperation with GullLars but he's been busy lately so I might do it myself.
(with iometer patterns, charts etc)
Here's a link to the iometer config file we've been using a lot lately. Link
Save the results to a separate file and post the results.
The test is read only (except for creating the test file) and performs QD tests from 1 to 128 on 512B to 64KB block sizes.
(total runtime is about 34minuttes)
You need to select the drive to test and set the size of the testfile. (4x the size of the cache on caching controllers, otherwise just use the default 1GB filesize)
I guess you mean the synchronous 4K random (QD=1)? If I understand that test correctly, it synchronously reads random 4K blocks. Since 4K is much less than your stripe (chunk) size, I think it will only be reading from one drive at a time (unlike, say, 512K which could read from 8 drives in parallel if you have 64K chunk size). If 4K random QD=1 is only reading from one drive at a time, then it cannot scale. Have you seen any results posted where 4K QD=1 did scale with additional drives in the array?
On the other hand, 4K random QD=32 can and does scale, since the controller and drives are more than capable of reading multiple 4K blocks in parallel, and moreso as you add drives to the array. Interestingly, I've noticed that with single SSDs, even the 4K random does not scale with QD (i.e., QD=1 and QD=32 are the same) with Indilinx, Toshiba and Samsung controllers, but it does scale with Intel, Micron, and Sandforce controllers.
Quite right at some extent johnw.
The 4K random write does scale, the 4K random read does not scale.
As fas as I have seen the raid0 decreases the 4k random read performance by 3 to 10% and in some rare cases %1.
http://www.xtremesystems.org/forums/...30&postcount=1
http://www.xtremesystems.org/forums/...34&postcount=1
http://www.xtremesystems.org/forums/...61&postcount=1
I have never seen a controller that scales the random read speed. The only thing that can scale 4k random read speeds are special controllers with cache like the Areca 1231ML-4G.
http://www.xtremesystems.org/forums/...0&postcount=46
Edit: http://www.newegg.com/Product/Produc...&nm_mc=OTC-RSS
It is not possible to run the winsat disk assessment Tool on dynamic disks.
Attachment 102042
"Unable to open the drive or device.
This drive may be allocated across more than one physical disk. Before re-running the Disk assessment, check that your drive is not in a software RAID configuration and is allocated on only one physical disk."
sounds like a very good point :)..!!Quote:
I guess you mean the synchronous 4K random (QD=1)? If I understand that test correctly, it synchronously reads random 4K blocks. Since 4K is much less than your stripe (chunk) size, I think it will only be reading from one drive at a time (unlike, say, 512K which could read from 8 drives in parallel if you have 64K chunk size). If 4K random QD=1 is only reading from one drive at a time, then it cannot scale.
mberslin - wow! 264k 4k random (1GBps) is one freaking outstanding results!
pcmark vantage: 20814
puts me like 14th or something in top 20 (not sure why it doesn't show up in that list?)
http://service.futuremark.com/compare?pcmv=293661
it will take time to show up. it will.
Excellent job bro! keep pushing :) i think the FastPath that is coming will help you tremendously.
@stevero---Everest is the benchmark i refer to. of the 'easy' GUI based benchmarks it is one of the best imo. also, it measures down lower on latency than other benchmarks (hdtune etc.)
here is mine
@computurd: my 5970 came today new vantage score: 22863
http://service.futuremark.com/compare?pcmv=293983
that puts me #6, i'm coming for the top!
Wow, excellent run and congrats!
Are you still on the 9211 or did you switch to the 9260?
I see it now - 9260 - well done!
and still only at 4.4!
You may be able to get some more out of your proc.
215 bclock may be a doable do if you can control the heat.
CompuT, Anvil - what else should he tweak?
It would be great to see a 9260 based system break into the top 5 (currently all Areca's).
I haven't even pushed it is the thing, though the temps are high somewhere near 70 which scares me a bit, but by not pushed it I mean I haven't tried anything that has even crashed me, my 4.4 is linx/prime95 stable, on thursday I'll be switching over to a 480 for cpu that will probably help, this cpu seems quite good.
Edit: somewhere near 70 at full load of course, idle in high 30's
I had a i7 965 that would run all day at 90 no problem - (when on air) - to prove prime stable I ran at 96 for 8 hours and it ran perfect both then and forever after that - no problem.
For benching - don't worry if you hit 90 for an hour or 2 - it shouldn't hurt it.
What is a 480? 4x120 reservoir?
edit - stupid me - radiator of course!
4x120 rad, I stuck it on a 120.3 because I punctured the 120.4 with too long of screw :(
I have the replacement now just being lazy about switching it, I don't have all my fans installed or anything, this thing should be a real beast when finished, hopefully this weekend or next.
i read the 81 posts! hahahahah nice setup mbreslin
120.3 is ok...double the fans :D
Yeah I have fans for push/pull only push fans are on now, this weekend or so will be 120.4 with push/pull for cpu/nb/sb and 120.3 with push/pull for 5970+5870
@steve also the 9260 is 256k stripe atm but my previous benches show it will perform better at 64k so all combined I should easily make top5
ok here is the thing..
Im comparing ur results with mine using the 9211 with 8 intels and i see this..
Green mine Red yours.
for me.. i would use the 9211 for Vantage :/ now its ur case
@hondacity: thanks! I have a build log though I haven't updated for a bit.. http://www.xtremesystems.org/forums/...d.php?t=246366
@tilt: I'd love to know where I'm going wrong with the 9211, can you tell me step by step what you turn off in w7 before a new bench or point me to the proper post, there is so much info out there and half of it seems like rubbish, thanks.
i dont have any C300 and i dont know who they responce to LSI's controllers. You have to experiment every aspect of ur pc. take notes use multiple settings and then compine them to make the perfect result. There is no guid unfortunetly about PCmark Vantage. My rig for expample is not performing at all in PCmark Vantage and belive me i have more RAW power from everyone here.
But for me 9211 with the latency that i have ( 0,04-0,05ms) is better than 9260. For gaming and everyday use and to live longer HDD controller is the 9260.
With the 9211 just leave the superfetch ON, that gain me 753 points. but that is just for me.
Good luck with the testing.
@mbreslin: Is it possible to test a single drive without cache and then 8 drives without cache using HDtune 4.01 read and write?
http://www.hdtune.com/
Here is an example: Single C300 drive taken from a review.
http://www.legitreviews.com/images/r...00_hdtune3.jpg
Thanks in advance,
@steve: http://service.futuremark.com/compare?pcmv=295688 24331 #4 in vantage still haven't really pushed anything, 920 still at 4.4, 2k or so points from putting 9260 back on 64k stripe size.
nice result. have you experimented with pcie overclocking? also have you seen if you can adjust the pcie packet expansion size on your motherboard? that can yield great results. try upping the voltage to your pcie bus if at all possible a tad. however make sure that you have it cooled effectively. oh man im racking my mind for helpful insight here...i will give more advice if i can think of it :)
please tell me you are on the latest firmware for the 9260-8i? that is crucial same with 9211 both newest firmwares are very very good! my best settings for vantage are Always read ahead, and with cache. try also to experiment with the NCQ both on and off, it can effect arrays differently. the main thing is patience and a notepad :) you jsut have to tweak like crazy if you have the time for it you can narrow it down to some pretty specific settings rather quickly, best of luck and man i am happy to see a 9260 up there in the top 5!!!!! great job man keep it up :)
Latest firmware/driver for both 9211/9260.
I will do a lot of experimenting and pcie is at 105 so I have quite a ways to go, I have no doubts I'll get there I just don't have a ton of time, tons of things I can still push, pcie can go much higher, I have a good set of dominator gt 2000mhz c7 only running at 1600 so I can push that, graphics cards are at stock and on water so should oc well. 980x ordered, I just missed stock somehow so will be a week or so until I get that, then push it to the max.
AND fastpath is still coming for the 9211 so that will be interesting as well.
#4 and lots of fun yet :)
Which controller is best to take for everday use?
Better take the LSI 9211 over the 9260?
Or better take the LSI 9260?
And is that onboard cache needed for ssd setups? Or preform its better without cache?
9260 seems to perform better for me, I think drive latency has a lot to do with it, and yes performance is much better with cache on for me.
can't argue with tilt's success with the 9211 and intels. Both seem great to me.
i have both and personally for me i use the 9260.