RAID5 file server build advice

[XS Janus]

I did some 8 drive tests and they were better.
I'll post results soon.

@Gruber: I will try how the controller works in 16x slot. But I was told on 2cpu thread that there shouldn't be any problems.

@jcool: I went for the lowest possible power option. the XOR you mentioned is very fast. I would very much like you to make a RAID5 config of some sort and do some back to back comparison with 9650 you have.

Speed will ot be a limiting factor in media fileserver. The only time it will be written on it will be when I offload some PC on it, and that will be over network so I won't be seeing >60MiBs anyway.
Read speeds wit 8 drives are more than good.

The only thing that coulld be benefitial IMO would be faster raid level migration, rebuild times and verifications. And I really wich someone could make those tests as those numbers are nowhere to be found.

And please do this on both controllers: make a large folder with pics and such files and copy it on the array itself. THAT is the ultimate test. R/W at the same time! Do the same with a large 4GB single file (best would be video).
If you can't do Raid5 then at least Raid0, but that wont tell us much how much difference is between each XOR when calculating parity. :/

[jcool]

Yeah, I will get you guys some benches if I can find the time. Right now I'm trying to figure out whether to get a few more Savvios to go Raid 5 or 10 with my mainrig. I got a hell of a deal on them (about half the price), only "problem" is that I'm totally satisfied with single drive performance as it is. Never been running so smooth before, the combination of a 15k drive and totally overpowered XOR with cache is just too great already.

I'm planning on a full encryption on my fileserver, that's why I'm gonna go the speedy route. I'm not gonna have mission-critical data lying around in the open.

[XS Janus]

OK, here is the most important stuff:

Real world results comparing 8 drives and 4 drive RAID5 using all the stripe sizes available to me.
Improvement was tangible, especially in simultaneous R/W usage.


vs.

I light of this the smart thing would be to choose a 256k stripe for a video fileserver, no?

[Mr.Guvernment]

i would of expected higher then 50ish MB on a raid 5, i know things are toned down for raid 5 due to the way it works... but that much? or is ICH9R just not good on raid 5 (come to think of it i recall hearing something like that, raid 5 and ICH9 just dont perform too awsome..)

[XS Janus]

What do you mean 50MB/s and what ich9R? Where have you read that?
These are done on a 3ware controller, and Raid5 with 8 drive is giving me 115-122MB/s i na simultaneous read and write operation. (copying a large file onto the array itself)
Read tests I cannot do cause I don't have a disk setup that can write the files faster than my Raid5 array can read them. Even the writes are capped by my single 150GB Raptor.

[Mr.Guvernment]

AID5 using 4xWD1000FYPS drives

HD Tach: Each setup run several times and using different stripe sizes, as indicated on the pics themselves.

54-60mb

[Gruber]

Janus : did you manage to try the 3ware on the G33M's x16 pci-e slot ?
I am eager to see if its working and eventually get that card

Also have you run any HD Tach / HD Tune / Atto benchmarks with the 8 drive setup ?

Atto's results with the 4 drive setup has some nice results.
Not to mention your test has been done with "real life" example of copying small/large files from to the same array,unlike many benchmarks which show only synthetic results.

There is a nice review of 9650se here :
http://www.xbitlabs.com/articles/sto...0se.html#sect0

The conclusion is a very confusing one

[XS Janus]

@Mr.Guvernment: You are right, I need to check that. I did the 8 drives too. need to see what it scores. But you are right, it does look very low. Like single drive performance slow! Strange cause the same setup scores !70MiB/ in real life and that is with R/W task.

@Gruber: I tried putting it in 16x, but I routed the cables already and it is to TIGHT to reach ---2mm to short, and my card was REALLY bent, so I dared not powering it up that way On top of all that it was HOT that day and after trying to force it I had to undress myself down to my shorts and just stop myself and quit doing what I was doing before braking something

I will try it when I come more mentally prepared for the task at hand, I thought it would be a simple task

I think I have all the benches (+ HDTune pro write tests) run on 8 drives as well, but only 256k, but that one seems to be the fastest anyway.
I'll post them soon.

[lefy]

can someone please remind me what tests I should perform on a new drive? in general and before adding it to an array. i thought I read something here about it, or maybe it was at avsforums, but I cannot find it for the life of me.

thanks for any help

[Surago]

Interesting thread this, been a great read for past 3 hours lol.

only "problem" is that I'm totally satisfied with single drive performance as it is. Never been running so smooth before, the combination of a 15k drive and totally overpowered XOR with cache is just too great already.

If you only have a single drive, then wouldn't the XOR engine be idle? As what would it be performing parity for?

[XS Janus]

@Gruber: The controller works in x16 slot well, detects the drives and all

My Raid0 and Raid5 results using the 4 drives were definitely off.
I had one drive fail on me and it was one of the first 4 I acquired and did the tests with.
This thread explains what happened to it: link

I'm re-doing the tests as i type this.
Raid5 256 stripe is now +10% in hd-tach - other stripe sizes were less
Raid0 16k stripe is now +13%
Raid0 256 stripe is now +10%
I will test some real life results to see if there are any differences.
Still those raid5 results of 70Mb/s in hd-tach seem low for 4 drives who on their own score ~55-60, especially since my read/write tests when I copy files around on the array are better than synthetic scores in hd-tach.

@lefy: After that episode I tested all the drives using WD diag extended test. I will also run them all through Seatools long tests just to be sure. Each test takes about 4-4.5h
If i was do do it all over again from scratch I would first test all the drives with long tests, then make various arrays and even do couple of RAID5 initializations and then redo all the extended tests on all the drives one by one.
One could hope that this 1-2 week test process and drive usage would weed out any weak links.
As my drive apparently died during my fileserver birth pains

@Surago: I'm glad we kept you busy
Regarding the XOR usage: I to think that you would stress that XOR only by hooking up lots of drives in Raid0 or by using fast SSDs. Of course for parity Raids like levels 5,6 it is a VERY good buy.

Recently I learned by talking to a guy on hardforum, that it really shows its strength when migrating you array. Like when you want to add one or more drive to your existing array. It does calculations 4 times as fast.
My 3ware calculates at 3-4MiB/ and 5805 card does that at a rate of 16MiB/s.
On a large array that would make a difference between risking your data for weeks during migration versus less than a week on a faster card.
Since I already bought mine I opted to build RAID5 with all 8 drives and not to risk some failure if i started with 4 and upgraded as i needed them.

[m^2]

Regarding the XOR usage: I to think that you would stress that XOR only by hooking up lots of drives in Raid0 or by using fast SSDs.

RAID 0 and XOR have nothing in common. Fast array stresses IO processor, but not the XOR part.

[jcool]

Interesting thread this, been a great read for past 3 hours lol.



If you only have a single drive, then wouldn't the XOR engine be idle? As what would it be performing parity for?

It does not
I meant to write IO CPU. But the XOR will get stressed soon enough, as soon as I get my Raid 5 up and running.

[XS Janus]

Which drives are you getting jcool?

and how many of them?

[jcool]

Well, I wanted to grab 4 Samsung F1 1TB. But since the 1TB F1 I used for a short-term storage (until the fileserver is up and running) died on me after only 2 weeks of operation, I won't ever go near these drives again. Stupid Samsung, lost 900GB of data, most of which wasn't backuped (yet)

I think I'll be waiting for WDs new 750GB and 1TB 334GB platter RE drives. WD RE's have never failed me or any of my customers (100+ sold, not one defective).

[XS Janus]

Then I must be one of the lucky ones.

First a 500GB GreenPower line drive that I wanted to use a a C:/ arrived dead. so I RMA that one.
and then soon as it came next issue:
In my post #161 you have a link that explains what happened to my 1TB RE2 drive, Thankfully building this thing is taking me so long so I weeded out the weak link, but still.
Let this be a lesson to all of us. Test Your drives HARD before you do anything important with them. I will grill the rest of mine until a replacement for a faulty unit comes.

[jcool]

Yeah, I'm thinking mHDD loop for a few days for each drive b4 I put my data on it from now on...
I think I'm getting more Savvios, at least these SAS drives don't die.

[lefy]

can someone please remind me what tests I should perform on a new drive? in general and before adding it to an array. i thought I read something here about it, or maybe it was at avsforums, but I cannot find it for the life of me.

thanks for any help

just to follow up found it here

After witnessing SATA/ATA drive failures first hand, I now do something along the lines of what Nexsan does. I put brand new drives through a DOD wipe, a week straight of SDStress, then a final DOD wipe. I find the same failure rates as I mentioned previously (failure rate 2/32 1wk of stress).

[XS Janus]

Thanks for the followup.
Anybody got a ling to download this SD Stress?
Or some idea how to use it?

[lefy]

Thanks for the followup.
Anybody got a ling to download this SD Stress?
Or some idea how to use it?

still trying to figure out that part myself

supermicro how to
msdn how to
it can be run under Driver Test Manager (DTM) aka Windows Logo Kit (WLK)
microsoft wlk info

I couldn't get the wlk to install. said "not enough storage is available to process this command". quick search suggested adding RAM. I upped from 2GB to 4GB and got the same error when trying to install. no dice yet

[jcool]

Hey guys,

almost forgot about this thread - almost
Got my fileserver up and running now, using the Adaptec 5405 and 4x WD6400AAKS SATA drives (dirt cheap but they rock).

Anyway, here's how it looks, full size raid 5, 256kb stripe, write cache enabled:





Really impressed with the controller here, single drives do around 90MB/s read/write.

If you want to see more benches, click the Adaptec 5405 link in my sig.. just scroll down, got raid 0, 10 and 6

[XS Janus]

Very good results!
That is one nice and modern speedy setup.
And for a right price to.

[jcool]

Thanks! You can have that, too.. just need another controller

[NapalmV5]

Adaptec 5405 4x WD6400AAKS raid 5

^ nice 0% cpu - dual 1.2GHz

vs mine:

hpt rr3510: 4x wd6400aaks raid5



edit: long bench vs long bench ^

[jcool]

Wow, your read speed's even better.. maybe new HDTach is faster?
Any idea how your highpoint fares with writes?