RAID And You (A Guide To RAID-0/1/5/6/xx)

[stevecs]

Actually, found a couple web documents here that may prove to be interesting to others creating arrays:

Draft write up of "silent" data errors (not only for disk but the entire chain of the process):
http://indico.cern.ch/getFile.py/acc...r&confId=13797

Failure rates on drives:
http://www.usenix.org/events/fast07/...tml/index.html

Another failure rate study:
http://209.85.163.132/papers/disk_failures.pdf

Pretty much shows that:
1) if you need to have data accurate you need to _ALWAYS_ verify it before, during, after, use as no subsystem will give you correct data 100% of the time.

2) MTTF, AFR, are not good paper specs to "buy" off of.

3) Older drives than rated lifetime or drives that have reported an error/reallocation should be replaced/removed due to the dramatically higher failure rates.

[Serra]

All good points, the google document is one I'll put emphasis on as something people should see (don't know why I didn't link it in my post).

To be fair though, most sources refer to enterprise use - ie. a home computer will be very unlikely to see a fraction of the same kind of read use, and an even smaller fraction of write use. I wouldn't bet my life on it, but I wouldn't be surprised if a good fraction of current drives could outlive me if used in conservative, desktop environments.

[CyberDruid]

Stll looking for "fishfacefignewton"

Thanks for your hard work--a nice legible FAQ.

[stevecs]

I must have never had a "conservative, desktop environment" at home then. I have gone through well over 200-250 drives in the past 20 years at home. Come to think of it maybe even more as the 4GB drives back in the late 80's I had a good 30 of those, and the 96MB drives from the early 80's I had bunches of those boat anchors. Then went to the 18's, 36's, 72's, then jumped to pata then sata drives.

Suffice it to say two items: you are probably correct that I am probably not typical of a home user, but there are probably others who are digital pack-rats on these forums. and two the documents show the statistical error/failure issues on drives (and barring a similar study for home-only drives to contradict due to environmental reasons all we can do is take this as indicative of the base drive failure rates) which would apply equally to the enterprise and homes.

[Serra]

Stll looking for "fishfacefignewton"

Thanks for your hard work--a nice legible FAQ.

Heh, reading this I just realized it got edited out when I made a text swap with a changed version I had on a notepad... but don't worry, it's back now

[Polizei]

Very nice. I was thinking about a RAID01 array, but this guide diverted me to RAID10. Thanks.

For that, I thank the OP with this. It may help some RAID newbies understand RAID.

[Machinus]

2e. (As A Result Of The Above) - Placing Partitions For Speed
Alright, now that we know that the physical outside of the disk (or the "start" as the computer sees it") is actually read faster, does anyone want to take a guess at what that means for you speed-wise? Well just in case, it means that partitions containing data which you would ideally prefer to give access-speed preference to (ie. swap and page file partitions) should be placed at the "start" of the disk for best performance.

This is independent of RAID, so does this mean that all first partitions will be faster than the later ones?

If I make two 75GB partitions, will the first one be faster because it's on the outer half of the disk?

[Serra]

This is independent of RAID, so does this mean that all first partitions will be faster than the later ones?

If I make two 75GB partitions, will the first one be faster because it's on the outer half of the disk?

In effect, yes. It should be noted though that transitioning between the two would be a pain... that is to say, don't split your partitions if you're going to use the "slower" (closer to the center of the physical drive) partition for background tasks or anything, you'll take an awful head seek hit.

As a more extreme example, someone here - I forget who - has I think 4x 75GB raptors in RAID-0 and actually partitioned them such that he only uses 20GB of each drive. Very expensive prospect, but he does get fantastic results from them.

With all that said, I do have to be sure to make sure to add in a reminder that this works because most of the time peoples drives are either not perfectly defragmented (pushing everything to the outer bits when possible) or - as will happen - even defragmented partitions still sometimes throw bits of stuff towards the middle for some reason or other. If you're only using say 4GB of space and it's all on the outside, partitioning your drive won't really do you anything. Mostly it's ensuring that you can state that over the entire size of your partition your access time is approximately equal, which you can't say with an unpartitioned drive.

For a good idea where to partition, you can effectively run a bench like HDTach and see where it starts drooping beyond what you would consider acceptable and just chop it there. I'm sure someone would also suggest calculating cylinder barriers and whatnot, but that's maybe going a little far.

[TopherTony]

wow i just sat down and read the entire thing, and I gotta say that was fantastic! I thought I knew about raid but this really helped clarify a lot a mysteries that I used to just wave off as 'its just RAID magic'

[Burner27]

If you are converting video from one format to another (i.e. DVD to Ipod, DiVX to DVD, ETC) what is the best scenario to use to convert:

Single HDD to another single physical HDD
RAID 0 Array to single HDD
Single HDD to RAID 0 array


Thx

[Serra]

If you are converting video from one format to another (i.e. DVD to Ipod, DiVX to DVD, ETC) what is the best scenario to use to convert:

...

As it should be a sequential read/write, I'd say probably one of the RAID-0 options... but which is going to depend on where the bottleneck comes in (read/write/or compute). If it's a read bottleneck (doubtful), read from the RAID array... write, write to the RAID array... compute, might as well go single to single for all the difference it will make. H.264 conversion for example is very processor intensive... other conversions, less so. Depends on the software being used and end format.

[Burner27]

As it should be a sequential read/write, I'd say probably one of the RAID-0 options... but which is going to depend on where the bottleneck comes in (read/write/or compute). If it's a read bottleneck (doubtful), read from the RAID array... write, write to the RAID array... compute, might as well go single to single for all the difference it will make. H.264 conversion for example is very processor intensive... other conversions, less so. Depends on the software being used and end format.

But never read/write to the same drive whether it is a RAID array or single HDD?

[stevecs]

Correct going to/from the same drive will hurt unless you have another bottleneck that's masking it from you (like say, your cpu in the transcoding process can't process the data fast enough). And depending on your direction of converting (ie, mpeg-2 -> h.264/mpeg4) you want the faster drive to be the mpeg-2. So raid-0 on that side and perhaps a single drive on the h.264 side. At the drive level it doesn't really care what the data is, just the size of the request. In this case you're compressing something more so you'll take say 4x the request size coming in (mpeg-2) and are going to be writing out 1x (assuming 4:1 compression/recoding). So your output drive will be doing 1/4 the amount of work.

[Sgt_Strider]

I'm really interested in setting up a RAID 0 array with 2 drives. I'm concerned about the lost of data when the array becomes damaged. How often does this occur? I guess the smart thing would be to save important data to either a single drive or a RAID 1 array.

[stevecs]

It occurs often enough that if you're worried about it at all I would not suggest you use it, or follow a strict backup regime and plan on having to restore your system. RAID0 is not for anyone who cares about their data.

[Sgt_Strider]

It occurs often enough that if you're worried about it at all I would not suggest you use it, or follow a strict backup regime and plan on having to restore your system. RAID0 is not for anyone who cares about their data.

If what you say is true then how can video editors cope with it? Or they use RAID 5/6 and or RAID 10?

[Polizei]

I would suspect they use RAID 10 when working with files, and then RAID5 or 6 for saving them.

Possibly RAID50 or 60 if they only want one array.

[Sgt_Strider]

I'm thinking of buying a DFI Blood Iron P35 motherboard sometime next week. The board said it supports RAID 5. Does it mean that it's a hardware solution? If so, how does the performance on this motherboard compare to a dedicated RAID 5 card? Would I get respectable performance gain with the onboard RAID 5 solution compared to a single drive?

[stevecs]

Video editors (at least professional ones) either 1) only used raid 0 for transitory data with the understanding that it will be lost, or 2) use raid-10 or raid-3. If it's only a single threaded app (like video capture et al) then raid-3 is used often. raid-0 seems to be in the popular mind-set for people here and for gamers looking at just certain types of performance metrics without regard to reliability or if that metric actually applies to their use (ie, hdtune ). It's an unfortunate trend that I'm seeing over the past 5-7 years where people are not truly understanding their apps nor their subsystems.

As for on-board DFI (or any other say <$500 MB) no, on-board raid on the DFI is a what's loosely termed as 'fake raid' Speed is hard to tell as it's very dependent on the on-board implementation and the type of raid. RAID-5 for reads would be similar to a raid-0 w/ 1 less drive. writes is what the killer is when you are writing less than a full stripe as in that case every 1 OS write = 2 reads & 2 writes for a raid-5 to accomplish. (raid-6 is 3 reads and 3 writes). Also with the on-board of fake-raid solutions they don't have any parity offload so your cpu is doing that work (for rebuilds & writes, there is no parity checking on reads when the array is not degraded).

[The0men]

Hello. I have read, and found your "fishfacefignewton" (lol you put it after you say to look for it, to make people read it again anyway? lol). It is a very informative post. Thank you for your considerable time to post this.

I have a couple of questions. Firstly, where does HDD indexing come into all of this (I have it disabled in msconfig) but I was wondering if this is helpful to performance, or only under certain circumstances, or it is just more microsoft rubbish and not usefull at all.

My other question is, while I'm normally pretty confident in using RAID-0 (Have been for several year) I have never come accross a problem. With my latest machine I did not bother to use RAID at first, then when windows was due to be reformatted I decided to go back to using RAID-0. (I had it once on this machine without a problem)
I now am getting BSOD's but I am not sure if it was because my RAID was a dodgy instal (I will explain) or if perhaps I damaged something when I pulled system apart to mod the case and instal new watercooling gear.
When I mention dodgy instal, at first it would not read the floppy I created, after already preparing to instal and deleting the old partitions and adding the 2 HDD's into an array. However I found out I could not remake the floppy disk, because not one of our four computers with floppy drives were working (the actual floppy drives would not even register an inserted disk).. So I retried the disk where I was up to (after pressing F6 to add drivers) and this time it recognised and worked. But during the install of windows it failed to find the file "IDECOI.DLL". So I just skipped that file. I then searched it on my motherboard drivers disk and found dozens of the same file name. So I took the one from the location "drivers/chipset/32bit/ide/winxp/sataraid" and used that for the instal. Perhaps this is what is giving me BSOD's, but on the off chance it is somehting else I dont want to reformat only to find I have the same error. Also if that is the error how can I stop this from reoccuring.

Any help would be greatly appreciated.


Just to add, I am using the exact same overclocking profile I saved onto my bios as I used to have, and it was totally stable before reformat, I am also running orthos now to make sure I do not have something undervolted or to much overclocked.

[stevecs]

"HD Indexing" or similar terms are generally application or OS based methods of just reading the data from the filesystem and storing it in another file that is contiguous on disk (ie can be read faster than doing a bunch of seeks or random i/o). It doesn't really have much to do w/ raid itself it's more of a general method to help cut down seek penalties on drives. At least the ones such as in msconfig that is.

Could really be anything from that description, but generally I would lean toward a overclock or MB issue (either bios settings causing instability or the MB/chipsets on the flaky side). Overclocks are not stable over the entire life of a system they change one setting that may work fine when you get it may be too aggresive and begin to show signs of data corruption over time or the added stress (mechanical (heat related et al)) is enough to push it over the edge.

I probably wouldn't really trust a system in that state and would start a regime of testing to see where the problem lies (set everything back to defaults, and start testing one type of setup/component at a time until you can reproduce the problem). Once it's found I would reload though from scratch as you may have corrupted data.

[The0men]

Well I will let Orthos run for a couple of hours, but unless I have seated my NB waterblock oddly (which would be easy thing to do) nothing seems to be out of the ordinary. Orthos has run for one hour now, CPU and mobo temps are cool, it is a cool evening, and the profile is actually my backed off overclock which I only normally use on wicked hot days, which today is not one of them. It is not the same as in sig, which is 24/7 stable, well before the changes. But anyhow, I will run orthos for a few more hours and see what happens. If it is fine then perhaps it is simpily I have used the wrong file to replace the missing one, if my NB is getting to hot normally windows restarts when using orthos in the first 15 minuts, and it has never led to a BSOD.

The problem normally occurs during normal internet surfing while on msn and listening to media player 10. I am using the older Internet explorer that comes with SP2 with windows. But like I said, I have never come across a single problem on this machine.

Perhaps after orthos I will do a error scan with HD tune or something.

Thank you very much for reply.

[The0men]

Well I orthose with overclock for four hours no problem nothing is hot. So then I use computer for normal use, and once again it crashes with IE, MSN and media player open.

So following your advise I put all setting's back to default apart from making sure raid is enabled again. So I orthos for four hours, and once again no problem. Open IE to look at XS, talking on MSN and listneing to music on media player, crashes once again even with all settings default.

The crash is not even occuring during a time of the computer being under load, just when I'm using it normally. It has only done since I installed RAID, though I have never ever had this issue, so I expect now that it must be because of a dodgy RAID install.

I have checked for errors using windows scan disk and HD tune error check, neither found a thing. I think my next step now is to just ref-ormat with no RAID this time and see if things go back to normal. I also thought it might be southbridge getting to hot, but just after crash it does not feel any more hot than ordinary. I guess I am at a loss of what to do so I will just re-format and hope for the best, and then maybe try round two with RAID-0 trying a different set of drivers.

[stevecs]

Sounds like either you have some corrupted files on your system (scan disk won't show corrupted files it only checks filesystem integrity not data integrity). Or you have a component that is flaking out (chipset, ram, cpu, et al). Orthos does not test everything.

Try a fresh install of the OS and see if that stabilizes.

[The0men]

Yer thats all I can do, plus without RAID, just to be on the safe side, If thats all bad still, I will re-seat NB waterblock, because I know from previous experience It cooks like theres no tomorrow. I also thought because of how petty the cooling is on this motherboard maybe even my SB is getting to hot, it didn't seem to hot though, and normally theyre not prone to getting hot.

Well I will try fresh install, then re-seat NB. And see how it happens. I do hope its just my chopped up raid installation.

Thank you for advice Stevecs