RAID And You (A Guide To RAID-0/1/5/6/xx)

As some of you may recall I had meant to post this very thing around March... well, here it is July and it's finally starting to get put up. I blame this on a combination of work, school, and losing the folder I had 3 of these posts already written in. And Neptune, that planet's never done anything for me.

I won't call my posts as thorough on the topic as an RFC - RFCs exist for a reason and are freely available. My goal here is to provide an understanding of the various RAID levels in enough detail that the average OCer can understand the pitfalls of RAID-0, learn the overlooked possibilities of RAID-1, and know why they need more advanced hardware to make effective use of RAID-5. Still, *if after reading everything carefully* you feel I have omitted anything crucial (quite likely in these early stages) or have gotten something wrong, feel free to post here to let me know. I will be sure to examine every post relating to the quality of this guide and try to make changes accordingly or at least justify why I did not make a change.

Some of you may wonder why I don't do a summary of all these RAID levels and technologies somewhere and basically just tell you how to set things up. The reasons are simple: tables and charts do not provide an understanding of the technology and it's not my job to tell you how to do things. All this guide is meant to do is inform you about different technologies available and allow you to draw your own informed conclusions as to what it best for your particular situation.

Master Index:
Post 1: Overview
Post 2: Revised Storage Technology Guide
Post 3: RAID-0
Post 4: RAID-1
Post 5: RAID-5 & -6
Post 6: RAID Level Combinations (coming soon)


Please note that this is NOT a "What To Buy" guide. Although I may make mention of some particular hard drives or controllers this post concerns the only the technologies used in hard drive systems. For advice on what controllers to get, check out the latest benchmarks available. This guide is intended to be informative only and that will not change until a controller company steps up and sends me something nice (j/k - or am I?).

** Heavy editing and additions to come, this first edition simply has RAID-related topics cut out of the original **

This section is meant to complement the rest of the guide by providing a primer for the technologies that are used in the background in RAID arrays.

Index:
Section 0: Some Quick Definitions To Get Out Of The Way
Section 1: Hard Drive Specification Types
Section 2: Physical Data Placement And Speed (A Must Read For Page File / Swap Partitions)
Section 3: Controller Duplexing and "Hot-S" Failover Mechanisms
Section 4: (Almost) Universal RAID Primer
Section 5: The Windows Page File [aka Virtual Memory]
Section 6: FAQ
Q. I have a SATA II drive and a motherboard that only supports SATA I. Will I see a boost in speed if I get a board that supports SATA II?
Q. Will I see a difference between a drive with an 8MB cache and a drive with a 16MB cache?
Q. What is Perpendicular Recording?
Q. I'm currently using a RAID-(whatever) array, Why would I still need backups?

Section 7: References / Further Reading



Section 0: Definitions Used
1. Access Time - The time it takes for the heads of a hard drive to get *to* the data, not to read it.
2. Read Time - The time it takes to read data once the heads have gotten to it.
3. Sequential Read - A perfectly defragmented disk reading a file reads blocks sequentially, in that it can read blocks A->Z in a single pass without having to find the data somewhere else on the drive
4. "small" file - For the purposes of this guide a "small" file is considered to be a file of such a size that it takes the heads longer to get to the starting point of the file than it does to read the file itself.
5. "large" file - This guide will consider a "large" file to be a file of such a size that the time it takes for the heads to reach the beginning point of the data is trivial compared to the time it takes to read it from the platter.
6. Random Seek - This term is actually quite rare in this guide, but prevelant in many others. In essence it is usually meant to suggest a the operation of looking for a file. This guide uses the term "small file" instead because I believe there is a crucial semantical difference between the two relating to the fact that if the seek time does not outweigh the read time it is not the major source of latency and therefore not the largest issue.



Section 1: Hard Drive Specification Types
1a. ATA/EIDE/PATA [hereby referred to as PATA]
1a-i. Master/Slave Relations
1b. SATA I/II
1c. SCSI


1a. ATA/EIDE/PATA [Speed <= 133MBytes/s (or <= 1Gbit/s)]
For various historical reasons, you may find these drives labelled as any of the above. For the sake of my wrists, we'll just call them the most appropriate (and currently manufacturer supported) PATA drives. These are the last-gen drives we all all cursed because the thick ribbon cables would frequently interfere with our airflow or just look plain ugly. The speeds of these drives varies according to which exact part of the standard the drive is from, but realistically you aren't particularly likely to see anything slower than ATA/133 speeds (133MB/s maximum transfer speeds) nowadays. Of all the drive types, this is the only one to which a Master/Slave configuration matters.

1a-i. Master/Slave Relationship
As noted above, a Master/Slave relationship is important ONLY when using PATA devices. Basically, for each channel (PATA plug on your motherboard), there can only be 1 "master" drive (including cables with a hard drive and CD-ROM plugged in). The actual use of this relationship is now out-dated, but it is good to know because if configured incorrectly, may cause 1, both, and - possibly - neither of your drives to stop working. To make this long and irrelevant story short, there is only one method you should ever use when configuring these devices: move your jumpers to "Cable Select". Modern PATA cables have a pin blocked for one device to indicate to it that it is the slave device and this configuration issue has not made a difference to how things operate since '97 or earlier.

1b. SATA I/II [Speed = 1.5Gb/s & 3.0Gbps]
SATA I/II drives use a different cabling standard that their PATA predecessors. Instead of a bulky, difficult ribbon cable, they each use a streamlined serial cable. Physically a SATA I and SATA II cable are identical (and interchangeable), the only effective difference lies in the ability of your motherboards chipset and hard drives ability to operate at one speed or the other. Thus electrically SATA I devices have a maximum theoretical operating speed of 1.5Gb/s, where SATA II devices have a maximum theoretical operating speed of 3Gb/s. Being fully backwards-compatable with SATA I, SATA II devices can be plugged into SATA I ports (and vice versa), though they will operate at the speed of the lower standard. Realistically, as of March 2007 no commercially available hard drive I am aware of is able to saturate even at burst the SATA I bandwidth, so lets just be clear right now that the WD Raptors are not slowed down by being only SATA I devices.

1c. SCSI [Speed = Dependant upon standard and number of devices]
SCSI technologies are beyond the scope of this guide, and so will only briefly be mentioned. These are the drives every enthusiast dreams of: 10,000rpm+ drive speeds, enough bandwidth to fit 4 drives utilizing their full speed before any bottleneck occurs, and a heat tolerance that would make a Prescott P4 jealous. Of course, the considerably higher cost, lower capacity, and specialized settings/requirements have ensured that this drive is off-limits to most individuals. Beyond this, there are so many different SCSI types, cables, and considerations that one could (and many have) devote entire web pages to this topic.


Section 2: Physical Data Placement And Speed
2a. A Quick Guide To Rotational Platter Physics
2b. Multiple Zone Bit Recording / Zoned Bit Recording
2c. CAV (Constant Angular Velocity)
2d. CLV (Constant Linear Velocity)
2e. (As A Result Of The Above) - Placing Partitions For Speed


2a. A Quick Guide To Rotational Platter Motion Physics
In order to properly understand the rest of this section, this is pretty much a "must read". Luckily, it's not a hard concept. Okay, the basic premise is that if you have something carving out a circular rotation, the matter farthest from the center of rotation will be moving faster than the matter closer to the center. Why? Well because the matter farther from the center requires the same amount of time to create one full revolution, but it travels a greater distance. Hence, it must be moving faster. Now apply this to a hard drive. What you basically get is that *assuming* the platter speed remained the same accross the entire volume regardless of head placement, the outer edge of the disk will be going notably faster than the inside. To help muddy the waters, computers actually see hard drives outer edges as being the "start" of the disk and records inwards towards the "end".

This is one of the actual primary reasons why disks seem to slow down as more information is put on them... it takes more time to get the same amount of information from the sections of the disk closer to the center than it does from the outside (though the discussion on this gets deeper in the next subsections).

2b. Multiple Zone Bit Recording / Zoned Bit Recording
Effectively, this is the name used to describe how data is organized on a hard drive (in tracks and zones) and why the data density is at least approximately the same throughout a disk. I would explain in detail, but I was able to find a webpage devoted to the topic which not only covers it better than I could, but *also* includes benchmarks (albeit on a 3.8GB disk... but the idea should be the same). The webpage is at: http://www.pcguide.com/ref/hdd/geom/tracksZBR-c.html

2c. CAV (Constant Angular Velocity)
Basically this is the acronym used to describe most hard drives since the early 90's and many (but not all) CD drives nowadays: the disk rotates at a fixed speed regardless of where the read/write heads are.

2d. CLV (Constant Linear Velocity)
This is the older style of hard drive design wherein depending upon which zone is being read the rotational speed of the hard drive changes. Effectively, this means that your hard drive has a "max" speed, but slows down sometimes. As far as I know this is no longer used in any medium (instead now the drive stays at the same speed, but less is accomplished as the heads move inwards on the drive (or towards the end of the drive as the computer sees it)).
This is used in some CD drives but should no longer be seen in hard drives.

2e. (As A Result Of The Above) - Placing Partitions For Speed
Alright, now that we know that the physical outside of the disk (or the "start" as the computer sees it") is actually read faster, does anyone want to take a guess at what that means for you speed-wise? Well just in case, it means that partitions containing data which you would ideally prefer to give access-speed preference to (ie. swap and page file partitions) should be placed at the "start" of the disk for best performance.


Section 3: Controller Duplexing and "Hot-S" Failover Mechanisms
3a. Controller Duplexing
3b. Hot-Swap
3c. Hot-Spare


3a. Controller Duplexing
This is a topic which can be approached from two directions: redundancy / fault tolerance and speed. The basic physical idea behind both is that if you are planning on building an array (or combination array) of disks, instead of plugging all disks that will be used into one hard drive controlling device, you could use multiple controllers instead and span the disks between them.

Commercial environments that have strict uptime requirements for example often use this because although they may have implemented a RAID array to prevent any one disk failing from affecting their uptime, if the array controller dies, it doesn't matter if the data is still retreivable - the data can't be accessed. A simple example could involve two PCI hard drive cards in a computer with a two-disk RAID-1 array. Instead of plugging both hard drives into one card, each hard drive is plugged into its own card instead, thus preparing the computer for the failure of the hard drive or hard drive controller.

But this is Xtreme Systems, so let's focus more on the speed issue. Lets say that you have the money and desire to build a RAID 0/5 array (data is broken into two stripe streams, each of which is stored in a seperate RAID-5 array, providing redundancy to the data) with 6 disks. Now you could go out and find a controller that can handle nested RAID levels for you and that has the power necessary to do all the calculations for each of your 3 arrays... or you could get two controllers and unload half the work to each. Basically, you would set up 3 drives on each controller as a RAID-5 array, and then create a RAID-0 array between the two controllers (which will each appear as a single drive to any external card/onboard solution/software solution). By doing this, you can not only buy much cheaper cards (each card can afford to have lower specs and fewer ports), but you get a speed advantage as well. Because neither card only has to worry about its own RAID array (not both) the latency for calculations is effectively cut in half, any onboard memory the controller card may have is now effectively doubled, and the total bandwidth is as well (each card having its own connection to the motherboard). While certainly an expensive proposition, controller duplexing is a must for any enthusiast with more money than things to do with it.

3b.Hot Swap
Hot Swapping is the ability to remove a component and replace it while the computer is still running. Depending on the system, this could realistically apply to any component (though for a motherboard replacement, you're looking at more of a clustinger technology rather than a single-computer environment). In reference to a discussion on hard drive technologies, this would mean that your hard drive controller allows you to remove a dead disk and insert a new one in its place and have it detected without shutting down the system. This is the most useful in environments where hot sparing is not an option, uptime must be maintained, and a fault-tolerant RAID array exists (though the last part is optional). For examples and special considerations, see subsection 4c - Rebuilding Failed Arrays.

3c.Hot Spare
Hot Sparing works somewhat like Hot Swapping but more automated. When a controller is configured to use Hot Sparing, once a hard drive failure is detected a user-predetermined disk (or set of disks or array of disks) that was previously unused by the controller springs into action and replaces the failed disk. It should be noted here that although the disks functionality has been automatically replaced, once the disk fails you no longer have the spare set to use if another disk fails, and therefore it is adviseable that you remove the dead disk and replace it with a new, working one as soon as possible and configure your controller to use the new disk as the hot spare.


Section 4: (Almost) Universal RAID Primer
4a. What is RAID (as a whole)?
4b. What is a RAID controller?
4c. What Does "stripe size" Mean?
4d. What is the best stripe size for me?
4e. A Quick-and-Dirty RAID Level Comparison


4a. What is RAID (as a whole)?
RAID stands for Redundant Array of Inexpensive Disks. In older literature the "I" used to stand for "Independant" in most publications, but given the decline in prices over the years the word Inexpensive eventually came to the forefront. Nomenclature aside though, all levels of RAID have a common goal: to team together multiple physical hard drives to provide better performance and present this team (or Array) as a single disk to the Operating System. This is not to be confused with JBOD (Just a Bunch Of Disks), whose purpose is to bundle multiple hard drives together logically but which simply fills one, then the other, then the other, etc.

RAID has a number of flavors, or levels, and each level has a number associated with it. Today RAID levels 0, 1, 5, and 6 are common, while 3, 4, and 7 are not. Despite the naming convention, of all the levels that exist, level 0 is not actually redundant in any way.

4b. What is a RAID controller?
A RAID controller is either a hardware or software device which performs the operations necessary to bind the multiple hard drives together, makes them work according to the various RAID specifications, and presents the array as a single hard drive to the Operating System.

Software RAID controllers (as found in most onboard solutions and add-on cards that cost under $300US) use the computer processor to make any calculations necessary to perform this function. For some RAID levels this is acceptable as their processor overhead is quite low, while for others it is unacceptable.

Hardware RAID controllers on the other hand contain their own processing engines capable of performing all of the tasks required. This eases the burned on the rest of the system and provides a faster response time. Most hardware controllers also contain a certain amount of cache, and many contain RAM (often expandable) which allows the controller to aggressively seek data it thinks may be requested and supply it to the computer at a speed greater than would otherwise be possible from querying the hard drives themselves for it.

4c. What Does "stripe size" Mean?
When creating a RAID array one of the first choices a person has to make (after how many disks will be included) is what stripe size to use. The "stripe size" basically refers to how large the blocks will be that data is broken up into before writing to a disk (and, conversely, how large each block of data to be read from a disk will be). The values available are typically between 16k and 256k, with many manufacturers recommending a value of 64k be chosen. This may not seem like a terribly relevant piece of information, but it will become more valuable when we start drilling down into the specifics of RAID.


4d. What is the Best Stripe Size for me?
I'll say it right now - I hate the topic of chosing a RAID stripe size. Everyone wants a clear-cut answer for what their stripe size, but the honest truth is that there is none. The best I can really offer is that companies such as Areca, Promise, Highpoint, and Intel have all conducted their own tests and have all independently concluded that they should set their products to use a 64k default. From there it's up to you as to whether you want to go larger or not, and I'll offer some guiding tips on how to decide:

Quote:

---

Originally Posted by From Soulburner, not sure of original source

Decreasing Stripe Size: As stripe size is decreased, files are broken into smaller and smaller pieces. This increases the number of drives that an average file will use to hold all the blocks containing the data of that file, theoretically increasing transfer performance, but decreasing positioning performance.

Increasing Stripe Size: Increasing the stripe size of the array does the opposite of decreasing it, of course. Fewer drives are required to store files of a given size, so transfer performance decreases. However, if the controller is optimized to allow it, the requirement for fewer drives allows the drives not needed for a particular access to be used for another one, improving positioning performance.


So what should you use for a stripe size? The best way to find out is to try different values: empirical evidence is the best for this particular problem. Also, as with most "performance optimizing endeavors", don't overestimate the difference in performance between different stripe sizes; it can be significant, particularly if contrasting values from opposite ends of the spectrum like 4k and 256k, but the difference often isn't all that large between similar values.

---

4e. A Quick-and-Dirty RAID Level Comparison


Section 5: The Windows Page File [aka Virtual Memory]
5a. What is a Page File? Why do I need one?
5b.Okay, I've decided to have a Page File... what are the best practices?


5a. What is a Page File? Why do I need one?
A Page File is a volume on your hard drive that your operating system treats somewhat like RAM, in that it stores pieces of applications that it may need while the application is running. This is important because the majority of programs that exist do not use all of their code at any one time, and so loading the entire program into memory would be an extremely inefficient use of prime real estate (and indeed, it program may not *fit* in the first place). Consider for example your operating system. While some of you may have enough RAM in your rigs to handle it, *I* for one couldn't load all the parts of my operating system that are put into RAM/the page file and any useful application into the amount of RAM I have. It also comes in handy in the event of an unexpected system shutdown. If possible, Windows will try to save what is currently in its RAM to the page file so that you do not lose any work you had been doing. Lastly, if you are a mobile user who uses the hibernate function, the page file is where all of your RAM is copied when going into hibernate mode.

The question of whether a person should use a paging file is one that comes up with regularly around XS. Strictly speaking, no you do not need a paging file to run Windows, and this is where a lot of confusion comes in. People feel that because they do not have a paging file programs are loaded entirely into memory and/or that some form of overhead related to a system process is eliminated. These are false suppositions. Without a paging file your program is not loaded entirely into memory and there is no service you disable by not using one (it's simply not used, the algorithms are still calculated). What is your result from not having a page file then? For starters you have less control over where your less frequently accessed bits of applications are stored and can find that they are located at slower ends of your hard drive or that the parts that would have been loaded into your page file are fragmented. Further, for those of you who have multiple hard drives, your page file must be read off of your primary (and theoretically more heavily loaded) hard drive, which further increases read times. For an explanation of these downsides (and how to correct them) see section 3b.

5b. Okay, I've decided to have a Page File... What are the best practices?
After the question of whether one needs a page file or not has been addressed (3a), there are three primary Page File best practices left to go through before deciding upon an implementation plan: Size and Location.

Size:
Microsoft suggests that your page file be set to 1.5x the amount of RAM in your system. They suggest this particular rule of thumb because a hibernation or system dump can copy the entire contents of your RAM into your page file (which may have also had other information in it) and because systems which have insufficient RAM may require a large page file if they are using an application that requires application data be RAM-resident. Frankly, with the cost of hard drive space nowadays, I say go with 1.5x. If you happen to have 4+ GB of RAM and don't feel that you don't require 6+ GB of Page File, you're probably right.

Regardless of the size you feel your Page File should be, it is suggested that you set the maximum and minimum sizes be set to the same value. This prevents Windows from dynamically chaging the size of your page file (which you don't need, because you already figured out how much you need - right?) which can result in fragmentation and unnecessary disk overhead at critical times (ie. making your page file larger when it needs it to be - if it was large enough in the first place, that's a step that would not need to happen).

Location:
You have three options for the location of your Page File. The first is to put it on a different partition on your primary (boot) drive. By locating your Page File in its own partition (according to Microsoft) your page file will not become fragmented. Whether that's true or not, by partitioning your hard drive with a seperate Page File partition you do get to put it at a marginally faster part of your hard drive. The second option you have is to put your Page File on a second, less frequently used hard drive. This is an excellent option because (obviously) without any applications or systems using the hard drive, your Page File will have the full bandwidth of the hard drive bus at its disposal. Your final option is to split your hard drive accross two hard drives, with some on your boot partition and some on a seperate partition of your second drive. According to Microsoft this is the *best* option. According to the Triforce of Wisdom employed by the technical writers in the Microsoft Technet, the algorithm that decides where to put data to and from will always put data on the hard drive which is used less frequently. Personally, I really don't put a lot of stock into this. It also offeres a benefit useful for code debugging, but I'll leave that as being out of the scope of this guide.

Please be sure to also read the section on physical data placement and its affect on speed if you plan on using a seperate page file partition.

Section 6: FAQ

Q. I have a SATA II drive and a motherboard that only supports SATA I. Will I see a boost in speed if I get a board that supports SATA II?
A. The short answer: No. The longer-than-short-but-not-long answer: Yes, slightly. The long anwer: Although there will be an increase in speed, unless there's something very wrong you shouldn't notice it. This is because, strictly speaking, consumer-grade hard drives are not able to actually read data from their platters fast enough to saturate even the PATA/133 bus yet for a sustained period of time, let alone a SATA I / II bus. At best, most 7200rpm drives can peak at PATA speeds when bursting, but sustained speeds are significantly lower. With that being said, your data will *technically* move its way from the top of the SATA cable to the bottom almost twice as fast, but that is absolutely not the bottleneck of the system and so the difference is effectively zero.

Q. Will I see a difference between a drive with an 8MB cache and a drive with a 16MB cache?
A. Like anything else with hard drives, it depends. If you just cruise around the Internet on your computer you could probably have a 1MB cache and never really notice. If, on the other hand, you do other work that involves predictable reads, you will see more of a difference. For clarity, when I say 'predictable', I mean your computer anticipates what data it needs and holds it in cache before you specifically request it (ie. it takes the cluster you asked for and gives it to you, but may hold the next few clusters in cache because chances are you will want them too). But will you see the difference in your everday life? If your a gamer you may notice something depending on your games of choice (but I wouldn't hold my breath), if you do a lot of file manipulation it certainly won't hurt, and if you're all about benchmarks then definitely yes.

Q. What is Perpendicular Recording?
A. http://www.hitachigst.com/hdd/resear...Animation.html
Sorry, I'm just not even going to try to compete with that video. The only thing I will add is that it also allows for faster reads as well because it covers more bits with every sweep of the same speed than it would have if the disk was not using perpendicular recording.

Q. I'm currently using a RAID-(whatever) array, Why would I still need backups?
A. Although you may have a super-redundant RAID-6+1+1+6+1 array with 12 hot spare drives standing by, all on the fastest controllers ever produced jam-packed with ECC memory, there's still a chance you'll lose your data. I'm not talking about loss from a single hard drive, or even a dozen hard drives all at once. What I am talking about is data corruption. If somewhere in your data processing an unstable system component (ie. overclocked processor or memory) produced invalid results and writes those to the hard drive, your data is corrupted and no amount of redundancy will bring it back. Similarly, RAID also doesn't do anything to hamper viruses. Your only defense against either of these is to make backups at regular intervals, and to be safe you should be sure to keep old copies for a period of time (because, for example, you may discover you picked up a virus three weeks ago and didn't notice... if you have overwritten your last backup, you're still going to be infected no matter what).

Section 7: References / Further Reading
http://en.wikipedia.org/wiki/Hard_disk
http://en.wikipedia.org/wiki/Parallel_ATA
http://en.wikipedia.org/wiki/SATA#SATA_3.0_Gb.2Fs
http://support.microsoft.com/kb/314482 <- Windows Page File Information
http://www.pcguide.com/ref/hdd/geom/tracksZBR-c.html <- Zone Bit Recording / How Physical Placement Affects Speed

Index:
Section 1: Overview
Section 2: How it Works 1 - The Basics
Section 3: How it Works 2 - More Advanced
Section 4: Side Notes
Section 5: Advantages (summary)
Section 6: Disadvantages (summary)
Section 7: Rules of Thumb
Section 8: FAQ (various)
Section 9: Future of RAID-0
Section 10: References / Further Reading


Section 1: Overview


RAID-0 is the simplest RAID level that exists, and runner up for the most misunderstood (see RAID-1 for that). The basic idea is elegant in its simplicity: all hard drives in the array work simultaneously to read and write all data. The primary advantage of this technique is obvious - namely that it can allow data read and write operations to complete more quickly than a single disk could without any disk space being lost or partitions needed. The disadvantages though become somewhat more technical in nature, and is why this RAID level is so misunderstood. The rest of this examination of RAID-0 will begin with the basics of RAID-0 then delving into the more technical details so as to provide an understanding of just what the disadvantages are and illustrate the many situations when a RAID-0 array is not the end-all, be-all of speed as many enthusiasts blindly believe.

To simplify my job, all examples used will assume an array with 3 hard drives. It is possible to create an array with only 2 (or for that matter more than 3), however it has been my experience that people need an example with more than 2 drives before they really understand, and 3 is the next most convenient number.


Section 2: How it Works 1 - The Basics
2a. Writing to the Array
2b. Reading from the Array


Fortunately for the author, the processes involved with RAID-0 are relatively easy to explain. This section is broken up into two main parts; how data is written to an array and how data is read from it (because a hard drive cannot do both simultaneously that situation will not be considered).

2a. Writing to the Array:
When your computer has generated data that it needs to write to a regular single disk, that data is simply laid down on the platter in its entirety. When data is to be written to a RAID-0 array however the data is instead broken into equal, fixed-size blocks by the array controller (either in hardware or software) before it is written. Once the data has been broken down into these fixed-size blocks the RAID controller simply sends them to each hard drive in rotating sequence until no blocks remain to be written. In this fashion each drive in our example 3-drive set needs only to do 1/3rd of the total work.

What happens next is best illustrated by example. Lets say you have a piece of data which the controller has broken into blocks A - I and you have 3 disks in your array. The result would look something like this:

Disk 1: A , D , G
Disk 2: B , E , H
Disk 3: C , F , I

And so voila - you have just cut the time it takes to write the data into one third versus a single disk! (or so it appears - be sure to read on)

2b. Reading from the Array:
Reading from the array works much like writing - each disk simply reads off what parts of the whole data set that it knows and the RAID controller puts them back together. Using the same example as above, Disk 1 would read blocks A, D, G, Disk 2 would read B, E, H, and so on. Again, much like above you *appear* to have completed the same operation as a single disk would in one third the time (but as above, be sure to read on - it is not necessarily as it seems).


Section 3: How it Works 2 - More Advanced
3a. Reading a "small" File
3b. Reading a "large" File
3c. Reading a "large" NON-SEQUENTIAL File
3d. Writing to the Array
3e. Reading & Writing from/to the Array (ie. uncompressing a RAR file)


Below you will find an explanation that goes into some of the finer details of RAID-0 read and write operations. This section strongly requires that you have read and understood the first How it Works section, as well as the introductory post about basic hard drive mechanics themselves. Unfortunately for the author once you dig down a bit into the mechanics of how a RAID-0 array reads data, you necessarily run into issues relating to the fact that read sizes and disk states greatly affect the outcome. This subsection must itself then be broken down into a number of different possibilities, and each examined independently. A summary will be provided later that lumps these details together however and will demonstrate how they will affect performance in the real world. Two cases that relate to both reading and writing from the array will be explained first.

Case 0: The array has spun down or is starting fresh
It is imporant to the operation of RAID-0 to note that in the event the array is either just starting up (ie. you're starting your computer and reading your OS from it) or has spun down (ie. you have not used your array in awhile and it spun down), the array will - on average - start up slower than any random single hard disk. The reason is fairly intuitive: if we assume that there is variation in spin up time among disks (and there is), the law of averages dictates that an array of multiple disks will probably have at least one disk that is slower than the other(s). The consequence of this is that because for an array to be useful all disks must be fully spun up, it will take longer on average for an array to spin up versus a regular disk AND the more disks there are in the array the longer (on average) this will take.

While this particular delay may not be too long itself, remember that the latency you experience is an accumulation of various events, and this will add into that equation.

This section assumes that disks are currently spun up, operating at full speed. See Case 0 above for an add-on about what would occur if they were not.

3a. Reading a "small" file
This is the most important scenario to consider when weighing the various RAID options in your mind. A "small" file for the purposes of this discussion is one which meets the following criteria:

The size of the file is small enough such that it would take a greater amount of time to reach the point where the data begins than it would to read the data itself once the head has reached the start point.

Now we will again use the law of averages. On average, lets say it takes your hard drives head 10ms to reach any random data point from any other random point on a disk. As more drives are added to your RAID array the law of averages suggests it is likely that one drive will reach the data faster than another, and the more drives there are in the array the more likely it becomes that there is a larger difference between the fastest and slowest drives in the array. It will now be given that a file cannot properly be read from a number of drives until all drives have arrived at the data points. As such, if we define a small file as one where it takes longer to get to the data point than it takes to read the data, then the logical conclusions are:
1. The largest source of delay in reading this file will be to get to the data itself and,
2. This delay will, on average, be longer in an array than on any given single disk and,
3. This delay will increase with the number of disks

3b. Reading a "large" SEQUENTIAL file
A sequential file is a file laid out on your disks such that if a disk requires blocks A, B, C, then they are laid out next to each other and no additional seeking is necessary after reading A or B, it can just continue reading. This occurs on well defragmented disks. This case is the reverse to the above. In this case we're assuming that a file is of such size that the time it takes to reach the data start point is trivial compared to the time it takes to read the data. Using similar reasoning as above, we draw two new conclusions:
1. The largest source of delay in reading this file will be to read the data itself and,
2. This delay will then, on average, be shorter in an array than on any given single disk due to the fact that multiple disks are breaking up the work and,
3. Performance will increase almost proportionately with the number of disks (be warned however that the minimum file size necessary for a large sequential read will increase at the same rate)

3c. Reading a "large" ***NON***-SEQUENTIAL file
This case is included for completeness only. As you may have realized, a non-sequential file is (unlike the above) a file that is fragmented, causing the heads of your disk to seek between file reads. In this case the amount of fragmentation will cause the seek latency issue to become a larger factor... if enough fragmentation occurs, you may even be looking at the issue described in subsection 3a.

3d. Writing to the Array:
Fortunately the cases for writing to the array are effectively the same as those for reading from the array. Just as with reading, seek time is the key component to the downside of RAID-0, which will otherwise scale fairly well (to a point) with the number of disks.

3e. Reading & Writing From/To the Array: (ie. unRARing files)
Another important note about the mechanics of RAID-0 is that it fails utterly when dealing with read/write operations (such as uncompressing a large file back to disk) as compared to a solution where both disks in the array are independent and one is used to read from and another is written to. The explanation for this behavior is simple: using the RAID array the operation will perform a portion of the read, then have to reposition heads to write a bit of data, then move the heads again, etc. very frequently and incurs a very large penalty from the seek time. Single disks, by comparison, have the freedom to simply read and write with no seek time hindrance.


Section 4: Side Notes
4a. When Drives go Bad...
4b. Software vs. Hardware RAID-0
4c. RAID-0 Performance Scaling With # of Drives


4a. When Drives go Bad...
You lose all your datas. There is no way to recover data from a failed array. And when you think about it, the chances of any one drive failing is FxN,where F is the chance of any one drive and N is the number of drives in the array... so the more drives you have, the more likely you are to lose all your data.

4b. Software vs. Hardware RAID-0
Because RAID-0 requires no logic beyond the division of data to be written into blocks of pre-determined size, this array type alone stands as a type which receives next to no benefit from a hardware controller versus a software-based controller. For example, an AMD Athlon 4000+ (single core) processor using software RAID over 3 drives only accumulates approximately 2-3&#37; more utilization than the same rig with a hardware controller... and given that recent system advances have come so far since that point, the current difference should be less than 1-2% for heavy I/O.

4c. RAID-0 Performance Scaling With # of Drives
One of the things I have always hated seeing here is people who have RAID-0 arrays with 7 or 8 drives. Aside from the clear danger of disk failure, RAID-0 may scale well with a few drives, but much less so as you add more. For example, versus one disk and assuming theoretical maximums:

1 Disk = Baseline
2 Disks = 1/2 Time Decrease = 50% performance increase vs. 1 disk
3 Disks = 2/3 Time Decrease = 16% performance increase vs. 2 disks
4 Disks = 3/4 Time Decrease = 9% performance increase vs. 3 disks
5 Disks = 4/5 Time Decrease = 5% performance increase vs. 4 disks
6 Disks = 5/6 Time Decrease = 3% performance increase vs. 5 disks

And one must take into account the fact that as the number of drives increase, so too does the minimum size of the file required to be considered a "large" file. Add in the fact that overhead alone accounts for a few % of performance and you can see that past 3 disks your *theoretical maximum* increase is sitting in the low to mid single digit range.


Section 5: Advantages (summary)

In short,
1. Fastest "large" file reads versus single disk or any other RAID type
2. Fastest "large" file writes versus single disk or any other RAID type
3. Trivial CPU impact even with software RAID (and hence #4)
4. Cheapest array type per GB (no hardware cost, no drives lost to parity bits)


Section 6: Disadvantages (summary)

In short,
1. Poor "small" file read performance
2. Poor "small" file write performance
3. No fault tolerance - one disk dies, it's all gone


Section 7: Rules of Thumb
7a. When to use RAID-0
7b. When NOT to use RAID-0
7c. One Thing you *DO* Need


7a. When to use RAID-0
Use RAID-0 when:
1. You do not care about fault tolerance
2. You work with "large" files often, except
2a. do not use RAID-0 if your work will see files read from and written to the hard drive in the same operation
3. You want a boost in startup time for many OS's (but this applies to many RAID levels, not just 0)

7b. When Not To use RAID-0
1. You care that your data will be lost if any drive fails
2. You're just an average desktop wanderer - most desktop work involves very small files
3. You like to do video or other large file manipulation from and to the same source

7c. One Thing You *Do* Need
You *DO* need a good disk defragmenting program. RAID-0 suffers with fragmentation, so something beyond the simple Windows Defragment tool is highly suggested. My personal favorite is Diskeeper, but O&O is another widely used program.


Section 8: FAQ (various)


Q: Is it true you shouldn't mix hard drive types?
A: Yes. You will be constrained by the slowest drive in your array so it makes no real sense to mix two different hard drives with different properties

Q: What's better, a 10k rpm drive or a 7.2k rpm drive for RAID-0?
A: For some reason some people think this may make a difference. With RAID-0 as with anything else, a faster drive is always faster.

Q: What's better, a single 10k rpm drive or an array of 2x 7.2k rpm drives?
A: For small file/average desktop use, the 10k rpm drive will be better because it is actively better for small file work, but the larger the files get that you're looking to play with, the more the array will start to shine.

Q: Will a RAID-0 array help with game X or application Y?
A: I have neither the time nor inclination to look up and examine ever game/apps file usage patterns and thus cannot answer any type of question like this.

Q: Why don't you ever define a file size for "large"/"small"?
A: Because it's fully dependant upong the hard drive in use. The faster the drive is for seeking, the smaller a file will have to be to be considered "small" and the faster it reads the larger a file will have to be to be considered "large"

Q: Why don't you provide more benchmarks?
A: Right now, this guide is new and some are coming. With that being said, to truly illustrate all points of all RAID levels comparatively, someone needs to invent a new benchmarking program. Until that happens, there really isn't a great application out there eyt for that kind of thing because it's all so heavily dependant on usage patterns, fragmentation, etc. Even a users hardware setup (less with RAID-0, but more with RAID 1/5/6) can make a monumental difference with the existence of or lack of a few key code optimizations. When in doubt, test it out.

Q: How do I implement RAID-0 on my motherboard/OS?
A: Given the hundreds of motherboards that exist and dozens of operating systems, I have no intention of even trying to explore this issue. RTFM.


Section 9: Future of RAID-0

As drives shift from platter-based to solid-state disks (and as solid state disks improve in quality/speed), RAID-0 will correspondingly undergo a massive shift. With Solid State disks access time is a nonissue, eliminating the idea of "small" files, and as solid state disks contain no moving parts their failure rate *should* (one would hope) be noticeably smaller.


Section 10: References / Further Reading


Coming Soon.

Index:
Section 1: Overview
Section 2: How it Works 1 - The Basics
Section 3: File Reading Models
Section 4: How it Works 2 - More Advanced
Section 5: Side Notes
Section 6: Advantages (summary)
Section 7: Disadvantages (summary)
Section 8: Rules of Thumb
Section 9: FAQ (various)
Section 10: Future of RAID-1
Section 11: References / Further Reading


Section 1: Overview


In the broad strokes, RAID-1 (or mirroring) is the simplest RAID form there is. It requires two disks which are basically exact copies of one another. The RAID controller simply duplicates the stream of data that was to be written to the array and sends one stream to each hard drive. This provides a number of advantages, such as full fault tolerance, potentially amazing data read speeds, and the ability to create the array without formatting... but the cost is that you lose half of your total potential storage capacity (one disk simply mirrors the other), it is not scalable in terms of the ability to add in more disks to increase capacity, and suffers a penalty in write speeds.

You may notice that throughout this thread many references are made to the complexity of RAID-1, yet I also just said that in the broad strokes it is the simplest RAID form there is. Both are true. The real complexity comes into play when you start to examine the various methods that exist to read data from a RAID-1 array that are not available in any other RAID model, which you will find explored in more detail in Section 3: File Reading Models (performance for various file sizes is in How it Works 2).


Section 2: How it Works 1 - The Basics
2a. Writing to the Mirror
2b. Reading from the Mirror


2a. Writing to the Mirror
As mentioned in the introduction, the basic writing mechanism is quite simple: the RAID controller simply duplicates the stream of data that was to be written to the array and sends one stream to each hard drive. Still, for a more graphical representation I'll draw a little diagram showing the result on the physical disks of the processor requesting data blocks A, B, C, D be written to the array (where the array is, as you'll recall from earlier sections, the name for the logical grouping of disks):

Disk 1: A, B, C, D
Disk 2: A, B, C, D

2b. Reading from the Mirror
Reading from the mirror is where RAID-1 becomes more complicated, and I will highly suggest reading the "File Reading Models" section for a more thorough explanation of the nuances and different methods that can occur. For now it suffices to say that there are two major models which can be used depending on the type of controller being used:

1. All data is read from only one disk
2. Data can be read from both disks

As you may have guessed, the second model has the potential to provide significantly better request processing.


Section 3: File Reading Models
3a. Single Disk
3b. Both Disks - Per Job Load Balancing
3c. Both Disks - Per "Stripe" Load Balancing
3d. Both Disks - Read Optimizations - Elevator Seek
3e. Both Disks - Read Optimizations - Shortest Seek First


The potential strength of RAID-1 comes entirely from the variety of read methods available. Some methods provide no benefit over a single disk solution, while others are able to handily beat the speed of a RAID-0 array

3a. Single Disk
The single disk read model is one you will find in many basic RAID chipsets and is solely responsible for the bad rap that RAID-1 receives from the enthusiast community. As the name implies, although both disks hold the same data, only one disk is used to read data from (and though some implementations change this disk over from time to time, the result is the same). This is a gross inefficiency and reduces read times to give approximately the same result as a single disk which is not in an array. As a small bit of consolation, it should be noted that even with cheap onboard RAID chipsets this particular RAID implementation uses less CPU time than any other.

3b. Both Disks - Per Job Load Balancing
Per Job load balancing requests read data from different disks based on various criteria (see below). For example, say you as a user were going to open two files simultaneously. Each file (or "job") is then read from a different hard drive. Already you can see how this is a significant improvement over the single disk model - instead of reading two jobs off one disk, one job is read off each disk and the total is completed in half the time. Another advantage to this type of Load Balancing is that it is a popular method employed by many software-based RAID card manufacturers, and may even be employed by many operating systems' software-based RAID arrays, yet offers no more a performance hit than 1-2&#37; on the CPU. This makes it a very cost-efficient solution. The various criteria that controllers can use to determine how to assign jobs include (but are not limited to): disk queue length (common), elevator seek (see 3d, often used in conjunction with disk queue length), shortest seek first (also used with disk queue length), and basic round robin.

The "disadvantage" of this type of load balancing is that it doesn't often do much for a desktop user because most people do not encounter issues of a large read request interrupting smaller read requests in their day-to-day browsing.

It is notable that this type of load balancing is that it inherently lends itself to multi-user environments where one users large request will not make a second user wait a long time for his small request to be filled. If, however, you find yourself in a situation where you have one program that does intensive reading and it interrupts your small side-requests (or vice versa), RAID-1 with Per-Job Load Balancing may be the way to go for you.

3c. Both Disks - Per Stripe Load Balancing
For lack of a better name I have decided to call this Per Stripe Load Balancing, though in reality a mirrored array really has no need for "stripes" in the same way that RAID-0 and RAID-5/6 do. In this scenario both drives can team up for each request that comes in, with one drive reading (for example) all the odd blocks of data and one drive reading all of the even blocks. This method effectively allows for RAID-0 style read speeds for large file read jobs while ensuring that if one disk fails you do not lose all of your data. Like Per Job Load Balancing, Per Stripe Load Balancing can also make use of various optimizations such as Elevator seek and Shortest Seek First which actually overcome the RAID-0 small file seek issue, and can vault RAID-1 into the throne of the fastest disk reading solution.

3d. Both Disks - Read Optimizations - Elevator Seek
The best explanation of this optimization comes from Wikipedia, and rather than plagarize, I'll simply quote them:

Quote:

---

Originally Posted by Wikipedia

From an implementation perspective, the drive maintains a buffer of pending read/write requests, along with the associated cylinder number of the request. Lower cylinder numbers indicate that the cylinder is closest to the spindle, and higher numbers indicate the cylinder is further away. When a new request arrives while the drive is idle, the initial arm/head movement will be in the direction of the cylinder where the data is stored, either in or out. As additional requests arrive, requests are serviced only in the current direction of arm movement until there are no further requests in that direction. When this happens, the direction of the arm reverses, and the requests that were remaining in the opposite direction are serviced, and so on.

---

This particular optimization is seen extensively in both software and hardware implementations. It does not really require any additional processing power and allows a RAID-1 two disk array to overcome the largest pitfall of a two-disk RAID-0 array - small file access time.

3e. Both Disks - Read Optimizations - Shortest Seek First
I'll start off by stating that I myself have never specifically seen this implemented in RAID-1, but it does get spoken of and is probably in place somewhere. This particular algorithm is designed to provide the shortest access times possible by buffering requests and tying their positions to cylinder data. Each hard drive then selects jobs by choosing those jobs who are closes to the current read head, providing exceptional service time. The downside is that if requests continue coming in this method could starve requests for areas of the disk that are further from the head. This limitation is often overcome by micro-optimizations concerning when the head should service other requests. In theory this method has the potential to provide the shortest seek times of any optimization, but thanks to its complexity and limited implementation very little data about its actual performance exists.


Section 4: How it Works 2 - More Advanced
4a. Reading a "small" File
4b. Reading a "large" File
4c. Reading a "large" NON-SEQUENTIAL File
4d. Writing to the Array
4e. Reading & Writing from/to the Array (ie. uncompressing a large RAR)


Please Note: This section *requires* an understanding of Section 3.

4a. Reading a "small" File
Without implementing any optimizations, any type of non-optimized read pattern will yield the same result - performance about equal to a regular single disk. With that being said however, any non-integrated solution (meaning either hardware or software controller seperated from the motherboard) should implement at least an Elevator Seek algorithm, at which time performance with small files excels. In fact, RAID-1 is the only RAID level which performs better than a single disk on average with random seeks (with optimizations, of course). In my own testing (which was by no means conclusive) I found that just implementing elevator seek in software alone resulted in a 10% reduction in average seek time versus a single disk (and let's not forget that other array types are actively worse than a single disk).

4b. Reading a "large" File
Obviously the read scheme will make a difference here. Single Disk read models will not help at all, and on a per-file basis neither will per-job load balancing, but per stripe load balancing will result in 2-disk RAID-0 speeds for large file reads.

4c. Reading a "large" NON-SEQUENTIAL File
Take everything from 4a and 4b and add them together here. A proper per striple load balancing scheme with an elevator seek will provide not only RAID-0 like speeds for the sequential reading bits, but can also reduce seek time. It should be noted however that to make effective use of this requires a somewhat more sophisticated piece of controller logic to allow one drives heads to go to one piece of fragmented data while having the other drives heads go to a different piece of fragmented data and keep track of it all. In reality I have not seen any data to conclusively show me that any hardware was able to show a difference one way or another versus RAID-0 for a test like this.

4d. Writing to the Array
Writing to the array is where RAID-1 takes a hit. Because one disk will always write slower than the other and write job will not complete until both are done, it will on average take longer than a single disk would.

4e. Reading & Writing from/to the Array (ie. uncompressing a large RAR)
Like any other method there is simply no benefit to using RAID-1 for this purpose. This is something best left for using multipe single drives for.


Section 5: Side Notes
5a. When Drives go Bad...
5b. Software vs. Hardware RAID-1


5a. When Drives go Bad...
Your data is fully protected. You can have one of your two drives die on you and the other contains a full working copy of all your data. Some controllers allow a drive to fail without interrupting your service, while others will require a reboot. To rebuild the array simply find another hard drive and re-create it, generally there should be no formatting required. Rebuilding the array will take a fairly significant amount of time however, equivalent to however long it takes the new drive to write all the data that is on the old drive onto its platters.

5b. Software vs. Hardware RAID-1
While there is no specific requirement to use hardware RAID-1, there is a persuasive arguement. Software RAID-1 requires very few computer resources and has the ability to implement any form of optimization that the programmers could come up with without any real additional processing requirements. However, the main split comes in when you look at the type of load balancing that different methods offer. To date all software load balancing I have seen uses per-job load balancing, as do many cards (the Areca 1210, for example)... but if "per-stripe" load balancing is what you are looking for, a hardware RAID controller is almost certainly what you will need.


Section 6: Advantages (summary)


1. Complete single disk failure redundancy
2. Low cost (hardware controller not necessary, though read section 5b)
3. Potential for faster read access than a 2-disk RAID-0 array overall (see 4a, 4b)


Section 7: Disadvantages (summary)


1. High cost per GB (you lose 1/2 your potential storage capacity)
2. Write speed is lower than a single disk on average
3. Does not protect against data corruption


Section 8: Rules of Thumb
7a. When to use RAID-1
7b. When NOT to use RAID-1


7a. When to use RAID-1
Use RAID-1 when:
1. You want the fastest read speed you can pull out of two identical drives
2. You have a game or application that would greatly benefit from a reduction in seek time (lots of small file reading going on)
3. When data preservation is important to you

7b. When NOT to use RAID-1
1. When you care that you're losing half your storage capacity
2. When you will be using the array for lots of writing


Section 9: FAQ


Q: Should I just read the RAID-0 FAQ first because it's boring to repeat the same questions and answers?
A: Yes.


Section 10: Future of RAID-1


As SSD drives come out and hardware that can provide XOR calculations becomes more mainstream, RAID-1 will slowly die out. The fact is that as seek times stop becoming an issue and platter mechanics disappear the speed benefit of RAID-1 will disappear as well, and redundancy will likely be covered more by RAID-5 and RAID-6 arrays, which offer more storage capacity overall.


Section 11: References / Further Reading


http://en.wikipedia.org/wiki/Elevator_algorithm

Index:
Section 1: Overview
Section 2: How it Works 1 - The Basics
Section 3: How it Works 2 - More Advanced
Section 4: Hardware and Performance
Section 5: Side Notes
Section 6: Advantages (summary)
Section 7: Disadvantages (summary)
Section 8: Rules of Thumb
Section 9: FAQ (various)
Section 10: Future of RAID-5/6
Section 11: References / Further Reading


Section 1: Overview


As an astute reader, you have no doubt determined that unlike the last two posts, this single section serves as a guide for two different RAID levels. Is this because I'm lazy and wanted to save myself some time by only going into half the quality each level deserves? No. It's because at the heart of i RAID-5 and RAID-6 are not too dissimilar to one another... in fact, RAID-6 is often seen as simply being an extension of RAID-5. As such, this post will primarily concern RAID-5 and make note of any areas where RAID-6 deviates.

Both RAID-5 and RAID-6 have a fairly idealized goal: to not only increase file read and write performance by utilizing multiple hard drives, but provide fault tolerance as well. While the latter goal is simply a necessary operation of these levels, the former may require significant investment depending on the level of performance expected. As such, both RAID-5 and RAID-6 are typically seen only in business environments which can tolerate the higher costs of ownership.


Section 2: How it Works 1 - The Basics
2a. Informational
2b. Parity
2c. Writing to the Array
2d. Reading from the Array


2a. Informational
To operate properly, RAID-5 requires a minimum of 3 hard drives in an array and RAID-6 requires 4. The reason behind this is that RAID-5 was designed to handle the failure of one hard drive, while RAID-6 was designed to handle the failure of up to two. For either to have less than the minimum number the situation would have to fall into one of two categories: either there is only a single disk and thus no RAID array is possible OR there are only two disks in the initial array and RAID-0 or -1 would be the better option (depending on goals).

2b. Parity
Unlike the other levels examined thus far, both RAID-5 and RAID-6 require the use of parity information. This parity information is effectively only used when a disk fails and allows the remaining disks to continue functioning and re-construct all of the information that was on the failed drive. Although we will explore how this parity information is generated in the first place in a later subsection, it suffices to say for now that it requires an amount of space equal to one full hard drive for RAID-5 and two for RAID-6. Thus,

4x 320GB drives in RAID-5 = 960GB of useable space (320GB lost)
5x 400GB drives in RAID-5 = 1.6TB of useable space (400GB lost)

4x 320GB drives in RAID-6 = 640GB of useable space (640GB lost)
5x 400GB drives in RAID-6 = 1.2TB of useable space (800GB lost)

2c. Writing to the Array
The chain of events involved go something like this (broadly - much more detail in the advanced section):
1. Data that needs to be written arrives at the RAID controller and is split into stripes
2. Parity information is calculated for each data stripe
3. For each data stripe, data is written on all but one of the drives, and on the drive it wasn't placed on the parity information is written
3*. The hard drive that the parity data is written on is changed on a round-robin basis per stripe written

Using 4 disks and a RAID-5 array, if the original data to be written were split into stripes A, B, C, D, E, then each stripe would be split into (n-1 = 3) blocks (A1-A3, B1-B3, etc). A parity block is then calculated for each stripe and is incorporated into a "full" stripe, which is written across the disks, with the parity block (denoted by the 'p') being distributed in round-robin fashion.
Disk 1: Ap , B1 , C1 , D1 , Ep
Disk 2: A1 , Bp , C2 , D2 , E1
Disk 3: A2 , B2 , Cp , D3 , E2
Disk 4: A3 , B3 , C3 , Dp , E3

2d. Reading from the Array
Thankfully, reading from an array is much simpler than writing to it. When reading from the array, only the data blocks (and not the parity blocks) are read, then re-combined by the controller to form a usable data stream.


Section 3: How it Works 2 - More Advanced
3-. Please Read
3a. What is an XOR?
3b. The Update Write Function
3c. The Regenerate Function
3d. The Rebuild Function


3-. Please Read
RAID-5/6 can be an extremely complicated beast depending upon the depth you want to go into. Like RAID-1, various manufacturers can offer different optimizations and to some degree different implementations (ie. on-disk versus on-controller partial XOR)... and the fact of the matter is that short of a 20 page document, I could not explain it fully. As such, while this section will attempt to detail the basic mechanism and I have provided a few links to more detailed pages (one so detailed it even outlines the protocol format for various operations) which I would suggest as a read for anyone who is really interested.

Also, it will suffice to say for the rest of this section that RAID-6 will be considered as a duplicate occurrence of RAID-5 operation using another parity block. It's not entirely true, but for the extent RAID-6 is seen in enthusiast PCs it will suffice.

Finally, I would like to acknowledge that the steps for subsections 2a/b/c are effectively paraphrases of the information found in the reference section (so shoot me, I found very well-written reference pages).

3a. What is an XOR?
An XOR is a mathematical operation also known as an Exclusive OR in many circles. For a full explanation of XOR and all of its applications, see http://en.wikipedia.org/wiki/Xor

The basic mathematical idea is that:
0 XOR 0 = 0
0 XOR 1 = 1
1 XOR 0 = 1
1 XOR 1 = 0

And it should be noted that when talking about multi-bit XORs, each bit is simply matched up with a bit below and XOR'd individually. Example:

10010110 XOR 01001110 = 11011000

The reason this is important to RAID-5 is that as it turns out if you XOR any number (n) of binary strings bit-wise (resulting in n original strings + 1 parity string) by XORing the first string with the second and the third string with the result of the first two etc, you can recreate the data lost by performing an XOR on the strings you have left and the final XOR result will be the string that was lost.

3b. The Update Write Function
The Update Write Function is used to write new data to a data stripe and update the parity information in parity stripes (note: on a per-stripe basis, each disk can be considered either a data drive or a parity drive). To perform a write operation to a RAID 5 disk array, it is necessary to perform what is referred to as a "Read-Modify-Writeback" operation. Several steps must be performed:

1. Get the new data to be written to the data disk
2. Read old block contents (data that will be replaced) into internal buffers from the data disk
3. Read old blocks corresponding parity information into internal buffer from the parity disk
4. XOR the old data block contents and the old parity information (removes the old data's contribution to parity) [wondering why this happens? read on]
5. Compute new parity information using by XORing (1) with the XOR result from (4)
6. Write the new parity to the parity disk
7. Write the new data to the data disk
8. Signify I/O completion via interrupt

An excellent diagram of how this works is found on page 5 of the Intel page mentioned in the references for this section.

There now remain only two things to explain relating to RAID-5 write operations:

Q. What happens when you take more than just a single data disk and parity disk into the equation (RAID-5 does require a minimum of 3 disks)?
A. Steps 1->8 occur for each disk in order. The parity block remains the same size because only XOR operations are performed upon it. This is one of the instances where optimizations can come in, with any decent controller being able to read all necessary data from all drives and computing XOR's in memory before finally telling the parity disk to read/write only once.

Q. Why step 4 takes place (why old data is read and used)?
A. Step 4 occurs because the data update may be small - say the size of only the stripe on one disk. If a slight change like that were to occur and step 4 did not take place, then the data from all disks would have to be read and parity calculated from all of them rather than simply from the single disk. Although it seems like it adds an unnecessary second XOR, in reality it prevents many more potential XOR operations.

3c. The Regenerate Function
The regenerate function is used to recreate a data block that cannot be read from a data device (ie. a disk has failed but data is still being read). In this scenario, the distinction between data and parity is nonexistent because neither means anything without the other. The sequence is as follows:

1. Data is read from drive 1
2. Data is read from drive 2
3. The data read from drive 1 and 2 are XOR'd
4. The XOR result is XOR'd with the data from drive 3 (if applicable)
5. Step 4 repeats as many times as is necessary until no drives remain
6. The final XOR result is the data that was on the failed drive

3d. The Rebuild Function
The rebuild function is similar to the regenerate function, except that the last XOR result is written to the formerly unreadable device (ie. a drive has failed and the controller is rebuilding the array on a fresh drive that has been installed in its place). Follow the sequence above and add a 7th step where the final XOR result is written to the new drive.


Section 4: Hardware and Performance
4a. Why do People Say RAID-5/6 is Resource Intensive?
4b. Bottleneck Sources
4c. Recommendations


4a. Why do People Say RAID-5/6 is Resource Intensive?
Because it is. While no single function that occurs in a RAID read or write may require too much in the way of resources, they do require a significant amount more than RAID-0 or non-RAID arrays and the cumulative effect over a larger array with an intensive operation is staggering, not to mention the effect these operations have on latency for small requests.

4b. Bottleneck Sources
Case 1: Software RAID
Your main bottleneck is going to be the fact that your CPU has to do all the work and no functions are streamlined or designed for... ultimately your CPU has to deal with nearly everything and because it wasn't designed to, it also has to deal with everything that helped it deal with the data in the first place. There are few situations where software RAID-5/6 could be a viable solutions, and fewer where it would be recommended, so we're not going to go there.

In hardware, many of today’s RAID controllers implement a hardware XOR engine - two in some cases - which can calculate XOR's at wire speed (meaning effectively instantaneous) and hence with such controllers the XOR itself is removed as the key bottleneck. What then emerges as the largest bottlenecks are:
1. Speed of the drives themselves as they are used to read/write data (though this is unavoidable)
2. Speed of the memory chips that are being used by the XOR engine for its data buffering on the controller. This one becomes the largest bottleneck because with the exception of distance of wire travel itself (also unavoidable and effectively a non-source of latency anyway), it is the only other source of latency. Every time data is to be XOR'd, it has to first sit in a buffer, and once done sit there again. This should become a smaller concern as time goes on, but it is still a concern. For example, for 250MBps of throughput, 4ns memory is required, and the keen observer will note that while many companies will list the clock speed of on-board processors, they fail to specify memory speed (not that I would particularly worry about this unless you really had the time and money to do so).

4c. Recommendations
My first recommendation is to find all your hookups with RAID card companies and have them send me units for testing. That aside, price is realistically going to be the largest determining factor in what you can (or cannot) get as good RAID controllers are not cheap, so I'll offer only two simple guidelines:

1. Make sure to find a hardware RAID controller, a software RAID controller won't do too much for RAID-5
2. Make sure to check the bandwidth of the bus the card is going to be on (PCI 2.2, PCI-E, PCI-X) and the potential bandwidth of each drive and ensure that if all fires perfectly you have room on the bus to spare so it doesn't bottleneck your system
3. If possible, purchase hard drives which can perform XOR operations in their buffers (this should be true of many SCSI drives today). This will remove a number of buffer and transfer latency-inducing operations per write operation.


Section 5: What if a Drive Fails?


RTFM. Your controller will come with some form of documentation on exactly what steps to take, and I suggest following them. Procedures are different from product line to product line, but in general if a drive in your RAID-5/6 array fails, your setup will fall into one of three major categories: one which can rebuild without a reboot, one which has hot-swap capabilities, and one which has a hot-spare built in. In any case, once the failure is detected and the failed drive removed and replaced with a working one (except in the case of the hot spare, where a new one is already in there) the Rebuild write function will begin. This is an extremely disk and resource intensive process, which may very well make the disk system unusably unresponsive depending upon your setup. I recommend you take this time to floss your teeth, as despite what Listerine may think, flossing is an extremely important part of oral health.


Section 6: Advantages (summary)

1. Complete protection against the loss of 1 disk (RAID-5) or two disks (RAID-6)
2. Read speeds are comparable by and large to RAID-0. Although with the same number of drives RAID-0 would have one additional disk worth of usable space to read from, lets not forget that RAID-5/6 do read from the same number of drives... the only difference is they effectively skip over a single stripe block (parity stripes) every so often. When you additionally take into account that RAID-5/6 require 3-4 drives minimum and that even theoretical read gains drop to single digits for RAID-0 by the 4th drive, distinctions are very difficult to see.


Section 7: Disadvantages (summary)

1. Cost of hardware controllers for a worthwhile implementation
2. Cost per GB increases in your disk subsystem because at least one if not two disks worth of space is lost to parity


Section 8: Rules of Thumb
8a. When to use RAID-5
8b. When to use RAID-6


8a. When to use RAID-5
Use RAID-5 if you meet the following criteria:
1. You have a few hundred dollars to spend on a hardware disk controller
2. You require large file read performance greater than what you would find with a single disk
3. You require data availability in the face of disk failure
4. You require more than one disk worth of space

8a. When to use RAID-6
Use RAID-6 if:
1. You meet all criteria for RAID-5
2. Your data is so important that the fact that another disk failure occuring while a failed disk is being replaced is not tolerable
3. You have more than just a few hundred dollars to spend on a hardware disk controller


Section 9: FAQ (various)


I have not really encountered any questions to post here. Please note that I will not answer questions about setting up or using RAID-5 on any chipset/hardware controller - you have to read the manuals people.


Section 10: Future of RAID-5/6


RAID-5/6 has a bright future. Like RAID-0 it will benefit from a shift in hard drives towards solid state disks reducing latency, and as memory chips become faster and cheaper controller cards will (we all hope) become cheaper, faster, and carry larger amounts of on-board cache as well. Drives have already emerged which can perform partial XOR functions in their own cache, eliminating a large bottleneck, and as they become more common hopefully their prices will decrease as well.


Section 11: References / Further Reading


Read these two in order:
http://en.wikipedia.org/wiki/Xor
http://download.intel.com/design/sto...s/30094601.pdf

Then read through some of these:
http://findarticles.com/p/articles/m..._61620918/pg_1
http://findarticles.com/p/articles/m..._60300939/pg_1
http://www.t11.org/t10/document.07/07-113r0.pdf
http://t10.t10.org/ftp/t10/document.94/94-111r6.pdf

Index:
Section 1: Overview
Section 2: Examples
- 2a. RAID-00
- 2b. RAID-01
- 2c. RAID-10
- 2d. RAID-11
Section 3: FAQ
Q: What is the effective difference between RAID-01 and RAID-10?



Section 1: Overview


Sometimes we want the best of both worlds. Sometimes "good enough" just isn't good enough. It's times like these when we look to combinations of RAID levels to offer us increased speeds and protection. Combination levels are pretty much what they sound like - multiple layers combined to behave in a way that neither could on its own. These
combinations do not necessarily have to use different levels, or even only two RAID levels... indeed, you could have a RAID-1 array of RAID-1 arrays, or a RAID-5 array of RAID-1 arrays which are themselves comprised of RAID-0 arrays. Your choices and the nesting you opt for are - practically speaking - limitless (as the costs will generally become oppressive long before you are constained by the technologies available).

How it Works
The first trick to master when dealing with RAID combinations is to wrap your head around the basic concept of creating an array using other arrays as constituents instead of hard drives. The simplest way to think of a RAID combination level is to break it down and just work it through piece-by-piece. Popular nomenclature (which I oppose but will still use here) states that the level furthest to the LEFT of the level will be the one which is actually comprised of hard drives, and as you move further right you begin creating arrays which are made of the arrays which make use of the physical drives.

For example, the level RAID-0+1 (aka RAID-01) could be expanded to:

A RAID-1 array which mirrors a RAID-0 array on to another RAID-0 array.

To go to a slightly more complex model, the level RAID-0+1+1 (aka RAID-011) could also be stated to be:

A RAID-1 array which mirrors a RAID-1 array to another RAID-1 array, both of which are themselves comprised of two seperate RAID-0 arrays

Pictoral examples:
http://upload.wikimedia.org/wikipedi...RAID_0%2B1.png [RAID-01]
http://upload.wikimedia.org/wikipedi...px-RAID_10.png [RAID-10]
http://upload.wikimedia.org/wikipedi..._0%2B3.svg.png [RAID-03]

It should be noted that you can technically arrange it such that each constituent array is made of different types of hard drives... however it is frowned upon because the best performance will come from matched drive, as so that will not be considered.


Section 2: Examples


Strictly speaking I believe once a person has seen a few pictures of how RAID combinations work and have been given a basic explanation they should be able to work out the details themselves as far as how many drives are required and what sort of benefits will be seen... but as we live in a world where expecting others to do their own homework as often as not results in misinformation I have created this section. Below you should find examples of a number of RAID combinations, as well as a brief synopsys of what benefits they offer and why, how many drives are required, and any special notes I may think of. As well, somewhere below is hidden the made up word "fishfacefignewton", which I expect anyone who seriously wants to read this full thread should be able to find on their own without using CTRL-F. If you look for but cannot find it, I would submit a proper exercise would be to read this entire guide from start to finish (not just this post) until you do because clearly you have not paid enough attention.

Note: It will be assume here that RAID-1 will not load balance a single job, as that is not likely to be the case

2a. RAID-00
Min. Drives Required: 4 (RAID-0 requires 2, and this is two instances of it combined)
Protection offered: None

One of the more confusing RAID combination levels, RAID-00 really doesn't offer anything versus a regular RAID-0 array with the same number of drives except more overhead (not a lot, but it's there). Basically it takes your data, breaks it into a number of stripes equal to the number of constituent arrays, and each constituent array then breaks off its stripe into sub-stripes and writes them to its array. The combined effect is that your data is still broken into the same number of stripes and laid down in what is effectively the same manner as before, only you will have incurred buffer latency and a computational hit. Worse, your data still has no protection so you can't even say you got that at least.

Recommendation:
RAID-00 is not recommended for any implementation, unless you want to humiliate yourself.

2b. RAID-01
Min. Drives Required: 4 (RAID-1 requires two arrays, each made of a RAID-0 array which itself requires at least 2 drives)
Protection Offered: One full RAID-0 array may fail

This is not a somewhat popular choice of nested RAID levels in some business roles, and deservedly so. It leverages the load-balancing and redundancy of RAID-1 and combines it with the raw throughput power of RAID-0, though it should be noted that the primary advantages will only be seen when the load balancing is in effect. Theoretically the performance can be stated as:

The performance of a RAID-01 array will closely approximate that of either component RAID-0 array on its own for single read requests, but will provide superior overall handling of multiple simultaneous read requests. Write operations will be constrained to the speed of the slowest constituent hard drive.

The breakdown is as follows:
- For a single read request, the request will only be made to one of the RAID-0 arrays (unless your controller allows for single-job RAID-1 load
balancing, which it likely does not) and thus can provide no speed advantage over a single RAID-0 array.
- For multiple separate, simultaneous read requests it is likely the RAID-1 logic will allow load balancing. Thus for request A one of the RAID-0 arrays will be used, while request B will be filled by the other.
- For a write requests, the array will be constrained by the speed of the slowest of all hard drives used, because a write operation is not complete until it is complete in all drives.

Recommendation:
RAID-01 is recommended for environments where high volumes of read requests are encountered. Web and (read only) file servers are two prime candidates for this combination because they are likely to encounter multiple simultaneous requests which need prompt service. It should be noted however that RAID-10 is the preferred version of this type of nesting, as the volume of data which must be regenerated after a disk failure is lower.

Primary Disadvantages:
1. RAID-01 arrays rebuild slowly compared to RAID-10 arrays by virtue of the fact that they must rebuild an entire RAID-0 set of disks (versus a single disk) and;
2. RAID-01 arrays cannot make use of any RAID-1 enhancement (ie. elevator seek) beyond basic load balancing.
For both of these reasons, RAID-10 is the suggested configuration for most environments.

2c. RAID-10
Min. Drives Required: 4 (RAID-0 requires two arrays, each made of a RAID-1 array which itself requires 2 drives)
Protection Offered: Any 1 drive in any constituent RAID-1 array may fail

RAID-10 is a popular choice among many enthusiasts, though that doesn't necessarily mean it's actually the "best". It does offer fair data protection against potentially two or more disk failures (depending on the number of RAID-1 arrays in use), and also an increase in speed due to making use of RAID-0s striping, but in reality the behavior is very similar to that of RAID-01 (with two differences I will mention later). Theoretically the performance can be stated as:

The performance of a RAID-10 array will closely approximate that of a RAID-0 array with half the total number of drives for single read requests - except with possibly better access time - and will provide superior overall handling of multiple simultaneous read requests. Write operations will be constrained by the speed of the slowest constituent hard drive.

The breakdown is as follows:
- For a single request, the request will be divided and sent to each constituent RAID-1 array. The RAID-1 controller may then select the drive closest to the information access point to retreive the data, thus delivering a faster access speed than an ordinary RAID-0 array.
- For multiple read requests, the RAID-0 logic *MAY* (vendor discretion) pass along the full requests simultaneously to each RAID-1 controller, which may then implement per-job load balancing. This in effect puts all drives to use and can offer much higher total throughput versus a single request. It should be noted that if the software or controller does not allow requests to be passed until the first request is fulfilled at the RAID-0 or earlier level, this advantage in multiple request load balancing will be lost.
- For a write requests, the array will be constrained by the speed of the slowest of all hard drives used, because a write operation is not complete until it is complete in all drives.

Recommendation:
Although the costs can quickly accumulate for RAID-10, I would have no problem recommending it for gaming and workstation use, and assuming the controller could gracefully handle multiple requests (not one at a time), then for web/file server use as well. It is preferable to use RAID-10 to RAID-01 in many instances where uptime is critical because less data regeneration must occur.

Please Note: If your controller does not support RAID-1 optimizations such as elevator seek, this level will offer little compared to its potential.

2d. RAID-11
Number of drives required: 4
Protection offered: Up to 3 drives may fail (75% loss tolerated!!)

A classic. RAID-11 is, as the name implies, a mirrored set of arrays where each component array is itself a set of mirrored disks. The redundacy advantages are obvious, the speed advantages are fairly straightforward, and the loss of potential disk space is outrageous. Theoretically the performance can be stated as:

The performance of a RAID-11 array for a single read request will equal that of a single disk, though with a lower average seek time. Multiple mirrors enable higher levels of multiple simultaneous requests with little degradation in speed as opposed to other nested RAID levels. Write operations will be constrained by the speed of the slowest constituent hard drive.

The breakdown is as follows:
- Single read requests will only be serviced by a single disk, but RAID controller optimizations may allow that controller to use its disk which is in the best position to fullfil the order. Thus the average seek time should be less than that of a single disk. fishfacefignewton.
- Multiple read requests will be load balanced between the two sets of RAID-1 disk arrays, providing the same benefits as regular RAID-1, but after that the disk arrays themselves will also load balance, providing non-degraded service for up to 4 requests, and various queing optimizations in RAID-1 should lean towards improved scalability of service speed with number of requests versus other nested array types (assuming 4 disks only, of course).
- For a write requests, the array will be constrained by the speed of the slowest of all hard drives used, because a write operation is not complete until it is complete in all drives.

Recommendation:
On a per-GB level, this is extremely hard to justify for home use. Businesses with high continuity needs for some files may implement this in some places, but realistically there's a point where you just have to learn to do backups regularly.



Section 3: FAQ


Q: What is the effective difference between RAID-01 and RAID-10?
A: RAID-01 and RAID-10 do share some distinct differences. For enterprise setups, the largest factor to keep in mind is the array rebuild time in the event of a disk failure. Should a disk fail, a RAID-10 array will only have to rebuild a single disk, from its partner single disk. For RAID-01 however it must rebuild an entire RAID-0 array... which necessarily means 2 or more disks. For home users, the primary factor which makes RAID-10 a better option is the fact that if your RAID card implements proper RAID-1 optimizations, access times should decrease versus a RAID-01 array by virtue of the fact. To explain this a little better, in a RAID-10 array, it is RAID-1 arrays which are imposed on the physical drives themselves and hence can track which drives will have the shortest access times to what data. With a RAID-01 array however, it is the RAID-0 arrays which are imposed on the physical drives, and the RAID-1 array cannot implement any optimizations beyond simple load-balancing.

More coming, I just feel my time would be better spent reviewing this collection of posts right now for accuracy, grammar, omissions, etc rather than adding on paragraphs about combinations that no-one here will make any use of in their personal life

Good information :)

:eek:
Great content. Keep it comming

Very Nice, needs to be sticky.

Great guide, was really informative. Needs to be stickied.

Thanks for the replies guys! I do promise the RAID-5/6 and combination sections are coming soon... it's definitely taken longer than I expected to put them together. And I know there's still a lot of cleanup to do, but hopefully I'll be able to get on that sooner than later.

Don't forget the paranoid classic of raid 6+1

Great posts, always fun to read about RAID :)

Very nice, helped me understand RAID 0 & 1 a lot better. Thanks a lot

stuck

Although later than advertised, the hotly anticipated RAID-5/6 section has been posted.

Edit: Still to be included is a brief description of what will occur when you purchase a SCSI drive which can perform XOR operations in its cache and a controller which can take advantage of that (in all fairness I don't expect many people to rush out to pay the kind of money this requires and hence it is of lower priority to me).

Need an update:up:

Raid 0 was done today but I can't see my temps on that hd :(

That's normal. Most controllers do not pass the HD temperature (nvRaid, IIRC does, as do 3ware controllers). Sux, but you must use the admin tool for other controllers to see the temps.

Decent overview, however a couple items should be mentioned:

- RAID is for hardware failures, no raid level will protect you against a software/logical failure (ie, deleting a file, or formatting the array, et al) Always keep backups regardless to what you're doing on a raid

- In my searches no raid (hardware/software) does parity checking on READ (please correct if you know of something that does). This basically means that there is a chance (and that chance increases with the size of the array) of the disk sending out bad data that is not caught which is then sent to the host. In situations which need to read data/update/and write back this will corrupt data with NO ERROR on a raid (as it assumes it's always good). Ie, request blocks 0-100, block 98 has a transient data glitch from the drive (bits flipped, random error that gets past the ~10^15 ecc check or happens at the controller interface). Since the drive does not report it as an error as it was not noticed, and the controller does not check the data, this gets to the host. the host may update data from block 1 say and write the whole stripe back. The raid card will then re-calculate the parity on that 'new' stripe (with the original error) and now you have a parity checked error stripe on your array which will be there for all future accesses.

- RAID6 provides over raid-5 in error correction. With RAID-5 you have only TWO sources of correct data (what is read from the stripe and what is it's calculated parity data to re-construct). If they do not match, which is the 'wrong' one. Flip a coin basically and you'll be right 50&#37; of the time. With RAID6 you have THREE sources of data so if it's wrong you have a tie-breaker.

Quote:

---

Originally Posted by stevecs

Decent overview, however a couple items should be mentioned:

- RAID is for hardware failures, no raid level will protect you against a software/logical failure (ie, deleting a file, or formatting the array, et al) Always keep backups regardless to what you're doing on a raid

- In my searches no raid (hardware/software) does parity checking on READ (please correct if you know of something that does). This basically means that there is a chance (and that chance increases with the size of the array) of the disk sending out bad data that is not caught which is then sent to the host. In situations which need to read data/update/and write back this will corrupt data with NO ERROR on a raid (as it assumes it's always good). Ie, request blocks 0-100, block 98 has a transient data glitch from the drive (bits flipped, random error that gets past the ~10^15 ecc check or happens at the controller interface). Since the drive does not report it as an error as it was not noticed, and the controller does not check the data, this gets to the host. the host may update data from block 1 say and write the whole stripe back. The raid card will then re-calculate the parity on that 'new' stripe (with the original error) and now you have a parity checked error stripe on your array which will be there for all future accesses.

- RAID6 provides over raid-5 in error correction. With RAID-5 you have only TWO sources of correct data (what is read from the stripe and what is it's calculated parity data to re-construct). If they do not match, which is the 'wrong' one. Flip a coin basically and you'll be right 50% of the time. With RAID6 you have THREE sources of data so if it's wrong you have a tie-breaker.

---

I'm a little out of it tonight (switching from day to night shifts ATM), but once I'm back on track I'll do my best to work that in as it's all true and does highlight the fact that even with the craziest RAID setup in the world, there's always a reason to do backups.

Actually, even backup is no guarantee that the data backed up is valid to begin with ;)

Pretty nice guide you have there.

I just set up RAID 0 last night :D

Installing windows and my games to it one of these days. If its still slow (highly doubt it), i'll juse use a single drive...second drive as an external.

Nice post. Can you describe the difference the between raid 0+1, raid10.
Thanks

Some small updates made about backups, a few sentence cleanups.

Quote:

---

Originally Posted by elec999

Nice post. Can you describe the difference the between raid 0+1, raid10.
Thanks

---

Ask and ye shall receive. Check out the new Section 3 of the RAID-xx post, and the added notes to the bottom of the RAID-01 and RAID-11 subsections.