Some RAID Benches (5, 6, 50, and 60): Areca 1680, VRaptors, 256 bit AES, etc.

[Speederlander]

Playing around with different configurations, figure I would make the data available to all.

System basics:
Vista Ult 64-bit
Supermicro X7DWA-N
2x Harpertown 5420's BSEL'd to 3GHz
16GB 800MHz Kingston FB-DIMM
Areca 1680ix-16 with battery back-up
8x 300GB Velociraptor HDDs

Areca specs (latest firmware):
Controller Name ARC-1680
Firmware Version V1.45 2008-09-16
BOOT ROM Version V1.45 2008-09-15
SAS Firmware Version 4.4.3.0
Main Processor 1.2GHz IOP348 C1
CPU ICache Size 32KBytes
CPU DCache Size 32KBytes/Write Back
CPU SCache Size 512KBytes/Write Back
System Memory 512MB/533MHz/ECC

[Speederlander]

8 Drive RAID 0, 128K Stripe, 120GB Volume, Write Back, Disk Write Cache On, OS Caching On, NCQ On

[Speederlander]

8 Drive RAID 0, 128K Stripe, 120GB Volume, Write Back, Disk Write Cache Off, OS Caching Off, NCQ On

[Speederlander]

8 Drive RAID 0, 128K Stripe, 120GB Volume, Write Back, Disk Write Cache Off, OS Caching Off, NCQ Off

NOTE: No further tests with NCQ off will be performed.

[Speederlander]

8 Drive RAID 10, 128K Stripe, 120GB Volume, Write Back, Disk Write Cache On, OS Caching On

[Speederlander]

****reserved

[Speederlander]

*****reserved

[Speederlander]

******reserved

[Speederlander]

*******reserved

[Speederlander]

********reserved

[Speederlander]

*********reserved

[Speederlander]

**********reserved

[Speederlander]

***********reserved

[Speederlander]

************reserved

[stevecs]

You want to turn off write caching in windows so that your OS/system cache doesn't colour the results too much (it always will unless doing direct I/O but it will cut down the majority of it's effects). Also set disk write cache mode to explicitly OFF on the areca (so drives don't colour results, this means that writes will only be cached on the card's ram IF you have a BBU (no bbu/no caching). That way at least you don't have any ambiguity.

Also if you could do something like simple crystaldiskmark I/O checks as it has random read/write just make sure you select the 1GB test size and 9x average (would be better if you did it before the 2GB cache upgrade). it would be better to run iozone/xdd but that's going to be much more intensive and you probably don't want to waste the time on it.

Aggressive read-ahead will affect random I/O (more aggressive the lower your random I/O results will be but better for streaming).

since you're on windows and looks like running against a partitioned space, are you aligning the partitions to your stripe size? (32K start for XP/earlier or 1MB for vista/2008).

[Speederlander]

You want to turn off write caching in windows so that your OS/system cache doesn't colour the results too much (it always will unless doing direct I/O but it will cut down the majority of it's effects). Also set disk write cache mode to explicitly OFF on the areca (so drives don't colour results, this means that writes will only be cached on the card's ram IF you have a BBU (no bbu/no caching). That way at least you don't have any ambiguity.

I do have bbu.
I will turn off windows write caching (it is off already I believe as this is fresh OS install).
I will turn off disk caching on the areca.

Also if you could do something like simple crystaldiskmark I/O checks as it has random read/write just make sure you select the 1GB test size and 9x average (would be better if you did it before the 2GB cache upgrade). it would be better to run iozone/xdd but that's going to be much more intensive and you probably don't want to waste the time on it.

Downloading crystaldiskmark now. I will include it.
I'll try iozone. Please provide settings as I have no previous experience with it.

Aggressive read-ahead will affect random I/O (more aggressive the lower your random I/O results will be but better for streaming).

I'll include results for both if possible.

What would you like to see these set to (for better random IO, etc characterization):

HDD Read Ahead Cache
Enabled

Volume Data Read Ahead
Aggressive

Disk Write Cache Mode
Auto

since you're on windows and looks like running against a partitioned space, are you aligning the partitions to your stripe size? (32K start for XP/earlier or 1MB for vista/2008).

Drives are unpartitioned. HDTach write tests wouldn't function on anything with partitions. I haven't traditionally messed too much with aligning partitions. Feel free to provide a bit more info and I will add those results as well if time permits.

[stevecs]

Actually, if you're going to d/l text tools, xdd would be better to show the different performance profiles of the raids faster. http://www.ioperformance.com/products.htm

for your specs you should use the following (note, create a file ~64GB in size or larger I have it called S0 below). Replace XXX for queue depth with desired depth (I usually test all by factors of 2 (1,2,4,8,16,32,64,128) request size is 128 sectors (64K) (as I was mainly testing a network file server here and that was the normal size I saw with my workload which is the max w/ smb v1).

XDD -verbose -op read -target S0 -blocksize 512 -reqsize 128 -mbytes 32768 -passes 3 -seek random -seek range 64000000 -queuedepth XXX
XDD -verbose -op read -target S0 -blocksize 512 -reqsize 128 -mbytes 32768 -passes 3 -seek random -seek range 64000000 -queuedepth XXX

XDD mainly provides IOPS readings on the subsystem. crystal mark 'sort of' does the same but much more limited, however it will provide at least an indication of subsystem performance.

---
Good that the drives are not partitioned (then you don't have to worry about alignment) that's basically how I run all my file systems here at home under unix (no partition tables, just raw file systems) very helpful when you change the underlaying subsystem often. Anyway if you don't have partitions then you have to change the target w/ XDD above to not go to a file on a device but the device itself just like you do w/ hdtach.

[Speederlander]

Actually, if you're going to d/l text tools, xdd would be better to show the different performance profiles of the raids faster. http://www.ioperformance.com/products.htm

for your specs you should use the following (note, create a file ~64GB in size or larger I have it called S0 below). Replace XXX for queue depth with desired depth (I usually test all by factors of 2 (1,2,4,8,16,32,64,128) request size is 128 sectors (64K) (as I was mainly testing a network file server here and that was the normal size I saw with my workload which is the max w/ smb v1).

XDD -verbose -op read -target S0 -blocksize 512 -reqsize 128 -mbytes 32768 -passes 3 -seek random -seek range 64000000 -queuedepth XXX
XDD -verbose -op read -target S0 -blocksize 512 -reqsize 128 -mbytes 32768 -passes 3 -seek random -seek range 64000000 -queuedepth XXX

XDD mainly provides IOPS readings on the subsystem. crystal mark 'sort of' does the same but much more limited, however it will provide at least an indication of subsystem performance.

---
Good that the drives are not partitioned (then you don't have to worry about alignment) that's basically how I run all my file systems here at home under unix (no partition tables, just raw file systems) very helpful when you change the underlaying subsystem often. Anyway if you don't have partitions then you have to change the target w/ XDD above to not go to a file on a device but the device itself just like you do w/ hdtach.

Alright. Won't have xdd results until after I get back from vacation next week though. The manual looks long enough that I want to read through it in some detail before using. Is that source code included with the zip just reference? I don't have visual studio installed and likely have no intention of compiling anything for these tests due to the time it is already going to take me to complete what I have committed to.

[stevecs]

The main item is to turn off write cache on the drives and OS as those do not have a battery backup unit to retain data in case of a power loss which is generally how a redundant array should be installed (if done properly). The read ahead caches can be enabled as they do not have any affect on the subsystem reliability (the write does as without it you will screw up your subsystem in the case of a power failure). The only issue with read caches I was just commenting on was that with them enabled you are not seeing physically what the drives can do natively, if that's important to your results leave them on. What 'aggressive' does on the areca is read in more sectors after the actual request in anticipation. What this means is that say you have a request to read 32K and have other requests in the queue. What will happen is that the 32K request will be fulfilled and then ADDITIONAL data will be read to anticipate another request which may or may not come in, but it will HOLD OFF on executing the other requests until it's done (ie, helps with streaming type requests and small queue sizes but actually hurts performance the more random/more requests you have). Whether you want to test that or not is up to your needs I was just trying to make it clear what that really did.

[stevecs]

Yes, you get a pre-compiled binary that you can run from the command line. for example here's a run I just did against my 2nd raw partition here (raid-10/8 savvios in sig/ 8K stripe size on areca) this is only a read as I have data on that array that I couldn't move around.

(BTW/ to do it against a physical disk you need to do this in windows:

D:\Temp\xdd\bin>xdd.exe -verbose -op read -targets 1 \\.\\PhysicalDrive2 -dio -blocksize 512 -reqsize 128 -mbytes 8192 -passes 3 -se
ek random -seek range 64000000 -queuedepth 1

The physical disk number I just got from everest (subtract 1 from list as it really starts at 0)though I just noticed that when using physical discs in windows (XP X64 at least) you can't use a queuedepth > 1 so below I used files (2GB each) in the ntfs filesystem (NON-alligned as I'm using 8K stripe size w/ 8 drives in raid-10 (32K/(8drives/2)) filesystem starts at sector 63.) Will be fixing this shortly once I get some sleep.

D:\Temp\xdd\bin>xdd.exe -verbose -op read -targets 1 S32GiB -dio -blocksize 512 -reqsize 128 -mbytes 8192 -passes 3 -seek rand
ek range 64000000 -queuedepth 128

IOIOIOIOIOIOIOIOIOIOI XDD version 6.5.013007.0001 IOIOIOIOIOIOIOIOIOIOIOI
xdd - I/O Performance Inc. Copyright 1992-2007
Starting time for this run, Sat Dec 20 21:43:04 2008

ID for this run, 'No ID Specified'
Maximum Process Priority, disabled
Passes, 3
Pass Delay in seconds, 0
Maximum Error Threshold, 0
Target Offset, 0
I/O Synchronization, 0
Total run-time limit in seconds, 0
Output file name, stdout
CSV output file name,
Error output file name, stderr
Pass seek randomization, disabled
File write synchronization, disabled
Pass synchronization barriers, enabled
Number of Targets, 1
Number of I/O Threads, 128

Computer Name, ANALYTICAL, User Name, stcost
Operating System Info: NT 5.2 Build 3790 Service Pack 2
Page size in bytes, 4096
Number of processors on this system, 4
Megabytes of physical memory, 4095
Seconds before starting, 0
        Target[0] Q[0], S32GiB
                Target directory, "./"
                Process ID, 4512
                Thread ID, 4512
                Processor, all/any
                Read/write ratio, 100.00,  0.00
                Throttle in MB/sec,   0.00
                Per-pass time limit in seconds, 0
                Blocksize in bytes, 512
                Request size, 128, blocks, 65536, bytes
                Number of Requests, 1024
                Start offset, 0
                Number of MegaBytes, 8192
                Pass Offset in blocks, 0
                I/O memory buffer is a normal memory buffer
                I/O memory buffer alignment in bytes, 4096
                Data pattern in buffer, '0x00'
                Data buffer verification is disabled.
                Direct I/O, enabled
                Seek pattern, queued_interleaved
                Seek range, 64000000
                Preallocation, 0
                Queue Depth, 128
                Timestamping, disabled
                Delete file, disabled

                     T  Q       Bytes      Ops    Time      Rate      IOPS   Latency     %CPU  OP_Type    ReqSize
TARGET   PASS0001    0 128    8589934592 131072 109.014    78.797    1202.35    0.0008     3.86   read       65536
TARGET   PASS0002    0 128    8589934592 131072 109.936    78.136    1192.26    0.0008     3.58   read       65536
TARGET   PASS0003    0 128    8589934592 131072 109.674    78.322    1195.10    0.0008     3.84   read       65536
TARGET   Average     0 128   25769803776 393216 327.624    78.657    1200.21    0.0008     3.76   read       65536
         Combined    1 128   25769803776 393216 328.000    78.566    1198.83    0.0008     3.69   read       65536
Ending time for this run, Sat Dec 20 21:48:34 2008

[alfaunits]

Thanks for clarifying the Aggressive setting, I was wondering what that option really does. (what is conservative BTW?)
Though, I find it doesn't matter which ReadAhead I choose, the streaming reads always have the same speed.

[XS Janus]

That's a lot of testing!
Be sure to compile all you final data into a easily understandable table so we can compare all the results more easily.

Thanks for all the effort!

[m^2]

It seems we'll have a really good thread. Thank you for this.

[stevecs]

conservative/aggressive/normal are just labels for varying degrees to the same thing, basically aggressive reads more sectors than 'normal'; conservative reads less. I don't have the exact numbers from areca on it (never really bothered to ask for them). disabled turns it off all together which is that the drive will only read what the request was for which is best for worst-case (100% random) workloads.

[Speederlander]

Out for vacation. I'll post more results when I get back. Thanks.