Some RAID Benches (5, 6, 50, and 60): Areca 1680, VRaptors, 256 bit AES, etc.

Printable View

Show 100 post(s) from this thread on one page

12-20-2008, 02:22 PM
Speederlander

2 Attachment(s)

Some RAID Benches: 0, 10, 60 with Areca 1680 and 8 Velociraptors

Playing around with different configurations, figure I would make the data available to all.

System basics:
Vista Ult 64-bit
Supermicro X7DWA-N
2x Harpertown 5420's BSEL'd to 3GHz
16GB 800MHz Kingston FB-DIMM
Areca 1680ix-16 with battery back-up
8x 300GB Velociraptor HDDs

Areca specs (latest firmware):
Controller Name ARC-1680
Firmware Version V1.45 2008-09-16
BOOT ROM Version V1.45 2008-09-15
SAS Firmware Version 4.4.3.0
Main Processor 1.2GHz IOP348 C1
CPU ICache Size 32KBytes
CPU DCache Size 32KBytes/Write Back
CPU SCache Size 512KBytes/Write Back
System Memory 512MB/533MHz/ECC
12-20-2008, 02:24 PM
Speederlander

3 Attachment(s)

8 Drive RAID 0, 128K Stripe, 120GB Volume, Write Back, Disk Write Cache On, OS Caching On, NCQ On
12-20-2008, 02:25 PM
Speederlander

3 Attachment(s)

8 Drive RAID 0, 128K Stripe, 120GB Volume, Write Back, Disk Write Cache Off, OS Caching Off, NCQ On
12-20-2008, 02:28 PM
Speederlander

3 Attachment(s)

8 Drive RAID 0, 128K Stripe, 120GB Volume, Write Back, Disk Write Cache Off, OS Caching Off, NCQ Off

NOTE: No further tests with NCQ off will be performed.
12-20-2008, 02:35 PM
Speederlander

3 Attachment(s)

8 Drive RAID 10, 128K Stripe, 120GB Volume, Write Back, Disk Write Cache On, OS Caching On
12-20-2008, 02:38 PM
Speederlander

****reserved
12-20-2008, 02:40 PM
Speederlander

*****reserved
12-20-2008, 02:44 PM
Speederlander

******reserved
12-20-2008, 02:46 PM
Speederlander

*******reserved
12-20-2008, 02:47 PM
Speederlander

********reserved
12-20-2008, 02:50 PM
Speederlander

*********reserved
12-20-2008, 02:53 PM
Speederlander

**********reserved
12-20-2008, 02:55 PM
Speederlander

***********reserved
12-20-2008, 02:58 PM
Speederlander

************reserved
12-20-2008, 03:57 PM
stevecs

You want to turn off write caching in windows so that your OS/system cache doesn't colour the results too much (it always will unless doing direct I/O but it will cut down the majority of it's effects). Also set disk write cache mode to explicitly OFF on the areca (so drives don't colour results, this means that writes will only be cached on the card's ram IF you have a BBU (no bbu/no caching). That way at least you don't have any ambiguity.

Also if you could do something like simple crystaldiskmark I/O checks as it has random read/write just make sure you select the 1GB test size and 9x average (would be better if you did it before the 2GB cache upgrade). it would be better to run iozone/xdd but that's going to be much more intensive and you probably don't want to waste the time on it.

Aggressive read-ahead will affect random I/O (more aggressive the lower your random I/O results will be but better for streaming).

since you're on windows and looks like running against a partitioned space, are you aligning the partitions to your stripe size? (32K start for XP/earlier or 1MB for vista/2008).
12-20-2008, 04:36 PM
Speederlander

Quote:

Originally Posted by stevecs

You want to turn off write caching in windows so that your OS/system cache doesn't colour the results too much (it always will unless doing direct I/O but it will cut down the majority of it's effects). Also set disk write cache mode to explicitly OFF on the areca (so drives don't colour results, this means that writes will only be cached on the card's ram IF you have a BBU (no bbu/no caching). That way at least you don't have any ambiguity.

I do have bbu.
I will turn off windows write caching (it is off already I believe as this is fresh OS install).
I will turn off disk caching on the areca.

Quote:

Also if you could do something like simple crystaldiskmark I/O checks as it has random read/write just make sure you select the 1GB test size and 9x average (would be better if you did it before the 2GB cache upgrade). it would be better to run iozone/xdd but that's going to be much more intensive and you probably don't want to waste the time on it.

Downloading crystaldiskmark now. I will include it.
I'll try iozone. Please provide settings as I have no previous experience with it.

Quote:

Aggressive read-ahead will affect random I/O (more aggressive the lower your random I/O results will be but better for streaming).

I'll include results for both if possible.

What would you like to see these set to (for better random IO, etc characterization):

Quote:

HDD Read Ahead Cache
Enabled

Volume Data Read Ahead
Aggressive

Disk Write Cache Mode
Auto

Quote:

since you're on windows and looks like running against a partitioned space, are you aligning the partitions to your stripe size? (32K start for XP/earlier or 1MB for vista/2008).

Drives are unpartitioned. HDTach write tests wouldn't function on anything with partitions. I haven't traditionally messed too much with aligning partitions. Feel free to provide a bit more info and I will add those results as well if time permits.
12-20-2008, 04:56 PM
stevecs

Actually, if you're going to d/l text tools, xdd would be better to show the different performance profiles of the raids faster. http://www.ioperformance.com/products.htm

for your specs you should use the following (note, create a file ~64GB in size or larger I have it called S0 below). Replace XXX for queue depth with desired depth (I usually test all by factors of 2 (1,2,4,8,16,32,64,128) request size is 128 sectors (64K) (as I was mainly testing a network file server here and that was the normal size I saw with my workload which is the max w/ smb v1).

XDD -verbose -op read -target S0 -blocksize 512 -reqsize 128 -mbytes 32768 -passes 3 -seek random -seek range 64000000 -queuedepth XXX
XDD -verbose -op read -target S0 -blocksize 512 -reqsize 128 -mbytes 32768 -passes 3 -seek random -seek range 64000000 -queuedepth XXX

XDD mainly provides IOPS readings on the subsystem. crystal mark 'sort of' does the same but much more limited, however it will provide at least an indication of subsystem performance.

---
Good that the drives are not partitioned (then you don't have to worry about alignment) that's basically how I run all my file systems here at home under unix (no partition tables, just raw file systems) very helpful when you change the underlaying subsystem often. Anyway if you don't have partitions then you have to change the target w/ XDD above to not go to a file on a device but the device itself just like you do w/ hdtach.
12-20-2008, 05:02 PM
Speederlander

Quote:

Originally Posted by stevecs

Actually, if you're going to d/l text tools, xdd would be better to show the different performance profiles of the raids faster. http://www.ioperformance.com/products.htm

for your specs you should use the following (note, create a file ~64GB in size or larger I have it called S0 below). Replace XXX for queue depth with desired depth (I usually test all by factors of 2 (1,2,4,8,16,32,64,128) request size is 128 sectors (64K) (as I was mainly testing a network file server here and that was the normal size I saw with my workload which is the max w/ smb v1).

XDD -verbose -op read -target S0 -blocksize 512 -reqsize 128 -mbytes 32768 -passes 3 -seek random -seek range 64000000 -queuedepth XXX
XDD -verbose -op read -target S0 -blocksize 512 -reqsize 128 -mbytes 32768 -passes 3 -seek random -seek range 64000000 -queuedepth XXX

XDD mainly provides IOPS readings on the subsystem. crystal mark 'sort of' does the same but much more limited, however it will provide at least an indication of subsystem performance.

---
Good that the drives are not partitioned (then you don't have to worry about alignment) that's basically how I run all my file systems here at home under unix (no partition tables, just raw file systems) very helpful when you change the underlaying subsystem often. Anyway if you don't have partitions then you have to change the target w/ XDD above to not go to a file on a device but the device itself just like you do w/ hdtach.

Alright. Won't have xdd results until after I get back from vacation next week though. The manual looks long enough that I want to read through it in some detail before using. Is that source code included with the zip just reference? I don't have visual studio installed and likely have no intention of compiling anything for these tests due to the time it is already going to take me to complete what I have committed to.
12-20-2008, 06:56 PM
stevecs

The main item is to turn off write cache on the drives and OS as those do not have a battery backup unit to retain data in case of a power loss which is generally how a redundant array should be installed (if done properly). The read ahead caches can be enabled as they do not have any affect on the subsystem reliability (the write does as without it you will screw up your subsystem in the case of a power failure). The only issue with read caches I was just commenting on was that with them enabled you are not seeing physically what the drives can do natively, if that's important to your results leave them on. What 'aggressive' does on the areca is read in more sectors after the actual request in anticipation. What this means is that say you have a request to read 32K and have other requests in the queue. What will happen is that the 32K request will be fulfilled and then ADDITIONAL data will be read to anticipate another request which may or may not come in, but it will HOLD OFF on executing the other requests until it's done (ie, helps with streaming type requests and small queue sizes but actually hurts performance the more random/more requests you have). Whether you want to test that or not is up to your needs I was just trying to make it clear what that really did.
12-20-2008, 07:49 PM
stevecs

Yes, you get a pre-compiled binary that you can run from the command line. for example here's a run I just did against my 2nd raw partition here (raid-10/8 savvios in sig/ 8K stripe size on areca) this is only a read as I have data on that array that I couldn't move around.

(BTW/ to do it against a physical disk you need to do this in windows:

D:\Temp\xdd\bin>xdd.exe -verbose -op read -targets 1 \\.\\PhysicalDrive2 -dio -blocksize 512 -reqsize 128 -mbytes 8192 -passes 3 -se
ek random -seek range 64000000 -queuedepth 1

The physical disk number I just got from everest (subtract 1 from list as it really starts at 0)though I just noticed that when using physical discs in windows (XP X64 at least) you can't use a queuedepth > 1 so below I used files (2GB each) in the ntfs filesystem (NON-alligned as I'm using 8K stripe size w/ 8 drives in raid-10 (32K/(8drives/2)) filesystem starts at sector 63.) Will be fixing this shortly once I get some sleep. ;)

Code:

D:\Temp\xdd\bin>xdd.exe -verbose -op read -targets 1 S32GiB -dio -blocksize 512 -reqsize 128 -mbytes 8192 -passes 3 -seek rand ek range 64000000 -queuedepth 128 IOIOIOIOIOIOIOIOIOIOI XDD version 6.5.013007.0001 IOIOIOIOIOIOIOIOIOIOIOI xdd - I/O Performance Inc. Copyright 1992-2007 Starting time for this run, Sat Dec 20 21:43:04 2008 ID for this run, 'No ID Specified' Maximum Process Priority, disabled Passes, 3 Pass Delay in seconds, 0 Maximum Error Threshold, 0 Target Offset, 0 I/O Synchronization, 0 Total run-time limit in seconds, 0 Output file name, stdout CSV output file name, Error output file name, stderr Pass seek randomization, disabled File write synchronization, disabled Pass synchronization barriers, enabled Number of Targets, 1 Number of I/O Threads, 128 Computer Name, ANALYTICAL, User Name, stcost Operating System Info: NT 5.2 Build 3790 Service Pack 2 Page size in bytes, 4096 Number of processors on this system, 4 Megabytes of physical memory, 4095 Seconds before starting, 0 Target[0] Q[0], S32GiB Target directory, "./" Process ID, 4512 Thread ID, 4512 Processor, all/any Read/write ratio, 100.00, 0.00 Throttle in MB/sec, 0.00 Per-pass time limit in seconds, 0 Blocksize in bytes, 512 Request size, 128, blocks, 65536, bytes Number of Requests, 1024 Start offset, 0 Number of MegaBytes, 8192 Pass Offset in blocks, 0 I/O memory buffer is a normal memory buffer I/O memory buffer alignment in bytes, 4096 Data pattern in buffer, '0x00' Data buffer verification is disabled. Direct I/O, enabled Seek pattern, queued_interleaved Seek range, 64000000 Preallocation, 0 Queue Depth, 128 Timestamping, disabled Delete file, disabled T Q Bytes Ops Time Rate IOPS Latency %CPU OP_Type ReqSize TARGET PASS0001 0 128 8589934592 131072 109.014 78.797 1202.35 0.0008 3.86 read 65536 TARGET PASS0002 0 128 8589934592 131072 109.936 78.136 1192.26 0.0008 3.58 read 65536 TARGET PASS0003 0 128 8589934592 131072 109.674 78.322 1195.10 0.0008 3.84 read 65536 TARGET Average 0 128 25769803776 393216 327.624 78.657 1200.21 0.0008 3.76 read 65536 Combined 1 128 25769803776 393216 328.000 78.566 1198.83 0.0008 3.69 read 65536 Ending time for this run, Sat Dec 20 21:48:34 2008
12-21-2008, 01:04 AM
alfaunits

Thanks for clarifying the Aggressive setting, I was wondering what that option really does. (what is conservative BTW?)
Though, I find it doesn't matter which ReadAhead I choose, the streaming reads always have the same speed.
12-21-2008, 06:01 AM
XS Janus

That's a lot of testing! :)
Be sure to compile all you final data into a easily understandable table so we can compare all the results more easily.

Thanks for all the effort!
:)
12-21-2008, 06:07 AM
m^2

It seems we'll have a really good thread. Thank you for this.
12-21-2008, 06:19 AM
stevecs

conservative/aggressive/normal are just labels for varying degrees to the same thing, basically aggressive reads more sectors than 'normal'; conservative reads less. I don't have the exact numbers from areca on it (never really bothered to ask for them). disabled turns it off all together which is that the drive will only read what the request was for which is best for worst-case (100% random) workloads.
12-21-2008, 09:25 AM
Speederlander

Out for vacation. I'll post more results when I get back. Thanks.

Show 100 post(s) from this thread on one page