Some RAID Benches (5, 6, 50, and 60): Areca 1680, VRaptors, 256 bit AES, etc.

Printable View

12-20-2008, 02:22 PM
Speederlander

2 Attachment(s)

Some RAID Benches: 0, 10, 60 with Areca 1680 and 8 Velociraptors

Playing around with different configurations, figure I would make the data available to all.

System basics:
Vista Ult 64-bit
Supermicro X7DWA-N
2x Harpertown 5420's BSEL'd to 3GHz
16GB 800MHz Kingston FB-DIMM
Areca 1680ix-16 with battery back-up
8x 300GB Velociraptor HDDs

Areca specs (latest firmware):
Controller Name ARC-1680
Firmware Version V1.45 2008-09-16
BOOT ROM Version V1.45 2008-09-15
SAS Firmware Version 4.4.3.0
Main Processor 1.2GHz IOP348 C1
CPU ICache Size 32KBytes
CPU DCache Size 32KBytes/Write Back
CPU SCache Size 512KBytes/Write Back
System Memory 512MB/533MHz/ECC
12-20-2008, 02:24 PM
Speederlander

3 Attachment(s)

8 Drive RAID 0, 128K Stripe, 120GB Volume, Write Back, Disk Write Cache On, OS Caching On, NCQ On
12-20-2008, 02:25 PM
Speederlander

3 Attachment(s)

8 Drive RAID 0, 128K Stripe, 120GB Volume, Write Back, Disk Write Cache Off, OS Caching Off, NCQ On
12-20-2008, 02:28 PM
Speederlander

3 Attachment(s)

8 Drive RAID 0, 128K Stripe, 120GB Volume, Write Back, Disk Write Cache Off, OS Caching Off, NCQ Off

NOTE: No further tests with NCQ off will be performed.
12-20-2008, 02:35 PM
Speederlander

3 Attachment(s)

8 Drive RAID 10, 128K Stripe, 120GB Volume, Write Back, Disk Write Cache On, OS Caching On
12-20-2008, 02:38 PM
Speederlander

****reserved
12-20-2008, 02:40 PM
Speederlander

*****reserved
12-20-2008, 02:44 PM
Speederlander

******reserved
12-20-2008, 02:46 PM
Speederlander

*******reserved
12-20-2008, 02:47 PM
Speederlander

********reserved
12-20-2008, 02:50 PM
Speederlander

*********reserved
12-20-2008, 02:53 PM
Speederlander

**********reserved
12-20-2008, 02:55 PM
Speederlander

***********reserved
12-20-2008, 02:58 PM
Speederlander

************reserved
12-20-2008, 03:57 PM
stevecs

You want to turn off write caching in windows so that your OS/system cache doesn't colour the results too much (it always will unless doing direct I/O but it will cut down the majority of it's effects). Also set disk write cache mode to explicitly OFF on the areca (so drives don't colour results, this means that writes will only be cached on the card's ram IF you have a BBU (no bbu/no caching). That way at least you don't have any ambiguity.

Also if you could do something like simple crystaldiskmark I/O checks as it has random read/write just make sure you select the 1GB test size and 9x average (would be better if you did it before the 2GB cache upgrade). it would be better to run iozone/xdd but that's going to be much more intensive and you probably don't want to waste the time on it.

Aggressive read-ahead will affect random I/O (more aggressive the lower your random I/O results will be but better for streaming).

since you're on windows and looks like running against a partitioned space, are you aligning the partitions to your stripe size? (32K start for XP/earlier or 1MB for vista/2008).
12-20-2008, 04:36 PM
Speederlander

Quote:

Originally Posted by stevecs

You want to turn off write caching in windows so that your OS/system cache doesn't colour the results too much (it always will unless doing direct I/O but it will cut down the majority of it's effects). Also set disk write cache mode to explicitly OFF on the areca (so drives don't colour results, this means that writes will only be cached on the card's ram IF you have a BBU (no bbu/no caching). That way at least you don't have any ambiguity.

I do have bbu.
I will turn off windows write caching (it is off already I believe as this is fresh OS install).
I will turn off disk caching on the areca.

Quote:

Also if you could do something like simple crystaldiskmark I/O checks as it has random read/write just make sure you select the 1GB test size and 9x average (would be better if you did it before the 2GB cache upgrade). it would be better to run iozone/xdd but that's going to be much more intensive and you probably don't want to waste the time on it.

Downloading crystaldiskmark now. I will include it.
I'll try iozone. Please provide settings as I have no previous experience with it.

Quote:

Aggressive read-ahead will affect random I/O (more aggressive the lower your random I/O results will be but better for streaming).

I'll include results for both if possible.

What would you like to see these set to (for better random IO, etc characterization):

Quote:

HDD Read Ahead Cache
Enabled

Volume Data Read Ahead
Aggressive

Disk Write Cache Mode
Auto

Quote:

since you're on windows and looks like running against a partitioned space, are you aligning the partitions to your stripe size? (32K start for XP/earlier or 1MB for vista/2008).

Drives are unpartitioned. HDTach write tests wouldn't function on anything with partitions. I haven't traditionally messed too much with aligning partitions. Feel free to provide a bit more info and I will add those results as well if time permits.
12-20-2008, 04:56 PM
stevecs

Actually, if you're going to d/l text tools, xdd would be better to show the different performance profiles of the raids faster. http://www.ioperformance.com/products.htm

for your specs you should use the following (note, create a file ~64GB in size or larger I have it called S0 below). Replace XXX for queue depth with desired depth (I usually test all by factors of 2 (1,2,4,8,16,32,64,128) request size is 128 sectors (64K) (as I was mainly testing a network file server here and that was the normal size I saw with my workload which is the max w/ smb v1).

XDD -verbose -op read -target S0 -blocksize 512 -reqsize 128 -mbytes 32768 -passes 3 -seek random -seek range 64000000 -queuedepth XXX
XDD -verbose -op read -target S0 -blocksize 512 -reqsize 128 -mbytes 32768 -passes 3 -seek random -seek range 64000000 -queuedepth XXX

XDD mainly provides IOPS readings on the subsystem. crystal mark 'sort of' does the same but much more limited, however it will provide at least an indication of subsystem performance.

---
Good that the drives are not partitioned (then you don't have to worry about alignment) that's basically how I run all my file systems here at home under unix (no partition tables, just raw file systems) very helpful when you change the underlaying subsystem often. Anyway if you don't have partitions then you have to change the target w/ XDD above to not go to a file on a device but the device itself just like you do w/ hdtach.
12-20-2008, 05:02 PM
Speederlander

Quote:

Originally Posted by stevecs

Actually, if you're going to d/l text tools, xdd would be better to show the different performance profiles of the raids faster. http://www.ioperformance.com/products.htm

for your specs you should use the following (note, create a file ~64GB in size or larger I have it called S0 below). Replace XXX for queue depth with desired depth (I usually test all by factors of 2 (1,2,4,8,16,32,64,128) request size is 128 sectors (64K) (as I was mainly testing a network file server here and that was the normal size I saw with my workload which is the max w/ smb v1).

XDD -verbose -op read -target S0 -blocksize 512 -reqsize 128 -mbytes 32768 -passes 3 -seek random -seek range 64000000 -queuedepth XXX
XDD -verbose -op read -target S0 -blocksize 512 -reqsize 128 -mbytes 32768 -passes 3 -seek random -seek range 64000000 -queuedepth XXX

XDD mainly provides IOPS readings on the subsystem. crystal mark 'sort of' does the same but much more limited, however it will provide at least an indication of subsystem performance.

---
Good that the drives are not partitioned (then you don't have to worry about alignment) that's basically how I run all my file systems here at home under unix (no partition tables, just raw file systems) very helpful when you change the underlaying subsystem often. Anyway if you don't have partitions then you have to change the target w/ XDD above to not go to a file on a device but the device itself just like you do w/ hdtach.

Alright. Won't have xdd results until after I get back from vacation next week though. The manual looks long enough that I want to read through it in some detail before using. Is that source code included with the zip just reference? I don't have visual studio installed and likely have no intention of compiling anything for these tests due to the time it is already going to take me to complete what I have committed to.
12-20-2008, 06:56 PM
stevecs

The main item is to turn off write cache on the drives and OS as those do not have a battery backup unit to retain data in case of a power loss which is generally how a redundant array should be installed (if done properly). The read ahead caches can be enabled as they do not have any affect on the subsystem reliability (the write does as without it you will screw up your subsystem in the case of a power failure). The only issue with read caches I was just commenting on was that with them enabled you are not seeing physically what the drives can do natively, if that's important to your results leave them on. What 'aggressive' does on the areca is read in more sectors after the actual request in anticipation. What this means is that say you have a request to read 32K and have other requests in the queue. What will happen is that the 32K request will be fulfilled and then ADDITIONAL data will be read to anticipate another request which may or may not come in, but it will HOLD OFF on executing the other requests until it's done (ie, helps with streaming type requests and small queue sizes but actually hurts performance the more random/more requests you have). Whether you want to test that or not is up to your needs I was just trying to make it clear what that really did.
12-20-2008, 07:49 PM
stevecs

Yes, you get a pre-compiled binary that you can run from the command line. for example here's a run I just did against my 2nd raw partition here (raid-10/8 savvios in sig/ 8K stripe size on areca) this is only a read as I have data on that array that I couldn't move around.

(BTW/ to do it against a physical disk you need to do this in windows:

D:\Temp\xdd\bin>xdd.exe -verbose -op read -targets 1 \\.\\PhysicalDrive2 -dio -blocksize 512 -reqsize 128 -mbytes 8192 -passes 3 -se
ek random -seek range 64000000 -queuedepth 1

The physical disk number I just got from everest (subtract 1 from list as it really starts at 0)though I just noticed that when using physical discs in windows (XP X64 at least) you can't use a queuedepth > 1 so below I used files (2GB each) in the ntfs filesystem (NON-alligned as I'm using 8K stripe size w/ 8 drives in raid-10 (32K/(8drives/2)) filesystem starts at sector 63.) Will be fixing this shortly once I get some sleep. ;)

Code:

D:\Temp\xdd\bin>xdd.exe -verbose -op read -targets 1 S32GiB -dio -blocksize 512 -reqsize 128 -mbytes 8192 -passes 3 -seek rand ek range 64000000 -queuedepth 128 IOIOIOIOIOIOIOIOIOIOI XDD version 6.5.013007.0001 IOIOIOIOIOIOIOIOIOIOIOI xdd - I/O Performance Inc. Copyright 1992-2007 Starting time for this run, Sat Dec 20 21:43:04 2008 ID for this run, 'No ID Specified' Maximum Process Priority, disabled Passes, 3 Pass Delay in seconds, 0 Maximum Error Threshold, 0 Target Offset, 0 I/O Synchronization, 0 Total run-time limit in seconds, 0 Output file name, stdout CSV output file name, Error output file name, stderr Pass seek randomization, disabled File write synchronization, disabled Pass synchronization barriers, enabled Number of Targets, 1 Number of I/O Threads, 128 Computer Name, ANALYTICAL, User Name, stcost Operating System Info: NT 5.2 Build 3790 Service Pack 2 Page size in bytes, 4096 Number of processors on this system, 4 Megabytes of physical memory, 4095 Seconds before starting, 0 Target[0] Q[0], S32GiB Target directory, "./" Process ID, 4512 Thread ID, 4512 Processor, all/any Read/write ratio, 100.00, 0.00 Throttle in MB/sec, 0.00 Per-pass time limit in seconds, 0 Blocksize in bytes, 512 Request size, 128, blocks, 65536, bytes Number of Requests, 1024 Start offset, 0 Number of MegaBytes, 8192 Pass Offset in blocks, 0 I/O memory buffer is a normal memory buffer I/O memory buffer alignment in bytes, 4096 Data pattern in buffer, '0x00' Data buffer verification is disabled. Direct I/O, enabled Seek pattern, queued_interleaved Seek range, 64000000 Preallocation, 0 Queue Depth, 128 Timestamping, disabled Delete file, disabled T Q Bytes Ops Time Rate IOPS Latency %CPU OP_Type ReqSize TARGET PASS0001 0 128 8589934592 131072 109.014 78.797 1202.35 0.0008 3.86 read 65536 TARGET PASS0002 0 128 8589934592 131072 109.936 78.136 1192.26 0.0008 3.58 read 65536 TARGET PASS0003 0 128 8589934592 131072 109.674 78.322 1195.10 0.0008 3.84 read 65536 TARGET Average 0 128 25769803776 393216 327.624 78.657 1200.21 0.0008 3.76 read 65536 Combined 1 128 25769803776 393216 328.000 78.566 1198.83 0.0008 3.69 read 65536 Ending time for this run, Sat Dec 20 21:48:34 2008
12-21-2008, 01:04 AM
alfaunits

Thanks for clarifying the Aggressive setting, I was wondering what that option really does. (what is conservative BTW?)
Though, I find it doesn't matter which ReadAhead I choose, the streaming reads always have the same speed.
12-21-2008, 06:01 AM
XS Janus

That's a lot of testing! :)
Be sure to compile all you final data into a easily understandable table so we can compare all the results more easily.

Thanks for all the effort!
:)
12-21-2008, 06:07 AM
m^2

It seems we'll have a really good thread. Thank you for this.
12-21-2008, 06:19 AM
stevecs

conservative/aggressive/normal are just labels for varying degrees to the same thing, basically aggressive reads more sectors than 'normal'; conservative reads less. I don't have the exact numbers from areca on it (never really bothered to ask for them). disabled turns it off all together which is that the drive will only read what the request was for which is best for worst-case (100% random) workloads.
12-21-2008, 09:25 AM
Speederlander

Out for vacation. I'll post more results when I get back. Thanks.
12-21-2008, 03:17 PM
XS Janus

Sh:@t dude!
Way to tease! :D
12-21-2008, 05:11 PM
tiro_uspsss

nice thread! :yepp: :toast: :bows: :up:
12-22-2008, 09:18 PM
Gogeta

Thank you for the time its going to take to run all of these benches and compile the results.

With regard to I/O testing, how does bonnie++ compare? The OP probably won't run Linux for his testing, but I think its a valid question while he enjoys some vacation time.
12-23-2008, 07:28 AM
stevecs

Bonnie++ is a fast test (ie, akin to hdtune; atto; crystaldiskmark; et al. for windows that are shown here often) not something that will provide you real good data for comparison. I use it mainly as a first step to get a 50,000 foot view and then start running real tests tailored toward what I am trying to bench. If you are running it yourself, normally what you want to do is run it 5-9 times, remove the best & worst run and average the rest to help remove some of the fluctuations.
12-30-2008, 12:38 AM
Speederlander

Quote:

Originally Posted by stevecs

Yes, you get a pre-compiled binary that you can run from the command line. for example here's a run I just did against my 2nd raw partition here (raid-10/8 savvios in sig/ 8K stripe size on areca) this is only a read as I have data on that array that I couldn't move around.

(BTW/ to do it against a physical disk you need to do this in windows:

D:\Temp\xdd\bin>xdd.exe -verbose -op read -targets 1 \\.\\PhysicalDrive2 -dio -blocksize 512 -reqsize 128 -mbytes 8192 -passes 3 -se
ek random -seek range 64000000 -queuedepth 1

The physical disk number I just got from everest (subtract 1 from list as it really starts at 0)though I just noticed that when using physical discs in windows (XP X64 at least) you can't use a queuedepth > 1 so below I used files (2GB each) in the ntfs filesystem (NON-alligned as I'm using 8K stripe size w/ 8 drives in raid-10 (32K/(8drives/2)) filesystem starts at sector 63.) Will be fixing this shortly once I get some sleep. ;)

Code:

D:\Temp\xdd\bin>xdd.exe -verbose -op read -targets 1 S32GiB -dio -blocksize 512 -reqsize 128 -mbytes 8192 -passes 3 -seek rand ek range 64000000 -queuedepth 128 IOIOIOIOIOIOIOIOIOIOI XDD version 6.5.013007.0001 IOIOIOIOIOIOIOIOIOIOIOI xdd - I/O Performance Inc. Copyright 1992-2007 Starting time for this run, Sat Dec 20 21:43:04 2008 ID for this run, 'No ID Specified' Maximum Process Priority, disabled Passes, 3 Pass Delay in seconds, 0 Maximum Error Threshold, 0 Target Offset, 0 I/O Synchronization, 0 Total run-time limit in seconds, 0 Output file name, stdout CSV output file name, Error output file name, stderr Pass seek randomization, disabled File write synchronization, disabled Pass synchronization barriers, enabled Number of Targets, 1 Number of I/O Threads, 128 Computer Name, ANALYTICAL, User Name, stcost Operating System Info: NT 5.2 Build 3790 Service Pack 2 Page size in bytes, 4096 Number of processors on this system, 4 Megabytes of physical memory, 4095 Seconds before starting, 0 Target[0] Q[0], S32GiB Target directory, "./" Process ID, 4512 Thread ID, 4512 Processor, all/any Read/write ratio, 100.00, 0.00 Throttle in MB/sec, 0.00 Per-pass time limit in seconds, 0 Blocksize in bytes, 512 Request size, 128, blocks, 65536, bytes Number of Requests, 1024 Start offset, 0 Number of MegaBytes, 8192 Pass Offset in blocks, 0 I/O memory buffer is a normal memory buffer I/O memory buffer alignment in bytes, 4096 Data pattern in buffer, '0x00' Data buffer verification is disabled. Direct I/O, enabled Seek pattern, queued_interleaved Seek range, 64000000 Preallocation, 0 Queue Depth, 128 Timestamping, disabled Delete file, disabled T Q Bytes Ops Time Rate IOPS Latency %CPU OP_Type ReqSize TARGET PASS0001 0 128 8589934592 131072 109.014 78.797 1202.35 0.0008 3.86 read 65536 TARGET PASS0002 0 128 8589934592 131072 109.936 78.136 1192.26 0.0008 3.58 read 65536 TARGET PASS0003 0 128 8589934592 131072 109.674 78.322 1195.10 0.0008 3.84 read 65536 TARGET Average 0 128 25769803776 393216 327.624 78.657 1200.21 0.0008 3.76 read 65536 Combined 1 128 25769803776 393216 328.000 78.566 1198.83 0.0008 3.69 read 65536 Ending time for this run, Sat Dec 20 21:48:34 2008

OK, so I'm trying to target my G: RAID10 drive and I can't seem to get the syntax correct. What was S32GiB in your example?
12-30-2008, 04:36 AM
stevecs

That was a file named "32GiB" (which was 32GiB in size) that I created on that partition. The syntax should be the same or is it blowing up as you don't have a file created there for it to use?
12-30-2008, 09:53 AM
Speederlander

Quote:

Originally Posted by stevecs

That was a file named "32GiB" (which was 32GiB in size) that I created on that partition. The syntax should be the same or is it blowing up as you don't have a file created there for it to use?

Yeah, didn't have the file. It was odd as it seemed to create the file the very first time I ran it but it's been failing since then. Maybe I accidently had a file of that name the first time from something else... Do you create the file ahead of time then on the target? Simple text file?

NCQ on, off or post results for both is time permits?
12-30-2008, 10:02 AM
stevecs

Yeah, sort of, I created it on my unix box (dd if=/dev/zero of=32GiB bs=1M count=32768) as that's where I was at the time and then just moved it over. it doesn't really matter where you create it or what's in it, it's just a pointer to blocks that xdd will operate on. I left NCQ on for the drives I have here, if you have time it would be interesting to see what type of affect it has).
12-30-2008, 10:05 AM
Speederlander

Quote:

Originally Posted by stevecs

Yeah, sort of, I created it on my unix box (dd if=/dev/zero of=32GiB bs=1M count=32768) as that's where I was at the time and then just moved it over. it doesn't really matter where you create it or what's in it, it's just a pointer to blocks that xdd will operate on. I left NCQ on for the drives I have here, if you have time it would be interesting to see what type of affect it has).

Here are the variables I am looking at testing against to see the impact. I obviously can't vary all of them against each other without taking 2 months to complete testing. So I need to choose a few to set constant and not mess with. Any suggestions?

Areca Settings:
Sata NCQ Support: on/off
HDD Read Ahead Cache: enabled/disabled
Volume Data Read Ahead: disabled/conservative/normal/aggresive
HDD Queue Depth: 1/2/4/8/16/32
Disk Write Cache Mode: Auto/Enabled/Disabled
Volume Cache Mode: Write Through/Write Back
Volume Stripe Size: 32/64/128

OS (Vista 64) Settings:
Enable Write Caching On Disk: on/off
Enable Advanced Performance: on/off

Also:
Can try with BBU plugged in or unplugged (for 512meg cache)
Can compare 512meg cache results to 2GB (2GB only with BBU plugged in)
12-30-2008, 10:53 AM
stevecs

Looks like a good overview for just base settings and their affects. stripe sizes, file system alignments et al will as you mentioned take quite a bit longer to plot out (I know, it takes about 4-6 months to do a full gamut test when I do them on arrays here).

The BBU itself won't provide anything different except for data integrity which you are not testing. You can change the cache sizes and then manually set your "Disk Write Cache Mode". Auto means (turn on write cache if BBU is present". So you can manually just turn it on or off on the card for testing purposes.

You probably want to reboot when setting these items just to avoid any issues w/ firmware not applying settings immediately. Volume stripe sizes are going to be a killer, you probably want to either save those for another test or at least last unless you are planning on limiting it to only a specific raid type (to get real value you will want to do all permutations which is what takes a lot of time). I would probably approach it in two modes, first with a raid-0 and then use that to test out all combinations of the areca settings (with OS settings off then on for each test) with the exception of stripe size (use the default of 64k). Then do the same with a pass-through disk (single disk but attached to the areca) this would be a baseline and show comparison to the multiplication affects of striping. After that then do tests with other raid levels (as the base settings are not germane to a raid-type but are low-level and would apply equally).
12-30-2008, 12:52 PM
Speederlander

Quote:

Originally Posted by stevecs

Yeah, sort of, I created it on my unix box (dd if=/dev/zero of=32GiB bs=1M count=32768) as that's where I was at the time and then just moved it over. it doesn't really matter where you create it or what's in it, it's just a pointer to blocks that xdd will operate on. I left NCQ on for the drives I have here, if you have time it would be interesting to see what type of affect it has).

Alright, so I just want to run a test case to solve my problem with the non-existing target.

I have a folder called xdd on my G: drive (the Raid 10 drive) with the xdd bin folder and everything else in it.

g:\xdd\bin>xdd.exe -op read -targets 1 _________

I want to target that drive and both the volume and the logical drive are named RAID10. No combo I have tried gets me anything other than "The system cannot find the file specified". I have tried files in the g:\xdd\bin directory and the drive itself.

I know I'm missing something simple. I'm pretty sure it's a syntax issue.
12-30-2008, 01:00 PM
Dean

When you say "newest firmware" are you talking about firmware on your RAID controller?

Awesome post, man. :)

Thanks,

-Dean
12-30-2008, 01:06 PM
Speederlander

Quote:

Originally Posted by Dean

When you say "newest firmware" are you talking about firmware on your RAID controller?

Yes. But I am going to redo the organization. The old firmware results will go away. I just happened to be in the middle of doing updates when I started and wanted to see any difference in results between old and new.
12-30-2008, 02:32 PM
stevecs

@speederlander: you just need to create a file then for it to run against. You can grab a windows version of DD at http://www.chrysocome.net/dd or use any other tool to create a file for xdd to point to. or you can use the -createnewfiles option which will take more time (it actually creates a new file per pass). So something like:

xdd.exe -verbose -op read -targets 1 S32GiB -createnewfiles -dio -blocksize 512 -reqsize 128 -mbytes 8192 -passes 3 -seek random -seek range 64000000 -queuedepth 128
12-31-2008, 09:51 AM
Gogeta

Speederlander, will you be posting updates with results as the tests complete, or in one large sum at the end of testing?
12-31-2008, 10:01 AM
Speederlander

Quote:

Originally Posted by Gogeta

Speederlander, will you be posting updates with results as the tests complete, or in one large sum at the end of testing?

Updates as I go.
12-31-2008, 10:10 AM
Speederlander

1 Attachment(s)

Quote:

Originally Posted by stevecs

@speederlander: you just need to create a file then for it to run against. You can grab a windows version of DD at http://www.chrysocome.net/dd or use any other tool to create a file for xdd to point to. or you can use the -createnewfiles option which will take more time (it actually creates a new file per pass). So something like:

xdd.exe -verbose -op read -targets 1 S32GiB -createnewfiles -dio -blocksize 512 -reqsize 128 -mbytes 8192 -passes 3 -seek random -seek range 64000000 -queuedepth 128

Alright, I obviously have lost my ability to use the command line. Hence iometer for the near-term. :p:

Disclosure: I freely admit I am not an expert when it comes to the nitty gritty details of storage access patterns, etc.

Do you have any specific pattern(s) you would like to see on iometer?

I need some feedback as to what is most appropriate for good characterization.

How about:
IOMeter Parameters:

Maximum Disk Size: 16777216
Starting Disk Sector: 0
# of Outstanding I/Os: 1
Test Connection Rate: skip

Access Pattern:
Transfer Request Size: 4KB
Percent of Access Specification: 100%
Percent Read/Write Distribution: 67% Read/33% Write
Percent Random/Sequential Distribution: 100% Random/0% Seq
Burstiness: 0ms/1 IOs
Align I/Os on Sector Boundaries
Reply Size: No Reply

Test Set-up:
Run Time: 3 Minutes
Ramp Up: 30 seconds
Cycling Options: Cycle # of Outstanding I/Os -- run step outstanding I/Os on all disks at a time
# of Outstanding I/Os: start: 1 end: 32 power: 2, exponential stepping

TEST RUN:
12-31-2008, 07:15 PM
stevecs

Quote:

Alright, I obviously have lost my ability to use the command line.

blame it on new-years early partying :)

as for iometer I haven't used it much, but you seem to have the general gist. I would suggest that you want to check a couple request sizes, your 4K that you have is good as that's your page size with non-itanium intel chips (ie, similar to what you would have with say a pagefile access). 64K would be good for a file server (smb1) with large client base, then whatever you want to show how it performs with your normal (or modeled) request size.

I would also allow queue depth to go to 256 or at least 128. The areca will spread that out and the 1680 has a queue depth of 256 (most drives allow for at least 32 each) this will basically saturate the drive (worst case).

The read/write distribution you have (67/33) is decent for a file server, but you probably want a pure 100/0 0/100 test for comparison sake.
12-31-2008, 09:29 PM
Speederlander

2 Attachment(s)

Test Schedule: (may throw in stripe variations as well...)
01-01-2009, 04:33 PM
Speederlander

First results posted to reserved post spots at the beginning of the thread.

Next test to be RAID 0 with disk write caching off and OS caching off.
01-02-2009, 01:02 AM
Speederlander

Comparison of RAID 0 results:

*removed for editing*
01-02-2009, 05:21 AM
stevecs

You may want mention in your column headers (for future lookups) "RAID 0 Write Cache [On|Off]" so people will know it's not read cache or whatnot. Also just for clarification you have MBps which indicates Mega = 1,000,000 bytes. If That's what you mean great. if you actually mean 1,048,576 bytes you should use the MiBps abbreviation. I haven't had a chance to go through the data in detail but looks good so far. :)
01-02-2009, 09:08 AM
Speederlander

Quote:

Originally Posted by stevecs

You may want mention in your column headers (for future lookups) "RAID 0 Write Cache [On|Off]" so people will know it's not read cache or whatnot. Also just for clarification you have MBps which indicates Mega = 1,000,000 bytes. If That's what you mean great. if you actually mean 1,048,576 bytes you should use the MiBps abbreviation. I haven't had a chance to go through the data in detail but looks good so far. :)

Thanks. The MBps header is the iometer header from their output csv file.

With respect to the headers, you mean the thing I posted in post #45? That was just a quick and dirty. It's going away. The information in the first posts of the thread where I am putting all the data is sufficient, yes?
01-02-2009, 10:02 AM
Speederlander

Added results for caching off and NCQ off. Interesting impact. NCQ stay on in any event for the remainder of tests.

RAID 10 next I think.
01-02-2009, 07:10 PM
Speederlander

Should I be comparing 8 drive RAID 10 to 4 drive RAID 0 to see the RAID 10 scaling?
01-02-2009, 07:25 PM
stevecs

We would expect that RAID-0 (4 drive) should handle the same number of write operations as raid-10 (8 drive) but both RAID-10 (4 drive) and raid-0 (4 drive) should exhibit close to the same number of read operations (not exact as it depends on where you fall in relation to your stripe size) but it would be interesting to see both a 8 & 4 way raid-10 compared against a 4-way raid-0.
01-02-2009, 07:38 PM
Speederlander

First RAID 10 results are now posted.
01-02-2009, 07:41 PM
Speederlander

Quote:

Originally Posted by stevecs

We would expect that RAID-0 (4 drive) should handle the same number of write operations as raid-10 (8 drive) but both RAID-10 (4 drive) and raid-0 (4 drive) should exhibit close to the same number of read operations (not exact as it depends on where you fall in relation to your stripe size) but it would be interesting to see both a 8 & 4 way raid-10 compared against a 4-way raid-0.

Well, I'm seeing an 8-drive RAID 10 lag behind 8-drive RAID 0 in reads. I'll try to do some variations on drive sizes with a 4 drive RAID 10 and a 4 drive RAID 0 once all the RAID 10 stuff is finished. I do plan on doing some stripe variation on the 10 and 60, with 128, 64 and 32 all run. One of those could even things out.

I will have some questions as to how the striping functions with a RAID 50/60 later I think.
01-03-2009, 01:07 PM
stevecs

And I can expect that for general requests. Basically it should come down to the distribution of the requested blocks on the subsystem. If you're requesting blocks that fall on the 'same disk' for example in a raid-0 you should see 2x that for a raid-1 (10) as you have two copies. However if you are NOT doing that (ie, block request does not hit the same spindles) then you will see more from a raid-0. I don't use iometer so I don't know how you would show such a scenario.

And if you have time you may want to show the write/read differences between say a raid 5/6 of 4 drives and a raid-50/60 of 8 drives. The writes should increase by 2 (assuming perfectly balanced/random workload). reads should decrease (as you have less data drives, more parity drives) especially with such small arrays (4/8 drives)
01-05-2009, 08:55 PM
[XC] riptide

My eyes! Need-graphs-WTF-BBQ! :D

Great stuff... you might run a few graphs at the end. :up:
01-08-2009, 10:50 AM
Speederlander

Quote:

Originally Posted by [XC] riptide

My eyes! Need-graphs-WTF-BBQ! :D

Great stuff... you might run a few graphs at the end. :up:

Had big performance issues actually with this firmware. Just updated to 1.46 beta firm and performance is greatly altered (improved a lot in some cases, not so good for large queue depth situations). Unfortunately most of these numbers are out the door. :(

Once I figure out my path forward I'll probably be creating a new thread.
01-08-2009, 03:39 PM
stevecs

What issues were you running into with the firmware? (also was it really the firmware or the transport or what?)
01-08-2009, 05:15 PM
Speederlander

Quote:

Originally Posted by stevecs

What issues were you running into with the firmware? (also was it really the firmware or the transport or what?)

Every sequential number I got, no matter what the benchmark, including iometer, seemed too low for an array of 8 velociraptors. I was also seeing a lot of settling; big variations in bench results from one run to the next, always moving to worse performance, sometimes going down 20% or more between run 1 and run 5 or 6. I was assuming it was just a function of SATA not playing so well on an Areca 1680 SAS controller. I bought the 1680 for RAID 6 performance anyway so I mostly didn't care. But it was still bugging me so I started looking a little closer. I found the 1.46 firmware beta (firm and boot only) and took a chance. I'm seeing up to 20% or better improvement in low queue depth sequential reads and writes, minor improvement in low queue depth random reads and writes, and an equivalent fall off (going from minor to pretty bad) in high queue depth operations. No more result fall-off now either. They are very stable.

All times are GMT -8. The time now is 12:26 AM.

XtremeSystems