Some RAID Benches (5, 6, 50, and 60): Areca 1680, VRaptors, 256 bit AES, etc.

[XS Janus]

Sh:@t dude!
Way to tease!

[tiro_uspsss]

nice thread!

[Gogeta]

Thank you for the time its going to take to run all of these benches and compile the results.

With regard to I/O testing, how does bonnie++ compare? The OP probably won't run Linux for his testing, but I think its a valid question while he enjoys some vacation time.

[stevecs]

Bonnie++ is a fast test (ie, akin to hdtune; atto; crystaldiskmark; et al. for windows that are shown here often) not something that will provide you real good data for comparison. I use it mainly as a first step to get a 50,000 foot view and then start running real tests tailored toward what I am trying to bench. If you are running it yourself, normally what you want to do is run it 5-9 times, remove the best & worst run and average the rest to help remove some of the fluctuations.

[Speederlander]

Yes, you get a pre-compiled binary that you can run from the command line. for example here's a run I just did against my 2nd raw partition here (raid-10/8 savvios in sig/ 8K stripe size on areca) this is only a read as I have data on that array that I couldn't move around.

(BTW/ to do it against a physical disk you need to do this in windows:

D:\Temp\xdd\bin>xdd.exe -verbose -op read -targets 1 \\.\\PhysicalDrive2 -dio -blocksize 512 -reqsize 128 -mbytes 8192 -passes 3 -se
ek random -seek range 64000000 -queuedepth 1

The physical disk number I just got from everest (subtract 1 from list as it really starts at 0)though I just noticed that when using physical discs in windows (XP X64 at least) you can't use a queuedepth > 1 so below I used files (2GB each) in the ntfs filesystem (NON-alligned as I'm using 8K stripe size w/ 8 drives in raid-10 (32K/(8drives/2)) filesystem starts at sector 63.) Will be fixing this shortly once I get some sleep.

D:\Temp\xdd\bin>xdd.exe -verbose -op read -targets 1 S32GiB -dio -blocksize 512 -reqsize 128 -mbytes 8192 -passes 3 -seek rand
ek range 64000000 -queuedepth 128

IOIOIOIOIOIOIOIOIOIOI XDD version 6.5.013007.0001 IOIOIOIOIOIOIOIOIOIOIOI
xdd - I/O Performance Inc. Copyright 1992-2007
Starting time for this run, Sat Dec 20 21:43:04 2008

ID for this run, 'No ID Specified'
Maximum Process Priority, disabled
Passes, 3
Pass Delay in seconds, 0
Maximum Error Threshold, 0
Target Offset, 0
I/O Synchronization, 0
Total run-time limit in seconds, 0
Output file name, stdout
CSV output file name,
Error output file name, stderr
Pass seek randomization, disabled
File write synchronization, disabled
Pass synchronization barriers, enabled
Number of Targets, 1
Number of I/O Threads, 128

Computer Name, ANALYTICAL, User Name, stcost
Operating System Info: NT 5.2 Build 3790 Service Pack 2
Page size in bytes, 4096
Number of processors on this system, 4
Megabytes of physical memory, 4095
Seconds before starting, 0
        Target[0] Q[0], S32GiB
                Target directory, "./"
                Process ID, 4512
                Thread ID, 4512
                Processor, all/any
                Read/write ratio, 100.00,  0.00
                Throttle in MB/sec,   0.00
                Per-pass time limit in seconds, 0
                Blocksize in bytes, 512
                Request size, 128, blocks, 65536, bytes
                Number of Requests, 1024
                Start offset, 0
                Number of MegaBytes, 8192
                Pass Offset in blocks, 0
                I/O memory buffer is a normal memory buffer
                I/O memory buffer alignment in bytes, 4096
                Data pattern in buffer, '0x00'
                Data buffer verification is disabled.
                Direct I/O, enabled
                Seek pattern, queued_interleaved
                Seek range, 64000000
                Preallocation, 0
                Queue Depth, 128
                Timestamping, disabled
                Delete file, disabled

                     T  Q       Bytes      Ops    Time      Rate      IOPS   Latency     %CPU  OP_Type    ReqSize
TARGET   PASS0001    0 128    8589934592 131072 109.014    78.797    1202.35    0.0008     3.86   read       65536
TARGET   PASS0002    0 128    8589934592 131072 109.936    78.136    1192.26    0.0008     3.58   read       65536
TARGET   PASS0003    0 128    8589934592 131072 109.674    78.322    1195.10    0.0008     3.84   read       65536
TARGET   Average     0 128   25769803776 393216 327.624    78.657    1200.21    0.0008     3.76   read       65536
         Combined    1 128   25769803776 393216 328.000    78.566    1198.83    0.0008     3.69   read       65536
Ending time for this run, Sat Dec 20 21:48:34 2008

OK, so I'm trying to target my G: RAID10 drive and I can't seem to get the syntax correct. What was S32GiB in your example?

[stevecs]

That was a file named "32GiB" (which was 32GiB in size) that I created on that partition. The syntax should be the same or is it blowing up as you don't have a file created there for it to use?

[Speederlander]

That was a file named "32GiB" (which was 32GiB in size) that I created on that partition. The syntax should be the same or is it blowing up as you don't have a file created there for it to use?

Yeah, didn't have the file. It was odd as it seemed to create the file the very first time I ran it but it's been failing since then. Maybe I accidently had a file of that name the first time from something else... Do you create the file ahead of time then on the target? Simple text file?

NCQ on, off or post results for both is time permits?

[stevecs]

Yeah, sort of, I created it on my unix box (dd if=/dev/zero of=32GiB bs=1M count=32768) as that's where I was at the time and then just moved it over. it doesn't really matter where you create it or what's in it, it's just a pointer to blocks that xdd will operate on. I left NCQ on for the drives I have here, if you have time it would be interesting to see what type of affect it has).

[Speederlander]

Yeah, sort of, I created it on my unix box (dd if=/dev/zero of=32GiB bs=1M count=32768) as that's where I was at the time and then just moved it over. it doesn't really matter where you create it or what's in it, it's just a pointer to blocks that xdd will operate on. I left NCQ on for the drives I have here, if you have time it would be interesting to see what type of affect it has).

Here are the variables I am looking at testing against to see the impact. I obviously can't vary all of them against each other without taking 2 months to complete testing. So I need to choose a few to set constant and not mess with. Any suggestions?

Areca Settings:
Sata NCQ Support: on/off
HDD Read Ahead Cache: enabled/disabled
Volume Data Read Ahead: disabled/conservative/normal/aggresive
HDD Queue Depth: 1/2/4/8/16/32
Disk Write Cache Mode: Auto/Enabled/Disabled
Volume Cache Mode: Write Through/Write Back
Volume Stripe Size: 32/64/128


OS (Vista 64) Settings:
Enable Write Caching On Disk: on/off
Enable Advanced Performance: on/off


Also:
Can try with BBU plugged in or unplugged (for 512meg cache)
Can compare 512meg cache results to 2GB (2GB only with BBU plugged in)

[stevecs]

Looks like a good overview for just base settings and their affects. stripe sizes, file system alignments et al will as you mentioned take quite a bit longer to plot out (I know, it takes about 4-6 months to do a full gamut test when I do them on arrays here).

The BBU itself won't provide anything different except for data integrity which you are not testing. You can change the cache sizes and then manually set your "Disk Write Cache Mode". Auto means (turn on write cache if BBU is present". So you can manually just turn it on or off on the card for testing purposes.

You probably want to reboot when setting these items just to avoid any issues w/ firmware not applying settings immediately. Volume stripe sizes are going to be a killer, you probably want to either save those for another test or at least last unless you are planning on limiting it to only a specific raid type (to get real value you will want to do all permutations which is what takes a lot of time). I would probably approach it in two modes, first with a raid-0 and then use that to test out all combinations of the areca settings (with OS settings off then on for each test) with the exception of stripe size (use the default of 64k). Then do the same with a pass-through disk (single disk but attached to the areca) this would be a baseline and show comparison to the multiplication affects of striping. After that then do tests with other raid levels (as the base settings are not germane to a raid-type but are low-level and would apply equally).

[Speederlander]

Yeah, sort of, I created it on my unix box (dd if=/dev/zero of=32GiB bs=1M count=32768) as that's where I was at the time and then just moved it over. it doesn't really matter where you create it or what's in it, it's just a pointer to blocks that xdd will operate on. I left NCQ on for the drives I have here, if you have time it would be interesting to see what type of affect it has).

Alright, so I just want to run a test case to solve my problem with the non-existing target.

I have a folder called xdd on my G: drive (the Raid 10 drive) with the xdd bin folder and everything else in it.

g:\xdd\bin>xdd.exe -op read -targets 1 _________

I want to target that drive and both the volume and the logical drive are named RAID10. No combo I have tried gets me anything other than "The system cannot find the file specified". I have tried files in the g:\xdd\bin directory and the drive itself.

I know I'm missing something simple. I'm pretty sure it's a syntax issue.

[Dean]

When you say "newest firmware" are you talking about firmware on your RAID controller?

Awesome post, man.

Thanks,

-Dean

[Speederlander]

When you say "newest firmware" are you talking about firmware on your RAID controller?

Yes. But I am going to redo the organization. The old firmware results will go away. I just happened to be in the middle of doing updates when I started and wanted to see any difference in results between old and new.

[stevecs]

@speederlander: you just need to create a file then for it to run against. You can grab a windows version of DD at http://www.chrysocome.net/dd or use any other tool to create a file for xdd to point to. or you can use the -createnewfiles option which will take more time (it actually creates a new file per pass). So something like:

xdd.exe -verbose -op read -targets 1 S32GiB -createnewfiles -dio -blocksize 512 -reqsize 128 -mbytes 8192 -passes 3 -seek random -seek range 64000000 -queuedepth 128

[Gogeta]

Speederlander, will you be posting updates with results as the tests complete, or in one large sum at the end of testing?

[Speederlander]

Speederlander, will you be posting updates with results as the tests complete, or in one large sum at the end of testing?

Updates as I go.

[Speederlander]

@speederlander: you just need to create a file then for it to run against. You can grab a windows version of DD at http://www.chrysocome.net/dd or use any other tool to create a file for xdd to point to. or you can use the -createnewfiles option which will take more time (it actually creates a new file per pass). So something like:

xdd.exe -verbose -op read -targets 1 S32GiB -createnewfiles -dio -blocksize 512 -reqsize 128 -mbytes 8192 -passes 3 -seek random -seek range 64000000 -queuedepth 128

Alright, I obviously have lost my ability to use the command line. Hence iometer for the near-term.

Disclosure: I freely admit I am not an expert when it comes to the nitty gritty details of storage access patterns, etc.

Do you have any specific pattern(s) you would like to see on iometer?

I need some feedback as to what is most appropriate for good characterization.


How about:
IOMeter Parameters:

Maximum Disk Size: 16777216
Starting Disk Sector: 0
# of Outstanding I/Os: 1
Test Connection Rate: skip

Access Pattern:
Transfer Request Size: 4KB
Percent of Access Specification: 100%
Percent Read/Write Distribution: 67% Read/33% Write
Percent Random/Sequential Distribution: 100% Random/0% Seq
Burstiness: 0ms/1 IOs
Align I/Os on Sector Boundaries
Reply Size: No Reply


Test Set-up:
Run Time: 3 Minutes
Ramp Up: 30 seconds
Cycling Options: Cycle # of Outstanding I/Os -- run step outstanding I/Os on all disks at a time
# of Outstanding I/Os: start: 1 end: 32 power: 2, exponential stepping


TEST RUN:

[stevecs]

Alright, I obviously have lost my ability to use the command line.

blame it on new-years early partying

as for iometer I haven't used it much, but you seem to have the general gist. I would suggest that you want to check a couple request sizes, your 4K that you have is good as that's your page size with non-itanium intel chips (ie, similar to what you would have with say a pagefile access). 64K would be good for a file server (smb1) with large client base, then whatever you want to show how it performs with your normal (or modeled) request size.

I would also allow queue depth to go to 256 or at least 128. The areca will spread that out and the 1680 has a queue depth of 256 (most drives allow for at least 32 each) this will basically saturate the drive (worst case).

The read/write distribution you have (67/33) is decent for a file server, but you probably want a pure 100/0 0/100 test for comparison sake.

[Speederlander]

Test Schedule: (may throw in stripe variations as well...)

[Speederlander]

First results posted to reserved post spots at the beginning of the thread.

Next test to be RAID 0 with disk write caching off and OS caching off.

[Speederlander]

Comparison of RAID 0 results:


*removed for editing*

[stevecs]

You may want mention in your column headers (for future lookups) "RAID 0 Write Cache [On|Off]" so people will know it's not read cache or whatnot. Also just for clarification you have MBps which indicates Mega = 1,000,000 bytes. If That's what you mean great. if you actually mean 1,048,576 bytes you should use the MiBps abbreviation. I haven't had a chance to go through the data in detail but looks good so far.

[Speederlander]

You may want mention in your column headers (for future lookups) "RAID 0 Write Cache [On|Off]" so people will know it's not read cache or whatnot. Also just for clarification you have MBps which indicates Mega = 1,000,000 bytes. If That's what you mean great. if you actually mean 1,048,576 bytes you should use the MiBps abbreviation. I haven't had a chance to go through the data in detail but looks good so far.

Thanks. The MBps header is the iometer header from their output csv file.

With respect to the headers, you mean the thing I posted in post #45? That was just a quick and dirty. It's going away. The information in the first posts of the thread where I am putting all the data is sufficient, yes?

[Speederlander]

Added results for caching off and NCQ off. Interesting impact. NCQ stay on in any event for the remainder of tests.

RAID 10 next I think.

[Speederlander]

Should I be comparing 8 drive RAID 10 to 4 drive RAID 0 to see the RAID 10 scaling?