Some RAID Benches (5, 6, 50, and 60): Areca 1680, VRaptors, 256 bit AES, etc.

Printable View

Show 100 post(s) from this thread on one page

12-21-2008, 03:17 PM
XS Janus

Sh:@t dude!
Way to tease! :D
12-21-2008, 05:11 PM
tiro_uspsss

nice thread! :yepp: :toast: :bows: :up:
12-22-2008, 09:18 PM
Gogeta

Thank you for the time its going to take to run all of these benches and compile the results.

With regard to I/O testing, how does bonnie++ compare? The OP probably won't run Linux for his testing, but I think its a valid question while he enjoys some vacation time.
12-23-2008, 07:28 AM
stevecs

Bonnie++ is a fast test (ie, akin to hdtune; atto; crystaldiskmark; et al. for windows that are shown here often) not something that will provide you real good data for comparison. I use it mainly as a first step to get a 50,000 foot view and then start running real tests tailored toward what I am trying to bench. If you are running it yourself, normally what you want to do is run it 5-9 times, remove the best & worst run and average the rest to help remove some of the fluctuations.
12-30-2008, 12:38 AM
Speederlander

Quote:

Originally Posted by stevecs

Yes, you get a pre-compiled binary that you can run from the command line. for example here's a run I just did against my 2nd raw partition here (raid-10/8 savvios in sig/ 8K stripe size on areca) this is only a read as I have data on that array that I couldn't move around.

(BTW/ to do it against a physical disk you need to do this in windows:

D:\Temp\xdd\bin>xdd.exe -verbose -op read -targets 1 \\.\\PhysicalDrive2 -dio -blocksize 512 -reqsize 128 -mbytes 8192 -passes 3 -se
ek random -seek range 64000000 -queuedepth 1

The physical disk number I just got from everest (subtract 1 from list as it really starts at 0)though I just noticed that when using physical discs in windows (XP X64 at least) you can't use a queuedepth > 1 so below I used files (2GB each) in the ntfs filesystem (NON-alligned as I'm using 8K stripe size w/ 8 drives in raid-10 (32K/(8drives/2)) filesystem starts at sector 63.) Will be fixing this shortly once I get some sleep. ;)

Code:

D:\Temp\xdd\bin>xdd.exe -verbose -op read -targets 1 S32GiB -dio -blocksize 512 -reqsize 128 -mbytes 8192 -passes 3 -seek rand ek range 64000000 -queuedepth 128 IOIOIOIOIOIOIOIOIOIOI XDD version 6.5.013007.0001 IOIOIOIOIOIOIOIOIOIOIOI xdd - I/O Performance Inc. Copyright 1992-2007 Starting time for this run, Sat Dec 20 21:43:04 2008 ID for this run, 'No ID Specified' Maximum Process Priority, disabled Passes, 3 Pass Delay in seconds, 0 Maximum Error Threshold, 0 Target Offset, 0 I/O Synchronization, 0 Total run-time limit in seconds, 0 Output file name, stdout CSV output file name, Error output file name, stderr Pass seek randomization, disabled File write synchronization, disabled Pass synchronization barriers, enabled Number of Targets, 1 Number of I/O Threads, 128 Computer Name, ANALYTICAL, User Name, stcost Operating System Info: NT 5.2 Build 3790 Service Pack 2 Page size in bytes, 4096 Number of processors on this system, 4 Megabytes of physical memory, 4095 Seconds before starting, 0 Target[0] Q[0], S32GiB Target directory, "./" Process ID, 4512 Thread ID, 4512 Processor, all/any Read/write ratio, 100.00, 0.00 Throttle in MB/sec, 0.00 Per-pass time limit in seconds, 0 Blocksize in bytes, 512 Request size, 128, blocks, 65536, bytes Number of Requests, 1024 Start offset, 0 Number of MegaBytes, 8192 Pass Offset in blocks, 0 I/O memory buffer is a normal memory buffer I/O memory buffer alignment in bytes, 4096 Data pattern in buffer, '0x00' Data buffer verification is disabled. Direct I/O, enabled Seek pattern, queued_interleaved Seek range, 64000000 Preallocation, 0 Queue Depth, 128 Timestamping, disabled Delete file, disabled T Q Bytes Ops Time Rate IOPS Latency %CPU OP_Type ReqSize TARGET PASS0001 0 128 8589934592 131072 109.014 78.797 1202.35 0.0008 3.86 read 65536 TARGET PASS0002 0 128 8589934592 131072 109.936 78.136 1192.26 0.0008 3.58 read 65536 TARGET PASS0003 0 128 8589934592 131072 109.674 78.322 1195.10 0.0008 3.84 read 65536 TARGET Average 0 128 25769803776 393216 327.624 78.657 1200.21 0.0008 3.76 read 65536 Combined 1 128 25769803776 393216 328.000 78.566 1198.83 0.0008 3.69 read 65536 Ending time for this run, Sat Dec 20 21:48:34 2008

OK, so I'm trying to target my G: RAID10 drive and I can't seem to get the syntax correct. What was S32GiB in your example?
12-30-2008, 04:36 AM
stevecs

That was a file named "32GiB" (which was 32GiB in size) that I created on that partition. The syntax should be the same or is it blowing up as you don't have a file created there for it to use?
12-30-2008, 09:53 AM
Speederlander

Quote:

Originally Posted by stevecs

That was a file named "32GiB" (which was 32GiB in size) that I created on that partition. The syntax should be the same or is it blowing up as you don't have a file created there for it to use?

Yeah, didn't have the file. It was odd as it seemed to create the file the very first time I ran it but it's been failing since then. Maybe I accidently had a file of that name the first time from something else... Do you create the file ahead of time then on the target? Simple text file?

NCQ on, off or post results for both is time permits?
12-30-2008, 10:02 AM
stevecs

Yeah, sort of, I created it on my unix box (dd if=/dev/zero of=32GiB bs=1M count=32768) as that's where I was at the time and then just moved it over. it doesn't really matter where you create it or what's in it, it's just a pointer to blocks that xdd will operate on. I left NCQ on for the drives I have here, if you have time it would be interesting to see what type of affect it has).
12-30-2008, 10:05 AM
Speederlander

Quote:

Originally Posted by stevecs

Yeah, sort of, I created it on my unix box (dd if=/dev/zero of=32GiB bs=1M count=32768) as that's where I was at the time and then just moved it over. it doesn't really matter where you create it or what's in it, it's just a pointer to blocks that xdd will operate on. I left NCQ on for the drives I have here, if you have time it would be interesting to see what type of affect it has).

Here are the variables I am looking at testing against to see the impact. I obviously can't vary all of them against each other without taking 2 months to complete testing. So I need to choose a few to set constant and not mess with. Any suggestions?

Areca Settings:
Sata NCQ Support: on/off
HDD Read Ahead Cache: enabled/disabled
Volume Data Read Ahead: disabled/conservative/normal/aggresive
HDD Queue Depth: 1/2/4/8/16/32
Disk Write Cache Mode: Auto/Enabled/Disabled
Volume Cache Mode: Write Through/Write Back
Volume Stripe Size: 32/64/128

OS (Vista 64) Settings:
Enable Write Caching On Disk: on/off
Enable Advanced Performance: on/off

Also:
Can try with BBU plugged in or unplugged (for 512meg cache)
Can compare 512meg cache results to 2GB (2GB only with BBU plugged in)
12-30-2008, 10:53 AM
stevecs

Looks like a good overview for just base settings and their affects. stripe sizes, file system alignments et al will as you mentioned take quite a bit longer to plot out (I know, it takes about 4-6 months to do a full gamut test when I do them on arrays here).

The BBU itself won't provide anything different except for data integrity which you are not testing. You can change the cache sizes and then manually set your "Disk Write Cache Mode". Auto means (turn on write cache if BBU is present". So you can manually just turn it on or off on the card for testing purposes.

You probably want to reboot when setting these items just to avoid any issues w/ firmware not applying settings immediately. Volume stripe sizes are going to be a killer, you probably want to either save those for another test or at least last unless you are planning on limiting it to only a specific raid type (to get real value you will want to do all permutations which is what takes a lot of time). I would probably approach it in two modes, first with a raid-0 and then use that to test out all combinations of the areca settings (with OS settings off then on for each test) with the exception of stripe size (use the default of 64k). Then do the same with a pass-through disk (single disk but attached to the areca) this would be a baseline and show comparison to the multiplication affects of striping. After that then do tests with other raid levels (as the base settings are not germane to a raid-type but are low-level and would apply equally).
12-30-2008, 12:52 PM
Speederlander

Quote:

Originally Posted by stevecs

Yeah, sort of, I created it on my unix box (dd if=/dev/zero of=32GiB bs=1M count=32768) as that's where I was at the time and then just moved it over. it doesn't really matter where you create it or what's in it, it's just a pointer to blocks that xdd will operate on. I left NCQ on for the drives I have here, if you have time it would be interesting to see what type of affect it has).

Alright, so I just want to run a test case to solve my problem with the non-existing target.

I have a folder called xdd on my G: drive (the Raid 10 drive) with the xdd bin folder and everything else in it.

g:\xdd\bin>xdd.exe -op read -targets 1 _________

I want to target that drive and both the volume and the logical drive are named RAID10. No combo I have tried gets me anything other than "The system cannot find the file specified". I have tried files in the g:\xdd\bin directory and the drive itself.

I know I'm missing something simple. I'm pretty sure it's a syntax issue.
12-30-2008, 01:00 PM
Dean

When you say "newest firmware" are you talking about firmware on your RAID controller?

Awesome post, man. :)

Thanks,

-Dean
12-30-2008, 01:06 PM
Speederlander

Quote:

Originally Posted by Dean

When you say "newest firmware" are you talking about firmware on your RAID controller?

Yes. But I am going to redo the organization. The old firmware results will go away. I just happened to be in the middle of doing updates when I started and wanted to see any difference in results between old and new.
12-30-2008, 02:32 PM
stevecs

@speederlander: you just need to create a file then for it to run against. You can grab a windows version of DD at http://www.chrysocome.net/dd or use any other tool to create a file for xdd to point to. or you can use the -createnewfiles option which will take more time (it actually creates a new file per pass). So something like:

xdd.exe -verbose -op read -targets 1 S32GiB -createnewfiles -dio -blocksize 512 -reqsize 128 -mbytes 8192 -passes 3 -seek random -seek range 64000000 -queuedepth 128
12-31-2008, 09:51 AM
Gogeta

Speederlander, will you be posting updates with results as the tests complete, or in one large sum at the end of testing?
12-31-2008, 10:01 AM
Speederlander

Quote:

Originally Posted by Gogeta

Speederlander, will you be posting updates with results as the tests complete, or in one large sum at the end of testing?

Updates as I go.
12-31-2008, 10:10 AM
Speederlander

1 Attachment(s)

Quote:

Originally Posted by stevecs

@speederlander: you just need to create a file then for it to run against. You can grab a windows version of DD at http://www.chrysocome.net/dd or use any other tool to create a file for xdd to point to. or you can use the -createnewfiles option which will take more time (it actually creates a new file per pass). So something like:

xdd.exe -verbose -op read -targets 1 S32GiB -createnewfiles -dio -blocksize 512 -reqsize 128 -mbytes 8192 -passes 3 -seek random -seek range 64000000 -queuedepth 128

Alright, I obviously have lost my ability to use the command line. Hence iometer for the near-term. :p:

Disclosure: I freely admit I am not an expert when it comes to the nitty gritty details of storage access patterns, etc.

Do you have any specific pattern(s) you would like to see on iometer?

I need some feedback as to what is most appropriate for good characterization.

How about:
IOMeter Parameters:

Maximum Disk Size: 16777216
Starting Disk Sector: 0
# of Outstanding I/Os: 1
Test Connection Rate: skip

Access Pattern:
Transfer Request Size: 4KB
Percent of Access Specification: 100%
Percent Read/Write Distribution: 67% Read/33% Write
Percent Random/Sequential Distribution: 100% Random/0% Seq
Burstiness: 0ms/1 IOs
Align I/Os on Sector Boundaries
Reply Size: No Reply

Test Set-up:
Run Time: 3 Minutes
Ramp Up: 30 seconds
Cycling Options: Cycle # of Outstanding I/Os -- run step outstanding I/Os on all disks at a time
# of Outstanding I/Os: start: 1 end: 32 power: 2, exponential stepping

TEST RUN:
12-31-2008, 07:15 PM
stevecs

Quote:

Alright, I obviously have lost my ability to use the command line.

blame it on new-years early partying :)

as for iometer I haven't used it much, but you seem to have the general gist. I would suggest that you want to check a couple request sizes, your 4K that you have is good as that's your page size with non-itanium intel chips (ie, similar to what you would have with say a pagefile access). 64K would be good for a file server (smb1) with large client base, then whatever you want to show how it performs with your normal (or modeled) request size.

I would also allow queue depth to go to 256 or at least 128. The areca will spread that out and the 1680 has a queue depth of 256 (most drives allow for at least 32 each) this will basically saturate the drive (worst case).

The read/write distribution you have (67/33) is decent for a file server, but you probably want a pure 100/0 0/100 test for comparison sake.
12-31-2008, 09:29 PM
Speederlander

2 Attachment(s)

Test Schedule: (may throw in stripe variations as well...)
01-01-2009, 04:33 PM
Speederlander

First results posted to reserved post spots at the beginning of the thread.

Next test to be RAID 0 with disk write caching off and OS caching off.
01-02-2009, 01:02 AM
Speederlander

Comparison of RAID 0 results:

*removed for editing*
01-02-2009, 05:21 AM
stevecs

You may want mention in your column headers (for future lookups) "RAID 0 Write Cache [On|Off]" so people will know it's not read cache or whatnot. Also just for clarification you have MBps which indicates Mega = 1,000,000 bytes. If That's what you mean great. if you actually mean 1,048,576 bytes you should use the MiBps abbreviation. I haven't had a chance to go through the data in detail but looks good so far. :)
01-02-2009, 09:08 AM
Speederlander

Quote:

Originally Posted by stevecs

You may want mention in your column headers (for future lookups) "RAID 0 Write Cache [On|Off]" so people will know it's not read cache or whatnot. Also just for clarification you have MBps which indicates Mega = 1,000,000 bytes. If That's what you mean great. if you actually mean 1,048,576 bytes you should use the MiBps abbreviation. I haven't had a chance to go through the data in detail but looks good so far. :)

Thanks. The MBps header is the iometer header from their output csv file.

With respect to the headers, you mean the thing I posted in post #45? That was just a quick and dirty. It's going away. The information in the first posts of the thread where I am putting all the data is sufficient, yes?
01-02-2009, 10:02 AM
Speederlander

Added results for caching off and NCQ off. Interesting impact. NCQ stay on in any event for the remainder of tests.

RAID 10 next I think.
01-02-2009, 07:10 PM
Speederlander

Should I be comparing 8 drive RAID 10 to 4 drive RAID 0 to see the RAID 10 scaling?

Show 100 post(s) from this thread on one page