A question came up recently with a network file server disk I/O load and performances under different raid types, I didn't have anything saved previously so I ran some simple tests with their request size (this was for large files shared over a SMB v1 network, so block sizes being limited to 64KiB) to illustrate some raid points to them. Figured it may be of interest here as it shows nicely the performance of the raids w/ random workloads and how the raid levels handle them everything else being equal.
This is not intended to be an absolute (depends on raid implementation, and other items) this is mainly good though for illustration purposes to help people get their minds around concepts like data instance redundancy (ie raid-1 or 10 is has more iops than any other raid for reads), where the write penalties show for the parity raids et al.
all OS & drive write caching is disabled only the areca card has a write cache with the BBU & normal read-ahead but data size is 24GiB and card has just 512MiB to mitigate that. This is more akin to show how it would perform under proper deployment.
RAID 0:
Code:
Areca 1680ix-24 512MB Cache;
HDD Read Ahead Cache: Enabled
Volume Data Read Ahead: Normal
HDD Queue Depth: 32
Disk Write Cache Mode: Disabled
RAID 0; 8KiB Stripe Size; 8 x ST973451SS (73GB 2.5" SFF SAS 15K Savvio)
Windows XP x64 SP2, NTFS, 32GB Partition; no allignment (file system start sector 63)
=====================================================================================================
H:\xdd\bin>xdd.exe -verbose -op read -targets 1 S24GiB -dio -blocksize 512 -reqsize 128 -mbytes 8192 -passes 3 -seek random -seek ra
nge 50000000 -queuedepth 128
IOIOIOIOIOIOIOIOIOIOI XDD version 6.5.013007.0001 IOIOIOIOIOIOIOIOIOIOIOI
xdd - I/O Performance Inc. Copyright 1992-2007
Starting time for this run, Mon Dec 22 11:23:07 2008
ID for this run, 'No ID Specified'
Maximum Process Priority, disabled
Passes, 3
Pass Delay in seconds, 0
Maximum Error Threshold, 0
Target Offset, 0
I/O Synchronization, 0
Total run-time limit in seconds, 0
Output file name, stdout
CSV output file name,
Error output file name, stderr
Pass seek randomization, disabled
File write synchronization, disabled
Pass synchronization barriers, enabled
Number of Targets, 1
Number of I/O Threads, 128
Computer Name, ANALYTICAL, User Name, stcost
Operating System Info: NT 5.2 Build 3790 Service Pack 2
Page size in bytes, 4096
Number of processors on this system, 4
Megabytes of physical memory, 4095
Seconds before starting, 0
Target[0] Q[0], S24GiB
Target directory, "./"
Process ID, 2376
Thread ID, 2376
Processor, all/any
Read/write ratio, 100.00, 0.00
Throttle in MB/sec, 0.00
Per-pass time limit in seconds, 0
Blocksize in bytes, 512
Request size, 128, blocks, 65536, bytes
Number of Requests, 1024
Start offset, 0
Number of MegaBytes, 8192
Pass Offset in blocks, 0
I/O memory buffer is a normal memory buffer
I/O memory buffer alignment in bytes, 4096
Data pattern in buffer, '0x00'
Data buffer verification is disabled.
Direct I/O, enabled
Seek pattern, queued_interleaved
Seek range, 50000000
Preallocation, 0
Queue Depth, 128
Timestamping, disabled
Delete file, disabled
T Q Bytes Ops Time Rate IOPS Latency %CPU OP_Type ReqSize
TARGET PASS0001 0 128 8589934592 131072 166.256 51.667 788.38 0.0013 2.36 read 65536
TARGET PASS0002 0 128 8589934592 131072 166.541 51.578 787.02 0.0013 2.33 read 65536
TARGET PASS0003 0 128 8589934592 131072 166.186 51.689 788.71 0.0013 2.72 read 65536
TARGET Average 0 128 25769803776 393216 498.751 51.669 788.40 0.0013 2.47 read 65536
Combined 1 128 25769803776 393216 498.751 51.669 788.40 0.0013 2.43 read 65536
Ending time for this run, Mon Dec 22 11:31:28 2008
H:\xdd\bin>xdd.exe -verbose -op write -targets 1 S24GiB -dio -blocksize 512 -reqsize 128 -mbytes 8192 -passes 3 -seek random -seek r
ange 50000000 -queuedepth 128
IOIOIOIOIOIOIOIOIOIOI XDD version 6.5.013007.0001 IOIOIOIOIOIOIOIOIOIOIOI
xdd - I/O Performance Inc. Copyright 1992-2007
Starting time for this run, Mon Dec 22 11:35:06 2008
ID for this run, 'No ID Specified'
Maximum Process Priority, disabled
Passes, 3
Pass Delay in seconds, 0
Maximum Error Threshold, 0
Target Offset, 0
I/O Synchronization, 0
Total run-time limit in seconds, 0
Output file name, stdout
CSV output file name,
Error output file name, stderr
Pass seek randomization, disabled
File write synchronization, disabled
Pass synchronization barriers, enabled
Number of Targets, 1
Number of I/O Threads, 128
Computer Name, ANALYTICAL, User Name, stcost
Operating System Info: NT 5.2 Build 3790 Service Pack 2
Page size in bytes, 4096
Number of processors on this system, 4
Megabytes of physical memory, 4095
Seconds before starting, 0
Target[0] Q[0], S24GiB
Target directory, "./"
Process ID, 2464
Thread ID, 2464
Processor, all/any
Read/write ratio, 0.00, 100.00
Throttle in MB/sec, 0.00
Per-pass time limit in seconds, 0
Blocksize in bytes, 512
Request size, 128, blocks, 65536, bytes
Number of Requests, 1024
Start offset, 0
Number of MegaBytes, 8192
Pass Offset in blocks, 0
I/O memory buffer is a normal memory buffer
I/O memory buffer alignment in bytes, 4096
Data pattern in buffer, '0x00'
Data buffer verification is disabled.
Direct I/O, enabled
Seek pattern, queued_interleaved
Seek range, 50000000
Preallocation, 0
Queue Depth, 128
Timestamping, disabled
Delete file, disabled
T Q Bytes Ops Time Rate IOPS Latency %CPU OP_Type ReqSize
TARGET PASS0001 0 128 8589934592 131072 240.651 35.695 544.66 0.0018 2.10 write 65536
TARGET PASS0002 0 128 8589934592 131072 246.432 34.857 531.88 0.0019 2.27 write 65536
TARGET PASS0003 0 128 8589934592 131072 246.922 34.788 530.82 0.0019 1.97 write 65536
TARGET Average 0 128 25769803776 393216 734.004 35.109 535.71 0.0019 2.12 write 65536
Combined 1 128 25769803776 393216 734.004 35.109 535.71 0.0019 2.12 write 65536
Ending time for this run, Mon Dec 22 11:47:21 2008
RAID 1+0 (10):
Code:
Areca 1680ix-24 512MB Cache;
HDD Read Ahead Cache: Enabled
Volume Data Read Ahead: Normal
HDD Queue Depth: 32
Disk Write Cache Mode: Disabled
RAID 1+0; 8KiB Stripe Size; 8 x ST973451SS (73GB 2.5" SFF SAS 15K Savvio)
Windows XP x64 SP2, NTFS, 32GB Partition; no allignment (file system start sector 63)
=====================================================================================================
H:\xdd\bin>xdd.exe -verbose -op read -targets 1 S24GiB -dio -blocksize 512 -reqsize 128 -mbytes 8192 -passes 3 -seek random -seek
nge 50000000 -queuedepth 128
IOIOIOIOIOIOIOIOIOIOI XDD version 6.5.013007.0001 IOIOIOIOIOIOIOIOIOIOIOI
xdd - I/O Performance Inc. Copyright 1992-2007
Starting time for this run, Sun Dec 21 15:37:11 2008
ID for this run, 'No ID Specified'
Maximum Process Priority, disabled
Passes, 3
Pass Delay in seconds, 0
Maximum Error Threshold, 0
Target Offset, 0
I/O Synchronization, 0
Total run-time limit in seconds, 0
Output file name, stdout
CSV output file name,
Error output file name, stderr
Pass seek randomization, disabled
File write synchronization, disabled
Pass synchronization barriers, enabled
Number of Targets, 1
Number of I/O Threads, 128
Computer Name, ANALYTICAL, User Name, stcost
Operating System Info: NT 5.2 Build 3790 Service Pack 2
Page size in bytes, 4096
Number of processors on this system, 4
Megabytes of physical memory, 4095
Seconds before starting, 0
Target[0] Q[0], S24GiB
Target directory, "./"
Process ID, 3952
Thread ID, 3952
Processor, all/any
Read/write ratio, 100.00, 0.00
Throttle in MB/sec, 0.00
Per-pass time limit in seconds, 0
Blocksize in bytes, 512
Request size, 128, blocks, 65536, bytes
Number of Requests, 1024
Start offset, 0
Number of MegaBytes, 8192
Pass Offset in blocks, 0
I/O memory buffer is a normal memory buffer
I/O memory buffer alignment in bytes, 4096
Data pattern in buffer, '0x00'
Data buffer verification is disabled.
Direct I/O, enabled
Seek pattern, queued_interleaved
Seek range, 50000000
Preallocation, 0
Queue Depth, 128
Timestamping, disabled
Delete file, disabled
T Q Bytes Ops Time Rate IOPS Latency %CPU OP_Type ReqSize
TARGET PASS0001 0 128 8589934592 131072 100.944 85.096 1298.46 0.0008 3.46 read 65536
TARGET PASS0002 0 128 8589934592 131072 101.129 84.941 1296.09 0.0008 3.55 read 65536
TARGET PASS0003 0 128 8589934592 131072 100.580 85.404 1303.16 0.0008 3.61 read 65536
TARGET Average 0 128 25769803776 393216 302.036 85.320 1301.88 0.0008 3.54 read 65536
Combined 1 128 25769803776 393216 302.036 85.320 1301.88 0.0008 3.49 read 65536
Ending time for this run, Sun Dec 21 15:42:15 2008
H:\xdd\bin>xdd.exe -verbose -op write -targets 1 S24GiB -dio -blocksize 512 -reqsize 128 -mbytes 8192 -passes 3 -seek random -seek r
ange 50000000 -queuedepth 128
IOIOIOIOIOIOIOIOIOIOI XDD version 6.5.013007.0001 IOIOIOIOIOIOIOIOIOIOIOI
xdd - I/O Performance Inc. Copyright 1992-2007
Starting time for this run, Sun Dec 21 15:44:02 2008
ID for this run, 'No ID Specified'
Maximum Process Priority, disabled
Passes, 3
Pass Delay in seconds, 0
Maximum Error Threshold, 0
Target Offset, 0
I/O Synchronization, 0
Total run-time limit in seconds, 0
Output file name, stdout
CSV output file name,
Error output file name, stderr
Pass seek randomization, disabled
File write synchronization, disabled
Pass synchronization barriers, enabled
Number of Targets, 1
Number of I/O Threads, 128
Computer Name, ANALYTICAL, User Name, stcost
Operating System Info: NT 5.2 Build 3790 Service Pack 2
Page size in bytes, 4096
Number of processors on this system, 4
Megabytes of physical memory, 4095
Seconds before starting, 0
Target[0] Q[0], S24GiB
Target directory, "./"
Process ID, 1600
Thread ID, 1600
Processor, all/any
Read/write ratio, 0.00, 100.00
Throttle in MB/sec, 0.00
Per-pass time limit in seconds, 0
Blocksize in bytes, 512
Request size, 128, blocks, 65536, bytes
Number of Requests, 1024
Start offset, 0
Number of MegaBytes, 8192
Pass Offset in blocks, 0
I/O memory buffer is a normal memory buffer
I/O memory buffer alignment in bytes, 4096
Data pattern in buffer, '0x00'
Data buffer verification is disabled.
Direct I/O, enabled
Seek pattern, queued_interleaved
Seek range, 50000000
Preallocation, 0
Queue Depth, 128
Timestamping, disabled
Delete file, disabled
T Q Bytes Ops Time Rate IOPS Latency %CPU OP_Type ReqSize
TARGET PASS0001 0 128 8589934592 131072 267.955 32.057 489.16 0.0020 1.79 write 65536
TARGET PASS0002 0 128 8589934592 131072 269.735 31.846 485.93 0.0021 1.77 write 65536
TARGET PASS0003 0 128 8589934592 131072 268.538 31.988 488.09 0.0020 1.80 write 65536
TARGET Average 0 128 25769803776 393216 806.226 31.964 487.72 0.0021 1.79 write 65536
Combined 1 128 25769803776 393216 806.226 31.964 487.72 0.0021 1.78 write 65536
Ending time for this run, Sun Dec 21 15:57:30 2008
RAID5:
Code:
Areca 1680ix-24 512MB Cache;
HDD Read Ahead Cache: Enabled
Volume Data Read Ahead: Normal
HDD Queue Depth: 32
Disk Write Cache Mode: Disabled
RAID 5; 8KiB Stripe Size; 8 x ST973451SS (73GB 2.5" SFF SAS 15K Savvio)
Windows XP x64 SP2, NTFS, 32GB Partition; no allignment (file system start sector 63)
=====================================================================================================
H:\xdd\bin>xdd.exe -verbose -op read -targets 1 S24GiB -dio -blocksize 512 -reqsize 128 -mbytes 8192 -passes 3 -seek random -seek ra
nge 50000000 -queuedepth 128
IOIOIOIOIOIOIOIOIOIOI XDD version 6.5.013007.0001 IOIOIOIOIOIOIOIOIOIOIOI
xdd - I/O Performance Inc. Copyright 1992-2007
Starting time for this run, Sun Dec 21 22:25:06 2008
ID for this run, 'No ID Specified'
Maximum Process Priority, disabled
Passes, 3
Pass Delay in seconds, 0
Maximum Error Threshold, 0
Target Offset, 0
I/O Synchronization, 0
Total run-time limit in seconds, 0
Output file name, stdout
CSV output file name,
Error output file name, stderr
Pass seek randomization, disabled
File write synchronization, disabled
Pass synchronization barriers, enabled
Number of Targets, 1
Number of I/O Threads, 128
Computer Name, ANALYTICAL, User Name, stcost
Operating System Info: NT 5.2 Build 3790 Service Pack 2
Page size in bytes, 4096
Number of processors on this system, 4
Megabytes of physical memory, 4095
Seconds before starting, 0
Target[0] Q[0], S24GiB
Target directory, "./"
Process ID, 3892
Thread ID, 3892
Processor, all/any
Read/write ratio, 100.00, 0.00
Throttle in MB/sec, 0.00
Per-pass time limit in seconds, 0
Blocksize in bytes, 512
Request size, 128, blocks, 65536, bytes
Number of Requests, 1024
Start offset, 0
Number of MegaBytes, 8192
Pass Offset in blocks, 0
I/O memory buffer is a normal memory buffer
I/O memory buffer alignment in bytes, 4096
Data pattern in buffer, '0x00'
Data buffer verification is disabled.
Direct I/O, enabled
Seek pattern, queued_interleaved
Seek range, 50000000
Preallocation, 0
Queue Depth, 128
Timestamping, disabled
Delete file, disabled
T Q Bytes Ops Time Rate IOPS Latency %CPU OP_Type ReqSize
TARGET PASS0001 0 128 8589934592 131072 171.366 50.126 764.87 0.0013 2.41 read 65536
TARGET PASS0002 0 128 8589934592 131072 170.789 50.296 767.45 0.0013 2.49 read 65536
TARGET PASS0003 0 128 8589934592 131072 171.240 50.163 765.43 0.0013 2.43 read 65536
TARGET Average 0 128 25769803776 393216 512.806 50.253 766.79 0.0013 2.44 read 65536
Combined 1 128 25769803776 393216 513.000 50.234 766.50 0.0013 2.40 read 65536
Ending time for this run, Sun Dec 21 22:33:40 2008
H:\xdd\bin>xdd.exe -verbose -op write -targets 1 S24GiB -dio -blocksize 512 -reqsize 128 -mbytes 8192 -passes 3 -seek random -seek r
ange 50000000 -queuedepth 128
IOIOIOIOIOIOIOIOIOIOI XDD version 6.5.013007.0001 IOIOIOIOIOIOIOIOIOIOIOI
xdd - I/O Performance Inc. Copyright 1992-2007
Starting time for this run, Sun Dec 21 22:33:59 2008
ID for this run, 'No ID Specified'
Maximum Process Priority, disabled
Passes, 3
Pass Delay in seconds, 0
Maximum Error Threshold, 0
Target Offset, 0
I/O Synchronization, 0
Total run-time limit in seconds, 0
Output file name, stdout
CSV output file name,
Error output file name, stderr
Pass seek randomization, disabled
File write synchronization, disabled
Pass synchronization barriers, enabled
Number of Targets, 1
Number of I/O Threads, 128
Computer Name, ANALYTICAL, User Name, stcost
Operating System Info: NT 5.2 Build 3790 Service Pack 2
Page size in bytes, 4096
Number of processors on this system, 4
Megabytes of physical memory, 4095
Seconds before starting, 0
Target[0] Q[0], S24GiB
Target directory, "./"
Process ID, 2364
Thread ID, 2364
Processor, all/any
Read/write ratio, 0.00, 100.00
Throttle in MB/sec, 0.00
Per-pass time limit in seconds, 0
Blocksize in bytes, 512
Request size, 128, blocks, 65536, bytes
Number of Requests, 1024
Start offset, 0
Number of MegaBytes, 8192
Pass Offset in blocks, 0
I/O memory buffer is a normal memory buffer
I/O memory buffer alignment in bytes, 4096
Data pattern in buffer, '0x00'
Data buffer verification is disabled.
Direct I/O, enabled
Seek pattern, queued_interleaved
Seek range, 50000000
Preallocation, 0
Queue Depth, 128
Timestamping, disabled
Delete file, disabled
T Q Bytes Ops Time Rate IOPS Latency %CPU OP_Type ReqSize
TARGET PASS0001 0 128 8589934592 131072 572.414 15.007 228.98 0.0044 0.89 write 65536
TARGET PASS0002 0 128 8589934592 131072 578.774 14.842 226.46 0.0044 0.87 write 65536
TARGET PASS0003 0 128 8589934592 131072 576.499 14.900 227.36 0.0044 0.89 write 65536
TARGET Average 0 128 25769803776 393216 1727.685 14.916 227.60 0.0044 0.88 write 65536
Combined 1 128 25769803776 393216 1727.685 14.916 227.60 0.0044 0.88 write 65536
Ending time for this run, Sun Dec 21 23:02:49 2008
RAID 6:
Code:
Areca 1680ix-24 512MB Cache;
HDD Read Ahead Cache: Enabled
Volume Data Read Ahead: Normal
HDD Queue Depth: 32
Disk Write Cache Mode: Disabled
RAID 6; 8KiB Stripe Size; 8 x ST973451SS (73GB 2.5" SFF SAS 15K Savvio)
Windows XP x64 SP2, NTFS, 32GB Partition; no allignment (file system start sector 63)
=====================================================================================================
H:\xdd\bin>xdd.exe -verbose -op read -targets 1 S24GiB -dio -blocksize 512 -reqsize 128 -mbytes 8192 -passes 3 -seek random -seek ra
nge 50000000 -queuedepth 128
IOIOIOIOIOIOIOIOIOIOI XDD version 6.5.013007.0001 IOIOIOIOIOIOIOIOIOIOIOI
xdd - I/O Performance Inc. Copyright 1992-2007
Starting time for this run, Sun Dec 21 19:30:13 2008
ID for this run, 'No ID Specified'
Maximum Process Priority, disabled
Passes, 3
Pass Delay in seconds, 0
Maximum Error Threshold, 0
Target Offset, 0
I/O Synchronization, 0
Total run-time limit in seconds, 0
Output file name, stdout
CSV output file name,
Error output file name, stderr
Pass seek randomization, disabled
File write synchronization, disabled
Pass synchronization barriers, enabled
Number of Targets, 1
Number of I/O Threads, 128
Computer Name, ANALYTICAL, User Name, stcost
Operating System Info: NT 5.2 Build 3790 Service Pack 2
Page size in bytes, 4096
Number of processors on this system, 4
Megabytes of physical memory, 4095
Seconds before starting, 0
Target[0] Q[0], S24GiB
Target directory, "./"
Process ID, 3160
Thread ID, 3160
Processor, all/any
Read/write ratio, 100.00, 0.00
Throttle in MB/sec, 0.00
Per-pass time limit in seconds, 0
Blocksize in bytes, 512
Request size, 128, blocks, 65536, bytes
Number of Requests, 1024
Start offset, 0
Number of MegaBytes, 8192
Pass Offset in blocks, 0
I/O memory buffer is a normal memory buffer
I/O memory buffer alignment in bytes, 4096
Data pattern in buffer, '0x00'
Data buffer verification is disabled.
Direct I/O, enabled
Seek pattern, queued_interleaved
Seek range, 50000000
Preallocation, 0
Queue Depth, 128
Timestamping, disabled
Delete file, disabled
T Q Bytes Ops Time Rate IOPS Latency %CPU OP_Type ReqSize
TARGET PASS0001 0 128 8589934592 131072 173.154 49.609 756.97 0.0013 2.45 read 65536
TARGET PASS0002 0 128 8589934592 131072 171.793 50.002 762.97 0.0013 2.33 read 65536
TARGET PASS0003 0 128 8589934592 131072 174.457 49.238 751.31 0.0013 1.97 read 65536
TARGET Average 0 128 25769803776 393216 518.649 49.686 758.15 0.0013 2.25 read 65536
Combined 1 128 25769803776 393216 519.000 49.653 757.64 0.0013 2.22 read 65536
Ending time for this run, Sun Dec 21 19:38:54 2008
H:\xdd\bin>xdd.exe -verbose -op write -targets 1 S24GiB -dio -blocksize 512 -reqsize 128 -mbytes 8192 -passes 3 -seek random -seek r
ange 50000000 -queuedepth 128
IOIOIOIOIOIOIOIOIOIOI XDD version 6.5.013007.0001 IOIOIOIOIOIOIOIOIOIOIOI
xdd - I/O Performance Inc. Copyright 1992-2007
Starting time for this run, Sun Dec 21 19:45:01 2008
ID for this run, 'No ID Specified'
Maximum Process Priority, disabled
Passes, 3
Pass Delay in seconds, 0
Maximum Error Threshold, 0
Target Offset, 0
I/O Synchronization, 0
Total run-time limit in seconds, 0
Output file name, stdout
CSV output file name,
Error output file name, stderr
Pass seek randomization, disabled
File write synchronization, disabled
Pass synchronization barriers, enabled
Number of Targets, 1
Number of I/O Threads, 128
Computer Name, ANALYTICAL, User Name, stcost
Operating System Info: NT 5.2 Build 3790 Service Pack 2
Page size in bytes, 4096
Number of processors on this system, 4
Megabytes of physical memory, 4095
Seconds before starting, 0
Target[0] Q[0], S24GiB
Target directory, "./"
Process ID, 2192
Thread ID, 2192
Processor, all/any
Read/write ratio, 0.00, 100.00
Throttle in MB/sec, 0.00
Per-pass time limit in seconds, 0
Blocksize in bytes, 512
Request size, 128, blocks, 65536, bytes
Number of Requests, 1024
Start offset, 0
Number of MegaBytes, 8192
Pass Offset in blocks, 0
I/O memory buffer is a normal memory buffer
I/O memory buffer alignment in bytes, 4096
Data pattern in buffer, '0x00'
Data buffer verification is disabled.
Direct I/O, enabled
Seek pattern, queued_interleaved
Seek range, 50000000
Preallocation, 0
Queue Depth, 128
Timestamping, disabled
Delete file, disabled
T Q Bytes Ops Time Rate IOPS Latency %CPU OP_Type ReqSize
TARGET PASS0001 0 128 8589934592 131072 702.788 12.223 186.50 0.0054 0.63 write 65536
TARGET PASS0002 0 128 8589934592 131072 708.464 12.125 185.01 0.0054 0.64 write 65536
TARGET PASS0003 0 128 8589934592 131072 712.876 12.050 183.86 0.0054 0.72 write 65536
TARGET Average 0 128 25769803776 393216 2124.127 12.132 185.12 0.0054 0.66 write 65536
Combined 1 128 25769803776 393216 2124.127 12.132 185.12 0.0054 0.66 write 65536
Ending time for this run, Sun Dec 21 20:20:28 2008
Bookmarks