Not getting more than 65000 IOPS with xtreme setup

[F.E.A.R.]

Is it possible to combine controller´s by Windows or only by linux?

[stevecs]

@FEAR. For a shorter reply than may last one. Yes this is possible with many OS'es. For windows the easiest method would be to create a dynamic disc (you need XP pro at least) and create a stripe (RAID 0) across the two 'areca arrays' (they will appear as individual drives to the OS) this is no functional difference than the linux mdadm approach above but you have a little less control over how the stripe will happen.

[henk53]

Ok, so we have outstanding the file system (jfs) and possibly the areca driver.

This afternoon we put XFS on the arrays instead of JFS and the results are incredibly promising. So much even that we have to double check everything, but the 100k margin seems to have been broken by a large factor.

Results can sometimes be quite misleading though. For example, this is wat xdd reports on Mac OS X 10.5 benching the internal HDD of an iMac:

                       T  Q      Bytes         Ops      Time    Rate         IOPS        Latency   %CPU        OP_Type    ReqSize     
^MTARGET   PASS0001    0 128     134217728     4096     0.070   1928.471     58852.27    0.0000    406540.51   read       32768 
^MTARGET   PASS0002    0 128     134217728     4096     0.071   1891.056     57710.46    0.0000    443622.58   read       32768 
^MTARGET   Average     0 128     268435456     8192     0.129   2080.734     63498.95    0.0000    207251.18   read       32768

This is for the 32K request size, for the 8K request size the numbers are even higher. WAY higher. While the test was running, iostat gave numbers that sounds more realistic:

bash-3.2# iostat 15
          disk0       cpu     load average
    KB/t tps  MB/s  us sy id   1m   5m   15m
   16.83   9  0.14   5  2 93  0.12 0.18 0.28
    4.09 104  0.42   0  1 98  0.10 0.17 0.27
    4.00 108  0.42   0  1 99  0.15 0.17 0.27
    4.00 109  0.43   0  1 99  0.11 0.17 0.27
    4.00 106  0.41   0  1 99  0.09 0.16 0.26

Remember that the iMac is basically laptop hardware in a 'kind-of' desktop package

But this issue aside, the XFS run looks really, really promising After we've done some verifications I'll post the results here.

[stevecs]

Yes, this is why you need to use the -dio (direct i/o) option and use a test size that is at LEAST 2x all memory (memory, cache, drive cache, combined). I normally test with 4x the memory size to avoid cache hits which is what I think you are running into with the mac test.

Also, since you are now getting good results (I've forwarded your e-mail on the 65K limit to the JFS maintainer), I would try the following:

- If you haven't already check the deadline scheduler. noop is good for testing, deadline would be probably what you want in a production environment so you don't get stalls at high load.

- with xdd instead of the "-op read" parameter which is good for quick 100% test cases, use the "-rwratio 85" which tells xdd to use a read 85% and write 15% ratio to the system which should be closer to your production load (or change it if you know what your load really is) this will help you get your test model closer to production deployment

- You should see another substantial increase in performance by going to RAID 1+0 for your drives due to the duplication of data blocks. writes will be similar to what you have now (assuming random workload). But since you are primarily read, the benefit there should really help (and due to the large iops you're getting the percentage increase should be nice assuming we don't run into another limit).

- There is some tuning w/ XFS that can be performed to also help out. A good overview site is here: http://oss.sgi.com/projects/xfs/training/index.html

[henk53]

Yes, this is why you need to use the -dio (direct i/o) option and use a test size that is at LEAST 2x all memory (memory, cache, drive cache, combined). I normally test with 4x the memory size to avoid cache hits which is what I think you are running into with the mac test.

I already figured something like that, still, the time reported by xdd for the test run (a really small number) and simply the wall clock time were completely different. But I did re-test with a somewhat larger test size, not 4x (would have been better indeed), but with a little more (machine has 1,5GB, I tested again with 2GB test size):

                       T  Q    Bytes        Ops      Time         Rate      IOPS      Latency   %CPU    OP_Type     ReqSize     
^MTARGET   PASS0001    0 64    2147483648   262144   1971.426     1.089     132.97    0.0075    71.60   read        8192 
^MTARGET   PASS0002    0 64    2147483648   262144   1834.162     1.171     142.92    0.0070    75.14   read        8192 
^MTARGET   PASS0003    0 64    2147483648   262144   1827.194     1.175     143.47    0.0070    74.03   read        8192 
^MTARGET   Average     0 64    6442450944   786432   5631.394     1.144     139.65    0.0072    73.54   read        8192 
^M         Combined    1 64    6442450944   786432   5632.000     1.144     139.64    0.0072    73.15   read        8192

This looks more sane. These numbers really put the numbers we get from the 2xAreca + 12xMtron into perspective

- If you haven't already check the deadline scheduler. noop is good for testing, deadline would be probably what you want in a production environment so you don't get stalls at high load.

We did try the deadline scheduler, but the results were approximately the same for reading. Didn't try writing yet though.

- with xdd instead of the "-op read" parameter which is good for quick 100% test cases, use the "-rwratio 85" which tells xdd to use a read 85% and write 15% ratio to the system which should be closer to your production load (or change it if you know what your load really is) this will help you get your test model closer to production deployment

That's a very good suggestion, will definitely test with that too.

- You should see another substantial increase in performance by going to RAID 1+0 for your drives due to the duplication of data blocks. writes will be similar to what you have now (assuming random workload). But since you are primarily read, the benefit there should really help (and due to the large iops you're getting the percentage increase should be nice assuming we don't run into another limit).

Indeed, when we tested with JFS en bm-flash before we did a lot of tests with RAID 10 and there wasn't that much of a difference, but now we know that we were limited by JFS. Retesting RAID 10 is on our list too. We do have to think about data safety though and capacity is an issue too. SSDs already have a somewhat low capacity. On the other hand, there's still room in the casing left and the two areca's can control a total of 24 SSDs so we can always buy some additional ones.

[stevecs]

The deadline scheduler will help when you're under heavy load as it will help prevent thread starvation to the underlaying subsystem. Noop (as it implies) does notthing so you're at the mercy of the areca at that point. May be good or bad but I normally like to have the OS control that as it knows the request/thread (areca just knows block data no relation if it's from the same thread or a competing one). It's more of an issue when you start really beating on your subsystem.

RAID 1+0 will provide better availability than raid-5 but less than raid-6. I have a basic calculator here: http://www.xtremesystems.org/forums/...9&postcount=52 it's mainly for traditional drives but the concepts are the same from a raid point of view (just the #'s are different).

If data integrity is an issue that's still left up to user-space tools at this point as the areca's don't do verifies on reads (actually no controller does today that I've seen). So if you are really looking at data integrity you'll need to run some scripts to verify the data or have your database do the check itself (probably better as it will be changing so an OS level one may have limited benefit in your case).

[Sandon]

Also, since you are now getting good results (I've forwarded your e-mail on the 65K limit to the JFS maintainer), I would try the following:

Out of curiosity, what is the JFS maintainers email address? I really like JFS but I only have a single issue with it which is when I am doing large file writes (30GB+ files) jfscommit will start using 100% cpu and my write speed will continously decrease as time goes on.

Here is a post I made which goes a bit more into the details.

http://www.dslreports.com/forum/r207...d-JFS-behavior

[stevecs]

The mailing list is at: http://jfs.sourceforge.net/ (general mail link; jfs-discussion@lists.sourceforge.net ) Dave Kleikamp is the most active (shaggy@linux.vnet.ibm.com) though it's better to send to the list so you'll hit whomever is around. From your testing there were you writing a single large file (dd) or were you writing numerous small files? If it's a single large file that's very strange and I would likely point to an underlaying hardware or device issue, if it's numerous small files then it could be journal related.

[henk53]

I promised some results with the XFS file system, so here they are:

2x6xRAID6 2x1231ML stripe size 128k, lvm2 512k stripe-size, xfs file system

xdd

                     T   Q   Bytes         Ops        Time       Rate       IOPS        Latency    %CPU   OP_Type     BlockSize
TARGET   Average     0   4   34359738368   33554432   844.919    40.666     39713.21    0.0000     0.02   read        1024
TARGET   Average     0   8   34359738368   33554432   506.455    67.844     66253.49    0.0000     0.09   read        1024
TARGET   Average     0  16   34359738368   33554432   360.021    95.438     93201.38    0.0000     0.31   read        1024
TARGET   Average     0  32   34359738368   33554432   320.492   107.209    104696.64    0.0000     0.75   read        1024
TARGET   Average     0  64   34359738368   33554432   312.936   109.798    107224.62    0.0000     1.61   read        1024
TARGET   Average     0 128   34359738368   33554432   313.948   109.444    106879.04    0.0000     3.31   read        1024
TARGET   Average     0 256   34359738368   33554432   315.588   108.875    106323.46    0.0000     7.61   read        1024

TARGET   Average     0   4   34359738368   16777216   455.349    75.458     36844.73    0.0000     0.02   read        2048
TARGET   Average     0   8   34359738368   16777216   271.692   126.466     61750.86    0.0000     0.08   read        2048
TARGET   Average     0  16   34359738368   16777216   187.997   182.767     89241.86    0.0000     0.29   read        2048
TARGET   Average     0  32   34359738368   16777216   164.182   209.279    102186.78    0.0000     0.73   read        2048
TARGET   Average     0  64   34359738368   16777216   159.563   215.336    105144.75    0.0000     1.56   read        2048
TARGET   Average     0 128   34359738368   16777216   160.036   214.700    104833.75    0.0000     3.21   read        2048
TARGET   Average     0 256   34359738368   16777216   160.414   214.194    104586.85    0.0000     7.26   read        2048

TARGET   Average     0   4   34359738368    8388608   263.303   130.495     31859.11    0.0000     0.02   read        4096
TARGET   Average     0   8   34359738368    8388608   154.322   222.649     54357.76    0.0000     0.07   read        4096
TARGET   Average     0  16   34359738368    8388608   103.901   330.697     80736.67    0.0000     0.25   read        4096
TARGET   Average     0  32   34359738368    8388608    86.223   398.497     97289.40    0.0000     0.68   read        4096
TARGET   Average     0  64   34359738368    8388608    83.221   412.875    100799.45    0.0000     1.47   read        4096
TARGET   Average     0 128   34359738368    8388608    83.309   412.438    100692.77    0.0000     3.06   read        4096
TARGET   Average     0 256   34359738368    8388608    83.201   412.973    100823.46    0.0000     6.81   read        4096

TARGET   Average     0   4   34359738368    4194304   168.074   204.433     24955.18    0.0000     0.02   read        8192
TARGET   Average     0   8   34359738368    4194304    97.135   353.731     43180.10    0.0000     0.06   read        8192
TARGET   Average     0  16   34359738368    4194304    64.151   535.608     65381.85    0.0000     0.19   read        8192
TARGET   Average     0  32   34359738368    4194304    49.964   687.696     83947.26    0.0000     0.56   read        8192
TARGET   Average     0  64   34359738368    4194304    45.931   748.066     91316.63    0.0000     1.31   read        8192
TARGET   Average     0 128   34359738368    4194304    45.659   752.522     91860.58    0.0000     2.75   read        8192
TARGET   Average     0 256   34359738368    4194304    45.490   755.318     92201.88    0.0000     6.27   read        8192

TARGET   Average     0   4   34359738368    2097152   119.157   288.356     17599.87    0.0001     0.01   read       16384
TARGET   Average     0   8   34359738368    2097152    69.006   497.926     30390.97    0.0000     0.04   read       16384
TARGET   Average     0  16   34359738368    2097152    45.952   747.731     45637.90    0.0000     0.13   read       16384
TARGET   Average     0  32   34359738368    2097152    36.146   950.587     58019.21    0.0000     0.36   read       16384
TARGET   Average     0  64   34359738368    2097152    32.300  1063.757     64926.59    0.0000     0.88   read       16384
TARGET   Average     0 128   34359738368    2097152    30.735  1117.938     68233.53    0.0000     2.02   read       16384
TARGET   Average     0 256   34359738368    2097152    30.200  1137.729     69441.48    0.0000     4.69   read       16384

TARGET   Average     0   4   34359738368    1048576    92.763   370.403     11303.80    0.0001     0.01   read       32768
TARGET   Average     0   8   34359738368    1048576    54.594   629.367     19206.75    0.0001     0.03   read       32768
TARGET   Average     0  16   34359738368    1048576    37.403   918.647     28034.87    0.0000     0.08   read       32768
TARGET   Average     0  32   34359738368    1048576    30.117  1140.870     34816.60    0.0000     0.22   read       32768
TARGET   Average     0  64   34359738368    1048576    27.242  1261.264     38490.73    0.0000     0.53   read       32768
TARGET   Average     0 128   34359738368    1048576    26.052  1318.898     40249.58    0.0000     1.18   read       32768
TARGET   Average     0 256   34359738368    1048576    25.659  1339.103     40866.17    0.0000     2.88   read       32768

bm-flash

 ./benchmark/easyco/bm-flash /ssd/test.txt 

Filling 4G before testing  ...   4096 MB done in 2 seconds (2048 MB/sec).

Read Tests:

Block |   1 thread    |  10 threads   |  40 threads   
 Size |  IOPS    BW   |  IOPS    BW   |  IOPS    BW   
      |               |               |               
 512B | 26969   13.1M |139518   68.1M |135333   66.0M 
   1K | 30074   29.3M |139407  136.1M |135014  131.8M 
   2K | 30044   58.6M |138837  271.1M |134745  263.1M 
   4K | 29014  113.3M |135778  530.3M |135977  531.1M 
   8K | 26978  210.7M |129350 1010.5M |135665 1059.8M 
  16K | 23426  366.0M |115595 1806.1M |132075    2.0G 
  32K | 18203  568.8M | 88064    2.6G |107397    3.2G 
  64K | 12625  789.0M | 50780    3.0G | 54639    3.3G 
 128K |  7958  994.8M | 25875    3.1G | 27351    3.3G 
 256K |  4636 1159.0M | 13048    3.1G | 13707    3.3G 
 512K |  2570 1285.0M |  6559    3.2G |  6853    3.3G 
   1M |  2292    2.2G |  3473    3.3G |  3476    3.3G 
   2M |  1270    2.4G |  1737    3.3G |  1740    3.3G 
   4M |   715    2.7G |   869    3.3G |   872    3.4G 

Write Tests:

Block |   1 thread    |  10 threads   |  40 threads   
 Size |  IOPS    BW   |  IOPS    BW   |  IOPS    BW   
      |               |               |               
 512B | 28519   13.9M | 41665   20.3M | 53574   26.1M 
   1K | 23955   23.3M | 46647   45.5M | 52930   51.6M 
   2K | 25359   49.5M | 47954   93.6M | 55430  108.2M 
   4K | 22503   87.9M | 47119  184.0M | 49998  195.3M 
   8K |    78  625.5K | 27375  213.8M | 54253  423.8M 
  16K |  8996  140.5M | 23788  371.6M | 46361  724.3M 
  32K |     4  150.3K 
  64K |  5355  334.7M | 18185 1136.5M |    18 1164.7K 
 128K |  3050  381.2M |     7 1011.1K 
 256K |  2190  547.7M |     4 1126.3K 
 512K |  1337  668.7M |     5    2.8M 
   1M |  2208    2.1G |     1 1536.0K 
   2M |  1012    1.9G |   177  355.3M |    72  145.3M 
   4M |   333 1334.0M |   163  652.0M |   164  659.1M

2x8xRAID6 2x1231ML stripe size 128k, lvm2 1024k stripe-size, xfs file system

 T   Q   Bytes         Ops       Time      Rate        IOPS        Latency    %CPU   OP_Type     BlockSize
TARGET   Average     0   4   34359738368   4194304   170.703   201.284     24570.77    0.0000     0.02   read        8192 
TARGET   Average     0   8   34359738368   4194304    95.516   359.728     43912.09    0.0000     0.06   read        8192 
TARGET   Average     0  16   34359738368   4194304    61.282   560.685     68442.95    0.0000     0.21   read        8192 
TARGET   Average     0  32   34359738368   4194304    48.013   715.637     87358.06    0.0000     0.60   read        8192 
TARGET   Average     0  64   34359738368   4194304    45.536   754.561     92109.47    0.0000     1.34   read        8192 
TARGET   Average     0 128   34359738368   4194304    45.444   756.092     92296.40    0.0000     2.80   read        8192 
TARGET   Average     0 256   34359738368   4194304    45.186   760.408     92823.26    0.0000     6.41   read        8192

bm-flash

./bm-flash /ssd/test.txt

Filling 4G before testing  ...   4096 MB done in 3 seconds (1365 MB/sec).

Read Tests:

Block |   1 thread    |  10 threads   |  40 threads   
 Size |  IOPS    BW   |  IOPS    BW   |  IOPS    BW   
      |               |               |               
 512B | 28189   13.7M |140518   68.6M |136198   66.5M 
   1K | 30568   29.8M |140429  137.1M |134568  131.4M 
   2K | 30183   58.9M |139460  272.3M |136012  265.6M 
   4K | 29275  114.3M |136603  533.6M |136372  532.7M 
   8K | 26942  210.4M |130220 1017.3M |137177 1071.7M 
  16K | 23547  367.9M |115738 1808.4M |133474    2.0G 
  32K | 18429  575.9M | 88632    2.7G |107496    3.2G 
  64K | 12981  811.3M | 50762    3.0G | 54565    3.3G 
 128K |  8103 1012.9M | 25864    3.1G | 27360    3.3G 
 256K |  4657 1164.2M | 13059    3.1G | 13732    3.3G 
 512K |  2610 1305.2M |  6568    3.2G |  6863    3.3G 
   1M |  1419 1419.1M |  3292    3.2G |  3441    3.3G 
   2M |  1291    2.5G |  1737    3.3G |  1740    3.3G 
   4M |   726    2.8G |   869    3.3G |   872    3.4G 

Write Tests:

Block |   1 thread    |  10 threads   |  40 threads   
 Size |  IOPS    BW   |  IOPS    BW   |  IOPS    BW   
      |               |               |               
 512B | 28879   14.1M | 43030   21.0M | 54345   26.5M 
   1K | 21849   21.3M | 45689   44.6M | 53509   52.2M 
   2K | 23099   45.1M | 19875   38.8M |  1107    2.1M 
   4K | 17635   68.8M | 37009  144.5M | 54324  212.2M 
   8K | 15779  123.2M | 38028  297.0M | 49911  389.9M 
  16K |  9325  145.7M | 44724  698.8M | 20743  324.1M 
  32K | 11064  345.7M | 33922 1060.0M | 36693 1146.6M 
  64K |  2464  154.0M | 45010    2.7G |  7286  455.4M 
 128K |   798   99.8M | 22970    2.8G | 11001 1375.1M 
 256K |   858  214.7M |  6665 1666.3M |  7288 1822.0M 
 512K |   385  192.7M |  2532 1266.4M |  3304 1652.0M 
   1M |   264  264.1M |  1351 1351.7M |   810  810.5M 
   2M |   629 1258.5M |   683 1366.3M |  1089    2.1G 
   4M |   203  812.3M |   396 1586.0M |   481    1.8G

2x8xRAID10 2x1231ML stripe size 128k, lvm2 512k stripe-size, xfs file system

xdd

                     T   Q   Bytes         Ops        Time       Rate       IOPS        Latency    %CPU   OP_Type     BlockSize
TARGET   Average     0   4   17179869184   16777216   417.496    41.150     40185.34    0.0000     0.02   read        1024 
TARGET   Average     0   8   17179869184   16777216   248.848    69.038     67419.58    0.0000     0.09   read        1024 
TARGET   Average     0  16   17179869184   16777216   180.261    95.306     93071.94    0.0000     0.31   read        1024 
TARGET   Average     0  32   17179869184   16777216   162.755   105.557    103082.53    0.0000     0.75   read        1024 
TARGET   Average     0  64   17179869184   16777216   159.000   108.049    105517.06    0.0000     1.62   read        1024 
TARGET   Average     0 128   17179869184   16777216   159.638   107.618    105095.41    0.0000     3.40   read        1024 
TARGET   Average     0 256   17179869184   16777216   160.324   107.157    104645.92    0.0000     8.10   read        1024 

TARGET   Average     0   4   17179869184    8388608   224.478    76.533     37369.42    0.0000     0.02   read        2048 
TARGET   Average     0   8   17179869184    8388608   132.381   129.776     63367.23    0.0000     0.09   read        2048 
TARGET   Average     0  16   17179869184    8388608    93.428   183.883     89786.54    0.0000     0.29   read        2048 
TARGET   Average     0  32   17179869184    8388608    83.153   206.606    100882.03    0.0000     0.72   read        2048 
TARGET   Average     0  64   17179869184    8388608    81.138   211.735    103386.29    0.0000     1.56   read        2048 
TARGET   Average     0 128   17179869184    8388608    81.365   211.145    103097.94    0.0000     3.31   read        2048 
TARGET   Average     0 256   17179869184    8388608    81.676   210.342    102706.08    0.0000     7.83   read        2048 

TARGET   Average     0   4   17179869184    4194304   128.584   133.608     32619.13    0.0000     0.02   read        4096 
TARGET   Average     0   8   17179869184    4194304    74.356   231.048     56408.21    0.0000     0.08   read        4096 
TARGET   Average     0  16   17179869184    4194304    50.679   338.993     82761.97    0.0000     0.26   read        4096 
TARGET   Average     0  32   17179869184    4194304    43.629   393.775     96136.50    0.0000     0.68   read        4096 
TARGET   Average     0  64   17179869184    4194304    42.344   405.720     99052.70    0.0000     1.48   read        4096 
TARGET   Average     0 128   17179869184    4194304    42.388   405.305     98951.40    0.0000     3.17   read        4096 
TARGET   Average     0 256   17179869184    4194304    42.463   404.583     98775.15    0.0000     7.38   read        4096 

TARGET   Average     0   4   17179869184    2097152    81.951   209.636     25590.31    0.0000     0.02   read        8192 
TARGET   Average     0   8   17179869184    2097152    46.336   370.769     45259.91    0.0000     0.06   read        8192 
TARGET   Average     0  16   17179869184    2097152    30.146   569.887     69566.23    0.0000     0.21   read        8192 
TARGET   Average     0  32   17179869184    2097152    24.319   706.428     86233.83    0.0000     0.61   read        8192 
TARGET   Average     0  64   17179869184    2097152    23.279   737.991     90086.83    0.0000     1.34   read        8192 
TARGET   Average     0 128   17179869184    2097152    23.227   739.662     90290.82    0.0000     2.91   read        8192 
TARGET   Average     0 256   17179869184    2097152    23.158   741.848     90557.62    0.0000     6.90   read        8192 

TARGET   Average     0   4   17179869184    1048576    57.798   297.240     18142.06    0.0001     0.01   read       16384 
TARGET   Average     0   8   17179869184    1048576    32.475   529.022     32288.94    0.0000     0.04   read       16384 
TARGET   Average     0  16   17179869184    1048576    20.634   832.610     50818.50    0.0000     0.15   read       16384 
TARGET   Average     0  32   17179869184    1048576    15.640  1098.428     67042.72    0.0000     0.46   read       16384 
TARGET   Average     0  64   17179869184    1048576    14.047  1223.041     74648.47    0.0000     1.15   read       16384 
TARGET   Average     0 128   17179869184    1048576    13.900  1235.937     75435.63    0.0000     2.52   read       16384 
TARGET   Average     0 256   17179869184    1048576    13.853  1240.174     75694.20    0.0000     6.15   read       16384 

TARGET   Average     0   4   17179869184     524288    44.724   384.130     11722.73    0.0001     0.01   read       32768 
TARGET   Average     0   8   17179869184     524288    25.168   682.612     20831.65    0.0000     0.03   read       32768 
TARGET   Average     0  16   17179869184     524288    16.317  1052.859     32130.71    0.0000     0.10   read       32768 
TARGET   Average     0  32   17179869184     524288    12.577  1365.923     41684.66    0.0000     0.29   read       32768 
TARGET   Average     0  64   17179869184     524288    11.236  1529.018     46661.94    0.0000     0.71   read       32768 
TARGET   Average     0 128   17179869184     524288    10.765  1595.947     48704.43    0.0000     1.63   read       32768 
TARGET   Average     0 256   17179869184     524288    10.745  1598.812     48791.86    0.0000     4.14   read       32768

bm-flash

easyco/bm-flash /ssd/test.txt 

Filling 4G before testing  ...   4096 MB done in 2 seconds (2048 MB/sec).

Read Tests:

Block |   1 thread    |  10 threads   |  40 threads   
 Size |  IOPS    BW   |  IOPS    BW   |  IOPS    BW   
      |               |               |               
 512B | 27860   13.6M |139120   67.9M |133423   65.1M 
   1K | 30351   29.6M |138892  135.6M |133319  130.1M 
   2K | 30407   59.3M |138095  269.7M |133254  260.2M 
   4K | 29376  114.7M |135356  528.7M |133530  521.6M 
   8K | 27056  211.3M |129178 1009.2M |134934 1054.1M 
  16K | 23527  367.6M |114792 1793.6M |130400    1.9G 
  32K | 18475  577.3M | 88310    2.6G |107442    3.2G 
  64K | 12940  808.7M | 50713    3.0G | 54531    3.3G 
 128K |  8183 1022.9M | 25923    3.1G | 27523    3.3G 
 256K |  4664 1166.0M | 13036    3.1G | 13720    3.3G 
 512K |  2625 1312.6M |  6545    3.1G |  6819    3.3G 
   1M |  2338    2.2G |  3473    3.3G |  3481    3.3G 
   2M |  1318    2.5G |  1737    3.3G |  1742    3.4G 
   4M |   713    2.7G |   869    3.3G |   873    3.4G 

Write Tests:

Block |   1 thread    |  10 threads   |  40 threads   
 Size |  IOPS    BW   |  IOPS    BW   |  IOPS    BW   
      |               |               |               
 512B | 29680   14.4M | 50779   24.7M | 57239   27.9M 
   1K | 28377   27.7M | 49372   48.2M | 56715   55.3M 
   2K | 27372   53.4M | 50698   99.0M | 56813  110.9M 
   4K | 27174  106.1M | 51021  199.3M | 56385  220.2M 
   8K | 24342  190.1M | 49089  383.5M | 53109  414.9M 
  16K | 18754  293.0M | 36595  571.7M | 43202  675.0M 
  32K |  8561  267.5M | 42592 1331.0M | 23903  746.9M 
  64K |  2425  151.6M | 42309    2.5G | 20235 1264.7M 
 128K |    27    3.4M | 24355    2.9G | 12316 1539.5M 
 256K |  1158  289.5M |  1965  491.3M |  6594 1648.5M 
 512K |   125   62.7M |  5127    2.5G |  1030  515.1M 
   1M |   582  582.7M |   624  624.7M |   526  526.5M 
   2M |   282  565.7M |   685 1370.3M |   955    1.8G 
   4M |   279 1118.3M |   646    2.5G |   449 1798.0M

And in case anyone care, some numbers for sequential performance (this is for the 2x8xRAID6):

dd if=/dev/zero of=/ssd/S1 bs=8k count=8M
8388608+0 records in
8388608+0 records out
68719476736 bytes (69 GB) copied, 72.9746 s, 942 MB/s

dd of=/dev/zero if=/ssd/S1 bs=8k
8388608+0 records in
8388608+0 records out
68719476736 bytes (69 GB) copied, 55.3226 s, 1.2 GB/s
time cp /ssd/S1 /ssd/S2

real	2m50.965s
user	0m1.036s
sys	1m37.226s

As can be seen, approximately 137K IOPS as reported by bm-flash and 92K IOPS as reported by xdd. There were some differences between the sequential performance, but since this thread is mainly about IOPS I won't go into that too much. There are already quite a lot of numbers posted here :P

Clearly, the hardware was indeed capable of doing much more then we initially got from JFS. The 65k barrier has also been broken, by a large margin

[stevecs]

Now that's very interesting, so you are showing pretty much zero improvement w/ RAID-10 (2x8) than with raid-6 (2x8)? And you're sure everything else is the same in both tests? You should be getting much more with a random read test unless we're hitting some other constraint.

[F.E.A.R.]

@FEAR. For a shorter reply than may last one. Yes this is possible with many OS'es. For windows the easiest method would be to create a dynamic disc (you need XP pro at least) and create a stripe (RAID 0) across the two 'areca arrays' (they will appear as individual drives to the OS) this is no functional difference than the linux mdadm approach above but you have a little less control over how the stripe will happen.

I need only XP Pro (and no additional software)?
I have XP x86 and Vista Ultimate x64.

And does it only run with identical controllers?

[nFo]

I need only XP Pro (and no additional software)?
I have XP x86 and Vista Ultimate x64.

And does it only run with identical controllers?

Software Raid I think.

[stevecs]

@FEAR The OS has no idea that the 'disk' it sees is on a controller or not. As long as your OS can see the 'drive' you can encapsulate it. So there is no requirement to have the same controller, though with multiple different vendor controllers you'll probably have different performance profiles and you will be taking up more driver space (driver per controller). Now, there may be limitations on booting a 'striped' OS drive I've never tried that but didn't see anything against it. I have not tried it in vista (don't run it) but would be surprised if they removed that feature that was there in xp pro.

@nfo, yes basically you're striping at the OS level but you're doing the raid at the hardware level.

[F.E.A.R.]

Ok, i will try it with XP Pro.
But where i can find this feature?
And which XP Pro i need? x86 or x64?

[stevecs]

Doesn't make a difference for 32/64bit it's the base code that does this. Just go into your computer management screen to the disk manager. convert your 'array' drive as presented by the areca (this is your raw full array on each card) to a dynamic disk. Then create a new volume you should have the options here to create a 'simple', 'spanned' and 'striped' volume. You want 'striped' you select the disks (your areca arrays) that you want to use for the stripes and then create the volume. You'll need to manually change the partition offset here to your data stripe width boundary so you will have an alligned partition just like with normal arrays under windows.

[F.E.A.R.]

Ok.
Which OS-Stripping performs better - XP Pro or Linux?

[stevecs]

Never seen a test between the two (nor any interest in doing such) as you normally pick the OS based on your application requirements not based on stripe performance. You have much more control over the linux one (you don't under the default windows) but you could also use VxFS (veritas clustering file system) which can be used on many platforms (basically does the same thing as your file system (replaces ntfs) and volume management (like lvm or dynamic discs).

[henk53]

As a followup and as a kind of summary to this topic, I finally found time to jot something down again: see here http://jdevelopment.nl/hardware/100k...dity-hardware/

Our thanks and acknowledgements to stevecs are included in the blog posting