Originally Posted by
stevecs
Got pulled into a project at work, been working /your/ hours for the past bit. :P
:P Got pulled into several other things too. Meanwhile my co-worker has reconfigured the machine; installed a second 1231 and rewired the drives, so now the machine physically has 6 drives attached to each 1231 and 4 drives to the 1680.
Your iostat there is very interesting with the average queue size. What do you see w/ running w/ bm_flash? I wonder if there is something up w/ xdd.
I'll look at the average queue size for the dual card config with bm_flash when I start testing with the striped configs again. It would be interesting to see what the results are there indeed.
Currently my co-worker has configured a RAID 6 array, so I've been running some tests against that. Although it's a little difficult to compare a single RAID 6 array to two LVM striped RAID 0 arrays, I did notice some new things.
For starters, for an xdd test run on a single RAID card, the number of IOPS as reported by iostat is now nearly exactly the same as what xdd reports. For various block sizes (request sizes) it appeared that the dual RAID 0 config was quite a bit faster for block sizes 2KiB and 4 KiB, but was about the same for block size 8KiB.
Here is the output (I edited the column names a little to correspond with bm-flash)
Code:
1x6xRAID6 noop
Threads Bytes Ops Time Rate IOPS Latency %CPU OP_Type BlockSize
TARGET Average 4 51539607552 50331648 1324.04 38.93 38013.58 0 0.02 read 1024
TARGET Average 8 51539607552 50331648 968.65 53.21 51960.7 0 0.06 read 1024
TARGET Average 16 51539607552 50331648 929.48 55.45 54150.42 0 0.13 read 1024
TARGET Average 32 51539607552 50331648 939.8 54.84 53555.54 0 0.26 read 1024
TARGET Average 64 51539607552 50331648 944.16 54.59 53308.57 0 0.53 read 1024
TARGET Average 128 51539607552 50331648 949.7 54.27 52997.57 0 1.12 read 1024
TARGET Average 256 51539607552 50331648 949.84 54.26 52989.66 0 3.13 read 1024
TARGET Average 128 51539607552 25165824 496.72 103.76 50663.57 0 1.06 read 2048
TARGET Average 4 51539607552 12582912 430.33 119.77 29239.91 0 0.01 read 4096
TARGET Average 8 51539607552 12582912 308.72 166.94 40757.88 0 0.04 read 4096
TARGET Average 16 51539607552 12582912 267.53 192.65 47033 0 0.11 read 4096
TARGET Average 32 51539607552 12582912 268.61 191.87 46843.96 0 0.22 read 4096
TARGET Average 64 51539607552 12582912 269.98 190.9 46607.29 0 0.46 read 4096
TARGET Average 128 51539607552 12582912 271.54 189.81 46339.54 0 0.96 read 4096
TARGET Average 256 51539607552 12582912 271.48 189.85 46349.2 0 2.75 read 4096
TARGET Average 4 51539607552 6291456 275.42 187.13 22843.46 0 0.01 read 8192
TARGET Average 8 51539607552 6291456 206.27 249.86 30500.83 0 0.03 read 8192
TARGET Average 16 51539607552 6291456 177.18 290.88 35508.34 0 0.08 read 8192
TARGET Average 32 51539607552 6291456 165.4 311.61 38037.68 0 0.18 read 8192
TARGET Average 64 51539607552 6291456 162.51 317.14 38713.44 0 0.37 read 8192
TARGET Average 128 51539607552 6291456 163.14 315.92 38564.38 0 0.8 read 8192
TARGET Average 256 51539607552 6291456 163.14 315.92 38564.66 0 2.37 read 8192
TARGET Average 4 51539607552 3145728 186.27 276.69 16887.81 0 0.01 read 16384
TARGET Average 8 51539607552 3145728 154.47 333.66 20365.24 0 0.02 read 16384
TARGET Average 16 51539607552 3145728 141.71 363.7 22198.17 0 0.05 read 16384
TARGET Average 32 51539607552 3145728 136.34 378.02 23072.76 0 0.1 read 16384
TARGET Average 64 51539607552 3145728 134.09 384.36 23459.45 0 0.22 read 16384
TARGET Average 128 51539607552 3145728 133.36 386.46 23587.77 0 0.46 read 16384
TARGET Average 256 51539607552 3145728 133.36 386.47 23588.3 0 1.53 read 16384
TARGET Average 4 51539607552 1572864 135.71 379.77 11589.64 0 0.01 read 32768
TARGET Average 8 51539607552 1572864 125.95 409.22 12488.43 0 0.01 read 32768
TARGET Average 16 51539607552 1572864 122.1 422.12 12882.15 0 0.03 read 32768
TARGET Average 32 51539607552 1572864 120.48 427.77 13054.61 0 0.06 read 32768
TARGET Average 64 51539607552 1572864 119.95 429.66 13112.26 0 0.13 read 32768
TARGET Average 128 51539607552 1572864 120.01 429.47 13106.26 0 0.27 read 32768
TARGET Average 256 51539607552 1572864 120.03 429.4 13104.1 0 0.94 read 32768
As can be seen, for 8KiB the IOPS range is from 22k to 38K here, while for the dual 8xRAID 0 (16 disks total) the range was 20k to 38k, which is nearly the same. However for 2KiB it's 29k to 46k here, while the earlier dual setup gave me 38k to 54k. Puzzling...
Did you run xdd against a couple ram disk partitions for comparison? also what file system do you have on here or are you testing against the raw device? if it's xfs what is your ag count & size?
Still haven't done the ram disk test, but hope to be able to do that soon. There's currently some write test running and it doesn't seem to be ending anytime soon 
The file system used for all tests is JFS.
Reply With Quote
Bookmarks