Not getting more than 65000 IOPS with xtreme setup

[henk53]

when you're running xdd w/ >32 threads what is your output for vmstat 5 while that's running? likewise when you do a iostat -m -x 15 (looking for averages, in vmstat to see if you have more context switches than interrupts for example or what % is in i/o wait). iostat can be interesting for service time & utilization. Though I would like to see if you're able a xdd read test against your ram disk to make sure we're not seeing a top limit in xdd here.

I did take a look at iostat earlier. Didn't pay a lot of attention to context switches there, but I did noticed that tps as reported by iostat was nearly equal to the iops reported by bm-flash, but differed from the iops reported by xdd. E.g. if bm-flash would report 60k, than that's basically what iostat is reporting too, but when iostat says 60k, xdd says ~54k.

I do some more testing tomorrow for sure. It's now 1:15 AM in Amsterdam

edit: sleep is so overrated :P so I did a quick test (8 threads, 8KiB request size) with XDD to look at iostat -m -x 15:

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.23    0.00    4.70   24.37    0.00   70.70

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.53    0.00    4.07     0.00     0.02     9.05     0.00    0.00   0.00   0.00
sda1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda2              0.00     0.53    0.00    4.07     0.00     0.02     9.05     0.00    0.00   0.00   0.00
sdb            1607.73     0.00 16649.67    0.00   116.30     0.00    14.31     5.71    0.34   0.06  99.71
sdc            1603.60     0.00 16636.60    0.00   116.16     0.00    14.30     2.43    0.15   0.06  92.77
dm-0              0.00     0.00    0.00    4.60     0.00     0.02     8.00     0.00    0.00   0.00   0.00
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-2              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
md0               0.00     0.00 39750.33    0.00   245.17     0.00    12.63     0.00    0.00   0.00   0.00

and a couple of lines from vmstat 5:

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 3  8   1572 16365168     48 15838276    0    0 14989  3616  131   95  0  3 85 12
 0  8   1572 16364920     48 15838300    0    0 239958     0 29942 63498  0  5 70 25
 0  8   1572 16364912     48 15838300    0    0 239763     0 29972 63240  0  5 70 25
 2  8   1572 16364912     48 15838300    0    0 235338     0 29507 61952  0  5 71 24
 2  8   1572 16364912     48 15838300    0    0 239374     0 29844 63689  0  5 70 26
 1  8   1572 16364912     48 15838300    0    0 239586     0 29948 64244  0  5 69 26

oh, and noticed that the request size was wrong in the script it's set to 8 (that's actually the number times your block size which is 512bytes so it's testing 4096 bytes (1 page)) not your 8K that you wanted which would be a request size of 16.

I did notice that, but thanks for mentioning it Is the block size in xdd btw only a unit for calculations, or does it actually have significance for the way load is put on the device? I.e. is block size 512 bytes and request size 16 the same as block size 1024 and request size 8, or do those differ?