Not getting more than 65000 IOPS with xtreme setup

[stevecs]

Yeah, NOOP is probably the best when using intelligent raid controllers, deadline is also helpful in rare cases where you completely saturate the raid controller's buss and you still want to have the system performing other tasks (but if you're waiting on I/O anyway as your sole bottleneck it's kind of moot). Another item you want to make sure of if you're really doing random I/O is to disable read-aheads on the controller (or at the very least set it to conservative) you don't want the controller to hold up the drive bus trying to read more sectors in that will most likely never be requested in a random workload environment so by disabling read-ahead you tell the controller to just retrieve the data that is being requested NOT any additional sectors.

Another comment here is that you probably want to use 4K block sizes not 512b (your memory page size is 4K, so that would be the most efficient (or multiples thereof like your 8K you're trying). smaller sizes are generally less efficient in terms of cpu.

If so then what this is really pointing to is a driver issue that may be capped by this or interrupts. Do you have your IRQ's for your raid cards on different cpu's? (what's your /proc/interrupts look like?) also, what is your setting in /sys/block/<raid device>/queue/nr_requests ?


edit: was interrupted, noticed that you did set your array to, your LVM stripe size for 8KiB should be 64KiB when using 8-way raid-0 (you want to use the data stripe width of an individual array for your LVM stripe size).

more mundane check as well, you are putting your cards into different I/O hubs on your server board right? (w/ 8KiB this is ~512MiB/s transfer rates)