Originally Posted by
stevecs
Another item you want to make sure of if you're really doing random I/O is to disable read-aheads on the controller (or at the very least set it to conservative) you don't want the controller to hold up the drive bus trying to read more sectors in that will most likely never be requested in a random workload environment so by disabling read-ahead you tell the controller to just retrieve the data that is being requested NOT any additional sectors.
Read-ahead is indeed enabled on the controller. I'll test again with these disabled.
Another comment here is that you probably want to use 4K block sizes not 512b (your memory page size is 4K, so that would be the most efficient (or multiples thereof like your 8K you're trying). smaller sizes are generally less efficient in terms of cpu.
Is this in reference to the heading in my result table "IOPS (bs=512b, 10 threads)" ?
I'm not really explicitly testing with 512b, bm-flash tests with various block sizes and starts with 512b. I just took the first line of the results 
So, 512b is just a reading. I could just as well pick the 8KiB line as a reading.
When you did the 8-way raid-0 (what stripe size?) that you want to allign your array to 8*stripe size offset, did you?
For the 8-way raid 0 setup I set the array stripe size on each controller to 8KiB and the LVM stripe size first to 64KiB and then in a second test to 8KiB. The lvm metadatasize (which I assume is what you mean with "allign your array to 8*stripe size offset) was set to 256.
Since in this setup LVM stripes two devices, which each consist of 8 disks in RAID 0, the data WIDTH would then be 2*8*8KiB=128KiB, right? Since the metadatasize of LVM was set to 256KiB, this should be aligned for the 128 data width.
If so then what this is really pointing to is a driver issue that may be capped by this or interrupts. Do you have your IRQ's for your raid cards on different cpu's? (what's your /proc/interrupts look like?) also, what is your setting in /sys/block/<raid device>/queue/nr_requests ?
/sys/block/<raid device>/queue/nr_requests for both RAID devices is set to 256.
/proc/interrupts shows the following:
Code:
cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7
0: 42 0 0 0 0 0 0 0 IO-APIC-edge timer
1: 1 0 1 2 1 1 1 1 IO-APIC-edge i8042
6: 0 1 0 0 0 1 0 0 IO-APIC-edge floppy
8: 0 1 0 0 0 0 0 0 IO-APIC-edge rtc0
9: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi acpi
12: 0 0 1 0 0 1 0 2 IO-APIC-edge i8042
14: 13 11 9 12 13 13 10 13 IO-APIC-edge ide0
20: 5 5 8 4 6 5 6 5 IO-APIC-fasteoi uhci_hcd:usb1
21: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb2
22: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb3
23: 24 23 21 26 25 25 24 24 IO-APIC-fasteoi ehci_hcd:usb4
48: 457 442 460 446 453 439 460 446 IO-APIC-fasteoi arcmsr
52: 14 24 14 22 16 29 16 22 IO-APIC-fasteoi arcmsr
1267: 0 0 0 0 0 1 0 0 PCI-MSI-edge eth0
1268: 7 7 6 5 10 8 6 8 PCI-MSI-edge eth0-rx3
1269: 18 18 19 19 4 6 6 4 PCI-MSI-edge eth0-rx2
1270: 9 12 7 9 13 11 10 11 PCI-MSI-edge eth0-rx1
1271: 26 23 24 22 20 27 19 22 PCI-MSI-edge eth0-rx0
1272: 33 40 37 40 47 42 50 49 PCI-MSI-edge eth0-tx0
NMI: 0 0 0 0 0 0 0 0 Non-maskable interrupts
LOC: 4012 3414 4110 7342 3026 3324 4046 3879 Local timer interrupts
RES: 289 128 320 193 160 85 215 434 Rescheduling interrupts
CAL: 326 477 471 443 412 458 471 442 function call interrupts
TLB: 272 279 966 934 261 272 920 911 TLB shootdowns
TRM: 0 0 0 0 0 0 0 0 Thermal event interrupts
THR: 0 0 0 0 0 0 0 0 Threshold APIC interrupts
SPU: 0 0 0 0 0 0 0 0 Spurious interrupts
Reply With Quote
Bookmarks