Not getting more than 65000 IOPS with xtreme setup

Hi,

Together with my co-workers I'm trying to create a storage solution capable of doing more than 100k IOPS. See http://jdevelopment.nl/hardware/one-dvd-per-second/ for our attempts thus far.

The problem we're currently facing is that we're unable to get more than ~65k iops for a single device. What we have been trying to do is combining two devices, with each build of of an high performance RAID controller (the 1680IX-12/4GB RAM) with 8 SSDs (mtron pro 7535) on each, into 1 device using either LVM or mdadm striping (raid 0).

In all of our tests the total number of IOPS we get is never higher than ~65k. However, if we put a simultaneous load on the two controllers configured as individual devices, we get a combined total of nearly exactly 100k IOPS (for blocksizes up to 16K, 20 testing threads total, 10 threads per individual device).

Because 65k is 2^16, this number feels like a maximum setting somewhere and not as a hardware limitation. I'm not sure where to look for such a setting though. We're running Debian Linux Lenny. So far I've read up a little on IO schedulers and kernel tuning parameters, but any help would be greatly appreciated.

very nice project. I look forward to more results.

I am not sure about the 65k limit... Napalm was messing around with MFT and was able to hit 300k+ IOPS with RAM:
http://www.xtremesystems.org/forums/...&postcount=558

Quote:

Originally Posted by One_Hertz

very nice project. I look forward to more results.

Thanks :) Next on our list is testing with a 1231ML, hopefully with two of those too.

Quote:

I am not sure about the 65k limit... Napalm was messing around with MFT and was able to hit 300k+ IOPS with RAM:
http://www.xtremesystems.org/forums/...&postcount=558

That's quite interesting indeed. We are working on Linux though so it's a little hard to compare any setting to that, but it does mean it might be an idea to do some testing on Windows. I'm not sure how to create s software RAID on Windows (I have very little experience with Windows), but I'm sure it can't be that hard ;)

Quote:

Originally Posted by One_Hertz

I am not sure about the 65k limit... Napalm was messing around with MFT and was able to hit 300k+ IOPS with RAM:
http://www.xtremesystems.org/forums/...&postcount=558

You realize the card cannot achieve that kind of speed even if it were a double ARC1231?:)
PCI-e x8 is capped at 2GB/s, and realistic limit is around 1.5GB/s.
I donno how he got the numbers, but they aren't real ;)

@OP: I think you won't be able to physically achieve more than that, and it has little to do with the card, but the Mtrons.
In best case scenario, the ranom I/O would be split across each drive in order, but in reality it will never happen ;( So you're going with mathematical probability here, which at 8 drives, presuming the stripe is the same as the block used for the random I/O, I think is 4 drives, i.e. after 5 I/Os you'll most probably hit one of the previous drives and thus cap the IOPs.

IIRC, a single Mtron does ~17000 random IOPs, so that makes perfect sense for 4x17000 = ~65000 IOPs.

Quote:

Originally Posted by alfaunits

You realize the card cannot achieve that kind of speed even if it were a double ARC1231?:)
PCI-e x8 is capped at 2GB/s, and realistic limit is around 1.5GB/s.
I donno how he got the numbers, but they aren't real ;)

@OP: I think you won't be able to physically achieve more than that, and it has little to do with the card, but the Mtrons.
In best case scenario, the ranom I/O would be split across each drive in order, but in reality it will never happen ;( So you're going with mathematical probability here, which at 8 drives, presuming the stripe is the same as the block used for the random I/O, I think is 4 drives, i.e. after 5 I/Os you'll most probably hit one of the previous drives and thus cap the IOPs.

IIRC, a single Mtron does ~17000 random IOPs, so that makes perfect sense for 4x17000 = ~65000 IOPs.

I did say it was from RAM not storage devices...

@ the second part: I might have read that wrong but I think they tried with 12 drives too and got the same 6Xk?

Very interesting study though a couple comments and observations:

First per your article you were using two ARC-1680IX-12's and using LVM2 to stripe at the OS level. What was your stripe size both for the LVM2 and for your array. LVM2 should be a multiple of your sub-array stripe size (probably equal to your stripe width of the base array that way each controller would have enough information to write full stripes which work here, but this is for large data not for 8K database blocks so some testing would need to be done. At least it should be => a multiple of your stripe size though). Also are you testing w/ a partition table on the array or raw (ie, alignment issues). Normally I test with no partition table (raw raid volume gets added to lvm, and then put a file system right on the lv).

In your write up you mentioned that you got ~3200MB/s out of two cards? That to me screams ram testing not disk testing. I could see with 2 controllers around 1800MB but not double that, unless I've mis-read what you were doing there.

As for IOPS from your results it really looks like a raid controller limit (IOP34x series) more than anything, but there are a couple things that can be tested.

- create a ram disk (system ram) and run your tests against that (with lvm2 & put a file system on it). This would rule out completely the drive/controller subsystem and still test your OS/filesystem. If you're still hitting 65K then it's something in kernel space (though honestly I've never heard of an issue like that before).
- if possible, try cards that DO NOT have a built-in expander (the -IX line from areca does, so you're going through that chip each time). The ARC-1680 doesn't have the expander which may improve performance a bit (also may help resolving some incompatibility issues you mentioned in the article)

Quote:

Originally Posted by alfaunits

@OP: I think you won't be able to physically achieve more than that, and it has little to do with the card, but the Mtrons.
In best case scenario, the ranom I/O would be split across each drive in order, but in reality it will never happen ;(

That's true, but logic tells me that eventually on average both devices would get a similar amount of random I/O. I might be able to monitor somewhere how much I/O each device would be getting during a load test, but I would be very surprised really if it appears that one device had to process 80% of the I/O while the other had to do only 20%.

Quote:

IIRC, a single Mtron does ~17000 random IOPs, so that makes perfect sense for 4x17000 = ~65000 IOPs.

Well, we were using a total of 16 drives, not 8 ;) Each controller had 8 Mtron's. We striped two controllers, for a total number of 16 disks.

Quote:

Originally Posted by stevecs

Very interesting study though a couple comments and observations:
First per your article you were using two ARC-1680IX-12's and using LVM2 to stripe at the OS level. What was your stripe size both for the LVM2 and for your array.

The stripe size for the array (set when creating the array in the areca bios) was 128k. For LVM we used the following command:

Code:

pvcreate --metadatasize 250 /dev/sdb /dev/sdc - vgcreate ssd /dev/sdb /dev/sdc - lvcreate -i2 -I128 -L447G -n ssd-striped ssd

Getting all the right alignment settings right proved to be rather tricky. I'm not 100% sure that we got everything absolutely right to be honest.

Quote:

In your write up you mentioned that you got ~3200MB/s out of two cards? That to me screams ram testing not disk testing. I could see with 2 controllers around 1800MB but not double that, unless I've mis-read what you were doing there.

The 3.3GB/s was reported by bm-flash, but only at very large block sizes (>256K) and with multiple threads (>10). This is most likely indeed the cache, although with smaller block sizes we weren't getting this speed.

Quote:

- create a ram disk (system ram) and run your tests against that (with lvm2 & put a file system on it).

Okay, what would you suggest as the best way to create a ram disk? I created one using GRUB (5GB), and got a lot more IOPS:

Code:

Filling 4G before testing ... 4096 MB done in 2 seconds (2048 MB/sec). Read Tests: Block | 1 thread | 10 threads | 40 threads Size | IOPS BW | IOPS BW | IOPS BW | | | 512B |207091 101.1M |202807 99.0M |204409 99.8M 1K |197413 192.7M |194549 189.9M |195600 191.0M 2K |184750 360.8M |180739 353.0M ^C

I have to find out how to create two RAM disks so LVM can be used on those.

Quote:

- if possible, try cards that DO NOT have a built-in expander (the -IX line from areca does, so you're going through that chip each time). The ARC-1680 doesn't have the expander which may improve performance a bit (also may help resolving some incompatibility issues you mentioned in the article)

That quite interesting to hear indeed. We just received the 1231ML and are going to test with that tomorrow. Currently it's well passed midnight here in Amsterdam, so I have to call it a day ;)

Quote:

Originally Posted by One_Hertz

I did say it was from RAM not storage devices...

That picture shows an Areca volume, from what I see..
And even then 8GB/s for memory copy is not something I would trust. Maybe it is possible, but I doubt it.

Quote:

the second part: I might have read that wrong but I think they tried with 12 drives too and got the same 6Xk?

Ye, I think probability kills them here as well. I don't remember it enough to give the exact formula and calculate, but I think it has more to do with that than anything else.
I.e. not the card, not the rest of the computer, but just the drives.

Quote:

Originally Posted by henk53

That's true, but logic tells me that eventually on average both devices would get a similar amount of random I/O. I might be able to monitor somewhere how much I/O each device would be getting during a load test, but I would be very surprised really if it appears that one device had to process 80% of the I/O while the other had to do only 20%.

Again if you RAID them in the OS, then the probability that you will hit the same physical array is 50%, and it won't allow the total random to be 2x, but 150%, which it is.

Quote:

Well, we were using a total of 16 drives, not 8 ;) Each controller had 8 Mtron's. We striped two controllers, for a total number of 16 disks.

Yes, that's how I calculated it ;)

Quote:

Originally Posted by henk53

The stripe size for the array (set when creating the array in the areca bios) was 128k. For LVM we used the following command:

Code:

pvcreate --metadatasize 250 /dev/sdb /dev/sdc - vgcreate ssd /dev/sdb /dev/sdc - lvcreate -i2 -I128 -L447G -n ssd-striped ssd

Getting all the right alignment settings right proved to be rather tricky. I'm not 100% sure that we got everything absolutely right to be honest.

Ok, first for small (and random I/O) this is surprising why you picked such a large stripe size. I would have assumed you would use 8K or 16K since you said your request size was fixed at 8K? This doesn't make much of a difference with reads but writes can be more of a hit. Unless the SSD's behave better with larger stripe sizes (cell writing more efficient or something? I haven't played w/ SSD's much to really dig to that level).

But your command string tells me that you are using raw devices (/dev/sdb, /dev/sdc) which is good (no partition tables). Your metadatasize though is strange, if you have a stripe size of 128K why is your meta data size 250? I could see 255 (ie, buffer fill to 256 which is 2x128) or even 127 (single stripe) but not 250. Your setting for striping of 128 (-I128) is something a bit larger than I would have started using assuming your request size of 8K You probably picked this as it was your stripe size for the underlaying array. That has some merit though if that really is your request size I would set both to 8K (or 128K if you picked that for a SSD write reason) (array stripe and this, OR if you're doing a lot of writes to set this to be stripe size*#data disks in base array. That way you increase the chance of writing a full stripe to the array and not using as much controller cache waiting to fill it out causing a read/modify/write partial stripe penalty.

Quote:

Originally Posted by henk53

Okay, what would you suggest as the best way to create a ram disk? I created one using GRUB (5GB), and got a lot more IOPS:

I have to find out how to create two RAM disks so LVM can be used on those.

This is much better though not as high as I would expect (how many banks do you have populated in the system? do you have any memory mirroring turned on or spare rows enabled?) Anyway it does prove at least that the i/o kernel subsystem is not likely the problem here at all. To do more testing you can create ram disks by:

Code:

#assuming /dev/ram0 doesn't already exist mknod -m 660 /dev/ram0 b 1 0 mknod -m 660 /dev/ram1 b 1 1 # size the disk the way you want it (8GiB below) dd if=/dev/zero of=/dev/ram0 bs=1M count=8K dd if=/dev/zero of=/dev/ram1 bs=1M count=8K

then treating /dev/ram0 /dev/ram1 as physical discs for you to work on (adding them to lvm putting a file system on them et al).

This should give you a max value that you can reach on your hardware. You'll be lower than this going through the I/O hubs and then the card controllers and any expanders.

Another item from reading your specs on the arecas I see that you have some settings set to auto that you may want to modify:

HDD Read Ahead: (assuming you mean HDD Read Ahead Cache? or volume data read ahead cache?) Anyway, for random workloads you want to turn this down or off especially if you're going to be loading the system with high levels of really random traffic. You /may/ get by with hdd read ahead cache enabled, and volume data read ahead set to conservative which gives a little volume buffering (helps with raid parity checks for example) without too much of a penalty with random workloads.

To test your theory on reasons for the choppy writes you can try disabling or removing the cache on the controller to see if it evens it out (or using smaller caches). Areca had some problems in the past w/ large caches that I thought were fixed but you may be running into something there (as you surmised filling the cache and then taking the time to write it out to the drives).

Quote:

Originally Posted by stevecs

Ok, first for small (and random I/O) this is surprising why you picked such a large stripe size. I would have assumed you would use 8K or 16K since you said your request size was fixed at 8K?

It basically is. The main (actually the only) application that will be using the storage is PostgreSQL. PostgreSQL defaults to using an 8Kb block size. This can be changed, but that requires recompiling the software.

We choose 128KB since that's what everybody recommends, including the supplier (ssdisk.eu). Nevertheless I plan to test with a RAID array stripe size of 4KB and an LVM stripe size of 8KB.

Quote:

Your metadatasize though is strange, if you have a stripe size of 128K why is your meta data size 250? I could see 255 (ie, buffer fill to 256 which is 2x128) or even 127 (single stripe) but not 250.

That's indeed a story of its own. The default metadatasize is 192, so I assume this is the minimum. Since a multiple of 128k was supposed to be the most efficient, the value should be 256. However, as it appears pvcreate does some automatic rounding up. When you set it to 256, you end up getting 320. 250 is thus a little trick. When you set it to 250, you end up with 256.

It appeared others found the same trick:

Quote:

LVM likes to allocate 192k for its header information, and 192k is not a multiple of 128k. So if you are creating file systems as logical volumes, and you want those volume to be properly aligned you have to tell LVM that it should reserve slightly more space for its meta-data, so that the physical extents that it allocates for its logical volumes are properly aligned. Unfortunately, the way this is done is slightly baroque:

# pvcreate –metadatasize 250k /dev/sdb2
Physical volume “/dev/sdb2″ successfully created

Why 250k and not 256k? I can’t tell you — sometimes the LVM tools aren’t terribly intuitive. However, you can test to make sure that physical extents start at the proper offset by using:

See: http://thunk.org/tytso/blog/2009/02/...se-block-size/

Quote:

This is much better though not as high as I would expect (how many banks do you have populated in the system?

The system is currently equipped with 32GB (which is 16GB less then we used for earlier testing) consisting of 8 modules of 4GB each.

Quote:

Code:

#assuming /dev/ram0 doesn't already exist mknod -m 660 /dev/ram0 b 1 0 mknod -m 660 /dev/ram1 b 1 1 # size the disk the way you want it (8GiB below) dd if=/dev/zero of=/dev/ram0 bs=1M count=8K dd if=/dev/zero of=/dev/ram1 bs=1M count=8K

I tried this, but it doesn't really work. This way the /dev/ramx is limited to 64mb. I finally did get it to work by setting the ram disk size in GRUB again (to 5GB) and simply using the existing /dev/ram0 /dev/ram 1 etc ;)

Creating an LVM stripe out of the two RAM disks gave me a slightly lower performance compared to just hitting a single RAM disk.

I also tried to stripe a RAM disk with an Areca, just for the experiment, and it gave me more then 64k, namely 80k. I'll summarize these findings below:

Code:

config | IOPS (bs=512b, 10 threads) ------------------------------------ Single RAM disk | 200k LVM striped RAM disk | 190k 2 individual 5xRAID0 | 100k (2x50k) lvm striped RAM+SSD | 80k 1 12xRAID0 | 63k 1 4xRAID0 | 63k 2 lvm striped 8xRAID0| 60K 2 lvm striped 4xRAID0| 60K

This table is for the 512b blocks size. The larger block sizes have slightly less IOPS, but the relative performance between the configurations are the same.

The single RAM disk shows that the system itself (kernel wise) is able to get more then ~60k IOPS. The test with putting load on two individual devices show that the hardware itself is capable of at least 100k IOPS (our target). This dual test includes the PCIe buses.

We also see that a single Areca controller can do at least 63K IOPS. This is either from it's cache or from the SSD, but the controller itself can obviously handle it. Striping an Areca controller with a RAM disk gives 80K and Striping two RAM disks gives 190K, so LVM doesn't limit the IOPS either.

Still, either increasing the number of disks on a single controller (e.g. from 4 to 12) or combining two controllers does not increase the number of IOPS.

It could of course well be that this is really due to the probability cause mentioned above, but one would think that with 40 testing threads all doing completely random IO the chance that all SSDs receive an equal amount of load would be fairly high.

Quote:

HDD Read Ahead: (assuming you mean HDD Read Ahead Cache? or volume data read ahead cache?)

It's HDD Read Ahead Cache ;) I tried to set this setting to disabled, but the results are about the same. The following is for 1x12xRAID0:

Code:

Filling 4G before testing ... 4096 MB done in 3 seconds (1365 MB/sec). Read Tests: Block | 1 thread | 10 threads | 40 threads Size | IOPS BW | IOPS BW | IOPS BW | | | 512B | 25633 12.5M | 64028 31.2M | 63074 30.7M 1K | 28228 27.5M | 63843 62.3M | 63060 61.5M 2K | 29437 57.4M | 63737 124.4M | 62826 122.7M 4K | 31409 122.6M | 63218 246.9M | 62516 244.2M 8K | 30017 234.5M | 62355 487.1M | 61603 481.2M 16K | 25898 404.6M | 61443 960.0M | 61167 955.7M 32K | 19703 615.7M | 51208 1600.2M | 50890 1590.3M

One setting that I'm desperate for to try is Queue Depth. Thus far I'm unable to locate where to set this setting. It's mentioned in the Areca firmware release notes:

Code:

Change Log For V1.44 Firmware 2007-4-26 * Add enclosure add/remove event * Add Battery Status for SAS (Battery is not monitored) 2007-5-8 * Add Queue Depth Setting for ARC1680 (default 16)

Does anyone know where this setting is supposed to be set on either the 1680 or 1231?

edt: I did find the setting "Hdd Queue Depth Setting" on our 1680IX. This was set to 32 (the max). No such setting can be found on the 1231 though.

Quote:

Originally Posted by henk53

We choose 128KB since that's what everybody recommends, including the supplier (ssdisk.eu). Nevertheless I plan to test with a RAID array stripe size of 4KB and an LVM stripe size of 8KB.

No, that would give you the reverse of what you want, since the LVM array would then ALWAYS read/write to both Areca arrays, and that is one thing you WANT to avoid.
So, choose a stripe size that is equal to the buffer size, i.e. 8K, and LVM stripe of 16K.

If you want to test whether the system can achieve more IOPS at the hardware level, create a random test yourself. Namely, for a 12 disk Mtron array, 8K stripe, single controller only: create software that creates 12 threads, each reading random location multiplied by it's thread number. SO each thread would read from the same drive always, but you would get 12 drive reads.
That would tell you if the limit is: in the controller (if it can't go over 65K) or the CPU/chipset/kernel (if it can't go over some other amount, say... 200K that you have already observed).
Also note that Areca uses sector 0x208 as the starting sector for its arrays! When you need to align the partitions, this must be taken into account :(

@henk53- If you are using 8KiB request sizes from postgresql then I would at least try 8KiB as a stripe size, I wouldn't use 4KiB as that would be 1/2 your request size so you would be splitting it across two drives (halving your subsystem's available iops which is opposite of what you want to do).

As for the lvm metadatasize rounding yes I'm aware of that and the default of 192KiB, that /should not/ increase to 320KiB if you are UNDER 256 (255KiB). Basically LVM ear-marks 3 blocks of 64KiB each by default for data. If you set it to 256 you are in the next block (remember this is 0 offset) so it will 'pad' that to 320 (256+64) as your new starting point for your data. So 255 should also work. The main point however is when you check the offset w/ pvs (pvs -o+pe_start) you should see your new starting point as long as it's aligned it doesn't really matter how you get there. :)

Now the question is (which I have NOT benchmarked myself) is should this be set to a multiple of your STRIPE size or a multiple of your data STRIPE WIDTH. (ie. assuming 5 drives in raid-5 with a 128K stripe size, your stripe width for the array would be 128*4 (minus 1 drive for parity) so 512KiB. By using your data stripe width for your LVM stripe size it /should/ increase your performance as you would have less stripe width fragmentation and cut down on the read/modify/write penalties per sub array. remember to set your metadatasize to your stripe width boundary not to your stripe size. It would prove to be very interesting if you could test that.

The numbers you get from your ram tests are very good and strongly point to the raid card being a bottleneck. Now why you don't get scale with multiple cards I think is due to how your i/o is being distributed and the low level updates (array & lvm striping) OR the areca driver. Another item to check as well is how do you have your interrupts distributed to your cpus' for the cards? do you have each card going to a different cpu & core?

As for queue depth I have not found anything on the ARC16xx cards that sets it per-drive, however you can set it per array (/sys/block/<device>/queue/nr_requests) max is 256 for the cards. At the same time here you may want to set your scheduler to deadline or noop if you haven't done so already (/sys/block/<device>/queue/scheduler )

Quote:

Originally Posted by alfaunits

You realize the card cannot achieve that kind of speed even if it were a double ARC1231?:)
PCI-e x8 is capped at 2GB/s, and realistic limit is around 1.5GB/s.
I donno how he got the numbers, but they aren't real ;)

@OP: I think you won't be able to physically achieve more than that, and it has little to do with the card, but the Mtrons.
In best case scenario, the ranom I/O would be split across each drive in order, but in reality it will never happen ;( So you're going with mathematical probability here, which at 8 drives, presuming the stripe is the same as the block used for the random I/O, I think is 4 drives, i.e. after 5 I/Os you'll most probably hit one of the previous drives and thus cap the IOPs.

IIRC, a single Mtron does ~17000 random IOPs, so that makes perfect sense for 4x17000 = ~65000 IOPs.

you realize that the results could be cached ?? :rolleyes: :)

the numbers are real i didnt just pullem out of my butt

Quote:

Originally Posted by One_Hertz

I did say it was from RAM not storage devices...

@ the second part: I might have read that wrong but I think they tried with 12 drives too and got the same 6Xk?

again its not the system ram.. controllers cache

Mate, even if the data is from the controller cache, that data cannot arrive faster than the PCI-e link allows, and that's 2GB/s. The cache is not in the system RAM for a dedicated controller.
And even then, those numbers would be impossible.

Quote:

Originally Posted by stevecs

Now the question is (which I have NOT benchmarked myself) is should this be set to a multiple of your STRIPE size or a multiple of your data STRIPE WIDTH.

Sorry for my ignorance, but is "STRIPE size" the array stripe size (the one I set in the Areca controller) and "STRIPE WIDTH" the LVM stripe size (the one I use for the pvcreate command) ?

Quote:

(ie. assuming 5 drives in raid-5 with a 128K stripe size, your stripe width for the array would be 128*4 (minus 1 drive for parity) so 512KiB.

Okay, so let me see if I get this right. If I thus set an array stripe size of 8KiB, then when using 5 drives in raid-5, I would use an LVM stripe size of 8*4= 32KiB.

The LVM command would then initially become something like this:

Code:

pvcreate --metadatasize 250 /dev/sdb /dev/sdc - vgcreate ssd /dev/sdb /dev/sdc - lvcreate -i2 -I32 -L447G -n ssd-striped ssd

Quote:

remember to set your metadatasize to your stripe width boundary not to your stripe size. It would prove to be very interesting if you could test that.

Hmmm, if I use metadatasize = 250, it will become 256 which is aligned with 32, Or do you mean something else here? I'm definitely willing to test this ;)

stripe size = the contiguous space assigned to each drive (what you set in the controller). stripe width is the # of drives in your array TIMES the stripe size. Now a data stripe width is the number of data drives per array (ie, raid-5 of 5 drives would have a stripe width of '5' or a data stripe width of 4, a raid-6 of 5 drives would have a stripe width of 6 and data stripe width of 4 (2 drives for parity).

Now for lvm you have to worry about your 192K initial metadata. So as you calculated if you have a raid-5 of 5 drives (4 data) your stripe width would be 32. Since 192KiB is divisible by 32 (6 stripes) you want to start your array AT 192K (so in this example you are alligned right out the gate you don't need to set a metadatasize argument). But you want your LVM stripe size (between arrays) to be a multiple of your individual array data stripe width (ie, 32KiB) That way you are not hopping between arrays and only writing partial stripes.

Now for argument if you have a stripe width of 128KiB for your array (same 5 drive raid-5) that would be 512KiB for data stripe width. So you want to use your metadatasize offset to START your partition at 512KiB (or multiple thereof), and use your lvm striping (between your arrays) to also be 512KiB (data stripe width of a single array) or a multiple thereof.

so

Code:

pvcreate --metadatasize X /dev/sdb /dev/sdc # X 511 or whatever it takes to start at 512KiB with padding) vgcreate ssd -s Y /dev/sdb /dev/sdc # Y here is your physical extent size which you probably want to be a multiple of your data stripe width default is 4MiB and that's fine as we our data stripe widths here are evenly divisable to that for 8K (32KiB) or 128K (512KiB) lvcreate -i2 -IZ -L447G -n ssd-striped ssd # Z here should be a multiple of your data stripe width (32KiB or 512KiB)

make sense?

Quote:

Originally Posted by stevecs

stripe size = the contiguous space assigned to each drive (what you set in the controller). stripe width is the # of drives in your array TIMES the stripe size. Now a data stripe width is the number of data drives per array (ie, raid-5 of 5 drives would have a stripe width of '5' or a data stripe width of 4, a raid-6 of 5 drives would have a stripe width of 6 and data stripe width of 4 (2 drives for parity).

Ah, ok. That's a very clear explanation. Thanks a lot. Your earlier suggestion of using the NOOP scheduler turned out to be quite beneficial. Using 12 disks on a single controller the number of IOPS increased from ~63k to ~71k.

Next to that I did some more testing, this time I build 2 RAID0 arrays, each consisting of 8 Mtrons on an Areca card. For the time being the first Areca card is a 1231ml and the other is a 1680ix. We ordered a second 1231ml, but it didn't arrive yet.

The array stripe size was set to 8KiB and the LVM stripe size to 64KiB. The metadatasize was kept at 256 KiB, which is a multiple of the data width you mentioned (8 disks in RAID0 * 8KiB stripe size = 64KiB). I'm not really sure why I choose 64KiB for the LVM stripe size, but I just did. I will re-test this soon with an 8KiB LVM stripe size. The scheduler used was the NOOP scheduler.

I tested the setup in three ways: the two devices individually, the two devices concurrently and the two devices striped via LVM. The results are as follows:

Code:

config | IOPS (bs=512b, 10 threads) ------------------------------------ 2 individual 8xRAID0 | 113k (2x56.5k) 1 8xRAID0 | 71k 2 lvm striped 8xRAID0| 65K

So with the smaller stripe size, the NOOP scheduler and the metadatasize alligned with the stripe WIDTH, I'm still seeing the same thing happening. Testing the devices concurrently gives me a very high number of IOPS, proving the hardware itself is quite capable. Testing an individual device gives a number of IOPS that is approximately 60% of the concurrent test. Finally, testing the LVM stripe gives an amount of IOPS that is about 10% lower then what a single device gives me.

Relatively these numbers really aren't unlike the numbers that I've seen before. In theory striping should increase performance (and it actually does for sequential IO), but for random IO I just don't see it happening.

Yeah, NOOP is probably the best when using intelligent raid controllers, deadline is also helpful in rare cases where you completely saturate the raid controller's buss and you still want to have the system performing other tasks (but if you're waiting on I/O anyway as your sole bottleneck it's kind of moot). Another item you want to make sure of if you're really doing random I/O is to disable read-aheads on the controller (or at the very least set it to conservative) you don't want the controller to hold up the drive bus trying to read more sectors in that will most likely never be requested in a random workload environment so by disabling read-ahead you tell the controller to just retrieve the data that is being requested NOT any additional sectors.

Another comment here is that you probably want to use 4K block sizes not 512b (your memory page size is 4K, so that would be the most efficient (or multiples thereof like your 8K you're trying). smaller sizes are generally less efficient in terms of cpu.

If so then what this is really pointing to is a driver issue that may be capped by this or interrupts. Do you have your IRQ's for your raid cards on different cpu's? (what's your /proc/interrupts look like?) also, what is your setting in /sys/block/<raid device>/queue/nr_requests ?

edit: was interrupted, noticed that you did set your array to, your LVM stripe size for 8KiB should be 64KiB when using 8-way raid-0 (you want to use the data stripe width of an individual array for your LVM stripe size).

more mundane check as well, you are putting your cards into different I/O hubs on your server board right? (w/ 8KiB this is ~512MiB/s transfer rates)

Quote:

Originally Posted by stevecs

Another item you want to make sure of if you're really doing random I/O is to disable read-aheads on the controller (or at the very least set it to conservative) you don't want the controller to hold up the drive bus trying to read more sectors in that will most likely never be requested in a random workload environment so by disabling read-ahead you tell the controller to just retrieve the data that is being requested NOT any additional sectors.

Read-ahead is indeed enabled on the controller. I'll test again with these disabled.

Quote:

Another comment here is that you probably want to use 4K block sizes not 512b (your memory page size is 4K, so that would be the most efficient (or multiples thereof like your 8K you're trying). smaller sizes are generally less efficient in terms of cpu.

Is this in reference to the heading in my result table "IOPS (bs=512b, 10 threads)" ?

I'm not really explicitly testing with 512b, bm-flash tests with various block sizes and starts with 512b. I just took the first line of the results ;) So, 512b is just a reading. I could just as well pick the 8KiB line as a reading.

Quote:

When you did the 8-way raid-0 (what stripe size?) that you want to allign your array to 8*stripe size offset, did you?

For the 8-way raid 0 setup I set the array stripe size on each controller to 8KiB and the LVM stripe size first to 64KiB and then in a second test to 8KiB. The lvm metadatasize (which I assume is what you mean with "allign your array to 8*stripe size offset) was set to 256.

Since in this setup LVM stripes two devices, which each consist of 8 disks in RAID 0, the data WIDTH would then be 2*8*8KiB=128KiB, right? Since the metadatasize of LVM was set to 256KiB, this should be aligned for the 128 data width.

Quote:

If so then what this is really pointing to is a driver issue that may be capped by this or interrupts. Do you have your IRQ's for your raid cards on different cpu's? (what's your /proc/interrupts look like?) also, what is your setting in /sys/block/<raid device>/queue/nr_requests ?

/sys/block/<raid device>/queue/nr_requests for both RAID devices is set to 256.

/proc/interrupts shows the following:

Code:

cat /proc/interrupts CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 0: 42 0 0 0 0 0 0 0 IO-APIC-edge timer 1: 1 0 1 2 1 1 1 1 IO-APIC-edge i8042 6: 0 1 0 0 0 1 0 0 IO-APIC-edge floppy 8: 0 1 0 0 0 0 0 0 IO-APIC-edge rtc0 9: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi acpi 12: 0 0 1 0 0 1 0 2 IO-APIC-edge i8042 14: 13 11 9 12 13 13 10 13 IO-APIC-edge ide0 20: 5 5 8 4 6 5 6 5 IO-APIC-fasteoi uhci_hcd:usb1 21: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb2 22: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb3 23: 24 23 21 26 25 25 24 24 IO-APIC-fasteoi ehci_hcd:usb4 48: 457 442 460 446 453 439 460 446 IO-APIC-fasteoi arcmsr 52: 14 24 14 22 16 29 16 22 IO-APIC-fasteoi arcmsr 1267: 0 0 0 0 0 1 0 0 PCI-MSI-edge eth0 1268: 7 7 6 5 10 8 6 8 PCI-MSI-edge eth0-rx3 1269: 18 18 19 19 4 6 6 4 PCI-MSI-edge eth0-rx2 1270: 9 12 7 9 13 11 10 11 PCI-MSI-edge eth0-rx1 1271: 26 23 24 22 20 27 19 22 PCI-MSI-edge eth0-rx0 1272: 33 40 37 40 47 42 50 49 PCI-MSI-edge eth0-tx0 NMI: 0 0 0 0 0 0 0 0 Non-maskable interrupts LOC: 4012 3414 4110 7342 3026 3324 4046 3879 Local timer interrupts RES: 289 128 320 193 160 85 215 434 Rescheduling interrupts CAL: 326 477 471 443 412 458 471 442 function call interrupts TLB: 272 279 966 934 261 272 920 911 TLB shootdowns TRM: 0 0 0 0 0 0 0 0 Thermal event interrupts THR: 0 0 0 0 0 0 0 0 Threshold APIC interrupts SPU: 0 0 0 0 0 0 0 0 Spurious interrupts

This is btw what /proc/interrupts shows during a load test on the LVM striped array:

Code:

cat /proc/interrupts CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 0: 42 0 0 0 0 0 0 0 IO-APIC-edge timer 1: 1 0 1 2 1 1 1 1 IO-APIC-edge i8042 6: 0 1 0 0 0 1 0 0 IO-APIC-edge floppy 8: 0 1 0 0 0 0 0 0 IO-APIC-edge rtc0 9: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi acpi 12: 0 0 1 0 0 1 0 2 IO-APIC-edge i8042 14: 13 11 9 12 13 13 10 13 IO-APIC-edge ide0 20: 5 5 8 4 6 5 6 5 IO-APIC-fasteoi uhci_hcd:usb1 21: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb2 22: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb3 23: 24 23 21 26 25 25 24 24 IO-APIC-fasteoi ehci_hcd:usb4 48: 336872 336984 336130 336383 337866 337167 337506 337323 IO-APIC-fasteoi arcmsr 52: 336991 336875 337740 337475 335995 336691 336368 336542 IO-APIC-fasteoi arcmsr 1267: 0 0 0 0 0 1 0 0 PCI-MSI-edge eth0 1268: 10 9 6 7 13 10 10 12 PCI-MSI-edge eth0-rx3 1269: 20 21 20 22 6 7 7 5 PCI-MSI-edge eth0-rx2 1270: 67 70 63 74 81 73 71 72 PCI-MSI-edge eth0-rx1 1271: 97 95 102 93 84 104 87 91 PCI-MSI-edge eth0-rx0 1272: 145 150 144 147 156 149 156 154 PCI-MSI-edge eth0-tx0 NMI: 0 0 0 0 0 0 0 0 Non-maskable interrupts LOC: 18164 17107 11112 20057 8536 8791 9079 8773 Local timer interrupts RES: 369121 327407 190796 206083 104982 104750 98572 92634 Rescheduling interrupts CAL: 328 479 473 445 414 460 473 442 function call interrupts TLB: 286 295 1028 1041 284 286 1014 994 TLB shootdowns TRM: 0 0 0 0 0 0 0 0 Thermal event interrupts THR: 0 0 0 0 0 0 0 0 Threshold APIC interrupts SPU: 0 0 0 0 0 0 0 0 Spurious interrupts

It seems clear that interrupts are divided equally over the total number of CPUs in the system, especially for the Areca drivers (arcmsr).

Your interrupts look good. much better than what I saw when doing some testing here w/ 10GbE cards which is why I brought it up.

Read-ahead should be disabled on the drive (if you have the option) and controller when doing random testing this will allow the drive to release back to the higher layers without hanging on trying to send more data than what was actually requested. (also true for the OS as well but let's focus on the atomic pieces first.)

Yes I was referring to your bm-flash header line. I just grabbed it here and haven't used it before, did you just grab this out of the blue to test or does this actually mimic your workload? and have you baselined it before on other arrays to verify it's accuracy?

If your array (raid-0 / 8 drives, 8KiB stripe size) that would give you a data stripe width of 64KiB which if you offset your array w/ metadatasize to allign to 256KiB which you said above you are alligned. Your LVM stripe width should be set to a multiple of a SINGLE sub-array so 64KiB. The LVM stirpe width indicates how much data is to be contained on each physical volume AS IT SEES IT. In this case that physical volume is actually your array which is comprised of multiple disks so you want it to be set to your data stripe width. Having 256KiB here is ok as it's a multiple of 64KiB but that means you will send 4 times your stripe width to each sub array, this could be good or bad (only way is to test) but ideally with a 100% random workload it should be best to have it equal to you data stripe width (that way you have a higher chance of distributing your workload to each controller).

You can try increasing your nr_requests (try 512 or 1024) to see if it has an effect. I don't believe that areca cards can handle more than 256, though I've not tested on an array that was capable of such high IOPS to verify without statistical error coming into play.

Another tool to test with would be XDD just to verify the numbers that you're getting w/ bm-flash. (http://www.ioperformance.com/products.htm)

fast script to help run it. This creates a 64GiB file (S0) to run against with 100% random read & write I/O w/ 8KiB request size across the range of the file you may want to increase that likewise if you wanted to increase the 16384 MiB to read/write to as you have a lot of ram but the -dio flag should do direct i/o anyway. Just as a comparison point.

Code:

#!/bin/bash ################################ CONTROLLER="ARC1680ix" RAID="R0" DISKS="D8" DRIVE="st31000340ns" SS="SS008k" FS="jfs" USERNAME=ftpadmin TMP="/var/ftp/tmp" XDD="/usr/local/bin/xdd.linux" XDDTARGET="${TMP}/S0" # XDD Tests echo "deleting old $XDDTARGET..." rm $XDDTARGET echo "creating new $XDDTARGET..." dd if=/dev/zero of=$XDDTARGET bs=1M count=65536 sync ; sleep 5 for QD in 1 2 4 8 16 32 64 128 256; do sync ; sleep 5 $XDD -verbose -op read -target S0 -blocksize 512 -reqsize 8 -mbytes 16384 -passes 5 -dio -seek random -seek range 128000000 -queuedepth $QD > $TMP/xdd-$CONTROLLER-$RAID-$DISKS-$DRIVE-$SS-$FS-READ-QD$QD.txt sync ; sleep 5 $XDD -verbose -op write -target S0 -blocksize 512 -reqsize 8 -mbytes 16384 -passes 5 -dio -seek random -seek range 128000000 -queuedepth $QD > $TMP/xdd-$CONTROLLER-$RAID-$DISKS-$DRIVE-$SS-$FS-WRITE-QD$QD.txt done echo "deleting old $XDDTARGET..." rm $XDDTARGET

Also have you opened up a query to areca yet on this to see if there is a driver limit at all?

Quote:

Originally Posted by stevecs

Yes I was referring to your bm-flash header line. I just grabbed it here and haven't used it before, did you just grab this out of the blue to test or does this actually mimic your workload?

We more or less grabbed it out of the blue. There is some justification for using it, but its thin. The thing is that we're obviously not professional testers or benchmarkers. In normal live I'm a Java (lead) developer. Most other tools we tried were quite cumbersome to setup, while bm-flash immediately gives results in understandable figures (IOPS and MB/s). We spend a lot of time on getting IOMeter to work on Linux, but it just doesn't work. The so-called GRUNT thread always hangs. Always...

Quote:

and have you baselined it before on other arrays to verify it's accuracy?

We did use it on another array (another server having a 1680 and 4 Mtron 7000's) but that's not really helping to test the accuracy.

Quote:

You can try increasing your nr_requests (try 512 or 1024) to see if it has an effect.

We did indeed try that, but this didn't made any difference.

Quote:

fast script to help run it.

Okay, I'm running the script and XDD (had to change -target S0 to -target $XDDTARGET ;)). Although it's still running, these are the first results:

Code:

IOIOIOIOIOIOIOIOIOIOI XDD version 6.5.013007.0001 IOIOIOIOIOIOIOIOIOIOIOI xdd - I/O Performance Inc. Copyright 1992-2007 Starting time for this run, Tue Mar 10 16:42:09 2009 ID for this run, 'No ID Specified' Maximum Process Priority, disabled Passes, 5 Pass Delay in seconds, 0 Maximum Error Threshold, 0 Target Offset, 0 I/O Synchronization, 0 Total run-time limit in seconds, 0 Output file name, stdout CSV output file name, Error output file name, stderr Pass seek randomization, disabled File write synchronization, disabled Pass synchronization barriers, enabled Number of Targets, 1 Number of I/O Threads, 1 Computer Name, mrhpgdb2, User Name, henk.dewit OS release and version, Linux 2.6.26-1-amd64 #1 SMP Sat Jan 10 17:57:00 UTC 2009 Machine hardware type, x86_64 Number of processors on this system, 1 Page size in bytes, 4096 Number of physical pages, 8255301 Megabytes of physical memory, 32247 Seconds before starting, 0 Target[0] Q[0], /ssd/S0 Target directory, "./" Process ID, 18681 Thread ID, -169722992 Processor, all/any Read/write ratio, 100.00, 0.00 Throttle in MB/sec, 0.00 Per-pass time limit in seconds, 0 Blocksize in bytes, 512 Request size, 8, blocks, 4096, bytes Number of Requests, 4194304 Start offset, 0 Number of MegaBytes, 16384 Pass Offset in blocks, 0 I/O memory buffer is a normal memory buffer I/O memory buffer alignment in bytes, 4096 Data pattern in buffer, '0x00' Data buffer verification is disabled. Direct I/O, enabled Seek pattern, random Seek range, 128000000 Preallocation, 0 Queue Depth, 1 Timestamping, disabled Delete file, disabled T Q Bytes Ops Time Rate IOPS Latency %CPU OP_Type ReqSize ^MTARGET PASS0001 0 1 17179869184 4194304 557.624 30.809 7521.75 0.0001 0.00 read 4096 ^MTARGET PASS0002 0 1 17179869184 4194304 558.737 30.748 7506.76 0.0001 0.00 read 4096 ^MTARGET PASS0003 0 1 17179869184 4194304 559.249 30.720 7499.88 0.0001 0.00 read 4096 ^MTARGET PASS0004 0 1 17179869184 4194304 558.915 30.738 7504.37 0.0001 0.00 read 4096 ^MTARGET PASS0005 0 1 17179869184 4194304 559.025 30.732 7502.89 0.0001 0.00 read 4096 ^MTARGET Average 0 1 85899345920 20971520 2793.549 30.749 7507.12 0.0001 0.00 read 4096 ^M Combined 1 1 85899345920 20971520 2793.549 30.749 7507.12 0.0001 0.00 read 4096

Quote:

Also have you opened up a query to areca yet on this to see if there is a driver limit at all?

Not yet about this issue. We had a couple of other outstanding issues with them, like the fact that a hot spare is not picked up when a 1680 contains two individual RAID sets. They initially replied, asking for more information, but after that nothing but silence. I'll try to contact them about this issue too though.

Yeah, just found the arcmsr.h and it's limited to 256 commands max for the controller:
#define ARCMSR_MAX_OUTSTANDING_CMD 256

sorry about the script, I modified it when I posted it to more match what you were trying to do so there may have been a couple typos. This first batch is showing 1 thread at a time (Q=1) if you want to skip all that and just run it w/ the max 256 outstanding threads (just don't do the for loop and set it to 256) as that will flood the array. though if you did let it run it should show scalability based on queue depth which may be interesting if there is a leveling off at some point which we can map back to a bottleneck.

Quote:

Originally Posted by stevecs

Yeah, just found the arcmsr.h and it's limited to 256 commands max for the controller:
#define ARCMSR_MAX_OUTSTANDING_CMD 256

I just took a look at the source too, but good find ;)

Quote:

if you want to skip all that and just run it w/ the max 256 outstanding threads (just don't do the for loop and set it to 256) as that will flood the array.

For a quick test I shortened the loop to 128 and 256 (and just two passes). Oh, and I also compiled the code. In the previous test I just executed the binary that was already in the /bin directory. These are the results for the 128 threads test:

Code:

IOIOIOIOIOIOIOIOIOIOI XDD version 6.5.013007.0001 IOIOIOIOIOIOIOIOIOIOIOI xdd - I/O Performance Inc. Copyright 1992-2007 Starting time for this run, Tue Mar 10 18:03:26 2009 ID for this run, 'No ID Specified' Maximum Process Priority, disabled Passes, 2 Pass Delay in seconds, 0 Maximum Error Threshold, 0 Target Offset, 0 I/O Synchronization, 0 Total run-time limit in seconds, 0 Output file name, stdout CSV output file name, Error output file name, stderr Pass seek randomization, disabled File write synchronization, disabled Pass synchronization barriers, enabled Number of Targets, 1 Number of I/O Threads, 128 Computer Name, mrhpgdb2, User Name, henk.dewit OS release and version, Linux 2.6.26-1-amd64 #1 SMP Sat Jan 10 17:57:00 UTC 2009 Machine hardware type, x86_64 Number of processors on this system, 1 Page size in bytes, 4096 Number of physical pages, 8255301 Megabytes of physical memory, 32247 Seconds before starting, 0 Target[0] Q[0], /ssd/S0 Target directory, "./" Process ID, 20815 Thread ID, 1141754192 Processor, all/any Read/write ratio, 100.00, 0.00 Throttle in MB/sec, 0.00 Per-pass time limit in seconds, 0 Blocksize in bytes, 512 Request size, 8, blocks, 4096, bytes Number of Requests, 32768 Start offset, 0 Number of MegaBytes, 16384 Pass Offset in blocks, 0 I/O memory buffer is a normal memory buffer I/O memory buffer alignment in bytes, 4096 Data pattern in buffer, '0x00' Data buffer verification is disabled. Direct I/O, enabled Seek pattern, queued_interleaved Seek range, 128000000 Preallocation, 0 Queue Depth, 128 Timestamping, disabled Delete file, disabled T Q Bytes Ops Time Rate IOPS Latency %CPU OP_Type ReqSize ^MTARGET PASS0001 0 128 17179869184 4194304 76.375 224.942 54917.49 0.0000 1.42 read 4096 ^MTARGET PASS0002 0 128 17179869184 4194304 76.593 224.300 54760.70 0.0000 1.24 read 4096 ^MTARGET Average 0 128 34359738368 8388608 152.927 224.681 54853.83 0.0000 1.33 read 4096 ^M Combined 1 128 34359738368 8388608 152.927 224.681 54853.83 0.0000 1.32 read 4096 Ending time for this run, Tue Mar 10 18:06:22 2009

The tool thus gives the number 54k for IOPS, which is distinctively different from the number that bm-flash gives me. I'll try to test with the same number of threads that bm-flash uses (10 resp. 40). I do wonder about one other thing and that's the fact that XDD reports the number of cpus on the system as 1, while in fact there are 8 (or 2 physical cpus).

edit:

The write test finally completed:

Code:

IOIOIOIOIOIOIOIOIOIOI XDD version 6.5.013007.0001 IOIOIOIOIOIOIOIOIOIOIOI xdd - I/O Performance Inc. Copyright 1992-2007 Starting time for this run, Tue Mar 10 18:06:28 2009 ID for this run, 'No ID Specified' Maximum Process Priority, disabled Passes, 2 Pass Delay in seconds, 0 Maximum Error Threshold, 0 Target Offset, 0 I/O Synchronization, 0 Total run-time limit in seconds, 0 Output file name, stdout CSV output file name, Error output file name, stderr Pass seek randomization, disabled File write synchronization, disabled Pass synchronization barriers, enabled Number of Targets, 1 Number of I/O Threads, 128 Computer Name, mrhpgdb2, User Name, henk.dewit OS release and version, Linux 2.6.26-1-amd64 #1 SMP Sat Jan 10 17:57:00 UTC 2009 Machine hardware type, x86_64 Number of processors on this system, 1 Page size in bytes, 4096 Number of physical pages, 8255301 Megabytes of physical memory, 32247 Seconds before starting, 0 Target[0] Q[0], /ssd/S0 Target directory, "./" Process ID, 20951 Thread ID, 1152395600 Processor, all/any Read/write ratio, 0.00, 100.00 Throttle in MB/sec, 0.00 Per-pass time limit in seconds, 0 Blocksize in bytes, 512 Request size, 8, blocks, 4096, bytes Number of Requests, 32768 Start offset, 0 Number of MegaBytes, 16384 Pass Offset in blocks, 0 I/O memory buffer is a normal memory buffer I/O memory buffer alignment in bytes, 4096 Data pattern in buffer, '0x00' Data buffer verification is disabled. Direct I/O, enabled Seek pattern, queued_interleaved Seek range, 128000000 Preallocation, 0 Queue Depth, 128 Timestamping, disabled Delete file, disabled T Q Bytes Ops Time Rate IOPS Latency %CPU OP_Type ReqSize ^MTARGET PASS0001 0 128 17179869184 4194304 2635.499 6.519 1591.46 0.0006 0.04 write 4096

It might be more clear to paste the figures separately from the overview data:

The write figures, 128 threads:

Code:

T Q Bytes Ops Time Rate IOPS Latency %CPU OP_Type ReqSize ^MTARGET PASS0001 0 128 17179869184 4194304 2635.499 6.519 1591.46 0.0006 0.04 write 4096 ^MTARGET PASS0002 0 128 17179869184 4194304 2911.876 5.900 1440.41 0.0007 0.04 write 4096 ^MTARGET Average 0 128 34359738368 8388608 5543.046 6.199 1513.36 0.0007 0.04 write 4096 ^M Combined 1 128 34359738368 8388608 5547.000 6.194 1512.28 0.0007 0.04 write 4096

The read figures for 256 threads:

Code:

T Q Bytes Ops Time Rate IOPS Latency %CPU OP_Type ReqSize ^MTARGET PASS0001 0 256 17179869184 4194304 337.441 50.912 12429.73 0.0001 0.86 read 4096 ^MTARGET PASS0002 0 256 17179869184 4194304 76.499 224.577 54828.35 0.0000 3.09 read 4096 ^MTARGET Average 0 256 34359738368 8388608 413.915 83.012 20266.49 0.0000 1.27 read 4096 ^M Combined 1 256 34359738368 8388608 413.915 83.012 20266.49 0.0000 1.27 read 4096

There is rather large performance difference between the first and second pass. When I did this run again (10 passes), I got a more consistent picture:

Code:

T Q Bytes Ops Time Rate IOPS Latency %CPU OP_Type ReqSize TARGET PASS0001 0 256 17179869184 4194304 76.641 224.159 54726.28 0.0000 3.95 read 4096 TARGET PASS0002 0 256 17179869184 4194304 76.726 223.912 54665.97 0.0000 3.22 read 4096 TARGET PASS0003 0 256 17179869184 4194304 76.594 224.297 54760.02 0.0000 3.20 read 4096 TARGET PASS0004 0 256 17179869184 4194304 76.691 224.013 54690.69 0.0000 3.26 read 4096 TARGET PASS0005 0 256 17179869184 4194304 76.680 224.045 54698.57 0.0000 3.24 read 4096 TARGET PASS0006 0 256 17179869184 4194304 76.702 223.983 54683.23 0.0000 3.22 read 4096 TARGET PASS0007 0 256 17179869184 4194304 76.708 223.966 54679.14 0.0000 3.26 read 4096 TARGET PASS0008 0 256 17179869184 4194304 76.259 225.283 55000.76 0.0000 3.27 read 4096 TARGET PASS0009 0 256 17179869184 4194304 76.835 223.596 54588.75 0.0000 3.22 read 4096 TARGET PASS0010 0 256 17179869184 4194304 76.295 225.177 54974.74 0.0000 3.24 read 4096 TARGET Average 0 256 171798691840 41943040 765.943 224.297 54760.01 0.0000 3.31 read 4096 Combined 1 256 171798691840 41943040 766.000 224.280 54755.93 0.0000 3.29 read 4096

Finally the write figures for 256 thread. This again took quite some time to complete.

Code:

T Q Bytes Ops Time Rate IOPS Latency %CPU OP_Type ReqSize ^MTARGET PASS0001 0 256 17179869184 4194304 2615.796 6.568 1603.45 0.0006 0.10 write 4096 ^MTARGET PASS0002 0 256 17179869184 4194304 2893.969 5.936 1449.33 0.0007 0.07 write 4096 ^MTARGET Average 0 256 34359738368 8388608 5508.566 6.238 1522.83 0.0007 0.09 write 4096 ^M Combined 1 256 34359738368 8388608 5509.000 6.237 1522.71 0.0007 0.08 write 4096

Ok, saw your comment on # procs and verified that here on a quad core system as well, so there's a bug with that. I sent an e-mail out to the author, it never really cropped up before (much lower IOPS so wasn't really saturating anything anyway). Though in your case even w/ 256 commands it's taking only ~3.5-4% cpu which is not bad at all. The write tests you can ignore/stop it's interesting to note that they are very low but then again I would have assumed that due to the nature of SSD's, more of a data point, but with databases you would probably do an initial load and most traffic would be read on these anyway right?

As for your read iops, xdd has historically been pretty good to give readings close to theoretical ones for performance in 'worst-case' (ie, 100% random workloads without caching) which is generally lower than a lot of number presented on the web. (most don't go for worst-case scenarios). Anyway without going through bm_flash code to see what it was doing the delta of ~20K iops could be due to caching or workload differences.

Can you try xdd against your 2x5GB ram disks w/ lvm? w/ your bm_flash you got ~190K, I want to see if we are running into another issue w/ the software for testing.

Also you may want to try a run w/ deadline as a scheduler (fast run either w/ bm_flash or xdd) to see how that compares to noop unless you've already tried that).

What slots do you have your cards in on your system? and do you have anything else in any other slots? (ideally running just the cards and it being headless) From the supermicro site it looks like J5 & J6 are the ones you want to use as they are tied directly to the MCH at full speed.

Also just found this doing some searching: http://www.usenix.org/events/lsf08/t...cardi_SATA.pdf

Quote:

Originally Posted by stevecs

though if you did let it run it should show scalability based on queue depth which may be interesting if there is a leveling off at some point which we can map back to a bottleneck.

Ok, that's indeed an interesting thing to test. I run the test for various queue depths with 10 passes for each depth, except for the 128 one which I copied from an earlier test. These are the results:

Code:

T Q Bytes Ops Time Rate IOPS Latency %CPU OP_Type ReqSize ^MTARGET Average 0 8 171798691840 41943040 1080.618 158.982 38813.93 0.0000 0.04 read 4096 ^MTARGET Average 0 16 171798691840 41943040 863.128 199.042 48594.24 0.0000 0.12 read 4096 ^MTARGET Average 0 32 171798691840 41943040 795.252 216.031 52741.85 0.0000 0.28 read 4096 ^MTARGET Average 0 64 171798691840 41943040 773.569 222.086 54220.14 0.0000 0.61 read 4096 ^MTARGET Average 0 128 34359738368 8388608 152.927 224.681 54853.83 0.0000 1.33 read 4096 ^MTARGET Average 0 256 171798691840 41943040 765.943 224.297 54760.01 0.0000 3.31 read 4096

So 32 threads already more or less saturate the controller and beyond 64 threads there really is no gain anymore.

when you're running xdd w/ >32 threads what is your output for vmstat 5 while that's running? likewise when you do a iostat -m -x 15 (looking for averages, in vmstat to see if you have more context switches than interrupts for example or what % is in i/o wait). iostat can be interesting for service time & utilization. Though I would like to see if you're able a xdd read test against your ram disk to make sure we're not seeing a top limit in xdd here.

oh, and noticed that the request size was wrong in the script it's set to 8 (that's actually the number times your block size which is 512bytes so it's testing 4096 bytes (1 page)) not your 8K that you wanted which would be a request size of 16. Shouldn't be much of a difference in numbers based on your earlier tests but to be aware.

Quote:

Originally Posted by stevecs

when you're running xdd w/ >32 threads what is your output for vmstat 5 while that's running? likewise when you do a iostat -m -x 15 (looking for averages, in vmstat to see if you have more context switches than interrupts for example or what % is in i/o wait). iostat can be interesting for service time & utilization. Though I would like to see if you're able a xdd read test against your ram disk to make sure we're not seeing a top limit in xdd here.

I did take a look at iostat earlier. Didn't pay a lot of attention to context switches there, but I did noticed that tps as reported by iostat was nearly equal to the iops reported by bm-flash, but differed from the iops reported by xdd. E.g. if bm-flash would report 60k, than that's basically what iostat is reporting too, but when iostat says 60k, xdd says ~54k.

I do some more testing tomorrow for sure. It's now 1:15 AM in Amsterdam ;)

edit: sleep is so overrated :P so I did a quick test (8 threads, 8KiB request size) with XDD to look at iostat -m -x 15:

Code:

avg-cpu: %user %nice %system %iowait %steal %idle 0.23 0.00 4.70 24.37 0.00 70.70 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util sda 0.00 0.53 0.00 4.07 0.00 0.02 9.05 0.00 0.00 0.00 0.00 sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sda2 0.00 0.53 0.00 4.07 0.00 0.02 9.05 0.00 0.00 0.00 0.00 sdb 1607.73 0.00 16649.67 0.00 116.30 0.00 14.31 5.71 0.34 0.06 99.71 sdc 1603.60 0.00 16636.60 0.00 116.16 0.00 14.30 2.43 0.15 0.06 92.77 dm-0 0.00 0.00 0.00 4.60 0.00 0.02 8.00 0.00 0.00 0.00 0.00 dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 md0 0.00 0.00 39750.33 0.00 245.17 0.00 12.63 0.00 0.00 0.00 0.00

and a couple of lines from vmstat 5:

Code:

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 3 8 1572 16365168 48 15838276 0 0 14989 3616 131 95 0 3 85 12 0 8 1572 16364920 48 15838300 0 0 239958 0 29942 63498 0 5 70 25 0 8 1572 16364912 48 15838300 0 0 239763 0 29972 63240 0 5 70 25 2 8 1572 16364912 48 15838300 0 0 235338 0 29507 61952 0 5 71 24 2 8 1572 16364912 48 15838300 0 0 239374 0 29844 63689 0 5 70 26 1 8 1572 16364912 48 15838300 0 0 239586 0 29948 64244 0 5 69 26

Quote:

oh, and noticed that the request size was wrong in the script it's set to 8 (that's actually the number times your block size which is 512bytes so it's testing 4096 bytes (1 page)) not your 8K that you wanted which would be a request size of 16.

I did notice that, but thanks for mentioning it :) Is the block size in xdd btw only a unit for calculations, or does it actually have significance for the way load is put on the device? I.e. is block size 512 bytes and request size 16 the same as block size 1024 and request size 8, or do those differ?

That's a high context switch count compared to interrupts, but (vmstat) you are in a i/o wait state for 25% but you have idle cpu cycles. Another item is interesting w/ the xdd (don't know if you noticed) but even though your iops stayed constant after say a queue depth of 32 the cpu time increased. In regards to that (or more accurately the # of processors detected which will allow us to use multiple targets on the same array and split the load up better to have more threads) I got a response back from Tom ruwart (author of xdd) and he fixed the # processor issue actually today. he should be sending out the corrected version when he gets back to a network.

As for block size calcs, yes that's mainly for unit calcs, ultimately it doesn't really have much to do w/ the sub system (for example under enterprise drives you normally don't have 512 byte blocks (well for systems yes, but sans, or large arrays it's usually 520 or I've even seen 524 bytes/block).

If you get a chance (I know it's late there) run a test of xdd against your ram disks just for comparison. Hopefully we'll get the updated xdd shortly. Not that it really matters in the end (ie, it doesn't really matter if we use xdd or bm-flash, just that xdd seems to have more granular controls for testing). There is no real industry standard benchmark. I personally like xdd as it is closer in line (at least w/ traditional media) to match worst case iops (what would be the worse-case, in reality it's never really /that/ bad but it gives a lower limit as to how it would perform so when I give numbers to clients I'm safe).

Not to add items (there's a lot outstanding to test already) but if space (capacity) is not an issue, you /may/ also increase iops by using two raid-10's. with raid0 or even raid-5 you are limited in that you have only one copy of the data, with raid-10 you have two copies of data so your raid controller should send the requests to the least busy subsystem drive. remember when you do this though that your data stripe width is 1/2 the drives.

Quote:

Originally Posted by stevecs

The write tests you can ignore/stop it's interesting to note that they are very low but then again I would have assumed that due to the nature of SSD's, more of a data point, but with databases you would probably do an initial load and most traffic would be read on these anyway right?

That's indeed the nature of most DB loads I think. We do have a process running that continuously inserts about 12 rows per second in some large table, but that only concerns one table. Most everything else is ~90% read, ~10 write.

Quote:

Can you try xdd against your 2x5GB ram disks w/ lvm? w/ your bm_flash you got ~190K, I want to see if we are running into another issue w/ the software for testing.

I'm going to try this soon. Currently the test server is occupied by a co-worker doing some other tests on it, but I will resume my testing soon. It would also be interesting to test what the xdd results are for testing a single Areca card. 'hopefully' these numbers will be lower then what xdd gives for the two Areca cards striped. I'll surely try the deadline scheduler too.

Quote:

What slots do you have your cards in on your system? and do you have anything else in any other slots? (ideally running just the cards and it being headless) From the supermicro site it looks like J5 & J6 are the ones you want to use as they are tied directly to the MCH at full speed.

There are no other things in any of the other slots. It's a pure headless server. The slots we put them in are the ones that are indeed connected directly to the MCH. Initially we put them in the slots that were connected to the south bridge and got performance numbers that were quite a bit lower. We also tried a hybrid approach with one card in a 'north bridge' slot and one card in a south bridge, but both cards in the north bridge slots gave the best results. It thus definitely pays off at this level to put your cards in the right slot.

I was a bit surprised though that not a lot of server motherboards are available that put emphasis on fast PCIe slots. The fastest one can get seems to be 2x PCIe8 and very few to almost none of the manufacturers actually advertise this point. In contrast, desktop motherboards these days can be get with multiple fast PCIe16 slots. This is a little different from the days that desktops computers came with slower PCI 32 bit/33Mhz slots, while the servers had the 64 bit/66Mhz versions.

Yeah, I've run into that problem for all platforms/vendors. I've even seen various 'stupid admin' designs where they put say a serial card in a 64bit/133mhz slot but then put the SAN hba in a 32bit/33mhz one. The board I've go my eyes on now is the Supermicro X8DAH+ as it has two tylersburg-36D chipsets on it (only one I've found so far w/ two). For I/O it's going to be a killer if they don't castrate something else. You don't really need more than 8x PCIe v1 speeds as of yet as the cards that are available can't handle more than that anyway (IOP34x are maxed out). What you're doing is generally what is done (putting multiple cards in a system and lvming/striping across them). Here at at the datacenter we don't have any SSD's really deployed to any of the clients, the largest bank of drives we have is about 500-600 (not including sans) but even that large one is not for a single db instance and is hooked up via 4gbit fc so there are a lot of built-in bottlenecks (doesn't help matters that they bought a lower-end sun box (should have had at least a T5440 for the sparc line).

Quote:

Originally Posted by stevecs

The board I've go my eyes on now is the Supermicro X8DAH+ as it has two tylersburg-36D chipsets on it (only one I've found so far w/ two). For I/O it's going to be a killer if they don't castrate something else.

It sounds interesting, thanks for the tip.

Quote:

You don't really need more than 8x PCIe v1 speeds as of yet as the cards that are available can't handle more than that anyway (IOP34x are maxed out).

Indeed, which is something else I don't understand. There don't seem to be any faster RAID cards coming any time soon. At least, no announcements have been made by either Intel, Areca or Adaptech. Clearly many of us are hitting a wall with the speed offered by the current generation of IOPS, but there doesn't seem to be any improvement in the short term.

Quote:

Here at at the datacenter we don't have any SSD's really deployed to any of the clients, the largest bank of drives we have is about 500-600 (not including sans)

Wow, that's something nice indeed to play with :P

I did have the opportunity to run some more tests today. I started with doing tests for an 8KiB request size. The numbers turned out to be significantly lower. Just to be sure that something else hadn't changed in the system (as I mentioned, my co-worker did some tests too), I re-run all the 4KiB tests and for every number of threads (queue depth) the same results as before were reported. To be really sure I then again re-run all the 8KiB tests (taking about 1 hour), but these gave exactly the same results as the earlier run.

Here they are. This is again for 8 disks per raid controller, 2 controllers, lvm stripped, 8KiB array stripe size, NOOP scheduler, average over 10 passes and thus 8KiB request size this time. I specifically checked that none of the passes were out of range, and none were. Every pass reported nearly the exact number of IOPS.

Code:

T Q Bytes Ops Time Rate IOPS Latency %CPU OP_Type ReqSize ^MTARGET Average 0 4 171798691840 20971520 1020.919 168.278 20541.81 0.0000 0.01 read 8192 ^MTARGET Average 0 8 171798691840 20971520 702.962 244.393 29833.08 0.0000 0.03 read 8192 ^MTARGET Average 0 16 171798691840 20971520 594.507 288.977 35275.51 0.0000 0.08 read 8192 ^MTARGET Average 0 32 171798691840 20971520 560.851 306.318 37392.34 0.0000 0.18 read 8192 ^MTARGET Average 0 64 171798691840 20971520 548.917 312.978 38205.30 0.0000 0.38 read 8192 ^MTARGET Average 0 128 171798691840 20971520 545.725 314.808 38428.76 0.0000 0.82 read 8192 ^MTARGET Average 0 256 171798691840 20971520 545.344 315.028 38455.59 0.0000 2.29 read 8192

As can be seen, in this case we already almost max out for 16 threads. Going beyond that only very marginally increases the IOPS.

For completeness I also tested with block size 2KiB, although that size isn't really important for my live load:

Code:

T Q Bytes Ops Time Rate IOPS Latency %CPU OP_Type ReqSize ^MTARGET Average 0 4 171798691840 83886080 2856.671 60.139 29364.98 0.0000 0.01 read 2048 ^MTARGET Average 0 8 171798691840 83886080 1813.242 94.747 46263.03 0.0000 0.05 read 2048 ^MTARGET Average 0 16 171798691840 83886080 1395.675 123.094 60104.29 0.0000 0.16 read 2048 ^MTARGET Average 0 32 171798691840 83886080 1258.457 136.515 66657.87 0.0000 0.40 read 2048 ^MTARGET Average 0 64 171798691840 83886080 1216.341 141.242 68965.91 0.0000 0.90 read 2048 ^MTARGET Average 0 128 171798691840 83886080 1205.534 142.508 69584.15 0.0000 1.91 read 2048 ^MTARGET Average 0 256 171798691840 83886080 1217.721 141.082 68887.78 0.0000 3.90 read 2048

In this case IOPS increase until 32 threads, but very slightly decrease after 128. With 128 threads I did notice an awkward a-synchronicity in the numbers reported by iostat:

Code:

Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util sda 0.00 0.00 0.00 0.13 0.00 0.00 8.00 0.00 4.00 2.00 0.03 sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sda2 0.00 0.00 0.00 0.13 0.00 0.00 8.00 0.00 4.00 2.00 0.03 sdb 0.00 0.00 35684.80 0.00 68.09 0.00 3.91 66.00 1.85 0.03 100.00 sdc 0.00 0.00 35833.27 0.00 68.39 0.00 3.91 4.30 0.12 0.03 99.07 dm-0 0.00 0.00 0.00 0.13 0.00 0.00 8.00 0.00 4.00 2.00 0.03 dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 md0 0.00 0.00 73157.07 0.00 139.68 0.00 3.91 0.00 0.00 0.00 0.00

avgqu-sz and await for sdb is way more than 10 times higher than that for sdc. With 256 threads I saw the same things. With 64 threads the difference was there too, but a little less (almost exactly a factor 10):

Code:

avg-cpu: %user %nice %system %iowait %steal %idle 0.68 0.00 17.21 69.68 0.00 12.42 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util sda 0.00 0.00 0.00 0.20 0.00 0.00 8.00 0.00 2.67 1.33 0.03 sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sda2 0.00 0.00 0.00 0.20 0.00 0.00 8.00 0.00 2.67 1.33 0.03 sdb 0.00 0.00 35177.27 0.00 67.12 0.00 3.91 47.15 1.34 0.03 100.00 sdc 0.00 0.00 35127.20 0.00 67.04 0.00 3.91 4.17 0.12 0.03 98.75 dm-0 0.00 0.00 0.00 0.20 0.00 0.00 8.00 0.00 2.67 1.33 0.03 dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 md0 0.00 0.00 71920.27 0.00 137.31 0.00 3.91 0.00 0.00 0.00 0.00

During the 64 threads run, vmstat showed this:

Code:

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 3 40 1572 12645792 52 15683252 0 0 9715 1252 20 17 0 3 85 12 7 63 1572 12645776 52 15683252 0 0 138445 0 50310 276128 1 17 12 70 4 46 1572 12645776 52 15683252 0 0 138288 0 50262 276661 0 17 13 69 1 53 1572 12645768 52 15683252 0 0 138387 0 50337 274764 1 17 13 69 2 40 1572 12645768 52 15683252 0 0 135370 0 49442 268276 2 17 13 68 2 55 1572 12645760 52 15683252 0 0 138110 0 50192 275198 1 18 11 71 3 45 1572 12645760 52 15683252 0 0 138575 0 50218 275155 1 17 12 70 5 53 1572 12645760 52 15683252 0 0 135070 1 49743 264514 1 17 14 69

Got pulled into a project at work, been working /your/ hours for the past bit. :P

Your iostat there is very interesting with the average queue size. What do you see w/ running w/ bm_flash? I wonder if there is something up w/ xdd.

Did you run xdd against a couple ram disk partitions for comparison? also what file system do you have on here or are you testing against the raw device? if it's xfs what is your ag count & size?

Guys, I lost you 10 posts ago, but when you finally figure it out, do let us know in plain English what the issue was (or where the bottleneck is :D)
:D :D :D

Quote:

Originally Posted by stevecs

Got pulled into a project at work, been working /your/ hours for the past bit. :P

:P Got pulled into several other things too. Meanwhile my co-worker has reconfigured the machine; installed a second 1231 and rewired the drives, so now the machine physically has 6 drives attached to each 1231 and 4 drives to the 1680.

Quote:

Your iostat there is very interesting with the average queue size. What do you see w/ running w/ bm_flash? I wonder if there is something up w/ xdd.

I'll look at the average queue size for the dual card config with bm_flash when I start testing with the striped configs again. It would be interesting to see what the results are there indeed.

Currently my co-worker has configured a RAID 6 array, so I've been running some tests against that. Although it's a little difficult to compare a single RAID 6 array to two LVM striped RAID 0 arrays, I did notice some new things.

For starters, for an xdd test run on a single RAID card, the number of IOPS as reported by iostat is now nearly exactly the same as what xdd reports. For various block sizes (request sizes) it appeared that the dual RAID 0 config was quite a bit faster for block sizes 2KiB and 4 KiB, but was about the same for block size 8KiB.

Here is the output (I edited the column names a little to correspond with bm-flash)

Code:

1x6xRAID6 noop Threads Bytes Ops Time Rate IOPS Latency %CPU OP_Type BlockSize TARGET Average 4 51539607552 50331648 1324.04 38.93 38013.58 0 0.02 read 1024 TARGET Average 8 51539607552 50331648 968.65 53.21 51960.7 0 0.06 read 1024 TARGET Average 16 51539607552 50331648 929.48 55.45 54150.42 0 0.13 read 1024 TARGET Average 32 51539607552 50331648 939.8 54.84 53555.54 0 0.26 read 1024 TARGET Average 64 51539607552 50331648 944.16 54.59 53308.57 0 0.53 read 1024 TARGET Average 128 51539607552 50331648 949.7 54.27 52997.57 0 1.12 read 1024 TARGET Average 256 51539607552 50331648 949.84 54.26 52989.66 0 3.13 read 1024 TARGET Average 128 51539607552 25165824 496.72 103.76 50663.57 0 1.06 read 2048 TARGET Average 4 51539607552 12582912 430.33 119.77 29239.91 0 0.01 read 4096 TARGET Average 8 51539607552 12582912 308.72 166.94 40757.88 0 0.04 read 4096 TARGET Average 16 51539607552 12582912 267.53 192.65 47033 0 0.11 read 4096 TARGET Average 32 51539607552 12582912 268.61 191.87 46843.96 0 0.22 read 4096 TARGET Average 64 51539607552 12582912 269.98 190.9 46607.29 0 0.46 read 4096 TARGET Average 128 51539607552 12582912 271.54 189.81 46339.54 0 0.96 read 4096 TARGET Average 256 51539607552 12582912 271.48 189.85 46349.2 0 2.75 read 4096 TARGET Average 4 51539607552 6291456 275.42 187.13 22843.46 0 0.01 read 8192 TARGET Average 8 51539607552 6291456 206.27 249.86 30500.83 0 0.03 read 8192 TARGET Average 16 51539607552 6291456 177.18 290.88 35508.34 0 0.08 read 8192 TARGET Average 32 51539607552 6291456 165.4 311.61 38037.68 0 0.18 read 8192 TARGET Average 64 51539607552 6291456 162.51 317.14 38713.44 0 0.37 read 8192 TARGET Average 128 51539607552 6291456 163.14 315.92 38564.38 0 0.8 read 8192 TARGET Average 256 51539607552 6291456 163.14 315.92 38564.66 0 2.37 read 8192 TARGET Average 4 51539607552 3145728 186.27 276.69 16887.81 0 0.01 read 16384 TARGET Average 8 51539607552 3145728 154.47 333.66 20365.24 0 0.02 read 16384 TARGET Average 16 51539607552 3145728 141.71 363.7 22198.17 0 0.05 read 16384 TARGET Average 32 51539607552 3145728 136.34 378.02 23072.76 0 0.1 read 16384 TARGET Average 64 51539607552 3145728 134.09 384.36 23459.45 0 0.22 read 16384 TARGET Average 128 51539607552 3145728 133.36 386.46 23587.77 0 0.46 read 16384 TARGET Average 256 51539607552 3145728 133.36 386.47 23588.3 0 1.53 read 16384 TARGET Average 4 51539607552 1572864 135.71 379.77 11589.64 0 0.01 read 32768 TARGET Average 8 51539607552 1572864 125.95 409.22 12488.43 0 0.01 read 32768 TARGET Average 16 51539607552 1572864 122.1 422.12 12882.15 0 0.03 read 32768 TARGET Average 32 51539607552 1572864 120.48 427.77 13054.61 0 0.06 read 32768 TARGET Average 64 51539607552 1572864 119.95 429.66 13112.26 0 0.13 read 32768 TARGET Average 128 51539607552 1572864 120.01 429.47 13106.26 0 0.27 read 32768 TARGET Average 256 51539607552 1572864 120.03 429.4 13104.1 0 0.94 read 32768

As can be seen, for 8KiB the IOPS range is from 22k to 38K here, while for the dual 8xRAID 0 (16 disks total) the range was 20k to 38k, which is nearly the same. However for 2KiB it's 29k to 46k here, while the earlier dual setup gave me 38k to 54k. Puzzling...

Quote:

Did you run xdd against a couple ram disk partitions for comparison? also what file system do you have on here or are you testing against the raw device? if it's xfs what is your ag count & size?

Still haven't done the ram disk test, but hope to be able to do that soon. There's currently some write test running and it doesn't seem to be ending anytime soon :( The file system used for all tests is JFS.

That is strange. Keep an eye on your queue depth to the base arrays (when you're doing plaid) if it looks the same w/ bm_flash that you saw w/ xdd (one base array taking 10x the queue). If it's lop-sided like that it could mean that there is something mis-alligned (allocation groups et al, w/ jfs you don't really have a way to force AG alignment (what is the ag size (jfs_tune -l <device>)? or something w/ lvm. If it only happens w/ xdd then we can narrow that down to the tool. I don't really want to put into the mix oracle's orion toolset here as it looks like a much lower level problem. When I get back from the office I'll check to see what's going on w/ the patched xdd as well.

Quote:

Originally Posted by stevecs

That is strange.

I'm encountering some more strange behavior. I've been ignoring writes a little until now, but started testing this again today. A single test pass was still running after some 5(!) hours. The total amount of data to be written would have been 16 GB. IO stat showed me this:

Code:

Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn sda 276.67 0.00 0.27 0 0 sdb 0.00 0.00 0.00 0 0 sdc 0.00 0.00 0.00 0 0 sdc1 0.00 0.00 0.00 0 0 sdc2 0.00 0.00 0.00 0 0 sdd 0.00 0.00 0.00 0 0 dm-0 0.00 0.00 0.00 0 0 dm-1 0.00 0.00 0.00 0 0 dm-2 0.00 0.00 0.00 0 0

I can imagine that sometimes the onboard cache on the Areca controller might need some time to empty itself, but speeds where consistently this low for the entire duration.

I canceled the test and tested with 16MB and 128MB of data to be written for a single pass.

The first pass of the following test was bizarrely slow again, but the second pass 'suddenly' picked up speed:

Code:

T Q Bytes Ops Time Rate IOPS Latency %CPU OP_Type ReqSize TARGET PASS0001 0 32 134217728 32768 99.731 1.346 328.56 0.0030 0.02 write 4096 TARGET PASS0002 0 32 134217728 32768 3.193 42.037 10262.91 0.0001 0.09 write 4096 TARGET PASS0003 0 32 134217728 32768 1.700 78.930 19269.97 0.0001 0.16 write 4096 TARGET PASS0004 0 32 134217728 32768 1.347 99.642 24326.67 0.0000 0.25 write 4096 TARGET PASS0005 0 32 134217728 32768 3.774 35.562 8682.02 0.0001 0.09 write 4096 TARGET Average 0 32 671088640 163840 108.036 6.212 1516.53 0.0007 0.02 write 4096 Combined 1 32 671088640 163840 109.000 6.157 1503.12 0.0007 0.01 write 4096

Maybe the controller happened to have a back log of writes to be done to the actual disks and it just happened that after the first pass the cache was available to accept new data again at a decent speed, but I don't think the cache should work that way, does it?

I'm running the script you gave me with one loop added, for the block size, and now the array seems to be writing at a rather normal speed again although a lot of time the figures between passes do differ greatly. E.g. :

Code:

T Q Bytes Ops Time Rate IOPS Latency %CPU OP_Type ReqSize ^MTARGET PASS0001 0 128 134217728 8192 3.236 41.474 2531.35 0.0004 5.74 write 16384 ^MTARGET PASS0002 0 128 134217728 8192 2.080 64.541 3939.27 0.0003 0.80 write 16384 ^MTARGET PASS0003 0 128 134217728 8192 0.429 312.631 19081.47 0.0001 0.81 write 16384 ^MTARGET PASS0004 0 128 134217728 8192 0.402 333.493 20354.77 0.0000 0.89 write 16384 ^MTARGET PASS0005 0 128 134217728 8192 0.372 360.663 22013.10 0.0000 1.27 write 16384 ^MTARGET Average 0 128 671088640 40960 6.455 103.968 6345.68 0.0002 0.80 write 16384 ^M Combined 1 128 671088640 40960 6.455 103.968 6345.68 0.0002 0.49 write 16384

This looks a lot like the 'gaps' I've seen in the results given by bm-flash. Namely, bm-flash doesn't test a total fixed amount of megabytes, but tests for some time and when that time elapses it moves on to the next test. I can't really wrap my head around this. I tried the 8Kib request size a couple of 100 times, and it was consistently fast. I won't dump the hundreds of lines of results of that here, but this is a snapshot from it:

Code:

TARGET PASS0023 0 32 134217728 16384 0.597 224.723 27431.99 0.0000 0.25 write 8192 TARGET PASS0024 0 32 134217728 16384 0.602 223.133 27237.94 0.0000 0.36 write 8192 TARGET PASS0025 0 32 134217728 16384 0.604 222.281 27133.87 0.0000 0.21 write 8192 TARGET PASS0026 0 32 134217728 16384 0.597 224.898 27453.31 0.0000 0.24 write 8192 TARGET PASS0027 0 32 134217728 16384 0.599 224.087 27354.35 0.0000 0.31 write 8192 TARGET PASS0028 0 32 134217728 16384 0.608 220.632 26932.66 0.0000 0.25 write 8192 TARGET PASS0029 0 32 134217728 16384 0.598 224.550 27410.87 0.0000 0.38 write 8192 TARGET PASS0030 0 32 134217728 16384 0.588 228.214 27858.21 0.0000 0.27 write 8192 TARGET PASS0031 0 32 134217728 16384 0.600 223.534 27286.88 0.0000 0.23 write 8192 TARGET PASS0032 0 32 134217728 16384 0.598 224.319 27382.65 0.0000 0.33 write 8192 TARGET PASS0033 0 32 134217728 16384 0.611 219.798 26830.78 0.0000 0.28 write 8192 TARGET PASS0034 0 32 134217728 16384 0.606 221.305 27014.77 0.0000 0.24 write 8192 TARGET PASS0035 0 32 134217728 16384 0.607 221.042 26982.61 0.0000 0.45 write 8192 TARGET PASS0036 0 32 134217728 16384 0.593 226.310 27625.74 0.0000 0.38 write 8192 TARGET PASS0037 0 32 134217728 16384 0.587 228.826 27932.83 0.0000 0.30 write 8192 TARGET PASS0038 0 32 134217728 16384 0.601 223.478 27280.02 0.0000 0.25 write 8192 TARGET PASS0039 0 32 134217728 16384 0.595 225.523 27529.70 0.0000 0.25 write 8192 TARGET PASS0040 0 32 134217728 16384 0.597 224.821 27443.98 0.0000 0.32 write 8192 TARGET PASS0041 0 32 134217728 16384 0.597 224.741 27434.24 0.0000 0.32 write 8192 TARGET PASS0042 0 32 134217728 16384 0.605 221.809 27076.29 0.0000 0.21 write 8192 TARGET PASS0043 0 32 134217728 16384 0.603 222.421 27151.00 0.0000 0.38 write 8192 TARGET PASS0044 0 32 134217728 16384 0.602 222.932 27213.32 0.0000 0.29 write 8192 TARGET PASS0045 0 32 134217728 16384 0.606 221.631 27054.61 0.0000 0.29 write 8192 TARGET PASS0046 0 32 134217728 16384 0.591 227.098 27721.89 0.0000 0.45 write 8192 TARGET PASS0047 0 32 134217728 16384 0.599 223.944 27336.92 0.0000 0.39 write 8192

Then I switched to the 2KiB request size again, and things slowed down again:

Code:

TARGET PASS0001 0 32 134217728 65536 7.288 18.415 8991.82 0.0001 0.13 write 2048 TARGET PASS0002 0 32 134217728 65536 7.002 19.169 9359.76 0.0001 0.06 write 2048 TARGET PASS0003 0 32 134217728 65536 2.404 55.823 27257.08 0.0000 0.25 write 2048 TARGET PASS0004 0 32 134217728 65536 31.388 4.276 2087.93 0.0005 0.02 write 2048 TARGET PASS0005 0 32 134217728 65536 76.843 1.747 852.86 0.0012 0.01 write 2048 TARGET PASS0006 0 32 134217728 65536 75.700 1.773 865.73 0.0012 0.01 write 2048 TARGET PASS0007 0 32 134217728 65536 42.965 3.124 1525.34 0.0007 0.01 write 2048

Thereafter I switched to 8KiB again, it first started slow and then was fast again:

Code:

TARGET PASS0001 0 32 134217728 16384 20.080 6.684 815.95 0.0012 0.03 write 8192 TARGET PASS0002 0 32 134217728 16384 3.438 39.035 4765.01 0.0002 0.18 write 8192 TARGET PASS0003 0 32 134217728 16384 0.662 202.631 24735.23 0.0000 0.37 write 8192 TARGET PASS0004 0 32 134217728 16384 0.626 214.401 26171.94 0.0000 0.24 write 8192 TARGET PASS0005 0 32 134217728 16384 0.603 222.636 27177.26 0.0000 0.42 write 8192 TARGET PASS0006 0 32 134217728 16384 0.608 220.797 26952.78 0.0000 0.33 write 8192 TARGET PASS0007 0 32 134217728 16384 0.603 222.688 27183.57 0.0000 0.23 write 8192 TARGET PASS0008 0 32 134217728 16384 0.603 222.475 27157.57 0.0000 0.27 write 8192 TARGET PASS0009 0 32 134217728 16384 0.616 218.035 26615.59 0.0000 0.21 write 8192

It almost looks like the alternation in writing between block sizes causes some serious (internal) 'trashing' effect or so to take place. I'm not sure yet though...

When I start the test for writing a total of 1024 MB, it also starts slow. This creeps me out a little. How does the RAID controller in advance 'know' that 1024 MB is going to be written. I take it that xdd writes this 1024 MB totally at random in the 64 GB file on which it operates. This is the result for writing 1024 MB total, which is way different from writing 128 MB a couple of hundred times right after each other:

Code:

TARGET PASS0001 0 32 1073741824 131072 352.669 3.045 371.66 0.0027 0.00 write 8192 TARGET PASS0002 0 32 1073741824 131072 19.656 54.627 6668.38 0.0001 0.06 write 8192 TARGET PASS0003 0 32 1073741824 131072 20.078 53.478 6528.10 0.0002 0.05 write 8192 TARGET PASS0004 0 32 1073741824 131072 26.033 41.246 5034.89 0.0002 0.03 write 8192 TARGET PASS0005 0 32 1073741824 131072 21.058 50.990 6224.40 0.0002 0.04 write 8192 TARGET PASS0006 0 32 1073741824 131072 21.431 50.103 6116.08 0.0002 0.04 write 8192 TARGET PASS0007 0 32 1073741824 131072 21.174 50.712 6190.37 0.0002 0.04 write 8192 TARGET PASS0008 0 32 1073741824 131072 30.574 35.119 4287.02 0.0002 0.03 write 8192 TARGET PASS0009 0 32 1073741824 131072 21.966 48.882 5966.98 0.0002 0.03 write 8192 TARGET PASS0010 0 32 1073741824 131072 19.553 54.916 6703.58 0.0001 0.05 write 8192 TARGET Average 0 32 10737418240 1310720 542.097 19.807 2417.87 0.0004 0.01 write 8192 Combined 1 32 10737418240 1310720 554.000 19.382 2365.92 0.0004 0.01 write 8192

(p.s. I hope it doesn't bother anyone that I'm 'dumping' so many figures here)

I wonder if we're seeing here some kind of profiling or wear leveling of the ssd's themselves with that result. The controller doesn't really have a 'backlog', it does have cache yes and will have some items that it needs to write out to the disks but it's not that large in/of itself nor would it take anywhere near that long unless the drive itself told it to hold off (taking too long to write or cell updates et al). To rule it out completely (ie, see if it's the SSD) just turn off the write cache or remove the ram from the controller (you have to reboot for it to take effect). you would be seeing the real speeds of the drives at that point.

Also we can put in a delay between passes w/ xdd something large enough to handle any type of outstanding i/o like 60 secs or whatever. (-delay <seconds>) which happens between passes which should clear out any delayed writes. also (should be doing this automatically as jfs does not have barrier support) but you could try -nobarrier w/ xdd as well to let all threads have free reign.

I've looked at the jfs layout specs (http://jfs.sourceforge.net/project/pub/jfslayout.pdf ) and don't really see anything here that would really (except by chance) be only on one disk opposed to another. with exception of the built-in journal. You may want to put the journal (128MB max) on another physical disk not the ssd array, could be another stand-alone ssd or whatever).

Code:

mkfs.jfs -J journal_dev /dev/external_journal # creates a journal on device /dev/external_journal mkfs.jfs -J device=/dev/external_journal /dev/jfs_device # attaches the external journal to the existing file system on /dev/jfs_device or mkfs.jfs -j /dev/external_journal /dev/jfs_device # if you're creating the array filesystem at the same time as you create the journal

did you ever send an e-mail over to areca to ask about any other types of limits in the driver?

Quote:

Originally Posted by stevecs

Did you run xdd against a couple ram disk partitions for comparison?

I now have some results from the xdd against the ram disks. I created a very basic LVM striped device, using:

Code:

pvcreate /dev/ram0 /dev/ram1 vgcreate lvram /dev/ram0 /dev/ram1 lvcreate -i2 -I128 -L5G -n ramdisk lvram mkfs.jfs /dev/lvram/ramdisk mount /dev/lvram/ramdisk /ramdiskstr/ -o noatime,nodiratime

I let xdd work on a 4GB file, testing an amount of data equal to ~16GB per pass:

Code:

dd if=/dev/zero of=$XDDTARGET bs=1M count=4096 sync ; sleep 5 $XDD -verbose -op read -target ${XDDTARGET} -blocksize 512 -reqsize $RQ -mbytes 16384 -passes 5 -dio -seek random -seek range 4000000 -queuedepth $QD

The results are this:

Code:

T Q Bytes Ops Time Rate IOPS Latency %CPU OP_Type ReqSize ^MTARGET PASS0001 0 128 17179869184 2097152 17.714 969.824 118386.69 0.0000 2.73 read 8192 ^MTARGET PASS0002 0 128 17179869184 2097152 18.182 944.893 115343.37 0.0000 2.16 read 8192 ^MTARGET PASS0003 0 128 17179869184 2097152 18.651 921.124 112441.89 0.0000 2.21 read 8192 ^MTARGET PASS0004 0 128 17179869184 2097152 18.490 929.159 113422.79 0.0000 2.20 read 8192 ^MTARGET PASS0005 0 128 17179869184 2097152 18.171 945.431 115409.08 0.0000 2.28 read 8192 ^MTARGET Average 0 128 85899345920 10485760 87.133 985.837 120341.44 0.0000 2.14 read 8192

While iostat -mx dm-3 ram0 ram1 15 showed this:

Code:

avg-cpu: %user %nice %system %iowait %steal %idle 0.62 0.00 14.41 0.00 0.00 84.97 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util ram0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 ram1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-3 0.00 0.00 115517.87 0.13 902.48 0.00 16.00 0.38 0.00 0.00 37.97

I do wonder why there's no traffic reports for ram0 and ram1, but I guess this is because of the way these device are implemented.

For comparison, the same xdd test on a single (non lvm) ram disk, with an ext2 fs (a jsf formatted ram disk couldn't be mounted on my system):

Code:

mke2fs -m 0 /dev/ram3 /dev/ram3 /ramdisksing/

gave the following results:

Code:

T Q Bytes Ops Time Rate IOPS Latency %CPU OP_Type ReqSize ^MTARGET PASS0001 0 128 17179869184 2097152 17.537 979.620 119582.57 0.0000 3.06 read 8192 ^MTARGET PASS0002 0 128 17179869184 2097152 18.119 948.153 115741.34 0.0000 2.26 read 8192 ^MTARGET PASS0003 0 128 17179869184 2097152 18.365 935.447 114190.29 0.0000 2.39 read 8192 ^MTARGET PASS0004 0 128 17179869184 2097152 17.933 957.981 116941.10 0.0000 2.27 read 8192 ^MTARGET PASS0005 0 128 17179869184 2097152 18.004 954.229 116483.08 0.0000 2.21 read 8192 ^MTARGET Average 0 128 85899345920 10485760 85.710 1002.204 122339.31 0.0000 2.20 read 8192

On earlier tests with striped and single ram disks, bm-flash reported 190k resp. 200k. My colleague set up the RAM disks then so I do have to ask how he exactly did that. In xdd however there seems to be no difference in performance. Since /dev/ram3 didn't show up in iostat there's nothing to report there.

I'll look at the performance for two RAID6 arrays on two 1231s next and will try to set the journal on a complete different disk (there is 1680 in the machine with 4 SSDs, I'll try to put the journal there, alternatively there are also 2 SAS HDDs on that 1680 that could it)

Ok, the xdd against ram is good as it shows it can calculate more than the iops we were seeing. I do not like that the averages of xdd are incorrect from your posting (iops, the average is higher than the individual passes, which unless there is some type of new math employed, is wrong. :) )

As for ram* not showing up and for not being able to mount a jfs ramdisk yes there are bugs in both functions under linux. I've noticed that before but didn't care enough to send out an e-mail at the time to the lists.

Move the journal to another ssd or spindle, max size is 128M so it doesn't need to be large (default when in-line is 32M which is what you have now) so if you want to rule that out completely you could try putting it on a 64M ramX disk (not good for production purposes obviously but gets it out of the way right now and doesn't introduce another delay point as we've tested that ram is > than your goal).

I probably wouldn't play too much w/ raid-6 here as that would be worse than raid-5 for iops even if you're doing just 10% writes. Seriously I would look at doing either a bunch of raid-1's and LVMing them at the OS level or RAID-10 on the arecas. (using even number disks you don't want raid-1E). And remember your offset. With a 'stripe size' under raid-10 your 'data disks' would be 1/2 of the drives in your array. so with 2x6 that would be 3 drives each. so with 8KiB stripe size that would be 24KiB so the lvm default of 192KiB is fine.

How to combine 2 Areca´s? (please step by step)

@FEAR. It's hard to give a step by step for all situations as you've probably seen even in this thread there are a lot of variables and they have an effect on the setup of a system. As a high-level simple process something like this for linux:

Create an array on each areca controller

Simple method here would be to have areca arrays of the same size
Your OS should see XX physical discs

where XX is the number of controllers in your system (max 4 for areca) you can do multiple arrays on a single controller but won't go into that, it's the same basic principle though.
Under linux you have two basic options
[INDENT]
mdadm (meta disk driver)

don't create any partition tables on the areca arrays (ie. /dev/sda only, no /dev/sda1 or whatever)
mdadm --create --help (displays the create help menu)
mdadm --create /dev/md0 --raid-level=0 --raid-devices=XX --chunk=XX </dev/arecaraid0> </dev/arecaraid1> </dev/arecaraidX>

set your chunk size to be a multiple of your areca data stripe width (array size minus parity disks TIMES your stripe size) if possible (best performance for writes) or at least equal the size of your areca stripe size.
put your file system of choice on your new meta-device, EXT3 caps at 8TiB (forget what others on the net have said, if you're going beyond this or think you may don't use EXT3, use JFS or XFS). XFS be careful if you have any type of power outages or system freezes as you have a higher chance to loose data.
mount and use your filesystem
If you set your chunk size to be your data stripe width you will have some pain in growing your file system if you add drives to the areca disk (your data stripe width would change and you can't change that at the mdadm level without a reformat). However you could change the size of your drives (ie, 500GB to 1TB or 1TB to 2TB and keep the same number of drives but change their sizes in which case your data stripe width remains constant. If you use a chunk size = your areca stripe size you can grow the array easier but you loose some write performance (assuming parity raids here). To grow you would:

add/replace drives to your areca controller
If you added more drives, expand your raidset
then modify your volumeset and you have the option to change the volumeset size this will keep your current data in place but just make the 'disk' larger
reboot so that linux sees the new areca volume sizes
mdadm --grow /dev/md0 (grows your metadisk array at the OS level)
Then grow the file system (for jfs: mount -o remount,resize </mntpoint> ; for xfs xfs_growfs </mntpoint> ) you do this with the file system mounted
you're done
LVM (Logical volume manager)

don't create any partition tables on the areca arrays (ie. /dev/sda only, no /dev/sda1 or whatever)
calculate out your starting offset. LVM uses by default 192KiB for meta data at the beginning of each physical volume (your areca arrays here). You want to find an offset that is the smallest multiple of your areca data stripe width * stripe size and is greater than or equal to 192KiB (LVM meta data). For example a raid-6 array of 10 drives with a 64KiB stripe size is 10-2 = 8 data drives 8*64 = 512KiB which is > 192KiB so you have to pad your lvm starting point to be at 512KiB.
pvcreate --metadatasize 511K --metadatacopies=2 <arecaraid0> <arecaraid1> <arecaraidX>
pvscan -v (shows volumes that were added)
pvs -o+pe_start (shows starting offset of volumes, this should be 512KiB in above example if not re-create with different offset, LVM 'padds' to a 64KiB boundary so it's a little fuzzy)
vgcreate --physicalextentsize X <volumegroupname> </dev/arecaraid0> </dev/arecaraid1> </dev/arecaraidX> (pick an extent size so that it's reasonable anticipating future growth. You're not limited to 65535 w/ LVM2, default size is 4MiB. I set mine to 1GiB in size as I plan to grow to 100TiB or so with the current array, and will probably move to 4GiB when I rebuild at that point)
lvcreate --stripes X --stripesize X --extents X --name <lvname> <vgname> </dev/arecaraid0> </dev/arecaraid1> </dev/arecaraidX> (--stripes here is the number of physical volumes to stripe across (raid cards/areca arrays); --stripesize needs to be 2^n and best case should be the data stripe width of your areca array assuming it 2^n, otherwise should be your base array stripe size; --extents is how big you want your logical volume to be. You list the areca physical volumes to indicate to lvm what underlaying devices it should pull extents from for your stripe, more important if you don't want to stripe across all physical disks for example, but it's good practice to enumerate what you want to happen explicitly)
put your file system of choice on your new meta-device, EXT3 caps at 8TiB (forget what others on the net have said, if you're going beyond this or think you may don't use EXT3, use JFS or XFS). XFS be careful if you have any type of power outages or system freezes as you have a higher chance to loose data.
mount and use your filesystem
If you set your chunk size to be your data stripe width you will have some pain in growing your file system if you add drives to the areca disk (your data stripe width would change and you can't change that at the mdadm level without a reformat). However you could change the size of your drives (ie, 500GB to 1TB or 1TB to 2TB and keep the same number of drives but change their sizes in which case your data stripe width remains constant. If you use a chunk size = your areca stripe size you can grow the array easier but you loose some write performance (assuming parity raids here). To grow you would:

add/replace drives to your areca controller
If you added more drives, expand your raidset
then modify your volumeset and you have the option to change the volumeset size this will keep your current data in place but just make the 'disk' larger
reboot so that linux sees the new areca volume sizes
pvresize -v -d </dev/arecaraidX> (do this for each array that you expanded on teh areca's this tells lvm to re-check the physical disk)
pvscan -v
vgdisplay <vgname>
lvresize --extents X </dev/logicalvolume> (this expands your logical volume by X extents)
Then grow the file system (for jfs: mount -o remount,resize </mntpoint> ; for xfs xfs_growfs </mntpoint> ) you do this with the file system mounted
you're done

Ok, a little long winded I guess but that's an overview.

Puh - english is not my element :eh:

Thx ;)

@stevecs that's a handy explanation. People are probably going to reference that forum post for some time ;)

Last week while residing on a remote location I locked myself out of the machine I'm testing when giving the reboot command. Something to do with the enumeration of the PCIe devices and the machine going to maintenance mode after a reboot :( Took some time to find a fix for this (using udev).

Anyway, my co-worker did some interesting further testing. He created a 2x6xRAID6@1231 config with stripe sizes for the array and lvm of respectively 8KiB/32Kib and 128KiB/512Kib. The results are rather interesting, as this is a more apples-to-apples comparison of the single controller vs the dual controller (the previous comparison I posted was a 1x6xRAID6@1231 vs 2x8xRAID0@1231+1680). Without further ado, these are the results:

2x6xRAID6 2x1231ML stripe size 8k, lvm2 stripe size 32k

Code:

T Q Bytes Ops Time Rate IOPS Latency %CPU OP_Type BlockSize TARGET Average 0 4 51539607552 50331648 1222.791 42.149 41161.29 0.0000 0.02 read 1024 TARGET Average 0 8 51539607552 50331648 798.801 64.521 63009.03 0.0000 0.09 read 1024 TARGET Average 0 16 51539607552 50331648 777.554 66.284 64730.74 0.0000 0.23 read 1024 TARGET Average 0 32 51539607552 50331648 783.667 65.767 64225.83 0.0000 0.46 read 1024 TARGET Average 0 64 51539607552 50331648 790.628 65.188 63660.31 0.0000 0.93 read 1024 TARGET Average 0 128 51539607552 50331648 800.083 64.418 62908.07 0.0000 1.90 read 1024 TARGET Average 0 256 51539607552 50331648 809.726 63.651 62158.89 0.0000 3.94 read 1024 TARGET Average 0 4 51539607552 25165824 667.203 77.247 37718.41 0.0000 0.02 read 2048 TARGET Average 0 8 51539607552 25165824 423.184 121.790 59467.75 0.0000 0.08 read 2048 TARGET Average 0 16 51539607552 25165824 395.697 130.250 63598.68 0.0000 0.23 read 2048 TARGET Average 0 32 51539607552 25165824 397.812 129.558 63260.62 0.0000 0.46 read 2048 TARGET Average 0 64 51539607552 25165824 401.566 128.347 62669.24 0.0000 0.93 read 2048 TARGET Average 0 128 51539607552 25165824 404.788 127.325 62170.33 0.0000 1.91 read 2048 TARGET Average 0 256 51539607552 25165824 408.990 126.017 61531.60 0.0000 3.94 read 2048 TARGET Average 0 4 51539607552 12582912 385.204 133.798 32665.58 0.0000 0.02 read 4096 TARGET Average 0 8 51539607552 12582912 240.319 214.463 52359.18 0.0000 0.07 read 4096 TARGET Average 0 16 51539607552 12582912 202.105 255.014 62259.27 0.0000 0.22 read 4096 TARGET Average 0 32 51539607552 12582912 203.047 253.831 61970.52 0.0000 0.46 read 4096 TARGET Average 0 64 51539607552 12582912 204.619 251.881 61494.38 0.0000 0.92 read 4096 TARGET Average 0 128 51539607552 12582912 207.224 248.714 60721.23 0.0000 1.90 read 4096 TARGET Average 0 256 51539607552 12582912 208.365 247.353 60388.91 0.0000 3.95 read 4096 TARGET Average 0 4 51539607552 6291456 237.350 217.146 26507.08 0.0000 0.01 read 8192 TARGET Average 0 8 51539607552 6291456 150.599 342.230 41776.13 0.0000 0.06 read 8192 TARGET Average 0 16 51539607552 6291456 113.680 453.374 55343.46 0.0000 0.19 read 8192 TARGET Average 0 32 51539607552 6291456 109.713 469.768 57344.75 0.0000 0.45 read 8192 TARGET Average 0 64 51539607552 6291456 110.428 466.725 56973.29 0.0000 0.92 read 8192 TARGET Average 0 128 51539607552 6291456 111.486 462.295 56432.49 0.0000 1.88 read 8192 TARGET Average 0 256 51539607552 6291456 112.282 459.018 56032.53 0.0000 3.94 read 8192 TARGET Average 0 4 51539607552 3145728 147.691 348.969 21299.38 0.0000 0.01 read 16384 TARGET Average 0 8 51539607552 3145728 101.245 509.059 31070.51 0.0000 0.04 read 16384 TARGET Average 0 16 51539607552 3145728 80.464 640.532 39094.97 0.0000 0.13 read 16384 TARGET Average 0 32 51539607552 3145728 71.336 722.494 44097.56 0.0000 0.33 read 16384 TARGET Average 0 64 51539607552 3145728 67.403 764.651 46670.57 0.0000 0.78 read 16384 TARGET Average 0 128 51539607552 3145728 65.747 783.905 47845.74 0.0000 1.72 read 16384 TARGET Average 0 256 51539607552 3145728 65.457 787.383 48058.06 0.0000 3.80 read 16384 TARGET Average 0 4 51539607552 1572864 96.325 535.060 16328.73 0.0001 0.01 read 32768 TARGET Average 0 8 51539607552 1572864 74.608 690.802 21081.60 0.0000 0.04 read 32768 TARGET Average 0 16 51539607552 1572864 64.893 794.225 24237.83 0.0000 0.09 read 32768 TARGET Average 0 32 51539607552 1572864 60.607 850.391 25951.87 0.0000 0.20 read 32768 TARGET Average 0 64 51539607552 1572864 58.769 876.980 26763.31 0.0000 0.45 read 32768 TARGET Average 0 128 51539607552 1572864 58.051 887.829 27094.39 0.0000 0.95 read 32768 TARGET Average 0 256 51539607552 1572864 57.957 889.272 27138.42 0.0000 2.54 read 32768

2x6xRAID6 2x1231ML stripe size 128k, lvm2 stripe size 512k

Code:

TARGET Average 0 4 51539607552 50331648 1220.183 42.239 41249.26 0.0000 0.02 read 1024 TARGET Average 0 8 51539607552 50331648 792.691 65.019 63494.66 0.0000 0.09 read 1024 TARGET Average 0 16 51539607552 50331648 769.496 66.978 65408.55 0.0000 0.23 read 1024 TARGET Average 0 32 51539607552 50331648 778.130 66.235 64682.83 0.0000 0.46 read 1024 TARGET Average 0 64 51539607552 50331648 786.188 65.556 64019.84 0.0000 0.93 read 1024 TARGET Average 0 128 51539607552 50331648 794.469 64.873 63352.55 0.0000 1.91 read 1024 TARGET Average 0 256 51539607552 50331648 801.456 64.307 62800.25 0.0000 3.97 read 1024 TARGET Average 0 4 51539607552 25165824 661.605 77.901 38037.56 0.0000 0.02 read 2048 TARGET Average 0 8 51539607552 25165824 413.202 124.732 60904.37 0.0000 0.09 read 2048 TARGET Average 0 16 51539607552 25165824 387.614 132.966 64924.99 0.0000 0.23 read 2048 TARGET Average 0 32 51539607552 25165824 390.090 132.122 64512.78 0.0000 0.46 read 2048 TARGET Average 0 64 51539607552 25165824 393.227 131.068 63998.13 0.0000 0.94 read 2048 TARGET Average 0 128 51539607552 25165824 397.461 129.672 63316.47 0.0000 1.90 read 2048 TARGET Average 0 256 51539607552 25165824 401.310 128.428 62709.14 0.0000 3.97 read 2048 TARGET Average 0 4 51539607552 12582912 384.675 133.982 32710.49 0.0000 0.02 read 4096 TARGET Average 0 8 51539607552 12582912 230.493 223.606 54591.28 0.0000 0.07 read 4096 TARGET Average 0 16 51539607552 12582912 194.767 264.622 64605.03 0.0000 0.23 read 4096 TARGET Average 0 32 51539607552 12582912 195.312 263.884 64424.74 0.0000 0.46 read 4096 TARGET Average 0 64 51539607552 12582912 197.903 260.428 63581.05 0.0000 0.94 read 4096 TARGET Average 0 128 51539607552 12582912 199.588 258.229 63044.28 0.0000 1.93 read 4096 TARGET Average 0 256 51539607552 12582912 201.143 256.234 62557.02 0.0000 4.00 read 4096 TARGET Average 0 4 51539607552 6291456 246.272 209.279 25546.79 0.0000 0.01 read 8192 TARGET Average 0 8 51539607552 6291456 144.297 357.178 43600.85 0.0000 0.05 read 8192 TARGET Average 0 16 51539607552 6291456 102.134 504.626 61599.88 0.0000 0.21 read 8192 TARGET Average 0 32 51539607552 6291456 100.701 511.809 62476.71 0.0000 0.46 read 8192 TARGET Average 0 64 51539607552 6291456 101.461 507.975 62008.70 0.0000 0.94 read 8192 TARGET Average 0 128 51539607552 6291456 102.220 504.200 61547.91 0.0000 1.93 read 8192 TARGET Average 0 256 51539607552 6291456 103.033 500.223 61062.36 0.0000 4.06 read 8192 TARGET Average 0 4 51539607552 3145728 175.645 293.431 17909.60 0.0001 0.01 read 16384 TARGET Average 0 8 51539607552 3145728 102.537 502.644 30678.98 0.0000 0.04 read 16384 TARGET Average 0 16 51539607552 3145728 68.935 747.656 45633.33 0.0000 0.13 read 16384 TARGET Average 0 32 51539607552 3145728 55.238 933.050 56948.82 0.0000 0.41 read 16384 TARGET Average 0 64 51539607552 3145728 53.488 963.582 58812.37 0.0000 0.95 read 16384 TARGET Average 0 128 51539607552 3145728 53.993 954.559 58261.68 0.0000 1.93 read 16384 TARGET Average 0 256 51539607552 3145728 53.813 957.749 58456.38 0.0000 4.07 read 16384 TARGET Average 0 4 51539607552 1572864 137.397 375.113 11447.55 0.0001 0.01 read 32768 TARGET Average 0 8 51539607552 1572864 81.315 633.823 19342.73 0.0001 0.02 read 32768 TARGET Average 0 16 51539607552 1572864 55.840 922.982 28167.16 0.0000 0.08 read 32768 TARGET Average 0 32 51539607552 1572864 45.307 1137.554 34715.39 0.0000 0.21 read 32768 TARGET Average 0 64 51539607552 1572864 40.859 1261.413 38495.26 0.0000 0.50 read 32768 TARGET Average 0 128 51539607552 1572864 39.018 1320.911 40311.00 0.0000 1.15 read 32768 TARGET Average 0 256 51539607552 1572864 38.446 1340.564 40910.76 0.0000 3.16 read 32768

On top of that, a test run with mdadm was done. This is (for us) just a theoretical test. Since our system administrator doesn't like mdadm, we're unable to use this in production. Nevertheless, it shows good results and might be interesting for anyone who is able/allowed to use mdadm:

2x6xRAID6 2x1231ML stripe size 128k, mdadm stripe width 512k (mdadm -Cv /dev/md0 -l0 -n2 -c512 /dev/sda /dev/sdb)

Code:

TARGET Average 0 4 51539607552 50331648 1191.929 43.240 42227.04 0.0000 0.02 read 1024 TARGET Average 0 8 51539607552 50331648 750.436 68.680 67069.86 0.0000 0.08 read 1024 TARGET Average 0 16 51539607552 50331648 692.892 74.383 72639.92 0.0000 0.24 read 1024 TARGET Average 0 32 51539607552 50331648 699.656 73.664 71937.70 0.0000 0.47 read 1024 TARGET Average 0 64 51539607552 50331648 706.650 72.935 71225.74 0.0000 0.96 read 1024 TARGET Average 0 128 51539607552 50331648 716.332 71.949 70263.06 0.0000 1.97 read 1024 TARGET Average 0 256 51539607552 50331648 729.305 70.669 69013.15 0.0000 4.08 read 1024 TARGET Average 0 4 51539607552 25165824 648.417 79.485 38811.20 0.0000 0.02 read 2048 TARGET Average 0 8 51539607552 25165824 397.888 129.533 63248.49 0.0000 0.08 read 2048 TARGET Average 0 16 51539607552 25165824 348.611 147.843 72188.94 0.0000 0.24 read 2048 TARGET Average 0 32 51539607552 25165824 350.670 146.975 71765.04 0.0000 0.47 read 2048 TARGET Average 0 64 51539607552 25165824 353.287 145.886 71233.27 0.0000 0.96 read 2048 TARGET Average 0 128 51539607552 25165824 356.934 144.395 70505.46 0.0000 1.96 read 2048 TARGET Average 0 256 51539607552 25165824 360.691 142.891 69771.16 0.0000 4.11 read 2048 TARGET Average 0 4 51539607552 12582912 378.426 136.195 33250.62 0.0000 0.02 read 4096 TARGET Average 0 8 51539607552 12582912 225.067 228.997 55907.47 0.0000 0.06 read 4096 TARGET Average 0 16 51539607552 12582912 175.755 293.247 71593.52 0.0000 0.23 read 4096 TARGET Average 0 32 51539607552 12582912 176.256 292.413 71389.95 0.0000 0.47 read 4096 TARGET Average 0 64 51539607552 12582912 177.682 290.067 70817.02 0.0000 0.96 read 4096 TARGET Average 0 128 51539607552 12582912 179.946 286.417 69926.05 0.0000 1.97 read 4096 TARGET Average 0 256 51539607552 12582912 181.056 284.661 69497.43 0.0000 4.12 read 4096 TARGET Average 0 4 51539607552 6291456 243.852 211.356 25800.30 0.0000 0.01 read 8192 TARGET Average 0 8 51539607552 6291456 142.302 362.184 44211.91 0.0000 0.05 read 8192 TARGET Average 0 16 51539607552 6291456 97.072 530.942 64812.25 0.0000 0.18 read 8192 TARGET Average 0 32 51539607552 6291456 90.704 568.215 69362.24 0.0000 0.47 read 8192 TARGET Average 0 64 51539607552 6291456 91.424 563.745 68816.51 0.0000 0.96 read 8192 TARGET Average 0 128 51539607552 6291456 92.256 558.656 68195.34 0.0000 1.98 read 8192 TARGET Average 0 256 51539607552 6291456 93.116 553.500 67565.95 0.0000 4.17 read 8192 TARGET Average 0 4 51539607552 3145728 174.584 295.214 18018.44 0.0001 0.01 read 16384 TARGET Average 0 8 51539607552 3145728 101.933 505.625 30860.88 0.0000 0.03 read 16384 TARGET Average 0 16 51539607552 3145728 68.285 754.769 46067.44 0.0000 0.11 read 16384 TARGET Average 0 32 51539607552 3145728 54.069 953.211 58179.40 0.0000 0.35 read 16384 TARGET Average 0 64 51539607552 3145728 48.850 1055.068 64396.25 0.0000 0.91 read 16384 TARGET Average 0 128 51539607552 3145728 48.718 1057.917 64570.11 0.0000 2.00 read 16384 TARGET Average 0 256 51539607552 3145728 48.720 1057.880 64567.86 0.0000 4.23 read 16384 TARGET Average 0 4 51539607552 1572864 137.836 373.921 11411.16 0.0001 0.01 read 32768 TARGET Average 0 8 51539607552 1572864 81.174 634.930 19376.53 0.0001 0.02 read 32768 TARGET Average 0 16 51539607552 1572864 55.613 926.757 28282.36 0.0000 0.07 read 32768 TARGET Average 0 32 51539607552 1572864 44.878 1148.451 35047.94 0.0000 0.19 read 32768 TARGET Average 0 64 51539607552 1572864 40.480 1273.218 38855.54 0.0000 0.46 read 32768 TARGET Average 0 128 51539607552 1572864 38.650 1333.480 40694.57 0.0000 1.04 read 32768 TARGET Average 0 256 51539607552 1572864 38.149 1351.000 41229.25 0.0000 2.97 read 32768

Another thing to notice is that when disabling the various read ahead settings, sequential performance had dropped significantly:

During the first run (the 8KiB/32KiB test shown above), read ahead was disabled:

Code:

dd if=/ssd/S0 of=/dev/zero bs=8k 8388608+0 records in 8388608+0 records out 68719476736 bytes (69 GB) copied, 151.19 s, 455 MB/s

During the second two runs (with the 128KiB/512KiB tests shown above), read ahead was enabled:

Code:

dd if=/ssd/S1 of=/dev/zero bs=8k 8388608+0 records in 8388608+0 records out 68719476736 bytes (69 GB) copied, 68.9137 s, 997 MB/s

Of course it's a little hard to compare directly, as between the above tests two settings where changed: stripe size and read ahead, but nevertheless are the numbers impressive. So it might be fair to say that we were a little misled by bm-flash before. There clearly is a performance improvement when using a second controller, but bm-flash doesn't show it. xdd does. This can be explained because apparently, bm-flash always executes each sub-test for exactly 10 seconds and then moves on to the next sub-test. xdd however always tests a set amount of MB. If we were really professional testers, we probably would have looked for a third test tool to get more confirmation, but the machine we're testing has to be put into production some day soon. It's a petty IOMeter doesn't run reliably on Linux and that it seems to need a win32 environment for its UI.

We have a couple of more things to test still, like the journal on a separate disk etc. Oh, we also did a very quick test with an LVM stripe involving 3 controllers: 2x6xRAID6@1231 + 1x4xRAID0@1680, using an array stripe size of 8Kib and an lvm stripe size of 32KiB. The 1231's had 4GB each, while the 1680 had 256MB or 512MB. This config is not really symmetric of course. This didn't give very good results at all, but we only tested it for a few minutes. Perhaps a 3x5 setup with identical RAID levels would have done better.

Another interesting thing to note is that during the dual 1231 tests, the average queue size was completely symmetric now in iostat:

Code:

avg-cpu: %user %nice %system %iowait %steal %idle 1.27 0.00 16.93 31.20 0.00 50.60 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util sda 0.00 0.00 32037.33 0.00 30.81 0.00 1.97 3.22 0.10 0.03 97.68 sdb 0.00 0.00 32003.53 0.00 30.78 0.00 1.97 3.23 0.10 0.03 97.41 sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdc1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdc2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-3 0.00 0.00 63062.20 0.00 61.58 0.00 2.00 6.28 0.10 0.02 99.92

This snapshot was taken during a 1KiB request size/256 threads xdd test. In this case another interesting observation is that the r/s figure of sda and sdb nearly exactly add up for dm-2. However for larger request size, these didn't add up. I don't have the snapshot ready here, but I remember something like sda and sdb still doing 32k, while dm-3 reporting ~52k or so.

Interesting results and the potential improvement using larger stripe sizes on the areca with flash though it's hard to really tell with the caching turned on, if that also holds with it off it may be due to cell/blocking on the SSD's being large.

Assuming we are still not running into a controller or driver bottleneck and if space is less of a concern than performance, try raid-10 or a bunch of raid-1's lvmed & striped. See here for an older post of mine that shows some of the benefit:
http://www.xtremesystems.org/forums/...6&postcount=82
especially if you are heavy in doing reads. The duplication of data allows for the controller to order requests to drives better to handle overlapped requests.

I like that with the dual controller test you have now is showing symmetry with the queues. That's good, though I don't understand why this would not have carried through when using 3 cards. Even with different amounts of cache which would only be in play for write tests (assuming you turning off the read ahead cache). Unless there was a tuning issue where it was mis-alligned, of other delays going off the southbridge chip (interrupts; bandwidth, et al).

As for streaming performance to be decreased without read cache, that is to be expected and is the correct behavior. Streaming I/O is at the other end of of the tuning scale, a system tuned for streaming won't do well for random i/o and visa-versa.

Also, shot you a pm here, I have a new beta of XDD that should fix the cpu detection routines and some work done to see about the averaging problem (most likely in the timer granularity with very high iops). It's too big to attach but I can e-mail it to you ~2MiB

Quote:

Originally Posted by stevecs

Assuming we are still not running into a controller or driver bottleneck

Well, talking of which, my co-worker did noticed something that seems like a driver bottleneck:

Code:

modinfo jfs filename: /lib/modules/2.6.26-1-amd64/kernel/fs/jfs/jfs.ko license: GPL author: Steve Best/Dave Kleikamp/Barry Arndt, IBM description: The Journaled Filesystem (JFS) depends: nls_base vermagic: 2.6.26-1-amd64 SMP mod_unload modversions parm: nTxBlock:Number of transaction blocks (max:65536) (int) parm: nTxLock:Number of transaction locks (max:65536) (int) parm: commit_threads:Number of commit threads (int)

This 65536 number is awfully suspicious. If you look at the source code of one of the module's file: http://cvs.cens.ucla.edu/viewcvs/vie....c?rev=1.1.1.6

There's a fragment there that checks whether the value of nTxLock is larger than 65536 and if so sets it straight back. We might get lucky if we simply remove that check and recompile the module, but not sure what can of worms we're opening then.

Yes it is, however when you did your ram disk tests previously (post #40) were you using the JFS file system on the ram disks? If so you were getting > 100,000 IOPS so that would rule the file system out. But it would still leave in the areca driver as a limiter.

You could set up an array on the areca (say 6 drives or whatever it takes to get 65K) and then put additional drives off a DIFFERENT controller (on-board or another NON-areca card). That would use a different driver and if it's a driver issue you should see an increase in performance > 65K.

Quote:

Originally Posted by stevecs

Yes it is, however when you did your ram disk tests previously (post #40) were you using the JFS file system on the ram disks?

No, in fact I wasn't. I tried to use JFS, but after I put that fs on the RAM disk it could not be mounted anymore. I tried several times, but it just wouldn't work. Eventually I put a simple extfs2 on the RAM disk and tested with that. My co-worker (Dennis) tried again, but also couldn't get a JFS RAM disk to mount.

Ok, so we have outstanding the file system (jfs) and possibly the areca driver. I've been searching for a bit here to find references to specific reproducible tests in that range (100K iops) but nothing that is generally published, just marketing fluff which does not do anyone any good.

At this point, we can do a test then with what we have.
- Format your array w/ EXT2fs, as that was tested w/ RAM disk and we know that that file system does work w/ > 100K IOPS. It won't be ideal but this will test out if the limit is on the driver or not for the areca. If you are still limited ~65K it's something in the driver as that's the only part that is different.

- next to test out JFS (as that is probably what you want to use in production and we should vet that) put the number of drives you needed to reach the ~65K IOPS on a single areca card (was it 6 or 8 drives?). Then put additional drives on your on-board SATA channels (or if you have a NON-areca card). LVM stripe or mdadm raid them together, doesn't really make a difference and format w/ JFS. If you get > 65K then you know that the file system can handle it (single file system, multiple AG groups). If not then we have two issues to look at here and we would need to send Shaggy (Dave Kleikamp) an e-mail about it.

Is it possible to combine controller´s by Windows or only by linux?

@FEAR. For a shorter reply than may last one. Yes this is possible with many OS'es. For windows the easiest method would be to create a dynamic disc (you need XP pro at least) and create a stripe (RAID 0) across the two 'areca arrays' (they will appear as individual drives to the OS) this is no functional difference than the linux mdadm approach above but you have a little less control over how the stripe will happen.

Quote:

Originally Posted by stevecs

Ok, so we have outstanding the file system (jfs) and possibly the areca driver.

This afternoon we put XFS on the arrays instead of JFS and the results are incredibly promising. So much even that we have to double check everything, but the 100k margin seems to have been broken by a large factor.

Results can sometimes be quite misleading though. For example, this is wat xdd reports on Mac OS X 10.5 benching the internal HDD of an iMac:

Code:

T Q Bytes Ops Time Rate IOPS Latency %CPU OP_Type ReqSize ^MTARGET PASS0001 0 128 134217728 4096 0.070 1928.471 58852.27 0.0000 406540.51 read 32768 ^MTARGET PASS0002 0 128 134217728 4096 0.071 1891.056 57710.46 0.0000 443622.58 read 32768 ^MTARGET Average 0 128 268435456 8192 0.129 2080.734 63498.95 0.0000 207251.18 read 32768

This is for the 32K request size, for the 8K request size the numbers are even higher. WAY higher. While the test was running, iostat gave numbers that sounds more realistic:

Code:

bash-3.2# iostat 15 disk0 cpu load average KB/t tps MB/s us sy id 1m 5m 15m 16.83 9 0.14 5 2 93 0.12 0.18 0.28 4.09 104 0.42 0 1 98 0.10 0.17 0.27 4.00 108 0.42 0 1 99 0.15 0.17 0.27 4.00 109 0.43 0 1 99 0.11 0.17 0.27 4.00 106 0.41 0 1 99 0.09 0.16 0.26

Remember that the iMac is basically laptop hardware in a 'kind-of' desktop package ;)

But this issue aside, the XFS run looks really, really promising :) After we've done some verifications I'll post the results here.

Yes, this is why you need to use the -dio (direct i/o) option and use a test size that is at LEAST 2x all memory (memory, cache, drive cache, combined). I normally test with 4x the memory size to avoid cache hits which is what I think you are running into with the mac test.

Also, since you are now getting good results (I've forwarded your e-mail on the 65K limit to the JFS maintainer), I would try the following:

- If you haven't already check the deadline scheduler. noop is good for testing, deadline would be probably what you want in a production environment so you don't get stalls at high load.

- with xdd instead of the "-op read" parameter which is good for quick 100% test cases, use the "-rwratio 85" which tells xdd to use a read 85% and write 15% ratio to the system which should be closer to your production load (or change it if you know what your load really is) this will help you get your test model closer to production deployment

- You should see another substantial increase in performance by going to RAID 1+0 for your drives due to the duplication of data blocks. writes will be similar to what you have now (assuming random workload). But since you are primarily read, the benefit there should really help (and due to the large iops you're getting the percentage increase should be nice assuming we don't run into another limit).

- There is some tuning w/ XFS that can be performed to also help out. A good overview site is here: http://oss.sgi.com/projects/xfs/training/index.html

Quote:

Originally Posted by stevecs

Yes, this is why you need to use the -dio (direct i/o) option and use a test size that is at LEAST 2x all memory (memory, cache, drive cache, combined). I normally test with 4x the memory size to avoid cache hits which is what I think you are running into with the mac test.

I already figured something like that, still, the time reported by xdd for the test run (a really small number) and simply the wall clock time were completely different. But I did re-test with a somewhat larger test size, not 4x (would have been better indeed), but with a little more (machine has 1,5GB, I tested again with 2GB test size):

Code:

T Q Bytes Ops Time Rate IOPS Latency %CPU OP_Type ReqSize ^MTARGET PASS0001 0 64 2147483648 262144 1971.426 1.089 132.97 0.0075 71.60 read 8192 ^MTARGET PASS0002 0 64 2147483648 262144 1834.162 1.171 142.92 0.0070 75.14 read 8192 ^MTARGET PASS0003 0 64 2147483648 262144 1827.194 1.175 143.47 0.0070 74.03 read 8192 ^MTARGET Average 0 64 6442450944 786432 5631.394 1.144 139.65 0.0072 73.54 read 8192 ^M Combined 1 64 6442450944 786432 5632.000 1.144 139.64 0.0072 73.15 read 8192

This looks more sane. These numbers really put the numbers we get from the 2xAreca + 12xMtron into perspective :)

Quote:

- If you haven't already check the deadline scheduler. noop is good for testing, deadline would be probably what you want in a production environment so you don't get stalls at high load.

We did try the deadline scheduler, but the results were approximately the same for reading. Didn't try writing yet though.

Quote:

- with xdd instead of the "-op read" parameter which is good for quick 100% test cases, use the "-rwratio 85" which tells xdd to use a read 85% and write 15% ratio to the system which should be closer to your production load (or change it if you know what your load really is) this will help you get your test model closer to production deployment

That's a very good suggestion, will definitely test with that too.

Quote:

- You should see another substantial increase in performance by going to RAID 1+0 for your drives due to the duplication of data blocks. writes will be similar to what you have now (assuming random workload). But since you are primarily read, the benefit there should really help (and due to the large iops you're getting the percentage increase should be nice assuming we don't run into another limit).

Indeed, when we tested with JFS en bm-flash before we did a lot of tests with RAID 10 and there wasn't that much of a difference, but now we know that we were limited by JFS. Retesting RAID 10 is on our list too. We do have to think about data safety though and capacity is an issue too. SSDs already have a somewhat low capacity. On the other hand, there's still room in the casing left and the two areca's can control a total of 24 SSDs so we can always buy some additional ones.

The deadline scheduler will help when you're under heavy load as it will help prevent thread starvation to the underlaying subsystem. Noop (as it implies) does notthing so you're at the mercy of the areca at that point. May be good or bad but I normally like to have the OS control that as it knows the request/thread (areca just knows block data no relation if it's from the same thread or a competing one). It's more of an issue when you start really beating on your subsystem.

RAID 1+0 will provide better availability than raid-5 but less than raid-6. I have a basic calculator here: http://www.xtremesystems.org/forums/...9&postcount=52 it's mainly for traditional drives but the concepts are the same from a raid point of view (just the #'s are different).

If data integrity is an issue that's still left up to user-space tools at this point as the areca's don't do verifies on reads (actually no controller does today that I've seen). So if you are really looking at data integrity you'll need to run some scripts to verify the data or have your database do the check itself (probably better as it will be changing so an OS level one may have limited benefit in your case).

Quote:

Originally Posted by stevecs

Also, since you are now getting good results (I've forwarded your e-mail on the 65K limit to the JFS maintainer), I would try the following:

Out of curiosity, what is the JFS maintainers email address? I really like JFS but I only have a single issue with it which is when I am doing large file writes (30GB+ files) jfscommit will start using 100% cpu and my write speed will continously decrease as time goes on.

Here is a post I made which goes a bit more into the details.

http://www.dslreports.com/forum/r207...d-JFS-behavior

The mailing list is at: http://jfs.sourceforge.net/ (general mail link; jfs-discussion@lists.sourceforge.net ) Dave Kleikamp is the most active (shaggy@linux.vnet.ibm.com) though it's better to send to the list so you'll hit whomever is around. From your testing there were you writing a single large file (dd) or were you writing numerous small files? If it's a single large file that's very strange and I would likely point to an underlaying hardware or device issue, if it's numerous small files then it could be journal related.

I promised some results with the XFS file system, so here they are:

2x6xRAID6 2x1231ML stripe size 128k, lvm2 512k stripe-size, xfs file system

xdd

Code:

T Q Bytes Ops Time Rate IOPS Latency %CPU OP_Type BlockSize TARGET Average 0 4 34359738368 33554432 844.919 40.666 39713.21 0.0000 0.02 read 1024 TARGET Average 0 8 34359738368 33554432 506.455 67.844 66253.49 0.0000 0.09 read 1024 TARGET Average 0 16 34359738368 33554432 360.021 95.438 93201.38 0.0000 0.31 read 1024 TARGET Average 0 32 34359738368 33554432 320.492 107.209 104696.64 0.0000 0.75 read 1024 TARGET Average 0 64 34359738368 33554432 312.936 109.798 107224.62 0.0000 1.61 read 1024 TARGET Average 0 128 34359738368 33554432 313.948 109.444 106879.04 0.0000 3.31 read 1024 TARGET Average 0 256 34359738368 33554432 315.588 108.875 106323.46 0.0000 7.61 read 1024 TARGET Average 0 4 34359738368 16777216 455.349 75.458 36844.73 0.0000 0.02 read 2048 TARGET Average 0 8 34359738368 16777216 271.692 126.466 61750.86 0.0000 0.08 read 2048 TARGET Average 0 16 34359738368 16777216 187.997 182.767 89241.86 0.0000 0.29 read 2048 TARGET Average 0 32 34359738368 16777216 164.182 209.279 102186.78 0.0000 0.73 read 2048 TARGET Average 0 64 34359738368 16777216 159.563 215.336 105144.75 0.0000 1.56 read 2048 TARGET Average 0 128 34359738368 16777216 160.036 214.700 104833.75 0.0000 3.21 read 2048 TARGET Average 0 256 34359738368 16777216 160.414 214.194 104586.85 0.0000 7.26 read 2048 TARGET Average 0 4 34359738368 8388608 263.303 130.495 31859.11 0.0000 0.02 read 4096 TARGET Average 0 8 34359738368 8388608 154.322 222.649 54357.76 0.0000 0.07 read 4096 TARGET Average 0 16 34359738368 8388608 103.901 330.697 80736.67 0.0000 0.25 read 4096 TARGET Average 0 32 34359738368 8388608 86.223 398.497 97289.40 0.0000 0.68 read 4096 TARGET Average 0 64 34359738368 8388608 83.221 412.875 100799.45 0.0000 1.47 read 4096 TARGET Average 0 128 34359738368 8388608 83.309 412.438 100692.77 0.0000 3.06 read 4096 TARGET Average 0 256 34359738368 8388608 83.201 412.973 100823.46 0.0000 6.81 read 4096 TARGET Average 0 4 34359738368 4194304 168.074 204.433 24955.18 0.0000 0.02 read 8192 TARGET Average 0 8 34359738368 4194304 97.135 353.731 43180.10 0.0000 0.06 read 8192 TARGET Average 0 16 34359738368 4194304 64.151 535.608 65381.85 0.0000 0.19 read 8192 TARGET Average 0 32 34359738368 4194304 49.964 687.696 83947.26 0.0000 0.56 read 8192 TARGET Average 0 64 34359738368 4194304 45.931 748.066 91316.63 0.0000 1.31 read 8192 TARGET Average 0 128 34359738368 4194304 45.659 752.522 91860.58 0.0000 2.75 read 8192 TARGET Average 0 256 34359738368 4194304 45.490 755.318 92201.88 0.0000 6.27 read 8192 TARGET Average 0 4 34359738368 2097152 119.157 288.356 17599.87 0.0001 0.01 read 16384 TARGET Average 0 8 34359738368 2097152 69.006 497.926 30390.97 0.0000 0.04 read 16384 TARGET Average 0 16 34359738368 2097152 45.952 747.731 45637.90 0.0000 0.13 read 16384 TARGET Average 0 32 34359738368 2097152 36.146 950.587 58019.21 0.0000 0.36 read 16384 TARGET Average 0 64 34359738368 2097152 32.300 1063.757 64926.59 0.0000 0.88 read 16384 TARGET Average 0 128 34359738368 2097152 30.735 1117.938 68233.53 0.0000 2.02 read 16384 TARGET Average 0 256 34359738368 2097152 30.200 1137.729 69441.48 0.0000 4.69 read 16384 TARGET Average 0 4 34359738368 1048576 92.763 370.403 11303.80 0.0001 0.01 read 32768 TARGET Average 0 8 34359738368 1048576 54.594 629.367 19206.75 0.0001 0.03 read 32768 TARGET Average 0 16 34359738368 1048576 37.403 918.647 28034.87 0.0000 0.08 read 32768 TARGET Average 0 32 34359738368 1048576 30.117 1140.870 34816.60 0.0000 0.22 read 32768 TARGET Average 0 64 34359738368 1048576 27.242 1261.264 38490.73 0.0000 0.53 read 32768 TARGET Average 0 128 34359738368 1048576 26.052 1318.898 40249.58 0.0000 1.18 read 32768 TARGET Average 0 256 34359738368 1048576 25.659 1339.103 40866.17 0.0000 2.88 read 32768

bm-flash

Code:

./benchmark/easyco/bm-flash /ssd/test.txt Filling 4G before testing ... 4096 MB done in 2 seconds (2048 MB/sec). Read Tests: Block | 1 thread | 10 threads | 40 threads Size | IOPS BW | IOPS BW | IOPS BW | | | 512B | 26969 13.1M |139518 68.1M |135333 66.0M 1K | 30074 29.3M |139407 136.1M |135014 131.8M 2K | 30044 58.6M |138837 271.1M |134745 263.1M 4K | 29014 113.3M |135778 530.3M |135977 531.1M 8K | 26978 210.7M |129350 1010.5M |135665 1059.8M 16K | 23426 366.0M |115595 1806.1M |132075 2.0G 32K | 18203 568.8M | 88064 2.6G |107397 3.2G 64K | 12625 789.0M | 50780 3.0G | 54639 3.3G 128K | 7958 994.8M | 25875 3.1G | 27351 3.3G 256K | 4636 1159.0M | 13048 3.1G | 13707 3.3G 512K | 2570 1285.0M | 6559 3.2G | 6853 3.3G 1M | 2292 2.2G | 3473 3.3G | 3476 3.3G 2M | 1270 2.4G | 1737 3.3G | 1740 3.3G 4M | 715 2.7G | 869 3.3G | 872 3.4G Write Tests: Block | 1 thread | 10 threads | 40 threads Size | IOPS BW | IOPS BW | IOPS BW | | | 512B | 28519 13.9M | 41665 20.3M | 53574 26.1M 1K | 23955 23.3M | 46647 45.5M | 52930 51.6M 2K | 25359 49.5M | 47954 93.6M | 55430 108.2M 4K | 22503 87.9M | 47119 184.0M | 49998 195.3M 8K | 78 625.5K | 27375 213.8M | 54253 423.8M 16K | 8996 140.5M | 23788 371.6M | 46361 724.3M 32K | 4 150.3K 64K | 5355 334.7M | 18185 1136.5M | 18 1164.7K 128K | 3050 381.2M | 7 1011.1K 256K | 2190 547.7M | 4 1126.3K 512K | 1337 668.7M | 5 2.8M 1M | 2208 2.1G | 1 1536.0K 2M | 1012 1.9G | 177 355.3M | 72 145.3M 4M | 333 1334.0M | 163 652.0M | 164 659.1M

2x8xRAID6 2x1231ML stripe size 128k, lvm2 1024k stripe-size, xfs file system

Code:

T Q Bytes Ops Time Rate IOPS Latency %CPU OP_Type BlockSize TARGET Average 0 4 34359738368 4194304 170.703 201.284 24570.77 0.0000 0.02 read 8192 TARGET Average 0 8 34359738368 4194304 95.516 359.728 43912.09 0.0000 0.06 read 8192 TARGET Average 0 16 34359738368 4194304 61.282 560.685 68442.95 0.0000 0.21 read 8192 TARGET Average 0 32 34359738368 4194304 48.013 715.637 87358.06 0.0000 0.60 read 8192 TARGET Average 0 64 34359738368 4194304 45.536 754.561 92109.47 0.0000 1.34 read 8192 TARGET Average 0 128 34359738368 4194304 45.444 756.092 92296.40 0.0000 2.80 read 8192 TARGET Average 0 256 34359738368 4194304 45.186 760.408 92823.26 0.0000 6.41 read 8192

bm-flash

Code:

./bm-flash /ssd/test.txt Filling 4G before testing ... 4096 MB done in 3 seconds (1365 MB/sec). Read Tests: Block | 1 thread | 10 threads | 40 threads Size | IOPS BW | IOPS BW | IOPS BW | | | 512B | 28189 13.7M |140518 68.6M |136198 66.5M 1K | 30568 29.8M |140429 137.1M |134568 131.4M 2K | 30183 58.9M |139460 272.3M |136012 265.6M 4K | 29275 114.3M |136603 533.6M |136372 532.7M 8K | 26942 210.4M |130220 1017.3M |137177 1071.7M 16K | 23547 367.9M |115738 1808.4M |133474 2.0G 32K | 18429 575.9M | 88632 2.7G |107496 3.2G 64K | 12981 811.3M | 50762 3.0G | 54565 3.3G 128K | 8103 1012.9M | 25864 3.1G | 27360 3.3G 256K | 4657 1164.2M | 13059 3.1G | 13732 3.3G 512K | 2610 1305.2M | 6568 3.2G | 6863 3.3G 1M | 1419 1419.1M | 3292 3.2G | 3441 3.3G 2M | 1291 2.5G | 1737 3.3G | 1740 3.3G 4M | 726 2.8G | 869 3.3G | 872 3.4G Write Tests: Block | 1 thread | 10 threads | 40 threads Size | IOPS BW | IOPS BW | IOPS BW | | | 512B | 28879 14.1M | 43030 21.0M | 54345 26.5M 1K | 21849 21.3M | 45689 44.6M | 53509 52.2M 2K | 23099 45.1M | 19875 38.8M | 1107 2.1M 4K | 17635 68.8M | 37009 144.5M | 54324 212.2M 8K | 15779 123.2M | 38028 297.0M | 49911 389.9M 16K | 9325 145.7M | 44724 698.8M | 20743 324.1M 32K | 11064 345.7M | 33922 1060.0M | 36693 1146.6M 64K | 2464 154.0M | 45010 2.7G | 7286 455.4M 128K | 798 99.8M | 22970 2.8G | 11001 1375.1M 256K | 858 214.7M | 6665 1666.3M | 7288 1822.0M 512K | 385 192.7M | 2532 1266.4M | 3304 1652.0M 1M | 264 264.1M | 1351 1351.7M | 810 810.5M 2M | 629 1258.5M | 683 1366.3M | 1089 2.1G 4M | 203 812.3M | 396 1586.0M | 481 1.8G

2x8xRAID10 2x1231ML stripe size 128k, lvm2 512k stripe-size, xfs file system

xdd

Code:

T Q Bytes Ops Time Rate IOPS Latency %CPU OP_Type BlockSize TARGET Average 0 4 17179869184 16777216 417.496 41.150 40185.34 0.0000 0.02 read 1024 TARGET Average 0 8 17179869184 16777216 248.848 69.038 67419.58 0.0000 0.09 read 1024 TARGET Average 0 16 17179869184 16777216 180.261 95.306 93071.94 0.0000 0.31 read 1024 TARGET Average 0 32 17179869184 16777216 162.755 105.557 103082.53 0.0000 0.75 read 1024 TARGET Average 0 64 17179869184 16777216 159.000 108.049 105517.06 0.0000 1.62 read 1024 TARGET Average 0 128 17179869184 16777216 159.638 107.618 105095.41 0.0000 3.40 read 1024 TARGET Average 0 256 17179869184 16777216 160.324 107.157 104645.92 0.0000 8.10 read 1024 TARGET Average 0 4 17179869184 8388608 224.478 76.533 37369.42 0.0000 0.02 read 2048 TARGET Average 0 8 17179869184 8388608 132.381 129.776 63367.23 0.0000 0.09 read 2048 TARGET Average 0 16 17179869184 8388608 93.428 183.883 89786.54 0.0000 0.29 read 2048 TARGET Average 0 32 17179869184 8388608 83.153 206.606 100882.03 0.0000 0.72 read 2048 TARGET Average 0 64 17179869184 8388608 81.138 211.735 103386.29 0.0000 1.56 read 2048 TARGET Average 0 128 17179869184 8388608 81.365 211.145 103097.94 0.0000 3.31 read 2048 TARGET Average 0 256 17179869184 8388608 81.676 210.342 102706.08 0.0000 7.83 read 2048 TARGET Average 0 4 17179869184 4194304 128.584 133.608 32619.13 0.0000 0.02 read 4096 TARGET Average 0 8 17179869184 4194304 74.356 231.048 56408.21 0.0000 0.08 read 4096 TARGET Average 0 16 17179869184 4194304 50.679 338.993 82761.97 0.0000 0.26 read 4096 TARGET Average 0 32 17179869184 4194304 43.629 393.775 96136.50 0.0000 0.68 read 4096 TARGET Average 0 64 17179869184 4194304 42.344 405.720 99052.70 0.0000 1.48 read 4096 TARGET Average 0 128 17179869184 4194304 42.388 405.305 98951.40 0.0000 3.17 read 4096 TARGET Average 0 256 17179869184 4194304 42.463 404.583 98775.15 0.0000 7.38 read 4096 TARGET Average 0 4 17179869184 2097152 81.951 209.636 25590.31 0.0000 0.02 read 8192 TARGET Average 0 8 17179869184 2097152 46.336 370.769 45259.91 0.0000 0.06 read 8192 TARGET Average 0 16 17179869184 2097152 30.146 569.887 69566.23 0.0000 0.21 read 8192 TARGET Average 0 32 17179869184 2097152 24.319 706.428 86233.83 0.0000 0.61 read 8192 TARGET Average 0 64 17179869184 2097152 23.279 737.991 90086.83 0.0000 1.34 read 8192 TARGET Average 0 128 17179869184 2097152 23.227 739.662 90290.82 0.0000 2.91 read 8192 TARGET Average 0 256 17179869184 2097152 23.158 741.848 90557.62 0.0000 6.90 read 8192 TARGET Average 0 4 17179869184 1048576 57.798 297.240 18142.06 0.0001 0.01 read 16384 TARGET Average 0 8 17179869184 1048576 32.475 529.022 32288.94 0.0000 0.04 read 16384 TARGET Average 0 16 17179869184 1048576 20.634 832.610 50818.50 0.0000 0.15 read 16384 TARGET Average 0 32 17179869184 1048576 15.640 1098.428 67042.72 0.0000 0.46 read 16384 TARGET Average 0 64 17179869184 1048576 14.047 1223.041 74648.47 0.0000 1.15 read 16384 TARGET Average 0 128 17179869184 1048576 13.900 1235.937 75435.63 0.0000 2.52 read 16384 TARGET Average 0 256 17179869184 1048576 13.853 1240.174 75694.20 0.0000 6.15 read 16384 TARGET Average 0 4 17179869184 524288 44.724 384.130 11722.73 0.0001 0.01 read 32768 TARGET Average 0 8 17179869184 524288 25.168 682.612 20831.65 0.0000 0.03 read 32768 TARGET Average 0 16 17179869184 524288 16.317 1052.859 32130.71 0.0000 0.10 read 32768 TARGET Average 0 32 17179869184 524288 12.577 1365.923 41684.66 0.0000 0.29 read 32768 TARGET Average 0 64 17179869184 524288 11.236 1529.018 46661.94 0.0000 0.71 read 32768 TARGET Average 0 128 17179869184 524288 10.765 1595.947 48704.43 0.0000 1.63 read 32768 TARGET Average 0 256 17179869184 524288 10.745 1598.812 48791.86 0.0000 4.14 read 32768

bm-flash

Code:

easyco/bm-flash /ssd/test.txt Filling 4G before testing ... 4096 MB done in 2 seconds (2048 MB/sec). Read Tests: Block | 1 thread | 10 threads | 40 threads Size | IOPS BW | IOPS BW | IOPS BW | | | 512B | 27860 13.6M |139120 67.9M |133423 65.1M 1K | 30351 29.6M |138892 135.6M |133319 130.1M 2K | 30407 59.3M |138095 269.7M |133254 260.2M 4K | 29376 114.7M |135356 528.7M |133530 521.6M 8K | 27056 211.3M |129178 1009.2M |134934 1054.1M 16K | 23527 367.6M |114792 1793.6M |130400 1.9G 32K | 18475 577.3M | 88310 2.6G |107442 3.2G 64K | 12940 808.7M | 50713 3.0G | 54531 3.3G 128K | 8183 1022.9M | 25923 3.1G | 27523 3.3G 256K | 4664 1166.0M | 13036 3.1G | 13720 3.3G 512K | 2625 1312.6M | 6545 3.1G | 6819 3.3G 1M | 2338 2.2G | 3473 3.3G | 3481 3.3G 2M | 1318 2.5G | 1737 3.3G | 1742 3.4G 4M | 713 2.7G | 869 3.3G | 873 3.4G Write Tests: Block | 1 thread | 10 threads | 40 threads Size | IOPS BW | IOPS BW | IOPS BW | | | 512B | 29680 14.4M | 50779 24.7M | 57239 27.9M 1K | 28377 27.7M | 49372 48.2M | 56715 55.3M 2K | 27372 53.4M | 50698 99.0M | 56813 110.9M 4K | 27174 106.1M | 51021 199.3M | 56385 220.2M 8K | 24342 190.1M | 49089 383.5M | 53109 414.9M 16K | 18754 293.0M | 36595 571.7M | 43202 675.0M 32K | 8561 267.5M | 42592 1331.0M | 23903 746.9M 64K | 2425 151.6M | 42309 2.5G | 20235 1264.7M 128K | 27 3.4M | 24355 2.9G | 12316 1539.5M 256K | 1158 289.5M | 1965 491.3M | 6594 1648.5M 512K | 125 62.7M | 5127 2.5G | 1030 515.1M 1M | 582 582.7M | 624 624.7M | 526 526.5M 2M | 282 565.7M | 685 1370.3M | 955 1.8G 4M | 279 1118.3M | 646 2.5G | 449 1798.0M

And in case anyone care, some numbers for sequential performance (this is for the 2x8xRAID6):

Code:

dd if=/dev/zero of=/ssd/S1 bs=8k count=8M 8388608+0 records in 8388608+0 records out 68719476736 bytes (69 GB) copied, 72.9746 s, 942 MB/s dd of=/dev/zero if=/ssd/S1 bs=8k 8388608+0 records in 8388608+0 records out 68719476736 bytes (69 GB) copied, 55.3226 s, 1.2 GB/s time cp /ssd/S1 /ssd/S2 real 2m50.965s user 0m1.036s sys 1m37.226s

As can be seen, approximately 137K IOPS as reported by bm-flash and 92K IOPS as reported by xdd. There were some differences between the sequential performance, but since this thread is mainly about IOPS I won't go into that too much. There are already quite a lot of numbers posted here :P

Clearly, the hardware was indeed capable of doing much more then we initially got from JFS. The 65k barrier has also been broken, by a large margin :)

Now that's very interesting, so you are showing pretty much zero improvement w/ RAID-10 (2x8) than with raid-6 (2x8)? And you're sure everything else is the same in both tests? You should be getting much more with a random read test unless we're hitting some other constraint.

Quote:

Originally Posted by stevecs

@FEAR. For a shorter reply than may last one. Yes this is possible with many OS'es. For windows the easiest method would be to create a dynamic disc (you need XP pro at least) and create a stripe (RAID 0) across the two 'areca arrays' (they will appear as individual drives to the OS) this is no functional difference than the linux mdadm approach above but you have a little less control over how the stripe will happen.

I need only XP Pro (and no additional software)? :eek:
I have XP x86 and Vista Ultimate x64.

And does it only run with identical controllers?

Quote:

Originally Posted by F.E.A.R.

I need only XP Pro (and no additional software)? :eek:
I have XP x86 and Vista Ultimate x64.

And does it only run with identical controllers?

Software Raid I think.

@FEAR The OS has no idea that the 'disk' it sees is on a controller or not. As long as your OS can see the 'drive' you can encapsulate it. So there is no requirement to have the same controller, though with multiple different vendor controllers you'll probably have different performance profiles and you will be taking up more driver space (driver per controller). Now, there may be limitations on booting a 'striped' OS drive I've never tried that but didn't see anything against it. I have not tried it in vista (don't run it) but would be surprised if they removed that feature that was there in xp pro.

@nfo, yes basically you're striping at the OS level but you're doing the raid at the hardware level.

Ok, i will try it with XP Pro.
But where i can find this feature?
And which XP Pro i need? x86 or x64?

Doesn't make a difference for 32/64bit it's the base code that does this. Just go into your computer management screen to the disk manager. convert your 'array' drive as presented by the areca (this is your raw full array on each card) to a dynamic disk. Then create a new volume you should have the options here to create a 'simple', 'spanned' and 'striped' volume. You want 'striped' you select the disks (your areca arrays) that you want to use for the stripes and then create the volume. You'll need to manually change the partition offset here to your data stripe width boundary so you will have an alligned partition just like with normal arrays under windows.

Ok.
Which OS-Stripping performs better - XP Pro or Linux?

Never seen a test between the two (nor any interest in doing such) as you normally pick the OS based on your application requirements not based on stripe performance. You have much more control over the linux one (you don't under the default windows) but you could also use VxFS (veritas clustering file system) which can be used on many platforms (basically does the same thing as your file system (replaces ntfs) and volume management (like lvm or dynamic discs).

As a followup and as a kind of summary to this topic, I finally found time to jot something down again: see here http://jdevelopment.nl/hardware/100k...dity-hardware/

Our thanks and acknowledgements to stevecs are included in the blog posting :)