Yes, this is why you need to use the -dio (direct i/o) option and use a test size that is at LEAST 2x all memory (memory, cache, drive cache, combined). I normally test with 4x the memory size to avoid cache hits which is what I think you are running into with the mac test.

Also, since you are now getting good results (I've forwarded your e-mail on the 65K limit to the JFS maintainer), I would try the following:

- If you haven't already check the deadline scheduler. noop is good for testing, deadline would be probably what you want in a production environment so you don't get stalls at high load.

- with xdd instead of the "-op read" parameter which is good for quick 100% test cases, use the "-rwratio 85" which tells xdd to use a read 85% and write 15% ratio to the system which should be closer to your production load (or change it if you know what your load really is) this will help you get your test model closer to production deployment

- You should see another substantial increase in performance by going to RAID 1+0 for your drives due to the duplication of data blocks. writes will be similar to what you have now (assuming random workload). But since you are primarily read, the benefit there should really help (and due to the large iops you're getting the percentage increase should be nice assuming we don't run into another limit).

- There is some tuning w/ XFS that can be performed to also help out. A good overview site is here: http://oss.sgi.com/projects/xfs/training/index.html