post your hard drive benchmarks :)

**WoLpH** · 01-27-2008, 12:36 PM

Originally Posted by stevecs

If you're not going to change your disk subsystem (ie, type of drives, or separate your work flows to different spindle groups et al) you're not going to see much at all of an improvement. Basically your computer MB/cpu, et al is generally NOT the bottleneck. It's mainly the drive subsystem that kills you (at least first, there are numerous bottlenecks in a system but that's a much bigger discussion a decent reference would be John Hennessy's/David Patterson's book "computer architecture: a quantitative approach")

My concern is mostly based on previous systems where the raid array would easily do 100MB/s and more but when transferring via the network the PCI bus was immediately completely saturated and because of that limiting the network transfers to 60MB/s. Because of the PCI-E lanes these restrictions obviously no longer apply (the shared PCI bus I mean), but since I haven't had the chance of testing a setup like this with a 'normal' motherboard I wasn't sure what to expect.

Before anyone can really point you in a direction a lot more information would be needed to model your type of access requirements and setup. I've just posted in the storage/raid thread some spreadsheets to do some quantitative analysis but they are only a start.

http://www.xtremesystems.org/forums/...=150176&page=2

Really comes down to is what is your workload type, and what performance level are you seeking. From that then you design the system to match. There's also the issue of disk subsystem utilization curves which come into play (Queuing theory & Markov models like the M/M/1 or M/M/m).

Fantastic information, this is something I can work with. Although the Queueing theory and Markov models usually only come into play when using higher loads, it's still a very interesting read, thank you very much.

As for my workload type, it's mostly transferring big files over the network (big as in 100MiB and more) to about 10 clients simultaneously so the queueing part will be quite simple and will most likely not give problems. My performance concerns are primarily related to the hardware limitations of the system.

It looks like I have enough information to start a proper analysis, perhaps I'll try calculate the Markov chain for it, it could give me some new insights.

**stevecs** · 01-27-2008, 01:39 PM

Correct on those models require a system in equilibrium and the number of incoming requests is unlimited among other items. but it sounds like you know that.

As for network transferring of large files another item that I've discovered recently is that switches are much more the bottleneck than normally thought for transferring large amounts of data. I've run into some tests with cisco switches (3750's, 3xxxx, 4xxxx et al (haven't tried the various 65xx's yet) that they cannot push full GbE speeds on their nics. They do generally OK in one direction but not in both (send/receive) looks like they are capped at 1GbE total per nic (ie one direction).

Then there is network filesystem overhead (like SMB/CIFS) which just suck at high speed performance (usually capped at around 60MiB/s or so per client connection).

**WoLpH** · 01-27-2008, 03:50 PM

Yes, I have some knowledge about queueing theory. As I'm currently programming with distributed systems I know a bit about multithreading and proper/fast ways to handle queues.

As for network limitations, that is one of the reasons I'm not a big fan of Cisco equipment, I'll take Foundry or Juniper over Cisco any day. Or, if money is a big issue, HP switches have been performing quite good for me aswell, 100-110MiB/s in both directions seems to be possible without the switch giving the least bit of problems (HP ProCurve 2824 or 2848 switches with 24 or 48 1GbE connections). The network won't be the limit here, perhaps the specific network cards though, as the onboard nic of my K8WE only does 75-80MiB/s while the 3c985 fiber channel nic does 120MiB/s easily.

And yes... network filesystems, I'm currently using NFS but I'm indeed not too happy about the performance. Perhaps I should consider switching to a virtual iSCSI system or ATA over Ethernet. But the last time I've tried ATA over Ethernet it was a bit unstable, it caused the kernel to lock up once in a while (and crashes in combination with ATA over Ethernet result in filesystem corruptions) so I'm not that eager to try it again.

Either way... I'm gonna look for a new motherboard with atleast 1x PCI-E 16x for the videocard, 1x PCI-E 8x for the raid card, and preferably PCI-X for the network cards (PCI-E fiber cards are still quite expensive at the moment, and I already have a load of PCI-X network cards lying around), if anyone has a tip I'm open for suggestions

I'm currently looking into Gigabyte but perhaps I'll have a look at Asus, Tyan or SuperMicro aswell.

**stevecs** · 01-27-2008, 04:06 PM

You may want to try NFS v4 if you're not already due to some of it's optimizations w/ command merging among other items. Bit more of a pain for clients/setup though. I haven't really tested that in a distributed environment yet though.

As for MB's I've used the asus ones (P5W64WS Pro) has 3 8x & 1 4x pci-e, (I've been trying to avoid pci-x as I'm looking more towards 10GbE and pci-X can't handle it). Main item though if you're looking for end-to-end you want an Intel based nic. I haven't found any of the other ones (broadcom, marvell, 3com, et al) able to push full wire speed in full duplex. For workstation classed boards both Tyan & Supermicro have been good in the past for me.

Good luck. and thanks for the tip on the juniper/foundary I'll check out some of those.

**WoLpH** · 01-27-2008, 08:33 PM

Thanks for the NFS v4 tip, I haven't tried switching to it yet. The total enviroment is not really distributed though, just some of the software I'm writing for work

10GbE sounds great to me, but it's still way to expensive imho, trunking a few gbit lines (dual or quad gbit connections) should be enough for now.

After your recommendation I've given Asus a good look and I think the P5E64WS Pro might be an option if I'd decide to switch to PCI-E, or the P5E3WS Pro if I'll stay with PCI-X. I'm looking at some Intel PRO/1000 PT dual port adapters right now, it seems they aren't too expensive. I'm actually considering a motherboard that would allow a little overclocking too, and I haven't found a Tyan/SuperMicro motherboard that allows that unfortunately (yeah, under windows with some tools, but I don't run windows

).

**BGVirtual** · 02-02-2008, 01:01 AM

4 x 150GB Raptors in Raid0, 128KB stripe size, Areca ARC-1210

**kemo** · 02-02-2008, 06:15 AM

Mien looks pretty good

**james bennett** · 02-08-2008, 08:43 AM

As you can see I have a Raptor. The transfer and burst levels aren't spectacular but, the access times are.

Click image for larger version.

Name: hd1.png
Views: 1171
Size: 38.1 KB
ID: 72027

**Boite de P4** · 02-10-2008, 10:37 AM

160Gb SATA @5400 rpm on my

32Mo

8Mo

**freeloader** · 02-10-2008, 11:46 AM

Two Hitachi Deskstar 320gb (7200rpm, 16mb cache) RAID 0

**Charles Wirth** · 02-10-2008, 02:32 PM

Here is a Skulltrail bench with onboard RAID

Vista Ultimate 64 bit
4x 150GB RAID 0

I need to get that Areca card.

**NiscoRacing** · 02-10-2008, 02:55 PM

This what mine 4x Samsung F1 32mb 1TB does in RAID0

**Fixxxer** · 02-10-2008, 05:02 PM

Just a single Seagate 7200.10 and it beats a craptor

**MotF Bane** · 02-10-2008, 06:18 PM

Seagate 7200.11 500GB.

Nisco, that's impressive.

**Charles Wirth** · 02-10-2008, 10:52 PM

Nisco, I was looking at those 32M drives, looking even better now.

**NapalmV5** · 02-12-2008, 05:01 AM

Fugger

get in touch with the peeps @ fusion-io to get a sample

the iodrive would be a great addition to skulltrail

at least thats what im planning on..

**Gen** · 02-12-2008, 10:03 AM

4x Seagate 7200.10 500gb
ICH9R 0+1 64kb stripe Write Cache Enable
P5K-E Q6600 @3.2 2GB
Vista x64

**james bennett** · 02-12-2008, 01:01 PM

Originally Posted by NiscoRacing

This what mine 4x Samsung F1 32mb 1TB does in RAID0

I want to set mine up in a striped set but, I can't afford another raptor at the moment. Your access times and transfer are incredible. Those are solid state drives right? Thats what is on my wish list next. Waiting for the price to come down.

**Pudi** · 02-12-2008, 04:05 PM

Originally Posted by james bennett

Those are solid state drives right?

That is not SSD !

That is :

**stoske** · 02-13-2008, 06:09 AM

2xSamsung 500GB 1xWD 500GB

Samsung SpinPoint T166 HD501LJ - 500GB

Samsung SpinPoint T166 HD501LJ - 500GB 2nd

WD5000AAKS

All tested on same rig:

Asus p5k-e wifi
4x1gb Geil 1ghz@1130
2140@3ghz

**james bennett** · 02-13-2008, 12:39 PM

Originally Posted by Pudi

That is not SSD !

That is :

I can't believe the transfer rates and the access time. It has to be reading it in a striped set right? (from the three disks). How else can it get such high transfer rates. I think I might be pricing this INSTEAD of solid states.