RAID5 file server build advice

[__Miguel_]

I also wondered about auto move function.
Do you know if you choosing "auto move when finished" option in torrent programs will prevent you from seeding that file as soon as downloaded has finished? (seems logical it will and that could cause issues with my ratio)

Well, I rarely use Torrents, so I'm not very familiar about that "auto-move" function, sorry. But, unless that function moves the files and updates the "read torrent+data from..." part (which would cause severe random reads on the RAID-5 array, and that's NEVER a good idea with RAID-5, hardware or otherwise), you'll automatically stop seeding that file, so yeah, that would pretty much kill your ratio.

It looks that 160 could be enough now. BUT as I find myself DLding more and more HD and huge torrents this "could" be a problem. My notebook drive is constantly full. However I never seed that much so offloading files to the array sooner would be an answer for a 160GB drive.
This said If I find a way to buy a WD320AAKS and be sure I'm getting a single platter one I will probably go for it and not look back.

What you said, only difference would be that the price should be about equal for both parts... hehehe

I think I would prefer manual offloading of files to. I don't think I could trust software for this, much. I mean moving files in windows is bad enough but moving it via got knows whose code... :/
Manual "Copy" option is my preferred way. Burned myself to many times enen with manual "move". One glitch and BAM! your directory/file is fried. whats been moved and what was not and what is complete is anybody's guess at that point. Joy for the whole afternoon...aahh...good times

lol I've never had issues with manual moves before. Unless on those files that were corrupted to start with (one of my server's memory sticks started missbehaving... it took me months to figure why I had to download some stuff twice, and sometimes video files had glitches...).

Granted, I don't move whole folders, only individual files (or groups of files), so that may well be the reason I have less problems... Though I'll keep that in mind.

We'll see... first I need to get my stuff back from RMA and figure out what went wrong with my 1st attempt to assemble it. (thank God I haven't bought the controller card or all the drives, cause I don't think they would have believed me they were all a bad batch, if you know what I mean )

Well, if you had both, you simply had to wait to install them again, unless you couldn't point the origin of the hardware. THAT would be a tough sell indeed, one top-of-the-line RAID controller AND 3/4/5 über-capacity (and price... lol) drives all failing at the same time... Can you say "not happening"?

Btw, one thing I've been thinking (run for the hills! lol): RAID controllers are basically dedicated CPUs and memory for the parity calculations (not even needed when reading, unless if the array is cripled). Also, it's very clear that parity calculation takes a fraction of today's CPUs computing power (see the links I posted a while back).

So, my point is, since there are usually no backup bateries on CPUs like there are on hardware RAID controllers, to keep data from being lost after a power outage, software RAID most likely calculates parity for every block written, and sends it immediately to the drive, to minimize data loss, without even considering storing it on RAM. This, of course, creates abismal performance for software RAID, when it could actually be the fastest configuration available...

So, am I too off on this? I don't think so...

Also, would it be possible to rewrite Intel or Microsoft's RAID driver to actually use system RAM as cache, before sending the data to the array? This would open up insane performance boosts on software RAID...

Cheers.

Miguel

[__Miguel_]

Btw, one thing I've been thinking (run for the hills! lol): RAID controllers are basically dedicated CPUs and memory for the parity calculations (not even needed when reading, unless if the array is cripled). Also, it's very clear that parity calculation takes a fraction of today's CPUs computing power (see the links I posted a while back).

So, my point is, since there are usually no backup bateries on CPUs like there are on hardware RAID controllers, to keep data from being lost after a power outage, software RAID most likely calculates parity for every block written, and sends it immediately to the drive, to minimize data loss, without even considering storing it on RAM. This, of course, creates abismal performance for software RAID, when it could actually be the fastest configuration available...

So, am I too off on this? I don't think so...

Also, would it be possible to rewrite Intel or Microsoft's RAID driver to actually use system RAM as cache, before sending the data to the array? This would open up insane performance boosts on software RAID...

Well, besides quoting myself (which is always weird to begin with), I have a little update on the whole "software RAID" vs. "hardware RAID" performance, and the why we see so different performances between them.

Granted, a dedicated CPU will always be better than a several times faster generic CPU. However, I've found an interesting controller which almost answers the questions I've risen on my last post.

For those who don't know what the hell I'm talking about, check here and here. The thing is, apparently there is one company which is trying to do exactly what I've been talking about: getting a generic purpose CPU to do the work of those I/O processors on the dedicated cards. The name of the company is RAIDCore, with its 4000 (apparently, not that good) and 5000 (just released) series controllers, and (most important) also the Intel (AMD on the works) driver replacement.

And guess what, its performance is not that bad, because instead of only pulling the absolute minimum CPU they can, they kinda go wild on that. If my readings are correct, the 4000 series cards could pack quite a punch, even when competing with dedicated I/O processor cards...

The only thing missing seems to be the cache. Cached writes with XOR-enabled and cache-enabled cards are through the roof, and that is very important with parity RAID levels. I've read that the RAIDCore cards could actually support cache, only there aren't any available models with that option (probably because of price concerns: a 4-port RAIDCore + software costs around €85, cache would at least double it). I just hoped there was an option to use the system RAM as cache, like IGPs do... it shouldn't be too hard, and the performance would be impressive...

So, what are your thoughts about this one? I might consider one of these, since a 4-port card with the special software is impressively cheap here (€85 is very nice, and the software itself is said to retail for $50, so...), and, combined with an ICHxR (presumably, ICH8R, ICH9R and ICH10R, because of the aparent 6-port requirement), I'd be able to go to a 10-drive array in no time, with support for a lot of interesting combos (like a kind of Matrix RAID, but with 10 drives). Only thing really missing for added safety is RAID-6, but I'm guessing it will be added to the software, sooner or later.

Also, another question. since I don't really need SATA for the system drive, since I won't be using an optic drive (except for the OS install, that is...), and since the IDE controller would be left there sitting, would it be a good idea to go IDE with the system drive? Like a WD1600AABS (IDE variant of the AAJS, I think)?

Cheers.

Miguel

[Xcel]

Very interesting stuff, I might be looking into the possibility of getting a hardware raid controller. I've kinda lost faith in the onboard controller... But I can't afford really expensive ones

[__Miguel_]

Very interesting stuff, I might be looking into the possibility of getting a hardware raid controller. I've kinda lost faith in the onboard controller... But I can't afford really expensive ones

Well, AFAIK "hardware RAID controller" and "affordable" aren't usually on the same sentence. Unless, of course, there is a "not" right before the "affordable" part... hehehe

The 8-port Areca and 3ware (as well as others) XOR-based controllers that have been discussed here are very expensive devices. Even 4-port controllers are not that cheap. The XOR engine and cache sends the price sky high...

Integrated is rather weird. Performance is also lacking, especially with random writes in parity arrays (more so because caching is not available, as I said).

The only apparently "heh" software-based RAID seems to be RAIDCore's Fulcrum engine, which allows the ICHxR ports to benefit from the same advantages of RAIDCore cards (like easy migration, even between systems, and also very nice expandability, up to 32 drives - currently only up to 14, since there are only 4 and 8-port cards available, plus the ICHxR's 6 ports).

I'd wait for some input from the experts on the subject, though.

Cheers.

Miguel