Guys, here is a challenge for you...regarding hard disks

[Anusha]

OK. This is the problem. You guys have to come up with a very good hard drive setup + data management solution, just like how they have the CPU caching system. Not very expensive (I can't give you a budget though), but yet again, very good performance very close to having the best performance from a PC hard disk setup.

Say for example, you go with 2 500GB Samsung F1 RAID-0 setup (i.e. the drive setup), partition it to three; one for system (Windows), one for programs and one for all your data (i.e. data management solution).

I guess you guys got the point. Show me/us what you propose.

Note: One requirement is be able to multitask very efficiently. Say for example, you are copying a huge file between disks or partitions, and it should give a minimum performance hit when doing other things simultaneously such as browsing a directory.

[Anusha]

Maybe if you guys don't have a clue, help me out with pointing where I can improve with my current setup. (All these researching is for my new complete rebuild)

Hard drives:

1. 250GBx2 in RAID-0
2. 500GB

The data is arranged in this manner. I follow the Vista's user profiles arrangement to the point

1. RAID volume (500GB)

1.1 C: drive (50GB)
Windows and Program Files

1.2 D: drive (200GB)
Games

1.3 E: drive (250GB)
User Profiles (everything except Public)
- I have the virtual machine hard drives stored in My Documents; which is pretty much the biggest disk accessing files.

2. Data drive (500GB single drive)

2.1 F: 500GB
Public profile, Backup images of C:

All my music files are in F:\Users\Public\Music
All my video files are in F:\Users\Public\Videos
All my picture files are in F:\Users\Public\Pictures
All my files that are currently being downloaded are in F:\Users\Public\Download
Almost all my documents are in F:\Users\Public\Documents

Show me areas where I should improve on.

P.S.
The only issue I see is where the individual user profiles saved in the RAID volume, which makes the hard disks have to do more seeks to load data from the user profiles. But anyway it is the case with the default installation of any OS, right? But the problem is, I can't take out the user profiles from E: as there is a lot of free space there which will not be used then. Sucks to have 250GB drives as the minimum + best for the system drive. A 74GB raptor would have done the trick, but...

[IanB]

OK. This is the problem. You guys have to come up with ...

Uh, why? Hasn't this been covered over and over in this forum?

But for what it's worth:



The arrows show the direction of information requests. As long as those don't point to partitions on the same physical disk, you'll get max speed by minimising seek times, and data in those requests can stream in parallel as it is going through different physical on-disk controllers. Each "drive" in the diagram could be a multiple-disk RAID array for even better throughput. If you aren't using an offboard RAID controller card, consider mirrors or RAID 0 for all for 6 disks total. The initial partitions (C/P/T) on each drive are relatively small (10-15% max) so you get a short-stroking bonus accessing data on those partitions for even faster access times for the most critical files, OS and program data.

Now just work out the amount of archivable data you want to store of each type (frequently accessed eg. torrents, infrequently accessed eg. backups, etc.) and that gives you the minimum sizes of the disks you need, as long as you keep those initial partitions no more than 15% of the total.

[Anusha]

@IanB

What do you think about my current setup in post #2? Should I take out the RAID and go ahead with the three drive scheme you've shown?

Wouldn't it be beneficial to only keep Windows in on drive (say a 37GB raptor), Programs and Games in another (74GB raptor), all other data maybe partitioned in to 2 for frequency of access in one huge 1TB or even a RAID-0ed big volume?

[IanB]

It seems you've got things rather about face there to me. I'm assuming that your F: drive is providing frequently accessed files to remote users (you mention downloads), so that is the drive that needs to have most availability and redundancy. It rather depends on the amount of data and the amount of money you have spare to throw at the problem, but a reasonable solution for the data you seem to have and the usage would be to use 2 disks as a RAID 1 mirror for your F: drive. This is at least as fast on reads as a single disk, provides good redundancy and may be as fast as a RAID 0 on reads with a good controller. However, if money is no object and you are providing a sharing service which must have very low downtime, upgrade to a good offboard controller card and go RAID6 (needs at least 4 disks) for more streaming speed with Enterprise-quality disks (Seagate ES or WD RE series) - 500GB ones are about the best bang-for-buck right now.

So you'll need at least one more 500GB drive to match your current one to make the mirror, or a new (faster? larger for expansion?) pair to make one from scratch. Remember you'll need at least one more 500GB+ drive to backup the data temporarily while you make the mirror, as you can't mirror an existing drive without losing all the data on it. Given the plummeting cost of 1TB drives, you're probably better off getting a pair of those, providing future expansion and backup space for other drives.

I'd then use the 2x 250GB drives as a second mirror to put only your games on. Same rationale: at least the same read speed as a single drive, maybe twice as fast, which is more important than the write speed. That's going to be the best way to reuse your smaller drives and makes sure they are set up redundantly if they are older to provide against failure. You also have a little expansion space there.

That then leaves the OS/apps drive. Short-stroking a larger drive gets you close to the same access times as a Raptor on this critical partition, and you then have lots of spare space for backups of the other drives. Redundancy won't hurt, even when the other drives are mirrored. I'd RAID 0 your 500GB drive with another identical one, so you are using the fastest first 10% of this set for your 50GB OS partition. Then add your user profiles partition (backed up on the F: RAID set) and use the rest as backup space.

So total extra cost would be 1 new 500GB to match your existing one and a matched pair of 750GB or 1TB drives depending on what you can afford. You'll need a mainboard with 6 RAID-capable SATA ports, or an offboard controller if you can afford it, but a newer ICH9R or ICH10R mainboard will be more than adequate for RAID 0 or RAID 1 with two disks in each set. 4 x 500GB Enterprise drives for RAID 6 would give you the same storage (at about the same cost) as 2 x 1TB drives in a mirror, but faster, so if you can afford the controller card go for it! You may not find RAID 6 available in anything less than an 8-port card, though, which would be very pricy.

EDIT: RAID 6 is safer than RAID 5 always, but for the same 4 disks RAID 10 (1+0) is an alternative with similar redundancy and identical price. I'm not sure of the relative risks of failure but I suspect the RAID 10 may be better and faster. No doubt stevecs can confirm that, if it isn't in the RAID sticky here. You'd also only need an extra 4-port controller if you don't have 8 RAID ports available on your mainboard, much cheaper...

[Anusha]

IanB thanks for the reply. Like I said, I am not going to upgrade my setup for THIS PC. This is just research information for the next PC.

I think I need to clear up one thing. I do not share files over the network normally. I used to; but not anymore. I keep the data in the Public profile because,
1. I like to stick to the Vista's shell folders structure AND
2. Sometimes I log in as other users, if something bad happens, and I need quick access to all the data. Public folder comes up on everyone's explorer quick links, so that works AND
3. there are very seldom cases that you might need to create a new use profile due to corruption and I don't want my *data* in that profile get deleted due to a mistake by me.

Please also note that I do not use a pagefile, as I've found with 4GB RAM and the things I do, I do not need a page file. (The only time I needed a pagefile was when I ran Crysis in DX9 mode with all enthusiast. Vista doesn't seem to like me using DX9; the RAM usage skyrocketed and was using 3.97GB in total once!!!)

The F: drive is where all my downloads go into (\Public\Downloads) and since I run the torrent client all the time, I don't want those disk accesses come in between my usual day to day tasks. Also better to separate it from the system drive to lower the deterioration of the disks.

Ah also, my 500GB drive is a Samsung 501LJ which is not the fastest drive out there. Here's a HDTune performance graph of it.



Here's the performance graph of the RAID-0 volume.



These two 250GB drives (HD250HJ) are very fast, faster than 250GB F1s (HD-252HJ) in transfer rate but a tad slower on the access time, but few hundred microseconds. hehe. (But I've seen my drives perform way faster on ICH9R; I'm on ICH8R)

Also, I use Diskeeper to defragment the drives (Set it and forget it), and I enabled I-FAAST on all the volumes to tweak the file locations on disk according to the usage patterns. But this sends some of the files to the slower end of the partitions, as they are hardly used (for example, I think the speech recognition files in Vista are sent to the end, as I've never used it - but I don't really know how it works, other than what they say in the *help*) Do you think using some other defragmenting tool will give lower seeks? PerfectDisk for example packs all the files towards the faster end of a partition, but there is no intelligent file placement as in Diskeeper afaik. Both of them (intelligent placement and packing files towards faster portion) are needed to lower the seek times IMHO.

Anyway, I think keeping the Public folder as it is in the F: drive + using Diskeeper gives the same seek performance of separating the files manually into partitions.

You are anyway proposing to get a large drive or drives and make a small partition for things that speed matters, right?

I guess the current data management place I'm having is good enough to the ideal case you've shown in the image.

What about a setup like this:
1. 150GB Velo Raptor for Windows and infrequently used data (say about 25GB for Windows and rest for data)
2. 150GB Velo Raptor for Programs + Games (partition it into two, say 50GB and 100GB and put programs in first partition? Since I don't play games all the time, and programs should get priority.)
3. 1TB Samsung F1 or anything fast enough as data drive (I will have the whole C:\Users in this drive as G:\Users?) I don't like to RAID-0 my data drive, because in case the array fails or i have to change the mobo, I will lose the data...unless backed up, but then again restoring them is a pain in the ass!) More concerned about the latter.

[NikoKiko]

@IanB

Hey IanB, I am thinking of following your recommendation here,
I'll be using a raid0 + another hard drive + 4-5Gb ram disk (have 8Gb of ram)
So i'll put the Temp/Pagefile on the ram disk but I was wondering should I put the OS on the raid0 and the "Program files" on the other hard drive or the opposite.
Since it looks like you have plenty of experience with this what would you say ? Which would benefit more from the raid performance ?
Thanks in advance!

[IanB]

Depends on the apps and data you are storing. If you have big apps (games) that need the streaming speed for loading a RAID 0 would give you, then that would be the better option. Otherwise the OS on the RAID should give snappier performance overall. You should also take the relative access times of the disks into account: the OS should generally go on the faster disks, since lots of small files loading/saving will be affected by slower access times more than outright streaming speed.

Remember to put the Internet cache in your RAMdrive, and point all your TEMP/TMP environment variables there too.

[khamsin]

Previously, I used to run 3-4 drives, all partitioned with careful distribution of data etc. Didn't help in a noticeable way. Now I use only a pagefile partition, the rest are 'whole' drives.

Currently, my setup is as follows
0: 640 GB: OS + program files.

1: 500 GB: Videos i.e. DVDs AVIs etc. Also music and some random collections of images. Most frequently used files also go here.

2: 500 GB: 10 GB partition for swap file. The rest for my non-sensitive and relatively static research data/school files. Torrents and random stuff.

3: 500 GB: Files from DSLR- RAW, TIF and JPEG. Destination folder for transcoded videos (ipod touch) from drive 1.

4: 250 GB: Truecrypt encrypted drive for important files - personal finances, medical records, proprietary scientific data etc.

Photoshop is set to use drives 1/2/4 as scratch drives.

The file system on all drives except #4 is maintained in superb shape by Diskeeper 2009 Pro running full automatic defrag mode. Barely any fragmentation at any given time. I-faast sometimes shuffles things around on a couple of the drives. I don't even check these days.

So, IMHO having large separate physical drives with the filesystem in good shape, is much more beneficial than partitioning. I could be wrong, but depending on how the logical drive corresponds to the physical drive, partitioning may make things worse in some cases by needlessly making the actuator arm travel a greater distance to pick up the files.

EDIT: I am not using any sort of RAID. I don't have enough confidence in RAID 0, and I don't have the inclination to deal with multi-level RAID solutions.