Multi-HD vs FILE Configuration

[ValueDriven]

Hello All. Pre-Thanks for your input!

I am building a new computer around an Intel Q6600 cpu. I have questions regarding the hard drive usage (File vs HD configuration). My main high computational requirement programs are video editing and photo editing (both non-professional). I do not game or watch movies on my computer at this time. (That's what large screen HDTV is for, just ask the kids. )

My question concerns optimizing the hard drive & FILE configuration, & prioritizing the multiple HD's relative speeds (assuming if not all the same) to the most important FILES. The idea is to minimize simultaneos system/program/data read/writes which become a bottleneck to processing times.

Trying to understand the available information, would this be at least a desirable, if not optimal, configuration:
1 fast drive for the OS. (Not too small as some applications require/default unchangeably to residence on the boot drive);
1 fast drive for the programs;
1 larger fast drive for capturing movies, (& storing data although speed necessity questionable for this storage?);
1 larger fast drive for processing the movie & picture data, aka the scratch disk;
1 (possibly small?) very fast drive for the OS page file (OS virtual memory).
All drives are PHYSICALLY separate, i.e., not partitions, except for possibly separating stored data from captured data on the same physical HD. A "physical" drive could be a multi-HD RAID drive.

As this would appear to require at least 5 physical drives, is there an alternative using fewer but close in efficiency? Is there a function I've missed that should be parsed out to another separate drive?

Which files should get fastest drives? I'm not interested in Computer or program startup times, just program processing, file copying, etc. times. Where would RAID help the most, or where should it be avoided due to the type of READ/WRITE activity?

Any suggestions on approx sizes, if necessary, is also welcome. (But, please leave discussion of various brand vs brand, or specific drive vs specfic drive discussion to a different thread.)

Obviously, VALUE is relative to each individual, so if we could keep the discussion focused on the technical configuration. However, to mention the relative gain or loss in performance for a relative cost or savings has technical merit, without making a value judgement for someone else.

FYI, I currently have purchased 2 150GB WD Raptors so far, & will be using WINDOWS XP PRO as the OS. I am considering purchasing one or more new drives in the 500GB+ range. But could change depending on your responses & my pocketbook! This could become my goal, rather than my immediate reality.

Thanks,

[james bennett]

Hard drives are the single slowest components on a computer if you have sufficent memory and a good processor. Striped volumes are the best set up for the fastest read/write speeds (ideally have to be set up on identical drives). If it were me I would make a partition for the O.S. (50gb) seeing as you cannot put the O.S. on a striped vol.. Set up a striped volume between the two raptors. Set up your page file on the stiped vol. and you have 195gbs left over for apps etc. If it weren't for the fact that you have to store movies I'd say you should be fine with what you have or maybe one more drive. Movies take ALOT of storage so, you need at least 750gb and you want that on a striped vol. Raptors don't have that much capacity. Another hard drive that I am very impressed with is the samsung F1. You can get 2 750gb drives for $300. By speading partitions over multiple drives files are being read/wrote by multiple drive controllers.
P.S. raptors are sweet, other than my memory upgrade getting a raptor was one of the most dramatic inprovements in performance for my system.

[Serra]

Trying to understand the available information, would this be at least a desirable, if not optimal, configuration:
1 fast drive for the OS. (Not too small as some applications require/default unchangeably to residence on the boot drive);

Sounds fair... most users are pretty happy with a single raptor for their main drive, especially when other programs are set off to other drives. Realistically though, if you aren't keeping any sizable programs on this drive, you can probably save some money here... I mean, all you'll be using it for is accessing shortcuts to programs on other drives and booting up, right? A raptor will give snappier response, but if you have no programs loaded here... it might be money that you don't really need to spend.

1 fast drive for the programs;

Choice of hard drive for this will depend on what the programs are and how you use them. For example, if you're a hardcore gamer then (depending on the game) you might want a RAID-0 array of raptors. If you're just going to load Internet Explorer, office applications, music/movie programs, etc... then there's really no benefit. Again, slightly faster response, but beyond the 1/10-second first seek, as you wouldn't be doing any time-critical additional seeks after, it would be smarter to get a drive that does better in throughput. More elaboration is probably needed here.

1 larger fast drive for capturing movies, (& storing data although speed necessity questionable for this storage?);

This should be pretty standard... just a large extra drive. Samsung F1 if you can afford it/find one... really any kind of 500+GB drive should be fine though.

1 larger fast drive for processing the movie & picture data, aka the scratch disk;

Disc suggestions depend on what you're going to be doing. If you're working rapidly with a lot of small files, you may as well get a 36GB raptor. If you're just going to rip the occasional movie, I probably wouldn't bother getting anything more than a cheap 7200rpm drive. Size considerations will also be very dependent on what you're actually doing.

1 (possibly small?) very fast drive for the OS page file (OS virtual memory).
All drives are PHYSICALLY separate, i.e., not partitions, except for possibly separating stored data from captured data on the same physical HD. A "physical" drive could be a multi-HD RAID drive.

Unless you are actually aware of a need for this drive, I'm sure you can do without it. Given you'll have one disc dedicated for use on opening icons, you'll have plenty of idle time on that disc. If you really want to be proper about it, get a raptor for your desktop and partition a space at the start of the disc (which is physically the outside, but logically the start) to use as the page file. A 4-6GB slice won't noticeably reduce your desktop usage and should be amply sufficient.


So if you can give us more details on how you access programs (programs, usage patterns, etc) we might be able to find something a little more definite for you.

And also...

[IanB]

1 fast drive for the OS. (Not too small as some applications require/default unchangeably to residence on the boot drive)

I am curious - which apps are you aware of that won't allow you to specify on which partition they should be installed? I can't think of any, except for OS-bundled apps like IE and OE. Anything of such an immense size that would make your fear of a boot drive being too small real would have a professional installer with the option to change the install location.

You are then left with the size of the OS only, plus any pagefile if you think it necessary, and for XP (not a lite version) that would leave you substantial change from 10GB even WITH a fairly well-loaded (OK, no games or Office) C:\Program Files - just how small a drive are you thinking of getting? Apart from Raptors, you can't buy anything under 250GB these days in PATA or SATA. So this idea that ANY drive might be too small for the OS is nonsensical.

Also, although this subject causes vigorous debate every time it comes up, while the pagefile should best be on a separate disk/spindle set from the OS, if you have sufficient RAM you can reduce it to absolute minimal size or remove it entirely. "Sufficient" is what is required to hold your working set of programs adn their data in a normal session plus some spare. I gave some detailed advice about how to find this number in a post that has apparently been pruned off this forum. The short version is: run a normal session doing whatever tasks you would normally do, then before shutdown open up Task Manager and look at the number for "Commit Charge - Peak". This is the max amount of Virtual Memory used by the system during the session. If this is lower than "Physical Memory - Total" then you don't need a pagefile, or certainly not a large one, so reduce it to minimal size. If it is larger than the physical memory then the difference is a guide to the actual amount of pagefile you need, just make sure to add some extra for spare. It depends on the RAM you have installed and the apps you use, there's no one answer for everyone.

[jcool]

I agree with serra, you don't need that extra disk just for the page file. I'd rather put that on your OS disk and go for something fast there.

Be sure to have an array with high read/write rates and LOTS of room for your processing drive, best combination of value, performance and security would be running a raid 10 with Samsungs 750GB F1 drives. Each of them gives you approx. 90MB/s read/write so you can expect quite some throughput when running 4 or more of these drives together.

[james bennett]

[If this is lower than "Physical Memory - Total" then you don't need a pagefile, or certainly not a large one, so reduce it to minimal size.

You are right to a point, he would be better off monitoring the page file over a long period of time. There might be some instances that occur only occasionally (virus scan, defrag, running multiple apps simultaniously) that would only be caught by long term monitoring. I would rather have a little extra Page than fall short. I know I have way more than I need but, the space is there and I would rather have it and not need it than need it and not have it. Then again I'm running XP X64 and it utililizes more virtual memory.

[james bennett]

I you can expect quite some throughput when running 4 or more of these drives together.

I would completely agree that getting a seperate drive just for page file is COMPLETELY unnessary. I would also agree with the samsungs (four), if it weren't for the fact that he already has two raptors. I guess he could try and sell them if push came to shove.

[jcool]

I'd use one of the raptors for the OS and one for the programs, or do a mirror or stripe for both. The capacity of these raptors obviously won't be enough to use them for capturing and processing video.

[james bennett]

Yep me too, I'd set up the O.S. on one and configure the remaining space between the two as a striped vol. (mirrored would only provide fault tolerance). I would put the page file and as many apps as I could on the striped one. Oh yeah, about the videos and the editing. Thats where he would need more storage. Like you said a couple of samsung F1's

[IanB]

Yep me too, I'd set up the O.S. on one and configure the remaining space between the two as a striped vol. (mirrored would only provide fault tolerance). I would put the page file and as many apps as I could on the striped one.

You defeat the value of the Raptors that way. The benefit of having a separate drive for each "function", such as OS, apps, pagefile etc., is that the drives can seek for data concurrently and stream data in parallel. That way the perceived access time and streaming speed is improved for the end user.

If you put the OS on one disk, leaving some space spare, and then stripe the remaining space with apps and other frequently accessed data like a pagefile, you now have the drive seeking two or three parts of the disk at the same time if it gets a request for app data and OS data and pagefile data, pretty typical in normal usage like loading an app. This means the heads are all over the disk surface and your seek times crash, massively affecting the perceived disk response. Also, you are now streaming two or three data blocks through a single physical disk interface, so instead of getting the best speed that disk can offer for one of the streams, you get a half or a third for all.

If you must use the remaining partition so as not to waste space, then probably best is to mirror instead and only place there archive data, that's being rarely accessed. That way there's little or no regular contention with the OS data access and you don't lose speed where you need it most. This is where Matrix RAID, or a similar controller function, works well - place a small fast striped partition or mirror at the drive start to maximise the regular seek times for OS or apps by short-stroking the drive, then put infrequently accessed data in the remaining space, RAIDed or not, so it doesn't conflict all the time. Using it for anything else will just waste your effort.

[james bennett]

[QUOTE=IanB;2797473]You defeat the value of the Raptors that way. The benefit of having a separate drive for each "function", such as OS, apps, pagefile etc., is that the drives can seek for data concurrently and stream data in parallel. That way the perceived access time and streaming speed is improved for the end user.

I never thought of it that way. I think I'm going to change my systems set up. I only have one raptor and one samsung so, wanting to take advantage of the raptors speed, I put my page file on a striped vol. between the two. I wish I had the cash to geet a couple of more raptors but, in the mean time. Do you think it would help my performance more if I used the samsung for paging? I mean I have 4gigs of physical memory how often can i be using the virtual memory.

[virtualrain]

[QUOTE=james bennett;2797648]

Do you think it would help my performance more if I used the samsung for paging? I mean I have 4gigs of physical memory how often can i be using the virtual memory.

I monitor my physical RAM usage and my PAGE file usage and right now with 4GB of physical RAM it shows I'm using 1.3GB of physical and 2.6GB of page file.

This is because Vista tries to balance the use of physical RAM between apps, disk cache and superfetch. It may decide that some of your physical RAM is better utilized loading an often used APP into memory and paging out some crap you haven't used in awhile. XP is more crude in this regard.

As for disk configurations... here's what I would suggest... in the interest of parallelism, you want 3 different physical drives...

1) OS and Apps. The OS is used to boot, the apps are used after boot. No sense putting these on different drives since they aren't accessed at the same time in general.
2) Files. This is where your video and pictures live
3) Scratch/Page. This is where temp crap will be written to in the editing process. Partition the first 10GB of this disk and use it for your page file for all drives. Use the other partition for scratch. In general, the data written to/from this disk will be in small chunks so performance of the HD isn't as imporant here (since most will have ample 8-16MB cache anyway).

If money is no object, get an Areca 1220 with 8 SATA ports and put each of these disks on a RAID-0 array with dual 150GB raptors each.

If this is too rich, at least get an Areca 1210 with 4 SATA ports and run #1 and #2 on RAID-0 arrays and use a 5th drive (Non RAID) for #3. This is what I do. The cache on the Areca card makes a huge difference... it's worth buying this card just for the cache, even if you don't use RAID. If the Scratch disk is really a bottleneck, then you could opt to put that on RAID0 on the Areca and run your OS/Apps on a stand-alone drive. I'm more of a gamer so the OS/Apps gets priority for me.

A couple of larger 500GB drives in RAID1 could be used for archival but these could be on a NAS, another PC, or in the same chassis... it doesn't matter because you will likely only backup to these overnight over GigE and after the first backup the time will be insignificant.

[james bennett]

Man that is sweet. I am a noob when it comes to xtreme systems or computers for that matter. I pick up alot here and there. Thats a server case right or did you build that youreslf? I've only been at this persuit of power and performance for about 10 months. I think I'm doing alright but, oh if I only had money to burn. I'll get one conponent and then something better comes down the pipe.

[virtualrain]

Man that is sweet. I am a noob when it comes to xtreme systems or computers for that matter. I pick up alot here and there. Thats a server case right or did you build that youreslf? I've only been at this persuit of power and performance for about 10 months. I think I'm doing alright but, oh if I only had money to burn. I'll get one conponent and then something better comes down the pipe.

Thanks... it's a Lian Li PC-343B cube case. Not sure what the original market was but it's a water-cooler's dream.

[iadstudio]

If it were me I would make a partition for the O.S. (50gb) seeing as you cannot put the O.S. on a striped vol..

Not true. I had xp and Vista both on a 3 drive RAID 0 stripe.

[james bennett]

Not true. I had xp and Vista both on a 3 drive RAID 0 stripe.

How do you install it. I thought I tried it once and it didn't recognize my striped volume. I would love to try it once I can afford some more drives.

[virtualrain]

How do you install it. I thought I tried it once and it didn't recognize my striped volume. I would love to try it once I can afford some more drives.

You need to select "Load Drivers" at the HD selection screen in Vista or with XP press F6 during setup to load drivers. You then point it to your RAID drivers on a floppy (XP) or some other media (Vista).

[james bennett]

[QUOTE=virtualrain;2797736][QUOTE=james bennett;2797648]
get an Areca 1210 with 4 SATA ports

I have a feeling what the answer will be but, say you had a budget of $400.Which would you choose to buy an Mtron 16gb SSD or areca 1210. Personally I'm partial to the SSD and then in the future getting the areca.

[virtualrain]

[QUOTE=james bennett;2800076][QUOTE=virtualrain;2797736]

get an Areca 1210 with 4 SATA ports

I have a feeling what the answer will be but, say you had a budget of $400.Which would you choose to buy an Mtron 16gb SSD or areca 1210. Personally I'm partial to the SSD and then in the future getting the areca.

Hmmm... are 16GB SSD's available now for $400? That's tempting... Although my current bloated Ultimate x64 install with all my apps and just a couple of games is consuming about 64GB... so for me I would get very little benefit from just a single 16GB SSD. If I could get a 64GB SSD for $400 it would be a no-brainer in favor of the SSD.

[james bennett]

[QUOTE=virtualrain;2800631][QUOTE=james bennett;2800076][QUOTE=virtualrain;2797736]

Hmmm... are 16GB SSD's available now for $400?

I looked it up last night, from the neostore.com, they are $385 for the MSD 6000sata 3.5". The issue of only being 16gb isn't a problem with me because when i back up all the data on my system it is only 14gb. The only game I have on it is call of duty 4 right now. I figured that should do until I am able to afford another SSD. I know I'll regret commiting to SSD's when this new fusion Iodrive becomes available. Then again I could always put them into my other computer.

[jcool]

[QUOTE=virtualrain;2800631][QUOTE=james bennett;2800076]

Hmmm... are 16GB SSD's available now for $400? That's tempting... Although my current bloated Ultimate x64 install with all my apps and just a couple of games is consuming about 64GB... so for me I would get very little benefit from just a single 16GB SSD. If I could get a 64GB SSD for $400 it would be a no-brainer in favor of the SSD.

You can get a 73GB 15k.1 Savvio for 400$
Will need a decent SAS controller though...

[james bennett]

[QUOTE=jcool;2802248][QUOTE=virtualrain;2800631]

You can get a 73GB 15k.1 Savvio for 400$
Will need a decent SAS controller though...

What kind of read/write stats does it have and what is the access time. I know that it cant be .1ms like an mtron.
I just looked it up pretty impressive but, as you said it leaves you still needing to spend more on a SAS controller. The money you put towards that could be going toward my second Mtron.

[jcool]

No, of course not. It's still a mechanical HDD, but the quickest for that matter. Fastest mechanical hdd available when it comes to access and IOPS, this is made on a REAL crappy controller (Promise TX2650, cost like 50 bucks and it's not even worth that much):

[james bennett]

Thats pretty sweet until you filled me in I though I had the fastest hard drive. The read/ write is about the same as the SSD I'm looking at, the clincher is the access times. This is a never ending battle though. (trying to stay on the cutting edge).I know that a few months down the road I'll be wanting a fusion Iodrive.

[jcool]

Jap, I love this drive. Just have to get a decent controller soon, this cheapo-promise is getting me nuts. It needs 3 minutes (!) to boot the hdd and it gives a horrible write performance.
Fastest "traditional" HDD in terms of access and IOPS is the savvio 15k.1 as mentioned, if we're talking transfers then its the brand new Cheetah 15k.6 300 (considering that the "old" 15k.5 delivers an average read of 135MB/s the 15k.6's performance will be sick)
The savvio does have a few benefits over the others though, being a 2,5" drive has its benefits:

- very quiet for a 15k drive, put it in a scythe quietdrive and it becomes inaudible (not XS-inaudible but _really_ inaudible)
- consumes less power (around 8W seek) hence less heat

It's made for 24/7 full-load operation, so if you do a lot of multitasking and stuff you can be certain it won't fail you after a certain period of time (the main reason against SSDs imo)