CEP2 and HDD speed.

Printable View

11-13-2010, 03:25 AM
fallwind

CEP2 and HDD speed.

I also posted this on the WCG forums but I know not everyone goes there, so here it is:

http://img688.imageshack.us/img688/9164/ssdvshdd.jpg

Just thought I'd post some results from a couple of my rigs running CEP2. The pic above shows BOINC's elapsed time on the left and BOINCviews CPU time on the right. The upper data set is from a 980x at 3.8GHz with BOINC installed on a traditional 7200RPM hard drive. The second data set is from a 980x at 4.5GHz with BOINC installed on an 40GB OCZ Vertex 2 SSD. The BOINC numbers look funny because I stretched the image a bit to get everything to line up better.

As you can see, the SSD equipped rig only loses 5-6 minutes per task for an efficiency of around 99%. The HDD rig is losing 35-50 minutes per task, for an efficiency of about 90%.

On a side note, it's only been a couple days, but so far I haven't had any problems running 100% CEP2's on the 3 machines I have dedicated to the project.
11-13-2010, 03:53 AM
Gamekiller

There's also a thread there about running it Ramdisk style. https://secure.worldcommunitygrid.or...d_thread,30244

I think it's kind of the same, right?
11-13-2010, 04:18 AM
fallwind

Yeah, pretty much. If you can swing it, a RAMdisk is the way to go since it's practically free and even faster than an SSD. The problem is the BOINC folders on my machine are currently over 7GB with a 1 day work buffer, CEP2 takes up alot of room...lol! You'd need a ton of RAM to fit that(plus a couple GB of free space) into a RAMdisk. It may be a more practical solution on a quad core with no HT and a minimal work buffer. But then again, I bet a regular HD could easily handle the I/O from 4 threads of CEP2.
11-13-2010, 04:35 AM
Metroid

My case, 8 work CEP2 units use 6 to 8GB of pagefile, 6 to 10GB of SSD and 1 to 2GB of ram. I never saw anything like. To run this properly with a RAMdisk would need 24GB of ram.
11-13-2010, 05:19 AM
haschioz

so what exactly does this tell us? a ssd is better for crunching? and the times clockspeed wise dont add up...your 3.8ghz machine is faster than it should be, maybe post a couple days of runtime and post them again with a similar clocked machine hdd vs ssd to compare.
11-13-2010, 06:18 AM
shoota

yeah i'm confused by this. i was under the impression that hard drives don't affect output
11-13-2010, 06:42 AM
Snow Crash

The efficiency issue is specific to running multiple CEP2 WUs at the same time ... CEP2 is I/O intensive.
Due to this, WCG by default limits 1 CEP2 in your que at a time.
You will only be effected by the aggregate I/O inefficiency if you went to your device profile and at the very bottom of the page told it to ignore the WCG limits.

I *think* this effect is limited to the Windows world ... they had released for Linux a long time ago.
11-13-2010, 07:53 AM
EtaCarinae

Quote:

As you can see, the SSD equipped rig only loses 5-6 minutes per task for an efficiency of around 99%. The HDD rig is losing 35-50 minutes per task, for an efficiency of about 90%.

Excellent news. I will put to test with CEP2 my new rig I am building which will have a Crucial 300 SSD (Sata 3). When you run 12 cores things add up and in absolute numbers 10% of a high number remains a high number.
11-13-2010, 11:15 AM
trn

Good findings, CEP2 WU's hammer the HD's. Probably the best way to run the project is mixed with other normal WU's vs running 8 or 12 CEP2 WU's at once.
11-13-2010, 05:47 PM
fallwind

Yes, I posted this thread as sort of a warning to XS crunchers. CEP2 has alot of HD activity and with the number of people running 980x's and SR2's here this information needs to be known. In the past you could run any old crappy HD on a beast of a rig and it would crunch just fine. That's not the case with CEP2. A mechanical drive can only handle about 4 concurrent CEP work units before you start losing CPU efficiency. Once you get to 8 or 12 of these runing at the same time, you lose alot of efficiency and risk OS crashes and WU errors.

Speaking of errors, the rig in the OP with the mechanical HD starting throwing "Task exited with zero status but no finished file - Restarting" errors today. I'll limit CEP2 to 4-5 threads till the SSD I ordered for it shows up next week.

Quote:

Originally Posted by haschioz

so what exactly does this tell us? a ssd is better for crunching? and the times clockspeed wise dont add up...your 3.8ghz machine is faster than it should be, maybe post a couple days of runtime and post them again with a similar clocked machine hdd vs ssd to compare.

SSD's are better for this particular project (CEP2). With all the others, HD speed doesn't make any difference.

Also, CEP2 work units are variable length so completion time is not and idicator of CPU speed.
11-13-2010, 06:06 PM
Nikolasz

I think boinc needs to be rewritten.
Too much hdd usage. Ramdisk is ok, is fast as memory speed......

I got single tasks takes 10gb of hdd usage, i only got 4gb fast memory other two comps that crunch 24/7 got 2 gb memory
11-13-2010, 06:14 PM
D_A

It's not BOINC it's the science application that has the issue, but CEP2 does have particularly high HDD usage for some reason.

Fallwind, correct me if I'm wrong, but I though excessive writes was particularly bad for SSDs. I'm not claiming to be an expert here, that was just my understanding.
11-13-2010, 06:19 PM
jcool

Quote:

Originally Posted by D_A

Fallwind, correct me if I'm wrong, but I though excessive writes was particularly bad for SSDs. I'm not claiming to be an expert here, that was just my understanding.

They are. For most (MLC-based) SSD anyway...
Thanks for the info. I was gonna try CEP2 on my SR-2 rig soon. Time for the SSD Raid (2x Vertex LE's) to show what it's made of :up:
11-13-2010, 06:36 PM
Metroid

SSD host writes before the CEP2 project 1.2T, after 5 days 8 work units 3.3T. :eek:

Pagefile is disabled so it uses the ram, now imagine if the pagefile was also enabled on the SSD :eek:
11-13-2010, 07:22 PM
fallwind

Quote:

Originally Posted by D_A

It's not BOINC it's the science application that has the issue, but CEP2 does have particularly high HDD usage for some reason.

Fallwind, correct me if I'm wrong, but I though excessive writes was particularly bad for SSDs. I'm not claiming to be an expert here, that was just my understanding.

You're absolutely correct. I have no clue what the effect could be on SSD life while crunching this project long term. That's why I'm equipping 3 rigs with small cheap SSD's just to contain the BOINC folders and I'll dedicate them to this project for the forseeable future. If an SSD dies, that's what the warranty is for....lol! :rofl:
11-13-2010, 08:26 PM
D_A

Since I have the sum total of ONE HDD that I bought new, I tend to get pretty good value out of them, but that also means I end up using a lot of older, slower drives. Of course if I kill one in a dedicated machine, who gives a rat's ass! :rofl:
11-14-2010, 04:03 AM
Rob_B

Quote:

Originally Posted by fallwind

I'll limit CEP2 to 4-5 threads till the SSD I ordered for it shows up next week.

Is there an option to limit that? Or do I just select other projects too and hope they get mixed up nicely?
11-14-2010, 04:10 AM
Nikolasz

Quote:

Originally Posted by D_A

It's not BOINC it's the science application that has the issue, but CEP2 does have particularly high HDD usage for some reason.

Fallwind, correct me if I'm wrong, but I though excessive writes was particularly bad for SSDs. I'm not claiming to be an expert here, that was just my understanding.

Totaly agree.
11-14-2010, 04:23 AM
fallwind

Quote:

Originally Posted by Rob_B

Is there an option to limit that? Or do I just select other projects too and hope they get mixed up nicely?

You can limit a machine to 2 CEP2 work units at a time total (in queue and/or in progress). You can find the option for this in your WCG device profiles. Other than that, there is no way to control how many come through other than manually suspending and resuming tasks as needed.
11-14-2010, 06:19 AM
SMTB1963

Quote:

Originally Posted by D_A

It's not BOINC it's the science application that has the issue, but CEP2 does have particularly high HDD usage for some reason.

Fallwind, correct me if I'm wrong, but I though excessive writes was particularly bad for SSDs. I'm not claiming to be an expert here, that was just my understanding.

So WCG applications are not bound by the "Task checkpoint to disk" setting in the BOINC software? This would explain why I was seeing constant write activity to a machine I had devoted to CEP, no matter what I set the checkpoint value to. I would also explain why it fried my USB drive so quickly.

This document from Toshiba says:

Quote:

To even begin to reach a conservative endurance limit of a 64GB MLC NAND-based SSD with wear-leveling technology, a mobile user would have to write approximately 40 Terabytes (forty trillion bytes) of data over the expected five-year life of the drive. That’s equal to approximately 22GB of new data per day, every day – or enough to fill 4.6 DVDs, or 32 CDs daily. With a 128GB drive, for example, the wear would be spread over a larger storage area, effectively doubling the average daily write limit to 44GB, or more than 9 DVDs.

Anybody have any idea what the write load per thread per day might be for CEP?
11-14-2010, 06:25 AM
EtaCarinae

On the mechanical HDD side I am running now CEP2 on 12 threads per machine on four machines Sata2. They have fast 10'000 rpm 150 and 300 GB Raptors. I run it also on one 8 core machine but that has a more slow 500 GB WD disk. In a few days 'l see how these rigs will behave. And later I am eager to compare with my new SSD rig.
11-14-2010, 05:53 PM
Metroid

My case since day 1 is 50~70% = 100%. So it does not take 10~12 hours to be completed. It takes around 6~8 hours. Is anyone similar to my case?
11-15-2010, 12:06 AM
fallwind

Metroid, yes that is normal. The work unit will progress to around 60% and then suddenly jump to 100% completion. There is an explanation for this behavior on WCG forums but I can't recall the reasoning.

==================

Just another FYI. I have also experienced OS crashing when the OS and BOINC are installed on the same SSD. This has happened both with Win 7 and Server 2003. If I put BOINC on the SSD all by itself and leave the OS on a mechanical HD or another SSD, everything is fine. This never happened with Ubuntu but Windows seems to have problems with extended, heavy disk useage.
11-15-2010, 12:34 AM
EtaCarinae

I have returned now a little over a hundred valid WUs. The Elapsed time per WU varies from 3.5 to 10 hours with all values in between. The majority is around 7-8 hours. These are running on 980X CPUs with HT at 3.7 and 4.0 Ghz with fast ram and disks.
11-15-2010, 12:44 AM
Metroid

Quote:

Originally Posted by fallwind

Metroid, yes that is normal. The work unit will progress to around 60% and then suddenly jump to 100% completion. There is an explanation for this behavior on WCG forums but I can't recall the reasoning.

Thanks :)

Quote:

Originally Posted by fallwind

Just another FYI. I have also experienced OS crashing when the OS and BOINC are installed on the same SSD. This has happened both with Win 7 and Server 2003. If I put BOINC on the SSD all by itself and leave the OS on a mechanical HD or another SSD, everything is fine. This never happened with Ubuntu but Windows seems to have problems with extended, heavy disk useage.

Strange, both my OS and BOINC shares the same SSD and I had no issues. I'm using Intel. Are you using Intel or Sandforce?
11-15-2010, 03:24 AM
fallwind

I've had crashes with both a Win 7/Intel/Indilinx combo and a Server 03/Intel/Sandforce machine.
11-15-2010, 02:05 PM
EtaCarinae

Quote:

Originally Posted by fallwind

I've had crashes with both a Win 7/Intel/Indilinx combo and a Server 03/Intel/Sandforce machine.

Interesting. I heard and read a lot about the excellent quality of the Sandforce controller for SSDs. It seems it is not so perfect.:D
11-15-2010, 05:43 PM
nanoprobe

FWIW I ran Linux on an SSD(OCZ 32GB) with an i7-860 and it crunched 8 WUs at a time without issues.
11-24-2010, 10:20 PM
josh1980

1 Attachment(s)

Holy hard drive thrashing batman!

So, I decided to do a little test with CEP2 work units tonight since I got my SR-2 rig up and running. I set up Boinc to crunch 16 threads of CEP2 simultaneously. I also installed HDDLED to monitor the read and write speeds and total bytes since last bootup. The intent was to help me win that sapphire badge asap! I also wanted to see how bad the hard drive thrashing would be.

Well, I accidentally left my computer crunching all night last night and this morning when I woke up I found that I had written 380GB of data to my C drive. The drive is an SSD, so I'm not too thrilled about my little oopsie. I decided that I'd move my boinc data folder to my trusty ANS-9010 with 32GB of RAM and set up a RAID-0. The drive is not used for anything except for Boinc data for the following numbers. All WUs were previously downloaded.

First I started up my gpugrid.net to get a baseline of how much data gpugrid.net reads/writes when it's crunching. After 2 hours the numbers were in: 350MB read, 317MB written.

Next, I unsuspended my WCG project. Within seconds 2.5GB of data was written. There was no network activity, the 2.5GB of data appears to be part of the process of starting a thread. I checked my folder size, and it had only increased by about 40MB from 2 hours before.

Then, I sat around for an hour. After an hour I took another look...

http://www.xtremesystems.org/forums/...1&d=1290665297

HOLY CRAP! Is this thing for real? 27GB!? Gentlemen, if you love your SSD, do not try to run CEP2 WUs on it. Running 16 threads appears to generate about 1.5GB per thread per hour. Perhaps these ACard RAMdrives have a very useful function for me after all. They don't "wear out" like SSDs. ;)

One thing to look at is the rate kb/sec of the screenshot. I sat and watched the write rate to see what was going on. It turns out that 8-17MB are written every second. Every second. Not intermittently, but constantly. For those of you wanting to crunch CEP2 work units, I highly recommend against using an SSD if you want your drive to have a reasonable life span. I did try changing the checkpoint time from 300 to 600 seconds, it didn't appreciably change the size of the writes.

I considered enabling the NTFS file compression for the boinc data directory. The files do compress quite well(about 40% saved). However, the extra CPU usage would slow down the performance of Boinc WUs.

One big penalty is taken when you suspend and restart the threads(such as during a reboot). For whatever reason, 2GB+ gets written to my drive every time I reboot or when I unsuspend the WCG project.

For those of you that DO use SSDs, I'd highly recommend minimizing the number of times you allow Boinc to suspend your tasks and try to minimize your reboots.

I'm leaving my cruncher on tonight to see what happens to the totals for reads and writes. Stay tuned!
11-25-2010, 03:19 AM
fallwind

Yep, the read/write activity is pretty crazy with CEP2. I'm using SSD's in 3 machines running this project, but I installed them as secondary drives, the only thing on them is BOINC, so the primary drive is spared the thrashing. I caught a good sale on the SSD's so I'm not too worried if they die.
11-25-2010, 04:18 AM
Metroid

1 Attachment(s)

Quote:

Originally Posted by josh1980

Well, I accidentally left my computer crunching all night last night and this morning when I woke up I found that I had written 380GB of data to my C drive.

I disabled the CEP2 project for a week, planning to start again in about 2~3 weeks.

My case was not different:

Quote:

Originally Posted by Metroid

SSD host writes before the CEP2 project 1.2T, after 5 days 8 work units 3.3T. :eek:

Pagefile is disabled so it uses the ram, now imagine if the pagefile was also enabled on the SSD :eek:

9~10 days 24/7 CEP2 8 work units.
11-25-2010, 04:36 AM
JimF

Quote:

Originally Posted by Metroid

SSD host writes before the CEP2 project 1.2T, after 5 days 8 work units 3.3T. :eek:

Pagefile is disabled so it uses the ram, now imagine if the pagefile was also enabled on the SSD :eek:

How much RAM do you have? I run a quad-core, with 8 GB RAM and a generation 1 SSD. It runs up to 4 CEP2 work units at a time, and while I see a lot of disk activity, it is nothing like that. I have the pagefile set to 1 GB minimum, and it has never increased beyond that. But I don't use that PC for anything except WCG and Folding@home, so no games or even web browsing.

And I am using Win7 64-bit, which can make use of all of the memory. Are you using a 64-bit OS?
My CPU efficiency on BoincTasks shows up at 96% on CEP2 jobs, so it is not thrashing much. But it took a firmware update on my SSD to get that.
11-25-2010, 06:05 AM
Metroid

1 Attachment(s)

Quote:

Originally Posted by JimF

I have the pagefile set to 1 GB minimum, and it has never increased beyond that.

Pagefile = hard drive space and speed + latency. The pagefile does not increase, in your case if the pagefile is used in full then the OS will use the ram memory but it will not tell you that the ram is being used and yes cached. It is good to leave the pagefile to a minimum, ram is faster than pagefile / hdd's and most ssd's. Only drawback for some programs is that ram is volatile and pagefile is a written cache.

Quote:

And I am using Win7 64-bit, which can make use of all of the memory. Are you using a 64-bit OS?

Yes.

Quote:

How much RAM do you have? I run a quad-core, with 8 GB RAM and a generation 1 SSD. It runs up to 4 CEP2 work units at a time, and while I see a lot of disk activity, it is nothing like that.

I'm using CrystalDiskInfo http://crystalmark.info/software/Cry...o/index-e.html to check host writes.

Commit is the ram being used.

Cached ram is there if needs to be used, the process may or may not have being killed.

Available - Cache = Free

I would be grateful if you could check how much cached ram you have while using BOINC with 4 CEP2 work units. In my case used to be between 800~1.280MB of cached ram per work unit.
11-25-2010, 06:33 AM
JimF

Quote:

Originally Posted by Metroid

Pagefile = hard drive space and speed + latency. The pagefile does not increase, in your case if the pagefile is used in full then the OS will use the ram memory but it will not tell you that the ram is being used and yes cached.

I have the maximum set at 4 GB. If it needs more, it should use more.

Quote:

I'm using CrystalDiskInfo http://crystalmark.info/software/Cry...o/index-e.html to check host writes.

I just tried it, but don't see much info on my SSD. Maybe because it is Gen 1, running in IDE mode?

Quote:

I would be grateful if you could check how much cached ram you have while using BOINC with 4 CEP2 work units. In my case used to be between 800~1.280MB of cached ram per work unit.

I am not sure how to get the ram per work unit, but I am running 2 CEP2 jobs now (along with CCW and HFCC), and Task Manager shows 2090 MB cached. Is that what you want?
11-25-2010, 07:07 AM
Metroid

Quote:

Originally Posted by JimF

I have the maximum set at 4 GB. If it needs more, it should use more.

The file is hidden in your hdd or ssd. It may only use more if you set it to System Managed Size which I would not recommend. If you set the pagefile then the Pagefile will not increase as it goes.

Quote:

I just tried it, but don't see much info on my SSD. Maybe because it is Gen 1, running in IDE mode?

Strange, could be, probably somebody that also has the generation 1 could help us with this. I'm using the AHCI mode.

Quote:

I am not sure how to get the ram per work unit, but I am running 2 CEP2 jobs now (along with CCW and HFCC), and Task Manager shows 2090 MB cached. Is that what you want?

Yes, that is enough. Now when both tasks get 100% don't set any other CEP2 work units, set others units now go to the task manager, you will see that your cached ram has decreased significantly, so that amount was freed from those processes.

That is just the ram. SSD used space = almost the same amount of cached ram.
11-25-2010, 07:32 AM
JimF

But I can look at disk activity in Resource Monitor, and the writes seem to be averaging about 1 MB/sec. Assuming it would be about 2 MB/sec for 4 CEP2 jobs, that is about 200 GB/day, which is nothing to sneeze at, but probably won't kill the SSD anytime soon. (There is in addition a lot of disk activity for about 5 minutes after a CEP2 job starts, but that can't affect the total much.)

I think the write endurance on these flash chips is something like 100,000 writes per cell (it probably goes down for the later generation drives), so with a 30 GB SSD and good wear-leveling, I should be good for 15,000 days or 40 years. There are factors that can affect that either way, but it should be long enough for my purposes.

PS - I have left out write amplification, which will reduce this by a factor of maybe 10 - I don't know what it is for these drives, but on the other hand most flash chips last well beyond the specified value.
11-26-2010, 09:02 AM
josh1980

Well, I left my machine unattended for 12 hours and racked up an amazing 359GB written to the drive! I'm going to leave CEP2 for this machine only and disable it on all of my other machines. The hard drive usage is just... crazy.

-Josh
11-26-2010, 04:16 PM
JimF

One way of reducing writes is by use of a write cache. And writing in larger blocks should help my Gen 1 SSD cope with all the I/O demands better (I hope). At any rate, FancyCache is in beta now, but free to try out for 90 days.
http://www.romexsoftware.com/en-us/f...che/index.html

One user reports that the use of delayed writes will extend the life of his SSD quite a bit:
http://www.romexsoftware.com/bbs2/en...php?f=25&t=647

I have used it thus far with limited success, since it likes to BSOD under certain circumstances that I won't go into here, but am going to try it again with different drivers. If you want to cache only a single volume (one partition on one disk), they recommend the Volumes version. If you want to cache all the partitions on the disk, or more than one disk, use the Disks version.

It is also discussed at some length on the OCZ forum with respect to the Vertex 2 by the way.
http://www.ocztechnologyforum.com/fo...ght=fancycache

Good luck, and remember that it is in beta.

PS - It works now, using the Intel SATA (IDE) drivers instead of the Microsoft ones, and probably more importantly using the Disks version rather than the Volumes version. It was suggested on their forum that the Disks version might be a simpler algorithm, less prone to crashes. But whether it actually reduces writes on CEP2 is another matter, since that will probably depend on whether new data can update old data in the cache before it is written to the disk, or various other factors that I know nothing about. But at least it should help consolidate the writes for less "thrashing" when multiple CEP2 jobs are running.
11-27-2010, 06:43 PM
josh1980

I'm currently at about 36 hours and 1.21TB!