Linux vs. Windows on HCC - a study

[jcool]

I originally posted this in artemm's thread, but since he actually did something different (HCC-only vs. all projects on Linux) I thought this deserved its own thread.

What I'd like to do here is find out which OS is best for crunching. This is done by comparing data from one or multiple machines. I started off by calculating some numbers for you - Windows 7 x64 vs. Ubuntu 10.04 x64 with no changes to the hardware (clockspeed etc).

All numbers are from the "result statistics" page on the WCG website. I just picked the first 15 (or mostly 30) valid HCC WUs, added them up, and divided through the number of WUs taken into account to generate an approximate average.

-----------------------------------------------------------

L5640: (Hexacore; Frequency: 3,4Ghz; Cache: 12MB L3; Threads: 12)

Runtime per WU on Win7 (15-WU average): 1,976h
Granted credit per WU on Win7 (15-WU average): 43,83

Credits per hour (per thread): 22,18
Credits per hour (CPU total): 266,17
-----------------------------------------------------------

Runtime per WU on Ubuntu 10.04 (30-WU average): 1,227h
Granted credit per WU on Ubuntu 10.04 (30-WU average): 22,82

Credits per hour (per thread): 18,6
Credits per hour (CPU total): 223,18

----------------------------------------------------------

Core i7 980X: (Hexacore; Frequency: 3,68Ghz; Cache: 12MB L3; Threads: 12)

Runtime per WU on Win7 (30-WU average): 1,876h
Claimed credit per WU on Win7 (30-WU average): 52,61
Granted credit per WU on Win7 (30-WU average): 42,82
Claimed vs. granted ratio on Win7 (30-WU average): 81,4%

Granted credits per hour (per thread): 22,83
Granted credits per hour (CPU total): 273,9
-----------------------------------------------------------

Runtime per WU on Ubuntu 10.04 (30-WU average): 1,072h
Claimed credit per WU on Ubuntu 10.04 (30-WU average): 30,71
Granted credit per WU on Ubuntu 10.04 (30-WU average): 22,55
Claimed vs. granted ratio on Ubuntu 10.04 (30-WU average): 73,4%

Credits per hour (per thread): 21,04
Credits per hour (CPU total): 252,43

So... what have we here.
From the looks of it, the Linux client is definitely more efficient. It completes the WUs a lot faster. However, the Windows client gets higher granted credit.
Now the real question is whether Linux and Windows client are getting the exact same work units.

Either way, I'm a little disappointed. I had hoped to present to everyone the "better" OS for crunching, yet it seems people have to choose between getting more science done (if the WUs are the same) or getting higher PPD.

--------------------------------------------------------------

To better illustrate my findings; I attached a graph. Higher is better except for runtime (which I multiplied by 20 to better show the difference in the chart)

[soya_crack]

I just can say that I tested the same thing with my X6 @3,6GHz and my results match with yours.

I really would like to know now, if the WUs are the same. If yes, the WCG/BOINC Credit systems would be total crap.

[jcool]

The BOINC inter-project credit system IS total crap.
Have a look at my sig... I got those 16 million on MW with a single HD5870 in a month or so. Huge point differences between projects have always been the case, and will always be the case I guess.

However, if the discrepancies are within WCG, they can (and should) do something about it.
I posted the question whether the WUs are the same over at WCG forums: https://secure.worldcommunitygrid.or...e,yes#lastpost

Let's see what they say.

[D_A]

Let's for a moment assume they are the same amount of work.

The credit is based on what the benchmark score is and how long the unit takes, with a "fudge factor" possibly but more of less that's it. If a Linux box is chewing through the equivalent amount of work in less time due to being more efficient, but has the same or close to the same benchmark scores, then it will logically claim less points per unit. If the benchmark scores are lower for Linux (which I suspect might be the case as the Windows compilers are more efficient for those processes) then it will also claim less points per hour. (That could be verified one way or the other if you have benchmarks for the same version client on both Win7 and Linux). I'd suggest that the credit per hour would be the metric to make the call on as it takes more factors into account and no matter how good the code is on any system, time is close enough to a constant pace. The difference in claimed v granted credit has me intrigued though. It's enough to swing things either way depending on what's causing the variation.

The credit systems on any project are, pretty much, crap anyway. At best they're only an educated guess of what the work is worth on an "average" system. At worst ...

[jcool]

Let's for a moment assume they are the same amount of work.

The credit is based on what the benchmark score is and how long the unit takes, with a "fudge factor" possibly but more of less that's it.
If a Linux box is chewing through the equivalent amount of work in less time due to being more efficient, but has the same or close to the same benchmark scores, then it will logically claim less points per unit. If the benchmark scores are lower for Linux (which I suspect might be the case as the Windows compilers are more efficient for those processes) then it will also claim less points per hour. (That could be verified one way or the other if you have benchmarks for the same version client on both Win7 and Linux). I'd suggest that the credit per hour would be the metric to make the call on as it takes more factors into account and no matter how good the code is on any system, time is close enough to a constant pace. The difference in claimed v granted credit has me intrigued though. It's enough to swing things either way depending on what's causing the variation.

The credit systems on any project are, pretty much, crap anyway. At best they're only an educated guess of what the work is worth on an "average" system. At worst ...

That makes sense.. benchmark score * runtime = score.
Interestingly enough, Windows doesn't score higher on the Benchmark, at least not overall. I just ran it on both machines, here's what I got:

L5640:

Ubuntu 2913/11816 FPU/Integer
Win 7 3123/10085 FPU Integer

980X:

Ubuntu 3184/12899 FPU/Integer
Win 7 3346/10776 FPU Integer

BOINC 6.10.45 on both Windows installs, BOINC 6.10.17 on both Linux installs

[onex]

Runtime per WU on Win7 (15-WU average): 1,976h
Runtime per WU on Ubuntu 10.04 (30-WU average): 1,227h

these are some great findings,
if the WU's are the same,

moving!

[jcool]

They seem to be the same, but I am still waiting for a definitive confirmation on that.

[onex]

They seem to be the same

how can you tell?
size?

E:
i think the code should be the same, only maybe a different file format,
after all the language should be universal and compilers -faster or slower- are just doing the translation job.

just wondering if they have different servers for both operating systems, and each WU, is going through an automated translation process,
they both should come from the same source,
could be that it's only changing the file format.

[jcool]

Well.. I can't tell for sure, as I keep saying.
But.. everyone on the WCG forums seems to say/think that the WUs are identical.. WU names certainly are identical. Ram usage is about the same as well (~30MB per thread) so it stands to reason that the work packages are, indeed, the same for any type of OS.

Edit: I am seeing a strange phenomenon on both Linux machines. The system monitor doesn't show 100% load on all cores.. there's always at least one core not fully loaded. Now, I have disabled any thermal management in bios so it can't be the CPU throttling. Any idea what else it could be?

[artemm]

That's really weird, jcool. Certainly does look like the CPU throttling itself. Swaps from thread to thread constantly with never any more than 11/12 load.

You sure your preferences in BOINC aren't being overriden to use <100% cores?

BTW

Core i7 980X: (Hexacore; Frequency: 3,68Ghz; Cache: 12MB L3; Threads: 12)

[Win7]Granted credits per hour (CPU total): 273,9
[Ubuntu]Credits per hour (CPU total): 252,43

252.43*(1+[1/12])= 273.4658333333333

Coincidence?

[jcool]

That's really weird, jcool. Certainly does look like the CPU throttling itself. Swaps from thread to thread constantly with never any more than 11/12 load.

You sure your preferences in BOINC aren't being overriden to use <100% cores?

BTW



252.43*(1+[1/12])= 273.4658333333333

Coincidence?

That pic was of the L5640. And yes I am sure, settings are right, Thermal throttling disabled in bios, temps are ok.

Here's a shot of the 980X, equally weird
But that system is on a nearly-full 4GB USB stick and is really slow to respond, so maybe it's the USB stick's fault (at least on the 980X rig).

[xVeinx]

Your better off switching to the server kernel in Ubuntu. You can go to synaptic in the admin menu, and search for kernel-server as I recall. The Desktop and Server kernel's are optimized differently, and the Server tends to be much better at maintaining full load.

[jcool]

Hey vein,

how about you join me in IRC and give me a noob-proof explanation on how to do that?

[don_xvi]

Does it have 12 active tasks? That'll help determine if it's a BOINC problem or a computer problem.
Are the advanced preferences all OK to run 100% and 12 cores?

[skinnee]

If you're going to run the server kernel for 'nix then you need to run Server 2008 R2 as well.

Presently, you're comparing two "Desktop" class OS installs.

[jcool]

I got it stabilised at 100% on the L5640 (no idea how.. didn't change anything).
The 980X still jumps around - I'll put the blame on the slow-as-crap USB stick though. Since we tried lots of stuff including a switch to the server kernel.
And it's definitely nothing trivial, I triple-checked BOINC settings and bios (besides, it runs fine on Win7 with the same Bios settings).

Oh well, the 980X is for a customer anyway, I was just testing it. The L5640 is mine, and stays on Linux for now.

[onex]

I got it stabilised at 100% on the L5640 (no idea how.. didn't change anything).

maybe a background process taking priority...

[jcool]

Maybe. Anyway, some news. Uplinger posted the following on the WCG forums:

HCC does not send out the same exact work unit to both windows and linux. It is as Randzo says, when the first work unit in the quorum is picked up, the OS is set for the wingman. This means linux compares with linux. Windows compares with windows....

A batch consists of about 1500 work units, these work units are split between all the OS's, so there is no discrimination based on batch level.

-Uplinger

Don't know what to make of that really.. So they are the same, and yet they aren't?

[onex]

as i said,
the work is being divided (by uplinger at the link) to all OS's from the same batch, they simply are not the same in the sense that each work is different,

the batch doesn't tell the difference between OS's,
a cruncher gets a WU so his partners are getting it only if they use the same OS.

they can't tell how many WU's would go to Linux users and how many would go to Windows,
the batch is simply a collective.
that's maybe the most reasonable solution.

as for Windows vs Linux optimization,
Linux is faster and it is quite known,
they can't optimize they're code (at least not as well) to make Windows client work faster as linux does, as it works different from the Linux client,
it's not about they're code,
it's about the OS.

[jspace]

To me it sounds like all of the WUs come from the same "pool" i.e. the same work gets done no matter the OS. Then that pool gets split up between operating systems and the results are compared from the same OS.
At least it makes sense to me.

To me, the fact that Linux is free is the biggest advantage. The cost of an OEM version of Windows can buy another power supply. While Windows can also be "free," that is either violation of copyright, which is bad, or there might be limits to the number of copies you can have activated. I've had pretty good uptime on 7, but every time I've used Linux, it seems to stay up forever. Unless I with it, ofc.
I'm not trying to take from what you guys are doing, I just think that in the long run it comes down to personal preferences.

[don_xvi]

I think what that means is that there's a big pool of WUs and they could go to either Windows or linux crunchers. When one is assigned to a computer, the OS family of that machine will be used for the quorum. So, they're the same in that they're all interchangeable and assigned to OS's at random, but unique in that you'll never see the same WU on both Windows and linux.

[onex]

shame we just can't confirm it with the developers .

[jcool]

Well... here's roughly a full day (12 CPU days) on the L5640 on HCC - Windows gets more credits, Linux does more work.
Line format is date - CPU time - granted credit - finished WUs (in case anyone is unfamiliar with it)

Win:
25.05.10 0:012:05:23:41 44,190 131

Linux:
27.05.10 0:012:12:12:21 38,221 251

And the 980X:

Win:
24.05.10 0:012:00:36:55 47,408 114

Linux (couldn't find a full 12 day entry yet):
28.05.10 0:011:01:43:40 37,226 242

[soya_crack]

What are the clockspeeds of these badass chips?

[jcool]

Pretty low, due to crap stock cooling and hot weather... see post 1 (3,68Ghz for the 980X, 3,4Ghz for the L5640).
Normal would be >4ghz on the 980X and 3,8 on the L5640