New Multi-Threaded Pi Program - Faster than SuperPi and PiFast

**Particle** · 03-19-2010, 10:24 PM

I appreciate your explanation. It's interesting to hear about the internals of a project like your own, being a programmer. I hadn't really paid much attention to what error I was getting once I realized it was a pattern. I ran it again just now though, and this is what it shows (attached).

I didn't realize you could set the memory when I started it. I'd actually launched it Monday morning just to serve as a stress test after making some cooling changes to the system. I decided to let it keep going. heh I've only got 8GB of memory in this box and need most of that for my daily stuff, as you can tell. Using an extra 2GB of memory for ycrunch caused a memory low condition while I was doing daily development and gaming. All your other guys doing 100B runs have rigs with much greater amounts of memory than I.

**poke349** · 03-19-2010, 10:30 PM

Originally Posted by Particle

I appreciate your explanation. It's interesting to hear about the internals of a project like your own, being a programmer. I hadn't really paid much attention to what error I was getting once I realized it was a pattern. I ran it again just now though, and this is what it shows (attached).

I didn't realize you could set the memory when I started it. I'd actually launched it Monday morning just to serve as a stress test after making some cooling changes to the system. I decided to let it keep going. heh I've only got 8GB of memory in this box and need most of that for my daily stuff, as you can tell. Using an extra 2GB of memory for ycrunch caused a memory low condition while I was doing daily development and gaming. All your other guys doing 100B runs have rigs with much greater amounts of memory than I.

What does the bottom of the validation file say?

Normally it is:

Code:

Last Digits:
3803750790 9491563108 2381689226 7224175329 0045253446  :  24,999,950
0786411592 4597806944 2455112852 2554677483 6191884322  :  25,000,000

Timer Sanity Check:      Passed
Frequency Sanity Check:  Passed
ECC Recovered Errors:    0

But you should be getting something else when it fails.

**Particle** · 03-19-2010, 10:31 PM

Timer sanity check failure.

Code:

Validation Version:  1.0

Processor(s):        Six-Core AMD Opteron(tm) Processor 2427
Logical Cores:       12
Physical Memory:     8,587,915,264 ( 8.00 GB )
CPU Frequency:       2,385,183,943

Program Version:     0.5.2 Build 9082 Alpha 3 (x64 SSE3 - Windows ~ Kasumi)
Constant:            Pi
Algorithm:           Chudnovsky Formula
Decimal Digits:      50,000,000
Hexdecimal Digits:   Disabled
Threading Mode:      16 threads
Computation Mode:    Ram Only
Swap Disks:          0
Working Memory:      332 MB

Start Time:          Sat Mar 20 01:19:53 2010
End Time:            Sat Mar 20 01:20:16 2010

Computation Time:    19.994 seconds
Total Time:          21.454 seconds

CPU Utilization:         825.39 %
Multi-core Efficiency:   68.78 %

Last Digits:
4127897300 0153683630 8346732220 0943329365 1632962502  :  49,999,950
5130045796 0464561703 2424263071 4554183801 7945652654  :  50,000,000

Timer Sanity Check:      Failed
Frequency Sanity Check:  Passed
ECC Recovered Errors:    0

----

Checksum:    53cc1527bc87df760d3dd7a2bb049be5848e39eb4325ca2c19572a8d6af91b1d

**poke349** · 03-19-2010, 10:36 PM

Originally Posted by Particle

Timer sanity check failure.

Code:

Validation Version:  1.0

Processor(s):        Six-Core AMD Opteron(tm) Processor 2427
Logical Cores:       12
Physical Memory:     8,587,915,264 ( 8.00 GB )
CPU Frequency:       2,385,183,943

Program Version:     0.5.2 Build 9082 Alpha 3 (x64 SSE3 - Windows ~ Kasumi)
Constant:            Pi
Algorithm:           Chudnovsky Formula
Decimal Digits:      50,000,000
Hexdecimal Digits:   Disabled
Threading Mode:      16 threads
Computation Mode:    Ram Only
Swap Disks:          0
Working Memory:      332 MB

Start Time:          Sat Mar 20 01:19:53 2010
End Time:            Sat Mar 20 01:20:16 2010

Computation Time:    19.994 seconds
Total Time:          21.454 seconds

CPU Utilization:         825.39 %
Multi-core Efficiency:   68.78 %

Last Digits:
4127897300 0153683630 8346732220 0943329365 1632962502  :  49,999,950
5130045796 0464561703 2424263071 4554183801 7945652654  :  50,000,000

Timer Sanity Check:      Failed
Frequency Sanity Check:  Passed
ECC Recovered Errors:    0

----

Checksum:    53cc1527bc87df760d3dd7a2bb049be5848e39eb4325ca2c19572a8d6af91b1d

That's interesting... I guess I overestimated the precision of the hardware timers on some of these motherboards...

Another thing to add to my to-do list for v0.5.3.

Anyways, I'm off for the night. Gotta catch a plane tomorrow morning.

**Particle** · 03-19-2010, 10:36 PM

Good night. Have a good trip!

**poke349** · 03-20-2010, 08:15 AM

Originally Posted by Particle

Good night. Have a good trip!

Thanks! Going home for spring break.

About to head to the airport. Guess I'll be spending the time on the plane migrating the program to the new thread API that I just knocked up.
(If I can manage to open up my 17-in. laptop on a seat tray...

)

Then I'll tweak the validation to make it less sensitive to clock variation...
(I might need your help for this, since I don't have a system with a very imprecise hardware timer like yours - assuming that's the problem.)

The last thing I have planned for v0.5.3 is a new conversion algorithm.
It's based on the same approach that was used to set the current Pi record and is 3 - 4x faster than one I'm currently using. So I guess the credit goes Fabrice Ballard for coming up with such an ingenious algorithm... (or whoever he got the idea from)

I doubt I'll finish this before I get back to school, since I'll probably be spending the whole break fixing my workstation and getting trashed with my high school friends...

With this new algorithm, the program should be 10 - 20% faster.
This gives me room to insert some new redundancy checks without making v0.5.3 any slower than v0.5.2.

**skycrane** · 03-20-2010, 02:16 PM

have a great time on spring break

remember, dont eat the worm,

**Particle** · 03-20-2010, 02:26 PM

Hey, some people do.

**poke349** · 03-22-2010, 09:38 AM

Originally Posted by Particle

Hey, some people do.

Spent all of yesterday debugging my workstation... No matter what I did, it always managed to fail Linpack (or some other stress test, but it fails Linpack the fastest).
I switched the PS... same results.
I switch the processors... same
Clock down to 2.4 GHz... same
I individually stress each core, and every combination of 2 and 4 cores... same
I test with only one processor. (and switch them around too)... same

Gonna try to get a mobo RMA today. I live 20min. from the Tyan RMA department.

Anyways... I finished migrating the program to the new thread library. It has the new protocol that checks if a thread has its task before it is terminated.
I haven't made any attempt to implement any sort of recovery measure yet... since that's more complicated.

It was a major change, but it looks stable for now. I need either my Core i7 rig (which is back in my dorm), or my workstation (which is down) to test it...

Lastly, here's some very large results that I got over the past few days:

W. Wong & R. Chang: 50 billion digits of Pi - 8 socket monster
Note that program hasn't been optimized for NUMA so the scaling is crap...

Code:

Constant :  Pi
Algorithm:  Chudnovsky Formula

Decimal Digits    :   50,000,000,000
Hexadecimal Digits:   41,524,101,187

Threads:    32
Mode   :    Basic Swap  ( Disks = 1 )

Start Time: Fri Mar 19 17:43:51 2010

Allocating and Reserving Memory...       100 GB
Constructing FFT lookup tables...
Setting Permissions for File Allocation...

Begin Computation:

Summing Series:  3,525,683,470 terms
Time:    52347.467 seconds  ( 14.541 hours )
InvSqrt...
Time:    2574.078 seconds  ( 0.715 hours )
Final Multiply...
Time:    3706.343 seconds  ( 1.030 hours )

Pi:  58627.890 seconds  ( 16.286 hours )

Writing Hexadecimal Digits:   41,524,101,188  digits written

Constructing Base Conversion Table:
Time:    2505.771 seconds  ( 0.696 hours )
Base Converting (Primary Cutting Parameters):
Time:    13961.596 seconds  ( 3.878 hours )

Writing Decimal Digits:   50,000,000,001  digits written

End Time:   Sat Mar 20 15:25:57 2010

Total Computation Time:                 75095.294 seconds  ( 20.860 hours )
Total Time (including writing digits):  78124.232 seconds  ( 21.701 hours )

CPU Utilization:        2010.91 %
Multi-core Efficiency:  62.8411 %

Last Digits:  Pi
3967497298 9229992389 7944203755 6895008004 9953150775  :  49,999,999,950
6599559400 3447556105 3766739199 8513398712 7510930042  :  50,000,000,000

Version:          0.5.2 Build 9082 Alpha 3 (x64 SSE3 - Windows ~ Kasumi)
Processor(s):     Quad-Core AMD Opteron(tm) Processor 8356
Logical Cores:    32
Physical Memory:  154,615,586,816 (  144 GB )
CPU Frequency:    2,300,188,015 Hz  (frequency may be inaccurate)

Result File: Validation - Pi - 50,000,000,000.txt


Press any key to continue . . .

Shigeru Kondo: 1 trillion digits of Sqrt(2) - New World Record
Below are the two computations that were used to compute and verify.
The Core i7 975 was actually overclocked to 4 GHz via multiplier. But the program can't detect non-stock multiplier frequencies. So it still reads at 3.33 GHz.

**poke349** · 03-27-2010, 11:27 PM

Finally got my workstation up and running again...

I drove the Tyan's RMA lab in Fremont, CA to drop off the mobo (and the CPUs, ram, and video card to make sure everything else was fine).
They confirmed a problem with the mobo and gave me a new one. So I went and picked it up on Friday. (along with all the other stuff I left with them)
Spent the rest of Friday putting it back together, reconfiguring the hardware and installing Win7.

Here's some pictures. (Click to enlarge)

The hardware changes were that I switched the video card with a GTS 250 and replaced the SSD with a 1.5 TB HD.
The RMA lab said that my old 9800 GTX+ was failing. (Though it still seems to work fine, I'm not gonna trust it anymore.)
The SSD was one of the old write-bricking ones. I couldn't stand it anymore so I turned it into a flash drive.

I also put cages on the Ultra Kaze fans.
They've spilled enough blood already... And I'm not kidding...

I've never actually run any v0.5.2 Advanced Swap computations on it because it went down shortly before I finished v0.5.2.

Now I can finally let it rip through and see what it can do...

Tomorrow... hopefully I'll have something - along with a sneak peak into v0.5.3.

**poke349** · 03-28-2010, 11:09 AM

And it's DONE!!!

(click to enlarge)

**Alpha** · 03-28-2010, 01:46 PM

Glad to hear you got your workstation up and running again! if i had more storage on mine, i would run some swap mode runs as well. all my drives are bound by a single gigabit connection to my server that has 6 1tb drives in it.

I haven't had the ultra Kazes draw blood yet....

**poke349** · 03-28-2010, 02:49 PM

Originally Posted by Alpha

Glad to hear you got your workstation up and running again! if i had more storage on mine, i would run some swap mode runs as well. all my drives are bound by a single gigabit connection to my server that has 6 1tb drives in it.

I haven't had the ultra Kazes draw blood yet....

Thanks!

Been itchin to run Advanced Swap Mode on my workstation since it went down in January.

Just that sheer power of ram... it's not the fastest machine, but it should still beat Dave's 12-core monster to 100b - provided he doesn't throw in a couple dozen VelociRaptors...

EDIT:
About the Ultra Kazes... Between me and two of my friends... we've had close to 10 deep cuts from Ultra Kazes (we have about 5 of them)... Each time... resulting in blood spewing out like a fountain or like in an Anime when someone gets beheaded... (sorry for being so graphic...)

yeah... painful... gross... whatever... totally worth the 2 dollars for the cages...

**Movieman** · 03-29-2010, 03:52 AM

Originally Posted by poke349

Thanks!

Been itchin to run Advanced Swap Mode on my workstation since it went down in January.

Just that sheer power of ram... it's not the fastest machine, but it should still beat Dave's 12-core monster to 100b - provided he doesn't throw in a couple dozen VelociRaptors...

EDIT:
About the Ultra Kazes... Between me and two of my friends... we've had close to 10 deep cuts from Ultra Kazes (we have about 5 of them)... Each time... resulting in blood spewing out like a fountain or like in an Anime when someone gets beheaded... (sorry for being so graphic...)

yeah... painful... gross... whatever... totally worth the 2 dollars for the cages...

In that machine right now is one 36 gig raptor as a boot drive and a WD Black 750 gig for storage..That's it and was that a challenge I heard?

**poke349** · 03-29-2010, 06:15 AM

Originally Posted by Movieman

In that machine right now is one 36 gig raptor as a boot drive and a WD Black 750 gig for storage..That's it and was that a challenge I heard?

THAT's RIGHT Dave!!! I'm challenging you to beat 31 hours on 100 billion digits.

If you accept, I'll send you the same build that I used, which is 0.5.3.9103.

So the question is: Can your 12 OC'ed and HT'ed cores make up for a lack of ram?

**Movieman** · 03-29-2010, 07:46 AM

Originally Posted by poke349

THAT's RIGHT Dave!!! I'm challenging you to beat 31 hours on 100 billion digits.

If you accept, I'll send you the same build that I used, which is 0.5.3.9103.

So the question is: Can your 12 OC'ed and HT'ed cores make up for a lack of ram?

so I need to go buy 12-4 gig sticks of DDR3-1333 ECC and maybe 10-2 tera drives correct...Ahhh, nope unless you want to send the check!

**poke349** · 03-29-2010, 08:37 AM

Originally Posted by Movieman

so I need to go buy 12-4 gig sticks of DDR3-1333 ECC and maybe 10-2 tera drives correct...Ahhh, nope unless you want to send the check!

Dave's afraid to lose... Dave's afraid to lose...

jk... *expects banhammer*

That would be overkill. I think 4 - 8 x 2TB drives will be enough. Extra ram probably isn't necessary...

**Particle** · 04-02-2010, 05:41 AM

It probably never came up since it's so old, but y-cruncher doesn't appear to run on Windows 2000 SP4. You might want to make a notice of that on the download page. Error in Kernel32.dll shortly after launch.

**poke349** · 04-02-2010, 06:31 AM

Originally Posted by Particle

It probably never came up since it's so old, but y-cruncher doesn't appear to run on Windows 2000 SP4. You might want to make a notice of that on the download page. Error in Kernel32.dll shortly after launch.

Good point

This seems to be the reason why it needs XP or later:

http://msdn.microsoft.com/en-us/libr...8VS.85%29.aspx

I think I almost accidentally used a Vista or later function once. But it was a good thing I took a good look at the function specs before I did it.

**poke349** · 04-02-2010, 12:24 PM

Dave... Try and beat this!!!

The validations don't work on these because they were run with build 9074 (which was before I fixed it).

Credit: Daniel Ghidali

Code:

Validation Version:  1.0

Processor(s):        Intel(R) Xeon(R) CPU X7560 @ 2.27GHz
Logical Cores:       32
Physical Memory:     137,424,424,960 ( 128 GB )
CPU Frequency:       2,261,038,496

Program Version:     0.5.2 Build 9074 Alpha 3 (x64 SSE4.1 - Windows ~ Ushio)
Constant:            Pi
Algorithm:           Chudnovsky Formula
Decimal Digits:      10,000,000,000
Hexdecimal Digits:   Disabled
Threading Mode:      32 threads
Computation Mode:    Ram Only
Swap Disks:          0
Working Memory:      46.2 GB

Start Time:          Fri Mar 05 23:14:54 2010

End Time:            Sat Mar 06 00:01:35 2010


Computation Time:    2,276.566 seconds
Total Time:          2,800.493 seconds

CPU Utilization:         2613.05 %
Multi-core Efficiency:   81.65 %

Last Digits:
9763261541 1423749758 2083180752 2573977719 9605119144  :  9,999,999,950
9403994581 8580686529 2375008092 3106244131 4758821220  :  10,000,000,000

Timer Sanity Check:      Passed
Frequency Sanity Check:  Passed
ECC Recovered Errors:    0

----

Checksum:    01c4398c710d6dbe00ce7f0778af8bec0ace5e4837666302196bfdaf5e576c10

Code:

Validation Version:  1.0

Processor(s):        Intel(R) Xeon(R) CPU X7560 @ 2.27GHz
Logical Cores:       32
Physical Memory:     137,424,424,960 ( 128 GB )
CPU Frequency:       2,261,038,352

Program Version:     0.5.2 Build 9074 Alpha 3 (x64 SSE4.1 - Windows ~ Ushio)
Constant:            Pi
Algorithm:           Chudnovsky Formula
Decimal Digits:      25,000,000,000
Hexdecimal Digits:   Disabled
Threading Mode:      32 threads
Computation Mode:    Ram Only
Swap Disks:          0
Working Memory:      116 GB

Start Time:          Sat Mar 06 09:06:41 2010

End Time:            Sat Mar 06 11:12:45 2010


Computation Time:    6,192.782 seconds
Total Time:          7,564.867 seconds

CPU Utilization:         2655.72 %
Multi-core Efficiency:   82.99 %

Last Digits:
2448547079 5329693979 7145627081 9204187454 9483487803  :  24,999,999,950
1309759846 5364560010 7388984278 8403481193 9913806533  :  25,000,000,000

Timer Sanity Check:      Passed
Frequency Sanity Check:  Passed
ECC Recovered Errors:    0

----

Checksum:    5dd78330683ccafd0bfdebf821112eb7b1308cf69ca057f713eb27cb1748e55a

**Movieman** · 04-02-2010, 12:40 PM

Now how the hell can I match that as I only have 12 gig.
Have this person do a one billion run and I'll roll over that 16 core/32 thread slowpoke like they are sitting still!

**poke349** · 04-03-2010, 10:26 AM

A couple of larger test runs for v0.5.3. Still a lot more tests to do...

10 billion on my old Pentium D @ 2.8 GHz

: (click to enlarge)

250 billion on my workstation: (click to enlarge)

As mentioned, v0.5.3 is about 10% - 15% faster than v0.5.2 depending on the computation size. (for Pi)
This time, the speedup is entirely algorithmic, so it is the same for both AMD and Intel, and across all processors.

This makes it the biggest speedup this program has had to date for all processors except on Core i7 between v0.4.2 - v0.4.3

- which I think is just enough to put it ahead of both PiFast 4.3 and QuickPi 4.5 for single-threaded runs (and using the same instruction sets for fairness).
It's still slower than TPI on anything with fewer than 4 cores... but at least it ties it on 4-core and beats it on 8-core.

Single-threaded performance was always (and still is) y-cruncher's achilles' heel. But at least it isn't as embarrassing as it used to be...

The other major change I made is in the stress-test feature:

It will now run in Below Normal priority. This gives the system a bit more responsiveness since y-cruncher had a tendency of starving everything else of CPU.
When one threads stops due to an error, the other thread will stop after it finishes (or fails) its current test. (So both threads will now stop after an error.)

**poke349** · 04-15-2010, 12:12 PM

I've attached some more results on quad-socket Beckton. (4 x 8 physical cores)
25m all the way up to 25b - All in RAM!!!

(Credit: Daniel Ghidali)

Dave's OC'd X5680's are indeed still faster for the sizes that he can run.
It's obvious that NUMA + low clock speed prevents the Becktons from scaling well on smaller sizes.

And also...

Version 0.5.3 is out!!!

Yeah, it took long enough...

There were a couple of bugs that needed several days to reproduce... So it took a hella long time to fix those...

2x faster Base Conversion -> 10% faster Pi
The stress test now runs in below normal priority.
The stress test will now stop both threads soon after either one detects an error.
New "Validation.txt" file that can be customized with your name/sn... (full support for unicode and international characters)
Automatic memory selection for Advanced Swap Mode. (it can still be manually overrided)
Added detection for the failed thread-creation bug that was discussed a few pages earlier. (Though I have yet to be able to properly test it...)
Decreased sensitivity of the cheat-detection. (too many false positives before)
New redundancy checks have been added to detect errors closer to when they occur.
Fixed: Junk output for computations larger than 41 billion digits on x86 and x86 SSE3.
Fixed: Possible stack-corruption for computations larger than 500 billion digits.
Fixed: Minor bugs in the interface.

Special thanks to Shigeru Kondo for helping me test this version.

Here's my first result from the public release of v0.5.3.

100 billion digits in 46 hours: (Down from 53 hours for v0.5.2.)
Core i7 920 @ 3.5 GHz - 12 GB DDR3 - 4 x 2 TB Hitachi
(click to enlarge)

**poke349** · 05-13-2010, 10:38 AM

Thought I give this thread a slight bump.

I've also done some new detailed multi-core scaling benchmarks on v0.5.3/v0.5.4. (I haven't done these since v0.4.3.)

http://www.numberworld.org/y-cruncher/#Benchmarks

v0.5.4 is the same speed as v0.5.3. So all benchmarks that I have posted of it are fully comparable with v0.5.3.
I don't plan on releasing it for a while, but if anyone is curious, it has the following changes:

Checkpointing - Allows Advanced Swap computations to be restarted in the middle of a computation. (This allows recovery from power outages, crashes, etc...)
More aggressive error-correction. For large computations, more than 90% of run-time is covered by at least one layer of ECC.

Checkpointing will increase the chance that a large computation will finish at all.
The error-correction will greatly increase the chance that a large computation will finish correctly.
(It seems that even on stable hardware, soft errors are common enough to be a problem for large computations...)
But as always, any error that is detected (and corrected) will trigger an ECC flag which will show up in the validation certificate.

I've also halted all optimizations for v0.5.x.
I'm in the process of rewriting a very large portion of the program and will be using this opportunity to experiment with some new algorithms and programming tricks that will scale with AVX and FMA. (So it will take at least several months.)