Bootable Benchmark Development Thread

[Renfield]

Bootable Utility for Balanced Benchmarking and Archiving (BUBBA)

For an introduction or detailed information on our project, see post #2.
This post will now be used to track product development, and to notify of testing releases.

Release time! Well, early testing release. This release may or may not work for users of the ASUS p5b series, due to the strange IDE controller. I know these are important boards (I own a p5b-deluxe wifi myself), so I apologize for the inconvienence.

You can find BUBBA at our website: www.OpenParasol.com (http://www.openparasol.com)

We appreciate any tests you can give us.
Thanks,
-Clark


Current development: (Pre-release dev. notes deleted for cleanliness.)
---------------------------------------------------
DEC 06, 2006 - Initial release verson.

Updates:
You may notice an incorrect integer result in the database. It's fairly obvious. My pentium D945 is showing up as 650 MIPS, due to a bug in the testing release used (which didn't go public). I apologize for the inaccuracy. This faulty result will be deleted shortly, and all future submissions to the database will be from stable releases only. Speaking of stable releases....
Next prerelease expected early January.
Key Changes:
- Fixed an intel bug showing large inaccuracy during third test.
- Advanced testing option selectable from menu. This will launch a seperate menu which allows the user to select only the tests he or she wishes to run. Tests not run will show in database as "Not Performed."
- Modern hardware compatibility.
- More network support, including PCMCIA cards,
- Faster, more stable database hosting.
- Backspace works.
- Working auto-restart, with prompt for CD removal.

[Renfield]

Hello. I'd like to tell a bit about who we are and what we intend to accomplish. Our team (myself: Clark, and Stephen: not registered here) are seeking to construct a cheatless, optimized, and accurate benchmarking utility, which runs from a bootable CD and uploads the results to a browsable web database. This project is being constructed as a senior-level independant research project in our Computer Science curriculum, University of Southern Mississippi.
I'm posting this here for two reasons.
One - To track the project and milestones from status to completion.
Two - To retrieve feedback from the enthusiast community.
Number two is by far the greater reason. We seek to create tamperproof accurate benchmark that runs optimally on all existing architectures. Obviously we don't have a large variety of test machines at our disposal. As such, we will seek community feedback concerning both our methods, and for testing.

I will now detail our project proposal.

Introduction:
Many benchmarking utilities are used to date, each with numerous approaches to measuring performance. Some of the more "professional" utilities available merely use computationally intensive games and programs originally designed for other purposes: some even at a high cost!
The majority of benchmarking applications currently available run in windows. This makes the results highly relative to the number of active processes running. There is also little standardization between operating systems. In some instances of benchmarks, ports to Linux or use of a small VM windows layer are discouraged, because of higher score yields. Furthermore, running in a fully user-controllable environment allows the user several opportunities to "cheat" the benchmark by using clock-slowing programs, or similar methods.
A greater problem is the lack of support for specific hardware optimizations. The true performance of one's hardware is rarely conveyed, as support for complex instruction sets, 64-bit calculation modes, multithreading, multiprocessor, and multi-core architectures are often absent. A common system stability and CPU benchmarking utility, Super Pi (typically used to calculate 32 million digits of pi) is a prime example of this issue. Because of a lack of support for multithreading, the user is often expected to run a number of instances of the program equal to the number of processors or cores the user's machine supports. This is both unintuitive, and generates inaccuracy, as there is latency between opening the multiple sessions.

Purpose:
We propose an optimized, fair, and cheat-free benchmarking utility for scoring the performance of one's CPU, and RAM; the utility also automatically uploads the results to a self-maintaining results and rankings server.
Our solution is launched from a bootable CD, containing a streamlined version of the Linux 2.6 kernel. A presented menu will allow the user to select his or her processor model to ensure that the most optimized kernel is loaded. Once boot is complete, the user is prompted for a name. After entry, benchmarks take place. When benchmarks are complete, the user's score is shown, and uploaded to the server, with user's name and basic machine specifications.
Once the server authenticates and verifies the score, the user is prompted to remove the CD, and press a key to reboot. This method prevents user tampering of scores.
Benchmarks: All tests will be coded to support any number of threads, so long as adequate memory for each thread exists, and run 64-bit mode on compatible CPUs.

Project Specifications:
We propose the following benchmarks:
--Under edit--

Score database:
The user will be presented with a front-end website, which will allow him or her to sort the rankings by each score type, or search for a user.

Community Involvement:
We will coordinate with the enthusiast community and primary users of benchmark applications while developing specifications and requirements for the benchmarking software, to ensure it meets the demands of its primary audience. The project will take on an iterative development cycle, based on community feedback, and feature requests. Ideally, we would work to comply with the needs of the intended user base to provide the most suitable hardware benchmarking application for analysis and competition.

We look forward to any feedback or commentary on existing features described above, as well as requested features.
Our timeline for this project is approximately 3 months.

Thanks,
Clark

[m0da]

I'd be glad to leave feedback on this project!

For your reserved benchmark, you might be able to add a hdd benchmark (I don't like these, but if you run out of ideas, you might as well).

Good luck!

[[cTx]Philosophy]

Count me in for it..
Sounds great to me..

[cirthix]

for the pi benchmark, DO NOT have it run to a specific number of digits. Have it run for a specified amount of TIME and have the score be the number of digits which were calculated. that way it also makes using a less stable, ut very fast rig a little harder.

[colfin22]

Great idea. I'm willin to help :D

[nn_step]

I'll do compatablity testing

[Renfield]

For your reserved benchmark, you might be able to add a hdd benchmark (I don't like these, but if you run out of ideas, you might as well).
Given our timeline for the project, I'd honestly not want to deal with another compatibility layer. As it stands, HDD drivers need not be loaded at all, since the scores are to be communicated directly to the server. If we had more time, I'd likely make this an optional (not affecting cumulative score) bench.

for the pi benchmark, DO NOT have it run to a specific number of digits. Have it run for a specified amount of TIME and have the score be the number of digits which were calculated. that way it also makes using a less stable, but very fast rig a little harder.

Hm. I see what you're saying.
I like this idea. Though, a score well into the millions would be pretty hard to read. I'd have to convert the score into a more concise and comparable form. Probably by use of a ratio, E.g. (Your Score)/(Some standardized score).
It'd likely mean rethinking the rest of the benchmarks to use the same method. But I like the concept of standardizing the time. Anyone benching would know exactly how long they'd have to keep their systems stable. I'll give it some thought.

Thanks, this is the sort of input we're hoping to recieve.

[ugp]

Sounds awesome.

[dave_graham]

i'll keep watching this thread but...for me, my "benchmarking" needs to be done using actual software that my systems are being built for (maya, lightwave, etc) a synthetic test can be accomplished by anything, really, so...what differentiates yours from 3dmark06? sandra 2007? sciencemark 2.x? etc. etc.

dave

[Renfield]

i'll keep watching this thread but...for me, my "benchmarking" needs to be done using actual software that my systems are being built for (maya, lightwave, etc) a synthetic test can be accomplished by anything, really, so...what differentiates yours from 3dmark06? sandra 2007? sciencemark 2.x? etc. etc.

dave
As for the first part. Yes, if you're building your system for a specific purpose, it makes sense to bench it in the environment you plan to run it in, with a score representative of your specific goal.
However, we can't look at everyone's specific needs, so we look to create a much more general benchmark.

As for the second part: Fair question.
We're certainly not looking to duplicate 3dmark06. We won't be rendering any fancy graphics on screen, or counting FPS, etc. Expect background calculations, and text telling you what's happening.
There's a few things about 3dmark that make it highly subjective. That sort of program is programmed in an API (directX) built for game designers and the such. It's a big set of upper level functions that execute lower level code at an efficency highly dependant on your drivers, and support code written by the engine developer. You'll find that most of 3dmark's synthetic benchmarks still fall into this description.
CPU Test 1
The test uses a high level of path finding complexity, tight AI synchronization intervals over 40 frames, locked to fixed frame rate of 2FPS with a Shader Profile of 2_0 and a resolution of 640x480.
...
Feature test - Vertex Shader (Simple)
This test does simple transformation and single light lighting on four high polygon sea monster models. Each sea monster has over one million vertices to transform and illuminate, so the total workload is quite substantial.

This is what I'll call "practical hardware ability." It tells you how games and applications programmed in a similar manner, using the same API will react.
What we're after is to find what you might call "potential hardware ability." This is discovered by writing lower level code, operating at a level that all drivers should function, without several layers of dependant code, which fully utilizes the GPU's processing ability. We're not initializing a full spectacular-looking graphics engine; instead we're constructing tests from the most basic GPU functions. This provides a result which is representative of the GPU's performance unbiased by software mode. It tells you the power of your GPU not only for games and advanced shaders, but for crunching numbers with application code such as GPUSort, or a GPU-utilizing distributed computing application.

Now, Sisoft Sandra, Superpi, and ScienceMark are closer to what we're doing. Once you get to the level where you're benchmarking by executing very basic functions, you're where we want to operate. We're not saying we're the first to offer large number of threads supported, or multi processors, or 64-bit calcuations: none of this in new to the table. However, what we are concerned with is a lack of standardization. Existing systems do an excellent job of generating accurate results within their operating environment. But that's not to say that Computer A's score won't be effectively higher than an equal-hardware computer B's score, because A's current environment has been stripped of services and other running processes. Just take a look at the any number of threads for optimizing your operating system to get better benchmarking scores.
That's the concept behind a bootable benchmarking system: Standardize loaded drivers, eliminate unneccessary services and processes, ensure that all systems are on an equal playing ground. And then, deliver results not dependant on software, OS compatibility, extra layers of code, etc. But rather representative of the hardware's full capability.
Sorry for the length of the reply. Hope it clarified our goals. If you still have any questions, don't hesitate.
Thanks,
Clark

[m0da]

Alright, it seems either I don't know much about drivers yet (VERY possible compared to your knowledge, I'm sure), or there's just something you've forgotten Renfield!
How will VGA cards operate at their full potential without drivers? Are you going to edit the current drivers somehow to be able to run with the bootable benchmark?
please explain thanks!

[Renfield]

Alright, it seems either I don't know much about drivers yet (VERY possible compared to your knowledge, I'm sure), or there's just something you've forgotten Renfield!
How will VGA cards operate at their full potential without drivers? Are you going to edit the current drivers somehow to be able to run with the bootable benchmark?
please explain thanks!

Hm, I suppose I was unclear somewhere, though I'm not sure exactly where. If you could, please quote me on where it became misleading that the benchmark will be running without VGA drivers, so that I can go clarify it.

VGA drivers, of course, will have to be loaded. The drivers are the actual interface code to the hardware. I mentioned earlier that HDD / raid drivers don't need to be loaded. This is because there exists no need to communicate with the harddrive. Since we're using the videocard, however, VGA drivers are a must.

Edit: forgot about the second part of your question. The benchmark CD boots a highly-optimized 2.6 linux kernel. We need only to detect and load the appropriate linux video driver. I understand that there may be support issues in some advanced functions of the linux drivers (especially ATI's) but this will be of little concern, as we're constructing tests from the most basic functions, with which those advanced ones are made. I elaborated more on the reasoning behind using low-level functionality, in response to dave_graham.

Thanks,
Clark

[m0da]

That linux part definitely clears it up. I understand now, thanks!

[STEvil]

what about chipset and modem/NIC drivers?

By the time you get done adding every driver to the OS bake, it would be better to just have each user make their own cd with the custom drivers they need (which will drive away a fair percentage of users).

[Renfield]

what about chipset and modem/NIC drivers?

By the time you get done adding every driver to the OS bake, it would be better to just have each user make their own cd with the custom drivers they need (which will drive away a fair percentage of users).

Like most linux distros, we'll dynamically load the NIC driver module.
My co-developer, Steve, elaborates:
We will include the majority of drivers for networking devices present in the kernel; however, [the benchmark] will only load the pre-compiled module after the test. The user will be given an option to automatically detect the network card, or to enter his or her own manual specifications. Most linux distributions handle this with small scripts that are freely available to implement in any project. Your input is greatly appreciated.

edit: I'll also note that NIC drivers are very, very small. At work I frequently use a multi-card template containing 50+ modem and ethernet drivers, with Symantec's Ghost. It runs off a single floppy.

Thanks,
Clark

[Console]

This sounds Cool , Defently will try it once you have some betas out

[Renfield]

Coding is moving along on the various benchmarks. I've decided to go with the suggestion of the explicitly timed benches. I've completed a rough initial coding of the pi and memory benchmarks with this method.
That said, how long would everyone be comfortable benching for? Keep in mind it should be a time that's long enough to throw off unstable machines, but not so long as to discourage use of the tool.
Please share your opinions.
Thanks,
Clark

[m0da]

Well... I usually would think of benching and stability of two totally different things.

I'd bench for 30min to an hour tops for a single run. An hour is actually almost pushing it.

For stability? Well... I'd hope 12 hours would be long enough, though if it should be higher like 18, I'd be fine with that.

Benching and stability together? Eh... I don't really know. 3 hours?
Hopefully others will have a better opinion than me lol

[Fr3ak]

When I think about spi, something like 5-10mins for a single bench would be what it should be.
Running it for 5 mins makes sure its not that unstable, depending on what is being computed.

I like the idea of te bootable benchmark. I hope its working like you planned.
Including a video card benchmark might be difficult to realise, because of the vid card drivers.

Excuse me, I didnt read everything now, but will do that tomorrow.
How about adding some sort of possibility to safe the result to a usb stick or hdd to upload it to your website, encrypted non editable files of course. There might be some configs, where the network drivers dont work, as there are some people who have pre released hardware.

[Renfield]

Thanks for the time length suggestions. We can write it how you want it. So, we're leaving this open to you.

In development news. We've gotten our first basic testing CD working. It's missing a lot of specialized support functions, but it's still an important first victory. I thought I would share these results, for our newly written pi and memory tests.
Our tests have been written with precision in mind. Results should not deviate much. For this post, I will not show the actual output, as it isn't yet formatted and looks like jargon. Instead, I present...

Benchmark Score Precision:
These statistics are calculated from a series of 10 runs on the same hardware.
(%Error = Standard Deviation / mean avg * 100% , lower is better)

Windows Environment
Pi Test: 0.29% Error
Sequential Memory Test: 0.93% Error
Random Access Memory Test: 0.32% Error

Our controlled Bootable Environment
Pi Test: 0.043% Error
Sequential Memory Test: 0.027% Error
Random Access Memory Test: 0.162% Error

How's that for proof of concept? :D
We look forward to bringing you a highly accurate, tamper-proof benchmark.
Thanks,
Clark

[Renfield]

Before I get to the main topic:
We will soon need some early-alpha testers.
We ask for these requirements:
- Own a digital camera.
- Own software capable of burning a .iso file to CD.
- Ability to type :p
Currently, we need to test on
- Multiple processor motherboards or
- Multiple core processors and
- AMD processors.
Just send me a PM at your leisure.

Now then...
Some heavy changes this week, as outlined in the first post. I'll be editing the project outline shortly to reflect these. The purpose of this post, however, is to offer some reasoning for these major changes, since they may not all be well-recieved, and to detail what you can expect without clogging up the change log with a lot of text.
Memory tests removed due to being CPU dependant.
Okay. I tried a few different schematics for taking the strain off of the CPU during memory testing, but the results all showed an unacceptible use of computational power required during the timing or scoring of the memory tests. Since we want this program to be compatible with both the archaic and the super, that level of dependancy couldn't be justified. Obviously, the speed of your memory will still affect the rest of these tests anyway.
Added in Practical FLOPS and Practical Millions Integer Operations tests.
Now, if I could highlight one word, it would be "practical." I'll explain. Most GFLOPS and MIPS tests you see from vendors are a theoretical yield. That is, you perform all calculations within the processor's fastest cache, never reaching main memory, getting I/O, paging, etc. If you have more than one core or processor, the vendor would multiply it out to represent scale, ignoring the impact of communications medium, bus links, message passing protocols, mutex locks, and many other technical issues which make the "max theoretical" nearly impossible to attain for useful computing, and meaningless as an actual determination of processing power. To illustrate this point, consider that the first P4 chips were touted with a theoretical power of 3.0 GFLOPS. Once it starts using main memory however, it's ability becomes roughly 0.15 GFLOPS. So that's the issue. To amend this, we've incorporated a fully threaded, MPI supported bench with accurate results for real-world FLOP and integer operation determation. You'll know how well your computer handles these tasks for real applications, and how it compares to others, as such.

Recoded Pi test back to variable time. This allows for a large increase in accuracy and comparibility.
When Cirthix made the suggestion to move to explicitly timed tests, it was something I really enjoyed. Knowing exactly how long a benchmark test would take would be appealing to the user, and it would promote standardization.
However, there's only one algorithm for pi that would allow for variable digits to be calcualted in a fixed time: the BBP algoirthm. It just so happened that we were already using that algorithm to support multiple cores, so transitioning to an explicit timer was an easy step.
But, managing a clock comparison ate a bit of processing power that could have been used better. That is, the main execution thread had to continuously check the current time to see if the time was up. By moving back to variable time, it allowed the main execution to also crunch digits instead of performing timekeeping. The timekeeping also posed the threat of overshoot. That is, it could miss the stopwatch by a few milliseconds, which kind of makes having a microsecond resolution timer a moot point. Also, the results, while very consistent, did little to accurately determine the processing power of a machine to a very precise measurement. This is because the BBP algorithm isn't linear. Larger digits, that is, take longer to calculate. So, the gap from 100 to 150 digits is much smaller than from 3000 to 3050 digits. Since the output was simply the number of digits calculated, it may be confusing that processors with a clear advantage over one another were not representing that lead in the numbers. So, the code was rewritten with a variable timer, and a fixed number of digits to calculate. That is, an even workload for all machines, which is more comparible.

Thought I'd clarify, given that removing features isn't usually a good thing.
Thanks,
-Clark

[nn_step]

why not something simple like a knoppix OS, with a set benchmarking suite and the use of a Thumbdrive to store scores

[Renfield]

why not something simple like a knoppix OS, with a set benchmarking suite and the use of a Thumbdrive to store scores
Basically, that's what we're doing. Only the linux boot is heavily stripped down. We don't need to load a GUI, or a number of processes and devices. By getting rid of the daemon clutter, we'll be able to avoid a lot of context-switching on the processor, giving more precise results.
The thumbdrive option is in consideration, however, the primary method for submitting results will be automated. Straight to our server.
The final release will be very simple. Insert a CD, boot, enter your name and team name, wait a few minutes, eject the CD and reboot. Then you can scope your scores on the webserver.
I hope the digital camera request didn't allude that it would be the only method of submitting results. We're still just approaching a first testing phase, so for the basic alpha test, not everything is incorporated.

[Killnine]

Sounds awesome. Look forward to watching your development.

However, I fear that compatibility in network devices and such will be an issue. Linux was always pissy (though its made LEAPS recently) about hardware, and you would really have to be on top of support.

Best of luck

[dave_graham]

i can provide some 2p/4 core testing along with 1p/4 core, etc. etc.

lemme know.

dave

[Renfield]

First, I'll apologize to those who asked to participate in an early beta. We found all the machines we needed to test on, and we're wrapping that up now.
We should have a release within the next few weeks, after cleaning up some server interaction and a couple minor compatibility issues.

In the meantime, here's an example of what you'll see upon booting the CD:


+------------------------------------------------------------+
| |
| ###### ## ## ###### ###### ##### |
| ####### ## ## ####### ####### ####### |
| ## ## ## ## ## ## ## ## ## ## |
| ## ## ## ## ## ## ## ## ## ## |
| ####### ## ## ####### ####### ######### |
| ######## ## ## ######## ######## ######### |
| ## ## ### ### ## ## ## ## ## ## |
| ######### ####### ######### ######### ## ## |
| ######## * ##### * ######## * ######## * ## ## * |
| |
+------------------------------------------------------------+
|+----------------------------------------------------------+|
|| Bootable Utility for Balanced Benchmarking and Archiving ||
|| PARASOL Software: Stephen Adamec, Clark Duplichien ||
|| ||
|| Press any key to Begin! ||
|+----------------------------------------------------------+|
+------------------------------------------------------------+
detected: 2x AMD Athlon(tm) MP 2400+ @ 2000.258MHz

Thanks for using BUBBA.
Please answer a few short questions.
Please enter your name as it will appear in the database: Renfield
Please enter the name of your team or organization: Parasol
Thanks, starting immediately!

Calculating Practical MFLOPS.
...............
Calculating Practical MIPS.
...............
Calculating 2,000 hex digits of pi, over 15 iterations
...............

BENCHMARK RESULTS FOR Renfield @ Parasol:
PI TEST
Average 237.65 Hex digits / sec completed, with + or - 0.06% error.
MILLIONS PRACTICAL FLOATING POINT OPS PER SECOND
Average 250.55 Practical MFLOPS completed, with + or - 0.62% error.
MILLIONS PRACTICAL INTEGER CALCULATIONS PER SECOND
Average 269.91 Practical MIPS completed, with + or - 0.08% error.

Upload these results (Y/N)? Y
Upload complete. Thanks for using BUBBA!
You may eject the CD and restart your system.


Also, to show how it scales on multithreading, here's what you get if I force to one thread on the same machine as abouve.

PI TEST
Average 119.57 hex digits / sec completed, with + or - 0.043% error.
MILLIONS PRACTICAL FLOATING POINT OPS PER SECOND
Average 128.91 Practical MFLOPS completed, with + or - 0.28% error.
MILLIONS PRACTICAL INTEGER CALCULATIONS PER SECOND
Average 135.01 Practical MIPS completed, with + or - 0.031% error.

You can tell right off, that it's very parallel. When allowed to autodetect to 2 cores, the results are just about twice as fast, as one would expect. I'm looking forward to seeing some mean results from the heavy multi-cores out there.

Edit: I'll note that the above test was run on a public linux application server. The actual % error from the boot CD should be significantly lower.

[Killnine]

Glad to see more progress on this. nice job, cant wait for the open testing. =D

[[XC] Teroedni]

Wow
seems sweet:D

[m0da]

glad to see updates!

[too_cold]

This looks very cool. I like the "putting everyone on a level playing field" aspect. It's a pain trying to streamline your system for benching. As for amount of time for the test to take, faster is better to me. 5 minutes or so as someone suggested seems pretty good.

[Renfield]

Release time! Well, early testing release. This release may or may not work for users of the ASUS p5b series, due to the strange IDE controller. I know these are important boards (I own a p5b-deluxe wifi myself), so I apologize for the inconvienence. At the moment, we're forced to release with limited compatibility so that we can get some results before our project is graded.
Well, you can find a gzipped iso file here: http://www.openparasol.com/bubba/
If you can't extract that, I reccomend 7-zip: http://www.7-zip.org/
If you need something to burn the CD, I reccomend burnatonce: http://www.burnatonce.com/

Just grab the .iso, burn it to a CD, pop in the burned CD and restart. You'll need to have a hard-line ethernet cable if you want to upload results. We apologize, but this release does not contain support for dialup or wireless access. Also, at the time of this posting, the online database isn't public viewable yet, but a frontend website should be up within 24 hours.

We appreciate any tests you can give us.
Thanks,
-Clark

[nn_step]

lean ISO
Does it have an MD5?

[Renfield]

lean ISO
Does it have an MD5?

Steve focused on keeping it very slim.
Unfortunately, no MD5 hash, sorry.

[nn_step]

Steve focused on keeping it very slim.
Unfortunately, no MD5 hash, sorry.
so you are telling me he can't take 10 seconds to use something like this
http://www.brandonstaggs.com/filecheckmd5.html
So that we can tell instantly if our downloads are perfect?

[Renfield]

so you are telling me he can't take 10 seconds to use something like this
http://www.brandonstaggs.com/filecheckmd5.html
So that we can tell instantly if our downloads are perfect?

I apologize. He was driving home at the moment. I'll have a link to a faster hosted iso, with an MD5 shortly.
edit: Done. You can find the MD5 along with the download at http://www.openparasol.com/bubba/

[Renfield]

Results are now viewable on the web by clicking this link (http://deadtrain.hopto.org/cgi-bin/retrieve.pl). We should have it looking a bit cleaner later.

I'm currently uploading a new version (and MD5) to http://openparasol.com/bubba which should now support most gigabit ethernet PCI cards.

[ahmad]

How soon do you need those results?

[nn_step]

I apologize. He was driving home at the moment. I'll have a link to a faster hosted iso, with an MD5 shortly.
edit: Done. You can find the MD5 along with the download at http://www.openparasol.com/bubba/
thank you very much :toast:
wonderful work

[Killnine]

Very nice. Its quick and to the point. Awesome job, I look forward to seeing more improvements on this product.

Two suggestions though:

1) Is there a way (i know, its linux) that backspace can simply mean "delete" in console. Its sort of confusing if you mess up a letter in your name or information.

2) My copy doesn't restart my computer when finished. It just says "you should restart now". Maybe having an option menu like "run test again" or "shutdown" would be nice.

3) Finally, maybe a menu to run the whole battery of tests, or the option to run tests individually would be nice, though I am not sure if it defeats the reason for creating this suite.

Thanks again, guys, nice work.

[Renfield]

Very nice. Its quick and to the point. Awesome job, I look forward to seeing more improvements on this product.

Two suggestions though:

1) Is there a way (i know, its linux) that backspace can simply mean "delete" in console. Its sort of confusing if you mess up a letter in your name or information.

2) My copy doesn't restart my computer when finished. It just says "you should restart now". Maybe having an option menu like "run test again" or "shutdown" would be nice.

3) Finally, maybe a menu to run the whole battery of tests, or the option to run tests individually would be nice, though I am not sure if it defeats the reason for creating this suite.

Thanks again, guys, nice work.

Actually, all of these are already on our to-do list. Now that our early alpha is out (and the course is over), we just took a bit of a breather before getting back to work.
Steve is already working on the backspace issue: I'm told he's already gotten the restart fixed (though that version isn't uploaded yet). He'll also be migrating the database to a more reliable server.
I'll get a menu structure on there, to let you select which tests you want to run (which was needed anyways because we'll be adding more non-critical tests down the road). A functional website is also overdue. Currently, if using certain intel chips, you may notice a precision error in the third test. I'll also be hammering that out.

[Killnine]

Any news on the benchmark? I would hate for this program to just disappear into thin air.

[Renfield]

Any news on the benchmark? I would hate for this program to just disappear into thin air.

Sorry about that. Yes, we've had all the peices of a new release ready to go since the 5th, but haven't had time to wrap it up and kick it out (as we've been scrambling to get a new project together for the coming semester). I'm meeting with Steve today concerning the coming semester's project, and I'll see if we can finalize this next BUBBA release. I'll put an edit here later with an ETA.

[Killnine]

Wow....still nothing? Is this project totally dead or what?

I would like to take a look at the source code, if it's available. Im no pro coder, but this seemed like a very worthwhile project, man.

[KTE]

Apart from the "regular" or "daily" online community, majorities are too busy with external affairs leaving no time for online, even if there be a set of new developments to boast about. ;)

I suspect the same happened with the two fella's on this project. From thier home page: http://openparasol.com/bubba.html

Status

6/1/2007 - BUBBA isn't dead. It just smells funny. No, seriously: Steve's been working on a better filesystem structure and is excited about putting a few new kernels in store. He also is working hard on his very own 64-bit port and recompiling tools.

5/20/2007 - The bootable software was completely overhauled. A suitable compact distribution, Damn Small Linux Not! (http://www.damnsmalllinux.org/dsl-n/), was chosen as the basis for the new benchmarking environment. Modifications have been made but are certainly not complete. The initial filesystem and linuxrc scripts are due to be redone, and until UnionFS makes the stable Linux tree, it's time for something else. The boot CD uses kernel 2.6.10 as of now; the next big project is to implement 2.6.21 and related modules.
Downloads Section

A test version of BUBBA 0.01 is now available. It is approximately 70 megabytes.

To run: After bootup, type "/bubba". Submittable results to a database aren't available yet.

Grab bubba0_01.iso (http://openparasol.com/bubba0_01.iso)

I was following and awaiting testing with this VERY eagerly since October. :(

[Killnine]

Yeah man, I was totally hoping this would continue development. It could be so awesome, just something you pop in and run. Dont have to worry about installing a fresh copy of windows, or anything.

I wish I could help these guys out but I DEFINITELY dont do the low-level programming required for such a project. Im in my own little .NET realm (hooray, C#).

[Eightballrj]

Renfield... I almost spit out my Coke when I read the first post and saw that you were from Southern. I am at MSU studying Mech Eng and working as a computer tech on campus. If you need an Alpha/Beta tester let me know.

Shoot me an email or a message on AIM. Good luck!!

[Renfield]

Holy thread resurrection.
Alright, well... first, I'll apologize for not updating this thread.
Our work also stopped for a few months due to work on a different project for school, codenamed WILMA (completed in May: eventually I'll get around to a thread for it).
Graduating, moving, new jobs, etc also don't help much on the free-time horizon.
To keep things short (since I need to sleep before work) before the next release, the code base needs a big overhaul to be more memory intensive (currently stresses CPU with small memory footprint), and include memory-specific testing, and the distro needs to be reworked for efficiency and hardware compatibility (Intel 965 / JMicron is still a no-go at last release).
We're still interested in the project, but we're both short on free time recently.
Thanks for checking up, and thanks for the email (I'm guessing it was one of ya'll). I'll see if I can't pound out some code this weekend.

[Eightballrj]

Are you still in MS Renfield?

[Killnine]

I am interested in this project still, PM sent.

[dinos22]

problem with this project i see if that fact that there is sooo much new hardware pouring in on this Forum and linux really sucks when it comes to supporting latest hardware

by the time these guys work out how to run different things we will be moving onto some new platform and the wait begins again.....how many months though heh

[laragirl83]

R.I.P. this topic... :(
Tough I was hoping...

[TheKarmakazi]

an awesome idea no doubt...

they'd need a team of people i think to work this out efficiently and keep it up to date...

thats whats hard about indie programming, most coders have a real job and family/friends etc that eat up their time. and then when u get a free minute, looking at pages of code isnt the most appealing thing...

[scottc19]

Excellent Idea... implementation seems difficult... but if you can pull it off props to you!:)

[MuffinFlavored]

Sounds awesome.
You could include video drivers for both ATI and NVIDIA on disk, and have only the appropriate loaded.

Then, you could create a simple OpenGL benchmark.

In the benchmark should be:
graphics card
RAM
hard drive
CPU

All with one standard for scoring.
And, with a minimal footprint left by the kernel being 100% loaded.

[Renfield]

Killnine woke me up. Here's hoping I don't get in trouble for thread necromancy.
Since there's still interest in this somewhat-buried project I feel that at least an update is warranted.
So.
I've gotten a request for the source code for continuing the project. Unfortunately the code has been thoroughly ripped apart since we last tooled around with it. It's not really packagable into one piece at the moment: the tests are a bit strewn about. With my coding partner and I both full-time employed on different parts of the country, and an onset of general apathy, it's been hard to get much done. What has been done? Well...
Steve did some amazing work optimizing the distro our application runs on. I won't embarass myself pretending to understand the kernel voodoo he does, but compatibility and efficiency took big leaps.
I replaced all the original benches with hand-multithreaded adaptations of the old whetstone and dhrystone tests, as well as an adaptation of an open source memory benchmark. So, as it stands, there's two CPU tests, a memory test, and the combined test (Pi test). The pi test needs to be re-written or replaced in order for the score to scale linearly, due to the nature of the calcuations. I have an idea about how to do this.
The dhry and whet stone tests don't really line up with the numbers from the conventional tests (which used a lot of crazy legacy C... One was ported from Algol), but that's okay in my book. Really, all that's needed is a series of good tests that score linearly higher on increasing speed machine. Why?
The numbers from the tests are going to be scaled and normalized such that they have equivalent ranks. The resulting sum of these normalized scores is to be the final score. A kind of one-number-fits-all mark.
I hear ravenous growling. Hear me out.
We set out on this project intending to develop a precise scoring application for competitive benchmarking. After researching numerous benchmarks, come to find out that hey: trying to find a way to get numbers to represent the power of a PC, on average, in a variety of applications, is about as wise a pursuit as trying to pry honey from a beehive. Naked. Therefore, I'm now looking at core performance: simple repeated calculations to stress CPU and memory. The small side of the pursuit is a highly precise repeatable number which represents the speed of a CPU.
And you want to talk about repeatable? (warning, bragging follows) Noone touches us in this aspect. I'm defining result error as the Standard Error Percentage of scores between runs (with 10 samples):
Our application in its bootable environment has an error of 0.063% (and that doesn't reflect the optimization work that Steve's done),
Our application ported to Windows XP professional running a firewall, an (inactive) anti-virus daemon, an open document file, an Internet messenger application, a web browser, and a printer driver daemon, while recording user key strokes and mouse input had a result error of 1.5%
For reference, SiSoft Sandra claims a 5-10% variation in results on their official website.

So, in review,
*Work hasn't really happened with the application recently.
*Code isn't in any shape to hand out (really sorry).
*Code is near completion (If I ever get to work on it again soon).
*We pwn the other benches in precision.

[Calmatory]

Hmm, few distros come with MemTest86+, and GRUB can boot it as a Linux kernel. Any way to replace the MemTest with own program which would run upon boot?

We seriously need an precise linux benchmark. We got SuperPI, yes, but really, wouldn't we want raw CPU bench, raw RAM bench and raw OpenGL bench? ...with multithreading of course. Well, drivers can be an issue, but then ATI/Nvidia could(yeah, never gonna happen) actually start putting some major effort in thedrives if Linux benching was considered as one "section"/aspect of benching(neitther is this going to happen), just like SuperPI/3Dmark etc.

Though, Linux isn't suitable for overclockers, not at all. :(

[MuffinFlavored]

Here, make it simple.

Grab the latest Linux kernel.

Disable support for everything except all networking devices, USB keyboards and mice, and RAID. (Goal of this would to be load as little RAM as possible)

Create (write/program) your benchmarks, have an entire fleet of them, do whatever.

Put the kernel on a CD, along with your benchmarks.
Upon a successful kernel boot up, ask for the username or whatever, and then run the benchmarks, and then display the results, and then submit them to your database.

This could go VERY well if you optimize everything and keep one goal in mind:
benchmarks over everything.
The kernel should just load the detected networking device so you can submit results to the database, USB keyboard (not even mouse, keep it console), and that should be it.

Now, for graphics.
Take DSL, strip it of everything except X Window System, and write some OpenGL programs.

Verdict:
Don't include printer support in an overclockers benchmark.
This can go far.

[Renfield]

Here, make it simple.
--Some stuff--


The guy on the distro side, Steve, has a pretty impressive solution. It loads pretty much nothing. Services were hand selected. Anything that needs to be loaded in order to function (e.g. HDD drivers, network drivers) are selected and configured runtime by scripts. This ensures it's as slim as possible. He's done some really impressive work. The guy knows linux kernels in a way I'm not even going to pretend to understand.
You mentioned DSL; I believe he's using DSL-N. http://damnsmalllinux.org/dsl-n/

I actually did get a bit of motivation. So, I'm giving the tests a good re-write. Steve is really busy with work and Parasol Video, so I'll be testing on the last stable ISO he gave me.

Will post if anything interesting happens.

Edit: Pretty much everything you said is exactly how the test versions have worked.

[MuffinFlavored]

Edit: Pretty much everything you said is exactly how the test versions have worked.

Verdict: You and Steve are my heroes, we think alike, this is going to rock, I am willing to help test or code anything. PM me if interested. I hope this succeeds.

:up:

[TiMiN8R]

Verdict: You and Steve are my heroes
:up:

Agreed, this looks like a very worthwhile project, I've been looking for something like this for years. Not being a coder I really need people like you!
Reading page 2 of this thread I was close to crying when it seemed to be dead or dying and then big relief...
Sure hope it picks up steam and keeps going.
I am willing to test, have several Intel boards and cpu's available.
Also, writing scores to a thumbdrive would be greatly appreciated as I frequently bench in my basement with no internet available.

[Renfield]

Agreed, this looks like a very worthwhile project, I've been looking for something like this for years. Not being a coder I really need people like you!
Reading page 2 of this thread I was close to crying when it seemed to be dead or dying and then big relief...
Sure hope it picks up steam and keeps going.
I am willing to test, have several Intel boards and cpu's available.
Also, writing scores to a thumbdrive would be greatly appreciated as I frequently bench in my basement with no internet available.

Before I answer your question, I'll make a progress note:
- Core BUBBA test functionality (CPU and memory testing) has been rewritten, and is in testing. There are now four tests. Three of the tests have been time-constrained. You won't need to wait a couple hours on a slow machine, and a fast machine isn't going to get away with running it in 10 seconds and claiming it's as stable as the long-running machine. Most of the test rewriting was to accommodate this. The only test that is not time-constrained is the memory test. This should be understandable, as constantly checking a clock would interfere with memory bandwidth determination.
The three timed tests run 150 seconds each, and the entire test suite completes in ~10 minutes on most machines.
- Environment needs to be remade, but this is in the future...
- I'm currently beginning development of the web-application portion. This will serve as the user interface on the web site, allowing for account creation, and personalization of run scores (e.g. adding comments to a run). It will also handle all the back-end stuff, including daemons that update the database from the uploaded run files, and something special I'm not yet going to talk about. :)

About the manual upload of the result file: We're trying to keep the image as slim as possible, but it did cross our mind that by limiting to only broadband connections, we're shooting ourselves in the foot when it comes to disconnected computers, those on dialup, and newer, possibly unsupported, network devices. Therefore, the workflow will probably look like this...
(This is my ideal workflow, and may differ slightly in reality)
1) User travels to the BUBBA website.
2) User registers with the BUBBA website, and is given a randomized personal identifying key. If the user loses it, he/she can retrieve it any time, but it's needed later.
3) User downloads the BUBBA image and burns it.
4) User pops in BUBBA CD on desired computer and restarts.
5) The BUBBA Benchmark opens, detects system specs, and the user is prompted to enter his identity key.
6) The tests run, and an encrypted results file is generated.
7) The user is asked to either submit the results over the network or save results to USB (possibly floppy) where he/she may upload them at the website.
8) The user restarts computer after removing the BUBBA CD, while the server pulls the file off the queue and adds it to the BUBBA database.
9) The user logs into the BUBBA website and sees his/her pwnage wreaking havoc on the OC community.

That's just the test workflow. There's a lot more that we'll have to offer.
So, assuming I don't get apathetic again, I'll keep you updated on progress.

[MuffinFlavored]

Before I answer your question, I'll make a progress note:
- Core BUBBA test functionality (CPU and memory testing) has been rewritten, and is in testing. There are now four tests. Three of the tests have been time-constrained. You won't need to wait a couple hours on a slow machine, and a fast machine isn't going to get away with running it in 10 seconds and claiming it's as stable as the long-running machine. Most of the test rewriting was to accommodate this. The only test that is not time-constrained is the memory test. This should be understandable, as constantly checking a clock would interfere with memory bandwidth determination.
The three timed tests run 150 seconds each, and the entire test suite completes in ~10 minutes on most machines.
- Environment needs to be remade, but this is in the future...
- I'm currently beginning development of the web-application portion. This will serve as the user interface on the web site, allowing for account creation, and personalization of run scores (e.g. adding comments to a run). It will also handle all the back-end stuff, including daemons that update the database from the uploaded run files, and something special I'm not yet going to talk about. :)

About the manual upload of the result file: We're trying to keep the image as slim as possible, but it did cross our mind that by limiting to only broadband connections, we're shooting ourselves in the foot when it comes to disconnected computers, those on dialup, and newer, possibly unsupported, network devices. Therefore, the workflow will probably look like this...
(This is my ideal workflow, and may differ slightly in reality)
1) User travels to the BUBBA website.
2) User registers with the BUBBA website, and is given a randomized personal identifying key. If the user loses it, he/she can retrieve it any time, but it's needed later.
3) User downloads the BUBBA image and burns it.
4) User pops in BUBBA CD on desired computer and restarts.
5) The BUBBA Benchmark opens, detects system specs, and the user is prompted to enter his identity key.
6) The tests run, and an encrypted results file is generated.
7) The user is asked to either submit the results over the network or save results to USB (possibly floppy) where he/she may upload them at the website.
8) The user restarts computer after removing the BUBBA CD, while the server pulls the file off the queue and adds it to the BUBBA database.
9) The user logs into the BUBBA website and sees his/her pwnage wreaking havoc on the OC community.

That's just the test workflow. There's a lot more that we'll have to offer.
So, assuming I don't get apathetic again, I'll keep you updated on progress.

Do it like this.
1. Go to BUBBA and register with a username.
2. Download BUBBA and get it onto media (make it USB bootable please).
3. Boot into BUBBA.
4. Ask for username.
5. Run tests.
6. Submit tests to the usernames record.

The web portion could be done in about:
4 HTML input fields:
1. username
2. e-mail address
3. password
4. verify password (everyone makes mistakes)

a PHP script to verify:
that the username is not already taken
that the e-mail address is not already registered to another username
that both passwords match

another PHP script to:
store all the data in a MySQL (or an equivalent) table with the correct fields

another PHP script to:
set cookie, allow logged in users site functions (if wanted)

and one PHP script to:
submit results (MySQL (or an equivalent))

and the other to:
view results (MySQL (or an equivalent))

Once again, I could help if needed. :)
Keep it simple.
I would remember MuffinFlavored over 52866, my XS ID.
Let there be 1 standard, username.
Done. :)

The BUBBA itself should go like this.
Boot
Get the following information:
CPU
-manufacture
-model
-front side bus speed
-multiplier
(possibly keep log of temps)
(possibly stepping or silly things like that)

GPU
-manufacture
-vendor
-model
-core clock
-shader clock
-memory clock
(possibly BIOS)
(possibly keep log of temps)

RAM
-manufacture
-model
(possibly chip type, like DG9 or something)
-type (DDR2, DDR3)
-speed
-timings

Motherboard
-manufacture
-model
-BIOS
-chipset


Hard Drive
-manufacture
-model(s)
-if array, and if so array type


Benchmarks should be totally independent. One should not effect the other very much.
1 for CPU
1 for RAM
1 for hard drive (optional, risky, you can't overclock hard drives, but SSD vendors would love the extreme people. :P)
1 for graphics

and then, after ALL that, 1 that combines them all.

There should be a benchmark that is result submittable, and then maybe, more of a stability test.
Unless, the benchmark is the equivalence of a stability test. :)

That should be what is submitted for every result.
If I missed something, let me know.
3DMark would be so much better if we knew exactly what clocks the results were obtained on the ORB.

P.S. These are just my thoughts/ideas.

[Renfield]

Lots of stuff


Re: Workflow
Don't get me wrong. User name and identity key are seperate. You log into the site with user name. Your results show up with your user name. But, you sure wouldn't want anyone else using your user name to submit terrible scores, right? Sure I could ask for password on the bootable CD, but then we'd be doing direct database transactions for authentication in the benchmark. That'd be bad. The ID key, at this point, seems like the easiest way to ensure that the person is who they say they are.
Alternatively
Scores could not be posted until the logged in user approves / comments them (currently, approval occurs on the CD). It's a little extra work on the user side. Let me know what you guys think. I want the app to be secure and easy for end user.

Re: Webapp suggestions
Yes, I could throw a few php scripts around. But, I prefer to have a robust, fault-tolerant, and secure application. I'm using a MVC Java layout: servlets, model classes, jsps and a JDBC secure connection to the database. Dev database is PostgreSQL, production will be MySQL based. The web application is going to provide efficient searchability, sortability, pagination, and easy user access. Quality over haste is the emphasis, but this is going along quickly and smoothly. Good progress on tonight's start.
Don't worry about implementation details. Assure you we have that down. ;)

Re: Data collection
number of procs, manufacturer, model, stock clockspeed, and current clockspeed are recorded. No desire to record a temperature graph, as the extra daemon would throw off the precision of the benchmark.

Re: Benchmark suggestions
Until we have an initial release, I'm keeping it at core marks: CPU and memory benches. CPU done. Mem done. HDD doable. VGA is a sensitive subject.

Always glad to hear suggestions and input.
Now, must sleep before work.

[[XC] gomeler]

Nothing wrong with the user having to approve the benchmark results. Would be rather nice, I wouldn't want the 10,000 loops I'd run to test any sort of tweaks to show up on the website. Being able to run a single test at a time would also be ideal, suites typically blow as you don't necessarily want the same thing for various tests. VGA tests would be ideal also, even a simple openGL app that measures FPS and converts into a score.

[MuffinFlavored]

Something I thought of:
Benchmarks 32-bit or 64-bit?

I would lean towards 64-bit, if you can utilize it.

Whipped this up in a quick state of though.

CREATE TABLE `results` (
`id` int(11) NOT NULL auto_increment,
`userid` int(11) NOT NULL,
`motherboard_manufacture` char(128) NOT NULL,
`motherboard_model` char(128) NOT NULL,
`motherboard_chipset` char(128) NOT NULL,
`cpu_manufacture` char(128) NOT NULL,
`cpu_model` char(128) NOT NULL,
`cpu_speed` char(128) NOT NULL,
`cpu_fsb` char(128) NOT NULL,
`cpu_multiplier` char(128) NOT NULL,
`cpu_cores` int(11) NOT NULL,
`cpu_number` int(11) NOT NULL,
`ram_manufacture` char(128) NOT NULL,
`ram_model` char(128) NOT NULL,
`ram_speed` char(128) NOT NULL,
`ram_timings` char(128) NOT NULL,
`hdd_manufacture` char(128) NOT NULL,
`hdd_model` char(128) NOT NULL,
`hdd_capacity` char(128) NOT NULL,
`hdd_rpm` char(128) NOT NULL,
`hdd_raid` char(128) NOT NULL,
`graphic_manufacture` char(128) NOT NULL,
`graphic_model` char(128) NOT NULL,
`graphic_core` char(128) NOT NULL,
`graphic_shader` char(128) NOT NULL,
`graphic_memory` char(128) NOT NULL,
`graphic_number` int(11) NOT NULL,
`cpu_benchmark_score` int(11) NOT NULL,
`ram_benchmark_score` int(11) NOT NULL,
`hdd_benchmark_score` int(11) NOT NULL,
`graphic_benchmark_score` int(11) NOT NULL,
`date` char(32) NOT NULL,
`time` char(16) NOT NULL,
PRIMARY KEY (`id`)
)


http://img186.imageshack.us/img186/6792/64899275qy1.jpg

[Renfield]

...

32. May have a separate 64-bit download in the future.

Re: database
Minimal specs are gathered about the user's computer. Maybe in a future version of the application I'll go hog wild. Right now it's much more basic. If a user wants to have more information, that's certainly attainable through the benchmark comments which the user can enter.
I'll say it again: Early version is going to be CPU and Memory tests. Not too worried about having to fight with GFX drivers yet. Partner is officially going full effort on Parasol Video, so this is a one-man show now. I'm not going to stress myself over every feature on a project I'm doing for fun.

Based on suggestions in this thread:
The user's identity key has been scrapped. Encrypted results file will be uploaded by application or user, tied to user name. User will need to approve and optionally comment a run when it enters the user's approval list.
The multiple CPU tests have been merged into a single test.

Keep the functionality and usability suggestions coming. However, while I appreciate the thoughts and help, please don't worry too much about code or implementation techniques. After 8 hours coding at work, I prefer not to come home to another tech spec or data model.

Thanks everyone.

[Renfield]

Hey guys.
Haven't really been putting a lot of effort into this project this week, and I don't want my own apathy affecting the project either.
So, if it comes down to me dropping the project again, I'm going make the source code available on openparasol.com That may happen this week. If so, anyone can feel free to pick up where we left off.
Just a heads up. I'll post when/if it happens.

[MuffinFlavored]

Sure.

Have you written the benchmarks? That seems to be the only problem for me. I do not know how to load 4 cores to 100% for 1 minute, along with the RAM.

I have a ~30mb (37mb including SSH, screen, w3m) KNOPPIX based disk I used to chroot into my Linux install when I was having trouble. That could very well be used as the base of the project.

I configured the kernel 2.6.24.3 to use only Intel (ICH9, probably ICH9R) south bridges, and I trimmed EVERYTHING else out. I could easily make every supported network adapter a module, and then add NVIDIA and other south bridges support.

[MuffinFlavored]

I just made the dumbest benchmarks ever.

Consumes all 4 cores of my Q6700, 100%.
Consumes the maximum amount of memory allowed.

http://img182.imageshack.us/img182/5200/55486712ku2.jpg


#include <stdio.h>
#include <windows.h>

void WINAPI loop() {
int i = 0;
//char* test;

while (1) {
/*if ((test = malloc(i)) == NULL) {
free(test);
i = 0;
}*/

i++;
}
}

void WINAPI memory() {
int i = 0;
char* test;

while (1) {
if ((test = malloc(i)) == NULL) {
free(test);
i = 0;
}

i++;
}
}
int main() {
HANDLE hThread[4];
DWORD dwThreadId[4];
int n = 0;

for (n = 0; n < 4; n++) {
if (n == 0) {
hThread[n] = CreateThread(NULL, 0, memory, NULL, 0, &dwThreadId[n]);
}

hThread[n] = CreateThread(NULL, 0, loop, NULL, 0, &dwThreadId[n]);
}

WaitForMultipleObjects(4, hThread, TRUE, INFINITE);

return 0;
}


I might as well make a small OpenGL application that just loads 1 texture a bunch of times and makes a bunch of little cubes.

Talk about stability.
Now if only I could assign something to the RAM instead of just consuming it.

Port this to Linux on a small footprinted setup, and you have yourself a bootable benchmark.

[Renfield]

...
Good luck with that.
Here. I don't have access to the latest source version on this box, but this is recent enough. It's the second to last version, the one with 3 seperate CPU benchmarks. I'll leave combining them as an exercise for the reader.

Download from http://www.openparasol.com/source/bubbaPrototype.cpp

What you should know:
For efficiency, the application has seperate threaded and non-threaded methods.
Each benchmark, except the memory test, is time controlled. The constant is at the top.
Each benchmark is performed 15 times. To account for processor throttling, only the last 10 are scored. The average is the retained score. Stats are also taken from the 10 runs to calculate % error.
There is a 2-second cooldown between each of the 15 runs. This helps create a stable play field for OEM machines which have thermal throttling on by default.
Bubba was intended not only for the extreme, but for the average user. This is why we sought to have a simple system. But, feel free to go bananas with any implementation you come up with. Eventually I'll come back to it to do it how I want.
Have fun.

P.S. Muffin: That's really not a benchmark if you're not measuring anything. Note that the way you've commented out your counter control, threads 1-3 will crash on int overflow. All CPU addition operations on thread zero are taking place in processor cache. Also, the memory may be consumed, but it's certainly not being stressed.
See, the only time you're triggering the free, is when its argument is null. free(null) performs no action. Say you've got 32 blocks of memory. Your algorithm is going "Reserve one block. Kay. Reserve two blocks. Kay (29 left). Reserve three blocks. Kay (26 left). Reserve four blocks. Kay (21 left). Reserve five blocks. Kay (16 left). Reserve six blocks. (10 left). Reserve Seven blocks (3 left). Reserve 8 blocks. O shi. Test = null. Free test, nothing happens. i = 0. Reserve 1 block (2 left) reserve 2 blocks (finally done). Reserve 3 blocks. O shi. Test = null. Reserve 1 block. O shi. Reserve 1 block. O shi. Reserve 1 block. O shi. Repeat ad infinitum.