help me build the perfect GPU machine

[antiacid]

So I'm working on a new project and I need a lot of GPU compute power. A LOT of power. I don't need CPU speed, I don't need ram, I don't need storage, I don't need nvidia cards. Here's what I figured so far:

part description
cpu am3 x2 250
board am3 890 fxa ud7
ram 2x1 gb
psu 1250w ocz
box haf X
vidcard 5870 x4
hdd 500gb blue

This setup is about 1700$CAD with used video cards or 2200$CAD new. I'm trying to optimize it.

One of my current ideas is to swap the board/box for something a little less expensive and run three instead of four cards. I can also go for 4x dual gpu cards and run this under linux for maximum performance (the task is not even related to crossfire, so I don't even need to use the link). However, if I do that I'll need to buy dual PSU and I don't think it'll come out ahead, price/performance -wise. This project will be air-cooled unless I can find really good deals on blocks and rads. Currently, finding this board is also tedious (there's a local shop who does it for 265$CAD but it's special order only).

Can you help me figure out what's the best deal in terms of performance per dollars invested, at as high of a performance as possible?

[ripken204]

it depends on what exactly you are trying to do with the gpus?
it doesnt seem to be gaming. so it's probably some type of calculations.
if so, what type of calculations?

[antiacid]

Ok I spent the night spreadsheeting and this seems like a better deal for me:

part description unit qty unit price qty
cpu am3 x2 250 1 $58.51 $58.51
board am3 890fxa-gd70 1 $199.00 $199.00
ram 2x1 gb 1 $23.96 $23.96
psu 1200 antec HCP 1 $267.58 $267.58
box haf X 1 $167.20 $167.20
vidcard 6990 4 $705.00 $2,820.00
hdd 500gb blue 1 $39.59 $39.59
total w/o taxes $3,575.84

Now I estimate this should pull about 1700w with the cards not overclocked. If I overclock, it's going to require 2 psu for sure. I remember reading a thread with a 1200-HCP doing 2200w so that's why I swapped that in.

watercooling
6990 block xspc 4 $99.95 $399.80
radiator mcr320 3 $49.99 $149.97
fans gt ap15 9 $14.99 $134.91
reservoir mcres rev2 1 $19.99 $19.99
pump mcp655 1 $69.99 $69.99
fittings BP 1/2 ID (pair) 7 $4.85 $33.95

WC total (no tax) $808.61


Is there anything I'm missing here? I found a cheaper board that still allows 4x video cards, I found possibly the only PSU that can run 4x6990 and I'm pretty sure the only issue is cooling so I might just be forced to spend an extra 800$ per server.

Can anyone find me a better deal on 2x PSU that would push about 1000w each?

[zalbard]

Antec HCP-1200 can pull up to 2200W, but continuously? Who's going to guarantee that it won't crap out in a few days?
You can see serious benchers using two of these not just for fun...
Why not pick up 4x 5970 instead? Cheaper, lower power draw, and they are just as good at OpenCL.
And you might need more RAM, depends on the application. Hard to give a particular advice here.
As for radiator, I'd look into something like MoRa 3 Lite. One pump will not do.
How are you going to connect your watercooled cards to each other? Think about it, you might need to use SLI fittings (yeah, they work for Radeons, too) or some kind of bridge.

[dairyFarmer]

This looks suspiciously like a Bitcoin mining rig. :p:

I dont have much experience with multi GPU's but i'd feel safer using two psu's than pushing one close to its limits. Especially with dual gpu cards.

4x 6990's in one case is going to be REALLY _hot_ and _noisey_. (for ~ 2700 Mhash/s?)

What do you need a hdd for? Just run linux off USB flash storage, and mine from there.

This is a similar sort of rig using one PSU
http://imageshack.us/f/543/img0709h.jpg/
Made using;
PSU: Silverstone Sst-St1500v2 1500w
Motherboard: MSI 890FXA-GD70

[antiacid]

Antec HCP-1200 can pull up to 2200W, but continuously? Who's going to guarantee that it won't crap out in a few days?
You can see serious benchers using two of these not just for fun...
That's a valid point. However, because of the price premium of that particular model, I'd be better off with 2x ocz 1250w gold rated. I think I should probably do that, I'll update my spreadsheets.



Why not pick up 4x 5970 instead? Cheaper, lower power draw, and they are just as good at OpenCL.
I would take a 5970 any day but no online retailer sells it anymore and ebay prices are up to 6990 levels. Even the more local B/S/T forums haven't yielded anything. As for the 6990, I found a shop 2 hours away that should receive a shipment from Hong Kong next monday so at least I know I can get at least 4, if not 8 on monday.



And you might need more RAM, depends on the application. Hard to give a particular advice here.
As for radiator, I'd look into something like MoRa 3 Lite. One pump will not do.
I'm fine with 2gb ram, if there's a decent deal for used 2x2gb ram I'll take it too. It's more a question of finding that deal.

As for the mora3, I can't find it in a canadian e-tailer and it's somewhat more expensive to get from the states (i.e. ppcs) with the customs and fees than getting 3xmcr320.

For the pump, I'll change it for a mcp35x (more head, should get me thru 4 blocks+3rads).



How are you going to connect your watercooled cards to each other? Think about it, you might need to use SLI fittings (yeah, they work for Radeons, too) or some kind of bridge.
I was going to get barbs and tubing to save on cost. Since I'm not planning on getting expensive EK blocks, I don't think their bridges will work on xspc cards.

[zalbard]

As for the mora3, I can't find it in a canadian e-tailer and it's somewhat more expensive to get from the states (i.e. ppcs) with the customs and fees than getting 3xmcr320.
Doesn't have to be MoRa 3 in particular. MoRa 2 and Phobya Xtreme are fine, too.

[antiacid]

4x 6990's in one case is going to be REALLY _hot_ and _noisey_. (for ~ 2700 Mhash/s?)

My 950/1200 5970 does 850MH/s and I dropped in a 5870 a couple of days ago and clocked it to 900/1200 in CCC and I get 1200MH/s. The 6990 basically has the same potential (or even a bit more if you have a clue) compared to the 5970 so I'm aiming at 3200MH/s per 8x gpu with air cooling, potentially 3500 with water. Noise / heat / power is not an issue as I'm moving into the perfect location in a couple of weeks. The first big rig is a bit more experimental ;)



What do you need a hdd for? Just run linux off USB flash storage

USB keys are slower to (re)boot and as you probably figured, downtime = money. Also, for 40$ it's more annoying to find a fast usb than not worry about it :)

[dairyFarmer]

I'm definately not saying you cant do it, but your estimates seem a bit optimistic in a couple of ways.

You could get 3000Mhash/s but that would be allude to you wanting to overclock (As ive seen 3x 6990 rigs that get almost exactly 2000Mhash/s). Also the water cooling gear alludes to overclock. However overclocked power draw swells with 4x 6990 so you might find it quite difficult to get stable?.

Also spending $800 on water cooling/overclocking, which will only gain you an additional ~300 Mhash/s if it all works okay.

You'd be better to spend that $800 on another 6990 in another basic rig?

RE: The hard drive, it will draw more power than the USB so if you were making multiple boxes there is a small power usage saving to be made, and downtime shouldnt be too different.

[antiacid]

I'm definately not saying you cant do it, but your estimates seem a bit optimistic in a couple of ways.

You could get 3000Mhash/s but that would be allude to you wanting to overclock (As ive seen 3x 6990 rigs that get almost exactly 2000Mhash/s). Also the water cooling gear alludes to overclock. However overclocked power draw swells with 4x 6990 so you might find it quite difficult to get stable?.

Also spending $800 on water cooling/overclocking, which will only gain you an additional ~300 Mhash/s if it all works okay.

You'd be better to spend that $800 on another 6990 in another basic rig?


I'm honestly not sure what people are doing with their hardware because that list on the wiki is underrated for the hardware I own. If they're off by 10-20% lower, and If I budget the top end reported + 5%, I'm pretty much spot on what I personally already produce. I guess people are reporting random results... If you're referring to those "builders" who supposedly offer a 3x6990 machine for 3800$ (lol I can fit 4 for the same price), they seem to be more marketing BS than real talk, from the pictures/posts I've seen. I basically disregard their performance numbers.

The 800$ on watercooling would result in about 500MH/s more, which is almost another single 6990, as far as I'm concerned. The thing is I'm trying to minimize system footprint for my project. There's a lot more to this than meet the eye ;)

splitting the hardware into 2 boxes wouldn't result in a higher perf/$ because of the additional costs of adding another cpu, more ram, another box (and if WC'ing, another pump/rad combo).

The task has a time-based diminishing return as far as I'm concerned. I chose to interpret it this way: start as big as you can!

[CharMz]

after buying this machine, how long would it take you to regain your money and start turning a profit? How many 24/7 days?

[antiacid]

after buying this machine, how long would it take you to regain your money and start turning a profit? How many 24/7 days?

That's a function of: cost of hardware, cost of electricity, cost of coins, uptime, pool used, difficulty, production rate, time period and taxes. Oh, and there's variance in the production rate (both from the machine and on the return from it). Then again, you could chose to rent a set amount of production capacity on a monthly fee and have a clearer idea of those numbers.

I wouldn't want to bore you with the details but I'm budgeting between 25d and 45d for a big rig. Then again, it's not like I have to sell the coins right away. I can stash until the virtual break-even point is even lower.

[CharMz]

That's a function of: cost of hardware, cost of electricity, cost of coins, uptime, pool used, difficulty, production rate, time period and taxes. Oh, and there's variance in the production rate (both from the machine and on the return from it). Then again, you could chose to rent a set amount of production capacity on a monthly fee and have a clearer idea of those numbers.

I wouldn't want to bore you with the details but I'm budgeting between 25d and 45d for a big rig. Then again, it's not like I have to sell the coins right away. I can stash until the virtual break-even point is even lower.

i see, bitcoins seems pretty interesting its popping up all over the internet...but i see it is also getting bad press for illegal items that can be bought with the coins =0

[antiacid]

i see, bitcoins seems pretty interesting its popping up all over the internet...but i see it is also getting bad press for illegal items that can be bought with the coins =0

You are indeed right. It is getting bad press because of Silk Roads. Then again, this is like saying 1000$ notes are getting bad press because they make large illegal purchases easier to handle... The bitcoin system itself is not illegal. What users chose to do (and declare to the taxman) is up to them :)

I think what I'll end up doing is buying the parts for that top-notch machine and test out the production rate over a week under each scenario (3 cards, 4 cards, 4 cards wc'ed) and then go from there for the following servers. Got to love the engineering design wheel!

[CharMz]

yeah keep me posted this bitcoin thing sounds really interesting, like with anything though, you need lots of money to make money

[antiacid]

Update: So I've been running a quad 6970 setup generating about 1650mh/s and as soon as I get my clamp meter (it's saint-jean-baptiste today here so closed stores) I'll be able to verify the wattage.

So far so good :)

[Craftyman.]

You best be on bitcoinpool :p:

Shameless plug I know I know.

A bit offtopic, where do you get your numbers from and do you have a number for an HD6990? from your post up there it looks like 700Mhash/s?

[antiacid]

You best be on bitcoinpool :p:

Shameless plug I know I know.

A bit offtopic, where do you get your numbers from and do you have a number for an HD6990? from your post up there it looks like 700Mhash/s?

So far I've been building myself tables with the prices of cards at shops I have access to (and I use best guesses to estimate the rest) so it looks like this:

Card table
card price $CAD power (watts) mh/s mh/s / $ mh/s / w
5770 $90.09 108 215 2.39 1.99
5830 $129.75 195 300 2.31 1.54
5850 $189.86 200 350 1.84 1.75
5870 $296.10 220 405 1.37 1.84
5970 $700.00 475 800 1.14 1.68
6750 $114.99 100 165 1.43 1.65
6770 $107.49 120 195 1.81 1.63
6850 $149.99 175 230 1.53 1.31
6870 $160.12 175 310 1.94 1.77
6950 $244.79 220 360 1.47 1.64
6970 $328.03 235 400 1.22 1.70
6990 $745.00 475 800 1.07 1.68

Now the 5xxx models I can't really buy new so the prices for those are more guidelines for when I get used parts, to know how the deal looks.

I'd like to point out that AMD CCC is a pain. To underclock the ram, I had to edit the profiles.xml with lower values because the sliders are retarded.

As for the pool, I've traditionally been with deepbit because they're large, got long-polling and have a decent UI. I'm open to suggestions if someone can demonstrate to me that I'd get more out of another pool! So far I'm only at 2.5-2.6 Gh/s but I've got parts to pump that up considerably ;)

I think that the next step in this setup should be to make a cluster of 24 units with a custom-built rack. I'll be playing on sketch-up and updating this thread for more advices since I'm still new at designing data-center'esque setups :P

[Craftyman.]

Well there's no "tax" at bitcoinpool, I can't compare the two as bitcoinpool is the only pool I've really tried. It's got about 300GH/s and I've heard payment is a bit slow but all the people are pretty helpful and the admins are great.