Ok, thanks for that. Full load core temp? I want to see the delta between core and ambient or core and water temp on SB-E, i'm assuming its about 30*c?Originally Posted by Kensek
Ok, thanks for that. Full load core temp? I want to see the delta between core and ambient or core and water temp on SB-E, i'm assuming its about 30*c?
Do you guys think a mobo will sustain eg 100% 1.55-1.6Vcore 24/7 ? This setups will run flatout all the time, not like we browse, game,... Don't think so... something will pop... I vote unrealistic 5.5Ghz, 5Ghz with aded cooling to the board might just cut it...
Question : Why do some overclockers switch into d*ckmode when money is involved
Remark : They call me Pro AsusSaaya yupp, I agree
No, not 24/7, but 5.5ghz 100% stable hopefully at max 1.55vcore. 0 degrees celcius chilled liquid on the cpu and normal water cooling on the vrm's.
sorry for the thread hijack alpha.
Im running one of the LittleDevil's Phase Change units 24/7/365 and cpu is running minium 30% load at all times, 100% load drops evap temps to around 27 celcius from 30 or 31 celcius.
LittleDevil's unit has been reliably running without problems exept controller problems.
Im not 100% shure what it consumes, forgot to plug my pc and ss into multimeter but should be in range 180-300 watts.
Not shure, it might pop or it might do without problems, around 5.0Ghz-5.2Ghz 24/7 should be running fine.
Last edited by rintamarotta; 07-17-2012 at 10:24 AM.
::: Desktop's - Intel *** Intel 2
2 x Xeon E5-2687W *** Intel i7 3930k
EVGA SR-X *** Asus Rampage IV Extreme
96Gb (12x8Gb) G.Skill Trident X DDR3-2400MHz 10-12-12-2N *** 32Gb (8x4Gb) G.Skill Trident X DDR3-2666 10-12-12-2N
3 x Zotac GTX 680 4Gb + EK-FC680 GTX Acetal *** 3 x EVGA GeForce GTX780 + EK Titan XXL Edition waterblocks.
OCZ RevoDrive 3 x4 960Gb *** 4 x Samsung 840 Pro 512Gb
Avermedia LiveGamer HD capture card
Caselabs TX10-D
14 x 4 TB WD RE4 in RAID10+2Spare
4 x Corsair AX1200
::: Basement DataCenter :::
[*] Fibreoptic connection from operators core network
[*] Dell PowerConnect 2848 Ethernet Switch [*] Network Security Devices by Cisco
[*] Dell EqualLogic PS6500E 96Tb iSCSI SAN (40 2Tb Drives + 8 Spare Drives, Raid10+Spare Configuration, 40Tb fail safe storage)
[*] Additional SAN machines with FusionIO ioDrive Octal's (4 total Octals).
[*] 10 x Dual Xeon X5680, 12Gb DDR3, 2x100Gb Vertex 2 Pro Raid1 [*] 4 x Quad Xeon E7-4870, 96Gb DDR3, 2x100Gb Vertex 2 Pro Raid1
[*] Monster UPS unit incase power grid failure backed up by diesel powered generator.
Not really.
http://www.cs.man.ac.uk/~rizos/papers/Image3.gif
Your number of interconnect paths I think is O(n^2) or O(n^3) (I always forget which it is) but it's an exponential or power function. Which means that if you spend more time hopping between nodes to be able to get to and from the data (or moving said data); then you're not really doing actual useful work on it.
Suppose you have a problem decomposed into m-processes on m-processing cores, and that it takes F number of floating point operations at a rate of n floating point operations per second.
If m(i) processing core can perform all of the necessarily operations in < 1 second; (i.e. F < n); then your per-processor efficiency is only F/n/m. And if that's < 1.0, then you're "short cycling" the processor, and your computational waste is 1-(F/n/m).
Hmmm....okay.
If he's referring to my set up, then yes, I would be running it at 5.5 GHz at 100% 24/7. If he's talking about his own setup or yours, then the answer is as it is.No, not 24/7, but 5.5ghz 100% stable hopefully at max 1.55vcore. 0 degrees celcius chilled liquid on the cpu and normal water cooling on the vrm's.
sorry for the thread hijack alpha.
And that's okay. It's still an interesting thread. It's got a fair bit of really good information on here. I'm still a little antsy about running phase change 24/7 (because out of those 24 hours, I'm gone half the time, which also means that the system MUST have a self monitor and self-protect where it NEEDS to be able to shut itself off if there's anything wrong with it because I won't be remotely close to being able to get to it.)
And I just passed 2 kWh last month...
flow man:
du/dt + u dot del u = - del P / rho + v vector_Laplacian u
{\partial\mathbf{u}\over\partial t}+\mathbf{u}\cdot\nabla\mathbf{u} = -{\nabla P\over\rho} + \nu\nabla^2\mathbf{u}
Ok i finally desided to put that multimeter to check out what kind of watt's LD's SS uses.
At startup it takes around 700 watts and then drops down to 350-400watts, to take in mind that this SS is tuned for 300watt load its efficiency is at around 85%.
Add my PC's power usage that is run with the SS, i7 3770k + 16Gb DDR3 2133 + 3 x GTX680 + OCZ Revo 3 X2 240 + Watercooling for GPU's, it will get close to 1.3-1.6kW overclocked including the SS, i dont have much room left due my SR-X machine being in same wall power plug, i can go to 3.8kW with 240v 16amp fuse but i have two machines in same Caselabs TX10-D case so im near to limites of the fuse.
Last edited by rintamarotta; 07-21-2012 at 09:13 AM.
::: Desktop's - Intel *** Intel 2
2 x Xeon E5-2687W *** Intel i7 3930k
EVGA SR-X *** Asus Rampage IV Extreme
96Gb (12x8Gb) G.Skill Trident X DDR3-2400MHz 10-12-12-2N *** 32Gb (8x4Gb) G.Skill Trident X DDR3-2666 10-12-12-2N
3 x Zotac GTX 680 4Gb + EK-FC680 GTX Acetal *** 3 x EVGA GeForce GTX780 + EK Titan XXL Edition waterblocks.
OCZ RevoDrive 3 x4 960Gb *** 4 x Samsung 840 Pro 512Gb
Avermedia LiveGamer HD capture card
Caselabs TX10-D
14 x 4 TB WD RE4 in RAID10+2Spare
4 x Corsair AX1200
::: Basement DataCenter :::
[*] Fibreoptic connection from operators core network
[*] Dell PowerConnect 2848 Ethernet Switch [*] Network Security Devices by Cisco
[*] Dell EqualLogic PS6500E 96Tb iSCSI SAN (40 2Tb Drives + 8 Spare Drives, Raid10+Spare Configuration, 40Tb fail safe storage)
[*] Additional SAN machines with FusionIO ioDrive Octal's (4 total Octals).
[*] 10 x Dual Xeon X5680, 12Gb DDR3, 2x100Gb Vertex 2 Pro Raid1 [*] 4 x Quad Xeon E7-4870, 96Gb DDR3, 2x100Gb Vertex 2 Pro Raid1
[*] Monster UPS unit incase power grid failure backed up by diesel powered generator.
Hmm....so it's really not that much more power efficient than having a single system Xeon based system (with four DP nodes). Damn.
Back to the drawing board...
*edit*
In my other thread where I was testing the 3930K to 4.5 GHz, it settled in around like 230 W running IBT. So the jump from 4.5 GHz to 5-5.5GHz is like another 500 W? Wow...
Last edited by alpha754293; 07-24-2012 at 05:16 AM.
flow man:
du/dt + u dot del u = - del P / rho + v vector_Laplacian u
{\partial\mathbf{u}\over\partial t}+\mathbf{u}\cdot\nabla\mathbf{u} = -{\nabla P\over\rho} + \nu\nabla^2\mathbf{u}
whole system power or cpu? 3930k at 5ghz will top out about 300watts for cpu alone, maybe 320-350 at 5.5ghz, i think in reality its a bit less than that.
4.5ghz+ is where power efficiency goes out the window for sb-e, especially compared to the gain in processing power your getting. if you go with the 2 or more xeons at there max turbo frequency you will be in the sweet spot efficiency wise, maybe not brute power but damn fast all the same... you could always source some older x58 xeons and overclock them though, i don't think there is an enormous difference between 1366 and 2011.
Last edited by Liam_G; 07-25-2012 at 02:52 AM.
Whole system.
The price for the 1366 Xeons doesn't make it worth it. Actually, the models that I would be looking at, the X5690 is still more than a E5-2670. (Fewer cores, but faster vs. more cores and slower); but it's at about the same price point.
Although according to Anandtech, there IS quite a bit of difference between the two.
http://www.anandtech.com/show/5553/t...for-servers/12
flow man:
du/dt + u dot del u = - del P / rho + v vector_Laplacian u
{\partial\mathbf{u}\over\partial t}+\mathbf{u}\cdot\nabla\mathbf{u} = -{\nabla P\over\rho} + \nu\nabla^2\mathbf{u}
i was actually thinking of the w3580 at about 4.5ghz x 2 on the evga sr2, but like you said, ipc is about 20-30% less so 3960x is probably the better choice at the 12-16 threads mark, i'm just trying to find something you might actually be interested in buying, are you actually going to buy something or are we just going to talk about it?
seems to me there are a number of options, you should choose based on what you think you can utilize best, ie do you need more cores, more ghz, or can you utilize multiple systems
#1 one i7 3960x at 5ghz, pretty doable and relatively cheap, 12 threads at 5ghz
#2 two e5 xeon 2687's at 3.1ghz on the evga srx, i believe there may be minimal oc'ing on this board, like 5% or so. so you would have 32 threads at 3.1+ghz. pretty steep price though
#3 two x5670's at 4+ghz but about 30% less ipc than SB-EP so not sure how much you gain, unless these can get to 4.5ghz + then probably not enough, also only 24 threads
#4 start accumulating a farm of i7 3770ks at 4.8ghz if you can use it
#5 all out crazy build with 4 e5 2687's, start robbing banks now.
#6 overclock the hell out of a single 3770k or 3960x on phase change for some fun and to see where it gets you, although you probably won't get far above good h20
i could go on
personally i would buy the 3960x and run with it until haswell ep comes out, it's also likely intel will be pushing higher tdp server chips out by then to compete with nvidias gpgpu stuff. so they will clock higher and have a lot of threads
take it or leave it i guess, everything has been pretty much hashed out in this thread so if none of that suits your needs then just wait it out
Last edited by Liam_G; 07-27-2012 at 03:29 AM.
it is possible to run up to about 5.5Ghz under phase.. if you find a 55x cpu that is
running a 54x chip under phase at work, 1.55v loaded under prime95. Using a LD Cooling V10 for the test..
Well, it's still kind of up for debate. It's an optimization problem to an extent because between power, performance, space, and cooling, (and to somewhat of a lesser extent, noise), and cost, it's trying to find the best solution for ALL of those criteria.
HTT is likely going to be out. (I'll still need to run more tests, but chances are, it'll likely remain off since most of my preliminary testing for both FEA and CFD is showing that HTT slows the run down).
As long as the funds are available, I will be buying. But that means that I must do a LOT of research because if I start building the Supermicro 4-node system, the initial cost is quite high (chassis alone is $3k), but if I build it so that I would have very similar final performance levels (or estimated performance levels), from what I can tell; it'll likely cost about the same; except that having many smaller systems will consume more power. (My last hydro bill was $317, and I topped about 2200 kWh in terms of power consumption).
For most people, running just one or two systems isn't such a big deal. But when you're doing a crash sim that runs for 9 days, or a combustion CFD run that runs for 5 days; it adds up.
Two Xeon E5-2687s is quite expensive too. (~$500 difference per CPU).
Somewhat surprisingly, dual Xeon X5670s isn't that much cheaper. It still commands $1.1k per CPU. (a Xeon E5-2670 is only $1363, which given the prices for the other two, it really isn't that bad.)
One of the best alternatives might be for me to go with dual 3820s (since I can't get dual LGA1155 boards for dual 3770K). The processors are a lot cheaper and I can plug THAT into an EVGA SR-X. So that's the upside. Downside is that I would be limited in terms of max. RAM.
Remember that only the first node for the E5-2670 will cost me $6.5k ($3k for the chassis, $3k for the processors, and $0.5k for the RAM, hard drives I have already). Each additional node after that will only be $3.5k.
At the same time too, I'm also trying to balance between having lots of slower cores (lots of intercore traffic) vs. having fewer, faster cores, less intercore traffic, but I would then be limited by RAM for each node (and higher power consumption per node).
Is it possible to hit 5.5 GHz without phase cooling? I'm still debating whether the power consumption would be worth it or whether I should just "settle" with 4.5 GHz with conventional liquid cooling. (Lower total overall system power consumption).it is possible to run up to about 5.5Ghz under phase.. if you find a 55x cpu that is
running a 54x chip under phase at work, 1.55v loaded under prime95. Using a LD Cooling V10 for the test..
flow man:
du/dt + u dot del u = - del P / rho + v vector_Laplacian u
{\partial\mathbf{u}\over\partial t}+\mathbf{u}\cdot\nabla\mathbf{u} = -{\nabla P\over\rho} + \nu\nabla^2\mathbf{u}
I'm tempted to try for 5ghz but I would like to stay under 85c under load and who knows what kind of voltage I will need for 5ghz.
What kind of cooling are you using? Air? Liquid? Phase? DDEC?I'm tempted to try for 5ghz but I would like to stay under 85c under load and who knows what kind of voltage I will need for 5ghz.
flow man:
du/dt + u dot del u = - del P / rho + v vector_Laplacian u
{\partial\mathbf{u}\over\partial t}+\mathbf{u}\cdot\nabla\mathbf{u} = -{\nabla P\over\rho} + \nu\nabla^2\mathbf{u}
Alright, I just did some number crunching for some of the various scenarios listed above (with some minor changes).
Couple of assumptions - because I don't know how well my simulations are going to respond to the difference processors (and whether it's single processor or multiple processors (questions concerning NUMA/SMP performance comes to mind)) and also the whole if the problem are broken down into more cores, that there's going to be much more intercore traffic/communication (since MPI is explicit for passing variable data around); and I don't have a way of measuring what that traffic volume or bandwidth requirement is, so I can't use that as a benchmark either.
And also because of varying clockspeeds, I'm going to "rank/rate" the various options ONLY by the TOTAL CLOCKSPEED (i.e. n-processors * x-clockspeed). It's the only thing that I can use that would REMOTELY be close to being a fair comparison. This is only a comparison for the cost of hardware only. No considerations for space, power, or cooling is taken into effect/factored into the calculation (which I could add that, but I'll do that (maybe) later.) Not also, that HTT is NOT taken into the calculation. Only just the physical core/# of physical FPUs from the chip level (i.e. FPADD/FPMUL is not treated as separate FPUs)
Total cost is calculated including shipping and/or tax (where applicable) to within continental lower 48 states as of Sun Aug 12 10:49 EDT 2012.
So here are the options:
#1) 4-DP node Supermicro blade, 2 RU, 1100 W 1+1 redundant PSU.
First node + chassis:
Dual Xeon E5-2670 (2.6 GHz, 8-core, HTT), 8 * Kingston DDR3-1600 8 GB ECC Reg. RAM, 2* Intel 520 240 GB SSD SATA 6 Gbps, Supermicro 2027TR-HTRF 2U barebones chassis (includes power supply).
$6,620.37
each additional node:
Dual Xeon E5-2670 (2.6 GHz, 8-core, HTT), 8 * Kingston DDR3-1600 8 GB ECC Reg. RAM, 2* Intel 520 240 GB SSD SATA 6 Gbps
$3,704.33
Total cost: $17,733.36
Total "performance" rating/index: 166.4 GHz (2.6 GHz/processor * 8 cores/processor * 2 processors/node * 4 nodes/chassis)
Option #2) dual 3820 SR-X * 6
Each system:
Dual Core i7 3820 (3.6 GHz, 4-core, HTT), EVGA SR-X, 8 * Kingston DDR3-1600 8 GB, 2* Intel 520 240 GB SSD SATA 6 Gbps, Corsair Carbide 500R White, Corsair AX850 PSU, PNY GTX560
$2,362.44
Total cost: $18,899.51
Total performance rating: 172.8 GHz
If OC'd to 5 GHz: (it'll be *4 instead of *6)
Total cost: $9,449.76
Total performance rating: 160 GHz
Option #3) Single 3930K * 9
Core i7 3930K (3.2 GHz, 6-core, HTT), Asus Sabertooth X79, 8 * Kingston DDR3-1600 8 GB, 2* Intel 520 240 GB SSD SATA 6 Gbps, Corsair Carbide 500R White, Corsair AX850 PSU, PNY GTX560
$2,073.85
Total cost: $18,664.64
Total performance rating: 172.8 GHz
If OC'd to 4.5 GHz: (it'll be *6 instead of *9)
Total cost: $12,443.10
Total performance: 162 GHz
Option 4) dual Xeon E5-2687W SR-X (there are other DP Xeon options available, but sticking with the EVGA SR-X for consistency)
Dual Xeon E5-2687W (3.1 GHz, 8-core, HTT), EVGA SR-X, 8 * Kingston DDR3-1600 8 GB, 2* Intel 520 240 GB SSD SATA 6 Gbps, Corsair Carbide 500R White, Corsair AX850 PSU, PNY GTX560
$5,487.27
Total cost: $16,461.81
Total performance: 148.8 GHz
You might be able to get a small OC out of it, but it'll still take about three of these systems to even remotely match the performance of Option 1.
Option 5) dual X5670 (still sticking with the EVGA SR series board, but again, there are logs of dual LGA1366 options as well)
Dual X5670 (2.93 GHz, 6-core, HTT), 8 * Kingston DDR2-667 8 GB ECC Reg., 2* Intel 520 240 GB SSD SATA 6 Gbps, Corsair Carbide 500R White, Corsair AX850 PSU, PNY GTX560
$5,909.46
Total cost: $29,547.30
Total performance: 175.8 GHz
So there's the cost breakdown and approximate performance numbers for the different options.
If you want to see the spreadsheet, I'll have to try and figure out a way to upload it to Google docs or something...
flow man:
du/dt + u dot del u = - del P / rho + v vector_Laplacian u
{\partial\mathbf{u}\over\partial t}+\mathbf{u}\cdot\nabla\mathbf{u} = -{\nabla P\over\rho} + \nu\nabla^2\mathbf{u}
Now comes the space, power, and cooling calculations. So, a quick couple of things - in order to somewhat simplify the calculations in order to get a representative "feel" for how the different options playout; and they might not be representative of actual real-world conditions and that there's obviously more than one solution to at least some of these calculations, but it's really just a ball-park figure that gets plugged into the optimization problem. I can ALWAYS make the individual terms of the equation in terms of how it's calcaluted/derived A LOT harder, but seeing as how people have already b*tched about how I'm an engineer and most people here aren't (or at least not necessarily so), so I'll try to simplify it somewhat.
So here are the assumptions:
Space is measured in square feet. I'm picking an arbitrary price of $500/ft^2/month. Footprint is calculated STRICTLY from the width and length of the chassis. Stacking is not considered (but very likely a possible solution, of course). Oh yeah, dimensions are either from the store or from the manufacturer (whichever was easier for me to find while I'm doing this). I picked the price because I have no idea how much office space retails for (say...in the middle of downtown Manhattan). I don't really care either. It's all relative anyways. (i.e. my per-sq.ft. cost at home is obviously less, but the ratio between any two option is constant.)
Cooling is taken at the same power level as the PSU rating. I don't want to have to get into actually calculating the cooling power output. Too lazy. And a bit too complicated. And that way I don't have to answer/attempt to explain my picking a "rule-of-thumb" factor of the input power.
Power is the PSU rating. Obviously what's drawn at the wall is different, but it varies with load. Also gives the maximum condition. Sort of.
Electricity cost is pegged at $0.10/kWh. (The cost of electricity right now in Ontario is a little big lower than that, but it's got a weighted formula - which again, just really adds complexity. So I rounded up and kinda figured that it'll at least also partially factor in price increases.) The system is also going to be assumed to run at the full power output 24/7 and cost is calculated on a monthly average of 30 days (24 hrs*30 days) and also for a 5-year period as well.
*edit* And this is AFTER the OC'd has been taken into account. If I don't, the numbers for Options 2, 3, and 4, will just get worse.
Anyways, onwards:
Option 1) Only one chassis
Space: 26.75" * 17.25" (2.229 ft * 1.4375 ft = 3.2 ft^2) * $500/ft^2/month = $1,602.214
Power/cooling: 1100 W + 1100 W = 2200 W = 2.2 kW @ $0.1/kWh = $0.22/hr * 24 = $5.28/day = $158.40 /month * 60 months = $9,504
Space+Power+Cooling for 5 years: $105,636.80
Total 5 year cost of ownership (SWaP + ownership): $123,370.17
Option 2) 4 850 W chassis.
Space: 20.5" * 8.1" (1.7083 ft * 0.675 ft * 4 = 4.6125 ft^2) * $500/ft^2/month = $2,306.25
Power/cooling: 4*850 W *2 = 6.8 kW @ $0.1/kWh = $0.68/hr * 24 = $16.32/day = $489.60/month * 60 months = $29,376
Space+Power+Cooling for 5 years: $167,751
Total 5 year cost of ownership (SWaP + ownership): $177,200.76
Option 3) 6 850 W chassis.
Space: 20.5" * 8.1" (1.7083 ft * 0.675 ft * 6 = 6.9175 ft^2) * $500/ft^2/month = $3,459.375
Power/cooling: 6*850 W *2 = 10.2 kW @ $0.1/kWh = $1.02/hr * 24 = $24.48/day = $734.40/month * 60 months = $44,064
Space+Power+Cooling for 5 years: $251,626.50
Total 5 year cost of ownership (SWaP + ownership): $264,069.60
Option 4) 3 850 W chassis.
Space: 20.5" * 8.1" (1.7083 ft * 0.675 ft * 3 = 3.459375 ft^2) * $500/ft^2/month = $1,729.688
Power/cooling: 3*850 W *2 = 5.1 kW @ $0.1/kWh = $0.51/hr * 24 = $12.24/day = $367.20/month * 60 months = $22,032
Space+Power+Cooling for 5 years: $125,813.3
Total 5 year cost of ownership (SWaP + ownership): $142,275.06
Option 5) 5 850 W chassis.
Space: 20.5" * 8.1" (1.7083 ft * 0.675 ft * 5 = 5.765625 ft^2) * $500/ft^2/month = $2,882.813
Power/cooling: 5*850 W *2 = 8.5 kW @ $0.1/kWh = $8.50/hr * 24 = $20.40/day = $612.00/month * 60 months = $36,720
Space+Power+Cooling for 5 years: $209,688.80
Total 5 year cost of ownership (SWaP + ownership): $239,236.05
Now, can you stack some of those cases to reduce the footprint cost, sure. Absolutely. But at that point, I'd probably be going to AT LEAST 3U rackmounts and the cost goes up per chassis.
Again, there are many different options/solutions, but this just gives an idea as to what the TCOs are for the various options.
I'm sure that if you want to give a quick run-down/run-through for what the costs would look like if you were to stack the cases, you can do that. But the bulk of the costs actually will come from power and because you've got so many more systems, it'll likely overshadow the cost of real estate. But if you're really that interested, you can always run the numbers yourself if you want to stack the Carbide cases.
Last edited by alpha754293; 08-12-2012 at 08:02 AM.
flow man:
du/dt + u dot del u = - del P / rho + v vector_Laplacian u
{\partial\mathbf{u}\over\partial t}+\mathbf{u}\cdot\nabla\mathbf{u} = -{\nabla P\over\rho} + \nu\nabla^2\mathbf{u}
Just water cooling.
Cool. (No pun intended.)
flow man:
du/dt + u dot del u = - del P / rho + v vector_Laplacian u
{\partial\mathbf{u}\over\partial t}+\mathbf{u}\cdot\nabla\mathbf{u} = -{\nabla P\over\rho} + \nu\nabla^2\mathbf{u}
up and running 3930k and a Rampage Formula just hit 5ghz well impressed. Cooled with H80i amazing little kit for the price.
Uploaded with ImageShack.us
Posting Permissions
Reply With Quote
Bookmarks