BOINC servers down

[sierra_bound]

You may have noticed the BOINC servers are down. My half-day supply on WU's has almost run out.:(

http://www.worldcommunitygrid.org/forums/wcg/viewthread?thread=10703

[[XC] DragonOrta]

If you end up running out of WCG work, feel free to load up F@H while you're waiting. :D

[njkid32]

Yep I am totally out of WU's... All I have to say is WTF!!!

[[XC] Jaco]

what the hell is going on , servers are down for like 11hours now:rolleyes:

[[XC] serlv]

I have to take a really big dump... ;)

[joshd]

I have to take a really big dump... ;)

yea... and I need to take it NOW! :eek:

[meshmesh]

Yes. It has been down too long indeed. Coming up to the 12th hour?
It appears they have some serious issue. Check the language of the announcement, over five hours ago.
There is a problem with the BOINC servers that is preventing agents to connect to them. You'll see a 'Project is down' message, if you look at the messages tab.

We are working on the problem but do not have an estimated time when they'll be back up.

We'll keep you posted. Hopefully everyone can survive on their task queue for the duration.

Never read this kind of language before there. Alther (WCG Tech), apparantly at the time of the post, was still hunting for the cause of the fault after six hours from inception, and of caurse he has no idea when he can get it fixed, since he does not know what the actual problem is!

I don't know who Alther is. All I know is that he is probably on his own for hours to come. Hopefully someone will come to his rescue sometime in the morning (if they have not off for the holidays already and left it to him). :)

Luckly for me, I have suspended WCG on all rigs since yesterday and fully switched to help our XS F@H brothers till end of the 26th. At which time I would be switching back to WCG to finish the couple of WUs remaining just before their deadline.

May I suggest, that if you like, just do the same:
Suspend all BOINC activity.
Shut down BOINC service
Install F@H client in service mode as per SparkyJJO step-by-step instructions in the second post of this (http://www.xtremesystems.org/forums/showthread.php?t=125551) thread. Piece of cake compared to boinc install, and runs with no fuss what so ever.

A quick comment:
1)Your member name need to be unique. You can make sure of that by searching the name at the F@H database (http://fah-web.stanford.edu/cgi-bin/main.py?qtype=userstats) at their website. However, it took me too long to get it done because their server was overloaded and kept refusing my connection. May I suggest, if you don't have an account already, to just use something like XS_yourusername. That way you know you will be unique and skip the check step.
2)If you have multiple pcs, repeat the instructions per pc, starting with machine 1,2,... per core again.

Guys, as we stand today, XS needs at least 100K FAH extra points per day for the next five days if are to win this one. That translates to about 900K WCG points per day. I know that some of us will be shutting off for veccation etc, but I think that if the rest of us switch everything we have ASAP, we will just scrape by. This race is as tight as they ever get. Trust me guys, every bit helps. We can do it if we all switch over there today.

I don't need to remind you of our latest discussions about XS members, etc... All I would say is this: we start with ourselves. Thanks.

[davidletterboyz]

Hi...I got this error and annoyed me. Came here and found the server is down. I switched to F@H...now running the first WU. LOL

[L0$t Pr0PhEt]

or switch to D²OL, not that big wu's :)

[meshmesh]

Just to update everyone, knreed just posted on WCG. They do have a real problem.
Our servers are located at a hosting facility in Boulder, CO (USA). Unfortunately, the problem affecting the BOINC servers requires someone to have physical access to the servers. Boulder is in the middle of a significant snow storm and no-one is able to travel at this time. Additionally due to the snow storm the hosting facility is operating with significanly reduced staff. As a result it may be awhile before we are able to resolve this problem

So I guess they are having an extra white Xmas :)
Solution: see my post above for the time being.

Cheers.

[[XC] itznfb]

our WU's better not expire :mad:

[sierra_bound]

"Our servers are located at a hosting facility in Boulder, CO (USA). Unfortunately, the problem affecting the BOINC servers requires someone to have physical access to the servers. Boulder is in the middle of a significant snow storm and no-one is able to travel at this time. Additionally due to the snow storm the hosting facility is operating with significanly reduced staff. As a result it may be awhile before we are able to resolve this problem."
This explanation makes absolutely no sense. Someone needs to access the servers, but there is significantly reduced staff. Does that mean no one is there?

[njkid32]

I am switching to F@H till this is resolved:) I just dont see the point in letting this machine sit here and do nothing..

[heater918]

This explanation makes absolutely no sense. Someone needs to access the servers, but there is significantly reduced staff. Does that mean no one is there?

Polite way of saying only the cleaning staff is available :D

[meshmesh]

It probably meant that when the national weather center issued an alert for the snow blizard, the staff did not want to get stuck, because they know that the next shift will not come to replace them.

So instead of being snowed in for a day or so with no food etc..., they just left the poor security guard alone and ran home before the police shuts the highways down.

Happens everywhere. At the end of the day, they are just humans. The question to ask is why does WCG need to access the servers at all? Don't they have KVM / remote power recycle / virtual media over IP? This is nearly twenty hours and counting. Very bad indeed.

[lawrywild]

Server Is Up For Me!!

[sierra_bound]

I will be patient. But I'm looking at web traffic cams of Boulder right now. Most of the main roads have been cleared and look driveable to me. This is not like the days when you had to cross the Rockies in covered wagons.

[meshmesh]

Server Is Up For Me!!

Oh really? You are on Boinc or UD ? I cannot tell at the moment because I am crunching the F@H till wednesday.

Edit: Problem solved. Posted on WCG :cheer:
The outage has been resolved. The hosting facility was able to get someone to reboot the server. This allowed us to connect remotely and fix the problem. Work is being sent out now and results are being validated.

That's crazy. I guess that "someone" in the facility is in big trouble come tomorrow. Hmm, fifty thousand hours of crunching time lost? :slap:

[lawrywild]

BOINC ofcourse :)

[sierra_bound]

Yeah, it looks like it's back up. Downloading more WU's now.:)

[lawrywild]

I still had about half a day of WU's left because I download enough for 2 days lol :D

[[XC] Jaco]

amateurs

[gr8golf]

Where do I change the amount of WU's downloaded? I'd like to change it so as to have 3 days worth on hand.

[sierra_bound]

Under Device Profile - Advanced Options, change "Connect to network about every" to however long you want. For example, setting it to 3.0 would give you a three days supply.

[lawrywild]

amateurs

:confused:

Anyway, server's down again, niiice :rolleyes:

[[XC] Jaco]

yes , i just noticed to

:upset:

[[XC] flat-four]

well, this works out perfect for a little F@H crunching. Got my 4 accessable boxes switched over

[mike047]

well, this works out perfect for a little F@H crunching. Got my 4 accessable boxes switched over

Good work.

[gr8golf]

Under Device Profile - Advanced Options, change "Connect to network about every" to however long you want. For example, setting it to 3.0 would give you a three days supply.

Does that also control how often the results are transmitted back to WCG?

[mike047]

Does that also control how often the results are transmitted back to WCG?


I believe the work return is on a time period cycle. If you choose 3.0 days, it will maintain 3 days worth of work backlog. I keep mine at 2.1 days.

[[XC] mysticmerlin]

Does that also control how often the results are transmitted back to WCG?

Yes it does as far as I know.

[mike047]

Yes it does as far as I know.


mine seem to check and return work on a 4 hour cycle....maybe it is my setups????

[lawrywild]

AFAIK in BOINC you can go Advanced -> Reminder frequency and I think that changes how often you connect automatically

[vowkilmer]

Anyone else still down? I am using BIONC.

[[XC] itznfb]

i was up, now down again :(

[jspace]

Logs for me say It was up at 4:00 cst. The only system I really have to worry about running out of WU is my main rig, I have one doing einstein and climate prediction (for vault points of course;) ) and another helping with F@H. I geuss I only need to connect once so :shrug:

[[XC] gomeler]

awesome, the day after I leave Atlanta and BOINC goes down, thankfully 2 of the 3 machine have enough work for 5-6 days but my main rig was crushing those WU queues in 18-22 hours, hopefully the F@H core will stay busy.

[gr8golf]

mine seem to check and return work on a 4 hour cycle....maybe it is my setups????

If you go in and check that setting it will probably say .3 - that appears to be the default and seems about right. I've changed mine to 2.0 so that I can have more WU's on hand for these outages.

[sierra_bound]

i was up, now down again :(
Website is down too. They need to get their act together.

[meshmesh]

This is very bad timing. For all we know it may drag for days now. The sever balked in the first place. They reset it. That does not solve the problem. They still need to hunt for the cause. In the meantime it will keep balking on them and everytime they need to call the datacenter to get someone to go there and reboot it.

WCG have been very busy squeezing in a whole lot of work on the servers in the past two weeks, in an attempt to get it all done before the holiday (why?). Even this new project they launched a few days ago did not go through enough beta (if any), and they have problems with it from day one. Ok, somewhere along the line something got broken. They will fix it. But since today is already the 22nd, may I suggest you save your self the grief and don't waste your CPU cycles. Go do some FAH till things stabilize.

If you do have a lot of work in your queue, and worried about missing the deadline (I assume you know there is a 48 hour grace period) or you don't want to suspend boinc for a few days, may I recommend that at least you disable network communication (network not available in the advanced menu). Boinc won't keep trying to contact the servers and would not log those error messages. The last thing you want is for WCG to loose the results you returned in the middle of this mess. Just crunch the WU and hold the results back till the dust settles on this one.

As for the length of BOINC work queue: as stated above, the "connect to server every" preference setting (for the specific profile the host is assigned to) impacts the length of the work queue (for those who are interested, detailed info on boinc work-fetch policy can be found here (http://boinc.berkeley.edu/sched.php)). The default setting is 0.1 days. Boinc client makes sure that there is enough work for the CPU (including what it is actually crunching) for this amount of time. In this case (assuming 24/7 and 100% wcg project share), when the remaining time on the WUs being crunched plus the remaining work in the queue is less than 2.4 hours, it goes out and gets a single WU and adds it to the queue. Then it rechecks whether it needs more, and so on.

Given what happened, I will be increasing my setting from 0.5 days to 2.0 days. Recommend you do the same.

AFAIK in BOINC you can go Advanced -> Reminder frequency and I think that changes how often you connect automatically
There is a setting in the preference that forces BOINC to get permission every time it wants to connect to the servers. I think you can also specify a time where network access is available. This is there for people on dial up. If you have an always-on connection you want boinc to connect automatically as needed without asking. So no need to change those default settings. BTW, which client version are you using?

[Martijn]

There are some guys over at D2OL that might want your help if you ran out of WUs... :)

[littleowl]

Website is down too. They need to get their act together.


They just can't handle the extreme power of XS!!!! :fact:

[DAK1640]

C'Mon team...help us out at F@H...:toast:

[[XC] mysticmerlin]

site is up go change the stuff quick!

[meshmesh]

site is up go change the stuff quick!
:)
Go to preferences and change the "contact server every" to 2.0 days in the device profile specific to this host (if not using default). And don't forget to save. In boinc manager hit the "update" button.

[lawrywild]

New message now.. :confused:

22/12/2006 12:52:50|World Community Grid|Scheduler request failed: HTTP internal server error
22/12/2006 12:52:50|World Community Grid|Deferring scheduler requests for 2 hours, 54 minutes and 2 seconds

[[XC] MarioMaster]

i wondered why my cpu wasn't at 100% damn must've been a mess down there

[[XC] ruane]

All my computers have run out of work units, even the Pentium !!! has finished its work units that usually take 15 hours to complete! Any more News? I've put the Athlon on Folding@Home for a bit until the servers come back up!

Ruan

[njkid32]

This is really bad... Makes me wonder how competent the WCG management is...

[sierra_bound]

They have no idea when it will be back up.

http://www.worldcommunitygrid.org/forums/wcg/viewthread?thread=10703

"As of the time of this post, the BOINC server remains down. I can't provide a good estimate of when it will be back up."
This is really bad... Makes me wonder how competent the WCG management is...
It shouldn't take this long to resolve a server issue. I've switched one machine to D2OL. May switch more later.

[meshmesh]

I see that knreed and nelsoc are both on it. But coming up on 45 hours since this first started (10:30 PM 21 Dec - UTC) means they don't know what the problem is. My guess, no IBM Xmas party this year. This never happened before AFAIK. WCG has been very reliable. Yes, it would go down for a few hours here and there, but no more, and in most cases it is intentional by them. Hope they can fix it soon somehow.

Anyway, maybe it is that other project race good fortune. :)

[sierra_bound]

New message from WCG admin:

"We are working on a solution at this time. The worst case scenario is that the BOINC servers will be down until Tuesday, December 26, 2006. The best case scenario is that the servers will be up later today. We will know more at 6pm central time (USA). In the meantime, UD continues to work fine."

[gr8golf]

But coming up on 45 hours since this first started (10:30 PM 21 Dec - UTC) means they don't know what the problem is. My guess, no IBM Xmas party this year.

My guess is that this doesn't come on line until that first week in January. I doubt anyone is going to give up holiday time because it's down. Maybe I'll look into f@h and move everything over. What a pain in the @$$

[[XC] Lead Head]

My guess is that this doesn't come on line until that first week in January. I doubt anyone is going to give up holiday time because it's down. Maybe I'll look into f@h and move everything over. What a pain in the @$$

Thats team spirit! Move everything over to a different project, because a project is experiencing some of the first major downtime it has had.:mad:

[phelan1777]

aaaaaaaaaawwwwwwwwwwwww phewy!

[[XC] hipno650]

i hope it gets up soon cuz i want to do some stablity testing on a sempy system while i have it. could put up some nice numbers:D

[cold_realms]

Jumping onto F@H while I wait for WU's in WCG

[cold_realms]

Anyone know how to get F@H to use 2 cores within windows?

EDIT: Never mind found it, damn console versions :)

[[XC] itznfb]

well.... i'm completely out of work units.

[sierra_bound]

Update from WCG admin:

"The servers have been repaired. The problem turned out to be a faulty hard-drive which has been replaced. Now comes the hard part of rebuilding the hard-drive. This is going to be very time consuming. Early estimates are that it will take from 8 to 72 hours because much of the data has to come from a tape backup system. We will provide an update at least once each day. Once again, this only affects BOINC users. We apologize for this inconvenience."

[Movieman]

After reading all above: Perfect time to help the F@H guys..I'm not asking that anyone with a huge farm do this. I understand the work involved..
maybe if everyone can toss 3-5 PC's on Folding till the 26th..
Just my thinking, as always, feel free to ignore this old fool!:D

[sierra_bound]

I would, except I hate F@H. Long story about why. I realize our F@H team is involved in a battle with another team. But I never viewed my folding as contributing to a competition. I'll probably switch a few more machines to D2OL.

[Fr3ak]

Because of not working WCG, I tried to change all worldommunitygrid.com xlms to .org ones, because it kept telling me I am using the wrong url.
That resulted in the lost of work of more than a day.

After that I thought, I can test another Sempron I have laying around here, so I wanted to swap the CPUs.
After getting the naked Sempron out, I noticed that some parts of the die were glued on the base of the heatsink.
That liquid metal stuff gets hard as metal when it gets hot. I had to lap the heatsink to get rid of it. Metal polish alone didnt help. The CPU is dead for sure, I trested it again, but its not working anymore.
Too bad, because I wanted to run it on dice in january to get a even higher OC than 100% which it was able on air.
Now to make it even worse. The 20 weeks newer Sempron clocks like crap and I cant even run 2666mhz at 1.65V. With the old one I was able to run 2800Mhz at 1.55V WCG stable.

I am kind of pissed of and think I will take a short brake.
I am thinking about locking all my S754 stuff up, sledgehammer the new Sempron and get something new.
But as it will be one of my hopefully few mainboard swaps in 07 , I dont know what to get. C2D and AM2 requires DDR2 Ram I cant affort + there are no good budget CPUs out. Maybe the X2 3800+EE which can be had for 100euro. Too bad current boards are no HTT3 boards and K8L ready, otherwise I think I would make the switch.

I never killed a CPU before and I dont think it was me not being carefull enough when I took the HSF off ;/

I wont be working on my Linux install either right now, because I cant compile anything with a unstable CPU. I will try to find stable settings to do that, but I am not happy overclocking this POS right now ;(

[an0nym0us]

My guess is that this doesn't come on line until that first week in January. I doubt anyone is going to give up holiday time because it's down. Maybe I'll look into f@h and move everything over. What a pain in the @$$
tell me about it man... i've got 30 REMOTE machines if i have to switch projects i'm gonna be PISSED. i joined WCG for the great guys and camaraderie here i busted my a$$ to get these machines 24/7 100% crunching for WCG. at least i hit 1m points before the servers went down. meh, i thought i had two more rigs to add to my acct tonight but this sh*t is just aggravating.

[gr8golf]

Thats team spirit! Move everything over to a different project, because a project is experiencing some of the first major downtime it has had.:mad:

Easy there big fella! I've been getting PM's and lots of forum requests to help out with the F@H race, and have not moved anything over because I AM thinking this is the project to be focused on. It also seems like switching a bunch of workstations over is a lot of hassle if the WCG system is about come online again, but if they are going to be offline for a long time I may as well be working on the ultimate goal - defeating disease. RIGHT? :slap:

Besides - since when does someone that has 150K points get to take someone that has 4,000,000 out behind the woodshed? :fact:

[[XC] hipno650]

i feel your pain man. most of my rigs are not on location so only the local rigs are going to f@h. i would encourage people to go to f@h ONLY for a shot period of time we can't have people staying there and not coming back when they are done. this is also the first problem we have had i with wcg and it is explainable all of boinc is down because of one HDD failing we all know what thats like and trying to bring the files back. it's hard and takes time. lets cut them some slack here then go back on the offensive once things are back online.

[smopoim86]

Mine still isn't working. Keeps saying
12/23/2006 1:07:28 PM|World Community Grid|Scheduler request failed: HTTP file not found

[lawrywild]

I get the same message, about 3 hours until I run out..

[mike047]

I get the same message, about 3 hours until I run out..


There are a lot of other "BOINC" projects you can attach to very easily;
"temporarily"[edit]
Take your pick here;

http://boinc.berkeley.edu/projects.php

I started some Tanpaku, it will be good for our "vault" standings also.

Or FAH if you can install.

[njkid32]

I am only on F@H for a week or so just to help out. I'll be back on WCG soon enough. I guess it came at a good time cuz if I wasnt running F@H I wouldnt be doing anything...

[Martijn]

I am only on F@H for a week or so just to help out. I'll be back on WCG soon enough. I guess it came at a good time cuz if I wasnt running F@H I wouldnt be doing anything...
Yes, this is indeed a 'good' thing for the F@H team :D

[STEvil]

Because of not working WCG, I tried to change all worldommunitygrid.com xlms to .org ones, because it kept telling me I am using the wrong url.
That resulted in the lost of work of more than a day.

After that I thought, I can test another Sempron I have laying around here, so I wanted to swap the CPUs.
After getting the naked Sempron out, I noticed that some parts of the die were glued on the base of the heatsink.
That liquid metal stuff gets hard as metal when it gets hot. I had to lap the heatsink to get rid of it. Metal polish alone didnt help. The CPU is dead for sure, I trested it again, but its not working anymore.
Too bad, because I wanted to run it on dice in january to get a even higher OC than 100% which it was able on air.
Now to make it even worse. The 20 weeks newer Sempron clocks like crap and I cant even run 2666mhz at 1.65V. With the old one I was able to run 2800Mhz at 1.55V WCG stable.

I am kind of pissed of and think I will take a short brake.
I am thinking about locking all my S754 stuff up, sledgehammer the new Sempron and get something new.
But as it will be one of my hopefully few mainboard swaps in 07 , I dont know what to get. C2D and AM2 requires DDR2 Ram I cant affort + there are no good budget CPUs out. Maybe the X2 3800+EE which can be had for 100euro. Too bad current boards are no HTT3 boards and K8L ready, otherwise I think I would make the switch.

I never killed a CPU before and I dont think it was me not being carefull enough when I took the HSF off ;/

I wont be working on my Linux install either right now, because I cant compile anything with a unstable CPU. I will try to find stable settings to do that, but I am not happy overclocking this POS right now ;(

The SMD capacitors and resistor packs came off? Pics per chance? The cpu can probably be repaired with the proper parts and some work...

[sierra_bound]

Update from WCG admin:

"We just got the BOINC servers up and running and you should start seeing normal operation patterns at this time, Knreed is doing a few more tests and will be posting another update later. Happy Holidays!!"

From knreed:
"There were stats accumulated by users on BOINC that were not correctly imported into the website database. In a few minutes we are going to re-run the stats since the outage to correctly import these stats."

[[XC] hipno650]

:woot: :woot: :woot: :woot: online!!!! finaly my slowest rigs and off site rigs can go to work again!

[[XC] leviathan18]

uploading all ther work downloading more

[[XC] hipno650]

ya i took a major wu dump!!! and that was only the local comps as well. but i won't be back to fun power untill the f@h race is done. then i can put it back on my p4 2.4ghz 4200x2 at 2.6ghz my main rig and the 1.9ghz barton im testing for a friend.

[sierra_bound]

Latest message:

"The stats have been imported correctly now and all data is up to date.

Over the next couple days BOINC will likely send out 4th copy of workunits due to workunits that timed out during the outage. Trying to prevent this behavior would be tricky and the likelyhood of introducing a new problem would be high enough that the risk is not worth it. As a result we are going to just have to let the 4th copy go out. This behavior will be largerly limited to the next 24 hours.

We have extended the grace period for how long you are sure to get credit for work returned after the deadline. Anybody returning late work over the next few days will be able to get credit for the work they have done.

Thank you all for your patience and we apprecate your commitment to the science behind World Community Grid."

[[XC] Jaco]

all right :cool:

[lawrywild]

lol I changed it to contact every 4.0 days and now have this huuuuuuge list of work :D

[cupra]

aua, now my servers can rest a little bit :bananal:

[Martijn]

Imagine how many points we have coming days if we dump our results :eek:

[lawrywild]

yeh lol I got 1700 boinc pts so far today and have 2 pages of pending validation results :)

[Rinsewand]

ahh nice to see they care! My conroe is on F@H till one of the teams hits 10.5mil, and then everythings back on here again!

RwD

[Bootup05]

:eek: Not again!

The "Project is down" message is back:(

EDIT:Wow, its back already:D :) :p:

[meshmesh]

:eek: Not again!

The "Project is down" message is back:(

EDIT:Wow, its back already:D :) :p:

If you notice that you found out this on Wednesday at 10:16 AM UTC.

This happens every week and it is not the only period this happens in.
But we know that it is one of those normal scheduled event that should take a small amount of time. BOINC will give error communication messages and will not be able to upload results or get new work during these events. Just in case someone does not know:

Database is offline everyday for results/stats updates. Takes up to an hour, starting on 00:06 UTC, 06:06 UTC, 12:06 UTC, 18:06 UTC

The whole system is brought down twice a week on sunday and wednesday for a full database backup (thank heavens for that) and other system maintenance. Takes up to two hours starting on 09:00 UTC

So depending on your time zone, expect these events to occur regularly. Just make sure your preference settings keeps at least half a day of work in the task queue. After the holiday incident, mine is now set for 2 days.

[sierra_bound]

From WCG admin, in case anyone was wondering.:)

"Starting Jan 10, the statistics updates will be run only at 0:00 UTC and 12:00 UTC. This is a temporary reduction in the update frequency and should only last a few days."