We got a billionaire more......OldChap...:up:
And he is chasing Hixies and Turtles....what have we done to you :shrug:;):D
Printable View
We got a billionaire more......OldChap...:up:
And he is chasing Hixies and Turtles....what have we done to you :shrug:;):D
:up: Oldchap.. Nice going!:clap:
Congrats :up:
OT..
Looking at numbers today.
You know if weall got together on one day we could take the #1 position for the dailies on the team scores.
Just have to cordinate whenand it's VERY doable.:D
Quote:
Originally Posted by Movieman
I agree! I think the final tally was almost 45million? :eek: Not a bad day :D
Let's pick a day for a 24 hour marathon that's good for everybody :) :hehe:
I'm in....I have about 5 mill to give in a day.....yeah I know, I'm not giving what I could, but I'm a Turtle, and that's an obligation which I take very serious. :D
Well, as said.....name a day, and I'll find my wheels... I know they are somewhere in my shell. :D
Thanks all,
Now that I have this on 1 rig in 2 vm's things should steady down to about 9-10mill/day and not interfere too much with normal browsing.......
That said .....Would take about a week to stockpile a couple of extra machines for the purposes of a top score for one day ;)
congats OC!
we could do the marathon but my current firepower es close to null
It doesn't sound as it's going to be that esay, and we have to set a date some weeks ahead.
So how about the 1st next month. Starting when the statspage over at alex's site turns to the 1st of october.
ok, so 7PM on sept 31st( East coast time) we hit them heavy..:up:
I am up for it :hehe: :DQuote:
Originally Posted by Movieman
We should try to get a hold of moddolicous and see if he can drop 5mil down for us if only for a day :)
1st October it is then :up: Won't be much fun surfing the net that day though ;)
I can only offer little.. Borging my high school's computers is hard as hell.
I'll PM him..Quote:
Originally Posted by DeadlyFire
I had that..MJ12 on 4 machines here and 2 mins to open a web page:rofl:Quote:
Originally Posted by OldChap
Bring what you can,it all adds up!:up:Quote:
Originally Posted by lkiller123
Just a reminder to have your reserved buckets in order, so that you won't run dry, IF the server should run dry......
Just a heads up : )
I have mine all maxxed.. I leeaned the hard way..Quote:
Originally Posted by Frisch
Poor Alex,he works so hard and HATES when he runs out!:rofl:
http://img837.imageshack.us/img837/9317/bko.jpg:DQuote:
Originally Posted by Movieman
ROTF very good Frisch!
seeing the situation right now this is a good point! :up: I only had 300 reserved per node...bumping up to 500 :DQuote:
Originally Posted by Frisch
Mr. Movieman, Dave himself, hitting the double digits(that's billions kids! :cool:)
Quote:
Nickname: Movieman
URLs crawled:10,001,536,066
Data (MB)*: 261,248,240
Mighty congrats Dave!! :clap: :up: :up: :up:
I'm just glad he's on our Team...:DQuote:
Originally Posted by DeadlyFire
Congrats, Dave.
That is a rather big number.....Great Work Dave :up:
When the project was young, no one imagined that individuals would hit those numbers. It was a rush just when the WHOLE project reached 10 Billion...
Just a perspective on things....
Great run, Dave.
Thanks to all..
Been chasing that number for a VERY long time!:rofl:
He's almost caught himself a turtle :eek: Run frisch run! :DQuote:
Originally Posted by Frisch
I myself have just passed the 5 billion mark :toast2: on to the next 5! :D
http://i.imgur.com/g1vs7.png
Gratz OC and DF! I havent been able to get my rig working again, plus hefty power bill made me be a little more considerate with what I use :(Quote:
Originally Posted by DeadlyFire
Building a new Phenom II X6 rig over the next few weeks (sigged, ha), so hopefully get my crunch box updated with the E8400 and WCG/MJ12 running 24/7 ish :)
You know I can't run....you're hurting my turtle feelings :DQuote:
Originally Posted by DeadlyFire
Gratz Deadly...it's out of this world..:up:
Welcome back mekoa, there's always room for another net "cruncher" :)Quote:
Originally Posted by Mekoa
See you when you have things up and running. :up:
Thanks Frisch. Still chasing that 1bill marker!
Hopefully get a chance tonight or tomorrow to dismantle my old rig and see whats wrong!
Now how the hell did I miss this?Quote:
Originally Posted by DeadlyFire
Congrats on the FIVE Billion!:clap:
Thanks all :) Come back soon Mekoa! :up:
Good one Deadly,
You see those flippers going 15 to the dozen :)
Maybe next year I can try to get after those two greedy pie eaters ....wonders how long I have to wait for a faster connection???
Turtle 1,454 mill
OC 1,452 mill
http://img257.imageshack.us/img257/7953/turqm.jpg
:p:
..Don't make too much noise now, a hare by the name of Hixie is still snoozing :D
Speaking of snoozing, parts arrived today and Phenom is up and running.. all be it me screwing up with forgetting TRUE's mount horizontally as standard and having to use the AMD stock cooler (bleah). Awaiting BTK from USA and TRUE Black goes on!Quote:
Originally Posted by DeadlyFire
This does mean my project for this week is to rebuild my crunch box with my E8400!
Will need to check power bill after a bit but hopefully I can keep it on again (should be more efficient than an X2 6000?).
Maybe this update....Midnight here...had 2 hours without power today so the turtle had to sweat just a little longer :DQuote:
Originally Posted by Frisch
Good push there my friend
Turtle 1,458 mill
OC 1,457 mill
http://img190.imageshack.us/img190/609/57481742.jpg
A little forth and back here, but my tinnitus is killing me after keeping it up and running 24, so I will let you slide by ;):D............the metalic noise from the HDD is killing my ears.....:(
BUT....some hours work still to come, so WHEN......is the question...:D
OC 1,462 Mill
Turtle 1,460 Mill
http://img513.imageshack.us/img513/9005/dxfbd.png
:D:up:
Hey, go put your feet up Mr turtle...you deserve a sit down after that effort :up:
Now then, when this supercomputer week is over I must try to go back to running two instances of this but this time on separate drives one native with 2500aaks and one vm with samsung f3 1TB the machine is a x3350 @3.6
Here's a sight for sore eyes! :D Nice to see XS on top even without a breakneck speed marathon :)
http://imgur.com/Z9rYO.png
:cheer::cheer::cheer::cheer::cheer::cheer::cheer:Quote:
Originally Posted by DeadlyFire
:D
Refic is trying to change it by last update though...still in front @ 10pm and only one update to go ....good work everyone
Not doing marathon but I am currently using some firepower I have at hand. I could use some extra firepower but it would strain this way too much and I might get scolded by clients and by provider.
There you go
http://www.majestic12.co.uk/projects...fo.php?id=2532
EDIT: I am adding a 1 mbit line now :P
We led to the end :cool: :DQuote:
Originally Posted by OldChap
http://imgur.com/rTav0.png
Oh yea!!! :up::up::up:
Looks like we're on top again today(as of 10am EST) :D
Anyone tried the beta yet? maybe the others are and it is helping us :)
You might be right. The new loading server is serving up buckets that aren't spectacular in quality. The old 'hand-fed' buckets server that 1.6.12 nodes still use have 85%+ success. Alex mentioned the new node will have better quality buckets once recrawl of bad data is finished(couple weeks tops?). On the bright side, no more dreaded "NoMoreURLsAvailable" messages :DQuote:
Originally Posted by OldChap
Turned on my MJ12, and had a look at the stats....
Well, there's us, and there's refic.........122 mill !!! ...
155mill total for the day :eek::DQuote:
Originally Posted by Frisch
Big milestone:
Name: XtremeSystems
URLs crawled: 30,011,997,950
Data (MB)*: 766,566,997
Congratulations to everyone who helped us get here!! :):cheer::cheer::party::cheer::cheer:
oh come on I can't be the only one impressed :p:
:eleph::eleph::party::party::bounces::bounces: :bday2::party2::banana::banana2::toast::toast: :YIPPIE::YIPPIE::cheer::cheer::cheer::rofl:Quote:
Originally Posted by DeadlyFire
"I'm not impressed, it's just showing how strong we are....others should be impressed, and make this thread glow of :eek2:"Quote:
Originally Posted by DeadlyFire
:D
It's alright I guess...
Yay! missed this completely ...WELL DONE TEAM :D :up:
There we go :D
we need to catch up refic at least... he has reached the PB
LOL! little late to reply to that but better than never.Quote:
Originally Posted by DeadlyFire
This hare isn't snoozing, it's crawling as fast as it can. I know it's not much, and my monthly output is only a fraction of what i used to output in an hour, but that's all i can do without my 100mbps line :(
And if it regains its speed......http://www.karin-lisbeth.dk/images/e/elmer/001.jpgQuote:
Originally Posted by hixie
:D
Either way it's still good to have you back back Hixie :up:
We'll take whatever you can add and say thanks..:up:Quote:
Originally Posted by hixie
I've been crawling for quite a while ... just not large figures, and been too busy to post here.
BTW dave, why aren't you going to vegas? I'll be there with a booth this year.
Finally have conquered a quarter of the pie http://i.imgur.com/jy5qi.gif Tasty URLs! :D
http://i.min.us/ibHDzQ.png
1.7 final node released:
You can increase max reserve buckets from 1 to 3 for more sustained crawling :up:Quote:
v1.7.0 5/02/11
+ Support for new generation of central server in parallel with current
! Better handling of redirects
! Better control of domain counts during crawling
! Improved analysis of crawl errors
! Fixed rare issue with empty indexed data written incorrectly
! Mono - support for alternative spawning of archiver, .NET 2.0 build is now the only one available
! Mono builds - Less junk in log on communication errors
! Mono - better logic for handling multiple crawlers on same box
! Added MaxPriorityBuckets parameters to options
! Max reserved (pre-cache) buckets is forced to 1 in order to enable more efficient crawling on whole distributed network
! New SQLite build used (once run database won't be backwards compatible with 1.6.x series)
! Bundled 64-bit SQLite build with Mono distributions
! Mono builds now support https protocol crawling
! Reduced number of messages printed by default (can still be shown if Warnings mode is on)
! Put a limit on barrel archiving to avoid create too many temporary files in odd data sets
Well done dude! keep it upQuote:
Originally Posted by DeadlyFire
I'm trying :DQuote:
Originally Posted by Duh
btw..where has that 30mb goodness from Dave gone to?? :shrug:
I expect he is in the same boat as the rest of us...apart from being away just now.... waiting for the new server to be up in order to redo settings to maximise output. I see we can start in on this tomorrow ...he said hopefully.
Well done on the numbers by the way
Two issues:Quote:
Originally Posted by DeadlyFire
First is I haven't been able to get more than 6mbit total from 3 machines since Alex made those changes.
Second is I have one machine down.
Swapped cpu's that had been in that machine and fine and now won't post..
Clueless as to why and that was my main machine..:shrug:
Dave : use a linux box ( ubuntu). Installation takes 5 mins at most with no previous knowledge.Quote:
Originally Posted by Movieman
Also: refic has made a script to install mono ( the toughest part of linux install).
We can guide you and you can call DF who knows it quite a bit :P.
The setup of the ubntu box ( of mj12) could take you 30-40 mins) and you can use all the 30 mbit from a single quad core ( old q6000 would doing it with no sweat)
A short answer: No!:rofl:Quote:
Originally Posted by Duh
I'm a windows guy and no linux in this house!:p:
Dave, under the 'More Crawler' tab in options, try setting 'Maximum deep crawl buckets' to 0 and 'Maximum priority buckets' to 50(or even 100 if 50 works ok). Also, Alex just released the 1.7 final version which lets you raise your reserved buckets from 1 to 3, which should use up more of your connection while crawling so upgrading should give you a boost.
regarding linux, it's very easy to install and run BUT it's a bit of a pain in the a$$ because mono is less stable than .NET under Windows so you constantly have to watch for bugs(not fun).
http://i.imgur.com/L7WRY.png
Congrats on 11 Billion Dave! :up::up:
I got hit by a nasty concoction of viruses and malware last week, just finished painstakingly reinstalling Windows and everything else(though I have made a clone image of my HD and am going to do monthly backups in case something goes wrong again! :)).
sh.. DF.. I always have such sh.. I wonder why alex would not include an AV+malware. .. he could even sell a list of webs containing that sh..
It was my first virus in a year or so, not a bad ratio :D good news is the infection I got wasn't related at all to MJ12 :up: Alex has said before anything crawled is never run or executed so nothing to worry about :)Quote:
Originally Posted by Duh
Quote:
Originally Posted by DeadlyFire
i know its not executed but if he includes an AV+malware he could detect which website has sh.. inside and sell such to AV companies ;)
Congrats on 2 Billion OldChap!! :clap::up:
Cheers, It did seem that it would never happen during December-January but I got there in the end even though I now have to stop during the day on my main "unlimited" line. apparently unlimited to them means if you dl more than 8Gigs a day between 9am and 9pm they will terminate the connection.
Big number, I'll try and follow your grand example, and reach 2 billion.......in Turtle tempo : ) But when I'm there you're long gone, so I'll shout it out loud with the wind, so you can hear me up there in front..... : )
Congrats !
Gratz all! Missed a few milestones it seems.
Shame I still cant get my weird error issues fixed, boo :( Crunching WCG 24/7 at the mo on 6 cores though. Gotta love AMD for hex!
Will grab this build over the next week to see if the errors have disappeared, unlikely though. Bar that, must be my ISP blocking traffic or something :(
Hi Mekoa :) It's been really quiet around here, welcome back :up:
When did you start having those error issues? Was it after the big beta node update around December?
Hey Deadly :DQuote:
Originally Posted by DeadlyFire
Its the below thread, mostly started after I upgraded to Windows 7. Have no issues with internet using anything else, just MJ12 (same ISP as before also).
http://www.xtremesystems.org/forums/...d.php?t=261795
I've just signed up :D
And now have a question. I'm on a 4096/512 line and after setting M12 to use 100% of the line I have done 21,500 after 3h15m which seems low. I've yet to see the download rate pass around 650 and it usually hovers around 200 to 300, often dipping down to under 50 (last edit, I've just seen download as 9, 0, 83, 14). What's up with that?
EDIT: In those three and a bot hours I've used 250MB down/15MB up. I KNOW that I can pull around 1.4GB/hour over HTTP/FTP and upload around 150MB/hour. Nothing else is using the network other than a bit of browsing (no flash/very few images).
Hey oj, tell us what you chose for crawler settings.
I put my laptop to work on a 8meg/0.7meg line with 100 workers and 35 buckets a few days ago and it seems to run ok but took 6 days for 9mill
On my 50.1.75 line I rarely see more than 25kbits sec this one seems to run at maybe 2
After the next time the NIC goes down, go into control panel and open the 'network and sharing' button. Then on the left side 'change adapter settings' and then find your NIC in the next window. Right-click on it and click 'status' and then the 'details' button. Take note of the IPv4 address before and after the crash and the lease obtain/expire time. It might show no change at all but it's worth a try to see what happens.Quote:
Originally Posted by Mekoa
What OC said ^^ What operating system are you running? For a 4mbit line I'd suggested 100 max # workers and 20-50 max open buckets(start low and keep raising until you're satisfied with the speed). The general idea is keep a low amount of buckets open but a higher number of workers. Also, under the 'More crawler' tab within options, I would set maximum priority buckets to something low like 5 or 10, if you don't have a beastly router those kinds of buckets can cause the router to hang. Be sure to check if your ISP allows unlimited or large amounts of downloading!!Quote:
Originally Posted by [XC] Oj101
Happy crawling and thanks for joining! :up::welcome:
One HUGE milestone for one of our smaller yet more determined crawlers :up: 100mil may not seem that big for some of us but for someone who chugs along at 100k urls/day it's a helluva milestone
http://i.imgur.com/u7Bby.png
100,028,744
Congrats eme! :clap::clap:
:up: congratulations :up:
DeadlyFire : you are very right. Congratz to eme ! Very well done man!
I followed the Majestic12 guide (the sticky) to the T, as while all options are explained a lot are still over my head :D
I'm running a Billion BiPAC 7300GA which can handle several tens of thousands of seeds are leeches connected in uTorrent so these packets shouldn't be a concern. Current router uptime is several days so it's not causing issues as far as I can tell :)
Good to hear :up: If you're willing to search, there's tons of tips and advice scattered all over this subforum, just finding it is a pain in the a$$ :rofl: If you have any problems with your node or router shout it out here. Thanks again for joining :)Quote:
Originally Posted by [XC] Oj101
A good day for us yesterday, 40mil/over 1TB crawled and second place :clap: :
http://i.imgur.com/nokDc.png
Good work guys :up:
I'll be glad when I can look at my output and it stays constant again though....Just now one day can be 50% more than another just now
Quote:
Originally Posted by OldChap
we will solve that ;)
I am chasing you now, you better watch your six :p
I had a weekend of discovery.....
My original setup of running 2 nodes at 350/100 to max out my connection has still been going at it quite well but with the current mix of buckets I find my connection (at least the upload which limits me) to be only partially used and the download was sometimes high sometimes low. As an interim measure I had increased buckets to 125 for slightly better performance.
I have now added a third node at 300/100 and the upload is now balls to the wall whilst the download is staying pretty constant. First full day yesterday and much better numbers.
I will report back here if my percentages suffer but overnight at least they are still in the 80's
Congrats elvis on hitting the Billionaires club!! :clap::clap::clap:
http://i.imgur.com/61sR9.png
1,000,733,396 to be precise :up:
:woot: http://www.dogproductshop.co.uk/smil.../party0019.gif http://www.dogproductshop.co.uk/smil.../party0018.gif A bit of a push going on lately....Congrats.
Quote:
Originally Posted by DeadlyFire
:blushes: many thanks DF :)
shoota and shootas mom have joined into the team ;)
WHAT THE F:::::.
http://i.imgur.com/t2UwO.png
is it just me ??
Just today i run out of ritalin so I might be just me ( not joking)
April Fool's joke :rofl: I almost spat out my coffee when I saw 3 Moviemans in this thread alone :ROTF:Quote:
Originally Posted by Duh