Majestic 12 Milestones.

[Frisch]

This Thread should have been made a long time ago...but better late than never.

This Thread is for milestones, set in Majestic 12.

movieman (Dave) made it to 2,500,000,000 today :shocked:

DeadlyFire hits 500,000,000 today :clap:

Congrats to both. :up:

Today they are kings. Tomorrow you can harass them again. :D

[Movieman]

Thank you..
And Deadly Fire hit 500 mil a lot faster than I did!:up:
I had some issues with the clover and have been testing it without MJ12 on it the past 2 days so my numbers are off a bit.. Just the quad and the FX51 on MJ12 right now.

[Frisch]

Thank you..
And Deadly Fire hit 500 mil a lot faster than I did!:up:
I had some issues with the clover and have been testing it without MJ12 on it the past 2 days so my numbers are off a bit.. Just the quad and the FX51 on MJ12 right now.

When you are a bit off.....you still spank the most of us......:cool:


PS

God damn, Dave....you sigged a Turtle....a proud Turtle can't walk straight, it will bump into everything, when its head is in the sky. ;) :D

[[XC] riptide]

A fine sig it is indeed.

[Movieman]

riptide;2948154']A fine sig it is indeed.

It grabbed my heart. The two days my daughters were born were the best in my life and that line says it better than I can.

[DeadlyFire]

Great thread idea Frish! :up: Finally got #5 overall on XS!!! :D :D #4 is mine in 7.5 days and then the long march on the 'Big 3' :stick: :p:



Thank you..
And Deadly Fire hit 500 mil a lot faster than I did!:up:
I had some issues with the clover and have been testing it without MJ12 on it the past 2 days so my numbers are off a bit.. Just the quad and the FX51 on MJ12 right now.

I still remember this thread (http://www.xtremesystems.org/forums/showthread.php?t=167129)like it's yesterday http://img339.imageshack.us/img339/9507/touchedbr8.gif

[Mekoa]

Gratz to both! Almighty milestones to hit :)

[Mekoa]

Mekoa;2948893']Gratz to both! Almighty milestones to hit :)

I hit 300mill last night, not a big mile stone but getting there :)

[DeadlyFire]

Mekoa;2952941']I hit 300mill last night, not a big mile stone but getting there :)

Nice Job! That's a 3rd of a billion! :D :clap: :up:


Also, I'd like to say thanks to Gamekiller, Jspace and Mrslinky who crawl 24/7 with us but don't receive as much attention, thanks alot guys!! :up:

[Mekoa]

Nice Job! That's a 3rd of a billion! :D :clap: :up:


Also, I'd like to say thanks to Gamekiller, Jspace and Mrslinky who crawl 24/7 with us but don't receive as much attention, thanks alot guys!! :up:

Big thumbs up to all who help out with MJ12! :clap:

Doesnt matter if you do 1000 URLs or 10'000'000. Every little helps :)

[Movieman]

Mekoa;2952941']I hit 300mill last night, not a big mile stone but getting there :)
ALL the milestones are big ones IMO and congrats on the big 300!:up:

Nice Job! That's a 3rd of a billion! :D :clap: :up:


Also, I'd like to say thanks to Gamekiller, Jspace and Mrslinky who crawl 24/7 with us but don't receive as much attention, thanks alot guys!! :up:

Agreed, it really is a team effort and thats really what matters!:clap:

BTW; Speaking of numbers, did anyone notice that deadly pumped almost 9 MILLION the other day!:shocked:

[Mekoa]

ALL the milestones are big ones IMO and congrats on the big 300!:up:


Agreed, it really is a team effort and thats really what matters!:clap:

BTW; Speaking of numbers, did anyone notice that deadly pumped almost 9 MILLION the other day!:shocked:

9 mill? Eep. GJ Deadly :D

[DeadlyFire]

BTW; Speaking of numbers, did anyone notice that deadly pumped almost 9 MILLION the other day!:shocked:



I've been doing a little experimenting with 'special' machines and multiple hard drive setups :D I've been able to produce a max 400k/hour. Now if I could play around some more and hit 500K/hour it would be pure awesome! :slobber: :woot:

[Movieman]

I've been doing a little experimenting with 'special' machines and multiple hard drive setups :D I've been able to produce a max 400k/hour. Now if I could play around some more and hit 500K/hour it would be pure awesome! :slobber: :woot:

So let us in on your secrets..:D

[DeadlyFire]

So let us in on your secrets..:D


Hehe, well I found having two hard drives and running two instances of MJ12 in two VM's on one hard drive and then running the original instance of MJ12 on a separate hard drive greatly reduced the freezing that occurred when all of sudden 2 or more instances of MJ12 started to archive. Before, I saw that if I ran 2 or more instances of MJ12 on one disk, the hard drive would go nuts trying to meet the demands of all 3 instances of MJ12 and would crash at times. I tried running up to 8 instances of MJ12 but found 5 machines is enough to saturate 20 Mbps(20-30 seconds to open a webpage=sweet spot :D) Gonna go for a 24 hour suicide run tomorrow to see what happens :hehe:

[Frisch]

Hehe, well I found having two hard drives and running two instances of MJ12 in two VM's on one hard drive and then running the original instance of MJ12 on a separate hard drive greatly reduced the freezing that occurred when all of sudden 2 or more instances of MJ12 started to archive. Before, I saw that if I ran 2 or more instances of MJ12 on one disk, the hard drive would go nuts trying to meet the demands of all 3 instances of MJ12 and would crash at times. I tried running up to 8 instances of MJ12 but found 5 machines is enough to saturate 20 Mbps(20-30 seconds to open a webpage=sweet spot :D) Gonna go for a 24 hour suicide run tomorrow to see what happens :hehe:

Does that mean, Turtle will keep his 3rd, if you succeed.......

Let me give you a cheer.....

GO KILL YOURSELF :cheer: :cheer: :cheer: :cheer: ;) :D

Mekoa, congrats on the 300 mill......:up:

PS
jspace..slinky, and gamekiller....334 mill together.....just to get a clear picture of how much it all adds up to. :up:

[DeadlyFire]

Does that mean, Turtle will keep his 3rd, if you succeed.......

Let me give you a cheer.....

GO KILL YOURSELF :cheer: :cheer: :cheer: :cheer: ;) :D



I see :hm: :D If I can hit 9-10 mil on the 24 hour run I'll be more than happy :wasntme: I'm surprised that we're hitting 20mil even with Hixie awol for a while :eek:

[Movieman]

I see :hm: :D If I can hit 9-10 mil on the 24 hour run I'll be more than happy :wasntme: I'm surprised that we're hitting 20mil even with Hixie awol for a while :eek:

That's 'TEAM" PAL..all of us together can make it happen even with one player away.:up:

[Mekoa]

That's 'TEAM" PAL..all of us together can make it happen even with one player away.:up:

Wait till hes back (couple of days, wee) I smell 30 mill \o/

[Frisch]

Just to get on the list. Turtle made 800 mill, with one blue eye from the Hixie Hare

http://www.nategreenwall.com/images/turtleandhare2.jpg

[Mekoa]

Just to get on the list. Turtle made 800 mill, with one blue eye from the Hixie Hare

Gratz on 800mill Mr Turtle :D

[DeadlyFire]

Ooh 800mil! Nice :D :clap: :clap:





edit: Arrghh!! I came to within ~50k of breaking my all time daily record today! :mad: I will hit that 10mil mark even if it kills me!!!! :D

[Frisch]

Congratulations, Hixie ....... 1,000,000,000

:cheer2:

[DeadlyFire]

1 BILLION! :eek::eek: Insane milestone, Congrats Hixie!! :clap::clap::clap::clap::clap:

[hixie]

Oh, did i? Wasn't paying attention :p: shop getting quite busy :(

[Mekoa]

2 Hixie 1,000,853,269 20,414,788

Hixie machine! GRATZ :D

[DeadlyFire]

Nickname: Deadly_Fire
URLs crawled: 1,001,620,670
Data (MB): 24,400,805

That's right, 1 billion by yours truly :woot: Now the run on Hixie and Dave :p:

[Mekoa]

Gratz mate :) Long time coming!

My machine fell over a short while ago and unfortunatly I dont have the money to replace some bits... also power bills here are going up a lot (I dont pay it but people who do complain lol).

Just started a new job as well.. maybe rebuilt the machine at some point :)

[Frisch]

Nickname: Deadly_Fire
URLs crawled: 1,001,620,670
Data (MB): 24,400,805

That's right, 1 billion by yours truly :woot: Now the run on Hixie and Dave :p:

Nothing to see, nothing but brag :rolleyes: :p: :D


:up:

[Frisch]

After we Turtles, have been on earth for more than 270 million years, new and bigger numbers needed to be added to our BRAG club.

We are proud to enter this Xtreme brag club, with 1 Billion, which by far exceeds our already 270 million brag years on earth.

Thank you, for reading this brag. :D

[DeadlyFire]

After we Turtles, have been on earth for more than 270 million years, new and bigger numbers needed to be added to our BRAG club.

We are proud to enter this Xtreme brag club, with 1 Billion, which by far exceeds our already 270 million brag years on earth.

Thank you, for reading this brag. :D


I'd say you've got 1,000,385,399 reasons to brag! :p:

One big mighty CONGRATS to you Frisch!!!! :up::up: That is a huge milestone!! :D

:clap: :clap: :clap: :clap: :clap: :clap: :clap: :clap: :clap: :clap: :clap: :clap: :clap: :clap: :clap: :clap: :clap: :clap:

[Movieman]

Nothing to see, nothing but brag :rolleyes: :p: :D


:up:

Congratulations my friend! Big number and huge milestone!:clap:

[Frisch]

I'd say you've got 1,000,385,399 reasons to brag! :p:

One big mighty CONGRATS to you Frisch!!!! :up::up: That is a huge milestone!! :D

:clap: :clap: :clap: :clap: :clap: :clap: :clap: :clap: :clap: :clap: :clap: :clap: :clap: :clap: :clap: :clap: :clap: :clap:


Congratulations my friend! Big number and huge milestone!:clap:

Thanks...champions !! :D

Brag on.

[Frisch]

Dave has now downloaded :

101,227,325 MB

That's 101 Tera or 101,000 Giga


Insane !!!!

Congratz !!!!

[DeadlyFire]

# Nick URLsdone Data(MB)
4 Movieman 6,001,226,691 140,986,518

Congrats Dave!!! :clap::clap::clap::up:

[Movieman]

Thanks guys and I meant to get over here to say congrats to frisch the other day and got sidetracked so Congratulations you turtle you!:D

The teams doing well the past few weeks..
kicking arse and taking names and watch the scores at the end of the day.
Making a run at PY222( just for the day..I'm not totally nuts!):rofl:

[Frisch]

The man is eating all the pie........and I'm sure he needs it with all that activity.....


Congrats Dave...it's insane...guess i have said it before, but you won't stop, will you...;):D

[DeadlyFire]

The man is eating all the pie........and I'm sure he needs it with all that activity.....


Congrats Dave...it's insane...guess i have said it before, but you won't stop, will you...;):D

You'll have to wrestle the CAT 5 cables from his cold dead hands!! :p::D

[Mekoa]

Gratz Dave!

6bill is one almighty milestone to hit :)

[Duh]

I really feel good for our MM efforts as well as the turtle :)

Id been having issues with my box ( high speed). At this moment I had signed up for another server/box and Id been flagged as fraudulent ( as I had signed while using a VPN which has NYcity IP as I work with it and when they tried to call home my mom answered and she could not recall the numbers of verification).

I emailed them and their response was so slow that I decided to post at a forum which I also like ( in another degree) and got flamed just as if I was adding pressure to the provider for posting that stuff in a public forum.


http://www.webhostingtalk.com/showthread.php?t=901689

elvis1 is my nick @WHT and the nick with which I am involved in MJ12 project ( and forum). you guys know me. our movieman can tell ( as he is an admin) that my IP changes from 190 to 67 lots of times. I was willing to donate the whole TB of BW as this box was meant as an extra failover point ( so I would not use it unless my main box goes down)

Sorry for not congrating as much as needed the old man ,his crazy friend Frisch ( the lil DSL maniac) and the rest of the team mates who did a bunch for the project

[DeadlyFire]

Together we hit 36mil yesterday, a 2-year high :clap: I'm up for breaking some records, anybody else? :D

[Duh]

Together we hit 36mil yesterday, a 2-year high :clap: I'm up for breaking some records, anybody else? :D

:cool::cool::cool::cool::cool: nice stuff for sure...

Apart from my lappie and vps box I recruited 2 friends ( 1 mbit and 3mbits ) :)

http://www.majestic12.co.uk/stats/allurls_by_day.php?w=800&h=400&id=2532

thats me :)

[DeadlyFire]

:cool::cool::cool::cool::cool: nice stuff for sure...

Apart from my lappie and vps box I recruited 2 friends ( 1 mbit and 3mbits ) :)

http://www.majestic12.co.uk/stats/allurls_by_day.php?w=800&h=400&id=2532

thats me :)

good man :clap::up:

[Movieman]

Together we hit 36mil yesterday, a 2-year high :clap: I'm up for breaking some records, anybody else? :D

I just saw that Xs is actually leading free-dc today!:clap:
And that your kicking my ass sideways, backwards and upside down!:rofl:

Congrats pal!:clap:
Let's get everyone here motivated and make some numbers! :cheer:

[DeadlyFire]

I just saw that Xs is actually leading free-dc today!:clap:
And that your kicking my ass sideways, backwards and upside down!:rofl:

Congrats pal!:clap:
Let's get everyone here motivated and make some numbers! :cheer:

D@mn straight! It's nice seeing XS back in the top 3 daily! :party:

Right now I think I've met my daily match with Fugitif, he's 'only' got a 30mb line but matching me URL-wise :yawn: I've also noticed that with adding more clients it's necessary to lower the amount of workers per node, I did a suicide run with 5 machines with 450 workers each, couldn't last a half hour before the router told me to make like a tree and go **** myself :ROTF: 200 workers per node is working very nicely though with no router hangs whatsoever. Gonna experiment with adding more 'special' machines and see how that scales :up::up:

before I forget, skycrane and redrumy get your butts in gear, we need you guys! :D

[Frisch]

Together we hit 36mil yesterday, a 2-year high :clap: I'm up for breaking some records, anybody else? :D

The fact is, that we for the first time hit 1 tera in one day. The data return in the old records was less. Said in another way : Had this been back then, we had beaten the url record aswell.

:up:

[Duh]

I see our little space being very hot right now.. will strain my 5 bucks puppy a tad more ;)


edit: I have already pushed about 124 is less than 2 days according to my logs.. the problem with it is that I am hitting the limits of hardware and might piss of my provider if I eat so many resources :p
. so I prefer to have a long term solution rather than puching it's face and not seeing it again .

[Mekoa]

Good job all.

I still need to configure my main node, sure I can crank a bit more out of 5.7mbit.

Had to turn off my E8400 machine due to power requirements (running 1kw 24 hours a day strains the old bit power bill).

[skycrane]

deadly fire, ill have my 50/30 line installed on jan 7 :) and i promise to max it out.

the only other thing ill be doing is stomping on helheim in dimes :) or atleast trying to...i come close to that now on satelite.
and trying to keep 19 rigs afloat with some BOINC work :D

[Duh]

deadly fire, ill have my 50/30 line installed on jan 7 :) and i promise to max it out.

the only other thing ill be doing is stomping on helheim in dimes :) or atleast trying to...i come close to that now on satelite.
and trying to keep 19 rigs afloat with some BOINC work :D

lotsa power there.. 19 rigs is an enormous amount.. your connection is also impressive

[DeadlyFire]

deadly fire, ill have my 50/30 line installed on jan 7 :) and i promise to max it out.

the only other thing ill be doing is stomping on helheim in dimes :) or atleast trying to...i come close to that now on satelite.
and trying to keep 19 rigs afloat with some BOINC work :D

Let me know if you need any tips on tweaking MJ12 :) I've got 5 machines set up with 220 workers each, about 25-30 buckets per node and it's taking up 45-50mb of my line :up: Good luck with getting your new line in place! :)

[skycrane]

that sounds just where i want to be

[Duh]

merry xmas Mj pals :)

[DeadlyFire]

Merry Christmas all :toast:

[Frisch]

merry xmas Mj pals :)


Merry Christmas all :toast:

:yepp:

So this is Christmas
And what have you done?
Another year over
And a new one just begun

And so this is Christmas
I hope you have fun
The near and the dear one
The old and the young.

A very Merry Christmas
And a Happy New Year
Let's hope it's a good one
Without any fear

And so this is Christmas
For weak and for strong
For rich and the poor ones
The road is so long

And so Happy Christmas
For black and for white
For yellow and red ones
Let's stop all the fight.

A very Merry Christmas
And a Happy New Year
Let's hope it's a good one
Without any fear

So this is Christmas
And what have you done?
Another year over
And a new one just begun

And so Happy Christmas
I hope you have fun
The near and the dear one
The old and the young.

A very Merry Christmas
And a Happy New Year
Let's hope it's a good one
Without any fear

War is over!
If you want it
War is over!
War is over, Now!

[DeadlyFire]

:yepp:

...So this is Christmas...

I usually get tired of hearing Christmas songs on the radio but this song I will listen to over and over without getting tired of it :)

btw, looks like we're having a URL shortage, otherwise I must be dreaming because I'm not only in second place but ahead of refic as well :rofl:

[Frisch]

I usually get tired of hearing Christmas songs on the radio but this song I will listen to over and over without getting tired of it :)

btw, looks like we're having a URL shortage, otherwise I must be dreaming because I'm not only in second place but ahead of refic as well :rofl:

We are in shortage, so i'll fire my machine up for the whole day, as I have 500 in reserve....let's have some fun, while the others are out of the game :D

[DeadlyFire]

We are in shortage, so i'll fire my machine up for the whole day, as I have 500 in reserve....let's have some fun, while the others are out of the game :D

I'll drink to that!! :toast: :hehe:

[Frisch]

Yes...we are #1 at the moment....that's a looong time ago, we've been there.....damn I regret i lost 2 - 3 hours of the night.

But who cares....we're on top for the day..:D

[Frisch]

Final yesterday....evidence for something to remember : )

http://img46.imageshack.us/img46/7165/statk.png

[DeadlyFire]

Final yesterday....evidence for something to remember : )

http://img46.imageshack.us/img46/7165/statk.png

Was nice indeed :toast2:

[DeadlyFire]

# Nick URLs done Data (MB)

1 Movieman 7,005,096,980 170,591,373


Congrats Dave!! massive milestone :eek: :up::clap::clap:

[Duh]

# Nick URLs done Data (MB)

1 Movieman 7,005,096,980 170,591,373


Congrats Dave!! massive milestone :eek: :up::clap::clap:

is that 7 PB? :shocked::shocked::shocked:

I will get a 10Mbit box ASAP and do some nice numbers :D .. my long time challenge is our Dutch friend (M.. ) :p:

[Martijn]

is that 7 PB? :shocked::shocked::shocked:

I will get a 10Mbit box ASAP and do some nice numbers :D .. my long time challenge is our Dutch friend (M.. ) :p:

I think it's time to shell out the extra €10/month to double our speed to 60Mb then :wasntme:

[DeadlyFire]

I think it's time to shell out the extra €10/month to double our speed to 60Mb then :wasntme:

do it! :hehe: :D

[Frisch]

Martijn 500 mill :eleph: :up:

[DeadlyFire]

Congrats Martijn! :clap::clap:

[Duh]

I think it's time to shell out the extra €10/month to double our speed to 60Mb then :wasntme:

I can't believe I am so dumb :D. I 've anticipated my next move and challenged a Dutch dude who pays 10 bucks and has his BW doubled :shocked: :p::p::p:

I've already passed you in the day to day basis ;) :D . 10 Euros and you can double your speed, that is funny :ROTF::rofl: . What pissed me off is that you only have sammy on your dsl line and you are doing some nice numbers. I've got: lappie, friends ( 2) and 3 VPS. Will soon add some more firepower :). I even passed elhoim who has an unmetered 100 mbit box ;).

I am about to reach the 200 M but that does not cut it :)

[Martijn]

I can't believe I am so dumb :D. I 've anticipated my next move and challenged a Dutch dude who pays 10 bucks and has his BW doubled :shocked: :p::p::p:

I've already passed you in the day to day basis ;) :D . 10 Euros and you can double your speed, that is funny :ROTF::rofl: . What pissed me off is that you only have sammy on your dsl line and you are doing some nice numbers. I've got: lappie, friends ( 2) and 3 VPS. Will soon add some more firepower :). I even passed elhoim who has an unmetered 100 mbit box ;).

I am about to reach the 200 M but that does not cut it :)

I could easily double my output/day, but well... I like to also use my internet connection for eh, other legal purposes ;)

[Martijn]

Screw the downloads, I'm ramping up :hehe:

[Duh]

sh.. you are getitng closer.. you are 19 in the dailies and I am 10.I will hire an extra VPS ASAP. Just called the GF to fire up her connection :cool:

[Martijn]

sh.. you are getitng closer.. you are 19 in the dailies and I am 10.I will hire an extra VPS ASAP. Just called the GF to fire up her connection :cool:

It's a shame I'm not at university the next 2 weeks :(

[Martijn]

sh.. you are getitng closer.. you are 19 in the dailies and I am 10.I will hire an extra VPS ASAP. Just called the GF to fire up her connection :cool:

Hey, what's happening with you? Having trouble keeping up? :ROTF:

[Duh]

my VPS's are having issues :S . I will buy some sh ASAP and you will notice ( I will make sure you do :slash::slash:)

[Duh]

while our beloved friend Martijn is cheating (http://www.majestic12.co.uk/projects/dsearch/userinfo.php?id=1094)j/k ( I have peeped into his setup and even takes a rig to the university to pump out the precious BW of a Dutch University) I am planning to get some horse power and see how he can handle that ( he will have to study another career so he can steal BW from that University also :P)

[Martijn]

while our beloved friend Martijn is cheating (http://www.majestic12.co.uk/projects/dsearch/userinfo.php?id=1094)j/k ( I have peeped into his setup and even takes a rig to the university to pump out the precious BW of a Dutch University) I am planning to get some horse power and see how he can handle that ( he will have to study another career so he can steal BW from that University also :P)

It's going to be a lot easier for you I suppose, I am only running MJ12 at uni and on Sammy now. In the past week my download-queue has risen to 300GB :shocked:

If only I could run a rig at uni 24/7... 10000/10000 is just :slobber:

[Duh]

why not take a serious server there than :banana::banana::banana::banana::banana: err sammy.. a dual w3520 would not do that bad 10000/10000 is 10 Gbit pipe? I can get such infra (10 gbit) for something like 3.5 k u$.. :)

[Martijn]

The University computers are E6850s, they aren't too bad. However, since I don't live at university it's impossible for me to set up a seedbox. Should I ever go and live there I'll still have a 100/100 pipe, but it's shared with the rest of the building. The problem is just that I don't have money left to go there.

[DeadlyFire]

2,000,000,000 for me! :D :D :)




....however, I am having nightmares (http://www.majestic12.co.uk/projects/dsearch/userinfo.php?id=1149)recently :help: :sofa:

[hixie]

congrats mate :up: more than twice of what i have done, i should be ashamed :p:

I've got more fire power coming soon, deciding between a IPCOP or an Untangled machine. IPCOP seems to use less resources, but Untangled has some neat features like monitoring which machine in my network is hogging my machine's bandwidth, so i can go and smack that person on the head :slap:

[DeadlyFire]

congrats mate :up: more than twice of what i have done, i should be ashamed :p:

I've got more fire power coming soon, deciding between a IPCOP or an Untangled machine. IPCOP seems to use less resources, but Untangled has some neat features like monitoring which machine in my network is hogging my machine's bandwidth, so i can go and smack that person on the head :slap:

Thanks! :D

I thought about setting up an old machine as a linux router but I have no more spare parts laying around :doh: I'm thinking about getting one of those ultra min-pc's (http://www.newegg.com/Product/Product.aspx?Item=N82E16883103228) for ~$200. It's not very powerful but for internet routing purposes it's overkill(most retail home routers can't compare with only 16-128 megs of ram/4-32 megs of flash ram and maybe a 200-500 Mhz processor at most). I've read more than I'd like into the guts of routers and the most performance limiting factor seems to be the ram size and my router has 64mb :rolleyes: A mini with a 1.6ghz CPU and a gig of ram would more than satisfy all my nodes :D

[hixie]

firmware is actually the limiting factor, processing power and ram plays a huge role in theory but most of it is locked away by the firmware. Modified firmware helps quite a bit.

In my experience with Smoothwall & IPCOP, an old Dell P3 that i got paid to take away was waaaayyyy more than i needed (Dimes and MJ-12)
It was a P3 with 128MB RAM and i think a 20GB Quantum Fireball HDD

Back in the days i was running dimes, i had something like 30 clients smoothie didn't even break a sweat, on a normal router 15 clients would crash the router and 25 would crash the modem, had to install Smoothie and get my ISP to upgrade my modem.


EDIT: Just checked my old stats, i had 53 clients!

[DeadlyFire]

firmware is actually the limiting factor, processing power and ram plays a huge role in theory but most of it is locked away by the firmware. Modified firmware helps quite a bit.

In my experience with Smoothwall & IPCOP, an old Dell P3 that i got paid to take away was waaaayyyy more than i needed (Dimes and MJ-12)
It was a P3 with 128MB RAM and i think a 20GB Quantum Fireball HDD

Back in the days i was running dimes, i had something like 30 clients smoothie didn't even break a sweat, on a normal router 15 clients would crash the router and 25 would crash the modem, had to install Smoothie and get my ISP to upgrade my modem.


EDIT: Just checked my old stats, i had 53 clients!

Wow!! Thanks for the insights :eek: :up:

I assume you know your way pretty well around setting up smoothwall/M0n0wall/pfsense/ipcop? Any suggestions on which is the better of the bunch? I'm very eager to setup my own router but I have almost no experience in linux(unless it's a gui-based installation).

[hixie]

Wow!! Thanks for the insights :eek: :up:

I assume you know your way pretty well around setting up smoothwall/M0n0wall/pfsense/ipcop? Any suggestions on which is the better of the bunch? I'm very eager to setup my own router but I have almost no experience in linux(unless it's a gui-based installation).

I'm not very good with linux but setting up a smoothie or ipcop is so simple even dave could figure it out without a manual. They have a simple 16bit colour GUI for installation, after everything is installed you can unplug the monitor, keyboard and mouse. Accessing all router functions are just the same as a normal router via a browser with a nice GUI.

I've only used smoothie and IPCOP so i can't comment on the others but i do lean towards IPCOP a little bit more as i find it's easier to find add-ons, and it also has a 'blue zone' for wireless.

If you do setup an IPCOP box make sure you install and setup the advanced QoS add-on, this way normal web browsing won't be affected by MJ-12.

[DeadlyFire]

I'm not very good with linux but setting up a smoothie or ipcop is so simple even dave could figure it out without a manual. They have a simple 16bit colour GUI for installation, after everything is installed you can unplug the monitor, keyboard and mouse. Accessing all router functions are just the same as a normal router via a browser with a nice GUI.

I've only used smoothie and IPCOP so i can't comment on the others but i do lean towards IPCOP a little bit more as i find it's easier to find add-ons, and it also has a 'blue zone' for wireless.

If you do setup an IPCOP box make sure you install and setup the advanced QoS add-on, this way normal web browsing won't be affected by MJ-12.

I did a little googling and it looks very easy like you said. I'm wondering how I can hookup the 3 pc's in my house plus my Vonage phone box up to share the line? Maybe turn my current router into a bridge? Or buy a switch?

I have 1 more question about the hardware part of the process; does the box you setup as the firewall require 2 NIC's? (I'm assuming 1 will be needed for the external WAN connection that will be connected at the wall and another that will connect from the firewall box to my computer/switch/bridged-router).

[Anarki]

This is weird, I am just putting together a pfsense box myself for all my routing needs as I am typing this :D

Trust me I'm no *nix expert but its so easy to setup, there are tons of tutorials out there to help you.

I ended up using my dual P3 box with 2GB of RAM, I brought a CF to IDE adaptor and am running a 2GB CF card for pfsense (modified embedded version - less read/writes to the CF card but with full VGA support)

It's just formatting the CF card as I type this, next up is to install the 2 Intel Pro100 server NIC's I bought for £2 each brand new :yepp:

I'll post up pics of when its all done.

[DeadlyFire]

This is weird, I am just putting together a pfsense box myself for all my routing needs as I am typing this :D

Trust me I'm no *nix expert but its so easy to setup, there are tons of tutorials out there to help you.

I ended up using my dual P3 box with 2GB of RAM, I brought a CF to IDE adaptor and am running a 2GB CF card for pfsense (modified embedded version - less read/writes to the CF card but with full VGA support)

It's just formatting the CF card as I type this, next up is to install the 2 Intel Pro100 server NIC's I bought for £2 each brand new :yepp:

I'll post up pics of when its all done.

Nice! :)

[hixie]

I did a little googling and it looks very easy like you said. I'm wondering how I can hookup the 3 pc's in my house plus my Vonage phone box up to share the line? Maybe turn my current router into a bridge? Or buy a switch?

I have 1 more question about the hardware part of the process; does the box you setup as the firewall require 2 NIC's? (I'm assuming 1 will be needed for the external WAN connection that will be connected at the wall and another that will connect from the firewall box to my computer/switch/bridged-router).

Yes you will need at least 2 NICs, one for WAN one for LAN. If you have 3-4 NICs then you could setup wireless and also a DMZ (Demilitarized Zone)

Get a good switch, at a minimum get an unmanaged switch. Don't buy a hub, they're pretty dumb and get relay the signals. Switches do some wizardry and witchcraft so it's faster with high loads.
I don't know what's the difference but my cheap plastic brought from a shop hub didn't do very well, my more expensive SMC rackmount switch made a huge difference.


This is weird, I am just putting together a pfsense box myself for all my routing needs as I am typing this :D

Trust me I'm no *nix expert but its so easy to setup, there are tons of tutorials out there to help you.

I ended up using my dual P3 box with 2GB of RAM, I brought a CF to IDE adaptor and am running a 2GB CF card for pfsense (modified embedded version - less read/writes to the CF card but with full VGA support)

It's just formatting the CF card as I type this, next up is to install the 2 Intel Pro100 server NIC's I bought for £2 each brand new :yepp:

I'll post up pics of when its all done.

:eek: 2GB of RAM, i remember once running smoothie on 64MB RAM and i was still fine.

[Frisch]

2,000,000,000 for me! :D :D :)



:up:

I thought you'd just drop by, but somehow you have shipped around 800 mill, since the cat dragged you in .....:D

[DeadlyFire]

:up:

I thought you'd just drop by, but somehow you have shipped around 800 mill, since the cat dragged you in .....:D

Has it been 800mil already?? Time flies fast :eek::D

[Mekoa]

Has it been 800mil already?? Time flies fast :eek::D

That it does.

Good to see you again Hixie also!

Im just crunching away to get to 700mill...

[DeadlyFire]

Dave, for God's sake, slow down!!! :p::p::D

[Movieman]

Dave, for God's sake, slow down!!! :p::p::D

Just saw that myself.. Nice way to start the day but really an overflow from yesterday when his upload server was down.
10+mill in the first 6 hours of the day!:p:
I've learned to config the machines to cover my ass when this happens..:up:

Good to see all the new and old faces..
We got ourselves a real team going and I just looked and we are #1 for the day so far..:rofl:

[Martijn]

I only just found out MJ12 had crashed and failed to restart a few days ago... 15M down the drain :(

[Duh]

Just saw that myself.. Nice way to start the day but really an overflow from yesterday when his upload server was down.
10+mill in the first 6 hours of the day!:p:
I've learned to config the machines to cover my ass when this happens..:up:

Good to see all the new and old faces..
We got ourselves a real team going and I just looked and we are #1 for the day so far..:rofl:

I ´ve seen you save some buckets ( for the just if)... well done :D

off topic: is auto merge doable to be installed on XS MM?

[Duh]

I only just found out MJ12 had crashed and failed to restart a few days ago... 15M down the drain :(

No sh I realised you were not making any points . I was gonna ask if someone got news about ya pal :cool:

[Martijn]

No sh I realised you were not making any points . I was gonna ask if someone got news about ya pal :cool:

I'm fine, it's just been extremely busy the last few months. Sadly it doesn't look like it's going to change any time soon :(

[DeadlyFire]

100 terabytes crawled for me :toast2: on to the next 100! :D

[OldChap]

Grats man ...that is one big number! :up: I won't ever catch you with this connection but I will reach that eventually...thankfully there are some smaller targets in the meantime

[Duh]

100 terabytes crawled for me :toast2: on to the next 100! :D

way to go! congrats. as someone posted google's db is 100 pb , so keep crawling :D

[Frisch]

We got a billionaire more......OldChap...:up:

And he is chasing Hixies and Turtles....what have we done to you :shrug:;):D

[Movieman]

:up: Oldchap.. Nice going!:clap:

[DeadlyFire]

Congrats :up:

[Movieman]

OT..
Looking at numbers today.
You know if weall got together on one day we could take the #1 position for the dailies on the team scores.
Just have to cordinate whenand it's VERY doable.:D

[DeadlyFire]

OT..
Looking at numbers today.
You know if weall got together on one day we could take the #1 position for the dailies on the team scores.
Just have to cordinate whenand it's VERY doable.:D


I agree! I think the final tally was almost 45million? :eek: Not a bad day :D

Let's pick a day for a 24 hour marathon that's good for everybody :) :hehe:

[Frisch]

I'm in....I have about 5 mill to give in a day.....yeah I know, I'm not giving what I could, but I'm a Turtle, and that's an obligation which I take very serious. :D


Well, as said.....name a day, and I'll find my wheels... I know they are somewhere in my shell. :D

[OldChap]

Thanks all,

Now that I have this on 1 rig in 2 vm's things should steady down to about 9-10mill/day and not interfere too much with normal browsing.......

That said .....Would take about a week to stockpile a couple of extra machines for the purposes of a top score for one day ;)

[Duh]

congats OC!
we could do the marathon but my current firepower es close to null

[Frisch]

It doesn't sound as it's going to be that esay, and we have to set a date some weeks ahead.

So how about the 1st next month. Starting when the statspage over at alex's site turns to the 1st of october.

[Movieman]

ok, so 7PM on sept 31st( East coast time) we hit them heavy..:up:

[DeadlyFire]

ok, so 7PM on sept 31st( East coast time) we hit them heavy..:up:

I am up for it :hehe: :D

We should try to get a hold of moddolicous and see if he can drop 5mil down for us if only for a day :)

[OldChap]

1st October it is then :up: Won't be much fun surfing the net that day though ;)

[lkiller123]

I can only offer little.. Borging my high school's computers is hard as hell.

[Movieman]

I am up for it :hehe: :D

We should try to get a hold of moddolicous and see if he can drop 5mil down for us if only for a day :)
I'll PM him..

1st October it is then :up: Won't be much fun surfing the net that day though ;)
I had that..MJ12 on 4 machines here and 2 mins to open a web page:rofl:

I can only offer little.. Borging my high school's computers is hard as hell.

Bring what you can,it all adds up!:up:

[Frisch]

Just a reminder to have your reserved buckets in order, so that you won't run dry, IF the server should run dry......

Just a heads up : )

[Movieman]

Just a reminder to have your reserved buckets in order, so that you won't run dry, IF the server should run dry......

Just a heads up : )

I have mine all maxxed.. I leeaned the hard way..
Poor Alex,he works so hard and HATES when he runs out!:rofl:

[Frisch]

I have mine all maxxed.. I leeaned the hard way..
Poor Alex,he works so hard and HATES when he runs out!:rofl:

http://img837.imageshack.us/img837/9317/bko.jpg:D

[Duh]

ROTF very good Frisch!

[DeadlyFire]

Just a reminder to have your reserved buckets in order, so that you won't run dry, IF the server should run dry......

Just a heads up : )

seeing the situation right now this is a good point! :up: I only had 300 reserved per node...bumping up to 500 :D

[DeadlyFire]

Mr. Movieman, Dave himself, hitting the double digits(that's billions kids! :cool:)


Nickname: Movieman

URLs crawled:10,001,536,066

Data (MB)*: 261,248,240



Mighty congrats Dave!! :clap: :up: :up: :up:

[Frisch]

Mr. Movieman, Dave himself, hitting the double digits(that's billions kids! :cool:)
Mighty congrats Dave!! :clap: :up: :up: :up:

I'm just glad he's on our Team...:D

Congrats, Dave.

[OldChap]

That is a rather big number.....Great Work Dave :up:

[Frisch]

When the project was young, no one imagined that individuals would hit those numbers. It was a rush just when the WHOLE project reached 10 Billion...

Just a perspective on things....

Great run, Dave.

[Movieman]

Thanks to all..
Been chasing that number for a VERY long time!:rofl:

[DeadlyFire]

We got a billionaire more......OldChap...:up:

And he is chasing Hixies and Turtles....what have we done to you :shrug:;):D
He's almost caught himself a turtle :eek: Run frisch run! :D


I myself have just passed the 5 billion mark :toast2: on to the next 5! :D

http://i.imgur.com/g1vs7.png

[Mekoa]

He's almost caught himself a turtle :eek: Run frisch run! :D


I myself have just passed the 5 billion mark :toast2: on to the next 5! :D

http://i.imgur.com/g1vs7.png

Gratz OC and DF! I havent been able to get my rig working again, plus hefty power bill made me be a little more considerate with what I use :(

Building a new Phenom II X6 rig over the next few weeks (sigged, ha), so hopefully get my crunch box updated with the E8400 and WCG/MJ12 running 24/7 ish :)

[Frisch]

He's almost caught himself a turtle :eek: Run frisch run! :D


I myself have just passed the 5 billion mark :toast2: on to the next 5! :D

http://i.imgur.com/g1vs7.png

You know I can't run....you're hurting my turtle feelings :D

Gratz Deadly...it's out of this world..:up:


Gratz OC and DF! I havent been able to get my rig working again, plus hefty power bill made me be a little more considerate with what I use :(

Building a new Phenom II X6 rig over the next few weeks (sigged, ha), so hopefully get my crunch box updated with the E8400 and WCG/MJ12 running 24/7 ish :)

Welcome back mekoa, there's always room for another net "cruncher" :)

See you when you have things up and running. :up:

[Mekoa]

Thanks Frisch. Still chasing that 1bill marker!

Hopefully get a chance tonight or tomorrow to dismantle my old rig and see whats wrong!

[Movieman]

He's almost caught himself a turtle :eek: Run frisch run! :D


I myself have just passed the 5 billion mark :toast2: on to the next 5! :D

http://i.imgur.com/g1vs7.png

Now how the hell did I miss this?
Congrats on the FIVE Billion!:clap:

[DeadlyFire]

Thanks all :) Come back soon Mekoa! :up:

[OldChap]

Good one Deadly,

You see those flippers going 15 to the dozen :)

Maybe next year I can try to get after those two greedy pie eaters ....wonders how long I have to wait for a faster connection???

[Frisch]

Turtle 1,454 mill
OC 1,452 mill

http://img257.imageshack.us/img257/7953/turqm.jpg

[DeadlyFire]

:p:

..Don't make too much noise now, a hare by the name of Hixie is still snoozing :D

[Mekoa]

:p:

..Don't make too much noise now, a hare by the name of Hixie is still snoozing :D

Speaking of snoozing, parts arrived today and Phenom is up and running.. all be it me screwing up with forgetting TRUE's mount horizontally as standard and having to use the AMD stock cooler (bleah). Awaiting BTK from USA and TRUE Black goes on!

This does mean my project for this week is to rebuild my crunch box with my E8400!

Will need to check power bill after a bit but hopefully I can keep it on again (should be more efficient than an X2 6000?).

[OldChap]

Turtle 1,454 mill
OC 1,452 mill

http://img257.imageshack.us/img257/7953/turqm.jpg

Maybe this update....Midnight here...had 2 hours without power today so the turtle had to sweat just a little longer :D

Good push there my friend

[Frisch]

Turtle 1,458 mill
OC 1,457 mill

http://img190.imageshack.us/img190/609/57481742.jpg

A little forth and back here, but my tinnitus is killing me after keeping it up and running 24, so I will let you slide by ;):D............the metalic noise from the HDD is killing my ears.....:(

BUT....some hours work still to come, so WHEN......is the question...:D

[Frisch]

OC 1,462 Mill
Turtle 1,460 Mill

http://img513.imageshack.us/img513/9005/dxfbd.png


:D:up:

[OldChap]

Hey, go put your feet up Mr turtle...you deserve a sit down after that effort :up:

Now then, when this supercomputer week is over I must try to go back to running two instances of this but this time on separate drives one native with 2500aaks and one vm with samsung f3 1TB the machine is a x3350 @3.6

[DeadlyFire]

Here's a sight for sore eyes! :D Nice to see XS on top even without a breakneck speed marathon :)


http://imgur.com/Z9rYO.png

[Frisch]

Here's a sight for sore eyes! :D Nice to see XS on top even without a breakneck speed marathon :)


http://imgur.com/Z9rYO.png

:cheer::cheer::cheer::cheer::cheer::cheer::cheer:

:D

[OldChap]

Refic is trying to change it by last update though...still in front @ 10pm and only one update to go ....good work everyone

[Duh]

Not doing marathon but I am currently using some firepower I have at hand. I could use some extra firepower but it would strain this way too much and I might get scolded by clients and by provider.
There you go

http://www.majestic12.co.uk/projects/dsearch/userinfo.php?id=2532

EDIT: I am adding a 1 mbit line now :P

[DeadlyFire]

Refic is trying to change it by last update though...still in front @ 10pm and only one update to go ....good work everyone

We led to the end :cool: :D


http://imgur.com/rTav0.png

[OldChap]

Oh yea!!! :up::up::up:

[Frisch]

http://itsnotahickie.com/wp-content/uploads/2009/08/success_baby1.jpg?w=300


:D

[DeadlyFire]

Looks like we're on top again today(as of 10am EST) :D

[OldChap]

Anyone tried the beta yet? maybe the others are and it is helping us :)

[DeadlyFire]

Anyone tried the beta yet? maybe the others are and it is helping us :)

You might be right. The new loading server is serving up buckets that aren't spectacular in quality. The old 'hand-fed' buckets server that 1.6.12 nodes still use have 85%+ success. Alex mentioned the new node will have better quality buckets once recrawl of bad data is finished(couple weeks tops?). On the bright side, no more dreaded "NoMoreURLsAvailable" messages :D

[Frisch]

Turned on my MJ12, and had a look at the stats....

Well, there's us, and there's refic.........122 mill !!! ...

[DeadlyFire]

Turned on my MJ12, and had a look at the stats....

Well, there's us, and there's refic.........122 mill !!! ...

155mill total for the day :eek::D

[DeadlyFire]

Big milestone:


Name: XtremeSystems

URLs crawled: 30,011,997,950

Data (MB)*: 766,566,997

Congratulations to everyone who helped us get here!! :):cheer::cheer::party::cheer::cheer:

[DeadlyFire]

oh come on I can't be the only one impressed :p:

[Movieman]

Big milestone:


Name: XtremeSystems

URLs crawled: 30,011,997,950

Data (MB)*: 766,566,997

Congratulations to everyone who helped us get here!! :):cheer::cheer::party::cheer::cheer:

:eleph::eleph::party::party::bounces::bounces: :bday2::party2::banana::banana2::toast::toast: :YIPPIE::YIPPIE::cheer::cheer::cheer::rofl:

[Frisch]

oh come on I can't be the only one impressed :p:

"I'm not impressed, it's just showing how strong we are....others should be impressed, and make this thread glow of :eek2:"

:D

[OldChap]

It's alright I guess...

Yay! missed this completely ...WELL DONE TEAM :D :up:

[DeadlyFire]

There we go :D

[Duh]

we need to catch up refic at least... he has reached the PB

[hixie]

:p:

..Don't make too much noise now, a hare by the name of Hixie is still snoozing :D

LOL! little late to reply to that but better than never.

This hare isn't snoozing, it's crawling as fast as it can. I know it's not much, and my monthly output is only a fraction of what i used to output in an hour, but that's all i can do without my 100mbps line :(

[Frisch]

LOL! little late to reply to that but better than never.

This hare isn't snoozing, it's crawling as fast as it can. I know it's not much, and my monthly output is only a fraction of what i used to output in an hour, but that's all i can do without my 100mbps line :(

And if it regains its speed......http://www.karin-lisbeth.dk/images/e/elmer/001.jpg



:D

[DeadlyFire]

Either way it's still good to have you back back Hixie :up:

[Movieman]

LOL! little late to reply to that but better than never.

This hare isn't snoozing, it's crawling as fast as it can. I know it's not much, and my monthly output is only a fraction of what i used to output in an hour, but that's all i can do without my 100mbps line :(

We'll take whatever you can add and say thanks..:up:

[hixie]

I've been crawling for quite a while ... just not large figures, and been too busy to post here.

BTW dave, why aren't you going to vegas? I'll be there with a booth this year.

[DeadlyFire]

Finally have conquered a quarter of the pie http://i.imgur.com/jy5qi.gif Tasty URLs! :D

http://i.min.us/ibHDzQ.png

[DeadlyFire]

1.7 final node released:


v1.7.0 5/02/11
+ Support for new generation of central server in parallel with current
! Better handling of redirects
! Better control of domain counts during crawling
! Improved analysis of crawl errors
! Fixed rare issue with empty indexed data written incorrectly
! Mono - support for alternative spawning of archiver, .NET 2.0 build is now the only one available
! Mono builds - Less junk in log on communication errors
! Mono - better logic for handling multiple crawlers on same box
! Added MaxPriorityBuckets parameters to options
! Max reserved (pre-cache) buckets is forced to 1 in order to enable more efficient crawling on whole distributed network
! New SQLite build used (once run database won't be backwards compatible with 1.6.x series)
! Bundled 64-bit SQLite build with Mono distributions
! Mono builds now support https protocol crawling
! Reduced number of messages printed by default (can still be shown if Warnings mode is on)
! Put a limit on barrel archiving to avoid create too many temporary files in odd data sets



You can increase max reserve buckets from 1 to 3 for more sustained crawling :up:

[Duh]

Finally have conquered a quarter of the pie


Well done dude! keep it up

[DeadlyFire]

Well done dude! keep it up

I'm trying :D

btw..where has that 30mb goodness from Dave gone to?? :shrug:

[OldChap]

I expect he is in the same boat as the rest of us...apart from being away just now.... waiting for the new server to be up in order to redo settings to maximise output. I see we can start in on this tomorrow ...he said hopefully.

Well done on the numbers by the way

[Movieman]

I'm trying :D

btw..where has that 30mb goodness from Dave gone to?? :shrug:

Two issues:
First is I haven't been able to get more than 6mbit total from 3 machines since Alex made those changes.
Second is I have one machine down.
Swapped cpu's that had been in that machine and fine and now won't post..
Clueless as to why and that was my main machine..:shrug:

[Duh]

Two issues:
First is I haven't been able to get more than 6mbit total from 3 machines since Alex made those changes.
Second is I have one machine down.
Swapped cpu's that had been in that machine and fine and now won't post..
Clueless as to why and that was my main machine..:shrug:

Dave : use a linux box ( ubuntu). Installation takes 5 mins at most with no previous knowledge.

Also: refic has made a script to install mono ( the toughest part of linux install).

We can guide you and you can call DF who knows it quite a bit :P.

The setup of the ubntu box ( of mj12) could take you 30-40 mins) and you can use all the 30 mbit from a single quad core ( old q6000 would doing it with no sweat)

[Movieman]

Dave : use a linux box ( ubuntu). Installation takes 5 mins at most with no previous knowledge.

Also: refic has made a script to install mono ( the toughest part of linux install).

We can guide you and you can call DF who knows it quite a bit :P.

The setup of the ubntu box ( of mj12) could take you 30-40 mins) and you can use all the 30 mbit from a single quad core ( old q6000 would doing it with no sweat)

A short answer: No!:rofl:
I'm a windows guy and no linux in this house!:p:

[DeadlyFire]

Dave, under the 'More Crawler' tab in options, try setting 'Maximum deep crawl buckets' to 0 and 'Maximum priority buckets' to 50(or even 100 if 50 works ok). Also, Alex just released the 1.7 final version which lets you raise your reserved buckets from 1 to 3, which should use up more of your connection while crawling so upgrading should give you a boost.

regarding linux, it's very easy to install and run BUT it's a bit of a pain in the a$$ because mono is less stable than .NET under Windows so you constantly have to watch for bugs(not fun).

[DeadlyFire]

http://i.imgur.com/L7WRY.png

Congrats on 11 Billion Dave! :up::up:

[DeadlyFire]

I got hit by a nasty concoction of viruses and malware last week, just finished painstakingly reinstalling Windows and everything else(though I have made a clone image of my HD and am going to do monthly backups in case something goes wrong again! :)).

[Duh]

sh.. DF.. I always have such sh.. I wonder why alex would not include an AV+malware. .. he could even sell a list of webs containing that sh..

[DeadlyFire]

sh.. DF.. I always have such sh.. I wonder why alex would not include an AV+malware. .. he could even sell a list of webs containing that sh..

It was my first virus in a year or so, not a bad ratio :D good news is the infection I got wasn't related at all to MJ12 :up: Alex has said before anything crawled is never run or executed so nothing to worry about :)

[Duh]

It was my first virus in a year or so, not a bad ratio :D good news is the infection I got wasn't related at all to MJ12 :up: Alex has said before anything crawled is never run or executed so nothing to worry about :)


i know its not executed but if he includes an AV+malware he could detect which website has sh.. inside and sell such to AV companies ;)

[DeadlyFire]

Congrats on 2 Billion OldChap!! :clap::up:

[OldChap]

Cheers, It did seem that it would never happen during December-January but I got there in the end even though I now have to stop during the day on my main "unlimited" line. apparently unlimited to them means if you dl more than 8Gigs a day between 9am and 9pm they will terminate the connection.

[Frisch]

Big number, I'll try and follow your grand example, and reach 2 billion.......in Turtle tempo : ) But when I'm there you're long gone, so I'll shout it out loud with the wind, so you can hear me up there in front..... : )

Congrats !

[Mekoa]

Gratz all! Missed a few milestones it seems.

Shame I still cant get my weird error issues fixed, boo :( Crunching WCG 24/7 at the mo on 6 cores though. Gotta love AMD for hex!

Will grab this build over the next week to see if the errors have disappeared, unlikely though. Bar that, must be my ISP blocking traffic or something :(

[DeadlyFire]

Hi Mekoa :) It's been really quiet around here, welcome back :up:

When did you start having those error issues? Was it after the big beta node update around December?

[Mekoa]

Hi Mekoa :) It's been really quiet around here, welcome back :up:

When did you start having those error issues? Was it after the big beta node update around December?

Hey Deadly :D

Its the below thread, mostly started after I upgraded to Windows 7. Have no issues with internet using anything else, just MJ12 (same ISP as before also).

http://www.xtremesystems.org/forums/showthread.php?t=261795

[[XC] Oj101]

I've just signed up :D

[[XC] Oj101]

And now have a question. I'm on a 4096/512 line and after setting M12 to use 100% of the line I have done 21,500 after 3h15m which seems low. I've yet to see the download rate pass around 650 and it usually hovers around 200 to 300, often dipping down to under 50 (last edit, I've just seen download as 9, 0, 83, 14). What's up with that?

EDIT: In those three and a bot hours I've used 250MB down/15MB up. I KNOW that I can pull around 1.4GB/hour over HTTP/FTP and upload around 150MB/hour. Nothing else is using the network other than a bit of browsing (no flash/very few images).

[OldChap]

Hey oj, tell us what you chose for crawler settings.

I put my laptop to work on a 8meg/0.7meg line with 100 workers and 35 buckets a few days ago and it seems to run ok but took 6 days for 9mill

On my 50.1.75 line I rarely see more than 25kbits sec this one seems to run at maybe 2

[DeadlyFire]

Hey Deadly :D

Its the below thread, mostly started after I upgraded to Windows 7. Have no issues with internet using anything else, just MJ12 (same ISP as before also).

http://www.xtremesystems.org/forums/showthread.php?t=261795

After the next time the NIC goes down, go into control panel and open the 'network and sharing' button. Then on the left side 'change adapter settings' and then find your NIC in the next window. Right-click on it and click 'status' and then the 'details' button. Take note of the IPv4 address before and after the crash and the lease obtain/expire time. It might show no change at all but it's worth a try to see what happens.




Oj101;4759387']And now have a question. I'm on a 4096/512 line and after setting M12 to use 100% of the line I have done 21,500 after 3h15m which seems low. I've yet to see the download rate pass around 650 and it usually hovers around 200 to 300, often dipping down to under 50 (last edit, I've just seen download as 9, 0, 83, 14). What's up with that?

EDIT: In those three and a bot hours I've used 250MB down/15MB up. I KNOW that I can pull around 1.4GB/hour over HTTP/FTP and upload around 150MB/hour. Nothing else is using the network other than a bit of browsing (no flash/very few images).

What OC said ^^ What operating system are you running? For a 4mbit line I'd suggested 100 max # workers and 20-50 max open buckets(start low and keep raising until you're satisfied with the speed). The general idea is keep a low amount of buckets open but a higher number of workers. Also, under the 'More crawler' tab within options, I would set maximum priority buckets to something low like 5 or 10, if you don't have a beastly router those kinds of buckets can cause the router to hang. Be sure to check if your ISP allows unlimited or large amounts of downloading!!

Happy crawling and thanks for joining! :up::welcome:

[DeadlyFire]

One HUGE milestone for one of our smaller yet more determined crawlers :up: 100mil may not seem that big for some of us but for someone who chugs along at 100k urls/day it's a helluva milestone

http://i.imgur.com/u7Bby.png

100,028,744

Congrats eme! :clap::clap:

[OldChap]

:up: congratulations :up:

[Duh]

DeadlyFire : you are very right. Congratz to eme ! Very well done man!

[[XC] Oj101]

I followed the Majestic12 guide (the sticky) to the T, as while all options are explained a lot are still over my head :D

I'm running a Billion BiPAC 7300GA which can handle several tens of thousands of seeds are leeches connected in uTorrent so these packets shouldn't be a concern. Current router uptime is several days so it's not causing issues as far as I can tell :)

[DeadlyFire]

Oj101;4761884']I followed the Majestic12 guide (the sticky) to the T, as while all options are explained a lot are still over my head :D

I'm running a Billion BiPAC 7300GA which can handle several tens of thousands of seeds are leeches connected in uTorrent so these packets shouldn't be a concern. Current router uptime is several days so it's not causing issues as far as I can tell :)

Good to hear :up: If you're willing to search, there's tons of tips and advice scattered all over this subforum, just finding it is a pain in the a$$ :rofl: If you have any problems with your node or router shout it out here. Thanks again for joining :)

[DeadlyFire]

A good day for us yesterday, 40mil/over 1TB crawled and second place :clap: :

http://i.imgur.com/nokDc.png

[OldChap]

Good work guys :up:

I'll be glad when I can look at my output and it stays constant again though....Just now one day can be 50% more than another just now

[Duh]

Good work guys :up:

I'll be glad when I can look at my output and it stays constant again though....Just now one day can be 50% more than another just now


we will solve that ;)

I am chasing you now, you better watch your six :p

[OldChap]

I had a weekend of discovery.....

My original setup of running 2 nodes at 350/100 to max out my connection has still been going at it quite well but with the current mix of buckets I find my connection (at least the upload which limits me) to be only partially used and the download was sometimes high sometimes low. As an interim measure I had increased buckets to 125 for slightly better performance.

I have now added a third node at 300/100 and the upload is now balls to the wall whilst the download is staying pretty constant. First full day yesterday and much better numbers.

I will report back here if my percentages suffer but overnight at least they are still in the 80's

[DeadlyFire]

Congrats elvis on hitting the Billionaires club!! :clap::clap::clap:

http://i.imgur.com/61sR9.png


1,000,733,396 to be precise :up:

[OldChap]

:woot: http://www.dogproductshop.co.uk/smile/party/party0019.gif http://www.dogproductshop.co.uk/smile/party/party0018.gif A bit of a push going on lately....Congrats.

[Duh]

Congrats elvis on hitting the Billionaires club!! :clap::clap::clap:

http://i.imgur.com/61sR9.png


1,000,733,396 to be precise :up:



:blushes: many thanks DF :)

shoota and shootas mom have joined into the team ;)

[Duh]

WHAT THE F:::::.



http://i.imgur.com/t2UwO.png


is it just me ??

Just today i run out of ritalin so I might be just me ( not joking)

[DeadlyFire]

WHAT THE F:::::.



http://i.imgur.com/t2UwO.png


is it just me ??

Just today i run out of ritalin so I might be just me ( not joking)

April Fool's joke :rofl: I almost spat out my coffee when I saw 3 Moviemans in this thread alone :ROTF:

[DeadlyFire]

I just looked at the XS team stats page (http://www.majestic12.co.uk/projects/dsearch/teaminfo.php?id=60) and we have 10 active crawlers today! :eek:


http://i.imgur.com/QgMRF.png


:clap::clap::up:

[hixie]

:woot::woot::woot:

[Frisch]

I just looked at the XS team stats page (http://www.majestic12.co.uk/projects/dsearch/teaminfo.php?id=60) and we have 10 active crawlers today! :eek:


http://i.imgur.com/QgMRF.png


:clap::clap::up:

Look who's the last......weeeeeeeeeeeeeeeeeeeeeee :D

[Mekoa]

Grats to all for some huge milestones! :cheer:

I really need to sit down when I get time and see if I can fix my clients :(

Edit: Installed new 1.7.0 client, appears to be running well!

[DeadlyFire]

Good to have you back Mekoa! :):)

today that makes it 11 :D

http://i.imgur.com/Qwl6C.png

[Mekoa]

Good to have you back Mekoa! :):)

today that makes it 11 :D

http://i.imgur.com/Qwl6C.png

I have a habit of speakin too soon. It started dropping tons of errors around 2 hours in. Main was Conn Errors or Retries. Boo :(

Either I'm getting really crap buckets or something is stopping my traffic from going through.

[DeadlyFire]

In the options menu, under the Crawler tab, what do you have set for the last section(Hard errors)? also, what is your 'max priority buckets' set to?(under the 'more crawler' tab)

[Mekoa]

In the options menu, under the Crawler tab, what do you have set for the last section(Hard errors)? also, what is your 'max priority buckets' set to?(under the 'more crawler' tab)

Threshold 10, all options checked.

Max Priority Buckets set to -1

Edit: Dropped threshold to 1 as a test, still generating lots of Conn Errors.

[DeadlyFire]

Threshold 10, all options checked.

Max Priority Buckets set to -1

Edit: Dropped threshold to 1 as a test, still generating lots of Conn Errors.

Try setting threshold to 100 and priority buckets to 0 temporarily. Check the results over the next couple hours(it will take some time for remaining priority buckets to clear out).

[Mekoa]

Try setting threshold to 100 and priority buckets to 0 temporarily. Check the results over the next couple hours(it will take some time for remaining priority buckets to clear out).

Changed and running now. I left it on after snoozing, it destroyed my internet within windows. Couldnt hit a single webpage, even though no bandwidth was being used.

[DeadlyFire]

Changed and running now. I left it on after snoozing, it destroyed my internet within windows. Couldnt hit a single webpage, even though no bandwidth was being used.

That's the priority buckets causing that, Dave experienced that as well. I would start with those set to 0 then slowly raise it(maybe by 1 at a time).

[Mekoa]

That's the priority buckets causing that, Dave experienced that as well. I would start with those set to 0 then slowly raise it(maybe by 1 at a time).

Im guessing thats probably what my original issue was then. Been running a few hours now with around 78% success and I can still use interwebs!

Cheers Deadly, could be on to a winner here.

[DeadlyFire]

Great! :) If you're willing you can try to raise it to 1 and see how it goes. If it makes the connection unusable or causes huge amount of errors then stick to 0.

[OldChap]

I'm doing this too now. I restarted and sent back 8 late buckets. That should be first in first out if you ask me with the priority buckets getting done with all the rest.

I keep running into too many buckets open too which must be all about the priority buckets setting not working properly.

Normally on my connection I get a state table size of around 40,000, Last night it was pushing 100,000 so just to see what would happen I upped the limit to 1,000,000 and it immediately went to 130,000

Just now although it will run this is broken as most controls don't work.

[DeadlyFire]

Yep definitely the priority buckets :( I say set it to lowest possible or 0 even(i'll add another 20 across my nodes to cover your share of those ;)) and call it a day.

[Duh]

so current best is to set it to 1?

I did not know that threshold trick :)

thx

[hixie]

Anyone used untangled before? the attack blocker blocks 90% of MJ-12's activities. I've been getting mad errors.

[DeadlyFire]

Can't say that I have but I did look into it before choosing pfSense. I know in pfSense you can install packages to do network IDS, not sure about Untangle :shrug:

[hixie]

i kinda have untangled already setup and running in the office ... don't want to have to change everything. If i do change over i'll have to get another machine to have pfsense running and do a straight change over, with minimum down time.

UPDATE: i made an exception for my computer through the attack blocker, but cause untangled is on transparent bridge mode i have no control over the local IPs and all IPs are renewed every 24hrs :(