one point five billion..

[Movieman]

Nick........... URLsdone... Data(MB)
Movieman 1,500,133,293 25,327,232 :hrhr:

[Frisch]

:shocked: :D I brought some ballons....now where's the beers

http://www.thesweetbasket.com/images/balloon/Mixed%20Ballons.bmp

[Movieman]

:shocked: :D I brought some ballons....now where's the beers

http://www.thesweetbasket.com/images/balloon/Mixed%20Ballons.bmp

I went looking for some beer but the stores were empty..
Told me some crazy turtle ran through guzzling it all down!:D
Next stop: Refic..figure 40-50 days and he's history..
Then for our friend..Carl..:devil:

[Mekoa]

Woo, go go go.

Im just ever so slightly behind you guys...... only a margin of about 1.45billion :P

[Movieman]

Woo, go go go.

Im just ever so slightly behind you guys...... only a margin of about 1.45billion :P

You may not beleive this but it's true. I don't care if you do 1,000 a day or 5 million, you came to help and thats good enough for me.
My Thanks!:up:

[Mekoa]

I just slapped Fedora on my 2nd machine but I cant get MJ12 to run on it (Mono site not working) - sigh! Must fix :D

1000 a day would annoy me though, prefer to do a mill or so ^^

[Paladin]

I just slapped Fedora on my 2nd machine but I cant get MJ12 to run on it (Mono site not working) - sigh! Must fix :D

Let me know how that goes. I had gotten Mono-MJ12 running, but wasn't happy with it's production (especially since Linux should be more efficient than Winbloat). Not sure which version of Mono I have installed -- I'm guessing not the ancient version that's suggested.

[Mekoa]

Let me know how that goes. I had gotten Mono-MJ12 running, but wasn't happy with it's production (especially since Linux should be more efficient than Winbloat). Not sure which version of Mono I have installed -- I'm guessing not the ancient version that's suggested.

Ended up dropping Server 2003 on it instead. Linux may be more efficient and nice and things, but I know how to use Windoze far better :<

Crunching away quite happily 24/7 now :)

[Movieman]

This has been a long chase. Just looking at the dates here. Almost 5 months.
Just coming up on 2.5 billion now and here's the best part: :hrhr:

[Mekoa]

Weee go go go Dave.

Dunno what mine would be for that lol. Dont think I can because he beats me everyday ^^

[Movieman]

Mekoa;2932562']Weee go go go Dave.

Dunno what mine would be for that lol. Dont think I can because he beats me everyday ^^

He was cranking close to 10 mil a day for awhile but slowed. I think he lost his office machines.
I decided way back that I was going to run him down no matter how long it took. Will be a 7 month rundown with my line essentially maxxed over 90&#37; of the time.
I'll never catch PY222.. Cripes, if he quit today it would take me three years to catch him at this rate.

[Mekoa]

Ive tried runing MJ12 on my machines at work, but due to local policies, it just crashes the program.

Corporate IT at its best!

[Movieman]

Mekoa;2932593']Ive tried runing MJ12 on my machines at work, but due to local policies, it just crashes the program.

Corporate IT at its best!

Yea, some can and some can't..thats life.
I wish I had the $$ to add a second line and run it at 100% on MJ12.
Maxxed with nothing else happenning I can JUST make 10 mil a day with good buckets but realistically I have to use some bandwidth for myself and Dear Old on a daily basis so 6 mil a day is a reasonable number for me.

[Mekoa]

Yea, some can and some can't..thats life.
I wish I had the $$ to add a second line and run it at 100% on MJ12.
Maxxed with nothing else happenning I can JUST make 10 mil a day with good buckets but realistically I have to use some bandwidth for myself and Dear Old on a daily basis so 6 mil a day is a reasonable number for me.

150 machines at my work, 20/20 fibre line with a 20/5 backup.

Somehow I think they would notice the huge bandwidth lol. Wish I could fire it up in one of our data centres. Several gigabit lines I think with more PC's and servers than I can count.

Im lucky if I hit 2mill per day atm. Single machine on an 8mbit line. Been playing a lot of Warcraft recently though so have to stop crawling for a bit as it lags me something awful :(

[Movieman]

Mekoa;2932600']150 machines at my work, 20/20 fibre line with a 20/5 backup.

Somehow I think they would notice the huge bandwidth lol. Wish I could fire it up in one of our data centres. Several gigabit lines I think with more PC's and servers than I can count.

Im lucky if I hit 2mill per day atm. Single machine on an 8mbit line. Been playing a lot of Warcraft recently though so have to stop crawling for a bit as it lags me something awful :(

Shame you can't grab them at night unless they are moving data 24/7..
As to warcraft, I understand, we all have things we do and MJ12 has to be second to those things.

[Mekoa]

Shame you can't grab them at night unless they are moving data 24/7..
As to warcraft, I understand, we all have things we do and MJ12 has to be second to those things.

Thing is, it used to be OK. Seems to whore up more bandwidth now. Perhaps due to better network efficiency with 1.6.0 or something?

I run DIMES 24/7 on cruncher also, maybe thats using up more bandwidth also. Not to mention Im syncing a bit lower too. Sigh, wtb your Verizon connection!

[Movieman]

Mekoa;2932610']Thing is, it used to be OK. Seems to whore up more bandwidth now. Perhaps due to better network efficiency with 1.6.0 or something?

I run DIMES 24/7 on cruncher also, maybe thats using up more bandwidth also. Not to mention Im syncing a bit lower too. Sigh, wtb your Verizon connection!

Could be, I have 1.6 on one machine and 1.5 on the other 2.
Killer here is I have a 15/2 line, they are offerring a 15/15 but that doesn't do a thing for MJ12 and 2mbit is fine for me for upload speed.
They offer 20/5 in other areas but not here.
They do have this but too rich for my wallet:
30 Mbps download
15 Mbps upload
$139.95/month

[Mekoa]

Could be, I have 1.6 on one machine and 1.5 on the other 2.
Killer here is I have a 15/2 line, they are offerring a 15/15 but that doesn't do a thing for MJ12 and 2mbit is fine for me for upload speed.
They offer 20/5 in other areas but not here.
They do have this but too rich for my wallet:
30 Mbps download
15 Mbps upload
$139.95/month

Nice line, hefty price though.

My line is theoretical maximum 24mbit down but you need to have very low noise margin to get that.

Im now synced at 9250/1300 which is about normal for me. Not bad for an unlimited connection at £18 a month.

[Movieman]

Mekoa;2932632']Nice line, hefty price though.

My line is theoretical maximum 24mbit down but you need to have very low noise margin to get that.

Im now synced at 9250/1300 which is about normal for me. Not bad for an unlimited connection at £18 a month.

Hey, it all beats the old 33600 modems we used to use..:rofl:
I can remember sending a 1 gig set of files to a guy at 2.2K..
a month but it got there!:up:

[[XC] riptide]

Well done Dave.

MAXIMUM CRAWL

[Movieman]

riptide;2932654']Well done Dave.

MAXIMUM CRAWL

Thanks pal.. Not quite there yet..few more days.. looking like I can do app 2.2 billion/year at this rate but I think the project will be over in the next year tops..maybe 6-8 months.
I'll have to be happy with #2 spot..sniffle-sniffle..
No one can beat PY222 in any app that is BW..
He's doing 17-18 mil a day now.
I couldn't do that with 2 of my fios lines...

[[XC] riptide]

Hmmm. I used to have a unintentional knack of getting good points for bad download. hmmm Wonder is there a way to 'replicate' such conditions on your setups? :hehe: http://stats.free-dc.org/new/teamstats.php?proj=maj&team=XtremeSystems
See my peers around me? I'm no. 37 on teh list.

[DeadlyFire]

No one can beat PY222 in any app that is BW..
He's doing 17-18 mil a day now.
I couldn't do that with 2 of my fios lines...

The trick is teamwork, FreeDC won't leave tire treads on us with our powerhouse team :) (as long as py222 doesnt crank up to 40 mil a day *knocks on wood*




This has been a long chase. Just looking at the dates here. Almost 5 months.
Just coming up on 2.5 billion now and here's the best part: :hrhr:

Go get em! :up: :up: :D


Speaking of conquests, there's a certain hard shelled friend that I have my sights on :hehe: :p:

http://img337.imageshack.us/img337/901/weeexy2.png

#5 is mine in a little over a week! :woot:

[DeadlyFire]

riptide;2932679']Hmmm. I used to have a unintentional knack of getting good points for bad download. hmmm Wonder is there a way to 'replicate' such conditions on your setups? :hehe: http://stats.free-dc.org/new/teamstats.php?proj=maj&team=XtremeSystems
See my peers around me? I'm no. 37 on teh list.


Almost 12 mil url's in under 93 gigs?! :eek::up:

[[XC] riptide]

Almost 12 mil url's in under 93 gigs?! :eek::up:

LOL. Well. Again it was not intentional. I suspect its because of having an improper MTU size and generally high error rate. Most of my crawling was done by wireless. Hence dropped packets, errors etc. Particularly timeout error.

However, you may consider that my comment above is not 100&#37;. I don;t really expect anyone to purposely degrade the standard of their output just to get more points.



But it sure was a teaser. :D:D Now Scale up 93gigs : 12mil to Daves figures. Then work out what he could do per day on the new ratio :lol: JUst to put it in perspective, on the same rate since he started, Dav would have almost 6 Billion. :yepp: And hit 20mil /day :yepp:

[DeadlyFire]

riptide;2932702']LOL. Well. Again it was not intentional. I suspect its because of having an improper MTU size and generally high error rate. Most of my crawling was done by wireless. Hence dropped packets, errors etc. Particularly timeout error.

However, you may consider that my comment above is not 100%. I don;t really expect anyone to purposely degrade the standard of their output just to get more points.



But it sure was a teaser. :D:D Now Scale up 93gigs : 12mil to Daves figures. Then work out what he could do per day on the new ratio :lol: JUst to put it in perspective, on the same rate since he started, Dav would have almost 6 Billion. :yepp: And hit 20mil /day :yepp:


Yes, scary but in a mischievous way :D I remember a thread on the MJ12 forum about a user pulling this intentionally, Alex banned his sorry a$$ fast :cool: But he came back and tried to pull it again lol :confused: A little puzzling since this is a voluntary project where you donate your bandwidth for a good cause and cheating helps no one(except the stats whore)

[Mekoa]

My conquests dont look quite as shiny as Dave's heh.

[Movieman]

riptide;2932702']LOL. Well. Again it was not intentional. I suspect its because of having an improper MTU size and generally high error rate. Most of my crawling was done by wireless. Hence dropped packets, errors etc. Particularly timeout error.

However, you may consider that my comment above is not 100&#37;. I don;t really expect anyone to purposely degrade the standard of their output just to get more points.



But it sure was a teaser. :D:D Now Scale up 93gigs : 12mil to Daves figures. Then work out what he could do per day on the new ratio :lol: JUst to put it in perspective, on the same rate since he started, Dav would have almost 6 Billion. :yepp: And hit 20mil /day :yepp:

nah, I'd rather do 6 mil a day and give him good info than 20 mil a day and give him bad.
There was talk there about switching to just credit for "good" urls and I posted against changing the rules in the middle of the game and Alex agreed although it might have been just to placate some people.
BUT I wouldn't take advantage of that. I like to win but only if it's fair so I take what they send me and exclude nothing..
When you know your playing the game 100% honest it makes winning taste all the more better!:up:

[[XC] riptide]

^^^ Couldn't agree more.

Just to show the state of the link I have. My max MTU size that I can use is 572. AS opposed to 1500 for most normal lines. The tel tale 'acid test' is that hotmail or bebo won't work on a unmodified TCP stack. Cos their servers don't do MTU discovery (I think its a precation against DDOS attacks).

AS to you guys with large pipes. Have you ever considered your settings? Particularly you Recieve Window? If its too low, a large line will suffer. You need a massive Recieve Window to fully use your FIOS dave?

[DeadlyFire]

riptide;2933101']

AS to you guys with large pipes. Have you ever considered your settings? Particularly you Recieve Window? If its too low, a large line will suffer. You need a massive Recieve Window to fully use your FIOS dave?


How would you know what to set the window size to? It depends on the latency to the servers you most often communicate to and with MJ12 it jumps all over the place :confused: Maybe set the receive window to the worst case scenario and monitor the results? Anywho I'm to try this out :D

[Mekoa]

How would you know what to set the window size to? It depends on the latency to the servers you most often communicate to and with MJ12 it jumps all over the place :confused: Maybe set the receive window to the worst case scenario and monitor the results? Anywho I'm to try this out :D

I thought RWIN was determined by the speed of your connection and a multiple of MTU minus overhead?

[[XC] riptide]

Corrrect mostly on both accounts. You can set a very large RWIN. I do not think there will be a risk of too high. The point been that Default XP come with a RWIN set too low.

Now, On Vista I beleive there is dynamic RWIN adjustment. So on Vista your ok! :)

Also guys, you could try turn off TimeStamping as this add another 12bytes to each packet. This 12 bytes is not data but more signalling overhead, if you will. Timestamping I believe is used in corporate enterprise solutions mostly.

Guys, DL "TCPOptimizer". Its free. http://www.speedguide.net/downloads.php


Also run this test on your MJ machines. http://www.dslreports.com/tweaksTweak test.

[Mekoa]

I used TCP Optimizer on my XP machines, havent bothered with it on Vista as of yet. Most of the settings TCP Opt changes arent even present in Vista registry!

[[XC] riptide]

What size is your ideal MTU? And did you calculate your RWIN?

[Mekoa]

riptide;2935076']What size is your ideal MTU? And did you calculate your RWIN?

1500, no.

Sure RWIN on XP was 512000 ish. Cant remember the exact numbers. Cant remember exactly how its calculated either!

[DeadlyFire]

Using the optimal settings option in the optimizer I set the mtu and rwin to 1500 and 1,027,840. MJ12 doesn't seem to have had any impact but success are around 88-90% and hourly url's bumped up to 330k from 315k so it didn't hurt.

[Mekoa]

Screeny of my TCP Opt settings for current connection of 9mbit.

Need to run it on my 24/7 box. Hasnt been optimized!

[[XC] riptide]

Nice one deadlyly fire. Glad it made a bit of difference!

Now. For completion. Mekoa/Deadly. Go to Largest MTU, and run a test. Then post results here.

[Mekoa]

Im already on my 1500 MTU mate. Cant go any higher as itll fragment the packet :(

[[XC] riptide]

Mekoa;2935584']Im already on my 1500 MTU mate. Cant go any higher as itll fragment the packet :(

:rolleyes: YES! 1500 is teh largest anyways!!!! Just run the test!

I know your Scottish and all, but stay with me on this. :yepp: :D

[Mekoa]

riptide;2935821']:rolleyes: YES! 1500 is teh largest anyways!!!! Just run the test!

I know your Scottish and all, but stay with me on this. :yepp: :D

Hopefully be able to test it over an 8 hour period tonight (12pm - 8am). Ill restart MJ12 in 39 mins.

And btw, my MTU has always been set to 1500 :P Enforced by the router :D

[[XC] riptide]

Mekoa! Go to 'largest MTU' on TCPOptomiser. It takes like 2 seconds. Not eight hours. And do it while MJ12 is running.

It will TELL you what your recommended MTU SHOULD BE!

Most of the time it will be 1500/1472.

[Mekoa]

riptide;2936130']Mekoa! Go to 'largest MTU' on TCPOptomiser. It takes like 2 seconds. Not eight hours. And do it while MJ12 is running.

It will TELL you what your recommended MTU SHOULD BE!

Most of the time it will be 1500/1472.

I know how to use TCP Opt ;)

As I said, its 1500(1472)-28 for ICMP and IP overheads. I meant I would test over a period of MJ12 for 8 hours after I ran TCP Optimizer to see if it made a difference.

Either Ive got a better batch of URLs or success rate is up (90.7%) doing approx 110k URLs an hour which is a wee bit above what it used to be.

My ISPs MTU is 1500.

[[XC] riptide]

LOL> Ah right. I get ya! ;)

[Mekoa]

riptide;2937292']LOL> Ah right. I get ya! ;)

Reason I started at midnight was for ease of hour counting lol. Still unsure if its just a better batch of URLs or tweaks made a bit of a difference.

PY222 is still almsot beating us singlehandedly :( WTB their connection!

[Mekoa]

One thing I have noticed recently, but not sure why its doing it, is that I seem to be doing less URLs for more MB. See below.

Edit: Something is definetly wrong. Moddolicious has 170mill more URLs crawled than me, but only 200Gb more downlaoded than me? Sorcerer is 50mill ahead but 1mill MB less?

Halp!

[[XC] riptide]

LOL. Lower error rates? You downloading big pages on each bucket.

[DeadlyFire]

Mekoa;2937316']

PY222 is still almsot beating us singlehandedly :( WTB their connection!


I dunno, Hixie is down today yet our team is still managing to outpace PY222 by at least 1.5mil :D

[Frisch]

I dunno, Hixie is down today yet our team is still managing to outpace PY222 by at least 1.5mil :D

That's the strength of this team. You can take some days off, and still we're on top.

Imagine a day with all on full speed. :D

[Movieman]

I think Hixie is away from the 20th-30th in Taiwan so his numbers will be down for that time.
We can compensate if we all get on in force!:up:

[Mekoa]

riptide;2937688']LOL. Lower error rates? You downloading big pages on each bucket.

Not really sure tbh. Unsure how I would even check lol. Screeny of client below for what its currently crunching at.

[[XC] riptide]

IIRC doesn't the client on another tab show the average page size? Bucket watcher?

[Mekoa]

riptide;2938289']IIRC doesn't the client on another tab show the average page size? Bucket watcher?

Aye, most are between 30-40kb per page on my current buckets.

[Frisch]

Speaking of conquests, there's a certain hard shelled friend that I have my sights on :hehe: :p:



You want me bad, according to the numbers you're producing at the moment.

Come and get me. :yawn2: :p: :D

[Mekoa]

You want me bad, according to the numbers you're producing at the moment.

Come and get me. :yawn2: :p: :D

Sounds like a challenge! 500mill to go lol.

[hixie]

Guys i won't be producing until 2nd of March. Have a little something do to in Taiwan :wasntme:
Hopefully, Frisch will be able to pass me before then, so i can overtake him again :rofl:

[Mekoa]

Guys i won't be producing until 2nd of March. Have a little something do to in Taiwan :wasntme:
Hopefully, Frisch will be able to pass me before then, so i can overtake him again :rofl:

2nd March? Eek thats next year! ;)

Dave said you were away until beginning of May I think. Sure rest of us can chip in and keep Free DC at bay!

[Frisch]

Mekoa;2942224']Sounds like a challenge! 500mill to go lol.
You too....dear God...My shell is too sexy !! :D


Guys i won't be producing until 2nd of March. Have a little something do to in Taiwan :wasntme:
Hopefully, Frisch will be able to pass me before then, so i can overtake him again :rofl:

2nd of May, i will be at around 815...so no chance for getting your ugly butt back where i would like it.......behind me...not in front, mooning me !!!!!

Oh...and guys....
LEAVE MY SEXY SHELL ALONE :D

[hixie]

Mekoa;2942347']2nd March? Eek thats next year! ;)

Dave said you were away until beginning of May I think. Sure rest of us can chip in and keep Free DC at bay!
LOL, i was supposed to say 2nd of May. Was busy at work so i didn't double check what i wrote :p: Had to build 3 machines in 3 hours.
I get back to HK on the 1st, but late at night. So i won't get my machine back up til the 2nd.


You too....dear God...My shell is too sexy !! :D



2nd of May, i will be at around 815...so no chance for getting your ugly butt back where i would like it.......behind me...not in front, mooning me !!!!!

Oh...and guys....
LEAVE MY SEXY SHELL ALONE :D
We bunnies love harassing turtles :rofl:

[Mekoa]

LOL, i was supposed to say 2nd of May. Was busy at work so i didn't double check what i wrote :p: Had to build 3 machines in 3 hours.
I get back to HK on the 1st, but late at night. So i won't get my machine back up til the 2nd.


We bunnies love harassing turtles :rofl:

At the speeds I crawl at, think my bunny is missing a few legs and an eye.

On a lighter note... interesting URL for a worker to hang on lol...

[[XC] riptide]

Must look up that site. :D

[Mekoa]

riptide;2942928']Must look up that site. :D

I did to see what it was... nothing 'jumps out' shall we say and most of it is in what I believe is Japanese.

Maybe my machine likes that kind of website? Stopped to pause for a look, lol.

[[XC] riptide]

Yes. its a Japanese dating site or something. LOL. Seems rather erotic also.! :D