one point five billion..

[DeadlyFire]

LOL. Well. Again it was not intentional. I suspect its because of having an improper MTU size and generally high error rate. Most of my crawling was done by wireless. Hence dropped packets, errors etc. Particularly timeout error.

However, you may consider that my comment above is not 100%. I don;t really expect anyone to purposely degrade the standard of their output just to get more points.



But it sure was a teaser. Now Scale up 93gigs : 12mil to Daves figures. Then work out what he could do per day on the new ratio JUst to put it in perspective, on the same rate since he started, Dav would have almost 6 Billion. And hit 20mil /day

Yes, scary but in a mischievous way I remember a thread on the MJ12 forum about a user pulling this intentionally, Alex banned his sorry a$$ fast But he came back and tried to pull it again lol A little puzzling since this is a voluntary project where you donate your bandwidth for a good cause and cheating helps no one(except the stats whore)

[Mekoa]

My conquests dont look quite as shiny as Dave's heh.

[Movieman]

LOL. Well. Again it was not intentional. I suspect its because of having an improper MTU size and generally high error rate. Most of my crawling was done by wireless. Hence dropped packets, errors etc. Particularly timeout error.

However, you may consider that my comment above is not 100%. I don;t really expect anyone to purposely degrade the standard of their output just to get more points.



But it sure was a teaser. Now Scale up 93gigs : 12mil to Daves figures. Then work out what he could do per day on the new ratio JUst to put it in perspective, on the same rate since he started, Dav would have almost 6 Billion. And hit 20mil /day

nah, I'd rather do 6 mil a day and give him good info than 20 mil a day and give him bad.
There was talk there about switching to just credit for "good" urls and I posted against changing the rules in the middle of the game and Alex agreed although it might have been just to placate some people.
BUT I wouldn't take advantage of that. I like to win but only if it's fair so I take what they send me and exclude nothing..
When you know your playing the game 100% honest it makes winning taste all the more better!

[[XC] riptide]

^^^ Couldn't agree more.

Just to show the state of the link I have. My max MTU size that I can use is 572. AS opposed to 1500 for most normal lines. The tel tale 'acid test' is that hotmail or bebo won't work on a unmodified TCP stack. Cos their servers don't do MTU discovery (I think its a precation against DDOS attacks).

AS to you guys with large pipes. Have you ever considered your settings? Particularly you Recieve Window? If its too low, a large line will suffer. You need a massive Recieve Window to fully use your FIOS dave?

[DeadlyFire]

AS to you guys with large pipes. Have you ever considered your settings? Particularly you Recieve Window? If its too low, a large line will suffer. You need a massive Recieve Window to fully use your FIOS dave?

How would you know what to set the window size to? It depends on the latency to the servers you most often communicate to and with MJ12 it jumps all over the place Maybe set the receive window to the worst case scenario and monitor the results? Anywho I'm to try this out

[Mekoa]

How would you know what to set the window size to? It depends on the latency to the servers you most often communicate to and with MJ12 it jumps all over the place Maybe set the receive window to the worst case scenario and monitor the results? Anywho I'm to try this out

I thought RWIN was determined by the speed of your connection and a multiple of MTU minus overhead?

[[XC] riptide]

Corrrect mostly on both accounts. You can set a very large RWIN. I do not think there will be a risk of too high. The point been that Default XP come with a RWIN set too low.

Now, On Vista I beleive there is dynamic RWIN adjustment. So on Vista your ok!

Also guys, you could try turn off TimeStamping as this add another 12bytes to each packet. This 12 bytes is not data but more signalling overhead, if you will. Timestamping I believe is used in corporate enterprise solutions mostly.

Guys, DL "TCPOptimizer". Its free. http://www.speedguide.net/downloads.php


Also run this test on your MJ machines. http://www.dslreports.com/tweaksTweak test.

[Mekoa]

I used TCP Optimizer on my XP machines, havent bothered with it on Vista as of yet. Most of the settings TCP Opt changes arent even present in Vista registry!

[[XC] riptide]

What size is your ideal MTU? And did you calculate your RWIN?

[Mekoa]

What size is your ideal MTU? And did you calculate your RWIN?

1500, no.

Sure RWIN on XP was 512000 ish. Cant remember the exact numbers. Cant remember exactly how its calculated either!

[DeadlyFire]

Using the optimal settings option in the optimizer I set the mtu and rwin to 1500 and 1,027,840. MJ12 doesn't seem to have had any impact but success are around 88-90% and hourly url's bumped up to 330k from 315k so it didn't hurt.

[Mekoa]

Screeny of my TCP Opt settings for current connection of 9mbit.

Need to run it on my 24/7 box. Hasnt been optimized!

[[XC] riptide]

Nice one deadlyly fire. Glad it made a bit of difference!

Now. For completion. Mekoa/Deadly. Go to Largest MTU, and run a test. Then post results here.

[Mekoa]

Im already on my 1500 MTU mate. Cant go any higher as itll fragment the packet

[[XC] riptide]

Im already on my 1500 MTU mate. Cant go any higher as itll fragment the packet

YES! 1500 is teh largest anyways!!!! Just run the test!

I know your Scottish and all, but stay with me on this.

[Mekoa]

YES! 1500 is teh largest anyways!!!! Just run the test!

I know your Scottish and all, but stay with me on this.

Hopefully be able to test it over an 8 hour period tonight (12pm - 8am). Ill restart MJ12 in 39 mins.

And btw, my MTU has always been set to 1500 :P Enforced by the router

[[XC] riptide]

Mekoa! Go to 'largest MTU' on TCPOptomiser. It takes like 2 seconds. Not eight hours. And do it while MJ12 is running.

It will TELL you what your recommended MTU SHOULD BE!

Most of the time it will be 1500/1472.

[Mekoa]

Mekoa! Go to 'largest MTU' on TCPOptomiser. It takes like 2 seconds. Not eight hours. And do it while MJ12 is running.

It will TELL you what your recommended MTU SHOULD BE!

Most of the time it will be 1500/1472.

I know how to use TCP Opt

As I said, its 1500(1472)-28 for ICMP and IP overheads. I meant I would test over a period of MJ12 for 8 hours after I ran TCP Optimizer to see if it made a difference.

Either Ive got a better batch of URLs or success rate is up (90.7%) doing approx 110k URLs an hour which is a wee bit above what it used to be.

My ISPs MTU is 1500.

[[XC] riptide]

LOL> Ah right. I get ya!

[Mekoa]

LOL> Ah right. I get ya!

Reason I started at midnight was for ease of hour counting lol. Still unsure if its just a better batch of URLs or tweaks made a bit of a difference.

PY222 is still almsot beating us singlehandedly WTB their connection!

[Mekoa]

One thing I have noticed recently, but not sure why its doing it, is that I seem to be doing less URLs for more MB. See below.

Edit: Something is definetly wrong. Moddolicious has 170mill more URLs crawled than me, but only 200Gb more downlaoded than me? Sorcerer is 50mill ahead but 1mill MB less?

Halp!

[[XC] riptide]

LOL. Lower error rates? You downloading big pages on each bucket.

[DeadlyFire]

PY222 is still almsot beating us singlehandedly WTB their connection!

I dunno, Hixie is down today yet our team is still managing to outpace PY222 by at least 1.5mil

[Frisch]

I dunno, Hixie is down today yet our team is still managing to outpace PY222 by at least 1.5mil

That's the strength of this team. You can take some days off, and still we're on top.

Imagine a day with all on full speed.

[Movieman]

I think Hixie is away from the 20th-30th in Taiwan so his numbers will be down for that time.
We can compensate if we all get on in force!