Just joined up!

[CrazyJerry]

Movieman urged me to join up with the addition of me new internet connection.


http://www.speedtest.net/result/1043692486.png

http://www.speedtest.net/result/1059591355.png

I have been reading through the posts and went through the tweaks. Is this beneficial to run this on multiple workstations?

[DeadlyFire]

Welcome Jerry!! :welcome: Yes running many nodes helps to saturate your line and crawl more URLs. Is that 100mbit line uncapped? :eek::eek: *starts calculating days til I'm steamrolled:p:

[CrazyJerry]

Welcome Jerry!! :welcome: Yes running many nodes helps to saturate your line and crawl more URLs. Is that 100mbit line uncapped? :eek::eek: *starts calculating days til I'm steamrolled:p:

Yes I was told it was uncapped. No one in the area is currently on the same segment and was told it was not "regulated" yet. Just hope all of my activity doesn't arouse suspicion. :-)

[DeadlyFire]

Yes I was told it was uncapped. No one in the area is currently on the same segment and was told it was not "regulated" yet. Just hope all of my activity doesn't arouse suspicion. :-)

Very cool :up: If you need help or suggestions on the client settings let us know!

[OldChap]

I reckon you can do some damage with that connection even if you just run on it overnight I'm thinking you could overtake me in weeks few.

Glad to have you on board :up:

[CrazyJerry]

Very cool :up: If you need help or suggestions on the client settings let us know!

I see some of your downstreams are 12,812kbs. How can I achieve speeds like this? Are you only crawling specific domains?

I used the basic sticky guide i saw on the main page. Could you recommend any settings for my connection type?

[CrazyJerry]

I reckon you can do some damage with that connection even if you just run on it overnight I'm thinking you could overtake me in weeks few.

Glad to have you on board :up:

I have one goal to start off. This is to run over Movieman :rofl:

I will be going live with this connection after the first of the year for all of my users. We are currently utilizing 2 t'1s for internet bandwidth. The users will be happy with 20mb and till think it is blazing fast. Just need to setup some scheduling with my offsite backups and i'll be good to utilize this connection to its full potential. Work does come before play...

[DeadlyFire]

I see some of your downstreams are 12,812kbs. How can I achieve speeds like this? Are you only crawling specific domains?

I used the basic sticky guide i saw on the main page. Could you recommend any settings for my connection type?

Are you going by the stats page on the Majestic 12 site? Those speed readings aren't entirely accurate because they take a 'snapshot' of the speed at a point in time and speeds within the node bounce all over the place(generally not steady). Though the bottom speed shows a rough approximate speed of the user's connection(my stat page (http://www.majestic12.co.uk/projects/dsearch/userinfo.php?id=2038) shows 43mb currently, I have a 50mb line). I have .ca .org .us and .com currently set as my domains. Post a screenshot of the 'crawler' tab under Tools>Options in the client and we can check your settings :up:


I have one goal to start off. This is to run over Movieman :rofl:


Dave might regret recruiting you after hearing that :D He's already got me hounding after him, let alone 2 of us :p:

[lkiller123]

Since you have a fast connection, you can set the async worker tab to "999" and the amount of open buckets top "333." Those values are all under the crawler tab.:up:

[CrazyJerry]

Are you going by the stats page on the Majestic 12 site? Those speed readings aren't entirely accurate because they take a 'snapshot' of the speed at a point in time and speeds within the node bounce all over the place(generally not steady). Though the bottom speed shows a rough approximate speed of the user's connection(my stat page (http://www.majestic12.co.uk/projects/dsearch/userinfo.php?id=2038) shows 43mb currently, I have a 50mb line). I have .ca .org .us and .com currently set as my domains. Post a screenshot of the 'crawler' tab under Tools>Options in the client and we can check your settings :up:

Screeny Attached bellow!



Dave might regret recruiting you after hearing that :D He's already got me hounding after him, let alone 2 of us :p:
I sent Dave a PM of my speed tests he told me about MJ12 I signed up. But it's time for some friendly competition. I am setting up a few rigs at other sites to run at night. They all have a minimum of 30mb down. I may call and see if we can get in on better promotional deals. Just saying if I get a good deal whats more bandwidth? :D

[lkiller123]

You definitely will need more async workers and buckets.

Check my post above, you would need to set 999 async workers and 333 open buckets. You might need to install MJ12 to more computers since 999 is the max you can go, that means each rig will crawl at about 12mbit/s

[DeadlyFire]

lkiller is right. Try putting your number of workers to greater than 350 and number of open URL buckets to higher than 300. You can set both even higher but I tested running only 1 node on my connection and after 300-350 or so workers the gains are very minimal if any. What kind of router do you have? A connection that big will swamp the NAT table on most consumer routers and force you to reset them every couple hours. A linux firewall/routing solution is probably the most powerful thing you can use next to a multi-thousand $ Cisco router.

[CrazyJerry]

A connection that big will swamp the NAT table on most consumer routers and force you to reset them every couple hours. A linux firewall/routing solution is probably the most powerful thing you can use next to a multi-thousand $ Cisco router.

For the time being I am using a Cisco 871 and am looking to demo a Sonicwall NSA.