Well I setup MJ12 last night and got it crankin'...
Woke up this morning (about 7.5 hours later) and between my two comps I've crawled 733,000 URL's. Is that good, bad, or indifferent?
Well I setup MJ12 last night and got it crankin'...
Woke up this morning (about 7.5 hours later) and between my two comps I've crawled 733,000 URL's. Is that good, bad, or indifferent?
That's estimated about 2,4 mill a day.
There's a number of ifs here, where you are crawling (domains), and how much you allow the line to use, amount of workers, and open url's.
This is a fidling program.
~Opinion about Dogs~~~~~~~Hug~~~~~~~~
~~~~
________________________________________________
See I don't have anything advanced setup... Both boxes allow up to 100 workers each and 20 URL buckets.
My primary box is able to use 24mbit down and 2mbit up, my Dell box is able to use 18mbit down and 1.5mbit up. I have noticed however that I never see either one using more than about 8mbit, and my average is about 2.5mbit on both boxes. Is this just due to the servers being in the UK and me being in the US?
2.4mil a good number?
No, not if you are running those lines, didn't know you had that at your disposal.
FragTek I have a meeting in five, movieman, or whoever is online will guide you, and if not i'll be back later.
But one thing, it has nothing to do with the servers, cause you are crawling the net, the only thing you have to do with the servers, is getting url's, and upload them back.
You'll never hit your top, the average will always stay somewhere below the top, but try to set your domains to US only, and nearby. Under tools you can select preferred domains.
Last edited by Frisch; 03-23-2007 at 06:20 AM.
~Opinion about Dogs~~~~~~~Hug~~~~~~~~
~~
________________________________________________
Yeah I currently have a 30/2.5 line and in about a week or so I'll have 60/5 at my disposal so I need to get this setup to utilize as much of that as possible.
When I get home I'll try setting my domains to US only and see if that helps out a bit.
Is there a "recommended" or known good number of workers and buckets?
Thx allGlad to be of service!
FYI: I managed to do ~1 mil/day out of a 4/1 line. MJ12 is a lot of tweaking/testing, but I got crapped by my ISP...
Main rig:
CPU: I7 920C0 @ 3.6Ghz (180*20)
Mobo: DFI UT X58 T3eH8
RAM: 12GB OCZ DDR3-1600 Platinum
GPU/LCD: GeForce GTX280 + GeForce 8600GTS (Quad LCDs)
Intel X25-M G2 80GB, 12TB storage
PSU/Case: Corsair AX850, Silverstone TJ07
What are the main things I should be tweaking to try and get the most out of it?
FIRST we need to get you to join the team!Originally Posted by FragTek
Your cranking those numbers and not on any team from what I see at FreeDC stats page!
I saw you put up almost 3 mil today..nice start!
Crunch with us, the XS WCG team
The XS WCG team needs your support.
A good project with good goals.
Come join us,get that warm fuzzy feeling that you've done something good for mankind.
How do I join the team?!?!?FIRST we need to get you to join the team!
Your cranking those numbers and not on any team from what I see at FreeDC stats page!
I saw you put up almost 3 mil today..nice start!I couldn't figure that out either, yes, i'm a re-re.
3 mil, not to bad I guess... I can't seem to get my main box to average any higher than about 5mbit downstream though
Last edited by Frisch; 04-18-2007 at 12:15 PM.
~Opinion about Dogs~~~~~~~Hug~~~~~~~~
~~
________________________________________________
Thank you oh Turtle from on high!( or is that turtle thats always high?)
Crunch with us, the XS WCG team
The XS WCG team needs your support.
A good project with good goals.
Come join us,get that warm fuzzy feeling that you've done something good for mankind.
To fully understand the soul of a Turtle, you need to realize the slowness, of movement in real time.Thank you oh Turtle from on high!( or is that turtle thats always high?)
It's thoughts are as a weapon designed to distract the real time, by letting it's awareness be way ahead of it's physical presents, you could misunderstand this as being high, in the terms of it reacting ahead of the actually event.
Is this making any sense for the normal mind, no, not at all, and that's why.........
It's a killer tank on four extremely sweet feet.
Last edited by Frisch; 03-23-2007 at 07:39 AM.
~Opinion about Dogs~~~~~~~Hug~~~~~~~~
~~
________________________________________________
as if I can actually make sense of this drivel after being up all night?To fully understand the soul of a Turtle, you need to realize the slowness, of movement in real time.
It's thoughts are as a weapon designed to distract the real time, by letting it's awareness be way ahead of it's physical presents, you could misunderstand this as being high, in the terms of it reacting ahead of the actually event.
Is this making any sense for the normal mind, no, not at all, and that's way.........
It's a killer tank on four extremely sweet feet.
Be kind and remember that at my age brain cells need lots of coffee to understand turtlespeak..
Crunch with us, the XS WCG team
The XS WCG team needs your support.
A good project with good goals.
Come join us,get that warm fuzzy feeling that you've done something good for mankind.
The drivel, is what make us superior.
PS
You quoted my misspelling you evil man, it's really hard to grab a pen, not to talk about writing, being a Turtle.
~Opinion about Dogs~~~~~~~Hug~~~~~~~~
~~
________________________________________________
OK I think I'm in! Someone double check me and make sure
Now to see how I can get more than 3mbps average out of MJ12....
Now we need you back on MJ12..Yea, we're relentless!OK I think I'm in! Someone double check me and make sure
Now to see how I can get more than 3mbps average out of MJ12....
Did I see you say you had a 60/5 line coming?
You on FIOS also?
You don't even want to know what that line would do.
Maybe 15 mil a day.
Crunch with us, the XS WCG team
The XS WCG team needs your support.
A good project with good goals.
Come join us,get that warm fuzzy feeling that you've done something good for mankind.
McIntosh managed 15m/day from 20MbNow we need you back on MJ12..Yea, we're relentless!
Did I see you say you had a 60/5 line coming?
You on FIOS also?
You don't even want to know what that line would do.
Maybe 15 mil a day.
You should triple it
Main rig:
CPU: I7 920C0 @ 3.6Ghz (180*20)
Mobo: DFI UT X58 T3eH8
RAM: 12GB OCZ DDR3-1600 Platinum
GPU/LCD: GeForce GTX280 + GeForce 8600GTS (Quad LCDs)
Intel X25-M G2 80GB, 12TB storage
PSU/Case: Corsair AX850, Silverstone TJ07
Posting Permissions
Reply With Quote
Bookmarks