Decided to run down to zero and delete the old MJ 12 completely.
I've just loaded MJ12node v1.7.0 BETA 2
http://www.majestic12.co.uk/files/mj...et2_v170b2.zip
Now we must wait and see what the implications are....one thing though ...still doing a lot of re-crawl just now so I expect the numbers will drop through the floor
My Biggest Fear Is When I die, My Wife Sells All My Stuff For What I Told Her I Paid For It.79 SB threads and 32 IB Threads across 4 rigs 111 threads Crunching!!
Can you post a screenshot of the Crawler tab of the Options menu? If I understand correctly reserved buckets are going to be severely limited in the new nodeAlso, how often do you see NoMoreURLs in the log(if you do)?
You won't have the option, it will be locked at 1. I had the feeling it would be coming in a version somewhere along the line...and this is it. But if there's no message about empty server, it's ok. BUT, what I don't like, is that it is set at 1, it should be something like 5, as sometimes the server is busy, and you will have a situation with your node screaming for work. This I predict will be a message that will appear a lot in the new node if set at 1.Originally Posted by DeadlyFire
Can you post a screenshot of the Crawler tab of the Options menu? If I understand correctly reserved buckets are going to be severely limited in the new node
~Opinion about Dogs~~~~~~~Hug~~~~~~~~
~~~~
________________________________________________
Wait till after I switch over to night crawling for the screenshot...there are, as has been said, no reserved buckets although the option is still there in options>crawler.
MoMoreUrl's available comes up all the time but only for a secnd or two at the bottom of the screen
My Biggest Fear Is When I die, My Wife Sells All My Stuff For What I Told Her I Paid For It.79 SB threads and 32 IB Threads across 4 rigs 111 threads Crunching!!
If possible can you post a screenshot of the bandwidth graph(under the Charts tab with the 1min period) on the old node vs the new node? I want to see if the lack of reserved buckets has any big impact on steady crawling.
JMHO but I'm staying with the old version..
Crunch with us, the XS WCG team
The XS WCG team needs your support.
A good project with good goals.
Come join us,get that warm fuzzy feeling that you've done something good for mankind.
I started an old instance on another machine but "noUrlsAvailable" so no comparisons here
I don't think it is worth posting anything much yet as 90% is recrawl
I have been enjoying a steady load of some 30Meg as shown by my router but just now it is up and down like a whores drawers....anything fron 2Meg to 35Meg (see attached) and I have had to change all my crawler settings upwards to accommodate the recrawl
Dave has it right for now...... stay with the original at least till the make up of the buckets is more normal
What you are looking at here is Night crawl @~30Meg, day crawl @~ 10Meg, some more Night followed by running down the remaining buckets then just after 12 I loaded up the new beta version...every picture tells a story huh?
My Biggest Fear Is When I die, My Wife Sells All My Stuff For What I Told Her I Paid For It.79 SB threads and 32 IB Threads across 4 rigs 111 threads Crunching!!
Is that last part of the chart running @30meg?If so steady crawling is gonna be hard to do without adding more nodes
edit: spoke too fast, when the more normal buckets start appearing on the 1.7 node I'll see how it looks.
Move over I'm coming with youJMHO but I'm staying with the old version..
Last edited by DeadlyFire; 12-05-2010 at 03:25 PM.
My one time 30meg overnight running is currently sitting @ ~5meg the single bucket at a time is starving me ...earlier I saw 13 buckets on one and 2 buckets on the other machine (normally 25 ish during the day) lots of rejected buckets due to the url's being duplicated in other buckets.
Once the re-crawl is finished though this could be good although I think I would like to see everyone getting a 10 bucket cache minimum if only to deal with the re-crawling that is included on the new server. Some of these I'm getting now have been only 7url's in a bucket
My Biggest Fear Is When I die, My Wife Sells All My Stuff For What I Told Her I Paid For It.79 SB threads and 32 IB Threads across 4 rigs 111 threads Crunching!!
Since the 1.6.12 node is starving half the time I said the hell with it and migrated all my nodes to the beta. Across 5 nodes I'm seeing a semi-steady 20-25mb(usually a steady 50) as long as there are buckets available. Can't wait to get out of these URL badlands
Posting Permissions
Reply With Quote
Bookmarks