Printable View
Decided to run down to zero and delete the old MJ 12 completely.
I've just loaded MJ12node v1.7.0 BETA 2
http://www.majestic12.co.uk/files/mj...et2_v170b2.zip
Now we must wait and see what the implications are....one thing though ...still doing a lot of re-crawl just now so I expect the numbers will drop through the floor
Can you post a screenshot of the Crawler tab of the Options menu? If I understand correctly reserved buckets are going to be severely limited in the new node :( Also, how often do you see NoMoreURLs in the log(if you do)?
You won't have the option, it will be locked at 1. I had the feeling it would be coming in a version somewhere along the line...and this is it. But if there's no message about empty server, it's ok. BUT, what I don't like, is that it is set at 1, it should be something like 5, as sometimes the server is busy, and you will have a situation with your node screaming for work. This I predict will be a message that will appear a lot in the new node if set at 1.Quote:
Originally Posted by DeadlyFire
Wait till after I switch over to night crawling for the screenshot...there are, as has been said, no reserved buckets although the option is still there in options>crawler.
MoMoreUrl's available comes up all the time but only for a secnd or two at the bottom of the screen
If possible can you post a screenshot of the bandwidth graph(under the Charts tab with the 1min period) on the old node vs the new node? I want to see if the lack of reserved buckets has any big impact on steady crawling.Quote:
Originally Posted by OldChap
JMHO but I'm staying with the old version..:D
I started an old instance on another machine but "noUrlsAvailable" so no comparisons here
I don't think it is worth posting anything much yet as 90% is recrawl
I have been enjoying a steady load of some 30Meg as shown by my router but just now it is up and down like a whores drawers....anything fron 2Meg to 35Meg (see attached) and I have had to change all my crawler settings upwards to accommodate the recrawl
Dave has it right for now...... stay with the original at least till the make up of the buckets is more normal
http://img528.imageshack.us/img528/1...dandnewrrd.jpg
What you are looking at here is Night crawl @~30Meg, day crawl @~ 10Meg, some more Night followed by running down the remaining buckets then just after 12 I loaded up the new beta version...every picture tells a story huh?
Is that last part of the chart running @30meg? :eek: If so steady crawling is gonna be hard to do without adding more nodes :cussing:Quote:
Originally Posted by OldChap
edit: spoke too fast, when the more normal buckets start appearing on the 1.7 node I'll see how it looks.
Move over I'm coming with you :D Though since 1.6.12 requires the manual bucket adding process, I don't know if Alex will continue supporting it or if 1.7 will be mandatory if we want to see any more buckets :shrug:Quote:
Originally Posted by Movieman
My one time 30meg overnight running is currently sitting @ ~5meg the single bucket at a time is starving me ...earlier I saw 13 buckets on one and 2 buckets on the other machine (normally 25 ish during the day) lots of rejected buckets due to the url's being duplicated in other buckets.
Once the re-crawl is finished though this could be good although I think I would like to see everyone getting a 10 bucket cache minimum if only to deal with the re-crawling that is included on the new server. Some of these I'm getting now have been only 7url's in a bucket
Since the 1.6.12 node is starving half the time I said the hell with it and migrated all my nodes to the beta. Across 5 nodes I'm seeing a semi-steady 20-25mb(usually a steady 50) as long as there are buckets available. Can't wait to get out of these URL badlands :brick: