Results 1 to 10 of 10

Thread: Took the plunge!

  1. #1
    Xtreme Legend
    Join Date
    Mar 2008
    Location
    Plymouth (UK)
    Posts
    5,285

    Took the plunge!

    Decided to run down to zero and delete the old MJ 12 completely.

    I've just loaded MJ12node v1.7.0 BETA 2

    http://www.majestic12.co.uk/files/mj...et2_v170b2.zip

    Now we must wait and see what the implications are....one thing though ...still doing a lot of re-crawl just now so I expect the numbers will drop through the floor


    My Biggest Fear Is When I die, My Wife Sells All My Stuff For What I Told Her I Paid For It.
    79 SB threads and 32 IB Threads across 4 rigs 111 threads Crunching!!

  2. #2
    Xtreme Enthusiast
    Join Date
    May 2007
    Posts
    648
    Can you post a screenshot of the Crawler tab of the Options menu? If I understand correctly reserved buckets are going to be severely limited in the new node Also, how often do you see NoMoreURLs in the log(if you do)?

  3. #3
    It is
    Join Date
    Feb 2005
    Location
    Copenhagen Denmark
    Posts
    3,058
    Quote Originally Posted by DeadlyFire View Post
    Can you post a screenshot of the Crawler tab of the Options menu? If I understand correctly reserved buckets are going to be severely limited in the new node Also, how often do you see NoMoreURLs in the log(if you do)?
    You won't have the option, it will be locked at 1. I had the feeling it would be coming in a version somewhere along the line...and this is it. But if there's no message about empty server, it's ok. BUT, what I don't like, is that it is set at 1, it should be something like 5, as sometimes the server is busy, and you will have a situation with your node screaming for work. This I predict will be a message that will appear a lot in the new node if set at 1.
    ~Opinion about Dogs~~~~~~~Hug~~~~~~~~
    ~~~~
    ________________________________________________

  4. #4
    Xtreme Legend
    Join Date
    Mar 2008
    Location
    Plymouth (UK)
    Posts
    5,285
    Wait till after I switch over to night crawling for the screenshot...there are, as has been said, no reserved buckets although the option is still there in options>crawler.

    MoMoreUrl's available comes up all the time but only for a secnd or two at the bottom of the screen


    My Biggest Fear Is When I die, My Wife Sells All My Stuff For What I Told Her I Paid For It.
    79 SB threads and 32 IB Threads across 4 rigs 111 threads Crunching!!

  5. #5
    Xtreme Enthusiast
    Join Date
    May 2007
    Posts
    648
    Quote Originally Posted by OldChap View Post
    Wait till after I switch over to night crawling for the screenshot...there are, as has been said, no reserved buckets although the option is still there in options>crawler.

    MoMoreUrl's available comes up all the time but only for a secnd or two at the bottom of the screen
    If possible can you post a screenshot of the bandwidth graph(under the Charts tab with the 1min period) on the old node vs the new node? I want to see if the lack of reserved buckets has any big impact on steady crawling.

  6. #6
    V3 Xeons coming soon!
    Join Date
    Nov 2005
    Location
    New Hampshire
    Posts
    33,457
    JMHO but I'm staying with the old version..
    Crunch with us, the XS WCG team
    The XS WCG team needs your support.
    A good project with good goals.
    Come join us,get that warm fuzzy feeling that you've done something good for mankind.

    Quote Originally Posted by Frisch View Post
    If you have lost faith in humanity, then hold a newborn in your hands.

  7. #7
    Xtreme Legend
    Join Date
    Mar 2008
    Location
    Plymouth (UK)
    Posts
    5,285
    I started an old instance on another machine but "noUrlsAvailable" so no comparisons here

    I don't think it is worth posting anything much yet as 90% is recrawl

    I have been enjoying a steady load of some 30Meg as shown by my router but just now it is up and down like a whores drawers....anything fron 2Meg to 35Meg (see attached) and I have had to change all my crawler settings upwards to accommodate the recrawl

    Dave has it right for now...... stay with the original at least till the make up of the buckets is more normal



    What you are looking at here is Night crawl @~30Meg, day crawl @~ 10Meg, some more Night followed by running down the remaining buckets then just after 12 I loaded up the new beta version...every picture tells a story huh?


    My Biggest Fear Is When I die, My Wife Sells All My Stuff For What I Told Her I Paid For It.
    79 SB threads and 32 IB Threads across 4 rigs 111 threads Crunching!!

  8. #8
    Xtreme Enthusiast
    Join Date
    May 2007
    Posts
    648
    Quote Originally Posted by OldChap View Post

    What you are looking at here is Night crawl @~30Meg, day crawl @~ 10Meg, some more Night followed by running down the remaining buckets then just after 12 I loaded up the new beta version...every picture tells a story huh?
    Is that last part of the chart running @30meg? If so steady crawling is gonna be hard to do without adding more nodes
    edit: spoke too fast, when the more normal buckets start appearing on the 1.7 node I'll see how it looks.

    Quote Originally Posted by Movieman View Post
    JMHO but I'm staying with the old version..
    Move over I'm coming with you Though since 1.6.12 requires the manual bucket adding process, I don't know if Alex will continue supporting it or if 1.7 will be mandatory if we want to see any more buckets
    Last edited by DeadlyFire; 12-05-2010 at 02:25 PM.

  9. #9
    Xtreme Legend
    Join Date
    Mar 2008
    Location
    Plymouth (UK)
    Posts
    5,285
    My one time 30meg overnight running is currently sitting @ ~5meg the single bucket at a time is starving me ...earlier I saw 13 buckets on one and 2 buckets on the other machine (normally 25 ish during the day) lots of rejected buckets due to the url's being duplicated in other buckets.

    Once the re-crawl is finished though this could be good although I think I would like to see everyone getting a 10 bucket cache minimum if only to deal with the re-crawling that is included on the new server. Some of these I'm getting now have been only 7url's in a bucket


    My Biggest Fear Is When I die, My Wife Sells All My Stuff For What I Told Her I Paid For It.
    79 SB threads and 32 IB Threads across 4 rigs 111 threads Crunching!!

  10. #10
    Xtreme Enthusiast
    Join Date
    May 2007
    Posts
    648
    Since the 1.6.12 node is starving half the time I said the hell with it and migrated all my nodes to the beta. Across 5 nodes I'm seeing a semi-steady 20-25mb(usually a steady 50) as long as there are buckets available. Can't wait to get out of these URL badlands

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •