And these buckets are still very much trash _mcintosh_, please don't fret.
When we get a literally "fresh" batch of URLs to crawl then you should see much higher speeds. I've had 3 good buckets open at once and no more than that and was topping out my 4mbps connection easily but today I've been sitting on 7 or 8 open buckets and about 2mbps used. As it sits right now I've been crawling for the past hour at around 500kbps.
Just tough it through these buckets and then if that doesn't end up helping we can go through other venues such as making sure your TCPIP.CONFIG file is set right and what kinda of router you have and even down to what NIC cards you have.
-Jeff
Bookmarks