Starvation Mode???

[jspace]

Hello, guys. Sorry for wasting your time by posting this but I have a slight problem. MJ12 refuses to get more URLs. Heres the log:
[11/13/2006 10:37:59 PM] Start init node...
[11/13/2006 10:37:59 PM] Loading webserver config from: F:\mj12node_win32_v130\webserver.xml
[11/13/2006 10:37:59 PM] Init: trying to connect to database...
[11/13/2006 10:37:59 PM] Init: connected to database!
[11/13/2006 10:37:59 PM] Init: running base SQL...
[11/13/2006 10:37:59 PM] Init: running base SQL done!
[11/13/2006 10:37:59 PM] Init WebCrawler...
[11/13/2006 10:37:59 PM] HKom is DISABLED in this release
[11/13/2006 10:37:59 PM] Using COM call to change max-threads!
[11/13/2006 10:37:59 PM] Maximum workers: 200, set threadpool to: 200 (was: 200), IO threads: 1000 (was: 1000)
[11/13/2006 10:37:59 PM] Cleaning database
[11/13/2006 10:37:59 PM] Starting main threads
[11/13/2006 10:37:59 PM] MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+) started with 200 workers.
[11/13/2006 10:37:59 PM] done!
[11/13/2006 10:37:59 PM] Finished init node...
[11/13/2006 10:38:05 PM] Could not get URL via HTTP: http://www.majestic12.co.uk/projects/dsearch/mj12node/banlist.txt Error: System.TypeInitializationException: The type initializer for "System.Net.HttpWebRequest" threw an exception. ---> System.Configuration.ConfigurationException: Unrecognized attribute 'useUnsafeHeaderParsing' (F:\mj12node_win32_v130\MJ12node.exe.config line 7)
at System.Configuration.HandlerBase.CheckForUnrecogni zedAttributes(XmlNode node)
at System.Net.Configuration.NetConfigurationHandler.C reate(Object parent, Object configContext, XmlNode section)
at System.Configuration.ConfigurationRecord.EvaluateR ecursive(IConfigurationSectionHandler factory, Object config, String[] keys, Int32 iKey, XmlTextReader reader)
at System.Configuration.ConfigurationRecord.EvaluateR ecursive(IConfigurationSectionHandler factory, Object config, String[] keys, Int32 iKey, XmlTextReader reader)
at System.Configuration.ConfigurationRecord.Evaluate( String configKey)
at System.Configuration.ConfigurationRecord.ResolveCo nfig(String configKey)
at System.Configuration.ConfigurationRecord.GetConfig (String configKey)
at System.Configuration.DefaultConfigurationSystem.Sy stem.Configuration.IConfigurationSystem.GetConfig( String configKey)
at System.Configuration.ConfigurationSettings.GetConf ig(String sectionName)
at System.Net.HttpWebRequest.GetDefaultMaximumRespons eHeadersLength()
at System.Net.HttpWebRequest..cctor()
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest..ctor(Uri uri)
at System.Net.HttpRequestCreator.Create(Uri Uri)
at System.Net.WebRequest.Create(Uri requestUri, Boolean useUriBase)
at System.Net.WebRequest.Create(String requestUriString)
at Majestic12.Utils.GetURL(String sURL, String p_sTargetFileName, String sUserAgent, Boolean bWriteAsBarrel)
[11/13/2006 10:38:05 PM] Failed to get fresh ban list: System.Exception: Could not get file via HTTP! ---> System.TypeInitializationException: The type initializer for "System.Net.HttpWebRequest" threw an exception. ---> System.Configuration.ConfigurationException: Unrecognized attribute 'useUnsafeHeaderParsing' (F:\mj12node_win32_v130\MJ12node.exe.config line 7)
at System.Configuration.HandlerBase.CheckForUnrecogni zedAttributes(XmlNode node)
at System.Net.Configuration.NetConfigurationHandler.C reate(Object parent, Object configContext, XmlNode section)
at System.Configuration.ConfigurationRecord.EvaluateR ecursive(IConfigurationSectionHandler factory, Object config, String[] keys, Int32 iKey, XmlTextReader reader)
at System.Configuration.ConfigurationRecord.EvaluateR ecursive(IConfigurationSectionHandler factory, Object config, String[] keys, Int32 iKey, XmlTextReader reader)
at System.Configuration.ConfigurationRecord.Evaluate( String configKey)
at System.Configuration.ConfigurationRecord.ResolveCo nfig(String configKey)
at System.Configuration.ConfigurationRecord.GetConfig (String configKey)
at System.Configuration.DefaultConfigurationSystem.Sy stem.Configuration.IConfigurationSystem.GetConfig( String configKey)
at System.Configuration.ConfigurationSettings.GetConf ig(String sectionName)
at System.Net.HttpWebRequest.GetDefaultMaximumRespons eHeadersLength()
at System.Net.HttpWebRequest..cctor()
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest..ctor(Uri uri)
at System.Net.HttpRequestCreator.Create(Uri Uri)
at System.Net.WebRequest.Create(Uri requestUri, Boolean useUriBase)
at System.Net.WebRequest.Create(String requestUriString)
at Majestic12.Utils.GetURL(String sURL, String p_sTargetFileName, String sUserAgent, Boolean bWriteAsBarrel)
--- End of inner exception stack trace ---
at Majestic12.Utils.GetURL(String sURL, String p_sTargetFileName, String sUserAgent, Boolean bWriteAsBarrel)
at Majestic12.Utils.GetURL(String sURL, String p_sTargetFileName, String sUserAgent)
at Majestic12.WebCrawler.UpdateBanList()
[11/13/2006 10:38:11 PM] Got response: Busy try later
11/13/2006 10:39:11 PM] We are in starvation mode but we will try again since 60 seconds elapsed since last check: 11/13/2006 10:39:11 PM - 11/13/2006 10:38:09 PM
[11/13/2006 10:40:12 PM] We are in starvation mode but we will try again since 60 seconds elapsed since last check: 11/13/2006 10:40:12 PM - 11/13/2006 10:39:11 PM
[11/13/2006 10:40:30 PM] Could not get URL via HTTP: http://majestic12.kicks-ass.org/data/urls/prepared/F5FD790CB5AC607955EE7F30A85BE284.gz Error: System.TypeInitializationException: The type initializer for "System.Net.HttpWebRequest" threw an exception. ---> System.Configuration.ConfigurationException: Unrecognized attribute 'useUnsafeHeaderParsing' (F:\mj12node_win32_v130\MJ12node.exe.config line 7)
at System.Configuration.HandlerBase.CheckForUnrecogni zedAttributes(XmlNode node)
at System.Net.Configuration.NetConfigurationHandler.C reate(Object parent, Object configContext, XmlNode section)
at System.Configuration.ConfigurationRecord.EvaluateR ecursive(IConfigurationSectionHandler factory, Object config, String[] keys, Int32 iKey, XmlTextReader reader)
at System.Configuration.ConfigurationRecord.EvaluateR ecursive(IConfigurationSectionHandler factory, Object config, String[] keys, Int32 iKey, XmlTextReader reader)
at System.Configuration.ConfigurationRecord.Evaluate( String configKey)
at System.Configuration.ConfigurationRecord.ResolveCo nfig(String configKey)
at System.Configuration.ConfigurationRecord.GetConfig (String configKey)
at System.Configuration.DefaultConfigurationSystem.Sy stem.Configuration.IConfigurationSystem.GetConfig( String configKey)
at System.Configuration.ConfigurationSettings.GetConf ig(String sectionName)
at System.Net.HttpWebRequest.GetDefaultMaximumRespons eHeadersLength()
at System.Net.HttpWebRequest..cctor()
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest..ctor(Uri uri)
at System.Net.HttpRequestCreator.Create(Uri Uri)
at System.Net.WebRequest.Create(Uri requestUri, Boolean useUriBase)
at System.Net.WebRequest.Create(String requestUriString)
at Majestic12.Utils.GetURL(String sURL, String p_sTargetFileName, String sUserAgent, Boolean bWriteAsBarrel)
[11/13/2006 10:40:30 PM] Failed to get new URLs: System.Exception: Could not get file via HTTP! ---> System.TypeInitializationException: The type initializer for "System.Net.HttpWebRequest" threw an exception. ---> System.Configuration.ConfigurationException: Unrecognized attribute 'useUnsafeHeaderParsing' (F:\mj12node_win32_v130\MJ12node.exe.config line 7)
at System.Configuration.HandlerBase.CheckForUnrecogni zedAttributes(XmlNode node)
at System.Net.Configuration.NetConfigurationHandler.C reate(Object parent, Object configContext, XmlNode section)
at System.Configuration.ConfigurationRecord.EvaluateR ecursive(IConfigurationSectionHandler factory, Object config, String[] keys, Int32 iKey, XmlTextReader reader)
at System.Configuration.ConfigurationRecord.EvaluateR ecursive(IConfigurationSectionHandler factory, Object config, String[] keys, Int32 iKey, XmlTextReader reader)
at System.Configuration.ConfigurationRecord.Evaluate( String configKey)
at System.Configuration.ConfigurationRecord.ResolveCo nfig(String configKey)
at System.Configuration.ConfigurationRecord.GetConfig (String configKey)
at System.Configuration.DefaultConfigurationSystem.Sy stem.Configuration.IConfigurationSystem.GetConfig( String configKey)
at System.Configuration.ConfigurationSettings.GetConf ig(String sectionName)
at System.Net.HttpWebRequest.GetDefaultMaximumRespons eHeadersLength()
at System.Net.HttpWebRequest..cctor()
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest..ctor(Uri uri)
at System.Net.HttpRequestCreator.Create(Uri Uri)
at System.Net.WebRequest.Create(Uri requestUri, Boolean useUriBase)
at System.Net.WebRequest.Create(String requestUriString)
at Majestic12.Utils.GetURL(String sURL, String p_sTargetFileName, String sUserAgent, Boolean bWriteAsBarrel)
--- End of inner exception stack trace ---
at Majestic12.Utils.GetURL(String sURL, String p_sTargetFileName, String sUserAgent, Boolean bWriteAsBarrel)
at Majestic12.Utils.GetURL(String sURL, String p_sTargetFileName, String sUserAgent)
at Majestic12.WebCrawler.CheckForMoreURLs()
[11/13/2006 10:41:13 PM] We are in starvation mode but we will try again since 60 seconds elapsed since last check: 11/13/2006 10:41:13 PM - 11/13/2006 10:40:12 PM
[11/13/2006 10:42:15 PM] We are in starvation mode but we will try again since 60 seconds elapsed since last check: 11/13/2006 10:42:15 PM - 11/13/2006 10:41:13 PM

Not really knowing much about how MJ12 works, I need your help to start crawling again. (For the team of course;))

[Frisch]

Starvation mode means that either you are not able to contact the server, asking for more url's, or the most commen, you have to many buckets with a low number of domains.
Starvation mode is then activated, and it will contact the server for new buckets with a high number of domains to crawl.

The only solution I can give you, is to try and open option and try some other country domains, if the server is short on your choice (that's if you have taken one)

[jspace]

I didn't have any selected other than that checkbox at the bottom. I checked all domains and it still hasn't gotten anything so far. Still says same errors.

Meh, I'm getting too tired to think. *sleeps*

[[XC] moddolicous]

What cleint are you using? Can you post a pic of the activity screen?

[Mc Intosh]

Try setting max. open buckets to a higher number.

[mad mikee]

I put MJ-12 on a 3g gc-ramdisk and let it rip.

Did 175k in 45 min, then suddenly starts unloading all my buckets (14-16) and then has the gall to say 'in starvation mode' :soap:
Shut it down, move it to my raptor, start it back up and all is fine, but SLOWER :brick: :shrug:

Going to see if it is the OS (putting it on linux tonite)...

[Vapor]

Mine was into starvation as well when I returned to my room....