Starvation Mode???

[jspace]

Hello, guys. Sorry for wasting your time by posting this but I have a slight problem. MJ12 refuses to get more URLs. Heres the log:

[11/13/2006 10:37:59 PM] Start init node...
[11/13/2006 10:37:59 PM] Loading webserver config from: F:\mj12node_win32_v130\webserver.xml
[11/13/2006 10:37:59 PM] Init: trying to connect to database...
[11/13/2006 10:37:59 PM] Init: connected to database!
[11/13/2006 10:37:59 PM] Init: running base SQL...
[11/13/2006 10:37:59 PM] Init: running base SQL done!
[11/13/2006 10:37:59 PM] Init WebCrawler...
[11/13/2006 10:37:59 PM] HKom is DISABLED in this release
[11/13/2006 10:37:59 PM] Using COM call to change max-threads!
[11/13/2006 10:37:59 PM] Maximum workers: 200, set threadpool to: 200 (was: 200), IO threads: 1000 (was: 1000)
[11/13/2006 10:37:59 PM] Cleaning database
[11/13/2006 10:37:59 PM] Starting main threads
[11/13/2006 10:37:59 PM] MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+) started with 200 workers.
[11/13/2006 10:37:59 PM] done!
[11/13/2006 10:37:59 PM] Finished init node...
[11/13/2006 10:38:05 PM] Could not get URL via HTTP: http://www.majestic12.co.uk/projects...de/banlist.txt Error: System.TypeInitializationException: The type initializer for "System.Net.HttpWebRequest" threw an exception. ---> System.Configuration.ConfigurationException: Unrecognized attribute 'useUnsafeHeaderParsing' (F:\mj12node_win32_v130\MJ12node.exe.config line 7)
at System.Configuration.HandlerBase.CheckForUnrecogni zedAttributes(XmlNode node)
at System.Net.Configuration.NetConfigurationHandler.C reate(Object parent, Object configContext, XmlNode section)
at System.Configuration.ConfigurationRecord.EvaluateR ecursive(IConfigurationSectionHandler factory, Object config, String[] keys, Int32 iKey, XmlTextReader reader)
at System.Configuration.ConfigurationRecord.EvaluateR ecursive(IConfigurationSectionHandler factory, Object config, String[] keys, Int32 iKey, XmlTextReader reader)
at System.Configuration.ConfigurationRecord.Evaluate( String configKey)
at System.Configuration.ConfigurationRecord.ResolveCo nfig(String configKey)
at System.Configuration.ConfigurationRecord.GetConfig (String configKey)
at System.Configuration.DefaultConfigurationSystem.Sy stem.Configuration.IConfigurationSystem.GetConfig( String configKey)
at System.Configuration.ConfigurationSettings.GetConf ig(String sectionName)
at System.Net.HttpWebRequest.GetDefaultMaximumRespons eHeadersLength()
at System.Net.HttpWebRequest..cctor()
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest..ctor(Uri uri)
at System.Net.HttpRequestCreator.Create(Uri Uri)
at System.Net.WebRequest.Create(Uri requestUri, Boolean useUriBase)
at System.Net.WebRequest.Create(String requestUriString)
at Majestic12.Utils.GetURL(String sURL, String p_sTargetFileName, String sUserAgent, Boolean bWriteAsBarrel)
[11/13/2006 10:38:05 PM] Failed to get fresh ban list: System.Exception: Could not get file via HTTP! ---> System.TypeInitializationException: The type initializer for "System.Net.HttpWebRequest" threw an exception. ---> System.Configuration.ConfigurationException: Unrecognized attribute 'useUnsafeHeaderParsing' (F:\mj12node_win32_v130\MJ12node.exe.config line 7)
at System.Configuration.HandlerBase.CheckForUnrecogni zedAttributes(XmlNode node)
at System.Net.Configuration.NetConfigurationHandler.C reate(Object parent, Object configContext, XmlNode section)
at System.Configuration.ConfigurationRecord.EvaluateR ecursive(IConfigurationSectionHandler factory, Object config, String[] keys, Int32 iKey, XmlTextReader reader)
at System.Configuration.ConfigurationRecord.EvaluateR ecursive(IConfigurationSectionHandler factory, Object config, String[] keys, Int32 iKey, XmlTextReader reader)
at System.Configuration.ConfigurationRecord.Evaluate( String configKey)
at System.Configuration.ConfigurationRecord.ResolveCo nfig(String configKey)
at System.Configuration.ConfigurationRecord.GetConfig (String configKey)
at System.Configuration.DefaultConfigurationSystem.Sy stem.Configuration.IConfigurationSystem.GetConfig( String configKey)
at System.Configuration.ConfigurationSettings.GetConf ig(String sectionName)
at System.Net.HttpWebRequest.GetDefaultMaximumRespons eHeadersLength()
at System.Net.HttpWebRequest..cctor()
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest..ctor(Uri uri)
at System.Net.HttpRequestCreator.Create(Uri Uri)
at System.Net.WebRequest.Create(Uri requestUri, Boolean useUriBase)
at System.Net.WebRequest.Create(String requestUriString)
at Majestic12.Utils.GetURL(String sURL, String p_sTargetFileName, String sUserAgent, Boolean bWriteAsBarrel)
--- End of inner exception stack trace ---
at Majestic12.Utils.GetURL(String sURL, String p_sTargetFileName, String sUserAgent, Boolean bWriteAsBarrel)
at Majestic12.Utils.GetURL(String sURL, String p_sTargetFileName, String sUserAgent)
at Majestic12.WebCrawler.UpdateBanList()
[11/13/2006 10:38:11 PM] Got response: Busy try later
11/13/2006 10:39:11 PM] We are in starvation mode but we will try again since 60 seconds elapsed since last check: 11/13/2006 10:39:11 PM - 11/13/2006 10:38:09 PM
[11/13/2006 10:40:12 PM] We are in starvation mode but we will try again since 60 seconds elapsed since last check: 11/13/2006 10:40:12 PM - 11/13/2006 10:39:11 PM
[11/13/2006 10:40:30 PM] Could not get URL via HTTP: http://majestic12.kicks-ass.org/data...F30A85BE284.gz Error: System.TypeInitializationException: The type initializer for "System.Net.HttpWebRequest" threw an exception. ---> System.Configuration.ConfigurationException: Unrecognized attribute 'useUnsafeHeaderParsing' (F:\mj12node_win32_v130\MJ12node.exe.config line 7)
at System.Configuration.HandlerBase.CheckForUnrecogni zedAttributes(XmlNode node)
at System.Net.Configuration.NetConfigurationHandler.C reate(Object parent, Object configContext, XmlNode section)
at System.Configuration.ConfigurationRecord.EvaluateR ecursive(IConfigurationSectionHandler factory, Object config, String[] keys, Int32 iKey, XmlTextReader reader)
at System.Configuration.ConfigurationRecord.EvaluateR ecursive(IConfigurationSectionHandler factory, Object config, String[] keys, Int32 iKey, XmlTextReader reader)
at System.Configuration.ConfigurationRecord.Evaluate( String configKey)
at System.Configuration.ConfigurationRecord.ResolveCo nfig(String configKey)
at System.Configuration.ConfigurationRecord.GetConfig (String configKey)
at System.Configuration.DefaultConfigurationSystem.Sy stem.Configuration.IConfigurationSystem.GetConfig( String configKey)
at System.Configuration.ConfigurationSettings.GetConf ig(String sectionName)
at System.Net.HttpWebRequest.GetDefaultMaximumRespons eHeadersLength()
at System.Net.HttpWebRequest..cctor()
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest..ctor(Uri uri)
at System.Net.HttpRequestCreator.Create(Uri Uri)
at System.Net.WebRequest.Create(Uri requestUri, Boolean useUriBase)
at System.Net.WebRequest.Create(String requestUriString)
at Majestic12.Utils.GetURL(String sURL, String p_sTargetFileName, String sUserAgent, Boolean bWriteAsBarrel)
[11/13/2006 10:40:30 PM] Failed to get new URLs: System.Exception: Could not get file via HTTP! ---> System.TypeInitializationException: The type initializer for "System.Net.HttpWebRequest" threw an exception. ---> System.Configuration.ConfigurationException: Unrecognized attribute 'useUnsafeHeaderParsing' (F:\mj12node_win32_v130\MJ12node.exe.config line 7)
at System.Configuration.HandlerBase.CheckForUnrecogni zedAttributes(XmlNode node)
at System.Net.Configuration.NetConfigurationHandler.C reate(Object parent, Object configContext, XmlNode section)
at System.Configuration.ConfigurationRecord.EvaluateR ecursive(IConfigurationSectionHandler factory, Object config, String[] keys, Int32 iKey, XmlTextReader reader)
at System.Configuration.ConfigurationRecord.EvaluateR ecursive(IConfigurationSectionHandler factory, Object config, String[] keys, Int32 iKey, XmlTextReader reader)
at System.Configuration.ConfigurationRecord.Evaluate( String configKey)
at System.Configuration.ConfigurationRecord.ResolveCo nfig(String configKey)
at System.Configuration.ConfigurationRecord.GetConfig (String configKey)
at System.Configuration.DefaultConfigurationSystem.Sy stem.Configuration.IConfigurationSystem.GetConfig( String configKey)
at System.Configuration.ConfigurationSettings.GetConf ig(String sectionName)
at System.Net.HttpWebRequest.GetDefaultMaximumRespons eHeadersLength()
at System.Net.HttpWebRequest..cctor()
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest..ctor(Uri uri)
at System.Net.HttpRequestCreator.Create(Uri Uri)
at System.Net.WebRequest.Create(Uri requestUri, Boolean useUriBase)
at System.Net.WebRequest.Create(String requestUriString)
at Majestic12.Utils.GetURL(String sURL, String p_sTargetFileName, String sUserAgent, Boolean bWriteAsBarrel)
--- End of inner exception stack trace ---
at Majestic12.Utils.GetURL(String sURL, String p_sTargetFileName, String sUserAgent, Boolean bWriteAsBarrel)
at Majestic12.Utils.GetURL(String sURL, String p_sTargetFileName, String sUserAgent)
at Majestic12.WebCrawler.CheckForMoreURLs()
[11/13/2006 10:41:13 PM] We are in starvation mode but we will try again since 60 seconds elapsed since last check: 11/13/2006 10:41:13 PM - 11/13/2006 10:40:12 PM
[11/13/2006 10:42:15 PM] We are in starvation mode but we will try again since 60 seconds elapsed since last check: 11/13/2006 10:42:15 PM - 11/13/2006 10:41:13 PM

Not really knowing much about how MJ12 works, I need your help to start crawling again. (For the team of course)

[Frisch]

Starvation mode means that either you are not able to contact the server, asking for more url's, or the most commen, you have to many buckets with a low number of domains.
Starvation mode is then activated, and it will contact the server for new buckets with a high number of domains to crawl.

The only solution I can give you, is to try and open option and try some other country domains, if the server is short on your choice (that's if you have taken one)

[jspace]

I didn't have any selected other than that checkbox at the bottom. I checked all domains and it still hasn't gotten anything so far. Still says same errors.

Meh, I'm getting too tired to think. *sleeps*

[[XC] moddolicous]

What cleint are you using? Can you post a pic of the activity screen?

[Mc Intosh]

Try setting max. open buckets to a higher number.

[mad mikee]

I put MJ-12 on a 3g gc-ramdisk and let it rip.

Did 175k in 45 min, then suddenly starts unloading all my buckets (14-16) and then has the gall to say 'in starvation mode'
Shut it down, move it to my raptor, start it back up and all is fine, but SLOWER

Going to see if it is the OS (putting it on linux tonite)...

[Vapor]

Mine was into starvation as well when I returned to my room....