Hey XS we got a Majestic 12 team !

[[XC] moddolicous]

Haha, I forgot how much fun majestic12 is (I'll be making random appearances until I'm home in late June). Going over the last couple posts, I think I can chime in. Oldchap, your router is going to be the weakest link in your chain for a while. Back when I was a contributor, I had an IPCOP machine set up. It was basically an old Pentium 2 machine with 128mb of ram, and I set that up as the router, attached to a switch. 10mil a day from me was easy if I remember correctly (not sure). After your router, either the cpu or hard drive will be the limiting factor. I am not sure how an ssd would hold up to MJ12, cause of all the writing.

[OldChap]

Ok then guys....I have some ddr2 and a 5050e so I need to buy a MB and PSU to build a router...the psu could be almost anything from a power point of view but the MB ...I've been thinking of a Gigabyte GA-MA770 UD3 (lots of pcie x1) thoughts anyone?

The final question is about NIC's...... any makes to watch for or avoid?

Incidentally, it is always the same rig that falls over ...usually with conn errors...tonight I will try it instead of my daily rig so still just running 2 rigs...I already stopped GPUGRID on it so here goes nothing

[DeadlyFire]

Ok then guys....I have some ddr2 and a 5050e so I need to buy a MB and PSU to build a router...the psu could be almost anything from a power point of view but the MB ...I've been thinking of a Gigabyte GA-MA770 UD3 (lots of pcie x1) thoughts anyone?

The final question is about NIC's...... any makes to watch for or avoid?

Incidentally, it is always the same rig that falls over ...usually with conn errors...tonight I will try it instead of my daily rig so still just running 2 rigs...I already stopped GPUGRID on it so here goes nothing

funny you mention this because I'm currently building my own router right now any parts will do, for the NICs stick to Intel, they're the tried and true, and easily supported by linux firewall/routing software(pfSense, IPcop, smoothwall, etc). I'm planning on installing pfSense on my router because it has a lot of configure-ability and seems to be the most updated. The limiting factor in your router will probably be the memory, cpu then hard drive. Routing eats up a lot or ram(home routers usually don't have more than 64-128mb and MJ12 puts a beating on them).

With your own router you need two nic cards, one will be for incoming internet WAN connection from the wall to your router(nic #1), and the other will be outgoing from the router(nic #2) to either your switch, your old router(if you can put it in bridged/switch mode) or directly to a computer(if more than 1 computer needs internet, use either a switch, turn your old router into 'switch mode' or simply add more NIC cards to your router). I decided to add more NIC cards to my router so there is less overhead. Intel NIC cards are about $30 on newegg, but you can get them much cheaper on ebay. I bought 5 of these from one seller and he gave me a bulk discount($8 per nic card). If you go for the multiple nic cards option, make sure your motherboard has enough PCI-E/PCI slots! If you decide to get a switch, you can find one for under $50 at newegg

Here's a diagram to visualize it:

[OldChap]

Mem = 2gig, 5050e = 2* 2.6gig, HD = 80gig 5400 sata (8M cache?) run on ubuntu? pfSense (what OS for this?)....this is my idea just now

Broadcom are the most common NIC's on ebayUK (at least cheaper ones anyway) Heard of them?

[DeadlyFire]

Mem = 2gig, 5050e = 2* 2.6gig, HD = 80gig 5400 sata (8M cache?) run on ubuntu? pfSense (what OS for this?)....this is my idea just now

Broadcom are the most common NIC's on ebayUK (at least cheaper ones anyway) Heard of them?

Yup I've heard of them,as long as they work OK then it should be fine; I would post here which one you plan to buy to rule out any incompatibilities before buying. pfSense is based on FreeBSD so it is its own OS basically(no need to install an OS). That 2 gigs ram will definitely be very handy. HD and CPU are good to go as well.

Here's a list of FreeBSD compatible NICs http://www.freebsd.org/releases/7.2R....html#ETHERNET

[DeadlyFire]

Haha, I forgot how much fun majestic12 is (I'll be making random appearances until I'm home in late June).

Welcome back

[[XC] moddolicous]

Thanks. I can help a little bit with the machine issue. Your machine has PLENTY of overhead for a router (like I said, I used a pentium 2 and it handled he workload fine). I used the IPCOP distro, which worked awesome. The only reason I had to stop it was that I used a hard drive I found, and it was bad, so once the power went out, the OS install got corrupted, and it was no good anymore Towards the end of last summer I was going to put together a Pfsense machine, just cause I liked the idea of trying something new, and it also sounded much easier to set up a wireless ap from Pfsense rather than IPCOP. Things got in the way, and it never happened. That being said, I am going to try again once I get home.

[Frisch]

Hello moddo............

Moddo is the author of our guide.

Are you back to get a majestic fix........

[=[PULSAR]=]

Well looks like I have everything up and running, wish I had more bandwidth. I used to have 40Mbps and now the ISP scaled it back to roughly 15Mbps Either way up and running looks like I'll be getting roughly 450k daily...not bad How are some of you on upward of 1 million, multiple locations running?

[DeadlyFire]

Well looks like I have everything up and running, wish I had more bandwidth. I used to have 40Mbps and now the ISP scaled it back to roughly 15Mbps Either way up and running looks like I'll be getting roughly 450k daily...not bad How are some of you on upward of 1 million, multiple locations running?

Nice Welcome

[[XC] moddolicous]

Well looks like I have everything up and running, wish I had more bandwidth. I used to have 40Mbps and now the ISP scaled it back to roughly 15Mbps Either way up and running looks like I'll be getting roughly 450k daily...not bad How are some of you on upward of 1 million, multiple locations running?

Welcome to the team! From a 15Mbps line, you should be able to get more than 450k daily. How many workers do you have it set for? If you take a screenshot of the connection, crawler and misc tabs, I can give you some recommendations as far as settings if you want.

[=[PULSAR]=]

Welcome to the team! From a 15Mbps line, you should be able to get more than 450k daily. How many workers do you have it set for? If you take a screenshot of the connection, crawler and misc tabs, I can give you some recommendations as far as settings if you want.

Okay thanks I'll post some up in a little bit, I feel like I have a pretty strong connection I can download things at up to 3 Megabytes/seconds I'm having a tough time understanding it I should easily be in the millions.

[=[PULSAR]=]

Okay here they are...

[[XC] moddolicous]

The only things I would really think about changing is maybe increase your upload speed. It wont really hurt anything, but it could help. I don't think the client will ever max out your upload speed, so even if you set it for something ridiculous, you should be fine. I also like to check the DO NOT USE PROXY FOR Uploading, even though it probably doesnt do anything. Crawler page?

[OldChap]

Well... I dumped the Gigabyte ep45t ud3r rig as even at 100/30 it still locks up after a few hours. Now I have the Supermicro DP board running and it already went longer that the Giga ever did.

My main problem just now seems to be upload speed.... lots of very sloooow uploads are not helping .....this means I will be limited to around 8mill/day....still playing with crawler settings though

[DeadlyFire]

Hey PULSAR here are some suggestions:






In the upload speed field I wasn't sure what you have for your upload speed so I put 1Mbps. If your speed is lower it will use whatever you have up to 1Mbps; If your speed is higher than 1Mbps than increase it to that amount.

My main problem just now seems to be upload speed.... lots of very sloooow uploads are not helping .....this means I will be limited to around 8mill/day....still playing with crawler settings though

Can you try running a speedtest to check your upload speed? speedtest.net

[[XC] moddolicous]

I agree with Deadlyfire (I couldn't see your crawler page yesterday for some reason ) Definitely change the fixed upload chunk and increase it, unless you ran the upload benchmark and that came out best for you. You can lie about your upload speed to Majestic12. In my experience, it never reaches whatever I set, so even if I set it for 15Mbps upload, it will just upload as fast as my connection allows. You can also lower the Robots txt flush delay and URL flush delay to 0. Alexc said that this is a setting from the old days, but I feel like it can't hurt. Also, if you use this machine daily (or someone does), put everything on normal except upload (leave that on high always). If this is dedicated, then I would put everything on above normal, except upload (leave on high). Whenever I put certain functions on high, the program seems to hang for some reason
Change your settings, restart the client, and report back in ~3hours and tell us how it is going

[=[PULSAR]=]

It's going much better already, I think the big thing I was missing was restarting the client. Didn't realize you had to, to have the changes take effect.

EDIT: After 10 mins I found my average is going to be roughly 6.2 million PPD.

[OldChap]

Deadly,

while crawling

normal non crawling


The common report along the bottom of the crawling screen is "upload stopped due to a connectivity problem will try later" and in the log " 05/06/2010 21:30:03] Could not connect to server to start uploading, looks like it's down, will retry later." appears for many lines one after another.

Where can I see how many buckets are waiting to upload?

EDIT: Alex reminded me that I'm running this concurrent with WCG so there is bound to be some interaction

[=[PULSAR]=]

Well I am pretty sure my ISP started questioning my bandwidth after a day of use. After changing some settings I was running at great speeds average of 16k kbits/seconds. After about 2 hours straight of watching it the bandwidth has stop to a dead crawl 200 kbits/second. Checked my internet bandwidth with multiple speed tests and getting 20k kbits/second. Ugh I think they are starting to block certain things.

[DeadlyFire]

Well I am pretty sure my ISP started questioning my bandwidth after a day of use. After changing some settings I was running at great speeds average of 16k kbits/seconds. After about 2 hours straight of watching it the bandwidth has stop to a dead crawl 200 kbits/second. Checked my internet bandwidth with multiple speed tests and getting 20k kbits/second. Ugh I think they are starting to block certain things.

That is definitely possible Some ISPs hate bandwidth junkies and MJ12 eats up bandwidth like crazy. I would try to lower your settings so you can still crawl but remain under the radar(lower the amount of workers to 50-100 and watch the crawl rate and data usage in MJ12). It is also possible that your ISP has a set data limit(ex.20 gigs) and once you reach it they downgrade your speed until a certain condition is met(24 hour limit, usage drops, etc.) I'd call your ISP to see if you can find out this limit and how you can stay under it without angering the ISP gods

Deadly,

The common report along the bottom of the crawling screen is "upload stopped due to a connectivity problem will try later" and in the log " 05/06/2010 21:30:03] Could not connect to server to start uploading, looks like it's down, will retry later." appears for many lines one after another.

Where can I see how many buckets are waiting to upload?

EDIT: Alex reminded me that I'm running this concurrent with WCG so there is bound to be some interaction

Your upload speed looks good so that can be most likely ruled out. I've already replied on Alex's forum but those upload errors you see happen some of the time. As long as they do not prevent you from uploading completely(i.e. if it causes you to have a huge backlog of uploads waiting to upload), then I would not worry about it.


Here's a visual on how a bucket upload backlog looks like:


NO BACKLOG:




BACKLOG:



The two numbers refer to the number of buckets that are waiting to be uploaded and what the size of the total buckets waiting to be uploaded is in megabytes. In the case above, there are in total 5 buckets, with a combined size of 10MB waiting to be uploaded. As long as the backlog isn't bigger than say, ~50-100MB then there's nothing to worry about. Looking at my log right now I can see about a dozen "upload timeout" messages.

[=[PULSAR]=]

That is definitely possible Some ISPs hate bandwidth junkies and MJ12 eats up bandwidth like crazy. I would try to lower your settings so you can still crawl but remain under the radar(lower the amount of workers to 50-100 and watch the crawl rate and data usage in MJ12). It is also possible that your ISP has a set data limit(ex.20 gigs) and once you reach it they downgrade your speed until a certain condition is met(24 hour limit, usage drops, etc.) I'd call your ISP to see if you can find out this limit and how you can stay under it without angering the ISP gods

Just looked it up and spoke to someone I am capped at 200GB download/upload a month. So far MJ12 has only used 34GB since yesterday? I can't imagine where the rest went. I only run WCG and GPUgrid, other than that I just surf. I don't torrent or anything. Hmmmm and then when ask if they could check where I stand on monthly usage they couldn't tell me.

[OldChap]

from alex

Router gone down again in our office - ******* piece of ****.

Give me 2 hours to deal with it - needs to attend some urgent stuff first I am afraid

[DeadlyFire]

Alex's router is unfortunately being hammered and keeps crashing It's probably because of the increase in downstream bandwidth capacity in the last few weeks and higher crawling rate as a project overall so it's not all doom and gloom. Hopefully he gets the router issue sorted out soon

[Movieman]

I'll add to the grief.
I usually runMJ12 on mytwo Harpertown machines and Sammy.
One Harper sold, the other came down to night tobe cleaned and to get ready to sell soI'm at one machine pulling 15mbit vs 3 pulling 26-29mbit until I rearrange things here.