Server

[Duh]

How about gettin a cheap ovh 100 mbit server and start teaching google a good lesson?

There are some pretty good deals and we could pay them quite easily if we team up.

:up:

J

[[XC] riptide]

Explain that? You get a server??

[Duh]

I dont have one at the moment but thought that if we team up we might be able to hace a cheap and efficient server

http://www.ovh.co.uk/individual/products/rps1.xml

from what I ve been told at the forums ( ovh forums)the expected monthly transfer is 2.5 tb ( quite good for the price).

regards and happy new year

J

[ripken204]

i find this post to be quite funny.
and to make it even funnier, look at the specs on that server.

[Duh]

i find this post to be quite funny.
and to make it even funnier, look at the specs on that server.
your post sucks dude :down: . I am trying to make something nice and you walk by and post nonsense with no backup at all. Sorry but I will ask mods to take a look at this post as I dont thinks its constructive or that goes with XS´s spirit.
Please explain why you consider it funny.

Here is my backup of why your post has only destructive purposes :


You will need to have unmetered broadband connection (you can limit the amount of bandwidth to be used), reasonably fast CPU (Pentium-II 400 Mhz+ or better) and 256Mb of RAM. Actual usage of memory depends on your settings, with default ones resulting in 50-60Mb usage at any time.

source:http://www.majestic12.co.uk/projects/dsearch/download.php

au revoir

[ripken204]

well asking people to pay for a server will not go through well at all.
the specs on that server you posted are not good either, you need high performance servers for search engines, this one will participate an extremely small amount in this project.
and the search engine doesn't even work to begin with, it only times out for me.

[Duh]

well asking people to pay for a server will not go through well at all.
the specs on that server you posted are not good either, you need high performance servers for search engines, this one will participate an extremely small amount in this project.
and the search engine doesn't even work to begin with, it only times out for me.
your wording is nonsense. to proove you are wrong you have the clearest example : google..

second : HAVE YOU READ THIS????? its only a client running


You will need to have unmetered broadband connection (you can limit the amount of bandwidth to be used), reasonably fast CPU (Pentium-II 400 Mhz+ or better) and 256Mb of RAM. Actual usage of memory depends on your settings, with default ones resulting in 50-60Mb usage at any time.

your posts are completely :down:

[[XC] riptide]

well asking people to pay for a server will not go through well at all.
the specs on that server you posted are not good either, you need high performance servers for search engines, this one will participate an extremely small amount in this project.
and the search engine doesn't even work to begin with, it only times out for me.

Easy now lads.

Ripken... I ran MJ-12 before on a P2 400Mhz laptop. It ran ok actually.Not great. But ok and was enough for 2Meg down DSL. The servers that are for hire on this page actually fit the bill for this project.

[Duh]

thanks riptide!

I am not asking to donate me $. I am proposing to donate to someone who wants to join those lil bucks ( 10eu/month) and pay for the server in order to make use of a technology that I think that might help enormously the project with 0 hassle and with just some cents from us. I do not want to keep your money :)

[Movieman]

Just to back up what riptide said; We have worked this project for close to 3 years now and as friends. If someone has an idea and one of us doesn't like it we don't jump on the guy we talk as friends.
Also remember it takes two to argue. If someone does dump on your idea ignore it. No sense getting mad.
Now lets PM some people, send them a link to the project and see if we can get some more bodies on this and get the team back to where it used to be.:up:

[[XC] riptide]

How much is 2.5TB these days in terms of points/urls? is it still about 40k per URL?

EDIT: If you look at the other options on that link, you can custom and hire serious dual quad socket machines. :) Its costs alot however. I'd have to do my sums to see would it be worth it vs. buying equip. and paying power. IIRC py222 over at Free-DC bought datacentre processing. Someone said that 3rd hand...somewhere.

[Movieman]

riptide;3548037']How much is 2.5TB these days in terms of points/urls? is it still about 40k per URL?

EDIT: If you look at the other options on that link, you can custom and hire serious dual quad socket machines. :) Its costs alot however. I'd have to do my sums to see would it be worth it vs. buying equip. and paying power. IIRC py222 over at Free-DC bought datacentre processing. Someone said that 3rd hand...somewhere.

I run two machines on MJ12.. The Sammy and the Q6600.. Both also run WCG.
I get between 5-6 million URLS a day between both machines on a 15mbit/2mbit line..
I did try it for awhile on the dual clover with a raptor in it and that worked app 10% faster for processing but in reality even the little sammy will use over 10mbit all by itself..
I'm telling you, that little machine can handle almost anything you can throw at it. One thing I did see was that I got a HUGE performance increase from it when I went from an old 40 gig IDE drive to a single 36 gig Raptor..
Man it was like you woke the sucker up..

[hixie]

Setting a server up is the easy part, i have some dual socket parts here almost enough for a complete system. The real problem is maxing out that 100Mbps line your talking about.

Only one .net client is allowed on each windows platform, and each MJ-12 client is limited to just over 10Mbps due to a bug in .net framework.
To get around it you'll need to load around 10 VMs onto the machine in order to in theory max out that 100Mbps line. However, their are always inefficiencies and from personal experience i know for a fact that it's so much easier to run 10 separate celeron/sepron machines than to run a single more powerful machine with 10 VMs.

The next problem is your router and modem combo, handling all those concurrent connection efficiently you'll need to get a smoothwall/IPCOP router setup, and hope to god that your DSL/ADSL modem can handle that many concurrent connections.

And the next question you should ask yourself is "Is 100Mbps really going to make that much of a difference?" The answer is yes and no, for MJ-12 ping times are so important, my old line had a huge amount of bandwidth but the ISP sacrificed some ping time to get that bandwidth, my new ISP has a bandwidth that only 1/10th my old line but ping times are about 3-4 times faster.
So the real answer really boils down to the balance between ping times, bandwidth and the amount of machines and VMs.

[[XC] riptide]

^^^ Ya. I didn't know Ping times was a big factor. Interesting. What way are teh Vm's limited? CPU power used processing the 10mbit lines archiving, and bucket/crawling handling? Also had not newer versions of .Net framwork increased the cap on network handling?

[Duh]

Just to back up what riptide said; We have worked this project for close to 3 years now and as friends. If someone has an idea and one of us doesn't like it we don't jump on the guy we talk as friends.
Also remember it takes two to argue. If someone does dump on your idea ignore it. No sense getting mad.
Now lets PM some people, send them a link to the project and see if we can get some more bodies on this and get the team back to where it used to be.:up:


Thanks my Irish friend :). I might have got confused when I saw his post count ( I had into my variables that he knew our spirit and that he knew what we do).

I excuse myself to ripken204 if I happened to be disrespectful.


Setting a server up is the easy part, i have some dual socket parts here almost enough for a complete system. The real problem is maxing out that 100Mbps line your talking about.

Only one .net client is allowed on each windows platform, and each MJ-12 client is limited to just over 10Mbps due to a bug in .net framework.
To get around it you'll need to load around 10 VMs onto the machine in order to in theory max out that 100Mbps line. However, their are always inefficiencies and from personal experience i know for a fact that it's so much easier to run 10 separate celeron/sepron machines than to run a single more powerful machine with 10 VMs.

The next problem is your router and modem combo, handling all those concurrent connection efficiently you'll need to get a smoothwall/IPCOP router setup, and hope to god that your DSL/ADSL modem can handle that many concurrent connections.

And the next question you should ask yourself is "Is 100Mbps really going to make that much of a difference?" The answer is yes and no, for MJ-12 ping times are so important, my old line had a huge amount of bandwidth but the ISP sacrificed some ping time to get that bandwidth, my new ISP has a bandwidth that only 1/10th my old line but ping times are about 3-4 times faster.
So the real answer really boils down to the balance between ping times, bandwidth and the amount of machines and VMs.
Wow. You do know a lot about the project !!!!!! I thought about opening one single instance and cross fingers :S. Even though the 100mbits is shared it must do some serious damage :).
If processing power is limiting us to create more instances through virtualization we can get the atom cpu listed in the link I ve provided for some lil bucks more :)

[hixie]

riptide;3548390']^^^ Ya. I didn't know Ping times was a big factor. Interesting. What way are teh Vm's limited? CPU power used processing the 10mbit lines archiving, and bucket/crawling handling? Also had not newer versions of .Net framwork increased the cap on network handling?

I think the VM's are most likely to be limited by their 10Mbps virtual network, and the amount of concurrent connection they can handle.
CPU power isn't that big a problem really, and Q6600 handles 4 clients with some to spare.
The latest versions of .net framework (.net 3.5 SP1) did improve that cap quite a bit, but would still only peak to about 15Mbps.


Wow. You do know a lot about the project !!!!!! I thought about opening one single instance and cross fingers :S. Even though the 100mbits is shared it must do some serious damage :).
If processing power is limiting us to create more instances through virtualization we can get the atom cpu listed in the link I ve provided for some lil bucks more :)

The Atom does support HT which would help the VMs, but i really doubt the atom can handle 2 fully optimized clients. Oh and believe me i can make a celeron/sempron cry because of MJ-12.
To fully utilize a 100Mbps line you'll either need almost 20 machines or one really powerful machine (dual socket quads) and VM the crap out of it.

EDIT: You don't want to max your CPU, when the CPU is maxed the errors increase.

[[XC] riptide]

I think the VM's are most likely to be limited by their 10Mbps virtual network, and the amount of concurrent connection they can handle.


1st the default adapter on Sun Virtual Box is 100Mps ;) I know Vmware default is 10.

2nd You can tweak the tcpip.sys file on each VM to handle the concurrent connections issue. I had to something similar to all my VM's in Dimes once.

[hixie]

riptide;3552462']1st the default adapter on Sun Virtual Box is 100Mps ;) I know Vmware default is 10.

2nd You can tweak the tcpip.sys file on each VM to handle the concurrent connections issue. I had to something similar to all my VM's in Dimes once.

Is that the one within the VM's OS?

[[XC] riptide]

Is that the one within the VM's OS?

Yes of course. AND the host OS. No point tweaking one without teh other. ;) I had a MTU issue that effeted both VM's and host before. Fixing the Host tcpip.sys did not resolve teh VM issue. I had to tweak every damn VM... :shakes:

EDIT: IIRC Deadlyfire or Mekoa I think reported at one stage up to 10% increase with a tweaked tcpip.sys on their own PC's that were running. They didn't update us further about it. Basically increasing the TCP Recieve window on WinXP machines can go along way on big fat lines... like yours :D

[hixie]

riptide;3554288']Yes of course. AND the host OS. No point tweaking one without teh other. ;) I had a MTU issue that effeted both VM's and host before. Fixing the Host tcpip.sys did not resolve teh VM issue. I had to tweak every damn VM... :shakes:

EDIT: IIRC Deadlyfire or Mekoa I think reported at one stage up to 10% increase with a tweaked tcpip.sys on their own PC's that were running. They didn't update us further about it. Basically increasing the TCP Recieve window on WinXP machines can go along way on big fat lines... like yours :D

That fat line is no more now :( it wasn't really that good. Great when you had friends on the same network and you want to share a dozen files (that are not illegal :rolleyes:) other then that it was crap, internet was slow service was crap luckily i moved house and they didn't cover that area so they were forced to cut my plan short. Even now i still get a monthly statement with big red letters saying i owe them money. I'll never be using Hong Kong Broadband again.

[DeadlyFire]

riptide;3554288']

EDIT: IIRC Deadlyfire or Mekoa I think reported at one stage up to 10% increase with a tweaked tcpip.sys on their own PC's that were running. They didn't update us further about it. Basically increasing the TCP Recieve window on WinXP machines can go along way on big fat lines... like yours :D


Yep very true, however at some point set it too high and the NAT table on your router will overflow and in my case my router chokes and loses connection for about 30 seconds and then it's back to normal again. This only happens when I have more than 100-120 concurrent connections, which the router feverishly tries to juggle before passing out :( Looking at the security log I see "NAT Error : connection pool is full. No connection created" repeated dozens of times :shakes:

I tried to take Verizon's Actiontec MI424-WR router out of the loop and plugged in the CAT5 cable directly from the ONT(big fancy box that converts light signals into digital) to the Ethernet port on my PC. After that I can max my connection out with 220 workers, no problems because I set my 'total number of concurrent connections' in the tcpip.sys file to 800 and there is no router overhead to slow things down.

I'm looking at buying a D-Link DGL-4300 router(though I'd love a Cisco 2800 series more :D) From googling it seems to be one of the cheaper, consumer-level routers with a huge NAT table(vs. the 1kb NAT table size on my Actiontec (http://www.smallnetbuilder.com/component/option,com_chart/Itemid,189/chart,124/) now that chokes on more than ~96 simultaneous connections :down:).

[Frisch]

The server dialog

Hi, I'm a server, how can I help you ?

Hi, server, could you please let me fiddle a little, with your register ?

No, that's not possible, as I'm the master of the net-universe, fiddling would cause severe damage to my superior ruling, and create big black holes in the net-cosmos.

[Duh]

I think the VM's are most likely to be limited by their 10Mbps virtual network, and the amount of concurrent connection they can handle.
CPU power isn't that big a problem really, and Q6600 handles 4 clients with some to spare.
The latest versions of .net framework (.net 3.5 SP1) did improve that cap quite a bit, but would still only peak to about 15Mbps.



The Atom does support HT which would help the VMs, but i really doubt the atom can handle 2 fully optimized clients. Oh and believe me i can make a celeron/sempron cry because of MJ-12.
To fully utilize a 100Mbps line you'll either need almost 20 machines or one really powerful machine (dual socket quads) and VM the crap out of it.

EDIT: You don't want to max your CPU, when the CPU is maxed the errors increase.


I guess its useless for our project then.. too bad as I did want to receive your money and run away with your billions of $ ( madoff mode off) . I was joking of course.

at the rest of the posts: my head is crippled and my lil knowledge about the stuff you are talking about makes me unable to comprehend :S

[Duh]

Thread revival!
GUYS: a friend of mine has a dedi box with OVH and has been performing very very good ( he is quite new and is in the top 20) .
As the one I mentioned before; we can get a very nice machine for pennies. We only need 20 dollars. This guy will have a very strong VPS ( 1gb of ram , 375 gb of hdd and a shared 100 mbit line ( which is used by external guys from pur box and the 3 of us :)

we only need the guy who wants to join us can translate contents of our curremt one and helps us config a vpn, some web sites and so on :)

[Mekoa]

Greetings!

Yeah I tweaked my local tcpip.sys and some other settings and got around 10-15% more. Sure deadly got more than me as his connection was better.
TCPReceiveWindow was one setting that allowed larger transfers. Think it worked better for me as im below the .NET cap (was around 8mbit).

Unfortunately, now that I moved 100 yards up the road my net is a third of the speed it was :( Lucky if I pull 750k a day!

Pretty sure it was mentioned above, but need VM's to run more than the .NET bandwidth cap (or several machines).

Dave,

Did you get that much of a bonus from the single 36GB raptor? My crunch box, bless its cotton socks, is using an old 120gb Maxtor POS, but I took the raptor out of my main rig due to space constraints.

May fire it in that box, reinstall XP64 and go crunch mad again :)

Cheers,

Mek.

[Duh]

the box from my friend sits in a debian box so I guess that mod won´t help.. IIRC dave has downgraded his connection speed

[Mekoa]

Dave was on FiOS for a while, 15mbit down I think. His output still seems decent though?

My connection got annihiliated when I moved into my new flat. Was 8mbit, now 2.5 :( Was only 200 yards up the road!

[Duh]

Mekoa;3957746']Dave was on FiOS for a while, 15mbit down I think. His output still seems decent though?

My connection got annihiliated when I moved into my new flat. Was 8mbit, now 2.5 :( Was only 200 yards up the road!

not that bad m8. I am in 1mbit connection and most likely will remain that way for some time.. I am willing to get a server ( multi purpose) and this way be able to help than what I am currently.

I am willing to get into a c2d with 2gb of ram 100mbit ( shared).. we ( a handful of friends and I ) will pay in total 35 Euros, not that bad, huh?

I want to kick over 2 tb per month :)

[Mekoa]

€35 a month? Thats rather nice tbh... I pay around that here just for my phoneline and ADSL2+ lol

[Duh]

Mekoa;3984859']€35 a month? Thats rather nice tbh... I pay around that here just for my phoneline and ADSL2+ lol

yeah, a real bargain ( not for me as we have a tremendous parity of something like 5 to 1 to the euro)