The Red Library DLV (DC) project thread + guide.

[[XC]melymel]

http://www.readyresponse.org/index.php?item=distributed&opt=2

The Red Library DLV project, or Red Library for short, is a long run DC project by XS that uses Bandwidth not CPU power and increases our DC Vault rank. Here is what the people who create the project give as an explanation;


What is Red Library?
Red Library is Ready Response's search engine with a difference - every single link in the database is hand picked (at present 7000+ links) and categorized. It is a tremendous resource and used by many other sites/visitors worldwide, but as with all search engines there is always a possibility that a link in the database will become a broken a link (404).



What is this projects purpose?

The idea behind this project was to make a client application which downloads links from the Red Library server, validates them (i.e. checks to see if they are 404 or not) and then returns the results back to Red Library. The Red Library server will then double check all the links that come back reported as 404, and remove the links if they are found to be down over a period of 7 days or longer. Basically - the more people that run the client, the less bad links will be present in Red Library!


Although this project is not seen as overly useful it uses near 0 cpu power and instead relies on bandwidth like MJ-12 and DIMES so you can run a cpu intensive DC project such as rosetta along side it. Red Library however has the advantage of using less bandwidth per client than either MJ-12 or DIMES. I have also only seen mine use 15mb of ram (in task manager) per client so many can be run on one boxen.

SETUP GUIDE

1)First you must register by following this link here (http://www.readyresponse.org/index.php?item=distributed&opt=3) and await them to email you your password. You then have to join the Xtreme Systems team this can be done by clicking Join on the team page here (http://www.readyresponse.org/index.php?item=distributed&opt=12&tid=11)


Once you have joined the team and recieved the email from ready response containing your password you can proceed to download and setup the client.


2)You can download the client from this page here (http://www.readyresponse.org/index.php?item=distributed&opt=7) at the moment there are only clients for windows and mac but they are working on a linux client. You only have to download the client once even if you want to run more than one of them.


3)Once you have downloaded unzipped and installed the client you simply fill in your username and password in the boxes (you can cut and paste(ctrl+v) your password in to save trouble)) and then select how often and how much you want the client to work. This can be done through the 2 drop down boxes to the right of where you filled in your username and password. To get the most out of your client in the top drop down box labelled interval select 5 mins meaning it will work every 5 mins, next in the box below it, labbeled links, select the maximum of 10 so now the client will validate 10 links every 5 mins, this is the most one client can do.


Now this is the way I add multiple clients;

4)To replicate the client simply copy the entire folder that the client you just setup is in and paste it somewhere else (may be wise to keep all clients in one folder). Now enter the client folder you just pasted and rename the "Red-LibDLV.exe" to something else "A.exe","B.exe"etc. Now you can simply open the clients up and they will run in unison, repeat this for however many clients you want.



A problem you may encounter is the taskbar fills with clients as hixie found out :p: so remember you can simply drag it up to stack it like this http://img72.imageshack.us/img72/6236/tbarzk6.png (http://imageshack.us) :D

Another way to solve the taskbar issue is to hide the icons, thank Martijn for the following:


This works agains the icons in the taskbar:

Rightclick on taskbar, select properties, press customize, then modify your icons' preferences.

http://img432.imageshack.us/img432/2037/propertiesxx4.th.jpg (http://img432.imageshack.us/my.php?image=propertiesxx4.jpg)

Now lets take number 1 in DC vault miscellaneous section :toast:

[vaio]

Uh oh :p:

[[XC]melymel]

Uh oh :p:

You guys have a 5.5million point headstart use it wisely the XS steamrollers about to get going, we'll be seeing you in a couple of weeks :p:

:toast:

[hixie]

I'm running Red Library at around 50% power now, just gotta sort out that taskbar problem :(

[vaio]

You guys have a 5.5million point headstart use it wisely the XS steamrollers about to get going, we'll be seeing you in a couple of weeks :p:

:toast:

You'll soon pass us........most of our team got bored with the 3 client restriction and gave up on it.
Just one or two of us part time now....we were 2nd til recently, be out of the top 10 soon :eek: :D

Have fun :)

[[XC]melymel]

You'll soon pass us........most of our team got bored with the 3 client restriction and gave up on it.
Just one or two of us part time now....we were 2nd til recently, be out of the top 10 soon :eek: :D

Have fun :)

Give them note then that the restriction has been lifted so you can run as many clients as you want on one rig!

The restriction was what put most of us off too and why we didnt really take it seriously. Hope to see you giving us some good competition now :D :toast:

[vaio]

We be a small team.......most of what we do have goes into MJ12......and we're still getting caned there too.

We can't compete with the big guns :bsod: :bsod: :bsod: (ain't surrendering though :p: )

typo edit.

edit 2.......oops, sorry I keep smapping up your dc threads.

[hixie]

I'm sure some underdog team will give us a little poke in the future

[[XC]melymel]

I almost forgot anyone with suggestions/ questions/ additions/ ways to better the guide, just post here and i'll add/edit it to the original post :toast:

[Martijn]

This works agains the icons in the taskbar:

Rightclick on taskbar, select properties, press customize, then modify your icons' preferences.

http://img432.imageshack.us/img432/2037/propertiesxx4.th.jpg (http://img432.imageshack.us/my.php?image=propertiesxx4.jpg)

[[XC]melymel]

Cheers Martijn :toast: I have added it to the top for those that don't scroll down pages :p: I can't wait until they get the linux client windows is driving me mad. :mad:

[Bloody_Sorcerer]

tossed up 10 clients, we'll see if it impacts MJ12 at all. if it does i'm dumpin DLV

[[XC] riptide]

Thanks Mely! This vault section is becoming a one stop resource for a lot of projects and how to get started!

[[XC]thewildblue]

Nice work melymel. I will sticky. I reckon I must have over 50 clients running now over 3 machines but I dont seem to get consistant points. Gonna try and add anoter 20 to 40 today depending on how I get on.

On a side note since yesterday same time we went from 15th producer overall to 4th....way to go team.

[[XC]melymel]

Cheers for stickying it :toast:

For some reason none of my clients seem to have done anything today although I have 100 running and they're all validating :confused: I have just fired up another instance of MJ-12 that has v. high errors so i may try closing that and see if it helps.

[[XC]thewildblue]

Mine are sporadic, did 9000 on the last update that I checked.....I cant remember how many I have running now....lol

[[XC]melymel]

Mine are sporadic, did 9000 on the last update that I checked.....I cant remember how many I have running now....lol

How long have they been running now? Mine appeared to have settled down to 9K per update last night after a few hours of fluctuant output but then i went and opened MJ-12 :p:

I actually did something last update after closing both MJ-12 clients so to me it seems there may be a conflict if running MJ-12 + lots of RL clients although I was getting 100K updates in MJ-12 so it may just have been sucking up all the bandwidth i'll leave MJ-12 off for now and see if it picks up anymore.

:toast:

[[XC]thewildblue]

Well since yesterday morning for the early ones and this for some of the later ones.

We jumped 2 places today.....w00t

[Martijn]

Exactly how much bandwith does one client use? I understand it can fluctuate a little depending on the size of the pages, but I'm sure someone can give me an idea?

[Steensen]

I would like to know how mutch bandwith it uses too.

Is it "only" checking for 404's on the www, or is there something else it does?

[[XC]thewildblue]

Well I have about 80 clients running and I dont even notice them on my connection, where-as 60 dimes would show up. Seems it only checks links. It does 10 links every 5 mins at maximum settings.

[Martijn]

That is nothing... I'll try and pop in a few clients as soon as I can fire up my MJ-12 rig again...

[[XC]melymel]

Warnign people I had to stop MJ-12 because it wouldn't run with RL. I don't know if MJ-12 was hogging all my connection or if my nic's couldn't handle it but closing MJ-12 fixed the problem!!

I'm testing my laptop atm because i dont think it was doing anything :p: so my output will be up and down for a while, also my friend wanted me to kick his a$$ on enemy territory breifely :p:

:toast:

[[XC]melymel]

:woot: my laptop did something now and i'm hoping to match you for the top pie James so watch out ;).

I find 60-65 clients is the maximum you can fit on 512mb and run other things and so at the moment have 12,500 per update. I don't know whats up with the scores today but it seems we have yesterdays output added up aswell giving us 1st,3rd and 6th overall :rofl:
http://img86.imageshack.us/img86/2183/rlxk6.png (http://imageshack.us)

The good thing is today we're only 12,000 behind Free-DC so one extra person could make the difference and get us into 3rd daily, so come on you old DIMERS we need some more help!!!!

:toast:

[[XC]thewildblue]

Wow whats your usage like, Im finding that running 30 on a 1gb system is taking up 1.2gb in mem usage. So its swapping a little. What version of .framework you running. Im only using 1.1 Wonder if that is why I have such high usage.

I am nowwhere near maxxed out, Im just down on rigs because of the heat, have another 5gb of ram in systems waiting to abused with red library....lol

[[XC]melymel]

Well my main rig has settled down on 58 clients whilst allowing me to use it and with only 512mb ram it uses 1gb of swap file and my laptop with 61 clients uses 1.25gb of swapfile with its 512mb ram. :D I use 1.1 framework too, I had to stop rosetta on my main boxen though because it was too memory hungry so that's on SOB for now. Unforunately I'll be selling this main boxen soon so my production will drop again. Hopefully should be replaced by a 939 box atleast

Cheers for joining today Helheim :toast: We could use more help now too because PCZ has rejoined Free-DC and PY222 hasn't even bothered to run it yet and if he does i think we're in trouble. :p:

P.S. I don't understand how swapfile works or anything but couldn't we just allocate say 20gb of hard drive space for it and run a few hundred clients on a dedicated box or does it not work like that?

:toast:

[[XC] riptide]

Is swapfile limited to 4gig!? Anybody! Not sure?:confused:

[[XC]thewildblue]

More in swap file will make the system run like a dog unless you have a maybe a raptor setup in raid0, but dont plan on doing much else on the system.

[[XC]melymel]

Have we crashed the red library server as I cant log into ready response and there are no stats, I hope they don't reinstate the limit again. :(

I played with the swap file today and yes you can only get 4gb max but I find only 2gb is useable but it allows 80+ clients on my laptop's 512mb. I took my main box down for a final spi 1m run before selling it and still managed 8.1K updates thanks to that laptop alone :p: :toast:

[[XC] riptide]

Mely! 1000 posts! Well done! (Ya big spam maestro :rofl:)

[[XC]melymel]

Mely! 1000 posts! Well done! (Ya big spam maestro :rofl:)

:woot: good spot on the 1,000!!! Atleast all the keys I click in my posts add to XS DC now :D

Now off to delete an old post :p:

[Martijn]

Atleast all the keys I click in my posts add to XS DC now :D



That reminds me I still have to dump 50k keys :cool:

[hixie]

I have my page file set to 4GB on a seperate partition which is hidden in my computer. Got over 100 clients running on it now, and i sorted my stability problem so i should have a more consistent output for Red Library and MJ-12 now.

[[XC]melymel]

Incase no one sees the other Red Library thread:

XtremeSystems is in the top 10!!!

:YIPPIE::YIPPIE::YIPPIE:

:toast:

[em99010pepe]

How to start multiple instances with only one click?

Just create a batch file and put this:

Start /min c:\dc\red\red1\RedLib-DLV1.exe
ping -n 30 127.0.0.1 > NUL
Start /min c:\dc\red\red2\RedLib-DLV2.exe
ping -n 30 127.0.0.1 > NUL
Start /min c:\dc\red\red3\RedLib-DLV3.exe
ping -n 30 127.0.0.1 > NUL

etc (two lines for each instance)

-n 30 means 30 seconds delay between each start.
Change path for each client.

It works!!!

Cheers,

Carlos

[[XC]thewildblue]

Thanks, any chance of telling me a quick way to replicate clients, currently copy,paste and rename.

[em99010pepe]

Thanks, any chance of telling me a quick way to replicate clients, currently copy,paste and rename.

Please check this free-dc thread (http://www.free-dc.org/forum/showthread.php?t=11459).

[hixie]

Rasberry pie!!! :banana: :banana:

[[XC]thewildblue]

Hixie how many clients are you running ?

We really need to ramp up production if we are to even keep up with ars and free-dc.

[[XC]melymel]

We do really need more people and more output only 6 member really running it isn't enough :(, Also Free-DC have called on the big guns just look at their updates they're higher than what I have done in total today :eek: . I'm struggling on my laptop and p3 600 loaded with 30 clients at the moment so my output's dropped right off. When DAK gets back from vacation I reckon we could convince him into running this for us and you know what happens when he gets into a project :p: . But yeah we need more members especially considering we arent sure that they wont reimpose the limit so we may only have a limited time to do something... :toast:

[[XC]thewildblue]

Well I increased a little today. So lets see how it goes. Im trying 60 clients on a machine with 1.25gb ram. Its got r@h on as well so lets see how it does.

Maybe we should get ready to pimp the rosetta section after 7th of August as we could do with some more members in the DC Vault area.

[Jose]

Incase no one sees the other Red Library thread:

XtremeSystems is in the top 10!!!

:YIPPIE::YIPPIE::YIPPIE:

:toast:

It is now past the Top 5!!!!!

Red Library DLV Project xtremesystems ( +1) 4
8,888.888889 (+370.370370) :woot: :woot: :woot:

[[XC]melymel]

It is now past the Top 5!!!!!

Red Library DLV Project xtremesystems ( +1) 4
8,888.888889 (+370.370370) :woot: :woot: :woot:

:woot:
Well 6th in reality but in the vault it says 4th :D. Unfortunately I guess the 3 ahead of us are AMD users,F-DC and Ars all of which aren't going to be easy.

James I agree with the pimping rosetta after casp we have a lot of projects now all of which could use more power maybe we should create a pimp team and plan how to go about it. :D

:toast:

P.S. twb: 30K updates :eek: great work, have you got anything else up your sleeve to add? :D

[[XC]thewildblue]

I have a few more to add, hehe. I have a rig here with 2 gig of ram running nix, but I cant get vmware to work otherwise would have another 100red clients already.

[hixie]

I have 1GB RAM and 4GB swap, 157 clients seems to be my max if i open anymore they error in 5 mins and close down. My pagefile usage is 3.50GB.
Might be adding another machine tommorow, depending if i can get off work early tommorrow.

EDIT: Someone has been stealing my rasberry pie :( :slapass:

[mcflurry4321]

Why am I unable to run more than like 25 or 30 clients on a 512 mb of ram computer?

[[XC]thewildblue]

Try adjusting your swap file to a larger size.

I Dual booted that nix box ( I had forgotten) so I now have another 150 clients running on there. Has anyone maxxed out a 2gb box yet ?

[[XC]thewildblue]

Just put up some info in the "Dimes" area if you know what I mean.....a potential tweak for all you RLers.

[mcflurry4321]

Just put up some info in the "Dimes" area if you know what I mean.....a potential tweak for all you RLers.

I don't see it....

[mcflurry4321]

Sorry for the double post, but I have another question. On one of my computers each Red Library client is taking approximately 5k memory, but on another of my computers each client is taking between 15k to 20k. Anyone else noticing this?

[[XC]thewildblue]

It should settle down somewhat after a while. If not just end the process and start it again.

mcflurry ygpm

[[XC]thewildblue]

My production is going to be down for some time Im afraid guys, my dad had a motorcycle accident last night, he is ok but he almost snapped his right foot off, I was first on the scene and the bone was just there glaring at me. Hopefully he will be able to keep it, thankfully he hasnt got another injuries, he was going quite quick from what I understand and the bike just seized up and had him off. Hopefully he will be out of hospital in a couple of weeks.

Well as my dad was self employed and a builder there is no longer an income coming in so things like crunching are a bit of a luxury now and I will have to do my best to help them out.

[[XC]melymel]

When things like that arise you need to get the priorities right and obviously it's your dad and money > crunching. It sounds a real nasty accident and I truely hope your Dad gets to keep his foot :( .

Anyway I hope he gets better soon and makes a full recovery and don't worry at all about crunching James.

:toast:

[hixie]

Ouch, i hope your dad gets well soon.
Crunching and crawling can wait, i'll try to increase my production to make up for you.

[[XC] DragonOrta]

Now seems like as good a time as any to join up w/ Red Library.

[mcflurry4321]

Now seems like as good a time as any to join up w/ Red Library.

Damned right! :p:

[[XC]melymel]

Cheers for the help DO we need all we can get. I've had to back down my production as I only have a 4gb HDD and 512mb of ram on my p3 and the poor thing jst kept dying or was unuseable :lol:

P.S. I'll wave you past mcflurry when you overtake shortly.

:toast:

[mcflurry4321]

P.S. I'll wave you past mcflurry when you overtake shortly.
:toast:

Why thank you :D

[[XC] DragonOrta]

I can only run 4 instances on my laptop, because I get runtime errors on them if I run any more. I'll get it running on my 2gb X2 soon though.

[hixie]

DragonOrta, try installing Windows NDP2.0 pack. My runtime error limit went from 157 client to 160, just by installing that...

[[XC]melymel]

How many clients are people running?

I decided to power up a few clients today and read on free-dc that there is a 9 client limit however 7 is all I can load up, anymore just give the maximum of 3 client message. I noticed twb and jspace are both out producing me so i'm wondering how many clients your running. Also is anybody else going to bother to run this? :toast:

[hixie]

I'm not going to run this seriously now, maybe one or two clients at the most.
The 3 client limit is rather stupid.

[BandofBrothers]

just to let you guys know, I am back on board with this project.

Will throw a few clients on this rig now:) :)

[[XC] riptide]

Red Library is long dead. I'm unstickying this thread.