Well I setup MJ12 last night and got it crankin'...
Woke up this morning (about 7.5 hours later) and between my two comps I've crawled 733,000 URL's. Is that good, bad, or indifferent?
Well I setup MJ12 last night and got it crankin'...
Woke up this morning (about 7.5 hours later) and between my two comps I've crawled 733,000 URL's. Is that good, bad, or indifferent?
That's estimated about 2,4 mill a day.
There's a number of ifs here, where you are crawling (domains), and how much you allow the line to use, amount of workers, and open url's.
This is a fidling program.
~Opinion about Dogs~~~~~~~Hug~~~~~~~~
~~~~
________________________________________________
See I don't have anything advanced setup... Both boxes allow up to 100 workers each and 20 URL buckets.
My primary box is able to use 24mbit down and 2mbit up, my Dell box is able to use 18mbit down and 1.5mbit up. I have noticed however that I never see either one using more than about 8mbit, and my average is about 2.5mbit on both boxes. Is this just due to the servers being in the UK and me being in the US?
2.4mil a good number?
No, not if you are running those lines, didn't know you had that at your disposal.
FragTek I have a meeting in five, movieman, or whoever is online will guide you, and if not i'll be back later.
But one thing, it has nothing to do with the servers, cause you are crawling the net, the only thing you have to do with the servers, is getting url's, and upload them back.
You'll never hit your top, the average will always stay somewhere below the top, but try to set your domains to US only, and nearby. Under tools you can select preferred domains.
Last edited by Frisch; 03-23-2007 at 06:20 AM.
~Opinion about Dogs~~~~~~~Hug~~~~~~~~
~~~~
________________________________________________
Yeah I currently have a 30/2.5 line and in about a week or so I'll have 60/5 at my disposal so I need to get this setup to utilize as much of that as possible.
When I get home I'll try setting my domains to US only and see if that helps out a bit.
Is there a "recommended" or known good number of workers and buckets?
Thx all Glad to be of service!
FYI: I managed to do ~1 mil/day out of a 4/1 line. MJ12 is a lot of tweaking/testing, but I got crapped by my ISP...
Main rig:
CPU: I7 920C0 @ 3.6Ghz (180*20)
Mobo: DFI UT X58 T3eH8
RAM: 12GB OCZ DDR3-1600 Platinum
GPU/LCD: GeForce GTX280 + GeForce 8600GTS (Quad LCDs)
Intel X25-M G2 80GB, 12TB storage
PSU/Case: Corsair AX850, Silverstone TJ07
What are the main things I should be tweaking to try and get the most out of it?
Crunch with us, the XS WCG team
The XS WCG team needs your support.
A good project with good goals.
Come join us,get that warm fuzzy feeling that you've done something good for mankind.
Crunch with us, the XS WCG team
The XS WCG team needs your support.
A good project with good goals.
Come join us,get that warm fuzzy feeling that you've done something good for mankind.
To fully understand the soul of a Turtle, you need to realize the slowness, of movement in real time.
It's thoughts are as a weapon designed to distract the real time, by letting it's awareness be way ahead of it's physical presents, you could misunderstand this as being high, in the terms of it reacting ahead of the actually event.
Is this making any sense for the normal mind, no, not at all, and that's why.........
It's a killer tank on four extremely sweet feet.
Last edited by Frisch; 03-23-2007 at 07:39 AM.
~Opinion about Dogs~~~~~~~Hug~~~~~~~~
~~~~
________________________________________________
Crunch with us, the XS WCG team
The XS WCG team needs your support.
A good project with good goals.
Come join us,get that warm fuzzy feeling that you've done something good for mankind.
The drivel, is what make us superior.
PS
You quoted my misspelling you evil man, it's really hard to grab a pen, not to talk about writing, being a Turtle.
~Opinion about Dogs~~~~~~~Hug~~~~~~~~
~~~~
________________________________________________
OK I think I'm in! Someone double check me and make sure
Now to see how I can get more than 3mbps average out of MJ12....
Crunch with us, the XS WCG team
The XS WCG team needs your support.
A good project with good goals.
Come join us,get that warm fuzzy feeling that you've done something good for mankind.
Main rig:
CPU: I7 920C0 @ 3.6Ghz (180*20)
Mobo: DFI UT X58 T3eH8
RAM: 12GB OCZ DDR3-1600 Platinum
GPU/LCD: GeForce GTX280 + GeForce 8600GTS (Quad LCDs)
Intel X25-M G2 80GB, 12TB storage
PSU/Case: Corsair AX850, Silverstone TJ07
Bookmarks