Best tools for making a spider?

[Zoltan (ZEE4)]

Evan Leibovitch wrote:
> This isn't meant to start a flamewar, honestly.
>
> I'm just wondering if there are some languages that are either 
> optimized or otherwise more suitable than others for the specific task 
> of writing a specialized web spider.
>
> So far the people I've spoken to say this is a perfect task for Ruby, 
> but I don't know it at all. There is no previous baggage on this 
> particular project so we can choose any tool we want -- but once 
> chosen I'd like to stay with it. Having some existing open source 
> templates or existing code to build upon is always nice too.
>
> I'm not going to be the programmer (we do want working code, after all 
> :-) ) but I do have some say in the tech to be used. Any suggestions 
> or pointers of where to explore this particular question are appreciated.
>
> - Evan
>
> -- 
> The Toronto Linux Users Group.      Meetings: http://gtalug.org/
> TLUG requests: Linux topics, No HTML, wrap text below 80 columns
> How to UNSUBSCRIBE: http://gtalug.org/wiki/Mailing_lists
>
>
>
>
Googling "php pear spider" or "php spider" turns up a bunch of results. 
I'd take a look at some of these projects for ideas. Also, the reason I 
included "pear" in the search is that the pear library (pear.php.net) 
has a lot of pre-written code for all kinds of things, including networking.

Zoltan


www.yyztech.ca - reviews. news. cybercafes.

--
The Toronto Linux Users Group.      Meetings: http://gtalug.org/
TLUG requests: Linux topics, No HTML, wrap text below 80 columns
How to UNSUBSCRIBE: http://gtalug.org/wiki/Mailing_lists