Spiders and crawlers

Evan Leibovitch evan-ieNeDk6JonTYtjvyW6yDsg at public.gmane.org
Thu Apr 1 21:56:35 UTC 2010


Hi all,

I'm looking to implement a spidering system intended to look through a bunch
of catalog websites, in order to track changes to those catalogs (with the
help of a backend MySQL system).

The Wikipedia entry for "web crawler" returns a lot of interesting choices;
I'm wondering is anyone here has experience in either writing one or using
an existing open source one. I'm hoping for something that is reasonably
configurable so that one doesn't need to know a language like C or Java to
make minor config changes.

Any help is appreciated.

-- 
Evan Leibovitch
evan-ieNeDk6JonTYtjvyW6yDsg at public.gmane.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gtalug.org/pipermail/legacy/attachments/20100401/04d78782/attachment.html>


More information about the Legacy mailing list