Skip to main content.
home | support | download

Back to List Archive

external spider

From: Mark Greenaway <mark(at)not-real.cres20.anu.edu.au>
Date: Thu Apr 08 2004 - 05:28:56 GMT
OK I am not that familar with perl
does anyone habe a modified copy of spider.pl or swishspider that allows
swish-e to index off-site of external links as well as local ones.
As in my previous post, I only want to access a single file on a number of
web sites to index on a series of meta tags.
Like a distributed database.

I would like to have a single file with alink to all the main pages on sites
that contain my metatags.  Then you point the spider at that file and it indexes
them all.

I have only managed it using multiple servers in config.pl @servers line
I would really like to have it just do the file with links and have
max_depth set so it just does the top level of the links only.

Any ideas or clues would be greatly appreciated.  As I said though my experience
with perl is very limited.
Received on Wed Apr 7 22:28:56 2004