I want to set up a distributed database like structure using metatags
I wanted a list of sites containing info on their main page
I then want to index these sites
I have it working using individual servers configured in the config.pl
and combine them all when defining @servers
This is very clumsy, particularly as the number of sites increases.
If I could have a file, with just all the links in it, and then index
that file it would do the trick
However swish-e with both -S prog and -S http doesn't follow external
or off-site links.
I want to do this but only to that one file (max_depth=0)
How can I modify spider.pl or swishspider.pl to allow me to do this
Any help would be appreciated
Received on Wed Apr 7 17:27:46 2004