Skip to main content.
home | support | download

Back to List Archive

Re: index depth 1 off-site links

From: Bill Moseley <moseley(at)>
Date: Thu Apr 08 2004 - 02:04:44 GMT
On Wed, Apr 07, 2004 at 05:27:04PM -0700, Mark Greenaway wrote:
> I wanted a list of sites containing info on their main page
> I then want to index these sites
> I have it working using individual servers configured in the
> and combine them all when defining @servers
> This is very clumsy, particularly as the number of sites increases.

Well, one advantage of having a perl code as a config file is you can
programmatically create the config.  So your spider config could do
something like this untested bit of code:

open URLS, "/path/to/list/of/urls" or die $!;
while ( my $url = <URLS> ) {
    push @servers, build_config( $url );

were build_config() returns a hash ref of your parameters.

Would that work?

Bill Moseley
Received on Wed Apr 7 19:04:44 2004