Skip to main content.
home | support | download

Back to List Archive

Re: Change to spider.pl to support Basic authentication

From: Bill Moseley <moseley(at)not-real.hank.org>
Date: Sat Feb 02 2002 - 14:12:01 GMT
On Fri, 1 Feb 2002, Darryl Friesen wrote:

> I looked through the archives and docs a bit (granted, not very thoroughly)
> and didn't find too much on spidering sites protected by Basic
> authentication, so I made this 2 line change to the spider.pl program that
> came with SWISH-E 2.1-dev-25.  (sorry about the long lines here)
> 
> [moondog] diff spider.orig.pl spider.pl
> 268a269
> >     $request->authorization_basic( split(':', $server->{credentials}, 2) )
> if ($server->{credentials});
> 661a663
> >         $request->authorization_basic( split(':', $server->{credentials},
> 2) )  if ($server->{credentials});
> 
> and you need to add an extra config option to your spider.config file like
> this:
> 
>     credentials => 'username:password'

Thanks!

I'm inclined to say let's use the method David suggested:

     http://user:pass@swish-e.org/protected

because you can specify more than one URL in the spider config file.  But,
for probably most cases people are only spidering one URL.

How about this:  Use credentials in the url, if found, or use credentials
specified in the config like above, or, if "credential => 'prompt:30' it
will prompt for username and password but only wait 30 seconds?



-- 
Bill Moseley moseley@hank.org
Received on Sat Feb 2 14:12:30 2002