if you're using spider.pl to spider your website, I suspect that 
configuring it to use authentication with the LWP NTLM module should do 
the trick.

http://search.cpan.org/~gaas/libwww-perl/lib/LWP/Authen/Ntlm.pm

Make sure you read the spider.pl docs for how to set up LWP and 
authentication.

pek

Gertjan Hofman scribbled on 7/6/06 1:53 PM:
> Dear SWISH users
> 
> I am slowly my initial swish-e difficulties but I need
> NTLM authentication to be able to index our intranet
> site. Digging around in the archive I see only a
> somewhat cryptic comment that 'ntlm authentication has
> to be done in the conf file'. It seems there samba
> tools around that could do this. Has this sucessfully
> been implemented in SWISH-e and does any one have a
> reference ?
> 
> Much appreciated
> 
> Gertjan
> 
> 
> __________________________________________________
> Do You Yahoo!?
> Tired of spam?  Yahoo! Mail has the best spam protection around 
> http://mail.yahoo.com 
> 

-- 
Peter Karman  .  http://peknet.com/  .  peter(at)not-real.peknet.com