david_42@hughes.net wrote on 10/09/2008 09:42 AM:
> Peter,
> 
> Thanks for the pointer. I knew that -S http was depreciated.  I switched to the 
> -S prog and was able to use robots.txt to exclude the directories causing the 
> problem.  A really simple solution that a small amount of documentation (of the 
> spider.pl) on the site would have made clear.

excellent. Mind sending a documentation patch so the next person doesn't
suffer your fate? You can patch against svn trunk here:

http://svn.swish-e.org/swish-e/trunk/prog-bin/spider.pl.in


-- 
Peter Karman  .  peter(at)not-real.peknet.com  .  http://peknet.com/

_______________________________________________
Users mailing list
Users@lists.swish-e.org
http://lists.swish-e.org/listinfo/users