At 09:27 AM 05/02/02 -0700, Hsiao Ketung Contr 61 CS/SCBN wrote:
>What does robots.txt does and
>what's your suggestion ?
Google is your friend.
If you were to use -S prog with spider.pl you can tell it to ignore
robots.txt. But, I'd suggest you try to get -S http method working first
before trying to tackle the -S prog / spider.pl setup with swish.
Received on Thu May 2 16:51:00 2002