Skip to main content.
home | support | download

Back to List Archive

robots.txt

From: J Robinson <jrobinson852(at)not-real.yahoo.com>
Date: Mon Oct 31 2005 - 14:42:06 GMT
Hello;

I've been getting some complaints from people on our
intranet that spider.pl isn't obeying Disallows in
robots.txt.

Any tips on how I can debug this? Is there a debug
flag for spider.pl that shows robots.txt being parsed
and/or urls being matched against it, or anything like
that?

Thanks in advance,
  jrobinson


	
		
__________________________________ 
Yahoo! Mail - PC Magazine Editors' Choice 2005 
http://mail.yahoo.com
Received on Mon Oct 31 06:42:27 2005