Hello;

I've been getting some complaints from people on our
intranet that spider.pl isn't obeying Disallows in
robots.txt.

Any tips on how I can debug this? Is there a debug
flag for spider.pl that shows robots.txt being parsed
and/or urls being matched against it, or anything like
that?

Thanks in advance,
  jrobinson


	
		
__________________________________ 
Yahoo! Mail - PC Magazine Editors' Choice 2005 
http://mail.yahoo.com