On Wed, Sep 15, 2004 at 03:49:42PM -0700, SRE wrote:
> Richard is saying he thinks SWISH should print the URL of
> the file it was attempting to spider when the problem occurred.
> I tend to agree with him. 

Well, when I saw that message I knew which URL that was.  I suspect
people familiar with robots.txt would get this:

    RobotRules: Unexpected line: Sutton

But, spider.pl might be spidering more than one site, so it would be
nice to have the URL printed.  I'll submit a patch to WWW::RobotRules.



-- 
Bill Moseley
moseley@hank.org

Unsubscribe from or help with the swish-e list: 
   http://swish-e.org/Discussion/

Help with Swish-e:
   http://swish-e.org/current/docs
   swish-e@sunsite.berkeley.edu