On Wed, Sep 15, 2004 at 03:49:42PM -0700, SRE wrote:
> Richard is saying he thinks SWISH should print the URL of
> the file it was attempting to spider when the problem occurred.
> I tend to agree with him.
Well, when I saw that message I knew which URL that was. I suspect
people familiar with robots.txt would get this:
RobotRules: Unexpected line: Sutton
But, spider.pl might be spidering more than one site, so it would be
nice to have the URL printed. I'll submit a patch to WWW::RobotRules.
Unsubscribe from or help with the swish-e list:
Help with Swish-e:
Received on Thu Sep 16 06:59:24 2004