On Sep 15, 2004, at 1:35 PM, Bill Moseley wrote:
> On Wed, Sep 15, 2004 at 01:28:23PM -0700, Richard Morin wrote:
>> Clearly, some program found "Sutton" somewhere and wasn't happy,
>> but this isn't enough information to allow the user to debug
>> anything. Could we have a more comprehensive error message?
> No, sorry. That's not a message generated from any of the code we
> control -- rather from (I suspect) the module that parsers the
> robots.txt file. Do you have an invalid line in your robots.txt file
> with the word "Sutton"?
The spidering script certainly knows where it's looking, at
any given time. Does the module not return a status code? Sigh.
In any case, your question about my robots.txt file presumes
that I have a robots.txt file (as far as I know, I do not).
If I had the URL, I'd know where to start looking...
Received on Wed Sep 15 13:45:38 2004