On Wed, Sep 15, 2004 at 01:37:50PM -0700, Richard Morin wrote:
> The spidering script certainly knows where it's looking, at
> any given time.  Does the module not return a status code?  Sigh.

perldoc LWP::RobotUA for how it works.  Like I said, it's not
spider.pl that's fetching that file.  It would be nice if that module
gave a better message -- you could bring that up on the LWP list as
it only makes sense if you understand the context of the message.

> In any case, your question about my robots.txt file presumes
> that I have a robots.txt file (as far as I know, I do not).
> If I had the URL, I'd know where to start looking...

Well that's easy: it's http://<yourserver>/robots.txt

-- 
Bill Moseley
moseley@hank.org

Unsubscribe from or help with the swish-e list: 
   http://swish-e.org/Discussion/

Help with Swish-e:
   http://swish-e.org/current/docs
   swish-e@sunsite.berkeley.edu