Skip to main content.
home | support | download

Back to List Archive

Re: robots.txt File

From: Michael Derrick Whang <mwhang(at)>
Date: Thu Apr 26 2001 - 06:34:03 GMT
Thanks Dave for the input...I'll look into the suggested site.


On Wed, 25 Apr 2001, Dave Stevens wrote:

> >I recently received a FILE NOT FOUND error on the server log for a
> >robots.txt file.
> >
> >Any clues to why SWISH-e is looking for this file?
> A robots.txt file is a file you put on your site to control the behavior of
> Web spiders, or robots.  For more info on robots.txt files work, surf
> .  Well behaved spiders look for the file to see
> if it's OK to index the site.
> FWIW, I'd suggest upgrading to 2.x if at all possible.  The difference is
> astounding, in terms of speed and stability.
> Dave
Received on Thu Apr 26 06:35:21 2001