Skip to main content.
home | support | download

Back to List Archive

Re: Does the <!-- Swishcommand noindex --> work whe

From: <Cleveland(at)not-real.mail.winnefox.org>
Date: Wed Jun 25 2003 - 14:48:50 GMT
Hello,

> # don't allow spidering of these specific files:
> Disallow: /otherpdfdir/pdfdoc.pdf
> Disallow: /otherpdfdir/pdfdoc2.pdf

I created a spider.txt file:
User-agent: *
Disallow: /citydirs/1857/1857full.pdf

and put it at http://www.oshkoshpubliclibrary.org/robots.txt

So, then I ran the spider again and noticed this:

+Fetched 2 Cnt: 22
http://www.oshkoshpubliclibrary.org/citydirs/1857/1857full.pdf 200 OK
application/pdf 5781897
parent:http://www.oshkoshpubliclibrary.org/citydirs/browse.html

Do I have something wrong in robots.txt?


--
Jody Cleveland
(cleveland@mail.winnefox.org) 
Received on Wed Jun 25 14:48:55 2003