Hello,

> # don't allow spidering of these specific files:
> Disallow: /otherpdfdir/pdfdoc.pdf
> Disallow: /otherpdfdir/pdfdoc2.pdf

I created a spider.txt file:
User-agent: *
Disallow: /citydirs/1857/1857full.pdf

and put it at http://www.oshkoshpubliclibrary.org/robots.txt

So, then I ran the spider again and noticed this:

+Fetched 2 Cnt: 22
http://www.oshkoshpubliclibrary.org/citydirs/1857/1857full.pdf 200 OK
application/pdf 5781897
parent:http://www.oshkoshpubliclibrary.org/citydirs/browse.html

Do I have something wrong in robots.txt?


--
Jody Cleveland
(cleveland@mail.winnefox.org)