Hello,
> Try this out (changing "site" to be your site). This is
> from the WWW::RobotRules man page.
>
> moseley@bumby:~/apache$ cat r.pl
Ok, I made that file, ran it, and got this output:
[cleveland@storm cleveland]$ perl r.pl
==========
User-agent: *
Disallow: /citydirs/1857/1857full.pdf
Disallow: /citydirs/1866/1866full.pdf
Disallow: /citydirs/1868/1868full.pdf
Disallow: /citydirs/1869/1869full.pdf
Disallow: /citydirs/1872/1872full.pdf
Disallow: /citydirs/1876/1876full.pdf
Disallow: /citydirs/1879/1879full.pdf
Disallow: /citydirs/1880/1880full.pdf
Disallow: /citydirs/1883/1883full.pdf
Disallow: /citydirs/1884/1884full.pdf
Disallow: /citydirs/1886/1886full.pdf
Disallow: /citydirs/1889/1889full.pdf
Disallow: /citydirs/1891/1891full.pdf
Disallow: /citydirs/1893/1893full.pdf=========
not allowed
http://www.oshkoshpubliclibrary.org/citydirs/1857/1857full.pdf
allowed http://www.oshkoshpubliclibrary.org/citydirs/1857/1857fullx.pdf
So, that works fine. But, if I spider with swish-e, it still doesn't
skip those files. Is there something else I'm missing?
Jody
Received on Fri Jun 27 13:11:57 2003