Skip to main content.
home | support | download

Back to List Archive

Robots.txt

From: David VanHook <dvanhook(at)not-real.mshanken.com>
Date: Thu Jul 24 2003 - 19:20:56 GMT
Sorry, can't find this in the docs or discussion list anyplace:

I need to set up robots.txt to disallow everybody except SWISH from
spidering my server.  I'm using spider.pl called through S-prog.  What is
the user-agent name for the SWISH spider?

Here's what I thought robots.txt would look like -- but this makes SWISH
abort as soon as it starts spidering:

User-agent: spider.pl
Disallow:

User-agent: *
Disallow: /


What's the appropriate "name" for spider.pl when run via S-prog?

Thanks --

David VanHook
Received on Thu Jul 24 19:21:12 2003