Skip to main content.
home | support | download

Back to List Archive

Disallow in Robots.txt

From: James <swish.enhanced(at)>
Date: Mon Jan 08 2007 - 09:18:07 GMT
Is there a way for other web-masters to disallow Swish-e from crawling their
site(s) and is there a way to declare what bot I am?  For instance, I always
put the following in my robots.txt files for my web-sites:

User-agent: psbot
Disallow: /

Is there some kind of configuration file that declares what bot (User-agent)
I am (when using Swish-e) and can that be changed to something I customize
and something I can declare publicly so that anyone can disallow my user

I ask these things in general because I know that Swish-e has a polite
spider, obeying Robots.txt and noindex, nofollow directives.

Due to deletion of content types excluded from this list by policy,
this multipart message was reduced to a single part, and from there
to a plain text message.
Received on Mon Jan 8 01:18:07 2007