Skip to main content.
home | support | download

Back to List Archive

Re: Exclusions for HTTP indexing method?

From: Ron Samuel Klatchko <rsk(at)>
Date: Tue Jan 11 2000 - 17:57:17 GMT
"Halasz, Steve" wrote:
> I think that Swish-e is brilliant. However it doesn't seem to allow
> exclusion of certain directories or files when indexing with the HTTP
> method. I'm hoping for some way to do exclusions along the lines of
> FileRules pathname /some/directory
> but for HTTP.

You can use robot exclusion rules for this.  Create a file called
robots.txt in your web servers document root (or add to it if it's
already there):

User-agent: swishspider
Disallow: /some/directory

           Ron Samuel Klatchko - Software Jester
            Brightmail Inc -
Received on Tue Jan 11 13:10:22 2000