Skip to main content.
home | support | download

Back to List Archive

Re: an exclusion question

From: Mark Gaulin <gaulin(at)>
Date: Thu Jan 28 1999 - 15:00:56 GMT
Yann Stettler <> has a patch that
does just what you are asking. I do not see it on the ftp site
so you may want to contact him directly.

(We should get that HTTP NoContents patch posted, yes?)


At 12:32 PM 1/27/99 -0800, you wrote:
>Now that I've got the spider working, I'd like him to ignore some files.
>Specifically I'd like it to ignore files of the form *.wwwstat.html.  I
>tried adding .wwwstat.html to the NoContents directive but that reduced my
>contents to 0 since it excluded all .html files :-).  Other than stuffing
>them off in some other directory with I could then exclude with robots.txt,
>is there another way (when spidering) to tell it to ignore files that match
>a certain pattern?
>Bruce Bowler                             207.633.9600 (voice)
>Research Associate                       207.633.9641 (fax)
>Bigelow Laboratory for Ocean Sciences
>West Boothbay Harbor ME  04575 
Received on Thu Jan 28 07:03:21 1999