Skip to main content.
home | support | download

Back to List Archive

Re: Indexing protected area

From: Jordan Hayes <jmhayes(at)>
Date: Thu Dec 07 2006 - 03:39:34 GMT
One thing you might consider, if you control the Apache server, is to 
put an alias in the URL tree that duplicates access to the content. 
I've found that once I start spidering a site, the statistics go all to 
heck, so filtering out your spidering can be a useful thing -- putting 
them under a different URL (or a different virtual server, even better!) 
is a good way to go.

Received on Wed Dec 6 19:39:36 2006