Skip to main content.
home | support | download

Back to List Archive

Intrusive spiders/crawlers

From: Paul Thomas <paul(at)>
Date: Sat Jun 09 2001 - 19:06:04 GMT

Sorry if this post is off topic, I couldn't resist calling
upon the collective crawler wisdom and experience of the 
knowledgeable people on this list. I

I have some web browseable archives on my web server. I use
a robot.txt file to designate off-limits web directories and
also tried protecting private archive directories with .htaccess
files. However there are still some spiders that just come
through and gobble everything up anyways.

Any suggestions how to keep these intruders out?


--Paul T.

"So enjoy yourselves, dabble in sound, paint your pictures, turn
on your mind and gently float downstream..." -- George Martin, Beatles's
Received on Sat Jun 9 19:25:26 2001