Skip to main content.
home | support | download

Back to List Archive

Re: question about robots.txt

From: Greg Fenton <greg_fenton(at)not-real.yahoo.com>
Date: Wed Jul 28 2004 - 04:33:12 GMT
--- Bill Conlon <bill@tothept.com> wrote:
> I have a robots.txt file:
> 

Is the robots.txt file located at /robots.txt ?

Maybe the spider can't get to it because of authentication (I can't get
to http://beowulf3.tothept.com/robots.txt )



=====
Greg Fenton
greg_fenton@yahoo.com


	
		
__________________________________
Do you Yahoo!?
New and Improved Yahoo! Mail - 100MB free storage!
http://promotions.yahoo.com/new_mail 
Received on Tue Jul 27 21:33:53 2004