Skip to main content.
home | support | download

Back to List Archive

Re: Indexing files without an extension

From: Bill Moseley <moseley(at)not-real.hank.org>
Date: Tue Feb 07 2006 - 06:02:47 GMT
On Mon, Feb 06, 2006 at 11:30:56PM -0600, dennis lastor wrote:
> When run the spider I get the following error(s):
> 
> Summary for: http://Internal_Site/WebHome
> Connection: Close: 1  (1.0/sec)
>       Unique URLs: 1  (1.0/sec)
>        robots.txt: 1  (1.0/sec)

robots.txt is telling it not to index that page.


-- 
Bill Moseley
moseley@hank.org

Unsubscribe from or help with the swish-e list: 
   http://swish-e.org/Discussion/

Help with Swish-e:
   http://swish-e.org/current/docs
   swish-e@sunsite.berkeley.edu
Received on Mon Feb 6 22:02:47 2006