Skip to main content.
home | support | download

Back to List Archive

Re: Re: Spider Design Flaw!

From: David Norris <dave(at)>
Date: Sat Feb 26 2000 - 15:33:58 GMT
Bill Moseley wrote:
> >  <META HTTP-EQUIV="Refresh" CONTENT="1;
> >URL=">
> Ouch!  You have two one second browser redirects just to get to that one page.

Hmm, (excuse my pet peeve ;) I'd agree with Bill that this is an
administration problem more than a spider problem...  Although, I would
never use, recommend, nor even hint at the existence of a META redirect
(gritting teeth as I type this) many folks do for various (good, bad,
indifferent) reasons.

> Might I recommend that you get rid of the META refresh tags and use
> Apache's redirect features.

( does appear to be running a semi-recent Apache 
 according to 'lynx -head -dump')

>   Redirect /search/database/index.html

I would not quite do exactly that...  Apache's default redirect status
is Temporary (HTTP 302?).  You want to have Apache tell the browser that
this is a permanent (HTTP 301) change (in the event of a miracle and the
browser actually follows specs by auto-updating it's bookmarks).  

The Apache syntax:
  Redirect [status code] url-path url

For example:
  Redirect Permanent /search/database/index.html


  Redirect 301 /search/database/index.html

And, then patching the spider somehow since this may be an issue with
folks indexing servers where they have no control.

,David Norris
  Open Server Architecture Project -
  Dave's Web -
  ICQ Universal Internet Number - 412039
  E-Mail -
Received on Sat Feb 26 07:31:02 2000