Skip to main content.
home | support | download

Back to List Archive

Re: Spider Design Flaw!

From: PropheZine Owner <bob(at)not-real.prophezine.com>
Date: Sun Feb 27 2000 - 13:27:33 GMT
Hi:

I understand about the server command to tell the browser but I was under
the impression that this may be server specific.  So we chose the meta tag
route for transportability sake.  Not that we have moved the site often but
we have moved portions to other servers on occasion.

I was using Apache on a server in California but I just moved the site and
believe it is Red Hat Linux that is used now.  At least when I telnet in
that is what it says.

Until 1 1/2 weeks ago PropheZine ran on 3 servers: California, Canada and NY
(2 gig on the server in NY).  I combined the California and Canadian sites
into one.  So transportability is important.

Although I am using the FS method currently I want to play and learn the
HTTP method for the future.

Thanks for the great software.  I along with everyone else appreciate the
effort you put into this.    code.

Bob

-----Original Message-----
From: swish-e@sunsite.berkeley.edu
[mailto:swish-e@sunsite.berkeley.edu]On Behalf Of David Norris
Sent: Saturday, February 26, 2000 10:32 AM
To: Multiple recipients of list
Subject: [SWISH-E] Re: Spider Design Flaw!


Bill Moseley wrote:
> >  <META HTTP-EQUIV="Refresh" CONTENT="1;
> >URL=http://www.prophezine.com/search/database/archives.html">
> Ouch!  You have two one second browser redirects just to get to that one
page.

Hmm, (excuse my pet peeve ;) I'd agree with Bill that this is an
administration problem more than a spider problem...  Although, I would
never use, recommend, nor even hint at the existence of a META redirect
(gritting teeth as I type this) many folks do for various (good, bad,
indifferent) reasons.

> Might I recommend that you get rid of the META refresh tags and use
> Apache's redirect features.

(prophezine.com does appear to be running a semi-recent Apache
 according to 'lynx -head -dump http://www.prophezine.com/')

>   Redirect /search/database/index.html
> http://www.prophezine.com/search/database/archives.shtml

I would not quite do exactly that...  Apache's default redirect status
is Temporary (HTTP 302?).  You want to have Apache tell the browser that
this is a permanent (HTTP 301) change (in the event of a miracle and the
browser actually follows specs by auto-updating it's bookmarks).

The Apache syntax:
  Redirect [status code] url-path url

For example:
  Redirect Permanent /search/database/index.html
http://www.prophezine.com/search/database/archives.shtml

or

  Redirect 301 /search/database/index.html
http://www.prophezine.com/search/database/archives.shtml

And, then patching the spider somehow since this may be an issue with
folks indexing servers where they have no control.

--
,David Norris
  Open Server Architecture Project - http://www.opensa.org/
  Dave's Web - http://www.webaugur.com/dave/
  ICQ Universal Internet Number - 412039
  E-Mail - dave@webaugur.com
Received on Sun Feb 27 08:32:52 2000