On Thu, Jun 03, 2004 at 01:12:58PM -0700, Justin Tang wrote:
> Sorry I wasn't clear enough.  The example where I encountered is where I get
> the authority: 123.com, but if I go to http://123.com it would redirect to
> http://www.123.com, yet spider.pl does not follow it.  Rather it read
> through http://123.com and finishes.  I was wondering if there is anyway
> around it.  Thank you.

See "same_hosts" in perldoc spider.pl

I wish I understood why Apache's Redirect directive requires a fully
qualified URL instead of just a path.  Seems like if a Host: header is
supplied then it could build up the Location: from that and then someone
coming in on 123.com wouldn't get redirected to www.123.com.


-- 
Bill Moseley
moseley@hank.org