On Thu, Jun 03, 2004 at 01:12:58PM -0700, Justin Tang wrote:
> Sorry I wasn't clear enough. The example where I encountered is where I get
> the authority: 123.com, but if I go to http://123.com it would redirect to
> http://www.123.com, yet spider.pl does not follow it. Rather it read
> through http://123.com and finishes. I was wondering if there is anyway
> around it. Thank you.
See "same_hosts" in perldoc spider.pl
I wish I understood why Apache's Redirect directive requires a fully
qualified URL instead of just a path. Seems like if a Host: header is
supplied then it could build up the Location: from that and then someone
coming in on 123.com wouldn't get redirected to www.123.com.
Received on Thu Jun 3 14:46:38 2004