Skip to main content.
home | support | download

Back to List Archive

Re: URL-fixing with callback routines for spider.pl

From: koszalekopalek <koszalekopalek(at)not-real.interia.pl>
Date: Thu May 19 2005 - 18:51:29 GMT
Bill Conlon wrote:
> have you turned on debugging and compared the response headers that 
> spider.pl sees with those that a 'normal' user agent gets?

Err, not yet.

> Maybe your server is configured to respond differently to certain user 
> agents?

Actually I thought of fiddling with the user
agent in spider.pl:

330     # set default agent for log files
331
332     $server->{agent} ||= 'swish-e spider 2.2 http://swish-e.org/';

I guess I'll try that tomorrow. Thanks for all
your help!

A.

----------------------------------------------------------------------
Startuj z INTERIA.PL! >>> http://link.interia.pl/f186c 
Received on Thu May 19 11:51:30 2005