Skip to main content.
home | support | download

Back to List Archive

Re: HTTP spidering - zero results

From: Angel Parn <angel(at)>
Date: Mon Jun 12 2000 - 10:53:54 GMT
> My guess would be that the Content Type header is not exactly what the
> spider expects.  You can grab the HTTP headers with Lynx:
> lynx -head -dump http://someurl/here/
> If that is the problem then you may need to modify the spider.  I think
> there are patches for this in the patches index.  There should be a few
> theads in the archive regarding this, as well.

Lynx response:

HTTP/1.1 200 OK
Date: Mon, 12 Jun 2000 10:49:56 GMT
Server: Apache/1.3.6 (Unix) PHP/3.0.12 mod_perl/1.21 mod_ssl/2.2.8
Connection: close
Content-Type: text/html

Seems bad. What I have to do ?

Received on Mon Jun 12 06:59:58 2000