Skip to main content.
home | support | download

Back to List Archive

Re: HTTP spidering - zero results

From: David Norris <dave(at)not-real.webaugur.com>
Date: Mon Jun 12 2000 - 10:27:46 GMT
Angel Parn wrote:
>     -i http://www.mysite.com/index.php3?date=2000/06/10

My guess would be that the Content Type header is not exactly what the
spider expects.  You can grab the HTTP headers with Lynx:

lynx -head -dump http://someurl/here/

If that is the problem then you may need to modify the spider.  I think
there are patches for this in the patches index.  There should be a few
theads in the archive regarding this, as well.

-- 
,David Norris
  Dave's Web - http://www.webaugur.com/dave/
  Dave's Weather - http://www.webaugur.com/dave/wx
  ICQ Universal Internet Number - 412039
  E-Mail - dave@webaugur.com
Received on Mon Jun 12 03:25:56 2000