Skip to main content.
home | support | download

Back to List Archive

Re: tracking the completeness of spidering?

From: Bill Conlon <bill(at)not-real.tothept.com>
Date: Thu Dec 16 2004 - 21:08:20 GMT
You probably don't need the headers, but you can find out whats 
happening  by placing

         debug => DEBUG_URL | DEBUG_SKIPPED | DEBUG_FAILED | 
DEBUG_HEADERS,

in your @servers

bill

On Thursday, December 16, 2004, at 11:25  AM, Richard Morin wrote:

> Some pages seem to be missing from my indexes, so I'm
> wondering if the spidering process is failing in some
> silent manner.
>
> I have considered adding some code to spider.config
> (e.g., to tally visited pages and report the results),
> but before I go there, I thought I should ask if there
> is an easier way.  Clues?  Suggestions?
>
> -r
>
Received on Thu Dec 16 13:08:26 2004