You probably don't need the headers, but you can find out whats
happening by placing
debug => DEBUG_URL | DEBUG_SKIPPED | DEBUG_FAILED |
in your @servers
On Thursday, December 16, 2004, at 11:25 AM, Richard Morin wrote:
> Some pages seem to be missing from my indexes, so I'm
> wondering if the spidering process is failing in some
> silent manner.
> I have considered adding some code to spider.config
> (e.g., to tally visited pages and report the results),
> but before I go there, I thought I should ask if there
> is an easier way. Clues? Suggestions?
Received on Thu Dec 16 13:08:26 2004