Skip to main content.
home | support | download

Back to List Archive

Reading the spider report

From: Justin Tang <justin.tang(at)not-real.positionresearch.com>
Date: Wed Jun 02 2004 - 17:33:23 GMT
Hi:
  After running spider.pl, it produced a summary of the site it had just
spidered.  I was wondering if there is a guide to reading the summary, as to
what each field means.  This is what the report looks like:

Summary for: http://www.123.com/

    Duplicates: 796  (1.5/sec)
MD5 Duplicates:   1  (0.0/sec)
Off-site links: 164  (0.3/sec)
       Skipped: 114  (0.2/sec)
   Unique URLs: 108  (0.2/sec)
    robots.txt:   3  (0.0/sec)

Thanks!

-Justin
Received on Wed Jun 2 10:33:24 2004