Skip to main content.
home | support | download

Back to List Archive

Re: use_md5 fails

From: Bill Moseley <moseley(at)not-real.hank.org>
Date: Thu Feb 20 2003 - 07:11:12 GMT
On Wed, 19 Feb 2003, Thomas McDonald wrote:

> Well I started using the spider.pl because the swishspider just didn't seem
> to be doing the trick.  Any idea why use_md5 is failing me?  I have two urls
> that point to the same page:

Was this solved with the cookies?

If not, the great thing about perl is it's easy to test with.  For
example, I'd just search for md5 in spider.pl and add some print STDERR
statements and print out the md5 keys from the docs and see if they are
the same or different.  If they are the same then there's some bug that is
not seeing that (I believe there's a debug option (DEBUG_SKIPPED?) that
will print out why a doc is skipped, including because of md5.  If they
are different then maybe print the docs to a file and md5 or diff them and
see what happens.

You don't have to spider the entire site to test, just list the two URLS
as an array ref in the spider confile file.

I'm off with a slow connection, using Windows, and short on sleep so I'm
not testing my above suggestions this time.  Sorry.  Let us know what you
find out.

Thanks,

-- 
Bill Moseley moseley@hank.org
Received on Thu Feb 20 07:11:41 2003