On Wed, Jan 07, 2004 at 10:14:02AM -0800, Aaron Bazar wrote:
> FYI,
> 
> I have used spider.pl to grab over 100,000 documents over a period of 16
> hours. I have, at other times, run into some issues where spider.pl starts
> using up a huge chunk of memory. I never was able to determine why it
> happened sometimes and not others.

Maybe a different issue, but there was a problem a while back where the
spider with some versions of LWP didn't free memory.  This was discussed
on both the swish-e list and the LWP list.  Was it possible you were
using different machines or different versions of LWP?


-- 
Bill Moseley
moseley@hank.org