I've updated spider.pl from a just downloaded version and the problem has
The matter was that when spider.pl was making the summary for a server(a
normal server) it crashed.
Now it's allright.
>Come to think of it, you are probably right. I have not seen the issue in
>long time, and in the meanwhile I am sure I have updated LWP to the newest
>version. Anyway, my main point is that the spider definitely is not
>It is actually quite versatile.
>[mailto:firstname.lastname@example.org]On Behalf Of Bill Moseley
>Sent: Wednesday, January 07, 2004 1:25 PM
>To: Multiple recipients of list
>Subject: [SWISH-E] RE: Server or documet limit on spider.pl
>On Wed, Jan 07, 2004 at 10:14:02AM -0800, Aaron Bazar wrote:
>> I have used spider.pl to grab over 100,000 documents over a period of 16
>> hours. I have, at other times, run into some issues where spider.pl
>> using up a huge chunk of memory. I never was able to determine why it
>> happened sometimes and not others.
>Maybe a different issue, but there was a problem a while back where the
>spider with some versions of LWP didn't free memory. This was discussed
>on both the swish-e list and the LWP list. Was it possible you were
>using different machines or different versions of LWP?
Txat euskalduna >>> http://www.euskalerria.org/solasgunea
Received on Fri Jan 9 11:24:18 2004