Skip to main content.
home | support | download

Back to List Archive

RE: Server or documet limit on spider.pl

From: Lee Thompson <lee(at)not-real.kaim.com>
Date: Wed Jan 07 2004 - 18:01:42 GMT
I saw that happen when the spider ran into a page requiring a username
and password.

Lee Thompson



-----Original Message-----
From: swish-e@sunsite.berkeley.edu [mailto:swish-e@sunsite.berkeley.edu]
On Behalf Of Ander
Sent: Wednesday, January 07, 2004 10:46 AM
To: Multiple recipients of list
Subject: [SWISH-E] Server or documet limit on spider.pl


Hi all:

I'm using spider.pl to index a list of servers, which I create
dinamically  (from a database). When we have 2500 documents indexed
(more or less), spidering (and indexing, of course) stops.

It looks like spider.pl has a limit and It stops. But I've been looking
for any limitation there could be, but I don't find anything sospicious.

Can anyone help me?. 
Received on Wed Jan 7 18:01:56 2004