Skip to main content.
home | support | download

Back to List Archive

Re: Running out of memory...

From: Bill Moseley <moseley(at)not-real.hank.org>
Date: Tue Apr 12 2005 - 16:40:14 GMT
On Tue, Apr 12, 2005 at 01:30:08AM -0700, CheHax wrote:
> Indexing Data Source: "External-Program"
> Indexing "./pl/full.pl"
> External Program found: ./pl/full.pl
> err: Ran out of memory (could not allocate 262144 more bytes)!

Looks similar as Peter mentioned, but swish allocates chunks of memory
pools in some operations so it's not too unusual to see the same
memory request.

> It happened after 14 hours of indexing.
> The problem is that we have 4GB of memory, but it happened before this
> hard limit was reached, as the system (windows server 2003) shows that
> the memory usage caped at 2.6GB. We have defined a max swap size of
> 4GB so programs can use up to 8GB minus memory used by the system.

Swish asks the OS for memory and if the OS won't allocate it then
you will get that message.  It's that simple.

Is there some other problem when indexing that many records?  I don't
know.  But, my first guess would be that the message is correct.


> We are trying to index 23 millions records, which are all quite small
> since we are using meta tags only (and group of meta tags).

I think I mentioned to you before that you might have problems trying
to index that many records.

> We launched Swish-e with the -e directive so that it would use hard
> drives instead of memory. Also note that temporary files were 730MB
> big for properties and 450MB for idx.temp file when the program
> crashed.

-e will help to a point, but still uses memory for each doc.  It's not
designed to be completely scalable.  That's one of the reasons google
can charge the big money.

    http://www.searchtools.com/tools/google-app.html

Or check out one of those other options I suggested on March 31st.

You might try indexing in smaller chunks -- say a million docs each,
and then start merging.  But, I suspect you won't have satisfactory
results.



-- 
Bill Moseley
moseley@hank.org

Unsubscribe from or help with the swish-e list: 
   http://swish-e.org/Discussion/

Help with Swish-e:
   http://swish-e.org/current/docs
   swish-e@sunsite.berkeley.edu
Received on Tue Apr 12 09:40:15 2005