Skip to main content.
home | support | download

Back to List Archive

Re: Merge improvement (was: Bug2?)

From: Bill Moseley <moseley(at)not-real.hank.org>
Date: Wed Aug 07 2002 - 00:43:22 GMT
At 07:07 AM 08/02/02 -0700, Nikolay Raspopov wrote:
>Hello!
>
>I try merge two indexes 700000 words and 40000 words, after allocating 1.2
>Gb (!!!) of swapfile swish-e crash with "out of memory" error. Help.

I just uploaded to cvs a patch by Jose to improve merge.  It will be in the
next swish-daily (in about 7 hours from now).

Merge now uses memory more like normal indexing, plus can be used with -e
to reduce memory usage even more (which with Jose's recent -e update is
also much faster, and recommended).

Can you test and post your results?

BTW - You note above that you have a file with 700K words.  Think carefully
if you need to index so many "words" -- will people really search all
those?  People sometimes index a record's unique ID, which is not really
needed since if you know the record's ID you don't really need to search
for it.  Filtering out words that you know will not be searched can help
speed up the indexing process.


-- 
Bill Moseley
mailto:moseley@hank.org
Received on Wed Aug 7 00:46:54 2002