Skip to main content.
home | support | download

Back to List Archive

Re: Memory issues even with -e

From: Bill Moseley <moseley(at)not-real.hank.org>
Date: Thu Oct 14 2004 - 03:59:14 GMT
On Wed, Oct 13, 2004 at 08:57:44AM -0700, Tuc wrote:
> 	I'm trying to index a few large sites, which I copy locally using
> "webcopy". Once I finish the copy, I run it with "-e" . It ran for 12 or so

12 hours is a long time to wait.

Are you indexing large files, or just a lot of files?

> 	I saw that I could do it by individual directory, then use the "-M"
> to merge, or allow the searches to use "-f". I think that if I do the "-M"
> that even with "-e" it will cause the memory allocation issue. And I'm 
> afraid with the "-f" that the search will take too long to join them all.

I'm not sure if -M will help that much.  You may find the memory
requirements are similar.

Using -f with multiple indexes shouldn't be that much slower.  There's
the overhead of opening the extra indexes.  If you keep the indexes
open between requests then you can avoid that.  Searching will be
somewhat slower than searching a single index, but I wouldn't expect
much.  If you try and sort by a property other than rank and output a
large result set than that can be slower -- swish has to read the
property file for each result in that case.

Maybe Jose will have more to offer.  You could also try using a
development snapshot (swish-daily build) -- I think Jose has done dome
work on the indexing code.

Jose, would you think the btree backend would deal with the large data
sets any better?

-- 
Bill Moseley
moseley@hank.org

Unsubscribe from or help with the swish-e list: 
   http://swish-e.org/Discussion/

Help with Swish-e:
   http://swish-e.org/current/docs
   swish-e@sunsite.berkeley.edu
Received on Wed Oct 13 20:59:28 2004