> On Mon, 15 Jul 2002, Michael wrote:
> > merges are slower than 2.05 -- hard to say quantitatively now. My 
> > guess is around 2:1 for files as above at least. I used to be able to 
> > merge the whole thing in a day or two but based on the benchmark of 
> > above, it would take 80-100 hrs or more.
> 
> I really can not remember.  IIRC, merge in 2.1-dev works like it did
> in 2.05, but 2.1's indexing is much faster and uses less RAM.  Merge
> in 2.1 does not take advantage of all the compression features of
> normal indexing.  Hopefully that will be fixed sometime...
> 
> The index format is different with 2.1, so that might be one reason
> it's slower than 2.05.
> 
> You have two options over merging.  One is to index everything at
> once, and the other is to specify more than one index file on the
> command line when searching.
> 
> And you can use -e if you are short on RAM.
> 

Neither of those is particularly appealing. We have 4-5 years worth 
of data and accumulate new data daily. The indexes are broken down 
by directory per month so not merging would imply searching up to 50 
index files for a full search. Merging monthly is OK, but takes a 
LONGGGG time even for a one month add but since swish does not 
have an "exclude dir/file" switch, indexing the entire site minus the 
last two months directories is not doable, but is easily accomplished 
with a merge.

I tried a whole site merge just to measure the time, it took about 45 
minutes to create a 90meg index file in economy mode.
-- LESS THAN A MERGE for 3megs + 30 megs which takes 90 minutes. This 
does not seem right.

I'd like some suggestions...

Michael
Michael@Insulin-Pumpers.org