Alrighty then,

I set kern.maxdsiz="2G" and fired up a incremental index of about 200K 
files (with -e). everything went fine! Feeling emboldened, I started the 
same thing with ~ 600K files. It hasn't run out of memory in the final 
stage, but it has been sitting here:

no words removed.
Writing main index...
Sorting words ...
Sorting 1,253,775 words alphabetically
Writing header ...
Writing index entries ...
   Writing word text: Complete
1,253,775 unique words indexed.
Sorting property: ipnstockweight

for about 2 hours. i am thinking that i will be pronouncing it dead 
shortly and i guess trying it without -e, see if the 2G process limit 
helps there.

of course, i can always just live without incremental, but at this point, 
it is fair to say that I am mental about incremental...(get it?).

Brad
---------------------
Brad Miele
VP Technology
IPNStock.com
866 476 7862 x902
bmiele@ipnstock.com

On Fri, 20 Oct 2006, Peter Karman wrote:

>
>
> David L Norris scribbled on 10/19/06 9:44 PM:
>
>> But if your OS and CPU are not both 64-bit (and swish-e also isn't
>> 64-bit) there's probably a hard limit of 2 GB per process.
>>
>
> Brad, Mr Intrepid Hero, is this box a 64-bit machine? Is that version of FreeBSD
> a 64-bit OS? I suspect Dave might be on to something here. I see this issue (2g
> ram limit) frequently right now with one of our db servers, where it's stuck on
> a 32bit machine and runs out of memory even though there's plenty of ram available.
>
> Here's a little C program you can use to test the mem limit on your machine:
>
> ---------------- snip here ------8<------------------------
> #include <stdio.h>
> #include <stdlib.h>
>
> int main ( void )
> {
>     size_t siz = 100 * 1024 * 1024 ;
>     size_t idx = 1 ;
>     void *ptr ;
>
>     for (;;)
>     {
>         ptr = malloc ( siz * idx );
>         if ( ! ptr )
>             break ;
>         free ( ptr );
>         idx ++ ;
>     }
>     printf ( "Max malloc %d * 100 MB \n", idx - 1 );
>     return ( 0 );
> }
> ---------------- snip here ------8<------------------------
>
> just compile it like:
>
>  cc -i checkmem.c -o checkmem
>
> and run:
>
>  ./checkmem
>
> that'll tell you how much memory a single process can allocate on your machine.
>
> -- 
> Peter Karman  .  http://peknet.com/  .  peter(at)not-real.peknet.com
>
>