Skip to main content.
home | support | download

Back to List Archive

Re: New index proccess in development version

From: <jmruiz(at)>
Date: Wed Aug 29 2001 - 17:04:25 GMT
HI Bill,

I think that the latest code is not well optimized with very long files.
21 MB after pdftotext (Chris data)is too much... 
I need to improve it.

I have tested the unfreed memory issue in some OSs with a very
simple program as you told me. I have tested it in solaris 2.6.

The program is:

int main()
char *p;
int pid =getpid();
char ps[100];

sprintf(ps,"ps -p %d -o osz",pid);

p=malloc(100000000); /* ~ 100 MB */
p=malloc(10000000); /* ~ 10 MB */

In solaris I get the following:
 103  << Before malloc
13311  << After malloc
13311   << after first free
13311   << second malloc - memory reused
13311   << after second  free

So, in Solaris, the memory is not returned to the OS, even if we do
not longer use part of it. 

So, for a very long file, the new code can be using a lot of memory
and, even freed and no longer used by swish, this memory will not
be returned to the OS.


On 29 Aug 2001, at 7:49, Bill Moseley wrote:

> Jose,
> When I was indexing I had similar upper memory usage between the
> different versions -- just that the newer version got to the upper
> memory limit and stayed there.  But maybe that depends on the type of
> documents that are being indexed?  In other words, you think it's
> possible that Chris has files that are making swish use more memory
> than in the previous version?
> Is it really true that BSD doesn't free the memory?  I just assumed
> that the reported number was not reduced after freeing memory.
Received on Wed Aug 29 17:10:38 2001