Skip to main content.
home | support | download

Back to List Archive

Re: A swish-e performance question

From: Ziegenhorn, Eric <eric.ziegenhorn(at)>
Date: Wed Oct 29 2003 - 18:08:17 GMT
On 17 Oct 2003 at 15:33, Patrick O'Lone wrote:

> All,
> > 
> > > There is a 2GB limit index file. So I think that you will reach
> > > the limit of swish easily. But, the solution must be easy. You can
> > > hack the source the code and make it 64 bit. The trick is easy.
> > > Look for fopen, fclose, fseek, fwrite... routines in the code,
> > > mainly in db_native.c
> > >
> > > I have seen this question before. Anyone in the list has made
> > > anything? If not, and if I find some time I will try to add this
> > > feature. Now, I am fighting against the PHP module.
> > >
> Can't you just compile with the -D_FILE_OFFSET_BITS=64
> -D_LARGEFILE_SOURCE? Shouldn't this just make it 64-bit assuming the
> file system the index resides on is LFS-aware? You shouldn't have to
> go through the code and convert fopen() to fopen64().

>Nope. There must be also changed the offset types. fseek and ftell
>uses a long (usually 32 bit). 
>Basically, fseek anf ftell should be moved to fseeko and ftello. This
>uses an off_t type instead of a long type. So there are more things
>to change to allow swish internal data to hold the 64 bit offsets.
>I am working on this issue now. As Bill points, this may be a feature
>for a future 2.4.1.
>This is true for UNIX/LINUX system but Ms Win32 is a different story
>as Dave told me.

Does anyone know that if Swish-e was changed to handle large files over 2GB
if memory usage would likely still be a limiting factor?  I'm on a machine
with 1.5GB RAM, but I'm trying to index 60GB of files.  Anyone know if this
will be possible in the near future or how I could make it possible?  Any
guesses on a release date for 2.4.1 with 64-bit file support?  I may be able
to work on it if it seems acheivable in the near term.

Received on Wed Oct 29 18:20:59 2003