Skip to main content.
home | support | download

Back to List Archive

Re: Disabling zlib support

From: Bill Moseley <moseley(at)not-real.hank.org>
Date: Tue May 25 2004 - 21:44:34 GMT
On Tue, May 25, 2004 at 02:05:33PM -0700, William Osborne wrote:
> Ok, since swish indexes created on Windows apparently cannot be used on a
> Net-BSD 64-bit Alpha system.  I have decided to rebuild my indexes on the
> Net-BSD system (not as convienient but still do-able).
> 
> Swish-e built the indexes fine, but when I try to search the index I get
> the following message:
> 
> Warning: Failed to uncompress Property. zlib uncompress returned: -5. 
> uncompressed size: 17546 buf_len: -2024
> 
> I searched the archives of the swish-e mailing list and found a post that
> mentioned zlib compressed properties may not work properly on 64-bit
> systems.  So, I decided to rebuild swish-e, this time disabling zlib
> support.
> 
> I ran configure --without-zlib and near the end of the transcript, I got a
> message that said "Disabling compression support".  So I ran "make",
> followed by "make test" and "make install".

I think you can also just disable zlib compression by 

    PropCompressionLevel 0


> I rebuilt the index, and tried searching again.  I got the same message
> saying "Failed to uncompress property".
> 
> Why is swish-e still building compressed indexes when compiled with
> --without-zlib?  Is there a source file I need to edit to disable zlib
> support?

Because it's not really a zlib problem, I think.

Here's the long answer since I'm not sure of the right answer.


Here's basically how it works ( db_native.c: DB_ReadProperty_Native )

Swish wants to display a property for a given file number.  It first
reads in a table from the main index that gives the seek positions  in
the .prop file of all the properties for the given file number.

Then swish seeks to the specific property location in the .prop file and
reads an integer off the disk.  This is the length of the property as
stored on disk.

Next, another integer is read off disk.  This is the *uncompressed* size
of the property.  Swish uses that number later to create a buffer big
enough for zlib to do it's uncompressing in.

Then swish-e creates a buffer big enough to hold the compressed property
and reads the compressed property from disk.

Now swish-e has the property in (possibly) compressed format.  Then if
the compressed and uncompressed sizes are different swish creates
another buffer for the final uncompressed size and passes that and the
compressed buffer to zlib.

So, back to your problem.  The error message is showing a buffer size of
-2024.  Clearly, that's wrong.  So, either the wrong size is written to
the index when indexing, or it's wrong when read back off disk.  And my
guess is what's happening is that swish-e is reading from the wrong
position in the index file.  Another possibility is that there's a
problem in the way the integers are compressed/uncompressed to the
property file (they are compressed and written in network order to make
the index [mostly] portable).

Why would that happen?  Well, one possibility would be if using the
wrong libswish-e when indexing vs. searching.  Doesn't sound likely.
Possible if your environment changes when indexing to searching.

Other possibility?  I'm not sure.  Time to get in there and debug.
I think you and one other person has reported that error message lately,
so it's likely a bug in there and not just something odd about your data.

What I'd need is a (hopefully) single file that will demonstrate the
problem.  Then I'd use gdb and a few print statements to look at the
seek pointers.  It's not hard -- you can probably look at db_native.c
and find where to add a few printf's.

I'd print out the file number, the meta ID number and the compressed and
uncompressed length and the seek positions when storing (indexing) and
when searching and see if they make sense.  I'd first start by looking
at the two integers that are stored and see if they are read back the
same way.

So, think you can get me a very small sample set that will demonstrate
the problem?





-- 
Bill Moseley
moseley@hank.org
Received on Tue May 25 14:44:34 2004