Skip to main content.
home | support | download

Back to List Archive

Re: Why does Swish-E hate my Opteron?

From: Bill Moseley <moseley(at)not-real.hank.org>
Date: Fri Sep 10 2004 - 20:15:04 GMT
On Fri, Sep 10, 2004 at 12:01:14PM -0700, Fennec Foxen wrote:
> Hmm. This seems to have an effect on the indexing process only, not
> the search. Should I be doing something else to show property reads on
> search, or will search just never tell me these things? :)

It's showing both.  When indexing it writes the props, and at the end
of indexing it reads them back in for sorting.  Which makes me wonder
why you don't get the error when indexing but you do when searching.

Also, when you do a test like below make sure it actually generates
the error you expect -- it may be that it doesn't show up with just
one file -- maybe you have to index a large number of files before
the problem shows up.

Also, if it's NOT happening on indexing but only on searching then
that makes me wonder if you are not somehow using a different
libswish-e for indexing than when searching.  That's been the case in
the past with other with similar problems.  (Still, there seems to be
a 64bit platform issue someplace.)

Unfortunately, you might need to capture a lot of output to see the
problem.  If that's the case maybe a perl script could look for the
unmatched numbers (i.e. read size or read location for a given
file:propIDX don't match the read size/location).  Again, I'm just
guessing that this is the problem.

Another possibility would be some unrelated code is causing a buffer
overflow and trashing some bit of data.

But, the basic problem is that zlib is being called with invalid
numbers and those should come from the property file correctly (which
the output from DEBUG_PROP should show).

You might want to capture indexing then use gdb when searching to
catch the error.  Might be something I'd try, too.

Sorry for not having better suggestions.  My memory is a but fuzzy --
maybe once I see that UPS tracking number I'll remember more....

Jose may need to help on this one, which would likely be Monday before
he's online again.




> 
> Here is the (one-file) indexing operation output.
> ---------
> Write Prop: file 1  PropIDX 0  (meta 6) seek: 8
> data=[uncompressed_len: 0 (2 bytes), prop_data: (24 bytes)]
> Write Prop: file 1  PropIDX 2  (meta 8) seek: 34
> data=[uncompressed_len: 0 (2 bytes), prop_data: (8 bytes)]
> Write Prop: file 1  PropIDX 3  (meta 9) seek: 44
> data=[uncompressed_len: 0 (2 bytes), prop_data: (8 bytes)]
> Writing seek positions to index for file 1
>   PropIDX: 0  data=[seek: 8]  main index location: 802168 for 8 bytes
> (one print long)
>   PropIDX: 1  data=[seek: 0]  main index location: 802176 for 8 bytes
> (one print long)
>   PropIDX: 2  data=[seek: 34]  main index location: 802184 for 8 bytes
> (one print long)
>   PropIDX: 3  data=[seek: 44]  main index location: 802192 for 8 bytes
> (one print long)
> 
> Removing very common words...
> no words removed.
> Writing main index...
> Sorting words ...
> Sorting 1,483 words alphabetically
> Writing header ...
> Writing index entries ...
>   Writing word text: Complete
>   Writing word hash: Complete
>   Writing word data: Complete
> 1,483 unique words indexed.
> Sorting property: swishdocpath
> Fetching seek positions for file 1
>  property index table at 802168, this file at 802168
>    PropIDX: 0  data[Seek: 8] at seek 802168 read 8 bytes (one readlong)
>    PropIDX: 1  data[Seek: 0] at seek 802176 read 8 bytes (one readlong)
>    PropIDX: 2  data[Seek: 34] at seek 802184 read 8 bytes (one readlong)
>    PropIDX: 3  data[Seek: 44] at seek 802192 read 8 bytes (one readlong)
> Fetching filenum: 1 propIDX: 0 at seek: 8
>  Fetched uncompressed length of 0 (2 bytes storage), now fetching 24
> prop bytes from 10
> Fetching filenum: 1 propIDX: 2 at seek: 34
>  Fetched uncompressed length of 0 (2 bytes storage), now fetching 8
> prop bytes from 36
> Fetching filenum: 1 propIDX: 3 at seek: 44
>  Fetched uncompressed length of 0 (2 bytes storage), now fetching 8
> prop bytes from 46
> 4 properties sorted.
> 1 file indexed.  79,656 total bytes.  5,840 total words.
> Elapsed time: 00:00:00 CPU time: 00:00:00
> Indexing done!
> 

-- 
Bill Moseley
moseley@hank.org

Unsubscribe from or help with the swish-e list: 
   http://swish-e.org/Discussion/

Help with Swish-e:
   http://swish-e.org/current/docs
   swish-e@sunsite.berkeley.edu
Received on Fri Sep 10 13:15:18 2004