Skip to main content.
home | support | download

Back to List Archive

How well does swish-e scale?

From: Omar Siddique <omar(at)not-real.umbc.edu>
Date: Mon May 04 1998 - 16:08:10 GMT
Hi, I recently was trying to set up swish-e 1.1 to index the user web
pages of this educational site.  I generated a swish conf listing all
the user web directories without any problems, and ran through
small-scale tests.  However, when I try to index the whole set of user
pages (roughly 2500 directories with varying numbers of indexable files
in them), it tends to just "stop" after a while.  It'll be going
along speedily, and then will start slowing down more and more.  I left
it running for over 3 days
at one point, and it had only made it through about 1200 of the user
directories.

The machine it's running on (the web server) is reasonably fast (Indy
R4400SC/100) with 96MB of
RAM and 128MB of swap.   The machine isn't running out of swap (although
it is going into swap by as much as 50-60 megs when it's at 1200
users) doing this.

So, is what I'm trying to do not do-able with swish-e?  Does something
in swish-e's design (maybe it needs to rewrite a big chunk of the index
file in memory every time it adds something) make it not
scalable to the level of what I'm doing?   And if it does manage to
index it, would it be too slow in
searching the index (the size of all the stuff being indexed is probably
100 or so megs)?

Any ideas?  I was hoping swish-e would do the trick, since excite for
web servers had failed abysmally (buggy and always died after a certain
number of entries).

Thanks for any advice!

-omar
Received on Mon May 4 09:17:07 1998