Skip to main content.
home | support | download

Back to List Archive

Re: Limiting load?

From: Jordan Hayes <jmhayes(at)not-real.speakeasy.net>
Date: Sat Sep 27 2003 - 20:23:12 GMT
> Is there a way to limit the load a swish-e process
> will drive the machine to while indexing, particularly
> while writing the word information?

I think swish-e (correct me if I'm wrong!) does it's indexing in memory,
so you can slow it down by renice'ing it (on Unix at least), but then
writes out the index all at once ... in Unix at least, if other
processes are waiting for that disk, they'll pile up and raise the load.
Slowing this down just makes it hurt more, because if you jump around on
the disk your buffer caches are going to go all to heck.  But I suppose
you could check for an option in db.c:write_word() and sleep a little
..

But I'm curious about this request: it sounds like you're seeing too
much of your machine getting used by indexing, so maybe you're indexing
too much too often?

I have a decent sized collection: about 160,000 documents that are an
archive of a mailing list; 87MB total properties, 223MB of index.  I've
decided to have three indexes: one for the current week, one for the
current year except the current week and one for everything except the
current year.  This way I can update the smallest index every few
minutes or so, the weekly one once per week and the major one only once
per year: maybe you need to segment your data out more and save some
time there?

/jordan
Received on Sat Sep 27 20:23:19 2003