Skip to main content.
home | support | download

Back to List Archive

Re: [swish-e] spidering and incremental index

From: Peter Karman <peter(at)not-real.peknet.com>
Date: Thu May 01 2008 - 03:07:24 GMT
david webster wrote on 4/30/08 8:14 PM:
> This is for "latest" as of about 2 weeks ago.  Would the 2.6 or ? be better
> for what I'm trying to do >> spidering xxx sites on each of 6 applications
> .. all on 1 server?
> 

You are using the 2.4.x (2.5.6) snapshot then.

Which one is "better" (2.4 or 2.6) for your particular needs is up to you. 
Here's a brief summary:

The btree/incremental features in 2.4.x are still labeled experimental. From a 
development perspective, when 2.4.6 finally comes out it is expected to be the 
last 2.4.x release. So no new dev is expected on the 2.4 incremental feature. I 
know some folks use it; I do not know how successfully (stories welcome).

The 2.6 branch is a complete rewrite of the index storage to use Berkeley DB. It 
is the only index format supported, so 2.6 indexes are completely incompatible 
with 2.4.x indexes. There are a few open bugs on the 2.6 branch[1] but initial 
tests show it is as fast as 2.4 and offers a more transparent index format.

I fully expect that there will be a 3.0 backend option that uses the 2.6 format, 
so that you can upgrade from 2.6 to 3.0 without needing to reindex. (A bold 
claim I will no doubt regret making some day.)

And finally, I can give you no timelines for any 2.6 or 3.0 development except 
to say that the more people who use 2.6, the more bugs will be revealed and thus 
opportunities for fixing them.

I hope that gives you some sense of the development roadmap (at least as it 
exists in my brain).


[1] http://dev.swish-e.org/report/1

-- 
Peter Karman  .  http://peknet.com/  .  peter(at)not-real.peknet.com
_______________________________________________
Users mailing list
Users@lists.swish-e.org
http://lists.swish-e.org/listinfo/users
Received on Wed Apr 30 23:06:54 2008