Is it possible to do some kind of incremental update (via spider or file system)? If not, how often would you reccommend indexing a large (1,000,000+ hits a month) site?
Also, this is a problem for me doing a large number of sites at once. We're trying to develop a search engine for our entire program which has around 100 domains. I started it last night and when I got in in the morning, it had frozen, swish-e was using 630MB of RAM, and the computer was barely usable. I'm using a 500Mhz P3, so it's not unexpected for something like this so happen, but if I could do 5-10 a night, and then incrementally add more to the index, it'd be great.
Received on Tue Jun 8 19:47:32 2004