>> Lets say I add a new page. Do I have to spider the whole
>> site again to index the 1 page?
> Mostly, yes.
There are other solutions to this. For instance, if you know in advance
what part of your site is going to have changes, and which site doesn't
change as much, you can make multiple indexes. If your content is
somehow date-based, like this:
[ ... ]
then you could come up with a scheme that only spiders the most recent
directory on a regular basis, and builds an index for the other, more
static parts, when you get a new "hot" directory.
That's just an example, but knowing how your content changes can direct
you toward more efficient spidering.
Received on Wed Nov 9 08:07:27 2005