I am curious on how the max_depth setting works in spider.pl and sub
domains. For example if i index the url www.somesite.com/sub/ and set the
max_depth to 2 will the spider stay within the sub folder for links or will
it look inside somesite.com?
I've done a few tests and it seems to go back up into root folders at
certain times, i assume when it needs more links? Can anyone explain how it
traverses the pages and if it is possible to limit the spider to only take
links from the sub domain?
MSN Hotmail is evolving – check out the new Windows Live Mail
Received on Fri Jan 12 06:34:56 2007