Skip to main content.
home | support | download

Back to List Archive

Behavior of max_depth in spider.pl

From: andy rosbrook <andy_rosbrook(at)not-real.hotmail.com>
Date: Fri Jan 12 2007 - 14:34:41 GMT
Hello all,

I am curious on how the max_depth setting works in spider.pl and sub 
domains. For example if i index the url www.somesite.com/sub/ and set the 
max_depth to 2 will the spider stay within the sub folder for links or will 
it look inside somesite.com?

I've done a few tests and it seems to go back up into root folders at 
certain times, i assume when it needs more links? Can anyone explain how it 
traverses the pages and if it is possible to limit the spider to only take 
links from the sub domain?

thanks
andy

_________________________________________________________________
MSN Hotmail is evolving  check out the new Windows Live Mail 
http://ideas.live.com
Received on Fri Jan 12 06:34:56 2007