At 03:22 PM 08/13/02 -0700, John P. Rouillard wrote:
>It looks like there is a bug in spider.pl. An attempt to set a no_index
>attribute on the base_url using the test_url function fails.
Well, it's not really a bug as the docs say:
You cannot use the server flags:
So, you need to set those in a test_response call-back. test_url is a way
to avoid fetching the document completely. You need test_response since
you don't want to index it, but you still want to follow links in that
document. Thus, you still need to fetch that doc.
Does that help?
Received on Tue Aug 13 22:38:21 2002