At 03:22 PM 08/13/02 -0700, John P. Rouillard wrote:
>
>It looks like there is a bug in spider.pl. An attempt to set a no_index
>attribute on the base_url using the test_url function fails.

Well, it's not really a bug as the docs say:

=item test_url

..

You cannot use the server flags:

    no_contents
    no_index
    no_spider

So, you need to set those in a test_response call-back.  test_url is a way
to avoid fetching the document completely.  You need test_response since
you don't want to index it, but you still want to follow links in that
document.  Thus, you still need to fetch that doc.

Does that help?


-- 
Bill Moseley
mailto:moseley@hank.org