Skip to main content.
home | support | download

Back to List Archive

Re: Bug in spider.pl setting no_index from test_url

From: Bill Moseley <moseley(at)not-real.hank.org>
Date: Tue Aug 13 2002 - 22:34:49 GMT
At 03:22 PM 08/13/02 -0700, John P. Rouillard wrote:
>
>It looks like there is a bug in spider.pl. An attempt to set a no_index
>attribute on the base_url using the test_url function fails.

Well, it's not really a bug as the docs say:

=item test_url

..

You cannot use the server flags:

    no_contents
    no_index
    no_spider

So, you need to set those in a test_response call-back.  test_url is a way
to avoid fetching the document completely.  You need test_response since
you don't want to index it, but you still want to follow links in that
document.  Thus, you still need to fetch that doc.

Does that help?


-- 
Bill Moseley
mailto:moseley@hank.org
Received on Tue Aug 13 22:38:21 2002