Another Option I would like to see added, is the IndexOnly option for HTTP.
Like:

IndexOnly .html .htm

I fail to see why this option should be for the filesystem only. I really
think that for webspidering the same rules should apply, or at least this
very important one.

- Mark

        System Administrator Asarian-host.org