Skip to main content.
home | support | download

Back to List Archive

RE: Getting the right files indexed the right way

From: Rob de Santos AFANA <rdesantos(at)>
Date: Thu Jan 29 2004 - 03:27:53 GMT
> That's for the -S http method of indexing so it doesn't apply 
> to -S prog with

OK, that wasn't clear in the config section of the documentation on
Should I specify additional URL's in the spider config file then?
> Are there any links to follow in /album/?

Ah, that could be the hangup... there are about 14 subdirectories with
jpegs in them... but I don't think there is any explicit link elsewhere
to it.  The links are dynamically generated by the application.  I'm
reluctant to open the application directory up in robots.txt not because
of swish-e but because of certain poorly behaved spiders out there. 

I could place a dummy index.html file in there with nothing but links to
the subdirectories, that might work. 


-Rob de Santos
-Columbus, Ohio USA
Chairman of the Board,
Australian Football Association of North America (AFANA)
ph: 1-888-4AFANA1 (North America) (1-888-423-2621)
ph: 1-614-338-0002 (outside NA)  
e-mail: rdesantos(at)   web: <>
Received on Wed Jan 28 19:27:54 2004