Skip to main content.
home | support | download

Back to List Archive

RE: HTTP Crawler

From: David L Norris <dave(at)not-real.webaugur.com>
Date: Thu May 02 2002 - 05:35:47 GMT
On Wed, 2002-05-01 at 18:37, Hsiao Ketung Contr 61 CS/SCBN wrote:
> But if I run the following (from src directory)
> ./swishspider . http://my-intranet-server-name/tmp.html.

Is there a robot control file blocking the URL?
  http://my-intranet-server-name/robots.txt

-- 
 David Norris
  Dave's Web - http://www.webaugur.com/dave/
  Augury Net - http://augur.homeip.net/
  ICQ - 412039
Received on Thu May 2 05:35:57 2002