Skip to main content.
home | support | download

Back to List Archive

Re: [SWISH-E:116] Re: Indexing off-site html

From: Paul J. Lucas <pjl(at)not-real.ptolemy.arc.nasa.gov>
Date: Sun Jan 11 1998 - 02:00:59 GMT
On Fri, 9 Jan 1998, Jim Winstead wrote:

> It indeed isn't difficult at all. One place to steal a handy fopen()
> wrapper that handles URLs from is PHP3 (http://www.php.net/) in the
> functions/fsock.c file.
> 
> Combined with some code that reads the files to index from another file,
> you don't even really need to building the spidering intelligence into
> swish.

	Except that's internet-hostile.  Please respect the robot
	exclusion standard.

	- Paul J. Lucas
	  NASA Ames Research Center		Caelum Research Corporation
	  Moffett Field, California		San Jose, California
	  <pjl AT ptolemy DOT arc DOT nasa DOT gov>
Received on Sat Jan 10 18:09:11 1998