Skip to main content.
home | support | download

Back to List Archive

spider.pl's delay_sec & keep_alive

From: J Robinson <jrobinson852(at)not-real.yahoo.com>
Date: Thu Jan 20 2005 - 13:22:50 GMT
The current spider.pl documents say:

=item keep_alive

..

"When a connection is not closed the spider does not
wait the "delay_sec" time when making the next
request.  In other words, there is no delay in
requesting documents while the connection is open."

Seems to me that the delay_sec should be respected
even if the connection is keep-alive. Just because a
connection to a server is kept alive doesn't mean that
fetching pages doesn't cause a load on the server!

Perhaps a separate keep_alive_delay_sec could be used?
Or just have spider.pl keep using the delay_sec
anyway?

Interested to hear what others think!
jrobinson


		
__________________________________ 
Do you Yahoo!? 
All your favorites on one personal page  Try My Yahoo!
http://my.yahoo.com 
Received on Thu Jan 20 05:22:53 2005