Skip to main content.
home | support | download

Back to List Archive's delay_sec & keep_alive

From: J Robinson <jrobinson852(at)>
Date: Thu Jan 20 2005 - 13:22:50 GMT
The current documents say:

=item keep_alive


"When a connection is not closed the spider does not
wait the "delay_sec" time when making the next
request.  In other words, there is no delay in
requesting documents while the connection is open."

Seems to me that the delay_sec should be respected
even if the connection is keep-alive. Just because a
connection to a server is kept alive doesn't mean that
fetching pages doesn't cause a load on the server!

Perhaps a separate keep_alive_delay_sec could be used?
Or just have keep using the delay_sec

Interested to hear what others think!

Do you Yahoo!? 
All your favorites on one personal page  Try My Yahoo! 
Received on Thu Jan 20 05:22:53 2005