The current spider.pl documents say:
"When a connection is not closed the spider does not
wait the "delay_sec" time when making the next
request. In other words, there is no delay in
requesting documents while the connection is open."
Seems to me that the delay_sec should be respected
even if the connection is keep-alive. Just because a
connection to a server is kept alive doesn't mean that
fetching pages doesn't cause a load on the server!
Perhaps a separate keep_alive_delay_sec could be used?
Or just have spider.pl keep using the delay_sec
Interested to hear what others think!
Do you Yahoo!?
All your favorites on one personal page – Try My Yahoo!
Received on Thu Jan 20 05:22:53 2005