Skip to main content.
home | support | download

Back to List Archive

request delay problem with spider.pl

From: Aliasgar Dahodwala <g_adahodwala(at)not-real.umassd.edu>
Date: Tue Jul 05 2005 - 16:14:47 GMT
Hi All:

I am running swish-e 2.4.3 on a redhat linux box. I am using the 
included spider.pl script to spider my website.

My problem: When i enable the keep_alive directive of the spider program 
and set the delay_sec to 5, the spider fetches the pages at blazing 
speed ignoring the delay_sec directive, and after going through around 
5000 pages it then catches up on all the delay, it stops fetching any 
more pages and just keeps sleeping for 5 seconds each. After a long wait 
it continues from where it left off.

I am not sure if anyone else is facing similar problems, and if anyone 
has a work around to using the keep_alive directive and also obey the 
delay_sec directive. Any help is appreciated.

Thanks in advance,
Regards,
Aliasgar.
Received on Tue Jul 5 09:14:54 2005