Skip to main content.
home | support | download

Back to List Archive

RE: indexing templates and data files for dynamic pages

From: Halasz, Steve <skh2(at)not-real.cornell.edu>
Date: Wed Mar 15 2000 - 22:49:28 GMT
Nathaniel,

We don't want protected pages to show up in search results. So I have
excluded the spider from looking at those pages using the robots.txt file in
the server document root.

Is the session ID set as a cookie? You might be able to hack the SWISH
spider to send a valid session cookie in its request headers.

-Steve

> -----Original Message-----
> From: Nathaniel Meyers [mailto:Nathaniel.Meyers@segasoft.com]
> Sent: Wednesday, March 15, 2000 5:07 PM
> To: 'Halasz, Steve'
> Subject: RE: [SWISH-E] indexing templates and data files for dynamic
> pages
> 
> 
> It might.  We'll give that a try.  I wonder though ... We 
> have many pages
> that are accessable only when the user has logged on and been given a
> session id (else the script defaults to a login screen).  I 
> wonder if we run
> swish without a session ID if it will attempt to index these 
> pages on the
> content of the log in screen.  Did you ever have a problem like this?
> 
> Thanks for the response.
> -Nathaniel
> 
> >-----Original Message-----
> >From: Halasz, Steve [mailto:skh2@cornell.edu]
> >Sent: Wednesday, March 15, 2000 1:57 PM
> >To: 'Nathaniel.Meyers@segasoft.com'
> >Subject: RE: [SWISH-E] indexing templates and data files for dynamic
> >pages
> >
> >
> >Will running SWISH in http mode solve your problem? I do this
> >for a site
> >that is entirely dynamic pages and it works fine. Although it
> >takes a little
> >longer and probably hits the server harder too.
> >
> >put -S http on the command line and make IndexDir in the
> >config file the URL
> >of where you want to start spidering.
> >
> >-Steve
> >
> >> -----Original Message-----
> >> From: Nathaniel Meyers [mailto:Nathaniel.Meyers@segasoft.com]
> >> Sent: Wednesday, March 15, 2000 4:37 PM
> >> To: Multiple recipients of list
> >> Subject: [SWISH-E] indexing templates and data files for
> >dynamic pages
> >>
> >>
> >> I'm wondering if anyone has had any experience indexing pages
> >> on a site that
> >> pretty much has no static pages (nearly all pages are
> >> dynamically generated
> >> from template files).  It looks like it shouldn't be too 
> much work to
> >> develope something that will do this but I wonder if anyone
> >> has already
> >> attempted it.
> >>
> >> -Nathaniel Meyers
> >> Heat.Net
> >>
> >
> 
Received on Wed Mar 15 17:49:55 2000