Skip to main content.
home | support | download

Back to List Archive

Re: spider a database

From: Aaron Bazar <aaronb(at)not-real.spamcop.net>
Date: Fri Nov 04 2005 - 20:27:34 GMT
 
You will probably get better support from swish-e.org... but from the sounds
of it you should use use included spider program to spider your web site...
You would not mess at all with /var/lib/mysql or /home/yc/www... You would
put the web site URL in the spider's config file.

Spidering a database can work too, but is likely a bit more difficult to set
up. You would need to write a program that queries your database and pipes
the results to swish-e. Once you write the program, it is much faster than
using the spider, but, again, it is likely to be a pain.


Regards,

Aaron Bazar
http://www.acompshop.com/

-----Original Message-----
From: swish-e@sunsite3.berkeley.edu [mailto:swish-e@sunsite3.berkeley.edu]
On Behalf Of Michael Porcaro
Sent: Friday, November 04, 2005 3:18 PM
To: Multiple recipients of list
Subject: [SWISH-E] Re: spider a database

Please bear with me here and thank you for your patience.  I looked at your
link and searched around.  By searching, I assume that swish-e can spider
databases, I wasn't really sure about this before.  I came across this
document.  Is this the right thing to read, in order to figure out how to
spider my dynamic pages?

http://swish-e.org/docs/swish-config.html#directives_for_the_http_access
_method_only

Also, I am confused as to where I should direct the config file to spider
the dynamic links.  Let's say I want to spider this particular
file:

http://www.youngcomposers.com/forum/Piano-Music-f50.html

Piano-Music-f50.html is actually a php generated file with an html alias,
but I don't know where to direct swish-e to spider this file.
When I spider the files under /home/yc/www/forum (my local site for
www.youngcomposers.com), all it does is spider the files that run the forum,
not the actual content dynamic pages, such as "Piano-Music-f50.html" or
equivalently http://www.youngcomposers.com/forum/index.php?showforum=50

So I guess my basic question would be, what is the address of my dynamic
files?  A very poor guess is, my database files are located here:

/var/lib/mysql/

But is this the address to spider?  Or do I spider /home/yc/www/forum
instead?  

-----Original Message-----
From: swish-e@sunsite3.berkeley.edu
[mailto:swish-e@sunsite3.berkeley.edu] On Behalf Of Bill Moseley
Sent: Friday, November 04, 2005 12:56 PM
To: Multiple recipients of list
Subject: [SWISH-E] Re: spider a database

On Fri, Nov 04, 2005 at 09:50:35AM -0800, Michael Porcaro wrote:
> Hi guys,
> 
> I know how to spider physical files on a server, but I am having
trouble
> spidering or indexing my dynamic files.  I have a big forum, and it is 
> all run by a mysql database (all dynamic files).  Any insight on how
to
> spider my dynamic files run by mysql database?  Thank you.

It was once said: if you can use swish you can then learn how to use swish.

http://swish-e.org/search/index.html?query=mysql&submit=search


--
Bill Moseley
moseley@hank.org

Unsubscribe from or help with the swish-e list: 
   http://swish-e.org/Discussion/

Help with Swish-e:
   http://swish-e.org/current/docs
   swish-e@sunsite.berkeley.edu
Received on Fri Nov 4 12:27:35 2005