On 1/10/07, Peter Karman <peter@peknet.com> wrote:
>
>
> Bill Moseley scribbled on 1/10/07 8:09 AM:
> > On Wed, Jan 10, 2007 at 05:15:27AM -0800, James wrote:
> >> Good morning again!
> >>
> >> I have another question.  I am pretty sure someone on this discussion list
> >> has some knowledge about User-Agents (bots in particular) that seem to use
> >> Mozilla/2.0 or Mozilla/5.0.  For instance, Ask seems to use Mozilla/2.0 and
> >> Google seems to use Mozilla/5.0.  Do you know what this means?  Are they
> >> somehow running their spider through Mozilla?  Do they have Mozilla
> >> installed on their server to do this somehow?  Are there advantages to
> >> this?  I wondered if the Swish-e spider could be enhanced by doing this.
> >> For instance, maybe this would solve the UTF-8 issue?  Maybe it would solve
> >> other issues too.  I am speaking out of "ignorance" because this aspect of
> >> Google and Ask are not known to me.  Perhaps someone could help out in this
> >> area.  Bill?
> >
> > Ask and Google actually use swish-e for all their spidering and
> > indexing and don't want anyone to know, so they change the user agent
> > string.
> >
>
> actually, Bill is way off here. Ask and Google have openly admitted they use
> swish-e.
>
> ;)
>
> seriously though, James, you might want to read up on UTF-8 and user agents
> before posting here. And I found this helpful advice:
>
> http://www.mikeash.com/getting_answers.html

Thanks, I'll check out the link.  By the way, I did take the time to
read up on UTF-8 and user agents.  But since there is a plethora of
information and since you guys are the experts, I am asking you
because I figure that you will be able to speed up my learning and/or
point me to some information that you are already aware of.  That's
why novices seek out help from experts in forums and discussion
groups, right?  Believe me, I have spent hours and hours already,
before even posting, trying to find useful information, even on
Mozilla's own site.

>
> and in regard to the user agent question: I believe that one reason bots
> identify themselves as particular user agents is because they want to receive
> the same responses that the server would hand out to those non-bot agents.

So, is this a real benefit to the Swish-e Spider and how would it be
accomplished?
>
> --
> Peter Karman

Thanks for your time, Peter,

James