On 1/10/07, Peter Karman <email@example.com> wrote:
> Bill Moseley scribbled on 1/10/07 8:09 AM:
> > On Wed, Jan 10, 2007 at 05:15:27AM -0800, James wrote:
> >> Good morning again!
> >> I have another question. I am pretty sure someone on this discussion list
> >> has some knowledge about User-Agents (bots in particular) that seem to use
> >> Mozilla/2.0 or Mozilla/5.0. For instance, Ask seems to use Mozilla/2.0 and
> >> Google seems to use Mozilla/5.0. Do you know what this means? Are they
> >> somehow running their spider through Mozilla? Do they have Mozilla
> >> installed on their server to do this somehow? Are there advantages to
> >> this? I wondered if the Swish-e spider could be enhanced by doing this.
> >> For instance, maybe this would solve the UTF-8 issue? Maybe it would solve
> >> other issues too. I am speaking out of "ignorance" because this aspect of
> >> Google and Ask are not known to me. Perhaps someone could help out in this
> >> area. Bill?
> > Ask and Google actually use swish-e for all their spidering and
> > indexing and don't want anyone to know, so they change the user agent
> > string.
> actually, Bill is way off here. Ask and Google have openly admitted they use
> seriously though, James, you might want to read up on UTF-8 and user agents
> before posting here. And I found this helpful advice:
Thanks, I'll check out the link. By the way, I did take the time to
read up on UTF-8 and user agents. But since there is a plethora of
information and since you guys are the experts, I am asking you
because I figure that you will be able to speed up my learning and/or
point me to some information that you are already aware of. That's
why novices seek out help from experts in forums and discussion
groups, right? Believe me, I have spent hours and hours already,
before even posting, trying to find useful information, even on
Mozilla's own site.
> and in regard to the user agent question: I believe that one reason bots
> identify themselves as particular user agents is because they want to receive
> the same responses that the server would hand out to those non-bot agents.
So, is this a real benefit to the Swish-e Spider and how would it be
> Peter Karman
Thanks for your time, Peter,
Received on Wed Jan 10 06:48:29 2007