On Wed, Jan 10, 2007 at 07:14:01AM -0800, James wrote:
> So, you think that it is a case of them just putting that information
> into their user / server agent string to trick the server into
> thinking they are viewing through a browser? That's an interesting
> thought. So, maybe they aren't running any spiders through browsers.
> Or is it still possible that they do run the spider through a browser?
The user agent string is just sent with a request. The server can
decided what to do with it -- the vast majority ignore it. That's
what I'd recommend doing, too.
Do you have an actual problem with this? I mean, do you know for a
*fact* (not because you guess so) that some server is not sending you
a document based on the user agent string? Have you discussed this
with that site's admin?
Even if you do have a real problem, then just set the agent string. I
think we covered how to do that now.
Unsubscribe from or help with the swish-e list:
Help with Swish-e:
Received on Wed Jan 10 07:31:47 2007