Bill Moseley wrote:

> The problem is if the content-length is reported in characters
> not bytes, then swish would end up reading in the wrong number of
> bytes.
> 
> One would expect if the number was reported in characters that if
> anything the length would be too low.  But you are seeing:
> 
>     Warning: Unknown header line: 'th-Name:
> 
> which looks like the content-length header was reporting too many
> bytes.  That's where I'm a bit confused.

It seems I copy-pasted the wrong error message. That was from an experiment I 
was doing where I modified the content-length header to see what was up. The 
real error was something like this (from memory now)

   Warning: Unknown header line: 'lumi/'

Which as part of the URL for the Path-Name. But it seems like it's still the 
same problem. It was reporting a content-length that was too long.

> The spider gets the content from LWP with:
> 
>         my $content = $response->decoded_content;

Would there be a problem if the site didn't correctly send a Charset header? 
Some of the pages I was spidering did not do that.

> So $content is characters at that point.  That's what you want -- you
> want characters in your Perl program and octets on the outside.

Is it possible that something in my environment was messing it up? I was calling 
swish-e via system() from a UTF8-ified system.

But regardless, I need to be able to show UTF-8 characters safely in the 
descriptions of my search results, so escaping those into HTML entities is 
needed for that. And after changing the content I need to recalculate the 
content-length header anyway, right?

-- 
Michael Peters
Plus Three, LP

_______________________________________________
Users mailing list
Users@lists.swish-e.org
http://lists.swish-e.org/listinfo/users