UTF8 encoding & Bash

[Lennart Sorensen]

On Wed, Jul 29, 2009 at 09:48:53PM -0400, Madison Kelly wrote:
> Hi all,
>
>   I've got a problem that I *thought* was perl, but turns out to be more 
> fundamental. I switched to bash and the problem remains...
>
>   There is a program called 'fantasdic' which takes a Japanese character 
> and returns information on it. When it outputs to STDOUT it's fine, but 
> when the output is redirected to a file, 0 bytes are written. This only 
> happens with certain characters, no less.
>
>   For example;
>
> -----------------------------------------------
> #!/bin/bash
>
> echo "### Start 右 ###"
> /usr/bin/fantasdic -o Japanese 右
>
> echo "### Start 林 ###"
> /usr/bin/fantasdic -o Japanese 林
> -----------------------------------------------
>
>   Run this (as a script or as individual commands) and the output is  
> clearly there and complete.
>
>   However, if you redirect the output to a file, the first character  
> will write out to the file where the second one will not. Actually, most  
> of the output from the first character will write... it seems to  
> double-encode at one point and bail, but I digress as I suspect solving  
> the second command's problem will solve the first. :)
>
>   Any idea what's going on here?

The program stupidly requries access to X to run.   If it has a command
line interface i addition to the X one, it should at least run without
access to X when used on the command line.

Seems someone wrote that without realizing what libs they were pulling in.

It also generates ^M at the end of lines, which is not unix compatible.
Except the last line which contains two pieces of non UTF8 garbage.
Looks like a bug for sure.

Of course it may also be missing the output and not using it in the
normal buffered way which might explain why the output is truncated.

Even running just the first fantasdic command gets truncated, never mind
the rest.  Seems like it is just in general a very buggy program.
It clearly doesn't use normal buffered output, nor does it seem to
understand proper line endings.

-- 
Len Sorensen
--
The Toronto Linux Users Group.      Meetings: http://gtalug.org/
TLUG requests: Linux topics, No HTML, wrap text below 80 columns
How to UNSUBSCRIBE: http://gtalug.org/wiki/Mailing_lists