Re: Supporting SJIS as a database encoding
Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
From: Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
To: tsunakawa.takay@jp.fujitsu.com
Cc: hlinnaka@iki.fi, tgl@sss.pgh.pa.us, ishii@sraoss.co.jp,
pgsql-hackers@postgresql.org
Date: 2016-09-07T07:13:04Z
Lists: pgsql-hackers
Hello,
At Tue, 6 Sep 2016 03:43:46 +0000, "Tsunakawa, Takayuki" <tsunakawa.takay@jp.fujitsu.com> wrote in <0A3221C70F24FB45833433255569204D1F5E66CE@G01JPEXMBYT05>
> > From: pgsql-hackers-owner@postgresql.org
> > [mailto:pgsql-hackers-owner@postgresql.org] On Behalf Of Kyotaro
> > HORIGUCHI
> Implementing radix tree code, then redefining the format of mapping table
> > to suppot radix tree, then modifying mapping generator script are needed.
> >
> > If no one oppse to this, I'll do that.
>
> +100
> Great analysis and your guts. I very much appreciate your trial!
Thanks, by the way, there's another issue related to SJIS
conversion. MS932 has several characters that have multiple code
points. By converting texts in this encoding to and from Unicode
causes a round-trop problem. For example,
8754(ROMAN NUMERICAL I in NEC specials)
=> U+2160(ROMAN NUMERICAL I)
=> FA4A (ROMAN NUMERICA I in IBM extension)
My counting said that 398 characters are affected by this kind of
replacement. Addition to that, "GAIJI" (Private usage area) is
not allowed. Is this meet your purpose?
regards,
--
Kyotaro Horiguchi
NTT Open Source Software Center
Commits
-
Use radix tree for character encoding conversions.
- aeed17d00037 10.0 landed
-
Small fixes to the Perl scripts to create unicode conversion tables.
- bc1686f3f653 10.0 landed
-
Rewrite the perl scripts to produce our Unicode conversion tables.
- 1de9cc0dcca6 10.0 landed
-
Remove leading zeros, for consistency with other map files.
- 6c303223be34 10.0 landed
-
Remove code points < 0x80 from character conversion tables.
- 2c09c93ce1b8 10.0 landed