Horton

Re: Supporting SJIS as a database encoding

Heikki Linnakangas <hlinnaka@iki.fi>

View thread View on postgr.es
From: Heikki Linnakangas <hlinnaka@iki.fi>
To: "Tsunakawa, Takayuki" <tsunakawa.takay@jp.fujitsu.com>
Cc: Tom Lane <tgl@sss.pgh.pa.us>, 'Tatsuo Ishii' <ishii@sraoss.co.jp>, "pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Date: 2016-09-05T16:38:33Z
Lists: pgsql-hackers 
On 09/05/2016 05:47 PM, Tom Lane wrote:
> "Tsunakawa, Takayuki" <tsunakawa.takay@jp.fujitsu.com> writes:
>> Before digging into the problem, could you share your impression on
>> whether PostgreSQL can support SJIS?  Would it be hopeless?
>
> I think it's pretty much hopeless.

Agreed.

But one thing that would help a little, would be to optimize the UTF-8 
-> SJIS conversion. It uses a very generic routine, with a binary search 
over a large array of mappings. I bet you could do better than that, 
maybe using a hash table or a radix tree instead of the large 
binary-searched array.

- Heikki

Commits

  1. Use radix tree for character encoding conversions.

  2. Small fixes to the Perl scripts to create unicode conversion tables.

  3. Rewrite the perl scripts to produce our Unicode conversion tables.

  4. Remove leading zeros, for consistency with other map files.

  5. Remove code points < 0x80 from character conversion tables.