Re: Supporting SJIS as a database encoding

Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>

From: "Tsunakawa, Takayuki" <tsunakawa.takay@jp.fujitsu.com>
To: 'Tatsuo Ishii' <ishii@sraoss.co.jp>
Cc: "pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Date: 2016-09-05T08:00:06Z
Lists: pgsql-hackers
> From: pgsql-hackers-owner@postgresql.org
> [mailto:pgsql-hackers-owner@postgresql.org] On Behalf Of Tatsuo Ishii
> > But what I'm wondering is why PostgreSQL doesn't support SJIS.  Was there
> any technical difficulty?  Is there anything you are worried about if adding
> SJIS?
> 
> Yes, there's a technical difficulty with backend code. In many places it
> is assumed that any string is "ASCII compatible", which means no ASCII
> character is used as a part of multi byte string. Here is such a random
> example from src/backend/util/adt/varlena.c:
> 
> 	/* Else, it's the traditional escaped style */
> 	for (bc = 0, tp = inputText; *tp != '\0'; bc++)
> 	{
> 		if (tp[0] != '\\')
> 			tp++;
> 
> Sometimes SJIS uses '\' as the second byte of it.

Thanks, I'll try to understand the seriousness of the problem as I don't have good knowledge of character sets.  But your example seems to be telling everything about the difficulty...

Before digging into the problem, could you share your impression on whether PostgreSQL can support SJIS?  Would it be hopeless?  Can't we find any direction to go?  Can I find relevant source code by searching specific words like "ASCII", "HIGH_BIT", "\\" etc?

Regards
Takayuki Tsunakawa




Commits

  1. Use radix tree for character encoding conversions.

  2. Small fixes to the Perl scripts to create unicode conversion tables.

  3. Rewrite the perl scripts to produce our Unicode conversion tables.

  4. Remove leading zeros, for consistency with other map files.

  5. Remove code points < 0x80 from character conversion tables.