Re: Supporting SJIS as a database encoding
Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>
From: "Tsunakawa, Takayuki" <tsunakawa.takay@jp.fujitsu.com>
To: 'Tatsuo Ishii' <ishii@sraoss.co.jp>
Cc: "pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Date: 2016-09-05T08:00:06Z
Lists: pgsql-hackers
> From: pgsql-hackers-owner@postgresql.org
> [mailto:pgsql-hackers-owner@postgresql.org] On Behalf Of Tatsuo Ishii
> > But what I'm wondering is why PostgreSQL doesn't support SJIS. Was there
> any technical difficulty? Is there anything you are worried about if adding
> SJIS?
>
> Yes, there's a technical difficulty with backend code. In many places it
> is assumed that any string is "ASCII compatible", which means no ASCII
> character is used as a part of multi byte string. Here is such a random
> example from src/backend/util/adt/varlena.c:
>
> /* Else, it's the traditional escaped style */
> for (bc = 0, tp = inputText; *tp != '\0'; bc++)
> {
> if (tp[0] != '\\')
> tp++;
>
> Sometimes SJIS uses '\' as the second byte of it.
Thanks, I'll try to understand the seriousness of the problem as I don't have good knowledge of character sets. But your example seems to be telling everything about the difficulty...
Before digging into the problem, could you share your impression on whether PostgreSQL can support SJIS? Would it be hopeless? Can't we find any direction to go? Can I find relevant source code by searching specific words like "ASCII", "HIGH_BIT", "\\" etc?
Regards
Takayuki Tsunakawa
Commits
-
Use radix tree for character encoding conversions.
- aeed17d00037 10.0 landed
-
Small fixes to the Perl scripts to create unicode conversion tables.
- bc1686f3f653 10.0 landed
-
Rewrite the perl scripts to produce our Unicode conversion tables.
- 1de9cc0dcca6 10.0 landed
-
Remove leading zeros, for consistency with other map files.
- 6c303223be34 10.0 landed
-
Remove code points < 0x80 from character conversion tables.
- 2c09c93ce1b8 10.0 landed