Re: Supporting SJIS as a database encoding

Tom Lane <tgl@sss.pgh.pa.us>

From: Tom Lane <tgl@sss.pgh.pa.us>
To: "Tsunakawa, Takayuki" <tsunakawa.takay@jp.fujitsu.com>
Cc: "'Tatsuo Ishii'" <ishii@sraoss.co.jp>, "pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Date: 2016-09-05T14:47:15Z
Lists: pgsql-hackers
"Tsunakawa, Takayuki" <tsunakawa.takay@jp.fujitsu.com> writes:
> Before digging into the problem, could you share your impression on
> whether PostgreSQL can support SJIS?  Would it be hopeless?

I think it's pretty much hopeless.  Even if we were willing to make every
bit of code that looks for '\' and other specific at-risk characters
multi-byte aware (with attendant speed penalties), we could expect that
third-party extensions would still contain vulnerable code.  More, we
could expect that new bugs of the same ilk would get introduced all the
time.  Many such bugs would amount to security problems.  So the amount of
effort and vigilance required seems out of proportion to the benefits.

Most of the recent discussion about allowed backend encodings has run
more in the other direction, ie, "why don't we disallow everything but
UTF8 and get rid of all the infrastructure for multiple backend
encodings?".  I'm not personally in favor of that, but there are very
few hackers who want to add any more overhead in this area.

			regards, tom lane


Commits

  1. Use radix tree for character encoding conversions.

  2. Small fixes to the Perl scripts to create unicode conversion tables.

  3. Rewrite the perl scripts to produce our Unicode conversion tables.

  4. Remove leading zeros, for consistency with other map files.

  5. Remove code points < 0x80 from character conversion tables.