Re: Radix tree for character conversion
Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
From: Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
To: hlinnaka@iki.fi
Cc: tgl@sss.pgh.pa.us, michael.paquier@gmail.com, daniel@yesql.se,
peter.eisentraut@2ndquadrant.com, robertmhaas@gmail.com,
tsunakawa.takay@jp.fujitsu.com, ishii@sraoss.co.jp,
pgsql-hackers@postgresql.org
Date: 2017-03-21T04:10:48Z
Lists: pgsql-hackers
Hello,
At Fri, 17 Mar 2017 13:03:35 +0200, Heikki Linnakangas <hlinnaka@iki.fi> wrote in <01efd334-b839-0450-1b63-f2dea9326a7e@iki.fi>
> On 03/17/2017 07:19 AM, Kyotaro HORIGUCHI wrote:
> > I would like to use convert() function. It can be a large
> > PL/PgSQL function or a series of "SELECT convert(...)"s. The
> > latter is doable on-the-fly (by not generating/storing the whole
> > script).
> >
> > | -- Test for SJIS->UTF-8 conversion
> > | ...
> > | SELECT convert('\0000', 'SJIS', 'UTF-8'); -- results in error
> > | ...
> > | SELECT convert('\897e', 'SJIS', 'UTF-8');
>
> Makes sense.
>
> >> You could then run those SQL statements against old and new server
> >> version, and verify that you get the same results.
> >
> > Including the result files in the repository will make this easy
> > but unacceptably bloats. Put mb/Unicode/README.sanity_check?
>
> Yeah, a README with instructions on how to do sounds good. No need to
> include the results in the repository, you can run the script against
> an older version when you need something to compare with.
Ok, I'll write a small script to generate a set of "conversion
dump" and try to write README.sanity_check describing how to use
it.
regards,
--
Kyotaro Horiguchi
NTT Open Source Software Center
Commits
-
Use radix tree for character encoding conversions.
- aeed17d00037 10.0 landed
-
Small fixes to the Perl scripts to create unicode conversion tables.
- bc1686f3f653 10.0 landed
-
Rewrite the perl scripts to produce our Unicode conversion tables.
- 1de9cc0dcca6 10.0 landed
-
Remove leading zeros, for consistency with other map files.
- 6c303223be34 10.0 landed
-
Remove code points < 0x80 from character conversion tables.
- 2c09c93ce1b8 10.0 landed