Re: Radix tree for character conversion

Heikki Linnakangas <hlinnaka@iki.fi>

From: Heikki Linnakangas <hlinnaka@iki.fi>
To: Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>, michael.paquier@gmail.com
Cc: daniel@yesql.se, peter.eisentraut@2ndquadrant.com, robertmhaas@gmail.com, tsunakawa.takay@jp.fujitsu.com, tgl@sss.pgh.pa.us, ishii@sraoss.co.jp, pgsql-hackers@postgresql.org
Date: 2017-03-13T18:48:07Z
Lists: pgsql-hackers
On 03/06/2017 10:16 AM, Kyotaro HORIGUCHI wrote:
> At Fri, 3 Mar 2017 12:53:04 +0900, Michael Paquier <michael.paquier@gmail.com> wrote in <CAB7nPqSQaLozFNg+5Tf9s1TZs2pcE-GHhnMG31qnsusV9vMUOw@mail.gmail.com>
>> On Thu, Mar 2, 2017 at 2:20 PM, Kyotaro HORIGUCHI
>> <horiguchi.kyotaro@lab.ntt.co.jp> wrote:
>>> 5) Just remove plain map files and all related code. Addition to
>>>    that, Makefile stores hash digest of authority files in
>>>    Unicode/authoriy_hashes.txt or something that is managed by
>>>    git.
>>
>> That may be an idea to check for differences across upstream versions.
>> But that sounds like a separate discussion to me.
>
> Fine with me either.

I did some more kibitzing here and there, and committed. Thanks everyone!

I agree the new maps should just replace the old maps altogether, so 
committed that way. I also moved the combined map files to the same .map 
files as the main radix trees. Seems more clear that way to me.

I changed the to/from_unicode properties back to a single direction 
property, with Perl constants BOTH, TO_UNICODE and FROM_UNICODE, per 
your alternative suggestion upthread. Seems more clear to me.

It would be nice to run the map_checker tool one more time, though, to 
verify that the mappings match those from PostgreSQL 9.6. Just to be 
sure, and after that the map checker can go to the dustbin.

- Heikki



Commits

  1. Use radix tree for character encoding conversions.

  2. Small fixes to the Perl scripts to create unicode conversion tables.

  3. Rewrite the perl scripts to produce our Unicode conversion tables.

  4. Remove leading zeros, for consistency with other map files.

  5. Remove code points < 0x80 from character conversion tables.