Re: Radix tree for character conversion

Tom Lane <tgl@sss.pgh.pa.us>

From: Tom Lane <tgl@sss.pgh.pa.us>
To: Robert Haas <robertmhaas@gmail.com>
Cc: Heikki Linnakangas <hlinnaka@iki.fi>, Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>, "Tsunakawa, Takayuki" <tsunakawa.takay@jp.fujitsu.com>, Tatsuo Ishii <ishii@sraoss.co.jp>, "pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Date: 2016-10-07T16:19:23Z
Lists: pgsql-hackers
Robert Haas <robertmhaas@gmail.com> writes:
> On Fri, Oct 7, 2016 at 6:46 AM, Heikki Linnakangas <hlinnaka@iki.fi> wrote:
>> Ouch. We should find and document an authoritative source for all the
>> mappings we have...
>> 
>> I think the next steps here are:
>> 
>> 1. Find an authoritative source for all the existing mappings.
>> 2. Generate the radix tree files directly from the authoritative sources,
>> instead of the existing *.map files.
>> 3. Completely replace the existing binary-search code with this.

> It might be best to convert using the existing map files, and then
> update the mappings later.  Otherwise, when things break, you won't
> know what to blame.

I think I went through this exercise last year or so, and updated the
notes about the authoritative sources where I was able to find one.
In the remaining cases, I believe that the maps have been intentionally
tweaked and we should be cautious about undoing that.  Tatsuo-san might
remember more about why they are the way they are.

			regards, tom lane


Commits

  1. Use radix tree for character encoding conversions.

  2. Small fixes to the Perl scripts to create unicode conversion tables.

  3. Rewrite the perl scripts to produce our Unicode conversion tables.

  4. Remove leading zeros, for consistency with other map files.

  5. Remove code points < 0x80 from character conversion tables.