Horton

Re: Radix tree for character conversion

Robert Haas <robertmhaas@gmail.com>

View thread View on postgr.es
From: Robert Haas <robertmhaas@gmail.com>
To: Heikki Linnakangas <hlinnaka@iki.fi>
Cc: Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>, "Tsunakawa, Takayuki" <tsunakawa.takay@jp.fujitsu.com>, Tom Lane <tgl@sss.pgh.pa.us>, Tatsuo Ishii <ishii@sraoss.co.jp>, "pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Date: 2016-10-07T15:55:10Z
Lists: pgsql-hackers 
On Fri, Oct 7, 2016 at 6:46 AM, Heikki Linnakangas <hlinnaka@iki.fi> wrote:
> Ouch. We should find and document an authoritative source for all the
> mappings we have...
>
> I think the next steps here are:
>
> 1. Find an authoritative source for all the existing mappings.
> 2. Generate the radix tree files directly from the authoritative sources,
> instead of the existing *.map files.
> 3. Completely replace the existing binary-search code with this.

It might be best to convert using the existing map files, and then
update the mappings later.  Otherwise, when things break, you won't
know what to blame.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Commits

  1. Use radix tree for character encoding conversions.

  2. Small fixes to the Perl scripts to create unicode conversion tables.

  3. Rewrite the perl scripts to produce our Unicode conversion tables.

  4. Remove leading zeros, for consistency with other map files.

  5. Remove code points < 0x80 from character conversion tables.