Re: Radix tree for character conversion

Heikki Linnakangas <hlinnaka@iki.fi>

From: Heikki Linnakangas <hlinnaka@iki.fi>
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>, michael.paquier@gmail.com, daniel@yesql.se, peter.eisentraut@2ndquadrant.com, robertmhaas@gmail.com, tsunakawa.takay@jp.fujitsu.com, ishii@sraoss.co.jp, pgsql-hackers@postgresql.org
Date: 2017-03-13T19:07:39Z
Lists: pgsql-hackers
On 03/13/2017 08:53 PM, Tom Lane wrote:
> Heikki Linnakangas <hlinnaka@iki.fi> writes:
>> It would be nice to run the map_checker tool one more time, though, to
>> verify that the mappings match those from PostgreSQL 9.6.
>
> +1
>
>> Just to be sure, and after that the map checker can go to the dustbin.
>
> Hm, maybe we should keep it around for the next time somebody has a bright
> idea in this area?

The map checker compares old-style maps with the new radix maps. The 
next time 'round, we'll need something that compares the radix maps with 
the next great thing. Not sure how easy it would be to adapt.

Hmm. A somewhat different approach might be more suitable for testing 
across versions, though. We could modify the perl scripts slightly to 
print out SQL statements that exercise every mapping. For every 
supported conversion, the SQL script could:

1. create a database in the source encoding.
2. set client_encoding='<target encoding>'
3. SELECT a string that contains every character in the source encoding.

You could then run those SQL statements against old and new server 
version, and verify that you get the same results.

- Heikki



Commits

  1. Use radix tree for character encoding conversions.

  2. Small fixes to the Perl scripts to create unicode conversion tables.

  3. Rewrite the perl scripts to produce our Unicode conversion tables.

  4. Remove leading zeros, for consistency with other map files.

  5. Remove code points < 0x80 from character conversion tables.