Re: Radix tree for character conversion

Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>

From: Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
To: hlinnaka@iki.fi
Cc: daniel@yesql.se, peter.eisentraut@2ndquadrant.com, robertmhaas@gmail.com, tsunakawa.takay@jp.fujitsu.com, tgl@sss.pgh.pa.us, ishii@sraoss.co.jp, pgsql-hackers@postgresql.org
Date: 2017-01-10T11:22:23Z
Lists: pgsql-hackers

Attachments

Hello, I found a bug in my portion while rebasing.

The attached patches apply on top of the current master HEAD, not
on Heikki's previous one. And separated into 4 parts.

At Tue, 13 Dec 2016 15:11:03 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> wrote in <20161213.151103.157484378.horiguchi.kyotaro@lab.ntt.co.jp>
> > Apart from the aboves, I have some trivial comments on the new
> > version.
> > 
> > 
> > 1. If we decide not to use old-style maps, UtfToLocal no longer
> >   need to take void * as map data. (Patch 0001)

I changed the pointer type wrongly. Combined maps are of the type
*_combined.

> > 2. "use Data::Dumper" doesn't seem necessary. (Patch 0002)
> > 3. A comment contains a superfluous comma. (Patch 0002) (The last
> >    byte of the first line below)
> > 4. The following code doesn't seem so perl'ish.
> > 5. download_srctxts.sh is no longer needed. (No patch)
> 
> 6. Fixed some inconsistent indentation/folding.
> 7. Fix handling of $verbose.
> 8. Sort segments using leading bytes.

The attached files are the following. This patchset is not
complete missing changes of map files. The change is tremendously
large but generatable.

0001-Add-missing-semicolon.patch

  UCS_to_EUC_JP.pl has a line missing teminating semicolon. This
  doesn't harm but surely a syntax error. This patch fixes it.
  This might should be a separate patch.

0002-Correct-reference-resolution-syntax.patch

  convutils.pm has lines with different syntax of reference
  resolution. This unifies the syntax.

0003-Apply-pgperltidy-on-src-backend-utils-mb-Unicode.patch

  Before adding radix tree stuff, applied pgperltidy and inserted
  format-skipping pragma for the parts where perltidy seems to do
  too much.

0004-Use-radix-tree-for-character-conversion.patch

  Radix tree body.


The unattached fifth patch is generated by the following steps.

[$(TOP)]$ ./configure
[Unicode]$ make
[Unicode]$ make distclean
[Unicode]$ git add .
[Unicode]$ commit 
=== COMMITE MESSSAGE
Replace map files with radix tree files.

These encodings no longer uses the former map files and uses new radix
tree files.
===


regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

Commits

  1. Use radix tree for character encoding conversions.

  2. Small fixes to the Perl scripts to create unicode conversion tables.

  3. Rewrite the perl scripts to produce our Unicode conversion tables.

  4. Remove leading zeros, for consistency with other map files.

  5. Remove code points < 0x80 from character conversion tables.