Horton

Thread

Commits

  1. Use radix tree for character encoding conversions.

  2. Small fixes to the Perl scripts to create unicode conversion tables.

  3. Rewrite the perl scripts to produce our Unicode conversion tables.

  4. Remove leading zeros, for consistency with other map files.

  5. Remove code points < 0x80 from character conversion tables.

  1. Supporting SJIS as a database encoding

    Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com> — 2016-09-05T07:23:21Z

    Hello,

I'd like to propose adding SJIS as a database encoding.  You may wonder why SJIS is still necessary in the world of Unicode.  The purpose is to achieve comparable performance when migrating legacy database systems from other DBMSs without little modification of applications.

Recently, we failed to migrate some customer's legacy database from DBMS-X to PostgreSQL.  That customer wished for PostgreSQL, but PostgreSQL couldn't meet the performance requirement.

The system uses DBMS-X with the database character set being SJIS.  The main applications are written in embedded SQL, which require SJIS in their host variables.  They insisted they cannot use UTF8 for the host variables because that would require large modification of applications due to character handling.  So no character set conversion is necessary between the clients and the server.

On the other hand, PostgreSQL doesn't support SJIS as a database encoding.  Therefore, character set conversion from UTF-8 to SJIS has to be performed.  The batch application runs millions of SELECTS each of which retrieves more than 100 columns.  And many of those columns are of character type.

If PostgreSQL supports SJIS, PostgreSQL will match or outperform the performance of DBMS-X with regard to the applications.  We confirmed it by using psql to run a subset of the batch processing.  When the client encoding is SJIS, one FETCH of 10,000 rows took about 500ms.  When the client encoding is UTF8 (the same as the database encoding), the same FETCH took 270ms.

Supporting SJIS may somewhat regain attention to PostgreSQL here in Japan, in the context of database migration.  BTW, MySQL supports SJIS as a database encoding.  PostgreSQL used to be the most popular open source database in Japan, but MySQL is now more popular.


But what I'm wondering is why PostgreSQL doesn't support SJIS.  Was there any technical difficulty?  Is there anything you are worried about if adding SJIS?

I'd like to write a patch for adding SJIS if there's no strong objection.  I'd appreciate it if you could let me know good design information to add a server encoding (e.g. the URL of the most recent patch to add a new server encoding)

Regards
Takayuki Tsunakawa

  2. Re: Supporting SJIS as a database encoding

    Tatsuo Ishii <ishii@sraoss.co.jp> — 2016-09-05T07:38:54Z

    > But what I'm wondering is why PostgreSQL doesn't support SJIS.  Was there any technical difficulty?  Is there anything you are worried about if adding SJIS?

Yes, there's a technical difficulty with backend code. In many places
it is assumed that any string is "ASCII compatible", which means no
ASCII character is used as a part of multi byte string. Here is such a
random example from src/backend/util/adt/varlena.c:

	/* Else, it's the traditional escaped style */
	for (bc = 0, tp = inputText; *tp != '\0'; bc++)
	{
		if (tp[0] != '\\')
			tp++;

Sometimes SJIS uses '\' as the second byte of it.

Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp

  3. Re: Supporting SJIS as a database encoding

    Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com> — 2016-09-05T08:00:06Z

    > From: pgsql-hackers-owner@postgresql.org
> [mailto:pgsql-hackers-owner@postgresql.org] On Behalf Of Tatsuo Ishii
> > But what I'm wondering is why PostgreSQL doesn't support SJIS.  Was there
> any technical difficulty?  Is there anything you are worried about if adding
> SJIS?
> 
> Yes, there's a technical difficulty with backend code. In many places it
> is assumed that any string is "ASCII compatible", which means no ASCII
> character is used as a part of multi byte string. Here is such a random
> example from src/backend/util/adt/varlena.c:
> 
> 	/* Else, it's the traditional escaped style */
> 	for (bc = 0, tp = inputText; *tp != '\0'; bc++)
> 	{
> 		if (tp[0] != '\\')
> 			tp++;
> 
> Sometimes SJIS uses '\' as the second byte of it.

Thanks, I'll try to understand the seriousness of the problem as I don't have good knowledge of character sets.  But your example seems to be telling everything about the difficulty...

Before digging into the problem, could you share your impression on whether PostgreSQL can support SJIS?  Would it be hopeless?  Can't we find any direction to go?  Can I find relevant source code by searching specific words like "ASCII", "HIGH_BIT", "\\" etc?

Regards
Takayuki Tsunakawa

  4. Re: Supporting SJIS as a database encoding

    Tatsuo Ishii <ishii@sraoss.co.jp> — 2016-09-05T08:19:25Z

    > Before digging into the problem, could you share your impression on whether PostgreSQL can support SJIS?  Would it be hopeless?  Can't we find any direction to go?  Can I find relevant source code by searching specific words like "ASCII", "HIGH_BIT", "\\" etc?

For starters, you could grep "multibyte".

Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp

  5. Re: Supporting SJIS as a database encoding

    Tom Lane <tgl@sss.pgh.pa.us> — 2016-09-05T14:47:15Z

    "Tsunakawa, Takayuki" <tsunakawa.takay@jp.fujitsu.com> writes:
> Before digging into the problem, could you share your impression on
> whether PostgreSQL can support SJIS?  Would it be hopeless?

I think it's pretty much hopeless.  Even if we were willing to make every
bit of code that looks for '\' and other specific at-risk characters
multi-byte aware (with attendant speed penalties), we could expect that
third-party extensions would still contain vulnerable code.  More, we
could expect that new bugs of the same ilk would get introduced all the
time.  Many such bugs would amount to security problems.  So the amount of
effort and vigilance required seems out of proportion to the benefits.

Most of the recent discussion about allowed backend encodings has run
more in the other direction, ie, "why don't we disallow everything but
UTF8 and get rid of all the infrastructure for multiple backend
encodings?".  I'm not personally in favor of that, but there are very
few hackers who want to add any more overhead in this area.

			regards, tom lane

  6. Re: Supporting SJIS as a database encoding

    Heikki Linnakangas <hlinnaka@iki.fi> — 2016-09-05T16:38:33Z

    On 09/05/2016 05:47 PM, Tom Lane wrote:
> "Tsunakawa, Takayuki" <tsunakawa.takay@jp.fujitsu.com> writes:
>> Before digging into the problem, could you share your impression on
>> whether PostgreSQL can support SJIS?  Would it be hopeless?
>
> I think it's pretty much hopeless.

Agreed.

But one thing that would help a little, would be to optimize the UTF-8 
-> SJIS conversion. It uses a very generic routine, with a binary search 
over a large array of mappings. I bet you could do better than that, 
maybe using a hash table or a radix tree instead of the large 
binary-searched array.

- Heikki

  7. Re: Supporting SJIS as a database encoding

    Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com> — 2016-09-06T02:51:13Z

    From: Tom Lane [mailto:tgl@sss.pgh.pa.us]
> "Tsunakawa, Takayuki" <tsunakawa.takay@jp.fujitsu.com> writes:
> > Before digging into the problem, could you share your impression on
> > whether PostgreSQL can support SJIS?  Would it be hopeless?
> 
> I think it's pretty much hopeless.  Even if we were willing to make every
> bit of code that looks for '\' and other specific at-risk characters
> multi-byte aware (with attendant speed penalties), we could expect that
> third-party extensions would still contain vulnerable code.  More, we could
> expect that new bugs of the same ilk would get introduced all the time.
> Many such bugs would amount to security problems.  So the amount of effort
> and vigilance required seems out of proportion to the benefits.

Hmm, this sounds like a death sentence.  But as I don't have good knowledge of character set handling yet, I'm not completely convinced about why PostgreSQL cannot support SJIS.  I wonder why and how other DBMSs support SJIS and what's the difference of the implementation.  Using multibyte-functions like mb... to process characters would solve the problem?  Isn't the current implementation blocking the support of other character sets that have similar characteristics?  I'll learn the character set handling...

> Most of the recent discussion about allowed backend encodings has run more
> in the other direction, ie, "why don't we disallow everything but
> UTF8 and get rid of all the infrastructure for multiple backend encodings?".
> I'm not personally in favor of that, but there are very few hackers who
> want to add any more overhead in this area.

Personally, I totally agree.  I want non-Unicode character sets to disappear from the world.  But the real business doesn't seem to forgive the lack of SJIS...

Regards
Takayuki Tsunakawa

  8. Re: Supporting SJIS as a database encoding

    Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com> — 2016-09-06T02:57:01Z

    From: pgsql-hackers-owner@postgresql.org
> [mailto:pgsql-hackers-owner@postgresql.org] On Behalf Of Heikki
> But one thing that would help a little, would be to optimize the UTF-8
> -> SJIS conversion. It uses a very generic routine, with a binary search
> over a large array of mappings. I bet you could do better than that, maybe
> using a hash table or a radix tree instead of the large binary-searched
> array.

That sounds worth pursuing.  Thanks!


Regards
Takayuki Tsunakawa

  9. Re: Supporting SJIS as a database encoding

    Tom Lane <tgl@sss.pgh.pa.us> — 2016-09-06T03:10:14Z

    "Tsunakawa, Takayuki" <tsunakawa.takay@jp.fujitsu.com> writes:
> Using multibyte-functions like mb... to process characters would solve
> the problem?

Well, sure.  The problem is (1) finding all the places that need that
(I'd estimate dozens to hundreds of places in the core code, and then
there's the question of extensions); (2) preventing new
non-multibyte-aware code from being introduced after you've fixed those
places; and (3) the performance penalties you'd take, because a lot of
those places are bottlenecks and it's much cheaper to not worry about
character lengths in an inner loop.

> Isn't the current implementation blocking the support of
> other character sets that have similar characteristics?

Sure, SJIS is not the only encoding that we consider frontend-only.
See

https://www.postgresql.org/docs/devel/static/multibyte.html#MULTIBYTE-CHARSET-SUPPORTED

			regards, tom lane

  10. Re: Supporting SJIS as a database encoding

    Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> — 2016-09-06T03:29:04Z

    Hello,

At Mon, 5 Sep 2016 19:38:33 +0300, Heikki Linnakangas <hlinnaka@iki.fi> wrote in <529db688-72fc-1ca2-f898-b0b99e30076f@iki.fi>
> On 09/05/2016 05:47 PM, Tom Lane wrote:
> > "Tsunakawa, Takayuki" <tsunakawa.takay@jp.fujitsu.com> writes:
> >> Before digging into the problem, could you share your impression on
> >> whether PostgreSQL can support SJIS?  Would it be hopeless?
> >
> > I think it's pretty much hopeless.
> 
> Agreed.

+1, even as a user of SJIS:)

> But one thing that would help a little, would be to optimize the UTF-8
> -> SJIS conversion. It uses a very generic routine, with a binary
> search over a large array of mappings. I bet you could do better than
> that, maybe using a hash table or a radix tree instead of the large
> binary-searched array.

I'm very impressed by the idea. Mean number of iterations for
binsearch on current conversion table with 8000 characters is
about 13 and the table size is under 100kBytes (maybe).

A three-level array with 2 byte values will take about 1.6~2MB of memory.

A radix tree for UTF-8->some-encoding conversion requires about,
or up to.. (using 1 byte index to point the next level)

(1 *  ((7f + 1) +
      (df - c2 + 1) * (bf - 80 + 1) +
      (ef - e0 + 1) * (bf - 80 + 1)^2)) = 67 kbytes.

SJIS characters are 2byte length at longest so about 8000
characters takes extra 16 k Bytes. And some padding space will be
added on them.

As the result, radix tree seems to be promising because of small
requirement of additional memory and far less comparisons.  Also
Big5 and other encodings including EUC-* will get benefit from
it.

Implementing radix tree code, then redefining the format of
mapping table to suppot radix tree, then modifying mapping
generator script are needed.

If no one oppse to this, I'll do that.

regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

  11. Re: Supporting SJIS as a database encoding

    Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com> — 2016-09-06T03:43:46Z

    > From: pgsql-hackers-owner@postgresql.org
> [mailto:pgsql-hackers-owner@postgresql.org] On Behalf Of Kyotaro
> HORIGUCHI
Implementing radix tree code, then redefining the format of mapping table
> to suppot radix tree, then modifying mapping generator script are needed.
> 
> If no one oppse to this, I'll do that.

+100
Great analysis and your guts.  I very much appreciate your trial!

Regards
Takayuki Tsunakawa

  12. Re: Supporting SJIS as a database encoding

    Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> — 2016-09-07T07:13:04Z

    Hello,

At Tue, 6 Sep 2016 03:43:46 +0000, "Tsunakawa, Takayuki" <tsunakawa.takay@jp.fujitsu.com> wrote in <0A3221C70F24FB45833433255569204D1F5E66CE@G01JPEXMBYT05>
> > From: pgsql-hackers-owner@postgresql.org
> > [mailto:pgsql-hackers-owner@postgresql.org] On Behalf Of Kyotaro
> > HORIGUCHI
> Implementing radix tree code, then redefining the format of mapping table
> > to suppot radix tree, then modifying mapping generator script are needed.
> > 
> > If no one oppse to this, I'll do that.
> 
> +100
> Great analysis and your guts.  I very much appreciate your trial!

Thanks, by the way, there's another issue related to SJIS
conversion.  MS932 has several characters that have multiple code
points. By converting texts in this encoding to and from Unicode
causes a round-trop problem. For example,

8754(ROMAN NUMERICAL I in NEC specials)
  => U+2160(ROMAN NUMERICAL I)
    => FA4A (ROMAN NUMERICA I in IBM extension)

My counting said that 398 characters are affected by this kind of
replacement. Addition to that, "GAIJI" (Private usage area) is
not allowed. Is this meet your purpose?

regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

  13. Re: Supporting SJIS as a database encoding

    Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com> — 2016-09-07T07:39:09Z

    From: pgsql-hackers-owner@postgresql.org
> [mailto:pgsql-hackers-owner@postgresql.org] On Behalf Of Kyotaro
> Thanks, by the way, there's another issue related to SJIS conversion.  MS932
> has several characters that have multiple code points. By converting texts
> in this encoding to and from Unicode causes a round-trop problem. For
> example,
> 
> 8754(ROMAN NUMERICAL I in NEC specials)
>   => U+2160(ROMAN NUMERICAL I)
>     => FA4A (ROMAN NUMERICA I in IBM extension)
> 
> My counting said that 398 characters are affected by this kind of replacement.
> Addition to that, "GAIJI" (Private usage area) is not allowed. Is this meet
> your purpose?

Supporting GAIJI is not a requirement as far as I know.  Thank you for sharing information.

# I realize my lack of knowledge about character sets...

Regards
Takayuki Tsunakawa

  14. Re: Supporting SJIS as a database encoding

    Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> — 2016-09-08T06:35:46Z

    Hello,

At Wed, 07 Sep 2016 16:13:04 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> wrote in <20160907.161304.112519789.horiguchi.kyotaro@lab.ntt.co.jp>
> > Implementing radix tree code, then redefining the format of mapping table
> > > to suppot radix tree, then modifying mapping generator script are needed.
> > > 
> > > If no one oppse to this, I'll do that.

So, I did that as a PoC. The radix tree takes a little less than
100k bytes (far smaller than expected:) and it is defnitely
faster than binsearch.


The attached patch does the following things.

- Defines a struct for static radix tree
  (utf_radix_tree). Currently it supports up to 3-byte encodings.

- Adds a map generator script UCS_to_SJIS_radix.pl, which
  generates utf8_to_sjis_radix.map from utf8_to_sjis.map.

- Adds a new conversion function utf8_to_sjis_radix.

- Modifies UtfToLocal so as to allow map to be NULL.

- Modifies utf8_to_sjis to use the new conversion function
  instead of ULmapSJIS.


The followings are to be done.

- utf8_to_sjis_radix could be more generic.

- SJIS->UTF8 is not implemented but it would be easily done since
  there's no difference in using the radix tree mechanism.
  (but the output character is currently assumed to be 2-byte long)

- It doesn't support 4-byte codes so this is not applicable to
  sjis_2004. Extending the radix tree to support 4-byte wouldn't
  be hard.


The following is the result of a simple test.

=# create table t (a text); alter table t alter column a storage plain;
=# insert into t values ('... 7130 cahracters containing (I believe) all characters in SJIS encoding');
=# insert into t values ('... 7130 cahracters containing (I believe) all characters in SJIS encoding');

# Doing that twice is just my mistake.

$ export PGCLIENTENCODING=SJIS

$ time psql postgres -c '
$ psql -c '\encoding' postgres
SJIS

<Using radix tree>
$ time psql postgres -c 'select t.a from t, generate_series(0, 9999)' > /dev/null

real	0m22.696s
user	0m16.991s
sys	0m0.182s>

Using binsearch the result for the same operation was 
real	0m35.296s
user	0m17.166s
sys	0m0.216s

Returning in UTF-8 bloats the result string by about 1.5 times so
it doesn't seem to make sense comparing with it. But it takes
real = 47.35s.

regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

  15. Re: Supporting SJIS as a database encoding

    Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com> — 2016-09-08T07:09:51Z

    From: pgsql-hackers-owner@postgresql.org
> [mailto:pgsql-hackers-owner@postgresql.org] On Behalf Of Kyotaro
> HORIGUCHI
> <Using radix tree>
> $ time psql postgres -c 'select t.a from t, generate_series(0, 9999)' >
> /dev/null
> 
> real	0m22.696s
> user	0m16.991s
> sys	0m0.182s>
> 
> Using binsearch the result for the same operation was
> real	0m35.296s
> user	0m17.166s
> sys	0m0.216s
> 
> Returning in UTF-8 bloats the result string by about 1.5 times so it doesn't
> seem to make sense comparing with it. But it takes real = 47.35s.

Cool, 36% speedup!  Does this difference vary depending on the actual characters used, e.g. the speedup would be greater if most of the characters are ASCII?

Regards
Takayuki Tsunakawa

  16. Re: Supporting SJIS as a database encoding

    Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> — 2016-09-13T02:04:27Z

    At Thu, 8 Sep 2016 07:09:51 +0000, "Tsunakawa, Takayuki" <tsunakawa.takay@jp.fujitsu.com> wrote in <0A3221C70F24FB45833433255569204D1F5E7D4A@G01JPEXMBYT05>
> From: pgsql-hackers-owner@postgresql.org
> > [mailto:pgsql-hackers-owner@postgresql.org] On Behalf Of Kyotaro
> > HORIGUCHI
> > <Using radix tree>
> > $ time psql postgres -c 'select t.a from t, generate_series(0, 9999)' >
> > /dev/null
> > 
> > real	0m22.696s
> > user	0m16.991s
> > sys	0m0.182s>
> > 
> > Using binsearch the result for the same operation was
> > real	0m35.296s
> > user	0m17.166s
> > sys	0m0.216s
> > 
> > Returning in UTF-8 bloats the result string by about 1.5 times so it doesn't
> > seem to make sense comparing with it. But it takes real = 47.35s.
> 
> Cool, 36% speedup!  Does this difference vary depending on the actual characters used, e.g. the speedup would be greater if most of the characters are ASCII?

Binsearch on JIS X 0208 always needs about 10 times of comparison
and bisecting and the radix tree requires three hops on arrays
for most of the characters and two hops for some. In sort, this
effect won't be differ among 2 and 3 byte characters in UTF-8.

The translation speed of ASCII cahracters (U+20 - U+7f) is not
affected by the character conversion mechanism. They are just
copied without conversion.

As the result, there's no speedup if the output consists only of
ASCII characters and maximum speedup when the output consists
only of 2 byte UTF-8 characters.

regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

  17. Re: Supporting SJIS as a database encoding

    Heikki Linnakangas <hlinnaka@iki.fi> — 2016-09-13T08:44:01Z

    On 09/08/2016 09:35 AM, Kyotaro HORIGUCHI wrote:
> Returning in UTF-8 bloats the result string by about 1.5 times so
> it doesn't seem to make sense comparing with it. But it takes
> real = 47.35s.

Nice!

I was hoping that this would also make the binaries smaller. A few dozen 
kB of storage is perhaps not a big deal these days, but still. And 
smaller tables would also consume less memory and CPU cache.

I removed the #include "../../Unicode/utf8_to_sjis.map" line, so that 
the old table isn't included anymore, compiled, and ran "strip 
utf8_and_sjis.so". Without this patch, it's 126 kB, and with it, it's 
160 kB. So the radix tree takes a little bit more space.

That's not too bad, and I'm sure we could live with that, but with a few 
simple tricks, we could do better. First, since all the values we store 
in the tree are < 0xffff, we could store them in int16 instead of int32, 
and halve the size of the table right off the bat. That won't work for 
all encodings, of course, but it might be worth it to have two versions 
of the code, one for int16 and another for int32.

Another trick is to eliminate redundancies in the tables. Many of the 
tables contain lots of zeros, as in:

>   /*   c3xx */{
>     /*   c380 */ 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
>     /*   c388 */ 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
>     /*   c390 */ 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x817e,
>     /*   c398 */ 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
>     /*   c3a0 */ 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
>     /*   c3a8 */ 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
>     /*   c3b0 */ 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x8180,
>     /*   c3b8 */ 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000
>   },

and

>   /* e388xx */{
>     /* e38880 */ 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
>     /* e38888 */ 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
>     /* e38890 */ 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
>     /* e38898 */ 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
>     /* e388a0 */ 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
>     /* e388a8 */ 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
>     /* e388b0 */ 0x0000, 0xfa58, 0x878b, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
>     /* e388b8 */ 0x0000, 0x878c, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000
>   },

You could overlay the last row of the first table, which is all zeros, 
with the first row of the second table, which is also all zeros. (Many 
of the tables have a lot more zero-rows than this example.)

But yes, this patch looks very promising in general. I think we should 
switch over to radix trees for the all the encodings.

- Heikki

  18. Re: Supporting SJIS as a database encoding

    Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> — 2016-09-21T06:14:27Z

    Hello,

At Tue, 13 Sep 2016 11:44:01 +0300, Heikki Linnakangas <hlinnaka@iki.fi> wrote in <7ff67a45-a53e-4d38-e25d-3a121afea47c@iki.fi>
> On 09/08/2016 09:35 AM, Kyotaro HORIGUCHI wrote:
> > Returning in UTF-8 bloats the result string by about 1.5 times so
> > it doesn't seem to make sense comparing with it. But it takes
> > real = 47.35s.
> 
> Nice!

Thanks!

> I was hoping that this would also make the binaries smaller. A few
> dozen kB of storage is perhaps not a big deal these days, but
> still. And smaller tables would also consume less memory and CPU
> cache.

Agreed.

> I removed the #include "../../Unicode/utf8_to_sjis.map" line, so that
> the old table isn't included anymore, compiled, and ran "strip
> utf8_and_sjis.so". Without this patch, it's 126 kB, and with it, it's
> 160 kB. So the radix tree takes a little bit more space.
> 
> That's not too bad, and I'm sure we could live with that, but with a
> few simple tricks, we could do better. First, since all the values we
> store in the tree are < 0xffff, we could store them in int16 instead
> of int32, and halve the size of the table right off the bat. won't work
> for all encodings, of course, but it might be worth it to
> have two versions of the code, one for int16 and another for int32.

That's right. I used int imprudently. All of the character in the
patch, and most of characters in other than Unicode-related
encodings are in 2 bytes. 3 bytes characters can be in separate
table in the struct for the case. Othersise two or more versions
of the structs is possible since currently the radix struct is
utf8_and_sjis's own in spite of the fact that it is in pg_wchar.h
in the patch.

> Another trick is to eliminate redundancies in the tables. Many of the
> tables contain lots of zeros, as in:
> 
> >   /*   c3xx */{
...
> >     0x817e,
> >     /* c398 */ 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
> >     0x0000,
> >     /* c3a0 */ 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
> >     0x0000,
> >     /* c3a8 */ 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
> >     0x0000,
> >     /* c3b0 */ 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
> >     0x8180,
> >     /* c3b8 */ 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
> >     0x0000
> >   },
> 
> and
> 
> >   /* e388xx */{
> >     /* e38880 */ 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
> >     0x0000,
> >     /* e38888 */ 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
> >     0x0000,
> >     /* e38890 */ 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
> >     0x0000,
> >     /* e38898 */ 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
> >     0x0000,
> >     /* e388a0 */ 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000,
> >     0x0000,
...
> >   },
> 
> You could overlay the last row of the first table, which is all zeros,
> with the first row of the second table, which is also all zeros. (Many
> of the tables have a lot more zero-rows than this example.)

Yes, the bunch of zeros was annoyance. Several or many
compression techniques are available in exchange for some
additional CPU time. But the technique you suggested doesn't
need such sacrifice, sounds nice.

> But yes, this patch looks very promising in general. I think we should
> switch over to radix trees for the all the encodings.

The result was more than I expected for a character set with
about 7000 characters. We can expect certain amount of advangate
even for character sets that have less than a hundred of
characters.


I'll work on this for the next CF.

Thanks.

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

  19. Re: Supporting SJIS as a database encoding

    Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> — 2016-10-07T08:32:00Z

    Hello, did this.

As a result, radix tree is about 1.5 times faster and needs a
half memory.

At Wed, 21 Sep 2016 15:14:27 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> wrote in <20160921.151427.265121484.horiguchi.kyotaro@lab.ntt.co.jp>
> I'll work on this for the next CF.

The radix conversion function and map conversion script became
more generic than the previous state. So I could easily added
radix conversion of EUC_JP in addition to SjiftJIS.

nm -S said that the size of radix tree data for sjis->utf8
conversion is 34kB and that for utf8->sjis is 46kB.  (eucjp->utf8
57kB, utf8->eucjp 93kB) LUmapSJIS and ULmapSJIS was 62kB and
59kB, and LUmapEUC_JP and ULmapEUC_JP was 106kB and 105kB. If I'm
not missing something, radix tree is faster and require less
memory.

A simple test where 'select '7070 sjis chars' x 100' (I'm not
sure, but the size is 1404kB) on local connection shows that this
is fast enough.

radix:  real 0m0.285s / user 0m0.199s / sys 0m0.006s
master: real 0m0.418s / user 0m0.180s / sys 0m0.004s

To make sure, the result of a test of sending the same amount of
ASCII string (1404kB) on SJIS and UTF8(no-conversion) encoding is
as follows.

ascii/utf8-sjis: real 0m0.220s / user 0m0.176s / sys 0m0.011s
ascii/utf8-utf8: real 0m0.137s / user 0m0.111s / sys 0m0.008s

======
Random discussions -

Currently the tree structure is devided into several elements,
One for 2-byte, other ones for 3-byte and 4-byte codes and output
table. The other than the last one is logically and technically
merged into single table but it makes the generator script far
complex than the current complexity. I no longer want to play
hide'n seek with complex perl object..

It might be better that combining this as a native feature of the
core. Currently the helper function is in core but that function
is given as conv_func on calling LocalToUtf.

Current implement uses *.map files of pg_utf_to_local as
input. It seems not good but the radix tree files is completely
uneditable. Provide custom made loading functions for every
source instead of load_chartable() would be the way to go.

# However, for example utf8_to_sjis.map, it doesn't seem to have
# generated from the source mentioned in UCS_to_SJIS.pl

I'm not sure that compilers other than gcc accepts generated map
file content.

The RADIXTREE.pm is in rather older style but seem no problem.

I haven't tried this for charsets that contains 4-byte code.

I haven't consider charset with conbined characters. I don't
think it is needed so immediately.

Though I believe that this is easily applied to other
conversions, I tried this only with character sets that I know
about it.


regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

  20. Radix tree for character conversion

    Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> — 2016-10-07T08:36:06Z

    This is a differnet topic from the original thread so I renamed
the subject and repost. Sorry for duplicate posting.

======================
Hello, I did this.

As a result, radix tree is about 1.5 times faster and needs a
half memory.

At Wed, 21 Sep 2016 15:14:27 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> wrote in <20160921.151427.265121484.horiguchi.kyotaro@lab.ntt.co.jp>
> I'll work on this for the next CF.

The radix conversion function and map conversion script became
more generic than the previous state. So I could easily added
radix conversion of EUC_JP in addition to SjiftJIS.

nm -S said that the size of radix tree data for sjis->utf8
conversion is 34kB and that for utf8->sjis is 46kB.  (eucjp->utf8
57kB, utf8->eucjp 93kB) LUmapSJIS and ULmapSJIS was 62kB and
59kB, and LUmapEUC_JP and ULmapEUC_JP was 106kB and 105kB. If I'm
not missing something, radix tree is faster and require less
memory.

A simple test where 'select '7070 sjis chars' x 100' (I'm not
sure, but the size is 1404kB) on local connection shows that this
is fast enough.

radix:  real 0m0.285s / user 0m0.199s / sys 0m0.006s
master: real 0m0.418s / user 0m0.180s / sys 0m0.004s

To make sure, the result of a test of sending the same amount of
ASCII string (1404kB) on SJIS and UTF8(no-conversion) encoding is
as follows.

ascii/utf8-sjis: real 0m0.220s / user 0m0.176s / sys 0m0.011s
ascii/utf8-utf8: real 0m0.137s / user 0m0.111s / sys 0m0.008s

======
Random discussions -

Currently the tree structure is devided into several elements,
One for 2-byte, other ones for 3-byte and 4-byte codes and output
table. The other than the last one is logically and technically
merged into single table but it makes the generator script far
complex than the current complexity. I no longer want to play
hide'n seek with complex perl object..

It might be better that combining this as a native feature of the
core. Currently the helper function is in core but that function
is given as conv_func on calling LocalToUtf.

Current implement uses *.map files of pg_utf_to_local as
input. It seems not good but the radix tree files is completely
uneditable. Provide custom made loading functions for every
source instead of load_chartable() would be the way to go.

# However, for example utf8_to_sjis.map, it doesn't seem to have
# generated from the source mentioned in UCS_to_SJIS.pl

I'm not sure that compilers other than gcc accepts generated map
file content.

The RADIXTREE.pm is in rather older style but seem no problem.

I haven't tried this for charsets that contains 4-byte code.

I haven't consider charset with conbined characters. I don't
think it is needed so immediately.

Though I believe that this is easily applied to other
conversions, I tried this only with character sets that I know
about it.


regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

  21. Re: Radix tree for character conversion

    Heikki Linnakangas <hlinnaka@iki.fi> — 2016-10-07T10:46:31Z

    On 10/07/2016 11:36 AM, Kyotaro HORIGUCHI wrote:
> The radix conversion function and map conversion script became
> more generic than the previous state. So I could easily added
> radix conversion of EUC_JP in addition to SjiftJIS.
>
> nm -S said that the size of radix tree data for sjis->utf8
> conversion is 34kB and that for utf8->sjis is 46kB.  (eucjp->utf8
> 57kB, utf8->eucjp 93kB) LUmapSJIS and ULmapSJIS was 62kB and
> 59kB, and LUmapEUC_JP and ULmapEUC_JP was 106kB and 105kB. If I'm
> not missing something, radix tree is faster and require less
> memory.

Cool!

> Currently the tree structure is devided into several elements,
> One for 2-byte, other ones for 3-byte and 4-byte codes and output
> table. The other than the last one is logically and technically
> merged into single table but it makes the generator script far
> complex than the current complexity. I no longer want to play
> hide'n seek with complex perl object..

I think that's OK. There isn't really anything to gain by merging them.

> It might be better that combining this as a native feature of the
> core. Currently the helper function is in core but that function
> is given as conv_func on calling LocalToUtf.

Yeah, I think we want to completely replace the current binary-search 
based code with this. I would rather maintain just one mechanism.

> Current implement uses *.map files of pg_utf_to_local as
> input. It seems not good but the radix tree files is completely
> uneditable. Provide custom made loading functions for every
> source instead of load_chartable() would be the way to go.
>
> # However, for example utf8_to_sjis.map, it doesn't seem to have
> # generated from the source mentioned in UCS_to_SJIS.pl

Ouch. We should find and document an authoritative source for all the 
mappings we have...

I think the next steps here are:

1. Find an authoritative source for all the existing mappings.
2. Generate the radix tree files directly from the authoritative 
sources, instead of the existing *.map files.
3. Completely replace the existing binary-search code with this.

- Heikki

  22. Re: Radix tree for character conversion

    Robert Haas <robertmhaas@gmail.com> — 2016-10-07T15:55:10Z

    On Fri, Oct 7, 2016 at 6:46 AM, Heikki Linnakangas <hlinnaka@iki.fi> wrote:
> Ouch. We should find and document an authoritative source for all the
> mappings we have...
>
> I think the next steps here are:
>
> 1. Find an authoritative source for all the existing mappings.
> 2. Generate the radix tree files directly from the authoritative sources,
> instead of the existing *.map files.
> 3. Completely replace the existing binary-search code with this.

It might be best to convert using the existing map files, and then
update the mappings later.  Otherwise, when things break, you won't
know what to blame.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  23. Re: Radix tree for character conversion

    Tom Lane <tgl@sss.pgh.pa.us> — 2016-10-07T16:19:23Z

    Robert Haas <robertmhaas@gmail.com> writes:
> On Fri, Oct 7, 2016 at 6:46 AM, Heikki Linnakangas <hlinnaka@iki.fi> wrote:
>> Ouch. We should find and document an authoritative source for all the
>> mappings we have...
>> 
>> I think the next steps here are:
>> 
>> 1. Find an authoritative source for all the existing mappings.
>> 2. Generate the radix tree files directly from the authoritative sources,
>> instead of the existing *.map files.
>> 3. Completely replace the existing binary-search code with this.

> It might be best to convert using the existing map files, and then
> update the mappings later.  Otherwise, when things break, you won't
> know what to blame.

I think I went through this exercise last year or so, and updated the
notes about the authoritative sources where I was able to find one.
In the remaining cases, I believe that the maps have been intentionally
tweaked and we should be cautious about undoing that.  Tatsuo-san might
remember more about why they are the way they are.

			regards, tom lane

  24. Re: Radix tree for character conversion

    Heikki Linnakangas <hlinnaka@iki.fi> — 2016-10-07T21:37:28Z

    On 10/07/2016 06:55 PM, Robert Haas wrote:
> On Fri, Oct 7, 2016 at 6:46 AM, Heikki Linnakangas <hlinnaka@iki.fi> wrote:
>> Ouch. We should find and document an authoritative source for all the
>> mappings we have...
>>
>> I think the next steps here are:
>>
>> 1. Find an authoritative source for all the existing mappings.
>> 2. Generate the radix tree files directly from the authoritative sources,
>> instead of the existing *.map files.
>> 3. Completely replace the existing binary-search code with this.
>
> It might be best to convert using the existing map files, and then
> update the mappings later.  Otherwise, when things break, you won't
> know what to blame.

I was thinking that we keep the mappings unchanged, but figure out where 
we got the mappings we have. An authoritative source may well be "file X 
from unicode, with the following tweaks: ...". As long as we have some 
way of representing that, in text files, or in perl code, that's OK.

What I don't want is that the current *.map files are turned into the 
authoritative source files, that we modify by hand. There are no 
comments in them, for starters, which makes hand-editing cumbersome. It 
seems that we have edited some of them by hand already, but we should 
rectify that.

- Heikki

  25. Re: Radix tree for character conversion

    Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> — 2016-10-21T08:33:21Z

    Hello, this is new version of radix charconv.

At Sat, 8 Oct 2016 00:37:28 +0300, Heikki Linnakangas <hlinnaka@iki.fi> wrote in <6d85d710-9554-a928-29ff-b2d3b80b01c9@iki.fi>
> What I don't want is that the current *.map files are turned into the
> authoritative source files, that we modify by hand. There are no
> comments in them, for starters, which makes hand-editing
> cumbersome. It seems that we have edited some of them by hand already,
> but we should rectify that.

Agreed. So, I identifed source files of each character for EUC_JP
and SJIS conversions to clarify what has been done on them.

SJIS conversion is made from CP932.TXT and 8 additional
conversions for UTF8->SJIS and none for SJIS->UTF8.

EUC_JP is made from CP932.TXT and JIS0212.TXT. JIS0201.TXT and
JIS0208.TXT are useless. It adds 83 or 86 (different by
direction) conversion entries.

http://unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/CP932.TXT
http://unicode.org/Public/MAPPINGS/OBSOLETE/EASTASIA/JIS/JIS0212.TXT

Now the generator scripts don't use *.map as source and in turn
generates old-style map files as well as radix tree files.

For convenience, UCS_to_(SJIS|EUC_JP).pl takes parater --flat and
-v. The format generates the old-style flat map as well as radix
map file and additional -v adds source description for each line
in the flat map file.

During working on this, EUC_JP map lacks some conversions but it
is another issue.

regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

  26. Re: Radix tree for character conversion

    Heikki Linnakangas <hlinnaka@iki.fi> — 2016-10-25T09:23:48Z

    On 10/21/2016 11:33 AM, Kyotaro HORIGUCHI wrote:
> Hello, this is new version of radix charconv.
>
> At Sat, 8 Oct 2016 00:37:28 +0300, Heikki Linnakangas <hlinnaka@iki.fi> wrote in <6d85d710-9554-a928-29ff-b2d3b80b01c9@iki.fi>
>> What I don't want is that the current *.map files are turned into the
>> authoritative source files, that we modify by hand. There are no
>> comments in them, for starters, which makes hand-editing
>> cumbersome. It seems that we have edited some of them by hand already,
>> but we should rectify that.
>
> Agreed. So, I identifed source files of each character for EUC_JP
> and SJIS conversions to clarify what has been done on them.
>
> SJIS conversion is made from CP932.TXT and 8 additional
> conversions for UTF8->SJIS and none for SJIS->UTF8.
>
> EUC_JP is made from CP932.TXT and JIS0212.TXT. JIS0201.TXT and
> JIS0208.TXT are useless. It adds 83 or 86 (different by
> direction) conversion entries.
>
> http://unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/CP932.TXT
> http://unicode.org/Public/MAPPINGS/OBSOLETE/EASTASIA/JIS/JIS0212.TXT
>
> Now the generator scripts don't use *.map as source and in turn
> generates old-style map files as well as radix tree files.
>
> For convenience, UCS_to_(SJIS|EUC_JP).pl takes parater --flat and
> -v. The format generates the old-style flat map as well as radix
> map file and additional -v adds source description for each line
> in the flat map file.
>
> During working on this, EUC_JP map lacks some conversions but it
> is another issue.

Thanks!

I'd reallly like to clean up all the current perl scripts, before we 
start to do the radix tree stuff. I worked through the rest of the 
conversions, and fixed/hacked the perl scripts so that they faithfully 
re-produce the mapping tables that we have in the repository currently. 
Whether those are the best mappings or not, or whether we should update 
them based on some authoritative source is another question, but let's 
try to nail down the process of creating the mapping tables.

Tom Lane looked into this in Nov 2015 
(https://www.postgresql.org/message-id/28825.1449076551%40sss.pgh.pa.us). 
This is a continuation of that, to actually fix the scripts. This patch 
series doesn't change any of the mappings, only the way we produce the 
mapping tables.

Our UHC conversion tables contained a lot more characters than the 
CP949.TXT file it's supposedly based on. I rewrote the script to use 
"windows-949-2000.xml" file, from the ICU project, as the source 
instead. It's a much closer match to our mapping tables, containing all 
but one of the additional characters. We were already using 
gb-18030-2000.xml as the source in UCS_GB18030.pl, so parsing ICU's XML 
files isn't a new thing.

The GB2312.TXT source file seems to have disappeared from the Unicode 
consortium's FTP site. I changed the UCS_to_EUC_CN.pl script to use 
gb-18030-2000.xml as the source instead. GB-18030 is an extension of 
GB-2312, UCS_to_EUC_CN.pl filters out the additional characters that are 
not in GB-2312.

This now forms a reasonable basis for switching to radix tree. Every 
mapping table is now generated by the print_tables() perl function in 
convutils.pm. To switch to a radix tree, you just need to swap that 
function with one that produces a radix tree instead of the 
current-format mapping tables.

The perl scripts are still quite messy. For example, I lost the checks 
for duplicate mappings somewhere along the way - that ought to be put 
back. My Perl skills are limited.


This is now an orthogonal discussion, and doesn't need to block the 
radix tree work, but we should consider what we want to base our mapping 
tables on. Perhaps we could use the XML files from ICU as the source for 
all of the mappings?

ICU seems to use a BSD-like license, so we could even include the XML 
files in our repository. Actually, looking at 
http://www.unicode.org/copyright.html#License, I think we could include 
the *.TXT files in our repository, too, if we wanted to. The *.TXT files 
are found under www.unicode.org/Public/, so that license applies. I 
think that has changed somewhat recently, because the comments in our 
perl scripts claim that the license didn't allow that.

- Heikki

  27. Re: Radix tree for character conversion

    Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> — 2016-10-27T07:23:37Z

    Hello, thank you very much for the work. My work became quite
easier with it.

At Tue, 25 Oct 2016 12:23:48 +0300, Heikki Linnakangas <hlinnaka@iki.fi> wrote in <08e7892a-d55c-eefe-76e6-7910bc8dd1f3@iki.fi>
> I'd reallly like to clean up all the current perl scripts, before we
> start to do the radix tree stuff. I worked through the rest of the
> conversions, and fixed/hacked the perl scripts so that they faithfully
> re-produce the mapping tables that we have in the repository
> currently. Whether those are the best mappings or not, or whether we
> should update them based on some authoritative source is another
> question, but let's try to nail down the process of creating the
> mapping tables.
> 
> Tom Lane looked into this in Nov 2015
> (https://www.postgresql.org/message-id/28825.1449076551%40sss.pgh.pa.us). This
> is a continuation of that, to actually fix the scripts. This patch
> series doesn't change any of the mappings, only the way we produce the
> mapping tables.
> 
> > Our UHC conversion tables contained a lot more characters than the
> CP949.TXT file it's supposedly based on. I rewrote the script to use
> "windows-949-2000.xml" file, from the ICU project, as the source
> instead. It's a much closer match to our mapping tables, containing
> all but one of the additional characters. We were already using
> gb-18030-2000.xml as the source in UCS_GB18030.pl, so parsing ICU's
> XML files isn't a new thing.
> 
> The GB2312.TXT source file seems to have disappeared from the Unicode
> consortium's FTP site. I changed the UCS_to_EUC_CN.pl script to use
> gb-18030-2000.xml as the source instead. GB-18030 is an extension of
> GB-2312, UCS_to_EUC_CN.pl filters out the additional characters that
> are not in GB-2312.
> 
> This now forms a reasonable basis for switching to radix tree. Every
> mapping table is now generated by the print_tables() perl function in
> convutils.pm. To switch to a radix tree, you just need to swap that
> function with one that produces a radix tree instead of the
> current-format mapping tables.

RADIXCONV.pm is merged into convutils.pm and the manner to
resolve reference is unified from $$x{} to $x->{}. (subroutine
call by '&' is not unified..) Now radix trees files are written
by the function with similar interface.

print_radix_trees($script_name, $encoding, \@mapping);

> The perl scripts are still quite messy. For example, I lost the checks
> for duplicate mappings somewhere along the way - that ought to be put
> back. My Perl skills are limited.

Perl scripts are to be messy, I believe. Anyway the duplicate
check as been built into the sub print_radix_trees. Maybe the
same check is needed by some plain map files but it would be just
duplication for the maps having radix tree.

The attached patches apply on top your patches and changes all
possible conversions to use radix tree (combined characters are
still using old-method). Addition to that, because of the
difficult-to-verify nature of the radix-tree data, I added
map_chekcer (make mapcheck) to check them agaist plain maps.

I have briefly checked with real characters for
SJIS/EUC-JP/BIG5/ISO8859-13 and radix conversion seems to work
correctly for them.

> This is now an orthogonal discussion, and doesn't need to block the
> radix tree work, but we should consider what we want to base our
> mapping tables on. Perhaps we could use the XML files from ICU as the
> source for all of the mappings?
> 
> ICU seems to use a BSD-like license, so we could even include the XML
> files in our repository. Actually, looking at
> http://www.unicode.org/copyright.html#License, I think we could
> include the *.TXT files in our repository, too, if we wanted to. The
> *.TXT files are found under www.unicode.org/Public/, so that license
> applies. I think that has changed somewhat recently, because the
> comments in our perl scripts claim that the license didn't allow that.

For the convenience, all the required files are downloaded by
typing 'make download-texts'.


In the following document,

http://unicode.org/Public/ReadMe.txt

| Terms of Use
| 	http://www.unicode.org/copyright.html

http://www.unicode.org/copyright.html

| EXHIBIT 1
| UNICODE, INC. LICENSE AGREEMENT - DATA FILES AND SOFTWARE
| Unicode Data Files include all data files under the directories
| http://www.unicode.org/Public/, http://www.unicode.org/reports/,
...

| COPYRIGHT AND PERMISSION NOTICE
| 
| Copyright (c) 1991-2016 Unicode, Inc. All rights reserved.
| Distributed under the Terms of Use in http://www.unicode.org/copyright.html.
| 
| Permission is hereby granted, free of charge, to any person obtaining
| a copy of the Unicode data files and any associated documentation
| (the "Data Files") or Unicode software and any associated documentation
| (the "Software") to deal in the Data Files or Software
| without restriction, including without limitation the rights to use,
| copy, modify, merge, publish, distribute, and/or sell copies of
| the Data Files or Software, and to permit persons to whom the Data Files
| or Software are furnished to do so, provided that either
| (a) this copyright and permission notice appear with all copies
| of the Data Files or Software, or
| (b) this copyright and permission notice appear in associated
| Documentation.

Perhaps we can put the files into our repositoy by providing some
notifications.

regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

  28. Re: Radix tree for character conversion

    Robert Haas <robertmhaas@gmail.com> — 2016-10-28T13:18:08Z

    On Thu, Oct 27, 2016 at 3:23 AM, Kyotaro HORIGUCHI
<horiguchi.kyotaro@lab.ntt.co.jp> wrote:
> | COPYRIGHT AND PERMISSION NOTICE
> |
> | Copyright (c) 1991-2016 Unicode, Inc. All rights reserved.
> | Distributed under the Terms of Use in http://www.unicode.org/copyright.html.
> |
> | Permission is hereby granted, free of charge, to any person obtaining
> | a copy of the Unicode data files and any associated documentation
> | (the "Data Files") or Unicode software and any associated documentation
> | (the "Software") to deal in the Data Files or Software
> | without restriction, including without limitation the rights to use,
> | copy, modify, merge, publish, distribute, and/or sell copies of
> | the Data Files or Software, and to permit persons to whom the Data Files
> | or Software are furnished to do so, provided that either
> | (a) this copyright and permission notice appear with all copies
> | of the Data Files or Software, or
> | (b) this copyright and permission notice appear in associated
> | Documentation.
>
> Perhaps we can put the files into our repositoy by providing some
> notifications.

Uggh, I don't much like advertising clauses.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  29. Re: Radix tree for character conversion

    Tom Lane <tgl@sss.pgh.pa.us> — 2016-10-28T13:42:25Z

    Robert Haas <robertmhaas@gmail.com> writes:
> On Thu, Oct 27, 2016 at 3:23 AM, Kyotaro HORIGUCHI
> <horiguchi.kyotaro@lab.ntt.co.jp> wrote:
>> Perhaps we can put the files into our repositoy by providing some
>> notifications.

> Uggh, I don't much like advertising clauses.

Even if the license were exactly compatible with ours, I'd be -1 on
bloating our tarballs with these files.  They're large and only a
tiny fraction of developers, let alone end users, will ever care
to look at them.

I think it's fine as long as we have a README file that explains
where to get them.  (I'm not even very thrilled with the proposed
auto-download script, as it makes undesirable assumptions about
which internet tools you use, not to mention that it won't work
at all on Windows.)

I'd actually vote for getting rid of the reference files we
have in the tree now (src/backend/utils/mb/Unicode/*txt), on
the same grounds.  That's 600K of stuff that does not need to
be in our tarballs.

			regards, tom lane

  30. Re: Radix tree for character conversion

    David Fetter <david@fetter.org> — 2016-10-28T17:12:39Z

    On Fri, Oct 28, 2016 at 09:18:08AM -0400, Robert Haas wrote:
> On Thu, Oct 27, 2016 at 3:23 AM, Kyotaro HORIGUCHI
> <horiguchi.kyotaro@lab.ntt.co.jp> wrote:
> > | COPYRIGHT AND PERMISSION NOTICE
> > |
> > | Copyright (c) 1991-2016 Unicode, Inc. All rights reserved.
> > | Distributed under the Terms of Use in http://www.unicode.org/copyright.html.
> > |
> > | Permission is hereby granted, free of charge, to any person obtaining
> > | a copy of the Unicode data files and any associated documentation
> > | (the "Data Files") or Unicode software and any associated documentation
> > | (the "Software") to deal in the Data Files or Software
> > | without restriction, including without limitation the rights to use,
> > | copy, modify, merge, publish, distribute, and/or sell copies of
> > | the Data Files or Software, and to permit persons to whom the Data Files
> > | or Software are furnished to do so, provided that either
> > | (a) this copyright and permission notice appear with all copies
> > | of the Data Files or Software, or
> > | (b) this copyright and permission notice appear in associated
> > | Documentation.
> >
> > Perhaps we can put the files into our repositoy by providing some
> > notifications.
> 
> Uggh, I don't much like advertising clauses.

Your dislike is pretty common.

Might it be worth reaching out to the Unicode consortium about this?
They may well have added that as boilerplate without really
considering the effects, and they even have a popup that specifically
addresses licensing.

http://www.unicode.org/reporting.html

Best,
David.
-- 
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778  AIM: dfetter666  Yahoo!: dfetter
Skype: davidfetter      XMPP: david(dot)fetter(at)gmail(dot)com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

  31. Re: Radix tree for character conversion

    Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> — 2016-10-31T00:59:15Z

    Hello,

At Fri, 28 Oct 2016 09:42:25 -0400, Tom Lane <tgl@sss.pgh.pa.us> wrote in <13049.1477662145@sss.pgh.pa.us>
> Robert Haas <robertmhaas@gmail.com> writes:
> > On Thu, Oct 27, 2016 at 3:23 AM, Kyotaro HORIGUCHI
> > <horiguchi.kyotaro@lab.ntt.co.jp> wrote:
> >> Perhaps we can put the files into our repositoy by providing some
> >> notifications.
> 
> > Uggh, I don't much like advertising clauses.
> 
> Even if the license were exactly compatible with ours, I'd be -1 on
> bloating our tarballs with these files.  They're large and only a
> tiny fraction of developers, let alone end users, will ever care
> to look at them.

I understood that the intention of Heikki's suggestion, that is,
these might be included in PostgreSQL's repository, is looking
for a kind of stability, or consistency. The source files are not
revision-mangaged. In case where the authorities get unwanted
changes or no longer avaiable, .map files have to be edited
irelevantly from the authority files maybe from the reason that
. Actually some map files have lost their authority file or other
map files have got several direct modifications. We will be free
from such disturbance by containing "frozen" authority files.

On the other hand, I also agree that the advertising or
additional bloats of source repositiry are a nuisance.

> I think it's fine as long as we have a README file that explains
> where to get them.  (I'm not even very thrilled with the proposed
> auto-download script, as it makes undesirable assumptions about
> which internet tools you use, not to mention that it won't work
> at all on Windows.)

Mmm. It would be a pain in the neck. Some of the files are
already stored in "OBSOLETE" directory in the Unicode consortium
ftp site, and one of them has been vanished and available from
another place, a part of ICU source tree. On the other hand map
files are assumed to be generated from the scripts and are to
discuraged to be directly edited. Radix map files are uneditable
and currently made from the authority files. If some authority
files are gone, the additional edit have to be done directly onto
map files, and they are in turn to be the authority for radix
files.  (it's quite easy to chage the authority to current map
files, though).

By the way, the following phrase of the terms of license.

http://www.unicode.org/copyright.html

| COPYRIGHT AND PERMISSION NOTICE
| 
| Copyright (c) 1991-2016 Unicode, Inc. All rights reserved.
| Distributed under the Terms of Use in http://www.unicode.org/copyright.html.
| 
| Permission is hereby granted, free of charge, to any person obtaining
| a copy of the Unicode data files and any associated documentation
| (the "Data Files") or Unicode software and any associated documentation
| (the "Software") to deal in the Data Files or Software
| without restriction, including without limitation the rights to use,
| copy, modify, merge, publish, distribute, and/or sell copies of
| the Data Files or Software, and to permit persons to whom the Data Files
| or Software are furnished to do so, provided that either
| (a) this copyright and permission notice appear with all copies
| of the Data Files or Software, or
| (b) this copyright and permission notice appear in associated
| Documentation.

I'm afraid that the map (and _radix.map files) are the translates
of the "Data Files", and 'translate' is a part of 'modify'.

Either the notice is necessary or not, if we decide to wipe the
'true' authority out from our source files, I'd like to make the
map files (preferably with comments) as the second authority,
_radix.map files are to be getenerated from them, since they're
not editable.

> I'd actually vote for getting rid of the reference files we
> have in the tree now (src/backend/utils/mb/Unicode/*txt), on
> the same grounds.  That's 600K of stuff that does not need to
> be in our tarballs.

Anyway, I'd like to register this as an item of this CF.

regares,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

  32. Re: Radix tree for character conversion

    Daniel Gustafsson <daniel@yesql.se> — 2016-10-31T16:11:17Z

    > On 27 Oct 2016, at 09:23, Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> wrote:
> 
> Hello, thank you very much for the work. My work became quite
> easier with it.
> 
> At Tue, 25 Oct 2016 12:23:48 +0300, Heikki Linnakangas <hlinnaka@iki.fi> wrote in <08e7892a-d55c-eefe-76e6-7910bc8dd1f3@iki.fi>
>> 
>> [..]
>> The perl scripts are still quite messy. For example, I lost the checks
>> for duplicate mappings somewhere along the way - that ought to be put
>> back. My Perl skills are limited.
> 
> Perl scripts are to be messy, I believe. Anyway the duplicate
> check as been built into the sub print_radix_trees. Maybe the
> same check is needed by some plain map files but it would be just
> duplication for the maps having radix tree.

I took a small stab at doing some cleaning of the Perl scripts, mainly around
using the more modern (well, modern as in +15 years old) form for open(..),
avoiding global filehandles for passing scalar references and enforcing use
strict.  Some smaller typos and fixes were also included.  It seems my Perl has
become a bit rusty so I hope the changes make sense.  The produced files are
identical with these patches applied, they are merely doing cleaning as opposed
to bugfixing.

The attached patches are against the 0001-0006 patches from Heikki and you in
this series of emails, the separation is intended to make them easier to read.

cheers ./daniel

  33. Re: Radix tree for character conversion

    Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> — 2016-11-04T07:34:33Z

    Thank you for looling this.

At Mon, 31 Oct 2016 17:11:17 +0100, Daniel Gustafsson <daniel@yesql.se> wrote in <3FC648B5-2B7F-4585-9615-207A44B730A9@yesql.se>
> > On 27 Oct 2016, at 09:23, Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> wrote:
> > Perl scripts are to be messy, I believe. Anyway the duplicate
> > check as been built into the sub print_radix_trees. Maybe the
> > same check is needed by some plain map files but it would be just
> > duplication for the maps having radix tree.
> 
> I took a small stab at doing some cleaning of the Perl scripts, mainly around
> using the more modern (well, modern as in +15 years old) form for open(..),
> avoiding global filehandles for passing scalar references and enforcing use
> strict.  Some smaller typos and fixes were also included.  It seems my Perl has
> become a bit rusty so I hope the changes make sense.  The produced files are
> identical with these patches applied, they are merely doing cleaning as opposed
> to bugfixing.
> 
> The attached patches are against the 0001-0006 patches from Heikki and you in
> this series of emails, the separation is intended to make them easier to read.

I'm not sure how the discussion about this goes, these patches
makes me think about coding style of Perl.

The distinction between executable script and library is by
intention with an obscure basis. Existing scripts don't get less
modification, but library uses more restricted scopes to get rid
of the troubles caused by using global scopes. But I don't have a
clear preference on that. The TAP test scripts takes OO notations
but I'm not sure convutils.pl also be better to take the same
notation. It would be rarely edited hereafter and won't gets
grown any more.

As far as I see the obvious bug fixes in the patchset are the
following,

- 0007: load_maptable fogets to close input file.
- 0010: commment for load_maptables is wrong.
- 0011: hash reference is incorrectly dereferenced

All other fixes other than the above three seem to be styling or
syntax-generation issues and I don't know whether any
recommendation exists...


regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

  34. Re: Radix tree for character conversion

    Daniel Gustafsson <daniel@yesql.se> — 2016-11-07T11:32:55Z

    > On 04 Nov 2016, at 08:34, Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> wrote:
> 
> Thank you for looling this.

And thank you for taking the time to read my patches!

> At Mon, 31 Oct 2016 17:11:17 +0100, Daniel Gustafsson <daniel@yesql.se> wrote in <3FC648B5-2B7F-4585-9615-207A44B730A9@yesql.se>
>>> On 27 Oct 2016, at 09:23, Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> wrote:
>>> Perl scripts are to be messy, I believe. Anyway the duplicate
>>> check as been built into the sub print_radix_trees. Maybe the
>>> same check is needed by some plain map files but it would be just
>>> duplication for the maps having radix tree.
>> 
>> I took a small stab at doing some cleaning of the Perl scripts, mainly around
>> using the more modern (well, modern as in +15 years old) form for open(..),
>> avoiding global filehandles for passing scalar references and enforcing use
>> strict.  Some smaller typos and fixes were also included.  It seems my Perl has
>> become a bit rusty so I hope the changes make sense.  The produced files are
>> identical with these patches applied, they are merely doing cleaning as opposed
>> to bugfixing.
>> 
>> The attached patches are against the 0001-0006 patches from Heikki and you in
>> this series of emails, the separation is intended to make them easier to read.
> 
> I'm not sure how the discussion about this goes, these patches
> makes me think about coding style of Perl.

Some of this can absolutely be considered style and more or less down to
personal preference.  I haven’t seen any coding conventions for Perl so I
assume it’s down to consensus among the committers.  My rationale for these
patches in the first place was that I perceived this thread to partly want to
clean up the code and make it more modern Perl.

> The distinction between executable script and library is by
> intention with an obscure basis. Existing scripts don't get less
> modification, but library uses more restricted scopes to get rid
> of the troubles caused by using global scopes. But I don't have a
> clear preference on that. The TAP test scripts takes OO notations
> but I'm not sure convutils.pl also be better to take the same
> notation. It would be rarely edited hereafter and won't gets
> grown any more.

I think the current convutils module is fine and converting it to OO would be
overkill.

> As far as I see the obvious bug fixes in the patchset are the
> following,

Agreed, with some comments:

> - 0007: load_maptable fogets to close input file.

An interesting note on this is that it’s not even a bug =) Since $in is a
scalar reference, there is no need to explicitly close() the filehandle since
the reference counter will close it on leaving scope, but there’s no harm in
doing it ourselves and it also makes for less confusion for anyone not familiar
with Perl internals.

> - 0010: commment for load_maptables is wrong.

There is also a fix for a typo in make_mapchecker.pl

> - 0011: hash reference is incorrectly dereferenced
> 
> All other fixes other than the above three seem to be styling or
> syntax-generation issues and I don't know whether any
> recommendation exists…

I think there are some more fixes that arent styling/syntax remaining.  I’ll go
through the patches one by one:

0007 - While this might be considered styling/syntax, my $0.02 is that it’s not
and instead a worthwhile change.  I’ll illustrate with an example from the
patch in question:

Using a bareword global variable in open() for the filehandle was replaced with
the three-part form in 5.6 and is now even actively discouraged from in the
Perl documentation (and has been so since the 5.20 docs).  The problem is that
they are global and can thus easily clash, so easily that the 0007 patch
actually fixes one such occurrence:

print_radix_map() opens the file in the global filehandle OUT and passes it to
print_radix_table() with the typeglob *OUT; print_radit_table() in turn passes
the filehandle to print_segmented_table() which writes to the file using the
parameter $hd, except in one case where it uses the global OUT variable without
knowing it will be the right file.  This is where the hunk below in 0007 comes
in:

-               print OUT "$line\n";
+               print { $$hd } "$line\n";

In this case OUT references the right file and it produces the right result,
but it illustrates how easy it is to get wrong (which can cause very subtle
bugs).  So, when poking at this code we might as well, IMHO, use what is today
in Perl considered the right way to deal with filehandle references.

Using implicit filemodes can also introduce bugs when opening filenames passed
in from the outside as we do in UCS_to_most.pl.  Considering the use case of
these scripts it’s obviously quite low on the list of risks but still.

0008 - I don’t think there are any recommendations whether or not to use use
strict; in the codebase, there certainly are lots of scripts not doing it.
Personally I think it’s good hygiene to always use strict but here it might
just be janitorial nitpicking (which I too am guilty of liking..  =)).

0009 - local $var; is to provide a temporary value of $var, where $var exists,
for the current scope (and was mostly used back in Perl 4).  Since we are
passing by value to ucs2utf(), and creating $utf inside it, using my to create
the variable is the right thing even though the end result is the same.

0010 and 0011 are already dealt with above.

So to summarize, I think there are a few more (while not all) hunks that are of
interest which aren’t just syntax/style which can serve to make the code easer
to read/work with down line should we need to.

cheers ./daniel

  35. Re: Radix tree for character conversion

    Daniel Gustafsson <daniel@yesql.se> — 2016-11-07T16:19:29Z

    > On 07 Nov 2016, at 12:32, Daniel Gustafsson <daniel@yesql.se> wrote:
> 
>> On 04 Nov 2016, at 08:34, Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> wrote:
>> 
>> I'm not sure how the discussion about this goes, these patches
>> makes me think about coding style of Perl.
> 
> Some of this can absolutely be considered style and more or less down to
> personal preference.  I haven’t seen any coding conventions for Perl so I
> assume it’s down to consensus among the committers.

Actually, scratch that; there is of course a perltidy profile in the pgindent
directory.  I should avoid sending email before coffee..

cheers ./daniel

  36. Re: Radix tree for character conversion

    Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> — 2016-11-08T01:43:56Z

    Hello,

At Mon, 7 Nov 2016 12:32:55 +0100, Daniel Gustafsson <daniel@yesql.se> wrote in <EE8775B6-BE30-459D-9DDB-F3D0B3FF573D@yesql.se>
> > On 04 Nov 2016, at 08:34, Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> wrote:
> > I'm not sure how the discussion about this goes, these patches
> > makes me think about coding style of Perl.
> 
> Some of this can absolutely be considered style and more or less down to
> personal preference.  I haven’t seen any coding conventions for Perl so I
> assume it’s down to consensus among the committers.  My rationale for these
> patches in the first place was that I perceived this thread to partly want to
> clean up the code and make it more modern Perl.
> 
> > The distinction between executable script and library is by
> > intention with an obscure basis. Existing scripts don't get less
> > modification, but library uses more restricted scopes to get rid
> > of the troubles caused by using global scopes. But I don't have a
> > clear preference on that. The TAP test scripts takes OO notations
> > but I'm not sure convutils.pl also be better to take the same
> > notation. It would be rarely edited hereafter and won't gets
> > grown any more.
> 
> I think the current convutils module is fine and converting it to OO would be
> overkill.

Agreed.

> > As far as I see the obvious bug fixes in the patchset are the
> > following,
> 
> Agreed, with some comments:
> 
> > - 0007: load_maptable fogets to close input file.
> 
> An interesting note on this is that it’s not even a bug =) Since $in is a
> scalar reference, there is no need to explicitly close() the filehandle since
> the reference counter will close it on leaving scope, but there’s no harm in
> doing it ourselves and it also makes for less confusion for anyone not familiar
> with Perl internals.

Wow. I didn't know that perl has such a hidden-OO
feature. Nevertheless, implicit close is not friendly to who are
not familiar with newer perl.

Your comment led me to confirm the requirement to build PostgreSQL.

https://www.postgresql.org/docs/devel/static/install-requirements.html

| Perl 5.8 or later is needed to build from a Git checkout, or if
| you changed the input files for any of the build steps that use
| Perl scripts. If building on Windows you will need Perl in any
| case. Perl is also required to run some test suites.

So, we should assume Perl 5.8 (released in 2002!) on build
time. And actually 5.10 on RedHat 6.4, 5.16 on my
environment(ContOS 7.2), and the official doc is at 5.24. Active
perl is 5.24. According to this, we should use syntax supported
as of 5.8 and/but not obsolete until 5.24, then to follow the
latest convention. But not OO. (But I can't squeeze out a
concrete syntax set out of this requirements :( )


> > - 0010: commment for load_maptables is wrong.
> 
> There is also a fix for a typo in make_mapchecker.pl
> 
> > - 0011: hash reference is incorrectly dereferenced
> > 
> > All other fixes other than the above three seem to be styling or
> > syntax-generation issues and I don't know whether any
> > recommendation exists…
> 
> I think there are some more fixes that arent styling/syntax remaining.  I’ll go
> through the patches one by one:
> 
> 0007 - While this might be considered styling/syntax, my $0.02 is that it’s not
> and instead a worthwhile change.  I’ll illustrate with an example from the
> patch in question:
> 
> Using a bareword global variable in open() for the filehandle was replaced with
> the three-part form in 5.6 and is now even actively discouraged from in the
> Perl documentation (and has been so since the 5.20 docs).  The problem is that
> they are global and can thus easily clash, so easily that the 0007 patch
> actually fixes one such occurrence:

That's what should be adopted in the criteria above.

  - Don't use bareword globals.
  - Use open() with separate MODE argument.

> print_radix_map() opens the file in the global filehandle OUT and passes it to
> print_radix_table() with the typeglob *OUT; print_radit_table() in turn passes
> the filehandle to print_segmented_table() which writes to the file using the
> parameter $hd, except in one case where it uses the global OUT variable without
> knowing it will be the right file.  This is where the hunk below in 0007 comes
> in:
> 
> -               print OUT "$line\n";
> +               print { $$hd } "$line\n";
> 
> In this case OUT references the right file and it produces the right result,
> but it illustrates how easy it is to get wrong (which can cause very subtle
> bugs).  So, when poking at this code we might as well, IMHO, use what is today
> in Perl considered the right way to deal with filehandle references.

Thanks for the detail. Ok, I'll change the style so in the next
patch.

> Using implicit filemodes can also introduce bugs when opening filenames passed
> in from the outside as we do in UCS_to_most.pl.  Considering the use case of
> these scripts it’s obviously quite low on the list of risks but still.

Ok, I'll do that.

> 0008 - I don’t think there are any recommendations whether or not to use use
> strict; in the codebase, there certainly are lots of scripts not doing it.
> Personally I think it’s good hygiene to always use strict but here it might
> just be janitorial nitpicking (which I too am guilty of liking..  =)).

I used strict as an amulet (or armor) not to type incorrect
symbols. Breaking well-working scripts by adding strict is not
reasonable but using it in new or heavily rewritten scripts are
reasonable. I changed my mind to use newer style in rewriting
existing scripts.

> 0009 - local $var; is to provide a temporary value of $var, where $var exists,
> for the current scope (and was mostly used back in Perl 4).  Since we are
> passing by value to ucs2utf(), and creating $utf inside it, using my to create
> the variable is the right thing even though the end result is the same.

Yes, you're right. The point was that the deferrence between
lexical and dynamic scopes doesn't make any difference in the
context. But I'll rewrite it according to the new policy.

> 0010 and 0011 are already dealt with above.
> 
> So to summarize, I think there are a few more (while not all) hunks that are of
> interest which aren’t just syntax/style which can serve to make the code easer
> to read/work with down line should we need to.

Addition to this, I'll remove existing authority files and modify
radix generator so that it can read plain map files in the next
patch.

Thank you for the significant comments.

regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

  37. Re: Radix tree for character conversion

    Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> — 2016-11-08T02:02:58Z

    Hello,

At Mon, 7 Nov 2016 17:19:29 +0100, Daniel Gustafsson <daniel@yesql.se> wrote in <39E295B9-7391-40B6-911D-FE852E4604BD@yesql.se>
> > On 07 Nov 2016, at 12:32, Daniel Gustafsson <daniel@yesql.se> wrote:
> > 
> >> On 04 Nov 2016, at 08:34, Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> wrote:
> >> 
> >> I'm not sure how the discussion about this goes, these patches
> >> makes me think about coding style of Perl.
> > 
> > Some of this can absolutely be considered style and more or less down to
> > personal preference.  I haven’t seen any coding conventions for Perl so I
> > assume it’s down to consensus among the committers.
> 
> Actually, scratch that; there is of course a perltidy profile in the pgindent
> directory.  I should avoid sending email before coffee..

Hmm.  Somehow perl-mode on my Emacs is stirring with
ununderstandable indentation and I manually correct them so it is
highly probable that the style of this patch is not compatible
with the defined style. Anyway it is better that pgindent
generates smaller patch so I'll try it.

regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

  38. Re: Radix tree for character conversion

    Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> — 2016-11-08T11:21:22Z

    Hello, this is the revising patch applies on top of the previous
patch.

Differences on map files are enormous but useless for discussion
so they aren't included in this. (but can be generated)

This still doesn't remove three .txt/.xml files since it heavily
bloats the patch. I'm planning that they are removed in the final
shape. All authority files including the removed files are
automatically downloaded by the Makefile in this patch.

At Tue, 08 Nov 2016 10:43:56 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> wrote in <20161108.104356.265607041.horiguchi.kyotaro@lab.ntt.co.jp>
> https://www.postgresql.org/docs/devel/static/install-requirements.html
> 
> | Perl 5.8 or later is needed to build from a Git checkout, or if
> | you changed the input files for any of the build steps that use
> | Perl scripts. If building on Windows you will need Perl in any
> | case. Perl is also required to run some test suites.
> 
> So, we should assume Perl 5.8 (released in 2002!) on build
> time. And actually 5.10 on RedHat 6.4, 5.16 on my
> environment(ContOS 7.2), and the official doc is at 5.24. Active
> perl is 5.24. According to this, we should use syntax supported
> as of 5.8 and/but not obsolete until 5.24, then to follow the
> latest convention. But not OO. (But I can't squeeze out a
> concrete syntax set out of this requirements :( )
...(forget this for a while..)

Finally the attached patch contains most of (virtually all of)
Daniel's suggestion and some modification by pgperltidy.

> Addition to this, I'll remove existing authority files and modify
> radix generator so that it can read plain map files in the next
> patch.

So, I think the attached are in rather modern shape.

At Tue, 08 Nov 2016 11:02:58 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> wrote in <20161108.110258.59832499.horiguchi.kyotaro@lab.ntt.co.jp>
> Hmm.  Somehow perl-mode on my Emacs is stirring with
> ununderstandable indentation and I manually correct them so it is
> highly probable that the style of this patch is not compatible
> with the defined style. Anyway it is better that pgindent
> generates smaller patch so I'll try it.

The attached are applied pgperltidy. Several regions such like
additional character list are marked not to be edited.

One concern is what to leave by 'make distclen' and 'make
maintainer-clean'. The former should remove authority *.TXT files
since it shouldn't be in source archive. On the other hand it is
more convenient that the latter leaves them. This seems somewhat
strange but I can't come up with better behavior for now.

regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

  39. Re: Radix tree for character conversion

    Daniel Gustafsson <daniel@yesql.se> — 2016-11-08T13:06:29Z

    > On 08 Nov 2016, at 12:21, Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> wrote:
> 
> Hello, this is the revising patch applies on top of the previous
> patch.
> 
> ...
> 
> Finally the attached patch contains most of (virtually all of)
> Daniel's suggestion and some modification by pgperltidy.

Reading over this it looks good to me.  I did spot one thing I had missed
before though, the error message below should be referencing the scalar
variable ‘direction' unless I’m missing something:

-	die "unacceptable direction : %direction"
+	die "unacceptable direction : $direction"
 	  if ($direction ne "to_unicode" && $direction ne "from_unicode");

With this, I would consider this ready for committer.

>> Addition to this, I'll remove existing authority files and modify
>> radix generator so that it can read plain map files in the next
>> patch.
> 
> So, I think the attached are in rather modern shape.

+1, nice work!

cheers ./daniel

  40. Re: Radix tree for character conversion

    Peter Eisentraut <peter.eisentraut@2ndquadrant.com> — 2016-11-08T16:37:58Z

    On 10/31/16 12:11 PM, Daniel Gustafsson wrote:
> I took a small stab at doing some cleaning of the Perl scripts, mainly around
> using the more modern (well, modern as in +15 years old) form for open(..),
> avoiding global filehandles for passing scalar references and enforcing use
> strict.  Some smaller typos and fixes were also included.  It seems my Perl has
> become a bit rusty so I hope the changes make sense.  The produced files are
> identical with these patches applied, they are merely doing cleaning as opposed
> to bugfixing.
> 
> The attached patches are against the 0001-0006 patches from Heikki and you in
> this series of emails, the separation is intended to make them easier to read.

Cool.  See also here:
https://www.postgresql.org/message-id/55E52225.4040305%40gmx.net

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  41. Re: Radix tree for character conversion

    Daniel Gustafsson <daniel@yesql.se> — 2016-11-09T01:19:01Z

    > On 08 Nov 2016, at 17:37, Peter Eisentraut <peter.eisentraut@2ndquadrant.com> wrote:
> 
> On 10/31/16 12:11 PM, Daniel Gustafsson wrote:
>> I took a small stab at doing some cleaning of the Perl scripts, mainly around
>> using the more modern (well, modern as in +15 years old) form for open(..),
>> avoiding global filehandles for passing scalar references and enforcing use
>> strict.  Some smaller typos and fixes were also included.  It seems my Perl has
>> become a bit rusty so I hope the changes make sense.  The produced files are
>> identical with these patches applied, they are merely doing cleaning as opposed
>> to bugfixing.
>> 
>> The attached patches are against the 0001-0006 patches from Heikki and you in
>> this series of emails, the separation is intended to make them easier to read.
> 
> Cool.  See also here:
> https://www.postgresql.org/message-id/55E52225.4040305%40gmx.net

Nice, not having hacked much Perl in quite a while I had all but forgotten
about perlcritic.

Running it on the current version of the patchset yields mostly warnings on
string values used in the require “convutils.pm” statement.  There were however
two more interesting reports: one more open() call not using the three
parameter form and an instance of map which alters the input value.  The latter
is not causing an issue since we don’t use the input list past the map but
fixing it seems like good form.

Attached is a patch that addresses the perlcritic reports (running without any
special options).

cheers ./daniel

  42. Re: Radix tree for character conversion

    Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> — 2016-11-09T08:38:53Z

    Hello, thank you for polishing this.

At Wed, 9 Nov 2016 02:19:01 +0100, Daniel Gustafsson <daniel@yesql.se> wrote in <80F34F25-BF6D-4BCD-9C38-42ED10D3F453@yesql.se>
> > On 08 Nov 2016, at 17:37, Peter Eisentraut <peter.eisentraut@2ndquadrant.com> wrote:
> > 
> > On 10/31/16 12:11 PM, Daniel Gustafsson wrote:
> >> I took a small stab at doing some cleaning of the Perl scripts, mainly around
> >> using the more modern (well, modern as in +15 years old) form for open(..),
> >> avoiding global filehandles for passing scalar references and enforcing use
> >> strict.  Some smaller typos and fixes were also included.  It seems my Perl has
> >> become a bit rusty so I hope the changes make sense.  The produced files are
> >> identical with these patches applied, they are merely doing cleaning as opposed
> >> to bugfixing.
> >> 
> >> The attached patches are against the 0001-0006 patches from Heikki and you in
> >> this series of emails, the separation is intended to make them easier to read.
> > 
> > Cool.  See also here:
> > https://www.postgresql.org/message-id/55E52225.4040305%40gmx.net

> Nice, not having hacked much Perl in quite a while I had all but forgotten
> about perlcritic.

I tried it on CentOS7. Installation failed saying that
Module::Build is too old. It is yum-inatlled so removed it and
installed it with CPAN. Again failed with many 'Could not create
MYMETA files'. Then tried to install CPAN::Meta and it failed
saying that CPAN::Meta::YAML is too *new*. That sucks.

So your patch is greately helpfull. Thank you.

| -my @mapnames = map { s/\.map//; $_ } values %plainmaps;
| +my @mapnames = map { my $m = $_; $m =~ s/\.map//; $m } values %plainmaps;

It surprised me to know that perlcritic does such things.

> Running it on the current version of the patchset yields mostly warnings on
> string values used in the require “convutils.pm” statement.  There were however
> two more interesting reports: one more open() call not using the three
> parameter form and an instance of map which alters the input value. 

Sorry for overlooking it.

> The latter
> is not causing an issue since we don’t use the input list past the map but
> fixing it seems like good form.

Agreed.

> Attached is a patch that addresses the perlcritic reports (running without any
> special options).

Thanks. The attached patch contains the patch by perlcritic.

0001,2,3 are Heikki's patch that are not modified since it is
first proposed. It's a bit too big so I don't attach them to this
mail (again).

https://www.postgresql.org/message-id/08e7892a-d55c-eefe-76e6-7910bc8dd1f3@iki.fi

0004 is radix-tree stuff, applies on top of the three patches
above.

There's a hidden fifth patch which of 20MB in size. But it is
generated by running make in the Unicode directory.

[$(TOP)]$ ./configure ...
[$(TOP)]$ make
[Unicode]$ make
[Unicode]$ make distclean
[Unicode]$ git add .
[Unicode]$ commit 
=== COMMITE MESSSAGE
Replace map files with radix tree files.

These encodings no longer uses the former map files and uses new radix
tree files. All existing authority files in this directory are removed.
===

regards,

  43. Re: Radix tree for character conversion

    Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> — 2016-11-28T10:25:15Z

    Hello.

I'll be off line until at least next Monday. So I move this to
the next CF by myself.

At Wed, 09 Nov 2016 17:38:53 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> wrote in <20161109.173853.77274443.horiguchi.kyotaro@lab.ntt.co.jp>
> Hello, thank you for polishing this.
> 
> At Wed, 9 Nov 2016 02:19:01 +0100, Daniel Gustafsson <daniel@yesql.se> wrote in <80F34F25-BF6D-4BCD-9C38-42ED10D3F453@yesql.se>
> > > On 08 Nov 2016, at 17:37, Peter Eisentraut <peter.eisentraut@2ndquadrant.com> wrote:
> > > 
> > > On 10/31/16 12:11 PM, Daniel Gustafsson wrote:
> > >> I took a small stab at doing some cleaning of the Perl scripts, mainly around
> > >> using the more modern (well, modern as in +15 years old) form for open(..),
> > >> avoiding global filehandles for passing scalar references and enforcing use
> > >> strict.  Some smaller typos and fixes were also included.  It seems my Perl has
> > >> become a bit rusty so I hope the changes make sense.  The produced files are
> > >> identical with these patches applied, they are merely doing cleaning as opposed
> > >> to bugfixing.
> > >> 
> > >> The attached patches are against the 0001-0006 patches from Heikki and you in
> > >> this series of emails, the separation is intended to make them easier to read.
> > > 
> > > Cool.  See also here:
> > > https://www.postgresql.org/message-id/55E52225.4040305%40gmx.net
> 
> > Nice, not having hacked much Perl in quite a while I had all but forgotten
> > about perlcritic.
> 
> I tried it on CentOS7. Installation failed saying that
> Module::Build is too old. It is yum-inatlled so removed it and
> installed it with CPAN. Again failed with many 'Could not create
> MYMETA files'. Then tried to install CPAN::Meta and it failed
> saying that CPAN::Meta::YAML is too *new*. That sucks.
> 
> So your patch is greately helpfull. Thank you.
> 
> | -my @mapnames = map { s/\.map//; $_ } values %plainmaps;
> | +my @mapnames = map { my $m = $_; $m =~ s/\.map//; $m } values %plainmaps;
> 
> It surprised me to know that perlcritic does such things.
> 
> > Running it on the current version of the patchset yields mostly warnings on
> > string values used in the require “convutils.pm” statement.  There were however
> > two more interesting reports: one more open() call not using the three
> > parameter form and an instance of map which alters the input value. 
> 
> Sorry for overlooking it.
> 
> > The latter
> > is not causing an issue since we don’t use the input list past the map but
> > fixing it seems like good form.
> 
> Agreed.
> 
> > Attached is a patch that addresses the perlcritic reports (running without any
> > special options).
> 
> Thanks. The attached patch contains the patch by perlcritic.
> 
> 0001,2,3 are Heikki's patch that are not modified since it is
> first proposed. It's a bit too big so I don't attach them to this
> mail (again).
> 
> https://www.postgresql.org/message-id/08e7892a-d55c-eefe-76e6-7910bc8dd1f3@iki.fi
> 
> 0004 is radix-tree stuff, applies on top of the three patches
> above.
> 
> There's a hidden fifth patch which of 20MB in size. But it is
> generated by running make in the Unicode directory.
> 
> [$(TOP)]$ ./configure ...
> [$(TOP)]$ make
> [Unicode]$ make
> [Unicode]$ make distclean
> [Unicode]$ git add .
> [Unicode]$ commit 
> === COMMITE MESSSAGE
> Replace map files with radix tree files.
> 
> These encodings no longer uses the former map files and uses new radix
> tree files. All existing authority files in this directory are removed.
> ===

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

  44. Re: Radix tree for character conversion

    Heikki Linnakangas <hlinnaka@iki.fi> — 2016-11-30T12:59:44Z

    On 10/31/2016 06:11 PM, Daniel Gustafsson wrote:
>> On 27 Oct 2016, at 09:23, Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> wrote:
>>
>> At Tue, 25 Oct 2016 12:23:48 +0300, Heikki Linnakangas <hlinnaka@iki.fi> wrote in <08e7892a-d55c-eefe-76e6-7910bc8dd1f3@iki.fi>
>>>
>>> [..]
>>> The perl scripts are still quite messy. For example, I lost the checks
>>> for duplicate mappings somewhere along the way - that ought to be put
>>> back. My Perl skills are limited.
>>
>> Perl scripts are to be messy, I believe. Anyway the duplicate
>> check as been built into the sub print_radix_trees. Maybe the
>> same check is needed by some plain map files but it would be just
>> duplication for the maps having radix tree.
>
> I took a small stab at doing some cleaning of the Perl scripts, mainly around
> using the more modern (well, modern as in +15 years old) form for open(..),
> avoiding global filehandles for passing scalar references and enforcing use
> strict.  Some smaller typos and fixes were also included.  It seems my Perl has
> become a bit rusty so I hope the changes make sense.  The produced files are
> identical with these patches applied, they are merely doing cleaning as opposed
> to bugfixing.
>
> The attached patches are against the 0001-0006 patches from Heikki and you in
> this series of emails, the separation is intended to make them easier to read.

Thanks! Patches 0001-0003 seem to have been mostly unchanged for the 
later discussion and everyone seems to be happy with those patches, so I 
picked the parts of these cleanups of yours that applied to my patches 
0001-0003, and pushed those. I'll continue reviewing the rest..

- Heikki

  45. Re: Radix tree for character conversion

    Heikki Linnakangas <hlinnaka@iki.fi> — 2016-12-02T20:07:07Z

    On 11/09/2016 10:38 AM, Kyotaro HORIGUCHI wrote:
> Thanks. The attached patch contains the patch by perlcritic.
>
> 0001,2,3 are Heikki's patch that are not modified since it is
> first proposed. It's a bit too big so I don't attach them to this
> mail (again).
>
> https://www.postgresql.org/message-id/08e7892a-d55c-eefe-76e6-7910bc8dd1f3@iki.fi

I've now pushed these preliminary patches, with the applicable fixes 
from you and Daniel. The attached patch is now against git master.

> 0004 is radix-tree stuff, applies on top of the three patches
> above.

I've spent the last couple of days reviewing this. While trying to 
understand how it works, I ended up dismantling, rewriting, and putting 
back together most of the added perl code. Attached is a new version, 
with more straightforward logic, making it more understandable. I find 
it more understandable, anyway, I hope it's not only because I wrote it 
myself :-). Let me know what you think.

In particular, I found the explanations of flat and segmented tables 
really hard to understand. So in this version, the radix trees for a 
conversion are stored completely in one large array. Leaf and 
intermediate levels are all in the same array. When reading this 
version, please note that I'm not sure if I mean the same thing with 
"segment" that you did in your version.

I moved the "lower" and "upper" values in the structs. Also, there are 
now also separate "lower" and "upper" values for the leaf levels of the 
trees, for 1- 2-, 3- and 4-byte inputs. This made a huge difference to 
the size of gb18030_to_utf8_radix.map, in particular: the source file 
shrank from about 2 MB to 1/2 MB. In that conversion, the valid range 
for the last byte of 2-byte inputs is 0x40-0xfe, and the valid range for 
the last byte of 4-byte inputs is 0x30-0x39. With the old patch version, 
the "chars" range was therefore 0x30-0xfe, to cover both of those, and 
most of the array was filled with zeros. With this new patch version, we 
store separate ranges for those, and can leave out most of the zeros.

There's a segment full of zeros at the beginning of each conversion 
array now. The purpose of that is that when traversing the radix tree, 
you don't need to check each intermediate value for 0. If you follow a 0 
offset, it simply points to the dummy all-zeros segments in the 
beginning. Seemed like a good idea to shave some cycles, although I'm 
not sure if it made much difference in reality.

I optimized pg_mb_radix_conv() a bit, too. We could do more. For 
example, I think it would save some cycles to have specialized versions 
of UtfToLocal and LocalToUtf, moving the tests for whether a combined 
character map and/or conversion callback is used, out of the loop. They 
feel a bit ugly too, in their current form...

I need a break now, but I'll try to pick this up again some time next 
week. Meanwhile, please have a look and tell me what you think.

- Heikki

  46. Re: Radix tree for character conversion

    Alvaro Herrera <alvherre@2ndquadrant.com> — 2016-12-02T20:18:46Z

    Heikki Linnakangas wrote:
> On 11/09/2016 10:38 AM, Kyotaro HORIGUCHI wrote:
> > Thanks. The attached patch contains the patch by perlcritic.
> > 
> > 0001,2,3 are Heikki's patch that are not modified since it is
> > first proposed. It's a bit too big so I don't attach them to this
> > mail (again).
> > 
> > https://www.postgresql.org/message-id/08e7892a-d55c-eefe-76e6-7910bc8dd1f3@iki.fi
> 
> I've now pushed these preliminary patches, with the applicable fixes from
> you and Daniel. The attached patch is now against git master.

Is this the Nov. 30th commit?  Because I don't see any other commits
from you.

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  47. Re: Radix tree for character conversion

    Heikki Linnakangas <hlinnaka@iki.fi> — 2016-12-02T20:49:52Z

    On 12/02/2016 10:18 PM, Alvaro Herrera wrote:
> Heikki Linnakangas wrote:
>> On 11/09/2016 10:38 AM, Kyotaro HORIGUCHI wrote:
>>> Thanks. The attached patch contains the patch by perlcritic.
>>>
>>> 0001,2,3 are Heikki's patch that are not modified since it is
>>> first proposed. It's a bit too big so I don't attach them to this
>>> mail (again).
>>>
>>> https://www.postgresql.org/message-id/08e7892a-d55c-eefe-76e6-7910bc8dd1f3@iki.fi
>>
>> I've now pushed these preliminary patches, with the applicable fixes from
>> you and Daniel. The attached patch is now against git master.
>
> Is this the Nov. 30th commit?  Because I don't see any other commits
> from you.

Yes. Sorry for the confusion.

- Heikki

  48. Re: Radix tree for character conversion

    Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> — 2016-12-05T10:29:54Z

    Hello, thank you for reviewing this.

I compared mine and yours. The new patch works fine and gives
smaller radix map files. It seems also to me more readable.

At Fri, 2 Dec 2016 22:07:07 +0200, Heikki Linnakangas <hlinnaka@iki.fi> wrote in <da58a154-0b28-802e-5e82-5a205f53926e@iki.fi>
> On 11/09/2016 10:38 AM, Kyotaro HORIGUCHI wrote:
> > Thanks. The attached patch contains the patch by perlcritic.
> >
> > 0001,2,3 are Heikki's patch that are not modified since it is
> > first proposed. It's a bit too big so I don't attach them to this
> > mail (again).
> >
> > https://www.postgresql.org/message-id/08e7892a-d55c-eefe-76e6-7910bc8dd1f3@iki.fi
> 
> I've now pushed these preliminary patches, with the applicable fixes
> from you and Daniel. The attached patch is now against git master.

Thanks for committing them.

> > 0004 is radix-tree stuff, applies on top of the three patches
> > above.
> 
> I've spent the last couple of days reviewing this. While trying to
> understand how it works, I ended up dismantling, rewriting, and
> putting back together most of the added perl code. 

I might have been putting too much in one structure and a bit too
eager to conceal lower level from the upper level.

> Attached is a new
> version, with more straightforward logic, making it more
> understandable. I find it more understandable, anyway, I hope it's not
> only because I wrote it myself :-). Let me know what you think.

First, thank you for the refactoring(?).

I didn't intend to replace all of .map files with radix files at
first. Finally my patch removes all old-style map file but I
haven't noticed that. So removing the old bsearch code seems
reasonable. Avoiding redundant decomposition of multibyte
characters into bytes seems reasonable from the view of
efficiency.

The new patch decomposes the structured pg_mb_radix_tree into a
series of (basically) plain member variables in a struct. I'm not
so in favor of the style but a radix tree of at least 4-levels is
easily read in the style and maybe the code to handle it is
rather easily readable. (So +1 for it)

> In particular, I found the explanations of flat and segmented tables
> really hard to understand. So in this version, the radix trees for a
> conversion are stored completely in one large array. Leaf and
> intermediate levels are all in the same array. When reading this
> version, please note that I'm not sure if I mean the same thing with
> "segment" that you did in your version.
> I moved the "lower" and "upper" values in the structs. Also, there are
> now also separate "lower" and "upper" values for the leaf levels of
> the trees, for 1- 2-, 3- and 4-byte inputs. This made a huge

The "segment" there seems to mean definitely the same to
mine. Flattening the on-memory structure is fine from the same
reason to the above.

> difference to the size of gb18030_to_utf8_radix.map, in particular:
> the source file shrank from about 2 MB to 1/2 MB. In that conversion,
> the valid range for the last byte of 2-byte inputs is 0x40-0xfe, and
> the valid range for the last byte of 4-byte inputs is 0x30-0x39. With
> the old patch version, the "chars" range was therefore 0x30-0xfe, to
> cover both of those, and most of the array was filled with zeros. With
> this new patch version, we store separate ranges for those, and can
> leave out most of the zeros.

Great. I agree that the (logically) devided chartable is
significantly space-efficient.

> There's a segment full of zeros at the beginning of each conversion
> array now. The purpose of that is that when traversing the radix tree,
> you don't need to check each intermediate value for 0. If you follow a
> 0 offset, it simply points to the dummy all-zeros segments in the
> beginning. Seemed like a good idea to shave some cycles, although I'm
> not sure if it made much difference in reality.

And I like the zero page.

> I optimized pg_mb_radix_conv() a bit, too. We could do more. For
> example, I think it would save some cycles to have specialized
> versions of UtfToLocal and LocalToUtf, moving the tests for whether a
> combined character map and/or conversion callback is used, out of the
> loop. They feel a bit ugly too, in their current form...

Hmm. Maybe decomposing iiso in pg_mb_radix_conv is faster than
pushing extra 3 (or 4) parameters into the stack (or it's wrong
if they are passed using registers?) but I'm not sure.

> I need a break now, but I'll try to pick this up again some time next
> week. Meanwhile, please have a look and tell me what you think.

Thank you very much for the big effort on this patch.

Apart from the aboves, I have some trivial comments on the new
version.


1. If we decide not to use old-style maps, UtfToLocal no longer
  need to take void * as map data. (Patch 0001)

2. "use Data::Dumper" doesn't seem necessary. (Patch 0002)

3. A comment contains a superfluous comma. (Patch 0002) (The last
   byte of the first line below)
 > ### The segments are written out physically to one big array in the final,
 > ### step, but logically, they form a radix tree. Or rather, four radix

4. The following code doesn't seem so perl'ish.

   >  for (my $i=0; $i <= 0xff; $i++)
   >  {
   >    my $val = $seg->{values}->{$i};
   >    if ($val)
   >    {
   >      $this_min = $i if (!defined $this_min || $i < $this_min);

   Refraining from proposing extreme perl, the following would be
   reasonable as an equivalent. (Patch 0002)

   foreach $i (keys $seg->{values})
   {
      
   The 0002 patch contains the following change but this might be
   kinda extreme..

   -    for (my $i=0; $i <= 0xff; $i++)
   +    while ((my $i, my $val) = each $map)

4. download_srctxts.sh is no longer needed. (No patch)


I'll put more consideration on the new version and put another
version later.

regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

  49. Re: Radix tree for character conversion

    Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> — 2016-12-13T06:11:03Z

    Hello, I looked on this closer.

The attached is the revised version of this patch.

At Mon, 05 Dec 2016 19:29:54 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> wrote in <20161205.192954.121855559.horiguchi.kyotaro@lab.ntt.co.jp>
> Apart from the aboves, I have some trivial comments on the new
> version.
> 
> 
> 1. If we decide not to use old-style maps, UtfToLocal no longer
>   need to take void * as map data. (Patch 0001)
> 2. "use Data::Dumper" doesn't seem necessary. (Patch 0002)
> 3. A comment contains a superfluous comma. (Patch 0002) (The last
>    byte of the first line below)
> 4. The following code doesn't seem so perl'ish.
> 4. download_srctxts.sh is no longer needed. (No patch)

6. Fixed some inconsistent indentation/folding.
7. Fix handling of $verbose.
8. Sort segments using leading bytes.

regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

  50. Re: Radix tree for character conversion

    Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> — 2017-01-10T11:22:23Z

    Hello, I found a bug in my portion while rebasing.

The attached patches apply on top of the current master HEAD, not
on Heikki's previous one. And separated into 4 parts.

At Tue, 13 Dec 2016 15:11:03 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> wrote in <20161213.151103.157484378.horiguchi.kyotaro@lab.ntt.co.jp>
> > Apart from the aboves, I have some trivial comments on the new
> > version.
> > 
> > 
> > 1. If we decide not to use old-style maps, UtfToLocal no longer
> >   need to take void * as map data. (Patch 0001)

I changed the pointer type wrongly. Combined maps are of the type
*_combined.

> > 2. "use Data::Dumper" doesn't seem necessary. (Patch 0002)
> > 3. A comment contains a superfluous comma. (Patch 0002) (The last
> >    byte of the first line below)
> > 4. The following code doesn't seem so perl'ish.
> > 5. download_srctxts.sh is no longer needed. (No patch)
> 
> 6. Fixed some inconsistent indentation/folding.
> 7. Fix handling of $verbose.
> 8. Sort segments using leading bytes.

The attached files are the following. This patchset is not
complete missing changes of map files. The change is tremendously
large but generatable.

0001-Add-missing-semicolon.patch

  UCS_to_EUC_JP.pl has a line missing teminating semicolon. This
  doesn't harm but surely a syntax error. This patch fixes it.
  This might should be a separate patch.

0002-Correct-reference-resolution-syntax.patch

  convutils.pm has lines with different syntax of reference
  resolution. This unifies the syntax.

0003-Apply-pgperltidy-on-src-backend-utils-mb-Unicode.patch

  Before adding radix tree stuff, applied pgperltidy and inserted
  format-skipping pragma for the parts where perltidy seems to do
  too much.

0004-Use-radix-tree-for-character-conversion.patch

  Radix tree body.


The unattached fifth patch is generated by the following steps.

[$(TOP)]$ ./configure
[Unicode]$ make
[Unicode]$ make distclean
[Unicode]$ git add .
[Unicode]$ commit 
=== COMMITE MESSSAGE
Replace map files with radix tree files.

These encodings no longer uses the former map files and uses new radix
tree files.
===


regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

  51. Re: Radix tree for character conversion

    Ishii Ayumi <ayumi.ishii.pg@gmail.com> — 2017-01-25T10:18:26Z

    HI,

I patched 4 patchset and run "make", but I got failed.
Is this a bug or my mistake ?
I'm sorry if I'm wrong.

[$(TOP)]$ patch -p1 < ../0001-Add-missing-semicolon.patch
[$(TOP)]$ patch -p1 < ../0002-Correct-reference-resolution-syntax.patch
[$(TOP)]$ patch -p1 <
../0003-Apply-pgperltidy-on-src-backend-utils-mb-Unicode.patch
[$(TOP)]$ patch -p1 < ../0004-Use-radix-tree-for-character-conversion.patch
[$(TOP)]$ ./configure
[Unicode]$ make
'/usr/bin/perl' UCS_to_most.pl
Type of arg 1 to keys must be hash (not hash element) at convutils.pm
line 443, near "})
        "
Type of arg 1 to values must be hash (not hash element) at
convutils.pm line 596, near "})
        "
Type of arg 1 to each must be hash (not private variable) at
convutils.pm line 755, near "$map)
        "
Compilation failed in require at UCS_to_most.pl line 19.
make: *** [iso8859_2_to_utf8.map] Error 255

Regars,
-- 
Ayumi Ishii

  52. Re: Radix tree for character conversion

    Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> — 2017-01-26T07:16:56Z

    Hello, thank you for looking this.

At Wed, 25 Jan 2017 19:18:26 +0900, Ishii Ayumi <ayumi.ishii.pg@gmail.com> wrote in <CAOu5J714+w-TRSNHbsS+aBVE5LdsR3CEZ6w4QLQ=9NrAJNavTA@mail.gmail.com>
> I patched 4 patchset and run "make", but I got failed.
> Is this a bug or my mistake ?
> I'm sorry if I'm wrong.
> 
> [$(TOP)]$ patch -p1 < ../0001-Add-missing-semicolon.patch
> [$(TOP)]$ patch -p1 < ../0002-Correct-reference-resolution-syntax.patch
> [$(TOP)]$ patch -p1 <
> ../0003-Apply-pgperltidy-on-src-backend-utils-mb-Unicode.patch
> [$(TOP)]$ patch -p1 < ../0004-Use-radix-tree-for-character-conversion.patch
> [$(TOP)]$ ./configure
> [Unicode]$ make

The directory src/backend/mb/uilts/Unicode is not built as a part
of the top-level build, and it would be preferable that the
preexisting map files are removed.

$ cd src/backend/utils/mb/Unicode
$ make distclean      # this would require ./configure
$ make maintainer-clean
$ cd ../../../../..   # go to top
$ make clean
(make'ing here will give you an error saying a .map file is not found)
$ cd  src/backend/utils/mb/Unicode  # again
$ make
$ cd ../../../../..   # go to top
$ make

This steps still suceeds for me, even with the patches on the
current master.

The cause of the the following errors seems the other things.

> '/usr/bin/perl' UCS_to_most.pl
> Type of arg 1 to keys must be hash (not hash element) at convutils.pm
> line 443, near "})
>         "
> Type of arg 1 to values must be hash (not hash element) at
> convutils.pm line 596, near "})
>         "
> Type of arg 1 to each must be hash (not private variable) at
> convutils.pm line 755, near "$map)
>         "
> Compilation failed in require at UCS_to_most.pl line 19.
> make: *** [iso8859_2_to_utf8.map] Error 255

Surely perl 5.8.9 complained just as above but 5.16
doesn't. Google told me that at least 5.10 behaves as the same
way. The *current* requirement for perl verion to build is 5.8.

https://www.postgresql.org/docs/current/static/install-requirements.html

Fortunately, only three lines of change suffices 5.8 so I've
chosen to flatter perl 5.8.

As the result, no changes has been made on 0001-0003 so I
attached only 0004 to this mail.

regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

  53. Re: Radix tree for character conversion

    Michael Paquier <michael.paquier@gmail.com> — 2017-01-26T07:28:16Z

    On Tue, Jan 10, 2017 at 8:22 PM, Kyotaro HORIGUCHI
<horiguchi.kyotaro@lab.ntt.co.jp> wrote:
> [...patch...]

Nobody has showed up yet to review this patch, so I am giving it a shot.

The patch file sizes are scary at first sight, but after having a look:
 36 files changed, 1411 insertions(+), 54398 deletions(-)
Yes that's a surprise, something like git diff --irreversible-delete
would have helped as most of the diffs are just caused by 3 files
being deleted in patch 0004, making 50k lines going to the abyss of
deletion.

> Hello, I found a bug in my portion while rebasing.

Right, that's 0001. Nice catch.

> The attached files are the following. This patchset is not
> complete missing changes of map files. The change is tremendously
> large but generatable.
>
> 0001-Add-missing-semicolon.patch
>
>   UCS_to_EUC_JP.pl has a line missing teminating semicolon. This
>   doesn't harm but surely a syntax error. This patch fixes it.
>   This might should be a separate patch.

This requires a back-patch. This makes me wonder how long this script
has actually not run...

> 0002-Correct-reference-resolution-syntax.patch
>
>   convutils.pm has lines with different syntax of reference
>   resolution. This unifies the syntax.

Yes that looks right to me. I am the best perl guru on this list but
looking around $$var{foo} is bad, ${$var}{foo} is better, and
$var->{foo} is even better. This also generates no diffs when running
make in src/backend/utils/mb/Unicode/. So no objections to that.

> 0003-Apply-pgperltidy-on-src-backend-utils-mb-Unicode.patch
>
>   Before adding radix tree stuff, applied pgperltidy and inserted
>   format-skipping pragma for the parts where perltidy seems to do
>   too much.

Which version of perltidy did you use? Looking at the archives, the
perl code is cleaned up with a specific version, v20090616. See
https://www.postgresql.org/message-id/20151204054322.GA2070309@tornado.leadboat.com
for example on the matter. As perltidy changes over time, this may be
a sensitive change if done this way.

> 0004-Use-radix-tree-for-character-conversion.patch
>
>   Radix tree body.

Well, here a lot of diffs could have been saved.

> The unattached fifth patch is generated by the following steps.
>
> [$(TOP)]$ ./configure
> [Unicode]$ make
> [Unicode]$ make distclean
> [Unicode]$ git add .
> [Unicode]$ commit
> === COMMITE MESSSAGE
> Replace map files with radix tree files.
>
> These encodings no longer uses the former map files and uses new radix
> tree files.
> ===

OK, I can see that working, with 200k of maps generated.. So going
through the important bits of this jungle..

+/*
+ * radix tree conversion function - this should be identical to the function in
+ * ../conv.c with the same name
+ */
+static inline uint32
+pg_mb_radix_conv(const pg_mb_radix_tree *rt,
+                int l,
+                unsigned char b1,
+                unsigned char b2,
+                unsigned char b3,
+                unsigned char b4)
This is not nice. Having a duplication like that is a recipe to forget
about it as this patch introduces a dependency with conv.c and the
radix tree generation.

Having a .gitignore in Unicode/ would be nice, particularly to avoid
committing map_checker.

A README documenting things may be welcome, or at least comments at
the top of map_checker.c. Why is map_checker essential? What does it
do? There is no way to understand that easily, except that it includes
a "radix tree conversion function", and that it performs sanity checks
on the radix trees to be sure that they are on a good shape. But as
this something that one would guess only after looking at your patch
and the code (at least I will sleep less stupid tonight after reading
this stuff).

--- a/src/backend/utils/mb/Unicode/UCS_to_SJIS.pl
+++ b/src/backend/utils/mb/Unicode/UCS_to_SJIS.pl
 # Drop these SJIS codes from the source for UTF8=>SJIS conversion
 #<<< do not let perltidy touch this
-my @reject_sjis =(
+my @reject_sjis = (
    0xed40..0xeefc, 0x8754..0x875d, 0x878a, 0x8782,
-   0x8784, 0xfa5b, 0xfa54, 0x8790..0x8792, 0x8795..0x8797,
+   0x8784, 0xfa5b, 0xfa54, 0x8790..0x8792, 0x8795..0x8797,
    0x879a..0x879c
-);
+   );
This is not generated, it would be nice to drop the noise from the patch.

Here is another one:
-       $i->{code} = $jis | (
-           $jis < 0x100
-           ? 0x8e00
-           : ($sjis >= 0xeffd ? 0x8f8080 : 0x8080));
-
+#<<< do not let perltidy touch this
+       $i->{code} = $jis | ($jis < 0x100 ? 0x8e00:
+                            ($sjis >= 0xeffd ? 0x8f8080 : 0x8080));
+#>>>

        if (l == 2)
        {
-           iutf = *utf++ << 8;
-           iutf |= *utf++;
+           b3 = *utf++;
+           b4 = *utf++;
        }
Ah, OK. This conversion is important so as it performs a minimum of
bitwise operations. Yes let's keep that. That's pretty cool to get a
faster operation.
-- 
Michael

  54. Re: Radix tree for character conversion

    Michael Paquier <michael.paquier@gmail.com> — 2017-01-26T07:29:10Z

    On Wed, Jan 25, 2017 at 7:18 PM, Ishii Ayumi <ayumi.ishii.pg@gmail.com> wrote:
> I patched 4 patchset and run "make", but I got failed.
> Is this a bug or my mistake ?
> I'm sorry if I'm wrong.
>
> [$(TOP)]$ patch -p1 < ../0001-Add-missing-semicolon.patch
> [$(TOP)]$ patch -p1 < ../0002-Correct-reference-resolution-syntax.patch
> [$(TOP)]$ patch -p1 <
> ../0003-Apply-pgperltidy-on-src-backend-utils-mb-Unicode.patch
> [$(TOP)]$ patch -p1 < ../0004-Use-radix-tree-for-character-conversion.patch
> [$(TOP)]$ ./configure
> [Unicode]$ make
> '/usr/bin/perl' UCS_to_most.pl
> Type of arg 1 to keys must be hash (not hash element) at convutils.pm
> line 443, near "})
>         "
> Type of arg 1 to values must be hash (not hash element) at
> convutils.pm line 596, near "})
>         "
> Type of arg 1 to each must be hash (not private variable) at
> convutils.pm line 755, near "$map)
>         "
> Compilation failed in require at UCS_to_most.pl line 19.
> make: *** [iso8859_2_to_utf8.map] Error 255

Hm, I am not sure what you are missing. I was able to get things to build.
-- 
Michael

  55. Re: Radix tree for character conversion

    Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> — 2017-01-26T12:17:37Z

    At Thu, 26 Jan 2017 16:29:10 +0900, Michael Paquier <michael.paquier@gmail.com> wrote in <CAB7nPqQd860tOC17O3Qs3+dzZTYbXrXxVD9Tfph0pJ9LAYZ=Ww@mail.gmail.com>
> On Wed, Jan 25, 2017 at 7:18 PM, Ishii Ayumi <ayumi.ishii.pg@gmail.com> wrote:
> > I patched 4 patchset and run "make", but I got failed.
> > Is this a bug or my mistake ?
> > I'm sorry if I'm wrong.
> >
> > [$(TOP)]$ patch -p1 < ../0001-Add-missing-semicolon.patch
> > [$(TOP)]$ patch -p1 < ../0002-Correct-reference-resolution-syntax.patch
> > [$(TOP)]$ patch -p1 <
> > ../0003-Apply-pgperltidy-on-src-backend-utils-mb-Unicode.patch
> > [$(TOP)]$ patch -p1 < ../0004-Use-radix-tree-for-character-conversion.patch
> > [$(TOP)]$ ./configure
> > [Unicode]$ make
> > '/usr/bin/perl' UCS_to_most.pl
> > Type of arg 1 to keys must be hash (not hash element) at convutils.pm
> > line 443, near "})
> >         "
> > Type of arg 1 to values must be hash (not hash element) at
> > convutils.pm line 596, near "})
> >         "
> > Type of arg 1 to each must be hash (not private variable) at
> > convutils.pm line 755, near "$map)
> >         "
> > Compilation failed in require at UCS_to_most.pl line 19.
> > make: *** [iso8859_2_to_utf8.map] Error 255
> 
> Hm, I am not sure what you are missing. I was able to get things to build.

As I posted, it should be caused by older perl, at least 5.8
complains so and 5.16 doesn't.

regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

  56. Re: Radix tree for character conversion

    Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> — 2017-01-26T12:42:12Z

    Thank you for looking this.

At Thu, 26 Jan 2017 16:28:16 +0900, Michael Paquier <michael.paquier@gmail.com> wrote in <CAB7nPqREL1fsDBGv4zRvaXY+UKtS0wzkamJcnYhX0--OZvpUUQ@mail.gmail.com>
> On Tue, Jan 10, 2017 at 8:22 PM, Kyotaro HORIGUCHI
> <horiguchi.kyotaro@lab.ntt.co.jp> wrote:
> > [...patch...]
> 
> Nobody has showed up yet to review this patch, so I am giving it a shot.
> 
> The patch file sizes are scary at first sight, but after having a look:
>  36 files changed, 1411 insertions(+), 54398 deletions(-)
> Yes that's a surprise, something like git diff --irreversible-delete
> would have helped as most of the diffs are just caused by 3 files
> being deleted in patch 0004, making 50k lines going to the abyss of
> deletion.

Thank you. Good to hear that. I'll try that at the next chance.

> > Hello, I found a bug in my portion while rebasing.
> 
> Right, that's 0001. Nice catch.
> 
> > The attached files are the following. This patchset is not
> > complete missing changes of map files. The change is tremendously
> > large but generatable.
> >
> > 0001-Add-missing-semicolon.patch
> >
> >   UCS_to_EUC_JP.pl has a line missing teminating semicolon. This
> >   doesn't harm but surely a syntax error. This patch fixes it.
> >   This might should be a separate patch.
> 
> This requires a back-patch. This makes me wonder how long this script
> has actually not run...
> 
> > 0002-Correct-reference-resolution-syntax.patch
> >
> >   convutils.pm has lines with different syntax of reference
> >   resolution. This unifies the syntax.
> 
> Yes that looks right to me.

Yes, I thoght that the three patches can be back-patched, a kind
of bug fix.

> I am the best perl guru on this list but
> looking around $$var{foo} is bad, ${$var}{foo} is better, and
> $var->{foo} is even better. This also generates no diffs when running
> make in src/backend/utils/mb/Unicode/. So no objections to that.

Thank you for the explanation. I think no '$$'s is left alone.

> > 0003-Apply-pgperltidy-on-src-backend-utils-mb-Unicode.patch
> >
> >   Before adding radix tree stuff, applied pgperltidy and inserted
> >   format-skipping pragma for the parts where perltidy seems to do
> >   too much.
> 
> Which version of perltidy did you use? Looking at the archives, the
> perl code is cleaned up with a specific version, v20090616. See
> https://www.postgresql.org/message-id/20151204054322.GA2070309@tornado.leadboat.com
> for example on the matter. As perltidy changes over time, this may be
> a sensitive change if done this way.

Hmm. I will make a confirmation on that.. tomorrow.

> > 0004-Use-radix-tree-for-character-conversion.patch
> >
> >   Radix tree body.
> 
> Well, here a lot of diffs could have been saved.
> 
> > The unattached fifth patch is generated by the following steps.
> >
> > [$(TOP)]$ ./configure
> > [Unicode]$ make
> > [Unicode]$ make distclean
> > [Unicode]$ git add .
> > [Unicode]$ commit
> > === COMMITE MESSSAGE
> > Replace map files with radix tree files.
> >
> > These encodings no longer uses the former map files and uses new radix
> > tree files.
> > ===
> 
> OK, I can see that working, with 200k of maps generated.. So going
> through the important bits of this jungle..

Many thaks for the exploration.

> +/*
> + * radix tree conversion function - this should be identical to the function in
> + * ../conv.c with the same name
> + */
> +static inline uint32
> +pg_mb_radix_conv(const pg_mb_radix_tree *rt,
> +                int l,
> +                unsigned char b1,
> +                unsigned char b2,
> +                unsigned char b3,
> +                unsigned char b4)
> This is not nice. Having a duplication like that is a recipe to forget
> about it as this patch introduces a dependency with conv.c and the
> radix tree generation.

Mmmmm. I agree to you, but conv.c contains unwanted reference to
elog or other sutff of the core. Separating the function in a
dedicate source file named such as "../pg_mb_radix_conv.c" will
work. If it is not so bad, I'll do that in the next version.

> Having a .gitignore in Unicode/ would be nice, particularly to avoid
> committing map_checker.
> 
> A README documenting things may be welcome, or at least comments at
> the top of map_checker.c. Why is map_checker essential? What does it
> do? There is no way to understand that easily, except that it includes
> a "radix tree conversion function", and that it performs sanity checks
> on the radix trees to be sure that they are on a good shape. But as
> this something that one would guess only after looking at your patch
> and the code (at least I will sleep less stupid tonight after reading
> this stuff).

Okay, I'll do that.

> --- a/src/backend/utils/mb/Unicode/UCS_to_SJIS.pl
> +++ b/src/backend/utils/mb/Unicode/UCS_to_SJIS.pl
>  # Drop these SJIS codes from the source for UTF8=>SJIS conversion
>  #<<< do not let perltidy touch this
> -my @reject_sjis =(
> +my @reject_sjis = (
>     0xed40..0xeefc, 0x8754..0x875d, 0x878a, 0x8782,
> -   0x8784, 0xfa5b, 0xfa54, 0x8790..0x8792, 0x8795..0x8797,
> +   0x8784, 0xfa5b, 0xfa54, 0x8790..0x8792, 0x8795..0x8797,
>     0x879a..0x879c
> -);
> +   );
> This is not generated, it would be nice to drop the noise from the patch.

Mmm. I'm not sure how this is generated but I'll care for that.

> Here is another one:
> -       $i->{code} = $jis | (
> -           $jis < 0x100
> -           ? 0x8e00
> -           : ($sjis >= 0xeffd ? 0x8f8080 : 0x8080));
> -
> +#<<< do not let perltidy touch this
> +       $i->{code} = $jis | ($jis < 0x100 ? 0x8e00:
> +                            ($sjis >= 0xeffd ? 0x8f8080 : 0x8080));
> +#>>>

Ok. Will revert this.

>         if (l == 2)
>         {
> -           iutf = *utf++ << 8;
> -           iutf |= *utf++;
> +           b3 = *utf++;
> +           b4 = *utf++;
>         }
> Ah, OK. This conversion is important so as it performs a minimum of
> bitwise operations. Yes let's keep that. That's pretty cool to get a
> faster operation.

It is Heikki's work:p

I'll address them and repost the next version sooner.

regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

  57. Re: Radix tree for character conversion

    Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> — 2017-01-27T08:33:57Z

    Hi, this is an intermediate report without a patch.

At Thu, 26 Jan 2017 21:42:12 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> wrote in <20170126.214212.111556326.horiguchi.kyotaro@lab.ntt.co.jp>
> > > 0003-Apply-pgperltidy-on-src-backend-utils-mb-Unicode.patch
> > >
> > >   Before adding radix tree stuff, applied pgperltidy and inserted
> > >   format-skipping pragma for the parts where perltidy seems to do
> > >   too much.
> > 
> > Which version of perltidy did you use? Looking at the archives, the
> > perl code is cleaned up with a specific version, v20090616. See
> > https://www.postgresql.org/message-id/20151204054322.GA2070309@tornado.leadboat.com
> > for example on the matter. As perltidy changes over time, this may be
> > a sensitive change if done this way.
> 
> Hmm. I will make a confirmation on that.. tomorrow.

My perltidy -v said "v20121207'. Anyway, I gave up to apply
perltidy by myself. So I'll just drop 0003 and new 0004 (name
changed to 0003) is made immediately on 0002.

> > > 0004-Use-radix-tree-for-character-conversion.patch
> > >
> > >   Radix tree body.
> > 
> > Well, here a lot of diffs could have been saved.
> > 
> > > The unattached fifth patch is generated by the following steps.
> > >
> > > [$(TOP)]$ ./configure
> > > [Unicode]$ make
> > > [Unicode]$ make distclean
> > > [Unicode]$ git add .
> > > [Unicode]$ commit
> > > === COMMITE MESSSAGE
> > > Replace map files with radix tree files.
> > >
> > > These encodings no longer uses the former map files and uses new radix
> > > tree files.
> > > ===
> > 
> > OK, I can see that working, with 200k of maps generated.. So going
> > through the important bits of this jungle..
> 
> Many thaks for the exploration.
> 
> > +/*
> > + * radix tree conversion function - this should be identical to the function in
> > + * ../conv.c with the same name
..
> > This is not nice. Having a duplication like that is a recipe to forget
> > about it as this patch introduces a dependency with conv.c and the
> > radix tree generation.

In the attatched patch, mb/char_conveter.c which contains one
inline function is created and it is includ'ed from mb/conv.c and
mb/Unicode/map_checker.c.

> > Having a .gitignore in Unicode/ would be nice, particularly to avoid
> > committing map_checker.

I missed this.  I added .gitignore to ignore map_checker stuff
and authority files and old-style map files.

> > A README documenting things may be welcome, or at least comments at
> > the top of map_checker.c. Why is map_checker essential? What does it
> > do? There is no way to understand that easily, except that it includes
> > a "radix tree conversion function", and that it performs sanity checks
> > on the radix trees to be sure that they are on a good shape. But as
> > this something that one would guess only after looking at your patch
> > and the code (at least I will sleep less stupid tonight after reading
> > this stuff).
> 
> Okay, I'll do that.

The patch has not been provided yet, I'm going to put the
following comment just before the main() in map_checker.c.

/*
 * The old-style plain map files were error-resistant due to its
 * straight-forward way for generation from authority files. In contrast the
 * radix tree maps are generated by a rather complex calculation and have a
 * complex, hard-to-confirm format.
 *
 * This program runs sanity check of the radix tree maps by confirming all
 * characters in the plain map files to be converted to the same code by the
 * corresponding radix tree map.
 *
 * All map files are included by map_checker.h that is generated by the script
 * make_mapchecker.pl as the variable mappairs.
 *
 */


I'll do the following things later.

> > --- a/src/backend/utils/mb/Unicode/UCS_to_SJIS.pl
> > +++ b/src/backend/utils/mb/Unicode/UCS_to_SJIS.pl
> >  # Drop these SJIS codes from the source for UTF8=>SJIS conversion
> >  #<<< do not let perltidy touch this
> > -my @reject_sjis =(
> > +my @reject_sjis = (
> >     0xed40..0xeefc, 0x8754..0x875d, 0x878a, 0x8782,
> > -   0x8784, 0xfa5b, 0xfa54, 0x8790..0x8792, 0x8795..0x8797,
> > +   0x8784, 0xfa5b, 0xfa54, 0x8790..0x8792, 0x8795..0x8797,
> >     0x879a..0x879c
> > -);
> > +   );
> > This is not generated, it would be nice to drop the noise from the patch.
> 
> Mmm. I'm not sure how this is generated but I'll care for that.
> 
> > Here is another one:
> > -       $i->{code} = $jis | (
> > -           $jis < 0x100
> > -           ? 0x8e00
> > -           : ($sjis >= 0xeffd ? 0x8f8080 : 0x8080));
> > -
> > +#<<< do not let perltidy touch this
> > +       $i->{code} = $jis | ($jis < 0x100 ? 0x8e00:
> > +                            ($sjis >= 0xeffd ? 0x8f8080 : 0x8080));
> > +#>>>
> 
> Ok. Will revert this.
> 
> >         if (l == 2)
> >         {
> > -           iutf = *utf++ << 8;
> > -           iutf |= *utf++;
> > +           b3 = *utf++;
> > +           b4 = *utf++;
> >         }
> > Ah, OK. This conversion is important so as it performs a minimum of
> > bitwise operations. Yes let's keep that. That's pretty cool to get a
> > faster operation.
> 
> It is Heikki's work:p
> 
> I'll address them and repost the next version sooner.

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

  58. Re: Radix tree for character conversion

    Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> — 2017-01-30T06:37:38Z

    Hello, this is the revised version of character conversion using radix tree.

At Fri, 27 Jan 2017 17:33:57 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> wrote in <20170127.173357.221584433.horiguchi.kyotaro@lab.ntt.co.jp>
> Hi, this is an intermediate report without a patch.
> 
> At Thu, 26 Jan 2017 21:42:12 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> wrote in <20170126.214212.111556326.horiguchi.kyotaro@lab.ntt.co.jp>
> > > > 0003-Apply-pgperltidy-on-src-backend-utils-mb-Unicode.patch
> > > >
> > > >   Before adding radix tree stuff, applied pgperltidy and inserted
> > > >   format-skipping pragma for the parts where perltidy seems to do
> > > >   too much.
> > > 
> > > Which version of perltidy did you use? Looking at the archives, the
> > > perl code is cleaned up with a specific version, v20090616. See
> > > https://www.postgresql.org/message-id/20151204054322.GA2070309@tornado.leadboat.com
> > > for example on the matter. As perltidy changes over time, this may be
> > > a sensitive change if done this way.
> 
> My perltidy -v said "v20121207'. Anyway, I gave up to apply
> perltidy by myself. So I'll just drop 0003 and new 0004 (name
> changed to 0003) is made immediately on 0002.

I'm not sure what to handle this so I just removed the perltidy
stuff from this patchset.

> > > > 0004-Use-radix-tree-for-character-conversion.patch
> > > >
> > > >   Radix tree body.
> > > 
> > > Well, here a lot of diffs could have been saved.
> > > 
> > > > The unattached fifth patch is generated by the following steps.
> > > >
> > > > [$(TOP)]$ ./configure
> > > > [Unicode]$ make
> > > > [Unicode]$ make distclean
> > > > [Unicode]$ git add .
> > > > [Unicode]$ commit
> > > > === COMMITE MESSSAGE
> > > > Replace map files with radix tree files.
> > > >
> > > > These encodings no longer uses the former map files and uses new radix
> > > > tree files.
> > > > ===
> > > 
> > > OK, I can see that working, with 200k of maps generated.. So going
> > > through the important bits of this jungle..
> > 
> > Many thaks for the exploration.
> > 
> > > +/*
> > > + * radix tree conversion function - this should be identical to the function in
> > > + * ../conv.c with the same name
> ..
> > > This is not nice. Having a duplication like that is a recipe to forget
> > > about it as this patch introduces a dependency with conv.c and the
> > > radix tree generation.
> 
> In the attatched patch, mb/char_conveter.c which contains one
> inline function is created and it is includ'ed from mb/conv.c and
> mb/Unicode/map_checker.c.
> 
> > > Having a .gitignore in Unicode/ would be nice, particularly to avoid
> > > committing map_checker.
> 
> I missed this.  I added .gitignore to ignore map_checker stuff
> and authority files and old-style map files.
> 
> > > A README documenting things may be welcome, or at least comments at
> > > the top of map_checker.c. Why is map_checker essential? What does it
> > > do? There is no way to understand that easily, except that it includes
> > > a "radix tree conversion function", and that it performs sanity checks
> > > on the radix trees to be sure that they are on a good shape. But as
> > > this something that one would guess only after looking at your patch
> > > and the code (at least I will sleep less stupid tonight after reading
> > > this stuff).
> > 
> > Okay, I'll do that.
> 
> The patch has not been provided yet, I'm going to put the
> following comment just before the main() in map_checker.c.
> 
> /*
>  * The old-style plain map files were error-resistant due to its
>  * straight-forward way for generation from authority files. In contrast the
>  * radix tree maps are generated by a rather complex calculation and have a
>  * complex, hard-to-confirm format.
>  *
>  * This program runs sanity check of the radix tree maps by confirming all
>  * characters in the plain map files to be converted to the same code by the
>  * corresponding radix tree map.
>  *
>  * All map files are included by map_checker.h that is generated by the script
>  * make_mapchecker.pl as the variable mappairs.
>  *
>  */
> 
> 
> I'll do the following things later.

The following is the continuation.

> > --- a/src/backend/utils/mb/Unicode/UCS_to_SJIS.pl
> > +++ b/src/backend/utils/mb/Unicode/UCS_to_SJIS.pl
> >  # Drop these SJIS codes from the source for UTF8=>SJIS conversion
> >  #<<< do not let perltidy touch this
> > -my @reject_sjis =(
> > +my @reject_sjis = (
> >     0xed40..0xeefc, 0x8754..0x875d, 0x878a, 0x8782,
> > -   0x8784, 0xfa5b, 0xfa54, 0x8790..0x8792, 0x8795..0x8797,
> > +   0x8784, 0xfa5b, 0xfa54, 0x8790..0x8792, 0x8795..0x8797,
> >     0x879a..0x879c
> > -);
> > +   );
> > This is not generated, it would be nice to drop the noise from the patch.
> 
> Mmm. I'm not sure how this is generated but I'll care for that.

I don't still understand what what the intermediate diff comes
from but copy-n-pasting from master silenced it...

> > Here is another one:
> > -       $i->{code} = $jis | (
> > -           $jis < 0x100
> > -           ? 0x8e00
> > -           : ($sjis >= 0xeffd ? 0x8f8080 : 0x8080));
> > -
> > +#<<< do not let perltidy touch this
> > +       $i->{code} = $jis | ($jis < 0x100 ? 0x8e00:
> > +                            ($sjis >= 0xeffd ? 0x8f8080 : 0x8080));
> > +#>>>
> 
> Ok. Will revert this.

The "previous" code (prefixed with a minus sign) is "my"
perltidy's work. Preltidy step is just removed from the patchset.

> >         if (l == 2)
> >         {
> > -           iutf = *utf++ << 8;
> > -           iutf |= *utf++;
> > +           b3 = *utf++;
> > +           b4 = *utf++;
> >         }
> > Ah, OK. This conversion is important so as it performs a minimum of
> > bitwise operations. Yes let's keep that. That's pretty cool to get a
> > faster operation.
> 
> It is Heikki's work:p
> 
> I'll address them and repost the next version sooner.


Finally, the patchset had the following changes from the previous
shape.

- Avoid syntaxes perl 5.8 complains about

- The perltidy step has been removed.

- pg_mb_radix_conv is now a separate .c file (but included from
  other c files)

- Added Unicode/.gitignore. The line for [~#] might be needless.

- The patchset are made with --irreversible-delete. This is just
  what I wanted (but counldn't find by myself..)

- Semicolon-fix patch(0001) gets several additional fixes.



This patchset consists of four patches. The first two are bug
fixes back-patchable to older versions. The third one is the
patch that adds radix tree feature. The forth one is not attached
to this mail but generatable.

0001-Add-missing-semicolon.patch

  Adds missing semicolon found in three files.

0002-Correct-reference-resolution-syntax.patch

  Changes reference syntax to more preferable style.

0003-Use-radix-tree-for-character-conversion.patch

  Radix tree conversion patch. The size has been reduced from
  1.6MB to 91KB.

0004: Replace map files

  This is not attached but generatable. This shouldn't fail even
  on the environment with perl 5.8.

  [$(TOP)]$ ./configure
  [$(TOP)]$ cd src/backend/utils/mb/Unicode
  [Unicode]$ make distclean maintainer-clean all
  [Unicode]$ make mapcheck
  ...
  All radix trees are perfect!
  [Unicode]$ make distclean
  [Unicode]$ git add .
  [Unicode]$ git commit
  ...

  The size of the forth patch was about 7.6MB using --irreversible-delete.

regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

  59. Re: Radix tree for character conversion

    Michael Paquier <michael.paquier@gmail.com> — 2017-01-31T03:25:46Z

    On Mon, Jan 30, 2017 at 3:37 PM, Kyotaro HORIGUCHI
<horiguchi.kyotaro@lab.ntt.co.jp> wrote:
> Hello, this is the revised version of character conversion using radix tree.

Thanks for the new version, I'll look at it once I am done with the
cleanup of the current CF. For now I have moved it to the CF 2017-03.
-- 
Michael

  60. Re: Radix tree for character conversion

    Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> — 2017-01-31T10:06:09Z

    At Tue, 31 Jan 2017 12:25:46 +0900, Michael Paquier <michael.paquier@gmail.com> wrote in <CAB7nPqSGWmHwZqaOA--EHCveG9m77pjWRzxZ6B5iUSJ7GKz-4w@mail.gmail.com>
> On Mon, Jan 30, 2017 at 3:37 PM, Kyotaro HORIGUCHI
> <horiguchi.kyotaro@lab.ntt.co.jp> wrote:
> > Hello, this is the revised version of character conversion using radix tree.
> 
> Thanks for the new version, I'll look at it once I am done with the
> cleanup of the current CF. For now I have moved it to the CF 2017-03.

Agreed. Thank you.

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

  61. Re: Radix tree for character conversion

    Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> — 2017-02-03T04:18:26Z

    Tnanks to that Heikki have pushed the first two patches and a
part of the third, only one patch is remaining now.

# Sorry for not separating KOI8 stuffs.

At Tue, 31 Jan 2017 19:06:09 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> wrote in <20170131.190609.254672218.horiguchi.kyotaro@lab.ntt.co.jp>
> > Thanks for the new version, I'll look at it once I am done with the
> > cleanup of the current CF. For now I have moved it to the CF 2017-03.
> 
> Agreed. Thank you.

Attached is the latest version on the current master (555494d).

Note: since this patch is created by git diff --irreversble-delete,
three files mb/Unicode/*.(txt|xml) to be deleted are left alone.

regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

  62. Re: Radix tree for character conversion

    Michael Paquier <michael.paquier@gmail.com> — 2017-02-22T07:06:14Z

    On Fri, Feb 3, 2017 at 1:18 PM, Kyotaro HORIGUCHI
<horiguchi.kyotaro@lab.ntt.co.jp> wrote:
> Thanks to that Heikki have pushed the first two patches and a
> part of the third, only one patch is remaining now.
>
> # Sorry for not separating KOI8 stuffs.
>
> At Tue, 31 Jan 2017 19:06:09 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> wrote in <20170131.190609.254672218.horiguchi.kyotaro@lab.ntt.co.jp>
>> > Thanks for the new version, I'll look at it once I am done with the
>> > cleanup of the current CF. For now I have moved it to the CF 2017-03.
>>
>> Agreed. Thank you.
>
> Attached is the latest version on the current master (555494d).
>
> Note: since this patch is created by git diff --irreversble-delete,
> three files mb/Unicode/*.(txt|xml) to be deleted are left alone.

Thanks for the rebase. I have been spending sore time looking at this
patch. The new stuff in convutils.pm is by far the interesting part of
the patch, where the building of the radix trees using a byte
structure looks in pretty good shape after eyeballing the logic for a
couple of hours.

+# ignore backup files of editors
+/*[~#]
+
This does not belong to Postgres core code. You could always set up
that in a global exclude file with core.excludefiles.

In order to conduct sanity checks on the shape of the radix tree maps
compared to the existing maps, having map_checker surely makes sense.
Now in the final result I don't think we need it. The existing map
files ought to be replaced by their radix versions at the end, and
map_checker should be removed. This leads to a couple of
simplifications, like Makefile, and reduces the maintenance to one
mechanism.

+sub print_radix_trees
+{
+   my ($this_script, $csname, $charset) = @_;
+
+   &print_radix_map($this_script, $csname, "from_unicode", $charset, 78);
+   &print_radix_map($this_script, $csname, "to_unicode",   $charset, 78);
+}
There is no need for the table width to be defined as a variable (5th
argument). Similarly, to_unicode/from_unicode require checks in
multiple places, this could be just a simple boolean flag. Or if you
want to go to the road of non-simple things, you could have two
arguments: an origin and a target. If one is UTF8 the other is the
mapping name.

+sub dump_charset
+{
+   my ($list, $filt) = @_;
+
+   foreach my $i (@$list)
+   {
+       next if (defined $filt && !&$filt($i));
+       if (!defined $i->{ucs}) { $i->{ucs} = &utf2ucs($i->{utf8}); }
+       printf "ucs=%x, code=%x, direction=%s %s:%d %s\n",
+         $i->{ucs}, $i->{code}, $i->{direction},
+         $i->{f},   $i->{l},    $i->{comment};
+   }
+}
This is used nowhere. Perhaps it was useful for debugging at some point?

+# make_charmap - convert charset table to charmap hash
+#     with checking duplicate source code
Maybe this should be "with checking of duplicated source codes".

+# print_radix_map($this_script, $csname, $direction, \%charset, $tblwidth)
+#
+# this_script - the name of the *caller script* of this feature
$this_script is not needed at all, you could just use basename($0) and
reduce the number of arguments of the different functions of the
stack.

+   ### amount of zeros that can be ovarlaid.
s/ovarlaid/overlaid.

+# make_mapchecker.pl - Gerates map_checker.h file included by map_checker.c
s/gerates/generates/

+           if (s < 0x80)
+           {
+               fprintf(stderr, "\nASCII character ? (%x)", s);
+               exit(1);
+           }
Most likely a newline at the end of the error string is better here.

+           $charmap{ ucs2utf($src) } = $dst;
+       }
+
+   }
Unnecessary newline here.
-- 
Michael

  63. Re: Radix tree for character conversion

    Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> — 2017-02-27T08:37:33Z

    Thank you for the comment.

At Wed, 22 Feb 2017 16:06:14 +0900, Michael Paquier <michael.paquier@gmail.com> wrote in <CAB7nPqRTQ+7ZjxuPTbsr18MXvW7mTd29mN+91N7AG8fe5aCeAA@mail.gmail.com>
> Thanks for the rebase. I have been spending sore time looking at this
> patch. The new stuff in convutils.pm is by far the interesting part of
> the patch, where the building of the radix trees using a byte
> structure looks in pretty good shape after eyeballing the logic for a
> couple of hours.
> 
> +# ignore backup files of editors
> +/*[~#]
> +
> This does not belong to Postgres core code. You could always set up
> that in a global exclude file with core.excludefiles.

Thank you for letting me know about it. I removed that.

> In order to conduct sanity checks on the shape of the radix tree maps
> compared to the existing maps, having map_checker surely makes sense.
> Now in the final result I don't think we need it. The existing map
> files ought to be replaced by their radix versions at the end, and
> map_checker should be removed. This leads to a couple of
> simplifications, like Makefile, and reduces the maintenance to one
> mechanism.

Hmm.. Though I don't remember clearly what the radix map of the
first version looked like, the current radix map seems
human-readable for me. It might be by practice or by additional
comments in map files. Anyway I removed all of the stuff so as
not to generate the plain maps. But I didn't change the names of
_radix.map and just commented out the line to output the plain
maps in UCS_to_*.pl.  Combined maps are still in the plain format
so print_tables was changed to take character tables separately
for regular (non-combined) characters and combined characters.

> +sub print_radix_trees
> +{
> +   my ($this_script, $csname, $charset) = @_;
> +
> +   &print_radix_map($this_script, $csname, "from_unicode", $charset, 78);
> +   &print_radix_map($this_script, $csname, "to_unicode",   $charset, 78);
> +}
> There is no need for the table width to be defined as a variable (5th
> argument).

The table width was already useless.. Removed.

> Similarly, to_unicode/from_unicode require checks in
> multiple places, this could be just a simple boolean flag.

The direction is a tristate (to/from/both) variable so cannot be
replaced with a boolean. But I agree that comparing with free
string is not so good. This is a change already committed in the
master but it is changed in the attached patch.

# Perhaps it is easier to read in string form..

> Or if you want to go to the road of non-simple things, you
> could have two arguments: an origin and a target. If one is
> UTF8 the other is the mapping name.

Mmmm. It seems (even) to me to give more harm than good.  I can
guess two alternatives for this.

- Split the property {direction} into two boolean properties
  {to_unicode} and {from_unicode}.

- Make the {direction} property an integer and compared with
  defined constants $BOTH, $TO_UNICODE and $FROM_UNICODE using
  the '=' operator.

I choosed the former in this patch.

> +sub dump_charset
> +{
> +   my ($list, $filt) = @_;
> +
> +   foreach my $i (@$list)
> +   {
> +       next if (defined $filt && !&$filt($i));
> +       if (!defined $i->{ucs}) { $i->{ucs} = &utf2ucs($i->{utf8}); }
> +       printf "ucs=%x, code=%x, direction=%s %s:%d %s\n",
> +         $i->{ucs}, $i->{code}, $i->{direction},
> +         $i->{f},   $i->{l},    $i->{comment};
> +   }
> +}
> This is used nowhere. Perhaps it was useful for debugging at some point?

Yes, it was quite useful. Removed.

> +# make_charmap - convert charset table to charmap hash
> +#     with checking duplicate source code
> Maybe this should be "with checking of duplicated source codes".

Even though I'm not good English writer, 'duplicated codes' looks
as multiple copies of the original 'code' (for me, of
course). And the 'checking' is a (pure) verbal noun (means not a
deverbal noun) so 'of' is not required. But, of course, I'm not
sure which sounds more natural as English.

This comment is not changed.

> +# print_radix_map($this_script, $csname, $direction, \%charset, $tblwidth)
> +#
> +# this_script - the name of the *caller script* of this feature
> $this_script is not needed at all, you could just use basename($0) and
> reduce the number of arguments of the different functions of the
> stack.

I avoided relying on global stuff by that but I accept the
suggestion. Fixed in this version.

> +   ### amount of zeros that can be ovarlaid.
> s/ovarlaid/overlaid.
> 
> +# make_mapchecker.pl - Gerates map_checker.h file included by map_checker.c
> s/gerates/generates/

make_mapcheker.pl is a tool only for map_checker.c so this files
is removed.

> +           if (s < 0x80)
> +           {
> +               fprintf(stderr, "\nASCII character ? (%x)", s);
> +               exit(1);
> +           }
> Most likely a newline at the end of the error string is better here.

map_checker.c is removed.

> +           $charmap{ ucs2utf($src) } = $dst;
> +       }
> +
> +   }
> Unnecessary newline here.

removed in convutils.pm.

Since Makefile ignores old .map files, the steps to generate a
patch for map files was a bit chaged.

$ rm *.map
$ make distclean maintainer-clean all
$ make distclean
$ git add .
$ git commit

regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

  64. Re: Radix tree for character conversion

    Robert Haas <robertmhaas@gmail.com> — 2017-02-28T02:30:22Z

    On Mon, Feb 27, 2017 at 2:07 PM, Kyotaro HORIGUCHI
<horiguchi.kyotaro@lab.ntt.co.jp> wrote:
>> +# make_charmap - convert charset table to charmap hash
>> +#     with checking duplicate source code
>> Maybe this should be "with checking of duplicated source codes".
>
> Even though I'm not good English writer, 'duplicated codes' looks
> as multiple copies of the original 'code' (for me, of
> course). And the 'checking' is a (pure) verbal noun (means not a
> deverbal noun) so 'of' is not required. But, of course, I'm not
> sure which sounds more natural as English

The problem is that, because "checking" is a noun in this sentence, it
can't be followed by a direct object so you need "of" to connect
"checking" with the thing that is being checked.  However, what I
would do is rearrange this sentence slightly as to use "checking" as a
verb, like this:

convert charset table to charmap hash, checking for duplicate source
codes along the way

While I don't think Michael's suggestion is wrong, I find the above a
little more natural.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  65. Re: Radix tree for character conversion

    Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> — 2017-02-28T04:19:10Z

    Hello,

At Tue, 28 Feb 2017 08:00:22 +0530, Robert Haas <robertmhaas@gmail.com> wrote in <CA+TgmoYheGx+knBQAMBm+nr8Cnr7e2RZm1BEwgK5AGMx4MKv9A@mail.gmail.com>
> On Mon, Feb 27, 2017 at 2:07 PM, Kyotaro HORIGUCHI
> <horiguchi.kyotaro@lab.ntt.co.jp> wrote:
> >> +# make_charmap - convert charset table to charmap hash
> >> +#     with checking duplicate source code
> >> Maybe this should be "with checking of duplicated source codes".
> >
> > Even though I'm not good English writer, 'duplicated codes' looks
> > as multiple copies of the original 'code' (for me, of
> > course). And the 'checking' is a (pure) verbal noun (means not a
> > deverbal noun) so 'of' is not required. But, of course, I'm not
> > sure which sounds more natural as English
> 
> The problem is that, because "checking" is a noun in this sentence, it
> can't be followed by a direct object so you need "of" to connect
> "checking" with the thing that is being checked.  However, what I
> would do is rearrange this sentence slightly as to use "checking" as a
> verb, like this:
> 
> convert charset table to charmap hash, checking for duplicate source
> codes along the way
> 
> While I don't think Michael's suggestion is wrong, I find the above a
> little more natural.

Thank you for the suggestion and explantion. I leaned that,
maybe.  I'll send the version with the revised comment.

regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

  66. Re: Radix tree for character conversion

    Michael Paquier <michael.paquier@gmail.com> — 2017-02-28T06:20:06Z

    On Mon, Feb 27, 2017 at 5:37 PM, Kyotaro HORIGUCHI
<horiguchi.kyotaro@lab.ntt.co.jp> wrote:
> At Wed, 22 Feb 2017 16:06:14 +0900, Michael Paquier <michael.paquier@gmail.com> wrote in <CAB7nPqRTQ+7ZjxuPTbsr18MXvW7mTd29mN+91N7AG8fe5aCeAA@mail.gmail.com>
>> In order to conduct sanity checks on the shape of the radix tree maps
>> compared to the existing maps, having map_checker surely makes sense.
>> Now in the final result I don't think we need it. The existing map
>> files ought to be replaced by their radix versions at the end, and
>> map_checker should be removed. This leads to a couple of
>> simplifications, like Makefile, and reduces the maintenance to one
>> mechanism.
>
> Hmm.. Though I don't remember clearly what the radix map of the
> first version looked like, the current radix map seems
> human-readable for me. It might be by practice or by additional
> comments in map files. Anyway I removed all of the stuff so as
> not to generate the plain maps. But I didn't change the names of
> _radix.map and just commented out the line to output the plain
> maps in UCS_to_*.pl.  Combined maps are still in the plain format
> so print_tables was changed to take character tables separately
> for regular (non-combined) characters and combined characters.

Do others have thoughts to offer on the matter? I would think that the
new radix maps should just replace by the old plain ones, and that the
only way to build the maps going forward is to use the new methods.
The radix trees is the only thing used in the backend code as well
(conv.c). We could keep the way to build the old maps, with the
map_checker in module out of the core code. FWIW, I am fine to add the
old APIs in my plugin repository on github and have the sanity checks
in that as well. And of course also publish on this thread a module to
do that.

>> Or if you want to go to the road of non-simple things, you
>> could have two arguments: an origin and a target. If one is
>> UTF8 the other is the mapping name.
>
> Mmmm. It seems (even) to me to give more harm than good.  I can
> guess two alternatives for this.
>
> - Split the property {direction} into two boolean properties
>   {to_unicode} and {from_unicode}.
>
> - Make the {direction} property an integer and compared with
>   defined constants $BOTH, $TO_UNICODE and $FROM_UNICODE using
>   the '=' operator.
>
> I choosed the former in this patch.

Fine for me.

>> +           $charmap{ ucs2utf($src) } = $dst;
>> +       }
>> +
>> +   }
>> Unnecessary newline here.
>
> removed in convutils.pm.
>
> Since Makefile ignores old .map files, the steps to generate a
> patch for map files was a bit chaged.
>
> $ rm *.map
> $ make distclean maintainer-clean all
> $ make distclean
> $ git add .
> $ git commit

+# ignore generated files
+/map_checker
+/map_checker.h
[...]
+map_checker.h: make_mapchecker.pl $(MAPS) $(RADIXMAPS)
+   $(PERL) $<
+
+map_checker.o: map_checker.c map_checker.h ../char_converter.c
+
+map_checker: map_checker.o
With map_checker out of the game, those things are not needed.

+++ b/src/backend/utils/mb/char_converter.c
@@ -0,0 +1,116 @@
+/*-------------------------------------------------------------------------
+ *
+ *   Character converter function using radix tree
In the simplified version of the patch, pg_mb_radix_conv() being only
needed in conv.c I think that this could just be a static local
routine.

-#include "../../Unicode/utf8_to_koi8r.map"
-#include "../../Unicode/koi8r_to_utf8.map"
-#include "../../Unicode/utf8_to_koi8u.map"
-#include "../../Unicode/koi8u_to_utf8.map"
+#include "../../Unicode/utf8_to_koi8r_radix.map"
+#include "../../Unicode/koi8r_to_utf8_radix.map"
+#include "../../Unicode/utf8_to_koi8u_radix.map"
+#include "../../Unicode/koi8u_to_utf8_radix.map"
FWIW, I am fine to use those new names as include points.

-distclean: clean
+distclean:
    rm -f $(TEXTS)
-maintainer-clean: distclean
+# maintainer-clean intentionally leaves $(TEXTS)
+maintainer-clean:
Why is that? There is also a useless diff down that code block.

+conv.o: conv.c char_converter.c
This also can go away.

-print_tables("EUC_JIS_2004", \@all, 1);
+# print_tables("EUC_JIS_2004", \@regular, undef, 1);
+print_radix_trees("EUC_JIS_2004", \@regular);
+print_tables("EUC_JIS_2004", undef, \@combined, 1);
[...]
 sub print_tables
 {
-   my ($charset, $table, $verbose) = @_;
+   my ($charset, $regular, $combined, $verbose) = @_;
print_tables is only used for combined maps, you could remove $regular
from it and just keep $combined around, perhaps renaming print_tables
to print_combined_maps?
-- 
Michael

  67. Re: Radix tree for character conversion

    Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> — 2017-02-28T08:34:02Z

    At Tue, 28 Feb 2017 15:20:06 +0900, Michael Paquier <michael.paquier@gmail.com> wrote in <CAB7nPqR49krGP6qaaKaL2v3HCnn+dnzv8Dq_ySGbDSr6b_ywrw@mail.gmail.com>
> On Mon, Feb 27, 2017 at 5:37 PM, Kyotaro HORIGUCHI
> <horiguchi.kyotaro@lab.ntt.co.jp> wrote:
> > At Wed, 22 Feb 2017 16:06:14 +0900, Michael Paquier <michael.paquier@gmail.com> wrote in <CAB7nPqRTQ+7ZjxuPTbsr18MXvW7mTd29mN+91N7AG8fe5aCeAA@mail.gmail.com>
> >> In order to conduct sanity checks on the shape of the radix tree maps
> >> compared to the existing maps, having map_checker surely makes sense.
> >> Now in the final result I don't think we need it. The existing map
> >> files ought to be replaced by their radix versions at the end, and
> >> map_checker should be removed. This leads to a couple of
> >> simplifications, like Makefile, and reduces the maintenance to one
> >> mechanism.
> >
> > Hmm.. Though I don't remember clearly what the radix map of the
> > first version looked like, the current radix map seems
> > human-readable for me. It might be by practice or by additional
> > comments in map files. Anyway I removed all of the stuff so as
> > not to generate the plain maps. But I didn't change the names of
> > _radix.map and just commented out the line to output the plain
> > maps in UCS_to_*.pl.  Combined maps are still in the plain format
> > so print_tables was changed to take character tables separately
> > for regular (non-combined) characters and combined characters.
> 
> Do others have thoughts to offer on the matter? I would think that the
> new radix maps should just replace by the old plain ones, and that the
> only way to build the maps going forward is to use the new methods.
> The radix trees is the only thing used in the backend code as well
> (conv.c). We could keep the way to build the old maps, with the
> map_checker in module out of the core code. FWIW, I am fine to add the
> old APIs in my plugin repository on github and have the sanity checks
> in that as well. And of course also publish on this thread a module to
> do that.

I couldn't make out my mind to move to radix tree completely, but
UtfToLocal/LocalToUtf no longer handle the "plain map"s for
non-combined character so they have lost their planground. Okay,
I think I removed all the trace of the plain map era.

Every characters in a mapping has a comment that describes what
the character is or where it is defined. This information is no
longer useful (radix map doesn't have a plance to show it) but
I left it for debug use. (This might just be justification..)

> > - Split the property {direction} into two boolean properties
> >   {to_unicode} and {from_unicode}.
> >
> > - Make the {direction} property an integer and compared with
> >   defined constants $BOTH, $TO_UNICODE and $FROM_UNICODE using
> >   the '=' operator.
> >
> > I choosed the former in this patch.
> 
> Fine for me.

Thanks.

> >> +           $charmap{ ucs2utf($src) } = $dst;
> >> +       }
> >> +
> >> +   }
> >> Unnecessary newline here.
> >
> > removed in convutils.pm.
> >
> > Since Makefile ignores old .map files, the steps to generate a
> > patch for map files was a bit chaged.
> >
> > $ rm *.map
> > $ make distclean maintainer-clean all
> > $ make distclean
> > $ git add .
> > $ git commit
> 
> +# ignore generated files
> +/map_checker
> +/map_checker.h
> [...]
> +map_checker.h: make_mapchecker.pl $(MAPS) $(RADIXMAPS)
> +   $(PERL) $<
> +
> +map_checker.o: map_checker.c map_checker.h ../char_converter.c
> +
> +map_checker: map_checker.o
> With map_checker out of the game, those things are not needed.

Ouch! Thanks for pointing out it. Removed.

> +++ b/src/backend/utils/mb/char_converter.c
> @@ -0,0 +1,116 @@
> +/*-------------------------------------------------------------------------
> + *
> + *   Character converter function using radix tree
> In the simplified version of the patch, pg_mb_radix_conv() being only
> needed in conv.c I think that this could just be a static local
> routine.
> 
> -#include "../../Unicode/utf8_to_koi8r.map"
> -#include "../../Unicode/koi8r_to_utf8.map"
> -#include "../../Unicode/utf8_to_koi8u.map"
> -#include "../../Unicode/koi8u_to_utf8.map"
> +#include "../../Unicode/utf8_to_koi8r_radix.map"
> +#include "../../Unicode/koi8r_to_utf8_radix.map"
> +#include "../../Unicode/utf8_to_koi8u_radix.map"
> +#include "../../Unicode/koi8u_to_utf8_radix.map"
> FWIW, I am fine to use those new names as include points.
> 
> -distclean: clean
> +distclean:
>     rm -f $(TEXTS)
> -maintainer-clean: distclean
> +# maintainer-clean intentionally leaves $(TEXTS)
> +maintainer-clean:
> Why is that? There is also a useless diff down that code block.

It *was* for convenience but now it is automatically downloaded
so such distinction donsn't offer anything good. Changed it to
remove $(TEXTS).

> +conv.o: conv.c char_converter.c
> This also can go away.

Touching char_converter.c will be ignored if it is removed. Did
you mistake it for map_checker?

> -print_tables("EUC_JIS_2004", \@all, 1);
> +# print_tables("EUC_JIS_2004", \@regular, undef, 1);
> +print_radix_trees("EUC_JIS_2004", \@regular);
> +print_tables("EUC_JIS_2004", undef, \@combined, 1);
> [...]
>  sub print_tables
>  {
> -   my ($charset, $table, $verbose) = @_;
> +   my ($charset, $regular, $combined, $verbose) = @_;
> print_tables is only used for combined maps, you could remove $regular
> from it and just keep $combined around, perhaps renaming print_tables
> to print_combined_maps?

Renamed to print_combied_maps.

And the code-comment pointed in the comment by the previous mail
is rewritten as Robert's suggestion.

regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

  68. Re: Radix tree for character conversion

    Michael Paquier <michael.paquier@gmail.com> — 2017-03-01T05:34:23Z

    On Tue, Feb 28, 2017 at 5:34 PM, Kyotaro HORIGUCHI
<horiguchi.kyotaro@lab.ntt.co.jp> wrote:
> At Tue, 28 Feb 2017 15:20:06 +0900, Michael Paquier <michael.paquier@gmail.com> wrote in <CAB7nPqR49krGP6qaaKaL2v3HCnn+dnzv8Dq_ySGbDSr6b_ywrw@mail.gmail.com>
>> +conv.o: conv.c char_converter.c
>> This also can go away.
>
> Touching char_converter.c will be ignored if it is removed. Did
> you mistake it for map_checker?

That was not what I meant: as pg_mb_radix_conv() is only used in
conv.c, it may be better to just remove completely char_converter.c.

> And the code-comment pointed in the comment by the previous mail
> is rewritten as Robert's suggestion.

Fine for me.

-distclean: clean
+distclean:
    rm -f $(TEXTS)

-maintainer-clean: distclean
-   rm -f $(MAPS)
-
+maintainer-clean:
+   rm -f $(TEXTS) $(MAPS)
Well, I would have assumed that this should not change..

The last version of the patch looks in rather good shape to me, we are
also sure that the sanity checks on the old maps and the new maps
match per the previous runs with map_checker. One thing that still
need some extra opinions is what to do with the old maps:
1) Just remove them, replacing the old maps by the new radix tree maps.
2) Keep them around in the backend code, even if they are useless.
3) Use a GUC to be able to switch from one to the other, giving a
fallback method in case of emergency.
4) Use an extension module to store the old maps with as well the
previous build code, so as sanity checks can still be performed on the
new maps.

I would vote for 2), to reduce long term maintenance burdens and after
seeing all the sanity checks that have been done in previous versions.
-- 
Michael

  69. Re: Radix tree for character conversion

    Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> — 2017-03-02T05:20:00Z

    At Wed, 1 Mar 2017 14:34:23 +0900, Michael Paquier <michael.paquier@gmail.com> wrote in <CAB7nPqQ_4n+FDWi5Xiueo68i=fTmdg1Wx+y6XWWX=8rAhKRtFw@mail.gmail.com>
> On Tue, Feb 28, 2017 at 5:34 PM, Kyotaro HORIGUCHI
> <horiguchi.kyotaro@lab.ntt.co.jp> wrote:
> > At Tue, 28 Feb 2017 15:20:06 +0900, Michael Paquier <michael.paquier@gmail.com> wrote in <CAB7nPqR49krGP6qaaKaL2v3HCnn+dnzv8Dq_ySGbDSr6b_ywrw@mail.gmail.com>
> >> +conv.o: conv.c char_converter.c
> >> This also can go away.
> >
> > Touching char_converter.c will be ignored if it is removed. Did
> > you mistake it for map_checker?
> 
> That was not what I meant: as pg_mb_radix_conv() is only used in
> conv.c, it may be better to just remove completely char_converter.c.

Ouch! You're right. Sorry for my short-sight. char_converter.c is
removed and related description in Makefile is removed.

> > And the code-comment pointed in the comment by the previous mail
> > is rewritten as Robert's suggestion.
> 
> Fine for me.
> 
> -distclean: clean
> +distclean:
>     rm -f $(TEXTS)
> 
> -maintainer-clean: distclean
> -   rm -f $(MAPS)
> -
> +maintainer-clean:
> +   rm -f $(TEXTS) $(MAPS)
> Well, I would have assumed that this should not change..

I should have reverted there but actually the patch somehow does
the different thing.. Surely reverted it this time.

> The last version of the patch looks in rather good shape to me, we are
> also sure that the sanity checks on the old maps and the new maps
> match per the previous runs with map_checker.

Agreed.

>  One thing that still
> need some extra opinions is what to do with the old maps:
> 1) Just remove them, replacing the old maps by the new radix tree maps.
> 2) Keep them around in the backend code, even if they are useless.
> 3) Use a GUC to be able to switch from one to the other, giving a
> fallback method in case of emergency.
> 4) Use an extension module to store the old maps with as well the
> previous build code, so as sanity checks can still be performed on the
> new maps.
> 
> I would vote for 2), to reduce long term maintenance burdens and after
> seeing all the sanity checks that have been done in previous versions.

I don't vote 3 and 4. And I did 1 in the last patch.

About 2, any change in the authority files rarely but possiblly
happens. Even in the case the plain map files are no longer can
be generated. (but can with a bit tweak of convutils.pm) If the
radix-tree file generator is under a suspicion, the "plain" map
file generator (and the map_checker) or some other means to
sanity check might be required.

That being said, when something occurs in radix files, we can
find it in the radix file and can find the corresponding lines in
the aurhority file. The remaining problem is the case where some
substantial change in authority files doesn't affect radix
files. We can detect such mistake by detecting changes in
authority files. So I propose the 5th option.

5) Just remove plain map files and all related code. Addition to
   that, Makefile stores hash digest of authority files in
   Unicode/authoriy_hashes.txt or something that is managed by
   git.

This digest may differ among platforms (typically from cr/nl
difference) but we can assume *nix for the usage.

I will send the next version after this discussion is settled.


regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

  70. Re: Radix tree for character conversion

    Michael Paquier <michael.paquier@gmail.com> — 2017-03-03T03:53:04Z

    On Thu, Mar 2, 2017 at 2:20 PM, Kyotaro HORIGUCHI
<horiguchi.kyotaro@lab.ntt.co.jp> wrote:
> 5) Just remove plain map files and all related code. Addition to
>    that, Makefile stores hash digest of authority files in
>    Unicode/authoriy_hashes.txt or something that is managed by
>    git.

That may be an idea to check for differences across upstream versions.
But that sounds like a separate discussion to me.

> This digest may differ among platforms (typically from cr/nl
> difference) but we can assume *nix for the usage.
>
> I will send the next version after this discussion is settled.

Sure. There is not much point to move on without Heikki's opinion at
least, or anybody else like Ishii-san or Tom who are familiar with
this code. I would think that Heikki would be the committer to pick up
this change though.
-- 
Michael

  71. Re: Radix tree for character conversion

    Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> — 2017-03-06T08:16:09Z

    Hello,

At Fri, 3 Mar 2017 12:53:04 +0900, Michael Paquier <michael.paquier@gmail.com> wrote in <CAB7nPqSQaLozFNg+5Tf9s1TZs2pcE-GHhnMG31qnsusV9vMUOw@mail.gmail.com>
> On Thu, Mar 2, 2017 at 2:20 PM, Kyotaro HORIGUCHI
> <horiguchi.kyotaro@lab.ntt.co.jp> wrote:
> > 5) Just remove plain map files and all related code. Addition to
> >    that, Makefile stores hash digest of authority files in
> >    Unicode/authoriy_hashes.txt or something that is managed by
> >    git.
> 
> That may be an idea to check for differences across upstream versions.
> But that sounds like a separate discussion to me.

Fine with me either.

> > This digest may differ among platforms (typically from cr/nl
> > difference) but we can assume *nix for the usage.
> >
> > I will send the next version after this discussion is settled.
> 
> Sure. There is not much point to move on without Heikki's opinion at
> least, or anybody else like Ishii-san or Tom who are familiar with
> this code. I would think that Heikki would be the committer to pick up
> this change though.

So, this is the latest version of this patch in the shape of the
option 1.


| need some extra opinions is what to do with the old maps:
| 1) Just remove them, replacing the old maps by the new radix tree maps.
| 2) Keep them around in the backend code, even if they are useless.
| 3) Use a GUC to be able to switch from one to the other, giving a
| fallback method in case of emergency.
| 4) Use an extension module to store the old maps with as well the
| previous build code, so as sanity checks can still be performed on the
| new maps.


regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

  72. Re: Radix tree for character conversion

    Heikki Linnakangas <hlinnaka@iki.fi> — 2017-03-13T18:48:07Z

    On 03/06/2017 10:16 AM, Kyotaro HORIGUCHI wrote:
> At Fri, 3 Mar 2017 12:53:04 +0900, Michael Paquier <michael.paquier@gmail.com> wrote in <CAB7nPqSQaLozFNg+5Tf9s1TZs2pcE-GHhnMG31qnsusV9vMUOw@mail.gmail.com>
>> On Thu, Mar 2, 2017 at 2:20 PM, Kyotaro HORIGUCHI
>> <horiguchi.kyotaro@lab.ntt.co.jp> wrote:
>>> 5) Just remove plain map files and all related code. Addition to
>>>    that, Makefile stores hash digest of authority files in
>>>    Unicode/authoriy_hashes.txt or something that is managed by
>>>    git.
>>
>> That may be an idea to check for differences across upstream versions.
>> But that sounds like a separate discussion to me.
>
> Fine with me either.

I did some more kibitzing here and there, and committed. Thanks everyone!

I agree the new maps should just replace the old maps altogether, so 
committed that way. I also moved the combined map files to the same .map 
files as the main radix trees. Seems more clear that way to me.

I changed the to/from_unicode properties back to a single direction 
property, with Perl constants BOTH, TO_UNICODE and FROM_UNICODE, per 
your alternative suggestion upthread. Seems more clear to me.

It would be nice to run the map_checker tool one more time, though, to 
verify that the mappings match those from PostgreSQL 9.6. Just to be 
sure, and after that the map checker can go to the dustbin.

- Heikki

  73. Re: Radix tree for character conversion

    Tom Lane <tgl@sss.pgh.pa.us> — 2017-03-13T18:53:17Z

    Heikki Linnakangas <hlinnaka@iki.fi> writes:
> I did some more kibitzing here and there, and committed. Thanks everyone!

111 files changed, 147742 insertions(+), 367346 deletions(-)

Nice.

> It would be nice to run the map_checker tool one more time, though, to 
> verify that the mappings match those from PostgreSQL 9.6.

+1

> Just to be sure, and after that the map checker can go to the dustbin.

Hm, maybe we should keep it around for the next time somebody has a bright
idea in this area?

			regards, tom lane

  74. Re: Radix tree for character conversion

    Heikki Linnakangas <hlinnaka@iki.fi> — 2017-03-13T19:07:39Z

    On 03/13/2017 08:53 PM, Tom Lane wrote:
> Heikki Linnakangas <hlinnaka@iki.fi> writes:
>> It would be nice to run the map_checker tool one more time, though, to
>> verify that the mappings match those from PostgreSQL 9.6.
>
> +1
>
>> Just to be sure, and after that the map checker can go to the dustbin.
>
> Hm, maybe we should keep it around for the next time somebody has a bright
> idea in this area?

The map checker compares old-style maps with the new radix maps. The 
next time 'round, we'll need something that compares the radix maps with 
the next great thing. Not sure how easy it would be to adapt.

Hmm. A somewhat different approach might be more suitable for testing 
across versions, though. We could modify the perl scripts slightly to 
print out SQL statements that exercise every mapping. For every 
supported conversion, the SQL script could:

1. create a database in the source encoding.
2. set client_encoding='<target encoding>'
3. SELECT a string that contains every character in the source encoding.

You could then run those SQL statements against old and new server 
version, and verify that you get the same results.

- Heikki

  75. Re: Radix tree for character conversion

    Michael Paquier <michael.paquier@gmail.com> — 2017-03-13T23:59:42Z

    On Tue, Mar 14, 2017 at 4:07 AM, Heikki Linnakangas <hlinnaka@iki.fi> wrote:
> On 03/13/2017 08:53 PM, Tom Lane wrote:
>> Heikki Linnakangas <hlinnaka@iki.fi> writes:
>>>
>>> It would be nice to run the map_checker tool one more time, though, to
>>> verify that the mappings match those from PostgreSQL 9.6.
>>
>> +1

Nice to login and see that committed!
-- 
Michael

  76. Re: Radix tree for character conversion

    Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> — 2017-03-17T05:19:31Z

    Thank you for committing this.

At Mon, 13 Mar 2017 21:07:39 +0200, Heikki Linnakangas <hlinnaka@iki.fi> wrote in <d5b70078-9f57-0f63-3462-1e564a57739f@iki.fi>
> On 03/13/2017 08:53 PM, Tom Lane wrote:
> > Heikki Linnakangas <hlinnaka@iki.fi> writes:
> >> It would be nice to run the map_checker tool one more time, though, to
> >> verify that the mappings match those from PostgreSQL 9.6.
> >
> > +1
> >
> >> Just to be sure, and after that the map checker can go to the dustbin.
> >
> > Hm, maybe we should keep it around for the next time somebody has a
> > bright
> > idea in this area?
> 
> The map checker compares old-style maps with the new radix maps. The
> next time 'round, we'll need something that compares the radix maps
> with the next great thing. Not sure how easy it would be to adapt.
> 
> Hmm. A somewhat different approach might be more suitable for testing
> across versions, though. We could modify the perl scripts slightly to
> print out SQL statements that exercise every mapping. For every
> supported conversion, the SQL script could:
> 
> 1. create a database in the source encoding.
> 2. set client_encoding='<target encoding>'
> 3. SELECT a string that contains every character in the source
> encoding.

There are many encodings that can be client-encoding but cannot
be database-encoding. And some encodings such as UTF-8 has
several one-way conversion. If we do something like this, it
would be as the following.

1. Encoding test
1-1. create a database in UTF-8
1-2. set client_encoding='<source encoding>'
1-3. INSERT all characters defined in the source encoding.
1-4. set client_encoding='UTF-8'
1-5. SELECT a string that contains every character in UTF-8.
2. Decoding test

.... sucks!


I would like to use convert() function. It can be a large
PL/PgSQL function or a series of "SELECT convert(...)"s. The
latter is doable on-the-fly (by not generating/storing the whole
script).

| -- Test for SJIS->UTF-8 conversion
| ...
| SELECT convert('\0000', 'SJIS', 'UTF-8'); -- results in error
| ...
| SELECT convert('\897e', 'SJIS', 'UTF-8');

> You could then run those SQL statements against old and new server
> version, and verify that you get the same results.

Including the result files in the repository will make this easy
but unacceptably bloats. Put mb/Unicode/README.sanity_check?

regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

  77. Re: Radix tree for character conversion

    Heikki Linnakangas <hlinnaka@iki.fi> — 2017-03-17T11:03:35Z

    On 03/17/2017 07:19 AM, Kyotaro HORIGUCHI wrote:
> At Mon, 13 Mar 2017 21:07:39 +0200, Heikki Linnakangas <hlinnaka@iki.fi> wrote in <d5b70078-9f57-0f63-3462-1e564a57739f@iki.fi>
>> Hmm. A somewhat different approach might be more suitable for testing
>> across versions, though. We could modify the perl scripts slightly to
>> print out SQL statements that exercise every mapping. For every
>> supported conversion, the SQL script could:
>>
>> 1. create a database in the source encoding.
>> 2. set client_encoding='<target encoding>'
>> 3. SELECT a string that contains every character in the source
>> encoding.
>
> There are many encodings that can be client-encoding but cannot
> be database-encoding.

Good point.

> I would like to use convert() function. It can be a large
> PL/PgSQL function or a series of "SELECT convert(...)"s. The
> latter is doable on-the-fly (by not generating/storing the whole
> script).
>
> | -- Test for SJIS->UTF-8 conversion
> | ...
> | SELECT convert('\0000', 'SJIS', 'UTF-8'); -- results in error
> | ...
> | SELECT convert('\897e', 'SJIS', 'UTF-8');

Makes sense.

>> You could then run those SQL statements against old and new server
>> version, and verify that you get the same results.
>
> Including the result files in the repository will make this easy
> but unacceptably bloats. Put mb/Unicode/README.sanity_check?

Yeah, a README with instructions on how to do sounds good. No need to 
include the results in the repository, you can run the script against an 
older version when you need something to compare with.

- Heikki

  78. Re: Radix tree for character conversion

    Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> — 2017-03-21T04:10:48Z

    Hello,

At Fri, 17 Mar 2017 13:03:35 +0200, Heikki Linnakangas <hlinnaka@iki.fi> wrote in <01efd334-b839-0450-1b63-f2dea9326a7e@iki.fi>
> On 03/17/2017 07:19 AM, Kyotaro HORIGUCHI wrote:
> > I would like to use convert() function. It can be a large
> > PL/PgSQL function or a series of "SELECT convert(...)"s. The
> > latter is doable on-the-fly (by not generating/storing the whole
> > script).
> >
> > | -- Test for SJIS->UTF-8 conversion
> > | ...
> > | SELECT convert('\0000', 'SJIS', 'UTF-8'); -- results in error
> > | ...
> > | SELECT convert('\897e', 'SJIS', 'UTF-8');
> 
> Makes sense.
> 
> >> You could then run those SQL statements against old and new server
> >> version, and verify that you get the same results.
> >
> > Including the result files in the repository will make this easy
> > but unacceptably bloats. Put mb/Unicode/README.sanity_check?
> 
> Yeah, a README with instructions on how to do sounds good. No need to
> include the results in the repository, you can run the script against
> an older version when you need something to compare with.

Ok, I'll write a small script to generate a set of "conversion
dump" and try to write README.sanity_check describing how to use
it.

regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

  79. Re: Radix tree for character conversion

    Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> — 2017-03-23T03:13:07Z

    At Tue, 21 Mar 2017 13:10:48 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> wrote in <20170321.131048.150321071.horiguchi.kyotaro@lab.ntt.co.jp>
> At Fri, 17 Mar 2017 13:03:35 +0200, Heikki Linnakangas <hlinnaka@iki.fi> wrote in <01efd334-b839-0450-1b63-f2dea9326a7e@iki.fi>
> > On 03/17/2017 07:19 AM, Kyotaro HORIGUCHI wrote:
> > > I would like to use convert() function. It can be a large
> > > PL/PgSQL function or a series of "SELECT convert(...)"s. The
> > > latter is doable on-the-fly (by not generating/storing the whole
> > > script).
> > >
> > > | -- Test for SJIS->UTF-8 conversion
> > > | ...
> > > | SELECT convert('\0000', 'SJIS', 'UTF-8'); -- results in error
> > > | ...
> > > | SELECT convert('\897e', 'SJIS', 'UTF-8');
> > 
> > Makes sense.
> > 
> > >> You could then run those SQL statements against old and new server
> > >> version, and verify that you get the same results.
> > >
> > > Including the result files in the repository will make this easy
> > > but unacceptably bloats. Put mb/Unicode/README.sanity_check?
> > 
> > Yeah, a README with instructions on how to do sounds good. No need to
> > include the results in the repository, you can run the script against
> > an older version when you need something to compare with.
> 
> Ok, I'll write a small script to generate a set of "conversion
> dump" and try to write README.sanity_check describing how to use
> it.

I found that there's no way to identify the character domain of a
conversion on SQL interface. Unconditionally giving from 0 to
0xffffffff as a bytea string yields too-bloat result by containg
many bogus lines.  (If \x40 is a character, convert() also
accepts \x4040, \x404040 and \x40404040)

One more annoyance is the fact that mappings and conversion
procedures are not in one-to-one correspondence. The
corresnponcence is hidden in conversion_procs/*.c files so we
should extract it from them or provide as knowledge. Both don't
seem good.

Finally, it seems that I have no choice than resurrecting
map_checker. The exactly the same one no longer works but
map_dumper.c with almost the same structure will work.

If no one objects to adding map_dumper.c and
gen_mapdumper_header.pl (tentavie name, of course), I'll make a
patch to do that.

Any suggestions?

regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

  80. Re: Radix tree for character conversion

    Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> — 2017-03-27T10:05:43Z

    Hmm, things are bit different.

At Thu, 23 Mar 2017 12:13:07 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> wrote in <20170323.121307.241436413.horiguchi.kyotaro@lab.ntt.co.jp>
> > Ok, I'll write a small script to generate a set of "conversion
> > dump" and try to write README.sanity_check describing how to use
> > it.
> 
> I found that there's no way to identify the character domain of a
> conversion on SQL interface. Unconditionally giving from 0 to
> 0xffffffff as a bytea string yields too-bloat result by containg
> many bogus lines.  (If \x40 is a character, convert() also
> accepts \x4040, \x404040 and \x40404040)
> 
> One more annoyance is the fact that mappings and conversion
> procedures are not in one-to-one correspondence. The
> corresnponcence is hidden in conversion_procs/*.c files so we
> should extract it from them or provide as knowledge. Both don't
> seem good.
> 
> Finally, it seems that I have no choice than resurrecting
> map_checker. The exactly the same one no longer works but
> map_dumper.c with almost the same structure will work.
> 
> If no one objects to adding map_dumper.c and
> gen_mapdumper_header.pl (tentavie name, of course), I'll make a
> patch to do that.

The scirpt or executable should be compatible between versions
but pg_mb_radix_conv is not. On the other hand more upper level
API reuiqres server stuff.

Finally I made an extension that dumps encoding conversion.

encoding_dumper('SJIS', 'UTF-8') or encoding_dumper(35, 6)

Then it returns the following output consists of two BYTEAs.

 srccode | dstcode  
---------+----------
 \x01    | \x01
 \x02    | \x02
...
 \xfc4a  | \xe9b899
 \xfc4b  | \xe9bb91
(7914 rows)

This returns in a very short time but doesn't when srccode
extends to 4 bytes. As an extreme example the following,

> =# select * from encoding_dumper('UTF-8', 'LATIN1');

takes over 2 minutes to return only 255 rows. We cannot determine
the exact domain without looking into map data so the function
cannot do other than looping through all the four-byte values.
Providing a function that gives the domain for a conversion was a
mess, especially for artithmetic-conversions. The following query
took 94 minutes to give 25M lines/125MB.  In short, that's a
crap. (the first attached)

SELECT x.conname, y.srccode, y.dstcode
FROM
 (
    SELECT conname, conforencoding, contoencoding
    FROM pg_conversion c
    WHERE pg_char_to_encoding('UTF-8') IN (c.conforencoding, c.contoencoding)
      AND pg_char_to_encoding('SQL_ASCII')
          NOT IN (c.conforencoding, c.contoencoding)) as x,
 LATERAL (
   SELECT srccode, dstcode
   FROM  encoding_dumper(x.conforencoding, x.contoencoding)) as y
ORDER BY x.conforencoding, x.contoencoding, y.srccode;


As the another way, I added a measure to generate plain mapping
lists corresponding to .map files (similar to old maps but
simpler) and this finishes the work within a second.

$ make mapdumps

If we will not shortly change the framework of mapped character
conversion, the dumper program may be useful but I'm not sure
this is reasonable as sanity check for future modifications.  In
the PoC, pg_mb_radix_tree() is copied into map_checker.c but this
needs to be a separate file again.  (the second attached)


regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center