Horton

Re: ICU locale validation / canonicalization

Peter Eisentraut <peter.eisentraut@enterprisedb.com>

View thread View on postgr.es
From: Peter Eisentraut <peter.eisentraut@enterprisedb.com>
To: Jeff Davis <pgsql@j-davis.com>, Robert Haas <robertmhaas@gmail.com>
Cc: pgsql-hackers@postgresql.org
Date: 2023-03-23T06:27:43Z
Lists: pgsql-hackers 
On 22.03.23 19:05, Jeff Davis wrote:
> On Tue, 2023-03-21 at 10:35 +0100, Peter Eisentraut wrote:
>> [PATCH v6 1/6] Support language tags in older ICU versions (53 and
>>    earlier).
>>
>> In pg_import_system_collations(), this is now redundant and can be
>> simplified:
>>
>> -               if (!pg_is_ascii(langtag) || !pg_is_ascii(iculocstr))
>> +               if (!pg_is_ascii(langtag) || !pg_is_ascii(langtag))
>>
>> icu_set_collation_attributes() needs more commenting about what is
>> going
>> on.  My guess is that uloc_canonicalize() converts from language tag
>> to
>> ICU locale ID, and then the existing logic to parse that apart would
>> apply.  Is that how it works?
> 
> Fixed the redundancy, added some comments, and committed 0001.

So, does uloc_canonicalize() always convert to ICU locale IDs?  What if 
you pass a language tag, does it convert it to ICU locale ID as well?

>> [PATCH v6 2/6] Wrap ICU ucol_open().

> So I'm inclined to just leave initdb alone in patches 0002 and 0003.

0002 and 0003 look ok to me now.

In 0002, the error "opening default collator is not supported", should 
that be an assert or an elog?  Is it reachable by the user?

You might want to check the declarations at the top of pg_ucol_open(). 
0003 reformats them after they were just added in 0002.  Maybe check 
that they are pgindent'ed in 0002 properly.

I don't understand patch 0004.  It seems to do two things, handle 
C/POSIX locale specifications and add an SQL-callable function.  Are 
those connected?

Commits

  1. Add missing source file to nls.mk

  2. Fix MSVC warning introduced in ea1db8ae70.

  3. Canonicalize ICU locale names to language tags.

  4. Validate ICU locales.

  5. initdb: emit message when using default ICU locale.

  6. initdb: replace check_icu_locale() with default_icu_locale().

  7. Fix error inconsistency in older ICU versions.

  8. Avoid potential UCollator leak for older ICU versions.

  9. pg_locale.c: change ereport() to elog().

  10. Handle the "und" locale in ICU versions 54 and older.

  11. Wrap ICU ucol_open().

  12. Support language tags in older ICU versions (53 and earlier).

  13. Improve support for UNICODE collation on older ICU

  14. Use ICU by default at initdb time.