Horton

Re: ICU locale validation / canonicalization

Peter Eisentraut <peter.eisentraut@enterprisedb.com>

View thread View on postgr.es
From: Peter Eisentraut <peter.eisentraut@enterprisedb.com>
To: Jeff Davis <pgsql@j-davis.com>, Robert Haas <robertmhaas@gmail.com>
Cc: pgsql-hackers@postgresql.org
Date: 2023-03-13T07:25:46Z
Lists: pgsql-hackers 
On 09.03.23 21:17, Jeff Davis wrote:
>> Personally, I'm not on board with this behavior:
>>
>> => CREATE COLLATION test (provider = icu, locale =
>> 'de@collation=phonebook');
>> NOTICE:  00000: using language tag "de-u-co-phonebk" for locale
>> "de@collation=phonebook"
>>
>> I mean, maybe that is a thing we want to do somehow sometime, to
>> migrate
>> people to the "new" spellings, but the old ones aren't wrong.
> 
> I see what you mean; I'm not sure the best thing to do here. We are
> adjusting the string passed by the user, and it feels like some users
> might want to know that. It's a NOTICE, not a WARNING, so it's not
> meant to imply that it's wrong.

For clarification, I wasn't complaining about the notice, but about the 
automatic conversion from old-style ICU locale ID to language tag.

>> It also doesn't appear to address
>> how to handle ICU before version 54.
> 
> Do you have a specific concern here?

What we had discussed a while ago in one of these threads is that ICU 
before version 54 do not support keyword lists, and we have custom code 
to do that parsing ourselves, but we don't have code to do the same for 
language tags.  Therefore, if I understand this right, if we 
automatically convert ICU locale IDs to language tags, as shown above, 
then we break support for such locales in those older ICU versions.

Commits

  1. Add missing source file to nls.mk

  2. Fix MSVC warning introduced in ea1db8ae70.

  3. Canonicalize ICU locale names to language tags.

  4. Validate ICU locales.

  5. initdb: emit message when using default ICU locale.

  6. initdb: replace check_icu_locale() with default_icu_locale().

  7. Fix error inconsistency in older ICU versions.

  8. Avoid potential UCollator leak for older ICU versions.

  9. pg_locale.c: change ereport() to elog().

  10. Handle the "und" locale in ICU versions 54 and older.

  11. Wrap ICU ucol_open().

  12. Support language tags in older ICU versions (53 and earlier).

  13. Improve support for UNICODE collation on older ICU

  14. Use ICU by default at initdb time.