Horton

Re: ICU locale validation / canonicalization

Peter Eisentraut <peter.eisentraut@enterprisedb.com>

View thread View on postgr.es
From: Peter Eisentraut <peter.eisentraut@enterprisedb.com>
To: Jeff Davis <pgsql@j-davis.com>, Robert Haas <robertmhaas@gmail.com>
Cc: pgsql-hackers@postgresql.org
Date: 2023-02-20T14:46:23Z
Lists: pgsql-hackers 
On 10.02.23 18:53, Jeff Davis wrote:
> To represent ICU locale strings in the catalog consistently, we have
> two choices, which as far as I can tell are equivalent:
> 
> 1. ICU format Locale IDs. These are more readable, and still specified
> (albeit non-standard).
> 
> 2. BCP47 language tags. These are standardized, there's better
> validation with "strict" mode, and we are already using them.
> 
> Honestly I don't think it's hugely important which one we pick. But
> being consistent is important, so we need to pick one, and BCP 47 seems
> like the better option to me.

I found some discussion about this from when ICU support was first 
added.  See this message as a starting point: 
https://www.postgresql.org/message-id/flat/5291804b-169e-3ba9-fdaf-fae8e7d2d959%402ndquadrant.com#96acb7eb9299c2ca64dbabcf58e11a90

There isn't much detail there, but the discussion and the current code 
seem pretty convinced that

a) BCP47 tags are preferred, and
b) They don't work with ICU versions before 54.

I can't locate the source for claim b) anymore.  However, it seems 
pretty clear that there is some cutoff, even if it isn't exactly 54.

I would support transitioning this forward somehow, but we would need to 
know exactly what the impact would be.

Commits

  1. Add missing source file to nls.mk

  2. Fix MSVC warning introduced in ea1db8ae70.

  3. Canonicalize ICU locale names to language tags.

  4. Validate ICU locales.

  5. initdb: emit message when using default ICU locale.

  6. initdb: replace check_icu_locale() with default_icu_locale().

  7. Fix error inconsistency in older ICU versions.

  8. Avoid potential UCollator leak for older ICU versions.

  9. pg_locale.c: change ereport() to elog().

  10. Handle the "und" locale in ICU versions 54 and older.

  11. Wrap ICU ucol_open().

  12. Support language tags in older ICU versions (53 and earlier).

  13. Improve support for UNICODE collation on older ICU

  14. Use ICU by default at initdb time.