Horton
Menu

Re: ICU for global collation

Marina Polyakova <m.polyakova@postgrespro.ru>

View thread
From: Marina Polyakova <m.polyakova@postgrespro.ru>
To: Peter Eisentraut <peter.eisentraut@enterprisedb.com>
Cc: Kyotaro Horiguchi <horikyota.ntt@gmail.com>, pgsql-hackers@postgresql.org, pryzby@telsasoft.com, rjuju123@gmail.com, daniel@manitou-mail.org, AndrewBille@gmail.com, michael@paquier.xyz
Date: 2022-09-22T18:06:07Z
Lists: pgsql-hackers

Attachments

On 2022-09-21 17:53, Peter Eisentraut wrote:
> Committed with that test, thanks.  I think that covers all the ICU
> issues you reported for PG15 for now?

I thought about the order of the ICU checks - if it is ok to check that 
the selected encoding is supported by ICU after printing all the locale 
& encoding information, why not to move almost all the ICU checks 
here?..

Examples of the work of the attached patch:

1. ICU locale vs supported encoding:

1.1.

$ initdb --encoding sql-ascii --locale-provider icu hoge
...
initdb: error: encoding mismatch
initdb: detail: The encoding you selected (SQL_ASCII) is not supported 
with the ICU provider.
initdb: hint: Rerun initdb and either do not specify an encoding 
explicitly, or choose a matching combination.

1.2. (like before)

$ initdb --encoding sql-ascii --icu-locale en-US hoge
initdb: error: --icu-locale cannot be specified unless locale provider 
"icu" is chosen

$ createdb --encoding sql-ascii --icu-locale en-US hoge
createdb: error: database creation failed: ERROR:  ICU locale cannot be 
specified unless locale provider is ICU

2. For builds without ICU:

2.1.

$ initdb --locale-provider icu hoge
...
initdb: error: ICU is not supported in this build

$ createdb --locale-provider icu hoge
createdb: error: database creation failed: ERROR:  ICU is not supported 
in this build

2.2. (like before)

$ initdb --icu-locale en-US hoge
initdb: error: --icu-locale cannot be specified unless locale provider 
"icu" is chosen

$ createdb --icu-locale en-US hoge
createdb: error: database creation failed: ERROR:  ICU locale cannot be 
specified unless locale provider is ICU

2.3.

$ createdb --locale-provider icu --icu-locale en-US --encoding sql-ascii 
hoge
createdb: error: database creation failed: ERROR:  ICU is not supported 
in this build

4. About errors in initdb:

4.1. If icu_locale is not specified, but it is required, then we get 
this:

$ initdb --locale-provider icu hoge
The files belonging to this database system will be owned by user 
"marina".
This user must also own the server process.

The database cluster will be initialized with this locale configuration:
   provider:    icu
   LC_COLLATE:  en_US.UTF-8
   LC_CTYPE:    en_US.UTF-8
   LC_MESSAGES: en_US.UTF-8
   LC_MONETARY: ru_RU.UTF-8
   LC_NUMERIC:  ru_RU.UTF-8
   LC_TIME:     ru_RU.UTF-8
The default database encoding has been set to "UTF8".
initdb: error: ICU locale must be specified

Almost the same if ICU is not supported in this build:

$ initdb --locale-provider icu hoge
The files belonging to this database system will be owned by user 
"marina".
This user must also own the server process.

The database cluster will be initialized with this locale configuration:
   provider:    icu
   LC_COLLATE:  en_US.UTF-8
   LC_CTYPE:    en_US.UTF-8
   LC_MESSAGES: en_US.UTF-8
   LC_MONETARY: ru_RU.UTF-8
   LC_NUMERIC:  ru_RU.UTF-8
   LC_TIME:     ru_RU.UTF-8
The default database encoding has been set to "UTF8".
initdb: error: ICU is not supported in this build

4.2. If icu_locale is specified for the wrong provider, the error will 
be at the beginning of the program start as before:

$ initdb --icu-locale en-US hoge
initdb: error: --icu-locale cannot be specified unless locale provider 
"icu" is chosen

-- 
Marina Polyakova
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Add CHECK_FOR_INTERRUPTS while restoring changes during decoding.

  2. Improve ICU option handling in CREATE DATABASE

  3. Don't allow creation of database with ICU locale with unsupported encoding

  4. Make locale option behavior more consistent

  5. pg_dump: Dump colliculocale

  6. Remove further unwanted linker flags from perl_embed_ldflags

  7. Doc: document possible need to raise kernel's somaxconn limit.

  8. Reduce warnings with -Wshadow=compatible-local builds

  9. Remove redundant spaces in _outA_Expr() output

  10. Fix outdated --help message for postgres -f

  11. pg_upgrade: Fix version comparison for global ICU support

  12. Silence -Wmaybe-uninitialized compiler warning in dbcommands.c.

  13. Add option to use ICU as global locale provider

  14. DefineCollation() code cleanup

  15. Change collate and ctype fields to type text

  16. Call pg_newlocale_from_collation() also with default collation

  17. Use libc version as a collation version on glibc systems.

  18. Silence -Wmaybe-uninitialized compiler warnings in dbcommands.c.

  19. Make LC_COLLATE and LC_CTYPE database-level settings. Collation and