Horton
Menu

Re: ICU for global collation

Peter Eisentraut <peter.eisentraut@2ndquadrant.com>

View thread
From: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
To: Daniel Verite <daniel@manitou-mail.org>
Cc: pgsql-hackers <pgsql-hackers@postgresql.org>
Date: 2019-11-06T10:09:57Z
Lists: pgsql-hackers 
On 2019-11-01 19:18, Daniel Verite wrote:
> Even if the FTS code is improved in that matter, any extension code
> with libc functions depending on LC_CTYPE is still going to be
> potentially problematic. In particular when it happens to be set
> to a different encoding than the database.

I think the answer here is that extension code must not do that, at 
least in ways that potentially interact with other parts of the 
(collation-aware) database system.  For example, libc and ICU might have 
different opinions about what is a letter, because of different versions 
of Unicode data in use.  That would then affect tokenization etc. in 
text search and elsewhere.  That's why things like isalpha have to go 
though ICU instead, if that is the collation provider in a particular 
context.

> Couldn't we simply invent per-database GUC options, as in
> ALTER DATABASE myicudb SET libc_lc_ctype TO 'value';
> ALTER DATABASE myicudb SET libc_lc_collate TO 'value';
> 
> where libc_lc_ctype/libc_lc_collate would specifically set
> the values in the LC_CTYPE and LC_COLLATE environment vars
> of any backend serving the corresponding database"?

We could do that as a transition measure to support extensions like you 
mention above.  But our own internal code should not have to rely on that.

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Add CHECK_FOR_INTERRUPTS while restoring changes during decoding.

  2. Improve ICU option handling in CREATE DATABASE

  3. Don't allow creation of database with ICU locale with unsupported encoding

  4. Make locale option behavior more consistent

  5. pg_dump: Dump colliculocale

  6. Remove further unwanted linker flags from perl_embed_ldflags

  7. Doc: document possible need to raise kernel's somaxconn limit.

  8. Reduce warnings with -Wshadow=compatible-local builds

  9. Remove redundant spaces in _outA_Expr() output

  10. Fix outdated --help message for postgres -f

  11. pg_upgrade: Fix version comparison for global ICU support

  12. Silence -Wmaybe-uninitialized compiler warning in dbcommands.c.

  13. Add option to use ICU as global locale provider

  14. DefineCollation() code cleanup

  15. Change collate and ctype fields to type text

  16. Call pg_newlocale_from_collation() also with default collation

  17. Use libc version as a collation version on glibc systems.

  18. Silence -Wmaybe-uninitialized compiler warnings in dbcommands.c.

  19. Make LC_COLLATE and LC_CTYPE database-level settings. Collation and