Horton
Menu

Re: Order changes in PG16 since ICU introduction

Jeff Davis <pgsql@j-davis.com>

View thread
From: Jeff Davis <pgsql@j-davis.com>
To: Daniel Verite <daniel@manitou-mail.org>
Cc: Matthias van de Meent <boekewurm+postgres@gmail.com>, Andrew Gierth <andrew@tao11.riddles.org.uk>, Peter Eisentraut <peter.eisentraut@enterprisedb.com>, Sandro Santilli <strk@kbt.io>, Tom Lane <tgl@sss.pgh.pa.us>, Regina Obe <lr@pcorp.us>, pgsql-hackers@lists.postgresql.org
Date: 2023-05-21T01:42:55Z
Lists: pgsql-hackers 
On Fri, 2023-05-19 at 21:13 +0200, Daniel Verite wrote:
> ISTM that if we want to go that route, we need the make the minimum
> changes at the user interface level and not any deeper, so that when
> (locale="C" OR locale="POSIX") AND the provider has not been
> specified,
> then the command (initdb and create database) act as if the user had
> specified provider=libc.

If we special case locale=C, but do nothing for locale=fr_FR, then I'm
not sure we've solved the problem. Andrew Gierth raised the issue here,
which he called "maximally confusing":

https://postgr.es/m/874jp9f5jo.fsf@news-spur.riddles.org.uk

That's why I feel that we need to make locale apply to whatever the
provider is, not just when it happens to be C.

> > (3) Support iculocale=C in the ICU provider using the memcmp()
> > path.
> 
> > In other words, if provider=icu and iculocale=C, lc_collate_is_c()
> > and
> > lc_ctpye_is_c() would both return true.
> 
> ICU does not provide a locale that behaves like that, and it doesn't
> feel right to pretend it does. It feels like attacking the problem
> at the wrong level.

I agree that #3 feels slightly wrong, but I think it's still a viable
option until we have consensus on something better.

> > (4) Create a new "none" provider (which has no locale and always
> > memcmp
> > semantics), and automatically change the provider to "none" if
> > provider=icu and iculocale=C.
> 
> It still uses libc/C for character classification and case changing,
> so "no locale" is technically not true.

The provider affects callers that have a pg_locale_t, such as the SQL-
callable lower() function. For those callers, the "none" provider uses
pg_ascii_tolower(), etc., not libc. That's why I called it "none" --
it's using simple internal postgres implementations instead of a
provider.

For callers that don't have a pg_locale_t, they may call libc functions
directly and rely on the server environment. But in those cases,
there's no way to set a provider at all, it's just relying on the
server environment. There aren't many of these cases, and hopefully we
can eliminate the reliance on the server environment over time.

If I'm missing something, let me know what cases you have in mind.

Regards,
	Jeff Davis

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Introduce "builtin" collation provider.

  2. ICU: do not convert locale 'C' to 'en-US-u-va-posix'.

  3. initdb: change default --locale-provider back to libc.

  4. CREATE DATABASE: make LOCALE apply to all collation providers.

  5. Remove read-only server settings lc_collate and lc_ctype

  6. ICU: use uloc_getDefault() for initdb.

  7. Doc fixes for commit 1e16af8ab5.

  8. Doc: fix some rendering problems in the PDF docs build.

  9. Doc improvements for language tags and custom ICU collations.

  10. ICU: check for U_STRING_NOT_TERMINATED_WARNING.

  11. Reduce icu_validation_level default to WARNING.

  12. Revert "ICU: do not convert locale 'C' to 'en-US-u-va-posix'."

  13. Fix initdb --no-locale.

  14. Avoid character classification in regex escape parsing.