Horton

Re: Built-in CTYPE provider

Noah Misch <noah@leadboat.com>

View thread View on postgr.es
From: Noah Misch <noah@leadboat.com>
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Jeff Davis <pgsql@j-davis.com>, Laurenz Albe <laurenz.albe@cybertec.at>, Peter Eisentraut <peter@eisentraut.org>, Daniel Verite <daniel@manitou-mail.org>, Jeremy Schneider <schneider@ardentperf.com>, Robert Haas <robertmhaas@gmail.com>, pgsql-hackers@postgresql.org
Date: 2024-07-18T23:39:08Z
Lists: pgsql-hackers 
On Thu, Jul 18, 2024 at 01:03:31PM -0400, Tom Lane wrote:
> This whole discussion seems quite bizarre to me.  In the first
> place, it is certain that Unicode will continue to evolve, and
> I can't believe that we'd just freeze pg_c_utf8 on the current
> definition forever.  Whether the first change happens in v18
> or years later doesn't seem like a particularly critical point.
> 
> In the second place, I cannot understand why pg_c_utf8 is being
> held to a mutability standard that we have never applied to any
> other locale-related functionality --- and indeed could not do
> so, since in most cases that functionality has been buried in
> libraries we don't control.  It seems to me to be already a

With libc and ICU providers, packagers have a way to avoid locale-related
behavior changes.  That's the "mutability standard" I want pg_c_utf8 to join.
pg_c_utf8 is the one provider where packagers can't opt out[1] of annual
pg_upgrade-time index scan breakage on affected expression indexes.

> great step forward that with pg_c_utf8, at least we can guarantee
> that the behavior won't change without us knowing about it.
> Noah's desire to revert the feature makes the mutability situation
> strictly worse, because people will have to continue to rely on
> OS-provided functionality that can change at any time.

I see:
- one step forward:
  "string1 < string2" won't change, forever, regardless of packager choices
- one step backward:
  "string ~ '[[:alpha:]]'" will change at pg_upgrade time, regardless of packager choices

I think one's perspective on the relative importance of the step forward and
the step backward depends on the sort of packages one uses today.  Consider a
user of Debian packages with locale!=C, doing Debian upgrades and pg_upgrade.
For that user, pg_c_utf8 gets far less index corruption than an ICU locale.
The step forward is a great step forward _for this user_, and the step
backward is in the noise next to the step forward.

I'm with a different kind of packager.  I don't tolerate index scans returning
wrong answers.  To achieve that, my libc and ICU aren't changing collation
behavior.  I suspect my packages won't offer a newer ICU behavior until
PostgreSQL gets support for multiple ICU library versions per database.  (SQL
Server, DB2 and Oracle already do.  I agree we can't freeze forever.  The
multiple-versions feature gets more valuable each year.)  _For this_ kind of
package, the step forward is a no-op.  The step backward is the sole effect on
this kind of package.

How much does that pair of perspectives explain the contrast between my
"revert" and your "great step forward"?  We may continue to disagree on the
ultimate decision, but I hope I can make my position cease to appear bizarre
to you.

Thanks,
nm


[1] Daniel Verite said packagers could patch src/Makefile.global.in and run
"make -C src/common/unicode update-unicode".  Editing src/Makefile.global.in
is modifying PostgreSQL, not configuring a packager-facing option.

Commits

  1. Support PG_UNICODE_FAST locale in the builtin collation provider.

  2. Support Unicode full case mapping and conversion.

  3. Fix test failures when language environment is not UTF-8.

  4. Add unicode_strtitle() for Unicode Default Case Conversion.

  5. Use version for builtin collations.

  6. Fix convert_case(), introduced in 5c40364dd6.

  7. Inline basic UTF-8 functions.

  8. Support C.UTF-8 locale in the new builtin collation provider.

  9. Fix another warning, introduced by 846311051e.

  10. Address more review comments on commit 2d819a08a1.

  11. Fix unreachable code warning from commit 2d819a08a1.

  12. Introduce "builtin" collation provider.

  13. Catalog changes preparing for builtin collation provider.

  14. Unicode case mapping tables and functions.

  15. Add Unicode property tables.

  16. Documentation update for Standard Collations.

  17. Cleanup for unicode-update build target and test.

  18. Shrink Unicode category table.

  19. Make some error strings more generic

  20. pg_upgrade: copy locale and encoding information to new cluster.

  21. Update Unicode data to Unicode 15.0.0

  22. Create a new type category for "internal use" types.