Horton

Re: Built-in CTYPE provider

Jeff Davis <pgsql@j-davis.com>

View thread View on postgr.es
From: Jeff Davis <pgsql@j-davis.com>
To: Jeremy Schneider <schneider@ardentperf.com>, Daniel Verite <daniel@manitou-mail.org>
Cc: Robert Haas <robertmhaas@gmail.com>, pgsql-hackers@postgresql.org
Date: 2024-01-09T22:55:44Z
Lists: pgsql-hackers 
On Mon, 2024-01-08 at 17:17 -0800, Jeremy Schneider wrote:

> I agree with merging the threads, even though it makes for a larger
> patch set. It would be great to get a unified "builtin" provider in
> place for the next major.

I believe that's possible and that this proposal is quite close (hoping
to get something in this 'fest). The tables I'm introducing have
exhaustive test coverage, so there's not a lot of risk there. And the
builtin provider itself is an optional feature, so it won't be
disruptive.

> 
> In the first list it seems that some callers might be influenced by a
> COLLATE clause or table definition while others always take the
> database
> default? It still seems a bit odd to me if different providers can be
> used for different parts of a single SQL.

Right, that can happen today, and my proposal doesn't change that.
Basically those are cases where the caller was never properly onboarded
to our collation system, like the ts_locale.c routines.

> Is there any reason we couldn't commit the minor cleanup (patch 0001)
> now? It's less than 200 lines and pretty straightforward.

Sure, I'll commit that fairly soon then.

> I wonder if, after a year of running the builtin provider in
> production,
> whether we might consider adding to the builtin provider a few
> locales
> with simple but more reasonable ordering for european and asian
> languages?

I won't rule that out completely, but there's a lot we would need to do
to get there. Even assuming we implement that perfectly, we'd need to
make sure it's a reasonable scope for Postgres as a project and that we
have more than one person willing to maintain it. Similar things have
been rejected before for similar reasons.

What I'm proposing for v17 is much simpler: basically some lookup
tables, which is just an extension of what we're already doing for
normalization.

> https://jeremyhussell.blogspot.com/2017/11/falsehoods-programmers-believe-about.html#main
> 
> Make sure to click the link to show the counterexamples and
> discussion,
> that's the best part.

Yes, it can be hard to reason about this stuff but I believe Unicode
provides a lot of good data and guidance to work from. If you think my
proposal relies on one of those assumptions let me know.

To the extent that I do rely on any of those assumptions, it's mostly
to match libc's "C.UTF-8" behavior.

Regards,
	Jeff Davis

Commits

  1. Support PG_UNICODE_FAST locale in the builtin collation provider.

  2. Support Unicode full case mapping and conversion.

  3. Fix test failures when language environment is not UTF-8.

  4. Add unicode_strtitle() for Unicode Default Case Conversion.

  5. Use version for builtin collations.

  6. Fix convert_case(), introduced in 5c40364dd6.

  7. Inline basic UTF-8 functions.

  8. Support C.UTF-8 locale in the new builtin collation provider.

  9. Fix another warning, introduced by 846311051e.

  10. Address more review comments on commit 2d819a08a1.

  11. Fix unreachable code warning from commit 2d819a08a1.

  12. Introduce "builtin" collation provider.

  13. Catalog changes preparing for builtin collation provider.

  14. Unicode case mapping tables and functions.

  15. Add Unicode property tables.

  16. Documentation update for Standard Collations.

  17. Cleanup for unicode-update build target and test.

  18. Shrink Unicode category table.

  19. Make some error strings more generic

  20. pg_upgrade: copy locale and encoding information to new cluster.

  21. Update Unicode data to Unicode 15.0.0

  22. Create a new type category for "internal use" types.