Horton

Re: Built-in CTYPE provider

Noah Misch <noah@leadboat.com>

View thread View on postgr.es
From: Noah Misch <noah@leadboat.com>
To: Jeff Davis <pgsql@j-davis.com>
Cc: Peter Eisentraut <peter@eisentraut.org>, Daniel Verite <daniel@manitou-mail.org>, Robert Haas <robertmhaas@gmail.com>, Jeremy Schneider <schneider@ardentperf.com>, pgsql-hackers@postgresql.org
Date: 2024-06-29T22:08:57Z
Lists: pgsql-hackers 
On Wed, Mar 20, 2024 at 05:13:26PM -0700, Jeff Davis wrote:
> On Tue, 2024-03-19 at 13:41 +0100, Peter Eisentraut wrote:
> > * v25-0002-Support-C.UTF-8-locale-in-the-new-builtin-collat.patch
> > 
> > Looks ok.
> 
> Committed.

>       <varlistentry>
> +      <term><literal>pg_c_utf8</literal></term>
> +      <listitem>
> +       <para>
> +        This collation sorts by Unicode code point values rather than natural
> +        language order.  For the functions <function>lower</function>,
> +        <function>initcap</function>, and <function>upper</function>, it uses
> +        Unicode simple case mapping.  For pattern matching (including regular
> +        expressions), it uses the POSIX Compatible variant of Unicode <ulink
> +        url="https://www.unicode.org/reports/tr18/#Compatibility_Properties">Compatibility
> +        Properties</ulink>.  Behavior is efficient and stable within a
> +        <productname>Postgres</productname> major version.  This collation is
> +        only available for encoding <literal>UTF8</literal>.
> +       </para>
> +      </listitem>
> +     </varlistentry>

lower(), initcap(), upper(), and regexp_matches() are PROVOLATILE_IMMUTABLE.
Until now, we've delegated that responsibility to the user.  The user is
supposed to somehow never update libc or ICU in a way that changes outcomes
from these functions.  Now that postgresql.org is taking that responsibility
for builtin C.UTF-8, how should we govern it?  I think the above text and [1]
convey that we'll update the Unicode data between major versions, making
functions like lower() effectively STABLE.  Is that right?

(This thread had some discussion[2] that datcollversion/collversion won't
necessarily change when a major versions changes lower() behavior.)

[1] https://postgr.es/m/7089acb3ebac0c1682a79c8bc16803cf06896fb9.camel@j-davis.com
[2] https://postgr.es/m/5a1ecc40539f36cac5b27a62739a45a49785ca54.camel@j-davis.com

Commits

  1. Support PG_UNICODE_FAST locale in the builtin collation provider.

  2. Support Unicode full case mapping and conversion.

  3. Fix test failures when language environment is not UTF-8.

  4. Add unicode_strtitle() for Unicode Default Case Conversion.

  5. Use version for builtin collations.

  6. Fix convert_case(), introduced in 5c40364dd6.

  7. Inline basic UTF-8 functions.

  8. Support C.UTF-8 locale in the new builtin collation provider.

  9. Fix another warning, introduced by 846311051e.

  10. Address more review comments on commit 2d819a08a1.

  11. Fix unreachable code warning from commit 2d819a08a1.

  12. Introduce "builtin" collation provider.

  13. Catalog changes preparing for builtin collation provider.

  14. Unicode case mapping tables and functions.

  15. Add Unicode property tables.

  16. Documentation update for Standard Collations.

  17. Cleanup for unicode-update build target and test.

  18. Shrink Unicode category table.

  19. Make some error strings more generic

  20. pg_upgrade: copy locale and encoding information to new cluster.

  21. Update Unicode data to Unicode 15.0.0

  22. Create a new type category for "internal use" types.