Horton

Thread

Commits

  1. Fix INITCAP() word boundaries for PG_UNICODE_FAST.

  2. Support PG_UNICODE_FAST locale in the builtin collation provider.

  3. Support Unicode full case mapping and conversion.

  1. Unicode full case mapping: PG_UNICODE_FAST, and standard-compliant UCS_BASIC

    Jeff Davis <pgsql@j-davis.com> — 2024-12-11T23:52:44Z

    Right now, the UCS_BASIC simply uses the "C" locale.

The standard for LOWER() and UPPER() define the behavior in terms of
Unicode. The specific requirement that "ß" shall uppercase to "SS" also
seems to imply Full Case Mapping.

Attached is a series of patches to implement full case mapping as the
locale PG_UNICODE_FAST.

The last patch in the series also changes UCS_BASIC to use that locale,
bringing it into compliance with the standard. However, this will break
existing users of the UCS_BASIC collation, so we may not want this at
all. If we do want it, we should be clear that affected expression
indexes using UCS_BASIC should be REINDEXed, or that users should
change the collation to "C" to get the previous behavior.

While Postgres uses the collation to define the behavior of LOWER() and
UPPER(), the standard doesn't mention it, so the behavior of those
functions is independent of the collation (according to the standard).
It wouldn't make sense to force the standard behavior on all the
collations, but it could make sense for the standard-defined UCS_BASIC
collation.

Regards,
	Jeff Davis

  2. Re: Unicode full case mapping: PG_UNICODE_FAST, and standard-compliant UCS_BASIC

    Jeff Davis <pgsql@j-davis.com> — 2024-12-16T20:49:14Z

    On Wed, 2024-12-11 at 15:52 -0800, Jeff Davis wrote:
> Attached is a series of patches to implement full case mapping as the
> locale PG_UNICODE_FAST.

Rebased and attached.

I'm having a doubt about the correctness, though. There's a statement
in SpecialCasing.txt:

  # IMPORTANT-when iota-subscript (0345) is uppercased or titlecased,
  #  the result will be incorrect unless the iota-subscript is moved to
the end
  #  of any sequence of combining marks. Otherwise, the accents will go
on the capital iota.
  #  This process can be achieved by first transforming the text to NFC
before casing.
  #  E.g. <alpha><iota_subscript><acute> is uppercased to
<ALPHA><acute><IOTA>

That requirement doesn't appear to exist in the Unicode standard
itself, nor is it implied from the mappings in the data files. And
based on the description, it only matters if the input is not
normalized in NFC (I believe NFD is also fine, because the combining
class of U+0345 is 240, higher than any other class). Furthermore, it
appears that the ICU root collation doesn't bother trying to implement
that requirement for non-normalized input.

There is a related requirement for caseless matching in the Unicode
standard[1] that requires normalization iff the source includes U+0345
(or any character which has U+0345 in its decomposition). The ICU
u_strFoldCase() function doesn't do that, either.

I'm not sure how important these requirements are, but I'm bringing
them up now because we can't change them after release, and they may be
technically incorrect for non-normalized input.

Regards,
	Jeff Davis

[1] Unicode 16.0 section 3.13.5 rule D145
https://www.unicode.org/versions/Unicode16.0.0/UnicodeStandard-16.0.pdf#G34145

  3. Re: Unicode full case mapping: PG_UNICODE_FAST, and standard-compliant UCS_BASIC

    Jeff Davis <pgsql@j-davis.com> — 2025-01-11T00:36:09Z

    On Mon, 2024-12-16 at 12:49 -0800, Jeff Davis wrote:
> On Wed, 2024-12-11 at 15:52 -0800, Jeff Davis wrote:
> > Attached is a series of patches to implement full case mapping as
> > the
> > locale PG_UNICODE_FAST.
> 
> Rebased and attached.

Rebased and attached v3.

Regards,
	Jeff Davis

  4. Re: Unicode full case mapping: PG_UNICODE_FAST, and standard-compliant UCS_BASIC

    Jeff Davis <pgsql@j-davis.com> — 2025-01-15T21:21:34Z

    On Fri, 2025-01-10 at 16:36 -0800, Jeff Davis wrote:
> On Mon, 2024-12-16 at 12:49 -0800, Jeff Davis wrote:
> > On Wed, 2024-12-11 at 15:52 -0800, Jeff Davis wrote:
> > > Attached is a series of patches to implement full case mapping as
> > > the
> > > locale PG_UNICODE_FAST.
> > 
> > Rebased and attached.
> 
> Rebased and attached v3.

I plan to commit 0001 and 0002 soon. There seems to be general
agreement that we want full case mapping[1], and we can potentially use
the infrastructure to extend the additional case variants into pattern
matching[2].

0003 requires more discussion.

Regards,
	Jeff Davis

[1]https://www.postgresql.org/message-id/27bb0e52-801d-4f73-a0a4-02cfdd4a9ada@eisentraut.org
[2]
https://www.postgresql.org/message-id/c10ed44c7e5dcbb7b4597889f02d029298f0c919.camel@j-davis.com

  5. Re: Unicode full case mapping: PG_UNICODE_FAST, and standard-compliant UCS_BASIC

    Jeff Davis <pgsql@j-davis.com> — 2025-01-18T00:06:20Z

    On Wed, 2025-01-15 at 13:21 -0800, Jeff Davis wrote:
> I plan to commit 0001 and 0002 soon.

Committed 0001 and 0002.

Upon reviewing the discussion threads, I removed the Unicode "adjust to
Cased" behavior when titlecasing. As Peter pointed out[1], it doesn't
match the documentation or expectations for INITCAP().

I also expanded the C tests a lot in 0001 so that it compares
exhaustively against ICU for single-codepoint strings, and also added
more multi-codepoint test strings.

I don't plan to commit 0003 in v18, so I'm considering this series to
be done for now. We can revisit whether UCS_BASIC should change
behavior in 19.

Regards,
	Jeff Davis

  6. Re: Unicode full case mapping: PG_UNICODE_FAST, and standard-compliant UCS_BASIC

    Jeff Davis <pgsql@j-davis.com> — 2025-01-18T00:40:07Z

    On Fri, 2025-01-17 at 16:06 -0800, Jeff Davis wrote:
> Upon reviewing the discussion threads, I removed the Unicode "adjust
> to
> Cased" behavior when titlecasing. As Peter pointed out[1], it doesn't
> match the documentation or expectations for INITCAP().

Forgot to add a link to the discussion. Here are some relevant
messages:

https://www.postgresql.org/message-id/4c9eea58-08a1-4629-a004-439e2cf12de8%40eisentraut.org
https://www.postgresql.org/message-id/610d7f1b-c68c-4eb8-a03d-1515da304c58%40manitou-mail.org

Regards,
	Jeff Davis

  7. Re: Unicode full case mapping: PG_UNICODE_FAST, and standard-compliant UCS_BASIC

    Noah Misch <noah@leadboat.com> — 2025-04-17T13:58:41Z

    On Fri, Jan 17, 2025 at 04:06:20PM -0800, Jeff Davis wrote:
> Committed 0001 and 0002.

> Upon reviewing the discussion threads, I removed the Unicode "adjust to
> Cased" behavior when titlecasing. As Peter pointed out[1], it doesn't
> match the documentation or expectations for INITCAP().

While commit d3d0983 changed most of the non-test pg_u_*() "bool posix"
arguments, it left a pg_u_isalnum(u, true) in strtitle_builtin() subroutine
initcap_wbnext().  The above paragraph may or may not be saying that's
intentional.  Example of the consequence at non-ASCII decimal digits:

SELECT
	str,
	re,
	regexp_count(str COLLATE pg_c_utf8, re) AS count_c_utf8,
	regexp_count(str COLLATE pg_unicode_fast, re) AS count_unicode_fast,
	regexp_count(str COLLATE unicode, re) AS count_unicode,
	initcap(str COLLATE pg_c_utf8) AS initcap_c_utf8,
	initcap(str COLLATE pg_unicode_fast) AS initcap_unicode_fast,
	initcap(str COLLATE unicode) AS initcap_unicode
FROM
	(VALUES (U&'foo\0661bar baz')) AS str_t(str),
	(VALUES ('[[:digit:]]')) AS re_t(re)
ORDER BY 1, 2;

str                  │ foo١bar baz
re                   │ [[:digit:]]
count_c_utf8         │ 0
count_unicode_fast   │ 1
count_unicode        │ 1
initcap_c_utf8       │ Foo١Bar Baz
initcap_unicode_fast │ Foo١Bar Baz
initcap_unicode      │ Foo١bar Baz

Should initcap_wbnext() pass in a locale-dependent "bool posix" argument like
the others calls the commit changed?  Related message from the development of
pg_c_utf8, which you shared downthread:
https://www.postgresql.org/message-id/610d7f1b-c68c-4eb8-a03d-1515da304c58%40manitou-mail.org


Long-term, pg_u_isword() should have a "bool posix" argument.  Currently, only
tests call that function.  If it got a non-test caller,
https://www.unicode.org/reports/tr18/#word would have pg_u_isword() follow the
choice of posix compatibility like pg_u_isalnum() does.

  8. Re: Unicode full case mapping: PG_UNICODE_FAST, and standard-compliant UCS_BASIC

    Jeff Davis <pgsql@j-davis.com> — 2025-04-19T19:30:57Z

    On Thu, 2025-04-17 at 06:58 -0700, Noah Misch wrote:
> Should initcap_wbnext() pass in a locale-dependent "bool posix"
> argument like
> the others calls the commit changed?

Yes, I believe you are correct. Patch and tests attached.

> Long-term, pg_u_isword() should have a "bool posix" argument. 
> Currently, only
> tests call that function.  If it got a non-test caller,
> https://www.unicode.org/reports/tr18/#word would have pg_u_isword()
> follow the
> choice of posix compatibility like pg_u_isalnum() does.

I based those functions on:

https://www.unicode.org/reports/tr18/#Compatibility_Properties

and the "word" class does not have a POSIX variant. But Postgres has
two documented definitions for "word" characters:

 * for regexes, alnum + "_"
 * for INITCAP(), just alnum

and the above definition doesn't match up with either one, which is why
we don't use it.

ICU INITCAP() uses the ICU definition of word boundaries, so doesn't
match our documentation.

We could adjust our documentation to allow for provider-dependent
definitions of word characters, which might be a good idea. But that
still doesn't quite capture ICU's more complex definition of word
boundaries.

Or, we could remove those unused functions for now, and figure out if
there's a reason to add them back later. They are probably adding more
confusion than anything.

Regards,
	Jeff Davis

  9. Re: Unicode full case mapping: PG_UNICODE_FAST, and standard-compliant UCS_BASIC

    Noah Misch <noah@leadboat.com> — 2025-04-20T12:53:22Z

    On Sat, Apr 19, 2025 at 12:30:57PM -0700, Jeff Davis wrote:
> On Thu, 2025-04-17 at 06:58 -0700, Noah Misch wrote:
> > Should initcap_wbnext() pass in a locale-dependent "bool posix"
> > argument like
> > the others calls the commit changed?
> 
> Yes, I believe you are correct. Patch and tests attached.

That patch is ready for commit.

> > Long-term, pg_u_isword() should have a "bool posix" argument. 
> > Currently, only
> > tests call that function.  If it got a non-test caller,
> > https://www.unicode.org/reports/tr18/#word would have pg_u_isword()
> > follow the
> > choice of posix compatibility like pg_u_isalnum() does.
> 
> I based those functions on:
> 
> https://www.unicode.org/reports/tr18/#Compatibility_Properties
> 
> and the "word" class does not have a POSIX variant.

I missed that distinction.  I withdraw this part.