Re: Pre-proposal: unicode normalized text

Jeff Davis <pgsql@j-davis.com>

From: Jeff Davis <pgsql@j-davis.com>

To: Peter Eisentraut <peter@eisentraut.org>, Robert Haas <robertmhaas@gmail.com>

Cc: pgsql-hackers@postgresql.org

Date: 2023-10-17T03:32:19Z

Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

Try again to fix the MSVC build
- 151ffcf6d8c1 17.0 landed
Additional unicode primitive functions.
- a02b37fc0832 17.0 landed

Attachments

v3-0001-Additional-unicode-primitive-functions.patch (text/x-patch) patch v3-0001

On Wed, 2023-10-11 at 08:56 +0200, Peter Eisentraut wrote:
> We need to be careful about precise terminology.  "Valid" has a
> defined 
> meaning for Unicode.  A byte sequence can be valid or not as UTF-8. 
> But 
> a string containing unassigned code points is not not-"valid" as
> Unicode.

New patch attached, function name is "unicode_assigned".

I believe the patch has utility as-is, but I've been brainstorming a
few more ideas that could build on it:

* Add a per-database option to enforce only storing assigned unicode
code points.

* (More radical) Add a per-database option to normalize all text in
NFC.

* Do character classification in Unicode rather than relying on
glibc/ICU. This would affect regex character classes, etc., but not
affect upper/lower/initcap nor collation. I did some experiments and
the General Category doesn't change a lot: a total of 197 characters
changed their General Category since Unicode 6.0.0, and only 5 since
ICU 11.0.0. I'm not quite sure how to expose this, but it seems like a
nicer way to handle it than tying it into the collation provider.

Regards,
	Jeff Davis