Horton

Thread

Commits

  1. Use libc version as a collation version on glibc systems.

  1. Custom Glibc collation version strings under LOCPATH

    Thomas Munro <thomas.munro@gmail.com> — 2025-06-04T04:03:27Z

    Hi,

One way to move to a newer glibc-based Linux distribution but keep the
locales working the same* without keeping the associated zombie C code
alive is to find the source system's collation definition source
files, compile them with the localedef on the target system and point
to the top-level directory with the environment variable LOCPATH.

That runs directly into the naivity of commit d5ac14f9's
gnu_get_libc_version() kludge.  So here's a patch that allows a brave
user of that recompilation technique to drop a custom version string
into a file called one of:

      * $LOCPATH/<collcollate>/LC_COLLATE.version
      * $LOCPATH/<collcollate>/version
      * $LOCPATH/LC_COLLATE.version
      * $LOCPATH/version

This way you can make your custom locales' reported version agree with
wherever they came from to skip those mismatch warnings, at whichever
granularity suits you.  Or you can design some other scheme for
labeling versions.  The attached POC shows this working, though it
lacks documentation for now as I wanted to float the general idea
first.

My preference would be for a tool-supported way for locale components
to report their own version with a new API[1], and I hope that someone
might eventually consider writing and proposing a patch to glibc for
that.  But in the meantime, I figured that users willing to compile
their own locale definitions for PostgreSQL's benefit might want to
drop their own version string into a text file.  The patch has no
effect otherwise, except for a few rare and harmless open() -> ENOENT
system calls if you have defined LOCPATH without supplying a custom
version file.

Returning gnu_get_libc_version() when you set LOCPATH is arguably a
bug and should at the very least be suppressed, I think.

*Of course you have to make sure you know what you're doing.  For
example we learned on this list of some tricky edge cases, mainly
around the treatment of Unicode-order sequences for eg C.UTF-8 which
began as buggy local patches in some distros' glibc C code, but at
least that case has been removed from our problem space by the new
built-in provider.  I'm interested in hearing about other concrete
examples of the locale-recompilation technique failing to be perfect,
and getting to the bottom of them; I have yet to hear of a real world
system that fails amcheck when using locale definitions ported in this
way.

[1] https://www.mail-archive.com/austin-group-l@opengroup.org/msg12849.html

  2. Re: Custom Glibc collation version strings under LOCPATH

    Peter Eisentraut <peter@eisentraut.org> — 2025-06-04T09:17:46Z

    On 04.06.25 06:03, Thomas Munro wrote:
> One way to move to a newer glibc-based Linux distribution but keep the
> locales working the same* without keeping the associated zombie C code
> alive is to find the source system's collation definition source
> files, compile them with the localedef on the target system and point
> to the top-level directory with the environment variable LOCPATH.
> 
> That runs directly into the naivity of commit d5ac14f9's
> gnu_get_libc_version() kludge.  So here's a patch that allows a brave
> user of that recompilation technique to drop a custom version string
> into a file called one of:
> 
>        * $LOCPATH/<collcollate>/LC_COLLATE.version
>        * $LOCPATH/<collcollate>/version
>        * $LOCPATH/LC_COLLATE.version
>        * $LOCPATH/version

Nice idea.

The patch looks mostly straightforward.

I wonder why you want to capture LOCPATH early in main.c.  It seems 
sufficient to look it up when needed?

  3. Re: Custom Glibc collation version strings under LOCPATH

    Thomas Munro <thomas.munro@gmail.com> — 2025-06-04T10:39:41Z

    On Wed, Jun 4, 2025 at 9:17 PM Peter Eisentraut <peter@eisentraut.org> wrote:
> I wonder why you want to capture LOCPATH early in main.c.  It seems
> sufficient to look it up when needed?

Right, it is setenv() that we're trying to avoid.  Updated.

  4. Re: Custom Glibc collation version strings under LOCPATH

    Joe Conway <mail@joeconway.com> — 2025-06-04T13:52:57Z

    On 6/4/25 00:03, Thomas Munro wrote:
> One way to move to a newer glibc-based Linux distribution but keep the
> locales working the same* without keeping the associated zombie C code
> alive is to find the source system's collation definition source
> files, compile them with the localedef on the target system and point
> to the top-level directory with the environment variable LOCPATH.

I don't think this works in all cases because I have seen where sorting 
was affected by C code rather than than data changes.

-- 
Joe Conway
PostgreSQL Contributors Team
Amazon Web Services: https://aws.amazon.com

  5. Re: Custom Glibc collation version strings under LOCPATH

    Joe Conway <mail@joeconway.com> — 2025-06-04T15:44:36Z

    On 6/4/25 09:52, Joe Conway wrote:
> On 6/4/25 00:03, Thomas Munro wrote:
>> One way to move to a newer glibc-based Linux distribution but keep the
>> locales working the same* without keeping the associated zombie C code
>> alive is to find the source system's collation definition source
>> files, compile them with the localedef on the target system and point
>> to the top-level directory with the environment variable LOCPATH.
> 
> I don't think this works in all cases because I have seen where sorting
> was affected by C code rather than than data changes.

Sorry I missed this part:

>> I'm interested in hearing about other concrete
>> examples of the locale-recompilation technique failing to be perfect,
>> and getting to the bottom of them; I have yet to hear of a real world
>> system that fails amcheck when using locale definitions ported in this
>> way.

If you go from anything pre-glibc-2.21 to post-glibc-2.21 I think you 
will find that even with the same data files you get a different sort. 
The same patch that caused the performance regression [1] (still present 
in up to date glibc) also cause changes in sort order via C code alone.


[1] https://sourceware.org/bugzilla/show_bug.cgi?id=18441

-- 
Joe Conway
PostgreSQL Contributors Team
Amazon Web Services: https://aws.amazon.com

  6. Re: Custom Glibc collation version strings under LOCPATH

    Thomas Munro <thomas.munro@gmail.com> — 2025-06-04T23:35:23Z

    On Thu, Jun 5, 2025 at 3:44 AM Joe Conway <mail@joeconway.com> wrote:
> On 6/4/25 09:52, Joe Conway wrote:
> > On 6/4/25 00:03, Thomas Munro wrote:
> >> I'm interested in hearing about other concrete
> >> examples of the locale-recompilation technique failing to be perfect,
> >> and getting to the bottom of them; I have yet to hear of a real world
> >> system that fails amcheck when using locale definitions ported in this
> >> way.
>
> If you go from anything pre-glibc-2.21 to post-glibc-2.21 I think you
> will find that even with the same data files you get a different sort.
> The same patch that caused the performance regression [1] (still present
> in up to date glibc) also cause changes in sort order via C code alone.

Will try.  And BTW I fully understand that your work on running parts
of pinned old glibc libraries is a bug-perfect solution to this.  But
I also want to explore other trade-off positions, for users who don't
want to run unmaintained C code.  In exchange for that paranoia you
have C code changes, intentional or unintentional, and I'd really like
to understand them better...  One thing that is definitely out of the
question is moving the compiled LC_COLLATE files between glibc
versions (the binary format clearly changes, sometimes it apparently
work, sometimes it doesn't at all).  That leads to the idea of
recompiling with localedef.  The source formats are standardised by
POSIX and *should* have the same meaning to any system, so now maybe
we're only talking about bugs (in theory, you should even be able to
move the source between unrelated Unixen, but I only care about glibc
here, and I have no doubt that there are extensions and quirks so
reality may fail to live up to the theory completely).  I've
personally analysed only one such case and chased it all the way down,
which is the support for strict codepoint ordering and the non-strict
local fudges that Debian et al shipped in some version range, so we
can't even really blame it on glibc, and yet it is/was in the wild so
we can't ignore it (thanks to Jeff for making that one irrelevant).
Finding more cases probably involves running something a little like
Jeremy's torture tests across a huge gallery of versions and
combinations of cross-version recompiled definitions.  Or something
like that...

  7. Re: Custom Glibc collation version strings under LOCPATH

    Joe Conway <mail@joeconway.com> — 2025-06-05T01:00:26Z

    On 6/4/25 19:35, Thomas Munro wrote:
> On Thu, Jun 5, 2025 at 3:44 AM Joe Conway <mail@joeconway.com> wrote:
>> If you go from anything pre-glibc-2.21 to post-glibc-2.21 I think you
>> will find that even with the same data files you get a different sort.
>> The same patch that caused the performance regression [1] (still present
>> in up to date glibc) also cause changes in sort order via C code alone.

> Finding more cases probably involves running something a little like
> Jeremy's torture tests across a huge gallery of versions and
> combinations of cross-version recompiled definitions.  Or something
> like that...

Sounds like great fun!

;-)

-- 
Joe Conway
PostgreSQL Contributors Team
Amazon Web Services: https://aws.amazon.com