Horton

Re: Collation versioning

Juan José Santamaría Flecha <juanjo.santamaria@gmail.com>

View thread View on postgr.es
From: Juan José Santamaría Flecha <juanjo.santamaria@gmail.com>
To: Thomas Munro <thomas.munro@gmail.com>
Cc: David Rowley <dgrowleyml@gmail.com>, Tom Lane <tgl@sss.pgh.pa.us>, Julien Rouhaud <rjuju123@gmail.com>, Peter Eisentraut <peter.eisentraut@2ndquadrant.com>, Michael Paquier <michael@paquier.xyz>, Robert Haas <robertmhaas@gmail.com>, Laurenz Albe <laurenz.albe@cybertec.at>, Douglas Doole <dougdoole@gmail.com>, Christoph Berg <myon@debian.org>, Pg Hackers <pgsql-hackers@postgresql.org>
Date: 2020-11-04T07:44:15Z
Lists: pgsql-hackers

Attachments

On Tue, Nov 3, 2020 at 10:49 PM Thomas Munro <thomas.munro@gmail.com> wrote:

>
> So we have:
>
> 1.  Windows locale names, like "English_United States.1252".  Windows
> still returns these from setlocale(), so they finish up in datcollate,
> and yet some relevant APIs don't accept them, at least on some
> machines.
>
> 2.  BCP 47/RFC 5646 language tags, like "en-US".  Windows uses these
> in relevant new APIs, including the case in point.
>
> 3.  Unix-style (XPG?  ISO/IEC 15897?) locale names, like "en_US"
> ("language[_territory[.codeset]][@modifier]").  These are used for
> message catalogues.
>
> We have a VS2015+ way of converting from form 1 to form 2 (and thence
> 3 by s/-/_/), and an older way.  Unfortunately, the new way looks a
> little too fuzzy: if i'm reading it right, search_locale_enum() might
> stop on either "en" or "en-AU", given "English_Australia", depending
> on the search order, no?


No, that is not the case. "English" could match any locale if the
enumeration order was to be changed in the future, right now the order is a
given (Language, Location), but "English_Australia" can only match  "en-AU".

This may be fine for the purpose of looking
> up error messages with gettext() (where there is only one English
> language message catalogue, we haven't got around to translating our
> errors into 'strayan yet), but it doesn't seem like a good way to look
> up the collation version; for all I know, "en" variants might change
> independently (I doubt it in practice, but in theory it's wrong).  We
> want the same algorithm that Windows uses internally to resolve the
> old style name to a collation; in other words we probably want
> something more like the code path that they took away in VS2015 :-(.
>

 We could create a static table with the conversion based on what was
discussed for commit a169155, please find attached a spreadsheet with the
comparison. This would require maintenance as new LCIDs are released [1].

[1]
https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-lcid/70feba9f-294e-491e-b6eb-56532684c37f

Regards,

Juan José Santamaría

Commits

  1. Doc: Document known problem with Windows collation versions.

  2. Add collation versions for FreeBSD.

  3. Tolerate version lookup failure for old style Windows locale names.

  4. Track collation versions for indexes.

  5. Add pg_depend.refobjversion.

  6. Remove pg_collation.collversion.

  7. Fix the MSVC build for versions 2015 and later.

  8. Add collation versions for Windows.

  9. Implement type regcollation

  10. Use libc version as a collation version on glibc systems.

  11. Make type "name" collation-aware.