Horton
Menu

Re: Order changes in PG16 since ICU introduction

Jeff Davis <pgsql@j-davis.com>

View thread
From: Jeff Davis <pgsql@j-davis.com>
To: Alexander Lakhin <exclusion@gmail.com>, Tom Lane <tgl@sss.pgh.pa.us>
Cc: Daniel Verite <daniel@manitou-mail.org>, Andrew Gierth <andrew@tao11.riddles.org.uk>, Peter Eisentraut <peter.eisentraut@enterprisedb.com>, Sandro Santilli <strk@kbt.io>, Regina Obe <lr@pcorp.us>, pgsql-hackers@lists.postgresql.org
Date: 2023-05-15T21:03:23Z
Lists: pgsql-hackers

Attachments

On Sat, 2023-05-13 at 13:00 +0300, Alexander Lakhin wrote:
> On the current master (after 455f948b0, and before f7faa9976, of
> course)
> I get an ASAN-detected failure with the following query:
> CREATE COLLATION col (provider = icu, locale = '123456789012');
> 

Thank you for the report!

ICU source specifically says:

  /**                                                                 
   * Useful constant for the maximum size of the language
     part of a locale ID.                                             
   * (including the terminating NULL).                                
   * @stable ICU 2.0                                                  
   */
  #define ULOC_LANG_CAPACITY 12

So the fact that it returning success without nul-terminating the
result is an ICU bug in my opinion, and I reported it here:

https://unicode-org.atlassian.net/browse/ICU-22394

Unfortunately that means we need to be a bit more paranoid and always
check for that warning, even if we have a preallocated buffer of the
"correct" size. It also means that both U_STRING_NOT_TERMINATED_WARNING
and U_BUFFER_OVERFLOW_ERROR will be user-facing errors (potentially
scary), unless we check for those errors each time and report specific
errors for them.

Another approach is to always check the length and loop using dynamic
allocation so that we never overflow (and forget about any static
buffers). That seems like overkill given that the problem case is
obviously invalid input; I think as long as we catch it and throw an
ERROR it's fine. But I can do this if others think it's worthwhile.

Patch attached. It just checks for the U_STRING_NOT_TERMINATED_WARNING
in a few places and turns it into an ERROR. It also cleans up the loop
for uloc_getLanguageTag() to check for U_STRING_NOT_TERMINATED_WARNING
rather than (U_SUCCESS(status) && len >= buflen).


-- 
Jeff Davis
PostgreSQL Contributor Team - AWS

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Introduce "builtin" collation provider.

  2. ICU: do not convert locale 'C' to 'en-US-u-va-posix'.

  3. initdb: change default --locale-provider back to libc.

  4. CREATE DATABASE: make LOCALE apply to all collation providers.

  5. Remove read-only server settings lc_collate and lc_ctype

  6. ICU: use uloc_getDefault() for initdb.

  7. Doc fixes for commit 1e16af8ab5.

  8. Doc: fix some rendering problems in the PDF docs build.

  9. Doc improvements for language tags and custom ICU collations.

  10. ICU: check for U_STRING_NOT_TERMINATED_WARNING.

  11. Reduce icu_validation_level default to WARNING.

  12. Revert "ICU: do not convert locale 'C' to 'en-US-u-va-posix'."

  13. Fix initdb --no-locale.

  14. Avoid character classification in regex escape parsing.