BUG #15548: Unaccent does not remove combining diacritical characters
The Post Office <noreply@postgresql.org>
From: PG Bug reporting form <noreply@postgresql.org>
To: pgsql-bugs@lists.postgresql.org
Cc: hugh@whtc.ca
Date: 2018-12-12T20:00:45Z
Lists: pgsql-bugs, pgsql-hackers
The following bug has been logged on the website: Bug reference: 15548 Logged by: Hugh Ranalli Email address: hugh@whtc.ca PostgreSQL version: 11.1 Operating system: Ubuntu 18.04 Description: Apparently Unicode has two ways of accenting a character: as a separate code point, which represents the base character and the accent, or as a "combining diacritical mark" (https://en.wikipedia.org/wiki/Combining_Diacritical_Marks), in which case the mark applies itself to the preceding character. For example, A followed by U+0300 displays À. However, unaccent is not removing these accents. SELECT unaccent(U&'A\0300'); should result in 'A', but instead results in 'À.' I'm running PostgreSQL 11.1, installed from the PostgreSQL repositories. I've read bug report #13440, and have tried with both the installed unaccent.rules as well as a new set generated by the generate_unaccent_rules.py distributed with the 11.1 source code: wget http://unicode.org/Public/7.0.0/ucd/UnicodeData.txt wget https://www.unicode.org/repos/cldr/trunk/common/transforms/Latin-ASCII.xml python generate_unaccent_rules.py --unicode-data-file UnicodeData.txt --latin-ascii-file Latin-ASCII.xml > unaccent.rules I see there have been some updates to generate_unaccent_rules.py to handle Greek and Vietnamese characters, but neither of them seem to address this issue. I'm happy to contribute a patch to handle these cases, but of course wanted to make sure this is desired behaviour, or if I am misunderstanding something somewhere. Thank you, Hugh Ranalli
Commits
-
Fix unaccent generation script in Windows
- 0afc0a784188 13.0 landed
-
Add combining characters to unaccent.rules.
- 456e3718e7b7 12.0 landed
-
Update unaccent rules with release 34 of CLDR for Latin-ASCII.xml
- e1c1d5444e43 12.0 landed
-
unaccent: Make generate_unaccent_rules.py Python 3 compatible
- 3d59da9ccdb9 12.0 landed
-
Convert unaccent tests to UTF-8
- b6f3649bba98 12.0 landed