Re: BUG #15548: Unaccent does not remove combining diacritical characters
Daniel Verite <daniel@manitou-mail.org>
From: "Daniel Verite" <daniel@manitou-mail.org>
To: hugh@whtc.ca,pgsql-bugs@lists.postgresql.org
Date: 2018-12-13T13:19:51Z
Lists: pgsql-bugs, pgsql-hackers
PG Bug reporting form wrote:
> Apparently Unicode has two ways of accenting a character: as a separate code
> point, which represents the base character and the accent, or as a
> "combining diacritical mark"
> (https://en.wikipedia.org/wiki/Combining_Diacritical_Marks)
Yes. See also https://en.wikipedia.org/wiki/Unicode_equivalence
In general, PostgreSQL leaves it to applications to normalize
Unicode strings so that they are all in the same canonical form,
either composed or decomposed.
> the mark applies itself to the preceding character. For example, A
> followed by U+0300 displays À. However, unaccent is not removing
> these accents.
Short of having the input normalized by the application, ISTM that the
best solution would be to provide functions to do it in Postgres, so
you'd just write for example:
unaccent(unicode_NFC(string))
Otherwise unaccent.rules can be customized. You may add replacements
for letter+diacritical sequences that are missing for the languages
you have to deal with. But doing it in general for all diacriticals
multiplied by all base characters seems unrealistic.
Best regards,
--
Daniel Vérité
PostgreSQL-powered mailer: http://www.manitou-mail.org
Twitter: @DanielVerite
Commits
-
Fix unaccent generation script in Windows
- 0afc0a784188 13.0 landed
-
Add combining characters to unaccent.rules.
- 456e3718e7b7 12.0 landed
-
Update unaccent rules with release 34 of CLDR for Latin-ASCII.xml
- e1c1d5444e43 12.0 landed
-
unaccent: Make generate_unaccent_rules.py Python 3 compatible
- 3d59da9ccdb9 12.0 landed
-
Convert unaccent tests to UTF-8
- b6f3649bba98 12.0 landed