Horton

Re: BUG #15548: Unaccent does not remove combining diacritical characters

Tom Lane <tgl@sss.pgh.pa.us>

View thread View on postgr.es
From: Tom Lane <tgl@sss.pgh.pa.us>
To: "Daniel Verite" <daniel@manitou-mail.org>
Cc: hugh@whtc.ca, pgsql-bugs@lists.postgresql.org
Date: 2018-12-13T15:05:42Z
Lists: pgsql-bugs, pgsql-hackers 
"Daniel Verite" <daniel@manitou-mail.org> writes:
> 	PG Bug reporting form wrote:
>> ... For example, A
>> followed by U+0300 displays À. However, unaccent is not removing
>> these accents.

> Short of having the input normalized by the application, ISTM that the
> best solution would be to provide functions to do it in Postgres, so
> you'd just write for example:
>     unaccent(unicode_NFC(string))

That might be worthwhile, but it seems independent of this issue.

> Otherwise unaccent.rules can be customized. You may add replacements
> for letter+diacritical sequences that are missing for the languages
> you have to deal with. But doing it in general for all diacriticals
> multiplied by all base characters seems unrealistic.

Hm, I thought the OP's proposal was just to make unaccent drop 
combining diacriticals independently of context, which'd avoid the
combinatorial-growth problem.

			regards, tom lane

Commits

  1. Fix unaccent generation script in Windows

  2. Add combining characters to unaccent.rules.

  3. Update unaccent rules with release 34 of CLDR for Latin-ASCII.xml

  4. unaccent: Make generate_unaccent_rules.py Python 3 compatible

  5. Convert unaccent tests to UTF-8