Re: BUG #15548: Unaccent does not remove combining diacritical characters

Hugh Ranalli <hugh@whtc.ca>

From: Hugh Ranalli <hugh@whtc.ca>
To: Daniel Verite <daniel@manitou-mail.org>
Cc: Tom Lane <tgl@sss.pgh.pa.us>, pgsql-bugs@lists.postgresql.org
Date: 2018-12-13T18:50:37Z
Lists: pgsql-bugs, pgsql-hackers
On Thu, 13 Dec 2018, 11:26 Daniel Verite <daniel@manitou-mail.org wrote:

>         Tom Lane wrote:
>
> > Hm, I thought the OP's proposal was just to make unaccent drop
> > combining diacriticals independently of context, which'd avoid the
> > combinatorial-growth problem.
>

That's what I was thinking. Given that the accent is separate from the
characters, simply dropping it should result in the correct unaccented
character.

>
> In that case, this could be achieved by simply appending the
> diacriticals themselves to unaccent.rules, since replacement of a
> string by an empty string is already supported as a rule.
> It doesn't seem like the current file has any of these, but from
> https://www.postgresql.org/docs/11/unaccent.html :
>
>  "Alternatively, if only one character is given on a line, instances
>  of that character are deleted; this is useful in languages where
>  accents are represented by separate characters"
>

Yes, I had read that in the docs, and that's the approach I planned to
take. I'll go ahead and develop a patch, then.

Best wishes,
Hugh

>

Commits

  1. Fix unaccent generation script in Windows

  2. Add combining characters to unaccent.rules.

  3. Update unaccent rules with release 34 of CLDR for Latin-ASCII.xml

  4. unaccent: Make generate_unaccent_rules.py Python 3 compatible

  5. Convert unaccent tests to UTF-8