Horton

Re: BUG #15548: Unaccent does not remove combining diacritical characters

Michael Paquier <michael@paquier.xyz>

View thread View on postgr.es
From: Michael Paquier <michael@paquier.xyz>
To: Hugh Ranalli <hugh@whtc.ca>
Cc: Tom Lane <tgl@sss.pgh.pa.us>, Alvaro Herrera <alvherre@2ndquadrant.com>, Peter Eisentraut <peter.eisentraut@2ndquadrant.com>, thomas.munro@enterprisedb.com, Daniel Verite <daniel@manitou-mail.org>, pgsql-bugs@lists.postgresql.org
Date: 2019-01-10T06:09:45Z
Lists: pgsql-bugs, pgsql-hackers 
On Wed, Jan 09, 2019 at 09:52:05PM -0500, Hugh Ranalli wrote:
> Thank you for putting so much effort into this. I think that looks great.
> When I was doing this, I discovered that I could parse both pre- and post-
> r29 versions, so I went with that, but I agree that there's probably no
> good reason to do so.

OK, committed then. I have yet to study yet the other part of the
proposal regarding diatritical characters.  Patch 3 has a conflict for
the regression tests, so a rebase would be needed.  That's not a big
deal though to resolve the conflict.  I am also a bit confused by the
newly-generated unaccent.rules.  Why nothing shows up for the second
column (around line 414 for example)?  Shouldn't we have mapping
characters?
--
Michael

Commits

  1. Fix unaccent generation script in Windows

  2. Add combining characters to unaccent.rules.

  3. Update unaccent rules with release 34 of CLDR for Latin-ASCII.xml

  4. unaccent: Make generate_unaccent_rules.py Python 3 compatible

  5. Convert unaccent tests to UTF-8