BUG #15548: Unaccent does not remove combining diacritical characters

The Post Office <noreply@postgresql.org>

From: PG Bug reporting form <noreply@postgresql.org>
To: pgsql-bugs@lists.postgresql.org
Cc: hugh@whtc.ca
Date: 2018-12-12T20:00:45Z
Lists: pgsql-bugs, pgsql-hackers
The following bug has been logged on the website:

Bug reference:      15548
Logged by:          Hugh Ranalli
Email address:      hugh@whtc.ca
PostgreSQL version: 11.1
Operating system:   Ubuntu 18.04
Description:        

Apparently Unicode has two ways of accenting a character: as a separate code
point, which represents the base character and the accent, or as a
"combining diacritical mark"
(https://en.wikipedia.org/wiki/Combining_Diacritical_Marks), in which case
the mark applies itself to the preceding character. For example, A followed
by U+0300 displays À. However, unaccent is not removing these accents. 

SELECT unaccent(U&'A\0300'); should result in 'A', but instead results in
'À.' I'm running PostgreSQL 11.1, installed from the PostgreSQL
repositories. I've read bug report #13440, and have tried with both the
installed unaccent.rules as well as a new set generated by the
generate_unaccent_rules.py distributed with the 11.1 source code:
  wget http://unicode.org/Public/7.0.0/ucd/UnicodeData.txt
  wget
https://www.unicode.org/repos/cldr/trunk/common/transforms/Latin-ASCII.xml
  python generate_unaccent_rules.py --unicode-data-file UnicodeData.txt
--latin-ascii-file  Latin-ASCII.xml > unaccent.rules

I see there have been some updates to generate_unaccent_rules.py to handle
Greek and Vietnamese characters, but neither of them seem to address this
issue. I'm happy to contribute a patch to handle these cases, but of course
wanted to make sure this is desired behaviour, or if I am misunderstanding
something somewhere.

Thank you,
Hugh Ranalli

Commits

  1. Fix unaccent generation script in Windows

  2. Add combining characters to unaccent.rules.

  3. Update unaccent rules with release 34 of CLDR for Latin-ASCII.xml

  4. unaccent: Make generate_unaccent_rules.py Python 3 compatible

  5. Convert unaccent tests to UTF-8