Horton

Re: BUG #15548: Unaccent does not remove combining diacritical characters

Ramanarayana <raam.soft@gmail.com>

View thread View on postgr.es
From: raam narayana <raam.soft@gmail.com>
To: pgsql-hackers@lists.postgresql.org
Cc: Hugh Ranalli <hugh@whtc.ca>
Date: 2019-02-10T20:06:25Z
Lists: pgsql-bugs, pgsql-hackers 
Hi,

After the latest commit in master branch, I was trying to test the python script. Ironically I still see that the output from the script is completely different from the unaccent.rules file content. Am I missing anything.My testing includes the following

Downloaded the following files

http://unicode.org/Public/8.0.0/ucd/UnicodeData.txt
 
http://unicode.org/cldr/trac/export/14746/tags/release-34/common/transforms/Latin-ASCII.xml

Executed the below python script

python generate_unaccent_rules.py --unicode-data-file UnicodeData.txt --latin-ascii-file  Latin-ASCII.xml > unaccent.rules 

I am using python 3.7.1 and running on Windows 10 Platform

The new status of this patch is: Needs review

Commits

  1. Fix unaccent generation script in Windows

  2. Add combining characters to unaccent.rules.

  3. Update unaccent rules with release 34 of CLDR for Latin-ASCII.xml

  4. unaccent: Make generate_unaccent_rules.py Python 3 compatible

  5. Convert unaccent tests to UTF-8