Horton

Re: BUG #15548: Unaccent does not remove combining diacritical characters

Thomas Munro <thomas.munro@enterprisedb.com>

View thread View on postgr.es
From: Thomas Munro <thomas.munro@enterprisedb.com>
To: raam narayana <raam.soft@gmail.com>
Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>, Hugh Ranalli <hugh@whtc.ca>
Date: 2019-02-10T20:44:01Z
Lists: pgsql-bugs, pgsql-hackers 
On Mon, Feb 11, 2019 at 7:07 AM raam narayana <raam.soft@gmail.com> wrote:
> After the latest commit in master branch, I was trying to test the python script. Ironically I still see that the output from the script is completely different from the unaccent.rules file content. Am I missing anything.My testing includes the following
>
> Downloaded the following files
>
> http://unicode.org/Public/8.0.0/ucd/UnicodeData.txt
>
> http://unicode.org/cldr/trac/export/14746/tags/release-34/common/transforms/Latin-ASCII.xml
>
> Executed the below python script
>
> python generate_unaccent_rules.py --unicode-data-file UnicodeData.txt --latin-ascii-file  Latin-ASCII.xml > unaccent.rules
>
> I am using python 3.7.1 and running on Windows 10 Platform
>
> The new status of this patch is: Needs review

Hi Raam,

How does it differ?  Can you please share the output you get?  I used
Python 2.7 on a Mac, exactly those input files, and my output matched
Hugh's.

-- 
Thomas Munro
http://www.enterprisedb.com

Commits

  1. Fix unaccent generation script in Windows

  2. Add combining characters to unaccent.rules.

  3. Update unaccent rules with release 34 of CLDR for Latin-ASCII.xml

  4. unaccent: Make generate_unaccent_rules.py Python 3 compatible

  5. Convert unaccent tests to UTF-8