Horton

Re: BUG #15548: Unaccent does not remove combining diacritical characters

Hugh Ranalli <hugh@whtc.ca>

View thread View on postgr.es
From: Hugh Ranalli <hugh@whtc.ca>
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Daniel Verite <daniel@manitou-mail.org>, thomas.munro@enterprisedb.com, pgsql-bugs@lists.postgresql.org
Date: 2018-12-15T19:05:07Z
Lists: pgsql-bugs, pgsql-hackers 
On Sat, 15 Dec 2018 at 13:44, Tom Lane <tgl@sss.pgh.pa.us> wrote:

> Hm.  Something funny is going on here.  When I fetch the two reference
> files from the URLs cited in the script, and do
>

> python2 generate_unaccent_rules.py --unicode-data-file UnicodeData.txt
> --latin-ascii-file Latin-ASCII.xml >newrules
>
> I get something that's bit-for-bit the same as what's in unaccent.rules.
> So there's clearly a platform difference between here and there.
>
> I'm using Python 2.6.6, which is what ships with RHEL6; have not tried
> it on anything newer.
>
Well, that's embarrassing. When I looked I couldn't see anything that
looked platform specific. I'm on Python 2.7.6, which shipped with Mint 17.
We use other versions of 2.7 on our production platforms. I'll take another
look, and check the URLs I am using.

Commits

  1. Fix unaccent generation script in Windows

  2. Add combining characters to unaccent.rules.

  3. Update unaccent rules with release 34 of CLDR for Latin-ASCII.xml

  4. unaccent: Make generate_unaccent_rules.py Python 3 compatible

  5. Convert unaccent tests to UTF-8