Horton

Re: BUG #15548: Unaccent does not remove combining diacritical characters

Michael Paquier <michael@paquier.xyz>

View thread View on postgr.es
From: Michael Paquier <michael@paquier.xyz>
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Thomas Munro <thomas.munro@enterprisedb.com>, hugh@whtc.ca, daniel@manitou-mail.org, PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Date: 2018-12-18T06:33:04Z
Lists: pgsql-bugs, pgsql-hackers 
On Tue, Dec 18, 2018 at 01:23:57AM -0500, Tom Lane wrote:
> Maybe I'm misremembering, but I thought that discussion was about the
> code files.  I am still mistrustful of non-ASCII in our code files.

Yes, that was in generate_unaccent_rules.py:
https://www.postgresql.org/message-id/25859.1535076450@sss.pgh.pa.us

> But for data and test files, we've been accepting UTF8 ever since the
> text-search-in-core stuff landed.  Heck, unaccent.rules itself is UTF8.

Okay, fine by me.
--
Michael

Commits

  1. Fix unaccent generation script in Windows

  2. Add combining characters to unaccent.rules.

  3. Update unaccent rules with release 34 of CLDR for Latin-ASCII.xml

  4. unaccent: Make generate_unaccent_rules.py Python 3 compatible

  5. Convert unaccent tests to UTF-8