Horton

Re: BUG #15548: Unaccent does not remove combining diacritical characters

Tom Lane <tgl@sss.pgh.pa.us>

View thread View on postgr.es
From: Tom Lane <tgl@sss.pgh.pa.us>
To: Hugh Ranalli <hugh@whtc.ca>
Cc: Daniel Verite <daniel@manitou-mail.org>, pgsql-bugs@lists.postgresql.org
Date: 2018-12-14T22:50:03Z
Lists: pgsql-bugs, pgsql-hackers 
Hugh Ranalli <hugh@whtc.ca> writes:
> I've attached a patch removes combining diacriticals. As with Latin and
> Greek letters, it uses ranges to restrict its activity.

Cool.  Please add it to the current CF so we don't forget about it:
https://commitfest.postgresql.org/21/

> I have not submitted a patch for unaccent.rules, as it seems that a rules
> file generated from generate_unaccent_rules.py will actually remove a large
> number of rules (even before my changes), such as replacing the copyright
> symbol © with (C), as well as other accented characters. It's probably
> worth asking if the shipped unaccent.rules should correspond to what the
> shipped generation utility produces, or not. I was surprised to see that it
> didn't.

Me too -- seems like that bears looking into.  Perhaps the script's
results are platform dependent -- what were you testing on?

			regards, tom lane

Commits

  1. Fix unaccent generation script in Windows

  2. Add combining characters to unaccent.rules.

  3. Update unaccent rules with release 34 of CLDR for Latin-ASCII.xml

  4. unaccent: Make generate_unaccent_rules.py Python 3 compatible

  5. Convert unaccent tests to UTF-8