Re: BUG #18216: Unaccent function is unable to remove accents (diacritic signs) from Japanese character 'ド'

Peter Eisentraut <peter@eisentraut.org>

From: Peter Eisentraut <peter@eisentraut.org>
To: shailesh.totale@sailpoint.com, pgsql-bugs@lists.postgresql.org
Date: 2023-11-29T09:13:54Z
Lists: pgsql-bugs
On 28.11.23 08:15, PG Bug reporting form wrote:
> PostgreSQL's unaccent module does not use Unicode normalisation, but only a
> simple search-and-replace dictionary. The dictionary, unaccent.rules
> (https://github.com/postgres/postgres/blob/master/contrib/unaccent/unaccent.rules)
>    , does not contain these Japanese  characters, thus  its unable to remove
> the diacritic signs.  Can someone please guide when we can expect these
> Japanese characters will be added.
> 
> Also tried to check with latest versions of Postgresql still the latest
> version does not have support for the Japanese characters.
> 
> https://pgpedia.info/u/unaccent.html

As the subsequent discussion shows, it's not quite clear to everybody 
what the exact mandate of the unaccent extension is.  Maybe we'll arrive 
at some conclusion.

In the meantime, I suggest you also consider solving this with 
collations.  You might find that those have a more principled approach 
to this problem, and they also have a lot of customization capabilities. 
  The documentation contains examples of accent-insensitive collations 
(e.g., [0]).  Maybe that will work for you, or serve as the basis for 
customization.

[0]: 
https://www.postgresql.org/docs/current/collation.html#COLLATION-NONDETERMINISTIC