Thread

Re: Proposal: tighten validation for legacy EUC encodings or document that accepted byte sequences may be unconvertible to UTF8

David G. Johnston <david.g.johnston@gmail.com> — 2026-05-02T03:28:31Z

On Friday, May 1, 2026, Zhongpu Chen <chenloveit@gmail.com> wrote:

> The issue is not specific to E'\\x..' literals. A normal COPY FROM data
> file with ENCODING 'EUC_CN' can create text rows that later cannot be
> retrieved with SELECT.
>
>  This suggests that input validation for EUC_CN is only structural, while
> the EUC_CN-to-UTF8 conversion table is stricter.
>

I suspect a lack of desire to maintain and ensure that specific values are
verified; or accepting the runtime cost to do so.  It is indeed
structural.  This point should probably be documented better.  But it’s
hard to feel too bad if the input claims it is providing verifiable EUC_CN
data then proceeds to supply data that lacks meaning in reality.  We are
happy to just store and return your data to you - but it’s unreasonable to
ask for it to be converted.  It would be nice for the database to provide
an extra layer of protection, so I’m not against the idea.  Either
automatically or or at least providing a function that could, say, be
called in a trigger for opt-in.  But definitely feels like a problematic
benefit-to-cost proposition.

David J.