Horton

Re: confusing / inefficient "need_transcoding" handling in copy

Michael Paquier <michael@paquier.xyz>

View thread View on postgr.es
From: Michael Paquier <michael@paquier.xyz>
To: Sutou Kouhei <kou@clear-code.com>
Cc: andres@anarazel.de, tgl@sss.pgh.pa.us, pgsql-hackers@postgresql.org, ishii@sraoss.co.jp
Date: 2024-02-13T21:56:16Z
Lists: pgsql-hackers 
On Thu, Feb 08, 2024 at 05:25:01PM +0900, Sutou Kouhei wrote:
> In <20240206222445.hzq22pb2nye7rm67@awork3.anarazel.de>
>   "Re: confusing / inefficient "need_transcoding" handling in copy" on Tue, 6 Feb 2024 14:24:45 -0800,
>   Andres Freund <andres@anarazel.de> wrote:
> 
>> One unfortunate issue: We don't have any tests verifying that COPY FROM
>> catches encoding issues.
> 
> How about the attached patch for it?
>
> +CREATE TABLE test (t text);
> +COPY test FROM stdin WITH (ENCODING 'EUC_JP');
> +こんにちは
> +\.
> +
> +DROP TABLE test;

We have a couple of non-ASCII characters in the tests, but I suspect
that this one will not be digested correctly everywhere, even if
EUC_JP should be OK to use for the check.  How about writing an
arbitrary sequence of bytes into a temporary file that gets used for 
the COPY FROM instead?  See for example how we do that with
abs_builddir in copy.sql.
--
Michael

Commits

  1. Add some tests for encoding conversion in COPY TO/FROM

  2. Improve COPY TO performance when server and client encodings match