Re: Make COPY format extendable: Extract COPY TO format implementations

Sutou Kouhei <kou@clear-code.com>

From: Sutou Kouhei <kou@clear-code.com>
To: david.g.johnston@gmail.com
Cc: sawada.mshk@gmail.com, tgl@sss.pgh.pa.us, zhjwpku@gmail.com, michael@paquier.xyz, pgsql-hackers@postgresql.org
Date: 2025-03-30T02:31:26Z
Lists: pgsql-hackers
Hi,

In <CAKFQuwYF7VnYcS9dkfvdzt-dgftMB1DV0bjRcNC8-4iYGS+gjw@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Sat, 29 Mar 2025 09:48:22 -0700,
  "David G. Johnston" <david.g.johnston@gmail.com> wrote:

> I don't think this module should be responsible for testing the validity of
> "qualified names in a string literal" behavior.  Having some of the tests
> use a schema qualification, and I'd suggest explicit
> double-quoting/case-folding, wouldn't hurt just to demonstrate it's
> possible, and how extensions should be referenced, but definitely don't
> need tests to prove the broken cases are indeed broken.  This relies on an
> existing API that has its own tests.  It is definitely pointlessly
> redundant to have 6 tests that only differ from 6 other tests in their use
> of a schema qualification.

You suggest the followings, right?

1. Add tests for "Schema.Name" with mixed cases
2. Remove the following 6 tests in
   src/test/modules/test_copy_format/sql/invalid.sql

   ----
   COPY public.test FROM stdin WITH (FORMAT 'test_schema.test_copy_format_wrong_input_type');
   COPY public.test TO stdout WITH (FORMAT 'test_schema.test_copy_format_wrong_input_type');
   COPY public.test FROM stdin WITH (FORMAT 'test_schema.test_copy_format_wrong_return_type');
   COPY public.test TO stdout WITH (FORMAT 'test_schema.test_copy_format_wrong_return_type');
   COPY public.test FROM stdin WITH (FORMAT 'test_schema.test_copy_format_wrong_return_value');
   COPY public.test TO stdout WITH (FORMAT 'test_schema.test_copy_format_wrong_return_value');
   ----

   because we have the following 6 tests:

   ----
   SET search_path = public,test_schema;
   COPY public.test FROM stdin WITH (FORMAT 'test_copy_format_wrong_input_type');
   COPY public.test TO stdout WITH (FORMAT 'test_copy_format_wrong_input_type');
   COPY public.test FROM stdin WITH (FORMAT 'test_copy_format_wrong_return_type');
   COPY public.test TO stdout WITH (FORMAT 'test_copy_format_wrong_return_type');
   COPY public.test FROM stdin WITH (FORMAT 'test_copy_format_wrong_return_value');
   COPY public.test TO stdout WITH (FORMAT 'test_copy_format_wrong_return_value');
   RESET search_path;
   ----
3. Remove the following tests because the behavior must be
   tested in other places:

   ----
   COPY public.test FROM stdin WITH (FORMAT 'nonexistent');
   COPY public.test TO stdout WITH (FORMAT 'nonexistent');
   COPY public.test FROM stdin WITH (FORMAT 'invalid.qualified.name');
   COPY public.test TO stdout WITH (FORMAT 'invalid.qualified.name');
   ----

Does it miss something?


1.: There is no difference between single-quoting and
    double-quoting here. Because the information what quote
    was used for the given FORMAT value isn't remained
    here. Should we update gram.y?

2.: I don't have a strong opinion for it. If nobody objects
    it, I'll remove them.

3.: I don't have a strong opinion for it. If nobody objects
    it, I'll remove them.


Thanks,
-- 
kou



Commits

  1. Refactor Copy{From|To}GetRoutine() to use pass-by-reference argument.

  2. Refactor COPY FROM to use format callback functions.

  3. Refactor COPY TO to use format callback functions.

  4. Another try to fix BF failure introduced in commit ddd5f4f54a.

  5. Revert "Refactor CopyReadAttributes{CSV,Text}() to use a callback in COPY FROM"

  6. Improve COPY TO performance when server and client encodings match

  7. Simplify signature of CopyAttributeOutCSV() in copyto.c

  8. Revert "Refactor CopyAttributeOut{CSV,Text}() to use a callback in COPY TO"

  9. Refactor CopyAttributeOut{CSV,Text}() to use a callback in COPY TO

  10. Refactor CopyReadAttributes{CSV,Text}() to use a callback in COPY FROM

  11. Add progress reporting of skipped tuples during COPY FROM.

  12. pgbench: Add \syncpipeline

  13. meson: Make gzip and tar optional

  14. Export the external file reader used in COPY FROM as APIs.