Horton

Re: Make COPY format extendable: Extract COPY TO format implementations

David G. Johnston <david.g.johnston@gmail.com>

View thread View on postgr.es
From: "David G. Johnston" <david.g.johnston@gmail.com>
To: Sutou Kouhei <kou@clear-code.com>
Cc: sawada.mshk@gmail.com, tgl@sss.pgh.pa.us, zhjwpku@gmail.com, michael@paquier.xyz, pgsql-hackers@postgresql.org
Date: 2025-03-29T16:48:22Z
Lists: pgsql-hackers 
On Wed, Mar 26, 2025 at 8:28 PM Sutou Kouhei <kou@clear-code.com> wrote:

>
> The attached v39 patch set uses the followings:
>
> 0001: Create copyto_internal.h and change COPY_XXX to
>       COPY_SOURCE_XXX and COPY_DEST_XXX accordingly.
>       (Same as 1. in your suggestion)
> 0002: Support custom format for both COPY TO and COPY FROM.
>       (Same as 2. in your suggestion)
> 0003: Expose necessary helper functions such as CopySendEndOfRow()
>       and add CopyFromSkipErrorRow().
>       (3. + 4. in your suggestion)
> 0004: Define handler functions for built-in formats.
>       (Not included in your suggestion)
> 0005: Documentation. (WIP)
>       (Same as 5. in your suggestion)
>
>
I don't think this module should be responsible for testing the validity of
"qualified names in a string literal" behavior.  Having some of the tests
use a schema qualification, and I'd suggest explicit
double-quoting/case-folding, wouldn't hurt just to demonstrate it's
possible, and how extensions should be referenced, but definitely don't
need tests to prove the broken cases are indeed broken.  This relies on an
existing API that has its own tests.  It is definitely pointlessly
redundant to have 6 tests that only differ from 6 other tests in their use
of a schema qualification.

I prefer keeping 0002 and 0004 separate.  In particular, keeping the design
choice of "unqualified internal format names ignore search_path" should
stand out as its own commit.

David J.

Commits

  1. Refactor Copy{From|To}GetRoutine() to use pass-by-reference argument.

  2. Refactor COPY FROM to use format callback functions.

  3. Refactor COPY TO to use format callback functions.

  4. Another try to fix BF failure introduced in commit ddd5f4f54a.

  5. Revert "Refactor CopyReadAttributes{CSV,Text}() to use a callback in COPY FROM"

  6. Improve COPY TO performance when server and client encodings match

  7. Simplify signature of CopyAttributeOutCSV() in copyto.c

  8. Revert "Refactor CopyAttributeOut{CSV,Text}() to use a callback in COPY TO"

  9. Refactor CopyAttributeOut{CSV,Text}() to use a callback in COPY TO

  10. Refactor CopyReadAttributes{CSV,Text}() to use a callback in COPY FROM

  11. Add progress reporting of skipped tuples during COPY FROM.

  12. pgbench: Add \syncpipeline

  13. meson: Make gzip and tar optional

  14. Export the external file reader used in COPY FROM as APIs.