Horton

Re: Make COPY format extendable: Extract COPY TO format implementations

David G. Johnston <david.g.johnston@gmail.com>

View thread View on postgr.es
From: "David G. Johnston" <david.g.johnston@gmail.com>
To: Masahiko Sawada <sawada.mshk@gmail.com>
Cc: Sutou Kouhei <kou@clear-code.com>, tgl@sss.pgh.pa.us, zhjwpku@gmail.com, michael@paquier.xyz, pgsql-hackers@postgresql.org
Date: 2025-03-31T20:42:16Z
Lists: pgsql-hackers 
On Mon, Mar 31, 2025 at 11:52 AM Masahiko Sawada <sawada.mshk@gmail.com>
wrote:

> On Sat, Mar 29, 2025 at 9:49 AM David G. Johnston
> <david.g.johnston@gmail.com> wrote:
> >
> > On Wed, Mar 26, 2025 at 8:28 PM Sutou Kouhei <kou@clear-code.com> wrote:
> >>
> >>
> >> The attached v39 patch set uses the followings:
> >>
> >> 0001: Create copyto_internal.h and change COPY_XXX to
> >>       COPY_SOURCE_XXX and COPY_DEST_XXX accordingly.
> >>       (Same as 1. in your suggestion)
> >> 0002: Support custom format for both COPY TO and COPY FROM.
> >>       (Same as 2. in your suggestion)
> >> 0003: Expose necessary helper functions such as CopySendEndOfRow()
> >>       and add CopyFromSkipErrorRow().
> >>       (3. + 4. in your suggestion)
> >> 0004: Define handler functions for built-in formats.
> >>       (Not included in your suggestion)
> >> 0005: Documentation. (WIP)
> >>       (Same as 5. in your suggestion)
> >>
> >
> > I prefer keeping 0002 and 0004 separate.  In particular, keeping the
> design choice of "unqualified internal format names ignore search_path"
> should stand out as its own commit.
>
> What is the point of having separate commits for already-agreed design
> choices? I guess that it would make it easier to revert that decision.
> But I think it makes more sense that if we agree with "unqualified
> internal format names ignore search_path" the original commit includes
> that decision and describes it in the commit message. If we want to
> change that design based on the discussion later on, we can have a
> separate commit that makes that change and has the link to the
> discussion.
>

Fair.  Comment withdrawn.  Though I was referring to the WIP patches; I
figured the final patch would squash this all together in any case.

David J.

Commits

  1. Refactor Copy{From|To}GetRoutine() to use pass-by-reference argument.

  2. Refactor COPY FROM to use format callback functions.

  3. Refactor COPY TO to use format callback functions.

  4. Another try to fix BF failure introduced in commit ddd5f4f54a.

  5. Revert "Refactor CopyReadAttributes{CSV,Text}() to use a callback in COPY FROM"

  6. Improve COPY TO performance when server and client encodings match

  7. Simplify signature of CopyAttributeOutCSV() in copyto.c

  8. Revert "Refactor CopyAttributeOut{CSV,Text}() to use a callback in COPY TO"

  9. Refactor CopyAttributeOut{CSV,Text}() to use a callback in COPY TO

  10. Refactor CopyReadAttributes{CSV,Text}() to use a callback in COPY FROM

  11. Add progress reporting of skipped tuples during COPY FROM.

  12. pgbench: Add \syncpipeline

  13. meson: Make gzip and tar optional

  14. Export the external file reader used in COPY FROM as APIs.