Horton

Re: Make COPY format extendable: Extract COPY TO format implementations

Li, Yong <yoli@ebay.com>

View thread View on postgr.es
From: "Li, Yong" <yoli@ebay.com>
To: Sutou Kouhei <kou@clear-code.com>
Cc: Tomas Vondra <tomas.vondra@enterprisedb.com>, Andres Freund <andres@anarazel.de>, Michael Paquier <michael@paquier.xyz>, "sawada.mshk@gmail.com" <sawada.mshk@gmail.com>, "zhjwpku@gmail.com" <zhjwpku@gmail.com>, "andrew@dunslane.net" <andrew@dunslane.net>, Nathan Bossart <nathandbossart@gmail.com>, "pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Date: 2024-07-26T08:57:42Z
Lists: pgsql-hackers 

> On Jul 25, 2024, at 12:51, Sutou Kouhei <kou@clear-code.com> wrote:
> 
> Hi,
> 
> THREAD SUMMARY:

Very nice summary.

> 
> Implementation:
> 
> The v18 patch set is the latest patch set. [6]
> It includes the following patches:
> 
> 0001: This adds a basic feature (Copy{From,To}Routine)
>      (This isn't enough for extending COPY format.
>      This just extracts minimal procedure sets to be
>      extendable as callback sets.)
> 0002: This uses Copy{From,To}Rountine for the existing
>      formats (text, csv and binary)
>      (This may not be committed because there is a
>      profiling related concern. See the following section
>      for details)
> 0003: This adds support for specifying custom format by
>      "COPY ... WITH (format 'my-format')"
>      (This also adds a test for this feature.)
> 0004: This exports Copy{From,To}StateData
>      (But this isn't enough to implement custom COPY
>      FROM/TO handlers as an extension.)
> 0005: This adds opaque member to Copy{From,To}StateData and
>      export some functions to read the next data and flush
>      the buffer
>      (We can implement a PoC Apache Arrow COPY FROM/TO
>      handler as an extension with this. [7])
> 
> Thanks,
> --
> kou
> 

This review is for 0001 only because the other patches are not ready
for commit.

The v18-0001 patch applies cleanly to HEAD. “make check-world” also
runs cleanly. The patch looks good for me.


Regards,
Yong

Commits

  1. Refactor Copy{From|To}GetRoutine() to use pass-by-reference argument.

  2. Refactor COPY FROM to use format callback functions.

  3. Refactor COPY TO to use format callback functions.

  4. Another try to fix BF failure introduced in commit ddd5f4f54a.

  5. Revert "Refactor CopyReadAttributes{CSV,Text}() to use a callback in COPY FROM"

  6. Improve COPY TO performance when server and client encodings match

  7. Simplify signature of CopyAttributeOutCSV() in copyto.c

  8. Revert "Refactor CopyAttributeOut{CSV,Text}() to use a callback in COPY TO"

  9. Refactor CopyAttributeOut{CSV,Text}() to use a callback in COPY TO

  10. Refactor CopyReadAttributes{CSV,Text}() to use a callback in COPY FROM

  11. Add progress reporting of skipped tuples during COPY FROM.

  12. pgbench: Add \syncpipeline

  13. meson: Make gzip and tar optional

  14. Export the external file reader used in COPY FROM as APIs.