Horton

Re: Make COPY format extendable: Extract COPY TO format implementations

Junwang Zhao <zhjwpku@gmail.com>

View thread View on postgr.es
From: Junwang Zhao <zhjwpku@gmail.com>
To: Sutou Kouhei <kou@clear-code.com>
Cc: kuroda.hayato@fujitsu.com, andrew@dunslane.net, michael@paquier.xyz, sawada.mshk@gmail.com, nathandbossart@gmail.com, pgsql-hackers@postgresql.org
Date: 2023-12-10T00:24:51Z
Lists: pgsql-hackers 
On Sun, Dec 10, 2023 at 4:44 AM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> Thanks for reviewing our latest patch!
>
> In
>  <TY3PR01MB9889C9234CD220A3A7075F0DF589A@TY3PR01MB9889.jpnprd01.prod.outlook.com>
>   "RE: Make COPY format extendable: Extract COPY TO format implementations" on Sat, 9 Dec 2023 02:43:49 +0000,
>   "Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com> wrote:
>
> > (I remember that this theme was talked at Japan PostgreSQL conference)
>
> Yes. I should have talked to you more at the conference...
> I will do it next time!
>
>
> Can we discuss how to proceed this improvement?
>
> There are 2 approaches for it:
>
> 1. Do the followings concurrently:
>    a. Implementing small changes that got a consensus and
>       merge them step-by-step
>       (e.g. We got a consensus that we need to extract the
>       current format related routines.)
>    b. Discuss design
>
>    (v1-v3 patches use this approach.)
>
> 2. Implement one (large) complete patch set with design
>    discussion and merge it
>
>    (v4- patches use this approach.)
>
> Which approach is preferred? (Or should we choose another
> approach?)
>
> I thought that 1. is preferred because it will reduce review
> cost. So I chose 1.
>
> If 2. is preferred, I'll use 2. (I'll add more changes to
> the latest patch.)
>
I'm ok with both, and I'd like to work with you for the parquet
extension, excited about this new feature, thanks for bringing
this up.

Forgive me for making so much noise about approach 2, I
just want to hear about more suggestions of the final shape
of this feature.

>
> Thanks,
> --
> kou



-- 
Regards
Junwang Zhao

Commits

  1. Refactor Copy{From|To}GetRoutine() to use pass-by-reference argument.

  2. Refactor COPY FROM to use format callback functions.

  3. Refactor COPY TO to use format callback functions.

  4. Another try to fix BF failure introduced in commit ddd5f4f54a.

  5. Revert "Refactor CopyReadAttributes{CSV,Text}() to use a callback in COPY FROM"

  6. Improve COPY TO performance when server and client encodings match

  7. Simplify signature of CopyAttributeOutCSV() in copyto.c

  8. Revert "Refactor CopyAttributeOut{CSV,Text}() to use a callback in COPY TO"

  9. Refactor CopyAttributeOut{CSV,Text}() to use a callback in COPY TO

  10. Refactor CopyReadAttributes{CSV,Text}() to use a callback in COPY FROM

  11. Add progress reporting of skipped tuples during COPY FROM.

  12. pgbench: Add \syncpipeline

  13. meson: Make gzip and tar optional

  14. Export the external file reader used in COPY FROM as APIs.