Horton

Re: Make COPY format extendable: Extract COPY TO format implementations

Masahiko Sawada <sawada.mshk@gmail.com>

View thread View on postgr.es
From: Masahiko Sawada <sawada.mshk@gmail.com>
To: Sutou Kouhei <kou@clear-code.com>
Cc: kuroda.hayato@fujitsu.com, zhjwpku@gmail.com, andrew@dunslane.net, michael@paquier.xyz, nathandbossart@gmail.com, pgsql-hackers@postgresql.org
Date: 2023-12-11T00:36:38Z
Lists: pgsql-hackers 
On Sun, Dec 10, 2023 at 5:44 AM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> Thanks for reviewing our latest patch!
>
> In
>  <TY3PR01MB9889C9234CD220A3A7075F0DF589A@TY3PR01MB9889.jpnprd01.prod.outlook.com>
>   "RE: Make COPY format extendable: Extract COPY TO format implementations" on Sat, 9 Dec 2023 02:43:49 +0000,
>   "Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com> wrote:
>
> > (I remember that this theme was talked at Japan PostgreSQL conference)
>
> Yes. I should have talked to you more at the conference...
> I will do it next time!
>
>
> Can we discuss how to proceed this improvement?
>
> There are 2 approaches for it:
>
> 1. Do the followings concurrently:
>    a. Implementing small changes that got a consensus and
>       merge them step-by-step
>       (e.g. We got a consensus that we need to extract the
>       current format related routines.)
>    b. Discuss design

It's preferable to make patches small for easy review. We can merge
them anytime before commit if necessary.

I think we need to discuss overall design about callbacks and how
extensions define a custom copy handler etc. It may require some PoC
patches. Once we have a consensus on overall design we polish patches
including the documentation changes and regression tests.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

Commits

  1. Refactor Copy{From|To}GetRoutine() to use pass-by-reference argument.

  2. Refactor COPY FROM to use format callback functions.

  3. Refactor COPY TO to use format callback functions.

  4. Another try to fix BF failure introduced in commit ddd5f4f54a.

  5. Revert "Refactor CopyReadAttributes{CSV,Text}() to use a callback in COPY FROM"

  6. Improve COPY TO performance when server and client encodings match

  7. Simplify signature of CopyAttributeOutCSV() in copyto.c

  8. Revert "Refactor CopyAttributeOut{CSV,Text}() to use a callback in COPY TO"

  9. Refactor CopyAttributeOut{CSV,Text}() to use a callback in COPY TO

  10. Refactor CopyReadAttributes{CSV,Text}() to use a callback in COPY FROM

  11. Add progress reporting of skipped tuples during COPY FROM.

  12. pgbench: Add \syncpipeline

  13. meson: Make gzip and tar optional

  14. Export the external file reader used in COPY FROM as APIs.