Horton

RE: Make COPY format extendable: Extract COPY TO format implementations

Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>

View thread View on postgr.es
From: "Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com>
To: 'Sutou Kouhei' <kou@clear-code.com>, "zhjwpku@gmail.com" <zhjwpku@gmail.com>
Cc: "andrew@dunslane.net" <andrew@dunslane.net>, "michael@paquier.xyz" <michael@paquier.xyz>, "sawada.mshk@gmail.com" <sawada.mshk@gmail.com>, "nathandbossart@gmail.com" <nathandbossart@gmail.com>, "pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Date: 2023-12-12T02:31:53Z
Lists: pgsql-hackers 
Dear Sutou-san, Junwang,

Sorry for the delay reply.

> 
> Can we discuss how to proceed this improvement?
> 
> There are 2 approaches for it:
> 
> 1. Do the followings concurrently:
>    a. Implementing small changes that got a consensus and
>       merge them step-by-step
>       (e.g. We got a consensus that we need to extract the
>       current format related routines.)
>    b. Discuss design
> 
>    (v1-v3 patches use this approach.)
> 
> 2. Implement one (large) complete patch set with design
>    discussion and merge it
> 
>    (v4- patches use this approach.)
> 
> Which approach is preferred? (Or should we choose another
> approach?)
> 
> I thought that 1. is preferred because it will reduce review
> cost. So I chose 1.

I'm ok to use approach 1, but could you please divide a large patch? E.g.,

0001. defines an infrastructure for copy-API
0002. adjusts current codes to use APIs
0003. adds a test module in src/test/modules or contrib.
...

This approach helps reviewers to see patches deeper. Separated patches can be
combined when they are close to committable.

Best Regards,
Hayato Kuroda
FUJITSU LIMITED

Commits

  1. Refactor Copy{From|To}GetRoutine() to use pass-by-reference argument.

  2. Refactor COPY FROM to use format callback functions.

  3. Refactor COPY TO to use format callback functions.

  4. Another try to fix BF failure introduced in commit ddd5f4f54a.

  5. Revert "Refactor CopyReadAttributes{CSV,Text}() to use a callback in COPY FROM"

  6. Improve COPY TO performance when server and client encodings match

  7. Simplify signature of CopyAttributeOutCSV() in copyto.c

  8. Revert "Refactor CopyAttributeOut{CSV,Text}() to use a callback in COPY TO"

  9. Refactor CopyAttributeOut{CSV,Text}() to use a callback in COPY TO

  10. Refactor CopyReadAttributes{CSV,Text}() to use a callback in COPY FROM

  11. Add progress reporting of skipped tuples during COPY FROM.

  12. pgbench: Add \syncpipeline

  13. meson: Make gzip and tar optional

  14. Export the external file reader used in COPY FROM as APIs.