Horton

Re: Make COPY format extendable: Extract COPY TO format implementations

Sutou Kouhei <kou@clear-code.com>

View thread View on postgr.es
From: Sutou Kouhei <kou@clear-code.com>
To: kuroda.hayato@fujitsu.com
Cc: zhjwpku@gmail.com, andrew@dunslane.net, michael@paquier.xyz, sawada.mshk@gmail.com, nathandbossart@gmail.com, pgsql-hackers@postgresql.org
Date: 2023-12-15T02:55:18Z
Lists: pgsql-hackers 
Hi,

In 
 <OS3PR01MB9882F023300EDC5AFD8A8339F58EA@OS3PR01MB9882.jpnprd01.prod.outlook.com>
  "RE: Make COPY format extendable: Extract COPY TO format implementations" on Tue, 12 Dec 2023 02:31:53 +0000,
  "Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com> wrote:

>> Can we discuss how to proceed this improvement?
>> 
>> There are 2 approaches for it:
>> 
>> 1. Do the followings concurrently:
>>    a. Implementing small changes that got a consensus and
>>       merge them step-by-step
>>       (e.g. We got a consensus that we need to extract the
>>       current format related routines.)
>>    b. Discuss design
>> 
>>    (v1-v3 patches use this approach.)
>> 
>> 2. Implement one (large) complete patch set with design
>>    discussion and merge it
>> 
>>    (v4- patches use this approach.)
>> 
>> Which approach is preferred? (Or should we choose another
>> approach?)
>> 
>> I thought that 1. is preferred because it will reduce review
>> cost. So I chose 1.
> 
> I'm ok to use approach 1, but could you please divide a large patch? E.g.,
> 
> 0001. defines an infrastructure for copy-API
> 0002. adjusts current codes to use APIs
> 0003. adds a test module in src/test/modules or contrib.
> ...
> 
> This approach helps reviewers to see patches deeper. Separated patches can be
> combined when they are close to committable.

It seems that I should have chosen another approach based on
comments so far:

3. Do the followings in order:
   a. Implement a workable (but maybe dirty and/or incomplete)
      implementation to discuss design like [1], discuss
      design with it and get a consensus on design
   b. Implement small patches based on the design

[1]: https://www.postgresql.org/message-id/CAD21AoCunywHird3GaPzWe6s9JG1wzxj3Cr6vGN36DDheGjOjA%40mail.gmail.com 

I'll implement a custom COPY FORMAT handler with [1] and
provide a feedback with the experience. (It's for a.)


Thanks,
-- 
kou

Commits

  1. Refactor Copy{From|To}GetRoutine() to use pass-by-reference argument.

  2. Refactor COPY FROM to use format callback functions.

  3. Refactor COPY TO to use format callback functions.

  4. Another try to fix BF failure introduced in commit ddd5f4f54a.

  5. Revert "Refactor CopyReadAttributes{CSV,Text}() to use a callback in COPY FROM"

  6. Improve COPY TO performance when server and client encodings match

  7. Simplify signature of CopyAttributeOutCSV() in copyto.c

  8. Revert "Refactor CopyAttributeOut{CSV,Text}() to use a callback in COPY TO"

  9. Refactor CopyAttributeOut{CSV,Text}() to use a callback in COPY TO

  10. Refactor CopyReadAttributes{CSV,Text}() to use a callback in COPY FROM

  11. Add progress reporting of skipped tuples during COPY FROM.

  12. pgbench: Add \syncpipeline

  13. meson: Make gzip and tar optional

  14. Export the external file reader used in COPY FROM as APIs.