Horton

Re: Make COPY format extendable: Extract COPY TO format implementations

Nathan Bossart <nathandbossart@gmail.com>

View thread View on postgr.es
From: Nathan Bossart <nathandbossart@gmail.com>
To: Sutou Kouhei <kou@clear-code.com>
Cc: pgsql-hackers@postgresql.org
Date: 2023-12-05T18:24:58Z
Lists: pgsql-hackers 
On Mon, Dec 04, 2023 at 03:35:48PM +0900, Sutou Kouhei wrote:
> I want to work on making COPY format extendable. I attach
> the first patch for it. I'll send more patches after this is
> merged.

Given the current discussion about adding JSON, I think this could be a
nice bit of refactoring that could ultimately open the door to providing
other COPY formats via shared libraries.

> In other words, this is just a refactoring for further
> changes to make COPY format extendable. If I use "complete
> the task and then request reviews for it" approach, it will
> be difficult to review because changes for it will be
> large. So I want to work on this step by step. Is it
> acceptable?

I think it makes sense to do this part independently, but we should be
careful to design this with the follow-up tasks in mind.

> So I measured COPY TO time with/without this change. You can
> see there is no significant loss of performance.
> 
> Data: Random 32 bit integers:
> 
>     CREATE TABLE data (int32 integer);
>     INSERT INTO data
>       SELECT random() * 10000
>         FROM generate_series(1, ${n_records});

Seems encouraging.  I assume the performance concerns stem from the use of
function pointers.  Or was there something else?

-- 
Nathan Bossart
Amazon Web Services: https://aws.amazon.com

Commits

  1. Refactor Copy{From|To}GetRoutine() to use pass-by-reference argument.

  2. Refactor COPY FROM to use format callback functions.

  3. Refactor COPY TO to use format callback functions.

  4. Another try to fix BF failure introduced in commit ddd5f4f54a.

  5. Revert "Refactor CopyReadAttributes{CSV,Text}() to use a callback in COPY FROM"

  6. Improve COPY TO performance when server and client encodings match

  7. Simplify signature of CopyAttributeOutCSV() in copyto.c

  8. Revert "Refactor CopyAttributeOut{CSV,Text}() to use a callback in COPY TO"

  9. Refactor CopyAttributeOut{CSV,Text}() to use a callback in COPY TO

  10. Refactor CopyReadAttributes{CSV,Text}() to use a callback in COPY FROM

  11. Add progress reporting of skipped tuples during COPY FROM.

  12. pgbench: Add \syncpipeline

  13. meson: Make gzip and tar optional

  14. Export the external file reader used in COPY FROM as APIs.