Horton

Re: Make COPY format extendable: Extract COPY TO format implementations

jian he <jian.universality@gmail.com>

View thread View on postgr.es
From: jian he <jian.universality@gmail.com>
To: Sutou Kouhei <kou@clear-code.com>
Cc: michael@paquier.xyz, andres@anarazel.de, sawada.mshk@gmail.com, zhjwpku@gmail.com, andrew@dunslane.net, nathandbossart@gmail.com, pgsql-hackers@postgresql.org
Date: 2024-03-13T08:00:46Z
Lists: pgsql-hackers 
On Mon, Mar 11, 2024 at 8:56 AM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <CACJufxEgn3=j-UWg-f2-DbLO+uVSKGcofpkX5trx+=YX6icSFg@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Mon, 11 Mar 2024 08:00:00 +0800,
>   jian he <jian.universality@gmail.com> wrote:
>
> > Hi, here are my cents:
> > Currently in v17, we have 3 extra functions within DoCopyTo
> > CopyToStart, one time, start, doing some preliminary work.
> > CopyToOneRow, doing the repetitive work, called many times, row by row.
> > CopyToEnd, one time doing the closing work.
> >
> > seems to need a function pointer for processing the format and other options.
> > or maybe the reason is we need a one time function call before doing DoCopyTo,
> > like one time initialization.
>
> I know that JSON format wants it but can we defer it? We can
> add more options later. I want to proceed this improvement
> step by step.
>
> More use cases will help us which callbacks are needed. We
> will be able to collect more use cases by providing basic
> callbacks.

I guess one of the ultimate goals would be that COPY can export data
to a customized format.
Let's say the customized format is "csv1", but it is just analogous to
the csv format.
people should be able to create an extension, with serval C functions,
then they can do `copy (select 1 ) to stdout (format 'csv1');`
but the output will be exact same as `copy (select 1 ) to stdout
(format 'csv');`

In such a scenario, we require a function akin to ProcessCopyOptions
to handle situations
where CopyFormatOptions->csv_mode is true, while the format is "csv1".

but CopyToStart is already within the DoCopyTo function, so you do
need an extra function pointer?
I do agree with the incremental improvement method.

Commits

  1. Refactor Copy{From|To}GetRoutine() to use pass-by-reference argument.

  2. Refactor COPY FROM to use format callback functions.

  3. Refactor COPY TO to use format callback functions.

  4. Another try to fix BF failure introduced in commit ddd5f4f54a.

  5. Revert "Refactor CopyReadAttributes{CSV,Text}() to use a callback in COPY FROM"

  6. Improve COPY TO performance when server and client encodings match

  7. Simplify signature of CopyAttributeOutCSV() in copyto.c

  8. Revert "Refactor CopyAttributeOut{CSV,Text}() to use a callback in COPY TO"

  9. Refactor CopyAttributeOut{CSV,Text}() to use a callback in COPY TO

  10. Refactor CopyReadAttributes{CSV,Text}() to use a callback in COPY FROM

  11. Add progress reporting of skipped tuples during COPY FROM.

  12. pgbench: Add \syncpipeline

  13. meson: Make gzip and tar optional

  14. Export the external file reader used in COPY FROM as APIs.