Horton

Thread

Commits

  1. Refactor Copy{From|To}GetRoutine() to use pass-by-reference argument.

  2. Refactor COPY FROM to use format callback functions.

  3. Refactor COPY TO to use format callback functions.

  4. Another try to fix BF failure introduced in commit ddd5f4f54a.

  5. Revert "Refactor CopyReadAttributes{CSV,Text}() to use a callback in COPY FROM"

  6. Improve COPY TO performance when server and client encodings match

  7. Simplify signature of CopyAttributeOutCSV() in copyto.c

  8. Revert "Refactor CopyAttributeOut{CSV,Text}() to use a callback in COPY TO"

  9. Refactor CopyAttributeOut{CSV,Text}() to use a callback in COPY TO

  10. Refactor CopyReadAttributes{CSV,Text}() to use a callback in COPY FROM

  11. Add progress reporting of skipped tuples during COPY FROM.

  12. pgbench: Add \syncpipeline

  13. meson: Make gzip and tar optional

  14. Export the external file reader used in COPY FROM as APIs.

  1. Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2023-12-04T06:35:48Z

    Hi,

I want to work on making COPY format extendable. I attach
the first patch for it. I'll send more patches after this is
merged.


Background:

Currently, COPY TO/FROM supports only "text", "csv" and
"binary" formats. There are some requests to support more
COPY formats. For example:

* 2023-11: JSON and JSON lines [1]
* 2022-04: Apache Arrow [2]
* 2018-02: Apache Avro, Apache Parquet and Apache ORC [3]

(FYI: I want to add support for Apache Arrow.)

There were discussions how to add support for more formats. [3][4]
In these discussions, we got a consensus about making COPY
format extendable.

But it seems that nobody works on this yet. So I want to
work on this. (If there is anyone who wants to work on this
together, I'm happy.)


Summary:

The attached patch introduces CopyToFormatOps struct that is
similar to TupleTableSlotOps for TupleTableSlot but
CopyToFormatOps is for COPY TO format. CopyToFormatOps has
routines to implement a COPY TO format.

The attached patch doesn't change:

* the current behavior (all existing tests are still passed
  without changing them)
* the existing "text", "csv" and "binary" format output
  implementations including local variable names (the
  attached patch just move them and adjust indent)
* performance (no significant loss of performance)

In other words, this is just a refactoring for further
changes to make COPY format extendable. If I use "complete
the task and then request reviews for it" approach, it will
be difficult to review because changes for it will be
large. So I want to work on this step by step. Is it
acceptable?

TODOs that should be done in subsequent patches:

* Add some CopyToState readers such as CopyToStateGetDest(),
  CopyToStateGetAttnums() and CopyToStateGetOpts()
  (We will need to consider which APIs should be exported.)
  (This is for implemeing COPY TO format by extension.)
* Export CopySend*() in src/backend/commands/copyto.c
  (This is for implemeing COPY TO format by extension.)
* Add API to register a new COPY TO format implementation
* Add "CREATE XXX" to register a new COPY TO format (or COPY
  TO/FROM format) implementation
  ("CREATE COPY HANDLER" was suggested in [5].)
* Same for COPY FROM


Performance:

We got a consensus about making COPY format extendable but
we should care about performance. [6]

> I think that step 1 ought to be to convert the existing
> formats into plug-ins, and demonstrate that there's no
> significant loss of performance.

So I measured COPY TO time with/without this change. You can
see there is no significant loss of performance.

Data: Random 32 bit integers:

    CREATE TABLE data (int32 integer);
    INSERT INTO data
      SELECT random() * 10000
        FROM generate_series(1, ${n_records});

The number of records: 100K, 1M and 10M

100K without this change:

    format,elapsed time (ms)
    text,22.527
    csv,23.822
    binary,24.806

100K with this change:

    format,elapsed time (ms)
    text,22.919
    csv,24.643
    binary,24.705

1M without this change:

    format,elapsed time (ms)
    text,223.457
    csv,233.583
    binary,242.687

1M with this change:

    format,elapsed time (ms)
    text,224.591
    csv,233.964
    binary,247.164

10M without this change:

    format,elapsed time (ms)
    text,2330.383
    csv,2411.394
    binary,2590.817

10M with this change:

    format,elapsed time (ms)
    text,2231.307
    csv,2408.067
    binary,2473.617


[1]: https://www.postgresql.org/message-id/flat/24e3ee88-ec1e-421b-89ae-8a47ee0d2df1%40joeconway.com#a5e6b8829f9a74dfc835f6f29f2e44c5
[2]: https://www.postgresql.org/message-id/flat/CAGrfaBVyfm0wPzXVqm0%3Dh5uArYh9N_ij%2BsVpUtDHqkB%3DVyB3jw%40mail.gmail.com
[3]: https://www.postgresql.org/message-id/flat/20180210151304.fonjztsynewldfba%40gmail.com
[4]: https://www.postgresql.org/message-id/flat/3741749.1655952719%40sss.pgh.pa.us#2bb7af4a3d2c7669f9a49808d777a20d
[5]: https://www.postgresql.org/message-id/20180211211235.5x3jywe5z3lkgcsr%40alap3.anarazel.de
[6]: https://www.postgresql.org/message-id/3741749.1655952719%40sss.pgh.pa.us


Thanks,
-- 
kou

  2. Re: Make COPY format extendable: Extract COPY TO format implementations

    Nathan Bossart <nathandbossart@gmail.com> — 2023-12-05T18:24:58Z

    On Mon, Dec 04, 2023 at 03:35:48PM +0900, Sutou Kouhei wrote:
> I want to work on making COPY format extendable. I attach
> the first patch for it. I'll send more patches after this is
> merged.

Given the current discussion about adding JSON, I think this could be a
nice bit of refactoring that could ultimately open the door to providing
other COPY formats via shared libraries.

> In other words, this is just a refactoring for further
> changes to make COPY format extendable. If I use "complete
> the task and then request reviews for it" approach, it will
> be difficult to review because changes for it will be
> large. So I want to work on this step by step. Is it
> acceptable?

I think it makes sense to do this part independently, but we should be
careful to design this with the follow-up tasks in mind.

> So I measured COPY TO time with/without this change. You can
> see there is no significant loss of performance.
> 
> Data: Random 32 bit integers:
> 
>     CREATE TABLE data (int32 integer);
>     INSERT INTO data
>       SELECT random() * 10000
>         FROM generate_series(1, ${n_records});

Seems encouraging.  I assume the performance concerns stem from the use of
function pointers.  Or was there something else?

-- 
Nathan Bossart
Amazon Web Services: https://aws.amazon.com

  3. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2023-12-06T02:44:47Z

    Hi,

Thanks for replying to this proposal!

In <20231205182458.GC2757816@nathanxps13>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Tue, 5 Dec 2023 12:24:58 -0600,
  Nathan Bossart <nathandbossart@gmail.com> wrote:

> I think it makes sense to do this part independently, but we should be
> careful to design this with the follow-up tasks in mind.

OK. I'll keep updating the "TODOs" section in the original
e-mail. It also includes design in the follow-up tasks. We
can discuss the design separately from the patches
submitting. (The current submitted patch just focuses on
refactoring but we can discuss the final design.)

> I assume the performance concerns stem from the use of
> function pointers.  Or was there something else?

I think so too.

The original e-mail that mentioned the performance concern
[1] didn't say about the reason but the use of function
pointers might be concerned.

If the currently supported formats ("text", "csv" and
"binary") are implemented as an extension, it may have more
concerns but we will keep them as built-in formats for
compatibility. So I think that no more concerns exist for
these formats.


[1]: https://www.postgresql.org/message-id/flat/3741749.1655952719%40sss.pgh.pa.us#2bb7af4a3d2c7669f9a49808d777a20d


Thanks,
-- 
kou

  4. Re: Make COPY format extendable: Extract COPY TO format implementations

    Junwang Zhao <zhjwpku@gmail.com> — 2023-12-06T03:18:35Z

    On Wed, Dec 6, 2023 at 10:45 AM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> Thanks for replying to this proposal!
>
> In <20231205182458.GC2757816@nathanxps13>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Tue, 5 Dec 2023 12:24:58 -0600,
>   Nathan Bossart <nathandbossart@gmail.com> wrote:
>
> > I think it makes sense to do this part independently, but we should be
> > careful to design this with the follow-up tasks in mind.
>
> OK. I'll keep updating the "TODOs" section in the original
> e-mail. It also includes design in the follow-up tasks. We
> can discuss the design separately from the patches
> submitting. (The current submitted patch just focuses on
> refactoring but we can discuss the final design.)
>
> > I assume the performance concerns stem from the use of
> > function pointers.  Or was there something else?
>
> I think so too.
>
> The original e-mail that mentioned the performance concern
> [1] didn't say about the reason but the use of function
> pointers might be concerned.
>
> If the currently supported formats ("text", "csv" and
> "binary") are implemented as an extension, it may have more
> concerns but we will keep them as built-in formats for
> compatibility. So I think that no more concerns exist for
> these formats.
>

For the modern formats(parquet, orc, avro, etc.), will they be
implemented as extensions or in core?

The patch looks good except for a pair of extra curly braces.

>
> [1]: https://www.postgresql.org/message-id/flat/3741749.1655952719%40sss.pgh.pa.us#2bb7af4a3d2c7669f9a49808d777a20d
>
>
> Thanks,
> --
> kou
>
>


-- 
Regards
Junwang Zhao

  5. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2023-12-06T06:19:08Z

    Hi,

In <CAEG8a3Jf7kPV3ez5OHu-pFGscKfVyd9KkubMF199etkfz=EPRg@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Wed, 6 Dec 2023 11:18:35 +0800,
  Junwang Zhao <zhjwpku@gmail.com> wrote:

> For the modern formats(parquet, orc, avro, etc.), will they be
> implemented as extensions or in core?

I think that they should be implemented as extensions
because they will depend of external libraries and may not
use C. For example, C++ will be used for Apache Parquet
because the official Apache Parquet C++ implementation
exists but the C implementation doesn't.

(I can implement an extension for Apache Parquet after we
complete this feature. I'll implement an extension for
Apache Arrow with the official Apache Arrow C++
implementation. And it's easy that we convert Apache Arrow
data to Apache Parquet with the official Apache Parquet
implementation.)

> The patch looks good except for a pair of extra curly braces.

Thanks for the review! I attach the v2 patch that removes
extra curly braces for "if (isnull)".


Thanks,
-- 
kou

  6. Re: Make COPY format extendable: Extract COPY TO format implementations

    Junwang Zhao <zhjwpku@gmail.com> — 2023-12-06T07:11:34Z

    On Wed, Dec 6, 2023 at 2:19 PM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <CAEG8a3Jf7kPV3ez5OHu-pFGscKfVyd9KkubMF199etkfz=EPRg@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Wed, 6 Dec 2023 11:18:35 +0800,
>   Junwang Zhao <zhjwpku@gmail.com> wrote:
>
> > For the modern formats(parquet, orc, avro, etc.), will they be
> > implemented as extensions or in core?
>
> I think that they should be implemented as extensions
> because they will depend of external libraries and may not
> use C. For example, C++ will be used for Apache Parquet
> because the official Apache Parquet C++ implementation
> exists but the C implementation doesn't.
>
> (I can implement an extension for Apache Parquet after we
> complete this feature. I'll implement an extension for
> Apache Arrow with the official Apache Arrow C++
> implementation. And it's easy that we convert Apache Arrow
> data to Apache Parquet with the official Apache Parquet
> implementation.)
>
> > The patch looks good except for a pair of extra curly braces.
>
> Thanks for the review! I attach the v2 patch that removes
> extra curly braces for "if (isnull)".
>
For the extra curly braces, I mean the following code block in
CopyToFormatBinaryStart:

+ {        <-- I thought this is useless?
+ /* Generate header for a binary copy */
+ int32 tmp;
+
+ /* Signature */
+ CopySendData(cstate, BinarySignature, 11);
+ /* Flags field */
+ tmp = 0;
+ CopySendInt32(cstate, tmp);
+ /* No header extension */
+ tmp = 0;
+ CopySendInt32(cstate, tmp);
+ }

>
> Thanks,
> --
> kou



-- 
Regards
Junwang Zhao

  7. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2023-12-06T07:28:34Z

    Hi,

In <CAEG8a3K9dE2gt3+K+h=DwTqMenR84aeYuYS+cty3SR3LAeDBAQ@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Wed, 6 Dec 2023 15:11:34 +0800,
  Junwang Zhao <zhjwpku@gmail.com> wrote:

> For the extra curly braces, I mean the following code block in
> CopyToFormatBinaryStart:
> 
> + {        <-- I thought this is useless?
> + /* Generate header for a binary copy */
> + int32 tmp;
> +
> + /* Signature */
> + CopySendData(cstate, BinarySignature, 11);
> + /* Flags field */
> + tmp = 0;
> + CopySendInt32(cstate, tmp);
> + /* No header extension */
> + tmp = 0;
> + CopySendInt32(cstate, tmp);
> + }

Oh, I see. I've removed and attach the v3 patch. In general,
I don't change variable name and so on in this patch. I just
move codes in this patch. But I also removed the "tmp"
variable for this case because I think that the name isn't
suitable for larger scope. (I think that "tmp" is acceptable
in a small scope like the above code.)

New code:

/* Generate header for a binary copy */
/* Signature */
CopySendData(cstate, BinarySignature, 11);
/* Flags field */
CopySendInt32(cstate, 0);
/* No header extension */
CopySendInt32(cstate, 0);


Thanks,
-- 
kou

  8. Re: Make COPY format extendable: Extract COPY TO format implementations

    Daniel Verite <daniel@manitou-mail.org> — 2023-12-06T12:31:59Z

    	Sutou Kouhei wrote:

> * 2022-04: Apache Arrow [2]
> * 2018-02: Apache Avro, Apache Parquet and Apache ORC [3]
> 
> (FYI: I want to add support for Apache Arrow.)
> 
> There were discussions how to add support for more formats. [3][4]
> In these discussions, we got a consensus about making COPY
> format extendable.


These formats seem all column-oriented whereas COPY is row-oriented
at the protocol level [1].
With regard to the procotol, how would it work to support these formats?


[1] https://www.postgresql.org/docs/current/protocol-flow.html#PROTOCOL-COPY


Best regards,
-- 
Daniel Vérité
https://postgresql.verite.pro/
Twitter: @DanielVerite

  9. Re: Make COPY format extendable: Extract COPY TO format implementations

    Junwang Zhao <zhjwpku@gmail.com> — 2023-12-06T14:07:51Z

    On Wed, Dec 6, 2023 at 3:28 PM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <CAEG8a3K9dE2gt3+K+h=DwTqMenR84aeYuYS+cty3SR3LAeDBAQ@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Wed, 6 Dec 2023 15:11:34 +0800,
>   Junwang Zhao <zhjwpku@gmail.com> wrote:
>
> > For the extra curly braces, I mean the following code block in
> > CopyToFormatBinaryStart:
> >
> > + {        <-- I thought this is useless?
> > + /* Generate header for a binary copy */
> > + int32 tmp;
> > +
> > + /* Signature */
> > + CopySendData(cstate, BinarySignature, 11);
> > + /* Flags field */
> > + tmp = 0;
> > + CopySendInt32(cstate, tmp);
> > + /* No header extension */
> > + tmp = 0;
> > + CopySendInt32(cstate, tmp);
> > + }
>
> Oh, I see. I've removed and attach the v3 patch. In general,
> I don't change variable name and so on in this patch. I just
> move codes in this patch. But I also removed the "tmp"
> variable for this case because I think that the name isn't
> suitable for larger scope. (I think that "tmp" is acceptable
> in a small scope like the above code.)
>
> New code:
>
> /* Generate header for a binary copy */
> /* Signature */
> CopySendData(cstate, BinarySignature, 11);
> /* Flags field */
> CopySendInt32(cstate, 0);
> /* No header extension */
> CopySendInt32(cstate, 0);
>
>
> Thanks,
> --
> kou

Hi Kou,

I read the thread[1] you posted and I think Andres's suggestion sounds great.

Should we extract both *copy to* and *copy from* for the first step, in that
case we can add the pg_copy_handler catalog smoothly later.

Attached V4 adds 'extract copy from' and it passed the cirrus ci,
please take a look.

I added a hook *copy_from_end* but this might be removed later if not used.

[1]: https://www.postgresql.org/message-id/20180211211235.5x3jywe5z3lkgcsr%40alap3.anarazel.de
-- 
Regards
Junwang Zhao

  10. Re: Make COPY format extendable: Extract COPY TO format implementations

    Junwang Zhao <zhjwpku@gmail.com> — 2023-12-06T14:32:14Z

    On Wed, Dec 6, 2023 at 8:32 PM Daniel Verite <daniel@manitou-mail.org> wrote:
>
>         Sutou Kouhei wrote:
>
> > * 2022-04: Apache Arrow [2]
> > * 2018-02: Apache Avro, Apache Parquet and Apache ORC [3]
> >
> > (FYI: I want to add support for Apache Arrow.)
> >
> > There were discussions how to add support for more formats. [3][4]
> > In these discussions, we got a consensus about making COPY
> > format extendable.
>
>
> These formats seem all column-oriented whereas COPY is row-oriented
> at the protocol level [1].
> With regard to the procotol, how would it work to support these formats?
>

They have kind of *RowGroup* concepts, a bunch of rows goes to a RowBatch
and the data of the same column goes together.

I think they should fit the COPY semantics and there are some  FDW out there for
these modern formats, like [1]. If we support COPY to deal with the
format, it will
be easier to interact with them(without creating
server/usermapping/foreign table).

[1]: https://github.com/adjust/parquet_fdw

>
> [1] https://www.postgresql.org/docs/current/protocol-flow.html#PROTOCOL-COPY
>
>
> Best regards,
> --
> Daniel Vérité
> https://postgresql.verite.pro/
> Twitter: @DanielVerite
>
>


-- 
Regards
Junwang Zhao

  11. Re: Make COPY format extendable: Extract COPY TO format implementations

    Michael Paquier <michael@paquier.xyz> — 2023-12-07T00:38:59Z

    On Wed, Dec 06, 2023 at 10:07:51PM +0800, Junwang Zhao wrote:
> I read the thread[1] you posted and I think Andres's suggestion sounds great.
> 
> Should we extract both *copy to* and *copy from* for the first step, in that
> case we can add the pg_copy_handler catalog smoothly later.
> 
> Attached V4 adds 'extract copy from' and it passed the cirrus ci,
> please take a look.
> 
> I added a hook *copy_from_end* but this might be removed later if not used.
> 
> [1]: https://www.postgresql.org/message-id/20180211211235.5x3jywe5z3lkgcsr%40alap3.anarazel.de

I was looking at the differences between v3 posted by Sutou-san and
v4 from you, seeing that:

+/* Routines for a COPY HANDLER implementation. */
+typedef struct CopyHandlerOps
 {
     /* Called when COPY TO is started. This will send a header. */
-    void        (*start) (CopyToState cstate, TupleDesc tupDesc);
+    void        (*copy_to_start) (CopyToState cstate, TupleDesc tupDesc);
 
     /* Copy one row for COPY TO. */
-    void        (*one_row) (CopyToState cstate, TupleTableSlot *slot);
+    void        (*copy_to_one_row) (CopyToState cstate, TupleTableSlot *slot);
 
     /* Called when COPY TO is ended. This will send a trailer. */
-    void        (*end) (CopyToState cstate);
-} CopyToFormatOps;
+    void        (*copy_to_end) (CopyToState cstate);
+
+    void        (*copy_from_start) (CopyFromState cstate, TupleDesc tupDesc);
+    bool        (*copy_from_next) (CopyFromState cstate, ExprContext *econtext,
+                                    Datum *values, bool *nulls);
+    void        (*copy_from_error_callback) (CopyFromState cstate);
+    void        (*copy_from_end) (CopyFromState cstate);
+} CopyHandlerOps;

And we've spent a good deal of time refactoring the copy code so as
the logic behind TO and FROM is split.  Having a set of routines that
groups both does not look like a step in the right direction to me,
and v4 is an attempt at solving two problems, while v3 aims to improve
one case.  It seems to me that each callback portion should be focused
on staying in its own area of the code, aka copyfrom*.c or copyto*.c.
--
Michael

  12. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2023-12-07T05:04:58Z

    Hi,

In <CAEG8a3LSRhK601Bn50u71BgfNWm4q3kv-o-KEq=hrbyLbY_EsA@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Wed, 6 Dec 2023 22:07:51 +0800,
  Junwang Zhao <zhjwpku@gmail.com> wrote:

> Should we extract both *copy to* and *copy from* for the first step, in that
> case we can add the pg_copy_handler catalog smoothly later.

I don't object it (mixing TO/FROM changes to one patch) but
it may make review difficult. Is it acceptable?

FYI: I planed that I implement TO part, and then FROM part,
and then unify TO/FROM parts if needed. [1]

> Attached V4 adds 'extract copy from' and it passed the cirrus ci,
> please take a look.

Thanks. Here are my comments:

> +		/*
> +			* Error is relevant to a particular line.
> +			*
> +			* If line_buf still contains the correct line, print it.
> +			*/
> +		if (cstate->line_buf_valid)

We need to fix the indentation.

> +CopyFromFormatBinaryStart(CopyFromState cstate, TupleDesc tupDesc)
> +{
> +	FmgrInfo   *in_functions;
> +	Oid		   *typioparams;
> +	Oid			in_func_oid;
> +	AttrNumber	num_phys_attrs;
> +
> +	/*
> +	 * Pick up the required catalog information for each attribute in the
> +	 * relation, including the input function, the element type (to pass to
> +	 * the input function), and info about defaults and constraints. (Which
> +	 * input function we use depends on text/binary format choice.)
> +	 */
> +	num_phys_attrs = tupDesc->natts;
> +	in_functions = (FmgrInfo *) palloc(num_phys_attrs * sizeof(FmgrInfo));
> +	typioparams = (Oid *) palloc(num_phys_attrs * sizeof(Oid));

We need to update the comment because defaults and
constraints aren't picked up here.

> +CopyFromFormatTextStart(CopyFromState cstate, TupleDesc tupDesc)
...
> +	/*
> +	 * Pick up the required catalog information for each attribute in the
> +	 * relation, including the input function, the element type (to pass to
> +	 * the input function), and info about defaults and constraints. (Which
> +	 * input function we use depends on text/binary format choice.)
> +	 */
> +	in_functions = (FmgrInfo *) palloc(num_phys_attrs * sizeof(FmgrInfo));
> +	typioparams = (Oid *) palloc(num_phys_attrs * sizeof(Oid));

ditto.


> @@ -1716,15 +1776,6 @@ BeginCopyFrom(ParseState *pstate,
>  		ReceiveCopyBinaryHeader(cstate);
>  	}

I think that this block should be moved to
CopyFromFormatBinaryStart() too. But we need to run it after
we setup inputs such as data_source_cb, pipe and filename...

+/* Routines for a COPY HANDLER implementation. */
+typedef struct CopyHandlerOps
+{
+	/* Called when COPY TO is started. This will send a header. */
+	void		(*copy_to_start) (CopyToState cstate, TupleDesc tupDesc);
+
+	/* Copy one row for COPY TO. */
+	void		(*copy_to_one_row) (CopyToState cstate, TupleTableSlot *slot);
+
+	/* Called when COPY TO is ended. This will send a trailer. */
+	void		(*copy_to_end) (CopyToState cstate);
+
+	void		(*copy_from_start) (CopyFromState cstate, TupleDesc tupDesc);
+	bool		(*copy_from_next) (CopyFromState cstate, ExprContext *econtext,
+			 					   Datum *values, bool *nulls);
+	void		(*copy_from_error_callback) (CopyFromState cstate);
+	void		(*copy_from_end) (CopyFromState cstate);
+} CopyHandlerOps;

It seems that "copy_" prefix is redundant. Should we use
"to_start" instead of "copy_to_start" and so on?

BTW, it seems that "COPY FROM (FORMAT json)" may not be implemented. [2]
We may need to care about NULL copy_from_* cases.


> I added a hook *copy_from_end* but this might be removed later if not used.

It may be useful to clean up resources for COPY FROM but the
patch doesn't call the copy_from_end. How about removing it
for now? We can add it and call it from EndCopyFrom() later?
Because it's not needed for now.

I think that we should focus on refactoring instead of
adding a new feature in this patch.


[1]: https://www.postgresql.org/message-id/20231204.153548.2126325458835528809.kou%40clear-code.com
[2]: https://www.postgresql.org/message-id/flat/CALvfUkBxTYy5uWPFVwpk_7ii2zgT07t3d-yR_cy4sfrrLU%3Dkcg%40mail.gmail.com


Thanks,
-- 
kou

  13. Re: Make COPY format extendable: Extract COPY TO format implementations

    Junwang Zhao <zhjwpku@gmail.com> — 2023-12-07T08:37:36Z

    On Thu, Dec 7, 2023 at 8:39 AM Michael Paquier <michael@paquier.xyz> wrote:
>
> On Wed, Dec 06, 2023 at 10:07:51PM +0800, Junwang Zhao wrote:
> > I read the thread[1] you posted and I think Andres's suggestion sounds great.
> >
> > Should we extract both *copy to* and *copy from* for the first step, in that
> > case we can add the pg_copy_handler catalog smoothly later.
> >
> > Attached V4 adds 'extract copy from' and it passed the cirrus ci,
> > please take a look.
> >
> > I added a hook *copy_from_end* but this might be removed later if not used.
> >
> > [1]: https://www.postgresql.org/message-id/20180211211235.5x3jywe5z3lkgcsr%40alap3.anarazel.de
>
> I was looking at the differences between v3 posted by Sutou-san and
> v4 from you, seeing that:
>
> +/* Routines for a COPY HANDLER implementation. */
> +typedef struct CopyHandlerOps
>  {
>      /* Called when COPY TO is started. This will send a header. */
> -    void        (*start) (CopyToState cstate, TupleDesc tupDesc);
> +    void        (*copy_to_start) (CopyToState cstate, TupleDesc tupDesc);
>
>      /* Copy one row for COPY TO. */
> -    void        (*one_row) (CopyToState cstate, TupleTableSlot *slot);
> +    void        (*copy_to_one_row) (CopyToState cstate, TupleTableSlot *slot);
>
>      /* Called when COPY TO is ended. This will send a trailer. */
> -    void        (*end) (CopyToState cstate);
> -} CopyToFormatOps;
> +    void        (*copy_to_end) (CopyToState cstate);
> +
> +    void        (*copy_from_start) (CopyFromState cstate, TupleDesc tupDesc);
> +    bool        (*copy_from_next) (CopyFromState cstate, ExprContext *econtext,
> +                                    Datum *values, bool *nulls);
> +    void        (*copy_from_error_callback) (CopyFromState cstate);
> +    void        (*copy_from_end) (CopyFromState cstate);
> +} CopyHandlerOps;
>
> And we've spent a good deal of time refactoring the copy code so as
> the logic behind TO and FROM is split.  Having a set of routines that
> groups both does not look like a step in the right direction to me,

The point of this refactor (from my view) is to make it possible to add new
copy handlers in extensions, just like access method. As Andres suggested,
a system catalog like *pg_copy_handler*, if we split TO and FROM into two
sets of routines, does that mean we have to create two catalog(
pg_copy_from_handler and pg_copy_to_handler)?

> and v4 is an attempt at solving two problems, while v3 aims to improve
> one case.  It seems to me that each callback portion should be focused
> on staying in its own area of the code, aka copyfrom*.c or copyto*.c.
> --
> Michael



-- 
Regards
Junwang Zhao

  14. Re: Make COPY format extendable: Extract COPY TO format implementations

    Junwang Zhao <zhjwpku@gmail.com> — 2023-12-07T08:46:53Z

    On Thu, Dec 7, 2023 at 1:05 PM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <CAEG8a3LSRhK601Bn50u71BgfNWm4q3kv-o-KEq=hrbyLbY_EsA@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Wed, 6 Dec 2023 22:07:51 +0800,
>   Junwang Zhao <zhjwpku@gmail.com> wrote:
>
> > Should we extract both *copy to* and *copy from* for the first step, in that
> > case we can add the pg_copy_handler catalog smoothly later.
>
> I don't object it (mixing TO/FROM changes to one patch) but
> it may make review difficult. Is it acceptable?
>
> FYI: I planed that I implement TO part, and then FROM part,
> and then unify TO/FROM parts if needed. [1]

I'm fine with step by step refactoring, let's just wait for more
suggestions.

>
> > Attached V4 adds 'extract copy from' and it passed the cirrus ci,
> > please take a look.
>
> Thanks. Here are my comments:
>
> > +             /*
> > +                     * Error is relevant to a particular line.
> > +                     *
> > +                     * If line_buf still contains the correct line, print it.
> > +                     */
> > +             if (cstate->line_buf_valid)
>
> We need to fix the indentation.
>
> > +CopyFromFormatBinaryStart(CopyFromState cstate, TupleDesc tupDesc)
> > +{
> > +     FmgrInfo   *in_functions;
> > +     Oid                *typioparams;
> > +     Oid                     in_func_oid;
> > +     AttrNumber      num_phys_attrs;
> > +
> > +     /*
> > +      * Pick up the required catalog information for each attribute in the
> > +      * relation, including the input function, the element type (to pass to
> > +      * the input function), and info about defaults and constraints. (Which
> > +      * input function we use depends on text/binary format choice.)
> > +      */
> > +     num_phys_attrs = tupDesc->natts;
> > +     in_functions = (FmgrInfo *) palloc(num_phys_attrs * sizeof(FmgrInfo));
> > +     typioparams = (Oid *) palloc(num_phys_attrs * sizeof(Oid));
>
> We need to update the comment because defaults and
> constraints aren't picked up here.
>
> > +CopyFromFormatTextStart(CopyFromState cstate, TupleDesc tupDesc)
> ...
> > +     /*
> > +      * Pick up the required catalog information for each attribute in the
> > +      * relation, including the input function, the element type (to pass to
> > +      * the input function), and info about defaults and constraints. (Which
> > +      * input function we use depends on text/binary format choice.)
> > +      */
> > +     in_functions = (FmgrInfo *) palloc(num_phys_attrs * sizeof(FmgrInfo));
> > +     typioparams = (Oid *) palloc(num_phys_attrs * sizeof(Oid));
>
> ditto.
>
>
> > @@ -1716,15 +1776,6 @@ BeginCopyFrom(ParseState *pstate,
> >               ReceiveCopyBinaryHeader(cstate);
> >       }
>
> I think that this block should be moved to
> CopyFromFormatBinaryStart() too. But we need to run it after
> we setup inputs such as data_source_cb, pipe and filename...
>
> +/* Routines for a COPY HANDLER implementation. */
> +typedef struct CopyHandlerOps
> +{
> +       /* Called when COPY TO is started. This will send a header. */
> +       void            (*copy_to_start) (CopyToState cstate, TupleDesc tupDesc);
> +
> +       /* Copy one row for COPY TO. */
> +       void            (*copy_to_one_row) (CopyToState cstate, TupleTableSlot *slot);
> +
> +       /* Called when COPY TO is ended. This will send a trailer. */
> +       void            (*copy_to_end) (CopyToState cstate);
> +
> +       void            (*copy_from_start) (CopyFromState cstate, TupleDesc tupDesc);
> +       bool            (*copy_from_next) (CopyFromState cstate, ExprContext *econtext,
> +                                                                  Datum *values, bool *nulls);
> +       void            (*copy_from_error_callback) (CopyFromState cstate);
> +       void            (*copy_from_end) (CopyFromState cstate);
> +} CopyHandlerOps;
>
> It seems that "copy_" prefix is redundant. Should we use
> "to_start" instead of "copy_to_start" and so on?
>
> BTW, it seems that "COPY FROM (FORMAT json)" may not be implemented. [2]
> We may need to care about NULL copy_from_* cases.
>
>
> > I added a hook *copy_from_end* but this might be removed later if not used.
>
> It may be useful to clean up resources for COPY FROM but the
> patch doesn't call the copy_from_end. How about removing it
> for now? We can add it and call it from EndCopyFrom() later?
> Because it's not needed for now.
>
> I think that we should focus on refactoring instead of
> adding a new feature in this patch.
>
>
> [1]: https://www.postgresql.org/message-id/20231204.153548.2126325458835528809.kou%40clear-code.com
> [2]: https://www.postgresql.org/message-id/flat/CALvfUkBxTYy5uWPFVwpk_7ii2zgT07t3d-yR_cy4sfrrLU%3Dkcg%40mail.gmail.com
>
>
> Thanks,
> --
> kou



-- 
Regards
Junwang Zhao

  15. Re: Make COPY format extendable: Extract COPY TO format implementations

    Andrew Dunstan <andrew@dunslane.net> — 2023-12-07T16:38:47Z

    On 2023-12-07 Th 03:37, Junwang Zhao wrote:
>
> The point of this refactor (from my view) is to make it possible to add new
> copy handlers in extensions, just like access method. As Andres suggested,
> a system catalog like *pg_copy_handler*, if we split TO and FROM into two
> sets of routines, does that mean we have to create two catalog(
> pg_copy_from_handler and pg_copy_to_handler)?



Surely not. Either have two fields, one for the TO handler and one for 
the FROM handler, or a flag on each row indicating if it's a FROM or TO 
handler.


cheers


andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

  16. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2023-12-07T19:27:14Z

    On Fri, Dec 8, 2023 at 1:39 AM Andrew Dunstan <andrew@dunslane.net> wrote:
>
>
> On 2023-12-07 Th 03:37, Junwang Zhao wrote:
> >
> > The point of this refactor (from my view) is to make it possible to add new
> > copy handlers in extensions, just like access method. As Andres suggested,
> > a system catalog like *pg_copy_handler*, if we split TO and FROM into two
> > sets of routines, does that mean we have to create two catalog(
> > pg_copy_from_handler and pg_copy_to_handler)?
>
>
>
> Surely not. Either have two fields, one for the TO handler and one for
> the FROM handler, or a flag on each row indicating if it's a FROM or TO
> handler.

True.

But why do we need a system catalog like pg_copy_handler in the first
place? I imagined that an extension can define a handler function
returning a set of callbacks and the parser can lookup the handler
function by name, like FDW and TABLESAMPLE.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  17. Re: Make COPY format extendable: Extract COPY TO format implementations

    Junwang Zhao <zhjwpku@gmail.com> — 2023-12-08T02:32:27Z

    On Fri, Dec 8, 2023 at 3:27 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Fri, Dec 8, 2023 at 1:39 AM Andrew Dunstan <andrew@dunslane.net> wrote:
> >
> >
> > On 2023-12-07 Th 03:37, Junwang Zhao wrote:
> > >
> > > The point of this refactor (from my view) is to make it possible to add new
> > > copy handlers in extensions, just like access method. As Andres suggested,
> > > a system catalog like *pg_copy_handler*, if we split TO and FROM into two
> > > sets of routines, does that mean we have to create two catalog(
> > > pg_copy_from_handler and pg_copy_to_handler)?
> >
> >
> >
> > Surely not. Either have two fields, one for the TO handler and one for
> > the FROM handler, or a flag on each row indicating if it's a FROM or TO
> > handler.

If we wrap the two fields into a single structure, that will still be in
copy.h, which I think is not necessary. A single routing wrapper should
be enough, the actual implementation still stays separate
copy_[to/from].c files.

>
> True.
>
> But why do we need a system catalog like pg_copy_handler in the first
> place? I imagined that an extension can define a handler function
> returning a set of callbacks and the parser can lookup the handler
> function by name, like FDW and TABLESAMPLE.
>
I can see FDW related utility commands but no TABLESAMPLE related,
and there is a pg_foreign_data_wrapper system catalog which has
a *fdwhandler* field.

If we want extensions to create a new copy handler, I think
something like pg_copy_hander should be necessary.

> Regards,
>
> --
> Masahiko Sawada
> Amazon Web Services: https://aws.amazon.com

I go one step further to implement the pg_copy_handler, attached V5 is
the implementation with some changes suggested by Kou.

You can also review this on this github pull request [1].

[1]: https://github.com/zhjwpku/postgres/pull/1/files

-- 
Regards
Junwang Zhao

  18. Re: Make COPY format extendable: Extract COPY TO format implementations

    Michael Paquier <michael@paquier.xyz> — 2023-12-08T05:17:42Z

    On Fri, Dec 08, 2023 at 10:32:27AM +0800, Junwang Zhao wrote:
> I can see FDW related utility commands but no TABLESAMPLE related,
> and there is a pg_foreign_data_wrapper system catalog which has
> a *fdwhandler* field.

+ */ +CATALOG(pg_copy_handler,4551,CopyHandlerRelationId)

Using a catalog is an over-engineered design.  Others have provided
hints about that upthread, but it would be enough to have one or two
handler types that are wrapped around one or two SQL *functions*, like
tablesamples.  It seems like you've missed it, but feel free to read
about tablesample-method.sgml, that explains how this is achieved for
tablesamples.

> If we want extensions to create a new copy handler, I think
> something like pg_copy_hander should be necessary.

A catalog is not necessary, that's the point, because it can be
replaced by a scan of pg_proc with the function name defined in a COPY
query (be it through a FORMAT, or different option in a DefElem).
An example of extension with tablesamples is contrib/tsm_system_rows/,
that just uses a function returning a tsm_handler: 
CREATE FUNCTION system_rows(internal)
RETURNS tsm_handler
AS 'MODULE_PATHNAME', 'tsm_system_rows_handler'
LANGUAGE C STRICT;

Then SELECT queries rely on the contents of the TABLESAMPLE clause to
find the set of callbacks it should use by calling the function.

+/* Routines for a COPY HANDLER implementation. */
+typedef struct CopyRoutine
+{

FWIW, I find weird the concept of having one handler for both COPY
FROM and COPY TO as each one of them has callbacks that are mutually
exclusive to the other, but I'm OK if there is a consensus of only
one.  So I'd suggest to use *two* NodeTags instead for a cleaner
split, meaning that we'd need two functions for each method.  My point
is that a custom COPY handler could just define a COPY TO handler or a
COPY FROM handler, though it mostly comes down to a matter of taste
regarding how clean the error handling becomes if one tries to use a
set of callbacks with a COPY type (TO or FROM) not matching it.
--
Michael

  19. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2023-12-08T06:42:06Z

    On Fri, Dec 8, 2023 at 2:17 PM Michael Paquier <michael@paquier.xyz> wrote:
>
> On Fri, Dec 08, 2023 at 10:32:27AM +0800, Junwang Zhao wrote:
> > I can see FDW related utility commands but no TABLESAMPLE related,
> > and there is a pg_foreign_data_wrapper system catalog which has
> > a *fdwhandler* field.
>
> + */ +CATALOG(pg_copy_handler,4551,CopyHandlerRelationId)
>
> Using a catalog is an over-engineered design.  Others have provided
> hints about that upthread, but it would be enough to have one or two
> handler types that are wrapped around one or two SQL *functions*, like
> tablesamples.  It seems like you've missed it, but feel free to read
> about tablesample-method.sgml, that explains how this is achieved for
> tablesamples.

Agreed. My previous example of FDW was not a good one, I missed something.

>
> > If we want extensions to create a new copy handler, I think
> > something like pg_copy_hander should be necessary.
>
> A catalog is not necessary, that's the point, because it can be
> replaced by a scan of pg_proc with the function name defined in a COPY
> query (be it through a FORMAT, or different option in a DefElem).
> An example of extension with tablesamples is contrib/tsm_system_rows/,
> that just uses a function returning a tsm_handler:
> CREATE FUNCTION system_rows(internal)
> RETURNS tsm_handler
> AS 'MODULE_PATHNAME', 'tsm_system_rows_handler'
> LANGUAGE C STRICT;
>
> Then SELECT queries rely on the contents of the TABLESAMPLE clause to
> find the set of callbacks it should use by calling the function.
>
> +/* Routines for a COPY HANDLER implementation. */
> +typedef struct CopyRoutine
> +{
>
> FWIW, I find weird the concept of having one handler for both COPY
> FROM and COPY TO as each one of them has callbacks that are mutually
> exclusive to the other, but I'm OK if there is a consensus of only
> one.  So I'd suggest to use *two* NodeTags instead for a cleaner
> split, meaning that we'd need two functions for each method.  My point
> is that a custom COPY handler could just define a COPY TO handler or a
> COPY FROM handler, though it mostly comes down to a matter of taste
> regarding how clean the error handling becomes if one tries to use a
> set of callbacks with a COPY type (TO or FROM) not matching it.

I tend to agree to have separate two functions for each method. But
given we implement it in tablesample-way, I think we need to make it
clear how to call one of the two functions depending on COPY TO and
FROM.

IIUC in tablesamples cases, we scan pg_proc to find the handler
function like system_rows(internal) by the method name specified in
the query. On the other hand, in COPY cases, the queries would be
going to be like:

COPY tab TO stdout WITH (format = 'arrow');
and
COPY tab FROM stdin WITH (format = 'arrow');

So a custom COPY extension would not be able to define SQL functions
just like arrow(internal) for example. We might need to define a rule
like the function returning copy_in/out_handler must be defined as
<method name>_to(internal) and <method_name>_from(internal).

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  20. Re: Make COPY format extendable: Extract COPY TO format implementations

    Michael Paquier <michael@paquier.xyz> — 2023-12-08T07:02:55Z

    On Fri, Dec 08, 2023 at 03:42:06PM +0900, Masahiko Sawada wrote:
> So a custom COPY extension would not be able to define SQL functions
> just like arrow(internal) for example. We might need to define a rule
> like the function returning copy_in/out_handler must be defined as
> <method name>_to(internal) and <method_name>_from(internal).

Yeah, I was wondering if there was a trick to avoid the input internal
argument conflict, but cannot recall something elegant on the top of
my mind.  Anyway, I'd be OK with any approach as long as it plays
nicely with the query integration, and that's FORMAT's DefElem with
its string value to do the function lookups.
--
Michael

  21. RE: Make COPY format extendable: Extract COPY TO format implementations

    Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com> — 2023-12-09T02:43:49Z

    Dear Junagn, Sutou-san,

Basically I agree your point - improving a extendibility is good.
(I remember that this theme was talked at Japan PostgreSQL conference)
Below are my comments for your patch.

01. General

Just to confirm - is it OK to partially implement APIs? E.g., only COPY TO is
available. Currently it seems not to consider a case which is not implemented.

02. General

It might be trivial, but could you please clarify how users can extend? Is it OK
to do below steps?

1. Create a handler function, via CREATE FUNCTION,
2. Register a handler, via new SQL (CREATE COPY HANDLER),
3. Specify the added handler as COPY ... FORMAT clause.

03. General

Could you please add document-related tasks to your TODO? I imagined like
fdwhandler.sgml.

04. General - copyright

For newly added files, the below copyright seems sufficient. See applyparallelworker.c.

```
 * Copyright (c) 2023, PostgreSQL Global Development Group
```

05. src/include/catalog/* files

IIUC, 8000 or higher OIDs should be used while developing a patch. src/include/catalog/unused_oids
would suggest a candidate which you can use.

06. copy.c

I felt that we can create files per copying methods, like copy_{text|csv|binary}.c,
like indexes.
How do other think?

07. fmt_to_name()

I'm not sure the function is really needed. Can we follow like get_foreign_data_wrapper_oid()
and remove the funciton?

08. GetCopyRoutineByName()

Should we use syscache for searching a catalog?

09. CopyToFormatTextSendEndOfRow(), CopyToFormatBinaryStart()

Comments still refer CopyHandlerOps, whereas it was renamed.

10. copy.h

Per foreign.h and fdwapi.h, should we add a new header file and move some APIs?

11. copy.h

```
-/* These are private in commands/copy[from|to].c */
-typedef struct CopyFromStateData *CopyFromState;
-typedef struct CopyToStateData *CopyToState;
```

Are above changes really needed?

12. CopyFormatOptions

Can we remove `bool binary` in future?

13. external functions

```
+extern void CopyToFormatTextStart(CopyToState cstate, TupleDesc tupDesc);
+extern void CopyToFormatTextOneRow(CopyToState cstate, TupleTableSlot *slot);
+extern void CopyToFormatTextEnd(CopyToState cstate);
+extern void CopyFromFormatTextStart(CopyFromState cstate, TupleDesc tupDesc);
+extern bool CopyFromFormatTextNext(CopyFromState cstate, ExprContext *econtext,
+
Datum *values, bool *nulls);
+extern void CopyFromFormatTextErrorCallback(CopyFromState cstate);
+
+extern void CopyToFormatBinaryStart(CopyToState cstate, TupleDesc tupDesc);
+extern void CopyToFormatBinaryOneRow(CopyToState cstate, TupleTableSlot *slot);
+extern void CopyToFormatBinaryEnd(CopyToState cstate);
+extern void CopyFromFormatBinaryStart(CopyFromState cstate, TupleDesc tupDesc);
+extern bool CopyFromFormatBinaryNext(CopyFromState cstate,
ExprContext *econtext,
+
  Datum *values, bool *nulls);
+extern void CopyFromFormatBinaryErrorCallback(CopyFromState cstate);
```

FYI - If you add files for {text|csv|binary}, these declarations can be removed.

Best Regards,
Hayato Kuroda
FUJITSU LIMITED

  22. Re: Make COPY format extendable: Extract COPY TO format implementations

    Junwang Zhao <zhjwpku@gmail.com> — 2023-12-09T08:39:11Z

    On Sat, Dec 9, 2023 at 10:43 AM Hayato Kuroda (Fujitsu)
<kuroda.hayato@fujitsu.com> wrote:
>
> Dear Junagn, Sutou-san,
>
> Basically I agree your point - improving a extendibility is good.
> (I remember that this theme was talked at Japan PostgreSQL conference)
> Below are my comments for your patch.
>
> 01. General
>
> Just to confirm - is it OK to partially implement APIs? E.g., only COPY TO is
> available. Currently it seems not to consider a case which is not implemented.
>
For partially implements, we can leave the hook as NULL, and check the NULL
at *ProcessCopyOptions* and report error if not supported.

> 02. General
>
> It might be trivial, but could you please clarify how users can extend? Is it OK
> to do below steps?
>
> 1. Create a handler function, via CREATE FUNCTION,
> 2. Register a handler, via new SQL (CREATE COPY HANDLER),
> 3. Specify the added handler as COPY ... FORMAT clause.
>
My original thought was option 2, but as Michael point, option 1 is
the right way
to go.

> 03. General
>
> Could you please add document-related tasks to your TODO? I imagined like
> fdwhandler.sgml.
>
> 04. General - copyright
>
> For newly added files, the below copyright seems sufficient. See applyparallelworker.c.
>
> ```
>  * Copyright (c) 2023, PostgreSQL Global Development Group
> ```
>
> 05. src/include/catalog/* files
>
> IIUC, 8000 or higher OIDs should be used while developing a patch. src/include/catalog/unused_oids
> would suggest a candidate which you can use.

Yeah, I will run renumber_oids.pl at last.

>
> 06. copy.c
>
> I felt that we can create files per copying methods, like copy_{text|csv|binary}.c,
> like indexes.
> How do other think?

Not sure about this, it seems others have put a lot of effort into
splitting TO and From.
Also like to hear from others.

>
> 07. fmt_to_name()
>
> I'm not sure the function is really needed. Can we follow like get_foreign_data_wrapper_oid()
> and remove the funciton?

I have referenced some code from greenplum, will remove this.

>
> 08. GetCopyRoutineByName()
>
> Should we use syscache for searching a catalog?
>
> 09. CopyToFormatTextSendEndOfRow(), CopyToFormatBinaryStart()
>
> Comments still refer CopyHandlerOps, whereas it was renamed.
>
> 10. copy.h
>
> Per foreign.h and fdwapi.h, should we add a new header file and move some APIs?
>
> 11. copy.h
>
> ```
> -/* These are private in commands/copy[from|to].c */
> -typedef struct CopyFromStateData *CopyFromState;
> -typedef struct CopyToStateData *CopyToState;
> ```
>
> Are above changes really needed?
>
> 12. CopyFormatOptions
>
> Can we remove `bool binary` in future?
>
> 13. external functions
>
> ```
> +extern void CopyToFormatTextStart(CopyToState cstate, TupleDesc tupDesc);
> +extern void CopyToFormatTextOneRow(CopyToState cstate, TupleTableSlot *slot);
> +extern void CopyToFormatTextEnd(CopyToState cstate);
> +extern void CopyFromFormatTextStart(CopyFromState cstate, TupleDesc tupDesc);
> +extern bool CopyFromFormatTextNext(CopyFromState cstate, ExprContext *econtext,
> +
> Datum *values, bool *nulls);
> +extern void CopyFromFormatTextErrorCallback(CopyFromState cstate);
> +
> +extern void CopyToFormatBinaryStart(CopyToState cstate, TupleDesc tupDesc);
> +extern void CopyToFormatBinaryOneRow(CopyToState cstate, TupleTableSlot *slot);
> +extern void CopyToFormatBinaryEnd(CopyToState cstate);
> +extern void CopyFromFormatBinaryStart(CopyFromState cstate, TupleDesc tupDesc);
> +extern bool CopyFromFormatBinaryNext(CopyFromState cstate,
> ExprContext *econtext,
> +
>   Datum *values, bool *nulls);
> +extern void CopyFromFormatBinaryErrorCallback(CopyFromState cstate);
> ```
>
> FYI - If you add files for {text|csv|binary}, these declarations can be removed.
>
> Best Regards,
> Hayato Kuroda
> FUJITSU LIMITED
>

Thanks for all the valuable suggestions.

-- 
Regards
Junwang Zhao

  23. Re: Make COPY format extendable: Extract COPY TO format implementations

    Hannu Krosing <hannuk@google.com> — 2023-12-09T11:38:46Z

    Hi Junwang

Please also see my presentation slides from last years PostgreSQL
Conference in Berlin (attached)

The main Idea is to make not just "format", but also "transport" and
"stream processing" extendable via virtual function tables.

Btw, will any of you here be in Prague next week ?
Would be a good opportunity to discuss this in person.


Best Regards
Hannu

On Sat, Dec 9, 2023 at 9:39 AM Junwang Zhao <zhjwpku@gmail.com> wrote:
>
> On Sat, Dec 9, 2023 at 10:43 AM Hayato Kuroda (Fujitsu)
> <kuroda.hayato@fujitsu.com> wrote:
> >
> > Dear Junagn, Sutou-san,
> >
> > Basically I agree your point - improving a extendibility is good.
> > (I remember that this theme was talked at Japan PostgreSQL conference)
> > Below are my comments for your patch.
> >
> > 01. General
> >
> > Just to confirm - is it OK to partially implement APIs? E.g., only COPY TO is
> > available. Currently it seems not to consider a case which is not implemented.
> >
> For partially implements, we can leave the hook as NULL, and check the NULL
> at *ProcessCopyOptions* and report error if not supported.
>
> > 02. General
> >
> > It might be trivial, but could you please clarify how users can extend? Is it OK
> > to do below steps?
> >
> > 1. Create a handler function, via CREATE FUNCTION,
> > 2. Register a handler, via new SQL (CREATE COPY HANDLER),
> > 3. Specify the added handler as COPY ... FORMAT clause.
> >
> My original thought was option 2, but as Michael point, option 1 is
> the right way
> to go.
>
> > 03. General
> >
> > Could you please add document-related tasks to your TODO? I imagined like
> > fdwhandler.sgml.
> >
> > 04. General - copyright
> >
> > For newly added files, the below copyright seems sufficient. See applyparallelworker.c.
> >
> > ```
> >  * Copyright (c) 2023, PostgreSQL Global Development Group
> > ```
> >
> > 05. src/include/catalog/* files
> >
> > IIUC, 8000 or higher OIDs should be used while developing a patch. src/include/catalog/unused_oids
> > would suggest a candidate which you can use.
>
> Yeah, I will run renumber_oids.pl at last.
>
> >
> > 06. copy.c
> >
> > I felt that we can create files per copying methods, like copy_{text|csv|binary}.c,
> > like indexes.
> > How do other think?
>
> Not sure about this, it seems others have put a lot of effort into
> splitting TO and From.
> Also like to hear from others.
>
> >
> > 07. fmt_to_name()
> >
> > I'm not sure the function is really needed. Can we follow like get_foreign_data_wrapper_oid()
> > and remove the funciton?
>
> I have referenced some code from greenplum, will remove this.
>
> >
> > 08. GetCopyRoutineByName()
> >
> > Should we use syscache for searching a catalog?
> >
> > 09. CopyToFormatTextSendEndOfRow(), CopyToFormatBinaryStart()
> >
> > Comments still refer CopyHandlerOps, whereas it was renamed.
> >
> > 10. copy.h
> >
> > Per foreign.h and fdwapi.h, should we add a new header file and move some APIs?
> >
> > 11. copy.h
> >
> > ```
> > -/* These are private in commands/copy[from|to].c */
> > -typedef struct CopyFromStateData *CopyFromState;
> > -typedef struct CopyToStateData *CopyToState;
> > ```
> >
> > Are above changes really needed?
> >
> > 12. CopyFormatOptions
> >
> > Can we remove `bool binary` in future?
> >
> > 13. external functions
> >
> > ```
> > +extern void CopyToFormatTextStart(CopyToState cstate, TupleDesc tupDesc);
> > +extern void CopyToFormatTextOneRow(CopyToState cstate, TupleTableSlot *slot);
> > +extern void CopyToFormatTextEnd(CopyToState cstate);
> > +extern void CopyFromFormatTextStart(CopyFromState cstate, TupleDesc tupDesc);
> > +extern bool CopyFromFormatTextNext(CopyFromState cstate, ExprContext *econtext,
> > +
> > Datum *values, bool *nulls);
> > +extern void CopyFromFormatTextErrorCallback(CopyFromState cstate);
> > +
> > +extern void CopyToFormatBinaryStart(CopyToState cstate, TupleDesc tupDesc);
> > +extern void CopyToFormatBinaryOneRow(CopyToState cstate, TupleTableSlot *slot);
> > +extern void CopyToFormatBinaryEnd(CopyToState cstate);
> > +extern void CopyFromFormatBinaryStart(CopyFromState cstate, TupleDesc tupDesc);
> > +extern bool CopyFromFormatBinaryNext(CopyFromState cstate,
> > ExprContext *econtext,
> > +
> >   Datum *values, bool *nulls);
> > +extern void CopyFromFormatBinaryErrorCallback(CopyFromState cstate);
> > ```
> >
> > FYI - If you add files for {text|csv|binary}, these declarations can be removed.
> >
> > Best Regards,
> > Hayato Kuroda
> > FUJITSU LIMITED
> >
>
> Thanks for all the valuable suggestions.
>
> --
> Regards
> Junwang Zhao
>
>

  24. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2023-12-09T20:44:07Z

    Hi,

Thanks for reviewing our latest patch!

In 
 <TY3PR01MB9889C9234CD220A3A7075F0DF589A@TY3PR01MB9889.jpnprd01.prod.outlook.com>
  "RE: Make COPY format extendable: Extract COPY TO format implementations" on Sat, 9 Dec 2023 02:43:49 +0000,
  "Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com> wrote:

> (I remember that this theme was talked at Japan PostgreSQL conference)

Yes. I should have talked to you more at the conference...
I will do it next time!


Can we discuss how to proceed this improvement?

There are 2 approaches for it:

1. Do the followings concurrently:
   a. Implementing small changes that got a consensus and
      merge them step-by-step
      (e.g. We got a consensus that we need to extract the
      current format related routines.)
   b. Discuss design

   (v1-v3 patches use this approach.)

2. Implement one (large) complete patch set with design
   discussion and merge it

   (v4- patches use this approach.)

Which approach is preferred? (Or should we choose another
approach?)

I thought that 1. is preferred because it will reduce review
cost. So I chose 1.

If 2. is preferred, I'll use 2. (I'll add more changes to
the latest patch.)


Thanks,
-- 
kou

  25. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2023-12-09T20:54:56Z

    Hi,

In <CAD21AoDkoGL6yJ_HjNOg9cU=aAdW8uQ3rSQOeRS0SX85LPPNwQ@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 8 Dec 2023 15:42:06 +0900,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

> So a custom COPY extension would not be able to define SQL functions
> just like arrow(internal) for example. We might need to define a rule
> like the function returning copy_in/out_handler must be defined as
> <method name>_to(internal) and <method_name>_from(internal).

We may not need to add "_to"/"_from" suffix by checking both
of argument type and return type. Because we use different
return type for copy_in/out_handler.

But the current LookupFuncName() family doesn't check return
type. If we use this approach, we need to improve the
current LookupFuncName() family too.


Thanks,
-- 
kou

  26. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2023-12-09T21:01:36Z

    Hi,

In <CAMT0RQRqVo4fGDWHqOn+wr_eoiXQVfyC=8-c=H=y6VcNxi6BvQ@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Sat, 9 Dec 2023 12:38:46 +0100,
  Hannu Krosing <hannuk@google.com> wrote:

> Please also see my presentation slides from last years PostgreSQL
> Conference in Berlin (attached)

Thanks for sharing your idea here.

> The main Idea is to make not just "format", but also "transport" and
> "stream processing" extendable via virtual function tables.

"Transport" and "stream processing" are out of scope in this
thread. How about starting new threads for them and discuss
them there?

> Btw, will any of you here be in Prague next week ?

Sorry. No.


Thanks,
-- 
kou

  27. Re: Make COPY format extendable: Extract COPY TO format implementations

    Junwang Zhao <zhjwpku@gmail.com> — 2023-12-10T00:24:51Z

    On Sun, Dec 10, 2023 at 4:44 AM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> Thanks for reviewing our latest patch!
>
> In
>  <TY3PR01MB9889C9234CD220A3A7075F0DF589A@TY3PR01MB9889.jpnprd01.prod.outlook.com>
>   "RE: Make COPY format extendable: Extract COPY TO format implementations" on Sat, 9 Dec 2023 02:43:49 +0000,
>   "Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com> wrote:
>
> > (I remember that this theme was talked at Japan PostgreSQL conference)
>
> Yes. I should have talked to you more at the conference...
> I will do it next time!
>
>
> Can we discuss how to proceed this improvement?
>
> There are 2 approaches for it:
>
> 1. Do the followings concurrently:
>    a. Implementing small changes that got a consensus and
>       merge them step-by-step
>       (e.g. We got a consensus that we need to extract the
>       current format related routines.)
>    b. Discuss design
>
>    (v1-v3 patches use this approach.)
>
> 2. Implement one (large) complete patch set with design
>    discussion and merge it
>
>    (v4- patches use this approach.)
>
> Which approach is preferred? (Or should we choose another
> approach?)
>
> I thought that 1. is preferred because it will reduce review
> cost. So I chose 1.
>
> If 2. is preferred, I'll use 2. (I'll add more changes to
> the latest patch.)
>
I'm ok with both, and I'd like to work with you for the parquet
extension, excited about this new feature, thanks for bringing
this up.

Forgive me for making so much noise about approach 2, I
just want to hear about more suggestions of the final shape
of this feature.

>
> Thanks,
> --
> kou



-- 
Regards
Junwang Zhao

  28. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2023-12-11T00:36:38Z

    On Sun, Dec 10, 2023 at 5:44 AM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> Thanks for reviewing our latest patch!
>
> In
>  <TY3PR01MB9889C9234CD220A3A7075F0DF589A@TY3PR01MB9889.jpnprd01.prod.outlook.com>
>   "RE: Make COPY format extendable: Extract COPY TO format implementations" on Sat, 9 Dec 2023 02:43:49 +0000,
>   "Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com> wrote:
>
> > (I remember that this theme was talked at Japan PostgreSQL conference)
>
> Yes. I should have talked to you more at the conference...
> I will do it next time!
>
>
> Can we discuss how to proceed this improvement?
>
> There are 2 approaches for it:
>
> 1. Do the followings concurrently:
>    a. Implementing small changes that got a consensus and
>       merge them step-by-step
>       (e.g. We got a consensus that we need to extract the
>       current format related routines.)
>    b. Discuss design

It's preferable to make patches small for easy review. We can merge
them anytime before commit if necessary.

I think we need to discuss overall design about callbacks and how
extensions define a custom copy handler etc. It may require some PoC
patches. Once we have a consensus on overall design we polish patches
including the documentation changes and regression tests.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  29. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2023-12-11T01:57:15Z

    On Sun, Dec 10, 2023 at 5:55 AM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <CAD21AoDkoGL6yJ_HjNOg9cU=aAdW8uQ3rSQOeRS0SX85LPPNwQ@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 8 Dec 2023 15:42:06 +0900,
>   Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> > So a custom COPY extension would not be able to define SQL functions
> > just like arrow(internal) for example. We might need to define a rule
> > like the function returning copy_in/out_handler must be defined as
> > <method name>_to(internal) and <method_name>_from(internal).
>
> We may not need to add "_to"/"_from" suffix by checking both
> of argument type and return type. Because we use different
> return type for copy_in/out_handler.
>
> But the current LookupFuncName() family doesn't check return
> type. If we use this approach, we need to improve the
> current LookupFuncName() family too.

IIUC we cannot create two same name functions with the same arguments
but a different return value type in the first place. It seems to me
to be an overkill to change such a design.

Another idea is to encapsulate copy_to/from_handler by a super class
like copy_handler. The handler function is called with an argument,
say copyto, and returns copy_handler encapsulating either
copy_to/from_handler depending on the argument.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  30. Re: Make COPY format extendable: Extract COPY TO format implementations

    Michael Paquier <michael@paquier.xyz> — 2023-12-11T10:19:40Z

    On Mon, Dec 11, 2023 at 10:57:15AM +0900, Masahiko Sawada wrote:
> IIUC we cannot create two same name functions with the same arguments
> but a different return value type in the first place. It seems to me
> to be an overkill to change such a design.

Agreed to not touch the logictics of LookupFuncName() for the sake of
this thread.  I have not checked the SQL specification, but I recall
that there are a few assumptions from the spec embedded in the lookup
logic particularly when it comes to specify a procedure name without
arguments.

> Another idea is to encapsulate copy_to/from_handler by a super class
> like copy_handler. The handler function is called with an argument,
> say copyto, and returns copy_handler encapsulating either
> copy_to/from_handler depending on the argument.

Yep, that's possible as well and can work as a cross-check between the
argument and the NodeTag assigned to the handler structure returned by
the function.

At the end, the final result of the patch should IMO include:
- Documentation about how one can register a custom copy_handler.
- Something in src/test/modules/, minimalistic still useful that can
be used as a template when one wants to implement their own handler.
The documentation should mention about this module.
- No need for SQL functions for all the in-core handlers: let's just
return pointers to them based on the options given.

It would be probably cleaner to split the patch so as the code is
refactored and evaluated with the in-core handlers first, and then
extended with the pluggable facilities and the function lookups.
--
Michael

  31. Re: Make COPY format extendable: Extract COPY TO format implementations

    Junwang Zhao <zhjwpku@gmail.com> — 2023-12-11T10:44:39Z

    On Sat, Dec 9, 2023 at 7:38 PM Hannu Krosing <hannuk@google.com> wrote:
>
> Hi Junwang
>
> Please also see my presentation slides from last years PostgreSQL
> Conference in Berlin (attached)

I read through the slides, really promising ideas, it's will be great
if we can get there at last.

>
> The main Idea is to make not just "format", but also "transport" and
> "stream processing" extendable via virtual function tables.
The code is really coupled, it is not easy to do all of these in one round,
it will be great if you have a POC patch.

>
> Btw, will any of you here be in Prague next week ?
> Would be a good opportunity to discuss this in person.
Sorry, no.

>
>
> Best Regards
> Hannu
>
> On Sat, Dec 9, 2023 at 9:39 AM Junwang Zhao <zhjwpku@gmail.com> wrote:
> >
> > On Sat, Dec 9, 2023 at 10:43 AM Hayato Kuroda (Fujitsu)
> > <kuroda.hayato@fujitsu.com> wrote:
> > >
> > > Dear Junagn, Sutou-san,
> > >
> > > Basically I agree your point - improving a extendibility is good.
> > > (I remember that this theme was talked at Japan PostgreSQL conference)
> > > Below are my comments for your patch.
> > >
> > > 01. General
> > >
> > > Just to confirm - is it OK to partially implement APIs? E.g., only COPY TO is
> > > available. Currently it seems not to consider a case which is not implemented.
> > >
> > For partially implements, we can leave the hook as NULL, and check the NULL
> > at *ProcessCopyOptions* and report error if not supported.
> >
> > > 02. General
> > >
> > > It might be trivial, but could you please clarify how users can extend? Is it OK
> > > to do below steps?
> > >
> > > 1. Create a handler function, via CREATE FUNCTION,
> > > 2. Register a handler, via new SQL (CREATE COPY HANDLER),
> > > 3. Specify the added handler as COPY ... FORMAT clause.
> > >
> > My original thought was option 2, but as Michael point, option 1 is
> > the right way
> > to go.
> >
> > > 03. General
> > >
> > > Could you please add document-related tasks to your TODO? I imagined like
> > > fdwhandler.sgml.
> > >
> > > 04. General - copyright
> > >
> > > For newly added files, the below copyright seems sufficient. See applyparallelworker.c.
> > >
> > > ```
> > >  * Copyright (c) 2023, PostgreSQL Global Development Group
> > > ```
> > >
> > > 05. src/include/catalog/* files
> > >
> > > IIUC, 8000 or higher OIDs should be used while developing a patch. src/include/catalog/unused_oids
> > > would suggest a candidate which you can use.
> >
> > Yeah, I will run renumber_oids.pl at last.
> >
> > >
> > > 06. copy.c
> > >
> > > I felt that we can create files per copying methods, like copy_{text|csv|binary}.c,
> > > like indexes.
> > > How do other think?
> >
> > Not sure about this, it seems others have put a lot of effort into
> > splitting TO and From.
> > Also like to hear from others.
> >
> > >
> > > 07. fmt_to_name()
> > >
> > > I'm not sure the function is really needed. Can we follow like get_foreign_data_wrapper_oid()
> > > and remove the funciton?
> >
> > I have referenced some code from greenplum, will remove this.
> >
> > >
> > > 08. GetCopyRoutineByName()
> > >
> > > Should we use syscache for searching a catalog?
> > >
> > > 09. CopyToFormatTextSendEndOfRow(), CopyToFormatBinaryStart()
> > >
> > > Comments still refer CopyHandlerOps, whereas it was renamed.
> > >
> > > 10. copy.h
> > >
> > > Per foreign.h and fdwapi.h, should we add a new header file and move some APIs?
> > >
> > > 11. copy.h
> > >
> > > ```
> > > -/* These are private in commands/copy[from|to].c */
> > > -typedef struct CopyFromStateData *CopyFromState;
> > > -typedef struct CopyToStateData *CopyToState;
> > > ```
> > >
> > > Are above changes really needed?
> > >
> > > 12. CopyFormatOptions
> > >
> > > Can we remove `bool binary` in future?
> > >
> > > 13. external functions
> > >
> > > ```
> > > +extern void CopyToFormatTextStart(CopyToState cstate, TupleDesc tupDesc);
> > > +extern void CopyToFormatTextOneRow(CopyToState cstate, TupleTableSlot *slot);
> > > +extern void CopyToFormatTextEnd(CopyToState cstate);
> > > +extern void CopyFromFormatTextStart(CopyFromState cstate, TupleDesc tupDesc);
> > > +extern bool CopyFromFormatTextNext(CopyFromState cstate, ExprContext *econtext,
> > > +
> > > Datum *values, bool *nulls);
> > > +extern void CopyFromFormatTextErrorCallback(CopyFromState cstate);
> > > +
> > > +extern void CopyToFormatBinaryStart(CopyToState cstate, TupleDesc tupDesc);
> > > +extern void CopyToFormatBinaryOneRow(CopyToState cstate, TupleTableSlot *slot);
> > > +extern void CopyToFormatBinaryEnd(CopyToState cstate);
> > > +extern void CopyFromFormatBinaryStart(CopyFromState cstate, TupleDesc tupDesc);
> > > +extern bool CopyFromFormatBinaryNext(CopyFromState cstate,
> > > ExprContext *econtext,
> > > +
> > >   Datum *values, bool *nulls);
> > > +extern void CopyFromFormatBinaryErrorCallback(CopyFromState cstate);
> > > ```
> > >
> > > FYI - If you add files for {text|csv|binary}, these declarations can be removed.
> > >
> > > Best Regards,
> > > Hayato Kuroda
> > > FUJITSU LIMITED
> > >
> >
> > Thanks for all the valuable suggestions.
> >
> > --
> > Regards
> > Junwang Zhao
> >
> >



-- 
Regards
Junwang Zhao

  32. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2023-12-11T14:31:29Z

    On Mon, Dec 11, 2023 at 7:19 PM Michael Paquier <michael@paquier.xyz> wrote:
>
> On Mon, Dec 11, 2023 at 10:57:15AM +0900, Masahiko Sawada wrote:
> > IIUC we cannot create two same name functions with the same arguments
> > but a different return value type in the first place. It seems to me
> > to be an overkill to change such a design.
>
> Agreed to not touch the logictics of LookupFuncName() for the sake of
> this thread.  I have not checked the SQL specification, but I recall
> that there are a few assumptions from the spec embedded in the lookup
> logic particularly when it comes to specify a procedure name without
> arguments.
>
> > Another idea is to encapsulate copy_to/from_handler by a super class
> > like copy_handler. The handler function is called with an argument,
> > say copyto, and returns copy_handler encapsulating either
> > copy_to/from_handler depending on the argument.
>
> Yep, that's possible as well and can work as a cross-check between the
> argument and the NodeTag assigned to the handler structure returned by
> the function.
>
> At the end, the final result of the patch should IMO include:
> - Documentation about how one can register a custom copy_handler.
> - Something in src/test/modules/, minimalistic still useful that can
> be used as a template when one wants to implement their own handler.
> The documentation should mention about this module.
> - No need for SQL functions for all the in-core handlers: let's just
> return pointers to them based on the options given.

Agreed.

> It would be probably cleaner to split the patch so as the code is
> refactored and evaluated with the in-core handlers first, and then
> extended with the pluggable facilities and the function lookups.

Agreed.

I've sketched the above idea including a test module in
src/test/module/test_copy_format, based on v2 patch. It's not splitted
and is dirty so just for discussion.


Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  33. Re: Make COPY format extendable: Extract COPY TO format implementations

    Junwang Zhao <zhjwpku@gmail.com> — 2023-12-12T02:09:03Z

    On Mon, Dec 11, 2023 at 10:32 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Mon, Dec 11, 2023 at 7:19 PM Michael Paquier <michael@paquier.xyz> wrote:
> >
> > On Mon, Dec 11, 2023 at 10:57:15AM +0900, Masahiko Sawada wrote:
> > > IIUC we cannot create two same name functions with the same arguments
> > > but a different return value type in the first place. It seems to me
> > > to be an overkill to change such a design.
> >
> > Agreed to not touch the logictics of LookupFuncName() for the sake of
> > this thread.  I have not checked the SQL specification, but I recall
> > that there are a few assumptions from the spec embedded in the lookup
> > logic particularly when it comes to specify a procedure name without
> > arguments.
> >
> > > Another idea is to encapsulate copy_to/from_handler by a super class
> > > like copy_handler. The handler function is called with an argument,
> > > say copyto, and returns copy_handler encapsulating either
> > > copy_to/from_handler depending on the argument.
> >
> > Yep, that's possible as well and can work as a cross-check between the
> > argument and the NodeTag assigned to the handler structure returned by
> > the function.
> >
> > At the end, the final result of the patch should IMO include:
> > - Documentation about how one can register a custom copy_handler.
> > - Something in src/test/modules/, minimalistic still useful that can
> > be used as a template when one wants to implement their own handler.
> > The documentation should mention about this module.
> > - No need for SQL functions for all the in-core handlers: let's just
> > return pointers to them based on the options given.
>
> Agreed.
>
> > It would be probably cleaner to split the patch so as the code is
> > refactored and evaluated with the in-core handlers first, and then
> > extended with the pluggable facilities and the function lookups.
>
> Agreed.
>
> I've sketched the above idea including a test module in
> src/test/module/test_copy_format, based on v2 patch. It's not splitted
> and is dirty so just for discussion.
>
The test_copy_format extension doesn't use the fields of CopyToState and
CopyFromState in this patch, I think we should move CopyFromStateData
and CopyToStateData to commands/copy.h, what do you think?

The framework in the patch LGTM.

>
> Regards,
>
> --
> Masahiko Sawada
> Amazon Web Services: https://aws.amazon.com



-- 
Regards
Junwang Zhao

  34. RE: Make COPY format extendable: Extract COPY TO format implementations

    Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com> — 2023-12-12T02:31:53Z

    Dear Sutou-san, Junwang,

Sorry for the delay reply.

> 
> Can we discuss how to proceed this improvement?
> 
> There are 2 approaches for it:
> 
> 1. Do the followings concurrently:
>    a. Implementing small changes that got a consensus and
>       merge them step-by-step
>       (e.g. We got a consensus that we need to extract the
>       current format related routines.)
>    b. Discuss design
> 
>    (v1-v3 patches use this approach.)
> 
> 2. Implement one (large) complete patch set with design
>    discussion and merge it
> 
>    (v4- patches use this approach.)
> 
> Which approach is preferred? (Or should we choose another
> approach?)
> 
> I thought that 1. is preferred because it will reduce review
> cost. So I chose 1.

I'm ok to use approach 1, but could you please divide a large patch? E.g.,

0001. defines an infrastructure for copy-API
0002. adjusts current codes to use APIs
0003. adds a test module in src/test/modules or contrib.
...

This approach helps reviewers to see patches deeper. Separated patches can be
combined when they are close to committable.

Best Regards,
Hayato Kuroda
FUJITSU LIMITED

  35. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2023-12-13T11:48:18Z

    On Tue, Dec 12, 2023 at 11:09 AM Junwang Zhao <zhjwpku@gmail.com> wrote:
>
> On Mon, Dec 11, 2023 at 10:32 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > On Mon, Dec 11, 2023 at 7:19 PM Michael Paquier <michael@paquier.xyz> wrote:
> > >
> > > On Mon, Dec 11, 2023 at 10:57:15AM +0900, Masahiko Sawada wrote:
> > > > IIUC we cannot create two same name functions with the same arguments
> > > > but a different return value type in the first place. It seems to me
> > > > to be an overkill to change such a design.
> > >
> > > Agreed to not touch the logictics of LookupFuncName() for the sake of
> > > this thread.  I have not checked the SQL specification, but I recall
> > > that there are a few assumptions from the spec embedded in the lookup
> > > logic particularly when it comes to specify a procedure name without
> > > arguments.
> > >
> > > > Another idea is to encapsulate copy_to/from_handler by a super class
> > > > like copy_handler. The handler function is called with an argument,
> > > > say copyto, and returns copy_handler encapsulating either
> > > > copy_to/from_handler depending on the argument.
> > >
> > > Yep, that's possible as well and can work as a cross-check between the
> > > argument and the NodeTag assigned to the handler structure returned by
> > > the function.
> > >
> > > At the end, the final result of the patch should IMO include:
> > > - Documentation about how one can register a custom copy_handler.
> > > - Something in src/test/modules/, minimalistic still useful that can
> > > be used as a template when one wants to implement their own handler.
> > > The documentation should mention about this module.
> > > - No need for SQL functions for all the in-core handlers: let's just
> > > return pointers to them based on the options given.
> >
> > Agreed.
> >
> > > It would be probably cleaner to split the patch so as the code is
> > > refactored and evaluated with the in-core handlers first, and then
> > > extended with the pluggable facilities and the function lookups.
> >
> > Agreed.
> >
> > I've sketched the above idea including a test module in
> > src/test/module/test_copy_format, based on v2 patch. It's not splitted
> > and is dirty so just for discussion.
> >
> The test_copy_format extension doesn't use the fields of CopyToState and
> CopyFromState in this patch, I think we should move CopyFromStateData
> and CopyToStateData to commands/copy.h, what do you think?

Yes, I basically agree with that, where we move CopyFromStateData to
might depend on how we define COPY FROM APIs though. I think we can
move CopyToStateData to copy.h at least.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  36. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2023-12-14T09:44:14Z

    Hi,

In <CAD21AoCvjGserrtEU=UcA3Mfyfe6ftf9OXPHv9fiJ9DmXMJ2nQ@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Mon, 11 Dec 2023 10:57:15 +0900,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

> IIUC we cannot create two same name functions with the same arguments
> but a different return value type in the first place. It seems to me
> to be an overkill to change such a design.

Oh, sorry. I didn't notice it.

> Another idea is to encapsulate copy_to/from_handler by a super class
> like copy_handler. The handler function is called with an argument,
> say copyto, and returns copy_handler encapsulating either
> copy_to/from_handler depending on the argument.

It's for using "${copy_format_name}" such as "json" and
"parquet" as a function name, right? If we use the
"${copy_format_name}" approach, we can't use function names
that are already used by tablesample method handler such as
"system" and "bernoulli" for COPY FORMAT name. Because both
of tablesample method handler function and COPY FORMAT
handler function use "(internal)" as arguments.

I think that tablesample method names and COPY FORMAT names
will not be conflicted but the limitation (using the same
namespace for tablesample method and COPY FORMAT) is
unnecessary limitation.

How about using prefix ("copy_to_${copy_format_name}" or
something) or suffix ("${copy_format_name}_copy_to" or
something) for function names? For example,
"copy_to_json"/"copy_from_json" for "json" COPY FORMAT.

("copy_${copy_format_name}" that returns copy_handler
encapsulating either copy_to/from_handler depending on the
argument may be an option.)


Thanks,
-- 
kou

  37. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2023-12-14T20:19:43Z

    On Thu, Dec 14, 2023 at 6:44 PM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <CAD21AoCvjGserrtEU=UcA3Mfyfe6ftf9OXPHv9fiJ9DmXMJ2nQ@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Mon, 11 Dec 2023 10:57:15 +0900,
>   Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> > IIUC we cannot create two same name functions with the same arguments
> > but a different return value type in the first place. It seems to me
> > to be an overkill to change such a design.
>
> Oh, sorry. I didn't notice it.
>
> > Another idea is to encapsulate copy_to/from_handler by a super class
> > like copy_handler. The handler function is called with an argument,
> > say copyto, and returns copy_handler encapsulating either
> > copy_to/from_handler depending on the argument.
>
> It's for using "${copy_format_name}" such as "json" and
> "parquet" as a function name, right?

Right.

> If we use the
> "${copy_format_name}" approach, we can't use function names
> that are already used by tablesample method handler such as
> "system" and "bernoulli" for COPY FORMAT name. Because both
> of tablesample method handler function and COPY FORMAT
> handler function use "(internal)" as arguments.
>
> I think that tablesample method names and COPY FORMAT names
> will not be conflicted but the limitation (using the same
> namespace for tablesample method and COPY FORMAT) is
> unnecessary limitation.

Presumably, such function name collisions are not limited to
tablesample and copy, but apply to all functions that have an
"internal" argument. To avoid collisions, extensions can be created in
a different schema than public. And note that built-in format copy
handler doesn't need to declare its handler function.

>
> How about using prefix ("copy_to_${copy_format_name}" or
> something) or suffix ("${copy_format_name}_copy_to" or
> something) for function names? For example,
> "copy_to_json"/"copy_from_json" for "json" COPY FORMAT.
>
> ("copy_${copy_format_name}" that returns copy_handler
> encapsulating either copy_to/from_handler depending on the
> argument may be an option.)

While there is a way to avoid collision as I mentioned above, I can
see the point that we might want to avoid using a generic function
name such as "arrow" and "parquet" as custom copy handler functions.
Adding a prefix or suffix would be one option but to give extensions
more flexibility, another option would be to support format = 'custom'
and add the "handler" option to specify a copy handler function name
to call. For example, COPY ... FROM ... WITH (FORMAT = 'custom',
HANDLER = 'arrow_copy_handler').

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  38. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2023-12-15T00:53:05Z

    Hi,

In <CAD21AoCZv3cVU+NxR2s9J_dWvjrS350GFFr2vMgCH8wWxQ5hTQ@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 15 Dec 2023 05:19:43 +0900,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

> To avoid collisions, extensions can be created in a
> different schema than public.

Thanks. I didn't notice it.

> And note that built-in format copy handler doesn't need to
> declare its handler function.

Right. I know it.

> Adding a prefix or suffix would be one option but to give extensions
> more flexibility, another option would be to support format = 'custom'
> and add the "handler" option to specify a copy handler function name
> to call. For example, COPY ... FROM ... WITH (FORMAT = 'custom',
> HANDLER = 'arrow_copy_handler').

Interesting. If we use this option, users can choose an COPY
FORMAT implementation they like from multiple
implementations. For example, a developer may implement a
COPY FROM FORMAT = 'json' handler with PostgreSQL's JSON
related API and another developer may implement a handler
with simdjson[1] which is a fast JSON parser. Users can
choose whichever they like.

But specifying HANDLER = '...' explicitly is a bit
inconvenient. Because only one handler will be installed in
most use cases. In the case, users don't need to choose one
handler.

If we choose this option, it may be better that we also
provide a mechanism that can work without HANDLER. Searching
a function by name like tablesample method does is an option.


[1]: https://github.com/simdjson/simdjson


Thanks,
-- 
kou

  39. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2023-12-15T02:55:18Z

    Hi,

In 
 <OS3PR01MB9882F023300EDC5AFD8A8339F58EA@OS3PR01MB9882.jpnprd01.prod.outlook.com>
  "RE: Make COPY format extendable: Extract COPY TO format implementations" on Tue, 12 Dec 2023 02:31:53 +0000,
  "Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com> wrote:

>> Can we discuss how to proceed this improvement?
>> 
>> There are 2 approaches for it:
>> 
>> 1. Do the followings concurrently:
>>    a. Implementing small changes that got a consensus and
>>       merge them step-by-step
>>       (e.g. We got a consensus that we need to extract the
>>       current format related routines.)
>>    b. Discuss design
>> 
>>    (v1-v3 patches use this approach.)
>> 
>> 2. Implement one (large) complete patch set with design
>>    discussion and merge it
>> 
>>    (v4- patches use this approach.)
>> 
>> Which approach is preferred? (Or should we choose another
>> approach?)
>> 
>> I thought that 1. is preferred because it will reduce review
>> cost. So I chose 1.
> 
> I'm ok to use approach 1, but could you please divide a large patch? E.g.,
> 
> 0001. defines an infrastructure for copy-API
> 0002. adjusts current codes to use APIs
> 0003. adds a test module in src/test/modules or contrib.
> ...
> 
> This approach helps reviewers to see patches deeper. Separated patches can be
> combined when they are close to committable.

It seems that I should have chosen another approach based on
comments so far:

3. Do the followings in order:
   a. Implement a workable (but maybe dirty and/or incomplete)
      implementation to discuss design like [1], discuss
      design with it and get a consensus on design
   b. Implement small patches based on the design

[1]: https://www.postgresql.org/message-id/CAD21AoCunywHird3GaPzWe6s9JG1wzxj3Cr6vGN36DDheGjOjA%40mail.gmail.com 

I'll implement a custom COPY FORMAT handler with [1] and
provide a feedback with the experience. (It's for a.)


Thanks,
-- 
kou

  40. Re: Make COPY format extendable: Extract COPY TO format implementations

    Junwang Zhao <zhjwpku@gmail.com> — 2023-12-15T03:27:30Z

    On Fri, Dec 15, 2023 at 8:53 AM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <CAD21AoCZv3cVU+NxR2s9J_dWvjrS350GFFr2vMgCH8wWxQ5hTQ@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 15 Dec 2023 05:19:43 +0900,
>   Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> > To avoid collisions, extensions can be created in a
> > different schema than public.
>
> Thanks. I didn't notice it.
>
> > And note that built-in format copy handler doesn't need to
> > declare its handler function.
>
> Right. I know it.
>
> > Adding a prefix or suffix would be one option but to give extensions
> > more flexibility, another option would be to support format = 'custom'
> > and add the "handler" option to specify a copy handler function name
> > to call. For example, COPY ... FROM ... WITH (FORMAT = 'custom',
> > HANDLER = 'arrow_copy_handler').
>
I like the prefix/suffix idea, easy to implement. *custom* is not a FORMAT,
and user has to know the name of the specific handler names, not
intuitive.

> Interesting. If we use this option, users can choose an COPY
> FORMAT implementation they like from multiple
> implementations. For example, a developer may implement a
> COPY FROM FORMAT = 'json' handler with PostgreSQL's JSON
> related API and another developer may implement a handler
> with simdjson[1] which is a fast JSON parser. Users can
> choose whichever they like.
Not sure about this, why not move Json copy handler to contrib
as an example for others, any extensions share the same format
function name and just install one? No bound would implement
another CSV or TEXT copy handler IMHO.
>
> But specifying HANDLER = '...' explicitly is a bit
> inconvenient. Because only one handler will be installed in
> most use cases. In the case, users don't need to choose one
> handler.
>
> If we choose this option, it may be better that we also
> provide a mechanism that can work without HANDLER. Searching
> a function by name like tablesample method does is an option.
>
>
> [1]: https://github.com/simdjson/simdjson
>
>
> Thanks,
> --
> kou



-- 
Regards
Junwang Zhao

  41. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2023-12-15T03:48:17Z

    On Fri, Dec 15, 2023 at 9:53 AM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <CAD21AoCZv3cVU+NxR2s9J_dWvjrS350GFFr2vMgCH8wWxQ5hTQ@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 15 Dec 2023 05:19:43 +0900,
>   Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> > To avoid collisions, extensions can be created in a
> > different schema than public.
>
> Thanks. I didn't notice it.
>
> > And note that built-in format copy handler doesn't need to
> > declare its handler function.
>
> Right. I know it.
>
> > Adding a prefix or suffix would be one option but to give extensions
> > more flexibility, another option would be to support format = 'custom'
> > and add the "handler" option to specify a copy handler function name
> > to call. For example, COPY ... FROM ... WITH (FORMAT = 'custom',
> > HANDLER = 'arrow_copy_handler').
>
> Interesting. If we use this option, users can choose an COPY
> FORMAT implementation they like from multiple
> implementations. For example, a developer may implement a
> COPY FROM FORMAT = 'json' handler with PostgreSQL's JSON
> related API and another developer may implement a handler
> with simdjson[1] which is a fast JSON parser. Users can
> choose whichever they like.
>
> But specifying HANDLER = '...' explicitly is a bit
> inconvenient. Because only one handler will be installed in
> most use cases. In the case, users don't need to choose one
> handler.
>
> If we choose this option, it may be better that we also
> provide a mechanism that can work without HANDLER. Searching
> a function by name like tablesample method does is an option.

Agreed. We can search the function by format name by default and the
user can optionally specify the handler function name in case where
the names of the installed custom copy handler collide. Probably the
handler option stuff could be a follow-up patch.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  42. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2023-12-15T04:45:31Z

    Hi,

In <CAEG8a3JuShA6g19Nt_Ejk15BrNA6PmeCbK7p81izZi71muGq3g@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 15 Dec 2023 11:27:30 +0800,
  Junwang Zhao <zhjwpku@gmail.com> wrote:

>> > Adding a prefix or suffix would be one option but to give extensions
>> > more flexibility, another option would be to support format = 'custom'
>> > and add the "handler" option to specify a copy handler function name
>> > to call. For example, COPY ... FROM ... WITH (FORMAT = 'custom',
>> > HANDLER = 'arrow_copy_handler').
>>
> I like the prefix/suffix idea, easy to implement. *custom* is not a FORMAT,
> and user has to know the name of the specific handler names, not
> intuitive.

Ah! I misunderstood this idea. "custom" is the special
format to use "HANDLER". I thought that we can use it like

   (FORMAT = 'arrow', HANDLER = 'arrow_copy_handler_impl1')

and

   (FORMAT = 'arrow', HANDLER = 'arrow_copy_handler_impl2')

.

>> Interesting. If we use this option, users can choose an COPY
>> FORMAT implementation they like from multiple
>> implementations. For example, a developer may implement a
>> COPY FROM FORMAT = 'json' handler with PostgreSQL's JSON
>> related API and another developer may implement a handler
>> with simdjson[1] which is a fast JSON parser. Users can
>> choose whichever they like.
> Not sure about this, why not move Json copy handler to contrib
> as an example for others, any extensions share the same format
> function name and just install one? No bound would implement
> another CSV or TEXT copy handler IMHO.

I should have used a different format not JSON as an example
for easy to understand. I just wanted to say that extension
developers can implement another implementation without
conflicting another implementation.


Thanks,
-- 
kou

  43. Re: Make COPY format extendable: Extract COPY TO format implementations

    Junwang Zhao <zhjwpku@gmail.com> — 2023-12-15T06:02:49Z

    On Fri, Dec 15, 2023 at 12:45 PM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <CAEG8a3JuShA6g19Nt_Ejk15BrNA6PmeCbK7p81izZi71muGq3g@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 15 Dec 2023 11:27:30 +0800,
>   Junwang Zhao <zhjwpku@gmail.com> wrote:
>
> >> > Adding a prefix or suffix would be one option but to give extensions
> >> > more flexibility, another option would be to support format = 'custom'
> >> > and add the "handler" option to specify a copy handler function name
> >> > to call. For example, COPY ... FROM ... WITH (FORMAT = 'custom',
> >> > HANDLER = 'arrow_copy_handler').
> >>
> > I like the prefix/suffix idea, easy to implement. *custom* is not a FORMAT,
> > and user has to know the name of the specific handler names, not
> > intuitive.
>
> Ah! I misunderstood this idea. "custom" is the special
> format to use "HANDLER". I thought that we can use it like
>
>    (FORMAT = 'arrow', HANDLER = 'arrow_copy_handler_impl1')
>
> and
>
>    (FORMAT = 'arrow', HANDLER = 'arrow_copy_handler_impl2')
>
> .
>
> >> Interesting. If we use this option, users can choose an COPY
> >> FORMAT implementation they like from multiple
> >> implementations. For example, a developer may implement a
> >> COPY FROM FORMAT = 'json' handler with PostgreSQL's JSON
> >> related API and another developer may implement a handler
> >> with simdjson[1] which is a fast JSON parser. Users can
> >> choose whichever they like.
> > Not sure about this, why not move Json copy handler to contrib
> > as an example for others, any extensions share the same format
> > function name and just install one? No bound would implement
> > another CSV or TEXT copy handler IMHO.
>
> I should have used a different format not JSON as an example
> for easy to understand. I just wanted to say that extension
> developers can implement another implementation without
> conflicting another implementation.

Yeah, I can see the value of the HANDLER option now. The possibility
of two extensions for the same format using same hanlder name should
be rare I guess ;)
>
>
> Thanks,
> --
> kou



-- 
Regards
Junwang Zhao

  44. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2023-12-21T09:35:04Z

    Hi,

In <CAD21AoCunywHird3GaPzWe6s9JG1wzxj3Cr6vGN36DDheGjOjA@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Mon, 11 Dec 2023 23:31:29 +0900,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

> I've sketched the above idea including a test module in
> src/test/module/test_copy_format, based on v2 patch. It's not splitted
> and is dirty so just for discussion.

I implemented a sample COPY TO handler for Apache Arrow that
supports only integer and text.

I needed to extend the patch:

1. Add an opaque space for custom COPY TO handler
   * Add CopyToState{Get,Set}Opaque()
   https://github.com/kou/postgres/commit/5a610b6a066243f971e029432db67152cfe5e944

2. Export CopyToState::attnumlist
   * Add CopyToStateGetAttNumList()
   https://github.com/kou/postgres/commit/15fcba8b4e95afa86edb3f677a7bdb1acb1e7688

3. Export CopySend*()
   * Rename CopySend*() to CopyToStateSend*() and export them
   * Exception: CopySendEndOfRow() to CopyToStateFlush() because
     it just flushes the internal buffer now.
   https://github.com/kou/postgres/commit/289a5640135bde6733a1b8e2c412221ad522901e

The attached patch is based on the Sawada-san's patch and
includes the above changes. Note that this patch is also
dirty so just for discussion.

My suggestions from this experience:

1. Split COPY handler to COPY TO handler and COPY FROM handler

   * CopyFormatRoutine is a bit tricky. An extension needs
     to create a CopyFormatRoutine node and
     a CopyToFormatRoutine node.

   * If we just require "copy_to_${FORMAT}(internal)"
     function and "copy_from_${FORMAT}(internal)" function,
     we can remove the tricky approach. And it also avoid
     name collisions with other handler such as tablesample
     handler.
     See also:
     https://www.postgresql.org/message-id/flat/20231214.184414.2179134502876898942.kou%40clear-code.com#af71f364d0a9f5c144e45b447e5c16c9

2. Need an opaque space like IndexScanDesc::opaque does

   * A custom COPY TO handler needs to keep its data

3. Export CopySend*()

   * If we like minimum API, we just need to export
     CopySendData() and CopySendEndOfRow(). But
     CopySend{String,Char,Int32,Int16}() will be convenient
     custom COPY TO handlers. (A custom COPY TO handler for
     Apache Arrow doesn't need them.)

Questions:

1. What value should be used for "format" in
   PgMsg_CopyOutResponse message?

   https://git.postgresql.org/gitweb/?p=postgresql.git;a=blob;f=src/backend/commands/copyto.c;h=c66a047c4a79cc614784610f385f1cd0935350f3;hb=9ca6e7b9411e36488ef539a2c1f6846ac92a7072#l144

   It's 1 for binary format and 0 for text/csv format.

   Should we make it customizable by custom COPY TO handler?
   If so, what value should be used for this?

2. Do we need more tries for design discussion for the first
   implementation? If we need, what should we try?


Thanks,
-- 
kou

  45. Re: Make COPY format extendable: Extract COPY TO format implementations

    Michael Paquier <michael@paquier.xyz> — 2023-12-22T01:00:24Z

    On Thu, Dec 21, 2023 at 06:35:04PM +0900, Sutou Kouhei wrote:
>    * If we just require "copy_to_${FORMAT}(internal)"
>      function and "copy_from_${FORMAT}(internal)" function,
>      we can remove the tricky approach. And it also avoid
>      name collisions with other handler such as tablesample
>      handler.
>      See also:
>      https://www.postgresql.org/message-id/flat/20231214.184414.2179134502876898942.kou%40clear-code.com#af71f364d0a9f5c144e45b447e5c16c9

Hmm.  I prefer the unique name approach for the COPY portions without
enforcing any naming policy on the function names returning the
handlers, actually, though I can see your point.

> 2. Need an opaque space like IndexScanDesc::opaque does
> 
>    * A custom COPY TO handler needs to keep its data

Sounds useful to me to have a private area passed down to the
callbacks.

> 3. Export CopySend*()
> 
>    * If we like minimum API, we just need to export
>      CopySendData() and CopySendEndOfRow(). But
>      CopySend{String,Char,Int32,Int16}() will be convenient
>      custom COPY TO handlers. (A custom COPY TO handler for
>      Apache Arrow doesn't need them.)

Hmm.  Not sure on this one.  This may come down to externalize the
manipulation of fe_msgbuf.  Particularly, could it be possible that
some custom formats don't care at all about the network order?

> Questions:
> 
> 1. What value should be used for "format" in
>    PgMsg_CopyOutResponse message?
> 
>    It's 1 for binary format and 0 for text/csv format.
> 
>    Should we make it customizable by custom COPY TO handler?
>    If so, what value should be used for this?

Interesting point.  It looks very tempting to give more flexibility to
people who'd like to use their own code as we have one byte in the
protocol but just use 0/1.  Hence it feels natural to have a callback
for that.

It also means that we may want to think harder about copy_is_binary in
libpq in the future step.  Now, having a backend implementation does
not need any libpq bits, either, because a client stack may just want
to speak the Postgres protocol directly.  Perhaps a custom COPY
implementation would be OK with how things are in libpq, as well,
tweaking its way through with just text or binary.

> 2. Do we need more tries for design discussion for the first
>    implementation? If we need, what should we try?

A makeNode() is used with an allocation in the current memory context
in the function returning the handler.  I would have assume that this
stuff returns a handler as a const struct like table AMs.
--
Michael

  46. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2023-12-22T01:23:28Z

    On Fri, Dec 22, 2023 at 10:00 AM Michael Paquier <michael@paquier.xyz> wrote:
>
> On Thu, Dec 21, 2023 at 06:35:04PM +0900, Sutou Kouhei wrote:
> >    * If we just require "copy_to_${FORMAT}(internal)"
> >      function and "copy_from_${FORMAT}(internal)" function,
> >      we can remove the tricky approach. And it also avoid
> >      name collisions with other handler such as tablesample
> >      handler.
> >      See also:
> >      https://www.postgresql.org/message-id/flat/20231214.184414.2179134502876898942.kou%40clear-code.com#af71f364d0a9f5c144e45b447e5c16c9
>
> Hmm.  I prefer the unique name approach for the COPY portions without
> enforcing any naming policy on the function names returning the
> handlers, actually, though I can see your point.

Yeah, another idea is to provide support functions to return a
CopyFormatRoutine wrapping either CopyToFormatRoutine or
CopyFromFormatRoutine. For example:

extern CopyFormatRoutine *MakeCopyToFormatRoutine(const
CopyToFormatRoutine *routine);

extensions can do like:

static const CopyToFormatRoutine testfmt_handler = {
    .type = T_CopyToFormatRoutine,
    .start_fn = testfmt_copyto_start,
    .onerow_fn = testfmt_copyto_onerow,
    .end_fn = testfmt_copyto_end
};

Datum
copy_testfmt_handler(PG_FUNCTION_ARGS)
{
    CopyFormatRoutine *routine = MakeCopyToFormatRoutine(&testfmt_handler);
    :

>
> > 2. Need an opaque space like IndexScanDesc::opaque does
> >
> >    * A custom COPY TO handler needs to keep its data
>
> Sounds useful to me to have a private area passed down to the
> callbacks.
>

+1

>
> > Questions:
> >
> > 1. What value should be used for "format" in
> >    PgMsg_CopyOutResponse message?
> >
> >    It's 1 for binary format and 0 for text/csv format.
> >
> >    Should we make it customizable by custom COPY TO handler?
> >    If so, what value should be used for this?
>
> Interesting point.  It looks very tempting to give more flexibility to
> people who'd like to use their own code as we have one byte in the
> protocol but just use 0/1.  Hence it feels natural to have a callback
> for that.

+1

>
> It also means that we may want to think harder about copy_is_binary in
> libpq in the future step.  Now, having a backend implementation does
> not need any libpq bits, either, because a client stack may just want
> to speak the Postgres protocol directly.  Perhaps a custom COPY
> implementation would be OK with how things are in libpq, as well,
> tweaking its way through with just text or binary.
>
> > 2. Do we need more tries for design discussion for the first
> >    implementation? If we need, what should we try?
>
> A makeNode() is used with an allocation in the current memory context
> in the function returning the handler.  I would have assume that this
> stuff returns a handler as a const struct like table AMs.

+1

The example I mentioned above does that.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  47. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2023-12-22T01:48:18Z

    On Thu, Dec 21, 2023 at 6:35 PM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <CAD21AoCunywHird3GaPzWe6s9JG1wzxj3Cr6vGN36DDheGjOjA@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Mon, 11 Dec 2023 23:31:29 +0900,
>   Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> > I've sketched the above idea including a test module in
> > src/test/module/test_copy_format, based on v2 patch. It's not splitted
> > and is dirty so just for discussion.
>
> I implemented a sample COPY TO handler for Apache Arrow that
> supports only integer and text.
>
> I needed to extend the patch:
>
> 1. Add an opaque space for custom COPY TO handler
>    * Add CopyToState{Get,Set}Opaque()
>    https://github.com/kou/postgres/commit/5a610b6a066243f971e029432db67152cfe5e944
>
> 2. Export CopyToState::attnumlist
>    * Add CopyToStateGetAttNumList()
>    https://github.com/kou/postgres/commit/15fcba8b4e95afa86edb3f677a7bdb1acb1e7688

I think we can move CopyToState to copy.h and we don't need to have
set/get functions for its fields.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  48. Re: Make COPY format extendable: Extract COPY TO format implementations

    Junwang Zhao <zhjwpku@gmail.com> — 2023-12-22T02:58:05Z

    On Thu, Dec 21, 2023 at 5:35 PM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <CAD21AoCunywHird3GaPzWe6s9JG1wzxj3Cr6vGN36DDheGjOjA@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Mon, 11 Dec 2023 23:31:29 +0900,
>   Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> > I've sketched the above idea including a test module in
> > src/test/module/test_copy_format, based on v2 patch. It's not splitted
> > and is dirty so just for discussion.
>
> I implemented a sample COPY TO handler for Apache Arrow that
> supports only integer and text.
>
> I needed to extend the patch:
>
> 1. Add an opaque space for custom COPY TO handler
>    * Add CopyToState{Get,Set}Opaque()
>    https://github.com/kou/postgres/commit/5a610b6a066243f971e029432db67152cfe5e944
>
> 2. Export CopyToState::attnumlist
>    * Add CopyToStateGetAttNumList()
>    https://github.com/kou/postgres/commit/15fcba8b4e95afa86edb3f677a7bdb1acb1e7688
>
> 3. Export CopySend*()
>    * Rename CopySend*() to CopyToStateSend*() and export them
>    * Exception: CopySendEndOfRow() to CopyToStateFlush() because
>      it just flushes the internal buffer now.
>    https://github.com/kou/postgres/commit/289a5640135bde6733a1b8e2c412221ad522901e
>
I guess the purpose of these helpers is to avoid expose CopyToState to
copy.h, but I
think expose CopyToState to user might make life easier, users might want to use
the memory contexts of the structure (though I agree not all the
fields are necessary
for extension handers).

> The attached patch is based on the Sawada-san's patch and
> includes the above changes. Note that this patch is also
> dirty so just for discussion.
>
> My suggestions from this experience:
>
> 1. Split COPY handler to COPY TO handler and COPY FROM handler
>
>    * CopyFormatRoutine is a bit tricky. An extension needs
>      to create a CopyFormatRoutine node and
>      a CopyToFormatRoutine node.
>
>    * If we just require "copy_to_${FORMAT}(internal)"
>      function and "copy_from_${FORMAT}(internal)" function,
>      we can remove the tricky approach. And it also avoid
>      name collisions with other handler such as tablesample
>      handler.
>      See also:
>      https://www.postgresql.org/message-id/flat/20231214.184414.2179134502876898942.kou%40clear-code.com#af71f364d0a9f5c144e45b447e5c16c9
>
> 2. Need an opaque space like IndexScanDesc::opaque does
>
>    * A custom COPY TO handler needs to keep its data

I once thought users might want to parse their own options, maybe this
is a use case for this opaque space.

For the name, I thought private_data might be a better candidate than
opaque, but I do not insist.
>
> 3. Export CopySend*()
>
>    * If we like minimum API, we just need to export
>      CopySendData() and CopySendEndOfRow(). But
>      CopySend{String,Char,Int32,Int16}() will be convenient
>      custom COPY TO handlers. (A custom COPY TO handler for
>      Apache Arrow doesn't need them.)

Do you use the arrow library to control the memory? Is there a way that
we can let the arrow use postgres' memory context? I'm not sure this
is necessary, just raise the question for discussion.
>
> Questions:
>
> 1. What value should be used for "format" in
>    PgMsg_CopyOutResponse message?
>
>    https://git.postgresql.org/gitweb/?p=postgresql.git;a=blob;f=src/backend/commands/copyto.c;h=c66a047c4a79cc614784610f385f1cd0935350f3;hb=9ca6e7b9411e36488ef539a2c1f6846ac92a7072#l144
>
>    It's 1 for binary format and 0 for text/csv format.
>
>    Should we make it customizable by custom COPY TO handler?
>    If so, what value should be used for this?
>
> 2. Do we need more tries for design discussion for the first
>    implementation? If we need, what should we try?
>
>
> Thanks,
> --
> kou

+PG_FUNCTION_INFO_V1(copy_testfmt_handler);
+Datum
+copy_testfmt_handler(PG_FUNCTION_ARGS)
+{
+ bool is_from = PG_GETARG_BOOL(0);
+ CopyFormatRoutine *cp = makeNode(CopyFormatRoutine);;
+

extra semicolon.

-- 
Regards
Junwang Zhao

  49. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-01-10T03:00:34Z

    Hi,

In <ZYTfqGppMc9e_w2k@paquier.xyz>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 22 Dec 2023 10:00:24 +0900,
  Michael Paquier <michael@paquier.xyz> wrote:

>> 3. Export CopySend*()
>> 
>>    * If we like minimum API, we just need to export
>>      CopySendData() and CopySendEndOfRow(). But
>>      CopySend{String,Char,Int32,Int16}() will be convenient
>>      custom COPY TO handlers. (A custom COPY TO handler for
>>      Apache Arrow doesn't need them.)
> 
> Hmm.  Not sure on this one.  This may come down to externalize the
> manipulation of fe_msgbuf.  Particularly, could it be possible that
> some custom formats don't care at all about the network order?

It means that all custom formats should control byte order
by themselves instead of using CopySendInt*() that always
use network byte order, right? It makes sense. Let's export
only CopySendData() and CopySendEndOfRow().


>> 1. What value should be used for "format" in
>>    PgMsg_CopyOutResponse message?
>> 
>>    It's 1 for binary format and 0 for text/csv format.
>> 
>>    Should we make it customizable by custom COPY TO handler?
>>    If so, what value should be used for this?
> 
> Interesting point.  It looks very tempting to give more flexibility to
> people who'd like to use their own code as we have one byte in the
> protocol but just use 0/1.  Hence it feels natural to have a callback
> for that.

OK. Let's add a callback something like:

typedef int16 (*CopyToGetFormat_function) (CopyToState cstate);

> It also means that we may want to think harder about copy_is_binary in
> libpq in the future step.  Now, having a backend implementation does
> not need any libpq bits, either, because a client stack may just want
> to speak the Postgres protocol directly.  Perhaps a custom COPY
> implementation would be OK with how things are in libpq, as well,
> tweaking its way through with just text or binary.

Can we defer this discussion after we commit a basic custom
COPY format handler mechanism?

>> 2. Do we need more tries for design discussion for the first
>>    implementation? If we need, what should we try?
> 
> A makeNode() is used with an allocation in the current memory context
> in the function returning the handler.  I would have assume that this
> stuff returns a handler as a const struct like table AMs.

If we use this approach, we can't use the Sawada-san's
idea[1] that provides a convenient API to hide
CopyFormatRoutine internal. The idea provides
MakeCopy{To,From}FormatRoutine(). They return a new
CopyFormatRoutine* with suitable is_from member. They can't
use static const CopyFormatRoutine because they may be called
multiple times in the same process.

We can use the satic const struct approach by choosing one
of the followings:

1. Use separated function for COPY {TO,FROM} format handlers
   as I suggested.

2. Don't provide convenient API. Developers construct
   CopyFormatRoutine by themselves. But it may be a bit
   tricky.

3. Similar to 2. but don't use a bit tricky approach (don't
   embed Copy{To,From}FormatRoutine nodes into
   CopyFormatRoutine).

   Use unified function for COPY {TO,FROM} format handlers
   but CopyFormatRoutine always have both of COPY {TO,FROM}
   format routines and these routines aren't nodes:

   typedef struct CopyToFormatRoutine
   {
           CopyToStart_function start_fn;
           CopyToOneRow_function onerow_fn;
           CopyToEnd_function end_fn;
   } CopyToFormatRoutine;

   /* XXX: just copied from COPY TO routines */
   typedef struct CopyFromFormatRoutine
   {
           CopyFromStart_function start_fn;
           CopyFromOneRow_function onerow_fn;
           CopyFromEnd_function end_fn;
   } CopyFromFormatRoutine;

   typedef struct CopyFormatRoutine
   {
           NodeTag		type;

           CopyToFormatRoutine	   to_routine;
           CopyFromFormatRoutine	   from_routine;
   } CopyFormatRoutine;

   ----

   static const CopyFormatRoutine testfmt_handler = {
       .type = T_CopyFormatRoutine,
       .to_routine = {
           .start_fn = testfmt_copyto_start,
           .onerow_fn = testfmt_copyto_onerow,
           .end_fn = testfmt_copyto_end,
       },
       .from_routine = {
           .start_fn = testfmt_copyfrom_start,
           .onerow_fn = testfmt_copyfrom_onerow,
           .end_fn = testfmt_copyfrom_end,
       },
   };

   PG_FUNCTION_INFO_V1(copy_testfmt_handler);
   Datum
   copy_testfmt_handler(PG_FUNCTION_ARGS)
   {
           PG_RETURN_POINTER(&testfmt_handler);
   }

4. ... other idea?


[1] https://www.postgresql.org/message-id/flat/CAD21AoDs9cOjuVbA_krGizAdc50KE%2BFjAuEXWF0NZwbMnc7F3Q%40mail.gmail.com#71bb03d9237252382b245dd33e705a3a


Thanks,
-- 
kou

  50. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-01-10T03:06:44Z

    Hi,

In <CAD21AoD=UapH4Wh06G6H5XAzPJ0iJg9YcW8r7E2UEJkZ8QsosA@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 22 Dec 2023 10:48:18 +0900,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

>> I needed to extend the patch:
>>
>> 1. Add an opaque space for custom COPY TO handler
>>    * Add CopyToState{Get,Set}Opaque()
>>    https://github.com/kou/postgres/commit/5a610b6a066243f971e029432db67152cfe5e944
>>
>> 2. Export CopyToState::attnumlist
>>    * Add CopyToStateGetAttNumList()
>>    https://github.com/kou/postgres/commit/15fcba8b4e95afa86edb3f677a7bdb1acb1e7688
> 
> I think we can move CopyToState to copy.h and we don't need to have
> set/get functions for its fields.

I don't object the idea if other PostgreSQL developers
prefer the approach. Is there any PostgreSQL developer who
objects that we export Copy{To,From}StateData as public API?


Thanks,
-- 
kou

  51. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-01-10T06:20:23Z

    Hi,

In <CAEG8a3+jG_NKOUmcxDyEX2xSggBXReZ4H=e3RFsUtedY88A03w@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 22 Dec 2023 10:58:05 +0800,
  Junwang Zhao <zhjwpku@gmail.com> wrote:

>> 1. Add an opaque space for custom COPY TO handler
>>    * Add CopyToState{Get,Set}Opaque()
>>    https://github.com/kou/postgres/commit/5a610b6a066243f971e029432db67152cfe5e944
>>
>> 2. Export CopyToState::attnumlist
>>    * Add CopyToStateGetAttNumList()
>>    https://github.com/kou/postgres/commit/15fcba8b4e95afa86edb3f677a7bdb1acb1e7688
>>
>> 3. Export CopySend*()
>>    * Rename CopySend*() to CopyToStateSend*() and export them
>>    * Exception: CopySendEndOfRow() to CopyToStateFlush() because
>>      it just flushes the internal buffer now.
>>    https://github.com/kou/postgres/commit/289a5640135bde6733a1b8e2c412221ad522901e
>>
> I guess the purpose of these helpers is to avoid expose CopyToState to
> copy.h,

Yes.

>         but I
> think expose CopyToState to user might make life easier, users might want to use
> the memory contexts of the structure (though I agree not all the
> fields are necessary
> for extension handers).

OK. I don't object it as I said in another e-mail:
https://www.postgresql.org/message-id/flat/20240110.120644.1876591646729327180.kou%40clear-code.com#d923173e9625c20319750155083cbd72

>> 2. Need an opaque space like IndexScanDesc::opaque does
>>
>>    * A custom COPY TO handler needs to keep its data
> 
> I once thought users might want to parse their own options, maybe this
> is a use case for this opaque space.

Good catch! I forgot to suggest a callback for custom format
options. How about the following API?

----
...
typedef bool (*CopyToProcessOption_function) (CopyToState cstate, DefElem *defel);

...
typedef bool (*CopyFromProcessOption_function) (CopyFromState cstate, DefElem *defel);

typedef struct CopyToFormatRoutine
{
	...
	CopyToProcessOption_function process_option_fn;
} CopyToFormatRoutine;

typedef struct CopyFromFormatRoutine
{
	...
	CopyFromProcessOption_function process_option_fn;
} CopyFromFormatRoutine;
----

----
diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c
index e7597894bf..1aa8b62551 100644
--- a/src/backend/commands/copy.c
+++ b/src/backend/commands/copy.c
@@ -416,6 +416,7 @@ void
 ProcessCopyOptions(ParseState *pstate,
 				   CopyFormatOptions *opts_out,
 				   bool is_from,
+				   void *cstate, /* CopyToState* for !is_from, CopyFromState* for is_from */
 				   List *options)
 {
 	bool		format_specified = false;
@@ -593,11 +594,19 @@ ProcessCopyOptions(ParseState *pstate,
 						 parser_errposition(pstate, defel->location)));
 		}
 		else
-			ereport(ERROR,
-					(errcode(ERRCODE_SYNTAX_ERROR),
-					 errmsg("option \"%s\" not recognized",
-							defel->defname),
-					 parser_errposition(pstate, defel->location)));
+		{
+			bool processed;
+			if (is_from)
+				processed = opts_out->from_ops->process_option_fn(cstate, defel);
+			else
+				processed = opts_out->to_ops->process_option_fn(cstate, defel);
+			if (!processed)
+				ereport(ERROR,
+						(errcode(ERRCODE_SYNTAX_ERROR),
+						 errmsg("option \"%s\" not recognized",
+								defel->defname),
+						 parser_errposition(pstate, defel->location)));
+		}
 	}
 
 	/*
----

> For the name, I thought private_data might be a better candidate than
> opaque, but I do not insist.

I don't have a strong opinion for this. Here are the number
of headers that use "private_data" and "opaque":

$ grep -r private_data --files-with-matches src/include | wc -l
6
$ grep -r opaque --files-with-matches src/include | wc -l
38

It seems that we use "opaque" than "private_data" in general.

but it seems that we use
"opaque" than "private_data" in our code.

> Do you use the arrow library to control the memory?

Yes.

>                                                     Is there a way that
> we can let the arrow use postgres' memory context?

Yes. Apache Arrow C++ provides a memory pool feature and we
can implement PostgreSQL's memory context based memory pool
for this. (But this is a custom COPY TO/FROM handler's
implementation details.)

>                                                    I'm not sure this
> is necessary, just raise the question for discussion.

Could you clarify what should we discuss? We should require
that COPY TO/FROM handlers should use PostgreSQL's memory
context for all internal memory allocations?

> +PG_FUNCTION_INFO_V1(copy_testfmt_handler);
> +Datum
> +copy_testfmt_handler(PG_FUNCTION_ARGS)
> +{
> + bool is_from = PG_GETARG_BOOL(0);
> + CopyFormatRoutine *cp = makeNode(CopyFormatRoutine);;
> +
> 
> extra semicolon.

I noticed it too :-)
But I ignored it because the current implementation is only
for discussion. We know that it may be dirty.


Thanks,
-- 
kou

  52. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2024-01-10T06:33:22Z

    On Wed, Jan 10, 2024 at 12:00 PM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <ZYTfqGppMc9e_w2k@paquier.xyz>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 22 Dec 2023 10:00:24 +0900,
>   Michael Paquier <michael@paquier.xyz> wrote:
>
> >> 3. Export CopySend*()
> >>
> >>    * If we like minimum API, we just need to export
> >>      CopySendData() and CopySendEndOfRow(). But
> >>      CopySend{String,Char,Int32,Int16}() will be convenient
> >>      custom COPY TO handlers. (A custom COPY TO handler for
> >>      Apache Arrow doesn't need them.)
> >
> > Hmm.  Not sure on this one.  This may come down to externalize the
> > manipulation of fe_msgbuf.  Particularly, could it be possible that
> > some custom formats don't care at all about the network order?
>
> It means that all custom formats should control byte order
> by themselves instead of using CopySendInt*() that always
> use network byte order, right? It makes sense. Let's export
> only CopySendData() and CopySendEndOfRow().
>
>
> >> 1. What value should be used for "format" in
> >>    PgMsg_CopyOutResponse message?
> >>
> >>    It's 1 for binary format and 0 for text/csv format.
> >>
> >>    Should we make it customizable by custom COPY TO handler?
> >>    If so, what value should be used for this?
> >
> > Interesting point.  It looks very tempting to give more flexibility to
> > people who'd like to use their own code as we have one byte in the
> > protocol but just use 0/1.  Hence it feels natural to have a callback
> > for that.
>
> OK. Let's add a callback something like:
>
> typedef int16 (*CopyToGetFormat_function) (CopyToState cstate);
>
> > It also means that we may want to think harder about copy_is_binary in
> > libpq in the future step.  Now, having a backend implementation does
> > not need any libpq bits, either, because a client stack may just want
> > to speak the Postgres protocol directly.  Perhaps a custom COPY
> > implementation would be OK with how things are in libpq, as well,
> > tweaking its way through with just text or binary.
>
> Can we defer this discussion after we commit a basic custom
> COPY format handler mechanism?
>
> >> 2. Do we need more tries for design discussion for the first
> >>    implementation? If we need, what should we try?
> >
> > A makeNode() is used with an allocation in the current memory context
> > in the function returning the handler.  I would have assume that this
> > stuff returns a handler as a const struct like table AMs.
>
> If we use this approach, we can't use the Sawada-san's
> idea[1] that provides a convenient API to hide
> CopyFormatRoutine internal. The idea provides
> MakeCopy{To,From}FormatRoutine(). They return a new
> CopyFormatRoutine* with suitable is_from member. They can't
> use static const CopyFormatRoutine because they may be called
> multiple times in the same process.
>
> We can use the satic const struct approach by choosing one
> of the followings:
>
> 1. Use separated function for COPY {TO,FROM} format handlers
>    as I suggested.
>
> 2. Don't provide convenient API. Developers construct
>    CopyFormatRoutine by themselves. But it may be a bit
>    tricky.
>
> 3. Similar to 2. but don't use a bit tricky approach (don't
>    embed Copy{To,From}FormatRoutine nodes into
>    CopyFormatRoutine).
>
>    Use unified function for COPY {TO,FROM} format handlers
>    but CopyFormatRoutine always have both of COPY {TO,FROM}
>    format routines and these routines aren't nodes:
>
>    typedef struct CopyToFormatRoutine
>    {
>            CopyToStart_function start_fn;
>            CopyToOneRow_function onerow_fn;
>            CopyToEnd_function end_fn;
>    } CopyToFormatRoutine;
>
>    /* XXX: just copied from COPY TO routines */
>    typedef struct CopyFromFormatRoutine
>    {
>            CopyFromStart_function start_fn;
>            CopyFromOneRow_function onerow_fn;
>            CopyFromEnd_function end_fn;
>    } CopyFromFormatRoutine;
>
>    typedef struct CopyFormatRoutine
>    {
>            NodeTag              type;
>
>            CopyToFormatRoutine     to_routine;
>            CopyFromFormatRoutine           from_routine;
>    } CopyFormatRoutine;
>
>    ----
>
>    static const CopyFormatRoutine testfmt_handler = {
>        .type = T_CopyFormatRoutine,
>        .to_routine = {
>            .start_fn = testfmt_copyto_start,
>            .onerow_fn = testfmt_copyto_onerow,
>            .end_fn = testfmt_copyto_end,
>        },
>        .from_routine = {
>            .start_fn = testfmt_copyfrom_start,
>            .onerow_fn = testfmt_copyfrom_onerow,
>            .end_fn = testfmt_copyfrom_end,
>        },
>    };
>
>    PG_FUNCTION_INFO_V1(copy_testfmt_handler);
>    Datum
>    copy_testfmt_handler(PG_FUNCTION_ARGS)
>    {
>            PG_RETURN_POINTER(&testfmt_handler);
>    }
>
> 4. ... other idea?

It's a just idea but the fourth idea is to provide a convenient macro
to make it easy to construct the CopyFormatRoutine. For example,

#define COPYTO_ROUTINE(...) (Node *) &(CopyToFormatRoutine) {__VA_ARGS__}

static const CopyFormatRoutine testfmt_copyto_handler = {
    .type = T_CopyFormatRoutine,
    .is_from = true,
    .routine = COPYTO_ROUTINE (
        .start_fn = testfmt_copyto_start,
        .onerow_fn = testfmt_copyto_onerow,
        .end_fn = testfmt_copyto_end
        )
};

Datum
copy_testfmt_handler(PG_FUNCTION_ARGS)
{
    PG_RETURN_POINTER(& testfmt_copyto_handler);
}

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  53. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-01-10T06:40:28Z

    Hi,

In <CAD21AoC_dhfS97DKwTL+2nvgBOYrmN9XVYrE8w2SuDgghb-yzg@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Wed, 10 Jan 2024 15:33:22 +0900,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

>> We can use the satic const struct approach by choosing one
>> of the followings:
>>
>> ...
>>
>> 4. ... other idea?
> 
> It's a just idea but the fourth idea is to provide a convenient macro
> to make it easy to construct the CopyFormatRoutine. For example,
> 
> #define COPYTO_ROUTINE(...) (Node *) &(CopyToFormatRoutine) {__VA_ARGS__}
> 
> static const CopyFormatRoutine testfmt_copyto_handler = {
>     .type = T_CopyFormatRoutine,
>     .is_from = true,
>     .routine = COPYTO_ROUTINE (
>         .start_fn = testfmt_copyto_start,
>         .onerow_fn = testfmt_copyto_onerow,
>         .end_fn = testfmt_copyto_end
>         )
> };
> 
> Datum
> copy_testfmt_handler(PG_FUNCTION_ARGS)
> {
>     PG_RETURN_POINTER(& testfmt_copyto_handler);
> }

Interesting. But I feel that it introduces another (a bit)
tricky mechanism...

BTW, we also need to set .type:

     .routine = COPYTO_ROUTINE (
         .type = T_CopyToFormatRoutine,
         .start_fn = testfmt_copyto_start,
         .onerow_fn = testfmt_copyto_onerow,
         .end_fn = testfmt_copyto_end
         )


Thanks,
-- 
kou

  54. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2024-01-10T07:53:48Z

    On Wed, Jan 10, 2024 at 3:40 PM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <CAD21AoC_dhfS97DKwTL+2nvgBOYrmN9XVYrE8w2SuDgghb-yzg@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Wed, 10 Jan 2024 15:33:22 +0900,
>   Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> >> We can use the satic const struct approach by choosing one
> >> of the followings:
> >>
> >> ...
> >>
> >> 4. ... other idea?
> >
> > It's a just idea but the fourth idea is to provide a convenient macro
> > to make it easy to construct the CopyFormatRoutine. For example,
> >
> > #define COPYTO_ROUTINE(...) (Node *) &(CopyToFormatRoutine) {__VA_ARGS__}
> >
> > static const CopyFormatRoutine testfmt_copyto_handler = {
> >     .type = T_CopyFormatRoutine,
> >     .is_from = true,
> >     .routine = COPYTO_ROUTINE (
> >         .start_fn = testfmt_copyto_start,
> >         .onerow_fn = testfmt_copyto_onerow,
> >         .end_fn = testfmt_copyto_end
> >         )
> > };
> >
> > Datum
> > copy_testfmt_handler(PG_FUNCTION_ARGS)
> > {
> >     PG_RETURN_POINTER(& testfmt_copyto_handler);
> > }
>
> Interesting. But I feel that it introduces another (a bit)
> tricky mechanism...

Right. On the other hand, I don't think the idea 3 is good for the
same reason Michael-san pointed out before[1][2].

>
> BTW, we also need to set .type:
>
>      .routine = COPYTO_ROUTINE (
>          .type = T_CopyToFormatRoutine,
>          .start_fn = testfmt_copyto_start,
>          .onerow_fn = testfmt_copyto_onerow,
>          .end_fn = testfmt_copyto_end
>          )

I think it's fine as the same is true for table AM.

[1] https://www.postgresql.org/message-id/ZXEUIy6wl4jHy6Nm%40paquier.xyz
[2] https://www.postgresql.org/message-id/ZXKm9tmnSPIVrqZz%40paquier.xyz

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  55. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-01-11T01:24:45Z

    Hi,

In <CAD21AoC4HVuxOrsX1fLwj=5hdEmjvZoQw6PJGzxqxHNnYSQUVQ@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Wed, 10 Jan 2024 16:53:48 +0900,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

>> Interesting. But I feel that it introduces another (a bit)
>> tricky mechanism...
> 
> Right. On the other hand, I don't think the idea 3 is good for the
> same reason Michael-san pointed out before[1][2].
>
> [1] https://www.postgresql.org/message-id/ZXEUIy6wl4jHy6Nm%40paquier.xyz
> [2] https://www.postgresql.org/message-id/ZXKm9tmnSPIVrqZz%40paquier.xyz

I think that the important part of the Michael-san's opinion
is "keep COPY TO implementation and COPY FROM implementation
separated for maintainability".

The patch focused in [1][2] uses one routine for both of
COPY TO and COPY FROM. If we use the approach, we need to
change one common routine from copyto.c and copyfrom.c (or
export callbacks from copyto.c and copyfrom.c and use them
in copyto.c to construct one common routine). It's
the problem.

The idea 3 still has separated routines for COPY TO and COPY
FROM. So I think that it still keeps COPY TO implementation
and COPY FROM implementation separated.

>> BTW, we also need to set .type:
>>
>>      .routine = COPYTO_ROUTINE (
>>          .type = T_CopyToFormatRoutine,
>>          .start_fn = testfmt_copyto_start,
>>          .onerow_fn = testfmt_copyto_onerow,
>>          .end_fn = testfmt_copyto_end
>>          )
> 
> I think it's fine as the same is true for table AM.

Ah, sorry. I should have said explicitly. I don't this that
it's not a problem. I just wanted to say that it's missing.


Defining one more static const struct instead of providing a
convenient (but a bit tricky) macro may be straightforward:

static const CopyToFormatRoutine testfmt_copyto_routine = {
    .type = T_CopyToFormatRoutine,
    .start_fn = testfmt_copyto_start,
    .onerow_fn = testfmt_copyto_onerow,
    .end_fn = testfmt_copyto_end
};

static const CopyFormatRoutine testfmt_copyto_handler = {
    .type = T_CopyFormatRoutine,
    .is_from = false,
    .routine = (Node *) &testfmt_copyto_routine
};


Thanks,
-- 
kou

  56. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-01-12T05:46:15Z

    Hi,

Here is the current summary for a this discussion to make
COPY format extendable. It's for reaching consensus and
starting implementing the feature. (I'll start implementing
the feature once we reach consensus.) If you have any
opinion, please share it.

Confirmed:

1.1 Making COPY format extendable will not reduce performance.
    [1]

Decisions:

2.1 Use separated handler for COPY TO and COPY FROM because
    our COPY TO implementation (copyto.c) and COPY FROM
    implementation (coypfrom.c) are separated.
    [2]

2.2 Don't use system catalog for COPY TO/FROM handlers. We can
    just use a function(internal) that returns a handler instead.
    [3]

2.3 The implementation must include documentation.
    [5]

2.4 The implementation must include test.
    [6]

2.5 The implementation should be consist of small patches
    for easy to review.
    [6]

2.7 Copy{To,From}State must have a opaque space for
    handlers.
    [8]

2.8 Export CopySendData() and CopySendEndOfRow() for COPY TO
    handlers.
    [8]

2.9 Make "format" in PgMsg_CopyOutResponse message
    extendable.
    [9]

2.10 Make makeNode() call avoidable in function(internal)
     that returns COPY TO/FROM handler.
     [9]

2.11 Custom COPY TO/FROM handlers must be able to parse
     their options.
     [11]

Discussing:

3.1 Should we use one function(internal) for COPY TO/FROM
    handlers or two function(internal)s (one is for COPY TO
    handler and another is for COPY FROM handler)?
    [4]

3.2 If we use separated function(internal) for COPY TO/FROM
    handlers, we need to define naming rule. For example,
    <method_name>_to(internal) for COPY TO handler and
    <method_name>_from(internal) for COPY FROM handler.
    [4]

3.3 Should we use prefix or suffix for function(internal)
    name to avoid name conflict with other handlers such as
    tablesample handlers?
    [7]

3.4 Should we export Copy{To,From}State? Or should we just
    provide getters/setters to access Copy{To,From}State
    internal?
    [10]


[1] https://www.postgresql.org/message-id/flat/20231204.153548.2126325458835528809.kou%40clear-code.com
[2] https://www.postgresql.org/message-id/flat/ZXEUIy6wl4jHy6Nm%40paquier.xyz
[3] https://www.postgresql.org/message-id/flat/CAD21AoAhcZkAp_WDJ4sSv_%2Bg2iCGjfyMFgeu7MxjnjX_FutZAg%40mail.gmail.com
[4] https://www.postgresql.org/message-id/flat/CAD21AoDkoGL6yJ_HjNOg9cU%3DaAdW8uQ3rSQOeRS0SX85LPPNwQ%40mail.gmail.com
[5] https://www.postgresql.org/message-id/flat/TY3PR01MB9889C9234CD220A3A7075F0DF589A%40TY3PR01MB9889.jpnprd01.prod.outlook.com
[6] https://www.postgresql.org/message-id/flat/ZXbiPNriHHyUrcTF%40paquier.xyz
[7] https://www.postgresql.org/message-id/flat/20231214.184414.2179134502876898942.kou%40clear-code.com
[8] https://www.postgresql.org/message-id/flat/20231221.183504.1240642084042888377.kou%40clear-code.com
[9] https://www.postgresql.org/message-id/flat/ZYTfqGppMc9e_w2k%40paquier.xyz
[10] https://www.postgresql.org/message-id/flat/CAD21AoD%3DUapH4Wh06G6H5XAzPJ0iJg9YcW8r7E2UEJkZ8QsosA%40mail.gmail.com
[11] https://www.postgresql.org/message-id/flat/20240110.152023.1920937326588672387.kou%40clear-code.com


Thanks,
-- 
kou

  57. Re: Make COPY format extendable: Extract COPY TO format implementations

    Junwang Zhao <zhjwpku@gmail.com> — 2024-01-12T06:40:41Z

    Hi,

On Wed, Jan 10, 2024 at 2:20 PM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <CAEG8a3+jG_NKOUmcxDyEX2xSggBXReZ4H=e3RFsUtedY88A03w@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 22 Dec 2023 10:58:05 +0800,
>   Junwang Zhao <zhjwpku@gmail.com> wrote:
>
> >> 1. Add an opaque space for custom COPY TO handler
> >>    * Add CopyToState{Get,Set}Opaque()
> >>    https://github.com/kou/postgres/commit/5a610b6a066243f971e029432db67152cfe5e944
> >>
> >> 2. Export CopyToState::attnumlist
> >>    * Add CopyToStateGetAttNumList()
> >>    https://github.com/kou/postgres/commit/15fcba8b4e95afa86edb3f677a7bdb1acb1e7688
> >>
> >> 3. Export CopySend*()
> >>    * Rename CopySend*() to CopyToStateSend*() and export them
> >>    * Exception: CopySendEndOfRow() to CopyToStateFlush() because
> >>      it just flushes the internal buffer now.
> >>    https://github.com/kou/postgres/commit/289a5640135bde6733a1b8e2c412221ad522901e
> >>
> > I guess the purpose of these helpers is to avoid expose CopyToState to
> > copy.h,
>
> Yes.
>
> >         but I
> > think expose CopyToState to user might make life easier, users might want to use
> > the memory contexts of the structure (though I agree not all the
> > fields are necessary
> > for extension handers).
>
> OK. I don't object it as I said in another e-mail:
> https://www.postgresql.org/message-id/flat/20240110.120644.1876591646729327180.kou%40clear-code.com#d923173e9625c20319750155083cbd72
>
> >> 2. Need an opaque space like IndexScanDesc::opaque does
> >>
> >>    * A custom COPY TO handler needs to keep its data
> >
> > I once thought users might want to parse their own options, maybe this
> > is a use case for this opaque space.
>
> Good catch! I forgot to suggest a callback for custom format
> options. How about the following API?
>
> ----
> ...
> typedef bool (*CopyToProcessOption_function) (CopyToState cstate, DefElem *defel);
>
> ...
> typedef bool (*CopyFromProcessOption_function) (CopyFromState cstate, DefElem *defel);
>
> typedef struct CopyToFormatRoutine
> {
>         ...
>         CopyToProcessOption_function process_option_fn;
> } CopyToFormatRoutine;
>
> typedef struct CopyFromFormatRoutine
> {
>         ...
>         CopyFromProcessOption_function process_option_fn;
> } CopyFromFormatRoutine;
> ----
>
> ----
> diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c
> index e7597894bf..1aa8b62551 100644
> --- a/src/backend/commands/copy.c
> +++ b/src/backend/commands/copy.c
> @@ -416,6 +416,7 @@ void
>  ProcessCopyOptions(ParseState *pstate,
>                                    CopyFormatOptions *opts_out,
>                                    bool is_from,
> +                                  void *cstate, /* CopyToState* for !is_from, CopyFromState* for is_from */
>                                    List *options)
>  {
>         bool            format_specified = false;
> @@ -593,11 +594,19 @@ ProcessCopyOptions(ParseState *pstate,
>                                                  parser_errposition(pstate, defel->location)));
>                 }
>                 else
> -                       ereport(ERROR,
> -                                       (errcode(ERRCODE_SYNTAX_ERROR),
> -                                        errmsg("option \"%s\" not recognized",
> -                                                       defel->defname),
> -                                        parser_errposition(pstate, defel->location)));
> +               {
> +                       bool processed;
> +                       if (is_from)
> +                               processed = opts_out->from_ops->process_option_fn(cstate, defel);
> +                       else
> +                               processed = opts_out->to_ops->process_option_fn(cstate, defel);
> +                       if (!processed)
> +                               ereport(ERROR,
> +                                               (errcode(ERRCODE_SYNTAX_ERROR),
> +                                                errmsg("option \"%s\" not recognized",
> +                                                               defel->defname),
> +                                                parser_errposition(pstate, defel->location)));
> +               }
>         }
>
>         /*
> ----

Looks good.

>
> > For the name, I thought private_data might be a better candidate than
> > opaque, but I do not insist.
>
> I don't have a strong opinion for this. Here are the number
> of headers that use "private_data" and "opaque":
>
> $ grep -r private_data --files-with-matches src/include | wc -l
> 6
> $ grep -r opaque --files-with-matches src/include | wc -l
> 38
>
> It seems that we use "opaque" than "private_data" in general.
>
> but it seems that we use
> "opaque" than "private_data" in our code.
>
> > Do you use the arrow library to control the memory?
>
> Yes.
>
> >                                                     Is there a way that
> > we can let the arrow use postgres' memory context?
>
> Yes. Apache Arrow C++ provides a memory pool feature and we
> can implement PostgreSQL's memory context based memory pool
> for this. (But this is a custom COPY TO/FROM handler's
> implementation details.)
>
> >                                                    I'm not sure this
> > is necessary, just raise the question for discussion.
>
> Could you clarify what should we discuss? We should require
> that COPY TO/FROM handlers should use PostgreSQL's memory
> context for all internal memory allocations?

Yes, handlers should use PostgreSQL's memory context, and I think
creating other memory context under CopyToStateData.copycontext
should be suggested for handler creators, so I proposed exporting
CopyToStateData to public header.
>
> > +PG_FUNCTION_INFO_V1(copy_testfmt_handler);
> > +Datum
> > +copy_testfmt_handler(PG_FUNCTION_ARGS)
> > +{
> > + bool is_from = PG_GETARG_BOOL(0);
> > + CopyFormatRoutine *cp = makeNode(CopyFormatRoutine);;
> > +
> >
> > extra semicolon.
>
> I noticed it too :-)
> But I ignored it because the current implementation is only
> for discussion. We know that it may be dirty.
>
>
> Thanks,
> --
> kou



-- 
Regards
Junwang Zhao

  58. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-01-15T06:23:50Z

    Hi,

In <CAEG8a3J02NzGBxG1rP9C4u7qRLOqUjSOdy3q5_5v__fydS3XcA@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 12 Jan 2024 14:40:41 +0800,
  Junwang Zhao <zhjwpku@gmail.com> wrote:

>> Could you clarify what should we discuss? We should require
>> that COPY TO/FROM handlers should use PostgreSQL's memory
>> context for all internal memory allocations?
> 
> Yes, handlers should use PostgreSQL's memory context, and I think
> creating other memory context under CopyToStateData.copycontext
> should be suggested for handler creators, so I proposed exporting
> CopyToStateData to public header.

I see.

We can provide a getter for CopyToStateData::copycontext if
we don't want to export CopyToStateData. Note that I don't
have a strong opinion whether we should export
CopyToStateData or not.


Thanks,
-- 
kou

  59. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-01-15T06:27:02Z

    Hi,

If there are no more comments for the current design, I'll
start implementing this feature with the following
approaches for "Discussing" items:

> 3.1 Should we use one function(internal) for COPY TO/FROM
>     handlers or two function(internal)s (one is for COPY TO
>     handler and another is for COPY FROM handler)?
>     [4]

I'll choose "one function(internal) for COPY TO/FROM handlers".

> 3.4 Should we export Copy{To,From}State? Or should we just
>     provide getters/setters to access Copy{To,From}State
>     internal?
>     [10]

I'll export Copy{To,From}State.


Thanks,
-- 
kou

In <20240112.144615.157925223373344229.kou@clear-code.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 12 Jan 2024 14:46:15 +0900 (JST),
  Sutou Kouhei <kou@clear-code.com> wrote:

> Hi,
> 
> Here is the current summary for a this discussion to make
> COPY format extendable. It's for reaching consensus and
> starting implementing the feature. (I'll start implementing
> the feature once we reach consensus.) If you have any
> opinion, please share it.
> 
> Confirmed:
> 
> 1.1 Making COPY format extendable will not reduce performance.
>     [1]
> 
> Decisions:
> 
> 2.1 Use separated handler for COPY TO and COPY FROM because
>     our COPY TO implementation (copyto.c) and COPY FROM
>     implementation (coypfrom.c) are separated.
>     [2]
> 
> 2.2 Don't use system catalog for COPY TO/FROM handlers. We can
>     just use a function(internal) that returns a handler instead.
>     [3]
> 
> 2.3 The implementation must include documentation.
>     [5]
> 
> 2.4 The implementation must include test.
>     [6]
> 
> 2.5 The implementation should be consist of small patches
>     for easy to review.
>     [6]
> 
> 2.7 Copy{To,From}State must have a opaque space for
>     handlers.
>     [8]
> 
> 2.8 Export CopySendData() and CopySendEndOfRow() for COPY TO
>     handlers.
>     [8]
> 
> 2.9 Make "format" in PgMsg_CopyOutResponse message
>     extendable.
>     [9]
> 
> 2.10 Make makeNode() call avoidable in function(internal)
>      that returns COPY TO/FROM handler.
>      [9]
> 
> 2.11 Custom COPY TO/FROM handlers must be able to parse
>      their options.
>      [11]
> 
> Discussing:
> 
> 3.1 Should we use one function(internal) for COPY TO/FROM
>     handlers or two function(internal)s (one is for COPY TO
>     handler and another is for COPY FROM handler)?
>     [4]
> 
> 3.2 If we use separated function(internal) for COPY TO/FROM
>     handlers, we need to define naming rule. For example,
>     <method_name>_to(internal) for COPY TO handler and
>     <method_name>_from(internal) for COPY FROM handler.
>     [4]
> 
> 3.3 Should we use prefix or suffix for function(internal)
>     name to avoid name conflict with other handlers such as
>     tablesample handlers?
>     [7]
> 
> 3.4 Should we export Copy{To,From}State? Or should we just
>     provide getters/setters to access Copy{To,From}State
>     internal?
>     [10]
> 
> 
> [1] https://www.postgresql.org/message-id/flat/20231204.153548.2126325458835528809.kou%40clear-code.com
> [2] https://www.postgresql.org/message-id/flat/ZXEUIy6wl4jHy6Nm%40paquier.xyz
> [3] https://www.postgresql.org/message-id/flat/CAD21AoAhcZkAp_WDJ4sSv_%2Bg2iCGjfyMFgeu7MxjnjX_FutZAg%40mail.gmail.com
> [4] https://www.postgresql.org/message-id/flat/CAD21AoDkoGL6yJ_HjNOg9cU%3DaAdW8uQ3rSQOeRS0SX85LPPNwQ%40mail.gmail.com
> [5] https://www.postgresql.org/message-id/flat/TY3PR01MB9889C9234CD220A3A7075F0DF589A%40TY3PR01MB9889.jpnprd01.prod.outlook.com
> [6] https://www.postgresql.org/message-id/flat/ZXbiPNriHHyUrcTF%40paquier.xyz
> [7] https://www.postgresql.org/message-id/flat/20231214.184414.2179134502876898942.kou%40clear-code.com
> [8] https://www.postgresql.org/message-id/flat/20231221.183504.1240642084042888377.kou%40clear-code.com
> [9] https://www.postgresql.org/message-id/flat/ZYTfqGppMc9e_w2k%40paquier.xyz
> [10] https://www.postgresql.org/message-id/flat/CAD21AoD%3DUapH4Wh06G6H5XAzPJ0iJg9YcW8r7E2UEJkZ8QsosA%40mail.gmail.com
> [11] https://www.postgresql.org/message-id/flat/20240110.152023.1920937326588672387.kou%40clear-code.com
> 
> 
> Thanks,
> -- 
> kou
> 
>

  60. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2024-01-15T07:03:41Z

    On Thu, Jan 11, 2024 at 10:24 AM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <CAD21AoC4HVuxOrsX1fLwj=5hdEmjvZoQw6PJGzxqxHNnYSQUVQ@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Wed, 10 Jan 2024 16:53:48 +0900,
>   Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> >> Interesting. But I feel that it introduces another (a bit)
> >> tricky mechanism...
> >
> > Right. On the other hand, I don't think the idea 3 is good for the
> > same reason Michael-san pointed out before[1][2].
> >
> > [1] https://www.postgresql.org/message-id/ZXEUIy6wl4jHy6Nm%40paquier.xyz
> > [2] https://www.postgresql.org/message-id/ZXKm9tmnSPIVrqZz%40paquier.xyz
>
> I think that the important part of the Michael-san's opinion
> is "keep COPY TO implementation and COPY FROM implementation
> separated for maintainability".
>
> The patch focused in [1][2] uses one routine for both of
> COPY TO and COPY FROM. If we use the approach, we need to
> change one common routine from copyto.c and copyfrom.c (or
> export callbacks from copyto.c and copyfrom.c and use them
> in copyto.c to construct one common routine). It's
> the problem.
>
> The idea 3 still has separated routines for COPY TO and COPY
> FROM. So I think that it still keeps COPY TO implementation
> and COPY FROM implementation separated.
>
> >> BTW, we also need to set .type:
> >>
> >>      .routine = COPYTO_ROUTINE (
> >>          .type = T_CopyToFormatRoutine,
> >>          .start_fn = testfmt_copyto_start,
> >>          .onerow_fn = testfmt_copyto_onerow,
> >>          .end_fn = testfmt_copyto_end
> >>          )
> >
> > I think it's fine as the same is true for table AM.
>
> Ah, sorry. I should have said explicitly. I don't this that
> it's not a problem. I just wanted to say that it's missing.

Thank you for pointing it out.

>
>
> Defining one more static const struct instead of providing a
> convenient (but a bit tricky) macro may be straightforward:
>
> static const CopyToFormatRoutine testfmt_copyto_routine = {
>     .type = T_CopyToFormatRoutine,
>     .start_fn = testfmt_copyto_start,
>     .onerow_fn = testfmt_copyto_onerow,
>     .end_fn = testfmt_copyto_end
> };
>
> static const CopyFormatRoutine testfmt_copyto_handler = {
>     .type = T_CopyFormatRoutine,
>     .is_from = false,
>     .routine = (Node *) &testfmt_copyto_routine
> };

Yeah, IIUC this is the option 2 you mentioned[1]. I think we can go
with this idea as it's the simplest. If we find a better way, we can
change it later. That is CopyFormatRoutine will be like:

typedef struct CopyFormatRoutine
{
    NodeTag     type;

    /* either CopyToFormatRoutine or CopyFromFormatRoutine */
    Node       *routine;
}           CopyFormatRoutine;

And the core can check the node type of the 'routine7 in the
CopyFormatRoutine returned by extensions.

Regards,

[1] https://www.postgresql.org/message-id/20240110.120034.501385498034538233.kou%40clear-code.com

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  61. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-01-16T02:53:00Z

    Hi,

In <CAD21AoB5x86TTyer90iSFivnSD8MFRU8V4ALzmQ=rQFw4QqiXQ@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Mon, 15 Jan 2024 16:03:41 +0900,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

>> Defining one more static const struct instead of providing a
>> convenient (but a bit tricky) macro may be straightforward:
>>
>> static const CopyToFormatRoutine testfmt_copyto_routine = {
>>     .type = T_CopyToFormatRoutine,
>>     .start_fn = testfmt_copyto_start,
>>     .onerow_fn = testfmt_copyto_onerow,
>>     .end_fn = testfmt_copyto_end
>> };
>>
>> static const CopyFormatRoutine testfmt_copyto_handler = {
>>     .type = T_CopyFormatRoutine,
>>     .is_from = false,
>>     .routine = (Node *) &testfmt_copyto_routine
>> };
> 
> Yeah, IIUC this is the option 2 you mentioned[1]. I think we can go
> with this idea as it's the simplest.
>
> [1] https://www.postgresql.org/message-id/20240110.120034.501385498034538233.kou%40clear-code.com

Ah, you're right. I forgot it...

>                  That is CopyFormatRoutine will be like:
> 
> typedef struct CopyFormatRoutine
> {
>     NodeTag     type;
> 
>     /* either CopyToFormatRoutine or CopyFromFormatRoutine */
>     Node       *routine;
> }           CopyFormatRoutine;
> 
> And the core can check the node type of the 'routine7 in the
> CopyFormatRoutine returned by extensions.

It makes sense.


If no more comments about the current design, I'll start
implementing this feature based on the current design.


Thanks,
-- 
kou

  62. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-01-24T05:49:36Z

    Hi,

I've implemented custom COPY format feature based on the
current design discussion. See the attached patches for
details.

I also implemented a PoC COPY format handler for Apache
Arrow with this implementation and it worked.
https://github.com/kou/pg-copy-arrow

The patches implement not only custom COPY TO format feature
but also custom COPY FROM format feature.

0001-0004 is for COPY TO and 0005-0008 is for COPY FROM.

For COPY TO:

0001: This adds CopyToRoutine and use it for text/csv/binary
formats. No implementation change. This just move codes.

0002: This adds support for adding custom COPY TO format by
"CREATE FUNCTION ${FORMAT_NAME}". This uses the same
approach provided by Sawada-san[1] but this doesn't
introduce a wrapper CopyRoutine struct for
Copy{To,From}Routine. Because I noticed that a wrapper
CopyRoutine struct is needless. Copy handler can just return
CopyToRoutine or CopyFromRtouine because both of them have
NodeTag. We can distinct a returned struct by the NodeTag.

[1] https://www.postgresql.org/message-id/CAD21AoCunywHird3GaPzWe6s9JG1wzxj3Cr6vGN36DDheGjOjA@mail.gmail.com

0003: This exports CopyToStateData. No implementation change
except CopyDest enum values. I changed COPY_ prefix to
COPY_DEST_ to avoid name conflict with CopySource enum
values. This just moves codes.

0004: This adds CopyToState::opaque and exports
CopySendEndOfRow(). CopySendEndOfRow() is renamed to
CopyToStateFlush().

For COPY FROM:

0005: Same as 0001 but for COPY FROM. This adds
CopyFromRoutine and use it for text/csv/binary formats. No
implementation change. This just move codes.

0006: Same as 0002 but for COPY FROM. This adds support for
adding custom COPY FROM format by "CREATE FUNCTION
${FORMAT_NAME}".

0007: Same as 0003 but for COPY FROM. This exports
CopyFromStateData. No implementation change except
CopySource enum values. I changed COPY_ prefix to
COPY_SOURCE_ to align CopyDest changes in 0003. This just
moves codes.

0008: Same as 0004 but for COPY FROM. This adds
CopyFromState::opaque and exports
CopyReadBinaryData(). CopyReadBinaryData() is renamed to
CopyFromStateRead().


Thanks,
-- 
kou

In <20240115.152702.2011620917962812379.kou@clear-code.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Mon, 15 Jan 2024 15:27:02 +0900 (JST),
  Sutou Kouhei <kou@clear-code.com> wrote:

> Hi,
> 
> If there are no more comments for the current design, I'll
> start implementing this feature with the following
> approaches for "Discussing" items:
> 
>> 3.1 Should we use one function(internal) for COPY TO/FROM
>>     handlers or two function(internal)s (one is for COPY TO
>>     handler and another is for COPY FROM handler)?
>>     [4]
> 
> I'll choose "one function(internal) for COPY TO/FROM handlers".
> 
>> 3.4 Should we export Copy{To,From}State? Or should we just
>>     provide getters/setters to access Copy{To,From}State
>>     internal?
>>     [10]
> 
> I'll export Copy{To,From}State.
> 
> 
> Thanks,
> -- 
> kou
> 
> In <20240112.144615.157925223373344229.kou@clear-code.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 12 Jan 2024 14:46:15 +0900 (JST),
>   Sutou Kouhei <kou@clear-code.com> wrote:
> 
>> Hi,
>> 
>> Here is the current summary for a this discussion to make
>> COPY format extendable. It's for reaching consensus and
>> starting implementing the feature. (I'll start implementing
>> the feature once we reach consensus.) If you have any
>> opinion, please share it.
>> 
>> Confirmed:
>> 
>> 1.1 Making COPY format extendable will not reduce performance.
>>     [1]
>> 
>> Decisions:
>> 
>> 2.1 Use separated handler for COPY TO and COPY FROM because
>>     our COPY TO implementation (copyto.c) and COPY FROM
>>     implementation (coypfrom.c) are separated.
>>     [2]
>> 
>> 2.2 Don't use system catalog for COPY TO/FROM handlers. We can
>>     just use a function(internal) that returns a handler instead.
>>     [3]
>> 
>> 2.3 The implementation must include documentation.
>>     [5]
>> 
>> 2.4 The implementation must include test.
>>     [6]
>> 
>> 2.5 The implementation should be consist of small patches
>>     for easy to review.
>>     [6]
>> 
>> 2.7 Copy{To,From}State must have a opaque space for
>>     handlers.
>>     [8]
>> 
>> 2.8 Export CopySendData() and CopySendEndOfRow() for COPY TO
>>     handlers.
>>     [8]
>> 
>> 2.9 Make "format" in PgMsg_CopyOutResponse message
>>     extendable.
>>     [9]
>> 
>> 2.10 Make makeNode() call avoidable in function(internal)
>>      that returns COPY TO/FROM handler.
>>      [9]
>> 
>> 2.11 Custom COPY TO/FROM handlers must be able to parse
>>      their options.
>>      [11]
>> 
>> Discussing:
>> 
>> 3.1 Should we use one function(internal) for COPY TO/FROM
>>     handlers or two function(internal)s (one is for COPY TO
>>     handler and another is for COPY FROM handler)?
>>     [4]
>> 
>> 3.2 If we use separated function(internal) for COPY TO/FROM
>>     handlers, we need to define naming rule. For example,
>>     <method_name>_to(internal) for COPY TO handler and
>>     <method_name>_from(internal) for COPY FROM handler.
>>     [4]
>> 
>> 3.3 Should we use prefix or suffix for function(internal)
>>     name to avoid name conflict with other handlers such as
>>     tablesample handlers?
>>     [7]
>> 
>> 3.4 Should we export Copy{To,From}State? Or should we just
>>     provide getters/setters to access Copy{To,From}State
>>     internal?
>>     [10]
>> 
>> 
>> [1] https://www.postgresql.org/message-id/flat/20231204.153548.2126325458835528809.kou%40clear-code.com
>> [2] https://www.postgresql.org/message-id/flat/ZXEUIy6wl4jHy6Nm%40paquier.xyz
>> [3] https://www.postgresql.org/message-id/flat/CAD21AoAhcZkAp_WDJ4sSv_%2Bg2iCGjfyMFgeu7MxjnjX_FutZAg%40mail.gmail.com
>> [4] https://www.postgresql.org/message-id/flat/CAD21AoDkoGL6yJ_HjNOg9cU%3DaAdW8uQ3rSQOeRS0SX85LPPNwQ%40mail.gmail.com
>> [5] https://www.postgresql.org/message-id/flat/TY3PR01MB9889C9234CD220A3A7075F0DF589A%40TY3PR01MB9889.jpnprd01.prod.outlook.com
>> [6] https://www.postgresql.org/message-id/flat/ZXbiPNriHHyUrcTF%40paquier.xyz
>> [7] https://www.postgresql.org/message-id/flat/20231214.184414.2179134502876898942.kou%40clear-code.com
>> [8] https://www.postgresql.org/message-id/flat/20231221.183504.1240642084042888377.kou%40clear-code.com
>> [9] https://www.postgresql.org/message-id/flat/ZYTfqGppMc9e_w2k%40paquier.xyz
>> [10] https://www.postgresql.org/message-id/flat/CAD21AoD%3DUapH4Wh06G6H5XAzPJ0iJg9YcW8r7E2UEJkZ8QsosA%40mail.gmail.com
>> [11] https://www.postgresql.org/message-id/flat/20240110.152023.1920937326588672387.kou%40clear-code.com
>> 
>> 
>> Thanks,
>> -- 
>> kou
>> 
>> 
> 
>

  63. Re: Make COPY format extendable: Extract COPY TO format implementations

    Michael Paquier <michael@paquier.xyz> — 2024-01-24T08:11:49Z

    On Wed, Jan 24, 2024 at 02:49:36PM +0900, Sutou Kouhei wrote:
> For COPY TO:
> 
> 0001: This adds CopyToRoutine and use it for text/csv/binary
> formats. No implementation change. This just move codes.

10M without this change:

    format,elapsed time (ms)
    text,1090.763
    csv,1136.103
    binary,1137.141

10M with this change:

    format,elapsed time (ms)
    text,1082.654
    csv,1196.991
    binary,1069.697

These numbers point out that binary is faster by 6%, csv is slower by
5%, while text stays around what looks like noise range.  That's not
negligible.  Are these numbers reproducible?  If they are, that could
be a problem for anybody doing bulk-loading of large data sets.  I am
not sure to understand where the improvement for binary comes from by
reading the patch, but perhaps perf would tell more for each format?
The loss with csv could be blamed on the extra manipulations of the
function pointers, likely.
--
Michael

  64. Re: Make COPY format extendable: Extract COPY TO format implementations

    Andrew Dunstan <andrew@dunslane.net> — 2024-01-24T12:15:55Z

    On 2024-01-24 We 03:11, Michael Paquier wrote:
> On Wed, Jan 24, 2024 at 02:49:36PM +0900, Sutou Kouhei wrote:
>> For COPY TO:
>>
>> 0001: This adds CopyToRoutine and use it for text/csv/binary
>> formats. No implementation change. This just move codes.
> 10M without this change:
>
>      format,elapsed time (ms)
>      text,1090.763
>      csv,1136.103
>      binary,1137.141
>
> 10M with this change:
>
>      format,elapsed time (ms)
>      text,1082.654
>      csv,1196.991
>      binary,1069.697
>
> These numbers point out that binary is faster by 6%, csv is slower by
> 5%, while text stays around what looks like noise range.  That's not
> negligible.  Are these numbers reproducible?  If they are, that could
> be a problem for anybody doing bulk-loading of large data sets.  I am
> not sure to understand where the improvement for binary comes from by
> reading the patch, but perhaps perf would tell more for each format?
> The loss with csv could be blamed on the extra manipulations of the
> function pointers, likely.


I don't think that's at all acceptable.

We've spent quite a lot of blood sweat and tears over the years to make 
COPY fast, and we should not sacrifice any of that lightly.


cheers


andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

  65. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-01-24T14:17:26Z

    Hi,

In <10025bac-158c-ffe7-fbec-32b42629121f@dunslane.net>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Wed, 24 Jan 2024 07:15:55 -0500,
  Andrew Dunstan <andrew@dunslane.net> wrote:

> 
> On 2024-01-24 We 03:11, Michael Paquier wrote:
>> On Wed, Jan 24, 2024 at 02:49:36PM +0900, Sutou Kouhei wrote:
>>> For COPY TO:
>>>
>>> 0001: This adds CopyToRoutine and use it for text/csv/binary
>>> formats. No implementation change. This just move codes.
>> 10M without this change:
>>
>>      format,elapsed time (ms)
>>      text,1090.763
>>      csv,1136.103
>>      binary,1137.141
>>
>> 10M with this change:
>>
>>      format,elapsed time (ms)
>>      text,1082.654
>>      csv,1196.991
>>      binary,1069.697
>>
>> These numbers point out that binary is faster by 6%, csv is slower by
>> 5%, while text stays around what looks like noise range.  That's not
>> negligible.  Are these numbers reproducible?  If they are, that could
>> be a problem for anybody doing bulk-loading of large data sets.  I am
>> not sure to understand where the improvement for binary comes from by
>> reading the patch, but perhaps perf would tell more for each format?
>> The loss with csv could be blamed on the extra manipulations of the
>> function pointers, likely.
> 
> 
> I don't think that's at all acceptable.
> 
> We've spent quite a lot of blood sweat and tears over the years to make COPY
> fast, and we should not sacrifice any of that lightly.

These numbers aren't reproducible. Because these benchmarks
executed on my normal machine not a machine only for
benchmarking. The machine runs another processes such as
editor and Web browser.

For example, here are some results with master
(94edfe250c6a200d2067b0debfe00b4122e9b11e):

Format,N records,Elapsed time (ms)
csv,10000000,1073.715
csv,10000000,1022.830
csv,10000000,1073.584
csv,10000000,1090.651
csv,10000000,1052.259

Here are some results with master + the 0001 patch:

Format,N records,Elapsed time (ms)
csv,10000000,1025.356
csv,10000000,1067.202
csv,10000000,1014.563
csv,10000000,1032.088
csv,10000000,1058.110


I uploaded my benchmark script so that you can run the same
benchmark on your machine:

https://gist.github.com/kou/be02e02e5072c91969469dbf137b5de5

Could anyone try the benchmark with master and master+0001?


Thanks,
-- 
kou

  66. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-01-24T14:20:22Z

    Hi,

In <20240124.144936.67229716500876806.kou@clear-code.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Wed, 24 Jan 2024 14:49:36 +0900 (JST),
  Sutou Kouhei <kou@clear-code.com> wrote:

> I've implemented custom COPY format feature based on the
> current design discussion. See the attached patches for
> details.

I forgot to mention one note. Documentation isn't included
in these patches. I'll write it after all (or some) patches
are merged. Is it OK?


Thanks,
-- 
kou

  67. Re: Make COPY format extendable: Extract COPY TO format implementations

    jian he <jian.universality@gmail.com> — 2024-01-25T02:53:58Z

    On Wed, Jan 24, 2024 at 10:17 PM Sutou Kouhei <kou@clear-code.com> wrote:
>
> I uploaded my benchmark script so that you can run the same
> benchmark on your machine:
>
> https://gist.github.com/kou/be02e02e5072c91969469dbf137b5de5
>
> Could anyone try the benchmark with master and master+0001?
>

sorry. I made a mistake. I applied v6, 0001 to 0008 all the patches.

my tests:
CREATE unlogged TABLE data (a bigint);
SELECT setseed(0.29);
INSERT INTO data SELECT random() * 10000 FROM generate_series(1, 1e7);

my setup:
meson setup --reconfigure ${BUILD} \
-Dprefix=${PG_PREFIX} \
-Dpgport=5462 \
-Dbuildtype=release \
-Ddocs_html_style=website \
-Ddocs_pdf=disabled \
-Dllvm=disabled \
-Dextra_version=_release_build

gcc version:  PostgreSQL 17devel_release_build on x86_64-linux,
compiled by gcc-11.4.0, 64-bit

apply your patch:
COPY data TO '/dev/null' WITH (FORMAT csv) \watch count=5
Time: 668.996 ms
Time: 596.254 ms
Time: 592.723 ms
Time: 591.663 ms
Time: 590.803 ms

not apply your patch, at git 729439607ad210dbb446e31754e8627d7e3f7dda
COPY data TO '/dev/null' WITH (FORMAT csv) \watch count=5
Time: 644.246 ms
Time: 583.075 ms
Time: 568.670 ms
Time: 569.463 ms
Time: 569.201 ms

I forgot to test other formats.

  68. Re: Make COPY format extendable: Extract COPY TO format implementations

    Michael Paquier <michael@paquier.xyz> — 2024-01-25T03:17:55Z

    On Wed, Jan 24, 2024 at 11:17:26PM +0900, Sutou Kouhei wrote:
> In <10025bac-158c-ffe7-fbec-32b42629121f@dunslane.net>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Wed, 24 Jan 2024 07:15:55 -0500,
>   Andrew Dunstan <andrew@dunslane.net> wrote:
>> We've spent quite a lot of blood sweat and tears over the years to make COPY
>> fast, and we should not sacrifice any of that lightly.

Clearly.

> I uploaded my benchmark script so that you can run the same
> benchmark on your machine:
> 
> https://gist.github.com/kou/be02e02e5072c91969469dbf137b5de5

Thanks, that saves time.  I am attaching it to this email as well, for
the sake of the archives if this link is removed in the future.

> Could anyone try the benchmark with master and master+0001?

Yep.  It is one point we need to settle before deciding what to do
with this patch set, and I've done so to reach my own conclusion.

I have a rather good machine at my disposal in the cloud, so I did a
few runs with HEAD and HEAD+0001, with PGDATA mounted on a tmpfs.
Here are some results for the 10M row case, as these should be the
least prone to noise, 5 runs each: 

master
text 10M  1732.570 1684.542 1693.430 1687.696 1714.845
csv 10M   1729.113 1724.926 1727.414 1726.237 1728.865
bin 10M   1679.097 1677.887 1676.764 1677.554 1678.120

master+0001
text 10M  1702.207 1654.818 1647.069 1690.568 1654.446
csv 10M   1764.939 1714.313 1712.444 1712.323 1716.952
bin 10M   1703.061 1702.719 1702.234 1703.346 1704.137

Hmm.  The point of contention in the patch is the change to use the
CopyToOneRow callback in CopyOneRowTo(), as we go through it for each
row and we should habe a worst-case scenario with a relation that has
a small attribute size.  The more rows, the more effect it would have.
The memory context switches and the StringInfo manipulations are
equally important, and there are a bunch of the latter, actually, with
optimizations around fe_msgbuf.

I've repeated a few runs across these two builds, and there is some
variance and noise, but I am going to agree with your point that the
effect 0001 cannot be seen.  Even HEAD is showing some noise.  So I am
discarding the concerns I had after seeing the numbers you posted
upthread.

+typedef bool (*CopyToProcessOption_function) (CopyToState cstate, DefElem *defel);
+typedef int16 (*CopyToGetFormat_function) (CopyToState cstate);
+typedef void (*CopyToStart_function) (CopyToState cstate, TupleDesc tupDesc);
+typedef void (*CopyToOneRow_function) (CopyToState cstate, TupleTableSlot *slot);
+typedef void (*CopyToEnd_function) (CopyToState cstate);

We don't really need a set of typedefs here, let's put the definitions
in the CopyToRoutine struct instead.

+extern CopyToRoutine CopyToRoutineText;
+extern CopyToRoutine CopyToRoutineCSV;
+extern CopyToRoutine CopyToRoutineBinary;

All that should IMO remain in copyto.c and copyfrom.c in the initial
patch doing the refactoring.  Why not using a fetch function instead
that uses a string in input?  Then you can call that once after
parsing the List of options in ProcessCopyOptions().

Introducing copyapi.h in the initial patch makes sense here for the TO
and FROM routines.

+/* All "text" and "csv" options are parsed in ProcessCopyOptions(). We may
+ * move the code to here later. */
Some areas, like this comment, are written in an incorrect format.

+            if (cstate->opts.csv_mode)
+                CopyAttributeOutCSV(cstate, colname, false,
+                                    list_length(cstate->attnumlist) == 1);
+            else
+                CopyAttributeOutText(cstate, colname);

You are right that this is not worth the trouble of creating a
different set of callbacks for CSV.  This makes the result cleaner.

+    getTypeBinaryOutputInfo(attr->atttypid, &out_func_oid, &isvarlena);
+    fmgr_info(out_func_oid, &cstate->out_functions[attnum - 1]);

Actually, this split is interesting.  It is possible for a custom
format to plug in a custom set of out functions.  Did you make use of
something custom for your own stuff?  Actually, could it make sense to
split the assignment of cstate->out_functions into its own callback?
Sure, that's part of the start phase, but at least it would make clear
that a custom method *has* to assign these OIDs to work.  The patch
implies that as a rule, without a comment that CopyToStart *must* set
up these OIDs.

I think that 0001 and 0005 should be handled first, as pieces
independent of the rest.  Then we could move on with 0002~0004 and
0006~0008.
--
Michael

  69. Re: Make COPY format extendable: Extract COPY TO format implementations

    Michael Paquier <michael@paquier.xyz> — 2024-01-25T03:28:01Z

    On Thu, Jan 25, 2024 at 10:53:58AM +0800, jian he wrote:
> apply your patch:
> COPY data TO '/dev/null' WITH (FORMAT csv) \watch count=5
> Time: 668.996 ms
> Time: 596.254 ms
> Time: 592.723 ms
> Time: 591.663 ms
> Time: 590.803 ms
> 
> not apply your patch, at git 729439607ad210dbb446e31754e8627d7e3f7dda
> COPY data TO '/dev/null' WITH (FORMAT csv) \watch count=5
> Time: 644.246 ms
> Time: 583.075 ms
> Time: 568.670 ms
> Time: 569.463 ms
> Time: 569.201 ms
> 
> I forgot to test other formats.

There can be some variance in the tests, so you'd better run much more
tests so as you can get a better idea of the mean.  Discarding the N
highest and lowest values also reduces slightly the effects of the
noise you would get across single runs.
--
Michael

  70. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2024-01-25T04:36:03Z

    On Wed, Jan 24, 2024 at 11:17 PM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <10025bac-158c-ffe7-fbec-32b42629121f@dunslane.net>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Wed, 24 Jan 2024 07:15:55 -0500,
>   Andrew Dunstan <andrew@dunslane.net> wrote:
>
> >
> > On 2024-01-24 We 03:11, Michael Paquier wrote:
> >> On Wed, Jan 24, 2024 at 02:49:36PM +0900, Sutou Kouhei wrote:
> >>> For COPY TO:
> >>>
> >>> 0001: This adds CopyToRoutine and use it for text/csv/binary
> >>> formats. No implementation change. This just move codes.
> >> 10M without this change:
> >>
> >>      format,elapsed time (ms)
> >>      text,1090.763
> >>      csv,1136.103
> >>      binary,1137.141
> >>
> >> 10M with this change:
> >>
> >>      format,elapsed time (ms)
> >>      text,1082.654
> >>      csv,1196.991
> >>      binary,1069.697
> >>
> >> These numbers point out that binary is faster by 6%, csv is slower by
> >> 5%, while text stays around what looks like noise range.  That's not
> >> negligible.  Are these numbers reproducible?  If they are, that could
> >> be a problem for anybody doing bulk-loading of large data sets.  I am
> >> not sure to understand where the improvement for binary comes from by
> >> reading the patch, but perhaps perf would tell more for each format?
> >> The loss with csv could be blamed on the extra manipulations of the
> >> function pointers, likely.
> >
> >
> > I don't think that's at all acceptable.
> >
> > We've spent quite a lot of blood sweat and tears over the years to make COPY
> > fast, and we should not sacrifice any of that lightly.
>
> These numbers aren't reproducible. Because these benchmarks
> executed on my normal machine not a machine only for
> benchmarking. The machine runs another processes such as
> editor and Web browser.
>
> For example, here are some results with master
> (94edfe250c6a200d2067b0debfe00b4122e9b11e):
>
> Format,N records,Elapsed time (ms)
> csv,10000000,1073.715
> csv,10000000,1022.830
> csv,10000000,1073.584
> csv,10000000,1090.651
> csv,10000000,1052.259
>
> Here are some results with master + the 0001 patch:
>
> Format,N records,Elapsed time (ms)
> csv,10000000,1025.356
> csv,10000000,1067.202
> csv,10000000,1014.563
> csv,10000000,1032.088
> csv,10000000,1058.110
>
>
> I uploaded my benchmark script so that you can run the same
> benchmark on your machine:
>
> https://gist.github.com/kou/be02e02e5072c91969469dbf137b5de5
>
> Could anyone try the benchmark with master and master+0001?
>

I've run a similar scenario:

create unlogged table test (a int);
insert into test select c from generate_series(1, 25000000) c;
copy test to '/tmp/result.csv' with (format csv); -- generates 230MB file

I've run it on HEAD and HEAD+0001 patch and here are the medians of 10
executions for each format:

HEAD:
binary 2930.353 ms
text 2754.852 ms
csv 2890.012 ms

HEAD w/ 0001 patch:
binary 2814.838 ms
text 2900.845 ms
csv 3015.210 ms

Hmm I can see a similar trend that Suto-san had; the binary format got
slightly faster whereas both text and csv format has small regression
(4%~5%). I think that the improvement for binary came from the fact
that we removed "if (cstate->opts.binary)" branches from the original
CopyOneRowTo(). I've experimented with a similar optimization for csv
and text format; have different callbacks for text and csv format and
remove "if (cstate->opts.csv_mode)" branches. I've attached a patch
for that. Here are results:

HEAD w/ 0001 patch + remove branches:
binary 2824.502 ms
text 2715.264 ms
csv 2803.381 ms

The numbers look better now. I'm not sure these are within a noise
range but it might be worth considering having different callbacks for
text and csv formats.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  71. Re: Make COPY format extendable: Extract COPY TO format implementations

    Michael Paquier <michael@paquier.xyz> — 2024-01-25T04:53:30Z

    On Thu, Jan 25, 2024 at 01:36:03PM +0900, Masahiko Sawada wrote:
> Hmm I can see a similar trend that Suto-san had; the binary format got
> slightly faster whereas both text and csv format has small regression
> (4%~5%). I think that the improvement for binary came from the fact
> that we removed "if (cstate->opts.binary)" branches from the original
> CopyOneRowTo(). I've experimented with a similar optimization for csv
> and text format; have different callbacks for text and csv format and
> remove "if (cstate->opts.csv_mode)" branches. I've attached a patch
> for that. Here are results:
> 
> HEAD w/ 0001 patch + remove branches:
> binary 2824.502 ms
> text 2715.264 ms
> csv 2803.381 ms
> 
> The numbers look better now. I'm not sure these are within a noise
> range but it might be worth considering having different callbacks for
> text and csv formats.

Interesting.

Your numbers imply a 0.3% speedup for text, 0.7% speedup for csv and
0.9% speedup for binary, which may be around the noise range assuming
a ~1% range.  While this does not imply a regression, that seems worth
the duplication IMO.  The patch had better document the reason why the
split is done, as well.

CopyFromTextOneRow() has also specific branches for binary and
non-binary removed in 0005, so assuming that I/O is not a bottleneck,
the operation would be faster because we would not evaluate this "if"
condition for each row.  Wouldn't we also see improvements for COPY
FROM with short row values, say when mounting PGDATA into a
tmpfs/ramfs?
--
Michael

  72. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2024-01-25T05:28:38Z

    On Thu, Jan 25, 2024 at 1:53 PM Michael Paquier <michael@paquier.xyz> wrote:
>
> On Thu, Jan 25, 2024 at 01:36:03PM +0900, Masahiko Sawada wrote:
> > Hmm I can see a similar trend that Suto-san had; the binary format got
> > slightly faster whereas both text and csv format has small regression
> > (4%~5%). I think that the improvement for binary came from the fact
> > that we removed "if (cstate->opts.binary)" branches from the original
> > CopyOneRowTo(). I've experimented with a similar optimization for csv
> > and text format; have different callbacks for text and csv format and
> > remove "if (cstate->opts.csv_mode)" branches. I've attached a patch
> > for that. Here are results:
> >
> > HEAD w/ 0001 patch + remove branches:
> > binary 2824.502 ms
> > text 2715.264 ms
> > csv 2803.381 ms
> >
> > The numbers look better now. I'm not sure these are within a noise
> > range but it might be worth considering having different callbacks for
> > text and csv formats.
>
> Interesting.
>
> Your numbers imply a 0.3% speedup for text, 0.7% speedup for csv and
> 0.9% speedup for binary, which may be around the noise range assuming
> a ~1% range.  While this does not imply a regression, that seems worth
> the duplication IMO.

Agreed. In addition to that, now that each format routine has its own
callbacks, there would be chances that we can do other optimizations
dedicated to the format type in the future if available.

>  The patch had better document the reason why the
> split is done, as well.

+1

>
> CopyFromTextOneRow() has also specific branches for binary and
> non-binary removed in 0005, so assuming that I/O is not a bottleneck,
> the operation would be faster because we would not evaluate this "if"
> condition for each row.  Wouldn't we also see improvements for COPY
> FROM with short row values, say when mounting PGDATA into a
> tmpfs/ramfs?

Probably. Seems worth evaluating.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  73. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-01-25T08:05:30Z

    Hi,

Thanks for trying these patches!

In <CACJufxF9NS3xQ2d79jN0V1CGvF7cR16uJo-C3nrY7vZrwvxF7w@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Thu, 25 Jan 2024 10:53:58 +0800,
  jian he <jian.universality@gmail.com> wrote:

> COPY data TO '/dev/null' WITH (FORMAT csv) \watch count=5

Wow! I didn't know the "\watch count="!
I'll use it.

> Time: 668.996 ms
> Time: 596.254 ms
> Time: 592.723 ms
> Time: 591.663 ms
> Time: 590.803 ms

It seems that 5 times isn't enough for this case as Michael
said. But thanks for trying!


Thanks,
-- 
kou

  74. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-01-25T08:45:43Z

    Hi,

In <ZbHS439y-Bs6HIAR@paquier.xyz>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Thu, 25 Jan 2024 12:17:55 +0900,
  Michael Paquier <michael@paquier.xyz> wrote:

> +typedef bool (*CopyToProcessOption_function) (CopyToState cstate, DefElem *defel);
> +typedef int16 (*CopyToGetFormat_function) (CopyToState cstate);
> +typedef void (*CopyToStart_function) (CopyToState cstate, TupleDesc tupDesc);
> +typedef void (*CopyToOneRow_function) (CopyToState cstate, TupleTableSlot *slot);
> +typedef void (*CopyToEnd_function) (CopyToState cstate);
> 
> We don't really need a set of typedefs here, let's put the definitions
> in the CopyToRoutine struct instead.

OK. I'll do it.

> +extern CopyToRoutine CopyToRoutineText;
> +extern CopyToRoutine CopyToRoutineCSV;
> +extern CopyToRoutine CopyToRoutineBinary;
> 
> All that should IMO remain in copyto.c and copyfrom.c in the initial
> patch doing the refactoring.  Why not using a fetch function instead
> that uses a string in input?  Then you can call that once after
> parsing the List of options in ProcessCopyOptions().

OK. How about the following for the fetch function
signature?

extern CopyToRoutine *GetBuiltinCopyToRoutine(const char *format);

We may introduce an enum and use it:

typedef enum CopyBuiltinFormat
{
	COPY_BUILTIN_FORMAT_TEXT = 0,
	COPY_BUILTIN_FORMAT_CSV,
	COPY_BUILTIN_FORMAT_BINARY,
} CopyBuiltinFormat;

extern CopyToRoutine *GetBuiltinCopyToRoutine(CopyBuiltinFormat format);

> +/* All "text" and "csv" options are parsed in ProcessCopyOptions(). We may
> + * move the code to here later. */
> Some areas, like this comment, are written in an incorrect format.

Oh, sorry. I assumed that the comment style was adjusted by
pgindent.

I'll use the following style:

/*
 * ...
 */

> +    getTypeBinaryOutputInfo(attr->atttypid, &out_func_oid, &isvarlena);
> +    fmgr_info(out_func_oid, &cstate->out_functions[attnum - 1]);
> 
> Actually, this split is interesting.  It is possible for a custom
> format to plug in a custom set of out functions.  Did you make use of
> something custom for your own stuff?

I didn't. My PoC custom COPY format handler for Apache Arrow
just handles integer and text for now. It doesn't use
cstate->out_functions because cstate->out_functions may not
return a valid binary format value for Apache Arrow. So it
formats each value by itself.

I'll chose one of them for a custom type (that isn't
supported by Apache Arrow, e.g. PostGIS types):

1. Report an unsupported error
2. Call output function for Apache Arrow provided by the
   custom type

>                                       Actually, could it make sense to
> split the assignment of cstate->out_functions into its own callback?

Yes. Because we need to use getTypeBinaryOutputInfo() for
"binary" and use getTypeOutputInfo() for "text" and "csv".

> Sure, that's part of the start phase, but at least it would make clear
> that a custom method *has* to assign these OIDs to work.  The patch
> implies that as a rule, without a comment that CopyToStart *must* set
> up these OIDs.

CopyToStart doesn't need to set up them if the handler
doesn't use cstate->out_functions.

> I think that 0001 and 0005 should be handled first, as pieces
> independent of the rest.  Then we could move on with 0002~0004 and
> 0006~0008.

OK. I'll focus on 0001 and 0005 for now. I'll restart
0002-0004/0006-0008 after 0001 and 0005 are accepted.


Thanks,
-- 
kou

  75. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-01-25T08:52:55Z

    Hi,

In <CAD21AoALxEZz33NpcSk99ad_DT3A2oFNMa2KNjGBCMVFeCiUaA@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Thu, 25 Jan 2024 13:36:03 +0900,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

>                 I've experimented with a similar optimization for csv
> and text format; have different callbacks for text and csv format and
> remove "if (cstate->opts.csv_mode)" branches. I've attached a patch
> for that. Here are results:
> 
> HEAD w/ 0001 patch + remove branches:
> binary 2824.502 ms
> text 2715.264 ms
> csv 2803.381 ms
> 
> The numbers look better now. I'm not sure these are within a noise
> range but it might be worth considering having different callbacks for
> text and csv formats.

Wow! Interesting. I tried the approach before but I didn't
see any difference by the approach. But it may depend on my
environment.

I'll import the approach to the next patch set so that
others can try the approach easily.


Thanks,
-- 
kou

  76. Re: Make COPY format extendable: Extract COPY TO format implementations

    Michael Paquier <michael@paquier.xyz> — 2024-01-25T23:35:19Z

    On Thu, Jan 25, 2024 at 05:45:43PM +0900, Sutou Kouhei wrote:
> In <ZbHS439y-Bs6HIAR@paquier.xyz>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Thu, 25 Jan 2024 12:17:55 +0900,
>   Michael Paquier <michael@paquier.xyz> wrote:
>> +extern CopyToRoutine CopyToRoutineText;
>> +extern CopyToRoutine CopyToRoutineCSV;
>> +extern CopyToRoutine CopyToRoutineBinary;
>> 
>> All that should IMO remain in copyto.c and copyfrom.c in the initial
>> patch doing the refactoring.  Why not using a fetch function instead
>> that uses a string in input?  Then you can call that once after
>> parsing the List of options in ProcessCopyOptions().
> 
> OK. How about the following for the fetch function
> signature?
> 
> extern CopyToRoutine *GetBuiltinCopyToRoutine(const char *format);

Or CopyToRoutineGet()?  I am not wedded to my suggestion, got a bad
history with naming things around here.

> We may introduce an enum and use it:
> 
> typedef enum CopyBuiltinFormat
> {
> 	COPY_BUILTIN_FORMAT_TEXT = 0,
> 	COPY_BUILTIN_FORMAT_CSV,
> 	COPY_BUILTIN_FORMAT_BINARY,
> } CopyBuiltinFormat;
> 
> extern CopyToRoutine *GetBuiltinCopyToRoutine(CopyBuiltinFormat format);

I am not sure that this is necessary as the option value is a string.

> Oh, sorry. I assumed that the comment style was adjusted by
> pgindent.

No worries, that's just something we get used to.  I tend to fix a lot
of these things by myself when editing patches.

>> +    getTypeBinaryOutputInfo(attr->atttypid, &out_func_oid, &isvarlena);
>> +    fmgr_info(out_func_oid, &cstate->out_functions[attnum - 1]);
>> 
>> Actually, this split is interesting.  It is possible for a custom
>> format to plug in a custom set of out functions.  Did you make use of
>> something custom for your own stuff?
> 
> I didn't. My PoC custom COPY format handler for Apache Arrow
> just handles integer and text for now. It doesn't use
> cstate->out_functions because cstate->out_functions may not
> return a valid binary format value for Apache Arrow. So it
> formats each value by itself.

I mean, if you use a custom output function, you could tweak things
even more with byteas or such..  If a callback is expected to do
something, like setting the output function OIDs in the start
callback, we'd better document it rather than letting that be implied.

>>                                       Actually, could it make sense to
>> split the assignment of cstate->out_functions into its own callback?
> 
> Yes. Because we need to use getTypeBinaryOutputInfo() for
> "binary" and use getTypeOutputInfo() for "text" and "csv".

Okay.  After sleeping on it, a split makes sense here, because it also
reduces the presence of TupleDesc in the start callback.

>> Sure, that's part of the start phase, but at least it would make clear
>> that a custom method *has* to assign these OIDs to work.  The patch
>> implies that as a rule, without a comment that CopyToStart *must* set
>> up these OIDs.
> 
> CopyToStart doesn't need to set up them if the handler
> doesn't use cstate->out_functions.

Noted.

>> I think that 0001 and 0005 should be handled first, as pieces
>> independent of the rest.  Then we could move on with 0002~0004 and
>> 0006~0008.
> 
> OK. I'll focus on 0001 and 0005 for now. I'll restart
> 0002-0004/0006-0008 after 0001 and 0005 are accepted.

Once you get these, I'd be interested in re-doing an evaluation of
COPY TO and more tests with COPY FROM while running Postgres on
scissors.  One thing I was thinking to use here is my blackhole_am for
COPY FROM:
https://github.com/michaelpq/pg_plugins/tree/main/blackhole_am

As per its name, it does nothing on INSERT, so you could create a
table using it as access method, and stress the COPY FROM execution
paths without having to mount Postgres on a tmpfs because the data is
sent to the void.  Perhaps it does not matter, but that moves the
tests to the bottlenecks we want to stress (aka the per-row callback
for large data sets).

I've switched the patch as waiting on author for now.  Thanks for your
perseverance here.  I understand that's not easy to follow up with
patches and reviews (^_^;)
--
Michael

  77. Re: Make COPY format extendable: Extract COPY TO format implementations

    Junwang Zhao <zhjwpku@gmail.com> — 2024-01-26T08:18:14Z

    On Thu, Jan 25, 2024 at 4:52 PM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <CAD21AoALxEZz33NpcSk99ad_DT3A2oFNMa2KNjGBCMVFeCiUaA@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Thu, 25 Jan 2024 13:36:03 +0900,
>   Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> >                 I've experimented with a similar optimization for csv
> > and text format; have different callbacks for text and csv format and
> > remove "if (cstate->opts.csv_mode)" branches. I've attached a patch
> > for that. Here are results:
> >
> > HEAD w/ 0001 patch + remove branches:
> > binary 2824.502 ms
> > text 2715.264 ms
> > csv 2803.381 ms
> >
> > The numbers look better now. I'm not sure these are within a noise
> > range but it might be worth considering having different callbacks for
> > text and csv formats.
>
> Wow! Interesting. I tried the approach before but I didn't
> see any difference by the approach. But it may depend on my
> environment.
>
> I'll import the approach to the next patch set so that
> others can try the approach easily.
>
>
> Thanks,
> --
> kou

Hi Kou-san,

In the current implementation, there is no way that one can check
incompatibility
options in ProcessCopyOptions, we can postpone the check in CopyFromStart
or CopyToStart, but I think it is a little bit late. Do you think
adding an extra
check for incompatible options hook is acceptable (PFA)?


-- 
Regards
Junwang Zhao

  78. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-01-26T08:32:46Z

    Hi,

In <CAEG8a3+-oG63GeG6v0L8EWi_8Fhuj9vJBhOteLxuBZwtun3GVA@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 26 Jan 2024 16:18:14 +0800,
  Junwang Zhao <zhjwpku@gmail.com> wrote:

> In the current implementation, there is no way that one can check
> incompatibility
> options in ProcessCopyOptions, we can postpone the check in CopyFromStart
> or CopyToStart, but I think it is a little bit late. Do you think
> adding an extra
> check for incompatible options hook is acceptable (PFA)?

Thanks for the suggestion! But I think that a custom handler
can do it in
CopyToProcessOption()/CopyFromProcessOption(). What do you
think about this? Or could you share a sample COPY TO/FROM
WITH() SQL you think?


Thanks,
-- 
kou

  79. Re: Make COPY format extendable: Extract COPY TO format implementations

    Junwang Zhao <zhjwpku@gmail.com> — 2024-01-26T08:41:50Z

    On Fri, Jan 26, 2024 at 4:32 PM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <CAEG8a3+-oG63GeG6v0L8EWi_8Fhuj9vJBhOteLxuBZwtun3GVA@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 26 Jan 2024 16:18:14 +0800,
>   Junwang Zhao <zhjwpku@gmail.com> wrote:
>
> > In the current implementation, there is no way that one can check
> > incompatibility
> > options in ProcessCopyOptions, we can postpone the check in CopyFromStart
> > or CopyToStart, but I think it is a little bit late. Do you think
> > adding an extra
> > check for incompatible options hook is acceptable (PFA)?
>
> Thanks for the suggestion! But I think that a custom handler
> can do it in
> CopyToProcessOption()/CopyFromProcessOption(). What do you
> think about this? Or could you share a sample COPY TO/FROM
> WITH() SQL you think?

CopyToProcessOption()/CopyFromProcessOption() can only handle
single option, and store the options in the opaque field,  but it can not
check the relation of two options, for example, considering json format,
the `header` option can not be handled by these two functions.

I want to find a way when the user specifies the header option, customer
handler can error out.

>
>
> Thanks,
> --
> kou



-- 
Regards
Junwang Zhao

  80. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-01-26T08:49:47Z

    Hi,

In <ZbLwNyOKbddno0Ue@paquier.xyz>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 26 Jan 2024 08:35:19 +0900,
  Michael Paquier <michael@paquier.xyz> wrote:

>> OK. How about the following for the fetch function
>> signature?
>> 
>> extern CopyToRoutine *GetBuiltinCopyToRoutine(const char *format);
> 
> Or CopyToRoutineGet()?  I am not wedded to my suggestion, got a bad
> history with naming things around here.

Thanks for the suggestion.
I rethink about this and use the following:

+extern void ProcessCopyOptionFormatTo(ParseState *pstate, CopyFormatOptions *opts_out, DefElem *defel);

It's not a fetch function. It sets CopyToRoutine opts_out
instead. But it hides CopyToRoutine* to copyto.c. Is it
acceptable?

>> OK. I'll focus on 0001 and 0005 for now. I'll restart
>> 0002-0004/0006-0008 after 0001 and 0005 are accepted.
> 
> Once you get these, I'd be interested in re-doing an evaluation of
> COPY TO and more tests with COPY FROM while running Postgres on
> scissors.  One thing I was thinking to use here is my blackhole_am for
> COPY FROM:
> https://github.com/michaelpq/pg_plugins/tree/main/blackhole_am

Thanks!

Could you evaluate the attached patch set with COPY FROM?

I attach v7 patch set. It includes only the 0001 and 0005
parts in v6 patch set because we focus on them for now.

0001: This is based on 0001 in v6.

Changes since v6:

* Fix comment style
* Hide CopyToRoutine{Text,CSV,Binary}
* Add more comments
* Eliminate "if (cstate->opts.csv_mode)" branches from "text"
  and "csv" callbacks
* Remove CopyTo*_function typedefs
* Update benchmark results in commit message but the results
  are measured on my environment that isn't suitable for
  accurate benchmark

0002: This is based on 0005 in v6.

Changes since v6:

* Fix comment style
* Hide CopyFromRoutine{Text,CSV,Binary}
* Add more comments
* Eliminate a "if (cstate->opts.csv_mode)" branch from "text"
  and "csv" callbacks
  * NOTE: We can eliminate more "if (cstate->opts.csv_mode)" branches
    such as one in NextCopyFromRawFields(). Should we do it
    in this feature improvement (make COPY format
    extendable)? Can we defer this as a separated improvement?
* Remove CopyFrom*_function typedefs



Thanks,
-- 
kou

  81. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-01-26T08:55:11Z

    Hi,

In <CAEG8a3KhS6s1XQgDSvc8vFTb4GkhBmS8TxOoVSDPFX+MPExxxQ@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 26 Jan 2024 16:41:50 +0800,
  Junwang Zhao <zhjwpku@gmail.com> wrote:

> CopyToProcessOption()/CopyFromProcessOption() can only handle
> single option, and store the options in the opaque field,  but it can not
> check the relation of two options, for example, considering json format,
> the `header` option can not be handled by these two functions.
> 
> I want to find a way when the user specifies the header option, customer
> handler can error out.

Ah, you want to use a built-in option (such as "header")
value from a custom handler, right? Hmm, it may be better
that we call CopyToProcessOption()/CopyFromProcessOption()
for all options including built-in options.


Thanks,
-- 
kou

  82. Re: Make COPY format extendable: Extract COPY TO format implementations

    Junwang Zhao <zhjwpku@gmail.com> — 2024-01-26T09:02:23Z

    On Fri, Jan 26, 2024 at 4:55 PM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <CAEG8a3KhS6s1XQgDSvc8vFTb4GkhBmS8TxOoVSDPFX+MPExxxQ@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 26 Jan 2024 16:41:50 +0800,
>   Junwang Zhao <zhjwpku@gmail.com> wrote:
>
> > CopyToProcessOption()/CopyFromProcessOption() can only handle
> > single option, and store the options in the opaque field,  but it can not
> > check the relation of two options, for example, considering json format,
> > the `header` option can not be handled by these two functions.
> >
> > I want to find a way when the user specifies the header option, customer
> > handler can error out.
>
> Ah, you want to use a built-in option (such as "header")
> value from a custom handler, right? Hmm, it may be better
> that we call CopyToProcessOption()/CopyFromProcessOption()
> for all options including built-in options.
>
Hmm, still I don't think it can handle all cases, since we don't know
the sequence of the options, we need all the options been parsed
before we check the compatibility of the options, or customer
handlers will need complicated logic to resolve that, which might
lead to ugly code :(

>
> Thanks,
> --
> kou



-- 
Regards
Junwang Zhao

  83. Re: Make COPY format extendable: Extract COPY TO format implementations

    Junwang Zhao <zhjwpku@gmail.com> — 2024-01-27T06:15:02Z

    Hi Kou-san,

On Fri, Jan 26, 2024 at 5:02 PM Junwang Zhao <zhjwpku@gmail.com> wrote:
>
> On Fri, Jan 26, 2024 at 4:55 PM Sutou Kouhei <kou@clear-code.com> wrote:
> >
> > Hi,
> >
> > In <CAEG8a3KhS6s1XQgDSvc8vFTb4GkhBmS8TxOoVSDPFX+MPExxxQ@mail.gmail.com>
> >   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 26 Jan 2024 16:41:50 +0800,
> >   Junwang Zhao <zhjwpku@gmail.com> wrote:
> >
> > > CopyToProcessOption()/CopyFromProcessOption() can only handle
> > > single option, and store the options in the opaque field,  but it can not
> > > check the relation of two options, for example, considering json format,
> > > the `header` option can not be handled by these two functions.
> > >
> > > I want to find a way when the user specifies the header option, customer
> > > handler can error out.
> >
> > Ah, you want to use a built-in option (such as "header")
> > value from a custom handler, right? Hmm, it may be better
> > that we call CopyToProcessOption()/CopyFromProcessOption()
> > for all options including built-in options.
> >
> Hmm, still I don't think it can handle all cases, since we don't know
> the sequence of the options, we need all the options been parsed
> before we check the compatibility of the options, or customer
> handlers will need complicated logic to resolve that, which might
> lead to ugly code :(
>

I have been working on a *COPY TO JSON* extension since yesterday,
which is based on your V6 patch set, I'd like to give you more input
so you can make better decisions about the implementation(with only
pg-copy-arrow you might not get everything considered).

V8 is based on V6, so anybody involved in the performance issue
should still review the V7 patch set.

0001-0008 is your original V6 implementations

0009 is some changes made by me, I changed CopyToGetFormat to
CopyToSendCopyBegin because pg_copy_json need to send different bytes
in SendCopyBegin, get the format code along is not enough, I once had
a thought that may be we should merge SendCopyBegin/SendCopyEnd into
CopyToStart/CopyToEnd but I don't do that in this patch. I have also
exported more APIs for extension usage.

00010 is the pg_copy_json extension, I think this should be a good
case which can utilize the *extendable copy format* feature, maybe we
should delete copy_test_format if we have this extension as an
example?

> >
> > Thanks,
> > --
> > kou
>
>
>
> --
> Regards
> Junwang Zhao



-- 
Regards
Junwang Zhao

  84. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2024-01-29T02:41:59Z

    On Fri, Jan 26, 2024 at 6:02 PM Junwang Zhao <zhjwpku@gmail.com> wrote:
>
> On Fri, Jan 26, 2024 at 4:55 PM Sutou Kouhei <kou@clear-code.com> wrote:
> >
> > Hi,
> >
> > In <CAEG8a3KhS6s1XQgDSvc8vFTb4GkhBmS8TxOoVSDPFX+MPExxxQ@mail.gmail.com>
> >   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 26 Jan 2024 16:41:50 +0800,
> >   Junwang Zhao <zhjwpku@gmail.com> wrote:
> >
> > > CopyToProcessOption()/CopyFromProcessOption() can only handle
> > > single option, and store the options in the opaque field,  but it can not
> > > check the relation of two options, for example, considering json format,
> > > the `header` option can not be handled by these two functions.
> > >
> > > I want to find a way when the user specifies the header option, customer
> > > handler can error out.
> >
> > Ah, you want to use a built-in option (such as "header")
> > value from a custom handler, right? Hmm, it may be better
> > that we call CopyToProcessOption()/CopyFromProcessOption()
> > for all options including built-in options.
> >
> Hmm, still I don't think it can handle all cases, since we don't know
> the sequence of the options, we need all the options been parsed
> before we check the compatibility of the options, or customer
> handlers will need complicated logic to resolve that, which might
> lead to ugly code :(
>

Does it make sense to pass only non-builtin options to the custom
format callback after parsing and evaluating the builtin options? That
is, we parse and evaluate only the builtin options and populate
opts_out first, then pass each rest option to the custom format
handler callback. The callback can refer to the builtin option values.
The callback is expected to return false if the passed option is not
supported. If one of the builtin formats is specified and the rest
options list has at least one option, we raise "option %s not
recognized" error.  IOW it's the core's responsibility to ranse the
"option %s not recognized" error, which is in order to raise a
consistent error message. Also, I think the core should check the
redundant options including bultiin and custom options.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  85. Re: Make COPY format extendable: Extract COPY TO format implementations

    Junwang Zhao <zhjwpku@gmail.com> — 2024-01-29T03:10:45Z

    On Mon, Jan 29, 2024 at 10:42 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Fri, Jan 26, 2024 at 6:02 PM Junwang Zhao <zhjwpku@gmail.com> wrote:
> >
> > On Fri, Jan 26, 2024 at 4:55 PM Sutou Kouhei <kou@clear-code.com> wrote:
> > >
> > > Hi,
> > >
> > > In <CAEG8a3KhS6s1XQgDSvc8vFTb4GkhBmS8TxOoVSDPFX+MPExxxQ@mail.gmail.com>
> > >   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 26 Jan 2024 16:41:50 +0800,
> > >   Junwang Zhao <zhjwpku@gmail.com> wrote:
> > >
> > > > CopyToProcessOption()/CopyFromProcessOption() can only handle
> > > > single option, and store the options in the opaque field,  but it can not
> > > > check the relation of two options, for example, considering json format,
> > > > the `header` option can not be handled by these two functions.
> > > >
> > > > I want to find a way when the user specifies the header option, customer
> > > > handler can error out.
> > >
> > > Ah, you want to use a built-in option (such as "header")
> > > value from a custom handler, right? Hmm, it may be better
> > > that we call CopyToProcessOption()/CopyFromProcessOption()
> > > for all options including built-in options.
> > >
> > Hmm, still I don't think it can handle all cases, since we don't know
> > the sequence of the options, we need all the options been parsed
> > before we check the compatibility of the options, or customer
> > handlers will need complicated logic to resolve that, which might
> > lead to ugly code :(
> >
>
> Does it make sense to pass only non-builtin options to the custom
> format callback after parsing and evaluating the builtin options? That
> is, we parse and evaluate only the builtin options and populate
> opts_out first, then pass each rest option to the custom format
> handler callback. The callback can refer to the builtin option values.

Yeah, I think this makes sense.

> The callback is expected to return false if the passed option is not
> supported. If one of the builtin formats is specified and the rest
> options list has at least one option, we raise "option %s not
> recognized" error.  IOW it's the core's responsibility to ranse the
> "option %s not recognized" error, which is in order to raise a
> consistent error message. Also, I think the core should check the
> redundant options including bultiin and custom options.

It would be good that core could check all the redundant options,
but where should core do the book-keeping of all the options? I have
no idea about this, in my implementation of pg_copy_json extension,
I handle redundant options by adding a xxx_specified field for each
xxx.

>
> Regards,
>
> --
> Masahiko Sawada
> Amazon Web Services: https://aws.amazon.com



-- 
Regards
Junwang Zhao

  86. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2024-01-29T03:21:48Z

    On Mon, Jan 29, 2024 at 12:10 PM Junwang Zhao <zhjwpku@gmail.com> wrote:
>
> On Mon, Jan 29, 2024 at 10:42 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > On Fri, Jan 26, 2024 at 6:02 PM Junwang Zhao <zhjwpku@gmail.com> wrote:
> > >
> > > On Fri, Jan 26, 2024 at 4:55 PM Sutou Kouhei <kou@clear-code.com> wrote:
> > > >
> > > > Hi,
> > > >
> > > > In <CAEG8a3KhS6s1XQgDSvc8vFTb4GkhBmS8TxOoVSDPFX+MPExxxQ@mail.gmail.com>
> > > >   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 26 Jan 2024 16:41:50 +0800,
> > > >   Junwang Zhao <zhjwpku@gmail.com> wrote:
> > > >
> > > > > CopyToProcessOption()/CopyFromProcessOption() can only handle
> > > > > single option, and store the options in the opaque field,  but it can not
> > > > > check the relation of two options, for example, considering json format,
> > > > > the `header` option can not be handled by these two functions.
> > > > >
> > > > > I want to find a way when the user specifies the header option, customer
> > > > > handler can error out.
> > > >
> > > > Ah, you want to use a built-in option (such as "header")
> > > > value from a custom handler, right? Hmm, it may be better
> > > > that we call CopyToProcessOption()/CopyFromProcessOption()
> > > > for all options including built-in options.
> > > >
> > > Hmm, still I don't think it can handle all cases, since we don't know
> > > the sequence of the options, we need all the options been parsed
> > > before we check the compatibility of the options, or customer
> > > handlers will need complicated logic to resolve that, which might
> > > lead to ugly code :(
> > >
> >
> > Does it make sense to pass only non-builtin options to the custom
> > format callback after parsing and evaluating the builtin options? That
> > is, we parse and evaluate only the builtin options and populate
> > opts_out first, then pass each rest option to the custom format
> > handler callback. The callback can refer to the builtin option values.
>
> Yeah, I think this makes sense.
>
> > The callback is expected to return false if the passed option is not
> > supported. If one of the builtin formats is specified and the rest
> > options list has at least one option, we raise "option %s not
> > recognized" error.  IOW it's the core's responsibility to ranse the
> > "option %s not recognized" error, which is in order to raise a
> > consistent error message. Also, I think the core should check the
> > redundant options including bultiin and custom options.
>
> It would be good that core could check all the redundant options,
> but where should core do the book-keeping of all the options? I have
> no idea about this, in my implementation of pg_copy_json extension,
> I handle redundant options by adding a xxx_specified field for each
> xxx.

What I imagined is that while parsing the all specified options, we
evaluate builtin options and we add non-builtin options to another
list. Then when parsing a non-builtin option, we check if this option
already exists in the list. If there is, we raise the "option %s not
recognized" error.". Once we complete checking all options, we pass
each option in the list to the callback.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  87. Re: Make COPY format extendable: Extract COPY TO format implementations

    Junwang Zhao <zhjwpku@gmail.com> — 2024-01-29T03:37:07Z

    On Mon, Jan 29, 2024 at 11:22 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Mon, Jan 29, 2024 at 12:10 PM Junwang Zhao <zhjwpku@gmail.com> wrote:
> >
> > On Mon, Jan 29, 2024 at 10:42 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > >
> > > On Fri, Jan 26, 2024 at 6:02 PM Junwang Zhao <zhjwpku@gmail.com> wrote:
> > > >
> > > > On Fri, Jan 26, 2024 at 4:55 PM Sutou Kouhei <kou@clear-code.com> wrote:
> > > > >
> > > > > Hi,
> > > > >
> > > > > In <CAEG8a3KhS6s1XQgDSvc8vFTb4GkhBmS8TxOoVSDPFX+MPExxxQ@mail.gmail.com>
> > > > >   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 26 Jan 2024 16:41:50 +0800,
> > > > >   Junwang Zhao <zhjwpku@gmail.com> wrote:
> > > > >
> > > > > > CopyToProcessOption()/CopyFromProcessOption() can only handle
> > > > > > single option, and store the options in the opaque field,  but it can not
> > > > > > check the relation of two options, for example, considering json format,
> > > > > > the `header` option can not be handled by these two functions.
> > > > > >
> > > > > > I want to find a way when the user specifies the header option, customer
> > > > > > handler can error out.
> > > > >
> > > > > Ah, you want to use a built-in option (such as "header")
> > > > > value from a custom handler, right? Hmm, it may be better
> > > > > that we call CopyToProcessOption()/CopyFromProcessOption()
> > > > > for all options including built-in options.
> > > > >
> > > > Hmm, still I don't think it can handle all cases, since we don't know
> > > > the sequence of the options, we need all the options been parsed
> > > > before we check the compatibility of the options, or customer
> > > > handlers will need complicated logic to resolve that, which might
> > > > lead to ugly code :(
> > > >
> > >
> > > Does it make sense to pass only non-builtin options to the custom
> > > format callback after parsing and evaluating the builtin options? That
> > > is, we parse and evaluate only the builtin options and populate
> > > opts_out first, then pass each rest option to the custom format
> > > handler callback. The callback can refer to the builtin option values.
> >
> > Yeah, I think this makes sense.
> >
> > > The callback is expected to return false if the passed option is not
> > > supported. If one of the builtin formats is specified and the rest
> > > options list has at least one option, we raise "option %s not
> > > recognized" error.  IOW it's the core's responsibility to ranse the
> > > "option %s not recognized" error, which is in order to raise a
> > > consistent error message. Also, I think the core should check the
> > > redundant options including bultiin and custom options.
> >
> > It would be good that core could check all the redundant options,
> > but where should core do the book-keeping of all the options? I have
> > no idea about this, in my implementation of pg_copy_json extension,
> > I handle redundant options by adding a xxx_specified field for each
> > xxx.
>
> What I imagined is that while parsing the all specified options, we
> evaluate builtin options and we add non-builtin options to another
> list. Then when parsing a non-builtin option, we check if this option
> already exists in the list. If there is, we raise the "option %s not
> recognized" error.". Once we complete checking all options, we pass
> each option in the list to the callback.

LGTM.

>
> Regards,
>
> --
> Masahiko Sawada
> Amazon Web Services: https://aws.amazon.com



-- 
Regards
Junwang Zhao

  88. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-01-29T06:03:32Z

    Hi,

In <CAEG8a3JDPks7XU5-NvzjzuKQYQqR8pDfS7CDGZonQTXfdWtnnw@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Sat, 27 Jan 2024 14:15:02 +0800,
  Junwang Zhao <zhjwpku@gmail.com> wrote:

> I have been working on a *COPY TO JSON* extension since yesterday,
> which is based on your V6 patch set, I'd like to give you more input
> so you can make better decisions about the implementation(with only
> pg-copy-arrow you might not get everything considered).

Thanks!

> 0009 is some changes made by me, I changed CopyToGetFormat to
> CopyToSendCopyBegin because pg_copy_json need to send different bytes
> in SendCopyBegin, get the format code along is not enough

Oh, I haven't cared about the case.
How about the following API instead?

static void
SendCopyBegin(CopyToState cstate)
{
	StringInfoData buf;

	pq_beginmessage(&buf, PqMsg_CopyOutResponse);
	cstate->opts.to_routine->CopyToFillCopyOutResponse(cstate, &buf);
	pq_endmessage(&buf);
	cstate->copy_dest = COPY_FRONTEND;
}

static void
CopyToJsonFillCopyOutResponse(CopyToState cstate, StringInfoData &buf)
{
	int16		format = 0;

	pq_sendbyte(&buf, format);      /* overall format */
	/*
	 * JSON mode is always one non-binary column
	 */
	pq_sendint16(&buf, 1);
	pq_sendint16(&buf, format);
}

> 00010 is the pg_copy_json extension, I think this should be a good
> case which can utilize the *extendable copy format* feature

It seems that it's convenient that we have one more callback
for initializing CopyToState::opaque. It's called only once
when Copy{To,From}Routine is chosen:

typedef struct CopyToRoutine
{
	void		(*CopyToInit) (CopyToState cstate);
...
};

void
ProcessCopyOptions(ParseState *pstate,
				   CopyFormatOptions *opts_out,
				   bool is_from,
				   void *cstate,
				   List *options)
{
...
	foreach(option, options)
	{
		DefElem    *defel = lfirst_node(DefElem, option);

		if (strcmp(defel->defname, "format") == 0)
		{
			...
			opts_out->to_routine = &CopyToRoutineXXX;
			opts_out->to_routine->CopyToInit(cstate);
			...
		}
	}
...
}


>                                                              maybe we
> should delete copy_test_format if we have this extension as an
> example?

I haven't read the COPY TO format json thread[1] carefully
(sorry), but we may add the JSON format as a built-in
format. If we do it, copy_test_format is useful to test the
extension API.

[1] https://www.postgresql.org/message-id/flat/CALvfUkBxTYy5uWPFVwpk_7ii2zgT07t3d-yR_cy4sfrrLU%3Dkcg%40mail.gmail.com


Thanks,
-- 
kou

  89. Re: Make COPY format extendable: Extract COPY TO format implementations

    Junwang Zhao <zhjwpku@gmail.com> — 2024-01-29T06:48:40Z

    On Mon, Jan 29, 2024 at 2:03 PM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <CAEG8a3JDPks7XU5-NvzjzuKQYQqR8pDfS7CDGZonQTXfdWtnnw@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Sat, 27 Jan 2024 14:15:02 +0800,
>   Junwang Zhao <zhjwpku@gmail.com> wrote:
>
> > I have been working on a *COPY TO JSON* extension since yesterday,
> > which is based on your V6 patch set, I'd like to give you more input
> > so you can make better decisions about the implementation(with only
> > pg-copy-arrow you might not get everything considered).
>
> Thanks!
>
> > 0009 is some changes made by me, I changed CopyToGetFormat to
> > CopyToSendCopyBegin because pg_copy_json need to send different bytes
> > in SendCopyBegin, get the format code along is not enough
>
> Oh, I haven't cared about the case.
> How about the following API instead?
>
> static void
> SendCopyBegin(CopyToState cstate)
> {
>         StringInfoData buf;
>
>         pq_beginmessage(&buf, PqMsg_CopyOutResponse);
>         cstate->opts.to_routine->CopyToFillCopyOutResponse(cstate, &buf);
>         pq_endmessage(&buf);
>         cstate->copy_dest = COPY_FRONTEND;
> }
>
> static void
> CopyToJsonFillCopyOutResponse(CopyToState cstate, StringInfoData &buf)
> {
>         int16           format = 0;
>
>         pq_sendbyte(&buf, format);      /* overall format */
>         /*
>          * JSON mode is always one non-binary column
>          */
>         pq_sendint16(&buf, 1);
>         pq_sendint16(&buf, format);
> }

Make sense to me.

>
> > 00010 is the pg_copy_json extension, I think this should be a good
> > case which can utilize the *extendable copy format* feature
>
> It seems that it's convenient that we have one more callback
> for initializing CopyToState::opaque. It's called only once
> when Copy{To,From}Routine is chosen:
>
> typedef struct CopyToRoutine
> {
>         void            (*CopyToInit) (CopyToState cstate);
> ...
> };

I like this, we can alloc private data in this hook.

>
> void
> ProcessCopyOptions(ParseState *pstate,
>                                    CopyFormatOptions *opts_out,
>                                    bool is_from,
>                                    void *cstate,
>                                    List *options)
> {
> ...
>         foreach(option, options)
>         {
>                 DefElem    *defel = lfirst_node(DefElem, option);
>
>                 if (strcmp(defel->defname, "format") == 0)
>                 {
>                         ...
>                         opts_out->to_routine = &CopyToRoutineXXX;
>                         opts_out->to_routine->CopyToInit(cstate);
>                         ...
>                 }
>         }
> ...
> }
>
>
> >                                                              maybe we
> > should delete copy_test_format if we have this extension as an
> > example?
>
> I haven't read the COPY TO format json thread[1] carefully
> (sorry), but we may add the JSON format as a built-in
> format. If we do it, copy_test_format is useful to test the
> extension API.

Yeah, maybe, I have no strong opinion here, pg_copy_json is
just a toy extension for discussion.

>
> [1] https://www.postgresql.org/message-id/flat/CALvfUkBxTYy5uWPFVwpk_7ii2zgT07t3d-yR_cy4sfrrLU%3Dkcg%40mail.gmail.com
>
>
> Thanks,
> --
> kou



-- 
Regards
Junwang Zhao

  90. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-01-29T09:45:23Z

    Hi,

In <CAEG8a3Jnmbjw82OiSvRK3v9XN2zSshsB8ew1mZCQDAkKq6r9YQ@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Mon, 29 Jan 2024 11:37:07 +0800,
  Junwang Zhao <zhjwpku@gmail.com> wrote:

>> > > Does it make sense to pass only non-builtin options to the custom
>> > > format callback after parsing and evaluating the builtin options? That
>> > > is, we parse and evaluate only the builtin options and populate
>> > > opts_out first, then pass each rest option to the custom format
>> > > handler callback. The callback can refer to the builtin option values.
>>
>> What I imagined is that while parsing the all specified options, we
>> evaluate builtin options and we add non-builtin options to another
>> list. Then when parsing a non-builtin option, we check if this option
>> already exists in the list. If there is, we raise the "option %s not
>> recognized" error.". Once we complete checking all options, we pass
>> each option in the list to the callback.

I implemented this idea and the following ideas:

1. Add init callback for initialization
2. Change GetFormat() to FillCopyXXXResponse()
   because JSON format always use 1 column
3. FROM only: Eliminate more cstate->opts.csv_mode branches
   (This is for performance.)

See the attached v9 patch set for details. Changes since v7:

0001:

* Move CopyToProcessOption() calls to the end of
  ProcessCopyOptions() for easy to option validation
* Add CopyToState::CopyToInit() and call it in
  ProcessCopyOptionFormatTo()
* Change CopyToState::CopyToGetFormat() to
  CopyToState::CopyToFillCopyOutResponse() and use it in
  SendCopyBegin()

0002:

* Move CopyFromProcessOption() calls to the end of
  ProcessCopyOptions() for easy to option validation
* Add CopyFromState::CopyFromInit() and call it in
  ProcessCopyOptionFormatFrom()
* Change CopyFromState::CopyFromGetFormat() to
  CopyFromState::CopyFromFillCopyOutResponse() and use it in
  ReceiveCopyBegin()
* Rename NextCopyFromRawFields() to
  NextCopyFromRawFieldsInternal() and pass the read
  attributes callback explicitly to eliminate more
  cstate->opts.csv_mode branches


Thanks,
-- 
kou

  91. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2024-01-30T02:11:59Z

    On Mon, Jan 29, 2024 at 6:45 PM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <CAEG8a3Jnmbjw82OiSvRK3v9XN2zSshsB8ew1mZCQDAkKq6r9YQ@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Mon, 29 Jan 2024 11:37:07 +0800,
>   Junwang Zhao <zhjwpku@gmail.com> wrote:
>
> >> > > Does it make sense to pass only non-builtin options to the custom
> >> > > format callback after parsing and evaluating the builtin options? That
> >> > > is, we parse and evaluate only the builtin options and populate
> >> > > opts_out first, then pass each rest option to the custom format
> >> > > handler callback. The callback can refer to the builtin option values.
> >>
> >> What I imagined is that while parsing the all specified options, we
> >> evaluate builtin options and we add non-builtin options to another
> >> list. Then when parsing a non-builtin option, we check if this option
> >> already exists in the list. If there is, we raise the "option %s not
> >> recognized" error.". Once we complete checking all options, we pass
> >> each option in the list to the callback.
>
> I implemented this idea and the following ideas:
>
> 1. Add init callback for initialization
> 2. Change GetFormat() to FillCopyXXXResponse()
>    because JSON format always use 1 column
> 3. FROM only: Eliminate more cstate->opts.csv_mode branches
>    (This is for performance.)
>
> See the attached v9 patch set for details. Changes since v7:
>
> 0001:
>
> * Move CopyToProcessOption() calls to the end of
>   ProcessCopyOptions() for easy to option validation
> * Add CopyToState::CopyToInit() and call it in
>   ProcessCopyOptionFormatTo()
> * Change CopyToState::CopyToGetFormat() to
>   CopyToState::CopyToFillCopyOutResponse() and use it in
>   SendCopyBegin()

Thank you for updating the patch! Here are comments on 0001 patch:

---
+        if (!format_specified)
+                /* Set the default format. */
+                ProcessCopyOptionFormatTo(pstate, opts_out, cstate, NULL);
+

I think we can pass "text" in this case instead of NULL. That way,
ProcessCopyOptionFormatTo doesn't need to handle NULL case.

We need curly brackets for this "if branch" as follows:

if (!format_specifed)
{
    /* Set the default format. */
    ProcessCopyOptionFormatTo(pstate, opts_out, cstate, NULL);
}

---
+        /* Process not built-in options. */
+        foreach(option, unknown_options)
+        {
+                DefElem    *defel = lfirst_node(DefElem, option);
+                bool           processed = false;
+
+                if (!is_from)
+                        processed =
opts_out->to_routine->CopyToProcessOption(cstate, defel);
+                if (!processed)
+                        ereport(ERROR,
+                                        (errcode(ERRCODE_SYNTAX_ERROR),
+                                         errmsg("option \"%s\" not recognized",
+                                                        defel->defname),
+                                         parser_errposition(pstate,
defel->location)));
+        }
+        list_free(unknown_options);

I think we can check the duplicated options in the core as we discussed.

---
+static void
+CopyToTextBasedInit(CopyToState cstate)
+{
+}

and

+static void
+CopyToBinaryInit(CopyToState cstate)
+{
+}

Do we really need separate callbacks for the same behavior? I think we
can have a common init function say CopyToBuitinInit() that does
nothing. Or we can make the init callback optional.

The same is true for process-option callback.

---
         List      *convert_select; /* list of column names (can be NIL) */
+        const          CopyToRoutine *to_routine;      /* callback
routines for COPY TO */
 } CopyFormatOptions;

I think CopyToStateData is a better place to have CopyToRoutine.
copy_data_dest_cb is also there.

---
-                        if (strcmp(fmt, "text") == 0)
-                                 /* default format */ ;
-                        else if (strcmp(fmt, "csv") == 0)
-                                opts_out->csv_mode = true;
-                        else if (strcmp(fmt, "binary") == 0)
-                                opts_out->binary = true;
+
+                        if (is_from)
+                        {
+                                char      *fmt = defGetString(defel);
+
+                                if (strcmp(fmt, "text") == 0)
+                                         /* default format */ ;
+                                else if (strcmp(fmt, "csv") == 0)
+                                {
+                                        opts_out->csv_mode = true;
+                                }
+                                else if (strcmp(fmt, "binary") == 0)
+                                {
+                                        opts_out->binary = true;
+                                }
                         else
-                                ereport(ERROR,
-
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
-                                                 errmsg("COPY format
\"%s\" not recognized", fmt\),
-
parser_errposition(pstate, defel->location)));
+                                ProcessCopyOptionFormatTo(pstate,
opts_out, cstate, defel);

The 0002 patch replaces the options checks with
ProcessCopyOptionFormatFrom(). However, both
ProcessCopyOptionFormatTo() and ProcessCOpyOptionFormatFrom() would
set format-related options such as opts_out->csv_mode etc, which seems
not elegant. IIUC the reason why we process only the "format" option
first is to set the callback functions and call the init callback. So
I think we don't necessarily need to do both setting callbacks and
setting format-related options together. Probably we can do only the
callback stuff first and then set format-related options in the
original place we used to do?

---
+static void
+CopyToTextBasedFillCopyOutResponse(CopyToState cstate, StringInfoData *buf)
+{
+        int16          format = 0;
+        int                    natts = list_length(cstate->attnumlist);
+        int                    i;
+
+        pq_sendbyte(buf, format);      /* overall format */
+        pq_sendint16(buf, natts);
+        for (i = 0; i < natts; i++)
+                pq_sendint16(buf, format);     /* per-column formats */
+}

This function and CopyToBinaryFillCopyOutResponse() fill three things:
overall format, the number of columns, and per-column formats. While
this approach is flexible, extensions will have to understand the
format of CopyOutResponse message. An alternative is to have one or
more callbacks that return these three things.

---
+        /* Get info about the columns we need to process. */
+        cstate->out_functions = (FmgrInfo *) palloc(num_phys_attrs *
sizeof(Fmgr\Info));
+        foreach(cur, cstate->attnumlist)
+        {
+                int                    attnum = lfirst_int(cur);
+                Oid                    out_func_oid;
+                bool           isvarlena;
+                Form_pg_attribute attr = TupleDescAttr(tupDesc, attnum - 1);
+
+                getTypeOutputInfo(attr->atttypid, &out_func_oid, &isvarlena);
+                fmgr_info(out_func_oid, &cstate->out_functions[attnum - 1]);
+        }

Is preparing the out functions an extension's responsibility? I
thought the core could prepare them based on the overall format
specified by extensions, as long as the overall format matches the
actual data format to send. What do you think?

---
+        /*
+         * Called when COPY TO via the PostgreSQL protocol is
started. This must
+         * fill buf as a valid CopyOutResponse message:
+         *
+         */
+        /*--
+         * +--------+--------+--------+--------+--------+   +--------+--------+
+         * | Format | N attributes    | Attr1's format  |...| AttrN's format  |
+         * +--------+--------+--------+--------+--------+   +--------+--------+
+         * 0: text                      0: text               0: text
+         * 1: binary                    1: binary             1: binary
+         */

I think this kind of diagram could be missed from being updated when
we update the CopyOutResponse format. It's better to refer to the
documentation instead.


Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  92. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-01-30T05:45:31Z

    Hi,

In <CAD21AoBmNiWwrspuedgAPgbAqsn7e7NoZYF6gNnYBf+gXEk9Mg@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Tue, 30 Jan 2024 11:11:59 +0900,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

> ---
> +        if (!format_specified)
> +                /* Set the default format. */
> +                ProcessCopyOptionFormatTo(pstate, opts_out, cstate, NULL);
> +
> 
> I think we can pass "text" in this case instead of NULL. That way,
> ProcessCopyOptionFormatTo doesn't need to handle NULL case.

Yes, we can do it. But it needs a DefElem allocation. Is it
acceptable?

> We need curly brackets for this "if branch" as follows:
> 
> if (!format_specifed)
> {
>     /* Set the default format. */
>     ProcessCopyOptionFormatTo(pstate, opts_out, cstate, NULL);
> }

Oh, sorry. I assumed that pgindent adjusts the style too.

> ---
> +        /* Process not built-in options. */
> +        foreach(option, unknown_options)
> +        {
> +                DefElem    *defel = lfirst_node(DefElem, option);
> +                bool           processed = false;
> +
> +                if (!is_from)
> +                        processed =
> opts_out->to_routine->CopyToProcessOption(cstate, defel);
> +                if (!processed)
> +                        ereport(ERROR,
> +                                        (errcode(ERRCODE_SYNTAX_ERROR),
> +                                         errmsg("option \"%s\" not recognized",
> +                                                        defel->defname),
> +                                         parser_errposition(pstate,
> defel->location)));
> +        }
> +        list_free(unknown_options);
> 
> I think we can check the duplicated options in the core as we discussed.

Oh, sorry. I missed the part. I'll implement it.

> ---
> +static void
> +CopyToTextBasedInit(CopyToState cstate)
> +{
> +}
> 
> and
> 
> +static void
> +CopyToBinaryInit(CopyToState cstate)
> +{
> +}
> 
> Do we really need separate callbacks for the same behavior? I think we
> can have a common init function say CopyToBuitinInit() that does
> nothing. Or we can make the init callback optional.
> 
> The same is true for process-option callback.

OK. I'll make them optional.

> ---
>          List      *convert_select; /* list of column names (can be NIL) */
> +        const          CopyToRoutine *to_routine;      /* callback
> routines for COPY TO */
>  } CopyFormatOptions;
> 
> I think CopyToStateData is a better place to have CopyToRoutine.
> copy_data_dest_cb is also there.

We can do it but ProcessCopyOptions() accepts NULL
CopyToState for file_fdw. Can we create an empty
CopyToStateData internally like we did for opts_out in
ProcessCopyOptions()? (But it requires exporting
CopyToStateData. We'll export it in a later patch but it's
not yet in 0001.)

> The 0002 patch replaces the options checks with
> ProcessCopyOptionFormatFrom(). However, both
> ProcessCopyOptionFormatTo() and ProcessCOpyOptionFormatFrom() would
> set format-related options such as opts_out->csv_mode etc, which seems
> not elegant. IIUC the reason why we process only the "format" option
> first is to set the callback functions and call the init callback. So
> I think we don't necessarily need to do both setting callbacks and
> setting format-related options together. Probably we can do only the
> callback stuff first and then set format-related options in the
> original place we used to do?

If we do it, we need to write the (strcmp(format, "csv") ==
0) condition in copyto.c and copy.c. I wanted to avoid it. I
think that the duplication (setting opts_out->csv_mode in
copyto.c and copyfrom.c) is not a problem. But it's not a
strong opinion. If (strcmp(format, "csv") == 0) duplication
is better than opts_out->csv_mode = true duplication, I'll
do it.

BTW, if we want to make the CSV format implementation more
modularized, we will remove opts_out->csv_mode, move CSV
related options to CopyToCSVProcessOption() and keep CSV
related options in its opaque space. For example,
opts_out->force_quote exists in COPY TO opaque space but
doesn't exist in COPY FROM opaque space because it's not
used in COPY FROM.


> +static void
> +CopyToTextBasedFillCopyOutResponse(CopyToState cstate, StringInfoData *buf)
> +{
> +        int16          format = 0;
> +        int                    natts = list_length(cstate->attnumlist);
> +        int                    i;
> +
> +        pq_sendbyte(buf, format);      /* overall format */
> +        pq_sendint16(buf, natts);
> +        for (i = 0; i < natts; i++)
> +                pq_sendint16(buf, format);     /* per-column formats */
> +}
> 
> This function and CopyToBinaryFillCopyOutResponse() fill three things:
> overall format, the number of columns, and per-column formats. While
> this approach is flexible, extensions will have to understand the
> format of CopyOutResponse message. An alternative is to have one or
> more callbacks that return these three things.

Yes, we can choose the approach. I don't have a strong
opinion on which approach to choose.

> +        /* Get info about the columns we need to process. */
> +        cstate->out_functions = (FmgrInfo *) palloc(num_phys_attrs *
> sizeof(Fmgr\Info));
> +        foreach(cur, cstate->attnumlist)
> +        {
> +                int                    attnum = lfirst_int(cur);
> +                Oid                    out_func_oid;
> +                bool           isvarlena;
> +                Form_pg_attribute attr = TupleDescAttr(tupDesc, attnum - 1);
> +
> +                getTypeOutputInfo(attr->atttypid, &out_func_oid, &isvarlena);
> +                fmgr_info(out_func_oid, &cstate->out_functions[attnum - 1]);
> +        }
> 
> Is preparing the out functions an extension's responsibility? I
> thought the core could prepare them based on the overall format
> specified by extensions, as long as the overall format matches the
> actual data format to send. What do you think?

Hmm. I want to keep the preparation as an extension's
responsibility. Because it's not needed for all formats. For
example, Apache Arrow FORMAT doesn't need it. And JSON
FORMAT doesn't need it too because it use
composite_to_json().

> +        /*
> +         * Called when COPY TO via the PostgreSQL protocol is
> started. This must
> +         * fill buf as a valid CopyOutResponse message:
> +         *
> +         */
> +        /*--
> +         * +--------+--------+--------+--------+--------+   +--------+--------+
> +         * | Format | N attributes    | Attr1's format  |...| AttrN's format  |
> +         * +--------+--------+--------+--------+--------+   +--------+--------+
> +         * 0: text                      0: text               0: text
> +         * 1: binary                    1: binary             1: binary
> +         */
> 
> I think this kind of diagram could be missed from being updated when
> we update the CopyOutResponse format. It's better to refer to the
> documentation instead.

It makes sense. I couldn't find the documentation when I
wrote it but I found it now...:
https://www.postgresql.org/docs/current/protocol-flow.html#PROTOCOL-COPY

Is there recommended comment style to refer a documentation?
"See doc/src/sgml/protocol.sgml for the CopyOutResponse
message details" is OK?


Thanks,
-- 
kou

  93. Re: Make COPY format extendable: Extract COPY TO format implementations

    Michael Paquier <michael@paquier.xyz> — 2024-01-30T07:20:54Z

    On Tue, Jan 30, 2024 at 02:45:31PM +0900, Sutou Kouhei wrote:
> In <CAD21AoBmNiWwrspuedgAPgbAqsn7e7NoZYF6gNnYBf+gXEk9Mg@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Tue, 30 Jan 2024 11:11:59 +0900,
>   Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> 
>> ---
>> +        if (!format_specified)
>> +                /* Set the default format. */
>> +                ProcessCopyOptionFormatTo(pstate, opts_out, cstate, NULL);
>> +
>> 
>> I think we can pass "text" in this case instead of NULL. That way,
>> ProcessCopyOptionFormatTo doesn't need to handle NULL case.
> 
> Yes, we can do it. But it needs a DefElem allocation. Is it
> acceptable?

I don't think that there is a need for a DelElem at all here?  While I
am OK with the choice of calling CopyToInit() in the
ProcessCopyOption*() routines that exist to keep the set of callbacks
local to copyto.c and copyfrom.c, I think that this should not bother
about setting opts_out->csv_mode or opts_out->csv_mode but just set 
the opts_out->{to,from}_routine callbacks.

>> +static void
>> +CopyToTextBasedInit(CopyToState cstate)
>> +{
>> +}
>> 
>> and
>> 
>> +static void
>> +CopyToBinaryInit(CopyToState cstate)
>> +{
>> +}
>> 
>> Do we really need separate callbacks for the same behavior? I think we
>> can have a common init function say CopyToBuitinInit() that does
>> nothing. Or we can make the init callback optional.

Keeping empty options does not strike as a bad idea, because this
forces extension developers to think about this code path rather than
just ignore it.  Now, all the Init() callbacks are empty for the
in-core callbacks, so I think that we should just remove it entirely
for now.  Let's keep the core patch a maximum simple.  It is always
possible to build on top of it depending on what people need.  It's
been mentioned that JSON would want that, but this also proves that we
just don't care about that for all the in-core callbacks, as well.  I
would choose a minimalistic design for now.

>> +        /* Get info about the columns we need to process. */
>> +        cstate->out_functions = (FmgrInfo *) palloc(num_phys_attrs *
>> sizeof(Fmgr\Info));
>> +        foreach(cur, cstate->attnumlist)
>> +        {
>> +                int                    attnum = lfirst_int(cur);
>> +                Oid                    out_func_oid;
>> +                bool           isvarlena;
>> +                Form_pg_attribute attr = TupleDescAttr(tupDesc, attnum - 1);
>> +
>> +                getTypeOutputInfo(attr->atttypid, &out_func_oid, &isvarlena);
>> +                fmgr_info(out_func_oid, &cstate->out_functions[attnum - 1]);
>> +        }
>> 
>> Is preparing the out functions an extension's responsibility? I
>> thought the core could prepare them based on the overall format
>> specified by extensions, as long as the overall format matches the
>> actual data format to send. What do you think?
> 
> Hmm. I want to keep the preparation as an extension's
> responsibility. Because it's not needed for all formats. For
> example, Apache Arrow FORMAT doesn't need it. And JSON
> FORMAT doesn't need it too because it use
> composite_to_json().

I agree that it could be really useful for extensions to be able to
force that.  We already know that for the in-core formats we've cared
about being able to enforce the way data is handled in input and
output.

> It makes sense. I couldn't find the documentation when I
> wrote it but I found it now...:
> https://www.postgresql.org/docs/current/protocol-flow.html#PROTOCOL-COPY
> 
> Is there recommended comment style to refer a documentation?
> "See doc/src/sgml/protocol.sgml for the CopyOutResponse
> message details" is OK?

There are a couple of places in the C code where we refer to SGML docs
when it comes to specific details, so using a method like that here to
avoid a duplication with the docs sounds sensible for me.

I would be really tempted to put my hands on this patch to put into
shape a minimal set of changes because I'm caring quite a lot about
the performance gains reported with the removal of the "if" checks in
the per-row callbacks, and that's one goal of this thread quite
independent on the extensibility.  Sutou-san, would you be OK with
that?
--
Michael

  94. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-01-30T08:15:11Z

    Hi,

In <ZbijVn9_51mljMAG@paquier.xyz>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Tue, 30 Jan 2024 16:20:54 +0900,
  Michael Paquier <michael@paquier.xyz> wrote:

>>> +        if (!format_specified)
>>> +                /* Set the default format. */
>>> +                ProcessCopyOptionFormatTo(pstate, opts_out, cstate, NULL);
>>> +
>>> 
>>> I think we can pass "text" in this case instead of NULL. That way,
>>> ProcessCopyOptionFormatTo doesn't need to handle NULL case.
>> 
>> Yes, we can do it. But it needs a DefElem allocation. Is it
>> acceptable?
> 
> I don't think that there is a need for a DelElem at all here?

We use defel->location for an error message. (We don't need
to set location for the default "text" DefElem.)

>                                                                While I
> am OK with the choice of calling CopyToInit() in the
> ProcessCopyOption*() routines that exist to keep the set of callbacks
> local to copyto.c and copyfrom.c, I think that this should not bother
> about setting opts_out->csv_mode or opts_out->csv_mode but just set 
> the opts_out->{to,from}_routine callbacks.

OK. I'll keep opts_out->{csv_mode,binary} in copy.c.

>                  Now, all the Init() callbacks are empty for the
> in-core callbacks, so I think that we should just remove it entirely
> for now.  Let's keep the core patch a maximum simple.  It is always
> possible to build on top of it depending on what people need.  It's
> been mentioned that JSON would want that, but this also proves that we
> just don't care about that for all the in-core callbacks, as well.  I
> would choose a minimalistic design for now.

OK. Let's remove Init() callbacks from the first patch set.

> I would be really tempted to put my hands on this patch to put into
> shape a minimal set of changes because I'm caring quite a lot about
> the performance gains reported with the removal of the "if" checks in
> the per-row callbacks, and that's one goal of this thread quite
> independent on the extensibility.  Sutou-san, would you be OK with
> that?

Yes, sure.
(We want to focus on the performance gains in the first
patch set and then focus on extensibility again, right?)

For the purpose, I think that the v7 patch set is more
suitable than the v9 patch set. The v7 patch set doesn't
include Init() callbacks, custom options validation support
or extra Copy{In,Out}Response support. But the v7 patch set
misses the removal of the "if" checks in
NextCopyFromRawFields() that exists in the v9 patch set. I'm
not sure how much performance will improve by this but it
may be worth a try.

Can I prepare the v10 patch set as "the v7 patch set" + "the
removal of the "if" checks in NextCopyFromRawFields()"?
(+ reverting opts_out->{csv_mode,binary} changes in
ProcessCopyOptions().)


Thanks,
-- 
kou

  95. Re: Make COPY format extendable: Extract COPY TO format implementations

    Michael Paquier <michael@paquier.xyz> — 2024-01-30T08:37:35Z

    On Tue, Jan 30, 2024 at 05:15:11PM +0900, Sutou Kouhei wrote:
> We use defel->location for an error message. (We don't need
> to set location for the default "text" DefElem.)

Yeah, but you should not need to have this error in the paths that set
the callback routines in opts_out if the same validation happens a few
lines before, in copy.c.

> Yes, sure.
> (We want to focus on the performance gains in the first
> patch set and then focus on extensibility again, right?)

Yep, exactly, the numbers are too good to just ignore.  I don't want
to hijack the thread, but I am really worried about the complexities
this thread is getting into because we are trying to shape the
callbacks in the most generic way possible based on *two* use cases.
This is going to be a never-ending discussion.  I'd rather get some
simple basics, and then we can discuss if tweaking the callbacks is
really necessary or not.  Even after introducing the pg_proc lookups
to get custom callbacks.

> For the purpose, I think that the v7 patch set is more
> suitable than the v9 patch set. The v7 patch set doesn't
> include Init() callbacks, custom options validation support
> or extra Copy{In,Out}Response support. But the v7 patch set
> misses the removal of the "if" checks in
> NextCopyFromRawFields() that exists in the v9 patch set. I'm
> not sure how much performance will improve by this but it
> may be worth a try.

Yeah..  The custom options don't seem like an absolute strong
requirement for the first shot with the callbacks or even the
possibility to retrieve the callbacks from a function call.  I mean,
you could provide some control with SET commands and a few GUCs, at
least, even if that would be strange.  Manipulations with a list of
DefElems is the intuitive way to have custom options at query level,
but we also have to guess the set of callbacks from this list of
DefElems coming from the query.  You see my point, I am not sure 
if it would be the best thing to process twice the options, especially
when it comes to decide if a DefElem should be valid or not depending
on the callbacks used.  Or we could use a kind of "special" DefElem
where we could store a set of key:value fed to a callback :)

> Can I prepare the v10 patch set as "the v7 patch set" + "the
> removal of the "if" checks in NextCopyFromRawFields()"?
> (+ reverting opts_out->{csv_mode,binary} changes in
> ProcessCopyOptions().)

Yep, if I got it that would make sense to me.  If you can do that,
that would help quite a bit.  :)
--
Michael

  96. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-01-31T05:11:22Z

    Hi,

In <Zbi1TwPfAvUpKqTd@paquier.xyz>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Tue, 30 Jan 2024 17:37:35 +0900,
  Michael Paquier <michael@paquier.xyz> wrote:

>> We use defel->location for an error message. (We don't need
>> to set location for the default "text" DefElem.)
> 
> Yeah, but you should not need to have this error in the paths that set
> the callback routines in opts_out if the same validation happens a few
> lines before, in copy.c.

Ah, yes. defel->location is used in later patches. For
example, it's used when a COPY handler for the specified
FORMAT isn't found.

>                           I am really worried about the complexities
> this thread is getting into because we are trying to shape the
> callbacks in the most generic way possible based on *two* use cases.
> This is going to be a never-ending discussion.  I'd rather get some
> simple basics, and then we can discuss if tweaking the callbacks is
> really necessary or not.  Even after introducing the pg_proc lookups
> to get custom callbacks.

I understand your concern. Let's introduce minimal callbacks
as the first step. I think that we completed our design
discussion for this feature. We can choose minimal callbacks
based on the discussion.

>         The custom options don't seem like an absolute strong
> requirement for the first shot with the callbacks or even the
> possibility to retrieve the callbacks from a function call.  I mean,
> you could provide some control with SET commands and a few GUCs, at
> least, even if that would be strange.  Manipulations with a list of
> DefElems is the intuitive way to have custom options at query level,
> but we also have to guess the set of callbacks from this list of
> DefElems coming from the query.  You see my point, I am not sure 
> if it would be the best thing to process twice the options, especially
> when it comes to decide if a DefElem should be valid or not depending
> on the callbacks used.  Or we could use a kind of "special" DefElem
> where we could store a set of key:value fed to a callback :)

Interesting. Let's remove custom options support from the
initial minimal callbacks.

>> Can I prepare the v10 patch set as "the v7 patch set" + "the
>> removal of the "if" checks in NextCopyFromRawFields()"?
>> (+ reverting opts_out->{csv_mode,binary} changes in
>> ProcessCopyOptions().)
> 
> Yep, if I got it that would make sense to me.  If you can do that,
> that would help quite a bit.  :)

I've prepared the v10 patch set. Could you try this?

Changes since the v7 patch set:

0001:

* Remove CopyToProcessOption() callback
* Remove CopyToGetFormat() callback
* Revert passing CopyToState to ProcessCopyOptions()
* Revert moving "opts_out->{csv_mode,binary} = true" to
  ProcessCopyOptionFormatTo()
* Change to receive "const char *format" instead "DefElem  *defel"
  by ProcessCopyOptionFormatTo()

0002:

* Remove CopyFromProcessOption() callback
* Remove CopyFromGetFormat() callback
* Change to receive "const char *format" instead "DefElem
  *defel" by ProcessCopyOptionFormatFrom()
* Remove "if (cstate->opts.csv_mode)" branches from
  NextCopyFromRawFields()



FYI: Here are Copy{From,To}Routine in the v10 patch set. I
think that only Copy{From,To}OneRow are minimal callbacks
for the performance gain. But can we keep Copy{From,To}Start
and Copy{From,To}End for consistency? We can remove a few
{csv_mode,binary} conditions by Copy{From,To}{Start,End}. It
doesn't depend on the number of COPY target tuples. So they
will not affect performance.

/* Routines for a COPY FROM format implementation. */
typedef struct CopyFromRoutine
{
	/*
	 * Called when COPY FROM is started. This will initialize something and
	 * receive a header.
	 */
	void		(*CopyFromStart) (CopyFromState cstate, TupleDesc tupDesc);

	/* Copy one row. It returns false if no more tuples. */
	bool		(*CopyFromOneRow) (CopyFromState cstate, ExprContext *econtext, Datum *values, bool *nulls);

	/* Called when COPY FROM is ended. This will finalize something. */
	void		(*CopyFromEnd) (CopyFromState cstate);
}			CopyFromRoutine;

/* Routines for a COPY TO format implementation. */
typedef struct CopyToRoutine
{
	/* Called when COPY TO is started. This will send a header. */
	void		(*CopyToStart) (CopyToState cstate, TupleDesc tupDesc);

	/* Copy one row for COPY TO. */
	void		(*CopyToOneRow) (CopyToState cstate, TupleTableSlot *slot);

	/* Called when COPY TO is ended. This will send a trailer. */
	void		(*CopyToEnd) (CopyToState cstate);
}			CopyToRoutine;




Thanks,
-- 
kou

  97. Re: Make COPY format extendable: Extract COPY TO format implementations

    Michael Paquier <michael@paquier.xyz> — 2024-01-31T05:39:54Z

    On Wed, Jan 31, 2024 at 02:11:22PM +0900, Sutou Kouhei wrote:
> Ah, yes. defel->location is used in later patches. For
> example, it's used when a COPY handler for the specified
> FORMAT isn't found.

I see.

> I've prepared the v10 patch set. Could you try this?

Thanks, I'm looking into that now.

> FYI: Here are Copy{From,To}Routine in the v10 patch set. I
> think that only Copy{From,To}OneRow are minimal callbacks
> for the performance gain. But can we keep Copy{From,To}Start
> and Copy{From,To}End for consistency? We can remove a few
> {csv_mode,binary} conditions by Copy{From,To}{Start,End}. It
> doesn't depend on the number of COPY target tuples. So they
> will not affect performance.

I think I'm OK to keep the start/end callbacks.  This makes the code
more consistent as a whole, as well.
--
Michael

  98. Re: Make COPY format extendable: Extract COPY TO format implementations

    Michael Paquier <michael@paquier.xyz> — 2024-02-01T01:57:58Z

    On Wed, Jan 31, 2024 at 02:39:54PM +0900, Michael Paquier wrote:
> Thanks, I'm looking into that now.

I have much to say about the patch, but for now I have begun running
some performance tests using the patches, because this thread won't
get far until we are sure that the callbacks do not impact performance
in some kind of worst-case scenario.  First, here is what I used to
setup a set of tables used for COPY FROM and COPY TO (requires [1] to
feed COPY FROM's data to the void, and note that default values is to
have a strict control on the size of the StringInfos used in the copy
paths):
CREATE EXTENSION blackhole_am;
CREATE OR REPLACE FUNCTION create_table_cols(tabname text, num_cols int)
RETURNS VOID AS
$func$
DECLARE
  query text;
BEGIN
  query := 'CREATE UNLOGGED TABLE ' || tabname || ' (';
  FOR i IN 1..num_cols LOOP
    query := query || 'a_' || i::text || ' int default 1';
    IF i != num_cols THEN
      query := query || ', ';
    END IF;
  END LOOP;
  query := query || ')';
  EXECUTE format(query);
END
$func$ LANGUAGE plpgsql;
-- Tables used for COPY TO
SELECT create_table_cols ('to_tab_1', 1);
SELECT create_table_cols ('to_tab_10', 10);
INSERT INTO to_tab_1 SELECT FROM generate_series(1, 10000000);
INSERT INTO to_tab_10 SELECT FROM generate_series(1, 10000000);
-- Data for COPY FROM
COPY to_tab_1 TO '/tmp/to_tab_1.bin' WITH (format binary);
COPY to_tab_10 TO '/tmp/to_tab_10.bin' WITH (format binary);
COPY to_tab_1 TO '/tmp/to_tab_1.txt' WITH (format text);
COPY to_tab_10 TO '/tmp/to_tab_10.txt' WITH (format text);
-- Tables used for COPY FROM
SELECT create_table_cols ('from_tab_1', 1);
SELECT create_table_cols ('from_tab_10', 10);
ALTER TABLE from_tab_1 SET ACCESS METHOD blackhole_am;
ALTER TABLE from_tab_10 SET ACCESS METHOD blackhole_am;

Then I have run a set of tests using HEAD, v7 and v10 with queries
like that (adapt them depending on the format and table):
COPY to_tab_1 TO '/dev/null' WITH (FORMAT text) \watch count=5
SET client_min_messages TO error; -- for blackhole_am
COPY from_tab_1 FROM '/tmp/to_tab_1.txt' with (FORMAT 'text') \watch count=5
COPY from_tab_1 FROM '/tmp/to_tab_1.bin' with (FORMAT 'binary') \watch count=5

All the patches have been compiled with -O2, without assertions, etc.
Postgres is run in tmpfs mode, on scissors, without fsync.  Unlogged
tables help a bit in focusing on the execution paths as we don't care
about WAL, of course.  I have also included v7 in the test of tests,
as this version uses more simple per-row callbacks.

And here are the results I get for text and binary (ms, average of 15
queries after discarding the three highest and three lowest values):
      test       | master |  v7  | v10  
-----------------+--------+------+------
 from_bin_1col   | 1575   | 1546 | 1575
 from_bin_10col  | 5364   | 5208 | 5230
 from_text_1col  | 1690   | 1715 | 1684
 from_text_10col | 4875   | 4793 | 4757
 to_bin_1col     | 1717   | 1730 | 1731
 to_bin_10col    | 7728   | 7707 | 7513
 to_text_1col    | 1710   | 1730 | 1698
 to_text_10col   | 5998   | 5960 | 5987

I am getting an interesting trend here in terms of a speedup between
HEAD and the patches with a table that has 10 attributes filled with
integers, especially for binary and text with COPY FROM.  COPY TO
binary also gets nice numbers, while text looks rather stable.  Hmm.

These were on my buildfarm animal, but we need to be more confident
about all this.  Could more people run these tests?  I am going to do
a second session on a local machine I have at hand and see what
happens.  Will publish the numbers here, the method will be the same.

[1]: https://github.com/michaelpq/pg_plugins/tree/main/blackhole_am
--
Michael

  99. Re: Make COPY format extendable: Extract COPY TO format implementations

    Junwang Zhao <zhjwpku@gmail.com> — 2024-02-01T03:43:07Z

    Hi Michael,

On Thu, Feb 1, 2024 at 9:58 AM Michael Paquier <michael@paquier.xyz> wrote:
>
> On Wed, Jan 31, 2024 at 02:39:54PM +0900, Michael Paquier wrote:
> > Thanks, I'm looking into that now.
>
> I have much to say about the patch, but for now I have begun running
> some performance tests using the patches, because this thread won't
> get far until we are sure that the callbacks do not impact performance
> in some kind of worst-case scenario.  First, here is what I used to
> setup a set of tables used for COPY FROM and COPY TO (requires [1] to
> feed COPY FROM's data to the void, and note that default values is to
> have a strict control on the size of the StringInfos used in the copy
> paths):
> CREATE EXTENSION blackhole_am;
> CREATE OR REPLACE FUNCTION create_table_cols(tabname text, num_cols int)
> RETURNS VOID AS
> $func$
> DECLARE
>   query text;
> BEGIN
>   query := 'CREATE UNLOGGED TABLE ' || tabname || ' (';
>   FOR i IN 1..num_cols LOOP
>     query := query || 'a_' || i::text || ' int default 1';
>     IF i != num_cols THEN
>       query := query || ', ';
>     END IF;
>   END LOOP;
>   query := query || ')';
>   EXECUTE format(query);
> END
> $func$ LANGUAGE plpgsql;
> -- Tables used for COPY TO
> SELECT create_table_cols ('to_tab_1', 1);
> SELECT create_table_cols ('to_tab_10', 10);
> INSERT INTO to_tab_1 SELECT FROM generate_series(1, 10000000);
> INSERT INTO to_tab_10 SELECT FROM generate_series(1, 10000000);
> -- Data for COPY FROM
> COPY to_tab_1 TO '/tmp/to_tab_1.bin' WITH (format binary);
> COPY to_tab_10 TO '/tmp/to_tab_10.bin' WITH (format binary);
> COPY to_tab_1 TO '/tmp/to_tab_1.txt' WITH (format text);
> COPY to_tab_10 TO '/tmp/to_tab_10.txt' WITH (format text);
> -- Tables used for COPY FROM
> SELECT create_table_cols ('from_tab_1', 1);
> SELECT create_table_cols ('from_tab_10', 10);
> ALTER TABLE from_tab_1 SET ACCESS METHOD blackhole_am;
> ALTER TABLE from_tab_10 SET ACCESS METHOD blackhole_am;
>
> Then I have run a set of tests using HEAD, v7 and v10 with queries
> like that (adapt them depending on the format and table):
> COPY to_tab_1 TO '/dev/null' WITH (FORMAT text) \watch count=5
> SET client_min_messages TO error; -- for blackhole_am
> COPY from_tab_1 FROM '/tmp/to_tab_1.txt' with (FORMAT 'text') \watch count=5
> COPY from_tab_1 FROM '/tmp/to_tab_1.bin' with (FORMAT 'binary') \watch count=5
>
> All the patches have been compiled with -O2, without assertions, etc.
> Postgres is run in tmpfs mode, on scissors, without fsync.  Unlogged
> tables help a bit in focusing on the execution paths as we don't care
> about WAL, of course.  I have also included v7 in the test of tests,
> as this version uses more simple per-row callbacks.
>
> And here are the results I get for text and binary (ms, average of 15
> queries after discarding the three highest and three lowest values):
>       test       | master |  v7  | v10
> -----------------+--------+------+------
>  from_bin_1col   | 1575   | 1546 | 1575
>  from_bin_10col  | 5364   | 5208 | 5230
>  from_text_1col  | 1690   | 1715 | 1684
>  from_text_10col | 4875   | 4793 | 4757
>  to_bin_1col     | 1717   | 1730 | 1731
>  to_bin_10col    | 7728   | 7707 | 7513
>  to_text_1col    | 1710   | 1730 | 1698
>  to_text_10col   | 5998   | 5960 | 5987
>
> I am getting an interesting trend here in terms of a speedup between
> HEAD and the patches with a table that has 10 attributes filled with
> integers, especially for binary and text with COPY FROM.  COPY TO
> binary also gets nice numbers, while text looks rather stable.  Hmm.
>
> These were on my buildfarm animal, but we need to be more confident
> about all this.  Could more people run these tests?  I am going to do
> a second session on a local machine I have at hand and see what
> happens.  Will publish the numbers here, the method will be the same.
>
> [1]: https://github.com/michaelpq/pg_plugins/tree/main/blackhole_am
> --
> Michael

I'm running the benchmark, but I got some strong numbers:

postgres=# \timing
Timing is on.
postgres=# COPY to_tab_10 TO '/dev/null' WITH (FORMAT binary) \watch count=15
COPY 10000000
Time: 3168.497 ms (00:03.168)
COPY 10000000
Time: 3255.464 ms (00:03.255)
COPY 10000000
Time: 3270.625 ms (00:03.271)
COPY 10000000
Time: 3285.112 ms (00:03.285)
COPY 10000000
Time: 3322.304 ms (00:03.322)
COPY 10000000
Time: 3341.328 ms (00:03.341)
COPY 10000000
Time: 3621.564 ms (00:03.622)
COPY 10000000
Time: 3700.911 ms (00:03.701)
COPY 10000000
Time: 3717.992 ms (00:03.718)
COPY 10000000
Time: 3708.350 ms (00:03.708)
COPY 10000000
Time: 3704.367 ms (00:03.704)
COPY 10000000
Time: 3724.281 ms (00:03.724)
COPY 10000000
Time: 3703.335 ms (00:03.703)
COPY 10000000
Time: 3728.629 ms (00:03.729)
COPY 10000000
Time: 3758.135 ms (00:03.758)

The first 6 rounds are like 10% better than the later 9 rounds, is this normal?

-- 
Regards
Junwang Zhao

  100. Re: Make COPY format extendable: Extract COPY TO format implementations

    Michael Paquier <michael@paquier.xyz> — 2024-02-01T03:49:59Z

    On Thu, Feb 01, 2024 at 10:57:58AM +0900, Michael Paquier wrote:
> And here are the results I get for text and binary (ms, average of 15
> queries after discarding the three highest and three lowest values):
>       test       | master |  v7  | v10  
> -----------------+--------+------+------
>  from_bin_1col   | 1575   | 1546 | 1575
>  from_bin_10col  | 5364   | 5208 | 5230
>  from_text_1col  | 1690   | 1715 | 1684
>  from_text_10col | 4875   | 4793 | 4757
>  to_bin_1col     | 1717   | 1730 | 1731
>  to_bin_10col    | 7728   | 7707 | 7513
>  to_text_1col    | 1710   | 1730 | 1698
>  to_text_10col   | 5998   | 5960 | 5987

Here are some numbers from a second local machine:
      test       | master |  v7  | v10  
-----------------+--------+------+------
 from_bin_1col   | 508    | 467  | 461
 from_bin_10col  | 2192   | 2083 | 2098
 from_text_1col  | 510    | 499  | 517
 from_text_10col | 1970   | 1678 | 1654
 to_bin_1col     | 575    | 577  | 573
 to_bin_10col    | 2680   | 2678 | 2722
 to_text_1col    | 516    | 506  | 527
 to_text_10col   | 2250   | 2245 | 2235

This is confirming a speedup with COPY FROM for both text and binary,
with more impact with a larger number of attributes.  That's harder to
conclude about COPY TO in both cases, but at least I'm not seeing any
regression even with some variance caused by what looks like noise.
We need more numbers from more people.  Sutou-san or Sawada-san, or
any volunteers?
--
Michael

  101. Re: Make COPY format extendable: Extract COPY TO format implementations

    Michael Paquier <michael@paquier.xyz> — 2024-02-01T03:56:22Z

    On Thu, Feb 01, 2024 at 11:43:07AM +0800, Junwang Zhao wrote:
> The first 6 rounds are like 10% better than the later 9 rounds, is this normal?

Even with HEAD?  Perhaps you have some OS cache eviction in play here?
FWIW, I'm not seeing any of that with longer runs after 7~ tries in a
loop of 15.
--
Michael

  102. Re: Make COPY format extendable: Extract COPY TO format implementations

    Junwang Zhao <zhjwpku@gmail.com> — 2024-02-01T04:20:11Z

    On Thu, Feb 1, 2024 at 11:56 AM Michael Paquier <michael@paquier.xyz> wrote:
>
> On Thu, Feb 01, 2024 at 11:43:07AM +0800, Junwang Zhao wrote:
> > The first 6 rounds are like 10% better than the later 9 rounds, is this normal?
>
> Even with HEAD?  Perhaps you have some OS cache eviction in play here?
> FWIW, I'm not seeing any of that with longer runs after 7~ tries in a
> loop of 15.

Yeah, with HEAD. I'm on ubuntu 22.04, I did not change any gucs, maybe I should
set a higher shared_buffers? But I dought that's related ;(


> --
> Michael



-- 
Regards
Junwang Zhao

  103. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-02-01T15:19:51Z

    Hi,

Thanks for preparing benchmark.

In <ZbsU53b3eEV-mMT3@paquier.xyz>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Thu, 1 Feb 2024 12:49:59 +0900,
  Michael Paquier <michael@paquier.xyz> wrote:

> On Thu, Feb 01, 2024 at 10:57:58AM +0900, Michael Paquier wrote:
>> And here are the results I get for text and binary (ms, average of 15
>> queries after discarding the three highest and three lowest values):
>>       test       | master |  v7  | v10  
>> -----------------+--------+------+------
>>  from_bin_1col   | 1575   | 1546 | 1575
>>  from_bin_10col  | 5364   | 5208 | 5230
>>  from_text_1col  | 1690   | 1715 | 1684
>>  from_text_10col | 4875   | 4793 | 4757
>>  to_bin_1col     | 1717   | 1730 | 1731
>>  to_bin_10col    | 7728   | 7707 | 7513
>>  to_text_1col    | 1710   | 1730 | 1698
>>  to_text_10col   | 5998   | 5960 | 5987
> 
> Here are some numbers from a second local machine:
>       test       | master |  v7  | v10  
> -----------------+--------+------+------
>  from_bin_1col   | 508    | 467  | 461
>  from_bin_10col  | 2192   | 2083 | 2098
>  from_text_1col  | 510    | 499  | 517
>  from_text_10col | 1970   | 1678 | 1654
>  to_bin_1col     | 575    | 577  | 573
>  to_bin_10col    | 2680   | 2678 | 2722
>  to_text_1col    | 516    | 506  | 527
>  to_text_10col   | 2250   | 2245 | 2235
> 
> This is confirming a speedup with COPY FROM for both text and binary,
> with more impact with a larger number of attributes.  That's harder to
> conclude about COPY TO in both cases, but at least I'm not seeing any
> regression even with some variance caused by what looks like noise.
> We need more numbers from more people.  Sutou-san or Sawada-san, or
> any volunteers?

Here are some numbers on my local machine (Note that my
local machine isn't suitable for benchmark as I said
before. Each number is median of "\watch 15" results):

1:
 direction     format  n_columns     master         v7        v10
        to       text          1   1077.254   1016.953   1028.434
        to        csv          1    1079.88   1055.545    1053.95
        to     binary          1   1051.247    1033.93    1003.44
        to       text         10   4373.168   3980.442    3955.94
        to        csv         10   4753.842     4719.2   4677.643
        to     binary         10   4598.374   4431.238   4285.757
      from       text          1    875.729    916.526    869.283
      from        csv          1    909.355   1001.277    918.655
      from     binary          1    872.943    907.778    859.433
      from       text         10   2594.429   2345.292   2587.603
      from        csv         10   2968.972   3039.544   2964.468
      from     binary         10    3072.01   3109.267   3093.983

2:
 direction     format  n_columns     master         v7        v10
        to       text          1   1061.908    988.768    978.291
        to        csv          1   1095.109   1037.015   1041.613
        to     binary          1   1076.992   1000.212    983.318
        to       text         10   4336.517   3901.833   3841.789
        to        csv         10   4679.411   4640.975   4570.774
        to     binary         10    4465.04   4508.063   4261.749
      from       text          1    866.689     917.54    830.417
      from        csv          1    917.973   1695.401    871.991
      from     binary          1    841.104   1422.012    820.786
      from       text         10   2523.607   3147.738   2517.505
      from        csv         10   2917.018   3042.685   2950.338
      from     binary         10   2998.051   3128.542   3018.954

3:
 direction     format  n_columns     master         v7        v10
        to       text          1   1021.168   1031.183    962.945
        to        csv          1   1076.549   1069.661   1060.258
        to     binary          1   1024.611   1022.143    975.768
        to       text         10    4327.24   3936.703   4049.893
        to        csv         10   4620.436   4531.676   4685.672
        to     binary         10   4457.165   4390.992   4301.463
      from       text          1    887.532    907.365    888.892
      from        csv          1    945.167    1012.29    895.921
      from     binary          1     853.06    854.652    849.661
      from       text         10   2660.509   2304.256   2527.071
      from        csv         10   2913.644   2968.204   2935.081
      from     binary         10   3020.812   3081.162   3090.803

I'll measure again on my local machine later. I'll stop
other processes such as Web browser, editor and so on as
much as possible when I do.


Thanks,
-- 
kou

  104. Re: Make COPY format extendable: Extract COPY TO format implementations

    Michael Paquier <michael@paquier.xyz> — 2024-02-01T21:51:02Z

    On Fri, Feb 02, 2024 at 12:19:51AM +0900, Sutou Kouhei wrote:
> Here are some numbers on my local machine (Note that my
> local machine isn't suitable for benchmark as I said
> before. Each number is median of "\watch 15" results):
>>
> I'll measure again on my local machine later. I'll stop
> other processes such as Web browser, editor and so on as
> much as possible when I do.

Thanks for compiling some numbers.  This is showing a lot of variance.
Expecially, these two lines in table 2 are showing surprising results
for v7:
  direction     format  n_columns     master         v7        v10
       from        csv          1    917.973   1695.401    871.991
       from     binary          1    841.104   1422.012    820.786

I am going to try to plug in some rusage() calls in the backend for
the COPY paths.  I hope that gives more precision about the backend
activity.  I'll post that with more numbers.
--
Michael

  105. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-02-02T00:40:56Z

    Hi,

In <ZbwSRsCqVS638Xjz@paquier.xyz>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 2 Feb 2024 06:51:02 +0900,
  Michael Paquier <michael@paquier.xyz> wrote:

> On Fri, Feb 02, 2024 at 12:19:51AM +0900, Sutou Kouhei wrote:
>> Here are some numbers on my local machine (Note that my
>> local machine isn't suitable for benchmark as I said
>> before. Each number is median of "\watch 15" results):
>>>
>> I'll measure again on my local machine later. I'll stop
>> other processes such as Web browser, editor and so on as
>> much as possible when I do.
> 
> Thanks for compiling some numbers.  This is showing a lot of variance.
> Expecially, these two lines in table 2 are showing surprising results
> for v7:
>   direction     format  n_columns     master         v7        v10
>        from        csv          1    917.973   1695.401    871.991
>        from     binary          1    841.104   1422.012    820.786

Here are more numbers:

1:
 direction     format  n_columns     master         v7        v10
        to       text          1   1053.844    978.998    956.575
        to        csv          1   1091.316   1020.584   1098.314
        to     binary          1   1034.685    969.224    980.458
        to       text         10   4216.264   3886.515   4111.417
        to        csv         10   4649.228   4530.882   4682.988
        to     binary         10   4219.228    4189.99   4211.942
      from       text          1    851.697    896.968    890.458
      from        csv          1    890.229    936.231     887.15
      from     binary          1    784.407     817.07    938.736
      from       text         10   2549.056   2233.899   2630.892
      from        csv         10   2809.441   2868.411   2895.196
      from     binary         10   2985.674   3027.522     3397.5

2:
 direction     format  n_columns     master         v7        v10
        to       text          1   1013.764   1011.968    940.855
        to        csv          1   1060.431   1065.468    1040.68
        to     binary          1   1013.652   1009.956    965.675
        to       text         10   4411.484   4031.571   3896.836
        to        csv         10   4739.625    4715.81   4631.002
        to     binary         10   4374.077   4357.942   4227.215
      from       text          1    955.078    922.346    866.222
      from        csv          1   1040.717    986.524    905.657
      from     binary          1    849.316    864.859    833.152
      from       text         10   2703.209   2361.651   2533.992
      from        csv         10    2990.35   3059.167   2930.632
      from     binary         10   3008.375   3368.714   3055.723

3:
 direction     format  n_columns     master         v7        v10
        to       text          1   1084.756   1003.822    994.409
        to        csv          1     1092.4   1062.536   1079.027
        to     binary          1   1046.774    994.168    993.633
        to       text         10    4363.51   3978.205   4124.359
        to        csv         10   4866.762   4616.001   4715.052
        to     binary         10   4382.412   4363.269   4213.456
      from       text          1    852.976    907.315    860.749
      from        csv          1    925.187    962.632    897.833
      from     binary          1    824.997    897.046    828.231
      from       text         10    2591.07   2358.541   2540.431
      from        csv         10   2907.033   3018.486   2915.997
      from     binary         10   3069.027    3209.21   3119.128

Other processes are stopped while I measure them. But I'm
not sure these numbers are more reliable than before...

> I am going to try to plug in some rusage() calls in the backend for
> the COPY paths.  I hope that gives more precision about the backend
> activity.  I'll post that with more numbers.

Thanks. It'll help us.


-- 
kou

  106. Re: Make COPY format extendable: Extract COPY TO format implementations

    Michael Paquier <michael@paquier.xyz> — 2024-02-02T00:52:04Z

    On Fri, Feb 02, 2024 at 06:51:02AM +0900, Michael Paquier wrote:
> I am going to try to plug in some rusage() calls in the backend for
> the COPY paths.  I hope that gives more precision about the backend
> activity.  I'll post that with more numbers.

And here they are with log_statement_stats enabled to get rusage() fot
these queries:
         test         |  user_s  | system_s | elapsed_s 
----------------------+----------+----------+-----------
 head_to_bin_1col     | 1.639761 | 0.007998 |  1.647762
 v7_to_bin_1col       | 1.645499 | 0.004003 |  1.649498
 v10_to_bin_1col      | 1.639466 | 0.004008 |  1.643488

 head_to_bin_10col    | 7.486369 | 0.056007 |  7.542485
 v7_to_bin_10col      | 7.314341 | 0.039990 |  7.354743
 v10_to_bin_10col     | 7.329355 | 0.052007 |  7.381408

 head_to_text_1col    | 1.581140 | 0.012000 |  1.593166
 v7_to_text_1col      | 1.615441 | 0.003992 |  1.619446
 v10_to_text_1col     | 1.613443 | 0.000000 |  1.613454

 head_to_text_10col   | 5.897014 | 0.011990 |  5.909063
 v7_to_text_10col     | 5.722872 | 0.016014 |  5.738979
 v10_to_text_10col    | 5.762286 | 0.011993 |  5.774265

 head_from_bin_1col   | 1.524038 | 0.020000 |  1.544046
 v7_from_bin_1col     | 1.551367 | 0.016015 |  1.567408
 v10_from_bin_1col    | 1.560087 | 0.016001 |  1.576115

 head_from_bin_10col  | 5.238444 | 0.139993 |  5.378595
 v7_from_bin_10col    | 5.170503 | 0.076021 |  5.246588
 v10_from_bin_10col   | 5.106496 | 0.112020 |  5.218565

 head_from_text_1col  | 1.664124 | 0.003998 |  1.668172
 v7_from_text_1col    | 1.720616 | 0.007990 |  1.728617
 v10_from_text_1col   | 1.683950 | 0.007990 |  1.692098

 head_from_text_10col | 4.859651 | 0.015996 |  4.875747
 v7_from_text_10col   | 4.775975 | 0.032000 |  4.808051
 v10_from_text_10col  | 4.737512 | 0.028012 |  4.765522
(24 rows)

I'm looking at this table, and what I can see is still a lot of
variance in the tests with tables involving 1 attribute.  However, a
second thing stands out to me here: there is a speedup with the
10-attribute case for all both COPY FROM and COPY TO, and both
formats.  The data posted at [1] is showing me the same trend.  In
short, let's move on with this split refactoring with the per-row
callbacks.  That clearly shows benefits.

[1] https://www.postgresql.org/message-id/Zbr6piWuVHDtFFOl@paquier.xyz
--
Michael

  107. Re: Make COPY format extendable: Extract COPY TO format implementations

    Michael Paquier <michael@paquier.xyz> — 2024-02-02T06:21:31Z

    On Fri, Feb 02, 2024 at 09:40:56AM +0900, Sutou Kouhei wrote:
> Thanks. It'll help us.

I have done a review of v10, see v11 attached which is still WIP, with
the patches for COPY TO and COPY FROM merged together.  Note that I'm
thinking to merge them into a single commit.

@@ -74,11 +75,11 @@ typedef struct CopyFormatOptions
     bool        convert_selectively;    /* do selective binary conversion? */
     CopyOnErrorChoice on_error; /* what to do when error happened */
     List       *convert_select; /* list of column names (can be NIL) */
+    const        CopyToRoutine *to_routine;    /* callback routines for COPY TO */
 } CopyFormatOptions;

Adding the routines to the structure for the format options is in my
opinion incorrect.  The elements of this structure are first processed
in the option deparsing path, and then we need to use the options to
guess which routines we need.  A more natural location is cstate
itself, so as the pointer to the routines is isolated within copyto.c
and copyfrom_internal.h.  My point is: the routines are an
implementation detail that the centralized copy.c has no need to know
about.  This also led to a strange separation with
ProcessCopyOptionFormatFrom() and ProcessCopyOptionFormatTo() to fit
the hole in-between.

The separation between cstate and the format-related fields could be
much better, though I am not sure if it is worth doing as it
introduces more duplication.  For example, max_fields and raw_fields
are specific to text and csv, while binary does not care much.
Perhaps this is just useful to be for custom formats.

copyapi.h needs more documentation, like what is expected for
extension developers when using these, what are the arguments, etc.  I
have added what I had in mind for now.

+typedef char *(*PostpareColumnValue) (CopyFromState cstate, char *string, int m);

CopyReadAttributes and PostpareColumnValue are also callbacks specific
to text and csv, except that they are used within the per-row
callbacks.  The same can be said about CopyAttributeOutHeaderFunction.
It seems to me that it would be less confusing to store pointers to
them in the routine structures, where the final picture involves not
having multiple layers of APIs like CopyToCSVStart,
CopyAttributeOutTextValue, etc.  These *have* to be documented
properly in copyapi.h, and this is much easier now that cstate stores
the routine pointers.  That would also make simpler function stacks.
Note that I have not changed that in the v11 attached.

This business with the extra callbacks required for csv and text is my
main point of contention, but I'd be OK once the model of the APIs is
more linear, with everything in Copy{From,To}State.  The changes would
be rather simple, and I'd be OK to put my hands on it.  Just,
Sutou-san, would you agree with my last point about these extra
callbacks?
--
Michael

  108. Re: Make COPY format extendable: Extract COPY TO format implementations

    Junwang Zhao <zhjwpku@gmail.com> — 2024-02-02T07:27:15Z

    On Fri, Feb 2, 2024 at 2:21 PM Michael Paquier <michael@paquier.xyz> wrote:
>
> On Fri, Feb 02, 2024 at 09:40:56AM +0900, Sutou Kouhei wrote:
> > Thanks. It'll help us.
>
> I have done a review of v10, see v11 attached which is still WIP, with
> the patches for COPY TO and COPY FROM merged together.  Note that I'm
> thinking to merge them into a single commit.
>
> @@ -74,11 +75,11 @@ typedef struct CopyFormatOptions
>      bool        convert_selectively;    /* do selective binary conversion? */
>      CopyOnErrorChoice on_error; /* what to do when error happened */
>      List       *convert_select; /* list of column names (can be NIL) */
> +    const        CopyToRoutine *to_routine;    /* callback routines for COPY TO */
>  } CopyFormatOptions;
>
> Adding the routines to the structure for the format options is in my
> opinion incorrect.  The elements of this structure are first processed
> in the option deparsing path, and then we need to use the options to
> guess which routines we need.  A more natural location is cstate
> itself, so as the pointer to the routines is isolated within copyto.c

I agree CopyToRoutine should be placed into CopyToStateData, but
why set it after ProcessCopyOptions, the implementation of
CopyToGetRoutine doesn't make sense if we want to support custom
format in the future.

Seems the refactor of v11 only considered performance but not
*extendable copy format*.

> and copyfrom_internal.h.  My point is: the routines are an
> implementation detail that the centralized copy.c has no need to know
> about.  This also led to a strange separation with
> ProcessCopyOptionFormatFrom() and ProcessCopyOptionFormatTo() to fit
> the hole in-between.
>
> The separation between cstate and the format-related fields could be
> much better, though I am not sure if it is worth doing as it
> introduces more duplication.  For example, max_fields and raw_fields
> are specific to text and csv, while binary does not care much.
> Perhaps this is just useful to be for custom formats.

I think those can be placed in format specific fields by utilizing the opaque
space, but yeah, this will introduce duplication.

>
> copyapi.h needs more documentation, like what is expected for
> extension developers when using these, what are the arguments, etc.  I
> have added what I had in mind for now.
>
> +typedef char *(*PostpareColumnValue) (CopyFromState cstate, char *string, int m);
>
> CopyReadAttributes and PostpareColumnValue are also callbacks specific
> to text and csv, except that they are used within the per-row
> callbacks.  The same can be said about CopyAttributeOutHeaderFunction.
> It seems to me that it would be less confusing to store pointers to
> them in the routine structures, where the final picture involves not
> having multiple layers of APIs like CopyToCSVStart,
> CopyAttributeOutTextValue, etc.  These *have* to be documented
> properly in copyapi.h, and this is much easier now that cstate stores
> the routine pointers.  That would also make simpler function stacks.
> Note that I have not changed that in the v11 attached.
>
> This business with the extra callbacks required for csv and text is my
> main point of contention, but I'd be OK once the model of the APIs is
> more linear, with everything in Copy{From,To}State.  The changes would
> be rather simple, and I'd be OK to put my hands on it.  Just,
> Sutou-san, would you agree with my last point about these extra
> callbacks?
> --
> Michael

If V7 and V10 have no performance reduction, then I think V6 is also
good with performance, since most of the time goes to CopyToOneRow
and CopyFromOneRow.

I just think we should take the *extendable* into consideration at
the beginning.

-- 
Regards
Junwang Zhao

  109. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-02-02T07:33:19Z

    Hi,

In <ZbyJ60Fd7CHt4m0i@paquier.xyz>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 2 Feb 2024 15:21:31 +0900,
  Michael Paquier <michael@paquier.xyz> wrote:

> I have done a review of v10, see v11 attached which is still WIP, with
> the patches for COPY TO and COPY FROM merged together.  Note that I'm
> thinking to merge them into a single commit.

OK. I don't have a strong opinion for commit unit.

> @@ -74,11 +75,11 @@ typedef struct CopyFormatOptions
>      bool        convert_selectively;    /* do selective binary conversion? */
>      CopyOnErrorChoice on_error; /* what to do when error happened */
>      List       *convert_select; /* list of column names (can be NIL) */
> +    const        CopyToRoutine *to_routine;    /* callback routines for COPY TO */
>  } CopyFormatOptions;
> 
> Adding the routines to the structure for the format options is in my
> opinion incorrect.  The elements of this structure are first processed
> in the option deparsing path, and then we need to use the options to
> guess which routines we need.

This was discussed with Sawada-san a bit before. [1][2]

[1] https://www.postgresql.org/message-id/flat/CAD21AoBmNiWwrspuedgAPgbAqsn7e7NoZYF6gNnYBf%2BgXEk9Mg%40mail.gmail.com#bfd19262d261c67058fdb8d64e6a723c
[2] https://www.postgresql.org/message-id/flat/20240130.144531.1257430878438173740.kou%40clear-code.com#fc55392d77f400fc74e42686fe7e348a

I kept the routines in CopyFormatOptions for custom option
processing. But I should have not cared about it in this
patch set because this patch set doesn't include custom
option processing.

So I'm OK that we move the routines to
Copy{From,To}StateData.

>         This also led to a strange separation with
> ProcessCopyOptionFormatFrom() and ProcessCopyOptionFormatTo() to fit
> the hole in-between.

They also for custom option processing. We don't need to
care about them in this patch set too.

> copyapi.h needs more documentation, like what is expected for
> extension developers when using these, what are the arguments, etc.  I
> have added what I had in mind for now.

Thanks! I'm not good at writing documentation in English...

> +typedef char *(*PostpareColumnValue) (CopyFromState cstate, char *string, int m);
> 
> CopyReadAttributes and PostpareColumnValue are also callbacks specific
> to text and csv, except that they are used within the per-row
> callbacks.  The same can be said about CopyAttributeOutHeaderFunction.
> It seems to me that it would be less confusing to store pointers to
> them in the routine structures, where the final picture involves not
> having multiple layers of APIs like CopyToCSVStart,
> CopyAttributeOutTextValue, etc.  These *have* to be documented
> properly in copyapi.h, and this is much easier now that cstate stores
> the routine pointers.  That would also make simpler function stacks.
> Note that I have not changed that in the v11 attached.
> 
> This business with the extra callbacks required for csv and text is my
> main point of contention, but I'd be OK once the model of the APIs is
> more linear, with everything in Copy{From,To}State.  The changes would
> be rather simple, and I'd be OK to put my hands on it.  Just,
> Sutou-san, would you agree with my last point about these extra
> callbacks?

I'm OK with the approach. But how about adding the extra
callbacks to Copy{From,To}StateData not
Copy{From,To}Routines like CopyToStateData::data_dest_cb and
CopyFromStateData::data_source_cb? They are only needed for
"text" and "csv". So we don't need to add them to
Copy{From,To}Routines to keep required callback minimum.

What is the better next action for us? Do you want to
complete the WIP v11 patch set by yourself (and commit it)?
Or should I take over it?


Thanks,
-- 
kou

  110. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-02-02T07:47:02Z

    Hi,

In <CAEG8a3LxnBwNRPRwvmimDvOkPvYL8pB1+rhLBnxjeddFt3MeNw@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 2 Feb 2024 15:27:15 +0800,
  Junwang Zhao <zhjwpku@gmail.com> wrote:

> I agree CopyToRoutine should be placed into CopyToStateData, but
> why set it after ProcessCopyOptions, the implementation of
> CopyToGetRoutine doesn't make sense if we want to support custom
> format in the future.
> 
> Seems the refactor of v11 only considered performance but not
> *extendable copy format*.

Right.
We focus on performance for now. And then we will focus on
extendability. [1]

[1] https://www.postgresql.org/message-id/flat/20240130.171511.2014195814665030502.kou%40clear-code.com#757a48c273f140081656ec8eb69f502b

> If V7 and V10 have no performance reduction, then I think V6 is also
> good with performance, since most of the time goes to CopyToOneRow
> and CopyFromOneRow.

Don't worry. I'll re-submit changes in the v6 patch set
again after the current patch set that focuses on
performance is merged.

> I just think we should take the *extendable* into consideration at
> the beginning.

Introducing Copy{To,From}Routine is also valuable for
extendability. We can improve extendability later. Let's
focus on only performance for now to introduce
Copy{To,From}Routine.


Thanks,
-- 
kou

  111. Re: Make COPY format extendable: Extract COPY TO format implementations

    Michael Paquier <michael@paquier.xyz> — 2024-02-02T08:04:28Z

    On Fri, Feb 02, 2024 at 04:33:19PM +0900, Sutou Kouhei wrote:
> Hi,
> 
> In <ZbyJ60Fd7CHt4m0i@paquier.xyz>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 2 Feb 2024 15:21:31 +0900,
>   Michael Paquier <michael@paquier.xyz> wrote:
> 
> > I have done a review of v10, see v11 attached which is still WIP, with
> > the patches for COPY TO and COPY FROM merged together.  Note that I'm
> > thinking to merge them into a single commit.
> 
> OK. I don't have a strong opinion for commit unit.
> 
> > @@ -74,11 +75,11 @@ typedef struct CopyFormatOptions
> >      bool        convert_selectively;    /* do selective binary conversion? */
> >      CopyOnErrorChoice on_error; /* what to do when error happened */
> >      List       *convert_select; /* list of column names (can be NIL) */
> > +    const        CopyToRoutine *to_routine;    /* callback routines for COPY TO */
> >  } CopyFormatOptions;
> > 
> > Adding the routines to the structure for the format options is in my
> > opinion incorrect.  The elements of this structure are first processed
> > in the option deparsing path, and then we need to use the options to
> > guess which routines we need.
> 
> This was discussed with Sawada-san a bit before. [1][2]
> 
> [1] https://www.postgresql.org/message-id/flat/CAD21AoBmNiWwrspuedgAPgbAqsn7e7NoZYF6gNnYBf%2BgXEk9Mg%40mail.gmail.com#bfd19262d261c67058fdb8d64e6a723c
> [2] https://www.postgresql.org/message-id/flat/20240130.144531.1257430878438173740.kou%40clear-code.com#fc55392d77f400fc74e42686fe7e348a
> 
> I kept the routines in CopyFormatOptions for custom option
> processing. But I should have not cared about it in this
> patch set because this patch set doesn't include custom
> option processing.

One idea I was considering is whether we should use a special value in
the "format" DefElem, say "custom:$my_custom_format" where it would be
possible to bypass the formay check when processing options and find
the routines after processing all the options.  I'm not wedded to
that, but attaching the routines to the state data is IMO the correct
thing, because this has nothing to do with CopyFormatOptions.

> So I'm OK that we move the routines to
> Copy{From,To}StateData.

Okay.

>> copyapi.h needs more documentation, like what is expected for
>> extension developers when using these, what are the arguments, etc.  I
>> have added what I had in mind for now.
> 
> Thanks! I'm not good at writing documentation in English...

No worries.

> I'm OK with the approach. But how about adding the extra
> callbacks to Copy{From,To}StateData not
> Copy{From,To}Routines like CopyToStateData::data_dest_cb and
> CopyFromStateData::data_source_cb? They are only needed for
> "text" and "csv". So we don't need to add them to
> Copy{From,To}Routines to keep required callback minimum.

And set them in cstate while we are in the Start routine, right?  Hmm.
Why not..  That would get rid of the multiples layers v11 has, which
is my pain point, and we have many fields in cstate that are already
used on a per-format basis.

> What is the better next action for us? Do you want to
> complete the WIP v11 patch set by yourself (and commit it)?
> Or should I take over it?

I was planning to work on that, but wanted to be sure how you felt
about the problem with text and csv first.
--
Michael

  112. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-02-02T08:46:18Z

    Hi,

In <ZbyiDHIrxRgzYT99@paquier.xyz>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 2 Feb 2024 17:04:28 +0900,
  Michael Paquier <michael@paquier.xyz> wrote:

> One idea I was considering is whether we should use a special value in
> the "format" DefElem, say "custom:$my_custom_format" where it would be
> possible to bypass the formay check when processing options and find
> the routines after processing all the options.  I'm not wedded to
> that, but attaching the routines to the state data is IMO the correct
> thing, because this has nothing to do with CopyFormatOptions.

Thanks for sharing your idea.
Let's discuss how to support custom options after we
complete the current performance changes.

>> I'm OK with the approach. But how about adding the extra
>> callbacks to Copy{From,To}StateData not
>> Copy{From,To}Routines like CopyToStateData::data_dest_cb and
>> CopyFromStateData::data_source_cb? They are only needed for
>> "text" and "csv". So we don't need to add them to
>> Copy{From,To}Routines to keep required callback minimum.
> 
> And set them in cstate while we are in the Start routine, right?

I imagined that it's done around the following part:

@@ -1418,6 +1579,9 @@ BeginCopyFrom(ParseState *pstate,
        /* Extract options from the statement node tree */
        ProcessCopyOptions(pstate, &cstate->opts, true /* is_from */ , options);
 
+       /* Set format routine */
+       cstate->routine = CopyFromGetRoutine(cstate->opts);
+
        /* Process the target relation */
        cstate->rel = rel;
 

Example1:

/* Set format routine */
cstate->routine = CopyFromGetRoutine(cstate->opts);
if (!cstate->opts.binary)
    if (cstate->opts.csv_mode)
        cstate->copy_read_attributes = CopyReadAttributesCSV;
    else
        cstate->copy_read_attributes = CopyReadAttributesText;

Example2:

static void
CopyFromSetRoutine(CopyFromState cstate)
{
    if (cstate->opts.csv_mode)
    {
        cstate->routine = &CopyFromRoutineCSV;
        cstate->copy_read_attributes = CopyReadAttributesCSV;
    }
    else if (cstate.binary)
        cstate->routine = &CopyFromRoutineBinary;
    else
    {
        cstate->routine = &CopyFromRoutineText;
        cstate->copy_read_attributes = CopyReadAttributesText;
    }
}

BeginCopyFrom()
{
    /* Set format routine */
    CopyFromSetRoutine(cstate);
}


But I don't object your original approach. If we have the
extra callbacks in Copy{From,To}Routines, I just don't use
them for my custom format extension.

>> What is the better next action for us? Do you want to
>> complete the WIP v11 patch set by yourself (and commit it)?
>> Or should I take over it?
> 
> I was planning to work on that, but wanted to be sure how you felt
> about the problem with text and csv first.

OK.
My opinion is the above. I have an idea how to implement it
but it's not a strong idea. You can choose whichever you like.


Thanks,
-- 
kou

  113. Re: Make COPY format extendable: Extract COPY TO format implementations

    Michael Paquier <michael@paquier.xyz> — 2024-02-05T07:14:08Z

    On Fri, Feb 02, 2024 at 05:46:18PM +0900, Sutou Kouhei wrote:
> Hi,
> 
> In <ZbyiDHIrxRgzYT99@paquier.xyz>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 2 Feb 2024 17:04:28 +0900,
>   Michael Paquier <michael@paquier.xyz> wrote:
> 
> > One idea I was considering is whether we should use a special value in
> > the "format" DefElem, say "custom:$my_custom_format" where it would be
> > possible to bypass the formay check when processing options and find
> > the routines after processing all the options.  I'm not wedded to
> > that, but attaching the routines to the state data is IMO the correct
> > thing, because this has nothing to do with CopyFormatOptions.
> 
> Thanks for sharing your idea.
> Let's discuss how to support custom options after we
> complete the current performance changes.
> 
> >> I'm OK with the approach. But how about adding the extra
> >> callbacks to Copy{From,To}StateData not
> >> Copy{From,To}Routines like CopyToStateData::data_dest_cb and
> >> CopyFromStateData::data_source_cb? They are only needed for
> >> "text" and "csv". So we don't need to add them to
> >> Copy{From,To}Routines to keep required callback minimum.
> > 
> > And set them in cstate while we are in the Start routine, right?
> 
> I imagined that it's done around the following part:
> 
> @@ -1418,6 +1579,9 @@ BeginCopyFrom(ParseState *pstate,
>         /* Extract options from the statement node tree */
>         ProcessCopyOptions(pstate, &cstate->opts, true /* is_from */ , options);
>  
> +       /* Set format routine */
> +       cstate->routine = CopyFromGetRoutine(cstate->opts);
> +
>         /* Process the target relation */
>         cstate->rel = rel;
>  
> 
> Example1:
> 
> /* Set format routine */
> cstate->routine = CopyFromGetRoutine(cstate->opts);
> if (!cstate->opts.binary)
>     if (cstate->opts.csv_mode)
>         cstate->copy_read_attributes = CopyReadAttributesCSV;
>     else
>         cstate->copy_read_attributes = CopyReadAttributesText;
> 
> Example2:
> 
> static void
> CopyFromSetRoutine(CopyFromState cstate)
> {
>     if (cstate->opts.csv_mode)
>     {
>         cstate->routine = &CopyFromRoutineCSV;
>         cstate->copy_read_attributes = CopyReadAttributesCSV;
>     }
>     else if (cstate.binary)
>         cstate->routine = &CopyFromRoutineBinary;
>     else
>     {
>         cstate->routine = &CopyFromRoutineText;
>         cstate->copy_read_attributes = CopyReadAttributesText;
>     }
> }
> 
> BeginCopyFrom()
> {
>     /* Set format routine */
>     CopyFromSetRoutine(cstate);
> }
> 
> 
> But I don't object your original approach. If we have the
> extra callbacks in Copy{From,To}Routines, I just don't use
> them for my custom format extension.
> 
> >> What is the better next action for us? Do you want to
> >> complete the WIP v11 patch set by yourself (and commit it)?
> >> Or should I take over it?
> > 
> > I was planning to work on that, but wanted to be sure how you felt
> > about the problem with text and csv first.
> 
> OK.
> My opinion is the above. I have an idea how to implement it
> but it's not a strong idea. You can choose whichever you like.

So, I've looked at all that today, and finished by applying two
patches as of 2889fd23be56 and 95fb5b49024a to get some of the
weirdness with the workhorse routines out of the way.  Both have added
callbacks assigned in their respective cstate data for text and csv.
As this is called within the OneRow routine, I can live with that.  If
there is an opposition to that, we could just attach it within the
routines.  The CopyAttributeOut routines had a strange argument
layout, actually, the flag for the quotes is required as a header uses
no quotes, but there was little point in the "single arg" case, so
I've removed it.

I am attaching a v12 which is close to what I want it to be, with
much more documentation and comments.  There are two things that I've
changed compared to the previous versions though:
1) I have added a callback to set up the input and output functions
rather than attach that in the Start callback.  These routines are now
called once per argument, where we know that the argument is valid.
The callbacks are in charge of filling the FmgrInfos.  There are some
good reasons behind that:
- No need for plugins to think about how to allocate this data.  v11
and other versions were doing things the wrong way by allocating this
stuff in the wrong memory context as we switch to the COPY context
when we are in the Start routines.
- This avoids attisdropped problems, and we have a long history of
bugs regarding that.  I'm ready to bet that custom formats would get
that wrong.
2) I have backpedaled on the postpare callback, which did not bring
much in clarity IMO while being a CSV-only callback.  Note that we
have in copyfromparse.c more paths that are only for CSV but the past
versions of the patch never cared about that.  This makes the text and
CSV implementations much closer to each other, as a result.

I had mixed feelings about CopySendEndOfRow() being split to
CopyToTextSendEndOfRow() to send the line terminations when sending a
CSV/text row, but I'm OK with that at the end.  v12 is mostly about
moving code around at this point, making it kind of straight-forward
to follow as the code blocks are the same.  I'm still planning to do a
few more measurements, just lacked of time.  Let me know if you have
comments about all that.
--
Michael

  114. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-02-05T09:05:15Z

    Hi,

In <ZcCKwAeFrlOqPBuN@paquier.xyz>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Mon, 5 Feb 2024 16:14:08 +0900,
  Michael Paquier <michael@paquier.xyz> wrote:

> So, I've looked at all that today, and finished by applying two
> patches as of 2889fd23be56 and 95fb5b49024a to get some of the
> weirdness with the workhorse routines out of the way.

Thanks!

> As this is called within the OneRow routine, I can live with that.  If
> there is an opposition to that, we could just attach it within the
> routines.

I don't object the approach.

> I am attaching a v12 which is close to what I want it to be, with
> much more documentation and comments.  There are two things that I've
> changed compared to the previous versions though:
> 1) I have added a callback to set up the input and output functions
> rather than attach that in the Start callback.

I'm OK with this. I just don't use them in Apache Arrow COPY
FORMAT extension.

> - No need for plugins to think about how to allocate this data.  v11
> and other versions were doing things the wrong way by allocating this
> stuff in the wrong memory context as we switch to the COPY context
> when we are in the Start routines.

Oh, sorry. I missed it when I moved them.

> 2) I have backpedaled on the postpare callback, which did not bring
> much in clarity IMO while being a CSV-only callback.  Note that we
> have in copyfromparse.c more paths that are only for CSV but the past
> versions of the patch never cared about that.  This makes the text and
> CSV implementations much closer to each other, as a result.

Ah, sorry. I forgot to eliminate cstate->opts.csv_mode in
CopyReadLineText(). The postpare callback is for
optimization. If it doesn't improve performance, we don't
need to introduce it.

We may want to try eliminating cstate->opts.csv_mode in
CopyReadLineText() for performance. But we don't need to
do this in introducing CopyFromRoutine. We can defer it.

So I don't object removing the postpare callback.

>                                              Let me know if you have
> comments about all that.

Here are some comments for the patch:

+	/*
+	 * Called when COPY FROM is started to set up the input functions
+	 * associated to the relation's attributes writing to.  `fmgr_info` can be

fmgr_info ->
finfo

+	 * optionally filled to provide the catalog information of the input
+	 * function.  `typioparam` can be optinally filled to define the OID of

optinally ->
optionally

+	 * the type to pass to the input function.  `atttypid` is the OID of data
+	 * type used by the relation's attribute.
+	 */
+	void		(*CopyFromInFunc) (Oid atttypid, FmgrInfo *finfo,
+								   Oid *typioparam);

How about passing CopyFromState cstate too like other
callbacks for consistency?

+	/*
+	 * Copy one row to a set of `values` and `nulls` of size tupDesc->natts.
+	 *
+	 * 'econtext' is used to evaluate default expression for each column that
+	 * is either not read from the file or is using the DEFAULT option of COPY

or is ->
or

(I'm not sure...)

+	 * FROM.  It is NULL if no default values are used.
+	 *
+	 * Returns false if there are no more tuples to copy.
+	 */
+	bool		(*CopyFromOneRow) (CopyFromState cstate, ExprContext *econtext,
+								   Datum *values, bool *nulls);

+typedef struct CopyToRoutine
+{
+	/*
+	 * Called when COPY TO is started to set up the output functions
+	 * associated to the relation's attributes reading from.  `fmgr_info` can

fmgr_info ->
finfo

+	 * be optionally filled. `atttypid` is the OID of data type used by the
+	 * relation's attribute.
+	 */
+	void		(*CopyToOutFunc) (Oid atttypid, FmgrInfo *finfo);

How about passing CopyToState cstate too like other
callbacks for consistency?


@@ -200,4 +204,10 @@ extern void ReceiveCopyBinaryHeader(CopyFromState cstate);
 extern int	CopyReadAttributesCSV(CopyFromState cstate);
 extern int	CopyReadAttributesText(CopyFromState cstate);
 
+/* Callbacks for CopyFromRoutine->OneRow */

CopyFromRoutine->OneRow ->
CopyFromRoutine->CopyFromOneRow

+extern bool CopyFromTextOneRow(CopyFromState cstate, ExprContext *econtext,
+							   Datum *values, bool *nulls);
+extern bool CopyFromBinaryOneRow(CopyFromState cstate, ExprContext *econtext,
+								 Datum *values, bool *nulls);
+
 #endif							/* COPYFROM_INTERNAL_H */

+/*
+ * CopyFromTextStart

CopyFromTextStart ->
CopyFromBinaryStart

+ *
+ * Start of COPY FROM for binary format.
+ */
+static void
+CopyFromBinaryStart(CopyFromState cstate, TupleDesc tupDesc)
+{
+	/* Read and verify binary header */
+	ReceiveCopyBinaryHeader(cstate);
+}
+
+/*
+ * CopyFromTextEnd

CopyFromTextEnd ->
CopyFromBinaryEnd

+ *
+ * End of COPY FROM for binary format.
+ */
+static void
+CopyFromBinaryEnd(CopyFromState cstate)
+{
+	/* nothing to do */
+}


diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 91433d439b..d02a7773e3 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -473,6 +473,7 @@ ConvertRowtypeExpr
 CookedConstraint
 CopyDest
 CopyFormatOptions
+CopyFromRoutine
 CopyFromState
 CopyFromStateData
 CopyHeaderChoice
@@ -482,6 +483,7 @@ CopyMultiInsertInfo
 CopyOnErrorChoice
 CopySource
 CopyStmt
+CopyToRoutine
 CopyToState
 CopyToStateData
 Cost

Wow! I didn't know that we need to update typedefs.list when
I add a "typedef struct".


Thanks,
-- 
kou

  115. Re: Make COPY format extendable: Extract COPY TO format implementations

    Andres Freund <andres@anarazel.de> — 2024-02-05T18:21:18Z

    Hi,

Have you benchmarked the performance effects of 2889fd23be5 ? I'd not at all
be surprised if it lead to a measurable performance regression.

I think callbacks for individual attributes is the wrong approach - the
dispatch needs to happen at a higher level, otherwise there are too many
indirect function calls.

Greetings,

Andres Freund

  116. Re: Make COPY format extendable: Extract COPY TO format implementations

    Michael Paquier <michael@paquier.xyz> — 2024-02-05T23:48:55Z

    On Mon, Feb 05, 2024 at 06:05:15PM +0900, Sutou Kouhei wrote:
> In <ZcCKwAeFrlOqPBuN@paquier.xyz>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Mon, 5 Feb 2024 16:14:08 +0900,
>   Michael Paquier <michael@paquier.xyz> wrote:
>> 2) I have backpedaled on the postpare callback, which did not bring
>> much in clarity IMO while being a CSV-only callback.  Note that we
>> have in copyfromparse.c more paths that are only for CSV but the past
>> versions of the patch never cared about that.  This makes the text and
>> CSV implementations much closer to each other, as a result.
> 
> Ah, sorry. I forgot to eliminate cstate->opts.csv_mode in
> CopyReadLineText(). The postpare callback is for
> optimization. If it doesn't improve performance, we don't
> need to introduce it.

No worries.

> We may want to try eliminating cstate->opts.csv_mode in
> CopyReadLineText() for performance. But we don't need to
> do this in introducing CopyFromRoutine. We can defer it.
> 
> So I don't object removing the postpare callback.

Rather related, but there has been a comment from Andres about this
kind of splits a few hours ago, so perhaps this is for the best:
https://www.postgresql.org/message-id/20240205182118.h5rkbnjgujwzuxip%40awork3.anarazel.de

I'll reply to this one in a bit.

>>                                              Let me know if you have
>> comments about all that.
> 
> Here are some comments for the patch:

Thanks.  My head was spinning after reading the diffs more than 20
times :)

> fmgr_info ->
> finfo
> optinally ->
> optionally
> CopyFromRoutine->OneRow ->
> CopyFromRoutine->CopyFromOneRow
> CopyFromTextStart ->
> CopyFromBinaryStart
> CopyFromTextEnd ->
> CopyFromBinaryEnd

Fixed all these.

> How about passing CopyFromState cstate too like other
> callbacks for consistency?

Yes, I was wondering a bit if this can be useful for the custom
formats.

> +	/*
> +	 * Copy one row to a set of `values` and `nulls` of size tupDesc->natts.
> +	 *
> +	 * 'econtext' is used to evaluate default expression for each column that
> +	 * is either not read from the file or is using the DEFAULT option of COPY
> 
> or is ->
> or

"or is" is correct here IMO.

> Wow! I didn't know that we need to update typedefs.list when
> I add a "typedef struct".

That's for the automated indentation.  This is a habit I have when it
comes to work on shaping up patches to avoid weird diffs with pgindent
and new structure names.  It's OK to forget about it :)

Attaching a v13 for now.
--
Michael

  117. Re: Make COPY format extendable: Extract COPY TO format implementations

    Michael Paquier <michael@paquier.xyz> — 2024-02-06T01:01:36Z

    On Mon, Feb 05, 2024 at 10:21:18AM -0800, Andres Freund wrote:
> Have you benchmarked the performance effects of 2889fd23be5 ? I'd not at all
> be surprised if it lead to a measurable performance regression.

Yes, I was looking at runtimes and some profiles around CopyOneRowTo()
to see the effects that this has yesterday.  The principal point of
contention is CopyOneRowTo() where the callback is called once per
attribute, so more attributes stress it more.  The method I've used is
described in [1], where I've used up to 50 int attributes (fixed value
size to limit appendBinaryStringInfo) with 5 million rows, with
shared_buffers large enough that all the data fits in it, while
prewarming the whole.  Postgres runs on a tmpfs, and COPY TO is
redirected to /dev/null.

For reference, I still have some reports lying around (-g attached to
the backend process running the COPY TO queries with text format), so
here you go:
* At 95fb5b49024a:
-   83.04%    11.46%  postgres  postgres            [.] CopyOneRowTo
    - 71.58% CopyOneRowTo
       - 30.37% OutputFunctionCall
          + 27.77% int4out
       + 13.18% CopyAttributeOutText
       + 10.19% appendBinaryStringInfo
         3.76% 0xffffa7096234
         2.78% 0xffffa7096214
       + 2.49% CopySendEndOfRow
         1.21% int4out
         0.83% memcpy@plt
         0.76% 0xffffa7094ba8
         0.75% 0xffffa7094ba4
         0.69% pgstat_progress_update_param
         0.57% enlargeStringInfo
         0.52% 0xffffa7096204
         0.52% 0xffffa7094b8c
    + 11.46% _start
* At 2889fd23be56:
-   83.53%    14.24%  postgres  postgres            [.] CopyOneRowTo
    - 69.29% CopyOneRowTo
       - 29.89% OutputFunctionCall
          + 27.43% int4out
       - 12.89% CopyAttributeOutText
            pg_server_to_any
       + 9.31% appendBinaryStringInfo
         3.68% 0xffffa6940234
       + 2.74% CopySendEndOfRow
         2.43% 0xffffa6940214
         1.36% int4out
         0.74% 0xffffa693eba8
         0.73% pgstat_progress_update_param
         0.65% memcpy@plt
         0.53% MemoryContextReset
    + 14.24% _start

If you have concerns about that, I'm OK to revert, I'm not wedded to
this level of control.  Note that I've actually seen *better*
runtimes.

[1]: https://www.postgresql.org/message-id/Zbr6piWuVHDtFFOl@paquier.xyz

> I think callbacks for individual attributes is the wrong approach - the
> dispatch needs to happen at a higher level, otherwise there are too many
> indirect function calls.

Hmm.  Do you have concerns about v13 posted on [2] then?  If yes, then
I'd assume that this shuts down the whole thread or that it needs a
completely different approach, because we will multiply indirect
function calls that can control how data is generated for each row,
which is the original case that Sutou-san wanted to tackle.  There
could be many indirect calls with custom callbacks that control how
things should be processed at row-level, and COPY likes doing work
with loads of data.  The End, Start and In/OutFunc callbacks are
called only once per query, so these don't matter AFAIU.

[2]: https://www.postgresql.org/message-id/ZcFz59nJjQNjwgX0@paquier.xyz
--
Michael

  118. Re: Make COPY format extendable: Extract COPY TO format implementations

    Andres Freund <andres@anarazel.de> — 2024-02-06T01:41:25Z

    Hi,

On 2024-02-06 10:01:36 +0900, Michael Paquier wrote:
> On Mon, Feb 05, 2024 at 10:21:18AM -0800, Andres Freund wrote:
> > Have you benchmarked the performance effects of 2889fd23be5 ? I'd not at all
> > be surprised if it lead to a measurable performance regression.
>
> Yes, I was looking at runtimes and some profiles around CopyOneRowTo()
> to see the effects that this has yesterday.  The principal point of
> contention is CopyOneRowTo() where the callback is called once per
> attribute, so more attributes stress it more.

Right.


> If you have concerns about that, I'm OK to revert, I'm not wedded to
> this level of control.  Note that I've actually seen *better*
> runtimes.

I'm somewhat worried that handling the different formats at that level will
make it harder to improve copy performance - it's quite attrociously slow
right now. The more we reduce the per-row/field overhead, the more the
dispatch overhead will matter.



> [1]: https://www.postgresql.org/message-id/Zbr6piWuVHDtFFOl@paquier.xyz
>
> > I think callbacks for individual attributes is the wrong approach - the
> > dispatch needs to happen at a higher level, otherwise there are too many
> > indirect function calls.
>
> Hmm.  Do you have concerns about v13 posted on [2] then?

As is I'm indeed not a fan. It imo doesn't make sense to have an indirect
dispatch for *both* ->copy_attribute_out *and* ->CopyToOneRow. After all, when
in ->CopyToOneRow for text, we could know that we need to call
CopyAttributeOutText etc.


> If yes, then I'd assume that this shuts down the whole thread or that it
> needs a completely different approach, because we will multiply indirect
> function calls that can control how data is generated for each row, which is
> the original case that Sutou-san wanted to tackle.

I think it could be rescued fairly easily - remove the dispatch via
->copy_attribute_out().  To avoid duplicating code you could use a static
inline function that's used with constant arguments by both csv and text mode.

I think it might also be worth ensuring that future patches can move branches
like
	if (cstate->encoding_embeds_ascii)
	if (cstate->need_transcoding)
into the choice of per-row callback.


> The End, Start and In/OutFunc callbacks are called only once per query, so
> these don't matter AFAIU.

Right.

Greetings,

Andres Freund

  119. Re: Make COPY format extendable: Extract COPY TO format implementations

    Michael Paquier <michael@paquier.xyz> — 2024-02-06T02:41:06Z

    On Mon, Feb 05, 2024 at 05:41:25PM -0800, Andres Freund wrote:
> On 2024-02-06 10:01:36 +0900, Michael Paquier wrote:
>> If you have concerns about that, I'm OK to revert, I'm not wedded to
>> this level of control.  Note that I've actually seen *better*
>> runtimes.
> 
> I'm somewhat worried that handling the different formats at that level will
> make it harder to improve copy performance - it's quite attrociously slow
> right now. The more we reduce the per-row/field overhead, the more the
> dispatch overhead will matter.

Yep.  That's the hard part when it comes to design these callbacks.
We don't want something too high level because this leads to more code
duplication churns when someone wants to plug in its own routine set,
and we don't want to be at a too low level because of the indirect
calls as you said.  I'd like to think that the current CopyFromOneRow
offers a good balance here, avoiding the "if" branch with the binary
and non-binary paths.

>> Hmm.  Do you have concerns about v13 posted on [2] then?
> 
> As is I'm indeed not a fan. It imo doesn't make sense to have an indirect
> dispatch for *both* ->copy_attribute_out *and* ->CopyToOneRow. After all, when
> in ->CopyToOneRow for text, we could know that we need to call
> CopyAttributeOutText etc.

Right.

>> If yes, then I'd assume that this shuts down the whole thread or that it
>> needs a completely different approach, because we will multiply indirect
>> function calls that can control how data is generated for each row, which is
>> the original case that Sutou-san wanted to tackle.
> 
> I think it could be rescued fairly easily - remove the dispatch via
> ->copy_attribute_out().  To avoid duplicating code you could use a static
> inline function that's used with constant arguments by both csv and text mode.

Hmm.  So you basically mean to tweak the beginning of
CopyToTextOneRow() and CopyToTextStart() so as copy_attribute_out is
saved in a local variable outside of cstate and we'd save the "if"
checked for each attribute.  If I got that right, it would mean
something like the v13-0002 attached, on top of the v13-0001 of
upthread.  Is that what you meant?

> I think it might also be worth ensuring that future patches can move branches
> like
> 	if (cstate->encoding_embeds_ascii)
> 	if (cstate->need_transcoding)
> into the choice of per-row callback.

Yeah, I'm still not sure how much we should split CopyToStateData in
the initial patch set.  I'd like to think that the best result would
be to have in the state data an opaque (void *) that points to a
structure that can be set for each format, so as there is a clean
split between which variable gets set and used where (same remark
applies to COPY FROM with its raw_fields, raw_fields, for example).
--
Michael

  120. Re: Make COPY format extendable: Extract COPY TO format implementations

    Andres Freund <andres@anarazel.de> — 2024-02-06T05:46:42Z

    Hi,

On 2024-02-06 11:41:06 +0900, Michael Paquier wrote:
> On Mon, Feb 05, 2024 at 05:41:25PM -0800, Andres Freund wrote:
> > On 2024-02-06 10:01:36 +0900, Michael Paquier wrote:
> >> If you have concerns about that, I'm OK to revert, I'm not wedded to
> >> this level of control.  Note that I've actually seen *better*
> >> runtimes.
> > 
> > I'm somewhat worried that handling the different formats at that level will
> > make it harder to improve copy performance - it's quite attrociously slow
> > right now. The more we reduce the per-row/field overhead, the more the
> > dispatch overhead will matter.
> 
> Yep.  That's the hard part when it comes to design these callbacks.
> We don't want something too high level because this leads to more code
> duplication churns when someone wants to plug in its own routine set,
> and we don't want to be at a too low level because of the indirect
> calls as you said.  I'd like to think that the current CopyFromOneRow
> offers a good balance here, avoiding the "if" branch with the binary
> and non-binary paths.

One way to address code duplication is to use static inline helper functions
that do a lot of the work in a generic fashion, but where the compiler can
optimize the branches away, because it can do constant folding.


> >> If yes, then I'd assume that this shuts down the whole thread or that it
> >> needs a completely different approach, because we will multiply indirect
> >> function calls that can control how data is generated for each row, which is
> >> the original case that Sutou-san wanted to tackle.
> > 
> > I think it could be rescued fairly easily - remove the dispatch via
> > ->copy_attribute_out().  To avoid duplicating code you could use a static
> > inline function that's used with constant arguments by both csv and text mode.
> 
> Hmm.  So you basically mean to tweak the beginning of
> CopyToTextOneRow() and CopyToTextStart() so as copy_attribute_out is
> saved in a local variable outside of cstate and we'd save the "if"
> checked for each attribute.  If I got that right, it would mean
> something like the v13-0002 attached, on top of the v13-0001 of
> upthread.  Is that what you meant?

No - what I mean is that it doesn't make sense to have copy_attribute_out(),
as e.g. CopyToTextOneRow() already knows that it's dealing with text, so it
can directly call the right function. That does require splitting a bit more
between csv and text output, but I think that can be done without much
duplication.

Greetings,

Andres Freund

  121. Re: Make COPY format extendable: Extract COPY TO format implementations

    Michael Paquier <michael@paquier.xyz> — 2024-02-06T06:11:05Z

    On Mon, Feb 05, 2024 at 09:46:42PM -0800, Andres Freund wrote:
> No - what I mean is that it doesn't make sense to have copy_attribute_out(),
> as e.g. CopyToTextOneRow() already knows that it's dealing with text, so it
> can directly call the right function. That does require splitting a bit more
> between csv and text output, but I think that can be done without much
> duplication.

I am not sure to understand here.  In what is that different from
reverting 2889fd23be56 then mark CopyAttributeOutCSV and
CopyAttributeOutText as static inline?  Or you mean to merge
CopyAttributeOutText and CopyAttributeOutCSV together into a single
inlined function, reducing a bit code readability?  Both routines have
their own roadmap for encoding_embeds_ascii with quoting and escaping,
so keeping them separated looks kinda cleaner here.
--
Michael

  122. Re: Make COPY format extendable: Extract COPY TO format implementations

    Andres Freund <andres@anarazel.de> — 2024-02-06T23:33:36Z

    Hi,

On 2024-02-06 15:11:05 +0900, Michael Paquier wrote:
> On Mon, Feb 05, 2024 at 09:46:42PM -0800, Andres Freund wrote:
> > No - what I mean is that it doesn't make sense to have copy_attribute_out(),
> > as e.g. CopyToTextOneRow() already knows that it's dealing with text, so it
> > can directly call the right function. That does require splitting a bit more
> > between csv and text output, but I think that can be done without much
> > duplication.
> 
> I am not sure to understand here.  In what is that different from
> reverting 2889fd23be56 then mark CopyAttributeOutCSV and
> CopyAttributeOutText as static inline?

Well, you can't just do that, because there's only one caller, namely
CopyToTextOneRow(). What I am trying to suggest is something like the
attached, just a quick hacky POC. Namely to split out CSV support from
CopyToTextOneRow() by introducing CopyToCSVOneRow(), and to avoid code
duplication by moving the code into a new CopyToTextLikeOneRow().

I named it CopyToTextLike* here, because it seems confusing that some Text*
are used for both CSV and text and others are actually just for text. But if
were to go for that, we should go further.


To test the performnce effects I chose to remove the pointless encoding
"check" we're discussing in the other thread, as it makes it harder to see the
time differences due to the per-attribute code.  I did three runs of pgbench
-t of [1] and chose the fastest result for each.


With turbo mode and power saving disabled:

                          Avg Time
HEAD                       995.349
Remove Encoding Check      870.793
v13-0001                   869.678
Remove out callback        839.508

Greetings,

Andres Freund

[1] COPY (SELECT 1::int2,2::int2,3::int2,4::int2,5::int2,6::int2,7::int2,8::int2,9::int2,10::int2,11::int2,12::int2,13::int2,14::int2,15::int2,16::int2,17::int2,18::int2,19::int2,20::int2, generate_series(1, 1000000::int4)) TO '/dev/null';

  123. Re: Make COPY format extendable: Extract COPY TO format implementations

    Michael Paquier <michael@paquier.xyz> — 2024-02-07T04:33:18Z

    On Tue, Feb 06, 2024 at 03:33:36PM -0800, Andres Freund wrote:
> Well, you can't just do that, because there's only one caller, namely
> CopyToTextOneRow(). What I am trying to suggest is something like the
> attached, just a quick hacky POC. Namely to split out CSV support from
> CopyToTextOneRow() by introducing CopyToCSVOneRow(), and to avoid code
> duplication by moving the code into a new CopyToTextLikeOneRow().

Ah, OK.  Got it now.

> I named it CopyToTextLike* here, because it seems confusing that some Text*
> are used for both CSV and text and others are actually just for text. But if
> were to go for that, we should go further.

This can always be argued later.

> To test the performnce effects I chose to remove the pointless encoding
> "check" we're discussing in the other thread, as it makes it harder to see the
> time differences due to the per-attribute code.  I did three runs of pgbench
> -t of [1] and chose the fastest result for each.
> 
> With turbo mode and power saving disabled:
>                           Avg Time
> HEAD                       995.349
> Remove Encoding Check      870.793
> v13-0001                   869.678
> Remove out callback        839.508

Hmm.  That explains why I was not seeing any differences with this
callback then.  It seems to me that the order of actions to take is
clear, like:
- Revert 2889fd23be56 to keep a clean state of the tree, now done with
1aa8324b81fa.
- Dive into the strlen() issue, as it really looks like this can
create more simplifications for the patch discussed on this thread
with COPY TO.
- Revisit what we have here, looking at more profiles to see how HEAD
an v13 compare.  It looks like we are on a good path, but let's tackle
things one step at a time.
--
Michael

  124. Re: Make COPY format extendable: Extract COPY TO format implementations

    Michael Paquier <michael@paquier.xyz> — 2024-02-09T00:54:53Z

    On Thu, Feb 01, 2024 at 10:57:58AM +0900, Michael Paquier wrote:
> CREATE EXTENSION blackhole_am;

One thing I have forgotten here is to provide a copy of this AM for
future references, so here you go with a blackhole_am.tar.gz attached.
--
Michael

  125. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-02-09T04:19:50Z

    Hi,

In <ZcMIDgkdSrz5ibvf@paquier.xyz>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Wed, 7 Feb 2024 13:33:18 +0900,
  Michael Paquier <michael@paquier.xyz> wrote:

> Hmm.  That explains why I was not seeing any differences with this
> callback then.  It seems to me that the order of actions to take is
> clear, like:
> - Revert 2889fd23be56 to keep a clean state of the tree, now done with
> 1aa8324b81fa.

Done.

> - Dive into the strlen() issue, as it really looks like this can
> create more simplifications for the patch discussed on this thread
> with COPY TO.

Done: b619852086ed2b5df76631f5678f60d3bebd3745

> - Revisit what we have here, looking at more profiles to see how HEAD
> an v13 compare.  It looks like we are on a good path, but let's tackle
> things one step at a time.

Are you already working on this? Do you want me to write the
next patch based on the current master?


Thanks,
-- 
kou

  126. Re: Make COPY format extendable: Extract COPY TO format implementations

    Michael Paquier <michael@paquier.xyz> — 2024-02-09T04:21:34Z

    On Wed, Feb 07, 2024 at 01:33:18PM +0900, Michael Paquier wrote:
> Hmm.  That explains why I was not seeing any differences with this
> callback then.  It seems to me that the order of actions to take is
> clear, like:
> - Revert 2889fd23be56 to keep a clean state of the tree, now done with
> 1aa8324b81fa.
> - Dive into the strlen() issue, as it really looks like this can
> create more simplifications for the patch discussed on this thread
> with COPY TO.

This has been done this morning with b619852086ed.

> - Revisit what we have here, looking at more profiles to see how HEAD
> an v13 compare.  It looks like we are on a good path, but let's tackle
> things one step at a time.

And attached is a v14 that's rebased on HEAD.  While on it, I've
looked at more profiles and did more runtime checks.

Some runtimes, in (ms), average of 15 runs, 30 int attributes on 5M
rows as mentioned above:
COPY FROM  text   binary
HEAD       6066   7110
v14        6087   7105
COPY TO    text   binary
HEAD       6591   10161
v14        6508   10189

And here are some profiles, where I'm not seeing an impact at
row-level with the addition of the callbacks:
COPY FROM, text, master:
-   66.59%    16.10%  postgres  postgres            [.] NextCopyFrom                                                                                                       ▒    - 50.50% NextCopyFrom
       - 30.75% NextCopyFromRawFields
          + 15.93% CopyReadLine
            13.73% CopyReadAttributesText
       - 19.43% InputFunctionCallSafe
          + 13.49% int4in
            0.77% pg_strtoint32_safe
    + 16.10% _start
COPY FROM, text, v14:
-   66.42%     0.74%  postgres  postgres            [.] NextCopyFrom
    - 65.67% NextCopyFrom
       - 65.51% CopyFromTextOneRow
          - 30.25% NextCopyFromRawFields
             + 16.14% CopyReadLine
               13.40% CopyReadAttributesText
          - 18.96% InputFunctionCallSafe
             + 13.15% int4in
               0.70% pg_strtoint32_safe
    + 0.74% _start

COPY TO, binary, master
-   90.32%     7.14%  postgres  postgres            [.] CopyOneRowTo
    - 83.18% CopyOneRowTo
       + 60.30% SendFunctionCall
       + 10.99% appendBinaryStringInfo
       + 3.67% MemoryContextReset
       + 2.89% CopySendEndOfRow
         0.89% memcpy@plt
         0.66% 0xffffa052db5c
         0.62% enlargeStringInfo
         0.56% pgstat_progress_update_param
    + 7.14% _start
COPY TO, binary, v14
-   90.96%     0.21%  postgres  postgres            [.] CopyOneRowTo
    - 90.75% CopyOneRowTo
       - 81.86% CopyToBinaryOneRow
          + 59.17% SendFunctionCall
          + 10.56% appendBinaryStringInfo
            1.10% enlargeStringInfo
            0.59% int4send
            0.57% memcpy@plt
       + 3.68% MemoryContextReset
       + 2.83% CopySendEndOfRow
         1.13% appendBinaryStringInfo
         0.58% SendFunctionCall
         0.58% pgstat_progress_update_param

Are there any comments about this v14?  Sutou-san?

A next step I think we could take is to split the binary-only and the
text/csv-only data in each cstate into their own structure to make the
structure, with an opaque pointer that custom formats could use, but a
lot of fields are shared as well.  This patch is already complicated
enough IMO, so I'm OK to leave it out for the moment, and focus on
making this infra pluggable as a next step.
--
Michael

  127. Re: Make COPY format extendable: Extract COPY TO format implementations

    Michael Paquier <michael@paquier.xyz> — 2024-02-09T04:40:43Z

    On Fri, Feb 09, 2024 at 01:19:50PM +0900, Sutou Kouhei wrote:
> Are you already working on this? Do you want me to write the
> next patch based on the current master?

No need for a new patch, thanks.  I've spent some time today doing a
rebase and measuring the whole, without seeing a degradation with what
should be the worst cases for COPY TO and FROM:
https://www.postgresql.org/message-id/ZcWoTr1N0GELFA9E%40paquier.xyz
--
Michael

  128. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-02-09T07:32:05Z

    Hi,

In <ZcWoTr1N0GELFA9E@paquier.xyz>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 9 Feb 2024 13:21:34 +0900,
  Michael Paquier <michael@paquier.xyz> wrote:

>> - Revisit what we have here, looking at more profiles to see how HEAD
>> an v13 compare.  It looks like we are on a good path, but let's tackle
>> things one step at a time.
> 
> And attached is a v14 that's rebased on HEAD.

Thanks!

> A next step I think we could take is to split the binary-only and the
> text/csv-only data in each cstate into their own structure to make the
> structure, with an opaque pointer that custom formats could use, but a
> lot of fields are shared as well.

It'll make COPY code base cleaner but it may decrease
performance. How about just adding an opaque pointer to each
cstate as the next step and then try the split?

My suggestion:
1. Introduce Copy{To,From}Routine
   (We can do it based on the v14 patch.)
2. Add an opaque pointer to Copy{To,From}Routine
   (This must not have performance impact.)
3.a. Split format specific data to the opaque space
3.b. Add support for registering custom format handler by
     creating a function
4. ...

>                                    This patch is already complicated
> enough IMO, so I'm OK to leave it out for the moment, and focus on
> making this infra pluggable as a next step.

I agree with you.

> Are there any comments about this v14?  Sutou-san?

Here are my comments:


+	/* Set read attribute callback */
+	if (cstate->opts.csv_mode)
+		cstate->copy_read_attributes = CopyReadAttributesCSV;
+	else
+		cstate->copy_read_attributes = CopyReadAttributesText;

I think that we should not use this approach for
performance. We need to use "static inline" and constant
argument instead something like the attached
remove-copy-read-attributes.diff.

We have similar codes for
CopyReadLine()/CopyReadLineText(). The attached
remove-copy-read-attributes-and-optimize-copy-read-line.diff
also applies the same optimization to
CopyReadLine()/CopyReadLineText().

I hope that this improved performance of COPY FROM.

+/*
+ * Routines assigned to each format.
++

Garbage "+"

+ * CSV and text share the same implementation, at the exception of the
+ * copy_read_attributes callback.
+ */


+/*
+ * CopyToTextOneRow
+ *
+ * Process one row for text/CSV format.
+ */
+static void
+CopyToTextOneRow(CopyToState cstate,
+				 TupleTableSlot *slot)
+{
...
+			if (cstate->opts.csv_mode)
+				CopyAttributeOutCSV(cstate, string,
+									cstate->opts.force_quote_flags[attnum - 1]);
+			else
+				CopyAttributeOutText(cstate, string);
...

How about use "static inline" and constant argument approach
here too?

static inline void
CopyToTextBasedOneRow(CopyToState cstate,
					  TupleTableSlot *slot,
					  bool csv_mode)
{
...
			if (cstate->opts.csv_mode)
				CopyAttributeOutCSV(cstate, string,
									cstate->opts.force_quote_flags[attnum - 1]);
			else
				CopyAttributeOutText(cstate, string);
...
}

static void
CopyToTextOneRow(CopyToState cstate,
				 TupleTableSlot *slot,
				 bool csv_mode)
{
	CopyToTextBasedOneRow(cstate, slot, false);
}

static void
CopyToCSVOneRow(CopyToState cstate,
				TupleTableSlot *slot,
				bool csv_mode)
{
	CopyToTextBasedOneRow(cstate, slot, true);
}

static const CopyToRoutine CopyCSVRoutineText = {
	...
	.CopyToOneRow = CopyToCSVOneRow,
	...
};


Thanks,
-- 
kou

  129. Re: Make COPY format extendable: Extract COPY TO format implementations

    Michael Paquier <michael@paquier.xyz> — 2024-02-09T08:25:50Z

    On Fri, Feb 09, 2024 at 04:32:05PM +0900, Sutou Kouhei wrote:
> In <ZcWoTr1N0GELFA9E@paquier.xyz>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 9 Feb 2024 13:21:34 +0900,
>   Michael Paquier <michael@paquier.xyz> wrote:
>> A next step I think we could take is to split the binary-only and the
>> text/csv-only data in each cstate into their own structure to make the
>> structure, with an opaque pointer that custom formats could use, but a
>> lot of fields are shared as well.
> 
> It'll make COPY code base cleaner but it may decrease
> performance.

Perhaps, but I'm not sure, TBH.  But perhaps others can comment on
this point.  This surely needs to be studied closely.

> My suggestion:
> 1. Introduce Copy{To,From}Routine
>    (We can do it based on the v14 patch.)
> 2. Add an opaque pointer to Copy{To,From}Routine
>    (This must not have performance impact.)
> 3.a. Split format specific data to the opaque space
> 3.b. Add support for registering custom format handler by
>      creating a function
> 4. ...

4. is going to need 3.  At this point 3.b sounds like the main thing
to tackle first if we want to get something usable for the end-user
into this release, at least.  Still 2 is important for pluggability
as we pass the cstates across all the routines and custom formats want
to save their own data, so this split sounds OK.  I am not sure how
much of 3.a we really need to do for the in-core formats.

> I think that we should not use this approach for
> performance. We need to use "static inline" and constant
> argument instead something like the attached
> remove-copy-read-attributes.diff.

FWIW, using inlining did not show any performance change here.
Perhaps that's only because this is called in the COPY FROM path once
per row (even for the case of using 1 attribute with blackhole_am).
--
Michael

  130. Re: Make COPY format extendable: Extract COPY TO format implementations

    Andres Freund <andres@anarazel.de> — 2024-02-09T19:27:05Z

    Hi,

On 2024-02-09 13:21:34 +0900, Michael Paquier wrote:
> +static void
> +CopyFromTextInFunc(CopyFromState cstate, Oid atttypid,
> +				   FmgrInfo *finfo, Oid *typioparam)
> +{
> +	Oid			func_oid;
> +
> +	getTypeInputInfo(atttypid, &func_oid, typioparam);
> +	fmgr_info(func_oid, finfo);
> +}

FWIW, we should really change the copy code to initialize FunctionCallInfoData
instead of re-initializing that on every call, realy makes a difference
performance wise.


> +/*
> + * CopyFromTextStart
> + *
> + * Start of COPY FROM for text/CSV format.
> + */
> +static void
> +CopyFromTextStart(CopyFromState cstate, TupleDesc tupDesc)
> +{
> +	AttrNumber	attr_count;
> +
> +	/*
> +	 * If encoding conversion is needed, we need another buffer to hold the
> +	 * converted input data.  Otherwise, we can just point input_buf to the
> +	 * same buffer as raw_buf.
> +	 */
> +	if (cstate->need_transcoding)
> +	{
> +		cstate->input_buf = (char *) palloc(INPUT_BUF_SIZE + 1);
> +		cstate->input_buf_index = cstate->input_buf_len = 0;
> +	}
> +	else
> +		cstate->input_buf = cstate->raw_buf;
> +	cstate->input_reached_eof = false;
> +
> +	initStringInfo(&cstate->line_buf);

Seems kinda odd that we have a supposedly extensible API that then stores all
this stuff in the non-extensible CopyFromState.


> +	/* create workspace for CopyReadAttributes results */
> +	attr_count = list_length(cstate->attnumlist);
> +	cstate->max_fields = attr_count;

Why is this here? This seems like generic code, not text format specific.


> +	cstate->raw_fields = (char **) palloc(attr_count * sizeof(char *));
> +	/* Set read attribute callback */
> +	if (cstate->opts.csv_mode)
> +		cstate->copy_read_attributes = CopyReadAttributesCSV;
> +	else
> +		cstate->copy_read_attributes = CopyReadAttributesText;
> +}

Isn't this precisely repeating the mistake of 2889fd23be56?

And, why is this done here? Shouldn't this decision have been made prior to
even calling CopyFromTextStart()?

> +/*
> + * CopyFromTextOneRow
> + *
> + * Copy one row to a set of `values` and `nulls` for the text and CSV
> + * formats.
> + */

I'm very doubtful it's a good idea to combine text and CSV here. They have
basically no shared parsing code, so what's the point in sending them through
one input routine?


> +bool
> +CopyFromTextOneRow(CopyFromState cstate,
> +				   ExprContext *econtext,
> +				   Datum *values,
> +				   bool *nulls)
> +{
> +	TupleDesc	tupDesc;
> +	AttrNumber	attr_count;
> +	FmgrInfo   *in_functions = cstate->in_functions;
> +	Oid		   *typioparams = cstate->typioparams;
> +	ExprState **defexprs = cstate->defexprs;
> +	char	  **field_strings;
> +	ListCell   *cur;
> +	int			fldct;
> +	int			fieldno;
> +	char	   *string;
> +
> +	tupDesc = RelationGetDescr(cstate->rel);
> +	attr_count = list_length(cstate->attnumlist);
> +
> +	/* read raw fields in the next line */
> +	if (!NextCopyFromRawFields(cstate, &field_strings, &fldct))
> +		return false;
> +
> +	/* check for overflowing fields */
> +	if (attr_count > 0 && fldct > attr_count)
> +		ereport(ERROR,
> +				(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
> +				 errmsg("extra data after last expected column")));

It bothers me that we look to be ending up with different error handling
across the various output formats, particularly if they're ending up in
extensions. That'll make it harder to evolve this code in the future.


> +	fieldno = 0;
> +
> +	/* Loop to read the user attributes on the line. */
> +	foreach(cur, cstate->attnumlist)
> +	{
> +		int			attnum = lfirst_int(cur);
> +		int			m = attnum - 1;
> +		Form_pg_attribute att = TupleDescAttr(tupDesc, m);
> +
> +		if (fieldno >= fldct)
> +			ereport(ERROR,
> +					(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
> +					 errmsg("missing data for column \"%s\"",
> +							NameStr(att->attname))));
> +		string = field_strings[fieldno++];
> +
> +		if (cstate->convert_select_flags &&
> +			!cstate->convert_select_flags[m])
> +		{
> +			/* ignore input field, leaving column as NULL */
> +			continue;
> +		}
> +
> +		cstate->cur_attname = NameStr(att->attname);
> +		cstate->cur_attval = string;
> +
> +		if (cstate->opts.csv_mode)
> +		{

More unfortunate intermingling of multiple formats in a single routine.


> +
> +		if (cstate->defaults[m])
> +		{
> +			/*
> +			 * The caller must supply econtext and have switched into the
> +			 * per-tuple memory context in it.
> +			 */
> +			Assert(econtext != NULL);
> +			Assert(CurrentMemoryContext == econtext->ecxt_per_tuple_memory);
> +
> +			values[m] = ExecEvalExpr(defexprs[m], econtext, &nulls[m]);
> +		}

I don't think it's good that we end up with this code in different copy
implementations.

Greetings,

Andres Freund

  131. Re: Make COPY format extendable: Extract COPY TO format implementations

    Michael Paquier <michael@paquier.xyz> — 2024-02-10T01:02:25Z

    On Fri, Feb 09, 2024 at 11:27:05AM -0800, Andres Freund wrote:
> On 2024-02-09 13:21:34 +0900, Michael Paquier wrote:
>> +static void
>> +CopyFromTextInFunc(CopyFromState cstate, Oid atttypid,
>> +				   FmgrInfo *finfo, Oid *typioparam)
>> +{
>> +	Oid			func_oid;
>> +
>> +	getTypeInputInfo(atttypid, &func_oid, typioparam);
>> +	fmgr_info(func_oid, finfo);
>> +}
> 
> FWIW, we should really change the copy code to initialize FunctionCallInfoData
> instead of re-initializing that on every call, realy makes a difference
> performance wise.

You mean to initialize once its memory and let the internal routines
call InitFunctionCallInfoData for each attribute.  Sounds like a good
idea, doing that for HEAD before the main patch.  More impact with
more attributes.

>> +/*
>> + * CopyFromTextStart
>> + *
>> + * Start of COPY FROM for text/CSV format.
>> + */
>> +static void
>> +CopyFromTextStart(CopyFromState cstate, TupleDesc tupDesc)
>> +{
>> +	AttrNumber	attr_count;
>> +
>> +	/*
>> +	 * If encoding conversion is needed, we need another buffer to hold the
>> +	 * converted input data.  Otherwise, we can just point input_buf to the
>> +	 * same buffer as raw_buf.
>> +	 */
>> +	if (cstate->need_transcoding)
>> +	{
>> +		cstate->input_buf = (char *) palloc(INPUT_BUF_SIZE + 1);
>> +		cstate->input_buf_index = cstate->input_buf_len = 0;
>> +	}
>> +	else
>> +		cstate->input_buf = cstate->raw_buf;
>> +	cstate->input_reached_eof = false;
>> +
>> +	initStringInfo(&cstate->line_buf);
> 
> Seems kinda odd that we have a supposedly extensible API that then stores all
> this stuff in the non-extensible CopyFromState.

That relates to the introduction of the the opaque pointer mentioned 
upthread to point to a per-format structure, where we'd store data
specific to each format.

>> +	/* create workspace for CopyReadAttributes results */
>> +	attr_count = list_length(cstate->attnumlist);
>> +	cstate->max_fields = attr_count;
> 
> Why is this here? This seems like generic code, not text format specific.

We don't care about that for binary.

>> +/*
>> + * CopyFromTextOneRow
>> + *
>> + * Copy one row to a set of `values` and `nulls` for the text and CSV
>> + * formats.
>> + */
> 
> I'm very doubtful it's a good idea to combine text and CSV here. They have
> basically no shared parsing code, so what's the point in sending them through
> one input routine?

The code shared between text and csv involves a path called once per
attribute.  TBH, I am not sure how much of the NULL handling should be
put outside the per-row routine as these options are embedded in the
core options.  So I don't have a better idea on this one than what's
proposed here if we cannot dispatch the routine calls once per
attribute.

>> +	/* read raw fields in the next line */
>> +	if (!NextCopyFromRawFields(cstate, &field_strings, &fldct))
>> +		return false;
>> +
>> +	/* check for overflowing fields */
>> +	if (attr_count > 0 && fldct > attr_count)
>> +		ereport(ERROR,
>> +				(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
>> +				 errmsg("extra data after last expected column")));
> 
> It bothers me that we look to be ending up with different error handling
> across the various output formats, particularly if they're ending up in
> extensions. That'll make it harder to evolve this code in the future.

But different formats may have different requirements, including the
number of attributes detected vs expected.  That was not really
nothing me.

>> +		if (cstate->opts.csv_mode)
>> +		{
> 
> More unfortunate intermingling of multiple formats in a single
> routine.

Similar answer as a few paragraphs above.  Sutou-san was suggesting to
use an internal routine with fixed arguments instead, which would be
enough at the end with some inline instructions?

>> +
>> +		if (cstate->defaults[m])
>> +		{
>> +			/*
>> +			 * The caller must supply econtext and have switched into the
>> +			 * per-tuple memory context in it.
>> +			 */
>> +			Assert(econtext != NULL);
>> +			Assert(CurrentMemoryContext == econtext->ecxt_per_tuple_memory);
>> +
>> +			values[m] = ExecEvalExpr(defexprs[m], econtext, &nulls[m]);
>> +		}
> 
> I don't think it's good that we end up with this code in different copy
> implementations.

Yeah, still we don't care about that for binary.
--
Michael

  132. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-02-13T08:33:40Z

    Hi,

In <20240209192705.5qdilvviq3py2voq@awork3.anarazel.de>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 9 Feb 2024 11:27:05 -0800,
  Andres Freund <andres@anarazel.de> wrote:

>> +static void
>> +CopyFromTextInFunc(CopyFromState cstate, Oid atttypid,
>> +				   FmgrInfo *finfo, Oid *typioparam)
>> +{
>> +	Oid			func_oid;
>> +
>> +	getTypeInputInfo(atttypid, &func_oid, typioparam);
>> +	fmgr_info(func_oid, finfo);
>> +}
> 
> FWIW, we should really change the copy code to initialize FunctionCallInfoData
> instead of re-initializing that on every call, realy makes a difference
> performance wise.

How about the attached patch approach? If it's a desired
approach, I can also write a separated patch for COPY TO.

>> +	cstate->raw_fields = (char **) palloc(attr_count * sizeof(char *));
>> +	/* Set read attribute callback */
>> +	if (cstate->opts.csv_mode)
>> +		cstate->copy_read_attributes = CopyReadAttributesCSV;
>> +	else
>> +		cstate->copy_read_attributes = CopyReadAttributesText;
>> +}
> 
> Isn't this precisely repeating the mistake of 2889fd23be56?

What do you think about the approach in my previous mail's
attachments?
https://www.postgresql.org/message-id/flat/20240209.163205.704848659612151781.kou%40clear-code.com#dbb1f8d7f2f0e8fe3c7e37a757fcfc54

If it's a desired approach, I can prepare a v15 patch set
based on the v14 patch set and the approach.


I'll reply other comments later...


Thanks,
-- 
kou

  133. Re: Make COPY format extendable: Extract COPY TO format implementations

    Michael Paquier <michael@paquier.xyz> — 2024-02-14T03:28:38Z

    On Tue, Feb 13, 2024 at 05:33:40PM +0900, Sutou Kouhei wrote:
> Hi,
> 
> In <20240209192705.5qdilvviq3py2voq@awork3.anarazel.de>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 9 Feb 2024 11:27:05 -0800,
>   Andres Freund <andres@anarazel.de> wrote:
> 
>>> +static void
>>> +CopyFromTextInFunc(CopyFromState cstate, Oid atttypid,
>>> +				   FmgrInfo *finfo, Oid *typioparam)
>>> +{
>>> +	Oid			func_oid;
>>> +
>>> +	getTypeInputInfo(atttypid, &func_oid, typioparam);
>>> +	fmgr_info(func_oid, finfo);
>>> +}
>> 
>> FWIW, we should really change the copy code to initialize FunctionCallInfoData
>> instead of re-initializing that on every call, realy makes a difference
>> performance wise.
> 
> How about the attached patch approach? If it's a desired
> approach, I can also write a separated patch for COPY TO.

Hmm, I have not studied that much, but my first impression was that we
would not require any new facility in fmgr.c, but perhaps you're right
and it's more elegant to pass a InitFunctionCallInfoData this way.

PrepareInputFunctionCallInfo() looks OK as a name, but I'm less a fan
of PreparedInputFunctionCallSafe() and its "Prepared" part.  How about
something like ExecuteInputFunctionCallSafe()?

I may be able to look more at that next week, and I would surely check
the impact of that with a simple COPY query throttled by CPU (more
rows and more attributes the better).

>>> +	cstate->raw_fields = (char **) palloc(attr_count * sizeof(char *));
>>> +	/* Set read attribute callback */
>>> +	if (cstate->opts.csv_mode)
>>> +		cstate->copy_read_attributes = CopyReadAttributesCSV;
>>> +	else
>>> +		cstate->copy_read_attributes = CopyReadAttributesText;
>>> +}
>> 
>> Isn't this precisely repeating the mistake of 2889fd23be56?
> 
> What do you think about the approach in my previous mail's
> attachments?
> https://www.postgresql.org/message-id/flat/20240209.163205.704848659612151781.kou%40clear-code.com#dbb1f8d7f2f0e8fe3c7e37a757fcfc54
>
> If it's a desired approach, I can prepare a v15 patch set
> based on the v14 patch set and the approach.

Yes, this one looks like it's using the right angle: we don't rely
anymore in cstate to decide which CopyReadAttributes to use, the
routines do that instead.  Note that I've reverted 06bd311bce24 for
the moment, as this is just getting in the way of the main patch, and
that was non-optimal once there is a per-row callback.

> diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c
> index 41f6bc43e4..a43c853e99 100644
> --- a/src/backend/commands/copyfrom.c
> +++ b/src/backend/commands/copyfrom.c
> @@ -1691,6 +1691,10 @@ BeginCopyFrom(ParseState *pstate,
>  	/* We keep those variables in cstate. */
>  	cstate->in_functions = in_functions;
>  	cstate->typioparams = typioparams;
> +	if (cstate->opts.binary)
> +		cstate->fcinfo = PrepareInputFunctionCallInfo();
> +	else
> +		cstate->fcinfo = PrepareReceiveFunctionCallInfo();

Perhaps we'd better avoid more callbacks like that, for now.
--
Michael

  134. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-02-14T05:08:51Z

    Hi,

In <ZcwzZrrsTEJ7oJyq@paquier.xyz>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Wed, 14 Feb 2024 12:28:38 +0900,
  Michael Paquier <michael@paquier.xyz> wrote:

>> How about the attached patch approach? If it's a desired
>> approach, I can also write a separated patch for COPY TO.
> 
> Hmm, I have not studied that much, but my first impression was that we
> would not require any new facility in fmgr.c, but perhaps you're right
> and it's more elegant to pass a InitFunctionCallInfoData this way.

I'm not familiar with the fmgr.c related code base but it
seems that we abstract {,binary-}input function call by
fmgr.c. So I think that it's better that we follow the
design. (If there is a person who knows the fmgr.c related
code base, please help us.)

> PrepareInputFunctionCallInfo() looks OK as a name, but I'm less a fan
> of PreparedInputFunctionCallSafe() and its "Prepared" part.  How about
> something like ExecuteInputFunctionCallSafe()?

I understand the feeling. SQL uses "prepared" for "prepared
statement". There are similar function names such as
InputFunctionCall()/InputFunctionCallSafe()/DirectInputFunctionCallSafe(). They
execute (call) an input function but they use "call" not
"execute" for it... So "Execute...Call..." may be
redundant...

How about InputFunctionCallSafeWithInfo(),
InputFunctionCallSafeInfo() or
InputFunctionCallInfoCallSafe()?

> I may be able to look more at that next week, and I would surely check
> the impact of that with a simple COPY query throttled by CPU (more
> rows and more attributes the better).

Thanks!

>                            Note that I've reverted 06bd311bce24 for
> the moment, as this is just getting in the way of the main patch, and
> that was non-optimal once there is a per-row callback.

Thanks for sharing the information. I'll rebase on master
when I create the v15 patch.


>> diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c
>> index 41f6bc43e4..a43c853e99 100644
>> --- a/src/backend/commands/copyfrom.c
>> +++ b/src/backend/commands/copyfrom.c
>> @@ -1691,6 +1691,10 @@ BeginCopyFrom(ParseState *pstate,
>>  	/* We keep those variables in cstate. */
>>  	cstate->in_functions = in_functions;
>>  	cstate->typioparams = typioparams;
>> +	if (cstate->opts.binary)
>> +		cstate->fcinfo = PrepareInputFunctionCallInfo();
>> +	else
>> +		cstate->fcinfo = PrepareReceiveFunctionCallInfo();
> 
> Perhaps we'd better avoid more callbacks like that, for now.

I'll not use a callback for this. I'll not change this part
after we introduce Copy{To,From}Routine. cstate->fcinfo
isn't used some custom COPY format handlers such as Apache
Arrow handler like cstate->in_functions and
cstate->typioparams. But they will be always allocated. It's
a bit wasteful for those handlers but we may not care about
it. So we can always use "if (state->opts.binary)" condition
here.

BTW... This part was wrong... Sorry... It should be:


	if (cstate->opts.binary)
		cstate->fcinfo = PrepareReceiveFunctionCallInfo();
	else
		cstate->fcinfo = PrepareInputFunctionCallInfo();


Thanks,
-- 
kou

  135. Re: Make COPY format extendable: Extract COPY TO format implementations

    Michael Paquier <michael@paquier.xyz> — 2024-02-14T06:52:36Z

    On Wed, Feb 14, 2024 at 02:08:51PM +0900, Sutou Kouhei wrote:
> I understand the feeling. SQL uses "prepared" for "prepared
> statement". There are similar function names such as
> InputFunctionCall()/InputFunctionCallSafe()/DirectInputFunctionCallSafe(). They
> execute (call) an input function but they use "call" not
> "execute" for it... So "Execute...Call..." may be
> redundant...
> 
> How about InputFunctionCallSafeWithInfo(),
> InputFunctionCallSafeInfo() or
> InputFunctionCallInfoCallSafe()?

WithInfo() would not be a new thing.  There are a couple of APIs named
like this when manipulating catalogs, so that sounds kind of a good
choice from here.
--
Michael

  136. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-02-15T06:34:21Z

    Hi,

In <ZcxjNDtqNLvdz0f5@paquier.xyz>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Wed, 14 Feb 2024 15:52:36 +0900,
  Michael Paquier <michael@paquier.xyz> wrote:

>> How about InputFunctionCallSafeWithInfo(),
>> InputFunctionCallSafeInfo() or
>> InputFunctionCallInfoCallSafe()?
> 
> WithInfo() would not be a new thing.  There are a couple of APIs named
> like this when manipulating catalogs, so that sounds kind of a good
> choice from here.

Thanks for the info. Let's use InputFunctionCallSafeWithInfo().
See that attached patch:
v2-0001-Reuse-fcinfo-used-in-COPY-FROM.patch

I also attach a patch for COPY TO:
v1-0001-Reuse-fcinfo-used-in-COPY-TO.patch

I measured the COPY TO patch on my environment with:
COPY (SELECT 1::int2,2::int2,3::int2,4::int2,5::int2,6::int2,7::int2,8::int2,9::int2,10::int2,11::int2,12::int2,13::int2,14::int2,15::int2,16::int2,17::int2,18::int2,19::int2,20::int2, generate_series(1, 1000000::int4)) TO '/dev/null' \watch c=5

master:
740.066ms
734.884ms
738.579ms
734.170ms
727.953ms

patched:
730.714ms
741.483ms
714.149ms
715.436ms
713.578ms

It seems that it improves performance a bit but my
environment isn't suitable for benchmark. So they may not
be valid numbers.


Thanks,
-- 
kou

  137. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-02-15T06:51:29Z

    Hi,

In <20240213.173340.1518143507526518973.kou@clear-code.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Tue, 13 Feb 2024 17:33:40 +0900 (JST),
  Sutou Kouhei <kou@clear-code.com> wrote:

> I'll reply other comments later...

I've read other comments and my answers for them are same as
Michael's one.


I'll prepare the v15 patch with static inline functions and
fixed arguments after the fcinfo cache patches are merged. I
think that the v15 patch will be conflicted with fcinfo
cache patches.


Thanks,
-- 
kou

  138. Re: Make COPY format extendable: Extract COPY TO format implementations

    jian he <jian.universality@gmail.com> — 2024-02-15T09:09:20Z

    On Thu, Feb 15, 2024 at 2:34 PM Sutou Kouhei <kou@clear-code.com> wrote:
>
>
> Thanks for the info. Let's use InputFunctionCallSafeWithInfo().
> See that attached patch:
> v2-0001-Reuse-fcinfo-used-in-COPY-FROM.patch
>
> I also attach a patch for COPY TO:
> v1-0001-Reuse-fcinfo-used-in-COPY-TO.patch
>
> I measured the COPY TO patch on my environment with:
> COPY (SELECT 1::int2,2::int2,3::int2,4::int2,5::int2,6::int2,7::int2,8::int2,9::int2,10::int2,11::int2,12::int2,13::int2,14::int2,15::int2,16::int2,17::int2,18::int2,19::int2,20::int2, generate_series(1, 1000000::int4)) TO '/dev/null' \watch c=5
>
> master:
> 740.066ms
> 734.884ms
> 738.579ms
> 734.170ms
> 727.953ms
>
> patched:
> 730.714ms
> 741.483ms
> 714.149ms
> 715.436ms
> 713.578ms
>
> It seems that it improves performance a bit but my
> environment isn't suitable for benchmark. So they may not
> be valid numbers.

My environment is slow (around 10x) but consistent.
I see around 2-3 percent increase consistently.
(with patch 7369.068 ms, without patch 7574.802 ms)

the patchset looks good in my eyes, i can understand it.
however I cannot apply it cleanly against the HEAD.

+/*
+ * Prepare callinfo for InputFunctionCallSafeWithInfo to reuse one callinfo
+ * instead of initializing it for each call. This is for performance.
+ */
+FunctionCallInfoBaseData *
+PrepareInputFunctionCallInfo(void)
+{
+ FunctionCallInfoBaseData *fcinfo;
+
+ fcinfo = (FunctionCallInfoBaseData *) palloc(SizeForFunctionCallInfo(3));

just wondering, I saw other similar places using palloc0,
do we need to use palloc0?

  139. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-02-15T09:15:54Z

    Hi,

In <CACJufxE=m8kMC92JpaqNMg02P_Pi1sZJ1w=xNec0=j_W6d9GDw@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Thu, 15 Feb 2024 17:09:20 +0800,
  jian he <jian.universality@gmail.com> wrote:

> My environment is slow (around 10x) but consistent.
> I see around 2-3 percent increase consistently.
> (with patch 7369.068 ms, without patch 7574.802 ms)

Thanks for sharing your numbers! It will help us to
determine whether these changes improve performance or not.

> the patchset looks good in my eyes, i can understand it.
> however I cannot apply it cleanly against the HEAD.

Hmm, I used 9bc1eee988c31e66a27e007d41020664df490214 as the
base version. But both patches based on the same
revision. So we may not be able to apply both patches at
once cleanly.

> +/*
> + * Prepare callinfo for InputFunctionCallSafeWithInfo to reuse one callinfo
> + * instead of initializing it for each call. This is for performance.
> + */
> +FunctionCallInfoBaseData *
> +PrepareInputFunctionCallInfo(void)
> +{
> + FunctionCallInfoBaseData *fcinfo;
> +
> + fcinfo = (FunctionCallInfoBaseData *) palloc(SizeForFunctionCallInfo(3));
> 
> just wondering, I saw other similar places using palloc0,
> do we need to use palloc0?

I think that we don't need to use palloc0() here because the
following InitFunctionCallInfoData() call initializes all
members explicitly.


Thanks,
-- 
kou

  140. Re: Make COPY format extendable: Extract COPY TO format implementations

    Michael Paquier <michael@paquier.xyz> — 2024-02-22T06:44:16Z

    On Thu, Feb 15, 2024 at 03:34:21PM +0900, Sutou Kouhei wrote:
> It seems that it improves performance a bit but my
> environment isn't suitable for benchmark. So they may not
> be valid numbers.

I was comparing what you have here, and what's been attached by Andres
at [1] and the top of the changes on my development branch at [2]
(v3-0008, mostly).  And, it strikes me that there is no need to do any
major changes in any of the callbacks proposed up to v13 and v14 in
this thread, as all the changes proposed want to plug in more data
into each StateData for COPY FROM and COPY TO, the best part being
that v3-0008 can just reuse the proposed callbacks as-is.  v1-0001
from Sutou-san would need one slight tweak in the per-row callback,
still that's minor.

I have been spending more time on the patch to introduce the COPY
APIs, leading me to the v15 attached, where I have replaced the
previous attribute callbacks for the output representation and the
reads with hardcoded routines that should be optimized by compilers,
and I have done more profiling with -O2.  I'm aware of the disparities
in the per-row and start callbacks for the text/csv cases as well as
the default expressions, but these are really format-dependent with
their own assumptions so splitting them is something that makes
limited sense to me.  I've also looks at externalizing some of the
error handling, though the result was not that beautiful, so what I
got here is what makes the callbacks leaner and easier to work with.

First, some results for COPY FROM using the previous tests (30 int
attributes, running on scissors, data sent to blackhole_am, etc.) in
NextCopyFrom() which becomes the hot-spot:
* Using v15:
  Children      Self  Command   Shared Object       Symbol
-   66.42%     0.71%  postgres  postgres            [.] NextCopyFrom
    - 65.70% NextCopyFrom
       - 65.49% CopyFromTextLikeOneRow
          + 19.29% InputFunctionCallSafe
          + 15.81% CopyReadLine
            13.89% CopyReadAttributesText
    + 0.71% _start
* Using HEAD (today's 011d60c4352c):
  Children      Self  Command   Shared Object       Symbol
-   67.09%    16.64%  postgres  postgres            [.] NextCopyFrom
    - 50.45% NextCopyFrom
       - 30.89% NextCopyFromRawFields
          + 16.26% CopyReadLine
            13.59% CopyReadAttributesText
       + 19.24% InputFunctionCallSafe
    + 16.64% _start

In this case, I have been able to limit the effects of the per-row
callback by making NextCopyFromRawFields() local to copyfromparse.c
while applying some inlining to it.  This brings me to a different
point, why don't we do this change independently on HEAD?  It's not 
really complicated to make NextCopyFromRawFields show high in the
profiles.  I was looking at external projects, and noticed that
there's nothing calling NextCopyFromRawFields() directly.

Second, some profiles with COPY TO (30 int integers, running on
scissors) where data is sent /dev/null:
* Using v15:
  Children      Self  Command   Shared Object       Symbol
-   85.61%     0.34%  postgres  postgres            [.] CopyOneRowTo
    - 85.26% CopyOneRowTo
       - 75.86% CopyToTextOneRow
          + 36.49% OutputFunctionCall
          + 10.53% appendBinaryStringInfo
            9.66% CopyAttributeOutText
            1.34% int4out
            0.92% 0xffffa9803be8
            0.79% enlargeStringInfo
            0.77% memcpy@plt
            0.69% 0xffffa9803be4
       + 3.12% CopySendEndOfRow
         2.81% CopySendChar
         0.95% pgstat_progress_update_param
         0.95% appendBinaryStringInfo
         0.55% MemoryContextReset
* Using HEAD (today's 011d60c4352c):
  Children      Self  Command   Shared Object       Symbol
-   80.35%    14.23%  postgres  postgres            [.] CopyOneRowTo
    - 66.12% CopyOneRowTo
       + 35.40% OutputFunctionCall
       + 11.00% appendBinaryStringInfo
         8.38% CopyAttributeOutText
       + 2.98% CopySendEndOfRow
         1.52% int4out
         0.88% pgstat_progress_update_param
         0.87% 0xffff8ab32be8
         0.74% memcpy@plt
         0.68% enlargeStringInfo
         0.61% 0xffff8ab32be4
         0.51% MemoryContextReset
    + 14.23% _start

The increase in CopyOneRowTo from 80% to 85% worries me but I am not
quite sure how to optimize that with the current structure of the
code, so the dispatch caused by per-row callback is noticeable in
what's my worst test case.  I am not quite sure how to avoid that,
TBH.  A result that has been puzzling me is that I am getting faster
runtimes with v15 (6232ms in average) vs HEAD (6550ms) at 5M rows with
COPY TO for what led to these profiles (for tests without perf
attached to the backends).

Any thoughts overall?

[1]: https://www.postgresql.org/message-id/20240218015955.rmw5mcmobt5hbene%40awork3.anarazel.de
[2]: https://www.postgresql.org/message-id/ZcWoTr1N0GELFA9E@paquier.xyz
--
Michael

  141. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-02-22T09:39:48Z

    Hi,

In <ZdbtQJ-p5H1_EDwE@paquier.xyz>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Thu, 22 Feb 2024 15:44:16 +0900,
  Michael Paquier <michael@paquier.xyz> wrote:

> I was comparing what you have here, and what's been attached by Andres
> at [1] and the top of the changes on my development branch at [2]
> (v3-0008, mostly).  And, it strikes me that there is no need to do any
> major changes in any of the callbacks proposed up to v13 and v14 in
> this thread, as all the changes proposed want to plug in more data
> into each StateData for COPY FROM and COPY TO, the best part being
> that v3-0008 can just reuse the proposed callbacks as-is.  v1-0001
> from Sutou-san would need one slight tweak in the per-row callback,
> still that's minor.

I think so too. But I thought that some minor conflicts will
be happen with this and the v15. So I worked on this before
the v15.

We agreed that this optimization doesn't block v15: [1]
So we can work on the v15 without this optimization for now.

[1] https://www.postgresql.org/message-id/flat/20240219195351.5vy7cdl3wxia66kg%40awork3.anarazel.de#20f9677e074fb0f8c5bb3994ef059a15

> I have been spending more time on the patch to introduce the COPY
> APIs, leading me to the v15 attached, where I have replaced the
> previous attribute callbacks for the output representation and the
> reads with hardcoded routines that should be optimized by compilers,
> and I have done more profiling with -O2.

Thanks! I wanted to work on it but I didn't have enough time
for it in a few days...

I've reviewed the v15.

----
> @@ -751,8 +751,9 @@ CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes)
>   *
>   * NOTE: force_not_null option are not applied to the returned fields.
>   */
> -bool
> -NextCopyFromRawFields(CopyFromState cstate, char ***fields, int *nfields)
> +static bool

"inline" is missing here.

> +NextCopyFromRawFields(CopyFromState cstate, char ***fields, int *nfields,
> +					  bool is_csv)
>  {
>  	int			fldct;
----

How about adding "is_csv" to CopyReadline() and
CopyReadLineText() too?

----
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 25b8d4bc52..79fabecc69 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -150,8 +150,8 @@ static const char BinarySignature[11] = "PGCOPY\n\377\r\n\0";
 
 
 /* non-export function prototypes */
-static bool CopyReadLine(CopyFromState cstate);
-static bool CopyReadLineText(CopyFromState cstate);
+static inline bool CopyReadLine(CopyFromState cstate, bool is_csv);
+static inline bool CopyReadLineText(CopyFromState cstate, bool is_csv);
 static inline int CopyReadAttributesText(CopyFromState cstate);
 static inline int CopyReadAttributesCSV(CopyFromState cstate);
 static Datum CopyReadBinaryAttribute(CopyFromState cstate, FmgrInfo *flinfo,
@@ -770,7 +770,7 @@ NextCopyFromRawFields(CopyFromState cstate, char ***fields, int *nfields,
 		tupDesc = RelationGetDescr(cstate->rel);
 
 		cstate->cur_lineno++;
-		done = CopyReadLine(cstate);
+		done = CopyReadLine(cstate, is_csv);
 
 		if (cstate->opts.header_line == COPY_HEADER_MATCH)
 		{
@@ -823,7 +823,7 @@ NextCopyFromRawFields(CopyFromState cstate, char ***fields, int *nfields,
 	cstate->cur_lineno++;
 
 	/* Actually read the line into memory here */
-	done = CopyReadLine(cstate);
+	done = CopyReadLine(cstate, is_csv);
 
 	/*
 	 * EOF at start of line means we're done.  If we see EOF after some
@@ -1133,8 +1133,8 @@ NextCopyFrom(CopyFromState cstate, ExprContext *econtext,
  * by newline.  The terminating newline or EOF marker is not included
  * in the final value of line_buf.
  */
-static bool
-CopyReadLine(CopyFromState cstate)
+static inline bool
+CopyReadLine(CopyFromState cstate, bool is_csv)
 {
 	bool		result;
 
@@ -1142,7 +1142,7 @@ CopyReadLine(CopyFromState cstate)
 	cstate->line_buf_valid = false;
 
 	/* Parse data and transfer into line_buf */
-	result = CopyReadLineText(cstate);
+	result = CopyReadLineText(cstate, is_csv);
 
 	if (result)
 	{
@@ -1209,8 +1209,8 @@ CopyReadLine(CopyFromState cstate)
 /*
  * CopyReadLineText - inner loop of CopyReadLine for text mode
  */
-static bool
-CopyReadLineText(CopyFromState cstate)
+static inline bool
+CopyReadLineText(CopyFromState cstate, bool is_csv)
 {
 	char	   *copy_input_buf;
 	int			input_buf_ptr;
@@ -1226,7 +1226,7 @@ CopyReadLineText(CopyFromState cstate)
 	char		quotec = '\0';
 	char		escapec = '\0';
 
-	if (cstate->opts.csv_mode)
+	if (is_csv)
 	{
 		quotec = cstate->opts.quote[0];
 		escapec = cstate->opts.escape[0];
@@ -1306,7 +1306,7 @@ CopyReadLineText(CopyFromState cstate)
 		prev_raw_ptr = input_buf_ptr;
 		c = copy_input_buf[input_buf_ptr++];
 
-		if (cstate->opts.csv_mode)
+		if (is_csv)
 		{
 			/*
 			 * If character is '\\' or '\r', we may need to look ahead below.
@@ -1345,7 +1345,7 @@ CopyReadLineText(CopyFromState cstate)
 		}
 
 		/* Process \r */
-		if (c == '\r' && (!cstate->opts.csv_mode || !in_quote))
+		if (c == '\r' && (!is_csv || !in_quote))
 		{
 			/* Check for \r\n on first line, _and_ handle \r\n. */
 			if (cstate->eol_type == EOL_UNKNOWN ||
@@ -1373,10 +1373,10 @@ CopyReadLineText(CopyFromState cstate)
 					if (cstate->eol_type == EOL_CRNL)
 						ereport(ERROR,
 								(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
-								 !cstate->opts.csv_mode ?
+								 !is_csv ?
 								 errmsg("literal carriage return found in data") :
 								 errmsg("unquoted carriage return found in data"),
-								 !cstate->opts.csv_mode ?
+								 !is_csv ?
 								 errhint("Use \"\\r\" to represent carriage return.") :
 								 errhint("Use quoted CSV field to represent carriage return.")));
 
@@ -1390,10 +1390,10 @@ CopyReadLineText(CopyFromState cstate)
 			else if (cstate->eol_type == EOL_NL)
 				ereport(ERROR,
 						(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
-						 !cstate->opts.csv_mode ?
+						 !is_csv ?
 						 errmsg("literal carriage return found in data") :
 						 errmsg("unquoted carriage return found in data"),
-						 !cstate->opts.csv_mode ?
+						 !is_csv ?
 						 errhint("Use \"\\r\" to represent carriage return.") :
 						 errhint("Use quoted CSV field to represent carriage return.")));
 			/* If reach here, we have found the line terminator */
@@ -1401,15 +1401,15 @@ CopyReadLineText(CopyFromState cstate)
 		}
 
 		/* Process \n */
-		if (c == '\n' && (!cstate->opts.csv_mode || !in_quote))
+		if (c == '\n' && (!is_csv || !in_quote))
 		{
 			if (cstate->eol_type == EOL_CR || cstate->eol_type == EOL_CRNL)
 				ereport(ERROR,
 						(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
-						 !cstate->opts.csv_mode ?
+						 !is_csv ?
 						 errmsg("literal newline found in data") :
 						 errmsg("unquoted newline found in data"),
-						 !cstate->opts.csv_mode ?
+						 !is_csv ?
 						 errhint("Use \"\\n\" to represent newline.") :
 						 errhint("Use quoted CSV field to represent newline.")));
 			cstate->eol_type = EOL_NL;	/* in case not set yet */
@@ -1421,7 +1421,7 @@ CopyReadLineText(CopyFromState cstate)
 		 * In CSV mode, we only recognize \. alone on a line.  This is because
 		 * \. is a valid CSV data value.
 		 */
-		if (c == '\\' && (!cstate->opts.csv_mode || first_char_in_line))
+		if (c == '\\' && (!is_csv || first_char_in_line))
 		{
 			char		c2;
 
@@ -1454,7 +1454,7 @@ CopyReadLineText(CopyFromState cstate)
 
 					if (c2 == '\n')
 					{
-						if (!cstate->opts.csv_mode)
+						if (!is_csv)
 							ereport(ERROR,
 									(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
 									 errmsg("end-of-copy marker does not match previous newline style")));
@@ -1463,7 +1463,7 @@ CopyReadLineText(CopyFromState cstate)
 					}
 					else if (c2 != '\r')
 					{
-						if (!cstate->opts.csv_mode)
+						if (!is_csv)
 							ereport(ERROR,
 									(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
 									 errmsg("end-of-copy marker corrupt")));
@@ -1479,7 +1479,7 @@ CopyReadLineText(CopyFromState cstate)
 
 				if (c2 != '\r' && c2 != '\n')
 				{
-					if (!cstate->opts.csv_mode)
+					if (!is_csv)
 						ereport(ERROR,
 								(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
 								 errmsg("end-of-copy marker corrupt")));
@@ -1508,7 +1508,7 @@ CopyReadLineText(CopyFromState cstate)
 				result = true;	/* report EOF */
 				break;
 			}
-			else if (!cstate->opts.csv_mode)
+			else if (!is_csv)
 			{
 				/*
 				 * If we are here, it means we found a backslash followed by
----

> In this case, I have been able to limit the effects of the per-row
> callback by making NextCopyFromRawFields() local to copyfromparse.c
> while applying some inlining to it.  This brings me to a different
> point, why don't we do this change independently on HEAD?

Does this mean that changing

bool NextCopyFromRawFields(CopyFromState cstate, char ***fields, int *nfields)

to (adding "static")

static bool NextCopyFromRawFields(CopyFromState cstate, char ***fields, int *nfields)

not  (adding "static" and "bool is_csv")

static bool NextCopyFromRawFields(CopyFromState cstate, char ***fields, int *nfields, bool is_csv)

improves performance?

If so, adding the change independently on HEAD makes
sense. But I don't know why that improves
performance... Inlining?

>                                                            It's not 
> really complicated to make NextCopyFromRawFields show high in the
> profiles.  I was looking at external projects, and noticed that
> there's nothing calling NextCopyFromRawFields() directly.

It means that we can hide NextCopyFromRawFields() without
breaking compatibility (because nobody uses it), right?

If so, I also think that we can change
NextCopyFromRawFields() directly.

If we assume that someone (not public code) may use it, we
can create a new internal function and use it something
like:

----
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 7cacd0b752..b1515ead82 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -751,8 +751,8 @@ CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes)
  *
  * NOTE: force_not_null option are not applied to the returned fields.
  */
-bool
-NextCopyFromRawFields(CopyFromState cstate, char ***fields, int *nfields)
+static bool
+NextCopyFromRawFieldsInternal(CopyFromState cstate, char ***fields, int *nfields)
 {
 	int			fldct;
 	bool		done;
@@ -840,6 +840,12 @@ NextCopyFromRawFields(CopyFromState cstate, char ***fields, int *nfields)
 	return true;
 }
 
+bool
+NextCopyFromRawFields(CopyFromState cstate, char ***fields, int *nfields)
+{
+	return NextCopyFromRawFieldsInternal(cstate, fields, nfields);
+}
+
 /*
  * Read next tuple from file for COPY FROM. Return false if no more tuples.
  *
----


Thanks,
-- 
kou

  142. Re: Make COPY format extendable: Extract COPY TO format implementations

    Michael Paquier <michael@paquier.xyz> — 2024-03-01T05:31:38Z

    On Thu, Feb 22, 2024 at 06:39:48PM +0900, Sutou Kouhei wrote:
> If so, adding the change independently on HEAD makes
> sense. But I don't know why that improves
> performance... Inlining?

I guess so.  It does not make much of a difference, though.  The thing
is that the dispatch caused by the custom callbacks called for each
row is noticeable in any profiles I'm taking (not that much in the
worst-case scenarios, still a few percents), meaning that this impacts
the performance for all the in-core formats (text, csv, binary) as
long as we refactor text/csv/binary to use the routines of copyapi.h.
I don't really see a way forward, except if we don't dispatch the
in-core formats to not impact the default cases.  That makes the code
a bit less elegant, but equally efficient for the existing formats.
--
Michael

  143. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-03-01T06:29:17Z

    Hi,

In <20240222.183948.518018047578925034.kou@clear-code.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Thu, 22 Feb 2024 18:39:48 +0900 (JST),
  Sutou Kouhei <kou@clear-code.com> wrote:

> How about adding "is_csv" to CopyReadline() and
> CopyReadLineText() too?

I tried this on my environment. This is a change for COPY
FROM not COPY TO but this decreases COPY TO
performance with [1]... Hmm...

master:   697.693 msec (the best case)
v15:      576.374 msec (the best case)
v15+this: 593.559 msec (the best case)

[1] COPY (SELECT 1::int2,2::int2,3::int2,4::int2,5::int2,6::int2,7::int2,8::int2,9::int2,10::int2,11::int2,12::int2,13::int2,14::int2,15::int2,16::int2,17::int2,18::int2,19::int2,20::int2, generate_series(1, 1000000::int4)) TO '/dev/null' \watch c=15

So I think that v15 is good.


perf result of master:

# Children      Self  Command   Shared Object      Symbol                                   
# ........  ........  ........  .................  .........................................
#
    31.39%    14.54%  postgres  postgres           [.] CopyOneRowTo
            |--17.00%--CopyOneRowTo
            |          |--10.61%--FunctionCall1Coll
            |          |           --8.40%--int2out
            |          |                     |--2.58%--pg_ltoa
            |          |                     |           --0.68%--pg_ultoa_n
            |          |                     |--1.11%--pg_ultoa_n
            |          |                     |--0.83%--AllocSetAlloc
            |          |                     |--0.69%--__memcpy_avx_unaligned_erms (inlined)
            |          |                     |--0.58%--FunctionCall1Coll
            |          |                      --0.55%--memcpy@plt
            |          |--3.25%--appendBinaryStringInfo
            |          |           --0.56%--pg_ultoa_n
            |           --0.69%--CopyAttributeOutText

perf result of v15:

# Children      Self  Command   Shared Object      Symbol                                   
# ........  ........  ........  .................  .........................................
#
    25.60%    10.47%  postgres  postgres           [.] CopyToTextOneRow
            |--15.39%--CopyToTextOneRow
            |          |--10.44%--FunctionCall1Coll
            |          |          |--7.25%--int2out
            |          |          |          |--2.60%--pg_ltoa
            |          |          |          |           --0.71%--pg_ultoa_n
            |          |          |          |--0.90%--FunctionCall1Coll
            |          |          |          |--0.84%--pg_ultoa_n
            |          |          |           --0.66%--AllocSetAlloc
            |          |          |--0.79%--ExecProjectSet
            |          |           --0.68%--int4out
            |          |--2.50%--appendBinaryStringInfo
            |           --0.53%--CopyAttributeOutText


The profiles on Michael's environment [2] showed that
CopyOneRow() % was increased by v15. But it
(CopyToTextOneRow() % not CopyOneRow() %) wasn't increased
by v15. It's decreased instead.

[2] https://www.postgresql.org/message-id/flat/ZdbtQJ-p5H1_EDwE%40paquier.xyz#6439e6ad574f2d47cd7220e9bfed3889

So I think that v15 doesn't have performance regression but
my environment isn't suitable for benchmark...


Thanks,
-- 
kou

  144. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-03-01T06:44:43Z

    Hi,

In <ZeFoOprWyKU6gpkP@paquier.xyz>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 1 Mar 2024 14:31:38 +0900,
  Michael Paquier <michael@paquier.xyz> wrote:

> I guess so.  It does not make much of a difference, though.  The thing
> is that the dispatch caused by the custom callbacks called for each
> row is noticeable in any profiles I'm taking (not that much in the
> worst-case scenarios, still a few percents), meaning that this impacts
> the performance for all the in-core formats (text, csv, binary) as
> long as we refactor text/csv/binary to use the routines of copyapi.h.
> I don't really see a way forward, except if we don't dispatch the
> in-core formats to not impact the default cases.  That makes the code
> a bit less elegant, but equally efficient for the existing formats.

It's an option based on your profile result but your
execution result also shows that v15 is faster than HEAD [1]:

> I am getting faster runtimes with v15 (6232ms in average)
> vs HEAD (6550ms) at 5M rows with COPY TO

[1] https://www.postgresql.org/message-id/flat/ZdbtQJ-p5H1_EDwE%40paquier.xyz#6439e6ad574f2d47cd7220e9bfed3889

I think that faster runtime is beneficial than mysterious
profile for users. So I think that we can merge v15 to
master.


Thanks,
-- 
kou

  145. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-03-04T05:11:08Z

    Hi,

In <20240301.154443.618034282613922707.kou@clear-code.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 01 Mar 2024 15:44:43 +0900 (JST),
  Sutou Kouhei <kou@clear-code.com> wrote:

>> I guess so.  It does not make much of a difference, though.  The thing
>> is that the dispatch caused by the custom callbacks called for each
>> row is noticeable in any profiles I'm taking (not that much in the
>> worst-case scenarios, still a few percents), meaning that this impacts
>> the performance for all the in-core formats (text, csv, binary) as
>> long as we refactor text/csv/binary to use the routines of copyapi.h.
>> I don't really see a way forward, except if we don't dispatch the
>> in-core formats to not impact the default cases.  That makes the code
>> a bit less elegant, but equally efficient for the existing formats.
> 
> It's an option based on your profile result but your
> execution result also shows that v15 is faster than HEAD [1]:
> 
>> I am getting faster runtimes with v15 (6232ms in average)
>> vs HEAD (6550ms) at 5M rows with COPY TO
> 
> [1] https://www.postgresql.org/message-id/flat/ZdbtQJ-p5H1_EDwE%40paquier.xyz#6439e6ad574f2d47cd7220e9bfed3889
> 
> I think that faster runtime is beneficial than mysterious
> profile for users. So I think that we can merge v15 to
> master.

If this is a blocker of making COPY format extendable, can
we defer moving the existing text/csv/binary format
implementations to Copy{From,To}Routine for now as Michael
suggested to proceed making COPY format extendable? (Can we
add Copy{From,To}Routine without changing the existing
text/csv/binary format implementations?)

I attach a patch for it.

There is a large hunk for CopyOneRowTo() that is caused by
indent change. I also attach "...-w.patch" that uses "git
-w" to remove space only changes. "...-w.patch" is only for
review. We should use .patch without -w for push.


Thanks,
-- 
kou

  146. Re: Make COPY format extendable: Extract COPY TO format implementations

    Michael Paquier <michael@paquier.xyz> — 2024-03-05T06:16:33Z

    On Mon, Mar 04, 2024 at 02:11:08PM +0900, Sutou Kouhei wrote:
> If this is a blocker of making COPY format extendable, can
> we defer moving the existing text/csv/binary format
> implementations to Copy{From,To}Routine for now as Michael
> suggested to proceed making COPY format extendable? (Can we
> add Copy{From,To}Routine without changing the existing
> text/csv/binary format implementations?)

Yeah, I assume that it would be the way to go so as we don't do any
dispatching in default cases.  A different approach that could be done
is to hide some of the parts of binary and text/csv in inline static
functions that are equivalent to the routine callbacks.  That's
similar to the previous versions of the patch set, but if we come back
to the argument that there is a risk of blocking optimizations of more
of the local areas of the per-row processing in NextCopyFrom() and
CopyOneRowTo(), what you have sounds like a good balance.

CopyOneRowTo() could do something like that to avoid the extra
indentation:
if (cstate->routine)
{
    cstate->routine->CopyToOneRow(cstate, slot);
    MemoryContextSwitchTo(oldcontext);
    return;
}

NextCopyFrom() does not need to be concerned by that.

> I attach a patch for it.

> There is a large hunk for CopyOneRowTo() that is caused by
> indent change. I also attach "...-w.patch" that uses "git
> -w" to remove space only changes. "...-w.patch" is only for
> review. We should use .patch without -w for push.

I didn't know this trick.  That's indeed nice..  I may use that for
other stuff to make patches more presentable to the eyes.  And that's
available as well with `git diff`.

If we basically agree about this part, how would the rest work out
with this set of APIs and the possibility to plug in a custom value
for FORMAT to do a pg_proc lookup, including an example of how these
APIs can be used?
--
Michael

  147. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-03-05T08:18:08Z

    Hi,

In <Zea4wXxpYaX64e_p@paquier.xyz>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Tue, 5 Mar 2024 15:16:33 +0900,
  Michael Paquier <michael@paquier.xyz> wrote:

> CopyOneRowTo() could do something like that to avoid the extra
> indentation:
> if (cstate->routine)
> {
>     cstate->routine->CopyToOneRow(cstate, slot);
>     MemoryContextSwitchTo(oldcontext);
>     return;
> }

OK. The v17 patch uses this style. Others are same as the
v16.

> I didn't know this trick.  That's indeed nice..  I may use that for
> other stuff to make patches more presentable to the eyes.  And that's
> available as well with `git diff`.

:-)

> If we basically agree about this part, how would the rest work out
> with this set of APIs and the possibility to plug in a custom value
> for FORMAT to do a pg_proc lookup, including an example of how these
> APIs can be used?

I'll send the following patches after this patch is
merged. They are based on the v6 patch[1]:

1. Add copy_handler
   * This also adds a pg_proc lookup for custom FORMAT
   * This also adds a test for copy_handler
2. Export CopyToStateData
   * We need it to implement custom copy TO handler
3. Add needed APIs to implement custom copy TO handler
   * Add CopyToStateData::opaque
   * Export CopySendEndOfRow()
4. Export CopyFromStateData
   * We need it to implement custom copy FROM handler
5. Add needed APIs to implement custom copy FROM handler
   * Add CopyFromStateData::opaque
   * Export CopyReadBinaryData()

[1] https://www.postgresql.org/message-id/flat/20240124.144936.67229716500876806.kou%40clear-code.com#f1ad092fc5e81fe38d3c376559efd52c


Thanks,
-- 
kou

  148. Re: Make COPY format extendable: Extract COPY TO format implementations

    Michael Paquier <michael@paquier.xyz> — 2024-03-06T06:34:04Z

    On Tue, Mar 05, 2024 at 05:18:08PM +0900, Sutou Kouhei wrote:
> I'll send the following patches after this patch is
> merged.

I am not sure that my schedule is on track to allow that for this
release, unfortunately, especially with all the other items to review
and discuss to make this thread feature-complete.  There should be
a bit more than four weeks until the feature freeze (date not set in
stone, should be around the 8th of April AoE), but I have less than
the half due to personal issues.  Perhaps if somebody jumps on this
thread, that will be possible..

> They are based on the v6 patch[1]:
> 
> 1. Add copy_handler
>    * This also adds a pg_proc lookup for custom FORMAT
>    * This also adds a test for copy_handler
> 2. Export CopyToStateData
>    * We need it to implement custom copy TO handler
> 3. Add needed APIs to implement custom copy TO handler
>    * Add CopyToStateData::opaque
>    * Export CopySendEndOfRow()
> 4. Export CopyFromStateData
>    * We need it to implement custom copy FROM handler
> 5. Add needed APIs to implement custom copy FROM handler
>    * Add CopyFromStateData::opaque
>    * Export CopyReadBinaryData()

Hmm.  Sounds like a good plan for a split.
--
Michael

  149. Re: Make COPY format extendable: Extract COPY TO format implementations

    Michael Paquier <michael@paquier.xyz> — 2024-03-07T06:32:01Z

    On Wed, Mar 06, 2024 at 03:34:04PM +0900, Michael Paquier wrote:
> I am not sure that my schedule is on track to allow that for this
> release, unfortunately, especially with all the other items to review
> and discuss to make this thread feature-complete.  There should be
> a bit more than four weeks until the feature freeze (date not set in
> stone, should be around the 8th of April AoE), but I have less than
> the half due to personal issues.  Perhaps if somebody jumps on this
> thread, that will be possible..

While on it, here are some profiles based on HEAD and v17 with the
previous tests (COPY TO /dev/null, COPY FROM data sent to the void).

COPY FROM, text format with 30 attributes and HEAD:
-   66.53%    16.33%  postgres  postgres            [.] NextCopyFrom
    - 50.20% NextCopyFrom
       - 30.83% NextCopyFromRawFields
          + 16.09% CopyReadLine
            13.72% CopyReadAttributesText
       + 19.11% InputFunctionCallSafe
    + 16.33% _start
COPY FROM, text format with 30 attributes and v17:
-   66.60%    16.10%  postgres  postgres            [.] NextCopyFrom
    - 50.50% NextCopyFrom
       - 30.44% NextCopyFromRawFields
          + 15.71% CopyReadLine
            13.73% CopyReadAttributesText
       + 19.81% InputFunctionCallSafe
    + 16.10% _start

COPY TO, text format with 30 attributes and HEAD:
-   79.55%    15.54%  postgres  postgres            [.] CopyOneRowTo
    - 64.01% CopyOneRowTo
       + 30.01% OutputFunctionCall
       + 11.71% appendBinaryStringInfo
         9.36% CopyAttributeOutText
       + 3.03% CopySendEndOfRow
         1.65% int4out
         1.01% 0xffff83e46be4
         0.93% 0xffff83e46be8
         0.93% memcpy@plt
         0.87% pgstat_progress_update_param
         0.78% enlargeStringInfo
         0.67% 0xffff83e46bb4
         0.66% 0xffff83e46bcc
         0.57% MemoryContextReset
    + 15.54% _start
COPY TO, text format with 30 attributes and v17:
-   79.35%    16.08%  postgres  postgres            [.] CopyOneRowTo
    - 62.27% CopyOneRowTo
       + 28.92% OutputFunctionCall
       + 10.88% appendBinaryStringInfo
         9.54% CopyAttributeOutText
       + 3.03% CopySendEndOfRow
         1.60% int4out
         0.97% pgstat_progress_update_param
         0.95% 0xffff8c46cbe8
         0.89% memcpy@plt
         0.87% 0xffff8c46cbe4
         0.79% enlargeStringInfo
         0.64% 0xffff8c46cbcc
         0.61% 0xffff8c46cbb4
         0.58% MemoryContextReset
    + 16.08% _start

So, in short, and that's not really a surprise, there is no effect
once we use the dispatching with the routines only when a format would
want to plug-in with the APIs, but a custom format would still have a
penalty of a few percents for both if bottlenecked on CPU.
--
Michael

  150. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-03-08T00:22:54Z

    Hi,

In <ZelfYatRdVZN3FbE@paquier.xyz>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Thu, 7 Mar 2024 15:32:01 +0900,
  Michael Paquier <michael@paquier.xyz> wrote:

> While on it, here are some profiles based on HEAD and v17 with the
> previous tests (COPY TO /dev/null, COPY FROM data sent to the void).
> 
...
> 
> So, in short, and that's not really a surprise, there is no effect
> once we use the dispatching with the routines only when a format would
> want to plug-in with the APIs, but a custom format would still have a
> penalty of a few percents for both if bottlenecked on CPU.

Thanks for sharing these profiles!
I agree with you.

This shows that the v17 approach doesn't affect the current
text/csv/binary implementations. (The v17 approach just adds
2 new structs, Copy{From,To}Rountine, without changing the
current text/csv/binary implementations.)

Can we push the v17 patch and proceed following
implementations? Could someone (especially a PostgreSQL
committer) take a look at this for double-check?


Thanks,
-- 
kou

  151. Re: Make COPY format extendable: Extract COPY TO format implementations

    jian he <jian.universality@gmail.com> — 2024-03-11T00:00:00Z

    On Fri, Mar 8, 2024 at 8:23 AM Sutou Kouhei <kou@clear-code.com> wrote:
>
>
> This shows that the v17 approach doesn't affect the current
> text/csv/binary implementations. (The v17 approach just adds
> 2 new structs, Copy{From,To}Rountine, without changing the
> current text/csv/binary implementations.)
>
> Can we push the v17 patch and proceed following
> implementations? Could someone (especially a PostgreSQL
> committer) take a look at this for double-check?
>

Hi, here are my cents:
Currently in v17, we have 3 extra functions within DoCopyTo
CopyToStart, one time, start, doing some preliminary work.
CopyToOneRow, doing the repetitive work, called many times, row by row.
CopyToEnd, one time doing the closing work.

seems to need a function pointer for processing the format and other options.
or maybe the reason is we need a one time function call before doing DoCopyTo,
like one time initialization.

We can placed the function pointer after:
`
cstate = BeginCopyTo(pstate, rel, query, relid,
stmt->filename, stmt->is_program,
NULL, stmt->attlist, stmt->options);
`


generally in v17, the code pattern looks like this.
if (cstate->opts.binary)
{
/* handle binary format */
}
else if (cstate->routine)
{
/* custom code, make the copy format extensible */
}
else
{
/* handle non-binary, (csv or text) format */
}
maybe we need another bool flag like `bool buildin_format`.
if the copy format is {csv|text|binary}  then buildin_format is true else false.

so the code pattern would be:
if (cstate->opts.binary)
{
/* handle binary format */
}
else if (cstate->routine && !buildin_format)
{
/* custom code, make the copy format extensible */
}
else
{
/* handle non-binary, (csv or text) format */
}

otherwise the {CopyToRoutine| CopyFromRoutine} needs a function pointer
to distinguish native copy format and extensible supported format,
like I mentioned above?

  152. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-03-11T00:56:24Z

    Hi,

In <CACJufxEgn3=j-UWg-f2-DbLO+uVSKGcofpkX5trx+=YX6icSFg@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Mon, 11 Mar 2024 08:00:00 +0800,
  jian he <jian.universality@gmail.com> wrote:

> Hi, here are my cents:
> Currently in v17, we have 3 extra functions within DoCopyTo
> CopyToStart, one time, start, doing some preliminary work.
> CopyToOneRow, doing the repetitive work, called many times, row by row.
> CopyToEnd, one time doing the closing work.
> 
> seems to need a function pointer for processing the format and other options.
> or maybe the reason is we need a one time function call before doing DoCopyTo,
> like one time initialization.

I know that JSON format wants it but can we defer it? We can
add more options later. I want to proceed this improvement
step by step.

More use cases will help us which callbacks are needed. We
will be able to collect more use cases by providing basic
callbacks.

> generally in v17, the code pattern looks like this.
> if (cstate->opts.binary)
> {
> /* handle binary format */
> }
> else if (cstate->routine)
> {
> /* custom code, make the copy format extensible */
> }
> else
> {
> /* handle non-binary, (csv or text) format */
> }
> maybe we need another bool flag like `bool buildin_format`.
> if the copy format is {csv|text|binary}  then buildin_format is true else false.
> 
> so the code pattern would be:
> if (cstate->opts.binary)
> {
> /* handle binary format */
> }
> else if (cstate->routine && !buildin_format)
> {
> /* custom code, make the copy format extensible */
> }
> else
> {
> /* handle non-binary, (csv or text) format */
> }
> 
> otherwise the {CopyToRoutine| CopyFromRoutine} needs a function pointer
> to distinguish native copy format and extensible supported format,
> like I mentioned above?

Hmm. I may miss something but I think that we don't need the
bool flag. Because we don't set cstate->routine for native
copy formats. So we can distinguish native copy format and
extensible supported format by checking only
cstate->routine.


Thanks,
-- 
kou

  153. Re: Make COPY format extendable: Extract COPY TO format implementations

    jian he <jian.universality@gmail.com> — 2024-03-13T08:00:46Z

    On Mon, Mar 11, 2024 at 8:56 AM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <CACJufxEgn3=j-UWg-f2-DbLO+uVSKGcofpkX5trx+=YX6icSFg@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Mon, 11 Mar 2024 08:00:00 +0800,
>   jian he <jian.universality@gmail.com> wrote:
>
> > Hi, here are my cents:
> > Currently in v17, we have 3 extra functions within DoCopyTo
> > CopyToStart, one time, start, doing some preliminary work.
> > CopyToOneRow, doing the repetitive work, called many times, row by row.
> > CopyToEnd, one time doing the closing work.
> >
> > seems to need a function pointer for processing the format and other options.
> > or maybe the reason is we need a one time function call before doing DoCopyTo,
> > like one time initialization.
>
> I know that JSON format wants it but can we defer it? We can
> add more options later. I want to proceed this improvement
> step by step.
>
> More use cases will help us which callbacks are needed. We
> will be able to collect more use cases by providing basic
> callbacks.

I guess one of the ultimate goals would be that COPY can export data
to a customized format.
Let's say the customized format is "csv1", but it is just analogous to
the csv format.
people should be able to create an extension, with serval C functions,
then they can do `copy (select 1 ) to stdout (format 'csv1');`
but the output will be exact same as `copy (select 1 ) to stdout
(format 'csv');`

In such a scenario, we require a function akin to ProcessCopyOptions
to handle situations
where CopyFormatOptions->csv_mode is true, while the format is "csv1".

but CopyToStart is already within the DoCopyTo function, so you do
need an extra function pointer?
I do agree with the incremental improvement method.

  154. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-03-15T08:37:54Z

    Hi,

In <CACJufxFbffGaxW1LiTNEQAPcuvP1s7GL1Ghi--kbSqsjwh7XeA@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Wed, 13 Mar 2024 16:00:46 +0800,
  jian he <jian.universality@gmail.com> wrote:

>> More use cases will help us which callbacks are needed. We
>> will be able to collect more use cases by providing basic
>> callbacks.

> Let's say the customized format is "csv1", but it is just analogous to
> the csv format.
> people should be able to create an extension, with serval C functions,
> then they can do `copy (select 1 ) to stdout (format 'csv1');`
> but the output will be exact same as `copy (select 1 ) to stdout
> (format 'csv');`

Thanks for sharing one use case but I think that we need
real-world use cases to consider our APIs.

For example, JSON support that is currently discussing in
another thread is a real-world use case. My Apache Arrow
support is also another real-world use case.


Thanks,
-- 
kou

  155. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-03-20T14:27:32Z

    Hi,

Could someone review the v17 patch to proceed this?

The v17 patch:
https://www.postgresql.org/message-id/flat/20240305.171808.667980402249336456.kou%40clear-code.com#d2ee079b75ebcf00c410300ecc4a357a

Some profiles by Michael:
https://www.postgresql.org/message-id/flat/ZelfYatRdVZN3FbE%40paquier.xyz#eccfd1a0131af93c48026d691cc247f4

Thanks,
-- 
kou

In <20240308.092254.359611633589181574.kou@clear-code.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 08 Mar 2024 09:22:54 +0900 (JST),
  Sutou Kouhei <kou@clear-code.com> wrote:

> Hi,
> 
> In <ZelfYatRdVZN3FbE@paquier.xyz>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Thu, 7 Mar 2024 15:32:01 +0900,
>   Michael Paquier <michael@paquier.xyz> wrote:
> 
>> While on it, here are some profiles based on HEAD and v17 with the
>> previous tests (COPY TO /dev/null, COPY FROM data sent to the void).
>> 
> ...
>> 
>> So, in short, and that's not really a surprise, there is no effect
>> once we use the dispatching with the routines only when a format would
>> want to plug-in with the APIs, but a custom format would still have a
>> penalty of a few percents for both if bottlenecked on CPU.
> 
> Thanks for sharing these profiles!
> I agree with you.
> 
> This shows that the v17 approach doesn't affect the current
> text/csv/binary implementations. (The v17 approach just adds
> 2 new structs, Copy{From,To}Rountine, without changing the
> current text/csv/binary implementations.)
> 
> Can we push the v17 patch and proceed following
> implementations? Could someone (especially a PostgreSQL
> committer) take a look at this for double-check?
> 
> 
> Thanks,
> -- 
> kou
> 
>

  156. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-04-10T08:16:26Z

    Hi Andres,

Could you take a look at this? I think that you don't want
to touch the current text/csv/binary implementations. The
v17 patch approach doesn't touch the current text/csv/binary
implementations. What do you think about this approach?


Thanks,
-- 
kou

In <20240320.232732.488684985873786799.kou@clear-code.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Wed, 20 Mar 2024 23:27:32 +0900 (JST),
  Sutou Kouhei <kou@clear-code.com> wrote:

> Hi,
> 
> Could someone review the v17 patch to proceed this?
> 
> The v17 patch:
> https://www.postgresql.org/message-id/flat/20240305.171808.667980402249336456.kou%40clear-code.com#d2ee079b75ebcf00c410300ecc4a357a
> 
> Some profiles by Michael:
> https://www.postgresql.org/message-id/flat/ZelfYatRdVZN3FbE%40paquier.xyz#eccfd1a0131af93c48026d691cc247f4
> 
> Thanks,
> -- 
> kou
> 
> In <20240308.092254.359611633589181574.kou@clear-code.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 08 Mar 2024 09:22:54 +0900 (JST),
>   Sutou Kouhei <kou@clear-code.com> wrote:
> 
>> Hi,
>> 
>> In <ZelfYatRdVZN3FbE@paquier.xyz>
>>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Thu, 7 Mar 2024 15:32:01 +0900,
>>   Michael Paquier <michael@paquier.xyz> wrote:
>> 
>>> While on it, here are some profiles based on HEAD and v17 with the
>>> previous tests (COPY TO /dev/null, COPY FROM data sent to the void).
>>> 
>> ...
>>> 
>>> So, in short, and that's not really a surprise, there is no effect
>>> once we use the dispatching with the routines only when a format would
>>> want to plug-in with the APIs, but a custom format would still have a
>>> penalty of a few percents for both if bottlenecked on CPU.
>> 
>> Thanks for sharing these profiles!
>> I agree with you.
>> 
>> This shows that the v17 approach doesn't affect the current
>> text/csv/binary implementations. (The v17 approach just adds
>> 2 new structs, Copy{From,To}Rountine, without changing the
>> current text/csv/binary implementations.)
>> 
>> Can we push the v17 patch and proceed following
>> implementations? Could someone (especially a PostgreSQL
>> committer) take a look at this for double-check?
>> 
>> 
>> Thanks,
>> -- 
>> kou
>> 
>> 
> 
>

  157. Re: Make COPY format extendable: Extract COPY TO format implementations

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2024-07-19T12:40:05Z

    Hello Kouhei-san,

I think it'd be helpful if you could post a patch status, i.e. a message
re-explaininig what it aims to achieve, summary of the discussion so
far, and what you think are the open questions. Otherwise every reviewer
has to read the whole thread to learn this.

FWIW I realize there are other related patches, and maybe some of the
discussion is happening on those threads. But that's just another reason
to post the summary here - as a reviewer I'm not going to read random
other patches that "might" have relevant info.

-----

The way I understand it, the ultimate goal is to allow extensions to
define formats using CREATE XYZ. And I agree that would be a very
valuable feature. But the proposed patch does not do that, right? It
only does some basic things at the C level, there's no DDL etc.

Per the commit message, none of the existing formats (text/csv/binary)
is implemented as "copy routine". IMHO that's a bit strange, because
that's exactly what I'd expect this patch to do - to define all the
infrastructure (catalogs, ...) and switch the existing formats to it.

Yes, the patch will be larger, but it'll also simplify some of the code
(right now there's a bunch of branches to handle these "old" formats).
How would you even know the new code is correct, when there's nothing
using using the "copy routine" branch?

In fact, doesn't this mean that the benchmarks presented earlier are not
very useful? We still use the old code, except there are a couple "if"
branches that are never taken? I don't think this measures the new
approach would not be slower once everything gets to be copy routine.

Or what am I missing?

Also, how do we know this API is suitable for the alternative formats?
For example you mentioned Arrow, and I suppose people will want to add
support for other column-oriented formats. I assume that will require
stashing a batch of rows (or some other internal state) somewhere, but
does the proposed API plan for that?

My guess would be we'll need to add a "private_data" pointer to the
CopyFromStateData/CopyToStateData structs, but maybe I'm wrong.

Also, won't the alternative formats require custom parameters. For
example, for column-oriented-formats it might be useful to specify a
stripe size (rows per batch), etc. I'm not saying this patch needs to
implement that, but maybe the API should expect it?

-----

To sum this up, what I think needs to happen for this patch to move forward:

1) Switch the existing formats to the new API, to validate the API works
at least for them, allow testing and benchmarking the code.

2) Try implementing some of the more exotic formats (column-oriented) to
test the API works for those too.

3) Maybe try implementing a PoC version to do the DDL, so that it
actually is extensible.

It's not my intent to "move the goalposts" - I think it's fine if the
patches (2) and (3) are just PoC, to validate (1) goes in the right
direction. For example, it's fine if (2) just hard-codes the new format
next to the build-in ones - that's not something we'd commit, I think,
but for validation of (1) it's good enough.

Most of the DDL stuff can probably be "copied" from FDW handlers. It's
pretty similar, and the "routine" idea is what FDW does too. It probably
also shows a good way to "initialize" the routine, etc.


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  158. Re: Make COPY format extendable: Extract COPY TO format implementations

    Li, Yong <yoli@ebay.com> — 2024-07-22T07:11:15Z

    Hi Kou,

I tried to follow the thread but had to skip quite some discussions in the middle part of the thread. From what I read, it appears to me that there were a lot of back-and-forth discussions on the specific implementation details (i.e. do not touch existing format implementation), performance concerns and how to split the patches to make it more manageable.

My understanding is that the provided v17 patch aims to achieve the followings:
- Retain existing format implementations as built-in formats, and do not go through the new interface for them.
- Make sure that there is no sign of performance degradation.
- Refactoring the existing code to make it easier and possible to make copy handlers extensible. However, some of the infrastructure work that are required to make copy handler extensible are intentionally delayed for future patches. Some of the work were proposed as patches in earlier messages, but they were not explicitly referenced in recent messages.

Overall, the current v17 patch applies cleanly to HEAD. “make check-world” also runs cleanly. If my understanding of the current status of the patch is correct, the patch looks good to me.


Regards,
Yong

  159. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-07-22T07:45:40Z

    Hi Tomas,

Thanks for joining this thread!

In <257d5573-07da-48c3-ac07-e047e7a65e99@enterprisedb.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 19 Jul 2024 14:40:05 +0200,
  Tomas Vondra <tomas.vondra@enterprisedb.com> wrote:

> I think it'd be helpful if you could post a patch status, i.e. a message
> re-explaininig what it aims to achieve, summary of the discussion so
> far, and what you think are the open questions. Otherwise every reviewer
> has to read the whole thread to learn this.

It makes sense. It seems your questions covers all important
points in this thread. So my answers of your questions
summarize the latest information.

> FWIW I realize there are other related patches, and maybe some of the
> discussion is happening on those threads. But that's just another reason
> to post the summary here - as a reviewer I'm not going to read random
> other patches that "might" have relevant info.

It makes sense too. To clarify it, other threads are
unrelated. We can focus on only this thread for this propose.

> The way I understand it, the ultimate goal is to allow extensions to
> define formats using CREATE XYZ.

Right.

>                   But the proposed patch does not do that, right? It
> only does some basic things at the C level, there's no DDL etc.

Right. The latest patch set includes only the basic things
for the first implementation.

> Per the commit message, none of the existing formats (text/csv/binary)
> is implemented as "copy routine".

Right.

>                                   IMHO that's a bit strange, because
> that's exactly what I'd expect this patch to do - to define all the
> infrastructure (catalogs, ...) and switch the existing formats to it.

We did it in the v1-v15 patch sets. But the v16/v17 patch
sets remove it because of a profiling result. (It's
described later.)

In general, we don't want to decrease the current
performance of the existing formats:

https://www.postgresql.org/message-id/flat/10025bac-158c-ffe7-fbec-32b42629121f%40dunslane.net#81cf82c219f2f2d77a616bbf5e511a5c

> We've spent quite a lot of blood sweat and tears over the years to make
> COPY fast, and we should not sacrifice any of that lightly.

The v15 patch set is faster than HEAD but there is a
mysterious profiling result:

https://www.postgresql.org/message-id/flat/ZdbtQJ-p5H1_EDwE%40paquier.xyz#6439e6ad574f2d47cd7220e9bfed3889

> The increase in CopyOneRowTo from 80% to 85% worries me
...
>                                                  I am getting faster
> runtimes with v15 (6232ms in average) vs HEAD (6550ms).

I think that it's not a blocker because the v15 patch set
approach is faster. But someone may think that it's a
blocker. So the v16 or later patch sets don't include codes
to use this extension mechanism for the existing formats. We
can work on it after we introduce the basic features if it's
valuable.

> How would you even know the new code is correct, when there's nothing
> using using the "copy routine" branch?

We can't test it only with the v16/v17 patch set
changes. But we can do it by adding more changes we did in
the v6 patch set.
https://www.postgresql.org/message-id/flat/20240124.144936.67229716500876806.kou%40clear-code.com#f1ad092fc5e81fe38d3c376559efd52c

If we should commit the basic changes with tests, I can
adjust the test mechanism in v6 patch set and add it to the
latest patch set. But it needs CREATE XYZ mechanism and
so on too. Is it OK?

> In fact, doesn't this mean that the benchmarks presented earlier are not
> very useful? We still use the old code, except there are a couple "if"
> branches that are never taken? I don't think this measures the new
> approach would not be slower once everything gets to be copy routine.

Here is a benchmark result with the v17 and HEAD:

https://www.postgresql.org/message-id/flat/ZelfYatRdVZN3FbE%40paquier.xyz#eccfd1a0131af93c48026d691cc247f4

It shows that no performance difference for the existing
formats.

The added mechanism may be slower than the existing formats
mechanism but it's not a blocker. Because it's never
performance regression. (Because this is a new feature.)

We can improve it later if it's needed.

> Also, how do we know this API is suitable for the alternative formats?

The v6 patch set has more APIs built on this API. These APIs
are for implementing the alternative formats.

https://www.postgresql.org/message-id/flat/20240124.144936.67229716500876806.kou%40clear-code.com#f1ad092fc5e81fe38d3c376559efd52c

This is an Apache Arrow format implementation based on the
v6 patch set: https://github.com/kou/pg-copy-arrow

> For example you mentioned Arrow, and I suppose people will want to add
> support for other column-oriented formats. I assume that will require
> stashing a batch of rows (or some other internal state) somewhere, but
> does the proposed API plan for that?
>
> My guess would be we'll need to add a "private_data" pointer to the
> CopyFromStateData/CopyToStateData structs, but maybe I'm wrong.

I think so too. The v6 patch set has a "private_data"
pointer. But the v17 patch set doesn't have it because the
v17 patch set has only basic changes. We'll add it and other
features in the following patches:

https://www.postgresql.org/message-id/flat/20240305.171808.667980402249336456.kou%40clear-code.com

> I'll send the following patches after this patch is
> merged. They are based on the v6 patch[1]:
> 
> 1. Add copy_handler
>    * This also adds a pg_proc lookup for custom FORMAT
>    * This also adds a test for copy_handler
> 2. Export CopyToStateData
>    * We need it to implement custom copy TO handler
> 3. Add needed APIs to implement custom copy TO handler
>    * Add CopyToStateData::opaque
>    * Export CopySendEndOfRow()
> 4. Export CopyFromStateData
>    * We need it to implement custom copy FROM handler
> 5. Add needed APIs to implement custom copy FROM handler
>    * Add CopyFromStateData::opaque
>    * Export CopyReadBinaryData()

"Copy{To,From}StateDate::opaque" are the "private_data"
pointer in the v6 patch.

> Also, won't the alternative formats require custom parameters. For
> example, for column-oriented-formats it might be useful to specify a
> stripe size (rows per batch), etc. I'm not saying this patch needs to
> implement that, but maybe the API should expect it?

Yes. The v6 patch set also has the API. But we want to
minimize API set as much as possible in the first
implementation.

https://www.postgresql.org/message-id/flat/Zbi1TwPfAvUpKqTd%40paquier.xyz#00abc60c5a1ad9eee395849b7b5a5e0d

>                           I am really worried about the complexities
> this thread is getting into because we are trying to shape the
> callbacks in the most generic way possible based on *two* use cases.
> This is going to be a never-ending discussion.  I'd rather get some
> simple basics, and then we can discuss if tweaking the callbacks is
> really necessary or not.

And I agree with this approach.

> 1) Switch the existing formats to the new API, to validate the API works
> at least for them, allow testing and benchmarking the code.

I want to keep the current style for the first
implementation to avoid affecting the existing formats
performance. If it's not allowed to move forward this
proposal, could someone help us to solve the mysterious
result (why are %s of CopyOneRowTo() different?) in the
following v15 patch set benchmark result?

https://www.postgresql.org/message-id/flat/ZdbtQJ-p5H1_EDwE%40paquier.xyz#6439e6ad574f2d47cd7220e9bfed3889

> 2) Try implementing some of the more exotic formats (column-oriented) to
> test the API works for those too.
>
> 3) Maybe try implementing a PoC version to do the DDL, so that it
> actually is extensible.
> 
> It's not my intent to "move the goalposts" - I think it's fine if the
> patches (2) and (3) are just PoC, to validate (1) goes in the right
> direction. For example, it's fine if (2) just hard-codes the new format
> next to the build-in ones - that's not something we'd commit, I think,
> but for validation of (1) it's good enough.
>
> Most of the DDL stuff can probably be "copied" from FDW handlers. It's
> pretty similar, and the "routine" idea is what FDW does too. It probably
> also shows a good way to "initialize" the routine, etc.

Is the v6 patch set enough for it?
https://www.postgresql.org/message-id/flat/20240124.144936.67229716500876806.kou%40clear-code.com#f1ad092fc5e81fe38d3c376559efd52c

Or should we do it based on the v17 patch set? If so, I'll
work on it now. It was a plan that I'll do after the v17
patch set is merged.


Thanks,
-- 
kou

  160. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-07-22T08:01:49Z

    Hi Yong,

Thanks for joining this thread!

In <453D52D4-2AC5-49F6-928D-79F8A4C0850E@ebay.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Mon, 22 Jul 2024 07:11:15 +0000,
  "Li, Yong" <yoli@ebay.com> wrote:

> My understanding is that the provided v17 patch aims to achieve the followings:
> - Retain existing format implementations as built-in formats, and do not go through the new interface for them.
> - Make sure that there is no sign of performance degradation.
> - Refactoring the existing code to make it easier and possible to make copy handlers extensible. However, some of the infrastructure work that are required to make copy handler extensible are intentionally delayed for future patches. Some of the work were proposed as patches in earlier messages, but they were not explicitly referenced in recent messages.

Right.

Sorry for bothering you. As Tomas suggested, I should have
prepared the current summary.

My last e-mail summarized the current information:
https://www.postgresql.org/message-id/flat/20240722.164540.889091645042390373.kou%40clear-code.com#0be14c4eeb041e70438ab7a423b728da

It also shows that your understanding is right.


Thanks,
-- 
kou

  161. Re: Make COPY format extendable: Extract COPY TO format implementations

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2024-07-22T12:36:40Z

    On 7/22/24 09:45, Sutou Kouhei wrote:
> Hi Tomas,
> 
> Thanks for joining this thread!
> 
> In <257d5573-07da-48c3-ac07-e047e7a65e99@enterprisedb.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 19 Jul 2024 14:40:05 +0200,
>   Tomas Vondra <tomas.vondra@enterprisedb.com> wrote:
> 
>> I think it'd be helpful if you could post a patch status, i.e. a message
>> re-explaininig what it aims to achieve, summary of the discussion so
>> far, and what you think are the open questions. Otherwise every reviewer
>> has to read the whole thread to learn this.
> 
> It makes sense. It seems your questions covers all important
> points in this thread. So my answers of your questions
> summarize the latest information.
> 

Thanks for the summary/responses. I still think it'd be better to post a
summary as a separate message, not as yet another post responding to
someone else. If I was reading the thread, I would not have noticed this
is meant to be a summary. I'd even consider putting a "THREAD SUMMARY"
title on the first line, or something like that. Up to you, of course.


As for the patch / decisions, thanks for the responses and explanations.
But I still find it hard to review / make judgements about the approach
based on the current version of the patch :-( Yes, it's entirely
possible earlier versions did something interesting - e.g. it might have
implemented the existing formats to the new approach. Or it might have a
private pointer in v6. But how do I know why it was removed? Was it
because it's unnecessary for the initial version? Or was it because it
turned out to not work?

And when reviewing a patch, I really don't want to scavenge through old
patch versions, looking for random parts. Not only because I don't know
what to look for, but also because it'll be harder and harder to make
those old versions work, as the patch moves evolves.

My suggestions would be to maintain this as a series of patches, making
incremental changes, with the "more complex" or "more experimental"
parts larger in the series. For example, I can imagine doing this:

0001 - minimal version of the patch (e.g. current v17)
0002 - switch existing formats to the new interface
0003 - extend the interface to add bits needed for columnar formats
0004 - add DML to create/alter/drop custom implementations
0005 - minimal patch with extension adding support for Arrow

Or something like that. The idea is that we still have a coherent story
of what we're trying to do, and can discuss the incremental changes
(easier than looking at a large patch). It's even possible to commit
earlier parts before the later parts are quite cleanup up for commit.
And some changes changes may not be even meant for commit (e.g. the
extension) but as guidance / validation for the earlier parts.


I do realize this might look like I'm requiring you to do more work.
Sorry about that. I'm just thinking about how to move the patch forward
and convince myself the approach is OK. Also, it's what I think works
quite well for other patches discussed on this mailing list (I do this
for various patches I submitted, for example). And I'm not even sure it
actually is more work.


As for the performance / profiling issues, I've read the reports and I'm
not sure I see something tremendously wrong. Yes, there are differences,
but 5% change can easily be noise, shift in binary layout, etc.

Unfortunately, there's not much information about what exactly the tests
did, context (hardware, ...). So I don't know, really. But if you share
enough information on how to reproduce this, I'm willing to take a look
and investigate.


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  162. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-07-24T08:30:59Z

    Hi,

In <9172d4eb-6de0-4c6d-beab-8210b7a2219b@enterprisedb.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Mon, 22 Jul 2024 14:36:40 +0200,
  Tomas Vondra <tomas.vondra@enterprisedb.com> wrote:

> Thanks for the summary/responses. I still think it'd be better to post a
> summary as a separate message, not as yet another post responding to
> someone else. If I was reading the thread, I would not have noticed this
> is meant to be a summary. I'd even consider putting a "THREAD SUMMARY"
> title on the first line, or something like that. Up to you, of course.

It makes sense. I'll do it as a separated e-mail.

> My suggestions would be to maintain this as a series of patches, making
> incremental changes, with the "more complex" or "more experimental"
> parts larger in the series. For example, I can imagine doing this:
> 
> 0001 - minimal version of the patch (e.g. current v17)
> 0002 - switch existing formats to the new interface
> 0003 - extend the interface to add bits needed for columnar formats
> 0004 - add DML to create/alter/drop custom implementations
> 0005 - minimal patch with extension adding support for Arrow
> 
> Or something like that. The idea is that we still have a coherent story
> of what we're trying to do, and can discuss the incremental changes
> (easier than looking at a large patch). It's even possible to commit
> earlier parts before the later parts are quite cleanup up for commit.
> And some changes changes may not be even meant for commit (e.g. the
> extension) but as guidance / validation for the earlier parts.

OK. I attach the v18 patch set:

0001: add a basic feature (Copy{From,To}Routine)
      (same as the v17 but it's based on the current master)
0002: use Copy{From,To}Rountine for the existing formats
      (this may not be committed because there is a
      profiling related concern)
0003: add support for specifying custom format by "COPY
      ... WITH (format 'my-format')"
      (this also has a test)
0004: export Copy{From,To}StateData
      (but this isn't enough to implement custom COPY
      FROM/TO handlers as an extension)
0005: add opaque member to Copy{From,To}StateData and export
      some functions to read the next data and flush the buffer
      (we can implement a PoC Apache Arrow COPY FROM/TO
      handler as an extension with this)

https://github.com/kou/pg-copy-arrow is a PoC Apache Arrow
COPY FROM/TO handler as an extension.


Notes:

* 0002: We use "static inline" and "constant argument" for
  optimization.
* 0002: This hides NextCopyFromRawFields() in a public
  header because it's not used in PostgreSQL and we want to
  use "static inline" for it. If it's a problem, we can keep
  it and create an internal function for "static inline".
* 0003: We use "CREATE FUNCTION" to register a custom COPY
  FROM/TO handler. It's the same approach as tablesample.
* 0004 and 0005: We can mix them but this patch set split
  them for easy to review. 0004 just moves the existing
  codes. It doesn't change the existing codes.
* PoC: I provide it as a separated repository instead of a
  patch because an extension exists as a separated project
  in general. If it's a problem, I can provide it as a patch
  for contrib/.
* This patch set still has minimal Copy{From,To}Routine. For
  example, custom COPY FROM/TO handlers can't process their
  own options with this patch set. We may add more callbacks
  to Copy{From,To}Routine later based on real world use-cases.

> Unfortunately, there's not much information about what exactly the tests
> did, context (hardware, ...). So I don't know, really. But if you share
> enough information on how to reproduce this, I'm willing to take a look
> and investigate.

Thanks. Here is related information based on the past
e-mails from Michael:

* Use -O2 for optimization build flag
  ("meson setup --buildtype=release" may be used)
* Use tmpfs for PGDATA
* Disable fsync
* Run on scissors (what is "scissors" in this context...?)
  https://www.postgresql.org/message-id/flat/Zbr6piWuVHDtFFOl%40paquier.xyz#dbbec4d5c54ef2317be01a54abaf495c
* Unlogged table may be used
* Use a table that has 30 integer columns (*1)
* Use 5M rows (*2)
* Use '/dev/null' for COPY TO (*3)
* Use blackhole_am for COPY FROM (*4)
  https://github.com/michaelpq/pg_plugins/tree/main/blackhole_am
* perf is used but used options are unknown (sorry)

(*1) This SQL may be used to create the table:

CREATE OR REPLACE FUNCTION create_table_cols(tabname text, num_cols int)
RETURNS VOID AS
$func$
DECLARE
  query text;
BEGIN
  query := 'CREATE UNLOGGED TABLE ' || tabname || ' (';
  FOR i IN 1..num_cols LOOP
    query := query || 'a_' || i::text || ' int default 1';
    IF i != num_cols THEN
      query := query || ', ';
    END IF;
  END LOOP;
  query := query || ')';
  EXECUTE format(query);
END
$func$ LANGUAGE plpgsql;
SELECT create_table_cols ('to_tab_30', 30);
SELECT create_table_cols ('from_tab_30', 30);

(*2) This SQL may be used to insert 5M rows:

INSERT INTO to_tab_30 SELECT FROM generate_series(1, 5000000);

(*3) This SQL may be used for COPY TO:

COPY to_tab_30 TO '/dev/null' WITH (FORMAT text);

(*4) This SQL may be used for COPY FROM:

CREATE EXTENSION blackhole_am;
ALTER TABLE from_tab_30 SET ACCESS METHOD blackhole_am;
COPY to_tab_30 TO '/tmp/to_tab_30.txt' WITH (FORMAT text);
COPY from_tab_30 FROM '/tmp/to_tab_30.txt' WITH (FORMAT text);


If there is enough information, could you try?


Thanks,
-- 
kou

  163. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-07-25T04:51:38Z

    Hi,

THREAD SUMMARY:

Proposal:

How about making COPY format extendable?


Background:

Currently, COPY TO/FROM supports only "text", "csv" and
"binary" formats. There are some requests to support more
COPY formats. For example:


* 2023-11: JSON and JSON lines [1]
* 2022-04: Apache Arrow [2]
* 2018-02: Apache Avro, Apache Parquet and Apache ORC [3]

There were discussions how to add support for more formats. [3][4]
In these discussions, we got a consensus about making COPY
format extendable.

[1]: https://www.postgresql.org/message-id/flat/24e3ee88-ec1e-421b-89ae-8a47ee0d2df1%40joeconway.com#a5e6b8829f9a74dfc835f6f29f2e44c5
[2]: https://www.postgresql.org/message-id/flat/CAGrfaBVyfm0wPzXVqm0%3Dh5uArYh9N_ij%2BsVpUtDHqkB%3DVyB3jw%40mail.gmail.com
[3]: https://www.postgresql.org/message-id/flat/20180210151304.fonjztsynewldfba%40gmail.com
[4]: https://www.postgresql.org/message-id/flat/3741749.1655952719%40sss.pgh.pa.us#2bb7af4a3d2c7669f9a49808d777a20d


Concerns:

* Performance: If we make COPY format extendable, it will
  introduce some overheads. We don't want to loss our
  optimization efforts for the current implementations by
  this. [5]
* Extendability: We don't know which API set is enough for
  custom COPY format implementations yet. We don't want to
  provide too much APIs to reduce maintenance cost.

[5]: https://www.postgresql.org/message-id/3741749.1655952719%40sss.pgh.pa.us


Implementation:

The v18 patch set is the latest patch set. [6]
It includes the following patches:

0001: This adds a basic feature (Copy{From,To}Routine)
      (This isn't enough for extending COPY format.
      This just extracts minimal procedure sets to be
      extendable as callback sets.)
0002: This uses Copy{From,To}Rountine for the existing
      formats (text, csv and binary)
      (This may not be committed because there is a
      profiling related concern. See the following section
      for details)
0003: This adds support for specifying custom format by
      "COPY ... WITH (format 'my-format')"
      (This also adds a test for this feature.)
0004: This exports Copy{From,To}StateData
      (But this isn't enough to implement custom COPY
      FROM/TO handlers as an extension.)
0005: This adds opaque member to Copy{From,To}StateData and
      export some functions to read the next data and flush
      the buffer
      (We can implement a PoC Apache Arrow COPY FROM/TO
      handler as an extension with this. [7])

[6]: https://www.postgresql.org/message-id/flat/20240724.173059.909782980111496972.kou%40clear-code.com
[7]: https://github.com/kou/pg-copy-arrow


Implementation notes:

* 0002: We use "static inline" and "constant argument" for
  optimization.
* 0002: This hides NextCopyFromRawFields() in a public
  header because it's not used in PostgreSQL and we want to
  use "static inline" for it. If it's a problem, we can keep
  it and create an internal function for "static inline".
* 0003: We use "CREATE FUNCTION" to register a custom COPY
  FROM/TO handler. It's the same approach as tablesample.
* 0004 and 0005: We can mix them but this patch set split
  them for easy to review. 0004 just moves the existing
  codes. It doesn't change the existing codes.
* PoC: I provide it as a separated repository instead of a
  patch because an extension exists as a separated project
  in general. If it's a problem, I can provide it as a patch
  for contrib/.
* This patch set still has minimal Copy{From,To}Routine. For
  example, custom COPY FROM/TO handlers can't process their
  own options with this patch set. We may add more callbacks
  to Copy{From,To}Routine later based on real world use-cases.


Performance concern:

We have a benchmark result and a profile for the change that
uses Copy{From,To}Routine for the existing formats. [8] They
are based on the v15 patch but there are no significant
difference between the v15 patch and v18 patch set.

These results show the followings:

* Runtime: The patched version is faster than HEAD.
  * The patched version: 6232ms in average
  * HEAD: 6550ms in average
* Profile: The patched version spends more percents than
  HEAD in a core function.
  * The patched version: 85.61% in CopyOneRowTo()
  * HEAD: 80.35% in CopyOneRowTo()

[8]: https://www.postgresql.org/message-id/flat/ZdbtQJ-p5H1_EDwE%40paquier.xyz


Here are related information for this benchmark/profile:

* Use -O2 for optimization build flag
  ("meson setup --buildtype=release" may be used)
* Use tmpfs for PGDATA
* Disable fsync
* Run on scissors (what is "scissors" in this context...?)   [9]
* Unlogged table may be used
* Use a table that has 30 integer columns (*1)
* Use 5M rows (*2)
* Use '/dev/null' for COPY TO (*3)
* Use blackhole_am for COPY FROM (*4)
  https://github.com/michaelpq/pg_plugins/tree/main/blackhole_am
* perf is used but used options are unknown (sorry)


(*1) This SQL may be used to create the table:

CREATE OR REPLACE FUNCTION create_table_cols(tabname text, num_cols int)
RETURNS VOID AS
$func$
DECLARE
  query text;
BEGIN
  query := 'CREATE UNLOGGED TABLE ' || tabname || ' (';
  FOR i IN 1..num_cols LOOP
    query := query || 'a_' || i::text || ' int default 1';
    IF i != num_cols THEN
      query := query || ', ';
    END IF;
  END LOOP;
  query := query || ')';
  EXECUTE format(query);
END
$func$ LANGUAGE plpgsql;
SELECT create_table_cols ('to_tab_30', 30);
SELECT create_table_cols ('from_tab_30', 30);


(*2) This SQL may be used to insert 5M rows:

INSERT INTO to_tab_30 SELECT FROM generate_series(1, 5000000);


(*3) This SQL may be used for COPY TO:

COPY to_tab_30 TO '/dev/null' WITH (FORMAT text);


(*4) This SQL may be used for COPY FROM:

CREATE EXTENSION blackhole_am;
ALTER TABLE from_tab_30 SET ACCESS METHOD blackhole_am;
COPY to_tab_30 TO '/tmp/to_tab_30.txt' WITH (FORMAT text);
COPY from_tab_30 FROM '/tmp/to_tab_30.txt' WITH (FORMAT text);


[9]: https://www.postgresql.org/message-id/flat/Zbr6piWuVHDtFFOl%40paquier.xyz#dbbec4d5c54ef2317be01a54abaf495c


Thanks,
-- 
kou

  164. Re: Make COPY format extendable: Extract COPY TO format implementations

    Li, Yong <yoli@ebay.com> — 2024-07-26T08:57:42Z

    
> On Jul 25, 2024, at 12:51, Sutou Kouhei <kou@clear-code.com> wrote:
> 
> Hi,
> 
> THREAD SUMMARY:

Very nice summary.

> 
> Implementation:
> 
> The v18 patch set is the latest patch set. [6]
> It includes the following patches:
> 
> 0001: This adds a basic feature (Copy{From,To}Routine)
>      (This isn't enough for extending COPY format.
>      This just extracts minimal procedure sets to be
>      extendable as callback sets.)
> 0002: This uses Copy{From,To}Rountine for the existing
>      formats (text, csv and binary)
>      (This may not be committed because there is a
>      profiling related concern. See the following section
>      for details)
> 0003: This adds support for specifying custom format by
>      "COPY ... WITH (format 'my-format')"
>      (This also adds a test for this feature.)
> 0004: This exports Copy{From,To}StateData
>      (But this isn't enough to implement custom COPY
>      FROM/TO handlers as an extension.)
> 0005: This adds opaque member to Copy{From,To}StateData and
>      export some functions to read the next data and flush
>      the buffer
>      (We can implement a PoC Apache Arrow COPY FROM/TO
>      handler as an extension with this. [7])
> 
> Thanks,
> --
> kou
> 

This review is for 0001 only because the other patches are not ready
for commit.

The v18-0001 patch applies cleanly to HEAD. “make check-world” also
runs cleanly. The patch looks good for me.


Regards,
Yong

  165. Re: Make COPY format extendable: Extract COPY TO format implementations

    Junwang Zhao <zhjwpku@gmail.com> — 2024-07-28T14:49:47Z

    Hi Sutou,

On Wed, Jul 24, 2024 at 4:31 PM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <9172d4eb-6de0-4c6d-beab-8210b7a2219b@enterprisedb.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Mon, 22 Jul 2024 14:36:40 +0200,
>   Tomas Vondra <tomas.vondra@enterprisedb.com> wrote:
>
> > Thanks for the summary/responses. I still think it'd be better to post a
> > summary as a separate message, not as yet another post responding to
> > someone else. If I was reading the thread, I would not have noticed this
> > is meant to be a summary. I'd even consider putting a "THREAD SUMMARY"
> > title on the first line, or something like that. Up to you, of course.
>
> It makes sense. I'll do it as a separated e-mail.
>
> > My suggestions would be to maintain this as a series of patches, making
> > incremental changes, with the "more complex" or "more experimental"
> > parts larger in the series. For example, I can imagine doing this:
> >
> > 0001 - minimal version of the patch (e.g. current v17)
> > 0002 - switch existing formats to the new interface
> > 0003 - extend the interface to add bits needed for columnar formats
> > 0004 - add DML to create/alter/drop custom implementations
> > 0005 - minimal patch with extension adding support for Arrow
> >
> > Or something like that. The idea is that we still have a coherent story
> > of what we're trying to do, and can discuss the incremental changes
> > (easier than looking at a large patch). It's even possible to commit
> > earlier parts before the later parts are quite cleanup up for commit.
> > And some changes changes may not be even meant for commit (e.g. the
> > extension) but as guidance / validation for the earlier parts.
>
> OK. I attach the v18 patch set:
>
> 0001: add a basic feature (Copy{From,To}Routine)
>       (same as the v17 but it's based on the current master)
> 0002: use Copy{From,To}Rountine for the existing formats
>       (this may not be committed because there is a
>       profiling related concern)
> 0003: add support for specifying custom format by "COPY
>       ... WITH (format 'my-format')"
>       (this also has a test)
> 0004: export Copy{From,To}StateData
>       (but this isn't enough to implement custom COPY
>       FROM/TO handlers as an extension)
> 0005: add opaque member to Copy{From,To}StateData and export
>       some functions to read the next data and flush the buffer
>       (we can implement a PoC Apache Arrow COPY FROM/TO
>       handler as an extension with this)
>
> https://github.com/kou/pg-copy-arrow is a PoC Apache Arrow
> COPY FROM/TO handler as an extension.
>
>
> Notes:
>
> * 0002: We use "static inline" and "constant argument" for
>   optimization.
> * 0002: This hides NextCopyFromRawFields() in a public
>   header because it's not used in PostgreSQL and we want to
>   use "static inline" for it. If it's a problem, we can keep
>   it and create an internal function for "static inline".
> * 0003: We use "CREATE FUNCTION" to register a custom COPY
>   FROM/TO handler. It's the same approach as tablesample.
> * 0004 and 0005: We can mix them but this patch set split
>   them for easy to review. 0004 just moves the existing
>   codes. It doesn't change the existing codes.
> * PoC: I provide it as a separated repository instead of a
>   patch because an extension exists as a separated project
>   in general. If it's a problem, I can provide it as a patch
>   for contrib/.
> * This patch set still has minimal Copy{From,To}Routine. For
>   example, custom COPY FROM/TO handlers can't process their
>   own options with this patch set. We may add more callbacks
>   to Copy{From,To}Routine later based on real world use-cases.
>
> > Unfortunately, there's not much information about what exactly the tests
> > did, context (hardware, ...). So I don't know, really. But if you share
> > enough information on how to reproduce this, I'm willing to take a look
> > and investigate.
>
> Thanks. Here is related information based on the past
> e-mails from Michael:
>
> * Use -O2 for optimization build flag
>   ("meson setup --buildtype=release" may be used)
> * Use tmpfs for PGDATA
> * Disable fsync
> * Run on scissors (what is "scissors" in this context...?)
>   https://www.postgresql.org/message-id/flat/Zbr6piWuVHDtFFOl%40paquier.xyz#dbbec4d5c54ef2317be01a54abaf495c
> * Unlogged table may be used
> * Use a table that has 30 integer columns (*1)
> * Use 5M rows (*2)
> * Use '/dev/null' for COPY TO (*3)
> * Use blackhole_am for COPY FROM (*4)
>   https://github.com/michaelpq/pg_plugins/tree/main/blackhole_am
> * perf is used but used options are unknown (sorry)
>
> (*1) This SQL may be used to create the table:
>
> CREATE OR REPLACE FUNCTION create_table_cols(tabname text, num_cols int)
> RETURNS VOID AS
> $func$
> DECLARE
>   query text;
> BEGIN
>   query := 'CREATE UNLOGGED TABLE ' || tabname || ' (';
>   FOR i IN 1..num_cols LOOP
>     query := query || 'a_' || i::text || ' int default 1';
>     IF i != num_cols THEN
>       query := query || ', ';
>     END IF;
>   END LOOP;
>   query := query || ')';
>   EXECUTE format(query);
> END
> $func$ LANGUAGE plpgsql;
> SELECT create_table_cols ('to_tab_30', 30);
> SELECT create_table_cols ('from_tab_30', 30);
>
> (*2) This SQL may be used to insert 5M rows:
>
> INSERT INTO to_tab_30 SELECT FROM generate_series(1, 5000000);
>
> (*3) This SQL may be used for COPY TO:
>
> COPY to_tab_30 TO '/dev/null' WITH (FORMAT text);
>
> (*4) This SQL may be used for COPY FROM:
>
> CREATE EXTENSION blackhole_am;
> ALTER TABLE from_tab_30 SET ACCESS METHOD blackhole_am;
> COPY to_tab_30 TO '/tmp/to_tab_30.txt' WITH (FORMAT text);
> COPY from_tab_30 FROM '/tmp/to_tab_30.txt' WITH (FORMAT text);
>
>
> If there is enough information, could you try?
>
Thanks for updating the patches, I applied them and test
in my local machine, I did not use tmpfs in my test, I guess
if I run the tests enough rounds, the OS will cache the
pages, below is my numbers(I run each test 30 times, I
count for the last 10 ones):

HEAD                                PATCHED

COPY to_tab_30 TO '/dev/null' WITH (FORMAT text);

5628.280 ms                   5679.860 ms
5583.144 ms                   5588.078 ms
5604.444 ms                   5628.029 ms
5617.133 ms                   5613.926 ms
5575.570 ms                   5601.045 ms
5634.828 ms                   5616.409 ms
5693.489 ms                   5637.434 ms
5585.857 ms                   5609.531 ms
5613.948 ms                   5643.629 ms
5610.394 ms                   5580.206 ms

COPY from_tab_30 FROM '/tmp/to_tab_30.txt' WITH (FORMAT text);

3929.955 ms                   4050.895 ms
3909.061 ms                   3890.156 ms
3940.272 ms                   3927.614 ms
3907.535 ms                   3925.560 ms
3952.719 ms                   3942.141 ms
3933.751 ms                   3904.250 ms
3958.274 ms                   4025.581 ms
3937.065 ms                   3894.149 ms
3949.896 ms                   3933.878 ms
3925.399 ms                   3936.170 ms

I did not see obvious performance degradation, maybe it's
because I did not use tmpfs, but I think this OTH means
that the *function call* and *if branch* added for each row
is not the bottleneck of the whole execution path.

In 0001,

+typedef struct CopyFromRoutine
+{
+ /*
+ * Called when COPY FROM is started to set up the input functions
+ * associated to the relation's attributes writing to.  `finfo` can be
+ * optionally filled to provide the catalog information of the input
+ * function.  `typioparam` can be optionally filled to define the OID of
+ * the type to pass to the input function.  `atttypid` is the OID of data
+ * type used by the relation's attribute.

+typedef struct CopyToRoutine
+{
+ /*
+ * Called when COPY TO is started to set up the output functions
+ * associated to the relation's attributes reading from.  `finfo` can be
+ * optionally filled.  `atttypid` is the OID of data type used by the
+ * relation's attribute.

The second comment has a simplified description for `finfo`, I think it
should match the first by:

`finfo` can be optionally filled to provide the catalog information of the
output function.

After I post the patch diffs, the gmail grammer shows some hints that
it should be *associated with* rather than *associated to*, but I'm
not sure about this one.

I think the patches are in good shape, I can help to do some
further tests if needed, thanks for working on this.

>
> Thanks,
> --
> kou



-- 
Regards
Junwang Zhao

  166. Re: Make COPY format extendable: Extract COPY TO format implementations

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2024-07-29T12:17:08Z

    On 7/25/24 06:51, Sutou Kouhei wrote:
> Hi,
> 
> ...
>
> Here are related information for this benchmark/profile:
> 
> * Use -O2 for optimization build flag
>   ("meson setup --buildtype=release" may be used)
> * Use tmpfs for PGDATA
> * Disable fsync
> * Run on scissors (what is "scissors" in this context...?)   [9]
> * Unlogged table may be used
> * Use a table that has 30 integer columns (*1)
> * Use 5M rows (*2)
> * Use '/dev/null' for COPY TO (*3)
> * Use blackhole_am for COPY FROM (*4)
>   https://github.com/michaelpq/pg_plugins/tree/main/blackhole_am
> * perf is used but used options are unknown (sorry)
> 
> 
> (*1) This SQL may be used to create the table:
> 
> CREATE OR REPLACE FUNCTION create_table_cols(tabname text, num_cols int)
> RETURNS VOID AS
> $func$
> DECLARE
>   query text;
> BEGIN
>   query := 'CREATE UNLOGGED TABLE ' || tabname || ' (';
>   FOR i IN 1..num_cols LOOP
>     query := query || 'a_' || i::text || ' int default 1';
>     IF i != num_cols THEN
>       query := query || ', ';
>     END IF;
>   END LOOP;
>   query := query || ')';
>   EXECUTE format(query);
> END
> $func$ LANGUAGE plpgsql;
> SELECT create_table_cols ('to_tab_30', 30);
> SELECT create_table_cols ('from_tab_30', 30);
> 
> 
> (*2) This SQL may be used to insert 5M rows:
> 
> INSERT INTO to_tab_30 SELECT FROM generate_series(1, 5000000);
> 
> 
> (*3) This SQL may be used for COPY TO:
> 
> COPY to_tab_30 TO '/dev/null' WITH (FORMAT text);
> 
> 
> (*4) This SQL may be used for COPY FROM:
> 
> CREATE EXTENSION blackhole_am;
> ALTER TABLE from_tab_30 SET ACCESS METHOD blackhole_am;
> COPY to_tab_30 TO '/tmp/to_tab_30.txt' WITH (FORMAT text);
> COPY from_tab_30 FROM '/tmp/to_tab_30.txt' WITH (FORMAT text);
> 

Thanks for the benchmark instructions and updated patches. Very helpful!

I wrote a simple script to automate the benchmark - it just runs these
tests with different parameters (number of columns and number of
imported/exported rows). See the run.sh attachment, along with two CSV
results from current master and with all patches applied.

The attached PDF has a simple summary, with a median duration for each
combination, and a comparison (patched/master). The results are from my
laptop, so it's probably noisy, and it would be good to test it on a
more realistic hardware (for perf-sensitive things).

- For COPY FROM there is no difference - the results are within 1% of
master, and there's no systemic difference.

- For COPY TO it's a different story, though. There's a pretty clear
regression, by ~5%. It's a bit interesting the correlation with the
number of columns is not stronger ...

I did do some basic profiling, and the perf diff looks like this:

# Event 'task-clock:upppH'
#
# Baseline  Delta Abs  Shared Object  Symbol

# ........  .........  .............
.........................................
#
    13.34%    -12.94%  postgres       [.] CopyOneRowTo
              +10.75%  postgres       [.] CopyToTextOneRow
     4.31%     +2.84%  postgres       [.] pg_ltoa
    10.96%     +1.15%  postgres       [.] CopySendChar
     8.68%     +0.78%  postgres       [.] AllocSetAlloc
    10.89%     -0.70%  postgres       [.] CopyAttributeOutText
     5.01%     -0.47%  postgres       [.] enlargeStringInfo
     4.95%     -0.42%  postgres       [.] OutputFunctionCall
     5.29%     -0.37%  postgres       [.] int4out
     5.90%     -0.31%  postgres       [.] appendBinaryStringInfo
               +0.29%  postgres       [.] CopyToStateFlush
     0.27%     -0.27%  postgres       [.] memcpy@plt

Not particularly surprising that CopyToTextOneRow has +11%, but that's
because it's a new function. The perf difference is perhaps due to
pg_ltoa/CopySendChar, but not sure why.

I also did some flamegraph - attached is for master, patched and diff.

It's interesting the main change in the flamegraphs is CopyToStateFlush
pops up on the left side. Because, what is that about? That is a thing
introduced in the 0005 patch, so maybe the regression is not strictly
about the existing formats moving to the new API, but due to something
else in a later version of the patch?

It would be good do run the tests for each patch in the series, and then
see when does the regression actually appear.

FWIW None of this actually proves this is an issue in practice. No one
will be exporting into /dev/null or importing into blackhole, and I'd
bet the difference gets way smaller for more realistic cases.

regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  167. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-07-30T02:58:24Z

    Hi,

In <CAEG8a3+KN=uofw5ksnCwh5s3m_VcfFYd=jTzcpO5uVLBHwSQEg@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Sun, 28 Jul 2024 22:49:47 +0800,
  Junwang Zhao <zhjwpku@gmail.com> wrote:

> Thanks for updating the patches, I applied them and test
> in my local machine, I did not use tmpfs in my test, I guess
> if I run the tests enough rounds, the OS will cache the
> pages, below is my numbers(I run each test 30 times, I
> count for the last 10 ones):
> 
> HEAD                                PATCHED
> 
> COPY to_tab_30 TO '/dev/null' WITH (FORMAT text);
> 
> 5628.280 ms                   5679.860 ms
> 5583.144 ms                   5588.078 ms
> 5604.444 ms                   5628.029 ms
> 5617.133 ms                   5613.926 ms
> 5575.570 ms                   5601.045 ms
> 5634.828 ms                   5616.409 ms
> 5693.489 ms                   5637.434 ms
> 5585.857 ms                   5609.531 ms
> 5613.948 ms                   5643.629 ms
> 5610.394 ms                   5580.206 ms
> 
> COPY from_tab_30 FROM '/tmp/to_tab_30.txt' WITH (FORMAT text);
> 
> 3929.955 ms                   4050.895 ms
> 3909.061 ms                   3890.156 ms
> 3940.272 ms                   3927.614 ms
> 3907.535 ms                   3925.560 ms
> 3952.719 ms                   3942.141 ms
> 3933.751 ms                   3904.250 ms
> 3958.274 ms                   4025.581 ms
> 3937.065 ms                   3894.149 ms
> 3949.896 ms                   3933.878 ms
> 3925.399 ms                   3936.170 ms
> 
> I did not see obvious performance degradation, maybe it's
> because I did not use tmpfs, but I think this OTH means
> that the *function call* and *if branch* added for each row
> is not the bottleneck of the whole execution path.

Thanks for sharing your numbers. I agree with there are no
obvious performance degradation.


> In 0001,
> 
> +typedef struct CopyFromRoutine
> +{
> + /*
> + * Called when COPY FROM is started to set up the input functions
> + * associated to the relation's attributes writing to.  `finfo` can be
> + * optionally filled to provide the catalog information of the input
> + * function.  `typioparam` can be optionally filled to define the OID of
> + * the type to pass to the input function.  `atttypid` is the OID of data
> + * type used by the relation's attribute.
> 
> +typedef struct CopyToRoutine
> +{
> + /*
> + * Called when COPY TO is started to set up the output functions
> + * associated to the relation's attributes reading from.  `finfo` can be
> + * optionally filled.  `atttypid` is the OID of data type used by the
> + * relation's attribute.
> 
> The second comment has a simplified description for `finfo`, I think it
> should match the first by:
> 
> `finfo` can be optionally filled to provide the catalog information of the
> output function.

Good catch. I'll update it as suggested in the next patch set.

> After I post the patch diffs, the gmail grammer shows some hints that
> it should be *associated with* rather than *associated to*, but I'm
> not sure about this one.

Thanks. I'll use "associated with".

> I think the patches are in good shape, I can help to do some
> further tests if needed, thanks for working on this.

Thanks!

-- 
kou

  168. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-07-30T07:13:06Z

    Hi,

In <26541788-8853-4d93-86cd-5f701b13ae51@enterprisedb.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Mon, 29 Jul 2024 14:17:08 +0200,
  Tomas Vondra <tomas.vondra@enterprisedb.com> wrote:

> I wrote a simple script to automate the benchmark - it just runs these
> tests with different parameters (number of columns and number of
> imported/exported rows). See the run.sh attachment, along with two CSV
> results from current master and with all patches applied.

Thanks. I also used the script with some modifications:

1. Create a test database automatically
2. Enable blackhole_am automatically
3. Create create_table_cols() automatically

I attach it. I also attach results of master and patched. My
results are from my desktop. So it's probably noisy.

> - For COPY FROM there is no difference - the results are within 1% of
> master, and there's no systemic difference.
> 
> - For COPY TO it's a different story, though. There's a pretty clear
> regression, by ~5%. It's a bit interesting the correlation with the
> number of columns is not stronger ...

My results showed different trend:

- COPY FROM: Patched is about 15-20% slower than master
- COPY TO: Patched is a bit faster than master

Here are some my numbers:

type	n_cols	n_rows	diff	master		patched
----------------------------------------------------------
TO	5	1	100.56%	218.376000	219.609000
FROM	5	1	113.33%	168.493000	190.954000
...
TO	5	5	100.60%	1037.773000	1044.045000
FROM	5	5	116.46%	767.966000	894.377000
...
TO	5	10	100.15%	2092.245000	2095.472000
FROM	5	10	115.91%	1508.160000	1748.130000
TO	10	1	98.62%	353.087000	348.214000
FROM	10	1	118.65%	260.551000	309.133000
...
TO	10	5	96.89%	1724.061000	1670.427000
FROM	10	5	119.92%	1224.098000	1467.941000
...
TO	10	10	98.70%	3444.291000	3399.538000
FROM	10	10	118.79%	2462.314000	2924.866000
TO	15	1	97.71%	492.082000	480.802000
FROM	15	1	115.59%	347.820000	402.033000
...
TO	15	5	98.32%	2402.419000	2362.140000
FROM	15	5	115.48%	1657.594000	1914.245000
...
TO	15	10	96.91%	4830.319000	4681.145000
FROM	15	10	115.09%	3304.798000	3803.542000
TO	20	1	96.05%	629.828000	604.939000
FROM	20	1	118.50%	438.673000	519.839000
...
TO	20	5	97.15%	3084.210000	2996.331000
FROM	20	5	115.35%	2110.909000	2435.032000
...
TO	25	1	98.29%	764.779000	751.684000
FROM	25	1	115.13%	519.686000	598.301000
...
TO	25	5	94.08%	3843.996000	3616.614000
FROM	25	5	115.62%	2554.008000	2952.928000
...
TO	25	10	97.41%	7504.865000	7310.549000
FROM	25	10	117.25%	4994.463000	5856.029000
TO	30	1	94.39%	906.324000	855.503000
FROM	30	1	119.60%	604.110000	722.491000
...
TO	30	5	96.50%	4419.907000	4265.417000
FROM	30	5	116.97%	2932.883000	3430.556000
...
TO	30	10	94.39%	8974.878000	8470.991000
FROM	30	10	117.84%	5800.793000	6835.900000
----

See the attached diff.txt for full numbers.
I also attach scripts to generate the diff.txt. Here is the
command line I used:

----
ruby diff.rb <(ruby aggregate.rb master.result) <(ruby aggregate.rb patched.result) | tee diff.txt
----

My environment:

* Debian GNU/Linux sid
* gcc (Debian 13.3.0-2) 13.3.0
* AMD Ryzen 9 3900X 12-Core Processor

I'll look into this.

If someone is interested in this proposal, could you share
your numbers?

> It's interesting the main change in the flamegraphs is CopyToStateFlush
> pops up on the left side. Because, what is that about? That is a thing
> introduced in the 0005 patch, so maybe the regression is not strictly
> about the existing formats moving to the new API, but due to something
> else in a later version of the patch?

Ah, making static CopySendEndOfRow() a to non-static function
(CopyToStateFlush()) may be the reason of this. Could you
try the attached v19 patch? It changes the 0005 patch:

* It reverts the static change
* It adds a new non-static function that just exports
  CopySendEndOfRow()


Thanks,
-- 
kou

  169. Re: Make COPY format extendable: Extract COPY TO format implementations

    Tomas Vondra <tomas.vondra@enterprisedb.com> — 2024-07-30T09:51:37Z

    
On 7/30/24 09:13, Sutou Kouhei wrote:
> Hi,
> 
> In <26541788-8853-4d93-86cd-5f701b13ae51@enterprisedb.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Mon, 29 Jul 2024 14:17:08 +0200,
>   Tomas Vondra <tomas.vondra@enterprisedb.com> wrote:
> 
>> I wrote a simple script to automate the benchmark - it just runs these
>> tests with different parameters (number of columns and number of
>> imported/exported rows). See the run.sh attachment, along with two CSV
>> results from current master and with all patches applied.
> 
> Thanks. I also used the script with some modifications:
> 
> 1. Create a test database automatically
> 2. Enable blackhole_am automatically
> 3. Create create_table_cols() automatically
> 
> I attach it. I also attach results of master and patched. My
> results are from my desktop. So it's probably noisy.
> 
>> - For COPY FROM there is no difference - the results are within 1% of
>> master, and there's no systemic difference.
>>
>> - For COPY TO it's a different story, though. There's a pretty clear
>> regression, by ~5%. It's a bit interesting the correlation with the
>> number of columns is not stronger ...
> 
> My results showed different trend:
> 
> - COPY FROM: Patched is about 15-20% slower than master
> - COPY TO: Patched is a bit faster than master
> 
> Here are some my numbers:
> 
> type	n_cols	n_rows	diff	master		patched
> ----------------------------------------------------------
> TO	5	1	100.56%	218.376000	219.609000
> FROM	5	1	113.33%	168.493000	190.954000
> ...
> TO	5	5	100.60%	1037.773000	1044.045000
> FROM	5	5	116.46%	767.966000	894.377000
> ...
> TO	5	10	100.15%	2092.245000	2095.472000
> FROM	5	10	115.91%	1508.160000	1748.130000
> TO	10	1	98.62%	353.087000	348.214000
> FROM	10	1	118.65%	260.551000	309.133000
> ...
> TO	10	5	96.89%	1724.061000	1670.427000
> FROM	10	5	119.92%	1224.098000	1467.941000
> ...
> TO	10	10	98.70%	3444.291000	3399.538000
> FROM	10	10	118.79%	2462.314000	2924.866000
> TO	15	1	97.71%	492.082000	480.802000
> FROM	15	1	115.59%	347.820000	402.033000
> ...
> TO	15	5	98.32%	2402.419000	2362.140000
> FROM	15	5	115.48%	1657.594000	1914.245000
> ...
> TO	15	10	96.91%	4830.319000	4681.145000
> FROM	15	10	115.09%	3304.798000	3803.542000
> TO	20	1	96.05%	629.828000	604.939000
> FROM	20	1	118.50%	438.673000	519.839000
> ...
> TO	20	5	97.15%	3084.210000	2996.331000
> FROM	20	5	115.35%	2110.909000	2435.032000
> ...
> TO	25	1	98.29%	764.779000	751.684000
> FROM	25	1	115.13%	519.686000	598.301000
> ...
> TO	25	5	94.08%	3843.996000	3616.614000
> FROM	25	5	115.62%	2554.008000	2952.928000
> ...
> TO	25	10	97.41%	7504.865000	7310.549000
> FROM	25	10	117.25%	4994.463000	5856.029000
> TO	30	1	94.39%	906.324000	855.503000
> FROM	30	1	119.60%	604.110000	722.491000
> ...
> TO	30	5	96.50%	4419.907000	4265.417000
> FROM	30	5	116.97%	2932.883000	3430.556000
> ...
> TO	30	10	94.39%	8974.878000	8470.991000
> FROM	30	10	117.84%	5800.793000	6835.900000
> ----
> 
> See the attached diff.txt for full numbers.
> I also attach scripts to generate the diff.txt. Here is the
> command line I used:
> 
> ----
> ruby diff.rb <(ruby aggregate.rb master.result) <(ruby aggregate.rb patched.result) | tee diff.txt
> ----
> 
> My environment:
> 
> * Debian GNU/Linux sid
> * gcc (Debian 13.3.0-2) 13.3.0
> * AMD Ryzen 9 3900X 12-Core Processor
> 
> I'll look into this.
> 
> If someone is interested in this proposal, could you share
> your numbers?
> 

I'm on Fedora 40 with gcc 14.1, on Intel i7-9750H. But it's running on
Qubes OS, so it's really in a VM which makes it noisier. I'll try to do
more benchmarks on a regular hw, but that may take a couple days.

I decided to do the benchmark for individual parts of the patch series.
The attached PDF shows results for master (label 0000) and the 0001-0005
patches, along with relative performance difference between the patches.
The color scale is the same as before - red = bad, green = good.

There are pretty clear differences between the patches and "direction"
of the COPY. I'm sure it does depend on the hardware - I tried running
this on rpi5 (with 32-bits), and it looks very different. There might be
a similar behavior difference between Intel and Ryzen, but my point is
that when looking for regressions, looking at these "per patch" charts
can be very useful (as it reduces the scope of changes that might have
caused the regression).

>> It's interesting the main change in the flamegraphs is CopyToStateFlush
>> pops up on the left side. Because, what is that about? That is a thing
>> introduced in the 0005 patch, so maybe the regression is not strictly
>> about the existing formats moving to the new API, but due to something
>> else in a later version of the patch?
> 
> Ah, making static CopySendEndOfRow() a to non-static function
> (CopyToStateFlush()) may be the reason of this. Could you
> try the attached v19 patch? It changes the 0005 patch:
> 

Perhaps, that's possible.

> * It reverts the static change
> * It adds a new non-static function that just exports
>   CopySendEndOfRow()
> 

I'll try to benchmark this later, when the other machine is available.


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  170. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-08-01T10:54:12Z

    Hi,

In <b1c8c9fa-06c5-4b60-a2b3-d2b4bedbbde9@enterprisedb.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Tue, 30 Jul 2024 11:51:37 +0200,
  Tomas Vondra <tomas.vondra@enterprisedb.com> wrote:

> I decided to do the benchmark for individual parts of the patch series.
> The attached PDF shows results for master (label 0000) and the 0001-0005
> patches, along with relative performance difference between the patches.
> The color scale is the same as before - red = bad, green = good.
> 
> There are pretty clear differences between the patches and "direction"
> of the COPY. I'm sure it does depend on the hardware - I tried running
> this on rpi5 (with 32-bits), and it looks very different. There might be
> a similar behavior difference between Intel and Ryzen, but my point is
> that when looking for regressions, looking at these "per patch" charts
> can be very useful (as it reduces the scope of changes that might have
> caused the regression).

Thanks.
The numbers on your environment shows that there are
performance problems in the following cases in the v18 patch
set:

1. 0001 + TO
2. 0005 + TO

There are +-~3% differences in FROM cases. They may be noise.
+~6% differences in TO cases may not be noise.

I also tried another benchmark with the v19 (not v18) patch
set with "Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz" not "AMD
Ryzen 9 3900X 12-Core Processor".

The attached PDF visualized my numbers like your PDF but red
= bad, green = good. -30 (blue) means 70% (faster) and 30
(red) means 130% (slower).

0001 + TO is a bit slower like your numbers. Other TO cases
are a bit faster.
0002 + FROM is very slower. Other FROM cases are slower with
less records but a bit faster with many records.

I'll re-run it with "AMD Ryzen 9 3900X 12-Core Processor".


FYI: I've created a repository to push benchmark scripts:
https://gitlab.com/ktou/pg-bench


Thanks,
-- 
kou

  171. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-08-04T22:20:12Z

    Hi,

I re-ran the benchmark(*) with the v19 patch set and the
following CPUs:

1. AMD Ryzen 9 3900X 12-Core Processor
2. Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz

(*)
* Use tables that have {5,10,15,20,25,30} integer columns
* Use tables that have {1,2,3,4,5,6,7,8,9,10}M rows
* Use '/dev/null' for COPY TO
* Use blackhole_am for COPY FROM

See the attached graphs for details.

Notes:
* X-axis is the number of columns
* Y-axis is the number of M rows
* Z-axis is the elapsed time percent (smaller is faster,
  e.g. 99% is a bit faster than the HEAD and 101% is a bit
  slower than the HEAD)
* Z-ranges aren't same (The Ryzen case uses about 79%-121%
  but the Intel case uses about 91%-111%)
* Red means the patch is slower than HEAD
* Blue means the patch is faster than HEAD
* The upper row shows FROM results
* The lower row shows TO results

Here are summaries based on the results:

For FROM:
* With Ryzen: It shows that negative performance impact
* With Intel: It shows that negative performance impact with
  1-5M rows and positive performance impact with 6M-10M rows
For TO:
* With Ryzen: It shows that positive performance impact
* With Intel: It shows that positive performance impact

Here are insights based on the results:

* 0001 (that introduces Copy{From,To}Routine} and adds some
  "if () {...}" for them but the existing formats still
  doesn't use them) has a bit negative performance impact
* 0002 (that migrates the existing codes to
  Copy{From,To}Routine} based implementations) has positive
  performance impact
  * For FROM: Negative impact by 0001 and positive impact by
    0002 almost balanced
    * We should use both of 0001 and 0002 than only 0001
    * With Ryzon: It's a bit slower than HEAD. So we may not
      want to reject this propose for FROM
    * With Intel:
      * With 1-5M rows: It's a bit slower than HEAD
      * With 6-10M rows: It's a bit faster than HEAD
  * For TO: Positive impact by 0002 is larger than negative
    impact by 0002
    * We should use both of 0001 and 0002 than only 0001
* 0003 (that makes Copy{From,To}Routine Node) has a bit
  negative performance impact
  * But I don't know why. This doesn't change per row
    related codes. Increasing Copy{From,To}Routine size
    (NodeTag is added) may be related.
* 0004 (that moves Copy{From,To}StateData to copyapi.h)
  doesn't have impact
  * It makes sense because this doesn't change any
    implementations.
* 0005 (that add "void *opaque" to Copy{From,To}StateData)
  has a bit negative impact for FROM and a bit positive
  impact for TO
  * But I don't know why. This doesn't change per row
    related codes. Increasing Copy{From,To}StateData size
    ("void *opaque" is added) may be related.


How to proceed this proposal?

* Do we need more numbers to judge this proposal?
  * If so, could someone help us?
* There is no negative performance impact for TO with both
  of Ryzen and Intel based on my results. Can we merge only
  the TO part?
  * Can we defer the FROM part? Should we proceed this
    proposal with both of the FROM and TO part?
* Could someone provide a hint why the FROM part is more
  slower with Ryzen?

(If nobody responds to this, this proposal will get stuck
again. If you're interested in this proposal, could you help
us?)


How to run this benchmark on your machine:

$ cd your-postgres
$ git switch -c copy-format-extendable
$ git am v19-*.patch
$ git clone https://gitlab.com/ktou/pg-bench.git ../pg-bench
$ ../pg-bench/bench.sh copy-format-extendable ../pg-bench/copy-format-extendable/run.sh
(This will take about 5 hours...)

If you want to visualize your results on your machine:

$ sudo gem install ruby-gr
$ ../pg-bench/visualize.rb 5

If you share your results to me, I can visualize it and
share.


Thanks,
-- 
kou

  172. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2024-09-27T23:33:13Z

    Hi,

On Sun, Aug 4, 2024 at 3:20 PM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> I re-ran the benchmark(*) with the v19 patch set and the
> following CPUs:
>
> 1. AMD Ryzen 9 3900X 12-Core Processor
> 2. Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz
>
> (*)
> * Use tables that have {5,10,15,20,25,30} integer columns
> * Use tables that have {1,2,3,4,5,6,7,8,9,10}M rows
> * Use '/dev/null' for COPY TO
> * Use blackhole_am for COPY FROM
>
> See the attached graphs for details.
>
> Notes:
> * X-axis is the number of columns
> * Y-axis is the number of M rows
> * Z-axis is the elapsed time percent (smaller is faster,
>   e.g. 99% is a bit faster than the HEAD and 101% is a bit
>   slower than the HEAD)
> * Z-ranges aren't same (The Ryzen case uses about 79%-121%
>   but the Intel case uses about 91%-111%)
> * Red means the patch is slower than HEAD
> * Blue means the patch is faster than HEAD
> * The upper row shows FROM results
> * The lower row shows TO results
>
> Here are summaries based on the results:
>
> For FROM:
> * With Ryzen: It shows that negative performance impact
> * With Intel: It shows that negative performance impact with
>   1-5M rows and positive performance impact with 6M-10M rows
> For TO:
> * With Ryzen: It shows that positive performance impact
> * With Intel: It shows that positive performance impact
>
> Here are insights based on the results:
>
> * 0001 (that introduces Copy{From,To}Routine} and adds some
>   "if () {...}" for them but the existing formats still
>   doesn't use them) has a bit negative performance impact
> * 0002 (that migrates the existing codes to
>   Copy{From,To}Routine} based implementations) has positive
>   performance impact
>   * For FROM: Negative impact by 0001 and positive impact by
>     0002 almost balanced
>     * We should use both of 0001 and 0002 than only 0001
>     * With Ryzon: It's a bit slower than HEAD. So we may not
>       want to reject this propose for FROM
>     * With Intel:
>       * With 1-5M rows: It's a bit slower than HEAD
>       * With 6-10M rows: It's a bit faster than HEAD
>   * For TO: Positive impact by 0002 is larger than negative
>     impact by 0002
>     * We should use both of 0001 and 0002 than only 0001
> * 0003 (that makes Copy{From,To}Routine Node) has a bit
>   negative performance impact
>   * But I don't know why. This doesn't change per row
>     related codes. Increasing Copy{From,To}Routine size
>     (NodeTag is added) may be related.
> * 0004 (that moves Copy{From,To}StateData to copyapi.h)
>   doesn't have impact
>   * It makes sense because this doesn't change any
>     implementations.
> * 0005 (that add "void *opaque" to Copy{From,To}StateData)
>   has a bit negative impact for FROM and a bit positive
>   impact for TO
>   * But I don't know why. This doesn't change per row
>     related codes. Increasing Copy{From,To}StateData size
>     ("void *opaque" is added) may be related.

I was surprised that the 0005 patch made COPY FROM slower (with fewer
rows) and COPY TO faster overall in spite of just adding one struct
field and some functions.

I'm interested in why the performance trends of COPY FROM are
different between fewer than 6M rows and more than 6M rows.

>
> How to proceed this proposal?
>
> * Do we need more numbers to judge this proposal?
>   * If so, could someone help us?
> * There is no negative performance impact for TO with both
>   of Ryzen and Intel based on my results. Can we merge only
>   the TO part?
>   * Can we defer the FROM part? Should we proceed this
>     proposal with both of the FROM and TO part?
> * Could someone provide a hint why the FROM part is more
>   slower with Ryzen?
>

Separating the patches into two parts (one is for COPY TO and another
one is for COPY FROM) could be a good idea. It would help reviews and
investigate performance regression in COPY FROM cases. And I think we
can commit them separately.

Also, could you please rebase the patches as they conflict with the
current HEAD? I'll run some benchmarks on my environment as well.

Regards,


--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  173. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-09-28T15:56:45Z

    Hi,

In <CAD21AoCwMmwLJ8PQLnZu0MbB4gDJiMvWrHREQD4xRp3-F2RU2Q@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 27 Sep 2024 16:33:13 -0700,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

>> * 0005 (that add "void *opaque" to Copy{From,To}StateData)
>>   has a bit negative impact for FROM and a bit positive
>>   impact for TO
>>   * But I don't know why. This doesn't change per row
>>     related codes. Increasing Copy{From,To}StateData size
>>     ("void *opaque" is added) may be related.
> 
> I was surprised that the 0005 patch made COPY FROM slower (with fewer
> rows) and COPY TO faster overall in spite of just adding one struct
> field and some functions.

Me too...

> I'm interested in why the performance trends of COPY FROM are
> different between fewer than 6M rows and more than 6M rows.

My hypothesis:

With this patch set:
  1. One row processing is faster than master.
  2. Non row related processing is slower than master.

If we have many rows, 1. impact is greater than 2. impact.


> Separating the patches into two parts (one is for COPY TO and another
> one is for COPY FROM) could be a good idea. It would help reviews and
> investigate performance regression in COPY FROM cases. And I think we
> can commit them separately.
> 
> Also, could you please rebase the patches as they conflict with the
> current HEAD?

OK. I've prepared 2 patch sets:

v20: It just rebased on master. It still mixes COPY TO and
COPY FROM implementations.

v21: It's based on v20 but splits COPY TO implementations
and COPY FROM implementations.
0001-0005 includes only COPY TO related changes.
0006-0010 includes only COPY FROM related changes.

(v21 0001 + 0006) == (v20 v0001),
(v21 0002 + 0007) == (v20 v0002) and so on.

>               I'll run some benchmarks on my environment as well.

Thanks. It's very helpful.


Thanks,
-- 
kou

  174. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2024-10-07T22:23:08Z

    On Sat, Sep 28, 2024 at 8:56 AM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <CAD21AoCwMmwLJ8PQLnZu0MbB4gDJiMvWrHREQD4xRp3-F2RU2Q@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 27 Sep 2024 16:33:13 -0700,
>   Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> >> * 0005 (that add "void *opaque" to Copy{From,To}StateData)
> >>   has a bit negative impact for FROM and a bit positive
> >>   impact for TO
> >>   * But I don't know why. This doesn't change per row
> >>     related codes. Increasing Copy{From,To}StateData size
> >>     ("void *opaque" is added) may be related.
> >
> > I was surprised that the 0005 patch made COPY FROM slower (with fewer
> > rows) and COPY TO faster overall in spite of just adding one struct
> > field and some functions.
>
> Me too...
>
> > I'm interested in why the performance trends of COPY FROM are
> > different between fewer than 6M rows and more than 6M rows.
>
> My hypothesis:
>
> With this patch set:
>   1. One row processing is faster than master.
>   2. Non row related processing is slower than master.
>
> If we have many rows, 1. impact is greater than 2. impact.
>
>
> > Separating the patches into two parts (one is for COPY TO and another
> > one is for COPY FROM) could be a good idea. It would help reviews and
> > investigate performance regression in COPY FROM cases. And I think we
> > can commit them separately.
> >
> > Also, could you please rebase the patches as they conflict with the
> > current HEAD?
>
> OK. I've prepared 2 patch sets:
>
> v20: It just rebased on master. It still mixes COPY TO and
> COPY FROM implementations.
>
> v21: It's based on v20 but splits COPY TO implementations
> and COPY FROM implementations.
> 0001-0005 includes only COPY TO related changes.
> 0006-0010 includes only COPY FROM related changes.
>
> (v21 0001 + 0006) == (v20 v0001),
> (v21 0002 + 0007) == (v20 v0002) and so on.
>
> >               I'll run some benchmarks on my environment as well.
>

Thank you for updating the patches!

I've run the same benchmark script on my various machines (Mac, Linux
(with Intel CPU and Ryzen CPU) and Raspberry Pi etc). I've not
investigated the results in depth yet but let me share the results.
Please find the attached file, extensible_copy_benchmark_20241007.pdf.

In the benchmark, I've applied the v20 patch set and 'master' in the
result refers to a19f83f87966. And I disabled CPU turbo boost where
possible. Overall, v20 patch got a similar or better performance in
both COPY FROM and COPY TO compared to master except for on MacOS. I'm
not sure that changes made to master since the last benchmark run by
Tomas and Suto-san might contribute to these results. I'll try to
investigate the performance regression that happened on MacOS. I think
that other performance differences in my results seem to be within
noises and could be acceptable. Of course, it would be great if others
also could try to run benchmark tests.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  175. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-10-08T08:39:18Z

    Hi,

In <CAD21AoD67TAO6KkBecKBsLgR1tgYJS1AwiN9NQJSLE0WYw8pDA@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Mon, 7 Oct 2024 15:23:08 -0700,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

> I've run the same benchmark script on my various machines (Mac, Linux
> (with Intel CPU and Ryzen CPU) and Raspberry Pi etc). I've not
> investigated the results in depth yet but let me share the results.
> Please find the attached file, extensible_copy_benchmark_20241007.pdf.
> 
> In the benchmark, I've applied the v20 patch set and 'master' in the
> result refers to a19f83f87966. And I disabled CPU turbo boost where
> possible. Overall, v20 patch got a similar or better performance in
> both COPY FROM and COPY TO compared to master except for on MacOS. I'm
> not sure that changes made to master since the last benchmark run by
> Tomas and Suto-san might contribute to these results. I'll try to
> investigate the performance regression that happened on MacOS. I think
> that other performance differences in my results seem to be within
> noises and could be acceptable. Of course, it would be great if others
> also could try to run benchmark tests.

Thanks! They're interesting...

I've run the same benchmark script for the v20 patch set on
the same Intel Core machine when I used for the v19 patch
set:
https://www.postgresql.org/message-id/20240805.072012.2006870620510018355.kou%40clear-code.com

See the attached v{19,20}-*-result.pdf. (v19-*-result.pdf is
the same PDF attached in the above e-mail.) There are no
significant differences. So I think that there are no
related changes in master...


Thanks,
-- 
kou

  176. Re: Make COPY format extendable: Extract COPY TO format implementations

    Michael Paquier <michael@paquier.xyz> — 2024-10-09T03:34:14Z

    On Mon, Oct 07, 2024 at 03:23:08PM -0700, Masahiko Sawada wrote:
> In the benchmark, I've applied the v20 patch set and 'master' in the
> result refers to a19f83f87966. And I disabled CPU turbo boost where
> possible. Overall, v20 patch got a similar or better performance in
> both COPY FROM and COPY TO compared to master except for on MacOS.
> I'm not sure that changes made to master since the last benchmark run by
> Tomas and Suto-san might contribute to these results.

Don't think so.  FWIW, I have been looking at the set of tests with
previous patch versions around v7 and v10 I have done, and did notice
a similar pattern where COPY FROM was getting slightly better for text
and binary.  It did not look like only noise involved, and it was
kind of reproducible.  As long as we avoid the function pointer
redirection for the per-row processing when dealing with in-core
formats, we should be fine as far as I understand.  That's what the
latest patch set is doing based on a read of v21.

> I'll try to investigate the performance regression that happened on MacOS.

I don't have a good explanation for this one.  Did you mount the data
folder on a tmpfs and made sure that all the workloads were
CPU-bounded?

> I think that other performance differences in my results seem to be within
> noises and could be acceptable. Of course, it would be great if others
> also could try to run benchmark tests.

Yeah.  At 1~2% it could be noise, but there are reproducible 1~2%
evolutions.  In the good sense here, it means.
--
Michael

  177. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2024-10-10T22:55:34Z

    On Tue, Oct 8, 2024 at 8:34 PM Michael Paquier <michael@paquier.xyz> wrote:
>
> On Mon, Oct 07, 2024 at 03:23:08PM -0700, Masahiko Sawada wrote:
> > In the benchmark, I've applied the v20 patch set and 'master' in the
> > result refers to a19f83f87966. And I disabled CPU turbo boost where
> > possible. Overall, v20 patch got a similar or better performance in
> > both COPY FROM and COPY TO compared to master except for on MacOS.
> > I'm not sure that changes made to master since the last benchmark run by
> > Tomas and Suto-san might contribute to these results.
>
> Don't think so.  FWIW, I have been looking at the set of tests with
> previous patch versions around v7 and v10 I have done, and did notice
> a similar pattern where COPY FROM was getting slightly better for text
> and binary.  It did not look like only noise involved, and it was
> kind of reproducible.  As long as we avoid the function pointer
> redirection for the per-row processing when dealing with in-core
> formats, we should be fine as far as I understand.  That's what the
> latest patch set is doing based on a read of v21.

Yeah, what v21 patch is doing makes sense to me.

>
> > I'll try to investigate the performance regression that happened on MacOS.
>
> I don't have a good explanation for this one.  Did you mount the data
> folder on a tmpfs and made sure that all the workloads were
> CPU-bounded?

Yes, I used tmpfs and workloads were CPU-bound.

>
> > I think that other performance differences in my results seem to be within
> > noises and could be acceptable. Of course, it would be great if others
> > also could try to run benchmark tests.
>
> Yeah.  At 1~2% it could be noise, but there are reproducible 1~2%
> evolutions.  In the good sense here, it means.

In real workloads, COPY FROM/TO operations would be more disk I/O
bound. I think that 1~2% performance differences that were shown in
CPU-bound workload would not be a problem in practice.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  178. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2024-11-05T06:19:07Z

    On Thu, Oct 10, 2024 at 3:55 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Tue, Oct 8, 2024 at 8:34 PM Michael Paquier <michael@paquier.xyz> wrote:
> >
> > On Mon, Oct 07, 2024 at 03:23:08PM -0700, Masahiko Sawada wrote:
> > > In the benchmark, I've applied the v20 patch set and 'master' in the
> > > result refers to a19f83f87966. And I disabled CPU turbo boost where
> > > possible. Overall, v20 patch got a similar or better performance in
> > > both COPY FROM and COPY TO compared to master except for on MacOS.
> > > I'm not sure that changes made to master since the last benchmark run by
> > > Tomas and Suto-san might contribute to these results.
> >
> > Don't think so.  FWIW, I have been looking at the set of tests with
> > previous patch versions around v7 and v10 I have done, and did notice
> > a similar pattern where COPY FROM was getting slightly better for text
> > and binary.  It did not look like only noise involved, and it was
> > kind of reproducible.  As long as we avoid the function pointer
> > redirection for the per-row processing when dealing with in-core
> > formats, we should be fine as far as I understand.  That's what the
> > latest patch set is doing based on a read of v21.
>
> Yeah, what v21 patch is doing makes sense to me.

I've further investigated the performance regression, and found out it
might be relevant that the compiler doesn't inline the
CopyFromTextLikeOneRow() function. It might be worth testing with
pg_attribute_always_inline instead of 'inline' as below:

diff --git a/src/backend/commands/copyfromparse.c
b/src/backend/commands/copyfromparse.c
index 1a5e1ef711..9f8f839d6c 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -861,7 +861,7 @@ NextCopyFromRawFields(CopyFromState cstate, char
***fields, int *nfields, bool i
  *
  * Workhorse for CopyFromTextOneRow() and CopyFromCSVOneRow().
  */
-static inline bool
+static pg_attribute_always_inline bool
 CopyFromTextLikeOneRow(CopyFromState cstate,
                       ExprContext *econtext,
                       Datum *values,
diff --git a/src/backend/commands/copyto.c b/src/backend/commands/copyto.c
index aa99459b26..fde2d41316 100644
--- a/src/backend/commands/copyto.c
+++ b/src/backend/commands/copyto.c
@@ -170,7 +170,7 @@ CopyToTextLikeOutFunc(CopyToState cstate, Oid
atttypid, FmgrInfo *finfo)
  *
  * Workhorse for CopyToTextOneRow() and CopyToCSVOneRow().
  */
-static inline void
+static pg_attribute_always_inline void
 CopyToTextLikeOneRow(CopyToState cstate,
                     TupleTableSlot *slot,
                     bool is_csv)

In my environment (RHEL 8.9, Intel Xeon Platinum 8375C, EC2 m6id.metal
instance, GCC 12.2.0), the performance got better with
pg_attribute_always_inline. I've confirmed that for the case where
there is a performance regression, that function was not actually
inlined by checking the object file.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  179. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-11-05T08:43:28Z

    Hi,

In <CAD21AoAdj-EJOH1o2fTLke-uskSvuenT--fKW9nkLzYcLwU_eg@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Mon, 4 Nov 2024 22:19:07 -0800,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

> I've further investigated the performance regression, and found out it
> might be relevant that the compiler doesn't inline the
> CopyFromTextLikeOneRow() function. It might be worth testing with
> pg_attribute_always_inline instead of 'inline' as below:

Wow! Good catch!

I've rebased on the current master and updated the v20 and
v21 patch sets with "pg_attribute_always_inline" not
"inline".

The v22 patch set is for the v20 patch set.
(TO/FROM changes are in one commit.)

The v23 patch set is for the v21 patch set.
(TO/FROM changes are separated for easy to merge only FROM
or TO part.)


I'll run benchmark on my environment again.


Thanks,
-- 
kou

  180. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-11-14T07:19:48Z

    Hi,

In <20241105.174328.1705956947135248653.kou@clear-code.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Tue, 05 Nov 2024 17:43:28 +0900 (JST),
  Sutou Kouhei <kou@clear-code.com> wrote:

>> I've further investigated the performance regression, and found out it
>> might be relevant that the compiler doesn't inline the
>> CopyFromTextLikeOneRow() function. It might be worth testing with
>> pg_attribute_always_inline instead of 'inline' as below:
> 
> Wow! Good catch!
> 
> I've rebased on the current master and updated the v20 and
> v21 patch sets with "pg_attribute_always_inline" not
> "inline".
> 
> The v22 patch set is for the v20 patch set.
> (TO/FROM changes are in one commit.)
> 
> The v23 patch set is for the v21 patch set.
> (TO/FROM changes are separated for easy to merge only FROM
> or TO part.)

I've run benchmark on my machine that has "Intel(R) Core(TM)
i7-3770 CPU @ 3.40GHz".

Summary:
* "pg_attribute_always_inline" is effective for the "COPY
  FROM" part
* "pg_attribute_always_inline" may not be needed for the
  "COPY TO" part


v20-result.pdf: This is the same result PDF attached in
https://www.postgresql.org/message-id/20241008.173918.995935870630354246.kou%40clear-code.com
. This is the base line for "pg_attribute_always_inline"
change.

v22-result.pdf: This is a new result PDF for the v22 patch
set.

COPY FROM:

0001: The v22 patch set is slower than HEAD. This just
introduces "routine" abstraction. It increases overhead. So
this is expected.

0002-0005: The v22 patch set is faster than HEAD for all
cases. The v20 patch set is slower than HEAD for smaller
data. This shows that "pg_attribute_always_inline" for the
"COPY FROM" part is effective on my machine too.


COPY TO:

0001: The v22 patch set is slower than HEAD. This is
as expected for the same reason as COPY FROM.

0002-0004: The v22 patch set is slower than HEAD. (The v20
patch set is faster than HEAD.) This is not expected.

0005: The v22 patch set is faster than HEAD. This is
expected. But 0005 just exports some functions. It doesn't
change existing logic. So it's strange...

This shows "pg_attribute_always_inline" is needless for the
"COPY TO" part.


I also tried the v24 patch set:
* The "COPY FROM" part is same as the v22 patch set
  ("pg_attribute_always_inline" is used.)
* The "COPY TO" part is same as the v20 patch set
  ("pg_attribute_always_inline" is NOT used.)


(I think that the v24 patch set isn't useful for others. So
I don't share it here. If you're interested in it, I'll
share it here.)

v24-result.pdf:

COPY FROM: The same trend as the v22 patch set result. It's
expected because the "COPY FROM" part is the same as the v22
patch set.

COPY TO: The v24 patch set is faster than the v22 patch set
but the v24 patch set isn't same trend as the v20 patch
set. This is not expected because the "COPY TO" part is the
same as the v20 patch set.


Anyway, the 0005 "COPY TO" parts are always faster than
HEAD. So we can use either "pg_attribute_always_inline" or
"inline".


Summary:
* "pg_attribute_always_inline" is effective for the "COPY
  FROM" part
* "pg_attribute_always_inline" may not be needed for the
  "COPY TO" part


Can we proceed this proposal with these results? Or should
we wait for more benchmark results?


Thanks,
-- 
kou

  181. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2024-11-15T00:04:41Z

    On Wed, Nov 13, 2024 at 11:19 PM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <20241105.174328.1705956947135248653.kou@clear-code.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Tue, 05 Nov 2024 17:43:28 +0900 (JST),
>   Sutou Kouhei <kou@clear-code.com> wrote:
>
> >> I've further investigated the performance regression, and found out it
> >> might be relevant that the compiler doesn't inline the
> >> CopyFromTextLikeOneRow() function. It might be worth testing with
> >> pg_attribute_always_inline instead of 'inline' as below:
> >
> > Wow! Good catch!
> >
> > I've rebased on the current master and updated the v20 and
> > v21 patch sets with "pg_attribute_always_inline" not
> > "inline".
> >
> > The v22 patch set is for the v20 patch set.
> > (TO/FROM changes are in one commit.)
> >
> > The v23 patch set is for the v21 patch set.
> > (TO/FROM changes are separated for easy to merge only FROM
> > or TO part.)
>
> I've run benchmark on my machine that has "Intel(R) Core(TM)
> i7-3770 CPU @ 3.40GHz".
>
> Summary:
> * "pg_attribute_always_inline" is effective for the "COPY
>   FROM" part
> * "pg_attribute_always_inline" may not be needed for the
>   "COPY TO" part
>
>
> v20-result.pdf: This is the same result PDF attached in
> https://www.postgresql.org/message-id/20241008.173918.995935870630354246.kou%40clear-code.com
> . This is the base line for "pg_attribute_always_inline"
> change.
>
> v22-result.pdf: This is a new result PDF for the v22 patch
> set.
>
> COPY FROM:
>
> 0001: The v22 patch set is slower than HEAD. This just
> introduces "routine" abstraction. It increases overhead. So
> this is expected.
>
> 0002-0005: The v22 patch set is faster than HEAD for all
> cases. The v20 patch set is slower than HEAD for smaller
> data. This shows that "pg_attribute_always_inline" for the
> "COPY FROM" part is effective on my machine too.
>
>
> COPY TO:
>
> 0001: The v22 patch set is slower than HEAD. This is
> as expected for the same reason as COPY FROM.
>
> 0002-0004: The v22 patch set is slower than HEAD. (The v20
> patch set is faster than HEAD.) This is not expected.
>
> 0005: The v22 patch set is faster than HEAD. This is
> expected. But 0005 just exports some functions. It doesn't
> change existing logic. So it's strange...
>
> This shows "pg_attribute_always_inline" is needless for the
> "COPY TO" part.
>
>
> I also tried the v24 patch set:
> * The "COPY FROM" part is same as the v22 patch set
>   ("pg_attribute_always_inline" is used.)
> * The "COPY TO" part is same as the v20 patch set
>   ("pg_attribute_always_inline" is NOT used.)
>
>
> (I think that the v24 patch set isn't useful for others. So
> I don't share it here. If you're interested in it, I'll
> share it here.)
>
> v24-result.pdf:
>
> COPY FROM: The same trend as the v22 patch set result. It's
> expected because the "COPY FROM" part is the same as the v22
> patch set.
>
> COPY TO: The v24 patch set is faster than the v22 patch set
> but the v24 patch set isn't same trend as the v20 patch
> set. This is not expected because the "COPY TO" part is the
> same as the v20 patch set.
>
>
> Anyway, the 0005 "COPY TO" parts are always faster than
> HEAD. So we can use either "pg_attribute_always_inline" or
> "inline".
>
>
> Summary:
> * "pg_attribute_always_inline" is effective for the "COPY
>   FROM" part
> * "pg_attribute_always_inline" may not be needed for the
>   "COPY TO" part
>
>
> Can we proceed this proposal with these results? Or should
> we wait for more benchmark results?

Thank you for sharing the benchmark test results! I think these
results are good for us to proceed. I'll closely look at COPY TO
results and review v22 patch sets.

Regards,


-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  182. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2024-11-19T01:02:41Z

    On Thu, Nov 14, 2024 at 4:04 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Wed, Nov 13, 2024 at 11:19 PM Sutou Kouhei <kou@clear-code.com> wrote:
> >
> > Hi,
> >
> > In <20241105.174328.1705956947135248653.kou@clear-code.com>
> >   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Tue, 05 Nov 2024 17:43:28 +0900 (JST),
> >   Sutou Kouhei <kou@clear-code.com> wrote:
> >
> > >> I've further investigated the performance regression, and found out it
> > >> might be relevant that the compiler doesn't inline the
> > >> CopyFromTextLikeOneRow() function. It might be worth testing with
> > >> pg_attribute_always_inline instead of 'inline' as below:
> > >
> > > Wow! Good catch!
> > >
> > > I've rebased on the current master and updated the v20 and
> > > v21 patch sets with "pg_attribute_always_inline" not
> > > "inline".
> > >
> > > The v22 patch set is for the v20 patch set.
> > > (TO/FROM changes are in one commit.)
> > >
> > > The v23 patch set is for the v21 patch set.
> > > (TO/FROM changes are separated for easy to merge only FROM
> > > or TO part.)
> >
> > I've run benchmark on my machine that has "Intel(R) Core(TM)
> > i7-3770 CPU @ 3.40GHz".
> >
> > Summary:
> > * "pg_attribute_always_inline" is effective for the "COPY
> >   FROM" part
> > * "pg_attribute_always_inline" may not be needed for the
> >   "COPY TO" part
> >
> >
> > v20-result.pdf: This is the same result PDF attached in
> > https://www.postgresql.org/message-id/20241008.173918.995935870630354246.kou%40clear-code.com
> > . This is the base line for "pg_attribute_always_inline"
> > change.
> >
> > v22-result.pdf: This is a new result PDF for the v22 patch
> > set.
> >
> > COPY FROM:
> >
> > 0001: The v22 patch set is slower than HEAD. This just
> > introduces "routine" abstraction. It increases overhead. So
> > this is expected.
> >
> > 0002-0005: The v22 patch set is faster than HEAD for all
> > cases. The v20 patch set is slower than HEAD for smaller
> > data. This shows that "pg_attribute_always_inline" for the
> > "COPY FROM" part is effective on my machine too.
> >
> >
> > COPY TO:
> >
> > 0001: The v22 patch set is slower than HEAD. This is
> > as expected for the same reason as COPY FROM.
> >
> > 0002-0004: The v22 patch set is slower than HEAD. (The v20
> > patch set is faster than HEAD.) This is not expected.
> >
> > 0005: The v22 patch set is faster than HEAD. This is
> > expected. But 0005 just exports some functions. It doesn't
> > change existing logic. So it's strange...
> >
> > This shows "pg_attribute_always_inline" is needless for the
> > "COPY TO" part.
> >
> >
> > I also tried the v24 patch set:
> > * The "COPY FROM" part is same as the v22 patch set
> >   ("pg_attribute_always_inline" is used.)
> > * The "COPY TO" part is same as the v20 patch set
> >   ("pg_attribute_always_inline" is NOT used.)
> >
> >
> > (I think that the v24 patch set isn't useful for others. So
> > I don't share it here. If you're interested in it, I'll
> > share it here.)
> >
> > v24-result.pdf:
> >
> > COPY FROM: The same trend as the v22 patch set result. It's
> > expected because the "COPY FROM" part is the same as the v22
> > patch set.
> >
> > COPY TO: The v24 patch set is faster than the v22 patch set
> > but the v24 patch set isn't same trend as the v20 patch
> > set. This is not expected because the "COPY TO" part is the
> > same as the v20 patch set.
> >
> >
> > Anyway, the 0005 "COPY TO" parts are always faster than
> > HEAD. So we can use either "pg_attribute_always_inline" or
> > "inline".
> >
> >
> > Summary:
> > * "pg_attribute_always_inline" is effective for the "COPY
> >   FROM" part
> > * "pg_attribute_always_inline" may not be needed for the
> >   "COPY TO" part
> >
> >
> > Can we proceed this proposal with these results? Or should
> > we wait for more benchmark results?
>
> Thank you for sharing the benchmark test results! I think these
> results are good for us to proceed. I'll closely look at COPY TO
> results and review v22 patch sets.

I have a question about v22. We use pg_attribute_always_inline for
some functions to avoid function call overheads. Applying it to
CopyToTextLikeOneRow() and CopyFromTextLikeOneRow() are legitimate as
we've discussed. But there are more function where the patch applied
it to:

-bool
-NextCopyFromRawFields(CopyFromState cstate, char ***fields, int *nfields)
+static pg_attribute_always_inline bool
+NextCopyFromRawFields(CopyFromState cstate, char ***fields, int
*nfields, bool is_csv)

-static bool
-CopyReadLineText(CopyFromState cstate)
+static pg_attribute_always_inline bool
+CopyReadLineText(CopyFromState cstate, bool is_csv)

+static pg_attribute_always_inline void
+CopyToTextLikeSendEndOfRow(CopyToState cstate)

I think it's out of scope of this patch even if these changes are
legitimate. Is there any reason for these changes?

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  183. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-11-19T01:31:15Z

    Hi,

In <CAD21AoC=DX5QQVb27C6UdpPfY-F=-PGnQ1u6rWo69DV=4EtDdw@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Mon, 18 Nov 2024 17:02:41 -0800,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

> I have a question about v22. We use pg_attribute_always_inline for
> some functions to avoid function call overheads. Applying it to
> CopyToTextLikeOneRow() and CopyFromTextLikeOneRow() are legitimate as
> we've discussed. But there are more function where the patch applied
> it to:
> 
> -bool
> -NextCopyFromRawFields(CopyFromState cstate, char ***fields, int *nfields)
> +static pg_attribute_always_inline bool
> +NextCopyFromRawFields(CopyFromState cstate, char ***fields, int
> *nfields, bool is_csv)
> 
> -static bool
> -CopyReadLineText(CopyFromState cstate)
> +static pg_attribute_always_inline bool
> +CopyReadLineText(CopyFromState cstate, bool is_csv)
> 
> +static pg_attribute_always_inline void
> +CopyToTextLikeSendEndOfRow(CopyToState cstate)
> 
> I think it's out of scope of this patch even if these changes are
> legitimate. Is there any reason for these changes?

Yes for NextCopyFromRawFields() and CopyReadLineText().
No for CopyToTextLikeSendEndOfRow().

NextCopyFromRawFields() and CopyReadLineText() have "bool
is_csv". So I think that we should use
pg_attribute_always_inline (or inline) like
CopyToTextLikeOneRow() and CopyFromTextLikeOneRow(). I think
that it's not out of scope of this patch because it's a part
of CopyToTextLikeOneRow() and CopyFromTextLikeOneRow()
optimization.

Note: The optimization is based on "bool is_csv" parameter
and constant "true"/"false" argument function call. If we
can inline this function call, all "if (is_csv)" checks in
the function are removed.

pg_attribute_always_inline (or inline) for
CopyToTextLikeSendEndOfRow() is out of scope of this
patch. You're right.

I think that inlining CopyToTextLikeSendEndOfRow() is better
because it's called per row. But it's not related to the
optimization.


Should I create a new patch set without
pg_attribute_always_inline/inline for
CopyToTextLikeSendEndOfRow()? Or could you remove it when
you push?


Thanks,
-- 
kou

  184. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2024-11-19T04:44:25Z

    On Mon, Nov 18, 2024 at 5:31 PM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <CAD21AoC=DX5QQVb27C6UdpPfY-F=-PGnQ1u6rWo69DV=4EtDdw@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Mon, 18 Nov 2024 17:02:41 -0800,
>   Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> > I have a question about v22. We use pg_attribute_always_inline for
> > some functions to avoid function call overheads. Applying it to
> > CopyToTextLikeOneRow() and CopyFromTextLikeOneRow() are legitimate as
> > we've discussed. But there are more function where the patch applied
> > it to:
> >
> > -bool
> > -NextCopyFromRawFields(CopyFromState cstate, char ***fields, int *nfields)
> > +static pg_attribute_always_inline bool
> > +NextCopyFromRawFields(CopyFromState cstate, char ***fields, int
> > *nfields, bool is_csv)
> >
> > -static bool
> > -CopyReadLineText(CopyFromState cstate)
> > +static pg_attribute_always_inline bool
> > +CopyReadLineText(CopyFromState cstate, bool is_csv)
> >
> > +static pg_attribute_always_inline void
> > +CopyToTextLikeSendEndOfRow(CopyToState cstate)
> >
> > I think it's out of scope of this patch even if these changes are
> > legitimate. Is there any reason for these changes?
>
> Yes for NextCopyFromRawFields() and CopyReadLineText().
> No for CopyToTextLikeSendEndOfRow().
>
> NextCopyFromRawFields() and CopyReadLineText() have "bool
> is_csv". So I think that we should use
> pg_attribute_always_inline (or inline) like
> CopyToTextLikeOneRow() and CopyFromTextLikeOneRow(). I think
> that it's not out of scope of this patch because it's a part
> of CopyToTextLikeOneRow() and CopyFromTextLikeOneRow()
> optimization.
>
> Note: The optimization is based on "bool is_csv" parameter
> and constant "true"/"false" argument function call. If we
> can inline this function call, all "if (is_csv)" checks in
> the function are removed.

Understood, thank you for pointing this out.

>
> pg_attribute_always_inline (or inline) for
> CopyToTextLikeSendEndOfRow() is out of scope of this
> patch. You're right.
>
> I think that inlining CopyToTextLikeSendEndOfRow() is better
> because it's called per row. But it's not related to the
> optimization.
>
>
> Should I create a new patch set without
> pg_attribute_always_inline/inline for
> CopyToTextLikeSendEndOfRow()? Or could you remove it when
> you push?

Since I'm reviewing the patch and the patch organization I'll include it.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  185. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2024-11-20T22:14:27Z

    On Mon, Nov 18, 2024 at 8:44 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Mon, Nov 18, 2024 at 5:31 PM Sutou Kouhei <kou@clear-code.com> wrote:
> >
> > Hi,
> >
> > In <CAD21AoC=DX5QQVb27C6UdpPfY-F=-PGnQ1u6rWo69DV=4EtDdw@mail.gmail.com>
> >   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Mon, 18 Nov 2024 17:02:41 -0800,
> >   Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > > I have a question about v22. We use pg_attribute_always_inline for
> > > some functions to avoid function call overheads. Applying it to
> > > CopyToTextLikeOneRow() and CopyFromTextLikeOneRow() are legitimate as
> > > we've discussed. But there are more function where the patch applied
> > > it to:
> > >
> > > -bool
> > > -NextCopyFromRawFields(CopyFromState cstate, char ***fields, int *nfields)
> > > +static pg_attribute_always_inline bool
> > > +NextCopyFromRawFields(CopyFromState cstate, char ***fields, int
> > > *nfields, bool is_csv)
> > >
> > > -static bool
> > > -CopyReadLineText(CopyFromState cstate)
> > > +static pg_attribute_always_inline bool
> > > +CopyReadLineText(CopyFromState cstate, bool is_csv)
> > >
> > > +static pg_attribute_always_inline void
> > > +CopyToTextLikeSendEndOfRow(CopyToState cstate)
> > >
> > > I think it's out of scope of this patch even if these changes are
> > > legitimate. Is there any reason for these changes?
> >
> > Yes for NextCopyFromRawFields() and CopyReadLineText().
> > No for CopyToTextLikeSendEndOfRow().
> >
> > NextCopyFromRawFields() and CopyReadLineText() have "bool
> > is_csv". So I think that we should use
> > pg_attribute_always_inline (or inline) like
> > CopyToTextLikeOneRow() and CopyFromTextLikeOneRow(). I think
> > that it's not out of scope of this patch because it's a part
> > of CopyToTextLikeOneRow() and CopyFromTextLikeOneRow()
> > optimization.
> >
> > Note: The optimization is based on "bool is_csv" parameter
> > and constant "true"/"false" argument function call. If we
> > can inline this function call, all "if (is_csv)" checks in
> > the function are removed.
>
> Understood, thank you for pointing this out.
>
> >
> > pg_attribute_always_inline (or inline) for
> > CopyToTextLikeSendEndOfRow() is out of scope of this
> > patch. You're right.
> >
> > I think that inlining CopyToTextLikeSendEndOfRow() is better
> > because it's called per row. But it's not related to the
> > optimization.
> >
> >
> > Should I create a new patch set without
> > pg_attribute_always_inline/inline for
> > CopyToTextLikeSendEndOfRow()? Or could you remove it when
> > you push?
>
> Since I'm reviewing the patch and the patch organization I'll include it.
>

I've extracted the changes to refactor COPY TO/FROM to use the format
callback routines from v23 patch set, which seems to be a better patch
split to me. Also, I've reviewed these changes and made some changes
on top of them. The attached patches are:

0001: make COPY TO use CopyToRoutine.
0002: minor changes to 0001 patch. will be fixed up.
0003: make COPY FROM use CopyFromRoutine.
0004: minor changes to 0003 patch. will be fixed up.

I've confirmed that v24 has a similar performance improvement to v23.
Please check these extractions and minor change suggestions.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  186. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-11-21T02:55:31Z

    Hi,

In <CAD21AoA1s0nzjGU9t3N_uNdg3SZeOxXyH3rQfxYFEN3Y7JrKRQ@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Wed, 20 Nov 2024 14:14:27 -0800,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

> I've extracted the changes to refactor COPY TO/FROM to use the format
> callback routines from v23 patch set, which seems to be a better patch
> split to me. Also, I've reviewed these changes and made some changes
> on top of them. The attached patches are:
> 
> 0001: make COPY TO use CopyToRoutine.
> 0002: minor changes to 0001 patch. will be fixed up.
> 0003: make COPY FROM use CopyFromRoutine.
> 0004: minor changes to 0003 patch. will be fixed up.
> 
> I've confirmed that v24 has a similar performance improvement to v23.
> Please check these extractions and minor change suggestions.

Thanks. Here are my comments:

0002:

+/* TEXT format */

"text" may be better than "TEXT". We use "text" not "TEXT"
in other places.

+static const CopyToRoutine CopyToRoutineText = {
+	.CopyToStart = CopyToTextLikeStart,
+	.CopyToOutFunc = CopyToTextLikeOutFunc,
+	.CopyToOneRow = CopyToTextOneRow,
+	.CopyToEnd = CopyToTextLikeEnd,
+};

+/* BINARY format */

"binary" may be better than "BINARY". We use "binary" not
"BINARY" in other places.

+static const CopyToRoutine CopyToRoutineBinary = {
+	.CopyToStart = CopyToBinaryStart,
+	.CopyToOutFunc = CopyToBinaryOutFunc,
+	.CopyToOneRow = CopyToBinaryOneRow,
+	.CopyToEnd = CopyToBinaryEnd,
+};

+/* Return COPY TO routines for the given option */

option ->
options

+static const CopyToRoutine *
+CopyToGetRoutine(CopyFormatOptions opts)


0003:

diff --git a/src/include/commands/copyapi.h b/src/include/commands/copyapi.h
index 99981b1579..224fda172e 100644
--- a/src/include/commands/copyapi.h
+++ b/src/include/commands/copyapi.h
@@ -56,4 +56,46 @@ typedef struct CopyToRoutine
 	void		(*CopyToEnd) (CopyToState cstate);
 } CopyToRoutine;
 
+/*
+ * API structure for a COPY FROM format implementation.	 Note this must be

Should we remove a tab character here?

0004:

diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c
index e77986f9a9..7f1de8a42b 100644
--- a/src/backend/commands/copyfrom.c
+++ b/src/backend/commands/copyfrom.c
@@ -106,31 +106,65 @@ typedef struct CopyMultiInsertInfo
 /* non-export function prototypes */
 static void ClosePipeFromProgram(CopyFromState cstate);
 
-
 /*
- * CopyFromRoutine implementations for text and CSV.
+ * built-in format-specific routines. One-row callbacks are defined in

built-in ->
Built-in

 /*
- * CopyFromTextLikeInFunc
- *
- * Assign input function data for a relation's attribute in text/CSV format.
+ * COPY FROM routines for built-in formats.
++

"+" ->
" *"

+/* TEXT format */

TEXT -> text?

+/* BINARY format */

BINARY -> binary?

+/* Return COPY FROM routines for the given option */

option ->
options

+static const CopyFromRoutine *
+CopyFromGetRoutine(CopyFormatOptions opts)

diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index 0447c4df7e..5416583e94 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c

+static bool CopyFromTextLikeOneRow(CopyFromState cstate, ExprContext *econtext,
+								   Datum *values, bool *nulls, bool is_csv);

Oh, I didn't know that we don't need inline in a function
declaration.
 
@@ -1237,7 +1219,7 @@ CopyReadLine(CopyFromState cstate, bool is_csv)
 /*
  * CopyReadLineText - inner loop of CopyReadLine for text mode
  */
-static pg_attribute_always_inline bool
+static bool
 CopyReadLineText(CopyFromState cstate, bool is_csv)

Is this an intentional change?
CopyReadLineText() has "bool in_csv".



Thanks,
-- 
kou

  187. Re: Make COPY format extendable: Extract COPY TO format implementations

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2024-11-21T10:41:18Z

    I ran `make headerscheck` after these patches and it reported a few
problems:

/pgsql/source/master/src/tools/pginclude/headerscheck /pgsql/source/master /pgsql/build/master
In file included from /tmp/headerscheck.xdG40Y/test.c:2:
/pgsql/source/master/src/include/commands/copyapi.h:76:44: error: unknown type name ‘CopyFromState’; did you mean ‘CopyToState’?
   76 |         void            (*CopyFromInFunc) (CopyFromState cstate, Oid atttypid,
      |                                            ^~~~~~~~~~~~~
      |                                            CopyToState
/pgsql/source/master/src/include/commands/copyapi.h:87:43: error: unknown type name ‘CopyFromState’; did you mean ‘CopyToState’?
   87 |         void            (*CopyFromStart) (CopyFromState cstate, TupleDesc tupDesc);
      |                                           ^~~~~~~~~~~~~
      |                                           CopyToState
/pgsql/source/master/src/include/commands/copyapi.h:98:44: error: unknown type name ‘CopyFromState’; did you mean ‘CopyToState’?
   98 |         bool            (*CopyFromOneRow) (CopyFromState cstate, ExprContext *econtext,
      |                                            ^~~~~~~~~~~~~
      |                                            CopyToState
/pgsql/source/master/src/include/commands/copyapi.h:102:41: error: unknown type name ‘CopyFromState’; did you mean ‘CopyToState’?
  102 |         void            (*CopyFromEnd) (CopyFromState cstate);
      |                                         ^~~~~~~~~~~~~
      |                                         CopyToState
/pgsql/source/master/src/include/commands/copyapi.h:103:1: warning: no semicolon at end of struct or union
  103 | } CopyFromRoutine;
      | ^

I think the fix should be the attached.

-- 
Álvaro Herrera               48°01'N 7°57'E  —  https://www.EnterpriseDB.com/
"In Europe they call me Niklaus Wirth; in the US they call me Nickel's worth.
 That's because in Europe they call me by name, and in the US by value!"

  188. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2024-11-22T20:44:19Z

    On Thu, Nov 21, 2024 at 2:41 AM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
>
> I ran `make headerscheck` after these patches and it reported a few
> problems:
>
> /pgsql/source/master/src/tools/pginclude/headerscheck /pgsql/source/master /pgsql/build/master
> In file included from /tmp/headerscheck.xdG40Y/test.c:2:
> /pgsql/source/master/src/include/commands/copyapi.h:76:44: error: unknown type name ‘CopyFromState’; did you mean ‘CopyToState’?
>    76 |         void            (*CopyFromInFunc) (CopyFromState cstate, Oid atttypid,
>       |                                            ^~~~~~~~~~~~~
>       |                                            CopyToState
> /pgsql/source/master/src/include/commands/copyapi.h:87:43: error: unknown type name ‘CopyFromState’; did you mean ‘CopyToState’?
>    87 |         void            (*CopyFromStart) (CopyFromState cstate, TupleDesc tupDesc);
>       |                                           ^~~~~~~~~~~~~
>       |                                           CopyToState
> /pgsql/source/master/src/include/commands/copyapi.h:98:44: error: unknown type name ‘CopyFromState’; did you mean ‘CopyToState’?
>    98 |         bool            (*CopyFromOneRow) (CopyFromState cstate, ExprContext *econtext,
>       |                                            ^~~~~~~~~~~~~
>       |                                            CopyToState
> /pgsql/source/master/src/include/commands/copyapi.h:102:41: error: unknown type name ‘CopyFromState’; did you mean ‘CopyToState’?
>   102 |         void            (*CopyFromEnd) (CopyFromState cstate);
>       |                                         ^~~~~~~~~~~~~
>       |                                         CopyToState
> /pgsql/source/master/src/include/commands/copyapi.h:103:1: warning: no semicolon at end of struct or union
>   103 | } CopyFromRoutine;
>       | ^
>
> I think the fix should be the attached.

Thank you for the report and providing the patch! The fix looks good
to me. I'll incorporate this patch in the next version.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  189. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2024-11-22T21:01:06Z

    On Wed, Nov 20, 2024 at 6:55 PM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <CAD21AoA1s0nzjGU9t3N_uNdg3SZeOxXyH3rQfxYFEN3Y7JrKRQ@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Wed, 20 Nov 2024 14:14:27 -0800,
>   Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> > I've extracted the changes to refactor COPY TO/FROM to use the format
> > callback routines from v23 patch set, which seems to be a better patch
> > split to me. Also, I've reviewed these changes and made some changes
> > on top of them. The attached patches are:
> >
> > 0001: make COPY TO use CopyToRoutine.
> > 0002: minor changes to 0001 patch. will be fixed up.
> > 0003: make COPY FROM use CopyFromRoutine.
> > 0004: minor changes to 0003 patch. will be fixed up.
> >
> > I've confirmed that v24 has a similar performance improvement to v23.
> > Please check these extractions and minor change suggestions.
>
> Thanks. Here are my comments:

Thank you for the comments!

>
> 0002:
>
> +/* TEXT format */
>
> "text" may be better than "TEXT". We use "text" not "TEXT"
> in other places.
>
> +static const CopyToRoutine CopyToRoutineText = {
> +       .CopyToStart = CopyToTextLikeStart,
> +       .CopyToOutFunc = CopyToTextLikeOutFunc,
> +       .CopyToOneRow = CopyToTextOneRow,
> +       .CopyToEnd = CopyToTextLikeEnd,
> +};
>
> +/* BINARY format */
>
> "binary" may be better than "BINARY". We use "binary" not
> "BINARY" in other places.
>
> +static const CopyToRoutine CopyToRoutineBinary = {
> +       .CopyToStart = CopyToBinaryStart,
> +       .CopyToOutFunc = CopyToBinaryOutFunc,
> +       .CopyToOneRow = CopyToBinaryOneRow,
> +       .CopyToEnd = CopyToBinaryEnd,
> +};
>
> +/* Return COPY TO routines for the given option */
>
> option ->
> options
>
> +static const CopyToRoutine *
> +CopyToGetRoutine(CopyFormatOptions opts)

Fixed all the above comments for 0002 patch.

>
>
> 0003:
>
> diff --git a/src/include/commands/copyapi.h b/src/include/commands/copyapi.h
> index 99981b1579..224fda172e 100644
> --- a/src/include/commands/copyapi.h
> +++ b/src/include/commands/copyapi.h
> @@ -56,4 +56,46 @@ typedef struct CopyToRoutine
>         void            (*CopyToEnd) (CopyToState cstate);
>  } CopyToRoutine;
>
> +/*
> + * API structure for a COPY FROM format implementation.         Note this must be
>
> Should we remove a tab character here?

Fixed.

>
> 0004:
>
> diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c
> index e77986f9a9..7f1de8a42b 100644
> --- a/src/backend/commands/copyfrom.c
> +++ b/src/backend/commands/copyfrom.c
> @@ -106,31 +106,65 @@ typedef struct CopyMultiInsertInfo
>  /* non-export function prototypes */
>  static void ClosePipeFromProgram(CopyFromState cstate);
>
> -
>  /*
> - * CopyFromRoutine implementations for text and CSV.
> + * built-in format-specific routines. One-row callbacks are defined in
>
> built-in ->
> Built-in
>
>  /*
> - * CopyFromTextLikeInFunc
> - *
> - * Assign input function data for a relation's attribute in text/CSV format.
> + * COPY FROM routines for built-in formats.
> ++
>
> "+" ->
> " *"
>
> +/* TEXT format */
>
> TEXT -> text?
>
> +/* BINARY format */
>
> BINARY -> binary?
>
> +/* Return COPY FROM routines for the given option */
>
> option ->
> options
>
> +static const CopyFromRoutine *
> +CopyFromGetRoutine(CopyFormatOptions opts)

Fixed.

>
> diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
> index 0447c4df7e..5416583e94 100644
> --- a/src/backend/commands/copyfromparse.c
> +++ b/src/backend/commands/copyfromparse.c
>
> +static bool CopyFromTextLikeOneRow(CopyFromState cstate, ExprContext *econtext,
> +                                                                  Datum *values, bool *nulls, bool is_csv);
>
> Oh, I didn't know that we don't need inline in a function
> declaration.

Removed this function declaration as it's not necessarily necessary.

>
> @@ -1237,7 +1219,7 @@ CopyReadLine(CopyFromState cstate, bool is_csv)
>  /*
>   * CopyReadLineText - inner loop of CopyReadLine for text mode
>   */
> -static pg_attribute_always_inline bool
> +static bool
>  CopyReadLineText(CopyFromState cstate, bool is_csv)
>
> Is this an intentional change?
> CopyReadLineText() has "bool in_csv".

Yes, I'm not sure it's really necessary to make it inline since the
benchmark results don't show much difference. Probably this is because
the function has 'is_csv' in some 'if' branches but the compiler
cannot optimize out the whole 'if' branches as most 'if' branches
check 'is_csv' and other variables.

I've attached the v25 patches that squashed the minor changes I made
in v24 and incorporated all comments I got so far. I think these two
patches are in good shape. Could you rebase remaining patches on top
of them so that we can see the big picture of this feature?

Regarding exposing the structs such as CopyToStateData, v22-0004 patch
moves most of all copy-related structs to copyapi.h from copyto.c,
copyfrom_internal.h, and copy.h, which seems odd to me. I think we can
expose CopyToStateData (and related structs) in a new file
copyto_internal.h and keep other structs in the original header files.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  190. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-11-25T02:06:20Z

    Hi,

In <CAD21AoBNfKDbJnu-zONNpG820ZXYC0fuTSLrJ-UdRqU4qp2wog@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 22 Nov 2024 13:01:06 -0800,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

>> @@ -1237,7 +1219,7 @@ CopyReadLine(CopyFromState cstate, bool is_csv)
>>  /*
>>   * CopyReadLineText - inner loop of CopyReadLine for text mode
>>   */
>> -static pg_attribute_always_inline bool
>> +static bool
>>  CopyReadLineText(CopyFromState cstate, bool is_csv)
>>
>> Is this an intentional change?
>> CopyReadLineText() has "bool in_csv".
> 
> Yes, I'm not sure it's really necessary to make it inline since the
> benchmark results don't show much difference. Probably this is because
> the function has 'is_csv' in some 'if' branches but the compiler
> cannot optimize out the whole 'if' branches as most 'if' branches
> check 'is_csv' and other variables.

I see. If explicit "inline" isn't related to performance, we
don't need explicit "inline".

> I've attached the v25 patches that squashed the minor changes I made
> in v24 and incorporated all comments I got so far. I think these two
> patches are in good shape. Could you rebase remaining patches on top
> of them so that we can see the big picture of this feature?

OK. I'll work on it.

> Regarding exposing the structs such as CopyToStateData, v22-0004 patch
> moves most of all copy-related structs to copyapi.h from copyto.c,
> copyfrom_internal.h, and copy.h, which seems odd to me. I think we can
> expose CopyToStateData (and related structs) in a new file
> copyto_internal.h and keep other structs in the original header files.

Custom COPY format extensions need to use
CopyToStateData/CopyFromStateData. For example,
CopyToStateData::rel is used to retrieve table schema. If we
move CopyToStateData to copyto_internal.h not copyapi.h,
custom COPY format extensions need to include
copyto_internal.h. I feel that it's strange that extensions
need to use internal headers.

What is your real concern? If you don't want to export
CopyToStateData/CopyFromStateData entirely, we can provide
accessors only for some members of them.

FYI: We discussed this in the past. For example:
https://www.postgresql.org/message-id/flat/20240115.152350.1128880926282754664.kou%40clear-code.com#1b523fb95e8fb46702f5568ae19e3649


Thanks,
-- 
kou

  191. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-11-25T06:01:50Z

    Hi,

In <20241125.110620.313152541320718947.kou@clear-code.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Mon, 25 Nov 2024 11:06:20 +0900 (JST),
  Sutou Kouhei <kou@clear-code.com> wrote:

>> I've attached the v25 patches that squashed the minor changes I made
>> in v24 and incorporated all comments I got so far. I think these two
>> patches are in good shape. Could you rebase remaining patches on top
>> of them so that we can see the big picture of this feature?
> 
> OK. I'll work on it.

I've attached the v26 patch set:

0001: It's same as 0001 in the v25 patch set.
0002: It's same as 0002 in the v25 patch set.
0003: It's same as 0003 in the v23 patch set.
      This parses the "format" option and adds support for
      custom TO handler.
0004: It's same as 0004 in the v23 patch set.
      This exports CopyToStateData. But the following
      enums/structs/functions aren't moved to copyapi.h from
      copy.h:
      * CopyHeaderChoice
      * CopyOnErrorChoice
      * CopyLogVerbosityChoice
      * CopyFormatOptions
      * copy_data_dest_cb()
0005: It's same as 0005 in the v23 patch set.
      This adds missing APIs to implement custom TO handler
      as an extension.
0006: It's same as 0008 in the v23 patch set.
      This adds support for custom FROM handler.
0007: It's same as 0009 in the v23 patch set.
      This exports CopyFromStateData.
0008: It's same as 0010 in the v23 patch set.
      This adds missing APIs to implement custom FROM handler
      as an extension.

I've also updated https://github.com/kou/pg-copy-arrow for
the current API.

I think that we can merge only 0001/0002 as the first
step. Because they don't change the current behavior and
they improve performance. We can merge other patches after
that.

>> Regarding exposing the structs such as CopyToStateData, v22-0004 patch
>> moves most of all copy-related structs to copyapi.h from copyto.c,
>> copyfrom_internal.h, and copy.h, which seems odd to me. I think we can
>> expose CopyToStateData (and related structs) in a new file
>> copyto_internal.h and keep other structs in the original header files.
> 
> Custom COPY format extensions need to use
> CopyToStateData/CopyFromStateData. For example,
> CopyToStateData::rel is used to retrieve table schema. If we
> move CopyToStateData to copyto_internal.h not copyapi.h,
> custom COPY format extensions need to include
> copyto_internal.h. I feel that it's strange that extensions
> need to use internal headers.
> 
> What is your real concern? If you don't want to export
> CopyToStateData/CopyFromStateData entirely, we can provide
> accessors only for some members of them.

The v26 patch set still exports
CopyToStateData/CopyFromStateData in copyapi.h not
copy{to,from}_internal.h. But I didn't move the following
enums/structs/functions:

* CopyHeaderChoice
* CopyOnErrorChoice
* CopyLogVerbosityChoice
* CopyFormatOptions
* copy_data_dest_cb()

What do you think about this approach?


Thanks,
-- 
kou

  192. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2024-11-26T07:10:50Z

    On Sun, Nov 24, 2024 at 6:06 PM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <CAD21AoBNfKDbJnu-zONNpG820ZXYC0fuTSLrJ-UdRqU4qp2wog@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 22 Nov 2024 13:01:06 -0800,
>   Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> >> @@ -1237,7 +1219,7 @@ CopyReadLine(CopyFromState cstate, bool is_csv)
> >>  /*
> >>   * CopyReadLineText - inner loop of CopyReadLine for text mode
> >>   */
> >> -static pg_attribute_always_inline bool
> >> +static bool
> >>  CopyReadLineText(CopyFromState cstate, bool is_csv)
> >>
> >> Is this an intentional change?
> >> CopyReadLineText() has "bool in_csv".
> >
> > Yes, I'm not sure it's really necessary to make it inline since the
> > benchmark results don't show much difference. Probably this is because
> > the function has 'is_csv' in some 'if' branches but the compiler
> > cannot optimize out the whole 'if' branches as most 'if' branches
> > check 'is_csv' and other variables.
>
> I see. If explicit "inline" isn't related to performance, we
> don't need explicit "inline".
>
> > I've attached the v25 patches that squashed the minor changes I made
> > in v24 and incorporated all comments I got so far. I think these two
> > patches are in good shape. Could you rebase remaining patches on top
> > of them so that we can see the big picture of this feature?
>
> OK. I'll work on it.
>
> > Regarding exposing the structs such as CopyToStateData, v22-0004 patch
> > moves most of all copy-related structs to copyapi.h from copyto.c,
> > copyfrom_internal.h, and copy.h, which seems odd to me. I think we can
> > expose CopyToStateData (and related structs) in a new file
> > copyto_internal.h and keep other structs in the original header files.
>
> Custom COPY format extensions need to use
> CopyToStateData/CopyFromStateData. For example,
> CopyToStateData::rel is used to retrieve table schema. If we
> move CopyToStateData to copyto_internal.h not copyapi.h,
> custom COPY format extensions need to include
> copyto_internal.h. I feel that it's strange that extensions
> need to use internal headers.
>
> What is your real concern? If you don't want to export
> CopyToStateData/CopyFromStateData entirely, we can provide
> accessors only for some members of them.

I'm not against exposing CopyToStateData and CopyFromStateData. My
concern is that if we move all copy-related structs to copyapi.h,
other copy-related files would need to include copyapi.h even if the
file is not related to copy format APIs. IMO copyapi.h should have
only copy-format-API-related variables structs such as CopyFromRoutine
and CopyToRoutine and functions that custom COPY format extension can
utilize to access data source and destination, such as CopyGetData().

When it comes to CopyToStateData and CopyFromStateData, I feel that
they have mixed fields of common fields (e.g., rel, num_errors, and
transition_capture) and format-specific fields (e.g., input_buf,
line_buf, and eol_type). While it makes sense to me that custom copy
format extensions can access the common fields, it seems odd to me
that they can access text-and-csv-format-specific fields such as
input_buf. We might want to sort out these fields but it could be a
huge task.

Also, I realized that CopyFromTextLikeOneRow() does input function
calls and handle soft errors based on ON_ERROR and LOG_VERBOSITY
options. I think these should be done in the core side.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  193. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-11-27T07:53:44Z

    Hi,

In <CAD21AoBW5dEv=Gd2iF_BYNZGEsF=3KTG7fpq=vP5qwpC1CAOeA@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Mon, 25 Nov 2024 23:10:50 -0800,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

>> Custom COPY format extensions need to use
>> CopyToStateData/CopyFromStateData. For example,
>> CopyToStateData::rel is used to retrieve table schema. If we
>> move CopyToStateData to copyto_internal.h not copyapi.h,
>> custom COPY format extensions need to include
>> copyto_internal.h. I feel that it's strange that extensions
>> need to use internal headers.
>>
>> What is your real concern? If you don't want to export
>> CopyToStateData/CopyFromStateData entirely, we can provide
>> accessors only for some members of them.
> 
> I'm not against exposing CopyToStateData and CopyFromStateData. My
> concern is that if we move all copy-related structs to copyapi.h,
> other copy-related files would need to include copyapi.h even if the
> file is not related to copy format APIs. IMO copyapi.h should have
> only copy-format-API-related variables structs such as CopyFromRoutine
> and CopyToRoutine and functions that custom COPY format extension can
> utilize to access data source and destination, such as CopyGetData().
> 
> When it comes to CopyToStateData and CopyFromStateData, I feel that
> they have mixed fields of common fields (e.g., rel, num_errors, and
> transition_capture) and format-specific fields (e.g., input_buf,
> line_buf, and eol_type). While it makes sense to me that custom copy
> format extensions can access the common fields, it seems odd to me
> that they can access text-and-csv-format-specific fields such as
> input_buf. We might want to sort out these fields but it could be a
> huge task.

I understand you concern.

How about using Copy{To,From}StateData::opaque to store
text-and-csv-format-specific data? I feel that this
refactoring doesn't block the 0001/0002 patches. Do you
think that this is a blocker of the 0001/0002 patches?

I think that this may block the 0004/0007 patches that
export Copy{To,From}StateData. But we can work on it after
we merge the 0004/0007 patches. Which is preferred?


> Also, I realized that CopyFromTextLikeOneRow() does input function
> calls and handle soft errors based on ON_ERROR and LOG_VERBOSITY
> options. I think these should be done in the core side.

How about extracting the following part in NextCopyFrom() as
a function and provide it for extensions?

----
				Assert(cstate->opts.on_error != COPY_ON_ERROR_STOP);

				cstate->num_errors++;

				if (cstate->opts.log_verbosity == COPY_LOG_VERBOSITY_VERBOSE)
				{
					/*
					 * Since we emit line number and column info in the below
					 * notice message, we suppress error context information
					 * other than the relation name.
					 */
					Assert(!cstate->relname_only);
					cstate->relname_only = true;

					if (cstate->cur_attval)
					{
						char	   *attval;

						attval = CopyLimitPrintoutLength(cstate->cur_attval);
						ereport(NOTICE,
								errmsg("skipping row due to data type incompatibility at line %llu for column \"%s\": \"%s\"",
									   (unsigned long long) cstate->cur_lineno,
									   cstate->cur_attname,
									   attval));
						pfree(attval);
					}
					else
						ereport(NOTICE,
								errmsg("skipping row due to data type incompatibility at line %llu for column \"%s\": null input",
									   (unsigned long long) cstate->cur_lineno,
									   cstate->cur_attname));

					/* reset relname_only */
					cstate->relname_only = false;
				}
----

See the attached v27 patch set for this idea.

0001-0008 are almost same as the v26 patch set.
("format" -> "FORMAT" in COPY test changes are included.)

0009 exports the above code as
CopyFromSkipErrorRow(). Extensions should call it when they
use errsave() for a soft error in CopyFromOneRow callback.


Thanks,
-- 
kou

  194. Re: Make COPY format extendable: Extract COPY TO format implementations

    Junwang Zhao <zhjwpku@gmail.com> — 2024-11-27T11:49:17Z

    Hi,

On Mon, Nov 25, 2024 at 2:02 PM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <20241125.110620.313152541320718947.kou@clear-code.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Mon, 25 Nov 2024 11:06:20 +0900 (JST),
>   Sutou Kouhei <kou@clear-code.com> wrote:
>
> >> I've attached the v25 patches that squashed the minor changes I made
> >> in v24 and incorporated all comments I got so far. I think these two
> >> patches are in good shape. Could you rebase remaining patches on top
> >> of them so that we can see the big picture of this feature?
> >
> > OK. I'll work on it.
>
> I've attached the v26 patch set:
>
> 0001: It's same as 0001 in the v25 patch set.
> 0002: It's same as 0002 in the v25 patch set.
> 0003: It's same as 0003 in the v23 patch set.
>       This parses the "format" option and adds support for
>       custom TO handler.
> 0004: It's same as 0004 in the v23 patch set.
>       This exports CopyToStateData. But the following
>       enums/structs/functions aren't moved to copyapi.h from
>       copy.h:
>       * CopyHeaderChoice
>       * CopyOnErrorChoice
>       * CopyLogVerbosityChoice
>       * CopyFormatOptions
>       * copy_data_dest_cb()
> 0005: It's same as 0005 in the v23 patch set.
>       This adds missing APIs to implement custom TO handler
>       as an extension.
> 0006: It's same as 0008 in the v23 patch set.
>       This adds support for custom FROM handler.
> 0007: It's same as 0009 in the v23 patch set.
>       This exports CopyFromStateData.
> 0008: It's same as 0010 in the v23 patch set.
>       This adds missing APIs to implement custom FROM handler
>       as an extension.
>
> I've also updated https://github.com/kou/pg-copy-arrow for
> the current API.
>
> I think that we can merge only 0001/0002 as the first
> step. Because they don't change the current behavior and
> they improve performance. We can merge other patches after
> that.
>
> >> Regarding exposing the structs such as CopyToStateData, v22-0004 patch
> >> moves most of all copy-related structs to copyapi.h from copyto.c,
> >> copyfrom_internal.h, and copy.h, which seems odd to me. I think we can
> >> expose CopyToStateData (and related structs) in a new file
> >> copyto_internal.h and keep other structs in the original header files.
> >
> > Custom COPY format extensions need to use
> > CopyToStateData/CopyFromStateData. For example,
> > CopyToStateData::rel is used to retrieve table schema. If we
> > move CopyToStateData to copyto_internal.h not copyapi.h,
> > custom COPY format extensions need to include
> > copyto_internal.h. I feel that it's strange that extensions
> > need to use internal headers.
> >
> > What is your real concern? If you don't want to export
> > CopyToStateData/CopyFromStateData entirely, we can provide
> > accessors only for some members of them.
>
> The v26 patch set still exports
> CopyToStateData/CopyFromStateData in copyapi.h not
> copy{to,from}_internal.h. But I didn't move the following
> enums/structs/functions:
>
> * CopyHeaderChoice
> * CopyOnErrorChoice
> * CopyLogVerbosityChoice
> * CopyFormatOptions
> * copy_data_dest_cb()
>
> What do you think about this approach?
>
>
> Thanks,
> --
> kou

I just gave this another round of benchmarking tests. I'd like to
share the number,
since COPY TO has some performance drawbacks, I test only COPY TO. I
use the run.sh Tomas provided earlier but use pgbench with a custom script, you
can find it here[0].

I tested 3 branches:

1. the master branch
2. all v26 patch sets applied
3. Emitting JSON to file using COPY TO v13 patch set[1], this add some
if branch in CopyOneRowTo, so I was expecting this slower than master

2 can be about -3%~+3% compared to 1, but what surprised me is that 3
is always better than 1 & 2.

I reviewed the patch set of 3 and I don't see any magic.

You can see the detailed results here[2], I can not upload files so I
just shared the google doc link, ping me if you can not open the link.

[0]: https://github.com/pghacking/scripts/tree/main/extensible_copy
[1]: https://www.postgresql.org/message-id/CACJufxH8J0uD-inukxAmd3TVwt-b-y7d7hLGSBdEdLXFGJLyDA%40mail.gmail.com
[2]: https://docs.google.com/spreadsheets/d/1wJPXZF4LHe34X9IU1pLG7rI9sCkSy2dEkdj7w7avTqM/edit?usp=sharing


-- 
Regards
Junwang Zhao

  195. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-11-28T06:16:17Z

    Hi,

In <CAEG8a3LUBcvjwqgt6AijJmg67YN_b_NZ4Kzoxc_dH4rpAq0pKg@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Wed, 27 Nov 2024 19:49:17 +0800,
  Junwang Zhao <zhjwpku@gmail.com> wrote:

> I just gave this another round of benchmarking tests. I'd like to
> share the number,
> since COPY TO has some performance drawbacks, I test only COPY TO. I
> use the run.sh Tomas provided earlier but use pgbench with a custom script, you
> can find it here[0].
> 
> I tested 3 branches:
> 
> 1. the master branch
> 2. all v26 patch sets applied
> 3. Emitting JSON to file using COPY TO v13 patch set[1], this add some
> if branch in CopyOneRowTo, so I was expecting this slower than master
> 
> 2 can be about -3%~+3% compared to 1, but what surprised me is that 3
> is always better than 1 & 2.
> 
> I reviewed the patch set of 3 and I don't see any magic.
> 
> You can see the detailed results here[2], I can not upload files so I
> just shared the google doc link, ping me if you can not open the link.
> 
> [0]: https://github.com/pghacking/scripts/tree/main/extensible_copy
> [1]: https://www.postgresql.org/message-id/CACJufxH8J0uD-inukxAmd3TVwt-b-y7d7hLGSBdEdLXFGJLyDA%40mail.gmail.com
> [2]: https://docs.google.com/spreadsheets/d/1wJPXZF4LHe34X9IU1pLG7rI9sCkSy2dEkdj7w7avTqM/edit?usp=sharing

Thanks for sharing your numbers.

1. and 2. shows that there is at least no significant
performance regression.

I see the patch set of 3. and I think that the result
(there is no performance difference between 1. and 3.) isn't
strange. The patch set adds some if branches but they aren't
used with "text" format at least in per row process.

Thanks,
-- 
kou

  196. Re: Make COPY format extendable: Extract COPY TO format implementations

    Junwang Zhao <zhjwpku@gmail.com> — 2024-11-28T11:02:57Z

    On Thu, Nov 28, 2024 at 2:16 PM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <CAEG8a3LUBcvjwqgt6AijJmg67YN_b_NZ4Kzoxc_dH4rpAq0pKg@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Wed, 27 Nov 2024 19:49:17 +0800,
>   Junwang Zhao <zhjwpku@gmail.com> wrote:
>
> > I just gave this another round of benchmarking tests. I'd like to
> > share the number,
> > since COPY TO has some performance drawbacks, I test only COPY TO. I
> > use the run.sh Tomas provided earlier but use pgbench with a custom script, you
> > can find it here[0].
> >
> > I tested 3 branches:
> >
> > 1. the master branch
> > 2. all v26 patch sets applied
> > 3. Emitting JSON to file using COPY TO v13 patch set[1], this add some
> > if branch in CopyOneRowTo, so I was expecting this slower than master
> >
> > 2 can be about -3%~+3% compared to 1, but what surprised me is that 3
> > is always better than 1 & 2.
> >
> > I reviewed the patch set of 3 and I don't see any magic.
> >
> > You can see the detailed results here[2], I can not upload files so I
> > just shared the google doc link, ping me if you can not open the link.
> >
> > [0]: https://github.com/pghacking/scripts/tree/main/extensible_copy
> > [1]: https://www.postgresql.org/message-id/CACJufxH8J0uD-inukxAmd3TVwt-b-y7d7hLGSBdEdLXFGJLyDA%40mail.gmail.com
> > [2]: https://docs.google.com/spreadsheets/d/1wJPXZF4LHe34X9IU1pLG7rI9sCkSy2dEkdj7w7avTqM/edit?usp=sharing
>
> Thanks for sharing your numbers.
>
> 1. and 2. shows that there is at least no significant
> performance regression.

Agreed.

>
> I see the patch set of 3. and I think that the result
> (there is no performance difference between 1. and 3.) isn't
> strange. The patch set adds some if branches but they aren't
> used with "text" format at least in per row process.

It is not used in "text" format, but it adds some assembly code
to the CopyOneRowTo function, so this will have some impact
on the cpu i cache I guess.

There is difference between 1 and 3, 3 is always better than 1
upto 4% improvement, I forgot to mention that the comparisons
are in *sheet2*.

>
> Thanks,
> --
> kou



-- 
Regards
Junwang Zhao

  197. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2024-11-29T01:07:13Z

    Hi,

In <CAEG8a3+BmNeEOLmApOCyktYbiZW=s95dvpod_FxJS+3ieVZQ7w@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Thu, 28 Nov 2024 19:02:57 +0800,
  Junwang Zhao <zhjwpku@gmail.com> wrote:

>> > I tested 3 branches:
>> >
>> > 1. the master branch
>> > 2. all v26 patch sets applied
>> > 3. Emitting JSON to file using COPY TO v13 patch set[1], this add some
>> > if branch in CopyOneRowTo, so I was expecting this slower than master
>> >
>> > You can see the detailed results here[2], I can not upload files so I
>> > just shared the google doc link, ping me if you can not open the link.
>> >
>> > [1]: https://www.postgresql.org/message-id/CACJufxH8J0uD-inukxAmd3TVwt-b-y7d7hLGSBdEdLXFGJLyDA%40mail.gmail.com
>> > [2]: https://docs.google.com/spreadsheets/d/1wJPXZF4LHe34X9IU1pLG7rI9sCkSy2dEkdj7w7avTqM/edit?usp=sharing
>>
>> Thanks for sharing your numbers.
>>
>> 1. and 2. shows that there is at least no significant
>> performance regression.
> 
> Agreed.

Can we focus on only 1. and 2. in this thread?

>> I see the patch set of 3. and I think that the result
>> (there is no performance difference between 1. and 3.) isn't
>> strange. The patch set adds some if branches but they aren't
>> used with "text" format at least in per row process.
> 
> It is not used in "text" format, but it adds some assembly code
> to the CopyOneRowTo function, so this will have some impact
> on the cpu i cache I guess.
> 
> There is difference between 1 and 3, 3 is always better than 1
> upto 4% improvement

Can we discuss 1. and 3. in the [1] thread?

(Anyway, we may want to confirm whether these numbers are
reproducible or not as the first step.)

>                      I forgot to mention that the comparisons
> are in *sheet2*.

Thanks. I missed it.


Thanks,
-- 
kou

  198. Re: Make COPY format extendable: Extract COPY TO format implementations

    Junwang Zhao <zhjwpku@gmail.com> — 2024-11-29T02:15:04Z

    On Fri, Nov 29, 2024 at 9:07 AM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <CAEG8a3+BmNeEOLmApOCyktYbiZW=s95dvpod_FxJS+3ieVZQ7w@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Thu, 28 Nov 2024 19:02:57 +0800,
>   Junwang Zhao <zhjwpku@gmail.com> wrote:
>
> >> > I tested 3 branches:
> >> >
> >> > 1. the master branch
> >> > 2. all v26 patch sets applied
> >> > 3. Emitting JSON to file using COPY TO v13 patch set[1], this add some
> >> > if branch in CopyOneRowTo, so I was expecting this slower than master
> >> >
> >> > You can see the detailed results here[2], I can not upload files so I
> >> > just shared the google doc link, ping me if you can not open the link.
> >> >
> >> > [1]: https://www.postgresql.org/message-id/CACJufxH8J0uD-inukxAmd3TVwt-b-y7d7hLGSBdEdLXFGJLyDA%40mail.gmail.com
> >> > [2]: https://docs.google.com/spreadsheets/d/1wJPXZF4LHe34X9IU1pLG7rI9sCkSy2dEkdj7w7avTqM/edit?usp=sharing
> >>
> >> Thanks for sharing your numbers.
> >>
> >> 1. and 2. shows that there is at least no significant
> >> performance regression.
> >
> > Agreed.
>
> Can we focus on only 1. and 2. in this thread?
>
> >> I see the patch set of 3. and I think that the result
> >> (there is no performance difference between 1. and 3.) isn't
> >> strange. The patch set adds some if branches but they aren't
> >> used with "text" format at least in per row process.
> >
> > It is not used in "text" format, but it adds some assembly code
> > to the CopyOneRowTo function, so this will have some impact
> > on the cpu i cache I guess.
> >
> > There is difference between 1 and 3, 3 is always better than 1
> > upto 4% improvement
>
> Can we discuss 1. and 3. in the [1] thread?

This thread and [1] thread are kind of interleaved, I chose this thread
to share the numbers because I think this feature should be committed
first and then adapt the *copy to json* as a contrib module.

Committers on this thread seem worried about the performance
drawback, so what I tried to do is that *if 2 is slightly worse than 1,
but better than 3*, then we can commit 2 first, but I did not get
the expected number.

>
> (Anyway, we may want to confirm whether these numbers are
> reproducible or not as the first step.)
>
> >                      I forgot to mention that the comparisons
> > are in *sheet2*.
>
> Thanks. I missed it.
>
>
> Thanks,
> --
> kou



-- 
Regards
Junwang Zhao

  199. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2025-01-23T09:09:29Z

    Hi,

In <CAEG8a3+-3fAmiwD5NmE7W4j5-=HLs2OEexQNW9-fB=j=mdxgDQ@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 29 Nov 2024 10:15:04 +0800,
  Junwang Zhao <zhjwpku@gmail.com> wrote:

> This thread and [1] thread are kind of interleaved, I chose this thread
> to share the numbers because I think this feature should be committed
> first and then adapt the *copy to json* as a contrib module.

I agree with you.

> Committers on this thread seem worried about the performance
> drawback, so what I tried to do is that *if 2 is slightly worse than 1,
> but better than 3*, then we can commit 2 first, but I did not get
> the expected number.

Could you break down which patch in the v13 patch set[1]
affected? If we can find which change improves performance,
we can use the approach in this patch set too.

[1]: https://www.postgresql.org/message-id/CACJufxH8J0uD-inukxAmd3TVwt-b-y7d7hLGSBdEdLXFGJLyDA%40mail.gmail.com


Thanks,
-- 
kou

  200. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2025-01-23T09:12:10Z

    Hi,

I noticed that the last patch set (v27) can't be applied to
the current master. I've rebased on the current master and
created v28 patch set. No code change.


Thanks,
-- 
kou

  201. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2025-01-28T23:00:03Z

    On Thu, Jan 23, 2025 at 1:12 AM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> I noticed that the last patch set (v27) can't be applied to
> the current master. I've rebased on the current master and
> created v28 patch set. No code change.

Thank you for updating the patch!

While 0001 and 0002 look good to me overall, we still need to polish
subsequent patches. Here are review comments:

---
I still find that it would not be a good idea to move all copy-related
struct definitions to copyapi.h because we need to include copyapi.h
file into a .c file even if the file is not related to the custom copy
format routines. I think that copyapi.h should have only the
definitions of CopyToRoutine and CopyFromRoutine as well as some
functions related to the custom copy format. Here is an idea:

- CopyToState and CopyFromState are defined in copyto_internal.h (new
file) and copyfrom_internal.h, respectively.
- These two files #include's copy.h and other necessary header files.
- copyapi.h has only CopyToRoutine and CopyFromRoutine and #include's
both copyfrom_internal.h and copyto_internal.h.
- copyto.c, copyfrom.c and copyfromparse.c #include copyapi.h

Some advantages of this idea:

- we can keep both CopyToState and CopyFromState private in _internal.h files.
- custom format extension can include copyapi.h to provide a custom
copy format routine and to access the copy state data.
- copy-related .c files won't need to include copyapi.h if they don't
use custom copy format routines.

---
The 0008 patch introduces CopyFromStateRead(). While it would be a
good start, I think we can consider sorting out low-level
communication functions more. For example, CopyReadBinaryData() uses
the internal 64kB buffer but some custom copy format extensions might
want to use a larger buffer in its own implementation, which would
require exposing CopyGetData() etc. Given that we might expose more
functions to provide more ways for extensions, we might want to rename
CopyFromStateRead().

---
While we get the format routines for custom formats in
ProcessCopyOptionFormat(), we do that for built-in formats in
BeginCopyTo(), which seems odd to me. I think we can have
CopyToGetRoutine() responsible for getting CopyToRoutine for built-in
formats as well as custom format. The same is true for
CopyFromRoutine.

---
Copy[To|From]Routine for built-in formats are missing to set the node type.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  202. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2025-01-30T15:42:13Z

    Hi,

In <CAD21AoDyBJrCsh5vNFWcRmS0_XKCCCP4gLzZnLCayYccLpaBfw@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Tue, 28 Jan 2025 15:00:03 -0800,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

> While 0001 and 0002 look good to me overall, we still need to polish
> subsequent patches. Here are review comments:

I attached the v29 patch set that applied your suggestions:

Refactoring:
0001-0002: There are some trivial changes (copyright year
           change and some comment fixes)

COPY TO related:
0003: Applied your copyto_internal.h related,
      CopyToGetRoutine() related and built-in CopyToRoutine
      suggestions
0004: Applied your copyto_internal.h related suggestion
0005: No change

COPY FROM related:
0006: Applied your copyfrom_internal.h related,
      CopyFromGetRoutine() related and built-in CopyFromRoutine
      suggestions
0007: Applied your copyfrom_internal.h related suggestion
0008: Applied your CopyFromStateRead() related suggestion
0009: No change


> I still find that it would not be a good idea to move all copy-related
> struct definitions to copyapi.h because we need to include copyapi.h
> file into a .c file even if the file is not related to the custom copy
> format routines. I think that copyapi.h should have only the
> definitions of CopyToRoutine and CopyFromRoutine as well as some
> functions related to the custom copy format. Here is an idea:
> 
> - CopyToState and CopyFromState are defined in copyto_internal.h (new
> file) and copyfrom_internal.h, respectively.
> - These two files #include's copy.h and other necessary header files.
> - copyapi.h has only CopyToRoutine and CopyFromRoutine and #include's
> both copyfrom_internal.h and copyto_internal.h.
> - copyto.c, copyfrom.c and copyfromparse.c #include copyapi.h
> 
> Some advantages of this idea:
> 
> - we can keep both CopyToState and CopyFromState private in _internal.h files.
> - custom format extension can include copyapi.h to provide a custom
> copy format routine and to access the copy state data.
> - copy-related .c files won't need to include copyapi.h if they don't
> use custom copy format routines.

Hmm. I thought Copy{To,From}State are "public" API not
"private" API for extensions. Because extensions need to use
at least Copy{To,From}State::opaque directly. If we want to
make Copy{To,From}State private, I think that we should
provide getter/setter for needed members of
Copy{To,From}State such as
Copy{To,From}State{Get,Set}Opaque().

It's a design in the v2 patch set:
https://www.postgresql.org/message-id/20231221.183504.1240642084042888377.kou%40clear-code.com

We discussed that we can make CopyToState public:
https://www.postgresql.org/message-id/CAD21AoD%3DUapH4Wh06G6H5XAzPJ0iJg9YcW8r7E2UEJkZ8QsosA%40mail.gmail.com

What does "private" mean here? I thought that it means that
"PostgreSQL itself can use it". But it seems that you mean
that "PostgreSQL itself and custom format extensions can use
it but other extensions can't use it".

I'm not familiar with "_internal.h" in PostgreSQL but is
"_internal.h" for the latter "private" mean?


> The 0008 patch introduces CopyFromStateRead(). While it would be a
> good start, I think we can consider sorting out low-level
> communication functions more. For example, CopyReadBinaryData() uses
> the internal 64kB buffer but some custom copy format extensions might
> want to use a larger buffer in its own implementation, which would
> require exposing CopyGetData() etc. Given that we might expose more
> functions to provide more ways for extensions, we might want to rename
> CopyFromStateRead().

This suggests that we just need a low-level CopyGetData()
not a high-level CopyReadBinaryData() as the first step,
right?

I agree that we should start from a minimal API set.

I've renamed CopyFromStateRead() to CopyFromStateGetData()
because it wraps CopyGetData() now.


> While we get the format routines for custom formats in
> ProcessCopyOptionFormat(), we do that for built-in formats in
> BeginCopyTo(), which seems odd to me. I think we can have
> CopyToGetRoutine() responsible for getting CopyToRoutine for built-in
> formats as well as custom format. The same is true for
> CopyFromRoutine.

I like the current design because we don't need to export
CopyToGetBuiltinRoutine() (we can use static for
CopyToGetBuiltinRoutine()) but I applied your
suggestion. Because it's not a strong opinion.


> Copy[To|From]Routine for built-in formats are missing to set the node type.

Oh, sorry. I missed this.


Thanks,
-- 
kou

  203. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2025-01-31T22:25:34Z

    On Thu, Jan 30, 2025 at 7:42 AM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <CAD21AoDyBJrCsh5vNFWcRmS0_XKCCCP4gLzZnLCayYccLpaBfw@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Tue, 28 Jan 2025 15:00:03 -0800,
>   Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> > While 0001 and 0002 look good to me overall, we still need to polish
> > subsequent patches. Here are review comments:
>
> I attached the v29 patch set that applied your suggestions:
>
> Refactoring:
> 0001-0002: There are some trivial changes (copyright year
>            change and some comment fixes)
>
> COPY TO related:
> 0003: Applied your copyto_internal.h related,
>       CopyToGetRoutine() related and built-in CopyToRoutine
>       suggestions
> 0004: Applied your copyto_internal.h related suggestion
> 0005: No change
>
> COPY FROM related:
> 0006: Applied your copyfrom_internal.h related,
>       CopyFromGetRoutine() related and built-in CopyFromRoutine
>       suggestions
> 0007: Applied your copyfrom_internal.h related suggestion
> 0008: Applied your CopyFromStateRead() related suggestion
> 0009: No change
>
>
> > I still find that it would not be a good idea to move all copy-related
> > struct definitions to copyapi.h because we need to include copyapi.h
> > file into a .c file even if the file is not related to the custom copy
> > format routines. I think that copyapi.h should have only the
> > definitions of CopyToRoutine and CopyFromRoutine as well as some
> > functions related to the custom copy format. Here is an idea:
> >
> > - CopyToState and CopyFromState are defined in copyto_internal.h (new
> > file) and copyfrom_internal.h, respectively.
> > - These two files #include's copy.h and other necessary header files.
> > - copyapi.h has only CopyToRoutine and CopyFromRoutine and #include's
> > both copyfrom_internal.h and copyto_internal.h.
> > - copyto.c, copyfrom.c and copyfromparse.c #include copyapi.h
> >
> > Some advantages of this idea:
> >
> > - we can keep both CopyToState and CopyFromState private in _internal.h files.
> > - custom format extension can include copyapi.h to provide a custom
> > copy format routine and to access the copy state data.
> > - copy-related .c files won't need to include copyapi.h if they don't
> > use custom copy format routines.
>
> Hmm. I thought Copy{To,From}State are "public" API not
> "private" API for extensions. Because extensions need to use
> at least Copy{To,From}State::opaque directly. If we want to
> make Copy{To,From}State private, I think that we should
> provide getter/setter for needed members of
> Copy{To,From}State such as
> Copy{To,From}State{Get,Set}Opaque().
>
> It's a design in the v2 patch set:
> https://www.postgresql.org/message-id/20231221.183504.1240642084042888377.kou%40clear-code.com
>
> We discussed that we can make CopyToState public:
> https://www.postgresql.org/message-id/CAD21AoD%3DUapH4Wh06G6H5XAzPJ0iJg9YcW8r7E2UEJkZ8QsosA%40mail.gmail.com

I think that CopyToState and CopyFromState are not APIs but the
execution states. I'm not against exposing CopyToState and
CopyFromState. What I'd like to avoid is that we end up adding
everything (including new fields we add in the future) related to copy
operation to copyapi.h, leading to include copyapi.h into files that
are not related to custom format api. fdwapi.h and tsmapi.h as
examples have only a struct having a bunch of callbacks but not the
execution state data such as SampScanState are not defined there.

>
> What does "private" mean here? I thought that it means that
> "PostgreSQL itself can use it". But it seems that you mean
> that "PostgreSQL itself and custom format extensions can use
> it but other extensions can't use it".
>
> I'm not familiar with "_internal.h" in PostgreSQL but is
> "_internal.h" for the latter "private" mean?

My understanding is that we don't strictly prohibit _internal.h from
being included in out of core files. For example, file_fdw.c includes
copyfrom_internal.h in order to access some fields of CopyFromState.

If the name with _internal.h is the problem, we can rename them to
copyfrom.h and copyto.h. It makes sense to me that the code that needs
to access the internal of the copy execution state include _internal.h
header, though.

> > While we get the format routines for custom formats in
> > ProcessCopyOptionFormat(), we do that for built-in formats in
> > BeginCopyTo(), which seems odd to me. I think we can have
> > CopyToGetRoutine() responsible for getting CopyToRoutine for built-in
> > formats as well as custom format. The same is true for
> > CopyFromRoutine.
>
> I like the current design because we don't need to export
> CopyToGetBuiltinRoutine() (we can use static for
> CopyToGetBuiltinRoutine()) but I applied your
> suggestion. Because it's not a strong opinion.

I meant that ProcessCopyOptionFormat() doesn't not necessarily get the
routine. An idea is that in ProcessCopyOptionFormat() we just get the
OID of the handler function, and then set up the format routine in
BeginCopyTo(). I've attached a patch for this idea (applied on top of
0009).

Also, please check some regression test failures on cfbot.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  204. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2025-01-31T23:10:23Z

    Hi,

In <CAD21AoBpWFU4k-_bwrTq0AkFSAdwQqhAsSW188STmu9HxLJ0nQ@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 31 Jan 2025 14:25:34 -0800,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

> I think that CopyToState and CopyFromState are not APIs but the
> execution states. I'm not against exposing CopyToState and
> CopyFromState. What I'd like to avoid is that we end up adding
> everything (including new fields we add in the future) related to copy
> operation to copyapi.h, leading to include copyapi.h into files that
> are not related to custom format api. fdwapi.h and tsmapi.h as
> examples have only a struct having a bunch of callbacks but not the
> execution state data such as SampScanState are not defined there.

Thanks for sharing examples. But it seems that
fdwapi.h/tsmapi.h (ForeignScanState/SampleScanSate) are not
good examples. It seems that PostgreSQL uses
nodes/execnodes.h for all *ScanState. It seems that the
sparation is not related to *api.h usage.

> My understanding is that we don't strictly prohibit _internal.h from
> being included in out of core files. For example, file_fdw.c includes
> copyfrom_internal.h in order to access some fields of CopyFromState.
> 
> If the name with _internal.h is the problem, we can rename them to
> copyfrom.h and copyto.h. It makes sense to me that the code that needs
> to access the internal of the copy execution state include _internal.h
> header, though.

Thanks for sharing the file_fdw.c example. I'm OK with
_internal.h suffix because PostgreSQL doesn't prohibit
_internal.h usage by extensions as you mentioned.

>> > While we get the format routines for custom formats in
>> > ProcessCopyOptionFormat(), we do that for built-in formats in
>> > BeginCopyTo(), which seems odd to me. I think we can have
>> > CopyToGetRoutine() responsible for getting CopyToRoutine for built-in
>> > formats as well as custom format. The same is true for
>> > CopyFromRoutine.
>>
>> I like the current design because we don't need to export
>> CopyToGetBuiltinRoutine() (we can use static for
>> CopyToGetBuiltinRoutine()) but I applied your
>> suggestion. Because it's not a strong opinion.
> 
> I meant that ProcessCopyOptionFormat() doesn't not necessarily get the
> routine. An idea is that in ProcessCopyOptionFormat() we just get the
> OID of the handler function, and then set up the format routine in
> BeginCopyTo(). I've attached a patch for this idea (applied on top of
> 0009).

Oh, sorry. I misunderstood your suggestion. I understand
what you suggested by the patch. Thanks.

If we use the approach, we can't show error position when a
custom COPY format handler function returns invalid routine
because DefElem for the "format" option isn't available in
BeginCopyTo(). Is it acceptable? If it's acceptable, let's
use the approach.

> Also, please check some regression test failures on cfbot.

Oh, sorry. I forgot to follow function name change in
0009. I attach the v30 patch set that fixes it in 0009.


Thanks,
-- 
kou

  205. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2025-02-01T00:34:52Z

    On Fri, Jan 31, 2025 at 3:10 PM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <CAD21AoBpWFU4k-_bwrTq0AkFSAdwQqhAsSW188STmu9HxLJ0nQ@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 31 Jan 2025 14:25:34 -0800,
>   Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> > I think that CopyToState and CopyFromState are not APIs but the
> > execution states. I'm not against exposing CopyToState and
> > CopyFromState. What I'd like to avoid is that we end up adding
> > everything (including new fields we add in the future) related to copy
> > operation to copyapi.h, leading to include copyapi.h into files that
> > are not related to custom format api. fdwapi.h and tsmapi.h as
> > examples have only a struct having a bunch of callbacks but not the
> > execution state data such as SampScanState are not defined there.
>
> Thanks for sharing examples. But it seems that
> fdwapi.h/tsmapi.h (ForeignScanState/SampleScanSate) are not
> good examples. It seems that PostgreSQL uses
> nodes/execnodes.h for all *ScanState. It seems that the
> sparation is not related to *api.h usage.

I didn't mean these examples perfectly apply the copyapi.h case.
Again, what I'd like to avoid is that we end up adding everything
(including new fields we add in the future) related to copy operation
to copyapi.h. For example, with v28 that moves both CopyFromState and
CopyToState to copyapi.h, file_fdw.c includes unrelated CopyToState
struct via copyfrom_internal.h -> copyapi.h. In addition to that, both
copyfrom.c and copyfrom_internal.h did the same, which made me think
copyfrom_internal.h mostly no longer plays its role. I'm very welcome
to other ideas too if they could achieve the same goal.

>
> > My understanding is that we don't strictly prohibit _internal.h from
> > being included in out of core files. For example, file_fdw.c includes
> > copyfrom_internal.h in order to access some fields of CopyFromState.
> >
> > If the name with _internal.h is the problem, we can rename them to
> > copyfrom.h and copyto.h. It makes sense to me that the code that needs
> > to access the internal of the copy execution state include _internal.h
> > header, though.
>
> Thanks for sharing the file_fdw.c example. I'm OK with
> _internal.h suffix because PostgreSQL doesn't prohibit
> _internal.h usage by extensions as you mentioned.
>
> >> > While we get the format routines for custom formats in
> >> > ProcessCopyOptionFormat(), we do that for built-in formats in
> >> > BeginCopyTo(), which seems odd to me. I think we can have
> >> > CopyToGetRoutine() responsible for getting CopyToRoutine for built-in
> >> > formats as well as custom format. The same is true for
> >> > CopyFromRoutine.
> >>
> >> I like the current design because we don't need to export
> >> CopyToGetBuiltinRoutine() (we can use static for
> >> CopyToGetBuiltinRoutine()) but I applied your
> >> suggestion. Because it's not a strong opinion.
> >
> > I meant that ProcessCopyOptionFormat() doesn't not necessarily get the
> > routine. An idea is that in ProcessCopyOptionFormat() we just get the
> > OID of the handler function, and then set up the format routine in
> > BeginCopyTo(). I've attached a patch for this idea (applied on top of
> > 0009).
>
> Oh, sorry. I misunderstood your suggestion. I understand
> what you suggested by the patch. Thanks.
>
> If we use the approach, we can't show error position when a
> custom COPY format handler function returns invalid routine
> because DefElem for the "format" option isn't available in
> BeginCopyTo(). Is it acceptable? If it's acceptable, let's
> use the approach.

I think we can live with it. All errors happening while processing the
copy options don't necessarily show the error position.

> Oh, sorry. I forgot to follow function name change in
> 0009. I attach the v30 patch set that fixes it in 0009.

Thank you for updating the patch! I'll review these patches.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  206. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2025-02-01T10:12:01Z

    Hi,

In <CAD21AoA3KMddnjxY1hxth3f4f1wo8a8i2icgK6GEKqXNR_e6jA@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 31 Jan 2025 16:34:52 -0800,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

> Again, what I'd like to avoid is that we end up adding everything
> (including new fields we add in the future) related to copy operation
> to copyapi.h. For example, with v28 that moves both CopyFromState and
> CopyToState to copyapi.h, file_fdw.c includes unrelated CopyToState
> struct via copyfrom_internal.h -> copyapi.h. In addition to that, both
> copyfrom.c and copyfrom_internal.h did the same, which made me think
> copyfrom_internal.h mostly no longer plays its role. I'm very welcome
> to other ideas too if they could achieve the same goal.

For the propose, copyapi.h should not include
copy{to,from}_internal.h. If we do it, copyto.c includes
CopyFromState and copyfrom*.c include CopyToState.

What do you think about the following change? Note that
extensions must include copy{to,from}_internal.h explicitly
in addition to copyapi.h.

-----
diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c
index 10f80ef3654..a2dc2d04407 100644
--- a/src/backend/commands/copy.c
+++ b/src/backend/commands/copy.c
@@ -23,6 +23,8 @@
 #include "access/xact.h"
 #include "catalog/pg_authid.h"
 #include "commands/copyapi.h"
+#include "commands/copyto_internal.h"
+#include "commands/copyfrom_internal.h"
 #include "commands/defrem.h"
 #include "executor/executor.h"
 #include "mb/pg_wchar.h"
diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c
index 3f6b0031d94..7bcf1c6544b 100644
--- a/src/backend/commands/copyfrom.c
+++ b/src/backend/commands/copyfrom.c
@@ -29,6 +29,7 @@
 #include "access/xact.h"
 #include "catalog/namespace.h"
 #include "commands/copyapi.h"
+#include "commands/copyfrom_internal.h"
 #include "commands/progress.h"
 #include "commands/trigger.h"
 #include "executor/execPartition.h"
diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c
index b016f43a711..7296745d6d2 100644
--- a/src/backend/commands/copyfromparse.c
+++ b/src/backend/commands/copyfromparse.c
@@ -63,6 +63,7 @@
 #include <sys/stat.h>
 
 #include "commands/copyapi.h"
+#include "commands/copyfrom_internal.h"
 #include "commands/progress.h"
 #include "executor/executor.h"
 #include "libpq/libpq.h"
diff --git a/src/backend/commands/copyto.c b/src/backend/commands/copyto.c
index da281f32950..a69771ea6da 100644
--- a/src/backend/commands/copyto.c
+++ b/src/backend/commands/copyto.c
@@ -20,6 +20,7 @@
 
 #include "access/tableam.h"
 #include "commands/copyapi.h"
+#include "commands/copyto_internal.h"
 #include "commands/progress.h"
 #include "executor/execdesc.h"
 #include "executor/executor.h"
diff --git a/src/include/commands/copyapi.h b/src/include/commands/copyapi.h
index 389f887b2c1..dfab62372a7 100644
--- a/src/include/commands/copyapi.h
+++ b/src/include/commands/copyapi.h
@@ -14,8 +14,7 @@
 #ifndef COPYAPI_H
 #define COPYAPI_H
 
-#include "commands/copyto_internal.h"
-#include "commands/copyfrom_internal.h"
+#include "commands/copy.h"
 
 /*
  * API structure for a COPY TO format implementation. Note this must be
diff --git a/src/test/modules/test_copy_format/test_copy_format.c b/src/test/modules/test_copy_format/test_copy_format.c
index d72d5c33c1b..c05d65557a9 100644
--- a/src/test/modules/test_copy_format/test_copy_format.c
+++ b/src/test/modules/test_copy_format/test_copy_format.c
@@ -14,6 +14,8 @@
 #include "postgres.h"
 
 #include "commands/copyapi.h"
+#include "commands/copyfrom_internal.h"
+#include "commands/copyto_internal.h"
 #include "commands/defrem.h"
 
 PG_MODULE_MAGIC;
-----

>> If we use the approach, we can't show error position when a
>> custom COPY format handler function returns invalid routine
>> because DefElem for the "format" option isn't available in
>> BeginCopyTo(). Is it acceptable? If it's acceptable, let's
>> use the approach.
> 
> I think we can live with it. All errors happening while processing the
> copy options don't necessarily show the error position.

OK. I attach the v31 patch set that uses this
approach. Mainly, 0003 and 0006 were changed. The v31 patch
set also includes the above
copyapi.h/copy{to,from}_internal.h related changes.

If we have a feature that returns a function name from Oid,
we can improve error messages by including function name
(format name) when a custom format handler function returns
not Copy{To,From}Routine...


Thanks,
-- 
kou

  207. Re: Make COPY format extendable: Extract COPY TO format implementations

    Vladlen Popolitov <v.popolitov@postgrespro.ru> — 2025-02-03T06:38:04Z

    Sutou Kouhei писал(а) 2025-02-01 17:12:
> Hi,
> 

Hi
I would like to inform about the security breach in your design of COPY 
TO/FROM.

You use FORMAT option to add new formats, filling it with routine name
in shared library. As result any caller can call any routine in 
PostgreSQL kernel.
I think, it will start competition, who can find most dangerous routine
to call just from COPY FROM command.

  Standard PostgreSQL realisation for new methods to use USING keyword. 
Every
new method could have own options (FORMAT is option of internal 'copy 
from/to'
methods), it assumes some SetOptions interface, that defines
an options structure according to the new method requirements.

  I agree with the general direction of the extensibility, but it should 
be secure
and consistent.

-- 
Best regards,

Vladlen Popolitov.

  208. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2025-02-04T06:29:27Z

    Hi,

In <d838025aceeb19c9ff1db702fa55cabf@postgrespro.ru>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Mon, 03 Feb 2025 13:38:04 +0700,
  Vladlen Popolitov <v.popolitov@postgrespro.ru> wrote:

> I would like to inform about the security breach in your design of
> COPY TO/FROM.

Thanks! I didn't notice it.

> You use FORMAT option to add new formats, filling it with routine name
> in shared library. As result any caller can call any routine in
> PostgreSQL kernel.

We require "FORMAT_NAME(internal)" signature:

----
	funcargtypes[0] = INTERNALOID;
	handlerOid = LookupFuncName(list_make1(makeString(format)), 1,
								funcargtypes, true);
----

So any caller can call only routines that use the signature.

Should we add more checks for security? If so, what checks
are needed?

For example, does requiring a prefix such as "copy_" (use
"copy_json" for "json" format) improve security?

For example, we need to register a handler explicitly
(CREATE ACCESS METHOD) when we want to use a new access
method. Should we require an explicit registration for
custom COPY format too?


>  Standard PostgreSQL realisation for new methods to use USING
>  keyword. Every
> new method could have own options (FORMAT is option of internal 'copy
> from/to'
> methods),

Ah, I didn't think about USING.

You suggest "COPY ... USING json" not "COPY ... FORMAT json"
like "CREATE INDEX ... USING custom_index", right? It will
work. If we use this interface, we should reject "COPY
... FORMAT ... USING" (both of FORMAT/USING are specified).


>           it assumes some SetOptions interface, that defines
> an options structure according to the new method requirements.

Sorry. I couldn't find the SetOptions interface in source
code. I found only AT_SetOptions. Did you mean it by "some
SetOptions interface"?

I'm familiar with only access method. It has
IndexAmRoutine::amoptions. Is it a SetOptions interface
example?


FYI: The current patch set doesn't have custom options
support yet. Because we want to start from a minimal feature
set. But we'll add support for custom options eventually.


Thanks,
-- 
kou

  209. Re: Make COPY format extendable: Extract COPY TO format implementations

    Vladlen Popolitov <v.popolitov@postgrespro.ru> — 2025-02-04T10:46:10Z

    Sutou Kouhei писал(а) 2025-02-04 13:29:
Hi
> Hi,
> 
> In <d838025aceeb19c9ff1db702fa55cabf@postgrespro.ru>
>   "Re: Make COPY format extendable: Extract COPY TO format 
> implementations" on Mon, 03 Feb 2025 13:38:04 +0700,
>   Vladlen Popolitov <v.popolitov@postgrespro.ru> wrote:
> 
>> I would like to inform about the security breach in your design of
>> COPY TO/FROM.
> 
> Thanks! I didn't notice it.
> 
>> You use FORMAT option to add new formats, filling it with routine name
>> in shared library. As result any caller can call any routine in
>> PostgreSQL kernel.
> 
> We require "FORMAT_NAME(internal)" signature:
> 
> ----
> 	funcargtypes[0] = INTERNALOID;
> 	handlerOid = LookupFuncName(list_make1(makeString(format)), 1,
> 								funcargtypes, true);
> ----
> 
> So any caller can call only routines that use the signature.
> 
> Should we add more checks for security? If so, what checks
> are needed?
> 
> For example, does requiring a prefix such as "copy_" (use
> "copy_json" for "json" format) improve security?
> 
> For example, we need to register a handler explicitly
> (CREATE ACCESS METHOD) when we want to use a new access
> method. Should we require an explicit registration for
> custom COPY format too?
> 
> 

I think, in case of USING PostgreSQL kernel will call corresponding 
handler,
and it looks secure - the same as for table and index methods handlers.

>>  Standard PostgreSQL realisation for new methods to use USING
>>  keyword. Every
>> new method could have own options (FORMAT is option of internal 'copy
>> from/to'
>> methods),
> 
> Ah, I didn't think about USING.
> 
> You suggest "COPY ... USING json" not "COPY ... FORMAT json"
> like "CREATE INDEX ... USING custom_index", right? It will
> work. If we use this interface, we should reject "COPY
> ... FORMAT ... USING" (both of FORMAT/USING are specified).
> 
> 
I cannot recommend about rejecting, I do not know details
of realisation of this part of code. Just idea - FORMAT value
could be additional option to copy handler or NULL
if it is omitted.
  If you add extensibility, than every handler will be the
extension, that can handle one or more formats.

>>           it assumes some SetOptions interface, that defines
>> an options structure according to the new method requirements.
> 
> Sorry. I couldn't find the SetOptions interface in source
> code. I found only AT_SetOptions. Did you mean it by "some
> SetOptions interface"?
Yes.
> I'm familiar with only access method. It has
> IndexAmRoutine::amoptions. Is it a SetOptions interface
> example?
Yes. I think, it would be compatible with other modules
of source code and could use the same code base to process
options of COPY TO/FROM
> 
> FYI: The current patch set doesn't have custom options
> support yet. Because we want to start from a minimal feature
> set. But we'll add support for custom options eventually.
Sorry for disturbing. I did not have intention to stop your
patch, I would like to point to that details as early as
possible.


-- 
Best regards,

Vladlen Popolitov.

  210. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2025-02-05T01:32:07Z

    On Tue, Feb 4, 2025 at 2:46 AM Vladlen Popolitov
<v.popolitov@postgrespro.ru> wrote:
>
> Sutou Kouhei писал(а) 2025-02-04 13:29:
> Hi
> > Hi,
> >
> > In <d838025aceeb19c9ff1db702fa55cabf@postgrespro.ru>
> >   "Re: Make COPY format extendable: Extract COPY TO format
> > implementations" on Mon, 03 Feb 2025 13:38:04 +0700,
> >   Vladlen Popolitov <v.popolitov@postgrespro.ru> wrote:
> >
> >> I would like to inform about the security breach in your design of
> >> COPY TO/FROM.
> >
> > Thanks! I didn't notice it.
> >
> >> You use FORMAT option to add new formats, filling it with routine name
> >> in shared library. As result any caller can call any routine in
> >> PostgreSQL kernel.
> >
> > We require "FORMAT_NAME(internal)" signature:
> >
> > ----
> >       funcargtypes[0] = INTERNALOID;
> >       handlerOid = LookupFuncName(list_make1(makeString(format)), 1,
> >                                                               funcargtypes, true);
> > ----
> >
> > So any caller can call only routines that use the signature.
> >
> > Should we add more checks for security? If so, what checks
> > are needed?
> >
> > For example, does requiring a prefix such as "copy_" (use
> > "copy_json" for "json" format) improve security?
> >
> > For example, we need to register a handler explicitly
> > (CREATE ACCESS METHOD) when we want to use a new access
> > method. Should we require an explicit registration for
> > custom COPY format too?
> >
> >
>
> I think, in case of USING PostgreSQL kernel will call corresponding
> handler,
> and it looks secure - the same as for table and index methods handlers.

IIUC even with custom copy format patches, we call the corresponding
handler function to get the routines, which is essentially similar to
what we do for table AM, index AM, and tablesample.I don't think we
allow users to call any routine in PostgreSQL core via custom FORMAT
option.

BTW we need to check if the return value type of the handler function
is copy_handler.

>
> >>  Standard PostgreSQL realisation for new methods to use USING
> >>  keyword. Every
> >> new method could have own options (FORMAT is option of internal 'copy
> >> from/to'
> >> methods),
> >
> > Ah, I didn't think about USING.
> >
> > You suggest "COPY ... USING json" not "COPY ... FORMAT json"
> > like "CREATE INDEX ... USING custom_index", right? It will
> > work. If we use this interface, we should reject "COPY
> > ... FORMAT ... USING" (both of FORMAT/USING are specified).
> >
> >
> I cannot recommend about rejecting, I do not know details
> of realisation of this part of code. Just idea - FORMAT value
> could be additional option to copy handler or NULL
> if it is omitted.
>   If you add extensibility, than every handler will be the
> extension, that can handle one or more formats.

Hmm, if we use the USING clause to specify the format type, we end up
having two ways to specify the format type (e.g., 'COPY ... USING
text' and 'COPY .. WITH (format = text)'), which seems to confuse
users.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  211. Re: Make COPY format extendable: Extract COPY TO format implementations

    Vladlen Popolitov <v.popolitov@postgrespro.ru> — 2025-02-05T02:19:27Z

    Masahiko Sawada писал(а) 2025-02-05 08:32:
> On Tue, Feb 4, 2025 at 2:46 AM Vladlen Popolitov

>> >>  Standard PostgreSQL realisation for new methods to use USING
>> >>  keyword. Every
>> >> new method could have own options (FORMAT is option of internal 'copy
>> >> from/to'
>> >> methods),
>> >
>> > Ah, I didn't think about USING.
>> >
>> > You suggest "COPY ... USING json" not "COPY ... FORMAT json"
>> > like "CREATE INDEX ... USING custom_index", right? It will
>> > work. If we use this interface, we should reject "COPY
>> > ... FORMAT ... USING" (both of FORMAT/USING are specified).
>> >
>> >
>> I cannot recommend about rejecting, I do not know details
>> of realisation of this part of code. Just idea - FORMAT value
>> could be additional option to copy handler or NULL
>> if it is omitted.
>>   If you add extensibility, than every handler will be the
>> extension, that can handle one or more formats.
> 
> Hmm, if we use the USING clause to specify the format type, we end up
> having two ways to specify the format type (e.g., 'COPY ... USING
> text' and 'COPY .. WITH (format = text)'), which seems to confuse
> users.
WITH clause has list of options defined by copy method define in USING.
The clause WITH (format=text) has options defined for default copy 
method,
but other methods will define own options. Probably they do not need
the word 'format' in options. The same as in index access methods.
  For example, copy method parquete:
COPY ... USING parquete WITH (row_group_size=1000000)
  copy method parquete need and will define the word 'row_group_size'
in options, the word 'format' will be wrong for it.
-- 
Best regards,

Vladlen Popolitov.

  212. Re: Make COPY format extendable: Extract COPY TO format implementations

    Michael Paquier <michael@paquier.xyz> — 2025-02-05T05:10:15Z

    On Sat, Feb 01, 2025 at 07:12:01PM +0900, Sutou Kouhei wrote:
> For the propose, copyapi.h should not include
> copy{to,from}_internal.h. If we do it, copyto.c includes
> CopyFromState and copyfrom*.c include CopyToState.
> 
> What do you think about the following change? Note that
> extensions must include copy{to,from}_internal.h explicitly
> in addition to copyapi.h.

I was just looking at bit at this series of patch labelled with v31,
to see what is happening here.

In 0001, we have that:

+	/* format-specific routines */
+	const CopyToRoutine *routine;
[...]
-	CopySendEndOfRow(cstate);
+	cstate->routine->CopyToOneRow(cstate, slot);

Having a callback where the copy state is processed once per row is
neat in terms of design for the callbacks and what extensions can do,
and this is much better than what 2889fd23be5 has attempted (later
reverted in 1aa8324b81fa) because we don't do indirect function calls
for each attribute.  Still, I have a question here: what happens for a
COPY TO that involves one attribute, a short field size like an int2
and many rows (the more rows the more pronounced the effect, of
course)?  Could this level of indirection still be the cause of some
regressions in a case like that?  This is the worst case I can think
about, on top of my mind, and I am not seeing tests with few
attributes like this one, where we would try to make this callback as
hot as possible.  This is a performance-sensitive area.
--
Michael

  213. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2025-02-05T06:10:35Z

    On Tue, Feb 4, 2025 at 6:19 PM Vladlen Popolitov
<v.popolitov@postgrespro.ru> wrote:
>
> Masahiko Sawada писал(а) 2025-02-05 08:32:
> > On Tue, Feb 4, 2025 at 2:46 AM Vladlen Popolitov
>
> >> >>  Standard PostgreSQL realisation for new methods to use USING
> >> >>  keyword. Every
> >> >> new method could have own options (FORMAT is option of internal 'copy
> >> >> from/to'
> >> >> methods),
> >> >
> >> > Ah, I didn't think about USING.
> >> >
> >> > You suggest "COPY ... USING json" not "COPY ... FORMAT json"
> >> > like "CREATE INDEX ... USING custom_index", right? It will
> >> > work. If we use this interface, we should reject "COPY
> >> > ... FORMAT ... USING" (both of FORMAT/USING are specified).
> >> >
> >> >
> >> I cannot recommend about rejecting, I do not know details
> >> of realisation of this part of code. Just idea - FORMAT value
> >> could be additional option to copy handler or NULL
> >> if it is omitted.
> >>   If you add extensibility, than every handler will be the
> >> extension, that can handle one or more formats.
> >
> > Hmm, if we use the USING clause to specify the format type, we end up
> > having two ways to specify the format type (e.g., 'COPY ... USING
> > text' and 'COPY .. WITH (format = text)'), which seems to confuse
> > users.
> WITH clause has list of options defined by copy method define in USING.
> The clause WITH (format=text) has options defined for default copy
> method,
> but other methods will define own options. Probably they do not need
> the word 'format' in options. The same as in index access methods.
>   For example, copy method parquete:
> COPY ... USING parquete WITH (row_group_size=1000000)
>   copy method parquete need and will define the word 'row_group_size'
> in options, the word 'format' will be wrong for it.

I think it's orthological between the syntax and options passed to the
custom format extension. For example, even if we specify the options
like "COPY ... WITH (format 'parquet', row_group_size '1000000',
on_error 'ignore)", we can pass only non-built-in options (i.e. only
row_group_size) to the extension.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  214. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2025-02-05T06:20:51Z

    On Tue, Feb 4, 2025 at 9:10 PM Michael Paquier <michael@paquier.xyz> wrote:
>
> On Sat, Feb 01, 2025 at 07:12:01PM +0900, Sutou Kouhei wrote:
> > For the propose, copyapi.h should not include
> > copy{to,from}_internal.h. If we do it, copyto.c includes
> > CopyFromState and copyfrom*.c include CopyToState.
> >
> > What do you think about the following change? Note that
> > extensions must include copy{to,from}_internal.h explicitly
> > in addition to copyapi.h.
>
> I was just looking at bit at this series of patch labelled with v31,
> to see what is happening here.
>
> In 0001, we have that:
>
> +       /* format-specific routines */
> +       const CopyToRoutine *routine;
> [...]
> -       CopySendEndOfRow(cstate);
> +       cstate->routine->CopyToOneRow(cstate, slot);
>
> Having a callback where the copy state is processed once per row is
> neat in terms of design for the callbacks and what extensions can do,
> and this is much better than what 2889fd23be5 has attempted (later
> reverted in 1aa8324b81fa) because we don't do indirect function calls
> for each attribute.  Still, I have a question here: what happens for a
> COPY TO that involves one attribute, a short field size like an int2
> and many rows (the more rows the more pronounced the effect, of
> course)?  Could this level of indirection still be the cause of some
> regressions in a case like that?  This is the worst case I can think
> about, on top of my mind, and I am not seeing tests with few
> attributes like this one, where we would try to make this callback as
> hot as possible.  This is a performance-sensitive area.

FYI when Sutou-san last measured the performance[1], it showed a
slight speed up even with fewer columns (5 columns) in both COPY TO
and COPY FROM cases. The callback design has not changed since then.
But it would be a good idea to run the benchmark with a table having a
single small size column.

Regards,

[1] https://www.postgresql.org/message-id/20241114.161948.1677325020727842666.kou%40clear-code.com

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  215. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2025-02-05T07:31:56Z

    Hi,

In <eb59c12bb36207c65f23719f255eb69b@postgrespro.ru>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Tue, 04 Feb 2025 17:46:10 +0700,
  Vladlen Popolitov <v.popolitov@postgrespro.ru> wrote:

> I think, in case of USING PostgreSQL kernel will call corresponding
> handler,
> and it looks secure - the same as for table and index methods
> handlers.

We use similar approach that is used by table sampling
method. We can use a new table sampling method by just
adding a "method_name(internal) RETURNS tsm_handler"
function. Is it not secure too?

>  If you add extensibility, than every handler will be the
> extension, that can handle one or more formats.

Hmm. It may be a needless extensibility. Is it useful? I
feel that it increases complexity when we implement a custom
format handler. We can just implement one handler per custom
format. If we want to share implementation details in
multiple handlers, we can just share internal C
functions.

If we require one handler per custom format, it'll simpler
than one handler for multiple custom formats.

>>>           it assumes some SetOptions interface, that defines
>>> an options structure according to the new method requirements.
>> Sorry. I couldn't find the SetOptions interface in source
>> code. I found only AT_SetOptions. Did you mean it by "some
>> SetOptions interface"?
> Yes.
>> I'm familiar with only access method. It has
>> IndexAmRoutine::amoptions. Is it a SetOptions interface
>> example?
> Yes. I think, it would be compatible with other modules
> of source code and could use the same code base to process
> options of COPY TO/FROM

Thanks. I thought that there is a common interface pattern
for SetOptions. But it seems that it's a feature that is
implemented in many extension points.

If we implement custom options support eventually, does it
satisfy the "SetOptions interface"?


Thanks,
-- 
kou

  216. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2025-02-05T07:37:25Z

    Hi,

In <CAD21AoDCH1io_dGtsmnmZ4bUWfdPhEUe_8VQNvi31+78Pt7KdQ@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Tue, 4 Feb 2025 17:32:07 -0800,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

> BTW we need to check if the return value type of the handler function
> is copy_handler.

Oh, can we do it without calling a function? It seems that
FmgrInfo doesn't have return value type information. Should
we read pg_catalog.pg_proc or something for it?


Thanks,
-- 
kou

  217. Re: Make COPY format extendable: Extract COPY TO format implementations

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2025-02-05T11:49:14Z

    On 2025-Feb-03, Vladlen Popolitov wrote:

> You use FORMAT option to add new formats, filling it with routine name
> in shared library. As result any caller can call any routine in PostgreSQL
> kernel.
> I think, it will start competition, who can find most dangerous routine
> to call just from COPY FROM command.

Hah.

Maybe it would be a better UI to require that COPY format handlers are
registered explicitly before they can be used:

 CREATE ACCESS METHOD copy_yaml TYPE copy HANDLER copy_yaml_handler;

... and then when the FORMAT is not recognized as one of the hardcoded
methods, we go look in pg_am for one with amtype='c' and the given name.
That gives you the function that initializes the Copy state.

This is convenient enough because system administrators can add COPY
formats that anyone can use, and doesn't allow to call arbitrary
functions via COPY.

-- 
Álvaro Herrera               48°01'N 7°57'E  —  https://www.EnterpriseDB.com/
"I can't go to a restaurant and order food because I keep looking at the
fonts on the menu.  Five minutes later I realize that it's also talking
about food" (Donald Knuth)

  218. Re: Make COPY format extendable: Extract COPY TO format implementations

    Vladlen Popolitov <v.popolitov@postgrespro.ru> — 2025-02-05T15:26:14Z

    Álvaro Herrera писал(а) 2025-02-05 18:49:
> On 2025-Feb-03, Vladlen Popolitov wrote:
> 
>> You use FORMAT option to add new formats, filling it with routine name
>> in shared library. As result any caller can call any routine in 
>> PostgreSQL
>> kernel.
>> I think, it will start competition, who can find most dangerous 
>> routine
>> to call just from COPY FROM command.
> 
> Hah.
> 
> Maybe it would be a better UI to require that COPY format handlers are
> registered explicitly before they can be used:
> 
>  CREATE ACCESS METHOD copy_yaml TYPE copy HANDLER copy_yaml_handler;
> 
> ... and then when the FORMAT is not recognized as one of the hardcoded
> methods, we go look in pg_am for one with amtype='c' and the given 
> name.
> That gives you the function that initializes the Copy state.
> 
> This is convenient enough because system administrators can add COPY
> formats that anyone can use, and doesn't allow to call arbitrary
> functions via COPY.

Yes! It is what I propose. This looks much safer and already used in 
access methods creation.

-- 
Best regards,

Vladlen Popolitov.

  219. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2025-02-05T18:30:08Z

    On Wed, Feb 5, 2025 at 3:49 AM Álvaro Herrera <alvherre@alvh.no-ip.org> wrote:
>
> On 2025-Feb-03, Vladlen Popolitov wrote:
>
> > You use FORMAT option to add new formats, filling it with routine name
> > in shared library. As result any caller can call any routine in PostgreSQL
> > kernel.
> > I think, it will start competition, who can find most dangerous routine
> > to call just from COPY FROM command.
>
> Hah.
>
> Maybe it would be a better UI to require that COPY format handlers are
> registered explicitly before they can be used:
>
>  CREATE ACCESS METHOD copy_yaml TYPE copy HANDLER copy_yaml_handler;
>
> ... and then when the FORMAT is not recognized as one of the hardcoded
> methods, we go look in pg_am for one with amtype='c' and the given name.
> That gives you the function that initializes the Copy state.
>
> This is convenient enough because system administrators can add COPY
> formats that anyone can use, and doesn't allow to call arbitrary
> functions via COPY.

I think that the patch needs to check if the function's result type is
COPY_HANDLEROID by using get_func_rettype(), before calling it. But
with this check, we can prevent arbitrary functions from being called
via COPY. Why do we need to extend CREATE ACCESS METHOD too for that
purpose?

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  220. Re: Make COPY format extendable: Extract COPY TO format implementations

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2025-02-05T19:26:20Z

    On 2025-Feb-05, Masahiko Sawada wrote:

> I think that the patch needs to check if the function's result type is
> COPY_HANDLEROID by using get_func_rettype(), before calling it. But
> with this check, we can prevent arbitrary functions from being called
> via COPY. Why do we need to extend CREATE ACCESS METHOD too for that
> purpose?

It's a nicer UI than a bare CREATE FUNCTION, but perhaps it is overkill.
IIRC the reason we require CREATE ACCESS METHOD for table AMs is so that
we acquire a pg_am entry with an OID that can be referenced from
elsewhere, for instance you can't drop an AM if tables are using it; but
you can't use COPY in rules or anything like that that's going to be
stored permanently.  Perhaps you're right that we don't need this for
extensible COPY FORMAT.

-- 
Álvaro Herrera         PostgreSQL Developer  —  https://www.EnterpriseDB.com/

  221. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2025-02-05T20:29:44Z

    On Tue, Feb 4, 2025 at 11:37 PM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <CAD21AoDCH1io_dGtsmnmZ4bUWfdPhEUe_8VQNvi31+78Pt7KdQ@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Tue, 4 Feb 2025 17:32:07 -0800,
>   Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> > BTW we need to check if the return value type of the handler function
> > is copy_handler.
>
> Oh, can we do it without calling a function?

Yes.

> It seems that
> FmgrInfo doesn't have return value type information. Should
> we read pg_catalog.pg_proc or something for it?

Yes, we can do like what we do for TABLESAMPLE for example:

    /* check that handler has correct return type */
    if (get_func_rettype(handlerOid) != TSM_HANDLEROID)
        ereport(ERROR,
                (errcode(ERRCODE_WRONG_OBJECT_TYPE),
                 errmsg("function %s must return type %s",
                        NameListToString(rts->method), "tsm_handler"),
                 parser_errposition(pstate, rts->location)));

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  222. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2025-02-06T12:06:31Z

    Hi,

In <CAD21AoDCd9pKZ2XMOUmnmteC60NYBLr80FWY56Nn3NEbxVxdeQ@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Wed, 5 Feb 2025 12:29:44 -0800,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

>> It seems that
>> FmgrInfo doesn't have return value type information. Should
>> we read pg_catalog.pg_proc or something for it?
> 
> Yes, we can do like what we do for TABLESAMPLE for example:
> 
>     /* check that handler has correct return type */
>     if (get_func_rettype(handlerOid) != TSM_HANDLEROID)
>         ereport(ERROR,
>                 (errcode(ERRCODE_WRONG_OBJECT_TYPE),
>                  errmsg("function %s must return type %s",
>                         NameListToString(rts->method), "tsm_handler"),
>                  parser_errposition(pstate, rts->location)));

Thanks! I didn't know get_func_rettype().

I attach the v32 patch set. 0003 is only changed. The
following check is added. It's similar to TABLESAMPLE's one.


	/* check that handler has correct return type */
	if (get_func_rettype(handlerOid) != COPY_HANDLEROID)
		ereport(ERROR,
				(errcode(ERRCODE_WRONG_OBJECT_TYPE),
				 errmsg("function %s must return type %s",
						format, "copy_handler"),
				 parser_errposition(pstate, defel->location)));


Thanks,
-- 
kou

  223. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2025-02-07T13:01:17Z

    Hi,

In <CAD21AoBkDE4JwjPgcLxSEwqu3nN4VXjkYS9vpRQDwA2GwNQCsg@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Tue, 4 Feb 2025 22:20:51 -0800,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

>> I was just looking at bit at this series of patch labelled with v31,
>> to see what is happening here.
>>
>> In 0001, we have that:
>>
>> +       /* format-specific routines */
>> +       const CopyToRoutine *routine;
>> [...]
>> -       CopySendEndOfRow(cstate);
>> +       cstate->routine->CopyToOneRow(cstate, slot);
>>
>> Having a callback where the copy state is processed once per row is
>> neat in terms of design for the callbacks and what extensions can do,
>> and this is much better than what 2889fd23be5 has attempted (later
>> reverted in 1aa8324b81fa) because we don't do indirect function calls
>> for each attribute.  Still, I have a question here: what happens for a
>> COPY TO that involves one attribute, a short field size like an int2
>> and many rows (the more rows the more pronounced the effect, of
>> course)?  Could this level of indirection still be the cause of some
>> regressions in a case like that?  This is the worst case I can think
>> about, on top of my mind, and I am not seeing tests with few
>> attributes like this one, where we would try to make this callback as
>> hot as possible.  This is a performance-sensitive area.
> 
> FYI when Sutou-san last measured the performance[1], it showed a
> slight speed up even with fewer columns (5 columns) in both COPY TO
> and COPY FROM cases. The callback design has not changed since then.
> But it would be a good idea to run the benchmark with a table having a
> single small size column.
> 
> [1] https://www.postgresql.org/message-id/20241114.161948.1677325020727842666.kou%40clear-code.com

I measured v31 patch set with 1,6,11,16,21,26,31 int2
columns. See the attached PDF for 0001 and 0002 result.

0001 - to:

It's faster than master when the number of rows are
1,000,000-5,000,000.

It's almost same as master when the number of rows are
6,000,000-10,000,000.

There is no significant slow down when the number of columns
is 1.

0001 - from:

0001 doesn't change COPY FROM code. So the differences are
not real difference.

0002 - to:

0002 doesn't change COPY TO code. So "0001 - to" and "0002 -
to" must be the same result. But 0002 is faster than master
for all cases. It shows that the CopyToOneRow() approach
doesn't have significant slow down.

0002 - from:

0002 changes COPY FROM code. So this may have performance
impact.

It's almost same as master when data is smaller
((1,000,000-2,000,000 rows) or (3,000,000 rows and 1,6,11,16
columns)).

It's faster than master when data is larger.

There is no significant slow down by 0002.


Thanks,
-- 
kou

  224. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2025-02-20T23:28:26Z

    On Fri, Feb 7, 2025 at 5:01 AM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <CAD21AoBkDE4JwjPgcLxSEwqu3nN4VXjkYS9vpRQDwA2GwNQCsg@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Tue, 4 Feb 2025 22:20:51 -0800,
>   Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> >> I was just looking at bit at this series of patch labelled with v31,
> >> to see what is happening here.
> >>
> >> In 0001, we have that:
> >>
> >> +       /* format-specific routines */
> >> +       const CopyToRoutine *routine;
> >> [...]
> >> -       CopySendEndOfRow(cstate);
> >> +       cstate->routine->CopyToOneRow(cstate, slot);
> >>
> >> Having a callback where the copy state is processed once per row is
> >> neat in terms of design for the callbacks and what extensions can do,
> >> and this is much better than what 2889fd23be5 has attempted (later
> >> reverted in 1aa8324b81fa) because we don't do indirect function calls
> >> for each attribute.  Still, I have a question here: what happens for a
> >> COPY TO that involves one attribute, a short field size like an int2
> >> and many rows (the more rows the more pronounced the effect, of
> >> course)?  Could this level of indirection still be the cause of some
> >> regressions in a case like that?  This is the worst case I can think
> >> about, on top of my mind, and I am not seeing tests with few
> >> attributes like this one, where we would try to make this callback as
> >> hot as possible.  This is a performance-sensitive area.
> >
> > FYI when Sutou-san last measured the performance[1], it showed a
> > slight speed up even with fewer columns (5 columns) in both COPY TO
> > and COPY FROM cases. The callback design has not changed since then.
> > But it would be a good idea to run the benchmark with a table having a
> > single small size column.
> >
> > [1] https://www.postgresql.org/message-id/20241114.161948.1677325020727842666.kou%40clear-code.com
>
> I measured v31 patch set with 1,6,11,16,21,26,31 int2
> columns. See the attached PDF for 0001 and 0002 result.
>
> 0001 - to:
>
> It's faster than master when the number of rows are
> 1,000,000-5,000,000.
>
> It's almost same as master when the number of rows are
> 6,000,000-10,000,000.
>
> There is no significant slow down when the number of columns
> is 1.
>
> 0001 - from:
>
> 0001 doesn't change COPY FROM code. So the differences are
> not real difference.
>
> 0002 - to:
>
> 0002 doesn't change COPY TO code. So "0001 - to" and "0002 -
> to" must be the same result. But 0002 is faster than master
> for all cases. It shows that the CopyToOneRow() approach
> doesn't have significant slow down.
>
> 0002 - from:
>
> 0002 changes COPY FROM code. So this may have performance
> impact.
>
> It's almost same as master when data is smaller
> ((1,000,000-2,000,000 rows) or (3,000,000 rows and 1,6,11,16
> columns)).
>
> It's faster than master when data is larger.
>
> There is no significant slow down by 0002.
>

Thank you for sharing the benchmark results. That looks good to me.

Looking at the 0001 patch again, I have a question: we have
CopyToTextLikeOneRow() for both CSV and text format:

+/* Implementation of the per-row callback for text format */
+static void
+CopyToTextOneRow(CopyToState cstate, TupleTableSlot *slot)
+{
+       CopyToTextLikeOneRow(cstate, slot, false);
+}
+
+/* Implementation of the per-row callback for CSV format */
+static void
+CopyToCSVOneRow(CopyToState cstate, TupleTableSlot *slot)
+{
+       CopyToTextLikeOneRow(cstate, slot, true);
+}

These two functions pass different is_csv value to that function,
which is used as follows:

+                       if (is_csv)
+                               CopyAttributeOutCSV(cstate, string,
+
 cstate->opts.force_quote_flags[attnum - 1]);
+                       else
+                               CopyAttributeOutText(cstate, string);

However, we can know whether the format is CSV or text by checking
cstate->opts.csv_mode instead of passing is_csv. That way, we can
directly call CopyToTextLikeOneRow() but not via CopyToCSVOneRow() or
CopyToTextOneRow(). It would not help performance since we already
inline CopyToTextLikeOneRow(), but it looks simpler.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  225. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2025-02-21T02:48:12Z

    Hi,

In <CAD21AoAni3cKToPfdShTsc0NmaJOtbJuUb=skyz3Udj7HZY7dA@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Thu, 20 Feb 2025 15:28:26 -0800,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

> Looking at the 0001 patch again, I have a question: we have
> CopyToTextLikeOneRow() for both CSV and text format:
> 
> +/* Implementation of the per-row callback for text format */
> +static void
> +CopyToTextOneRow(CopyToState cstate, TupleTableSlot *slot)
> +{
> +       CopyToTextLikeOneRow(cstate, slot, false);
> +}
> +
> +/* Implementation of the per-row callback for CSV format */
> +static void
> +CopyToCSVOneRow(CopyToState cstate, TupleTableSlot *slot)
> +{
> +       CopyToTextLikeOneRow(cstate, slot, true);
> +}
> 
> These two functions pass different is_csv value to that function,
> which is used as follows:
> 
> +                       if (is_csv)
> +                               CopyAttributeOutCSV(cstate, string,
> +
>  cstate->opts.force_quote_flags[attnum - 1]);
> +                       else
> +                               CopyAttributeOutText(cstate, string);
> 
> However, we can know whether the format is CSV or text by checking
> cstate->opts.csv_mode instead of passing is_csv. That way, we can
> directly call CopyToTextLikeOneRow() but not via CopyToCSVOneRow() or
> CopyToTextOneRow(). It would not help performance since we already
> inline CopyToTextLikeOneRow(), but it looks simpler.

This means the following, right?

1. We remove CopyToTextOneRow() and CopyToCSVOneRow()
2. We remove "bool is_csv" parameter from CopyToTextLikeOneRow()
   and use cstate->opts.csv_mode in CopyToTextLikeOneRow()
   instead of is_csv
3. We use CopyToTextLikeOneRow() for
   CopyToRoutineText::CopyToOneRow and
   CopyToRoutineCSV::CopyToOneRow

If we use this approach, we can't remove the following
branch in compile time:

+			if (is_csv)
+				CopyAttributeOutCSV(cstate, string,
+									cstate->opts.force_quote_flags[attnum - 1]);
+			else
+				CopyAttributeOutText(cstate, string);

We can remove the branch in compile time with the current
approach (constant argument + inline).

It may have a negative performance impact because the "if"
is used many times with large data. (That's why we choose
the constant argument + inline approach in this thread.)


Thanks,
-- 
kou

  226. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2025-02-25T22:05:28Z

    On Thu, Feb 20, 2025 at 6:48 PM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <CAD21AoAni3cKToPfdShTsc0NmaJOtbJuUb=skyz3Udj7HZY7dA@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Thu, 20 Feb 2025 15:28:26 -0800,
>   Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> > Looking at the 0001 patch again, I have a question: we have
> > CopyToTextLikeOneRow() for both CSV and text format:
> >
> > +/* Implementation of the per-row callback for text format */
> > +static void
> > +CopyToTextOneRow(CopyToState cstate, TupleTableSlot *slot)
> > +{
> > +       CopyToTextLikeOneRow(cstate, slot, false);
> > +}
> > +
> > +/* Implementation of the per-row callback for CSV format */
> > +static void
> > +CopyToCSVOneRow(CopyToState cstate, TupleTableSlot *slot)
> > +{
> > +       CopyToTextLikeOneRow(cstate, slot, true);
> > +}
> >
> > These two functions pass different is_csv value to that function,
> > which is used as follows:
> >
> > +                       if (is_csv)
> > +                               CopyAttributeOutCSV(cstate, string,
> > +
> >  cstate->opts.force_quote_flags[attnum - 1]);
> > +                       else
> > +                               CopyAttributeOutText(cstate, string);
> >
> > However, we can know whether the format is CSV or text by checking
> > cstate->opts.csv_mode instead of passing is_csv. That way, we can
> > directly call CopyToTextLikeOneRow() but not via CopyToCSVOneRow() or
> > CopyToTextOneRow(). It would not help performance since we already
> > inline CopyToTextLikeOneRow(), but it looks simpler.
>
> This means the following, right?
>
> 1. We remove CopyToTextOneRow() and CopyToCSVOneRow()
> 2. We remove "bool is_csv" parameter from CopyToTextLikeOneRow()
>    and use cstate->opts.csv_mode in CopyToTextLikeOneRow()
>    instead of is_csv
> 3. We use CopyToTextLikeOneRow() for
>    CopyToRoutineText::CopyToOneRow and
>    CopyToRoutineCSV::CopyToOneRow
>
> If we use this approach, we can't remove the following
> branch in compile time:
>
> +                       if (is_csv)
> +                               CopyAttributeOutCSV(cstate, string,
> +                                                                       cstate->opts.force_quote_flags[attnum - 1]);
> +                       else
> +                               CopyAttributeOutText(cstate, string);
>
> We can remove the branch in compile time with the current
> approach (constant argument + inline).
>
> It may have a negative performance impact because the "if"
> is used many times with large data. (That's why we choose
> the constant argument + inline approach in this thread.)
>

Thank you for the explanation, I missed that fact. I'm fine with having is_csv.

The first two patches are refactoring patches (+ small performance
improvements). I've reviewed these patches again and attached the
updated patches. I reorganized the function order and updated comments
etc. I find that these patches are reasonably ready to push. Could you
review these versions? I'm going to push them, barring objections and
further comments.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  227. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2025-02-25T23:51:59Z

    Hi,

In <CAD21AoBjzkL2Lv7j4teaHBZvNmKctQtH6X71kN_sj6Fm-+VvJQ@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Tue, 25 Feb 2025 14:05:28 -0800,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

> The first two patches are refactoring patches (+ small performance
> improvements). I've reviewed these patches again and attached the
> updated patches. I reorganized the function order and updated comments
> etc. I find that these patches are reasonably ready to push. Could you
> review these versions? I'm going to push them, barring objections and
> further comments.

Sure. Here are some minor comments:

0001:

Commit message:

> or CSV mode. The performance benchmark results showed ~5% performance
> gain intext or CSV mode.

intext -> in text

> --- a/src/backend/commands/copyto.c
> +++ b/src/backend/commands/copyto.c

> @@ -20,6 +20,7 @@

>  #include "commands/copy.h"
> +#include "commands/copyapi.h"

We can remove '#include "commands/copy.h"' because it's
included in copyapi.h. (0002 does it.)

> @@ -254,6 +502,35 @@ CopySendEndOfRow(CopyToState cstate)

> +/*
> + * Wrapper function of CopySendEndOfRow for text and CSV formats. Sends the
> + * the line termination and do common appropriate things for the end of row.
> + */

Sends the the line ->
Sends the line

> --- /dev/null
> +++ b/src/include/commands/copyapi.h

> +	/* End a COPY TO. This callback is called once at the end of COPY FROM */

The last "." is missing: ... COPY FROM.

0002:

Commit message:

> This change is a preliminary step towards making the COPY TO command
> extensible in terms of output formats.

COPY TO -> COPY FROM

> --- a/src/backend/commands/copyfromparse.c
> +++ b/src/backend/commands/copyfromparse.c

> @@ -1087,7 +1132,7 @@ NextCopyFrom(CopyFromState cstate, ExprContext *econtext,

>  static bool
> -CopyReadLine(CopyFromState cstate)
> +CopyReadLine(CopyFromState cstate, bool is_csv)

> @@ -1163,7 +1208,7 @@ CopyReadLine(CopyFromState cstate)

>  static bool
> -CopyReadLineText(CopyFromState cstate)
> +CopyReadLineText(CopyFromState cstate, bool is_csv)

We may want to add a comment why we don't use "inline" nor
"pg_attribute_always_inline" here:

https://www.postgresql.org/message-id/CAD21AoBNfKDbJnu-zONNpG820ZXYC0fuTSLrJ-UdRqU4qp2wog%40mail.gmail.com

> Yes, I'm not sure it's really necessary to make it inline since the
> benchmark results don't show much difference. Probably this is because
> the function has 'is_csv' in some 'if' branches but the compiler
> cannot optimize out the whole 'if' branches as most 'if' branches
> check 'is_csv' and other variables.

Or we can add "inline" not "pg_attribute_always_inline" here
as a hint for compiler.

> --- a/src/include/commands/copyapi.h
> +++ b/src/include/commands/copyapi.h

> @@ -52,4 +52,50 @@ typedef struct CopyToRoutine

> +	/* End a COPY FROM. This callback is called once at the end of COPY FROM */

The last "." is missing: ... COPY FROM.


I think that these patches are ready to push too.

Thanks,
-- 
kou

  228. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2025-02-26T01:14:43Z

    On Tue, Feb 25, 2025 at 3:52 PM Sutou Kouhei <kou@clear-code.com> wrote:
>
>

Thank you for reviewing the patches. I've addressed comments except
for the following comment:

> > --- a/src/backend/commands/copyfromparse.c
> > +++ b/src/backend/commands/copyfromparse.c
>
> > @@ -1087,7 +1132,7 @@ NextCopyFrom(CopyFromState cstate, ExprContext *econtext,
>
> >  static bool
> > -CopyReadLine(CopyFromState cstate)
> > +CopyReadLine(CopyFromState cstate, bool is_csv)
>
> > @@ -1163,7 +1208,7 @@ CopyReadLine(CopyFromState cstate)
>
> >  static bool
> > -CopyReadLineText(CopyFromState cstate)
> > +CopyReadLineText(CopyFromState cstate, bool is_csv)
>
> We may want to add a comment why we don't use "inline" nor
> "pg_attribute_always_inline" here:
>
> https://www.postgresql.org/message-id/CAD21AoBNfKDbJnu-zONNpG820ZXYC0fuTSLrJ-UdRqU4qp2wog%40mail.gmail.com
>
> > Yes, I'm not sure it's really necessary to make it inline since the
> > benchmark results don't show much difference. Probably this is because
> > the function has 'is_csv' in some 'if' branches but the compiler
> > cannot optimize out the whole 'if' branches as most 'if' branches
> > check 'is_csv' and other variables.
>
> Or we can add "inline" not "pg_attribute_always_inline" here
> as a hint for compiler.

I think we should not add inline unless we see a performance
improvement. Also, I find that it would be independent with this
refactoring so we can add it later if needed.

I've attached updated patches.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  229. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2025-02-26T02:08:26Z

    Hi,

In <CAD21AoB3TiyuCcu02itGktUE6L4YGqwWT_LRtYrFkW7xedoe+g@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Tue, 25 Feb 2025 17:14:43 -0800,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

> I've attached updated patches.

Thanks.

I found one more missing last ".":

0002:

> --- a/src/backend/commands/copyfrom.c
> +++ b/src/backend/commands/copyfrom.c

> @@ -106,6 +106,145 @@ typedef struct CopyMultiInsertInfo

> +/*
> + * Built-in format-specific routines. One-row callbacks are defined in
> + * copyfromparse.c
> + */

copyfromparse.c -> copyfromparse.c.


Could you push them?


Thanks,
-- 
kou

  230. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2025-02-27T23:24:26Z

    On Tue, Feb 25, 2025 at 6:08 PM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <CAD21AoB3TiyuCcu02itGktUE6L4YGqwWT_LRtYrFkW7xedoe+g@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Tue, 25 Feb 2025 17:14:43 -0800,
>   Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> > I've attached updated patches.
>
> Thanks.
>
> I found one more missing last ".":
>
> 0002:
>
> > --- a/src/backend/commands/copyfrom.c
> > +++ b/src/backend/commands/copyfrom.c
>
> > @@ -106,6 +106,145 @@ typedef struct CopyMultiInsertInfo
>
> > +/*
> > + * Built-in format-specific routines. One-row callbacks are defined in
> > + * copyfromparse.c
> > + */
>
> copyfromparse.c -> copyfromparse.c.
>
>
> Could you push them?
>

Pushed the 0001 patch.

Regarding the 0002 patch, I realized we stopped exposing
NextCopyFromRawFields() function:

 --- a/src/include/commands/copy.h
+++ b/src/include/commands/copy.h
@@ -107,8 +107,6 @@ extern CopyFromState BeginCopyFrom(ParseState
*pstate, Relation rel, Node *where
 extern void EndCopyFrom(CopyFromState cstate);
 extern bool NextCopyFrom(CopyFromState cstate, ExprContext *econtext,
                         Datum *values, bool *nulls);
-extern bool NextCopyFromRawFields(CopyFromState cstate,
-                                 char ***fields, int *nfields);

I think that this change is not relevant with the refactoring and
probably we should keep it exposed as extension might be using it.
Considering that we added pg_attribute_always_inline to the function,
does it work even if we omit pg_attribute_always_inline to its
function declaration in the copy.h file?

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  231. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2025-02-28T03:57:09Z

    Hi,

In <CAD21AoDABLkUTTOwWa1he6gbc=nM46COMu-BvWjc_i6USnNbHw@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Thu, 27 Feb 2025 15:24:26 -0800,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

> Pushed the 0001 patch.

Thanks!

> Regarding the 0002 patch, I realized we stopped exposing
> NextCopyFromRawFields() function:
> 
>  --- a/src/include/commands/copy.h
> +++ b/src/include/commands/copy.h
> @@ -107,8 +107,6 @@ extern CopyFromState BeginCopyFrom(ParseState
> *pstate, Relation rel, Node *where
>  extern void EndCopyFrom(CopyFromState cstate);
>  extern bool NextCopyFrom(CopyFromState cstate, ExprContext *econtext,
>                          Datum *values, bool *nulls);
> -extern bool NextCopyFromRawFields(CopyFromState cstate,
> -                                 char ***fields, int *nfields);
> 
> I think that this change is not relevant with the refactoring and
> probably we should keep it exposed as extension might be using it.
> Considering that we added pg_attribute_always_inline to the function,
> does it work even if we omit pg_attribute_always_inline to its
> function declaration in the copy.h file?

Unfortunately, no. The inline + constant argument
optimization requires "static".

How about the following?

static pg_attribute_always_inline bool
NextCopyFromRawFieldsInternal(CopyFromState cstate, char ***fields, int *nfields, bool is_csv)
{
    ...
}

bool
NextCopyFromRawFields(CopyFromState cstate, char ***fields, int *nfields)
{
    return NextCopyFromRawFieldsInternal(cstate, fields, nfields, cstate->opts.csv_mode);
}


Thanks,
-- 
kou

  232. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2025-02-28T19:50:39Z

    On Thu, Feb 27, 2025 at 7:57 PM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <CAD21AoDABLkUTTOwWa1he6gbc=nM46COMu-BvWjc_i6USnNbHw@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Thu, 27 Feb 2025 15:24:26 -0800,
>   Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> > Pushed the 0001 patch.
>
> Thanks!
>
> > Regarding the 0002 patch, I realized we stopped exposing
> > NextCopyFromRawFields() function:
> >
> >  --- a/src/include/commands/copy.h
> > +++ b/src/include/commands/copy.h
> > @@ -107,8 +107,6 @@ extern CopyFromState BeginCopyFrom(ParseState
> > *pstate, Relation rel, Node *where
> >  extern void EndCopyFrom(CopyFromState cstate);
> >  extern bool NextCopyFrom(CopyFromState cstate, ExprContext *econtext,
> >                          Datum *values, bool *nulls);
> > -extern bool NextCopyFromRawFields(CopyFromState cstate,
> > -                                 char ***fields, int *nfields);
> >
> > I think that this change is not relevant with the refactoring and
> > probably we should keep it exposed as extension might be using it.
> > Considering that we added pg_attribute_always_inline to the function,
> > does it work even if we omit pg_attribute_always_inline to its
> > function declaration in the copy.h file?
>
> Unfortunately, no. The inline + constant argument
> optimization requires "static".
>
> How about the following?
>
> static pg_attribute_always_inline bool
> NextCopyFromRawFieldsInternal(CopyFromState cstate, char ***fields, int *nfields, bool is_csv)
> {
>     ...
> }
>
> bool
> NextCopyFromRawFields(CopyFromState cstate, char ***fields, int *nfields)
> {
>     return NextCopyFromRawFieldsInternal(cstate, fields, nfields, cstate->opts.csv_mode);
> }
>

Thank you for the confirmation.

I initially thought it would be acceptable to stop
NextCopyFromRawFields exposed since NextCopyFrom() could serve as an
alternative. For example, the NextCopyFromRawFields() function was
originally exposed in commit 8ddc05fb01ee2c primarily to support
extension modules like file_fdw but file_fdw wasn't utilizing this
API. I pushed the patch without the above change. Unfortunately, this
commit subsequently broke file_text_array_fdw[1] and made BF animal
crake unhappy[2].

Upon examining file_text_array_fdw more closely, I realized that
NextCopyFrom() may not be a suitable replacement for
NextCopyFromRawFields() in certain scenarios. Specifically,
NextCopyFrom() assumes that the caller has prior knowledge of the
source data's column count, making it inadequate for cases where
extensions like file_text_array_fdw need to construct an array of
source data with an unknown number of columns. In such situations,
NextCopyFromRawFields() proves to be more practical. Given these
considerations, I'm now leaning towards implementing the proposed
change. Thoughts?

Regards,

[1] https://github.com/adunstan/file_text_array_fdw
[2] https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=crake&dt=2025-02-28%2018%3A47%3A02

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  233. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2025-02-28T21:58:21Z

    Hi,

In <CAD21AoDr13=dx+k8gmQnR5_bY+NskyN4mbSWN0KhQncL6xuPMA@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 28 Feb 2025 11:50:39 -0800,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

> I initially thought it would be acceptable to stop
> NextCopyFromRawFields exposed since NextCopyFrom() could serve as an
> alternative. For example, the NextCopyFromRawFields() function was
> originally exposed in commit 8ddc05fb01ee2c primarily to support
> extension modules like file_fdw but file_fdw wasn't utilizing this
> API. I pushed the patch without the above change. Unfortunately, this
> commit subsequently broke file_text_array_fdw[1] and made BF animal
> crake unhappy[2].
>
> [1] https://github.com/adunstan/file_text_array_fdw
> [2] https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=crake&dt=2025-02-28%2018%3A47%3A02

Thanks for the try!

> Upon examining file_text_array_fdw more closely, I realized that
> NextCopyFrom() may not be a suitable replacement for
> NextCopyFromRawFields() in certain scenarios. Specifically,
> NextCopyFrom() assumes that the caller has prior knowledge of the
> source data's column count, making it inadequate for cases where
> extensions like file_text_array_fdw need to construct an array of
> source data with an unknown number of columns. In such situations,
> NextCopyFromRawFields() proves to be more practical. Given these
> considerations, I'm now leaning towards implementing the proposed
> change. Thoughts?

You suggest that we re-export NextCopyFromRawFields() (as a
wrapper of static inline version) for backward
compatibility, right? It makes sense. We should keep
backward compatibility because there is a use-case of
NextCopyFromRawFields().


Thanks,
-- 
kou

  234. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2025-02-28T22:00:18Z

    On Fri, Feb 28, 2025 at 1:58 PM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <CAD21AoDr13=dx+k8gmQnR5_bY+NskyN4mbSWN0KhQncL6xuPMA@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 28 Feb 2025 11:50:39 -0800,
>   Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> > I initially thought it would be acceptable to stop
> > NextCopyFromRawFields exposed since NextCopyFrom() could serve as an
> > alternative. For example, the NextCopyFromRawFields() function was
> > originally exposed in commit 8ddc05fb01ee2c primarily to support
> > extension modules like file_fdw but file_fdw wasn't utilizing this
> > API. I pushed the patch without the above change. Unfortunately, this
> > commit subsequently broke file_text_array_fdw[1] and made BF animal
> > crake unhappy[2].
> >
> > [1] https://github.com/adunstan/file_text_array_fdw
> > [2] https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=crake&dt=2025-02-28%2018%3A47%3A02
>
> Thanks for the try!
>
> > Upon examining file_text_array_fdw more closely, I realized that
> > NextCopyFrom() may not be a suitable replacement for
> > NextCopyFromRawFields() in certain scenarios. Specifically,
> > NextCopyFrom() assumes that the caller has prior knowledge of the
> > source data's column count, making it inadequate for cases where
> > extensions like file_text_array_fdw need to construct an array of
> > source data with an unknown number of columns. In such situations,
> > NextCopyFromRawFields() proves to be more practical. Given these
> > considerations, I'm now leaning towards implementing the proposed
> > change. Thoughts?
>
> You suggest that we re-export NextCopyFromRawFields() (as a
> wrapper of static inline version) for backward
> compatibility, right? It makes sense. We should keep
> backward compatibility because there is a use-case of
> NextCopyFromRawFields().

Yes, I've submitted the patch to re-export that function[1]. Could you
review it?

Regards,

[1] https://www.postgresql.org/message-id/CAD21AoBA414Q76LthY65NJfWbjOxXn1bdFFsD_NBhT2wPUS1SQ%40mail.gmail.com

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  235. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2025-02-28T22:17:37Z

    Hi,

In <CAD21AoB=FBiUB-ER7dmyE-QBBytUxqmv-sgbeP0DKTvYKXsOEA@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 28 Feb 2025 14:00:18 -0800,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

> Yes, I've submitted the patch to re-export that function[1]. Could you
> review it?
> 
> [1] https://www.postgresql.org/message-id/CAD21AoBA414Q76LthY65NJfWbjOxXn1bdFFsD_NBhT2wPUS1SQ%40mail.gmail.com

Sure! Done!

https://www.postgresql.org/message-id/20250301.071641.1257013931056303227.kou%40clear-code.com


Thanks,
-- 
kou

  236. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2025-03-01T02:50:09Z

    Hi,

Our 0001/0002 patches were merged into master. I've rebased
on master. Can we discuss how to proceed rest patches?

The contents of them aren't changed but I'll show a summary
of them again:

0001-0003 are for COPY TO and 0004-0007 are for COPY FROM.

For COPY TO:

0001: Add support for adding custom COPY TO format. This
uses tablesample like handler approach. We've discussed
other approaches such as USING+CREATE XXX approach but it
seems that other approaches are overkill for this case.

See also: https://www.postgresql.org/message-id/flat/d838025aceeb19c9ff1db702fa55cabf%40postgrespro.ru#caca2799effc859f82f40ee8bec531d8

0002: Export CopyToStateData to implement custom COPY TO
format as extension.

0003: Export a function and add a private space to
CopyToStateData to implement custom COPY TO format as
extension.

We may want to squash 0002 and 0003 but splitting them will
be easy to review. Because 0002 just moves existing codes
(with some rename) and 0003 just adds some codes. If we
squash 0002 and 0003, moving and adding are mixed.

For COPY FROM:

0004: This is COPY FROM version of 0001.

0005: 0002 has COPY_ prefix -> COPY_DEST_ prefix change for
enum CopyDest. This is similar change for enum CopySource.

0006: This is COPY FROM version of 0003.

0007: This is for easy to implement "ON_ERROR stop" and
"LOG_VERBOSITY verbose" in extension.

We may want to squash 0005-0007 like for 0002-0003.


Thanks,
-- 
kou

  237. Re: Make COPY format extendable: Extract COPY TO format implementations

    Junwang Zhao <zhjwpku@gmail.com> — 2025-03-02T07:40:11Z

    On Sat, Mar 1, 2025 at 10:50 AM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> Our 0001/0002 patches were merged into master. I've rebased
> on master. Can we discuss how to proceed rest patches?
>
> The contents of them aren't changed but I'll show a summary
> of them again:
>
> 0001-0003 are for COPY TO and 0004-0007 are for COPY FROM.
>
> For COPY TO:
>
> 0001: Add support for adding custom COPY TO format. This
> uses tablesample like handler approach. We've discussed
> other approaches such as USING+CREATE XXX approach but it
> seems that other approaches are overkill for this case.
>
> See also: https://www.postgresql.org/message-id/flat/d838025aceeb19c9ff1db702fa55cabf%40postgrespro.ru#caca2799effc859f82f40ee8bec531d8
>
> 0002: Export CopyToStateData to implement custom COPY TO
> format as extension.
>
> 0003: Export a function and add a private space to
> CopyToStateData to implement custom COPY TO format as
> extension.
>
> We may want to squash 0002 and 0003 but splitting them will
> be easy to review. Because 0002 just moves existing codes
> (with some rename) and 0003 just adds some codes. If we
> squash 0002 and 0003, moving and adding are mixed.
>
> For COPY FROM:
>
> 0004: This is COPY FROM version of 0001.
>
> 0005: 0002 has COPY_ prefix -> COPY_DEST_ prefix change for
> enum CopyDest. This is similar change for enum CopySource.
>
> 0006: This is COPY FROM version of 0003.
>
> 0007: This is for easy to implement "ON_ERROR stop" and
> "LOG_VERBOSITY verbose" in extension.
>
> We may want to squash 0005-0007 like for 0002-0003.
>
>
> Thanks,
> --
> kou


While review another thread (Emitting JSON to file using COPY TO),
I found the recently committed patches on this thread pass the
CopyFormatOptions struct directly rather a pointer of the struct
as a function parameter of CopyToGetRoutine and CopyFromGetRoutine.

Then I took a quick look at the newly rebased patch set and
found Sutou has already fixed this issue.

I'm wondering if we should fix it as a separate commit as
it seems like an oversight of previous patches?

-- 
Regards
Junwang Zhao

  238. Re: Make COPY format extendable: Extract COPY TO format implementations

    Tom Lane <tgl@sss.pgh.pa.us> — 2025-03-02T16:27:20Z

    Junwang Zhao <zhjwpku@gmail.com> writes:
> While review another thread (Emitting JSON to file using COPY TO),
> I found the recently committed patches on this thread pass the
> CopyFormatOptions struct directly rather a pointer of the struct
> as a function parameter of CopyToGetRoutine and CopyFromGetRoutine.

Coverity is unhappy about that too:

/srv/coverity/git/pgsql-git/postgresql/src/backend/commands/copyto.c: 177 in CopyToGetRoutine()
171     	.CopyToOneRow = CopyToBinaryOneRow,
172     	.CopyToEnd = CopyToBinaryEnd,
173     };
174     
175     /* Return a COPY TO routine for the given options */
176     static const CopyToRoutine *
>>>     CID 1643911:  Performance inefficiencies  (PASS_BY_VALUE)
>>>     Passing parameter opts of type "CopyFormatOptions" (size 184 bytes) by value, which exceeds the low threshold of 128 bytes.
177     CopyToGetRoutine(CopyFormatOptions opts)
178     {
179     	if (opts.csv_mode)
180     		return &CopyToRoutineCSV;

(and likewise for CopyFromGetRoutine).  I realize that these
functions aren't called often enough for performance to be an
overriding concern, but it still seems like poor style.

> Then I took a quick look at the newly rebased patch set and
> found Sutou has already fixed this issue.

+1, except I'd suggest declaring the parameters as
"const CopyFormatOptions *opts".

			regards, tom lane

  239. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2025-03-03T00:19:12Z

    Hi,

In <3191030.1740932840@sss.pgh.pa.us>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Sun, 02 Mar 2025 11:27:20 -0500,
  Tom Lane <tgl@sss.pgh.pa.us> wrote:

>> While review another thread (Emitting JSON to file using COPY TO),
>> I found the recently committed patches on this thread pass the
>> CopyFormatOptions struct directly rather a pointer of the struct
>> as a function parameter of CopyToGetRoutine and CopyFromGetRoutine.
> 
> Coverity is unhappy about that too:
> 
> /srv/coverity/git/pgsql-git/postgresql/src/backend/commands/copyto.c: 177 in CopyToGetRoutine()
> 171     	.CopyToOneRow = CopyToBinaryOneRow,
> 172     	.CopyToEnd = CopyToBinaryEnd,
> 173     };
> 174     
> 175     /* Return a COPY TO routine for the given options */
> 176     static const CopyToRoutine *
>>>>     CID 1643911:  Performance inefficiencies  (PASS_BY_VALUE)
>>>>     Passing parameter opts of type "CopyFormatOptions" (size 184 bytes) by value, which exceeds the low threshold of 128 bytes.
> 177     CopyToGetRoutine(CopyFormatOptions opts)
> 178     {
> 179     	if (opts.csv_mode)
> 180     		return &CopyToRoutineCSV;
> 
> (and likewise for CopyFromGetRoutine).  I realize that these
> functions aren't called often enough for performance to be an
> overriding concern, but it still seems like poor style.
> 
>> Then I took a quick look at the newly rebased patch set and
>> found Sutou has already fixed this issue.
> 
> +1, except I'd suggest declaring the parameters as
> "const CopyFormatOptions *opts".

Thanks for pointing out this (and sorry for missing this in
our reviews...)!

How about the attached patch?

I'll rebase the v35 patch set after this is fixed.


Thanks,
-- 
kou

  240. Re: Make COPY format extendable: Extract COPY TO format implementations

    Junwang Zhao <zhjwpku@gmail.com> — 2025-03-03T01:53:45Z

    On Mon, Mar 3, 2025 at 8:19 AM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <3191030.1740932840@sss.pgh.pa.us>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Sun, 02 Mar 2025 11:27:20 -0500,
>   Tom Lane <tgl@sss.pgh.pa.us> wrote:
>
> >> While review another thread (Emitting JSON to file using COPY TO),
> >> I found the recently committed patches on this thread pass the
> >> CopyFormatOptions struct directly rather a pointer of the struct
> >> as a function parameter of CopyToGetRoutine and CopyFromGetRoutine.
> >
> > Coverity is unhappy about that too:
> >
> > /srv/coverity/git/pgsql-git/postgresql/src/backend/commands/copyto.c: 177 in CopyToGetRoutine()
> > 171           .CopyToOneRow = CopyToBinaryOneRow,
> > 172           .CopyToEnd = CopyToBinaryEnd,
> > 173     };
> > 174
> > 175     /* Return a COPY TO routine for the given options */
> > 176     static const CopyToRoutine *
> >>>>     CID 1643911:  Performance inefficiencies  (PASS_BY_VALUE)
> >>>>     Passing parameter opts of type "CopyFormatOptions" (size 184 bytes) by value, which exceeds the low threshold of 128 bytes.
> > 177     CopyToGetRoutine(CopyFormatOptions opts)
> > 178     {
> > 179           if (opts.csv_mode)
> > 180                   return &CopyToRoutineCSV;
> >
> > (and likewise for CopyFromGetRoutine).  I realize that these
> > functions aren't called often enough for performance to be an
> > overriding concern, but it still seems like poor style.
> >
> >> Then I took a quick look at the newly rebased patch set and
> >> found Sutou has already fixed this issue.
> >
> > +1, except I'd suggest declaring the parameters as
> > "const CopyFormatOptions *opts".
>
> Thanks for pointing out this (and sorry for missing this in
> our reviews...)!
>
> How about the attached patch?

Looking good, thanks

>
> I'll rebase the v35 patch set after this is fixed.
>
>
> Thanks,
> --
> kou



-- 
Regards
Junwang Zhao

  241. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2025-03-03T19:06:39Z

    On Sun, Mar 2, 2025 at 4:19 PM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <3191030.1740932840@sss.pgh.pa.us>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Sun, 02 Mar 2025 11:27:20 -0500,
>   Tom Lane <tgl@sss.pgh.pa.us> wrote:
>
> >> While review another thread (Emitting JSON to file using COPY TO),
> >> I found the recently committed patches on this thread pass the
> >> CopyFormatOptions struct directly rather a pointer of the struct
> >> as a function parameter of CopyToGetRoutine and CopyFromGetRoutine.
> >
> > Coverity is unhappy about that too:
> >
> > /srv/coverity/git/pgsql-git/postgresql/src/backend/commands/copyto.c: 177 in CopyToGetRoutine()
> > 171           .CopyToOneRow = CopyToBinaryOneRow,
> > 172           .CopyToEnd = CopyToBinaryEnd,
> > 173     };
> > 174
> > 175     /* Return a COPY TO routine for the given options */
> > 176     static const CopyToRoutine *
> >>>>     CID 1643911:  Performance inefficiencies  (PASS_BY_VALUE)
> >>>>     Passing parameter opts of type "CopyFormatOptions" (size 184 bytes) by value, which exceeds the low threshold of 128 bytes.
> > 177     CopyToGetRoutine(CopyFormatOptions opts)
> > 178     {
> > 179           if (opts.csv_mode)
> > 180                   return &CopyToRoutineCSV;
> >
> > (and likewise for CopyFromGetRoutine).  I realize that these
> > functions aren't called often enough for performance to be an
> > overriding concern, but it still seems like poor style.
> >
> >> Then I took a quick look at the newly rebased patch set and
> >> found Sutou has already fixed this issue.
> >
> > +1, except I'd suggest declaring the parameters as
> > "const CopyFormatOptions *opts".
>
> Thanks for pointing out this (and sorry for missing this in
> our reviews...)!
>
> How about the attached patch?
>
> I'll rebase the v35 patch set after this is fixed.

Thank you for reporting the issue and making the patch.

I agree with the fix and the patch looks good to me. I've updated the
commit message and am going to push, barring any objections.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  242. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2025-03-05T00:06:08Z

    Hi,

In <CAD21AoAwOP7p6LgmkPGqPuJ5KbJPPQsSZsFzwCDguwzr9F677Q@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Mon, 3 Mar 2025 11:06:39 -0800,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

> I agree with the fix and the patch looks good to me. I've updated the
> commit message and am going to push, barring any objections.

Thanks!

I've rebased the patch set. Here is a summary again:

> 0001-0003 are for COPY TO and 0004-0007 are for COPY FROM.
>
> For COPY TO:
>
> 0001: Add support for adding custom COPY TO format. This
> uses tablesample like handler approach. We've discussed
> other approaches such as USING+CREATE XXX approach but it
> seems that other approaches are overkill for this case.
>
> See also: https://www.postgresql.org/message-id/flat/d838025aceeb19c9ff1db702fa55cabf%40postgrespro.ru#caca2799effc859f82f40ee8bec531d8
>
> 0002: Export CopyToStateData to implement custom COPY TO
> format as extension.
>
> 0003: Export a function and add a private space to
> CopyToStateData to implement custom COPY TO format as
> extension.
>
> We may want to squash 0002 and 0003 but splitting them will
> be easy to review. Because 0002 just moves existing codes
> (with some rename) and 0003 just adds some codes. If we
> squash 0002 and 0003, moving and adding are mixed.
>
> For COPY FROM:
>
> 0004: This is COPY FROM version of 0001.
>
> 0005: 0002 has COPY_ prefix -> COPY_DEST_ prefix change for
> enum CopyDest. This is similar change for enum CopySource.
>
> 0006: This is COPY FROM version of 0003.
>
> 0007: This is for easy to implement "ON_ERROR stop" and
> "LOG_VERBOSITY verbose" in extension.
>
> We may want to squash 0005-0007 like for 0002-0003.



Thanks,
-- 
kou

  243. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2025-03-17T20:50:03Z

    On Tue, Mar 4, 2025 at 4:06 PM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <CAD21AoAwOP7p6LgmkPGqPuJ5KbJPPQsSZsFzwCDguwzr9F677Q@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Mon, 3 Mar 2025 11:06:39 -0800,
>   Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> > I agree with the fix and the patch looks good to me. I've updated the
> > commit message and am going to push, barring any objections.
>
> Thanks!
>
> I've rebased the patch set. Here is a summary again:

Thank you for updating the patches. Here are some review comments on
the 0001 patch:

+   if (strcmp(format, "text") == 0)
+   {
+       /* "csv_mode == false && binary == false" means "text" */
+       return;
+   }
+   else if (strcmp(format, "csv") == 0)
+   {
+       opts_out->csv_mode = true;
+       return;
+   }
+   else if (strcmp(format, "binary") == 0)
+   {
+       opts_out->binary = true;
+       return;
+   }
+
+   /* custom format */
+   if (!is_from)
+   {
+       funcargtypes[0] = INTERNALOID;
+       handlerOid = LookupFuncName(list_make1(makeString(format)), 1,
+                                   funcargtypes, true);
+   }
+   if (!OidIsValid(handlerOid))
+       ereport(ERROR,
+               (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+                errmsg("COPY format \"%s\" not recognized", format),
+                parser_errposition(pstate, defel->location)));

I think that built-in formats also need to have their handler
functions. This seems to be a conventional way for customizable
features such as tablesample and access methods, and we can simplify
this function.

---
I think we need to update the documentation to describe how users can
define the handler functions and what each callback function is
responsible for.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  244. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2025-03-19T02:56:17Z

    Hi,

In <CAD21AoDU=bYRDDY8MzCXAfg4h9XTeTBdM-wVJaO1t4UcseCpuA@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Mon, 17 Mar 2025 13:50:03 -0700,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

> I think that built-in formats also need to have their handler
> functions. This seems to be a conventional way for customizable
> features such as tablesample and access methods, and we can simplify
> this function.

OK. 0008 in the attached v37 patch set does it.

> I think we need to update the documentation to describe how users can
> define the handler functions and what each callback function is
> responsible for.

I agree with it but we haven't finalized public APIs yet. Can
we defer it after we finalize public APIs? (Proposed public
APIs exist in 0003, 0006 and 0007.)

And could someone help (take over if possible) writing a
document for this feature? I'm not good at writing a
document in English... 0009 in the attached v37 patch set
has a draft of it. It's based on existing documents in
doc/src/sgml/ and *.h.

0001-0007 aren't changed from v36 patch set.


Thanks,
-- 
kou

  245. Re: Make COPY format extendable: Extract COPY TO format implementations

    David G. Johnston <david.g.johnston@gmail.com> — 2025-03-20T00:49:49Z

    On Tue, Mar 18, 2025 at 7:56 PM Sutou Kouhei <kou@clear-code.com> wrote:

> And could someone help (take over if possible) writing a
> document for this feature? I'm not good at writing a
> document in English... 0009 in the attached v37 patch set
> has a draft of it. It's based on existing documents in
> doc/src/sgml/ and *.h.
>
>
I haven't touched the innards of the structs aside from changing
programlisting to synopsis.  And redoing the two section opening paragraphs
to better integrate with the content in the chapter opening.

The rest I kinda went to town on...

David J.

diff --git a/doc/src/sgml/copy-handler.sgml b/doc/src/sgml/copy-handler.sgml
index f602debae6..9d2897a104 100644
--- a/doc/src/sgml/copy-handler.sgml
+++ b/doc/src/sgml/copy-handler.sgml
@@ -10,56 +10,72 @@
  <para>
   <productname>PostgreSQL</productname> supports
   custom <link linkend="sql-copy"><literal>COPY</literal></link>
-  handlers. The <literal>COPY</literal> handlers can use different copy
format
-  instead of built-in <literal>text</literal>, <literal>csv</literal>
-  and <literal>binary</literal>.
+  handlers; adding additional <replaceable>format_name</replaceable>
options
+  to the <literal>FORMAT</literal> clause.
  </para>

  <para>
-  At the SQL level, a table sampling method is represented by a single SQL
-  function, typically implemented in C, having the signature
-<programlisting>
-format_name(internal) RETURNS copy_handler
-</programlisting>
-  The name of the function is the same name appearing in
-  the <literal>FORMAT</literal> option. The <type>internal</type> argument
is
-  a dummy that simply serves to prevent this function from being called
-  directly from an SQL command. The real argument is <literal>bool
-  is_from</literal>. If the handler is used by <literal>COPY
FROM</literal>,
-  it's <literal>true</literal>. If the handler is used by <literal>COPY
-  FROM</literal>, it's <literal>false</literal>.
+  At the SQL level, a copy handler method is represented by a single SQL
+  function (see <xref linkend="sql-createfunction"/>), typically
implemented in
+  C, having the signature
+<synopsis>
+<replaceable>format_name</replaceable>(internal) RETURNS
<literal>copy_handler</literal>
+</synopsis>
+  The function's name is then accepted as a valid
<replaceable>format_name</replaceable>.
+  The return pseudo-type <literal>copy_handler</literal> informs the
system that
+  this function needs to be registered as a copy handler.
+  The <type>internal</type> argument is a dummy that prevents
+  this function from being called directly from an SQL command.  As the
+  handler implementation must be server-lifetime immutable; this SQL
function's
+  volatility should be marked immutable.  The
<literal>link_symbol</literal>
+  for this function is the name of the implementation function, described
next.
  </para>

  <para>
-  The function must return <type>CopyFromRoutine *</type> when
-  the <literal>is_from</literal> argument is <literal>true</literal>.
-  The function must return <type>CopyToRoutine *</type> when
-  the <literal>is_from</literal> argument is <literal>false</literal>.
+  The implementation function signature expected for the function named
+  in the <literal>link_symbol</literal> is:
+<synopsis>
+Datum
+<replaceable>copy_format_handler</replaceable>(PG_FUNCTION_ARGS)
+</synopsis>
+  The convention for the name is to replace the word
+  <replaceable>format</replaceable> in the placeholder above with the
value given
+  to <replaceable>format_name</replaceable> in the SQL function.
+  The first argument is a <type>boolean</type> that indicates whether the
handler
+  must provide a pointer to its implementation for <literal>COPY
FROM</literal>
+  (a <type>CopyFromRoutine *</type>). If <literal>false</literal>, the
handler
+  must provide a pointer to its implementation of <literal>COPY
TO</literal>
+  (a <type>CopyToRoutine *</type>).  These structs are declared in
+  <filename>src/include/commands/copyapi.h</filename>.
  </para>

  <para>
-  The <type>CopyFromRoutine</type> and <type>CopyToRoutine</type> struct
types
-  are declared in <filename>src/include/commands/copyapi.h</filename>,
-  which see for additional details.
+  The structs hold pointers to implementation functions for
+  initializing, starting, processing rows, and ending a copy operation.
+  The specific structures vary a bit between <literal>COPY FROM</literal>
and
+  <literal>COPY TO</literal> so the next two sections describes each
+  in detail.
  </para>

  <sect1 id="copy-handler-from">
   <title>Copy From Handler</title>

   <para>
-   The <literal>COPY</literal> handler function for <literal>COPY
-   FROM</literal> returns a <type>CopyFromRoutine</type> struct containing
-   pointers to the functions described below. All functions are required.
+   The opening to this chapter describes how the executor will call the
+   main handler function with, in this case,
+   a <type>boolean</type> <literal>true</literal>, and expect to receive a
+   <type>CopyFromRoutine *</type> <type>Datum</type>.  This section
describes
+   the components of the <type>CopyFromRoutine</type> struct.
   </para>

   <para>
-<programlisting>
+<synopsis>
 void
 CopyFromInFunc(CopyFromState cstate,
                Oid atttypid,
                FmgrInfo *finfo,
                Oid *typioparam);
-</programlisting>
+</synopsis>

    This sets input function information for the
    given <literal>atttypid</literal> attribute. This function is called
once
@@ -110,11 +126,11 @@ CopyFromInFunc(CopyFromState cstate,
   </para>

   <para>
-<programlisting>
+<synopsis>
 void
 CopyFromStart(CopyFromState cstate,
               TupleDesc tupDesc);
-</programlisting>
+</synopsis>

    This starts a <literal>COPY FROM</literal>. This function is called
once at
    the beginning of <literal>COPY FROM</literal>.
@@ -144,13 +160,13 @@ CopyFromStart(CopyFromState cstate,
   </para>

   <para>
-<programlisting>
+<synopsis>
 bool
 CopyFromOneRow(CopyFromState cstate,
                ExprContext *econtext,
                Datum *values,
                bool *nulls);
-</programlisting>
+</synopsis>

    This reads one row from the source and fill <literal>values</literal>
    and <literal>nulls</literal>. If there is one or more tuples to be read,
@@ -202,10 +218,10 @@ CopyFromOneRow(CopyFromState cstate,
   </para>

   <para>
-<programlisting>
+<synopsis>
 void
 CopyFromEnd(CopyFromState cstate);
-</programlisting>
+</synopsis>

    This ends a <literal>COPY FROM</literal>. This function is called once
at
    the end of <literal>COPY FROM</literal>.
@@ -232,18 +248,20 @@ CopyFromEnd(CopyFromState cstate);
   <title>Copy To Handler</title>

   <para>
-   The <literal>COPY</literal> handler function for <literal>COPY
-   TO</literal> returns a <type>CopyToRoutine</type> struct containing
-   pointers to the functions described below. All functions are required.
+   The opening to this chapter describes how the executor will call the
+   main handler function with, in this case,
+   a <type>boolean</type> <literal>false</literal>, and expect to receive a
+   <type>CopyInRoutine *</type> <type>Datum</type>.  This section describes
+   the components of the <type>CopyInRoutine</type> struct.
   </para>

   <para>
-<programlisting>
+<synopsis>
 void
 CopyToOutFunc(CopyToState cstate,
               Oid atttypid,
               FmgrInfo *finfo);
-</programlisting>
+</synopsis>

    This sets output function information for the
    given <literal>atttypid</literal> attribute. This function is called
once
@@ -284,11 +302,11 @@ CopyToOutFunc(CopyToState cstate,
   </para>

   <para>
-<programlisting>
+<synopsis>
 void
 CopyToStart(CopyToState cstate,
             TupleDesc tupDesc);
-</programlisting>
+</synopsis>

    This starts a <literal>COPY TO</literal>. This function is called once
at
    the beginning of <literal>COPY TO</literal>.
@@ -316,11 +334,11 @@ CopyToStart(CopyToState cstate,
   </para>

   <para>
-<programlisting>
+<synopsis>
 bool
 CopyToOneRow(CopyToState cstate,
              TupleTableSlot *slot);
-</programlisting>
+</synopsis>

    This writes one row stored in <literal>slot</literal> to the
destination.

@@ -347,10 +365,10 @@ CopyToOneRow(CopyToState cstate,
   </para>

   <para>
-<programlisting>
+<synopsis>
 void
 CopyToEnd(CopyToState cstate);
-</programlisting>
+</synopsis>

    This ends a <literal>COPY TO</literal>. This function is called once at
    the end of <literal>COPY TO</literal>.

  246. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2025-03-20T01:24:55Z

    Hi,

In <CAKFQuwaMAFMHqxDXR=SxA0mDjdmntrwxZd2w=nSruLNFH-OzLw@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Wed, 19 Mar 2025 17:49:49 -0700,
  "David G. Johnston" <david.g.johnston@gmail.com> wrote:

>> And could someone help (take over if possible) writing a
>> document for this feature? I'm not good at writing a
>> document in English... 0009 in the attached v37 patch set
>> has a draft of it. It's based on existing documents in
>> doc/src/sgml/ and *.h.
>>
>>
> I haven't touched the innards of the structs aside from changing
> programlisting to synopsis.  And redoing the two section opening paragraphs
> to better integrate with the content in the chapter opening.
> 
> The rest I kinda went to town on...

Thanks!!! It's very helpful!!!

I've applied your patch. 0009 is only changed.

Thanks,
-- 
kou

  247. Re: Make COPY format extendable: Extract COPY TO format implementations

    David G. Johnston <david.g.johnston@gmail.com> — 2025-03-22T00:31:54Z

    On Tue, Mar 18, 2025 at 7:56 PM Sutou Kouhei <kou@clear-code.com> wrote:

> Hi,
>
> In <CAD21AoDU=bYRDDY8MzCXAfg4h9XTeTBdM-wVJaO1t4UcseCpuA@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format
> implementations" on Mon, 17 Mar 2025 13:50:03 -0700,
>   Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> > I think that built-in formats also need to have their handler
> > functions. This seems to be a conventional way for customizable
> > features such as tablesample and access methods, and we can simplify
> > this function.
>
> OK. 0008 in the attached v37 patch set does it.
>
>
tl/dr;

We need to exclude from our SQL function search any function that doesn't
declare copy_handler as its return type.
("function text must return type copy_handler" is not an acceptable error
message)

We need to accept identifiers in FORMAT and parse the optional catalog,
schema, and object name portions.
(Restrict our function search to the named schema if provided.)

Detail:

Fun thing...(not sure how much of this is covered above: I do see, but
didn't scour, the security discussion):

-- create some poison
create function public.text(internal) returns boolean language c as
'/home/davidj/gotya/gotya', 'gotit';
CREATE FUNCTION

-- inject it
postgres=# set search_path to public,pg_catalog;
SET

-- watch it die
postgres=# copy (select 1) to stdout (format text);
ERROR:  function text must return type copy_handler
LINE 1: copy (select 1) to stdout (format text);

I'm especially concerned about extensions here.

We shouldn't be locating any SQL function that doesn't have a copy_handler
return type.  Unfortunately, LookupFuncName seems incapable of doing what
we want here.  I suggest we create a new lookup routine where we can
specify the return argument type as a required element.  That would cleanly
mitigate the denial-of-service attack/accident vector demonstrated above
(the text returning function should have zero impact on how this feature
behaves).  If someone does create a handler SQL function without using
copy_handler return type we'd end up showing "COPY format 'david' not
recognized" - a developer should be able to figure out they didn't put a
correct return type on their handler function and that is why the system
did not register it.

A second concern is simply people wanting to name things the same; or, why
namespaces were invented.

Can we just accept a proper identifier after FORMAT so we can use
schema-qualified names?

(FORMAT "davescopyformat"."david")

We can special case the internal schema-less names and internally force
pg_catalog to avoid them being shadowed.

David J.

  248. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2025-03-22T05:07:50Z

    On Fri, Mar 21, 2025 at 5:32 PM David G. Johnston
<david.g.johnston@gmail.com> wrote:
>
> On Tue, Mar 18, 2025 at 7:56 PM Sutou Kouhei <kou@clear-code.com> wrote:
>>
>> Hi,
>>
>> In <CAD21AoDU=bYRDDY8MzCXAfg4h9XTeTBdM-wVJaO1t4UcseCpuA@mail.gmail.com>
>>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Mon, 17 Mar 2025 13:50:03 -0700,
>>   Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>>
>> > I think that built-in formats also need to have their handler
>> > functions. This seems to be a conventional way for customizable
>> > features such as tablesample and access methods, and we can simplify
>> > this function.
>>
>> OK. 0008 in the attached v37 patch set does it.
>>
>
> tl/dr;
>
> We need to exclude from our SQL function search any function that doesn't declare copy_handler as its return type.
> ("function text must return type copy_handler" is not an acceptable error message)
>
> We need to accept identifiers in FORMAT and parse the optional catalog, schema, and object name portions.
> (Restrict our function search to the named schema if provided.)
>
> Detail:
>
> Fun thing...(not sure how much of this is covered above: I do see, but didn't scour, the security discussion):
>
> -- create some poison
> create function public.text(internal) returns boolean language c as '/home/davidj/gotya/gotya', 'gotit';
> CREATE FUNCTION
>
> -- inject it
> postgres=# set search_path to public,pg_catalog;
> SET
>
> -- watch it die
> postgres=# copy (select 1) to stdout (format text);
> ERROR:  function text must return type copy_handler
> LINE 1: copy (select 1) to stdout (format text);
>
> I'm especially concerned about extensions here.
>
> We shouldn't be locating any SQL function that doesn't have a copy_handler return type.  Unfortunately, LookupFuncName seems incapable of doing what we want here.  I suggest we create a new lookup routine where we can specify the return argument type as a required element.  That would cleanly mitigate the denial-of-service attack/accident vector demonstrated above (the text returning function should have zero impact on how this feature behaves).  If someone does create a handler SQL function without using copy_handler return type we'd end up showing "COPY format 'david' not recognized" - a developer should be able to figure out they didn't put a correct return type on their handler function and that is why the system did not register it.

Just to be clear, the patch checks the function's return type before calling it:

       funcargtypes[0] = INTERNALOID;
       handlerOid = LookupFuncName(list_make1(makeString(format)), 1,

funcargtypes, true);
       if (!OidIsValid(handlerOid))
               ereport(ERROR,
                               (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
                                errmsg("COPY format \"%s\" not
recognized", format),
                                parser_errposition(pstate, defel->location)));

       /* check that handler has correct return type */
       if (get_func_rettype(handlerOid) != COPY_HANDLEROID)
               ereport(ERROR,
                               (errcode(ERRCODE_WRONG_OBJECT_TYPE),
                                errmsg("function %s must return type %s",
                                               format, "copy_handler"),
                                parser_errposition(pstate, defel->location)));

So would changing the error message to like "COPY format 'text' not
recognized" untangle your concern?

FYI the same is true for TABLESAMPLE; it invokes a function with the
specified method name and checks the returned Node type:

=# select * from pg_class tablesample text (0);
ERROR:  function text must return type tsm_handler

A difference between TABLESAMPLE and COPY format is that the former
accepts a qualified name but the latter doesn't:

=# create extension tsm_system_rows ;
=# create schema s1;
=# create function s1.system_rows(internal) returns void language c as
'tsm_system_rows.so', 'tsm_system_rows_handler';
=# \df *.system_rows
                          List of functions
 Schema |    Name     | Result data type | Argument data types | Type
--------+-------------+------------------+---------------------+------
 public | system_rows | tsm_handler      | internal            | func
 s1     | system_rows | void             | internal            | func
(2 rows)
postgres(1:1194923)=# select count(*) from pg_class tablesample system_rows(0);
 count
-------
     0
(1 row)

postgres(1:1194923)=# select count(*) from pg_class tablesample
s1.system_rows(0);
ERROR:  function s1.system_rows must return type tsm_handler

> A second concern is simply people wanting to name things the same; or, why namespaces were invented.

Yeah, I think that the custom COPY format should support qualified
names at least.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  249. Re: Make COPY format extendable: Extract COPY TO format implementations

    David G. Johnston <david.g.johnston@gmail.com> — 2025-03-22T05:23:56Z

    On Friday, March 21, 2025, Masahiko Sawada <sawada.mshk@gmail.com> wrote:

> On Fri, Mar 21, 2025 at 5:32 PM David G. Johnston
> <david.g.johnston@gmail.com> wrote:
> >
> > On Tue, Mar 18, 2025 at 7:56 PM Sutou Kouhei <kou@clear-code.com> wrote:
> >>
> >> Hi,
> >>
> >> In <CAD21AoDU=bYRDDY8MzCXAfg4h9XTeTBdM-wVJaO1t4UcseCpuA@mail.gmail.com>
> >>   "Re: Make COPY format extendable: Extract COPY TO format
> implementations" on Mon, 17 Mar 2025 13:50:03 -0700,
> >>   Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >>
> >> > I think that built-in formats also need to have their handler
> >> > functions. This seems to be a conventional way for customizable
> >> > features such as tablesample and access methods, and we can simplify
> >> > this function.
> >>
> >> OK. 0008 in the attached v37 patch set does it.
> >>
> >
> > tl/dr;
> >
> > We need to exclude from our SQL function search any function that
> doesn't declare copy_handler as its return type.
> > ("function text must return type copy_handler" is not an acceptable
> error message)
> >
> > We need to accept identifiers in FORMAT and parse the optional catalog,
> schema, and object name portions.
> > (Restrict our function search to the named schema if provided.)
> >
> > Detail:
> >
> > Fun thing...(not sure how much of this is covered above: I do see, but
> didn't scour, the security discussion):
> >
> > -- create some poison
> > create function public.text(internal) returns boolean language c as
> '/home/davidj/gotya/gotya', 'gotit';
> > CREATE FUNCTION
> >
> > -- inject it
> > postgres=# set search_path to public,pg_catalog;
> > SET
> >
> > -- watch it die
> > postgres=# copy (select 1) to stdout (format text);
> > ERROR:  function text must return type copy_handler
> > LINE 1: copy (select 1) to stdout (format text);
> >
> > I'm especially concerned about extensions here.
> >
> > We shouldn't be locating any SQL function that doesn't have a
> copy_handler return type.  Unfortunately, LookupFuncName seems incapable of
> doing what we want here.  I suggest we create a new lookup routine where we
> can specify the return argument type as a required element.  That would
> cleanly mitigate the denial-of-service attack/accident vector demonstrated
> above (the text returning function should have zero impact on how this
> feature behaves).  If someone does create a handler SQL function without
> using copy_handler return type we'd end up showing "COPY format 'david' not
> recognized" - a developer should be able to figure out they didn't put a
> correct return type on their handler function and that is why the system
> did not register it.
>
> Just to be clear, the patch checks the function's return type before
> calling it:
>
>        funcargtypes[0] = INTERNALOID;
>        handlerOid = LookupFuncName(list_make1(makeString(format)), 1,
>
> funcargtypes, true);
>        if (!OidIsValid(handlerOid))
>                ereport(ERROR,
>                                (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
>                                 errmsg("COPY format \"%s\" not
> recognized", format),
>                                 parser_errposition(pstate,
> defel->location)));
>
>        /* check that handler has correct return type */
>        if (get_func_rettype(handlerOid) != COPY_HANDLEROID)
>                ereport(ERROR,
>                                (errcode(ERRCODE_WRONG_OBJECT_TYPE),
>                                 errmsg("function %s must return type %s",
>                                                format, "copy_handler"),
>                                 parser_errposition(pstate,
> defel->location)));
>
> So would changing the error message to like "COPY format 'text' not
> recognized" untangle your concern?


In my example above copy should not fail at all.  The text function created
in public that returns Boolean would never be seen and the real one in
pg_catalog would then be found and behave as expected.


>
> FYI the same is true for TABLESAMPLE; it invokes a function with the
> specified method name and checks the returned Node type:
>
> =# select * from pg_class tablesample text (0);
> ERROR:  function text must return type tsm_handler


Then this would benefit from the new function I suggest creating since it
apparently has the same, IMO, bug.

David J.

  250. Re: Make COPY format extendable: Extract COPY TO format implementations

    David G. Johnston <david.g.johnston@gmail.com> — 2025-03-22T17:11:10Z

    On Fri, Mar 21, 2025 at 10:23 PM David G. Johnston <
david.g.johnston@gmail.com> wrote:

> Then this would benefit from the new function I suggest creating since it
> apparently has the same, IMO, bug.
>
>
Concretely like I posted here:
https://www.postgresql.org/message-id/CAKFQuwYBTcK+uW-BYFChHP8HYj0R5+UpytGmdqEvP9PHCSZ+-g@mail.gmail.com

David J.

  251. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2025-03-23T09:01:59Z

    On Wed, Mar 19, 2025 at 6:25 PM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <CAKFQuwaMAFMHqxDXR=SxA0mDjdmntrwxZd2w=nSruLNFH-OzLw@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Wed, 19 Mar 2025 17:49:49 -0700,
>   "David G. Johnston" <david.g.johnston@gmail.com> wrote:
>
> >> And could someone help (take over if possible) writing a
> >> document for this feature? I'm not good at writing a
> >> document in English... 0009 in the attached v37 patch set
> >> has a draft of it. It's based on existing documents in
> >> doc/src/sgml/ and *.h.
> >>
> >>
> > I haven't touched the innards of the structs aside from changing
> > programlisting to synopsis.  And redoing the two section opening paragraphs
> > to better integrate with the content in the chapter opening.
> >
> > The rest I kinda went to town on...
>
> Thanks!!! It's very helpful!!!
>
> I've applied your patch. 0009 is only changed.

Thank you for updating the patches. I've reviewed the main part of
supporting the custom COPY format. Here are some random comments:

---
+/*
+ * Process the "format" option.
+ *
+ * This function checks whether the option value is a built-in format such as
+ * "text" and "csv" or not. If the option value isn't a built-in format, this
+ * function finds a COPY format handler that returns a CopyToRoutine (for
+ * is_from == false) or CopyFromRountine (for is_from == true). If no COPY
+ * format handler is found, this function reports an error.
+ */

I think this comment needs to be updated as the part "If the option
value isn't ..." is no longer true.

I think we don't necessarily need to create a separate function
ProcessCopyOptionFormat for processing the format option.

We need more regression tests for handling the given format name. For example,

- more various input patterns.
- a function with the specified format name exists but it returns an
unexpected Node.
- looking for a handler function in a different namespace.
etc.

---
I think that we should accept qualified names too as the format name
like tablesample does. That way, different extensions implementing the
same format can be used.

---
+        if (routine == NULL || !IsA(routine, CopyFromRoutine))
+                ereport(
+                                ERROR,
+                                (errcode(
+
ERRCODE_INVALID_PARAMETER_VALUE),
+                                 errmsg("COPY handler function "
+                                                "%u did not return "
+                                                "CopyFromRoutine struct",
+                                                opts->handler)));

It's not conventional to put a new line between 'ereport(' and 'ERROR'
(similarly between 'errcode(' and 'ERRCODE_...'. Also, we don't need
to split the error message into multiple lines as it's not long.

---
+        if (routine == NULL || !IsA(routine, CopyToRoutine))
+                ereport(
+                                ERROR,
+                                (errcode(
+
ERRCODE_INVALID_PARAMETER_VALUE),
+                                 errmsg("COPY handler function "
+                                                "%u did not return "
+                                                "CopyToRoutine struct",
+                                                opts->handler)));

Same as the above comment.

---
+  descr => 'pseudo-type for the result of a copy to/from method function',

s/method function/format function/

---
+        Oid                    handler;                /* handler
function for custom format routine */

'handler' is used also for built-in formats.

---
+static void
+CopyFromInFunc(CopyFromState cstate, Oid atttypid,
+                           FmgrInfo *finfo, Oid *typioparam)
+{
+        ereport(NOTICE, (errmsg("CopyFromInFunc: atttypid=%d", atttypid)));
+}

OIDs could be changed across major versions even for built-in types. I
think it's better to avoid using it for tests.

---
+static void
+CopyToOneRow(CopyToState cstate, TupleTableSlot *slot)
+{
+        ereport(NOTICE, (errmsg("CopyToOneRow: tts_nvalid=%u",
slot->tts_nvalid)));
+}

Similar to the above comment, the field name 'tts_nvalid' might also
be changed in the future, let's use another name.

---
+static const CopyFromRoutine CopyFromRoutineTestCopyFormat = {
+        .type = T_CopyFromRoutine,
+        .CopyFromInFunc = CopyFromInFunc,
+        .CopyFromStart = CopyFromStart,
+        .CopyFromOneRow = CopyFromOneRow,
+        .CopyFromEnd = CopyFromEnd,
+};

I'd suggest not using the same function names as the fields.

---
+/*
+ * Export CopySendEndOfRow() for extensions. We want to keep
+ * CopySendEndOfRow() as a static function for
+ * optimization. CopySendEndOfRow() calls in this file may be optimized by a
+ * compiler.
+ */
+void
+CopyToStateFlush(CopyToState cstate)
+{
+        CopySendEndOfRow(cstate);
+}

Is there any reason to use a different name for public functions?

---
+/*
+ * Export CopyGetData() for extensions. We want to keep CopyGetData() as a
+ * static function for optimization. CopyGetData() calls in this file may be
+ * optimized by a compiler.
+ */
+int
+CopyFromStateGetData(CopyFromState cstate, void *dest, int minread,
int maxread)
+{
+        return CopyGetData(cstate, dest, minread, maxread);
+}
+

The same as the above comment.

---
+        /* For custom format implementation */
+        void      *opaque;                     /* private space */

How about renaming 'private'?

---
I've not reviewed the documentation patch yet but I think the patch
seems to miss the updates to the description of the FORMAT option in
the COPY command section.

---
I think we can reorganize the patch set as follows:

1. Create copyto_internal.h and change COPY_XXX to COPY_SOURCE_XXX and
COPY_DEST_XXX accordingly.
2. Support custom format for both COPY TO and COPY FROM.
3. Expose necessary helper functions such as CopySendEndOfRow().
4. Add CopyFromSkipErrorRow().
5. Documentation.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  252. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2025-03-25T00:45:10Z

    On Wed, Mar 19, 2025 at 6:25 PM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <CAKFQuwaMAFMHqxDXR=SxA0mDjdmntrwxZd2w=nSruLNFH-OzLw@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Wed, 19 Mar 2025 17:49:49 -0700,
>   "David G. Johnston" <david.g.johnston@gmail.com> wrote:
>
> >> And could someone help (take over if possible) writing a
> >> document for this feature? I'm not good at writing a
> >> document in English... 0009 in the attached v37 patch set
> >> has a draft of it. It's based on existing documents in
> >> doc/src/sgml/ and *.h.
> >>
> >>
> > I haven't touched the innards of the structs aside from changing
> > programlisting to synopsis.  And redoing the two section opening paragraphs
> > to better integrate with the content in the chapter opening.
> >
> > The rest I kinda went to town on...
>
> Thanks!!! It's very helpful!!!
>
> I've applied your patch. 0009 is only changed.

FYI I've implemented an extension to add JSON Lines format as a custom
COPY format[1] to check the usability of the COPY format APIs. I think
that the exposed APIs are fairly simple and minimum. I didn't find the
deficiency and excess of exposed APIs for helping extensions but I
find that it would be better to describe what the one-row callback
should do to utilize the abstracted destination. For example, in order
to use CopyToStateFlush() to write out to the destination, extensions
should write the data to cstate->fe_msgbuf. We expose
CopyToStateFlush() but not for any functions to write data there such
as CopySendString(). It was a bit inconvenient to me but I managed to
write the data directly there by #include'ing copyto_internal.h.

Regards,

[1] https://github.com/MasahikoSawada/pg_copy_jsonlines

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  253. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2025-03-27T03:28:40Z

    Hi,

In <CAD21AoAfWrjpTDJ0garVUoXY0WC3Ud4Cu51q+ccWiotm1uo_2A@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Sun, 23 Mar 2025 02:01:59 -0700,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

> ---
> +/*
> + * Process the "format" option.
> + *
> + * This function checks whether the option value is a built-in format such as
> + * "text" and "csv" or not. If the option value isn't a built-in format, this
> + * function finds a COPY format handler that returns a CopyToRoutine (for
> + * is_from == false) or CopyFromRountine (for is_from == true). If no COPY
> + * format handler is found, this function reports an error.
> + */
> 
> I think this comment needs to be updated as the part "If the option
> value isn't ..." is no longer true.
> 
> I think we don't necessarily need to create a separate function
> ProcessCopyOptionFormat for processing the format option.

Hmm. I think that this separated function will increase
readability by reducing indentation. But I've removed the
separation as you suggested. So the comment is also removed
entirely.

0002 includes this.

> We need more regression tests for handling the given format name. For example,
> 
> - more various input patterns.
> - a function with the specified format name exists but it returns an
> unexpected Node.
> - looking for a handler function in a different namespace.
> etc.

I've added the following tests:

* Wrong input type handler without namespace
* Wrong input type handler with namespace
* Wrong return type handler without namespace
* Wrong return type handler with namespace
* Wrong return value (Copy*Routine isn't returned) handler without namespace
* Wrong return value (Copy*Routine isn't returned) handler with namespace
* Nonexistent handler
* Invalid qualified name
* Valid handler without namespace and without search_path
* Valid handler without namespace and with search_path
* Valid handler with namespace

0002 also includes this.

> I think that we should accept qualified names too as the format name
> like tablesample does. That way, different extensions implementing the
> same format can be used.

Implemented. It's implemented after parsing SQL. Is it OK?
(It seems that tablesample does it in parsing SQL.)

Because "WITH (FORMAT XXX)" is processed as a generic option
in gram.y. All generic options are processed as strings. So
I keep this.

Syntax is "COPY ... WITH (FORMAT 'NAMESPACE.HANDLER_NAME')"
not "COPY ... WITH (FORMAT 'NAMESPACE'.'HANDLER_NAME')"
because of this choice.


0002 also includes this.

> ---
> +        if (routine == NULL || !IsA(routine, CopyFromRoutine))
> +                ereport(
> +                                ERROR,
> +                                (errcode(
> +
> ERRCODE_INVALID_PARAMETER_VALUE),
> +                                 errmsg("COPY handler function "
> +                                                "%u did not return "
> +                                                "CopyFromRoutine struct",
> +                                                opts->handler)));
> 
> It's not conventional to put a new line between 'ereport(' and 'ERROR'
> (similarly between 'errcode(' and 'ERRCODE_...'. Also, we don't need
> to split the error message into multiple lines as it's not long.

Oh, sorry. I can't remember why I used this... I think I
trusted pgindent...

> ---
> +  descr => 'pseudo-type for the result of a copy to/from method function',
> 
> s/method function/format function/

Good catch. I used "handler function" not "format function"
because we use "handler" in other places.

> ---
> +        Oid                    handler;                /* handler
> function for custom format routine */
> 
> 'handler' is used also for built-in formats.

Updated in 0004.

> ---
> +static void
> +CopyFromInFunc(CopyFromState cstate, Oid atttypid,
> +                           FmgrInfo *finfo, Oid *typioparam)
> +{
> +        ereport(NOTICE, (errmsg("CopyFromInFunc: atttypid=%d", atttypid)));
> +}
> 
> OIDs could be changed across major versions even for built-in types. I
> think it's better to avoid using it for tests.

Oh, I didn't know it. I've changed to use type name instead
of OID. It'll be more stable than OID.

> ---
> +static void
> +CopyToOneRow(CopyToState cstate, TupleTableSlot *slot)
> +{
> +        ereport(NOTICE, (errmsg("CopyToOneRow: tts_nvalid=%u",
> slot->tts_nvalid)));
> +}
> 
> Similar to the above comment, the field name 'tts_nvalid' might also
> be changed in the future, let's use another name.

Hmm. If the field name is changed, we need to change this
code. So changing tests too isn't strange. Anyway, I used
more generic text.

> ---
> +static const CopyFromRoutine CopyFromRoutineTestCopyFormat = {
> +        .type = T_CopyFromRoutine,
> +        .CopyFromInFunc = CopyFromInFunc,
> +        .CopyFromStart = CopyFromStart,
> +        .CopyFromOneRow = CopyFromOneRow,
> +        .CopyFromEnd = CopyFromEnd,
> +};
> 
> I'd suggest not using the same function names as the fields.

OK. I've added "Test" prefix.

> ---
> +/*
> + * Export CopySendEndOfRow() for extensions. We want to keep
> + * CopySendEndOfRow() as a static function for
> + * optimization. CopySendEndOfRow() calls in this file may be optimized by a
> + * compiler.
> + */
> +void
> +CopyToStateFlush(CopyToState cstate)
> +{
> +        CopySendEndOfRow(cstate);
> +}
> 
> Is there any reason to use a different name for public functions?

In this patch set, I use "CopyFrom"/"CopyTo" prefixes for
public APIs for custom COPY FORMAT handler extensions. It
will help understanding related APIs. Is it strange in
PostgreSQL?

> ---
> +        /* For custom format implementation */
> +        void      *opaque;                     /* private space */
> 
> How about renaming 'private'?

We should not use "private" because it's a keyword in
C++. If we use "private" here, we can't include this file
from C++ code.

> ---
> I've not reviewed the documentation patch yet but I think the patch
> seems to miss the updates to the description of the FORMAT option in
> the COPY command section.

I defer this for now. We can revisit the last documentation
patch after we finalize our API. (Or could someone help us?)

> I think we can reorganize the patch set as follows:
> 
> 1. Create copyto_internal.h and change COPY_XXX to COPY_SOURCE_XXX and
> COPY_DEST_XXX accordingly.
> 2. Support custom format for both COPY TO and COPY FROM.
> 3. Expose necessary helper functions such as CopySendEndOfRow().
> 4. Add CopyFromSkipErrorRow().
> 5. Documentation.

The attached v39 patch set uses the followings:

0001: Create copyto_internal.h and change COPY_XXX to
      COPY_SOURCE_XXX and COPY_DEST_XXX accordingly.
      (Same as 1. in your suggestion)
0002: Support custom format for both COPY TO and COPY FROM.
      (Same as 2. in your suggestion)
0003: Expose necessary helper functions such as CopySendEndOfRow()
      and add CopyFromSkipErrorRow().
      (3. + 4. in your suggestion)
0004: Define handler functions for built-in formats.
      (Not included in your suggestion)
0005: Documentation. (WIP)
      (Same as 5. in your suggestion)

We can merge 0001 quickly, right?


Thanks,
-- 
kou

  254. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2025-03-29T05:37:03Z

    On Wed, Mar 26, 2025 at 8:28 PM Sutou Kouhei <kou@clear-code.com> wrote:
>
> > We need more regression tests for handling the given format name. For example,
> >
> > - more various input patterns.
> > - a function with the specified format name exists but it returns an
> > unexpected Node.
> > - looking for a handler function in a different namespace.
> > etc.
>
> I've added the following tests:
>
> * Wrong input type handler without namespace
> * Wrong input type handler with namespace
> * Wrong return type handler without namespace
> * Wrong return type handler with namespace
> * Wrong return value (Copy*Routine isn't returned) handler without namespace
> * Wrong return value (Copy*Routine isn't returned) handler with namespace
> * Nonexistent handler
> * Invalid qualified name
> * Valid handler without namespace and without search_path
> * Valid handler without namespace and with search_path
> * Valid handler with namespace

Probably we can merge these newly added four files into one .sql file?

Also we need to add some comments to describe what these queries test.
For example, it's not clear to me at a glance what queries in
no-schema.sql are going to test as there is no comment there.

>
> 0002 also includes this.
>
> > I think that we should accept qualified names too as the format name
> > like tablesample does. That way, different extensions implementing the
> > same format can be used.
>
> Implemented. It's implemented after parsing SQL. Is it OK?
> (It seems that tablesample does it in parsing SQL.)

I think it's okay.

One problem in the following chunk I can see is:

+           qualified_format = stringToQualifiedNameList(format, NULL);
+           DeconstructQualifiedName(qualified_format, &schema, &fmt);
+           if (!schema || strcmp(schema, "pg_catalog") == 0)
+           {
+               if (strcmp(fmt, "csv") == 0)
+                   opts_out->csv_mode = true;
+               else if (strcmp(fmt, "binary") == 0)
+                   opts_out->binary = true;
+           }

Non-qualified names depend on the search_path value so it's not
necessarily a built-in format. If the user specifies 'csv' with
seach_patch = 'myschema, pg_catalog', the COPY command unnecessarily
sets csv_mode true. I think we can instead check if the retrieved
handler function's OID matches the built-in formats' ones. Also, it's
weired to me that cstate has csv_mode and binary fields even though
the format should have already been known by the callback functions.

Regarding the documentation for the existing options, it says "...
only when not using XXX format." some places, where XXX can be
replaced with binary or CSV. Once we support custom formats, 'non-CSV
mode' would actually include custom formats as well, so we need to
update the description too.

>
> > ---
> > +static void
> > +CopyToOneRow(CopyToState cstate, TupleTableSlot *slot)
> > +{
> > +        ereport(NOTICE, (errmsg("CopyToOneRow: tts_nvalid=%u",
> > slot->tts_nvalid)));
> > +}
> >
> > Similar to the above comment, the field name 'tts_nvalid' might also
> > be changed in the future, let's use another name.
>
> Hmm. If the field name is changed, we need to change this
> code.

Yes, but if we use independe name in the NOTICE message we would not
need to update the expected files.

>
> > ---
> > +/*
> > + * Export CopySendEndOfRow() for extensions. We want to keep
> > + * CopySendEndOfRow() as a static function for
> > + * optimization. CopySendEndOfRow() calls in this file may be optimized by a
> > + * compiler.
> > + */
> > +void
> > +CopyToStateFlush(CopyToState cstate)
> > +{
> > +        CopySendEndOfRow(cstate);
> > +}
> >
> > Is there any reason to use a different name for public functions?
>
> In this patch set, I use "CopyFrom"/"CopyTo" prefixes for
> public APIs for custom COPY FORMAT handler extensions. It
> will help understanding related APIs. Is it strange in
> PostgreSQL?

I see your point. Probably we need to find a better name as the name
CopyToStateFlush doesn't sound well like this API should be called
only once at the end of a row (IOW user might try to call it multiple
times to 'flush' the state while processing a row). How about
CopyToEndOfRow()?

>
> > ---
> > +        /* For custom format implementation */
> > +        void      *opaque;                     /* private space */
> >
> > How about renaming 'private'?
>
> We should not use "private" because it's a keyword in
> C++. If we use "private" here, we can't include this file
> from C++ code.

Understood.

>
> > ---
> > I've not reviewed the documentation patch yet but I think the patch
> > seems to miss the updates to the description of the FORMAT option in
> > the COPY command section.
>
> I defer this for now. We can revisit the last documentation
> patch after we finalize our API. (Or could someone help us?)
>
> > I think we can reorganize the patch set as follows:
> >
> > 1. Create copyto_internal.h and change COPY_XXX to COPY_SOURCE_XXX and
> > COPY_DEST_XXX accordingly.
> > 2. Support custom format for both COPY TO and COPY FROM.
> > 3. Expose necessary helper functions such as CopySendEndOfRow().
> > 4. Add CopyFromSkipErrorRow().
> > 5. Documentation.
>
> The attached v39 patch set uses the followings:
>
> 0001: Create copyto_internal.h and change COPY_XXX to
>       COPY_SOURCE_XXX and COPY_DEST_XXX accordingly.
>       (Same as 1. in your suggestion)
> 0002: Support custom format for both COPY TO and COPY FROM.
>       (Same as 2. in your suggestion)
> 0003: Expose necessary helper functions such as CopySendEndOfRow()
>       and add CopyFromSkipErrorRow().
>       (3. + 4. in your suggestion)
> 0004: Define handler functions for built-in formats.
>       (Not included in your suggestion)
> 0005: Documentation. (WIP)
>       (Same as 5. in your suggestion)

Can we merge 0002 and 0004?

> We can merge 0001 quickly, right?

I don't think it makes sense to push only 0001 as it's a completely
preliminary patch for subsequent patches. It would be prudent to push
it once other patches are ready too.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  255. Re: Make COPY format extendable: Extract COPY TO format implementations

    David G. Johnston <david.g.johnston@gmail.com> — 2025-03-29T05:57:56Z

    On Friday, March 28, 2025, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
>
> One problem in the following chunk I can see is:
>
> +           qualified_format = stringToQualifiedNameList(format, NULL);
> +           DeconstructQualifiedName(qualified_format, &schema, &fmt);
> +           if (!schema || strcmp(schema, "pg_catalog") == 0)
> +           {
> +               if (strcmp(fmt, "csv") == 0)
> +                   opts_out->csv_mode = true;
> +               else if (strcmp(fmt, "binary") == 0)
> +                   opts_out->binary = true;
> +           }
>
> Non-qualified names depend on the search_path value so it's not
> necessarily a built-in format. If the user specifies 'csv' with
> seach_patch = 'myschema, pg_catalog', the COPY command unnecessarily
> sets csv_mode true. I think we can instead check if the retrieved
> handler function's OID matches the built-in formats' ones. Also, it's
> weired to me that cstate has csv_mode and binary fields even though
> the format should have already been known by the callback functions.
>

I considered it a feature that a schema-less reference to text, csv, or
binary always resolves to the core built-in handlers.  As does an
unspecified format default of text.

To use an extension that chooses to override that format name would require
schema qualification.

David J.

  256. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2025-03-29T08:57:53Z

    Hi,

In <CAD21AoBKMNsO+b6wahb6847xwFci1JCfV+JykoMziVgiFxB6cw@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 28 Mar 2025 22:37:03 -0700,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

>> I've added the following tests:
>>
>> * Wrong input type handler without namespace
>> * Wrong input type handler with namespace
>> * Wrong return type handler without namespace
>> * Wrong return type handler with namespace
>> * Wrong return value (Copy*Routine isn't returned) handler without namespace
>> * Wrong return value (Copy*Routine isn't returned) handler with namespace
>> * Nonexistent handler
>> * Invalid qualified name
>> * Valid handler without namespace and without search_path
>> * Valid handler without namespace and with search_path
>> * Valid handler with namespace
> 
> Probably we can merge these newly added four files into one .sql file?

I know that src/test/regress/sql/ uses this style (one .sql
file includes many test patterns in one large category). I
understand that the style is preferable in
src/test/regress/sql/ because src/test/regress/sql/ has
tests for many categories.

But do we need to follow the style in
src/test/modules/*/sql/ too? If we use the style in
src/test/modules/*/sql/, we need to have only one .sql in
src/test/modules/*/sql/ because src/test/modules/*/ are for
each category.

And the current .sql per sub-category style is easy to debug
(at least for me). For example, if we try qualified name
cases on debugger, we can use "\i sql/schema.sql" instead of
extracting target statements from .sql that includes many
unrelated statements. (Or we can use "\i sql/all.sql" and
many GDB "continue"s.)

BTW, it seems that src/test/modules/test_ddl_deparse/sql/
uses .sql per sub-category style. Should we use one .sql
file for sql/test/modules/test_copy_format/sql/? If it's
required for merging this patch set, I'll do it.

> Also we need to add some comments to describe what these queries test.
> For example, it's not clear to me at a glance what queries in
> no-schema.sql are going to test as there is no comment there.

Hmm. You refer no_schema.sql in 0002, right?

----
CREATE EXTENSION test_copy_format;
CREATE TABLE public.test (a smallint, b integer, c bigint);
INSERT INTO public.test VALUES (1, 2, 3), (12, 34, 56), (123, 456, 789);
COPY public.test FROM stdin WITH (FORMAT 'test_copy_format');
\.
COPY public.test TO stdout WITH (FORMAT 'test_copy_format');
DROP TABLE public.test;
DROP EXTENSION test_copy_format;
----

In general, COPY FORMAT tests focus on "COPY FROM WITH
(FORMAT ...)" and "COPY TO WITH (FORMAT ...)". And the file
name "no_schema" shows that it doesn't use qualified
name. Based on this, I feel that the above content is very
straightforward without any comment.

What should we add as comments? For example, do we need the
following comments?

----
-- This extension includes custom COPY handler: test_copy_format
CREATE EXTENSION test_copy_format;
-- Test data
CREATE TABLE public.test (a smallint, b integer, c bigint);
INSERT INTO public.test VALUES (1, 2, 3), (12, 34, 56), (123, 456, 789);
-- Use custom COPY handler, test_copy_format, without
-- schema for FROM.
COPY public.test FROM stdin WITH (FORMAT 'test_copy_format');
\.
-- Use custom COPY handler, test_copy_format, without
-- schema for TO.
COPY public.test TO stdout WITH (FORMAT 'test_copy_format');
-- Cleanup
DROP TABLE public.test;
DROP EXTENSION test_copy_format;
----

> One problem in the following chunk I can see is:
> 
> +           qualified_format = stringToQualifiedNameList(format, NULL);
> +           DeconstructQualifiedName(qualified_format, &schema, &fmt);
> +           if (!schema || strcmp(schema, "pg_catalog") == 0)
> +           {
> +               if (strcmp(fmt, "csv") == 0)
> +                   opts_out->csv_mode = true;
> +               else if (strcmp(fmt, "binary") == 0)
> +                   opts_out->binary = true;
> +           }
> 
> Non-qualified names depend on the search_path value so it's not
> necessarily a built-in format. If the user specifies 'csv' with
> seach_patch = 'myschema, pg_catalog', the COPY command unnecessarily
> sets csv_mode true.

I think that we should always use built-in COPY handlers for
(not-qualified) "text", "csv" and "binary" for
compatibility. If we allow custom COPY handlers for
(not-qualified) "text", "csv" and "binary", pg_dump or
existing dump may be broken. Because we must use the same
COPY handler when we dump (COPY TO) and we restore (COPY
FROM).

BTW, the current implementation always uses
pg_catalog.{text,csv,binary} for (not-qualified) "text",
"csv" and "binary" even when there are
myschema.{text,csv,binary}. See
src/test/modules/test_copy_format/sql/builtin.sql. But I
haven't looked into it why...

>                     I think we can instead check if the retrieved
> handler function's OID matches the built-in formats' ones.

I agree that the approach is clear than the current
implementation. I'll use it when I create the next patch
set.

>                                                            Also, it's
> weired to me that cstate has csv_mode and binary fields even though
> the format should have already been known by the callback functions.

You refer CopyFomratOptions::{csv_mode,binary} not
Copy{To,From}StateData, right? And you suggest that we
should replace all opts.csv_mode and opts.binary with
opts.handler == F_CSV and opts.handler == F_BINARY, right?

We can do it but I suggest that we do it as a refactoring
(or cleanup) in a separated patch for easy to review.

> Regarding the documentation for the existing options, it says "...
> only when not using XXX format." some places, where XXX can be
> replaced with binary or CSV. Once we support custom formats, 'non-CSV
> mode' would actually include custom formats as well, so we need to
> update the description too.

I agree with you.

>> > ---
>> > +/*
>> > + * Export CopySendEndOfRow() for extensions. We want to keep
>> > + * CopySendEndOfRow() as a static function for
>> > + * optimization. CopySendEndOfRow() calls in this file may be optimized by a
>> > + * compiler.
>> > + */
>> > +void
>> > +CopyToStateFlush(CopyToState cstate)
>> > +{
>> > +        CopySendEndOfRow(cstate);
>> > +}
>> >
>> > Is there any reason to use a different name for public functions?
>>
>> In this patch set, I use "CopyFrom"/"CopyTo" prefixes for
>> public APIs for custom COPY FORMAT handler extensions. It
>> will help understanding related APIs. Is it strange in
>> PostgreSQL?
> 
> I see your point. Probably we need to find a better name as the name
> CopyToStateFlush doesn't sound well like this API should be called
> only once at the end of a row (IOW user might try to call it multiple
> times to 'flush' the state while processing a row). How about
> CopyToEndOfRow()?

CopyToStateFlush() can be called multiple times in a row. It
can also be called only once with multiple rows. Because it
just flushes the current buffer.

Existing CopySendEndOfRow() is called at the end of a
row. (Buffer is flushed at the end of row.) So I think that
the "EndOfRow" was chosen.

Some custom COPY handlers may not be row based. For example,
Apache Arrow COPY handler doesn't flush buffer for each row.
So, we should provide "flush" API not "end of row" API for
extensibility.

>> The attached v39 patch set uses the followings:
>>
>> 0001: Create copyto_internal.h and change COPY_XXX to
>>       COPY_SOURCE_XXX and COPY_DEST_XXX accordingly.
>>       (Same as 1. in your suggestion)
>> 0002: Support custom format for both COPY TO and COPY FROM.
>>       (Same as 2. in your suggestion)
>> 0003: Expose necessary helper functions such as CopySendEndOfRow()
>>       and add CopyFromSkipErrorRow().
>>       (3. + 4. in your suggestion)
>> 0004: Define handler functions for built-in formats.
>>       (Not included in your suggestion)
>> 0005: Documentation. (WIP)
>>       (Same as 5. in your suggestion)
> 
> Can we merge 0002 and 0004?

Can we do it when we merge this patch set if it's still
desirable at the time? Because:

* I think that separated 0002 and 0004 patches are easier to
  review than squashed 0002 and 0004 patch.
* I still think that someone may don't like defining COPY
  handlers for built-in formats. If we don't define COPY
  handlers for built-in formats finally, we can just drop
  0004.

>> We can merge 0001 quickly, right?
> 
> I don't think it makes sense to push only 0001 as it's a completely
> preliminary patch for subsequent patches. It would be prudent to push
> it once other patches are ready too.

Hmm. I feel that 0001 is a refactoring category patch like
merged patches. In general, distinct enum value names are
easier to understand.

BTW, does the "other patches" include the documentation
patch...?


Thanks,
-- 
kou

  257. Re: Make COPY format extendable: Extract COPY TO format implementations

    David G. Johnston <david.g.johnston@gmail.com> — 2025-03-29T16:12:00Z

    On Sat, Mar 29, 2025 at 1:57 AM Sutou Kouhei <kou@clear-code.com> wrote:


> * I still think that someone may don't like defining COPY
>   handlers for built-in formats. If we don't define COPY
>   handlers for built-in formats finally, we can just drop
>   0004.
>

We should (and usually do) dog-food APIs when reasonable and this situation
seems quite reasonable.  I'd push back quite a bit about publishing this
without any internal code leveraging it.


> >> We can merge 0001 quickly, right?
> >
> > I don't think it makes sense to push only 0001 as it's a completely
> > preliminary patch for subsequent patches. It would be prudent to push
> > it once other patches are ready too.
>
> Hmm. I feel that 0001 is a refactoring category patch like
> merged patches. In general, distinct enum value names are
> easier to understand.
>
>
I'm for pushing 0001.  We've had copyfrom_internal.h for a while now and
this seems like a simple refactor to make that area of the code cleaner via
symmetry.

David J.

  258. Re: Make COPY format extendable: Extract COPY TO format implementations

    David G. Johnston <david.g.johnston@gmail.com> — 2025-03-29T16:48:22Z

    On Wed, Mar 26, 2025 at 8:28 PM Sutou Kouhei <kou@clear-code.com> wrote:

>
> The attached v39 patch set uses the followings:
>
> 0001: Create copyto_internal.h and change COPY_XXX to
>       COPY_SOURCE_XXX and COPY_DEST_XXX accordingly.
>       (Same as 1. in your suggestion)
> 0002: Support custom format for both COPY TO and COPY FROM.
>       (Same as 2. in your suggestion)
> 0003: Expose necessary helper functions such as CopySendEndOfRow()
>       and add CopyFromSkipErrorRow().
>       (3. + 4. in your suggestion)
> 0004: Define handler functions for built-in formats.
>       (Not included in your suggestion)
> 0005: Documentation. (WIP)
>       (Same as 5. in your suggestion)
>
>
I don't think this module should be responsible for testing the validity of
"qualified names in a string literal" behavior.  Having some of the tests
use a schema qualification, and I'd suggest explicit
double-quoting/case-folding, wouldn't hurt just to demonstrate it's
possible, and how extensions should be referenced, but definitely don't
need tests to prove the broken cases are indeed broken.  This relies on an
existing API that has its own tests.  It is definitely pointlessly
redundant to have 6 tests that only differ from 6 other tests in their use
of a schema qualification.

I prefer keeping 0002 and 0004 separate.  In particular, keeping the design
choice of "unqualified internal format names ignore search_path" should
stand out as its own commit.

David J.

  259. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2025-03-30T02:31:26Z

    Hi,

In <CAKFQuwYF7VnYcS9dkfvdzt-dgftMB1DV0bjRcNC8-4iYGS+gjw@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Sat, 29 Mar 2025 09:48:22 -0700,
  "David G. Johnston" <david.g.johnston@gmail.com> wrote:

> I don't think this module should be responsible for testing the validity of
> "qualified names in a string literal" behavior.  Having some of the tests
> use a schema qualification, and I'd suggest explicit
> double-quoting/case-folding, wouldn't hurt just to demonstrate it's
> possible, and how extensions should be referenced, but definitely don't
> need tests to prove the broken cases are indeed broken.  This relies on an
> existing API that has its own tests.  It is definitely pointlessly
> redundant to have 6 tests that only differ from 6 other tests in their use
> of a schema qualification.

You suggest the followings, right?

1. Add tests for "Schema.Name" with mixed cases
2. Remove the following 6 tests in
   src/test/modules/test_copy_format/sql/invalid.sql

   ----
   COPY public.test FROM stdin WITH (FORMAT 'test_schema.test_copy_format_wrong_input_type');
   COPY public.test TO stdout WITH (FORMAT 'test_schema.test_copy_format_wrong_input_type');
   COPY public.test FROM stdin WITH (FORMAT 'test_schema.test_copy_format_wrong_return_type');
   COPY public.test TO stdout WITH (FORMAT 'test_schema.test_copy_format_wrong_return_type');
   COPY public.test FROM stdin WITH (FORMAT 'test_schema.test_copy_format_wrong_return_value');
   COPY public.test TO stdout WITH (FORMAT 'test_schema.test_copy_format_wrong_return_value');
   ----

   because we have the following 6 tests:

   ----
   SET search_path = public,test_schema;
   COPY public.test FROM stdin WITH (FORMAT 'test_copy_format_wrong_input_type');
   COPY public.test TO stdout WITH (FORMAT 'test_copy_format_wrong_input_type');
   COPY public.test FROM stdin WITH (FORMAT 'test_copy_format_wrong_return_type');
   COPY public.test TO stdout WITH (FORMAT 'test_copy_format_wrong_return_type');
   COPY public.test FROM stdin WITH (FORMAT 'test_copy_format_wrong_return_value');
   COPY public.test TO stdout WITH (FORMAT 'test_copy_format_wrong_return_value');
   RESET search_path;
   ----
3. Remove the following tests because the behavior must be
   tested in other places:

   ----
   COPY public.test FROM stdin WITH (FORMAT 'nonexistent');
   COPY public.test TO stdout WITH (FORMAT 'nonexistent');
   COPY public.test FROM stdin WITH (FORMAT 'invalid.qualified.name');
   COPY public.test TO stdout WITH (FORMAT 'invalid.qualified.name');
   ----

Does it miss something?


1.: There is no difference between single-quoting and
    double-quoting here. Because the information what quote
    was used for the given FORMAT value isn't remained
    here. Should we update gram.y?

2.: I don't have a strong opinion for it. If nobody objects
    it, I'll remove them.

3.: I don't have a strong opinion for it. If nobody objects
    it, I'll remove them.


Thanks,
-- 
kou

  260. Re: Make COPY format extendable: Extract COPY TO format implementations

    David G. Johnston <david.g.johnston@gmail.com> — 2025-03-31T17:05:34Z

    On Wed, Mar 26, 2025 at 8:28 PM Sutou Kouhei <kou@clear-code.com> wrote:

>
> > ---
> > +static const CopyFromRoutine CopyFromRoutineTestCopyFormat = {
> > +        .type = T_CopyFromRoutine,
> > +        .CopyFromInFunc = CopyFromInFunc,
> > +        .CopyFromStart = CopyFromStart,
> > +        .CopyFromOneRow = CopyFromOneRow,
> > +        .CopyFromEnd = CopyFromEnd,
> > +};
> >
>
>
In trying to document the current API I'm strongly disliking it.  Namely,
the amount of internal code an extension needs to care about/copy-paste to
create a working handler.

Presently, pg_type defines text and binary I/O routines and the text/csv
formats use the text I/O while binary uses binary I/O - for all
attributes.  The CopyFromInFunc API allows for each attribute to somehow
have its I/O format individualized.  But I don't see how that is practical
or useful, and it adds burden on API users.

I suggest we remove both .CopyFromInFunc and .CopyFromStart/End and add a
property to CopyFromRoutine (.ioMode?) with values of either Copy_IO_Text
or Copy_IO_Binary and then just branch to either:

CopyFromTextLikeInFunc & CopyFromTextLikeStart/End
or
CopyFromBinaryInFunc & CopyFromStart/End

So, in effect, the only method an extension needs to write is converting
to/from the 'serialized' form to the text/binary form (text being near
unanimous).

In a similar manner, the amount of boilerplate within CopyFromOneRow seems
undesirable from an API perspective.

cstate->cur_attname = NameStr(att->attname);
cstate->cur_attval = string;

if (string != NULL)
    nulls[m] = false;

if (cstate->defaults[m])
{
    /* We must have switched into the per-tuple memory context */
    Assert(econtext != NULL);
    Assert(CurrentMemoryContext == econtext->ecxt_per_tuple_memory);

    values[m] = ExecEvalExpr(defexprs[m], econtext, &nulls[m]);
}

/*
    * If ON_ERROR is specified with IGNORE, skip rows with soft errors
    */
else if (!InputFunctionCallSafe(&in_functions[m],
                                string,
                                typioparams[m],
                                att->atttypmod,
                                (Node *) cstate->escontext,
                                &values[m]))
{
    CopyFromSkipErrorRow(cstate);
    return true;
}

cstate->cur_attname = NULL;
cstate->cur_attval = NULL;

It seems to me that CopyFromOneRow could simply produce a *string
collection,
one cell per attribute, and NextCopyFrom could do all of the above on a
for-loop over *string

The pg_type and pg_proc catalogs are not extensible so the API can and
should be limited to
producing the, usually text, values that are ready to be passed into the
text I/O routines and all
the work to find and use types and functions left in the template code.

I haven't looked at COPY TO but I am expecting much the same. The API
should simply receive
the content of the type I/O output routine (binary or text as it dictates)
for each output attribute, by
row, and be expected to take those values and produce a final output from
them.

David J.

  261. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2025-03-31T18:51:30Z

    On Sat, Mar 29, 2025 at 9:49 AM David G. Johnston
<david.g.johnston@gmail.com> wrote:
>
> On Wed, Mar 26, 2025 at 8:28 PM Sutou Kouhei <kou@clear-code.com> wrote:
>>
>>
>> The attached v39 patch set uses the followings:
>>
>> 0001: Create copyto_internal.h and change COPY_XXX to
>>       COPY_SOURCE_XXX and COPY_DEST_XXX accordingly.
>>       (Same as 1. in your suggestion)
>> 0002: Support custom format for both COPY TO and COPY FROM.
>>       (Same as 2. in your suggestion)
>> 0003: Expose necessary helper functions such as CopySendEndOfRow()
>>       and add CopyFromSkipErrorRow().
>>       (3. + 4. in your suggestion)
>> 0004: Define handler functions for built-in formats.
>>       (Not included in your suggestion)
>> 0005: Documentation. (WIP)
>>       (Same as 5. in your suggestion)
>>
>
> I prefer keeping 0002 and 0004 separate.  In particular, keeping the design choice of "unqualified internal format names ignore search_path" should stand out as its own commit.

What is the point of having separate commits for already-agreed design
choices? I guess that it would make it easier to revert that decision.
But I think it makes more sense that if we agree with "unqualified
internal format names ignore search_path" the original commit includes
that decision and describes it in the commit message. If we want to
change that design based on the discussion later on, we can have a
separate commit that makes that change and has the link to the
discussion.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  262. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2025-03-31T19:35:23Z

    On Sat, Mar 29, 2025 at 1:57 AM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <CAD21AoBKMNsO+b6wahb6847xwFci1JCfV+JykoMziVgiFxB6cw@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 28 Mar 2025 22:37:03 -0700,
>   Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> >> I've added the following tests:
> >>
> >> * Wrong input type handler without namespace
> >> * Wrong input type handler with namespace
> >> * Wrong return type handler without namespace
> >> * Wrong return type handler with namespace
> >> * Wrong return value (Copy*Routine isn't returned) handler without namespace
> >> * Wrong return value (Copy*Routine isn't returned) handler with namespace
> >> * Nonexistent handler
> >> * Invalid qualified name
> >> * Valid handler without namespace and without search_path
> >> * Valid handler without namespace and with search_path
> >> * Valid handler with namespace
> >
> > Probably we can merge these newly added four files into one .sql file?
>
> I know that src/test/regress/sql/ uses this style (one .sql
> file includes many test patterns in one large category). I
> understand that the style is preferable in
> src/test/regress/sql/ because src/test/regress/sql/ has
> tests for many categories.
>
> But do we need to follow the style in
> src/test/modules/*/sql/ too? If we use the style in
> src/test/modules/*/sql/, we need to have only one .sql in
> src/test/modules/*/sql/ because src/test/modules/*/ are for
> each category.
>
> And the current .sql per sub-category style is easy to debug
> (at least for me). For example, if we try qualified name
> cases on debugger, we can use "\i sql/schema.sql" instead of
> extracting target statements from .sql that includes many
> unrelated statements. (Or we can use "\i sql/all.sql" and
> many GDB "continue"s.)
>
> BTW, it seems that src/test/modules/test_ddl_deparse/sql/
> uses .sql per sub-category style. Should we use one .sql
> file for sql/test/modules/test_copy_format/sql/? If it's
> required for merging this patch set, I'll do it.

I'm not sure that the regression test queries are categorized in the
same way as in test_ddl_deparse. While the former have separate .sql
files for different types of inputs (valid inputs and invalid inputs
etc.) , which seems finer graind, the latter has .sql files for each
DDL command.

Most of the queries under test_copy_format/sql verifies the input
patterns of the FORMAT option. I find that the regression tests
included in that directory probably should focus on testing new
callback APIs and some regression tests for FORMAT option handling can
be moved into the normal regression test suite (e.g., in copy.sql or a
new file like copy_format.sql). IIUC testing for invalid input
patterns can be done even without creating artificial wrong handler
functions.

>
> > Also we need to add some comments to describe what these queries test.
> > For example, it's not clear to me at a glance what queries in
> > no-schema.sql are going to test as there is no comment there.
>
> Hmm. You refer no_schema.sql in 0002, right?
>
> ----
> CREATE EXTENSION test_copy_format;
> CREATE TABLE public.test (a smallint, b integer, c bigint);
> INSERT INTO public.test VALUES (1, 2, 3), (12, 34, 56), (123, 456, 789);
> COPY public.test FROM stdin WITH (FORMAT 'test_copy_format');
> \.
> COPY public.test TO stdout WITH (FORMAT 'test_copy_format');
> DROP TABLE public.test;
> DROP EXTENSION test_copy_format;
> ----
>
> In general, COPY FORMAT tests focus on "COPY FROM WITH
> (FORMAT ...)" and "COPY TO WITH (FORMAT ...)". And the file
> name "no_schema" shows that it doesn't use qualified
> name. Based on this, I feel that the above content is very
> straightforward without any comment.
>
> What should we add as comments? For example, do we need the
> following comments?
>
> ----
> -- This extension includes custom COPY handler: test_copy_format
> CREATE EXTENSION test_copy_format;
> -- Test data
> CREATE TABLE public.test (a smallint, b integer, c bigint);
> INSERT INTO public.test VALUES (1, 2, 3), (12, 34, 56), (123, 456, 789);
> -- Use custom COPY handler, test_copy_format, without
> -- schema for FROM.
> COPY public.test FROM stdin WITH (FORMAT 'test_copy_format');
> \.
> -- Use custom COPY handler, test_copy_format, without
> -- schema for TO.
> COPY public.test TO stdout WITH (FORMAT 'test_copy_format');
> -- Cleanup
> DROP TABLE public.test;
> DROP EXTENSION test_copy_format;
> ----

I'd like to see in the comment what the tests expect. Taking the
following queries as an example,

COPY public.test FROM stdin WITH (FORMAT 'test_copy_format');
\.
COPY public.test TO stdout WITH (FORMAT 'test_copy_format');

it would help readers understand the test case better if we have a
comment like for example:

-- Specify the custom format name without schema. We test if both
-- COPY TO and COPY FROM can find the correct handler function
-- in public schema.

>
> > One problem in the following chunk I can see is:
> >
> > +           qualified_format = stringToQualifiedNameList(format, NULL);
> > +           DeconstructQualifiedName(qualified_format, &schema, &fmt);
> > +           if (!schema || strcmp(schema, "pg_catalog") == 0)
> > +           {
> > +               if (strcmp(fmt, "csv") == 0)
> > +                   opts_out->csv_mode = true;
> > +               else if (strcmp(fmt, "binary") == 0)
> > +                   opts_out->binary = true;
> > +           }
> >
> > Non-qualified names depend on the search_path value so it's not
> > necessarily a built-in format. If the user specifies 'csv' with
> > seach_patch = 'myschema, pg_catalog', the COPY command unnecessarily
> > sets csv_mode true.
>
> I think that we should always use built-in COPY handlers for
> (not-qualified) "text", "csv" and "binary" for
> compatibility. If we allow custom COPY handlers for
> (not-qualified) "text", "csv" and "binary", pg_dump or
> existing dump may be broken. Because we must use the same
> COPY handler when we dump (COPY TO) and we restore (COPY
> FROM).

I agreed.

>
> BTW, the current implementation always uses
> pg_catalog.{text,csv,binary} for (not-qualified) "text",
> "csv" and "binary" even when there are
> myschema.{text,csv,binary}. See
> src/test/modules/test_copy_format/sql/builtin.sql. But I
> haven't looked into it why...

Sorry, I don't follow that. IIUC test_copy_format extension doesn't
create a handler function in myschema schema, and SQLs in builtin.sql
seem to work as expected (specifying a non-qualified built-in format
unconditionally uses the built-in format).

>
> >                                                            Also, it's
> > weired to me that cstate has csv_mode and binary fields even though
> > the format should have already been known by the callback functions.
>
> You refer CopyFomratOptions::{csv_mode,binary} not
> Copy{To,From}StateData, right?

Yes. I referred to the wrong one.

> And you suggest that we
> should replace all opts.csv_mode and opts.binary with
> opts.handler == F_CSV and opts.handler == F_BINARY, right?
>
> We can do it but I suggest that we do it as a refactoring
> (or cleanup) in a separated patch for easy to review.

I think that csv_mode and binary are used mostly in
ProcessCopyOptions() so probably we can use local variables for that.
I find there are two other places where to use csv_mode:
NextCopyFromRawFields() and CopyToTextLikeStart(). I think we can
simply check the handler function's OID there, or we can define macros
like COPY_FORMAT_IS_TEXT/CSV/BINARY checking the OID and use them
there.

>
> >> > ---
> >> > +/*
> >> > + * Export CopySendEndOfRow() for extensions. We want to keep
> >> > + * CopySendEndOfRow() as a static function for
> >> > + * optimization. CopySendEndOfRow() calls in this file may be optimized by a
> >> > + * compiler.
> >> > + */
> >> > +void
> >> > +CopyToStateFlush(CopyToState cstate)
> >> > +{
> >> > +        CopySendEndOfRow(cstate);
> >> > +}
> >> >
> >> > Is there any reason to use a different name for public functions?
> >>
> >> In this patch set, I use "CopyFrom"/"CopyTo" prefixes for
> >> public APIs for custom COPY FORMAT handler extensions. It
> >> will help understanding related APIs. Is it strange in
> >> PostgreSQL?
> >
> > I see your point. Probably we need to find a better name as the name
> > CopyToStateFlush doesn't sound well like this API should be called
> > only once at the end of a row (IOW user might try to call it multiple
> > times to 'flush' the state while processing a row). How about
> > CopyToEndOfRow()?
>
> CopyToStateFlush() can be called multiple times in a row. It
> can also be called only once with multiple rows. Because it
> just flushes the current buffer.
>
> Existing CopySendEndOfRow() is called at the end of a
> row. (Buffer is flushed at the end of row.) So I think that
> the "EndOfRow" was chosen.
>
> Some custom COPY handlers may not be row based. For example,
> Apache Arrow COPY handler doesn't flush buffer for each row.
> So, we should provide "flush" API not "end of row" API for
> extensibility.

Okay, understood.

>
> >> We can merge 0001 quickly, right?
> >
> > I don't think it makes sense to push only 0001 as it's a completely
> > preliminary patch for subsequent patches. It would be prudent to push
> > it once other patches are ready too.
>
> Hmm. I feel that 0001 is a refactoring category patch like
> merged patches. In general, distinct enum value names are
> easier to understand.

Right, but the patches that have already been merged contributed to
speed up COPY commands, but 0001 patch also introduces
copyto_internal.h, which is not used by anyone in a case where the
custom copy format patch is not merged. Without adding
copyto_internal.h changing enum value names less makes sense to me.

> BTW, does the "other patches" include the documentation
> patch...?

I think that when pushing the main custom COPY format patch, we need
to include the documentation changes into it.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  263. Re: Make COPY format extendable: Extract COPY TO format implementations

    David G. Johnston <david.g.johnston@gmail.com> — 2025-03-31T20:42:16Z

    On Mon, Mar 31, 2025 at 11:52 AM Masahiko Sawada <sawada.mshk@gmail.com>
wrote:

> On Sat, Mar 29, 2025 at 9:49 AM David G. Johnston
> <david.g.johnston@gmail.com> wrote:
> >
> > On Wed, Mar 26, 2025 at 8:28 PM Sutou Kouhei <kou@clear-code.com> wrote:
> >>
> >>
> >> The attached v39 patch set uses the followings:
> >>
> >> 0001: Create copyto_internal.h and change COPY_XXX to
> >>       COPY_SOURCE_XXX and COPY_DEST_XXX accordingly.
> >>       (Same as 1. in your suggestion)
> >> 0002: Support custom format for both COPY TO and COPY FROM.
> >>       (Same as 2. in your suggestion)
> >> 0003: Expose necessary helper functions such as CopySendEndOfRow()
> >>       and add CopyFromSkipErrorRow().
> >>       (3. + 4. in your suggestion)
> >> 0004: Define handler functions for built-in formats.
> >>       (Not included in your suggestion)
> >> 0005: Documentation. (WIP)
> >>       (Same as 5. in your suggestion)
> >>
> >
> > I prefer keeping 0002 and 0004 separate.  In particular, keeping the
> design choice of "unqualified internal format names ignore search_path"
> should stand out as its own commit.
>
> What is the point of having separate commits for already-agreed design
> choices? I guess that it would make it easier to revert that decision.
> But I think it makes more sense that if we agree with "unqualified
> internal format names ignore search_path" the original commit includes
> that decision and describes it in the commit message. If we want to
> change that design based on the discussion later on, we can have a
> separate commit that makes that change and has the link to the
> discussion.
>

Fair.  Comment withdrawn.  Though I was referring to the WIP patches; I
figured the final patch would squash this all together in any case.

David J.

  264. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2025-04-04T06:42:48Z

    Hi,

In <CAKFQuwbhSssKTJyeYo9rn20zffV3L7wdQSbEQ8zwRfC=uXLkVA@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Mon, 31 Mar 2025 10:05:34 -0700,
  "David G. Johnston" <david.g.johnston@gmail.com> wrote:

>              The CopyFromInFunc API allows for each attribute to somehow
> have its I/O format individualized.  But I don't see how that is practical
> or useful, and it adds burden on API users.

If an extension want to use I/O routines, it can use the
CopyFromInFunc API. Otherwise it can provide an empty
function.

For example,
https://github.com/MasahikoSawada/pg_copy_jsonlines/blob/master/copy_jsonlines.c
uses the CopyFromInFunc API but
https://github.com/kou/pg-copy-arrow/blob/main/copy_arrow.cc
uses an empty function for the CopyFromInFunc API.

The "it adds burden" means that "defining an empty function
is inconvenient", right? See also our past discussion for
this design:

https://www.postgresql.org/message-id/ZbijVn9_51mljMAG%40paquier.xyz

> Keeping empty options does not strike as a bad idea, because this
> forces extension developers to think about this code path rather than
> just ignore it.


> I suggest we remove both .CopyFromInFunc and .CopyFromStart/End and add a
> property to CopyFromRoutine (.ioMode?) with values of either Copy_IO_Text
> or Copy_IO_Binary and then just branch to either:
> 
> CopyFromTextLikeInFunc & CopyFromTextLikeStart/End
> or
> CopyFromBinaryInFunc & CopyFromStart/End
> 
> So, in effect, the only method an extension needs to write is converting
> to/from the 'serialized' form to the text/binary form (text being near
> unanimous).

I object this API. If we choose this API, we can create only
custom COPY formats that compatible with PostgreSQL's
text/binary form. For example, the above jsonlines format
and Apache Arrow format aren't implemented. It's meaningless
to introduce this custom COPY format mechanism with the
suggested API.

> It seems to me that CopyFromOneRow could simply produce a *string
> collection,
> one cell per attribute, and NextCopyFrom could do all of the above on a
> for-loop over *string

You suggest that we use a string collection instead of a
Datum collection in CopyFromOneRow() and convert a string
collection to a Datum collection in NextCopyFrom(), right?

I object this API. Because it has needless string <-> Datum
conversion overhead. For example,
https://github.com/MasahikoSawada/pg_copy_jsonlines/blob/master/copy_jsonlines.c
parses a JSON value to Datum. If we use this API, we need to
convert parsed Datum to string in an extension and
NextCopyFrom() re-converts the converted string to
Datum. It will slow down custom COPY format.

I want this custom COPY format feature for performance. So
APIs that require needless overhead for non text/csv/binary
formats isn't acceptable to me.


Thanks,
-- 
kou

  265. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2025-04-04T08:38:39Z

    Hi,

In <CAD21AoDOcYah-nREv09BB3ZoB-k+Yf1XUfJcDMoq=LLvV1v75w@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Mon, 31 Mar 2025 12:35:23 -0700,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

> Most of the queries under test_copy_format/sql verifies the input
> patterns of the FORMAT option. I find that the regression tests
> included in that directory probably should focus on testing new
> callback APIs and some regression tests for FORMAT option handling can
> be moved into the normal regression test suite (e.g., in copy.sql or a
> new file like copy_format.sql). IIUC testing for invalid input
> patterns can be done even without creating artificial wrong handler
> functions.

Can we clarify what should we do for the next patch set
before we create the next patch set? Are the followings
correct?

1. Move invalid input patterns in
   src/test/modules/test_copy_format/sql/invalid.sql to
   src/test/regress/sql/copy.sql as much as possible.
   * We can do only 4 patterns in 16 patterns.
   * Other tests in
     src/test/modules/test_copy_format/sql/*.sql depend on
     custom COPY handler for test. So we can't move to
     src/test/regress/sql/copy.sql.
2. Create
   src/test/modules/test_copy_format/sql/test_copy_format.sql
   and move all contents in existing *.sql to the file 

> I'd like to see in the comment what the tests expect. Taking the
> following queries as an example,
> 
> COPY public.test FROM stdin WITH (FORMAT 'test_copy_format');
> \.
> COPY public.test TO stdout WITH (FORMAT 'test_copy_format');
> 
> it would help readers understand the test case better if we have a
> comment like for example:
> 
> -- Specify the custom format name without schema. We test if both
> -- COPY TO and COPY FROM can find the correct handler function
> -- in public schema.

I agree with you that the comment is useful when we use
src/test/modules/test_copy_format/sql/test_copy_format.sql
for all tests. (I feel that it's redundant when we use
no_schema.sql.) I'll add it when I create
test_copy_format.sql in the next patch set.

>> BTW, the current implementation always uses
>> pg_catalog.{text,csv,binary} for (not-qualified) "text",
>> "csv" and "binary" even when there are
>> myschema.{text,csv,binary}. See
>> src/test/modules/test_copy_format/sql/builtin.sql. But I
>> haven't looked into it why...
> 
> Sorry, I don't follow that. IIUC test_copy_format extension doesn't
> create a handler function in myschema schema, and SQLs in builtin.sql
> seem to work as expected (specifying a non-qualified built-in format
> unconditionally uses the built-in format).

Ah, sorry. I should have not used "myschema." in the text
with builtin.sql reference. I just wanted to say "qualified
text,csv,binary formats" by "myschema.{text,csv,binary}". In
builtin.sql uses "public" schema not "myschema"
schema. Sorry.

Yes. builtin.sql works as expected but I don't know why. I
don't add any special codes for them. If "test_copy_format"
exists in public schema, "FORMAT 'test_copy_format'" uses
it. But if "text" exists in public schema, "FORMAT 'text'"
doesn't uses it. ("pg_catalog.text" is used instead.)

>> We can do it but I suggest that we do it as a refactoring
>> (or cleanup) in a separated patch for easy to review.
> 
> I think that csv_mode and binary are used mostly in
> ProcessCopyOptions() so probably we can use local variables for that.
> I find there are two other places where to use csv_mode:
> NextCopyFromRawFields() and CopyToTextLikeStart(). I think we can
> simply check the handler function's OID there, or we can define macros
> like COPY_FORMAT_IS_TEXT/CSV/BINARY checking the OID and use them
> there.

We need this change for "ready for merge", right?


Can we clarify items should be resolved for "ready for
merge"?

Are the followings correct?

1. Move invalid input patterns in
   src/test/modules/test_copy_format/sql/invalid.sql to
   src/test/regress/sql/copy.sql as much as possible.
2. Create
   src/test/modules/test_copy_format/sql/test_copy_format.sql
   and move all contents in existing *.sql to the file.
3. Add comments what the tests expect to
   src/test/modules/test_copy_format/sql/test_copy_format.sql.
4. Remove CopyFormatOptions::{binary,csv_mode}.
5. Squash the "Support custom format" patch and the "Define
   handler functions for built-in formats" patch.
   * Could you do it when you push it? Or is it required for
     "ready for merge"?
6. Use handler OID for detecting the default built-in format
   instead of comparing the given format as string.
7. Update documentation.

There are 3 unconfirmed suggested changes for tests in:
https://www.postgresql.org/message-id/20250330.113126.433742864258096312.kou%40clear-code.com

Here are my opinions for them:

> 1.: There is no difference between single-quoting and
>     double-quoting here. Because the information what quote
>     was used for the given FORMAT value isn't remained
>     here. Should we update gram.y?
> 
> 2.: I don't have a strong opinion for it. If nobody objects
>     it, I'll remove them.
> 
> 3.: I don't have a strong opinion for it. If nobody objects
>     it, I'll remove them.

Is the 1. required for "ready for merge"? If so, is there
any suggestion? I don't have a strong opinion for it.

If there are no more opinions for 2. and 3., I'll remove
them.



Thanks,
-- 
kou

  266. Re: Make COPY format extendable: Extract COPY TO format implementations

    jian he <jian.universality@gmail.com> — 2025-04-06T11:29:46Z

    On Thu, Mar 27, 2025 at 11:29 AM Sutou Kouhei <kou@clear-code.com> wrote:
> We can merge 0001 quickly, right?

I did a brief review of v39-0001 and v39-0002.

text:
COPY_FILE
COPY_FRONTEND
still appear on comments in copyfrom_internal.h and copyto.c,
Should it be removed?


+#include "commands/copyto_internal.h"
#include "commands/progress.h"
#include "executor/execdesc.h"
#include "executor/executor.h"
#include "executor/tuptable.h"

"copyto_internal.h" already include:

#include "executor/execdesc.h"
#include "executor/tuptable.h"
so you should removed
"
#include "executor/execdesc.h"
#include "executor/tuptable.h"
"
in copyto.c.



CREATE FUNCTION test_copy_format(internal)
    RETURNS copy_handler
    AS 'MODULE_PATHNAME', 'test_copy_format'
    LANGUAGE C;
src/backend/commands/copy.c: ProcessCopyOptions
            if (strcmp(fmt, "text") == 0)
                 /* default format */ ;
            else if (strcmp(fmt, "csv") == 0)
                opts_out->csv_mode = true;
            else if (strcmp(fmt, "binary") == 0)
                opts_out->binary = true;
            else
            {
                List       *qualified_format;
                ....
            }
what if our customized format name is one of "csv", "binary", "text",
then that ELSE branch will never be reached.
then our customized format is being shadowed?


https://www.postgresql.org/docs/current/error-message-reporting.html
"The extra parentheses were required before PostgreSQL version 12, but
are now optional."

means that
    ereport(NOTICE, (errmsg("CopyFromInFunc: attribute: %s",
format_type_be(atttypid))));
can change to
    ereport(NOTICE, errmsg("CopyFromInFunc: attribute: %s",
format_type_be(atttypid)));

all
ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), ....

can also be simplified to

ereport(ERROR, errcode(ERRCODE_INVALID_PARAMETER_VALUE), ....

  267. Re: Make COPY format extendable: Extract COPY TO format implementations

    David G. Johnston <david.g.johnston@gmail.com> — 2025-04-06T16:20:56Z

    On Sun, Apr 6, 2025 at 4:30 AM jian he <jian.universality@gmail.com> wrote:

>
> CREATE FUNCTION test_copy_format(internal)
>     RETURNS copy_handler
>     AS 'MODULE_PATHNAME', 'test_copy_format'
>     LANGUAGE C;
> src/backend/commands/copy.c: ProcessCopyOptions
>             if (strcmp(fmt, "text") == 0)
>                  /* default format */ ;
>             else if (strcmp(fmt, "csv") == 0)
>                 opts_out->csv_mode = true;
>             else if (strcmp(fmt, "binary") == 0)
>                 opts_out->binary = true;
>             else
>             {
>                 List       *qualified_format;
>                 ....
>             }
> what if our customized format name is one of "csv", "binary", "text",
> then that ELSE branch will never be reached.
> then our customized format is being shadowed?
>
>
Yes.  The user of your extension can specify a schema name to get access to
your conflicting format name choice but all the existing code out there
that relied on text/csv/binary being the built-in options continue to
behave the same no matter the search_path.

David J.

  268. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2025-04-07T06:34:18Z

    Hi,

In <CACJufxG=njY32g=YAF4T6rvXySN56VFbYt4ffjLTRBYQTKPAFg@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Sun, 6 Apr 2025 19:29:46 +0800,
  jian he <jian.universality@gmail.com> wrote:

> I did a brief review of v39-0001 and v39-0002.
> 
> text:
> COPY_FILE
> COPY_FRONTEND
> still appear on comments in copyfrom_internal.h and copyto.c,
> Should it be removed?

Good catch!
I found them in copy{from,to}_internal.h but couldn't find
them in copyto.c. It's a typo, right?

We should update them instead of removing them. I'll update
them in the next patch set.


> +#include "commands/copyto_internal.h"
> #include "commands/progress.h"
> #include "executor/execdesc.h"
> #include "executor/executor.h"
> #include "executor/tuptable.h"
> 
> "copyto_internal.h" already include:
> 
> #include "executor/execdesc.h"
> #include "executor/tuptable.h"
> so you should removed
> "
> #include "executor/execdesc.h"
> #include "executor/tuptable.h"
> "
> in copyto.c.

You're right. I'll update this too in the next patch set.

> CREATE FUNCTION test_copy_format(internal)
>     RETURNS copy_handler
>     AS 'MODULE_PATHNAME', 'test_copy_format'
>     LANGUAGE C;
> src/backend/commands/copy.c: ProcessCopyOptions
>             if (strcmp(fmt, "text") == 0)
>                  /* default format */ ;
>             else if (strcmp(fmt, "csv") == 0)
>                 opts_out->csv_mode = true;
>             else if (strcmp(fmt, "binary") == 0)
>                 opts_out->binary = true;
>             else
>             {
>                 List       *qualified_format;
>                 ....
>             }
> what if our customized format name is one of "csv", "binary", "text",
> then that ELSE branch will never be reached.
> then our customized format is being shadowed?

Right. We should not use customized format handlers to keep
backward compatibility.

> https://www.postgresql.org/docs/current/error-message-reporting.html
> "The extra parentheses were required before PostgreSQL version 12, but
> are now optional."
> 
> means that
>     ereport(NOTICE, (errmsg("CopyFromInFunc: attribute: %s",
> format_type_be(atttypid))));
> can change to
>     ereport(NOTICE, errmsg("CopyFromInFunc: attribute: %s",
> format_type_be(atttypid)));
> 
> all
> ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), ....
> 
> can also be simplified to
> 
> ereport(ERROR, errcode(ERRCODE_INVALID_PARAMETER_VALUE), ....

Oh, I didn't notice it. Can we do it as a separated patch
because we have many codes that use this style in
copy*.c. The separated patch should update this style at
once.


Thanks,
-- 
kou

  269. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2025-04-24T06:44:55Z

    On Fri, Apr 4, 2025 at 1:38 AM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <CAD21AoDOcYah-nREv09BB3ZoB-k+Yf1XUfJcDMoq=LLvV1v75w@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Mon, 31 Mar 2025 12:35:23 -0700,
>   Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> > Most of the queries under test_copy_format/sql verifies the input
> > patterns of the FORMAT option. I find that the regression tests
> > included in that directory probably should focus on testing new
> > callback APIs and some regression tests for FORMAT option handling can
> > be moved into the normal regression test suite (e.g., in copy.sql or a
> > new file like copy_format.sql). IIUC testing for invalid input
> > patterns can be done even without creating artificial wrong handler
> > functions.
>
> Can we clarify what should we do for the next patch set
> before we create the next patch set? Are the followings
> correct?
>
> 1. Move invalid input patterns in
>    src/test/modules/test_copy_format/sql/invalid.sql to
>    src/test/regress/sql/copy.sql as much as possible.
>    * We can do only 4 patterns in 16 patterns.
>    * Other tests in
>      src/test/modules/test_copy_format/sql/*.sql depend on
>      custom COPY handler for test. So we can't move to
>      src/test/regress/sql/copy.sql.
> 2. Create
>    src/test/modules/test_copy_format/sql/test_copy_format.sql
>    and move all contents in existing *.sql to the file

Agreed.

>
> >> We can do it but I suggest that we do it as a refactoring
> >> (or cleanup) in a separated patch for easy to review.
> >
> > I think that csv_mode and binary are used mostly in
> > ProcessCopyOptions() so probably we can use local variables for that.
> > I find there are two other places where to use csv_mode:
> > NextCopyFromRawFields() and CopyToTextLikeStart(). I think we can
> > simply check the handler function's OID there, or we can define macros
> > like COPY_FORMAT_IS_TEXT/CSV/BINARY checking the OID and use them
> > there.
>
> We need this change for "ready for merge", right?

I think so.

> Can we clarify items should be resolved for "ready for
> merge"?
>
> Are the followings correct?
>
> 1. Move invalid input patterns in
>    src/test/modules/test_copy_format/sql/invalid.sql to
>    src/test/regress/sql/copy.sql as much as possible.
> 2. Create
>    src/test/modules/test_copy_format/sql/test_copy_format.sql
>    and move all contents in existing *.sql to the file.
> 3. Add comments what the tests expect to
>    src/test/modules/test_copy_format/sql/test_copy_format.sql.
> 4. Remove CopyFormatOptions::{binary,csv_mode}.

Agreed with the above items.

> 5. Squash the "Support custom format" patch and the "Define
>    handler functions for built-in formats" patch.
>    * Could you do it when you push it? Or is it required for
>      "ready for merge"?

Let's keep them for now.

> 6. Use handler OID for detecting the default built-in format
>    instead of comparing the given format as string.
> 7. Update documentation.

Agreed.

>
> There are 3 unconfirmed suggested changes for tests in:
> https://www.postgresql.org/message-id/20250330.113126.433742864258096312.kou%40clear-code.com
>
> Here are my opinions for them:
>
> > 1.: There is no difference between single-quoting and
> >     double-quoting here. Because the information what quote
> >     was used for the given FORMAT value isn't remained
> >     here. Should we update gram.y?
> >
> > 2.: I don't have a strong opinion for it. If nobody objects
> >     it, I'll remove them.
> >
> > 3.: I don't have a strong opinion for it. If nobody objects
> >     it, I'll remove them.
>
> Is the 1. required for "ready for merge"? If so, is there
> any suggestion? I don't have a strong opinion for it.
>
> If there are no more opinions for 2. and 3., I'll remove
> them.

Agreed.

I think we would still need some rounds of reviews but the patch is
getting in good shape.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  270. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2025-04-25T12:45:34Z

    Hi,

I've updated the patch set. See the attached v40 patch set.

In <CAD21AoAXzwPC7jjPMTcT80hnzmPa2SUJkiqdYHweEY8sZscEMA@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Wed, 23 Apr 2025 23:44:55 -0700,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

>> Are the followings correct?
>>
>> 1. Move invalid input patterns in
>>    src/test/modules/test_copy_format/sql/invalid.sql to
>>    src/test/regress/sql/copy.sql as much as possible.
>> 2. Create
>>    src/test/modules/test_copy_format/sql/test_copy_format.sql
>>    and move all contents in existing *.sql to the file.
>> 3. Add comments what the tests expect to
>>    src/test/modules/test_copy_format/sql/test_copy_format.sql.
>> 4. Remove CopyFormatOptions::{binary,csv_mode}.
> 
> Agreed with the above items.

Done except 1. because 1. is removed by 3. in the following
list:

----
>> There are 3 unconfirmed suggested changes for tests in:
>> https://www.postgresql.org/message-id/20250330.113126.433742864258096312.kou%40clear-code.com
>>
>> Here are my opinions for them:
>>
>> > 1.: There is no difference between single-quoting and
>> >     double-quoting here. Because the information what quote
>> >     was used for the given FORMAT value isn't remained
>> >     here. Should we update gram.y?
>> >
>> > 2.: I don't have a strong opinion for it. If nobody objects
>> >     it, I'll remove them.
>> >
>> > 3.: I don't have a strong opinion for it. If nobody objects
>> >     it, I'll remove them.
----

0005 is added for 4. Could you squash 0004 ("Use copy
handler for bult-in formats") and 0005 ("Remove
CopyFormatOptions::{binary,csv_mode}") if needed when you
push?

>> 6. Use handler OID for detecting the default built-in format
>>    instead of comparing the given format as string.

Done.

>> 7. Update documentation.

Could someone help this? 0007 is the draft commit for this.

>> There are 3 unconfirmed suggested changes for tests in:
>> https://www.postgresql.org/message-id/20250330.113126.433742864258096312.kou%40clear-code.com
>>
>> Here are my opinions for them:
>>
>> > 1.: There is no difference between single-quoting and
>> >     double-quoting here. Because the information what quote
>> >     was used for the given FORMAT value isn't remained
>> >     here. Should we update gram.y?
>> >
>> > 2.: I don't have a strong opinion for it. If nobody objects
>> >     it, I'll remove them.
>> >
>> > 3.: I don't have a strong opinion for it. If nobody objects
>> >     it, I'll remove them.
>>
>> Is the 1. required for "ready for merge"? If so, is there
>> any suggestion? I don't have a strong opinion for it.
>>
>> If there are no more opinions for 2. and 3., I'll remove
>> them.
> 
> Agreed.

1.: I didn't do anything. Because there is no suggestion.

2., 3.: Done.

> I think we would still need some rounds of reviews but the patch is
> getting in good shape.

I hope that this is completed in this year...


Thanks,
-- 
kou

  271. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2025-05-01T19:15:30Z

    On Fri, Apr 25, 2025 at 5:45 AM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> I've updated the patch set. See the attached v40 patch set.
>
> In <CAD21AoAXzwPC7jjPMTcT80hnzmPa2SUJkiqdYHweEY8sZscEMA@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Wed, 23 Apr 2025 23:44:55 -0700,
>   Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> >> Are the followings correct?
> >>
> >> 1. Move invalid input patterns in
> >>    src/test/modules/test_copy_format/sql/invalid.sql to
> >>    src/test/regress/sql/copy.sql as much as possible.
> >> 2. Create
> >>    src/test/modules/test_copy_format/sql/test_copy_format.sql
> >>    and move all contents in existing *.sql to the file.
> >> 3. Add comments what the tests expect to
> >>    src/test/modules/test_copy_format/sql/test_copy_format.sql.
> >> 4. Remove CopyFormatOptions::{binary,csv_mode}.
> >
> > Agreed with the above items.
>
> Done except 1. because 1. is removed by 3. in the following
> list:
>
> ----
> >> There are 3 unconfirmed suggested changes for tests in:
> >> https://www.postgresql.org/message-id/20250330.113126.433742864258096312.kou%40clear-code.com
> >>
> >> Here are my opinions for them:
> >>
> >> > 1.: There is no difference between single-quoting and
> >> >     double-quoting here. Because the information what quote
> >> >     was used for the given FORMAT value isn't remained
> >> >     here. Should we update gram.y?
> >> >
> >> > 2.: I don't have a strong opinion for it. If nobody objects
> >> >     it, I'll remove them.
> >> >
> >> > 3.: I don't have a strong opinion for it. If nobody objects
> >> >     it, I'll remove them.
> ----
>
> 0005 is added for 4. Could you squash 0004 ("Use copy
> handler for bult-in formats") and 0005 ("Remove
> CopyFormatOptions::{binary,csv_mode}") if needed when you
> push?
>
> >> 6. Use handler OID for detecting the default built-in format
> >>    instead of comparing the given format as string.
>
> Done.
>
> >> 7. Update documentation.
>
> Could someone help this? 0007 is the draft commit for this.
>
> >> There are 3 unconfirmed suggested changes for tests in:
> >> https://www.postgresql.org/message-id/20250330.113126.433742864258096312.kou%40clear-code.com
> >>
> >> Here are my opinions for them:
> >>
> >> > 1.: There is no difference between single-quoting and
> >> >     double-quoting here. Because the information what quote
> >> >     was used for the given FORMAT value isn't remained
> >> >     here. Should we update gram.y?
> >> >
> >> > 2.: I don't have a strong opinion for it. If nobody objects
> >> >     it, I'll remove them.
> >> >
> >> > 3.: I don't have a strong opinion for it. If nobody objects
> >> >     it, I'll remove them.
> >>
> >> Is the 1. required for "ready for merge"? If so, is there
> >> any suggestion? I don't have a strong opinion for it.
> >>
> >> If there are no more opinions for 2. and 3., I'll remove
> >> them.
> >
> > Agreed.
>
> 1.: I didn't do anything. Because there is no suggestion.
>
> 2., 3.: Done.

Thank you for updating the patches.

One of the primary considerations we need to address is the treatment
of the specified format name. The current patch set utilizes built-in
formats (namely 'csv', 'text', and 'binary') when the format name is
either unqualified or explicitly specified with 'pg_catalog' as the
schema. In all other cases, we search for custom format handler
functions based on the search_path. To be frank, I have reservations
about this interface design, as the dependence of the specified custom
format name on the search_path could potentially confuse users.

In light of these concerns, I've been contemplating alternative
interface designs. One promising approach would involve registering
custom copy formats via a C function during module loading
(specifically, in _PG_init()). This method would require extension
authors to invoke a registration function, say
RegisterCustomCopyFormat(), in _PG_init() as follows:

JsonLinesFormatId = RegisterCustomCopyFormat("jsonlines",
                                             &JsonLinesCopyToRoutine,
                                             &JsonLinesCopyFromRoutine);

The registration function would validate the format name and store it
in TopMemoryContext. It would then return a unique identifier that can
be used subsequently to reference the custom copy format extension.

Custom copy format modules could be loaded through
shared_preload_libraries, session_preload_libraries, or the LOAD
command. Extensions could register their own options within this
framework, for example:

RegisterCustomCopyFormatOption(JsonLinesFormatId,
    "custom_option",
    custom_option_handler);

This approach offers several advantages: it would eliminate the
search_path issue, provide greater flexibility, and potentially
simplify the overall interface for users and developers alike. We
might be able to provide a view showing the registered custom COPY
format in the future. Also, these interfaces align with other
customizable functionalities such as custom rmgr, custom lwlock,
custom waitevent, and custom EXPLAIN option etc.

Feedback is very welcome.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  272. Re: Make COPY format extendable: Extract COPY TO format implementations

    Michael Paquier <michael@paquier.xyz> — 2025-05-01T23:03:50Z

    On Thu, May 01, 2025 at 12:15:30PM -0700, Masahiko Sawada wrote:
> In light of these concerns, I've been contemplating alternative
> interface designs. One promising approach would involve registering
> custom copy formats via a C function during module loading
> (specifically, in _PG_init()). This method would require extension
> authors to invoke a registration function, say
> RegisterCustomCopyFormat(), in _PG_init() as follows:
> 
> JsonLinesFormatId = RegisterCustomCopyFormat("jsonlines",
>                                              &JsonLinesCopyToRoutine,
>                                              &JsonLinesCopyFromRoutine);
> 
> The registration function would validate the format name and store it
> in TopMemoryContext. It would then return a unique identifier that can
> be used subsequently to reference the custom copy format extension.

Hmm.  How much should we care about the observability of the COPY
format used by a given backend?  Storing this information in a
backend's TopMemoryContext is OK to get the extensibility basics to
work, but could it make sense to use some shmem state to allocate a
uint32 ID that could be shared by all backends.  Contrary to EXPLAIN,
COPY commands usually run for a very long time, so I am wondering if 
these APIs should be designed so as it would be possible to monitor
the format used.  One layer where the format information could be made
available is the progress reporting view for COPY, for example.  I can
also imagine a pgstats kind where we do COPY stats aggregates, with a
per-format pgstats kind, and sharing a fixed ID across multiple
backends is relevant (when flushing the stats at shutdown, we would
use a name/ID mapping like replication slots).

I don't think that this needs to be relevant for the option part, just 
for the format where, I suspect, we should store in a shmem array
based on the ID allocated the name of the format, the library of the
callback and the function name fed to load_external_function().

Note that custom LWLock and wait events use a shmem state for
monitoring purposes, where we are able to do ID->format name lookups 
as much as format->ID lookups.  Perhaps it's OK not to do that for
COPY, but I am wondering if we'd better design things from scratch
with states in shmem state knowing that COPY is a long-running
operation, and that if one mixes multiple formats they would most
likely want to know which formats are bottlenecks, through SQL.  Cloud
providers would love that.

> This approach offers several advantages: it would eliminate the
> search_path issue, provide greater flexibility, and potentially
> simplify the overall interface for users and developers alike. We
> might be able to provide a view showing the registered custom COPY
> format in the future. Also, these interfaces align with other
> customizable functionalities such as custom rmgr, custom lwlock,
> custom waitevent, and custom EXPLAIN option etc.

Yeah, agreed with the search_path concerns.  We are getting better at
making areas of Postgres more pluggable lately, having a loading path
where we don't have any of these potential issues by design matters.
--
Michael

  273. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2025-05-02T22:52:49Z

    On Thu, May 1, 2025 at 4:04 PM Michael Paquier <michael@paquier.xyz> wrote:
>
> On Thu, May 01, 2025 at 12:15:30PM -0700, Masahiko Sawada wrote:
> > In light of these concerns, I've been contemplating alternative
> > interface designs. One promising approach would involve registering
> > custom copy formats via a C function during module loading
> > (specifically, in _PG_init()). This method would require extension
> > authors to invoke a registration function, say
> > RegisterCustomCopyFormat(), in _PG_init() as follows:
> >
> > JsonLinesFormatId = RegisterCustomCopyFormat("jsonlines",
> >                                              &JsonLinesCopyToRoutine,
> >                                              &JsonLinesCopyFromRoutine);
> >
> > The registration function would validate the format name and store it
> > in TopMemoryContext. It would then return a unique identifier that can
> > be used subsequently to reference the custom copy format extension.
>
> Hmm.  How much should we care about the observability of the COPY
> format used by a given backend?  Storing this information in a
> backend's TopMemoryContext is OK to get the extensibility basics to
> work, but could it make sense to use some shmem state to allocate a
> uint32 ID that could be shared by all backends.  Contrary to EXPLAIN,
> COPY commands usually run for a very long time, so I am wondering if
> these APIs should be designed so as it would be possible to monitor
> the format used.  One layer where the format information could be made
> available is the progress reporting view for COPY, for example.  I can
> also imagine a pgstats kind where we do COPY stats aggregates, with a
> per-format pgstats kind, and sharing a fixed ID across multiple
> backends is relevant (when flushing the stats at shutdown, we would
> use a name/ID mapping like replication slots).
>
> I don't think that this needs to be relevant for the option part, just
> for the format where, I suspect, we should store in a shmem array
> based on the ID allocated the name of the format, the library of the
> callback and the function name fed to load_external_function().
>
> Note that custom LWLock and wait events use a shmem state for
> monitoring purposes, where we are able to do ID->format name lookups
> as much as format->ID lookups.  Perhaps it's OK not to do that for
> COPY, but I am wondering if we'd better design things from scratch
> with states in shmem state knowing that COPY is a long-running
> operation, and that if one mixes multiple formats they would most
> likely want to know which formats are bottlenecks, through SQL.  Cloud
> providers would love that.

Good point. It would make sense to have such information as a map on
shmem. It might be better to use dshash here since a custom copy
format module can be loaded at runtime. Or we can use dynahash with
large enough elements.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  274. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2025-05-03T02:19:58Z

    Hi,

In <CAD21AoBuEqcz2_+dpA3WTiDUF=FgudPBKwM+nvH+qHT-k4p5mA@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Thu, 1 May 2025 12:15:30 -0700,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

> One of the primary considerations we need to address is the treatment
> of the specified format name. The current patch set utilizes built-in
> formats (namely 'csv', 'text', and 'binary') when the format name is
> either unqualified or explicitly specified with 'pg_catalog' as the
> schema. In all other cases, we search for custom format handler
> functions based on the search_path. To be frank, I have reservations
> about this interface design, as the dependence of the specified custom
> format name on the search_path could potentially confuse users.

How about requiring schema for all custom formats?

Valid:

  COPY ... TO ... (FORMAT 'text');
  COPY ... TO ... (FORMAT 'my_schema.jsonlines');

Invalid:

  COPY ... TO ... (FORMAT 'jsonlines'); -- no schema
  COPY ... TO ... (FORMAT 'pg_catalog.text'); -- needless schema

If we require "schema" for all custom formats, we don't need
to depend on search_path.

> In light of these concerns, I've been contemplating alternative
> interface designs. One promising approach would involve registering
> custom copy formats via a C function during module loading
> (specifically, in _PG_init()). This method would require extension
> authors to invoke a registration function, say
> RegisterCustomCopyFormat(), in _PG_init() as follows:
> 
> JsonLinesFormatId = RegisterCustomCopyFormat("jsonlines",
>                                              &JsonLinesCopyToRoutine,
>                                              &JsonLinesCopyFromRoutine);
> 
> The registration function would validate the format name and store it
> in TopMemoryContext. It would then return a unique identifier that can
> be used subsequently to reference the custom copy format extension.

I don't object the suggested interface because I don't have
a strong opinion how to implement this feature.

Why do we need to assign a unique ID? For performance? For
RegisterCustomCopyFormatOption()?

I think that we don't need to use it so much in COPY. We
don't need to use format name and assigned ID after we
retrieve a corresponding Copy{To,From}Routine. Because all
needed information are in Copy{To,From}Routine.

>          Extensions could register their own options within this
> framework, for example:
> 
> RegisterCustomCopyFormatOption(JsonLinesFormatId,
>     "custom_option",
>     custom_option_handler);

Can we defer to discuss how to add support for custom
options while we focus on the first implementation? Earlier
patch sets with the current approach had custom options
support but it's removed in the first implementation.

(BTW, I think that it's not a good API because we want COPY
FROM only options and COPY TO only options something like
"compression level".)

> This approach offers several advantages: it would eliminate the
> search_path issue, provide greater flexibility, and potentially
> simplify the overall interface for users and developers alike.

What contributes to the "flexibility"? Developers can call
multiple Register* functions in _PG_Init(), right?


Thanks,
-- 
kou

  275. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2025-05-03T02:24:06Z

    Hi,

In <CAD21AoB82+MoP_RJ=zzhO9KaHK4LbfGjORkre34C7g-xsCdegQ@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 2 May 2025 15:52:49 -0700,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

>> Hmm.  How much should we care about the observability of the COPY
>> format used by a given backend?  Storing this information in a
>> backend's TopMemoryContext is OK to get the extensibility basics to
>> work, but could it make sense to use some shmem state to allocate a
>> uint32 ID that could be shared by all backends.  Contrary to EXPLAIN,
>> COPY commands usually run for a very long time, so I am wondering if
>> these APIs should be designed so as it would be possible to monitor
>> the format used.  One layer where the format information could be made
>> available is the progress reporting view for COPY, for example.  I can
>> also imagine a pgstats kind where we do COPY stats aggregates, with a
>> per-format pgstats kind, and sharing a fixed ID across multiple
>> backends is relevant (when flushing the stats at shutdown, we would
>> use a name/ID mapping like replication slots).
>>
>> I don't think that this needs to be relevant for the option part, just
>> for the format where, I suspect, we should store in a shmem array
>> based on the ID allocated the name of the format, the library of the
>> callback and the function name fed to load_external_function().
>>
>> Note that custom LWLock and wait events use a shmem state for
>> monitoring purposes, where we are able to do ID->format name lookups
>> as much as format->ID lookups.  Perhaps it's OK not to do that for
>> COPY, but I am wondering if we'd better design things from scratch
>> with states in shmem state knowing that COPY is a long-running
>> operation, and that if one mixes multiple formats they would most
>> likely want to know which formats are bottlenecks, through SQL.  Cloud
>> providers would love that.
> 
> Good point. It would make sense to have such information as a map on
> shmem. It might be better to use dshash here since a custom copy
> format module can be loaded at runtime. Or we can use dynahash with
> large enough elements.

If we don't need to assign an ID for each format, can we
avoid it? If we implement it, is this approach more complex
than the current table sampling method like approach?


Thanks,
-- 
kou

  276. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2025-05-03T04:38:32Z

    On Fri, May 2, 2025 at 7:20 PM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <CAD21AoBuEqcz2_+dpA3WTiDUF=FgudPBKwM+nvH+qHT-k4p5mA@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Thu, 1 May 2025 12:15:30 -0700,
>   Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> > One of the primary considerations we need to address is the treatment
> > of the specified format name. The current patch set utilizes built-in
> > formats (namely 'csv', 'text', and 'binary') when the format name is
> > either unqualified or explicitly specified with 'pg_catalog' as the
> > schema. In all other cases, we search for custom format handler
> > functions based on the search_path. To be frank, I have reservations
> > about this interface design, as the dependence of the specified custom
> > format name on the search_path could potentially confuse users.
>
> How about requiring schema for all custom formats?
>
> Valid:
>
>   COPY ... TO ... (FORMAT 'text');
>   COPY ... TO ... (FORMAT 'my_schema.jsonlines');
>
> Invalid:
>
>   COPY ... TO ... (FORMAT 'jsonlines'); -- no schema
>   COPY ... TO ... (FORMAT 'pg_catalog.text'); -- needless schema
>
> If we require "schema" for all custom formats, we don't need
> to depend on search_path.

I'm concerned that users cannot use the same format name in the FORMAT
option depending on which schema the handler function is created.

>
> > In light of these concerns, I've been contemplating alternative
> > interface designs. One promising approach would involve registering
> > custom copy formats via a C function during module loading
> > (specifically, in _PG_init()). This method would require extension
> > authors to invoke a registration function, say
> > RegisterCustomCopyFormat(), in _PG_init() as follows:
> >
> > JsonLinesFormatId = RegisterCustomCopyFormat("jsonlines",
> >                                              &JsonLinesCopyToRoutine,
> >                                              &JsonLinesCopyFromRoutine);
> >
> > The registration function would validate the format name and store it
> > in TopMemoryContext. It would then return a unique identifier that can
> > be used subsequently to reference the custom copy format extension.
>
> I don't object the suggested interface because I don't have
> a strong opinion how to implement this feature.
>
> Why do we need to assign a unique ID? For performance? For
> RegisterCustomCopyFormatOption()?

I think it's required for monitoring purposes for example. For
instance, we can set the format ID in the progress information and the
progress view can fetch the format name by the ID so that users can
see what format is being used in the COPY command.

>
> >          Extensions could register their own options within this
> > framework, for example:
> >
> > RegisterCustomCopyFormatOption(JsonLinesFormatId,
> >     "custom_option",
> >     custom_option_handler);
>
> Can we defer to discuss how to add support for custom
> options while we focus on the first implementation? Earlier
> patch sets with the current approach had custom options
> support but it's removed in the first implementation.

I think we can skip the custom option patch for the first
implementation but still need to discuss how we will be able to
implement it to understand the big picture of this feature. Otherwise
we could end up going the wrong direction.

>
> (BTW, I think that it's not a good API because we want COPY
> FROM only options and COPY TO only options something like
> "compression level".)

Why does this matter in terms of API? I think that even with this API
we can pass is_from to the option handler function so that it
validates the option based on it.

>
> > This approach offers several advantages: it would eliminate the
> > search_path issue, provide greater flexibility, and potentially
> > simplify the overall interface for users and developers alike.
>
> What contributes to the "flexibility"? Developers can call
> multiple Register* functions in _PG_Init(), right?

I think that with a tablesample-like approach we need to do everything
based on one handler function and callbacks returned from it whereas
there is no such limitation with C API style.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  277. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2025-05-03T04:56:39Z

    Hi,

In <CAD21AoBGRFStdVbHUcxL0QB8wn92J3Sn-6x=RhsSMuhepRH0NQ@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 2 May 2025 21:38:32 -0700,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

>> How about requiring schema for all custom formats?
>>
>> Valid:
>>
>>   COPY ... TO ... (FORMAT 'text');
>>   COPY ... TO ... (FORMAT 'my_schema.jsonlines');
>>
>> Invalid:
>>
>>   COPY ... TO ... (FORMAT 'jsonlines'); -- no schema
>>   COPY ... TO ... (FORMAT 'pg_catalog.text'); -- needless schema
>>
>> If we require "schema" for all custom formats, we don't need
>> to depend on search_path.
> 
> I'm concerned that users cannot use the same format name in the FORMAT
> option depending on which schema the handler function is created.

I'm not sure that it's a problem or not. If users want to
use the same format name, they can install the handler
function to the same schema.

>> Why do we need to assign a unique ID? For performance? For
>> RegisterCustomCopyFormatOption()?
> 
> I think it's required for monitoring purposes for example. For
> instance, we can set the format ID in the progress information and the
> progress view can fetch the format name by the ID so that users can
> see what format is being used in the COPY command.

How about setting the format name instead of the format ID
in the progress information?

> I think we can skip the custom option patch for the first
> implementation but still need to discuss how we will be able to
> implement it to understand the big picture of this feature. Otherwise
> we could end up going the wrong direction.

I think that we don't need to discuss it deeply because we
have many options with this approach. We can call C
functions in _PG_Init(). I think that this feature will not
be a blocker of this approach.

>> (BTW, I think that it's not a good API because we want COPY
>> FROM only options and COPY TO only options something like
>> "compression level".)
> 
> Why does this matter in terms of API? I think that even with this API
> we can pass is_from to the option handler function so that it
> validates the option based on it.

If we choose the API, each custom format developer needs to
handle the case in handler function. For example, if we pass
information whether this option is only for TO to
PostgreSQL, ProcessCopyOptions() not handler functions can
handle it.

Anyway, I think that we don't need to discuss this deeply
for now.

>> What contributes to the "flexibility"? Developers can call
>> multiple Register* functions in _PG_Init(), right?
> 
> I think that with a tablesample-like approach we need to do everything
> based on one handler function and callbacks returned from it whereas
> there is no such limitation with C API style.

Thanks for clarifying it. It seems that my understanding is
correct.

I hope that the flexibility is needed flexibility and too
much flexibility doesn't introduce too much complexity.


Thanks,
-- 
kou

  278. Re: Make COPY format extendable: Extract COPY TO format implementations

    David G. Johnston <david.g.johnston@gmail.com> — 2025-05-03T05:36:06Z

    On Thursday, May 1, 2025, Masahiko Sawada <sawada.mshk@gmail.com> wrote:

>
> In light of these concerns, I've been contemplating alternative
> interface designs. One promising approach would involve registering
> custom copy formats via a C function during module loading
> (specifically, in _PG_init()). This method would require extension
> authors to invoke a registration function, say
> RegisterCustomCopyFormat(), in _PG_init() as follows:
>
> JsonLinesFormatId = RegisterCustomCopyFormat("jsonlines",
>                                              &JsonLinesCopyToRoutine,
>                                              &JsonLinesCopyFromRoutine);
>
> The registration function would validate the format name and store it
> in TopMemoryContext. It would then return a unique identifier that can
> be used subsequently to reference the custom copy format extension.
>

How does this fix the search_path concern?  Are query writers supposed to
put JsonLinesFormatId into their queries?  Or are you just prohibiting a
DBA from ever installing an extension that wants to register a format name
that is already registered so that no namespace is ever required?

ISTM accommodating a namespace for formats is required just like we do for
virtually every other named object in the system.  At least, if we want to
play nice with extension authors.  It doesn’t have to be within the
existing pg_proc scope, we can create a new scope if desired, but
abolishing it seems unwise.

It would be more consistent with established policy if we didn’t make
exceptions for text/csv/binary - if the DBA permits a text format to exist
in a different schema and that schema appears first in the search_path,
unqualified references to text would resolve to the non-core handler.  We
already protect ourselves with safe search_paths.  This is really no
different than if someone wanted to implement a now() function and people
are putting pg_catalog from of existing usage.  It’s the DBAs problem, not
ours.

David J.

  279. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2025-05-03T05:57:53Z

    On Fri, May 2, 2025 at 10:36 PM David G. Johnston
<david.g.johnston@gmail.com> wrote:
>
> On Thursday, May 1, 2025, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>>
>>
>> In light of these concerns, I've been contemplating alternative
>> interface designs. One promising approach would involve registering
>> custom copy formats via a C function during module loading
>> (specifically, in _PG_init()). This method would require extension
>> authors to invoke a registration function, say
>> RegisterCustomCopyFormat(), in _PG_init() as follows:
>>
>> JsonLinesFormatId = RegisterCustomCopyFormat("jsonlines",
>>                                              &JsonLinesCopyToRoutine,
>>                                              &JsonLinesCopyFromRoutine);
>>
>> The registration function would validate the format name and store it
>> in TopMemoryContext. It would then return a unique identifier that can
>> be used subsequently to reference the custom copy format extension.
>
>
> How does this fix the search_path concern?  Are query writers supposed to put JsonLinesFormatId into their queries?  Or are you just prohibiting a DBA from ever installing an extension that wants to register a format name that is already registered so that no namespace is ever required?

Users can specify "jsonlines", passed in the first argument to the
register function, to the COPY FORMAT option in this case.  While
JsonLinesFormatId is reserved for internal operations such as module
processing and monitoring, any attempt to load another custom COPY
format module named 'jsonlines' will result in an error.

> ISTM accommodating a namespace for formats is required just like we do for virtually every other named object in the system.  At least, if we want to play nice with extension authors.  It doesn’t have to be within the existing pg_proc scope, we can create a new scope if desired, but abolishing it seems unwise.
>
> It would be more consistent with established policy if we didn’t make exceptions for text/csv/binary - if the DBA permits a text format to exist in a different schema and that schema appears first in the search_path, unqualified references to text would resolve to the non-core handler.  We already protect ourselves with safe search_paths.  This is really no different than if someone wanted to implement a now() function and people are putting pg_catalog from of existing usage.  It’s the DBAs problem, not ours.

I'm concerned about allowing multiple 'text' format implementations
with identical names within the database, as this could lead to
considerable confusion. When users specify 'text', it would be more
logical to guarantee that the built-in 'text' format is consistently
used. This principle aligns with other customizable components, such
as custom resource managers, wait events, lightweight locks, and
custom scans. These components maintain their built-in data/types and
explicitly prevent the registration of duplicate names.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  280. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2025-05-03T06:02:25Z

    On Fri, May 2, 2025 at 9:56 PM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <CAD21AoBGRFStdVbHUcxL0QB8wn92J3Sn-6x=RhsSMuhepRH0NQ@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 2 May 2025 21:38:32 -0700,
>   Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> >> How about requiring schema for all custom formats?
> >>
> >> Valid:
> >>
> >>   COPY ... TO ... (FORMAT 'text');
> >>   COPY ... TO ... (FORMAT 'my_schema.jsonlines');
> >>
> >> Invalid:
> >>
> >>   COPY ... TO ... (FORMAT 'jsonlines'); -- no schema
> >>   COPY ... TO ... (FORMAT 'pg_catalog.text'); -- needless schema
> >>
> >> If we require "schema" for all custom formats, we don't need
> >> to depend on search_path.
> >
> > I'm concerned that users cannot use the same format name in the FORMAT
> > option depending on which schema the handler function is created.
>
> I'm not sure that it's a problem or not. If users want to
> use the same format name, they can install the handler
> function to the same schema.
>
> >> Why do we need to assign a unique ID? For performance? For
> >> RegisterCustomCopyFormatOption()?
> >
> > I think it's required for monitoring purposes for example. For
> > instance, we can set the format ID in the progress information and the
> > progress view can fetch the format name by the ID so that users can
> > see what format is being used in the COPY command.
>
> How about setting the format name instead of the format ID
> in the progress information?

The progress view can know only numbers. We need to extend the
progress view infrastructure so that we can pass other data types.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  281. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2025-05-03T06:20:46Z

    Hi,

In <CAD21AoDnY2fhC7tp7jpn24AuwkeW-0YjFEtZbEfPwg8YcH6bAw@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 2 May 2025 23:02:25 -0700,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

> The progress view can know only numbers. We need to extend the
> progress view infrastructure so that we can pass other data types.

Sorry. Could you tell me what APIs referred here?
pgstat_progress_*() functions in
src/include/utils/backend_progress.h?


Thanks,
-- 
kou

  282. Re: Make COPY format extendable: Extract COPY TO format implementations

    David G. Johnston <david.g.johnston@gmail.com> — 2025-05-03T06:37:36Z

    On Friday, May 2, 2025, Masahiko Sawada <sawada.mshk@gmail.com> wrote:

>
> I'm concerned about allowing multiple 'text' format implementations
> with identical names within the database, as this could lead to
> considerable confusion. When users specify 'text', it would be more
> logical to guarantee that the built-in 'text' format is consistently
> used.


Do you want to only give text/csv/binary this special treatment or also any
future format name we ever decide to implement in core.  If an extension
takes up “xml” and we try to do that in core do we fail an upgrade because
of the conflict, and make it impossible to actually use said extension?

This principle aligns with other customizable components, such
> as custom resource managers, wait events, lightweight locks, and
> custom scans. These components maintain their built-in data/types and
> explicitly prevent the registration of duplicate names.
>

I am totally lost on how any of those resemble this feature.

I’m all for registration to enable additional options and features - but am
against moving away from turning format into a namespaced identifier.  This
is a query-facing feature where namespaces are common and fundamentally
required.  I have some sympathy for the fact that until now one could not
prefix text/binary/csv with pg_catalog to be fully safe, but in reality
DBAs/query authors either put pg_catalog first in their search_path or make
an informed decision when they deviate.  That is the established precedent
relevant to this feature.  The power, and responsibility for education,
lies with the user.

David J.

  283. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2025-05-03T06:37:46Z

    On Fri, May 2, 2025 at 11:20 PM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <CAD21AoDnY2fhC7tp7jpn24AuwkeW-0YjFEtZbEfPwg8YcH6bAw@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 2 May 2025 23:02:25 -0700,
>   Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> > The progress view can know only numbers. We need to extend the
> > progress view infrastructure so that we can pass other data types.
>
> Sorry. Could you tell me what APIs referred here?
> pgstat_progress_*() functions in
> src/include/utils/backend_progress.h?

The progress information is stored in PgBackendStatus defined in
backend_status.h:

    /*
     * Command progress reporting.  Any command which wishes can advertise
     * that it is running by setting st_progress_command,
     * st_progress_command_target, and st_progress_param[].
     * st_progress_command_target should be the OID of the relation which the
     * command targets (we assume there's just one, as this is meant for
     * utility commands), but the meaning of each element in the
     * st_progress_param array is command-specific.
     */
    ProgressCommandType st_progress_command;
    Oid         st_progress_command_target;
    int64       st_progress_param[PGSTAT_NUM_PROGRESS_PARAM];

Then the progress view maps the numbers to the corresponding strings:

CREATE VIEW pg_stat_progress_copy AS
    SELECT
        S.pid AS pid, S.datid AS datid, D.datname AS datname,
        S.relid AS relid,
        CASE S.param5 WHEN 1 THEN 'COPY FROM'
                      WHEN 2 THEN 'COPY TO'
                      END AS command,
        CASE S.param6 WHEN 1 THEN 'FILE'
                      WHEN 2 THEN 'PROGRAM'
                      WHEN 3 THEN 'PIPE'
                      WHEN 4 THEN 'CALLBACK'
                      END AS "type",
        S.param1 AS bytes_processed,
        S.param2 AS bytes_total,
        S.param3 AS tuples_processed,
        S.param4 AS tuples_excluded,
        S.param7 AS tuples_skipped
    FROM pg_stat_get_progress_info('COPY') AS S
        LEFT JOIN pg_database D ON S.datid = D.oid;

So the idea is that the backend process sets the format ID somewhere
in st_progress_param, and then the progress view calls a SQL function,
say pg_stat_get_copy_format_name(), with the format ID that returns
the corresponding format name.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  284. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2025-05-03T07:54:36Z

    On Fri, May 2, 2025 at 11:37 PM David G. Johnston
<david.g.johnston@gmail.com> wrote:
>
> On Friday, May 2, 2025, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>>
>>
>> I'm concerned about allowing multiple 'text' format implementations
>> with identical names within the database, as this could lead to
>> considerable confusion. When users specify 'text', it would be more
>> logical to guarantee that the built-in 'text' format is consistently
>> used.
>
>
> Do you want to only give text/csv/binary this special treatment or also any future format name we ever decide to implement in core.  If an extension takes up “xml” and we try to do that in core do we fail an upgrade because of the conflict, and make it impossible to actually use said extension?

I guess that's an extension author's responsibility to upgrade its
extension so as to work with the new PostgreSQL version, or carefully
choose the format name. They can even name
'[extension_name].[format_name]' as a format name. Even with the
current patch design (i.e., search_path affects handler function
lookups), users would end up using the built-in 'xml' format without
notice after upgrade, no? I guess that could introduce another
problem.

I think that we need to ensure that if users specify text/csv/binary
the built-in formats are always used, to keep backward compatibility.

>
>> This principle aligns with other customizable components, such
>> as custom resource managers, wait events, lightweight locks, and
>> custom scans. These components maintain their built-in data/types and
>> explicitly prevent the registration of duplicate names.
>
>
> I am totally lost on how any of those resemble this feature.
>
> I’m all for registration to enable additional options and features - but am against moving away from turning format into a namespaced identifier.  This is a query-facing feature where namespaces are common and fundamentally required.

That's a fair concern. But isn't the format name ultimately just an
option value, but not like a database object? As I mentioned above, I
think we need to keep backward compatibility but treating the built-in
formats special seems inconsistent with common name resolution
behavior.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  285. Re: Make COPY format extendable: Extract COPY TO format implementations

    David G. Johnston <david.g.johnston@gmail.com> — 2025-05-03T14:42:08Z

    On Saturday, May 3, 2025, Masahiko Sawada <sawada.mshk@gmail.com> wrote:


> I think that we need to ensure that if users specify text/csv/binary
> the built-in formats are always used, to keep backward compatibility.


That was my original thinking, but it’s inconsistent with how functions
behave today.  We don’t promise that installing extensions won’t cause
existing code to change.


>
>
> > I’m all for registration to enable additional options and features - but
> am against moving away from turning format into a namespaced identifier.
> This is a query-facing feature where namespaces are common and
> fundamentally required.
>
> That's a fair concern. But isn't the format name ultimately just an
> option value, but not like a database object?


We get to decide that.  And deciding in favor of “extensible database
object in a namespace’ makes more sense - leveraging all that pre-existing
design to play more nicely with extensions and give DBAs control.  The SQL
command to add one is “create function” instead of “create copy format”.

David J.

  286. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2025-05-04T05:02:51Z

    On Sat, May 3, 2025 at 7:42 AM David G. Johnston
<david.g.johnston@gmail.com> wrote:
>
> On Saturday, May 3, 2025, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
>>
>> I think that we need to ensure that if users specify text/csv/binary
>> the built-in formats are always used, to keep backward compatibility.
>
>
> That was my original thinking, but it’s inconsistent with how functions behave today.  We don’t promise that installing extensions won’t cause existing code to change.

I'm skeptical about whether that's an acceptable backward
compatibility breakage.

>> > I’m all for registration to enable additional options and features - but am against moving away from turning format into a namespaced identifier.  This is a query-facing feature where namespaces are common and fundamentally required.
>>
>> That's a fair concern. But isn't the format name ultimately just an
>> option value, but not like a database object?
>
>
> We get to decide that.  And deciding in favor of “extensible database object in a namespace’ makes more sense - leveraging all that pre-existing design to play more nicely with extensions and give DBAs control.  The SQL command to add one is “create function” instead of “create copy format”.

I still don't fully understand why the FORMAT value alone needs to be
treated like a schema-qualified object. If the concern is about name
conflict with future built-in formats, I would argue that the same
concern applies to custom EXPLAIN options and logical decoding
plugins. To me, the benefit of treating the COPY FORMAT value as a
schema-qualified object seems limited. Meanwhile, the risk of not
protecting built-in formats like 'text', 'csv', and 'binary' is
significant. If those names can be shadowed by extension via
search_patch, we lose backward compatibility.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  287. Re: Make COPY format extendable: Extract COPY TO format implementations

    David G. Johnston <david.g.johnston@gmail.com> — 2025-05-04T05:27:36Z

    On Saturday, May 3, 2025, Masahiko Sawada <sawada.mshk@gmail.com> wrote:

> On Sat, May 3, 2025 at 7:42 AM David G. Johnston
> <david.g.johnston@gmail.com> wrote:
> >
> > On Saturday, May 3, 2025, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> >>
> >> I think that we need to ensure that if users specify text/csv/binary
> >> the built-in formats are always used, to keep backward compatibility.
> >
> >
> > That was my original thinking, but it’s inconsistent with how functions
> behave today.  We don’t promise that installing extensions won’t cause
> existing code to change.
>
> I'm skeptical about whether that's an acceptable backward
> compatibility breakage.


I’m skeptical you are correctly defining what backward-compatibility
requires.

Well, the only potential breakage is that we are searching for a matching
function by signature without first limiting the mandated return type.  But
that is solve-able should anyone else see the problem as well.

The global format name has its merits but neither it nor the namespaced
format option suffer from breaking compatibility or policy.


>
> I still don't fully understand why the FORMAT value alone needs to be
> treated like a schema-qualified object. If the concern is about name
> conflict with future built-in formats, I would argue that the same
> concern applies to custom EXPLAIN options and logical decoding
> plugins.


>
Then maybe we have the same “problem” in those places.


>
> To me, the benefit of treating the COPY FORMAT value as a
> schema-qualified object seems limited. Meanwhile, the risk of not
> protecting built-in formats like 'text', 'csv', and 'binary' is
> significant.


Really? You think lots of extensions are going to choose to use these
values even if they are permitted?  Or are you concerned about attack
surfaces?


> If those names can be shadowed by extension via
> search_patch, we lose backward compatibility.
>

This is not a definition of backward-compatibility that I am familiar with.

If anything the ability for a DBA to arrange for such shadowing would be a
feature enhancement.  They can drop-in a more efficient or desirable
implementation without having to change query code.

In any case, I’m doubtful either of us can make a convincing enough
argument to sway the other fully.  Both options are plausible, IMO.  Others
need to chime in.

David J.

  288. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2025-05-09T08:51:27Z

    Hi,

In <CAD21AoD9CBjh4u6jdiE0tG-jvejw-GJN8fUPoQSVhKh36HW2NQ@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 2 May 2025 23:37:46 -0700,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

> The progress information is stored in PgBackendStatus defined in
> backend_status.h:
> 
>     /*
>      * Command progress reporting.  Any command which wishes can advertise
>      * that it is running by setting st_progress_command,
>      * st_progress_command_target, and st_progress_param[].
>      * st_progress_command_target should be the OID of the relation which the
>      * command targets (we assume there's just one, as this is meant for
>      * utility commands), but the meaning of each element in the
>      * st_progress_param array is command-specific.
>      */
>     ProgressCommandType st_progress_command;
>     Oid         st_progress_command_target;
>     int64       st_progress_param[PGSTAT_NUM_PROGRESS_PARAM];
> 
> Then the progress view maps the numbers to the corresponding strings:
> 
> CREATE VIEW pg_stat_progress_copy AS
>     SELECT
>         S.pid AS pid, S.datid AS datid, D.datname AS datname,
>         S.relid AS relid,
>         CASE S.param5 WHEN 1 THEN 'COPY FROM'
>                       WHEN 2 THEN 'COPY TO'
>                       END AS command,
>         CASE S.param6 WHEN 1 THEN 'FILE'
>                       WHEN 2 THEN 'PROGRAM'
>                       WHEN 3 THEN 'PIPE'
>                       WHEN 4 THEN 'CALLBACK'
>                       END AS "type",
>         S.param1 AS bytes_processed,
>         S.param2 AS bytes_total,
>         S.param3 AS tuples_processed,
>         S.param4 AS tuples_excluded,
>         S.param7 AS tuples_skipped
>     FROM pg_stat_get_progress_info('COPY') AS S
>         LEFT JOIN pg_database D ON S.datid = D.oid;

Thanks. I didn't know about how to implement
pg_stat_progress_copy.

> So the idea is that the backend process sets the format ID somewhere
> in st_progress_param, and then the progress view calls a SQL function,
> say pg_stat_get_copy_format_name(), with the format ID that returns
> the corresponding format name.

Does it work when we use session_preload_libraries or the
LOAD command? If we have 2 sessions and both of them load
"jsonlines" COPY FORMAT extensions, what will be happened?

For example:

1. Session 1: Register "jsonlines"
2. Session 2: Register "jsonlines"
              (Should global format ID <-> format name mapping
              be updated?)
3. Session 2: Close this session.
              Unregister "jsonlines".
              (Can we unregister COPY FORMAT extension?)
              (Should global format ID <-> format name mapping
              be updated?)
4. Session 1: Close this session.
              Unregister "jsonlines".
              (Can we unregister COPY FORMAT extension?)
              (Should global format ID <-> format name mapping
              be updated?)

Thanks,
-- 
kou

  289. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2025-05-09T09:41:37Z

    Hi,

In <CAKFQuwaRDXANaL+QcT6LZRAem4rwkSwv9v+viv_mcR+Rex3quA@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Sat, 3 May 2025 22:27:36 -0700,
  "David G. Johnston" <david.g.johnston@gmail.com> wrote:

> In any case, I’m doubtful either of us can make a convincing enough
> argument to sway the other fully.  Both options are plausible, IMO.  Others
> need to chime in.

I may misunderstand but here is the current summary, right?

Proposed approaches to register custom COPY formats:
a. Create a function that has the same name of custom COPY
   format
b. Call a register function from _PG_init()

FYI: I proposed c. approach that uses a. but it always
requires schema name for format name in other e-mail.


Users can register the same format name:
a. Yes
   * Users can distinct the same format name by schema name
   * If format name doesn't have schema name, the used
     format depends on search_path
     * Pros:
       * Using schema for it is consistent with other
         PostgreSQL mechanisms
       * Custom format never conflict with built-in
         format. For example, an extension register "xml" and
         PostgreSQL adds "xml" later, they are never
         conflicted because PostgreSQL's "xml" is registered
         to pg_catalog.
     * Cons: Different format may be used with the same
       input. For example, "jsonlines" may choose
       "jsonlines" implemented by extension X or implemented
       by extension Y when search_path is different.
b. No
   * Users can use "${schema}.${name}" for format name
     that mimics PostgreSQL's builtin schema (but it's just
     a string)


Built-in formats (text/csv/binary) should be able to
overwritten by extensions:
a. (The current patch is no but David's answer is) Yes
   * Pros: Users can use drop-in replacement faster
     implementation without changing input
   * Cons: Users may overwrite them accidentally.
     It may break pg_dump result.
     (This is called as "backward incompatibility.")
b. No


Are there any missing or wrong items? If we can summarize
the current discussion here correctly, others will be able
to chime in this discussion. (At least I can do it.)


Thanks,
-- 
kou

  290. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2025-05-10T00:57:35Z

    On Fri, May 9, 2025 at 2:41 AM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <CAKFQuwaRDXANaL+QcT6LZRAem4rwkSwv9v+viv_mcR+Rex3quA@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Sat, 3 May 2025 22:27:36 -0700,
>   "David G. Johnston" <david.g.johnston@gmail.com> wrote:
>
> > In any case, I’m doubtful either of us can make a convincing enough
> > argument to sway the other fully.  Both options are plausible, IMO.  Others
> > need to chime in.
>
> I may misunderstand but here is the current summary, right?

Thank you for summarizing the discussion.

>
> Proposed approaches to register custom COPY formats:
> a. Create a function that has the same name of custom COPY
>    format
> b. Call a register function from _PG_init()
>
> FYI: I proposed c. approach that uses a. but it always
> requires schema name for format name in other e-mail.

With approach (c), do you mean that we require users to change all
FORMAT option values like from 'text' to 'pg_catalog.text' after the
upgrade? Or are we exempt the built-in formats?

>
> Users can register the same format name:
> a. Yes
>    * Users can distinct the same format name by schema name
>    * If format name doesn't have schema name, the used
>      format depends on search_path
>      * Pros:
>        * Using schema for it is consistent with other
>          PostgreSQL mechanisms
>        * Custom format never conflict with built-in
>          format. For example, an extension register "xml" and
>          PostgreSQL adds "xml" later, they are never
>          conflicted because PostgreSQL's "xml" is registered
>          to pg_catalog.
>      * Cons: Different format may be used with the same
>        input. For example, "jsonlines" may choose
>        "jsonlines" implemented by extension X or implemented
>        by extension Y when search_path is different.
> b. No
>    * Users can use "${schema}.${name}" for format name
>      that mimics PostgreSQL's builtin schema (but it's just
>      a string)
>
>
> Built-in formats (text/csv/binary) should be able to
> overwritten by extensions:
> a. (The current patch is no but David's answer is) Yes
>    * Pros: Users can use drop-in replacement faster
>      implementation without changing input
>    * Cons: Users may overwrite them accidentally.
>      It may break pg_dump result.
>      (This is called as "backward incompatibility.")
> b. No

The summary matches my understanding. I think the second point is
important. If we go with a tablesample-like API, I agree with David's
point that all FORMAT values including the built-in formats should
depend on the search_path value. While it provides a similar user
experience to other database objects, there is a possibility that a
COPY with built-in format could work differently on v19 than v18 or
earlier depending on the search_path value.

> Are there any missing or wrong items?

I think the approach (b) provides more flexibility than (a) in terms
of API design as with (a) we need to do everything based on one
handler function and callbacks.

> If we can summarize
> the current discussion here correctly, others will be able
> to chime in this discussion. (At least I can do it.)

+1

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  291. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2025-05-10T04:29:23Z

    On Fri, May 9, 2025 at 1:51 AM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <CAD21AoD9CBjh4u6jdiE0tG-jvejw-GJN8fUPoQSVhKh36HW2NQ@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 2 May 2025 23:37:46 -0700,
>   Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> > The progress information is stored in PgBackendStatus defined in
> > backend_status.h:
> >
> >     /*
> >      * Command progress reporting.  Any command which wishes can advertise
> >      * that it is running by setting st_progress_command,
> >      * st_progress_command_target, and st_progress_param[].
> >      * st_progress_command_target should be the OID of the relation which the
> >      * command targets (we assume there's just one, as this is meant for
> >      * utility commands), but the meaning of each element in the
> >      * st_progress_param array is command-specific.
> >      */
> >     ProgressCommandType st_progress_command;
> >     Oid         st_progress_command_target;
> >     int64       st_progress_param[PGSTAT_NUM_PROGRESS_PARAM];
> >
> > Then the progress view maps the numbers to the corresponding strings:
> >
> > CREATE VIEW pg_stat_progress_copy AS
> >     SELECT
> >         S.pid AS pid, S.datid AS datid, D.datname AS datname,
> >         S.relid AS relid,
> >         CASE S.param5 WHEN 1 THEN 'COPY FROM'
> >                       WHEN 2 THEN 'COPY TO'
> >                       END AS command,
> >         CASE S.param6 WHEN 1 THEN 'FILE'
> >                       WHEN 2 THEN 'PROGRAM'
> >                       WHEN 3 THEN 'PIPE'
> >                       WHEN 4 THEN 'CALLBACK'
> >                       END AS "type",
> >         S.param1 AS bytes_processed,
> >         S.param2 AS bytes_total,
> >         S.param3 AS tuples_processed,
> >         S.param4 AS tuples_excluded,
> >         S.param7 AS tuples_skipped
> >     FROM pg_stat_get_progress_info('COPY') AS S
> >         LEFT JOIN pg_database D ON S.datid = D.oid;
>
> Thanks. I didn't know about how to implement
> pg_stat_progress_copy.
>
> > So the idea is that the backend process sets the format ID somewhere
> > in st_progress_param, and then the progress view calls a SQL function,
> > say pg_stat_get_copy_format_name(), with the format ID that returns
> > the corresponding format name.
>
> Does it work when we use session_preload_libraries or the
> LOAD command? If we have 2 sessions and both of them load
> "jsonlines" COPY FORMAT extensions, what will be happened?
>
> For example:
>
> 1. Session 1: Register "jsonlines"
> 2. Session 2: Register "jsonlines"
>               (Should global format ID <-> format name mapping
>               be updated?)
> 3. Session 2: Close this session.
>               Unregister "jsonlines".
>               (Can we unregister COPY FORMAT extension?)
>               (Should global format ID <-> format name mapping
>               be updated?)
> 4. Session 1: Close this session.
>               Unregister "jsonlines".
>               (Can we unregister COPY FORMAT extension?)
>               (Should global format ID <-> format name mapping
>               be updated?)

I imagine that only for progress reporting purposes, I think session 1
and 2 can have different format IDs for the same 'jsonlines' if they
load it by LOAD command. They can advertise the format IDs on the
shmem and we can also provide a SQL function for the progress view
that can get the format name by the format ID.

Considering the possibility that we might want to use the format ID
also in the cumulative statistics, we might want to strictly provide
the unique format ID for each custom format as the format IDs are
serialized to the pgstat file. One possible way to implement it is
that we manage the custom format IDs in a wiki page like we do for
custom cumulative statistics and custom RMGR[1][2]. That is, a custom
format extension registers the format name along with the format ID
that is pre-registered in the wiki page or the format ID (e.g. 128)
indicating under development. If either the format name or format ID
conflict with an already registered custom format extension, the
registration function raises an error. And we preallocate enough
format IDs for built-in formats.

As for unregistration, I think that  even if we provide an
unregisteration API, it ultimately depends on whether or not custom
format extensions call it in _PG_fini().

Regards,

[1] https://wiki.postgresql.org/wiki/CustomCumulativeStats
[2] https://wiki.postgresql.org/wiki/CustomWALResourceManagers

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  292. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2025-05-26T01:04:05Z

    Hi,

In <CAD21AoBrSTmPyDai_QVR-XOe7PL722Dazm70A+FpvGy2hfSV9g@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 9 May 2025 17:57:35 -0700,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

>> Proposed approaches to register custom COPY formats:
>> a. Create a function that has the same name of custom COPY
>>    format
>> b. Call a register function from _PG_init()
>>
>> FYI: I proposed c. approach that uses a. but it always
>> requires schema name for format name in other e-mail.
> 
> With approach (c), do you mean that we require users to change all
> FORMAT option values like from 'text' to 'pg_catalog.text' after the
> upgrade? Or are we exempt the built-in formats?

The latter. 'text' must be accepted because existing pg_dump
results use 'text'. If we reject 'text', it's a big
incompatibility. (We can't dump on old PostgreSQL and
restore to new PostgreSQL.)


>> Users can register the same format name:
>> a. Yes
>>    * Users can distinct the same format name by schema name
>>    * If format name doesn't have schema name, the used
>>      format depends on search_path
>>      * Pros:
>>        * Using schema for it is consistent with other
>>          PostgreSQL mechanisms
>>        * Custom format never conflict with built-in
>>          format. For example, an extension register "xml" and
>>          PostgreSQL adds "xml" later, they are never
>>          conflicted because PostgreSQL's "xml" is registered
>>          to pg_catalog.
>>      * Cons: Different format may be used with the same
>>        input. For example, "jsonlines" may choose
>>        "jsonlines" implemented by extension X or implemented
>>        by extension Y when search_path is different.
>> b. No
>>    * Users can use "${schema}.${name}" for format name
>>      that mimics PostgreSQL's builtin schema (but it's just
>>      a string)
>>
>>
>> Built-in formats (text/csv/binary) should be able to
>> overwritten by extensions:
>> a. (The current patch is no but David's answer is) Yes
>>    * Pros: Users can use drop-in replacement faster
>>      implementation without changing input
>>    * Cons: Users may overwrite them accidentally.
>>      It may break pg_dump result.
>>      (This is called as "backward incompatibility.")
>> b. No
> 
> The summary matches my understanding. I think the second point is
> important. If we go with a tablesample-like API, I agree with David's
> point that all FORMAT values including the built-in formats should
> depend on the search_path value. While it provides a similar user
> experience to other database objects, there is a possibility that a
> COPY with built-in format could work differently on v19 than v18 or
> earlier depending on the search_path value.

Thanks for sharing additional points.

David said that the additional point case is a
responsibility or DBA not PostgreSQL, right?


As I already said, I don't have a strong opinion on which
approach is better. My opinion for the (important) second
point is no. I feel that the pros of a. isn't realistic. If
users want to improve text/csv/binary performance (or
something), they should improve PostgreSQL itself instead of
replacing it as an extension. (Or they should create another
custom copy format such as "faster_text" not "text".)


So I'm OK with the approach b.

>> Are there any missing or wrong items?
> 
> I think the approach (b) provides more flexibility than (a) in terms
> of API design as with (a) we need to do everything based on one
> handler function and callbacks.

Thanks for sharing this missing point.

I have a concern that the flexibility may introduce needless
complexity. If it's not a real concern, I'm OK with the
approach b.


>> If we can summarize
>> the current discussion here correctly, others will be able
>> to chime in this discussion. (At least I can do it.)
> 
> +1

Are there any more people who are interested in custom COPY
FORMAT implementation design? If no more people, let's
decide it by us.


Thanks,
-- 
kou

  293. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2025-05-26T01:27:20Z

    Hi,

In <CAD21AoAY_h-9nuhs14e3cyO_A2rH7==zuq+NPHkn9ggwyaXnPQ@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 9 May 2025 21:29:23 -0700,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

>> > So the idea is that the backend process sets the format ID somewhere
>> > in st_progress_param, and then the progress view calls a SQL function,
>> > say pg_stat_get_copy_format_name(), with the format ID that returns
>> > the corresponding format name.
>>
>> Does it work when we use session_preload_libraries or the
>> LOAD command? If we have 2 sessions and both of them load
>> "jsonlines" COPY FORMAT extensions, what will be happened?
>>
>> For example:
>>
>> 1. Session 1: Register "jsonlines"
>> 2. Session 2: Register "jsonlines"
>>               (Should global format ID <-> format name mapping
>>               be updated?)
>> 3. Session 2: Close this session.
>>               Unregister "jsonlines".
>>               (Can we unregister COPY FORMAT extension?)
>>               (Should global format ID <-> format name mapping
>>               be updated?)
>> 4. Session 1: Close this session.
>>               Unregister "jsonlines".
>>               (Can we unregister COPY FORMAT extension?)
>>               (Should global format ID <-> format name mapping
>>               be updated?)
> 
> I imagine that only for progress reporting purposes, I think session 1
> and 2 can have different format IDs for the same 'jsonlines' if they
> load it by LOAD command. They can advertise the format IDs on the
> shmem and we can also provide a SQL function for the progress view
> that can get the format name by the format ID.
> 
> Considering the possibility that we might want to use the format ID
> also in the cumulative statistics, we might want to strictly provide
> the unique format ID for each custom format as the format IDs are
> serialized to the pgstat file. One possible way to implement it is
> that we manage the custom format IDs in a wiki page like we do for
> custom cumulative statistics and custom RMGR[1][2]. That is, a custom
> format extension registers the format name along with the format ID
> that is pre-registered in the wiki page or the format ID (e.g. 128)
> indicating under development. If either the format name or format ID
> conflict with an already registered custom format extension, the
> registration function raises an error. And we preallocate enough
> format IDs for built-in formats.
> 
> As for unregistration, I think that  even if we provide an
> unregisteration API, it ultimately depends on whether or not custom
> format extensions call it in _PG_fini().

Thanks for sharing your idea.

With the former ID issuing approach, it seems that we need a
global format ID <-> name mapping and a per session
registered format name list. The custom COPY FORMAT register
function rejects the same format name, right? If we support
both of shared_preload_libraries and
session_preload_libraries/LOAD, we have different life time
custom formats. It may introduce a complexity with the ID
issuing approach.

With the latter static ID approach, how to implement a
function that converts format ID to format name? PostgreSQL
itself doesn't know ID <-> name mapping in the Wiki page. It
seems that custom COPY FORMAT implementation needs to
register its name to PostgreSQL by itself.


Thanks,
-- 
kou

  294. Re: Make COPY format extendable: Extract COPY TO format implementations

    Michael Paquier <michael@paquier.xyz> — 2025-06-12T02:33:52Z

    On Mon, May 26, 2025 at 10:04:05AM +0900, Sutou Kouhei wrote:
> As I already said, I don't have a strong opinion on which
> approach is better. My opinion for the (important) second
> point is no. I feel that the pros of a. isn't realistic. If
> users want to improve text/csv/binary performance (or
> something), they should improve PostgreSQL itself instead of
> replacing it as an extension. (Or they should create another
> custom copy format such as "faster_text" not "text".)

Patches welcome.  Andres may have a TODO board regarding that, I
think.

> So I'm OK with the approach b.

Here is an opinion.

Approach (b), that uses _PG_init() a function to register a custom
format has the merit to be simple to implement and secure by "design",
because it depends only on the fact that we can do a lookup based on
the string defined in one or more DefElems.  Adding a dependendy to
search_path as you say could lead to surprising results. 

Using a shared ID when a COPY method is registered (like extension
wait events) or an ID that's static in a backend (like EXPLAIN
extensibility does) is an implementation difference that can be useful
for monitoring, and only that AFAIK.  If you want to implement
method-based statistics for COPY, you will want to allocate one stats
kind for each COPY method, because the stats stored will be aggregates
of the COPY methods.  The stats kind ID is something that should not
be linked to the COPY method ID, because the stats kind ID is
registered in its own dedicated path, and it would be hardcoded in the
library where the COPY callbacks are defined.  So you could have a
stats kind with a fixed ID, and a COPY method ID that's linked to each
backend like EXPLAIN does.

One factor to take into account is how much freedom we are OK with
giving to users when it comes to the deployment of custom COPY
methods, and how popular these would be.  Cloud is popular these days,
so folks may want to be able to define pointers to functions that are
run in something else than C, as long as the language is trusted.  My
take on this part is that we are not going to see many formats out
there that would benefit from these callbacks, so asking for people to
deploy a .so on disk that can only be LOAD'ed or registered with one
of the preloading GUCs should be enough to satisfy most users, even if
the barrier entry to get that only a cloud instead like RDS or Azure
is higher.  This has also the benefit in giving more control on the
COPY internals to cloud providers, as they are the ones who would be
in charge of saying if they're OK with a dedicated .so or not.  Not
the users themselves.  We've had a lot of bad PR and false CVEs in the
past with COPY FROM/TO PROGRAM and the fact that it requires
superusers.  Having something in this area that gives more freedom to
the user with something like approach (a) (SQL functions allowed to
define the callback) will, I suspect, bite us back hard.

So, my opinion is to rely on _PG_init(), with a shared ID if you want
to expose the method used somewhere for monitoring tools.  You could
as well implement the simpler set of APIs that allocates IDs local to
each backend, like EXPLAIN, then consider later if shared IDs are
really needed.  The registration APIs don't have to be fixed in time
across releases, they can be always improved in steps as required.
What matters is ABI compatibility in the same major version once it is
released.
--
Michael

  295. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2025-06-12T17:00:12Z

    On Wed, Jun 11, 2025 at 7:34 PM Michael Paquier <michael@paquier.xyz> wrote:
>
> On Mon, May 26, 2025 at 10:04:05AM +0900, Sutou Kouhei wrote:
> > As I already said, I don't have a strong opinion on which
> > approach is better. My opinion for the (important) second
> > point is no. I feel that the pros of a. isn't realistic. If
> > users want to improve text/csv/binary performance (or
> > something), they should improve PostgreSQL itself instead of
> > replacing it as an extension. (Or they should create another
> > custom copy format such as "faster_text" not "text".)
>
> Patches welcome.  Andres may have a TODO board regarding that, I
> think.
>
> > So I'm OK with the approach b.
>
> Here is an opinion.

Thank you for the comments.

>
> Approach (b), that uses _PG_init() a function to register a custom
> format has the merit to be simple to implement and secure by "design",
> because it depends only on the fact that we can do a lookup based on
> the string defined in one or more DefElems.  Adding a dependendy to
> search_path as you say could lead to surprising results.
>
> Using a shared ID when a COPY method is registered (like extension
> wait events) or an ID that's static in a backend (like EXPLAIN
> extensibility does) is an implementation difference that can be useful
> for monitoring, and only that AFAIK.  If you want to implement
> method-based statistics for COPY, you will want to allocate one stats
> kind for each COPY method, because the stats stored will be aggregates
> of the COPY methods.  The stats kind ID is something that should not
> be linked to the COPY method ID, because the stats kind ID is
> registered in its own dedicated path, and it would be hardcoded in the
> library where the COPY callbacks are defined.  So you could have a
> stats kind with a fixed ID, and a COPY method ID that's linked to each
> backend like EXPLAIN does.

Good point.

>
> One factor to take into account is how much freedom we are OK with
> giving to users when it comes to the deployment of custom COPY
> methods, and how popular these would be.  Cloud is popular these days,
> so folks may want to be able to define pointers to functions that are
> run in something else than C, as long as the language is trusted.  My
> take on this part is that we are not going to see many formats out
> there that would benefit from these callbacks, so asking for people to
> deploy a .so on disk that can only be LOAD'ed or registered with one
> of the preloading GUCs should be enough to satisfy most users, even if
> the barrier entry to get that only a cloud instead like RDS or Azure
> is higher.  This has also the benefit in giving more control on the
> COPY internals to cloud providers, as they are the ones who would be
> in charge of saying if they're OK with a dedicated .so or not.  Not
> the users themselves.  We've had a lot of bad PR and false CVEs in the
> past with COPY FROM/TO PROGRAM and the fact that it requires
> superusers.  Having something in this area that gives more freedom to
> the user with something like approach (a) (SQL functions allowed to
> define the callback) will, I suspect, bite us back hard.

That's a valid point and I agree.

>
> So, my opinion is to rely on _PG_init(), with a shared ID if you want
> to expose the method used somewhere for monitoring tools.  You could
> as well implement the simpler set of APIs that allocates IDs local to
> each backend, like EXPLAIN, then consider later if shared IDs are
> really needed.  The registration APIs don't have to be fixed in time
> across releases, they can be always improved in steps as required.
> What matters is ABI compatibility in the same major version once it is
> released.

+1 to start with a simpler set of APIs.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  296. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2025-06-16T23:50:37Z

    Hi,

In <CAD21AoBwxgfkMYxgPWyrLG-r8-ptVKjd=jhncY_QAaVJYhQQdw@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Thu, 12 Jun 2025 10:00:12 -0700,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

>> So, my opinion is to rely on _PG_init(), with a shared ID if you want
>> to expose the method used somewhere for monitoring tools.  You could
>> as well implement the simpler set of APIs that allocates IDs local to
>> each backend, like EXPLAIN, then consider later if shared IDs are
>> really needed.  The registration APIs don't have to be fixed in time
>> across releases, they can be always improved in steps as required.
>> What matters is ABI compatibility in the same major version once it is
>> released.
> 
> +1 to start with a simpler set of APIs.

OK. I'll implement the initial version with this
design. (Allocating IDs local not shared.)


Thanks,
-- 
kou

  297. Re: Make COPY format extendable: Extract COPY TO format implementations

    Michael Paquier <michael@paquier.xyz> — 2025-06-17T00:38:54Z

    On Tue, Jun 17, 2025 at 08:50:37AM +0900, Sutou Kouhei wrote:
> OK. I'll implement the initial version with this
> design. (Allocating IDs local not shared.)

Sounds good to me.  Thanks Sutou-san!
--
Michael

  298. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2025-06-18T03:59:20Z

    Hi,

In <aFC5HmZHU5NCPuTL@paquier.xyz>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Tue, 17 Jun 2025 09:38:54 +0900,
  Michael Paquier <michael@paquier.xyz> wrote:

> On Tue, Jun 17, 2025 at 08:50:37AM +0900, Sutou Kouhei wrote:
>> OK. I'll implement the initial version with this
>> design. (Allocating IDs local not shared.)
> 
> Sounds good to me.  Thanks Sutou-san!

I've attached the v41 patch set that uses the C API approach
with local (not shared) COPY routine management.

0001: This is same as 0001 in the v40 patch set. It just
      cleans up CopySource and CopyDest enums.
0002: This is the initial version of this approach.

Here are some discussion points:

1. This provides 2 registration APIs
   (RegisterCopy{From,To}Routine(name, routine)) instead of
   1 registration API (RegisterCopyFormat(name,
   from_routine, to_routine)).

   It's for simple implementation and easy to extend without
   breaking APIs in the future. (And some formats may
   provide only FROM routine or TO routine.)

   Is this design acceptable?

   FYI: RegisterCopy{From,To}Routine() uses the same logic
   as RegisterExtensionExplainOption().

2. This allocates IDs internally but doesn't provide APIs
   that get them. Because it's not needed for now.

   We can provide GetExplainExtensionId() like API when we
   need it.
        
   Is this design acceptable?

3. I want to register the built-in COPY {FROM,TO} routines
   in the PostgreSQL initialization phase. Where should we
   do it? In 0002, it's done in InitPostgres() but I'm not
   sure whether it's a suitable location or not.

4. 0002 adds CopyFormatOptions::routine as union:

   @@ -87,9 +91,14 @@ typedef struct CopyFormatOptions
           CopyLogVerbosityChoice log_verbosity;   /* verbosity of logged messages */
           int64           reject_limit;   /* maximum tolerable number of errors */
           List       *convert_select; /* list of column names (can be NIL) */
   +       union
   +       {
   +               const struct CopyFromRoutine *from; /* for COPY FROM */
   +               const struct CopyToRoutine *to; /* for COPY TO */
   +       }                       routine;                /* routine to process the specified format */
    } CopyFormatOptions;

   Because one of Copy{From,To}Routine is only needed at
   once. Is this union usage strange in PostgreSQL?

5. 0002 adds InitializeCopy{From,To}Routines() and
   GetCopy{From,To}Routine() that aren't used by COPY
   {FROM,TO} routine implementations to copyapi.h. Should we
   move them to other .h? If so, which .h should be used for
   them?


Thanks,
-- 
kou

  299. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2025-06-24T02:59:17Z

    On Wed, Jun 18, 2025 at 12:59 PM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <aFC5HmZHU5NCPuTL@paquier.xyz>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Tue, 17 Jun 2025 09:38:54 +0900,
>   Michael Paquier <michael@paquier.xyz> wrote:
>
> > On Tue, Jun 17, 2025 at 08:50:37AM +0900, Sutou Kouhei wrote:
> >> OK. I'll implement the initial version with this
> >> design. (Allocating IDs local not shared.)
> >
> > Sounds good to me.  Thanks Sutou-san!
>
> I've attached the v41 patch set that uses the C API approach
> with local (not shared) COPY routine management.
>
> 0001: This is same as 0001 in the v40 patch set. It just
>       cleans up CopySource and CopyDest enums.
> 0002: This is the initial version of this approach.

Thank you for updating the patches!

> Here are some discussion points:
>
> 1. This provides 2 registration APIs
>    (RegisterCopy{From,To}Routine(name, routine)) instead of
>    1 registration API (RegisterCopyFormat(name,
>    from_routine, to_routine)).
>
>    It's for simple implementation and easy to extend without
>    breaking APIs in the future. (And some formats may
>    provide only FROM routine or TO routine.)
>
>    Is this design acceptable?

With the single registration API idea, we can register the custom
format routine that supports only FROM routine using the API like:

RegisterCopyRoutine("new-format", NewFormatFromRoutine, NULL);

Compared to this approach, what points do you think having separate
two registration APIs is preferable in terms of extendability and API
compatibility? I think it would be rather confusing for example if
each COPY TO routine and COPY FROM routine is registered by different
extensions with the same format name.

>    FYI: RegisterCopy{From,To}Routine() uses the same logic
>    as RegisterExtensionExplainOption().

I'm concerned that the patch has duplicated logics for the
registration of COPY FROM and COPY TO.

>
> 2. This allocates IDs internally but doesn't provide APIs
>    that get them. Because it's not needed for now.
>
>    We can provide GetExplainExtensionId() like API when we
>    need it.
>
>    Is this design acceptable?

+1

>
> 3. I want to register the built-in COPY {FROM,TO} routines
>    in the PostgreSQL initialization phase. Where should we
>    do it? In 0002, it's done in InitPostgres() but I'm not
>    sure whether it's a suitable location or not.

InitPostgres() is not a correct function as it's a process
initialization function. Probably we don't necessarily need to
register the built-in formats in the same way as custom formats. A
simple solution would be to have separate arrays for built-in formats
and custom formats and have the GetCopy[To|From]Routine() search both
arrays (built-in array first).

> 4. 0002 adds CopyFormatOptions::routine as union:
>
>    @@ -87,9 +91,14 @@ typedef struct CopyFormatOptions
>            CopyLogVerbosityChoice log_verbosity;   /* verbosity of logged messages */
>            int64           reject_limit;   /* maximum tolerable number of errors */
>            List       *convert_select; /* list of column names (can be NIL) */
>    +       union
>    +       {
>    +               const struct CopyFromRoutine *from; /* for COPY FROM */
>    +               const struct CopyToRoutine *to; /* for COPY TO */
>    +       }                       routine;                /* routine to process the specified format */
>     } CopyFormatOptions;
>
>    Because one of Copy{From,To}Routine is only needed at
>    once. Is this union usage strange in PostgreSQL?

I think we can live with having two fields as there are other options
that are used only in either COPY FROM or COPY TO.

>
> 5. 0002 adds InitializeCopy{From,To}Routines() and
>    GetCopy{From,To}Routine() that aren't used by COPY
>    {FROM,TO} routine implementations to copyapi.h. Should we
>    move them to other .h? If so, which .h should be used for
>    them?

As I commented at 3, I think it's better to avoid dynamically
registering the built-in formats.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  300. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2025-06-24T05:11:50Z

    Hi,

In <CAD21AoA57owo6qYTPTxOtCjDmcuj1tGL1aGs95cvEnoLQvwF0A@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Tue, 24 Jun 2025 11:59:17 +0900,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

>> 1. This provides 2 registration APIs
>>    (RegisterCopy{From,To}Routine(name, routine)) instead of
>>    1 registration API (RegisterCopyFormat(name,
>>    from_routine, to_routine)).
>>
>>    It's for simple implementation and easy to extend without
>>    breaking APIs in the future. (And some formats may
>>    provide only FROM routine or TO routine.)
>>
>>    Is this design acceptable?
> 
> With the single registration API idea, we can register the custom
> format routine that supports only FROM routine using the API like:
> 
> RegisterCopyRoutine("new-format", NewFormatFromRoutine, NULL);
> 
> Compared to this approach, what points do you think having separate
> two registration APIs is preferable in terms of extendability and API
> compatibility?

It's natural to add more related APIs with this
approach. The single registration API provides one feature
by one operation. If we use the RegisterCopyRoutine() for
FROM and TO formats API, it's not natural that we add more
related APIs. In this case, some APIs may provide multiple
features by one operation and other APIs may provide single
feature by one operation. Developers may be confused with
the API. For example, developers may think "what does mean
NULL here?" or "can we use NULL here?" for
"RegisterCopyRoutine("new-format", NewFormatFromRoutine,
NULL)".


>                I think it would be rather confusing for example if
> each COPY TO routine and COPY FROM routine is registered by different
> extensions with the same format name.

Hmm, I don't think so. Who is confused by the case? DBA?
Users who use COPY? Why is it confused?

Do you assume the case that the same name is used for
different format? For example, "json" is used for JSON Lines
for COPY FROM and and normal JSON for COPY TO by different
extensions.

>>    FYI: RegisterCopy{From,To}Routine() uses the same logic
>>    as RegisterExtensionExplainOption().
> 
> I'm concerned that the patch has duplicated logics for the
> registration of COPY FROM and COPY TO.

We can implement a convenient routine that can be used for
RegisterExtensionExplainOption() and
RegisterCopy{From,To}Routine() if it's needed.

>> 3. I want to register the built-in COPY {FROM,TO} routines
>>    in the PostgreSQL initialization phase. Where should we
>>    do it? In 0002, it's done in InitPostgres() but I'm not
>>    sure whether it's a suitable location or not.
> 
> InitPostgres() is not a correct function as it's a process
> initialization function. Probably we don't necessarily need to
> register the built-in formats in the same way as custom formats. A
> simple solution would be to have separate arrays for built-in formats
> and custom formats and have the GetCopy[To|From]Routine() search both
> arrays (built-in array first).

We had a discussion that we should dog-food APIs:

https://www.postgresql.org/message-id/flat/CAKFQuwaCHhrS%2BRE4p_OO6d7WEskd9b86-2cYcvChNkrP%2B7PJ7A%40mail.gmail.com#e6d1cdd04dac53eafe34b784ac47b68b

> We should (and usually do) dog-food APIs when reasonable
> and this situation seems quite reasonable.

In this case, we don't need to dog-food APIs, right?

>> 4. 0002 adds CopyFormatOptions::routine as union:
>>
>>    @@ -87,9 +91,14 @@ typedef struct CopyFormatOptions
>>            CopyLogVerbosityChoice log_verbosity;   /* verbosity of logged messages */
>>            int64           reject_limit;   /* maximum tolerable number of errors */
>>            List       *convert_select; /* list of column names (can be NIL) */
>>    +       union
>>    +       {
>>    +               const struct CopyFromRoutine *from; /* for COPY FROM */
>>    +               const struct CopyToRoutine *to; /* for COPY TO */
>>    +       }                       routine;                /* routine to process the specified format */
>>     } CopyFormatOptions;
>>
>>    Because one of Copy{From,To}Routine is only needed at
>>    once. Is this union usage strange in PostgreSQL?
> 
> I think we can live with having two fields as there are other options
> that are used only in either COPY FROM or COPY TO.

OK.


Thanks,
-- 
kou

  301. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2025-06-24T06:24:23Z

    On Tue, Jun 24, 2025 at 2:11 PM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <CAD21AoA57owo6qYTPTxOtCjDmcuj1tGL1aGs95cvEnoLQvwF0A@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Tue, 24 Jun 2025 11:59:17 +0900,
>   Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> >> 1. This provides 2 registration APIs
> >>    (RegisterCopy{From,To}Routine(name, routine)) instead of
> >>    1 registration API (RegisterCopyFormat(name,
> >>    from_routine, to_routine)).
> >>
> >>    It's for simple implementation and easy to extend without
> >>    breaking APIs in the future. (And some formats may
> >>    provide only FROM routine or TO routine.)
> >>
> >>    Is this design acceptable?
> >
> > With the single registration API idea, we can register the custom
> > format routine that supports only FROM routine using the API like:
> >
> > RegisterCopyRoutine("new-format", NewFormatFromRoutine, NULL);
> >
> > Compared to this approach, what points do you think having separate
> > two registration APIs is preferable in terms of extendability and API
> > compatibility?
>
> It's natural to add more related APIs with this
> approach. The single registration API provides one feature
> by one operation. If we use the RegisterCopyRoutine() for
> FROM and TO formats API, it's not natural that we add more
> related APIs. In this case, some APIs may provide multiple
> features by one operation and other APIs may provide single
> feature by one operation. Developers may be confused with
> the API. For example, developers may think "what does mean
> NULL here?" or "can we use NULL here?" for
> "RegisterCopyRoutine("new-format", NewFormatFromRoutine,
> NULL)".

We can document it in the comment for the registration function.

>
>
> >                I think it would be rather confusing for example if
> > each COPY TO routine and COPY FROM routine is registered by different
> > extensions with the same format name.
>
> Hmm, I don't think so. Who is confused by the case? DBA?
> Users who use COPY? Why is it confused?
>
> Do you assume the case that the same name is used for
> different format? For example, "json" is used for JSON Lines
> for COPY FROM and and normal JSON for COPY TO by different
> extensions.

Suppose that extension-A implements only CopyToRoutine for the
custom-format-X with the format name 'myformat' and extension-B
implements only CopyFromRoutine for the custom-format-Y with the same
name, if users load both extension-A and extension-B, it seems to me
that extension-A registers the custom-format-X format as 'myformat'
only with CopyToRoutine, and extension-B overwrites the 'myformat'
registration by adding custom-format-Y's CopyFromRoutine. However, if
users register extension-C that implements both routines with the
format name 'myformat', they can register neither extension-A nor
extension-B, which seems to me that we don't allow overwriting the
registration in this case.

I think the core issue appears to be the internal management of custom
format entries but the current patch does enable registration
overwriting in the former case (extension-A and extension-B case).

>
> >>    FYI: RegisterCopy{From,To}Routine() uses the same logic
> >>    as RegisterExtensionExplainOption().
> >
> > I'm concerned that the patch has duplicated logics for the
> > registration of COPY FROM and COPY TO.
>
> We can implement a convenient routine that can be used for
> RegisterExtensionExplainOption() and
> RegisterCopy{From,To}Routine() if it's needed.

I meant there are duplicated codes in COPY FROM and COPY TO. For
instance, RegisterCopyFromRoutine() and RegisterCopyToRoutine() have
the same logic.

>
> >> 3. I want to register the built-in COPY {FROM,TO} routines
> >>    in the PostgreSQL initialization phase. Where should we
> >>    do it? In 0002, it's done in InitPostgres() but I'm not
> >>    sure whether it's a suitable location or not.
> >
> > InitPostgres() is not a correct function as it's a process
> > initialization function. Probably we don't necessarily need to
> > register the built-in formats in the same way as custom formats. A
> > simple solution would be to have separate arrays for built-in formats
> > and custom formats and have the GetCopy[To|From]Routine() search both
> > arrays (built-in array first).
>
> We had a discussion that we should dog-food APIs:
>
> https://www.postgresql.org/message-id/flat/CAKFQuwaCHhrS%2BRE4p_OO6d7WEskd9b86-2cYcvChNkrP%2B7PJ7A%40mail.gmail.com#e6d1cdd04dac53eafe34b784ac47b68b
>
> > We should (and usually do) dog-food APIs when reasonable
> > and this situation seems quite reasonable.
>
> In this case, we don't need to dog-food APIs, right?

Yes, I think so.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  302. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2025-06-24T07:10:09Z

    Hi,

In <CAD21AoC8-d=GF-hOvGqUyq2xFg=QGpYfCiWJbcp4wcn0UidrPw@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Tue, 24 Jun 2025 15:24:23 +0900,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

>> It's natural to add more related APIs with this
>> approach. The single registration API provides one feature
>> by one operation. If we use the RegisterCopyRoutine() for
>> FROM and TO formats API, it's not natural that we add more
>> related APIs. In this case, some APIs may provide multiple
>> features by one operation and other APIs may provide single
>> feature by one operation. Developers may be confused with
>> the API. For example, developers may think "what does mean
>> NULL here?" or "can we use NULL here?" for
>> "RegisterCopyRoutine("new-format", NewFormatFromRoutine,
>> NULL)".
> 
> We can document it in the comment for the registration function.

I think that API that can be understandable without the
additional note is better API than API that needs some
notes.

Why do you suggest the RegisterCopyRoutine("new-format",
NewFormatFromRoutine, NewFormatToRoutine) API? You want to
remove the duplicated codes in
RegisterCopy{From,To}Routine(), right? I think that we can
do it by creating a convenient function that has the
duplicated codes extracted from
RegisterCopy{From,To}Routine() and
RegisterExtensionExplainOption().

BTW, what do you think about my answer (one feature by one
operation API is more extendable API) for your question
(extendability and API compatibility)?

> Suppose that extension-A implements only CopyToRoutine for the
> custom-format-X with the format name 'myformat' and extension-B
> implements only CopyFromRoutine for the custom-format-Y with the same
> name, if users load both extension-A and extension-B, it seems to me
> that extension-A registers the custom-format-X format as 'myformat'
> only with CopyToRoutine, and extension-B overwrites the 'myformat'
> registration by adding custom-format-Y's CopyFromRoutine. However, if
> users register extension-C that implements both routines with the
> format name 'myformat', they can register neither extension-A nor
> extension-B, which seems to me that we don't allow overwriting the
> registration in this case.

Do you assume that users use extension-A, extension-B and
extension-C without reading their documentation? If users
read their documentation before users use them, users can
know all of them use the same format name 'myformat' and
which extension provides Copy{From,To}Routine.

In this case, these users (who don't read documentation)
will be confused with the RegisterCopyRoutine("new-format",
NewFormatFromRoutine, NewFormatToRoutine) API too. Do we
really need to care about this case?

> I think the core issue appears to be the internal management of custom
> format entries but the current patch does enable registration
> overwriting in the former case (extension-A and extension-B case).

This is the same behavior as existing custom EXPLAIN option
implementation. Should we use different behavior here?

>> >>    FYI: RegisterCopy{From,To}Routine() uses the same logic
>> >>    as RegisterExtensionExplainOption().
>> >
>> > I'm concerned that the patch has duplicated logics for the
>> > registration of COPY FROM and COPY TO.
>>
>> We can implement a convenient routine that can be used for
>> RegisterExtensionExplainOption() and
>> RegisterCopy{From,To}Routine() if it's needed.
> 
> I meant there are duplicated codes in COPY FROM and COPY TO. For
> instance, RegisterCopyFromRoutine() and RegisterCopyToRoutine() have
> the same logic.

Yes, I understand it. I wanted to say that we can remove the
duplicated codes by introducing a RegisterSomething()
function that can be used by
RegisterExtensionExplainOption() and
RegisterCopy{From,To}Routine():

void
RegisterSomething(...)
{
  /* Common codes in RegisterExtensionExplainOption() and
     RegisterCopy{From,To}Routine()
     ...
   */
}

void
RegisterExtensionExplainOption(...)
{
  RegisterSomething(...);
}

void
RegisterCopyFromRoutine(...)
{
  RegisterSomething(...);
}

void
RegisterCopyToRoutine(...)
{
  RegisterSomething(...);
}

You think that this approach can't remove the duplicated
codes, right?

>> > InitPostgres() is not a correct function as it's a process
>> > initialization function. Probably we don't necessarily need to
>> > register the built-in formats in the same way as custom formats. A
>> > simple solution would be to have separate arrays for built-in formats
>> > and custom formats and have the GetCopy[To|From]Routine() search both
>> > arrays (built-in array first).
>>
>> We had a discussion that we should dog-food APIs:
>>
>> https://www.postgresql.org/message-id/flat/CAKFQuwaCHhrS%2BRE4p_OO6d7WEskd9b86-2cYcvChNkrP%2B7PJ7A%40mail.gmail.com#e6d1cdd04dac53eafe34b784ac47b68b
>>
>> > We should (and usually do) dog-food APIs when reasonable
>> > and this situation seems quite reasonable.
>>
>> In this case, we don't need to dog-food APIs, right?
> 
> Yes, I think so.

OK. I don't have a strong opinion for it. If nobody objects
it, I'll do it when I update the patch set.


Thanks,
-- 
kou

  303. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2025-06-24T15:48:46Z

    On Tue, Jun 24, 2025 at 4:10 PM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <CAD21AoC8-d=GF-hOvGqUyq2xFg=QGpYfCiWJbcp4wcn0UidrPw@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Tue, 24 Jun 2025 15:24:23 +0900,
>   Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> >> It's natural to add more related APIs with this
> >> approach. The single registration API provides one feature
> >> by one operation. If we use the RegisterCopyRoutine() for
> >> FROM and TO formats API, it's not natural that we add more
> >> related APIs. In this case, some APIs may provide multiple
> >> features by one operation and other APIs may provide single
> >> feature by one operation. Developers may be confused with
> >> the API. For example, developers may think "what does mean
> >> NULL here?" or "can we use NULL here?" for
> >> "RegisterCopyRoutine("new-format", NewFormatFromRoutine,
> >> NULL)".
> >
> > We can document it in the comment for the registration function.
>
> I think that API that can be understandable without the
> additional note is better API than API that needs some
> notes.

I don't see much difference in this case.

>
> Why do you suggest the RegisterCopyRoutine("new-format",
> NewFormatFromRoutine, NewFormatToRoutine) API? You want to
> remove the duplicated codes in
> RegisterCopy{From,To}Routine(), right?

No. I think that if extensions are likely to support both
CopyToRoutine and CopyFromRoutine in most cases, it would be simpler
to register the custom format using a single API. Registering
CopyToRoutine and CopyFromRoutine separately seems redundant to me.

>
> BTW, what do you think about my answer (one feature by one
> operation API is more extendable API) for your question
> (extendability and API compatibility)?

Could you provide some examples? It seems to me that even if we
provide the single API for the registration we can provide other APIs
differently. For example, if we want to provide an API to register a
custom option, we can provide RegisterCopyToOption() and
RegisterCopyFromOption().

>
> > Suppose that extension-A implements only CopyToRoutine for the
> > custom-format-X with the format name 'myformat' and extension-B
> > implements only CopyFromRoutine for the custom-format-Y with the same
> > name, if users load both extension-A and extension-B, it seems to me
> > that extension-A registers the custom-format-X format as 'myformat'
> > only with CopyToRoutine, and extension-B overwrites the 'myformat'
> > registration by adding custom-format-Y's CopyFromRoutine. However, if
> > users register extension-C that implements both routines with the
> > format name 'myformat', they can register neither extension-A nor
> > extension-B, which seems to me that we don't allow overwriting the
> > registration in this case.
>
> Do you assume that users use extension-A, extension-B and
> extension-C without reading their documentation? If users
> read their documentation before users use them, users can
> know all of them use the same format name 'myformat' and
> which extension provides Copy{From,To}Routine.
>
> In this case, these users (who don't read documentation)
> will be confused with the RegisterCopyRoutine("new-format",
> NewFormatFromRoutine, NewFormatToRoutine) API too. Do we
> really need to care about this case?

My point is about the consistency of registration behavior. I think
that we should raise an error if the custom format name that an
extension tries to register already exists. Therefore I'm not sure why
installing extension-A+B is okay but installing extension-C+A or
extension-C+B is not okay? We can think that's an extension-A's choice
not to implement CopyFromRoutine for the 'myformat' format so
extension-B should not change it.

>
> > I think the core issue appears to be the internal management of custom
> > format entries but the current patch does enable registration
> > overwriting in the former case (extension-A and extension-B case).
>
> This is the same behavior as existing custom EXPLAIN option
> implementation. Should we use different behavior here?

I think that unlike custom EXPLAIN options, it's better to raise an
error or a warning if the custom format name (or combination of format
name and COPY direction) that an extension tries to register already
exists.

> >> >>    FYI: RegisterCopy{From,To}Routine() uses the same logic
> >> >>    as RegisterExtensionExplainOption().
> >> >
> >> > I'm concerned that the patch has duplicated logics for the
> >> > registration of COPY FROM and COPY TO.
> >>
> >> We can implement a convenient routine that can be used for
> >> RegisterExtensionExplainOption() and
> >> RegisterCopy{From,To}Routine() if it's needed.
> >
> > I meant there are duplicated codes in COPY FROM and COPY TO. For
> > instance, RegisterCopyFromRoutine() and RegisterCopyToRoutine() have
> > the same logic.
>
> Yes, I understand it. I wanted to say that we can remove the
> duplicated codes by introducing a RegisterSomething()
> function that can be used by
> RegisterExtensionExplainOption() and
> RegisterCopy{From,To}Routine():
>
> void
> RegisterSomething(...)
> {
>   /* Common codes in RegisterExtensionExplainOption() and
>      RegisterCopy{From,To}Routine()
>      ...
>    */
> }
>
> void
> RegisterExtensionExplainOption(...)
> {
>   RegisterSomething(...);
> }
>
> void
> RegisterCopyFromRoutine(...)
> {
>   RegisterSomething(...);
> }
>
> void
> RegisterCopyToRoutine(...)
> {
>   RegisterSomething(...);
> }
>
> You think that this approach can't remove the duplicated
> codes, right?

Well, no, I just meant we don't need to do that. Custom EXPLAIN option
and custom COPY format are different features and have different
requirements. I think while we don't need to remove duplicates between
them at least at this stage we need to remove the duplicate between
COPY TO registration code and COPY TO's one.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  304. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2025-06-25T07:35:47Z

    Hi,

In <CAD21AoC19fV5Ujs-1r24MNU+hwTQUeZMEnaJDjSFwHLMMdFi0Q@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Wed, 25 Jun 2025 00:48:46 +0900,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

>> >> It's natural to add more related APIs with this
>> >> approach. The single registration API provides one feature
>> >> by one operation. If we use the RegisterCopyRoutine() for
>> >> FROM and TO formats API, it's not natural that we add more
>> >> related APIs. In this case, some APIs may provide multiple
>> >> features by one operation and other APIs may provide single
>> >> feature by one operation. Developers may be confused with
>> >> the API. For example, developers may think "what does mean
>> >> NULL here?" or "can we use NULL here?" for
>> >> "RegisterCopyRoutine("new-format", NewFormatFromRoutine,
>> >> NULL)".
>> >
>> > We can document it in the comment for the registration function.
>>
>> I think that API that can be understandable without the
>> additional note is better API than API that needs some
>> notes.
> 
> I don't see much difference in this case.

OK. It seems that we can't agree on which API is better.

I've implemented your idea as the v42 patch set. Can we
proceed this proposal with this approach? What is the next
step?

> No. I think that if extensions are likely to support both
> CopyToRoutine and CopyFromRoutine in most cases, it would be simpler
> to register the custom format using a single API. Registering
> CopyToRoutine and CopyFromRoutine separately seems redundant to me.

I don't think so. In general, extensions are implemented
step by step. Extension developers will not implement
CopyToRoutine and CopyFromRoutine at once even if extensions
implement both of CopyToRoutine and CopyFromRoutine
eventually.

> Could you provide some examples? It seems to me that even if we
> provide the single API for the registration we can provide other APIs
> differently. For example, if we want to provide an API to register a
> custom option, we can provide RegisterCopyToOption() and
> RegisterCopyFromOption().

Yes. We can mix different style APIs. In general, consistent
style APIs is easier to use than mixed style APIs. If it's
not an important point in PostgreSQL API design, my point is
meaningless. (Sorry, I'm not familiar with PostgreSQL API
design.)

> My point is about the consistency of registration behavior. I think
> that we should raise an error if the custom format name that an
> extension tries to register already exists. Therefore I'm not sure why
> installing extension-A+B is okay but installing extension-C+A or
> extension-C+B is not okay? We can think that's an extension-A's choice
> not to implement CopyFromRoutine for the 'myformat' format so
> extension-B should not change it.

I think that it's the users' responsibility. I think that
it's more convenient that users can mix extension-A+B (A
provides only TO format and B provides only FROM format)
than users can't mix them. I think that extension-A doesn't
want to prohibit FROM format in the case. Extension-A just
doesn't care about FROM format.

FYI: Both of extension-C+A and extension-C+B are OK when we
update not raising an error existing format.


Thanks,
-- 
kou

  305. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2025-06-30T06:00:45Z

    On Wed, Jun 25, 2025 at 4:35 PM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <CAD21AoC19fV5Ujs-1r24MNU+hwTQUeZMEnaJDjSFwHLMMdFi0Q@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Wed, 25 Jun 2025 00:48:46 +0900,
>   Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> >> >> It's natural to add more related APIs with this
> >> >> approach. The single registration API provides one feature
> >> >> by one operation. If we use the RegisterCopyRoutine() for
> >> >> FROM and TO formats API, it's not natural that we add more
> >> >> related APIs. In this case, some APIs may provide multiple
> >> >> features by one operation and other APIs may provide single
> >> >> feature by one operation. Developers may be confused with
> >> >> the API. For example, developers may think "what does mean
> >> >> NULL here?" or "can we use NULL here?" for
> >> >> "RegisterCopyRoutine("new-format", NewFormatFromRoutine,
> >> >> NULL)".
> >> >
> >> > We can document it in the comment for the registration function.
> >>
> >> I think that API that can be understandable without the
> >> additional note is better API than API that needs some
> >> notes.
> >
> > I don't see much difference in this case.
>
> OK. It seems that we can't agree on which API is better.
>
> I've implemented your idea as the v42 patch set. Can we
> proceed this proposal with this approach? What is the next
> step?

I'll review the patches. In the meanwhile could you update the
documentation accordingly?

>
> > No. I think that if extensions are likely to support both
> > CopyToRoutine and CopyFromRoutine in most cases, it would be simpler
> > to register the custom format using a single API. Registering
> > CopyToRoutine and CopyFromRoutine separately seems redundant to me.
>
> I don't think so. In general, extensions are implemented
> step by step. Extension developers will not implement
> CopyToRoutine and CopyFromRoutine at once even if extensions
> implement both of CopyToRoutine and CopyFromRoutine
> eventually.

Hmm, I think if the extension eventually implements both directions,
it would make sense to provide the single API.

>
> > Could you provide some examples? It seems to me that even if we
> > provide the single API for the registration we can provide other APIs
> > differently. For example, if we want to provide an API to register a
> > custom option, we can provide RegisterCopyToOption() and
> > RegisterCopyFromOption().
>
> Yes. We can mix different style APIs. In general, consistent
> style APIs is easier to use than mixed style APIs. If it's
> not an important point in PostgreSQL API design, my point is
> meaningless. (Sorry, I'm not familiar with PostgreSQL API
> design.)

As far as I know, there is no standard for PostgreSQL API design, but
I don't find any weirdness in this design.

>
> > My point is about the consistency of registration behavior. I think
> > that we should raise an error if the custom format name that an
> > extension tries to register already exists. Therefore I'm not sure why
> > installing extension-A+B is okay but installing extension-C+A or
> > extension-C+B is not okay? We can think that's an extension-A's choice
> > not to implement CopyFromRoutine for the 'myformat' format so
> > extension-B should not change it.
>
> I think that it's the users' responsibility. I think that
> it's more convenient that users can mix extension-A+B (A
> provides only TO format and B provides only FROM format)
> than users can't mix them. I think that extension-A doesn't
> want to prohibit FROM format in the case. Extension-A just
> doesn't care about FROM format.
>
> FYI: Both of extension-C+A and extension-C+B are OK when we
> update not raising an error existing format.

I want to keep the basic design that one custom format comes from one
extension because it's straightforward for both of us and users and
easy to maintain format ID. IIUC we somewhat agreed on this design in
the previous API design (TABLESAMPLE like API).

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  306. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2025-07-13T18:28:16Z

    On Mon, Jun 30, 2025 at 3:00 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Wed, Jun 25, 2025 at 4:35 PM Sutou Kouhei <kou@clear-code.com> wrote:
> >
> > Hi,
> >
> > In <CAD21AoC19fV5Ujs-1r24MNU+hwTQUeZMEnaJDjSFwHLMMdFi0Q@mail.gmail.com>
> >   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Wed, 25 Jun 2025 00:48:46 +0900,
> >   Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > >> >> It's natural to add more related APIs with this
> > >> >> approach. The single registration API provides one feature
> > >> >> by one operation. If we use the RegisterCopyRoutine() for
> > >> >> FROM and TO formats API, it's not natural that we add more
> > >> >> related APIs. In this case, some APIs may provide multiple
> > >> >> features by one operation and other APIs may provide single
> > >> >> feature by one operation. Developers may be confused with
> > >> >> the API. For example, developers may think "what does mean
> > >> >> NULL here?" or "can we use NULL here?" for
> > >> >> "RegisterCopyRoutine("new-format", NewFormatFromRoutine,
> > >> >> NULL)".
> > >> >
> > >> > We can document it in the comment for the registration function.
> > >>
> > >> I think that API that can be understandable without the
> > >> additional note is better API than API that needs some
> > >> notes.
> > >
> > > I don't see much difference in this case.
> >
> > OK. It seems that we can't agree on which API is better.
> >
> > I've implemented your idea as the v42 patch set. Can we
> > proceed this proposal with this approach? What is the next
> > step?
>
> I'll review the patches.

I've reviewed the 0001 and 0002 patches. The API implemented in the
0002 patch looks good to me, but I'm concerned about the capsulation
of copy state data. With the v42 patches, we pass the whole
CopyToStateData to the extension codes, but most of the fields in
CopyToStateData are internal working state data that shouldn't be
exposed to extensions. I think we need to sort out which fields are
exposed or not. That way, it would be safer and we would be able to
avoid exposing copyto_internal.h and extensions would not need to
include copyfrom_internal.h.

I've implemented a draft patch for that idea. In the 0001 patch, I
moved fields that are related to internal working state from
CopyToStateData to CopyToExectuionData. COPY routine APIs pass a
pointer of CopyToStateData but extensions can access only fields
except for CopyToExectuionData. In the 0002 patch, I've implemented
the registration API and some related APIs based on your v42 patch.
I've made similar changes to COPY FROM codes too.

The patch is a very PoC phase and we would need to scrutinize the
fields that should or should not be exposed. Feedback is very welcome.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  307. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2025-07-14T08:38:03Z

    Hi,

In <CAD21AoB0Z3gkOGALK3pXYmGTWATVvgDAmn-yXGp2mX64S-YrSw@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Mon, 14 Jul 2025 03:28:16 +0900,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

> I've reviewed the 0001 and 0002 patches. The API implemented in the
> 0002 patch looks good to me, but I'm concerned about the capsulation
> of copy state data. With the v42 patches, we pass the whole
> CopyToStateData to the extension codes, but most of the fields in
> CopyToStateData are internal working state data that shouldn't be
> exposed to extensions. I think we need to sort out which fields are
> exposed or not. That way, it would be safer and we would be able to
> avoid exposing copyto_internal.h and extensions would not need to
> include copyfrom_internal.h.

FYI: We discussed this so far. For example:

https://www.postgresql.org/message-id/flat/CAD21AoD%3DUapH4Wh06G6H5XAzPJ0iJg9YcW8r7E2UEJkZ8QsosA%40mail.gmail.com

> I think we can move CopyToState to copy.h and we don't
> need to have set/get functions for its fields.

https://www.postgresql.org/message-id/flat/CAD21AoBpWFU4k-_bwrTq0AkFSAdwQqhAsSW188STmu9HxLJ0nQ%40mail.gmail.com

> > What does "private" mean here? I thought that it means that
> > "PostgreSQL itself can use it". But it seems that you mean
> > that "PostgreSQL itself and custom format extensions can use
> > it but other extensions can't use it".
> >
> > I'm not familiar with "_internal.h" in PostgreSQL but is
> > "_internal.h" for the latter "private" mean?
>
> My understanding is that we don't strictly prohibit _internal.h from
> being included in out of core files. For example, file_fdw.c includes
> copyfrom_internal.h in order to access some fields of CopyFromState.


In general, I agree that we should export only needed
information.

How about adding accessors instead of splitting
Copy{From,To}State to Copy{From,To}ExecutionData? If we use
the accessors approach, we can export only needed
information step by step without breaking ABI.

The built-in formats can keep using Copy{From,To}State
directly with the accessors approach. We can avoid any
performance regression of the built-in formats. If we split
Copy{From,To}State to Copy{From,To}ExecutionData,
performance may be changed.


> I've implemented a draft patch for that idea. In the 0001 patch, I
> moved fields that are related to internal working state from
> CopyToStateData to CopyToExectuionData. COPY routine APIs pass a
> pointer of CopyToStateData but extensions can access only fields
> except for CopyToExectuionData. In the 0002 patch, I've implemented
> the registration API and some related APIs based on your v42 patch.
> I've made similar changes to COPY FROM codes too.
> 
> The patch is a very PoC phase and we would need to scrutinize the
> fields that should or should not be exposed. Feedback is very welcome.

Based on our sample extensions [1][2], the following fields
may be minimal. I added "(*)" marks that exist in
Copy{From,To}StateDate in your patch. Other fields exist in
Copy{From,To}ExecutionData. We need to export them to
extensions. We can hide fields in Copy{From,To}StateData not
listed here.

FROM:

- attnumlist (*)
- bytes_processed
- cur_attname
- escontext
- in_functions (*)
- input_buf
- input_reached_eof
- line_buf
- opts (*)
- raw_buf
- raw_buf_index
- raw_buf_len
- rel (*)
- typioparams (*)

TO:

- attnumlist (*)
- fe_msgbuf
- opts (*)

[1] https://github.com/kou/pg-copy-arrow/
[2] https://github.com/MasahikoSawada/pg_copy_jsonlines/


Thanks,
-- 
kou

  308. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2025-07-15T07:54:13Z

    Hi,

In <20250714.173803.865595983884510428.kou@clear-code.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Mon, 14 Jul 2025 17:38:03 +0900 (JST),
  Sutou Kouhei <kou@clear-code.com> wrote:

>> I've reviewed the 0001 and 0002 patches. The API implemented in the
>> 0002 patch looks good to me, but I'm concerned about the capsulation
>> of copy state data. With the v42 patches, we pass the whole
>> CopyToStateData to the extension codes, but most of the fields in
>> CopyToStateData are internal working state data that shouldn't be
>> exposed to extensions. I think we need to sort out which fields are
>> exposed or not. That way, it would be safer and we would be able to
>> avoid exposing copyto_internal.h and extensions would not need to
>> include copyfrom_internal.h.

> In general, I agree that we should export only needed
> information.
> 
> How about adding accessors instead of splitting
> Copy{From,To}State to Copy{From,To}ExecutionData? If we use
> the accessors approach, we can export only needed
> information step by step without breaking ABI.

Another idea: We'll add Copy{From,To}State::opaque
eventually. (For example, the v40-0003 patch includes it.)

How about using it to hide fields only for built-in formats?


Thanks,
-- 
kou

  309. Re: Make COPY format extendable: Extract COPY TO format implementations

    Andres Freund <andres@anarazel.de> — 2025-07-15T12:36:58Z

    Hi,

On 2025-07-14 03:28:16 +0900, Masahiko Sawada wrote:
> I've reviewed the 0001 and 0002 patches. The API implemented in the
> 0002 patch looks good to me, but I'm concerned about the capsulation
> of copy state data. With the v42 patches, we pass the whole
> CopyToStateData to the extension codes, but most of the fields in
> CopyToStateData are internal working state data that shouldn't be
> exposed to extensions. I think we need to sort out which fields are
> exposed or not. That way, it would be safer and we would be able to
> avoid exposing copyto_internal.h and extensions would not need to
> include copyfrom_internal.h.
> 
> I've implemented a draft patch for that idea. In the 0001 patch, I
> moved fields that are related to internal working state from
> CopyToStateData to CopyToExectuionData. COPY routine APIs pass a
> pointer of CopyToStateData but extensions can access only fields
> except for CopyToExectuionData. In the 0002 patch, I've implemented
> the registration API and some related APIs based on your v42 patch.
> I've made similar changes to COPY FROM codes too.

I've not followed the development of this patch - but I continue to be
concerned about the performance impact it has as-is and the amount of COPY
performance improvements it forecloses.

This seems to add yet another layer of indirection to a lot of hot functions
like CopyGetData() etc.

Greetings,

Andres Freund

  310. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2025-07-17T20:33:13Z

    On Tue, Jul 15, 2025 at 5:37 AM Andres Freund <andres@anarazel.de> wrote:
>
> Hi,
>
> On 2025-07-14 03:28:16 +0900, Masahiko Sawada wrote:
> > I've reviewed the 0001 and 0002 patches. The API implemented in the
> > 0002 patch looks good to me, but I'm concerned about the capsulation
> > of copy state data. With the v42 patches, we pass the whole
> > CopyToStateData to the extension codes, but most of the fields in
> > CopyToStateData are internal working state data that shouldn't be
> > exposed to extensions. I think we need to sort out which fields are
> > exposed or not. That way, it would be safer and we would be able to
> > avoid exposing copyto_internal.h and extensions would not need to
> > include copyfrom_internal.h.
> >
> > I've implemented a draft patch for that idea. In the 0001 patch, I
> > moved fields that are related to internal working state from
> > CopyToStateData to CopyToExectuionData. COPY routine APIs pass a
> > pointer of CopyToStateData but extensions can access only fields
> > except for CopyToExectuionData. In the 0002 patch, I've implemented
> > the registration API and some related APIs based on your v42 patch.
> > I've made similar changes to COPY FROM codes too.
>
> I've not followed the development of this patch - but I continue to be
> concerned about the performance impact it has as-is and the amount of COPY
> performance improvements it forecloses.
>
> This seems to add yet another layer of indirection to a lot of hot functions
> like CopyGetData() etc.
>

The most refactoring works have been done by commit 7717f6300 and
2e4127b6d with a slight performance gain. At this stage, we're trying
to introduce the registration API so that extensions can provide their
callbacks to the core. Some functions required for I/O such as
CopyGetData() and CopySendEndOfRow() would be exposed but I'm not
going to add additional indirection function call layers.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  311. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2025-07-17T20:44:11Z

    On Tue, Jul 15, 2025 at 12:54 AM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <20250714.173803.865595983884510428.kou@clear-code.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Mon, 14 Jul 2025 17:38:03 +0900 (JST),
>   Sutou Kouhei <kou@clear-code.com> wrote:
>
> >> I've reviewed the 0001 and 0002 patches. The API implemented in the
> >> 0002 patch looks good to me, but I'm concerned about the capsulation
> >> of copy state data. With the v42 patches, we pass the whole
> >> CopyToStateData to the extension codes, but most of the fields in
> >> CopyToStateData are internal working state data that shouldn't be
> >> exposed to extensions. I think we need to sort out which fields are
> >> exposed or not. That way, it would be safer and we would be able to
> >> avoid exposing copyto_internal.h and extensions would not need to
> >> include copyfrom_internal.h.
>
> > In general, I agree that we should export only needed
> > information.
> >
> > How about adding accessors instead of splitting
> > Copy{From,To}State to Copy{From,To}ExecutionData? If we use
> > the accessors approach, we can export only needed
> > information step by step without breaking ABI.

Yeah, while it can export required fields without breaking ABI, I'm
concerned that setter and getter functions could be bloated if we need
to have them for many fields.

>
> Another idea: We'll add Copy{From,To}State::opaque
> eventually. (For example, the v40-0003 patch includes it.)
>
> How about using it to hide fields only for built-in formats?

What is the difference between your idea and splitting CopyToState
into CopyToState and CopyToExecutionData?

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  312. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2025-07-18T09:49:12Z

    Hi,

In <CAD21AoAQkjU=o0nX4y0jtX0BnsrqA04g2ABqrUwjT88YeEWarA@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Thu, 17 Jul 2025 13:33:13 -0700,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

>> I've not followed the development of this patch - but I continue to be
>> concerned about the performance impact it has as-is and the amount of COPY
>> performance improvements it forecloses.
>>
>> This seems to add yet another layer of indirection to a lot of hot functions
>> like CopyGetData() etc.
>>
> 
> The most refactoring works have been done by commit 7717f6300 and
> 2e4127b6d with a slight performance gain. At this stage, we're trying
> to introduce the registration API so that extensions can provide their
> callbacks to the core. Some functions required for I/O such as
> CopyGetData() and CopySendEndOfRow() would be exposed but I'm not
> going to add additional indirection function call layers.

I think Andres is talking about any indirection not only
indirection function call. In this case, "cstate->XXX" ->
"cstate->edata->XXX".

It's also mentioned in my e-mail. I'm not sure whether it
has performance impact but it's better that we benchmark to
confirm whether there is any performance impact or not with
the Copy{From,To}ExecutionData approach.


Thanks,
-- 
kou

  313. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2025-07-18T10:05:53Z

    Hi,

In <CAD21AoAZL2RzPM4RLOJKm_73z5LXq2_VOVF+S+T0tnbjHdWTFA@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Thu, 17 Jul 2025 13:44:11 -0700,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

>> > How about adding accessors instead of splitting
>> > Copy{From,To}State to Copy{From,To}ExecutionData? If we use
>> > the accessors approach, we can export only needed
>> > information step by step without breaking ABI.
> 
> Yeah, while it can export required fields without breaking ABI, I'm
> concerned that setter and getter functions could be bloated if we need
> to have them for many fields.

In general, I choose this approach in my projects even when
I need to define many accessors. Because I can hide
implementation details from users. I can change
implementation details without breaking API/ABI.

But PostgreSQL isn't my project. Is there any guideline for
PostgreSQL API(/ABI?) design that we can refer for this
case?

FYI: We need to export at least the following fields:

https://www.postgresql.org/message-id/flat/20250714.173803.865595983884510428.kou%40clear-code.com#78fdbccf89742f856aa2cf95eaf42032

> FROM:
> 
> - attnumlist (*)
> - bytes_processed
> - cur_attname
> - escontext
> - in_functions (*)
> - input_buf
> - input_reached_eof
> - line_buf
> - opts (*)
> - raw_buf
> - raw_buf_index
> - raw_buf_len
> - rel (*)
> - typioparams (*)
> 
> TO:
> 
> - attnumlist (*)
> - fe_msgbuf
> - opts (*)


Here are pros/cons of the Copy{From,To}ExecutionData
approach, right?

Pros:
1. We can hide internal data from extensions

Cons:
1. Built-in format routines need to refer fields via
   Copy{From,To}ExecutionData.
   * This MAY has performance impact. If there is no
     performance impact, this is not a cons.
2. API/ABI compatibility will be broken when we change
   exported fields.
   * I'm not sure whether this is a cons in the PostgreSQL
     design.

Here are pros/cons of the accessors approach:

Pros:
1. We can hide internal data from extensions
2. We can export new fields change field names
   without breaking API/ABI compatibility
3. We don't need to change built-in format routines.
   So we can assume that there is no performance impact.

Cons:
1. We may need to define many accessors
   * I'm not sure whether this is a cons in the PostgreSQL
     design.

>> Another idea: We'll add Copy{From,To}State::opaque
>> eventually. (For example, the v40-0003 patch includes it.)
>>
>> How about using it to hide fields only for built-in formats?
> 
> What is the difference between your idea and splitting CopyToState
> into CopyToState and CopyToExecutionData?

1. We don't need to manage 2 similar data for built-in
   formats and extensions.
   * Build-in formats use CopyToExecutionData and extensions
     use opaque.
2. We can introduce registration API now.
   * We can work on this topic AFTER we introduce
     registration API.
   * e.g.: Add registration API -> Add opaque -> Use opaque
     for internal fields (we will benchmark this
     implementation at this time)


Thanks,
-- 
kou

  314. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2025-07-28T19:33:28Z

    On Fri, Jul 18, 2025 at 3:05 AM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <CAD21AoAZL2RzPM4RLOJKm_73z5LXq2_VOVF+S+T0tnbjHdWTFA@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Thu, 17 Jul 2025 13:44:11 -0700,
>   Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> >> > How about adding accessors instead of splitting
> >> > Copy{From,To}State to Copy{From,To}ExecutionData? If we use
> >> > the accessors approach, we can export only needed
> >> > information step by step without breaking ABI.
> >
> > Yeah, while it can export required fields without breaking ABI, I'm
> > concerned that setter and getter functions could be bloated if we need
> > to have them for many fields.
>
> In general, I choose this approach in my projects even when
> I need to define many accessors. Because I can hide
> implementation details from users. I can change
> implementation details without breaking API/ABI.
>
> But PostgreSQL isn't my project. Is there any guideline for
> PostgreSQL API(/ABI?) design that we can refer for this
> case?

As far as I know there is no official guideline for PostgreSQL API and
ABI design, but I've never seen structs having more than 10 getter and
setter in PostgreSQL source code.

>
> FYI: We need to export at least the following fields:
>
> https://www.postgresql.org/message-id/flat/20250714.173803.865595983884510428.kou%40clear-code.com#78fdbccf89742f856aa2cf95eaf42032
>
> > FROM:
> >
> > - attnumlist (*)
> > - bytes_processed
> > - cur_attname
> > - escontext
> > - in_functions (*)
> > - input_buf
> > - input_reached_eof
> > - line_buf
> > - opts (*)
> > - raw_buf
> > - raw_buf_index
> > - raw_buf_len
> > - rel (*)
> > - typioparams (*)
> >
> > TO:
> >
> > - attnumlist (*)
> > - fe_msgbuf
> > - opts (*)

I think we can think of the minimum list of fields that we need to
expose. For instance, fields used for buffered reads for COPY FROM
such as input_buf and raw_buf related fields don't necessarily need to
be exposed as extension can implement it in its own way. We can start
with the supporting simple copy format extensions like that read and
parse the binary data from the data source and fill 'values' and
'nulls' arrays as output. Considering these facts, it might be
sufficient for copy format extensions if they could access 'rel',
'attnumlist', and 'opts' in both COPY FROM and COPY TO (and
CopyFromErrorCallback related fields for COPY FROM).

Apart from this, we might want to reorganize CopyFromStateData fields
and CopyToStateData fields since they have mixed fields of general
purpose fields for COPY operations (e.g., num_defaults, whereClause,
and range_table) and built-in format specific fields (e.g., line_buf
and input_buf). Text and CSV formats are using some fields for parsing
fields with buffered reads so one idea is that we move related fields
to another struct so that both built-in formats (text and CSV) and
external extensions that want to use the buffered reads for text
parsing can use this functionality.

> Here are pros/cons of the Copy{From,To}ExecutionData
> approach, right?
>
> Pros:
> 1. We can hide internal data from extensions
>
> Cons:
> 1. Built-in format routines need to refer fields via
>    Copy{From,To}ExecutionData.
>    * This MAY has performance impact. If there is no
>      performance impact, this is not a cons.
> 2. API/ABI compatibility will be broken when we change
>    exported fields.
>    * I'm not sure whether this is a cons in the PostgreSQL
>      design.
>
> Here are pros/cons of the accessors approach:
>
> Pros:
> 1. We can hide internal data from extensions
> 2. We can export new fields change field names
>    without breaking API/ABI compatibility
> 3. We don't need to change built-in format routines.
>    So we can assume that there is no performance impact.
>
> Cons:
> 1. We may need to define many accessors
>    * I'm not sure whether this is a cons in the PostgreSQL
>      design.

I agree with the summary.

> >> Another idea: We'll add Copy{From,To}State::opaque
> >> eventually. (For example, the v40-0003 patch includes it.)
> >>
> >> How about using it to hide fields only for built-in formats?
> >
> > What is the difference between your idea and splitting CopyToState
> > into CopyToState and CopyToExecutionData?
>
> 1. We don't need to manage 2 similar data for built-in
>    formats and extensions.
>    * Build-in formats use CopyToExecutionData and extensions
>      use opaque.
> 2. We can introduce registration API now.
>    * We can work on this topic AFTER we introduce
>      registration API.
>    * e.g.: Add registration API -> Add opaque -> Use opaque
>      for internal fields (we will benchmark this
>      implementation at this time)

What if we find performance overhead in built-in format cases after
introducing opaque data? I personally would like to avoid merging the
registration API (i.e., supporting custom copy formats) while being
unsure about the overall design ahead and the potential performance
impact by following patches.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  315. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2025-07-29T05:19:36Z

    On Mon, Jul 28, 2025 at 12:33 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Fri, Jul 18, 2025 at 3:05 AM Sutou Kouhei <kou@clear-code.com> wrote:
> >
> > Hi,
> >
> > In <CAD21AoAZL2RzPM4RLOJKm_73z5LXq2_VOVF+S+T0tnbjHdWTFA@mail.gmail.com>
> >   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Thu, 17 Jul 2025 13:44:11 -0700,
> >   Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > >> > How about adding accessors instead of splitting
> > >> > Copy{From,To}State to Copy{From,To}ExecutionData? If we use
> > >> > the accessors approach, we can export only needed
> > >> > information step by step without breaking ABI.
> > >
> > > Yeah, while it can export required fields without breaking ABI, I'm
> > > concerned that setter and getter functions could be bloated if we need
> > > to have them for many fields.
> >
> > In general, I choose this approach in my projects even when
> > I need to define many accessors. Because I can hide
> > implementation details from users. I can change
> > implementation details without breaking API/ABI.
> >
> > But PostgreSQL isn't my project. Is there any guideline for
> > PostgreSQL API(/ABI?) design that we can refer for this
> > case?
>
> As far as I know there is no official guideline for PostgreSQL API and
> ABI design, but I've never seen structs having more than 10 getter and
> setter in PostgreSQL source code.
>
> >
> > FYI: We need to export at least the following fields:
> >
> > https://www.postgresql.org/message-id/flat/20250714.173803.865595983884510428.kou%40clear-code.com#78fdbccf89742f856aa2cf95eaf42032
> >
> > > FROM:
> > >
> > > - attnumlist (*)
> > > - bytes_processed
> > > - cur_attname
> > > - escontext
> > > - in_functions (*)
> > > - input_buf
> > > - input_reached_eof
> > > - line_buf
> > > - opts (*)
> > > - raw_buf
> > > - raw_buf_index
> > > - raw_buf_len
> > > - rel (*)
> > > - typioparams (*)
> > >
> > > TO:
> > >
> > > - attnumlist (*)
> > > - fe_msgbuf
> > > - opts (*)
>
> I think we can think of the minimum list of fields that we need to
> expose. For instance, fields used for buffered reads for COPY FROM
> such as input_buf and raw_buf related fields don't necessarily need to
> be exposed as extension can implement it in its own way. We can start
> with the supporting simple copy format extensions like that read and
> parse the binary data from the data source and fill 'values' and
> 'nulls' arrays as output. Considering these facts, it might be
> sufficient for copy format extensions if they could access 'rel',
> 'attnumlist', and 'opts' in both COPY FROM and COPY TO (and
> CopyFromErrorCallback related fields for COPY FROM).
>
> Apart from this, we might want to reorganize CopyFromStateData fields
> and CopyToStateData fields since they have mixed fields of general
> purpose fields for COPY operations (e.g., num_defaults, whereClause,
> and range_table) and built-in format specific fields (e.g., line_buf
> and input_buf). Text and CSV formats are using some fields for parsing
> fields with buffered reads so one idea is that we move related fields
> to another struct so that both built-in formats (text and CSV) and
> external extensions that want to use the buffered reads for text
> parsing can use this functionality.

So probably it might be worth refactoring the codes in terms of:

1. hiding internal data from format callbacks
2. separating format-specific fields from the main state data.

I categorized the fields in CopyFromStateData. I think there are
roughly three different kind of fields mixed there:

1. fields used only the core (not by format callback)
- filename
- is_program
- whereClause
- cur_relname
- copycontext
- defmap
- num_defaults
- volatile_defexprs
- range_table
- rtrperminfos
- qualexpr
- transition_capture

2. fields used by both the core and format callbacks
- rel
- attnumlist
- cur_lineno
- cur_attname
- cur_attval
- relname_only
- num_errors
- opts
- in_functions
- typioparams
- escontext
- defexprs
- Input-related fields
    - copy_src
    - copy_file
    - fe_msgbuf
    - data_source_cb
    - byteprocessed

3. built-in format specific fields (mostly for text and csv)
- eol_type
- defaults
- Encoding related fields
    - file_encoding
    - need_transcoding
    - conversion_proc
- convert_select_flags
- raw data pointers
    - max_fields
    - raw_fields
- attribute_buf
- line_buf related fields
    - line_buf
    - line_buf_valid
- input_buf related fields
    - input_buf
    - input_buf_index
    - input_buf_len
    - input_reached_eof
    - input_reached_error
- raw_buf related fields
    - raw_buf
    - raw_buf_index
    - raw_buf_len
    - raw_reached_eof

The fields in 1 are mostly static fields, and the fields in 2 and 3
are likely to be accessed in hot functions during COPY FROM. Would it
be a good idea to restructure these fields so that we can hide the
fields in 1 from callback functions and having the fields in 3 in a
separate format-specific struct that can be accessed via an opaque
pointer? But could the latter change potentially cause performance
overheads?

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  316. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2025-08-14T06:36:54Z

    Hi,

In <CAD21AoBa0Wm3C2H12jaqkvLidP2zEhsC+gf=3w7XiA4LQnvx0g@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Mon, 28 Jul 2025 22:19:36 -0700,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

> The fields in 1 are mostly static fields, and the fields in 2 and 3
> are likely to be accessed in hot functions during COPY FROM. Would it
> be a good idea to restructure these fields so that we can hide the
> fields in 1 from callback functions and having the fields in 3 in a
> separate format-specific struct that can be accessed via an opaque
> pointer? But could the latter change potentially cause performance
> overheads?

Yes. It changes memory layout (1 continuous memory chunk ->
2 separated memory chunks) and introduces indirect member
accesses (x->y -> x->z->y). They may not have performance
impact but we need to measure it if we want to use this
approach.


BTW, how about the following approach?

copyapi.h:

typedef struct CopyToStateData
{
	/* public members */
	/* ... */
} CopyToStateData;

copyto.c:

typedef struct CopyToStateInternalData
{
	CopyToStateData parent;

	/* private members */
	/* ... */
} CopyToStateInternalData;

We export CopyToStateData only with public members. We don't
export CopyToStateInternalData that has members only for
built-in formats.

CopyToStateInternalData has the same memory layout as
CopyToStateData. So we can use CopyToStateInternalData as
CopyToStateData.

We use CopyToStateData not CopyToStateInternalData in public
API. We cast CopyToStateData to CopyToStateInternalData when
we need to use private members:

static void
CopySendData(CopyToState cstate, const void *databuf, int datasize)
{
	CopyToStateInternal cstate_internal = (CopyToStateInternal) cstate;
	appendBinaryStringInfo(cstate_internal->fe_msgbuf, databuf, datasize);
}

It's still direct member access.


With this approach, we can keep the same memory layout (1
continuous memory chunk) and direct member access. I think
that this approach doesn't have performance impact.

See the attached patch for PoC of this approach.

Drawback: This approach always allocates
CopyToStateInternalData not CopyToStateData. So we need to
allocate needless memories for extensions. But this will
prevent performance regression of built-in formats. Is it
acceptable?


Thanks,
-- 
kou

  317. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2025-09-08T21:08:16Z

    On Wed, Aug 13, 2025 at 11:37 PM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <CAD21AoBa0Wm3C2H12jaqkvLidP2zEhsC+gf=3w7XiA4LQnvx0g@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Mon, 28 Jul 2025 22:19:36 -0700,
>   Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> > The fields in 1 are mostly static fields, and the fields in 2 and 3
> > are likely to be accessed in hot functions during COPY FROM. Would it
> > be a good idea to restructure these fields so that we can hide the
> > fields in 1 from callback functions and having the fields in 3 in a
> > separate format-specific struct that can be accessed via an opaque
> > pointer? But could the latter change potentially cause performance
> > overheads?
>
> Yes. It changes memory layout (1 continuous memory chunk ->
> 2 separated memory chunks) and introduces indirect member
> accesses (x->y -> x->z->y).

I think fields accessed by the hot functions are limited. If we
assemble fields required by hot functions into one struct and pass it
to them can we deal with the latter? For example, we assemble the
fields in 3 I mentioned above (i.e., built-in format specific fields)
into say CopyFromStateBuiltin and pass it to CopyReadLine() function
and the following functions, instead of CopyFromState. Since there are
some places where we need to access to CopyFromState (e.g.,
CopyGetData()), CopyFromStateBuiltin needs to have a pointer to
CopyFromState as well.

>  They may not have performance
> impact but we need to measure it if we want to use this
> approach.

Agreed.

> BTW, how about the following approach?
>
> copyapi.h:
>
> typedef struct CopyToStateData
> {
>         /* public members */
>         /* ... */
> } CopyToStateData;
>
> copyto.c:
>
> typedef struct CopyToStateInternalData
> {
>         CopyToStateData parent;
>
>         /* private members */
>         /* ... */
> } CopyToStateInternalData;
>
> We export CopyToStateData only with public members. We don't
> export CopyToStateInternalData that has members only for
> built-in formats.
>
> CopyToStateInternalData has the same memory layout as
> CopyToStateData. So we can use CopyToStateInternalData as
> CopyToStateData.
>
> We use CopyToStateData not CopyToStateInternalData in public
> API. We cast CopyToStateData to CopyToStateInternalData when
> we need to use private members:
>
> static void
> CopySendData(CopyToState cstate, const void *databuf, int datasize)
> {
>         CopyToStateInternal cstate_internal = (CopyToStateInternal) cstate;
>         appendBinaryStringInfo(cstate_internal->fe_msgbuf, databuf, datasize);
> }
>
> It's still direct member access.
>
>
> With this approach, we can keep the same memory layout (1
> continuous memory chunk) and direct member access. I think
> that this approach doesn't have performance impact.
>
> See the attached patch for PoC of this approach.
>
> Drawback: This approach always allocates
> CopyToStateInternalData not CopyToStateData. So we need to
> allocate needless memories for extensions. But this will
> prevent performance regression of built-in formats. Is it
> acceptable?

While this approach could make sense to avoid potential performance
overheads for built-in format, I find it's somewhat odd that
extensions cannot allocate memory for its working state having
CopyToStateData as the base type.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  318. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2025-09-09T02:50:14Z

    Hi,

In <CAD21AoCCjKA77xkUxx59qJ8an_G_58Mry_EtCEcFgd=g9N2xew@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Mon, 8 Sep 2025 14:08:16 -0700,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

>> > The fields in 1 are mostly static fields, and the fields in 2 and 3
>> > are likely to be accessed in hot functions during COPY FROM. Would it
>> > be a good idea to restructure these fields so that we can hide the
>> > fields in 1 from callback functions and having the fields in 3 in a
>> > separate format-specific struct that can be accessed via an opaque
>> > pointer? But could the latter change potentially cause performance
>> > overheads?
>>
>> Yes. It changes memory layout (1 continuous memory chunk ->
>> 2 separated memory chunks) and introduces indirect member
>> accesses (x->y -> x->z->y).
> 
> I think fields accessed by the hot functions are limited. If we
> assemble fields required by hot functions into one struct and pass it
> to them can we deal with the latter? For example, we assemble the
> fields in 3 I mentioned above (i.e., built-in format specific fields)
> into say CopyFromStateBuiltin and pass it to CopyReadLine() function
> and the following functions, instead of CopyFromState. Since there are
> some places where we need to access to CopyFromState (e.g.,
> CopyGetData()), CopyFromStateBuiltin needs to have a pointer to
> CopyFromState as well.

It can change indirect member accesses (built-in format
specific members can be direct access but other members in
CopyFromState are indirect access) but it doesn't change 2
separated memory chunks.

If this approach has performance impact and it's caused by
indirect member accesses for built-in format specific
members, your suggestion will work. If performance impact is
caused by another reason, your suggestion may not work.

Anyway, we need to measure performance to proceed with this
approach. If we can confirm that this approach doesn't have
any performance impact, we can use the original your idea.

Do you have any idea how to measure performance of this
approach?

We did it when we introduce Copy{To,From}Routine. But it was
difficult to evaluate the results:

* I don't have machines for stable benchmark results
  * We may not be able to use them for the final decision
* Most results showed performance improvement but
  there was a result showed mysterious result[1]

[1] https://www.postgresql.org/message-id/flat/CAEG8a3LUBcvjwqgt6AijJmg67YN_b_NZ4Kzoxc_dH4rpAq0pKg%40mail.gmail.com

>> Drawback: This approach always allocates
>> CopyToStateInternalData not CopyToStateData. So we need to
>> allocate needless memories for extensions. But this will
>> prevent performance regression of built-in formats. Is it
>> acceptable?
> 
> While this approach could make sense to avoid potential performance
> overheads for built-in format, I find it's somewhat odd that
> extensions cannot allocate memory for its working state having
> CopyToStateData as the base type.

Is it important? We'll provide a opaque member for
extensions. Extensions should use the opaque member instead
of extending Copy{From,To}StateData.


I don't object your approach but we need a good way to
measure performance. If we use this approach, we can omit it
for now and we can revisit your approach later without
breaking compatibility. How about using this approach if we
can't find a good way to measure performance?


Thanks,
-- 
kou

  319. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2025-09-09T20:15:43Z

    On Mon, Sep 8, 2025 at 7:50 PM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <CAD21AoCCjKA77xkUxx59qJ8an_G_58Mry_EtCEcFgd=g9N2xew@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Mon, 8 Sep 2025 14:08:16 -0700,
>   Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> >> > The fields in 1 are mostly static fields, and the fields in 2 and 3
> >> > are likely to be accessed in hot functions during COPY FROM. Would it
> >> > be a good idea to restructure these fields so that we can hide the
> >> > fields in 1 from callback functions and having the fields in 3 in a
> >> > separate format-specific struct that can be accessed via an opaque
> >> > pointer? But could the latter change potentially cause performance
> >> > overheads?
> >>
> >> Yes. It changes memory layout (1 continuous memory chunk ->
> >> 2 separated memory chunks) and introduces indirect member
> >> accesses (x->y -> x->z->y).
> >
> > I think fields accessed by the hot functions are limited. If we
> > assemble fields required by hot functions into one struct and pass it
> > to them can we deal with the latter? For example, we assemble the
> > fields in 3 I mentioned above (i.e., built-in format specific fields)
> > into say CopyFromStateBuiltin and pass it to CopyReadLine() function
> > and the following functions, instead of CopyFromState. Since there are
> > some places where we need to access to CopyFromState (e.g.,
> > CopyGetData()), CopyFromStateBuiltin needs to have a pointer to
> > CopyFromState as well.
>
> It can change indirect member accesses (built-in format
> specific members can be direct access but other members in
> CopyFromState are indirect access) but it doesn't change 2
> separated memory chunks.

Right. IIUC the latter point is related to cache locality. But I'm not
sure how much the latter point affects the performance as currently we
don't declare fields to CopyFromState while carefully considering the
cache locality even today. Separating a single memory into multiple
chunks could even have a positive effect on it.

>
> If this approach has performance impact and it's caused by
> indirect member accesses for built-in format specific
> members, your suggestion will work. If performance impact is
> caused by another reason, your suggestion may not work.
>
> Anyway, we need to measure performance to proceed with this
> approach. If we can confirm that this approach doesn't have
> any performance impact, we can use the original your idea.
>
> Do you have any idea how to measure performance of this
> approach?

I think we can start with measuring the entire COPY execution time
with several scenarios as we did previously. For example, reading a
huge file with a single column value and with many columns etc.

>
> We did it when we introduce Copy{To,From}Routine. But it was
> difficult to evaluate the results:
>
> * I don't have machines for stable benchmark results
>   * We may not be able to use them for the final decision
> * Most results showed performance improvement but
>   there was a result showed mysterious result[1]

Perhaps measuring cache-misses help to see how changes could affect
the performance?

>
> [1] https://www.postgresql.org/message-id/flat/CAEG8a3LUBcvjwqgt6AijJmg67YN_b_NZ4Kzoxc_dH4rpAq0pKg%40mail.gmail.com
>
> >> Drawback: This approach always allocates
> >> CopyToStateInternalData not CopyToStateData. So we need to
> >> allocate needless memories for extensions. But this will
> >> prevent performance regression of built-in formats. Is it
> >> acceptable?
> >
> > While this approach could make sense to avoid potential performance
> > overheads for built-in format, I find it's somewhat odd that
> > extensions cannot allocate memory for its working state having
> > CopyToStateData as the base type.
>
> Is it important? We'll provide a opaque member for
> extensions. Extensions should use the opaque member instead
> of extending Copy{From,To}StateData.

I think yes, because it could be a blocker for future improvements
that might require a large field to CopyFrom/ToStateData.

> I don't object your approach but we need a good way to
> measure performance. If we use this approach, we can omit it
> for now and we can revisit your approach later without
> breaking compatibility. How about using this approach if we
> can't find a good way to measure performance?

I think it would be better to hear more opinions about this idea and
then make a decision, rather than basing our decision on whether or
not we can measure its performance, so we can be more confident in the
idea we have chosen. While this idea has the above downside, it could
make sense because we always allocate the entire CopyFrom/ToStateData
even today in spite of some fields being not used at all in binary
format and it requires less implementation costs to hide the
for-core-only fields. On the other hand, another possible idea is that
we have three different structs for categories 1 (core-only), 2 (core
and extensions), and 3 (extension-only), and expose only 2 that has a
void pointer to 3. The core can allocate the memory for 1 that embeds
2 at the beginning of the fields. While this design looks cleaner and
we can minimize overheads due to indirect references, it would require
more implementation costs. Which method we choose, I think we need
performance measurements in several scenarios to check if performance
regressions don't happen unexpectedly.


Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  320. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2025-09-10T02:40:58Z

    Hi,

In <CAD21AoCidyfKcpf9-f2Np8kWgkM09c4TjnS1h1hcO_-CCbjeqw@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Tue, 9 Sep 2025 13:15:43 -0700,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

>> I don't object your approach but we need a good way to
>> measure performance. If we use this approach, we can omit it
>> for now and we can revisit your approach later without
>> breaking compatibility. How about using this approach if we
>> can't find a good way to measure performance?
> 
> I think it would be better to hear more opinions about this idea and
> then make a decision, rather than basing our decision on whether or
> not we can measure its performance, so we can be more confident in the
> idea we have chosen. While this idea has the above downside, it could
> make sense because we always allocate the entire CopyFrom/ToStateData
> even today in spite of some fields being not used at all in binary
> format and it requires less implementation costs to hide the
> for-core-only fields. On the other hand, another possible idea is that
> we have three different structs for categories 1 (core-only), 2 (core
> and extensions), and 3 (extension-only), and expose only 2 that has a
> void pointer to 3. The core can allocate the memory for 1 that embeds
> 2 at the beginning of the fields. While this design looks cleaner and
> we can minimize overheads due to indirect references, it would require
> more implementation costs. Which method we choose, I think we need
> performance measurements in several scenarios to check if performance
> regressions don't happen unexpectedly.

OK. So the next step is collecting more opinions, right?

Could you add key people in this area to Cc to hear their
opinions? I'm not familiar with key people in the PostgreSQL
community...


Thanks,
-- 
kou

  321. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2025-09-10T07:36:38Z

    On Tue, Sep 9, 2025 at 7:41 PM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <CAD21AoCidyfKcpf9-f2Np8kWgkM09c4TjnS1h1hcO_-CCbjeqw@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Tue, 9 Sep 2025 13:15:43 -0700,
>   Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> >> I don't object your approach but we need a good way to
> >> measure performance. If we use this approach, we can omit it
> >> for now and we can revisit your approach later without
> >> breaking compatibility. How about using this approach if we
> >> can't find a good way to measure performance?
> >
> > I think it would be better to hear more opinions about this idea and
> > then make a decision, rather than basing our decision on whether or
> > not we can measure its performance, so we can be more confident in the
> > idea we have chosen. While this idea has the above downside, it could
> > make sense because we always allocate the entire CopyFrom/ToStateData
> > even today in spite of some fields being not used at all in binary
> > format and it requires less implementation costs to hide the
> > for-core-only fields. On the other hand, another possible idea is that
> > we have three different structs for categories 1 (core-only), 2 (core
> > and extensions), and 3 (extension-only), and expose only 2 that has a
> > void pointer to 3. The core can allocate the memory for 1 that embeds
> > 2 at the beginning of the fields. While this design looks cleaner and
> > we can minimize overheads due to indirect references, it would require
> > more implementation costs. Which method we choose, I think we need
> > performance measurements in several scenarios to check if performance
> > regressions don't happen unexpectedly.
>
> OK. So the next step is collecting more opinions, right?
>
> Could you add key people in this area to Cc to hear their
> opinions? I'm not familiar with key people in the PostgreSQL
> community...

How about another idea like we move format-specific data to another
struct that embeds CopyFrom/ToStateData at the first field and have
CopyFrom/ToStart callback return memory with the size of that
struct?It resolves the concerns about adding an extra indirection
layer and extensions doesn't need to allocate memory for unnecessary
fields (used only for built-in formats). While extensions can access
the internal fields I think we can live with that given that there are
some similar precedents such as table AM's scan descriptions.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  322. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2025-09-11T05:46:27Z

    Hi,

In <CAD21AoBb3t7EcsjYT4w68p9OfMNwWTYsbSVaSRY6DRhi7sNRFg@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Wed, 10 Sep 2025 00:36:38 -0700,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

> How about another idea like we move format-specific data to another
> struct that embeds CopyFrom/ToStateData at the first field and have
> CopyFrom/ToStart callback return memory with the size of that
> struct?It resolves the concerns about adding an extra indirection
> layer and extensions doesn't need to allocate memory for unnecessary
> fields (used only for built-in formats). While extensions can access
> the internal fields I think we can live with that given that there are
> some similar precedents such as table AM's scan descriptions.

The another idea looks like the following, right?

struct CopyToStateBuiltInData
{
  struct CopyToStateData parent;

  /* Members for built-in formats */
  ...;
}

typedef CopyToState *(*CopyToStart) (void);

CopyToState
BeginCopyTo(..., CopyToStart copy_to_start)
{
  ...;

  /* Allocate workspace and zero all fields */
  cstate = copy_to_start();
  ...;
}

This idea will almost work. But we can't know which
CopyToStart should be used before we parse "FORMAT" option
of COPY.

If we can iterate options twice in BeginCopy{To,From}(), we
can know it. For example:

BeginCopyTo(...)
{
  ...;

  CopyToStart copy_to_start = NULL;
  foreach(option, options)
  {
    DefElem  *defel = lfirst_node(DefElem, option);

    if (strcmp(defel->defname, "format") == 0)
    {
       char *fmt = defGetString(defel);
       if (strcmp(fmt, "text") == 0 ||
           strcmp(fmt, "csv") == 0 ||
           strcmp(fmt, "binary") == 0) {
         /* Use the builtin cstate */
       } else {
         copy_to_start = /* Detect CopyToStart for custom format */;
       }
    }
  }
  if (copy_to_start)
    cstate = copy_to_start();
  else 
    cstate = (CopyToStateData *) palloc0(sizeof(CopyToStateBuiltInData));
  ...;
}

(It may be better that we add
Copy{To,From}Routine::Copy{To,From}Allocate() instead of
CopyToStart callback.)

I think that this is acceptable because this must be a light
process. This must not have negative performance impact.


Thanks,
-- 
kou

  323. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2025-09-11T20:41:26Z

    On Wed, Sep 10, 2025 at 10:46 PM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <CAD21AoBb3t7EcsjYT4w68p9OfMNwWTYsbSVaSRY6DRhi7sNRFg@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Wed, 10 Sep 2025 00:36:38 -0700,
>   Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> > How about another idea like we move format-specific data to another
> > struct that embeds CopyFrom/ToStateData at the first field and have
> > CopyFrom/ToStart callback return memory with the size of that
> > struct?It resolves the concerns about adding an extra indirection
> > layer and extensions doesn't need to allocate memory for unnecessary
> > fields (used only for built-in formats). While extensions can access
> > the internal fields I think we can live with that given that there are
> > some similar precedents such as table AM's scan descriptions.
>
> The another idea looks like the following, right?
>
> struct CopyToStateBuiltInData
> {
>   struct CopyToStateData parent;
>
>   /* Members for built-in formats */
>   ...;
> }
>
> typedef CopyToState *(*CopyToStart) (void);
>
> CopyToState
> BeginCopyTo(..., CopyToStart copy_to_start)
> {
>   ...;
>
>   /* Allocate workspace and zero all fields */
>   cstate = copy_to_start();
>   ...;
> }

Right.

> This idea will almost work. But we can't know which
> CopyToStart should be used before we parse "FORMAT" option
> of COPY.
>
> If we can iterate options twice in BeginCopy{To,From}(), we
> can know it. For example:
>
> BeginCopyTo(...)
> {
>   ...;
>
>   CopyToStart copy_to_start = NULL;
>   foreach(option, options)
>   {
>     DefElem  *defel = lfirst_node(DefElem, option);
>
>     if (strcmp(defel->defname, "format") == 0)
>     {
>        char *fmt = defGetString(defel);
>        if (strcmp(fmt, "text") == 0 ||
>            strcmp(fmt, "csv") == 0 ||
>            strcmp(fmt, "binary") == 0) {
>          /* Use the builtin cstate */
>        } else {
>          copy_to_start = /* Detect CopyToStart for custom format */;
>        }
>     }
>   }
>   if (copy_to_start)
>     cstate = copy_to_start();
>   else
>     cstate = (CopyToStateData *) palloc0(sizeof(CopyToStateBuiltInData));
>   ...;
> }
>
> (It may be better that we add
> Copy{To,From}Routine::Copy{To,From}Allocate() instead of
> CopyToStart callback.)

I think we can use a local variable of CopyFormatOptions and memcpy it
to the opts of the returned cstate.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  324. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2025-09-12T00:07:52Z

    Hi,

In <CAD21AoCfqD=f2ELqPxg62+_QADhHi_kJXCDMhAerBtvxudd-xQ@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Thu, 11 Sep 2025 13:41:26 -0700,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

> I think we can use a local variable of CopyFormatOptions and memcpy it
> to the opts of the returned cstate.

It'll work too. Can we proceed this proposal with this
approach? Should we collect more opinions before we proceed?
If so, Could you add key people in this area to Cc to hear
their opinions?


Thanks,
-- 
kou

  325. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2025-09-15T17:00:18Z

    On Thu, Sep 11, 2025 at 5:07 PM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <CAD21AoCfqD=f2ELqPxg62+_QADhHi_kJXCDMhAerBtvxudd-xQ@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Thu, 11 Sep 2025 13:41:26 -0700,
>   Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> > I think we can use a local variable of CopyFormatOptions and memcpy it
> > to the opts of the returned cstate.
>
> It'll work too. Can we proceed this proposal with this
> approach? Should we collect more opinions before we proceed?
> If so, Could you add key people in this area to Cc to hear
> their opinions?

Since we don't have a single decision-maker, we should proceed through
consensus-building and careful evaluation of each approach. I see that
several senior hackers are already included in this thread, which is
excellent. Since you and I, who have been involved in these
discussions, agreed with this approach, I believe we can proceed with
this direction. If anyone proposes alternative solutions that we find
more compelling, we might have to change the approach.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  326. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2025-10-03T07:06:50Z

    Hi,

In <CAD21AoADXWgdizS0mV5w8wdfftDRsm8sUtNW=CzYYS1OhjFD2A@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Mon, 15 Sep 2025 10:00:18 -0700,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

>> > I think we can use a local variable of CopyFormatOptions and memcpy it
>> > to the opts of the returned cstate.
>>
>> It'll work too. Can we proceed this proposal with this
>> approach? Should we collect more opinions before we proceed?
>> If so, Could you add key people in this area to Cc to hear
>> their opinions?
> 
> Since we don't have a single decision-maker, we should proceed through
> consensus-building and careful evaluation of each approach. I see that
> several senior hackers are already included in this thread, which is
> excellent. Since you and I, who have been involved in these
> discussions, agreed with this approach, I believe we can proceed with
> this direction. If anyone proposes alternative solutions that we find
> more compelling, we might have to change the approach.

OK. There is no objection for now.

How about the attached patch? The patch uses the approach
only for CopyToStateData. If this looks good, I can do it
for CopyFromStateData too.

This patch splits CopyToStateData to

* CopyToStateData
* CopyToStateInternalData
* CopyToStateBuiltinData

structs.

This is based on the category described in
https://www.postgresql.org/message-id/flat/CAD21AoBa0Wm3C2H12jaqkvLidP2zEhsC%2Bgf%3D3w7XiA4LQnvx0g%40mail.gmail.com#85cb988b0bec243d1e8dce699e02e009
:

> 1. fields used only the core (not by format callback)
> 2. fields used by both the core and format callbacks
> 3. built-in format specific fields (mostly for text and csv)

CopyToStateInternalData is for 1.
CopyToStateData is for 2.
CopyToStateBuiltinData is for 3.


This patch adds CopyToRoutine::CopyToGetStateSize() that
returns size of state struct for the routine. For example,
Built-in formats use sizeof(CopyToStateBuiltinData) for it.

BeginCopyTo() allocates sizeof(CopyToStateInternalData) +
CopyToGetStateSize() size continuous memory and uses the
front part as CopyToStateInternalData and the back part as
CopyToStateData/CopyToStateBuilinData.


Thanks,
-- 
kou

  327. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2025-10-13T21:40:31Z

    On Fri, Oct 3, 2025 at 12:06 AM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <CAD21AoADXWgdizS0mV5w8wdfftDRsm8sUtNW=CzYYS1OhjFD2A@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Mon, 15 Sep 2025 10:00:18 -0700,
>   Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> >> > I think we can use a local variable of CopyFormatOptions and memcpy it
> >> > to the opts of the returned cstate.
> >>
> >> It'll work too. Can we proceed this proposal with this
> >> approach? Should we collect more opinions before we proceed?
> >> If so, Could you add key people in this area to Cc to hear
> >> their opinions?
> >
> > Since we don't have a single decision-maker, we should proceed through
> > consensus-building and careful evaluation of each approach. I see that
> > several senior hackers are already included in this thread, which is
> > excellent. Since you and I, who have been involved in these
> > discussions, agreed with this approach, I believe we can proceed with
> > this direction. If anyone proposes alternative solutions that we find
> > more compelling, we might have to change the approach.
>
> OK. There is no objection for now.
>
> How about the attached patch? The patch uses the approach
> only for CopyToStateData. If this looks good, I can do it
> for CopyFromStateData too.
>
> This patch splits CopyToStateData to
>
> * CopyToStateData
> * CopyToStateInternalData
> * CopyToStateBuiltinData
>
> structs.
>
> This is based on the category described in
> https://www.postgresql.org/message-id/flat/CAD21AoBa0Wm3C2H12jaqkvLidP2zEhsC%2Bgf%3D3w7XiA4LQnvx0g%40mail.gmail.com#85cb988b0bec243d1e8dce699e02e009
> :
>
> > 1. fields used only the core (not by format callback)
> > 2. fields used by both the core and format callbacks
> > 3. built-in format specific fields (mostly for text and csv)
>
> CopyToStateInternalData is for 1.
> CopyToStateData is for 2.
> CopyToStateBuiltinData is for 3.
>
>
> This patch adds CopyToRoutine::CopyToGetStateSize() that
> returns size of state struct for the routine. For example,
> Built-in formats use sizeof(CopyToStateBuiltinData) for it.
>
> BeginCopyTo() allocates sizeof(CopyToStateInternalData) +
> CopyToGetStateSize() size continuous memory and uses the
> front part as CopyToStateInternalData and the back part as
> CopyToStateData/CopyToStateBuilinData.

Thank you for drafting the idea!

The patch refactors the CopyToStateData so that we can both hide
internal-use-only fields from extensions and extension can use its own
state data, while not adding extra indirection layers. TBH I'm really
not sure we must fully hide internal fields from extensions. Other
extendable components seem not to strictly hide internal information
from extensions. I'd suggest starting with only the latter point. That
is, we merge fields in CopyToStateInternalData to CopyToStateData.
What do you think?

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  328. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2025-10-14T02:15:24Z

    Hi,

In <CAD21AoBkA=g=PN17r_iieru+vLyLtGZ8WvohgANa2vzsMfMogQ@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Mon, 13 Oct 2025 14:40:31 -0700,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

> The patch refactors the CopyToStateData so that we can both hide
> internal-use-only fields from extensions and extension can use its own
> state data, while not adding extra indirection layers. TBH I'm really
> not sure we must fully hide internal fields from extensions. Other
> extendable components seem not to strictly hide internal information
> from extensions. I'd suggest starting with only the latter point. That
> is, we merge fields in CopyToStateInternalData to CopyToStateData.
> What do you think?

OK. Let's follow the existing style. How about the attached
patch? It merges CopyToStateInternalData to CopyToStateData.


Thanks,
-- 
kou

  329. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2025-10-29T20:41:39Z

    On Mon, Oct 13, 2025 at 7:15 PM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <CAD21AoBkA=g=PN17r_iieru+vLyLtGZ8WvohgANa2vzsMfMogQ@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Mon, 13 Oct 2025 14:40:31 -0700,
>   Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> > The patch refactors the CopyToStateData so that we can both hide
> > internal-use-only fields from extensions and extension can use its own
> > state data, while not adding extra indirection layers. TBH I'm really
> > not sure we must fully hide internal fields from extensions. Other
> > extendable components seem not to strictly hide internal information
> > from extensions. I'd suggest starting with only the latter point. That
> > is, we merge fields in CopyToStateInternalData to CopyToStateData.
> > What do you think?
>
> OK. Let's follow the existing style. How about the attached
> patch? It merges CopyToStateInternalData to CopyToStateData.
>

The basic idea of this patch makes sense to me.

Andres, I believe that the current idea deals with your concerns about
performance overheads. Particularly, we separate format-specific
fields (c.f. CopyToStateBuiltinData struct in the patch) from the
commonly-used fields (c.f., CopyToStateData struct), but the whole
fields are stored in the contiguous memory. While the patch needs to
be polished much, could you review if the basic idea of this patch
addresses your concern?

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  330. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2025-11-14T20:19:47Z

    On Wed, Oct 29, 2025 at 1:41 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Mon, Oct 13, 2025 at 7:15 PM Sutou Kouhei <kou@clear-code.com> wrote:
> >
> > Hi,
> >
> > In <CAD21AoBkA=g=PN17r_iieru+vLyLtGZ8WvohgANa2vzsMfMogQ@mail.gmail.com>
> >   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Mon, 13 Oct 2025 14:40:31 -0700,
> >   Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > > The patch refactors the CopyToStateData so that we can both hide
> > > internal-use-only fields from extensions and extension can use its own
> > > state data, while not adding extra indirection layers. TBH I'm really
> > > not sure we must fully hide internal fields from extensions. Other
> > > extendable components seem not to strictly hide internal information
> > > from extensions. I'd suggest starting with only the latter point. That
> > > is, we merge fields in CopyToStateInternalData to CopyToStateData.
> > > What do you think?
> >
> > OK. Let's follow the existing style. How about the attached
> > patch? It merges CopyToStateInternalData to CopyToStateData.
> >
>
> The basic idea of this patch makes sense to me.
>

This thread has involved extensive discussion, and the patch needs to
be rebased. I'd like to summarize the current status of this patch and
our discussions. I've attached updated patches that implement the
whole ideas of this feature to help provide a clearer overall picture.

In commits 2e4127b6d and 7717f6300, we refactored COPY TO/FROM code to
use a set of callbacks for format-specific operations like data
parsing. These callbacks are currently not exposed and are only used
by built-in formats. The next step is to allow extensions to register
their own format implementations, which these attached patches
accomplish. The registration API can be called in _PG_init() as
follows to enable users to specify a custom format name ('jsonlines'
in this example) in the FORMAT option:

RegisterCopyCustomFormat("jsonlines",
                       &JsonLinesCopyFromRoutine,
                       &JsonLinesCopyToRoutine);

However, before introducing the registration API, we need to resolve
an issue with how we currently use a monolithic struct
(Copy{From,To}StateData) to store COPY TO/FROM state data. This struct
currently contains both format-agnostic fields (e.g., target relation
and source file) and format-specific fields (e.g., input buffers and
EOL type). Patches 0001 and 0002 reorganize these fields to separate
them. Specifically, format-specific fields are moved to a new struct
while Copy{From,To}StateData retains format-agnostic fields, as shown
here (e.g., COPY TO case):

typedef struct CopyToStateData
{
    /* format-specific routines */
    const struct CopyToRoutine *routine;

    /* low-level state data */
    CopyDest    copy_dest;      /* type of copy source/destination */
    FILE       *copy_file;      /* used if copy_dest == COPY_DEST_FILE */
    StringInfo  fe_msgbuf;      /* used for all dests during COPY TO */

    (snip)

    /*
     * Working state
     */
    MemoryContext copycontext;  /* per-copy execution context */

    FmgrInfo   *out_functions;  /* lookup info for output functions */
    MemoryContext rowcontext;   /* per-row evaluation context */
    uint64      bytes_processed;    /* number of bytes processed so far */
} CopyToStateData;

typedef struct CopyToStateTextLike
{
    CopyToStateData base; /* embedded */

    int             file_encoding;
    bool            need_transcoding;
    bool            encoding_embeds_ascii;
} CopyToStateTextLike;

Extensions must specify their required state struct size (like
CopyToStateTextLike for built-in formats) using new callbacks
Copy{From,To}EstimateStateSpace, allowing the core to allocate the
appropriate amount of memory. This approach offers two advantages:

- Format processing implementations only use the memory they need
- No additional pointer traversal compared to using an opaque pointer
for format-specific data

Patches 0003 through 0006 implement the following:

0003: Introduces the RegisterCustomCopyFormat() API
0004 and 0005: Enable custom format implementations to register their
own COPY command options
0006: Adds regression tests

 With these patches, here's what we can do using the 'jsonlines'
format extension:

-- COPY TO
CREATE TABLE jl (id int, a text, b jsonb);
INSERT INTO jl VALUES (1, 'hello', '{"test" : [1, true, {"num" :
42}]}'::jsonb), (2, 'hello world', 'true'), (999, null, '{"a" : 1}');
TABLE jl;
 id  |      a      |                b
-----+-------------+----------------------------------
   1 | hello       | {"test": [1, true, {"num": 42}]}
   2 | hello world | true
 999 |             | {"a": 1}
(3 rows)

COPY jl TO '/tmp/test.jsonl' WITH (format 'jsonlines');
\! cat /tmp/test.jsonl
{"id":1,"a":"hello","b":{"test": [1, true, {"num": 42}]}}
{"id":2,"a":"hello world","b":true}
{"id":999,"a":null,"b":{"a": 1}}

-- COPY FROM
CREATE TABLE jl_load (id int, a text, b jsonb);
COPY jl_load FROM '/tmp/test.jsonl' WITH (format 'jsonlines');

-- COPY TO/FROM with custom options
COPY jl TO '/tmp/jl.jsonl.gz' WITH (format 'jsonlines', compression
'gzip', compression_detail 'level=2');
COPY jl FROM '/tmp/jl.jsonl.gz' WITH (format 'jsonlines');

To demonstrate the functionality of both current and new APIs,
Suto-san and I have created several experimental custom COPY format
extensions:

Apache Arrow (developed by Sutou-san): https://github.com/kou/pg-copy-arrow

JSON Lines (developed by me):
https://github.com/MasahikoSawada/pg_custom_copy_formats/blob/master/jsonlines.c

These implementations serve as good examples of how extensions can use
these APIs to define custom COPY formats.

After offline discussions with Sutou-san, we believe the current APIs
work well, particularly for text-based formats, though we still need
to verify there are no performance regressions.

One potential improvement would be adding support for random file
access in COPY FROM operations. For example, with parquet files, it
would be much more efficient to read the footer section first since it
contains metadata, allowing selective reading of necessary file
sections. The current sequential read API (CopyFromGetData()) requires
reading all data to access the metadata.

For future consideration, we could look into supporting file
reading/writing from external sources like S3. While this is outside
the scope of this patch, we discussed that allowing the core to
delegate I/O operations to custom format implementations might be a
good starting point. We can discuss this in a separate thread.

I welcome your feedback on these proposed changes and APIs to help
move this patch forward.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  331. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2025-11-17T06:40:13Z

    Hi,

In <CAD21AoDCEfe0PQhMEx8G1rpS7RrzGCJPobeqm3Mpn2bgbUH9nQ@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 14 Nov 2025 12:19:47 -0800,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

> This thread has involved extensive discussion, and the patch needs to
> be rebased. I'd like to summarize the current status of this patch and
> our discussions. I've attached updated patches that implement the
> whole ideas of this feature to help provide a clearer overall picture.

Thanks!

I'm not sure whether we should include option parsing
feature to this patch's scope or not but I'm OK with this
approach.

Here are my review comments but they are minor
comments. They don't require design change.

0001:

diff --git a/src/backend/commands/copyto.c b/src/backend/commands/copyto.c
index cef452584e5..6a0a66507ba 100644
--- a/src/backend/commands/copyto.c
+++ b/src/backend/commands/copyto.c
@@ -155,6 +120,7 @@ static void CopySendInt16(CopyToState cstate, int16 val);
 
 /* text format */
 static const CopyToRoutine CopyToRoutineText = {
+	.CopyToEstimateStateSpace = CopyToEstimateStateTextLike,

How about including "Space"
(CopyToEstimateStateSpaceTextLike)?

How about renaming this to
"CopyToTextLikeEstimateStateSpace" because other functions
use "CopyToTextLikeXXX" style.

@@ -171,62 +138,77 @@ static const CopyToRoutine CopyToRoutineCSV = {
...
 /* Implementation of the start callback for text and CSV formats */
 static void
-CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc)
+CopyToTextLikeStart(CopyToState ccstate, TupleDesc tupDesc)
 {
+	CopyToStateTextLike *cstate = (CopyToStateTextLike *) ccstate;

How about always using "cstate" for "CopyToState"? In other
functions in this patch, "CopyToState" is referred "cstate"
or "cctate".

We can use "state" or something for
"CopyToStateTextLike". (0002 uses "state" for
"CopyFromStateBuiltins".)

+		cstate->base.opts.null_print_client = pg_server_to_any(cstate->base.opts.null_print,
+															   cstate->base.opts.null_print_len,
+															   cstate->file_encoding);

We can use "ccstate->" instead of "cstate->base." in this
function.

@@ -614,6 +603,54 @@ EndCopy(CopyToState cstate)
 	pfree(cstate);
 }
 
+/*
+ * Allocate COPY TO state data based on the format's EsimateStateSpace
+ * callback.
+ */
+static CopyToState
+create_copyto_state(ParseState *pstate, List *options)

"Esimate" ->
"Estimate"

How about using CamelCase like "CreateCopyToState" for
function name like other functions in this file?

+	Size		req_size;

"state_size" may be better.

@@ -1233,16 +1246,16 @@ CopyOneRowTo(CopyToState cstate, TupleTableSlot *slot)
...
 static void
-CopyAttributeOutText(CopyToState cstate, const char *string)
+CopyAttributeOutText(CopyToStateTextLike * cstate, const char *string)
 {

CopyAttributeOutText(CopyToState cstate, const char *string)
{
	CopyToStateTextLike *state = (CopyToStateTextLike *)cstate;

may reduce "cstate->base." and "(CopyToState) cstate" in
this function.

BTW, could you remove a needless space after "*" in
"CopyToStateTextLike * cstate"?


0002:

diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c
index 12781963b4f..0c51e5ba5e1 100644
--- a/src/backend/commands/copyfrom.c
+++ b/src/backend/commands/copyfrom.c
@@ -129,6 +131,7 @@ static void CopyFromBinaryEnd(CopyFromState cstate);
 
 /* text format */
 static const CopyFromRoutine CopyFromRoutineText = {
+	.CopyFromEstimateStateSpace = CopyFromBuiltinsEstimateSpace,

How about including "State"
("CopyFromBuiltinsEstimateStateSpace")?

@@ -145,54 +149,129 @@ static const CopyFromRoutine CopyFromRoutineCSV = {
...
+/*
+ * Common routine to initialize CopyFromStateBuiltins data.
+ */
+static void
+initialize_copyfrom_bultins_state(CopyFromStateBuiltins * state, TupleDesc tupDesc)

* How about using CamelCase like other functions in this file?
* How about using the same name as the struct?

InitializeCopyFromStateBuiltins?

@@ -1379,8 +1462,8 @@ CopyFrom(CopyFromState cstate)
 					/* Add this tuple to the tuple buffer */
 					CopyMultiInsertInfoStore(&multiInsertInfo,
 											 resultRelInfo, myslot,
-											 cstate->line_buf.len,
-											 cstate->cur_lineno);
+											 rowinfo.tuplen,
+											 rowinfo.lineno);

How about passing "CopyFromRowInfo *" instead of
"rowinfo.tuplen" and "rowinfo.lineno"?

@@ -1512,6 +1595,50 @@ CopyFrom(CopyFromState cstate)
 	return processed;
 }
 
+static CopyFromState
+create_copyfrom_state(ParseState *pstate, List *options)

How about using CamelCase like other functions in this file?

CreateCopyFromState?

@@ -1138,19 +1156,29 @@ CopyFromBinaryOneRow(CopyFromState cstate, ExprContext *econtext, Datum *values,
...
+	if (rowinfo)
+	{
+		/*
+		 * XXX: We used to use line_buf.len but we don't actually use line_buf
+		 * in binary format.
+		 */
+		rowinfo->lineno = cstate->base.cur_lineno;
+		rowinfo->tuplen = cstate->line_buf.len;
 	}

How about always setting "0" to "rowinfo->tuplen" instead of
using "cstate->line_buf.len"?

diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h
index d75a70715a4..30a1d2bff6e 100644
--- a/src/include/commands/copy.h
+++ b/src/include/commands/copy.h
@@ -48,6 +48,12 @@ typedef enum CopyLogVerbosityChoice
...
+typedef struct CopyFromRowInfo
+{
+	uint64		lineno;
+	int			tuplen;
+}			CopyFromRowInfo;

I don't have a strong opinion nor alternative but I'm not
sure whether this name is suitable or not...


diff --git a/src/include/commands/copyapi.h b/src/include/commands/copyapi.h
index 3b9982d54b8..c3d2199a0b6 100644
--- a/src/include/commands/copyapi.h
+++ b/src/include/commands/copyapi.h
@@ -99,12 +104,11 @@ typedef struct CopyFromRoutine
 	 * Returns false if there are no more tuples to read.
 	 */
 	bool		(*CopyFromOneRow) (CopyFromState cstate, ExprContext *econtext,
-								   Datum *values, bool *nulls);
+								   Datum *values, bool *nulls, CopyFromRowInfo * rowinfo);

Can we add some docstrings for "rowinfo"?


diff --git a/src/include/commands/copystate.h b/src/include/commands/copystate.h
index 7561083a323..145dccd0f8f 100644
--- a/src/include/commands/copystate.h
+++ b/src/include/commands/copystate.h
@@ -65,4 +67,99 @@ typedef struct CopyToStateData
...
+/*
+ * Represents the different source cases we need to worry about at
+ * the bottom level
+ */
+typedef enum CopySource
+{
+	COPY_FILE,					/* from file (or a piped program) */
+	COPY_FRONTEND,				/* from frontend */
+	COPY_CALLBACK,				/* from callback function */
+} CopySource;

Can we use "COPY_SOURCE_" prefix instead of "COPY_" prefix
such as "COPY_SOURCE_FILE"?


0003:

diff --git a/src/backend/commands/copy_custom_format.c b/src/backend/commands/copy_custom_format.c
new file mode 100644
index 00000000000..8bef6e779ac
--- /dev/null
+++ b/src/backend/commands/copy_custom_format.c

+/*
+ * Returns true if the given custom format name is registered.
+ */
+bool
+FindCustomCopyFormat(const char *fmt_name)
+{
+	for (int i = 0; i < CopyCustomFormatAssigned; i++)
+	{
+		if (strcmp(CopyCustomFormatArray[i].fmt_name, fmt_name) == 0)
+			return true;
+	}
+
+	return false;
+}

How about using other word than "Find" here? I expect that
"FindXXX()" returns a found value instead of "whether found
or not" as bool.

CustomCopyFormatExists()?

+bool
+GetCustomCopyToRoutine(const char *fmt_name, const CopyToRoutine **to_routine)
+{
+	for (int i = 0; i < CopyCustomFormatAssigned; i++)
+	{
+		if (strcmp(CopyCustomFormatArray[i].fmt_name, fmt_name) == 0)
+		{
+			*to_routine = CopyCustomFormatArray[i].to_routine;
+			return true;
+		}
+	}
+
+	return false;
+}

How about returning "const CopyToRoutine *" instead of
"bool"? We can use "FindCustomCopyFormat()" whether the
given format name exists or not.

+bool
+GetCustomCopyFromRoutine(const char *fmt_name, const CopyFromRoutine **from_routine)

Ditto.

diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h
index 30a1d2bff6e..82f07b05823 100644
--- a/src/include/commands/copy.h
+++ b/src/include/commands/copy.h
@@ -120,6 +120,12 @@ extern uint64 CopyFrom(CopyFromState cstate);
 
 extern DestReceiver *CreateCopyDestReceiver(void);
 
+extern void ProcessCopyBuiltinOptions(List *options, CopyFormatOptions *opts_out,
+									  bool is_from, List **other_options, ParseState *pstate);

It seems that this change should exist in 0004.


0004:

diff --git a/src/backend/commands/copyto.c b/src/backend/commands/copyto.c
index b1b3ae141eb..ea31fa911f9 100644
--- a/src/backend/commands/copyto.c
+++ b/src/backend/commands/copyto.c
@@ -1570,3 +1571,32 @@ CreateCopyDestReceiver(void)
 
 	return (DestReceiver *) self;
 }
+
+static void
+ProcessCopyToOptions(CopyToState cstate, List *options, ParseState *pstate)
+{
+	bool		temp_state = false;
+	List	   *other_options = NIL;
+	CopyFormatOptions *opts;
+
+	if (cstate == NULL)
+	{
+		cstate = create_copyto_state(pstate, options);
+		temp_state = true;
+	}
+
+	opts = &cstate->opts;
+
+	ProcessCopyBuiltinOptions(options, opts, false, &other_options, pstate);
+
+	foreach_node(DefElem, option, options)

options ->
other_options

(This change exists in 0005.)


0006:

diff --git a/src/test/modules/test_copy_custom_format/test_copy_custom_format.c b/src/test/modules/test_copy_custom_format/test_copy_custom_format.c
new file mode 100644
index 00000000000..a63390e875b
--- /dev/null
+++ b/src/test/modules/test_copy_custom_format/test_copy_custom_format.c


+static Size
+TestCopyToEsimateSpace(void)
+{
+	return sizeof(TestCopyToState);
+}
+
+static Size
+TestCopyFromEsimateSpace(void)
+{
+	return sizeof(TestCopyFromState);
+}

EsimateSpace ->
EstimateStateSpace

+	if (strcmp(option->defname, "common_int") == 0)
+	{
+		int			val = defGetInt32(option);
+
+		opt->common_int = val;
+
+		return true;
+	}
+	else if (strcmp(option->defname, "common_bool") == 0)
+	{
+		bool		val = defGetBoolean(option);
+
+		opt->common_bool = val;
+
+		return true;

We may not need to use local variables:

opt->common_int = defGetInt32(option);
opt->common_bool = defGetBoolean(option);


> One potential improvement would be adding support for random file
> access in COPY FROM operations. For example, with parquet files, it
> would be much more efficient to read the footer section first since it
> contains metadata, allowing selective reading of necessary file
> sections. The current sequential read API (CopyFromGetData()) requires
> reading all data to access the metadata.

This is outside the scope of this patch but I've created
a custom COPY format implementation for Apache Parquet:
https://github.com/kou/pg-copy-parquet

We need to start parsing from footer as mentioned above. So
the implementation reads all data before it starts parsing:

https://github.com/kou/pg-copy-parquet/blob/7da367ea81d8964f5045fe0b1514a798d4ecbbc7/copy_parquet.cc#L410-L434

If we have random access API, we don't need to read all
data. It improves performance.

Anyway, this is outside the scope of this patch. We can
discuss this in a separated thread after we merge this
patch.


Thanks,
-- 
kou

  332. Re: Make COPY format extendable: Extract COPY TO format implementations

    Tomas Vondra <tomas@vondra.me> — 2025-11-17T17:04:46Z

    On 11/14/25 21:19, Masahiko Sawada wrote:
> After offline discussions with Sutou-san, we believe the current APIs
> work well, particularly for text-based formats, though we still need
> to verify there are no performance regressions.

I got pinged about this patch off-list. I won't have capacity to do a
proper review, anytime soon, but I got a bit of time to do a simple
benchmark (which seems useful as that was one of the concerns in this
thread, it seems).

Attached is a script that does COPY TO/FROM with the built-in formats,
on table with 1, 10 and 100 integer columns. The data sets are between
10 and 1M rows. The table is UNLOGGED, to eliminate WAL overhead.

The attached PDF summarizes results from my ryzen machine, for master
and patched build. The final columns are comparison (i.e. copy/master)
of the timings. Values >100% are regressions (marked as red).

It seems quite "red", but it's not particularly conclusive. The
differences are mostly within 5%, and that could be caused e.g. by
changes to binary layout. And some of the cases got faster too.

It might be interesting to get results from other machines. The script
may need some adjustments, but it should be too difficult.

The other thing Andres was concerned about is "the amount of COPY
performance improvements it forecloses". I have no opinion on that, as
it depends on what improvements Andres envisioned.


regards

-- 
Tomas Vondra

  333. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2025-12-02T02:39:57Z

    Hi,

In <c36d218a-bb38-42b9-9076-cb75b8984a39@vondra.me>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Mon, 17 Nov 2025 18:04:46 +0100,
  Tomas Vondra <tomas@vondra.me> wrote:

> I got pinged about this patch off-list. I won't have capacity to do a
> proper review, anytime soon, but I got a bit of time to do a simple
> benchmark (which seems useful as that was one of the concerns in this
> thread, it seems).

Thanks!!!

I also do the same condition benchmark on my Mac mini:

Machine:

* Apple M1 (8 core)
* Memory: 16GB
* macOS Sequoia (15.6)

Parameters:

* N integer columns: 1 10 100
* N rows: 10 100 1000 10000 10000 100000 1000000
* Formats: text csv binary
* Operations: FROM TO
* Patches: 0001 0002 0003 0004 0005 0006

I ran 5 times for each parameter set and choose the median
elapsed time.

I used
https://gitlab.com/ktou/pg-bench/-/blob/main/copy-format-extendable/run.sh
. I attach it.

I measured 6 times. See the attached mac-mini-result-${N}.{csv,pdf}.

A PDF has 3 columns. They are text, csv and binary formats
from left to right.

1st row uses COPY FROM and 0001 patch.
2nd row uses COPY TO and 0001 patch.

3rd row uses COPY FROM and 0002 patch.
4th row uses COPY TO and 0002 patch.

...

Each heatmap visualizes (${elapsed_time_patch} /
${elapsed_time_master}) * 100. 100 > (red) means slower and
100 < (blue) means faster.

It seems that they don't show any reproducible trends.

For example, the binary cases in mac-mini-result-2.pdf show
that patched cases are always slower but the binary cases in
mac-mini-result-{1,6}.pdf show that most patched cases are
faster. The binary cases in mac-mini-result-{1,5,6}.pdf show
that 0006 patch is slower than 0005 patch but the binary
cases in mac-mini-result-{3,4}.pdf don't show it.

Another example, the text cases in mac-mini-result-1.pdf
show that patched cases are always slower but the text cases
in mac-mini-result-1.pdf show that most patched cases are
faster.

I hope that these numbers help to proceed this proposal.


Thanks,
-- 
kou

  334. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2025-12-18T23:43:07Z

    On Mon, Dec 1, 2025 at 6:41 PM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <c36d218a-bb38-42b9-9076-cb75b8984a39@vondra.me>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Mon, 17 Nov 2025 18:04:46 +0100,
>   Tomas Vondra <tomas@vondra.me> wrote:
>
> > I got pinged about this patch off-list. I won't have capacity to do a
> > proper review, anytime soon, but I got a bit of time to do a simple
> > benchmark (which seems useful as that was one of the concerns in this
> > thread, it seems).
>
> Thanks!!!
>
> I also do the same condition benchmark on my Mac mini:
>
> Machine:
>
> * Apple M1 (8 core)
> * Memory: 16GB
> * macOS Sequoia (15.6)
>
> Parameters:
>
> * N integer columns: 1 10 100
> * N rows: 10 100 1000 10000 10000 100000 1000000
> * Formats: text csv binary
> * Operations: FROM TO
> * Patches: 0001 0002 0003 0004 0005 0006
>
> I ran 5 times for each parameter set and choose the median
> elapsed time.
>
> I used
> https://gitlab.com/ktou/pg-bench/-/blob/main/copy-format-extendable/run.sh
> . I attach it.
>
> I measured 6 times. See the attached mac-mini-result-${N}.{csv,pdf}.
>
> A PDF has 3 columns. They are text, csv and binary formats
> from left to right.
>
> 1st row uses COPY FROM and 0001 patch.
> 2nd row uses COPY TO and 0001 patch.
>
> 3rd row uses COPY FROM and 0002 patch.
> 4th row uses COPY TO and 0002 patch.
>
> ...
>
> Each heatmap visualizes (${elapsed_time_patch} /
> ${elapsed_time_master}) * 100. 100 > (red) means slower and
> 100 < (blue) means faster.
>
> It seems that they don't show any reproducible trends.
>
> For example, the binary cases in mac-mini-result-2.pdf show
> that patched cases are always slower but the binary cases in
> mac-mini-result-{1,6}.pdf show that most patched cases are
> faster. The binary cases in mac-mini-result-{1,5,6}.pdf show
> that 0006 patch is slower than 0005 patch but the binary
> cases in mac-mini-result-{3,4}.pdf don't show it.
>
> Another example, the text cases in mac-mini-result-1.pdf
> show that patched cases are always slower but the text cases
> in mac-mini-result-1.pdf show that most patched cases are
> faster.
>
> I hope that these numbers help to proceed this proposal.

Thank you for sharing the performance test results! I'll run the same
benchmark tests on my environment.

Looking at these results, it seems that 0001-from-binary cases and
0006-to-binary cases are slower throughout the six results?

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  335. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2026-03-27T01:36:40Z

    Hi,

In <CAD21AoCLxUhQ0uBjDKXvCEtJBCfF13Ru_7u-Qrrsu+0PPUqcPQ@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Thu, 18 Dec 2025 15:43:07 -0800,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

> Looking at these results, it seems that 0001-from-binary cases and
> 0006-to-binary cases are slower throughout the six results?

Good point. I didn't notice them. But I feel that it's not
related to the patch set. Because 0001 doesn't change COPY
FROM related code. 0001 just changes COPY TO related
code. And 0006 just adds tests. 0006 doesn't change
implementations.


BTW, how to proceed this proposal? It seems that we can't
proceed this proposal without PostgreSQL committers'
attentions but it seems that it's difficult.


Thanks,
-- 
kou

  336. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2026-06-23T01:06:07Z

    Hi,

On Thu, Mar 26, 2026 at 6:36 PM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <CAD21AoCLxUhQ0uBjDKXvCEtJBCfF13Ru_7u-Qrrsu+0PPUqcPQ@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Thu, 18 Dec 2025 15:43:07 -0800,
>   Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> > Looking at these results, it seems that 0001-from-binary cases and
> > 0006-to-binary cases are slower throughout the six results?
>
> Good point. I didn't notice them. But I feel that it's not
> related to the patch set. Because 0001 doesn't change COPY
> FROM related code. 0001 just changes COPY TO related
> code. And 0006 just adds tests. 0006 doesn't change
> implementations.
>
>
> BTW, how to proceed this proposal? It seems that we can't
> proceed this proposal without PostgreSQL committers'
> attentions but it seems that it's difficult.

Sorry for going quiet on this for a while -- I haven't had time to
work on it until now.

After more thought, I'd like to keep the custom-format changes to the
bare minimum and not disturb the existing built-in format processing.

In particular, I've dropped the earlier rework that split
CopyToStateData / CopyFromStateData to hide built-in-specific fields
from extensions. That was my own idea, but I no longer think it pays
off: the fields it hid (raw_buf, line_buf, the input buffers, etc.)
are only ever used by the built-in text/CSV/binary parsers, and a
custom format never touches them -- so visible or not, nothing depends
on them, while splitting the struct is invasive to the existing format
processing. Touching the Copy state structs is fine in itself; it's
the hiding that wasn't worth the cost.

Instead, each state struct just gets one opaque pointer for a custom
format to keep its own state, and the existing code paths are left
alone.

Updated patches attached:

- 0001 moves CopyFromStateData and CopyToStateData to a new
copy_state.h, so extensions can implement their routines without
including the *_internal.h headers. It also drops file_fdw.c's
dependency on copyfrom_internal.h.
- 0002 introduces the registration API and the opaque per-format
pointer in both structs.
- 0003 adds a callback to validate the COPY options as a whole, called
after all options are processed.
- 0004 adds the regression tests.

I'd like to proceed in this direction barring objections. Feedback is
very welcome.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  337. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2026-06-23T05:41:49Z

    Hi,

Thanks for restarting this.

In <CAD21AoCnA7vayZAOmwVqTSOyWfyBhyxH7mBb4UzjskF-eZ+_Jg@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Mon, 22 Jun 2026 18:06:07 -0700,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

> After more thought, I'd like to keep the custom-format changes to the
> bare minimum and not disturb the existing built-in format processing.

+1

> Updated patches attached:
> 
> - 0001 moves CopyFromStateData and CopyToStateData to a new
> copy_state.h, so extensions can implement their routines without
> including the *_internal.h headers. It also drops file_fdw.c's
> dependency on copyfrom_internal.h.

+1

> - 0002 introduces the registration API and the opaque per-format
> pointer in both structs.

> --- /dev/null
> +++ b/src/backend/commands/copyapi.c

> +bool
> +GetCopyCustomFormatRoutines(const char *name, const CopyToRoutine **to,
> +							const CopyFromRoutine **from, ProcessOneOptionFn * option_fn)

How about returning CopyCustomFormatEntry instead? The
function name is "Get...Routines" but it also returns
ProcessOneOptionFn. "Get...Routines" is a bit strange.

> --- a/src/include/commands/copyapi.h
> +++ b/src/include/commands/copyapi.h

> @@ -102,4 +103,40 @@ typedef struct CopyFromRoutine
> ...
> +typedef bool (*ProcessOneOptionFn) (CopyFormatOptions *opts, bool is_from,
> +									DefElem *option);

How about adding "Copy" keyword to the type name such as
"ProcessOneCopyOptionFn" because this is only for COPY format?

> --- a/src/include/commands/copy.h
> +++ b/src/include/commands/copy.h

> @@ -58,7 +58,16 @@ typedef enum CopyFormat
> ...
> +#define CopyFormatBuiltins(format) ((format) != COPY_FORMAT_CUSTOM)

How about renaming this to CopyFormatIsBuiltin() or
something? "...Builtins" is a bit strange because this
returns a boolean.

> - 0003 adds a callback to validate the COPY options as a whole, called
> after all options are processed.

> --- a/src/include/commands/copyapi.h
> +++ b/src/include/commands/copyapi.h
> @@ -120,6 +120,15 @@ typedef struct CopyFromRoutine
> ...
> +typedef void (*ValidateOptionsFn) (CopyFormatOptions *opts, bool is_from);

How about adding "Copy" keyword like "ValidateCopyOptionsFn"?

> - 0004 adds the regression tests.

> --- /dev/null
> +++ b/src/test/modules/test_copy_custom_format/test_copy_custom_format.c

> @@ -0,0 +1,169 @@
> ...
> +TestCopyProcessOneOption(CopyFormatOptions *opts, bool is_from, DefElem *option)
> +{
> +	TestCopyOptions *t = (TestCopyOptions *) opts->format_private_opts;
> +
> +	if (t == NULL)
> +	{
> +		t = palloc0_object(TestCopyOptions);
> +		opts->format_private_opts = (void *) t;
> +	}

This is not a blocker but we may want to add
InitializeCopyOptions callback for this.


Thanks,
-- 
kou

  338. Re: Make COPY format extendable: Extract COPY TO format implementations

    Masahiko Sawada <sawada.mshk@gmail.com> — 2026-06-24T01:15:10Z

    On Mon, Jun 22, 2026 at 10:41 PM Sutou Kouhei <kou@clear-code.com> wrote:
>
> > - 0002 introduces the registration API and the opaque per-format
> > pointer in both structs.
>
> > --- /dev/null
> > +++ b/src/backend/commands/copyapi.c
>
> > +bool
> > +GetCopyCustomFormatRoutines(const char *name, const CopyToRoutine **to,
> > +                                                     const CopyFromRoutine **from, ProcessOneOptionFn * option_fn)
>
> How about returning CopyCustomFormatEntry instead? The
> function name is "Get...Routines" but it also returns
> ProcessOneOptionFn. "Get...Routines" is a bit strange.

Agreed.

>
> > --- a/src/include/commands/copyapi.h
> > +++ b/src/include/commands/copyapi.h
>
> > @@ -102,4 +103,40 @@ typedef struct CopyFromRoutine
> > ...
> > +typedef bool (*ProcessOneOptionFn) (CopyFormatOptions *opts, bool is_from,
> > +                                                                     DefElem *option);
>
> How about adding "Copy" keyword to the type name such as
> "ProcessOneCopyOptionFn" because this is only for COPY format?

Agreed.

>
> > --- a/src/include/commands/copy.h
> > +++ b/src/include/commands/copy.h
>
> > @@ -58,7 +58,16 @@ typedef enum CopyFormat
> > ...
> > +#define CopyFormatBuiltins(format) ((format) != COPY_FORMAT_CUSTOM)
>
> How about renaming this to CopyFormatIsBuiltin() or
> something? "...Builtins" is a bit strange because this
> returns a boolean.

Agreed.

>
> > - 0003 adds a callback to validate the COPY options as a whole, called
> > after all options are processed.
>
> > --- a/src/include/commands/copyapi.h
> > +++ b/src/include/commands/copyapi.h
> > @@ -120,6 +120,15 @@ typedef struct CopyFromRoutine
> > ...
> > +typedef void (*ValidateOptionsFn) (CopyFormatOptions *opts, bool is_from);
>
> How about adding "Copy" keyword like "ValidateCopyOptionsFn"?

Agreed.

>
> > - 0004 adds the regression tests.
>
> > --- /dev/null
> > +++ b/src/test/modules/test_copy_custom_format/test_copy_custom_format.c
>
> > @@ -0,0 +1,169 @@
> > ...
> > +TestCopyProcessOneOption(CopyFormatOptions *opts, bool is_from, DefElem *option)
> > +{
> > +     TestCopyOptions *t = (TestCopyOptions *) opts->format_private_opts;
> > +
> > +     if (t == NULL)
> > +     {
> > +             t = palloc0_object(TestCopyOptions);
> > +             opts->format_private_opts = (void *) t;
> > +     }
>
> This is not a blocker but we may want to add
> InitializeCopyOptions callback for this.

It would save just a few lines. It might be worth having the
initialization callback if it would enable extensions to do what
cannot be done with the current proposed callbacks.

Thank you for reviewing the patches! I've attached updated patches.
I'll verify that the new API works well with an experimental custom
copy format extension.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

  339. Re: Make COPY format extendable: Extract COPY TO format implementations

    Sutou Kouhei <kou@clear-code.com> — 2026-06-26T03:21:50Z

    Hi,

In <CAD21AoBLbmxT6xMHkDjfvBZEJUo1VbCd-vWhwX9WXbpMgaGW_A@mail.gmail.com>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Tue, 23 Jun 2026 18:15:10 -0700,
  Masahiko Sawada <sawada.mshk@gmail.com> wrote:

> Thank you for reviewing the patches! I've attached updated patches.

+1

I have only a few minor comments:

0002:

> --- /dev/null
> +++ b/src/backend/commands/copyapi.c

> +void
> +RegisterCopyCustomFormat(const char *name, const CopyToRoutine *to,
> +						 const CopyFromRoutine *from, ProcessOneCopyOptionFn option_fn)

How about using "const CopyCustomFormatEntry *" instead of
"to", "from" and "option_fn"? If we use
CopyCustomFormatEntry here, we don't need change the
signature of this function when we add more items.

> +const CopyCustomFormatEntry *
> +GetCopyCustomFormatRoutines(const char *name)

How about renaming this to "...FormatEntry" from
"...FormatRoutines"?

> --- a/src/include/commands/copy.h
> +++ b/src/include/commands/copy.h

> +#define CopyFormatIsBuiltins(format) ((format) != COPY_FORMAT_CUSTOM)

How about removing the last "s" ("...IsBuiltin") because
this processes only one format?

> +	const struct CopyCustomFormatEntry *custom_format_ent;

It may be better that we don't abbreviate "_entry" to "_ent"
here for readability. It seems that we use this abbreviation
only in a few places:

$ git grep '_ent;' src/
src/backend/replication/logical/reorderbuffer.c:		ReorderBufferTupleCidEnt *new_ent;
src/backend/utils/cache/catcache.c:	CatCInProgress in_progress_ent;
src/backend/utils/cache/catcache.c:	catcache_in_progress_stack = &in_progress_ent;
src/backend/utils/cache/catcache.c:			CatCInProgress in_progress_ent;
src/backend/utils/cache/catcache.c:			catcache_in_progress_stack = &in_progress_ent;


> I'll verify that the new API works well with an experimental custom
> copy format extension.

I think that we need to provide more APIs to read/write data
like we did in v40-0003 to implement a custom copy format
extension:
https://www.postgresql.org/message-id/flat/20250425.214534.1841428689427124725.kou%40clear-code.com

At least https://github.com/kou/pg-copy-arrow needs them.


Thanks,
-- 
kou