Horton

Re: Make COPY format extendable: Extract COPY TO format implementations

Masahiko Sawada <sawada.mshk@gmail.com>

View thread View on postgr.es
From: Masahiko Sawada <sawada.mshk@gmail.com>
To: "David G. Johnston" <david.g.johnston@gmail.com>
Cc: Sutou Kouhei <kou@clear-code.com>, "tgl@sss.pgh.pa.us" <tgl@sss.pgh.pa.us>, "zhjwpku@gmail.com" <zhjwpku@gmail.com>, "michael@paquier.xyz" <michael@paquier.xyz>, "pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Date: 2025-05-04T05:02:51Z
Lists: pgsql-hackers 
On Sat, May 3, 2025 at 7:42 AM David G. Johnston
<david.g.johnston@gmail.com> wrote:
>
> On Saturday, May 3, 2025, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
>>
>> I think that we need to ensure that if users specify text/csv/binary
>> the built-in formats are always used, to keep backward compatibility.
>
>
> That was my original thinking, but it’s inconsistent with how functions behave today.  We don’t promise that installing extensions won’t cause existing code to change.

I'm skeptical about whether that's an acceptable backward
compatibility breakage.

>> > I’m all for registration to enable additional options and features - but am against moving away from turning format into a namespaced identifier.  This is a query-facing feature where namespaces are common and fundamentally required.
>>
>> That's a fair concern. But isn't the format name ultimately just an
>> option value, but not like a database object?
>
>
> We get to decide that.  And deciding in favor of “extensible database object in a namespace’ makes more sense - leveraging all that pre-existing design to play more nicely with extensions and give DBAs control.  The SQL command to add one is “create function” instead of “create copy format”.

I still don't fully understand why the FORMAT value alone needs to be
treated like a schema-qualified object. If the concern is about name
conflict with future built-in formats, I would argue that the same
concern applies to custom EXPLAIN options and logical decoding
plugins. To me, the benefit of treating the COPY FORMAT value as a
schema-qualified object seems limited. Meanwhile, the risk of not
protecting built-in formats like 'text', 'csv', and 'binary' is
significant. If those names can be shadowed by extension via
search_patch, we lose backward compatibility.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

Commits

  1. Refactor Copy{From|To}GetRoutine() to use pass-by-reference argument.

  2. Refactor COPY FROM to use format callback functions.

  3. Refactor COPY TO to use format callback functions.

  4. Another try to fix BF failure introduced in commit ddd5f4f54a.

  5. Revert "Refactor CopyReadAttributes{CSV,Text}() to use a callback in COPY FROM"

  6. Improve COPY TO performance when server and client encodings match

  7. Simplify signature of CopyAttributeOutCSV() in copyto.c

  8. Revert "Refactor CopyAttributeOut{CSV,Text}() to use a callback in COPY TO"

  9. Refactor CopyAttributeOut{CSV,Text}() to use a callback in COPY TO

  10. Refactor CopyReadAttributes{CSV,Text}() to use a callback in COPY FROM

  11. Add progress reporting of skipped tuples during COPY FROM.

  12. pgbench: Add \syncpipeline

  13. meson: Make gzip and tar optional

  14. Export the external file reader used in COPY FROM as APIs.