Horton

Re: Make COPY format extendable: Extract COPY TO format implementations

David G. Johnston <david.g.johnston@gmail.com>

View thread View on postgr.es
From: "David G. Johnston" <david.g.johnston@gmail.com>
To: Masahiko Sawada <sawada.mshk@gmail.com>
Cc: Sutou Kouhei <kou@clear-code.com>, "tgl@sss.pgh.pa.us" <tgl@sss.pgh.pa.us>, "zhjwpku@gmail.com" <zhjwpku@gmail.com>, "michael@paquier.xyz" <michael@paquier.xyz>, "pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Date: 2025-05-04T05:27:36Z
Lists: pgsql-hackers 
On Saturday, May 3, 2025, Masahiko Sawada <sawada.mshk@gmail.com> wrote:

> On Sat, May 3, 2025 at 7:42 AM David G. Johnston
> <david.g.johnston@gmail.com> wrote:
> >
> > On Saturday, May 3, 2025, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> >>
> >> I think that we need to ensure that if users specify text/csv/binary
> >> the built-in formats are always used, to keep backward compatibility.
> >
> >
> > That was my original thinking, but it’s inconsistent with how functions
> behave today.  We don’t promise that installing extensions won’t cause
> existing code to change.
>
> I'm skeptical about whether that's an acceptable backward
> compatibility breakage.


I’m skeptical you are correctly defining what backward-compatibility
requires.

Well, the only potential breakage is that we are searching for a matching
function by signature without first limiting the mandated return type.  But
that is solve-able should anyone else see the problem as well.

The global format name has its merits but neither it nor the namespaced
format option suffer from breaking compatibility or policy.


>
> I still don't fully understand why the FORMAT value alone needs to be
> treated like a schema-qualified object. If the concern is about name
> conflict with future built-in formats, I would argue that the same
> concern applies to custom EXPLAIN options and logical decoding
> plugins.


>
Then maybe we have the same “problem” in those places.


>
> To me, the benefit of treating the COPY FORMAT value as a
> schema-qualified object seems limited. Meanwhile, the risk of not
> protecting built-in formats like 'text', 'csv', and 'binary' is
> significant.


Really? You think lots of extensions are going to choose to use these
values even if they are permitted?  Or are you concerned about attack
surfaces?


> If those names can be shadowed by extension via
> search_patch, we lose backward compatibility.
>

This is not a definition of backward-compatibility that I am familiar with.

If anything the ability for a DBA to arrange for such shadowing would be a
feature enhancement.  They can drop-in a more efficient or desirable
implementation without having to change query code.

In any case, I’m doubtful either of us can make a convincing enough
argument to sway the other fully.  Both options are plausible, IMO.  Others
need to chime in.

David J.

Commits

  1. Refactor Copy{From|To}GetRoutine() to use pass-by-reference argument.

  2. Refactor COPY FROM to use format callback functions.

  3. Refactor COPY TO to use format callback functions.

  4. Another try to fix BF failure introduced in commit ddd5f4f54a.

  5. Revert "Refactor CopyReadAttributes{CSV,Text}() to use a callback in COPY FROM"

  6. Improve COPY TO performance when server and client encodings match

  7. Simplify signature of CopyAttributeOutCSV() in copyto.c

  8. Revert "Refactor CopyAttributeOut{CSV,Text}() to use a callback in COPY TO"

  9. Refactor CopyAttributeOut{CSV,Text}() to use a callback in COPY TO

  10. Refactor CopyReadAttributes{CSV,Text}() to use a callback in COPY FROM

  11. Add progress reporting of skipped tuples during COPY FROM.

  12. pgbench: Add \syncpipeline

  13. meson: Make gzip and tar optional

  14. Export the external file reader used in COPY FROM as APIs.