Horton

Re: Make COPY format extendable: Extract COPY TO format implementations

Masahiko Sawada <sawada.mshk@gmail.com>

View thread View on postgr.es
From: Masahiko Sawada <sawada.mshk@gmail.com>
To: "David G. Johnston" <david.g.johnston@gmail.com>
Cc: Sutou Kouhei <kou@clear-code.com>, "tgl@sss.pgh.pa.us" <tgl@sss.pgh.pa.us>, "zhjwpku@gmail.com" <zhjwpku@gmail.com>, "michael@paquier.xyz" <michael@paquier.xyz>, "pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Date: 2025-05-03T07:54:36Z
Lists: pgsql-hackers 
On Fri, May 2, 2025 at 11:37 PM David G. Johnston
<david.g.johnston@gmail.com> wrote:
>
> On Friday, May 2, 2025, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>>
>>
>> I'm concerned about allowing multiple 'text' format implementations
>> with identical names within the database, as this could lead to
>> considerable confusion. When users specify 'text', it would be more
>> logical to guarantee that the built-in 'text' format is consistently
>> used.
>
>
> Do you want to only give text/csv/binary this special treatment or also any future format name we ever decide to implement in core.  If an extension takes up “xml” and we try to do that in core do we fail an upgrade because of the conflict, and make it impossible to actually use said extension?

I guess that's an extension author's responsibility to upgrade its
extension so as to work with the new PostgreSQL version, or carefully
choose the format name. They can even name
'[extension_name].[format_name]' as a format name. Even with the
current patch design (i.e., search_path affects handler function
lookups), users would end up using the built-in 'xml' format without
notice after upgrade, no? I guess that could introduce another
problem.

I think that we need to ensure that if users specify text/csv/binary
the built-in formats are always used, to keep backward compatibility.

>
>> This principle aligns with other customizable components, such
>> as custom resource managers, wait events, lightweight locks, and
>> custom scans. These components maintain their built-in data/types and
>> explicitly prevent the registration of duplicate names.
>
>
> I am totally lost on how any of those resemble this feature.
>
> I’m all for registration to enable additional options and features - but am against moving away from turning format into a namespaced identifier.  This is a query-facing feature where namespaces are common and fundamentally required.

That's a fair concern. But isn't the format name ultimately just an
option value, but not like a database object? As I mentioned above, I
think we need to keep backward compatibility but treating the built-in
formats special seems inconsistent with common name resolution
behavior.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

Commits

  1. Refactor Copy{From|To}GetRoutine() to use pass-by-reference argument.

  2. Refactor COPY FROM to use format callback functions.

  3. Refactor COPY TO to use format callback functions.

  4. Another try to fix BF failure introduced in commit ddd5f4f54a.

  5. Revert "Refactor CopyReadAttributes{CSV,Text}() to use a callback in COPY FROM"

  6. Improve COPY TO performance when server and client encodings match

  7. Simplify signature of CopyAttributeOutCSV() in copyto.c

  8. Revert "Refactor CopyAttributeOut{CSV,Text}() to use a callback in COPY TO"

  9. Refactor CopyAttributeOut{CSV,Text}() to use a callback in COPY TO

  10. Refactor CopyReadAttributes{CSV,Text}() to use a callback in COPY FROM

  11. Add progress reporting of skipped tuples during COPY FROM.

  12. pgbench: Add \syncpipeline

  13. meson: Make gzip and tar optional

  14. Export the external file reader used in COPY FROM as APIs.