Horton

Re: Make COPY format extendable: Extract COPY TO format implementations

Masahiko Sawada <sawada.mshk@gmail.com>

View thread View on postgr.es
From: Masahiko Sawada <sawada.mshk@gmail.com>
To: Sutou Kouhei <kou@clear-code.com>
Cc: david.g.johnston@gmail.com, tgl@sss.pgh.pa.us, zhjwpku@gmail.com, michael@paquier.xyz, pgsql-hackers@postgresql.org
Date: 2025-05-03T06:02:25Z
Lists: pgsql-hackers 
On Fri, May 2, 2025 at 9:56 PM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <CAD21AoBGRFStdVbHUcxL0QB8wn92J3Sn-6x=RhsSMuhepRH0NQ@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 2 May 2025 21:38:32 -0700,
>   Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> >> How about requiring schema for all custom formats?
> >>
> >> Valid:
> >>
> >>   COPY ... TO ... (FORMAT 'text');
> >>   COPY ... TO ... (FORMAT 'my_schema.jsonlines');
> >>
> >> Invalid:
> >>
> >>   COPY ... TO ... (FORMAT 'jsonlines'); -- no schema
> >>   COPY ... TO ... (FORMAT 'pg_catalog.text'); -- needless schema
> >>
> >> If we require "schema" for all custom formats, we don't need
> >> to depend on search_path.
> >
> > I'm concerned that users cannot use the same format name in the FORMAT
> > option depending on which schema the handler function is created.
>
> I'm not sure that it's a problem or not. If users want to
> use the same format name, they can install the handler
> function to the same schema.
>
> >> Why do we need to assign a unique ID? For performance? For
> >> RegisterCustomCopyFormatOption()?
> >
> > I think it's required for monitoring purposes for example. For
> > instance, we can set the format ID in the progress information and the
> > progress view can fetch the format name by the ID so that users can
> > see what format is being used in the COPY command.
>
> How about setting the format name instead of the format ID
> in the progress information?

The progress view can know only numbers. We need to extend the
progress view infrastructure so that we can pass other data types.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

Commits

  1. Refactor Copy{From|To}GetRoutine() to use pass-by-reference argument.

  2. Refactor COPY FROM to use format callback functions.

  3. Refactor COPY TO to use format callback functions.

  4. Another try to fix BF failure introduced in commit ddd5f4f54a.

  5. Revert "Refactor CopyReadAttributes{CSV,Text}() to use a callback in COPY FROM"

  6. Improve COPY TO performance when server and client encodings match

  7. Simplify signature of CopyAttributeOutCSV() in copyto.c

  8. Revert "Refactor CopyAttributeOut{CSV,Text}() to use a callback in COPY TO"

  9. Refactor CopyAttributeOut{CSV,Text}() to use a callback in COPY TO

  10. Refactor CopyReadAttributes{CSV,Text}() to use a callback in COPY FROM

  11. Add progress reporting of skipped tuples during COPY FROM.

  12. pgbench: Add \syncpipeline

  13. meson: Make gzip and tar optional

  14. Export the external file reader used in COPY FROM as APIs.