Horton

Re: Make COPY format extendable: Extract COPY TO format implementations

Masahiko Sawada <sawada.mshk@gmail.com>

View thread View on postgr.es
From: Masahiko Sawada <sawada.mshk@gmail.com>
To: Sutou Kouhei <kou@clear-code.com>
Cc: michael@paquier.xyz, zhjwpku@gmail.com, pgsql-hackers@postgresql.org
Date: 2025-02-28T22:00:18Z
Lists: pgsql-hackers 
On Fri, Feb 28, 2025 at 1:58 PM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> In <CAD21AoDr13=dx+k8gmQnR5_bY+NskyN4mbSWN0KhQncL6xuPMA@mail.gmail.com>
>   "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 28 Feb 2025 11:50:39 -0800,
>   Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> > I initially thought it would be acceptable to stop
> > NextCopyFromRawFields exposed since NextCopyFrom() could serve as an
> > alternative. For example, the NextCopyFromRawFields() function was
> > originally exposed in commit 8ddc05fb01ee2c primarily to support
> > extension modules like file_fdw but file_fdw wasn't utilizing this
> > API. I pushed the patch without the above change. Unfortunately, this
> > commit subsequently broke file_text_array_fdw[1] and made BF animal
> > crake unhappy[2].
> >
> > [1] https://github.com/adunstan/file_text_array_fdw
> > [2] https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=crake&dt=2025-02-28%2018%3A47%3A02
>
> Thanks for the try!
>
> > Upon examining file_text_array_fdw more closely, I realized that
> > NextCopyFrom() may not be a suitable replacement for
> > NextCopyFromRawFields() in certain scenarios. Specifically,
> > NextCopyFrom() assumes that the caller has prior knowledge of the
> > source data's column count, making it inadequate for cases where
> > extensions like file_text_array_fdw need to construct an array of
> > source data with an unknown number of columns. In such situations,
> > NextCopyFromRawFields() proves to be more practical. Given these
> > considerations, I'm now leaning towards implementing the proposed
> > change. Thoughts?
>
> You suggest that we re-export NextCopyFromRawFields() (as a
> wrapper of static inline version) for backward
> compatibility, right? It makes sense. We should keep
> backward compatibility because there is a use-case of
> NextCopyFromRawFields().

Yes, I've submitted the patch to re-export that function[1]. Could you
review it?

Regards,

[1] https://www.postgresql.org/message-id/CAD21AoBA414Q76LthY65NJfWbjOxXn1bdFFsD_NBhT2wPUS1SQ%40mail.gmail.com

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

Commits

  1. Refactor Copy{From|To}GetRoutine() to use pass-by-reference argument.

  2. Refactor COPY FROM to use format callback functions.

  3. Refactor COPY TO to use format callback functions.

  4. Another try to fix BF failure introduced in commit ddd5f4f54a.

  5. Revert "Refactor CopyReadAttributes{CSV,Text}() to use a callback in COPY FROM"

  6. Improve COPY TO performance when server and client encodings match

  7. Simplify signature of CopyAttributeOutCSV() in copyto.c

  8. Revert "Refactor CopyAttributeOut{CSV,Text}() to use a callback in COPY TO"

  9. Refactor CopyAttributeOut{CSV,Text}() to use a callback in COPY TO

  10. Refactor CopyReadAttributes{CSV,Text}() to use a callback in COPY FROM

  11. Add progress reporting of skipped tuples during COPY FROM.

  12. pgbench: Add \syncpipeline

  13. meson: Make gzip and tar optional

  14. Export the external file reader used in COPY FROM as APIs.