Horton

Re: Make COPY format extendable: Extract COPY TO format implementations

Masahiko Sawada <sawada.mshk@gmail.com>

View thread View on postgr.es
From: Masahiko Sawada <sawada.mshk@gmail.com>
To: Sutou Kouhei <kou@clear-code.com>
Cc: zhjwpku@gmail.com, michael@paquier.xyz, pgsql-hackers@postgresql.org
Date: 2025-01-28T23:00:03Z
Lists: pgsql-hackers 
On Thu, Jan 23, 2025 at 1:12 AM Sutou Kouhei <kou@clear-code.com> wrote:
>
> Hi,
>
> I noticed that the last patch set (v27) can't be applied to
> the current master. I've rebased on the current master and
> created v28 patch set. No code change.

Thank you for updating the patch!

While 0001 and 0002 look good to me overall, we still need to polish
subsequent patches. Here are review comments:

---
I still find that it would not be a good idea to move all copy-related
struct definitions to copyapi.h because we need to include copyapi.h
file into a .c file even if the file is not related to the custom copy
format routines. I think that copyapi.h should have only the
definitions of CopyToRoutine and CopyFromRoutine as well as some
functions related to the custom copy format. Here is an idea:

- CopyToState and CopyFromState are defined in copyto_internal.h (new
file) and copyfrom_internal.h, respectively.
- These two files #include's copy.h and other necessary header files.
- copyapi.h has only CopyToRoutine and CopyFromRoutine and #include's
both copyfrom_internal.h and copyto_internal.h.
- copyto.c, copyfrom.c and copyfromparse.c #include copyapi.h

Some advantages of this idea:

- we can keep both CopyToState and CopyFromState private in _internal.h files.
- custom format extension can include copyapi.h to provide a custom
copy format routine and to access the copy state data.
- copy-related .c files won't need to include copyapi.h if they don't
use custom copy format routines.

---
The 0008 patch introduces CopyFromStateRead(). While it would be a
good start, I think we can consider sorting out low-level
communication functions more. For example, CopyReadBinaryData() uses
the internal 64kB buffer but some custom copy format extensions might
want to use a larger buffer in its own implementation, which would
require exposing CopyGetData() etc. Given that we might expose more
functions to provide more ways for extensions, we might want to rename
CopyFromStateRead().

---
While we get the format routines for custom formats in
ProcessCopyOptionFormat(), we do that for built-in formats in
BeginCopyTo(), which seems odd to me. I think we can have
CopyToGetRoutine() responsible for getting CopyToRoutine for built-in
formats as well as custom format. The same is true for
CopyFromRoutine.

---
Copy[To|From]Routine for built-in formats are missing to set the node type.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

Commits

  1. Refactor Copy{From|To}GetRoutine() to use pass-by-reference argument.

  2. Refactor COPY FROM to use format callback functions.

  3. Refactor COPY TO to use format callback functions.

  4. Another try to fix BF failure introduced in commit ddd5f4f54a.

  5. Revert "Refactor CopyReadAttributes{CSV,Text}() to use a callback in COPY FROM"

  6. Improve COPY TO performance when server and client encodings match

  7. Simplify signature of CopyAttributeOutCSV() in copyto.c

  8. Revert "Refactor CopyAttributeOut{CSV,Text}() to use a callback in COPY TO"

  9. Refactor CopyAttributeOut{CSV,Text}() to use a callback in COPY TO

  10. Refactor CopyReadAttributes{CSV,Text}() to use a callback in COPY FROM

  11. Add progress reporting of skipped tuples during COPY FROM.

  12. pgbench: Add \syncpipeline

  13. meson: Make gzip and tar optional

  14. Export the external file reader used in COPY FROM as APIs.