Horton

Re: Make COPY format extendable: Extract COPY TO format implementations

Michael Paquier <michael@paquier.xyz>

View thread View on postgr.es
From: Michael Paquier <michael@paquier.xyz>
To: Junwang Zhao <zhjwpku@gmail.com>
Cc: Masahiko Sawada <sawada.mshk@gmail.com>, Andrew Dunstan <andrew@dunslane.net>, Sutou Kouhei <kou@clear-code.com>, nathandbossart@gmail.com, pgsql-hackers@postgresql.org
Date: 2023-12-08T05:17:42Z
Lists: pgsql-hackers 
On Fri, Dec 08, 2023 at 10:32:27AM +0800, Junwang Zhao wrote:
> I can see FDW related utility commands but no TABLESAMPLE related,
> and there is a pg_foreign_data_wrapper system catalog which has
> a *fdwhandler* field.

+ */ +CATALOG(pg_copy_handler,4551,CopyHandlerRelationId)

Using a catalog is an over-engineered design.  Others have provided
hints about that upthread, but it would be enough to have one or two
handler types that are wrapped around one or two SQL *functions*, like
tablesamples.  It seems like you've missed it, but feel free to read
about tablesample-method.sgml, that explains how this is achieved for
tablesamples.

> If we want extensions to create a new copy handler, I think
> something like pg_copy_hander should be necessary.

A catalog is not necessary, that's the point, because it can be
replaced by a scan of pg_proc with the function name defined in a COPY
query (be it through a FORMAT, or different option in a DefElem).
An example of extension with tablesamples is contrib/tsm_system_rows/,
that just uses a function returning a tsm_handler: 
CREATE FUNCTION system_rows(internal)
RETURNS tsm_handler
AS 'MODULE_PATHNAME', 'tsm_system_rows_handler'
LANGUAGE C STRICT;

Then SELECT queries rely on the contents of the TABLESAMPLE clause to
find the set of callbacks it should use by calling the function.

+/* Routines for a COPY HANDLER implementation. */
+typedef struct CopyRoutine
+{

FWIW, I find weird the concept of having one handler for both COPY
FROM and COPY TO as each one of them has callbacks that are mutually
exclusive to the other, but I'm OK if there is a consensus of only
one.  So I'd suggest to use *two* NodeTags instead for a cleaner
split, meaning that we'd need two functions for each method.  My point
is that a custom COPY handler could just define a COPY TO handler or a
COPY FROM handler, though it mostly comes down to a matter of taste
regarding how clean the error handling becomes if one tries to use a
set of callbacks with a COPY type (TO or FROM) not matching it.
--
Michael

Commits

  1. Refactor Copy{From|To}GetRoutine() to use pass-by-reference argument.

  2. Refactor COPY FROM to use format callback functions.

  3. Refactor COPY TO to use format callback functions.

  4. Another try to fix BF failure introduced in commit ddd5f4f54a.

  5. Revert "Refactor CopyReadAttributes{CSV,Text}() to use a callback in COPY FROM"

  6. Improve COPY TO performance when server and client encodings match

  7. Simplify signature of CopyAttributeOutCSV() in copyto.c

  8. Revert "Refactor CopyAttributeOut{CSV,Text}() to use a callback in COPY TO"

  9. Refactor CopyAttributeOut{CSV,Text}() to use a callback in COPY TO

  10. Refactor CopyReadAttributes{CSV,Text}() to use a callback in COPY FROM

  11. Add progress reporting of skipped tuples during COPY FROM.

  12. pgbench: Add \syncpipeline

  13. meson: Make gzip and tar optional

  14. Export the external file reader used in COPY FROM as APIs.