Horton

Re: Make COPY format extendable: Extract COPY TO format implementations

Sutou Kouhei <kou@clear-code.com>

View thread View on postgr.es
From: Sutou Kouhei <kou@clear-code.com>
To: michael@paquier.xyz
Cc: sawada.mshk@gmail.com, david.g.johnston@gmail.com, tgl@sss.pgh.pa.us, zhjwpku@gmail.com, pgsql-hackers@postgresql.org
Date: 2025-06-18T03:59:20Z
Lists: pgsql-hackers

Attachments

Hi,

In <aFC5HmZHU5NCPuTL@paquier.xyz>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Tue, 17 Jun 2025 09:38:54 +0900,
  Michael Paquier <michael@paquier.xyz> wrote:

> On Tue, Jun 17, 2025 at 08:50:37AM +0900, Sutou Kouhei wrote:
>> OK. I'll implement the initial version with this
>> design. (Allocating IDs local not shared.)
> 
> Sounds good to me.  Thanks Sutou-san!

I've attached the v41 patch set that uses the C API approach
with local (not shared) COPY routine management.

0001: This is same as 0001 in the v40 patch set. It just
      cleans up CopySource and CopyDest enums.
0002: This is the initial version of this approach.

Here are some discussion points:

1. This provides 2 registration APIs
   (RegisterCopy{From,To}Routine(name, routine)) instead of
   1 registration API (RegisterCopyFormat(name,
   from_routine, to_routine)).

   It's for simple implementation and easy to extend without
   breaking APIs in the future. (And some formats may
   provide only FROM routine or TO routine.)

   Is this design acceptable?

   FYI: RegisterCopy{From,To}Routine() uses the same logic
   as RegisterExtensionExplainOption().

2. This allocates IDs internally but doesn't provide APIs
   that get them. Because it's not needed for now.

   We can provide GetExplainExtensionId() like API when we
   need it.
        
   Is this design acceptable?

3. I want to register the built-in COPY {FROM,TO} routines
   in the PostgreSQL initialization phase. Where should we
   do it? In 0002, it's done in InitPostgres() but I'm not
   sure whether it's a suitable location or not.

4. 0002 adds CopyFormatOptions::routine as union:

   @@ -87,9 +91,14 @@ typedef struct CopyFormatOptions
           CopyLogVerbosityChoice log_verbosity;   /* verbosity of logged messages */
           int64           reject_limit;   /* maximum tolerable number of errors */
           List       *convert_select; /* list of column names (can be NIL) */
   +       union
   +       {
   +               const struct CopyFromRoutine *from; /* for COPY FROM */
   +               const struct CopyToRoutine *to; /* for COPY TO */
   +       }                       routine;                /* routine to process the specified format */
    } CopyFormatOptions;

   Because one of Copy{From,To}Routine is only needed at
   once. Is this union usage strange in PostgreSQL?

5. 0002 adds InitializeCopy{From,To}Routines() and
   GetCopy{From,To}Routine() that aren't used by COPY
   {FROM,TO} routine implementations to copyapi.h. Should we
   move them to other .h? If so, which .h should be used for
   them?


Thanks,
-- 
kou

Commits

  1. Refactor Copy{From|To}GetRoutine() to use pass-by-reference argument.

  2. Refactor COPY FROM to use format callback functions.

  3. Refactor COPY TO to use format callback functions.

  4. Another try to fix BF failure introduced in commit ddd5f4f54a.

  5. Revert "Refactor CopyReadAttributes{CSV,Text}() to use a callback in COPY FROM"

  6. Improve COPY TO performance when server and client encodings match

  7. Simplify signature of CopyAttributeOutCSV() in copyto.c

  8. Revert "Refactor CopyAttributeOut{CSV,Text}() to use a callback in COPY TO"

  9. Refactor CopyAttributeOut{CSV,Text}() to use a callback in COPY TO

  10. Refactor CopyReadAttributes{CSV,Text}() to use a callback in COPY FROM

  11. Add progress reporting of skipped tuples during COPY FROM.

  12. pgbench: Add \syncpipeline

  13. meson: Make gzip and tar optional

  14. Export the external file reader used in COPY FROM as APIs.