Horton

Re: Make COPY format extendable: Extract COPY TO format implementations

Sutou Kouhei <kou@clear-code.com>

View thread View on postgr.es
From: Sutou Kouhei <kou@clear-code.com>
To: v.popolitov@postgrespro.ru
Cc: sawada.mshk@gmail.com, zhjwpku@gmail.com, michael@paquier.xyz, pgsql-hackers@postgresql.org
Date: 2025-02-05T07:31:56Z
Lists: pgsql-hackers 
Hi,

In <eb59c12bb36207c65f23719f255eb69b@postgrespro.ru>
  "Re: Make COPY format extendable: Extract COPY TO format implementations" on Tue, 04 Feb 2025 17:46:10 +0700,
  Vladlen Popolitov <v.popolitov@postgrespro.ru> wrote:

> I think, in case of USING PostgreSQL kernel will call corresponding
> handler,
> and it looks secure - the same as for table and index methods
> handlers.

We use similar approach that is used by table sampling
method. We can use a new table sampling method by just
adding a "method_name(internal) RETURNS tsm_handler"
function. Is it not secure too?

>  If you add extensibility, than every handler will be the
> extension, that can handle one or more formats.

Hmm. It may be a needless extensibility. Is it useful? I
feel that it increases complexity when we implement a custom
format handler. We can just implement one handler per custom
format. If we want to share implementation details in
multiple handlers, we can just share internal C
functions.

If we require one handler per custom format, it'll simpler
than one handler for multiple custom formats.

>>>           it assumes some SetOptions interface, that defines
>>> an options structure according to the new method requirements.
>> Sorry. I couldn't find the SetOptions interface in source
>> code. I found only AT_SetOptions. Did you mean it by "some
>> SetOptions interface"?
> Yes.
>> I'm familiar with only access method. It has
>> IndexAmRoutine::amoptions. Is it a SetOptions interface
>> example?
> Yes. I think, it would be compatible with other modules
> of source code and could use the same code base to process
> options of COPY TO/FROM

Thanks. I thought that there is a common interface pattern
for SetOptions. But it seems that it's a feature that is
implemented in many extension points.

If we implement custom options support eventually, does it
satisfy the "SetOptions interface"?


Thanks,
-- 
kou

Commits

  1. Refactor Copy{From|To}GetRoutine() to use pass-by-reference argument.

  2. Refactor COPY FROM to use format callback functions.

  3. Refactor COPY TO to use format callback functions.

  4. Another try to fix BF failure introduced in commit ddd5f4f54a.

  5. Revert "Refactor CopyReadAttributes{CSV,Text}() to use a callback in COPY FROM"

  6. Improve COPY TO performance when server and client encodings match

  7. Simplify signature of CopyAttributeOutCSV() in copyto.c

  8. Revert "Refactor CopyAttributeOut{CSV,Text}() to use a callback in COPY TO"

  9. Refactor CopyAttributeOut{CSV,Text}() to use a callback in COPY TO

  10. Refactor CopyReadAttributes{CSV,Text}() to use a callback in COPY FROM

  11. Add progress reporting of skipped tuples during COPY FROM.

  12. pgbench: Add \syncpipeline

  13. meson: Make gzip and tar optional

  14. Export the external file reader used in COPY FROM as APIs.