Re: Make COPY format extendable: Extract COPY TO format implementations
Sutou Kouhei <kou@clear-code.com>
Hi,
In <CAD21AoD9CBjh4u6jdiE0tG-jvejw-GJN8fUPoQSVhKh36HW2NQ@mail.gmail.com>
"Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 2 May 2025 23:37:46 -0700,
Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> The progress information is stored in PgBackendStatus defined in
> backend_status.h:
>
> /*
> * Command progress reporting. Any command which wishes can advertise
> * that it is running by setting st_progress_command,
> * st_progress_command_target, and st_progress_param[].
> * st_progress_command_target should be the OID of the relation which the
> * command targets (we assume there's just one, as this is meant for
> * utility commands), but the meaning of each element in the
> * st_progress_param array is command-specific.
> */
> ProgressCommandType st_progress_command;
> Oid st_progress_command_target;
> int64 st_progress_param[PGSTAT_NUM_PROGRESS_PARAM];
>
> Then the progress view maps the numbers to the corresponding strings:
>
> CREATE VIEW pg_stat_progress_copy AS
> SELECT
> S.pid AS pid, S.datid AS datid, D.datname AS datname,
> S.relid AS relid,
> CASE S.param5 WHEN 1 THEN 'COPY FROM'
> WHEN 2 THEN 'COPY TO'
> END AS command,
> CASE S.param6 WHEN 1 THEN 'FILE'
> WHEN 2 THEN 'PROGRAM'
> WHEN 3 THEN 'PIPE'
> WHEN 4 THEN 'CALLBACK'
> END AS "type",
> S.param1 AS bytes_processed,
> S.param2 AS bytes_total,
> S.param3 AS tuples_processed,
> S.param4 AS tuples_excluded,
> S.param7 AS tuples_skipped
> FROM pg_stat_get_progress_info('COPY') AS S
> LEFT JOIN pg_database D ON S.datid = D.oid;
Thanks. I didn't know about how to implement
pg_stat_progress_copy.
> So the idea is that the backend process sets the format ID somewhere
> in st_progress_param, and then the progress view calls a SQL function,
> say pg_stat_get_copy_format_name(), with the format ID that returns
> the corresponding format name.
Does it work when we use session_preload_libraries or the
LOAD command? If we have 2 sessions and both of them load
"jsonlines" COPY FORMAT extensions, what will be happened?
For example:
1. Session 1: Register "jsonlines"
2. Session 2: Register "jsonlines"
(Should global format ID <-> format name mapping
be updated?)
3. Session 2: Close this session.
Unregister "jsonlines".
(Can we unregister COPY FORMAT extension?)
(Should global format ID <-> format name mapping
be updated?)
4. Session 1: Close this session.
Unregister "jsonlines".
(Can we unregister COPY FORMAT extension?)
(Should global format ID <-> format name mapping
be updated?)
Thanks,
--
kou
Commits
-
Refactor Copy{From|To}GetRoutine() to use pass-by-reference argument.
- bacbc4863b3b 18.0 landed
-
Refactor COPY FROM to use format callback functions.
- 7717f6300693 18.0 landed
-
Refactor COPY TO to use format callback functions.
- 2e4127b6d2d8 18.0 landed
-
Another try to fix BF failure introduced in commit ddd5f4f54a.
- 9bc1eee988c3 17.0 cited
-
Revert "Refactor CopyReadAttributes{CSV,Text}() to use a callback in COPY FROM"
- 06bd311bce24 17.0 landed
-
Improve COPY TO performance when server and client encodings match
- b619852086ed 17.0 cited
-
Simplify signature of CopyAttributeOutCSV() in copyto.c
- b9d6038d7048 17.0 landed
-
Revert "Refactor CopyAttributeOut{CSV,Text}() to use a callback in COPY TO"
- 1aa8324b81fa 17.0 landed
-
Refactor CopyAttributeOut{CSV,Text}() to use a callback in COPY TO
- 2889fd23be56 17.0 landed
-
Refactor CopyReadAttributes{CSV,Text}() to use a callback in COPY FROM
- 95fb5b49024a 17.0 landed
-
Add progress reporting of skipped tuples during COPY FROM.
- 729439607ad2 17.0 cited
-
pgbench: Add \syncpipeline
- 94edfe250c6a 17.0 cited
-
meson: Make gzip and tar optional
- 9ca6e7b9411e 17.0 cited
-
Export the external file reader used in COPY FROM as APIs.
- 8ddc05fb01ee 9.1.0 cited