Add tuples_skipped to pg_stat_progress_copy

torikoshia <torikoshia@oss.nttdata.com>

From: torikoshia <torikoshia@oss.nttdata.com>
To: Pgsql Hackers <pgsql-hackers@postgresql.org>
Date: 2024-01-17T05:22:03Z
Lists: pgsql-hackers

Attachments

Hi,

132de9968840c introduced SAVE_ERROR_TO option to COPY and enabled to 
skip malformed data, but there is no way to watch the number of skipped 
rows during COPY.

Attached patch adds tuples_skipped to pg_stat_progress_copy, which 
counts the number of skipped tuples because source data is malformed.
If SAVE_ERROR_TO is not specified, this column remains zero.

The advantage would be that users can quickly notice and stop COPYing 
when there is a larger amount of skipped data than expected, for 
example.

As described in commit log, it is expected to add more choices for 
SAVE_ERROR_TO like 'log' and using such options may enable us to know 
the number of skipped tuples during COPY, but exposed in 
pg_stat_progress_copy would be easier to monitor.


What do you think?

-- 
Regards,

--
Atsushi Torikoshi
NTT DATA Group Corporation

Commits

  1. Add progress reporting of skipped tuples during COPY FROM.