Re: Parallel copy

Amit Kapila <amit.kapila16@gmail.com>

From: Amit Kapila <amit.kapila16@gmail.com>

To: vignesh C <vignesh21@gmail.com>

Cc: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>

Date: 2020-10-29T09:24:41Z

Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

Allow WaitLatch() to be used without a latch.
- 733fa9aa51c5 14.0 cited
Add %P to log_line_prefix for parallel group leader
- b8fdee7d0ca8 14.0 cited
Include replication origins in SQL functions for commit timestamp
- b1e48bbe64a4 14.0 cited
Avoid useless buffer allocations during binary COPY FROM.
- cd22d3cdb9bd 14.0 cited

On Thu, Oct 29, 2020 at 11:45 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Tue, Oct 27, 2020 at 7:06 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> [latest version]
>
> I think the parallel-safety checks in this patch
> (v9-0002-Allow-copy-from-command-to-process-data-from-file) are
> incomplete and wrong.
>

One more point, I have noticed that some time back [1], I have given
one suggestion related to the way workers process the set of lines
(aka chunk). I think you can try by increasing the chunk size to say
100, 500, 1000 and use some shared counter to remember the number of
chunks processed.

[1] - https://www.postgresql.org/message-id/CAA4eK1L-Xgw1zZEbGePmhBBWmEmLFL6rCaiOMDPnq2GNMVz-sg%40mail.gmail.com

-- 
With Regards,
Amit Kapila.