Horton

Re: Parallel copy

Heikki Linnakangas <hlinnaka@iki.fi>

View thread
From: Heikki Linnakangas <hlinnaka@iki.fi>
To: vignesh C <vignesh21@gmail.com>, Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2020-10-30T16:52:37Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Allow WaitLatch() to be used without a latch.

  2. Add %P to log_line_prefix for parallel group leader

  3. Include replication origins in SQL functions for commit timestamp

  4. Avoid useless buffer allocations during binary COPY FROM.

Attachments

On 30/10/2020 18:36, Heikki Linnakangas wrote:
> Whether the leader process finds the EOLs or the worker processes, it's
> pretty clear that it needs to be done ASAP, for a chunk at a time,
> because that cannot be done in parallel. I think some refactoring in
> CopyReadLine() and friends would be in order. It probably would be
> faster, or at least not slower, to find all the EOLs in a block in one
> tight loop, even when parallel copy is not used.

Something like the attached. It passes the regression tests, but it's 
quite incomplete. It's missing handing of "\." as end-of-file marker, 
and I haven't tested encoding conversions at all, for starters. Quick 
testing suggests that this a little bit faster than the current code, 
but the difference is small; I had to use a "WHERE false" option to 
really see the difference.

The crucial thing here is that there's a new function, ParseLinesText(), 
to find all end-of-line characters in a buffer in one go. In this patch, 
it's used against 'raw_buf', but with parallel copy, you could point it 
at a block in shared memory instead.

- Heikki