Horton

Re: Parallel copy

Ants Aasma <ants@cybertec.at>

View thread
From: Ants Aasma <ants@cybertec.at>
To: Thomas Munro <thomas.munro@gmail.com>
Cc: Amit Kapila <amit.kapila16@gmail.com>, Alastair Turner <minion@decodable.me>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2020-02-18T07:13:45Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Allow WaitLatch() to be used without a latch.

  2. Add %P to log_line_prefix for parallel group leader

  3. Include replication origins in SQL functions for commit timestamp

  4. Avoid useless buffer allocations during binary COPY FROM.

On Tue, 18 Feb 2020 at 04:40, Thomas Munro <thomas.munro@gmail.com> wrote:
> +1.  That sort of two-queue scheme is exactly how I sketched out a
> multi-consumer queue for a hypothetical Parallel Scatter node.  It
> probably gets a bit trickier when the payload has to be broken up into
> fragments to wrap around the "data" buffer N times.

At least for copy it should be easy enough - it already has to handle reading
data block by block. If worker updates its position while doing so the reader
can wrap around the data buffer.

There will be no parallelism while one worker is buffering up a line larger
than the data buffer, but that doesn't seem like a major issue. Once the line is
buffered and begins inserting next worker can start buffering the next tuple.

Regards,
Ants Aasma