Horton

Re: Parallel copy

Amit Kapila <amit.kapila16@gmail.com>

View thread
From: Amit Kapila <amit.kapila16@gmail.com>
To: Ants Aasma <ants@cybertec.at>
Cc: Alastair Turner <minion@decodable.me>, Thomas Munro <thomas.munro@gmail.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2020-02-18T13:21:29Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Allow WaitLatch() to be used without a latch.

  2. Add %P to log_line_prefix for parallel group leader

  3. Include replication origins in SQL functions for commit timestamp

  4. Avoid useless buffer allocations during binary COPY FROM.

On Tue, Feb 18, 2020 at 5:59 PM Ants Aasma <ants@cybertec.at> wrote:
>
> On Tue, 18 Feb 2020 at 12:20, Amit Kapila <amit.kapila16@gmail.com> wrote:
> > This is something similar to what I had also in mind for this idea.  I
> > had thought of handing over complete chunk (64K or whatever we
> > decide).  The one thing that slightly bothers me is that we will add
> > some additional overhead of copying to and from shared memory which
> > was earlier from local process memory.  And, the tokenization (finding
> > line boundaries) would be serial.  I think that tokenization should be
> > a small part of the overall work we do during the copy operation, but
> > will do some measurements to ascertain the same.
>
> I don't think any extra copying is needed.
>

I am talking about access to shared memory instead of the process
local memory.  I understand that an extra copy won't be required.

> The reader can directly
> fread()/pq_copymsgbytes() into shared memory, and the workers can run
> CopyReadLineText() inner loop directly off of the buffer in shared memory.
>

I am slightly confused here.  AFAIU, the for(;;) loop in
CopyReadLineText is about finding the line endings which we thought
that the reader process will do.



-- 
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com