Horton

Re: Parallel copy

Amit Kapila <amit.kapila16@gmail.com>

View thread
From: Amit Kapila <amit.kapila16@gmail.com>
To: vignesh C <vignesh21@gmail.com>
Cc: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>, Rafia Sabih <rafia.pghackers@gmail.com>, Ashutosh Sharma <ashu.coek88@gmail.com>, Andres Freund <andres@anarazel.de>, Robert Haas <robertmhaas@gmail.com>, Ants Aasma <ants@cybertec.at>, Tomas Vondra <tomas.vondra@2ndquadrant.com>, Alastair Turner <minion@decodable.me>, Thomas Munro <thomas.munro@gmail.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2020-10-09T05:12:38Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Allow WaitLatch() to be used without a latch.

  2. Add %P to log_line_prefix for parallel group leader

  3. Include replication origins in SQL functions for commit timestamp

  4. Avoid useless buffer allocations during binary COPY FROM.

On Thu, Oct 8, 2020 at 12:14 AM vignesh C <vignesh21@gmail.com> wrote:
>
> On Mon, Sep 28, 2020 at 12:19 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> >
> > I am convinced by the reason given by Kyotaro-San in that another
> > thread [1] and performance data shown by Peter that this can't be an
> > independent improvement and rather in some cases it can do harm. Now,
> > if you need it for a parallel-copy path then we can change it
> > specifically to the parallel-copy code path but I don't understand
> > your reason completely.
> >
>
> Whenever we need data to be populated, we will get a new data block &
> pass it to CopyGetData to populate the data. In case of file copy, the
> server will completely fill the data block. We expect the data to be
> filled completely. If data is available it will completely load the
> complete data block in case of file copy. There is no scenario where
> even if data is present a partial data block will be returned except
> for EOF or no data available. But in case of STDIN data copy, even
> though there is 8K data available in data block & 8K data available in
> STDIN, CopyGetData will return as soon as libpq buffer data is more
> than the minread. We will pass new data block every time to load data.
> Every time we pass an 8K data block but CopyGetData loads a few bytes
> in the new data block & returns. I wanted to keep the same data
> population logic for both file copy & STDIN copy i.e copy full 8K data
> blocks & then the populated data can be required. There is an
> alternative solution I can have some special handling in case of STDIN
> wherein the existing data block can be passed with the index from
> where the data should be copied. Thoughts?
>

What you are proposing as an alternative solution, isn't that what we
are doing without the patch? IIUC, you require this because of your
corresponding changes to handle COPY_NEW_FE in CopyReadLine(), is that
right? If so, what is the difficulty in making it behave similar to
the non-parallel case?

-- 
With Regards,
Amit Kapila.