Horton

Re: Parallel copy

Amit Kapila <amit.kapila16@gmail.com>

View thread
From: Amit Kapila <amit.kapila16@gmail.com>
To: Robert Haas <robertmhaas@gmail.com>
Cc: Andres Freund <andres@anarazel.de>, Ants Aasma <ants@cybertec.at>, vignesh C <vignesh21@gmail.com>, Tomas Vondra <tomas.vondra@2ndquadrant.com>, Alastair Turner <minion@decodable.me>, Thomas Munro <thomas.munro@gmail.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2020-05-15T04:19:19Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Allow WaitLatch() to be used without a latch.

  2. Add %P to log_line_prefix for parallel group leader

  3. Include replication origins in SQL functions for commit timestamp

  4. Avoid useless buffer allocations during binary COPY FROM.

On Fri, May 15, 2020 at 1:51 AM Robert Haas <robertmhaas@gmail.com> wrote:
>
> On Thu, May 14, 2020 at 2:18 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > To support that, we need to consider a few things.
> > a. Currently, we increment the command counter each time we take a key
> > share lock on a tuple during trigger execution.  I am really not sure
> > if this is required during Copy command execution or we can just
> > increment it once for the copy.   If we need to increment the command
> > counter just once for copy command then for the parallel copy we can
> > ensure that we do it just once at the end of the parallel copy but if
> > not then we might need some special handling.
>
> My sense is that it would be a lot more sensible to do it at the
> *beginning* of the parallel operation. Once we do it once, we
> shouldn't ever do it again; that's how it works now. Deferring it
> until later seems much more likely to break things.
>

AFAIU, we always increment the command counter after executing the
command.  Why do we want to do it differently here?

> > b.  Another point is that after inserting rows we record CTIDs of the
> > tuples in the event queue and then once all tuples are processed we
> > call FK trigger for each CTID.  Now, with parallelism, the FK checks
> > will be processed once the worker processed one chunk.  I don't see
> > any problem with it but still, this will be a bit different from what
> > we do in serial case.  Do you see any problem with this?
>
> I think there could be some problems here. For instance, suppose that
> there are two entries for different workers for the same CTID.
>

First, let me clarify the CTID I have used in my email are for the
table in which insertion is happening which means FK table.  So, in
such a case, we can't have the same CTIDs queued for different
workers.  Basically, we use CTID to fetch the row from FK table later
and form a query to lock (in KEY SHARE mode) the corresponding tuple
in PK table.  Now, it is possible that two different workers try to
lock the same row of PK table.  I am not clear what problem group
locking can have in this case because these are non-conflicting locks.
Can you please elaborate a bit more?

-- 
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com