Horton

Re: Parallel copy

Kyotaro Horiguchi <horikyota.ntt@gmail.com>

View thread
From: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
To: amit.kapila16@gmail.com
Cc: andrew.dunstan@2ndquadrant.com, minion@decodable.me, thomas.munro@gmail.com, pgsql-hackers@lists.postgresql.org
Date: 2020-02-18T10:59:22Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Allow WaitLatch() to be used without a latch.

  2. Add %P to log_line_prefix for parallel group leader

  3. Include replication origins in SQL functions for commit timestamp

  4. Avoid useless buffer allocations during binary COPY FROM.

At Tue, 18 Feb 2020 15:59:36 +0530, Amit Kapila <amit.kapila16@gmail.com> wrote in 
> On Tue, Feb 18, 2020 at 7:28 AM Kyotaro Horiguchi
> <horikyota.ntt@gmail.com> wrote:
> >
> > In an extreme case, if we didn't see a QUOTE in a chunk, we cannot
> > know the chunk is in a quoted section or not, until all the past
> > chunks are parsed.  After all we are forced to parse fully
> > sequentially as far as we allow QUOTE.
> >
> 
> Right, I think the benefits of this as compared to single reader idea
> would be (a) we can save accessing shared memory for the most part of
> the chunk (b) for non-csv mode, even the tokenization (finding line
> boundaries) would also be parallel.   OTOH, doing processing
> differently for csv and non-csv mode might not be good.

Agreed. So I think it's a good point of compromize.

> > On the other hand, if we allowed "COPY t FROM f WITH (FORMAT CSV,
> > QUOTE '')" in order to signal that there's no quoted section in the
> > file then all chunks would be fully concurrently parsable.
> >
> 
> Yeah, if we can provide such an option, we can probably make parallel
> csv processing equivalent to non-csv.  However, users might not like
> this as I think in some cases it won't be easier for them to tell
> whether the file has quoted fields or not.  I am not very sure of this
> point.

I'm not sure how large portion of the usage contains quoted sections,
so I'm not sure how it is useful..

regards.

-- 
Kyotaro Horiguchi
NTT Open Source Software Center