Horton

Re: Parallel copy

Tomas Vondra <tomas.vondra@2ndquadrant.com>

View thread
From: Tomas Vondra <tomas.vondra@2ndquadrant.com>
To: Andres Freund <andres@anarazel.de>
Cc: Ants Aasma <ants@cybertec.at>, Amit Kapila <amit.kapila16@gmail.com>, Alastair Turner <minion@decodable.me>, Thomas Munro <thomas.munro@gmail.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2020-02-25T16:00:51Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Allow WaitLatch() to be used without a latch.

  2. Add %P to log_line_prefix for parallel group leader

  3. Include replication origins in SQL functions for commit timestamp

  4. Avoid useless buffer allocations during binary COPY FROM.

On Sun, Feb 23, 2020 at 05:09:51PM -0800, Andres Freund wrote:
>Hi,
>
>On 2020-02-19 11:38:45 +0100, Tomas Vondra wrote:
>> I generally agree with the impression that parsing CSV is tricky and
>> unlikely to benefit from parallelism in general. There may be cases with
>> restrictions making it easier (e.g. restrictions on the format) but that
>> might be a bit too complex to start with.
>>
>> For example, I had an idea to parallelise the planning by splitting it
>> into two phases:
>
>FWIW, I think we ought to rewrite our COPY parsers before we go for
>complex schemes. They're way slower than a decent green-field
>CSV/... parser.
>

Yep, that's quite possible.

>
>> The one piece of information I'm missing here is at least a very rough
>> quantification of the individual steps of CSV processing - for example
>> if parsing takes only 10% of the time, it's pretty pointless to start by
>> parallelising this part and we should focus on the rest. If it's 50% it
>> might be a different story. Has anyone done any measurements?
>
>Not recently, but I'm pretty sure that I've observed CSV parsing to be
>way more than 10%.
>

Perhaps. I guess it'll depend on the CSV file (number of fields, ...),
so I still think we need to do some measurements first. I'm willing to
do that, but (a) I doubt I'll have time for that until after 2020-03,
and (b) it'd be good to agree on some set of typical CSV files.

regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services