Horton

Re: Parallel copy

Tomas Vondra <tomas.vondra@2ndquadrant.com>

View thread
From: Tomas Vondra <tomas.vondra@2ndquadrant.com>
To: Amit Kapila <amit.kapila16@gmail.com>
Cc: David Fetter <david@fetter.org>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2020-02-20T13:36:02Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Allow WaitLatch() to be used without a latch.

  2. Add %P to log_line_prefix for parallel group leader

  3. Include replication origins in SQL functions for commit timestamp

  4. Avoid useless buffer allocations during binary COPY FROM.

On Thu, Feb 20, 2020 at 04:11:39PM +0530, Amit Kapila wrote:
>On Thu, Feb 20, 2020 at 5:12 AM David Fetter <david@fetter.org> wrote:
>>
>> On Fri, Feb 14, 2020 at 01:41:54PM +0530, Amit Kapila wrote:
>> > This work is to parallelize the copy command and in particular "Copy
>> > <table_name> from 'filename' Where <condition>;" command.
>>
>> Apropos of the initial parsing issue generally, there's an interesting
>> approach taken here: https://github.com/robertdavidgraham/wc2
>>
>
>Thanks for sharing.  I might be missing something, but I can't figure
>out how this can help here.  Does this in some way help to allow
>multiple workers to read and tokenize the chunks?
>

I think the wc2 is showing that maybe instead of parallelizing the
parsing, we might instead try using a different tokenizer/parser and
make the implementation more efficient instead of just throwing more
CPUs on it.

I don't know if our code is similar to what wc does, maytbe parsing
csv is more complicated than what wc does.

regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services