Horton

Re: Parallel copy

Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>

View thread
From: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
To: Greg Nancarrow <gregn4422@gmail.com>
Cc: vignesh C <vignesh21@gmail.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2020-09-17T05:36:16Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Allow WaitLatch() to be used without a latch.

  2. Add %P to log_line_prefix for parallel group leader

  3. Include replication origins in SQL functions for commit timestamp

  4. Avoid useless buffer allocations during binary COPY FROM.

On Wed, Sep 16, 2020 at 1:20 PM Greg Nancarrow <gregn4422@gmail.com> wrote:
>
> Fortunately I have been given permission to share the exact table
> definition and data I used, so you can check the behaviour and timings
> on your own test machine.
>

Thanks Greg for the script. I ran your test case and I didn't observe
any increase in exec time with 1 worker, indeed, we have benefitted a
few seconds from 0 to 1 worker as expected.

Execution time is in seconds. Each test case is executed 3 times on
release build. Each time the data directory is recreated.

Case 1: 1000000 rows, 2GB
With Patch, default configuration, 0 worker: 88.933, 92.261, 88.423
With Patch, default configuration, 1 worker: 73.825, 74.583, 72.678

With Patch, custom configuration, 0 worker: 76.191, 78.160, 78.822
With Patch, custom configuration, 1 worker: 61.289, 61.288, 60.573

Case 2: 2550000 rows, 5GB
With Patch, default configuration, 0 worker: 246.031, 188.323, 216.683
With Patch, default configuration, 1 worker: 156.299, 153.293, 170.307

With Patch, custom configuration, 0 worker: 197.234, 195.866, 196.049
With Patch, custom configuration, 1 worker: 157.173, 158.287, 157.090

[1] - Custom configuration is set up to ensure that no other processes
influence the results. The postgresql.conf used:
shared_buffers = 40GB
synchronous_commit = off
checkpoint_timeout = 1d
max_wal_size = 24GB
min_wal_size = 15GB
autovacuum = off
max_worker_processes = 32
max_parallel_maintenance_workers = 24
max_parallel_workers = 32

With Regards,
Bharath Rupireddy.
EnterpriseDB: http://www.enterprisedb.com