Re: WIP/PoC for parallel backup

Suraj Kharage <suraj.kharage@enterprisedb.com>

From: Suraj Kharage <suraj.kharage@enterprisedb.com>

To: Robert Haas <robertmhaas@gmail.com>

Cc: Rushabh Lathia <rushabh.lathia@gmail.com>, Ahsan Hadi <ahsan.hadi@gmail.com>, Amit Kapila <amit.kapila16@gmail.com>, David Zhang <david.zhang@highgo.ca>, Asif Rehman <asifr.rehman@gmail.com>, Kashif Zeeshan <kashif.zeeshan@enterprisedb.com>, Rajkumar Raghuwanshi <rajkumar.raghuwanshi@enterprisedb.com>, Jeevan Chalke <jeevan.chalke@enterprisedb.com>, PostgreSQL Hackers <pgsql-hackers@postgresql.org>

Date: 2020-05-22T06:03:29Z

Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

Fix failures in incremental_sort due to number of workers
- 23ba3b5ee278 13.0 cited
In jsonb_plpython.c, suppress warning message from gcc 10.
- a06921816370 13.0 cited
Fix minor problems with non-exclusive backup cleanup.
- 303640199d04 13.0 cited

Attachments

pg_stat_local_bk_j4.txt (text/plain)
pg_stat_local_normal_backup.txt (text/plain)

On Thu, May 21, 2020 at 7:12 PM Robert Haas <robertmhaas@gmail.com> wrote:

>
> So, basically, when we go from 1 process to 4, the additional
> processes spend all of their time waiting rather than doing any useful
> work, and that's why there is no performance benefit. Presumably, the
> reason they spend all their time waiting for ClientRead/ClientWrite is
> because the network between the two machines is saturated, so adding
> more processes that are trying to use it at maximum speed just leads
> to spending more time waiting for it to be available.
>
> Do we have the same results for the local backup case, where the patch
> helped?
>

Here is the result for local backup case (100GB data). Attaching the
captured logs.

The total number of events (pg_stat_activity) captured during local runs:
- 82 events for normal backups
- 31 events for parallel backups (-j 4)

BaseBackupRead wait event numbers: (newly added)
24 - in normal backups
14 - in parallel backup (-j 4)

ClientWrite wait event numbers:
8 - in normal backup
43 - in parallel backups

ClientRead wait event numbers:
0 - ClientRead in normal backup
32 - ClientRead in parallel backups for diff processes.


-- 
--

Thanks & Regards,
Suraj kharage,
EnterpriseDB Corporation,
The Postgres Database Company.