Re: WIP/PoC for parallel backup

Asif Rehman <asifr.rehman@gmail.com>

From: Asif Rehman <asifr.rehman@gmail.com>

To: Amit Kapila <amit.kapila16@gmail.com>

Cc: Kashif Zeeshan <kashif.zeeshan@enterprisedb.com>, Robert Haas <robertmhaas@gmail.com>, Rajkumar Raghuwanshi <rajkumar.raghuwanshi@enterprisedb.com>, Jeevan Chalke <jeevan.chalke@enterprisedb.com>, PostgreSQL Hackers <pgsql-hackers@postgresql.org>

Date: 2020-04-21T07:29:47Z

Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

Fix failures in incremental_sort due to number of workers
- 23ba3b5ee278 13.0 cited
In jsonb_plpython.c, suppress warning message from gcc 10.
- a06921816370 13.0 cited
Fix minor problems with non-exclusive backup cleanup.
- 303640199d04 13.0 cited

Hi,

I did some tests a while back, and here are the results. The tests were
done to simulate
a live database environment using pgbench.

machine configuration used for this test:
Instance Type:    t2.xlarge
Volume Type  :    io1
Memory (MiB) :    16384
vCPU #           :    4
Architecture    :    X86_64
IOP                 :    16000
Database Size (GB) :    102

The setup consist of 3 machines.
- one for database instances
- one for pg_basebackup client and
- one for pgbench with some parallel workers, simulating SELECT loads.

                                   basebackup | 4 workers | 8 Workers  | 16
workers
Backup Duration(Min):       69.25    |  20.44      | 19.86          | 20.15
(pgbench running with 50 parallel client simulating SELECT load)

Backup Duration(Min):       154.75   |  49.28     | 45.27         | 20.35
(pgbench running with 100 parallel client simulating SELECT load)



On Tue, Apr 21, 2020 at 9:27 AM Amit Kapila <amit.kapila16@gmail.com> wrote:

> On Tue, Apr 14, 2020 at 8:07 PM Asif Rehman <asifr.rehman@gmail.com>
> wrote:
>
>>
>> I forgot to make a check for no-manifest. Fixed. Attached is the updated
>> patch.
>>
>>
> Have we done any performance testing with this patch to see the benefits?
> If so, can you point me to the results? If not, then can we perform some
> tests on large backups to see the benefits of this patch/idea?
>
> --
> With Regards,
> Amit Kapila.
> EnterpriseDB: http://www.enterprisedb.com
>


-- 
--
Asif Rehman
Highgo Software (Canada/China/Pakistan)
URL : www.highgo.ca