Re: WIP/PoC for parallel backup

Robert Haas <robertmhaas@gmail.com>

From: Robert Haas <robertmhaas@gmail.com>

To: Kashif Zeeshan <kashif.zeeshan@enterprisedb.com>

Cc: Rajkumar Raghuwanshi <rajkumar.raghuwanshi@enterprisedb.com>, Ahsan Hadi <ahsan.hadi@gmail.com>, Asif Rehman <asifr.rehman@gmail.com>, Jeevan Chalke <jeevan.chalke@enterprisedb.com>, PostgreSQL Hackers <pgsql-hackers@postgresql.org>

Date: 2020-04-02T11:48:33Z

Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

Fix failures in incremental_sort due to number of workers
- 23ba3b5ee278 13.0 cited
In jsonb_plpython.c, suppress warning message from gcc 10.
- a06921816370 13.0 cited
Fix minor problems with non-exclusive backup cleanup.
- 303640199d04 13.0 cited

On Thu, Apr 2, 2020 at 7:30 AM Kashif Zeeshan <
kashif.zeeshan@enterprisedb.com> wrote:

> The backup failed with errors "error: could not connect to server: could
> not look up local user ID 1000: Too many open files" when the
> max_wal_senders was set to 2000.
> The errors generated for the workers starting from backup worke=1017.
>

It wasn't the fact that you set max_wal_senders to 2000. It was the fact
that you specified 1990 parallel workers. By so doing, you overloaded the
machine, which is why everything failed. That's to be expected.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company