Re: WIP/PoC for parallel backup

Asif Rehman <asifr.rehman@gmail.com>

From: Asif Rehman <asifr.rehman@gmail.com>

To: Robert Haas <robertmhaas@gmail.com>

Cc: Jeevan Chalke <jeevan.chalke@enterprisedb.com>, Ibrar Ahmed <ibrar.ahmad@gmail.com>, PostgreSQL Hackers <pgsql-hackers@postgresql.org>

Date: 2019-12-12T15:19:57Z

Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

Fix failures in incremental_sort due to number of workers
- 23ba3b5ee278 13.0 cited
In jsonb_plpython.c, suppress warning message from gcc 10.
- a06921816370 13.0 cited
Fix minor problems with non-exclusive backup cleanup.
- 303640199d04 13.0 cited

Attachments

0007-parallel-backup-documentation_v7.patch (application/octet-stream) patch v7-0007
0004-Parallel-Backup-Backend-Replication-commands_v7.patch (application/octet-stream) patch v7-0004
0003-Refactor-some-basebackup-code-to-increase-reusabilit_v7.patch (application/octet-stream) patch v7-0003
0006-parallel-backup-testcase_v7.patch (application/octet-stream) patch v7-0006
0005-Parallel-Backup-pg_basebackup_v7.patch (application/octet-stream) patch v7-0005
0002-Rename-sizeonly-to-dryrun-for-few-functions-in-baseb_v7.patch (application/octet-stream) patch v7-0002
0001-removed-PG_ENSURE_ERROR_CLEANUP-macro-from-basebacku_v7.patch (application/octet-stream) patch v7-0001

On Tue, Dec 10, 2019 at 7:34 PM Asif Rehman <asifr.rehman@gmail.com> wrote:

>
>
> On Thu, Nov 28, 2019 at 12:57 AM Robert Haas <robertmhaas@gmail.com>
> wrote:
>
>> On Wed, Nov 27, 2019 at 3:38 AM Jeevan Chalke
>> <jeevan.chalke@enterprisedb.com> wrote:
>> > I am still not sure why we need SEND_BACKUP_FILELIST as a separate
>> command.
>> > Can't we return the file list with START_BACKUP itself?
>>
>> I had the same thought, but I think it's better to keep them separate.
>> Somebody might want to use the SEND_BACKUP_FILELIST command for
>> something other than a backup (I actually think it should be called
>> just SEND_FILE_LIST)
>
>
> Sure. Thanks for the recommendation. To keep the function names in sync, I
> intend to do following the
> following renamings:
> - SEND_BACKUP_FILES --> SEND_FILES
> - SEND_BACKUP_FILELIST -->  SEND_FILE_LIST
>
> . Somebody might want to start a backup without
>> getting a file list because they're going to copy the files at the FS
>> level. Somebody might want to get a list of files to process after
>> somebody else has started the backup on another connection. Or maybe
>> nobody wants to do any of those things, but it doesn't seem to cost us
>> much of anything to split the commands, so I think we should.
>>
>
> +1
>
>
I have updated the patches (v7 attached) and have taken care of all issues
pointed by Jeevan, additionally
ran the pgindent on each patch. Furthermore, Command names have been
renamed as suggested and I
have simplified the SendFiles function. Client can only request the regular
files, any other kind such as
directories or symlinks will be skipped, the client will be responsible for
taking care of such.


--
Asif Rehman
Highgo Software (Canada/China/Pakistan)
URL : www.highgo.ca