Thread

PgBackRest Full backup and N/W reliability

KK CHN <kkchn.in@gmail.com> — 2024-08-29T06:21:54Z

List,

I am doing a full backup  using PgBackRest from a production server to
Reposerver.

My connection is  IPSec VPN over ILL  ( 8 Mbps link) between the
 Production DB Server and  the remote RepoServer.

I understood the  bottleneck of 8 Mbps between servers. (Server NICs 10Gbps
and switch)

Query : I have started the  backup command  and it is running (may go for
hours and days as link speed is minimal) .
If the link disconnected or Network error  happens before completion of the
backup command  Definitely the option is to reissue the  backup command
again.

If so, does the backup process start  again from  scratch ?   or it resumes
from  where the backup process is stopped   ?

If it starts from scratch I am afraid that  I can''t complete the initial
full backup never :(

  Or is there a work around if the network connectivity is lost in between
?


Any suggestions much appreciated

Thank you ,
Krishane

[root@dbtest pgbackrest]# sudo -u postgres pgbackrest --stanza=Repo
--log-level-console=info backup

2024-08-29 10:55:27.729 P00   INFO: backup command begin 2.52.1: --delta
--exec-id=523103-56943986 --log-level-console=info --log-level-file=debug
--pg1-host=10.15.0.202 --pg1-host-user=enterprisedb
--pg1-path=/data/edb/as16/data --pg-version-force=16 --process-max=5
--repo1-block --repo1-bundle --repo1-cipher-pass=<redacted>
--repo1-cipher-type=aes-256-cbc --repo1-path=/data/DB_BKUPS
--repo1-retention-diff=2 --repo1-retention-full=2 --stanza=Repo --start-fast
WARN: no prior backup exists, incr backup has been changed to full
2024-08-29 10:55:30.589 P00   INFO: execute non-exclusive backup start:
backup begins after the requested immediate checkpoint completes
2024-08-29 10:55:31.543 P00   INFO: backup start archive =
00000001000000850000004C, lsn = 85/4C0007F8
2024-08-29 10:55:31.543 P00   INFO: check archive for prior segment
00000001000000850000004B


ON Repo Server:
[root@dbtest backup]# date
Thursday 29 August 2024 10:58:08 AM IST
[root@dbtest backup]# du -h
165M    ./Repo
165M

[root@dbtest backup]# date
Thursday 29 August 2024 11:37:03 AM IST
[root@dbtest backup]# du -h
1.9G    ./Repo
1.9G

ON  Production Server    /data/edb/as16/data    directory size is   500 GB

Re: PgBackRest Full backup and N/W reliability

Greg Sabino Mullane <htamfids@gmail.com> — 2024-08-29T13:24:00Z

On Thu, Aug 29, 2024 at 2:21 AM KK CHN <kkchn.in@gmail.com> wrote:

> I am doing a full backup  using PgBackRest from a production server to
> Reposerver.
>
...

> If so, does the backup process start  again from  scratch ?   or it
> resumes from  where the backup process is stopped   ?
>

It resumes. You will see a message like this:

WARN: resumable backup 20240829-091727F of same type exists -- invalid
files will be removed then the backup will resume

Any suggestions much appreciated
>

Boost your process-max as high as you are able to speed up your backup time.

Cheers,
Greg

Re: PgBackRest Full backup and N/W reliability

KK CHN <kkchn.in@gmail.com> — 2024-08-29T16:07:39Z

On Thu, Aug 29, 2024 at 6:54 PM Greg Sabino Mullane <htamfids@gmail.com>
wrote:

> On Thu, Aug 29, 2024 at 2:21 AM KK CHN <kkchn.in@gmail.com> wrote:
>
>> I am doing a full backup  using PgBackRest from a production server to
>> Reposerver.
>>
> ...
>
>> If so, does the backup process start  again from  scratch ?   or it
>> resumes from  where the backup process is stopped   ?
>>
>
> It resumes. You will see a message like this:
>
> WARN: resumable backup 20240829-091727F of same type exists -- invalid
> files will be removed then the backup will resume
>
> Any suggestions much appreciated
>>
>
> Boost your process-max as high as you are able to speed up your backup
> time.
>
>
what will be the ideal process-max number to use ?

 Once I update the process-max param in pgbackrest.conf ,  do I need to
stop and  start the pgback rest ?

Or just editing the pgbackrest.conf  on the fly will reflect the increased
process-max  numbers advantage ?

Is there a limit for proces max. setting ?

> Cheers,
> Greg
>
>
>

Re: PgBackRest Full backup and N/W reliability

Ron <ronljohnsonjr@gmail.com> — 2024-08-29T18:10:45Z

On Thu, Aug 29, 2024 at 12:08 PM KK CHN <kkchn.in@gmail.com> wrote:

>
> On Thu, Aug 29, 2024 at 6:54 PM Greg Sabino Mullane <htamfids@gmail.com>
> wrote:
>
>> On Thu, Aug 29, 2024 at 2:21 AM KK CHN <kkchn.in@gmail.com> wrote:
>>
>>> I am doing a full backup  using PgBackRest from a production server to
>>> Reposerver.
>>>
>> ...
>>
>>> If so, does the backup process start  again from  scratch ?   or it
>>> resumes from  where the backup process is stopped   ?
>>>
>>
>> It resumes. You will see a message like this:
>>
>> WARN: resumable backup 20240829-091727F of same type exists -- invalid
>> files will be removed then the backup will resume
>>
>> Any suggestions much appreciated
>>>
>>
>> Boost your process-max as high as you are able to speed up your backup
>> time.
>>
>>
> what will be the ideal process-max number to use ?
>

We can't answer that.  It depends on what else your PG and PgBackRest
servers are doing when "pgbackrest backup" runs.


> Once I update the process-max param in pgbackrest.conf ,  do I need to
> stop and  start the pgback rest ?
>

Or just editing the pgbackrest.conf on the fly will reflect the increased
> process-max  numbers advantage ?
>

"pgbackrest backup" reads the config file when it starts.


> Is there a limit for proces max. setting ?
>

I think you need managed DBA services.

-- 
Death to America, and butter sauce.
Iraq lobster!