Horton

Duplicate history file?

Kyotaro Horiguchi <horikyota.ntt@gmail.com>

View thread View on postgr.es
From: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
To: pgsql-hackers@lists.postgresql.org
Cc: tatsuro.yamada.tf@nttcom.co.jp
Date: 2021-05-31T07:58:25Z
Lists: pgsql-hackers 
So, I started a thread for this topic diverged from the following
thread.

https://www.postgresql.org/message-id/4698027d-5c0d-098f-9a8e-8cf09e36a555@nttcom.co.jp_1

At Mon, 31 May 2021 11:52:05 +0900, Tatsuro Yamada <tatsuro.yamada.tf@nttcom.co.jp> wrote in 
> Since the above behavior is different from the behavior of the
> test command in the following example in postgresql.conf, I think
> we should write a note about this example.
> 
> # e.g. 'test ! -f /mnt/server/archivedir/%f && cp %p
> # /mnt/server/archivedir/%f'
>
> Let me describe the problem we faced.
> - When archive_mode=always, archive_command is (sometimes) executed
>   in a situation where the history file already exists on the standby
>   side.
> 
> - In this case, if "test ! -f" is written in the archive_command of
>   postgresql.conf on the standby side, the command will keep failing.
> 
>   Note that this problem does not occur when archive_mode=on.
> 
> So, what should we do for the user? I think we should put some notes
> in postgresql.conf or in the documentation. For example, something
> like this:

I'm not sure about the exact configuration you have in mind, but that
would happen on the cascaded standby in the case where the upstream
promotes. In this case, the history file for the new timeline is
archived twice.  walreceiver triggers archiving of the new history
file at the time of the promotion, then startup does the same when it
restores the file from archive.  Is it what you complained about?

The same workaround using the alternative archive script works for the
case.

We could check pg_wal before fetching archive, however, archiving is
not controlled so strictly that duplicate archiving never happens and
I think we choose possible duplicate archiving than having holes in
archive. (so we suggest the "test ! -f" script)

> ====
> Note: If you use archive_mode=always, the archive_command on the
> standby side should not be used "test ! -f".
> ====

It could be one workaround. However, I would suggest not to overwrite
existing files (with a file with different content) to protect archive
from corruption.

We might need to write that in the documentation...

regards.

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

Commits

  1. Further refinement of stuck_on_old_timeline recovery test

  2. Adjust new test case to set wal_keep_segments.

  3. Adjust new test case to set wal_keep_size.

  4. Fix corner case failure of new standby to follow new primary.

  5. Back-port a few PostgresNode.pm methods.

  6. Allow PostgresNode.pm's backup method to accept backup_options.

  7. Add end-to-end testing of pg_basebackup's tar-format output.

  8. Allow PostgresNode.pm tests to wait for catchup

  9. Delay reading timeline history file until it's fetched from master.