Re: BUG #18575: Sometimes pg_rewind mistakenly assumes that nothing needs to be done.

On 08/08/2024 14:58, Georgy Shelkovy wrote:
> this is good log of previous run for comparison
Thanks! That confirms my theory that the minRecoveryPoint in the "bad" 
case is at the LSN where the histories diverged, while in the "good" 
case, it's later. So pg_rewind is not wrong when it says that no rewind 
is required. It's still confusing though. Here's a visualization of the 
scenarios:

Legend:

TLI 1: this the WAL produced on the master
TLI 2: WAL produced on standby1
*: point of divergence
^: minRecoveryPoint on standby2

Good:

             -----------  TLI 2
            /
----------*------------  TLI
                    ^
Bad:

             -----------  TLI 2
            /
----------*------------ TLI 1
           ^

There's a third possibility, which actually produces an assertion 
failure. I was able to reproduce this case by adding some sleeps in the 
script and in walreceiver startup code:


             -----------  TLI 2
            /
----------*------------ TLI 1
     ^

pg_rewind: Source timeline history:
pg_rewind: 1: 0/0 - 0/1138F00
pg_rewind: 2: 0/1138F00 - 0/0
pg_rewind: Target timeline history:
pg_rewind: 1: 0/0 - 0/0
pg_rewind: servers diverged at WAL location 0/1138F00 on timeline 1
pg_rewind: ../src/bin/pg_rewind/pg_rewind.c:443: int main(int, char **): 
Assertion `target_wal_endrec == divergerec' failed.

Except for the assertion failure, I think that's essentially the same as 
the "Bad" case. On a non-assertion build, pg_rewind would report "no 
rewind required" which seems correct.


So it seems true that rewind is not required in those cases. However, if 
the WAL is already written on the standby's disk, just not replayed yet, 
then when you restart the server, it will replay the WAL from timeline 
1. That does seem surprising. Perhaps pg_rewind should just update the 
minRecoveryPoint and minRecoveryTLI in control file in that case, to 
force WAL recovery to follow the timeline switch to TLI 2.

I will try to write a TAP test for the "Bad" and the assertion failure 
case, fix the assertion failure, and test how updating the 
minRecoveryPoint would behave.

-- 
Heikki Linnakangas
Neon (https://neon.tech)