pg_rewind WAL segments deletion pitfall

Hello,

It seems for me that there is currently a pitfall in the pg_rewind
implementation.

Imagine the following situation:


There is a cluster consisting of a primary with the following
configuration: wal_level=‘replica’, archive_mode=‘on’ and a replica.

   1. The primary that is not fast enough in archiving WAL segments (e.g.
   network issues, high CPU/Disk load...)
   2. The primary fails
   3. The replica is promoted
   4. We are not lucky enough, the new and the old primary’s timelines
   diverged, we need to run pg_rewind
   5. We are even less lucky: the old primary still has some WAL segments
   with .ready signal files that were generated before the point of divergence
   and were not archived. (e.g. 000000020004D20200000095.done,
   000000020004D20200000096.ready, 000000020004D20200000097.ready,
   000000020004D20200000098.ready)
   6. The promoted primary runs for some time and recycles the old WAL
   segments.
   7. We revive the old primary and try to rewind it
   8. When pg_rewind finished successfully, we see that the WAL segments
   with .ready files are removed, because they were already absent on the
   promoted replica. We end up in a situation where we completely lose some
   WAL segments, even though we had a clear sign that they were not
archived and
   more importantly, pg_rewind read these segments while collecting
   information about the data blocks.
   9. The old primary fails to start because of the missing WAL segments
   (more strictly, the records between the last common checkpoint and the
   point of divergence) with the following log record: "ERROR:  requested WAL
   segment 000000020004D20200000096 has already been removed"


In this situation, after pg_rewind:
archived:

000000020004D20200000095

000000020004D20200000099.partial

000000030004D20200000099


the following segments are lost:

000000020004D20200000096

000000020004D20200000097

000000020004D20200000098


Thus, my thoughts are: why can’t pg_rewind be a little bit wiser in terms
of creating filemap for WALs? Can it preserve the WAL segments that contain
those potentially lost records (> the last common checkpoint and  < the
point of divergence) on the target? (see the patch attached)


If I am missing something however, please correct me or explain why it is
not possible to implement this straightforward solution.


Thank you,

Polina Bungina