Horton

Re: prevent immature WAL streaming

Alvaro Herrera <alvherre@alvh.no-ip.org>

View thread View on postgr.es
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
To: Andres Freund <andres@anarazel.de>
Cc: Pg Hackers <pgsql-hackers@lists.postgresql.org>, "Bossart, Nathan" <bossartn@amazon.com>, Kyotaro Horiguchi <horikyota.ntt@gmail.com>, 蔡梦娟(玊于) <mengjuan.cmj@alibaba-inc.com>, Jakub Wartak <Jakub.Wartak@tomtom.com>
Date: 2021-08-31T13:56:30Z
Lists: pgsql-hackers 
On 2021-Aug-30, Andres Freund wrote:

> I'm doubtful that the approach of adding awareness of record boundaries
> is a good path to go down:

Honestly, I do not like it one bit and if I can avoid relying on them
while making the whole thing work correctly, I am happy.  Clearly it
wasn't a problem for the ancient recovery-only WAL design, but as soon
as we added replication on top the whole issue of continuation records
became a bug.

I do think that the code should be first correct and second performant,
though.
 
> - There are very similar issues with promotions of replicas (consider
>   what happens if we need to promote with the end of local WAL spanning
>   a segment boundary, and what happens to cascading replicas). We have
>   some logic to try to deal with that, but it's pretty grotty and I
>   think incomplete.

Ouch, I hadn't thought of cascading replicas.

> - It seems to make some future optimizations harder - we should work
>   towards replicating data sooner, rather than the opposite. Right now
>   that's a major bottleneck around syncrep.

Absolutely.

> I think a better approach might be to handle this on the WAL layout
> level. What if we never overwrite partial records but instead just
> skipped over them during decoding?

Maybe this is a workable approach, let's work it out fully.

Let me see if I understand what you mean:
* We would remove the logic to inhibit archiving and streaming-
  replicating the tail end of a split WAL record; that logic deals with
  bytes only, so doesn't have to be aware of record boundaries.
* On WAL replay, we ignore records that are split across a segment
  boundary and whose checksum does not match.
* On WAL write ... ?

How do we detect after recovery that a record that was being written,
and potentially was sent to the archive, needs to be "skipped"?

-- 
Álvaro Herrera              Valdivia, Chile  —  https://www.EnterpriseDB.com/

Commits

  1. Fix determination of broken LSN in OVERWRITTEN_CONTRECORD

  2. Fix instability in 026_overwrite_contrecord.pl test.

  3. Fix typo

  4. Change recently added test code for stability

  5. Remove unstable, unnecessary test; fix typo

  6. Repair two portability oversights of new test

  7. Fix WAL replay in presence of an incomplete record

  8. Document XLOG_INCLUDE_XID a little better

  9. Revert "Avoid creating archive status ".ready" files too early"

  10. Avoid creating archive status ".ready" files too early

  11. Fix scenario where streaming standby gets stuck at a continuation record.

  12. Teach xlogreader to follow timeline switches

  13. Forbid using pg_xlogfile_name() and pg_xlogfile_name_offset() during