Horton

Re: prevent immature WAL streaming

Fujii Masao <masao.fujii@oss.nttdata.com>

View thread View on postgr.es
From: Fujii Masao <masao.fujii@oss.nttdata.com>
To: Andres Freund <andres@anarazel.de>, Alvaro Herrera <alvherre@alvh.no-ip.org>
Cc: Pg Hackers <pgsql-hackers@lists.postgresql.org>, "Bossart, Nathan" <bossartn@amazon.com>, Kyotaro Horiguchi <horikyota.ntt@gmail.com>, 蔡梦娟(玊于) <mengjuan.cmj@alibaba-inc.com>, Jakub Wartak <Jakub.Wartak@tomtom.com>
Date: 2021-09-01T02:34:34Z
Lists: pgsql-hackers 

On 2021/09/01 0:53, Andres Freund wrote:
> I was thinking that on a normal WAL write we'd do nothing. Instead we would
> have dedicated code at the end of recovery that, if the WAL ends in a partial
> record, changes the page following the "valid" portion of the WAL to indicate
> that an incomplete record is to be skipped.

Agreed!



> Of course, we need to be careful to not weaken WAL validity checking too
> much. How about the following:
> 
> If we're "aborting" a continued record, we set XLP_FIRST_IS_ABORTED_PARTIAL on
> the page at which we do so (i.e. the page after the valid end of the WAL).

When do you expect that XLP_FIRST_IS_ABORTED_PARTIAL is set? It's set
when recovery finds a a partially-flushed segment-spanning record?
But maybe we cannot do that (i.e., cannot overwrite the page) because
the page that the flag is set in might have already been archived. No?


> I think we can just read the WAL and see if it ends with a partial
> record. It'd add a bit of complication to the error checking in xlogreader,
> because we'd likely want to treat verification from page headers a bit
> different from verification due to record data. But that seems doable.

Yes.


> Does this make sense?

Yes, I think!


Regards,

-- 
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION

Commits

  1. Fix determination of broken LSN in OVERWRITTEN_CONTRECORD

  2. Fix instability in 026_overwrite_contrecord.pl test.

  3. Fix typo

  4. Change recently added test code for stability

  5. Remove unstable, unnecessary test; fix typo

  6. Repair two portability oversights of new test

  7. Fix WAL replay in presence of an incomplete record

  8. Document XLOG_INCLUDE_XID a little better

  9. Revert "Avoid creating archive status ".ready" files too early"

  10. Avoid creating archive status ".ready" files too early

  11. Fix scenario where streaming standby gets stuck at a continuation record.

  12. Teach xlogreader to follow timeline switches

  13. Forbid using pg_xlogfile_name() and pg_xlogfile_name_offset() during