Horton

Re: prevent immature WAL streaming

Kyotaro Horiguchi <horikyota.ntt@gmail.com>

View thread View on postgr.es
From: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
To: bossartn@amazon.com
Cc: robertmhaas@gmail.com, alvherre@alvh.no-ip.org, pgsql-hackers@lists.postgresql.org, mengjuan.cmj@alibaba-inc.com, Jakub.Wartak@tomtom.com
Date: 2021-08-26T00:40:09Z
Lists: pgsql-hackers 
At Wed, 25 Aug 2021 18:18:59 +0000, "Bossart, Nathan" <bossartn@amazon.com> wrote in 
> On 8/25/21, 5:33 AM, "alvherre@alvh.no-ip.org" <alvherre@alvh.no-ip.org> wrote:
> > On 2021-Aug-24, Bossart, Nathan wrote:
> >> Another interesting thing I see is that the boundary stored in
> >> earliestSegBoundary is not necessarily the earliest one.  It's just
> >> the first one that has been registered.  I did this for simplicity for
> >> the .ready file fix, but I can see it causing problems here.
> >
> > Hmm, is there really a problem here?  Surely the flush point cannot go
> > past whatever has been written.  If somebody is writing an earlier
> > section of WAL, then we cannot move the flush pointer to a later
> > position.  So it doesn't matter if the earliest point we have registered
> > is the true earliest -- we only care for it to be the earliest that is
> > past the flush point.
> 
> Let's say we have the following situation (F = flush, E = earliest
> registered boundary, and L = latest registered boundary), and let's
> assume that each segment has a cross-segment record that ends in the
> next segment.
> 
>         F     E                                         L
>         |-----|-----|-----|-----|-----|-----|-----|-----|
>            1     2     3     4     5     6     7     8
> 
> Then, we write out WAL to disk and create .ready files as needed.  If
> we didn't flush beyond the latest registered boundary, the latest
> registered boundary now becomes the earliest boundary.
> 
>                           F                             E
>         |-----|-----|-----|-----|-----|-----|-----|-----|
>            1     2     3     4     5     6     7     8
> 
> At this point, the earliest segment boundary past the flush point is
> before the "earliest" boundary we are tracking.

We know we have some cross-segment records in the regin [E L] so we
cannot add a .ready file if flush is in the region. I haven't looked
the latest patch (or I may misunderstand the discussion here) but I
think we shouldn't move E before F exceeds previous (or in the first
picture above) L.  Things are done that way in my ancient proposal in
[1].

https://www.postgresql.org/message-id/attachment/117052/v4-0001-Avoid-archiving-a-WAL-segment-that-continues-to-t.patch
+    if (LogwrtResult.Write < firstSegContRecStart ||
+        lastSegContRecEnd <= LogwrtResult.Write)
+    {
        <notify the last segment>

By doing so, at the time of the second picutre, the pointers are set as:

               E           F                             L
         |-----|-----|-----|-----|-----|-----|-----|-----|
            1     2     3     4     5     6     7     8

Then the poiter are cleard at the time F reaches L,

                                                         F
         |-----|-----|-----|-----|-----|-----|-----|-----|
            1     2     3     4     5     6     7     8

Isn't this work correctly?  As I think I mentioned in the thread, I
don't think we don't have so many (more than several, specifically)
segments in a region [E L].

[1] https://www.postgresql.org/message-id/20201216.110120.887433782054853494.horikyota.ntt%40gmail.com

regards.

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

Commits

  1. Fix determination of broken LSN in OVERWRITTEN_CONTRECORD

  2. Fix instability in 026_overwrite_contrecord.pl test.

  3. Fix typo

  4. Change recently added test code for stability

  5. Remove unstable, unnecessary test; fix typo

  6. Repair two portability oversights of new test

  7. Fix WAL replay in presence of an incomplete record

  8. Document XLOG_INCLUDE_XID a little better

  9. Revert "Avoid creating archive status ".ready" files too early"

  10. Avoid creating archive status ".ready" files too early

  11. Fix scenario where streaming standby gets stuck at a continuation record.

  12. Teach xlogreader to follow timeline switches

  13. Forbid using pg_xlogfile_name() and pg_xlogfile_name_offset() during