Re: [HACKERS] Restricting maximum keep segments by repslots

Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>

From: Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
To: peter.eisentraut@2ndquadrant.com
Cc: sawada.mshk@gmail.com, pgsql-hackers@lists.postgresql.org, thomas.munro@enterprisedb.com, sk@zsrv.org, michael.paquier@gmail.com, andres@anarazel.de
Date: 2018-09-13T09:29:31Z
Lists: pgsql-hackers

Attachments

Hello.

Thank you for the comments, Sawada-san, Peter.

At Mon, 10 Sep 2018 19:52:24 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> wrote in <20180910.195224.22629595.horiguchi.kyotaro@lab.ntt.co.jp>
> At Thu, 6 Sep 2018 22:32:21 +0200, Peter Eisentraut <peter.eisentraut@2ndquadrant.com> wrote in <29bbd79d-696b-509e-578a-0fc38a3b9405@2ndquadrant.com>
> Thanks for pointing that. That's a major cause of confusion. Does
> the following make sense?
> 
> > Specify the maximum size of WAL files that <link
> > linkend="streaming-replication-slots">replication slots</link>
> > are allowed to retain in the <filename>pg_wal</filename>
> > directory at checkpoint time.  If
> > <varname>max_slot_wal_keep_size</varname> is zero (the
> > default), replication slots retain unlimited size of WAL files.
> + If restart_lsn of a replication slot gets behind more than that
> + bytes from the current LSN, the standby using the slot may not
> + be able to reconnect due to removal of required WAL records.
...
> > Also, I don't think 0 is a good value for the default behavior.  0 would
> > mean that a slot is not allowed to retain any more WAL than already
> > exists anyway.  Maybe we don't want to support that directly, but it's a
> > valid configuration.  So maybe use -1 for infinity.
> 
> In realtion to the reply just sent to Sawada-san, remain of a
> slot can be at most 16MB in the 0 case with the default segment
> size. So you're right in this sense. Will fix in the coming
> version. Thanks.

I did the following thinkgs in the new version.

- Changed the disable (or infinite) and default value of
  max_slot_wal_keep_size to -1 from 0.
  (patch 1, 2. guc.c, xlog.c: GetOldestKeepSegment())

- Fixed documentation for max_slot_wal_keep_size tomention what
  happnes when WAL exceeds the size, and additional rewrites.
  (patch 4, catalogs.sgml, config.sgml)

- Folded parameter list of GetOldestKeepSegment().
  (patch 1, 2. xlog.c)

- Provided the plural form of errdetail of checkpoint-time
  warning.  (patch 1, xlog.c: KeepLogSeg())

- Some cosmetic change and small refactor.
  (patch 1, 2, 3)

regards.

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

Commits

  1. Save slot's restart_lsn when invalidated due to size

  2. Fix checkpoint signalling

  3. Check slot->restart_lsn validity in a few more places

  4. Allow users to limit storage reserved by replication slots

  5. Remove header noise from test_decoding test

  6. Rework WAL-reading supporting structs

  7. Flip argument order in XLogSegNoOffsetToRecPtr