Horton

Thread

Commits

  1. Document units for max_slot_wal_keep_size

  1. max_slot_wal_keep_size unit is not specified

    The Post Office <noreply@postgresql.org> — 2021-11-22T17:41:51Z

    The following documentation comment has been logged on the website:

Page: https://www.postgresql.org/docs/14/runtime-config-replication.html
Description:

The unit (I assume it's MB) of max_slot_wal_keep_size is not explicitly
specified in the docs. If it's intentional then please, disregards, but I've
not been able to deduct that from this page alone. Thanks!

  2. Re: max_slot_wal_keep_size unit is not specified

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2021-11-23T14:21:05Z

    On 2021-Nov-22, PG Doc comments form wrote:

> The unit (I assume it's MB) of max_slot_wal_keep_size is not explicitly
> specified in the docs. If it's intentional then please, disregards, but I've
> not been able to deduct that from this page alone. Thanks!

Well, that's embarrasing.  I'll see about fixing it.

It is a size-based unit.  You would typically specify some unit (say,
MB) together with the number; internally, because the way this works is
in terms of whole files, it is rounded down to an integer number of WAL
segments.  If you don't specify a unit, it is taken to be a number of
megabytes.

I wonder why did we make it round down rather than up.  Does this mean
that if you have max_slot_wal_keep_size=8MB and wal segments of 16 MB,
the slot gets invalidated with more than zero reserved segments?

Thanks for reporting.

-- 
Álvaro Herrera         PostgreSQL Developer  —  https://www.EnterpriseDB.com/
"I'm impressed how quickly you are fixing this obscure issue. I came from 
MS SQL and it would be hard for me to put into words how much of a better job
you all are doing on [PostgreSQL]."
 Steve Midgley, http://archives.postgresql.org/pgsql-sql/2008-08/msg00000.php

  3. Re: max_slot_wal_keep_size unit is not specified

    Kyotaro Horiguchi <horikyota.ntt@gmail.com> — 2021-11-24T00:50:13Z

    At Tue, 23 Nov 2021 11:21:05 -0300, Alvaro Herrera <alvherre@alvh.no-ip.org> wrote in 
> On 2021-Nov-22, PG Doc comments form wrote:
> 
> > The unit (I assume it's MB) of max_slot_wal_keep_size is not explicitly
> > specified in the docs. If it's intentional then please, disregards, but I've
> > not been able to deduct that from this page alone. Thanks!
> 
> Well, that's embarrasing.  I'll see about fixing it.

(facepalm..)

> It is a size-based unit.  You would typically specify some unit (say,
> MB) together with the number; internally, because the way this works is
> in terms of whole files, it is rounded down to an integer number of WAL
> segments.  If you don't specify a unit, it is taken to be a number of
> megabytes.
> 
> I wonder why did we make it round down rather than up.  Does this mean
> that if you have max_slot_wal_keep_size=8MB and wal segments of 16 MB,
> the slot gets invalidated with more than zero reserved segments?

That rounding behavior is the way we are generally doing on similar
variables. It is based on the behavior of ConvertToXSegs(). So
max_wal_size for example is rounded-up the same way.

regards.

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

  4. Re: max_slot_wal_keep_size unit is not specified

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2021-11-25T14:37:37Z

    On 2021-Nov-24, Kyotaro Horiguchi wrote:

> At Tue, 23 Nov 2021 11:21:05 -0300, Alvaro Herrera <alvherre@alvh.no-ip.org> wrote in 

> > I wonder why did we make it round down rather than up.  Does this mean
> > that if you have max_slot_wal_keep_size=8MB and wal segments of 16 MB,
> > the slot gets invalidated with more than zero reserved segments?
> 
> That rounding behavior is the way we are generally doing on similar
> variables. It is based on the behavior of ConvertToXSegs(). So
> max_wal_size for example is rounded-up the same way.

After thinking on it some more, I think rounding down is correct.  If
somebody sets max_slot_wal_keep_size=40MB, then by keeping the limit at
32 MB (round down) we're honoring that request better than if we made it
48 MB (round up).  But in any case, I think it doesn't matter much:
users need a *rough* size limit, not a super-precise one.  After all,
this is only verified at checkpoint time.  We're probably dealing with a
few hundreds of megabytes at least, so 16 MB one way or the other don't
change things much.

Here's a proposed patch.

-- 
Álvaro Herrera           39°49'30"S 73°17'W  —  https://www.EnterpriseDB.com/
"Las mujeres son como hondas:  mientras más resistencia tienen,
 más lejos puedes llegar con ellas"  (Jonas Nightingale, Leap of Faith)

  5. Re: max_slot_wal_keep_size unit is not specified

    Kyotaro Horiguchi <horikyota.ntt@gmail.com> — 2021-11-26T07:03:09Z

    At Thu, 25 Nov 2021 11:37:37 -0300, Alvaro Herrera <alvherre@alvh.no-ip.org> wrote in 
> On 2021-Nov-24, Kyotaro Horiguchi wrote:
> 
> > At Tue, 23 Nov 2021 11:21:05 -0300, Alvaro Herrera <alvherre@alvh.no-ip.org> wrote in 
> 
> > > I wonder why did we make it round down rather than up.  Does this mean
> > > that if you have max_slot_wal_keep_size=8MB and wal segments of 16 MB,
> > > the slot gets invalidated with more than zero reserved segments?
> > 
> > That rounding behavior is the way we are generally doing on similar
> > variables. It is based on the behavior of ConvertToXSegs(). So
> > max_wal_size for example is rounded-up the same way.

(no.. rounded-down X)

> After thinking on it some more, I think rounding down is correct.  If
> somebody sets max_slot_wal_keep_size=40MB, then by keeping the limit at
> 32 MB (round down) we're honoring that request better than if we made it
> 48 MB (round up).  But in any case, I think it doesn't matter much:
> users need a *rough* size limit, not a super-precise one.  After all,
> this is only verified at checkpoint time.  We're probably dealing with a
> few hundreds of megabytes at least, so 16 MB one way or the other don't
> change things much.

I thought exactly on that line at development.

Regarding the proposed description, we have never explained how the
same kind of values specified in megabytes is converted into internal
units.  If we add that explanation there, we might want the same for
max_wal_size, min_wal_size and wal_keep_size.

> Here's a proposed patch.

+        The value is rounded down to an integer number of WAL files
+        according to <varname>wal_segment_size</varname>.

The same section starts with the following sentences.

>        Specify the maximum size of WAL files
>        that <link linkend="streaming-replication-slots">replication
>        slots</link> are allowed to retain in the <filename>pg_wal</filename>
>        directory at checkpoint time.

FWIW I thought (or intended) that "the maximum size of WAL files that
slots are *allowed to retain*" connotes the same to the sentences
cited above.

Premising "16 MB one way or the other don't change things much", I'm
not sure we really need to be particular about a few megabites
difference especially only for this specific variable.

regards.

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

  6. Re: max_slot_wal_keep_size unit is not specified

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2021-11-26T17:36:25Z

    On 2021-Nov-26, Kyotaro Horiguchi wrote:

> Regarding the proposed description, we have never explained how the
> same kind of values specified in megabytes is converted into internal
> units.  If we add that explanation there, we might want the same for
> max_wal_size, min_wal_size and wal_keep_size.

True.  Since this was for backpatch, it seemed best to copy the existing
entries instead of changing them.

> The same section starts with the following sentences.
> 
> >        Specify the maximum size of WAL files
> >        that <link linkend="streaming-replication-slots">replication
> >        slots</link> are allowed to retain in the <filename>pg_wal</filename>
> >        directory at checkpoint time.
> 
> FWIW I thought (or intended) that "the maximum size of WAL files that
> slots are *allowed to retain*" connotes the same to the sentences
> cited above.

Right you are -- that addition was redundant, so I pushed without that.

> Premising "16 MB one way or the other don't change things much", I'm
> not sure we really need to be particular about a few megabites
> difference especially only for this specific variable.

Yeah.

-- 
Álvaro Herrera              Valdivia, Chile  —  https://www.EnterpriseDB.com/