Horton

Thread

Commits

  1. Fix initdb's handling of min_wal_size and max_wal_size.

  2. Reduce memory leakage in initdb.

  3. Add "-c name=value" switch to initdb.

  1. Set arbitrary GUC options during initdb

    Tom Lane <tgl@sss.pgh.pa.us> — 2023-01-25T21:25:19Z

    The attached patch responds to the discussion at [1] about how
we ought to offer a way to set any server GUC from the initdb
command line.  Currently, if for some reason the server won't
start with default parameters, the only way to get through initdb
is to change the installed version of postgresql.conf.sample.
And even that is just a kluge, because the initial probes to
choose max_connections and shared_buffers will all fail, causing
initdb to choose rock-bottom-minimum values of those settings.
You can fix that up after the fact if you notice it, but you
might not.

So this invents an initdb switch "-c NAME=VALUE" just like the
one that the server itself has long had.  The specified settings
are applied on the command line of the initial probe calls
(which happen before we've made any config files), and then they
are added to postgresql.auto.conf, which causes them to take
effect for the bootstrap backend runs as well as subsequent
postmaster starts.

I also invented "--set NAME=VALUE", mainly because just about
every other initdb switch has a long form.  The server itself
doesn't have that spelling, so I'm not wedded to that part.

Thoughts?

			regards, tom lane

[1] https://www.postgresql.org/message-id/flat/17757-dbdfc1f1c954a6db%40postgresql.org

  2. Re: Set arbitrary GUC options during initdb

    Andres Freund <andres@anarazel.de> — 2023-01-25T22:03:39Z

    Hi,

On 2023-01-25 16:25:19 -0500, Tom Lane wrote:
> The attached patch responds to the discussion at [1] about how
> we ought to offer a way to set any server GUC from the initdb
> command line.

Are you thinking of backpatching this, to offer the people affected by the
issue in [1] a way out?


> So this invents an initdb switch "-c NAME=VALUE" just like the
> one that the server itself has long had.

I still am +1 on the idea. I've actually wanted this for development purposes
a couple times...


> The specified settings are applied on the command line of the initial probe
> calls (which happen before we've made any config files), and then they are
> added to postgresql.auto.conf, which causes them to take effect for the
> bootstrap backend runs as well as subsequent postmaster starts.

I think this means that if you set e.g. max_connections as an initdb
parameter, the probes won't do much. Probably fine?


Perhaps worth memorializing the priority of the -c options in a test?
E.g. setting shared_buffers = 20MB or so and then testing that that's the
value when starting the server?


> I also invented "--set NAME=VALUE", mainly because just about
> every other initdb switch has a long form.  The server itself
> doesn't have that spelling, so I'm not wedded to that part.

Fine with me, but also fine to leave out.

Greetings,

Andres Freund

  3. Re: Set arbitrary GUC options during initdb

    Tom Lane <tgl@sss.pgh.pa.us> — 2023-01-25T22:21:14Z

    Andres Freund <andres@anarazel.de> writes:
> On 2023-01-25 16:25:19 -0500, Tom Lane wrote:
>> The attached patch responds to the discussion at [1] about how
>> we ought to offer a way to set any server GUC from the initdb
>> command line.

> Are you thinking of backpatching this, to offer the people affected by the
> issue in [1] a way out?

We could ... it's a new feature for sure, but it seems quite unlikely
to break things for anybody not using it.

>> The specified settings are applied on the command line of the initial probe
>> calls (which happen before we've made any config files), and then they are
>> added to postgresql.auto.conf, which causes them to take effect for the
>> bootstrap backend runs as well as subsequent postmaster starts.

> I think this means that if you set e.g. max_connections as an initdb
> parameter, the probes won't do much. Probably fine?

Right, the probed value will be overridden.

> Perhaps worth memorializing the priority of the -c options in a test?
> E.g. setting shared_buffers = 20MB or so and then testing that that's the
> value when starting the server?

Given that it's written into postgresql.auto.conf, I imagine that
we have test coverage of that point already.

There is a more subtle issue, which is that -c max_connections or
-c shared_buffers should override the probe values *during the
probe steps*.  My first thought about implementation had been to
create postgresql.auto.conf right off the bat, but that would
fail to have this property because server command line overrides
config file.  I can't think of any very portable way to check
that though.

			regards, tom lane

  4. Re: Set arbitrary GUC options during initdb

    Peter Eisentraut <peter.eisentraut@enterprisedb.com> — 2023-01-27T14:48:53Z

    On 25.01.23 22:25, Tom Lane wrote:
> So this invents an initdb switch "-c NAME=VALUE" just like the
> one that the server itself has long had.

This seems useful.

> The specified settings
> are applied on the command line of the initial probe calls
> (which happen before we've made any config files), and then they
> are added to postgresql.auto.conf, which causes them to take
> effect for the bootstrap backend runs as well as subsequent
> postmaster starts.

I would have expected them to be edited into postgresql.conf.  What are 
the arguments for one or the other?

Btw., something that I have had in my notes for a while, but with this 
it would now be officially exposed:  Not all options can be safely set 
during bootstrap.  For example,

     initdb -D data -c track_commit_timestamp=on

will fail an assertion.  This might be an exception, or there might be 
others.

  5. Re: Set arbitrary GUC options during initdb

    Isaac Morland <isaac.morland@gmail.com> — 2023-01-27T15:24:16Z

    On Fri, 27 Jan 2023 at 09:49, Peter Eisentraut <
peter.eisentraut@enterprisedb.com> wrote:

> On 25.01.23 22:25, Tom Lane wrote:
> > So this invents an initdb switch "-c NAME=VALUE" just like the
> > one that the server itself has long had.
>
> This seems useful.
>
> > The specified settings
> > are applied on the command line of the initial probe calls
> > (which happen before we've made any config files), and then they
> > are added to postgresql.auto.conf, which causes them to take
> > effect for the bootstrap backend runs as well as subsequent
> > postmaster starts.
>
> I would have expected them to be edited into postgresql.conf.  What are
> the arguments for one or the other?
>

That would be my expectation also. I believe that is how it works now for
options which can be set by initdb, such as locale and port. I view
postgresql.auto.conf being for temporary changes, or changes related to
different instances within a replication setup, or whatever other uses
people come up with - but not for the permanent configuration established
by initdb.

In particular, I would be surprised if removing a postgresql.auto.conf
completely disabled an instance. Obviously, in my replication setup
example, the replication would be broken, but the basic operation of the
instance would still be possible.

Also, if somebody wants to put a change in postgresql.auto.conf, they can
easily do it using ALTER SYSTEM once the instance is running, or by just
writing out their own postgresql.auto.conf before starting it. Putting a
change in postgresql.conf programmatically is a bit of a pain.

  6. Re: Set arbitrary GUC options during initdb

    Robert Haas <robertmhaas@gmail.com> — 2023-01-27T15:29:37Z

    On Wed, Jan 25, 2023 at 4:26 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> So this invents an initdb switch "-c NAME=VALUE" just like the
> one that the server itself has long had.

HUGE +1 from me. This will, I think, be extremely convenient in many situations.

> The specified settings
> are applied on the command line of the initial probe calls
> (which happen before we've made any config files), and then they
> are added to postgresql.auto.conf, which causes them to take
> effect for the bootstrap backend runs as well as subsequent
> postmaster starts.

I agree with others that it would seem more natural to edit them in
postgresql.conf itself, but I also think it doesn't matter nearly as
much as getting the feature in some form.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  7. Re: Set arbitrary GUC options during initdb

    Tom Lane <tgl@sss.pgh.pa.us> — 2023-01-27T15:34:05Z

    Peter Eisentraut <peter.eisentraut@enterprisedb.com> writes:
> On 25.01.23 22:25, Tom Lane wrote:
>> The specified settings
>> are applied on the command line of the initial probe calls
>> (which happen before we've made any config files), and then they
>> are added to postgresql.auto.conf, which causes them to take
>> effect for the bootstrap backend runs as well as subsequent
>> postmaster starts.

> I would have expected them to be edited into postgresql.conf.  What are 
> the arguments for one or the other?

TBH, the driving reason was that the string-munging code we have in
initdb isn't up to snuff for that: it wants to substitute for an
exactly-known string, which we won't have in the case of an
arbitrary GUC.

One idea if we want to make it work like that could be to stop
trying to edit out the default value, and instead make the file
contents look like, say,

#huge_pages = try                   # on, off, or try
huge_pages = off                    # set by initdb

Then we just need to be able to find the GUC's entry.

> Btw., something that I have had in my notes for a while, but with this 
> it would now be officially exposed:  Not all options can be safely set 
> during bootstrap.  For example,
>      initdb -D data -c track_commit_timestamp=on
> will fail an assertion.  This might be an exception, or there might be 
> others.

Interesting.  We'd probably want to sprinkle some more
do-nothing-in-bootstrap-mode tests as we discover that sort of thing.

			regards, tom lane

  8. Re: Set arbitrary GUC options during initdb

    Robert Haas <robertmhaas@gmail.com> — 2023-01-27T15:41:51Z

    On Fri, Jan 27, 2023 at 10:34 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> One idea if we want to make it work like that could be to stop
> trying to edit out the default value, and instead make the file
> contents look like, say,
>
> #huge_pages = try                   # on, off, or try
> huge_pages = off                    # set by initdb

How about just making replace_token() a little smarter, and maybe renaming it?

The idea is that instead of:

replace_token(conflines, "#max_connections = 100", repltok);

You'd write something like:

replace_guc_value(conflines, "max_connections", repltok);

Which would look for a line matching /^#max_connections\s+=\s/, and
then identify everything following that point up to the first #. It
would replace all that stuff with repltok, but if the replacement is
shorter than the original, it would pad with spaces to get back to the
original length. And otherwise it would add a single space, so that if
you set a super long GUC value there's still at least one space
between the end of the value and the comment that follows.

There might be some quoting-related problems with this idea, not sure.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  9. Re: Set arbitrary GUC options during initdb

    Tom Lane <tgl@sss.pgh.pa.us> — 2023-01-27T15:53:42Z

    Robert Haas <robertmhaas@gmail.com> writes:
> The idea is that instead of:

> replace_token(conflines, "#max_connections = 100", repltok);

> You'd write something like:

> replace_guc_value(conflines, "max_connections", repltok);

> Which would look for a line matching /^#max_connections\s+=\s/, and
> then identify everything following that point up to the first #. It
> would replace all that stuff with repltok, but if the replacement is
> shorter than the original, it would pad with spaces to get back to the
> original length. And otherwise it would add a single space, so that if
> you set a super long GUC value there's still at least one space
> between the end of the value and the comment that follows.

Well, yeah, I was trying to avoid writing that ;-).  There's even
one more wrinkle: we might already have removed the initial '#',
if one does say "-c max_connections=N", because this logic won't
know whether the -c switch matches one of initdb's predetermined
substitutions.

> There might be some quoting-related problems with this idea, not sure.

'#' in a value might confuse it, but we could probably take the last '#'
not the first.

Anyway, it seems like I gotta work harder.  I'll produce a
new patch.

			regards, tom lane

  10. Re: Set arbitrary GUC options during initdb

    David G. Johnston <david.g.johnston@gmail.com> — 2023-01-27T16:01:51Z

    On Fri, Jan 27, 2023 at 8:53 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:

> Robert Haas <robertmhaas@gmail.com> writes:
> > The idea is that instead of:
>
> > replace_token(conflines, "#max_connections = 100", repltok);
>
> > You'd write something like:
>
> > replace_guc_value(conflines, "max_connections", repltok);
>
> > Which would look for a line matching /^#max_connections\s+=\s/, and
> > then identify everything following that point up to the first #. It
> > would replace all that stuff with repltok, but if the replacement is
> > shorter than the original, it would pad with spaces to get back to the
> > original length. And otherwise it would add a single space, so that if
> > you set a super long GUC value there's still at least one space
> > between the end of the value and the comment that follows.
>
> Well, yeah, I was trying to avoid writing that ;-).  There's even
> one more wrinkle: we might already have removed the initial '#',
> if one does say "-c max_connections=N", because this logic won't
> know whether the -c switch matches one of initdb's predetermined
> substitutions.
>
> > There might be some quoting-related problems with this idea, not sure.
>
> '#' in a value might confuse it, but we could probably take the last '#'
> not the first.
>
> Anyway, it seems like I gotta work harder.  I'll produce a
> new patch.
>
>
How about just adding a "section" to the end of the file as needed:

# AdHoc Settings Specified During InitDB
max_connections=75
...

David J.

  11. Re: Set arbitrary GUC options during initdb

    Tom Lane <tgl@sss.pgh.pa.us> — 2023-01-27T16:35:48Z

    "David G. Johnston" <david.g.johnston@gmail.com> writes:
> On Fri, Jan 27, 2023 at 8:53 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> Anyway, it seems like I gotta work harder.  I'll produce a
>> new patch.

> How about just adding a "section" to the end of the file as needed:

> # AdHoc Settings Specified During InitDB
> max_connections=75
> ...

Nah, I think that would be impossibly confusing.  One way or another
the live setting has to be near where the GUC is documented.

We will have to do add-at-the-end for custom GUCs, of course,
but in that case there's no matching comment to confuse you.

			regards, tom lane

  12. Re: Set arbitrary GUC options during initdb

    Tom Lane <tgl@sss.pgh.pa.us> — 2023-01-27T20:02:04Z

    I wrote:
>>> Anyway, it seems like I gotta work harder.  I'll produce a
>>> new patch.

The string-hacking was fully as tedious as I expected.  However, the
output looks pretty nice, and this does have the advantage that the
pre-programmed substitutions become a lot more robust: they are no
longer dependent on the initdb code exactly matching what is in
postgresql.conf.sample.

			regards, tom lane

  13. Re: Set arbitrary GUC options during initdb

    Robert Haas <robertmhaas@gmail.com> — 2023-01-27T20:05:20Z

    On Fri, Jan 27, 2023 at 3:02 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> The string-hacking was fully as tedious as I expected.  However, the
> output looks pretty nice, and this does have the advantage that the
> pre-programmed substitutions become a lot more robust: they are no
> longer dependent on the initdb code exactly matching what is in
> postgresql.conf.sample.

Awesome! Thank you very much.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  14. Re: Set arbitrary GUC options during initdb

    Peter Eisentraut <peter.eisentraut@enterprisedb.com> — 2023-03-22T06:45:18Z

    On 27.01.23 21:02, Tom Lane wrote:
> I wrote:
>>>> Anyway, it seems like I gotta work harder.  I'll produce a
>>>> new patch.
> 
> The string-hacking was fully as tedious as I expected.  However, the
> output looks pretty nice, and this does have the advantage that the
> pre-programmed substitutions become a lot more robust: they are no
> longer dependent on the initdb code exactly matching what is in
> postgresql.conf.sample.

This patch looks good to me.  It's a very nice simplification of the 
initdb.c code, even without the new feature.

I found that the addition of

#include <ctype.h>

didn't appear to be necessary.  Maybe it was required before 
guc_value_requires_quotes() was changed?

I would remove the

#if DEF_PGPORT != 5432

This was in the previous code too, but now if we remove it, then we 
don't have any more hardcoded 5432 left, which seems like a nice 
improvement in cleanliness.

  15. Re: Set arbitrary GUC options during initdb

    Tom Lane <tgl@sss.pgh.pa.us> — 2023-03-22T17:04:35Z

    Peter Eisentraut <peter.eisentraut@enterprisedb.com> writes:
> This patch looks good to me.  It's a very nice simplification of the 
> initdb.c code, even without the new feature.

Thanks for looking!

> I found that the addition of
> #include <ctype.h>
> didn't appear to be necessary.  Maybe it was required before 
> guc_value_requires_quotes() was changed?

There's still an isspace() added by the patch ... oh, the #include
is not needed because port.h includes ctype.h.  That's spectacularly
awful from an include-footprint standpoint, but I can't say that
I want to go fix it right this minute.

> I would remove the
> #if DEF_PGPORT != 5432
> This was in the previous code too, but now if we remove it, then we 
> don't have any more hardcoded 5432 left, which seems like a nice 
> improvement in cleanliness.

Hm.  That'll waste a few cycles during initdb; not sure if the extra
cleanliness is worth it.  It's not like that number is going to change.

			regards, tom lane

  16. Re: Set arbitrary GUC options during initdb

    Tom Lane <tgl@sss.pgh.pa.us> — 2023-03-22T18:33:09Z

    I wrote:
> Peter Eisentraut <peter.eisentraut@enterprisedb.com> writes:
>> I would remove the
>> #if DEF_PGPORT != 5432
>> This was in the previous code too, but now if we remove it, then we 
>> don't have any more hardcoded 5432 left, which seems like a nice 
>> improvement in cleanliness.

> Hm.  That'll waste a few cycles during initdb; not sure if the extra
> cleanliness is worth it.  It's not like that number is going to change.

After further thought I did it as you suggest.  I think the only case
where we really care about shaving milliseconds from initdb is in debug
builds (e.g. buildfarm), which very likely get built with nondefault
DEF_PGPORT anyway.

I did get a bee in my bonnet about how replace_token (and now
replace_guc_value) leak memory like there's no tomorrow.  The leakage
amounts to about a megabyte per run according to valgrind, and it's
not going anywhere but up as we add more calls of those functions.
So I made a quick change to redefine them in a less leak-prone way.

			regards, tom lane

  17. Re: Set arbitrary GUC options during initdb

    Andres Freund <andres@anarazel.de> — 2023-03-22T20:07:51Z

    Hi,

This commit unfortunately broke --wal-segsize. If I use a slightly larger than
the default setting, I get:

initdb --wal-segsize 64 somepath
running bootstrap script ... 2023-03-22 13:06:41.282 PDT [639848] FATAL:  "min_wal_size" must be at least twice "wal_segment_size"

Greetings,

Andres Freund

  18. Re: Set arbitrary GUC options during initdb

    Tom Lane <tgl@sss.pgh.pa.us> — 2023-03-22T20:29:16Z

    Andres Freund <andres@anarazel.de> writes:
> This commit unfortunately broke --wal-segsize. If I use a slightly larger than
> the default setting, I get:
> initdb --wal-segsize 64 somepath
> running bootstrap script ... 2023-03-22 13:06:41.282 PDT [639848] FATAL:  "min_wal_size" must be at least twice "wal_segment_size"

[ confused... ]  Oh, I see the problem.  This:

	/* set default max_wal_size and min_wal_size */
	snprintf(repltok, sizeof(repltok), "min_wal_size = %s",
			 pretty_wal_size(DEFAULT_MIN_WAL_SEGS));
	conflines = replace_token(conflines, "#min_wal_size = 80MB", repltok);

looks like it's setting a compile-time-constant value of min_wal_size;
at least that's what I thought it was doing when I revised the code.
But it isn't, because somebody had the brilliant idea of making
pretty_wal_size() depend on the wal_segment_size_mb variable.

Will fix, thanks for report.

			regards, tom lane

  19. Re: Set arbitrary GUC options during initdb

    Peter Eisentraut <peter.eisentraut@enterprisedb.com> — 2023-04-14T08:29:53Z

    On 27.01.23 15:48, Peter Eisentraut wrote:
> Btw., something that I have had in my notes for a while, but with this 
> it would now be officially exposed:  Not all options can be safely set 
> during bootstrap.  For example,
> 
>      initdb -D data -c track_commit_timestamp=on
> 
> will fail an assertion.  This might be an exception, or there might be 
> others.

I ran a test across all changeable boolean parameters with initdb 
setting it to the opposite of their default.  The only one besides 
track_commit_timestamp that caused initdb to not complete was 
default_transaction_read_only, which is to be expected.

We should fix track_commit_timestamp, but it doesn't look like there is 
wider impact.  (Obviously, this tested only boolean settings.  If 
someone wants to fuzz-test the others ...)