Thread

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Amit Kapila <amit.kapila@huawei.com> — 2013-01-18T10:05:26Z

On Monday, January 14, 2013 6:48 PM Boszormenyi Zoltan wrote:
2013-01-14 07:47 keltezéssel, Amit kapila írta:
> On Sunday, January 13, 2013 2:45 PM Boszormenyi Zoltan wrote:
> 2013-01-13 05:49 keltezéssel, Amit kapila írta:
>> On Sunday, January 13, 2013 12:41 AM Tom Lane wrote:
>> Boszormenyi Zoltan <zb@cybertec.at> writes:

>>>> I can think of below ways to generate tmp file
>>>> 1. Generate session specific tmp file name during operation. This will
be similar to what is
>>>>      currently being done in OpenTemporaryFileinTablespace() to
generate a tempfilename.
>>>> 2. Generate global tmp file name. For this we need to maintain global
counter.
>>>> 3. Always generate temp file with same name "postgresql.auto.conf.tmp",
as at a time only one
>>>>      session is allowed to operate.
>>> What's wrong with mkstemp("config_dir/postgresql.auto.conf.XXXXXX")
>>> that returns   a file descriptor for an already created file with a
unique name?
>> I think for Windows exactly same behavior API might not exist, we might
need to use GetTempFileName.
>> It will generate some different kind of name, which means cleanup also
need to take care of different kind of names.
>> So if this is right, I think it might not be very appropriate to use it.

> In that case (and also because the LWLock still serialize the whole
procedure)
> you can use GetTempFileName() on Windows and tempnam(3) for non-Windows.
> GetTempFileName() and tempnam() return the constructed temporary file name
> out of the directory and the filename prefix components. 

On using mktemp, linux compilation gives below warning
warning: the use of `mktemp' is dangerous, better use `mkstemp' 

So I planned to use mkstemp.

In Windows, there is an api _mktemp_s, followed by open call, behaves in a similar way as mkstemp available in linux. 
The code is changed to use of mkstemp functionality to generate a unique temporary file name instead of .lock file. 
Please check this part of code, I am not very comfortable with using this API.  

Removed the code which is used for deleting the .lock file in case of backend crash or during server startup. 

Added a new function RemoveAutoConfTmpFiles used for deleting the temp files left out any during previous 
postmaster session in the server startup. 

In SendDir(), used sizeof() -1

With Regards,
Amit Kapila.

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Boszormenyi Zoltan <zb@cybertec.at> — 2013-01-18T20:08:30Z

Hi,

comments below.

2013-01-18 11:05 keltezéssel, Amit kapila írta:
> On using mktemp, linux compilation gives below warning
> warning: the use of `mktemp' is dangerous, better use `mkstemp'
>
> So I planned to use mkstemp.

Good.

> In Windows, there is an api _mktemp_s, followed by open call, behaves in a similar way as mkstemp available in linux.
> The code is changed to use of mkstemp functionality to generate a unique temporary file name instead of .lock file.
> Please check this part of code, I am not very comfortable with using this API.

I read the documentation of _mktemp_s() at

http://msdn.microsoft.com/en-us/library/t8ex5e91%28v=vs.80%29.aspx

The comment says it's only for C++, but I can compile your patch using
the MinGW cross-compiler under Fedora 18. So either the MSDN comment
is false, or MinGW provides this function outside C++. More research is
needed on this.

However, I am not sure whether Cygwin provides the mkstemp() call or not.
Searching... Found bugzilla reports against mkstemp on Cygwin. So, yes, it does
and your code would clash with it. In port/dirmod.c:

----8<--------8<--------8<--------8<--------8<----
#if defined(WIN32) || defined(__CYGWIN__)

...

/*
  *      pgmkstemp
  */
int
pgmkstemp (char *TmpFileName)
{
         int err;

         err = _mktemp_s(TmpFileName, strlen(TmpFileName) + 1);
         if (err != 0)
                 return -1;

         return open(TmpFileName, O_CREAT | O_RDWR | O_EXCL, S_IRUSR | S_IWUSR);
}

/* We undefined these above; now redefine for possible use below */
#define rename(from, to)                pgrename(from, to)
#define unlink(path)                    pgunlink(path)
#define mkstemp(tempfilename)   pgmkstemp(tempfilename)
#endif   /* defined(WIN32) || defined(__CYGWIN__) */
----8<--------8<--------8<--------8<--------8<----

pgmkstemp() should be defined in its own block inside

#if defined(WIN32) && !defined(__CYGWIN__)
...
#endif

Same thing applies to src/include/port.h:

----8<--------8<--------8<--------8<--------8<----
#if defined(WIN32) || defined(__CYGWIN__)
/*
  *      Win32 doesn't have reliable rename/unlink during concurrent access.
  */
extern int      pgrename(const char *from, const char *to);
extern int      pgunlink(const char *path);
extern int pgmkstemp (char *TmpFileName);

/* Include this first so later includes don't see these defines */
#ifdef WIN32_ONLY_COMPILER
#include <io.h>
#endif

#define rename(from, to)                pgrename(from, to)
#define unlink(path)                    pgunlink(path)
#define mkstemp(tempfilename)   pgmkstemp(tempfilename)
#endif   /* defined(WIN32) || defined(__CYGWIN__) */
----8<--------8<--------8<--------8<--------8<----

> Removed the code which is used for deleting the .lock file in case of backend crash or during server startup.

Good.

> Added a new function RemoveAutoConfTmpFiles used for deleting the temp files left out any during previous
> postmaster session in the server startup.

Good.

> In SendDir(), used sizeof() -1

Good.

Since you have these defined:

+ #define PG_AUTOCONF_DIR "config_dir"
+ #define PG_AUTOCONF_FILENAME "postgresql.auto.conf"
+ #define PG_AUTOCONF_SAMPLE_TMPFILE "postgresql.auto.conf.XXXXXX"
+ #define PG_AUTOCONF_TMP_FILE    "postgresql.auto.conf."

then the hardcoded values can also be changed everywhere. But to kill
them in initdb.c, these #define's must be in pg_config_manual.h instead of
guc.h.

Most notably, postgresql.conf.sample contains:
     #include_dir = 'auto.conf.d'
and initdb replaces it with
     include_dir = 'config_dir'.
I prefer the auto.conf.d directory name. After killing all hardcoded
"config_dir", changing one #define is all it takes to change it.

There are two blocks of code that Tom commented as being "over-engineered":

+       /* Frame auto conf name and auto conf sample temp file name */
+       snprintf(Filename,sizeof(Filename),"%s/%s", PG_AUTOCONF_DIR, PG_AUTOCONF_FILENAME);
+       StrNCpy(AutoConfFileName, ConfigFileName, sizeof(AutoConfFileName));
+       get_parent_directory(AutoConfFileName);
+       join_path_components(AutoConfFileName, AutoConfFileName, Filename);
+       canonicalize_path(AutoConfFileName);
+
+       snprintf(Filename,sizeof(Filename),"%s/%s", PG_AUTOCONF_DIR, 
PG_AUTOCONF_SAMPLE_TMPFILE);
+       StrNCpy(AutoConfTmpFileName, ConfigFileName, sizeof(AutoConfTmpFileName));
+       get_parent_directory(AutoConfTmpFileName);
+       join_path_components(AutoConfTmpFileName, AutoConfTmpFileName, Filename);
+       canonicalize_path(AutoConfTmpFileName);

You can simply do
         snprintf(AutoConfFileName, sizeof(AutoConfFileName), "%s/%s/%s",
                                                 data_directory,
                                                 PG_AUTOCONF_DIR,
PG_AUTOCONF_FILENAME);
         snprintf(AutoConfTmpFileName, sizeof(AutoConfTmpFileName),"%s/%s/%s",
                                                 data_directory,
                                                 PG_AUTOCONF_DIR,
PG_AUTOCONF_SAMPLE_TMPFILE);

Also, mkstemp() and _mktemp_s() modify the file name in place,
so the XXXXXX suffix becomes something else. You can pass the
array pointer to write_auto_conf_file() together with the returned fd
so the ereport() calls can report the actual file name, not the template.

You removed the static keyword from before AbsoluteConfigLocation(),
resulting in this compiler warning:

In file included from guc.c:9274:0:
guc-file.l:382:1: warning: no previous prototype for 'AbsoluteConfigLocation' 
[-Wmissing-prototypes]

In validate_conf_option() in guc.c, you had "return NULL;" and "return 0;" mixed.
Since this function returns a pointer, I changed all to "return NULL;".

void-returning function write_auto_conf_file() in guc.c had a "return;"
at the end. It's not needed.

I have fixed these in the attached patch and also changed wording
of the documentation, some comments and the error messages to
be more expressive or more concise. The difference between your
patch and mine is also attached.

Best regards,
Zoltán Böszörményi

-- 
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de
      http://www.postgresql.at/

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Tom Lane <tgl@sss.pgh.pa.us> — 2013-01-18T20:32:15Z

Boszormenyi Zoltan <zb@cybertec.at> writes:
> 2013-01-18 11:05 keltezssel, Amit kapila rta:
>> On using mktemp, linux compilation gives below warning
>> warning: the use of `mktemp' is dangerous, better use `mkstemp'
>> 
>> So I planned to use mkstemp.

> Good.

On my HPUX box, the man page disapproves of both, calling them obsolete
(and this man page is old enough to vote, I believe...)

Everywhere else that we need to do something like this, we just use our
own PID to disambiguate, ie
	sprintf(tempfilename, "/path/to/file.%d", (int) getpid());
There is no need to deviate from that pattern or introduce portability
issues, since we can reasonably assume that no non-Postgres programs are
creating files in this directory.

			regards, tom lane

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Boszormenyi Zoltan <zb@cybertec.at> — 2013-01-18T20:48:25Z

2013-01-18 21:32 keltezéssel, Tom Lane írta:
> Boszormenyi Zoltan <zb@cybertec.at> writes:
>> 2013-01-18 11:05 keltezéssel, Amit kapila írta:
>>> On using mktemp, linux compilation gives below warning
>>> warning: the use of `mktemp' is dangerous, better use `mkstemp'
>>>
>>> So I planned to use mkstemp.
>> Good.
> On my HPUX box, the man page disapproves of both, calling them obsolete
> (and this man page is old enough to vote, I believe...)
>
> Everywhere else that we need to do something like this, we just use our
> own PID to disambiguate, ie
> 	sprintf(tempfilename, "/path/to/file.%d", (int) getpid());
> There is no need to deviate from that pattern or introduce portability
> issues, since we can reasonably assume that no non-Postgres programs are
> creating files in this directory.

Thanks for the enlightenment, I will post a new version soon.

>
> 			regards, tom lane
>
>


-- 
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de
      http://www.postgresql.at/

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Boszormenyi Zoltan <zb@cybertec.at> — 2013-01-18T21:07:40Z

2013-01-18 21:48 keltezéssel, Boszormenyi Zoltan írta:
> 2013-01-18 21:32 keltezéssel, Tom Lane írta:
>> Boszormenyi Zoltan <zb@cybertec.at> writes:
>>> 2013-01-18 11:05 keltezéssel, Amit kapila írta:
>>>> On using mktemp, linux compilation gives below warning
>>>> warning: the use of `mktemp' is dangerous, better use `mkstemp'
>>>>
>>>> So I planned to use mkstemp.
>>> Good.
>> On my HPUX box, the man page disapproves of both, calling them obsolete
>> (and this man page is old enough to vote, I believe...)

Then we have to at least consider what this old {p|s}age says... ;-)

>>
>> Everywhere else that we need to do something like this, we just use our
>> own PID to disambiguate, ie
>>     sprintf(tempfilename, "/path/to/file.%d", (int) getpid());
>> There is no need to deviate from that pattern or introduce portability
>> issues, since we can reasonably assume that no non-Postgres programs are
>> creating files in this directory.
>
> Thanks for the enlightenment, I will post a new version soon.

Here it is.

Best regards,
Zoltán Böszörményi

-- 
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de
      http://www.postgresql.at/

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Amit Kapila <amit.kapila@huawei.com> — 2013-01-19T06:35:29Z

On Saturday, January 19, 2013 2:37 AM Boszormenyi Zoltan wrote:
2013-01-18 21:48 keltezéssel, Boszormenyi Zoltan írta:
> 2013-01-18 21:32 keltezéssel, Tom Lane írta:
>> Boszormenyi Zoltan <zb@cybertec.at> writes:
>>> 2013-01-18 11:05 keltezéssel, Amit kapila írta:
>>>>> On using mktemp, linux compilation gives below warning
>>>>> warning: the use of `mktemp' is dangerous, better use `mkstemp'
>>>>
>>>>> So I planned to use mkstemp.
>>>> Good.
>>> On my HPUX box, the man page disapproves of both, calling them obsolete
>>> (and this man page is old enough to vote, I believe...)

>Then we have to at least consider what this old {p|s}age says... ;-)

>>
>>> Everywhere else that we need to do something like this, we just use our
>>> own PID to disambiguate, ie
>>>     sprintf(tempfilename, "/path/to/file.%d", (int) getpid());
>>> There is no need to deviate from that pattern or introduce portability
>>> issues, since we can reasonably assume that no non-Postgres programs are
>>> creating files in this directory.
>
>> Thanks for the enlightenment, I will post a new version soon.

> Here it is.

Thanks a ton for providing better modified version. 
I shall test it on Monday once and then we can conclude on it.

The only point in modifications, I am not comfortable is that in error messages you have 
mentioned (automatic configuration directory). I am not sure if we should use work "automatic"
for config_dir or just use configuration directory.
However I shall keep as it is if no one else has any objection. We can let committer decide about it.

With Regards,
Amit Kapila.

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Craig Ringer <craig@2ndquadrant.com> — 2013-01-21T02:03:32Z

On 01/19/2013 04:08 AM, Boszormenyi Zoltan wrote:
>
> However, I am not sure whether Cygwin provides the mkstemp() call or not.
> Searching... Found bugzilla reports against mkstemp on Cygwin.
Is Cygwin a platform that should be targeted for the server backend
these days?

I can understand making sure that libpq works on Cygwin, but is there
any reason at all to run a Pg server backend on Cygwin rather than as
native Windows binaries?

-- 
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Craig Ringer <craig@2ndquadrant.com> — 2013-01-21T02:05:44Z

On 01/21/2013 10:03 AM, Craig Ringer wrote:
> On 01/19/2013 04:08 AM, Boszormenyi Zoltan wrote:
>> However, I am not sure whether Cygwin provides the mkstemp() call or not.
>> Searching... Found bugzilla reports against mkstemp on Cygwin.
> Is Cygwin a platform that should be targeted for the server backend
> these days?
>
> I can understand making sure that libpq works on Cygwin, but is there
> any reason at all to run a Pg server backend on Cygwin rather than as
> native Windows binaries?

I'm not suggesting immediately dropping working support, since this is
so trivially worked around. I'm just wondering why anybody cares about
the platform.

-- 
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Amit Kapila <amit.kapila@huawei.com> — 2013-01-21T03:17:42Z

On Monday, January 21, 2013 7:36 AM Craig Ringer wrote:
> On 01/21/2013 10:03 AM, Craig Ringer wrote:
> > On 01/19/2013 04:08 AM, Boszormenyi Zoltan wrote:
> >> However, I am not sure whether Cygwin provides the mkstemp() call or
> not.
> >> Searching... Found bugzilla reports against mkstemp on Cygwin.
> > Is Cygwin a platform that should be targeted for the server backend
> > these days?
> >
> > I can understand making sure that libpq works on Cygwin, but is there
> > any reason at all to run a Pg server backend on Cygwin rather than as
> > native Windows binaries?
> 
> I'm not suggesting immediately dropping working support, since this is
> so trivially worked around. I'm just wondering why anybody cares about
> the platform.

We have avoided the use of mkstemp with small native implementation so now
it won't be problem 
for any platform.

With Regards,
Amit Kapila.

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Magnus Hagander <magnus@hagander.net> — 2013-01-21T07:17:10Z

On Jan 21, 2013 3:06 AM, "Craig Ringer" <craig@2ndquadrant.com> wrote:
>
> On 01/21/2013 10:03 AM, Craig Ringer wrote:
> > On 01/19/2013 04:08 AM, Boszormenyi Zoltan wrote:
> >> However, I am not sure whether Cygwin provides the mkstemp() call or
not.
> >> Searching... Found bugzilla reports against mkstemp on Cygwin.
> > Is Cygwin a platform that should be targeted for the server backend
> > these days?
> >
> > I can understand making sure that libpq works on Cygwin, but is there
> > any reason at all to run a Pg server backend on Cygwin rather than as
> > native Windows binaries?
>
> I'm not suggesting immediately dropping working support, since this is
> so trivially worked around. I'm just wondering why anybody cares about
> the platform.

I have suggested similar before, and been voted down :) iirc Andrew uses
it, no? Either way, the consensus earlier had been that as long as it
doesn't require major surgery or blocks something else, we should try to
keep it working. And as you say this sounds like something that can be
handled trivially, I think now is not the time.

/Magnus

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Andrew Dunstan <andrew@dunslane.net> — 2013-01-21T08:01:55Z

On 01/21/2013 02:17 AM, Magnus Hagander wrote:
>
>
> On Jan 21, 2013 3:06 AM, "Craig Ringer" <craig@2ndquadrant.com 
> <mailto:craig@2ndquadrant.com>> wrote:
> >
> > On 01/21/2013 10:03 AM, Craig Ringer wrote:
> > > On 01/19/2013 04:08 AM, Boszormenyi Zoltan wrote:
> > >> However, I am not sure whether Cygwin provides the mkstemp() call 
> or not.
> > >> Searching... Found bugzilla reports against mkstemp on Cygwin.
> > > Is Cygwin a platform that should be targeted for the server backend
> > > these days?
> > >
> > > I can understand making sure that libpq works on Cygwin, but is there
> > > any reason at all to run a Pg server backend on Cygwin rather than as
> > > native Windows binaries?
> >
> > I'm not suggesting immediately dropping working support, since this is
> > so trivially worked around. I'm just wondering why anybody cares about
> > the platform.
>
> I have suggested similar before, and been voted down :) iirc Andrew 
> uses it, no? Either way, the consensus earlier had been that as long 
> as it doesn't require major surgery or blocks something else, we 
> should try to keep it working. And as you say this sounds like 
> something that can be handled trivially, I think now is not the time.
>
>


No, I only use the client. But then I support plenty of things I don't use.

cheers

andrew

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Magnus Hagander <magnus@hagander.net> — 2013-01-21T12:05:16Z

On Mon, Jan 21, 2013 at 9:01 AM, Andrew Dunstan <andrew@dunslane.net> wrote:
>
> On 01/21/2013 02:17 AM, Magnus Hagander wrote:
>
>>
>>
>> On Jan 21, 2013 3:06 AM, "Craig Ringer" <craig@2ndquadrant.com
>> <mailto:craig@2ndquadrant.com>> wrote:
>> >
>> > On 01/21/2013 10:03 AM, Craig Ringer wrote:
>> > > On 01/19/2013 04:08 AM, Boszormenyi Zoltan wrote:
>> > >> However, I am not sure whether Cygwin provides the mkstemp() call or
>> > >> not.
>> > >> Searching... Found bugzilla reports against mkstemp on Cygwin.
>> > > Is Cygwin a platform that should be targeted for the server backend
>> > > these days?
>> > >
>> > > I can understand making sure that libpq works on Cygwin, but is there
>> > > any reason at all to run a Pg server backend on Cygwin rather than as
>> > > native Windows binaries?
>> >
>> > I'm not suggesting immediately dropping working support, since this is
>> > so trivially worked around. I'm just wondering why anybody cares about
>> > the platform.
>>
>> I have suggested similar before, and been voted down :) iirc Andrew uses
>> it, no? Either way, the consensus earlier had been that as long as it
>> doesn't require major surgery or blocks something else, we should try to
>> keep it working. And as you say this sounds like something that can be
>> handled trivially, I think now is not the time.
>>
>
> No, I only use the client. But then I support plenty of things I don't use.

Oh, I somehow thought you were. And yes, we all support things we
don't use - but it certainly helps if there is *someone* out there who
uses it. Having a buildfarm animal (which we do) only goes so far...

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Andrew Dunstan <andrew@dunslane.net> — 2013-01-21T12:55:24Z

On 01/21/2013 07:05 AM, Magnus Hagander wrote:
>> No, I only use the client. But then I support plenty of things I don't use.
> Oh, I somehow thought you were. And yes, we all support things we
> don't use - but it certainly helps if there is *someone* out there who
> uses it. Having a buildfarm animal (which we do) only goes so far...
>


It is being used. I get emails from time to time from people asking 
about it.

cheers

andrew

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Amit Kapila <amit.kapila@huawei.com> — 2013-01-22T12:32:07Z

On Saturday, January 19, 2013 2:37 AM Boszormenyi Zoltan wrote:
2013-01-18 21:48 keltezéssel, Boszormenyi Zoltan írta:
> 2013-01-18 21:32 keltezéssel, Tom Lane írta:
>> Boszormenyi Zoltan <zb@cybertec.at> writes:
>>> 2013-01-18 11:05 keltezéssel, Amit kapila írta:
>>>>> On using mktemp, linux compilation gives below warning
>>>>> warning: the use of `mktemp' is dangerous, better use `mkstemp'
>>>>


>>
>>> Everywhere else that we need to do something like this, we just use our
>>> own PID to disambiguate, ie
>>>     sprintf(tempfilename, "/path/to/file.%d", (int) getpid());
>>> There is no need to deviate from that pattern or introduce portability
>>> issues, since we can reasonably assume that no non-Postgres programs are
>>> creating files in this directory.
>
>> Thanks for the enlightenment, I will post a new version soon.

> Here it is.

The patch sent by you works fine.
It needs small modification as below:

The "auto.conf.d" directory should follow the postgresql.conf file directory not the data_directory. 
The same is validated while parsing the postgresql.conf configuration file. 

Patch is changed to use the postgresql.conf file directory as below. 

StrNCpy(ConfigFileDir, ConfigFileName, sizeof(ConfigFileDir)); 
get_parent_directory(ConfigFileDir); 
/* Frame auto conf name and auto conf sample temp file name */ 
snprintf(AutoConfFileName, sizeof(AutoConfFileName), "%s/%s/%s", 
                                        ConfigFileDir, 
                                        PG_AUTOCONF_DIR, 
                                        PG_AUTOCONF_FILENAME); 

This closes all comments raised till now for this patch.
Kindly let me know if you feel something is missing?

With Regards,
Amit Kapila.

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Boszormenyi Zoltan <zb@cybertec.at> — 2013-01-22T13:39:36Z

2013-01-22 13:32 keltezéssel, Amit kapila írta:
> On Saturday, January 19, 2013 2:37 AM Boszormenyi Zoltan wrote:
> 2013-01-18 21:48 keltezéssel, Boszormenyi Zoltan írta:
>> 2013-01-18 21:32 keltezéssel, Tom Lane írta:
>>> Boszormenyi Zoltan <zb@cybertec.at> writes:
>>>> 2013-01-18 11:05 keltezéssel, Amit kapila írta:
>>>>>> On using mktemp, linux compilation gives below warning
>>>>>> warning: the use of `mktemp' is dangerous, better use `mkstemp'
>
>>>> Everywhere else that we need to do something like this, we just use our
>>>> own PID to disambiguate, ie
>>>>      sprintf(tempfilename, "/path/to/file.%d", (int) getpid());
>>>> There is no need to deviate from that pattern or introduce portability
>>>> issues, since we can reasonably assume that no non-Postgres programs are
>>>> creating files in this directory.
>>> Thanks for the enlightenment, I will post a new version soon.
>> Here it is.
> The patch sent by you works fine.
> It needs small modification as below:
>
> The "auto.conf.d" directory should follow the postgresql.conf file directory not the data_directory.
> The same is validated while parsing the postgresql.conf configuration file.
>
> Patch is changed to use the postgresql.conf file directory as below.
>
> StrNCpy(ConfigFileDir, ConfigFileName, sizeof(ConfigFileDir));
> get_parent_directory(ConfigFileDir);
> /* Frame auto conf name and auto conf sample temp file name */
> snprintf(AutoConfFileName, sizeof(AutoConfFileName), "%s/%s/%s",
>                                          ConfigFileDir,
>                                          PG_AUTOCONF_DIR,
>                                          PG_AUTOCONF_FILENAME);

Maybe it's just me but I prefer to have identical
settings across all replicated servers. But I agree
that there can be use cases with different setups.

All in all, this change makes it necessary to run the
same SET PERSISTENT statements on all slave servers,
too, to make them identical again if the configuration
is separated from the data directory (like on Debian
or Ubuntu using the default packages). This needs to be
documented explicitly.

>
> This closes all comments raised till now for this patch.
> Kindly let me know if you feel something is missing?

I can't think of anything else.

Best regards,
Zoltán Böszörményi

-- 
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de
http://www.postgresql.at/

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Amit Kapila <amit.kapila@huawei.com> — 2013-01-22T14:07:55Z

On Tuesday, January 22, 2013 7:10 PM Zoltán Böszörményi wrote:
> 2013-01-22 13:32 keltezéssel, Amit kapila írta:
> > On Saturday, January 19, 2013 2:37 AM Boszormenyi Zoltan wrote:
> > 2013-01-18 21:48 keltezéssel, Boszormenyi Zoltan írta:
> >> 2013-01-18 21:32 keltezéssel, Tom Lane írta:
> >>> Boszormenyi Zoltan <zb@cybertec.at> writes:
> >>>> 2013-01-18 11:05 keltezéssel, Amit kapila írta:
> >>>>>> On using mktemp, linux compilation gives below warning
> >>>>>> warning: the use of `mktemp' is dangerous, better use `mkstemp'
> >
> >>>> Everywhere else that we need to do something like this, we just
> use our
> >>>> own PID to disambiguate, ie
> >>>>      sprintf(tempfilename, "/path/to/file.%d", (int) getpid());
> >>>> There is no need to deviate from that pattern or introduce
> portability
> >>>> issues, since we can reasonably assume that no non-Postgres
> programs are
> >>>> creating files in this directory.
> >>> Thanks for the enlightenment, I will post a new version soon.
> >> Here it is.
> > The patch sent by you works fine.
> > It needs small modification as below:
> >
> > The "auto.conf.d" directory should follow the postgresql.conf file
> directory not the data_directory.
> > The same is validated while parsing the postgresql.conf configuration
> file.
> >
> > Patch is changed to use the postgresql.conf file directory as below.
> >
> > StrNCpy(ConfigFileDir, ConfigFileName, sizeof(ConfigFileDir));
> > get_parent_directory(ConfigFileDir);
> > /* Frame auto conf name and auto conf sample temp file name */
> > snprintf(AutoConfFileName, sizeof(AutoConfFileName), "%s/%s/%s",
> >                                          ConfigFileDir,
> >                                          PG_AUTOCONF_DIR,
> >                                          PG_AUTOCONF_FILENAME);
> 
> Maybe it's just me but I prefer to have identical
> settings across all replicated servers. But I agree
> that there can be use cases with different setups.
> 
> All in all, this change makes it necessary to run the
> same SET PERSISTENT statements on all slave servers,
> too, to make them identical again if the configuration
> is separated from the data directory (like on Debian
> or Ubuntu using the default packages). This needs to be
> documented explicitly.

SET PERSISTENT command needs to run on all slave servers even if the path we
take w.r.t data directory
unless user takes backup after command. 

> >
> > This closes all comments raised till now for this patch.
> > Kindly let me know if you feel something is missing?
> 
> I can't think of anything else.

In that case can you mark it as "Ready For Committer"

With Regards,
Amit Kapila.

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Fujii Masao <masao.fujii@gmail.com> — 2013-01-22T14:54:56Z

On Tue, Jan 22, 2013 at 11:07 PM, Amit Kapila <amit.kapila@huawei.com> wrote:
> On Tuesday, January 22, 2013 7:10 PM Zoltán Böszörményi wrote:
>> 2013-01-22 13:32 keltezéssel, Amit kapila írta:
>> > On Saturday, January 19, 2013 2:37 AM Boszormenyi Zoltan wrote:
>> > 2013-01-18 21:48 keltezéssel, Boszormenyi Zoltan írta:
>> >> 2013-01-18 21:32 keltezéssel, Tom Lane írta:
>> >>> Boszormenyi Zoltan <zb@cybertec.at> writes:
>> >>>> 2013-01-18 11:05 keltezéssel, Amit kapila írta:
>> >>>>>> On using mktemp, linux compilation gives below warning
>> >>>>>> warning: the use of `mktemp' is dangerous, better use `mkstemp'
>> >
>> >>>> Everywhere else that we need to do something like this, we just
>> use our
>> >>>> own PID to disambiguate, ie
>> >>>>      sprintf(tempfilename, "/path/to/file.%d", (int) getpid());
>> >>>> There is no need to deviate from that pattern or introduce
>> portability
>> >>>> issues, since we can reasonably assume that no non-Postgres
>> programs are
>> >>>> creating files in this directory.
>> >>> Thanks for the enlightenment, I will post a new version soon.
>> >> Here it is.
>> > The patch sent by you works fine.
>> > It needs small modification as below:
>> >
>> > The "auto.conf.d" directory should follow the postgresql.conf file
>> directory not the data_directory.
>> > The same is validated while parsing the postgresql.conf configuration
>> file.
>> >
>> > Patch is changed to use the postgresql.conf file directory as below.
>> >
>> > StrNCpy(ConfigFileDir, ConfigFileName, sizeof(ConfigFileDir));
>> > get_parent_directory(ConfigFileDir);
>> > /* Frame auto conf name and auto conf sample temp file name */
>> > snprintf(AutoConfFileName, sizeof(AutoConfFileName), "%s/%s/%s",
>> >                                          ConfigFileDir,
>> >                                          PG_AUTOCONF_DIR,
>> >                                          PG_AUTOCONF_FILENAME);
>>
>> Maybe it's just me but I prefer to have identical
>> settings across all replicated servers. But I agree
>> that there can be use cases with different setups.
>>
>> All in all, this change makes it necessary to run the
>> same SET PERSISTENT statements on all slave servers,
>> too, to make them identical again if the configuration
>> is separated from the data directory (like on Debian
>> or Ubuntu using the default packages). This needs to be
>> documented explicitly.
>
> SET PERSISTENT command needs to run on all slave servers even if the path we
> take w.r.t data directory
> unless user takes backup after command.

Is it safe to write something in the directory other than data directory
via SQL?

postgres user usually has the write permission for the configuration
directory like /etc/postgresql?

>> > This closes all comments raised till now for this patch.
>> > Kindly let me know if you feel something is missing?
>>
>> I can't think of anything else.

For removing
+   a configuration entry from <filename>postgresql.auto.conf</filename> file,
+   use <command>RESET PERSISTENT</command>. The values will be effective
+   after reload of server configuration (SIGHUP) or server startup.

The description of RESET PERSISTENT is in the document, but it
seems not to be implemented.

Regards,

-- 
Fujii Masao

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Fujii Masao <masao.fujii@gmail.com> — 2013-01-22T16:44:04Z

On Tue, Jan 22, 2013 at 11:54 PM, Fujii Masao <masao.fujii@gmail.com> wrote:
> On Tue, Jan 22, 2013 at 11:07 PM, Amit Kapila <amit.kapila@huawei.com> wrote:
>> On Tuesday, January 22, 2013 7:10 PM Zoltán Böszörményi wrote:
>>> 2013-01-22 13:32 keltezéssel, Amit kapila írta:
>>> > On Saturday, January 19, 2013 2:37 AM Boszormenyi Zoltan wrote:
>>> > 2013-01-18 21:48 keltezéssel, Boszormenyi Zoltan írta:
>>> >> 2013-01-18 21:32 keltezéssel, Tom Lane írta:
>>> >>> Boszormenyi Zoltan <zb@cybertec.at> writes:
>>> >>>> 2013-01-18 11:05 keltezéssel, Amit kapila írta:
>>> >>>>>> On using mktemp, linux compilation gives below warning
>>> >>>>>> warning: the use of `mktemp' is dangerous, better use `mkstemp'
>>> >
>>> >>>> Everywhere else that we need to do something like this, we just
>>> use our
>>> >>>> own PID to disambiguate, ie
>>> >>>>      sprintf(tempfilename, "/path/to/file.%d", (int) getpid());
>>> >>>> There is no need to deviate from that pattern or introduce
>>> portability
>>> >>>> issues, since we can reasonably assume that no non-Postgres
>>> programs are
>>> >>>> creating files in this directory.
>>> >>> Thanks for the enlightenment, I will post a new version soon.
>>> >> Here it is.
>>> > The patch sent by you works fine.
>>> > It needs small modification as below:
>>> >
>>> > The "auto.conf.d" directory should follow the postgresql.conf file
>>> directory not the data_directory.
>>> > The same is validated while parsing the postgresql.conf configuration
>>> file.
>>> >
>>> > Patch is changed to use the postgresql.conf file directory as below.
>>> >
>>> > StrNCpy(ConfigFileDir, ConfigFileName, sizeof(ConfigFileDir));
>>> > get_parent_directory(ConfigFileDir);
>>> > /* Frame auto conf name and auto conf sample temp file name */
>>> > snprintf(AutoConfFileName, sizeof(AutoConfFileName), "%s/%s/%s",
>>> >                                          ConfigFileDir,
>>> >                                          PG_AUTOCONF_DIR,
>>> >                                          PG_AUTOCONF_FILENAME);
>>>
>>> Maybe it's just me but I prefer to have identical
>>> settings across all replicated servers. But I agree
>>> that there can be use cases with different setups.
>>>
>>> All in all, this change makes it necessary to run the
>>> same SET PERSISTENT statements on all slave servers,
>>> too, to make them identical again if the configuration
>>> is separated from the data directory (like on Debian
>>> or Ubuntu using the default packages). This needs to be
>>> documented explicitly.
>>
>> SET PERSISTENT command needs to run on all slave servers even if the path we
>> take w.r.t data directory
>> unless user takes backup after command.
>
> Is it safe to write something in the directory other than data directory
> via SQL?
>
> postgres user usually has the write permission for the configuration
> directory like /etc/postgresql?
>
>>> > This closes all comments raised till now for this patch.
>>> > Kindly let me know if you feel something is missing?
>>>
>>> I can't think of anything else.
>
> For removing
> +   a configuration entry from <filename>postgresql.auto.conf</filename> file,
> +   use <command>RESET PERSISTENT</command>. The values will be effective
> +   after reload of server configuration (SIGHUP) or server startup.
>
> The description of RESET PERSISTENT is in the document, but it
> seems not to be implemented.

I read only document part in the patch and did simple test.
Here are the comments:

storage.sgml needs to be updated.

Doesn't auto.conf.d need to be explained in config.sgml?

When I created my own configuration file in auto.conf.d and
started the server, I got the following LOG message. This
means that users should not create any their own configuration
file there? Why shouldn't they?  I think that it's more useful to
allow users to put also their own configuration files in auto.conf.d.
Then users can include any configuration file without adding
new include derective into postgresql.conf because auto.conf.d
has already been included.

LOG:  unexpected file found in automatic configuration directory:
"/data/auto.conf.d/hoge"

When I removed postgresql.auto.conf and restarted the server,
I got the following warning message. This is not correct because
I didn't remove "auto.conf.d" from postgresql.conf. What I removed
is only postgresql.auto.conf.

WARNING:  Default "auto.conf.d" is not present in postgresql.conf.
Configuration parameters changed with SET PERSISTENT command will not
be effective.

Regards,

-- 
Fujii Masao

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Amit Kapila <amit.kapila@huawei.com> — 2013-01-23T03:24:56Z

On Tuesday, January 22, 2013 8:25 PM Fujii Masao wrote:
> On Tue, Jan 22, 2013 at 11:07 PM, Amit Kapila <amit.kapila@huawei.com>
> wrote:
> > On Tuesday, January 22, 2013 7:10 PM Zoltán Böszörményi wrote:
> >> 2013-01-22 13:32 keltezéssel, Amit kapila írta:
> >> > On Saturday, January 19, 2013 2:37 AM Boszormenyi Zoltan wrote:
> >> > 2013-01-18 21:48 keltezéssel, Boszormenyi Zoltan írta:
> >> >> 2013-01-18 21:32 keltezéssel, Tom Lane írta:
> >> >>> Boszormenyi Zoltan <zb@cybertec.at> writes:
> >> >>>> 2013-01-18 11:05 keltezéssel, Amit kapila írta:
> >> >>>>>> On using mktemp, linux compilation gives below warning
> >> >>>>>> warning: the use of `mktemp' is dangerous, better use
> `mkstemp'
> >> >
> >> >>>> Everywhere else that we need to do something like this, we just
> >> use our
> >> >>>> own PID to disambiguate, ie
> >> >>>>      sprintf(tempfilename, "/path/to/file.%d", (int) getpid());
> >> >>>> There is no need to deviate from that pattern or introduce
> >> portability
> >> >>>> issues, since we can reasonably assume that no non-Postgres
> >> programs are
> >> >>>> creating files in this directory.
> >> >>> Thanks for the enlightenment, I will post a new version soon.
> >> >> Here it is.
> >> > The patch sent by you works fine.
> >> > It needs small modification as below:
> >> >
> >> > The "auto.conf.d" directory should follow the postgresql.conf file
> >> directory not the data_directory.
> >> > The same is validated while parsing the postgresql.conf
> configuration
> >> file.
> >> >
> >> > Patch is changed to use the postgresql.conf file directory as
> below.
> >> >
> >> > StrNCpy(ConfigFileDir, ConfigFileName, sizeof(ConfigFileDir));
> >> > get_parent_directory(ConfigFileDir);
> >> > /* Frame auto conf name and auto conf sample temp file name */
> >> > snprintf(AutoConfFileName, sizeof(AutoConfFileName), "%s/%s/%s",
> >> >                                          ConfigFileDir,
> >> >                                          PG_AUTOCONF_DIR,
> >> >                                          PG_AUTOCONF_FILENAME);
> >>
> >> Maybe it's just me but I prefer to have identical
> >> settings across all replicated servers. But I agree
> >> that there can be use cases with different setups.
> >>
> >> All in all, this change makes it necessary to run the
> >> same SET PERSISTENT statements on all slave servers,
> >> too, to make them identical again if the configuration
> >> is separated from the data directory (like on Debian
> >> or Ubuntu using the default packages). This needs to be
> >> documented explicitly.
> >
> > SET PERSISTENT command needs to run on all slave servers even if the
> path we
> > take w.r.t data directory
> > unless user takes backup after command.
> 
> Is it safe to write something in the directory other than data
> directory
> via SQL?
> 
> postgres user usually has the write permission for the configuration
> directory like /etc/postgresql?

As postgresql.conf will also be in same path and if user can change that,
then what's the problem with postgresql.auto.conf

Do you see any security risk?

> >> > This closes all comments raised till now for this patch.
> >> > Kindly let me know if you feel something is missing?
> >>
> >> I can't think of anything else.
> 
> For removing
> +   a configuration entry from
> <filename>postgresql.auto.conf</filename> file,
> +   use <command>RESET PERSISTENT</command>. The values will be
> effective
> +   after reload of server configuration (SIGHUP) or server startup.
> 
> The description of RESET PERSISTENT is in the document, but it
> seems not to be implemented.

This command support has been removed from patch due to parser issues, so it
should be removed from documentation as well. I shall fix this along with
other issues raised by you.

With Regards,
Amit Kapila.

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Amit Kapila <amit.kapila@huawei.com> — 2013-01-23T09:18:19Z

On Tuesday, January 22, 2013 10:14 PM Fujii Masao wrote:
> On Tue, Jan 22, 2013 at 11:54 PM, Fujii Masao <masao.fujii@gmail.com>
> wrote:
> > On Tue, Jan 22, 2013 at 11:07 PM, Amit Kapila
> <amit.kapila@huawei.com> wrote:
> >> On Tuesday, January 22, 2013 7:10 PM Zoltán Böszörményi wrote:
> >>> 2013-01-22 13:32 keltezéssel, Amit kapila írta:
> >>> > On Saturday, January 19, 2013 2:37 AM Boszormenyi Zoltan wrote:
> >>> > 2013-01-18 21:48 keltezéssel, Boszormenyi Zoltan írta:
> >>> >> 2013-01-18 21:32 keltezéssel, Tom Lane írta:
> >>> >>> Boszormenyi Zoltan <zb@cybertec.at> writes:
> >>> >>>> 2013-01-18 11:05 keltezéssel, Amit kapila írta:
> >>> >>>>>> On using mktemp, linux compilation gives below warning
> >>> >>>>>> warning: the use of `mktemp' is dangerous, better use
> `mkstemp'
> >>> >
> >>> >>>> Everywhere else that we need to do something like this, we
> just
> >>> use our
> >>> >>>> own PID to disambiguate, ie
> >>> >>>>      sprintf(tempfilename, "/path/to/file.%d", (int)
> getpid());
> >>> >>>> There is no need to deviate from that pattern or introduce
> >>> portability
> >>> >>>> issues, since we can reasonably assume that no non-Postgres
> >>> programs are
> >>> >>>> creating files in this directory.
> >>> >>> Thanks for the enlightenment, I will post a new version soon.
> >>> >> Here it is.
> >>> > The patch sent by you works fine.
> >>> > It needs small modification as below:
> >>> >
> >>> > The "auto.conf.d" directory should follow the postgresql.conf
> file
> >>> directory not the data_directory.
> >>> > The same is validated while parsing the postgresql.conf
> configuration
> >>> file.
> >>> >
> >>> > Patch is changed to use the postgresql.conf file directory as
> below.
> >>> >
> >>> > StrNCpy(ConfigFileDir, ConfigFileName, sizeof(ConfigFileDir));
> >>> > get_parent_directory(ConfigFileDir);
> >>> > /* Frame auto conf name and auto conf sample temp file name */
> >>> > snprintf(AutoConfFileName, sizeof(AutoConfFileName), "%s/%s/%s",
> >>> >                                          ConfigFileDir,
> >>> >                                          PG_AUTOCONF_DIR,
> >>> >                                          PG_AUTOCONF_FILENAME);
> >>>
> >>> Maybe it's just me but I prefer to have identical
> >>> settings across all replicated servers. But I agree
> >>> that there can be use cases with different setups.
> >>>
> >>> All in all, this change makes it necessary to run the
> >>> same SET PERSISTENT statements on all slave servers,
> >>> too, to make them identical again if the configuration
> >>> is separated from the data directory (like on Debian
> >>> or Ubuntu using the default packages). This needs to be
> >>> documented explicitly.
> >>
> >> SET PERSISTENT command needs to run on all slave servers even if the
> path we
> >> take w.r.t data directory
> >> unless user takes backup after command.
> >
> > Is it safe to write something in the directory other than data
> directory
> > via SQL?
> >
> > postgres user usually has the write permission for the configuration
> > directory like /etc/postgresql?
> >
> >>> > This closes all comments raised till now for this patch.
> >>> > Kindly let me know if you feel something is missing?
> >>>
> >>> I can't think of anything else.
> >
> > For removing
> > +   a configuration entry from
> <filename>postgresql.auto.conf</filename> file,
> > +   use <command>RESET PERSISTENT</command>. The values will be
> effective
> > +   after reload of server configuration (SIGHUP) or server startup.
> >
> > The description of RESET PERSISTENT is in the document, but it
> > seems not to be implemented.
> 
> I read only document part in the patch and did simple test.
> Here are the comments:
> 
> storage.sgml needs to be updated.
> 
> Doesn't auto.conf.d need to be explained in config.sgml?

I shall update both these files.
 
> When I created my own configuration file in auto.conf.d and
> started the server, I got the following LOG message. This
> means that users should not create any their own configuration
> file there? Why shouldn't they?

It can be allowed, but for now it has been kept such that automatically
generated conf files will be
present in this directory, thats what the name 'auto.conf.d' suggests.

>  I think that it's more useful to
> allow users to put also their own configuration files in auto.conf.d.
> Then users can include any configuration file without adding
> new include derective into postgresql.conf because auto.conf.d
> has already been included.
> 
> LOG:  unexpected file found in automatic configuration directory:
> "/data/auto.conf.d/hoge"

Personally I don't have objection in getting other user specific conf files
in this directory. 
But I think then we should name this directory also differently as it was
initially (conf_dir) in the Patch.
I would like to see opinion of others also in this matter, so that later I
don't need to
change it back to what currently it is.

> When I removed postgresql.auto.conf and restarted the server,
> I got the following warning message. This is not correct because
> I didn't remove "auto.conf.d" from postgresql.conf. What I removed
> is only postgresql.auto.conf.
> 
> WARNING:  Default "auto.conf.d" is not present in postgresql.conf.
> Configuration parameters changed with SET PERSISTENT command will not
> be effective.

How about changing it to below message:

WARNING:  File 'postgresql.auto.conf' is not accessible, either file
'postgresql.auto.conf' or folder '%s' doesn't exist or default "auto.conf.d"
is not present in postgresql.conf. 
Configuration parameters changed with SET PERSISTENT command will not be
effective.

The reason I have kept single error message for all issues related to
'postgresql.auto.conf' is to keep code little simpler, else I need to
distinguish for each particular case.

With Regards,
Amit Kapila.

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Fujii Masao <masao.fujii@gmail.com> — 2013-01-23T15:52:13Z

On Wed, Jan 23, 2013 at 12:24 PM, Amit Kapila <amit.kapila@huawei.com> wrote:
> On Tuesday, January 22, 2013 8:25 PM Fujii Masao wrote:
>> On Tue, Jan 22, 2013 at 11:07 PM, Amit Kapila <amit.kapila@huawei.com>
>> wrote:
>> > On Tuesday, January 22, 2013 7:10 PM Zoltán Böszörményi wrote:
>> >> 2013-01-22 13:32 keltezéssel, Amit kapila írta:
>> >> > On Saturday, January 19, 2013 2:37 AM Boszormenyi Zoltan wrote:
>> >> > 2013-01-18 21:48 keltezéssel, Boszormenyi Zoltan írta:
>> >> >> 2013-01-18 21:32 keltezéssel, Tom Lane írta:
>> >> >>> Boszormenyi Zoltan <zb@cybertec.at> writes:
>> >> >>>> 2013-01-18 11:05 keltezéssel, Amit kapila írta:
>> >> >>>>>> On using mktemp, linux compilation gives below warning
>> >> >>>>>> warning: the use of `mktemp' is dangerous, better use
>> `mkstemp'
>> >> >
>> >> >>>> Everywhere else that we need to do something like this, we just
>> >> use our
>> >> >>>> own PID to disambiguate, ie
>> >> >>>>      sprintf(tempfilename, "/path/to/file.%d", (int) getpid());
>> >> >>>> There is no need to deviate from that pattern or introduce
>> >> portability
>> >> >>>> issues, since we can reasonably assume that no non-Postgres
>> >> programs are
>> >> >>>> creating files in this directory.
>> >> >>> Thanks for the enlightenment, I will post a new version soon.
>> >> >> Here it is.
>> >> > The patch sent by you works fine.
>> >> > It needs small modification as below:
>> >> >
>> >> > The "auto.conf.d" directory should follow the postgresql.conf file
>> >> directory not the data_directory.
>> >> > The same is validated while parsing the postgresql.conf
>> configuration
>> >> file.
>> >> >
>> >> > Patch is changed to use the postgresql.conf file directory as
>> below.
>> >> >
>> >> > StrNCpy(ConfigFileDir, ConfigFileName, sizeof(ConfigFileDir));
>> >> > get_parent_directory(ConfigFileDir);
>> >> > /* Frame auto conf name and auto conf sample temp file name */
>> >> > snprintf(AutoConfFileName, sizeof(AutoConfFileName), "%s/%s/%s",
>> >> >                                          ConfigFileDir,
>> >> >                                          PG_AUTOCONF_DIR,
>> >> >                                          PG_AUTOCONF_FILENAME);
>> >>
>> >> Maybe it's just me but I prefer to have identical
>> >> settings across all replicated servers. But I agree
>> >> that there can be use cases with different setups.
>> >>
>> >> All in all, this change makes it necessary to run the
>> >> same SET PERSISTENT statements on all slave servers,
>> >> too, to make them identical again if the configuration
>> >> is separated from the data directory (like on Debian
>> >> or Ubuntu using the default packages). This needs to be
>> >> documented explicitly.
>> >
>> > SET PERSISTENT command needs to run on all slave servers even if the
>> path we
>> > take w.r.t data directory
>> > unless user takes backup after command.
>>
>> Is it safe to write something in the directory other than data
>> directory
>> via SQL?
>>
>> postgres user usually has the write permission for the configuration
>> directory like /etc/postgresql?
>
> As postgresql.conf will also be in same path and if user can change that,
> then what's the problem with postgresql.auto.conf

If the configuration directory like /etc is owned by root and only root has
a write permission for it, the user running PostgreSQL server would not
be able to update postgresql.auto.conf there. OTOH, even in that case,
a user can switch to root and update the configuration file there. I'm not
sure whether the configuration directory is usually writable by the user
running PostgreSQL server in Debian or Ubuntu, though.

> Do you see any security risk?

I have no idea. I just wondered that because some functions like pg_file_write()
in adminpack are restricted to write something in the directory other
than $PGDATA.

>> >> > This closes all comments raised till now for this patch.
>> >> > Kindly let me know if you feel something is missing?
>> >>
>> >> I can't think of anything else.
>>
>> For removing
>> +   a configuration entry from
>> <filename>postgresql.auto.conf</filename> file,
>> +   use <command>RESET PERSISTENT</command>. The values will be
>> effective
>> +   after reload of server configuration (SIGHUP) or server startup.
>>
>> The description of RESET PERSISTENT is in the document, but it
>> seems not to be implemented.
>
> This command support has been removed from patch due to parser issues, so it
> should be removed from documentation as well. I shall fix this along with
> other issues raised by you.

Okay.

Regards,

-- 
Fujii Masao

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Fujii Masao <masao.fujii@gmail.com> — 2013-01-23T16:21:07Z

On Wed, Jan 23, 2013 at 6:18 PM, Amit Kapila <amit.kapila@huawei.com> wrote:
> On Tuesday, January 22, 2013 10:14 PM Fujii Masao wrote:
>> When I removed postgresql.auto.conf and restarted the server,
>> I got the following warning message. This is not correct because
>> I didn't remove "auto.conf.d" from postgresql.conf. What I removed
>> is only postgresql.auto.conf.
>>
>> WARNING:  Default "auto.conf.d" is not present in postgresql.conf.
>> Configuration parameters changed with SET PERSISTENT command will not
>> be effective.
>
> How about changing it to below message:
>
> WARNING:  File 'postgresql.auto.conf' is not accessible, either file
> 'postgresql.auto.conf' or folder '%s' doesn't exist or default "auto.conf.d"
> is not present in postgresql.conf.
> Configuration parameters changed with SET PERSISTENT command will not be
> effective.

Or we should suppress such a warning message in the case where
postgresql.auto.conf doesn't exist? SET PERSISTENT creates that
file automatically if it doesn't exist. So we can expect that configuration
parameters changed with SET PERSISTENT WILL be effective.

This warning message implies that the line "include_dir 'auto.conf.d'"
must not be removed from postgresql.conf? If so, we should warn that
in both document and postgresql.conf.sample? Or we should hard-code
so that something like auto.conf.d is always included even when that
include_dir directive doesn't exist?

Regards,

-- 
Fujii Masao

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Alvaro Herrera <alvherre@2ndquadrant.com> — 2013-01-23T18:20:51Z

Fujii Masao escribió:
> On Wed, Jan 23, 2013 at 12:24 PM, Amit Kapila <amit.kapila@huawei.com> wrote:

> >> Is it safe to write something in the directory other than data
> >> directory
> >> via SQL?
> >>
> >> postgres user usually has the write permission for the configuration
> >> directory like /etc/postgresql?
> >
> > As postgresql.conf will also be in same path and if user can change that,
> > then what's the problem with postgresql.auto.conf
> 
> If the configuration directory like /etc is owned by root and only root has
> a write permission for it, the user running PostgreSQL server would not
> be able to update postgresql.auto.conf there. OTOH, even in that case,
> a user can switch to root and update the configuration file there. I'm not
> sure whether the configuration directory is usually writable by the user
> running PostgreSQL server in Debian or Ubuntu, though.

Yes it is -- the /etc/postgresql/<version>/<cluster> directory (where
postgresql.conf resides) is owned by user postgres.

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Andres Freund <andres@2ndquadrant.com> — 2013-01-23T20:25:48Z

On 2013-01-22 12:32:07 +0000, Amit kapila wrote:
> This closes all comments raised till now for this patch.
> Kindly let me know if you feel something is missing?

I am coming late to this patch, so bear with me if I repeat somethign
said elsewhere.

Review comments of cursory pass through the patch:
* most comments are hard to understand. I know the problem of that
  being hard for a non-native speaker by heart, but I think another pass
  over them would be good thing.
* The gram.y changes arround set_rest_(more|common) seem pretty confused
  to me. E.g. its not possible anymore to set the timezone for a
  function. And why is it possible to persistently set the search path,
  but not client encoding? Why is FROM CURRENT in set_rest_more?
* set_config_file should elog(ERROR), not return on an unhandled
  setstmt->kind
* why are you creating AutoConfFileName if its not stat'able? It seems
  better to simply skip parsing the old file in that case
* Writing the temporary file to .$pid seems like a bad idea, better use
  one file for that, SET PERSISTENT is protected by an exclusive lock
  anyway.
* the write sequence should be:
  * fsync(tempfile)
  * fsync(directory)
  * rename(tempfile, configfile)
  * fsync(configfile)
  * fsync(directory)
* write_auto_conf_file should probably escape quoted values?
* coding style should be adhered to more closesly, there are many
  if (pointer) which should be if (pointer != NULL), single-line blocks
  enclosed in curlies which shouldn't, etc.
* replace_auto_config_file and surrounding functions need more comments
  in the header
* the check that prevents persistent SETs in a transaction should rather
  be in utility.c and use PreventTransactionChain like most of the
  others that need to do that (c.f. T_CreatedbStmt).

I think this patch is a good bit away of being ready for committer...

Greetings,

Andres Freund

-- 
 Andres Freund	                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Amit Kapila <amit.kapila@huawei.com> — 2013-01-24T05:55:47Z

On Wednesday, January 23, 2013 9:51 PM Fujii Masao wrote:
> On Wed, Jan 23, 2013 at 6:18 PM, Amit Kapila <amit.kapila@huawei.com>
> wrote:
> > On Tuesday, January 22, 2013 10:14 PM Fujii Masao wrote:
> >> When I removed postgresql.auto.conf and restarted the server,
> >> I got the following warning message. This is not correct because
> >> I didn't remove "auto.conf.d" from postgresql.conf. What I removed
> >> is only postgresql.auto.conf.
> >>
> >> WARNING:  Default "auto.conf.d" is not present in postgresql.conf.
> >> Configuration parameters changed with SET PERSISTENT command will
> not
> >> be effective.
> >
> > How about changing it to below message:
> >
> > WARNING:  File 'postgresql.auto.conf' is not accessible, either file
> > 'postgresql.auto.conf' or folder '%s' doesn't exist or default
> "auto.conf.d"
> > is not present in postgresql.conf.
> > Configuration parameters changed with SET PERSISTENT command will not
> be
> > effective.
> 
> Or we should suppress such a warning message in the case where
> postgresql.auto.conf doesn't exist? SET PERSISTENT creates that
> file automatically if it doesn't exist. So we can expect that
> configuration
> parameters changed with SET PERSISTENT WILL be effective.

This Warning (message) can come during startup or reload, so if at that time
postgresql.auto.conf doesn't exist the parameters set by SET PERSISTENT
previous to startup or reload will not be effective.
So IMO, we can think to change part of this message as below:

"Configuration parameters changed before start of server with SET PERSISTENT
command will not be effective."
Any other idea for change in message?
 
> This warning message implies that the line "include_dir 'auto.conf.d'"
> must not be removed from postgresql.conf? If so, we should warn that
> in both document and postgresql.conf.sample? 
  
 I shall fix this as below:

Change current WARNING in postgresql.conf.sample as below:

# This includes the default configuration directory included to support 
# SET PERSISTENT statement. 
# 
# WARNING:  User should not remove below include_dir or directory
auto.conf.d, 
            as both are required to make SET PERSISTENT command work.
            Any configuration parameter values specified after this line 
#           will override the values set by SET PERSISTENT statement.

Also Update this in runtime.sgml under heading "Starting the Database
Server"

> Or we should hard-code
> so that something like auto.conf.d is always included even when that
> include_dir directive doesn't exist?

I think this can create problem for user who intentionally wants to remove
this directory inclusion. 

With Regards,
Amit Kapila.

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Amit Kapila <amit.kapila@huawei.com> — 2013-01-24T06:01:16Z

On Wednesday, January 23, 2013 11:51 PM Alvaro Herrera wrote:
> Fujii Masao escribió:
> > On Wed, Jan 23, 2013 at 12:24 PM, Amit Kapila
> <amit.kapila@huawei.com> wrote:
> 
> > >> Is it safe to write something in the directory other than data
> > >> directory
> > >> via SQL?
> > >>
> > >> postgres user usually has the write permission for the
> configuration
> > >> directory like /etc/postgresql?
> > >
> > > As postgresql.conf will also be in same path and if user can change
> that,
> > > then what's the problem with postgresql.auto.conf
> >
> > If the configuration directory like /etc is owned by root and only
> root has
> > a write permission for it, the user running PostgreSQL server would
> not
> > be able to update postgresql.auto.conf there. OTOH, even in that
> case,
> > a user can switch to root and update the configuration file there.
> I'm not
> > sure whether the configuration directory is usually writable by the
> user
> > running PostgreSQL server in Debian or Ubuntu, though.
> 
> Yes it is -- the /etc/postgresql/<version>/<cluster> directory (where
> postgresql.conf resides) is owned by user postgres.

So in that case we can consider postgresql.auto.conf to be in path w.r.t
postgresql.conf instead of data directory.

With Regards,
Amit Kapila.

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Amit Kapila <amit.kapila@huawei.com> — 2013-01-24T11:15:42Z

On Thursday, January 24, 2013 1:56 AM Andres Freund wrote:
> On 2013-01-22 12:32:07 +0000, Amit kapila wrote:
> > This closes all comments raised till now for this patch.
> > Kindly let me know if you feel something is missing?
> 
> I am coming late to this patch, so bear with me if I repeat somethign
> said elsewhere.

Thanks for looking at patch.

> Review comments of cursory pass through the patch:
> * most comments are hard to understand. I know the problem of that
>   being hard for a non-native speaker by heart, but I think another
> pass
>   over them would be good thing.

I shall give another pass to improve the comments in code.

> * The gram.y changes arround set_rest_(more|common) seem pretty
> confused
>   to me. 

>E.g. its not possible anymore to set the timezone for a  function. 

What do you exactly mean by this part of comment.

> And why is it possible to persistently set the search path,
>   but not client encoding? Why is FROM CURRENT in set_rest_more?

I think the main reason was some of the commands can work only in
transaction 
and as SET Persistent cannot work inside transaction block, so I had kept
some seggregation. 
I shall reply you the exact reason in separate mail.

> * set_config_file should elog(ERROR), not return on an unhandled
>   setstmt->kind

Shall fix it.

> * why are you creating AutoConfFileName if its not stat'able? It seems
>   better to simply skip parsing the old file in that case

Sure, it's better to handle as you are suggesting.

> * Writing the temporary file to .$pid seems like a bad idea, better use
>   one file for that, SET PERSISTENT is protected by an exclusive lock
>   anyway.

  I think we can use one temporary file, infact that was one of the ways I
have asked in one of the previous mails. 
  However Tom and Zoltan felt this is better way to do it. 
  I can think of one reason where it will be better to have .$pid, and that
is if some one has opened the 
  file manually, then all other sessions can fail (in WINDOWS). Infact this
is one of test Zoltan had performed.

> * the write sequence should be:
>   * fsync(tempfile)
>   * fsync(directory)
>   * rename(tempfile, configfile)
>   * fsync(configfile)
>   * fsync(directory)

Why do we need fsync(directory) and fsync(configfile)?

> * write_auto_conf_file should probably escape quoted values?
  Can you please elaborate more, I am not able to understand your point?

> * coding style should be adhered to more closesly, there are many
>   if (pointer) which should be if (pointer != NULL), 

Are you pointing in function  validate_conf_option(const char *name, char
*value) 
  for below usage: 
  +                                 if (value)

> single-line blocks
>   enclosed in curlies which shouldn't, etc.
Shall fix.

> * replace_auto_config_file and surrounding functions need more comments
>   in the header

  Sure, shall add more comments in function header of following functions: 
  validate_conf_option() 
  RemoveAutoConfTmpFiles() 
  write_auto_conf_file() 
  replace_auto_config_value()

> * the check that prevents persistent SETs in a transaction should
> rather
>   be in utility.c and use PreventTransactionChain like most of the
>   others that need to do that (c.f. T_CreatedbStmt).

  We can move the check in utility.c, but if we use PreventTransactionChain,
then it will be disallowed 
  in functions as well. But the original idea was to not support in
transaction blocks only. 
  So I feel use of current function IsTransactionBlock() should be okay.

With Regards,
Amit Kapila.

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Andres Freund <andres@2ndquadrant.com> — 2013-01-24T11:54:31Z

On 2013-01-24 16:45:42 +0530, Amit Kapila wrote:
> > * The gram.y changes arround set_rest_(more|common) seem pretty
> > confused
> >   to me.
>
> >E.g. its not possible anymore to set the timezone for a function.
>
> What do you exactly mean by this part of comment.

The set_rest_more production is used in FunctionSetResetClause and youve
removed capabilities from it.

> > And why is it possible to persistently set the search path,
> >   but not client encoding? Why is FROM CURRENT in set_rest_more?
>
> I think the main reason was some of the commands can work only in
> transaction  and as SET Persistent cannot work inside transaction block, so I had kept
> some seggregation.

Yes, I can see reasons for doing this, I just think the split isn't
correct as youve done it.

> > * Writing the temporary file to .$pid seems like a bad idea, better use
> >   one file for that, SET PERSISTENT is protected by an exclusive lock
> >   anyway.
>
>   I think we can use one temporary file, infact that was one of the ways I
> have asked in one of the previous mails.
>   However Tom and Zoltan felt this is better way to do it.

The have? I didn't read it like that. The file can only ever written by
a running postmaster and we already have code that ensures that. There's
absolutely no need for the tempfile to have a nondeterministic
name. That makes cleanup way easier as well.

>   I can think of one reason where it will be better to have .$pid, and that
> is if some one has opened the
>   file manually, then all other sessions can fail (in WINDOWS). Infact this
> is one of test Zoltan had performed.

Why on earth should somebody have the tempfile open? There's an
exclusive lock around writing the file in your code and if anybody
interferes like that with postgres' temporary data you have far bigger
problems than SET PERSISTENT erroring out.

> > * the write sequence should be:
> >   * fsync(tempfile)
> >   * fsync(directory)
> >   * rename(tempfile, configfile)
> >   * fsync(configfile)
> >   * fsync(directory)
>
> Why do we need fsync(directory) and fsync(configfile)?

So they don't vanish /get corrupted after a crash? The above is the
canonically safe way to safely write a file without an invalid file ever
being visible.

> > * write_auto_conf_file should probably escape quoted values?
>   Can you please elaborate more, I am not able to understand your
>   point?

You do something like (don't have the code right here)
if (quoted)
   appendStringInfo("= \"%s\"", value).
what happens if value contains a "?

> > * coding style should be adhered to more closesly, there are many
> >   if (pointer) which should be if (pointer != NULL),
>
> Are you pointing in function  validate_conf_option(const char *name, char  *value)
>   for below usage:
>   +                                 if (value)

For example, yes.

> > * the check that prevents persistent SETs in a transaction should
> > rather
> >   be in utility.c and use PreventTransactionChain like most of the
> >   others that need to do that (c.f. T_CreatedbStmt).
>
>   We can move the check in utility.c, but if we use PreventTransactionChain,
>   then it will be disallowed in functions as well. But the original idea was to not support in
>   transaction blocks only.
>   So I feel use of current function IsTransactionBlock() should be okay.

I don't think its even remotely safe to allow doing this from a
function. Doing it there explicitly allows doing it in a transaction.

Should we find out its safe, fine, relaxing restrictions lateron is no
problem, imposing ones after introducing a feature is.

Greetings,

Andres Freund

--
 Andres Freund	                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Amit Kapila <amit.kapila@huawei.com> — 2013-01-24T13:07:29Z

On Thursday, January 24, 2013 5:25 PM Andres Freund wrote:
> On 2013-01-24 16:45:42 +0530, Amit Kapila wrote:
> > > * The gram.y changes arround set_rest_(more|common) seem pretty
> > > confused
> > >   to me.
> >
> > >E.g. its not possible anymore to set the timezone for a function.
> >
> > What do you exactly mean by this part of comment.
> 
> The set_rest_more production is used in FunctionSetResetClause and
> youve
> removed capabilities from it.

set_rest_more has set_reset_common, so there should be no problem.

set_rest_more:        /* Generic SET syntaxes: */ 
                        set_rest_common 
                        | var_name FROM CURRENT_P 
                                { 
                                        VariableSetStmt *n =
makeNode(VariableSetStmt); 
                                        n->kind = VAR_SET_CURRENT; 
                                        n->name = $1; 
                                        $$ = n; 
                                } 

> > > And why is it possible to persistently set the search path,
> > >   but not client encoding? Why is FROM CURRENT in set_rest_more?
> >
> > I think the main reason was some of the commands can work only in
> > transaction  and as SET Persistent cannot work inside transaction
> block, so I had kept
> > some seggregation.
> 
> Yes, I can see reasons for doing this, I just think the split isn't
> correct as youve done it.
> 
> > > * Writing the temporary file to .$pid seems like a bad idea, better
> use
> > >   one file for that, SET PERSISTENT is protected by an exclusive
> lock
> > >   anyway.
> >
> >   I think we can use one temporary file, infact that was one of the
> ways I
> > have asked in one of the previous mails.
> >   However Tom and Zoltan felt this is better way to do it.

> The have? I didn't read it like that. The file can only ever written by
> a running postmaster and we already have code that ensures that.
> There's
> absolutely no need for the tempfile to have a nondeterministic
> name. That makes cleanup way easier as well.

Sure, I understand that cleaning will be easier. So IIUC you are suggesting,
we should just keep
name as postgresql.auto.conf.tmp

In general, please read the below mail where it has been suggested to use
.$pid
http://www.postgresql.org/message-id/28379.1358541135@sss.pgh.pa.us


> >   I can think of one reason where it will be better to have .$pid,
> and that
> > is if some one has opened the
> >   file manually, then all other sessions can fail (in WINDOWS).
> Infact this
> > is one of test Zoltan had performed.
> 
> Why on earth should somebody have the tempfile open? There's an
> exclusive lock around writing the file in your code and if anybody
> interferes like that with postgres' temporary data you have far bigger
> problems than SET PERSISTENT erroring out.

I am also not sure, but may be some people do for test purpose. 


> > > * the write sequence should be:
> > >   * fsync(tempfile)
> > >   * fsync(directory)
> > >   * rename(tempfile, configfile)
> > >   * fsync(configfile)
> > >   * fsync(directory)
> >
> > Why do we need fsync(directory) and fsync(configfile)?
> 
> So they don't vanish /get corrupted after a crash? The above is the
> canonically safe way to safely write a file without an invalid file
> ever
> being visible.

Do you think there will be problem if we just use as below: 
* fsync(tempfile)
* rename(tempfile, configfile)

I have seen such kind of use elsewhere also in code (writeTimeLineHistory())

> > > * write_auto_conf_file should probably escape quoted values?
> >   Can you please elaborate more, I am not able to understand your
> >   point?
> 
> You do something like (don't have the code right here)
> if (quoted)
>    appendStringInfo("= \"%s\"", value).
> what happens if value contains a "?

I think this case is missed. I shall take care to handle the case when value
contains such value.


> > > * coding style should be adhered to more closesly, there are many
> > >   if (pointer) which should be if (pointer != NULL),
> >
> > Are you pointing in function  validate_conf_option(const char *name,
> char  *value)
> >   for below usage:
> >   +                                 if (value)
> 
> For example, yes.

Okay, I shall fix these, but I have seen such code at other places
(set_config_option), that's why I have kept it like how it is currently.

> 
> > > * the check that prevents persistent SETs in a transaction should
> > > rather
> > >   be in utility.c and use PreventTransactionChain like most of the
> > >   others that need to do that (c.f. T_CreatedbStmt).
> >
> >   We can move the check in utility.c, but if we use
> PreventTransactionChain,
> >   then it will be disallowed in functions as well. But the original
> idea was to not support in
> >   transaction blocks only.
> >   So I feel use of current function IsTransactionBlock() should be
> okay.
> 
> I don't think its even remotely safe to allow doing this from a
> function. Doing it there explicitly allows doing it in a transaction.

As SET command is allowed inside a function, so I don't think without any
reason we should disallow SET PERSISTENT in function. 
The reason why it's not allowed in transaction is that we have to delete the
temporary file in transaction end or at rollback which could have made the
logic much more complex.

> Should we find out its safe, fine, relaxing restrictions lateron is no
> problem, imposing ones after introducing a feature is.

I have discussed most of these restrictions in this thread. But there can be
few which I could have missed.

With Regards,
Amit Kapila.

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Andres Freund <andres@2ndquadrant.com> — 2013-01-24T13:20:36Z

On 2013-01-24 18:37:29 +0530, Amit Kapila wrote:
> On Thursday, January 24, 2013 5:25 PM Andres Freund wrote:
> > On 2013-01-24 16:45:42 +0530, Amit Kapila wrote:
> > > > * The gram.y changes arround set_rest_(more|common) seem pretty
> > > > confused
> > > >   to me.
> > >
> > > >E.g. its not possible anymore to set the timezone for a function.
> > >
> > > What do you exactly mean by this part of comment.
> > 
> > The set_rest_more production is used in FunctionSetResetClause and
> > youve
> > removed capabilities from it.
> 
> set_rest_more has set_reset_common, so there should be no problem.
> 
> set_rest_more:        /* Generic SET syntaxes: */ 
>                         set_rest_common 
>                         | var_name FROM CURRENT_P 
>                                 { 
>                                         VariableSetStmt *n = makeNode(VariableSetStmt); 
>                                         n->kind = VAR_SET_CURRENT; 
>                                         n->name = $1; 
>                                         $$ = n; 
>                                 } 

True. I still think that the split youve made isn't right as-is.

> > > > * Writing the temporary file to .$pid seems like a bad idea, better use
> > > >   one file for that, SET PERSISTENT is protected by an exclusive lock
> > > >   anyway.
> > >
> > >   I think we can use one temporary file, infact that was one of the ways I
> > > have asked in one of the previous mails.
> > >   However Tom and Zoltan felt this is better way to do it.

> > The have? I didn't read it like that. The file can only ever written by
> > a running postmaster and we already have code that ensures that.
> > There's
> > absolutely no need for the tempfile to have a nondeterministic
> > name. That makes cleanup way easier as well.
> 
> Sure, I understand that cleaning will be easier. So IIUC you are suggesting,
> we should just keep
> name as postgresql.auto.conf.tmp
> 
> In general, please read the below mail where it has been suggested to use
> .$pid
> http://www.postgresql.org/message-id/28379.1358541135@sss.pgh.pa.us

That was in the context of your use of mkstemp() as far as I read
it. We use constantly named temp files in other locations as well.


> > Why on earth should somebody have the tempfile open? There's an
> > exclusive lock around writing the file in your code and if anybody
> > interferes like that with postgres' temporary data you have far bigger
> > problems than SET PERSISTENT erroring out.
> 
> I am also not sure, but may be some people do for test purpose. 

That seems like an completely pointless reasoning.

> > > > * the write sequence should be:
> > > >   * fsync(tempfile)
> > > >   * fsync(directory)
> > > >   * rename(tempfile, configfile)
> > > >   * fsync(configfile)
> > > >   * fsync(directory)
> > >
> > > Why do we need fsync(directory) and fsync(configfile)?
> > 
> > So they don't vanish /get corrupted after a crash? The above is the
> > canonically safe way to safely write a file without an invalid file
> > ever
> > being visible.
> 
> Do you think there will be problem if we just use as below: 
> * fsync(tempfile)
> * rename(tempfile, configfile)
> 
> I have seen such kind of use elsewhere also in code (writeTimeLineHistory())

Yea, there are some places where the code isn't entirely safe. No reason
to introduce more of those.
> > > > * the check that prevents persistent SETs in a transaction should
> > > > rather
> > > >   be in utility.c and use PreventTransactionChain like most of the
> > > >   others that need to do that (c.f. T_CreatedbStmt).
> > >
> > >   We can move the check in utility.c, but if we use
> > PreventTransactionChain,
> > >   then it will be disallowed in functions as well. But the original
> > idea was to not support in
> > >   transaction blocks only.
> > >   So I feel use of current function IsTransactionBlock() should be
> > okay.
> > 
> > I don't think its even remotely safe to allow doing this from a
> > function. Doing it there explicitly allows doing it in a transaction.
> 
> As SET command is allowed inside a function, so I don't think without any
> reason we should disallow SET PERSISTENT in function. 
> The reason why it's not allowed in transaction is that we have to delete the
> temporary file in transaction end or at rollback which could have made the
> logic much more complex.

Yea, and allowing use in functions doesn't help you with that at all:

Consider the following plpgsql function:

$$
BEGIN
    SET PERSISTENT foo = 'bar';
    RAISE ERROR 'blub';
END;
$$

Now you have a SET PERSISTENT that succeeded although the transaction
failed.

Greetings,

Andres Freund

-- 
 Andres Freund	                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Alvaro Herrera <alvherre@2ndquadrant.com> — 2013-01-24T13:37:34Z

Amit Kapila escribió:
> On Wednesday, January 23, 2013 11:51 PM Alvaro Herrera wrote:

> > Yes it is -- the /etc/postgresql/<version>/<cluster> directory (where
> > postgresql.conf resides) is owned by user postgres.
> 
> So in that case we can consider postgresql.auto.conf to be in path w.r.t
> postgresql.conf instead of data directory.

That seems sensible to me.

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Amit Kapila <amit.kapila@huawei.com> — 2013-01-24T15:40:15Z

On Thursday, January 24, 2013 6:51 PM Andres Freund wrote:
> On 2013-01-24 18:37:29 +0530, Amit Kapila wrote:
> > On Thursday, January 24, 2013 5:25 PM Andres Freund wrote:
> > > On 2013-01-24 16:45:42 +0530, Amit Kapila wrote:
> > > > > * The gram.y changes arround set_rest_(more|common) seem pretty
> > > > > confused
> > > > >   to me.
> > > >
> > > > >E.g. its not possible anymore to set the timezone for a
> function.
> > > >
> > > > What do you exactly mean by this part of comment.
> > >
> > > The set_rest_more production is used in FunctionSetResetClause and
> > > youve
> > > removed capabilities from it.
> >
> > set_rest_more has set_reset_common, so there should be no problem.
> >
> > set_rest_more:        /* Generic SET syntaxes: */
> >                         set_rest_common
> >                         | var_name FROM CURRENT_P
> >                                 {
> >                                         VariableSetStmt *n =
> makeNode(VariableSetStmt);
> >                                         n->kind = VAR_SET_CURRENT;
> >                                         n->name = $1;
> >                                         $$ = n;
> >                                 }
> 
> True. I still think that the split youve made isn't right as-is.

I am working on to provide the exact reasoning of split and if some variable
is not appropriate,
I shall do the appropriate movement.

> > > > > * Writing the temporary file to .$pid seems like a bad idea,
> better use
> > > > >   one file for that, SET PERSISTENT is protected by an
> exclusive lock
> > > > >   anyway.
> > > >
> > > >   I think we can use one temporary file, infact that was one of
> the ways I
> > > > have asked in one of the previous mails.
> > > >   However Tom and Zoltan felt this is better way to do it.
> 
> > > The have? I didn't read it like that. The file can only ever
> written by
> > > a running postmaster and we already have code that ensures that.
> > > There's
> > > absolutely no need for the tempfile to have a nondeterministic
> > > name. That makes cleanup way easier as well.
> >
> > Sure, I understand that cleaning will be easier. So IIUC you are
> suggesting,
> > we should just keep
> > name as postgresql.auto.conf.tmp
> >
> > In general, please read the below mail where it has been suggested to
> use
> > .$pid
> > http://www.postgresql.org/message-id/28379.1358541135@sss.pgh.pa.us
> 
> That was in the context of your use of mkstemp() as far as I read
> it. We use constantly named temp files in other locations as well.

Okay, as I told you that I have also proposed initially, so as now you have
pointed it specifically,
I shall do it that way unless somebody will have another strong point of not
doing it this way. 
 
> > > Why on earth should somebody have the tempfile open? There's an
> > > exclusive lock around writing the file in your code and if anybody
> > > interferes like that with postgres' temporary data you have far
> bigger
> > > problems than SET PERSISTENT erroring out.
> >
> > I am also not sure, but may be some people do for test purpose.
> 
> That seems like an completely pointless reasoning.

I have no specific reason, so I shall do it the way you have suggested.

> > > > > * the write sequence should be:
> > > > >   * fsync(tempfile)
> > > > >   * fsync(directory)
> > > > >   * rename(tempfile, configfile)
> > > > >   * fsync(configfile)
> > > > >   * fsync(directory)
> > > >
> > > > Why do we need fsync(directory) and fsync(configfile)?
> > >
> > > So they don't vanish /get corrupted after a crash? The above is the
> > > canonically safe way to safely write a file without an invalid file
> > > ever
> > > being visible.
> >
> > Do you think there will be problem if we just use as below:
> > * fsync(tempfile)
> > * rename(tempfile, configfile)
> >
> > I have seen such kind of use elsewhere also in code
> (writeTimeLineHistory())
> 
> Yea, there are some places where the code isn't entirely safe. No
> reason
> to introduce more of those.

Okay, I will check this and let you know if I have any doubts.

> > > > > * the check that prevents persistent SETs in a transaction
> should
> > > > > rather
> > > > >   be in utility.c and use PreventTransactionChain like most of
> the
> > > > >   others that need to do that (c.f. T_CreatedbStmt).
> > > >
> > > >   We can move the check in utility.c, but if we use
> > > PreventTransactionChain,
> > > >   then it will be disallowed in functions as well. But the
> original
> > > idea was to not support in
> > > >   transaction blocks only.
> > > >   So I feel use of current function IsTransactionBlock() should
> be
> > > okay.
> > >
> > > I don't think its even remotely safe to allow doing this from a
> > > function. Doing it there explicitly allows doing it in a
> transaction.
> >
> > As SET command is allowed inside a function, so I don't think without
> any
> > reason we should disallow SET PERSISTENT in function.
> > The reason why it's not allowed in transaction is that we have to
> delete the
> > temporary file in transaction end or at rollback which could have
> made the
> > logic much more complex.
> 
> Yea, and allowing use in functions doesn't help you with that at all:
> 
> Consider the following plpgsql function:
> 
> $$
> BEGIN
>     SET PERSISTENT foo = 'bar';
>     RAISE ERROR 'blub';
> END;
> $$
> 
> Now you have a SET PERSISTENT that succeeded although the transaction
> failed.

I think the core point of this command behavior is that it is not related to
transaction boundary.
So now the question is in such case if we don't allow this command inside
transaction block, why to allow inside
Function?
When user starts transaction and runs this command and then few other
commands and now if he rollback, this command cannot be rolledback.
OTOH in function there is no explicit way to issue rollback, only an error
can cause rollback which is same as when running outside block.

With Regards,
Amit Kapila.

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Tom Lane <tgl@sss.pgh.pa.us> — 2013-01-24T16:22:52Z

Andres Freund <andres@2ndquadrant.com> writes:
> On 2013-01-24 16:45:42 +0530, Amit Kapila wrote:
>>> * Writing the temporary file to .$pid seems like a bad idea, better use
>>> one file for that, SET PERSISTENT is protected by an exclusive lock
>>> anyway.

>> I think we can use one temporary file, infact that was one of the ways I
>> have asked in one of the previous mails.
>> However Tom and Zoltan felt this is better way to do it.

> The have? I didn't read it like that. The file can only ever written by
> a running postmaster and we already have code that ensures that. There's
> absolutely no need for the tempfile to have a nondeterministic
> name. That makes cleanup way easier as well.

Say again?  Surely the temp file is being written by whichever backend
is executing SET PERSISTENT, and there could be more than one.

Or, if it isn't and you really do mean that this patch is getting the
postmaster process involved in executing SET PERSISTENT, that's going to
be sufficient grounds for rejection right there.  The only way that we
have to keep the postmaster reliable is to not have it doing more than
it absolutely must.

While I have not been paying very much attention to this patch, I have
to say that every time I've looked into the thread I've gotten an
overwhelming impression of overdesign and overcomplication.  This should
not be a very large or subtle patch.  Write a setting into a temp file,
rename it into place, done.

			regards, tom lane

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Andres Freund <andres@2ndquadrant.com> — 2013-01-24T16:27:32Z

On 2013-01-24 11:22:52 -0500, Tom Lane wrote:
> Andres Freund <andres@2ndquadrant.com> writes:
> > On 2013-01-24 16:45:42 +0530, Amit Kapila wrote:
> >>> * Writing the temporary file to .$pid seems like a bad idea, better use
> >>> one file for that, SET PERSISTENT is protected by an exclusive lock
> >>> anyway.
> 
> >> I think we can use one temporary file, infact that was one of the ways I
> >> have asked in one of the previous mails.
> >> However Tom and Zoltan felt this is better way to do it.
> 
> > The have? I didn't read it like that. The file can only ever written by
> > a running postmaster and we already have code that ensures that. There's
> > absolutely no need for the tempfile to have a nondeterministic
> > name. That makes cleanup way easier as well.
> 
> Say again?  Surely the temp file is being written by whichever backend
> is executing SET PERSISTENT, and there could be more than one.

Sure, but the patch acquires SetPersistentLock exlusively beforehand
which seems fine to me.


Any opinion whether its acceptable to allow SET PERSISTENT in functions?
It seems absurd to me to allow it, but Amit seems to be of another
opinion.

Greetings,

Andres Freund

-- 
 Andres Freund	                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Tom Lane <tgl@sss.pgh.pa.us> — 2013-01-24T17:02:59Z

Andres Freund <andres@2ndquadrant.com> writes:
> On 2013-01-24 11:22:52 -0500, Tom Lane wrote:
>> Say again?  Surely the temp file is being written by whichever backend
>> is executing SET PERSISTENT, and there could be more than one.

> Sure, but the patch acquires SetPersistentLock exlusively beforehand
> which seems fine to me.

Why should we have such a lock?  Seems like that will probably introduce
as many problems as it fixes.  Deadlock risk, blockages, etc.  It is not
necessary for atomicity, since rename() would be atomic already.

> Any opinion whether its acceptable to allow SET PERSISTENT in functions?
> It seems absurd to me to allow it, but Amit seems to be of another
> opinion.

Well, it's really a definitional question I think: do you expect that
subsequent failure of the transaction should cause such a SET to roll
back?

I think it would be entirely self-consistent to define SET PERSISTENT as
a nontransactional operation.  Then the implementation would just be to
write the file immediately when the command is executed, and there's no
particular reason why it can't be allowed inside a transaction block.

If you want it to be transactional, then the point of disallowing it in
transaction blocks (or functions) would be to not have a very large
window between writing the file and committing.  But it's still possible
that the transaction would fail somewhere in there, leading to the
inconsistent outcome that the transaction reports failing but we applied
the SET anyway.  I do agree that it would be nonsensical to allow SET
PERSISTENT in functions but not transaction blocks.

Another approach is to remember the requested setting until somewhere in
the pre-commit sequence, and then try to do the file write at that time.
I'm not terribly thrilled with that approach, though, because (a) it
only narrows the window for an inconsistent outcome, it doesn't remove
it entirely; (b) there are already too many things that want to be the
"last thing done before commit"; and (c) it adds complexity and overhead
that I'd just as soon this patch not add.  But if you want S.P. to be
transactional and allowed inside functions, I think this would be the
only acceptable implementation.

			regards, tom lane

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Andres Freund <andres@anarazel.de> — 2013-01-24T17:24:42Z

On 2013-01-24 12:02:59 -0500, Tom Lane wrote:
> Andres Freund <andres@2ndquadrant.com> writes:
> > On 2013-01-24 11:22:52 -0500, Tom Lane wrote:
> >> Say again?  Surely the temp file is being written by whichever backend
> >> is executing SET PERSISTENT, and there could be more than one.
> 
> > Sure, but the patch acquires SetPersistentLock exlusively beforehand
> > which seems fine to me.
> 
> Why should we have such a lock?  Seems like that will probably introduce
> as many problems as it fixes.  Deadlock risk, blockages, etc.  It is not
> necessary for atomicity, since rename() would be atomic already.

Well, the patch acquires it as-is, so I made the judgement based on
that.
But I think that lock isn't neccesarily a bad idea. While the
replacement of the values clearly is atomic due to the rename I think
there's another confusing behaviour lurking:

Backend A: starts
Backend B: starts
Backend A: reads the config
Backend B: reads the config
Backend A: does SET PERSISTENT foobar =..;
Backend B: does SET PERSISTENT foobar =..;

Now B overwrites the config change A has made as they are all stored in
the same file.

So the only safe way to do this seems to be:

LWLockAcquire(SetPersistentLock, LW_EXCLUSIVE);
ProcessConfigFile();
validate_option();
write_rename();
LWLockRelease();

We can decide not to care about the above case but the window isn't that
small if no reload is implied and so this seems to be overly grotty.

> > Any opinion whether its acceptable to allow SET PERSISTENT in functions?
> > It seems absurd to me to allow it, but Amit seems to be of another
> > opinion.
> 
> Well, it's really a definitional question I think: do you expect that
> subsequent failure of the transaction should cause such a SET to roll
> back?
> 
> I think it would be entirely self-consistent to define SET PERSISTENT as
> a nontransactional operation.  Then the implementation would just be to
> write the file immediately when the command is executed, and there's no
> particular reason why it can't be allowed inside a transaction block.

Thats how its implemented atm except for not allowing it in transactions.

I think the reason I am weary of allowing it inside transaction is that
I think the config file needs to be reloaded before writing the new one
and it seems dangerous to me to reload the config in all the possible
situations a function can be called.

Greetings,

Andres Freund

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Tom Lane <tgl@sss.pgh.pa.us> — 2013-01-24T17:30:02Z

Andres Freund <andres@anarazel.de> writes:
> Backend A: does SET PERSISTENT foobar =..;
> Backend B: does SET PERSISTENT foobar =..;

> Now B overwrites the config change A has made as they are all stored in
> the same file.

Say what?  I thought the plan was one setting per file, so that we don't
get involved in having to parse-and-edit the file contents.  What was
all that argument about a new directory, if we're only using one file?

If we are using just one file, then I agree a lock would be needed to
synchronize updates.  But that seems to add a lot of complication
elsewhere.

			regards, tom lane

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Andres Freund <andres@2ndquadrant.com> — 2013-01-24T17:39:33Z

On 2013-01-24 12:30:02 -0500, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> > Backend A: does SET PERSISTENT foobar =..;
> > Backend B: does SET PERSISTENT foobar =..;
>
> > Now B overwrites the config change A has made as they are all stored in
> > the same file.
>
> Say what?  I thought the plan was one setting per file, so that we don't
> get involved in having to parse-and-edit the file contents.  What was
> all that argument about a new directory, if we're only using one file?

I initially thought that as well (and voted for it) but after reskimming
the thread and reading the patch that doesn't seem to be the case unless
its implemented in a way I don't understand:

+#define PG_AUTOCONF_DIR            "auto.conf.d"
+#define PG_AUTOCONF_FILENAME       "postgresql.auto.conf"

+   /* Frame auto conf name and auto conf sample temp file name */
+   snprintf(AutoConfFileName, sizeof(AutoConfFileName), "%s/%s/%s",
+                       ConfigFileDir,
+                       PG_AUTOCONF_DIR,
+                       PG_AUTOCONF_FILENAME);
+   snprintf(AutoConfTmpFileName, sizeof(AutoConfTmpFileName),"%s/%s/%s.%d",
+                       ConfigFileDir,
+                       PG_AUTOCONF_DIR,
+                       PG_AUTOCONF_FILENAME,
+                       (int) getpid());
+

I don't understand what the conf.d is all about either if only one file
is going to be used.

> If we are using just one file, then I agree a lock would be needed to
> synchronize updates.  But that seems to add a lot of complication
> elsewhere.

More people seem to have voted for the single file approach but I still
haven't understood why...

Greetings,

Andres Freund

--
 Andres Freund	                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Amit Kapila <amit.kapila@huawei.com> — 2013-01-25T03:02:17Z

On Thursday, January 24, 2013 10:33 PM Tom Lane wrote:
> Andres Freund <andres@2ndquadrant.com> writes:
> > On 2013-01-24 11:22:52 -0500, Tom Lane wrote:
> >> Say again?  Surely the temp file is being written by whichever
> backend
> >> is executing SET PERSISTENT, and there could be more than one.
> 
> > Sure, but the patch acquires SetPersistentLock exlusively beforehand
> > which seems fine to me.
> 
> Why should we have such a lock?  Seems like that will probably
> introduce
> as many problems as it fixes.  Deadlock risk, blockages, etc.  It is
> not
> necessary for atomicity, since rename() would be atomic already.
> 
> > Any opinion whether its acceptable to allow SET PERSISTENT in
> functions?
> > It seems absurd to me to allow it, but Amit seems to be of another
> > opinion.
> 
> Well, it's really a definitional question I think: do you expect that
> subsequent failure of the transaction should cause such a SET to roll
> back?
> 
> I think it would be entirely self-consistent to define SET PERSISTENT
> as
> a nontransactional operation.  Then the implementation would just be to
> write the file immediately when the command is executed, and there's no
> particular reason why it can't be allowed inside a transaction block.
> 
> If you want it to be transactional, then the point of disallowing it in
> transaction blocks (or functions) would be to not have a very large
> window between writing the file and committing.  But it's still
> possible
> that the transaction would fail somewhere in there, leading to the
> inconsistent outcome that the transaction reports failing but we
> applied
> the SET anyway.  I do agree that it would be nonsensical to allow SET
> PERSISTENT in functions but not transaction blocks.

As we have agreed on before that this command has to be non-transactional,
so
lets stick on that and about allowing it in functions and not in transaction
blocks,
I had some reasoning behind it as below, but as it is not making much sense 
we can disallow it for both transaction blocks as well as functions:

Reasoning for disallowing in transaction blocks and allowing in functions:
----------------------------------------------------------------------------
--
" I think the core point of this command behavior is that it is not related
to transaction boundary.
So now the question is in such case if we don't allow this command inside
transaction block, why to allow inside Function?
When user starts transaction and runs this command and then few other
commands and now if he rollback, this command cannot be rolledback.
OTOH in function there is no explicit way to issue rollback, only an error
can cause rollback which is same as when running outside block."

> Say what?  I thought the plan was one setting per file, so that we don't
get involved > in having to parse-and-edit the file contents.  
> What was all that argument about a new directory, if we're only using one
file?

A new directory (named as auto.conf.d, initially in the patch named as
config_dir) is for having automatically generated configuration files.
Also it was suggested previously that it is better to have this file in
directory instead of standalone file.

> If we are using just one file, then I agree a lock would be needed to
synchronize 
> updates.  But that seems to add a lot of complication elsewhere.

There were 2 ways to protect the file write, 
a. one is by having lock SetPersistentLock
b. another is by opening file in X mode and if open fails wait and retry for
sometime
   and then give error.
As suggested in this thread I have implemented as Approach-1.
What is the complication it can add, can we resolve that?

With Regards,
Amit Kapila.

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Amit Kapila <amit.kapila@huawei.com> — 2013-01-25T10:36:00Z

On Thursday, January 24, 2013 6:51 PM Andres Freund wrote:
> On 2013-01-24 18:37:29 +0530, Amit Kapila wrote:
> > On Thursday, January 24, 2013 5:25 PM Andres Freund wrote:
> > > On 2013-01-24 16:45:42 +0530, Amit Kapila wrote:
> > > > > * The gram.y changes arround set_rest_(more|common) seem pretty
> > > > > confused
> > > > >   to me.
> > > >
> > > > >E.g. its not possible anymore to set the timezone for a
> function.
> > > >
> > > > What do you exactly mean by this part of comment.
> > >
> > > The set_rest_more production is used in FunctionSetResetClause and
> > > youve
> > > removed capabilities from it.
> >
> > set_rest_more has set_reset_common, so there should be no problem.
> >
> > set_rest_more:        /* Generic SET syntaxes: */
> >                         set_rest_common
> >                         | var_name FROM CURRENT_P
> >                                 {
> >                                         VariableSetStmt *n =
> makeNode(VariableSetStmt);
> >                                         n->kind = VAR_SET_CURRENT;
> >                                         n->name = $1;
> >                                         $$ = n;
> >                                 }
> 
> True. I still think that the split youve made isn't right as-is.

We can move FROM CURRENT and client_encoding to common part. 
Other syntax (ROLE, SESSION AUTHORIZATION, TRANSACTION SNAPSHOT) doesn't
exist in postgresql.conf and requires a transaction to set and during reload
we don't have transaction, so we should not allow them. 

With Regards,
Amit Kapila.

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Amit Kapila <amit.kapila@huawei.com> — 2013-01-26T05:24:39Z


Thanks to both (Andres and Fujii Masao) of you for review.
Please find the patch attached with this mail which contains fixes for the
comments raised by you. 
Kindly confirm if the fixes are okay or if I have missed any point raised by
you.

On Thursday, January 24, 2013 1:56 AM Andres Freund wrote:
> On 2013-01-22 12:32:07 +0000, Amit kapila wrote:
> > This closes all comments raised till now for this patch.
> > Kindly let me know if you feel something is missing?
> 
> I am coming late to this patch, so bear with me if I repeat somethign
> said elsewhere.
> 
> Review comments of cursory pass through the patch:
> * most comments are hard to understand. I know the problem of that
>   being hard for a non-native speaker by heart, but I think another
> pass
>   over them would be good thing.
  I have changed the comments in most parts of code.

> * The gram.y changes arround set_rest_(more|common) seem pretty
> confused
>   to me. E.g. its not possible anymore to set the timezone for a
>   function. And why is it possible to persistently set the search path,
>   but not client encoding? Why is FROM CURRENT in set_rest_more?

client encoding and FROM CURRENT syntax will be allowed in SET PERSISTENT. 
 Others cannot be moved as they need transaction to set them and they are 
 even not present in postgresql.conf file

> * set_config_file should elog(ERROR), not return on an unhandled
>   setstmt->kind

Fixed.

> * why are you creating AutoConfFileName if its not stat'able? It seems
>   better to simply skip parsing the old file in that case

Fixed.

> * Writing the temporary file to .$pid seems like a bad idea, better use
>   one file for that, SET PERSISTENT is protected by an exclusive lock
>   anyway.

now only one file (postgresql.auto.conf.temp) will be created

> * the write sequence should be:
>   * fsync(tempfile)
>   * fsync(directory)
>   * rename(tempfile, configfile)
>   * fsync(configfile)
>   * fsync(directory)
to do fsync of directory we need to open the directory and then do the
fsync. 
  rename is an atomic operation, so it will always point to either new or
old file. 
  I am just not sure whether to modify the code to handle this scenario. 
  So for now I have added the comment at that place.

> * write_auto_conf_file should probably escape quoted values?

  Fixed. Changed write_auto_conf_file() to handle it.

> * coding style should be adhered to more closesly, there are many
>   if (pointer) which should be if (pointer != NULL), single-line blocks
>   enclosed in curlies which shouldn't, etc.

   Fixed.

> * replace_auto_config_file and surrounding functions need more comments
>   in the header

Added more comments.

> * the check that prevents persistent SETs in a transaction should
> rather
>   be in utility.c and use PreventTransactionChain like most of the
>   others that need to do that (c.f. T_CreatedbStmt).

Fixed. Now this command will not work for transaction blocks or Functions.

Fujii Masao Comments
---------------------
> When I removed postgresql.auto.conf and restarted the server, I got 
> the following warning message. This is not correct because I didn't 
> remove "auto.conf.d" from postgresql.conf. What I removed is only 
> postgresql.auto.conf.
>
> WARNING:  Default "auto.conf.d" is not present in postgresql.conf.
> Configuration parameters changed with SET PERSISTENT command will not 
> be effective.
>

Warning message is changed as per discussion in another mail.

> Is it safe to write something in the directory other than data 
> directory via SQL?

As discussed, it is okay to write in config directory, so I have not changed
anything for this.

> The description of RESET PERSISTENT is in the document, but it seems 
> not to be implemented.

RESET PERSISTENT description is removed from patch.

> storage.sgml needs to be updated.
> Doesn't auto.conf.d need to be explained in config.sgml?

Updated both storage.sgml and config.sgml

> This warning message implies that the line "include_dir 'auto.conf.d'"
> must not be removed from postgresql.conf? If so, we should warn that in
both document and postgresql.conf.sample?

I have modified the WARNING in postgresql.conf.sample, I have thought of
modifying runtime.sgml but not sure if that is the right place, can you
please suggest which document would be better to have this warning?

With Regards,
Amit Kapila.

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Robert Haas <robertmhaas@gmail.com> — 2013-01-26T12:46:50Z

On Thu, Jan 24, 2013 at 12:39 PM, Andres Freund <andres@2ndquadrant.com> wrote:
> More people seem to have voted for the single file approach but I still
> haven't understood why...

Me neither.  Having an include directory seems good, but I can't think
why we'd want to clutter it up with a bajillion automatically
generated files.  One .auto file that gets overwritten at need seems
way nicer.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Kevin Grittner <kgrittn@ymail.com> — 2013-01-26T15:52:51Z

Robert Haas <robertmhaas@gmail.com> wrote:
> Andres Freund <andres@2ndquadrant.com> wrote:
>> More people seem to have voted for the single file approach but I still
>> haven't understood why...
>
> Me neither.  Having an include directory seems good, but I can't think
> why we'd want to clutter it up with a bajillion automatically
> generated files.  One .auto file that gets overwritten at need seems
> way nicer.

The single-value per file approach punts the concurrency issues and
ordering issues to the OS, which might significantly simplify the
patch.  I'm not sure that the niceness of the all-in-one-file
approach justifies the extra code.  If you ever want to view them
all in one file as of some moment in time, there is always:

  cat * | less

-Kevin

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Andres Freund <andres@2ndquadrant.com> — 2013-01-26T16:27:28Z

On 2013-01-26 07:46:50 -0500, Robert Haas wrote:
> On Thu, Jan 24, 2013 at 12:39 PM, Andres Freund <andres@2ndquadrant.com> wrote:
> > More people seem to have voted for the single file approach but I still
> > haven't understood why...
> 
> Me neither.  Having an include directory seems good, but I can't think
> why we'd want to clutter it up with a bajillion automatically
> generated files.  One .auto file that gets overwritten at need seems
> way nicer.

I unfortunately think you misunderstood what I meant. I don't really see
that many arguments for putting all the variables in a single file. That
is I think the one-value-per-file is preferrable.

The primary reason is that the multiple values per file is just
considerably harder to implement without POLA. In order to not loose
values set in this or other backends you basically need to do something
like:

1) exlusive lock
2) reload config file (to update in memory structures)
3) check new variable
4) write config file (needs to be done atomically)
5) optionally reload config file
6) reload lock

Where as you can get acceptable behaviour in the single-value-per-file
approach by:
1) check new config variable
2) write new single-variable file
3) optionally reload config file

Not requiring a global lock while reading the configuration seems to
make stuff noticeably simpler.

The other advantages for a single value approach I know of are:
* way easier to integrate into other systems, no need to parse the whole
  file for them
* easier to get yourself out of problems if you screwed up and set a bad
  value (shared_buffers=16TB instead of GB ;))

Whereas the only real argument I can see for the all-in-one-file
approach I can see is that it will perform better.

Greetings,

Andres Freund

-- 
 Andres Freund	                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Tom Lane <tgl@sss.pgh.pa.us> — 2013-01-26T17:01:59Z

Robert Haas <robertmhaas@gmail.com> writes:
> On Thu, Jan 24, 2013 at 12:39 PM, Andres Freund <andres@2ndquadrant.com> wrote:
>> More people seem to have voted for the single file approach but I still
>> haven't understood why...

> Me neither.  Having an include directory seems good, but I can't think
> why we'd want to clutter it up with a bajillion automatically
> generated files.  One .auto file that gets overwritten at need seems
> way nicer.

IMO an include directory containing just one file is silly.  If we're
going with the single-file approach, let's lose the directory altogether
and just store the file at $PGDATA/postgresql.conf.auto.

			regards, tom lane

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Craig Ringer <craig@2ndquadrant.com> — 2013-01-26T23:47:32Z

On 01/27/2013 01:01 AM, Tom Lane wrote:
> Robert Haas <robertmhaas@gmail.com> writes:
>> On Thu, Jan 24, 2013 at 12:39 PM, Andres Freund <andres@2ndquadrant.com> wrote:
>>> More people seem to have voted for the single file approach but I still
>>> haven't understood why...
>> Me neither.  Having an include directory seems good, but I can't think
>> why we'd want to clutter it up with a bajillion automatically
>> generated files.  One .auto file that gets overwritten at need seems
>> way nicer.
> IMO an include directory containing just one file is silly.  If we're
> going with the single-file approach, let's lose the directory altogether
> and just store the file at $PGDATA/postgresql.conf.auto.
Wasn't part of the reason for having the config dir to make package
managers' lives easier and make it easier to script updates to
postgresql.conf? For the use of things like pg_wrapper?

I think the config dir has value even if a single .auto file is used, so
that packages can drop their own config snippets into it. For example,
if I installed a packaged extension that had its own postgresql.conf
changes or new GUCs, I'd want it to be able to drop that into the
configdir, not have to script changes to my postgresql.conf.

-- 
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Robert Haas <robertmhaas@gmail.com> — 2013-01-27T04:37:57Z

On Sat, Jan 26, 2013 at 6:47 PM, Craig Ringer <craig@2ndquadrant.com> wrote:
> On 01/27/2013 01:01 AM, Tom Lane wrote:
>> Robert Haas <robertmhaas@gmail.com> writes:
>>> On Thu, Jan 24, 2013 at 12:39 PM, Andres Freund <andres@2ndquadrant.com> wrote:
>>>> More people seem to have voted for the single file approach but I still
>>>> haven't understood why...
>>> Me neither.  Having an include directory seems good, but I can't think
>>> why we'd want to clutter it up with a bajillion automatically
>>> generated files.  One .auto file that gets overwritten at need seems
>>> way nicer.
>> IMO an include directory containing just one file is silly.  If we're
>> going with the single-file approach, let's lose the directory altogether
>> and just store the file at $PGDATA/postgresql.conf.auto.
> Wasn't part of the reason for having the config dir to make package
> managers' lives easier and make it easier to script updates to
> postgresql.conf? For the use of things like pg_wrapper?
>
> I think the config dir has value even if a single .auto file is used, so
> that packages can drop their own config snippets into it. For example,
> if I installed a packaged extension that had its own postgresql.conf
> changes or new GUCs, I'd want it to be able to drop that into the
> configdir, not have to script changes to my postgresql.conf.

That was my understanding.  But I just work here.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Amit Kapila <amit.kapila@huawei.com> — 2013-01-27T05:58:04Z

On Sunday, January 27, 2013 10:07 AM Robert Haas wrote:
On Sat, Jan 26, 2013 at 6:47 PM, Craig Ringer <craig@2ndquadrant.com> wrote:
> On 01/27/2013 01:01 AM, Tom Lane wrote:
>> Robert Haas <robertmhaas@gmail.com> writes:
>>> On Thu, Jan 24, 2013 at 12:39 PM, Andres Freund <andres@2ndquadrant.com> wrote:
>>>>> More people seem to have voted for the single file approach but I still
>>>>> haven't understood why...
>>>> Me neither.  Having an include directory seems good, but I can't think
>>>> why we'd want to clutter it up with a bajillion automatically
>>>> generated files.  One .auto file that gets overwritten at need seems
>>>> way nicer.
>>> IMO an include directory containing just one file is silly.  If we're
>>> going with the single-file approach, let's lose the directory altogether
>>> and just store the file at $PGDATA/postgresql.conf.auto.
>> Wasn't part of the reason for having the config dir to make package
>> managers' lives easier and make it easier to script updates to
>> postgresql.conf? For the use of things like pg_wrapper?
>
>> I think the config dir has value even if a single .auto file is used, so
>> that packages can drop their own config snippets into it. For example,
>> if I installed a packaged extension that had its own postgresql.conf
>> changes or new GUCs, I'd want it to be able to drop that into the
>> configdir, not have to script changes to my postgresql.conf.

> That was my understanding.  But I just work here.

Current implementation of patch is to have directory and single file.
However if user puts other files, it gives LOG: unexpected file found in automatic configuration directory .
As now Craig has explained the usecase for having other files and previously Fujii Masao has also reported similar
usecase, I think it is better to remove the LOG message from code.
Currently the directory is named as auto.conf.d, so if we allow users to place other files, is this name okay
or shall we change it to something like config_dir?

With Regards,
Amit Kapila.

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Boszormenyi Zoltan <zb@cybertec.at> — 2013-02-11T14:21:13Z

2013-01-24 18:02 keltezéssel, Tom Lane írta:
> Andres Freund <andres@2ndquadrant.com> writes:
>> On 2013-01-24 11:22:52 -0500, Tom Lane wrote:
>>> Say again?  Surely the temp file is being written by whichever backend
>>> is executing SET PERSISTENT, and there could be more than one.
>> Sure, but the patch acquires SetPersistentLock exlusively beforehand
>> which seems fine to me.
> Why should we have such a lock?  Seems like that will probably introduce
> as many problems as it fixes.  Deadlock risk, blockages, etc.  It is not
> necessary for atomicity, since rename() would be atomic already.

There is a problem when running SET PERSISTENT for different GUCs
in parallel. All happen to read the same original file, and only one
setting ends up in the result if you rely only on the rename() being atomic.
The LWLock provides the serialization for that problem.

>
>> Any opinion whether its acceptable to allow SET PERSISTENT in functions?
>> It seems absurd to me to allow it, but Amit seems to be of another
>> opinion.
> Well, it's really a definitional question I think: do you expect that
> subsequent failure of the transaction should cause such a SET to roll
> back?
>
> I think it would be entirely self-consistent to define SET PERSISTENT as
> a nontransactional operation.  Then the implementation would just be to
> write the file immediately when the command is executed, and there's no
> particular reason why it can't be allowed inside a transaction block.
>
> If you want it to be transactional, then the point of disallowing it in
> transaction blocks (or functions) would be to not have a very large
> window between writing the file and committing.  But it's still possible
> that the transaction would fail somewhere in there, leading to the
> inconsistent outcome that the transaction reports failing but we applied
> the SET anyway.  I do agree that it would be nonsensical to allow SET
> PERSISTENT in functions but not transaction blocks.
>
> Another approach is to remember the requested setting until somewhere in
> the pre-commit sequence, and then try to do the file write at that time.
> I'm not terribly thrilled with that approach, though, because (a) it
> only narrows the window for an inconsistent outcome, it doesn't remove
> it entirely; (b) there are already too many things that want to be the
> "last thing done before commit"; and (c) it adds complexity and overhead
> that I'd just as soon this patch not add.  But if you want S.P. to be
> transactional and allowed inside functions, I think this would be the
> only acceptable implementation.
>
> 			regards, tom lane
>
>


-- 
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de
      http://www.postgresql.at/

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Andres Freund <andres@2ndquadrant.com> — 2013-02-11T14:25:17Z

On 2013-02-11 15:21:13 +0100, Boszormenyi Zoltan wrote:
> 2013-01-24 18:02 keltezéssel, Tom Lane írta:
> >Andres Freund <andres@2ndquadrant.com> writes:
> >>On 2013-01-24 11:22:52 -0500, Tom Lane wrote:
> >>>Say again?  Surely the temp file is being written by whichever backend
> >>>is executing SET PERSISTENT, and there could be more than one.
> >>Sure, but the patch acquires SetPersistentLock exlusively beforehand
> >>which seems fine to me.
> >Why should we have such a lock?  Seems like that will probably introduce
> >as many problems as it fixes.  Deadlock risk, blockages, etc.  It is not
> >necessary for atomicity, since rename() would be atomic already.
> 
> There is a problem when running SET PERSISTENT for different GUCs
> in parallel. All happen to read the same original file, and only one
> setting ends up in the result if you rely only on the rename() being atomic.
> The LWLock provides the serialization for that problem.

Tom was voting for one-setting-per-file, in that case the problem
doesn't exist.

Greetings,

Andres Freund

-- 
 Andres Freund	                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Boszormenyi Zoltan <zb@cybertec.at> — 2013-02-11T14:27:23Z

2013-01-26 13:46 keltezéssel, Robert Haas írta:
> On Thu, Jan 24, 2013 at 12:39 PM, Andres Freund <andres@2ndquadrant.com> wrote:
>> More people seem to have voted for the single file approach but I still
>> haven't understood why...
> Me neither.  Having an include directory seems good, but I can't think
> why we'd want to clutter it up with a bajillion automatically
> generated files.  One .auto file that gets overwritten at need seems
> way nicer.
>

Sarcasm at work. Nice. :-)

-- 
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de
      http://www.postgresql.at/

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Boszormenyi Zoltan <zb@cybertec.at> — 2013-02-11T14:33:01Z

2013-02-11 15:25 keltezéssel, Andres Freund írta:
> On 2013-02-11 15:21:13 +0100, Boszormenyi Zoltan wrote:
>> 2013-01-24 18:02 keltezéssel, Tom Lane írta:
>>> Andres Freund <andres@2ndquadrant.com> writes:
>>>> On 2013-01-24 11:22:52 -0500, Tom Lane wrote:
>>>>> Say again?  Surely the temp file is being written by whichever backend
>>>>> is executing SET PERSISTENT, and there could be more than one.
>>>> Sure, but the patch acquires SetPersistentLock exlusively beforehand
>>>> which seems fine to me.
>>> Why should we have such a lock?  Seems like that will probably introduce
>>> as many problems as it fixes.  Deadlock risk, blockages, etc.  It is not
>>> necessary for atomicity, since rename() would be atomic already.
>> There is a problem when running SET PERSISTENT for different GUCs
>> in parallel. All happen to read the same original file, and only one
>> setting ends up in the result if you rely only on the rename() being atomic.
>> The LWLock provides the serialization for that problem.
> Tom was voting for one-setting-per-file, in that case the problem
> doesn't exist.

I voted for the one-file approach and was arguing from the POV
of the current implementation.

-- 
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de
      http://www.postgresql.at/

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Josh Berkus <josh@agliodbs.com> — 2013-02-11T19:17:16Z

On 02/11/2013 06:33 AM, Boszormenyi Zoltan wrote:
> 2013-02-11 15:25 keltezéssel, Andres Freund írta:
>> On 2013-02-11 15:21:13 +0100, Boszormenyi Zoltan wrote:
>>> 2013-01-24 18:02 keltezéssel, Tom Lane írta:
>>>> Andres Freund <andres@2ndquadrant.com> writes:
>>>>> On 2013-01-24 11:22:52 -0500, Tom Lane wrote:
>>>>>> Say again?  Surely the temp file is being written by whichever
>>>>>> backend
>>>>>> is executing SET PERSISTENT, and there could be more than one.
>>>>> Sure, but the patch acquires SetPersistentLock exlusively beforehand
>>>>> which seems fine to me.
>>>> Why should we have such a lock?  Seems like that will probably
>>>> introduce
>>>> as many problems as it fixes.  Deadlock risk, blockages, etc.  It is
>>>> not
>>>> necessary for atomicity, since rename() would be atomic already.
>>> There is a problem when running SET PERSISTENT for different GUCs
>>> in parallel. All happen to read the same original file, and only one
>>> setting ends up in the result if you rely only on the rename() being
>>> atomic.
>>> The LWLock provides the serialization for that problem.
>> Tom was voting for one-setting-per-file, in that case the problem
>> doesn't exist.
> 
> I voted for the one-file approach and was arguing from the POV
> of the current implementation.

I thought we discussed this ad naseum, and decided to go with the
one-single-file approach for the first round, since we already had an
implementation for that. I still think that's the right approach to take
with this patch; if it doesn't work out, we can go do
one-file-per-setting in 9.4.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Andres Freund <andres@2ndquadrant.com> — 2013-02-11T19:24:28Z

On 2013-02-11 11:17:16 -0800, Josh Berkus wrote:
> On 02/11/2013 06:33 AM, Boszormenyi Zoltan wrote:
> > 2013-02-11 15:25 keltezéssel, Andres Freund írta:
> >> On 2013-02-11 15:21:13 +0100, Boszormenyi Zoltan wrote:
> >>> 2013-01-24 18:02 keltezéssel, Tom Lane írta:
> >>>> Andres Freund <andres@2ndquadrant.com> writes:
> >>>>> On 2013-01-24 11:22:52 -0500, Tom Lane wrote:
> >>>>>> Say again?  Surely the temp file is being written by whichever
> >>>>>> backend
> >>>>>> is executing SET PERSISTENT, and there could be more than one.
> >>>>> Sure, but the patch acquires SetPersistentLock exlusively beforehand
> >>>>> which seems fine to me.
> >>>> Why should we have such a lock?  Seems like that will probably
> >>>> introduce
> >>>> as many problems as it fixes.  Deadlock risk, blockages, etc.  It is
> >>>> not
> >>>> necessary for atomicity, since rename() would be atomic already.
> >>> There is a problem when running SET PERSISTENT for different GUCs
> >>> in parallel. All happen to read the same original file, and only one
> >>> setting ends up in the result if you rely only on the rename() being
> >>> atomic.
> >>> The LWLock provides the serialization for that problem.
> >> Tom was voting for one-setting-per-file, in that case the problem
> >> doesn't exist.
> > 
> > I voted for the one-file approach and was arguing from the POV
> > of the current implementation.
> 
> I thought we discussed this ad naseum, and decided to go with the
> one-single-file approach for the first round, since we already had an
> implementation for that. I still think that's the right approach to take
> with this patch; if it doesn't work out, we can go do
> one-file-per-setting in 9.4.

Well, several people (at least Tom, I, and I think Zoltan as well) think
that the one-file approach is considerably more complex.

Check
http://www.postgresql.org/message-id/20130126162728.GA5482@awork2.anarazel.de
and related messages for some of the problems. Most of which are
unhandled in the current patch, i.e. currently you *will* loose changes
made in concurrent sessions.

Greetings,

Andres Freund

-- 
 Andres Freund	                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Craig Ringer <craig@2ndquadrant.com> — 2013-02-12T01:53:33Z

On 02/12/2013 03:24 AM, Andres Freund wrote:
> Well, several people (at least Tom, I, and I think Zoltan as well)
> think that the one-file approach is considerably more complex. Check
> http://www.postgresql.org/message-id/20130126162728.GA5482@awork2.anarazel.de
> and related messages for some of the problems. Most of which are
> unhandled in the current patch, i.e. currently you *will* loose
> changes made in concurrent sessions.
We need to read all files in the directory anyway to make the configdir
approach useful for system package managers that want to create config
files for extensions, akin to how /etc/yum.repos.ds/ ,
/etc/apt/sources.list.d/ , /etc/apache2/conf.d/ and so on work.

OTOH, it'll be easier for a sysadmin to understand what's going on if
Pg's auto-written config is a single file, so any system-installed
settings aren't lost in a giant clutter of Pg-created files.

It seems that SQL-level configuration and the config dir feature are
actually more separate than it might've appeared.

-- 
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Amit Kapila <amit.kapila@huawei.com> — 2013-02-12T03:54:19Z

On Tuesday, February 12, 2013 12:54 AM Andres Freund wrote:
> On 2013-02-11 11:17:16 -0800, Josh Berkus wrote:
> > On 02/11/2013 06:33 AM, Boszormenyi Zoltan wrote:
> > > 2013-02-11 15:25 keltezéssel, Andres Freund írta:
> > >> On 2013-02-11 15:21:13 +0100, Boszormenyi Zoltan wrote:
> > >>> 2013-01-24 18:02 keltezéssel, Tom Lane írta:
> > >>>> Andres Freund <andres@2ndquadrant.com> writes:
> > >>>>> On 2013-01-24 11:22:52 -0500, Tom Lane wrote:
> > >>>>>> Say again?  Surely the temp file is being written by whichever
> > >>>>>> backend
> > >>>>>> is executing SET PERSISTENT, and there could be more than one.
> > >>>>> Sure, but the patch acquires SetPersistentLock exlusively
> beforehand
> > >>>>> which seems fine to me.
> > >>>> Why should we have such a lock?  Seems like that will probably
> > >>>> introduce
> > >>>> as many problems as it fixes.  Deadlock risk, blockages, etc.
> It is
> > >>>> not
> > >>>> necessary for atomicity, since rename() would be atomic already.
> > >>> There is a problem when running SET PERSISTENT for different GUCs
> > >>> in parallel. All happen to read the same original file, and only
> one
> > >>> setting ends up in the result if you rely only on the rename()
> being
> > >>> atomic.
> > >>> The LWLock provides the serialization for that problem.
> > >> Tom was voting for one-setting-per-file, in that case the problem
> > >> doesn't exist.
> > >
> > > I voted for the one-file approach and was arguing from the POV
> > > of the current implementation.
> >
> > I thought we discussed this ad naseum, and decided to go with the
> > one-single-file approach for the first round, since we already had an
> > implementation for that. I still think that's the right approach to
> take
> > with this patch; if it doesn't work out, we can go do
> > one-file-per-setting in 9.4.
> 
> Well, several people (at least Tom, I, and I think Zoltan as well)
> think
> that the one-file approach is considerably more complex.

Zoltan has reviewed this patch very thoroughly, I have never seen a comment
from him that the current
Patch (one-file-all-settings) approach is not as good as having
one-file-per-setting approach.
Zoltan, please correct me, If I am wrong.

What I could understand from mails is Tom has initially suggested to have
one-file-per-setting but for the current patch  
(one-file-all-settings) he was telling that if we wanted to go for single
file approach, then there
is no need for separate directory, but for that CRAIG has given a scenario
where separate directory is useful.

Also Robert has suggested from the beginning that (one-file-all-settings) is
better approach.

It took months of discussion to reach the consensus of this level, if we
again want to change approach,
then I think it will be tough to touch this feature again.

I think it would be better if we could try to see if there are any problems
in existing patch which cannot be handled because of it's current design,
then it will make more sense to conclude on changing approach.

> Check
> http://www.postgresql.org/message-
> id/20130126162728.GA5482@awork2.anarazel.de
> and related messages for some of the problems. Most of which are
> unhandled in the current patch, i.e. currently you *will* loose changes
> made in concurrent sessions.

How will somebody loose changes in concurrent sessions?
Each session currently waits with LWLock if some other session is operating
on file. Also after having lock they don't acquire any other lock, so there
should be no chances of any other problem.

With Regards,
Amit Kapila.

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Boszormenyi Zoltan <zb@cybertec.at> — 2013-02-12T05:53:51Z

2013-02-12 04:54 keltezéssel, Amit Kapila írta:
> On Tuesday, February 12, 2013 12:54 AM Andres Freund wrote:
>> On 2013-02-11 11:17:16 -0800, Josh Berkus wrote:
>>> On 02/11/2013 06:33 AM, Boszormenyi Zoltan wrote:
>>>> 2013-02-11 15:25 keltezéssel, Andres Freund írta:
>>>>> On 2013-02-11 15:21:13 +0100, Boszormenyi Zoltan wrote:
>>>>>> 2013-01-24 18:02 keltezéssel, Tom Lane írta:
>>>>>>> Andres Freund <andres@2ndquadrant.com> writes:
>>>>>>>> On 2013-01-24 11:22:52 -0500, Tom Lane wrote:
>>>>>>>>> Say again?  Surely the temp file is being written by whichever
>>>>>>>>> backend
>>>>>>>>> is executing SET PERSISTENT, and there could be more than one.
>>>>>>>> Sure, but the patch acquires SetPersistentLock exlusively
>> beforehand
>>>>>>>> which seems fine to me.
>>>>>>> Why should we have such a lock?  Seems like that will probably
>>>>>>> introduce
>>>>>>> as many problems as it fixes.  Deadlock risk, blockages, etc.
>> It is
>>>>>>> not
>>>>>>> necessary for atomicity, since rename() would be atomic already.
>>>>>> There is a problem when running SET PERSISTENT for different GUCs
>>>>>> in parallel. All happen to read the same original file, and only
>> one
>>>>>> setting ends up in the result if you rely only on the rename()
>> being
>>>>>> atomic.
>>>>>> The LWLock provides the serialization for that problem.
>>>>> Tom was voting for one-setting-per-file, in that case the problem
>>>>> doesn't exist.
>>>> I voted for the one-file approach and was arguing from the POV
>>>> of the current implementation.
>>> I thought we discussed this ad naseum, and decided to go with the
>>> one-single-file approach for the first round, since we already had an
>>> implementation for that. I still think that's the right approach to
>> take
>>> with this patch; if it doesn't work out, we can go do
>>> one-file-per-setting in 9.4.
>> Well, several people (at least Tom, I, and I think Zoltan as well)
>> think
>> that the one-file approach is considerably more complex.
> Zoltan has reviewed this patch very thoroughly, I have never seen a comment
> from him that the current
> Patch (one-file-all-settings) approach is not as good as having
> one-file-per-setting approach.
> Zoltan, please correct me, If I am wrong.

I cannot recall arguing for the one-file-per-GUC way.
But I haven't re-read my mails in this thread, either.

> What I could understand from mails is Tom has initially suggested to have
> one-file-per-setting but for the current patch
> (one-file-all-settings) he was telling that if we wanted to go for single
> file approach, then there
> is no need for separate directory, but for that CRAIG has given a scenario
> where separate directory is useful.
>
> Also Robert has suggested from the beginning that (one-file-all-settings) is
> better approach.
>
>
> It took months of discussion to reach the consensus of this level, if we
> again want to change approach,
> then I think it will be tough to touch this feature again.
>
> I think it would be better if we could try to see if there are any problems
> in existing patch which cannot be handled because of it's current design,
> then it will make more sense to conclude on changing approach.
>
>> Check
>> http://www.postgresql.org/message-
>> id/20130126162728.GA5482@awork2.anarazel.de
>> and related messages for some of the problems. Most of which are
>> unhandled in the current patch, i.e. currently you *will* loose changes
>> made in concurrent sessions.

This mail lists this order for the single file approach:

> 1) exlusive lock
> 2) reload config file (to update in memory structures)
> 3) check new variable
> 4) write config file (needs to be done atomically)
> 5) optionally reload config file
> 6) reload lock

The patch does it this way:
1. check new variable. No problem with that, validation for proper GUC name,
    type of the value, etc. can be done outside the lock.
2. grab lock
3. parse the config file
4. write the new config file
5. release lock

Reloading the config file is intentionally not done, it's even documented.
You can do SELECT pg_reload_conf() after SET PERSISTENT if you need it.

> How will somebody loose changes in concurrent sessions?
> Each session currently waits with LWLock if some other session is operating
> on file. Also after having lock they don't acquire any other lock, so there
> should be no chances of any other problem.

Specifically, LWLockAcquire() is called first, then ParseConfigFp()
in a PG_TRY() block, so reading the original postgresql.auto.conf
is serialized. No chance to lose changes done in parallel.

Best regards,
Zoltán Böszörményi

-- 
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de
      http://www.postgresql.at/

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Amit Kapila <amit.kapila@huawei.com> — 2013-02-12T09:27:51Z

On Tuesday, February 12, 2013 11:24 AM Boszormenyi Zoltan wrote:
> 2013-02-12 04:54 keltezéssel, Amit Kapila írta:
> > On Tuesday, February 12, 2013 12:54 AM Andres Freund wrote:
> >> On 2013-02-11 11:17:16 -0800, Josh Berkus wrote:
> >>> On 02/11/2013 06:33 AM, Boszormenyi Zoltan wrote:
> >>>> 2013-02-11 15:25 keltezéssel, Andres Freund írta:
> >>>>> On 2013-02-11 15:21:13 +0100, Boszormenyi Zoltan wrote:
> >>>>>> 2013-01-24 18:02 keltezéssel, Tom Lane írta:
> >>>>>>> Andres Freund <andres@2ndquadrant.com> writes:
> >>>>>>>> On 2013-01-24 11:22:52 -0500, Tom Lane wrote:
> >>>>>>>>> Say again?  Surely the temp file is being written by
> whichever
> >>>>>>>>> backend
> >>>>>>>>> is executing SET PERSISTENT, and there could be more than
> one.
> >>>>>>>> Sure, but the patch acquires SetPersistentLock exlusively
> >> beforehand
> >>>>>>>> which seems fine to me.
> >>>>>>> Why should we have such a lock?  Seems like that will probably
> >>>>>>> introduce
> >>>>>>> as many problems as it fixes.  Deadlock risk, blockages, etc.
> >> It is
> >>>>>>> not
> >>>>>>> necessary for atomicity, since rename() would be atomic
> already.
> >>>>>> There is a problem when running SET PERSISTENT for different
> GUCs
> >>>>>> in parallel. All happen to read the same original file, and only
> >> one
> >>>>>> setting ends up in the result if you rely only on the rename()
> >> being
> >>>>>> atomic.
> >>>>>> The LWLock provides the serialization for that problem.
> >>>>> Tom was voting for one-setting-per-file, in that case the problem
> >>>>> doesn't exist.
> >>>> I voted for the one-file approach and was arguing from the POV
> >>>> of the current implementation.
> >>> I thought we discussed this ad naseum, and decided to go with the
> >>> one-single-file approach for the first round, since we already had
> an
> >>> implementation for that. I still think that's the right approach to
> >> take
> >>> with this patch; if it doesn't work out, we can go do
> >>> one-file-per-setting in 9.4.
> >> Well, several people (at least Tom, I, and I think Zoltan as well)
> >> think
> >> that the one-file approach is considerably more complex.
> > Zoltan has reviewed this patch very thoroughly, I have never seen a
> comment
> > from him that the current
> > Patch (one-file-all-settings) approach is not as good as having
> > one-file-per-setting approach.
> > Zoltan, please correct me, If I am wrong.
> 
> I cannot recall arguing for the one-file-per-GUC way.
> But I haven't re-read my mails in this thread, either.

Thanks for confirmation. I also don't think if ever have argument over this
approach.
 
> > What I could understand from mails is Tom has initially suggested to
> have
> > one-file-per-setting but for the current patch
> > (one-file-all-settings) he was telling that if we wanted to go for
> single
> > file approach, then there
> > is no need for separate directory, but for that CRAIG has given a
> scenario
> > where separate directory is useful.
> >
> > Also Robert has suggested from the beginning that (one-file-all-
> settings) is
> > better approach.
> >
> >
> > It took months of discussion to reach the consensus of this level, if
> we
> > again want to change approach,
> > then I think it will be tough to touch this feature again.
> >
> > I think it would be better if we could try to see if there are any
> problems
> > in existing patch which cannot be handled because of it's current
> design,
> > then it will make more sense to conclude on changing approach.
> >
> >> Check
> >> http://www.postgresql.org/message-
> >> id/20130126162728.GA5482@awork2.anarazel.de
> >> and related messages for some of the problems. Most of which are
> >> unhandled in the current patch, i.e. currently you *will* loose
> changes
> >> made in concurrent sessions.
> 
> This mail lists this order for the single file approach:
> 
> > 1) exlusive lock
> > 2) reload config file (to update in memory structures)
> > 3) check new variable
> > 4) write config file (needs to be done atomically)
> > 5) optionally reload config file
> > 6) reload lock
> 
> The patch does it this way:
> 1. check new variable. No problem with that, validation for proper GUC
> name,
>     type of the value, etc. can be done outside the lock.
> 2. grab lock
> 3. parse the config file
> 4. write the new config file
> 5. release lock

Before step-3, we open the auto conf file and in step-3 parse will update
in-memory structures.
So I think Andres's Step-2 is same as the missing step and Step-3 mentioned
by you.

> Reloading the config file is intentionally not done, it's even
> documented.
> You can do SELECT pg_reload_conf() after SET PERSISTENT if you need it.
> 
> > How will somebody loose changes in concurrent sessions?
> > Each session currently waits with LWLock if some other session is
> operating
> > on file. Also after having lock they don't acquire any other lock, so
> there
> > should be no chances of any other problem.
> 
> Specifically, LWLockAcquire() is called first, then ParseConfigFp()
> in a PG_TRY() block, so reading the original postgresql.auto.conf
> is serialized. No chance to lose changes done in parallel.

True.

With Regards,
Amit Kapila.

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Andres Freund <andres@2ndquadrant.com> — 2013-02-12T11:25:19Z

On 2013-02-12 14:57:51 +0530, Amit Kapila wrote:
> On Tuesday, February 12, 2013 11:24 AM Boszormenyi Zoltan wrote:
> > 2013-02-12 04:54 keltezéssel, Amit Kapila írta:
> > > On Tuesday, February 12, 2013 12:54 AM Andres Freund wrote:
> > > Zoltan has reviewed this patch very thoroughly

Sorry, the patch as referenced in the commitfest app is *far* from being
thoroughly reviewed. Its not even remotely ready for commit. I
absolutely don't want to say *anything* against Zoltans reviews or
such. I have looked over the patch before as well, but I found several
severe issues in a 5min look.

> > > I have never seen a comment
> > > from him that the current
> > > Patch (one-file-all-settings) approach is not as good as having
> > > one-file-per-setting approach.
> > > Zoltan, please correct me, If I am wrong.
> > 
> > I cannot recall arguing for the one-file-per-GUC way.
> > But I haven't re-read my mails in this thread, either.

Zoltan, mis(read|remembered) something, sorry.

> Thanks for confirmation. I also don't think if ever have argument over this
> approach.

No idea what youre trying to say here though?

> > >> Check
> > >> http://www.postgresql.org/message-
> > >> id/20130126162728.GA5482@awork2.anarazel.de
> > >> and related messages for some of the problems. Most of which are
> > >> unhandled in the current patch, i.e. currently you *will* loose
> > changes
> > >> made in concurrent sessions.
> > 
> > This mail lists this order for the single file approach:
> > 
> > > 1) exlusive lock
> > > 2) reload config file (to update in memory structures)
> > > 3) check new variable
> > > 4) write config file (needs to be done atomically)
> > > 5) optionally reload config file
> > > 6) reload lock
> > 
> > The patch does it this way:
> > 1. check new variable. No problem with that, validation for proper GUC
> > name,
> >     type of the value, etc. can be done outside the lock.
> > 2. grab lock
> > 3. parse the config file
> > 4. write the new config file
> > 5. release lock

1) You need to grab the lock before the value is checked since some
   variables are interdependent (e.g. log_statement_stats, wal_level,
   archive_mode) and thus the check needs to be made after preventing
   concurrent changes.
2) You need to apply the current config file for exactly the same
   reason before checking the new value. Also
   validate_conf_option/find_option doesn't work appropriately without
   an up2date config file. E.g. CURRENT does absurd things without but I
   think there are other cases as well.

I am not saying its impossible to do the one-file approach correctly, I
just think its harder while not being more useful.

> > Reloading the config file is intentionally not done, it's even
> > documented.  You can do SELECT pg_reload_conf() after SET PERSISTENT
> > if you need it.

Not being done and it being documented as not doing so doesn't make it
correct :P
I think a SET having no immediate results is confusing. Especially if I
am right and we do need to apply previous config changes before doing
the next SET. But I don't have *that* strong feelings about it.

> > Specifically, LWLockAcquire() is called first, then ParseConfigFp()
> > in a PG_TRY() block, so reading the original postgresql.auto.conf
> > is serialized. No chance to lose changes done in parallel.

Not a fundamental issue, but I just noticed LWLockRelease isn't called
in the PG_CATCH branch in set_config_file. There's also an ereport()
which needs to be moved into the PG_TRY to avoid exiting with a held
lock.

I think you also forgot to adjust copyfuncs.c et al for your
VariableSetStmt change (addition of is_persistent).

You should also check for GUC_DISALLOW_IN_FILE in validate_conf_option,
its not the same as PGC_INTERNAL.

What do you mean by "framing" a variable? Surrounding it by ""?

It might be worth adding a check that setting and especially resetting a
user-defined variable works correctly. I.e. persistently setting foo.bar
= 'blub' and then resetting it again.

You cannot use the name passed in via the VariableSetStmt in
set_config_file, you should use the one in 'record' as returned by
find_option (via validate_conf_option) as it handles the correct
name. Otherwise you will get into problems with stuff like TimeZone and
similar.

I think validate_conf_option duplicates far too much code. You need to
unify parts of set_config_option with validate_conf_option.

Greetings,

Andres Freund

-- 
 Andres Freund	                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Amit Kapila <amit.kapila@huawei.com> — 2013-02-12T14:49:43Z

On Tuesday, February 12, 2013 4:55 PM Andres Freund wrote:
> On 2013-02-12 14:57:51 +0530, Amit Kapila wrote:
> > On Tuesday, February 12, 2013 11:24 AM Boszormenyi Zoltan wrote:
> > > 2013-02-12 04:54 keltezéssel, Amit Kapila írta:
> > > > On Tuesday, February 12, 2013 12:54 AM Andres Freund wrote:
> > > > Zoltan has reviewed this patch very thoroughly
> 
> Sorry, the patch as referenced in the commitfest app is *far* from
> being
> thoroughly reviewed. Its not even remotely ready for commit. I
> absolutely don't want to say *anything* against Zoltans reviews or
> such. I have looked over the patch before as well, but I found several
> severe issues in a 5min look.

> > > > I have never seen a comment
> > > > from him that the current
> > > > Patch (one-file-all-settings) approach is not as good as having
> > > > one-file-per-setting approach.
> > > > Zoltan, please correct me, If I am wrong.
> > >
> > > I cannot recall arguing for the one-file-per-GUC way.
> > > But I haven't re-read my mails in this thread, either.
> 
> Zoltan, mis(read|remembered) something, sorry.
> 
> > Thanks for confirmation. I also don't think if ever have argument
> over this
> > approach.
> 
> No idea what youre trying to say here though?


Forget it, this is not important.
Just wanted to say, there is no confusion about approach between 2 of us.

> > > >> Check
> > > >> http://www.postgresql.org/message-
> > > >> id/20130126162728.GA5482@awork2.anarazel.de
> > > >> and related messages for some of the problems. Most of which are
> > > >> unhandled in the current patch, i.e. currently you *will* loose
> > > changes
> > > >> made in concurrent sessions.
> > >
> > > This mail lists this order for the single file approach:
> > >
> > > > 1) exlusive lock
> > > > 2) reload config file (to update in memory structures)
> > > > 3) check new variable
> > > > 4) write config file (needs to be done atomically)
> > > > 5) optionally reload config file
> > > > 6) reload lock
> > >
> > > The patch does it this way:
> > > 1. check new variable. No problem with that, validation for proper
> GUC
> > > name,
> > >     type of the value, etc. can be done outside the lock.
> > > 2. grab lock
> > > 3. parse the config file
> > > 4. write the new config file
> > > 5. release lock
> 
> 1) You need to grab the lock before the value is checked since some
>    variables are interdependent (e.g. log_statement_stats, wal_level,
>    archive_mode) and thus the check needs to be made after preventing
>    concurrent changes.

This can happen if we do any SIGHUP after the command, otherwise it will
have old value only.

> 2) You need to apply the current config file for exactly the same
>    reason before checking the new value. Also
>    validate_conf_option/find_option doesn't work appropriately without
>    an up2date config file. E.g. CURRENT does absurd things without but
> I
>    think there are other cases as well.

At this moment, I am not able to think through this, could you explain by
small example.

I am thinking that shall we remove check hook function and do other
validation only, as this will be done at time
of reload, similar to what will get done when user manually edits the
postgresql.conf file.


> I am not saying its impossible to do the one-file approach correctly, I
> just think its harder while not being more useful.
> 
> > > Reloading the config file is intentionally not done, it's even
> > > documented.  You can do SELECT pg_reload_conf() after SET
> PERSISTENT
> > > if you need it.
> 
> Not being done and it being documented as not doing so doesn't make it
> correct :P
> I think a SET having no immediate results is confusing. Especially if I
> am right and we do need to apply previous config changes before doing
> the next SET. But I don't have *that* strong feelings about it.

I don't think any such expectation should be there, as with this feature
(SET PERSISTENT),
we will allow user to change the settings in file with command instead of
manually editing the file.

> > > Specifically, LWLockAcquire() is called first, then ParseConfigFp()
> > > in a PG_TRY() block, so reading the original postgresql.auto.conf
> > > is serialized. No chance to lose changes done in parallel.
> 
> Not a fundamental issue, but I just noticed LWLockRelease isn't called
> in the PG_CATCH branch in set_config_file. There's also an ereport()
> which needs to be moved into the PG_TRY to avoid exiting with a held
> lock.

I think rollback/abort transaction due to error will handle release of
locks.

> I think you also forgot to adjust copyfuncs.c et al for your
> VariableSetStmt change (addition of is_persistent).

It is there in _copyVariableSetStmt() function.

> You should also check for GUC_DISALLOW_IN_FILE in validate_conf_option,
> its not the same as PGC_INTERNAL.

I shall update the patch to address this issue.

> What do you mean by "framing" a variable? Surrounding it by ""?

Sorry, I am not able to find "framing" in quotes.

> It might be worth adding a check that setting and especially resetting
> a
> user-defined variable works correctly. I.e. persistently setting
> foo.bar
> = 'blub' and then resetting it again.

I shall update the patch to address this issue.

> You cannot use the name passed in via the VariableSetStmt in
> set_config_file, you should use the one in 'record' as returned by
> find_option (via validate_conf_option) as it handles the correct
> name. Otherwise you will get into problems with stuff like TimeZone and
> similar.

I shall check.

> I think validate_conf_option duplicates far too much code. You need to
> unify parts of set_config_option with validate_conf_option.

I had thought of doing this initially, but found it might be little tricky
as duplicate part is not one single chunk.
I shall check, if it can be done in a reasonable way.

Thank you for feedback.

With Regards,
Amit Kapila.

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Andres Freund <andres@2ndquadrant.com> — 2013-02-12T15:32:24Z

On 2013-02-12 20:19:43 +0530, Amit Kapila wrote:
> On Tuesday, February 12, 2013 4:55 PM Andres Freund wrote:
> > On 2013-02-12 14:57:51 +0530, Amit Kapila wrote:
> > > On Tuesday, February 12, 2013 11:24 AM Boszormenyi Zoltan wrote:
> > > > This mail lists this order for the single file approach:
> > > >
> > > > > 1) exlusive lock
> > > > > 2) reload config file (to update in memory structures)
> > > > > 3) check new variable
> > > > > 4) write config file (needs to be done atomically)
> > > > > 5) optionally reload config file
> > > > > 6) reload lock
> > > >
> > > > The patch does it this way:
> > > > 1. check new variable. No problem with that, validation for proper
> > GUC
> > > > name,
> > > >     type of the value, etc. can be done outside the lock.
> > > > 2. grab lock
> > > > 3. parse the config file
> > > > 4. write the new config file
> > > > 5. release lock
> > 
> > 1) You need to grab the lock before the value is checked since some
> >    variables are interdependent (e.g. log_statement_stats, wal_level,
> >    archive_mode) and thus the check needs to be made after preventing
> >    concurrent changes.
> 
> This can happen if we do any SIGHUP after the command, otherwise it will
> have old value only.

Yes, and thats a problem imo.

SET PERSISTENT log_statement_stats = true;
restart;
SET PERSISTENT log_statement_stats = false;
SET PERSISTENT log_parser_stats = true;
ERROR...

thats why I think the config file needs to be processed.

> > 2) You need to apply the current config file for exactly the same
> >    reason before checking the new value. Also
> >    validate_conf_option/find_option doesn't work appropriately without
> >    an up2date config file. E.g. CURRENT does absurd things without but
> > I
> >    think there are other cases as well.
> 
> At this moment, I am not able to think through this, could you explain by
> small example.

The most trivial one I can think of is:

Transaction A: SET PERSISTENT blub = 'bar';
Transaction B: SET PERSISTENT blub FROM CURRENT;

> I am thinking that shall we remove check hook function and do other
> validation only, as this will be done at time
> of reload, similar to what will get done when user manually edits the
> postgresql.conf file.

Why? The user isn't editing the file by hand for a reason.

> > I am not saying its impossible to do the one-file approach correctly, I
> > just think its harder while not being more useful.
> > 
> > > > Reloading the config file is intentionally not done, it's even
> > > > documented.  You can do SELECT pg_reload_conf() after SET
> > PERSISTENT
> > > > if you need it.
> > 
> > Not being done and it being documented as not doing so doesn't make it
> > correct :P
> > I think a SET having no immediate results is confusing. Especially if I
> > am right and we do need to apply previous config changes before doing
> > the next SET. But I don't have *that* strong feelings about it.
> 
> I don't think any such expectation should be there, as with this feature
> (SET PERSISTENT),
> we will allow user to change the settings in file with command instead of
> manually editing the file.

I don't see why that follows. The users *do* want something different,
otherwise they would hand-edit the file.

> > > > Specifically, LWLockAcquire() is called first, then ParseConfigFp()
> > > > in a PG_TRY() block, so reading the original postgresql.auto.conf
> > > > is serialized. No chance to lose changes done in parallel.
> > 
> > Not a fundamental issue, but I just noticed LWLockRelease isn't called
> > in the PG_CATCH branch in set_config_file. There's also an ereport()
> > which needs to be moved into the PG_TRY to avoid exiting with a held
> > lock.
>
> I think rollback/abort transaction due to error will handle release of
> locks.

Yes, in a proper transaction abort this is done but for a utility
command it might be possible to get there without a StartTransaction
being done. I don't immediately see how, but I wouldn't want to rely on
it, especially as doing it is simple.

> > I think you also forgot to adjust copyfuncs.c et al for your
> > VariableSetStmt change (addition of is_persistent).
> 
> It is there in _copyVariableSetStmt() function.

Oh, sorry, skipped over it somehow.

> > What do you mean by "framing" a variable? Surrounding it by ""?
> 
> Sorry, I am not able to find "framing" in quotes.

The quotes were just there to quote the word ;). I was referring to the
following comment:

+ 	/*
+ 	 * The "auto.conf.d" directory should follow the postgresql.conf file
+ 	 * directory not the data_directory. The same is validated while parsing
+ 	 * the postgresql.conf configuration file. So while framing the
+ 	 * postgresql.auto.conf and it's temp file we need to follow the
+ 	 * postgresql.conf file directory.
+ 	 */

> > I think validate_conf_option duplicates far too much code. You need to
> > unify parts of set_config_option with validate_conf_option.
> 
> I had thought of doing this initially, but found it might be little tricky
> as duplicate part is not one single chunk.
> I shall check, if it can be done in a reasonable way.

I think calling validate_conf_option() from set_config_option and
removing the now redundant validation from there should do the trick.

Greetings,

Andres Freund

-- 
 Andres Freund	                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Tom Lane <tgl@sss.pgh.pa.us> — 2013-02-12T15:45:28Z

Andres Freund <andres@2ndquadrant.com> writes:
> On 2013-02-12 20:19:43 +0530, Amit Kapila wrote:
>> On Tuesday, February 12, 2013 4:55 PM Andres Freund wrote:
>>> 1) You need to grab the lock before the value is checked since some
>>> variables are interdependent (e.g. log_statement_stats, wal_level,
>>> archive_mode) and thus the check needs to be made after preventing
>>> concurrent changes.

>> This can happen if we do any SIGHUP after the command, otherwise it will
>> have old value only.

> Yes, and thats a problem imo.

That sounds to me like an entirely unreasonable requirement to put on
the patch.  There is no way to positively guarantee that a value with
interdependencies will load successfully into other sessions, so why
try to enforce it at all?  I note also that trying to make the value
active in the current session doesn't necessarily result in a meaningful
configuration --- what if there's an active session-level SET, for
instance?  You can't just override that.

(I've said this before, but this discussion smells of overdesign every
time I look into the thread.  I can't help thinking this is a big patch
with a small patch struggling to get out.)

			regards, tom lane

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Andres Freund <andres@2ndquadrant.com> — 2013-02-12T16:05:55Z

On 2013-02-12 10:45:28 -0500, Tom Lane wrote:
> Andres Freund <andres@2ndquadrant.com> writes:
> > On 2013-02-12 20:19:43 +0530, Amit Kapila wrote:
> >> On Tuesday, February 12, 2013 4:55 PM Andres Freund wrote:
> >>> 1) You need to grab the lock before the value is checked since some
> >>> variables are interdependent (e.g. log_statement_stats, wal_level,
> >>> archive_mode) and thus the check needs to be made after preventing
> >>> concurrent changes.
> 
> >> This can happen if we do any SIGHUP after the command, otherwise it will
> >> have old value only.
> 
> > Yes, and thats a problem imo.
> 
> That sounds to me like an entirely unreasonable requirement to put on
> the patch.  There is no way to positively guarantee that a value with
> interdependencies will load successfully into other sessions, so why
> try to enforce it at all?  I note also that trying to make the value
> active in the current session doesn't necessarily result in a meaningful
> configuration --- what if there's an active session-level SET, for
> instance?  You can't just override that.

Fair point. The reason I think its reasonable to put more effort into
validation is that I am pretty sure that users will expect that its hard
to get into a state where the server doesn't restart anymore. And
getting 80% of that is better than 50%.
And it seems required to reload the configuration anyway to make FROM
CURRENT work properly. Obviously we can also just document that it
throws away previously set values.

There's also the point that I find it suprising behaviour that SET
PERSISTENT doesn't change the active value unless a manual
pg_reload_conf() is issued.

If we decide that those are not important enough warts to care, yes,
there is no reason for any complication here.

> (I've said this before, but this discussion smells of overdesign every
> time I look into the thread.  I can't help thinking this is a big patch
> with a small patch struggling to get out.)

To be honest, I don't really think thats fair to the patch. I don't see
much that can be made smaller as long as the "one file for all
persistent settings" dogma is upheld which more people seem to vote for
(exluding me).

There's the duplication around validate_conf_option and some comment
cleanup, but otherwise there really isn't that much code that looks
superflous for a simple version?

Greetings,

Andres Freund

-- 
 Andres Freund	                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Amit Kapila <amit.kapila@huawei.com> — 2013-02-12T16:55:49Z

On Tuesday, February 12, 2013 9:02 PM Andres Freund wrote:
> On 2013-02-12 20:19:43 +0530, Amit Kapila wrote:
> > On Tuesday, February 12, 2013 4:55 PM Andres Freund wrote:
> > > On 2013-02-12 14:57:51 +0530, Amit Kapila wrote:
> > > > On Tuesday, February 12, 2013 11:24 AM Boszormenyi Zoltan wrote:
> > > > > This mail lists this order for the single file approach:
> > > > >
> > > > > > 1) exlusive lock
> > > > > > 2) reload config file (to update in memory structures)
> > > > > > 3) check new variable
> > > > > > 4) write config file (needs to be done atomically)
> > > > > > 5) optionally reload config file
> > > > > > 6) reload lock
> > > > >
> > > > > The patch does it this way:
> > > > > 1. check new variable. No problem with that, validation for
> proper
> > > GUC
> > > > > name,
> > > > >     type of the value, etc. can be done outside the lock.
> > > > > 2. grab lock
> > > > > 3. parse the config file
> > > > > 4. write the new config file
> > > > > 5. release lock
> > >
> > > 1) You need to grab the lock before the value is checked since some
> > >    variables are interdependent (e.g. log_statement_stats,
> wal_level,
> > >    archive_mode) and thus the check needs to be made after
> preventing
> > >    concurrent changes.
> >
> > This can happen if we do any SIGHUP after the command, otherwise it
> will
> > have old value only.
> 
> Yes, and thats a problem imo.
> 
> SET PERSISTENT log_statement_stats = true;
> restart;
> SET PERSISTENT log_statement_stats = false;
> SET PERSISTENT log_parser_stats = true;
> ERROR...
> 
> thats why I think the config file needs to be processed.
> 
> > > 2) You need to apply the current config file for exactly the same
> > >    reason before checking the new value. Also
> > >    validate_conf_option/find_option doesn't work appropriately
> without
> > >    an up2date config file. E.g. CURRENT does absurd things without
> but
> > > I
> > >    think there are other cases as well.
> >
> > At this moment, I am not able to think through this, could you
> explain by
> > small example.
> 
> The most trivial one I can think of is:
> 
> Transaction A: SET PERSISTENT blub = 'bar';
> Transaction B: SET PERSISTENT blub FROM CURRENT;
> 
> > I am thinking that shall we remove check hook function and do other
> > validation only, as this will be done at time
> > of reload, similar to what will get done when user manually edits the
> > postgresql.conf file.
> 
> Why? The user isn't editing the file by hand for a reason.
> 
> > > I am not saying its impossible to do the one-file approach
> correctly, I
> > > just think its harder while not being more useful.
> > >
> > > > > Reloading the config file is intentionally not done, it's even
> > > > > documented.  You can do SELECT pg_reload_conf() after SET
> > > PERSISTENT
> > > > > if you need it.
> > >
> > > Not being done and it being documented as not doing so doesn't make
> it
> > > correct :P
> > > I think a SET having no immediate results is confusing. Especially
> if I
> > > am right and we do need to apply previous config changes before
> doing
> > > the next SET. But I don't have *that* strong feelings about it.
> >
> > I don't think any such expectation should be there, as with this
> feature
> > (SET PERSISTENT),
> > we will allow user to change the settings in file with command
> instead of
> > manually editing the file.
> 
> I don't see why that follows. The users *do* want something different,
> otherwise they would hand-edit the file.
> 
> > > > > Specifically, LWLockAcquire() is called first, then
> ParseConfigFp()
> > > > > in a PG_TRY() block, so reading the original
> postgresql.auto.conf
> > > > > is serialized. No chance to lose changes done in parallel.
> > >
> > > Not a fundamental issue, but I just noticed LWLockRelease isn't
> called
> > > in the PG_CATCH branch in set_config_file. There's also an
> ereport()
> > > which needs to be moved into the PG_TRY to avoid exiting with a
> held
> > > lock.
> >
> > I think rollback/abort transaction due to error will handle release
> of
> > locks.
> 
> Yes, in a proper transaction abort this is done but for a utility
> command it might be possible to get there without a StartTransaction
> being done. I don't immediately see how, but I wouldn't want to rely on
> it, especially as doing it is simple.

True, doing it is simple but without reason can raise questions again like
why is it released when it
will be done as part of abort.
I will think more on it to see if there can be any way it can reach without
StartTransaction.

> > > I think you also forgot to adjust copyfuncs.c et al for your
> > > VariableSetStmt change (addition of is_persistent).
> >
> > It is there in _copyVariableSetStmt() function.
> 
> Oh, sorry, skipped over it somehow.
> 
> > > What do you mean by "framing" a variable? Surrounding it by ""?
> >
> > Sorry, I am not able to find "framing" in quotes.
> 
> The quotes were just there to quote the word ;). I was referring to the
> following comment:
> 
> + 	/*
> + 	 * The "auto.conf.d" directory should follow the postgresql.conf
> file
> + 	 * directory not the data_directory. The same is validated while
> parsing
> + 	 * the postgresql.conf configuration file. So while framing the
> + 	 * postgresql.auto.conf and it's temp file we need to follow the
> + 	 * postgresql.conf file directory.
> + 	 */

Shall we change it as one of below:
a. "framing the path for postgresql.auto.conf ... "
b. "framing the name for postgresql.auto.conf ... "
c. "generating the path for postgresql.auto.conf ..."
d. any other suggestion?

> > > I think validate_conf_option duplicates far too much code. You need
> to
> > > unify parts of set_config_option with validate_conf_option.
> >
> > I had thought of doing this initially, but found it might be little
> tricky
> > as duplicate part is not one single chunk.
> > I shall check, if it can be done in a reasonable way.
> 
> I think calling validate_conf_option() from set_config_option and
> removing the now redundant validation from there should do the trick.

Okay.

I have intentionally not replied some parts of message as I want to wait if
there can be conclusion 
for the points you have mentioned in another mail.

With Regards,
Amit Kapila.

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Josh Berkus <josh@agliodbs.com> — 2013-02-12T18:03:27Z

> To be honest, I don't really think thats fair to the patch. I don't see
> much that can be made smaller as long as the "one file for all
> persistent settings" dogma is upheld which more people seem to vote for
> (exluding me).

I'm not a fan of "one file to rule them all" either.  Hovever, I'm
strongly in favor of anything that can get into 9.3.  So if we can make
a 9.3 patch out of this by doing "file-per-setting", then I'm all for it.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Amit Kapila <amit.kapila@huawei.com> — 2013-02-13T12:59:15Z

On Tuesday, February 12, 2013 9:02 PM Andres Freund wrote:
> On 2013-02-12 20:19:43 +0530, Amit Kapila wrote:
> > On Tuesday, February 12, 2013 4:55 PM Andres Freund wrote:
> > > On 2013-02-12 14:57:51 +0530, Amit Kapila wrote:
> > > > On Tuesday, February 12, 2013 11:24 AM Boszormenyi Zoltan wrote:
> > > > > This mail lists this order for the single file approach:
> > > > >
> > > > > > 1) exlusive lock
> > > > > > 2) reload config file (to update in memory structures)
> > > > > > 3) check new variable
> > > > > > 4) write config file (needs to be done atomically)
> > > > > > 5) optionally reload config file
> > > > > > 6) reload lock
> > > > >
> > > > > The patch does it this way:
> > > > > 1. check new variable. No problem with that, validation for
> proper
> > > GUC
> > > > > name,
> > > > >     type of the value, etc. can be done outside the lock.
> > > > > 2. grab lock
> > > > > 3. parse the config file
> > > > > 4. write the new config file
> > > > > 5. release lock
> > >
> > > 1) You need to grab the lock before the value is checked since some
> > >    variables are interdependent (e.g. log_statement_stats,
> wal_level,
> > >    archive_mode) and thus the check needs to be made after
> preventing
> > >    concurrent changes.
> >
> > This can happen if we do any SIGHUP after the command, otherwise it
> will
> > have old value only.
> 
> Yes, and thats a problem imo.
> 
> SET PERSISTENT log_statement_stats = true;
> restart;
> SET PERSISTENT log_statement_stats = false;
> SET PERSISTENT log_parser_stats = true;
> ERROR...
> 
> thats why I think the config file needs to be processed.
> 
> > > 2) You need to apply the current config file for exactly the same
> > >    reason before checking the new value. Also
> > >    validate_conf_option/find_option doesn't work appropriately
> without
> > >    an up2date config file. E.g. CURRENT does absurd things without
> but
> > > I
> > >    think there are other cases as well.
> >
> > At this moment, I am not able to think through this, could you
> explain by
> > small example.
> 
> The most trivial one I can think of is:
> 
> Transaction A: SET PERSISTENT blub = 'bar';
> Transaction B: SET PERSISTENT blub FROM CURRENT;
> 
> > I am thinking that shall we remove check hook function and do other
> > validation only, as this will be done at time
> > of reload, similar to what will get done when user manually edits the
> > postgresql.conf file.
> 
> Why? The user isn't editing the file by hand for a reason.
> 
> > > I am not saying its impossible to do the one-file approach
> correctly, I
> > > just think its harder while not being more useful.
> > >
> > > > > Reloading the config file is intentionally not done, it's even
> > > > > documented.  You can do SELECT pg_reload_conf() after SET
> > > PERSISTENT
> > > > > if you need it.
> > >
> > > Not being done and it being documented as not doing so doesn't make
> it
> > > correct :P
> > > I think a SET having no immediate results is confusing. Especially
> if I
> > > am right and we do need to apply previous config changes before
> doing
> > > the next SET. But I don't have *that* strong feelings about it.
> >
> > I don't think any such expectation should be there, as with this
> feature
> > (SET PERSISTENT),
> > we will allow user to change the settings in file with command
> instead of
> > manually editing the file.
> 
> I don't see why that follows. The users *do* want something different,
> otherwise they would hand-edit the file.
> 
> > > > > Specifically, LWLockAcquire() is called first, then
> ParseConfigFp()
> > > > > in a PG_TRY() block, so reading the original
> postgresql.auto.conf
> > > > > is serialized. No chance to lose changes done in parallel.
> > >
> > > Not a fundamental issue, but I just noticed LWLockRelease isn't
> called
> > > in the PG_CATCH branch in set_config_file. There's also an
> ereport()
> > > which needs to be moved into the PG_TRY to avoid exiting with a
> held
> > > lock.
> >
> > I think rollback/abort transaction due to error will handle release
> of
> > locks.
> 
> Yes, in a proper transaction abort this is done but for a utility
> command it might be possible to get there without a StartTransaction
> being done. I don't immediately see how, but I wouldn't want to rely on
> it, especially as doing it is simple.

We cannot directly call LWLockRelease in catch block as it will lead to
Assert failure.
The reason is errfinish will set InterruptHoldOffCount to 0, and now in
LWLockRelease, 
when it will call RESUME_INTERRUPTS it will check InterruptHoldOffCount
which leads to Assertion failure.

To handle it, we need to call HOLD_INTERRUPTS before it similar to
LWLockReleaseAll().
So I am not sure if we are not sure of scenario, we should add such calls in
Catch block.

> > > I think you also forgot to adjust copyfuncs.c et al for your
> > > VariableSetStmt change (addition of is_persistent).
> >
> > It is there in _copyVariableSetStmt() function.
> 
> Oh, sorry, skipped over it somehow.
> 
> > > What do you mean by "framing" a variable? Surrounding it by ""?
> >
> > Sorry, I am not able to find "framing" in quotes.
> 
> The quotes were just there to quote the word ;). I was referring to the
> following comment:
> 
> + 	/*
> + 	 * The "auto.conf.d" directory should follow the postgresql.conf
> file
> + 	 * directory not the data_directory. The same is validated while
> parsing
> + 	 * the postgresql.conf configuration file. So while framing the
> + 	 * postgresql.auto.conf and it's temp file we need to follow the
> + 	 * postgresql.conf file directory.
> + 	 */
> 
> > > I think validate_conf_option duplicates far too much code. You need
> to
> > > unify parts of set_config_option with validate_conf_option.
> >
> > I had thought of doing this initially, but found it might be little
> tricky
> > as duplicate part is not one single chunk.
> > I shall check, if it can be done in a reasonable way.
> 
> I think calling validate_conf_option() from set_config_option and
> removing the now redundant validation from there should do the trick.

I have tried it, it will not be straight forward, please see some problems
which we need to handle:
1. few checks like GUC_DISALLOW_IN_FILE will be different for persistent
function 
   - we can address this by passing ispersistent flag 
2. Cannot call validate_config_option directly, because the below part of
code is 
   tightly coupled. 
   a. if (value) 
      .. 
      else if() 
      .. 
      else 

      Now if the value part is already checked in validate_config_option,
adjusting this loop will be tricky. 
   b. some of the variable like newextra are required outside function, so
extra parameter 
      needs to be passed. 
   c. memory free issues, as newextra will be required in function
set_config_option, however set_config_file 
      doesn't need it, so it would be better to free that memory inside
validate_config_option 

3. Extra parameter elevel would be required in validate_conf_option. 

4. Function set_config_option is used from many places, so impact can be
higher if we 
   do too many adjustments. 

Considering all above, I think it might not be advisable to remove common
code of validate_conf_option


> It might be worth adding a check that setting and especially resetting a
user-defined variable works correctly. I.e. 
> persistently setting foo.bar = 'blub' and then resetting it again.

What exactly you want in this handling?

With Regards,
Amit Kapila.

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Amit Kapila <amit.kapila@huawei.com> — 2013-02-14T05:46:39Z

On Tuesday, February 12, 2013 4:55 PM Andres Freund wrote:
> On 2013-02-12 14:57:51 +0530, Amit Kapila wrote:
> > On Tuesday, February 12, 2013 11:24 AM Boszormenyi Zoltan wrote:
> > > 2013-02-12 04:54 keltezéssel, Amit Kapila írta:
> > > > On Tuesday, February 12, 2013 12:54 AM Andres Freund wrote:
> > > > Zoltan has reviewed this patch very thoroughly
> 

> > >
> > > This mail lists this order for the single file approach:
> > >
> > > > 1) exlusive lock
> > > > 2) reload config file (to update in memory structures)
> > > > 3) check new variable
> > > > 4) write config file (needs to be done atomically)
> > > > 5) optionally reload config file
> > > > 6) reload lock
> > >
> > > The patch does it this way:
> > > 1. check new variable. No problem with that, validation for proper
> GUC
> > > name,
> > >     type of the value, etc. can be done outside the lock.
> > > 2. grab lock
> > > 3. parse the config file
> > > 4. write the new config file
> > > 5. release lock
> 
> 1) You need to grab the lock before the value is checked since some
>    variables are interdependent (e.g. log_statement_stats, wal_level,
>    archive_mode) and thus the check needs to be made after preventing
>    concurrent changes.

> SET PERSISTENT log_statement_stats = true; restart; SET PERSISTENT
log_statement_stats = false; SET PERSISTENT log_parser_stats = true; ERROR

As we are not loading configuration at end of each command, so this will be
a valid case.

> 2) You need to apply the current config file for exactly the same
>    reason before checking the new value. Also
>    validate_conf_option/find_option doesn't work appropriately without
>    an up2date config file. E.g. CURRENT does absurd things without but
> I
>    think there are other cases as well.

> Transaction A: SET PERSISTENT blub = 'bar'; Transaction B: SET PERSISTENT
blub FROM CURRENT;

This behavior is not only for CURRENT, but any other setting will have same
behavior.
It just retains the last set value for any parameter before restart.

> I am not saying its impossible to do the one-file approach correctly, I
> just think its harder while not being more useful.

IMHO, let's try to see if there are any fundamental problems with this
Approach, then it
will make much more sense to change the Approach.

> > > Reloading the config file is intentionally not done, it's even
> > > documented.  You can do SELECT pg_reload_conf() after SET
> PERSISTENT
> > > if you need it.
> 
> Not being done and it being documented as not doing so doesn't make it
> correct :P
> I think a SET having no immediate results is confusing. Especially if I
> am right and we do need to apply previous config changes before doing
> the next SET. But I don't have *that* strong feelings about it.
> 
> > > Specifically, LWLockAcquire() is called first, then ParseConfigFp()
> > > in a PG_TRY() block, so reading the original postgresql.auto.conf
> > > is serialized. No chance to lose changes done in parallel.
> 
> Not a fundamental issue, but I just noticed LWLockRelease isn't called
> in the PG_CATCH branch in set_config_file. There's also an ereport()
> which needs to be moved into the PG_TRY to avoid exiting with a held
> lock.

We cannot directly call LWLockRelease in catch block as it will lead to
Assert failure.
The reason is errfinish will set InterruptHoldOffCount to 0, and now in
LWLockRelease, when it will call RESUME_INTERRUPTS it will check
InterruptHoldOffCount which leads to Assertion failure.

To handle it, we need to call HOLD_INTERRUPTS before it similar to
LWLockReleaseAll().
So I am not sure if we should add such calls in Catch block.

If you are very sure that such scenario exist where it can reach without
transaction then
we can handle it in above way.

 
> I think you also forgot to adjust copyfuncs.c et al for your
> VariableSetStmt change (addition of is_persistent).


> You should also check for GUC_DISALLOW_IN_FILE in validate_conf_option,
> its not the same as PGC_INTERNAL.

Fixed.
 
> What do you mean by "framing" a variable? Surrounding it by ""?

Changed the line to "faraming the name for".
Here framing means constructing the variable with different inputs.

> It might be worth adding a check that setting and especially resetting
> a
> user-defined variable works correctly. I.e. persistently setting
> foo.bar
> = 'blub' and then resetting it again.

Can you please elaborate a little bit more on this as I am not able to
clearly 
Understand.

> You cannot use the name passed in via the VariableSetStmt in
> set_config_file, you should use the one in 'record' as returned by
> find_option (via validate_conf_option) as it handles the correct
> name. Otherwise you will get into problems with stuff like TimeZone and
> similar.

The name passed in via the VariableSetStmt always is lowercase, which will
be same
same as per current postgresql.conf. 
Doing it is very simple, but I am not able to understand what is the exact
problem you are 
seeing in the current way.

> I think validate_conf_option duplicates far too much code. You need to
> unify parts of set_config_option with validate_conf_option.

I have changed few things in validate_conf_option:
a. comments on top of function
b. memory was not getting freed for one variable.

I have tried to unify, it doesn't seem to be straight forward, please see
some problems which we need to handle:
1. few checks like GUC_DISALLOW_IN_FILE will be different for persistent
function 
   - we can address this by passing ispersistent flag 2. Cannot call
validate_config_option directly, because the below part of code is 
   tightly coupled. 
   a. if (value) 
      .. 
      else if() 
      .. 
      else 

      Now if the value part is already checked in validate_config_option,
adjusting this loop will be tricky. 
   b. some of the variable like newextra are required outside function, so
extra parameter 
      needs to be passed. 
   c. memory free issues, as newextra will be required in function
set_config_option, however set_config_file 
      doesn't need it, so it would be better to free that memory inside
validate_config_option 

3. Extra parameter elevel would be required in validate_conf_option. 

4. Function set_config_option is used from many places, so impact can be
higher if we 
   do too many adjustments. 

Considering all above, I think it might not be advisable to remove common
code of validate_conf_option.

With Regards,
Amit Kapila.

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Greg Smith <greg@2ndquadrant.com> — 2013-03-04T01:07:48Z

This submission didn't have any listed reviewers in the CF app.  I added 
Zoltan and Andres since both of you went through the usual review steps 
and have given lots of feedback.

There are two main sets of issues that keep popping up with this feature:

-The implementation seems too long, at around 2189 lines of diff.  I 
have a few ideas for how things might be trimmed below.  I do appreciate 
that a good part of the length is a long set of regression tests, 
relative to what a feature like this normally gets.

-It might be possible to get a simpler implementation with a file per 
setting.

I can make a pass over cleaning up the wording in your comments and 
documentation.  There are enough coding issues that I think that should 
wait until another rev of the patch first.

= Directory vs. single file =

The main reason I've advocated a configuration file directory is to try 
and suggest a standard for tool generated config files to co-exist in. 
This particular feature doesn't *need* that.  But in the long term I was 
hoping to have more tools that can write out .conf files without having 
to read and understand every existing file that's in there.  It doesn't 
make that big of a difference whether this particular one lives in 
$PGDATA/postgresql.auto.conf or $PGDATA/postgresql.auto.conf.  The main 
reason for the directory is to make the second such tool not further 
clutter the main $PGDATA directory.

I would like to see the name of the directory be conf.d instead of 
auto.conf.d though.  What's "auto" about it?  Using that word just adds 
a source of confusion.  The same problem exists with the file name 
itself.  I was hoping for conf.d/persistent.conf here, and no use of the 
word "auto" in the code itself.

What I don't see a lot of value in is the config file per setting idea. 
  I was hoping SET PERSISTENT put all of its changes into once place. 
It should be obvious where they came from, and that people can't edit 
that file manually without breaking the feature.  To me something like 
persistent.conf gives that impression, while postgresql.auto.conf sounds 
like the output from some auto-tuning tool.

= Length reduction =

I'm not sure why you are opening the old auto config file with 
ParseConfigFp.  Can't you just navigate the existing GUCs in memory and 
directly write the new one out?  If someone is going to manually edit 
this file and use SET PERSISTENT, they're going to end up in trouble 
regardless.  I don't think it's really worth the extra complexity needed 
to try and handle that case.

It sounds like you've thought a good bit about how to eliminate all the 
code in the validate_conf_option function.  I think that idea may take 
some more thinking to try and be creative about it.  Have you looked at 
whether there's any way to refactor this to be smaller by modifying 
guc.c to break out parts of its implementation needed here?

Is it possible to modify your changes to src/backend/parser/gram.y to 
produce a smaller diff?  There looks to be a good amount of code (~25 
lines) that aren't really changed that I can see.  I suspect they might 
have been hit with an editor whitespace change.

= General code issues =

There is some trivial bit rot on your v10 here:

1 out of 8 hunks FAILED -- saving rejects to file 
src/backend/parser/gram.y.rej
1 out of 3 hunks FAILED -- saving rejects to file 
src/bin/initdb/initdb.c.rej

Your change to gram.y was broken by PROGRAM being added.  I'm not sure 
why the initdb.c one doesn't apply; whitespace issue maybe?  As written 
right now, even with fixing those two, I'm getting this compiler error:

gram.y:12694.27-36: warning: type clash on default action: <keyword> != <>
gram.y:1306.31-40: symbol PERSISTENT is used, but is not defined as a 
token and has no rules

This combination in src/test/regress/sql/persistent.sql a few times 
makes me nervous:

+ select pg_reload_conf();
+ select pg_sleep(1);

Why the pause?  There are some regression tests with sleeps in them, 
such as the stats collector and the timezone tests.  I would expect 
pg_reload_conf would not return until its configuration is stable, so 
this is a bit of a surprise.

-- 
Greg Smith   2ndQuadrant US    greg@2ndQuadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Josh Berkus <josh@agliodbs.com> — 2013-03-04T01:25:29Z

> = Directory vs. single file =
> 
> The main reason I've advocated a configuration file directory is to try
> and suggest a standard for tool generated config files to co-exist in.
> This particular feature doesn't *need* that.  But in the long term I was
> hoping to have more tools that can write out .conf files without having
> to read and understand every existing file that's in there.  It doesn't
> make that big of a difference whether this particular one lives in
> $PGDATA/postgresql.auto.conf or $PGDATA/postgresql.auto.conf.  The main
> reason for the directory is to make the second such tool not further
> clutter the main $PGDATA directory.
> 
> I would like to see the name of the directory be conf.d instead of
> auto.conf.d though.  What's "auto" about it?  Using that word just adds
> a source of confusion.  The same problem exists with the file name
> itself.  I was hoping for conf.d/persistent.conf here, and no use of the
> word "auto" in the code itself.

my $0.03: I agree with Greg that using the directory is a good idea, and
that his naming is an improvement.

> What I don't see a lot of value in is the config file per setting idea.
>  I was hoping SET PERSISTENT put all of its changes into once place. It
> should be obvious where they came from, and that people can't edit that
> file manually without breaking the feature.  To me something like
> persistent.conf gives that impression, while postgresql.auto.conf sounds
> like the output from some auto-tuning tool.

I think most users would prefer a single file.  The reason
file-per-setting is being discussed is that it avoids a lot of locking
issues and code complexity.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Tom Lane <tgl@sss.pgh.pa.us> — 2013-03-04T01:34:12Z

Josh Berkus <josh@agliodbs.com> writes:
>> I would like to see the name of the directory be conf.d instead of
>> auto.conf.d though.  What's "auto" about it?  Using that word just adds
>> a source of confusion.  The same problem exists with the file name
>> itself.  I was hoping for conf.d/persistent.conf here, and no use of the
>> word "auto" in the code itself.

> my $0.03: I agree with Greg that using the directory is a good idea, and
> that his naming is an improvement.

Neither of those names is consistent with any other PGDATA subdirectory
name we use.  It should just be config, or perhaps pg_config, though the
latter risks confusion with the tool of the same name.

FWIW, I do think that having "auto" or some such in the file name(s)
would be a good idea, to help warn people off editing them manually.

			regards, tom lane

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Greg Smith <greg@2ndquadrant.com> — 2013-03-04T01:55:34Z

On 3/3/13 8:34 PM, Tom Lane wrote:
> Neither of those names is consistent with any other PGDATA subdirectory
> name we use.  It should just be config, or perhaps pg_config, though the
> latter risks confusion with the tool of the same name.

I'd be just as happy with config/ as the directory name.  I have a bias 
toward wanting this to look like Apache that I've been scolded for 
before, I forgot to avoid that this time.

> FWIW, I do think that having "auto" or some such in the file name(s)
> would be a good idea, to help warn people off editing them manually.

I can see that for the file name itself, as long as it's not in the 
directory name.  I don't like giving the idea that everything in there 
is automatically generated.  config/persistent-auto.conf or 
config/persist-auto.conf maybe?

The way files are sorted by name means that using 
config/auto-persist[ent].conf would make the file more likely to be 
processed before other things in that directory.  I could live with that 
order too.  I think it is important to name the file such that it 
suggests being connected to the SET PERSISTENT feature though.

-- 
Greg Smith   2ndQuadrant US    greg@2ndQuadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Craig Ringer <craig@2ndquadrant.com> — 2013-03-04T04:11:20Z

On 03/04/2013 09:07 AM, Greg Smith wrote:
> I'm not sure why you are opening the old auto config file with
> ParseConfigFp.  Can't you just navigate the existing GUCs in memory
> and directly write the new one out?  If someone is going to manually
> edit this file and use SET PERSISTENT, they're going to end up in
> trouble regardless.  I don't think it's really worth the extra
> complexity needed to try and handle that case.
Additionally, if you want to avoid silently overwriting user changes,
you could store a timestamp for when we last updated the persistent
config and compare it to the on-disk timestamp before writing. If they
don't match a warning would be issued and the config would be
overwritten anyway. There's a race, of course, but since the worst case
is that we fail to issue a warning it's a pretty harmless one.

As for the per-file vs single-file issue and concerns about locking
complexity: Can't we just use a global lock in shm to enforce that
exactly one backend at a time may be modifying the global configuration?
I don't see this ever becoming a realistic concern for concurrency and
performance, and the shm cost would be tiny.

-- 
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Craig Ringer <craig@2ndquadrant.com> — 2013-03-04T04:12:18Z

On 03/04/2013 09:34 AM, Tom Lane wrote:
> Josh Berkus <josh@agliodbs.com> writes:
>>> I would like to see the name of the directory be conf.d instead of
>>> auto.conf.d though.  What's "auto" about it?  Using that word just adds
>>> a source of confusion.  The same problem exists with the file name
>>> itself.  I was hoping for conf.d/persistent.conf here, and no use of the
>>> word "auto" in the code itself.
>> my $0.03: I agree with Greg that using the directory is a good idea, and
>> that his naming is an improvement.
> Neither of those names is consistent with any other PGDATA subdirectory
> name we use.  It should just be config, or perhaps pg_config, though the
> latter risks confusion with the tool of the same name.
>
> FWIW, I do think that having "auto" or some such in the file name(s)
> would be a good idea, to help warn people off editing them manually.
Yep; auto in the dir name seems unwise given that we want to allow tools
to drop config snippets in there, but auto in the file name would IMO
help people realise it shouldn't be edited directly.

-- 
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Amit Kapila <amit.kapila@huawei.com> — 2013-03-04T05:18:12Z

On Monday, March 04, 2013 7:26 AM Greg Smith wrote:
> On 3/3/13 8:34 PM, Tom Lane wrote:
> > Neither of those names is consistent with any other PGDATA
> subdirectory
> > name we use.  It should just be config, or perhaps pg_config, though
> the
> > latter risks confusion with the tool of the same name.
> 
> I'd be just as happy with config/ as the directory name.  I have a bias
> toward wanting this to look like Apache that I've been scolded for
> before, I forgot to avoid that this time.
> 
> > FWIW, I do think that having "auto" or some such in the file name(s)
> > would be a good idea, to help warn people off editing them manually.
> 
> I can see that for the file name itself, as long as it's not in the
> directory name.  I don't like giving the idea that everything in there
> is automatically generated.  config/persistent-auto.conf or
> config/persist-auto.conf maybe?
> 
> The way files are sorted by name means that using
> config/auto-persist[ent].conf would make the file more likely to be
> processed before other things in that directory.  I could live with
> that
> order too.  I think it is important to name the file such that it
> suggests being connected to the SET PERSISTENT feature though.

If we want to link it with SET PERSISTENT feature, we can choose name as you
suggest or
config/auto.persistent.conf?

If we see the current files postgresql.conf, recovery.conf etc, it seems
most are related
to the functionality it provides. For example recovery.conf suggests that it
contains settings for doing recovery.

The current name postgresql.auto.conf has been chosen to make it's
connection with user editable
postgresql.conf. I thought it will give impression to user that this
contains settings similar to postgresql.conf
but this file is auto generated, so it can't be editable by user.

With Regards,
Amit Kapila.

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Amit Kapila <amit.kapila@huawei.com> — 2013-03-04T05:23:54Z

On Monday, March 04, 2013 9:41 AM Craig Ringer wrote:
> On 03/04/2013 09:07 AM, Greg Smith wrote:
> > I'm not sure why you are opening the old auto config file with
> > ParseConfigFp.  Can't you just navigate the existing GUCs in memory
> > and directly write the new one out?  If someone is going to manually
> > edit this file and use SET PERSISTENT, they're going to end up in
> > trouble regardless.  I don't think it's really worth the extra
> > complexity needed to try and handle that case.
> Additionally, if you want to avoid silently overwriting user changes,
> you could store a timestamp for when we last updated the persistent
> config and compare it to the on-disk timestamp before writing. If they
> don't match a warning would be issued and the config would be
> overwritten anyway. There's a race, of course, but since the worst case
> is that we fail to issue a warning it's a pretty harmless one.


> As for the per-file vs single-file issue and concerns about locking
> complexity: Can't we just use a global lock in shm to enforce that
> exactly one backend at a time may be modifying the global
> configuration?

Yes, this is currently used.

> I don't see this ever becoming a realistic concern for concurrency and
> performance, and the shm cost would be tiny.

With Regards,
Amit Kapila.

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Amit Kapila <amit.kapila@huawei.com> — 2013-03-04T15:15:31Z

On Monday, March 04, 2013 6:38 AM Greg Smith wrote:
> This submission didn't have any listed reviewers in the CF app.  I
> added
> Zoltan and Andres since both of you went through the usual review steps
> and have given lots of feedback.

Thank you for review.
 
> There are two main sets of issues that keep popping up with this
> feature:
> 
> -The implementation seems too long, at around 2189 lines of diff.  I
> have a few ideas for how things might be trimmed below.  I do
> appreciate
> that a good part of the length is a long set of regression tests,
> relative to what a feature like this normally gets.
> 
> -It might be possible to get a simpler implementation with a file per
> setting.
> 
> I can make a pass over cleaning up the wording in your comments and
> documentation.  There are enough coding issues that I think that should
> wait until another rev of the patch first.

I have tried to completely handle all the comments, but need 1 more day.
 
> = Directory vs. single file =
> 
> The main reason I've advocated a configuration file directory is to try
> and suggest a standard for tool generated config files to co-exist in.
> This particular feature doesn't *need* that.  But in the long term I
> was
> hoping to have more tools that can write out .conf files without having
> to read and understand every existing file that's in there.  It doesn't
> make that big of a difference whether this particular one lives in
> $PGDATA/postgresql.auto.conf or $PGDATA/postgresql.auto.conf.  The main
> reason for the directory is to make the second such tool not further
> clutter the main $PGDATA directory.
> 
> I would like to see the name of the directory be conf.d instead of
> auto.conf.d though.  What's "auto" about it?  Using that word just adds
> a source of confusion.  The same problem exists with the file name
> itself.  I was hoping for conf.d/persistent.conf here, and no use of
> the
> word "auto" in the code itself.

We can name the directory as config.
For file, I am awaiting your reponse to select from below options
a. persistent.auto.conf
b. persist.auto.conf
c. auto.persist.conf
d. postgresql.auto.conf
 
> What I don't see a lot of value in is the config file per setting idea.
>   I was hoping SET PERSISTENT put all of its changes into once place.
> It should be obvious where they came from, and that people can't edit
> that file manually without breaking the feature.  To me something like
> persistent.conf gives that impression, while postgresql.auto.conf
> sounds
> like the output from some auto-tuning tool.
> 
> = Length reduction =
> 
> I'm not sure why you are opening the old auto config file with
> ParseConfigFp.  Can't you just navigate the existing GUCs in memory and
> directly write the new one out?  If someone is going to manually edit
> this file and use SET PERSISTENT, they're going to end up in trouble
> regardless.  I don't think it's really worth the extra complexity
> needed
> to try and handle that case.

This is for logic to see if variable already exist just replace it's value
at the same
location in .auto file, else append at end.
IIUC what you are suggesting is set first in memory then while writing just
check the memory, whatever
Is new value just write it.
if we just traverse in memory how will we know where this exist in file. 
This might have complication what if write to memory is success, but to file
it fails.

> It sounds like you've thought a good bit about how to eliminate all the
> code in the validate_conf_option function.  I think that idea may take
> some more thinking to try and be creative about it.  Have you looked at
> whether there's any way to refactor this to be smaller by modifying
> guc.c to break out parts of its implementation needed here?

Today I tried again by having single common function for both places, but
It is getting clumsy. So I can to write multiple functions for each
datatype?
 
> Is it possible to modify your changes to src/backend/parser/gram.y to
> produce a smaller diff?  There looks to be a good amount of code (~25
> lines) that aren't really changed that I can see.  I suspect they might
> have been hit with an editor whitespace change.

I will do pgindent. 
Can you point what are extra lines that can be removed. 

> = General code issues =
> 
> There is some trivial bit rot on your v10 here:
> 
> 1 out of 8 hunks FAILED -- saving rejects to file
> src/backend/parser/gram.y.rej
> 1 out of 3 hunks FAILED -- saving rejects to file
> src/bin/initdb/initdb.c.rej
> 
> Your change to gram.y was broken by PROGRAM being added.  I'm not sure
> why the initdb.c one doesn't apply; whitespace issue maybe?  As written
> right now, even with fixing those two, I'm getting this compiler error:
> 
> gram.y:12694.27-36: warning: type clash on default action: <keyword> !=
> <>
> gram.y:1306.31-40: symbol PERSISTENT is used, but is not defined as a
> token and has no rules

I shall take care of these, minor code related issues due to recent changes
in HEAD.

> This combination in src/test/regress/sql/persistent.sql a few times
> makes me nervous:
> 
> + select pg_reload_conf();
> + select pg_sleep(1);
> 
> Why the pause?  There are some regression tests with sleeps in them,
> such as the stats collector and the timezone tests.  I would expect
> pg_reload_conf would not return until its configuration is stable, so
> this is a bit of a surprise.

pg_reload_conf(), doesn't ensure that configuration has been propagated to
memory.
It just sends the SIGHUP signal to postmaster.
If I remove, it will fail the tests.

With Regards,
Amit Kapila.

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Amit Kapila <amit.kapila@huawei.com> — 2013-03-05T14:07:13Z

On Monday, March 04, 2013 8:46 PM Amit Kapila wrote:
> On Monday, March 04, 2013 6:38 AM Greg Smith wrote:
> > This submission didn't have any listed reviewers in the CF app.  I
> > added
> > Zoltan and Andres since both of you went through the usual review
> steps
> > and have given lots of feedback.
> 
> Thank you for review.

Fujii Masao has also done the review and test of patch multiple times.

> > There are two main sets of issues that keep popping up with this
> > feature:
> >
> > -The implementation seems too long, at around 2189 lines of diff.  I
> > have a few ideas for how things might be trimmed below.  I do
> > appreciate
> > that a good part of the length is a long set of regression tests,
> > relative to what a feature like this normally gets.
> >
> > -It might be possible to get a simpler implementation with a file per
> > setting.
> >
> > I can make a pass over cleaning up the wording in your comments and
> > documentation.  There are enough coding issues that I think that
> should
> > wait until another rev of the patch first.
> 
> I have tried to completely handle all the comments, but need 1 more
> day.
> 
> > = Directory vs. single file =
> >
> > The main reason I've advocated a configuration file directory is to
> try
> > and suggest a standard for tool generated config files to co-exist
> in.
> > This particular feature doesn't *need* that.  But in the long term I
> > was
> > hoping to have more tools that can write out .conf files without
> having
> > to read and understand every existing file that's in there.  It
> doesn't
> > make that big of a difference whether this particular one lives in
> > $PGDATA/postgresql.auto.conf or $PGDATA/postgresql.auto.conf.  The
> main
> > reason for the directory is to make the second such tool not further
> > clutter the main $PGDATA directory.
> >
> > I would like to see the name of the directory be conf.d instead of
> > auto.conf.d though.  What's "auto" about it?  Using that word just
> adds
> > a source of confusion.  The same problem exists with the file name
> > itself.  I was hoping for conf.d/persistent.conf here, and no use of
> > the
> > word "auto" in the code itself.
> 
> We can name the directory as config.
> For file, I am awaiting your reponse to select from below options
> a. persistent.auto.conf
> b. persist.auto.conf
> c. auto.persist.conf
> d. postgresql.auto.conf

In v11 patch, I have changed name of directory to config.
For file name, currently I have not changed, but if you feel it needs to be
changed, kindly suggest any one of above or if any other better you have in
mind.

> > What I don't see a lot of value in is the config file per setting
> idea.
> >   I was hoping SET PERSISTENT put all of its changes into once place.
> > It should be obvious where they came from, and that people can't edit
> > that file manually without breaking the feature.  To me something
> like
> > persistent.conf gives that impression, while postgresql.auto.conf
> > sounds
> > like the output from some auto-tuning tool.
> >
> > = Length reduction =
> >
> > I'm not sure why you are opening the old auto config file with
> > ParseConfigFp.  Can't you just navigate the existing GUCs in memory
> and
> > directly write the new one out?  If someone is going to manually edit
> > this file and use SET PERSISTENT, they're going to end up in trouble
> > regardless.  I don't think it's really worth the extra complexity
> > needed
> > to try and handle that case.
> 
> This is for logic to see if variable already exist just replace it's
> value
> at the same
> location in .auto file, else append at end.
> IIUC what you are suggesting is set first in memory then while writing
> just
> check the memory, whatever
> Is new value just write it.
> if we just traverse in memory how will we know where this exist in
> file.
> This might have complication what if write to memory is success, but to
> file
> it fails.
> 
> > It sounds like you've thought a good bit about how to eliminate all
> the
> > code in the validate_conf_option function.  I think that idea may
> take
> > some more thinking to try and be creative about it.  Have you looked
> at
> > whether there's any way to refactor this to be smaller by modifying
> > guc.c to break out parts of its implementation needed here?
> 
> Today I tried again by having single common function for both places,
> but
> It is getting clumsy. So I can to write multiple functions for each
> datatype?

Finally, I could come up with unified function with common parts together
based on
your suggestions.

> > Is it possible to modify your changes to src/backend/parser/gram.y to
> > produce a smaller diff?  There looks to be a good amount of code (~25
> > lines) that aren't really changed that I can see.  I suspect they
> might
> > have been hit with an editor whitespace change.
> 
> I will do pgindent.
> Can you point what are extra lines that can be removed.

I think it is mainly due to the changes done in gram.y.
 
> > = General code issues =
> >
> > There is some trivial bit rot on your v10 here:
> >
> > 1 out of 8 hunks FAILED -- saving rejects to file
> > src/backend/parser/gram.y.rej
> > 1 out of 3 hunks FAILED -- saving rejects to file
> > src/bin/initdb/initdb.c.rej
> >
> > Your change to gram.y was broken by PROGRAM being added.  I'm not
> sure
> > why the initdb.c one doesn't apply; whitespace issue maybe?  As
> written
> > right now, even with fixing those two, I'm getting this compiler
> error:
> >
> > gram.y:12694.27-36: warning: type clash on default action: <keyword>
> !=
> > <>
> > gram.y:1306.31-40: symbol PERSISTENT is used, but is not defined as a
> > token and has no rules
> 
> I shall take care of these, minor code related issues due to recent
> changes
> in HEAD.

Gram.y - PERSISTENT word is missed from one of the list
Initdb.c - structure subdirs, doesn't contain new directory.
Fixed both of the above.

> > This combination in src/test/regress/sql/persistent.sql a few times
> > makes me nervous:
> >
> > + select pg_reload_conf();
> > + select pg_sleep(1);
> >
> > Why the pause?  There are some regression tests with sleeps in them,
> > such as the stats collector and the timezone tests.  I would expect
> > pg_reload_conf would not return until its configuration is stable, so
> > this is a bit of a surprise.
> 
> pg_reload_conf(), doesn't ensure that configuration has been propagated
> to
> memory.
> It just sends the SIGHUP signal to postmaster.
> If I remove, it will fail the tests.

I have kept this intact.

With Regards,
Amit Kapila.

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Greg Smith <greg@2ndquadrant.com> — 2013-03-07T05:23:54Z

On 3/5/13 9:07 AM, Amit Kapila wrote:
> In v11 patch, I have changed name of directory to config.
> For file name, currently I have not changed, but if you feel it needs to be
> changed, kindly suggest any one of above or if any other better you have in
> mind.

This seems fine for now.  Hashing out exactly which name is best is not 
going to be the thing that determines whether this can be committed or not.

Your v11 applies cleanly for me too, and the code does look a lot nicer. 
  The code diff itself is down to 1300 lines and not nearly as 
disruptive, the rest is docs or the many regression tests you've added. 
  That's still not the sort of small change Tom was hoping this was 
possible, but it's closer to the right range.  It may not really be 
possible to make this much smaller.

I'm out of time for today, but I'll try to do some functional review 
work on this tomorrow if no one beats me to it.  I would find it very 
useful to me personally if this feature were committed, and we know 
there's plenty of user demand for it too.

-- 
Greg Smith   2ndQuadrant US    greg@2ndQuadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Amit Kapila <amit.kapila@huawei.com> — 2013-03-07T07:42:27Z

On Thursday, March 07, 2013 10:54 AM Greg Smith wrote:
> On 3/5/13 9:07 AM, Amit Kapila wrote:
> > In v11 patch, I have changed name of directory to config.
> > For file name, currently I have not changed, but if you feel it needs
> to be
> > changed, kindly suggest any one of above or if any other better you
> have in
> > mind.
> 
> This seems fine for now.  Hashing out exactly which name is best is not
> going to be the thing that determines whether this can be committed or
> not.
> 
> Your v11 applies cleanly for me too, and the code does look a lot
> nicer.
>   The code diff itself is down to 1300 lines and not nearly as
> disruptive, the rest is docs or the many regression tests you've added.

I also think the tests added for regression may be more than required.
Currently it is organized as below:

1. 3 type of parameters to use with set persistent. 
   a. parameters for which new connection is required (log_connections) 
   b. parameters for which server restart is required
   c. parameters for which reload is required

   I think in third category "c", there are many parameters, I think may be
3-4 parameters should be sufficient.
   The initial intention to keep more parameters is to have test it for
various type(Boolean, integer, float, string)

2. Set some of the parameters to default, then try reload and reconnect.
3. Again set those parameters to some value and re-verify by reload and
reconnect.
4. Test for set persistent command inside functions and transaction.
5. Set all parameters value to default.

I think tests for step-3 can be removed and tests for step-5 can be merged
to step-2.

If you think above optimization's to reduce number of tests are okay, then I
will 
update the patch.

>   That's still not the sort of small change Tom was hoping this was
> possible, but it's closer to the right range.  It may not really be
> possible to make this much smaller.
> 
> I'm out of time for today, but I'll try to do some functional review
> work on this tomorrow if no one beats me to it.  I would find it very
> useful to me personally if this feature were committed, and we know
> there's plenty of user demand for it too.

With Regards,
Amit Kapila.

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Greg Smith <greg@2ndquadrant.com> — 2013-03-10T15:12:51Z

On 3/7/13 2:42 AM, Amit Kapila wrote:
> I also think the tests added for regression may be more than required...
> If you think above optimization's to reduce number of tests are okay, then I
> will update the patch.

I was not trying to get you to remove regression tests.  I was just 
pointing out to everyone that the patch seems longer than it really is, 
because you put so many of them in there.  This is not a problem, and I 
didn't bring it up to say you should remove some of them.

This feature is close to being functional enough to make a lot of people 
happy.  Most of it works the way I've wanted it to for a few years now, 
an end point several people have pushed toward in pieces for years now. 
  There is a decent sized list of issues that I found under careful 
testing though.  Maybe all these can be resolved quickly and this moves 
on to commit candidate.  I'd like to see this feature hit the code base, 
but it feels like a good amount of work is left to reach there to me 
right now.

I'm focusing on functional rather than code quality review here.  My 
opinion on how the implementation is coded isn't worth very much 
anyway--I have a bad track record for correctly judging whether 
something is good quality when it starts messing with the GUC system. 
The patch has shrunk since original submission a bit, and it has many 
regression tests it passes.  It may still be objectionably long.  There 
is a good amount of cleanup in comments and documentation needed, and I 
will be happy to work on that part myself.  I wanted to get the 
functional issues outlined first though.  I would be shocked if this 
goes in without someone else wanting more code cleanup or 
simplification.  I don't know who is going to do that other than one of 
the committers though.

If Amit can address the functional ones, I can tweak the text/level of 
the log messages myself during the cleanup round I do.

= Functional changes =

There are a number of individually small but serious blocker items I 
think need to be changed in how this works.  I see 7 functional changes 
to be made before there's no surprising behavior here.  I have a lot of 
commentary about how the log/error messages from this might be improved 
in addition to that.

1) When you change a sighup or user setting, it writes the config file 
out.  But it does not signal for a reload.  Example:

$ psql -c "show work_mem" -x
work_mem | 1MB
$ psql -c "set persistent work_mem='2MB'"
SET
$ psql -c "show work_mem" -x
work_mem | 1MB

That the above doesn't leave work_mem set to 2MB for new sessions is 
surprising.  SET PERSISTENT can't do anything about postmaster 
parameters like shared_buffers.  But it should be able to change 
work_mem or a sighup parameter like checkpoint_segments.  The regression 
test examples call pg_reload_conf() in order to activate the changes. 
As a user, I would expect the config reload to be automatic after 
executing SET PERSISTENT on parameters that can be changed dynamically.

2) Once automatic activation is happening, a hint is really needed in 
cases where it can't happen.  I'd like to see a warning about that. 
There's one like this in the code already:

LOG:  parameter "shared_buffers" cannot be changed without restarting 
the server

Something like this would be appropriate for SET PERSISTENT:

LOG:  parameter "shared_buffers" cannot be changed without restarting 
the server.  The changed setting has been saved to 
config/postgresql.auto.conf.

3) The config/postgresql.auto.conf file (which I'd like to see named 
'config/persistent.auto.conf') is completely overwritten when a change 
is made.  When the file is written out, both at initdb time and when 
changes are made, it should have this as its first line:

# Do not edit this file manually!  It is overwritten by the SET 
PERSISTENT command.

4) There is one bad problem case left if I try to make this not work. 
If I do this:

-rm -rf config/
-Remove "include_dir config"
-Restart the server.  It gives me a warning that SET PERSISTENT won't 
work because at least one of the multiple things it needs are missing.
-mkdir config
-psql -c "set persistent checkpoint_segments='32'"

That command happily writes out postgresql.auto.conf, but since I 
removed the include_dir it doesn't matter.  It will never become active.

The check for the include_dir entry needs to happen each time SET 
PERSISTENT runs.  This is more important than noticing it at server 
start time.  In fact, I think if this problem case includes the WARNING 
about "include_dir config" not being in the postgresql.conf, the check 
that happens at server start (shown below) might even go away.  People 
who object to it as log file garbage will be happier, and users will be 
told about the feature being broken if they try to use it.

5) An error message appears every time you reload configuration, if some 
part of the SET PERSISTENT mechanism isn't functional.  This is going to 
be too much for some people.  And it's confusing when it happens as part 
an interactive psql session.  I have an example way down at the very end 
of this message.

6) Putting spaces into memory units results in broken config files:

$ psql -c "set persistent work_mem='2 MB'"
SET
$ cat $PGDATA/config/postgresql.auto.conf
work_mem = 2 MB

This is wrong and you'll get this on reload:

LOG:  syntax error in file 
"/Users/gsmith/pgwork/data/autoconf/config/postgresql.auto.conf" line 1, 
near token "MB"
LOG:  configuration file 
"/Users/gsmith/pgwork/data/autoconf/postgresql.conf" contains errors; no 
changes were applied

It needs to look like this:

work_mem = '2 MB'

A regression test for some cases with spaces in the middle should be 
added too.  This case is really weird, due to how the code is reading 
the existing postgresql.auto.conf file.  If I first manually fix the 
file, and then run a change again, it stays fixed!  Look at this weird 
sequence:

$ psql -c "set persistent work_mem='2 MB'"
SET
$ cat $PGDATA/config/postgresql.auto.conf
work_mem = 2 MB
[Here this setting is broken]
$ vi $PGDATA/config/postgresql.auto.conf
[Add quotes around the value, now it works]
$ cat $PGDATA/config/postgresql.auto.conf
work_mem = '2 MB'
$ pg_ctl reload
server signaled
$ psql -c "set persistent work_mem='4 MB'"
SET
$ cat $PGDATA/config/postgresql.auto.conf
work_mem = '4 MB'

Below I'll rant some more about how looking at what's in the existing 
file, rather than the in-memory GUCs, has some other surprising properties.

7) If I run SET PERSISTENT a lot concurrently, something happens that 
slows down connecting to the server.  Restarting the server makes it go 
away.  I have a pgbench test case demonstrating the problem below, in 
the "Concurrency" section.  I haven't tried to replicate it on another 
system yet.

= Tested features that work fine =

Entries added here are tracked as you'd expect:

$ psql -c "set persistent checkpoint_segments='32'"
$ pg_ctl reload
$ psql -xc "select name,setting,sourcefile,sourceline from pg_settings 
where name='checkpoint_segments'"
name       | checkpoint_segments
setting    | 32
sourcefile | /Users/gsmith/pgwork/data/autoconf/config/postgresql.auto.conf
sourceline | 1

When the postgresql.auto.conf file is missing and you use SET 
PERSISTENT, it quietly creates it when writing out the new setting.

= Concurrency and performance slowdown =

I made two pgbench scripts that adjust a guc, one user and the other 
sighup, to a random value:

random-join-limit.sql:

\setrandom limit 1 8
set persistent join_collapse_limit=:limit;
select pg_reload_conf();

random-segments.sql:

\setrandom segments 3 16
set persistent checkpoint_segments=:segments;
select pg_reload_conf();

I then fired off a bunch of these in parallel:

pgbench -n -f random-join-limit.sql -f random-segments.sql -c 8 -T 60

This ran at 1547 TPS on my 8 core Mac, so that's not bad.  No assertions 
and the config file was still valid at the end, which is a good sign the 
locking method isn't allowing utter chaos.  Without the pg_reload_conf() 
in the test files, it also completed.  With the reload happening in one 
file but not the other, things were also fine.

However, one thing I saw is that the server got significantly slower the 
more I ran this test script.  After a few minutes it was down to only 
400 TPS.  The delay shows up between when I run psql and when I get the 
prompt back.  Here's normal behavior:

$ time psql -c "select 1"
real	0m0.006s

And here's what I get after a single run of this pgbench hammering:

$ time psql -c "select 1"
real	0m0.800s

800ms?  The slowdown is all for psql to start and connect, it's not in 
the executor:

$ time psql -c "explain analyze select 1"
                                      QUERY PLAN
------------------------------------------------------------------------------------
  Result  (cost=0.00..0.01 rows=1 width=0) (actual time=0.001..0.001 
rows=1 loops=1)
  Total runtime: 0.039 ms
(2 rows)

real	0m0.807s

Stopping and restarting the server brings the performance back to 
normal.  Maybe this is one of those assertion build issues, but it 
smells like there could be nasty memory leak somewhere instead. 
Clearing whatever weirdness is going on here is a blocking issue.

= Loss of in-memory changes =

In this example, I change two settings, corrupt the file, and then 
change one of them a second time:

$ psql -c "set persistent checkpoint_segments='32'"
$ psql -c "set persistent work_mem='2MB'"
$ cat postgresql.auto.conf
checkpoint_segments = 32
work_mem = 2MB
$ rm postgresql.auto.conf
$ psql -c "set persistent work_mem='2MB'"
SET
$ cat postgresql.auto.conf
work_mem = 2MB
$ psql -xc "show checkpoint_segments"
checkpoint_segments | 3

That this happens isn't unreasonable, and I can live with the 
limitation.  The change to work_mem is lost, but it didn't have to be. 
A user might expect that in this case the last SET PERSISTENT would have 
written out a file with both the settings still intact.  The running GUC 
entries certainly knew that postgresql.auto.conf had two lines with 
these entries at that point.  All in-memory persistent changes could be 
dumped out to postgresql.auto.conf.  That's what I had hoped was 
possible in the implementation.

Amit may have this right though, because I think the code he's using is 
simpler and reliable than what I had in mind.  I'm really not sure which 
way is better.  This one isn't a blocker, and if this gets committed it 
could easily be nudged around later, as an internal change without a 
catversion bump.

= Error messages =

If you remove postgresql.auto.conf and restart the server, it gives a 
warning that SET PERSISTENT won't work until you put it back.  The error 
in this and several similar cases is pretty generic too:

WARNING:  File "postgresql.auto.conf" is not accessible, either file 
"postgresql.auto.conf" or folder "config" doesn't exist. or default 
"config" is not present in postgresql.conf.

It would be nice if the error were more specific, individually 
identifying which of these is the actual problem.  I can rewrite that 
long text entry to be more readable, but I think it should be a series 
of smaller error checks with their own individual messages instead.

If you remove postgresql.auto.conf then exeute "pg_ctl reload", it gives 
this error 6 times, which seems excessive.  Reducing how often it 
appears in the reload case would be nice.

Deleting the whole config directory gives this:

LOG:  could not open configuration directory 
"/Users/gsmith/pgwork/data/autoconf/config": No such file or directory
LOG:  configuration file 
"/Users/gsmith/pgwork/data/autoconf/postgresql.conf" contains errors; no 
changes were applied

If you now try to use the feature, the error message could be better.

$ psql -c "set persistent checkpoint_segments='32'"
ERROR:  Failed to open auto conf temp file 
"/Users/gsmith/pgwork/data/autoconf/config/postgresql.auto.conf.temp": 
No such file or directory

It would be nice to complain about the config directory being missing, 
as a first check before the file is opened.  Restarting the server in 
this situation throws the correct error in your face though:

LOG:  could not open configuration directory 
"/Users/gsmith/pgwork/data/autoconf/config": No such file or directory
FATAL:  configuration file 
"/Users/gsmith/pgwork/data/autoconf/postgresql.conf" contains errors

If you render this whole system unavailable by removing the "include_dir 
config", at every server start you'll see this:

WARNING:  File "postgresql.auto.conf" is not accessible, either file 
"postgresql.auto.conf" or folder "config" doesn't exist. or default 
"config" is not present in postgresql.conf.
	Configuration parameters changed before start/reload of server with SET 
PERSISTENT command will not be effective.
LOG:  database system was shut down at 2013-03-09 23:55:03 EST

This is a debatable design choice.  Some people might not want the file 
and remove it, but don't want to be nagged about it.  If people want to 
wipe out the file or directory and work the old way, without this 
feature available, that's fine and they can.  To me, helping users who 
accidentally break this is more important than reducing harmless warning 
messages for things people did intentionally.  WARNING might not be the 
right level for this though.  The existing checks like this I showed 
above use LOG for this sort of thing.

The bigger problem is that this message shows up whenever you reload the 
config too.  Watch this bizarre sequence:

gsmith=# select pg_reload_conf();
  pg_reload_conf
----------------
  t
(1 row)

gsmith=#
gsmith=# select 1;
WARNING:  File "postgresql.auto.conf" is not accessible, either file 
"postgresql.auto.conf" or folder "config" doesn't exist. or default 
"config" is not present in postgresql.conf.
Configuration parameters changed before start/reload of server with SET 
PERSISTENT command will not be effective.
  ?column?
----------
         1

And as I commented above, shifting more of the checks to SET PERISTENT 
time could eliminate this check from running at server start and reload 
altogether.  I would be fine with them *also* happening at server start. 
  But I could understand that other people might not like that.  And 
having this pop up on every reload, appearing to a client next to 
another statement altogether, that isn't acceptable though.

-- 
Greg Smith   2ndQuadrant US    greg@2ndQuadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Craig Ringer <craig@2ndquadrant.com> — 2013-03-11T00:02:13Z

On 03/10/2013 11:12 PM, Greg Smith wrote:
>
> 5) An error message appears every time you reload configuration, if
> some part of the SET PERSISTENT mechanism isn't functional.  This is
> going to be too much for some people.  And it's confusing when it
> happens as part an interactive psql session.  I have an example way
> down at the very end of this message. 
An error when `SET PERSISTENT` is actually run seems sufficient to me if
the `config` dir is absent.

If the config dir or auto config file exist but are not included, that
probably deserves a warning during startup.

-- 
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Amit Kapila <amit.kapila@huawei.com> — 2013-03-11T06:48:10Z

On Sunday, March 10, 2013 8:43 PM Greg Smith wrote:
> On 3/7/13 2:42 AM, Amit Kapila wrote:
> > I also think the tests added for regression may be more than
> required...
> > If you think above optimization's to reduce number of tests are okay,
> then I
> > will update the patch.
> 
> I was not trying to get you to remove regression tests.  I was just
> pointing out to everyone that the patch seems longer than it really is,
> because you put so many of them in there.  This is not a problem, and I
> didn't bring it up to say you should remove some of them.
> 
> This feature is close to being functional enough to make a lot of
> people
> happy.  Most of it works the way I've wanted it to for a few years now,
> an end point several people have pushed toward in pieces for years now.
>   There is a decent sized list of issues that I found under careful
> testing though.  Maybe all these can be resolved quickly and this moves
> on to commit candidate.  I'd like to see this feature hit the code
> base,
> but it feels like a good amount of work is left to reach there to me
> right now.

Thank you for a careful and detail testing.

> If Amit can address the functional ones, I can tweak the text/level of
> the log messages myself during the cleanup round I do.

Sure, I would address the functional issues.
 
> = Functional changes =
> 
> There are a number of individually small but serious blocker items I
> think need to be changed in how this works.  I see 7 functional changes
> to be made before there's no surprising behavior here.  I have a lot of
> commentary about how the log/error messages from this might be improved
> in addition to that.
> 
> 1) When you change a sighup or user setting, it writes the config file
> out.  But it does not signal for a reload.  Example:
> 
> $ psql -c "show work_mem" -x
> work_mem | 1MB
> $ psql -c "set persistent work_mem='2MB'"
> SET
> $ psql -c "show work_mem" -x
> work_mem | 1MB
> 
> That the above doesn't leave work_mem set to 2MB for new sessions is
> surprising.  SET PERSISTENT can't do anything about postmaster
> parameters like shared_buffers.  But it should be able to change
> work_mem or a sighup parameter like checkpoint_segments.  The
> regression
> test examples call pg_reload_conf() in order to activate the changes.
> As a user, I would expect the config reload to be automatic after
> executing SET PERSISTENT on parameters that can be changed dynamically.

I think we cannot guarantee even after calling pg_reload_conf(), as this is
an
asynchronous function call. 
We already once had a discussion about this point and as I can understand it
is 
concluded that it should not be handled. Refer the below mail:
http://www.postgresql.org/message-id/21869.1360683928@sss.pgh.pa.us


> 2) Once automatic activation is happening, a hint is really needed in
> cases where it can't happen.  I'd like to see a warning about that.
> There's one like this in the code already:
> 
> LOG:  parameter "shared_buffers" cannot be changed without restarting
> the server
> 
> Something like this would be appropriate for SET PERSISTENT:
> 
> LOG:  parameter "shared_buffers" cannot be changed without restarting
> the server.  The changed setting has been saved to
> config/postgresql.auto.conf.
> 
> 3) The config/postgresql.auto.conf file (which I'd like to see named
> 'config/persistent.auto.conf') is completely overwritten when a change
> is made.  When the file is written out, both at initdb time and when
> changes are made, it should have this as its first line:
> 
> # Do not edit this file manually!  It is overwritten by the SET
> PERSISTENT command.

2 things, you want as part of this comment:
a. change the name of .auto file to persistent.auto.conf
b. new comment in beginning of persistent.auto.conf file.

> 4) There is one bad problem case left if I try to make this not work.
> If I do this:
> 
> -rm -rf config/
> -Remove "include_dir config"
> -Restart the server.  It gives me a warning that SET PERSISTENT won't
> work because at least one of the multiple things it needs are missing.
> -mkdir config
> -psql -c "set persistent checkpoint_segments='32'"
> 
> That command happily writes out postgresql.auto.conf, but since I
> removed the include_dir it doesn't matter.  It will never become
> active.
> 
> The check for the include_dir entry needs to happen each time SET
> PERSISTENT runs.  This is more important than noticing it at server
> start time.  In fact, I think if this problem case includes the WARNING
> about "include_dir config" not being in the postgresql.conf, the check
> that happens at server start (shown below) might even go away.  People
> who object to it as log file garbage will be happier, and users will be
> told about the feature being broken if they try to use it.

This can be handled, but for this we need to parse the whole postgresql.conf
file
and then give the warning. This will increase the time of SET PERSISTENT
command.

Or the other way might be to note down the information about include dir at
startup and then use it, 
but implementation can get complicated for scenarios if somebody change the
postgresql.conf and reload it.

I will check whichever way is less complicated, I will handle it that way.

> 5) An error message appears every time you reload configuration, if
> some
> part of the SET PERSISTENT mechanism isn't functional.  This is going
> to
> be too much for some people.  And it's confusing when it happens as
> part
> an interactive psql session.  I have an example way down at the very
> end
> of this message.
> 
> 6) Putting spaces into memory units results in broken config files:
> 
> $ psql -c "set persistent work_mem='2 MB'"
> SET
> $ cat $PGDATA/config/postgresql.auto.conf
> work_mem = 2 MB
> 
> This is wrong and you'll get this on reload:
> 
> LOG:  syntax error in file
> "/Users/gsmith/pgwork/data/autoconf/config/postgresql.auto.conf" line
> 1,
> near token "MB"
> LOG:  configuration file
> "/Users/gsmith/pgwork/data/autoconf/postgresql.conf" contains errors;
> no
> changes were applied
> 
> It needs to look like this:
> 
> work_mem = '2 MB'
> 
> A regression test for some cases with spaces in the middle should be
> added too.  This case is really weird, due to how the code is reading
> the existing postgresql.auto.conf file.  If I first manually fix the
> file, and then run a change again, it stays fixed!  Look at this weird
> sequence:
> 
> $ psql -c "set persistent work_mem='2 MB'"
> SET
> $ cat $PGDATA/config/postgresql.auto.conf
> work_mem = 2 MB
> [Here this setting is broken]
> $ vi $PGDATA/config/postgresql.auto.conf
> [Add quotes around the value, now it works]
> $ cat $PGDATA/config/postgresql.auto.conf
> work_mem = '2 MB'
> $ pg_ctl reload
> server signaled
> $ psql -c "set persistent work_mem='4 MB'"
> SET
> $ cat $PGDATA/config/postgresql.auto.conf
> work_mem = '4 MB'

I shall fix this behavior.

> Below I'll rant some more about how looking at what's in the existing
> file, rather than the in-memory GUCs, has some other surprising
> properties.
> 
> 7) If I run SET PERSISTENT a lot concurrently, something happens that
> slows down connecting to the server.  Restarting the server makes it go
> away.  I have a pgbench test case demonstrating the problem below, in
> the "Concurrency" section.  I haven't tried to replicate it on another
> system yet.
> 
> = Tested features that work fine =
> 
> Entries added here are tracked as you'd expect:
> 
> $ psql -c "set persistent checkpoint_segments='32'"
> $ pg_ctl reload
> $ psql -xc "select name,setting,sourcefile,sourceline from pg_settings
> where name='checkpoint_segments'"
> name       | checkpoint_segments
> setting    | 32
> sourcefile |
> /Users/gsmith/pgwork/data/autoconf/config/postgresql.auto.conf
> sourceline | 1
> 
> When the postgresql.auto.conf file is missing and you use SET
> PERSISTENT, it quietly creates it when writing out the new setting.
> 
> = Concurrency and performance slowdown =
> 
> I made two pgbench scripts that adjust a guc, one user and the other
> sighup, to a random value:
> 
> random-join-limit.sql:
> 
> \setrandom limit 1 8
> set persistent join_collapse_limit=:limit;
> select pg_reload_conf();
> 
> random-segments.sql:
> 
> \setrandom segments 3 16
> set persistent checkpoint_segments=:segments;
> select pg_reload_conf();
> 
> I then fired off a bunch of these in parallel:
> 
> pgbench -n -f random-join-limit.sql -f random-segments.sql -c 8 -T 60
> 
> This ran at 1547 TPS on my 8 core Mac, so that's not bad.  No
> assertions
> and the config file was still valid at the end, which is a good sign
> the
> locking method isn't allowing utter chaos.  Without the
> pg_reload_conf()
> in the test files, it also completed.  With the reload happening in one
> file but not the other, things were also fine.
> 
> However, one thing I saw is that the server got significantly slower
> the
> more I ran this test script.  After a few minutes it was down to only
> 400 TPS.  The delay shows up between when I run psql and when I get the
> prompt back.  Here's normal behavior:
> 
> $ time psql -c "select 1"
> real	0m0.006s
> 
> And here's what I get after a single run of this pgbench hammering:
> 
> $ time psql -c "select 1"
> real	0m0.800s
> 	
> 800ms?  The slowdown is all for psql to start and connect, it's not in
> the executor:
> 
> $ time psql -c "explain analyze select 1"
>                                       QUERY PLAN
> -----------------------------------------------------------------------
> -------------
>   Result  (cost=0.00..0.01 rows=1 width=0) (actual time=0.001..0.001
> rows=1 loops=1)
>   Total runtime: 0.039 ms
> (2 rows)
> 
> real	0m0.807s
> 
> Stopping and restarting the server brings the performance back to
> normal.  Maybe this is one of those assertion build issues, but it
> smells like there could be nasty memory leak somewhere instead.
> Clearing whatever weirdness is going on here is a blocking issue.

Okay, I shall look into this issue.

> = Loss of in-memory changes =
> 
> In this example, I change two settings, corrupt the file, and then
> change one of them a second time:
> 
> $ psql -c "set persistent checkpoint_segments='32'"
> $ psql -c "set persistent work_mem='2MB'"
> $ cat postgresql.auto.conf
> checkpoint_segments = 32
> work_mem = 2MB
> $ rm postgresql.auto.conf
> $ psql -c "set persistent work_mem='2MB'"
> SET
> $ cat postgresql.auto.conf
> work_mem = 2MB
> $ psql -xc "show checkpoint_segments"
> checkpoint_segments | 3
> 
> That this happens isn't unreasonable, and I can live with the
> limitation.  The change to work_mem is lost, but it didn't have to be.
> A user might expect that in this case the last SET PERSISTENT would
> have
> written out a file with both the settings still intact.  The running
> GUC
> entries certainly knew that postgresql.auto.conf had two lines with
> these entries at that point.  All in-memory persistent changes could be
> dumped out to postgresql.auto.conf.  That's what I had hoped was
> possible in the implementation.
> 
> Amit may have this right though, because I think the code he's using is
> simpler and reliable than what I had in mind.  I'm really not sure
> which
> way is better.  This one isn't a blocker, and if this gets committed it
> could easily be nudged around later, as an internal change without a
> catversion bump.

In-memory values are not guaranteed after running SET PERSISTENT, even
though we
do pg_reload_conf(), so IMO we should leave this behavior as it is.

> = Error messages =
> 
> If you remove postgresql.auto.conf and restart the server, it gives a
> warning that SET PERSISTENT won't work until you put it back.  The
> error
> in this and several similar cases is pretty generic too:
> 
> WARNING:  File "postgresql.auto.conf" is not accessible, either file
> "postgresql.auto.conf" or folder "config" doesn't exist. or default
> "config" is not present in postgresql.conf.
> 
> It would be nice if the error were more specific, individually
> identifying which of these is the actual problem.  I can rewrite that
> long text entry to be more readable, but I think it should be a series
> of smaller error checks with their own individual messages instead.

You are right, we should change this by identifying actual problems.

> If you remove postgresql.auto.conf then exeute "pg_ctl reload", it
> gives
> this error 6 times, which seems excessive.  Reducing how often it
> appears in the reload case would be nice.

> Deleting the whole config directory gives this:
> 
> LOG:  could not open configuration directory
> "/Users/gsmith/pgwork/data/autoconf/config": No such file or directory
> LOG:  configuration file
> "/Users/gsmith/pgwork/data/autoconf/postgresql.conf" contains errors;
> no
> changes were applied
> 
> If you now try to use the feature, the error message could be better.
> 
> $ psql -c "set persistent checkpoint_segments='32'"
> ERROR:  Failed to open auto conf temp file
> "/Users/gsmith/pgwork/data/autoconf/config/postgresql.auto.conf.temp":
> No such file or directory
> 
> It would be nice to complain about the config directory being missing,
> as a first check before the file is opened.  Restarting the server in
> this situation throws the correct error in your face though:
> 
> LOG:  could not open configuration directory
> "/Users/gsmith/pgwork/data/autoconf/config": No such file or directory
> FATAL:  configuration file
> "/Users/gsmith/pgwork/data/autoconf/postgresql.conf" contains errors
> 
> If you render this whole system unavailable by removing the
> "include_dir
> config", at every server start you'll see this:
> 
> WARNING:  File "postgresql.auto.conf" is not accessible, either file
> "postgresql.auto.conf" or folder "config" doesn't exist. or default
> "config" is not present in postgresql.conf.
> 	Configuration parameters changed before start/reload of server
> with SET
> PERSISTENT command will not be effective.
> LOG:  database system was shut down at 2013-03-09 23:55:03 EST
> 
> This is a debatable design choice.  Some people might not want the file
> and remove it, but don't want to be nagged about it.  If people want to
> wipe out the file or directory and work the old way, without this
> feature available, that's fine and they can.  To me, helping users who
> accidentally break this is more important than reducing harmless
> warning
> messages for things people did intentionally.  WARNING might not be the
> right level for this though.  The existing checks like this I showed
> above use LOG for this sort of thing.
> 
> The bigger problem is that this message shows up whenever you reload
> the
> config too.  Watch this bizarre sequence:
> 
> gsmith=# select pg_reload_conf();
>   pg_reload_conf
> ----------------
>   t
> (1 row)
> 
> gsmith=#
> gsmith=# select 1;
> WARNING:  File "postgresql.auto.conf" is not accessible, either file
> "postgresql.auto.conf" or folder "config" doesn't exist. or default
> "config" is not present in postgresql.conf.
> Configuration parameters changed before start/reload of server with SET
> PERSISTENT command will not be effective.
>   ?column?
> ----------
>          1
> 
> And as I commented above, shifting more of the checks to SET PERISTENT
> time could eliminate this check from running at server start and reload
> altogether.  I would be fine with them *also* happening at server
> start.
>   But I could understand that other people might not like that.  And
> having this pop up on every reload, appearing to a client next to
> another statement altogether, that isn't acceptable though.

Agreed, I will add the check at time of SET PERSISTENT command.
Also I will try to remove this message at time of reload.


With Regards,
Amit Kapila.

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Greg Smith <greg@2ndquadrant.com> — 2013-03-11T07:39:58Z

On 3/11/13 2:48 AM, Amit Kapila wrote:
>> 1) When you change a sighup or user setting, it writes the config file
>> out.  But it does not signal for a reload.  Example:
>
> I think we cannot guarantee even after calling pg_reload_conf(), as this is
> an
> asynchronous function call.
> We already once had a discussion about this point and as I can understand it
> is
> concluded that it should not be handled. Refer the below mail:
> http://www.postgresql.org/message-id/21869.1360683928@sss.pgh.pa.us

I wasn't complaining that the change isn't instant.  I understand that 
can't be done.  But I think the signal to reload should be sent.  If 
people execute SET PERSISTENT, and it doesn't actually do anything until 
the server is next restarted, they will be very surprised.  It's OK if 
it doesn't do anything for a second, or until new sessions connect, 
because that's just how SIGHUP/session variables work.  That's a 
documentation issue.  Not reloading the config at all, I think that's 
going to trigger a ton of future support problems.

> 2 things, you want as part of this comment:
> a. change the name of .auto file to persistent.auto.conf
> b. new comment in beginning of persistent.auto.conf file.

Right.  config/persistent.auto.conf seems like it addresses everyone's 
ideas on how the file should be named.  If there's a warning not to edit 
it in the file, too, it would be making obvious to users what is 
happening.  The file handles persistent changes, it's automatically 
configured, and it's a config file.  Putting "postgresql" in the name 
seems unnecessary to me.

>> The check for the include_dir entry needs to happen each time SET
>> PERSISTENT runs.
> This can be handled, but for this we need to parse the whole postgresql.conf
> file
> and then give the warning. This will increase the time of SET PERSISTENT
> command.

I don't think that's a problem.  I can run the command 1800 times per 
second on my laptop right now.  If it could only run once per second, 
that would still be fast enough.  Doing this the most reliable way it 
can be handled is the important part; if that happens to be the slowest 
way, too, that is acceptable.

-- 
Greg Smith   2ndQuadrant US    greg@2ndQuadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Fujii Masao <masao.fujii@gmail.com> — 2013-03-11T17:32:04Z

On Mon, Mar 11, 2013 at 4:39 PM, Greg Smith <greg@2ndquadrant.com> wrote:
> On 3/11/13 2:48 AM, Amit Kapila wrote:
>>>
>>> 1) When you change a sighup or user setting, it writes the config file
>>> out.  But it does not signal for a reload.  Example:
>>
>>
>> I think we cannot guarantee even after calling pg_reload_conf(), as this
>> is
>> an
>> asynchronous function call.
>> We already once had a discussion about this point and as I can understand
>> it
>> is
>> concluded that it should not be handled. Refer the below mail:
>> http://www.postgresql.org/message-id/21869.1360683928@sss.pgh.pa.us
>
>
> I wasn't complaining that the change isn't instant.  I understand that can't
> be done.  But I think the signal to reload should be sent.  If people
> execute SET PERSISTENT, and it doesn't actually do anything until the server
> is next restarted, they will be very surprised.  It's OK if it doesn't do
> anything for a second, or until new sessions connect, because that's just
> how SIGHUP/session variables work.  That's a documentation issue.  Not
> reloading the config at all, I think that's going to trigger a ton of future
> support problems.

I agree with you if SET PERSISTENT reloads only postgresql.auto.conf.
But in current conf reload mechanism, other configuration files like
pg_hba.conf are also reloaded when pg_read_conf() is executed. Probably
I don't like this behavior. Users might get surprised that the configuration
changes by their manual operation (by not SET PERSISTENT) are also
activated by SET PERSISTENT.

And, this change would make the patch bigger...

>>> The check for the include_dir entry needs to happen each time SET
>>> PERSISTENT runs.
>>
>> This can be handled, but for this we need to parse the whole
>> postgresql.conf
>>
>> file
>> and then give the warning. This will increase the time of SET PERSISTENT
>> command.
>
>
> I don't think that's a problem.  I can run the command 1800 times per second
> on my laptop right now.  If it could only run once per second, that would
> still be fast enough.  Doing this the most reliable way it can be handled is
> the important part; if that happens to be the slowest way, too, that is
> acceptable.

Why should the include_dir entry for SET PERSISTENT exist in postgresql.conf?
Is there any use case that users change that include_dir setting? If not, what
about hardcoding the include_dir entry for SET PERSISTENT in the core?
If we do this, we don't need to handle this problem at all.

Regards,

-- 
Fujii Masao

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Josh Berkus <josh@agliodbs.com> — 2013-03-11T17:52:12Z

> I agree with you if SET PERSISTENT reloads only postgresql.auto.conf.
> But in current conf reload mechanism, other configuration files like
> pg_hba.conf are also reloaded when pg_read_conf() is executed. Probably
> I don't like this behavior. Users might get surprised that the configuration
> changes by their manual operation (by not SET PERSISTENT) are also
> activated by SET PERSISTENT.

I'm not sure I see this as a problem if it's well-documented.  Basically
you're saying that if a user does:

1. make some changes to pg_hba.conf
2. make some changes via SET PERSISTENT
3. pg_hba.conf changes will be automatically loaded

It's a little surprising for some users, but if the behavior is
well-documented, then I really don't see it as a problem.  Consider the
parallel of this:

1. make some changes to postgresql.conf
2. make some changes via SET PERSISTENT
3. manual pg.conf changes are automatically loaded

I don't see this path as a problem ... in fact, I kind of think that as
many users will expect the above as be surprised by it.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Greg Stark <stark@mit.edu> — 2013-03-11T19:19:32Z

On Mon, Mar 11, 2013 at 7:39 AM, Greg Smith <greg@2ndquadrant.com> wrote:

> I wasn't complaining that the change isn't instant.  I understand that can't
> be done.  But I think the signal to reload should be sent.  If people
> execute SET PERSISTENT, and it doesn't actually do anything until the server
> is next restarted, they will be very surprised.  It's OK if it doesn't do
> anything for a second, or until new sessions connect, because that's just
> how SIGHUP/session variables work.  That's a documentation issue.  Not
> reloading the config at all, I think that's going to trigger a ton of future
> support problems.

Think also about the case where someone wants to change multiple
values together and having just some set and not others would be
inconsistent.

I can see you're right about surprising users but is there not another
way to solve the same problem without making that impossible?

-- 
greg

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Craig Ringer <craig@2ndquadrant.com> — 2013-03-11T22:27:55Z

On 03/12/2013 03:19 AM, Greg Stark wrote:
> Think also about the case where someone wants to change multiple
> values together and having just some set and not others would be
> inconsistent.
Yeah, that's a killer. The reload would need to be scheduled for COMMIT
time, it can't be done by `SET PERSISTENT` directly.

-- 
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Amit Kapila <amit.kapila@huawei.com> — 2013-03-12T03:52:27Z

On Monday, March 11, 2013 11:02 PM Fujii Masao wrote:
> On Mon, Mar 11, 2013 at 4:39 PM, Greg Smith <greg@2ndquadrant.com>
> wrote:
> > On 3/11/13 2:48 AM, Amit Kapila wrote:
> >>>
> >>> 1) When you change a sighup or user setting, it writes the config
> file
> >>> out.  But it does not signal for a reload.  Example:
> >>
> >>
> >> I think we cannot guarantee even after calling pg_reload_conf(), as
> this
> >> is
> >> an
> >> asynchronous function call.
> >> We already once had a discussion about this point and as I can
> understand
> >> it
> >> is
> >> concluded that it should not be handled. Refer the below mail:
> >> http://www.postgresql.org/message-id/21869.1360683928@sss.pgh.pa.us
> >
> >
> > I wasn't complaining that the change isn't instant.  I understand
> that can't
> > be done.  But I think the signal to reload should be sent.  If people
> > execute SET PERSISTENT, and it doesn't actually do anything until the
> server
> > is next restarted, they will be very surprised.  It's OK if it
> doesn't do
> > anything for a second, or until new sessions connect, because that's
> just
> > how SIGHUP/session variables work.  That's a documentation issue.
> Not
> > reloading the config at all, I think that's going to trigger a ton of
> future
> > support problems.
> 
> I agree with you if SET PERSISTENT reloads only postgresql.auto.conf.
> But in current conf reload mechanism, other configuration files like
> pg_hba.conf are also reloaded when pg_read_conf() is executed. Probably
> I don't like this behavior. Users might get surprised that the
> configuration
> changes by their manual operation (by not SET PERSISTENT) are also
> activated by SET PERSISTENT.
> 
> And, this change would make the patch bigger...

it's more of issue in consistency as pg_reload_conf() is asynchronous.
 
> >>> The check for the include_dir entry needs to happen each time SET
> >>> PERSISTENT runs.
> >>
> >> This can be handled, but for this we need to parse the whole
> >> postgresql.conf
> >>
> >> file
> >> and then give the warning. This will increase the time of SET
> PERSISTENT
> >> command.
> >
> >
> > I don't think that's a problem.  I can run the command 1800 times per
> second
> > on my laptop right now.  If it could only run once per second, that
> would
> > still be fast enough.  Doing this the most reliable way it can be
> handled is
> > the important part; if that happens to be the slowest way, too, that
> is
> > acceptable.
> 
> Why should the include_dir entry for SET PERSISTENT exist in
> postgresql.conf?
> Is there any use case that users change that include_dir setting? If
> not, what
> about hardcoding the include_dir entry for SET PERSISTENT in the core?
> If we do this, we don't need to handle this problem at all.

It will be for users who want to use the configuration settings as per
previous version, means by manually 
setting in conf file.

With Regards,
Amit Kapila.

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Amit Kapila <amit.kapila@huawei.com> — 2013-03-12T05:16:53Z

> From: gsstark@gmail.com [mailto:gsstark@gmail.com] On Behalf Of Greg
> Stark
> Sent: Tuesday, March 12, 2013 12:50 AM
> To: Greg Smith
> Cc: Amit Kapila; Andres Freund; Boszormenyi Zoltan; pgsql-
> hackers@postgresql.org
> Subject: Re: Re: Proposal for Allow postgresql.conf values to be
> changed via SQL [review]
> 
> On Mon, Mar 11, 2013 at 7:39 AM, Greg Smith <greg@2ndquadrant.com>
> wrote:
> 
> > I wasn't complaining that the change isn't instant.  I understand
> that can't
> > be done.  But I think the signal to reload should be sent.  If people
> > execute SET PERSISTENT, and it doesn't actually do anything until the
> server
> > is next restarted, they will be very surprised.  It's OK if it
> doesn't do
> > anything for a second, or until new sessions connect, because that's
> just
> > how SIGHUP/session variables work.  That's a documentation issue.
> Not
> > reloading the config at all, I think that's going to trigger a ton of
> future
> > support problems.
> 
> Think also about the case where someone wants to change multiple
> values together and having just some set and not others would be
> inconsistent.

Do you mean to say that because some variables can only be set after restart
can lead to 
inconsistency, or is it because of asynchronous nature of pg_reload_conf()?

> I can see you're right about surprising users but is there not another
> way to solve the same problem without making that impossible?


With Regards,
Amit Kapila.

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Greg Smith <greg@2ndquadrant.com> — 2013-03-12T09:06:36Z

On 3/11/13 12:19 PM, Greg Stark wrote:
> Think also about the case where someone wants to change multiple
> values together and having just some set and not others would be
> inconsistent.

Isn't that an argument for syntax to make an exception though?  If 
starting from a blank slate I would say this case should be:

SET PERSISTENT (POSTPONE) maintenance_work_mem='2GB';
SET PERSISTENT work_mem='2GB';

That's jumping right over a few rounds of simpler ways to do this, and 
just going right to the approach we know allows adding more such options 
later with minimal grammar impact.

 From the perspective of what a useful minimal commit looks like, I 
would think 90% of the use cases here expect immediate reload of just 
the postgresql.conf--but probably not pg_hba.conf--and any other 
refinement could wait until a next release.

-- 
Greg Smith   2ndQuadrant US    greg@2ndQuadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Greg Stark <stark@mit.edu> — 2013-03-12T15:36:44Z

On Tue, Mar 12, 2013 at 9:06 AM, Greg Smith <greg@2ndquadrant.com> wrote:
> That's jumping right over a few rounds of simpler ways to do this, and just
> going right to the approach we know allows adding more such options later
> with minimal grammar impact.

As Craig intimated, the minimal grammar impact would be simply

BEGIN;
set persistent maintenance_work_mem='2GB';
set persistent work_mem='2GB';
COMMIT;

Sending the sighup at transaction end seems like a fairly safe thing
to do too. It's hard to imagine it failing and if it did the worst
case would be that other backends would still have the old values too.

To be honest I went through the gucs last night looking for a better
example than this and didn't really find any compelling examples. The
best I could come up with was adjusting several query costs in tandem.
Hardly the end of the world if it didn't work. The worst case would be
some security sensitive option alongside a second parameter for
limiting its scope, but I couldn't find any such examples in a quick
search. (And that's probably a good thing)

-- 
greg

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Tom Lane <tgl@sss.pgh.pa.us> — 2013-03-12T16:07:16Z

Greg Stark <stark@mit.edu> writes:
> As Craig intimated, the minimal grammar impact would be simply

> BEGIN;
> set persistent maintenance_work_mem='2GB';
> set persistent work_mem='2GB';
> COMMIT;

> Sending the sighup at transaction end seems like a fairly safe thing
> to do too. It's hard to imagine it failing and if it did the worst
> case would be that other backends would still have the old values too.

This would only be sane if we also postponed writing the file until
commit.  I don't know what is in the patch at the moment, but I remember
that we had talked of executing the file write immediately when SET
PERSISTENT is issued.

Maybe it's worth going in that direction just to cover the
multiple-updates case more conveniently.

			regards, tom lane

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Amit Kapila <amit.kapila@huawei.com> — 2013-03-13T03:14:02Z

On Tuesday, March 12, 2013 9:37 PM Tom Lane wrote:
> Greg Stark <stark@mit.edu> writes:
> > As Craig intimated, the minimal grammar impact would be simply
> 
> > BEGIN;
> > set persistent maintenance_work_mem='2GB';
> > set persistent work_mem='2GB';
> > COMMIT;
> 
> > Sending the sighup at transaction end seems like a fairly safe thing
> > to do too. It's hard to imagine it failing and if it did the worst
> > case would be that other backends would still have the old values
> too.
> 
> This would only be sane if we also postponed writing the file until
> commit.  I don't know what is in the patch at the moment, but I
> remember
> that we had talked of executing the file write immediately when SET
> PERSISTENT is issued.

Currently the patch handles it by executing the file write immediately when
SET
PERSISTENT is issued.
You have pointed at that time it's better to write file immediately as if we
postpone
it to commit time, it can have some other complications.

The only doubt I have in doing pg_reload_conf() is about the point raised by
Fujii Masao yesterday,
that is it will load all other configurations apart from what had been
changed by SET PERSISTENT.
Are you okay with it, if we implement this and document such behavior
clearly?

With Regards,
Amit Kapila.

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Craig Ringer <craig@2ndquadrant.com> — 2013-03-13T06:42:18Z

On 03/12/2013 06:27 AM, Craig Ringer wrote:
>> > Think also about the case where someone wants to change multiple
>> > values together and having just some set and not others would be
>> > inconsistent.
> Yeah, that's a killer. The reload would need to be scheduled for COMMIT
> time, it can't be done by `SET PERSISTENT` directly.
Thinking about this some more, I'm not sure this is a good idea.

Right now, SET takes effect immediately. Always, without exception. 
Delaying SET PERSISTENT's effects until commit would make it
inconsistent with SET's normal behaviour.

However, *not* delaying it would make it another quirky
not-transactional not-atomic command. That's OK, but if it's not going
to follow transactional semantics it should not be allowed to run within
a transaction, like VACUUM .

Writing the changes immediately but deferring the reload until commit
seems to be the worst of those two worlds.

-- 
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Boszormenyi Zoltan <zb@cybertec.at> — 2013-03-13T08:09:24Z

2013-03-13 07:42 keltezéssel, Craig Ringer írta:
> On 03/12/2013 06:27 AM, Craig Ringer wrote:
>>>> Think also about the case where someone wants to change multiple
>>>> values together and having just some set and not others would be
>>>> inconsistent.
>> Yeah, that's a killer. The reload would need to be scheduled for COMMIT
>> time, it can't be done by `SET PERSISTENT` directly.
> Thinking about this some more, I'm not sure this is a good idea.
>
> Right now, SET takes effect immediately. Always, without exception.
> Delaying SET PERSISTENT's effects until commit would make it
> inconsistent with SET's normal behaviour.
>
> However, *not* delaying it would make it another quirky
> not-transactional not-atomic command. That's OK, but if it's not going
> to follow transactional semantics it should not be allowed to run within
> a transaction, like VACUUM .
>
> Writing the changes immediately but deferring the reload until commit
> seems to be the worst of those two worlds.

I was thinking about it a little. There is a hook that runs at the end
of (sub-)transactions. It can be abused for this purpose to make
SET PERSISTENT transactional. The subtransactions can also stack
these settings, by forgetting settings upon ROLLBACK [ TO SAVEPOINT ]
and overwriting previous settings upon COMMIT and RELEASE SAVEPOINT.
All it needs is a list and maintaining intermediate pointers when entering
into a new level of SAVEPOINT. The functions to register such hooks are
in src/include/access/xact.h:

extern void RegisterXactCallback(XactCallback callback, void *arg);
extern void UnregisterXactCallback(XactCallback callback, void *arg);
extern void RegisterSubXactCallback(SubXactCallback callback, void *arg);
extern void UnregisterSubXactCallback(SubXactCallback callback, void *arg);

The per-session local effect can also be immediate, it just needs
the normal SET and SET PERSISTENT code paths to be unified.

On the other hand, it would make a lot of sense to implement it
as one setting per file or extending the  code to allow modifying
settings in bulk. The one setting per file should be easier and
it would also allow extensions to drop their settings automatically
into the automatic config directory. I don't know who mentioned
this idea about extensions but it also came up a while back.

Best regards,
Zoltán Böszörményi

-- 
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de
      http://www.postgresql.at/

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Boszormenyi Zoltan <zb@cybertec.at> — 2013-03-13T09:31:53Z

2013-03-13 09:09 keltezéssel, Boszormenyi Zoltan írta:
> 2013-03-13 07:42 keltezéssel, Craig Ringer írta:
>> On 03/12/2013 06:27 AM, Craig Ringer wrote:
>>>>> Think also about the case where someone wants to change multiple
>>>>> values together and having just some set and not others would be
>>>>> inconsistent.
>>> Yeah, that's a killer. The reload would need to be scheduled for COMMIT
>>> time, it can't be done by `SET PERSISTENT` directly.
>> Thinking about this some more, I'm not sure this is a good idea.
>>
>> Right now, SET takes effect immediately. Always, without exception.
>> Delaying SET PERSISTENT's effects until commit would make it
>> inconsistent with SET's normal behaviour.
>>
>> However, *not* delaying it would make it another quirky
>> not-transactional not-atomic command. That's OK, but if it's not going
>> to follow transactional semantics it should not be allowed to run within
>> a transaction, like VACUUM .
>>
>> Writing the changes immediately but deferring the reload until commit
>> seems to be the worst of those two worlds.
>
> I was thinking about it a little. There is a hook that runs at the end
> of (sub-)transactions. It can be abused for this purpose to make
> SET PERSISTENT transactional. The subtransactions can also stack
> these settings, by forgetting settings upon ROLLBACK [ TO SAVEPOINT ]
> and overwriting previous settings upon COMMIT and RELEASE SAVEPOINT.

By "overwriting at RELEASE SAVEPOINT" I meant only the memory copy
so the value that was set last is remembered at the same level of subxact.
The final settings make it to the configuration file at COMMIT time.

> All it needs is a list and maintaining intermediate pointers when entering
> into a new level of SAVEPOINT. The functions to register such hooks are
> in src/include/access/xact.h:
>
> extern void RegisterXactCallback(XactCallback callback, void *arg);
> extern void UnregisterXactCallback(XactCallback callback, void *arg);
> extern void RegisterSubXactCallback(SubXactCallback callback, void *arg);
> extern void UnregisterSubXactCallback(SubXactCallback callback, void *arg);
>
> The per-session local effect can also be immediate, it just needs
> the normal SET and SET PERSISTENT code paths to be unified.
>
> On the other hand, it would make a lot of sense to implement it
> as one setting per file or extending the  code to allow modifying
> settings in bulk. The one setting per file should be easier and
> it would also allow extensions to drop their settings automatically
> into the automatic config directory. I don't know who mentioned
> this idea about extensions but it also came up a while back.
>
> Best regards,
> Zoltán Böszörményi
>


-- 
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de
      http://www.postgresql.at/

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Andres Freund <andres@2ndquadrant.com> — 2013-03-13T12:39:41Z

On 2013-03-12 10:46:53 +0530, Amit Kapila wrote:
> Do you mean to say that because some variables can only be set after restart
> can lead to 
> inconsistency, or is it because of asynchronous nature of pg_reload_conf()?

As long as SET PERSISTENT cannot be executed inside a transaction - or
only takes effect after its end - there doesn't seem to be any problem
executing ProcessConfigFile() directly.

The reason its not executed directly is that it is normally called
during query execution and it wouldn't be nice changing stuff after half
of a query has been processed.

Greetings,

Andres Freund

-- 
 Andres Freund	                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Andres Freund <andres@2ndquadrant.com> — 2013-03-13T12:45:24Z

On 2013-03-13 09:09:24 +0100, Boszormenyi Zoltan wrote:
> 2013-03-13 07:42 keltezéssel, Craig Ringer írta:
> >On 03/12/2013 06:27 AM, Craig Ringer wrote:
> >>>>Think also about the case where someone wants to change multiple
> >>>>values together and having just some set and not others would be
> >>>>inconsistent.
> >>Yeah, that's a killer. The reload would need to be scheduled for COMMIT
> >>time, it can't be done by `SET PERSISTENT` directly.
> >Thinking about this some more, I'm not sure this is a good idea.
> >
> >Right now, SET takes effect immediately. Always, without exception.
> >Delaying SET PERSISTENT's effects until commit would make it
> >inconsistent with SET's normal behaviour.
> >
> >However, *not* delaying it would make it another quirky
> >not-transactional not-atomic command. That's OK, but if it's not going
> >to follow transactional semantics it should not be allowed to run within
> >a transaction, like VACUUM .
> >
> >Writing the changes immediately but deferring the reload until commit
> >seems to be the worst of those two worlds.
> 
> I was thinking about it a little. There is a hook that runs at the end
> of (sub-)transactions. It can be abused for this purpose to make
> SET PERSISTENT transactional. The subtransactions can also stack
> these settings, by forgetting settings upon ROLLBACK [ TO SAVEPOINT ]
> and overwriting previous settings upon COMMIT and RELEASE SAVEPOINT.
> All it needs is a list and maintaining intermediate pointers when entering
> into a new level of SAVEPOINT. The functions to register such hooks are
> in src/include/access/xact.h:
> 
> extern void RegisterXactCallback(XactCallback callback, void *arg);
> extern void UnregisterXactCallback(XactCallback callback, void *arg);
> extern void RegisterSubXactCallback(SubXactCallback callback, void *arg);
> extern void UnregisterSubXactCallback(SubXactCallback callback, void *arg);

(sub)xact commit/abort already calls AtEOXact_GUC(commit, nestlevel). So you
wouldn't even need that. It seems we could add another value to enum
GucStackState, like GUC_SET_PERSISTENT - and process those only if commit &&
nestlevel == 1.
Everytime you see one with commit && nestlevel > 1 you put them into them into
the stack one level up.

This seems like its somewhat in line with the way SET LOCAL is implemented?

> On the other hand, it would make a lot of sense to implement it
> as one setting per file or extending the  code to allow modifying
> settings in bulk. The one setting per file should be easier and
> it would also allow extensions to drop their settings automatically
> into the automatic config directory. I don't know who mentioned
> this idea about extensions but it also came up a while back.

I still think one-setting-per-file is the right way to go, yes.

Greetings,

Andres Freund

-- 
 Andres Freund	                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Amit Kapila <amit.kapila@huawei.com> — 2013-03-13T13:08:12Z

On Wednesday, March 13, 2013 6:10 PM Andres Freund wrote:
> On 2013-03-12 10:46:53 +0530, Amit Kapila wrote:
> > Do you mean to say that because some variables can only be set after
> restart
> > can lead to
> > inconsistency, or is it because of asynchronous nature of
> pg_reload_conf()?
> 
> As long as SET PERSISTENT cannot be executed inside a transaction - or
> only takes effect after its end - there doesn't seem to be any problem
> executing ProcessConfigFile() directly.

Do you mean to say we call directly ProcessConfigFile() at end of SET
PERSISTENT instead 
Of pg_reload_conf() but in that case would it load the variables for other
backends?


With Regards,
Amit Kapila.

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Andres Freund <andres@2ndquadrant.com> — 2013-03-13T13:13:40Z

On 2013-03-13 18:38:12 +0530, Amit Kapila wrote:
> On Wednesday, March 13, 2013 6:10 PM Andres Freund wrote:
> > On 2013-03-12 10:46:53 +0530, Amit Kapila wrote:
> > > Do you mean to say that because some variables can only be set after
> > restart
> > > can lead to
> > > inconsistency, or is it because of asynchronous nature of
> > pg_reload_conf()?
> > 
> > As long as SET PERSISTENT cannot be executed inside a transaction - or
> > only takes effect after its end - there doesn't seem to be any problem
> > executing ProcessConfigFile() directly.
> 
> Do you mean to say we call directly ProcessConfigFile() at end of SET
> PERSISTENT instead 
> Of pg_reload_conf() but in that case would it load the variables for other
> backends?

I'd say do both. Yes, we would evaluate config potentially twice. Who
cares. Messages inside non-postmaster environments are only output at DEBUG2
anyway.

Greetings,

Andres Freund

-- 
 Andres Freund	                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Amit Kapila <amit.kapila@huawei.com> — 2013-03-13T13:22:48Z

On Wednesday, March 13, 2013 6:44 PM Andres Freund wrote:
> On 2013-03-13 18:38:12 +0530, Amit Kapila wrote:
> > On Wednesday, March 13, 2013 6:10 PM Andres Freund wrote:
> > > On 2013-03-12 10:46:53 +0530, Amit Kapila wrote:
> > > > Do you mean to say that because some variables can only be set
> after
> > > restart
> > > > can lead to
> > > > inconsistency, or is it because of asynchronous nature of
> > > pg_reload_conf()?
> > >
> > > As long as SET PERSISTENT cannot be executed inside a transaction -
> or
> > > only takes effect after its end - there doesn't seem to be any
> problem
> > > executing ProcessConfigFile() directly.
> >
> > Do you mean to say we call directly ProcessConfigFile() at end of SET
> > PERSISTENT instead
> > Of pg_reload_conf() but in that case would it load the variables for
> other
> > backends?
> 
> I'd say do both. Yes, we would evaluate config potentially twice. Who
> cares. Messages inside non-postmaster environments are only output at
> DEBUG2
> anyway.

I could see your point, when you say do both that you want that in current
session,
the values will be immediately available which can make user happy.
However if there is any error during function ProcessConfigFile(), it could
be little
inconvenient for user as the setting would have been done in file but memory
processing 
has created problem.
Ideally, there should not be any error in function ProcessConfigFile(), but
might be some palloc or such call
would be sufficient to make user unhappy.

With Regards,
Amit Kapila.

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Andres Freund <andres@2ndquadrant.com> — 2013-03-13T14:35:51Z

On 2013-03-13 18:52:48 +0530, Amit Kapila wrote:
> On Wednesday, March 13, 2013 6:44 PM Andres Freund wrote:
> > On 2013-03-13 18:38:12 +0530, Amit Kapila wrote:
> > > On Wednesday, March 13, 2013 6:10 PM Andres Freund wrote:
> > > > On 2013-03-12 10:46:53 +0530, Amit Kapila wrote:
> > > > > Do you mean to say that because some variables can only be set
> > after
> > > > restart
> > > > > can lead to
> > > > > inconsistency, or is it because of asynchronous nature of
> > > > pg_reload_conf()?
> > > >
> > > > As long as SET PERSISTENT cannot be executed inside a transaction -
> > or
> > > > only takes effect after its end - there doesn't seem to be any
> > problem
> > > > executing ProcessConfigFile() directly.
> > >
> > > Do you mean to say we call directly ProcessConfigFile() at end of SET
> > > PERSISTENT instead
> > > Of pg_reload_conf() but in that case would it load the variables for
> > other
> > > backends?
> > 
> > I'd say do both. Yes, we would evaluate config potentially twice. Who
> > cares. Messages inside non-postmaster environments are only output at
> > DEBUG2
> > anyway.
> 
> I could see your point, when you say do both that you want that in current
> session,
> the values will be immediately available which can make user happy.
> However if there is any error during function ProcessConfigFile(), it could
> be little  inconvenient for user as the setting would have been done in file but memory
> processing has created problem.

But thats pretty independent from this? If anything it allows for
*better* reporting of problems since you could convert the log level to
WARNING if ProcessConfigFile() is executed in foreground - which at least
interactive sessions normally will noramlly be displayed for the user.

If you do don't do it immediately you're in the same situation after the
pg_reload_config(), just that the user won't see any error messages.

There is something I am more worried about which is that it ight be bad
if a postmaster child adapts new values bfore postmaster does. I right
now can't think of any new dangers since the reverse is already true...

Greetings,

Andres Freund

-- 
 Andres Freund	                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Boszormenyi Zoltan <zb@cybertec.at> — 2013-03-13T20:28:34Z

2013-03-13 13:45 keltezéssel, Andres Freund írta:
> On 2013-03-13 09:09:24 +0100, Boszormenyi Zoltan wrote:
>> 2013-03-13 07:42 keltezéssel, Craig Ringer írta:
>>> On 03/12/2013 06:27 AM, Craig Ringer wrote:
>>>>>> Think also about the case where someone wants to change multiple
>>>>>> values together and having just some set and not others would be
>>>>>> inconsistent.
>>>> Yeah, that's a killer. The reload would need to be scheduled for COMMIT
>>>> time, it can't be done by `SET PERSISTENT` directly.
>>> Thinking about this some more, I'm not sure this is a good idea.
>>>
>>> Right now, SET takes effect immediately. Always, without exception.
>>> Delaying SET PERSISTENT's effects until commit would make it
>>> inconsistent with SET's normal behaviour.
>>>
>>> However, *not* delaying it would make it another quirky
>>> not-transactional not-atomic command. That's OK, but if it's not going
>>> to follow transactional semantics it should not be allowed to run within
>>> a transaction, like VACUUM .
>>>
>>> Writing the changes immediately but deferring the reload until commit
>>> seems to be the worst of those two worlds.
>> I was thinking about it a little. There is a hook that runs at the end
>> of (sub-)transactions. It can be abused for this purpose to make
>> SET PERSISTENT transactional. The subtransactions can also stack
>> these settings, by forgetting settings upon ROLLBACK [ TO SAVEPOINT ]
>> and overwriting previous settings upon COMMIT and RELEASE SAVEPOINT.
>> All it needs is a list and maintaining intermediate pointers when entering
>> into a new level of SAVEPOINT. The functions to register such hooks are
>> in src/include/access/xact.h:
>>
>> extern void RegisterXactCallback(XactCallback callback, void *arg);
>> extern void UnregisterXactCallback(XactCallback callback, void *arg);
>> extern void RegisterSubXactCallback(SubXactCallback callback, void *arg);
>> extern void UnregisterSubXactCallback(SubXactCallback callback, void *arg);
> (sub)xact commit/abort already calls AtEOXact_GUC(commit, nestlevel). So you
> wouldn't even need that.

Yes, thanks.

>   It seems we could add another value to enum
> GucStackState, like GUC_SET_PERSISTENT - and process those only if commit &&
> nestlevel == 1.

Maybe it's not needed, only enum GucAction needs a new
GUC_ACTION_PERSISTENT value since that's what has any
business in push_old_value(). Adding two new members to
GucStack like these are enough
     bool has_persistent;
     config_var_value persistent;
and SET PERSISTENT can be treated as GUC_SET. push_old_value()
can merge GUC values set in the same transaction level.

> Everytime you see one with commit && nestlevel > 1 you put them into them into
> the stack one level up.
>
> This seems like its somewhat in line with the way SET LOCAL is implemented?
>
>> On the other hand, it would make a lot of sense to implement it
>> as one setting per file or extending the  code to allow modifying
>> settings in bulk. The one setting per file should be easier and
>> it would also allow extensions to drop their settings automatically
>> into the automatic config directory. I don't know who mentioned
>> this idea about extensions but it also came up a while back.
> I still think one-setting-per-file is the right way to go, yes.
>
> Greetings,
>
> Andres Freund
>


-- 
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de
      http://www.postgresql.at/

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Amit Kapila <amit.kapila@huawei.com> — 2013-03-14T12:53:19Z

On Wednesday, March 13, 2013 8:06 PM Andres Freund wrote:
> On 2013-03-13 18:52:48 +0530, Amit Kapila wrote:
> > On Wednesday, March 13, 2013 6:44 PM Andres Freund wrote:
> > > On 2013-03-13 18:38:12 +0530, Amit Kapila wrote:
> > > > On Wednesday, March 13, 2013 6:10 PM Andres Freund wrote:
> > > > > On 2013-03-12 10:46:53 +0530, Amit Kapila wrote:
> > > > > > Do you mean to say that because some variables can only be
> set
> > > after
> > > > > restart
> > > > > > can lead to
> > > > > > inconsistency, or is it because of asynchronous nature of
> > > > > pg_reload_conf()?
> > > > >
> > > > > As long as SET PERSISTENT cannot be executed inside a
> transaction -
> > > or
> > > > > only takes effect after its end - there doesn't seem to be any
> > > problem
> > > > > executing ProcessConfigFile() directly.
> > > >
> > > > Do you mean to say we call directly ProcessConfigFile() at end of
> SET
> > > > PERSISTENT instead
> > > > Of pg_reload_conf() but in that case would it load the variables
> for
> > > other
> > > > backends?
> > >
> > > I'd say do both. Yes, we would evaluate config potentially twice.
> Who
> > > cares. Messages inside non-postmaster environments are only output
> at
> > > DEBUG2
> > > anyway.
> >
> > I could see your point, when you say do both that you want that in
> current
> > session,
> > the values will be immediately available which can make user happy.
> > However if there is any error during function ProcessConfigFile(), it
> could
> > be little  inconvenient for user as the setting would have been done
> in file but memory
> > processing has created problem.
> 
> But thats pretty independent from this? If anything it allows for
> *better* reporting of problems since you could convert the log level to
> WARNING if ProcessConfigFile() is executed in foreground - which at
> least
> interactive sessions normally will noramlly be displayed for the user.

I checked in ProcessConfigFile(), error level will be DEBUG2 for
IsUnderPostmaster.
So it should not be a problem.
There is another observation from Greg Smith's test-7 (refer his Review
mail), that if 
we call pg_reload_conf() after SET Persistent, it is probably causing some
memory leak which leads to delay
of further connections.
I am checking that problem that why calling pg_reload_conf() leads to memory
increase which further causes delay in connection.

I think once the reason for same is clear, we can decide on how to load conf
file after command execution.

With Regards,
Amit Kapila.

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Amit Kapila <amit.kapila@huawei.com> — 2013-03-14T13:27:52Z

On Monday, March 11, 2013 12:18 PM Amit Kapila wrote:
> On Sunday, March 10, 2013 8:43 PM Greg Smith wrote:
> > On 3/7/13 2:42 AM, Amit Kapila wrote:
> > > I also think the tests added for regression may be more than
> > required...
> > > If you think above optimization's to reduce number of tests are
> okay,
> > then I
> > > will update the patch.
> >
> > I was not trying to get you to remove regression tests.  I was just
> > pointing out to everyone that the patch seems longer than it really
> is,
> > 1) When you change a sighup or user setting, it writes the config
> file
> > out.  But it does not signal for a reload.  Example:
> >
> > $ psql -c "show work_mem" -x
> > work_mem | 1MB
> > $ psql -c "set persistent work_mem='2MB'"
> > SET
> > $ psql -c "show work_mem" -x
> > work_mem | 1MB
> >
> > That the above doesn't leave work_mem set to 2MB for new sessions is
> > surprising.  SET PERSISTENT can't do anything about postmaster
> > parameters like shared_buffers.  But it should be able to change
> > work_mem or a sighup parameter like checkpoint_segments.  The
> > regression
> > test examples call pg_reload_conf() in order to activate the changes.
> > As a user, I would expect the config reload to be automatic after
> > executing SET PERSISTENT on parameters that can be changed
> dynamically.
> 
> I think we cannot guarantee even after calling pg_reload_conf(), as
> this is an
> asynchronous function call.
> We already once had a discussion about this point and as I can
> understand it is
> concluded that it should not be handled. Refer the below mail:
> http://www.postgresql.org/message-id/21869.1360683928@sss.pgh.pa.us
> 
> 
> > 2) Once automatic activation is happening, a hint is really needed in
> > cases where it can't happen.  I'd like to see a warning about that.
> > There's one like this in the code already:
> >
> > LOG:  parameter "shared_buffers" cannot be changed without restarting
> > the server
> >
> > Something like this would be appropriate for SET PERSISTENT:
> >
> > LOG:  parameter "shared_buffers" cannot be changed without restarting
> > the server.  The changed setting has been saved to
> > config/postgresql.auto.conf.

I have not updated code to address 1 and 2, because as per my observation
the 
reason for your test-7 is that you have called pg_reload_conf(), after each
test.
So I feel calling directly in code will also lead to same problem.
I will hold to change code for this till that issue is resolved.

> > 3) The config/postgresql.auto.conf file (which I'd like to see named
> > 'config/persistent.auto.conf') is completely overwritten when a
> change
> > is made.  When the file is written out, both at initdb time and when
> > changes are made, it should have this as its first line:
> >
> > # Do not edit this file manually!  It is overwritten by the SET
> > PERSISTENT command.
> 
> 2 things, you want as part of this comment:
> a. change the name of .auto file to persistent.auto.conf
> b. new comment in beginning of persistent.auto.conf file.

Fixed both 2a and 2b.

> > 4) There is one bad problem case left if I try to make this not work.
> > If I do this:
> >
> > -rm -rf config/
> > -Remove "include_dir config"
> > -Restart the server.  It gives me a warning that SET PERSISTENT won't
> > work because at least one of the multiple things it needs are
> missing.
> > -mkdir config
> > -psql -c "set persistent checkpoint_segments='32'"
> >
> > That command happily writes out postgresql.auto.conf, but since I
> > removed the include_dir it doesn't matter.  It will never become
> > active.
> >
> > The check for the include_dir entry needs to happen each time SET
> > PERSISTENT runs.  This is more important than noticing it at server
> > start time.  In fact, I think if this problem case includes the
> WARNING
> > about "include_dir config" not being in the postgresql.conf, the
> check
> > that happens at server start (shown below) might even go away.
> People
> > who object to it as log file garbage will be happier, and users will
> be
> > told about the feature being broken if they try to use it.
> 
> This can be handled, but for this we need to parse the whole
> postgresql.conf file
> and then give the warning. This will increase the time of SET
> PERSISTENT command.

Fixed by parsing the file and if include_dir is missing, then give Warning
to user.
In case user deletes the folder, ERROR will be thrown, because in anycase it
cannot perform the 
Command if folder is deleted.
If persistent.auto.conf file is missing, then don't raise any error as SET
PERSISTENT command will
create the file.
 
> > 5) An error message appears every time you reload configuration, if
> > some
> > part of the SET PERSISTENT mechanism isn't functional.  This is going
> > to
> > be too much for some people.  And it's confusing when it happens as
> > part
> > an interactive psql session.  I have an example way down at the very
> > end
> > of this message.

Now for reload there will be no error, as we handled the same in SET
PERSISTENT command.
at time of startup it will LOG the message.

> > 6) Putting spaces into memory units results in broken config files:
> >
> > $ psql -c "set persistent work_mem='2 MB'"
> > SET
> > $ cat $PGDATA/config/postgresql.auto.conf
> > work_mem = 2 MB
> >
> > This is wrong and you'll get this on reload:
> >
> > LOG:  syntax error in file
> > "/Users/gsmith/pgwork/data/autoconf/config/postgresql.auto.conf" line
> > 1,
> > near token "MB"
> > LOG:  configuration file
> > "/Users/gsmith/pgwork/data/autoconf/postgresql.conf" contains errors;
> > no
> > changes were applied
> >
> > It needs to look like this:
> >
> > work_mem = '2 MB'
> >
> > A regression test for some cases with spaces in the middle should be
> > added too.  This case is really weird, due to how the code is reading
> > the existing postgresql.auto.conf file.  If I first manually fix the
> > file, and then run a change again, it stays fixed!  Look at this
> weird
> > sequence:
> >
> > $ psql -c "set persistent work_mem='2 MB'"
> > SET
> > $ cat $PGDATA/config/postgresql.auto.conf
> > work_mem = 2 MB
> > [Here this setting is broken]
> > $ vi $PGDATA/config/postgresql.auto.conf
> > [Add quotes around the value, now it works]
> > $ cat $PGDATA/config/postgresql.auto.conf
> > work_mem = '2 MB'
> > $ pg_ctl reload
> > server signaled
> > $ psql -c "set persistent work_mem='4 MB'"
> > SET
> > $ cat $PGDATA/config/postgresql.auto.conf
> > work_mem = '4 MB'
> 
> I shall fix this behavior.

Fixed. Now all values will be in quotes, as at that point it is difficult to
distinguish
as per current implementation.

> >
> > 7) If I run SET PERSISTENT a lot concurrently, something happens that
> > slows down connecting to the server.  Restarting the server makes it
> go
> > away.  I have a pgbench test case demonstrating the problem below, in
> > the "Concurrency" section.  I haven't tried to replicate it on
> another
> > system yet.
> >
> > = Tested features that work fine =
> >
> > Entries added here are tracked as you'd expect:
> >
> > $ psql -c "set persistent checkpoint_segments='32'"
> > $ pg_ctl reload
> > $ psql -xc "select name,setting,sourcefile,sourceline from
> pg_settings
> > where name='checkpoint_segments'"
> > name       | checkpoint_segments
> > setting    | 32
> > sourcefile |
> > /Users/gsmith/pgwork/data/autoconf/config/postgresql.auto.conf
> > sourceline | 1
> >
> > When the postgresql.auto.conf file is missing and you use SET
> > PERSISTENT, it quietly creates it when writing out the new setting.
> >
> > = Concurrency and performance slowdown =
> >
> > I made two pgbench scripts that adjust a guc, one user and the other
> > sighup, to a random value:
> >
> > random-join-limit.sql:
> >
> > \setrandom limit 1 8
> > set persistent join_collapse_limit=:limit;
> > select pg_reload_conf();
> >
> > random-segments.sql:
> >
> > \setrandom segments 3 16
> > set persistent checkpoint_segments=:segments;
> > select pg_reload_conf();
> >
> > I then fired off a bunch of these in parallel:
> >
> > pgbench -n -f random-join-limit.sql -f random-segments.sql -c 8 -T 60
> >
> > This ran at 1547 TPS on my 8 core Mac, so that's not bad.  No
> > assertions
> > and the config file was still valid at the end, which is a good sign
> > the
> > locking method isn't allowing utter chaos.  Without the
> > pg_reload_conf()
> > in the test files, it also completed.  With the reload happening in
> one
> > file but not the other, things were also fine.
> >
> > However, one thing I saw is that the server got significantly slower
> > the
> > more I ran this test script.  After a few minutes it was down to only
> > 400 TPS.  The delay shows up between when I run psql and when I get
> the
> > prompt back.  Here's normal behavior:
> >
> > $ time psql -c "select 1"
> > real	0m0.006s
> >
> > And here's what I get after a single run of this pgbench hammering:
> >
> > $ time psql -c "select 1"
> > real	0m0.800s
> >
> > 800ms?  The slowdown is all for psql to start and connect, it's not
> in
> > the executor:

I suspect this is due to pg_reload_conf(). I am looking into it.
However could you please try once without pg_reload_conf(). 

> > = Error messages =
> >
> > If you remove postgresql.auto.conf and restart the server, it gives a
> > warning that SET PERSISTENT won't work until you put it back.  The
> > error
> > in this and several similar cases is pretty generic too:
> >
> > WARNING:  File "postgresql.auto.conf" is not accessible, either file
> > "postgresql.auto.conf" or folder "config" doesn't exist. or default
> > "config" is not present in postgresql.conf.
> >
> > It would be nice if the error were more specific, individually
> > identifying which of these is the actual problem.  I can rewrite that
> > long text entry to be more readable, but I think it should be a
> series
> > of smaller error checks with their own individual messages instead.
> 
> You are right, we should change this by identifying actual problems.
> 
> > If you remove postgresql.auto.conf then exeute "pg_ctl reload", it
> > gives
> > this error 6 times, which seems excessive.  Reducing how often it
> > appears in the reload case would be nice.
> 
> > Deleting the whole config directory gives this:
> >
> > LOG:  could not open configuration directory
> > "/Users/gsmith/pgwork/data/autoconf/config": No such file or
> directory
> > LOG:  configuration file
> > "/Users/gsmith/pgwork/data/autoconf/postgresql.conf" contains errors;
> > no
> > changes were applied
> >
> > If you now try to use the feature, the error message could be better.
> >
> > $ psql -c "set persistent checkpoint_segments='32'"
> > ERROR:  Failed to open auto conf temp file
> >
> "/Users/gsmith/pgwork/data/autoconf/config/postgresql.auto.conf.temp":
> > No such file or directory
> >
> > It would be nice to complain about the config directory being
> missing,
> > as a first check before the file is opened.  Restarting the server in
> > this situation throws the correct error in your face though:
> >
> > LOG:  could not open configuration directory
> > "/Users/gsmith/pgwork/data/autoconf/config": No such file or
> directory
> > FATAL:  configuration file
> > "/Users/gsmith/pgwork/data/autoconf/postgresql.conf" contains errors
> >
> > If you render this whole system unavailable by removing the
> > "include_dir
> > config", at every server start you'll see this:
> >
> > WARNING:  File "postgresql.auto.conf" is not accessible, either file
> > "postgresql.auto.conf" or folder "config" doesn't exist. or default
> > "config" is not present in postgresql.conf.
> > 	Configuration parameters changed before start/reload of server
> > with SET
> > PERSISTENT command will not be effective.
> > LOG:  database system was shut down at 2013-03-09 23:55:03 EST
> >
> > This is a debatable design choice.  Some people might not want the
> file
> > and remove it, but don't want to be nagged about it.  If people want
> to
> > wipe out the file or directory and work the old way, without this
> > feature available, that's fine and they can.  To me, helping users
> who
> > accidentally break this is more important than reducing harmless
> > warning
> > messages for things people did intentionally.  WARNING might not be
> the
> > right level for this though.  The existing checks like this I showed
> > above use LOG for this sort of thing.
> >
> > The bigger problem is that this message shows up whenever you reload
> > the
> > config too.  Watch this bizarre sequence:
> >
> > gsmith=# select pg_reload_conf();
> >   pg_reload_conf
> > ----------------
> >   t
> > (1 row)
> >
> > gsmith=#
> > gsmith=# select 1;
> > WARNING:  File "postgresql.auto.conf" is not accessible, either file
> > "postgresql.auto.conf" or folder "config" doesn't exist. or default
> > "config" is not present in postgresql.conf.
> > Configuration parameters changed before start/reload of server with
> SET
> > PERSISTENT command will not be effective.
> >   ?column?
> > ----------
> >          1
> >
> > And as I commented above, shifting more of the checks to SET
> PERISTENT
> > time could eliminate this check from running at server start and
> reload
> > altogether.  I would be fine with them *also* happening at server
> > start.
> >   But I could understand that other people might not like that.  And
> > having this pop up on every reload, appearing to a client next to
> > another statement altogether, that isn't acceptable though.
> 
> Agreed, I will add the check at time of SET PERSISTENT command.
> Also I will try to remove this message at time of reload.

Done as explained above.

With Regards,
Amit Kapila.

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Boszormenyi Zoltan <zb@cybertec.at> — 2013-03-14T23:48:04Z

2013-03-13 21:28 keltezéssel, Boszormenyi Zoltan írta:
> 2013-03-13 13:45 keltezéssel, Andres Freund írta:
>> On 2013-03-13 09:09:24 +0100, Boszormenyi Zoltan wrote:
>>> 2013-03-13 07:42 keltezéssel, Craig Ringer írta:
>>>> On 03/12/2013 06:27 AM, Craig Ringer wrote:
>>>>>>> Think also about the case where someone wants to change multiple
>>>>>>> values together and having just some set and not others would be
>>>>>>> inconsistent.
>>>>> Yeah, that's a killer. The reload would need to be scheduled for COMMIT
>>>>> time, it can't be done by `SET PERSISTENT` directly.
>>>> Thinking about this some more, I'm not sure this is a good idea.
>>>>
>>>> Right now, SET takes effect immediately. Always, without exception.
>>>> Delaying SET PERSISTENT's effects until commit would make it
>>>> inconsistent with SET's normal behaviour.
>>>>
>>>> However, *not* delaying it would make it another quirky
>>>> not-transactional not-atomic command. That's OK, but if it's not going
>>>> to follow transactional semantics it should not be allowed to run within
>>>> a transaction, like VACUUM .
>>>>
>>>> Writing the changes immediately but deferring the reload until commit
>>>> seems to be the worst of those two worlds.
>>> I was thinking about it a little. There is a hook that runs at the end
>>> of (sub-)transactions. It can be abused for this purpose to make
>>> SET PERSISTENT transactional. The subtransactions can also stack
>>> these settings, by forgetting settings upon ROLLBACK [ TO SAVEPOINT ]
>>> and overwriting previous settings upon COMMIT and RELEASE SAVEPOINT.
>>> All it needs is a list and maintaining intermediate pointers when entering
>>> into a new level of SAVEPOINT. The functions to register such hooks are
>>> in src/include/access/xact.h:
>>>
>>> extern void RegisterXactCallback(XactCallback callback, void *arg);
>>> extern void UnregisterXactCallback(XactCallback callback, void *arg);
>>> extern void RegisterSubXactCallback(SubXactCallback callback, void *arg);
>>> extern void UnregisterSubXactCallback(SubXactCallback callback, void *arg);
>> (sub)xact commit/abort already calls AtEOXact_GUC(commit, nestlevel). So you
>> wouldn't even need that.
>
> Yes, thanks.
>
>>   It seems we could add another value to enum
>> GucStackState, like GUC_SET_PERSISTENT - and process those only if commit &&
>> nestlevel == 1.
>
> Maybe it's not needed, only enum GucAction needs a new
> GUC_ACTION_PERSISTENT value since that's what has any
> business in push_old_value(). Adding two new members to
> GucStack like these are enough
>     bool has_persistent;
>     config_var_value persistent;
> and SET PERSISTENT can be treated as GUC_SET. push_old_value()
> can merge GUC values set in the same transaction level.

It seems both were needed. The attached patch makes
SET PERSISTENT transactional and uses one setting per file.
It uses the currently existing parsing and validating code
and because of this, the patch is half the size of v12 from Amit.

Best regards,
Zoltán Böszörményi

>
>> Everytime you see one with commit && nestlevel > 1 you put them into them into
>> the stack one level up.
>>
>> This seems like its somewhat in line with the way SET LOCAL is implemented?
>>
>>> On the other hand, it would make a lot of sense to implement it
>>> as one setting per file or extending the  code to allow modifying
>>> settings in bulk. The one setting per file should be easier and
>>> it would also allow extensions to drop their settings automatically
>>> into the automatic config directory. I don't know who mentioned
>>> this idea about extensions but it also came up a while back.
>> I still think one-setting-per-file is the right way to go, yes.
>>
>> Greetings,
>>
>> Andres Freund
>>
>
>


-- 
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de
      http://www.postgresql.at/

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Boszormenyi Zoltan <zb@cybertec.at> — 2013-03-15T09:20:47Z

2013-03-15 00:48 keltezéssel, Boszormenyi Zoltan írta:
> 2013-03-13 21:28 keltezéssel, Boszormenyi Zoltan írta:
>> 2013-03-13 13:45 keltezéssel, Andres Freund írta:
>>> On 2013-03-13 09:09:24 +0100, Boszormenyi Zoltan wrote:
>>>> 2013-03-13 07:42 keltezéssel, Craig Ringer írta:
>>>>> On 03/12/2013 06:27 AM, Craig Ringer wrote:
>>>>>>>> Think also about the case where someone wants to change multiple
>>>>>>>> values together and having just some set and not others would be
>>>>>>>> inconsistent.
>>>>>> Yeah, that's a killer. The reload would need to be scheduled for COMMIT
>>>>>> time, it can't be done by `SET PERSISTENT` directly.
>>>>> Thinking about this some more, I'm not sure this is a good idea.
>>>>>
>>>>> Right now, SET takes effect immediately. Always, without exception.
>>>>> Delaying SET PERSISTENT's effects until commit would make it
>>>>> inconsistent with SET's normal behaviour.
>>>>>
>>>>> However, *not* delaying it would make it another quirky
>>>>> not-transactional not-atomic command. That's OK, but if it's not going
>>>>> to follow transactional semantics it should not be allowed to run within
>>>>> a transaction, like VACUUM .
>>>>>
>>>>> Writing the changes immediately but deferring the reload until commit
>>>>> seems to be the worst of those two worlds.
>>>> I was thinking about it a little. There is a hook that runs at the end
>>>> of (sub-)transactions. It can be abused for this purpose to make
>>>> SET PERSISTENT transactional. The subtransactions can also stack
>>>> these settings, by forgetting settings upon ROLLBACK [ TO SAVEPOINT ]
>>>> and overwriting previous settings upon COMMIT and RELEASE SAVEPOINT.
>>>> All it needs is a list and maintaining intermediate pointers when entering
>>>> into a new level of SAVEPOINT. The functions to register such hooks are
>>>> in src/include/access/xact.h:
>>>>
>>>> extern void RegisterXactCallback(XactCallback callback, void *arg);
>>>> extern void UnregisterXactCallback(XactCallback callback, void *arg);
>>>> extern void RegisterSubXactCallback(SubXactCallback callback, void *arg);
>>>> extern void UnregisterSubXactCallback(SubXactCallback callback, void *arg);
>>> (sub)xact commit/abort already calls AtEOXact_GUC(commit, nestlevel). So you
>>> wouldn't even need that.
>>
>> Yes, thanks.
>>
>>>   It seems we could add another value to enum
>>> GucStackState, like GUC_SET_PERSISTENT - and process those only if commit &&
>>> nestlevel == 1.
>>
>> Maybe it's not needed, only enum GucAction needs a new
>> GUC_ACTION_PERSISTENT value since that's what has any
>> business in push_old_value(). Adding two new members to
>> GucStack like these are enough
>>     bool has_persistent;
>>     config_var_value persistent;
>> and SET PERSISTENT can be treated as GUC_SET. push_old_value()
>> can merge GUC values set in the same transaction level.
>
> It seems both were needed. The attached patch makes
> SET PERSISTENT transactional and uses one setting per file.
> It uses the currently existing parsing and validating code
> and because of this, the patch is half the size of v12 from Amit.

The only missing piece is the check for superuser.


>
> Best regards,
> Zoltán Böszörményi
>
>>
>>> Everytime you see one with commit && nestlevel > 1 you put them into them into
>>> the stack one level up.
>>>
>>> This seems like its somewhat in line with the way SET LOCAL is implemented?
>>>
>>>> On the other hand, it would make a lot of sense to implement it
>>>> as one setting per file or extending the  code to allow modifying
>>>> settings in bulk. The one setting per file should be easier and
>>>> it would also allow extensions to drop their settings automatically
>>>> into the automatic config directory. I don't know who mentioned
>>>> this idea about extensions but it also came up a while back.
>>> I still think one-setting-per-file is the right way to go, yes.
>>>
>>> Greetings,
>>>
>>> Andres Freund
>>>
>>
>>
>
>
>
>


-- 
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de
      http://www.postgresql.at/

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review] (Fix memory growth)

Amit Kapila <amit.kapila@huawei.com> — 2013-03-15T10:57:29Z

On Thursday, March 14, 2013 6:58 PM Amit Kapila wrote:
> On Monday, March 11, 2013 12:18 PM Amit Kapila wrote:
> > On Sunday, March 10, 2013 8:43 PM Greg Smith wrote:
> > > On 3/7/13 2:42 AM, Amit Kapila wrote:
> > > > I also think the tests added for regression may be more than
> > > required...
> > > > If you think above optimization's to reduce number of tests are
> > okay,
> > > then I
> > > > will update the patch.
> > > 7) If I run SET PERSISTENT a lot concurrently, something happens
> > > that slows down connecting to the server.  Restarting the server
> > > makes it
> > go
> > > away.  I have a pgbench test case demonstrating the problem below,
> > > in the "Concurrency" section.  I haven't tried to replicate it on
> > another
> > > system yet.
> > >
> > > = Tested features that work fine =
> > >
> > > Entries added here are tracked as you'd expect:
> > >
> > > $ psql -c "set persistent checkpoint_segments='32'"
> > > $ pg_ctl reload
> > > $ psql -xc "select name,setting,sourcefile,sourceline from
> > pg_settings
> > > where name='checkpoint_segments'"
> > > name       | checkpoint_segments
> > > setting    | 32
> > > sourcefile |
> > > /Users/gsmith/pgwork/data/autoconf/config/postgresql.auto.conf
> > > sourceline | 1
> > >
> > > When the postgresql.auto.conf file is missing and you use SET
> > > PERSISTENT, it quietly creates it when writing out the new setting.
> > >
> > > = Concurrency and performance slowdown =
> > >
> > > I made two pgbench scripts that adjust a guc, one user and the
> other
> > > sighup, to a random value:
> > >
> > > random-join-limit.sql:
> > >
> > > \setrandom limit 1 8
> > > set persistent join_collapse_limit=:limit; select pg_reload_conf();
> > >
> > > random-segments.sql:
> > >
> > > \setrandom segments 3 16
> > > set persistent checkpoint_segments=:segments; select
> > > pg_reload_conf();
> > >
> > > I then fired off a bunch of these in parallel:
> > >
> > > pgbench -n -f random-join-limit.sql -f random-segments.sql -c 8 -T
> > > 60
> > >
> > > This ran at 1547 TPS on my 8 core Mac, so that's not bad.  No
> > > assertions and the config file was still valid at the end, which is
> > > a good sign the locking method isn't allowing utter chaos.  Without
> > > the
> > > pg_reload_conf()
> > > in the test files, it also completed.  With the reload happening in
> > one
> > > file but not the other, things were also fine.
> > >
> > > However, one thing I saw is that the server got significantly
> slower
> > > the more I ran this test script.  After a few minutes it was down
> to
> > > only 400 TPS.  The delay shows up between when I run psql and when
> I
> > > get
> > the
> > > prompt back.  Here's normal behavior:
> > >
> > > $ time psql -c "select 1"
> > > real	0m0.006s
> > >
> > > And here's what I get after a single run of this pgbench hammering:
> > >
> > > $ time psql -c "select 1"
> > > real	0m0.800s
> > >
> > > 800ms?  The slowdown is all for psql to start and connect, it's not
> > in
> > > the executor:
> 
> I suspect this is due to pg_reload_conf(). I am looking into it.
> However could you please try once without pg_reload_conf().

The memory growth is because of pg_reload_conf(). 
There were 2 sources of memory growth:
1. Load_ident - Memory context was not getting deleted
2. ProcessConfigFile - Most of the memory for parsing and processing is
allocated under TopMemoryContext and is not getting reset,
                       which leads to this problem. For now I have fixed by
doing this in separate context.

The above 2 fixes are in attached patch 'ctx_growth_fix_v1'.

Apart from this, I have fixed issues 1 and 2 reported by you in attached
version 'set_persistent_v13', by sending signal at end of command
and checking for parameters which cannot be loaded.

With this, all the issues reported by you are addressed. 

Thanks to you for so careful test and review of this patch.

With Regards,
Amit Kapila.

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Greg Smith <greg@2ndquadrant.com> — 2013-03-17T16:05:55Z

On 3/14/13 4:48 PM, Boszormenyi Zoltan wrote:
> The attached patch makes
> SET PERSISTENT transactional and uses one setting per file.
> It uses the currently existing parsing and validating code
> and because of this, the patch is half the size of v12 from Amit.

That's not a completely fair comparison, because you lost all the 
regression testing code too.  This does look like a usefully tighter 
implementation in a few ways, so good progress on that.

I still can't see any reason to prefer this "one setting per file" idea. 
  As I see it, that is pushing the complexity toward the user in a bad 
way, seemingly just so it's easier to implement.  Most of my customers 
now use tools like Puppet to manage their PostgreSQL configuration.  I 
do not want to have this conversation:

Greg:  "You can use SET PERSISTENT to modify settings instead of 
changing the postgresql.conf"

User:  "That's useful.  How do we adjust Puppet to make sure it picks up 
the changes?"

Greg:  "You just scan this config directory and add every file that 
appears in there!  Each setting will be in its own file."

User:  "<shocked look>  It creates new files?  That isn't going to fit 
into our version control approach easily.  Can we disable this feature 
so no one accidentally uses it?"

I'm not exaggerating here--"one setting per file" makes this feature 
less than useless to me.  It becomes something where I will have to 
waste time defending against people using it.  I'd prefer to not have 
this at all than to do it that way.

That we're breaking these settings off into their own file, instead of 
trying to edit the postgresql.conf, to me is a pragmatic trade-off to 
keep the implementation from being really complicated.  It's also a step 
forward in a larger series for how to improve configuration management. 
  Just because that change introduces an entire directory being scanned, 
I don't see that as an excuse to clutter it with a long list of files too.

-- 
Greg Smith   2ndQuadrant US    greg@2ndQuadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com

Re: Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Boszormenyi Zoltan <zb@cybertec.at> — 2013-03-17T19:49:13Z

2013-03-17 17:05 keltezéssel, Greg Smith írta:
> On 3/14/13 4:48 PM, Boszormenyi Zoltan wrote:
>> The attached patch makes
>> SET PERSISTENT transactional and uses one setting per file.
>> It uses the currently existing parsing and validating code
>> and because of this, the patch is half the size of v12 from Amit.
>
> That's not a completely fair comparison, because you lost all the regression testing 
> code too.

True.

> This does look like a usefully tighter implementation in a few ways, so good progress on 
> that.
>
> I still can't see any reason to prefer this "one setting per file" idea.  As I see it, 
> that is pushing the complexity toward the user in a bad way, seemingly just so it's 
> easier to implement.  Most of my customers now use tools like Puppet to manage their 
> PostgreSQL configuration.  I do not want to have this conversation:
>
> Greg:  "You can use SET PERSISTENT to modify settings instead of changing the 
> postgresql.conf"
>
> User:  "That's useful.  How do we adjust Puppet to make sure it picks up the changes?"
>
> Greg:  "You just scan this config directory and add every file that appears in there!  
> Each setting will be in its own file."
>
> User:  "<shocked look>  It creates new files?  That isn't going to fit into our version 
> control approach easily.  Can we disable this feature so no one accidentally uses it?"
>
> I'm not exaggerating here--"one setting per file" makes this feature less than useless 
> to me.  It becomes something where I will have to waste time defending against people 
> using it.  I'd prefer to not have this at all than to do it that way.
>
> That we're breaking these settings off into their own file, instead of trying to edit 
> the postgresql.conf, to me is a pragmatic trade-off to keep the implementation from 
> being really complicated.  It's also a step forward in a larger series for how to 
> improve configuration management.  Just because that change introduces an entire 
> directory being scanned, I don't see that as an excuse to clutter it with a long list of 
> files too.

OK. I just wanted to show an alternative implementation.

I admit, I haven't read all mails from this thread so I don't know
how important for this feature to be non-transactional, or
whether it would be better to be transactional.

The one-file-to-rule-them-all approach can also be added to this
patch as well, but synchronizing transactions over parsing and
rewriting the extra file would decrease the relaxed effects of
synchronous_commit. On the other hand, writing out one file
per setting, although atomic to the level of one file, cannot be
guaranteed to be atomic as a whole for all settings changed
in the transaction without some synchronization.

As far as I can see, AtEOXact_GUC() is called outside the
critical section and is not synchronized any other way.
(Currently, there's no need to.)

Best regards,
Zoltán Böszörményi

-- 
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de
      http://www.postgresql.at/

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Robert Haas <robertmhaas@gmail.com> — 2013-03-21T18:38:12Z

On Tue, Mar 12, 2013 at 11:36 AM, Greg Stark <stark@mit.edu> wrote:
> On Tue, Mar 12, 2013 at 9:06 AM, Greg Smith <greg@2ndquadrant.com> wrote:
>> That's jumping right over a few rounds of simpler ways to do this, and just
>> going right to the approach we know allows adding more such options later
>> with minimal grammar impact.
>
> As Craig intimated, the minimal grammar impact would be simply
>
> BEGIN;
> set persistent maintenance_work_mem='2GB';
> set persistent work_mem='2GB';
> COMMIT;
>
> Sending the sighup at transaction end seems like a fairly safe thing
> to do too. It's hard to imagine it failing and if it did the worst
> case would be that other backends would still have the old values too.

I'm late to the party here, but my feeling is that we ought NOT to
send a SIGHUP automatically.  For various reasons already discussed,
that's not going to be completely transparent to the DBA.  And when
things are not fully transparent, IME, it's better not to do them
automatically.  DBAs like automation when it's 100% reliable - but
they don't like the database to automatically do things that can have
unforeseen side effects.

Also, while I think that MOST people will probably want a SIGHUP right
after SET PERSISTENT, I am not sure that EVERYONE will want that.  If
you want it and it doesn't happen automatically, you can always do it
by hand.  If you don't want it and it does happen automatically,
you're out of luck: go back to hand-editing config files.  I really
don't want to leave people with any more reasons to continue
hand-editing the config files than we truly must.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Greg Smith <greg@2ndquadrant.com> — 2013-03-21T19:05:44Z

On 3/21/13 2:38 PM, Robert Haas wrote:
> Also, while I think that MOST people will probably want a SIGHUP right
> after SET PERSISTENT, I am not sure that EVERYONE will want that.  If
> you want it and it doesn't happen automatically, you can always do it
> by hand.

This is a fair position, and since that's how the feature as written 
right now works that helps.  I think proceeding this way needs to hand 
some sort of hint back to the user though, telling them the change isn't 
active until SIGHUP.  The path I don't want to see if where someone uses 
SET PERSISTENT and can't figure out why nothing changed.  It should be 
as obvious as we can make it to someone that the explicit reload is 
necessary.

-- 
Greg Smith   2ndQuadrant US    greg@2ndQuadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Alvaro Herrera <alvherre@2ndquadrant.com> — 2013-03-21T19:17:54Z

Greg Smith escribió:
> On 3/21/13 2:38 PM, Robert Haas wrote:
> >Also, while I think that MOST people will probably want a SIGHUP right
> >after SET PERSISTENT, I am not sure that EVERYONE will want that.  If
> >you want it and it doesn't happen automatically, you can always do it
> >by hand.
> 
> This is a fair position, and since that's how the feature as written
> right now works that helps.  I think proceeding this way needs to
> hand some sort of hint back to the user though, telling them the
> change isn't active until SIGHUP.  The path I don't want to see if
> where someone uses SET PERSISTENT and can't figure out why nothing
> changed.  It should be as obvious as we can make it to someone that
> the explicit reload is necessary.

Maybe add some syntax to prevent the SIGHUP for the rare case where that
is wanted, say

SET PERSISTENT (reload=off) var=val;

(perhaps WITH at the end, dunno)

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Amit Kapila <amit.kapila@huawei.com> — 2013-03-22T02:39:20Z

On Friday, March 22, 2013 12:48 AM Alvaro Herrera wrote:
> Greg Smith escribió:
> > On 3/21/13 2:38 PM, Robert Haas wrote:
> > >Also, while I think that MOST people will probably want a SIGHUP
> right
> > >after SET PERSISTENT, I am not sure that EVERYONE will want that.
> If
> > >you want it and it doesn't happen automatically, you can always do
> it
> > >by hand.
> >
> > This is a fair position, and since that's how the feature as written
> > right now works that helps.  I think proceeding this way needs to
> > hand some sort of hint back to the user though, telling them the
> > change isn't active until SIGHUP.  The path I don't want to see if
> > where someone uses SET PERSISTENT and can't figure out why nothing
> > changed.  It should be as obvious as we can make it to someone that
> > the explicit reload is necessary.
> 
> Maybe add some syntax to prevent the SIGHUP for the rare case where
> that
> is wanted, say
> 
> SET PERSISTENT (reload=off) var=val;
> 
> (perhaps WITH at the end, dunno)

I think adding new syntax change is little scary for me, not for the matter
of implementation but for building consensus on syntax.
Can we do it as an enhancement later?
Other ways could be to either give Notice after each command or mention the
same in documentation clearly.
I feel giving Notice after every command doesn't look good, so may be we can
mention the same in documentation.

With Regards,
Amit Kapila.

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Alvaro Herrera <alvherre@2ndquadrant.com> — 2013-03-22T03:27:23Z

Amit Kapila escribió:
> On Friday, March 22, 2013 12:48 AM Alvaro Herrera wrote:

> > Maybe add some syntax to prevent the SIGHUP for the rare case where
> > that
> > is wanted, say
> > 
> > SET PERSISTENT (reload=off) var=val;
> > 
> > (perhaps WITH at the end, dunno)
> 
> I think adding new syntax change is little scary for me, not for the matter
> of implementation but for building consensus on syntax.

I cannot but agree on that point.

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Greg Smith <greg@2ndquadrant.com> — 2013-03-22T04:43:28Z

On 3/21/13 10:39 PM, Amit Kapila wrote:
> I think adding new syntax change is little scary for me, not for the matter
> of implementation but for building consensus on syntax.
> Can we do it as an enhancement later?

Yeah, I think the basics of this could be committed, but have later 
improvements that let you fine-tune when the reload happens better.

> Other ways could be to either give Notice after each command or mention the
> same in documentation clearly.
> I feel giving Notice after every command doesn't look good, so may be we can
> mention the same in documentation.

I think that NOTICE after every command is the only way we'll make sure 
to catch every user who should be notified about the feature's limitation.

I was thinking of something like this:

NOTICE:  New configuration is written but not applied yet.  Signal the 
server to reload settings using pg_reload_conf to activate the change.

Maybe that's a NOTICE plus a HINT.

-- 
Greg Smith   2ndQuadrant US    greg@2ndQuadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Amit Kapila <amit.kapila@huawei.com> — 2013-03-22T04:47:10Z

On Friday, March 22, 2013 8:57 AM Alvaro Herrera wrote:
> Amit Kapila escribió:
> > On Friday, March 22, 2013 12:48 AM Alvaro Herrera wrote:
> 
> > > Maybe add some syntax to prevent the SIGHUP for the rare case where
> > > that
> > > is wanted, say
> > >
> > > SET PERSISTENT (reload=off) var=val;
> > >
> > > (perhaps WITH at the end, dunno)
> >
> > I think adding new syntax change is little scary for me, not for the
> matter
> > of implementation but for building consensus on syntax.
> 
> I cannot but agree on that point.

Sorry, I don't get your point. 
Do you mean to say that you don't agree with me and want new syntax as
proposed by you to be implemented?

With Regards,
Amit Kapila.

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Tom Lane <tgl@sss.pgh.pa.us> — 2013-03-22T04:58:05Z

Greg Smith <greg@2ndQuadrant.com> writes:
> On 3/21/13 10:39 PM, Amit Kapila wrote:
>> I feel giving Notice after every command doesn't look good, so may be we can
>> mention the same in documentation.

> I think that NOTICE after every command is the only way we'll make sure 
> to catch every user who should be notified about the feature's limitation.

We have been around on that type of proposal before.  A command that
issues a NOTICE as part of its *standard* behavior is really not going
to fly; the annoyance, talking-down-to-the-user factor is too high.

IOW, if you think the command needs that, then its design is broken
and you need to do better.

			regards, tom lane

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Alvaro Herrera <alvherre@2ndquadrant.com> — 2013-03-22T14:02:45Z

Amit Kapila escribió:
> On Friday, March 22, 2013 8:57 AM Alvaro Herrera wrote:
> > Amit Kapila escribió:

> > > I think adding new syntax change is little scary for me, not for
> > > the matter of implementation but for building consensus on syntax.
> > 
> > I cannot but agree on that point.
> 
> Sorry, I don't get your point. 
> Do you mean to say that you don't agree with me and want new syntax as
> proposed by you to be implemented?

On the contrary, I was saying I agree with you on the difficulty on
getting consensus on this.

I don't know where I learned this phrase or even if it's in common
usage.  After a not-so-quick search I see it explained here:
http://www.perfectyourenglish.com/usage/but.htm

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Amit Kapila <amit.kapila@huawei.com> — 2013-03-23T01:50:52Z

On Friday, March 22, 2013 7:33 PM Alvaro Herrera wrote:
> Amit Kapila escribió:
> > On Friday, March 22, 2013 8:57 AM Alvaro Herrera wrote:
> > > Amit Kapila escribió:
> 
> > > > I think adding new syntax change is little scary for me, not for
> > > > the matter of implementation but for building consensus on
> syntax.
> > >
> > > I cannot but agree on that point.
> >
> > Sorry, I don't get your point.
> > Do you mean to say that you don't agree with me and want new syntax
> as
> > proposed by you to be implemented?
> 
> On the contrary, I was saying I agree with you on the difficulty on
> getting consensus on this.

Now based on final discussion, I will go-ahead and document the behavior
rather than using SIGHUP at end or giving NOTICE at end of command.
Let me know if anybody still feels otherwise.

Apart from this during Greg's testing, he found a performance problem with
running pg_reload_conf() along with my patch the reason for which actually
turns out to be a memory
growth. I have attempted to fix it in the patch ctx_growth_fix_v1.patch
which is currently attached in CF queue along with my Patch.
http://www.postgresql.org/message-id/004801ce216b$e37c3b30$aa74b190$@kapila@
huawei.com
I think that should be done irrespective of SET Persistent Patch.

> I don't know where I learned this phrase or even if it's in common
> usage.  After a not-so-quick search I see it explained here:
> http://www.perfectyourenglish.com/usage/but.htm

You were right, I was not able to understand.

With Regards,
Amit Kapila.

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Amit Kapila <amit.kapila@huawei.com> — 2013-03-25T12:22:11Z

On Saturday, March 23, 2013 7:21 AM Amit Kapila wrote:
> On Friday, March 22, 2013 7:33 PM Alvaro Herrera wrote:
> > Amit Kapila escribió:
> > > On Friday, March 22, 2013 8:57 AM Alvaro Herrera wrote:
> > > > Amit Kapila escribió:
> >
> > > > > I think adding new syntax change is little scary for me, not
> for
> > > > > the matter of implementation but for building consensus on
> > syntax.
> > > >
> > > > I cannot but agree on that point.
> > >
> > > Sorry, I don't get your point.
> > > Do you mean to say that you don't agree with me and want new syntax
> > as
> > > proposed by you to be implemented?
> >
> > On the contrary, I was saying I agree with you on the difficulty on
> > getting consensus on this.
> 
> Now based on final discussion, I will go-ahead and document the
> behavior
> rather than using SIGHUP at end or giving NOTICE at end of command.
> Let me know if anybody still feels otherwise.

Updated the patch to remove SIGHUP at end of command and updated docs
accordingly.

> Apart from this during Greg's testing, he found a performance problem
> with
> running pg_reload_conf() along with my patch the reason for which
> actually
> turns out to be a memory
> growth. I have attempted to fix it in the patch ctx_growth_fix_v1.patch
> which is currently attached in CF queue along with my Patch.
> http://www.postgresql.org/message-
> id/004801ce216b$e37c3b30$aa74b190$@kapila@
> huawei.com
> I think that should be done irrespective of SET Persistent Patch.

With Regards,
Amit Kapila.

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Greg Smith <greg@2ndquadrant.com> — 2013-04-01T07:33:27Z

At this point SET PERSISTENT is looking complete enough that some parts 
of it might be committed.  There's been a lot of useful progress in 
nailing down the early/obvious problems and what fundamental approach 
makes sense.  Accepting the whole thing still seems a bit too invasive 
to chew on this late in 9.3 though.  Unfortunately it's not until 
everything goes in that a really useful change happens.

What I'll mainly talk about here is how to break this into smaller 
pieces to consider as potential future commits, hopeful ones that might 
all go in early for 9.4.  There's no identified committer for this 
feature yet.  I think which committer (or committers, plural, given the 
number of pieces) is willing to take on the GUC changes, and how 
aggressively they want to consider the pieces of this, is going to 
determine the timeline for when the moving parts of this idea are adopted.

As for the code that seems high risk to me, there's a whole class of 
subtle problems possible in the memory management in particular.  The 
first stress test I did (just to try and break the feature) shook out 
what looks so far like a memory allocation concern in the existing 
codebase, before this new feature is even considered.  That didn't feel 
like a good sign to me.

Stepping back, I see four potential commits worth of changes here now, 
and I would consider them in this order:

1) Fix for pg_reload_conf memory context growth problem uncovered during 
testing.  This is a small patch and minor bug fix that could even go 
into 9.3.  The problem is not so severe it seems worth the disruption of 
backporting to earlier versions, and it wouldn't really hurt much to 
kick the problem forward to 9.4.

2) Change the default postgresql.conf to include a config directory. 
This has been one of my soapbox positions for a while now, and I think 
this feature is working well enough now to have proven the form of 
re-arrangement does something useful.  This is a small change to the 
PostgreSQL code that will ripple out to impact packaging.  It would be 
possible to do this part and face the main disruption for 9.3 even if 
the exact SET PERSISTENT implementation is pushed forward.  If there was 
not much else going on with packaging right now, I might even advocate 
that myself.  Unfortunately, right now "not much else going on" is the 
exact opposite of the situation all the packagers are in.  It's just a 
bad time to talk about it.

3) Rearrangement of GUC validation into validate_conf_option function. 
This is ~400 lines of code that could be changed as a zero-impact 
refactoring, one that just makes the SET PERSISTENT option easier to 
implement.

4) Implementation of SET PERSISTENT into a single file, 
config/persistent.auto.conf.  As written, a reasonable amount of error 
checking is done to reduce the odds of someone breaking its 
implementation without realizing it.  The responsibility of activating 
the new configuration using pg_reload_conf is pushed toward the user, 
documented but without an explicit warning.

All of these changes have different risk vs. reward trade-offs in them. 
  I'd guess the validation_config_option change (3) is the riskiest of 
this bunch, because it could easily break the standard GUC path in a 
major way even for people who don't use the feature.  It's not that much 
code, but it is going to take a good bit of committer level review to 
accept due to its risk.

-- 
Greg Smith   2ndQuadrant US    greg@2ndQuadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Amit Kapila <amit.kapila@huawei.com> — 2013-04-01T09:44:35Z

On Monday, April 01, 2013 1:03 PM Greg Smith wrote:
> At this point SET PERSISTENT is looking complete enough that some parts
> of it might be committed.  There's been a lot of useful progress in
> nailing down the early/obvious problems and what fundamental approach
> makes sense.  Accepting the whole thing still seems a bit too invasive
> to chew on this late in 9.3 though.  Unfortunately it's not until
> everything goes in that a really useful change happens.
> 
> What I'll mainly talk about here is how to break this into smaller
> pieces to consider as potential future commits, hopeful ones that might
> all go in early for 9.4.  There's no identified committer for this
> feature yet.  I think which committer (or committers, plural, given the
> number of pieces) is willing to take on the GUC changes, and how
> aggressively they want to consider the pieces of this, is going to
> determine the timeline for when the moving parts of this idea are
> adopted.
> 
> As for the code that seems high risk to me, there's a whole class of
> subtle problems possible in the memory management in particular.  The
> first stress test I did (just to try and break the feature) shook out
> what looks so far like a memory allocation concern in the existing
> codebase, before this new feature is even considered.  That didn't feel
> like a good sign to me.
> 
> Stepping back, I see four potential commits worth of changes here now,
> and I would consider them in this order:
> 
> 1) Fix for pg_reload_conf memory context growth problem uncovered
> during
> testing.  This is a small patch and minor bug fix that could even go
> into 9.3.  The problem is not so severe it seems worth the disruption
> of
> backporting to earlier versions, and it wouldn't really hurt much to
> kick the problem forward to 9.4.
> 
> 2) Change the default postgresql.conf to include a config directory.
> This has been one of my soapbox positions for a while now, and I think
> this feature is working well enough now to have proven the form of
> re-arrangement does something useful.  This is a small change to the
> PostgreSQL code that will ripple out to impact packaging.  It would be
> possible to do this part and face the main disruption for 9.3 even if
> the exact SET PERSISTENT implementation is pushed forward.  If there
> was
> not much else going on with packaging right now, I might even advocate
> that myself.  Unfortunately, right now "not much else going on" is the
> exact opposite of the situation all the packagers are in.  It's just a
> bad time to talk about it.
> 
> 3) Rearrangement of GUC validation into validate_conf_option function.
> This is ~400 lines of code that could be changed as a zero-impact
> refactoring, one that just makes the SET PERSISTENT option easier to
> implement.
> 
> 4) Implementation of SET PERSISTENT into a single file,
> config/persistent.auto.conf.  As written, a reasonable amount of error
> checking is done to reduce the odds of someone breaking its
> implementation without realizing it.  The responsibility of activating
> the new configuration using pg_reload_conf is pushed toward the user,
> documented but without an explicit warning.
> 
> All of these changes have different risk vs. reward trade-offs in them.
>   I'd guess the validation_config_option change (3) is the riskiest of
> this bunch, because it could easily break the standard GUC path in a
> major way even for people who don't use the feature.  It's not that
> much
> code, but it is going to take a good bit of committer level review to
> accept due to its risk.

I think in that case we can have 3 separate patches 
1. Memory growth defect fix
2. Default postgresql.conf to include config directory and SET Persistent
into single file implementation
3. Rearrangement of GUC validation into validate_conf_option function.

As already there is review happened for point 2 and point 1 is an existing
code defect fix, so in my opinion
Patch 1 and 2 should be considered for 9.3. 

With Regards,
Amit Kapila.

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Peter Eisentraut <peter_e@gmx.net> — 2013-04-02T16:19:05Z

I'm going to ignore most of the discussion that led up to this and give
this patch a fresh look.

+ <screen>
+ SET PERSISTENT max_connections To 10;
+ </screen>

The To should probably be capitalized.

I doubt this example actually works because changing max_connections
requires a restart.  Try to find a better example.

It's weird that SET LOCAL and SET SESSION actually *set* the value, and
the second key word determines how long the setting will last.  SET
PERSISTENT doesn't actually set the value.  I predict that this will be
a new favorite help-it-doesn't-work FAQ.

      <varlistentry>
       <term><literal>SCHEMA</literal></term>
       <listitem>
!       <para><literal>SET [ PERSISTENT ] SCHEMA
'<replaceable>value</>'</> is an alias for
         <literal>SET search_path TO <replaceable>value</></>.  Only one
         schema can be specified using this syntax.
        </para>

I don't think [ PERSISTENT ] needs to be added in these and similar
snippets.  We don't mention LOCAL etc. here either.

--- 34,41 ----
  <filename>postgresql.conf</filename>,
<filename>pg_hba.conf</filename>, and
  <filename>pg_ident.conf</filename> are traditionally stored in
  <varname>PGDATA</> (although in <productname>PostgreSQL</productname>
8.0 and
! later, it is possible to place them elsewhere). By default the
directory config is stored
! in <varname>PGDATA</>, however it should be kept along with
<filename>postgresql.conf</filename>.
  </para>

  <table tocentry="1" id="pgdata-contents-table">

This chunk doesn't make any sense to me.  "Should" is always tricky.
Why should I, and why might I not?

  <row>
+  <entry><filename>config</></entry>
+  <entry>Subdirectory containing automatically generated configuration
files</entry>
+ </row>
+
+ <row>
   <entry><filename>base</></entry>
   <entry>Subdirectory containing per-database subdirectories</entry>
  </row>

Only automatically generated ones?

        COPY_STRING_FIELD(name);
        COPY_NODE_FIELD(args);
        COPY_SCALAR_FIELD(is_local);
+       COPY_SCALAR_FIELD(is_persistent);

        return newnode;
  }

I suggest changing is_local into a new trivalued field that stores LOCAL
or SESSION or PERSISTENT.

                                        n->is_local = false;
                                        $$ = (Node *) n;
                                }
+                       | SET PERSISTENT set_persistent
+                               {
+                                       VariableSetStmt *n = $3;
+                                       n->is_persistent = true;
+                                       $$ = (Node *) n;
+                               }
                ;

  set_rest:

Why can't you use SET PERSISTENT set_rest?

*** a/src/backend/replication/basebackup.c
--- b/src/backend/replication/basebackup.c
***************
*** 755,760 **** sendDir(char *path, int basepathlen, bool sizeonly)
--- 755,766 ----
                                        strlen(PG_TEMP_FILE_PREFIX)) == 0)
                        continue;

+               /* skip auto conf temporary file */
+               if (strncmp(de->d_name,
+                                       PG_AUTOCONF_FILENAME ".",
+                                       sizeof(PG_AUTOCONF_FILENAME)) == 0)
+                       continue;
+

Maybe pg_basebackup should be taught to ignore certain kinds of
temporary files in general.  The file name shouldn't be hardcoded into
pg_basebackup.  This would effectively make the configuration file
naming scheme part of the replication protocol.  See other thread about
pg_basebackup client/server compatibility.  This needs to be generalized.

(Side thought: Does pg_basebackup copy editor temp and backup files?)

+           ereport(elevel,
+               (errmsg("Configuration parameters changed with SET
PERSISTENT command before start of server "
+                       "will not be effective as \"%s\"  file is not
accessible.", PG_AUTOCONF_FILENAME)));

I'm having trouble interpreting this: How can you change something with
SET PERSISTENT before the server starts?

Also: errmsg messages should start with a lowercase letter and should
generally be much shorter.  Please review other cases in your patch as well.

+   appendStringInfoString(&buf, "# Do not edit this file manually! "
+                      "It is overwritten by the SET PERSISTENT command
\n");

There is some punctuation or something missing at the end.

I suggest you break this into two lines.  It's pretty long.


I think the naming of these files is suboptimal:

+ #define PG_AUTOCONF_DIR                       "config"

"config" would seem to include pg_hba.conf and others.  Or
postgresql.conf for that matter.  Maybe I should move postgresql.conf
into config/.  Another great new FAQ.  The normal convention for this is
"postgresql.conf.d".  I don't see any reason not to use that.  One
reason that was brought up is that this doesn't match other things
currently in PGDATA, but (a) actually it does (hint: postgresql.conf),
and (b) neither does "config".

+ #define PG_AUTOCONF_FILENAME          "persistent.auto.conf"

"auto" sounds like the result of an automatic tuning tool.  Why not just
persistent.conf.  Or set-persistent.conf.  That makes the association
clear enough.

I don't know why the concept of a configuration directory is being
introduced here.  I mean, I'm all for that, but it seems completely
independent of this.  Some people wanted one setting per file, but
that's not what is happening here.  The name of the configuration file
is hardcoded throughout, so there is no value in reading all files from
a directory instead of just that one file.

There is a lot of confusion throughout the patch about whether this
config directory is for automatic files only or you can throw your own
stuff in there at will.  Another great FAQ.  And then the whole dance
about parsing postgresql.conf to make sure the include_dir is not
removed (=> FAQ).  Does it care whether it's at the beginning or the
end?  Should it?

How does all of this work when you move the configuration files to, say,
/etc/, but want to keep the SET PERSISTENT stuff under $PGDATA?  Lots of
moving parts there.

I think this could be much easier if you just read
$PGDATA/persistent.conf or whatever if present.  There is perhaps an
analogy to be found with the proposed new recovery.conf handling (e.g.,
always read if present after all other postgresql.conf processing or
something like that).

I'm thinking, this feature is ultimately intended to make things simpler
for average users.  But I see so many traps here.  It needs to be
simpler yet, and have less moving parts.

I'd almost like an option to turn this off for a server, because it has
so much potential for confusion.

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Greg Smith <greg@2ndquadrant.com> — 2013-04-02T20:53:05Z

On 4/1/13 5:44 AM, Amit Kapila wrote:

> I think in that case we can have 3 separate patches
> 1. Memory growth defect fix
> 2. Default postgresql.conf to include config directory and SET Persistent
> into single file implementation
> 3. Rearrangement of GUC validation into validate_conf_option function.
>
> As already there is review happened for point 2 and point 1 is an existing
> code defect fix, so in my opinion
> Patch 1 and 2 should be considered for 9.3.

They have been considered for 9.3, I just doubt they could get committed 
right now.  In order for this to go in as part of the very last 9.3 
feature changes (which is where we're at in the development cycle), 
you'd need to have a committer volunteer to take on the job of doing a 
second level of review on it.  And then that would have to happen 
without any other issues popping up.  That usually is not how it works. 
  Normally the first round of committer review finds another round of 
issues, and there's at least one more update before commit.  (Note that 
this is exactly what just happened today, with the review from Peter 
Eisentraut)

I'm not saying it's impossible for this feature to go in to 9.3, but I'm 
not seeing any committers volunteer to take on the job either.  I do 
want to see this feature go in--I'll update it for 9.4 even if you 
don't--but we're already into April.  There isn't much time left for 
9.3.  And the security release this week has gobbled up a good chunk of 
committer and packager time unexpectedly, which is just bad luck for 
your submission.

 From a process perspective, features that enter the last CF of a 
release that are very close to ready from the start have good odds of 
being committed.  You've done an excellent job of updating this in 
response to feedback, but it has involved a long list of changes so far. 
  It's fair to say this was still a rough feature at the start of CF 
2013-01, and now it's good but can be usefully polished a bit more.

For something of this size, going from rough feature to commit quality 
normally takes more than one CF.  I don't have any obvious errors to 
point out right now.  But I think there's still some room to improve on 
this before commit.  Andres mentioned on another thread that he thought 
merging some of your ideas with the version Zoltan did was useful to 
look at, and I was thinking of something similar.  This is close to 
being ready, and I hope you won't get discouraged just because it's 
probably going to slip to 9.4.

-- 
Greg Smith   2ndQuadrant US    greg@2ndQuadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Amit Kapila <amit.kapila@huawei.com> — 2013-04-03T06:24:56Z

On Tuesday, April 02, 2013 9:49 PM Peter Eisentraut wrote:
> I'm going to ignore most of the discussion that led up to this and give
> this patch a fresh look.

Thank you.

> + <screen>
> + SET PERSISTENT max_connections To 10;
> + </screen>
> 
> The To should probably be capitalized.

Okay, shall fix it.
 
> I doubt this example actually works because changing max_connections
> requires a restart.  Try to find a better example.

Any parameter's changed with this command can be effective either after
restart or SIGHUP.
So I will change it to SIGHUP parameter.

> It's weird that SET LOCAL and SET SESSION actually *set* the value, and
> the second key word determines how long the setting will last.  SET
> PERSISTENT doesn't actually set the value.  I predict that this will be
> a new favorite help-it-doesn't-work FAQ.
> 
>       <varlistentry>
>        <term><literal>SCHEMA</literal></term>
>        <listitem>
> !       <para><literal>SET [ PERSISTENT ] SCHEMA
> '<replaceable>value</>'</> is an alias for
>          <literal>SET search_path TO <replaceable>value</></>.  Only
> one
>          schema can be specified using this syntax.
>         </para>
> 
> I don't think [ PERSISTENT ] needs to be added in these and similar
> snippets.  We don't mention LOCAL etc. here either.

Agreed, shall fix this.

> --- 34,41 ----
>   <filename>postgresql.conf</filename>,
> <filename>pg_hba.conf</filename>, and
>   <filename>pg_ident.conf</filename> are traditionally stored in
>   <varname>PGDATA</> (although in <productname>PostgreSQL</productname>
> 8.0 and
> ! later, it is possible to place them elsewhere). By default the
> directory config is stored
> ! in <varname>PGDATA</>, however it should be kept along with
> <filename>postgresql.conf</filename>.
>   </para>
> 
>   <table tocentry="1" id="pgdata-contents-table">
> 
> This chunk doesn't make any sense to me.  "Should" is always tricky.
> Why should I, and why might I not?

I mean to say here, that user needs to move config directory and its
contents along with 
postgresql.conf. 
Shall we change as below: 
By default config directory is stored in PGDATA, however it needs to be kept
along with postgresql.conf


>   <row>
> +  <entry><filename>config</></entry>
> +  <entry>Subdirectory containing automatically generated configuration
> files</entry>
> + </row>
> +
> + <row>
>    <entry><filename>base</></entry>
>    <entry>Subdirectory containing per-database subdirectories</entry>
>   </row>
> 
> Only automatically generated ones?

No, any other files can also be present. 
How about change it as : 
Subdirectory containing generated configuration files. 
Any other suggestions?

This new directory's will be used to place generated files, 

>         COPY_STRING_FIELD(name);
>         COPY_NODE_FIELD(args);
>         COPY_SCALAR_FIELD(is_local);
> +       COPY_SCALAR_FIELD(is_persistent);
> 
>         return newnode;
>   }
> 
> I suggest changing is_local into a new trivalued field that stores
> LOCAL
> or SESSION or PERSISTENT.
> 
>                                         n->is_local = false;
>                                         $$ = (Node *) n;
>                                 }

Okay, I will change it.

> +                       | SET PERSISTENT set_persistent
> +                               {
> +                                       VariableSetStmt *n = $3;
> +                                       n->is_persistent = true;
> +                                       $$ = (Node *) n;
> +                               }
>                 ;
> 
>   set_rest:
> 
> Why can't you use SET PERSISTENT set_rest?

As SET PERSISTENT cannot be used with some of syntaxes, example (SESSION
AUTHORIZATION) and also it is not supportted inside transaction blocks.

> 
> *** a/src/backend/replication/basebackup.c
> --- b/src/backend/replication/basebackup.c
> ***************
> *** 755,760 **** sendDir(char *path, int basepathlen, bool sizeonly)
> --- 755,766 ----
>                                         strlen(PG_TEMP_FILE_PREFIX)) ==
> 0)
>                         continue;
> 
> +               /* skip auto conf temporary file */
> +               if (strncmp(de->d_name,
> +                                       PG_AUTOCONF_FILENAME ".",
> +                                       sizeof(PG_AUTOCONF_FILENAME))
> == 0)
> +                       continue;
> +
> 
> Maybe pg_basebackup should be taught to ignore certain kinds of
> temporary files in general.  The file name shouldn't be hardcoded into
> pg_basebackup.  This would effectively make the configuration file
> naming scheme part of the replication protocol.  See other thread about
> pg_basebackup client/server compatibility.  This needs to be
> generalized.

As we are just trying to ignore the file during backup, why it should effect
replication protocol? 
I mean protocol should be effected, if try to send something new or don't
send what is expected on the other side.
Also the same is done for team and backup label file.

> (Side thought: Does pg_basebackup copy editor temp and backup files?)

Currently pg_basebackup doesn't copy temp or backup files. The check I made
in code is similar to check for those two.

> 
> +           ereport(elevel,
> +               (errmsg("Configuration parameters changed with SET
> PERSISTENT command before start of server "
> +                       "will not be effective as \"%s\"  file is not
> accessible.", PG_AUTOCONF_FILENAME)));
> 
> I'm having trouble interpreting this: How can you change something with
> SET PERSISTENT before the server starts?

This is for the case when somebody starts the server second time, or i can
say restart server 
Example 
1. Startup Server, Connect client 
2. Change some settings with SET PERSISTENT 
3. Shutdown server 
4. Start Server; at this step if we don't find the file, the above error
message will appear.

> Also: errmsg messages should start with a lowercase letter and should
> generally be much shorter.  Please review other cases in your patch as
> well.

Okay. I shall take care of it.

> +   appendStringInfoString(&buf, "# Do not edit this file manually! "
> +                      "It is overwritten by the SET PERSISTENT command
> \n");
> 
> There is some punctuation or something missing at the end.
> 
> I suggest you break this into two lines.  It's pretty long.

Okay. I shall take care of it.
 
> 
> I think the naming of these files is suboptimal:
> 
> + #define PG_AUTOCONF_DIR                       "config"
> 
> "config" would seem to include pg_hba.conf and others.  Or
> postgresql.conf for that matter.  Maybe I should move postgresql.conf
> into config/.  Another great new FAQ.  The normal convention for this
> is
> "postgresql.conf.d".  I don't see any reason not to use that.  One
> reason that was brought up is that this doesn't match other things
> currently in PGDATA, but (a) actually it does (hint: postgresql.conf),
> and (b) neither does "config".

Reason was that "postgresql.conf.d" or "auto.conf.d" doesn't match existing
folder name, 
whereas config can have some similarity with global directory as far as I
can see. 

However changing name is not a big matter, it's more about for which name
there is more consensus.

> + #define PG_AUTOCONF_FILENAME          "persistent.auto.conf"
> 
> "auto" sounds like the result of an automatic tuning tool.  Why not
> just
> persistent.conf.  Or set-persistent.conf.  That makes the association
> clear enough.


The 'auto' keyword is kept to make it different from others, as it will be
generated by Command. 
It can give indication to user that this is not similar to other conf files
which he can directly edit.

> I don't know why the concept of a configuration directory is being
> introduced here.  I mean, I'm all for that, but it seems completely
> independent of this.  Some people wanted one setting per file, but
> that's not what is happening here.  The name of the configuration file
> is hardcoded throughout, so there is no value in reading all files from
> a directory instead of just that one file.
> 
> There is a lot of confusion throughout the patch about whether this
> config directory is for automatic files only or you can throw your own
> stuff in there at will.  Another great FAQ.  And then the whole dance
> about parsing postgresql.conf to make sure the include_dir is not
> removed (=> FAQ).  Does it care whether it's at the beginning or the
> end?  Should it?

This will also work as per existing behavior of #include_dir, which means
any configuration parameter values specified after this #include_dir 
will override the values set by SET PERSISTENT statement. The same is
mentioned as WARNING just before this include directive.

> How does all of this work when you move the configuration files to,
> say,
> /etc/, but want to keep the SET PERSISTENT stuff under $PGDATA?  Lots
> of
> moving parts there.

He needs to move config directory along with postgresql.conf.

> I think this could be much easier if you just read
> $PGDATA/persistent.conf or whatever if present.  There is perhaps an
> analogy to be found with the proposed new recovery.conf handling (e.g.,
> always read if present after all other postgresql.conf processing or
> something like that).

In current mechanism, user can revert to old usage (edit every thing in
postgresql.conf) just by commenting or removing 
include_dir = 'config' in postgresql.conf. 
OTOH if we just try to read if present mechanism, we might not be able to
tell to user that his file is missing and if he has made 
any changes by SET PERSISTENT they will not reflect after this startup.

> I'm thinking, this feature is ultimately intended to make things
> simpler
> for average users.  But I see so many traps here.  It needs to be
> simpler yet, and have less moving parts.
> 
> I'd almost like an option to turn this off for a server, because it has
> so much potential for confusion.

If user comments or removes below line in postgresql.conf, then this feature
will be OFF 
include_dir = 'config'


With Regards,
Amit Kapila.

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Amit Kapila <amit.kapila@huawei.com> — 2013-04-03T06:32:52Z

On Wednesday, April 03, 2013 2:23 AM Greg Smith wrote:
> On 4/1/13 5:44 AM, Amit Kapila wrote:
> 
> > I think in that case we can have 3 separate patches
> > 1. Memory growth defect fix
> > 2. Default postgresql.conf to include config directory and SET
> Persistent
> > into single file implementation
> > 3. Rearrangement of GUC validation into validate_conf_option
> function.
> >
> > As already there is review happened for point 2 and point 1 is an
> existing
> > code defect fix, so in my opinion
> > Patch 1 and 2 should be considered for 9.3.
> 
> They have been considered for 9.3, I just doubt they could get
> committed
> right now.  In order for this to go in as part of the very last 9.3
> feature changes (which is where we're at in the development cycle),
> you'd need to have a committer volunteer to take on the job of doing a
> second level of review on it.  And then that would have to happen
> without any other issues popping up.  That usually is not how it works.
>   Normally the first round of committer review finds another round of
> issues, and there's at least one more update before commit.  (Note that
> this is exactly what just happened today, with the review from Peter
> Eisentraut)
> 
> I'm not saying it's impossible for this feature to go in to 9.3, but
> I'm
> not seeing any committers volunteer to take on the job either.  I do
> want to see this feature go in--I'll update it for 9.4 even if you
> don't--but we're already into April.  There isn't much time left for
> 9.3.  And the security release this week has gobbled up a good chunk of
> committer and packager time unexpectedly, which is just bad luck for
> your submission.
> 
>  From a process perspective, features that enter the last CF of a
> release that are very close to ready from the start have good odds of
> being committed.  You've done an excellent job of updating this in
> response to feedback, but it has involved a long list of changes so
> far.
>   It's fair to say this was still a rough feature at the start of CF
> 2013-01, and now it's good but can be usefully polished a bit more.
> 
> For something of this size, going from rough feature to commit quality
> normally takes more than one CF.  I don't have any obvious errors to
> point out right now.  But I think there's still some room to improve on
> this before commit.  Andres mentioned on another thread that he thought
> merging some of your ideas with the version Zoltan did was useful to
> look at, and I was thinking of something similar.  This is close to
> being ready, and I hope you won't get discouraged just because it's
> probably going to slip to 9.4.

Thank you for keeping me motivated for this patch now and throughout the
last CF by providing valuable suggestions and comments.


With Regards,
Amit Kapila.

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Robert Haas <robertmhaas@gmail.com> — 2013-04-03T17:24:55Z

On Tue, Apr 2, 2013 at 12:19 PM, Peter Eisentraut <peter_e@gmx.net> wrote:
> It's weird that SET LOCAL and SET SESSION actually *set* the value, and
> the second key word determines how long the setting will last.  SET
> PERSISTENT doesn't actually set the value.  I predict that this will be
> a new favorite help-it-doesn't-work FAQ.

I think this is another argument against this particular syntax.  I
have always thought that something along the lines of ALTER SYSTEM
would be more appropriate.  ALTER DATABASE .. SET and ALTER ROLE ..
SET don't change the value immediately either, and nobody gets
confused about that to my knowledge.  But I can see where SET
PERSISTENT could cause that sort of confusion.

...Robert

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Tom Lane <tgl@sss.pgh.pa.us> — 2013-04-03T18:54:01Z

Robert Haas <robertmhaas@gmail.com> writes:
> On Tue, Apr 2, 2013 at 12:19 PM, Peter Eisentraut <peter_e@gmx.net> wrote:
>> It's weird that SET LOCAL and SET SESSION actually *set* the value, and
>> the second key word determines how long the setting will last.  SET
>> PERSISTENT doesn't actually set the value.  I predict that this will be
>> a new favorite help-it-doesn't-work FAQ.

> I think this is another argument against this particular syntax.  I
> have always thought that something along the lines of ALTER SYSTEM
> would be more appropriate.  ALTER DATABASE .. SET and ALTER ROLE ..
> SET don't change the value immediately either, and nobody gets
> confused about that to my knowledge.  But I can see where SET
> PERSISTENT could cause that sort of confusion.

Yeah, I think I argued for using the SET syntax to start with, but
I'm coming around to the position that SET PERSISTENT is too much
unlike the behavior of other varieties of SET.  ALTER is sounding
more attractive to me now.  Not sure about "ALTER SYSTEM" in particular
though --- it's not clear that that has any real merit other than
already existing as a keyword.  (Not that that's negligible.)
ALTER CONFIGURATION is another alternative using an existing keyword
that might be worth considering.

			regards, tom lane

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Robert Haas <robertmhaas@gmail.com> — 2013-04-03T21:21:36Z

On Wed, Apr 3, 2013 at 2:54 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Robert Haas <robertmhaas@gmail.com> writes:
>> On Tue, Apr 2, 2013 at 12:19 PM, Peter Eisentraut <peter_e@gmx.net> wrote:
>>> It's weird that SET LOCAL and SET SESSION actually *set* the value, and
>>> the second key word determines how long the setting will last.  SET
>>> PERSISTENT doesn't actually set the value.  I predict that this will be
>>> a new favorite help-it-doesn't-work FAQ.
>
>> I think this is another argument against this particular syntax.  I
>> have always thought that something along the lines of ALTER SYSTEM
>> would be more appropriate.  ALTER DATABASE .. SET and ALTER ROLE ..
>> SET don't change the value immediately either, and nobody gets
>> confused about that to my knowledge.  But I can see where SET
>> PERSISTENT could cause that sort of confusion.
>
> Yeah, I think I argued for using the SET syntax to start with, but
> I'm coming around to the position that SET PERSISTENT is too much
> unlike the behavior of other varieties of SET.  ALTER is sounding
> more attractive to me now.  Not sure about "ALTER SYSTEM" in particular
> though --- it's not clear that that has any real merit other than
> already existing as a keyword.  (Not that that's negligible.)
> ALTER CONFIGURATION is another alternative using an existing keyword
> that might be worth considering.

Yeah, I thought about something like that.  Aside from saving on
keywords, the reason I like ALTER SYSTEM or similar is that I suspect
there will be other system-wide things that we may want to let people
ALTER in the future, so I think that route might avoid an unnecessary
proliferation of top-level commands.  I am not, however, deadly
attached to the idea, if someone's got a good reason for preferring
something else.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Amit Kapila <amit.kapila@huawei.com> — 2013-04-04T09:01:38Z

On Thursday, April 04, 2013 2:52 AM Robert Haas wrote:
> On Wed, Apr 3, 2013 at 2:54 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> > Robert Haas <robertmhaas@gmail.com> writes:
> >> On Tue, Apr 2, 2013 at 12:19 PM, Peter Eisentraut <peter_e@gmx.net>
> wrote:
> >>> It's weird that SET LOCAL and SET SESSION actually *set* the value,
> and
> >>> the second key word determines how long the setting will last.  SET
> >>> PERSISTENT doesn't actually set the value.  I predict that this
> will be
> >>> a new favorite help-it-doesn't-work FAQ.
> >
> >> I think this is another argument against this particular syntax.  I
> >> have always thought that something along the lines of ALTER SYSTEM
> >> would be more appropriate.  ALTER DATABASE .. SET and ALTER ROLE ..
> >> SET don't change the value immediately either, and nobody gets
> >> confused about that to my knowledge.  But I can see where SET
> >> PERSISTENT could cause that sort of confusion.
> >
> > Yeah, I think I argued for using the SET syntax to start with, but
> > I'm coming around to the position that SET PERSISTENT is too much
> > unlike the behavior of other varieties of SET.  ALTER is sounding
> > more attractive to me now.  Not sure about "ALTER SYSTEM" in
> particular
> > though --- it's not clear that that has any real merit other than
> > already existing as a keyword.  (Not that that's negligible.)
> > ALTER CONFIGURATION is another alternative using an existing keyword
> > that might be worth considering.
> 
> Yeah, I thought about something like that.  Aside from saving on
> keywords, the reason I like ALTER SYSTEM or similar is that I suspect
> there will be other system-wide things that we may want to let people
> ALTER in the future, so I think that route might avoid an unnecessary
> proliferation of top-level commands.  I am not, however, deadly
> attached to the idea, if someone's got a good reason for preferring
> something else.

I think second parameter in SET command telling the scope should be fine. As
I could see Oracle
also has similar syntax for it's ALTER SYSTEM Command (Alter System Scope
[Memory|Spfile|Both]). 
Description in short:
SPFILE indicates that the change is made in the server parameter file. The
new 
setting takes effect when the database is next shut down and started up
again.
MEMORY indicates that the change is made in memory, takes effect
immediately, 
and persists until the database is shut down.

The only reason to show above example is that second parameter telling Scope
exists in other databases as well.


However if you are not convinced with above reasoning, then the Alter syntax
can be as follows:
ALTER SYSTEM SET configuration_parameter {TO | =} {value, | 'value'};

With Regards,
Amit Kapila.

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Amit Kapila <amit.kapila@huawei.com> — 2013-05-27T10:47:19Z

On Wednesday, April 03, 2013 11:55 AM Amit Kapila wote:
> On Tuesday, April 02, 2013 9:49 PM Peter Eisentraut wrote:

This feature was discussed for 9.3, but couldn't get committed. 
History for this patch is that in the last, Peter Eisentraut has given quite
a few review comments, most of which I have closed in the patch attached.

Apart from that Robert and Tom intended to use ALTER SYSTEM or variant of
ALTER rather than SET. Syntax can be as below:

ALTER SYSTEM SET configuration_parameter {TO | =} {value, | 'value'};

Some Comments of Peter E which needs to be discussed before changing are as
below:

1) Name of directory should be postgresql.conf.d rather than config.
2) Name of the file persistent.auto.conf should not have auto word.

I have not modified the code, as these had been previously discussed in this
mail chain and as a conclusion of that, I had kept the current names for
file and directory.

> > I'm going to ignore most of the discussion that led up to this and
> give
> > this patch a fresh look.
> 
> Thank you.
> 
> > + <screen>
> > + SET PERSISTENT max_connections To 10;
> > + </screen>
> >
> > The To should probably be capitalized.
> 
> Okay, shall fix it.

Changed to 
SET PERSISTENT checkpoint_timeout TO 600

> > I doubt this example actually works because changing max_connections
> > requires a restart.  Try to find a better example.
> 
> Any parameter's changed with this command can be effective either after
> restart or SIGHUP.
> So I will change it to SIGHUP parameter.
> 
> > It's weird that SET LOCAL and SET SESSION actually *set* the value,
> and
> > the second key word determines how long the setting will last.  SET
> > PERSISTENT doesn't actually set the value.  I predict that this will
> be
> > a new favorite help-it-doesn't-work FAQ.
> >
> >       <varlistentry>
> >        <term><literal>SCHEMA</literal></term>
> >        <listitem>
> > !       <para><literal>SET [ PERSISTENT ] SCHEMA
> > '<replaceable>value</>'</> is an alias for
> >          <literal>SET search_path TO <replaceable>value</></>.  Only
> > one
> >          schema can be specified using this syntax.
> >         </para>
> >
> > I don't think [ PERSISTENT ] needs to be added in these and similar
> > snippets.  We don't mention LOCAL etc. here either.
> 
> Agreed, shall fix this.

Removed from all similar places.

> > --- 34,41 ----
> >   <filename>postgresql.conf</filename>,
> > <filename>pg_hba.conf</filename>, and
> >   <filename>pg_ident.conf</filename> are traditionally stored in
> >   <varname>PGDATA</> (although in
> <productname>PostgreSQL</productname>
> > 8.0 and
> > ! later, it is possible to place them elsewhere). By default the
> > directory config is stored
> > ! in <varname>PGDATA</>, however it should be kept along with
> > <filename>postgresql.conf</filename>.
> >   </para>
> >
> >   <table tocentry="1" id="pgdata-contents-table">
> >
> > This chunk doesn't make any sense to me.  "Should" is always tricky.
> > Why should I, and why might I not?
> 
> I mean to say here, that user needs to move config directory and its
> contents along with
> postgresql.conf.
> Shall we change as below:
> By default config directory is stored in PGDATA, however it needs to be
> kept along with postgresql.conf

Changed as proposed above.

> 
> >   <row>
> > +  <entry><filename>config</></entry>
> > +  <entry>Subdirectory containing automatically generated
> configuration
> > files</entry>
> > + </row>
> > +
> > + <row>
> >    <entry><filename>base</></entry>
> >    <entry>Subdirectory containing per-database subdirectories</entry>
> >   </row>
> >
> > Only automatically generated ones?
> 
> No, any other files can also be present.
> How about change it as :
> Subdirectory containing generated configuration files.
> Any other suggestions?

Changed as above.

> This new directory's will be used to place generated files,
> 
> >         COPY_STRING_FIELD(name);
> >         COPY_NODE_FIELD(args);
> >         COPY_SCALAR_FIELD(is_local);
> > +       COPY_SCALAR_FIELD(is_persistent);
> >
> >         return newnode;
> >   }
> >
> > I suggest changing is_local into a new trivalued field that stores
> > LOCAL
> > or SESSION or PERSISTENT.
> >
> >                                         n->is_local = false;
> >                                         $$ = (Node *) n;
> >                                 }
> 
> Okay, I will change it.
> 
> > +                       | SET PERSISTENT set_persistent
> > +                               {
> > +                                       VariableSetStmt *n = $3;
> > +                                       n->is_persistent = true;
> > +                                       $$ = (Node *) n;
> > +                               }
> >                 ;
> >
> >   set_rest:
> >
> > Why can't you use SET PERSISTENT set_rest?
> 
> As SET PERSISTENT cannot be used with some of syntaxes, example
> (SESSION AUTHORIZATION) and also it is not supportted inside
> transaction blocks.
> 
> >
> > *** a/src/backend/replication/basebackup.c
> > --- b/src/backend/replication/basebackup.c
> > ***************
> > *** 755,760 **** sendDir(char *path, int basepathlen, bool sizeonly)
> > --- 755,766 ----
> >                                         strlen(PG_TEMP_FILE_PREFIX))
> ==
> > 0)
> >                         continue;
> >
> > +               /* skip auto conf temporary file */
> > +               if (strncmp(de->d_name,
> > +                                       PG_AUTOCONF_FILENAME ".",
> > +                                       sizeof(PG_AUTOCONF_FILENAME))
> > == 0)
> > +                       continue;
> > +
> >
> > Maybe pg_basebackup should be taught to ignore certain kinds of
> > temporary files in general.  The file name shouldn't be hardcoded
> into
> > pg_basebackup.  This would effectively make the configuration file
> > naming scheme part of the replication protocol.  See other thread
> about
> > pg_basebackup client/server compatibility.  This needs to be
> > generalized.
> 
> As we are just trying to ignore the file during backup, why it should
> effect replication protocol?
> I mean protocol should be effected, if try to send something new or
> don't send what is expected on the other side.
> Also the same is done for team and backup label file.
> 
> > (Side thought: Does pg_basebackup copy editor temp and backup files?)
> 
> Currently pg_basebackup doesn't copy temp or backup files. The check I
> made in code is similar to check for those two.
> 
> >
> > +           ereport(elevel,
> > +               (errmsg("Configuration parameters changed with SET
> > PERSISTENT command before start of server "
> > +                       "will not be effective as \"%s\"  file is not
> > accessible.", PG_AUTOCONF_FILENAME)));
> >
> > I'm having trouble interpreting this: How can you change something
> with
> > SET PERSISTENT before the server starts?
> 
> This is for the case when somebody starts the server second time, or i
> can say restart server
> Example
> 1. Startup Server, Connect client
> 2. Change some settings with SET PERSISTENT
> 3. Shutdown server
> 4. Start Server; at this step if we don't find the file, the above
> error message will appear.
> 
> > Also: errmsg messages should start with a lowercase letter and should
> > generally be much shorter.  Please review other cases in your patch
> as
> > well.
> 
> Okay. I shall take care of it.

Done, all error messages started with lower case letter. Apart from above
message, all other error messages are short, do you feel any other needs to
be shortened.

> > +   appendStringInfoString(&buf, "# Do not edit this file manually! "
> > +                      "It is overwritten by the SET PERSISTENT
> command
> > \n");
> >
> > There is some punctuation or something missing at the end.
> >
> > I suggest you break this into two lines.  It's pretty long.
> 
> Okay. I shall take care of it.


Done, broken into 2 lines.

> 
> >
> > I think the naming of these files is suboptimal:
> >
> > + #define PG_AUTOCONF_DIR                       "config"
> >
> > "config" would seem to include pg_hba.conf and others.  Or
> > postgresql.conf for that matter.  Maybe I should move postgresql.conf
> > into config/.  Another great new FAQ.  The normal convention for this
> > is
> > "postgresql.conf.d".  I don't see any reason not to use that.  One
> > reason that was brought up is that this doesn't match other things
> > currently in PGDATA, but (a) actually it does (hint:
> postgresql.conf),
> > and (b) neither does "config".
> 
> Reason was that "postgresql.conf.d" or "auto.conf.d" doesn't match
> existing folder name,
> whereas config can have some similarity with global directory as far as
> I can see.
> 
> However changing name is not a big matter, it's more about for which
> name there is more consensus.
> 
> > + #define PG_AUTOCONF_FILENAME          "persistent.auto.conf"
> >
> > "auto" sounds like the result of an automatic tuning tool.  Why not
> > just
> > persistent.conf.  Or set-persistent.conf.  That makes the association
> > clear enough.
> 
> 
> The 'auto' keyword is kept to make it different from others, as it will
> be generated by Command.
> It can give indication to user that this is not similar to other conf
> files which he can directly edit.
> 
> > I don't know why the concept of a configuration directory is being
> > introduced here.  I mean, I'm all for that, but it seems completely
> > independent of this.  Some people wanted one setting per file, but
> > that's not what is happening here.  The name of the configuration
> file
> > is hardcoded throughout, so there is no value in reading all files
> from
> > a directory instead of just that one file.
> >
> > There is a lot of confusion throughout the patch about whether this
> > config directory is for automatic files only or you can throw your
> own
> > stuff in there at will.  Another great FAQ.  And then the whole dance
> > about parsing postgresql.conf to make sure the include_dir is not
> > removed (=> FAQ).  Does it care whether it's at the beginning or the
> > end?  Should it?
> 
> This will also work as per existing behavior of #include_dir, which
> means any configuration parameter values specified after this
> #include_dir
> will override the values set by SET PERSISTENT statement. The same is
> mentioned as WARNING just before this include directive.
> 
> > How does all of this work when you move the configuration files to,
> > say,
> > /etc/, but want to keep the SET PERSISTENT stuff under $PGDATA?  Lots
> > of
> > moving parts there.
> 
> He needs to move config directory along with postgresql.conf.
> 
> > I think this could be much easier if you just read
> > $PGDATA/persistent.conf or whatever if present.  There is perhaps an
> > analogy to be found with the proposed new recovery.conf handling
> (e.g.,
> > always read if present after all other postgresql.conf processing or
> > something like that).
> 
> In current mechanism, user can revert to old usage (edit every thing in
> postgresql.conf) just by commenting or removing
> include_dir = 'config' in postgresql.conf.
> OTOH if we just try to read if present mechanism, we might not be able
> to tell to user that his file is missing and if he has made
> any changes by SET PERSISTENT they will not reflect after this startup.
> 
> > I'm thinking, this feature is ultimately intended to make things
> > simpler
> > for average users.  But I see so many traps here.  It needs to be
> > simpler yet, and have less moving parts.
> >
> > I'd almost like an option to turn this off for a server, because it
> has
> > so much potential for confusion.
> 
> If user comments or removes below line in postgresql.conf, then this
> feature will be OFF
> include_dir = 'config'

Do you think above way to switch OFF the feature is acceptable or do you
have any other idea?


Suggestions/Feedback?


With Regards,
Amit Kapila.

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Amit Kapila <amit.kapila@huawei.com> — 2013-06-05T11:24:28Z

On Monday, May 27, 2013 4:17 PM Amit Kapila wrote:
> On Wednesday, April 03, 2013 11:55 AM Amit Kapila wote:
> > On Tuesday, April 02, 2013 9:49 PM Peter Eisentraut wrote:
> 

There are 2 options to proceed for this patch for 9.4

1. Upload the SET PERSISTENT syntax patch for coming CF by fixing existing
review comments
2. Implement new syntax ALTER SYSTEM as proposed in below mail

Could you suggest me what could be best way to proceed for this patch?

> This feature was discussed for 9.3, but couldn't get committed.
> History for this patch is that in the last, Peter Eisentraut has given
> quite a few review comments, most of which I have closed in the patch
> attached.
> 
> Apart from that Robert and Tom intended to use ALTER SYSTEM or variant
> of ALTER rather than SET. Syntax can be as below:
> 
> ALTER SYSTEM SET configuration_parameter {TO | =} {value, | 'value'};
> 
> Some Comments of Peter E which needs to be discussed before changing
> are as
> below:
> 
> 1) Name of directory should be postgresql.conf.d rather than config.
> 2) Name of the file persistent.auto.conf should not have auto word.
> 
> I have not modified the code, as these had been previously discussed in
> this mail chain and as a conclusion of that, I had kept the current
> names for file and directory.
> 
> > > I'm going to ignore most of the discussion that led up to this and
> > give
> > > this patch a fresh look.
> >
> > Thank you.
> >
> > > + <screen>
> > > + SET PERSISTENT max_connections To 10; </screen>
> > >
> > > The To should probably be capitalized.
> >
> > Okay, shall fix it.
> 
> Changed to
> SET PERSISTENT checkpoint_timeout TO 600
> 
> > > I doubt this example actually works because changing
> max_connections
> > > requires a restart.  Try to find a better example.
> >
> > Any parameter's changed with this command can be effective either
> > after restart or SIGHUP.
> > So I will change it to SIGHUP parameter.
> >
> > > It's weird that SET LOCAL and SET SESSION actually *set* the value,
> > and
> > > the second key word determines how long the setting will last.  SET
> > > PERSISTENT doesn't actually set the value.  I predict that this
> will
> > be
> > > a new favorite help-it-doesn't-work FAQ.
> > >
> > >       <varlistentry>
> > >        <term><literal>SCHEMA</literal></term>
> > >        <listitem>
> > > !       <para><literal>SET [ PERSISTENT ] SCHEMA
> > > '<replaceable>value</>'</> is an alias for
> > >          <literal>SET search_path TO <replaceable>value</></>.
> Only
> > > one
> > >          schema can be specified using this syntax.
> > >         </para>
> > >
> > > I don't think [ PERSISTENT ] needs to be added in these and similar
> > > snippets.  We don't mention LOCAL etc. here either.
> >
> > Agreed, shall fix this.
> 
> Removed from all similar places.
> 
> > > --- 34,41 ----
> > >   <filename>postgresql.conf</filename>,
> > > <filename>pg_hba.conf</filename>, and
> > >   <filename>pg_ident.conf</filename> are traditionally stored in
> > >   <varname>PGDATA</> (although in
> > <productname>PostgreSQL</productname>
> > > 8.0 and
> > > ! later, it is possible to place them elsewhere). By default the
> > > directory config is stored ! in <varname>PGDATA</>, however it
> > > should be kept along with <filename>postgresql.conf</filename>.
> > >   </para>
> > >
> > >   <table tocentry="1" id="pgdata-contents-table">
> > >
> > > This chunk doesn't make any sense to me.  "Should" is always
> tricky.
> > > Why should I, and why might I not?
> >
> > I mean to say here, that user needs to move config directory and its
> > contents along with postgresql.conf.
> > Shall we change as below:
> > By default config directory is stored in PGDATA, however it needs to
> > be kept along with postgresql.conf
> 
> Changed as proposed above.
> 
> >
> > >   <row>
> > > +  <entry><filename>config</></entry>
> > > +  <entry>Subdirectory containing automatically generated
> > configuration
> > > files</entry>
> > > + </row>
> > > +
> > > + <row>
> > >    <entry><filename>base</></entry>
> > >    <entry>Subdirectory containing per-database
> subdirectories</entry>
> > >   </row>
> > >
> > > Only automatically generated ones?
> >
> > No, any other files can also be present.
> > How about change it as :
> > Subdirectory containing generated configuration files.
> > Any other suggestions?
> 
> Changed as above.
> 
> > This new directory's will be used to place generated files,
> >
> > >         COPY_STRING_FIELD(name);
> > >         COPY_NODE_FIELD(args);
> > >         COPY_SCALAR_FIELD(is_local);
> > > +       COPY_SCALAR_FIELD(is_persistent);
> > >
> > >         return newnode;
> > >   }
> > >
> > > I suggest changing is_local into a new trivalued field that stores
> > > LOCAL or SESSION or PERSISTENT.
> > >
> > >                                         n->is_local = false;
> > >                                         $$ = (Node *) n;
> > >                                 }
> >
> > Okay, I will change it.
> >
> > > +                       | SET PERSISTENT set_persistent
> > > +                               {
> > > +                                       VariableSetStmt *n = $3;
> > > +                                       n->is_persistent = true;
> > > +                                       $$ = (Node *) n;
> > > +                               }
> > >                 ;
> > >
> > >   set_rest:
> > >
> > > Why can't you use SET PERSISTENT set_rest?
> >
> > As SET PERSISTENT cannot be used with some of syntaxes, example
> > (SESSION AUTHORIZATION) and also it is not supportted inside
> > transaction blocks.
> >
> > >
> > > *** a/src/backend/replication/basebackup.c
> > > --- b/src/backend/replication/basebackup.c
> > > ***************
> > > *** 755,760 **** sendDir(char *path, int basepathlen, bool
> sizeonly)
> > > --- 755,766 ----
> > >
> strlen(PG_TEMP_FILE_PREFIX))
> > ==
> > > 0)
> > >                         continue;
> > >
> > > +               /* skip auto conf temporary file */
> > > +               if (strncmp(de->d_name,
> > > +                                       PG_AUTOCONF_FILENAME ".",
> > > +
> > > + sizeof(PG_AUTOCONF_FILENAME))
> > > == 0)
> > > +                       continue;
> > > +
> > >
> > > Maybe pg_basebackup should be taught to ignore certain kinds of
> > > temporary files in general.  The file name shouldn't be hardcoded
> > into
> > > pg_basebackup.  This would effectively make the configuration file
> > > naming scheme part of the replication protocol.  See other thread
> > about
> > > pg_basebackup client/server compatibility.  This needs to be
> > > generalized.
> >
> > As we are just trying to ignore the file during backup, why it should
> > effect replication protocol?
> > I mean protocol should be effected, if try to send something new or
> > don't send what is expected on the other side.
> > Also the same is done for team and backup label file.
> >
> > > (Side thought: Does pg_basebackup copy editor temp and backup
> > > files?)
> >
> > Currently pg_basebackup doesn't copy temp or backup files. The check
> I
> > made in code is similar to check for those two.
> >
> > >
> > > +           ereport(elevel,
> > > +               (errmsg("Configuration parameters changed with SET
> > > PERSISTENT command before start of server "
> > > +                       "will not be effective as \"%s\"  file is
> > > + not
> > > accessible.", PG_AUTOCONF_FILENAME)));
> > >
> > > I'm having trouble interpreting this: How can you change something
> > with
> > > SET PERSISTENT before the server starts?
> >
> > This is for the case when somebody starts the server second time, or
> i
> > can say restart server Example 1. Startup Server, Connect client 2.
> > Change some settings with SET PERSISTENT 3. Shutdown server 4. Start
> > Server; at this step if we don't find the file, the above error
> > message will appear.
> >
> > > Also: errmsg messages should start with a lowercase letter and
> > > should generally be much shorter.  Please review other cases in
> your
> > > patch
> > as
> > > well.
> >
> > Okay. I shall take care of it.
> 
> Done, all error messages started with lower case letter. Apart from
> above message, all other error messages are short, do you feel any
> other needs to be shortened.
> 
> > > +   appendStringInfoString(&buf, "# Do not edit this file manually!
> "
> > > +                      "It is overwritten by the SET PERSISTENT
> > command
> > > \n");
> > >
> > > There is some punctuation or something missing at the end.
> > >
> > > I suggest you break this into two lines.  It's pretty long.
> >
> > Okay. I shall take care of it.
> 
> 
> Done, broken into 2 lines.
> 
> >
> > >
> > > I think the naming of these files is suboptimal:
> > >
> > > + #define PG_AUTOCONF_DIR                       "config"
> > >
> > > "config" would seem to include pg_hba.conf and others.  Or
> > > postgresql.conf for that matter.  Maybe I should move
> > > postgresql.conf into config/.  Another great new FAQ.  The normal
> > > convention for this is "postgresql.conf.d".  I don't see any reason
> > > not to use that.  One reason that was brought up is that this
> > > doesn't match other things currently in PGDATA, but (a) actually it
> > > does (hint:
> > postgresql.conf),
> > > and (b) neither does "config".
> >
> > Reason was that "postgresql.conf.d" or "auto.conf.d" doesn't match
> > existing folder name, whereas config can have some similarity with
> > global directory as far as I can see.
> >
> > However changing name is not a big matter, it's more about for which
> > name there is more consensus.
> >
> > > + #define PG_AUTOCONF_FILENAME          "persistent.auto.conf"
> > >
> > > "auto" sounds like the result of an automatic tuning tool.  Why not
> > > just persistent.conf.  Or set-persistent.conf.  That makes the
> > > association clear enough.
> >
> >
> > The 'auto' keyword is kept to make it different from others, as it
> > will be generated by Command.
> > It can give indication to user that this is not similar to other conf
> > files which he can directly edit.
> >
> > > I don't know why the concept of a configuration directory is being
> > > introduced here.  I mean, I'm all for that, but it seems completely
> > > independent of this.  Some people wanted one setting per file, but
> > > that's not what is happening here.  The name of the configuration
> > file
> > > is hardcoded throughout, so there is no value in reading all files
> > from
> > > a directory instead of just that one file.
> > >
> > > There is a lot of confusion throughout the patch about whether this
> > > config directory is for automatic files only or you can throw your
> > own
> > > stuff in there at will.  Another great FAQ.  And then the whole
> > > dance about parsing postgresql.conf to make sure the include_dir is
> > > not removed (=> FAQ).  Does it care whether it's at the beginning
> or
> > > the end?  Should it?
> >
> > This will also work as per existing behavior of #include_dir, which
> > means any configuration parameter values specified after this
> > #include_dir will override the values set by SET PERSISTENT
> statement.
> > The same is mentioned as WARNING just before this include directive.
> >
> > > How does all of this work when you move the configuration files to,
> > > say, /etc/, but want to keep the SET PERSISTENT stuff under
> $PGDATA?
> > > Lots of moving parts there.
> >
> > He needs to move config directory along with postgresql.conf.
> >
> > > I think this could be much easier if you just read
> > > $PGDATA/persistent.conf or whatever if present.  There is perhaps
> an
> > > analogy to be found with the proposed new recovery.conf handling
> > (e.g.,
> > > always read if present after all other postgresql.conf processing
> or
> > > something like that).
> >
> > In current mechanism, user can revert to old usage (edit every thing
> > in
> > postgresql.conf) just by commenting or removing include_dir =
> 'config'
> > in postgresql.conf.
> > OTOH if we just try to read if present mechanism, we might not be
> able
> > to tell to user that his file is missing and if he has made any
> > changes by SET PERSISTENT they will not reflect after this startup.
> >
> > > I'm thinking, this feature is ultimately intended to make things
> > > simpler for average users.  But I see so many traps here.  It needs
> > > to be simpler yet, and have less moving parts.
> > >
> > > I'd almost like an option to turn this off for a server, because it
> > has
> > > so much potential for confusion.
> >
> > If user comments or removes below line in postgresql.conf, then this
> > feature will be OFF include_dir = 'config'
> 
> Do you think above way to switch OFF the feature is acceptable or do
> you have any other idea?
> 
> 
> Suggestions/Feedback?

With Regards,
Amit Kapila.

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Robert Haas <robertmhaas@gmail.com> — 2013-06-06T16:51:38Z

On Wed, Jun 5, 2013 at 7:24 AM, Amit Kapila <amit.kapila@huawei.com> wrote:
> On Monday, May 27, 2013 4:17 PM Amit Kapila wrote:
>> On Wednesday, April 03, 2013 11:55 AM Amit Kapila wote:
>> > On Tuesday, April 02, 2013 9:49 PM Peter Eisentraut wrote:
>>
>
> There are 2 options to proceed for this patch for 9.4
>
> 1. Upload the SET PERSISTENT syntax patch for coming CF by fixing existing
> review comments
> 2. Implement new syntax ALTER SYSTEM as proposed in below mail
>
> Could you suggest me what could be best way to proceed for this patch?

I'm still in favor of some syntax involving ALTER, because it's still
true that this behaves more like the existing GUC-setting commands
that use ALTER (which change configuration for future sessions) rather
the ones that use SET (which change the current settings for some
period of time).

But I can't speak for people who are not me.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Amit Kapila <amit.kapila@huawei.com> — 2013-06-07T04:14:42Z

On Thursday, June 06, 2013 10:22 PM Robert Haas wrote:
> On Wed, Jun 5, 2013 at 7:24 AM, Amit Kapila <amit.kapila@huawei.com>
> wrote:
> > On Monday, May 27, 2013 4:17 PM Amit Kapila wrote:
> >> On Wednesday, April 03, 2013 11:55 AM Amit Kapila wote:
> >> > On Tuesday, April 02, 2013 9:49 PM Peter Eisentraut wrote:
> >>
> >
> > There are 2 options to proceed for this patch for 9.4
> >
> > 1. Upload the SET PERSISTENT syntax patch for coming CF by fixing
> existing
> > review comments
> > 2. Implement new syntax ALTER SYSTEM as proposed in below mail
> >
> > Could you suggest me what could be best way to proceed for this
> patch?
> 
> I'm still in favor of some syntax involving ALTER, because it's still
> true that this behaves more like the existing GUC-setting commands
> that use ALTER (which change configuration for future sessions) rather
> the ones that use SET (which change the current settings for some
> period of time).


I will change the patch as per below syntax if there are no objections:

 ALTER SYSTEM SET configuration_parameter {TO | =} {value, | 'value'};


With Regards,
Amit Kapila.

ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila@huawei.com> — 2013-06-13T12:35:17Z

On Friday, June 07, 2013 9:45 AM Amit Kapila wrote:
> On Thursday, June 06, 2013 10:22 PM Robert Haas wrote:
> > On Wed, Jun 5, 2013 at 7:24 AM, Amit Kapila <amit.kapila@huawei.com>
> > wrote:
> > > On Monday, May 27, 2013 4:17 PM Amit Kapila wrote:
> > >> On Wednesday, April 03, 2013 11:55 AM Amit Kapila wote:
> > >> > On Tuesday, April 02, 2013 9:49 PM Peter Eisentraut wrote:
> > >>
> > >
> > > There are 2 options to proceed for this patch for 9.4
> > >
> > > 1. Upload the SET PERSISTENT syntax patch for coming CF by fixing
> > existing
> > > review comments
> > > 2. Implement new syntax ALTER SYSTEM as proposed in below mail
> > >
> > > Could you suggest me what could be best way to proceed for this
> > patch?
> >
> > I'm still in favor of some syntax involving ALTER, because it's still
> > true that this behaves more like the existing GUC-setting commands
> > that use ALTER (which change configuration for future sessions)
> rather
> > the ones that use SET (which change the current settings for some
> > period of time).
> 
> 
> I will change the patch as per below syntax if there are no objections:
> 
>  ALTER SYSTEM SET configuration_parameter {TO | =} {value, | 'value'};

Modified patch contains:

1. Syntax implemented is 
ALTER SYSTEM SET configuration_parameter {TO | =} {value, | 'value' |
DEFAULT};

If user specifies DEFAULT, it will remove entry from auto conf file.

2. File name to store settings set by ALTER SYSTEM command is still
persistent.auto.conf

3. Added a new page for Alter System command in docs. In compatability
section, I had mentioned that
   this statement is an PostgresQL extension. I had tried to search in
SQL-92 spec but couldn't find such command.

4. During test of this patch, I had observed one problem for PGC_BACKEND
parameters like log_connections.
   If I change these parameters directly in postgresql.conf and then do
pg_reload_conf() and reconnect, it will 
   still show the old value. This is observed only on WINDOWS. I will
investigate this problem and update you.
   Due to this problem, I had removed few test cases.

Kindly let me know your suggestions. 

With Regards,
Amit Kapila.

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Josh Berkus <josh@agliodbs.com> — 2013-06-13T21:47:03Z

On 06/13/2013 05:35 AM, Amit Kapila wrote:
> On Friday, June 07, 2013 9:45 AM Amit Kapila wrote:
>> On Thursday, June 06, 2013 10:22 PM Robert Haas wrote:
>>> On Wed, Jun 5, 2013 at 7:24 AM, Amit Kapila <amit.kapila@huawei.com>
>>> wrote:
>>>> On Monday, May 27, 2013 4:17 PM Amit Kapila wrote:
>>>>> On Wednesday, April 03, 2013 11:55 AM Amit Kapila wote:
>>>>>> On Tuesday, April 02, 2013 9:49 PM Peter Eisentraut wrote:
>>>>>
>>>>
>>>> There are 2 options to proceed for this patch for 9.4
>>>>
>>>> 1. Upload the SET PERSISTENT syntax patch for coming CF by fixing
>>> existing
>>>> review comments
>>>> 2. Implement new syntax ALTER SYSTEM as proposed in below mail
>>>>
>>>> Could you suggest me what could be best way to proceed for this
>>> patch?
>>>
>>> I'm still in favor of some syntax involving ALTER, because it's still
>>> true that this behaves more like the existing GUC-setting commands
>>> that use ALTER (which change configuration for future sessions)
>> rather
>>> the ones that use SET (which change the current settings for some
>>> period of time).
>>
>>
>> I will change the patch as per below syntax if there are no objections:
>>
>>  ALTER SYSTEM SET configuration_parameter {TO | =} {value, | 'value'};
> 
> Modified patch contains:
> 
> 1. Syntax implemented is 
> ALTER SYSTEM SET configuration_parameter {TO | =} {value, | 'value' |
> DEFAULT};
> 
> If user specifies DEFAULT, it will remove entry from auto conf file.
> 
> 2. File name to store settings set by ALTER SYSTEM command is still
> persistent.auto.conf

Why?  Shouldn't it just be auto.conf?  Or system.auto.conf?

I prefer auto.conf, personally.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila@huawei.com> — 2013-06-14T03:12:07Z

On Friday, June 14, 2013 3:17 AM Josh Berkus wrote:
> On 06/13/2013 05:35 AM, Amit Kapila wrote:
> > On Friday, June 07, 2013 9:45 AM Amit Kapila wrote:
> >> On Thursday, June 06, 2013 10:22 PM Robert Haas wrote:
> >>> On Wed, Jun 5, 2013 at 7:24 AM, Amit Kapila
> <amit.kapila@huawei.com>
> >>> wrote:
> >>>> On Monday, May 27, 2013 4:17 PM Amit Kapila wrote:
> >>>>> On Wednesday, April 03, 2013 11:55 AM Amit Kapila wote:
> >>>>>> On Tuesday, April 02, 2013 9:49 PM Peter Eisentraut wrote:
> >>>>>
> >>>>
> >>>> There are 2 options to proceed for this patch for 9.4
> >>>>
> >>>> 1. Upload the SET PERSISTENT syntax patch for coming CF by fixing
> >>> existing
> >>>> review comments
> >>>> 2. Implement new syntax ALTER SYSTEM as proposed in below mail
> >>>>
> >>>> Could you suggest me what could be best way to proceed for this
> >>> patch?
> >>>
> >>> I'm still in favor of some syntax involving ALTER, because it's
> still
> >>> true that this behaves more like the existing GUC-setting commands
> >>> that use ALTER (which change configuration for future sessions)
> >> rather
> >>> the ones that use SET (which change the current settings for some
> >>> period of time).
> >>
> >>
> >> I will change the patch as per below syntax if there are no
> objections:
> >>
> >>  ALTER SYSTEM SET configuration_parameter {TO | =} {value, |
> 'value'};
> >
> > Modified patch contains:
> >
> > 1. Syntax implemented is
> > ALTER SYSTEM SET configuration_parameter {TO | =} {value, | 'value' |
> > DEFAULT};
> >
> > If user specifies DEFAULT, it will remove entry from auto conf file.
> >
> > 2. File name to store settings set by ALTER SYSTEM command is still
> > persistent.auto.conf
> 
> Why?  Shouldn't it just be auto.conf?  Or system.auto.conf?

  I had kept same name as it was decided for 9.3, but now as command has changed so it makes more
  sense to change name as well. I think it can be one of what you suggested or postgresql.auto.conf.


> I prefer auto.conf, personally.
  Thanks. if no body has any other suggestion I will change it.
  
  I think one of system.auto.conf or postgresql.auto.conf is more appropriate because it either resembles command or existing configuration settings 
  file.

With Regards,
Amit Kapila.

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Boszormenyi Zoltan <zb@cybertec.at> — 2013-06-18T09:55:54Z

Hi,

review below.

2013-06-13 14:35 keltezéssel, Amit Kapila írta:
> On Friday, June 07, 2013 9:45 AM Amit Kapila wrote:
>> On Thursday, June 06, 2013 10:22 PM Robert Haas wrote:
>>> On Wed, Jun 5, 2013 at 7:24 AM, Amit Kapila <amit.kapila@huawei.com>
>>> wrote:
>>>> On Monday, May 27, 2013 4:17 PM Amit Kapila wrote:
>>>>> On Wednesday, April 03, 2013 11:55 AM Amit Kapila wote:
>>>>>> On Tuesday, April 02, 2013 9:49 PM Peter Eisentraut wrote:
>>>> There are 2 options to proceed for this patch for 9.4
>>>>
>>>> 1. Upload the SET PERSISTENT syntax patch for coming CF by fixing
>>> existing
>>>> review comments
>>>> 2. Implement new syntax ALTER SYSTEM as proposed in below mail
>>>>
>>>> Could you suggest me what could be best way to proceed for this
>>> patch?
>>>
>>> I'm still in favor of some syntax involving ALTER, because it's still
>>> true that this behaves more like the existing GUC-setting commands
>>> that use ALTER (which change configuration for future sessions)
>> rather
>>> the ones that use SET (which change the current settings for some
>>> period of time).
>>
>> I will change the patch as per below syntax if there are no objections:
>>
>>   ALTER SYSTEM SET configuration_parameter {TO | =} {value, | 'value'};
> Modified patch contains:
>
> 1. Syntax implemented is
> ALTER SYSTEM SET configuration_parameter {TO | =} {value, | 'value' |
> DEFAULT};
>
> If user specifies DEFAULT, it will remove entry from auto conf file.
>
> 2. File name to store settings set by ALTER SYSTEM command is still
> persistent.auto.conf
>
> 3. Added a new page for Alter System command in docs. In compatability
> section, I had mentioned that
>     this statement is an PostgresQL extension. I had tried to search in
> SQL-92 spec but couldn't find such command.
>
> 4. During test of this patch, I had observed one problem for PGC_BACKEND
> parameters like log_connections.
>     If I change these parameters directly in postgresql.conf and then do
> pg_reload_conf() and reconnect, it will
>     still show the old value. This is observed only on WINDOWS. I will
> investigate this problem and update you.
>     Due to this problem, I had removed few test cases.

I can't reproduce this error under Linux (Fedora 19/x86_64).

The bug might be somewhere else and not caused by your patch
if manually adding/removing "log_connections = on" from postgresql.conf
exhibits the same behaviour under Windows. (If I read it correctly,
you tested it exactly this way.) Does the current GIT version behave
the same way?

>
> Kindly let me know your suggestions.
>
> With Regards,
> Amit Kapila.

* Is the patch in a patch format which has context?

Yes.

* Does it apply cleanly to the current git master?

Yes.

* Does it include reasonable tests, necessary doc patches, etc?

Yes.

Read what the patch is supposed to do, and consider:

* Does the patch actually implement that?

Yes.

* Do we want that?

Yes.

* Do we already have it?

No.

* Does it follow SQL spec, or the community-agreed behavior?

No such spec, follows the agreed behaviour.

* Does it include pg_dump support (if applicable)?

N/A

* Are there dangers?

No.

* Have all the bases been covered?

According to the above note under Windows, not yet.

The name "persistent.auto.conf" is not yet set in stone.
postgresql.auto.conf might be better.

Apply the patch, compile it and test:

* Does the feature work as advertised?

Yes, with the noted exception above for log_connections.

* Are there corner cases the author has failed to consider?

I can't see any. It is through

* Are there any assertion failures or crashes?

No.

* Does the patch slow down simple tests?

No.

* If it claims to improve performance, does it?

N/A

* Does it slow down other things?

No.

* Does it follow the project coding guidelines 
<http://developer.postgresql.org/pgdocs/postgres/source.html>?

Mostly, yes. Some nits, though. At some places, you do:

- ParseConfigFile(ConfigFileName, NULL, true, 0, elevel, &head, &tail);
+ ParseConfigFile(ConfigFileName, NULL, true, 0, elevel, &head, &tail,NULL);

without a space between the last comma and the NULL keyword.

Also, in the comment above ParseConfigFile() there are unnecessary
white space changes and spaces at the end of the line.

* Are there portability issues?

No.

* Will it work on Windows/BSD etc?

It should. It doesn't use any platform-dependent code.

* Are the comments sufficient and accurate?

Yes.

* Does it do what it says, correctly?

Yes.

* Does it produce compiler warnings?

No.

* Can you make it crash?

No.

Best regards,
Zoltán Böszörményi

-- 
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de
      http://www.postgresql.at/

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Boszormenyi Zoltan <zb@cybertec.at> — 2013-06-18T09:56:48Z

2013-06-14 05:12 keltezéssel, Amit Kapila írta:
> On Friday, June 14, 2013 3:17 AM Josh Berkus wrote:
>> On 06/13/2013 05:35 AM, Amit Kapila wrote:
>>> On Friday, June 07, 2013 9:45 AM Amit Kapila wrote:
>>>> On Thursday, June 06, 2013 10:22 PM Robert Haas wrote:
>>>>> On Wed, Jun 5, 2013 at 7:24 AM, Amit Kapila
>> <amit.kapila@huawei.com>
>>>>> wrote:
>>>>>> On Monday, May 27, 2013 4:17 PM Amit Kapila wrote:
>>>>>>> On Wednesday, April 03, 2013 11:55 AM Amit Kapila wote:
>>>>>>>> On Tuesday, April 02, 2013 9:49 PM Peter Eisentraut wrote:
>>>>>> There are 2 options to proceed for this patch for 9.4
>>>>>>
>>>>>> 1. Upload the SET PERSISTENT syntax patch for coming CF by fixing
>>>>> existing
>>>>>> review comments
>>>>>> 2. Implement new syntax ALTER SYSTEM as proposed in below mail
>>>>>>
>>>>>> Could you suggest me what could be best way to proceed for this
>>>>> patch?
>>>>>
>>>>> I'm still in favor of some syntax involving ALTER, because it's
>> still
>>>>> true that this behaves more like the existing GUC-setting commands
>>>>> that use ALTER (which change configuration for future sessions)
>>>> rather
>>>>> the ones that use SET (which change the current settings for some
>>>>> period of time).
>>>>
>>>> I will change the patch as per below syntax if there are no
>> objections:
>>>>   ALTER SYSTEM SET configuration_parameter {TO | =} {value, |
>> 'value'};
>>> Modified patch contains:
>>>
>>> 1. Syntax implemented is
>>> ALTER SYSTEM SET configuration_parameter {TO | =} {value, | 'value' |
>>> DEFAULT};
>>>
>>> If user specifies DEFAULT, it will remove entry from auto conf file.
>>>
>>> 2. File name to store settings set by ALTER SYSTEM command is still
>>> persistent.auto.conf
>> Why?  Shouldn't it just be auto.conf?  Or system.auto.conf?
>    I had kept same name as it was decided for 9.3, but now as command has changed so it makes more
>    sense to change name as well. I think it can be one of what you suggested or postgresql.auto.conf.
>
>
>> I prefer auto.conf, personally.
>    Thanks. if no body has any other suggestion I will change it.
>    
>    I think one of system.auto.conf or postgresql.auto.conf is more appropriate because it either resembles command or existing configuration settings
>    file.

I like postgresql.auto.conf better.

>
> With Regards,
> Amit Kapila.
>
>
>


-- 
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de
      http://www.postgresql.at/

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila@huawei.com> — 2013-06-18T12:11:01Z

On Tuesday, June 18, 2013 3:26 PM Boszormenyi Zoltan wrote:
> Hi,

> review below.
  Thanks for the review.


>>>>>There are 2 options to proceed for this patch for 9.4

>>>>>1. Upload the SET PERSISTENT syntax patch for coming CF by fixing
>>>>>existing
>>>>>review comments
>>>>>2. Implement new syntax ALTER SYSTEM as proposed in below mail

>>>>>Could you suggest me what could be best way to proceed for this
>>>>>patch?

>>>>I'm still in favor of some syntax involving ALTER, because it's still
>>>>true that this behaves more like the existing GUC-setting commands
>>>>that use ALTER (which change configuration for future sessions)
>>>>rather
>>>>the ones that use SET (which change the current settings for some
>>>>period of time).


>>>I will change the patch as per below syntax if there are no objections:

>>>ALTER SYSTEM SET configuration_parameter {TO | =} {value, | 'value'};

>>Modified patch contains:

>>1. Syntax implemented is 
>>ALTER SYSTEM SET configuration_parameter {TO | =} {value, | 'value' |
>>DEFAULT};

>>If user specifies DEFAULT, it will remove entry from auto conf file.

>>2. File name to store settings set by ALTER SYSTEM command is still
>>persistent.auto.conf

>>3. Added a new page for Alter System command in docs. In compatability
>>section, I had mentioned that
>>   this statement is an PostgresQL extension. I had tried to search in
>>SQL-92 spec but couldn't find such command.

>>4. During test of this patch, I had observed one problem for PGC_BACKEND
>>parameters like log_connections.
 >>  If I change these parameters directly in postgresql.conf and then do
>>pg_reload_conf() and reconnect, it will 
>>   still show the old value. This is observed only on WINDOWS. I will
>>investigate this problem and update you.
>>   Due to this problem, I had removed few test cases.

> I can't reproduce this error under Linux (Fedora 19/x86_64).

> The bug might be somewhere else and not caused by your patch
> if manually adding/removing "log_connections = on" from postgresql.conf
> exhibits the same behaviour under Windows. (If I read it correctly,
> you tested it exactly this way.) Does the current GIT version behave
> the same way?

Yes, the current Git has problem which I had reported in below mail:

http://www.postgresql.org/message-id/009801ce68f7$3a746340$af5d29c0$@kapila@
huawei.com

This problem is not related to this patch, it occurs only on WINDOWS or
under EXEC_BACKEND flag.
I think we can discuss this problem in above mail thread.



> * Have all the bases been covered?

> According to the above note under Windows, not yet.

> The name "persistent.auto.conf" is not yet set in stone.
> postgresql.auto.conf might be better.

I think, the decision of name, we can leave to committer with below
possibilities, 
as it is very difficult to build consensus on any particular name.

Auto.conf
System.auto.conf
Postgresql.auto.conf
Persistent.auto.conf

> Apply the patch, compile it and test:


> * Does it follow the project coding guidelines?

> Mostly, yes. Some nits, though. At some places, you do:

> - ParseConfigFile(ConfigFileName, NULL, true, 0, elevel, &head, &tail);
> + ParseConfigFile(ConfigFileName, NULL, true, 0, elevel, &head,
&tail,NULL);

I think you mean ParseConfigFp()?

without a space between the last comma and the NULL keyword.

> Also, in the comment above ParseConfigFile() there are unnecessary
> white space changes and spaces at the end of the line.

Do you mean to say whitespaces in below text?

!  * and absolute-ifying the path name if necessary.

!  *                                                                        
!  * While parsing, it records if it has parsed persistent.auto.conf file.  
!  * This information can be used by the callers to ensure if the parameters
!  * set by ALTER SYSTEM SET command will be effective.
   */

With Regards,
Amit Kapila.

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Boszormenyi Zoltan <zb@cybertec.at> — 2013-06-18T14:09:05Z

2013-06-18 14:11 keltezéssel, Amit Kapila írta:
> On Tuesday, June 18, 2013 3:26 PM Boszormenyi Zoltan wrote:
>> Hi,
>> review below.
>    Thanks for the review.
>
>
>>>>>> There are 2 options to proceed for this patch for 9.4
>>>>>> 1. Upload the SET PERSISTENT syntax patch for coming CF by fixing
>>>>>> existing
>>>>>> review comments
>>>>>> 2. Implement new syntax ALTER SYSTEM as proposed in below mail
>>>>>> Could you suggest me what could be best way to proceed for this
>>>>>> patch?
>>>>> I'm still in favor of some syntax involving ALTER, because it's still
>>>>> true that this behaves more like the existing GUC-setting commands
>>>>> that use ALTER (which change configuration for future sessions)
>>>>> rather
>>>>> the ones that use SET (which change the current settings for some
>>>>> period of time).
>
>>>> I will change the patch as per below syntax if there are no objections:
>>>> ALTER SYSTEM SET configuration_parameter {TO | =} {value, | 'value'};
>>> Modified patch contains:
>>> 1. Syntax implemented is
>>> ALTER SYSTEM SET configuration_parameter {TO | =} {value, | 'value' |
>>> DEFAULT};
>>> If user specifies DEFAULT, it will remove entry from auto conf file.
>>> 2. File name to store settings set by ALTER SYSTEM command is still
>>> persistent.auto.conf
>>> 3. Added a new page for Alter System command in docs. In compatability
>>> section, I had mentioned that
>>>    this statement is an PostgresQL extension. I had tried to search in
>>> SQL-92 spec but couldn't find such command.
>>> 4. During test of this patch, I had observed one problem for PGC_BACKEND
>>> parameters like log_connections.
>   >>  If I change these parameters directly in postgresql.conf and then do
>>> pg_reload_conf() and reconnect, it will
>>>    still show the old value. This is observed only on WINDOWS. I will
>>> investigate this problem and update you.
>>>    Due to this problem, I had removed few test cases.
>> I can't reproduce this error under Linux (Fedora 19/x86_64).
>> The bug might be somewhere else and not caused by your patch
>> if manually adding/removing "log_connections = on" from postgresql.conf
>> exhibits the same behaviour under Windows. (If I read it correctly,
>> you tested it exactly this way.) Does the current GIT version behave
>> the same way?
> Yes, the current Git has problem which I had reported in below mail:
>
> http://www.postgresql.org/message-id/009801ce68f7$3a746340$af5d29c0$@kapila@
> huawei.com
>
> This problem is not related to this patch, it occurs only on WINDOWS or
> under EXEC_BACKEND flag.
> I think we can discuss this problem in above mail thread.
>
>
>
>> * Have all the bases been covered?
>> According to the above note under Windows, not yet.
>> The name "persistent.auto.conf" is not yet set in stone.
>> postgresql.auto.conf might be better.
> I think, the decision of name, we can leave to committer with below
> possibilities,
> as it is very difficult to build consensus on any particular name.
>
> Auto.conf
> System.auto.conf
> Postgresql.auto.conf
> Persistent.auto.conf
>
>> Apply the patch, compile it and test:
>
>> * Does it follow the project coding guidelines?
>> Mostly, yes. Some nits, though. At some places, you do:
>> - ParseConfigFile(ConfigFileName, NULL, true, 0, elevel, &head, &tail);
>> + ParseConfigFile(ConfigFileName, NULL, true, 0, elevel, &head,
> &tail,NULL);
>
> I think you mean ParseConfigFp()?

Sorry, yes.

>
> without a space between the last comma and the NULL keyword.
>
>> Also, in the comment above ParseConfigFile() there are unnecessary
>> white space changes and spaces at the end of the line.
> Do you mean to say whitespaces in below text?
>
> !  * and absolute-ifying the path name if necessary.
>
> !  *
> !  * While parsing, it records if it has parsed persistent.auto.conf file.
> !  * This information can be used by the callers to ensure if the parameters
> !  * set by ALTER SYSTEM SET command will be effective.
>     */

Yes.

Anyway, both would be fixed by a pgindent run. (I think.)

>
> With Regards,
> Amit Kapila.
>


-- 
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de
      http://www.postgresql.at/

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Josh Berkus <josh@agliodbs.com> — 2013-06-18T17:54:54Z

Amit,

> I think, the decision of name, we can leave to committer with below
> possibilities, 
> as it is very difficult to build consensus on any particular name.
> 
> Auto.conf
> System.auto.conf
> Postgresql.auto.conf
> Persistent.auto.conf

Reasons for "auto.conf" as a choice above all of the previous:

1) short
2) significantly different from "postgresql.conf"
3) near the beginning of the alphabet, so it should sort near the top if
there are other files in conf/ directory

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila@huawei.com> — 2013-06-19T05:58:26Z

On Tuesday, June 18, 2013 11:25 PM Josh Berkus wrote:
> Amit,
> 
> > I think, the decision of name, we can leave to committer with below
> > possibilities,
> > as it is very difficult to build consensus on any particular name.
> >
> > Auto.conf
> > System.auto.conf
> > Postgresql.auto.conf
> > Persistent.auto.conf
> 
> Reasons for "auto.conf" as a choice above all of the previous:
> 
> 1) short
> 2) significantly different from "postgresql.conf"
     What is the advantage to keep name significantly different from "postgresql.conf", 
     rather will it be not helpful for use if it is similar to "postgresql.conf"?
> 3) near the beginning of the alphabet, so it should sort near the top
> if
> there are other files in conf/ directory

How would user correlate auto.conf contains configuration parameters set by Alter System command?

I had kept persistent.auto.conf based on suggestion in this thread that user can easily correlate it with
SET PERSISTENT command. 
So based on that theory I think it would be better to name it as System.auto.conf or Postgresql.auto.conf

With Regards,
Amit Kapila.

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila@huawei.com> — 2013-06-19T07:20:19Z

On Tuesday, June 18, 2013 7:39 PM Boszormenyi Zoltan wrote:
> 2013-06-18 14:11 keltezéssel, Amit Kapila írta:
> > On Tuesday, June 18, 2013 3:26 PM Boszormenyi Zoltan wrote:
> >> Hi,
> >> review below.
> >    Thanks for the review.
> >
> >
> >>>>>> There are 2 options to proceed for this patch for 9.4
> >>>>>> 1. Upload the SET PERSISTENT syntax patch for coming CF by
> fixing
> >>>>>> existing
> >>>>>> review comments
> >>>>>> 2. Implement new syntax ALTER SYSTEM as proposed in below mail
> >>>>>> Could you suggest me what could be best way to proceed for this
> >>>>>> patch?
> >>>>> I'm still in favor of some syntax involving ALTER, because it's
> still
> >>>>> true that this behaves more like the existing GUC-setting
> commands
> >>>>> that use ALTER (which change configuration for future sessions)
> >>>>> rather
> >>>>> the ones that use SET (which change the current settings for some
> >>>>> period of time).
> >
> >>>> I will change the patch as per below syntax if there are no
> objections:
> >>>> ALTER SYSTEM SET configuration_parameter {TO | =} {value, |
> 'value'};
> >>> Modified patch contains:
> >>> 1. Syntax implemented is
> >>> ALTER SYSTEM SET configuration_parameter {TO | =} {value, | 'value'
> |
> >>> DEFAULT};
> >>> If user specifies DEFAULT, it will remove entry from auto conf
> file.
> >>> 2. File name to store settings set by ALTER SYSTEM command is still
> >>> persistent.auto.conf
> >>> 3. Added a new page for Alter System command in docs. In
> compatability
> >>> section, I had mentioned that
> >>>    this statement is an PostgresQL extension. I had tried to search
> in
> >>> SQL-92 spec but couldn't find such command.
> >>> 4. During test of this patch, I had observed one problem for
> PGC_BACKEND
> >>> parameters like log_connections.
> >   >>  If I change these parameters directly in postgresql.conf and
> then do
> >>> pg_reload_conf() and reconnect, it will
> >>>    still show the old value. This is observed only on WINDOWS. I
> will
> >>> investigate this problem and update you.
> >>>    Due to this problem, I had removed few test cases.
> >> I can't reproduce this error under Linux (Fedora 19/x86_64).
> >> The bug might be somewhere else and not caused by your patch
> >> if manually adding/removing "log_connections = on" from
> postgresql.conf
> >> exhibits the same behaviour under Windows. (If I read it correctly,
> >> you tested it exactly this way.) Does the current GIT version behave
> >> the same way?
> > Yes, the current Git has problem which I had reported in below mail:
> >
> > http://www.postgresql.org/message-
> id/009801ce68f7$3a746340$af5d29c0$@kapila@
> > huawei.com
> >
> > This problem is not related to this patch, it occurs only on WINDOWS
> or
> > under EXEC_BACKEND flag.
> > I think we can discuss this problem in above mail thread.
> >
> >
> >
> >> * Have all the bases been covered?
> >> According to the above note under Windows, not yet.
> >> The name "persistent.auto.conf" is not yet set in stone.
> >> postgresql.auto.conf might be better.
> > I think, the decision of name, we can leave to committer with below
> > possibilities,
> > as it is very difficult to build consensus on any particular name.
> >
> > Auto.conf
> > System.auto.conf
> > Postgresql.auto.conf
> > Persistent.auto.conf
> >
> >> Apply the patch, compile it and test:
> >
> >> * Does it follow the project coding guidelines?
> >> Mostly, yes. Some nits, though. At some places, you do:
> >> - ParseConfigFile(ConfigFileName, NULL, true, 0, elevel, &head,
> &tail);
> >> + ParseConfigFile(ConfigFileName, NULL, true, 0, elevel, &head,
> > &tail,NULL);
> >
> > I think you mean ParseConfigFp()?
> 
> Sorry, yes.
> 
> >
> > without a space between the last comma and the NULL keyword.
> >
> >> Also, in the comment above ParseConfigFile() there are unnecessary
> >> white space changes and spaces at the end of the line.
> > Do you mean to say whitespaces in below text?
> >
> > !  * and absolute-ifying the path name if necessary.
> >
> > !  *
> > !  * While parsing, it records if it has parsed persistent.auto.conf
> file.
> > !  * This information can be used by the callers to ensure if the
> parameters
> > !  * set by ALTER SYSTEM SET command will be effective.
> >     */
> 
> Yes.
> 
> Anyway, both would be fixed by a pgindent run. (I think.)

I think pgindent doesn't work on .l file, in anycase I had corrected it
manually.
Find the updated patch attached with this mail.

Kindly let me know if there is anything more needs to be done for this
patch.

With Regards,
Amit Kapila.

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Peter Eisentraut <peter_e@gmx.net> — 2013-06-19T17:48:27Z

On 6/13/13 5:47 PM, Josh Berkus wrote:
>> 2. File name to store settings set by ALTER SYSTEM command is still
>> > persistent.auto.conf
> Why?  Shouldn't it just be auto.conf?  Or system.auto.conf?
> 
> I prefer auto.conf, personally.

Well, not much about it is automatic, really.  It's just set elsewhere.

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Josh Berkus <josh@agliodbs.com> — 2013-06-19T17:49:42Z

On 06/19/2013 10:48 AM, Peter Eisentraut wrote:
> On 6/13/13 5:47 PM, Josh Berkus wrote:
>>> 2. File name to store settings set by ALTER SYSTEM command is still
>>>> persistent.auto.conf
>> Why?  Shouldn't it just be auto.conf?  Or system.auto.conf?
>>
>> I prefer auto.conf, personally.
> 
> Well, not much about it is automatic, really.  It's just set elsewhere.
> 

True, but can you think of a better word to mean "don't edit this by hand"?

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Peter Eisentraut <peter_e@gmx.net> — 2013-06-19T17:55:17Z

On 6/19/13 1:49 PM, Josh Berkus wrote:
> On 06/19/2013 10:48 AM, Peter Eisentraut wrote:
>> On 6/13/13 5:47 PM, Josh Berkus wrote:
>>>> 2. File name to store settings set by ALTER SYSTEM command is still
>>>>> persistent.auto.conf
>>> Why?  Shouldn't it just be auto.conf?  Or system.auto.conf?
>>>
>>> I prefer auto.conf, personally.
>>
>> Well, not much about it is automatic, really.  It's just set elsewhere.
>>
> 
> True, but can you think of a better word to mean "don't edit this by hand"?

generated_by_server.conf

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Magnus Hagander <magnus@hagander.net> — 2013-06-19T17:57:35Z

On Jun 19, 2013 7:55 PM, "Peter Eisentraut" <peter_e@gmx.net> wrote:
>
> On 6/19/13 1:49 PM, Josh Berkus wrote:
> > On 06/19/2013 10:48 AM, Peter Eisentraut wrote:
> >> On 6/13/13 5:47 PM, Josh Berkus wrote:
> >>>> 2. File name to store settings set by ALTER SYSTEM command is still
> >>>>> persistent.auto.conf
> >>> Why?  Shouldn't it just be auto.conf?  Or system.auto.conf?
> >>>
> >>> I prefer auto.conf, personally.
> >>
> >> Well, not much about it is automatic, really.  It's just set elsewhere.
> >>
> >
> > True, but can you think of a better word to mean "don't edit this by
hand"?
>
> generated_by_server.conf

System.conf?

/Magnus

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Peter Eisentraut <peter_e@gmx.net> — 2013-06-19T17:59:35Z

On 6/7/13 12:14 AM, Amit Kapila wrote:
> I will change the patch as per below syntax if there are no objections:
> 
>  ALTER SYSTEM SET configuration_parameter {TO | =} {value, | 'value'};

I do like using ALTER SYSTEM in general, but now that I think about it,
the 9.3 feature to create global settings in pg_db_role_setting would
also have been a candidate for exactly that same syntax if it had been
available.  In fact, if we do add ALTER SYSTEM, it might make sense to
recast that feature into that syntax.

It might be clearer to do something like ALTER SYSTEM SET EXTERNAL FILE
or something like that.  It's only a small syntax change, so don't worry
about it too much, but let's keep thinking about it.

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Alvaro Herrera <alvherre@2ndquadrant.com> — 2013-06-19T19:01:30Z

Magnus Hagander wrote:
> On Jun 19, 2013 7:55 PM, "Peter Eisentraut" <peter_e@gmx.net> wrote:

> > generated_by_server.conf
> 
> System.conf?

alter_system.conf ?

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Tom Lane <tgl@sss.pgh.pa.us> — 2013-06-19T22:23:02Z

Josh Berkus <josh@agliodbs.com> writes:
> True, but can you think of a better word to mean "don't edit this by hand"?

The file name is not nearly as important for that as putting in a
header comment "# Don't edit this file by hand."

			regards, tom lane

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Amit Kapila <amit.kapila@huawei.com> — 2013-06-20T03:47:31Z

On Wednesday, June 19, 2013 11:30 PM Peter Eisentraut wrote:
> On 6/7/13 12:14 AM, Amit Kapila wrote:
> > I will change the patch as per below syntax if there are no
> objections:
> >
> >  ALTER SYSTEM SET configuration_parameter {TO | =} {value, |
> 'value'};
> 
> I do like using ALTER SYSTEM in general, but now that I think about it,
> the 9.3 feature to create global settings in pg_db_role_setting would
> also have been a candidate for exactly that same syntax if it had been
> available.  In fact, if we do add ALTER SYSTEM, it might make sense to
> recast that feature into that syntax.
> 
> It might be clearer to do something like ALTER SYSTEM SET EXTERNAL FILE
> or something like that.  It's only a small syntax change, so don't
> worry
> about it too much, but let's keep thinking about it.

Is your intention to extend the syntax by EXTERNAL FILE is for specifying
SCOPE for configuration parameter?

ALTER SYSTEM SET SCOPE FILE
ALTER SYSTEM SET EXTENSION FILE


With Regards,
Amit Kapila.

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila@huawei.com> — 2013-06-20T03:51:50Z

On Thursday, June 20, 2013 3:53 AM Tom Lane wrote:
> Josh Berkus <josh@agliodbs.com> writes:
> > True, but can you think of a better word to mean "don't edit this by
> hand"?
> 
> The file name is not nearly as important for that as putting in a
> header comment "# Don't edit this file by hand."

Currently header comment is:
# Do not edit this file manually! 
# It will be overwritten by ALTER SYSTEM command.

Kindly let me know your suggestion about the above comment in header?

With Regards,
Amit Kapila.

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila@huawei.com> — 2013-06-20T04:18:28Z

On Thursday, June 20, 2013 12:32 AM Alvaro Herrera wrote:
> Magnus Hagander wrote:
> > On Jun 19, 2013 7:55 PM, "Peter Eisentraut" <peter_e@gmx.net> wrote:
> 
> > > generated_by_server.conf
> >
> > System.conf?
> 
> alter_system.conf ?

All the names proposed in this thread are as follows:

Auto.conf            - 1 Vote (Josh)
System.auto.conf     - 1 Vote (Josh)
Postgresql.auto.conf - 2 Votes (Zoltan, Amit)
Persistent.auto.conf - 0 Vote
generated_by_server.conf - 1 Vote (Peter E)
System.conf              - 1 Vote (Magnus)
alter_system.conf        - 1 Vote (Alvaro)

I had consolidated the list, so that it would be helpful for committer to
decide among proposed names for this file.


With Regards,
Amit Kapila.

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Alvaro Herrera <alvherre@2ndquadrant.com> — 2013-06-20T05:42:30Z

Amit Kapila wrote:

> Currently header comment is:
> # Do not edit this file manually! 
> # It will be overwritten by ALTER SYSTEM command.
> 
> Kindly let me know your suggestion about the above comment in header?

This seems perfect to me.

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Robert Haas <robertmhaas@gmail.com> — 2013-06-21T15:11:01Z

On Wed, Jun 19, 2013 at 1:59 PM, Peter Eisentraut <peter_e@gmx.net> wrote:
> On 6/7/13 12:14 AM, Amit Kapila wrote:
>> I will change the patch as per below syntax if there are no objections:
>>
>>  ALTER SYSTEM SET configuration_parameter {TO | =} {value, | 'value'};
>
> I do like using ALTER SYSTEM in general, but now that I think about it,
> the 9.3 feature to create global settings in pg_db_role_setting would
> also have been a candidate for exactly that same syntax if it had been
> available.  In fact, if we do add ALTER SYSTEM, it might make sense to
> recast that feature into that syntax.
>
> It might be clearer to do something like ALTER SYSTEM SET EXTERNAL FILE
> or something like that.  It's only a small syntax change, so don't worry
> about it too much, but let's keep thinking about it.

I think that anything we want to add should either go before SET or
after the value, such as:

ALTER SYSTEM SET configuration_parameter = 'value' IN FILE 'myconf.conf';
ALTER SYSTEM CONFIGURATION FILE SET configuration_parameter = 'file';
ALTER SYSTEM CONFIGURATION FILE 'myconf.conf' SET
configuration_parameter = 'file';

I agree we shouldn't back ourselves into a syntactic corner.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Fujii Masao <masao.fujii@gmail.com> — 2013-06-21T16:56:45Z

On Sat, Jun 22, 2013 at 12:11 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> On Wed, Jun 19, 2013 at 1:59 PM, Peter Eisentraut <peter_e@gmx.net> wrote:
>> On 6/7/13 12:14 AM, Amit Kapila wrote:
>>> I will change the patch as per below syntax if there are no objections:
>>>
>>>  ALTER SYSTEM SET configuration_parameter {TO | =} {value, | 'value'};
>>
>> I do like using ALTER SYSTEM in general, but now that I think about it,
>> the 9.3 feature to create global settings in pg_db_role_setting would
>> also have been a candidate for exactly that same syntax if it had been
>> available.  In fact, if we do add ALTER SYSTEM, it might make sense to
>> recast that feature into that syntax.
>>
>> It might be clearer to do something like ALTER SYSTEM SET EXTERNAL FILE
>> or something like that.  It's only a small syntax change, so don't worry
>> about it too much, but let's keep thinking about it.
>
> I think that anything we want to add should either go before SET or
> after the value, such as:
>
> ALTER SYSTEM SET configuration_parameter = 'value' IN FILE 'myconf.conf';
> ALTER SYSTEM CONFIGURATION FILE SET configuration_parameter = 'file';
> ALTER SYSTEM CONFIGURATION FILE 'myconf.conf' SET
> configuration_parameter = 'file';
>
> I agree we shouldn't back ourselves into a syntactic corner.

Maybe this idea has been already discussed, but why don't we just
add the function like update_config_file(parameter, value)? We can
commit the core of the patch with that function API first, and then
we can discuss the syntax and add another API like ALTER SYSTEM
later.

We now have set_config() function to set the parameter,
so adding the function for this feature should not be a surprise.
If the name of the function is, for example, update_conf_file,
most users would think that it's intuitive even if SIGHUP is not
automatically sent immediately. We don't need to emit
the message like 'This setting change will not be loaded until SIGHUP'.

Regards,

-- 
Fujii Masao

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Robert Haas <robertmhaas@gmail.com> — 2013-06-21T18:13:04Z

On Fri, Jun 21, 2013 at 12:56 PM, Fujii Masao <masao.fujii@gmail.com> wrote:
> On Sat, Jun 22, 2013 at 12:11 AM, Robert Haas <robertmhaas@gmail.com> wrote:
>> On Wed, Jun 19, 2013 at 1:59 PM, Peter Eisentraut <peter_e@gmx.net> wrote:
>>> On 6/7/13 12:14 AM, Amit Kapila wrote:
>>>> I will change the patch as per below syntax if there are no objections:
>>>>
>>>>  ALTER SYSTEM SET configuration_parameter {TO | =} {value, | 'value'};
>>>
>>> I do like using ALTER SYSTEM in general, but now that I think about it,
>>> the 9.3 feature to create global settings in pg_db_role_setting would
>>> also have been a candidate for exactly that same syntax if it had been
>>> available.  In fact, if we do add ALTER SYSTEM, it might make sense to
>>> recast that feature into that syntax.
>>>
>>> It might be clearer to do something like ALTER SYSTEM SET EXTERNAL FILE
>>> or something like that.  It's only a small syntax change, so don't worry
>>> about it too much, but let's keep thinking about it.
>>
>> I think that anything we want to add should either go before SET or
>> after the value, such as:
>>
>> ALTER SYSTEM SET configuration_parameter = 'value' IN FILE 'myconf.conf';
>> ALTER SYSTEM CONFIGURATION FILE SET configuration_parameter = 'file';
>> ALTER SYSTEM CONFIGURATION FILE 'myconf.conf' SET
>> configuration_parameter = 'file';
>>
>> I agree we shouldn't back ourselves into a syntactic corner.
>
> Maybe this idea has been already discussed, but why don't we just
> add the function like update_config_file(parameter, value)? We can
> commit the core of the patch with that function API first, and then
> we can discuss the syntax and add another API like ALTER SYSTEM
> later.
>
> We now have set_config() function to set the parameter,
> so adding the function for this feature should not be a surprise.
> If the name of the function is, for example, update_conf_file,
> most users would think that it's intuitive even if SIGHUP is not
> automatically sent immediately. We don't need to emit
> the message like 'This setting change will not be loaded until SIGHUP'.

I could certainly support that plan.  I'm satisfied with the ALTER
SYSTEM syntax and feel that might find other plausible uses, but
functional notation would be OK with me, too.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Robert Haas <robertmhaas@gmail.com> — 2013-06-21T18:18:59Z

On Thu, Jun 20, 2013 at 12:18 AM, Amit Kapila <amit.kapila@huawei.com> wrote:
> Auto.conf            - 1 Vote (Josh)
> System.auto.conf     - 1 Vote (Josh)
> Postgresql.auto.conf - 2 Votes (Zoltan, Amit)
> Persistent.auto.conf - 0 Vote
> generated_by_server.conf - 1 Vote (Peter E)
> System.conf              - 1 Vote (Magnus)
> alter_system.conf        - 1 Vote (Alvaro)
>
> I had consolidated the list, so that it would be helpful for committer to
> decide among proposed names for this file.

I'll also vote for postgresql.auto.conf.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila@huawei.com> — 2013-06-22T04:17:17Z

On Friday, June 21, 2013 11:48 PM Robert Haas wrote:
On Thu, Jun 20, 2013 at 12:18 AM, Amit Kapila <amit.kapila@huawei.com> wrote:
>> Auto.conf            - 1 Vote (Josh)
>> System.auto.conf     - 1 Vote (Josh)
>> Postgresql.auto.conf - 2 Votes (Zoltan, Amit)
>> Persistent.auto.conf - 0 Vote
>> generated_by_server.conf - 1 Vote (Peter E)
>> System.conf              - 1 Vote (Magnus)
>> alter_system.conf        - 1 Vote (Alvaro)
>
>> I had consolidated the list, so that it would be helpful for committer to
>> decide among proposed names for this file.

> I'll also vote for postgresql.auto.conf.

  Thanks to all of you for suggesting meaningful names. I will change the name of file to postgresql.auto.conf.
  Kindly let me know if there is any objection to it.

With Regards,
Amit Kapila.

Re: Proposal for Allow postgresql.conf values to be changed via SQL [review]

Amit Kapila <amit.kapila@huawei.com> — 2013-06-22T04:39:22Z

On Friday, June 21, 2013 11:43 PM Robert Haas wrote:
On Fri, Jun 21, 2013 at 12:56 PM, Fujii Masao <masao.fujii@gmail.com> wrote:
> On Sat, Jun 22, 2013 at 12:11 AM, Robert Haas <robertmhaas@gmail.com> wrote:
>> On Wed, Jun 19, 2013 at 1:59 PM, Peter Eisentraut <peter_e@gmx.net> wrote:
>>> On 6/7/13 12:14 AM, Amit Kapila wrote:
>>>>> I will change the patch as per below syntax if there are no objections:
>>>>>
>>>>>  ALTER SYSTEM SET configuration_parameter {TO | =} {value, | 'value'};
>>>>
>>>> I do like using ALTER SYSTEM in general, but now that I think about it,
>>>> the 9.3 feature to create global settings in pg_db_role_setting would
>>>> also have been a candidate for exactly that same syntax if it had been
>>>> available.  In fact, if we do add ALTER SYSTEM, it might make sense to
>>>> recast that feature into that syntax.
>>>>
>>>> It might be clearer to do something like ALTER SYSTEM SET EXTERNAL FILE
>>>> or something like that.  It's only a small syntax change, so don't worry
>>>> about it too much, but let's keep thinking about it.
>>>
>>> I think that anything we want to add should either go before SET or
>>> after the value, such as:
>>>
>>> ALTER SYSTEM SET configuration_parameter = 'value' IN FILE 'myconf.conf';
>>> ALTER SYSTEM CONFIGURATION FILE SET configuration_parameter = 'file';
>>> ALTER SYSTEM CONFIGURATION FILE 'myconf.conf' SET
>>> configuration_parameter = 'file';
>>>
>>> I agree we shouldn't back ourselves into a syntactic corner.
>>
>> Maybe this idea has been already discussed, but why don't we just
>> add the function like update_config_file(parameter, value)? We can
>> commit the core of the patch with that function API first, and then
>> we can discuss the syntax and add another API like ALTER SYSTEM
>> later.
>>
>> We now have set_config() function to set the parameter,
>> so adding the function for this feature should not be a surprise.
>> If the name of the function is, for example, update_conf_file,
>> most users would think that it's intuitive even if SIGHUP is not
>> automatically sent immediately. We don't need to emit
>> the message like 'This setting change will not be loaded until SIGHUP'.

> I could certainly support that plan.  I'm satisfied with the ALTER
> SYSTEM syntax and feel that might find other plausible uses, but
> functional notation would be OK with me, too.


I can update the patch to have a function as suggested by Fujii-san if it can be decided
that for the first committed version it will be sufficient.
OTOH already we already have consensus on ALTER SYSTEM syntax apart from few extra keywords to make it more meaningful/extendible. 
I think we can consider the current syntax (ALTER SYSTEM SET ..) and make that behavior as default for writing to auto file.
In future we can extend it with other keywords depending on usage.

With Regards,
Amit Kapila.

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila@huawei.com> — 2013-06-25T06:54:22Z

On Saturday, June 22, 2013 9:47 AM Amit kapila wrote:
> On Friday, June 21, 2013 11:48 PM Robert Haas wrote:
> On Thu, Jun 20, 2013 at 12:18 AM, Amit Kapila <amit.kapila@huawei.com>
> wrote:
> >> Auto.conf            - 1 Vote (Josh)
> >> System.auto.conf     - 1 Vote (Josh)
> >> Postgresql.auto.conf - 2 Votes (Zoltan, Amit)
> >> Persistent.auto.conf - 0 Vote
> >> generated_by_server.conf - 1 Vote (Peter E)
> >> System.conf              - 1 Vote (Magnus)
> >> alter_system.conf        - 1 Vote (Alvaro)
> >
> >> I had consolidated the list, so that it would be helpful for
> committer to
> >> decide among proposed names for this file.
> 
> > I'll also vote for postgresql.auto.conf.
> 
>   Thanks to all of you for suggesting meaningful names. I will change
> the name of file to postgresql.auto.conf.
>   Kindly let me know if there is any objection to it.

I have changed the file name to postgresql.auto.conf and I have changed the
name of SetPersistentLock to AutoFileLock.

Zoltan,

Could you please once check the attached Patch if you have time, as now all
the things are resolved except for small doubt in syntax extensibility 
which can be changed based on final decision.  


With Regards,
Amit Kapila.

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Josh Berkus <josh@agliodbs.com> — 2013-06-25T23:32:30Z

>> I'll also vote for postgresql.auto.conf.
> 
>   Thanks to all of you for suggesting meaningful names. I will change the name of file to postgresql.auto.conf.
>   Kindly let me know if there is any objection to it.

There is no name you could come up with for which there would not be an
objection.  Please plow ahead.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila@huawei.com> — 2013-06-26T04:51:00Z

On Wednesday, June 26, 2013 5:03 AM Josh Berkus wrote:
> 
> >> I'll also vote for postgresql.auto.conf.
> >
> >   Thanks to all of you for suggesting meaningful names. I will change
> the name of file to postgresql.auto.conf.
> >   Kindly let me know if there is any objection to it.
> 
> There is no name you could come up with for which there would not be an
> objection.  Please plow ahead.

Thanks. 
I had updated the patch and sent it.

With Regards,
Amit Kapila.

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Alvaro Herrera <alvherre@2ndquadrant.com> — 2013-07-02T21:27:39Z

Amit Kapila escribió:

> I have changed the file name to postgresql.auto.conf and I have changed the
> name of SetPersistentLock to AutoFileLock.
> 
> Zoltan,
> 
> Could you please once check the attached Patch if you have time, as now all
> the things are resolved except for small doubt in syntax extensibility 
> which can be changed based on final decision.  

There are a bunch of whitespace problems, as you would notice with "git
diff --check" or "git show --color".  Could you please clean that up?
Also, some of the indentation in various places is not right; perhaps
you could fix that by running pgindent over the source files you
modified (this is easily visible by eyeballing the git diff output;
stuff is misaligned in pretty obvious ways.)

Random minor other comments:

+     use of <xref linkend="SQL-ALTER SYSTEM">

this SGML link cannot possibly work.  Please run "make check" in the
doc/src/sgml directory.

Examples in alter_system.sgml have a typo "SYTEM".  Same file has "or or".

Also in that file,
      The name of an configuration parameter that exist in <filename>postgresql.conf</filename>.
the parameter needn't exist in that file to be settable, right?

 <refnamediv>
  <refname>ALTER SYSTEM</refname>
  <refpurpose>change a server configuration parameter</refpurpose>
 </refnamediv>

Perhaps "permanently change .."?

Also, some parameters require a restart, not a reload; maybe we should
direct the user somewhere else to learn how to freshen up the
configuration after calling the command.

+       /* skip auto conf temporary file */
+       if (strncmp(de->d_name,
+                   PG_AUTOCONF_FILENAME ".",
+                   sizeof(PG_AUTOCONF_FILENAME)) == 0)
+           continue;

What's the dot doing there?


+       if ((stat(AutoConfFileName, &st) == -1))
+           ereport(elevel,
+               (errmsg("configuration parameters changed with ALTER SYSTEM command before restart of server "
+                       "will not be effective as \"%s\"  file is not accessible.", PG_AUTOCONF_FILENAME)));
+       else
+           ereport(elevel,
+               (errmsg("configuration parameters changed with ALTER SYSTEM command before restart of server "
+                       "will not be effective as default include directive for  \"%s\" folder is not present in postgresql.conf", PG_AUTOCONF_DIR)));

These messages should be split.  Perhaps have the "will not be
effective" in the errmsg() and the reason as part of errdetail()?  Also,
I'm not really sure about doing another stat() on the file after parsing
failed; I think the parse routine should fill some failure context
information, instead of having this code trying to reproduce the failure
to know what to report.  Maybe something like the SlruErrorCause enum,
not sure.

Similarly,

+   /*
+     * record if the file currently being parsed is postgresql.auto.conf,
+     * so that it can be later used to give warning if it doesn't parse
+     * it.
+    */
+   snprintf(Filename,sizeof(Filename),"%s/%s", PG_AUTOCONF_DIR, PG_AUTOCONF_FILENAME);
+   ConfigAutoFileName = AbsoluteConfigLocation(Filename, ConfigFileName);
+   if (depth == 1 && strcmp(ConfigAutoFileName, config_file) == 0)
+       *is_config_dir_parsed = true;

this seems very odd.  The whole "is_config_dir_parsed" mechanism smells
strange to me, really.  I think the caller should be in charge of
keeping track of this, but I'm not sure.  ParseConfigFp() would have no
way of knowing, and in one place we're calling that routine precisely to
parse the auto file.

Thanks,

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Simon Riggs <simon@2ndquadrant.com> — 2013-07-03T06:12:23Z

On 22 June 2013 05:17, Amit kapila <amit.kapila@huawei.com> wrote:

> On Friday, June 21, 2013 11:48 PM Robert Haas wrote:
> On Thu, Jun 20, 2013 at 12:18 AM, Amit Kapila <amit.kapila@huawei.com>
> wrote:
> >> Auto.conf            - 1 Vote (Josh)
> >> System.auto.conf     - 1 Vote (Josh)
> >> Postgresql.auto.conf - 2 Votes (Zoltan, Amit)
> >> Persistent.auto.conf - 0 Vote
> >> generated_by_server.conf - 1 Vote (Peter E)
> >> System.conf              - 1 Vote (Magnus)
> >> alter_system.conf        - 1 Vote (Alvaro)
> >
> >> I had consolidated the list, so that it would be helpful for committer
> to
> >> decide among proposed names for this file.
>
> > I'll also vote for postgresql.auto.conf.
>
>   Thanks to all of you for suggesting meaningful names. I will change the
> name of file to postgresql.auto.conf.
>   Kindly let me know if there is any objection to it.
>

postgresql.auto.conf is similar enough to postgresql.conf that it will
prevent tab completion from working as it does now. As a result it will
cause people to bring that file up for editing when we wish to avoid that.

So for me the name is not arbitrary because of that point and we should
avoid the current choice.

Can we go for auto.postgresql.conf instead .... almost identical, just
significantly visually different and not interfering with tab completion?

-- 
 Simon Riggs                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila@huawei.com> — 2013-07-03T06:45:57Z

On Wednesday, July 03, 2013 11:42 AM Simon Riggs wrote:
On 22 June 2013 05:17, Amit kapila <amit.kapila@huawei.com> wrote:
On Friday, June 21, 2013 11:48 PM Robert Haas wrote:
On Thu, Jun 20, 2013 at 12:18 AM, Amit Kapila <amit.kapila@huawei.com>
wrote:
>>>> Auto.conf            - 1 Vote (Josh)
>>>> System.auto.conf     - 1 Vote (Josh)
>>>> Postgresql.auto.conf - 2 Votes (Zoltan, Amit)
>>>> Persistent.auto.conf - 0 Vote
>>>> generated_by_server.conf - 1 Vote (Peter E)
>>>> System.conf              - 1 Vote (Magnus)
>>>> alter_system.conf        - 1 Vote (Alvaro)
>
>>>> I had consolidated the list, so that it would be helpful for committer
to
>>>> decide among proposed names for this file.

>>> I'll also vote for postgresql.auto.conf.
>>  Thanks to all of you for suggesting meaningful names. I will change the
name of file to postgresql.auto.conf.
>>  Kindly let me know if there is any objection to it.

> postgresql.auto.conf is similar enough to postgresql.conf that it will
prevent tab completion from working as it does now. As a result it will
cause  
> people to bring that file up for editing when we wish to avoid that.

  This new file postgresql.auto.conf will be created inside a new directory
"config", so both will be in different directories. 
  Will there be any confusion with tab completion for different directories?

> So for me the name is not arbitrary because of that point and we should
avoid the current choice.
 
> Can we go for auto.postgresql.conf instead .... almost identical, just
significantly visually different and not interfering with tab completion?

With Regards,
Amit Kapila.

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila@huawei.com> — 2013-07-03T10:31:16Z

On Wednesday, July 03, 2013 2:58 AM Alvaro Herrera wrote:
> Amit Kapila escribió:
> 
> > I have changed the file name to postgresql.auto.conf and I have
> changed the
> > name of SetPersistentLock to AutoFileLock.
> >
> > Zoltan,
> >
> > Could you please once check the attached Patch if you have time, as
> now all
> > the things are resolved except for small doubt in syntax
> extensibility
> > which can be changed based on final decision.
> 
> There are a bunch of whitespace problems, as you would notice with "git
> diff --check" or "git show --color".  Could you please clean that up?
> Also, some of the indentation in various places is not right; perhaps
> you could fix that by running pgindent over the source files you
> modified (this is easily visible by eyeballing the git diff output;
> stuff is misaligned in pretty obvious ways.)
> 
> Random minor other comments:
> 
> +     use of <xref linkend="SQL-ALTER SYSTEM">
> 
> this SGML link cannot possibly work.  Please run "make check" in the
> doc/src/sgml directory.
> 
> Examples in alter_system.sgml have a typo "SYTEM".  Same file has "or
> or".

I will fix above issues.

> Also in that file,
>       The name of an configuration parameter that exist in
> <filename>postgresql.conf</filename>.
> the parameter needn't exist in that file to be settable, right?
> 
>  <refnamediv>
>   <refname>ALTER SYSTEM</refname>
>   <refpurpose>change a server configuration parameter</refpurpose>
>  </refnamediv>

Yes you are right. How about below line, same parameter description as SET
command 
Name of a settable run-time parameter. Available parameters are documented
in Chapter 18

> Perhaps "permanently change .."?

I am not sure, if adding "permanently" is appropriate here, as one could
also interpret it, that after parameter is changed with this command, it can
never be changed again.

> Also, some parameters require a restart, not a reload; maybe we should
> direct the user somewhere else to learn how to freshen up the
> configuration after calling the command.

Below link contains useful information in this regard: 
http://www.postgresql.org/docs/devel/static/config-setting.html 

Particularly refer below para in above link (the below text is for your
reference to see if above link is useful): 
The configuration file is reread whenever the main server process receives a
SIGHUP signal; this is most easily done by running pg_ctl reload from the
command-line or by calling the SQL function pg_reload_conf(). The main
server process also propagates this signal to all currently running server
processes so that existing sessions also get the new value. Alternatively,
you can send the signal to a single server process directly. Some parameters
can only be set at server start; any changes to their entries in the
configuration file will be ignored until the server is restarted. Invalid
parameter settings in the configuration file are likewise ignored (but
logged) during SIGHUP processing.

> 
> +       /* skip auto conf temporary file */
> +       if (strncmp(de->d_name,
> +                   PG_AUTOCONF_FILENAME ".",
> +                   sizeof(PG_AUTOCONF_FILENAME)) == 0)
> +           continue;
> 
> What's the dot doing there?

This was from previous version of patch, now it is not required, I will
remove it.

> 
> +       if ((stat(AutoConfFileName, &st) == -1))
> +           ereport(elevel,
> +               (errmsg("configuration parameters changed with ALTER
> SYSTEM command before restart of server "
> +                       "will not be effective as \"%s\"  file is not
> accessible.", PG_AUTOCONF_FILENAME)));
> +       else
> +           ereport(elevel,
> +               (errmsg("configuration parameters changed with ALTER
> SYSTEM command before restart of server "
> +                       "will not be effective as default include
> directive for  \"%s\" folder is not present in postgresql.conf",
> PG_AUTOCONF_DIR)));
> 
> These messages should be split.  Perhaps have the "will not be
> effective" in the errmsg() and the reason as part of errdetail()?

How about changing to below messages 

ereport(elevel, 
               (errmsg("configuration parameters changed with ALTER SYSTEM
command will not be effective") 
                errdetail("file \"%s\" containing configuration parameters
is not accessible.", PG_AUTOCONF_FILENAME))); 

ereport(elevel, 
               (errmsg("configuration parameters changed with ALTER SYSTEM
command will not be effective") 
                errdetail("default include directive for  \"%s\" folder is
not present in postgresql.conf", PG_AUTOCONF_DIR)));

> Also,
> I'm not really sure about doing another stat() on the file after
> parsing
> failed; I think the parse routine should fill some failure context
> information, instead of having this code trying to reproduce the
> failure
> to know what to report.  Maybe something like the SlruErrorCause enum,
> not sure.

It might not be feasible with current implementation as currently it just
note down whether it has parsed file with name 
postgresql.auto.conf and then in outer function takes decision based on that
parameter. 
However if we change current implementation for knowing whether it has
parsed postgresql.auto.conf, then might be it is possible. 
See in below point one suggestion to achieve the same.

> Similarly,
> 
> +   /*
> +     * record if the file currently being parsed is
> postgresql.auto.conf,
> +     * so that it can be later used to give warning if it doesn't
> parse
> +     * it.
> +    */
> +   snprintf(Filename,sizeof(Filename),"%s/%s", PG_AUTOCONF_DIR,
> PG_AUTOCONF_FILENAME);
> +   ConfigAutoFileName = AbsoluteConfigLocation(Filename,
> ConfigFileName);
> +   if (depth == 1 && strcmp(ConfigAutoFileName, config_file) == 0)
> +       *is_config_dir_parsed = true;
> 
> this seems very odd.  The whole "is_config_dir_parsed" mechanism smells
> strange to me, really.  I think the caller should be in charge of
> keeping track of this, but I'm not sure.  ParseConfigFp() would have no
> way of knowing, and in one place we're calling that routine precisely
> to
> parse the auto file.

How about changing it in way such that 
1. we introduce a new enum ConfigDirParseError (3 values a.
config_dir_not_parsed, config_dir_parsed, config_file_not_accessible)    and
a new static variable confdirerr. 
2. variable should be initialized with config_dir_not_parsed in function
ParseConfigFile() 
3. Now in function ParseConfigDirectory, we compare that passed directory
name is "config" , then we set confdirerr = config_dir_parsed 
4. Further in function ParseConfigDirectory, check during ReadDir, if it has
read file postgresql.auto.conf 

I think above should handle both error cases 
a. include directive for config file is commented or not present; 
b. file postgresql.auto.conf is not accessible

With Regards,
Amit Kapila.

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Simon Riggs <simon@2ndquadrant.com> — 2013-07-03T12:59:15Z

On 3 July 2013 07:45, Amit Kapila <amit.kapila@huawei.com> wrote:

> > postgresql.auto.conf is similar enough to postgresql.conf that it will
> prevent tab completion from working as it does now. As a result it will
> cause
> > people to bring that file up for editing when we wish to avoid that.
>
>   This new file postgresql.auto.conf will be created inside a new directory
> "config", so both will be in different directories.
>   Will there be any confusion with tab completion for different
> directories?

Tab completion will not be confused, no.

But I think everything else will be. How will I know that some settings
have been set by ALTER SYSTEM and some by other means?

Sounds to me like a recipe for chaos. I hope nobody is calling for a commit
on this anytime soon. I think the whole thing needs some careful thought
and review before we commit this.

 --
 Simon Riggs                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila@huawei.com> — 2013-07-03T13:29:31Z

On Wednesday, July 03, 2013 6:29 PM Simon Riggs wrote:
On 3 July 2013 07:45, Amit Kapila <amit.kapila@huawei.com> wrote:
 
>>> postgresql.auto.conf is similar enough to postgresql.conf that it will
prevent tab completion from working as it does now. As a result it will
cause
>>> people to bring that file up for editing when we wish to avoid that.
  
>>This new file postgresql.auto.conf will be created inside a new directory
>>"config", so both will be in different directories.
>>  Will there be any confusion with tab completion for different
directories?

> Tab completion will not be confused, no.

> But I think everything else will be. 
>How will I know that some settings have been set by ALTER SYSTEM and some
by other means?

Other means by hand editing postgresql.auto.conf?
If not we can check in pg_settings, it shows sourcefile. So if the setting
is from new file postgresql.auto.conf, this means it is set by ALTER SYSTEM.


With Regards,
Amit Kapila.

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila@huawei.com> — 2013-07-05T05:19:17Z

On Wednesday, July 03, 2013 2:58 AM Alvaro Herrera wrote:
> Amit Kapila escribió:
> 
> > I have changed the file name to postgresql.auto.conf and I have
> changed the
> > name of SetPersistentLock to AutoFileLock.
> >
> > Zoltan,
> >
> > Could you please once check the attached Patch if you have time, as
> now all
> > the things are resolved except for small doubt in syntax
> extensibility
> > which can be changed based on final decision.
> 
> There are a bunch of whitespace problems, as you would notice with "git
> diff --check" or "git show --color".  Could you please clean that up?

Fixed all whitespaces.

> Also, some of the indentation in various places is not right; perhaps
> you could fix that by running pgindent over the source files you
> modified (this is easily visible by eyeballing the git diff output;
> stuff is misaligned in pretty obvious ways.)

Fixed, by running pgindent


 
> Random minor other comments:
> 
> +     use of <xref linkend="SQL-ALTER SYSTEM">
> 
> this SGML link cannot possibly work.  Please run "make check" in the
> doc/src/sgml directory.

Fixed, make check passes now.

> Examples in alter_system.sgml have a typo "SYTEM".  Same file has "or
> or".

Fixed.

> Also in that file,
>       The name of an configuration parameter that exist in
> <filename>postgresql.conf</filename>.
> the parameter needn't exist in that file to be settable, right?

Changed to below text:
Name of a settable run-time parameter.  Available parameters are documented
in <xref linkend="runtime-config">.
   

>  <refnamediv>
>   <refname>ALTER SYSTEM</refname>
>   <refpurpose>change a server configuration parameter</refpurpose>
>  </refnamediv>
> 
> Perhaps "permanently change .."?

Not changed.

> 
> Also, some parameters require a restart, not a reload; maybe we should
> direct the user somewhere else to learn how to freshen up the
> configuration after calling the command.
> 
> +       /* skip auto conf temporary file */
> +       if (strncmp(de->d_name,
> +                   PG_AUTOCONF_FILENAME ".",
> +                   sizeof(PG_AUTOCONF_FILENAME)) == 0)
> +           continue;
> 
> What's the dot doing there?

Fixed, removed dot.
> 
> 
> +       if ((stat(AutoConfFileName, &st) == -1))
> +           ereport(elevel,
> +               (errmsg("configuration parameters changed with ALTER
> SYSTEM command before restart of server "
> +                       "will not be effective as \"%s\"  file is not
> accessible.", PG_AUTOCONF_FILENAME)));
> +       else
> +           ereport(elevel,
> +               (errmsg("configuration parameters changed with ALTER
> SYSTEM command before restart of server "
> +                       "will not be effective as default include
> directive for  \"%s\" folder is not present in postgresql.conf",
> PG_AUTOCONF_DIR)));
> 
> These messages should be split.  Perhaps have the "will not be
> effective" in the errmsg() and the reason as part of errdetail()?

Okay, changed as per suggestion.

> Also,
> I'm not really sure about doing another stat() on the file after
> parsing
> failed; I think the parse routine should fill some failure context
> information, instead of having this code trying to reproduce the
> failure
> to know what to report.  Maybe something like the SlruErrorCause enum,
> not sure.
> 
> Similarly,
> 
> +   /*
> +     * record if the file currently being parsed is
> postgresql.auto.conf,
> +     * so that it can be later used to give warning if it doesn't
> parse
> +     * it.
> +    */
> +   snprintf(Filename,sizeof(Filename),"%s/%s", PG_AUTOCONF_DIR,
> PG_AUTOCONF_FILENAME);
> +   ConfigAutoFileName = AbsoluteConfigLocation(Filename,
> ConfigFileName);
> +   if (depth == 1 && strcmp(ConfigAutoFileName, config_file) == 0)
> +       *is_config_dir_parsed = true;
> 
> this seems very odd.  The whole "is_config_dir_parsed" mechanism smells
> strange to me, really.  I think the caller should be in charge of
> keeping track of this, but I'm not sure.  ParseConfigFp() would have no
> way of knowing, and in one place we're calling that routine precisely
> to
> parse the auto file.

Changed by adding new enum AutoConfErrorCause. Now is_config_dir_parsed is
removed from code.
Kindly let me know if this way is better than previous?

Apart from above I have fixed one issue in function
AlterSystemSetConfigFile(), called FreeConfigVariables().

With Regards,
Amit Kapila.

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Fujii Masao <masao.fujii@gmail.com> — 2013-07-11T18:32:34Z

On Fri, Jul 5, 2013 at 2:19 PM, Amit Kapila <amit.kapila@huawei.com> wrote:
> On Wednesday, July 03, 2013 2:58 AM Alvaro Herrera wrote:
>> Amit Kapila escribió:
>>
>> > I have changed the file name to postgresql.auto.conf and I have
>> changed the
>> > name of SetPersistentLock to AutoFileLock.
>> >
>> > Zoltan,
>> >
>> > Could you please once check the attached Patch if you have time, as
>> now all
>> > the things are resolved except for small doubt in syntax
>> extensibility
>> > which can be changed based on final decision.
>>
>> There are a bunch of whitespace problems, as you would notice with "git
>> diff --check" or "git show --color".  Could you please clean that up?
>
> Fixed all whitespaces.
>
>> Also, some of the indentation in various places is not right; perhaps
>> you could fix that by running pgindent over the source files you
>> modified (this is easily visible by eyeballing the git diff output;
>> stuff is misaligned in pretty obvious ways.)
>
> Fixed, by running pgindent
>
>
>
>> Random minor other comments:
>>
>> +     use of <xref linkend="SQL-ALTER SYSTEM">
>>
>> this SGML link cannot possibly work.  Please run "make check" in the
>> doc/src/sgml directory.
>
> Fixed, make check passes now.
>
>> Examples in alter_system.sgml have a typo "SYTEM".  Same file has "or
>> or".
>
> Fixed.
>
>> Also in that file,
>>       The name of an configuration parameter that exist in
>> <filename>postgresql.conf</filename>.
>> the parameter needn't exist in that file to be settable, right?
>
> Changed to below text:
> Name of a settable run-time parameter.  Available parameters are documented
> in <xref linkend="runtime-config">.
>
>
>>  <refnamediv>
>>   <refname>ALTER SYSTEM</refname>
>>   <refpurpose>change a server configuration parameter</refpurpose>
>>  </refnamediv>
>>
>> Perhaps "permanently change .."?
>
> Not changed.
>
>>
>> Also, some parameters require a restart, not a reload; maybe we should
>> direct the user somewhere else to learn how to freshen up the
>> configuration after calling the command.
>>
>> +       /* skip auto conf temporary file */
>> +       if (strncmp(de->d_name,
>> +                   PG_AUTOCONF_FILENAME ".",
>> +                   sizeof(PG_AUTOCONF_FILENAME)) == 0)
>> +           continue;
>>
>> What's the dot doing there?
>
> Fixed, removed dot.
>>
>>
>> +       if ((stat(AutoConfFileName, &st) == -1))
>> +           ereport(elevel,
>> +               (errmsg("configuration parameters changed with ALTER
>> SYSTEM command before restart of server "
>> +                       "will not be effective as \"%s\"  file is not
>> accessible.", PG_AUTOCONF_FILENAME)));
>> +       else
>> +           ereport(elevel,
>> +               (errmsg("configuration parameters changed with ALTER
>> SYSTEM command before restart of server "
>> +                       "will not be effective as default include
>> directive for  \"%s\" folder is not present in postgresql.conf",
>> PG_AUTOCONF_DIR)));
>>
>> These messages should be split.  Perhaps have the "will not be
>> effective" in the errmsg() and the reason as part of errdetail()?
>
> Okay, changed as per suggestion.
>
>> Also,
>> I'm not really sure about doing another stat() on the file after
>> parsing
>> failed; I think the parse routine should fill some failure context
>> information, instead of having this code trying to reproduce the
>> failure
>> to know what to report.  Maybe something like the SlruErrorCause enum,
>> not sure.
>>
>> Similarly,
>>
>> +   /*
>> +     * record if the file currently being parsed is
>> postgresql.auto.conf,
>> +     * so that it can be later used to give warning if it doesn't
>> parse
>> +     * it.
>> +    */
>> +   snprintf(Filename,sizeof(Filename),"%s/%s", PG_AUTOCONF_DIR,
>> PG_AUTOCONF_FILENAME);
>> +   ConfigAutoFileName = AbsoluteConfigLocation(Filename,
>> ConfigFileName);
>> +   if (depth == 1 && strcmp(ConfigAutoFileName, config_file) == 0)
>> +       *is_config_dir_parsed = true;
>>
>> this seems very odd.  The whole "is_config_dir_parsed" mechanism smells
>> strange to me, really.  I think the caller should be in charge of
>> keeping track of this, but I'm not sure.  ParseConfigFp() would have no
>> way of knowing, and in one place we're calling that routine precisely
>> to
>> parse the auto file.
>
> Changed by adding new enum AutoConfErrorCause. Now is_config_dir_parsed is
> removed from code.
> Kindly let me know if this way is better than previous?
>
> Apart from above I have fixed one issue in function
> AlterSystemSetConfigFile(), called FreeConfigVariables().

I got the following compile warnings.

guc.c:5187: warning: no previous prototype for 'validate_conf_option'
preproc.y:7746.2-31: warning: type clash on default action: <str> != <>

The make installcheck failed when I ran it against the server with
wal_level = hot_standby. The regression.diff is

------------------------------------
*** 27,35 ****
  (1 row)

  show wal_level;
!  wal_level
! -----------
!  minimal
  (1 row)

  show authentication_timeout;
--- 27,35 ----
  (1 row)

  show wal_level;
!   wal_level
! -------------
!  hot_standby
  (1 row)

  show authentication_timeout;
------------------------------------

The regression test of ALTER SYSTEM calls pg_sleep(1) six times.
Those who dislike the regression test patches which were proposed
in this CF might dislike this repeat of pg_sleep(1) because it would
increase the time of regression test.

+		/* skip auto conf temporary file */
+		if (strncmp(de->d_name,
+					PG_AUTOCONF_FILENAME,
+					sizeof(PG_AUTOCONF_FILENAME)) == 0)
+			continue;

Why do we need to exclude the auto conf file from the backup?
I think that it should be included in the backup as well as normal
postgresql.conf.

+	autofile = fopen(path, PG_BINARY_W);
+	if (autofile == NULL)
+	{
+		fprintf(stderr, _("%s: could not open file \"%s\" for writing: %s\n"),
+				progname, path, strerror(errno));
+		exit_nicely();
+	}
+
+	if (fputs("# Do not edit this file manually! It is overwritten by
the ALTER SYSTEM command \n",
+			  autofile) < 0)
+	{
+		fprintf(stderr, _("%s: could not write file \"%s\": %s\n"),
+				progname, path, strerror(errno));
+		exit_nicely();
+	}
+
+	if (fclose(autofile))
+	{
+		fprintf(stderr, _("%s: could not close file \"%s\": %s\n"),
+				progname, path, strerror(errno));
+		exit_nicely();
+	}

You can simplify the code by using writefile().

Regards,

-- 
Fujii Masao

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila@huawei.com> — 2013-07-12T13:15:40Z

On Friday, July 12, 2013 12:02 AM Fujii Masao wrote:
On Fri, Jul 5, 2013 at 2:19 PM, Amit Kapila <amit.kapila@huawei.com> wrote:
> On Wednesday, July 03, 2013 2:58 AM Alvaro Herrera wrote:
>> Amit Kapila escribió:
>>

> I got the following compile warnings.

> guc.c:5187: warning: no previous prototype for 'validate_conf_option'
> preproc.y:7746.2-31: warning: type clash on default action: <str> != <>

I generally use windows as dev environment, it hasn't shown these warnings.
I shall check in linux and correct the same.

> The make installcheck failed when I ran it against the server with
> wal_level = hot_standby. The regression.diff is

> ------------------------------------
> *** 27,35 ****
>  (1 row)

>  show wal_level;
> !  wal_level
> ! -----------
> !  minimal
>   (1 row)

>  show authentication_timeout;
> --- 27,35 ----
>  (1 row)

>  show wal_level;
> !   wal_level
> ! -------------
> !  hot_standby
>   (1 row)

>   show authentication_timeout;
> ------------------------------------

The tests in alter_system.sql are dependent on default values of postgresql.conf, which is okay for make check 
but not for installcheck. I shall remove that dependency from testcases.


> The regression test of ALTER SYSTEM calls pg_sleep(1) six times.
> Those who dislike the regression test patches which were proposed
> in this CF might dislike this repeat of pg_sleep(1) because it would
> increase the time of regression test.

The sleep is used to ensure the effects of pg_reload_conf() can be visible. 
To avoid increasing time of regression tests, we can do one of below:

1) decrease the time for sleep, but not sure how to safely decide how much to sleep.
2) combine the tests such that only 1 or 2 sleep calls should be used.

what's your opinion?

> +               /* skip auto conf temporary file */
> +               if (strncmp(de->d_name,
> +                                       PG_AUTOCONF_FILENAME,
> +                                       sizeof(PG_AUTOCONF_FILENAME)) == 0)
> +                       continue;

> Why do we need to exclude the auto conf file from the backup?
> I think that it should be included in the backup as well as normal
> postgresql.conf.

  The original intention was to skip the autoconf temporary file which is created during AlterSystemSetConfigFile()
  for crash safety. I will change to exclude temp autoconf file.

> +       autofile = fopen(path, PG_BINARY_W);
> +       if (autofile == NULL)
> +       {
> +               fprintf(stderr, _("%s: could not open file \"%s\" for writing: %s\n"),
> +                               progname, path, strerror(errno));
> +               exit_nicely();
> +       }
> +
> +       if (fputs("# Do not edit this file manually! It is overwritten by
> the ALTER SYSTEM command \n",
> +                         autofile) < 0)
> +       {
> +               fprintf(stderr, _("%s: could not write file \"%s\": %s\n"),
> +                               progname, path, strerror(errno));
> +               exit_nicely();
> +       }
> +
> +       if (fclose(autofile))
> +       {
> +               fprintf(stderr, _("%s: could not close file \"%s\": %s\n"),
> +                               progname, path, strerror(errno));
> +               exit_nicely();
> +       }

> You can simplify the code by using writefile().

  Yes, it is better to use writefile(). I shall update the patch for same.

With Regards,
Amit Kapila

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Andrew Dunstan <andrew@dunslane.net> — 2013-07-12T15:10:28Z

On 07/12/2013 09:15 AM, Amit kapila wrote:
>> guc.c:5187: warning: no previous prototype for 'validate_conf_option'
>> preproc.y:7746.2-31: warning: type clash on default action: <str> != <>
> I generally use windows as dev environment, it hasn't shown these warnings.


Hackers, please take note. Assertions that nobody uses Windows for 
development are false.

cheers

andrew

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Josh Berkus <josh@agliodbs.com> — 2013-07-12T16:37:29Z

On 07/12/2013 06:15 AM, Amit kapila wrote:
> I generally use windows as dev environment, it hasn't shown these warnings.
> I shall check in linux and correct the same.

Really?  Hey, I'm gonna send you a lot of Windows-specific patches for
testing in the future ...

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila@huawei.com> — 2013-07-13T03:38:19Z

On Friday, July 12, 2013 10:07 PM Josh Berkus wrote:
On 07/12/2013 06:15 AM, Amit kapila wrote:
>> I generally use windows as dev environment, it hasn't shown these warnings.
>> I shall check in linux and correct the same.

> Really?  

  Yes.

> Hey, I'm gonna send you a lot of Windows-specific patches for
>testing in the future ...

I would be happy to participate as much as I can.

With Regards,
Amit Kapila.

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila@huawei.com> — 2013-07-15T13:43:19Z

On Friday, July 12, 2013 6:46 PM Amit kapila wrote:
> On Friday, July 12, 2013 12:02 AM Fujii Masao wrote:
> On Fri, Jul 5, 2013 at 2:19 PM, Amit Kapila <amit.kapila@huawei.com>
> wrote:
> > On Wednesday, July 03, 2013 2:58 AM Alvaro Herrera wrote:
> >> Amit Kapila escribió:
> >>
> 
> > I got the following compile warnings.
> 
> > guc.c:5187: warning: no previous prototype for 'validate_conf_option'
> > preproc.y:7746.2-31: warning: type clash on default action: <str> !=
> <>
 
    In gram.y semicolon (';') was missing, due to which it was not
generating proper preproc.y

> I generally use windows as dev environment, it hasn't shown these
> warnings.
> I shall check in linux and correct the same.

  Fixed.
> 
> > The make installcheck failed when I ran it against the server with
> > wal_level = hot_standby. The regression.diff is
> 
> > ------------------------------------
> > *** 27,35 ****
> >  (1 row)
> 
> >  show wal_level;
> > !  wal_level
> > ! -----------
> > !  minimal
> >   (1 row)
> 
> >  show authentication_timeout;
> > --- 27,35 ----
> >  (1 row)
> 
> >  show wal_level;
> > !   wal_level
> > ! -------------
> > !  hot_standby
> >   (1 row)
> 
> >   show authentication_timeout;
> > ------------------------------------
> 
> The tests in alter_system.sql are dependent on default values of
> postgresql.conf, which is okay for make check
> but not for installcheck. I shall remove that dependency from
> testcases.

  Removed all tests for which values were dependent on postgresql.conf
  a. removed check of values before reload
  b. removed parameters which can only set after server restart
  c. removed check for values after setting to default
 
> 
> > The regression test of ALTER SYSTEM calls pg_sleep(1) six times.
> > Those who dislike the regression test patches which were proposed
> > in this CF might dislike this repeat of pg_sleep(1) because it would
> > increase the time of regression test.
> 
> The sleep is used to ensure the effects of pg_reload_conf() can be
> visible.
> To avoid increasing time of regression tests, we can do one of below:
> 
> 1) decrease the time for sleep, but not sure how to safely decide how
> much to sleep.
> 2) combine the tests such that only 1 or 2 sleep calls should be used.
> 
> what's your opinion?

Modified to use 2 sleep calls.

> > +               /* skip auto conf temporary file */
> > +               if (strncmp(de->d_name,
> > +                                       PG_AUTOCONF_FILENAME,
> > +                                       sizeof(PG_AUTOCONF_FILENAME))
> == 0)
> > +                       continue;
> 
> > Why do we need to exclude the auto conf file from the backup?
> > I think that it should be included in the backup as well as normal
> > postgresql.conf.
> 
>   The original intention was to skip the autoconf temporary file which
> is created during AlterSystemSetConfigFile()
>   for crash safety. I will change to exclude temp autoconf file.

   I had modified the check to exclude postgresql.auto.conf.temp file. I
have used hardcoded ".temp" instead of macro,
as I don't find other use of same in code.
 
> > +       autofile = fopen(path, PG_BINARY_W);
> > +       if (autofile == NULL)
> > +       {
> > +               fprintf(stderr, _("%s: could not open file \"%s\" for
> writing: %s\n"),
> > +                               progname, path, strerror(errno));
> > +               exit_nicely();
> > +       }
> > +
> > +       if (fputs("# Do not edit this file manually! It is
> overwritten by
> > the ALTER SYSTEM command \n",
> > +                         autofile) < 0)
> > +       {
> > +               fprintf(stderr, _("%s: could not write file \"%s\":
> %s\n"),
> > +                               progname, path, strerror(errno));
> > +               exit_nicely();
> > +       }
> > +
> > +       if (fclose(autofile))
> > +       {
> > +               fprintf(stderr, _("%s: could not close file \"%s\":
> %s\n"),
> > +                               progname, path, strerror(errno));
> > +               exit_nicely();
> > +       }
> 
> > You can simplify the code by using writefile().
> 
>   Yes, it is better to use writefile(). I shall update the patch for
> same.

    Modified to use writefile().


With Regards,
Amit Kapila.

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Fujii Masao <masao.fujii@gmail.com> — 2013-07-15T18:21:59Z

On Mon, Jul 15, 2013 at 10:43 PM, Amit Kapila <amit.kapila@huawei.com> wrote:
> On Friday, July 12, 2013 6:46 PM Amit kapila wrote:
>> On Friday, July 12, 2013 12:02 AM Fujii Masao wrote:
>> On Fri, Jul 5, 2013 at 2:19 PM, Amit Kapila <amit.kapila@huawei.com>
>> wrote:
>> > On Wednesday, July 03, 2013 2:58 AM Alvaro Herrera wrote:
>> >> Amit Kapila escribió:
>> >>
>>
>> > I got the following compile warnings.
>>
>> > guc.c:5187: warning: no previous prototype for 'validate_conf_option'
>> > preproc.y:7746.2-31: warning: type clash on default action: <str> !=
>> <>
>
>     In gram.y semicolon (';') was missing, due to which it was not
> generating proper preproc.y
>
>> I generally use windows as dev environment, it hasn't shown these
>> warnings.
>> I shall check in linux and correct the same.
>
>   Fixed.
>>
>> > The make installcheck failed when I ran it against the server with
>> > wal_level = hot_standby. The regression.diff is
>>
>> > ------------------------------------
>> > *** 27,35 ****
>> >  (1 row)
>>
>> >  show wal_level;
>> > !  wal_level
>> > ! -----------
>> > !  minimal
>> >   (1 row)
>>
>> >  show authentication_timeout;
>> > --- 27,35 ----
>> >  (1 row)
>>
>> >  show wal_level;
>> > !   wal_level
>> > ! -------------
>> > !  hot_standby
>> >   (1 row)
>>
>> >   show authentication_timeout;
>> > ------------------------------------
>>
>> The tests in alter_system.sql are dependent on default values of
>> postgresql.conf, which is okay for make check
>> but not for installcheck. I shall remove that dependency from
>> testcases.
>
>   Removed all tests for which values were dependent on postgresql.conf
>   a. removed check of values before reload
>   b. removed parameters which can only set after server restart
>   c. removed check for values after setting to default
>
>>
>> > The regression test of ALTER SYSTEM calls pg_sleep(1) six times.
>> > Those who dislike the regression test patches which were proposed
>> > in this CF might dislike this repeat of pg_sleep(1) because it would
>> > increase the time of regression test.
>>
>> The sleep is used to ensure the effects of pg_reload_conf() can be
>> visible.
>> To avoid increasing time of regression tests, we can do one of below:
>>
>> 1) decrease the time for sleep, but not sure how to safely decide how
>> much to sleep.
>> 2) combine the tests such that only 1 or 2 sleep calls should be used.
>>
>> what's your opinion?
>
> Modified to use 2 sleep calls.
>
>> > +               /* skip auto conf temporary file */
>> > +               if (strncmp(de->d_name,
>> > +                                       PG_AUTOCONF_FILENAME,
>> > +                                       sizeof(PG_AUTOCONF_FILENAME))
>> == 0)
>> > +                       continue;
>>
>> > Why do we need to exclude the auto conf file from the backup?
>> > I think that it should be included in the backup as well as normal
>> > postgresql.conf.
>>
>>   The original intention was to skip the autoconf temporary file which
>> is created during AlterSystemSetConfigFile()
>>   for crash safety. I will change to exclude temp autoconf file.
>
>    I had modified the check to exclude postgresql.auto.conf.temp file. I
> have used hardcoded ".temp" instead of macro,
> as I don't find other use of same in code.
>
>> > +       autofile = fopen(path, PG_BINARY_W);
>> > +       if (autofile == NULL)
>> > +       {
>> > +               fprintf(stderr, _("%s: could not open file \"%s\" for
>> writing: %s\n"),
>> > +                               progname, path, strerror(errno));
>> > +               exit_nicely();
>> > +       }
>> > +
>> > +       if (fputs("# Do not edit this file manually! It is
>> overwritten by
>> > the ALTER SYSTEM command \n",
>> > +                         autofile) < 0)
>> > +       {
>> > +               fprintf(stderr, _("%s: could not write file \"%s\":
>> %s\n"),
>> > +                               progname, path, strerror(errno));
>> > +               exit_nicely();
>> > +       }
>> > +
>> > +       if (fclose(autofile))
>> > +       {
>> > +               fprintf(stderr, _("%s: could not close file \"%s\":
>> %s\n"),
>> > +                               progname, path, strerror(errno));
>> > +               exit_nicely();
>> > +       }
>>
>> > You can simplify the code by using writefile().
>>
>>   Yes, it is better to use writefile(). I shall update the patch for
>> same.
>
>     Modified to use writefile().

Thanks for updating the patch!

In the patch, ALTER SYSTEM SET validates the postgresql.conf.
I think this is overkill. While ALTER SYSTEM SET is being executed,
a user might be modifying the postgresql.conf. That validation
breaks this use case.

+# This includes the default configuration directory included to support
+# ALTER SYSTEM statement.
+#
+# WARNING: User should not remove below include_dir or directory config,
+#          as both are required to make ALTER SYSTEM command work.
+#          Any configuration parameter values specified after this line
+#          will override the values set by ALTER SYSTEM statement.
+#include_dir = 'config'

Why do we need to expose this setting to a user? As the warning says,
if a user *should not* remove this setting, I think we should not expose
it from the beginning and should always treat the postgresql.auto.conf
as included configuration file even if that setting is not in postgresql.conf.

Regards,

-- 
Fujii Masao

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila@huawei.com> — 2013-07-16T04:23:02Z

On Monday, July 15, 2013 11:51 PM Fujii Masao wrote:
On Mon, Jul 15, 2013 at 10:43 PM, Amit Kapila <amit.kapila@huawei.com> wrote:
> On Friday, July 12, 2013 6:46 PM Amit kapila wrote:
>> On Friday, July 12, 2013 12:02 AM Fujii Masao wrote:
>> On Fri, Jul 5, 2013 at 2:19 PM, Amit Kapila <amit.kapila@huawei.com>
>> wrote:
>> > On Wednesday, July 03, 2013 2:58 AM Alvaro Herrera wrote:
>> >> Amit Kapila escribió:
>> >>
>>

> Thanks for updating the patch!

> In the patch, ALTER SYSTEM SET validates the postgresql.conf.
> I think this is overkill. 

I assume you are talking about below code, if I am wrong please point me to the exact code you are reffering:

  /*
     * Verify if include_dir for postgresql.auto.conf file exists in
     * postgresql.conf. If it doesn't exist then warn the user that parameters
     * changed with this command will not be effective.
     */
    autoconf_errcause = AUTOCONF_DIR_NOT_PARSED;

    if (!ParseConfigFile(ConfigFileName, NULL, false, 0, LOG, &head, &tail))
        ereport(ERROR,
                (errmsg("configuration file \"%s\" contains errors;
parameters changed by "
                     "this command will not be effective", ConfigFileName)));

    /*
     * ignore the list of options as we are intersted to just ensure the
     * existence of include directive for config folder.
     */
    head = tail = NULL;

    if (autoconf_errcause == AUTOCONF_DIR_NOT_PARSED)
        ereport(WARNING,
                (errmsg("configuration parameters changed by this
command will not be effective"),
                 errdetail("default include directive for  \"%s\"
folder is not present in postgresql.conf", PG_AUTOCONF_DIR)));

This code is to parse postgresql.conf to check if include directive for config directory is present.


> While ALTER SYSTEM SET is being executed,
> a user might be modifying the postgresql.conf. That validation
> breaks this use case.

> +# This includes the default configuration directory included to support
> +# ALTER SYSTEM statement.
> +#
> +# WARNING: User should not remove below include_dir or directory config,
> +#          as both are required to make ALTER SYSTEM command work.
> +#          Any configuration parameter values specified after this line
> +#          will override the values set by ALTER SYSTEM statement.
> +#include_dir = 'config'

> Why do we need to expose this setting to a user? 

a) This can be a knob to turn this feature off. This has been asked by few people, 
   one of the mail link is mentioned below (refer towards end of mail in the link):
   http://www.postgresql.org/message-id/515B04F9.30900@gmx.net
    
b) In case user wants to change priority of parameters set by Alter System, he can move the
   include_dir up or down in postgresql.conf.

> As the warning says,
> if a user *should not* remove this setting, I think we should not expose
> it from the beginning and should always treat the postgresql.auto.conf
> as included configuration file even if that setting is not in postgresql.conf.

I think it will be usefull to keep include_dir for some users.

If you think that above use for keeping include_dir is not worth or it can be provided in some 
another way, then we can change this behavior and remove parsing of postgresql.conf from 
AlterSystemSetConfigFile() function.


Thank you very much for reviewing this patch so carefully and giving valuable feedback.


With Regards,
Amit Kapila.

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Alvaro Herrera <alvherre@2ndquadrant.com> — 2013-07-17T19:01:14Z

Amit kapila escribió:


> > +# This includes the default configuration directory included to support
> > +# ALTER SYSTEM statement.
> > +#
> > +# WARNING: User should not remove below include_dir or directory config,
> > +#          as both are required to make ALTER SYSTEM command work.
> > +#          Any configuration parameter values specified after this line
> > +#          will override the values set by ALTER SYSTEM statement.
> > +#include_dir = 'config'
> 
> > Why do we need to expose this setting to a user? 
> 
> a) This can be a knob to turn this feature off. This has been asked by few people, 
>    one of the mail link is mentioned below (refer towards end of mail in the link):
>    http://www.postgresql.org/message-id/515B04F9.30900@gmx.net
>     
> b) In case user wants to change priority of parameters set by Alter System, he can move the
>    include_dir up or down in postgresql.conf.

Both of these seem like they would make troubleshooters' lives more
difficult.  I think we should just parse the auto file automatically
after parsing postgresql.conf, without requiring the include directive
to be there.

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Josh Berkus <josh@agliodbs.com> — 2013-07-17T19:07:53Z

On 07/17/2013 12:01 PM, Alvaro Herrera wrote:
> Both of these seem like they would make troubleshooters' lives more
> difficult.  I think we should just parse the auto file automatically
> after parsing postgresql.conf, without requiring the include directive
> to be there.

Wait, I thought the auto file was going into the conf.d directory?


-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Robert Haas <robertmhaas@gmail.com> — 2013-07-17T19:30:32Z

On Fri, Jul 12, 2013 at 9:15 AM, Amit kapila <amit.kapila@huawei.com> wrote:
> The sleep is used to ensure the effects of pg_reload_conf() can be visible.

That strikes me as almost certain to result in random regression test
failures.  My experience, as a man who helps to maintain a very large
suite of regression tests, is that all time-based assumptions about
how long it will take some operation to complete are false, and that
failures are a lot more frequent than you think they will be.

This feature doesn't strike me as a good candidate for regression
testing anyway.  Keep in mind that people are intended to be able to
run make installcheck against a deployed system without messing it up.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Alvaro Herrera <alvherre@2ndquadrant.com> — 2013-07-17T21:05:40Z

Robert Haas escribió:

> This feature doesn't strike me as a good candidate for regression
> testing anyway.  Keep in mind that people are intended to be able to
> run make installcheck against a deployed system without messing it up.

This is my opinion as well.

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Andres Freund <andres@2ndquadrant.com> — 2013-07-17T22:17:35Z

On 2013-07-17 17:05:40 -0400, Alvaro Herrera wrote:
> Robert Haas escribió:
> 
> > This feature doesn't strike me as a good candidate for regression
> > testing anyway.  Keep in mind that people are intended to be able to
> > run make installcheck against a deployed system without messing it up.
> 
> This is my opinion as well.

Setting and resetting a GUC like pg_regress.test_var or so shouldn't be
too invasive. Some minimal testing seems like a good idea to me.

Greetings,

Andres Freund

-- 
 Andres Freund	                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Josh Berkus <josh@agliodbs.com> — 2013-07-17T22:28:58Z

On 07/17/2013 03:17 PM, Andres Freund wrote:
> On 2013-07-17 17:05:40 -0400, Alvaro Herrera wrote:
>> Robert Haas escribió:
>>
>>> This feature doesn't strike me as a good candidate for regression
>>> testing anyway.  Keep in mind that people are intended to be able to
>>> run make installcheck against a deployed system without messing it up.
>>
>> This is my opinion as well.
> 
> Setting and resetting a GUC like pg_regress.test_var or so shouldn't be
> too invasive. Some minimal testing seems like a good idea to me.

Also, we can "make check" even if we don't "make installcheck".

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Peter Eisentraut <peter_e@gmx.net> — 2013-07-18T00:39:26Z

If we are going to add ALTER SYSTEM, then I'd like to consider recasting
ALTER ROLE ALL SET in terms of ALTER SYSTEM as well, because the ALTER
ROLE ALL syntax was a hack.  At least it should be possible to consider
various levels of "system".  Who knows, maybe a future version will
allow propagating settings to standby servers?  I don't know what a good
terminology would be, but let's keep some options open.

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Peter Eisentraut <peter_e@gmx.net> — 2013-07-18T00:47:12Z

On Tue, 2013-07-16 at 04:23 +0000, Amit kapila wrote:
> 
> > Why do we need to expose this setting to a user? 
> 
> a) This can be a knob to turn this feature off. This has been asked by
> few people, 
>    one of the mail link is mentioned below (refer towards end of mail
> in the link):
>    http://www.postgresql.org/message-id/515B04F9.30900@gmx.net
>     
> b) In case user wants to change priority of parameters set by Alter
> System, he can move the
>    include_dir up or down in postgresql.conf. 

In that message I argue that the auto file should always be read
implicitly.

The comment about being able to turn this feature off was because as
presented it was too complicated and error-prone.  Removing the include
would incidentally not be a way to turn the feature off.  Turning the
feature of would require disabling the ALTER SYSTEM command.  I'm not
really asking for that functionality, but I'm really against having to
manually include the auto file, and also against the system nagging me
about including the file.

There is the slightly interesting question whether the auto file should
notionally be processed before or after the main postgresql.conf, but
that's a question we should answer now, not let it the users work it
out.

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila@huawei.com> — 2013-07-18T04:01:59Z

On Thursday, July 18, 2013 2:36 AM Alvaro Herrera wrote:
> Robert Haas escribió:
> 
> > This feature doesn't strike me as a good candidate for regression
> > testing anyway.  Keep in mind that people are intended to be able to
> > run make installcheck against a deployed system without messing it
> up.
> 
> This is my opinion as well.

Summarization of all tests in this patch:
1. Set of SIGHUP parameters in config file, use reload and sleep and then
check value of parameters to validate whether they are set.
2. Reset all parameters so that the system returns to previous state, use
reload and sleep
3. Negative scenario tests

As suggested by Robert that having timing based tests are not good candidate
for regression tests, which essentially means that all tests
as part of point 1 & 2 are not good candidates for testing. If tests
corresponding to point 1 && 2 are to be removed, then keeping some negative
tests
also doesn't seem to make much sense.

To cover Alter System functionality and code, we can have Alter System
commands without reload and sleep, but I think it might not be of much use
until we verify the same using Show.  

So I will remove the tests from this patch in next updated version if there
is no other useful way to achieve it.

On a side note, I had checked that in regression suite already few tests use
pg_sleep() and one of them in stats.sql is something similar. It also does
wait using pg_sleep() to let prior tests action get completed. 
I understand that having already at one place doesn't allow to use it at
other places, but still I think that if some tests using similar mechanism
are running successfully, we can add new one provided they ensure the timing
doesn't cause problem.

With Regards,
Amit Kapila.

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila@huawei.com> — 2013-07-18T04:53:51Z

On Thursday, July 18, 2013 12:31 AM Alvaro Herrera wrote:
> Amit kapila escribió:
> 
> 
> > > +# This includes the default configuration directory included to
> support
> > > +# ALTER SYSTEM statement.
> > > +#
> > > +# WARNING: User should not remove below include_dir or directory
> config,
> > > +#          as both are required to make ALTER SYSTEM command work.
> > > +#          Any configuration parameter values specified after this
> line
> > > +#          will override the values set by ALTER SYSTEM statement.
> > > +#include_dir = 'config'
> >
> > > Why do we need to expose this setting to a user?
> >
> > a) This can be a knob to turn this feature off. This has been asked
> by few people,
> >    one of the mail link is mentioned below (refer towards end of mail
> in the link):
> >    http://www.postgresql.org/message-id/515B04F9.30900@gmx.net
> >
> > b) In case user wants to change priority of parameters set by Alter
> System, he can move the
> >    include_dir up or down in postgresql.conf.
> 
> Both of these seem like they would make troubleshooters' lives more
> difficult.  I think we should just parse the auto file automatically
> after parsing postgresql.conf, without requiring the include directive
> to be there.

Okay, I shall modify the patch that way. However still the file will be in
config directory.
Now the 2 things regarding this needs to be taken care:

a. As raised by Peter Eisentraut, shall we do parsing of autofile before or
after postgresql.conf. By default I will keep it after until there is some
reason to do it before.
b. If the directory/file is missing while parsing issue warning message.

Kindly let me know your opinion about the same.

With Regards,
Amit Kapila.

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila@huawei.com> — 2013-07-18T11:25:46Z

On Thursday, July 18, 2013 12:38 AM Josh Berkus wrote:
> On 07/17/2013 12:01 PM, Alvaro Herrera wrote:
> > Both of these seem like they would make troubleshooters' lives more
> > difficult.  I think we should just parse the auto file automatically
> > after parsing postgresql.conf, without requiring the include
> directive
> > to be there.
> 
> Wait, I thought the auto file was going into the conf.d directory?

Auto file is going into config directory, but will that make any difference 
if we have to parse it automatically after postgresql.conf.

With Regards,
Amit Kapila.

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Josh Berkus <josh@agliodbs.com> — 2013-07-18T17:45:05Z

On 07/18/2013 04:25 AM, Amit Kapila wrote:
> On Thursday, July 18, 2013 12:38 AM Josh Berkus wrote:
>> On 07/17/2013 12:01 PM, Alvaro Herrera wrote:
>>> Both of these seem like they would make troubleshooters' lives more
>>> difficult.  I think we should just parse the auto file automatically
>>> after parsing postgresql.conf, without requiring the include
>> directive
>>> to be there.
>>
>> Wait, I thought the auto file was going into the conf.d directory?
> 
> Auto file is going into config directory, but will that make any difference 
> if we have to parse it automatically after postgresql.conf.

Well, I thought that the whole conf.d directory automatically got parsed
after postgresql.conf.  No?

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Fujii Masao <masao.fujii@gmail.com> — 2013-07-18T19:26:05Z

On Fri, Jul 19, 2013 at 2:45 AM, Josh Berkus <josh@agliodbs.com> wrote:
> On 07/18/2013 04:25 AM, Amit Kapila wrote:
>> On Thursday, July 18, 2013 12:38 AM Josh Berkus wrote:
>>> On 07/17/2013 12:01 PM, Alvaro Herrera wrote:
>>>> Both of these seem like they would make troubleshooters' lives more
>>>> difficult.  I think we should just parse the auto file automatically
>>>> after parsing postgresql.conf, without requiring the include
>>> directive
>>>> to be there.
>>>
>>> Wait, I thought the auto file was going into the conf.d directory?
>>
>> Auto file is going into config directory, but will that make any difference
>> if we have to parse it automatically after postgresql.conf.
>
> Well, I thought that the whole conf.d directory automatically got parsed
> after postgresql.conf.  No?

No, in the previous patch. We needed to set include_dir to 'config' (though
that's the default).

Regards,

-- 
Fujii Masao

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Alvaro Herrera <alvherre@2ndquadrant.com> — 2013-07-18T20:02:53Z

Fujii Masao escribió:
> On Fri, Jul 19, 2013 at 2:45 AM, Josh Berkus <josh@agliodbs.com> wrote:
> > On 07/18/2013 04:25 AM, Amit Kapila wrote:
> >> On Thursday, July 18, 2013 12:38 AM Josh Berkus wrote:
> >>> On 07/17/2013 12:01 PM, Alvaro Herrera wrote:
> >>>> Both of these seem like they would make troubleshooters' lives more
> >>>> difficult.  I think we should just parse the auto file automatically
> >>>> after parsing postgresql.conf, without requiring the include
> >>> directive
> >>>> to be there.
> >>>
> >>> Wait, I thought the auto file was going into the conf.d directory?
> >>
> >> Auto file is going into config directory, but will that make any difference
> >> if we have to parse it automatically after postgresql.conf.
> >
> > Well, I thought that the whole conf.d directory automatically got parsed
> > after postgresql.conf.  No?
> 
> No, in the previous patch. We needed to set include_dir to 'config' (though
> that's the default).

I know this has been discussed already, but I want to do a quick summary
of existing proposals, point out their drawbacks, and present a proposal
of my own.  Note I ignore the whole file-per-setting vs.
one-file-to-rule-them-all, because the latter has already been shot
down.  So live ideas floated here are:

1. have a config/postgresql.auto.conf file, require include_dir or
   include in postgresql.conf
   This breaks if user removes the config/ directory, or if the user
   removes the include_dir directive from postgresql.conf.  ALTER SYSTEM
   is in the business of doing mkdir() or failing altogether if the dir
   is not present.  Doesn't seem friendly.

2. have a conf.d/ directory, put the file therein; no include or
   include_dir directive is necessary.
   I think this is a separate patch and merits its own discussion.  This
   might be a good idea, but I don't think that this is the
   way to implement ALTER SYSTEM.  If users don't want to allow conf.d
   they can remove it, but that would break ALTER SYSTEM unnecessarily.
   Since they are separate features it seems to me that they should
   function independently.

I think we should just put the config directives of ALTER SYSTEM into a
file, not dir, alongside postgresql.conf; I would suggest
postgresql.auto.conf.  This file is parsed automatically after
postgresql.conf, without requiring an "include" directive in
postgresql.conf.  If the user does not want that file, they can remove
it; but a future ALTER SYSTEM will create the file again.  No need to
parse stuff to find out whether the directory exists, or the file
exists, or such nonsense.

I think the only drawback of this is that there's no way to disable
ALTER SYSTEM (i.e. there's no directory you can remove to prevent the
thing from working).  But is this a desirable property?  I mean, if we
want to disallow ALTER SYSTEM I think we should provide a direct way to
do that, perhaps allow_alter_system = off in postgresql.conf; but I
don't think we need to provide this, really, at least not in the first
implementation.

Note that the Debian packaging uses postgres-writable directories for
its configuration files, so the daemon is always able to create the file
in the config dir.

This seems the simplest way; config tools such as Puppet know they
always need to consider the postgresql.auto.conf file.

I think the whole business of parsing the file, and then verifying
whether the auto file has been parsed, is nonsensical and should be
removed from the patch.

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Josh Berkus <josh@agliodbs.com> — 2013-07-18T21:03:17Z

Alvaro,

> I think the only drawback of this is that there's no way to disable
> ALTER SYSTEM (i.e. there's no directory you can remove to prevent the
> thing from working).  But is this a desirable property?  I mean, if we
> want to disallow ALTER SYSTEM I think we should provide a direct way to
> do that, perhaps allow_alter_system = off in postgresql.conf; but I
> don't think we need to provide this, really, at least not in the first
> implementation.

Agreed that turning alter system off by deleting the directory is NOT a
desireable feature.  I'm also unclear on the desire to disable ALTER
SYSTEM; if someone has superuser access, then they can just use
pg_write_file to add config directives anyway, no?  So there's not any
security value in disabling it.  Maybe there's a case I'm not thinking of.

Of course, people *can* disable it by creating a blank
postgresql.auto.conf file in the right place and making it
non-writeable, if they want.

We are missing one feature, which is the ability to relocate the
postgresql.auto.conf file if relocating it is desireable according to
some sysadmin spec.  This kind of ties into another patch which was
discussed on this list -- the ability to relocate the recovery.conf
file.  Personally, I think that it would be better for the users if we
provided a way to relocate *all* conf files to the same master
directory, but that we don't need a way to relocate each config file
individually, but that's a longer discussion.

In other words, let's accept an ALTER SYSTEM patch which doesn't support
relocating, and then argue whether a second patch which supports
relocating is needed.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Alvaro Herrera <alvherre@2ndquadrant.com> — 2013-07-18T22:03:36Z

Josh Berkus escribió:

> We are missing one feature, which is the ability to relocate the
> postgresql.auto.conf file if relocating it is desireable according to
> some sysadmin spec.  This kind of ties into another patch which was
> discussed on this list -- the ability to relocate the recovery.conf
> file.

Well, postgresql.conf is already relocatable.  Is there any advantage in
being able to move postgresql.auto.conf to a different location than
postgresql.conf?  I don't see any.  I didn't follow the recovery.conf
discussion, but I imagine that the reason for wanting to be able to
relocate it is related to standby vs. master distinction.  This doesn't
apply to postgresql.auto.conf, I think, does it?

If people want to drill real down, they can always have a
postgresql.auto.conf that's a symlink. (In this light, we would ship an
empty postgresql.auto.conf in a freshly initdb'd system.  IIRC the
current patch already does that.)

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Greg Smith <greg@2ndquadrant.com> — 2013-07-19T00:31:09Z

On 7/18/13 4:02 PM, Alvaro Herrera wrote:

> I think we should just put the config directives of ALTER SYSTEM into a
> file, not dir, alongside postgresql.conf; I would suggest
> postgresql.auto.conf.  This file is parsed automatically after
> postgresql.conf, without requiring an "include" directive in
> postgresql.conf.

It is possible to build ALTER SYSTEM SET this way.  One of the goals of 
the implementation style used here wasn't just to accomplish that narrow 
feature though.  We keep running into situations where a standard, 
directory based configuration system would make things easier.  Each 
time that happens, someone comes up with another way to avoid doing it. 
  I think that approach contributes to stagnation in the terrible status 
quo of PostgreSQL configuration management.

I thought this was a good spot to try and re-draw this line because I 
don't want just one program that is able to create new configuration 
entries easily.  I want to see a whole universe of them.  ALTER SYSTEM 
SET, tuning helpers, replication helpers, logging helpers, vacuum 
schedulers.  All of them *could* just dump a simple file into a config 
directory with code anyone can write.  And having ALTER SYSTEM SET do 
that provides a strong precedent for how it can be done.  (I'd like to 
see initdb do that instead of hacking the system postgresql.conf as if 
sed-style edits were still the new hotness, but that's a future change)

My claim, and that's one informed by writing pgtune, is that by far the 
hardest part of writing a configuration addition tool is parsing a 
postgresql.conf file.  The expectation right now is that all changes 
must happen there.  Creating a new snippet of configuration settings is 
easy, but no tools know where to put one right now.  Instead we just 
keep coming up with single, hard-coded file names that people have to 
know in order to manage their installations.

> This seems the simplest way; config tools such as Puppet know they
> always need to consider the postgresql.auto.conf file.

Like this idea.  What administrators really want, whether they realize 
it or not, is to point Puppet at a configuration directory.  Then the 
problem of "what are the config files in the new version of Postgres" 
happens once more and then it's over.  Exactly what the files in there 
are named should be completely under control of the administrator.

We're never going to reach that unless we lead by example though.  The 
database's configuration pushes people toward using a small number of 
files with magic names--postgresql.conf, recovery.conf, and now 
postgresql.auto.conf in your proposal.  Meanwhile, all sensible 
UNIX-style projects with complicated configurations in text files has 
moved toward a configuration directory full of them.  Here are some 
directories on the last RHEL6 system I was editing configuration on this 
week:

/etc/httpd/conf.d/
/etc/munin/conf.d/
/etc/ld.so.conf.d/
/etc/munin/conf.d/
/etc/dovecot/conf.d/
/etc/yum/pluginconf.d/

Some of them didn't get the memo that the right standard name is conf.d 
now, but they're the minority.

It's fine to have a postgresql.conf file that you *can* make all your 
changes to, for people who want to stay in the old monolithic approach. 
  But if there were also a conf.d directory under there, many classes of 
administrator would breath a sign of relief.  With all the precedents 
people may have already ran into, with the right naming can be made 
discoverable and familiar to a lot of administrators.

Telling them instead that there's this new postgresql.auto.conf file 
that suddenly they have to worry about--I'd say don't even bother if 
that's how you want to do it.  That's making the problem I've been 
trying to simplify for five years now even worse.

> I think the whole business of parsing the file, and then verifying
> whether the auto file has been parsed, is nonsensical and should be
> removed from the patch.

That was just trying to keep people from screwing up their configuration 
while they get used to things.  That and some of the other sanity 
checking is not necessary, it was just trying to make the transition to 
the new configuration approach less error-prone.  I don't think anyone 
would disagree that the current patch is doing enough of that error 
checking work that the error checking itself is the most likely thing to 
break.

-- 
Greg Smith   2ndQuadrant US    greg@2ndQuadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Josh Berkus <josh@agliodbs.com> — 2013-07-19T03:36:02Z

Greg,

> I thought this was a good spot to try and re-draw this line because I
> don't want just one program that is able to create new configuration
> entries easily.  I want to see a whole universe of them.  ALTER SYSTEM
> SET, tuning helpers, replication helpers, logging helpers, vacuum
> schedulers.  All of them *could* just dump a simple file into a config
> directory with code anyone can write.  And having ALTER SYSTEM SET do
> that provides a strong precedent for how it can be done.  (I'd like to
> see initdb do that instead of hacking the system postgresql.conf as if
> sed-style edits were still the new hotness, but that's a future change)

Thank you.  I wanted to say this, but I couldn't find a way to express it.

> Some of them didn't get the memo that the right standard name is conf.d
> now, but they're the minority.

Apparently we didn't get the memo either.


-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila@huawei.com> — 2013-07-19T04:58:24Z

On Friday, July 19, 2013 1:33 AM Alvaro Herrera wrote:
> Fujii Masao escribió:
> > On Fri, Jul 19, 2013 at 2:45 AM, Josh Berkus <josh@agliodbs.com>
> wrote:
> > > On 07/18/2013 04:25 AM, Amit Kapila wrote:
> > >> On Thursday, July 18, 2013 12:38 AM Josh Berkus wrote:
> > >>> On 07/17/2013 12:01 PM, Alvaro Herrera wrote:
> > >>>> Both of these seem like they would make troubleshooters' lives
> more
> > >>>> difficult.  I think we should just parse the auto file
> automatically
> > >>>> after parsing postgresql.conf, without requiring the include
> > >>> directive
> > >>>> to be there.
> > >>>
> > >>> Wait, I thought the auto file was going into the conf.d
> directory?
> > >>
> > >> Auto file is going into config directory, but will that make any
> difference
> > >> if we have to parse it automatically after postgresql.conf.
> > >
> > > Well, I thought that the whole conf.d directory automatically got
> parsed
> > > after postgresql.conf.  No?
> >
> > No, in the previous patch. We needed to set include_dir to 'config'
> (though
> > that's the default).
> 
> I know this has been discussed already, but I want to do a quick
> summary
> of existing proposals, point out their drawbacks, and present a
> proposal
> of my own.  Note I ignore the whole file-per-setting vs.
> one-file-to-rule-them-all, because the latter has already been shot
> down.  So live ideas floated here are:
> 
> 1. have a config/postgresql.auto.conf file, require include_dir or
>    include in postgresql.conf
>    This breaks if user removes the config/ directory, or if the user
>    removes the include_dir directive from postgresql.conf.  ALTER
> SYSTEM
>    is in the business of doing mkdir() or failing altogether if the dir
>    is not present.  Doesn't seem friendly.
> 
> 2. have a conf.d/ directory, put the file therein; no include or
>    include_dir directive is necessary.
>    I think this is a separate patch and merits its own discussion.
> This
>    might be a good idea, but I don't think that this is the
>    way to implement ALTER SYSTEM.  If users don't want to allow conf.d
>    they can remove it, but that would break ALTER SYSTEM unnecessarily.
>    Since they are separate features it seems to me that they should
>    function independently.
> 
> I think we should just put the config directives of ALTER SYSTEM into a
> file, not dir, alongside postgresql.conf; I would suggest
> postgresql.auto.conf.  This file is parsed automatically after
> postgresql.conf, without requiring an "include" directive in
> postgresql.conf.  If the user does not want that file, they can remove
> it; but a future ALTER SYSTEM will create the file again.  

> No need to
> parse stuff to find out whether the directory exists, or the file
> exists, or such nonsense.

  I think this will be removed from patch, once we implement automatic
parsing of config/postgresql.auto.conf
  after parsing postgresql.conf
 

> I think the only drawback of this is that there's no way to disable
> ALTER SYSTEM (i.e. there's no directory you can remove to prevent the
> thing from working).  But is this a desirable property?  I mean, if we
> want to disallow ALTER SYSTEM I think we should provide a direct way to
> do that, perhaps allow_alter_system = off in postgresql.conf; but I
> don't think we need to provide this, really, at least not in the first
> implementation.
> 
> Note that the Debian packaging uses postgres-writable directories for
> its configuration files, so the daemon is always able to create the
> file
> in the config dir.
> 
> This seems the simplest way; config tools such as Puppet know they
> always need to consider the postgresql.auto.conf file.
 


> I think the whole business of parsing the file, and then verifying
> whether the auto file has been parsed, is nonsensical and should be
> removed from the patch.

  Okay, I understood your concern about checking during parsing to 
  find error whether directory/file exist.
  In the next version, I will remove this error.

With Regards,
Amit Kapila.

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila@huawei.com> — 2013-07-22T16:58:28Z

Amit Kapila wrote:
On Thursday, July 18, 2013 12:31 AM Alvaro Herrera wrote:
> Amit kapila escribió:
> 
> 
>> > > +# This includes the default configuration directory included to
>> support
>> > > +# ALTER SYSTEM statement.
>> > > +#
>> > > +# WARNING: User should not remove below include_dir or directory
>> config,
>> > > +#          as both are required to make ALTER SYSTEM command work.
>> > > +#          Any configuration parameter values specified after this
>> line
>> > > +#          will override the values set by ALTER SYSTEM statement.
>> > > +#include_dir = 'config'
>> >
>> > > Why do we need to expose this setting to a user?
>> >
>> > a) This can be a knob to turn this feature off. This has been asked
>> by few people,
>> >    one of the mail link is mentioned below (refer towards end of mail
>> in the link):
>> >    http://www.postgresql.org/message-id/515B04F9.30900@gmx.net
>> >
>> > b) In case user wants to change priority of parameters set by Alter
>> System, he can move the
>> >    include_dir up or down in postgresql.conf.
>> 
>> Both of these seem like they would make troubleshooters' lives more
>> difficult.  I think we should just parse the auto file automatically
>> after parsing postgresql.conf, without requiring the include directive
>> to be there.

> Okay, I shall modify the patch that way. However still the file will be in
> config directory.
> Now the 2 things regarding this needs to be taken care:

> a. As raised by Peter Eisentraut, shall we do parsing of autofile before or
> after postgresql.conf. By default I will keep it after until there is some
> reason to do it before.
> b. If the directory/file is missing while parsing issue warning message.



As per discussion, updated patch contains following changes:
1. Regression tests for Alter System are removed 
2. Parsed the auto file automatically after parsing postgresql.conf 
3. Removed addition of include directive in postgresql.conf 
4. Removed error handling for parsing errors

With Regards,
Amit Kapila.

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Greg Smith <greg@2ndquadrant.com> — 2013-07-22T19:06:26Z

On 7/22/13 12:58 PM, Amit kapila wrote:
> As per discussion, updated patch contains following changes:
> 1. Regression tests for Alter System are removed
> 2. Parsed the auto file automatically after parsing postgresql.conf
> 3. Removed addition of include directive in postgresql.conf
> 4. Removed error handling for parsing errors

These changes have shrunk the diff down to 1411 lines of code.

I'd like to identify which committer might take this on at this point. 
In a few respects this is "Ready For Committer" now, because the 
committer who takes this on is going to get a strong vote on how to 
resolve most of the remaining fundamental issues:

-Is this the point to finally commit to the config directory approach?

-If this does set the config directory usage precedent, is the name used 
for that appropriate?  Whoever suggested the change from "conf.d" to 
"config" made an error IMHO.  Every example I've found of other projects 
doing this style of config refactoring picked either "conf.d" or a 
unique, no two are alike name.  I'd rather not see Postgres add yet 
another unique one.  (I think the 'postgresql' part of 
postgresql.auto.conf as the name of the file is redundant too--what else 
would be in the Postgres config directory but postgresql files?--but 
that's not a major issue)

This could definitely use a round of committer level review of how the 
GUC handling is being done now too.  That part of the code seems to have 
settled, and things like using the new validate_conf_option could be 
committed even with other parts still being discussed.  Exactly how to 
best break this out into useful commits is another decision that really 
needs some input from the potential committer though.

-- 
Greg Smith   2ndQuadrant US    greg@2ndQuadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Alvaro Herrera <alvherre@2ndquadrant.com> — 2013-07-22T19:12:21Z

Greg Smith escribió:
> On 7/22/13 12:58 PM, Amit kapila wrote:
> >As per discussion, updated patch contains following changes:
> >1. Regression tests for Alter System are removed
> >2. Parsed the auto file automatically after parsing postgresql.conf
> >3. Removed addition of include directive in postgresql.conf
> >4. Removed error handling for parsing errors
> 
> These changes have shrunk the diff down to 1411 lines of code.

Nice.

> I'd like to identify which committer might take this on at this
> point.

I'm considering take a stab at it soonish, FWIW, if no other committer
steps up.

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Josh Berkus <josh@agliodbs.com> — 2013-07-22T23:35:47Z

All,

Christophe just discovered something with include files which is going
to cause issues with ALTER SYSTEM SET.

So, take as a hypothetical that you use the default postgresql.conf
file, which sets shared_buffers = 32MB.

Instead of editing this file, you do ALTER SYSTEM SET shared_buffers =
'1GB', which updates config/postgresql.auto.conf.

Then you restart PostgreSQL.

Everything is hunky-dory, until a later occasion where you *reload*
postgresql.  Then postgres startup hits the first "shared_buffers=32MB"
(in postgresql.conf), says "I can't change shared buffers on a reload",
and aborts before ever getting to postgresql.auto.conf.

This is a problem that exists now with includes in postgresql.conf, but
having ALTER SYSTEM SET will cause more users to hit it.  Seems like
we'll need to fix it before releasing 9.4.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Tom Lane <tgl@sss.pgh.pa.us> — 2013-07-22T23:56:19Z

Josh Berkus <josh@agliodbs.com> writes:
> Christophe just discovered something with include files which is going
> to cause issues with ALTER SYSTEM SET.

> So, take as a hypothetical that you use the default postgresql.conf
> file, which sets shared_buffers = 32MB.

> Instead of editing this file, you do ALTER SYSTEM SET shared_buffers =
> '1GB', which updates config/postgresql.auto.conf.

> Then you restart PostgreSQL.

> Everything is hunky-dory, until a later occasion where you *reload*
> postgresql.

Everything was already *not* hunky-dory as soon as you did that, since
a SIGHUP would have had the same problem.

I'd be inclined to think that ALTER SYSTEM SET should not be allowed to
modify any PGC_POSTMASTER parameters.

			regards, tom lane

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila@huawei.com> — 2013-07-23T05:01:40Z

> On Tuesday, July 23, 2013 5:26 AM Tom Lane wrote:
> Josh Berkus <josh@agliodbs.com> writes:
> > Christophe just discovered something with include files which is
> going
> > to cause issues with ALTER SYSTEM SET.
> 
> > So, take as a hypothetical that you use the default postgresql.conf
> > file, which sets shared_buffers = 32MB.
> 
> > Instead of editing this file, you do ALTER SYSTEM SET shared_buffers
> =
> > '1GB', which updates config/postgresql.auto.conf.
> 
> > Then you restart PostgreSQL.
> 
> > Everything is hunky-dory, until a later occasion where you *reload*
> > postgresql.
> 
> Everything was already *not* hunky-dory as soon as you did that, since
> a SIGHUP would have had the same problem.
> 
> I'd be inclined to think that ALTER SYSTEM SET should not be allowed to
> modify any PGC_POSTMASTER parameters.

  One way to fix the problem is that while parsing if the option already
exists, replace it.
  Something like below code

ParseConfigFp() 
{ 
.. 
.. 
/* replace the option value, if already exists in list */ 
                        for (item = *head_p; item != NULL; item =
item->next) 
                        { 
                                if (strcmp(item->name, opt_name) == 0) 
                                { 
                                        pfree(item->value); 
                                        item->value = pstrdup(opt_value); 
                                        replaced = true; 
                                        break; 
                                } 
                        } 
                        if(!replaced) 
                        { 
                                /* ordinary variable, append to list */ 
                                item = palloc(sizeof *item); 
                                item->name = opt_name; 
                                item->value = opt_value; 
                                item->filename = pstrdup(config_file); 
                                item->sourceline = ConfigFileLineno-1; 
                                item->next = NULL; 
                                if (*head_p == NULL) 
                                        *head_p = item; 
                                else 
                                        (*tail_p)->next = item; 
                                *tail_p = item; 
                        } 

.. 
.. 
}

There is overhead of traversing the list each time, but as this path is
traversed in less and non-performance critical operations, it can be
considered to fix the problem.

If you consider above as a non-trivial or not a right way to fix the
problem, 
then I can update the patch to disallow PGC_POSTMASTER parameters by ALTER
SYSTEM SET command.

With Regards,
Amit Kapila.

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Josh Berkus <josh@agliodbs.com> — 2013-07-23T21:42:59Z

> Everything was already *not* hunky-dory as soon as you did that, since
> a SIGHUP would have had the same problem.
> 
> I'd be inclined to think that ALTER SYSTEM SET should not be allowed to
> modify any PGC_POSTMASTER parameters.

That makes alter system set a bunch less useful, but might be
insurmountable.

Anyway, I also regard this as a problem we need to resolve now that we
have the config directory if we expect people to build autotuning tools.
 Once someone is relying on an autotuning tool which drops a file in
config/, it becomes a real problem that there are uncommented
PGC_POSTMASTER options in the default postgresql.conf.

I'm not sure what the solution to that problem should be, but I do know
that we're going to hear it a lot from users as includes and the config
directory get more common.  Certainly any solution which says "first,
manually edit postgresql.conf" makes the config directory into a kind of
joke.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila@huawei.com> — 2013-07-24T05:06:18Z

On Tuesday, July 23, 2013 5:06 AM Josh Berkus wrote:
> All,
> 
> Christophe just discovered something with include files which is going
> to cause issues with ALTER SYSTEM SET.
> 
> So, take as a hypothetical that you use the default postgresql.conf
> file, which sets shared_buffers = 32MB.
> 
> Instead of editing this file, you do ALTER SYSTEM SET shared_buffers =
> '1GB', which updates config/postgresql.auto.conf.
> 
> Then you restart PostgreSQL.
> 
> Everything is hunky-dory, until a later occasion where you *reload*
> postgresql.  Then postgres startup hits the first "shared_buffers=32MB"
> (in postgresql.conf), says "I can't change shared buffers on a reload",
> and aborts before ever getting to postgresql.auto.conf.

It doesn't abort after showing initial message "parameter %s cannot be changed without restart", rather it processes all remaining parameters.
We can even test it by setting 1 postmaster and 1 sighup variable, after reload, even though it log message for postmaster variable, but it will set
the sighup variable.

So in this the real problem is that there is some message in log which can create confusion, otherwise the behavior will be exactly what user wants as per Postgres specs.

To avoid the message, following can be done:
1. We need to make sure that in function ProcessConfigFile before calling set_config_option(), list of parameters is unique.
   a. it can be done while storing the elements in list as suggested by me in my yesterday's mail.
      http://www.postgresql.org/message-id/004c01ce8761$b8a4ab20$29ee0160$@kapila@huawei.com
   b. after parsing all the config files, make sure the list contain one entry for any variable 
      (giving priority to elements that come later)
2. As this is just a confusing message issue, we can even update the docs to explain this behavior.


With Regards,
Amit Kapila.

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Robert Haas <robertmhaas@gmail.com> — 2013-07-25T19:58:06Z

On Thu, Jul 18, 2013 at 6:03 PM, Alvaro Herrera
<alvherre@2ndquadrant.com> wrote:
> Josh Berkus escribió:
>
>> We are missing one feature, which is the ability to relocate the
>> postgresql.auto.conf file if relocating it is desireable according to
>> some sysadmin spec.  This kind of ties into another patch which was
>> discussed on this list -- the ability to relocate the recovery.conf
>> file.
>
> Well, postgresql.conf is already relocatable.  Is there any advantage in
> being able to move postgresql.auto.conf to a different location than
> postgresql.conf?  I don't see any.  I didn't follow the recovery.conf
> discussion, but I imagine that the reason for wanting to be able to
> relocate it is related to standby vs. master distinction.  This doesn't
> apply to postgresql.auto.conf, I think, does it?
>
> If people want to drill real down, they can always have a
> postgresql.auto.conf that's a symlink. (In this light, we would ship an
> empty postgresql.auto.conf in a freshly initdb'd system.  IIRC the
> current patch already does that.)

My thought is that people might put postgresql.conf in a directory
that only contains configuration files and isn't writeable by the
postgres user.  So I would expect to find postgresql.auto.conf in the
data directory always.

But in general +1 for your proposal.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Robert Haas <robertmhaas@gmail.com> — 2013-07-25T20:01:00Z

On Mon, Jul 22, 2013 at 7:56 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Josh Berkus <josh@agliodbs.com> writes:
>> Christophe just discovered something with include files which is going
>> to cause issues with ALTER SYSTEM SET.
>
>> So, take as a hypothetical that you use the default postgresql.conf
>> file, which sets shared_buffers = 32MB.
>
>> Instead of editing this file, you do ALTER SYSTEM SET shared_buffers =
>> '1GB', which updates config/postgresql.auto.conf.
>
>> Then you restart PostgreSQL.
>
>> Everything is hunky-dory, until a later occasion where you *reload*
>> postgresql.
>
> Everything was already *not* hunky-dory as soon as you did that, since
> a SIGHUP would have had the same problem.
>
> I'd be inclined to think that ALTER SYSTEM SET should not be allowed to
> modify any PGC_POSTMASTER parameters.

That significantly decreases the usefulness of ALTER SYSTEM without
actually preventing the underlying problem.  If having multiple
conflicting values for parameters in the config files doesn't work
cleanly, then we should fix that, not cripple this feature.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Alvaro Herrera <alvherre@2ndquadrant.com> — 2013-07-25T20:50:16Z

Robert Haas escribió:
> On Mon, Jul 22, 2013 at 7:56 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

> > I'd be inclined to think that ALTER SYSTEM SET should not be allowed to
> > modify any PGC_POSTMASTER parameters.
> 
> That significantly decreases the usefulness of ALTER SYSTEM without
> actually preventing the underlying problem.  If having multiple
> conflicting values for parameters in the config files doesn't work
> cleanly, then we should fix that, not cripple this feature.

I think the only solution that makes sense here is to comment out the
entry in postgresql.conf.  We weren't supposed to modify that file, but
in light of this problem I'm starting to be of the opinion that we ought
to allow this implementation do this one change.  I admit this is
somewhat problematic: consider that somebody having postgresql.conf open
in an editor while somebody else runs ALTER SYSTEM would see the file
changing underneath.  But at least in Vim, what happens is that the
editor warns you that the file has changed underneath, and asks you
whether you want to load the new version.

Now, this is not likely to be an easy thing to implement ...

Another option would be to separate config loading in two stages; first
read all the files, and only check and throw errors and warnings later,
when we know the definite place that's going to be the authoritative
definition of each variable.

I'm not comfortable with the idea that PGC_POSTMASTER variables should
be treated differently from others.

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Tom Lane <tgl@sss.pgh.pa.us> — 2013-07-25T20:58:55Z

Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> Robert Haas escribi:
>> On Mon, Jul 22, 2013 at 7:56 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>> I'd be inclined to think that ALTER SYSTEM SET should not be allowed to
>>> modify any PGC_POSTMASTER parameters.

>> That significantly decreases the usefulness of ALTER SYSTEM without
>> actually preventing the underlying problem.  If having multiple
>> conflicting values for parameters in the config files doesn't work
>> cleanly, then we should fix that, not cripple this feature.

> I think the only solution that makes sense here is to comment out the
> entry in postgresql.conf.

Actually, this is much ado about nothing.  If you actually put two
conflicting values for shared_buffers into postgresql.conf, you'll
find out that:

1. on initial start, the server just silently adopts the later one.

2. if you SIGHUP, you get:

LOG:  autovacuum launcher started
LOG:  received SIGHUP, reloading configuration files
LOG:  parameter "shared_buffers" cannot be changed without restarting the server
LOG:  configuration file "/home/postgres/version93/data/postgresql.conf" contains errors; unaffected changes were applied

We could possibly improve the code to not produce that log bleat, but
it's hardly a show-stopper as is.

			regards, tom lane

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Tom Lane <tgl@sss.pgh.pa.us> — 2013-07-25T21:02:19Z

Robert Haas <robertmhaas@gmail.com> writes:
> On Thu, Jul 18, 2013 at 6:03 PM, Alvaro Herrera
> <alvherre@2ndquadrant.com> wrote:
>> Josh Berkus escribi:
>>> We are missing one feature, which is the ability to relocate the
>>> postgresql.auto.conf file if relocating it is desireable according to
>>> some sysadmin spec.

> My thought is that people might put postgresql.conf in a directory
> that only contains configuration files and isn't writeable by the
> postgres user.  So I would expect to find postgresql.auto.conf in the
> data directory always.

Yeah, exactly.  I think putting it anywhere but $PGDATA is a bad idea,
and a sysadmin who thinks he knows better probably doesn't.

			regards, tom lane

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Josh Berkus <josh@agliodbs.com> — 2013-07-26T00:52:37Z

On 07/25/2013 02:02 PM, Tom Lane wrote:
> Robert Haas <robertmhaas@gmail.com> writes:
>> On Thu, Jul 18, 2013 at 6:03 PM, Alvaro Herrera
>> <alvherre@2ndquadrant.com> wrote:
>>> Josh Berkus escribi�:
>>>> We are missing one feature, which is the ability to relocate the
>>>> postgresql.auto.conf file if relocating it is desireable according to
>>>> some sysadmin spec.
> 
>> My thought is that people might put postgresql.conf in a directory
>> that only contains configuration files and isn't writeable by the
>> postgres user.  So I would expect to find postgresql.auto.conf in the
>> data directory always.
> 
> Yeah, exactly.  I think putting it anywhere but $PGDATA is a bad idea,
> and a sysadmin who thinks he knows better probably doesn't.

Please see Greg Smith's fairly strong arguments for putting it in the
config/ directory.


-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Alvaro Herrera <alvherre@2ndquadrant.com> — 2013-07-26T05:04:51Z

Josh Berkus escribió:
> On 07/25/2013 02:02 PM, Tom Lane wrote:
> > Robert Haas <robertmhaas@gmail.com> writes:
> > 
> >> My thought is that people might put postgresql.conf in a directory
> >> that only contains configuration files and isn't writeable by the
> >> postgres user.  So I would expect to find postgresql.auto.conf in the
> >> data directory always.
> > 
> > Yeah, exactly.  I think putting it anywhere but $PGDATA is a bad idea,
> > and a sysadmin who thinks he knows better probably doesn't.
> 
> Please see Greg Smith's fairly strong arguments for putting it in the
> config/ directory.

As far as I see, there are two argument he makes:

1. We ought to have conf.d/ (not config/) so that tools have a way to
   deploy snippets (e.g. pgtune)

2. we ought to have all the config files together so that versioning
   tools (Puppet) can just say "keep all files within directory X
   versioned" and not have to care about specific file names, etc.

I can buy (1), because that's a pretty common design for daemons
nowadays.  But I think that's its own patch, and there's no reason that
this patch should be messing with this.  I don't care all that much
about (2), but I have no problem with doing that.

So we could have two patches, first one that introduces a conf.d subdir
that's automatically parsed after postgresql.conf, and another one that
implements ALTER SYSTEM by using a file within conf.d.  The reason I say
we need a separate patch for conf.d is that I think it'd be easier to
argue about it in isolation, than having it be entangled with ALTER
SYSTEM stuff.  The main contention point I see is where conf.d lives;
the two options are in $PGDATA or together with postgresql.conf.  Tom
and Robert, above, say it should be in $PGDATA; but this goes against
Debian packaging and the Linux FHS (or whatever that thing is called).

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila@huawei.com> — 2013-07-26T11:10:25Z

On Friday, July 26, 2013 10:35 AM Alvaro Herrera wrote:
> Josh Berkus escribió:
> > On 07/25/2013 02:02 PM, Tom Lane wrote:
> > > Robert Haas <robertmhaas@gmail.com> writes:
> > >
> > >> My thought is that people might put postgresql.conf in a directory
> > >> that only contains configuration files and isn't writeable by the
> > >> postgres user.  So I would expect to find postgresql.auto.conf in
> the
> > >> data directory always.
> > >
> > > Yeah, exactly.  I think putting it anywhere but $PGDATA is a bad
> idea,
> > > and a sysadmin who thinks he knows better probably doesn't.
> >
> > Please see Greg Smith's fairly strong arguments for putting it in the
> > config/ directory.
> 
> As far as I see, there are two argument he makes:
> 
> 1. We ought to have conf.d/ (not config/) so that tools have a way to
>    deploy snippets (e.g. pgtune)
> 
> 2. we ought to have all the config files together so that versioning
>    tools (Puppet) can just say "keep all files within directory X
>    versioned" and not have to care about specific file names, etc.
> 
> 
> I can buy (1), because that's a pretty common design for daemons
> nowadays.  But I think that's its own patch, and there's no reason that
> this patch should be messing with this.  I don't care all that much
> about (2), but I have no problem with doing that.
> 
> So we could have two patches, first one that introduces a conf.d subdir
> that's automatically parsed after postgresql.conf, and another one that
> implements ALTER SYSTEM by using a file within conf.d.  The reason I
> say
> we need a separate patch for conf.d is that I think it'd be easier to
> argue about it in isolation, than having it be entangled with ALTER
> SYSTEM stuff.  The main contention point I see is where conf.d lives;
> the two options are in $PGDATA or together with postgresql.conf.  Tom
> and Robert, above, say it should be in $PGDATA; but this goes against
> Debian packaging and the Linux FHS (or whatever that thing is called).

Sure, I think we can split into 2 patches, but I doubt it will make it any
easier
to get this feature completed. 
The contention point can delay the first patch (automatic parse of conf.d)
which can
lead to further delay of ALTER SYSTEM SET patch and that will eventually
lead to its death.
	
With Regards,
Amit Kapila.

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Tom Lane <tgl@sss.pgh.pa.us> — 2013-07-26T12:48:32Z

Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> The main contention point I see is where conf.d lives;
> the two options are in $PGDATA or together with postgresql.conf.  Tom
> and Robert, above, say it should be in $PGDATA; but this goes against
> Debian packaging and the Linux FHS (or whatever that thing is called).

Ordinarily, if postgresql.conf is not in $PGDATA, it will be somewhere
that the postmaster does not (and should not) have write permissions
for.  I have no objection to inventiny a conf.d subdirectory, I just say
that it must be under $PGDATA.  The argument that this is against FHS
is utter nonsense, because anything we write there is not static
configuration, it's just data.

Come to think of it, maybe part of the reason we're having such a hard
time getting to consensus is that people are conflating the "snippet"
part with the "writable" part?  I mean, if you are thinking you want
system-management tools to be able to drop in configuration fragments as
separate files, there's a case to be made for a conf.d subdirectory that
lives somewhere that the postmaster can't necessarily write.  We just
mustn't confuse that with support for ALTER SYSTEM SET.  I strongly
believe that ALTER SYSTEM SET must not be designed to write anywhere
outside $PGDATA.

			regards, tom lane

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-07-26T19:19:02Z

* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> Come to think of it, maybe part of the reason we're having such a hard
> time getting to consensus is that people are conflating the "snippet"
> part with the "writable" part?  I mean, if you are thinking you want
> system-management tools to be able to drop in configuration fragments as
> separate files, there's a case to be made for a conf.d subdirectory that
> lives somewhere that the postmaster can't necessarily write.  We just
> mustn't confuse that with support for ALTER SYSTEM SET.  I strongly
> believe that ALTER SYSTEM SET must not be designed to write anywhere
> outside $PGDATA.

Agreed.  To continue that thought, I find it *very* unlikely that a
given environment would use *both* a tool like puppet to manage the
files in their conf.d *and* have people using ALTER SYSTEM SET.  You're
going to do one or the other, almost certainly; not the least of which
is because those are very likely two different teams and only one of
them is going to be responsible for the PG system config.

	Thanks,

		Stephen

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Dimitri Fontaine <dimitri@2ndquadrant.fr> — 2013-07-27T18:24:45Z

Tom Lane <tgl@sss.pgh.pa.us> writes:
> Come to think of it, maybe part of the reason we're having such a hard
> time getting to consensus is that people are conflating the "snippet"
> part with the "writable" part?  I mean, if you are thinking you want
> system-management tools to be able to drop in configuration fragments as
> separate files, there's a case to be made for a conf.d subdirectory that
> lives somewhere that the postmaster can't necessarily write.  We just
> mustn't confuse that with support for ALTER SYSTEM SET.  I strongly
> believe that ALTER SYSTEM SET must not be designed to write anywhere
> outside $PGDATA.

Agreed, +1.

I think we should have conf.d in the same place as postgresql.conf,
wherever that is for your distribution of choice (e.g. /etc), and
certainly ALTER SYSTEM SET should stay away from those places.

Come to think of it, the one-file-per-guc approach that received some
push back when targeting the conf.d place might well be acceptable when
targeting some place in $PGDATA, right? I still believe that's the best
technical way to have at it.

Regards,
-- 
Dimitri Fontaine
http://2ndQuadrant.fr     PostgreSQL : Expertise, Formation et Support

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila@huawei.com> — 2013-07-28T05:41:44Z

On Friday, July 26, 2013 6:18 PM Tom Lane wrote:
Alvaro Herrera <alvherre@2ndquadrant.com> writes:
>> The main contention point I see is where conf.d lives;
>> the two options are in $PGDATA or together with postgresql.conf.  Tom
>> and Robert, above, say it should be in $PGDATA; but this goes against
>> Debian packaging and the Linux FHS (or whatever that thing is called).

> Ordinarily, if postgresql.conf is not in $PGDATA, it will be somewhere
> that the postmaster does not (and should not) have write permissions
> for.  I have no objection to inventiny a conf.d subdirectory, I just say
> that it must be under $PGDATA.  The argument that this is against FHS
> is utter nonsense, because anything we write there is not static
> configuration, it's just data.

> Come to think of it, maybe part of the reason we're having such a hard
> time getting to consensus is that people are conflating the "snippet"
> part with the "writable" part?  I mean, if you are thinking you want
> system-management tools to be able to drop in configuration fragments as
> separate files, there's a case to be made for a conf.d subdirectory that
> lives somewhere that the postmaster can't necessarily write.  We just
> mustn't confuse that with support for ALTER SYSTEM SET.  I strongly
> believe that ALTER SYSTEM SET must not be designed to write anywhere
> outside $PGDATA.

I think if we can design conf.d separately for config files of management tools, then
it is better to have postgresql.auto.conf to be in $PGDATA rather than in 
$PGDATA/conf.d

Kindly let me know if you feel otherwise, else I will update and send patch
tomorrow.


With Regards,
Amit Kapila.

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Josh Berkus <josh@agliodbs.com> — 2013-07-29T01:34:06Z

> I think if we can design conf.d separately for config files of management tools, then
> it is better to have postgresql.auto.conf to be in $PGDATA rather than in 
> $PGDATA/conf.d

One of the biggest current complaints about recovery.conf from
Debian/Ubuntu users is the fact that it lives in $PGDATA.  Won't we just
get the same thing here?

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Tom Lane <tgl@sss.pgh.pa.us> — 2013-07-29T02:49:04Z

Josh Berkus <josh@agliodbs.com> writes:
>> I think if we can design conf.d separately for config files of management tools, then
>> it is better to have postgresql.auto.conf to be in $PGDATA rather than in 
>> $PGDATA/conf.d

> One of the biggest current complaints about recovery.conf from
> Debian/Ubuntu users is the fact that it lives in $PGDATA.  Won't we just
> get the same thing here?

I don't think that's the same case, but ... why exactly don't they like
recovery.conf, and can you show that the location of the file has
anything to do with the underlying complaint?  Personally I bet it's
more about the confusion between configuration and triggering functions.
Until we can get those things separated, using recovery.conf to argue
about file locations will result in nothing but confusion and bad
design.

			regards, tom lane

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Dimitri Fontaine <dimitri@2ndquadrant.fr> — 2013-07-29T05:47:54Z

Tom Lane <tgl@sss.pgh.pa.us> writes:
>> One of the biggest current complaints about recovery.conf from
>> Debian/Ubuntu users is the fact that it lives in $PGDATA.  Won't we just
>> get the same thing here?

I don't think so, see below.

> I don't think that's the same case, but ... why exactly don't they like
> recovery.conf, and can you show that the location of the file has
> anything to do with the underlying complaint?  Personally I bet it's
> more about the confusion between configuration and triggering functions.
> Until we can get those things separated, using recovery.conf to argue
> about file locations will result in nothing but confusion and bad
> design.

I think it's all about having a user-edited configuration file somewhere
else than /etc, and that keeping the triggering functions separated from
the setup itself would be a good thing to have.

I bet that if some facts and links are necessary they will appear soon,
now is not a right time for me to be searching for those facts…

Regards,
-- 
Dimitri Fontaine
http://2ndQuadrant.fr     PostgreSQL : Expertise, Formation et Support

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila@huawei.com> — 2013-07-29T11:47:57Z

On Sunday, July 28, 2013 11:12 AM Amit kapila wrote:
> On Friday, July 26, 2013 6:18 PM Tom Lane wrote:
> Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> >> The main contention point I see is where conf.d lives;
> >> the two options are in $PGDATA or together with postgresql.conf.
> Tom
> >> and Robert, above, say it should be in $PGDATA; but this goes
> against
> >> Debian packaging and the Linux FHS (or whatever that thing is
> called).
> 
> > Ordinarily, if postgresql.conf is not in $PGDATA, it will be
> somewhere
> > that the postmaster does not (and should not) have write permissions
> > for.  I have no objection to inventiny a conf.d subdirectory, I just
> say
> > that it must be under $PGDATA.  The argument that this is against FHS
> > is utter nonsense, because anything we write there is not static
> > configuration, it's just data.
> 
> > Come to think of it, maybe part of the reason we're having such a
> hard
> > time getting to consensus is that people are conflating the "snippet"
> > part with the "writable" part?  I mean, if you are thinking you want
> > system-management tools to be able to drop in configuration fragments
> as
> > separate files, there's a case to be made for a conf.d subdirectory
> that
> > lives somewhere that the postmaster can't necessarily write.  We just
> > mustn't confuse that with support for ALTER SYSTEM SET.  I strongly
> > believe that ALTER SYSTEM SET must not be designed to write anywhere
> > outside $PGDATA.
> 
> I think if we can design conf.d separately for config files of
> management tools, then
> it is better to have postgresql.auto.conf to be in $PGDATA rather than
> in
> $PGDATA/conf.d
> 
> Kindly let me know if you feel otherwise, else I will update and send
> patch
> tomorrow.

Modified patch to have postgresql.auto.conf in $PGDATA. Changes are as
below:

1. initdb to create auto file in $PGDATA
2. ProcessConfigFile to open auto file from data directory, special case
handling for initdb
3. AlterSystemSetConfigFile function to consider data directory as reference
for operating on auto file
4. modified comments in code and docs to remove usage of config directory
5. modified function write_auto_conf_file() such that even if there is no
configuration item to write, it should write header message.
   This is to handle case when there is only one parameter value and user
set it to default, before this modification ,it 
   will write empty file.

With Regards,
Amit Kapila.

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Cédric Villemain <cedric@2ndquadrant.com> — 2013-07-29T13:45:26Z

Le lundi 29 juillet 2013 13:47:57, Amit Kapila a écrit :
> On Sunday, July 28, 2013 11:12 AM Amit kapila wrote:
> > On Friday, July 26, 2013 6:18 PM Tom Lane wrote:
> > 
> > Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> > >> The main contention point I see is where conf.d lives;
> > >> the two options are in $PGDATA or together with postgresql.conf.
> > 
> > Tom
> > 
> > >> and Robert, above, say it should be in $PGDATA; but this goes
> > 
> > against
> > 
> > >> Debian packaging and the Linux FHS (or whatever that thing is
> > 
> > called).
> > 
> > > Ordinarily, if postgresql.conf is not in $PGDATA, it will be
> > 
> > somewhere
> > 
> > > that the postmaster does not (and should not) have write permissions
> > > for.  I have no objection to inventiny a conf.d subdirectory, I just
> > 
> > say
> > 
> > > that it must be under $PGDATA.  The argument that this is against FHS
> > > is utter nonsense, because anything we write there is not static
> > > configuration, it's just data.
> > > 
> > > Come to think of it, maybe part of the reason we're having such a
> > 
> > hard
> > 
> > > time getting to consensus is that people are conflating the "snippet"
> > > part with the "writable" part?  I mean, if you are thinking you want
> > > system-management tools to be able to drop in configuration fragments
> > 
> > as
> > 
> > > separate files, there's a case to be made for a conf.d subdirectory
> > 
> > that
> > 
> > > lives somewhere that the postmaster can't necessarily write.  We just
> > > mustn't confuse that with support for ALTER SYSTEM SET.  I strongly
> > > believe that ALTER SYSTEM SET must not be designed to write anywhere
> > > outside $PGDATA.
> > 
> > I think if we can design conf.d separately for config files of
> > management tools, then
> > it is better to have postgresql.auto.conf to be in $PGDATA rather than
> > in
> > $PGDATA/conf.d
> > 
> > Kindly let me know if you feel otherwise, else I will update and send
> > patch
> > tomorrow.
> 
> Modified patch to have postgresql.auto.conf in $PGDATA. Changes are as
> below:
> 
> 1. initdb to create auto file in $PGDATA
> 2. ProcessConfigFile to open auto file from data directory, special case
> handling for initdb
> 3. AlterSystemSetConfigFile function to consider data directory as
> reference for operating on auto file
> 4. modified comments in code and docs to remove usage of config directory
> 5. modified function write_auto_conf_file() such that even if there is no
> configuration item to write, it should write header message.
>    This is to handle case when there is only one parameter value and user
> set it to default, before this modification ,it
>    will write empty file.

I just read the patch, quickly.
You may split the patch thanks to validate_conf_option(), however it is not a 
rule on postgresql-hacker.

Why not harcode in ParseConfigFp() that we should parse the auto.conf file at 
the end  (and/or if USE_AUTO_CONF is not OFF)  instead of hacking 
ProcessConfigFile() with data_directory ? (data_directory should be set at this 
point) ... just thinking, a very convenient way to enable/disable that is just 
to add/remove the include directive in postgresql.conf. So no change should be 
required in ParseConf at all. Except maybe AbsoluteConfigLocation which should 
prefix the path to auto.conf.d with data_directory. What I like with the 
include directive is that Sysadmin can define some GUC *after* the auto.conf so 
he is sure those are not 'erased' by auto.conf (or by the DBA).

Also, it looks very interesting to stick to an one-file-for-many-GUC when we 
absolutely don't care : this file should (MUST ?) not be edited by hand.
The thing achieve is that it limits the access to ALTER SYSTEM. One file per 
GUC allows to LWlock only this GUC, isn't it ? (and also does not require 
machinery for holding old/new auto GUC, or at least more simple).

It also prevent usage of ALTER SYSTEM for a cluster (as in replication) 
because this is not WAL logged. But it can be easier if trying to manage only 
one GUC at a time.

I agree with Tom comment that this file(s) must be in data_directory. 
postgresql.auto.conf is useless, a "data_directory/auto.conf" (.d/ ?) is 
enough.

-- 
Cédric Villemain +33 (0)6 20 30 22 52
http://2ndQuadrant.fr/
PostgreSQL: Support 24x7 - Développement, Expertise et Formation

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Alvaro Herrera <alvherre@2ndquadrant.com> — 2013-07-29T16:03:21Z

> Why not harcode in ParseConfigFp() that we should parse the auto.conf file at 
> the end  (and/or if USE_AUTO_CONF is not OFF)  instead of hacking 
> ProcessConfigFile() with data_directory ? (data_directory should be set at this 
> point) ... just thinking, a very convenient way to enable/disable that is just 
> to add/remove the include directive in postgresql.conf. So no change should be 
> required in ParseConf at all. Except maybe AbsoluteConfigLocation which should 
> prefix the path to auto.conf.d with data_directory. What I like with the 
> include directive is that Sysadmin can define some GUC *after* the auto.conf so 
> he is sure those are not 'erased' by auto.conf (or by the DBA).

Why do you think DBAs would like an option to disable this feature?  I
see no point in that.  And being able to relocate the parsing of
auto.conf to be in the middle of postgresql.conf instead of at the end
... that seems nightmarish.  I mean, things are *already* nontrivial to
follow, I don't see what would can come from a DBA running ALTER SYSTEM
and wondering why their changes don't take.

> Also, it looks very interesting to stick to an one-file-for-many-GUC when we 
> absolutely don't care : this file should (MUST ?) not be edited by hand.
> The thing achieve is that it limits the access to ALTER SYSTEM. One file per 
> GUC allows to LWlock only this GUC, isn't it ? (and also does not require 
> machinery for holding old/new auto GUC, or at least more simple).

This has already been debated, and we have already reached consensus
(one file to rule them all).  I don't think it's a good idea to go over
all that discussion again.

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Josh Berkus <josh@agliodbs.com> — 2013-07-29T19:46:39Z

All,

Based on the ongoing discussion of this patch, I have moved it to 9.4CF2
(9-2013).

Mind you, it would be good to commit some version of it before September.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Euler Taveira <euler@timbira.com.br> — 2013-07-29T21:03:02Z

On 18-07-2013 17:02, Alvaro Herrera wrote:
> I think we should just put the config directives of ALTER SYSTEM into a
> file, not dir, alongside postgresql.conf; I would suggest
> postgresql.auto.conf.  This file is parsed automatically after
> postgresql.conf, without requiring an "include" directive in
> postgresql.conf.  If the user does not want that file, they can remove
> it; but a future ALTER SYSTEM will create the file again.  No need to
> parse stuff to find out whether the directory exists, or the file
> exists, or such nonsense.
> 
+1. Let's keep it as simple as possible. There will be the possibility
to revisit the directory and/or include* ideas.

> I think the only drawback of this is that there's no way to disable
> ALTER SYSTEM (i.e. there's no directory you can remove to prevent the
> thing from working).
> 
Disable ALTER SYSTEM? No, thanks. Change the postgresql.auto.conf
permissions or ownership is an elegant way to disable it.


-- 
   Euler Taveira                   Timbira - http://www.timbira.com.br/
   PostgreSQL: Consultoria, Desenvolvimento, Suporte 24x7 e Treinamento

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila@huawei.com> — 2013-07-30T03:27:12Z

On Monday, July 29, 2013 7:15 PM Cédric Villemain wrote:
> Le lundi 29 juillet 2013 13:47:57, Amit Kapila a écrit :
> > On Sunday, July 28, 2013 11:12 AM Amit kapila wrote:
> > > On Friday, July 26, 2013 6:18 PM Tom Lane wrote:
> > >
> > > Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> > > >> The main contention point I see is where conf.d lives; the two
> > > >> options are in $PGDATA or together with postgresql.conf.
> > >
> > > Tom
> > >
> > > >> and Robert, above, say it should be in $PGDATA; but this goes
> > >
> > > against
> > >
> > > >> Debian packaging and the Linux FHS (or whatever that thing is
> > >
> > > called).
> > >
> > > > Ordinarily, if postgresql.conf is not in $PGDATA, it will be
> > >
> > > somewhere
> > >
> > > > that the postmaster does not (and should not) have write
> > > > permissions for.  I have no objection to inventiny a conf.d
> > > > subdirectory, I just
> > >
> > > say
> > >
> > > > that it must be under $PGDATA.  The argument that this is against
> > > > FHS is utter nonsense, because anything we write there is not
> > > > static configuration, it's just data.
> > > >
> > > > Come to think of it, maybe part of the reason we're having such a
> > >
> > > hard
> > >
> > > > time getting to consensus is that people are conflating the
> "snippet"
> > > > part with the "writable" part?  I mean, if you are thinking you
> > > > want system-management tools to be able to drop in configuration
> > > > fragments
> > >
> > > as
> > >
> > > > separate files, there's a case to be made for a conf.d
> > > > subdirectory
> > >
> > > that
> > >
> > > > lives somewhere that the postmaster can't necessarily write.  We
> > > > just mustn't confuse that with support for ALTER SYSTEM SET.  I
> > > > strongly believe that ALTER SYSTEM SET must not be designed to
> > > > write anywhere outside $PGDATA.
> > >
> > > I think if we can design conf.d separately for config files of
> > > management tools, then it is better to have postgresql.auto.conf to
> > > be in $PGDATA rather than in $PGDATA/conf.d
> > >
> > > Kindly let me know if you feel otherwise, else I will update and
> > > send patch tomorrow.
> >
> > Modified patch to have postgresql.auto.conf in $PGDATA. Changes are
> as
> > below:
> >
> > 1. initdb to create auto file in $PGDATA 2. ProcessConfigFile to open
> > auto file from data directory, special case handling for initdb 3.
> > AlterSystemSetConfigFile function to consider data directory as
> > reference for operating on auto file 4. modified comments in code and
> > docs to remove usage of config directory 5. modified function
> > write_auto_conf_file() such that even if there is no configuration
> > item to write, it should write header message.
> >    This is to handle case when there is only one parameter value and
> > user set it to default, before this modification ,it
> >    will write empty file.
> 
> I just read the patch, quickly.


  Thank you for review.

> You may split the patch thanks to validate_conf_option(), however it is
> not a rule on postgresql-hacker.
  
  The review of the core functionality of patch has been done before the
introduction of function
  validate_conf_option() in the patch. It was introduced because there 
  were some common parts in core implementation of AlterSystem and
set_config_option().
  I am really not sure, after having multiple round of reviews by reviewers,
it can add
  significant value to split it.

> Why not harcode in ParseConfigFp() that we should parse the auto.conf
> file at the end  (and/or if USE_AUTO_CONF is not OFF)  instead of
> hacking
> ProcessConfigFile() with data_directory ? (data_directory should be set
> at this
> point) ... 

  No data_directory will not be set by that time incase of initdb, when
ProcessConfigFile()
  is called from SelectConfigFiles()

> just thinking, a very convenient way to enable/disable that
> is just to add/remove the include directive in postgresql.conf. So no
> change should be required in ParseConf at all. Except maybe
> AbsoluteConfigLocation which should prefix the path to auto.conf.d with
> data_directory. What I like with the include directive is that Sysadmin
> can define some GUC *after* the auto.conf so he is sure those are not
> 'erased' by auto.conf (or by the DBA).

I think earlier versions have this implementation, but later based on
suggestions, I have changed it to automatic parsing of auto file after
postgresql.conf


> Also, it looks very interesting to stick to an one-file-for-many-GUC
> when we absolutely don't care : this file should (MUST ?) not be edited
> by hand.
> The thing achieve is that it limits the access to ALTER SYSTEM. One
> file per GUC allows to LWlock only this GUC, isn't it ? (and also does
> not require machinery for holding old/new auto GUC, or at least more
> simple).
> 
> It also prevent usage of ALTER SYSTEM for a cluster (as in replication)
> because this is not WAL logged. But it can be easier if trying to
> manage only one GUC at a time.
> 
> I agree with Tom comment that this file(s) must be in data_directory.
> postgresql.auto.conf is useless, a "data_directory/auto.conf" (.d/ ?)
> is enough.

There were multiple suggestions for names, but I have kept name
postgresql.auto.conf based 
on more votes for it and consensus.


With Regards,
Amit Kapila.

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila@huawei.com> — 2013-07-30T04:13:51Z

On Tuesday, July 30, 2013 1:17 AM Josh Berkus wrote:
On Tuesday, July 30, 2013 1:17 AM Amit kapila wrote:
> All,
> 
> Based on the ongoing discussion of this patch, I have moved it to
> 9.4CF2
> (9-2013).

I had already sent the updated patch based on recent suggestions.

I am really not sure if something is pending from myside, 
so bit confused by the status of patch as Waiting On Author?

> Mind you, it would be good to commit some version of it before
> September.

True, I am also hoping for the same.

With Regards,
Amit Kapila.

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Cédric Villemain <cedric@2ndquadrant.com> — 2013-07-30T07:39:05Z

Le lundi 29 juillet 2013 18:03:21, Alvaro Herrera a écrit :
> > Why not harcode in ParseConfigFp() that we should parse the auto.conf
> > file at the end  (and/or if USE_AUTO_CONF is not OFF)  instead of
> > hacking ProcessConfigFile() with data_directory ? (data_directory should
> > be set at this point) ... just thinking, a very convenient way to
> > enable/disable that is just to add/remove the include directive in
> > postgresql.conf. So no change should be required in ParseConf at all.
> > Except maybe AbsoluteConfigLocation which should prefix the path to
> > auto.conf.d with data_directory. What I like with the include directive
> > is that Sysadmin can define some GUC *after* the auto.conf so he is sure
> > those are not 'erased' by auto.conf (or by the DBA).
> 
> Why do you think DBAs would like an option to disable this feature?  I
> see no point in that.  And being able to relocate the parsing of
> auto.conf to be in the middle of postgresql.conf instead of at the end
> ... that seems nightmarish.  I mean, things are *already* nontrivial to
> follow, I don't see what would can come from a DBA running ALTER SYSTEM
> and wondering why their changes don't take.

I don't find that hard to do nor to understand, but if that has already reach a 
consensus then let's do that.

> > Also, it looks very interesting to stick to an one-file-for-many-GUC when
> > we absolutely don't care : this file should (MUST ?) not be edited by
> > hand. The thing achieve is that it limits the access to ALTER SYSTEM.
> > One file per GUC allows to LWlock only this GUC, isn't it ? (and also
> > does not require machinery for holding old/new auto GUC, or at least
> > more simple).
> 
> This has already been debated, and we have already reached consensus
> (one file to rule them all).  I don't think it's a good idea to go over
> all that discussion again.

ok, I've only lost track for the consensus based on the technical objective.

-- 
Cédric Villemain +33 (0)6 20 30 22 52
http://2ndQuadrant.fr/
PostgreSQL: Support 24x7 - Développement, Expertise et Formation

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Cédric Villemain <cedric@2ndquadrant.com> — 2013-07-30T07:45:47Z

Le mardi 30 juillet 2013 05:27:12, Amit Kapila a écrit :
> On Monday, July 29, 2013 7:15 PM Cédric Villemain wrote:
> > Le lundi 29 juillet 2013 13:47:57, Amit Kapila a écrit :
> > > On Sunday, July 28, 2013 11:12 AM Amit kapila wrote:
> > > > On Friday, July 26, 2013 6:18 PM Tom Lane wrote:
> > > > 
> > > > Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> > > > >> The main contention point I see is where conf.d lives; the two
> > > > >> options are in $PGDATA or together with postgresql.conf.
> > > > 
> > > > Tom
> > > > 
> > > > >> and Robert, above, say it should be in $PGDATA; but this goes
> > > > 
> > > > against
> > > > 
> > > > >> Debian packaging and the Linux FHS (or whatever that thing is
> > > > 
> > > > called).
> > > > 
> > > > > Ordinarily, if postgresql.conf is not in $PGDATA, it will be
> > > > 
> > > > somewhere
> > > > 
> > > > > that the postmaster does not (and should not) have write
> > > > > permissions for.  I have no objection to inventiny a conf.d
> > > > > subdirectory, I just
> > > > 
> > > > say
> > > > 
> > > > > that it must be under $PGDATA.  The argument that this is against
> > > > > FHS is utter nonsense, because anything we write there is not
> > > > > static configuration, it's just data.
> > > > > 
> > > > > Come to think of it, maybe part of the reason we're having such a
> > > > 
> > > > hard
> > > > 
> > > > > time getting to consensus is that people are conflating the
> > 
> > "snippet"
> > 
> > > > > part with the "writable" part?  I mean, if you are thinking you
> > > > > want system-management tools to be able to drop in configuration
> > > > > fragments
> > > > 
> > > > as
> > > > 
> > > > > separate files, there's a case to be made for a conf.d
> > > > > subdirectory
> > > > 
> > > > that
> > > > 
> > > > > lives somewhere that the postmaster can't necessarily write.  We
> > > > > just mustn't confuse that with support for ALTER SYSTEM SET.  I
> > > > > strongly believe that ALTER SYSTEM SET must not be designed to
> > > > > write anywhere outside $PGDATA.
> > > > 
> > > > I think if we can design conf.d separately for config files of
> > > > management tools, then it is better to have postgresql.auto.conf to
> > > > be in $PGDATA rather than in $PGDATA/conf.d
> > > > 
> > > > Kindly let me know if you feel otherwise, else I will update and
> > > > send patch tomorrow.
> > > 
> > > Modified patch to have postgresql.auto.conf in $PGDATA. Changes are
> > 
> > as
> > 
> > > below:
> > > 
> > > 1. initdb to create auto file in $PGDATA 2. ProcessConfigFile to open
> > > auto file from data directory, special case handling for initdb 3.
> > > AlterSystemSetConfigFile function to consider data directory as
> > > reference for operating on auto file 4. modified comments in code and
> > > docs to remove usage of config directory 5. modified function
> > > write_auto_conf_file() such that even if there is no configuration
> > > item to write, it should write header message.
> > > 
> > >    This is to handle case when there is only one parameter value and
> > > 
> > > user set it to default, before this modification ,it
> > > 
> > >    will write empty file.
> > 
> > I just read the patch, quickly.
> 
>   Thank you for review.
> 
> > You may split the patch thanks to validate_conf_option(), however it is
> > not a rule on postgresql-hacker.
> 
>   The review of the core functionality of patch has been done before the
> introduction of function
>   validate_conf_option() in the patch. It was introduced because there
>   were some common parts in core implementation of AlterSystem and
> set_config_option().
>   I am really not sure, after having multiple round of reviews by
> reviewers, it can add
>   significant value to split it.

Yes, it just appeared that this part was a significant one in the patch. I 
understand that it is not interesting to split now.

> 
> > Why not harcode in ParseConfigFp() that we should parse the auto.conf
> > file at the end  (and/or if USE_AUTO_CONF is not OFF)  instead of
> > hacking
> > ProcessConfigFile() with data_directory ? (data_directory should be set
> > at this
> > point) ...
> 
>   No data_directory will not be set by that time incase of initdb, when
> ProcessConfigFile()
>   is called from SelectConfigFiles()

ok.

> > just thinking, a very convenient way to enable/disable that
> > is just to add/remove the include directive in postgresql.conf. So no
> > change should be required in ParseConf at all. Except maybe
> > AbsoluteConfigLocation which should prefix the path to auto.conf.d with
> > data_directory. What I like with the include directive is that Sysadmin
> > can define some GUC *after* the auto.conf so he is sure those are not
> > 'erased' by auto.conf (or by the DBA).
> 
> I think earlier versions have this implementation, but later based on
> suggestions, I have changed it to automatic parsing of auto file after
> postgresql.conf

I probably missed those suggestions.

> > Also, it looks very interesting to stick to an one-file-for-many-GUC
> > when we absolutely don't care : this file should (MUST ?) not be edited
> > by hand.
> > The thing achieve is that it limits the access to ALTER SYSTEM. One
> > file per GUC allows to LWlock only this GUC, isn't it ? (and also does
> > not require machinery for holding old/new auto GUC, or at least more
> > simple).
> > 
> > It also prevent usage of ALTER SYSTEM for a cluster (as in replication)
> > because this is not WAL logged. But it can be easier if trying to
> > manage only one GUC at a time.
> > 
> > I agree with Tom comment that this file(s) must be in data_directory.
> > postgresql.auto.conf is useless, a "data_directory/auto.conf" (.d/ ?)
> > is enough.
> 
> There were multiple suggestions for names, but I have kept name
> postgresql.auto.conf based
> on more votes for it and consensus.

yeah, I vote here for a shorter name, but it changes nothing to the patch.
-- 
Cédric Villemain +33 (0)6 20 30 22 52
http://2ndQuadrant.fr/
PostgreSQL: Support 24x7 - Développement, Expertise et Formation

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Greg Smith <greg@2ndquadrant.com> — 2013-07-30T12:22:31Z

On 7/28/13 10:49 PM, Tom Lane wrote:
> Josh Berkus <josh@agliodbs.com> writes:
>>> I think if we can design conf.d separately for config files of management tools, then
>>> it is better to have postgresql.auto.conf to be in $PGDATA rather than in
>>> $PGDATA/conf.d
>
>> One of the biggest current complaints about recovery.conf from
>> Debian/Ubuntu users is the fact that it lives in $PGDATA.  Won't we just
>> get the same thing here?
>
> I don't think that's the same case, but ... why exactly don't they like
> recovery.conf, and can you show that the location of the file has
> anything to do with the underlying complaint?

Here is a quick survey of config files:

RHEL6:
Cluster configuration file at /etc/sysconfig/pgsql, writeable by root. 
postgresql.conf is in $HOME/8.4/data

Debian Squeeze:
All configuration files in /etc/postgresql/8.4/main and are owned by the 
postgres user.

That last one shows why there's a complaint, and why it's 75% due to 
location rather than split role.  Debian users expect all of the config 
files to be relocated to /etc/postgresql or /etc/postgresql-common. 
Those files are also writeable by the postgres user.  recovery.conf 
doesn't fit that model, and people don't like that.  The admin who has 
to manage servers with puppet would be much happier if recovery.conf was 
in the /etc/postgresql tree they are already managing.  I get this 
complaint every single time I introduce an experienced Debian admin to 
PostgreSQL replication setup.  They aren't sure whether it's the 
packagers or PostgreSQL that is to blame for recovery.conf being in 
$PGDATA instead of /etc, but it's obvious to them someone has screwed up.

Given that Debian is the major example of a relocated postgresql.conf in 
the field, and that work has been done such that a) all config files are 
in /etc and b) they are all writeable by the postgres user, I don't 
understand why people are suggesting the auto.conf file or conf.d 
directory will go anywhere else on that platform anymore.  I think the 
only claims that are suggesting otherwise are thinking about an older, 
now unsupported version of the Debian packaging.  Debian packaging 
absolutely will want to relocate any new files added here into their 
/etc directory tree as well, where they will be writeable.  They will 
not go into the $PGDATA directory.

-- 
Greg Smith   2ndQuadrant US    greg@2ndQuadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Josh Berkus <josh@agliodbs.com> — 2013-07-30T16:55:28Z

Amit,

> I had already sent the updated patch based on recent suggestions.

Yes, but there is no consensus yet on certain fundamental issues, such as:

1. what directory should postgresql.auto.conf live in?

2. should admins be able to turn it "off"?

Once we have consensus on these issues, then hopefully we can commit
your patch.  I don't *think* anyone is arguing with the code at this stage.

Note that I count the name of the file as something on which we *have*
consensus.  While I personally don't like the name, I like arguing about
names even less.  We had a vote, let's stick with what we have.

> I am really not sure if something is pending from myside, 
> so bit confused by the status of patch as Waiting On Author?

It's our policy that patches which are under spec discussion are marked
"waiting on author".

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Alvaro Herrera <alvherre@2ndquadrant.com> — 2013-07-30T17:10:30Z

Josh Berkus escribió:
> Amit,
> 
> > I had already sent the updated patch based on recent suggestions.
> 
> Yes, but there is no consensus yet on certain fundamental issues, such as:
> 
> 1. what directory should postgresql.auto.conf live in?
> 
> 2. should admins be able to turn it "off"?
> 
> Once we have consensus on these issues, then hopefully we can commit
> your patch.  I don't *think* anyone is arguing with the code at this stage.

There's one other thing, raised by Cédric upthread, about the multiple
files vs. single file.  We decided on a single file because Greg said
many little files scattered in conf.d/ would be a mess; but if we're
going to have two new items, one conf.d/ for snippets from external
tools (living in the config directory) and one for ALTER SYSTEM (living
in PGDATA), then the idea of going back to one setting per file is
perhaps not bad.

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Greg Stark <stark@mit.edu> — 2013-07-30T17:28:28Z

On Tue, Jul 30, 2013 at 6:10 PM, Alvaro Herrera
<alvherre@2ndquadrant.com> wrote:
> one for ALTER SYSTEM (living
> in PGDATA), then the idea of going back to one setting per file is
> perhaps not bad.

Well more to the point, if we conf.d for sysadmins to drop in extra
snippets in a different place then we could drop conf.d in PGDATA
since there's no need for it any more and just have auto.conf in
PGDATA directly.

-- 
greg

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Josh Berkus <josh@agliodbs.com> — 2013-07-30T17:40:55Z

On 07/30/2013 10:10 AM, Alvaro Herrera wrote:
>> Once we have consensus on these issues, then hopefully we can commit
>> your patch.  I don't *think* anyone is arguing with the code at this stage.
> 
> There's one other thing, raised by Cédric upthread, about the multiple
> files vs. single file.  We decided on a single file because Greg said
> many little files scattered in conf.d/ would be a mess; but if we're

That's not what I get out of the discussion thread.  I believed we
settled on One File because that's the patch Amit wrote, and a patch in
the hand is worth two in WIP.

> going to have two new items, one conf.d/ for snippets from external
> tools (living in the config directory) and one for ALTER SYSTEM (living
> in PGDATA), then the idea of going back to one setting per file is
> perhaps not bad.

On 07/30/2013 10:28 AM, Greg Stark wrote:> On Tue, Jul 30, 2013 at 6:10
PM, Alvaro Herrera
> Well more to the point, if we conf.d for sysadmins to drop in extra
> snippets in a different place then we could drop conf.d in PGDATA
> since there's no need for it any more and just have auto.conf in
> PGDATA directly.

That assumes that the only reason we have for a conf.d is to support
auto.conf, which I don't agree with; I personally have a need for conf.d
which has nothing to do with auto.conf.

Also, see gsmith's point about forcing auto.conf to be in PGDATA as not
satisfactory for Debian users (and others).

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Greg Stark <stark@mit.edu> — 2013-07-30T18:12:35Z

On Tue, Jul 30, 2013 at 6:40 PM, Josh Berkus <josh@agliodbs.com> wrote:
>
> On 07/30/2013 10:28 AM, Greg Stark wrote:> On Tue, Jul 30, 2013 at 6:10
> PM, Alvaro Herrera
>> Well more to the point, if we conf.d for sysadmins to drop in extra
>> snippets in a different place then we could drop conf.d in PGDATA
>> since there's no need for it any more and just have auto.conf in
>> PGDATA directly.
>
> That assumes that the only reason we have for a conf.d is to support
> auto.conf, which I don't agree with; I personally have a need for conf.d
> which has nothing to do with auto.conf.

No, wait. We had two reasons for conf.d and we thought we could kill
two birds with one stone. It seems it's turning out that we can't kill
both birds with the same stone because we want to separate the user
maintained config from the automatically maintained config even
further than just having them in separate files. Now we need them in
separate directories.

If you have a need for conf.d which has nothing to do with auto.conf
then you presumably will want to be editing the system conf.d which
might be in PGDATA or might be in /etc or might be someplace else. I
totally agree that conf.d is useful for sysadmins as well as for
distribution authors.

But if we're going to insist that conf.d be in PGDATA then I'm saying
we don't need a second conf.d just to contain that one file. And if we
let distributions and sysadmins drop files in there then we'll be back
to square 0 with wanting to separate them.

I guess what I'm saying is that auto.conf shouldn't be in conf.d after
all. It should just be in PGDATA directly and trying to kill two birds
with one stone led us astray. Putting it in conf.d creates a raft of
problems of what if conf.d is moved and what if it's shared between
databases and what if the user puts files that come after it, etc.
Putting it directly in PGDATA lets us manage those issues however we
want and ties the automatic file to a specific database regardless of
how we handle conf.d in the future.

> Also, see gsmith's point about forcing auto.conf to be in PGDATA as not
> satisfactory for Debian users (and others).

Greg Smith's argument was about recovery.conf which is a file that
users are expected to edit. A file which user's are not expected to
edit and is maintained by the software is no more a configuration file
than pg_auth or pg_database which are actually being stored in the
database itself.

Having an automatically maintained file and a manually maintained file
is going to raise some tricky problems. In Oracle they have exactly
the same issue. In that case the automatically maintained one is
actually kept in a binary format. You can dump it out in text format
but unless you switch which one you're using it doesn't read the text
format file at all. I assume they did this because working out the
conflicts between the two was just too complex for users.

I think we're fine with allowing users to use both but we should try
to keep the two as separate as possible to avoid confusion. Keeping
the auto.conf inside conf.d sounds like it will actually confuse users
about which files they're supposed to edit and which belong to the
system.

-- 
greg

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Josh Berkus <josh@agliodbs.com> — 2013-07-30T18:31:55Z

On 07/30/2013 11:12 AM, Greg Stark wrote:
> But if we're going to insist that conf.d be in PGDATA then I'm saying
> we don't need a second conf.d just to contain that one file. And if we
> let distributions and sysadmins drop files in there then we'll be back
> to square 0 with wanting to separate them.

Ah, misunderstood your point.  We certainly don't need *two* conf.d's.

> Greg Smith's argument was about recovery.conf which is a file that
> users are expected to edit. A file which user's are not expected to
> edit and is maintained by the software is no more a configuration file
> than pg_auth or pg_database which are actually being stored in the
> database itself.

I don't think you understood GSmith's argument.  For Debian/Ubuntu
sysadmins, configuration files live in /etc/, *period*.  Even ones which
were automatically generated.  The packagers and scripters of that OS
have taken a significant amount of trouble to make that so, including
writing their own wrapper scripts around utilities for Postgres, Apache,
etc., in order to support putting all configuration files in /etc/.

Now we're proposing to break that *again*, by putting yet another
configuration file in PGDATA, and making it impossible to relocate.
That's fairly hostile to the concerns of possibly our most popular OSes.

Saying "it's not a configuration file because it's automatic" is rather
disingenuous.  If you can set work_mem via postgresql.auto.conf, it's a
configuration file, regardless of the means you used to set it.

> Having an automatically maintained file and a manually maintained file
> is going to raise some tricky problems. In Oracle they have exactly
> the same issue. In that case the automatically maintained one is
> actually kept in a binary format. You can dump it out in text format
> but unless you switch which one you're using it doesn't read the text
> format file at all. I assume they did this because working out the
> conflicts between the two was just too complex for users.

I'm not sure we won't end up in the same place as Oracle, eventually.
We'll see.

Re: allowing users to disable ALTER SYSTEM SET, I think we'll have to
implement that before 9.4. comes out, for the simple reason that users
of Puppet and other large-scale configuration management utilities will
demand it.  If you're managing 120 PostgreSQL servers using centrally
version-controlled Chef recipes, the last thing in the world you want is
having your DBAs bypass that via ALTER SYSTEM SET.

HOWEVER, I think adding the ability to disable ALTER SYSTEM SET can (and
should) be a second, separate patch.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Greg Stark <stark@mit.edu> — 2013-07-30T19:50:49Z

On Jul 30, 2013 7:32 PM, "Josh Berkus" <josh@agliodbs.com> wrote:
>
>
> I don't think you understood GSmith's argument.  For Debian/Ubuntu
> sysadmins, configuration files live in /etc/, *period*.  Even ones which
> were automatically generated.  The packagers and scripters of that OS
> have taken a significant amount of trouble to make that so, including
> writing their own wrapper scripts around utilities for Postgres, Apache,
> etc., in order to support putting all configuration files in /etc/.
>

I used to be a Debian package maintainer. I packaged kerberos and afs and a
number of smaller packages. I know why Debian separates config files.

If a file is automatically maintained then it's internal program state just
like the list of users, databases, tables, etc.

Programs keep their state in /var. It would be just as wrong to put
auto.conf in /etc as it is to put the existing postgresql.conf in PGDATA.

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-07-30T23:10:24Z

* Greg Stark (stark@mit.edu) wrote:
> I used to be a Debian package maintainer. I packaged kerberos and afs and a
> number of smaller packages. I know why Debian separates config files.

Ditto (though I suppose I'm still technically "active").

> If a file is automatically maintained then it's internal program state just
> like the list of users, databases, tables, etc.

Right.

> Programs keep their state in /var. It would be just as wrong to put
> auto.conf in /etc as it is to put the existing postgresql.conf in PGDATA.

Agreed.

Though, I have to say that I think we need "pg_hba.d" and "pg_ident.d"
more than we need a conf.d for postgresql.conf snippets..  And ditto for
auto.conf-style managed-through-PG options for pg_hba and pg_ident.
Apologies for not having been able to keep up with this thread fully, so
feel free to point out if this has already been discussed. :)

	Thanks,

		Stephen

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila@huawei.com> — 2013-08-01T09:05:46Z

On Tuesday, July 30, 2013 10:25 PM Josh Berkus wrote:
> Amit,
> 
> > I had already sent the updated patch based on recent suggestions.
> 
> Yes, but there is no consensus yet on certain fundamental issues, such
> as:
> 
> 1. what directory should postgresql.auto.conf live in?

  Current situation is that Greg Smith has provided a usecase for all config files to be present
  in location other than $PGDATA, but based on further suggestions given by Greg Stark and Stephen Frost
  it is clear that keeping postgresql.auto.conf will not harm the users of Debian.

  More to that point, I think if we try to address conf.d (for maintaining config files) along with this 
  Patch, it will not be possible to get every one on same plane.

  I think apart from Greg Stark and Stephen Frost other people (Tom, Robert) are of opinion that auto file should be placed in
  $PGDATA, so shouldn't we consider this as consensus and move forward for this point unless Tom or Robert have changed their view based on
  usecase provided by Greg Smith?

> 2. should admins be able to turn it "off"?

     IIRC there are not many people who want this feature to be turned off. Peter E. had asked seeing the complexity, but in general I don't think
     This is a blocking point for patch, am I missing something here?

> Once we have consensus on these issues, then hopefully we can commit
> your patch.  I don't *think* anyone is arguing with the code at this
> stage.

Sure, I understand that it is important to have consensus and agreement on above points.

With Regards,
Amit Kapila.

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Dimitri Fontaine <dimitri@2ndquadrant.fr> — 2013-08-01T12:12:27Z

Greg Stark <stark@mit.edu> writes:
> Greg Smith's argument was about recovery.conf which is a file that
> users are expected to edit. A file which user's are not expected to
> edit and is maintained by the software is no more a configuration file
> than pg_auth or pg_database which are actually being stored in the
> database itself.

+1

> I think we're fine with allowing users to use both but we should try
> to keep the two as separate as possible to avoid confusion. Keeping
> the auto.conf inside conf.d sounds like it will actually confuse users
> about which files they're supposed to edit and which belong to the
> system.

+1

I think we need both an ALTER SYSTEM SET implementation using files
somewhere in $PGDATA, and a separate conf.d facility that lives
alongside wherever the main postgresql.conf is maintained on your OS of
choice.

Also, now that we have decided to separate away those two fundamentally
different aspects of the configuration setup, I join Álvaro and Cédric
in thinking that we should review the implementation choice of the ALTER
SYSTEM SET facility, and vote for having one-file-per-GUC.

Regards,
-- 
Dimitri Fontaine
http://2ndQuadrant.fr     PostgreSQL : Expertise, Formation et Support

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Greg Stark <stark@mit.edu> — 2013-08-01T13:32:24Z

On Thu, Aug 1, 2013 at 1:12 PM, Dimitri Fontaine <dimitri@2ndquadrant.fr> wrote:
> we should review the implementation choice of the ALTER
> SYSTEM SET facility, and vote for having one-file-per-GUC.

Zombie crazy design idea arise!

I think people are going to laugh at us if an open source database
software can't manage a simple flat file database of settings,
especially one that is purely write-only and can be a simple dump of
settings that are set by alter system.

-- 
greg

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Dimitri Fontaine <dimitri@2ndquadrant.fr> — 2013-08-01T13:55:25Z

Greg Stark <stark@mit.edu> writes:
> I think people are going to laugh at us if an open source database
> software can't manage a simple flat file database of settings,
> especially one that is purely write-only and can be a simple dump of
> settings that are set by alter system.

So you say it's easier to implement per-GUC locking semantics correctly
when using a single file with multiple units of information that all are
of the same type? Interesting.

Maybe the storage should actually be a shared catalog, in fact.

Regards,
-- 
Dimitri Fontaine
http://2ndQuadrant.fr     PostgreSQL : Expertise, Formation et Support

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Andres Freund <andres@2ndquadrant.com> — 2013-08-01T14:02:57Z

On 2013-08-01 15:55:25 +0200, Dimitri Fontaine wrote:
> Greg Stark <stark@mit.edu> writes:
> > I think people are going to laugh at us if an open source database
> > software can't manage a simple flat file database of settings,
> > especially one that is purely write-only and can be a simple dump of
> > settings that are set by alter system.

Why would using a single-file solution imply that we cannot do the
other?

> So you say it's easier to implement per-GUC locking semantics correctly
> when using a single file with multiple units of information that all are
> of the same type? Interesting.

> Maybe the storage should actually be a shared catalog, in fact.

We can't read those early enough. Think a) of options that can only be
set at postmaster start b) think of crash recovery. We can't read the
catalog till we're in a consistent state.

Greetings,

Andres Freund

-- 
 Andres Freund	                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Greg Stark <stark@mit.edu> — 2013-08-01T14:17:04Z

We don't need per guc locking. This is the whole objection Tom had about
this patch being more complex than it has to be.

-- 
greg
On 1 Aug 2013 14:55, "Dimitri Fontaine" <dimitri@2ndquadrant.fr> wrote:

> Greg Stark <stark@mit.edu> writes:
> > I think people are going to laugh at us if an open source database
> > software can't manage a simple flat file database of settings,
> > especially one that is purely write-only and can be a simple dump of
> > settings that are set by alter system.
>
> So you say it's easier to implement per-GUC locking semantics correctly
> when using a single file with multiple units of information that all are
> of the same type? Interesting.
>
> Maybe the storage should actually be a shared catalog, in fact.
>
> Regards,
> --
> Dimitri Fontaine
> http://2ndQuadrant.fr     PostgreSQL : Expertise, Formation et Support
>
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers
>

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Andres Freund <andres@2ndquadrant.com> — 2013-08-01T14:19:35Z

On 2013-08-01 15:17:04 +0100, Greg Stark wrote:
> We don't need per guc locking. This is the whole objection Tom had about
> this patch being more complex than it has to be.

IIRC he objected to using locking *at all* because a simple
one-file-per-setting approach should be used.

Greetings,

Andres Freund

-- 
 Andres Freund	                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Greg Stark <stark@mit.edu> — 2013-08-01T14:40:22Z

Why isn't it enough to just dump out all variables with a source of alter
system to a text file? You can either have a single global lock around that
operation or write it to a new file and move it into place.

-- 
greg
On 1 Aug 2013 15:19, "Andres Freund" <andres@2ndquadrant.com> wrote:

> On 2013-08-01 15:17:04 +0100, Greg Stark wrote:
> > We don't need per guc locking. This is the whole objection Tom had about
> > this patch being more complex than it has to be.
>
> IIRC he objected to using locking *at all* because a simple
> one-file-per-setting approach should be used.
>
> Greetings,
>
> Andres Freund
>
> --
>  Andres Freund                     http://www.2ndQuadrant.com/
>  PostgreSQL Development, 24x7 Support, Training & Services
>

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

David Johnston <polobo@yahoo.com> — 2013-08-01T14:47:32Z

Minor request: could someone enlighten me as to why making the directory
location a compile-time option is undesirable. Packagers then can setup
whatever structure they desire when they compile their distributions. In
which case the discussion becomes what is a reasonable default and that can
be made with respect of other defaults that are in place for people that
would self-compile.

I get the "supporting users - telling them where to go to find these files"
aspect but I believe that ship has already sailed. The goal should be to
make it as easy as possible to allow distributions and/or individual users
to integrate PostgreSQL into their normal routine as possible. It isn't
like we are adding unneeded complexity since it is obvious from the
discussion that where files/directories are placed in the file system is a
major variable. Enforcing $PGDATA when we know Debian is going to be upset
doesn't seem to be that great an idea - it isn't like we are going to
suddenly make them realize they have been doing things incorrectly all this
time. I am not familiar with all of the configurations but I do recall that
the location of postgres.conf and related files is already distribution
specific so why shouldn't these extensions be as well?

Sorry if this was discussed previously; I'll go look deeper in the thread if
someone confirms that indeed it is.

David J.

--
View this message in context: http://postgresql.1045698.n5.nabble.com/Proposal-for-Allow-postgresql-conf-values-to-be-changed-via-SQL-tp5729917p5765892.html
Sent from the PostgreSQL - hackers mailing list archive at Nabble.com.

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Andres Freund <andres@2ndquadrant.com> — 2013-08-01T15:06:30Z

Hi,

On 2013-08-01 15:40:22 +0100, Greg Stark wrote:
> Why isn't it enough to just dump out all variables with a source of alter
> system to a text file? You can either have a single global lock around that
> operation or write it to a new file and move it into place.

It saves you from several complications:
* No need to iterate over all GUCs, figuring out which come from which
  source, when writing out the file.
* Less logic required when writing out a value, since you have an
  acceptable input ready.
* No need to make sure the autogenerated file is written out in the same
  order when adding/changing a setting (to make sure you can
  diff/version control it sensibly)
* No locking necessary, without locking concurrent changes can vanish.
* Way easier to delete a setting if it ends up stopping postgres from
  starting.

Greetings,

Andres Freund

PS: .oO(quoting?)

-- 
 Andres Freund	                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Josh Berkus <josh@agliodbs.com> — 2013-08-01T17:06:26Z

On 08/01/2013 07:47 AM, David Johnston wrote:
> Minor request: could someone enlighten me as to why making the directory
> location a compile-time option is undesirable.  Packagers then can setup
> whatever structure they desire when they compile their distributions.  In
> which case the discussion becomes what is a reasonable default and that can
> be made with respect of other defaults that are in place for people that
> would self-compile.

Hey, that's a good idea.  Anyone else?

On 08/01/2013 06:32 AM, Greg Stark wrote:> On Thu, Aug 1, 2013 at 1:12
PM, Dimitri Fontaine <dimitri@2ndquadrant.fr> wrote:
>> we should review the implementation choice of the ALTER
>> SYSTEM SET facility, and vote for having one-file-per-GUC.
>
> Zombie crazy design idea arise!
>
> I think people are going to laugh at us if an open source database
> software can't manage a simple flat file database of settings,
> especially one that is purely write-only and can be a simple dump of
> settings that are set by alter system.

While I find some value in the one-setting-per-file approach, there's
also some major issues with it.  And we already argued this out months
ago, and ended up with the current single-file approach.  Let's not
rehash the past infinitely, please?

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Dimitri Fontaine <dimitri@2ndquadrant.fr> — 2013-08-01T17:11:20Z

Josh Berkus <josh@agliodbs.com> writes:
> While I find some value in the one-setting-per-file approach, there's
> also some major issues with it.  And we already argued this out months
> ago, and ended up with the current single-file approach.  Let's not
> rehash the past infinitely, please?

We rehash because the situation did change *a lot*. We just decided that
the ALTER SYSTEM SET setup will live in PGDATA and will not have to be
edited by DBA nor sysadmin nor tools ever. We will have a separate
facility (conf.d) for that. As a result, I don't think there's any
issues left with one-setting-per-file now.

So yes, I think it makes sense to review our position and make the thing
as easy as possible to code and maintain. See Andres' list about that,
earlier today on this same thread.

Regards,
-- 
Dimitri Fontaine
http://2ndQuadrant.fr     PostgreSQL : Expertise, Formation et Support

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Josh Berkus <josh@agliodbs.com> — 2013-08-01T17:13:37Z

On 07/26/2013 12:19 PM, Stephen Frost wrote:
> Agreed.  To continue that thought, I find it *very* unlikely that a
> given environment would use *both* a tool like puppet to manage the
> files in their conf.d *and* have people using ALTER SYSTEM SET.  You're
> going to do one or the other, almost certainly; not the least of which
> is because those are very likely two different teams and only one of
> them is going to be responsible for the PG system config.

Ideally, yes.  And that's the reason why I think that we will need to
implement a way to disable ALTER SYSTEM SET in postgresql.conf before
9.4.0 is done.  The big-Puppet-management shops will demand it; they do
NOT want their DBAs setting unversioned settings in isolation on one
database server out of 200.

However, I can imagine "hybrid" approaches.  For example, my personal
main use for conf.d/ is to modify logging settings on a temporary basis.
 I can imagine DBAs using ALTER SYSTEM SET for this purpose as well,
that is just to turn log_min_duration_statement down to 0 and then back
up again.

In that case, most of the "live" settings would live in /etc/postgresql,
but some of the logging settings would be controlled with ALTER SYSTEM
SET>  That conceptually seems to work fine with the existing design, I
just wanted to bring it up as a likely use case.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Josh Berkus <josh@agliodbs.com> — 2013-08-01T17:16:59Z

Dimitri,

> We rehash because the situation did change *a lot*. We just decided that
> the ALTER SYSTEM SET setup will live in PGDATA and will not have to be
> edited by DBA nor sysadmin nor tools ever. We will have a separate
> facility (conf.d) for that. As a result, I don't think there's any
> issues left with one-setting-per-file now.

Let's please NOT call it conf.d if it's living in PGDATA and is not
meant to be edited by hand.  conf.d is for a directory of config files
created by users and external utilities, living in CONFIGDIR.

If we're going to have a directory of individual file settings in
PGDATA, let's be consistent with the existing command and call it
"system_set".

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Andres Freund <andres@2ndquadrant.com> — 2013-08-01T17:21:14Z

On 2013-08-01 10:13:37 -0700, Josh Berkus wrote:
> On 07/26/2013 12:19 PM, Stephen Frost wrote:
> > Agreed.  To continue that thought, I find it *very* unlikely that a
> > given environment would use *both* a tool like puppet to manage the
> > files in their conf.d *and* have people using ALTER SYSTEM SET.  You're
> > going to do one or the other, almost certainly; not the least of which
> > is because those are very likely two different teams and only one of
> > them is going to be responsible for the PG system config.
> 
> Ideally, yes.  And that's the reason why I think that we will need to
> implement a way to disable ALTER SYSTEM SET in postgresql.conf before
> 9.4.0 is done.  The big-Puppet-management shops will demand it; they do
> NOT want their DBAs setting unversioned settings in isolation on one
> database server out of 200.

They would need a setting that disables ALTER (DATABASE|USER) ... SET
... as well though. At least for some settings.

I don't think enforcing things on that level makes much sense.

Greetings,

Andres Freund

-- 
 Andres Freund	                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Andres Freund <andres@2ndquadrant.com> — 2013-08-01T17:24:43Z

On 2013-08-01 10:16:59 -0700, Josh Berkus wrote:
> Dimitri,
> 
> > We rehash because the situation did change *a lot*. We just decided that
> > the ALTER SYSTEM SET setup will live in PGDATA and will not have to be
> > edited by DBA nor sysadmin nor tools ever. We will have a separate
> > facility (conf.d) for that. As a result, I don't think there's any
> > issues left with one-setting-per-file now.
> 
> Let's please NOT call it conf.d if it's living in PGDATA and is not
> meant to be edited by hand.  conf.d is for a directory of config files
> created by users and external utilities, living in CONFIGDIR.

How nice that that's not what's being discussed here then. conf.d *IS*
the thing thats been proposed to be a separate feature from ALTER
SYSTEM. For the use case you describe.

Greetings,

Andres Freund

-- 
 Andres Freund	                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Dimitri Fontaine <dimitri@2ndquadrant.fr> — 2013-08-01T17:48:13Z

Josh Berkus <josh@agliodbs.com> writes:
> Let's please NOT call it conf.d if it's living in PGDATA and is not
> meant to be edited by hand.  conf.d is for a directory of config files
> created by users and external utilities, living in CONFIGDIR.

+1

Regards,
-- 
Dimitri Fontaine
http://2ndQuadrant.fr     PostgreSQL : Expertise, Formation et Support

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Josh Berkus <josh@agliodbs.com> — 2013-08-01T18:02:12Z

On 08/01/2013 10:24 AM, Andres Freund wrote:
>> Let's please NOT call it conf.d if it's living in PGDATA and is not
>> meant to be edited by hand.  conf.d is for a directory of config files
>> created by users and external utilities, living in CONFIGDIR.
> 
> How nice that that's not what's being discussed here then. conf.d *IS*
> the thing thats been proposed to be a separate feature from ALTER
> SYSTEM. For the use case you describe.

Some of the earlier emails sounded like that's exactly what was
proposed.  Glad to clarify then.

*if* we do file-per-setting, what do you think of the directory name
system_set or system_conf then?

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Andres Freund <andres@2ndquadrant.com> — 2013-08-01T18:09:47Z


Josh Berkus <josh@agliodbs.com> schrieb:
>On 08/01/2013 10:24 AM, Andres Freund wrote:
>>> Let's please NOT call it conf.d if it's living in PGDATA and is not
>>> meant to be edited by hand.  conf.d is for a directory of config
>files
>>> created by users and external utilities, living in CONFIGDIR.
>> 
>> How nice that that's not what's being discussed here then. conf.d
>*IS*
>> the thing thats been proposed to be a separate feature from ALTER
>> SYSTEM. For the use case you describe.
>
>Some of the earlier emails sounded like that's exactly what was
>proposed.  Glad to clarify then.
>
>*if* we do file-per-setting, what do you think of the directory name
>system_set or system_conf then?

Postgresql.auto.conf.d is what I'd propose, but the decision about that seems to be one of the smaller problems around this feature... That naming seems to make it sensible to extend other files (hba, ident) similarly at a later point.

Please excuse brevity and formatting - I am writing this on my mobile phone.

Andres Freund	                   http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-01T18:37:45Z

* Alvaro Herrera (alvherre@2ndquadrant.com) wrote:
> Andres Freund wrote:
> 
> > Postgresql.auto.conf.d is what I'd propose, but the decision about
> > that seems to be one of the smaller problems around this feature...
> > That naming seems to make it sensible to extend other files (hba,
> > ident) similarly at a later point.
> 
> I don't like this particular naming proposal, but I'm glad this patch
> finally seems to be getting somewhere.

Yeah, also not a fan.  We don't have any 'conf.d' directories in PGDATA
and I don't think we should start now.  I liked Josh's suggestions of
something like "system_conf".  For multiple independent config files, we
could use directories under that (eg: 'pgconf', 'pghba', 'pgident'..).

	Thanks,

		Stephen

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Dimitri Fontaine <dimitri@2ndquadrant.fr> — 2013-08-01T23:03:56Z

Andres Freund <andres@2ndquadrant.com> writes:
> They would need a setting that disables ALTER (DATABASE|USER) ... SET
> ... as well though. At least for some settings.
>
> I don't think enforcing things on that level makes much sense.

If only we could trigger some actions when a command is about to be
executed, in a way that it's easy for the user to implement whatever
policy he fancies…

Oh, maybe I should finish preparing those patches for Event Triggers to
be fully usable in 9.4 then ;)

Regards,
-- 
Dimitri Fontaine
http://2ndQuadrant.fr     PostgreSQL : Expertise, Formation et Support

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Andres Freund <andres@2ndquadrant.com> — 2013-08-01T23:16:08Z

On 2013-08-01 14:37:45 -0400, Stephen Frost wrote:
> * Alvaro Herrera (alvherre@2ndquadrant.com) wrote:
> > Andres Freund wrote:
> > 
> > > Postgresql.auto.conf.d is what I'd propose, but the decision about
> > > that seems to be one of the smaller problems around this feature...
> > > That naming seems to make it sensible to extend other files (hba,
> > > ident) similarly at a later point.
> > 
> > I don't like this particular naming proposal, but I'm glad this patch
> > finally seems to be getting somewhere.
> 
> Yeah, also not a fan.  We don't have any 'conf.d' directories in PGDATA
> and I don't think we should start now.  I liked Josh's suggestions of
> something like "system_conf".  For multiple independent config files, we
> could use directories under that (eg: 'pgconf', 'pghba', 'pgident'..).

People know what to expect from .d directories, that's why I suggested
it, don't feel really strongly about it. I dislike naming the
subdirectories pgconf/... et al though, we should reference the original
files name, instead of introducing new abbreviations.

Greetings,

Andres Freund

-- 
 Andres Freund	                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-02T00:33:49Z

* Andres Freund (andres@2ndquadrant.com) wrote:
> People know what to expect from .d directories, that's why I suggested
> it, don't feel really strongly about it. I dislike naming the
> subdirectories pgconf/... et al though, we should reference the original
> files name, instead of introducing new abbreviations.

*People* should not be messing around with files under $PGDATA- that's
half the point.  conf.d or other names which look like configuration
directories that an admin should modify are a *bad* idea.

	Thanks,

		Stephen

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Andres Freund <andres@2ndquadrant.com> — 2013-08-02T00:40:51Z

On 2013-08-01 20:33:49 -0400, Stephen Frost wrote:
> * Andres Freund (andres@2ndquadrant.com) wrote:
> > People know what to expect from .d directories, that's why I suggested
> > it, don't feel really strongly about it. I dislike naming the
> > subdirectories pgconf/... et al though, we should reference the original
> > files name, instead of introducing new abbreviations.
> 
> *People* should not be messing around with files under $PGDATA- that's
> half the point.  conf.d or other names which look like configuration
> directories that an admin should modify are a *bad* idea.

I personally consider readers of this list persons... And even people
not interested in internals will have to look in there if they set
something stupid before. Like setting max_connections higher than the
currently configured kernel's max number of semaphores. Or a good number
of other settings.

Greetings,

Andres Freund

-- 
 Andres Freund	                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-02T00:45:38Z

* Andres Freund (andres@2ndquadrant.com) wrote:
> I personally consider readers of this list persons... And even people
> not interested in internals will have to look in there if they set
> something stupid before. Like setting max_connections higher than the
> currently configured kernel's max number of semaphores. Or a good number
> of other settings.

And that's actually one of the issues that I have with this overall
approach..  If configurations can be set in 'data' areas which prevent
the system from starting, that's *bad*.  Very bad.  It means people will
not be able to trust PG to manage the configuration sanely and will have
a lot of distrust and fear of the ALTER SYSTEM capability.

Requiring users to go monkey around inside of a system data directory to
clean things up in order to get PG to come up is a situation we should
do our best to prevent from ever happening.

	Thanks,

		Stephen

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Andres Freund <andres@2ndquadrant.com> — 2013-08-02T00:53:26Z

On 2013-08-01 20:45:38 -0400, Stephen Frost wrote:
> * Andres Freund (andres@2ndquadrant.com) wrote:
> > I personally consider readers of this list persons... And even people
> > not interested in internals will have to look in there if they set
> > something stupid before. Like setting max_connections higher than the
> > currently configured kernel's max number of semaphores. Or a good number
> > of other settings.
> 
> And that's actually one of the issues that I have with this overall
> approach..  If configurations can be set in 'data' areas which prevent
> the system from starting, that's *bad*.  Very bad.  It means people will
> not be able to trust PG to manage the configuration sanely and will have
> a lot of distrust and fear of the ALTER SYSTEM capability.

I agree that we need to do reasonable checks, like running GUC
validators, but we simply can't control the overall system state. And
it's not like this are errors that you couldn't get before. And we
should (that's something to improve on) report the relevant guc + file
in many cases.

> Requiring users to go monkey around inside of a system data directory to
> clean things up in order to get PG to come up is a situation we should
> do our best to prevent from ever happening.

Even trying to do this completely will guarantee that this patch will
never, ever, suceed. There simply is no way to reliably detect problems
that have complex interactions with the rest of the system.

We can improve the detection rate of problems after some real world
experience. Don't make this unneccesarily complex.

Greetings,

Andres Freund

-- 
 Andres Freund	                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

David Johnston <polobo@yahoo.com> — 2013-08-02T01:06:20Z

Andres Freund-3 wrote
> Even trying to do this completely will guarantee that this patch will
> never, ever, suceed. There simply is no way to reliably detect problems
> that have complex interactions with the rest of the system.
> 
> We can improve the detection rate of problems after some real world
> experience. Don't make this unneccesarily complex.

Instead of prevention some thought to recovery should be considered then.

How about some form of persistence mechanism so that, before making these
kinds of changes, the admin can "save" the current configuration.  Then, in
a worse case-scenario, they could run something like "pg_ctl
--restore-persisted-configuration ..." to reset everything back the last
known good configuration.

A single-version save-restore routine for the configuration.  When restoring
you would want to keep the "current/non-working" configuration and
associated logging information - maybe archived somewhere along with the a
copy of the last known working version.  This would provide some level of
audit capability as well as a convenient way for someone to take that
archive and send it off to someone more knowledgeable for assistance. 
Having it auto-run at boot time - possibly to a different archive area than
when run manually - would be possible as well; so you'd have both the last
good boot configuration as well as whatever point-in-time configurations you
wish to save.







--
View this message in context: http://postgresql.1045698.n5.nabble.com/Proposal-for-Allow-postgresql-conf-values-to-be-changed-via-SQL-tp5729917p5765968.html
Sent from the PostgreSQL - hackers mailing list archive at Nabble.com.

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-02T01:06:49Z

* Andres Freund (andres@2ndquadrant.com) wrote:
> I agree that we need to do reasonable checks, like running GUC
> validators, but we simply can't control the overall system state. And
> it's not like this are errors that you couldn't get before. And we
> should (that's something to improve on) report the relevant guc + file
> in many cases.

You could get the errors before, sure, but when you did, you could read
the log output and go modify the *configuration files* (which things in
$PGDATA are *not*) and fix it and get the system back online.  If the
files in $PGDATA have to be modified to get the system online then they
are configuration files and should be in /etc.

> Even trying to do this completely will guarantee that this patch will
> never, ever, suceed. There simply is no way to reliably detect problems
> that have complex interactions with the rest of the system.

The patch will never be able to completely remove the need for external
config files, without changes to PG to deal with these conditions
better.

> We can improve the detection rate of problems after some real world
> experience. Don't make this unneccesarily complex.

Actually, putting it out there as "this can be used to modify anything
and means you can trivially make PG unstartable" is actually the wrong
move to make, imv.  Consider that, to deal with the issues caused, we'd
have to *remove* things from being modifyable through this function.
That's a whole lot harder to do from a backward-compatibility view than
adding things later as we improve PG to be able to still come up enough
to be useful even with configuration issues.

	Thanks,

		Stephen

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-02T01:15:16Z

* David Johnston (polobo@yahoo.com) wrote:
> How about some form of persistence mechanism so that, before making these
> kinds of changes, the admin can "save" the current configuration.  Then, in
> a worse case-scenario, they could run something like "pg_ctl
> --restore-persisted-configuration ..." to reset everything back the last
> known good configuration.

Yeah, I had considered the possibility that we would provide a tool to
manage the config in $PGDATA but I'm not really thrilled with that idea
either.

> A single-version save-restore routine for the configuration.  When restoring
> you would want to keep the "current/non-working" configuration and
> associated logging information - maybe archived somewhere along with the a
> copy of the last known working version.  This would provide some level of
> audit capability as well as a convenient way for someone to take that
> archive and send it off to someone more knowledgeable for assistance. 
> Having it auto-run at boot time - possibly to a different archive area than
> when run manually - would be possible as well; so you'd have both the last
> good boot configuration as well as whatever point-in-time configurations you
> wish to save.

Yeah, there's a lot of work involved in writing a whole system for
managing multiple configurations with history, diffs, etc..  Problems
which, really, our existing text-based config w/ a tool like puppet have
already solved.

	Thanks,

		Stephen

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Andres Freund <andres@2ndquadrant.com> — 2013-08-02T03:01:16Z

Hi,

FWIW, I think you've just put the final nail in the coffin of this
patch by raising the barriers unreasonably high.

> * Andres Freund (andres@2ndquadrant.com) wrote:
> > I agree that we need to do reasonable checks, like running GUC
> > validators, but we simply can't control the overall system state. And
> > it's not like this are errors that you couldn't get before. And we
> > should (that's something to improve on) report the relevant guc + file
> > in many cases.
> 
> You could get the errors before, sure, but when you did, you could read
> the log output and go modify the *configuration files* (which things in
> $PGDATA are *not*) and fix it and get the system back online.  If the
> files in $PGDATA have to be modified to get the system online then they
> are configuration files and should be in /etc.

That doesn't seem to be a logical consequence to me.

On 2013-08-01 21:06:49 -0400, Stephen Frost wrote:
> > Even trying to do this completely will guarantee that this patch will
> > never, ever, suceed. There simply is no way to reliably detect problems
> > that have complex interactions with the rest of the system.
> 
> The patch will never be able to completely remove the need for external
> config files, without changes to PG to deal with these conditions
> better.

That's not the goal of the patch as far as I understand it.

> > We can improve the detection rate of problems after some real world
> > experience. Don't make this unneccesarily complex.
> 
> Actually, putting it out there as "this can be used to modify anything
> and means you can trivially make PG unstartable" is actually the wrong
> move to make, imv.  Consider that, to deal with the issues caused, we'd
> have to *remove* things from being modifyable through this function.
> That's a whole lot harder to do from a backward-compatibility view than
> adding things later as we improve PG to be able to still come up enough
> to be useful even with configuration issues.

I think this chain of argument doesn't have much for it. There are
litteraly dozens of ways to break postgres from SQL which we don't even
try to defend against. Starting from DELETE FROM pg_class, ending with
COPYing files into the datadir. This is a database, not a children's
toy, and the feature *is* superuser only.

Anyway, I don't see much point arguing this further.

Greetings,

Andres Freund

-- 
 Andres Freund	                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-02T03:27:21Z

* Andres Freund (andres@2ndquadrant.com) wrote:
> FWIW, I think you've just put the final nail in the coffin of this
> patch by raising the barriers unreasonably high.

For my 2c, I don't think it's an unreasonable idea to actually
*consider* what options are available through this mechanism rather than
just presuming that it's a good idea to be able to modify anything,
including things that you wouldn't be able to fix after a restart w/o
hacking around in $PGDATA.

I also don't believe that limiting the set of options which can be
modified through this system is a particularly difficult thing to
implement.

> > * Andres Freund (andres@2ndquadrant.com) wrote:
> On 2013-08-01 21:06:49 -0400, Stephen Frost wrote:
> > > Even trying to do this completely will guarantee that this patch will
> > > never, ever, suceed. There simply is no way to reliably detect problems
> > > that have complex interactions with the rest of the system.
> > 
> > The patch will never be able to completely remove the need for external
> > config files, without changes to PG to deal with these conditions
> > better.
> 
> That's not the goal of the patch as far as I understand it.

The point above is that we will always need some amount of external
config file and, as such, we should probably consider which items should
really only be set in the *config* files and which can be set in either
place.

> I think this chain of argument doesn't have much for it. There are
> litteraly dozens of ways to break postgres from SQL which we don't even
> try to defend against. 

This is a strawman.  An admin doing "DELETE FROM pg_class;" or using
COPY to overwrite files in PG's data dir and doing "ALTER SYSTEM SET
shared_buffers = '2GB';", "ALTER SYSTEM SET port = 123;" or even "ALTER
SYSTEM SET data_directory = '/new/path/for/db';" (how would doing that
even make sense..?) are not nearly the same.  On the flip side, there's
not nearly as much risk around allowing log_line_prefix and friends to
be set through ALTER SYSTEM SET because it's pretty unlikely that such
a misconfiguration would cause PG to not start.

	Thanks,

		Stephen

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Peter Geoghegan <pg@heroku.com> — 2013-08-02T03:47:52Z

On Thu, Aug 1, 2013 at 8:27 PM, Stephen Frost <sfrost@snowman.net> wrote:
> The point above is that we will always need some amount of external
> config file and, as such, we should probably consider which items should
> really only be set in the *config* files and which can be set in either
> place.

What settings did you have in mind? Which ones ought to be only be
settable in config files in your estimation?


-- 
Peter Geoghegan

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Alvaro Herrera <alvherre@2ndquadrant.com> — 2013-08-02T04:13:05Z

> On Thu, Aug 1, 2013 at 8:27 PM, Stephen Frost <sfrost@snowman.net> wrote:
> > The point above is that we will always need some amount of external
> > config file and, as such, we should probably consider which items should
> > really only be set in the *config* files and which can be set in either
> > place.

I think this idea might make some sense: mark the settings (add a
flag bit in the guc.c tables) that can be changed via alter setting.  Or
perhaps add the bit to ones that can *not* be changed.

I don't think we need to promise exact compatibility on the set of
settings that can be changed via ALTER SYSTEM.  If you can change
setting XYZ in 9.4, and we find that it wasn't such a good idea and
have to disallow it in 9.5, well, too bad.  (Or perhaps even a minor
version.  Am I sacrilegious?)

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila@huawei.com> — 2013-08-02T05:58:47Z

On Friday, August 02, 2013 4:34 AM Dimitri Fontaine wrote:
> Andres Freund <andres@2ndquadrant.com> writes:
> > They would need a setting that disables ALTER (DATABASE|USER) ... SET
> > ... as well though. At least for some settings.
> >
> > I don't think enforcing things on that level makes much sense.
> 
> If only we could trigger some actions when a command is about to be
> executed, in a way that it's easy for the user to implement whatever
> policy he fancies…
> 
> Oh, maybe I should finish preparing those patches for Event Triggers to
> be fully usable in 9.4 then ;)

I think this can be one useful way to disable.

With Regards,
Amit Kapila.

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila@huawei.com> — 2013-08-02T07:23:17Z

On Friday, August 02, 2013 8:57 AM Stephen Frost wrote:
> * Andres Freund (andres@2ndquadrant.com) wrote:
> > FWIW, I think you've just put the final nail in the coffin of this
> > patch by raising the barriers unreasonably high.
> 
> For my 2c, I don't think it's an unreasonable idea to actually
> *consider* what options are available through this mechanism rather
> than just presuming that it's a good idea to be able to modify
> anything, including things that you wouldn't be able to fix after a
> restart w/o hacking around in $PGDATA.

I think if user has set any value wrong such that it doesn't allow server to
start,
we can provide an option similar to pg_resetxlog to reset the auto file.

How about with such an option user might loose some settings?
1. We can think of providing reset for an particular parameter.
2. Suggestions in docs that in case of such a scenario, he can remember
values from auto file and reset the settings.

As this can happen only in rare scenario's, I think it can make sense to
provide such option.

With Regards,
Amit Kapila.

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Cédric Villemain <cedric@2ndquadrant.com> — 2013-08-02T10:46:53Z

Le vendredi 2 août 2013 09:23:17, Amit Kapila a écrit :
> On Friday, August 02, 2013 8:57 AM Stephen Frost wrote:
> > * Andres Freund (andres@2ndquadrant.com) wrote:
> > > FWIW, I think you've just put the final nail in the coffin of this
> > > patch by raising the barriers unreasonably high.
> > 
> > For my 2c, I don't think it's an unreasonable idea to actually
> > *consider* what options are available through this mechanism rather
> > than just presuming that it's a good idea to be able to modify
> > anything, including things that you wouldn't be able to fix after a
> > restart w/o hacking around in $PGDATA.
> 
> I think if user has set any value wrong such that it doesn't allow server
> to start,
> we can provide an option similar to pg_resetxlog to reset the auto file.

While guessing what changes are safe is hard, it is easy to discover the GUCs 
preventing PostgreSQL from restarting correctly.
pg_ctl might be able to expose a clear message like : 
MSG: Params X and Y set by ALTER SYSTEM prevent PostgreSQL from starting
HINT: Issue pg_ctl --ignore-bad-GUC start

Note that the same may also allow postgresql to start with bad GUC value in 
postgresql.conf ....
So this is another topic (IMHO).

With the current idea of one-GUC-one-file in a PGDATA subdirectory it is *easy* 
to fix for any DBA or admin. However, note that I do prefer a simple 
'include_auto_conf=here|start|end|off' in postgresql.conf (by default at end of 
file, with value 'end' set). 

-- 
Cédric Villemain +33 (0)6 20 30 22 52
http://2ndQuadrant.fr/
PostgreSQL: Support 24x7 - Développement, Expertise et Formation

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila@huawei.com> — 2013-08-02T11:38:12Z

On Thursday, August 01, 2013 8:37 PM Andres Freund wrote:
> Hi,
> 
> On 2013-08-01 15:40:22 +0100, Greg Stark wrote:
> > Why isn't it enough to just dump out all variables with a source of
> alter
> > system to a text file? You can either have a single global lock
> around that
> > operation or write it to a new file and move it into place.
> 
> It saves you from several complications:
> * No need to iterate over all GUCs, figuring out which come from which
>   source, when writing out the file.

   Not all GUC's, only which are in auto file.

> * Less logic required when writing out a value, since you have an
>   acceptable input ready.
> * No need to make sure the autogenerated file is written out in the
> same
>   order when adding/changing a setting (to make sure you can
>   diff/version control it sensibly)
> * No locking necessary, without locking concurrent changes can vanish.
> * Way easier to delete a setting if it ends up stopping postgres from
>   starting.

    The logic needed in current patch for above points is not all that
complex that it
    needs to be thought of redesign until the basic idea of
one-file-per-setting scores high 
    over one-file-all-setting.

Below are some points in my mind due to which I have supported/implemented
one-file-all-setting approach:
a. I had heard quite a few times that Postgres has lot of files (each
relation has separate file) as compare to Oracle.
   Users feel that Oracle's table space approach is better.
b. While server start/Sighup, we needs to open/read/close each file
separately which in-itself seems to be overhead.

I believe what Greg Stark has said in his below mail link is the more
appropriate way and the current patch has done that way.
http://www.postgresql.org/message-id/CAM-w4HP7=a2VowyJUD0CAZL5b8FsaHymdQeouL
udSOhdnCw_zg@mail.gmail.com

Also when other commercial database (Oracle) can do it with single file,
users will try to compare with it.

I understand that our views are not matching on this point and I totally
respect your views, but not able to convince myself
for one-file-per-setting approach. Can you please once again think and see
if there is a viable way for moving forward.
I had modified the patch for many suggestions which had made it simpler and
if you have any idea to make one-file-all-settings
implementation better, then I can address it. 

With Regards,
Amit Kapila.

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila@huawei.com> — 2013-08-02T11:44:01Z

On Friday, August 02, 2013 4:17 PM Cédric Villemain wrote:
Le vendredi 2 août 2013 09:23:17, Amit Kapila a écrit :
> On Friday, August 02, 2013 8:57 AM Stephen Frost wrote:
> > * Andres Freund (andres@2ndquadrant.com) wrote:
>> > > FWIW, I think you've just put the final nail in the coffin of this
>> > > patch by raising the barriers unreasonably high.
>> > 
>> > For my 2c, I don't think it's an unreasonable idea to actually
>> > *consider* what options are available through this mechanism rather
>> > than just presuming that it's a good idea to be able to modify
>> > anything, including things that you wouldn't be able to fix after a
>> > restart w/o hacking around in $PGDATA.
> 
>> I think if user has set any value wrong such that it doesn't allow server
>> to start,
>> we can provide an option similar to pg_resetxlog to reset the auto file.
 
> While guessing what changes are safe is hard, it is easy to discover the
GUCs preventing PostgreSQL from restarting correctly.
> pg_ctl might be able to expose a clear message like : 
> MSG: Params X and Y set by ALTER SYSTEM prevent PostgreSQL from starting
> HINT: Issue pg_ctl --ignore-bad-GUC start
 
> Note that the same may also allow postgresql to start with bad GUC value
in postgresql.conf ....
> So this is another topic (IMHO).

  Yes, this can be viable option to ignore variable values that don't allow
server to start, also I agree with you that
  this can be a separate patch.
 
> With the current idea of one-GUC-one-file in a PGDATA subdirectory it is
*easy* to fix for any DBA or admin.

 
> However, note that I do prefer a simple
'include_auto_conf=here|start|end|off' in postgresql.conf (by default at end
of file, with value 'end' set). 
   Earlier patch has this implementation, but later based on suggestions, I
made it default.

With Regards,
Amit Kapila.

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-02T11:49:14Z

* Amit Kapila (amit.kapila@huawei.com) wrote:
> Below are some points in my mind due to which I have supported/implemented
> one-file-all-setting approach:
> a. I had heard quite a few times that Postgres has lot of files (each
> relation has separate file) as compare to Oracle.
>    Users feel that Oracle's table space approach is better.

This is completely unrelated to this discussion, imv.

> b. While server start/Sighup, we needs to open/read/close each file
> separately which in-itself seems to be overhead.

I also don't think performance of this particular operation should be a
high priority.

> I believe what Greg Stark has said in his below mail link is the more
> appropriate way and the current patch has done that way.
> http://www.postgresql.org/message-id/CAM-w4HP7=a2VowyJUD0CAZL5b8FsaHymdQeouL
> udSOhdnCw_zg@mail.gmail.com

He doesn't actually provide any reasoning for it.  That said, I've not
heard any particularly good reason for having a setting per file either.
This is an internal-to-PG data file and we should really implement it in
whichever way makes the most sense for us.  My general feeling is that
one file is simpler and sufficient for the postgresql.conf-like
parameters, but I wonder what we're going to do for pg_hba/pg_ident.
Those would be good to have multiple files for because (as we saw with
pg_authid) they could get quite large because they can have per-user
entries in them and rewriting a large file for every change would be
quite painful.

> Also when other commercial database (Oracle) can do it with single file,
> users will try to compare with it.

To make it clear- this isn't justification for this design.  Also, the
notion that Oracle's *configuration* is all done with a *single-file*
is..  laughable.

	Thanks,

		Stephen

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-02T11:52:52Z

* Amit Kapila (amit.kapila@huawei.com) wrote:
>   Yes, this can be viable option to ignore variable values that don't allow
> server to start, also I agree with you that
>   this can be a separate patch.

I disagree that this can be a separate patch.  Adding an option to not
allow certain GUCs from being modified through this mechanism should be
trivial.  Then, *after* we have such utility that will allow users to
fix a bad situation, we can consider relaxing those restrictions to
allow more values to be set.  I still contend that there will be some
GUCs that simply don't make any sense to have in the auto-conf area of
PGDATA.

	Thanks,

		Stephen

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila@huawei.com> — 2013-08-02T12:27:38Z

On Friday, August 02, 2013 5:19 PM Stephen Frost wrote:
> * Amit Kapila (amit.kapila@huawei.com) wrote:
> > Below are some points in my mind due to which I have
> > supported/implemented one-file-all-setting approach:
> > a. I had heard quite a few times that Postgres has lot of files (each
> > relation has separate file) as compare to Oracle.
> >    Users feel that Oracle's table space approach is better.
> 
> This is completely unrelated to this discussion, imv.
   The point I wanted to convey is that having more files for database in
general is not a great idea.

> > b. While server start/Sighup, we needs to open/read/close each file
> > separately which in-itself seems to be overhead.
> 
> I also don't think performance of this particular operation should be a
> high priority.
  
  If it makes startup taking more time, then isn't it a performance critical
path?
 
> > I believe what Greg Stark has said in his below mail link is the more
> > appropriate way and the current patch has done that way.
> > http://www.postgresql.org/message-id/CAM-
> w4HP7=a2VowyJUD0CAZL5b8FsaHym
> > dQeouL
> > udSOhdnCw_zg@mail.gmail.com
> 
> He doesn't actually provide any reasoning for it.  That said, I've not
> heard any particularly good reason for having a setting per file
> either.
> This is an internal-to-PG data file and we should really implement it
> in whichever way makes the most sense for us.  My general feeling is
> that one file is simpler and sufficient for the postgresql.conf-like
> parameters, 

Sure, I also feel the same that if it can be addressed with single file,
then lets do that way only.

> but I wonder what we're going to do for pg_hba/pg_ident.
> Those would be good to have multiple files for because (as we saw with
> pg_authid) they could get quite large because they can have per-user
> entries in them and rewriting a large file for every change would be
> quite painful.
> 
> > Also when other commercial database (Oracle) can do it with single
> > file, users will try to compare with it.
> 
> To make it clear- this isn't justification for this design. 


> Also, the
> notion that Oracle's *configuration* is all done with a *single-file*
> is..  laughable.

Not all Oracle's configuration, but Change of configuration parameters.
IIRC, before starting this feature I had checked Oracle's specs and it seems
to be
not doing with multiple files for Alter System. If you have doubt, I can
once again 
Verify it.

With Regards,
Amit Kapila.

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-02T12:41:09Z

* Amit Kapila (amit.kapila@huawei.com) wrote:
> > This is an internal-to-PG data file and we should really implement it
> > in whichever way makes the most sense for us.  My general feeling is
> > that one file is simpler and sufficient for the postgresql.conf-like
> > parameters, 
> 
> Sure, I also feel the same that if it can be addressed with single file,
> then lets do that way only.

We need to settle on one choice and then implement it, yes.  It
certainly doesn't make any sense to have two different ways to deal with
an internal-to-PG data structure.  Of course, that might argue for
making this file actually *be* like pg_authid is today; has that been
considered?  I'm guessing it's not practical because the point where we
need to read the config is before certain things have been set up to
allow reading from heap files, but it seems like something which should
at least be considered.  If we can make it work, then that may also
solve the pg_hba/pg_ident issue, which is about a bazillion times more
interesting than the mostly set-and-forget postgresql.conf settings.

Perhaps having the file be a heap file instead of anything a sysadmin
can be asked to go hack would also make it more clear that this is an
internal PG file which is to be managed only through PG and stop all
this arguing about how "oh, they can just fix it by twiddling things in
$PGDATA" is considered by some to be an acceptable solution.  Heck, it'd
also keep the number of files down while allowing more fine-grained
modifications and writes (iow, we wouldn't have to rewrite the whole
file every time..).

	Thanks,

		Stephen

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Tom Lane <tgl@sss.pgh.pa.us> — 2013-08-02T12:45:56Z

Stephen Frost <sfrost@snowman.net> writes:
> Perhaps having the file be a heap file instead of anything a sysadmin
> can be asked to go hack would also make it more clear that this is an
> internal PG file which is to be managed only through PG and stop all
> this arguing about how "oh, they can just fix it by twiddling things in
> $PGDATA" is considered by some to be an acceptable solution.

Whether you think it's acceptable or not, sometimes it's *necessary*.
What if you set a combination of parameters that prevents Postgres from
starting?

Even if we could make config-in-a-catalog work, which I'm very dubious
of, I would think it far too unsafe to use.

We could of course fix that problem by only storing "safe" parameters
in a catalog, but then you lose the supposed appeal of a single-file
solution.

			regards, tom lane

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Greg Smith <greg@2ndquadrant.com> — 2013-08-02T13:24:50Z

On 7/29/13 3:46 PM, Josh Berkus wrote:
> Based on the ongoing discussion of this patch, I have moved it to 9.4CF2
> (9-2013).
>
> Mind you, it would be good to commit some version of it before September.

Quite a bit of the patch adds some refactored GUC parameter validation 
code that seems necessary for this feature, and that's now where the 
controversial parts are either.  That's the part I thought should be 
looked at for commit now.  I'd like to get the change size Amit is 
carrying around and (and other people are reviewing) reduced here.

-- 
Greg Smith   2ndQuadrant US    greg@2ndQuadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Andres Freund <andres@2ndquadrant.com> — 2013-08-02T13:45:05Z

On 2013-08-02 08:41:09 -0400, Stephen Frost wrote:
> Perhaps having the file be a heap file instead of anything a sysadmin
> can be asked to go hack would also make it more clear that this is an
> internal PG file which is to be managed only through PG and stop all
> this arguing about how "oh, they can just fix it by twiddling things in
> $PGDATA" is considered by some to be an acceptable solution.  Heck, it'd
> also keep the number of files down while allowing more fine-grained
> modifications and writes (iow, we wouldn't have to rewrite the whole
> file every time..).

Crash recovery & PGC_SIGHUP|PGC_POSTMASTER…

Greetings,

Andres Freund

-- 
 Andres Freund	                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Greg Stark <stark@mit.edu> — 2013-08-02T14:04:13Z

Writing out each guc in a separate file is a singularly bad idea. It's
going out of our way to confuse users about what's going on and how
they're expected to interact with the settings files and it actively
makes it harder or nearly impossible to protect against simple
failures.

1) The whole reason for conf.d directories for things like Apache or
cron or whatever is so that other software can drop in snippets
without having to parse and edit a file in place. We *do not* want
users doing that inside PGDATA.

I'm not even clear we do want this in /etc since none of our GUC
options are repeatable things like Apache virtual servers. It actually
makes *more* sense for pg_hba than it does for gucs. I think we can
assume that in the future we'll have something like it however.

2) Directories are notoriously hard to version control, most version
control systems either don't do it at all or do a weak form of version
control on directories. Even if users tried to track the changes in
these files they'll likely find it difficult to tell when two settings
were changed together or in separate changes. On the other hand if all
the settings are in a single file then even the simplest form of
version control -- backup files -- would suffice.

If we just keep a backup copy of the settings file for each change
then it would be easy for people to diff from one version to another
and see all the changes together and easy for them to restore an old
copy if the current one isn't starting up. If they're in a million
tiny files then users would have to keep a backup copy of the whole
directory and dig thorugh a recursive diff of the whole directory. Or
they would have tons of backup files for different settings at
different times and need to figure out which ones were in effect at a
given time.

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Greg Smith <greg@2ndquadrant.com> — 2013-08-02T14:07:33Z

On 8/1/13 10:47 AM, David Johnston wrote:
> Minor request: could someone enlighten me as to why making the directory
> location a compile-time option is undesirable.

The ongoing argument here is whether to allow moving the directory at 
all, or to make it fixed to $PGDATA the way recovery.conf is.  If you 
accept that it should float, then it actually needs to be a start time 
option.  Software like Debian moves around the postgresql.conf like this:

pg_ctl -c config_file=/etc/postgresql/9.3/main/postgresql.conf ...

The way this argument is going the last few days, I'm starting to think 
that it's worth breaking this style of config directory setup out into 
its own feature now.

Whether or not ALTER SYSTEM SET takes advantage of the config directory 
or not seems a still raging question.  I've been coupling the two 
together because I think the design of ALTER SYSTEM SET should consider 
a config directory based approach.  But from the perspective of what can 
get committed first, the config directory really should go first.

-- 
Greg Smith   2ndQuadrant US    greg@2ndQuadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-02T14:54:42Z

* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> What if you set a combination of parameters that prevents Postgres from
> starting?

This was what I was trying to get at up-thread.  Things that prevent PG
from being able to start (or, really, which cause PG to be started in a
completely different mode, ala recovery.conf) need to be able to be
modified outside of PG and therefore should, imv, be considered
configuration parameters and therefore live outside of $PGDATA (when
installed from a distro, blah, blah).

> We could of course fix that problem by only storing "safe" parameters
> in a catalog, but then you lose the supposed appeal of a single-file
> solution.

I don't see "having them all in one file is more convenient for the
admin" as being a justification for the single-file approach, simply
because I don't consider the 'auto' file to be something that the admin
would be modifying.

Curiously, I've not heard any argument about what parameters are "safe"
and what aren't, though I was asked which ones I thought were safe and
which weren't.  Perhaps looking at the specific options that would
likely cause PG to not start would be useful to this discussion.

Off-hand, I see:

data_directory-
  Clearly we need to know this before starting, so it has to be defined
  somewhere and then passed to PG in some way.  Currently this is done
  in Ubuntu by having the init script read the directory out of the
  postgresql.conf, but it could also be passed through Ubuntu's
  "pg_ctl.conf", which takes a set of parameters to pass.  I will note
  here, though it may apply in other places also, that this part of the
  configuration is necessairly the distro's responsibility.

hba_file-
  Won't start without one.

ident_file-
  We *will* start without one.  We'll even start if it's got garbage in
  it.  I have to admit, I was a bit surprised that this behaves so
  differently from hba_file wrt dealing with existance / errors.

listen_addresses-
  Won't start if it's invalid, but if it's not there we'll just try to
  listen on localhost:5432, but if that fails we won't start.

port-
  Similar to listen_addresses

unix_socket_directories, unix_socket_group-
  Won't start if it's invalid, will start w/ a default if not set.

ssl_cert_file, ssl_key_file, ssl_ca_file, ssl_crl_file,
krb_server_keyfile, shared_preload_libraries-
  Won't start if it's invalid, not used if not set.

shared_buffers-
  Older versions won't start if it's set above the SHM limits on the
  system.  Even in 9.3+ though, if set too high, will either cause it to
  not start or to possible crash very quickly (eg: user set it to
  however much real memory they have in the system).

log_directory, log_filename-
  Won't start if set such that PG can't create the necessary directories
  or open its log file.

log_timezone, timezone, lc_messages, lc_monetary, etc-
  Won't start if set invalid- can we check validity of this when set
  through ALTER SYSTEM?

local_preload_libraries-
  Will start if it's set to something invalid, but then you can't
  connect because new backend starts will fail.

Now, there's certainly a whole slew of things which *can* be modified
w/o too much risk to being able to get PG started again.  Also, many of
the things above could probably be changed to deal with error cases and
keep starting up (eg: ssl_*).  Andrew pointed out that we can use
command-line arguments to override badly configured parameters, but I'm
worried that'd simply end up making those items be configured through
the distro scripts, and probably in a way that isn't as nice as
postgresql.conf, eg:

Ubuntu's pg_ctl.conf might become:
pg_ctl_options = '-o "-c listen_addresses=127.0.01.1" -o "-c port=5433"'

etc, or, more likely, a *new* config file being created to manage these
things which is completely distribution-specific...

	Thanks,

		Stephen

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-02T15:05:31Z

Greg,

* Greg Stark (stark@mit.edu) wrote:
> Writing out each guc in a separate file is a singularly bad idea. It's
> going out of our way to confuse users about what's going on and how
> they're expected to interact with the settings files and it actively
> makes it harder or nearly impossible to protect against simple
> failures.

I agree that having a separate file for each GUC is a bad idea.  That
said, I still contend that there's a difference between files in $PGDATA
and files found in /etc.

> 1) The whole reason for conf.d directories for things like Apache or
> cron or whatever is so that other software can drop in snippets
> without having to parse and edit a file in place. We *do not* want
> users doing that inside PGDATA.

Agreed.  That said, if users *want* a separate file per GUC in their
/etc, we can let them do that with the conf.d structure.

> I'm not even clear we do want this in /etc since none of our GUC
> options are repeatable things like Apache virtual servers. It actually
> makes *more* sense for pg_hba than it does for gucs. I think we can
> assume that in the future we'll have something like it however.

I tend to agree with this also, though I can imagine wanting to separate
things in a conf.d directory ala exim's conf.d directories, to allow
tools like puppet to manage certain things environment-wide (perhaps
krb_server_keyfile) while other configuration options are managed
locally.

> If we just keep a backup copy of the settings file for each change
> then it would be easy for people to diff from one version to another
> and see all the changes together and easy for them to restore an old
> copy if the current one isn't starting up. If they're in a million
> tiny files then users would have to keep a backup copy of the whole
> directory and dig thorugh a recursive diff of the whole directory. Or
> they would have tons of backup files for different settings at
> different times and need to figure out which ones were in effect at a
> given time.

This I don't entirely follow though.  Above, you don't want users
monkeying around in PGDATA dropping files, but it's ok for them to be
diffing them and, presumably, rewritting them with older version when
they feel the need to?

I agree that we should provide a way for users to get old versions of
their config and know when things changed, but, to be honest, I'd like
to see that kind of audit log information for various catalog tables
too- eg: pg_database.  If we had event triggers and they could be fired
for ALTER SYSTEM along with ALTER DATABASE, then we could have an audit
system with an audit table which takes who did what when, old value, new
value, etc..

	Thanks,

		Stephen

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Alvaro Herrera <alvherre@2ndquadrant.com> — 2013-08-02T15:36:42Z

Dimitri Fontaine escribió:
> Andres Freund <andres@2ndquadrant.com> writes:
> > They would need a setting that disables ALTER (DATABASE|USER) ... SET
> > ... as well though. At least for some settings.
> >
> > I don't think enforcing things on that level makes much sense.
> 
> If only we could trigger some actions when a command is about to be
> executed, in a way that it's easy for the user to implement whatever
> policy he fancies…
> 
> Oh, maybe I should finish preparing those patches for Event Triggers to
> be fully usable in 9.4 then ;)

I remind you that event triggers are not fired for global objects
such as databases and roles.  Do you intend to lift that restriction?

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Alvaro Herrera <alvherre@2ndquadrant.com> — 2013-08-02T17:34:58Z

Stephen Frost escribió:
> * Tom Lane (tgl@sss.pgh.pa.us) wrote:
> > What if you set a combination of parameters that prevents Postgres from
> > starting?
> 
> This was what I was trying to get at up-thread.  Things that prevent PG
> from being able to start (or, really, which cause PG to be started in a
> completely different mode, ala recovery.conf) need to be able to be
> modified outside of PG and therefore should, imv, be considered
> configuration parameters and therefore live outside of $PGDATA (when
> installed from a distro, blah, blah).

I think the way out of this situation is to have a postmaster and/or
pg_ctl switch that disables reading of ALTER SYSTEM settings.  Then the
DBA can do ALTER SYSTEM RESET until it works again.  No need to mess
with "data" files.

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Josh Berkus <josh@agliodbs.com> — 2013-08-02T17:55:35Z

On 08/02/2013 07:54 AM, Stephen Frost wrote:
> Curiously, I've not heard any argument about what parameters are "safe"
> and what aren't, though I was asked which ones I thought were safe and
> which weren't.  Perhaps looking at the specific options that would
> likely cause PG to not start would be useful to this discussion.

I really think this is the wrong approach.  If we start removing
"unsafe" parameters from ALTER SYSTEM SET, we basically hobble the
feature to the point of uselessness.  Out of the 15 or so parameters 80%
of our users touch, half of them are on your "unsafe" list.

A much simpler solution to the issue Stephen proposes is to have a way
to start up the server with all settings from ALTER SYSTEM SET disabled,
just like some software allows you to start it up in "safe mode".

Of course, automatically disabling the *individual* parameters would be
even better from a usability perspective.  This would be equally useful
for a manually-written postgresql.conf, i.e.:

"PostgreSQL is unable to start because we couldn't allocate all of the
memory you asked for.  Starting up with shared_buffers set to 32MB.".

... However, I'm not confident that we'll always be able to do that.
We'd also need to have a way to "kick and scream" so that sysadmins
would actually SEE it when the system disables a parameter.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-02T18:14:25Z

* Josh Berkus (josh@agliodbs.com) wrote:
> On 08/02/2013 07:54 AM, Stephen Frost wrote:
> > Curiously, I've not heard any argument about what parameters are "safe"
> > and what aren't, though I was asked which ones I thought were safe and
> > which weren't.  Perhaps looking at the specific options that would
> > likely cause PG to not start would be useful to this discussion.
> 
> I really think this is the wrong approach.  If we start removing
> "unsafe" parameters from ALTER SYSTEM SET, we basically hobble the
> feature to the point of uselessness.  Out of the 15 or so parameters 80%
> of our users touch, half of them are on your "unsafe" list.

They're on the 'unsafe' list because they're likely to *break* things.

I'm not at *all* surprised that the list comprises 80% of the parameters
that people actually modify/use, but that doesn't mean it's smart to let
them hack at those parameters remotely w/ no access to the host system
to fix things if they screw it up, which is really the case that I'm
thinking about here because it's what we're enabling through this
mechanism.  If the users already had access to the host system to go
modify the parameters in the config on the filesystem, they'd be likely
to just go *do* that, no?

> A much simpler solution to the issue Stephen proposes is to have a way
> to start up the server with all settings from ALTER SYSTEM SET disabled,
> just like some software allows you to start it up in "safe mode".

See above for why I'm not thrilled wih this approach, unless it was set
up to happen automatically, but you couldn't simply ignore *all* the
ALTER SYSTEM SET parameters because then you might not be able to
connect in due to some ALTER SYSTEM SET parameter being necessary for
remote connectivity or authentication.

> Of course, automatically disabling the *individual* parameters would be
> even better from a usability perspective.  This would be equally useful
> for a manually-written postgresql.conf, i.e.:
> 
> "PostgreSQL is unable to start because we couldn't allocate all of the
> memory you asked for.  Starting up with shared_buffers set to 32MB.".

Right; this is part of what I was getting at with the list- there are
definitely items on that list that we could start up without, were they
misconfigured.  The one question remaining from *that* however is if
there would be such a security impact from the config change that it
wouldn't be *safe* for us to do so (consider if we used our 'default'
pg_hba.conf, with 'trust' all over the place, in the event that we
couldn't load the system pg_hba.conf..).

> ... However, I'm not confident that we'll always be able to do that.
> We'd also need to have a way to "kick and scream" so that sysadmins
> would actually SEE it when the system disables a parameter.

Yes, this is also true.  When it comes to a 'safe mode', my original
inclination was to only allow connections over a unix socket from a user
running as the same Unix UID as the database is running as...  That's
not a solution for the Windows folks though, I'm afraid. :/

	Thanks,

		Stephen

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Tom Lane <tgl@sss.pgh.pa.us> — 2013-08-02T19:23:29Z

Stephen Frost <sfrost@snowman.net> writes:
> * Josh Berkus (josh@agliodbs.com) wrote:
>> A much simpler solution to the issue Stephen proposes is to have a way
>> to start up the server with all settings from ALTER SYSTEM SET disabled,
>> just like some software allows you to start it up in "safe mode".

> See above for why I'm not thrilled wih this approach, unless it was set
> up to happen automatically, but you couldn't simply ignore *all* the
> ALTER SYSTEM SET parameters because then you might not be able to
> connect in due to some ALTER SYSTEM SET parameter being necessary for
> remote connectivity or authentication.

Yeah, this approach is a nonstarter because there's no reason to assume
that a postmaster started with default parameters will start successfully,
or will be connectable-to if it does start.  Maybe there's another
postmaster hogging the default port, for instance.

			regards, tom lane

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-02T20:48:24Z

* Josh Berkus (josh@agliodbs.com) wrote:
> I really think this is the wrong approach.  If we start removing
> "unsafe" parameters from ALTER SYSTEM SET, we basically hobble the
> feature to the point of uselessness.  Out of the 15 or so parameters 80%
> of our users touch, half of them are on your "unsafe" list.

Reflecting on this a bit more, I'm curious what your list-of-15 is.

Many of the items on my list were file locations or other things which
generally require coordination with other groups (like the unix admins
or network admins) to change, eg: listen_addresses, port, SSL or
Kerberos file locations, etc.

There's quite a few parameters which I've changed that are "safe" and
internal-to-PG which weren't on my list- work_mem, maint_work_mem,
vacuum / autovacuum settings, effective_io_concurrency, wal_level,
sync_commit, checkpoint settings, max_wal_senders, planner costs,
logging settings (though this might have to be coordinated w/ the unix
admins unless splunk or similar is being used), etc.

	Thanks,

		Stephen

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Greg Smith <greg@2ndquadrant.com> — 2013-08-03T06:54:25Z

On 8/2/13 4:48 PM, Stephen Frost wrote:
> * Josh Berkus (josh@agliodbs.com) wrote:
>> I really think this is the wrong approach.  If we start removing
>> "unsafe" parameters from ALTER SYSTEM SET, we basically hobble the
>> feature to the point of uselessness.  Out of the 15 or so parameters 80%
>> of our users touch, half of them are on your "unsafe" list.
>
> Reflecting on this a bit more, I'm curious what your list-of-15 is.

You can get a top 15-ish list from 
https://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server

The list of things I see changed regularly that are on your unsafe list are:

listen_addresses, port, shared_buffers, log_directory, log_filename

Obvious missing thing from your unsafe list that is also changed 
regularly is max_connection.  I count 6 parameters that are both unsafe 
and changed regularly.

-- 
Greg Smith   2ndQuadrant US    greg@2ndQuadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Dimitri Fontaine <dimitri@2ndquadrant.fr> — 2013-08-03T15:04:40Z

Greg Stark <stark@mit.edu> writes:
> Writing out each guc in a separate file is a singularly bad idea. It's

I'm not buying into any of your arguments here, and have something to
add to that part:

> I'm not even clear we do want this in /etc since none of our GUC
> options are repeatable things like Apache virtual servers. It actually
> makes *more* sense for pg_hba than it does for gucs. I think we can
> assume that in the future we'll have something like it however.

Given a conf.d option in /etc it's then quite easy to add per-extension
configuration files in the packaging system, so that users don't have to
edit postgresql.conf for default values.

We still need some kind of repeatable settings that we don't have yet
for that to happen: cumulative setting of a "list" GUC such as
local_preload_libraries and suchlike.

Regards,
-- 
Dimitri Fontaine
http://2ndQuadrant.fr     PostgreSQL : Expertise, Formation et Support

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Dimitri Fontaine <dimitri@2ndquadrant.fr> — 2013-08-04T20:26:52Z

Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> I remind you that event triggers are not fired for global objects
> such as databases and roles.  Do you intend to lift that restriction?

That's not on my TODO list for 9.4. My understanding about implementing
that is:

  - we agree that it would be nice to have,
  - it requires a separate *shared* catalog for event triggers.

What I'm yet unsure about is that there's a consensus that the use cases
are worthy of a new shared catalog in the system. Also I didn't look how
hard it is to actually provide for it.

Regards,
-- 
Dimitri Fontaine
http://2ndQuadrant.fr     PostgreSQL : Expertise, Formation et Support

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila@huawei.com> — 2013-08-05T06:14:22Z

On Saturday, August 03, 2013 12:53 AM Tom Lane wrote:
> Stephen Frost <sfrost@snowman.net> writes:
> > * Josh Berkus (josh@agliodbs.com) wrote:
> >> A much simpler solution to the issue Stephen proposes is to have a
> way
> >> to start up the server with all settings from ALTER SYSTEM SET
> disabled,
> >> just like some software allows you to start it up in "safe mode".
> 
> > See above for why I'm not thrilled wih this approach, unless it was
> set
> > up to happen automatically, but you couldn't simply ignore *all* the
> > ALTER SYSTEM SET parameters because then you might not be able to
> > connect in due to some ALTER SYSTEM SET parameter being necessary for
> > remote connectivity or authentication.
> 
> Yeah, this approach is a nonstarter because there's no reason to assume
> that a postmaster started with default parameters will start
> successfully,
> or will be connectable-to if it does start.  Maybe there's another
> postmaster hogging the default port, for instance.

Okay, but user will always have option to start server with different value
of parameter (pg_ctl -o "-p 5434").

Now as a summarization we have below ways to move forward:

1. Provide a way for user to start server if not able to start due to
in-appropriate value of unsafe parameter
   a. already user has an option that he can mention value of any particular
parameter with which sever can start
   b. keep one backup copy of parameters, so that user can option to start
with that copy, else if that also doesn't work he
      can use point 'a'.

2. Don't allow unsafe parameters to be modified by ALTER SYSTEM
   a. List of un-safe parameters
   b. mechanism so that ALTER SYSTEM throws error for non-modifiable
parameters
   c. user can view non-modifiable parameters (may be in pg_settings)
   d. some way such that if user wants to take risk of server not getting
started, he should allow to modify such parameters.
      may be server is started with some specific option. This can reduce
the fear Josh had regarding this command to be not of much use.

I think if we choose Option-2, then one of the initial difficulty will be to
get an agreement on list of un-safe parameters.
I believe even if we want to go with Option-2, then in first cut the work
should be minimized. 

With Regards,
Amit Kapila.

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Tom Lane <tgl@sss.pgh.pa.us> — 2013-08-05T06:27:22Z

Amit Kapila <amit.kapila@huawei.com> writes:
> On Saturday, August 03, 2013 12:53 AM Tom Lane wrote:
>> Yeah, this approach is a nonstarter because there's no reason to assume
>> that a postmaster started with default parameters will start
>> successfully,
>> or will be connectable-to if it does start.  Maybe there's another
>> postmaster hogging the default port, for instance.

> Okay, but user will always have option to start server with different value
> of parameter (pg_ctl -o "-p 5434").

You're assuming that the user starts the postmaster manually.  On most
modern installations there's a few layers of scripting in there, which
might not be that easy to hack to add some command-line parameters,
even assuming that the DBA has sufficient wits about him to think of
this solution.  (When your postmaster unexpectedly fails to restart
at four in the morning, having to think of such an approach isn't what
you want to be doing.)

My point here is just that we should keep the parameter values in plain
text files, so that one possible solution is reverting a bogus change with
vi/emacs/your-weapon-of-choice.  If we "improve" matters so that the only
possible way to fix the parameter setting is via a running postmaster,
we've narrowed the number of escape routes that a frantic DBA will have.
And what would we have bought for that?  Not much.

			regards, tom lane

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila@huawei.com> — 2013-08-05T09:28:49Z

On Monday, August 05, 2013 11:57 AM Tom Lane wrote:
> Amit Kapila <amit.kapila@huawei.com> writes:
> > On Saturday, August 03, 2013 12:53 AM Tom Lane wrote:
> >> Yeah, this approach is a nonstarter because there's no reason to
> assume
> >> that a postmaster started with default parameters will start
> >> successfully,
> >> or will be connectable-to if it does start.  Maybe there's another
> >> postmaster hogging the default port, for instance.
> 
> > Okay, but user will always have option to start server with different
> value
> > of parameter (pg_ctl -o "-p 5434").
> 
> You're assuming that the user starts the postmaster manually.  On most
> modern installations there's a few layers of scripting in there, which
> might not be that easy to hack to add some command-line parameters,
> even assuming that the DBA has sufficient wits about him to think of
> this solution.  (When your postmaster unexpectedly fails to restart
> at four in the morning, having to think of such an approach isn't what
> you want to be doing.)
> 
> My point here is just that we should keep the parameter values in plain
> text files, 

Here by text files, do you mean to say you are expecting
file-per-guc-setting?

> so that one possible solution is reverting a bogus change
> with
> vi/emacs/your-weapon-of-choice.  If we "improve" matters so that the
> only
> possible way to fix the parameter setting is via a running postmaster,
> we've narrowed the number of escape routes that a frantic DBA will
> have.
> And what would we have bought for that?  Not much.

Although it is not advisable to edit this file manually, but I think in such
situations (postmaster doesn't start up due to inappropriate parameter
value) it can help user to come out of situation much easily.

My only point was to address the concerns regarding un-safe parameter
values.

With Regards,
Amit Kapila.

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Tom Lane <tgl@sss.pgh.pa.us> — 2013-08-05T12:18:33Z

Amit Kapila <amit.kapila@huawei.com> writes:
> On Monday, August 05, 2013 11:57 AM Tom Lane wrote:
>> My point here is just that we should keep the parameter values in plain
>> text files, 

> Here by text files, do you mean to say you are expecting
> file-per-guc-setting?

No, I don't think this argument really says much one way or the other
about one-file vs one-file-per-GUC.  I'm just lobbying against the
"put them in a catalog" idea.

			regards, tom lane

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Greg Stark <stark@mit.edu> — 2013-08-05T13:51:56Z

On Fri, Aug 2, 2013 at 4:05 PM, Stephen Frost <sfrost@snowman.net> wrote:
>> I'm not even clear we do want this in /etc since none of our GUC
>> options are repeatable things like Apache virtual servers. It actually
>> makes *more* sense for pg_hba than it does for gucs. I think we can
>> assume that in the future we'll have something like it however.
>
> I tend to agree with this also, though I can imagine wanting to separate
> things in a conf.d directory ala exim's conf.d directories, to allow
> tools like puppet to manage certain things environment-wide (perhaps
> krb_server_keyfile) while other configuration options are managed
> locally.

Extensions are actually a pretty good argument for why conf.d in /etc
(or wherever the non-auto-config is) is pretty important useful.
That's the kind of thing conf.d directories are meant for. A user can
install a package containing an extension and the extension would
automatically drop in the config entries needed in that directory.

-- 
greg

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-05T14:00:39Z

* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> Amit Kapila <amit.kapila@huawei.com> writes:
> > On Monday, August 05, 2013 11:57 AM Tom Lane wrote:
> >> My point here is just that we should keep the parameter values in plain
> >> text files, 
> 
> > Here by text files, do you mean to say you are expecting
> > file-per-guc-setting?
> 
> No, I don't think this argument really says much one way or the other
> about one-file vs one-file-per-GUC.  I'm just lobbying against the
> "put them in a catalog" idea.

imv there are things which should be in a catalog and things which
shouldn't.  I'm not sure that "can the postmaster start up if its wrong"
is necessairly the right test to use but the results tend to line up
with my expectations.  postgresql.conf seems to have a number of items
which don't even make a lot of sense to set server-side: enable_* being
a good example.  I know it *works* to set those in postgresql.conf, but
I dislike using postgresql.conf as documentation for what options exist.

I keep thinking that postgresql.conf should really be those items that
define how PG interacts with the OS (which tends to also line up with
"things the postmaster needs correct to start") and the rest of the
options should go somewhere else, perhaps as things which can be
configured through this ALTER SYSTEM option, which starts to sound more
like "default GUC values" for those items which take effect on
individual backend startup.

I've some more thoughts on this and will try to formulate them later
today.

	Thanks,

		Stephen

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-05T14:01:46Z

* Greg Stark (stark@mit.edu) wrote:
> On Fri, Aug 2, 2013 at 4:05 PM, Stephen Frost <sfrost@snowman.net> wrote:
> >> I'm not even clear we do want this in /etc since none of our GUC
> >> options are repeatable things like Apache virtual servers. It actually
> >> makes *more* sense for pg_hba than it does for gucs. I think we can
> >> assume that in the future we'll have something like it however.
> >
> > I tend to agree with this also, though I can imagine wanting to separate
> > things in a conf.d directory ala exim's conf.d directories, to allow
> > tools like puppet to manage certain things environment-wide (perhaps
> > krb_server_keyfile) while other configuration options are managed
> > locally.
> 
> Extensions are actually a pretty good argument for why conf.d in /etc
> (or wherever the non-auto-config is) is pretty important useful.
> That's the kind of thing conf.d directories are meant for. A user can
> install a package containing an extension and the extension would
> automatically drop in the config entries needed in that directory.

Agreed, though I think there should be a difference between "shared
library load" being added-to for extensions, and "random
extension-specific GUC"..

	Thanks,

		Stephen

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Bruce Momjian <bruce@momjian.us> — 2013-08-05T16:18:25Z

On Thu, Aug  1, 2013 at 03:40:22PM +0100, Greg Stark wrote:
> Why isn't it enough to just dump out all variables with a source of alter
> system to a text file? You can either have a single global lock around that
> operation or write it to a new file and move it into place.
> 
> --
> greg
> 
> On 1 Aug 2013 15:19, "Andres Freund" <andres@2ndquadrant.com> wrote:
> 
>     On 2013-08-01 15:17:04 +0100, Greg Stark wrote:
>     > We don't need per guc locking. This is the whole objection Tom had about
>     > this patch being more complex than it has to be.
> 
>     IIRC he objected to using locking *at all* because a simple
>     one-file-per-setting approach should be used.

I am unclear why we don't need a lock around _each_ GUC, i.e. if two
sessions try to modify the same GUC at the same time.  And if we need a
lock, seems we can have just one and write all the settings to one file
--- it is not like we have trouble doing locking, though this is
cluster-wide locking.

How would users handle renamed GUC variables, as we have done in the
past?  Would pg_dumpall dump the settings out?  Would unrecognized
settings throw an error, causing pg_upgrade to fail?

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + It's impossible for everything to be true. +

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Josh Berkus <josh@agliodbs.com> — 2013-08-05T16:41:52Z

All,

To limit the argument here, let's please not argue about things which
people already agree on.  So:

We seem to have consensus around:

A) the inadvisability of storing GUCs in a system catalog.

B) the utility of a conf.d in /etc/ which may have nothing to do with
ALTER SYSTEM SET

C) that any file for ALTER SYSTEM SET go in $PGDATA somewhere.

What we are still arguing about:

D) one-big-file vs. file-per-setting

E) whether "unsafe" settings or "restart" settings should be allowed in
ALTER SYSTEM SET.

F) whether admins need the ability to disable ALTER SYSTEM SET.

Since each of D, E and F issues are completely orthagonal to each other,
I suggest that maybe we argue them each out on their own threads?  I'll
start.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Andres Freund <andres@2ndquadrant.com> — 2013-08-05T16:46:11Z

On 2013-08-05 12:18:25 -0400, Bruce Momjian wrote:
> On Thu, Aug  1, 2013 at 03:40:22PM +0100, Greg Stark wrote:
> > Why isn't it enough to just dump out all variables with a source of alter
> > system to a text file? You can either have a single global lock around that
> > operation or write it to a new file and move it into place.
> > 
> > --
> > greg
> > 
> > On 1 Aug 2013 15:19, "Andres Freund" <andres@2ndquadrant.com> wrote:
> > 
> >     On 2013-08-01 15:17:04 +0100, Greg Stark wrote:
> >     > We don't need per guc locking. This is the whole objection Tom had about
> >     > this patch being more complex than it has to be.
> > 
> >     IIRC he objected to using locking *at all* because a simple
> >     one-file-per-setting approach should be used.
> 
> I am unclear why we don't need a lock around _each_ GUC, i.e. if two
> sessions try to modify the same GUC at the same time.  And if we need a
> lock, seems we can have just one and write all the settings to one file
> --- it is not like we have trouble doing locking, though this is
> cluster-wide locking.

If you have two sessions modifying the same variable, one is going to
win and overwrite the other's setting with or without locking around
GUCs unless you error out if somebody else holds the lock.

Greetings,

Andres Freund

-- 
 Andres Freund	                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Tom Lane <tgl@sss.pgh.pa.us> — 2013-08-05T16:53:24Z

Andres Freund <andres@2ndquadrant.com> writes:
> On 2013-08-05 12:18:25 -0400, Bruce Momjian wrote:
>> I am unclear why we don't need a lock around _each_ GUC, i.e. if two
>> sessions try to modify the same GUC at the same time.  And if we need a
>> lock, seems we can have just one and write all the settings to one file
>> --- it is not like we have trouble doing locking, though this is
>> cluster-wide locking.

> If you have two sessions modifying the same variable, one is going to
> win and overwrite the other's setting with or without locking around
> GUCs unless you error out if somebody else holds the lock.

The point of a lock is just to ensure that the end result is one valid
state or the other, and not something corrupt.  We would certainly need a
lock if we write to a single file.  With file-per-GUC, we could possibly
dispense with a lock if we depend on atomic file rename(); though whether
it's wise to assume that for Windows is unclear.  (Note that we ought to
write a temp file and rename it into place anyway, to avoid possibly
corrupting the existing file on out-of-disk-space.  The only thing that
needs discussion is whether to add an explicit lock around that.)

			regards, tom lane

Re: Unsafe GUCs and ALTER SYSTEM WAS: Re: ALTER SYSTEM SET

Josh Berkus <josh@agliodbs.com> — 2013-08-05T17:01:00Z

Stephen, all:

(forked from main ALTER SYSTEM discussion.  this thread is meant to
discuss only this question:

E) whether "unsafe" settings or "restart" settings should be allowed in
ALTER SYSTEM SET.)

On 08/02/2013 01:48 PM, Stephen Frost wrote:
> Reflecting on this a bit more, I'm curious what your list-of-15 is.

Based on serving literally hundreds of clients, the below are the
settings we change on client servers 50% or more of the time.  Other
settings I touch maybe 10% of the time.  THese are also, in general, the
settings which I modify when we create Puppet/Chef/Salt scripts for clients.

listen_addresses*@
shared_buffers*@
work_mem
maintenance_work_mem
effective_cache_size
synchronous_commit (because of amazon)
wal_buffers*@
checkpoint_segments*@
checkpoint_completion_target* (because of ext3)
autovacuum* (turn off for data warehouses,
             turn back on for very mistaken users)
stats_file_directory*@

replication/archiving settings as a set*@
wal_level, max_wal_senders, wal_keep_segments, hot_standby, archive_mode
and archive_command

logging settings as a set
logging_collector*
everything else

* = requires a cold start to change
@ potentially can cause failure to restart

Note that two of the settings, shared_buffers and wal_buffers, become
much less of an issue for restarting the system in 9.3.  Also, it's
possible that Heikki's automated WAL log management might deal with
out-of-disk-space better than currently, which makes that less of a risk.

However, you'll see that among the 11 core settings, 7 require a full
restart, and 5 could "potentially cause failure to restart".  That means
that from my perspective, ALTER SYSTEM SET is at least 45% useless if it
can't touch unsafe settngs, and 63% useless if it can't touch any
setting which requires a restart.  Adding the replication settings into
things makes stuff significantly worse that way, although ALTER SYSTEM
SET would be very useful for logging options provided that
logging_collector was turned on.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Bruce Momjian <bruce@momjian.us> — 2013-08-05T17:05:03Z

On Mon, Aug  5, 2013 at 12:53:24PM -0400, Tom Lane wrote:
> Andres Freund <andres@2ndquadrant.com> writes:
> > On 2013-08-05 12:18:25 -0400, Bruce Momjian wrote:
> >> I am unclear why we don't need a lock around _each_ GUC, i.e. if two
> >> sessions try to modify the same GUC at the same time.  And if we need a
> >> lock, seems we can have just one and write all the settings to one file
> >> --- it is not like we have trouble doing locking, though this is
> >> cluster-wide locking.
> 
> > If you have two sessions modifying the same variable, one is going to
> > win and overwrite the other's setting with or without locking around
> > GUCs unless you error out if somebody else holds the lock.
> 
> The point of a lock is just to ensure that the end result is one valid
> state or the other, and not something corrupt.  We would certainly need a
> lock if we write to a single file.  With file-per-GUC, we could possibly
> dispense with a lock if we depend on atomic file rename(); though whether
> it's wise to assume that for Windows is unclear.  (Note that we ought to
> write a temp file and rename it into place anyway, to avoid possibly
> corrupting the existing file on out-of-disk-space.  The only thing that
> needs discussion is whether to add an explicit lock around that.)

So my larger question is why a single-guc-per-file avoids corruption
while having all the gucs in a single file does not.  It seems the later
reduces the probability of lost updates, but does not eliminate it.

Also, should I be concerned that everyone removed my pg_upgrade question
in their replies.  :-O

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + It's impossible for everything to be true. +

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Andres Freund <andres@2ndquadrant.com> — 2013-08-05T17:08:12Z

On 2013-08-05 12:53:24 -0400, Tom Lane wrote:
> Andres Freund <andres@2ndquadrant.com> writes:
> > On 2013-08-05 12:18:25 -0400, Bruce Momjian wrote:
> >> I am unclear why we don't need a lock around _each_ GUC, i.e. if two
> >> sessions try to modify the same GUC at the same time.  And if we need a
> >> lock, seems we can have just one and write all the settings to one file
> >> --- it is not like we have trouble doing locking, though this is
> >> cluster-wide locking.
> 
> > If you have two sessions modifying the same variable, one is going to
> > win and overwrite the other's setting with or without locking around
> > GUCs unless you error out if somebody else holds the lock.
> 
> The point of a lock is just to ensure that the end result is one valid
> state or the other, and not something corrupt.

Sure. That's what I tried to explain. I've completely missed to mention
in this mail why the safe rename dance is sufficient to guarantee a
correct file with one-file-per-guc...

>  We would certainly need a
> lock if we write to a single file.  With file-per-GUC, we could possibly
> dispense with a lock if we depend on atomic file rename(); though whether
> it's wise to assume that for Windows is unclear.

Afaik it should be safe on anything NT based via Replacefile. Anything
else/earlier isn't supported anyways... I think there's already a custom
rename() implementation on windows?
Don't we already rely on atomic renames working for the control file?

> (Note that we ought to
> write a temp file and rename it into place anyway, to avoid possibly
> corrupting the existing file on out-of-disk-space.  The only thing that
> needs discussion is whether to add an explicit lock around that.)

Yes, agreed. All versions of the patch that I've read have done so luckily.

Greetings,

Andres Freund

-- 
 Andres Freund	                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Tom Lane <tgl@sss.pgh.pa.us> — 2013-08-05T17:16:10Z

Bruce Momjian <bruce@momjian.us> writes:
> So my larger question is why a single-guc-per-file avoids corruption
> while having all the gucs in a single file does not.

If it's file-per-GUC, then when two sessions try to write different GUCs,
there is no conflict.  When they try to write the same GUC, the end result
will be one value or the other (assuming use of atomic rename).  Which
seems fine.

If it's single-file, and we don't lock, then when two sessions try to
write different GUCs, one's update can be lost altogether, because
whichever one renames second didn't see the first one's update.  That
doesn't seem acceptable.

			regards, tom lane

Re: File-per-GUC WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Josh Berkus <josh@agliodbs.com> — 2013-08-05T17:21:56Z

(this discussion concerns issue (D), file-per-setting vs. one-big-file)

On 08/05/2013 10:16 AM, Tom Lane wrote:
> Bruce Momjian <bruce@momjian.us> writes:
>> So my larger question is why a single-guc-per-file avoids corruption
>> while having all the gucs in a single file does not.
> 
> If it's file-per-GUC, then when two sessions try to write different GUCs,
> there is no conflict.  When they try to write the same GUC, the end result
> will be one value or the other (assuming use of atomic rename).  Which
> seems fine.
> 
> If it's single-file, and we don't lock, then when two sessions try to
> write different GUCs, one's update can be lost altogether, because
> whichever one renames second didn't see the first one's update.  That
> doesn't seem acceptable.

I'll also point out that some of our settings only really "work" in
combinations of two or more settings.  For example, one doesn't want to
set archive_mode = on unless one is setting archive_command as well.
And generally if one sets sequential_page_cost, one is changing the
other cost parameters as well.  And logging parameters are generally
managed as a set.

So the case of two sessions both modifying ALTER SYSTEM SET, and one
succeeding for some-but-all-GUCS, and the other succeeding for
some-but-not-all-GUCs, would not be user-friendly or pretty, even if
each setting change succeeded or failed atomically.

Also, one of the reasons Amit went to one-big-file was the question of:
if each setting is changed independantly, how do we know when to send
the backend a reload()?  IIRC, anyway.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Bruce Momjian <bruce@momjian.us> — 2013-08-05T17:39:31Z

On Mon, Aug  5, 2013 at 01:16:10PM -0400, Tom Lane wrote:
> Bruce Momjian <bruce@momjian.us> writes:
> > So my larger question is why a single-guc-per-file avoids corruption
> > while having all the gucs in a single file does not.
> 
> If it's file-per-GUC, then when two sessions try to write different GUCs,
> there is no conflict.  When they try to write the same GUC, the end result
> will be one value or the other (assuming use of atomic rename).  Which
> seems fine.
> 
> If it's single-file, and we don't lock, then when two sessions try to
> write different GUCs, one's update can be lost altogether, because
> whichever one renames second didn't see the first one's update.  That
> doesn't seem acceptable.

Yes, I understand now --- with file-per-GUC, you can say one was later
than the other, but when changing two different GUCs, a single file
implementation doesn't have that logical clarity.

FYI, we will need to use create-rename even without the problem of
creating corrupted files because we need it to avoid backends reading
partially-written files.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + It's impossible for everything to be true. +

Re: File-per-GUC WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Bruce Momjian <bruce@momjian.us> — 2013-08-05T17:46:07Z

On Mon, Aug  5, 2013 at 10:21:56AM -0700, Josh Berkus wrote:
> (this discussion concerns issue (D), file-per-setting vs. one-big-file)
> 
> On 08/05/2013 10:16 AM, Tom Lane wrote:
> > Bruce Momjian <bruce@momjian.us> writes:
> >> So my larger question is why a single-guc-per-file avoids corruption
> >> while having all the gucs in a single file does not.
> > 
> > If it's file-per-GUC, then when two sessions try to write different GUCs,
> > there is no conflict.  When they try to write the same GUC, the end result
> > will be one value or the other (assuming use of atomic rename).  Which
> > seems fine.
> > 
> > If it's single-file, and we don't lock, then when two sessions try to
> > write different GUCs, one's update can be lost altogether, because
> > whichever one renames second didn't see the first one's update.  That
> > doesn't seem acceptable.
> 
> I'll also point out that some of our settings only really "work" in
> combinations of two or more settings.  For example, one doesn't want to
> set archive_mode = on unless one is setting archive_command as well.
> And generally if one sets sequential_page_cost, one is changing the
> other cost parameters as well.  And logging parameters are generally
> managed as a set.
> 
> So the case of two sessions both modifying ALTER SYSTEM SET, and one
> succeeding for some-but-all-GUCS, and the other succeeding for
> some-but-not-all-GUCs, would not be user-friendly or pretty, even if
> each setting change succeeded or failed atomically.

Wow, that is a good point.  I just mentioned in a previous email that
sessions are going to be created while this is going on and they can't
see partially-written files, so we need the create/rename dance.  One
new twist is that certain settings have to be _all_ set, or the backend
is going to throw an error.

> Also, one of the reasons Amit went to one-big-file was the question of:
> if each setting is changed independantly, how do we know when to send
> the backend a reload()?  IIRC, anyway.

Remember the backends starting up during this too.  A global lock is
looking unavoidable.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + It's impossible for everything to be true. +

Re: Unsafe GUCs and ALTER SYSTEM WAS: Re: ALTER SYSTEM SET

Stephen Frost <sfrost@snowman.net> — 2013-08-05T17:49:37Z

Josh,

* Josh Berkus (josh@agliodbs.com) wrote:
> Based on serving literally hundreds of clients, the below are the
> settings we change on client servers 50% or more of the time.  Other
> settings I touch maybe 10% of the time.  THese are also, in general, the
> settings which I modify when we create Puppet/Chef/Salt scripts for clients.

And that is actually half my point with all of this- these things are
often set by change management systems like puppet or chef and you can
be almost guaranteed that the folks running those pieces of the
infrastructure will be *very* unhappy if the DBA tries to adjust those
parameters independently because it'll almost certainly *not work*.  In
particular, that goes for things like listen_addresses and port, but
also shared_buffers, checkpoint_segments, wal_keep_segments, etc.

When it comes to things like the stats_file_directory, the admin will
need to be doing things on the host server anyway to set that up
(directories, permissions, etc), so it's unclear to me why they would
care to have the ability to modify these things through ALTER SYSTEM.

Also, to reiterate, what this patch is primairly adding, from my pov
anyway, is a way for DBAs to modify settings where they don't have
access to do so directly.  Today, you can prevent that from happening
while giving the DBA "superuser" (which is required generally because
we don't have sufficiently fine-grained control to avoid it) by making
postgresql.conf root-owned.  To be clear, I'm not suggesting that this
is some malicious act on the part of the DBA.

In the end, I'd prefer that we shrink up the config file to just those
items which really need to be in a config file, have a conf.d structure
which allows tools like puppet/chef to more easily break up and have
different components (eg: logging) configured specially for certain
classes of systems, and then move all of the internal-to-PG type of
config (enable_*, etc) into a catalog which is modified through an
'ALTER CLUSTER SET' type command, allowing DBAs to set parameters
across the entire cluster instead of just for a given database or role.

Josh, I really have to ask- are these people who are implementing puppet
to control these configs really clamoring to have an 'ALTER SYSTEM' PG
command to have to code against instead of dealing with text files?  I
feel like you're arguing for these parameters to be modifiable through
ALTER SYSTEM on the grounds that these parameters need to be set at some
point and in some way and not because having them set through ALTER
SYSTEM actually makes any *sense*.

	Thanks,

		Stephen

Re: File-per-GUC WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

David Johnston <polobo@yahoo.com> — 2013-08-05T17:52:52Z

Josh Berkus wrote
> (this discussion concerns issue (D), file-per-setting vs. one-big-file)
> 
> So the case of two sessions both modifying ALTER SYSTEM SET, and one
> succeeding for some-but-all-GUCS, and the other succeeding for
> some-but-not-all-GUCs, would not be user-friendly or pretty, even if
> each setting change succeeded or failed atomically.

Can the final file write occur only at "COMMIT;" with anything inside a
transaction simply staged up for later saving (or rollback).  The file write
phase as a whole then needs to be atomic and not just a single GUC-file.

Could the system read the last update timestamp of each GUC-file when the
original statement is executed and then re-read all of them at commit and
fail with some kind of serialization error if the last-update timestamp on
any of the files has changed?

I dislike the idea of any kind of automatic reload.  That said some kind of
"have their been any configuration changes since last reload?"
query/function makes sense.  In can be plugged into Nagios or similar to
warn if these changes are occurring but made live.

David J.




--
View this message in context: http://postgresql.1045698.n5.nabble.com/Proposal-for-Allow-postgresql-conf-values-to-be-changed-via-SQL-tp5729917p5766338.html
Sent from the PostgreSQL - hackers mailing list archive at Nabble.com.

Re: File-per-GUC WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Tom Lane <tgl@sss.pgh.pa.us> — 2013-08-05T17:56:40Z

Josh Berkus <josh@agliodbs.com> writes:
> I'll also point out that some of our settings only really "work" in
> combinations of two or more settings.  For example, one doesn't want to
> set archive_mode = on unless one is setting archive_command as well.
> And generally if one sets sequential_page_cost, one is changing the
> other cost parameters as well.  And logging parameters are generally
> managed as a set.

> So the case of two sessions both modifying ALTER SYSTEM SET, and one
> succeeding for some-but-all-GUCS, and the other succeeding for
> some-but-not-all-GUCs, would not be user-friendly or pretty, even if
> each setting change succeeded or failed atomically.

That is a killer point.  So really the value of the global lock is to
ensure serializability when transactions are updating multiple GUCs.

			regards, tom lane

Re: Re: File-per-GUC WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Bruce Momjian <bruce@momjian.us> — 2013-08-05T17:58:22Z

On Mon, Aug  5, 2013 at 10:52:52AM -0700, David Johnston wrote:
> Josh Berkus wrote
> > (this discussion concerns issue (D), file-per-setting vs. one-big-file)
> > 
> > So the case of two sessions both modifying ALTER SYSTEM SET, and one
> > succeeding for some-but-all-GUCS, and the other succeeding for
> > some-but-not-all-GUCs, would not be user-friendly or pretty, even if
> > each setting change succeeded or failed atomically.
> 
> Can the final file write occur only at "COMMIT;" with anything inside a
> transaction simply staged up for later saving (or rollback).  The file write
> phase as a whole then needs to be atomic and not just a single GUC-file.

Yes, the writes would only happen at commit, but I see of no way to have
the renames happen atomically.

> Could the system read the last update timestamp of each GUC-file when the
> original statement is executed and then re-read all of them at commit and
> fail with some kind of serialization error if the last-update timestamp on
> any of the files has changed?

That sounds like a mess --- clearly a global lock and a single file for
all GUCs would be perferrable.

> I dislike the idea of any kind of automatic reload.  That said some kind of
> "have their been any configuration changes since last reload?"
> query/function makes sense.  In can be plugged into Nagios or similar to
> warn if these changes are occurring but made live.

Who talked about automatic reload?  The bigger problem is new backends
starting and seeing partial state during the renames.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + It's impossible for everything to be true. +

Re: Unsafe GUCs and ALTER SYSTEM WAS: Re: ALTER SYSTEM SET

Josh Berkus <josh@agliodbs.com> — 2013-08-05T18:02:42Z

On 08/05/2013 10:49 AM, Stephen Frost wrote:
> Josh, I really have to ask- are these people who are implementing puppet
> to control these configs really clamoring to have an 'ALTER SYSTEM' PG
> command to have to code against instead of dealing with text files?  I
> feel like you're arguing for these parameters to be modifiable through
> ALTER SYSTEM on the grounds that these parameters need to be set at some
> point and in some way and not because having them set through ALTER
> SYSTEM actually makes any *sense*.

Nope.  ALTER SYSTEM, from my POV, is mainly for folks who *don't* use
Puppet/Chef/whatever.  Here's where I see ALTER SYSTEM being useful:

* invididually managed servers with out centralized management (i.e. one
DBA, one server).
* developer machines (i.e. laptops and vms)
* automated testing of tweaking performance parameters
* setting logging parameters temporarily on systems under centralized
management

For that reason, the only way in which I think it makes sense to try to
make ALTER SYSTEM set work together with Puppet/Chef is in the rather
limited context of modifying the logging settings for limited-time data
collection.  Mostly, ALTER SYSTEM SET is for systems were people
*aren't* using Puppet/Chef.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

Re: File-per-GUC WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-05T18:02:50Z

* Josh Berkus (josh@agliodbs.com) wrote:
> I'll also point out that some of our settings only really "work" in
> combinations of two or more settings.  For example, one doesn't want to
> set archive_mode = on unless one is setting archive_command as well.

And one doesn't want to set archive_command w/o doing some kind of setup
work on the server in question, perhaps setting up SSH keys or putting a
decent shell script in place to be called from archive_command.  That's
part of my issue w/ this- the parameters which can't be set up correctly
through sole use of "ALTER SYSTEM" should be excluded from it and
instead set up through a config file.

	Thanks,

		Stephen

Re: File-per-GUC WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Bruce Momjian <bruce@momjian.us> — 2013-08-05T18:04:42Z

On Mon, Aug  5, 2013 at 02:02:50PM -0400, Stephen Frost wrote:
> * Josh Berkus (josh@agliodbs.com) wrote:
> > I'll also point out that some of our settings only really "work" in
> > combinations of two or more settings.  For example, one doesn't want to
> > set archive_mode = on unless one is setting archive_command as well.
> 
> And one doesn't want to set archive_command w/o doing some kind of setup
> work on the server in question, perhaps setting up SSH keys or putting a
> decent shell script in place to be called from archive_command.  That's
> part of my issue w/ this- the parameters which can't be set up correctly
> through sole use of "ALTER SYSTEM" should be excluded from it and
> instead set up through a config file.

Yes, but many setting changes don't require file system changes.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + It's impossible for everything to be true. +

Re: File-per-GUC WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Josh Berkus <josh@agliodbs.com> — 2013-08-05T18:05:43Z

On 08/05/2013 11:02 AM, Stephen Frost wrote:
> * Josh Berkus (josh@agliodbs.com) wrote:
>> I'll also point out that some of our settings only really "work" in
>> combinations of two or more settings.  For example, one doesn't want to
>> set archive_mode = on unless one is setting archive_command as well.
> 
> And one doesn't want to set archive_command w/o doing some kind of setup
> work on the server in question, perhaps setting up SSH keys or putting a
> decent shell script in place to be called from archive_command.  That's
> part of my issue w/ this- the parameters which can't be set up correctly
> through sole use of "ALTER SYSTEM" should be excluded from it and
> instead set up through a config file.

I'm really trying to make progress on this discussion by separating it
into the distinct, orthagonal, arguments which need to be resolved
individually.  You are NOT helping by then intermixing the threads.  We
are already discussing that on "unsafe GUCs", please keep it there.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

Re: whichever ALTER SYSTEM thread is the right one

Stephen Frost <sfrost@snowman.net> — 2013-08-05T18:13:41Z

* Bruce Momjian (bruce@momjian.us) wrote:
> On Mon, Aug  5, 2013 at 02:02:50PM -0400, Stephen Frost wrote:
> > * Josh Berkus (josh@agliodbs.com) wrote:
> > > I'll also point out that some of our settings only really "work" in
> > > combinations of two or more settings.  For example, one doesn't want to
> > > set archive_mode = on unless one is setting archive_command as well.
> > 
> > And one doesn't want to set archive_command w/o doing some kind of setup
> > work on the server in question, perhaps setting up SSH keys or putting a
> > decent shell script in place to be called from archive_command.  That's
> > part of my issue w/ this- the parameters which can't be set up correctly
> > through sole use of "ALTER SYSTEM" should be excluded from it and
> > instead set up through a config file.
> 
> Yes, but many setting changes don't require file system changes.

Agreed, and those are the changes which would make sense to allow
through ALTER SYSTEM (or ALTER CLUSTER, or whatever)- settings which
don't require file system, network, etc, changes.

	Thanks,

		Stephen

Re: Unsafe GUCs and ALTER SYSTEM WAS: Re: ALTER SYSTEM SET

Stefan Kaltenbrunner <stefan@kaltenbrunner.cc> — 2013-08-05T18:14:45Z

On 08/05/2013 08:02 PM, Josh Berkus wrote:
> On 08/05/2013 10:49 AM, Stephen Frost wrote:
>> Josh, I really have to ask- are these people who are implementing puppet
>> to control these configs really clamoring to have an 'ALTER SYSTEM' PG
>> command to have to code against instead of dealing with text files?  I
>> feel like you're arguing for these parameters to be modifiable through
>> ALTER SYSTEM on the grounds that these parameters need to be set at some
>> point and in some way and not because having them set through ALTER
>> SYSTEM actually makes any *sense*.
> 
> Nope.  ALTER SYSTEM, from my POV, is mainly for folks who *don't* use
> Puppet/Chef/whatever.  Here's where I see ALTER SYSTEM being useful:
> 
> * invididually managed servers with out centralized management (i.e. one
> DBA, one server).
> * developer machines (i.e. laptops and vms)
> * automated testing of tweaking performance parameters
> * setting logging parameters temporarily on systems under centralized
> management

overridding the configuration system, that will just lead to very
confused sysadmins why something that was configurated now behaves
differently and I cause operational hazards because people _WILL_ forget
changing those "temporary only" settings back?

> 
> For that reason, the only way in which I think it makes sense to try to
> make ALTER SYSTEM set work together with Puppet/Chef is in the rather
> limited context of modifying the logging settings for limited-time data
> collection.  Mostly, ALTER SYSTEM SET is for systems were people
> *aren't* using Puppet/Chef.

I tend to disagree, the current approach of ALTER SYSTEM requiring
superuser basically means:

* in a few years from now people will just use superuser over the
network for almost all stuff "because its easy and I can click around in
$gui", having potential "unsafe" operations available over the network
will in turn cause a lot of actual downtime (in a lot of cases the
reason why people want remote management is because the don't have
physical/shell access - so if they break stuff they cannot fix)

* for classic IaaS/SaaS/DBaaS the ALTER SYSTEM seems to be mostly
useless in the current form - because most of them will not or cannot
hand out flat out superuser (like if you run a managed service you might
want customers to be able to tweak some stuff but say not
archive/pitr/replication stuff because the responsibility for backups is
with the hosting company)




Stefan

Re: Disabling ALTER SYSTEM SET WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters

Josh Berkus <josh@agliodbs.com> — 2013-08-05T18:16:24Z

All:

(this thread concerns argument (F) whether or not admins need a way to
disable ALTER SYSTEM SET)

On 08/01/2013 04:03 PM, Dimitri Fontaine wrote:> If only we could
trigger some actions when a command is about to be
> executed, in a way that it's easy for the user to implement whatever
> policy he fancies…
>
> Oh, maybe I should finish preparing those patches for Event Triggers to
> be fully usable in 9.4 then ;)

I don't see this as a solution at all.  "Mr. Sysadmin, we've given the
DBAs a new tool which allows them to override your version-controlled
database parameter settings.  You can turn it off, though, by using this
incredibly complicated, brand-new Event Trigger tool which requires
writing lots of SQL code to make work."

Per Stephen Frost's arguments, some system owners are going to be
opposed to allowing ALTER SYSTEM SET at all because it can mess systems
up and cause downtime.  Yes, that's already true of ALTER ROLE and ALTER
DATABASE, but ALTER SYSTEM SET expands this the ability of the DBA to
change setting substantially.  That's obviously its benefit, but it's
also clearly a misfeature for some system owners.  Also, to be blunt,
most DBAs/DEVs don't *know* about ALTER ROLE/DATABASE.

I don't think this is a small thing.  I really think we'll get a LOT of
blowback from sysadmins -- and maybe even refusals to upgrade -- if we
add ALTER SYSTEM SET in 9.4.0 with no easy way to disable it.  Having an
easy way to disable ALTER SYSTEM SET is *also* a good way to get out of
the whole situation of "I set shared_buffers to 100GB using ALTER SYSTEM
SET and now PostgreSQL won't start" problem.

As I've said before, I think the disable switch can be a follow-up patch
to the main ALTER SYSTEM SET patch.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

Re: Unsafe GUCs and ALTER SYSTEM WAS: Re: ALTER SYSTEM SET

Stefan Kaltenbrunner <stefan@kaltenbrunner.cc> — 2013-08-05T18:20:19Z

On 08/05/2013 07:01 PM, Josh Berkus wrote:
> Stephen, all:
> 
> (forked from main ALTER SYSTEM discussion.  this thread is meant to
> discuss only this question:
> 
> E) whether "unsafe" settings or "restart" settings should be allowed in
> ALTER SYSTEM SET.)
> 
> On 08/02/2013 01:48 PM, Stephen Frost wrote:
>> Reflecting on this a bit more, I'm curious what your list-of-15 is.
> 
> Based on serving literally hundreds of clients, the below are the
> settings we change on client servers 50% or more of the time.  Other
> settings I touch maybe 10% of the time.  THese are also, in general, the
> settings which I modify when we create Puppet/Chef/Salt scripts for clients.
> 
> listen_addresses*@
> shared_buffers*@
> work_mem
> maintenance_work_mem
> effective_cache_size
> synchronous_commit (because of amazon)
> wal_buffers*@
> checkpoint_segments*@
> checkpoint_completion_target* (because of ext3)
> autovacuum* (turn off for data warehouses,
>              turn back on for very mistaken users)
> stats_file_directory*@
> 
> replication/archiving settings as a set*@
> wal_level, max_wal_senders, wal_keep_segments, hot_standby, archive_mode
> and archive_command
> 
> logging settings as a set
> logging_collector*
> everything else
> 
> * = requires a cold start to change
> @ potentially can cause failure to restart
> 
> Note that two of the settings, shared_buffers and wal_buffers, become
> much less of an issue for restarting the system in 9.3.  Also, it's
> possible that Heikki's automated WAL log management might deal with
> out-of-disk-space better than currently, which makes that less of a risk.
> 
> However, you'll see that among the 11 core settings, 7 require a full
> restart, and 5 could "potentially cause failure to restart".  That means
> that from my perspective, ALTER SYSTEM SET is at least 45% useless if it
> can't touch unsafe settngs, and 63% useless if it can't touch any
> setting which requires a restart.  Adding the replication settings into
> things makes stuff significantly worse that way, although ALTER SYSTEM
> SET would be very useful for logging options provided that
> logging_collector was turned on.

not sure at all I agree with our "% useless" measure but we need to
consider that having all of those available over remote means they will
suddenly become "action at a distance" thingies, people will play with
them more and randomly change stuff, and a lot of those can break the
entire system because of say overrunning system resources. The same
thing can happen now just as well, but having them available from remote
will also result in tools doing this and people that have less
information about the hardware and system or what else is going on on
that box. Also we have to keep in mind that in most scenarios the
logfile and potentially reported errors/warnings there will be useless
because people will never see them...


Stefan

Re: Unsafe GUCs and ALTER SYSTEM WAS: Re: ALTER SYSTEM SET

Josh Berkus <josh@agliodbs.com> — 2013-08-05T18:21:57Z

On 08/05/2013 11:14 AM, Stefan Kaltenbrunner wrote:
> * in a few years from now people will just use superuser over the
> network for almost all stuff "because its easy and I can click around in
> $gui", having potential "unsafe" operations available over the network
> will in turn cause a lot of actual downtime (in a lot of cases the
> reason why people want remote management is because the don't have
> physical/shell access - so if they break stuff they cannot fix)

See thread "Disabling ALTER SYSTEM SET".

> * for classic IaaS/SaaS/DBaaS the ALTER SYSTEM seems to be mostly
> useless in the current form - because most of them will not or cannot
> hand out flat out superuser (like if you run a managed service you might
> want customers to be able to tweak some stuff but say not
> archive/pitr/replication stuff because the responsibility for backups is
> with the hosting company)

100% in agreement.  If someone thought we were serving DBAAS with this,
they haven't paid attention to the patch.

However, there are other places where ALTER SYSTEM SET will be valuable.
 For example, for anyone who wants to implement an autotuning utility.
For example, I'm writing a network utility which checks bgwriter stats
and tries adjusting settings over the network to improve checkpoint
issues.  Not having to SSH configuration files into place (and make sure
they're not overridden by other configuration files) would make writing
that script a *lot* easier.  Same thing with automated performance testing.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

Re: Disabling ALTER SYSTEM SET WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters

Bruce Momjian <bruce@momjian.us> — 2013-08-05T18:27:12Z

On Mon, Aug  5, 2013 at 11:16:24AM -0700, Josh Berkus wrote:
> I don't see this as a solution at all.  "Mr. Sysadmin, we've given the
> DBAs a new tool which allows them to override your version-controlled
> database parameter settings.  You can turn it off, though, by using this
> incredibly complicated, brand-new Event Trigger tool which requires
> writing lots of SQL code to make work."
> 
> Per Stephen Frost's arguments, some system owners are going to be
> opposed to allowing ALTER SYSTEM SET at all because it can mess systems
> up and cause downtime.  Yes, that's already true of ALTER ROLE and ALTER
> DATABASE, but ALTER SYSTEM SET expands this the ability of the DBA to
> change setting substantially.  That's obviously its benefit, but it's
> also clearly a misfeature for some system owners.  Also, to be blunt,
> most DBAs/DEVs don't *know* about ALTER ROLE/DATABASE.
> 
> I don't think this is a small thing.  I really think we'll get a LOT of
> blowback from sysadmins -- and maybe even refusals to upgrade -- if we
                             ----------------------------------

Really?  Is that a reasonable statement?

> add ALTER SYSTEM SET in 9.4.0 with no easy way to disable it.  Having an
> easy way to disable ALTER SYSTEM SET is *also* a good way to get out of
> the whole situation of "I set shared_buffers to 100GB using ALTER SYSTEM
> SET and now PostgreSQL won't start" problem.
> 
> As I've said before, I think the disable switch can be a follow-up patch
> to the main ALTER SYSTEM SET patch.

You are mixing the behaviors of disabling the ALTER SYSTEM SET command
with recognizing ALTER SYSTEM SET settings already made --- the later
would be to fix a problem with the server not starting.  However,
frankly, these are flat files, so I don't see a problem with having the
administrator modify the flat file.

Would disabling the ALTER SYSTEM SET command also disable recognizing
any ALTER SYSTEM SET commands already performed?  Maybe that was already
understood, but I missed that point.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + It's impossible for everything to be true. +

Re: Unsafe GUCs and ALTER SYSTEM WAS: Re: ALTER SYSTEM SET

Stephen Frost <sfrost@snowman.net> — 2013-08-05T18:28:44Z

* Josh Berkus (josh@agliodbs.com) wrote:
> Nope.  ALTER SYSTEM, from my POV, is mainly for folks who *don't* use
> Puppet/Chef/whatever.  

Ok, that's fine, but let's try to avoid making life difficult for those
who *do* use puppet/chef/whatever.  This capability runs a very high
risk of that by allowing a DBA who *isn't* a sysadmin to go modifying
things that depend on external-to-PG factors.

> Here's where I see ALTER SYSTEM being useful:
> 
> * invididually managed servers with out centralized management (i.e. one
> DBA, one server).
> * developer machines (i.e. laptops and vms)

The above strikes me as being already dealt with through pgAdmin and the
'admin pack', if the user wants a GUI to use for modifying these
parameters (which seems like what they'd primairly get out of ALTER
SYSTEM SET- pgAdmin, or whatever $gui wouldn't have to depend on the
admin pack).

> * automated testing of tweaking performance parameters

This sounds like you'd need tooling around it to make it work anyway,
which could probably handle modifying a text file, but even if not,
these paremeters may be on the 'safe' list.

> * setting logging parameters temporarily on systems under centralized
> management

This is the kind of argument that I could get behind- in an environment
where logs are shipped to a server where DBAs can view and analyze them,
being able to modify the logging parameters on the fly could be useful
(eg: log_min_duration_statement or similar).  As I tried to get at
up-thread, my concern is primairly around those parameters which can't
be set sensibly through ALTER SYSTEM because they depend on other
activities happening.

	Thanks,

		Stephen

Re: Unsafe GUCs and ALTER SYSTEM WAS: Re: ALTER SYSTEM SET

Stefan Kaltenbrunner <stefan@kaltenbrunner.cc> — 2013-08-05T18:30:11Z

On 08/05/2013 08:21 PM, Josh Berkus wrote:
> On 08/05/2013 11:14 AM, Stefan Kaltenbrunner wrote:
>> * in a few years from now people will just use superuser over the
>> network for almost all stuff "because its easy and I can click around in
>> $gui", having potential "unsafe" operations available over the network
>> will in turn cause a lot of actual downtime (in a lot of cases the
>> reason why people want remote management is because the don't have
>> physical/shell access - so if they break stuff they cannot fix)
> 
> See thread "Disabling ALTER SYSTEM SET".
> 
>> * for classic IaaS/SaaS/DBaaS the ALTER SYSTEM seems to be mostly
>> useless in the current form - because most of them will not or cannot
>> hand out flat out superuser (like if you run a managed service you might
>> want customers to be able to tweak some stuff but say not
>> archive/pitr/replication stuff because the responsibility for backups is
>> with the hosting company)
> 
> 100% in agreement.  If someone thought we were serving DBAAS with this,
> they haven't paid attention to the patch.
> 
> However, there are other places where ALTER SYSTEM SET will be valuable.
>  For example, for anyone who wants to implement an autotuning utility.
> For example, I'm writing a network utility which checks bgwriter stats
> and tries adjusting settings over the network to improve checkpoint
> issues.  Not having to SSH configuration files into place (and make sure
> they're not overridden by other configuration files) would make writing
> that script a *lot* easier.  Same thing with automated performance testing.

seems like an excessively narrow usecase to me - people doing that kind
of specific testing can easily do automation over ssh, and those are
very few vs. having to maintain a fairly complex piece of code in
postgresql core.
Nevertheless my main point is that people _WILL_ use this as a simple
convinience tool not fully understanding all the complex implications,
and in a few years from now running people with superuser by default
(because people will create "cool little tools say to change stuff from
my tray or using $IOS app" that have a little small comment "make sure
to create the user "WITH SUPERUSER" and people will follow like lemmings.


Stefan

Re: Unsafe GUCs and ALTER SYSTEM WAS: Re: ALTER SYSTEM SET

Josh Berkus <josh@agliodbs.com> — 2013-08-05T18:36:21Z

On 08/05/2013 11:28 AM, Stephen Frost wrote:
> * Josh Berkus (josh@agliodbs.com) wrote:
>> Nope.  ALTER SYSTEM, from my POV, is mainly for folks who *don't* use
>> Puppet/Chef/whatever.  
> 
> Ok, that's fine, but let's try to avoid making life difficult for those
> who *do* use puppet/chef/whatever.  This capability runs a very high
> risk of that by allowing a DBA who *isn't* a sysadmin to go modifying
> things that depend on external-to-PG factors.

See thread "Disabling ALTER SYSTEM SET".  In short, I agree with you.

> 
>> Here's where I see ALTER SYSTEM being useful:
>>
>> * invididually managed servers with out centralized management (i.e. one
>> DBA, one server).
>> * developer machines (i.e. laptops and vms)
> 
> The above strikes me as being already dealt with through pgAdmin and the
> 'admin pack', if the user wants a GUI to use for modifying these
> parameters (which seems like what they'd primairly get out of ALTER
> SYSTEM SET- pgAdmin, or whatever $gui wouldn't have to depend on the
> admin pack).

Except that forcing developers to install the admin pack and pgadmin to
get this functionality is a high barrier to entry exactly where we don't
want one.

> 
>> * automated testing of tweaking performance parameters
> 
> This sounds like you'd need tooling around it to make it work anyway,
> which could probably handle modifying a text file, but even if not,
> these paremeters may be on the 'safe' list.

Well, frankly, it's the main reason why *I* want ALTER SYSTEM SET.  It
makes my job writing automated testing scripts easier.  Certainly it was
possible before, but there's value in "easier".

And that's the reason I don't want you to take away the ability to
modify shared_buffers et. al.  ;-)

On 08/05/2013 11:30 AM, Stefan Kaltenbrunner wrote:> Nevertheless my
main point is that people _WILL_ use this as a simple
> convinience tool not fully understanding all the complex implications,
> and in a few years from now running people with superuser by default
> (because people will create "cool little tools say to change stuff from
> my tray or using $IOS app" that have a little small comment "make sure
> to create the user "WITH SUPERUSER" and people will follow like lemmings.

Most of our users not on Heroku are running with superuser as the app
user now.  Like, 95% of them based on my personal experience (because
our object permissions management sucks).  In that this feature will
further discourage people from having a separate application user,
there's some argument.  However, it's really an argument for not having
ALTER SYSTEM SET *at all* rather than restricting it to "safe" GUCs, no?

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

Re: Unsafe GUCs and ALTER SYSTEM WAS: Re: ALTER SYSTEM SET

Stephen Frost <sfrost@snowman.net> — 2013-08-05T18:44:28Z

* Josh Berkus (josh@agliodbs.com) wrote:
> On 08/05/2013 11:14 AM, Stefan Kaltenbrunner wrote:
> > * in a few years from now people will just use superuser over the
> > network for almost all stuff "because its easy and I can click around in
> > $gui", having potential "unsafe" operations available over the network
> > will in turn cause a lot of actual downtime (in a lot of cases the
> > reason why people want remote management is because the don't have
> > physical/shell access - so if they break stuff they cannot fix)
> 
> See thread "Disabling ALTER SYSTEM SET".

I'm really not thrilled with this "solution".

> However, there are other places where ALTER SYSTEM SET will be valuable.
>  For example, for anyone who wants to implement an autotuning utility.
> For example, I'm writing a network utility which checks bgwriter stats
> and tries adjusting settings over the network to improve checkpoint
> issues.  Not having to SSH configuration files into place (and make sure
> they're not overridden by other configuration files) would make writing
> that script a *lot* easier.  Same thing with automated performance testing.

I've done a fair bit of this myself and find templating postgresql.conf
isn't really all that hard and comes in handy for a lot things, not to
mention that it can then be integrated into a configuration management
system more easily..  Still, I'm fine w/ parameters which don't depend
on external things happening, which I think covers a lot of this
use-case.

	Thanks,

		Stephen

Re: Disabling ALTER SYSTEM SET WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters

Stephen Frost <sfrost@snowman.net> — 2013-08-05T18:52:40Z

* Bruce Momjian (bruce@momjian.us) wrote:
> On Mon, Aug  5, 2013 at 11:16:24AM -0700, Josh Berkus wrote:
> > I don't think this is a small thing.  I really think we'll get a LOT of
> > blowback from sysadmins -- and maybe even refusals to upgrade -- if we
>                              ----------------------------------
> 
> Really?  Is that a reasonable statement?

I don't believe Josh was saying that in jest..  I certainly believe that
it could happen.

> > As I've said before, I think the disable switch can be a follow-up patch
> > to the main ALTER SYSTEM SET patch.

Even if it's a different patch, it should go in at the same time, imv..

> You are mixing the behaviors of disabling the ALTER SYSTEM SET command
> with recognizing ALTER SYSTEM SET settings already made --- the later
> would be to fix a problem with the server not starting.  

That's a good point.

> However,
> frankly, these are flat files, so I don't see a problem with having the
> administrator modify the flat file.

Admins on Ubuntu or Debian or a host of their derivatives aren't going
to be looking in $PGDATA for config files that they have to hand-modify
to fix something the DBA did.  When they eventually figure it out,
they're going to be *very* unhappy.

> Would disabling the ALTER SYSTEM SET command also disable recognizing
> any ALTER SYSTEM SET commands already performed?  Maybe that was already
> understood, but I missed that point.

I don't have an answer to that one, though I do like the idea of a
switch that says "only read the settings from my postgresql.conf file".

		Thanks,

			Stephen

Re: Disabling ALTER SYSTEM SET WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters

Bruce Momjian <bruce@momjian.us> — 2013-08-05T19:02:24Z

On Mon, Aug  5, 2013 at 02:52:40PM -0400, Stephen Frost wrote:
> > However,
> > frankly, these are flat files, so I don't see a problem with having the
> > administrator modify the flat file.
> 
> Admins on Ubuntu or Debian or a host of their derivatives aren't going
> to be looking in $PGDATA for config files that they have to hand-modify
> to fix something the DBA did.  When they eventually figure it out,
> they're going to be *very* unhappy.

Well, can you assume that if you have a problem with one of your ALTER
SYSTEM SET commands, that disabling _all_ of them is going to get you a
running system?  I question that, e.g. port.  With postgresql.conf, you
can modify the bad entry, but how would that happen with ALTER SYSTEM
SET?

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + It's impossible for everything to be true. +

Re: Disabling ALTER SYSTEM SET WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters

Tom Lane <tgl@sss.pgh.pa.us> — 2013-08-05T19:38:34Z

Bruce Momjian <bruce@momjian.us> writes:
> Well, can you assume that if you have a problem with one of your ALTER
> SYSTEM SET commands, that disabling _all_ of them is going to get you a
> running system?  I question that, e.g. port.  With postgresql.conf, you
> can modify the bad entry, but how would that happen with ALTER SYSTEM
> SET?

I think we already have consensus that the settings will be in text files,
so that system breakage can be handled by editing the files.

What Josh seems to be concerned with in this thread is the question of
whether we should support an installation *policy decision* not to allow
ALTER SYSTEM SET.  Not because a particular set of parameters is broken,
but just because somebody is afraid the DBA might break things.  TBH
I'm not sure I buy that, at least not as long as ALTER SYSTEM is a
superuser feature.  There is nothing in Postgres that denies permissions
to superusers, and this doesn't seem like a very good place to start.

			regards, tom lane

Re: Disabling ALTER SYSTEM SET WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters

Alvaro Herrera <alvherre@2ndquadrant.com> — 2013-08-05T19:53:01Z

Tom Lane escribió:

> What Josh seems to be concerned with in this thread is the question of
> whether we should support an installation *policy decision* not to allow
> ALTER SYSTEM SET.  Not because a particular set of parameters is broken,
> but just because somebody is afraid the DBA might break things.  TBH
> I'm not sure I buy that, at least not as long as ALTER SYSTEM is a
> superuser feature.  There is nothing in Postgres that denies permissions
> to superusers, and this doesn't seem like a very good place to start.

Someone made an argument about this on IRC: GUI tool users are going to
want to use ALTER SYSTEM through point-and-click, and if all we offer is
superuser-level access to the feature, we're going to end up with a lot
of people running with superuser privileges just so that they are able
to tweak inconsequential settings.  This seems dangerous.

The other issue is that currently you can only edit a server's config if
you are logged in to it.  If we permit SQL-level access to that, and
somebody who doesn't have access to edit the files blocks themselves
out, there is no way for them to get a working system *at all*.

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Re: Disabling ALTER SYSTEM SET WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters

Dimitri Fontaine <dimitri@2ndquadrant.fr> — 2013-08-05T20:18:35Z

Hi,

I'm now completely lost in the current threads. Is there a single valid
use case for the feature we're trying to design? Who is the target
audience of this patch?

Josh Berkus <josh@agliodbs.com> writes:
> I don't see this as a solution at all.  "Mr. Sysadmin, we've given the
> DBAs a new tool which allows them to override your version-controlled
> database parameter settings.  You can turn it off, though, by using this
> incredibly complicated, brand-new Event Trigger tool which requires
> writing lots of SQL code to make work."

Well, given what has been said already and very recently again by Tom,
it's superuser only and installing a precedent wherein superuser is not
allowed to use a feature looks like a dead-end. You would have to make a
case that it's comparable to allow_system_table_mods.

If you believe that revoking ALTER SYSTEM SET privileges to superusers
isn't going to be accepted, I know of only two other paths to allow you
to implementing your own policy, including per-GUC policy and
non-superuser granting of ALTER SYSTEM SET in a controled fashion:

  - add per GUC GRANT/REVOKE capabilities to SETTINGs,
  - implement the same thing within an Event Trigger.

The former has been talked about lots of time already in the past and
I'm yet to see any kind of progress made about it despite plenty of user
support for the feature, the latter requires a shared catalog for global
object Event Triggers and maybe a flexible Extension that you can manage
by just dropping a configuration file into the PostgreSQL conf.d.

So when trying to be realistic the answer is "incredibly complicated"
because it involves a stored procedure to implement the local policy and
a command to enable the policy, really, I wonder who you're addressing
there. Certainly not DBA, so that must be sysadmins, who would be better
off without the feature in the first place if I'm understanding you.

Again, what are we trying to achieve?!

Regards,
-- 
Dimitri Fontaine
http://2ndQuadrant.fr     PostgreSQL : Expertise, Formation et Support

Re: Disabling ALTER SYSTEM SET WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters

Tom Lane <tgl@sss.pgh.pa.us> — 2013-08-05T20:24:23Z

Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> Tom Lane escribi:
>> What Josh seems to be concerned with in this thread is the question of
>> whether we should support an installation *policy decision* not to allow
>> ALTER SYSTEM SET.  Not because a particular set of parameters is broken,
>> but just because somebody is afraid the DBA might break things.  TBH
>> I'm not sure I buy that, at least not as long as ALTER SYSTEM is a
>> superuser feature.  There is nothing in Postgres that denies permissions
>> to superusers, and this doesn't seem like a very good place to start.

> Someone made an argument about this on IRC: GUI tool users are going to
> want to use ALTER SYSTEM through point-and-click, and if all we offer is
> superuser-level access to the feature, we're going to end up with a lot
> of people running with superuser privileges just so that they are able
> to tweak inconsequential settings.  This seems dangerous.

Agreed, but what else are you going to do?  You can't have random
unprivileged users changing settings that affect other users, even
if those settings are somehow "safe".  And what is more to the point
in this thread, having an additional shutoff that prevents even superusers
from doing it doesn't reduce the temptation for everyone to make
themselves superuser all the time.

> The other issue is that currently you can only edit a server's config if
> you are logged in to it.  If we permit SQL-level access to that, and
> somebody who doesn't have access to edit the files blocks themselves
> out, there is no way for them to get a working system *at all*.

True.  So, if they were smart enough to prevent themselves from changing
any settings remotely, they have no feature.  And if they weren't, having
such a blocking capability didn't really help them either.

These are both valid worries, but what design is going to make them
better?  Other than forgetting about ALTER SYSTEM entirely?

			regards, tom lane

Re: Disabling ALTER SYSTEM SET WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters

Rodrigo Gonzalez <rjgonzale.lists@gmail.com> — 2013-08-05T20:31:08Z

On Mon, 5 Aug 2013 15:53:01 -0400
Alvaro Herrera <alvherre@2ndquadrant.com> wrote:

> The other issue is that currently you can only edit a server's config
> if you are logged in to it.  If we permit SQL-level access to that,
> and somebody who doesn't have access to edit the files blocks
> themselves out, there is no way for them to get a working system *at
> all*.
> 

This is not a valid point, at least for me....basically all OSs should
disable any change to firewall if you are connected remotely cause you
can block yourself....giving power to users does not mean being their
babysitter....we should smart enough to use the power we receive...if
we are not smart...we should start thinking about other profession

And excuse my English, I hope the point is clear...

Best regards

Rodrigo Gonzalez

Re: Disabling ALTER SYSTEM SET WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters

Stefan Kaltenbrunner <stefan@kaltenbrunner.cc> — 2013-08-05T21:08:13Z

On 08/05/2013 10:18 PM, Dimitri Fontaine wrote:
> Hi,
> 
> I'm now completely lost in the current threads. Is there a single valid
> use case for the feature we're trying to design? Who is the target
> audience of this patch?

wonder about that myself...


> 
> Josh Berkus <josh@agliodbs.com> writes:
>> I don't see this as a solution at all.  "Mr. Sysadmin, we've given the
>> DBAs a new tool which allows them to override your version-controlled
>> database parameter settings.  You can turn it off, though, by using this
>> incredibly complicated, brand-new Event Trigger tool which requires
>> writing lots of SQL code to make work."
> 
> Well, given what has been said already and very recently again by Tom,
> it's superuser only and installing a precedent wherein superuser is not
> allowed to use a feature looks like a dead-end. You would have to make a
> case that it's comparable to allow_system_table_mods.
> 
> If you believe that revoking ALTER SYSTEM SET privileges to superusers
> isn't going to be accepted, I know of only two other paths to allow you
> to implementing your own policy, including per-GUC policy and
> non-superuser granting of ALTER SYSTEM SET in a controled fashion:
> 
>   - add per GUC GRANT/REVOKE capabilities to SETTINGs,

realistically I think this is what we "want"(fsvo) for this feature as a
prerequisite, however that also will make it fairly complex to use for
both humans and tools so not sure we would really gain anything...


>   - implement the same thing within an Event Trigger.
> 
> The former has been talked about lots of time already in the past and
> I'm yet to see any kind of progress made about it despite plenty of user
> support for the feature, the latter requires a shared catalog for global
> object Event Triggers and maybe a flexible Extension that you can manage
> by just dropping a configuration file into the PostgreSQL conf.d.
> 
> So when trying to be realistic the answer is "incredibly complicated"
> because it involves a stored procedure to implement the local policy and
> a command to enable the policy, really, I wonder who you're addressing
> there. Certainly not DBA, so that must be sysadmins, who would be better
> off without the feature in the first place if I'm understanding you.
> 
> Again, what are we trying to achieve?!

no idea - wondering about that myself...


Stefan

Re: Disabling ALTER SYSTEM SET WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters

Stefan Kaltenbrunner <stefan@kaltenbrunner.cc> — 2013-08-05T21:15:06Z

On 08/05/2013 09:53 PM, Alvaro Herrera wrote:
> Tom Lane escribió:
> 
>> What Josh seems to be concerned with in this thread is the question of
>> whether we should support an installation *policy decision* not to allow
>> ALTER SYSTEM SET.  Not because a particular set of parameters is broken,
>> but just because somebody is afraid the DBA might break things.  TBH
>> I'm not sure I buy that, at least not as long as ALTER SYSTEM is a
>> superuser feature.  There is nothing in Postgres that denies permissions
>> to superusers, and this doesn't seem like a very good place to start.
> 
> Someone made an argument about this on IRC: GUI tool users are going to
> want to use ALTER SYSTEM through point-and-click, and if all we offer is
> superuser-level access to the feature, we're going to end up with a lot
> of people running with superuser privileges just so that they are able
> to tweak inconsequential settings.  This seems dangerous.

indeed it is

> 
> The other issue is that currently you can only edit a server's config if
> you are logged in to it.  If we permit SQL-level access to that, and
> somebody who doesn't have access to edit the files blocks themselves
> out, there is no way for them to get a working system *at all*.

thinking more about that - is there _ANY_ prerequisite of an application
that can be completely reconfigured over a remote access protocol and
solved the reliability and security challenges of that to a reasonable
degree?


Stefan

Re: Disabling ALTER SYSTEM SET WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters

Josh Berkus <josh@agliodbs.com> — 2013-08-05T21:23:30Z

Stefan,

> thinking more about that - is there _ANY_ prerequisite of an application
> that can be completely reconfigured over a remote access protocol and
> solved the reliability and security challenges of that to a reasonable
> degree?

Good question!

I also think that, given the issues raised in discussion of ALTER SYSTEM
SET, we should poll a bit of the wider community.  I'll put something up
on my blog, unless someone can think of a better way.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

Re: File-per-GUC WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Bruce Momjian <bruce@momjian.us> — 2013-08-05T21:29:48Z

On Mon, Aug  5, 2013 at 01:56:40PM -0400, Tom Lane wrote:
> Josh Berkus <josh@agliodbs.com> writes:
> > I'll also point out that some of our settings only really "work" in
> > combinations of two or more settings.  For example, one doesn't want to
> > set archive_mode = on unless one is setting archive_command as well.
> > And generally if one sets sequential_page_cost, one is changing the
> > other cost parameters as well.  And logging parameters are generally
> > managed as a set.
> 
> > So the case of two sessions both modifying ALTER SYSTEM SET, and one
> > succeeding for some-but-all-GUCS, and the other succeeding for
> > some-but-not-all-GUCs, would not be user-friendly or pretty, even if
> > each setting change succeeded or failed atomically.
> 
> That is a killer point.  So really the value of the global lock is to
> ensure serializability when transactions are updating multiple GUCs.

Well, I think it is more than that.  The global lock will allow multiple
GUCs to be modified in a serializable fashion, but unless you have new
backends wait until that lock is clear, new backends are going to see
the directory in an inconsistent state if we keep a single-guc-per-file.

So, unless we want new backends to hang during ALTER SYSTEM SET
operations, I think we are going to need a global lock and all gucs in a
single file.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + It's impossible for everything to be true. +

Re: Disabling ALTER SYSTEM SET WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters

Bruce Momjian <bruce@momjian.us> — 2013-08-05T21:39:59Z

On Mon, Aug  5, 2013 at 03:53:01PM -0400, Alvaro Herrera wrote:
> The other issue is that currently you can only edit a server's config if
> you are logged in to it.  If we permit SQL-level access to that, and
> somebody who doesn't have access to edit the files blocks themselves
> out, there is no way for them to get a working system *at all*.

Well, if we want to give the administrator a way to disable honoring any
previously-defined ALTER SYSTEM SET commands, how would they do that
without OS access?  By definition, they can't connect via SQL, so what
would the API be?

Also, even if they could do it remotely, if they previously set
listen_addresses via ALTER SYSTEM SET, and we then disable all ALTER
SYSTEM SET settings, they still can't access the system because by
default Postgres will only listen on local sockets.

In summary, the SQL interface to configuration parameters is a
convenience, but I don't think it is ever going to be something that can
replace full file system access --- that is not a limitation of the
implemention of ALTER SYSTEM SET, but just something that is impossible.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + It's impossible for everything to be true. +

Re: Unsafe GUCs and ALTER SYSTEM WAS: Re: ALTER SYSTEM SET

Greg Stark <stark@mit.edu> — 2013-08-05T22:56:16Z

On Mon, Aug 5, 2013 at 7:02 PM, Josh Berkus <josh@agliodbs.com> wrote:
> Nope.  ALTER SYSTEM, from my POV, is mainly for folks who *don't* use
> Puppet/Chef/whatever.  Here's where I see ALTER SYSTEM being useful:
>
> * invididually managed servers with out centralized management (i.e. one
> DBA, one server).

Well any server that is treated as a one-off rather than part of a
cluster of identical servers. That will actually be true for a lot of
users even if they have a slave or two and a development machine or
three. It's only when you start managing multiple production servers
for multiple databasees that you start to really see the benefits of
Puppet/Chef in solving your problems once for every server.

Fwiw It's not hard to imagine Puppet/Chef modules that *do* work
through ALTER SYSTEM and there would even be advantages. They could
easily check what the current value is and even adjust the values
differently for each system based on the environment. This is a bit
trickier to manage when rolling out files than it is when connected to
an instance. I would not be surprised to see both styles be quite
common, possibly even in the same environment for different settings
(I'm not saying I recommend that though).

-- 
greg

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Greg Stark <stark@mit.edu> — 2013-08-05T23:17:14Z

On Mon, Aug 5, 2013 at 6:16 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> If it's file-per-GUC, then when two sessions try to write different GUCs,
> there is no conflict

Well it's not that there's no conflict. It's just that we don't detect
the conflict. If you raise shared_buffers and I calculate the optimal
work_mem based on the old shared_buffers and set work_mem I may well
set it too large. Or if I set multiple settings together and you set
one of them you'll undo my change and lose just part of my changes but
not stop me from setting the others inconsistently.

I think it's actually better to lose a whole set of changes than to
intermix two sets of changes in unpredictable ways.  And I would
suggest instead of doing a rename we just create a second link leaving
behind the original link as well. That means you wouldn't be losing
the other changes entirely you would have the history of the settings
as they were briefly at least.

-- 
greg

Re: File-per-GUC WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Andres Freund <andres@2ndquadrant.com> — 2013-08-06T00:12:51Z

On 2013-08-05 13:56:40 -0400, Tom Lane wrote:
> Josh Berkus <josh@agliodbs.com> writes:
> > I'll also point out that some of our settings only really "work" in
> > combinations of two or more settings.  For example, one doesn't want to
> > set archive_mode = on unless one is setting archive_command as well.
> > And generally if one sets sequential_page_cost, one is changing the
> > other cost parameters as well.  And logging parameters are generally
> > managed as a set.
> 
> > So the case of two sessions both modifying ALTER SYSTEM SET, and one
> > succeeding for some-but-all-GUCS, and the other succeeding for
> > some-but-not-all-GUCs, would not be user-friendly or pretty, even if
> > each setting change succeeded or failed atomically.
> 
> That is a killer point.  So really the value of the global lock is to
> ensure serializability when transactions are updating multiple GUCs.

I don't think a global lock helps very much WRT this. Likely the lock
will be released after a single SET finishes, in that case it doesn't
guarantee much for a set of SETs. Even if we were to make it be released
only at session end, the session that wants to set something conflicting
will only be blocked by wanting to set the lock, it won't be prevented
from doing so. The SET will then succeed after the other session finished.

I don't think we're designing a feature that's supposed to be used under
heavy concurrency here. If you have users/tools doing conflicting
actions as superusers you need to solve that by social means, not by
technical ones.

Greetings,

Andres Freund

-- 
 Andres Freund	                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: File-per-GUC WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Josh Berkus <josh@agliodbs.com> — 2013-08-06T01:17:16Z

On 08/05/2013 05:12 PM, Andres Freund wrote:
> I don't think we're designing a feature that's supposed to be used under
> heavy concurrency here. If you have users/tools doing conflicting
> actions as superusers you need to solve that by social means, not by
> technical ones.

+1

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Dimitri Fontaine <dimitri@2ndquadrant.fr> — 2013-08-06T09:14:33Z

Greg Stark <stark@mit.edu> writes:
> set it too large. Or if I set multiple settings together and you set
> one of them you'll undo my change and lose just part of my changes but
> not stop me from setting the others inconsistently.

So we need to be able to change more than one setting in a single
command and have a lock around the whole command, or maybe have ALTER
SYSTEM SET run in a transaction.

-- 
Dimitri Fontaine
http://2ndQuadrant.fr     PostgreSQL : Expertise, Formation et Support

Re: Disabling ALTER SYSTEM SET WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters

Cédric Villemain <cedric@2ndquadrant.com> — 2013-08-06T10:40:12Z

> > Again, what are we trying to achieve?!
> 
> no idea - wondering about that myself...

It seems we are trying to add grammar for modifying postgresql.conf.
Something we can already do easily in a standard extension, but without 
grammar changes.

Maybe better to provide a contrib/ to modify config, then design what we can 
achieve more with an ALTER SYSTEM command.
-- 
Cédric Villemain +33 (0)6 20 30 22 52
http://2ndQuadrant.fr/
PostgreSQL: Support 24x7 - Développement, Expertise et Formation

Re: Disabling ALTER SYSTEM SET WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters

Tom Lane <tgl@sss.pgh.pa.us> — 2013-08-06T11:59:14Z

=?iso-8859-1?q?C=E9dric_Villemain?= <cedric@2ndquadrant.com> writes:
>>> Again, what are we trying to achieve?!

>> no idea - wondering about that myself...

> It seems we are trying to add grammar for modifying postgresql.conf.
> Something we can already do easily in a standard extension, but without 
> grammar changes.

> Maybe better to provide a contrib/ to modify config, then design what we can 
> achieve more with an ALTER SYSTEM command.

Hmm ... putting the UI into a contrib module would neatly solve the
problem for people who don't want the functionality.  They just don't
install the module.  Another thought here is that it seems like we've
been discussing multiple scenarios that don't necessarily all have the
same best solution.  Maybe we should think in terms of multiple contrib
modules.

			regards, tom lane

Re: File-per-GUC WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Bruce Momjian <bruce@momjian.us> — 2013-08-06T12:08:49Z

On Mon, Aug  5, 2013 at 05:29:48PM -0400, Bruce Momjian wrote:
> > That is a killer point.  So really the value of the global lock is to
> > ensure serializability when transactions are updating multiple GUCs.
> 
> Well, I think it is more than that.  The global lock will allow multiple
> GUCs to be modified in a serializable fashion, but unless you have new
> backends wait until that lock is clear, new backends are going to see
> the directory in an inconsistent state if we keep a single-guc-per-file.
> 
> So, unless we want new backends to hang during ALTER SYSTEM SET
> operations, I think we are going to need a global lock and all gucs in a
> single file.

To clarify, I talked about new backends above, but on Unix and Windows,
new backends inherit the postmaster GUC settings, so it would only be
config file reload that would possibly see the single-guc-per-file in an
inconsistent state.  And we already have this possibility if someone
edits postgresql.conf, and while the editor is writing the file, a
backend is reading the file via a reload.

Now, I assume that ALTER SYSTEM SET would automatically issue a
pg_reload_conf(), so we would need to make sure that people modifying
multiple parameters that are related do it in a single transaction, and
that we issue a pg_reload_conf() only after all values have been
changed.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + It's impossible for everything to be true. +

Re: Disabling ALTER SYSTEM SET WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters

Bruce Momjian <bruce@momjian.us> — 2013-08-06T12:29:17Z

On Tue, Aug  6, 2013 at 12:40:12PM +0200, Cédric Villemain wrote:
> > > Again, what are we trying to achieve?!
> > 
> > no idea - wondering about that myself...
> 
> It seems we are trying to add grammar for modifying postgresql.conf.
> Something we can already do easily in a standard extension, but without 
> grammar changes.
> 
> Maybe better to provide a contrib/ to modify config, then design what we can 
> achieve more with an ALTER SYSTEM command.

Let's look at the problems:

*  remote users can lock themselves out of the server
*  interconnected GUC variables are complex to change
*  need a way to disable this feature

Given the above, I am not sure I see a way forward for ALTER SYSTEM SET.
One compromise that avoids the problems above would be to have a limited
feature called ALTER LOG SET that only controls logging parameters, but
I am not sure there is enough use-case there.  

This is not a zero-cost feature as we would be giving people _two_ ways
to configure Postgres, and two ways to find a fix if it is broken, so we
have to have a clear win to implement this.  Also, if you have broken
the system via ALTER SYSTEM SET, you might need to edit flat files to
fix it, adding further complexity and limitations if someone only knows
the SQL method of configuration.  Given that, and the problems above, I
reluctantly just don't see how this features moves us forward.

There seemed to be agreement on having a config.d, though.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + It's impossible for everything to be true. +

Re: Disabling ALTER SYSTEM SET WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters

Cédric Villemain <cedric@2ndquadrant.com> — 2013-08-06T12:36:22Z

> There seemed to be agreement on having a config.d, though.

Yes.
Also, the validate_conf_parameter() (or some similar name) Amit added in his 
patch sounds useful if an extension can use it (to check a GUC someone want to 
change, to check a configuration file, ...)
-- 
Cédric Villemain +33 (0)6 20 30 22 52
http://2ndQuadrant.fr/
PostgreSQL: Support 24x7 - Développement, Expertise et Formation

Re: File-per-GUC WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila@huawei.com> — 2013-08-06T13:00:18Z

On Tuesday, August 06, 2013 5:39 PM Bruce Momjian wrote:
> On Mon, Aug  5, 2013 at 05:29:48PM -0400, Bruce Momjian wrote:
> > > That is a killer point.  So really the value of the global lock is
> to
> > > ensure serializability when transactions are updating multiple
> GUCs.
> >
> > Well, I think it is more than that.  The global lock will allow
> multiple
> > GUCs to be modified in a serializable fashion, but unless you have
> new
> > backends wait until that lock is clear, new backends are going to see
> > the directory in an inconsistent state if we keep a single-guc-per-
> file.
> >
> > So, unless we want new backends to hang during ALTER SYSTEM SET
> > operations, I think we are going to need a global lock and all gucs
> in a
> > single file.
> 
> To clarify, I talked about new backends above, but on Unix and Windows,
> new backends inherit the postmaster GUC settings, so it would only be
> config file reload that would possibly see the single-guc-per-file in
> an
> inconsistent state.  And we already have this possibility if someone
> edits postgresql.conf, and while the editor is writing the file, a
> backend is reading the file via a reload.
> 
> Now, I assume that ALTER SYSTEM SET would automatically issue a
> pg_reload_conf(), so we would need to make sure that people modifying
> multiple parameters that are related do it in a single transaction, and
> that we issue a pg_reload_conf() only after all values have been
> changed.

Can't we leave this onus of conflict of changing dependent parameters on
user, such that he should do it carefully?
Other databases SQL Server, Oracle also has similar feature, so there user's
also must be handling by themselves.


With Regards,
Amit Kapila.

Re: File-per-GUC WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Bruce Momjian <bruce@momjian.us> — 2013-08-06T14:25:54Z

On Tue, Aug  6, 2013 at 06:30:18PM +0530, Amit Kapila wrote:
> > Now, I assume that ALTER SYSTEM SET would automatically issue a
> > pg_reload_conf(), so we would need to make sure that people modifying
> > multiple parameters that are related do it in a single transaction, and
> > that we issue a pg_reload_conf() only after all values have been
> > changed.
> 
> Can't we leave this onus of conflict of changing dependent parameters on
> user, such that he should do it carefully?
> Other databases SQL Server, Oracle also has similar feature, so there user's
> also must be handling by themselves.

Yes, we _can_ have the user deal with it, but we have to consider the
final user experience.  The initial experience will be, "Oh, great, no
more need to edit a configuration file,", but it might end with:

*  Oh, my server doesn't start and I don't have shell access
*  How do I undo an ALTER SYSTEM SET if the server can't be started
*  Why doesn't my server start?  postgresql.conf looks fine (see ALTER
   SYSTEM SET)
*  What things did I change via ALTER SYSTEM SET
*  How do I disable ALTER SYSTEM SET so only postgresql.conf works

Consider how long it took the hackers to realize all the interactions
involved;  we would have to effectively communicate that to users.

I realize other database systems have this, but those systems are not
known to be easy to use.  ALTER SYSTEM SET has the promise of making
Postgres easier to use, _and_ harder to use.

The question is what percentage of users are going to have a positive
experience with this feature, and what percentage are going to have a
negative or disastrous experience with it, or disable it?

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + It's impossible for everything to be true. +

Re: File-per-GUC WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-06T14:54:22Z

* Andres Freund (andres@2ndquadrant.com) wrote:
> I don't think we're designing a feature that's supposed to be used under
> heavy concurrency here. If you have users/tools doing conflicting
> actions as superusers you need to solve that by social means, not by
> technical ones.

If this actually gets used by puppet or another CMS, the chances of the
'social means' being successful drop drastically.

I agree that it doesn't need to work under heavy concurrency, but it
should do something sensible if it happens- perhaps even just throwing
an error if it can't acquire the lock immediately, warning the user that
some other process is trying to modify the config concurrently.

	Thanks,

		Stephen

Re: Disabling ALTER SYSTEM SET WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters

Stephen Frost <sfrost@snowman.net> — 2013-08-06T14:58:52Z

* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> =?iso-8859-1?q?C=E9dric_Villemain?= <cedric@2ndquadrant.com> writes:
> > Maybe better to provide a contrib/ to modify config, then design what we can 
> > achieve more with an ALTER SYSTEM command.
> 
> Hmm ... putting the UI into a contrib module would neatly solve the
> problem for people who don't want the functionality.  They just don't
> install the module.  Another thought here is that it seems like we've
> been discussing multiple scenarios that don't necessarily all have the
> same best solution.  Maybe we should think in terms of multiple contrib
> modules.

I'd certainly be happier with that- but it already exists, to some
extent, in the form of the pgAdmin "adminpack" contrib module.  Perhaps
we should be working on improving what we've already got there rather
than adding something completely new..

	Thanks,

		Stephen

Re: File-per-GUC WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Bruce Momjian <bruce@momjian.us> — 2013-08-06T15:06:36Z

On Tue, Aug  6, 2013 at 10:54:22AM -0400, Stephen Frost wrote:
> * Andres Freund (andres@2ndquadrant.com) wrote:
> > I don't think we're designing a feature that's supposed to be used under
> > heavy concurrency here. If you have users/tools doing conflicting
> > actions as superusers you need to solve that by social means, not by
> > technical ones.
> 
> If this actually gets used by puppet or another CMS, the chances of the
> 'social means' being successful drop drastically.
> 
> I agree that it doesn't need to work under heavy concurrency, but it
> should do something sensible if it happens- perhaps even just throwing
> an error if it can't acquire the lock immediately, warning the user that
> some other process is trying to modify the config concurrently.

My guess is that most users are going to do:

	SHOW log_min_duration_statement;
	SET log_min_duration_statement = 3;

i.e. there isn't going to be any way to _lock_ that value between
viewing it and setting it, and hence no way to warn users.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + It's impossible for everything to be true. +

Re: Disabling ALTER SYSTEM SET WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters

Bruce Momjian <bruce@momjian.us> — 2013-08-06T15:08:21Z

On Tue, Aug  6, 2013 at 10:58:52AM -0400, Stephen Frost wrote:
> * Tom Lane (tgl@sss.pgh.pa.us) wrote:
> > =?iso-8859-1?q?C=E9dric_Villemain?= <cedric@2ndquadrant.com> writes:
> > > Maybe better to provide a contrib/ to modify config, then design what we can 
> > > achieve more with an ALTER SYSTEM command.
> > 
> > Hmm ... putting the UI into a contrib module would neatly solve the
> > problem for people who don't want the functionality.  They just don't
> > install the module.  Another thought here is that it seems like we've
> > been discussing multiple scenarios that don't necessarily all have the
> > same best solution.  Maybe we should think in terms of multiple contrib
> > modules.
> 
> I'd certainly be happier with that- but it already exists, to some
> extent, in the form of the pgAdmin "adminpack" contrib module.  Perhaps
> we should be working on improving what we've already got there rather
> than adding something completely new..

Yes, "adminpack" has mostly read/write file I/O operations --- it would
need a higher-level API.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + It's impossible for everything to be true. +

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Boszormenyi Zoltan <zb@cybertec.at> — 2013-08-06T16:34:35Z

2013-08-05 16:01 keltezéssel, Stephen Frost írta:
> * Greg Stark (stark@mit.edu) wrote:
>> On Fri, Aug 2, 2013 at 4:05 PM, Stephen Frost <sfrost@snowman.net> wrote:
>>>> I'm not even clear we do want this in /etc since none of our GUC
>>>> options are repeatable things like Apache virtual servers. It actually
>>>> makes *more* sense for pg_hba than it does for gucs. I think we can
>>>> assume that in the future we'll have something like it however.
>>> I tend to agree with this also, though I can imagine wanting to separate
>>> things in a conf.d directory ala exim's conf.d directories, to allow
>>> tools like puppet to manage certain things environment-wide (perhaps
>>> krb_server_keyfile) while other configuration options are managed
>>> locally.
>> Extensions are actually a pretty good argument for why conf.d in /etc
>> (or wherever the non-auto-config is) is pretty important useful.
>> That's the kind of thing conf.d directories are meant for. A user can
>> install a package containing an extension and the extension would
>> automatically drop in the config entries needed in that directory.
> Agreed, though I think there should be a difference between "shared
> library load" being added-to for extensions, and "random
> extension-specific GUC"..

Now that you mention "shared library load", it may be a good idea
to add an "append-to-this-GUC" flag instead of overwriting the
previous value. Two GUCs may make use of it: shared_preload_libraries
and local_preload_libraries. It would make packagers' of extensions
and DBA's lives easier.

Best regards,
Zoltán Böszörményi

>
> 	Thanks,
>
> 		Stephen


-- 
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de
      http://www.postgresql.at/

Re: Disabling ALTER SYSTEM SET WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters

Josh Berkus <josh@agliodbs.com> — 2013-08-06T17:33:20Z

On 08/06/2013 05:29 AM, Bruce Momjian wrote:
> Let's look at the problems:
> 
> *  remote users can lock themselves out of the server
> *  interconnected GUC variables are complex to change
> *  need a way to disable this feature
> 
> Given the above, I am not sure I see a way forward for ALTER SYSTEM SET.
> One compromise that avoids the problems above would be to have a limited
> feature called ALTER LOG SET that only controls logging parameters, but
> I am not sure there is enough use-case there.  
> 
> This is not a zero-cost feature as we would be giving people _two_ ways
> to configure Postgres, and two ways to find a fix if it is broken, so we
> have to have a clear win to implement this.  Also, if you have broken
> the system via ALTER SYSTEM SET, you might need to edit flat files to
> fix it, adding further complexity and limitations if someone only knows
> the SQL method of configuration.  Given that, and the problems above, I
> reluctantly just don't see how this features moves us forward.

Well said.

I'd like to look at use cases, and let's see how ALTER SYSTEM SET
addresses or doesn't address these use cases.  I'd really like it if
some other folks also posted use cases they know of.

(1) Making is easier for GUIs to manage PostgreSQL settings.

(2) Enabling DBAAS services to give users limited control over settings.

(3) Making the "initial developer experience" better by not requiring
hand-editing of configuration files to play with settings.

(4) Making it easier to write automated testing scripts which test
changes to settings.

(5) Enabling new ways of writing Puppet/Chef/etc. scripts, which can
check a setting before changing it.

For the current patch, it fullfills these use cases as follows:

(1) works, but is ALSO one of the critical problems with the feature.

(2) does not work for this purpose.

(3) works for this purpose.

(4) works for this purpose.

(5) works for this purpose, although it's unclear if it's actually wanted.

Others?

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Bruce Momjian <bruce@momjian.us> — 2013-08-06T17:41:48Z

On Tue, Aug  6, 2013 at 06:34:35PM +0200, Boszormenyi Zoltan wrote:
> 2013-08-05 16:01 keltezéssel, Stephen Frost írta:
> >* Greg Stark (stark@mit.edu) wrote:
> >>On Fri, Aug 2, 2013 at 4:05 PM, Stephen Frost <sfrost@snowman.net> wrote:
> >>>>I'm not even clear we do want this in /etc since none of our GUC
> >>>>options are repeatable things like Apache virtual servers. It actually
> >>>>makes *more* sense for pg_hba than it does for gucs. I think we can
> >>>>assume that in the future we'll have something like it however.
> >>>I tend to agree with this also, though I can imagine wanting to separate
> >>>things in a conf.d directory ala exim's conf.d directories, to allow
> >>>tools like puppet to manage certain things environment-wide (perhaps
> >>>krb_server_keyfile) while other configuration options are managed
> >>>locally.
> >>Extensions are actually a pretty good argument for why conf.d in /etc
> >>(or wherever the non-auto-config is) is pretty important useful.
> >>That's the kind of thing conf.d directories are meant for. A user can
> >>install a package containing an extension and the extension would
> >>automatically drop in the config entries needed in that directory.
> >Agreed, though I think there should be a difference between "shared
> >library load" being added-to for extensions, and "random
> >extension-specific GUC"..
> 
> Now that you mention "shared library load", it may be a good idea
> to add an "append-to-this-GUC" flag instead of overwriting the
> previous value. Two GUCs may make use of it: shared_preload_libraries
> and local_preload_libraries. It would make packagers' of extensions
> and DBA's lives easier.

'search_path' might also use it, though we might need append and
prepend.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + It's impossible for everything to be true. +

Re: Disabling ALTER SYSTEM SET WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters

Bruce Momjian <bruce@momjian.us> — 2013-08-06T18:31:47Z

On Tue, Aug  6, 2013 at 10:33:20AM -0700, Josh Berkus wrote:
> On 08/06/2013 05:29 AM, Bruce Momjian wrote:
> > Let's look at the problems:
> > 
> > *  remote users can lock themselves out of the server
> > *  interconnected GUC variables are complex to change
> > *  need a way to disable this feature
> > 
> > Given the above, I am not sure I see a way forward for ALTER SYSTEM SET.
> > One compromise that avoids the problems above would be to have a limited
> > feature called ALTER LOG SET that only controls logging parameters, but
> > I am not sure there is enough use-case there.  
> > 
> > This is not a zero-cost feature as we would be giving people _two_ ways
> > to configure Postgres, and two ways to find a fix if it is broken, so we
> > have to have a clear win to implement this.  Also, if you have broken
> > the system via ALTER SYSTEM SET, you might need to edit flat files to
> > fix it, adding further complexity and limitations if someone only knows
> > the SQL method of configuration.  Given that, and the problems above, I
> > reluctantly just don't see how this features moves us forward.
> 
> Well said.

I wish I had good news to "say well".  ;-)

> I'd like to look at use cases, and let's see how ALTER SYSTEM SET
> addresses or doesn't address these use cases.  I'd really like it if
> some other folks also posted use cases they know of.
> 
> (1) Making is easier for GUIs to manage PostgreSQL settings.

One of the traps here is that while it makes it easier, it also could
trap the user if they don't have the knowledge to fix a problem because
would need to acquire the knowledge while they are trying to fix the
problem, rather then while they are making the initial change.

> (2) Enabling DBAAS services to give users limited control over settings.

Right.  This could be accomplished by giving users a function API for
certain features, and then marking the function as SECURITY DEFINER. 
However, I am unclear how to do this in a generic way.

Once neat idea would be to have a lookup table define which setting the
administrator wishes to allow for non-superusers.  If adminpack has an
API to change postgresql.conf, it would be easy to create a function
with SECURITY DEFINER permissions that SET lookup-allowed values in
postgresql.conf.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + It's impossible for everything to be true. +

Re: Disabling ALTER SYSTEM SET WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters

Alvaro Herrera <alvherre@2ndquadrant.com> — 2013-08-06T18:53:47Z

Josh Berkus escribió:

> (2) Enabling DBAAS services to give users limited control over settings.

> (5) Enabling new ways of writing Puppet/Chef/etc. scripts, which can
> check a setting before changing it.

Surely these two cases are better covered by conf.d.  For (2), DBaaS
providers could offer pre-cooked snippets that are dropped in conf.d at
the click of a button.  For (5) it should be obvious that conf.d is
better than ALTER SYSTEM.

Not clear on (4); I think conf.d would also solve that one.

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Re: Disabling ALTER SYSTEM SET WAS: Re: ALTER SYSTEM SET command to change postgresql.conf parameters

Claudio Freire <klaussfreire@gmail.com> — 2013-08-06T18:55:46Z

On Tue, Aug 6, 2013 at 3:31 PM, Bruce Momjian <bruce@momjian.us> wrote:
>> I'd like to look at use cases, and let's see how ALTER SYSTEM SET
>> addresses or doesn't address these use cases.  I'd really like it if
>> some other folks also posted use cases they know of.
>>
>> (1) Making is easier for GUIs to manage PostgreSQL settings.
>
> One of the traps here is that while it makes it easier, it also could
> trap the user if they don't have the knowledge to fix a problem because
> would need to acquire the knowledge while they are trying to fix the
> problem, rather then while they are making the initial change.

I think the more serious problem here is not about knowledge
acquisition, but access to problem-fixing means. As was said some
posts ago, if a DBA has access to a superuser account but not to
server configuration files, he can lock himself out for good. No
amount of knowledge will avail him/her. That's bad.

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Boszormenyi Zoltan <zb@cybertec.at> — 2013-08-06T19:52:52Z

2013-08-06 19:41 keltezéssel, Bruce Momjian írta:
> On Tue, Aug  6, 2013 at 06:34:35PM +0200, Boszormenyi Zoltan wrote:
>> 2013-08-05 16:01 keltezéssel, Stephen Frost írta:
>>> * Greg Stark (stark@mit.edu) wrote:
>>>> On Fri, Aug 2, 2013 at 4:05 PM, Stephen Frost <sfrost@snowman.net> wrote:
>>>>>> I'm not even clear we do want this in /etc since none of our GUC
>>>>>> options are repeatable things like Apache virtual servers. It actually
>>>>>> makes *more* sense for pg_hba than it does for gucs. I think we can
>>>>>> assume that in the future we'll have something like it however.
>>>>> I tend to agree with this also, though I can imagine wanting to separate
>>>>> things in a conf.d directory ala exim's conf.d directories, to allow
>>>>> tools like puppet to manage certain things environment-wide (perhaps
>>>>> krb_server_keyfile) while other configuration options are managed
>>>>> locally.
>>>> Extensions are actually a pretty good argument for why conf.d in /etc
>>>> (or wherever the non-auto-config is) is pretty important useful.
>>>> That's the kind of thing conf.d directories are meant for. A user can
>>>> install a package containing an extension and the extension would
>>>> automatically drop in the config entries needed in that directory.
>>> Agreed, though I think there should be a difference between "shared
>>> library load" being added-to for extensions, and "random
>>> extension-specific GUC"..
>> Now that you mention "shared library load", it may be a good idea
>> to add an "append-to-this-GUC" flag instead of overwriting the
>> previous value. Two GUCs may make use of it: shared_preload_libraries
>> and local_preload_libraries. It would make packagers' of extensions
>> and DBA's lives easier.
> 'search_path' might also use it, though we might need append and
> prepend.

Indeed. Although I was thinking along the lines of the GUC parser, so:

shared_preload_library += 'some_new_lib'

would append to the old value.

Maybe "<+=" is intuitive enough for prepending, or someone may
come up with a better idea.

But the extra flag would still be needed to indicate the GUC is a list,
so these new operators are usable on them and not on regular GUCs.

Best regards,
Zoltán Böszörményi

-- 
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de
      http://www.postgresql.at/

Re: Unsafe GUCs and ALTER SYSTEM WAS: Re: ALTER SYSTEM SET

Greg Smith <greg@2ndquadrant.com> — 2013-08-07T01:24:47Z

On 8/5/13 2:36 PM, Josh Berkus wrote:
> Most of our users not on Heroku are running with superuser as the app
> user now.  Like, 95% of them based on my personal experience (because
> our object permissions management sucks).

My percentage wouldn't be nearly that high.  95% of database installs 
done by developers?  That I could believe.  But setups done by 
ex-{Oracle,DB2,Sybase...} DBAs avoid superuser rights for the 
application, and that's around 1/3 of the systems I see.  In general I 
agree that there certainly are a lot of superuser only installs out 
there, I just don't think it's quite as bad as you're painting it.

The important thing to realize here is that ALTER SYSTEM SET is a 
convenience function.  It’s useful to the extent that it makes people 
feel more comfortable with changing things in the database.  We will 
never stop people from shooting themselves in the foot.  And if the 
barriers added for that purpose are too high, like disabling changes to 
shared_buffers altogether, all you’ll do is make this so restricted that 
it doesn’t satisfy anyone.

The original spec I outlined for how to implement this feature allowed 
disabling the whole thing.  Then it was just commenting out the 
includedir directive from the postgresql.conf.  Allowing people to 
disable use of this feature in a managed configuration environment is 
important.  Beyond that, I don’t expect that we’ll ever make this foolproof.

After arguing out this topic in person with Stephen last night, I’ve 
lumped ideas here into three major approaches that could be followed:

1) Don’t allow changing unsafe GUCs.

2) Provide a way for the server to start with bad setting and force the 
administrator to fix the problem they introduced.  Some sort of 
maintenance mode that only allows superuser connections would force 
someone to clean up this class of problem at next restart, while still 
giving them enough access to run a new ALTER SYSTEM SET command.

3) Require extra syntax to modify an unsafe GUC.

As far as fixing the bad settings goes, there’s already code in initdb 
to detect how high shared_buffers and max_connections can go.  Any bogus 
shared_buffers/max_connections value above the kernel limits could be 
worked around with that approach.

Here’s how I would try and communicate that a change is unsafe, only 
allowing it after proving user is paying some attention to the danger:

# ALTER SYSTEM SET shared_buffers = ‘8GB’;
NOTICE:  Changing shared_buffers only takes effect after a server restart.
ERROR:  Changing shared_buffers incorrectly can prevent the server from 
starting normally.  Use the FORCE option to modify the value anyway.

# ALTER SYSTEM SET shared_buffers = ‘8GB’ FORCE;
NOTICE:  Changing shared_buffers only takes effect after a server restart.
ALTER SYSTEM

Will bad examples pop up in the Internet that just use FORCE all the 
time?  Sure they will, and people will cut and paste them without paying 
attention.  I don't see why that possibility has to block this feature 
from being adopted though.  That line of thinking leads toward removing 
trust authentication, because that's similarly abused with cut and paste 
tutorials.

-- 
Greg Smith   2ndQuadrant US    greg@2ndQuadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com

Re: Unsafe GUCs and ALTER SYSTEM WAS: Re: ALTER SYSTEM SET

Bruce Momjian <bruce@momjian.us> — 2013-08-07T02:30:47Z

On Tue, Aug  6, 2013 at 09:24:47PM -0400, Greg Smith wrote:
> # ALTER SYSTEM SET shared_buffers = ‘8GB’ FORCE;
> NOTICE:  Changing shared_buffers only takes effect after a server restart.
> ALTER SYSTEM
> 
> Will bad examples pop up in the Internet that just use FORCE all the
> time?  Sure they will, and people will cut and paste them without
> paying attention.  I don't see why that possibility has to block
> this feature from being adopted though.  That line of thinking leads
> toward removing trust authentication, because that's similarly
> abused with cut and paste tutorials.

We already have six levels of GUC settings:

	postgresql.conf
	user
	database
	session
	function
	subtransaction

If we add ALTER SYSTEM SET and config.d, we would then have eight. 
ALTER SYSTEM SET seems to add an entirely new set of behaviors and
complexity.  Is that really what we want?

If we do this, perhaps we should unconditionally just print the file
name they have to delete to undo the operation in case the server
doesn't start;  I am unclear we can clearly identify all the GUC
settings that could cause a server not to start.  Also, I think we need
a SHOW SYSTEM command so users can see their settings via SQL.

FYI, ALTER SYSTEM SET is hitting the same problems we would have if
pg_hba.conf were set in SQL and in flat files.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + It's impossible for everything to be true. +

Re: Unsafe GUCs and ALTER SYSTEM WAS: Re: ALTER SYSTEM SET

Amit Kapila <amit.kapila@huawei.com> — 2013-08-07T03:40:31Z

> From: Greg Smith [mailto:greg@2ndQuadrant.com]
> Sent: Wednesday, August 07, 2013 6:55 AM
> To: Josh Berkus
> Cc: Stephen Frost; Bruce Momjian; Greg Stark; Andres Freund; Alvaro
> Herrera; Fujii Masao; Robert Haas; Amit Kapila; Dimitri Fontaine;
> pgsql-hackers@postgresql.org; Tom Lane
> Subject: Re: [HACKERS] Unsafe GUCs and ALTER SYSTEM WAS: Re: ALTER
> SYSTEM SET
> 
> On 8/5/13 2:36 PM, Josh Berkus wrote:
> > Most of our users not on Heroku are running with superuser as the app
> > user now.  Like, 95% of them based on my personal experience (because
> > our object permissions management sucks).
> 
> My percentage wouldn't be nearly that high.  95% of database installs
> done by developers?  That I could believe.  But setups done by
> ex-{Oracle,DB2,Sybase...} DBAs avoid superuser rights for the
> application, and that's around 1/3 of the systems I see.  In general I
> agree that there certainly are a lot of superuser only installs out
> there, I just don't think it's quite as bad as you're painting it.
> 
> The important thing to realize here is that ALTER SYSTEM SET is a
> convenience function.  It’s useful to the extent that it makes people
> feel more comfortable with changing things in the database.  We will
> never stop people from shooting themselves in the foot.  And if the
> barriers added for that purpose are too high, like disabling changes to
> shared_buffers altogether, all you’ll do is make this so restricted
> that
> it doesn’t satisfy anyone.
> 
> The original spec I outlined for how to implement this feature allowed
> disabling the whole thing.  Then it was just commenting out the
> includedir directive from the postgresql.conf.  Allowing people to
> disable use of this feature in a managed configuration environment is
> important.  Beyond that, I don’t expect that we’ll ever make this
> foolproof.
> 
> After arguing out this topic in person with Stephen last night, I’ve
> lumped ideas here into three major approaches that could be followed:

> 1) Don’t allow changing unsafe GUCs.
> 
> 2) Provide a way for the server to start with bad setting and force the
> administrator to fix the problem they introduced.  Some sort of
> maintenance mode that only allows superuser connections would force
> someone to clean up this class of problem at next restart, while still
> giving them enough access to run a new ALTER SYSTEM SET command.
> 
> 3) Require extra syntax to modify an unsafe GUC.

I also believe this is the right way to move forward and similar thoughts of mine are shared in mail below:
http://www.postgresql.org/message-id/00a301ce91a3$09226970$1b673c50$@kapila@huawei.com

I think for unsafe parameter's, for initial patch may be we can choose limited number of parameters. 
 
> As far as fixing the bad settings goes, there’s already code in initdb
> to detect how high shared_buffers and max_connections can go.  Any
> bogus
> shared_buffers/max_connections value above the kernel limits could be
> worked around with that approach.
> 
> Here’s how I would try and communicate that a change is unsafe, only
> allowing it after proving user is paying some attention to the danger:
> 
> # ALTER SYSTEM SET shared_buffers = ‘8GB’;
> NOTICE:  Changing shared_buffers only takes effect after a server
> restart.
> ERROR:  Changing shared_buffers incorrectly can prevent the server from
> starting normally.  Use the FORCE option to modify the value anyway.
> 
> # ALTER SYSTEM SET shared_buffers = ‘8GB’ FORCE;
> NOTICE:  Changing shared_buffers only takes effect after a server
> restart.
> ALTER SYSTEM
> 
> Will bad examples pop up in the Internet that just use FORCE all the
> time?  Sure they will, and people will cut and paste them without
> paying
> attention.  I don't see why that possibility has to block this feature
> from being adopted though.  That line of thinking leads toward removing
> trust authentication, because that's similarly abused with cut and
> paste
> tutorials.


With Regards,
Amit Kapila.

Re: Unsafe GUCs and ALTER SYSTEM WAS: Re: ALTER SYSTEM SET

Amit Kapila <amit.kapila@huawei.com> — 2013-08-07T03:44:02Z

On Wednesday, August 07, 2013 8:01 AM Bruce Momjian wrote:
> On Tue, Aug  6, 2013 at 09:24:47PM -0400, Greg Smith wrote:
> > # ALTER SYSTEM SET shared_buffers = ‘8GB’ FORCE;
> > NOTICE:  Changing shared_buffers only takes effect after a server
> restart.
> > ALTER SYSTEM
> >
> > Will bad examples pop up in the Internet that just use FORCE all the
> > time?  Sure they will, and people will cut and paste them without
> > paying attention.  I don't see why that possibility has to block
> > this feature from being adopted though.  That line of thinking leads
> > toward removing trust authentication, because that's similarly
> > abused with cut and paste tutorials.
> 
> We already have six levels of GUC settings:
> 
> 	postgresql.conf
> 	user
> 	database
> 	session
> 	function
> 	subtransaction
> 
> If we add ALTER SYSTEM SET and config.d, we would then have eight.
> ALTER SYSTEM SET seems to add an entirely new set of behaviors and
> complexity.  Is that really what we want?
> 
> If we do this, perhaps we should unconditionally just print the file
> name they have to delete to undo the operation in case the server
> doesn't start;  I am unclear we can clearly identify all the GUC
> settings that could cause a server not to start.  

> Also, I think we need
> a SHOW SYSTEM command so users can see their settings via SQL.

Although users can see the settings in pg_settings as it has sourcefile, but such a command can
be useful.

> FYI, ALTER SYSTEM SET is hitting the same problems we would have if
> pg_hba.conf were set in SQL and in flat files.

With Regards,
Amit Kapila.

Re: Unsafe GUCs and ALTER SYSTEM WAS: Re: ALTER SYSTEM SET

Alvaro Herrera <alvherre@2ndquadrant.com> — 2013-08-07T14:59:05Z

Bruce Momjian escribió:

> We already have six levels of GUC settings:
> 
> 	postgresql.conf
> 	user
> 	database
> 	session
> 	function
> 	subtransaction
> 
> If we add ALTER SYSTEM SET and config.d, we would then have eight. 

Actually, conf.d (not config.d) would just be reported as a file, and
the "file" column in the pg_settings view would show which file it is.
That's how it's done for postgresql.conf and any other file it includes,
so that seems a pretty reasonable thing to me.

> ALTER SYSTEM SET seems to add an entirely new set of behaviors and
> complexity.  Is that really what we want?

I had imagined that ALTER SYSTEM SET would be represented in the same
way as above, i.e. just be a different source file.  But thinking more
about it now, that doesn't make sense, because those files might be in a
completely different base directory, and the file names shouldn't even
be exposed to the user in the first place; so clearly ALTER SYSTEM
should show up differently in pg_settings.  Displaying each option's
full path seems useful for troubleshooting, as you say:

> If we do this, perhaps we should unconditionally just print the file
> name they have to delete to undo the operation in case the server
> doesn't start;

However, bear in mind that if the DBA is administering a server through
ALTER SYSTEM and they don't have shell access, they might just be
screwed if they bollix the system and they lose access.  Knowing what
file you have to delete does you no good if you can't actually delete it.

> I am unclear we can clearly identify all the GUC
> settings that could cause a server not to start.  Also, I think we need
> a SHOW SYSTEM command so users can see their settings via SQL.

Not sure about this.  SHOW normally just displays the current value,
nothing more.  If you want more details, there's the pg_settings view
with complete information.

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Re: Unsafe GUCs and ALTER SYSTEM WAS: Re: ALTER SYSTEM SET

Bruce Momjian <bruce@momjian.us> — 2013-08-07T15:08:34Z

On Wed, Aug  7, 2013 at 10:59:05AM -0400, Alvaro Herrera wrote:
> > If we do this, perhaps we should unconditionally just print the file
> > name they have to delete to undo the operation in case the server
> > doesn't start;
> 
> However, bear in mind that if the DBA is administering a server through
> ALTER SYSTEM and they don't have shell access, they might just be
> screwed if they bollix the system and they lose access.  Knowing what
> file you have to delete does you no good if you can't actually delete it.

My point was that if we tell them what file they have to edit to undo a
problem, then at least then know that there is chance this will require
shell access, and they can undo the change right there (with SET var =
DEFAULT?) e.g.:

WARNING:  If this change prevents the server from starting, you will
          need to edit postgresql.conf.auto to undo the change

> > I am unclear we can clearly identify all the GUC
> > settings that could cause a server not to start.  Also, I think we need
> > a SHOW SYSTEM command so users can see their settings via SQL.
> 
> Not sure about this.  SHOW normally just displays the current value,
> nothing more.  If you want more details, there's the pg_settings view
> with complete information.

The issue is that if we give users the ability to set values via SQL, we
should at least document how they can view the things they set via SQL,
and pg_settings doesn't do that because it just shows the _active_
value, which might have overridden the SQL-set value.

Right now we can see settings in postgresql.conf by viewing the file,
and ALTER USER/DATABASE via system tables.  Having no API to view
SQL-set values except viewing a flat file seems counter-productive.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + It's impossible for everything to be true. +

Re: Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila@huawei.com> — 2013-08-08T09:51:56Z

On Monday, August 05, 2013 10:12 PM Josh Berkus wrote:
> All,
> 
> To limit the argument here, let's please not argue about things which
> people already agree on.  So:
> 
> We seem to have consensus around:
> 
> A) the inadvisability of storing GUCs in a system catalog.
> 
> B) the utility of a conf.d in /etc/ which may have nothing to do with
> ALTER SYSTEM SET
> 
> C) that any file for ALTER SYSTEM SET go in $PGDATA somewhere.
> 
> What we are still arguing about:

Summarization of discussion, please add if I have missed or misunderstood any conclusive point:

> D) one-big-file vs. file-per-setting
     Here I think there are no points against one-big-file and not much in favor of file-per-setting.
     I believe there is enough support for one-big-file that there is no need to change it to file-per-setting.

> E) whether "unsafe" settings or "restart" settings should be allowed in
> ALTER SYSTEM SET.
     a. Settings should continue to be in flat file. This will allow user to change settings incase anything goes wrong.
     b. Few unsafe settings shouldn't be allowed to change, but there can be an option (new keyword FORCE) to change if user wants it.
        Initial set of parameters suggested by Stephen Frost:
        http://www.postgresql.org/message-id/20130802145442.GB2706@tamriel.snowman.net
     

> F) whether admins need the ability to disable ALTER SYSTEM SET.
     a. UI into contrib module, people who don't want doesn't include it
     b. Have an include file mechanism, so that user can comment the include in postgresql.conf and disable it.

Some new requirements
1. Show System command to separately show the values set by Alter System
2. append-to-this-GUC flag instead of overwriting the previous value.  
 
With Regards,
Amit Kapila.

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Robert Haas <robertmhaas@gmail.com> — 2013-08-08T13:33:07Z

On Sun, Aug 4, 2013 at 4:26 PM, Dimitri Fontaine <dimitri@2ndquadrant.fr> wrote:
> Alvaro Herrera <alvherre@2ndquadrant.com> writes:
>> I remind you that event triggers are not fired for global objects
>> such as databases and roles.  Do you intend to lift that restriction?
>
> That's not on my TODO list for 9.4. My understanding about implementing
> that is:
>
>   - we agree that it would be nice to have,
>   - it requires a separate *shared* catalog for event triggers.
>
> What I'm yet unsure about is that there's a consensus that the use cases
> are worthy of a new shared catalog in the system. Also I didn't look how
> hard it is to actually provide for it.

A new shared catalog wouldn't actually help, because the actual
procedure to be run has to live in pg_proc, which is not shared.  And
that has references to all sorts of other things (like pg_language)
that aren't shared either.

I think the question isn't really a technical one so much as one of
policy.  We could quite easily allow event triggers on shared objects.
 The reason I suggested that we NOT allow that is because then
operations on those objects would behave differently depending on
which database you've attached.  If you've attached a database with an
event trigger, you get the special behavior; otherwise, you don't.  I
feared creating user confusion, there, and the use cases seemed
marginal anyway.  But if there's a sufficient consensus that such a
thing is useful and non-confusing, I'll give way.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-09T12:44:05Z

* Robert Haas (robertmhaas@gmail.com) wrote:
> On Sun, Aug 4, 2013 at 4:26 PM, Dimitri Fontaine <dimitri@2ndquadrant.fr> wrote:
> > What I'm yet unsure about is that there's a consensus that the use cases
> > are worthy of a new shared catalog in the system. Also I didn't look how
> > hard it is to actually provide for it.
> 
> A new shared catalog wouldn't actually help, because the actual
> procedure to be run has to live in pg_proc, which is not shared.  And
> that has references to all sorts of other things (like pg_language)
> that aren't shared either.

A shared catalog which defined which *database* to run the trigger in,
with a way to fire off a new backend worker in that database and tell it
to run the trigger, might be interesting and would deal with the issue
that the trigger would behave differently depending on the database
connected to.  That would bring along other issues, of course, but it
seemed an interesting enough idea to mention.

	Thanks,

		Stephen

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Dimitri Fontaine <dimitri@2ndquadrant.fr> — 2013-08-09T13:08:45Z

Stephen Frost <sfrost@snowman.net> writes:
> A shared catalog which defined which *database* to run the trigger in,
> with a way to fire off a new backend worker in that database and tell it
> to run the trigger, might be interesting and would deal with the issue
> that the trigger would behave differently depending on the database
> connected to.  That would bring along other issues, of course, but it
> seemed an interesting enough idea to mention.

Yeah, I like that approach. The only drawback is that it requires having
PLproxy in core first, or something like Foreign Functions or something.

Regards,
-- 
Dimitri Fontaine
http://2ndQuadrant.fr     PostgreSQL : Expertise, Formation et Support

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

David Fetter <david@fetter.org> — 2013-08-09T14:58:16Z

On Fri, Aug 09, 2013 at 03:08:45PM +0200, Dimitri Fontaine wrote:
> Stephen Frost <sfrost@snowman.net> writes:
> > A shared catalog which defined which *database* to run the trigger
> > in, with a way to fire off a new backend worker in that database
> > and tell it to run the trigger, might be interesting and would
> > deal with the issue that the trigger would behave differently
> > depending on the database connected to.  That would bring along
> > other issues, of course, but it seemed an interesting enough idea
> > to mention.
> 
> Yeah, I like that approach. The only drawback is that it requires
> having PLproxy in core first, or something like Foreign Functions or
> something.

SQL/MED does define such an API.  Whether we find it useful enough to
make it the default way of doing things is a separate matter.  I'll do
some research.

Cheers,
David.
-- 
David Fetter <david@fetter.org> http://fetter.org/
Phone: +1 415 235 3778  AIM: dfetter666  Yahoo!: dfetter
Skype: davidfetter      XMPP: david.fetter@gmail.com
iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Robert Haas <robertmhaas@gmail.com> — 2013-08-16T17:03:10Z

On Fri, Aug 9, 2013 at 8:44 AM, Stephen Frost <sfrost@snowman.net> wrote:
> * Robert Haas (robertmhaas@gmail.com) wrote:
>> On Sun, Aug 4, 2013 at 4:26 PM, Dimitri Fontaine <dimitri@2ndquadrant.fr> wrote:
>> > What I'm yet unsure about is that there's a consensus that the use cases
>> > are worthy of a new shared catalog in the system. Also I didn't look how
>> > hard it is to actually provide for it.
>>
>> A new shared catalog wouldn't actually help, because the actual
>> procedure to be run has to live in pg_proc, which is not shared.  And
>> that has references to all sorts of other things (like pg_language)
>> that aren't shared either.
>
> A shared catalog which defined which *database* to run the trigger in,
> with a way to fire off a new backend worker in that database and tell it
> to run the trigger, might be interesting and would deal with the issue
> that the trigger would behave differently depending on the database
> connected to.  That would bring along other issues, of course, but it
> seemed an interesting enough idea to mention.

Eh, maybe.  I'm not sure there's enough use case for that to justify
the amount of infrastructure it would require.  I'm happy with the
recent enhancements to background workers, but there's an awful lot of
ground to cover between that and what you're proposing.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Alvaro Herrera <alvherre@2ndquadrant.com> — 2013-08-16T17:42:17Z

Robert Haas escribió:
> On Fri, Aug 9, 2013 at 8:44 AM, Stephen Frost <sfrost@snowman.net> wrote:
> > A shared catalog which defined which *database* to run the trigger in,
> > with a way to fire off a new backend worker in that database and tell it
> > to run the trigger, might be interesting and would deal with the issue
> > that the trigger would behave differently depending on the database
> > connected to.  That would bring along other issues, of course, but it
> > seemed an interesting enough idea to mention.
> 
> Eh, maybe.  I'm not sure there's enough use case for that to justify
> the amount of infrastructure it would require.  I'm happy with the
> recent enhancements to background workers, but there's an awful lot of
> ground to cover between that and what you're proposing.

Yeah, agreed.  There's a lot of infrastructure required for this; it
seems hard to justify it only to support disabling ALTER SYSTEM.

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila16@gmail.com> — 2013-08-17T06:10:26Z

On Fri, Aug 16, 2013 at 11:12 PM, Alvaro Herrera
<alvherre@2ndquadrant.com> wrote:
> Robert Haas escribió:
>> On Fri, Aug 9, 2013 at 8:44 AM, Stephen Frost <sfrost@snowman.net> wrote:
>> > A shared catalog which defined which *database* to run the trigger in,
>> > with a way to fire off a new backend worker in that database and tell it
>> > to run the trigger, might be interesting and would deal with the issue
>> > that the trigger would behave differently depending on the database
>> > connected to.  That would bring along other issues, of course, but it
>> > seemed an interesting enough idea to mention.
>>
>> Eh, maybe.  I'm not sure there's enough use case for that to justify
>> the amount of infrastructure it would require.  I'm happy with the
>> recent enhancements to background workers, but there's an awful lot of
>> ground to cover between that and what you're proposing.
>
> Yeah, agreed.  There's a lot of infrastructure required for this; it
> seems hard to justify it only to support disabling ALTER SYSTEM.

I think disabling ALTER SYSTEM can be more easily supported if we
choose one of below options:
a. UI into contrib module, people who don't want doesn't include it
b. Have an include file mechanism, so that user can comment the
include in postgresql.conf and disable it.

If I remember correctly, earlier you mentioned that by default auto
file should be parsed after postgresql.conf, but how about reverting
to previous mechanism of "include" such that if the file is mentioned
in include then it will be parsed, else will be ignored. I think this
can be reasonable way to disable.

With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-20T00:17:26Z

* Alvaro Herrera (alvherre@2ndquadrant.com) wrote:
> Robert Haas escribió:
> > On Fri, Aug 9, 2013 at 8:44 AM, Stephen Frost <sfrost@snowman.net> wrote:
> > > A shared catalog which defined which *database* to run the trigger in,
> > > with a way to fire off a new backend worker in that database and tell it
> > > to run the trigger, might be interesting and would deal with the issue
> > > that the trigger would behave differently depending on the database
> > > connected to.  That would bring along other issues, of course, but it
> > > seemed an interesting enough idea to mention.
> > 
> > Eh, maybe.  I'm not sure there's enough use case for that to justify
> > the amount of infrastructure it would require.  I'm happy with the
> > recent enhancements to background workers, but there's an awful lot of
> > ground to cover between that and what you're proposing.
> 
> Yeah, agreed.  There's a lot of infrastructure required for this; it
> seems hard to justify it only to support disabling ALTER SYSTEM.

This would be support for generic triggers on shared catalog updates,
which would certainly extend beyond just ALTER SYSTEM and into ALTER
DATABASE and ALTER ROLE..

	Thanks,

		Stephen

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-20T00:19:12Z

* Amit Kapila (amit.kapila16@gmail.com) wrote:
> I think disabling ALTER SYSTEM can be more easily supported if we
> choose one of below options:
> a. UI into contrib module, people who don't want doesn't include it
> b. Have an include file mechanism, so that user can comment the
> include in postgresql.conf and disable it.

Either of these would be good options, imv.  In my new world order, it'd
be a '#include' in the postgresql.conf- which is commented out by
default.

	Thanks,

		Stephen

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila16@gmail.com> — 2013-08-20T02:51:49Z

On Tue, Aug 20, 2013 at 5:49 AM, Stephen Frost <sfrost@snowman.net> wrote:
> * Amit Kapila (amit.kapila16@gmail.com) wrote:
>> I think disabling ALTER SYSTEM can be more easily supported if we
>> choose one of below options:
>> a. UI into contrib module, people who don't want doesn't include it
>> b. Have an include file mechanism, so that user can comment the
>> include in postgresql.conf and disable it.
>
> Either of these would be good options, imv.  In my new world order, it'd
> be a '#include' in the postgresql.conf- which is commented out by
> default.

If both are okay, then I would like to go with second option (include
file mechanism).
I think by default, we should allow auto file to be included and if
user faces any problem or otherwise,
then he can decide to disable it.

With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-20T02:57:28Z

* Amit Kapila (amit.kapila16@gmail.com) wrote:
> If both are okay, then I would like to go with second option (include
> file mechanism).
> I think by default, we should allow auto file to be included and if
> user faces any problem or otherwise,
> then he can decide to disable it.

If it's enabled by default, then we need to ship an 'auto' file which is
empty by default...  I don't particularly like that (implies that an
empty file is acceptable and not an error condition, which it otherwise
probably would be...).  I'm still not a big fan of enabling it by default
tho.

	Thanks,

		Stephen

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila16@gmail.com> — 2013-08-20T03:08:52Z

On Tue, Aug 20, 2013 at 8:27 AM, Stephen Frost <sfrost@snowman.net> wrote:
> * Amit Kapila (amit.kapila16@gmail.com) wrote:
>> If both are okay, then I would like to go with second option (include
>> file mechanism).
>> I think by default, we should allow auto file to be included and if
>> user faces any problem or otherwise,
>> then he can decide to disable it.
>
> If it's enabled by default, then we need to ship an 'auto' file which is
> empty by default...

initdb will create the empty auto file. If we don't enable by default,
then if user uses
ALTER SYSTEM and do sighup/restart, changed config parameter values
will not come into affect
until he manually enables it by removing '#' from '#include'.

> I don't particularly like that (implies that an
> empty file is acceptable and not an error condition, which it otherwise
> probably would be...).  I'm still not a big fan of enabling it by default
> tho.

With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-20T03:13:49Z

* Amit Kapila (amit.kapila16@gmail.com) wrote:
> initdb will create the empty auto file. If we don't enable by default,
> then if user uses
> ALTER SYSTEM and do sighup/restart, changed config parameter values
> will not come into affect
> until he manually enables it by removing '#' from '#include'.

That's a pretty simple thing to detect and return an ERROR case on
though.

	Thanks,

		Stephen

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila16@gmail.com> — 2013-08-20T03:51:26Z

On Tue, Aug 20, 2013 at 8:43 AM, Stephen Frost <sfrost@snowman.net> wrote:
> * Amit Kapila (amit.kapila16@gmail.com) wrote:
>> initdb will create the empty auto file. If we don't enable by default,
>> then if user uses
>> ALTER SYSTEM and do sighup/restart, changed config parameter values
>> will not come into affect
>> until he manually enables it by removing '#' from '#include'.
>
> That's a pretty simple thing to detect and return an ERROR case on
> though.

   Do you mean to say, that when user uses ALTER SYSTEM and include is
disabled, then we can give ERROR in
   ALTER SYSTEM, that to reflect the values, he need to enable it?
   Actually in previous version of patch, there was WARNING on such a condition.
   I agree that it can be detected and we issue Warning or Error, but
I think it will be an un-necessary inconvenience
   for user. However if you still vote for disable by default, I will
modify the patch that way only.

The other part which needs suggestions is about un-safe parameters:
I think for first version of feature, we can decide on some minimum
list of parameters that will be considere un-safe and user will not be
allowed to modify them with ALTER SYSTEM.
To start with we can consider below 4 parameters in that category and
later on keep adding more parameters:
data_directory, listen_addresses, port, shared_buffers.
These are from the list you provided at link:
http://www.postgresql.org/message-id/20130802145442.GB2706@tamriel.snowman.net

I have no strong reason why only these 4 parameters, we can choose
other parameters as well. The only thing is I want to make a base for
this feature and then later we can enhance it as per below points:
a. add more parameters to un-safe category
b. user can view non-modifiable parameters (may be in pg_settings)
c. some way such that if user wants to take risk of server not getting
started, he should allow to modify such parameters (may be with
additional syntax in ALTER SYSTEM .. FORCE as suggested by Greg
Smith).

With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-20T04:03:50Z

* Amit Kapila (amit.kapila16@gmail.com) wrote:
>    Do you mean to say, that when user uses ALTER SYSTEM and include is
> disabled, then we can give ERROR in
>    ALTER SYSTEM, that to reflect the values, he need to enable it?
>    Actually in previous version of patch, there was WARNING on such a condition.
>    I agree that it can be detected and we issue Warning or Error, but
> I think it will be an un-necessary inconvenience
>    for user. However if you still vote for disable by default, I will
> modify the patch that way only.

It hardly seems unnecessary..  Otherwise we accept a command that won't
actually do anything, yes?  We usually notice or warn, at least, on such
cases (see similar things around GRANT/REVOKE...).

> The other part which needs suggestions is about un-safe parameters:
> I think for first version of feature, we can decide on some minimum
> list of parameters that will be considere un-safe and user will not be
> allowed to modify them with ALTER SYSTEM.

I do feel like it's easier to add things to the 'allowed' set later,
should we see some good justification for it, than to disallow things
already allowed.

> To start with we can consider below 4 parameters in that category and
> later on keep adding more parameters:
> data_directory, listen_addresses, port, shared_buffers.

What about include directives?

	Thanks,

		Stephen

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila16@gmail.com> — 2013-08-20T04:33:54Z

On Tue, Aug 20, 2013 at 9:33 AM, Stephen Frost <sfrost@snowman.net> wrote:
> * Amit Kapila (amit.kapila16@gmail.com) wrote:
>>    Do you mean to say, that when user uses ALTER SYSTEM and include is
>> disabled, then we can give ERROR in
>>    ALTER SYSTEM, that to reflect the values, he need to enable it?
>>    Actually in previous version of patch, there was WARNING on such a condition.
>>    I agree that it can be detected and we issue Warning or Error, but
>> I think it will be an un-necessary inconvenience
>>    for user. However if you still vote for disable by default, I will
>> modify the patch that way only.
>
> It hardly seems unnecessary..  Otherwise we accept a command that won't
> actually do anything, yes?

Okay, let us decide how things will be if we disable by default:
   1. initdb won't create any empty auto file, rather the file will be
created first time user uses ALTER SYSTEM.
   2. Alter system should issue warning, if it detects that feature is
disabled (for this we need to error detection code
       during parsing of postgresql.conf as it was previously)
   3. postgresql.conf will contain include directive in below form:
       #include = 'postgresql.auto.conf'
       Whenever user wants to use Alter System, he needs to enable it
after first time using ALTER SYSTEM.

One question here, if somebody just enables it without using ALTER
SYSTEM, should it throw any error on not finding auto file or just
ignore it?

> We usually notice or warn, at least, on such
> cases (see similar things around GRANT/REVOKE...).
>
>> The other part which needs suggestions is about un-safe parameters:
>> I think for first version of feature, we can decide on some minimum
>> list of parameters that will be considere un-safe and user will not be
>> allowed to modify them with ALTER SYSTEM.
>
> I do feel like it's easier to add things to the 'allowed' set later,
> should we see some good justification for it, than to disallow things
> already allowed.

   Here the only consideration in my mind is about question of
usefulness of this feature and getting agreement on
   long list of un-safe parameters.
   As we are already proposing/planing to give FORCE option with which
user can change config parameters, the
   question of usefulness can be answered with that, but how about other?

>> To start with we can consider below 4 parameters in that category and
>> later on keep adding more parameters:
>> data_directory, listen_addresses, port, shared_buffers.
>
> What about include directives?

Sorry, I didn't get which parameters you are referring by include directives?


With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Alvaro Herrera <alvherre@2ndquadrant.com> — 2013-08-20T04:56:29Z

Amit Kapila escribió:

>    3. postgresql.conf will contain include directive in below form:
>        #include = 'postgresql.auto.conf'
>        Whenever user wants to use Alter System, he needs to enable it
> after first time using ALTER SYSTEM.

This seems wrong to me.  If the auto file is read by an include line in
postgresql.conf, what is its priority w.r.t. files placed in an
hypothetical conf.d directory?  Hopefully snippets put in conf.d/ by
puppet/chef will override the settings in postgresql.conf (i.e. conf.d/
should be processed after postgresql.conf, not before); and hopefully
ALTER SYSTEM will in turn override conf.d.  I see no way to have ALTER
SYSTEM handled by an include line, yet still have it override conf.d.

If we want to make ALTER SYSTEM disable-able from postgresql.conf, I
think it should be an explicit option, something like
enable_alter_system = on
or something like that.

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila16@gmail.com> — 2013-08-20T05:37:48Z

On Tue, Aug 20, 2013 at 10:26 AM, Alvaro Herrera
<alvherre@2ndquadrant.com> wrote:
> Amit Kapila escribió:
>
>>    3. postgresql.conf will contain include directive in below form:
>>        #include = 'postgresql.auto.conf'
>>        Whenever user wants to use Alter System, he needs to enable it
>> after first time using ALTER SYSTEM.
>
> This seems wrong to me.  If the auto file is read by an include line in
> postgresql.conf, what is its priority w.r.t. files placed in an
> hypothetical conf.d directory?  Hopefully snippets put in conf.d/ by
> puppet/chef will override the settings in postgresql.conf (i.e. conf.d/
> should be processed after postgresql.conf, not before); and hopefully
> ALTER SYSTEM will in turn override conf.d.  I see no way to have ALTER
> SYSTEM handled by an include line, yet still have it override conf.d.

I think the behavior of overriding can be controlled by location of
include directive of conf.d directory or include location of auto file
whichever is later will override.
User can change the priority by changing location of include directive.

> If we want to make ALTER SYSTEM disable-able from postgresql.conf, I
> think it should be an explicit option, something like
> enable_alter_system = on
> or something like that.

I think this would be an easier way to handle disabling, but the only
question is whether this feature deserves a new config parameter?

With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-20T12:51:21Z

* Amit Kapila (amit.kapila16@gmail.com) wrote:
> Okay, let us decide how things will be if we disable by default:
>    1. initdb won't create any empty auto file, rather the file will be
> created first time user uses ALTER SYSTEM.

I'm not particularly set on this..  Why not create the file initially?

>    2. Alter system should issue warning, if it detects that feature is
> disabled (for this we need to error detection code
>        during parsing of postgresql.conf as it was previously)

I agree that it should complain through a warning or perhaps an error
(to be honest, I like error more, but there may be good reasons against
that).

>    3. postgresql.conf will contain include directive in below form:
>        #include = 'postgresql.auto.conf'
>        Whenever user wants to use Alter System, he needs to enable it
> after first time using ALTER SYSTEM.

That I don't like..  You should be able to enable it and have things
"work" without having to run ALTER SYSTEM first..

> One question here, if somebody just enables it without using ALTER
> SYSTEM, should it throw any error on not finding auto file or just
> ignore it?

I'd prefer that it error if it can't find an included file.

> > What about include directives?
> 
> Sorry, I didn't get which parameters you are referring by include directives?

Literally, the above 'include' in postgresql.conf.  Would that only be
dealt with as a parse-time thing, or should it be more like a GUC which
could then be set through ALTER SYSTEM?

	Thanks,

		Stephen

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-20T13:08:08Z

* Alvaro Herrera (alvherre@2ndquadrant.com) wrote:
> Amit Kapila escribió:
> >    3. postgresql.conf will contain include directive in below form:
> >        #include = 'postgresql.auto.conf'
> >        Whenever user wants to use Alter System, he needs to enable it
> > after first time using ALTER SYSTEM.
> 
> This seems wrong to me.  If the auto file is read by an include line in
> postgresql.conf, what is its priority w.r.t. files placed in an
> hypothetical conf.d directory?  

This could be handled by a simple ordering with "last one wins".  I
don't particularly like that though..  My gut feeling is that I'd like
something to complain if there's more than one value set for the same
GUC..

> Hopefully snippets put in conf.d/ by
> puppet/chef will override the settings in postgresql.conf (i.e. conf.d/
> should be processed after postgresql.conf, not before); and hopefully
> ALTER SYSTEM will in turn override conf.d.  I see no way to have ALTER
> SYSTEM handled by an include line, yet still have it override conf.d.

With includedir and include directives, and postgresql.conf setting a
defined ordering, with last-wins, you could simply have the 'includedir'
for conf.d come before the 'include' for auto.conf.

> If we want to make ALTER SYSTEM disable-able from postgresql.conf, I
> think it should be an explicit option, something like
> enable_alter_system = on
> or something like that.

I really don't like this approach- I'd much rather we have a general
include mechanism than special "alter-system" GUCs.

	Thanks,

		Stephen

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila16@gmail.com> — 2013-08-20T13:13:50Z

On Tue, Aug 20, 2013 at 6:21 PM, Stephen Frost <sfrost@snowman.net> wrote:
> * Amit Kapila (amit.kapila16@gmail.com) wrote:
>> Okay, let us decide how things will be if we disable by default:
>>    1. initdb won't create any empty auto file, rather the file will be
>> created first time user uses ALTER SYSTEM.
>
> I'm not particularly set on this..  Why not create the file initially?
   If by default this feature needs to be disabled, then it is not
must to have at initdb time.
   OTOH if we want to enable it by default, then we need to get this
created at initdb time else it will give error during
   system startup (an error can occur during startup when it will
parse postgresql.conf and didn't find the file
   mentioned in include directive).

   Also you mentioned below line upthread which I understood as you
don't like idea of creating empty file at initdb
   time:
   "If it's enabled by default, then we need to ship an 'auto' file which is
    empty by default...  I don't particularly like that"


>>    2. Alter system should issue warning, if it detects that feature is
>> disabled (for this we need to error detection code
>>        during parsing of postgresql.conf as it was previously)
>
> I agree that it should complain through a warning or perhaps an error
> (to be honest, I like error more, but there may be good reasons against
> that).
>
>>    3. postgresql.conf will contain include directive in below form:
>>        #include = 'postgresql.auto.conf'
>>        Whenever user wants to use Alter System, he needs to enable it
>> after first time using ALTER SYSTEM.
>
> That I don't like..  You should be able to enable it and have things
> "work" without having to run ALTER SYSTEM first..

   This was actually linked to point 1 mentioned above, if we create
empty file at initdb time, then there should not be
   any problem.

   One other option to disable as suggested by Alvaro is have another
config parameter to enable/disable Alter
   System, if we can do that way, the solution will be much neater.

>> One question here, if somebody just enables it without using ALTER
>> SYSTEM, should it throw any error on not finding auto file or just
>> ignore it?
>
> I'd prefer that it error if it can't find an included file.
>
>> > What about include directives?
>>
>> Sorry, I didn't get which parameters you are referring by include directives?
>
> Literally, the above 'include' in postgresql.conf.  Would that only be
> dealt with as a parse-time thing, or should it be more like a GUC which
> could then be set through ALTER SYSTEM?

 I think if we choose to use include directive as a way to
enable/disable this feature, it will not be good to allow change of
this parameter with Alter System.


With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-20T13:25:14Z

* Amit Kapila (amit.kapila16@gmail.com) wrote:
> On Tue, Aug 20, 2013 at 6:21 PM, Stephen Frost <sfrost@snowman.net> wrote:
> > I'm not particularly set on this..  Why not create the file initially?
>    If by default this feature needs to be disabled, then it is not
> must to have at initdb time.

I don't see how the two are related.  You could never use two-phase
commit (could even disable it), yet we still create things in the data
directory to support it.  Having a file in the data directory which
isn't used by default isn't that big a deal, imv.

>    Also you mentioned below line upthread which I understood as you
> don't like idea of creating empty file at initdb
>    time:
>    "If it's enabled by default, then we need to ship an 'auto' file which is
>     empty by default...  I don't particularly like that"

What I didn't like was having an empty file be accepted as 'valid', but
you need to have some kind of bootstrap mechanism.  Telling users "run
this command and then ignore the warning" is certainly bad.  A better
option might be to have a *non-empty* auto.conf be generated, where all
it has in it is some kind of identifier, perhaps even just
"enable_alter_system = on" which we could then key off of to see if
ALTER SYSTEM has been set up to be allowed or not.

>  I think if we choose to use include directive as a way to
> enable/disable this feature, it will not be good to allow change of
> this parameter with Alter System.

I agree, but then we need to add it to the list of things ALTER SYSTEM
can't modify (if the include is implemented as a GUC, that is; I've not
looked).

	Thanks,

		Stephen

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila16@gmail.com> — 2013-08-20T14:09:51Z

On Tue, Aug 20, 2013 at 6:55 PM, Stephen Frost <sfrost@snowman.net> wrote:
> * Amit Kapila (amit.kapila16@gmail.com) wrote:
>> On Tue, Aug 20, 2013 at 6:21 PM, Stephen Frost <sfrost@snowman.net> wrote:
>> > I'm not particularly set on this..  Why not create the file initially?
>>    If by default this feature needs to be disabled, then it is not
>> must to have at initdb time.
>
> I don't see how the two are related.  You could never use two-phase
> commit (could even disable it), yet we still create things in the data
> directory to support it.  Having a file in the data directory which
> isn't used by default isn't that big a deal, imv.

   Yes, it can be done, what I wanted to say it is not a "must",
rather a good to have thing.

>>    Also you mentioned below line upthread which I understood as you
>> don't like idea of creating empty file at initdb
>>    time:
>>    "If it's enabled by default, then we need to ship an 'auto' file which is
>>     empty by default...  I don't particularly like that"
>
> What I didn't like was having an empty file be accepted as 'valid', but
> you need to have some kind of bootstrap mechanism.  Telling users "run
> this command and then ignore the warning" is certainly bad.  A better
> option might be to have a *non-empty* auto.conf be generated, where all
> it has in it is some kind of identifier, perhaps even just
> "enable_alter_system = on" which we could then key off of to see if
> ALTER SYSTEM has been set up to be allowed or not.

Wouldn't it be complicated to handle this way rather then by having
include directive.
If include directive is enabled then the autofile will be read else no
need to read it.

If we want to have separate identifier to judge ALTER SYSTEM is enable
or not, then it is better to go with a new GUC.

>>  I think if we choose to use include directive as a way to
>> enable/disable this feature, it will not be good to allow change of
>> this parameter with Alter System.
>
> I agree, but then we need to add it to the list of things ALTER SYSTEM
> can't modify (if the include is implemented as a GUC, that is; I've not
> looked).

   I have checked and it seems to be just parse time thing.

With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-20T14:21:48Z

* Amit Kapila (amit.kapila16@gmail.com) wrote:
> Wouldn't it be complicated to handle this way rather then by having
> include directive.

You misunderstood.  I was suggesting a setup along these lines:

postgresql.conf:
  # include = 'auto.conf'  # Disabled by default

auto.conf:
  ####################################################
  # MANAGED BY ALTER SYSTEM -- DO NOT MODIFY BY HAND
  ####################################################

  # auto.conf not processed unless included by postgresql.conf
  # If included by postgresql.conf, then this will turn on the
  # ALTER SYSTEM command.
  enable_alter_system = on 

Which would give us the ability to independently turn on/off ALTER
SYSTEM from including auto.conf.

	Thanks,

		Stephen

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila16@gmail.com> — 2013-08-20T14:41:44Z

On Tue, Aug 20, 2013 at 7:51 PM, Stephen Frost <sfrost@snowman.net> wrote:
> * Amit Kapila (amit.kapila16@gmail.com) wrote:
>> Wouldn't it be complicated to handle this way rather then by having
>> include directive.
>
> You misunderstood.  I was suggesting a setup along these lines:
>
> postgresql.conf:
>   # include = 'auto.conf'  # Disabled by default
>
> auto.conf:
>   ####################################################
>   # MANAGED BY ALTER SYSTEM -- DO NOT MODIFY BY HAND
>   ####################################################
>
>   # auto.conf not processed unless included by postgresql.conf
>   # If included by postgresql.conf, then this will turn on the
>   # ALTER SYSTEM command.
>   enable_alter_system = on
>
> Which would give us the ability to independently turn on/off ALTER
> SYSTEM from including auto.conf.

So let me try to explain what I understood from above:

1. enable_alter_system a new GUC whose default value =off.
2. Alter System will check this variable and return error (not
allowed), if this parameter is off.
3. Now if user enables include directive in postgresql.conf, it will
enable Alter System as value of
    enable_alter_system is on.
4. User can run Alter System command to disable Alter System
"enable_alter_system = off".
    Now even though include directive is enabled, but new Alter System
commands will not work, however
    existing parameter's take into effect on restart/sighup.


With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-20T14:44:49Z

* Amit Kapila (amit.kapila16@gmail.com) wrote:
> So let me try to explain what I understood from above:
> 
> 1. enable_alter_system a new GUC whose default value =off.
> 2. Alter System will check this variable and return error (not
> allowed), if this parameter is off.
> 3. Now if user enables include directive in postgresql.conf, it will
> enable Alter System as value of
>     enable_alter_system is on.
> 4. User can run Alter System command to disable Alter System
> "enable_alter_system = off".
>     Now even though include directive is enabled, but new Alter System
> commands will not work, however
>     existing parameter's take into effect on restart/sighup.

Yes.  Not sure that it'd be terribly likely for a user to do that, but
if they do it, so be it.

		Thanks,

			Stephen

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Alvaro Herrera <alvherre@2ndquadrant.com> — 2013-08-20T15:05:47Z

Stephen Frost escribió:
> * Alvaro Herrera (alvherre@2ndquadrant.com) wrote:

> > Hopefully snippets put in conf.d/ by
> > puppet/chef will override the settings in postgresql.conf (i.e. conf.d/
> > should be processed after postgresql.conf, not before); and hopefully
> > ALTER SYSTEM will in turn override conf.d.  I see no way to have ALTER
> > SYSTEM handled by an include line, yet still have it override conf.d.
> 
> With includedir and include directives, and postgresql.conf setting a
> defined ordering, with last-wins, you could simply have the 'includedir'
> for conf.d come before the 'include' for auto.conf.

Uh, I was thinking that conf.d would be read by the system
automatically, not because of an includedir line in postgresql.conf.

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-20T15:11:16Z

* Alvaro Herrera (alvherre@2ndquadrant.com) wrote:
> Stephen Frost escribió:
> > With includedir and include directives, and postgresql.conf setting a
> > defined ordering, with last-wins, you could simply have the 'includedir'
> > for conf.d come before the 'include' for auto.conf.
> 
> Uh, I was thinking that conf.d would be read by the system
> automatically, not because of an includedir line in postgresql.conf.

I'd much rather have an includedir directive than some hard-coded or
command-line option to read the directory..  The directory should live
in /etc/postgresql/X.Y/cluster/ on at least Debian derivatives..

	Thanks,

		Stephen

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Alvaro Herrera <alvherre@2ndquadrant.com> — 2013-08-20T15:14:38Z

Stephen Frost escribió:
> * Amit Kapila (amit.kapila16@gmail.com) wrote:
> > So let me try to explain what I understood from above:
> > 
> > 1. enable_alter_system a new GUC whose default value =off.
> > 2. Alter System will check this variable and return error (not
> > allowed), if this parameter is off.
> > 3. Now if user enables include directive in postgresql.conf, it will
> > enable Alter System as value of
> >     enable_alter_system is on.
> > 4. User can run Alter System command to disable Alter System
> > "enable_alter_system = off".
> >     Now even though include directive is enabled, but new Alter System
> > commands will not work, however
> >     existing parameter's take into effect on restart/sighup.
> 
> Yes.  Not sure that it'd be terribly likely for a user to do that, but
> if they do it, so be it.

With this design, if you put enable_alter_system=off in auto.conf, there
is no way for the user to enable alter system again short of editing a
file in the data directory.  I think this is one of the things that was
"forbidden" by policy; only files in the config directory needs to be
edited.

What I was proposing upthread is that enable_alter_system=off/on would
be present in postgresql.conf, and there is no include line for
auto.conf.  That way, if the user wishes to enable/disable the feature,
they need to edit postgresql.conf to do so.  ALTER SYSTEM doesn't offer
a way to disable itself.  If ALTER SYSTEM is disabled via
enable_alter_system=off in postgresql.conf, the settings in auto.conf
are not read.

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Alvaro Herrera <alvherre@2ndquadrant.com> — 2013-08-20T15:19:25Z

Stephen Frost escribió:
> * Alvaro Herrera (alvherre@2ndquadrant.com) wrote:
> > Stephen Frost escribió:
> > > With includedir and include directives, and postgresql.conf setting a
> > > defined ordering, with last-wins, you could simply have the 'includedir'
> > > for conf.d come before the 'include' for auto.conf.
> > 
> > Uh, I was thinking that conf.d would be read by the system
> > automatically, not because of an includedir line in postgresql.conf.
> 
> I'd much rather have an includedir directive than some hard-coded or
> command-line option to read the directory..  The directory should live
> in /etc/postgresql/X.Y/cluster/ on at least Debian derivatives..

The conf.d/ path would be relative to postgresql.conf, so there's no
need for Debian to patch anything.

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Tom Lane <tgl@sss.pgh.pa.us> — 2013-08-20T15:19:33Z

Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> What I was proposing upthread is that enable_alter_system=off/on would
> be present in postgresql.conf, and there is no include line for
> auto.conf.  That way, if the user wishes to enable/disable the feature,
> they need to edit postgresql.conf to do so.  ALTER SYSTEM doesn't offer
> a way to disable itself.  If ALTER SYSTEM is disabled via
> enable_alter_system=off in postgresql.conf, the settings in auto.conf
> are not read.

Personally, I'd get rid of any explicit "enable_alter_system" variable,
and instead have an include directive for auto.conf.  The functionality
you describe above is then available by commenting or uncommenting the
directive, plus the user has the option to decide where to put the
directive (and thus control which settings can or can't be overridden).
Anything else seems like it's going to be less flexible or else a lot
more complicated.

			regards, tom lane

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-20T15:23:06Z

* Alvaro Herrera (alvherre@2ndquadrant.com) wrote:
> With this design, if you put enable_alter_system=off in auto.conf, there
> is no way for the user to enable alter system again short of editing a
> file in the data directory.  I think this is one of the things that was
> "forbidden" by policy; only files in the config directory needs to be
> edited.

If you edit it by hand to begin with (setting that parameter to 'off')
then it's reasonable that you may have to edit it by hand again to fix
it.  If we don't want people to "lock themselves out" by using ALTER
SYSTEM to turn it off, then we just disallow that.

> What I was proposing upthread is that enable_alter_system=off/on would
> be present in postgresql.conf, and there is no include line for
> auto.conf.  

I really think that's a terrible approach, to be honest.  I want to see
an 'include' line in postgresql.conf for auto.conf, so the hapless
sysadmin who is trying to figure out what the crazy DBA did has some
clue what to look for.  "enable_alter_system" doesn't tell him diddly
about an 'auto.conf' file which is included in the system config.

> That way, if the user wishes to enable/disable the feature,
> they need to edit postgresql.conf to do so.  ALTER SYSTEM doesn't offer
> a way to disable itself.  

We can simply disallow ALTER SYSTEM from modifying enable_alter_system;
that strikes me as a reasonable thing to do anyway.  What I find a bit
more worrying is what happens if they decide to put
enable_alter_system=off into the postgresql.conf but keep the 'include'
line for auto.conf..  Which goes right back to the question that I had
before around if we want to complain when the same GUC is seen multiple
times during parsing.  It seems like there's no hope for it, given the
way this has been designed, because you *must* set certain parameters
and so you can't simply have them commented out, but those are likely to
be parameters which DBAs will want to change through ALTER SYSTEM.

	Thanks,

		Stephen

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-20T15:23:23Z

* Alvaro Herrera (alvherre@2ndquadrant.com) wrote:
> Stephen Frost escribió:
> > * Alvaro Herrera (alvherre@2ndquadrant.com) wrote:
> > > Stephen Frost escribió:
> > > > With includedir and include directives, and postgresql.conf setting a
> > > > defined ordering, with last-wins, you could simply have the 'includedir'
> > > > for conf.d come before the 'include' for auto.conf.
> > > 
> > > Uh, I was thinking that conf.d would be read by the system
> > > automatically, not because of an includedir line in postgresql.conf.
> > 
> > I'd much rather have an includedir directive than some hard-coded or
> > command-line option to read the directory..  The directory should live
> > in /etc/postgresql/X.Y/cluster/ on at least Debian derivatives..
> 
> The conf.d/ path would be relative to postgresql.conf, so there's no
> need for Debian to patch anything.

Uhhh, I really don't see that working, at all...

	Thanks,

		Stephen

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Tom Lane <tgl@sss.pgh.pa.us> — 2013-08-20T15:24:32Z

Stephen Frost <sfrost@snowman.net> writes:
> * Alvaro Herrera (alvherre@2ndquadrant.com) wrote:
>> The conf.d/ path would be relative to postgresql.conf, so there's no
>> need for Debian to patch anything.

> Uhhh, I really don't see that working, at all...

Why not?  conf.d is for installable files, AIUI.  What we need to
be writable is auto.conf, but I thought we'd agreed that would live
inside $PGDATA.

			regards, tom lane

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-20T15:33:51Z

* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> Stephen Frost <sfrost@snowman.net> writes:
> > * Alvaro Herrera (alvherre@2ndquadrant.com) wrote:
> >> The conf.d/ path would be relative to postgresql.conf, so there's no
> >> need for Debian to patch anything.
> 
> > Uhhh, I really don't see that working, at all...
> 
> Why not?  conf.d is for installable files, AIUI.  What we need to
> be writable is auto.conf, but I thought we'd agreed that would live
> inside $PGDATA.

I agree that auto.conf should live in $PGDATA, but I really don't like
the idea of conf.d being relative to some other existing file.  It
should be included through an 'includedir' option, not just picked up as
some magic directory name, and therefore consider the current
arrangement of parameters in Debian:

data_directory = '/var/lib/postgresql/9.2/main'
hba_file = '/etc/postgresql/9.2/main/pg_hba.conf'
ident_file = '/etc/postgresql/9.2/main/pg_ident.conf'

and postgres is started like so:

/usr/lib/postgresql/9.2/bin/postgres -D /var/lib/postgresql/9.2/main -c config_file=/etc/postgresql/9.2/main/postgresql.conf

With the proposed include line for auto.conf, which lives in $PGDATA,
we'd have:

include 'auto.conf'

Would we then have

includedir 'conf.d'

which is relative to postgresql.conf instead?

	Thanks,

		Stephen

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Alvaro Herrera <alvherre@2ndquadrant.com> — 2013-08-20T15:33:51Z

Stephen Frost escribió:
> * Alvaro Herrera (alvherre@2ndquadrant.com) wrote:
> > Stephen Frost escribió:

> > > I'd much rather have an includedir directive than some hard-coded or
> > > command-line option to read the directory..  The directory should live
> > > in /etc/postgresql/X.Y/cluster/ on at least Debian derivatives..
> > 
> > The conf.d/ path would be relative to postgresql.conf, so there's no
> > need for Debian to patch anything.
> 
> Uhhh, I really don't see that working, at all...

I don't understand why not.  Care to explain?

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-20T15:39:54Z

* Alvaro Herrera (alvherre@2ndquadrant.com) wrote:
> Stephen Frost escribió:
> > * Alvaro Herrera (alvherre@2ndquadrant.com) wrote:
> > > Stephen Frost escribió:
> 
> > > > I'd much rather have an includedir directive than some hard-coded or
> > > > command-line option to read the directory..  The directory should live
> > > > in /etc/postgresql/X.Y/cluster/ on at least Debian derivatives..
> > > 
> > > The conf.d/ path would be relative to postgresql.conf, so there's no
> > > need for Debian to patch anything.
> > 
> > Uhhh, I really don't see that working, at all...
> 
> I don't understand why not.  Care to explain?

Tried to in my other mail, but let me also point out that we
("PGDG"/Upstream) don't "own" the directory in which postgresql.conf
lives.  At least on Debian and relatives, that directory isn't under
$PGDATA and it already has other files in it beyond postgresql.conf or
even the other PostgreSQL config files of hba and ident.  Here's the
default directory setup on Debian for /etc/postgresql/9.2/main/:

-rw-r--r-- 1 postgres postgres   316 Jun 29 22:07 environment
-rw-r--r-- 1 postgres postgres   143 Jun 29 22:07 pg_ctl.conf
-rw-r----- 1 postgres postgres  4649 Jun 29 22:07 pg_hba.conf
-rw-r----- 1 postgres postgres  1636 Jun 29 22:07 pg_ident.conf
-rw-r--r-- 1 postgres postgres 19770 Jun 29 22:07 postgresql.conf
-rw-r--r-- 1 postgres postgres   378 Jun 29 22:07 start.conf

There's three other files there and some sysadmins may have already
created their own 'conf.d' directory, perhaps to use for building the
postgresql.conf file or similar.  We must have a way to disable the
conf.d option and a way to name it something other than 'conf.d', imv.

	Thanks,

		Stephen

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Alvaro Herrera <alvherre@2ndquadrant.com> — 2013-08-20T15:41:57Z

Stephen Frost escribió:

> I agree that auto.conf should live in $PGDATA, but I really don't like
> the idea of conf.d being relative to some other existing file.  It
> should be included through an 'includedir' option, not just picked up as
> some magic directory name, and therefore consider the current
> arrangement of parameters in Debian:
> 
> data_directory = '/var/lib/postgresql/9.2/main'
> hba_file = '/etc/postgresql/9.2/main/pg_hba.conf'
> ident_file = '/etc/postgresql/9.2/main/pg_ident.conf'
> 
> and postgres is started like so:
> 
> /usr/lib/postgresql/9.2/bin/postgres -D /var/lib/postgresql/9.2/main -c config_file=/etc/postgresql/9.2/main/postgresql.conf
> 
> With the proposed include line for auto.conf, which lives in $PGDATA,
> we'd have:
> 
> include 'auto.conf'
> 
> Would we then have
> 
> includedir 'conf.d'
> 
> which is relative to postgresql.conf instead?

Well, all the relative paths in include/includedir directives would be
relative to the directory specified by the -c config_file param, which
makes perfect sense.  So the conf.d would work fine in your example.

The only problem I see in your snippet above is the "include auto.conf"
line, which doesn't make any sense because that file would not be found.
Hence my proposal that the file ought to be read automatically, not via
an include line.  Sadly I don't think we cannot just make it an absolute
path, in case the data dir is moved or whatever; the only choice I see
would be to have something like
include-data 'auto.conf'
or something like that which tells the system that the file is not in
the config dir but in the data dir instead.  A nearby comment could let
the user know about this file being in the data directory instead of the
config directory.

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Alvaro Herrera <alvherre@2ndquadrant.com> — 2013-08-20T15:47:50Z

Stephen Frost escribió:

> Tried to in my other mail,

Yep, got it and replied, thanks.

> but let me also point out that we
> ("PGDG"/Upstream) don't "own" the directory in which postgresql.conf
> lives.  At least on Debian and relatives, that directory isn't under
> $PGDATA and it already has other files in it beyond postgresql.conf or
> even the other PostgreSQL config files of hba and ident.  Here's the
> default directory setup on Debian for /etc/postgresql/9.2/main/:
> 
> -rw-r--r-- 1 postgres postgres   316 Jun 29 22:07 environment
> -rw-r--r-- 1 postgres postgres   143 Jun 29 22:07 pg_ctl.conf
> -rw-r----- 1 postgres postgres  4649 Jun 29 22:07 pg_hba.conf
> -rw-r----- 1 postgres postgres  1636 Jun 29 22:07 pg_ident.conf
> -rw-r--r-- 1 postgres postgres 19770 Jun 29 22:07 postgresql.conf
> -rw-r--r-- 1 postgres postgres   378 Jun 29 22:07 start.conf
> 
> There's three other files there and some sysadmins may have already
> created their own 'conf.d' directory, perhaps to use for building the
> postgresql.conf file or similar.  We must have a way to disable the
> conf.d option and a way to name it something other than 'conf.d', imv.

Uhm.  I find it hard to care much for this position.  Surely config
files are not migrated automatically from one major version to the next,
so if somebody created a 9.3/main/conf.d directory, they will have to
change it when they migrate to 9.4.

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-20T16:22:08Z

* Alvaro Herrera (alvherre@2ndquadrant.com) wrote:
> Well, all the relative paths in include/includedir directives would be
> relative to the directory specified by the -c config_file param, which
> makes perfect sense.  So the conf.d would work fine in your example.

Why would include/includedir directives be relative to where the
'config_file' GUC is set to instead of relative to where all the other
GUCs in postgresql.conf are relative to?  That is a recipe for
confusion, imv.

Of course, the current situation is quite terrible anyway, imv.  Having
the GUCs be relative to whereever the user happens to run pg_ctl from is
pretty ugly- not to mention that the commented out 'defaults' won't
actually work if you uncomment them because the *actual* default/unset
value *is* in the data directory.  I'm starting to wish for a way to do
variable substitution inside postgresql.conf, so we could have defaults
that would actually work if uncommented (eg: $DataDir/pg_hba.conf).

That would help with auto.conf also.

> The only problem I see in your snippet above is the "include auto.conf"
> line, which doesn't make any sense because that file would not be found.

To be honest, I was considering 'include' to be relative to the data
directory and handled similar to how we process recovery.conf, but as we
consider paths in postgresql.conf to be relative to $PWD, that's not
ideal because it'd be different from the rest of the file references.

In any case, while the current situation sucks, I don't think we can go
changing how relative files are treated in postgresql.conf, nor should
we make the way a path is processed change depending on which GUC is
being set.

While I really like the 'include auto.conf' style, I'm starting to think
it may not be workable after all.  Another thing to consider is if the
user decides to change that include line..  What happens when the DBA
tries to do a 'ALTER SYSTEM'?  It'd still use the hard-coded auto.conf
file and happily update it, I imagine, but it won't actually get
included...

	Thanks,

		Stephen

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Alvaro Herrera <alvherre@2ndquadrant.com> — 2013-08-20T16:32:10Z

Stephen Frost escribió:
> * Alvaro Herrera (alvherre@2ndquadrant.com) wrote:
> > Well, all the relative paths in include/includedir directives would be
> > relative to the directory specified by the -c config_file param, which
> > makes perfect sense.  So the conf.d would work fine in your example.
> 
> Why would include/includedir directives be relative to where the
> 'config_file' GUC is set to instead of relative to where all the other
> GUCs in postgresql.conf are relative to?  That is a recipe for
> confusion, imv.
> 
> Of course, the current situation is quite terrible anyway, imv.  Having
> the GUCs be relative to whereever the user happens to run pg_ctl from is
> pretty ugly- not to mention that the commented out 'defaults' won't
> actually work if you uncomment them because the *actual* default/unset
> value *is* in the data directory.

Uh?  See the docs:
http://www.postgresql.org/docs/devel/static/config-setting.html#CONFIG-INCLUDES

 " ... the postgresql.conf file can contain include directives, ...
  If the file name is not an absolute path, it is taken as relative to
  the directory containing the referencing configuration file."


> I'm starting to wish for a way to do
> variable substitution inside postgresql.conf, so we could have defaults
> that would actually work if uncommented (eg: $DataDir/pg_hba.conf).
> 
> That would help with auto.conf also.

Grumble.  I don't think we need this.  At least, not for ALTER SYSTEM or
conf.d.

> > The only problem I see in your snippet above is the "include auto.conf"
> > line, which doesn't make any sense because that file would not be found.
> 
> To be honest, I was considering 'include' to be relative to the data
> directory and handled similar to how we process recovery.conf,

Well, recovery.conf is a special case that doesn't follow to normal
rules.

> but as we
> consider paths in postgresql.conf to be relative to $PWD, that's not
> ideal because it'd be different from the rest of the file references.

I don't know where you got that idea from, but it's wrong.

> While I really like the 'include auto.conf' style, I'm starting to think
> it may not be workable after all.  Another thing to consider is if the
> user decides to change that include line..  What happens when the DBA
> tries to do a 'ALTER SYSTEM'?  It'd still use the hard-coded auto.conf
> file and happily update it, I imagine, but it won't actually get
> included...

Well, this whole line of discussion started because I objected to the
whole code path that was trying to detect whether auto.conf had been
parsed, and raised a warning if ALTER SYSTEM was executed and the file
wasn't parsed.

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Tom Lane <tgl@sss.pgh.pa.us> — 2013-08-20T16:34:26Z

Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> Stephen Frost escribi:
>> While I really like the 'include auto.conf' style, I'm starting to think
>> it may not be workable after all.  Another thing to consider is if the
>> user decides to change that include line..  What happens when the DBA
>> tries to do a 'ALTER SYSTEM'?  It'd still use the hard-coded auto.conf
>> file and happily update it, I imagine, but it won't actually get
>> included...

> Well, this whole line of discussion started because I objected to the
> whole code path that was trying to detect whether auto.conf had been
> parsed, and raised a warning if ALTER SYSTEM was executed and the file
> wasn't parsed.

I really, really don't think that we should be trying to detect or prevent
any such thing.  If the user breaks it like that, he gets to keep both
pieces --- and who's to say it's broken, anyway?  Disabling ALTER SYSTEM
might have been exactly his intent.

			regards, tom lane

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-20T18:02:51Z

* Alvaro Herrera (alvherre@2ndquadrant.com) wrote:
> Stephen Frost escribió:
> > Of course, the current situation is quite terrible anyway, imv.  Having
> > the GUCs be relative to whereever the user happens to run pg_ctl from is
> > pretty ugly- not to mention that the commented out 'defaults' won't
> > actually work if you uncomment them because the *actual* default/unset
> > value *is* in the data directory.
> 
> Uh?  See the docs:
> http://www.postgresql.org/docs/devel/static/config-setting.html#CONFIG-INCLUDES
> 
>  " ... the postgresql.conf file can contain include directives, ...
>   If the file name is not an absolute path, it is taken as relative to
>   the directory containing the referencing configuration file."

And what about

http://www.postgresql.org/docs/9.3/static/runtime-config-file-locations.html

 ... ?

   "When setting any of these parameters, a relative path will be
   interpreted with respect to the directory in which postgres is
   started."

> > I'm starting to wish for a way to do
> > variable substitution inside postgresql.conf, so we could have defaults
> > that would actually work if uncommented (eg: $DataDir/pg_hba.conf).
> > 
> > That would help with auto.conf also.
> 
> Grumble.  I don't think we need this.  At least, not for ALTER SYSTEM or
> conf.d.

imv, it would be handy.  Along with a $ConfDir which is the
postgresql.conf location- it would certainly make things clearer about
what files are being included from where.

> > To be honest, I was considering 'include' to be relative to the data
> > directory and handled similar to how we process recovery.conf,
> 
> Well, recovery.conf is a special case that doesn't follow to normal
> rules.

I don't see why it should be.

> > but as we
> > consider paths in postgresql.conf to be relative to $PWD, that's not
> > ideal because it'd be different from the rest of the file references.
> 
> I don't know where you got that idea from, but it's wrong.

Not sure which you're referring to, but see above from the docs?  Also,
please tias..  I was amazed that we use $PWD also, but we do..

> Well, this whole line of discussion started because I objected to the
> whole code path that was trying to detect whether auto.conf had been
> parsed, and raised a warning if ALTER SYSTEM was executed and the file
> wasn't parsed.

I like the idea that we complain if ALTER SYSTEM ends up being
idempotent..  Not sure how far we should take that, but accepting
commands which end up doing nothing is bad, imv.

	Thanks,

		Stephen

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Alvaro Herrera <alvherre@2ndquadrant.com> — 2013-08-20T18:32:24Z

Stephen Frost escribió:

> And what about
> 
> http://www.postgresql.org/docs/9.3/static/runtime-config-file-locations.html
> 
>  ... ?
> 
>    "When setting any of these parameters, a relative path will be
>    interpreted with respect to the directory in which postgres is
>    started."

That's talking about the data_directory and the various foo_file
settings, though; it doesn't apply to the include settings.  Note
especially that config_file says it can only be set on the postgres
command line.  (I think a saner definition would have been to state that
relative paths are not allowed in the command line.  But that ship has
already sailed.  And relative paths seem useful in the config file; and
maintaining the distinction that they are allowed within the config file
but not in the command line might be awkward.)


(Uhm, when a command line contains stuff, is the stuff "in" the command
line or "on" it?)

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-20T19:17:09Z

* Alvaro Herrera (alvherre@2ndquadrant.com) wrote:
> Stephen Frost escribió:
> > http://www.postgresql.org/docs/9.3/static/runtime-config-file-locations.html
> 
> That's talking about the data_directory and the various foo_file
> settings, though; it doesn't apply to the include settings.  

Right- that's what I'm bitching about.  We have various references to
file locations, with defined handling of relative locations, and the
'include' system completely ignores that and instead invents its own
making the result a confusing mess.

> Note
> especially that config_file says it can only be set on the postgres
> command line.  (I think a saner definition would have been to state that
> relative paths are not allowed in the command line.  But that ship has
> already sailed.  And relative paths seem useful in the config file; and
> maintaining the distinction that they are allowed within the config file
> but not in the command line might be awkward.)

Relative paths based on $PWD are useful?  Really?  Not on my systems
anyway..

> (Uhm, when a command line contains stuff, is the stuff "in" the command
> line or "on" it?)

A just question- I vote 'on'. :)

	Thanks,

		Stephen

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Martijn van Oosterhout <kleptog@svana.org> — 2013-08-20T19:41:01Z

On Tue, Aug 20, 2013 at 11:23:06AM -0400, Stephen Frost wrote:
> > What I was proposing upthread is that enable_alter_system=off/on would
> > be present in postgresql.conf, and there is no include line for
> > auto.conf.  
> 
> I really think that's a terrible approach, to be honest.  I want to see
> an 'include' line in postgresql.conf for auto.conf, so the hapless
> sysadmin who is trying to figure out what the crazy DBA did has some
> clue what to look for.  "enable_alter_system" doesn't tell him diddly
> about an 'auto.conf' file which is included in the system config.

ISTM you want some kind of hybrid setting like:

#include_system auto.conf

which simultaneously does three things:

1. Sets the enable_alter_system flag
2. Indicates the file to use
3. Indicates the priority of the setting re other settings.

Comment it out, ALTER SYSTEM stop working. Put it back and it's
immediately clear what it means. And the user can control where the
settings go.

Syntax is a bit fugly though.

Have a nice day,
-- 
Martijn van Oosterhout   <kleptog@svana.org>   http://svana.org/kleptog/
> He who writes carelessly confesses thereby at the very outset that he does
> not attach much importance to his own thoughts.
   -- Arthur Schopenhauer

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Bruce Momjian <bruce@momjian.us> — 2013-08-21T01:59:40Z

On Tue, Aug 20, 2013 at 08:38:52AM +0530, Amit Kapila wrote:
> On Tue, Aug 20, 2013 at 8:27 AM, Stephen Frost <sfrost@snowman.net> wrote:
> > * Amit Kapila (amit.kapila16@gmail.com) wrote:
> >> If both are okay, then I would like to go with second option (include
> >> file mechanism).
> >> I think by default, we should allow auto file to be included and if
> >> user faces any problem or otherwise,
> >> then he can decide to disable it.
> >
> > If it's enabled by default, then we need to ship an 'auto' file which is
> > empty by default...
> 
> initdb will create the empty auto file. If we don't enable by default,
> then if user uses
> ALTER SYSTEM and do sighup/restart, changed config parameter values
> will not come into affect
> until he manually enables it by removing '#' from '#include'.

Just a heads-up, but a lot of C language folks are going to look at:

#include abc

and think that is enabled.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + It's impossible for everything to be true. +

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila16@gmail.com> — 2013-08-21T04:40:32Z

On Tue, Aug 20, 2013 at 10:02 PM, Alvaro Herrera
<alvherre@2ndquadrant.com> wrote:
> Stephen Frost escribió:
>> * Alvaro Herrera (alvherre@2ndquadrant.com) wrote:
>> > Well, all the relative paths in include/includedir directives would be
>> > relative to the directory specified by the -c config_file param, which
>> > makes perfect sense.  So the conf.d would work fine in your example.
>>
>> Why would include/includedir directives be relative to where the
>> 'config_file' GUC is set to instead of relative to where all the other
>> GUCs in postgresql.conf are relative to?  That is a recipe for
>> confusion, imv.
>>
>> Of course, the current situation is quite terrible anyway, imv.  Having
>> the GUCs be relative to whereever the user happens to run pg_ctl from is
>> pretty ugly- not to mention that the commented out 'defaults' won't
>> actually work if you uncomment them because the *actual* default/unset
>> value *is* in the data directory.
>
> Uh?  See the docs:
> http://www.postgresql.org/docs/devel/static/config-setting.html#CONFIG-INCLUDES
>
>  " ... the postgresql.conf file can contain include directives, ...
>   If the file name is not an absolute path, it is taken as relative to
>   the directory containing the referencing configuration file."

      You are right and I have checked code as well, it works in above
way for include directives.
      Now the question I have in mind is that even if we can't
directly use include directive to enable/disable Alter
      System and reading of auto file, how would a new GUC
enable_alter_system can solve all the things.
      It can allow/disallow Alter System command, but how about
reading of auto file?
      If we directly read auto file without considering
enable_alter_system, it can lead to chaos due to some un-safe
      values, on the other side if we want to consider
enable_alter_system before reading file, it can complicate the
      ProcessConfigFile() code.

With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila16@gmail.com> — 2013-08-21T04:43:30Z

On Wed, Aug 21, 2013 at 7:29 AM, Bruce Momjian <bruce@momjian.us> wrote:
> On Tue, Aug 20, 2013 at 08:38:52AM +0530, Amit Kapila wrote:
>> On Tue, Aug 20, 2013 at 8:27 AM, Stephen Frost <sfrost@snowman.net> wrote:
>> > * Amit Kapila (amit.kapila16@gmail.com) wrote:
>> >> If both are okay, then I would like to go with second option (include
>> >> file mechanism).
>> >> I think by default, we should allow auto file to be included and if
>> >> user faces any problem or otherwise,
>> >> then he can decide to disable it.
>> >
>> > If it's enabled by default, then we need to ship an 'auto' file which is
>> > empty by default...
>>
>> initdb will create the empty auto file. If we don't enable by default,
>> then if user uses
>> ALTER SYSTEM and do sighup/restart, changed config parameter values
>> will not come into affect
>> until he manually enables it by removing '#' from '#include'.
>
> Just a heads-up, but a lot of C language folks are going to look at:
>
> #include abc
>
> and think that is enabled.

True, but generally for conf and script file, symbol '#' is treated as
commented portion of content.


With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-21T14:52:01Z

Martijn,

* Martijn van Oosterhout (kleptog@svana.org) wrote:
> ISTM you want some kind of hybrid setting like:
> 
> #include_system auto.conf
> 
> which simultaneously does three things:
> 
> 1. Sets the enable_alter_system flag
> 2. Indicates the file to use
> 3. Indicates the priority of the setting re other settings.
> 
> Comment it out, ALTER SYSTEM stop working. Put it back and it's
> immediately clear what it means. And the user can control where the
> settings go.

Yeah, that's certainly an interesting idea.  I might call it
'auto_conf_file auto.conf' to avoid the '#include' concern and to
perhaps clarify that it's more than just a regular 'include'.

> Syntax is a bit fugly though.

Agreed.

	Thanks,

		Stephen
		(who is still unhappy about the GUC-specific handling for relative
		paths in postgresql.conf)

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila16@gmail.com> — 2013-08-22T03:36:22Z

On Wed, Aug 21, 2013 at 8:22 PM, Stephen Frost <sfrost@snowman.net> wrote:
> Martijn,
>
> * Martijn van Oosterhout (kleptog@svana.org) wrote:
>> ISTM you want some kind of hybrid setting like:
>>
>> #include_system auto.conf
>>
>> which simultaneously does three things:
>>
>> 1. Sets the enable_alter_system flag
>> 2. Indicates the file to use
>> 3. Indicates the priority of the setting re other settings.
>>
>> Comment it out, ALTER SYSTEM stop working. Put it back and it's
>> immediately clear what it means. And the user can control where the
>> settings go.
>
> Yeah, that's certainly an interesting idea.  I might call it
> 'auto_conf_file auto.conf' to avoid the '#include' concern and to
> perhaps clarify that it's more than just a regular 'include'.

   This can resolve the problem of whether to read auto file rather
cleanly, so the idea is:

Enable/Disable reading of auto file
-----------------------------------------------------
a. Have a new include in postresql.conf
    #include_auto_conf_file    postgresql.auto.conf
    as it is a special include, we can read this file relative to data
directory.

Enable/Disable Alter System command
-----------------------------------------------------------
This can be achieved in 3 ways:
a. Check before executing Alter System if include directive is
disabled, then just issue a warning to user and proceed with command.
b. Check before executing Alter System if include directive is
disabled, then just issue an error and stop.
c. Have a new guc enable_alter_system which will behave as described
in my previous mail and below:
   >1. enable_alter_system a new GUC whose default value =off.
   > 2. Alter System will check this variable and return error (not
   > allowed), if this parameter is off.
   > 3. Now if user enables include directive in postgresql.conf, it will
   > enable Alter System as value of enable_alter_system is on.
   > 4. User can run Alter System command to disable Alter System
   > "enable_alter_system = off".
   >     Now even though include directive is enabled, but new Alter System
   > commands will not work, however
   >     existing parameter's take into effect on restart/sighup.


With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-22T12:36:37Z

* Amit Kapila (amit.kapila16@gmail.com) wrote:
>    This can resolve the problem of whether to read auto file rather
> cleanly, so the idea is:
> 
> Enable/Disable reading of auto file
> -----------------------------------------------------
> a. Have a new include in postresql.conf
>     #include_auto_conf_file    postgresql.auto.conf
>     as it is a special include, we can read this file relative to data
> directory.
> 
> Enable/Disable Alter System command
> -----------------------------------------------------------
> This can be achieved in 3 ways:
> a. Check before executing Alter System if include directive is
> disabled, then just issue a warning to user and proceed with command.
> b. Check before executing Alter System if include directive is
> disabled, then just issue an error and stop.

It doesn't make sense for it to be a 'warning' with this- the
parameter specifies the file to use.  If you don't know what file to
use, how you can possibly do anything but return an error?

Note that I *like* that about this approach.

There are a few other considerations with this-

- What should the default be?  (Still thinking 'off' myself)
- What happens if the user specifies 'postgresql.conf'?  I'm thinking we
  would disallow such insanity (as that's what it is, imv..) by having
  an identifier in the file that this is the PG "auto conf" file.
- Should we have such an identifier in auto.conf to indicate that we
  created it, to prevent the user from setting it to something they
  shouldn't?
- What's the "bootstrap" mode; iow, if a user enables the option but the
  file doesn't exist, what do we do?  With this approach, I'd be
  inclined to say we simply create it and put the marker to indicate
  it's "our" file.
- Should we allow it to be outside of the data dir?  We could simply log
  an error and ignore the parameter if it's more than a simple filename.

There are probably other considerations also..

	Thanks,

		Stephen

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Bruce Momjian <bruce@momjian.us> — 2013-08-22T16:04:08Z

On Thu, Aug 22, 2013 at 08:36:37AM -0400, Stephen Frost wrote:
> * Amit Kapila (amit.kapila16@gmail.com) wrote:
> >    This can resolve the problem of whether to read auto file rather
> > cleanly, so the idea is:
> > 
> > Enable/Disable reading of auto file
> > -----------------------------------------------------
> > a. Have a new include in postresql.conf
> >     #include_auto_conf_file    postgresql.auto.conf
> >     as it is a special include, we can read this file relative to data
> > directory.

The big advantage of using 'include_auto_conf_file' and not simply
'include' is that we can issue an error from ALTER SYSTEM SET if that is
not set.

> > Enable/Disable Alter System command
> > -----------------------------------------------------------
> > This can be achieved in 3 ways:
> > a. Check before executing Alter System if include directive is
> > disabled, then just issue a warning to user and proceed with command.
> > b. Check before executing Alter System if include directive is
> > disabled, then just issue an error and stop.
> 
> It doesn't make sense for it to be a 'warning' with this- the
> parameter specifies the file to use.  If you don't know what file to
> use, how you can possibly do anything but return an error?

Agreed.  No sense in allowing users to add things to the 'auto' file
when the auto file is inactive.

> Note that I *like* that about this approach.
> 
> There are a few other considerations with this-
> 
> - What should the default be?  (Still thinking 'off' myself)

Probably, but we might need to wait until we have a final API for a
decision on that.

> - What happens if the user specifies 'postgresql.conf'?  I'm thinking we
>   would disallow such insanity (as that's what it is, imv..) by having
>   an identifier in the file that this is the PG "auto conf" file.

I am thinking they can't include a value equal to 'config_file', which
is normally postgresql.conf.  I am not a big fan of looking for special
text in files.  This might be complex to check, though, because of path
changes --- we might just disallow the basement from matching the
basename of config_file.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + It's impossible for everything to be true. +

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila16@gmail.com> — 2013-08-23T03:57:10Z

On Thu, Aug 22, 2013 at 6:06 PM, Stephen Frost <sfrost@snowman.net> wrote:
> * Amit Kapila (amit.kapila16@gmail.com) wrote:
>>    This can resolve the problem of whether to read auto file rather
>> cleanly, so the idea is:
>>
>> Enable/Disable reading of auto file
>> -----------------------------------------------------
>> a. Have a new include in postresql.conf
>>     #include_auto_conf_file    postgresql.auto.conf
>>     as it is a special include, we can read this file relative to data
>> directory.
>>
>> Enable/Disable Alter System command
>> -----------------------------------------------------------
>> This can be achieved in 3 ways:
>> a. Check before executing Alter System if include directive is
>> disabled, then just issue a warning to user and proceed with command.
>> b. Check before executing Alter System if include directive is
>> disabled, then just issue an error and stop.
>
> It doesn't make sense for it to be a 'warning' with this- the
> parameter specifies the file to use.  If you don't know what file to
> use, how you can possibly do anything but return an error?

   As the file and location are fixed, we can go-ahead and write to
it, but I think now we are deciding
   if someone disables include dir, then we can just disable Alter
System, so it is better to return error in such
   situation.

> Note that I *like* that about this approach.
>
> There are a few other considerations with this-
>
> - What should the default be?  (Still thinking 'off' myself)
     default 'off' is a safe option, as it won't allow users to make
any change to parameter values until/unless they
     read from manual, how to use it and what can go wrong, on the
other side it will be bit hassle for user to use this
     command. I think 'on' would be better.
> - What happens if the user specifies 'postgresql.conf'?  I'm thinking we
>   would disallow such insanity (as that's what it is, imv..) by having
>   an identifier in the file that this is the PG "auto conf" file.
          I think we can detect by name and give error.
> - Should we have such an identifier in auto.conf to indicate that we
>   created it, to prevent the user from setting it to something they
>   shouldn't?
     I think if user plays with this file manually, it can lead to
problems, that's why earlier we have
     decided to keep a note on top of file which will indicate, do not
edit this file manually.
     I believe that should be sufficient.

> - What's the "bootstrap" mode; iow, if a user enables the option but the
>   file doesn't exist, what do we do?  With this approach, I'd be
>   inclined to say we simply create it and put the marker to indicate
>   it's "our" file.

     Alter System will create the file if doesn't exist.

> - Should we allow it to be outside of the data dir?  We could simply log
>   an error and ignore the parameter if it's more than a simple filename.

     This should be an error, the file location and name will be fixed.


With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila16@gmail.com> — 2013-08-23T04:00:29Z

On Thu, Aug 22, 2013 at 9:34 PM, Bruce Momjian <bruce@momjian.us> wrote:
> On Thu, Aug 22, 2013 at 08:36:37AM -0400, Stephen Frost wrote:
>> * Amit Kapila (amit.kapila16@gmail.com) wrote:
>> >    This can resolve the problem of whether to read auto file rather
>> > cleanly, so the idea is:
>> >
>> > Enable/Disable reading of auto file
>> > -----------------------------------------------------
>> > a. Have a new include in postresql.conf
>> >     #include_auto_conf_file    postgresql.auto.conf
>> >     as it is a special include, we can read this file relative to data
>> > directory.
>
> The big advantage of using 'include_auto_conf_file' and not simply
> 'include' is that we can issue an error from ALTER SYSTEM SET if that is
> not set.
>
>> > Enable/Disable Alter System command
>> > -----------------------------------------------------------
>> > This can be achieved in 3 ways:
>> > a. Check before executing Alter System if include directive is
>> > disabled, then just issue a warning to user and proceed with command.
>> > b. Check before executing Alter System if include directive is
>> > disabled, then just issue an error and stop.
>>
>> It doesn't make sense for it to be a 'warning' with this- the
>> parameter specifies the file to use.  If you don't know what file to
>> use, how you can possibly do anything but return an error?
>
> Agreed.  No sense in allowing users to add things to the 'auto' file
> when the auto file is inactive.
>
>> Note that I *like* that about this approach.
>>
>> There are a few other considerations with this-
>>
>> - What should the default be?  (Still thinking 'off' myself)
>
> Probably, but we might need to wait until we have a final API for a
> decision on that.
>
>> - What happens if the user specifies 'postgresql.conf'?  I'm thinking we
>>   would disallow such insanity (as that's what it is, imv..) by having
>>   an identifier in the file that this is the PG "auto conf" file.
>
> I am thinking they can't include a value equal to 'config_file', which
> is normally postgresql.conf.  I am not a big fan of looking for special
> text in files.  This might be complex to check, though, because of path
> changes --- we might just disallow the basement from matching the
> basename of config_file.

   Right, I also think that as file and location are fixed, so it can
be detected with name.

With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-23T12:31:05Z

* Amit Kapila (amit.kapila16@gmail.com) wrote:
> On Thu, Aug 22, 2013 at 6:06 PM, Stephen Frost <sfrost@snowman.net> wrote:
> > * Amit Kapila (amit.kapila16@gmail.com) wrote:
> >> Enable/Disable reading of auto file
> >> -----------------------------------------------------
> >> a. Have a new include in postresql.conf
> >>     #include_auto_conf_file    postgresql.auto.conf
> >>     as it is a special include, we can read this file relative to data
> >> directory.
> >>
> >> Enable/Disable Alter System command
> >> -----------------------------------------------------------
> >> This can be achieved in 3 ways:
> >> a. Check before executing Alter System if include directive is
> >> disabled, then just issue a warning to user and proceed with command.
> >> b. Check before executing Alter System if include directive is
> >> disabled, then just issue an error and stop.
> >
> > It doesn't make sense for it to be a 'warning' with this- the
> > parameter specifies the file to use.  If you don't know what file to
> > use, how you can possibly do anything but return an error?
> 
>    As the file and location are fixed, we can go-ahead and write to
> it, but I think now we are deciding
>    if someone disables include dir, then we can just disable Alter
> System, so it is better to return error in such
>    situation.

It wouldn't be fixed with this approach.

> > Note that I *like* that about this approach.
> >
> > There are a few other considerations with this-
> >
> > - What should the default be?  (Still thinking 'off' myself)
>      default 'off' is a safe option, as it won't allow users to make
> any change to parameter values until/unless they
>      read from manual, how to use it and what can go wrong, on the
> other side it will be bit hassle for user to use this
>      command. I think 'on' would be better.

Yeah, no, I still think 'off' would be best for this particular option.

> > - What happens if the user specifies 'postgresql.conf'?  I'm thinking we
> >   would disallow such insanity (as that's what it is, imv..) by having
> >   an identifier in the file that this is the PG "auto conf" file.
>           I think we can detect by name and give error.
> > - Should we have such an identifier in auto.conf to indicate that we
> >   created it, to prevent the user from setting it to something they
> >   shouldn't?
>      I think if user plays with this file manually, it can lead to
> problems, that's why earlier we have
>      decided to keep a note on top of file which will indicate, do not
> edit this file manually.
>      I believe that should be sufficient.

I agree that having such a disclaimer at the top of the file is a good
idea.  I'm not completely convinced that's sufficient but it's certainly
better than nothing.

> > - What's the "bootstrap" mode; iow, if a user enables the option but the
> >   file doesn't exist, what do we do?  With this approach, I'd be
> >   inclined to say we simply create it and put the marker to indicate
> >   it's "our" file.
> 
>      Alter System will create the file if doesn't exist.

... Only if it's enabled though.

> > - Should we allow it to be outside of the data dir?  We could simply log
> >   an error and ignore the parameter if it's more than a simple filename.
> 
>      This should be an error, the file location and name will be fixed.

Not with this proposal...  If it's fixed then it makes no sense to make it
look like it can be modified.

	Thanks,

		Stephen

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila16@gmail.com> — 2013-08-23T13:11:04Z

On Fri, Aug 23, 2013 at 6:01 PM, Stephen Frost <sfrost@snowman.net> wrote:
> * Amit Kapila (amit.kapila16@gmail.com) wrote:
>> On Thu, Aug 22, 2013 at 6:06 PM, Stephen Frost <sfrost@snowman.net> wrote:
>> > * Amit Kapila (amit.kapila16@gmail.com) wrote:
>> >> Enable/Disable reading of auto file
>> >> -----------------------------------------------------
>> >> a. Have a new include in postresql.conf
>> >>     #include_auto_conf_file    postgresql.auto.conf
>> >>     as it is a special include, we can read this file relative to data
>> >> directory.
>> >>
>> >> Enable/Disable Alter System command
>> >> -----------------------------------------------------------
>> >> This can be achieved in 3 ways:
>> >> a. Check before executing Alter System if include directive is
>> >> disabled, then just issue a warning to user and proceed with command.
>> >> b. Check before executing Alter System if include directive is
>> >> disabled, then just issue an error and stop.
>> >
>> > It doesn't make sense for it to be a 'warning' with this- the
>> > parameter specifies the file to use.  If you don't know what file to
>> > use, how you can possibly do anything but return an error?
>>
>>    As the file and location are fixed, we can go-ahead and write to
>> it, but I think now we are deciding
>>    if someone disables include dir, then we can just disable Alter
>> System, so it is better to return error in such
>>    situation.
>
> It wouldn't be fixed with this approach.
>
>> > Note that I *like* that about this approach.
>> >
>> > There are a few other considerations with this-
>> >
>> > - What should the default be?  (Still thinking 'off' myself)
>>      default 'off' is a safe option, as it won't allow users to make
>> any change to parameter values until/unless they
>>      read from manual, how to use it and what can go wrong, on the
>> other side it will be bit hassle for user to use this
>>      command. I think 'on' would be better.
>
> Yeah, no, I still think 'off' would be best for this particular option.
>
>> > - What happens if the user specifies 'postgresql.conf'?  I'm thinking we
>> >   would disallow such insanity (as that's what it is, imv..) by having
>> >   an identifier in the file that this is the PG "auto conf" file.
>>           I think we can detect by name and give error.
>> > - Should we have such an identifier in auto.conf to indicate that we
>> >   created it, to prevent the user from setting it to something they
>> >   shouldn't?
>>      I think if user plays with this file manually, it can lead to
>> problems, that's why earlier we have
>>      decided to keep a note on top of file which will indicate, do not
>> edit this file manually.
>>      I believe that should be sufficient.
>
> I agree that having such a disclaimer at the top of the file is a good
> idea.  I'm not completely convinced that's sufficient but it's certainly
> better than nothing.
>
>> > - What's the "bootstrap" mode; iow, if a user enables the option but the
>> >   file doesn't exist, what do we do?  With this approach, I'd be
>> >   inclined to say we simply create it and put the marker to indicate
>> >   it's "our" file.
>>
>>      Alter System will create the file if doesn't exist.
>
> ... Only if it's enabled though.

        Yes.
>
>> > - Should we allow it to be outside of the data dir?  We could simply log
>> >   an error and ignore the parameter if it's more than a simple filename.
>>
>>      This should be an error, the file location and name will be fixed.
>
> Not with this proposal...  If it's fixed then it makes no sense to make it
> look like it can be modified.

   This proposal is to have a special include which user can only use
for enable/disable
   which means he can remove symbol '#' or add '#'.
   We cannot stop user from changing file name or add some different
location, but that can lead to problems.
   We can have a note on top of this include indicating it is only for
enable/disable.

   I think if we add more meaning to it, like allow user to change it,
handling and defining of that will be bit complex.

With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Martijn van Oosterhout <kleptog@svana.org> — 2013-08-24T12:38:36Z

On Fri, Aug 23, 2013 at 06:41:04PM +0530, Amit Kapila wrote:
> > Not with this proposal...  If it's fixed then it makes no sense to make it
> > look like it can be modified.
> 
>    This proposal is to have a special include which user can only use
> for enable/disable
>    which means he can remove symbol '#' or add '#'.
>    We cannot stop user from changing file name or add some different
> location, but that can lead to problems.
>    We can have a note on top of this include indicating it is only for
> enable/disable.

Note, my whole purpose for suggesting something like:

include_auto_conf_file    postgresql.auto.conf

is because I want the file location to be configurable. If I put in my
configuration:

include_auto_conf_file    /etc/postgresql/9.4/postgresql.auto.conf

it better put the options there. And if I comment the line out ALTER
SYSTEM should stop working.  If I put it at the beginning of the config
then any other option in the file will override it (which we can detect
and warn about).  If I put it at the end of the file, ALTER SYSTEM
overrides any statically configured options.

And I can imagine hosting providers putting the configuration for
memory usage, shared library directories and other such options after,
and options like cpu_cost, enable_merge_join, etc before.  That way
they have fine grained control over which options the user can set and
which not.

>    I think if we add more meaning to it, like allow user to change it,
> handling and defining of that will be bit complex.

Letting the user configure the location seems like common curtesy. Note
this line isn't in itself a GUC, so you can't configure it via ALTER
SYSTEM.

Have a nice day,
-- 
Martijn van Oosterhout   <kleptog@svana.org>   http://svana.org/kleptog/
> He who writes carelessly confesses thereby at the very outset that he does
> not attach much importance to his own thoughts.
   -- Arthur Schopenhauer

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila16@gmail.com> — 2013-08-25T05:00:53Z

On Sat, Aug 24, 2013 at 6:08 PM, Martijn van Oosterhout
<kleptog@svana.org> wrote:
> On Fri, Aug 23, 2013 at 06:41:04PM +0530, Amit Kapila wrote:
>> > Not with this proposal...  If it's fixed then it makes no sense to make it
>> > look like it can be modified.
>>
>>    This proposal is to have a special include which user can only use
>> for enable/disable
>>    which means he can remove symbol '#' or add '#'.
>>    We cannot stop user from changing file name or add some different
>> location, but that can lead to problems.
>>    We can have a note on top of this include indicating it is only for
>> enable/disable.
>
> Note, my whole purpose for suggesting something like:
>
> include_auto_conf_file    postgresql.auto.conf
>
> is because I want the file location to be configurable. If I put in my
> configuration:
>
> include_auto_conf_file    /etc/postgresql/9.4/postgresql.auto.conf
>
> it better put the options there. And if I comment the line out ALTER
> SYSTEM should stop working.  If I put it at the beginning of the config
> then any other option in the file will override it (which we can detect
> and warn about).  If I put it at the end of the file, ALTER SYSTEM
> overrides any statically configured options.
>
> And I can imagine hosting providers putting the configuration for
> memory usage, shared library directories and other such options after,
> and options like cpu_cost, enable_merge_join, etc before.  That way
> they have fine grained control over which options the user can set and
> which not.

   Thanks for your suggestion.
   Above usecase can be achieved even if the file is not configurable.

>>    I think if we add more meaning to it, like allow user to change it,
>> handling and defining of that will be bit complex.
>
> Letting the user configure the location seems like common curtesy. Note
> this line isn't in itself a GUC, so you can't configure it via ALTER
> SYSTEM.

In general yes it is better to give control to user for configuring
the location, but as this file will
be used for internal purpose and will be modified by system and not by
user, so it is better to
be in data directory. The similar thing was discussed previously on
this thread and it is suggested
to have this file in data directory.


With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-26T17:20:29Z

Martijn,

* Martijn van Oosterhout (kleptog@svana.org) wrote:
> Note, my whole purpose for suggesting something like:
> 
> include_auto_conf_file    postgresql.auto.conf
> 
> is because I want the file location to be configurable. If I put in my
> configuration:
> 
> include_auto_conf_file    /etc/postgresql/9.4/postgresql.auto.conf

I'm not following the use-case here.

Why would it make sense for a file which is entirely managed by PG
automatically to be in /etc?  Of course, by the same token you might ask
why it makes sense to let the user pick it at all, which holds some
merit- but I liked your idea because then an admin doesn't have to go
looking for the file but instead knows where it is.  Perhaps comments in
the file stating where the auto.conf lives would be sufficient, but
those are too often nuked. :(

> it better put the options there. And if I comment the line out ALTER
> SYSTEM should stop working.  If I put it at the beginning of the config
> then any other option in the file will override it (which we can detect
> and warn about).  If I put it at the end of the file, ALTER SYSTEM
> overrides any statically configured options.
> 
> And I can imagine hosting providers putting the configuration for
> memory usage, shared library directories and other such options after,
> and options like cpu_cost, enable_merge_join, etc before.  That way
> they have fine grained control over which options the user can set and
> which not.

For my part, I'd honestly rather have the first things found be what's
picked and later things be ignored.  If you want it controlled by ALTER
SYSTEM, then don't set it in postgresql.conf.  The problem with that is
there's no "bootstrap" mechanism without actually modifying such configs
or making the configs be in auto.conf to begin with, neither of which is
ideal, imv.

I really hate the idea that someone could configure 'X' in
postgresql.conf and because the auto.conf line is later in the file,
it's able to override the original setting.  Does not strike me as
intuitive at all.

	Thanks,

		Stephen

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila16@gmail.com> — 2013-08-27T03:34:00Z

On Mon, Aug 26, 2013 at 10:50 PM, Stephen Frost <sfrost@snowman.net> wrote:
> Martijn,
>
> * Martijn van Oosterhout (kleptog@svana.org) wrote:
>> Note, my whole purpose for suggesting something like:
>>
>> include_auto_conf_file    postgresql.auto.conf
>>
>> is because I want the file location to be configurable. If I put in my
>> configuration:
>>
>> include_auto_conf_file    /etc/postgresql/9.4/postgresql.auto.conf
>
> I'm not following the use-case here.
>
> Why would it make sense for a file which is entirely managed by PG
> automatically to be in /etc?  Of course, by the same token you might ask
> why it makes sense to let the user pick it at all, which holds some
> merit- but I liked your idea because then an admin doesn't have to go
> looking for the file but instead knows where it is.  Perhaps comments in
> the file stating where the auto.conf lives would be sufficient, but
> those are too often nuked. :(
>
>> it better put the options there. And if I comment the line out ALTER
>> SYSTEM should stop working.  If I put it at the beginning of the config
>> then any other option in the file will override it (which we can detect
>> and warn about).  If I put it at the end of the file, ALTER SYSTEM
>> overrides any statically configured options.
>>
>> And I can imagine hosting providers putting the configuration for
>> memory usage, shared library directories and other such options after,
>> and options like cpu_cost, enable_merge_join, etc before.  That way
>> they have fine grained control over which options the user can set and
>> which not.
>
> For my part, I'd honestly rather have the first things found be what's
> picked and later things be ignored.  If you want it controlled by ALTER
> SYSTEM, then don't set it in postgresql.conf.  The problem with that is
> there's no "bootstrap" mechanism without actually modifying such configs
> or making the configs be in auto.conf to begin with, neither of which is
> ideal, imv.
>
> I really hate the idea that someone could configure 'X' in
> postgresql.conf and because the auto.conf line is later in the file,
> it's able to override the original setting.  Does not strike me as
> intuitive at all.

This is currently how include mechanism works in postgresql.conf,
changing that for this special case can be costly and moreover the
specs for this patch were layout from beginning that way.

However, we can have some mechanism for appending values (if feasible)
as suggested in mail below, after initial patch is done
http://www.postgresql.org/message-id/52015414.9060900@cybertec.at


With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Bruce Momjian <bruce@momjian.us> — 2013-08-28T17:08:54Z

On Tue, Aug 27, 2013 at 09:04:00AM +0530, Amit Kapila wrote:
> > For my part, I'd honestly rather have the first things found be what's
> > picked and later things be ignored.  If you want it controlled by ALTER
> > SYSTEM, then don't set it in postgresql.conf.  The problem with that is
> > there's no "bootstrap" mechanism without actually modifying such configs
> > or making the configs be in auto.conf to begin with, neither of which is
> > ideal, imv.
> >
> > I really hate the idea that someone could configure 'X' in
> > postgresql.conf and because the auto.conf line is later in the file,
> > it's able to override the original setting.  Does not strike me as
> > intuitive at all.
> 
> This is currently how include mechanism works in postgresql.conf,
> changing that for this special case can be costly and moreover the
> specs for this patch were layout from beginning that way.

Agreed.  If you are worried about ALTER SYSTEM changing postgresql.conf
settings, you should move the include_auto line to the top of
postgresql.conf, but I don't think that should be the default.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + It's impossible for everything to be true. +

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-28T18:30:41Z

* Bruce Momjian (bruce@momjian.us) wrote:
> On Tue, Aug 27, 2013 at 09:04:00AM +0530, Amit Kapila wrote:
> > > I really hate the idea that someone could configure 'X' in
> > > postgresql.conf and because the auto.conf line is later in the file,
> > > it's able to override the original setting.  Does not strike me as
> > > intuitive at all.
> > 
> > This is currently how include mechanism works in postgresql.conf,
> > changing that for this special case can be costly and moreover the
> > specs for this patch were layout from beginning that way.
> 
> Agreed.  If you are worried about ALTER SYSTEM changing postgresql.conf
> settings, you should move the include_auto line to the top of
> postgresql.conf, but I don't think that should be the default.

While I appreciate that there are bootstrap-type issues with this, I
really don't like this idea of "later stuff can just override earlier
stuff".

include files and conf.d-style options are for breaking the config up,
not to allow you to override options because a file came later than an
earlier file.  Our particular implementation of config-file reading
happens to lend itself to later-definition-wins, but that's really
counter-intuitive for anyone unfamiliar with PG, imv.

	Thanks,

		Stephen

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Tom Lane <tgl@sss.pgh.pa.us> — 2013-08-28T18:37:12Z

Stephen Frost <sfrost@snowman.net> writes:
> While I appreciate that there are bootstrap-type issues with this, I
> really don't like this idea of "later stuff can just override earlier
> stuff".

> include files and conf.d-style options are for breaking the config up,
> not to allow you to override options because a file came later than an
> earlier file.  Our particular implementation of config-file reading
> happens to lend itself to later-definition-wins, but that's really
> counter-intuitive for anyone unfamiliar with PG, imv.

I don't follow this argument at all.  Do you know any software with text
config files that will act differently from this if the same setting is
listed twice?  "Last one wins" is certainly what I'd expect.

			regards, tom lane

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Bruce Momjian <bruce@momjian.us> — 2013-08-28T18:47:47Z

On Wed, Aug 28, 2013 at 02:30:41PM -0400, Stephen Frost wrote:
> * Bruce Momjian (bruce@momjian.us) wrote:
> > On Tue, Aug 27, 2013 at 09:04:00AM +0530, Amit Kapila wrote:
> > > > I really hate the idea that someone could configure 'X' in
> > > > postgresql.conf and because the auto.conf line is later in the file,
> > > > it's able to override the original setting.  Does not strike me as
> > > > intuitive at all.
> > > 
> > > This is currently how include mechanism works in postgresql.conf,
> > > changing that for this special case can be costly and moreover the
> > > specs for this patch were layout from beginning that way.
> > 
> > Agreed.  If you are worried about ALTER SYSTEM changing postgresql.conf
> > settings, you should move the include_auto line to the top of
> > postgresql.conf, but I don't think that should be the default.
> 
> While I appreciate that there are bootstrap-type issues with this, I
> really don't like this idea of "later stuff can just override earlier
> stuff".
> 
> include files and conf.d-style options are for breaking the config up,
> not to allow you to override options because a file came later than an
> earlier file.  Our particular implementation of config-file reading
> happens to lend itself to later-definition-wins, but that's really
> counter-intuitive for anyone unfamiliar with PG, imv.

Agreed, but I think this is a much larger issue than ALTER SYSTEM SET.

I think changing behavior to first-seen would only add to confusion. 
What we really need is a WARNING when a later postgresql.conf setting
overrides an earlier one, and ALTER SYSTEM SET's config file could
behave the same, so you would know right away when you were overriding
something in postgresql.conf that appeared earlier.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + It's impossible for everything to be true. +

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-28T19:10:29Z

* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> Stephen Frost <sfrost@snowman.net> writes:
> > While I appreciate that there are bootstrap-type issues with this, I
> > really don't like this idea of "later stuff can just override earlier
> > stuff".
> 
> > include files and conf.d-style options are for breaking the config up,
> > not to allow you to override options because a file came later than an
> > earlier file.  Our particular implementation of config-file reading
> > happens to lend itself to later-definition-wins, but that's really
> > counter-intuitive for anyone unfamiliar with PG, imv.
> 
> I don't follow this argument at all.  Do you know any software with text
> config files that will act differently from this if the same setting is
> listed twice?  "Last one wins" is certainly what I'd expect.

Have you tried having the same mount specified in multiple files in
fstab.d..?  Also, aiui (not a big exim fan personally), duplicate
definitions in an exim4/conf.d would result in an error.  Playing around
in apache2/sites-enabled, it appears to be "first wins" wrt virtual
hosts.

There's a number of cases where only a single value is being set and
subseqeunt files are 'additive' (eg: ld.so.conf.d), so they don't have
this issue.  Similar to that are script directories, which simply run a
set of scripts in the $DIR.d and, as it's the scripts themselves which
are the 'parameters', and being files in a directory, you can't
duplicate them.  Others (eg: pam.d) define the file name to be an
enclosing context, again preventing duplication by using the filename
itself.

There are counter-examples also; sysctl.d will replace what's already
been set, so perhaps it simply depends on an individual's experience.
For my part, I'd much prefer an error or warning saying "you've got
duplicate definitions of X" than a last-wins approach, though perhaps
that's just because I like things to be very explicit and clear.
Allowing multiple definitions of the same parameter to be valid isn't
'clear' to me.

	Thanks,

		Stephen

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-28T19:15:14Z

* Bruce Momjian (bruce@momjian.us) wrote:
> Agreed, but I think this is a much larger issue than ALTER SYSTEM SET.

Yeah, true.

> I think changing behavior to first-seen would only add to confusion. 
> What we really need is a WARNING when a later postgresql.conf setting
> overrides an earlier one, and ALTER SYSTEM SET's config file could
> behave the same, so you would know right away when you were overriding
> something in postgresql.conf that appeared earlier.

A warning would be nice.  If only everyone read the log files. :/

	Thanks,

		Stephen

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Alvaro Herrera <alvherre@2ndquadrant.com> — 2013-08-28T19:23:36Z

Stephen Frost escribió:

> There are counter-examples also; sysctl.d will replace what's already
> been set, so perhaps it simply depends on an individual's experience.
> For my part, I'd much prefer an error or warning saying "you've got
> duplicate definitions of X" than a last-wins approach, though perhaps
> that's just because I like things to be very explicit and clear.
> Allowing multiple definitions of the same parameter to be valid isn't
> 'clear' to me.

One of the elements that had to be in place for this whole thing of
multiple configuration files to be allowed was for pg_settings to
include precise information about, for each setting, where is its value
coming from.  I doubt any of the systems you mentioned have comparable
functionality.  If the admin has trouble figuring it out, he needs only
look into that view.

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Bruce Momjian <bruce@momjian.us> — 2013-08-28T19:23:53Z

On Wed, Aug 28, 2013 at 03:15:14PM -0400, Stephen Frost wrote:
> * Bruce Momjian (bruce@momjian.us) wrote:
> > Agreed, but I think this is a much larger issue than ALTER SYSTEM SET.
> 
> Yeah, true.
> 
> > I think changing behavior to first-seen would only add to confusion. 
> > What we really need is a WARNING when a later postgresql.conf setting
> > overrides an earlier one, and ALTER SYSTEM SET's config file could
> > behave the same, so you would know right away when you were overriding
> > something in postgresql.conf that appeared earlier.
> 
> A warning would be nice.  If only everyone read the log files. :/

I would expect ALTER SYSTEM SET to return a WARNING in such cases too.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + It's impossible for everything to be true. +

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Robert Haas <robertmhaas@gmail.com> — 2013-08-29T16:13:04Z

On Wed, Aug 28, 2013 at 3:10 PM, Stephen Frost <sfrost@snowman.net> wrote:
> * Tom Lane (tgl@sss.pgh.pa.us) wrote:
>> Stephen Frost <sfrost@snowman.net> writes:
>> > While I appreciate that there are bootstrap-type issues with this, I
>> > really don't like this idea of "later stuff can just override earlier
>> > stuff".
>>
>> > include files and conf.d-style options are for breaking the config up,
>> > not to allow you to override options because a file came later than an
>> > earlier file.  Our particular implementation of config-file reading
>> > happens to lend itself to later-definition-wins, but that's really
>> > counter-intuitive for anyone unfamiliar with PG, imv.
>>
>> I don't follow this argument at all.  Do you know any software with text
>> config files that will act differently from this if the same setting is
>> listed twice?  "Last one wins" is certainly what I'd expect.
>
> Have you tried having the same mount specified in multiple files in
> fstab.d..?  Also, aiui (not a big exim fan personally), duplicate
> definitions in an exim4/conf.d would result in an error.  Playing around
> in apache2/sites-enabled, it appears to be "first wins" wrt virtual
> hosts.
>
> There's a number of cases where only a single value is being set and
> subseqeunt files are 'additive' (eg: ld.so.conf.d), so they don't have
> this issue.  Similar to that are script directories, which simply run a
> set of scripts in the $DIR.d and, as it's the scripts themselves which
> are the 'parameters', and being files in a directory, you can't
> duplicate them.  Others (eg: pam.d) define the file name to be an
> enclosing context, again preventing duplication by using the filename
> itself.
>
> There are counter-examples also; sysctl.d will replace what's already
> been set, so perhaps it simply depends on an individual's experience.
> For my part, I'd much prefer an error or warning saying "you've got
> duplicate definitions of X" than a last-wins approach, though perhaps
> that's just because I like things to be very explicit and clear.
> Allowing multiple definitions of the same parameter to be valid isn't
> 'clear' to me.

This may be true, but I think it's got little if anything to do with
the current patch.  There are lots of things that I don't like about
our GUC machinery, too, but the goal of this thread shouldn't be to
fix them all, but to get a useful piece of functionality added.

The thing that we should keep in mind here is that it's *already*
possible - TODAY - for users to overwrite postgresql.conf with a new
set of settings.  pgAdmin has this functionality built-in; it only
requires that you install the "adminpack" contrib module, which we
ship.  According to Dave Page, pgAdmin will even offer to run CREATE
EXTENSION adminpack for you, if you haven't done that already.  Also
according to Dave Page, if two users try to use it concurrently, you
will get exactly the sort of awful results that you might expect.  If
it gets killed halfway through rewriting the file, too bad for you.

The sorts of watered-down half-features being proposed here are not
going to do anything to address that situation.  If there are
restrictions on what GUCs can be changed with this feature, or if the
feature is disabled by default, or if you can only use it when the
moon is full and you hop on your left foot while spinning around
sideways, then pgAdmin (and any other, similar tools) are just going
to keep doing what they do today in the same crappy, unsafe way they
currently do it.  Meanwhile, people who use psql are going to continue
to find themselves without a reasonable API for doing the same thing
that can be done easily using pgAdmin.

I think the goals of this patch should be to (1) let users of other
clients manipulate the startup configuration just as easily as we can
already do using pgAdmin and (2) remove some of the concurrency
hazards that pgAdmin introduces, for example by using locking and
atomic renames.  Restricting the functionality to something less than
what pgAdmin provides will just cause people to ignore the new
mechanism and use the one that already exists and, by and large,
works.  And trying to revise other aspects of how GUCs and config
files work as part of this effort is not reasonably related to this
patch, and should be kept out of the discussion.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-29T17:42:14Z

Robert,

* Robert Haas (robertmhaas@gmail.com) wrote:
> The sorts of watered-down half-features being proposed here are not
> going to do anything to address that situation.  If there are
> restrictions on what GUCs can be changed with this feature, or if the
> feature is disabled by default, or if you can only use it when the
> moon is full and you hop on your left foot while spinning around
> sideways, then pgAdmin (and any other, similar tools) are just going
> to keep doing what they do today in the same crappy, unsafe way they
> currently do it.  Meanwhile, people who use psql are going to continue
> to find themselves without a reasonable API for doing the same thing
> that can be done easily using pgAdmin.

To be honest, I don't find the arguments of "pgAdmin does it badly"
nor "psql users want this ability" (which I find difficult to believe)
to be particularlly compelling reasons to have a dangerous
implementation (even if it's better than 'adminpack' today) in core.

If it's in core rather than in contrib it's going to be deployed a great
deal farther with a large increase in userbase.  I've already stated
that if this is in contrib that my concerns are much less.

> I think the goals of this patch should be to (1) let users of other
> clients manipulate the startup configuration just as easily as we can
> already do using pgAdmin,

Which could be done already through use of adminpack..  The capabilities
exposed there could be used by other clients.  The fact that none of the
other clients have chosen to do so could be an indication that this
ability isn't terribly 'in demand'.

> and (2) remove some of the concurrency
> hazards that pgAdmin introduces, for example by using locking and
> atomic renames.

Why can't adminpack do that..?

> Restricting the functionality to something less than
> what pgAdmin provides will just cause people to ignore the new
> mechanism and use the one that already exists and, by and large,
> works.  And trying to revise other aspects of how GUCs and config
> files work as part of this effort is not reasonably related to this
> patch, and should be kept out of the discussion.

We're talking about modifying config files through an interface and you
wish to exclude all discussion about how those config files are handled.
That leads to a result that only an adminpack-like solution is an
option, to which I respond "use and improve adminpack then".  If we want
something that works well in *core* then I don't think we can exclude
how core reads and handles config files from the discussion.  We need a
real solution, not another adminpack-like hack.

	Thanks,

		Stephen

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Robert Haas <robertmhaas@gmail.com> — 2013-08-29T19:07:35Z

On Thu, Aug 29, 2013 at 1:42 PM, Stephen Frost <sfrost@snowman.net> wrote:
> To be honest, I don't find the arguments of "pgAdmin does it badly"
> nor "psql users want this ability" (which I find difficult to believe)
> to be particularlly compelling reasons to have a dangerous
> implementation (even if it's better than 'adminpack' today) in core.

I don't understand how you can find that difficult to believe.  I'm a
psql user and I want it.  Josh Berkus is a psql user and he wants it.
And there are numerous statements of support on these threads from
other people as well.  The sheer volume of discussion on this topic,
and the fact that it has not gone away after years of wrangling, is a
clear indication that people do, in fact, want it.

To be honest, I think the argument that this is dangerous is pretty
ridiculous.  AFAICS, the only argument anyone's advanced for this
being dangerous is the theory that you might accidentally put
something in your postgresql.conf file that makes the server not
start.  However, the reality is that the superuser has MANY, MANY ways
of killing the database cluster as things stand.  Consider the
ever-popular "DELETE FROM pg_proc".  That will not only render your
database unusable, but it's a hell of a lot harder to recover from
than anything you might do to postgresql.conf.  Therefore, from where
I'm sitting, this is like telling a head of state with the ability to
launch nuclear weapons which will destroy the planet that he isn't
allowed to bring his swiss army knife on board an aircraft.

Now, you can argue that people are more likely to render the database
nonfunctional by changing GUC settings than they are to do it by
corrupting the system catalogs, but I'm not sure I believe it.  We
can, after all, validate that any setting a user supplies is a valid
value for that setting before writing it out to the configuration
file.  It might still make the system fail to start if - for example -
it causes too many semaphores to be reserved, or something like that.
But why should we think that such mistakes will be common?  If
anything, it sounds less error-prone to me than hand-editing the
configuration file, where typing something like on_exit_error=maybe
will make the server fail to start.  We can eliminate that whole
category of error, at least for people who choose to use the new
tools.

If you're using the term "dangerous" to refer to a security exposure,
I concede there is some incremental exposure from allowing this by
default.  But it's not a terribly large additional exposure.  It's
already the case that if adminpack is available the super-user can do
whatever he or she wants, because she or he can write to arbitrary
files inside the data directory.  Even if not, for most intents and
purposes, ALTER DATABASE my_main_database SET whatever = thing is
functionally equivalent to modifying postgresql.conf.  Some settings
can't be modified that way, but so what?  AFAICS, about the worst
thing the user can do is ALTER SYSTEM SET shared_preload_libraries =
'rootkit'.  But if the user has the ability to install rootkit.so, the
sysadmin is already screwed.  And aside from that sort of thing, I
don't really see what can happen that's any worse than what a rogue
superuser can already do as things stand today.

> If it's in core rather than in contrib it's going to be deployed a great
> deal farther with a large increase in userbase.  I've already stated
> that if this is in contrib that my concerns are much less.

I don't really see a compelling reason why it can't or shouldn't be in
core.  But having something better in contrib would still be an
improvement on the status quo.

>> I think the goals of this patch should be to (1) let users of other
>> clients manipulate the startup configuration just as easily as we can
>> already do using pgAdmin,
>
> Which could be done already through use of adminpack..  The capabilities
> exposed there could be used by other clients.  The fact that none of the
> other clients have chosen to do so could be an indication that this
> ability isn't terribly 'in demand'.

Huh?  The problem with adminpack is that it doesn't let you modify
individual configuration settings.  All you can do is rewrite an
entire file.  I guess somebody could write a specialized client that
just uses that infrastructure to rewrite postgresql.conf.  For all I
know, someone has.  Even if not, I don't think that you can use that
to prove that people don't care about this feature.  If nobody cares,
why are there 400 emails on this topic?!

>> and (2) remove some of the concurrency
>> hazards that pgAdmin introduces, for example by using locking and
>> atomic renames.
>
> Why can't adminpack do that..?

It could.  But it doesn't.  We could improve it to do that, and that
might be worthwhile, but it still wouldn't be as nice as what's being
proposed here.

>> Restricting the functionality to something less than
>> what pgAdmin provides will just cause people to ignore the new
>> mechanism and use the one that already exists and, by and large,
>> works.  And trying to revise other aspects of how GUCs and config
>> files work as part of this effort is not reasonably related to this
>> patch, and should be kept out of the discussion.
>
> We're talking about modifying config files through an interface and you
> wish to exclude all discussion about how those config files are handled.
> That leads to a result that only an adminpack-like solution is an
> option, to which I respond "use and improve adminpack then".  If we want
> something that works well in *core* then I don't think we can exclude
> how core reads and handles config files from the discussion.  We need a
> real solution, not another adminpack-like hack.

I'm not sure what you mean about "all discussion about how those
config files are handled".  I think it's entirely reasonable to
discuss how where the automatically-written configuration settings
will be stored, and how they'll interact with manually-generated
files.  But IIUC you have proposed:

1. disabling the feature by default, and providing no way for it to be
turned on remotely, and
2. even when the feature is enabled, only allowing a subset of the
settings to be changed with it, and
3. also changing the interpretation of the config files so that we
have a first-wins rules instead of a last-wins rule.

If we do either #1 or #2, this won't be a plausible substitute for the
functionality that adminpack offers today.  #3 is a bad idea in my
book - it would break some of my existing benchmarking scripts, which
do initdb; cat >>$PGDATA/postgresql.conf <<EOM.  But even if it were a
good idea, it isn't a necessary prerequisite for this patch.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Andres Freund <andres@2ndquadrant.com> — 2013-08-29T22:25:46Z

On 2013-08-29 15:07:35 -0400, Robert Haas wrote:
> On Thu, Aug 29, 2013 at 1:42 PM, Stephen Frost <sfrost@snowman.net> wrote:
> > To be honest, I don't find the arguments of "pgAdmin does it badly"
> > nor "psql users want this ability" (which I find difficult to believe)
> > to be particularlly compelling reasons to have a dangerous
> > implementation (even if it's better than 'adminpack' today) in core.
> 
> I don't understand how you can find that difficult to believe.  I'm a
> psql user and I want it.  Josh Berkus is a psql user and he wants it.
> And there are numerous statements of support on these threads from
> other people as well.  The sheer volume of discussion on this topic,
> and the fact that it has not gone away after years of wrangling, is a
> clear indication that people do, in fact, want it.
> 
> To be honest, I think the argument that this is dangerous is pretty
> ridiculous.

+waytoomuch.

> > If it's in core rather than in contrib it's going to be deployed a great
> > deal farther with a large increase in userbase.  I've already stated
> > that if this is in contrib that my concerns are much less.
> 
> I don't really see a compelling reason why it can't or shouldn't be in
> core.  But having something better in contrib would still be an
> improvement on the status quo.

I don't see much argument for putting it into contrib. One class of
users this will benefit is relatively new ones, possibly using some
GUI. Adding some additional complexity for them to enable the feature
seems pointless to me.

If you don't want your installation to use it, tell you ops people not
to do so. They are superusers, they need to have the ability to follow
some rules you make up internally.

The energy wasted in a good part of this massive 550+ messages thread is
truly saddening. We all (c|sh)ould have spent that time making PG more
awesome instead.

Greetings,

Andres Freund

-- 
 Andres Freund	                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-29T22:42:13Z

On Thursday, August 29, 2013, Andres Freund wrote:

> On 2013-08-29 15:07:35 -0400, Robert Haas wrote:
> > I don't really see a compelling reason why it can't or shouldn't be in
> > core.  But having something better in contrib would still be an
> > improvement on the status quo.
>
> I don't see much argument for putting it into contrib. One class of
> users this will benefit is relatively new ones, possibly using some
> GUI. Adding some additional complexity for them to enable the feature
> seems pointless to me.

I keep wondering what this fantastic new GUI that isn't pgAdmin is, and why
it wouldn't be able to use the exact same mechanism that pgAdmin uses and
provides in a few simple steps to enable this.

> If you don't want your installation to use it, tell you ops people not
> to do so. They are superusers, they need to have the ability to follow
> some rules you make up internally.

The OPs people are the ones that will be upset with this because the DBAs
will be modifying configs which OPs rightfully claim as theirs. If they
have a way to prevent it then perhaps it's not terrible but they'd also
need to know to disable this new "feature". As for ALTER DATABASE- I would
be happier with encouraging use of that (or providing an ALTER CLUSTER) for
those thing it makes sense and works for and removing those from being in
postgresql.conf. I still feel things like listen_addresses shouldn't be
changed through this.

> The energy wasted in a good part of this massive 550+ messages thread is
> truly saddening. We all (c|sh)ould have spent that time making PG more
> awesome instead.
>

Perhaps not understood by all, but keeping PG awesome involves more than
adding every feature proposed- it also means saying no sometimes; to
features, to new GUCs, even to micro-optimizations when they're overly
complicated and offer only minimal or questionable improvements.

Thanks,

Stephen

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Andres Freund <andres@2ndquadrant.com> — 2013-08-29T23:10:15Z

On 2013-08-29 18:42:13 -0400, Stephen Frost wrote:
> On Thursday, August 29, 2013, Andres Freund wrote:
> > The energy wasted in a good part of this massive 550+ messages thread is
> > truly saddening. We all (c|sh)ould have spent that time making PG more
> > awesome instead.

> Perhaps not understood by all, but keeping PG awesome involves more than
> adding every feature proposed- it also means saying no sometimes; to
> features, to new GUCs, even to micro-optimizations when they're overly
> complicated and offer only minimal or questionable improvements.

But that doesn't mean that endlessly discussing in circles is a
worthwile thing to do. The discussion essentially hasn't progressed
towards concensus in the last months at all besides involving most
active hackers at some point. If anything it has become more
contentious.

Obviously lots of people have strong opinions. Now we need to make
decisions. Even if it ends up being ones I judge to be ridiculously bad.

> > On 2013-08-29 15:07:35 -0400, Robert Haas wrote:
> > > I don't really see a compelling reason why it can't or shouldn't be in
> > > core.  But having something better in contrib would still be an
> > > improvement on the status quo.
> >
> > I don't see much argument for putting it into contrib. One class of
> > users this will benefit is relatively new ones, possibly using some
> > GUI. Adding some additional complexity for them to enable the feature
> > seems pointless to me.

> I keep wondering what this fantastic new GUI that isn't pgAdmin is, and why
> it wouldn't be able to use the exact same mechanism that pgAdmin uses and
> provides in a few simple steps to enable this.

To quote Robert two mails up:

> Huh?  The problem with adminpack is that it doesn't let you modify
> individual configuration settings.  All you can do is rewrite an
> entire file.  I guess somebody could write a specialized client that
> just uses that infrastructure to rewrite postgresql.conf.  For all I
> know, someone has.  Even if not, I don't think that you can use that
> to prove that people don't care about this feature.  If nobody cares,
> why are there 400 emails on this topic?!

Also, doing it the adminpack way lacks even the most basic validity
checks. And that's not really changeable.

Presumably one major reason why we don't have other|good GUIs is that
it's ridicuously hard to make them work to an interesting extent with
the current infrastructure.

> > If you don't want your installation to use it, tell you ops people not
> > to do so. They are superusers, they need to have the ability to follow
> > some rules you make up internally.

> The OPs people are the ones that will be upset with this because the DBAs
> will be modifying configs which OPs rightfully claim as theirs. If they
> have a way to prevent it then perhaps it's not terrible but they'd also
> need to know to disable this new "feature". As for ALTER DATABASE- I would
> be happier with encouraging use of that (or providing an ALTER CLUSTER) for
> those thing it makes sense and works for and removing those from being in
> postgresql.conf. I still feel things like listen_addresses shouldn't be
> changed through this.

If they give out superuser access it has to be to people who can follow
rules. After all they don't DROP DATABASE; DELETE FROM pg_class; alter
passwords; use adminpack (changing postgresql.conf..); ... All of which
they can do.

Greetings,

Andres Freund

-- 
 Andres Freund	                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-30T00:07:23Z

On Thursday, August 29, 2013, Andres Freund wrote:
>
> To quote Robert two mails up:
>
> > Huh?  The problem with adminpack is that it doesn't let you modify
> > individual configuration settings.  All you can do is rewrite an
> > entire file.

That's clearly fixable.

> I guess somebody could write a specialized client that
> > just uses that infrastructure to rewrite postgresql.conf.  For all I
> > know, someone has.  Even if not, I don't think that you can use that
> > to prove that people don't care about this feature.  If nobody cares,
> > why are there 400 emails on this topic?!

Having 400 emails about it means it's contentious. That's quite different
from having a large demand. It does speak to the author's persistence as
well, but that shouldn't be a surprise.

> Also, doing it the adminpack way lacks even the most basic validity
> checks. And that's not really changeable.

I don't see why..?  Admin pack could certainly be modified to take a
parameter and do appropriate verification before locking an object and
rewriting the file. It's what we're being expected to do in core, after
all. Indeed, we can't even do validity checks on all the options, which is
the crux of what I'm concerned about.

> Presumably one major reason why we don't have other|good GUIs is that
> it's ridicuously hard to make them work to an interesting extent with
> the current infrastructure.

Yet no one has tried to improve admin pack?

> If they give out superuser access it has to be to people who can follow
> rules. After all they don't DROP DATABASE; DELETE FROM pg_class; alter
> passwords; use adminpack (changing postgresql.conf..); ... All of which
> they can do.
>

This completely misses, or perhaps just ignores, the point. Disallowing
super user access can be difficult because there's a lot of *normal* DBA
activities which can't be easily done without it (like changing table
ownership or similar).  The "createrole" option definitely improved things
but we aren't there yet. It's certainly easy to simply not install the
adminpack. The other concerns above are strawmen because they attack a
malicious DBA.  I'm not talking about malicious DBAs but rather a generally
knowledgable DBA who changed shared_buffers up too high and then leaves on
vacation, while the OPs guys need to do a database restart for whatever
reason and then discover it doesn't start.

I bring up these concerns because I have environments where I can see
exactly this happening and I have a hard time believing that I'm somehow
alone.

Thanks,

Stephen

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Robert Haas <robertmhaas@gmail.com> — 2013-08-30T00:56:03Z

On Thu, Aug 29, 2013 at 8:07 PM, Stephen Frost <sfrost@snowman.net> wrote:
> Having 400 emails about it means it's contentious. That's quite different
> from having a large demand. It does speak to the author's persistence as
> well, but that shouldn't be a surprise.

Yet you can't ignore the fact that many people on these threads want
some form of this.

>> Presumably one major reason why we don't have other|good GUIs is that
>> it's ridicuously hard to make them work to an interesting extent with
>> the current infrastructure.
>
> Yet no one has tried to improve admin pack?

No, but multiple people have tried to do ALTER SYSTEM .. SET.  Peter
had a crack at this problem, I fooled around with it (though I don't
think I ever got as far as publishing), and there were various others
as well (Greg Smith?).

> I'm not talking about malicious DBAs but rather a generally
> knowledgable DBA who changed shared_buffers up too high and then leaves on
> vacation, while the OPs guys need to do a database restart for whatever
> reason and then discover it doesn't start.

/me looks at Stephen incredulously.

In the first place, modifying postgresql.conf and not immediately
restarting the server to test your changes is probably the single most
basic DBA error imaginable.  I have a hard time viewing such a person
as generally knowledgeable.  I hope the DBA in question doesn't have
access to the switches, because he's probably the sort of guy who
reassigns switch ports and doesn't type "wr m" afterwards.

In the second place, the same guy can do the same thing today.  He
just has to use "vi".  In fact, I think this is a pretty common
failure mode in poorly-controlled environments where too many people
have access to the configuration files.  Now maybe you're going to
tell me that the ops guys can't modify the configuration file because
they only have SQL-level access, but then how are they managing to
restart the database?  They need to be able to run pg_ctl *as the
postgres user* to do that, and if they have shell access to that
account, all bets are off.

Sure, you can construct a scenario where this matters.  The ops guys
have "sudo postgres pg_ctl" access but adminpack isn't installed and
they have no other way to modify the configuration file.  But that's
just bizarre.  And if that's really the environment you have, then you
can install a loadable module that grabs ProcessUtility_hook and uses
it to forbid ALTER SYSTEM on that machine.  Hell, we can ship such a
thing in contrib.  Problem solved.  But it's surely too obscure a
combination of circumstances to justify disabling this by default.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-30T01:12:28Z

Robert,

Was working on replying to this, but got distracted..

* Robert Haas (robertmhaas@gmail.com) wrote:
> To be honest, I think the argument that this is dangerous is pretty
> ridiculous.  AFAICS, the only argument anyone's advanced for this
> being dangerous is the theory that you might accidentally put
> something in your postgresql.conf file that makes the server not
> start.  However, the reality is that the superuser has MANY, MANY ways
> of killing the database cluster as things stand.  Consider the
> ever-popular "DELETE FROM pg_proc".  

You will not find me argueing to allow that in normal operation, or most
direct-catalog hacking.  I'd be much happier if we had all the ALTER,
etc, options necessary to prevent any need to ever touch the catalogs.
Similairly, it'd be nice to have more permission options which reduce
the need for anyone to be superuser.

> Now, you can argue that people are more likely to render the database
> nonfunctional by changing GUC settings than they are to do it by
> corrupting the system catalogs, but I'm not sure I believe it.  We
> can, after all, validate that any setting a user supplies is a valid
> value for that setting before writing it out to the configuration
> file.  It might still make the system fail to start if - for example -
> it causes too many semaphores to be reserved, or something like that.
> But why should we think that such mistakes will be common?  If
> anything, it sounds less error-prone to me than hand-editing the
> configuration file, where typing something like on_exit_error=maybe
> will make the server fail to start.  We can eliminate that whole
> category of error, at least for people who choose to use the new
> tools.

That category of error is much more likely to get caught by the sysadmin
doing the restart after the update..  ALTER SYSTEM doesn't provide any
way to restart the DB to make sure it worked.  That, in turn, encourages
modification of the config parameters w/o a restart, which is a pretty
bad idea, imv.

> If you're using the term "dangerous" to refer to a security exposure,
> I concede there is some incremental exposure from allowing this by
> default.  But it's not a terribly large additional exposure.  It's
> already the case that if adminpack is available the super-user can do
> whatever he or she wants, because she or he can write to arbitrary
> files inside the data directory.  

This is only true if -contrib is installed, which a *lot* of admins
refuse to do, specifically because of such insane modules as this.
Therefore, I would argue that it's not nearly as small a change wrt
exposure as you believe.  Strictly speaking, I don't believe it's a huge
security hole or something, but I do think it's going to surprise admins
who aren't following the lists.

> Even if not, for most intents and
> purposes, ALTER DATABASE my_main_database SET whatever = thing is
> functionally equivalent to modifying postgresql.conf.  Some settings
> can't be modified that way, but so what?  AFAICS, about the worst
> thing the user can do is ALTER SYSTEM SET shared_preload_libraries =
> 'rootkit'.  

This is assuming a malicious user, and one who has access to the
filesystem, which are pretty big assumptions to be making.  I'm not
trying to make any argument about a malicious user and in general I
don't believe the users of this capability are people who have root
on the box.

> Huh?  The problem with adminpack is that it doesn't let you modify
> individual configuration settings.  All you can do is rewrite an
> entire file.  I guess somebody could write a specialized client that
> just uses that infrastructure to rewrite postgresql.conf.  For all I
> know, someone has.  Even if not, I don't think that you can use that
> to prove that people don't care about this feature.  If nobody cares,
> why are there 400 emails on this topic?!

Addressed this in my reply to Andres.

> > Why can't adminpack do that..?
> 
> It could.  But it doesn't.  We could improve it to do that, and that
> might be worthwhile, but it still wouldn't be as nice as what's being
> proposed here.

This wouldn't be the first time we've said "do it externally and when
we figure out a better answer then we'll put it into core."

> 1. disabling the feature by default, and providing no way for it to be
> turned on remotely, and

Yes, that's quite intentional because it would make someone knowledgable
about the *overall system* be aware of this rather than having it sprung
on them when they discover some change has made the database unable to
start.

> 2. even when the feature is enabled, only allowing a subset of the
> settings to be changed with it, and
> 3. also changing the interpretation of the config files so that we
> have a first-wins rules instead of a last-wins rule.
> 
> If we do either #1 or #2, this won't be a plausible substitute for the
> functionality that adminpack offers today.  

While I appreciate that the original thrust behind this may be trying to
replace adminpack, I find that module to be utterly horrendous in what
it does and if we don't replicate all that it can do, so much the
better.  Consider a similar proposal as ALTER SYSTEM which simply
rewrites pg_hba.conf, ala how adminpack could.  Would we punt on the
hard questions around things like how to specify the order in which
entries in pg_hba.conf are maintained?  We could have an ALTER HBA today
which takes a line number to insert a string of text into, but is it a
good idea or design?  Certainly not.  My point here is simply that
focusing on replacing adminpack is not the basis for a feature or
design.

> #3 is a bad idea in my
> book - it would break some of my existing benchmarking scripts, which
> do initdb; cat >>$PGDATA/postgresql.conf <<EOM.  But even if it were a
> good idea, it isn't a necessary prerequisite for this patch.

Too often do we consider sloppy handling like this, which is great for a
dev or testing environment, to be appropriate for production systems. :(

	Thanks,

		Stephen

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Peter Geoghegan <pg@heroku.com> — 2013-08-30T01:18:57Z

On Thu, Aug 29, 2013 at 9:13 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> I think the goals of this patch should be to (1) let users of other
> clients manipulate the startup configuration just as easily as we can
> already do using pgAdmin and (2) remove some of the concurrency
> hazards that pgAdmin introduces, for example by using locking and
> atomic renames.

+1

-- 
Peter Geoghegan

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-30T01:26:48Z

* Robert Haas (robertmhaas@gmail.com) wrote:
> On Thu, Aug 29, 2013 at 8:07 PM, Stephen Frost <sfrost@snowman.net> wrote:
> > I'm not talking about malicious DBAs but rather a generally
> > knowledgable DBA who changed shared_buffers up too high and then leaves on
> > vacation, while the OPs guys need to do a database restart for whatever
> > reason and then discover it doesn't start.
> 
> /me looks at Stephen incredulously.
> 
> In the first place, modifying postgresql.conf and not immediately
> restarting the server to test your changes is probably the single most
> basic DBA error imaginable.  

You're not modifying postgresql.conf with ALTER SYSTEM, now are you?
Admins are going to realize the need to restart or at least reload the
service after updating such, but a DBA who has focused mainly on
normalization, query optimization, etc, isn't going to have the same
experience that a sysadmin has.

A DBA updating a GUC, something they are likely to do frequently
day-in and day-out, isn't necessairly going to consider that it's a
reboot-needing change.  Indeed, it's not unreasonable to consider that
we may, some day, actually be able to change or increase
shared_buffers on the fly (or error in the trying); I'd think your
work with the dynamic backends would actually make that likely to
happen sooner rather than later.

> I have a hard time viewing such a person
> as generally knowledgeable.  I hope the DBA in question doesn't have
> access to the switches, because he's probably the sort of guy who
> reassigns switch ports and doesn't type "wr m" afterwards.

I'd consider the DBAs who I work with as generally knowledgable, and
thankfully also cautious enough to talk to me before making changes, but
they have superuser access (they need to be able to do database
'releases' to our production environments and those scripts need to
change table ownership and do other things which can be annoying to
maintain the permissions for) and certainly understand changing GUCs
(mostly things like work_mem).

> In the second place, the same guy can do the same thing today.  He
> just has to use "vi".  In fact, I think this is a pretty common
> failure mode in poorly-controlled environments where too many people
> have access to the configuration files.  Now maybe you're going to
> tell me that the ops guys can't modify the configuration file because
> they only have SQL-level access, but then how are they managing to
> restart the database?  They need to be able to run pg_ctl *as the
> postgres user* to do that, and if they have shell access to that
> account, all bets are off.

You seem to be confused here between the DBA/data team and the OPs or
Infrastructure team.  I'm making a clear distinction between them and
feel quite comfortable with it because it's the environment that I live
in today.  My job, in fact, is exactly to straddle that line and work
with both.

In the above example, the DBA hasn't got root on the box and can't
simply go change or modify postgresql.conf, not even with "vi", and is
true for most of the DBAs on my team, even though they have superuser
access in PG.

> Sure, you can construct a scenario where this matters.  The ops guys
> have "sudo postgres pg_ctl" access but adminpack isn't installed and
> they have no other way to modify the configuration file.  But that's
> just bizarre.  And if that's really the environment you have, then you
> can install a loadable module that grabs ProcessUtility_hook and uses
> it to forbid ALTER SYSTEM on that machine.  Hell, we can ship such a
> thing in contrib.  Problem solved.  But it's surely too obscure a
> combination of circumstances to justify disabling this by default.

It's not the OPs guy that I'm worried about using ALTER SYSTEM- I don't
expect them to have any clue about it or care about it, except where it
can be used to modify things under /etc which they, rightfully, consider
their domain.

	Thanks,

		Stephen

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Cédric Villemain <cedric@2ndquadrant.com> — 2013-08-30T08:20:48Z

Le jeudi 29 août 2013 18:42:13 Stephen Frost a écrit :
> On Thursday, August 29, 2013, Andres Freund wrote:
> > If you don't want your installation to use it, tell you ops people not
> > to do so. They are superusers, they need to have the ability to follow
> > some rules you make up internally.
> 
> The OPs people are the ones that will be upset with this because the DBAs
> will be modifying configs which OPs rightfully claim as theirs. If they
> have a way to prevent it then perhaps it's not terrible but they'd also
> need to know to disable this new "feature". As for ALTER DATABASE- I would
> be happier with encouraging use of that (or providing an ALTER CLUSTER) for
> those thing it makes sense and works for and removing those from being in
> postgresql.conf. I still feel things like listen_addresses shouldn't be
> changed through this.

ALTER ROLE ALL may be good enougth to handle every GUC that we can also remove 
from postgresql.conf (I suppose all GUC needing only a reload, not a restart). 
It may needs some improvement to handle changing default for ALL and adding 
new role.

> > The energy wasted in a good part of this massive 550+ messages thread is
> > truly saddening. We all (c|sh)ould have spent that time making PG more
> > awesome instead.
> 
> Perhaps not understood by all, but keeping PG awesome involves more than
> adding every feature proposed- it also means saying no sometimes; to
> features, to new GUCs, even to micro-optimizations when they're overly
> complicated and offer only minimal or questionable improvements.

Agreed, the current feature and proposal does not include pg_reload, and it 
introduces a full machinery we absolutely don't need.

Grammar can be added later when the feature is stable.

So far, we can achieve the goal by using adminpack, by using a file_fdw or a 
config_fdw. IMHO it is the job of a FDW to be able to handle atomic write or 
anything like that.

I've commented one of the proposed patch adding some helpers to validate GUC 
change, I claimed this part was good enough to be added without ALTER SYSTEM 
(so a contrib can use it).

-- 
Cédric Villemain +33 (0)6 20 30 22 52
http://2ndQuadrant.fr/
PostgreSQL: Support 24x7 - Développement, Expertise et Formation

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Dimitri Fontaine <dimitri@2ndquadrant.fr> — 2013-08-30T10:43:14Z

Stephen Frost <sfrost@snowman.net> writes:
> The OPs people are the ones that will be upset with this because the DBAs
> will be modifying configs which OPs rightfully claim as theirs.

If that's the problem you want to solve, there's no technical solution
that will put you at ease. That's a people and trust problem.

I don't think typical (or less typical) organisation should drive our
technical choices too much, and I'm pretty confident they didn't in the
past: pg_hba.conf is a file not because it's meant for this or that team
but because it makes sense technically to manage the settings to allow
using some resources *outside* of said resources.

We currently have no way that I know of to disable ALTER ROLE SET and
ALTER DATABASE SET effects, why do we need to provide that feature for
ALTER SYSTEM SET so much?

Regards,
-- 
Dimitri Fontaine
http://2ndQuadrant.fr     PostgreSQL : Expertise, Formation et Support

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Andres Freund <andres@2ndquadrant.com> — 2013-08-30T11:59:06Z

Hi,

On 2013-08-30 10:20:48 +0200, Cédric Villemain wrote:
> > > The energy wasted in a good part of this massive 550+ messages thread is
> > > truly saddening. We all (c|sh)ould have spent that time making PG more
> > > awesome instead.
> > 
> > Perhaps not understood by all, but keeping PG awesome involves more than
> > adding every feature proposed- it also means saying no sometimes; to
> > features, to new GUCs, even to micro-optimizations when they're overly
> > complicated and offer only minimal or questionable improvements.
> 
> Agreed, the current feature and proposal does not include pg_reload, and it
> introduces a full machinery we absolutely don't need.

The complexity in the last version of the patch I looked at wasn't in
the added grammar or pg_reload() (well, it didn't have that). It was the
logic to read (from memory)/write the config file and validate the
GUCs. That's needed even if you put it into some module. And it requires
support from guc.c/guc-file.l

> Grammar can be added later when the feature is stable.

Could you explain the advantages of this? It will require users to get
used to different interfaces and we will end up maintaining both just
about forever. And I don't see the grammar being that contentious?

Greetings,

Andres Freund

-- 
 Andres Freund	                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-30T12:48:21Z

* Dimitri Fontaine (dimitri@2ndQuadrant.fr) wrote:
> Stephen Frost <sfrost@snowman.net> writes:
> > The OPs people are the ones that will be upset with this because the DBAs
> > will be modifying configs which OPs rightfully claim as theirs.
> 
> If that's the problem you want to solve, there's no technical solution
> that will put you at ease. That's a people and trust problem.

I really just don't buy that- I've already put forward suggestions for
how to deal with it, but no one here seems to understand the
distinction.  Modifying listen_addresses through ALTER SYSTEM is akin to
ISC/bind allowing changes to its listen_addresses equivilant through
dynamic DNS updates.  Would it be possible to implement?  Sure.  Does it
make any sense?  Certainly not.

> I don't think typical (or less typical) organisation should drive our
> technical choices too much, and I'm pretty confident they didn't in the
> past: pg_hba.conf is a file not because it's meant for this or that team
> but because it makes sense technically to manage the settings to allow
> using some resources *outside* of said resources.

I'm all for moving pg_hba.conf into the database properly, actually.  We
already handle permissions and user access in the DB and that's one of
the DBA's responsibilities.  The same goes for pg_ident.conf.

> We currently have no way that I know of to disable ALTER ROLE SET and
> ALTER DATABASE SET effects, why do we need to provide that feature for
> ALTER SYSTEM SET so much?

Because we've got crap mixed into postgresql.conf which are bootstrap
configs needed to get the system started.  Those things, in my view
anyway, fall much more into the category of "resources which should be
managed outside the database" than pg_hba.conf.

	Thanks,

		Stephen

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-30T12:53:25Z

* Cédric Villemain (cedric@2ndquadrant.com) wrote:
> ALTER ROLE ALL may be good enougth to handle every GUC that we can also remove 
> from postgresql.conf (I suppose all GUC needing only a reload, not a restart). 
> It may needs some improvement to handle changing default for ALL and adding 
> new role.

Yes, one of the issues with the existing system is that you can't
specify a default to be applied to new roles.  Also, there are
parameters which are not per-role yet which it probably makes sense to
be in the database and we'd need a way to deal with that.  Although, at
the same time, considering it role based does make for a nice
distinction.  Unfortunately, those clammoring for this will argue that
it wouldn't replace adminpack and that they couldn't use it to modify
their /etc/network/interfaces file, which is the obvious next step to
all of this.

> Grammar can be added later when the feature is stable.

I tend to agree w/ Andres on this point- the grammar isn't really the
contentious part.  I think I see where you were going with this in that
excluding the grammar makes it able to live as a module, but that's a
different consideration.

> I've commented one of the proposed patch adding some helpers to validate GUC 
> change, I claimed this part was good enough to be added without ALTER SYSTEM 
> (so a contrib can use it).

Perhaps..

	Thanks,

		Stephen

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-30T12:55:17Z

* Andres Freund (andres@2ndquadrant.com) wrote:
> On 2013-08-30 10:20:48 +0200, Cédric Villemain wrote:
> > Grammar can be added later when the feature is stable.
> 
> Could you explain the advantages of this? It will require users to get
> used to different interfaces and we will end up maintaining both just
> about forever. And I don't see the grammar being that contentious?

The "different interfaces" issue already exists, to some extent, as
adminpack does exist already and this is clearly a different interface
from that.  That said, as I mentioned just now elsewhere, I agree that
the grammar itself (being rather simple anyway...) isn't the contentious
part of this.

	Thanks,

		Stephen

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Andres Freund <andres@2ndquadrant.com> — 2013-08-30T13:02:54Z

On 2013-08-30 08:48:21 -0400, Stephen Frost wrote:
> * Dimitri Fontaine (dimitri@2ndQuadrant.fr) wrote:
> > Stephen Frost <sfrost@snowman.net> writes:
> > > The OPs people are the ones that will be upset with this because the DBAs
> > > will be modifying configs which OPs rightfully claim as theirs.
> > 
> > If that's the problem you want to solve, there's no technical solution
> > that will put you at ease. That's a people and trust problem.
> 
> I really just don't buy that- I've already put forward suggestions for
> how to deal with it, but no one here seems to understand the
> distinction.  Modifying listen_addresses through ALTER SYSTEM is akin to
> ISC/bind allowing changes to its listen_addresses equivilant through
> dynamic DNS updates.  Would it be possible to implement?  Sure.  Does it
> make any sense?  Certainly not.

I very much want to change stuff like wal_level, listen_addresses and
shared_buffers via ALTER SYSTEM. Configuration variables like that
(PGC_POSTMASTER stuff mostly) are the prime reason why you actually need
to change postgresql.conf instead of changing per user/database
settings.
And you don't even need to do anything special to implement it. Because
it's already there.

> > We currently have no way that I know of to disable ALTER ROLE SET and
> > ALTER DATABASE SET effects, why do we need to provide that feature for
> > ALTER SYSTEM SET so much?
> 
> Because we've got crap mixed into postgresql.conf which are bootstrap
> configs needed to get the system started.  Those things, in my view
> anyway, fall much more into the category of "resources which should be
> managed outside the database" than pg_hba.conf.

I think the problem with your position in this thread is that you want
to overhaul the way our configuration works in a pretty radical
way. Which is fair enough, there certainly are deficiencies. But it's
not the topic of this thread.

Greetings,

Andres Freund

-- 
 Andres Freund	                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Andres Freund <andres@2ndquadrant.com> — 2013-08-30T13:06:27Z

On 2013-08-29 21:26:48 -0400, Stephen Frost wrote:
> > Sure, you can construct a scenario where this matters.  The ops guys
> > have "sudo postgres pg_ctl" access but adminpack isn't installed and
> > they have no other way to modify the configuration file.  But that's
> > just bizarre.  And if that's really the environment you have, then you
> > can install a loadable module that grabs ProcessUtility_hook and uses
> > it to forbid ALTER SYSTEM on that machine.  Hell, we can ship such a
> > thing in contrib.  Problem solved.  But it's surely too obscure a
> > combination of circumstances to justify disabling this by default.
> 
> It's not the OPs guy that I'm worried about using ALTER SYSTEM- I don't
> expect them to have any clue about it or care about it, except where it
> can be used to modify things under /etc which they, rightfully, consider
> their domain.

I think for the scenarios you describe it makes far, far much more sense
to add the ability to easily monitor for two things:
* on-disk configuration isn't the same as the currently loaded (not
  trivially possible yet)
* Configuration variables only come from locations that are approved for
  in your scenario (Already possible, we might want to make it even easier)

Greetings,

Andres Freund

-- 
 Andres Freund	                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Robert Haas <robertmhaas@gmail.com> — 2013-08-30T13:18:02Z

On Thu, Aug 29, 2013 at 9:12 PM, Stephen Frost <sfrost@snowman.net> wrote:
> You will not find me argueing to allow that in normal operation, or most
> direct-catalog hacking.  I'd be much happier if we had all the ALTER,
> etc, options necessary to prevent any need to ever touch the catalogs.
> Similairly, it'd be nice to have more permission options which reduce
> the need for anyone to be superuser.

Sure, I agree.  But you can't burden this patch with the job of
conforming to what you and I think the project policy ought to be, but
isn't.  Right now, as things stand today, the superuser is one step
short of God Almighty.  The only way in which we even attempt to
restrict the superuser is to try to keep her from hijacking the
postgres login account.  But we don't even try to do that very
thoroughly, as has recently been pointed out on other threads; there
are several plausible attack vectors for her to do just that.  This
patch fits right into that existing philosophy.  If we take this
patch, and I think we should, and later change our policy, then the
permissions around this may require adjustment to fit the new policy.
Fine!  But that policy change is properly a separate discussion.

>> Now, you can argue that people are more likely to render the database
>> nonfunctional by changing GUC settings than they are to do it by
>> corrupting the system catalogs, but I'm not sure I believe it.  We
>> can, after all, validate that any setting a user supplies is a valid
>> value for that setting before writing it out to the configuration
>> file.  It might still make the system fail to start if - for example -
>> it causes too many semaphores to be reserved, or something like that.
>> But why should we think that such mistakes will be common?  If
>> anything, it sounds less error-prone to me than hand-editing the
>> configuration file, where typing something like on_exit_error=maybe
>> will make the server fail to start.  We can eliminate that whole
>> category of error, at least for people who choose to use the new
>> tools.
>
> That category of error is much more likely to get caught by the sysadmin
> doing the restart after the update..

If you edit postgresql.conf manually today, you will have no warning
of ANY sort of error you might make.  With a feature like this, we can
catch a large subset of things you might do wrong *before we even
modify the file*.  Yes, there will be some that slip through, but an
imperfect mechanism for catching mistakes is better than no mechanism
at all.

> ALTER SYSTEM doesn't provide any
> way to restart the DB to make sure it worked.
> That, in turn, encourages
> modification of the config parameters w/o a restart, which is a pretty
> bad idea, imv.

"vi" doesn't provide any way to restart the DB to make sure it worked,
either, and never will.  However, ALTER SYSTEM could be extended in
exactly that way: we could have ALTER SYSTEM RESTART.  Of course some
people are likely to argue that's a bad idea, so we'd have to have
that discussion and think carefully about the issues, but there's
nothing intrinsic that restricts us from giving ALTER SYSTEM any
arbitrary set of capabilities we might want it to have.  Even if we
don't ever do that, we're no worse off than today.

Well, with one exception.  If we make it easier to modify the
configuration file, then people are more likely to do it, and thus
more likely to do it wrong.  But you could use that argument against
any change that improves ease-of-use.  And ease-of-use is a feature,
not a bug.

>> If you're using the term "dangerous" to refer to a security exposure,
>> I concede there is some incremental exposure from allowing this by
>> default.  But it's not a terribly large additional exposure.  It's
>> already the case that if adminpack is available the super-user can do
>> whatever he or she wants, because she or he can write to arbitrary
>> files inside the data directory.
>
> This is only true if -contrib is installed, which a *lot* of admins
> refuse to do, specifically because of such insane modules as this.
> Therefore, I would argue that it's not nearly as small a change wrt
> exposure as you believe.  Strictly speaking, I don't believe it's a huge
> security hole or something, but I do think it's going to surprise admins
> who aren't following the lists.

If we take the patch, it will be in the release notes, just like any
other feature.  I suspect there will be more people *pleasantly*
surprised than anything else.  If there's anything I've learned about
ease-of-use as a 10+-year veteran of PostgreSQL, it's that being able
to do everything via a database connection is really, really, really
convenient.  That's why foreign data wrappers are awesome.  Instead of
installing a foreign data wrapper for Oracle and pointing it at an
Oracle server and then sucking data down over the link to store or
process or whatever, you put all that logic in a client which knows
how to talk to both PostgreSQL and Oracle.  It turns out, though, that
having that capability *in the server* is really valuable to people.
It doesn't let you do anything you couldn't do otherwise, but it's
extremely convenient.  It's also why we keep hearing requests to put a
scheduler in core.  It's not that you *can't* schedule tasks using
cron or Scheduled Tasks on Windows; it's that it's much more
convenient to be able to do it all in one place, namely PostgreSQL.
It is 100% possible for us to tell all of our users, sorry, you've got
to keep using vi to change configuration settings, because we don't
trust you to do that via SQL.  But I think it's the WRONG decision.
Users like being trusted.  And it's not that hard for the DBA to
disable this or any other feature if he or she feels a real need to do
so.

> This wouldn't be the first time we've said "do it externally and when
> we figure out a better answer then we'll put it into core."

No, but I don't think there's anything terribly wrong with *this*
answer.  It's the product of a lot of people hashing out a lot of
issues, and I think it's pretty good.  Pushing things out externally
is appropriate when the best design is unclear and especially when
little or no core support is required, but I think we have a pretty
broad consensus on the general contours of how this should work, and
some core support is needed.  Someone could certainly do something
better than adminpack without core integration, but what's being
proposed here is better still.

>> 1. disabling the feature by default, and providing no way for it to be
>> turned on remotely, and
>
> Yes, that's quite intentional because it would make someone knowledgable
> about the *overall system* be aware of this rather than having it sprung
> on them when they discover some change has made the database unable to
> start.

We add new features in every release.  I don't see that this one is
dangerous enough that it needs to be hedged about with disclaimers we
don't attach to any of our other features.  My favorite example is the
release where we changed the default behavior of TRUNCATE so that it
cascades to child tables instead of affecting only the parent.  Talk
about springing changes on people that could have catastrophic
consequences.  People have to read the release notes before upgrading
to new major versions, or they will get surprised.  That's why we have
release notes, and that's why we have stable minor versions.

I would also just like to note in passing that I believe the shared
memory changes in 9.3 eliminate the #1 reason for "my database won't
start after a config change", which IME has always been that the
change pushed up the shared memory allocation just beyond what the OS
would allow.  That hazard affected a large number of different
configuration parameters, and it doesn't exist any more.  You can
still bump max_connections or max_autovacuum_workers or the new
max_worker_processes high enough to eat up all the system's
semaphores, and I'm sure someone has killed their system that way
before, but I've personally not run into it.  You might also increase
the size of shared memory so much that you actually overrun what's
available on the box, rather than just overrunning the limit on System
V shared memory, but that requires a much larger adjustment to the
parameter values and is therefore likely to be the result of abject
stupidity rather than simple bad luck.

> While I appreciate that the original thrust behind this may be trying to
> replace adminpack, I find that module to be utterly horrendous in what
> it does and if we don't replicate all that it can do, so much the
> better.  Consider a similar proposal as ALTER SYSTEM which simply
> rewrites pg_hba.conf, ala how adminpack could.  Would we punt on the
> hard questions around things like how to specify the order in which
> entries in pg_hba.conf are maintained?  We could have an ALTER HBA today
> which takes a line number to insert a string of text into, but is it a
> good idea or design?  Certainly not.  My point here is simply that
> focusing on replacing adminpack is not the basis for a feature or
> design.

I partially agree with that, but the point of mentioning adminpack is
that the horse is already halfway out of the barn.  I think this
feature is pretty good as designed and not half-baked at all, but even
if you feel otherwise it's surely a heck of a lot better than what
exists today.

>> #3 is a bad idea in my
>> book - it would break some of my existing benchmarking scripts, which
>> do initdb; cat >>$PGDATA/postgresql.conf <<EOM.  But even if it were a
>> good idea, it isn't a necessary prerequisite for this patch.
>
> Too often do we consider sloppy handling like this, which is great for a
> dev or testing environment, to be appropriate for production systems. :(

Well, IMHO, the solution to that is to stop relying on hand-edited
files.  I've never really liked the fact that postgresql.conf is a
text file rather than somehow part of the database guts, requiring a
special tool (that can do the appropriate validation) to make changes.
 But that ship has already sailed and doesn't seem likely to reverse
course any time soon.

...Robert

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-30T13:19:42Z

* Andres Freund (andres@2ndquadrant.com) wrote:
> On 2013-08-30 08:48:21 -0400, Stephen Frost wrote:
> > I really just don't buy that- I've already put forward suggestions for
> > how to deal with it, but no one here seems to understand the
> > distinction.  Modifying listen_addresses through ALTER SYSTEM is akin to
> > ISC/bind allowing changes to its listen_addresses equivilant through
> > dynamic DNS updates.  Would it be possible to implement?  Sure.  Does it
> > make any sense?  Certainly not.
> 
> I very much want to change stuff like wal_level, listen_addresses and
> shared_buffers via ALTER SYSTEM. Configuration variables like that
> (PGC_POSTMASTER stuff mostly) are the prime reason why you actually need
> to change postgresql.conf instead of changing per user/database
> settings.

wal_level and shared_buffers I can buy, but listen_addresses?  The most
typical change there is going from localhost -> '*', but you've got to
be on the box to do that.  Anything else and you're going to need to be
adding interfaces to the box anyway and hacking around in
/etc/network/interfaces or what-have-you.

> > Because we've got crap mixed into postgresql.conf which are bootstrap
> > configs needed to get the system started.  Those things, in my view
> > anyway, fall much more into the category of "resources which should be
> > managed outside the database" than pg_hba.conf.
> 
> I think the problem with your position in this thread is that you want
> to overhaul the way our configuration works in a pretty radical
> way. Which is fair enough, there certainly are deficiencies. But it's
> not the topic of this thread.

You and Robert both seem to be of the opinion that this hack which
brings postgresql.conf into the database via ALTER SYSTEM is a-ok
because it's moving us "forward" in someone's mind, but it really is
developing a system configuration management system which *looks* like a
first-class citizen when it actually falls woefully short of that.

There is absolutely no question in my mind that this will be a huge
support pain, from the first "ALTER SYSTEM SET shared_buffers = blah;
SHOW shared_buffers;" to the "why can't my database start?!?  it's
complaining it can't allocate memory but I keep changing postgresql.conf
and nothing works!"  I'm simply not convinced that this is moving us
forward nor that we will end up with more benefit than pain from it.

	Thanks,

		Stephen

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-30T13:26:32Z

* Andres Freund (andres@2ndquadrant.com) wrote:
> On 2013-08-29 21:26:48 -0400, Stephen Frost wrote:
> > It's not the OPs guy that I'm worried about using ALTER SYSTEM- I don't
> > expect them to have any clue about it or care about it, except where it
> > can be used to modify things under /etc which they, rightfully, consider
> > their domain.
> 
> I think for the scenarios you describe it makes far, far much more sense
> to add the ability to easily monitor for two things:
> * on-disk configuration isn't the same as the currently loaded (not
>   trivially possible yet)

That would certainly help.  I don't know that it needs to be technically
'trivial', but at least having check_postgres.pl or something which can
alert on it would be an improvement.  Clearly, that would be valuable
today (assuming it doesn't already exist somewhere..  it might).

> * Configuration variables only come from locations that are approved for
>   in your scenario (Already possible, we might want to make it even easier)

That an interesting notion; do you have something specific in mind..?
The easiest, imv anyway, would be that options set in postgresql.conf
can't be overridden, but that gets us into the bootstrap problem that
people seem to be concerned about.  It would also be a change to how
postgresql.conf is parsed today which some people would be annoyed by.
Having some configuration option which says what can be modified by
alter system doesn't strike me as a terribly good solution either.

	Thanks,

		Stephen

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Andres Freund <andres@2ndquadrant.com> — 2013-08-30T13:28:12Z

On 2013-08-30 09:19:42 -0400, Stephen Frost wrote:
> * Andres Freund (andres@2ndquadrant.com) wrote:
> > On 2013-08-30 08:48:21 -0400, Stephen Frost wrote:
> > > I really just don't buy that- I've already put forward suggestions for
> > > how to deal with it, but no one here seems to understand the
> > > distinction.  Modifying listen_addresses through ALTER SYSTEM is akin to
> > > ISC/bind allowing changes to its listen_addresses equivilant through
> > > dynamic DNS updates.  Would it be possible to implement?  Sure.  Does it
> > > make any sense?  Certainly not.
> > 
> > I very much want to change stuff like wal_level, listen_addresses and
> > shared_buffers via ALTER SYSTEM. Configuration variables like that
> > (PGC_POSTMASTER stuff mostly) are the prime reason why you actually need
> > to change postgresql.conf instead of changing per user/database
> > settings.
> 
> wal_level and shared_buffers I can buy, but listen_addresses?  The most
> typical change there is going from localhost -> '*', but you've got to
> be on the box to do that.  Anything else and you're going to need to be
> adding interfaces to the box anyway and hacking around in
> /etc/network/interfaces or what-have-you.

Even if it requires to be on the box locally, I only need libpq for
it. And it's not infrequent to allow additional, already configured
interfaces. And even if not, what's the point of prohibiting
listen_interfaces specifically? When the other very interesting
variables have the same dangers?
Doing this on a variable-by-variable basis will a) be a ridiculous
amount of effort, b) confuse users which may not share our judgement of
individual variables.

> > > Because we've got crap mixed into postgresql.conf which are bootstrap
> > > configs needed to get the system started.  Those things, in my view
> > > anyway, fall much more into the category of "resources which should be
> > > managed outside the database" than pg_hba.conf.
> > 
> > I think the problem with your position in this thread is that you want
> > to overhaul the way our configuration works in a pretty radical
> > way. Which is fair enough, there certainly are deficiencies. But it's
> > not the topic of this thread.
> 
> You and Robert both seem to be of the opinion that this hack which
> brings postgresql.conf into the database via ALTER SYSTEM is a-ok
> because it's moving us "forward" in someone's mind, but it really is
> developing a system configuration management system which *looks* like a
> first-class citizen when it actually falls woefully short of that.

It's what plenty of people want and it doesn't hurt people who do not
want it. Yes. I think that's a step forward.

> There is absolutely no question in my mind that this will be a huge
> support pain, from the first "ALTER SYSTEM SET shared_buffers = blah;
> SHOW shared_buffers;" to the "why can't my database start?!?  it's
> complaining it can't allocate memory but I keep changing postgresql.conf
> and nothing works!"  I'm simply not convinced that this is moving us
> forward nor that we will end up with more benefit than pain from it.

That will not show the changed shared_buffers. And it (afair) will throw
a WARNING that shared_buffers couldn't be adjusted at this point.

Greetings,

Andres Freund

-- 
 Andres Freund	                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Robert Haas <robertmhaas@gmail.com> — 2013-08-30T13:33:51Z

On Thu, Aug 29, 2013 at 9:26 PM, Stephen Frost <sfrost@snowman.net> wrote:
>> In the first place, modifying postgresql.conf and not immediately
>> restarting the server to test your changes is probably the single most
>> basic DBA error imaginable.
>
> You're not modifying postgresql.conf with ALTER SYSTEM, now are you?
> Admins are going to realize the need to restart or at least reload the
> service after updating such, but a DBA who has focused mainly on
> normalization, query optimization, etc, isn't going to have the same
> experience that a sysadmin has.

I refuse to reject any feature on the basis of the possibility that
people might use it without reading the documentation.  Nearly
anything anyone will ever propose is so idiot-proof that we can't
conceive of a scenario in which someone shoots their foot off with it.
 The question is whether the usefulness of the feature exceeds, by a
sufficient amount, the harm it's likely to do, and I think the answer
is clearly yes.

> A DBA updating a GUC, something they are likely to do frequently
> day-in and day-out, isn't necessairly going to consider that it's a
> reboot-needing change.  Indeed, it's not unreasonable to consider that
> we may, some day, actually be able to change or increase
> shared_buffers on the fly (or error in the trying); I'd think your
> work with the dynamic backends would actually make that likely to
> happen sooner rather than later.

I wouldn't hold my breath.  We have way too many separate,
variable-length data structures in shared memory.  Increasing
shared_buffers means making the lwlock array bigger, making the buffer
header array bigger, allocating more space for the buffer mapping
hash, etc.  I doubt we'd accept a design that involves each of those
data structures being a separate shared memory segment, but if they're
all in the same segment one after another, then you can't easily
extend them on the fly.

There are also a lot of problems around unmapping-and-remapping a
segment.  If the remap fails, it's going to be at least a FATAL, but
if you had any shared memory state in the unmapped segment (like a
buffer pin) that has to be unwound, you have to PANIC; there's no
other way to fix it.  My initial design for dynamic shared memory
allowed for an unbounded number of dynamic shared memory segments by
growing the control segment on the fly.  However, this introduced
PANIC hazards in corner cases that I couldn't get rid of, so now
there's a fairly generous but fixed-at-startup-time limit on how many
segments you can have.  In practice I don't think this matters much,
but it was a sobering reminder that the main shared memory segment,
with all of its inflexibility, has important reliability properties
that are hard to replicate in more dynamic scenarios.

> It's not the OPs guy that I'm worried about using ALTER SYSTEM- I don't
> expect them to have any clue about it or care about it, except where it
> can be used to modify things under /etc which they, rightfully, consider
> their domain.

Under the currently-proposed design, it can't be used to do any such
thing.  It can only be used to modify some auto.conf file which lives
in $PGDATA.  It's therefore no different from the ops perspective than
ALTER DATABASE or ALTER USER - except that it allows all settings to
be changed rather than only a subset.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Andres Freund <andres@2ndquadrant.com> — 2013-08-30T13:39:26Z

On 2013-08-30 09:26:32 -0400, Stephen Frost wrote:
> * Andres Freund (andres@2ndquadrant.com) wrote:
> > On 2013-08-29 21:26:48 -0400, Stephen Frost wrote:
> > > It's not the OPs guy that I'm worried about using ALTER SYSTEM- I don't
> > > expect them to have any clue about it or care about it, except where it
> > > can be used to modify things under /etc which they, rightfully, consider
> > > their domain.
> > 
> > I think for the scenarios you describe it makes far, far much more sense
> > to add the ability to easily monitor for two things:
> > * on-disk configuration isn't the same as the currently loaded (not
> >   trivially possible yet)
> 
> That would certainly help.  I don't know that it needs to be technically
> 'trivial', but at least having check_postgres.pl or something which can
> alert on it would be an improvement.  Clearly, that would be valuable
> today (assuming it doesn't already exist somewhere..  it might).

Technically trivial in the sense that it should be queryable from SQL
without having to write code in an untrusted PL ;).

I guess storing the file modification date along the file/location a GUC
is originating from would be good enough. Then you could write a query
using pg_stat_file() to make sure they are up2date.

> > * Configuration variables only come from locations that are approved for
> >   in your scenario (Already possible, we might want to make it even easier)
> 
> That an interesting notion; do you have something specific in mind..?
> The easiest, imv anyway, would be that options set in postgresql.conf
> can't be overridden, but that gets us into the bootstrap problem that
> people seem to be concerned about.  It would also be a change to how
> postgresql.conf is parsed today which some people would be annoyed by.
> Having some configuration option which says what can be modified by
> alter system doesn't strike me as a terribly good solution either.

I think changing the precedence of options in postgresql.conf has about
zero chance.

Currently you can do something like:
SELECT name FROM pg_settings WHERE sourcefile IS NOT NULL AND sourcefile <> '/etc/postgresql/9.2/main/postgresql.conf';

To be easily usable we should have two new columns in pg_settings:
* reset_sourcefile
* reset_sourceline
And an easy way to get the absolute path of
* postgresql.conf
* auto.conf or whatever we name it.

That would make it possible to easily write a query that works across
intallation that warns about any values stored in auto.conf, even if
they are overwritten by a per-user config or similar.

Greetings,

Andres Freund

-- 
 Andres Freund	                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Robert Haas <robertmhaas@gmail.com> — 2013-08-30T13:40:56Z

On Fri, Aug 30, 2013 at 8:53 AM, Stephen Frost <sfrost@snowman.net> wrote:
> Yes, one of the issues with the existing system is that you can't
> specify a default to be applied to new roles.  Also, there are
> parameters which are not per-role yet which it probably makes sense to
> be in the database and we'd need a way to deal with that.  Although, at
> the same time, considering it role based does make for a nice
> distinction.  Unfortunately, those clammoring for this will argue that
> it wouldn't replace adminpack and that they couldn't use it to modify
> their /etc/network/interfaces file, which is the obvious next step to
> all of this.

This is a straw-man.  adminpack doesn't allow reading or writing files
outside of the configured data and log directories, as a security
precaution.  But suppose it did.  If your permissions on
/etc/network/interfaces allow the postgres user to modify it, then you
pretty much deserve exactly what you get.  Likening this to the
feature being proposed is silly.  What is being asked for here is the
ability to easily modify system-wide settings from the SQL prompt,
just as today you can modify settings per-user or per-database.
That's not the same thing as rewriting the entire system
configuration.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-30T13:43:01Z

* Andres Freund (andres@2ndquadrant.com) wrote:
> On 2013-08-30 09:19:42 -0400, Stephen Frost wrote:
> > wal_level and shared_buffers I can buy, but listen_addresses?  The most
> > typical change there is going from localhost -> '*', but you've got to
> > be on the box to do that.  Anything else and you're going to need to be
> > adding interfaces to the box anyway and hacking around in
> > /etc/network/interfaces or what-have-you.
> 
> Even if it requires to be on the box locally, I only need libpq for
> it. And it's not infrequent to allow additional, already configured
> interfaces. And even if not, what's the point of prohibiting
> listen_interfaces specifically? When the other very interesting
> variables have the same dangers?

I'd like to see those dangers removed from the other very interesting
variables.  We're making progress towards that, for example with
shared_buffers.  I've commented on that already up-thread.  Hell, you
could even make things such that PG would start w/ a misconfigured
listen_addresses- but if we don't like that then I would argue that it
shouldn't be included here.  I'm not entirely sure how wal_level has the
same danger as the others..

> Doing this on a variable-by-variable basis will a) be a ridiculous
> amount of effort, b) confuse users which may not share our judgement of
> individual variables.

I don't think the effort involved is nearly as much as you claim and we
already have the issue that users don't like our choices around what can
be reconfigured on the fly and what can't (perhaps they understand that
there are technical challenges to some of them, but that doesn't make
them agree with them).

> > You and Robert both seem to be of the opinion that this hack which
> > brings postgresql.conf into the database via ALTER SYSTEM is a-ok
> > because it's moving us "forward" in someone's mind, but it really is
> > developing a system configuration management system which *looks* like a
> > first-class citizen when it actually falls woefully short of that.
> 
> It's what plenty of people want and it doesn't hurt people who do not
> want it. Yes. I think that's a step forward.

It will be quite interesting to see if people decide they really wanted
this once they actually *get* it.

> > There is absolutely no question in my mind that this will be a huge
> > support pain, from the first "ALTER SYSTEM SET shared_buffers = blah;
> > SHOW shared_buffers;" to the "why can't my database start?!?  it's
> > complaining it can't allocate memory but I keep changing postgresql.conf
> > and nothing works!"  I'm simply not convinced that this is moving us
> > forward nor that we will end up with more benefit than pain from it.
> 
> That will not show the changed shared_buffers. And it (afair) will throw
> a WARNING that shared_buffers couldn't be adjusted at this point.

Not showing the change is what I was getting at.  As has been said
elsewhere, throwing a warning on every interesting invokation of a
command can speak to it not being exactly 'ready'.

	Thanks,

		Stephen

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Andres Freund <andres@2ndquadrant.com> — 2013-08-30T13:47:45Z

On 2013-08-30 09:43:01 -0400, Stephen Frost wrote:
> * Andres Freund (andres@2ndquadrant.com) wrote:
> > On 2013-08-30 09:19:42 -0400, Stephen Frost wrote:
> > > wal_level and shared_buffers I can buy, but listen_addresses?  The most
> > > typical change there is going from localhost -> '*', but you've got to
> > > be on the box to do that.  Anything else and you're going to need to be
> > > adding interfaces to the box anyway and hacking around in
> > > /etc/network/interfaces or what-have-you.
> > 
> > Even if it requires to be on the box locally, I only need libpq for
> > it. And it's not infrequent to allow additional, already configured
> > interfaces. And even if not, what's the point of prohibiting
> > listen_interfaces specifically? When the other very interesting
> > variables have the same dangers?
> 
> I'd like to see those dangers removed from the other very interesting
> variables.  We're making progress towards that, for example with
> shared_buffers.  I've commented on that already up-thread.  Hell, you
> could even make things such that PG would start w/ a misconfigured
> listen_addresses- but if we don't like that then I would argue that it
> shouldn't be included here.

Yes, we're making progress. But that's an independent thing, partially
constrained by implementation complexity, partially constrained by cross
platform worries.
You're free to work on reducing the dangers around other variables, but
I don't understand in the very least why that should stop this feature.

> I'm not entirely sure how wal_level has the same danger as the
> others..

Try setting max_wal_senders = 10 and wal_level = minimal.

> > > There is absolutely no question in my mind that this will be a huge
> > > support pain, from the first "ALTER SYSTEM SET shared_buffers = blah;
> > > SHOW shared_buffers;" to the "why can't my database start?!?  it's
> > > complaining it can't allocate memory but I keep changing postgresql.conf
> > > and nothing works!"  I'm simply not convinced that this is moving us
> > > forward nor that we will end up with more benefit than pain from it.
> > 
> > That will not show the changed shared_buffers. And it (afair) will throw
> > a WARNING that shared_buffers couldn't be adjusted at this point.
> 
> Not showing the change is what I was getting at.  As has been said
> elsewhere, throwing a warning on every interesting invokation of a
> command can speak to it not being exactly 'ready'.

Then pg isn't ready already, because it has done that for a long
time. Set some PGC_POSTMASTER variable and reload the configuration.

Guiding a user that to perform required further action doesn't imply
nonreadyness in the least.

Greetings,

Andres Freund

-- 
 Andres Freund	                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Robert Haas <robertmhaas@gmail.com> — 2013-08-30T13:48:30Z

On Fri, Aug 30, 2013 at 9:19 AM, Stephen Frost <sfrost@snowman.net> wrote:
> You and Robert both seem to be of the opinion that this hack which
> brings postgresql.conf into the database via ALTER SYSTEM is a-ok
> because it's moving us "forward" in someone's mind, but it really is
> developing a system configuration management system which *looks* like a
> first-class citizen when it actually falls woefully short of that.

It's not a system configuration management system, and it doesn't
pretend to be.  It's an analogue of ALTER USER and ALTER DATABASE that
papers over their shortcomings, and a safer alternative to adminpack's
kludgy way of enabling the same functionality.

> There is absolutely no question in my mind that this will be a huge
> support pain, from the first "ALTER SYSTEM SET shared_buffers = blah;
> SHOW shared_buffers;"

They'd have the same problem with ALTER USER SET work_mem = blah.
You're acting like ALTER commands that don't take immediate effect are
something brand-new, but we've had them for years.

>  to the "why can't my database start?!?  it's
> complaining it can't allocate memory but I keep changing postgresql.conf
> and nothing works!"

As of 9.3, that failure mode doesn't really happen any more, unless
you maybe set shared_buffers equal to 100% of system memory.

> I'm simply not convinced that this is moving us
> forward nor that we will end up with more benefit than pain from it.

Fair enough, but I'm not convinced that we'll derive any pain at all
from it.  The existing similar features haven't been a notable source
of complaints AFAIK.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-30T13:49:12Z

* Robert Haas (robertmhaas@gmail.com) wrote:
> there's a fairly generous but fixed-at-startup-time limit on how many
> segments you can have.  In practice I don't think this matters much,
> but it was a sobering reminder that the main shared memory segment,
> with all of its inflexibility, has important reliability properties
> that are hard to replicate in more dynamic scenarios.

Why wouldn't it be possible to have the same arrangment for
shared_buffers, where you have more entires than you 'need' at startup
but which then allows you to add more shared segments later?  I do see
that we would need an additional bit of indirection to handle that,
which might be too much overhead, but the concept seems possible.  Isn't
that more-or-less how the kernel handles dynamic memory..?

> Under the currently-proposed design, it can't be used to do any such
> thing.  It can only be used to modify some auto.conf file which lives
> in $PGDATA.  It's therefore no different from the ops perspective than
> ALTER DATABASE or ALTER USER - except that it allows all settings to
> be changed rather than only a subset.

Claiming that modifying a file *included from a file in /etc* doesn't
modify things under /etc is disingenuous, imv.

	Thanks,

		Stephen

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-30T13:53:13Z

* Andres Freund (andres@2ndquadrant.com) wrote:
> Technically trivial in the sense that it should be queryable from SQL
> without having to write code in an untrusted PL ;).

hah.

> I guess storing the file modification date along the file/location a GUC
> is originating from would be good enough. Then you could write a query
> using pg_stat_file() to make sure they are up2date.

Wouldn't you want to actually look at the GUC and see if the value is
different also?  Just knowing that postgresql.conf changed doesn't mean
you want every value to say it's different...  It's entirely possible
that the file was rewritten or touch'd but the configuration is
identical.

> > > * Configuration variables only come from locations that are approved for
> > >   in your scenario (Already possible, we might want to make it even easier)
> > 
> > That an interesting notion; do you have something specific in mind..?
> > The easiest, imv anyway, would be that options set in postgresql.conf
> > can't be overridden, but that gets us into the bootstrap problem that
> > people seem to be concerned about.  It would also be a change to how
> > postgresql.conf is parsed today which some people would be annoyed by.
> > Having some configuration option which says what can be modified by
> > alter system doesn't strike me as a terribly good solution either.
> 
> I think changing the precedence of options in postgresql.conf has about
> zero chance.

Sadly, you're probably right.

> That would make it possible to easily write a query that works across
> intallation that warns about any values stored in auto.conf, even if
> they are overwritten by a per-user config or similar.

I had the impression that 'approved for' above meant something which
actually *enforced* it rather than just another monitoring check..

	Thanks,

		Stephen

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Robert Haas <robertmhaas@gmail.com> — 2013-08-30T13:54:52Z

On Fri, Aug 30, 2013 at 9:49 AM, Stephen Frost <sfrost@snowman.net> wrote:
> * Robert Haas (robertmhaas@gmail.com) wrote:
>> there's a fairly generous but fixed-at-startup-time limit on how many
>> segments you can have.  In practice I don't think this matters much,
>> but it was a sobering reminder that the main shared memory segment,
>> with all of its inflexibility, has important reliability properties
>> that are hard to replicate in more dynamic scenarios.
>
> Why wouldn't it be possible to have the same arrangment for
> shared_buffers, where you have more entires than you 'need' at startup
> but which then allows you to add more shared segments later?  I do see
> that we would need an additional bit of indirection to handle that,
> which might be too much overhead, but the concept seems possible.  Isn't
> that more-or-less how the kernel handles dynamic memory..?

Yeah, I think that something like would be possible, but the
synchronization would be tricky, and it wouldn't work at all with
System V shared memory or Windows shared memory, which apparently
can't be resized after creation.  Also, it would rely on the system
administrator having a sensible setting for
max_on_the_fly_shared_buffers and only having the wrong setting for
initial_shared_buffers_until_i_change_it_later, which might not cover
a very high percentage of the cases that arise in practice.

>> Under the currently-proposed design, it can't be used to do any such
>> thing.  It can only be used to modify some auto.conf file which lives
>> in $PGDATA.  It's therefore no different from the ops perspective than
>> ALTER DATABASE or ALTER USER - except that it allows all settings to
>> be changed rather than only a subset.
>
> Claiming that modifying a file *included from a file in /etc* doesn't
> modify things under /etc is disingenuous, imv.

I think you're getting way too hung up on the fact that the proposed
auto.conf will be stored as a flat file.  From your comments upthread,
I gather that you'd be rejoicing if it were a table.  The only reason
we're not doing that is because of the possibility that something in
auto.conf might keep the server from starting.  I don't think that's
gonna be very common now that we're mostly out from under the System V
shared memory limits, but we'll see.  As you point out, it might also
be worth fixing: maybe we can find some way to arrange things so that
the server will always be able to start up regardless of how badly
messed-up auto.conf is... but that's a different patch.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-30T13:55:43Z

* Robert Haas (robertmhaas@gmail.com) wrote:
> On Fri, Aug 30, 2013 at 8:53 AM, Stephen Frost <sfrost@snowman.net> wrote:
> > Yes, one of the issues with the existing system is that you can't
> > specify a default to be applied to new roles.  Also, there are
> > parameters which are not per-role yet which it probably makes sense to
> > be in the database and we'd need a way to deal with that.  Although, at
> > the same time, considering it role based does make for a nice
> > distinction.  Unfortunately, those clammoring for this will argue that
> > it wouldn't replace adminpack and that they couldn't use it to modify
> > their /etc/network/interfaces file, which is the obvious next step to
> > all of this.
> 
> This is a straw-man.  

It was intended to be and I wasn't particularly expecting a response to
the sarcasm, at the same time..

> Likening this to the
> feature being proposed is silly.  What is being asked for here is the
> ability to easily modify system-wide settings from the SQL prompt,
> just as today you can modify settings per-user or per-database.

/etc/network/interfaces would be considered part of the "system" by
some.. :)  PG isn't an OS, yet..

> That's not the same thing as rewriting the entire system
> configuration.

And here you're using the *other* meaning of system, no?  Glad there
won't be any confusion here! ;)  (yes, I'm kidding..).

	Thanks,

		Stephen

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-30T14:46:43Z

* Robert Haas (robertmhaas@gmail.com) wrote:
> I think you're getting way too hung up on the fact that the proposed
> auto.conf will be stored as a flat file.  From your comments upthread,
> I gather that you'd be rejoicing if it were a table.  

I'd be happy if it was a table which managed an *independent* set of
parameters from those used to bootstrap the system, but no one seems to
like breaking up the options between "things that can be sanely modified
without other OS changes" and "things which require OS support".

	Thanks,

		Stephen

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Tom Lane <tgl@sss.pgh.pa.us> — 2013-08-30T14:54:58Z

Stephen Frost <sfrost@snowman.net> writes:
> * Robert Haas (robertmhaas@gmail.com) wrote:
>> I think you're getting way too hung up on the fact that the proposed
>> auto.conf will be stored as a flat file.  From your comments upthread,
>> I gather that you'd be rejoicing if it were a table.  

> I'd be happy if it was a table which managed an *independent* set of
> parameters from those used to bootstrap the system, but no one seems to
> like breaking up the options between "things that can be sanely modified
> without other OS changes" and "things which require OS support".

I agree with Robert's comments upthread that if the new facility can't do
everything that can be done today by editing postgresql.conf, it's not
going to be adequate.  So I'm not in favor of having two sets of
parameters.  It's also not clear to me that we can make a reliable
distinction between parameters that can prevent a server restart vs those
that can't; or at least, the set of the latter will be much smaller than
one could wish.

			regards, tom lane

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Stephen Frost <sfrost@snowman.net> — 2013-08-30T15:15:49Z

Tom, all,

  I'm not one to give up a fight (I hope that's something ya'll like
  about me ;), but in this case I'm gonna have to concede.  Clearly, I'm
  in the minority about this, at least on the lists and among the active
  hackers.  Let me just say that I hope all the happy users of this will
  outweigh the complaints.  I suppose there's only one way to find out.

  	Thanks!

		Stephen

* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> Stephen Frost <sfrost@snowman.net> writes:
> > * Robert Haas (robertmhaas@gmail.com) wrote:
> >> I think you're getting way too hung up on the fact that the proposed
> >> auto.conf will be stored as a flat file.  From your comments upthread,
> >> I gather that you'd be rejoicing if it were a table.  
> 
> > I'd be happy if it was a table which managed an *independent* set of
> > parameters from those used to bootstrap the system, but no one seems to
> > like breaking up the options between "things that can be sanely modified
> > without other OS changes" and "things which require OS support".
> 
> I agree with Robert's comments upthread that if the new facility can't do
> everything that can be done today by editing postgresql.conf, it's not
> going to be adequate.  So I'm not in favor of having two sets of
> parameters.  It's also not clear to me that we can make a reliable
> distinction between parameters that can prevent a server restart vs those
> that can't; or at least, the set of the latter will be much smaller than
> one could wish.
> 
> 			regards, tom lane

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila16@gmail.com> — 2013-09-28T07:05:56Z

On Fri, Aug 30, 2013 at 8:24 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Stephen Frost <sfrost@snowman.net> writes:
>> * Robert Haas (robertmhaas@gmail.com) wrote:
>>> I think you're getting way too hung up on the fact that the proposed
>>> auto.conf will be stored as a flat file.  From your comments upthread,
>>> I gather that you'd be rejoicing if it were a table.
>
>> I'd be happy if it was a table which managed an *independent* set of
>> parameters from those used to bootstrap the system, but no one seems to
>> like breaking up the options between "things that can be sanely modified
>> without other OS changes" and "things which require OS support".
>
> I agree with Robert's comments upthread that if the new facility can't do
> everything that can be done today by editing postgresql.conf, it's not
> going to be adequate.  So I'm not in favor of having two sets of
> parameters.  It's also not clear to me that we can make a reliable
> distinction between parameters that can prevent a server restart vs those
> that can't; or at least, the set of the latter will be much smaller than
> one could wish.

Now as we have an agreement, I had updated patch for below left issues:
1. ALTER SYSTEM SET should be constrained to only set known GUCs, this
point has been discussed on another mail thread
   (http://www.postgresql.org/message-id/14857.1378523977@sss.pgh.pa.us)
   In function AlterSystemSetConfigFile(), when we try to get the
record using find_option(), pass second parameter as false which will
make sure
   if the parameter doesn't exist it will return NULL.
2. Some indentation issues.

With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Peter Eisentraut <peter_e@gmx.net> — 2013-09-30T15:37:09Z
```
On 9/28/13 3:05 AM, Amit Kapila wrote:
> Now as we have an agreement, I had updated patch for below left issues:

Regression tests are failing.
```

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila16@gmail.com> — 2013-09-30T19:26:04Z

On Mon, Sep 30, 2013 at 9:07 PM, Peter Eisentraut <peter_e@gmx.net> wrote:
> On 9/28/13 3:05 AM, Amit Kapila wrote:
>> Now as we have an agreement, I had updated patch for below left issues:
>
> Regression tests are failing.

   Thanks for informing. I am sorry for not running regression before
sending patch.

   Reason for failure was that source for GUC in new function
validate_conf_option() was hardcoded to PGC_S_FILE which was okay for
Alter System, but
   not for SET path. I had added new parameter source in this function
and get the value of source when this is called from SET path.

With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Haribabu kommi <haribabu.kommi@huawei.com> — 2013-11-13T10:35:51Z

On 01 October 2013 00:56 Amit Kapila wrote:
> On Mon, Sep 30, 2013 at 9:07 PM, Peter Eisentraut <peter_e@gmx.net>
> wrote:
> > On 9/28/13 3:05 AM, Amit Kapila wrote:
> >> Now as we have an agreement, I had updated patch for below left
> issues:
> >
> > Regression tests are failing.
> 
>    Thanks for informing. I am sorry for not running regression before
> sending patch.
> 
>    Reason for failure was that source for GUC in new function
> validate_conf_option() was hardcoded to PGC_S_FILE which was okay for
> Alter System, but
>    not for SET path. I had added new parameter source in this function
> and get the value of source when this is called from SET path.

Some of the initial observation of the patch are,
1. Patch is not applying against git head, needs a rebase.
2. Patch doesn't contain the tests.

I started reviewing the patch, will share the details once I finish.

Regards,
Hari babu.

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila16@gmail.com> — 2013-11-13T13:53:34Z

On Wed, Nov 13, 2013 at 4:05 PM, Haribabu kommi
<haribabu.kommi@huawei.com> wrote:
> On 01 October 2013 00:56 Amit Kapila wrote:
>> On Mon, Sep 30, 2013 at 9:07 PM, Peter Eisentraut <peter_e@gmx.net>
>> wrote:
>> > On 9/28/13 3:05 AM, Amit Kapila wrote:
>> >> Now as we have an agreement, I had updated patch for below left
>> issues:
>> >
>> > Regression tests are failing.
>>
>>    Thanks for informing. I am sorry for not running regression before
>> sending patch.
>>
>>    Reason for failure was that source for GUC in new function
>> validate_conf_option() was hardcoded to PGC_S_FILE which was okay for
>> Alter System, but
>>    not for SET path. I had added new parameter source in this function
>> and get the value of source when this is called from SET path.
>
> Some of the initial observation of the patch are,
> 1. Patch is not applying against git head, needs a rebase.
> 2. Patch doesn't contain the tests.
   It was intentional and as per feedback for this patch. As for
testing this feature, we need to put sleep after operation, so it was
suggested to remove tests.

> I started reviewing the patch, will share the details once I finish.
Thanks.

With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila16@gmail.com> — 2013-11-14T07:50:10Z

On Wed, Nov 13, 2013 at 7:23 PM, Amit Kapila <amit.kapila16@gmail.com> wrote:
> On Wed, Nov 13, 2013 at 4:05 PM, Haribabu kommi
> <haribabu.kommi@huawei.com> wrote:
>> On 01 October 2013 00:56 Amit Kapila wrote:
>>> On Mon, Sep 30, 2013 at 9:07 PM, Peter Eisentraut <peter_e@gmx.net>
>>> wrote:
>>> > On 9/28/13 3:05 AM, Amit Kapila wrote:
>>> >> Now as we have an agreement, I had updated patch for below left
>>> issues:
>>> >
>>
>> Some of the initial observation of the patch are,
>> 1. Patch is not applying against git head, needs a rebase.

Find the rebased version attached with this mail. I have removed
header inclusions of port.h and lwlock.h from guc.c, it works fine
without them.

>> 2. Patch doesn't contain the tests.
>    It was intentional and as per feedback for this patch. As for
> testing this feature, we need to put sleep after operation, so it was
> suggested to remove tests.


With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Peter Eisentraut <peter_e@gmx.net> — 2013-11-15T16:48:18Z

On 11/14/13, 2:50 AM, Amit Kapila wrote:
> Find the rebased version attached with this mail.

Doesn't build:

openjade  -wall -wno-unused-param -wno-empty -wfully-tagged -D . -D . -c /usr/share/sgml/docbook/stylesheet/dsssl/modular/catalog -d stylesheet.dsl -t sgml -i output-html -V html-index postgres.sgml
openjade:reference.sgml:61:3:E: cannot find "alter_system.sgml"; tried "ref/alter_system.sgml", "./alter_system.sgml", "./alter_system.sgml", "/usr/local/share/sgml/alter_system.sgml", "/usr/share/sgml/alter_system.sgml"
openjade:config.sgml:164:27:X: reference to non-existent ID "SQL-ALTERSYSTEM"
make[3]: *** [HTML.index] Error 1
make[3]: *** Deleting file `HTML.index'
osx -D. -x lower -i include-xslt-index postgres.sgml >postgres.xmltmp
osx:reference.sgml:61:3:E: cannot find "alter_system.sgml"; tried "ref/alter_system.sgml", "./alter_system.sgml", "/usr/local/share/sgml/alter_system.sgml", "/usr/share/sgml/alter_system.sgml"
osx:config.sgml:164:27:X: reference to non-existent ID "SQL-ALTERSYSTEM"
make[3]: *** [postgres.xml] Error 1

New file missing in patch?

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila16@gmail.com> — 2013-11-16T04:12:40Z

On Fri, Nov 15, 2013 at 10:18 PM, Peter Eisentraut <peter_e@gmx.net> wrote:
> On 11/14/13, 2:50 AM, Amit Kapila wrote:
>> Find the rebased version attached with this mail.
>
> Doesn't build:
>
> openjade  -wall -wno-unused-param -wno-empty -wfully-tagged -D . -D . -c /usr/share/sgml/docbook/stylesheet/dsssl/modular/catalog -d stylesheet.dsl -t sgml -i output-html -V html-index postgres.sgml
> openjade:reference.sgml:61:3:E: cannot find "alter_system.sgml"; tried "ref/alter_system.sgml", "./alter_system.sgml", "./alter_system.sgml", "/usr/local/share/sgml/alter_system.sgml", "/usr/share/sgml/alter_system.sgml"
> openjade:config.sgml:164:27:X: reference to non-existent ID "SQL-ALTERSYSTEM"
> make[3]: *** [HTML.index] Error 1
> make[3]: *** Deleting file `HTML.index'
> osx -D. -x lower -i include-xslt-index postgres.sgml >postgres.xmltmp
> osx:reference.sgml:61:3:E: cannot find "alter_system.sgml"; tried "ref/alter_system.sgml", "./alter_system.sgml", "/usr/local/share/sgml/alter_system.sgml", "/usr/share/sgml/alter_system.sgml"
> osx:config.sgml:164:27:X: reference to non-existent ID "SQL-ALTERSYSTEM"
> make[3]: *** [postgres.xml] Error 1
>
> New file missing in patch?

Oops, missed the new sgml file in patch, updated patch to include it.
Many thanks for checking it.

With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Haribabu kommi <haribabu.kommi@huawei.com> — 2013-11-16T11:05:04Z

On 16 November 2013 09:43 Amit Kapila wrote:
> On Fri, Nov 15, 2013 at 10:18 PM, Peter Eisentraut <peter_e@gmx.net>
> wrote:
> > On 11/14/13, 2:50 AM, Amit Kapila wrote:
> >> Find the rebased version attached with this mail.
> >
> > Doesn't build:
> >
> > openjade  -wall -wno-unused-param -wno-empty -wfully-tagged -D . -D .
> -c /usr/share/sgml/docbook/stylesheet/dsssl/modular/catalog -d
> stylesheet.dsl -t sgml -i output-html -V html-index postgres.sgml
> > openjade:reference.sgml:61:3:E: cannot find "alter_system.sgml";
> tried "ref/alter_system.sgml", "./alter_system.sgml",
> "./alter_system.sgml", "/usr/local/share/sgml/alter_system.sgml",
> "/usr/share/sgml/alter_system.sgml"
> > openjade:config.sgml:164:27:X: reference to non-existent ID "SQL-
> ALTERSYSTEM"
> > make[3]: *** [HTML.index] Error 1
> > make[3]: *** Deleting file `HTML.index'
> > osx -D. -x lower -i include-xslt-index postgres.sgml >postgres.xmltmp
> > osx:reference.sgml:61:3:E: cannot find "alter_system.sgml"; tried
> "ref/alter_system.sgml", "./alter_system.sgml",
> "/usr/local/share/sgml/alter_system.sgml",
> "/usr/share/sgml/alter_system.sgml"
> > osx:config.sgml:164:27:X: reference to non-existent ID "SQL-
> ALTERSYSTEM"
> > make[3]: *** [postgres.xml] Error 1
> >
> > New file missing in patch?
> 
> Oops, missed the new sgml file in patch, updated patch to include it.
> Many thanks for checking it.

1. Patch applied properly
2. No warnings in the compilation
3. Regress test passed.
4. Basic tests are passed.

Please find review comments:

+ *		ALTER SYSTEM SET
+ *
+ * Command to edit postgresql.conf
+ *****************************************************************************/

I feel it should be "Command to change the configuration parameter"
because this command is not edits the postgresql.conf file.

 			ereport(ERROR,
 					(errcode(ERRCODE_CONFIG_FILE_ERROR),
 					 errmsg("configuration file \"%s\" contains errors",
-							ConfigFileName)));
+							ErrorConfFile)));

The ErrorConfFile prints "postgresql.auto.conf" only if there is any parsing problem
with postgresql.auto.conf otherwise it always print "postgresql.conf" because of any other error.

+ * A stale temporary file may be left behind in case we crash.
+ * Such files are removed on the next server restart.

The above comment is wrong, the stale temporary file will be used
in the next ALTER SYSTEM command. I didn't find any code where it gets
deleted on the next server restart.

if any postmaster setting which are set by the alter system command which 
leads to failure of server start, what is the solution to user to proceed
further to start the server. As it is mentioned that the auto.conf file
shouldn't be edited manually.

Regards,
Hari babu.

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila16@gmail.com> — 2013-11-17T06:54:36Z

On Sat, Nov 16, 2013 at 4:35 PM, Haribabu kommi
<haribabu.kommi@huawei.com> wrote:
> On 16 November 2013 09:43 Amit Kapila wrote:
>> On Fri, Nov 15, 2013 at 10:18 PM, Peter Eisentraut <peter_e@gmx.net>
>> wrote:
>> > On 11/14/13, 2:50 AM, Amit Kapila wrote:
>> >> Find the rebased version attached with this mail.
>> >
> Please find review comments:
>
> + *             ALTER SYSTEM SET
> + *
> + * Command to edit postgresql.conf
> + *****************************************************************************/
>
> I feel it should be "Command to change the configuration parameter"
> because this command is not edits the postgresql.conf file.

   Changed the comment, but I think it is better to use persistently
in line suggested by you, so I have done that way.

>                         ereport(ERROR,
>                                         (errcode(ERRCODE_CONFIG_FILE_ERROR),
>                                          errmsg("configuration file \"%s\" contains errors",
> -                                                       ConfigFileName)));
> +                                                       ErrorConfFile)));
>
> The ErrorConfFile prints "postgresql.auto.conf" only if there is any parsing problem
> with postgresql.auto.conf otherwise it always print "postgresql.conf" because of any other error.

   Changed to ensure ErrorConfFile contains proper config file name.
   Note: I have not asssigned file name incase of error in below loop,
as file name in gconf is NULL in most cases and moreover this loops
over
            guc_variables which doesn't contain values/parameters from
auto.conf. So I don't think it is required to assign ErrorConfFile in
this loop.

ProcessConfigFile(GucContext context)
{
..
   for (i = 0; i < num_guc_variables; i++)
   {
       struct config_generic *gconf = guc_variables[i];

..
}

>
> + * A stale temporary file may be left behind in case we crash.
> + * Such files are removed on the next server restart.
>
> The above comment is wrong, the stale temporary file will be used
> in the next ALTER SYSTEM command. I didn't find any code where it gets
> deleted on the next server restart.

   Removed the comment from top of function and modified the comment
where file is getting opened.

>
> if any postmaster setting which are set by the alter system command which
> leads to failure of server start, what is the solution to user to proceed
> further to start the server. As it is mentioned that the auto.conf file
> shouldn't be edited manually.

Yeah, but in case of emergency user can change it to get server
started. Now the question is whether to mention it in documentation, I
think we can leave this decision to committer. If he thinks that it is
better to document then I will update it.

With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Haribabu kommi <haribabu.kommi@huawei.com> — 2013-11-18T12:58:38Z

On 17 November 2013 12:25 Amit Kapila wrote:
> On Sat, Nov 16, 2013 at 4:35 PM, Haribabu kommi
> <haribabu.kommi@huawei.com> wrote:
> > On 16 November 2013 09:43 Amit Kapila wrote:
> >> On Fri, Nov 15, 2013 at 10:18 PM, Peter Eisentraut <peter_e@gmx.net>
> >> wrote:
> >> > On 11/14/13, 2:50 AM, Amit Kapila wrote:
> >> >> Find the rebased version attached with this mail.
> >> >
> >                         ereport(ERROR,
> >
> (errcode(ERRCODE_CONFIG_FILE_ERROR),
> >                                          errmsg("configuration file
> \"%s\" contains errors",
> > -
> ConfigFileName)));
> > +
> > + ErrorConfFile)));
> >
> > The ErrorConfFile prints "postgresql.auto.conf" only if there is any
> > parsing problem with postgresql.auto.conf otherwise it always print
> "postgresql.conf" because of any other error.
> 
>    Changed to ensure ErrorConfFile contains proper config file name.
>    Note: I have not asssigned file name incase of error in below loop,
> as file name in gconf is NULL in most cases and moreover this loops
> over
>             guc_variables which doesn't contain values/parameters from
> auto.conf. So I don't think it is required to assign ErrorConfFile in
> this loop.
> 
> ProcessConfigFile(GucContext context)
> {
> ..
>    for (i = 0; i < num_guc_variables; i++)
>    {
>        struct config_generic *gconf = guc_variables[i];
> 
> ..
> }

Code changes are fine. 
If two or three errors are present in the configuration file, it prints the last error
Configuration parameter file only. Is it required to be mentioned in the documentation?

> >
> > if any postmaster setting which are set by the alter system command
> > which leads to failure of server start, what is the solution to user
> > to proceed further to start the server. As it is mentioned that the
> > auto.conf file shouldn't be edited manually.
> 
> Yeah, but in case of emergency user can change it to get server started.
> Now the question is whether to mention it in documentation, I think we
> can leave this decision to committer. If he thinks that it is better to
> document then I will update it.

Ok fine.

Regards,
Hari babu.

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila16@gmail.com> — 2013-11-18T14:31:23Z

On Mon, Nov 18, 2013 at 6:28 PM, Haribabu kommi
<haribabu.kommi@huawei.com> wrote:
> On 17 November 2013 12:25 Amit Kapila wrote:
>> On Sat, Nov 16, 2013 at 4:35 PM, Haribabu kommi
>> >> >> Find the rebased version attached with this mail.
>> >> >
>> >                         ereport(ERROR,
>> >
>> (errcode(ERRCODE_CONFIG_FILE_ERROR),
>> >                                          errmsg("configuration file
>> \"%s\" contains errors",
>> > -
>> ConfigFileName)));
>> > +
>> > + ErrorConfFile)));
>> >
>> > The ErrorConfFile prints "postgresql.auto.conf" only if there is any
>> > parsing problem with postgresql.auto.conf otherwise it always print
>> "postgresql.conf" because of any other error.
>>
>>    Changed to ensure ErrorConfFile contains proper config file name.
>>    Note: I have not asssigned file name incase of error in below loop,
>> as file name in gconf is NULL in most cases and moreover this loops
>> over
>>             guc_variables which doesn't contain values/parameters from
>> auto.conf. So I don't think it is required to assign ErrorConfFile in
>> this loop.
>>
>> ProcessConfigFile(GucContext context)
>> {
>> ..
>>    for (i = 0; i < num_guc_variables; i++)
>>    {
>>        struct config_generic *gconf = guc_variables[i];
>>
>> ..
>> }
>
> Code changes are fine.
> If two or three errors are present in the configuration file, it prints the last error
> Configuration parameter file only. Is it required to be mentioned in the documentation?

Do you mean to say parsing errors or some run-time error, could you
explain with example?

With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Haribabu kommi <haribabu.kommi@huawei.com> — 2013-11-18T15:01:39Z

On 18 November 2013 20:01 Amit Kapila wrote:
> On Mon, Nov 18, 2013 at 6:28 PM, Haribabu kommi
> <haribabu.kommi@huawei.com> wrote:
> > On 17 November 2013 12:25 Amit Kapila wrote:
> >> On Sat, Nov 16, 2013 at 4:35 PM, Haribabu kommi
> >> >> >> Find the rebased version attached with this mail.
> >> >> >
> >> >                         ereport(ERROR,
> >> >
> >> (errcode(ERRCODE_CONFIG_FILE_ERROR),
> >> >                                          errmsg("configuration
> file
> >> \"%s\" contains errors",
> >> > -
> >> ConfigFileName)));
> >> > +
> >> > + ErrorConfFile)));
> >> >
> >> > The ErrorConfFile prints "postgresql.auto.conf" only if there is
> >> > any parsing problem with postgresql.auto.conf otherwise it always
> >> > print
> >> "postgresql.conf" because of any other error.
> >>
> >>    Changed to ensure ErrorConfFile contains proper config file name.
> >>    Note: I have not asssigned file name incase of error in below
> >> loop, as file name in gconf is NULL in most cases and moreover this
> >> loops over
> >>             guc_variables which doesn't contain values/parameters
> >> from auto.conf. So I don't think it is required to assign
> >> ErrorConfFile in this loop.
> >>
> >> ProcessConfigFile(GucContext context) { ..
> >>    for (i = 0; i < num_guc_variables; i++)
> >>    {
> >>        struct config_generic *gconf = guc_variables[i];
> >>
> >> ..
> >> }
> >
> > Code changes are fine.
> > If two or three errors are present in the configuration file, it
> prints the last error
> > Configuration parameter file only. Is it required to be mentioned in
> the documentation?
> 
> Do you mean to say parsing errors or some run-time error, could you
> explain with example?

LOG:  parameter "shared_buffers" cannot be changed without restarting the server
LOG:  parameter "port" cannot be changed without restarting the server
LOG:  configuration file "/home/hari/installation/bin/../../data/postgresql.auto.conf" contains errors; unaffected changes were applied

The shared_buffers parameter is changed in postgresql.conf and port is changed in postgresql.auto.conf.
The error file displays the last error occurred file.

Is it required to mention the above behavior in the document?

Regards,
Hari babu

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila16@gmail.com> — 2013-11-19T04:28:34Z

On Mon, Nov 18, 2013 at 8:31 PM, Haribabu kommi
<haribabu.kommi@huawei.com> wrote:
> On 18 November 2013 20:01 Amit Kapila wrote:
>> > Code changes are fine.
>> > If two or three errors are present in the configuration file, it
>> prints the last error
>> > Configuration parameter file only. Is it required to be mentioned in
>> the documentation?
>>
>> Do you mean to say parsing errors or some run-time error, could you
>> explain with example?
>
> LOG:  parameter "shared_buffers" cannot be changed without restarting the server
> LOG:  parameter "port" cannot be changed without restarting the server
> LOG:  configuration file "/home/hari/installation/bin/../../data/postgresql.auto.conf" contains errors; unaffected changes were applied
>
> The shared_buffers parameter is changed in postgresql.conf and port is changed in postgresql.auto.conf.
> The error file displays the last error occurred file.

   This is only possible if user tries to change configuration
parameters both by Alter System command and manually as well and the
case above is
   not an error, it is just an information for user that there are
some parameters changed in config file which can only get reflected
after server restart.
   So definitely user can check log files to see which parameter's
doesn't get reflected if he is expecting some parameter to be changed,
but it's not
   changed. I think here even in logs if last file containing errors
is mentioned is not a big problem.

> Is it required to mention the above behavior in the document?

   It is better to show in log both the files rather than documenting
it, if the the above case is helpful for user which I don't think so,
also to handle case
   in code can complicate the error handling path of code a bit. So I
think we can leave this case as-is.

With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Haribabu kommi <haribabu.kommi@huawei.com> — 2013-11-19T08:36:34Z

On 19 November 2013 09:59 Amit Kapila wrote:
> On Mon, Nov 18, 2013 at 8:31 PM, Haribabu kommi
> <haribabu.kommi@huawei.com> wrote:
> > On 18 November 2013 20:01 Amit Kapila wrote:
> >> > Code changes are fine.
> >> > If two or three errors are present in the configuration file, it
> >> prints the last error
> >> > Configuration parameter file only. Is it required to be mentioned
> >> > in
> >> the documentation?
> >>
> >> Do you mean to say parsing errors or some run-time error, could you
> >> explain with example?
> >
> > LOG:  parameter "shared_buffers" cannot be changed without restarting
> > the server
> > LOG:  parameter "port" cannot be changed without restarting the
> server
> > LOG:  configuration file
> > "/home/hari/installation/bin/../../data/postgresql.auto.conf"
> contains
> > errors; unaffected changes were applied
> >
> > The shared_buffers parameter is changed in postgresql.conf and port
> is changed in postgresql.auto.conf.
> > The error file displays the last error occurred file.
> 
>    This is only possible if user tries to change configuration
> parameters both by Alter System command and manually as well and the
> case above is
>    not an error, it is just an information for user that there are some
> parameters changed in config file which can only get reflected after
> server restart.
>    So definitely user can check log files to see which parameter's
> doesn't get reflected if he is expecting some parameter to be changed,
> but it's not
>    changed. I think here even in logs if last file containing errors is
> mentioned is not a big problem.
> 
> > Is it required to mention the above behavior in the document?
> 
>    It is better to show in log both the files rather than documenting
> it, if the the above case is helpful for user which I don't think so,
> also to handle case
>    in code can complicate the error handling path of code a bit. So I
> think we can leave this case as-is.

Ok fine I marked the patch as ready for committer.

Regards,
Hari babu.

Re: ALTER SYSTEM SET command to change postgresql.conf parameters (RE: Proposal for Allow postgresql.conf values to be changed via SQL [review])

Amit Kapila <amit.kapila16@gmail.com> — 2013-11-19T14:50:25Z

On Tue, Nov 19, 2013 at 2:06 PM, Haribabu kommi
<haribabu.kommi@huawei.com> wrote:
> On 19 November 2013 09:59 Amit Kapila wrote:
>> On Mon, Nov 18, 2013 at 8:31 PM, Haribabu kommi
>> <haribabu.kommi@huawei.com> wrote:
>> > On 18 November 2013 20:01 Amit Kapila wrote:
>> >> > Code changes are fine.
>> >> > If two or three errors are present in the configuration file, it
>> >> prints the last error
>
> Ok fine I marked the patch as ready for committer.

   Thanks for review.

With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: ALTER SYSTEM SET command to change postgresql.conf parameters

Tatsuo Ishii <ishii@postgresql.org> — 2013-12-18T08:35:34Z

Hi,

I have looked into this because it's marked as "ready for committer".

> On Tue, Nov 19, 2013 at 2:06 PM, Haribabu kommi
> <haribabu.kommi@huawei.com> wrote:
>> On 19 November 2013 09:59 Amit Kapila wrote:
>>> On Mon, Nov 18, 2013 at 8:31 PM, Haribabu kommi
>>> <haribabu.kommi@huawei.com> wrote:
>>> > On 18 November 2013 20:01 Amit Kapila wrote:
>>> >> > Code changes are fine.
>>> >> > If two or three errors are present in the configuration file, it
>>> >> prints the last error
>>
>> Ok fine I marked the patch as ready for committer.
> 
>    Thanks for review.
> 
> With Regards,
> Amit Kapila.
> EnterpriseDB: http://www.enterprisedb.com

It looks like working as advertised. Great! However I have noticed a
few minor issues.

1) validate_conf_option

+/*
+ * Validates configuration parameter and value, by calling check hook functions
+ * depending on record's vartype. It validates if the parameter
+ * value given is in range of expected predefined value for that parameter.
+ *
+ * freemem - true indicates memory for newval and newextra will be
+ *			 freed in this function, false indicates it will be freed
+ *			 by caller.
+ * Return value:
+ *	1: the value is valid
+ *	0: the name or value is invalid
+ */
+int
+validate_conf_option(struct config_generic * record, const char *name,
+					 const char *value, GucSource source, int elevel,
+					 bool freemem, void *newval, void **newextra)

Is there any reason for the function returns int as it always returns
0 or 1. Maybe returns bool is better?

2) initdb.c

+	strcpy(tempautobuf, "# Do not edit this file manually! \n");
+	autoconflines[0] = pg_strdup(tempautobuf);
+	strcpy(tempautobuf, "# It will be overwritten by the ALTER SYSTEM command. \n");
+	autoconflines[1] = pg_strdup(tempautobuf);

Is there any reason to use "tempautobuf" here? I think we can simply change to this:

+	autoconflines[0] = pg_strdup("# Do not edit this file manually! \n");
+	autoconflines[1] = pg_strdup("# It will be overwritten by the ALTER SYSTEM command. \n");

3) initdb.c

It seems the memory allocated for autoconflines[0] and
autoconflines[1] by pg_strdup is never freed.

(I think there's similar problem with "conflines" as well, though it
was not introduced by the patch).

Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese: http://www.sraoss.co.jp

Re: ALTER SYSTEM SET command to change postgresql.conf parameters

Amit Kapila <amit.kapila16@gmail.com> — 2013-12-18T12:38:26Z

On Wed, Dec 18, 2013 at 2:05 PM, Tatsuo Ishii <ishii@postgresql.org> wrote:
> Hi,
>
> I have looked into this because it's marked as "ready for committer".
   Thank you.
> It looks like working as advertised. Great! However I have noticed a
> few minor issues.
>
> 1) validate_conf_option
>
> +/*
> + * Validates configuration parameter and value, by calling check hook functions
> + * depending on record's vartype. It validates if the parameter
> + * value given is in range of expected predefined value for that parameter.
> + *
> + * freemem - true indicates memory for newval and newextra will be
> + *                      freed in this function, false indicates it will be freed
> + *                      by caller.
> + * Return value:
> + *     1: the value is valid
> + *     0: the name or value is invalid
> + */
> +int
> +validate_conf_option(struct config_generic * record, const char *name,
> +                                        const char *value, GucSource source, int elevel,
> +                                        bool freemem, void *newval, void **newextra)
>
> Is there any reason for the function returns int as it always returns
> 0 or 1. Maybe returns bool is better?

No, return type should be bool, I have changed the same in attached patch.

> 2) initdb.c
>
> +       strcpy(tempautobuf, "# Do not edit this file manually! \n");
> +       autoconflines[0] = pg_strdup(tempautobuf);
> +       strcpy(tempautobuf, "# It will be overwritten by the ALTER SYSTEM command. \n");
> +       autoconflines[1] = pg_strdup(tempautobuf);
>
> Is there any reason to use "tempautobuf" here? I think we can simply change to this:
>
> +       autoconflines[0] = pg_strdup("# Do not edit this file manually! \n");
> +       autoconflines[1] = pg_strdup("# It will be overwritten by the ALTER SYSTEM command. \n");

You are right, I have changed code as per your suggestion.

> 3) initdb.c
>
> It seems the memory allocated for autoconflines[0] and
> autoconflines[1] by pg_strdup is never freed.

I think, it gets freed in writefile() in below code.

for (line = lines; *line != NULL; line++)
{
if (fputs(*line, out_file) < 0)
{
fprintf(stderr, _("%s: could not write file \"%s\": %s\n"),
progname, path, strerror(errno));
exit_nicely();
}
free(*line);
}


With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: ALTER SYSTEM SET command to change postgresql.conf parameters

Andrew Dunstan <andrew@dunslane.net> — 2013-12-18T14:00:40Z

On 12/18/2013 03:35 AM, Tatsuo Ishii wrote:
> 3) initdb.c
>
> It seems the memory allocated for autoconflines[0] and
> autoconflines[1] by pg_strdup is never freed.
>
> (I think there's similar problem with "conflines" as well, though it
> was not introduced by the patch).
>

Why would we care? initdb doesn't run for very long, and the memory will 
be freed when it exits, usually within a few seconds. My recollection 
from back when I originally rewrote initdb in C was that cleaning up the 
memory leaks wasn't worth the trouble.

cheers

andrew

Re: ALTER SYSTEM SET command to change postgresql.conf parameters

Tatsuo Ishii <ishii@postgresql.org> — 2013-12-18T14:55:41Z

>> Is there any reason for the function returns int as it always returns
>> 0 or 1. Maybe returns bool is better?
> 
> No, return type should be bool, I have changed the same in attached patch.

Confirmed.

>> 2) initdb.c
>>
>> +       strcpy(tempautobuf, "# Do not edit this file manually! \n");
>> +       autoconflines[0] = pg_strdup(tempautobuf);
>> +       strcpy(tempautobuf, "# It will be overwritten by the ALTER SYSTEM command. \n");
>> +       autoconflines[1] = pg_strdup(tempautobuf);
>>
>> Is there any reason to use "tempautobuf" here? I think we can simply change to this:
>>
>> +       autoconflines[0] = pg_strdup("# Do not edit this file manually! \n");
>> +       autoconflines[1] = pg_strdup("# It will be overwritten by the ALTER SYSTEM command. \n");
> 
> You are right, I have changed code as per your suggestion.

Confirmed.

>> 3) initdb.c
>>
>> It seems the memory allocated for autoconflines[0] and
>> autoconflines[1] by pg_strdup is never freed.
> 
> I think, it gets freed in writefile() in below code.

Oh, I see. Sorry for noise.

I have committed your patches. Thanks.
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese: http://www.sraoss.co.jp

Re: ALTER SYSTEM SET command to change postgresql.conf parameters

Amit Kapila <amit.kapila16@gmail.com> — 2013-12-19T03:08:41Z

On Wed, Dec 18, 2013 at 8:25 PM, Tatsuo Ishii <ishii@postgresql.org> wrote:
>>> Is there any reason for the function returns int as it always returns
>>> 0 or 1. Maybe returns bool is better?
>>
>
> I have committed your patches. Thanks.

Thank you very much.


With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: ALTER SYSTEM SET command to change postgresql.conf parameters

Fujii Masao <masao.fujii@gmail.com> — 2013-12-19T04:51:31Z

On Thu, Dec 19, 2013 at 12:08 PM, Amit Kapila <amit.kapila16@gmail.com> wrote:
> On Wed, Dec 18, 2013 at 8:25 PM, Tatsuo Ishii <ishii@postgresql.org> wrote:
>>>> Is there any reason for the function returns int as it always returns
>>>> 0 or 1. Maybe returns bool is better?
>>>
>>
>> I have committed your patches. Thanks.
>
> Thank you very much.

I found that the psql tab-completion for ALTER SYSTEM SET has not been
implemented yet.
Attached patch does that. Barring any objections, I will commit this patch.

Regards,

-- 
Fujii Masao

Re: ALTER SYSTEM SET command to change postgresql.conf parameters

Tatsuo Ishii <ishii@postgresql.org> — 2013-12-19T05:21:19Z

> I found that the psql tab-completion for ALTER SYSTEM SET has not been
> implemented yet.
> Attached patch does that. Barring any objections, I will commit this patch.

Good catch!

Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese: http://www.sraoss.co.jp

Re: ALTER SYSTEM SET command to change postgresql.conf parameters

Fujii Masao <masao.fujii@gmail.com> — 2013-12-19T17:35:28Z

On Thu, Dec 19, 2013 at 2:21 PM, Tatsuo Ishii <ishii@postgresql.org> wrote:
>> I found that the psql tab-completion for ALTER SYSTEM SET has not been
>> implemented yet.
>> Attached patch does that. Barring any objections, I will commit this patch.
>
> Good catch!

Committed.

Regards,

-- 
Fujii Masao

Re: ALTER SYSTEM SET command to change postgresql.conf parameters

Fujii Masao <masao.fujii@gmail.com> — 2013-12-22T09:31:43Z

On Fri, Dec 20, 2013 at 2:35 AM, Fujii Masao <masao.fujii@gmail.com> wrote:
> On Thu, Dec 19, 2013 at 2:21 PM, Tatsuo Ishii <ishii@postgresql.org> wrote:
>>> I found that the psql tab-completion for ALTER SYSTEM SET has not been
>>> implemented yet.
>>> Attached patch does that. Barring any objections, I will commit this patch.
>>
>> Good catch!
>
> Committed.

I found the bug of ALTER SYSTEM SET patch. The procedure to reproduce
it is here.

$ psql
=# ALTER SYSTEM SET shared_buffers = '512MB';
ALTER SYSTEM
=# \q
$ pg_ctl -D data reload
server signaled
2013-12-22 18:24:13 JST LOG:  received SIGHUP, reloading configuration files
2013-12-22 18:24:13 JST LOG:  parameter "shared_buffers" cannot be
changed without restarting the server
2013-12-22 18:24:13 JST LOG:  configuration file "X??" contains
errors; unaffected changes were applied

The configuration file name in the last log message is broken. This problem was
introduced by the ALTER SYSTEM SET patch.

>    FreeConfigVariables(head);
> <snip>
>        else if (apply)
>            ereport(elevel,
>                    (errcode(ERRCODE_CONFIG_FILE_ERROR),
>                     errmsg("configuration file \"%s\" contains errors; unaffected changes were applied",
>                            ErrorConfFile)));

The cause of the problem is that, in ProcessConfigFile(), the log
message including
the 'ErrorConfFile' is emitted after 'head' is free'd even though
'ErrorConfFile' points
to one of entry of the 'head'.

Regards,

-- 
Fujii Masao

Re: ALTER SYSTEM SET command to change postgresql.conf parameters

Amit Kapila <amit.kapila16@gmail.com> — 2013-12-22T15:02:58Z

On Sun, Dec 22, 2013 at 3:01 PM, Fujii Masao <masao.fujii@gmail.com> wrote:
> I found the bug of ALTER SYSTEM SET patch. The procedure to reproduce
> it is here.
>
> $ psql
> =# ALTER SYSTEM SET shared_buffers = '512MB';
> ALTER SYSTEM
> =# \q
> $ pg_ctl -D data reload
> server signaled
> 2013-12-22 18:24:13 JST LOG:  received SIGHUP, reloading configuration files
> 2013-12-22 18:24:13 JST LOG:  parameter "shared_buffers" cannot be
> changed without restarting the server
> 2013-12-22 18:24:13 JST LOG:  configuration file "X??" contains
> errors; unaffected changes were applied
>
> The configuration file name in the last log message is broken. This problem was
> introduced by the ALTER SYSTEM SET patch.
>
>>    FreeConfigVariables(head);
>> <snip>
>>        else if (apply)
>>            ereport(elevel,
>>                    (errcode(ERRCODE_CONFIG_FILE_ERROR),
>>                     errmsg("configuration file \"%s\" contains errors; unaffected changes were applied",
>>                            ErrorConfFile)));
>
> The cause of the problem is that, in ProcessConfigFile(), the log
> message including
> the 'ErrorConfFile' is emitted after 'head' is free'd even though
> 'ErrorConfFile' points
> to one of entry of the 'head'.

Your analysis is absolutely right.
The reason for this issue is that we want to display filename to which
erroneous parameter
belongs and in this case we are freeing the parameter info before actual error.
To fix, we need to save the filename value before freeing parameter list.

Please find the attached patch to fix this issue.

Note - In my m/c, I could not reproduce the issue. I think this is not
surprising because we
are not setting freed memory to NULL, so it can display anything
(sometimes right value as well)

With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: ALTER SYSTEM SET command to change postgresql.conf parameters

Amit Kapila <amit.kapila16@gmail.com> — 2013-12-23T17:57:04Z

On Sun, Dec 22, 2013 at 8:32 PM, Amit Kapila <amit.kapila16@gmail.com> wrote:
> On Sun, Dec 22, 2013 at 3:01 PM, Fujii Masao <masao.fujii@gmail.com> wrote:
>> I found the bug of ALTER SYSTEM SET patch. The procedure to reproduce
>> it is here.
>>
>> $ psql
>> =# ALTER SYSTEM SET shared_buffers = '512MB';
>> ALTER SYSTEM
>> =# \q
>> $ pg_ctl -D data reload
>> server signaled
>> 2013-12-22 18:24:13 JST LOG:  received SIGHUP, reloading configuration files
>> 2013-12-22 18:24:13 JST LOG:  parameter "shared_buffers" cannot be
>> changed without restarting the server
>> 2013-12-22 18:24:13 JST LOG:  configuration file "X??" contains
>> errors; unaffected changes were applied
>>
>> The configuration file name in the last log message is broken. This problem was
>> introduced by the ALTER SYSTEM SET patch.
>>
>>>    FreeConfigVariables(head);
>>> <snip>
>>>        else if (apply)
>>>            ereport(elevel,
>>>                    (errcode(ERRCODE_CONFIG_FILE_ERROR),
>>>                     errmsg("configuration file \"%s\" contains errors; unaffected changes were applied",
>>>                            ErrorConfFile)));
>>
>> The cause of the problem is that, in ProcessConfigFile(), the log
>> message including
>> the 'ErrorConfFile' is emitted after 'head' is free'd even though
>> 'ErrorConfFile' points
>> to one of entry of the 'head'.
>
> Your analysis is absolutely right.
> The reason for this issue is that we want to display filename to which
> erroneous parameter
> belongs and in this case we are freeing the parameter info before actual error.
> To fix, we need to save the filename value before freeing parameter list.
>
> Please find the attached patch to fix this issue.
>
> Note - In my m/c, I could not reproduce the issue. I think this is not
> surprising because we
> are not setting freed memory to NULL, so it can display anything
> (sometimes right value as well)

If you find that the provided patch doesn't fix the problem or you don't
find it appropriate due to some other reason, let me know the feedback.

I shall be able to provide updated patch early next as I will be on vacation for
remaining part of this week.

With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: ALTER SYSTEM SET command to change postgresql.conf parameters

Thom Brown <thom@linux.com> — 2013-12-23T18:42:26Z

Discussion around this topic has reached hundreds of messages, and
whilst I have failed to find mention of my following question, I
appreciate it may have already been discussed.

I have noticed that configuration parameters for extensions are only
supported if the server was started with one of the parameters already
being set in postgresql.conf:

# without any mention in postgresql.conf
postgres=# ALTER SYSTEM SET auto_explain.log_verbose = true;
ERROR:  unrecognized configuration parameter "auto_explain.log_verbose"

# with auto_explain.log_verbose = false added to postgresql.conf
postgres=# ALTER SYSTEM SET auto_explain.log_verbose = true;
ALTER SYSTEM

Removing the entry from postgresql.conf, restarting Postgres, setting
it to the default, then restarting again renders it unsettable again:

# removed entry from postgresql.conf and restarted

postgres=# ALTER SYSTEM SET auto_explain.log_verbose = default;
ALTER SYSTEM

# restart postgres

postgres=# ALTER SYSTEM SET auto_explain.log_verbose = true;
ERROR:  unrecognized configuration parameter "auto_explain.log_verbose"

Is this by design?

-- 
Thom

Re: ALTER SYSTEM SET command to change postgresql.conf parameters

Robert Haas <robertmhaas@gmail.com> — 2013-12-23T19:14:07Z

On Mon, Dec 23, 2013 at 1:42 PM, Thom Brown <thom@linux.com> wrote:
> Discussion around this topic has reached hundreds of messages, and
> whilst I have failed to find mention of my following question, I
> appreciate it may have already been discussed.
>
> I have noticed that configuration parameters for extensions are only
> supported if the server was started with one of the parameters already
> being set in postgresql.conf:
>
> # without any mention in postgresql.conf
> postgres=# ALTER SYSTEM SET auto_explain.log_verbose = true;
> ERROR:  unrecognized configuration parameter "auto_explain.log_verbose"
>
> # with auto_explain.log_verbose = false added to postgresql.conf
> postgres=# ALTER SYSTEM SET auto_explain.log_verbose = true;
> ALTER SYSTEM
>
> Removing the entry from postgresql.conf, restarting Postgres, setting
> it to the default, then restarting again renders it unsettable again:
>
> # removed entry from postgresql.conf and restarted
>
> postgres=# ALTER SYSTEM SET auto_explain.log_verbose = default;
> ALTER SYSTEM
>
> # restart postgres
>
> postgres=# ALTER SYSTEM SET auto_explain.log_verbose = true;
> ERROR:  unrecognized configuration parameter "auto_explain.log_verbose"
>
> Is this by design?

I would think that you'd need to have auto_explain loaded in the
backend where you're trying to make a change, but you shouldn't need
the setting to be present in postgresql.conf, I would think.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: ALTER SYSTEM SET command to change postgresql.conf parameters

Thom Brown <thom@linux.com> — 2013-12-23T19:19:56Z

On 23 December 2013 19:14, Robert Haas <robertmhaas@gmail.com> wrote:
> On Mon, Dec 23, 2013 at 1:42 PM, Thom Brown <thom@linux.com> wrote:
>> Discussion around this topic has reached hundreds of messages, and
>> whilst I have failed to find mention of my following question, I
>> appreciate it may have already been discussed.
>>
>> I have noticed that configuration parameters for extensions are only
>> supported if the server was started with one of the parameters already
>> being set in postgresql.conf:
>>
>> # without any mention in postgresql.conf
>> postgres=# ALTER SYSTEM SET auto_explain.log_verbose = true;
>> ERROR:  unrecognized configuration parameter "auto_explain.log_verbose"
>>
>> # with auto_explain.log_verbose = false added to postgresql.conf
>> postgres=# ALTER SYSTEM SET auto_explain.log_verbose = true;
>> ALTER SYSTEM
>>
>> Removing the entry from postgresql.conf, restarting Postgres, setting
>> it to the default, then restarting again renders it unsettable again:
>>
>> # removed entry from postgresql.conf and restarted
>>
>> postgres=# ALTER SYSTEM SET auto_explain.log_verbose = default;
>> ALTER SYSTEM
>>
>> # restart postgres
>>
>> postgres=# ALTER SYSTEM SET auto_explain.log_verbose = true;
>> ERROR:  unrecognized configuration parameter "auto_explain.log_verbose"
>>
>> Is this by design?
>
> I would think that you'd need to have auto_explain loaded in the
> backend where you're trying to make a change, but you shouldn't need
> the setting to be present in postgresql.conf, I would think.

This appears to be the case.  I hadn't set the library to be loaded in
the config.

I guess therefore it follows that arbitrary configuration parameters
aren't supported (e.g. moo.bark = 5), only pre-defined ones.

Thanks

Thom

Re: ALTER SYSTEM SET command to change postgresql.conf parameters

Robert Haas <robertmhaas@gmail.com> — 2013-12-23T19:35:52Z

On Mon, Dec 23, 2013 at 2:19 PM, Thom Brown <thom@linux.com> wrote:
>> I would think that you'd need to have auto_explain loaded in the
>> backend where you're trying to make a change, but you shouldn't need
>> the setting to be present in postgresql.conf, I would think.
>
> This appears to be the case.  I hadn't set the library to be loaded in
> the config.
>
> I guess therefore it follows that arbitrary configuration parameters
> aren't supported (e.g. moo.bark = 5), only pre-defined ones.

Yeah, and that's by design.  Otherwise, it would be too easy to set a
config parameter to a value that wasn't legal, and therefore make the
server fail to start.  moo.bark = 5 likely won't cause any problems,
but auto_explain.log_verbose = fasle could.  By insisting that the
module providing the GUC be loaded, we can sanity-check the value at
the time it gets set.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: ALTER SYSTEM SET command to change postgresql.conf parameters

Thom Brown <thom@linux.com> — 2013-12-23T19:36:38Z

On 23 December 2013 19:35, Robert Haas <robertmhaas@gmail.com> wrote:
> On Mon, Dec 23, 2013 at 2:19 PM, Thom Brown <thom@linux.com> wrote:
>>> I would think that you'd need to have auto_explain loaded in the
>>> backend where you're trying to make a change, but you shouldn't need
>>> the setting to be present in postgresql.conf, I would think.
>>
>> This appears to be the case.  I hadn't set the library to be loaded in
>> the config.
>>
>> I guess therefore it follows that arbitrary configuration parameters
>> aren't supported (e.g. moo.bark = 5), only pre-defined ones.
>
> Yeah, and that's by design.  Otherwise, it would be too easy to set a
> config parameter to a value that wasn't legal, and therefore make the
> server fail to start.  moo.bark = 5 likely won't cause any problems,
> but auto_explain.log_verbose = fasle could.  By insisting that the
> module providing the GUC be loaded, we can sanity-check the value at
> the time it gets set.

Alles klar. :)

-- 
Thom

Re: ALTER SYSTEM SET command to change postgresql.conf parameters

Fujii Masao <masao.fujii@gmail.com> — 2013-12-24T18:08:41Z

On Tue, Dec 24, 2013 at 2:57 AM, Amit Kapila <amit.kapila16@gmail.com> wrote:
> On Sun, Dec 22, 2013 at 8:32 PM, Amit Kapila <amit.kapila16@gmail.com> wrote:
>> On Sun, Dec 22, 2013 at 3:01 PM, Fujii Masao <masao.fujii@gmail.com> wrote:
>>> I found the bug of ALTER SYSTEM SET patch. The procedure to reproduce
>>> it is here.
>>>
>>> $ psql
>>> =# ALTER SYSTEM SET shared_buffers = '512MB';
>>> ALTER SYSTEM
>>> =# \q
>>> $ pg_ctl -D data reload
>>> server signaled
>>> 2013-12-22 18:24:13 JST LOG:  received SIGHUP, reloading configuration files
>>> 2013-12-22 18:24:13 JST LOG:  parameter "shared_buffers" cannot be
>>> changed without restarting the server
>>> 2013-12-22 18:24:13 JST LOG:  configuration file "X??" contains
>>> errors; unaffected changes were applied
>>>
>>> The configuration file name in the last log message is broken. This problem was
>>> introduced by the ALTER SYSTEM SET patch.
>>>
>>>>    FreeConfigVariables(head);
>>>> <snip>
>>>>        else if (apply)
>>>>            ereport(elevel,
>>>>                    (errcode(ERRCODE_CONFIG_FILE_ERROR),
>>>>                     errmsg("configuration file \"%s\" contains errors; unaffected changes were applied",
>>>>                            ErrorConfFile)));
>>>
>>> The cause of the problem is that, in ProcessConfigFile(), the log
>>> message including
>>> the 'ErrorConfFile' is emitted after 'head' is free'd even though
>>> 'ErrorConfFile' points
>>> to one of entry of the 'head'.
>>
>> Your analysis is absolutely right.
>> The reason for this issue is that we want to display filename to which
>> erroneous parameter
>> belongs and in this case we are freeing the parameter info before actual error.
>> To fix, we need to save the filename value before freeing parameter list.
>>
>> Please find the attached patch to fix this issue.
>>
>> Note - In my m/c, I could not reproduce the issue. I think this is not
>> surprising because we
>> are not setting freed memory to NULL, so it can display anything
>> (sometimes right value as well)
>
> If you find that the provided patch doesn't fix the problem or you don't
> find it appropriate due to some other reason, let me know the feedback.

When I read ProcessConfigFile() more carefully, I found another related
problem. The procedure to reproduce the problem is here.

--------------------
$ psql
=# ALTER SYSTEM SET shared_buffers = '256MB';
=# \q

$ echo "shared_buffers = '256MB'" >> $PGDATA/postgresql.conf
$ pg_ctl -D $PGDATA reload
2013-12-25 03:05:44 JST LOG:  parameter "shared_buffers" cannot be
changed without restarting the server
2013-12-25 03:05:44 JST LOG:  parameter "shared_buffers" cannot be
changed without restarting the server
2013-12-25 03:05:44 JST LOG:  configuration file
"/dav/alter-system/data/postgresql.auto.conf" contains errors;
unaffected changes were applied
--------------------

Both postgresql.conf and postgresql.auto.conf contain the setting of
the parameter that cannot be changed without restarting the server.
But only postgresql.auto.conf was logged, and which would be confusing,
I'm afraid. I think that this problem should be fixed together with the
problem that I reported before. Thought?

Regards,

-- 
Fujii Masao

Re: ALTER SYSTEM SET command to change postgresql.conf parameters

Robert Haas <robertmhaas@gmail.com> — 2013-12-26T23:31:44Z

On Tue, Dec 24, 2013 at 10:08 AM, Fujii Masao <masao.fujii@gmail.com> wrote:
> On Tue, Dec 24, 2013 at 2:57 AM, Amit Kapila <amit.kapila16@gmail.com> wrote:
>> On Sun, Dec 22, 2013 at 8:32 PM, Amit Kapila <amit.kapila16@gmail.com> wrote:
>>> On Sun, Dec 22, 2013 at 3:01 PM, Fujii Masao <masao.fujii@gmail.com> wrote:
>>>> I found the bug of ALTER SYSTEM SET patch. The procedure to reproduce
>>>> it is here.
>>>>
>>>> $ psql
>>>> =# ALTER SYSTEM SET shared_buffers = '512MB';
>>>> ALTER SYSTEM
>>>> =# \q
>>>> $ pg_ctl -D data reload
>>>> server signaled
>>>> 2013-12-22 18:24:13 JST LOG:  received SIGHUP, reloading configuration files
>>>> 2013-12-22 18:24:13 JST LOG:  parameter "shared_buffers" cannot be
>>>> changed without restarting the server
>>>> 2013-12-22 18:24:13 JST LOG:  configuration file "X??" contains
>>>> errors; unaffected changes were applied
>>>>
>>>> The configuration file name in the last log message is broken. This problem was
>>>> introduced by the ALTER SYSTEM SET patch.
>>>>
>>>>>    FreeConfigVariables(head);
>>>>> <snip>
>>>>>        else if (apply)
>>>>>            ereport(elevel,
>>>>>                    (errcode(ERRCODE_CONFIG_FILE_ERROR),
>>>>>                     errmsg("configuration file \"%s\" contains errors; unaffected changes were applied",
>>>>>                            ErrorConfFile)));
>>>>
>>>> The cause of the problem is that, in ProcessConfigFile(), the log
>>>> message including
>>>> the 'ErrorConfFile' is emitted after 'head' is free'd even though
>>>> 'ErrorConfFile' points
>>>> to one of entry of the 'head'.
>>>
>>> Your analysis is absolutely right.
>>> The reason for this issue is that we want to display filename to which
>>> erroneous parameter
>>> belongs and in this case we are freeing the parameter info before actual error.
>>> To fix, we need to save the filename value before freeing parameter list.
>>>
>>> Please find the attached patch to fix this issue.
>>>
>>> Note - In my m/c, I could not reproduce the issue. I think this is not
>>> surprising because we
>>> are not setting freed memory to NULL, so it can display anything
>>> (sometimes right value as well)
>>
>> If you find that the provided patch doesn't fix the problem or you don't
>> find it appropriate due to some other reason, let me know the feedback.
>
> When I read ProcessConfigFile() more carefully, I found another related
> problem. The procedure to reproduce the problem is here.
>
> --------------------
> $ psql
> =# ALTER SYSTEM SET shared_buffers = '256MB';
> =# \q
>
> $ echo "shared_buffers = '256MB'" >> $PGDATA/postgresql.conf
> $ pg_ctl -D $PGDATA reload
> 2013-12-25 03:05:44 JST LOG:  parameter "shared_buffers" cannot be
> changed without restarting the server
> 2013-12-25 03:05:44 JST LOG:  parameter "shared_buffers" cannot be
> changed without restarting the server
> 2013-12-25 03:05:44 JST LOG:  configuration file
> "/dav/alter-system/data/postgresql.auto.conf" contains errors;
> unaffected changes were applied
> --------------------
>
> Both postgresql.conf and postgresql.auto.conf contain the setting of
> the parameter that cannot be changed without restarting the server.
> But only postgresql.auto.conf was logged, and which would be confusing,
> I'm afraid. I think that this problem should be fixed together with the
> problem that I reported before. Thought?

I have run into this problem on many occasions because my benchmark
scripts usually append a hunk of stuff to postgresql.conf, and any
parameters that were already set generate this warning every time you
reload, because postgresql.conf now mentions that parameter twice.
I'm not quite sure what other problem you want to fix it together
with, and I'm not sure what the right fix is either, but +1 for coming
up with some solution that's better than what we have now.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: ALTER SYSTEM SET command to change postgresql.conf parameters

Amit Kapila <amit.kapila16@gmail.com> — 2013-12-29T05:32:26Z

On Tue, Dec 24, 2013 at 11:38 PM, Fujii Masao <masao.fujii@gmail.com> wrote:
>> On Sun, Dec 22, 2013 at 8:32 PM, Amit Kapila <amit.kapila16@gmail.com> wrote:
>>> On Sun, Dec 22, 2013 at 3:01 PM, Fujii Masao <masao.fujii@gmail.com> wrote:
>>>> I found the bug of ALTER SYSTEM SET patch. The procedure to reproduce
>>>> it is here.
>>>>
>>>> $ psql
>>>> =# ALTER SYSTEM SET shared_buffers = '512MB';
>>>> ALTER SYSTEM
>>>> =# \q
>>>> $ pg_ctl -D data reload
>>>> server signaled
>>>> 2013-12-22 18:24:13 JST LOG:  received SIGHUP, reloading configuration files
>>>> 2013-12-22 18:24:13 JST LOG:  parameter "shared_buffers" cannot be
>>>> changed without restarting the server
>>>> 2013-12-22 18:24:13 JST LOG:  configuration file "X??" contains
>>>> errors; unaffected changes were applied
>>>>
>>>
>>> Your analysis is absolutely right.
>>> The reason for this issue is that we want to display filename to which
>>> erroneous parameter
>>> belongs and in this case we are freeing the parameter info before actual error.
>>> To fix, we need to save the filename value before freeing parameter list.
>>>
>>> Please find the attached patch to fix this issue.
>
> When I read ProcessConfigFile() more carefully, I found another related
> problem. The procedure to reproduce the problem is here.
>
> --------------------
> $ psql
> =# ALTER SYSTEM SET shared_buffers = '256MB';
> =# \q
>
> $ echo "shared_buffers = '256MB'" >> $PGDATA/postgresql.conf
> $ pg_ctl -D $PGDATA reload
> 2013-12-25 03:05:44 JST LOG:  parameter "shared_buffers" cannot be
> changed without restarting the server
> 2013-12-25 03:05:44 JST LOG:  parameter "shared_buffers" cannot be
> changed without restarting the server
> 2013-12-25 03:05:44 JST LOG:  configuration file
> "/dav/alter-system/data/postgresql.auto.conf" contains errors;
> unaffected changes were applied
> --------------------
>
> Both postgresql.conf and postgresql.auto.conf contain the setting of
> the parameter that cannot be changed without restarting the server.
> But only postgresql.auto.conf was logged, and which would be confusing,
> I'm afraid.

The reason for above behaviour is that while applying configuration parameters,
it just note down the last file which has error and ignore others. Now
if we just
want to change behaviour for above case that it display both the files, then
during processing of parameters, we can save the errorconffile if it
is different
from previous. However, let us say if user uses include mechanism of conf files
and used new file for specifying some of the changed parameters, then again
similar thing can be observed.

I think here basic idea is user should avoid using multiple methods to change
configuration parameters, however we cannot stop them from doing the same.
So in some cases like above where user use multiple methods to change
configuration can add more work for him to understand and detect the error.

Also, some similar behaviour can be observed if user uses include mechanism
to specify some config parameters without even Alter System changes.

>I think that this problem should be fixed together with the
> problem that I reported before. Thought?

Here I think you might be worried that if we want to fix the behaviour reported
in this mail, it might overlap with the changes of previous fix. So I
think let us
first decide if we want to keep the current behaviour as it is or would like to
change it and if it needs change, what should be the new behaviour?

With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: ALTER SYSTEM SET command to change postgresql.conf parameters

Amit Kapila <amit.kapila16@gmail.com> — 2013-12-29T05:42:37Z

On Fri, Dec 27, 2013 at 5:01 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> On Tue, Dec 24, 2013 at 10:08 AM, Fujii Masao <masao.fujii@gmail.com> wrote:
>> When I read ProcessConfigFile() more carefully, I found another related
>> problem. The procedure to reproduce the problem is here.
>>
>> --------------------
>> $ psql
>> =# ALTER SYSTEM SET shared_buffers = '256MB';
>> =# \q
>>
>> $ echo "shared_buffers = '256MB'" >> $PGDATA/postgresql.conf
>> $ pg_ctl -D $PGDATA reload
>> 2013-12-25 03:05:44 JST LOG:  parameter "shared_buffers" cannot be
>> changed without restarting the server
>> 2013-12-25 03:05:44 JST LOG:  parameter "shared_buffers" cannot be
>> changed without restarting the server
>> 2013-12-25 03:05:44 JST LOG:  configuration file
>> "/dav/alter-system/data/postgresql.auto.conf" contains errors;
>> unaffected changes were applied
>> --------------------
>>
>> Both postgresql.conf and postgresql.auto.conf contain the setting of
>> the parameter that cannot be changed without restarting the server.
>> But only postgresql.auto.conf was logged, and which would be confusing,
>> I'm afraid. I think that this problem should be fixed together with the
>> problem that I reported before. Thought?
>
> I have run into this problem on many occasions because my benchmark
> scripts usually append a hunk of stuff to postgresql.conf, and any
> parameters that were already set generate this warning every time you
> reload, because postgresql.conf now mentions that parameter twice.
> I'm not quite sure what other problem you want to fix it together
> with,

The 2 problems are:

1. First is that if user changes the config parameter (for ex.
shared_buffers) both by manually
    editing postgresql.conf and by using Alter System command and then
reload the config,
    it will show the message for parameter 'shared_buffers' twice, but
will only show the last file
    where it exists. The detailed description of problem is in current mail.

2. The other problem is in some cases the name of file it display is
junk, problem and fix can be found
    at link:
    http://www.postgresql.org/message-id/CAA4eK1+-yD2p=+6tdj_xPruHN4m_CkvFr51aH0gv3SXLGOACGg@mail.gmail.com

>and I'm not sure what the right fix is either, but +1 for coming
> up with some solution that's better than what we have now.

With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: ALTER SYSTEM SET command to change postgresql.conf parameters

Robert Haas <robertmhaas@gmail.com> — 2014-01-06T14:28:25Z

On Sun, Dec 22, 2013 at 10:02 AM, Amit Kapila <amit.kapila16@gmail.com> wrote:
> On Sun, Dec 22, 2013 at 3:01 PM, Fujii Masao <masao.fujii@gmail.com> wrote:
>> I found the bug of ALTER SYSTEM SET patch. The procedure to reproduce
>> it is here.
>>
>> $ psql
>> =# ALTER SYSTEM SET shared_buffers = '512MB';
>> ALTER SYSTEM
>> =# \q
>> $ pg_ctl -D data reload
>> server signaled
>> 2013-12-22 18:24:13 JST LOG:  received SIGHUP, reloading configuration files
>> 2013-12-22 18:24:13 JST LOG:  parameter "shared_buffers" cannot be
>> changed without restarting the server
>> 2013-12-22 18:24:13 JST LOG:  configuration file "X??" contains
>> errors; unaffected changes were applied
>>
>> The configuration file name in the last log message is broken. This problem was
>> introduced by the ALTER SYSTEM SET patch.
>>
>>>    FreeConfigVariables(head);
>>> <snip>
>>>        else if (apply)
>>>            ereport(elevel,
>>>                    (errcode(ERRCODE_CONFIG_FILE_ERROR),
>>>                     errmsg("configuration file \"%s\" contains errors; unaffected changes were applied",
>>>                            ErrorConfFile)));
>>
>> The cause of the problem is that, in ProcessConfigFile(), the log
>> message including
>> the 'ErrorConfFile' is emitted after 'head' is free'd even though
>> 'ErrorConfFile' points
>> to one of entry of the 'head'.
>
> Your analysis is absolutely right.
> The reason for this issue is that we want to display filename to which
> erroneous parameter
> belongs and in this case we are freeing the parameter info before actual error.
> To fix, we need to save the filename value before freeing parameter list.
>
> Please find the attached patch to fix this issue.
>
> Note - In my m/c, I could not reproduce the issue. I think this is not
> surprising because we
> are not setting freed memory to NULL, so it can display anything
> (sometimes right value as well)

Couldn't we also handle this by postponing FreeConfigVariables until
after the if (error) block?

Also, what's the point of this:

+       snprintf(ConfigAutoFileName,sizeof(ConfigAutoFileName),"%s",
PG_AUTOCONF_FILENAME);

Why copy PG_AUTOCONF_FILENAME into another buffer?  Why not just pass
PG_AUTOCONF_FILENAME directly to ParseConfigFile?

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: ALTER SYSTEM SET command to change postgresql.conf parameters

Amit Kapila <amit.kapila16@gmail.com> — 2014-01-06T14:48:36Z

On Mon, Jan 6, 2014 at 7:58 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> On Sun, Dec 22, 2013 at 10:02 AM, Amit Kapila <amit.kapila16@gmail.com> wrote:
>> On Sun, Dec 22, 2013 at 3:01 PM, Fujii Masao <masao.fujii@gmail.com> wrote:
>>> I found the bug of ALTER SYSTEM SET patch. The procedure to reproduce
>>> it is here.
>>>
>>>>    FreeConfigVariables(head);
>>>> <snip>
>>>>        else if (apply)
>>>>            ereport(elevel,
>>>>                    (errcode(ERRCODE_CONFIG_FILE_ERROR),
>>>>                     errmsg("configuration file \"%s\" contains errors; unaffected changes were applied",
>>>>                            ErrorConfFile)));
>>>
>>> The cause of the problem is that, in ProcessConfigFile(), the log
>>> message including
>>> the 'ErrorConfFile' is emitted after 'head' is free'd even though
>>> 'ErrorConfFile' points
>>> to one of entry of the 'head'.
>>
>> Your analysis is absolutely right.
>> The reason for this issue is that we want to display filename to which
>> erroneous parameter
>> belongs and in this case we are freeing the parameter info before actual error.
>> To fix, we need to save the filename value before freeing parameter list.
>>
>> Please find the attached patch to fix this issue.
>>
>>
>
> Couldn't we also handle this by postponing FreeConfigVariables until
> after the if (error) block?

   Wouldn't doing that way can lead to bigger memory leak, if error level
   is ERROR. Though in current fix also it can leak memory but it will be
   just for ErrorConfFile_save. I think some similar case can happen for
   'pre_value' in code currently as well, that's why I have fixed it in a
   similar way in patch.


> Also, what's the point of this:
>
> +       snprintf(ConfigAutoFileName,sizeof(ConfigAutoFileName),"%s",
> PG_AUTOCONF_FILENAME);
>
> Why copy PG_AUTOCONF_FILENAME into another buffer?  Why not just pass
> PG_AUTOCONF_FILENAME directly to ParseConfigFile?

Initially, I think we were doing concat of folder and file name for
Autofile, that's why
the code was written that way, but currently it can be changed to way you are
suggesting.

With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: ALTER SYSTEM SET command to change postgresql.conf parameters

Robert Haas <robertmhaas@gmail.com> — 2014-01-06T19:22:10Z

On Mon, Jan 6, 2014 at 9:48 AM, Amit Kapila <amit.kapila16@gmail.com> wrote:
>> Couldn't we also handle this by postponing FreeConfigVariables until
>> after the if (error) block?
>
>    Wouldn't doing that way can lead to bigger memory leak, if error level
>    is ERROR. Though in current fix also it can leak memory but it will be
>    just for ErrorConfFile_save. I think some similar case can happen for
>    'pre_value' in code currently as well, that's why I have fixed it in a
>    similar way in patch.

I was assuming that error-recovery would reset the containing memory
context, but I'm not sure what memory context we're executing in at
this point.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: ALTER SYSTEM SET command to change postgresql.conf parameters

Amit Kapila <amit.kapila16@gmail.com> — 2014-01-07T04:37:29Z

On Tue, Jan 7, 2014 at 12:52 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> On Mon, Jan 6, 2014 at 9:48 AM, Amit Kapila <amit.kapila16@gmail.com> wrote:
>>> Couldn't we also handle this by postponing FreeConfigVariables until
>>> after the if (error) block?
>>
>>    Wouldn't doing that way can lead to bigger memory leak, if error level
>>    is ERROR. Though in current fix also it can leak memory but it will be
>>    just for ErrorConfFile_save. I think some similar case can happen for
>>    'pre_value' in code currently as well, that's why I have fixed it in a
>>    similar way in patch.
>
> I was assuming that error-recovery would reset the containing memory
> context, but I'm not sure what memory context we're executing in at
> this point.

This function is called from multiple places and based on when it would
get called the memory context varies. During Startup, it gets called in
Postmaster context and if some backend runs pg_reload_conf(), then
it will get called from other background processes (WAL Writer,
Checpointer, etc..) in their respective contexts (for WAL Writer, the
context will be WAL Writer, ..).

In current code, the only time it can go to error path with elevel as
ERROR is during Postmaster startup
(context == PGC_POSTMASTER), at which it will anyway upgrade
ERROR to FATAL, so it should not be a problem to move
function FreeConfigVariables() after error block check. However
in future, if someone added any more ERROR (the chances of which
seems to be quite less), it can cause leak, may be thats why original
code has been written that way.

If you think it's better to fix by moving FreeConfigVariables() after error
block check, then I can update the patch by doing so and incorporate other
change (directly use PG_AUTOCONF_FILENAME) suggested by you
as well?

With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: ALTER SYSTEM SET command to change postgresql.conf parameters

Robert Haas <robertmhaas@gmail.com> — 2014-01-08T18:56:46Z

On Mon, Jan 6, 2014 at 11:37 PM, Amit Kapila <amit.kapila16@gmail.com> wrote:
> On Tue, Jan 7, 2014 at 12:52 AM, Robert Haas <robertmhaas@gmail.com> wrote:
>> On Mon, Jan 6, 2014 at 9:48 AM, Amit Kapila <amit.kapila16@gmail.com> wrote:
>>>> Couldn't we also handle this by postponing FreeConfigVariables until
>>>> after the if (error) block?
>>>
>>>    Wouldn't doing that way can lead to bigger memory leak, if error level
>>>    is ERROR. Though in current fix also it can leak memory but it will be
>>>    just for ErrorConfFile_save. I think some similar case can happen for
>>>    'pre_value' in code currently as well, that's why I have fixed it in a
>>>    similar way in patch.
>>
>> I was assuming that error-recovery would reset the containing memory
>> context, but I'm not sure what memory context we're executing in at
>> this point.
>
> This function is called from multiple places and based on when it would
> get called the memory context varies. During Startup, it gets called in
> Postmaster context and if some backend runs pg_reload_conf(), then
> it will get called from other background processes (WAL Writer,
> Checpointer, etc..) in their respective contexts (for WAL Writer, the
> context will be WAL Writer, ..).
>
> In current code, the only time it can go to error path with elevel as
> ERROR is during Postmaster startup
> (context == PGC_POSTMASTER), at which it will anyway upgrade
> ERROR to FATAL, so it should not be a problem to move
> function FreeConfigVariables() after error block check. However
> in future, if someone added any more ERROR (the chances of which
> seems to be quite less), it can cause leak, may be thats why original
> code has been written that way.
>
> If you think it's better to fix by moving FreeConfigVariables() after error
> block check, then I can update the patch by doing so and incorporate other
> change (directly use PG_AUTOCONF_FILENAME) suggested by you
> as well?

Yeah, let's do it that way.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: ALTER SYSTEM SET command to change postgresql.conf parameters

Amit Kapila <amit.kapila16@gmail.com> — 2014-01-10T04:06:38Z

On Thu, Jan 9, 2014 at 12:26 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> On Mon, Jan 6, 2014 at 11:37 PM, Amit Kapila <amit.kapila16@gmail.com> wrote:
>> On Tue, Jan 7, 2014 at 12:52 AM, Robert Haas <robertmhaas@gmail.com> wrote:
>>> On Mon, Jan 6, 2014 at 9:48 AM, Amit Kapila <amit.kapila16@gmail.com> wrote:
>>>>> Couldn't we also handle this by postponing FreeConfigVariables until
>>>>> after the if (error) block?
>>>>
>>>>    Wouldn't doing that way can lead to bigger memory leak, if error level
>>>>    is ERROR. Though in current fix also it can leak memory but it will be
>>>>    just for ErrorConfFile_save. I think some similar case can happen for
>>>>    'pre_value' in code currently as well, that's why I have fixed it in a
>>>>    similar way in patch.
>>>
>>> I was assuming that error-recovery would reset the containing memory
>>> context, but I'm not sure what memory context we're executing in at
>>> this point.
>>
>>
>> In current code, the only time it can go to error path with elevel as
>> ERROR is during Postmaster startup
>> (context == PGC_POSTMASTER), at which it will anyway upgrade
>> ERROR to FATAL, so it should not be a problem to move
>> function FreeConfigVariables() after error block check. However
>> in future, if someone added any more ERROR (the chances of which
>> seems to be quite less), it can cause leak, may be thats why original
>> code has been written that way.
>>
>> If you think it's better to fix by moving FreeConfigVariables() after error
>> block check, then I can update the patch by doing so and incorporate other
>> change (directly use PG_AUTOCONF_FILENAME) suggested by you
>> as well?
>
> Yeah, let's do it that way.

Okay, done. Attached patch fixes both the display of wrong file name and
usage of PG_AUTOCONF_FILENAME.

With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

Re: ALTER SYSTEM SET command to change postgresql.conf parameters

Robert Haas <robertmhaas@gmail.com> — 2014-01-13T19:55:50Z

On Thu, Jan 9, 2014 at 11:06 PM, Amit Kapila <amit.kapila16@gmail.com> wrote:
> On Thu, Jan 9, 2014 at 12:26 AM, Robert Haas <robertmhaas@gmail.com> wrote:
>> On Mon, Jan 6, 2014 at 11:37 PM, Amit Kapila <amit.kapila16@gmail.com> wrote:
>>> On Tue, Jan 7, 2014 at 12:52 AM, Robert Haas <robertmhaas@gmail.com> wrote:
>>>> On Mon, Jan 6, 2014 at 9:48 AM, Amit Kapila <amit.kapila16@gmail.com> wrote:
>>>>>> Couldn't we also handle this by postponing FreeConfigVariables until
>>>>>> after the if (error) block?
>>>>>
>>>>>    Wouldn't doing that way can lead to bigger memory leak, if error level
>>>>>    is ERROR. Though in current fix also it can leak memory but it will be
>>>>>    just for ErrorConfFile_save. I think some similar case can happen for
>>>>>    'pre_value' in code currently as well, that's why I have fixed it in a
>>>>>    similar way in patch.
>>>>
>>>> I was assuming that error-recovery would reset the containing memory
>>>> context, but I'm not sure what memory context we're executing in at
>>>> this point.
>>>
>>>
>>> In current code, the only time it can go to error path with elevel as
>>> ERROR is during Postmaster startup
>>> (context == PGC_POSTMASTER), at which it will anyway upgrade
>>> ERROR to FATAL, so it should not be a problem to move
>>> function FreeConfigVariables() after error block check. However
>>> in future, if someone added any more ERROR (the chances of which
>>> seems to be quite less), it can cause leak, may be thats why original
>>> code has been written that way.
>>>
>>> If you think it's better to fix by moving FreeConfigVariables() after error
>>> block check, then I can update the patch by doing so and incorporate other
>>> change (directly use PG_AUTOCONF_FILENAME) suggested by you
>>> as well?
>>
>> Yeah, let's do it that way.
>
> Okay, done. Attached patch fixes both the display of wrong file name and
> usage of PG_AUTOCONF_FILENAME.

Committed with a comment change.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company