Thread

Commits

Rename "pg_clog" directory to "pg_xact".
- 88e66d193fba 10.0 landed
Improve error reporting in pg_upgrade's file copying/linking/rewriting.
- f002ed2b8e45 10.0 cited

Renaming of pg_xlog and pg_clog

Michael Paquier <michael.paquier@gmail.com> — 2016-08-26T02:39:29Z

Hi all,

I am relaunching $subject as 10 development will begin soon. As far as
I know, there is agreement that we can do something here. Among the
different proposals I have found:
- pg_clog renamed to pg_commit_status, pg_xact or pg_commit
- pg_xlog renamed to pg_xjournal, pg_wal or pg_journal

Another idea from Stephen
(https://www.postgresql.org/message-id/20160826003715.GG4028@tamriel.snowman.net)
would be to put everything that is temporary and not WAL-logged into a
single place to facilitate the filtering work of backup tools.

A straight renaming would be a simple patch (including pg_upgrade
part), and if we actually do it for 10.0 it would be good to do it now
instead of in 3 months. I don't mind writing a patch for it.

Now, one of the things discussed as well was that we may want to still
keep pg_xlog, and soft-link to pg_journal or whatever-the-new-name is
to not break the existing tools. Personally, I'd prefer a hard break.
That would not be complicated to fix for any tool maintainers.

Thoughts?
-- 
Michael

Re: Renaming of pg_xlog and pg_clog

Bruce Momjian <bruce@momjian.us> — 2016-08-26T02:55:39Z

On Fri, Aug 26, 2016 at 11:39:29AM +0900, Michael Paquier wrote:
> Now, one of the things discussed as well was that we may want to still
> keep pg_xlog, and soft-link to pg_journal or whatever-the-new-name is
> to not break the existing tools. Personally, I'd prefer a hard break.
> That would not be complicated to fix for any tool maintainers.

I agree on a hard break, unless we get pushback from users, and even
then, they can create the symlinks themselves.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+                     Ancient Roman grave inscription +

Re: Renaming of pg_xlog and pg_clog

Fujii Masao <masao.fujii@gmail.com> — 2016-08-26T03:58:14Z

On Fri, Aug 26, 2016 at 11:55 AM, Bruce Momjian <bruce@momjian.us> wrote:
> On Fri, Aug 26, 2016 at 11:39:29AM +0900, Michael Paquier wrote:
>> Now, one of the things discussed as well was that we may want to still
>> keep pg_xlog, and soft-link to pg_journal or whatever-the-new-name is
>> to not break the existing tools. Personally, I'd prefer a hard break.
>> That would not be complicated to fix for any tool maintainers.
>
> I agree on a hard break, unless we get pushback from users, and even
> then, they can create the symlinks themselves.

I strongly prefer symlink approach not to break many existing tools
and scripts.

Regards,

-- 
Fujii Masao

Re: Renaming of pg_xlog and pg_clog

Stephen Frost <sfrost@snowman.net> — 2016-08-26T04:23:02Z

* Bruce Momjian (bruce@momjian.us) wrote:
> On Fri, Aug 26, 2016 at 11:39:29AM +0900, Michael Paquier wrote:
> > Now, one of the things discussed as well was that we may want to still
> > keep pg_xlog, and soft-link to pg_journal or whatever-the-new-name is
> > to not break the existing tools. Personally, I'd prefer a hard break.
> > That would not be complicated to fix for any tool maintainers.
> 
> I agree on a hard break, unless we get pushback from users, and even
> then, they can create the symlinks themselves.

I'm in favor of the hard break generally, but I'd like David's input on
it, to be sure we're not missing something.

Thanks!

Stephen

Re: Renaming of pg_xlog and pg_clog

Christoph Berg <myon@debian.org> — 2016-08-26T10:44:46Z

Re: Fujii Masao 2016-08-26 <CAHGQGwHK2yimfLvG_WQ1Vrq2h+CMzgv5u6OEmxr-cbJRO+WKWQ@mail.gmail.com>
> > I agree on a hard break, unless we get pushback from users, and even
> > then, they can create the symlinks themselves.
> 
> I strongly prefer symlink approach not to break many existing tools
> and scripts.

Symlinks might actually be worse than removing the directories
altogether. If your backup tool fails because the pg_xlog directory is
gone, you'll hopefully notice, but if you end up with a backup that
consists merely of a copy of a symlink named pg_xlog, you might not
notice.

Christoph

Re: Renaming of pg_xlog and pg_clog

Magnus Hagander <magnus@hagander.net> — 2016-08-26T10:48:09Z

On Fri, Aug 26, 2016 at 12:44 PM, Christoph Berg <myon@debian.org> wrote:

> Re: Fujii Masao 2016-08-26 <CAHGQGwHK2yimfLvG_WQ1Vrq2h+
> CMzgv5u6OEmxr-cbJRO+WKWQ@mail.gmail.com>
> > > I agree on a hard break, unless we get pushback from users, and even
> > > then, they can create the symlinks themselves.
> >
> > I strongly prefer symlink approach not to break many existing tools
> > and scripts.
>
> Symlinks might actually be worse than removing the directories
> altogether. If your backup tool fails because the pg_xlog directory is
> gone, you'll hopefully notice, but if you end up with a backup that
> consists merely of a copy of a symlink named pg_xlog, you might not
> notice.
>

+1. It's *much* better to cause a clean break. That way people will notice
it, and can fix it (and it will be an easy fix).

An unclean break might leave people with things that look like they work,
but don't. That's a lot more dangerous.

Same reason I'm also +1 for Stephens suggestion to put all things that
should not be in a base backup into the same directory. That may break
things now, but it will simplify things down the road. And doing it at the
same time as renaming these things makes a lot of sense, because it causes
breakage that tool-builders *have* to look at, and then they will hopefully
also notice the other change.

-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

Re: Renaming of pg_xlog and pg_clog

Simon Riggs <simon@2ndquadrant.com> — 2016-08-26T11:31:49Z

On 26 August 2016 at 04:39, Michael Paquier <michael.paquier@gmail.com> wrote:
> Hi all,
>
> I am relaunching $subject as 10 development will begin soon. As far as
> I know, there is agreement that we can do something here. Among the
> different proposals I have found:
> - pg_clog renamed to pg_commit_status, pg_xact or pg_commit
> - pg_xlog renamed to pg_xjournal, pg_wal or pg_journal

Don't mean to be a party pooper, but what discussion and agreement are
we referring to here?

If we are going to suggest doing something we really should summarize
the reason for doing it rather than assume it is self evident, cos it
certainly isn't.

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Renaming of pg_xlog and pg_clog

Michael Paquier <michael.paquier@gmail.com> — 2016-08-26T12:12:56Z

On Fri, Aug 26, 2016 at 8:31 PM, Simon Riggs <simon@2ndquadrant.com> wrote:
> On 26 August 2016 at 04:39, Michael Paquier <michael.paquier@gmail.com> wrote:
>> Hi all,
>>
>> I am relaunching $subject as 10 development will begin soon. As far as
>> I know, there is agreement that we can do something here. Among the
>> different proposals I have found:
>> - pg_clog renamed to pg_commit_status, pg_xact or pg_commit
>> - pg_xlog renamed to pg_xjournal, pg_wal or pg_journal
>
> Don't mean to be a party pooper, but what discussion and agreement are
> we referring to here?
>
> If we are going to suggest doing something we really should summarize
> the reason for doing it rather than assume it is self evident, cos it
> certainly isn't.

This thread was the previous one on the matter:
http://www.postgresql.org/message-id/CAASwCXcVGma9KgEu-ESC6u928mW67noZvnawbPUSW7R7AN9UVg@mail.gmail.com

In short, with the current names, sometimes users think that pg_xlog
and pg_clog are just logs. And so it is fine to delete them to free up
space, corrupting their cluster, because they are just *logs*.
Personally I have seen that, and based on the thread I am not the only
one.
-- 
Michael

Re: Renaming of pg_xlog and pg_clog

Devrim GÜNDÜZ <devrim@gunduz.org> — 2016-08-26T12:25:14Z

Hi,

On Fri, 2016-08-26 at 21:12 +0900, Michael Paquier wrote:
> In short, with the current names, sometimes users think that pg_xlog
> and pg_clog are just logs. And so it is fine to delete them to free up
> space, corrupting their cluster, because they are just *logs*.

...and we also have "pg_logical", that includes a "log" keyword already...

Regards,

-- 
Devrim GÜNDÜZ
EnterpriseDB: http://www.enterprisedb.com
PostgreSQL Danışmanı/Consultant, Red Hat Certified Engineer
Twitter: @DevrimGunduz , @DevrimGunduzTR

Re: Renaming of pg_xlog and pg_clog

Euler Taveira <euler@timbira.com.br> — 2016-08-26T14:36:43Z

On 26-08-2016 09:25, Devrim Gündüz wrote:
> On Fri, 2016-08-26 at 21:12 +0900, Michael Paquier wrote:
>> In short, with the current names, sometimes users think that pg_xlog
>> and pg_clog are just logs. And so it is fine to delete them to free up
>> space, corrupting their cluster, because they are just *logs*.
> 
> ...and we also have "pg_logical", that includes a "log" keyword already...
> 
"clog" and "xlog" is almost "log"; "logical" is not. I don't imagine
people confusing "logical" meaning "log".


-- 
   Euler Taveira                   Timbira - http://www.timbira.com.br/
   PostgreSQL: Consultoria, Desenvolvimento, Suporte 24x7 e Treinamento

Re: Renaming of pg_xlog and pg_clog

Tom Lane <tgl@sss.pgh.pa.us> — 2016-08-26T14:44:54Z

Euler Taveira <euler@timbira.com.br> writes:
> On 26-08-2016 09:25, Devrim Gndz wrote:
>> ...and we also have "pg_logical", that includes a "log" keyword already...

> "clog" and "xlog" is almost "log"; "logical" is not. I don't imagine
> people confusing "logical" meaning "log".

Well, I dunno; people with a weak grasp of English might have an issue
there.  But I never liked that directory name anyway; my problem with
it is I read "pg_logical" and think "logical what?".  Naming things using
a disconnected adjective is not good, especially one with as many
potential applications as that one has.  If we're up for renaming things
under PGDATA, that one is high on my hit list.

			regards, tom lane

Re: Renaming of pg_xlog and pg_clog

Stephen Frost <sfrost@snowman.net> — 2016-08-26T14:50:33Z

* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> Euler Taveira <euler@timbira.com.br> writes:
> > On 26-08-2016 09:25, Devrim Gündüz wrote:
> >> ...and we also have "pg_logical", that includes a "log" keyword already...
> 
> > "clog" and "xlog" is almost "log"; "logical" is not. I don't imagine
> > people confusing "logical" meaning "log".
> 
> Well, I dunno; people with a weak grasp of English might have an issue
> there.  But I never liked that directory name anyway; my problem with
> it is I read "pg_logical" and think "logical what?".  Naming things using
> a disconnected adjective is not good, especially one with as many
> potential applications as that one has.  If we're up for renaming things
> under PGDATA, that one is high on my hit list.

iirc, pg_logical also has both temporary and non-temporary data in it
too.  That complicates things for backup utilities that are trying to
exclude all temporary files.  If we actually move all the temp files
into their own directory (or tree), then we're changing what's in
pg_logical anyway, so renaming it seems like a good idea.

Thanks!

Stephen

Re: Renaming of pg_xlog and pg_clog

Joshua D. Drake <jd@commandprompt.com> — 2016-08-26T15:11:12Z

On 08/26/2016 03:48 AM, Magnus Hagander wrote:

>
> Same reason I'm also +1 for Stephens suggestion to put all things that
> should not be in a base backup into the same directory. That may break
> things now, but it will simplify things down the road. And doing it at
> the same time as renaming these things makes a lot of sense, because it
> causes breakage that tool-builders *have* to look at, and then they will
> hopefully also notice the other change.

If this is done this fall, developers will have at least a year to fix 
their utilities.

JD




-- 
Command Prompt, Inc.                  http://the.postgres.company/
                         +1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Everyone appreciates your honesty, until you are honest with them.
Unless otherwise stated, opinions are my own.

Re: Renaming of pg_xlog and pg_clog

Joshua D. Drake <jd@commandprompt.com> — 2016-08-26T15:13:04Z

On 08/25/2016 07:39 PM, Michael Paquier wrote:
> Hi all,
>
> I am relaunching $subject as 10 development will begin soon. As far as
> I know, there is agreement that we can do something here. Among the
> different proposals I have found:
> - pg_clog renamed to pg_commit_status, pg_xact or pg_commit
> - pg_xlog renamed to pg_xjournal, pg_wal or pg_journal

I think the use of the pg_ prefix is redundant. Just a directory called: 
wal will do (for example).

In reference to pg_xlog specifically, it should be wal. It is the Write 
Ahead Log, not the Journal (although I recognize they can be 
synonymous). All the documentation talks about Write Ahead Log.

Sincerely,

JD

-- 
Command Prompt, Inc.                  http://the.postgres.company/
                         +1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Everyone appreciates your honesty, until you are honest with them.
Unless otherwise stated, opinions are my own.

Re: Renaming of pg_xlog and pg_clog

Magnus Hagander <magnus@hagander.net> — 2016-08-26T15:14:36Z

On Aug 26, 2016 5:13 PM, "Joshua D. Drake" <jd@commandprompt.com> wrote:
>
> On 08/25/2016 07:39 PM, Michael Paquier wrote:
>>
>> Hi all,
>>
>> I am relaunching $subject as 10 development will begin soon. As far as
>> I know, there is agreement that we can do something here. Among the
>> different proposals I have found:
>> - pg_clog renamed to pg_commit_status, pg_xact or pg_commit
>> - pg_xlog renamed to pg_xjournal, pg_wal or pg_journal
>
>
> I think the use of the pg_ prefix is redundant. Just a directory called:
wal will do (for example).
>
> In reference to pg_xlog specifically, it should be wal. It is the Write
Ahead Log, not the Journal (although I recognize they can be synonymous).
All the documentation talks about Write Ahead Log.
>

Yes, let's avoid inventing a *third* name for it, please. It's already bad
enough that we have both wal and xlog.

/Magnus

Re: Renaming of pg_xlog and pg_clog

Peter Eisentraut <peter.eisentraut@2ndquadrant.com> — 2016-08-26T15:53:21Z

On 8/25/16 10:39 PM, Michael Paquier wrote:
> I am relaunching $subject as 10 development will begin soon. As far as
> I know, there is agreement that we can do something here. Among the
> different proposals I have found:
> - pg_clog renamed to pg_commit_status, pg_xact or pg_commit
> - pg_xlog renamed to pg_xjournal, pg_wal or pg_journal

If we're going to do some renaming, then I suggest we do a
mini-file-system structure under $PGDATA, like

$PGDATA/etc
$PGDATA/log
$PGDATA/run (lock files etc.)
$PGDATA/tmp
$PGDATA/var

The names of all the things under "var" could still be refined, but it's
much less likely that users will confuse data with configuration or
plain logs under that scheme.

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Renaming of pg_xlog and pg_clog

Magnus Hagander <magnus@hagander.net> — 2016-08-26T15:58:45Z

On Aug 26, 2016 5:54 PM, "Peter Eisentraut" <
peter.eisentraut@2ndquadrant.com> wrote:
>
> On 8/25/16 10:39 PM, Michael Paquier wrote:
> > I am relaunching $subject as 10 development will begin soon. As far as
> > I know, there is agreement that we can do something here. Among the
> > different proposals I have found:
> > - pg_clog renamed to pg_commit_status, pg_xact or pg_commit
> > - pg_xlog renamed to pg_xjournal, pg_wal or pg_journal
>
> If we're going to do some renaming, then I suggest we do a
> mini-file-system structure under $PGDATA, like
>
> $PGDATA/etc
> $PGDATA/log
> $PGDATA/run (lock files etc.)
> $PGDATA/tmp
> $PGDATA/var
>
> The names of all the things under "var" could still be refined, but it's
> much less likely that users will confuse data with configuration or
> plain logs under that scheme

Interesting idea. I worry a bit that this might encourage distributions to
split it up into different places though, and I'm not sure we want to
encourage that..

/Magnus

Re: Renaming of pg_xlog and pg_clog

Tom Lane <tgl@sss.pgh.pa.us> — 2016-08-26T16:28:41Z

Magnus Hagander <magnus@hagander.net> writes:
> On Aug 26, 2016 5:54 PM, "Peter Eisentraut" <
> peter.eisentraut@2ndquadrant.com> wrote:
>> If we're going to do some renaming, then I suggest we do a
>> mini-file-system structure under $PGDATA, like
>> 
>> $PGDATA/etc
>> $PGDATA/log
>> $PGDATA/run (lock files etc.)
>> $PGDATA/tmp
>> $PGDATA/var
>> 
>> The names of all the things under "var" could still be refined, but it's
>> much less likely that users will confuse data with configuration or
>> plain logs under that scheme

> Interesting idea. I worry a bit that this might encourage distributions to
> split it up into different places though, and I'm not sure we want to
> encourage that..

Yeah, I'm afraid that these names are not as well standardized as Peter
probably wishes they were.  Also, I'd just as soon not move/rename things
that don't really need it.  If, for example, we decide to move
postgresql.conf to etc/postgresql.conf, that is going to break a metric
ton of stuff that doesn't need to get broken AFAICS.

			regards, tom lane

Re: Renaming of pg_xlog and pg_clog

Joshua D. Drake <jd@commandprompt.com> — 2016-08-26T20:02:55Z

On 08/26/2016 09:28 AM, Tom Lane wrote:
> Magnus Hagander <magnus@hagander.net> writes:
>> On Aug 26, 2016 5:54 PM, "Peter Eisentraut" <
>> peter.eisentraut@2ndquadrant.com> wrote:
>>> If we're going to do some renaming, then I suggest we do a
>>> mini-file-system structure under $PGDATA, like
>>>
>>> $PGDATA/etc
>>> $PGDATA/log
>>> $PGDATA/run (lock files etc.)
>>> $PGDATA/tmp
>>> $PGDATA/var
>>>
>>> The names of all the things under "var" could still be refined, but it's
>>> much less likely that users will confuse data with configuration or
>>> plain logs under that scheme
>
>> Interesting idea. I worry a bit that this might encourage distributions to
>> split it up into different places though, and I'm not sure we want to
>> encourage that..
>
> Yeah, I'm afraid that these names are not as well standardized as Peter
> probably wishes they were.  Also, I'd just as soon not move/rename things
> that don't really need it.  If, for example, we decide to move
> postgresql.conf to etc/postgresql.conf, that is going to break a metric
> ton of stuff that doesn't need to get broken AFAICS.

I am not so sure that is accurate. Yes, Windows is an outlying but any 
Unix person is going to easily understand etc log run. Further as Linux 
is by far our most run platform (outside of possibly Windows) it will 
feel right at home for the largest growing user base. Personally I 
really like the idea of:

$PGDATA/etc/postgresql.conf
$PGDATA/log/Thu.log
$PGDATA/run/postmaster.pid
$PGDATA/var/base/16758

Although... wouldn't run be under var?

JD


>
> 			regards, tom lane
>
>


-- 
Command Prompt, Inc.                  http://the.postgres.company/
                         +1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Everyone appreciates your honesty, until you are honest with them.

Re: Renaming of pg_xlog and pg_clog

Simon Riggs <simon@2ndquadrant.com> — 2016-08-26T20:18:22Z

On 26 August 2016 at 18:28, Tom Lane <tgl@sss.pgh.pa.us> wrote:

> Also, I'd just as soon not move/rename things that don't really need it.

+1

Let's leave everything exactly as it is now... but put a small README
in each directory to explain why files in it shouldn't be deleted to
make space.

That helps the few people who made such mistakes, but doesn't cause
massive change as a result.

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Renaming of pg_xlog and pg_clog

Peter Eisentraut <peter.eisentraut@2ndquadrant.com> — 2016-08-26T21:06:45Z

On 8/26/16 11:58 AM, Magnus Hagander wrote:
> $PGDATA/etc
>> $PGDATA/log
>> $PGDATA/run (lock files etc.)
>> $PGDATA/tmp
>> $PGDATA/var
>>
>> The names of all the things under "var" could still be refined, but it's
>> much less likely that users will confuse data with configuration or
>> plain logs under that scheme
> 
> Interesting idea. I worry a bit that this might encourage distributions
> to split it up into different places though, and I'm not sure we want to
> encourage that..

They already do this.  This would just formalize where we think
appropriate boundaries between the pieces are.

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Renaming of pg_xlog and pg_clog

Andres Freund <andres@anarazel.de> — 2016-08-26T21:08:30Z

On 2016-08-26 11:53:21 -0400, Peter Eisentraut wrote:
> On 8/25/16 10:39 PM, Michael Paquier wrote:
> > I am relaunching $subject as 10 development will begin soon. As far as
> > I know, there is agreement that we can do something here. Among the
> > different proposals I have found:
> > - pg_clog renamed to pg_commit_status, pg_xact or pg_commit
> > - pg_xlog renamed to pg_xjournal, pg_wal or pg_journal
> 
> If we're going to do some renaming, then I suggest we do a
> mini-file-system structure under $PGDATA, like
> 
> $PGDATA/etc
> $PGDATA/log
> $PGDATA/run (lock files etc.)
> $PGDATA/tmp
> $PGDATA/var
> 
> The names of all the things under "var" could still be refined, but it's
> much less likely that users will confuse data with configuration or
> plain logs under that scheme.

Splitting of ephemeral data seems to have a benefit, the rest seems more
like rather noisy busywork to me.

Re: Renaming of pg_xlog and pg_clog

Peter Eisentraut <peter.eisentraut@2ndquadrant.com> — 2016-08-26T21:11:00Z

On 8/26/16 12:28 PM, Tom Lane wrote:
> Also, I'd just as soon not move/rename things
> that don't really need it.

I'm just as happy with not changing anything.  But if we're going to
rename stuff, let's at least think about something slightly more
comprehensive.  Any rename is going to break a bunch of stuff.  But if
we break it in a way that reduces the need for future discussion or
changes, it would at least be a small win in the long run.

> If, for example, we decide to move
> postgresql.conf to etc/postgresql.conf, that is going to break a metric
> ton of stuff that doesn't need to get broken AFAICS.

Sure, I'd be content with leaving those in the top-level instead of say
"etc".

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Renaming of pg_xlog and pg_clog

Peter Eisentraut <peter.eisentraut@2ndquadrant.com> — 2016-08-26T21:15:41Z

On 8/26/16 4:02 PM, Joshua D. Drake wrote:
> Although... wouldn't run be under var?

Traditionally yes, but things are changing in this area, if you consider
the top-level file system of a modern Linux distribution.  One reason is
that "run" is/can be blown away at reboot.  This wouldn't be an entirely
useless feature for postgres: Can you tell otherwise which of the
various pid/lock/opts files you can delete if you have killed the
processes and want to eliminate any left-over state?  Is postmaster.opts
a configuration file or a state file?

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Renaming of pg_xlog and pg_clog

Andres Freund <andres@anarazel.de> — 2016-08-26T21:20:15Z

On 2016-08-26 17:11:00 -0400, Peter Eisentraut wrote:
> On 8/26/16 12:28 PM, Tom Lane wrote:
> > Also, I'd just as soon not move/rename things
> > that don't really need it.
>
> I'm just as happy with not changing anything.  But if we're going to
> rename stuff, let's at least think about something slightly more
> comprehensive.  Any rename is going to break a bunch of stuff.  But if
> we break it in a way that reduces the need for future discussion or
> changes, it would at least be a small win in the long run.

I do think there's an order of magnitude between the impact between
moving some and moving everything. And that's going to impact
cost/benefit calculations.

Moving e.g. all ephemeral files into a (possibly configurable) directory
is going to hardly impact anyone.  Renaming pg_logical into something
different (FWIW, it was originally named differently...) will hopefully
impact nobody, excepting some out of date file exclusion lists possibly.

But moving config files, and even pg_xlog (which we document to be
symlinkable somewhere else) imo is different.

The other thing is that the likelihood of getting anywhere by doing
radical one-off redesigns is approximately 0.

Re: Renaming of pg_xlog and pg_clog

Peter Eisentraut <peter.eisentraut@2ndquadrant.com> — 2016-08-26T21:31:14Z

On 8/26/16 5:20 PM, Andres Freund wrote:
> I do think there's an order of magnitude between the impact between
> moving some and moving everything. And that's going to impact
> cost/benefit calculations.
> 
> Moving e.g. all ephemeral files into a (possibly configurable) directory
> is going to hardly impact anyone.  Renaming pg_logical into something
> different (FWIW, it was originally named differently...) will hopefully
> impact nobody, excepting some out of date file exclusion lists possibly.
> 
> But moving config files, and even pg_xlog (which we document to be
> symlinkable somewhere else) imo is different.

I agree with all that.  But the subject line is specifically about
moving pg_xlog.  So if your opinion is that we shouldn't move pg_xlog,
then that is noted.  But if we were to move it, we can think about a
good place to move it to.

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Renaming of pg_xlog and pg_clog

Andres Freund <andres@anarazel.de> — 2016-08-26T21:34:06Z

On 2016-08-26 17:31:14 -0400, Peter Eisentraut wrote:
> I agree with all that.  But the subject line is specifically about
> moving pg_xlog.  So if your opinion is that we shouldn't move pg_xlog,
> then that is noted.  But if we were to move it, we can think about a
> good place to move it to.

I think it's probably worth moving pg_xlog, because the benefit also
includes preventing a few users from shooting themselves somewhere
vital. That's imo much less the case for some of the other moves.  But I
still don't think think a largescale reorganization is a good idea,
it'll just stall and nothing will happen.

Re: Renaming of pg_xlog and pg_clog

Michael Paquier <michael.paquier@gmail.com> — 2016-08-27T08:33:14Z

On Sat, Aug 27, 2016 at 6:34 AM, Andres Freund <andres@anarazel.de> wrote:
> On 2016-08-26 17:31:14 -0400, Peter Eisentraut wrote:
>> I agree with all that.  But the subject line is specifically about
>> moving pg_xlog.  So if your opinion is that we shouldn't move pg_xlog,
>> then that is noted.  But if we were to move it, we can think about a
>> good place to move it to.
>
> I think it's probably worth moving pg_xlog, because the benefit also
> includes preventing a few users from shooting themselves somewhere
> vital. That's imo much less the case for some of the other moves.  But I
> still don't think think a largescale reorganization is a good idea,
> it'll just stall and nothing will happen.

OK, so let's focus only on the renaming mentioned in $subject. So far
as I can see on this thread, here are the opinions of people who
clearly gave one:
- Rename them, hard break is OK: Michael P, Bruce, Stephen (depends on
David's input),  Magnus
- Rename them, hard break not OK: Fujii-san (perhaps do nothing?)
- Do nothing: Simon (add a README), Tom, Peter E

As far as I can see, there is a consensus to not rename pg_xlog to
pg_journal and avoid using a third meaning, but instead use pg_wal. I
guess that now the other renaming would be pg_clog -> pg_xact. Other
opinions? Forgot you here?
-- 
Michael

Re: Renaming of pg_xlog and pg_clog

Gavin Flower <gavinflower@archidevsys.co.nz> — 2016-08-27T09:13:00Z

On 27/08/16 20:33, Michael Paquier wrote:
> On Sat, Aug 27, 2016 at 6:34 AM, Andres Freund <andres@anarazel.de> wrote:
>> On 2016-08-26 17:31:14 -0400, Peter Eisentraut wrote:
>>> I agree with all that.  But the subject line is specifically about
>>> moving pg_xlog.  So if your opinion is that we shouldn't move pg_xlog,
>>> then that is noted.  But if we were to move it, we can think about a
>>> good place to move it to.
>> I think it's probably worth moving pg_xlog, because the benefit also
>> includes preventing a few users from shooting themselves somewhere
>> vital. That's imo much less the case for some of the other moves.  But I
>> still don't think think a largescale reorganization is a good idea,
>> it'll just stall and nothing will happen.
> OK, so let's focus only on the renaming mentioned in $subject. So far
> as I can see on this thread, here are the opinions of people who
> clearly gave one:
> - Rename them, hard break is OK: Michael P, Bruce, Stephen (depends on
> David's input),  Magnus
> - Rename them, hard break not OK: Fujii-san (perhaps do nothing?)
> - Do nothing: Simon (add a README), Tom, Peter E
>
> As far as I can see, there is a consensus to not rename pg_xlog to
> pg_journal and avoid using a third meaning, but instead use pg_wal. I
> guess that now the other renaming would be pg_clog -> pg_xact. Other
> opinions? Forgot you here?

I think if there are going to be things in pg that break software - for 
good reasons, like making future usage easier at the cost an initial 
sharp pain - then to do so in version '10.0.0' is very appropriate!  IMHO

And better to do so in 10.0.0 (especially if closely related), rather 
than 10.1.0 (or whatever the next version after that is named).  So, if 
other things might cause breakages, do so IN 10.0.0 - rather than hold 
back - assuming that there won't be hundreds or more major breakages!!!


Cheers,
Gavin

Re: Renaming of pg_xlog and pg_clog

Tom Lane <tgl@sss.pgh.pa.us> — 2016-08-27T16:50:08Z

Michael Paquier <michael.paquier@gmail.com> writes:
> OK, so let's focus only on the renaming mentioned in $subject. So far
> as I can see on this thread, here are the opinions of people who
> clearly gave one:
> - Rename them, hard break is OK: Michael P, Bruce, Stephen (depends on
> David's input),  Magnus
> - Rename them, hard break not OK: Fujii-san (perhaps do nothing?)
> - Do nothing: Simon (add a README), Tom, Peter E

Hm, if you read me as voting against renaming pg_xlog, that wasn't
the conclusion I meant to convey.  I'm against moving/renaming the
configuration files, because I think that will break a lot of users'
scripts and habits without really buying much.  But I'm for consolidating
all the files that should not be copied by backup tools into one
subdirectory, and I think that while we're doing that it would be sensible
to rename pg_xlog and pg_clog to something that doesn't sound like it's
scratch data.  I'm on the fence about whether pg_logical ought to get
renamed.

> As far as I can see, there is a consensus to not rename pg_xlog to
> pg_journal and avoid using a third meaning, but instead use pg_wal.

Yeah, +1 for pg_wal, we do not need yet another name for that.

> I guess that now the other renaming would be pg_clog -> pg_xact.

We already have pg_subtrans, so it seems like pg_trans is an obvious
suggestion.  I'm not sure whether the other precedent of pg_multixact
is a stronger one than that.

			regards, tom lane

Re: Renaming of pg_xlog and pg_clog

Alvaro Herrera <alvherre@2ndquadrant.com> — 2016-08-27T17:56:08Z

Michael Paquier wrote:
> On Sat, Aug 27, 2016 at 6:34 AM, Andres Freund <andres@anarazel.de> wrote:
> > On 2016-08-26 17:31:14 -0400, Peter Eisentraut wrote:
> >> I agree with all that.  But the subject line is specifically about
> >> moving pg_xlog.  So if your opinion is that we shouldn't move pg_xlog,
> >> then that is noted.  But if we were to move it, we can think about a
> >> good place to move it to.
> >
> > I think it's probably worth moving pg_xlog, because the benefit also
> > includes preventing a few users from shooting themselves somewhere
> > vital. That's imo much less the case for some of the other moves.  But I
> > still don't think think a largescale reorganization is a good idea,
> > it'll just stall and nothing will happen.
> 
> OK, so let's focus only on the renaming mentioned in $subject. So far
> as I can see on this thread, here are the opinions of people who
> clearly gave one:
> - Rename them, hard break is OK: Michael P, Bruce, Stephen (depends on
> David's input),  Magnus
> - Rename them, hard break not OK: Fujii-san (perhaps do nothing?)
> - Do nothing: Simon (add a README), Tom, Peter E

I'm for renaming too, but I'd go with Peter E's suggestion: move pg_xlog
to something like $PGDATA/var/wal or $PGDATA/srv/wal or something like that.

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Renaming of pg_xlog and pg_clog

Tom Lane <tgl@sss.pgh.pa.us> — 2016-08-27T18:11:33Z

Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> I'm for renaming too, but I'd go with Peter E's suggestion: move pg_xlog
> to something like $PGDATA/var/wal or $PGDATA/srv/wal or something like that.

I think that would make sense if we were to relocate *everything* under
PGDATA into some FHS-like subdirectory structure.  But I'm against moving
the config files for previously-stated reasons, and I doubt it makes sense
to adopt an FHS-like structure only in part.

			regards, tom lane

Re: Renaming of pg_xlog and pg_clog

Jim Nasby <jim.nasby@bluetreble.com> — 2016-08-29T00:45:47Z

On 8/26/16 4:08 PM, Andres Freund wrote:
> Splitting of ephemeral data seems to have a benefit, the rest seems more
> like rather noisy busywork to me.

People accidentally blowing away pg_clog or pg_xlog is a pretty common 
occurrence, and I don't think there's all that many tools that reference 
them. I think it's well worth renaming them.

I actually like the idea of re-organizing everything too. It would be 
easy to include a tool that puts symlinks in place to all the original 
locations if you need that.
-- 
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com
855-TREBLE2 (855-873-2532)   mobile: 512-569-9461

Re: Renaming of pg_xlog and pg_clog

Craig Ringer <craig.ringer@2ndquadrant.com> — 2016-08-29T05:36:43Z

On 29 Aug 2016 12:10 PM, "Jim Nasby" <Jim.Nasby@bluetreble.com> wrote:
>
> On 8/26/16 4:08 PM, Andres Freund wrote:
>>
>> Splitting of ephemeral data seems to have a benefit, the rest seems more
>> like rather noisy busywork to me.
>
>
> People accidentally blowing away pg_clog or pg_xlog is a pretty common
occurrence, and I don't think there's all that many tools that reference
them. I think it's well worth renaming them.

Yeah. I've seen it in BIG production users who really should have known
better.

People won't see a README in amongst 5000 xlog segments while freaking out
about the sever being down.

I don't care if it comes as part of some greater reorg or not but I'll be
really annoyed if scope creep lands up killing the original proposal to
just rename these dirs. I think that a simple rename should be done first.
Then if some greater reorg is to be done it can be done shortly after. The
only people that'll upset are folks tracking early 10.0 dev and they'll be
aware it's coming.

Re: Renaming of pg_xlog and pg_clog

Michael Paquier <michael.paquier@gmail.com> — 2016-08-29T06:30:17Z

On Mon, Aug 29, 2016 at 2:36 PM, Craig Ringer
<craig.ringer@2ndquadrant.com> wrote:
> I don't care if it comes as part of some greater reorg or not but I'll be
> really annoyed if scope creep lands up killing the original proposal to just
> rename these dirs. I think that a simple rename should be done first. Then
> if some greater reorg is to be done it can be done shortly after. The only
> people that'll upset are folks tracking early 10.0 dev and they'll be aware
> it's coming.

Okay, so let's do it. Attached are two patches:
- 0001 renames pg_clog to pg_trans. I have let clog.c with its current
name, as well as its structures. That's the mechanical patch, the ony
interesting part being in pg_upgrade.
- 0002 renames pg_xlog to pg_wal.
-- 
Michael

Re: Renaming of pg_xlog and pg_clog

Magnus Hagander <magnus@hagander.net> — 2016-08-29T07:04:39Z

On Mon, Aug 29, 2016 at 2:45 AM, Jim Nasby <Jim.Nasby@bluetreble.com> wrote:

> On 8/26/16 4:08 PM, Andres Freund wrote:
>
>> Splitting of ephemeral data seems to have a benefit, the rest seems more
>> like rather noisy busywork to me.
>>
>
> People accidentally blowing away pg_clog or pg_xlog is a pretty common
> occurrence, and I don't think there's all that many tools that reference
> them. I think it's well worth renaming them.
>

Pretty sure every single backup tool or script out there is referencing
pg_xlog. If it's not, then it's broken...

pg_clog is a different story of course.

-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

Re: Renaming of pg_xlog and pg_clog

Craig Ringer <craig.ringer@2ndquadrant.com> — 2016-08-29T08:28:07Z

On 29 August 2016 at 14:30, Michael Paquier <michael.paquier@gmail.com> wrote:
> On Mon, Aug 29, 2016 at 2:36 PM, Craig Ringer
> <craig.ringer@2ndquadrant.com> wrote:
>> I don't care if it comes as part of some greater reorg or not but I'll be
>> really annoyed if scope creep lands up killing the original proposal to just
>> rename these dirs. I think that a simple rename should be done first. Then
>> if some greater reorg is to be done it can be done shortly after. The only
>> people that'll upset are folks tracking early 10.0 dev and they'll be aware
>> it's coming.
>
> Okay, so let's do it. Attached are two patches:
> - 0001 renames pg_clog to pg_trans. I have let clog.c with its current
> name, as well as its structures. That's the mechanical patch, the ony
> interesting part being in pg_upgrade.
> - 0002 renames pg_xlog to pg_wal.

Is there any expectation that a 10.0 pg_basebackup should work on a
9.x server, or fail to work gracefully? There doesn't look to be any
version specific handling of the rename there.

Otherwise looks good.

Doesn't upset 'make check' or the TAP recovery suite.

Works:

src/test/regress/tmp_check/data $ ls
base    pg_commit_ts  pg_hba.conf    pg_logical    pg_notify
pg_serial     pg_stat      pg_subtrans  pg_trans     PG_VERSION
postgresql.auto.conf  postmaster.opts
global  pg_dynshmem   pg_ident.conf  pg_multixact  pg_replslot
pg_snapshots  pg_stat_tmp  pg_tblspc    pg_twophase  pg_wal
postgresql.conf

It leaves pg_xlogfilename etc with original names as it IMO should.
There's no need to break more than we have to and it's still the xlog.

The documentation on backup/restore might benefit from a note saying
that pg_wal was named pg_xlog prior to 10.0 so tools intended to work
on older versions should check PG_VERSION. Though in practice most
people who write new tools will target 10.0+ and people maintaining
older tools will know, so it's not a big deal.

I don't know if the renaming in XLogFileRead of XLOG_FROM_PG_XLOG =>
XLOG_FROM_PG_WAL  is really necessary, but tend to think it's good
since that define explicitly refers to the directory name, not
transaction logs in general.

The patch does not update the translations. I wonder if it's worth
doing so to save translators the hassle by sed'ing the following
lines:

-                                        errhint("The database server
will regularly poll the pg_xlog subdirectory to check for files placed
there.")));
+                                        errhint("The database server
will regularly poll the pg_wal subdirectory to check for files placed
there.")));

-                                (errmsg("could not open directory
\"%s\": %m", "pg_xlog")));
+                                (errmsg("could not open directory
\"%s\": %m", "pg_wal")));

with

    sed -i 's/\<pg_xlog\>/pg_wal/g' src/backend/po/*.po

?

src/backend/access/transam/README should probably have a note about the rename.

Looks like changes in pg_upgrade for clog are a bit more than the
mechanical changes elsewhere, but seem sensible to me. I haven't yet
done a test pg_upgrade run.

-- 
-- 
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Renaming of pg_xlog and pg_clog

Petr Jelinek <petr@2ndquadrant.com> — 2016-08-29T09:37:06Z

On 27/08/16 18:50, Tom Lane wrote:
> Michael Paquier <michael.paquier@gmail.com> writes:
>> OK, so let's focus only on the renaming mentioned in $subject. So far
>> as I can see on this thread, here are the opinions of people who
>> clearly gave one:
>> - Rename them, hard break is OK: Michael P, Bruce, Stephen (depends on
>> David's input),  Magnus
>> - Rename them, hard break not OK: Fujii-san (perhaps do nothing?)
>> - Do nothing: Simon (add a README), Tom, Peter E
>
> I'm against moving/renaming the
> configuration files, because I think that will break a lot of users'
> scripts and habits without really buying much.

I don't agree with this part, mainly because there is significant number 
of installations (everything that uses debian/ubuntu scripts) that 
already don't have configuration files inside the data directory.

On the other matters:
+1 for renaming pg_xlog to pg_wal and pg_clog to pg_xact/trans (don't 
care really which one)

And +1 for renaming pg_logical to something more reasonable.

-- 
   Petr Jelinek                  http://www.2ndQuadrant.com/
   PostgreSQL Development, 24x7 Support, Training & Services

Re: Renaming of pg_xlog and pg_clog

Michael Paquier <michael.paquier@gmail.com> — 2016-08-29T12:28:35Z

On Mon, Aug 29, 2016 at 5:28 PM, Craig Ringer
<craig.ringer@2ndquadrant.com> wrote:
> On 29 August 2016 at 14:30, Michael Paquier <michael.paquier@gmail.com> wrote:
>> On Mon, Aug 29, 2016 at 2:36 PM, Craig Ringer
>> <craig.ringer@2ndquadrant.com> wrote:
>>> I don't care if it comes as part of some greater reorg or not but I'll be
>>> really annoyed if scope creep lands up killing the original proposal to just
>>> rename these dirs. I think that a simple rename should be done first. Then
>>> if some greater reorg is to be done it can be done shortly after. The only
>>> people that'll upset are folks tracking early 10.0 dev and they'll be aware
>>> it's coming.
>>
>> Okay, so let's do it. Attached are two patches:
>> - 0001 renames pg_clog to pg_trans. I have let clog.c with its current
>> name, as well as its structures. That's the mechanical patch, the ony
>> interesting part being in pg_upgrade.
>> - 0002 renames pg_xlog to pg_wal.
>
> Is there any expectation that a 10.0 pg_basebackup should work on a
> 9.x server, or fail to work gracefully? There doesn't look to be any
> version specific handling of the rename there.

Oops. Per the docs:
pg_basebackup works with servers of the same or an older major
version, down to 9.1. However, WAL streaming mode (-X stream) only
works with server version 9.3 and later, and tar format mode
(--format=tar) of the current version only works with server version
9.5 or later.

So we need to do a bit better than what the patch proposes, but that's
actually just tweaking the things with pg_xlog/pg_wal depending on the
version of the target server.

> The patch does not update the translations. I wonder if it's worth
> doing so to save translators the hassle by sed'ing the following
> lines:
>     sed -i 's/\<pg_xlog\>/pg_wal/g' src/backend/po/*.po
> ?

Yes I was wondering about that... But concluded that normally the
translation updates would just do it. It is not complicated to update
if need be anyway.

> src/backend/access/transam/README should probably have a note about the rename.

Good idea.

> Looks like changes in pg_upgrade for clog are a bit more than the
> mechanical changes elsewhere, but seem sensible to me. I haven't yet
> done a test pg_upgrade run.

I have tested that FWIW using 10devel -> 10devel, 9.5 -> 10devel, 9.4
-> 10devel. There are versions of 10devel using pg_xlog and others
pg_wal if this patch is introduced. I am aware of the fact that
pg_upgrade supports upgrades for the same major version though it
seems to me that we would not want to support this scenario, which is
why it would be good to get this patch at the beginning of the dev
cycle.
-- 
Michael

Re: Renaming of pg_xlog and pg_clog

Craig Ringer <craig.ringer@2ndquadrant.com> — 2016-08-29T12:39:05Z

On 29 August 2016 at 20:28, Michael Paquier <michael.paquier@gmail.com> wrote:
> On Mon, Aug 29, 2016 at 5:28 PM, Craig Ringer
> <craig.ringer@2ndquadrant.com> wrote:
>> On 29 August 2016 at 14:30, Michael Paquier <michael.paquier@gmail.com> wrote:
>>> On Mon, Aug 29, 2016 at 2:36 PM, Craig Ringer
>>> <craig.ringer@2ndquadrant.com> wrote:
>>>> I don't care if it comes as part of some greater reorg or not but I'll be
>>>> really annoyed if scope creep lands up killing the original proposal to just
>>>> rename these dirs. I think that a simple rename should be done first. Then
>>>> if some greater reorg is to be done it can be done shortly after. The only
>>>> people that'll upset are folks tracking early 10.0 dev and they'll be aware
>>>> it's coming.
>>>
>>> Okay, so let's do it. Attached are two patches:
>>> - 0001 renames pg_clog to pg_trans. I have let clog.c with its current
>>> name, as well as its structures. That's the mechanical patch, the ony
>>> interesting part being in pg_upgrade.
>>> - 0002 renames pg_xlog to pg_wal.
>>
>> Is there any expectation that a 10.0 pg_basebackup should work on a
>> 9.x server, or fail to work gracefully? There doesn't look to be any
>> version specific handling of the rename there.
>
> Oops. Per the docs:
> pg_basebackup works with servers of the same or an older major
> version, down to 9.1. However, WAL streaming mode (-X stream) only
> works with server version 9.3 and later, and tar format mode
> (--format=tar) of the current version only works with server version
> 9.5 or later.
>
> So we need to do a bit better than what the patch proposes, but that's
> actually just tweaking the things with pg_xlog/pg_wal depending on the
> version of the target server.

Cool. I'll mark as waiting on author pending that.

It'll be good to get this footgun put away.

-- 
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Renaming of pg_xlog and pg_clog

Daniel Verite <daniel@manitou-mail.org> — 2016-08-29T13:42:31Z

	Craig Ringer wrote:

> People won't see a README in amongst 5000 xlog segments while
> freaking out about the sever being down.

Maybe they're more likely to google "pg_xlog".
BTW, renaming it will not help with respect to that, as
there's a pretty good corpus on knowledge linked to that
particular keyword.

The first google results of "pg_xlog" are, for me:

- Solving pg_xlog out of disk space problem on Postgres 
- PostgreSQL: Documentation: 9.1: WAL Internals
- Why is my pg_xlog directory so huge? - PostgreSQL
- Database Soup: Don't delete pg_xlog
...

That's spot-on. For each person deleting stuff in pg_xlog and then
regretting it, how many look it up in the above results and avoid
making the mistake? Will they find comparable results if the
directory is "wal" ?

Aside from that, we might also question how much of the excuse
"pg_xlog looked like it was just deletable logs" is a lie made up
after the fact, because anybody wrecking a database is not against
deflecting a bit of the blame to the software, that's human.

The truth being that they took the gamble that postgres
will somehow recover from the deletion, at the risk of possibly
loosing the latest transactions. That doesn't necessarily look
so bad when current transactions are failing anyway for lack of disk
space, users are yelling at you, and you're frantically searching for
anything to delete in the FS to come back online quickly.
Personally I'm quite skeptical of the *name* of the directory
playing that much of a role in this scenario.

Best regards,
-- 
Daniel Vérité
PostgreSQL-powered mailer: http://www.manitou-mail.org
Twitter: @DanielVerite

Re: Renaming of pg_xlog and pg_clog

Joshua D. Drake <jd@commandprompt.com> — 2016-08-29T14:48:13Z

On 08/27/2016 11:11 AM, Tom Lane wrote:
> Alvaro Herrera <alvherre@2ndquadrant.com> writes:
>> I'm for renaming too, but I'd go with Peter E's suggestion: move pg_xlog
>> to something like $PGDATA/var/wal or $PGDATA/srv/wal or something like that.
>
> I think that would make sense if we were to relocate *everything* under
> PGDATA into some FHS-like subdirectory structure.  But I'm against moving
> the config files for previously-stated reasons, and I doubt it makes sense
> to adopt an FHS-like structure only in part.

I think that is a very reasonable suggestion.

Also as a note to the idea that we make break things for external user 
space; the next version being v10 is the exact time to do that.

JD

>
> 			regards, tom lane
>
>


-- 
Command Prompt, Inc.                  http://the.postgres.company/
                         +1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Everyone appreciates your honesty, until you are honest with them.

Re: Renaming of pg_xlog and pg_clog

Joshua D. Drake <jd@commandprompt.com> — 2016-08-29T15:00:13Z

On 08/29/2016 12:04 AM, Magnus Hagander wrote:
> On Mon, Aug 29, 2016 at 2:45 AM, Jim Nasby <Jim.Nasby@bluetreble.com
> <mailto:Jim.Nasby@bluetreble.com>> wrote:
>
>     On 8/26/16 4:08 PM, Andres Freund wrote:
>
>         Splitting of ephemeral data seems to have a benefit, the rest
>         seems more
>         like rather noisy busywork to me.
>
>
>     People accidentally blowing away pg_clog or pg_xlog is a pretty
>     common occurrence, and I don't think there's all that many tools
>     that reference them. I think it's well worth renaming them.
>
>
> Pretty sure every single backup tool or script out there is referencing
> pg_xlog. If it's not, then it's broken...

No, not really. Consider a filesytem backup using archiving and base 
backups. It doesn't care one lick about pg_xlog. And I guarantee you 
that there are tons of people running a backup like that considering the 
same script would work all the way back to 8.2 (.1?).

Sincerely,

JD

-- 
Command Prompt, Inc.                  http://the.postgres.company/
                         +1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Everyone appreciates your honesty, until you are honest with them.

Re: Renaming of pg_xlog and pg_clog

Joshua D. Drake <jd@commandprompt.com> — 2016-08-29T15:04:58Z

On 08/29/2016 06:42 AM, Daniel Verite wrote:
>
> Aside from that, we might also question how much of the excuse
> "pg_xlog looked like it was just deletable logs" is a lie made up
> after the fact, because anybody wrecking a database is not against
> deflecting a bit of the blame to the software, that's human.
>
> The truth being that they took the gamble that postgres
> will somehow recover from the deletion, at the risk of possibly
> loosing the latest transactions. That doesn't necessarily look
> so bad when current transactions are failing anyway for lack of disk
> space, users are yelling at you, and you're frantically searching for
> anything to delete in the FS to come back online quickly.
> Personally I'm quite skeptical of the *name* of the directory
> playing that much of a role in this scenario.

Oh it makes perfect sense.

User who isn't a postgres/database person installs X software. X 
software uses PostgreSQL and you have read on the intertubes about how 
Postgres is awesome. So you install it, get everything up and running 
from the README.md on Github. You walk away.

A year later, after it becomes crucial to whatever it is you do the 
system crashes. You have no idea what is going on, you just set this up 
on some recycled server or VM. You log in, see that all the space and 
you find that you are using a ton of disk space. You look around for 
anything you can delete. You find a directory called pg_xlog, it says 
log, junior ignorant, don't want to be a sysadmin 101 says, "delete logs".

Boom, crushed server. Let us not forget that by far the number of 
PostgreSQL users we have, have never done ANYTHING with postgres except 
make it so something like Drupal can connect to it.

JD

>
> Best regards,
>

-- 
Command Prompt, Inc.                  http://the.postgres.company/
                         +1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Everyone appreciates your honesty, until you are honest with them.

Re: Renaming of pg_xlog and pg_clog

Tom Lane <tgl@sss.pgh.pa.us> — 2016-08-29T15:07:17Z

"Joshua D. Drake" <jd@commandprompt.com> writes:
> Also as a note to the idea that we make break things for external user 
> space; the next version being v10 is the exact time to do that.

Let's please drop this meme that "v10 is a great time to break things".
We don't want this to be any worse than any other major-version upgrade.
If we throw thirty different major incompatibilities in at once, we're
going to be hearing about how painful it was for the next decade, even if
any one of them individually would have been manageable.  Or, if we make
the pain factor too high, users will simply not upgrade, and we'll be
faced with demands that we support 9.6 forever.

			regards, tom lane

Re: Renaming of pg_xlog and pg_clog

David Steele <david@pgmasters.net> — 2016-08-29T15:18:38Z

On 8/27/16 4:33 AM, Michael Paquier wrote:
> On Sat, Aug 27, 2016 at 6:34 AM, Andres Freund <andres@anarazel.de> wrote:
>> On 2016-08-26 17:31:14 -0400, Peter Eisentraut wrote:
>>> I agree with all that.  But the subject line is specifically about
>>> moving pg_xlog.  So if your opinion is that we shouldn't move pg_xlog,
>>> then that is noted.  But if we were to move it, we can think about a
>>> good place to move it to.
>>
>> I think it's probably worth moving pg_xlog, because the benefit also
>> includes preventing a few users from shooting themselves somewhere
>> vital. That's imo much less the case for some of the other moves.  But I
>> still don't think think a largescale reorganization is a good idea,
>> it'll just stall and nothing will happen.
> 
> OK, so let's focus only on the renaming mentioned in $subject. So far
> as I can see on this thread, here are the opinions of people who
> clearly gave one:
> - Rename them, hard break is OK: Michael P, Bruce, Stephen (depends on
> David's input),  Magnus
> - Rename them, hard break not OK: Fujii-san (perhaps do nothing?)
> - Do nothing: Simon (add a README), Tom, Peter E
> 
> As far as I can see, there is a consensus to not rename pg_xlog to
> pg_journal and avoid using a third meaning, but instead use pg_wal. I
> guess that now the other renaming would be pg_clog -> pg_xact. Other
> opinions? Forgot you here?

I'm definitely +1 for a hard break (internal links are a pain).  Ideally
I'd would like to see:

pg_xlog -> pg_wal
pg_clog -> pg_xact
pg_logical -> pg_replgcl

pg_logical is a special case because it contains both ephemeral and
persistent files.  I'd like to see the temporary files in
pg_tmp/pg_replgcl (or whatever it gets renamed to) but that means that
pg_tmp must be on the same device to get atomic renames.  It would be
enough if pg_replgcl had a pgsql_tmp subdirectory so temp files are
excluded from backups with the current logic.

I'm also in favor of leaving configuration files where they are because
these are the files that are most likely to be checked/manipulated in
various user scripts (other than pg_xlog) of course.

As Stephen mentioned I would like to see purely ephemeral data moved
into its own directory.  Maybe $PGDATA/pg_tmp?

pg_subtrans -> pg_tmp/pg_subxact
pg_stat_tamp -> pg_tmp/pg_stat
pg_dynshmem -> pg_tmp/pg_dynshmem (or pg_shmem?)
pg_notify -> pg_tmp/pg_notify
pg_replslot -> pg_tmp/pg_repslot
pg_serial -> pg_tmp/pg_serial
pg_snapshots -> pg_tmp/pg_snapshot

It would be helpful if we had consistent temp directory naming
everywhere so it's clear what to exclude when it is not practical to put
files in the root pg_tmp.  As such I would rename pgsql_tmp to pg_tmp in
base and pg_tblspc/*/*/.

-- 
-David
david@pgmasters.net

Re: Renaming of pg_xlog and pg_clog

Andres Freund <andres@anarazel.de> — 2016-08-29T15:35:45Z

Hi,

On 2016-08-29 11:18:38 -0400, David Steele wrote:
> pg_logical -> pg_replgcl

That seems considerably worse.

> pg_replslot -> pg_tmp/pg_repslot

That's most certainly not ephemeral. Just because something isn't
generally appropriate on a standby, doesn't, by far, mean it's ephemeral.

Greetings,

Andres Freund

Re: Renaming of pg_xlog and pg_clog

Tom Lane <tgl@sss.pgh.pa.us> — 2016-08-29T15:41:03Z

Andres Freund <andres@anarazel.de> writes:
> On 2016-08-29 11:18:38 -0400, David Steele wrote:
>> pg_replslot -> pg_tmp/pg_repslot

> That's most certainly not ephemeral. Just because something isn't
> generally appropriate on a standby, doesn't, by far, mean it's ephemeral.

Do we need to account for both of those concepts explicitly?

The original point here was to simplify figuring out what needed to be
copied in base backups.  I'm not sure whether we really need a marker
for what's going to be flushed at restart.

			regards, tom lane

Re: Renaming of pg_xlog and pg_clog

David Steele <david@pgmasters.net> — 2016-08-29T16:07:51Z

On 8/29/16 11:35 AM, Andres Freund wrote:
> On 2016-08-29 11:18:38 -0400, David Steele wrote:
>> pg_logical -> pg_replgcl
> 
> That seems considerably worse.

Fair enough.  I was just trying to throw something out there to get rid
of the "log" in logical.

>> pg_replslot -> pg_tmp/pg_repslot
> 
> That's most certainly not ephemeral. Just because something isn't
> generally appropriate on a standby, doesn't, by far, mean it's ephemeral.

Yes, ephemeral was a poor choice of words.  I really meant "can be
excluded from backups".

-- 
-David
david@pgmasters.net

Re: Renaming of pg_xlog and pg_clog

Andres Freund <andres@anarazel.de> — 2016-08-29T16:09:24Z

On 2016-08-29 12:07:51 -0400, David Steele wrote:
> >> pg_replslot -> pg_tmp/pg_repslot
> > 
> > That's most certainly not ephemeral. Just because something isn't
> > generally appropriate on a standby, doesn't, by far, mean it's ephemeral.
> 
> Yes, ephemeral was a poor choice of words.  I really meant "can be
> excluded from backups".

Then it most certainly can't be called pg_tmp, that'll just invite
people to rm -rf it. And then they'll be screwed.

Re: Renaming of pg_xlog and pg_clog

Joshua D. Drake <jd@commandprompt.com> — 2016-08-29T16:37:44Z

On 08/29/2016 08:07 AM, Tom Lane wrote:
> "Joshua D. Drake" <jd@commandprompt.com> writes:
>> Also as a note to the idea that we make break things for external user
>> space; the next version being v10 is the exact time to do that.
>
> Let's please drop this meme that "v10 is a great time to break things".
> We don't want this to be any worse than any other major-version upgrade.

The moment you break backward compatibility, it will be worse. We are 
talking about breaking backward compatibility. So let's just accept it 
as it is, there is a mentality about a major jump. A major jump 
(9.6->10) is *the* perfect time to make world changing, changes.

> If we throw thirty different major incompatibilities in at once, we're
> going to be hearing about how painful it was for the next decade, even if
> any one of them individually would have been manageable.  Or, if we make
> the pain factor too high, users will simply not upgrade, and we'll be
> faced with demands that we support 9.6 forever.

Whiners always find a reason to whine.

Let's be on two feet here. I am not saying we should jump to using json 
notation for our next release. I am simply stating that any largish 
(even if it is a small patch) changes to expected behavior should be 
done with care. We have a window because no matter how much you yell, I 
yell, Magnus yells, or anybody else yells; the telling story will be, 
"10.0 is a huge jump from 9.6". Most people *WOULD NOT CARE* if we only 
changed one thing. They care that we jumped 4 releases. That type of 
communication implies BIG CHANGES.

Whether you like it or not. Whether I like it or not.

Sincerely,

JD

-- 
Command Prompt, Inc.                  http://the.postgres.company/
                         +1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Everyone appreciates your honesty, until you are honest with them.
Unless otherwise stated, opinions are my own.

Re: Renaming of pg_xlog and pg_clog

Fujii Masao <masao.fujii@gmail.com> — 2016-08-29T16:51:25Z

On Sat, Aug 27, 2016 at 5:33 PM, Michael Paquier
<michael.paquier@gmail.com> wrote:
> On Sat, Aug 27, 2016 at 6:34 AM, Andres Freund <andres@anarazel.de> wrote:
>> On 2016-08-26 17:31:14 -0400, Peter Eisentraut wrote:
>>> I agree with all that.  But the subject line is specifically about
>>> moving pg_xlog.  So if your opinion is that we shouldn't move pg_xlog,
>>> then that is noted.  But if we were to move it, we can think about a
>>> good place to move it to.
>>
>> I think it's probably worth moving pg_xlog, because the benefit also
>> includes preventing a few users from shooting themselves somewhere
>> vital. That's imo much less the case for some of the other moves.  But I
>> still don't think think a largescale reorganization is a good idea,
>> it'll just stall and nothing will happen.
>
> OK, so let's focus only on the renaming mentioned in $subject. So far
> as I can see on this thread, here are the opinions of people who
> clearly gave one:
> - Rename them, hard break is OK: Michael P, Bruce, Stephen (depends on
> David's input),  Magnus
> - Rename them, hard break not OK: Fujii-san (perhaps do nothing?)
> - Do nothing: Simon (add a README), Tom, Peter E

I vote for "do nothing".
First of all, I can't have much hope for that renaming the directories really
prevents "careless" users from wrongly deleting the important files.

Regards,

-- 
Fujii Masao

Re: Renaming of pg_xlog and pg_clog

Robert Haas <robertmhaas@gmail.com> — 2016-08-29T16:54:07Z

On Sat, Aug 27, 2016 at 2:03 PM, Michael Paquier
<michael.paquier@gmail.com> wrote:
> - Rename them, hard break is OK: Michael P, Bruce, Stephen (depends on
> David's input),  Magnus

I'm in favor of this.  But I don't like Peter's proposal to use a more
complicated structure.  As for the new names, how about pg_wal and
pg_xact?  I don't think "pg_trans" is as good; is it transactional or
transient or transport or transitive or what?

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: Renaming of pg_xlog and pg_clog

Daniel Verite <daniel@manitou-mail.org> — 2016-08-29T17:00:43Z

	Joshua D. Drake wrote:

> You log in, see that all the space and you find that you are using a
> ton of disk space. You look around for anything you can delete. You
> find a directory called pg_xlog, it says log, junior ignorant, don't
> want to be a sysadmin 101 says, "delete logs".

Yes, it happens. I don't deny the problem, but I'm wondering
about the wishful thinking we're possibly falling into here
concerning the solution.

Let's imagine that pg_xlog is named wal instead.
How does that help our user with the disk space problem?
Does that point to a path of resolution? I don't see it.
What do we think that user's next move will be?
After all, WAL means Write Ahead *Log*.

On the other hand, by decommissioning pg_xlog as a name,
that makes it obsolete in all presentations, tutorials, docs
that refer to that directory, and there are many of them.
There are years of confusion ahead with questions like
"where is that pg_xlog directory that I'm supposed to
monitor, or move into a different partition, etc...",
for a benefit that remains to be seen.

Best regards,
-- 
Daniel Vérité
PostgreSQL-powered mailer: http://www.manitou-mail.org
Twitter: @DanielVerite

Re: Renaming of pg_xlog and pg_clog

Joshua D. Drake <jd@commandprompt.com> — 2016-08-29T17:05:23Z

On 08/29/2016 10:00 AM, Daniel Verite wrote:

> Let's imagine that pg_xlog is named wal instead.
> How does that help our user with the disk space problem?
> Does that point to a path of resolution? I don't see it.
> What do we think that user's next move will be?
> After all, WAL means Write Ahead *Log*.

If, they look it up (which they would likely have to) they will see it 
isn't removable. :D That is the point I am making. If it is called xlog 
the brain says "logs". If it says wal the brain says, "What is wal?"

At that point they have to look it up and then if they still delete it; 
well that is what emergency rates are for :P

Sincerely,

JD



-- 
Command Prompt, Inc.                  http://the.postgres.company/
                         +1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Everyone appreciates your honesty, until you are honest with them.
Unless otherwise stated, opinions are my own.

Re: Renaming of pg_xlog and pg_clog

Bruce Momjian <bruce@momjian.us> — 2016-08-29T17:47:07Z

On Fri, Aug 26, 2016 at 05:14:36PM +0200, Magnus Hagander wrote:
> On Aug 26, 2016 5:13 PM, "Joshua D. Drake" <jd@commandprompt.com> wrote:
> >
> > On 08/25/2016 07:39 PM, Michael Paquier wrote:
> >>
> >> Hi all,
> >>
> >> I am relaunching $subject as 10 development will begin soon. As far as
> >> I know, there is agreement that we can do something here. Among the
> >> different proposals I have found:
> >> - pg_clog renamed to pg_commit_status, pg_xact or pg_commit
> >> - pg_xlog renamed to pg_xjournal, pg_wal or pg_journal
> >
> >
> > I think the use of the pg_ prefix is redundant. Just a directory called: wal
> will do (for example).
> >
> > In reference to pg_xlog specifically, it should be wal. It is the Write Ahead
> Log, not the Journal (although I recognize they can be synonymous). All the
> documentation talks about Write Ahead Log.
> >
> 
> Yes, let's avoid inventing a *third* name for it, please. It's already bad
> enough that we have both wal and xlog.

As far as removing 'pg_', consider that many users place pg_xlog on a
different device, so using "wal" would not be clear it was related to
Postgres. Perhaps we can have initdb --xlogdir use pg_wal, and maybe
document this suggestion.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+                     Ancient Roman grave inscription +

Re: Renaming of pg_xlog and pg_clog

Jim Nasby <jim.nasby@bluetreble.com> — 2016-08-30T00:27:29Z

On 8/27/16 1:11 PM, Tom Lane wrote:
> Alvaro Herrera <alvherre@2ndquadrant.com> writes:
>> I'm for renaming too, but I'd go with Peter E's suggestion: move pg_xlog
>> to something like $PGDATA/var/wal or $PGDATA/srv/wal or something like that.
>
> I think that would make sense if we were to relocate *everything* under
> PGDATA into some FHS-like subdirectory structure.  But I'm against moving
> the config files for previously-stated reasons, and I doubt it makes sense
> to adopt an FHS-like structure only in part.

What if we left symlinks for the config files? Or perhaps even better, 
provide a tool that will create them for people that actually need them.
-- 
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com
855-TREBLE2 (855-873-2532)   mobile: 512-569-9461

Re: Renaming of pg_xlog and pg_clog

Andres Freund <andres@anarazel.de> — 2016-08-30T00:34:27Z

On 2016-08-29 19:27:29 -0500, Jim Nasby wrote:
> On 8/27/16 1:11 PM, Tom Lane wrote:
> > Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> > > I'm for renaming too, but I'd go with Peter E's suggestion: move pg_xlog
> > > to something like $PGDATA/var/wal or $PGDATA/srv/wal or something like that.
> > 
> > I think that would make sense if we were to relocate *everything* under
> > PGDATA into some FHS-like subdirectory structure.  But I'm against moving
> > the config files for previously-stated reasons, and I doubt it makes sense
> > to adopt an FHS-like structure only in part.
> 
> What if we left symlinks for the config files? Or perhaps even better,
> provide a tool that will create them for people that actually need
> them.

See the thread around
http://archives.postgresql.org/message-id/20160826104446.n3cif4m7modslkrs%40msg.df7cb.de

Re: Renaming of pg_xlog and pg_clog

Peter Eisentraut <peter.eisentraut@2ndquadrant.com> — 2016-08-30T00:54:33Z

On 8/29/16 12:54 PM, Robert Haas wrote:
> As for the new names, how about pg_wal and
> pg_xact?  I don't think "pg_trans" is as good; is it transactional or
> transient or transport or transitive or what?

pg_transaction_status? (or pg_xact_status if you want to keep it short)

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Renaming of pg_xlog and pg_clog

Peter Eisentraut <peter.eisentraut@2ndquadrant.com> — 2016-08-30T00:58:01Z

On 8/28/16 8:45 PM, Jim Nasby wrote:
> People accidentally blowing away pg_clog or pg_xlog is a pretty common 
> occurrence, and I don't think there's all that many tools that reference 
> them. I think it's well worth renaming them.

I think a related problem is that the default log directory is called
"pg_log", which is very similar to those other two.  There is no quick
way to tell their meaning apart.

I would consider changing the default from "pg_log" to "log".  Then we'd
also be at the point where we can say, everything starting with "pg_" is
internal, don't touch it.

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Renaming of pg_xlog and pg_clog

Michael Paquier <michael.paquier@gmail.com> — 2016-08-30T02:04:31Z

On Mon, Aug 29, 2016 at 9:39 PM, Craig Ringer
<craig.ringer@2ndquadrant.com> wrote:
> Cool. I'll mark as waiting on author pending that.
>
> It'll be good to get this footgun put away.

OK, so done. I have put the renaming of pg_xlog to pg_wal on top patch
stack as that's the one making no discussion it seems: a lot of people
like pg_wal. I have added as well handling for the renaming in
pg_basebackup by using PQserverVersion. One thing to note is that a
connection needs to be made to the target server *before* creating the
soft link of pg_xlog/pg_wal because we need to know the version of the
target server. pg_upgrade is handled as well. And that's all in 0001.

Patch 0002 does pg_clog -> pg_trans, "trans" standing for
"transaction". Switching to pg_trans_status or pg_xact_status is not
that complicated to change anyway...
-- 
Michael

Re: Renaming of pg_xlog and pg_clog

Jim Nasby <jim.nasby@bluetreble.com> — 2016-09-02T23:49:06Z

On 8/29/16 7:34 PM, Andres Freund wrote:
>> What if we left symlinks for the config files? Or perhaps even better,
>> > provide a tool that will create them for people that actually need
>> > them.
> See the thread around
> http://archives.postgresql.org/message-id/20160826104446.n3cif4m7modslkrs%40msg.df7cb.de

Right, but I was referring only to the config files. AFAIK it should be 
safe for those to be symlinks.
-- 
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com
855-TREBLE2 (855-873-2532)   mobile: 512-569-9461

Re: Renaming of pg_xlog and pg_clog

Michael Paquier <michael.paquier@gmail.com> — 2016-09-29T08:17:48Z

On Tue, Aug 30, 2016 at 11:04 AM, Michael Paquier
<michael.paquier@gmail.com> wrote:
> On Mon, Aug 29, 2016 at 9:39 PM, Craig Ringer
> <craig.ringer@2ndquadrant.com> wrote:
>> Cool. I'll mark as waiting on author pending that.
>>
>> It'll be good to get this footgun put away.
>
> OK, so done. I have put the renaming of pg_xlog to pg_wal on top patch
> stack as that's the one making no discussion it seems: a lot of people
> like pg_wal. I have added as well handling for the renaming in
> pg_basebackup by using PQserverVersion. One thing to note is that a
> connection needs to be made to the target server *before* creating the
> soft link of pg_xlog/pg_wal because we need to know the version of the
> target server. pg_upgrade is handled as well. And that's all in 0001.
>
> Patch 0002 does pg_clog -> pg_trans, "trans" standing for
> "transaction". Switching to pg_trans_status or pg_xact_status is not
> that complicated to change anyway...

Any input to offer for those patches? If there is nothing happening, I
guess that the best move is just to move it to next CF. At least I can
see that the flow would be to get those renamings done.
-- 
Michael

Re: Renaming of pg_xlog and pg_clog

Thomas Munro <thomas.munro@enterprisedb.com> — 2016-09-30T04:32:58Z

On Thu, Sep 29, 2016 at 9:17 PM, Michael Paquier
<michael.paquier@gmail.com> wrote:
> On Tue, Aug 30, 2016 at 11:04 AM, Michael Paquier
> <michael.paquier@gmail.com> wrote:
>> On Mon, Aug 29, 2016 at 9:39 PM, Craig Ringer
>> <craig.ringer@2ndquadrant.com> wrote:
>>> Cool. I'll mark as waiting on author pending that.
>>>
>>> It'll be good to get this footgun put away.
>>
>> OK, so done. I have put the renaming of pg_xlog to pg_wal on top patch
>> stack as that's the one making no discussion it seems: a lot of people
>> like pg_wal. I have added as well handling for the renaming in
>> pg_basebackup by using PQserverVersion. One thing to note is that a
>> connection needs to be made to the target server *before* creating the
>> soft link of pg_xlog/pg_wal because we need to know the version of the
>> target server. pg_upgrade is handled as well. And that's all in 0001.
>>
>> Patch 0002 does pg_clog -> pg_trans, "trans" standing for
>> "transaction". Switching to pg_trans_status or pg_xact_status is not
>> that complicated to change anyway...
>
> Any input to offer for those patches? If there is nothing happening, I
> guess that the best move is just to move it to next CF. At least I can
> see that the flow would be to get those renamings done.

+1 for pg_xlog -> pg_wal.

Of the existing suggestions, would like to add my vote for the
following renames, matching pg_multixact:

pg_clog -> pg_xact
pg_subtrans -> pg_subxact

If longer names are on the table, I would consider expanding all three of those:

pg_clog -> pg_transaction
pg_subtrans -> pg_subtransaction
pg_multixact -> pg_multitransaction

They sound eminently non-deletable.

-- 
Thomas Munro
http://www.enterprisedb.com

Re: Renaming of pg_xlog and pg_clog

Michael Paquier <michael.paquier@gmail.com> — 2016-09-30T05:45:29Z

On Fri, Sep 30, 2016 at 1:32 PM, Thomas Munro
<thomas.munro@enterprisedb.com> wrote:
> +1 for pg_xlog -> pg_wal.

This is clearly the winner here and...

> Of the existing suggestions, would like to add my vote for the
> following renames, matching pg_multixact:
>
> pg_clog -> pg_xact
> pg_subtrans -> pg_subxact
>
> If longer names are on the table, I would consider expanding all three of those:
>
> pg_clog -> pg_transaction

This one sounds like the consensus by what I am reading upthread.

> pg_subtrans -> pg_subtransaction
> pg_multixact -> pg_multitransaction
>
> They sound eminently non-deletable.

Well the original complain was that users tend to remove the entries
that include *log. I am not sure if it is worth worrying about doing
that much.

Anyway, those patches have rotten because of the commits of yesterday
for initdb and basebackup.c (stuff I wrote and/or reviewed actually).
So attached is a rebased set. 0001 does the work for pg_wal, 0002 for
pg_transaction.

As there have been no reviews at code level, I am moving that to the next CF.
-- 
Michael

Re: Renaming of pg_xlog and pg_clog

Robert Haas <robertmhaas@gmail.com> — 2016-10-03T13:00:33Z

On Fri, Sep 30, 2016 at 1:45 AM, Michael Paquier
<michael.paquier@gmail.com> wrote:
> As there have been no reviews at code level, I am moving that to the next CF.

Code review of 0001:

I think the tests for PQserverVersion(conn) / 100 >= 1000 are strange.
I submit that either PQserverVersion(conn) >= 100000 or
PQserverVersion(conn) / 10000 >= 10 is an easier-to-understand test.
I vote for the first style.  So in pg_basebackup.c I'd do:

#define MINIMUM_VERSION_FOR_PG_WAL 100000
...
int version = PQserverVersion(conn);
...
if (version >= MINIMUM_VERSION_FOR_PG_WAL)
   /* do whatever */

Also, for this sort of thing:

+    if (!conn)
+        /* Error message already written in GetConnection() */
+        exit(1);

...because of the comment, I'd add braces around this.

Tom changed the error-reporting framework for pg_upgrade in
f002ed2b8e45286fe73a36e119cba2016ea0de19, so you'll need to do some
rebasing over that.  I haven't checked what the new conventions are,
but obviously you'll want to try to be consistent with them.

Other than those minor nitpicks, I think this looks OK.  I'm not sure
we have consensus on 0002, but 0001 seems to be popular with far more
people than not.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: Renaming of pg_xlog and pg_clog

Christoph Berg <myon@debian.org> — 2016-10-03T14:20:00Z

Re: Michael Paquier 2016-09-30 <CAB7nPqR=SZNo_=B1ukwCiNn7aWDcw_dV0z-LG4ys9WF1N4a=uQ@mail.gmail.com>

"pg_trans" is used in two places:

> -pg_clog records the commit status for each transaction that has been assigned
> +pg_trans records the commit status for each transaction that has been assigned

> -	/* copy old commit logs to new data dir */
> -	copy_subdir_files("pg_clog");
> +	/*
> +	 * Copy old commit logs to new data dir. pg_clog has been renamed to
> +	 * pg_trans in post-10 clusters.

(Fwiw, I'd prefer something shorter than the imho clumsy
pg_transaction. "pg_xact" sounded very fine for me, it also combines
nicely with pg_subxact and the existing pg_multixact.)

Christoph

Re: Renaming of pg_xlog and pg_clog

Tom Lane <tgl@sss.pgh.pa.us> — 2016-10-03T14:39:43Z

Robert Haas <robertmhaas@gmail.com> writes:
> I think the tests for PQserverVersion(conn) / 100 >= 1000 are strange.
> I submit that either PQserverVersion(conn) >= 100000 or
> PQserverVersion(conn) / 10000 >= 10 is an easier-to-understand test.
> I vote for the first style.

+1, that's the way most existing tests of this sort are written.

(Right at the moment, there's kind of a lot of zeroes there, but
once we get to version 11 it'll be less bad.)

			regards, tom lane

Re: Renaming of pg_xlog and pg_clog

Michael Paquier <michael.paquier@gmail.com> — 2016-10-04T05:48:40Z

On Mon, Oct 3, 2016 at 10:00 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> On Fri, Sep 30, 2016 at 1:45 AM, Michael Paquier
> <michael.paquier@gmail.com> wrote:
>> As there have been no reviews at code level, I am moving that to the next CF.
>
> Code review of 0001:
>
> I think the tests for PQserverVersion(conn) / 100 >= 1000 are strange.
> I submit that either PQserverVersion(conn) >= 100000 or
> PQserverVersion(conn) / 10000 >= 10 is an easier-to-understand test.
> I vote for the first style.  So in pg_basebackup.c I'd do:
>
> #define MINIMUM_VERSION_FOR_PG_WAL 100000
> ...
> int version = PQserverVersion(conn);
> ...
> if (version >= MINIMUM_VERSION_FOR_PG_WAL)
>    /* do whatever */

I have adopted the existing style of BaseBackup() in last versions,
but I don't mind as well doing the way you suggest, without changing
the stuff in BaseBackup() though, we'd still want a pg_basebackup
compiled with 10.1 to work with 10.2 or newer minor versions of 10.

> Also, for this sort of thing:
>
> +    if (!conn)
> +        /* Error message already written in GetConnection() */
> +        exit(1);
>
> ...because of the comment, I'd add braces around this.

Fixed. That was done without brackets on HEAD, but you are right that
is not PG-like. (There are other places in src/bin/pg_basebackup doing
the same thing, but I have not modified them to not create more diff
noise).

> Tom changed the error-reporting framework for pg_upgrade in
> f002ed2b8e45286fe73a36e119cba2016ea0de19, so you'll need to do some
> rebasing over that.  I haven't checked what the new conventions are,
> but obviously you'll want to try to be consistent with them.

strerror(errno) needs to be used instead of getErrorText() in the
context of this patch.

> Other than those minor nitpicks, I think this looks OK.  I'm not sure
> we have consensus on 0002, but 0001 seems to be popular with far more
> people than not.

Yes, pg_wal is fine as new name in replacement of pg_xlog. Now for the
pg_clog...

Re: Michael Paquier 2016-09-30
<CAB7nPqR=SZNo_=B1ukwCiNn7aWDcw_dV0z-LG4ys9WF1N4a=uQ@mail.gmail.com>
> "pg_trans" is used in two places:
> -pg_clog records the commit status for each transaction that has been assigned
> +pg_trans records the commit status for each transaction that has been assigned

> -     /* copy old commit logs to new data dir */
> -     copy_subdir_files("pg_clog");
> +     /*
> +      * Copy old commit logs to new data dir. pg_clog has been renamed to
> +      * pg_trans in post-10 clusters.

> (Fwiw, I'd prefer something shorter than the imho clumsy
> pg_transaction. "pg_xact" sounded very fine for me, it also combines
> nicely with pg_subxact and the existing pg_multixact.)

Thanks, fixed those.

So this is still open for votes. Here are the candidates and who
voiced for what:
- pg_transaction: Michael P, Thomas M. => Current 0002 is doing that.
- pg_xact: David S, Robert
- pg_trans: Tom
- pg_transaction_status: Peter E.

Attached is a new patch set. I have re-done tests with pg_upgrade with
9.5 -> HEAD and 9.6 -> HEAD, and pg_basebackup for symlink handling.
Actually in the latter case I have found a bug in 0001 introduced by
bc34223b because fsync_pgdata() is not aware of the version of the
server it is trying to fsync(). I think that the better way to address
that is to extend fsync_pgdata in file_utils.c with the server version
to fsync() like attached.
-- 
Michael

Re: Renaming of pg_xlog and pg_clog

David Steele <david@pgmasters.net> — 2016-10-04T13:07:06Z

On 10/4/16 1:48 AM, Michael Paquier wrote:

> So this is still open for votes. Here are the candidates and who
> voiced for what:
> - pg_transaction: Michael P, Thomas M. => Current 0002 is doing that.
> - pg_xact: David S, Robert
> - pg_trans: Tom
> - pg_transaction_status: Peter E.

Christoph also prefers pg_xact [1].

-- 
-David
david@pgmasters.net

[1]
https://www.postgresql.org/message-id/20161003141959.qhvd6roesj4kpgww@msg.df7cb.de

Re: Renaming of pg_xlog and pg_clog

Robert Haas <robertmhaas@gmail.com> — 2016-10-04T17:38:35Z

On Tue, Oct 4, 2016 at 1:48 AM, Michael Paquier
<michael.paquier@gmail.com> wrote:
> Yes, pg_wal is fine as new name in replacement of pg_xlog. Now for the
> pg_clog...

Of course the irony here is that "WAL" stands for "Write Ahead Log".
So we're renaming a directly that has "log" in the name (pg_xlog) to a
directory that has an "l" in the name which stands for "log" (pg_wal).
Whee!

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: Renaming of pg_xlog and pg_clog

Peter Eisentraut <peter.eisentraut@2ndquadrant.com> — 2016-10-12T15:00:50Z

On 10/4/16 1:48 AM, Michael Paquier wrote:
> So this is still open for votes. Here are the candidates and who
> voiced for what:
> - pg_transaction: Michael P, Thomas M. => Current 0002 is doing that.
> - pg_xact: David S, Robert
> - pg_trans: Tom
> - pg_transaction_status: Peter E.

I think this would all be a lot easier if we just moved all the pg_*
directories one directory down.  We did this with "global" many years
ago and it worked out well.  We didn't actually have to change the file
names, the top-level directory became cleaner, and no one was messing
around with the files anymore.  Adjusting external tools also becomes
easier.  There is so much lore out there about clog and xlog; it would
be annoying to break with that.

I can see a reason for not doing that with pg_xlog, because that is a
bit of an external interface, with initdb -X and such, and pg_wal is a
pretty good alternative name.  But no one deals with these other
directories directly, so just move them out of the way.

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Renaming of pg_xlog and pg_clog

Tom Lane <tgl@sss.pgh.pa.us> — 2016-10-12T15:22:06Z

Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:
> On 10/4/16 1:48 AM, Michael Paquier wrote:
>> So this is still open for votes. Here are the candidates and who
>> voiced for what:
>> - pg_transaction: Michael P, Thomas M. => Current 0002 is doing that.
>> - pg_xact: David S, Robert
>> - pg_trans: Tom
>> - pg_transaction_status: Peter E.

> I think this would all be a lot easier if we just moved all the pg_*
> directories one directory down.

The main problem we're trying to fix here is people thinking that
something with "log" in the name contains discardable data.  Just
relocating the directory without renaming it won't improve that.

The relocation aspect was meant to address another problem, which
is making it easier to distinguish which parts of the tree should
be copied by tools like pg_basebackup.  But that's not what Michael
is asking for votes on.

			regards, tom lane

Re: Renaming of pg_xlog and pg_clog

Stephen Frost <sfrost@snowman.net> — 2016-10-12T15:31:49Z

* David Steele (david@pgmasters.net) wrote:
> On 10/4/16 1:48 AM, Michael Paquier wrote:
> 
> > So this is still open for votes. Here are the candidates and who
> > voiced for what:
> > - pg_transaction: Michael P, Thomas M. => Current 0002 is doing that.
> > - pg_xact: David S, Robert
> > - pg_trans: Tom
> > - pg_transaction_status: Peter E.
> 
> Christoph also prefers pg_xact [1].

Since we're asking for votes, I also prefer pg_xact over the other
options listed here.

Thanks!

Stephen

Re: Renaming of pg_xlog and pg_clog

Michael Paquier <michael.paquier@gmail.com> — 2016-10-13T02:22:58Z

On Thu, Oct 13, 2016 at 12:31 AM, Stephen Frost <sfrost@snowman.net> wrote:
> * David Steele (david@pgmasters.net) wrote:
>> On 10/4/16 1:48 AM, Michael Paquier wrote:
>>
>> > So this is still open for votes. Here are the candidates and who
>> > voiced for what:
>> > - pg_transaction: Michael P, Thomas M. => Current 0002 is doing that.
>> > - pg_xact: David S, Robert
>> > - pg_trans: Tom
>> > - pg_transaction_status: Peter E.
>>
>> Christoph also prefers pg_xact [1].
>
> Since we're asking for votes, I also prefer pg_xact over the other
> options listed here.

OK. I can live with that as well. Attached are three patches. The
pg_xlog -> pg_wal move, the pg_clog -> pg_transaction move, and the
pg_clog -> pg_xact move. Only one survivor to be chosen among the last
two ones.
-- 
Michael

Re: Renaming of pg_xlog and pg_clog

Peter Eisentraut <peter.eisentraut@2ndquadrant.com> — 2016-10-14T04:38:06Z

On 10/12/16 11:22 AM, Tom Lane wrote:
> The main problem we're trying to fix here is people thinking that
> something with "log" in the name contains discardable data.  Just
> relocating the directory without renaming it won't improve that.

I think it would help if we moved it to something like
"internal/pg_xlog" and "internal/pg_clog".  Keep the name but move it
out of sight.

We have a tool called pg_xlogdump in the standard installation.  initdb
has an option --xlogdir, pg_basebackup has --xlog and others.  Renaming
the xlog directory would make this all a bit confusing, unless we're
prepared to rename the programs and options as well.

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Renaming of pg_xlog and pg_clog

Stephen Frost <sfrost@snowman.net> — 2016-10-14T11:35:23Z

* Peter Eisentraut (peter.eisentraut@2ndquadrant.com) wrote:
> On 10/12/16 11:22 AM, Tom Lane wrote:
> > The main problem we're trying to fix here is people thinking that
> > something with "log" in the name contains discardable data.  Just
> > relocating the directory without renaming it won't improve that.
> 
> I think it would help if we moved it to something like
> "internal/pg_xlog" and "internal/pg_clog".  Keep the name but move it
> out of sight.

I disagree that this will materially help with the issue.

> We have a tool called pg_xlogdump in the standard installation.  initdb
> has an option --xlogdir, pg_basebackup has --xlog and others.  Renaming
> the xlog directory would make this all a bit confusing, unless we're
> prepared to rename the programs and options as well.

pg_xlogdump is not a user-facing tool, frankly, so I don't believe we
should be terribly concerned about either leaving it named as-is or
renaming it.  I agree that we should consider adding alternative names
to the options for initdb and pg_basebackup.

Thanks!

Stephen

Re: Renaming of pg_xlog and pg_clog

Christoph Berg <myon@debian.org> — 2016-10-14T12:58:33Z

Re: Stephen Frost 2016-10-14 <20161014113523.GZ13284@tamriel.snowman.net>
> > I think it would help if we moved it to something like
> > "internal/pg_xlog" and "internal/pg_clog".  Keep the name but move it
> > out of sight.
> 
> I disagree that this will materially help with the issue.

And internal/base and internal/global and internal/pg_... because
these shouldn't be touched by the users either.

I don't think this would lead anywhere.

Christoph

Re: Renaming of pg_xlog and pg_clog

Stephen Frost <sfrost@snowman.net> — 2016-10-14T14:06:30Z

* Christoph Berg (myon@debian.org) wrote:
> Re: Stephen Frost 2016-10-14 <20161014113523.GZ13284@tamriel.snowman.net>
> > > I think it would help if we moved it to something like
> > > "internal/pg_xlog" and "internal/pg_clog".  Keep the name but move it
> > > out of sight.
> > 
> > I disagree that this will materially help with the issue.
> 
> And internal/base and internal/global and internal/pg_... because
> these shouldn't be touched by the users either.
> 
> I don't think this would lead anywhere.

It'd probably be easier to move the things that are *not* PG internal
(eg: config files, et al) *out* of the data directory and into somewhere
sensible, like /etc ...

Oh, wait ...

Thanks!

Stephen

Re: Renaming of pg_xlog and pg_clog

Jim Nasby <jim.nasby@bluetreble.com> — 2016-10-14T16:10:49Z

On 10/14/16 9:06 AM, Stephen Frost wrote:
>> > And internal/base and internal/global and internal/pg_... because
>> > these shouldn't be touched by the users either.
>> >
>> > I don't think this would lead anywhere.
> It'd probably be easier to move the things that are *not* PG internal
> (eg: config files, et al) *out* of the data directory and into somewhere
> sensible, like /etc ...

I do think it would be an improvement to segregate things users are 
expected to touch (*.conf and pg_log are what come to mind) from 
everything else, which could easily be done by moving everything else to 
an internal/ directory. I agree that's not much of an improvement for 
pg_[cx]log, but we could create internal/ as well as rename some things.
-- 
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com
855-TREBLE2 (855-873-2532)   mobile: 512-569-9461

Re: Renaming of pg_xlog and pg_clog

Stephen Frost <sfrost@snowman.net> — 2016-10-14T16:19:41Z

* Jim Nasby (Jim.Nasby@BlueTreble.com) wrote:
> On 10/14/16 9:06 AM, Stephen Frost wrote:
> >>> And internal/base and internal/global and internal/pg_... because
> >>> these shouldn't be touched by the users either.
> >>>
> >>> I don't think this would lead anywhere.
> >It'd probably be easier to move the things that are *not* PG internal
> >(eg: config files, et al) *out* of the data directory and into somewhere
> >sensible, like /etc ...
> 
> I do think it would be an improvement to segregate things users are
> expected to touch (*.conf and pg_log are what come to mind) from
> everything else, which could easily be done by moving everything
> else to an internal/ directory. I agree that's not much of an
> improvement for pg_[cx]log, but we could create internal/ as well as
> rename some things.

Apologis, I should have made it clear what I was getting at.

The Debian/Ubuntu packaging already does segregate out the *.conf and
pg_log into other locations based on the FHS.  On a Debian-based system,
there really shouldn't be anything that the user is expected to touch
under /var/lib (where the PG data directory lives).

Thanks!

Stephen

Re: Renaming of pg_xlog and pg_clog

Tom Lane <tgl@sss.pgh.pa.us> — 2016-10-14T16:56:21Z

Jim Nasby <Jim.Nasby@BlueTreble.com> writes:
> On 10/14/16 9:06 AM, Stephen Frost wrote:
>> It'd probably be easier to move the things that are *not* PG internal
>> (eg: config files, et al) *out* of the data directory and into somewhere
>> sensible, like /etc ...

> I do think it would be an improvement to segregate things users are 
> expected to touch (*.conf and pg_log are what come to mind) from 
> everything else, which could easily be done by moving everything else to 
> an internal/ directory. I agree that's not much of an improvement for 
> pg_[cx]log, but we could create internal/ as well as rename some things.

I can't get excited about that at all.  It will just get in the way of
the actually useful end goal we had for directory restructuring, which
was to separate data to be copied by pg_basebackup from data not to be
copied by it.  And in reality, people who don't understand that the
contents of PGDATA should be treated as "hands off unless documented
otherwise" are not going to be any less likely to shoot themselves in
the foot just because there's a directory named "internal" or
"here_be_dragons" or "keep_out_this_means_you" in the way.

I'd also suggest that an extra level of directory search to get at
database data files is not free.

			regards, tom lane

Re: Renaming of pg_xlog and pg_clog

Magnus Hagander <magnus@hagander.net> — 2016-10-14T17:02:55Z

On Fri, Oct 14, 2016 at 9:56 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

> Jim Nasby <Jim.Nasby@BlueTreble.com> writes:
> > On 10/14/16 9:06 AM, Stephen Frost wrote:
> >> It'd probably be easier to move the things that are *not* PG internal
> >> (eg: config files, et al) *out* of the data directory and into somewhere
> >> sensible, like /etc ...
>
> > I do think it would be an improvement to segregate things users are
> > expected to touch (*.conf and pg_log are what come to mind) from
> > everything else, which could easily be done by moving everything else to
> > an internal/ directory. I agree that's not much of an improvement for
> > pg_[cx]log, but we could create internal/ as well as rename some things.
>
> I can't get excited about that at all.  It will just get in the way of
> the actually useful end goal we had for directory restructuring, which
> was to separate data to be copied by pg_basebackup from data not to be
> copied by it.  And in reality, people who don't understand that the
> contents of PGDATA should be treated as "hands off unless documented
> otherwise" are not going to be any less likely to shoot themselves in
> the foot just because there's a directory named "internal" or
> "here_be_dragons" or "keep_out_this_means_you" in the way.


+1. I think we should move the don't-include-in-backup files to a separate
directory for *technical* reasons.

I don't think moving files that are already supposed to be internal into a
directory called internal is going to help much, an din particular not for
xlog. If we don't rename it, the problem will remain.

And for the "don't touch" part, if anything we should move the config files
into a subdirectory of PGDATA and not the other way around.

-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

Re: Renaming of pg_xlog and pg_clog

Magnus Hagander <magnus@hagander.net> — 2016-10-14T17:04:57Z

On Fri, Oct 14, 2016 at 4:35 AM, Stephen Frost <sfrost@snowman.net> wrote:

> * Peter Eisentraut (peter.eisentraut@2ndquadrant.com) wrote:
> > On 10/12/16 11:22 AM, Tom Lane wrote:
> > > The main problem we're trying to fix here is people thinking that
> > > something with "log" in the name contains discardable data.  Just
> > > relocating the directory without renaming it won't improve that.
> >
> > I think it would help if we moved it to something like
> > "internal/pg_xlog" and "internal/pg_clog".  Keep the name but move it
> > out of sight.
>
> I disagree that this will materially help with the issue.
>
> > We have a tool called pg_xlogdump in the standard installation.  initdb
> > has an option --xlogdir, pg_basebackup has --xlog and others.  Renaming
> > the xlog directory would make this all a bit confusing, unless we're
> > prepared to rename the programs and options as well.
>
> pg_xlogdump is not a user-facing tool, frankly, so I don't believe we
> should be terribly concerned about either leaving it named as-is or
> renaming it.  I agree that we should consider adding alternative names
> to the options for initdb and pg_basebackup.
>

Ugh. Changing the names of those options are probably going to break a
*lot* of things, a lot more than changing the names of the directories. Is
it really worth doing that? Especially if we are even more clear that
people should not be touching those directories in the first place.

Those are just the tip of the iceberg. We do use the term xlog in a lot of
places (almost as many as wal, but that's a different problem)

-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

Re: Renaming of pg_xlog and pg_clog

Oskari Saarenmaa <os@ohmu.fi> — 2016-10-14T17:08:33Z

14.10.2016, 07:38, Peter Eisentraut kirjoitti:
> On 10/12/16 11:22 AM, Tom Lane wrote:
>> The main problem we're trying to fix here is people thinking that
>> something with "log" in the name contains discardable data.  Just
>> relocating the directory without renaming it won't improve that.
>
> I think it would help if we moved it to something like
> "internal/pg_xlog" and "internal/pg_clog".  Keep the name but move it
> out of sight.
>
> We have a tool called pg_xlogdump in the standard installation.  initdb
> has an option --xlogdir, pg_basebackup has --xlog and others.  Renaming
> the xlog directory would make this all a bit confusing, unless we're
> prepared to rename the programs and options as well.

pg_receivexlog should probably be renamed, seeing how we have 
pg_recvlogical perhaps pg_recvwal would work?

The --xlog, -x, --xlog-method and -X flags for pg_basebackup are a bit 
confusing as it is.  Perhaps they can be kept around as deprecated 
aliases for a new --wal stream/fetch switch: I don't think we need new 
--wal and --wal-method switches.

pg_resetxlog should probably be called something different than just 
plain pg_resetwal to make it obvious that running it will cause data loss.

pg_xlogdump is a developer tool, users shouldn't care; it's hard enough 
to use as it is as it doesn't do anything useful when you try to point 
it to a recycled wal file.

All in all, I think this is a good opportunity to clarify what the tools 
are actually supposed to do and who should be running them.  As an 
author of a backup tool utilizing some of these tools & options I don't 
think renaming commands and/or arguments is a big deal -- we have to 
deal with a bunch of changes for every new major version anyway.

/ Oskari

Re: Renaming of pg_xlog and pg_clog

Christoph Berg <myon@debian.org> — 2016-10-14T17:19:02Z

Re: Stephen Frost 2016-10-14 <20161014113523.GZ13284@tamriel.snowman.net>
> > We have a tool called pg_xlogdump in the standard installation.  initdb
> > has an option --xlogdir, pg_basebackup has --xlog and others.  Renaming
> > the xlog directory would make this all a bit confusing, unless we're
> > prepared to rename the programs and options as well.
> 
> pg_xlogdump is not a user-facing tool, frankly, so I don't believe we
> should be terribly concerned about either leaving it named as-is or
> renaming it.  I agree that we should consider adding alternative names
> to the options for initdb and pg_basebackup.

There's actually another instance of "rename so people shoot their
feet less often" here: pg_resetxlog, which is a user-facing tool.
Folks on #postgresql have repeatedly been joking that it should rather
be named pg_eatmydata, given the number of "howtos" on the web that
make use of it. Maybe this is the chance to find a less
innocent-sounding name. (Or add a mandatory --rewind-my-data switch.)

Christoph

Re: Renaming of pg_xlog and pg_clog

Stephen Frost <sfrost@snowman.net> — 2016-10-14T18:48:14Z

* Magnus Hagander (magnus@hagander.net) wrote:
> On Fri, Oct 14, 2016 at 4:35 AM, Stephen Frost <sfrost@snowman.net> wrote:
> > * Peter Eisentraut (peter.eisentraut@2ndquadrant.com) wrote:
> > > On 10/12/16 11:22 AM, Tom Lane wrote:
> > > > The main problem we're trying to fix here is people thinking that
> > > > something with "log" in the name contains discardable data.  Just
> > > > relocating the directory without renaming it won't improve that.
> > >
> > > I think it would help if we moved it to something like
> > > "internal/pg_xlog" and "internal/pg_clog".  Keep the name but move it
> > > out of sight.
> >
> > I disagree that this will materially help with the issue.
> >
> > > We have a tool called pg_xlogdump in the standard installation.  initdb
> > > has an option --xlogdir, pg_basebackup has --xlog and others.  Renaming
> > > the xlog directory would make this all a bit confusing, unless we're
> > > prepared to rename the programs and options as well.
> >
> > pg_xlogdump is not a user-facing tool, frankly, so I don't believe we
> > should be terribly concerned about either leaving it named as-is or
> > renaming it.  I agree that we should consider adding alternative names
> > to the options for initdb and pg_basebackup.
> 
> Ugh. Changing the names of those options are probably going to break a
> *lot* of things, a lot more than changing the names of the directories. Is
> it really worth doing that? Especially if we are even more clear that
> people should not be touching those directories in the first place.

I would point out that I said "adding alternative names", not
"change/remove the existing options."

That said, while I like the general idea of "make everything referring
to transaction log/write-ahead xlog/xlog/wal/whatever refer in the same
way", I'm not sure that it's a really pressing issue or that changing
the name of the directory should drive those other changes.

> Those are just the tip of the iceberg. We do use the term xlog in a lot of
> places (almost as many as wal, but that's a different problem)

True, and yes, we should consider the places that already talk about
"wal", but, overall, I believe we can take this one step at a time and
don't have to do everything all at once simply because we're changing
the directory name.  We're not changing what it *is*, after all.

Thanks!

Stephen

Re: Renaming of pg_xlog and pg_clog

Stephen Frost <sfrost@snowman.net> — 2016-10-14T18:49:37Z

* Christoph Berg (myon@debian.org) wrote:
> Re: Stephen Frost 2016-10-14 <20161014113523.GZ13284@tamriel.snowman.net>
> > > We have a tool called pg_xlogdump in the standard installation.  initdb
> > > has an option --xlogdir, pg_basebackup has --xlog and others.  Renaming
> > > the xlog directory would make this all a bit confusing, unless we're
> > > prepared to rename the programs and options as well.
> > 
> > pg_xlogdump is not a user-facing tool, frankly, so I don't believe we
> > should be terribly concerned about either leaving it named as-is or
> > renaming it.  I agree that we should consider adding alternative names
> > to the options for initdb and pg_basebackup.
> 
> There's actually another instance of "rename so people shoot their
> feet less often" here: pg_resetxlog, which is a user-facing tool.

That is *not* a user-facing tool.

> Folks on #postgresql have repeatedly been joking that it should rather
> be named pg_eatmydata, given the number of "howtos" on the web that
> make use of it. Maybe this is the chance to find a less
> innocent-sounding name. (Or add a mandatory --rewind-my-data switch.)

Not sure on the best approach here, but I agree that it's a problem that
people think pg_resetxlog is some kind of recovery tool (hint: it's
not).

Thanks!

Stephen

Re: Renaming of pg_xlog and pg_clog

Magnus Hagander <magnus@hagander.net> — 2016-10-14T18:51:49Z

On Fri, Oct 14, 2016 at 11:48 AM, Stephen Frost <sfrost@snowman.net> wrote:

> * Magnus Hagander (magnus@hagander.net) wrote:
> > On Fri, Oct 14, 2016 at 4:35 AM, Stephen Frost <sfrost@snowman.net>
> wrote:
> > > * Peter Eisentraut (peter.eisentraut@2ndquadrant.com) wrote:
> > > > On 10/12/16 11:22 AM, Tom Lane wrote:
> > > > > The main problem we're trying to fix here is people thinking that
> > > > > something with "log" in the name contains discardable data.  Just
> > > > > relocating the directory without renaming it won't improve that.
> > > >
> > > > I think it would help if we moved it to something like
> > > > "internal/pg_xlog" and "internal/pg_clog".  Keep the name but move it
> > > > out of sight.
> > >
> > > I disagree that this will materially help with the issue.
> > >
> > > > We have a tool called pg_xlogdump in the standard installation.
> initdb
> > > > has an option --xlogdir, pg_basebackup has --xlog and others.
> Renaming
> > > > the xlog directory would make this all a bit confusing, unless we're
> > > > prepared to rename the programs and options as well.
> > >
> > > pg_xlogdump is not a user-facing tool, frankly, so I don't believe we
> > > should be terribly concerned about either leaving it named as-is or
> > > renaming it.  I agree that we should consider adding alternative names
> > > to the options for initdb and pg_basebackup.
> >
> > Ugh. Changing the names of those options are probably going to break a
> > *lot* of things, a lot more than changing the names of the directories.
> Is
> > it really worth doing that? Especially if we are even more clear that
> > people should not be touching those directories in the first place.
>
> I would point out that I said "adding alternative names", not
> "change/remove the existing options."
>

Oh, I missed that. Thanks for clarifying :)



> That said, while I like the general idea of "make everything referring
> to transaction log/write-ahead xlog/xlog/wal/whatever refer in the same
> way", I'm not sure that it's a really pressing issue or that changing
> the name of the directory should drive those other changes.
>

Agreed.



> > Those are just the tip of the iceberg. We do use the term xlog in a lot
> of
> > places (almost as many as wal, but that's a different problem)
>
> True, and yes, we should consider the places that already talk about
> "wal", but, overall, I believe we can take this one step at a time and
> don't have to do everything all at once simply because we're changing
> the directory name.  We're not changing what it *is*, after all.
>

Yeah, I think if we aim to fix everything at once, we'll get nothing done.
That old saying of not letting perfection getting in the way of progress
applies.

-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

Re: Renaming of pg_xlog and pg_clog

Bruce Momjian <bruce@momjian.us> — 2016-10-18T22:09:10Z

On Fri, Oct 14, 2016 at 07:19:02PM +0200, Christoph Berg wrote:
> Re: Stephen Frost 2016-10-14 <20161014113523.GZ13284@tamriel.snowman.net>
> > > We have a tool called pg_xlogdump in the standard installation.  initdb
> > > has an option --xlogdir, pg_basebackup has --xlog and others.  Renaming
> > > the xlog directory would make this all a bit confusing, unless we're
> > > prepared to rename the programs and options as well.
> > 
> > pg_xlogdump is not a user-facing tool, frankly, so I don't believe we
> > should be terribly concerned about either leaving it named as-is or
> > renaming it.  I agree that we should consider adding alternative names
> > to the options for initdb and pg_basebackup.
> 
> There's actually another instance of "rename so people shoot their
> feet less often" here: pg_resetxlog, which is a user-facing tool.
> Folks on #postgresql have repeatedly been joking that it should rather
> be named pg_eatmydata, given the number of "howtos" on the web that
> make use of it. Maybe this is the chance to find a less
> innocent-sounding name. (Or add a mandatory --rewind-my-data switch.)

I wonder how many of the uses of pg_resetxlog were caused by mistakenly
removing the pg_xlog directory.   My point is renaming pg_xlog might
reduce mistake use of pg_resetxlog.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

Re: Renaming of pg_xlog and pg_clog

Christoph Berg <myon@debian.org> — 2016-10-19T11:05:28Z

Re: Bruce Momjian 2016-10-19 <20161018220909.GA11661@momjian.us>
> > There's actually another instance of "rename so people shoot their
> > feet less often" here: pg_resetxlog, which is a user-facing tool.
> > Folks on #postgresql have repeatedly been joking that it should rather
> > be named pg_eatmydata, given the number of "howtos" on the web that
> > make use of it. Maybe this is the chance to find a less
> > innocent-sounding name. (Or add a mandatory --rewind-my-data switch.)
> 
> I wonder how many of the uses of pg_resetxlog were caused by mistakenly
> removing the pg_xlog directory.   My point is renaming pg_xlog might
> reduce mistake use of pg_resetxlog.

I don't think there's much of a connection. There are people who
clean up disk space by removing everything that sounds like log files.
For those people renaming the directories makes sense so they don't
have that idea.

There are other people who have random database startup problems, ask
google, and end up with applying some pg_resetxlog advice (that
doesn't necessarily fit their problem). For those people renaming
pg_resetxlog to something that sounds more like "it will break your
data, use as last resort only" might make sense. (Though I don't have
a good suggestion, and the cost of renaming utilities is higher than
renaming some internal directory names.)

The question would now be if there's people who used pg_resetxlog
because they thought it freed up disk space, and if renaming either
would have prevented that. I'm less sure about that.

(tl;dr: rename pg_xlog yes, rename pg_resetxlog only if we have a good
alternative.)

Christoph

Re: Renaming of pg_xlog and pg_clog

Robert Haas <robertmhaas@gmail.com> — 2016-10-20T15:11:50Z

On Wed, Oct 19, 2016 at 7:05 AM, Christoph Berg <myon@debian.org> wrote:
> (tl;dr: rename pg_xlog yes, rename pg_resetxlog only if we have a good
> alternative.)

I'm amused by the idea of a TL;DR in parentheses at the very bottom of
the email, but maybe I'm just easily amused.

One idea would be to rename pg_resetxlog to pg_resetwal.  I think
that's actually an improvement.  The "x" in "xlog" is not a clear
abbreviation for "write-ahead", especially when we also use "x" in
other contexts to mean "transaction" - think "xid".  Given our current
practices, we could justifiably rename pg_clog to pg_xlog -- after
all, that's where we log the status of the xacts.  Ugh.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: Renaming of pg_xlog and pg_clog

Robert Haas <robertmhaas@gmail.com> — 2016-10-20T15:35:42Z

On Wed, Oct 12, 2016 at 10:22 PM, Michael Paquier
<michael.paquier@gmail.com> wrote:
> OK. I can live with that as well. Attached are three patches. The
> pg_xlog -> pg_wal move, the pg_clog -> pg_transaction move, and the
> pg_clog -> pg_xact move. Only one survivor to be chosen among the last
> two ones.

Committed 0001.

To be honest, I don't really like either pg_transaction or pg_xact.
Neither name captures the fact that what we're really tracking here is
the transaction *status*.  pg_xact is slightly worse because it's a
poor abbreviation for transaction, but I think the argument against
even pg_transaction is similar to the one Tom previously levied
against pg_logical - viz. "logical what?".  The transaction themselves
are not stored in the directory, just the commit status.  The fact
that commit status is saved is the source of the "c" in "clog".

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: Renaming of pg_xlog and pg_clog

Tom Lane <tgl@sss.pgh.pa.us> — 2016-10-20T15:48:33Z

Robert Haas <robertmhaas@gmail.com> writes:
> One idea would be to rename pg_resetxlog to pg_resetwal.  I think
> that's actually an improvement.

This would fit in as part of a general plan to s/xlog/wal/g throughout
our user-visible names and documentation.  Which seems like a good idea
to me; there's no need to expose two different terms for the same thing.

(I don't feel a great need to unify the terminology in the code, though.
Just in stuff users see.)

			regards, tom lane

Re: Renaming of pg_xlog and pg_clog

Stephen Frost <sfrost@snowman.net> — 2016-10-20T16:05:00Z

* Robert Haas (robertmhaas@gmail.com) wrote:
> On Wed, Oct 12, 2016 at 10:22 PM, Michael Paquier
> <michael.paquier@gmail.com> wrote:
> > OK. I can live with that as well. Attached are three patches. The
> > pg_xlog -> pg_wal move, the pg_clog -> pg_transaction move, and the
> > pg_clog -> pg_xact move. Only one survivor to be chosen among the last
> > two ones.
> 
> Committed 0001.

Glad to see that happen, finally.

> To be honest, I don't really like either pg_transaction or pg_xact.

> Neither name captures the fact that what we're really tracking here is
> the transaction *status*.  pg_xact is slightly worse because it's a
> poor abbreviation for transaction, but I think the argument against
> even pg_transaction is similar to the one Tom previously levied
> against pg_logical - viz. "logical what?".  The transaction themselves
> are not stored in the directory, just the commit status.  The fact
> that commit status is saved is the source of the "c" in "clog".

This really needs to move forward also.

When it comes to the name, I tend to think of 'pg_xact' as saying "this
is where we persist info we need to keep about transactions."  Today
that's just the commit status info, but I could imagine that there
might, someday, be other things that go in there.  "pg_multixact" is
an example of something quite similar but does have more than just one
"thing."  Also, using "pg_xact" and then "pg_subxact" and "pg_multixact"
bring them all under one consistent naming scheme.

Thanks!

Stephen

Re: Renaming of pg_xlog and pg_clog

Stephen Frost <sfrost@snowman.net> — 2016-10-20T16:12:16Z

* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> Robert Haas <robertmhaas@gmail.com> writes:
> > One idea would be to rename pg_resetxlog to pg_resetwal.  I think
> > that's actually an improvement.
> 
> This would fit in as part of a general plan to s/xlog/wal/g throughout
> our user-visible names and documentation.  Which seems like a good idea
> to me; there's no need to expose two different terms for the same thing.
> 
> (I don't feel a great need to unify the terminology in the code, though.
> Just in stuff users see.)

+1 on the general change of xlog -> wal.

That said, I'd also like to see a --force or similar option or mechanism
put in place to reduce the risk of users trashing their system because
they think pg_resetwal is "safe." ("It's just gonna reset things to make
the database start again, should be fine.").

pg_destroydb almost seems like a better choice, though I suppose
'pg_clearwal' would be more acceptable.  Doesn't have quite the same
impact though.

Not sure on the best answer here, but it's definitely foot-gun that some
users have ended up using on themselves with depressing regularity.

Thanks!

Stephen

Re: Renaming of pg_xlog and pg_clog

Joshua D. Drake <jd@commandprompt.com> — 2016-10-20T16:19:42Z

On 10/20/2016 09:12 AM, Stephen Frost wrote:
> * Tom Lane (tgl@sss.pgh.pa.us) wrote:
>> Robert Haas <robertmhaas@gmail.com> writes:

> That said, I'd also like to see a --force or similar option or mechanism
> put in place to reduce the risk of users trashing their system because
> they think pg_resetwal is "safe." ("It's just gonna reset things to make
> the database start again, should be fine.").
>
> pg_destroydb almost seems like a better choice, though I suppose
> 'pg_clearwal' would be more acceptable.  Doesn't have quite the same
> impact though.

pg_dropwal

Users won't *drop* things they shouldn't on purpose (usually) but they 
will reset and will clear them. Destroydb isn't anymore accurate because 
it doesn't destroy it. Instead it makes it so I can log in again and see 
my data.

(Yes we all know the real implications with it but from a DUH user 
perspective...)

Sincerely,

JD


-- 
Command Prompt, Inc.                  http://the.postgres.company/
                         +1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Everyone appreciates your honesty, until you are honest with them.

Re: Renaming of pg_xlog and pg_clog

Robert Haas <robertmhaas@gmail.com> — 2016-10-20T16:29:47Z

On Thu, Oct 20, 2016 at 12:05 PM, Stephen Frost <sfrost@snowman.net> wrote:
>> To be honest, I don't really like either pg_transaction or pg_xact.
>
>> Neither name captures the fact that what we're really tracking here is
>> the transaction *status*.  pg_xact is slightly worse because it's a
>> poor abbreviation for transaction, but I think the argument against
>> even pg_transaction is similar to the one Tom previously levied
>> against pg_logical - viz. "logical what?".  The transaction themselves
>> are not stored in the directory, just the commit status.  The fact
>> that commit status is saved is the source of the "c" in "clog".
>
> This really needs to move forward also.
>
> When it comes to the name, I tend to think of 'pg_xact' as saying "this
> is where we persist info we need to keep about transactions."  Today
> that's just the commit status info, but I could imagine that there
> might, someday, be other things that go in there.  "pg_multixact" is
> an example of something quite similar but does have more than just one
> "thing."  Also, using "pg_xact" and then "pg_subxact" and "pg_multixact"
> bring them all under one consistent naming scheme.

I don't dispute the fact that you tend to think of it that way, but I
think it's a real stretch to say that "pg_xact" is a clear name from
the point of view of the uninitiated.  Now, maybe the point is to be a
little bit deliberately unclear, but "xact" for "transaction" is not a
lot better than "xlog" for "write-ahead log".  It's just arbitrary
abbreviations we made up and you either know what they mean or you
don't.  We could call it "pg_xkcd" and we wouldn't be removing much in
the way of clarity.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: Renaming of pg_xlog and pg_clog

Robert Haas <robertmhaas@gmail.com> — 2016-10-20T16:49:15Z

On Thu, Oct 20, 2016 at 12:12 PM, Stephen Frost <sfrost@snowman.net> wrote:
> That said, I'd also like to see a --force or similar option or mechanism
> put in place to reduce the risk of users trashing their system because
> they think pg_resetwal is "safe." ("It's just gonna reset things to make
> the database start again, should be fine.").

You know we already have that, right?

> pg_destroydb almost seems like a better choice, though I suppose
> 'pg_clearwal' would be more acceptable.  Doesn't have quite the same
> impact though.
>
> Not sure on the best answer here, but it's definitely foot-gun that some
> users have ended up using on themselves with depressing regularity.

Just to provide some perspective from the other side of this, I
handled a severity-1 escalation a few weeks ago where a user basically
called up and explained their situation and I told them based on all
of the facts and circumstances presented that running pg_resetxlog was
going to be the best way forward and they said that was pretty much
what they were expecting to hear, and did so.  I think there's far too
much anti-pg_resetxlog bias on this list.  The situation where you
need to use it is not common in general, but it's pretty common in
support cases that reach me.  I don't think I'd be exaggerating if I
said that 25% of the customer escalations I handle personally require
the use pg_resetxlog.  Of course, that's because by the time the
ticket gets to me, it's typically the case that all of the easy, neat
solutions have already been ruled out.

My experience is that users who are faced with this situation
typically understand that they are in a bad situation and when I
explain to them --- in a clear and direct way but without the sort of
hysterical exaggeration sometimes seen on this mailing list --- what
the consequences are, they are not surprised by those consequences and
they are willing to accept them because they know that the
alternatives are worse.  For example, consider a customer with a
mostly read-only 20TB database.  If that user runs pg_resetxlog, they
can be back up in 5 minutes and it is likely (though of course not
certain) that corruption will be minimal.  If they restore from
backup, they will be down for many more hours.  After pg_resetxlog,
they can restore from backup on another system or do the
dump-and-restore which I invariably recommend while they are up.  For
many customers, this is a good trade-off.  The decision, because of
the risks associated with it, is usually passed up from the DBA I'm
working with to some business leader.  If that business leader feels
that benefits of being up immediately rather than many hours later are
sufficient to justify the risk of a possibly-corrupted database,
running pg_resetxlog is a perfectly reasonable decision.

I have not yet had one DBA or business leader call back and say "hey,
that thing where we ran pg_resetxlog?  you should have discouraged us
more, because that was a disaster".  Now maybe those disasters have
happened and I am just not aware of them, but I think we should all
take a deep breath and remind ourselves that the database exists in
service to the business and not the other way around.  My job -- when
doing support -- is to tell you what your options are and what
consequences they may have, good and bad -- not to tell you how to run
your company.  Telling users that pg_resetxlog will "destroy your
database" is over-the-top hyperbole.  It's a sharp tool with risks and
benefits and it deserves to be presented exactly that way.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: Renaming of pg_xlog and pg_clog

Bruce Momjian <bruce@momjian.us> — 2016-10-20T17:39:31Z

On Thu, Oct 20, 2016 at 12:29:47PM -0400, Robert Haas wrote:
> > When it comes to the name, I tend to think of 'pg_xact' as saying "this
> > is where we persist info we need to keep about transactions."  Today
> > that's just the commit status info, but I could imagine that there
> > might, someday, be other things that go in there.  "pg_multixact" is
> > an example of something quite similar but does have more than just one
> > "thing."  Also, using "pg_xact" and then "pg_subxact" and "pg_multixact"
> > bring them all under one consistent naming scheme.
> 
> I don't dispute the fact that you tend to think of it that way, but I
> think it's a real stretch to say that "pg_xact" is a clear name from
> the point of view of the uninitiated.  Now, maybe the point is to be a
> little bit deliberately unclear, but "xact" for "transaction" is not a
> lot better than "xlog" for "write-ahead log".  It's just arbitrary
> abbreviations we made up and you either know what they mean or you
> don't.  We could call it "pg_xkcd" and we wouldn't be removing much in
> the way of clarity.

What is your suggestion for a name?  If you have none, I suggest we use
"pg_xact".

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

Re: Renaming of pg_xlog and pg_clog

Robert Haas <robertmhaas@gmail.com> — 2016-10-20T18:02:27Z

On Thu, Oct 20, 2016 at 1:39 PM, Bruce Momjian <bruce@momjian.us> wrote:
> On Thu, Oct 20, 2016 at 12:29:47PM -0400, Robert Haas wrote:
>> > When it comes to the name, I tend to think of 'pg_xact' as saying "this
>> > is where we persist info we need to keep about transactions."  Today
>> > that's just the commit status info, but I could imagine that there
>> > might, someday, be other things that go in there.  "pg_multixact" is
>> > an example of something quite similar but does have more than just one
>> > "thing."  Also, using "pg_xact" and then "pg_subxact" and "pg_multixact"
>> > bring them all under one consistent naming scheme.
>>
>> I don't dispute the fact that you tend to think of it that way, but I
>> think it's a real stretch to say that "pg_xact" is a clear name from
>> the point of view of the uninitiated.  Now, maybe the point is to be a
>> little bit deliberately unclear, but "xact" for "transaction" is not a
>> lot better than "xlog" for "write-ahead log".  It's just arbitrary
>> abbreviations we made up and you either know what they mean or you
>> don't.  We could call it "pg_xkcd" and we wouldn't be removing much in
>> the way of clarity.
>
> What is your suggestion for a name?  If you have none, I suggest we use
> "pg_xact".

I'm not sure.  pg_transaction_status would be clear, but it's long.
Is pg_xact actually better than pg_clog?

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: Renaming of pg_xlog and pg_clog

Tom Lane <tgl@sss.pgh.pa.us> — 2016-10-20T18:09:31Z

Robert Haas <robertmhaas@gmail.com> writes:
> Is pg_xact actually better than pg_clog?

Yes, because it doesn't contain the three letters "log".

We have the two precedents "pg_subtrans" and "pg_multixact", so
unless we want to get into renaming those too, I think "pg_trans"
and "pg_xact" are really the only options worth considering.

Personally I'd go for "pg_trans", but it's only a weak preference.

			regards, tom lane

Re: Renaming of pg_xlog and pg_clog

Robert Haas <robertmhaas@gmail.com> — 2016-10-20T18:15:03Z

On Thu, Oct 20, 2016 at 2:09 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Robert Haas <robertmhaas@gmail.com> writes:
>> Is pg_xact actually better than pg_clog?
>
> Yes, because it doesn't contain the three letters "log".

I figured somebody was going to say that.

> We have the two precedents "pg_subtrans" and "pg_multixact", so
> unless we want to get into renaming those too, I think "pg_trans"
> and "pg_xact" are really the only options worth considering.
>
> Personally I'd go for "pg_trans", but it's only a weak preference.

Heaven forfend we actually use enough characters to make it self-documenting.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: Renaming of pg_xlog and pg_clog

Tom Lane <tgl@sss.pgh.pa.us> — 2016-10-20T18:23:32Z

Robert Haas <robertmhaas@gmail.com> writes:
> On Thu, Oct 20, 2016 at 2:09 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> We have the two precedents "pg_subtrans" and "pg_multixact", so
>> unless we want to get into renaming those too, I think "pg_trans"
>> and "pg_xact" are really the only options worth considering.
>> 
>> Personally I'd go for "pg_trans", but it's only a weak preference.

> Heaven forfend we actually use enough characters to make it self-documenting.

$ ls $PGDATA
PG_VERSION     pg_dynshmem/   pg_notify/     pg_stat_tmp/  postgresql.auto.conf
base/          pg_hba.conf    pg_replslot/   pg_subtrans/  postgresql.conf
global/        pg_ident.conf  pg_serial/     pg_tblspc/    postmaster.opts
pg_clog/       pg_logical/    pg_snapshots/  pg_twophase/  postmaster.pid
pg_commit_ts/  pg_multixact/  pg_stat/       pg_wal/

I don't see one single one of those subdirectory names that I'd call
self-documenting.  Are you proposing we rename them all with carpal-
tunnel-syndrome-promoting names?

There's certainly some case to be made for renaming at least one of
"pg_subtrans" and "pg_multixact" so that these three similarly-purposed
subdirectories can all have similar names.  But I think on the whole
that's (a) fixing what ain't broken, and (b) making it even more unlikely
that we'll ever get to consensus on changing anything.  We've managed to
agree that we need to change the names ending in "log"; let's do that
and be happy that we've removed one foot-gun from the system.

			regards, tom lane

Re: Renaming of pg_xlog and pg_clog

Bruce Momjian <bruce@momjian.us> — 2016-10-20T18:34:16Z

On Thu, Oct 20, 2016 at 02:02:27PM -0400, Robert Haas wrote:
> On Thu, Oct 20, 2016 at 1:39 PM, Bruce Momjian <bruce@momjian.us> wrote:
> > On Thu, Oct 20, 2016 at 12:29:47PM -0400, Robert Haas wrote:
> >> > When it comes to the name, I tend to think of 'pg_xact' as saying "this
> >> > is where we persist info we need to keep about transactions."  Today
> >> > that's just the commit status info, but I could imagine that there
> >> > might, someday, be other things that go in there.  "pg_multixact" is
> >> > an example of something quite similar but does have more than just one
> >> > "thing."  Also, using "pg_xact" and then "pg_subxact" and "pg_multixact"
> >> > bring them all under one consistent naming scheme.
> >>
> >> I don't dispute the fact that you tend to think of it that way, but I
> >> think it's a real stretch to say that "pg_xact" is a clear name from
> >> the point of view of the uninitiated.  Now, maybe the point is to be a
> >> little bit deliberately unclear, but "xact" for "transaction" is not a
> >> lot better than "xlog" for "write-ahead log".  It's just arbitrary
> >> abbreviations we made up and you either know what they mean or you
> >> don't.  We could call it "pg_xkcd" and we wouldn't be removing much in
> >> the way of clarity.
> >
> > What is your suggestion for a name?  If you have none, I suggest we use
> > "pg_xact".
> 
> I'm not sure.  pg_transaction_status would be clear, but it's long.
> Is pg_xact actually better than pg_clog?

Uh, yeah, no "log".  Wasn't that the point?

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

Re: Renaming of pg_xlog and pg_clog

Robert Haas <robertmhaas@gmail.com> — 2016-10-20T18:35:14Z

On Thu, Oct 20, 2016 at 2:23 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Robert Haas <robertmhaas@gmail.com> writes:
>> On Thu, Oct 20, 2016 at 2:09 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>> We have the two precedents "pg_subtrans" and "pg_multixact", so
>>> unless we want to get into renaming those too, I think "pg_trans"
>>> and "pg_xact" are really the only options worth considering.
>>>
>>> Personally I'd go for "pg_trans", but it's only a weak preference.
>
>> Heaven forfend we actually use enough characters to make it self-documenting.
>
> $ ls $PGDATA
> PG_VERSION     pg_dynshmem/   pg_notify/     pg_stat_tmp/  postgresql.auto.conf
> base/          pg_hba.conf    pg_replslot/   pg_subtrans/  postgresql.conf
> global/        pg_ident.conf  pg_serial/     pg_tblspc/    postmaster.opts
> pg_clog/       pg_logical/    pg_snapshots/  pg_twophase/  postmaster.pid
> pg_commit_ts/  pg_multixact/  pg_stat/       pg_wal/
>
> I don't see one single one of those subdirectory names that I'd call
> self-documenting.  Are you proposing we rename them all with carpal-
> tunnel-syndrome-promoting names?

No.  Are you proposing that self-documenting names are a bad thing
rather than a good thing?

> There's certainly some case to be made for renaming at least one of
> "pg_subtrans" and "pg_multixact" so that these three similarly-purposed
> subdirectories can all have similar names.  But I think on the whole
> that's (a) fixing what ain't broken, and (b) making it even more unlikely
> that we'll ever get to consensus on changing anything.  We've managed to
> agree that we need to change the names ending in "log"; let's do that
> and be happy that we've removed one foot-gun from the system.

I agree that there is an overwhelming consensus in favor of getting
"log" out of the names, but I do not agree that the only two possible
alternative names are "pg_trans" and "pg_xact", which strikes me as
being akin to saying that the only two options for dinner tonight are
overripe peaches and lunch meats a week past the sell-by date.  If I
had to pick only between those two, I suppose I'd go with "pg_xact"
which is clear enough to someone familiar with PostgreSQL internals,
as opposed to "pg_trans" which I don't think will be clear even to
those people.  But I don't like either one very much.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: Renaming of pg_xlog and pg_clog

David Fetter <david@fetter.org> — 2016-10-20T18:52:39Z

On Thu, Oct 20, 2016 at 02:23:32PM -0400, Tom Lane wrote:
> Robert Haas <robertmhaas@gmail.com> writes:
> > On Thu, Oct 20, 2016 at 2:09 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> >> We have the two precedents "pg_subtrans" and "pg_multixact", so
> >> unless we want to get into renaming those too, I think "pg_trans"
> >> and "pg_xact" are really the only options worth considering.
> >> 
> >> Personally I'd go for "pg_trans", but it's only a weak preference.
> 
> > Heaven forfend we actually use enough characters to make it self-documenting.
> 
> $ ls $PGDATA
> PG_VERSION     pg_dynshmem/   pg_notify/     pg_stat_tmp/  postgresql.auto.conf
> base/          pg_hba.conf    pg_replslot/   pg_subtrans/  postgresql.conf
> global/        pg_ident.conf  pg_serial/     pg_tblspc/    postmaster.opts
> pg_clog/       pg_logical/    pg_snapshots/  pg_twophase/  postmaster.pid
> pg_commit_ts/  pg_multixact/  pg_stat/       pg_wal/
> 
> I don't see one single one of those subdirectory names that I'd call
> self-documenting.

That's a problem we should do something about, even if we can't do it
by renaming these all in one go.  At the very least, we can do this
for any new names.

> Are you proposing we rename them all with carpal-
> tunnel-syndrome-promoting names?

Are you saying that people are getting carpal tunnel syndrome from
hitting the tab key, which has been standard for completion in shells
for decades?  I'm pretty sure that doesn't actually happen.

> There's certainly some case to be made for renaming at least one of
> "pg_subtrans" and "pg_multixact" so that these three similarly-purposed
> subdirectories can all have similar names.  But I think on the whole
> that's (a) fixing what ain't broken, and (b) making it even more unlikely
> that we'll ever get to consensus on changing anything.  We've managed to
> agree that we need to change the names ending in "log"; let's do that
> and be happy that we've removed one foot-gun from the system.

Removing foot guns, un-sexy as it may be from a developer's
perspective, is very useful work.

Best,
David.
-- 
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778  AIM: dfetter666  Yahoo!: dfetter
Skype: davidfetter      XMPP: david(dot)fetter(at)gmail(dot)com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

Re: Renaming of pg_xlog and pg_clog

Stephen Frost <sfrost@snowman.net> — 2016-10-20T19:38:24Z

* David Fetter (david@fetter.org) wrote:
> On Thu, Oct 20, 2016 at 02:23:32PM -0400, Tom Lane wrote:
> > Robert Haas <robertmhaas@gmail.com> writes:
> > > On Thu, Oct 20, 2016 at 2:09 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> > >> We have the two precedents "pg_subtrans" and "pg_multixact", so
> > >> unless we want to get into renaming those too, I think "pg_trans"
> > >> and "pg_xact" are really the only options worth considering.
> > >> 
> > >> Personally I'd go for "pg_trans", but it's only a weak preference.
> > 
> > > Heaven forfend we actually use enough characters to make it self-documenting.
> > 
> > $ ls $PGDATA
> > PG_VERSION     pg_dynshmem/   pg_notify/     pg_stat_tmp/  postgresql.auto.conf
> > base/          pg_hba.conf    pg_replslot/   pg_subtrans/  postgresql.conf
> > global/        pg_ident.conf  pg_serial/     pg_tblspc/    postmaster.opts
> > pg_clog/       pg_logical/    pg_snapshots/  pg_twophase/  postmaster.pid
> > pg_commit_ts/  pg_multixact/  pg_stat/       pg_wal/
> > 
> > I don't see one single one of those subdirectory names that I'd call
> > self-documenting.
> 
> That's a problem we should do something about, even if we can't do it
> by renaming these all in one go.  At the very least, we can do this
> for any new names.

I disagree that excessivly long names for files or directories are
useful and should be fully self-documenting.  We should name our
directories with useful hints at what they are used for that remind
those who are knowledgable where things live.  Secondary to that is
using an approach to naming which avoid implying anything about the
directories to those who are *not* knowledgable, which leads us to the
current discussion regarding the removal of directories with 'log' in
the name.

Individuals who are not knowledgable in this area are not going to get
any more benefit from a directory named 'pg_trans' or
'pg_transaction_status' than one named 'pg_clog' or 'pg_xact' and
therefore this whole line of reasoning is a red herring.

Thanks!

Stephen

Re: Renaming of pg_xlog and pg_clog

Tom Lane <tgl@sss.pgh.pa.us> — 2016-10-20T19:46:20Z

Stephen Frost <sfrost@snowman.net> writes:
> * David Fetter (david@fetter.org) wrote:
>> On Thu, Oct 20, 2016 at 02:23:32PM -0400, Tom Lane wrote:
>>> I don't see one single one of those subdirectory names that I'd call
>>> self-documenting.

>> That's a problem we should do something about, even if we can't do it
>> by renaming these all in one go.  At the very least, we can do this
>> for any new names.

> I disagree that excessivly long names for files or directories are
> useful and should be fully self-documenting.

I'm mostly with Stephen on this.  As the names stand, they encourage
people to go look at the documentation,
https://www.postgresql.org/docs/devel/static/storage-file-layout.html
which will provide more information than you'd ever get out of any
reasonable directory name.

Having said that, I still don't like "pg_logical", but I suppose
renaming it would have more downsides than upsides.

			regards, tom lane

Re: Renaming of pg_xlog and pg_clog

Michael Paquier <michael.paquier@gmail.com> — 2016-10-21T00:00:58Z

On Fri, Oct 21, 2016 at 12:35 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> On Wed, Oct 12, 2016 at 10:22 PM, Michael Paquier
> <michael.paquier@gmail.com> wrote:
>> OK. I can live with that as well. Attached are three patches. The
>> pg_xlog -> pg_wal move, the pg_clog -> pg_transaction move, and the
>> pg_clog -> pg_xact move. Only one survivor to be chosen among the last
>> two ones.
>
> Committed 0001.

Thanks!
-- 
Michael

Re: Renaming of pg_xlog and pg_clog

Robert Haas <robertmhaas@gmail.com> — 2016-10-21T01:03:22Z

On Thu, Oct 20, 2016 at 3:46 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> I'm mostly with Stephen on this.  As the names stand, they encourage
> people to go look at the documentation,
> https://www.postgresql.org/docs/devel/static/storage-file-layout.html
> which will provide more information than you'd ever get out of any
> reasonable directory name.

Well, we could change them all to pg_a, pg_b, pg_c, pg_d, ... which
would encourage that even more strongly.  But I don't think that
proposal can be taken seriously.  Giving things meaningful names is a
good practice in almost every case.

> Having said that, I still don't like "pg_logical", but I suppose
> renaming it would have more downsides than upsides.

Remind me what your beef is?

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: Renaming of pg_xlog and pg_clog

Michael Paquier <michael.paquier@gmail.com> — 2016-10-21T01:24:56Z

On Fri, Oct 21, 2016 at 10:03 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> On Thu, Oct 20, 2016 at 3:46 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> I'm mostly with Stephen on this.  As the names stand, they encourage
>> people to go look at the documentation,
>> https://www.postgresql.org/docs/devel/static/storage-file-layout.html
>> which will provide more information than you'd ever get out of any
>> reasonable directory name.
>
> Well, we could change them all to pg_a, pg_b, pg_c, pg_d, ... which
> would encourage that even more strongly.  But I don't think that
> proposal can be taken seriously.  Giving things meaningful names is a
> good practice in almost every case.

Moving on with the topic of this thread... I would think that
"pg_xact" is what we are moving to rename pg_clog.

On top of that, after reading the thread, here are the options and
binaries that could be renamed for consistency with the renaming of
pg_xlog:
- pg_xlogdump -> pg_waldump
- pg_resetxlog -> pg_resetwal
- pg_receivexlog -> pg_receivewal
- initdb --xlogdir -> --waldir
- pg_basebackup --xlogmethod --xlogdir -> --walmethod --waldir
That's quite a number, and each change is trivial. Previous options
would be better kept for backward-compatibility. Personally, I see no
urge in changing all that and I am fine with just renaming the *log
directories of PGDATA for this thread. But as the point has been
raised, here are all things that could be done.
-- 
Michael

Re: Renaming of pg_xlog and pg_clog

David G. Johnston <david.g.johnston@gmail.com> — 2016-10-21T02:12:01Z

On Thu, Oct 20, 2016 at 6:03 PM, Robert Haas <robertmhaas@gmail.com> wrote:

> On Thu, Oct 20, 2016 at 3:46 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> > I'm mostly with Stephen on this.  As the names stand, they encourage
> > people to go look at the documentation,
> > https://www.postgresql.org/docs/devel/static/storage-file-layout.html
> > which will provide more information than you'd ever get out of any
> > reasonable directory name.
>
> Well, we could change them all to pg_a, pg_b, pg_c, pg_d, ... which
> would encourage that even more strongly.  But I don't think that
> proposal can be taken seriously.  Giving things meaningful names is a
> good practice in almost every case.
>

Those don't have the virtue of being at least somewhat 
m
nemonic
 like pg_xact.

I'll toss my lot in with Steven's and Tom's on this.

I have no problem continuing keeping with historical precedent and
allowing mnemonic abbreviations in our directory and file names at this
point.

David J.

Re: Renaming of pg_xlog and pg_clog

David Steele <david@pgmasters.net> — 2016-10-21T08:50:36Z

On 10/21/16 3:12 AM, David G. Johnston wrote:

> I have no problem continuing keeping with historical precedent and
> allowing mnemonic abbreviations in our directory and file names at this
> point.

I'm still in favor of pg_xact.  A search of the 9.6 docs brings up a 
number of hits for "xact": pg_last_xact_replay_timestamp(), 
pg_advisory_xact_lock(), pg_advisory_xact_lock_shared(), 
pg_last_committed_xact(),  pg_prepared_xacts(), etc. There are also 
numerous column names that have "xact" in them.

It's not just an arcane developer term when it shows up in a number of 
our user-facing functions/columns.

-- 
-David
david@pgmasters.net

Re: Renaming of pg_xlog and pg_clog

Stephen Frost <sfrost@snowman.net> — 2016-10-21T13:47:54Z

* Robert Haas (robertmhaas@gmail.com) wrote:
> On Thu, Oct 20, 2016 at 12:12 PM, Stephen Frost <sfrost@snowman.net> wrote:
> > That said, I'd also like to see a --force or similar option or mechanism
> > put in place to reduce the risk of users trashing their system because
> > they think pg_resetwal is "safe." ("It's just gonna reset things to make
> > the database start again, should be fine.").
> 
> You know we already have that, right?

Yes, but I was meaning an option which would be required to make
pg_resetxlog actually *do* anything.  In other words, I'd rather have it
report some info back to the user, if it's run without the
'--really-force' or what-have-you option, and only proceed with
clearing the WAL or rewriting pg_control when '--really-force' is used.

> > pg_destroydb almost seems like a better choice, though I suppose
> > 'pg_clearwal' would be more acceptable.  Doesn't have quite the same
> > impact though.
> >
> > Not sure on the best answer here, but it's definitely foot-gun that some
> > users have ended up using on themselves with depressing regularity.
> 
> Just to provide some perspective from the other side of this, I
[...]

I wasn't suggesting that we remove the capability.  There are certainly
use-cases for it, but, unfortunately, I've seen a number of cases where
users simply google'd an error that they got back when trying to start
PG and found someone saying "well, I got this error, but then I ran
pg_resetxlog, and now the database starts up again."

It likely doesn't help that the top links tend to be to mailing list
archives where pg_resetxlog was brought up.

Thanks!

Stephen

Re: Renaming of pg_xlog and pg_clog

Robert Haas <robertmhaas@gmail.com> — 2016-10-21T17:44:10Z

On Fri, Oct 21, 2016 at 9:47 AM, Stephen Frost <sfrost@snowman.net> wrote:
> * Robert Haas (robertmhaas@gmail.com) wrote:
>> On Thu, Oct 20, 2016 at 12:12 PM, Stephen Frost <sfrost@snowman.net> wrote:
>> > That said, I'd also like to see a --force or similar option or mechanism
>> > put in place to reduce the risk of users trashing their system because
>> > they think pg_resetwal is "safe." ("It's just gonna reset things to make
>> > the database start again, should be fine.").
>>
>> You know we already have that, right?
>
> Yes, but I was meaning an option which would be required to make
> pg_resetxlog actually *do* anything.  In other words, I'd rather have it
> report some info back to the user, if it's run without the
> '--really-force' or what-have-you option, and only proceed with
> clearing the WAL or rewriting pg_control when '--really-force' is used.

I don't think that the problem is that people are accidentally typing
"pg_resetxlog $PGDATA" and pressing return.  They're typing that on
purpose, and if you change the sequence of characters required to get
that effect, they'll just type the new thing instead. The problem is
that they don't understand when it's appropriate to use that command
and what the consequences are.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: Renaming of pg_xlog and pg_clog

Stephen Frost <sfrost@snowman.net> — 2016-10-21T18:03:27Z

* Robert Haas (robertmhaas@gmail.com) wrote:
> I don't think that the problem is that people are accidentally typing
> "pg_resetxlog $PGDATA" and pressing return.  They're typing that on
> purpose, and if you change the sequence of characters required to get
> that effect, they'll just type the new thing instead. The problem is
> that they don't understand when it's appropriate to use that command
> and what the consequences are.

I agree that they don't understand, and that's why I believe that making
the command name, or a required option, a bit more ominous would make
them pause and realize that maybe they want to consider other options
first.

This is not exactly unheard of- apt-get requires an entire phrase be to
be entered when you're asking it to do something extremely dangerous
(remove an essential package):

---------------
root@ionith:~# apt-get remove login
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages will be REMOVED:
  login
WARNING: The following essential packages will be removed.
This should NOT be done unless you know exactly what you are doing!
  login
0 upgraded, 0 newly installed, 1 to remove and 71 not upgraded.
After this operation, 1,212 kB disk space will be freed.
You are about to do something potentially harmful.
To continue type in the phrase 'Yes, do as I say!'
 ?] 
---------------

My thought would be to make pg_resetxlog do something more along the
lines of what pg_control does and have it, by default, just investigate
the state of things and complain about problems and then have it include
an option to actually reset things with an appropriately ominous name.
The goal there being, primairly, to give a way for users to get
information about why PG isn't starting or what it is complaining about,
with some additional information about what happens if they forcibly
reset the xlog, noting that it could very likely cause corruption, etc.

Thanks!

Stephen

Re: Renaming of pg_xlog and pg_clog

Alvaro Herrera <alvherre@2ndquadrant.com> — 2016-10-21T19:29:01Z

David Steele wrote:
> On 10/21/16 3:12 AM, David G. Johnston wrote:
> 
> > I have no problem continuing keeping with historical precedent and
> > allowing mnemonic abbreviations in our directory and file names at this
> > point.
> 
> I'm still in favor of pg_xact.  A search of the 9.6 docs brings up a number
> of hits for "xact": pg_last_xact_replay_timestamp(),
> pg_advisory_xact_lock(), pg_advisory_xact_lock_shared(),
> pg_last_committed_xact(),  pg_prepared_xacts(), etc. There are also numerous
> column names that have "xact" in them.

I'm +1 on pg_clog -> pg_xact.

Also +1 to renaming pg_subtrans to pg_subxact.

In general, +1 to short mnemonic meaningful names.  -1 to overly long
names, -1 to meaningless names.

Regarding pg_receivexlog I think I'd propose pg_recvwal rather than
pg_receivewal, because of pg_recvlogical.

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Renaming of pg_xlog and pg_clog

Michael Paquier <michael.paquier@gmail.com> — 2016-10-21T22:33:56Z

On Sat, Oct 22, 2016 at 4:29 AM, Alvaro Herrera
<alvherre@2ndquadrant.com> wrote:
> David Steele wrote:
>> On 10/21/16 3:12 AM, David G. Johnston wrote:
>>
>> > I have no problem continuing keeping with historical precedent and
>> > allowing mnemonic abbreviations in our directory and file names at this
>> > point.
>>
>> I'm still in favor of pg_xact.  A search of the 9.6 docs brings up a number
>> of hits for "xact": pg_last_xact_replay_timestamp(),
>> pg_advisory_xact_lock(), pg_advisory_xact_lock_shared(),
>> pg_last_committed_xact(),  pg_prepared_xacts(), etc. There are also numerous
>> column names that have "xact" in them.
>
> I'm +1 on pg_clog -> pg_xact.

So let's say that's the winner then.

> Also +1 to renaming pg_subtrans to pg_subxact.

Nice suggestion, good naming for consistency with the rest.
-- 
Michael

Re: Renaming of pg_xlog and pg_clog

Bruce Momjian <bruce@momjian.us> — 2016-10-22T15:58:09Z

On Sat, Oct 22, 2016 at 07:33:56AM +0900, Michael Paquier wrote:
> On Sat, Oct 22, 2016 at 4:29 AM, Alvaro Herrera
> <alvherre@2ndquadrant.com> wrote:
> > David Steele wrote:
> >> On 10/21/16 3:12 AM, David G. Johnston wrote:
> >>
> >> > I have no problem continuing keeping with historical precedent and
> >> > allowing mnemonic abbreviations in our directory and file names at this
> >> > point.
> >>
> >> I'm still in favor of pg_xact.  A search of the 9.6 docs brings up a number
> >> of hits for "xact": pg_last_xact_replay_timestamp(),
> >> pg_advisory_xact_lock(), pg_advisory_xact_lock_shared(),
> >> pg_last_committed_xact(),  pg_prepared_xacts(), etc. There are also numerous
> >> column names that have "xact" in them.
> >
> > I'm +1 on pg_clog -> pg_xact.
> 
> So let's say that's the winner then.
> 
> > Also +1 to renaming pg_subtrans to pg_subxact.
> 
> Nice suggestion, good naming for consistency with the rest.

Agreed.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

Re: Renaming of pg_xlog and pg_clog

David Steele <david@pgmasters.net> — 2016-10-22T16:00:58Z

On 10/22/16 6:58 PM, Bruce Momjian wrote:
> On Sat, Oct 22, 2016 at 07:33:56AM +0900, Michael Paquier wrote:
>> On Sat, Oct 22, 2016 at 4:29 AM, Alvaro Herrera
>>
>>> Also +1 to renaming pg_subtrans to pg_subxact.
>>
>> Nice suggestion, good naming for consistency with the rest.
>
> Agreed.

+1

-- 
-David
david@pgmasters.net

Re: Renaming of pg_xlog and pg_clog

Greg Stark <stark@mit.edu> — 2016-10-22T20:18:28Z

On Fri, Oct 21, 2016 at 9:03 PM, Stephen Frost <sfrost@snowman.net> wrote:
> WARNING: The following essential packages will be removed.
> This should NOT be done unless you know exactly what you are doing!
>   login
> 0 upgraded, 0 newly installed, 1 to remove and 71 not upgraded.
> After this operation, 1,212 kB disk space will be freed.
> You are about to do something potentially harmful.
> To continue type in the phrase 'Yes, do as I say!'

Another case was:
   glxgears -iacknowledgethatthistoolisnotabenchmark

Which was required to get it to print the FPS for a while. The problem
-- and also the advantage -- of this is that it's scriptable. That
means people can still put it in recipe books and scripts and others
can copy it without being aware what it does or even that they're
doing it.

I think the apt-get behaviour was specifically designed to ensure it
couldn't easily be put into a script which I would have said was
desirable -- except I suspect there are situations where Postgres
database scripts need to do a resetxlog. I'm not sure I can think of
any examples offhand but I wouldn't be too surprised.



-- 
greg

Re: Renaming of pg_xlog and pg_clog

Vik Fearing <vik@2ndquadrant.fr> — 2016-10-23T08:18:16Z

On 10/22/2016 06:00 PM, David Steele wrote:
> On 10/22/16 6:58 PM, Bruce Momjian wrote:
>> On Sat, Oct 22, 2016 at 07:33:56AM +0900, Michael Paquier wrote:
>>> On Sat, Oct 22, 2016 at 4:29 AM, Alvaro Herrera
>>>
>>>> Also +1 to renaming pg_subtrans to pg_subxact.
>>>
>>> Nice suggestion, good naming for consistency with the rest.
>>
>> Agreed.
> 
> +1

+1 from me, too.
-- 
Vik Fearing                                          +33 6 46 75 15 36
http://2ndQuadrant.fr     PostgreSQL : Expertise, Formation et Support

Re: Renaming of pg_xlog and pg_clog

Michael Paquier <michael.paquier@gmail.com> — 2016-10-23T12:34:07Z

On Sun, Oct 23, 2016 at 5:18 PM, Vik Fearing <vik@2ndquadrant.fr> wrote:
> On 10/22/2016 06:00 PM, David Steele wrote:
>> On 10/22/16 6:58 PM, Bruce Momjian wrote:
>>> On Sat, Oct 22, 2016 at 07:33:56AM +0900, Michael Paquier wrote:
>>>> On Sat, Oct 22, 2016 at 4:29 AM, Alvaro Herrera
>>>>
>>>>> Also +1 to renaming pg_subtrans to pg_subxact.
>>>>
>>>> Nice suggestion, good naming for consistency with the rest.
>>>
>>> Agreed.
>>
>> +1
>
> +1 from me, too.

OK, glad to see that we are reaching a conclusion to this thread.
Attached are two patches. 0001 renames pg_clog to pg_xact. I found a
comment in xact.c that was not yet updated. And 0002 renames
pg_subtrans to pg_subxact.
-- 
Michael

Re: Renaming of pg_xlog and pg_clog

Bruce Momjian <bruce@momjian.us> — 2016-10-24T13:37:59Z

On Sat, Oct 22, 2016 at 11:18:28PM +0300, Greg Stark wrote:
> On Fri, Oct 21, 2016 at 9:03 PM, Stephen Frost <sfrost@snowman.net> wrote:
> > WARNING: The following essential packages will be removed.
> > This should NOT be done unless you know exactly what you are doing!
> >   login
> > 0 upgraded, 0 newly installed, 1 to remove and 71 not upgraded.
> > After this operation, 1,212 kB disk space will be freed.
> > You are about to do something potentially harmful.
> > To continue type in the phrase 'Yes, do as I say!'
> 
> Another case was:
>    glxgears -iacknowledgethatthistoolisnotabenchmark
> 
> Which was required to get it to print the FPS for a while. The problem
> -- and also the advantage -- of this is that it's scriptable. That
> means people can still put it in recipe books and scripts and others
> can copy it without being aware what it does or even that they're
> doing it.
> 
> I think the apt-get behaviour was specifically designed to ensure it
> couldn't easily be put into a script which I would have said was
> desirable -- except I suspect there are situations where Postgres
> database scripts need to do a resetxlog. I'm not sure I can think of
> any examples offhand but I wouldn't be too surprised.

Yes, pg_upgrade has eight calls to pg_resetxlog to set various value.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

Re: Renaming of pg_xlog and pg_clog

Alvaro Herrera <alvherre@2ndquadrant.com> — 2016-10-24T14:59:42Z

Bruce Momjian wrote:
> On Sat, Oct 22, 2016 at 11:18:28PM +0300, Greg Stark wrote:

> > I think the apt-get behaviour was specifically designed to ensure it
> > couldn't easily be put into a script which I would have said was
> > desirable -- except I suspect there are situations where Postgres
> > database scripts need to do a resetxlog. I'm not sure I can think of
> > any examples offhand but I wouldn't be too surprised.
> 
> Yes, pg_upgrade has eight calls to pg_resetxlog to set various value.

There is one difference though, which is that the really destructive
use of pg_resetxlog is the one that removes pg_xlog files.  The other
uses that simply set flags in the control file are not as bad -- you can
simply go back to what the original value was.  I think we would only
need the obnoxious string requirement in the most dangerous mode.

Alternatively, we could separate the tool in two different executables.

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Renaming of pg_xlog and pg_clog

Bruce Momjian <bruce@momjian.us> — 2016-10-24T15:06:27Z

On Mon, Oct 24, 2016 at 11:59:42AM -0300, Alvaro Herrera wrote:
> Bruce Momjian wrote:
> > On Sat, Oct 22, 2016 at 11:18:28PM +0300, Greg Stark wrote:
> 
> > > I think the apt-get behaviour was specifically designed to ensure it
> > > couldn't easily be put into a script which I would have said was
> > > desirable -- except I suspect there are situations where Postgres
> > > database scripts need to do a resetxlog. I'm not sure I can think of
> > > any examples offhand but I wouldn't be too surprised.
> > 
> > Yes, pg_upgrade has eight calls to pg_resetxlog to set various value.
> 
> There is one difference though, which is that the really destructive
> use of pg_resetxlog is the one that removes pg_xlog files.  The other
> uses that simply set flags in the control file are not as bad -- you can
> simply go back to what the original value was.  I think we would only
> need the obnoxious string requirement in the most dangerous mode.
> 
> Alternatively, we could separate the tool in two different executables.

OK, but we are going need to document that this special flag is only
needed for certain option uses.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

Re: Renaming of pg_xlog and pg_clog

Alvaro Herrera <alvherre@2ndquadrant.com> — 2016-10-24T18:26:33Z

Bruce Momjian wrote:
> On Mon, Oct 24, 2016 at 11:59:42AM -0300, Alvaro Herrera wrote:

> > There is one difference though, which is that the really destructive
> > use of pg_resetxlog is the one that removes pg_xlog files.  The other
> > uses that simply set flags in the control file are not as bad -- you can
> > simply go back to what the original value was.  I think we would only
> > need the obnoxious string requirement in the most dangerous mode.

> OK, but we are going need to document that this special flag is only
> needed for certain option uses.

Perhaps, but not in too much detail; just that "users need to follow
on-screen indications" or some such would be enough.

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Renaming of pg_xlog and pg_clog

Haribabu Kommi <kommi.haribabu@gmail.com> — 2016-11-22T11:35:01Z

Hi Craig,

This is a gentle reminder.

you assigned as reviewer to the current patch in the 11-2016 commitfest.
But you haven't shared your review yet. Please share your review about
the patch. This will help us in smoother operation of commitfest.

Please Ignore if you already shared your review.


Regards,
Hari Babu
Fujitsu Australia

Re: Renaming of pg_xlog and pg_clog

Michael Paquier <michael.paquier@gmail.com> — 2016-11-29T04:35:13Z

On Tue, Nov 22, 2016 at 8:35 PM, Haribabu Kommi
<kommi.haribabu@gmail.com> wrote:
> Hi Craig,
>
> This is a gentle reminder.
>
> you assigned as reviewer to the current patch in the 11-2016 commitfest.
> But you haven't shared your review yet. Please share your review about
> the patch. This will help us in smoother operation of commitfest.
>
> Please Ignore if you already shared your review.

I have moved this CF entry to 2017-01, the remaining, still unreviewed
patch are for renaming pg_subxact and pg_clog.
-- 
Michael

Re: Renaming of pg_xlog and pg_clog

Michael Paquier <michael.paquier@gmail.com> — 2017-01-17T07:31:00Z

On Tue, Nov 29, 2016 at 1:35 PM, Michael Paquier
<michael.paquier@gmail.com> wrote:
> On Tue, Nov 22, 2016 at 8:35 PM, Haribabu Kommi
> <kommi.haribabu@gmail.com> wrote:
>> Hi Craig,
>>
>> This is a gentle reminder.
>>
>> you assigned as reviewer to the current patch in the 11-2016 commitfest.
>> But you haven't shared your review yet. Please share your review about
>> the patch. This will help us in smoother operation of commitfest.
>>
>> Please Ignore if you already shared your review.
>
> I have moved this CF entry to 2017-01, the remaining, still unreviewed
> patch are for renaming pg_subxact and pg_clog.

The second patch has rotten a little because of the backup
documentation. By the way, is something going to happen in the CF?
Craig, you are a reviewer of this patch.
-- 
Michael

Re: Renaming of pg_xlog and pg_clog

Michael Paquier <michael.paquier@gmail.com> — 2017-01-31T04:45:36Z

On Tue, Jan 17, 2017 at 4:31 PM, Michael Paquier
<michael.paquier@gmail.com> wrote:
> On Tue, Nov 29, 2016 at 1:35 PM, Michael Paquier
> <michael.paquier@gmail.com> wrote:
>> On Tue, Nov 22, 2016 at 8:35 PM, Haribabu Kommi
>> <kommi.haribabu@gmail.com> wrote:
>>> Hi Craig,
>>>
>>> This is a gentle reminder.
>>>
>>> you assigned as reviewer to the current patch in the 11-2016 commitfest.
>>> But you haven't shared your review yet. Please share your review about
>>> the patch. This will help us in smoother operation of commitfest.
>>>
>>> Please Ignore if you already shared your review.
>>
>> I have moved this CF entry to 2017-01, the remaining, still unreviewed
>> patch are for renaming pg_subxact and pg_clog.
>
> The second patch has rotten a little because of the backup
> documentation. By the way, is something going to happen in the CF?
> Craig, you are a reviewer of this patch.

Moved to CF 2017-03.
-- 
Michael

Re: Renaming of pg_xlog and pg_clog

David Steele <david@pgmasters.net> — 2017-03-16T15:19:54Z

On 1/17/17 2:31 AM, Michael Paquier wrote:
> On Tue, Nov 29, 2016 at 1:35 PM, Michael Paquier
> <michael.paquier@gmail.com> wrote:
>> On Tue, Nov 22, 2016 at 8:35 PM, Haribabu Kommi
>> <kommi.haribabu@gmail.com> wrote:
>>> Hi Craig,
>>>
>>> This is a gentle reminder.
>>>
>>> you assigned as reviewer to the current patch in the 11-2016 commitfest.
>>> But you haven't shared your review yet. Please share your review about
>>> the patch. This will help us in smoother operation of commitfest.
>>>
>>> Please Ignore if you already shared your review.
>>
>> I have moved this CF entry to 2017-01, the remaining, still unreviewed
>> patch are for renaming pg_subxact and pg_clog.
> 
> The second patch has rotten a little because of the backup
> documentation. By the way, is something going to happen in the CF?
> Craig, you are a reviewer of this patch.

This patch does not apply cleanly at cccbdde:

$ git apply ../other/0001-Rename-pg_clog-to-pg_xact.patch
error: doc/src/sgml/ref/pg_resetxlog.sgml: No such file or directory
error: patch failed: src/backend/postmaster/autovacuum.c:2468
error: src/backend/postmaster/autovacuum.c: patch does not apply

Marked "Waiting on Author".

I'd really like to see the rest of the renames happen for v10.  It seems
like the process got stalled after the pg_wal rename.

-- 
-David
david@pgmasters.net

Re: Renaming of pg_xlog and pg_clog

Michael Paquier <michael.paquier@gmail.com> — 2017-03-17T01:31:26Z

On Fri, Mar 17, 2017 at 12:19 AM, David Steele <david@pgmasters.net> wrote:
> This patch does not apply cleanly at cccbdde:
>
> $ git apply ../other/0001-Rename-pg_clog-to-pg_xact.patch
> error: doc/src/sgml/ref/pg_resetxlog.sgml: No such file or directory
> error: patch failed: src/backend/postmaster/autovacuum.c:2468
> error: src/backend/postmaster/autovacuum.c: patch does not apply

This has rotten again...

> Marked "Waiting on Author".
>
> I'd really like to see the rest of the renames happen for v10.  It seems
> like the process got stalled after the pg_wal rename.

Let's see what happens, attached are refreshed versions. Uncertainty
is the fun part of a CF.
-- 
Michael

Re: Renaming of pg_xlog and pg_clog

Robert Haas <robertmhaas@gmail.com> — 2017-03-17T02:17:49Z

On Thu, Mar 16, 2017 at 9:31 PM, Michael Paquier
<michael.paquier@gmail.com> wrote:
> On Fri, Mar 17, 2017 at 12:19 AM, David Steele <david@pgmasters.net> wrote:
>> This patch does not apply cleanly at cccbdde:
>>
>> $ git apply ../other/0001-Rename-pg_clog-to-pg_xact.patch
>> error: doc/src/sgml/ref/pg_resetxlog.sgml: No such file or directory
>> error: patch failed: src/backend/postmaster/autovacuum.c:2468
>> error: src/backend/postmaster/autovacuum.c: patch does not apply
>
> This has rotten again...
>
>> Marked "Waiting on Author".
>>
>> I'd really like to see the rest of the renames happen for v10.  It seems
>> like the process got stalled after the pg_wal rename.
>
> Let's see what happens, attached are refreshed versions. Uncertainty
> is the fun part of a CF.

I understand that the point of renaming pg_clog to pg_xact is that
pg_clog contains the dreaded letters l-o-g, which we hypothesize
causes DBAs to remove it.  (Alternate hypothesis: "So, that's what's
clogging my database!")

Renaming pg_subtrans to pg_subxact has no such redeeming properties.

More, with each of these renamings, we're further separating what
things are called in the code (xlog, clog, subtrans) with what they're
called in the filesystem (wal, xact, subxact).

So if we must rename pg_clog, OK, but can't we leave pg_subtrans
alone?  It's not hurting anybody.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: Renaming of pg_xlog and pg_clog

Michael Paquier <michael.paquier@gmail.com> — 2017-03-17T02:21:02Z

On Fri, Mar 17, 2017 at 11:17 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> I understand that the point of renaming pg_clog to pg_xact is that
> pg_clog contains the dreaded letters l-o-g, which we hypothesize
> causes DBAs to remove it.  (Alternate hypothesis: "So, that's what's
> clogging my database!")
>
> Renaming pg_subtrans to pg_subxact has no such redeeming properties.
>
> More, with each of these renamings, we're further separating what
> things are called in the code (xlog, clog, subtrans) with what they're
> called in the filesystem (wal, xact, subxact).
>
> So if we must rename pg_clog, OK, but can't we leave pg_subtrans
> alone?  It's not hurting anybody.

The only argument behind the renaming of pg_subtrans is really
consistency with pg_xact, because both deal with transactions. I don't
personally mind if this portion of the renaming is left off, as you
say anything labelled with "log" is at the origin of this thread.
-- 
Michael

Re: Renaming of pg_xlog and pg_clog

Robert Haas <robertmhaas@gmail.com> — 2017-03-17T13:58:36Z

On Thu, Mar 16, 2017 at 10:21 PM, Michael Paquier
<michael.paquier@gmail.com> wrote:
> On Fri, Mar 17, 2017 at 11:17 AM, Robert Haas <robertmhaas@gmail.com> wrote:
>> I understand that the point of renaming pg_clog to pg_xact is that
>> pg_clog contains the dreaded letters l-o-g, which we hypothesize
>> causes DBAs to remove it.  (Alternate hypothesis: "So, that's what's
>> clogging my database!")
>>
>> Renaming pg_subtrans to pg_subxact has no such redeeming properties.
>>
>> More, with each of these renamings, we're further separating what
>> things are called in the code (xlog, clog, subtrans) with what they're
>> called in the filesystem (wal, xact, subxact).
>>
>> So if we must rename pg_clog, OK, but can't we leave pg_subtrans
>> alone?  It's not hurting anybody.
>
> The only argument behind the renaming of pg_subtrans is really
> consistency with pg_xact, because both deal with transactions. I don't
> personally mind if this portion of the renaming is left off, as you
> say anything labelled with "log" is at the origin of this thread.

Fine!  I've committed the pg_clog renaming, but I'd really like to
draw the line here.  I'm not going to commit the pg_subtrans ->
pg_subxact naming and am -1 on anyone else doing so.  I think that
having the names of things in the code match what shows up in the file
system is an awfully desirable property which we should preserve
insofar as we can do so without compromising other goals.  Not
invalidating what users and DBAs think they know about how PostgreSQL
works is a good thing, too; there are a lot more people who know
something about the directory layout than will read this thread.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: Renaming of pg_xlog and pg_clog

Michael Paquier <michael.paquier@gmail.com> — 2017-03-17T14:03:08Z

On Fri, Mar 17, 2017 at 10:58 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> Fine!  I've committed the pg_clog renaming, but I'd really like to
> draw the line here.  I'm not going to commit the pg_subtrans ->
> pg_subxact naming and am -1 on anyone else doing so.  I think that
> having the names of things in the code match what shows up in the file
> system is an awfully desirable property which we should preserve
> insofar as we can do so without compromising other goals.  Not
> invalidating what users and DBAs think they know about how PostgreSQL
> works is a good thing, too; there are a lot more people who know
> something about the directory layout than will read this thread.

I'm cool with that, thanks for the commit.
-- 
Michael

Re: Renaming of pg_xlog and pg_clog

Fujii Masao <masao.fujii@gmail.com> — 2017-03-18T18:03:37Z

On Fri, Mar 17, 2017 at 11:03 PM, Michael Paquier
<michael.paquier@gmail.com> wrote:
> On Fri, Mar 17, 2017 at 10:58 PM, Robert Haas <robertmhaas@gmail.com> wrote:
>> Fine!  I've committed the pg_clog renaming, but I'd really like to
>> draw the line here.  I'm not going to commit the pg_subtrans ->
>> pg_subxact naming and am -1 on anyone else doing so.  I think that
>> having the names of things in the code match what shows up in the file
>> system is an awfully desirable property which we should preserve
>> insofar as we can do so without compromising other goals.  Not
>> invalidating what users and DBAs think they know about how PostgreSQL
>> works is a good thing, too; there are a lot more people who know
>> something about the directory layout than will read this thread.
>
> I'm cool with that, thanks for the commit.

monitoring.sgml still has some references to "clog". We should update them?

Regards,

-- 
Fujii Masao