Horton

Thread

Commits

  1. Manually un-break a few URLs that pgindent used to insist on splitting.

  2. Remove entab and associated detritus.

  3. Phase 3 of pgindent updates.

  4. Phase 2 of pgindent updates.

  5. Initial pgindent run with pg_bsd_indent version 2.0.

  6. Adjust pgindent script to use pg_bsd_indent 2.0.

  7. Final pgindent run with old pg_bsd_indent (version 1.3).

  1. Preliminary results for proposed new pgindent implementation

    Tom Lane <tgl@sss.pgh.pa.us> — 2017-05-19T03:00:40Z

    Over in this thread:
https://www.postgresql.org/message-id/flat/E1dAmxK-0006EE-1r%40gemulon.postgresql.org
we've been discussing switching to FreeBSD's version of "indent",
specifically the updated version that Piotr Stefaniak is working on,
as the basis for pgindent.  There seem to be a small number of bugs
that Piotr needs to fix, but overall it is looking remarkably good
to me.

To create a diff with not too much noise in it, I applied the hacks
(fixes?) already mentioned in the other thread, and I tweaked things
slightly to suppress changes in alignment of comments on #else and
#endif lines.  We might well want to reconsider that later, because it's
quite random right now (basically, #else comments are indented to column
33, #endif comments are indented to column 10, and for no reason at all
the latter do not have standard tab substitution).  But again, I'm trying
to drill down to the changes we want rather than the incidental ones.

At this point, the changes look pretty good, although still bulky.
I've attached a diff between current HEAD and re-indented HEAD
for anybody who wants to peruse all the details, but basically
what I'm seeing is:

* Improvements in formatting around sizeof and related constructs,
for example:

-       *phoned_word = palloc(sizeof(char) * strlen(word) +1);
+       *phoned_word = palloc(sizeof(char) * strlen(word) + 1);

* Likewise, operators after casts work better than before:

        res = shm_mq_send_bytes(mqh, sizeof(Size) - mqh->mqh_partial_bytes,
-                               ((char *) &nbytes) +mqh->mqh_partial_bytes,
+                               ((char *) &nbytes) + mqh->mqh_partial_bytes,
                                nowait, &bytes_written);

* Sane formatting of function typedefs, for example:

     * use buf as work area if NULL in-place copy
     */
    int         (*pull) (void *priv, PullFilter *src, int len,
-                                    uint8 **data_p, uint8 *buf, int buflen);
+                        uint8 **data_p, uint8 *buf, int buflen);
    void        (*free) (void *priv);
 };

* Non-typedef struct pointers are now formatted consistently, for example:
 
-mdcbuf_finish(struct MDCBufData * st)
+mdcbuf_finish(struct MDCBufData *st)

* Better handling of pointers with const/volatile qualifiers, for example:

-static bool shm_mq_wait_internal(volatile shm_mq *mq, PGPROC *volatile * ptr,
+static bool shm_mq_wait_internal(volatile shm_mq *mq, PGPROC *volatile *ptr,

* Better handling of PG_USED_FOR_ASSERTS_ONLY declarations, for example

    BlockNumber blkno;
    Buffer      buf;
-   Bucket new_bucket PG_USED_FOR_ASSERTS_ONLY = InvalidBucket;
+   Bucket      new_bucket PG_USED_FOR_ASSERTS_ONLY = InvalidBucket;
    bool        bucket_dirty = false;

* Corner cases where no space was left before a comment are fixed:

@@ -149,12 +149,12 @@ typedef struct RewriteStateData
    bool        rs_logical_rewrite;     /* do we need to do logical rewriting */
    TransactionId rs_oldest_xmin;       /* oldest xmin used by caller to
                                         * determine tuple visibility */
-   TransactionId rs_freeze_xid;/* Xid that will be used as freeze cutoff
-                                * point */
+   TransactionId rs_freeze_xid;    /* Xid that will be used as freeze cutoff
+                                    * point */
    TransactionId rs_logical_xmin;      /* Xid that will be used as cutoff
                                         * point for logical rewrites */
-   MultiXactId rs_cutoff_multi;/* MultiXactId that will be used as cutoff
-                                * point for multixacts */
+   MultiXactId rs_cutoff_multi;    /* MultiXactId that will be used as cutoff
+                                    * point for multixacts */
    MemoryContext rs_cxt;       /* for hash tables and entries and tuples in
                                 * them */
    XLogRecPtr  rs_begin_lsn;   /* XLogInsertLsn when starting the rewrite */


There are also changes that are not really wins, just different, but they
do arguably improve consistency.  One is that comments following "else"
and "case" are now always indented at least as far as column 33 (or
indent's -c parameter), whereas the old code would sometimes let them
slide to the left of that:

@@ -723,7 +723,7 @@ dblink_record_internal(FunctionCallInfo fcinfo, bool is_async)
                /* shouldn't happen */
                elog(ERROR, "wrong number of arguments");
        }
-       else    /* is_async */
+       else                    /* is_async */
        {
            /* get async result */
            conname = text_to_cstring(PG_GETARG_TEXT_PP(0));

This also happens for comments on a function's "}" close brace:

@@ -722,7 +722,7 @@ _metaphone(char *word,          /* IN */
    End_Phoned_Word;
 
    return (META_SUCCESS);
-}  /* END metaphone */
+}                              /* END metaphone */

It's hard to see these as anything except bug fixes, because the
old indent code certainly looks like it intends to always force
same-line columns to at least the -c column.  I haven't quite figured
out why it fails to, or where the new version fixed that.  Still,
it's a difference that isn't a particular benefit to us.

Another set of changes is slightly different handling of unrecognized
typedef names:

@@ -250,7 +250,7 @@ typedef enum
    PGSS_TRACK_NONE,            /* track no statements */
    PGSS_TRACK_TOP,             /* only top level statements */
    PGSS_TRACK_ALL              /* all statements, including nested ones */
-}  PGSSTrackLevel;
+}          PGSSTrackLevel;

The reason PGSSTrackLevel is "unrecognized" is that it's not in
typedefs.list, which is a deficiency in our typedef-collection
technology not in indent.  (I believe the problem is that there
are no variables declared with that typename, causing there to
not be any of the kind of symbol table entries we are looking for.)
The handling of such names was already slightly wonky, though;
note that the previous version was already differently indented
from what would happen if PGSSTrackLevel were known.

Another issue I'm noticing is that "extern C" declarations are
getting reformatted:

diff --git a/src/interfaces/ecpg/include/ecpgtype.h b/src/interfaces/ecpg/include/ecpgtype.h
index 7cc47e9..236d028 100644
--- a/src/interfaces/ecpg/include/ecpgtype.h
+++ b/src/interfaces/ecpg/include/ecpgtype.h
@@ -34,7 +34,7 @@
 #define _ECPGTYPE_H
 
 #ifdef __cplusplus
-extern     "C"
+extern "C"
 {
 #endif
 

The pgindent Perl script is fooling around with these already, and
I think what it's doing is probably not quite compatible with what
the new indent version does, but I haven't tracked it down yet.

All in all, this looks pretty darn good from here, and I'm thinking
we should push forward on it.

			regards, tom lane

  2. Re: Preliminary results for proposed new pgindent implementation

    Stephen Frost <sfrost@snowman.net> — 2017-05-19T11:03:54Z

    Tom, all,

* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> All in all, this looks pretty darn good from here, and I'm thinking
> we should push forward on it.

+1.

Thanks!

Stephen

  3. Re: Preliminary results for proposed new pgindent implementation

    Robert Haas <robertmhaas@gmail.com> — 2017-05-19T12:20:22Z

    On Thu, May 18, 2017 at 11:00 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> * Improvements in formatting around sizeof and related constructs,
> for example:
>
> * Likewise, operators after casts work better than before:
>
> * Sane formatting of function typedefs, for example:
>
> * Non-typedef struct pointers are now formatted consistently, for example:
>
> * Better handling of pointers with const/volatile qualifiers, for example:
>
> * Better handling of PG_USED_FOR_ASSERTS_ONLY declarations, for example
>
> * Corner cases where no space was left before a comment are fixed:

Those all sound like good things.

> Another set of changes is slightly different handling of unrecognized
> typedef names:
>
> @@ -250,7 +250,7 @@ typedef enum
>     PGSS_TRACK_NONE,            /* track no statements */
>     PGSS_TRACK_TOP,             /* only top level statements */
>     PGSS_TRACK_ALL              /* all statements, including nested ones */
> -}  PGSSTrackLevel;
> +}          PGSSTrackLevel;
>
> The reason PGSSTrackLevel is "unrecognized" is that it's not in
> typedefs.list, which is a deficiency in our typedef-collection
> technology not in indent.  (I believe the problem is that there
> are no variables declared with that typename, causing there to
> not be any of the kind of symbol table entries we are looking for.)
> The handling of such names was already slightly wonky, though;
> note that the previous version was already differently indented
> from what would happen if PGSSTrackLevel were known.

This, however, doesn't sound so good.  Isn't there some way this can be fixed?

> All in all, this looks pretty darn good from here, and I'm thinking
> we should push forward on it.

What does that exactly mean concretely?

We've talked about pulling pgindent into our main repo, or posting a
link to a tarball someplace.  An intermediate plan might be to give it
its own repo, but on git.postgresql.org, which seems like it might
give us the best of both worlds.  But I really want something that's
going to be easy to set up and configure.  It took me years to be
brave enough to get the current pgindent set up.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  4. Re: Preliminary results for proposed new pgindent implementation

    Tom Lane <tgl@sss.pgh.pa.us> — 2017-05-19T14:21:08Z

    Robert Haas <robertmhaas@gmail.com> writes:
> On Thu, May 18, 2017 at 11:00 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> The reason PGSSTrackLevel is "unrecognized" is that it's not in
>> typedefs.list, which is a deficiency in our typedef-collection
>> technology not in indent.  (I believe the problem is that there
>> are no variables declared with that typename, causing there to
>> not be any of the kind of symbol table entries we are looking for.)

> This, however, doesn't sound so good.  Isn't there some way this can be fixed?

I'm intending to look into it, but I think it's mostly independent of
whether we replace pgindent itself.  The existing code has the same
problem, really.

One brute-force way we could deal with the problem is to have a "manual"
list of names to be treated as typedefs, in addition to whatever the
buildfarm produces.  I see no other way than that to get, for instance,
simplehash.h's SH_TYPE to be formatted as a typedef.  There are also
some typedefs that don't get formatted correctly because they are only
used for wonky options that no existing typedef-reporting buildfarm member
builds.  Manual addition might be the path of least resistance there too.

Now the other side of this coin is that, by definition, such typedefs
are not getting used in a huge number of places.  If we just had to
live with it, it might not be awful.

>> All in all, this looks pretty darn good from here, and I'm thinking
>> we should push forward on it.

> What does that exactly mean concretely?

That means I plan to continue putting effort into it with the goal of
making a switchover sometime pretty darn soon.  We do not have a very
wide window for fooling with pgindent rules, IMO --- once v11 development
starts I think we can't touch it again (until this time next year).

> We've talked about pulling pgindent into our main repo, or posting a
> link to a tarball someplace.  An intermediate plan might be to give it
> its own repo, but on git.postgresql.org, which seems like it might
> give us the best of both worlds.  But I really want something that's
> going to be easy to set up and configure.  It took me years to be
> brave enough to get the current pgindent set up.

Yes, moving the goalposts on ease-of-use is an important consideration
here.  What that says to me is that we ought to pull FreeBSD indent
into our tree, and provide Makefile support that makes it easy for
any developer to build it and put it into their PATH.  (I suppose
that means support in the MSVC scripts too, but somebody else will
have to do that part.)

We should also think hard about getting rid of the entab dependency,
to eliminate the other nonstandard prerequisite program.  We could
either accept that that will result in some tab-vs-space changes in
our code, or try to convert those steps in pgindent into pure Perl,
or try to convince Piotr to add an option to indent that will make
it do tabs the way we want (ie use a space not a tab if the tab
would only move one space anyway).

Lastly (and I've said this before, but you pushed back on it at
the time), if we're doing this then we're going all in.  That
means reformatting the back branches to match too.  That diff
is already big enough to be a disaster for back-patching, and
we haven't even considered whether we want to let pgindent adopt
less-inconsistent rules for comment indentation.  So I think that
as soon as the dust has settled in HEAD, we back-patch the addition
of FreeBSD indent, and the changes in pgindent proper, and then
run pgindent in each supported back branch.

			regards, tom lane

  5. Re: Preliminary results for proposed new pgindent implementation

    Stephen Frost <sfrost@snowman.net> — 2017-05-19T15:05:26Z

    Tom, all,

* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> Robert Haas <robertmhaas@gmail.com> writes:
> > On Thu, May 18, 2017 at 11:00 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> >> The reason PGSSTrackLevel is "unrecognized" is that it's not in
> >> typedefs.list, which is a deficiency in our typedef-collection
> >> technology not in indent.  (I believe the problem is that there
> >> are no variables declared with that typename, causing there to
> >> not be any of the kind of symbol table entries we are looking for.)
> 
> > This, however, doesn't sound so good.  Isn't there some way this can be fixed?
> 
> I'm intending to look into it, but I think it's mostly independent of
> whether we replace pgindent itself.  The existing code has the same
> problem, really.
> 
> One brute-force way we could deal with the problem is to have a "manual"
> list of names to be treated as typedefs, in addition to whatever the
> buildfarm produces.  I see no other way than that to get, for instance,
> simplehash.h's SH_TYPE to be formatted as a typedef.  There are also
> some typedefs that don't get formatted correctly because they are only
> used for wonky options that no existing typedef-reporting buildfarm member
> builds.  Manual addition might be the path of least resistance there too.
> 
> Now the other side of this coin is that, by definition, such typedefs
> are not getting used in a huge number of places.  If we just had to
> live with it, it might not be awful.

Dealing with the typedef lists in general is a bit of a pain to get
right, to make sure that new just-written code gets correctly indented.
Perhaps we could find a way to incorporate the buildfarm typedef lists
and a manual list and a locally generated/provided set in a simpler
fashion in general.

> >> All in all, this looks pretty darn good from here, and I'm thinking
> >> we should push forward on it.
> 
> > What does that exactly mean concretely?
> 
> That means I plan to continue putting effort into it with the goal of
> making a switchover sometime pretty darn soon.  We do not have a very
> wide window for fooling with pgindent rules, IMO --- once v11 development
> starts I think we can't touch it again (until this time next year).

Agreed.

> > We've talked about pulling pgindent into our main repo, or posting a
> > link to a tarball someplace.  An intermediate plan might be to give it
> > its own repo, but on git.postgresql.org, which seems like it might
> > give us the best of both worlds.  But I really want something that's
> > going to be easy to set up and configure.  It took me years to be
> > brave enough to get the current pgindent set up.
> 
> Yes, moving the goalposts on ease-of-use is an important consideration
> here.  What that says to me is that we ought to pull FreeBSD indent
> into our tree, and provide Makefile support that makes it easy for
> any developer to build it and put it into their PATH.  (I suppose
> that means support in the MSVC scripts too, but somebody else will
> have to do that part.)

I'm not a huge fan of this, however.  Do we really need to carry around
the FreeBSD indent in our tree?  I had been expecting that these changes
would eventually result in a package that's available in the common
distributions (possibly from apt/yum.postgresql.org, at least until it's
in the main Debian-based and RHEL-based package systems).  Are you
thinking that we'll always have to have our own modified version?

> We should also think hard about getting rid of the entab dependency,
> to eliminate the other nonstandard prerequisite program.  We could
> either accept that that will result in some tab-vs-space changes in
> our code, or try to convert those steps in pgindent into pure Perl,
> or try to convince Piotr to add an option to indent that will make
> it do tabs the way we want (ie use a space not a tab if the tab
> would only move one space anyway).

What about perltidy itself..?  We don't include that in our tree either.

I do think it'd be good to if Piotr would add such an option, hopefully
that's agreeable.

> Lastly (and I've said this before, but you pushed back on it at
> the time), if we're doing this then we're going all in.  That
> means reformatting the back branches to match too.  That diff
> is already big enough to be a disaster for back-patching, and
> we haven't even considered whether we want to let pgindent adopt
> less-inconsistent rules for comment indentation.  So I think that
> as soon as the dust has settled in HEAD, we back-patch the addition
> of FreeBSD indent, and the changes in pgindent proper, and then
> run pgindent in each supported back branch.

Ugh.  This would be pretty painful, but I agree that back-patching
without doing re-indenting the back-branches would also suck, so I'm on
the fence about this.

Thanks!

Stephen

  6. Re: Preliminary results for proposed new pgindent implementation

    Tom Lane <tgl@sss.pgh.pa.us> — 2017-05-19T15:22:54Z

    Stephen Frost <sfrost@snowman.net> writes:
> * Tom Lane (tgl@sss.pgh.pa.us) wrote:
>> Yes, moving the goalposts on ease-of-use is an important consideration
>> here.  What that says to me is that we ought to pull FreeBSD indent
>> into our tree, and provide Makefile support that makes it easy for
>> any developer to build it and put it into their PATH.  (I suppose
>> that means support in the MSVC scripts too, but somebody else will
>> have to do that part.)

> I'm not a huge fan of this, however.  Do we really need to carry around
> the FreeBSD indent in our tree?  I had been expecting that these changes
> would eventually result in a package that's available in the common
> distributions (possibly from apt/yum.postgresql.org, at least until it's
> in the main Debian-based and RHEL-based package systems).  Are you
> thinking that we'll always have to have our own modified version?

I certainly would rather that our version matched something that's under
active maintenance someplace.  But it seems like there are two good
arguments for having a copy in our tree:

* easy accessibility for PG developers

* at any given time we need to be using a specific "blessed" version,
so that all developers can get equivalent results.  There's pretty much
no chance of that happening if we depend on distro-provided packages,
even if those share a common upstream.

We've had reasonably decent luck with tracking the tzcode/tzdata packages
as local copies, so I feel like we're not taking on anything unreasonable
if our model is that we'll occasionally (not oftener than once per year)
update our copy to recent upstream and then re-indent using that.

> What about perltidy itself..?  We don't include that in our tree either.

Not being much of a Perl guy, I don't care one way or the other about
perltidy.  Somebody else can work on that if it needs work.

			regards, tom lane

  7. Re: Preliminary results for proposed new pgindent implementation

    Heikki Linnakangas <hlinnaka@iki.fi> — 2017-05-19T15:42:10Z

    On 05/19/2017 06:05 PM, Stephen Frost wrote:
> * Tom Lane (tgl@sss.pgh.pa.us) wrote:
>> Robert Haas <robertmhaas@gmail.com> writes:
>>> On Thu, May 18, 2017 at 11:00 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>>> The reason PGSSTrackLevel is "unrecognized" is that it's not in
>>>> typedefs.list, which is a deficiency in our typedef-collection
>>>> technology not in indent.  (I believe the problem is that there
>>>> are no variables declared with that typename, causing there to
>>>> not be any of the kind of symbol table entries we are looking for.)
>>
>>> This, however, doesn't sound so good.  Isn't there some way this can be fixed?
>>
>> I'm intending to look into it, but I think it's mostly independent of
>> whether we replace pgindent itself.  The existing code has the same
>> problem, really.
>>
>> One brute-force way we could deal with the problem is to have a "manual"
>> list of names to be treated as typedefs, in addition to whatever the
>> buildfarm produces.  I see no other way than that to get, for instance,
>> simplehash.h's SH_TYPE to be formatted as a typedef.  There are also
>> some typedefs that don't get formatted correctly because they are only
>> used for wonky options that no existing typedef-reporting buildfarm member
>> builds.  Manual addition might be the path of least resistance there too.
>>
>> Now the other side of this coin is that, by definition, such typedefs
>> are not getting used in a huge number of places.  If we just had to
>> live with it, it might not be awful.
>
> Dealing with the typedef lists in general is a bit of a pain to get
> right, to make sure that new just-written code gets correctly indented.
> Perhaps we could find a way to incorporate the buildfarm typedef lists
> and a manual list and a locally generated/provided set in a simpler
> fashion in general.

You can get a pretty good typedefs list just by looking for the pattern 
"} <type name>;". Something like this:

grep -o -h -I --perl-regexp -r "}\W(\w+);" src/ contrib/ | perl -pe 
's/}\W(.*);/\1/' | sort | uniq > typedefs.list

It won't cover system headers and non-struct typedefs, but it catches 
those simplehash typedefs and PGSSTrackLevel. Maybe we should run that 
and merge the result with the typedef lists we collect in the buildfarm.

- Heikki

  8. Re: Preliminary results for proposed new pgindent implementation

    Tom Lane <tgl@sss.pgh.pa.us> — 2017-05-19T15:48:54Z

    Heikki Linnakangas <hlinnaka@iki.fi> writes:
> You can get a pretty good typedefs list just by looking for the pattern 
> "} <type name>;".

That's going to catch a lot of things that are just variables, though.
It might be all right as long as there was manual filtering after it.

			regards, tom lane

  9. Re: Preliminary results for proposed new pgindent implementation

    Bruce Momjian <bruce@momjian.us> — 2017-05-19T16:01:33Z

    On Fri, May 19, 2017 at 11:22:54AM -0400, Tom Lane wrote:
> We've had reasonably decent luck with tracking the tzcode/tzdata packages
> as local copies, so I feel like we're not taking on anything unreasonable
> if our model is that we'll occasionally (not oftener than once per year)
> update our copy to recent upstream and then re-indent using that.

I guess by having a copy in our tree we would overtly update our
version, rather than downloading whatever happens to be the most recent
version and finding changes by accident.

If we could download a specific version that had everything we need,
that would work too.

> > What about perltidy itself..?  We don't include that in our tree either.
> 
> Not being much of a Perl guy, I don't care one way or the other about
> perltidy.  Somebody else can work on that if it needs work.

We have agreed on a fixed version of perltidy and I added a download
link to pgindent/README.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

  10. Re: Preliminary results for proposed new pgindent implementation

    Heikki Linnakangas <hlinnaka@iki.fi> — 2017-05-19T16:10:17Z

    On 05/19/2017 06:48 PM, Tom Lane wrote:
> Heikki Linnakangas <hlinnaka@iki.fi> writes:
>> You can get a pretty good typedefs list just by looking for the pattern
>> "} <type name>;".
>
> That's going to catch a lot of things that are just variables, though.
> It might be all right as long as there was manual filtering after it.

At a quick glance, there are only a couple of them. This two cases 
caught my eye. In twophase.c:

static struct xllist
{
         StateFileChunk *head;           /* first data block in the chain */
         StateFileChunk *tail;           /* last block in chain */
         uint32          num_chunks;
         uint32          bytes_free;             /* free bytes left in 
tail block */
         uint32          total_len;              /* total data bytes in 
chain */
}       records;

And this in informix.c:

static struct
{
         long            val;
         int                     maxdigits;
         int                     digits;
         int                     remaining;
         char            sign;
         char       *val_string;
}       value;

IMHO it would actually be an improvement if there was a space rather 
than a tab there. But I'm not sure what else it would mess up to 
consider those typedef names. And those are awfully generic names; 
wouldn't hurt to rename them, anyway.

- Heikki

  11. Re: Preliminary results for proposed new pgindent implementation

    Robert Haas <robertmhaas@gmail.com> — 2017-05-19T16:21:52Z

    On Fri, May 19, 2017 at 11:22 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> I certainly would rather that our version matched something that's under
> active maintenance someplace.  But it seems like there are two good
> arguments for having a copy in our tree:
>
> * easy accessibility for PG developers
>
> * at any given time we need to be using a specific "blessed" version,
> so that all developers can get equivalent results.  There's pretty much
> no chance of that happening if we depend on distro-provided packages,
> even if those share a common upstream.

Yeah, but those advantages could also be gained by putting the
pgindent tree on git.postgresql.org in a separate repository.  Having
it in the same repository as the actual PostgreSQL code is not
required nor, in my opinion, particularly desirable.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  12. Re: Preliminary results for proposed new pgindent implementation

    Andres Freund <andres@anarazel.de> — 2017-05-19T16:30:08Z

    On 2017-05-19 12:21:52 -0400, Robert Haas wrote:
> On Fri, May 19, 2017 at 11:22 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> > I certainly would rather that our version matched something that's under
> > active maintenance someplace.  But it seems like there are two good
> > arguments for having a copy in our tree:
> >
> > * easy accessibility for PG developers
> >
> > * at any given time we need to be using a specific "blessed" version,
> > so that all developers can get equivalent results.  There's pretty much
> > no chance of that happening if we depend on distro-provided packages,
> > even if those share a common upstream.
> 
> Yeah, but those advantages could also be gained by putting the
> pgindent tree on git.postgresql.org in a separate repository.  Having
> it in the same repository as the actual PostgreSQL code is not
> required nor, in my opinion, particularly desirable.

I'm of the contrary opinion.  A lot of the regular churn due to pgindent
right now is because it's inconvenient to run.  Having to clone a
separate repository, compile that project, put it into PATH (fun if
there's multiple versions), run pgindent, discover typedefs.list is out
of date, update, run, ...  is pretty much a guarantee that'll continue.
If we had a make indent that computed local typedefs list, *added* new
but not removed old ones, we could get much closer to just always being
properly indented.

The cost of putting it somewhere blow src/tools/pgindent seems fairly
minor.

- Andres

  13. Re: Preliminary results for proposed new pgindent implementation

    Tom Lane <tgl@sss.pgh.pa.us> — 2017-05-19T16:31:14Z

    Robert Haas <robertmhaas@gmail.com> writes:
> On Fri, May 19, 2017 at 11:22 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> I certainly would rather that our version matched something that's under
>> active maintenance someplace.  But it seems like there are two good
>> arguments for having a copy in our tree:
>> 
>> * easy accessibility for PG developers
>> 
>> * at any given time we need to be using a specific "blessed" version,
>> so that all developers can get equivalent results.  There's pretty much
>> no chance of that happening if we depend on distro-provided packages,
>> even if those share a common upstream.

> Yeah, but those advantages could also be gained by putting the
> pgindent tree on git.postgresql.org in a separate repository.  Having
> it in the same repository as the actual PostgreSQL code is not
> required nor, in my opinion, particularly desirable.

It adds an extra step to what a developer has to do to get pgindent
up and running, so it doesn't seem to me like it's helping the goal
of reducing the setup overhead.

			regards, tom lane

  14. Re: Preliminary results for proposed new pgindent implementation

    Tom Lane <tgl@sss.pgh.pa.us> — 2017-05-19T16:38:13Z

    Heikki Linnakangas <hlinnaka@iki.fi> writes:
> On 05/19/2017 06:48 PM, Tom Lane wrote:
>> That's going to catch a lot of things that are just variables, though.
>> It might be all right as long as there was manual filtering after it.

> At a quick glance, there are only a couple of them. This two cases 
> caught my eye. In twophase.c:

> static struct xllist
> {
> ...
> }       records;

> IMHO it would actually be an improvement if there was a space rather 
> than a tab there.

Agreed, but if "records" were considered a typedef name, that would
likely screw up the formatting of code referencing it.  Maybe less
badly with this version of indent than our old one, not sure.

What I was just looking at is the possibility of absorbing struct
tags ("xllist" in the above) as if they were typedef names.  In
at least 95% of our usages, if a struct has a tag then the tag is
also the struct's typedef name.  The reason this is interesting
is that it looks like (on at least Linux and macOS) the debug info
captures struct tags even when it misses the corresponding typedef.
We could certainly create a coding rule that struct tags *must*
match struct typedef names for our own code, but I'm not sure what
violations of that convention might appear in system headers.

			regards, tom lane

  15. Re: Preliminary results for proposed new pgindent implementation

    Tom Lane <tgl@sss.pgh.pa.us> — 2017-05-19T16:49:10Z

    Andres Freund <andres@anarazel.de> writes:
> On 2017-05-19 12:21:52 -0400, Robert Haas wrote:
>> Yeah, but those advantages could also be gained by putting the
>> pgindent tree on git.postgresql.org in a separate repository.  Having
>> it in the same repository as the actual PostgreSQL code is not
>> required nor, in my opinion, particularly desirable.

> I'm of the contrary opinion.  A lot of the regular churn due to pgindent
> right now is because it's inconvenient to run.  Having to clone a
> separate repository, compile that project, put it into PATH (fun if
> there's multiple versions), run pgindent, discover typedefs.list is out
> of date, update, run, ...  is pretty much a guarantee that'll continue.
> If we had a make indent that computed local typedefs list, *added* new
> but not removed old ones, we could get much closer to just always being
> properly indented.

I hadn't really thought of automating it to that extent, but yeah,
that seems like an interesting prospect.

> The cost of putting it somewhere blow src/tools/pgindent seems fairly
> minor.

I think the main cost would be bloating distribution tarballs.  Although
we're talking about adding ~50K to tarballs that are already pushing 20MB,
so realistically who's going to notice?  If you want to cut the tarball
size, let's reopen the discussion about keeping release notes since the
dawn of time.

Also, having the support in distributed tarballs is not all bad, because
it would allow someone working from a tarball rather than a git pull
to have pgindent support.  Dunno if there are any such someones anymore,
but maybe they're out there.

			regards, tom lane

  16. Re: Preliminary results for proposed new pgindent implementation

    Alvaro Herrera <alvherre@2ndquadrant.com> — 2017-05-19T17:31:11Z

    Tom Lane wrote:
> Robert Haas <robertmhaas@gmail.com> writes:

> > Yeah, but those advantages could also be gained by putting the
> > pgindent tree on git.postgresql.org in a separate repository.  Having
> > it in the same repository as the actual PostgreSQL code is not
> > required nor, in my opinion, particularly desirable.
> 
> It adds an extra step to what a developer has to do to get pgindent
> up and running, so it doesn't seem to me like it's helping the goal
> of reducing the setup overhead.

I favor having indent in a separate repository in our Git server, for
these reasons

0. it's under our control (so we can change rules as we see fit)
1. we can have Piotr as a committer there
2. we can use the same pgindent version for all Pg branches

I'm thinking that whenever we change the indent rules, we would
re-indent supported back-branches, just as Tom's proposing we'd do now
(which I endorse).

We wouldn't change the rules often, but say if we leave some typedef
wonky behavior alone for now, and somebody happens to fix it in the
future, then the fix would apply not only to the current tree at the
time but also to all older trees, which makes sense.


Now, there is a process that can be followed to update a *patch* from an
"pre-indent upstream PG" to a "post-indent upstream PG", to avoid manual
work in rebasing the patch past pgindent.  This can easily be used on
branches that can be rebased, also (since they are essentially just a
collection of patches).  One problem that remains is that this doesn't
apply easily to branches that get merged without rebase.  I *think* it
should be possible to come up with a process that creates a merge commit
using pgindent, but I haven't tried.

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  17. Re: Preliminary results for proposed new pgindent implementation

    Tom Lane <tgl@sss.pgh.pa.us> — 2017-05-19T18:54:54Z

    I wrote:
> What I was just looking at is the possibility of absorbing struct
> tags ("xllist" in the above) as if they were typedef names.  In
> at least 95% of our usages, if a struct has a tag then the tag is
> also the struct's typedef name.  The reason this is interesting
> is that it looks like (on at least Linux and macOS) the debug info
> captures struct tags even when it misses the corresponding typedef.
> We could certainly create a coding rule that struct tags *must*
> match struct typedef names for our own code, but I'm not sure what
> violations of that convention might appear in system headers.

I did an experiment with seeing what would happen to the typedef list
if we included struct tags.  On my Linux box, that adds about 10%
more names (3343 instead of 3028).  A lot of them would be good to
have, but there are a lot of others that maybe not so much.  See
attached diff output.

I hesitate to suggest any rule as grotty as "take struct tags only
if they begin with an upper-case letter", but that would actually
work really well, looks like.

			regards, tom lane

  18. Re: Preliminary results for proposed new pgindent implementation

    Peter Eisentraut <peter.eisentraut@2ndquadrant.com> — 2017-05-19T19:12:32Z

    On 5/19/17 13:31, Alvaro Herrera wrote:
> I favor having indent in a separate repository in our Git server, for
> these reasons

I am also in favor of that.

> 0. it's under our control (so we can change rules as we see fit)
> 1. we can have Piotr as a committer there
> 2. we can use the same pgindent version for all Pg branches

3. We can use pgindent for external code.

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  19. Re: Preliminary results for proposed new pgindent implementation

    Peter Eisentraut <peter.eisentraut@2ndquadrant.com> — 2017-05-19T19:17:24Z

    On 5/19/17 11:22, Tom Lane wrote:
> I certainly would rather that our version matched something that's under
> active maintenance someplace.  But it seems like there are two good
> arguments for having a copy in our tree:

Is pgindent going to be indented by pgindent?

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  20. Re: Preliminary results for proposed new pgindent implementation

    Tom Lane <tgl@sss.pgh.pa.us> — 2017-05-19T20:01:36Z

    Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:
> On 5/19/17 11:22, Tom Lane wrote:
>> I certainly would rather that our version matched something that's under
>> active maintenance someplace.  But it seems like there are two good
>> arguments for having a copy in our tree:

> Is pgindent going to be indented by pgindent?

If we were going to keep it in our tree, I'd plan to add an exclusion
rule to keep pgindent from touching it, as we already have for assorted
other files that are copied from external projects.  However, it seems
like "keep it in a separate repo" is winning, so it's moot.

			regards, tom lane

  21. Re: Preliminary results for proposed new pgindent implementation

    Tom Lane <tgl@sss.pgh.pa.us> — 2017-06-16T14:51:04Z

    Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:
> On 5/19/17 13:31, Alvaro Herrera wrote:
>> I favor having indent in a separate repository in our Git server, for
>> these reasons

> I am also in favor of that.

>> 0. it's under our control (so we can change rules as we see fit)
>> 1. we can have Piotr as a committer there
>> 2. we can use the same pgindent version for all Pg branches

> 3. We can use pgindent for external code.

Now that we've about reached the point of actually making the change,
we need to come to a resolution on where we're keeping the new indent
code.  I thought that Alvaro's point 1 above (we can give Piotr a
commit bit) was the only really compelling argument for putting it
into a separate repo rather than into our main tree.  In other aspects
that's a loser --- in particular, it would be hard to have different
indent versions for different PG branches, if we chose to run things
that way.  However, I gather from Piotr's recent remarks[1] that
he's not actually excited about doing continuing maintenance on
indent, so that advantage now seems illusory.  In any case we'd
need to keep such a repo pretty well locked down: if it's changing,
and different developers pull from it at different times, then
we're going to have people working with different indent behaviors,
which will make nobody happy.

So I'm back to the position that we ought to stick the indent
code under src/tools/ in our main repo.  Is anyone really
seriously against that?

			regards, tom lane

[1] https://www.postgresql.org/message-id/VI1PR03MB119959F4B65F000CA7CD9F6BF2CC0%40VI1PR03MB1199.eurprd03.prod.outlook.com

  22. Re: Preliminary results for proposed new pgindent implementation

    Tom Lane <tgl@sss.pgh.pa.us> — 2017-06-16T17:10:50Z

    There was some discussion upthread about how we'd like pgindent not to
do weird things with string literals that wrap around the end of the
line a little bit.  I looked into that and found that it's actually
a generic behavior for any line that's within parentheses: normally,
such a line will get lined up with the parens, like this:

        foobar(baz,
               baz2,
               baz3,
               ...

but if the line would wrap when indented that much, and backing off
lets it not wrap, then it backs off.

I experimented with disabling that logic and just always aligning
to the paren indentation.  That fixes the weird cases with continued
string literals, but it also makes for a heck of a lot of other changes.
The full diff is too big to post here, but I've attached a selection
of diff hunks to give you an idea.  I'm not really sure if I like this
better than pgindent's traditional behavior --- but it's arguably less
confusing.

An intermediate position that we could consider is to disable the back-off
logic only when the line starts with a string literal.  I haven't actually
coded this but it looks like it would be easy, if grotty.

Or we could leave it alone.

Thoughts?

			regards, tom lane

  23. Re: Preliminary results for proposed new pgindent implementation

    Andres Freund <andres@anarazel.de> — 2017-06-16T17:19:46Z

    Hi,

On 2017-06-16 13:10:50 -0400, Tom Lane wrote:
> I experimented with disabling that logic and just always aligning
> to the paren indentation.  That fixes the weird cases with continued
> string literals, but it also makes for a heck of a lot of other changes.
> The full diff is too big to post here, but I've attached a selection
> of diff hunks to give you an idea.  I'm not really sure if I like this
> better than pgindent's traditional behavior --- but it's arguably less
> confusing.
> 
> An intermediate position that we could consider is to disable the back-off
> logic only when the line starts with a string literal.  I haven't actually
> coded this but it looks like it would be easy, if grotty.

I think the current logic is pretty horrible, primarily because it's so
hard to get to manually.  I could live with both of these proposed
changes, the selection of the changes you posted looks like it could be
improved by code changes, but that's obviously a large amount of work.
The heuristic also seems to make sense.

At this point however I wonder whether just moving to the new tool on
its own wouldn't be a big enough change - we could just delay that
decision until we've got the rest done at least.

- Andres

  24. Re: Preliminary results for proposed new pgindent implementation

    Tom Lane <tgl@sss.pgh.pa.us> — 2017-06-16T17:34:01Z

    Andres Freund <andres@anarazel.de> writes:
> I think the current logic is pretty horrible, primarily because it's so
> hard to get to manually.

Yes, I think that's really the big argument against it: no editor on
the face of the planet will indent code that way to start with.

> I could live with both of these proposed
> changes, the selection of the changes you posted looks like it could be
> improved by code changes, but that's obviously a large amount of work.

In the end, the only thing that fixes this sort of stuff is to be more
rigid about making the code fit into 80 columns to begin with.  I get
the impression though that a lot of people work in editor windows that
are wider than that, so the code looks fine to them when it slops over
a bit.

> At this point however I wonder whether just moving to the new tool on
> its own wouldn't be a big enough change - we could just delay that
> decision until we've got the rest done at least.

I'm torn between that approach and "let's just have one big flag day
and get it over with".  I think having the rules incrementally changing
from one release to the next will be a huge headache.

I do intend to apply the diffs to HEAD in multiple steps, just to
make them more reviewable.  But I think we should probably absorb
all the changes we want into v10, not leave some for later cycles.

			regards, tom lane

  25. Re: Preliminary results for proposed new pgindent implementation

    Bruce Momjian <bruce@momjian.us> — 2017-06-16T17:44:30Z

    On Fri, Jun 16, 2017 at 01:34:01PM -0400, Tom Lane wrote:
> > I could live with both of these proposed
> > changes, the selection of the changes you posted looks like it could be
> > improved by code changes, but that's obviously a large amount of work.
> 
> In the end, the only thing that fixes this sort of stuff is to be more
> rigid about making the code fit into 80 columns to begin with.  I get
> the impression though that a lot of people work in editor windows that
> are wider than that, so the code looks fine to them when it slops over
> a bit.

Yes, it is all about <80 column output.  The current pgindent does
everything possible to accomplish that --- the question is whether we
want uglier code to do it.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

  26. Re: Preliminary results for proposed new pgindent implementation

    Andres Freund <andres@anarazel.de> — 2017-06-16T18:02:12Z

    On 2017-06-16 13:44:30 -0400, Bruce Momjian wrote:
> On Fri, Jun 16, 2017 at 01:34:01PM -0400, Tom Lane wrote:
> > > I could live with both of these proposed
> > > changes, the selection of the changes you posted looks like it could be
> > > improved by code changes, but that's obviously a large amount of work.
> > 
> > In the end, the only thing that fixes this sort of stuff is to be more
> > rigid about making the code fit into 80 columns to begin with.  I get
> > the impression though that a lot of people work in editor windows that
> > are wider than that, so the code looks fine to them when it slops over
> > a bit.
> 
> Yes, it is all about <80 column output.  The current pgindent does
> everything possible to accomplish that --- the question is whether we
> want uglier code to do it.

For me personally the misindentation is way uglier than a too long line.
I think a number of those long-lines are there because pgindent
sometimes re-indents lines that are continuations of previous ones
pretty far, making it hard to reduce indentation.

- Andres

  27. Re: Preliminary results for proposed new pgindent implementation

    Andres Freund <andres@anarazel.de> — 2017-06-16T18:05:34Z

    On 2017-06-16 13:34:01 -0400, Tom Lane wrote:
> > I could live with both of these proposed
> > changes, the selection of the changes you posted looks like it could be
> > improved by code changes, but that's obviously a large amount of work.
> 
> In the end, the only thing that fixes this sort of stuff is to be more
> rigid about making the code fit into 80 columns to begin with.  I get
> the impression though that a lot of people work in editor windows that
> are wider than that, so the code looks fine to them when it slops over
> a bit.

That, but maybe also that it's often slightly too long line vs. weird
multiline mess.  A good number of things pgindent indents weirdly can be
prevented by just not adding a linebreak, which isn't a great fix...


> > At this point however I wonder whether just moving to the new tool on
> > its own wouldn't be a big enough change - we could just delay that
> > decision until we've got the rest done at least.
> 
> I'm torn between that approach and "let's just have one big flag day
> and get it over with".

I don't have a strong opinion on this.


> I think having the rules incrementally changing from one release to
> the next will be a huge headache.

Yea, I was more thinking of getting the new indent in, and then making
the followup decisions a few days after.


> I do intend to apply the diffs to HEAD in multiple steps, just to
> make them more reviewable.  But I think we should probably absorb
> all the changes we want into v10, not leave some for later cycles.

Btw, how much are you planning to backpatch these?

Greetings,

Andres Freund

  28. Re: Preliminary results for proposed new pgindent implementation

    Tom Lane <tgl@sss.pgh.pa.us> — 2017-06-16T18:11:58Z

    Andres Freund <andres@anarazel.de> writes:
> On 2017-06-16 13:44:30 -0400, Bruce Momjian wrote:
>> Yes, it is all about <80 column output.  The current pgindent does
>> everything possible to accomplish that --- the question is whether we
>> want uglier code to do it.

> For me personally the misindentation is way uglier than a too long line.

I'm coming around to that opinion too.  We have many source lines that
are a bit too long, or a lot too long if someone decided they didn't
want to split an error message across lines.  pgindent "fixes" that
in some places but not others (if it would have to go left of the
prevailing statement indent, it gives up and indents to the paren level
anyway).  On balance it's just weird.  Better to indent normally and
let the programmer decide if she wants to break the lines differently
to keep them from wrapping.

I assume though that Piotr wants an option to preserve that behavior.
I'm happy to write up a patch for bsdindent that adds a switch
controlling this, but is there any rhyme or reason to the way its
switches are named?

			regards, tom lane

  29. Re: Preliminary results for proposed new pgindent implementation

    Tom Lane <tgl@sss.pgh.pa.us> — 2017-06-16T18:23:00Z

    Andres Freund <andres@anarazel.de> writes:
> On 2017-06-16 13:34:01 -0400, Tom Lane wrote:
>> I do intend to apply the diffs to HEAD in multiple steps, just to
>> make them more reviewable.  But I think we should probably absorb
>> all the changes we want into v10, not leave some for later cycles.

> Btw, how much are you planning to backpatch these?

Well, that's something we need to discuss.  I originally argued for
back-patching the new rules, whatever they are (ie, run the new
pgindent on the back branches whenever we've agreed that the dust
has settled).  But I'm starting to realize that that's likely to
be horrid for anyone who's carrying out-of-tree patches, as I know
a lot of packagers do for instance.  We have to trade off our own
inconvenience in making back-patches against inconvenience to
people who are maintaining private patchsets.

One idea that occurs to me after a few minutes' thought is to
announce that we will reindent the back branches, but not till
around the time of v10 final release.  Once v10 is out, anybody
who's carrying a private patchset will be needing to think about
rebasing it on top of reindented code anyway, so dealing with that
in the back branches at the same time might be a bit less work.

Or we could leave the back branches alone and anticipate five
years worth of pain in back-patching.  I don't find that very
appetizing personally, but it might be the easiest sell to the
majority of the community, since very few of us do back-patching
work on a regular basis.

			regards, tom lane

  30. Re: Preliminary results for proposed new pgindent implementation

    Bruce Momjian <bruce@momjian.us> — 2017-06-16T18:42:38Z

    On Fri, Jun 16, 2017 at 02:23:00PM -0400, Tom Lane wrote:
> Well, that's something we need to discuss.  I originally argued for
> back-patching the new rules, whatever they are (ie, run the new
> pgindent on the back branches whenever we've agreed that the dust
> has settled).  But I'm starting to realize that that's likely to
> be horrid for anyone who's carrying out-of-tree patches, as I know
> a lot of packagers do for instance.  We have to trade off our own
> inconvenience in making back-patches against inconvenience to
> people who are maintaining private patchsets.

Can't they sync up to just before our pgindent commit and run pgindent
on their own code base?

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

  31. Re: Preliminary results for proposed new pgindent implementation

    Andres Freund <andres@anarazel.de> — 2017-06-16T18:54:06Z

    On 2017-06-16 14:42:38 -0400, Bruce Momjian wrote:
> On Fri, Jun 16, 2017 at 02:23:00PM -0400, Tom Lane wrote:
> > Well, that's something we need to discuss.  I originally argued for
> > back-patching the new rules, whatever they are (ie, run the new
> > pgindent on the back branches whenever we've agreed that the dust
> > has settled).  But I'm starting to realize that that's likely to
> > be horrid for anyone who's carrying out-of-tree patches, as I know
> > a lot of packagers do for instance.  We have to trade off our own
> > inconvenience in making back-patches against inconvenience to
> > people who are maintaining private patchsets.
> 
> Can't they sync up to just before our pgindent commit and run pgindent
> on their own code base?

That doesn't really help that much if you have a series of patches that
you want to keep independent, e.g. because you might want to submit to
postgres.  And you'll also get a bunch of annoying to resolve merge
conflicts, even if they're easier to resolve with that methodology.

- Andres

  32. Re: Preliminary results for proposed new pgindent implementation

    Bruce Momjian <bruce@momjian.us> — 2017-06-16T19:04:41Z

    On Fri, Jun 16, 2017 at 11:54:06AM -0700, Andres Freund wrote:
> On 2017-06-16 14:42:38 -0400, Bruce Momjian wrote:
> > On Fri, Jun 16, 2017 at 02:23:00PM -0400, Tom Lane wrote:
> > > Well, that's something we need to discuss.  I originally argued for
> > > back-patching the new rules, whatever they are (ie, run the new
> > > pgindent on the back branches whenever we've agreed that the dust
> > > has settled).  But I'm starting to realize that that's likely to
> > > be horrid for anyone who's carrying out-of-tree patches, as I know
> > > a lot of packagers do for instance.  We have to trade off our own
> > > inconvenience in making back-patches against inconvenience to
> > > people who are maintaining private patchsets.
> > 
> > Can't they sync up to just before our pgindent commit and run pgindent
> > on their own code base?
> 
> That doesn't really help that much if you have a series of patches that
> you want to keep independent, e.g. because you might want to submit to
> postgres.  And you'll also get a bunch of annoying to resolve merge
> conflicts, even if they're easier to resolve with that methodology.

I think we have to ask how much we want to make things easier for people
with modified but continually-updated Postgres trees vs. our
community-tree developers.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

  33. Re: Preliminary results for proposed new pgindent implementation

    Tom Lane <tgl@sss.pgh.pa.us> — 2017-06-16T19:56:47Z

    One other thing I'd like to do while we're changing this stuff is
to get rid of the need for entab/detab.  Right now, after doing
all the other work, my copy of pgindent is running the code through
detab and then entab so as to match the old decisions about how to
represent whitespace (ie, as spaces or tabs).  This is grotty as
can be.  I managed to tweak bsdindent so that its output matches
what entab would do, by dint of the attached patch, which implements
the rule "use a space instead of a tab if the tab would only move
one column and we don't need another tab after it".  (I think entab
is being weird with the second half of that rule, but if I remove it,
I get circa a thousand lines of invisible whitespace changes; probably
better not to deal with those.  With no patch at all, just letting
bsdindent do what it does now, there's circa ten thousand changed lines.)

Unless Piotr objects, I propose to add another switch to bsdindent
that selects this behavior, and then we can drop entab, removing
another impediment to getting pgindent working.

			regards, tom lane

  34. Re: Preliminary results for proposed new pgindent implementation

    Bruce Momjian <bruce@momjian.us> — 2017-06-16T20:10:42Z

    On Fri, Jun 16, 2017 at 03:56:47PM -0400, Tom Lane wrote:
> can be.  I managed to tweak bsdindent so that its output matches
> what entab would do, by dint of the attached patch, which implements
> the rule "use a space instead of a tab if the tab would only move
> one column and we don't need another tab after it".  (I think entab
> is being weird with the second half of that rule, but if I remove it,
> I get circa a thousand lines of invisible whitespace changes; probably
> better not to deal with those.  With no patch at all, just letting
> bsdindent do what it does now, there's circa ten thousand changed lines.)

Yeah, entab was designed to do that, via this C comment:

	/*
	 * Is the next character going to be a tab?  We do tab
	 * replacement in the current spot if the next char is
	 * going to be a tab and ignore min_spaces.
	 */


-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

  35. Re: Preliminary results for proposed new pgindent implementation

    Piotr Stefaniak <postgres@piotr-stefaniak.me> — 2017-06-16T20:30:21Z

    On 2017-06-16 21:56, Tom Lane wrote:
> One other thing I'd like to do while we're changing this stuff is
> to get rid of the need for entab/detab.  Right now, after doing
> all the other work, my copy of pgindent is running the code through
> detab and then entab so as to match the old decisions about how to
> represent whitespace (ie, as spaces or tabs).  This is grotty as
> can be.  I managed to tweak bsdindent so that its output matches
> what entab would do, by dint of the attached patch, which implements
> the rule "use a space instead of a tab if the tab would only move
> one column and we don't need another tab after it".  (I think entab
> is being weird with the second half of that rule, but if I remove it,
> I get circa a thousand lines of invisible whitespace changes; probably
> better not to deal with those.  With no patch at all, just letting
> bsdindent do what it does now, there's circa ten thousand changed lines.)
> 
> Unless Piotr objects, I propose to add another switch to bsdindent
> that selects this behavior, and then we can drop entab, removing
> another impediment to getting pgindent working.

I understand the reasoning, but this is a very specific need and I think
not at all universal for anyone else in the future. One of the bugs
listed in indent's manpage is that it "has more switches than ls(1)". So
currently I'm against pushing an option for the above upstream, to the
FreeBSD repository.

Why not add this to the already non-empty list of custom patches?

  36. Re: Preliminary results for proposed new pgindent implementation

    Piotr Stefaniak <postgres@piotr-stefaniak.me> — 2017-06-16T21:05:27Z

    On 2017-06-16 20:11, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
>> On 2017-06-16 13:44:30 -0400, Bruce Momjian wrote:
>>> Yes, it is all about <80 column output.  The current pgindent does
>>> everything possible to accomplish that --- the question is whether we
>>> want uglier code to do it.
> 
>> For me personally the misindentation is way uglier than a too long line.

> 
> I assume though that Piotr wants an option to preserve that behavior.
> I'm happy to write up a patch for bsdindent that adds a switch
> controlling this, but is there any rhyme or reason to the way its
> switches are named?

I don't want to preserve the current behavior at all, but I might need
to add an option for choosing one or the other if users of FreeBSD
indent protest.

I don't have a good name for it. The best I can do is -lpl ("-lp long
lines too").  Can I see the patch?

  37. Re: Preliminary results for proposed new pgindent implementation

    Tom Lane <tgl@sss.pgh.pa.us> — 2017-06-16T21:48:49Z

    Piotr Stefaniak <postgres@piotr-stefaniak.me> writes:
> On 2017-06-16 20:11, Tom Lane wrote:
>> I assume though that Piotr wants an option to preserve that behavior.
>> I'm happy to write up a patch for bsdindent that adds a switch
>> controlling this, but is there any rhyme or reason to the way its
>> switches are named?

> I don't want to preserve the current behavior at all, but I might need
> to add an option for choosing one or the other if users of FreeBSD
> indent protest.

> I don't have a good name for it. The best I can do is -lpl ("-lp long
> lines too").  Can I see the patch?

Here's a patch.  An alternative switch name might be -lpa ("-lp always")
but I'm not set on that.

			regards, tom lane

  38. Re: Preliminary results for proposed new pgindent implementation

    Tom Lane <tgl@sss.pgh.pa.us> — 2017-06-16T22:02:15Z

    Piotr Stefaniak <postgres@piotr-stefaniak.me> writes:
> On 2017-06-16 21:56, Tom Lane wrote:
>> Unless Piotr objects, I propose to add another switch to bsdindent
>> that selects this behavior, and then we can drop entab, removing
>> another impediment to getting pgindent working.

> I understand the reasoning, but this is a very specific need and I think
> not at all universal for anyone else in the future. One of the bugs
> listed in indent's manpage is that it "has more switches than ls(1)". So
> currently I'm against pushing an option for the above upstream, to the
> FreeBSD repository.

> Why not add this to the already non-empty list of custom patches?

Umm ... I thought the idea was to get to the point where the list of
custom patches *is* empty.  Except for carrying our own Makefile of
course.  I'd be sad if we needed a fork just for this.

What I'm testing with right now has just four differences from your repo:

1. This workaround for what I believe you agree is a bug:

-	    ps.in_decl = ps.decl_on_line = ps.last_token != type_def;
+	    ps.in_decl = ps.decl_on_line = true;

2. The long-lines adjustment I just sent you a patch for.

3. The tab-vs-space difference under discussion here.

4. A temporary hack affecting the indentation of comments on the same line
(forcing them to a multiple of 8 spaces even though tabsize is 4).  I have
every intention of dropping that one later; I just don't want to deal with
comment reindentation at the same time as these other things.

			regards, tom lane

  39. Re: Preliminary results for proposed new pgindent implementation

    Piotr Stefaniak <postgres@piotr-stefaniak.me> — 2017-06-16T22:52:53Z

    On 2017-06-17 00:02, Tom Lane wrote:
> Piotr Stefaniak <postgres@piotr-stefaniak.me> writes:
>> On 2017-06-16 21:56, Tom Lane wrote:
>>> Unless Piotr objects, I propose to add another switch to bsdindent
>>> that selects this behavior, and then we can drop entab, removing
>>> another impediment to getting pgindent working.
> 
>> I understand the reasoning, but this is a very specific need and I think
>> not at all universal for anyone else in the future. One of the bugs
>> listed in indent's manpage is that it "has more switches than ls(1)". So
>> currently I'm against pushing an option for the above upstream, to the
>> FreeBSD repository.
> 
>> Why not add this to the already non-empty list of custom patches?
> 
> Umm ... I thought the idea was to get to the point where the list of
> custom patches *is* empty.  Except for carrying our own Makefile of
> course.  I'd be sad if we needed a fork just for this.
> 
> What I'm testing with right now has just four differences from your repo:

There are also the "portability fixes" and they're the main problem.

I've simply removed things like capsicum or __FBSDID() because I thought
it wouldn't be a problem since Postgres will have its own copy of indent
anyway (so that its behavior is not a moving target). I can ifdef-out
them instead of removing entirely, I just didn't think it was important
anymore.

I expect to be in trouble for replacing err() and errx(), though.


> 1. This workaround for what I believe you agree is a bug:
> 
> -	    ps.in_decl = ps.decl_on_line = ps.last_token != type_def;
> +	    ps.in_decl = ps.decl_on_line = true;

That will need a proper fix...

> 2. The long-lines adjustment I just sent you a patch for.

That looks very good.

> 3. The tab-vs-space difference under discussion here.

I can be convinced to make it another option upstream. But I dislike it
nevertheless.

> 4. A temporary hack affecting the indentation of comments on the same line
> (forcing them to a multiple of 8 spaces even though tabsize is 4).  I have
> every intention of dropping that one later; I just don't want to deal with
> comment reindentation at the same time as these other things.

Great!

  40. Re: Preliminary results for proposed new pgindent implementation

    Tom Lane <tgl@sss.pgh.pa.us> — 2017-06-17T02:18:07Z

    Piotr Stefaniak <postgres@piotr-stefaniak.me> writes:
> On 2017-06-17 00:02, Tom Lane wrote:
>> What I'm testing with right now has just four differences from your repo:

> There are also the "portability fixes" and they're the main problem.

Fair enough.

> I've simply removed things like capsicum or __FBSDID() because I thought
> it wouldn't be a problem since Postgres will have its own copy of indent
> anyway (so that its behavior is not a moving target). I can ifdef-out
> them instead of removing entirely, I just didn't think it was important
> anymore.

We should be able to deal with those via some #define hackery, no?

> I expect to be in trouble for replacing err() and errx(), though.

Understood.  I think we could deal with this by providing err() and
errx() in a support file that would be part of our distribution but
not yours.

			regards, tom lane

  41. Re: Preliminary results for proposed new pgindent implementation

    Robert Haas <robertmhaas@gmail.com> — 2017-06-17T17:41:28Z

    On Fri, Jun 16, 2017 at 10:51 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> So I'm back to the position that we ought to stick the indent
> code under src/tools/ in our main repo.  Is anyone really
> seriously against that?

Is it under the same license as everything else?

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  42. Re: Preliminary results for proposed new pgindent implementation

    Tom Lane <tgl@sss.pgh.pa.us> — 2017-06-17T19:13:43Z

    Robert Haas <robertmhaas@gmail.com> writes:
> On Fri, Jun 16, 2017 at 10:51 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> So I'm back to the position that we ought to stick the indent
>> code under src/tools/ in our main repo.  Is anyone really
>> seriously against that?

> Is it under the same license as everything else?

Hm, now that you mention it, interesting point.  I was about to
answer that it's under the standard BSD license, but looking closer,
I see that these files still quote the old 4-clause BSD license text
(with the "advertising" clause).  We need to get them adjusted
to be 3-clause with no advertising.  Looking into a FreeBSD source
tree locally, I see that FreeBSD has removed the advertising clause
from all their files --- sometimes without even remembering to
renumber the old clause 4 to clause 3 ;-).  So Piotr needs to do
likewise to conform with FreeBSD policy.  Once he has, it'll be
identical text to the other BSD-origin files we have in our tree,
eg src/port/getopt.c.

Piotr: if you're unclear on the rationale for this, see the bottom of
https://www.freebsd.org/copyright/license.html

			regards, tom lane

  43. Re: Preliminary results for proposed new pgindent implementation

    Tom Lane <tgl@sss.pgh.pa.us> — 2017-06-17T19:55:11Z

    I wrote:
> Piotr Stefaniak <postgres@piotr-stefaniak.me> writes:
>> There are also the "portability fixes" and they're the main problem.

> Fair enough.

I spent some time looking into this.  I reverted your commits
198457848ae5c86bec3336a9437dd5aa30f480c2 (Replace err.h functions with
standard C equivalents) and fb10acb040b90bdcbad09defd303363db29257d1
(Remove inclusion of sys/cdefs.h) locally and tried to build without
those.  I've successfully worked around the err.h change by adding
cut-down versions of FreeBSD 11's err.h and err.c to the fileset
(see attached).  However, it's proving impossible to work around having
"#include <sys/cdefs.h>" as the first live code in the files.  I thought
maybe we could provide a dummy cdefs.h file, but that breaks things on
platforms where cdefs.h is a real thing and is relied on by other system
headers --- which includes both Linux and BSD.  It seems we would have
to have something like #ifdef HAVE_SYS_CDEFS_H, but that is already a
departure from FreeBSD practice.

So what I'm currently thinking is that we have to diverge from the
FreeBSD sources to the extent of removing #include <sys/cdefs.h>
and the __FBSDID() calls, and instead inserting #include "c.h" to
pick up PG's own portability definitions.  The thing that forced me
into the latter is that there seems no way to avoid compiler warnings
if we don't decorate the declarations of err() and errx() with noreturn
and printf-format attributes --- and we need c.h to provide portable
ways of writing those.  But there are probably other portability things
that we'll need c.h for, anyway, especially if we want to make it work
on Windows.  So I'm thinking this is a small and easily maintainable
difference from the upstream FreeBSD files.

When I inserted #include "c.h", I got duplicate-macro-definition warnings
about "true" and "false", so I would ask you to add this:

--- freebsd_indent/indent_globs.h	2017-06-16 11:06:53.329712682 -0400
+++ new/indent_globs.h	2017-06-17 14:45:41.388015754 -0400
@@ -43,8 +43,12 @@
 				 * of code */
 
 
+#ifndef false
 #define false 0
+#endif
+#ifndef true
 #define true  1
+#endif
 
 
 FILE       *input;		/* the fid for the input file */

Other than that, I think this is a workable compromise on the
portability questions.

			regards, tom lane

  44. Re: Preliminary results for proposed new pgindent implementation

    Peter Eisentraut <peter.eisentraut@2ndquadrant.com> — 2017-06-17T21:41:40Z

    On 6/16/17 10:51, Tom Lane wrote:
> So I'm back to the position that we ought to stick the indent
> code under src/tools/ in our main repo.  Is anyone really
> seriously against that?

I think it would be better to have it separate.

Other than for reasons of principle and general modularity of the world,
I would like this to be available separately for separate download,
packaging, etc. to it can be applied to extension projects without
having to download and build (a specific version of) PostgreSQL.  The
code formatting in extension projects is, um, questionable.  In fact, if
it's a better indent period, I would like to package it for the general
public.

If the vote is to put it into the tree, I would request not to do it in
PG10.  At this point, we should be winding things down and not open up
new areas of activity.  There is a chance that if this goes in (or
anywhere else), there will be a stream of requests along the lines of:
doesn't build on Windows, doesn't build on AIX, doesn't build on
PowerPC, doesn't build on this other Windows variant, the tests don't
run, the tests don't run on Windows, it doesn't build in vpath, it
doesn't work on the buildfarm, and so on.

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  45. Re: Preliminary results for proposed new pgindent implementation

    Piotr Stefaniak <postgres@piotr-stefaniak.me> — 2017-06-17T23:45:29Z

    On 2017-06-17 21:55, Tom Lane wrote:
> I spent some time looking into this.  I reverted your commits
> 198457848ae5c86bec3336a9437dd5aa30f480c2 (Replace err.h functions with
> standard C equivalents) and fb10acb040b90bdcbad09defd303363db29257d1
> (Remove inclusion of sys/cdefs.h) locally and tried to build without
> those.

I wanted to mirror that move, but forgot to not rebase the repository,
so I removed those two commits instead of committing their negatives.
Sorry about that.

> I've successfully worked around the err.h change by adding
> cut-down versions of FreeBSD 11's err.h and err.c to the fileset
> (see attached).

I thought about something like:
#ifdef __FreeBSD__
#include <err.h>
#define ERR(...) err(__VA_ARGS__)
#define ERRX(...) errx(__VA_ARGS__)
#else
#include "err.h"
#endif

and then call ERR() and ERRX() instead of err() and errx(). But that
requires C99. And I would have a very hard time convincing anyone that
it makes any sense from FreeBSD's perspective, since indent is part of
the base system, where <err.h> is guaranteed to exist.

Perhaps it would be best for everyone if indent was moved out of FreeBSD
base, so that portability arguments would make more sense. But that
would take time and some debate.

> However, it's proving impossible to work around having
> "#include <sys/cdefs.h>" as the first live code in the files.  I thought
> maybe we could provide a dummy cdefs.h file, but that breaks things on
> platforms where cdefs.h is a real thing and is relied on by other system
> headers --- which includes both Linux and BSD.  It seems we would have
> to have something like #ifdef HAVE_SYS_CDEFS_H, but that is already a
> departure from FreeBSD practice.

I was thinking if I could get away with putting those into #ifdef
__FreeBSD__ ... #endif. I think that it might be feasible unlike the
idea above. I could be wrong.

> So what I'm currently thinking is that we have to diverge from the
> FreeBSD sources to the extent of removing #include <sys/cdefs.h>
> and the __FBSDID() calls, and instead inserting #include "c.h" to
> pick up PG's own portability definitions.  The thing that forced me
> into the latter is that there seems no way to avoid compiler warnings
> if we don't decorate the declarations of err() and errx() with noreturn
> and printf-format attributes --- and we need c.h to provide portable
> ways of writing those.  But there are probably other portability things
> that we'll need c.h for, anyway, especially if we want to make it work
> on Windows.  So I'm thinking this is a small and easily maintainable
> difference from the upstream FreeBSD files.

That works for me.

> When I inserted #include "c.h", I got duplicate-macro-definition warnings
> about "true" and "false", so I would ask you to add this:

Done.

  46. Re: Preliminary results for proposed new pgindent implementation

    Tom Lane <tgl@sss.pgh.pa.us> — 2017-06-18T15:42:14Z

    Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:
> On 6/16/17 10:51, Tom Lane wrote:
>> So I'm back to the position that we ought to stick the indent
>> code under src/tools/ in our main repo.  Is anyone really
>> seriously against that?

> I think it would be better to have it separate.

> Other than for reasons of principle and general modularity of the world,
> I would like this to be available separately for separate download,
> packaging, etc. to it can be applied to extension projects without
> having to download and build (a specific version of) PostgreSQL.  The
> code formatting in extension projects is, um, questionable.  In fact, if
> it's a better indent period, I would like to package it for the general
> public.

Well, the direction I'm headed in for addressing the portability issues
is to make it depend on the usual PG build environment, notably c.h and
libpgport.  If we don't want it in-tree, it can be built using PGXS,
but it'll still require a PG installation somewhere in order to get built.
Making it independent of both FreeBSD and PG is a significantly larger
project, and one I don't personally intend to tackle.  (And, if someone
does tackle that, I don't exactly see why having our own copy in-tree
would stop them.)

However ... off-list discussion with Piotr indicates that he's unwilling
to touch the license text without permission from FreeBSD core and/or
legal teams.  While the 4-clause license is certainly no impediment to
using indent, we don't want any such text in our tree, so that seems
like a showstopper, at least until the license question is resolved.

Accordingly, I'll proceed with setting up a repo for it on
git.postgresql.org.

> If the vote is to put it into the tree, I would request not to do it in
> PG10.  At this point, we should be winding things down and not open up
> new areas of activity.

I'm confused by this.  Are you objecting to switching to the new indent
version for v10?

			regards, tom lane

  47. Re: Preliminary results for proposed new pgindent implementation

    Robert Haas <robertmhaas@gmail.com> — 2017-06-19T15:49:22Z

    On Sat, Jun 17, 2017 at 5:41 PM, Peter Eisentraut
<peter.eisentraut@2ndquadrant.com> wrote:
> On 6/16/17 10:51, Tom Lane wrote:
>> So I'm back to the position that we ought to stick the indent
>> code under src/tools/ in our main repo.  Is anyone really
>> seriously against that?
>
> I think it would be better to have it separate.

+1.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  48. Re: Preliminary results for proposed new pgindent implementation

    Stephen Frost <sfrost@snowman.net> — 2017-06-19T18:41:05Z

    * Robert Haas (robertmhaas@gmail.com) wrote:
> On Sat, Jun 17, 2017 at 5:41 PM, Peter Eisentraut
> <peter.eisentraut@2ndquadrant.com> wrote:
> > On 6/16/17 10:51, Tom Lane wrote:
> >> So I'm back to the position that we ought to stick the indent
> >> code under src/tools/ in our main repo.  Is anyone really
> >> seriously against that?
> >
> > I think it would be better to have it separate.
> 
> +1.

+1.

Thanks!

Stephen

  49. Re: Preliminary results for proposed new pgindent implementation

    Tom Lane <tgl@sss.pgh.pa.us> — 2017-06-19T19:27:47Z

    Stephen Frost <sfrost@snowman.net> writes:
> * Robert Haas (robertmhaas@gmail.com) wrote:
>> On Sat, Jun 17, 2017 at 5:41 PM, Peter Eisentraut
>> <peter.eisentraut@2ndquadrant.com> wrote:
>>> On 6/16/17 10:51, Tom Lane wrote:
>>>> So I'm back to the position that we ought to stick the indent
>>>> code under src/tools/ in our main repo.  Is anyone really
>>>> seriously against that?

>>> I think it would be better to have it separate.

>> +1.

> +1.

Given the license issues raised downthread, we have no choice in
the short term.  So I have a request in to create a separate repo
on git.postgresql.org (whose chain do I need to pull to get that
approved, btw?)

			regards, tom lane

  50. Re: Preliminary results for proposed new pgindent implementation

    Stephen Frost <sfrost@snowman.net> — 2017-06-19T19:31:00Z

    * Tom Lane (tgl@sss.pgh.pa.us) wrote:
> Stephen Frost <sfrost@snowman.net> writes:
> > * Robert Haas (robertmhaas@gmail.com) wrote:
> >> On Sat, Jun 17, 2017 at 5:41 PM, Peter Eisentraut
> >> <peter.eisentraut@2ndquadrant.com> wrote:
> >>> On 6/16/17 10:51, Tom Lane wrote:
> >>>> So I'm back to the position that we ought to stick the indent
> >>>> code under src/tools/ in our main repo.  Is anyone really
> >>>> seriously against that?
> 
> >>> I think it would be better to have it separate.
> 
> >> +1.
> 
> > +1.
> 
> Given the license issues raised downthread, we have no choice in
> the short term.  So I have a request in to create a separate repo
> on git.postgresql.org (whose chain do I need to pull to get that
> approved, btw?)

uhhhh, that would probably be pginfra in some capacity, but I don't
recall seeing any notification of such a request.

I will follow up with those responsible,  #blamemagnus

Thanks!

Stephen