Thread

1. Separate the result of \watch for each query execution (psql)

   Noboru Saito <noborusai@gmail.com> — 2022-02-21T05:19:18Z

   I need a way to separate the results of \watch for each query execution.
   
   There is only a blank line between the results of \watch.
   However, there is also a blank line after the title, which complicates
   the rules.
   
   My suggestion is to insert a "form feed(\f)" (preferably a newline)
   before the result and output it.
   Then, the output will be as follows.
   
       # select now(); \watch 1
       ^L                                          <--- add
       Thu Feb 17 11:52:05 2022 (every 1s)
   
       now
       -------------------------------
       2022-02-17 11:52:05.69394 + 09
       (1 row)
   
       ^L                                           <--- add
       Thu Feb 17 11:52:06 2022 (every 1s)
   
       now
       ------------------------------
       2022-02-17 11:52:06.96906 + 09
       (1 row)
   
   (^L is usually not displayed. It is visualized by passing through a
   filter such as `less`.)
   
   This is possible with the following simple patch.
   
   ```
   diff --git a/src/bin/psql/common.c b/src/bin/psql/common.c
   index d65b9a124f..ee9442d0a6 100644
   --- a/src/bin/psql/common.c
   +++ b/src/bin/psql/common.c
   @@ -646,10 +646,12 @@ PSQLexecWatch(const char *query, const
   printQueryOpt *opt, FILE *printQueryFout)
     switch (PQresultStatus(res))
     {
     case PGRES_TUPLES_OK:
   + fprintf(fout, "\f\n");
     printQuery(res, opt, fout, false, pset.logfile);
     break;
   
     case PGRES_COMMAND_OK:
   + fprintf(fout, "\f\n");
     fprintf(fout, "%s\n%s\n\n", opt->title, PQcmdStatus(res));
     break;
   
   ```
   
   I am developing a terminal pager ov (https://github.com/noborus/ov).
   It's a generic pager, but it has features suitable for use with psql.
   
   I found that \watch supports PAGER in PostgreSQL 15. That's great.
   
   ov can be received and displayed, but I want to display it from the
   beginning every time it is executed (like pspg).
   
   The current output is a bit difficult to clear and display for each result.
   

2. Re: Separate the result of \watch for each query execution (psql)

   Pavel Stehule <pavel.stehule@gmail.com> — 2022-02-21T05:52:18Z

   Hi
   
   po 21. 2. 2022 v 6:19 odesílatel Noboru Saito <noborusai@gmail.com> napsal:
   
   > I need a way to separate the results of \watch for each query execution.
   >
   > There is only a blank line between the results of \watch.
   > However, there is also a blank line after the title, which complicates
   > the rules.
   >
   > My suggestion is to insert a "form feed(\f)" (preferably a newline)
   > before the result and output it.
   > Then, the output will be as follows.
   >
   >     # select now(); \watch 1
   >     ^L                                          <--- add
   >     Thu Feb 17 11:52:05 2022 (every 1s)
   >
   >     now
   >     -------------------------------
   >     2022-02-17 11:52:05.69394 + 09
   >     (1 row)
   >
   >     ^L                                           <--- add
   >     Thu Feb 17 11:52:06 2022 (every 1s)
   >
   >     now
   >     ------------------------------
   >     2022-02-17 11:52:06.96906 + 09
   >     (1 row)
   >
   > (^L is usually not displayed. It is visualized by passing through a
   > filter such as `less`.)
   >
   > This is possible with the following simple patch.
   >
   > ```
   > diff --git a/src/bin/psql/common.c b/src/bin/psql/common.c
   > index d65b9a124f..ee9442d0a6 100644
   > --- a/src/bin/psql/common.c
   > +++ b/src/bin/psql/common.c
   > @@ -646,10 +646,12 @@ PSQLexecWatch(const char *query, const
   > printQueryOpt *opt, FILE *printQueryFout)
   >   switch (PQresultStatus(res))
   >   {
   >   case PGRES_TUPLES_OK:
   > + fprintf(fout, "\f\n");
   >   printQuery(res, opt, fout, false, pset.logfile);
   >   break;
   >
   >   case PGRES_COMMAND_OK:
   > + fprintf(fout, "\f\n");
   >   fprintf(fout, "%s\n%s\n\n", opt->title, PQcmdStatus(res));
   >   break;
   >
   > ```
   >
   > I am developing a terminal pager ov (https://github.com/noborus/ov).
   > It's a generic pager, but it has features suitable for use with psql.
   >
   > I found that \watch supports PAGER in PostgreSQL 15. That's great.
   >
   > ov can be received and displayed, but I want to display it from the
   > beginning every time it is executed (like pspg).
   >
   > The current output is a bit difficult to clear and display for each result.
   >
   
   I strongly agree. It was a lot of work to find a workable solution for
   pspg. Special chars that starting result and maybe other, that ending
   result can significantly increase robustness and can reduce code. I think
   it can be better to use form feed at the end of form - like it is semantic
   of form feed. You know, at this moment, the result is complete.
   https://en.wikipedia.org/wiki/Page_break
   
   I don't think using it by default can be the best. Lot of people don't use
   specialized pagers, but it can be set by \pset. Form feed should be used on
   end
   
   \pset formfeed [on, off]
   
   Ascii has few nice characters, that we can use
   
   1 .. SOX - start of header
   2 .. STX - start of text
   3 .. ETX - end of text
   
   Using it, it can reduce some heuristic in pspg, that is not fully 100% when
   border is not 2.
   
   But implementation of formfeed support can be a very good start. Any mark
   that can be used for synchronization can help a lot.
   
   Regards
   
   Pavel
   

3. Re: Separate the result of \watch for each query execution (psql)

   Noboru Saito <noborusai@gmail.com> — 2022-02-21T13:23:16Z

   Thank you for your reply.
   
   2022年2月21日(月) 14:52 Pavel Stehule <pavel.stehule@gmail.com>:
   
   >
   > Hi
   >
   > po 21. 2. 2022 v 6:19 odesílatel Noboru Saito <noborusai@gmail.com> napsal:
   >>
   >> I need a way to separate the results of \watch for each query execution.
   >>
   >> There is only a blank line between the results of \watch.
   >> However, there is also a blank line after the title, which complicates
   >> the rules.
   >>
   >> My suggestion is to insert a "form feed(\f)" (preferably a newline)
   >> before the result and output it.
   >> Then, the output will be as follows.
   >>
   >>     # select now(); \watch 1
   >>     ^L                                          <--- add
   >>     Thu Feb 17 11:52:05 2022 (every 1s)
   >>
   >>     now
   >>     -------------------------------
   >>     2022-02-17 11:52:05.69394 + 09
   >>     (1 row)
   >>
   >>     ^L                                           <--- add
   >>     Thu Feb 17 11:52:06 2022 (every 1s)
   >>
   >>     now
   >>     ------------------------------
   >>     2022-02-17 11:52:06.96906 + 09
   >>     (1 row)
   >>
   >> (^L is usually not displayed. It is visualized by passing through a
   >> filter such as `less`.)
   >>
   >> This is possible with the following simple patch.
   >>
   >> ```
   >> diff --git a/src/bin/psql/common.c b/src/bin/psql/common.c
   >> index d65b9a124f..ee9442d0a6 100644
   >> --- a/src/bin/psql/common.c
   >> +++ b/src/bin/psql/common.c
   >> @@ -646,10 +646,12 @@ PSQLexecWatch(const char *query, const
   >> printQueryOpt *opt, FILE *printQueryFout)
   >>   switch (PQresultStatus(res))
   >>   {
   >>   case PGRES_TUPLES_OK:
   >> + fprintf(fout, "\f\n");
   >>   printQuery(res, opt, fout, false, pset.logfile);
   >>   break;
   >>
   >>   case PGRES_COMMAND_OK:
   >> + fprintf(fout, "\f\n");
   >>   fprintf(fout, "%s\n%s\n\n", opt->title, PQcmdStatus(res));
   >>   break;
   >>
   >> ```
   >>
   >> I am developing a terminal pager ov (https://github.com/noborus/ov).
   >> It's a generic pager, but it has features suitable for use with psql.
   >>
   >> I found that \watch supports PAGER in PostgreSQL 15. That's great.
   >>
   >> ov can be received and displayed, but I want to display it from the
   >> beginning every time it is executed (like pspg).
   >>
   >> The current output is a bit difficult to clear and display for each result.
   >
   >
   > I strongly agree. It was a lot of work to find a workable solution for pspg. Special chars that starting result and maybe other, that ending result can significantly increase robustness and can reduce code. I think it can be better to use form feed at the end of form - like it is semantic of form feed. You know, at this moment, the result is complete. https://en.wikipedia.org/wiki/Page_break
   
   It's easier to print a form feed before the result, but it's okay at the end.
   
   > I don't think using it by default can be the best. Lot of people don't use specialized pagers, but it can be set by \pset. Form feed should be used on end
   >
   > \pset formfeed [on, off]
   
   I think it's a good idea to be able to switch with \pset.
   
   > Ascii has few nice characters, that we can use
   >
   > 1 .. SOX - start of header
   > 2 .. STX - start of text
   > 3 .. ETX - end of text
   >
   > Using it, it can reduce some heuristic in pspg, that is not fully 100% when border is not 2.
   >
   > But implementation of formfeed support can be a very good start. Any mark that can be used for synchronization can help a lot.
   >
   > Regards
   >
   > Pavel
   
   Thank you.
   
   
   
   

4. Re: Separate the result of \watch for each query execution (psql)

   Noboru Saito <noborusai@gmail.com> — 2022-02-25T04:23:31Z

   Hi,
   
   Pavel Stehule <pavel.stehule@gmail.com>:
   > > I strongly agree. It was a lot of work to find a workable solution for pspg. Special chars that starting result and maybe other, that ending result can significantly increase robustness and can reduce code. I think it can be better to use form feed at the end of form - like it is semantic of form feed. You know, at this moment, the result is complete. https://en.wikipedia.org/wiki/Page_break
   >
   > It's easier to print a form feed before the result, but it's okay at the end.
   >
   > > I don't think using it by default can be the best. Lot of people don't use specialized pagers, but it can be set by \pset. Form feed should be used on end
   > >
   > > \pset formfeed [on, off]
   >
   > I think it's a good idea to be able to switch with \pset.
   
   I have created a patch that allows you to turn it on and off in \pset.
   The attached patch adds the following features.
   
   
   Formfeed can be turned on with the command line option or \pset.
   Formfeed (\f\n) is output after the query execution result by \watch.
   
   I think the considerations are as follows.
   
   * Is formfeed output after the result, not before?
   * Is the formfeed output only "\f\n"?
   * Is the formfeed output only executed by \watch?
   * Is the name "formfeed" appropriate?
   
   If the formfeed is output before the query result,
   it will be better if the screen is reset when the formfeed is read.
   
   Furthermore, if the terminal clear string can be set instead of the
   formfeed character,
   the \watch output can be fixedly displayed without a pager.
   
   (I noticed that if I set it in the title, it would behave similarly in
   the current version.
       # \C '\033[2J;\033[0;0H'
       # SELECT now();\watch 1
   )
   
   Also, it may be good to output formfeed when outputting a file with
   `\o result.txt`
   other than \watch.
   

5. Re: Separate the result of \watch for each query execution (psql)

   Pavel Stehule <pavel.stehule@gmail.com> — 2022-02-25T04:41:41Z

   pá 25. 2. 2022 v 5:23 odesílatel Noboru Saito <noborusai@gmail.com> napsal:
   
   > Hi,
   >
   > Pavel Stehule <pavel.stehule@gmail.com>:
   > > > I strongly agree. It was a lot of work to find a workable solution for
   > pspg. Special chars that starting result and maybe other, that ending
   > result can significantly increase robustness and can reduce code. I think
   > it can be better to use form feed at the end of form - like it is semantic
   > of form feed. You know, at this moment, the result is complete.
   > https://en.wikipedia.org/wiki/Page_break
   > >
   > > It's easier to print a form feed before the result, but it's okay at the
   > end.
   > >
   > > > I don't think using it by default can be the best. Lot of people don't
   > use specialized pagers, but it can be set by \pset. Form feed should be
   > used on end
   > > >
   > > > \pset formfeed [on, off]
   > >
   > > I think it's a good idea to be able to switch with \pset.
   >
   > I have created a patch that allows you to turn it on and off in \pset.
   > The attached patch adds the following features.
   >
   >
   > Formfeed can be turned on with the command line option or \pset.
   > Formfeed (\f\n) is output after the query execution result by \watch.
   >
   > I think the considerations are as follows.
   >
   > * Is formfeed output after the result, not before?
   >
   
   We are talking about the first iteration. In the second and other iteration
   this question has no sense. You know the starting point. You don't know the
   endpoint. So I think so using formfeed on the end is good idea.
   
   * Is the formfeed output only "\f\n"?
   >
   
   yes
   
   
   > * Is the formfeed output only executed by \watch?
   >
   
   This is a good question. I think the implementation for \watch is a good
   start. But it can help with normal results too. In pspg it can be very
   useful for incremental load or for streaming mode. But if it will be used
   everywhere, then it should be used just for some specified pagers.
   
   
   
   > * Is the name "formfeed" appropriate?
   >
   
   If it will do work of formfeed, then the formfeed is good name.
   
   
   >
   > If the formfeed is output before the query result,
   > it will be better if the screen is reset when the formfeed is read.
   >
   
   I think, so you propose another feature - reset terminal sequence - it can
   be a good idea. Not for pspg, but generally, why not.
   
   Regards
   
   Pavel
   
   
   >
   > Furthermore, if the terminal clear string can be set instead of the
   > formfeed character,
   > the \watch output can be fixedly displayed without a pager.
   >
   > (I noticed that if I set it in the title, it would behave similarly in
   > the current version.
   >     # \C '\033[2J;\033[0;0H'
   >     # SELECT now();\watch 1
   > )
   >
   > Also, it may be good to output formfeed when outputting a file with
   > `\o result.txt`
   > other than \watch.
   >
   

6. Re: Separate the result of \watch for each query execution (psql)

   Noboru Saito <noborusai@gmail.com> — 2022-02-27T21:05:44Z

   Hi,
   
   2022年2月25日(金) 13:42 Pavel Stehule <pavel.stehule@gmail.com>:
   >
   >
   >
   > pá 25. 2. 2022 v 5:23 odesílatel Noboru Saito <noborusai@gmail.com> napsal:
   >>
   >> Hi,
   >>
   >> Pavel Stehule <pavel.stehule@gmail.com>:
   >> > > I strongly agree. It was a lot of work to find a workable solution for pspg. Special chars that starting result and maybe other, that ending result can significantly increase robustness and can reduce code. I think it can be better to use form feed at the end of form - like it is semantic of form feed. You know, at this moment, the result is complete. https://en.wikipedia.org/wiki/Page_break
   >> >
   >> > It's easier to print a form feed before the result, but it's okay at the end.
   >> >
   >> > > I don't think using it by default can be the best. Lot of people don't use specialized pagers, but it can be set by \pset. Form feed should be used on end
   >> > >
   >> > > \pset formfeed [on, off]
   >> >
   >> > I think it's a good idea to be able to switch with \pset.
   >>
   >> I have created a patch that allows you to turn it on and off in \pset.
   >> The attached patch adds the following features.
   >>
   >>
   >> Formfeed can be turned on with the command line option or \pset.
   >> Formfeed (\f\n) is output after the query execution result by \watch.
   >>
   >> I think the considerations are as follows.
   >>
   >> * Is formfeed output after the result, not before?
   >
   >
   > We are talking about the first iteration. In the second and other iteration this question has no sense. You know the starting point. You don't know the endpoint. So I think so using formfeed on the end is good idea.
   
   Yes. However, as you know, in the case of \ watch, there is a
   difference in waiting before and after the specified time.
   
   >> * Is the formfeed output only "\f\n"?
   >
   >
   > yes
   >
   >>
   >> * Is the formfeed output only executed by \watch?
   >
   >
   > This is a good question. I think the implementation for \watch is a good start. But it can help with normal results too. In pspg it can be very useful for incremental load or for streaming mode. But if it will be used everywhere, then it should be used just for some specified pagers.
   >
   >
   >>
   >> * Is the name "formfeed" appropriate?
   >
   >
   > If it will do work of formfeed, then the formfeed is good name.
   >
   >>
   >>
   >> If the formfeed is output before the query result,
   >> it will be better if the screen is reset when the formfeed is read.
   >
   >
   > I think, so you propose another feature - reset terminal sequence - it can be a good idea. Not for pspg, but generally, why not.
   
   Yes, it is. I was wondering if I could just add formfeed or add more features.
   
   Thank you.
   Following your advice, I will register this for the commitfest
    (I am registering for the commitfest for the first time. Thank you
   for your cooperation).
   
   
   
   

7. Re: Separate the result of \watch for each query execution (psql)

   Tom Lane <tgl@sss.pgh.pa.us> — 2022-02-28T22:46:43Z

   Noboru Saito <noborusai@gmail.com> writes:
   > I have created a patch that allows you to turn it on and off in \pset.
   > The attached patch adds the following features.
   > Formfeed can be turned on with the command line option or \pset.
   > Formfeed (\f\n) is output after the query execution result by \watch.
   
   Hmm ... I grant your use-case for this, but I think the patch
   is too narrow-minded, because it supposes that the only string
   anybody could wish to output between \watch commands is "\f\n".
   Once you open the floodgates of inserting formatting there,
   ISTM that people might want other things.
   
   Also, I'm not that thrilled with treating this as a \pset option,
   because it has nothing to do with formatting of normal query
   results.  (IMV anyway, perhaps others will disagree.)
   
   How about instead of defining fixed semantics, we invent a psql
   special variable that can contain a string to be output between
   \watch commands?  It looks like you could then set it through
   a command like
   
   \set WATCH_SEPARATOR '\f\n'
   
   (not wedded to that variable name, it's just the first idea
   that came to mind)
   
   Personally I'd not bother with inventing a specialized command-line
   option to set it, either.  There's already -v and friends.
   
   > * Is formfeed output after the result, not before?
   
   Or we could invent WATCH_BEFORE and WATCH_AFTER ...
   
   			regards, tom lane
   
   
   
   

8. Re: Separate the result of \watch for each query execution (psql)

   Pavel Stehule <pavel.stehule@gmail.com> — 2022-03-01T05:42:54Z

   po 28. 2. 2022 v 23:46 odesílatel Tom Lane <tgl@sss.pgh.pa.us> napsal:
   
   > Noboru Saito <noborusai@gmail.com> writes:
   > > I have created a patch that allows you to turn it on and off in \pset.
   > > The attached patch adds the following features.
   > > Formfeed can be turned on with the command line option or \pset.
   > > Formfeed (\f\n) is output after the query execution result by \watch.
   >
   > Hmm ... I grant your use-case for this, but I think the patch
   > is too narrow-minded, because it supposes that the only string
   > anybody could wish to output between \watch commands is "\f\n".
   > Once you open the floodgates of inserting formatting there,
   > ISTM that people might want other things.
   >
   > Also, I'm not that thrilled with treating this as a \pset option,
   > because it has nothing to do with formatting of normal query
   > results.  (IMV anyway, perhaps others will disagree.)
   >
   
   pspg (and ov pager too) supports streaming (pspg is used in another
   terminal than psql), and for this case, the marks can be useful for all
   modes.
   
   Regards
   
   Pavel
   
   
   >
   > How about instead of defining fixed semantics, we invent a psql
   > special variable that can contain a string to be output between
   > \watch commands?  It looks like you could then set it through
   > a command like
   >
   > \set WATCH_SEPARATOR '\f\n'
   >
   > (not wedded to that variable name, it's just the first idea
   > that came to mind)
   >
   > Personally I'd not bother with inventing a specialized command-line
   > option to set it, either.  There's already -v and friends.
   >
   > > * Is formfeed output after the result, not before?
   >
   > Or we could invent WATCH_BEFORE and WATCH_AFTER ...
   >
   >                         regards, tom lane
   >
   >
   >
   

9. Re: Separate the result of \watch for each query execution (psql)

   Noboru Saito <noborusai@gmail.com> — 2022-03-02T00:57:21Z

   Tom Lane <tgl@sss.pgh.pa.us>:
   > Noboru Saito <noborusai@gmail.com> writes:
   > > I have created a patch that allows you to turn it on and off in \pset.
   > > The attached patch adds the following features.
   > > Formfeed can be turned on with the command line option or \pset.
   > > Formfeed (\f\n) is output after the query execution result by \watch.
   >
   > Hmm ... I grant your use-case for this, but I think the patch
   > is too narrow-minded, because it supposes that the only string
   > anybody could wish to output between \watch commands is "\f\n".
   > Once you open the floodgates of inserting formatting there,
   > ISTM that people might want other things.
   >
   > Also, I'm not that thrilled with treating this as a \pset option,
   > because it has nothing to do with formatting of normal query
   > results.  (IMV anyway, perhaps others will disagree.)
   >
   > How about instead of defining fixed semantics, we invent a psql
   > special variable that can contain a string to be output between
   > \watch commands?  It looks like you could then set it through
   > a command like
   
   I understand because it's a problem I was worried about.
   However, we need a machine-readable string for PAGER.
   It didn't exist before, so newcomers will match it to a fixed string.
   The de facto standard is required even if it can be changed.
   
   > \set WATCH_SEPARATOR '\f\n'
   >
   > (not wedded to that variable name, it's just the first idea
   > that came to mind)
   
   I think it's better not to include "WATCH" in the name,
   as it may be used in the future other than \watch.
   
   > Personally I'd not bother with inventing a specialized command-line
   > option to set it, either.  There's already -v and friends.
   >
   > > * Is formfeed output after the result, not before?
   >
   > Or we could invent WATCH_BEFORE and WATCH_AFTER ...
   >
   >                         regards, tom lane
   
   It may be good to add it in the future.
   For now, I'm fine with either one.
   
   Thank you.
   
   
   
   

10. Re: Separate the result of \watch for each query execution (psql)

    Andres Freund <andres@anarazel.de> — 2022-03-22T01:15:40Z

    Hi,
    
    On 2022-02-25 13:23:31 +0900, Noboru Saito wrote:
    > I have created a patch that allows you to turn it on and off in \pset.
    
    The patch unfortunately causes tests to fail:
    https://cirrus-ci.com/task/5932406812180480
    diff -U3 /tmp/cirrus-ci-build/src/test/regress/expected/psql.out /tmp/cirrus-ci-build/src/test/regress/results/psql.out
    --- /tmp/cirrus-ci-build/src/test/regress/expected/psql.out	2022-03-21 09:55:55.875784000 +0000
    +++ /tmp/cirrus-ci-build/src/test/regress/results/psql.out	2022-03-21 09:59:51.950512000 +0000
    @@ -304,6 +304,7 @@
     fieldsep_zero            off
     footer                   on
     format                   aligned
    +formfeed                 off
     linestyle                ascii
     null                     ''
     numericlocale            off
    
    Greetings,
    
    Andres Freund
    
    
    
    

11. Re: Separate the result of \watch for each query execution (psql)

    Robert Haas <robertmhaas@gmail.com> — 2022-04-05T15:53:04Z

    On Mon, Mar 21, 2022 at 9:15 PM Andres Freund <andres@anarazel.de> wrote:
    > On 2022-02-25 13:23:31 +0900, Noboru Saito wrote:
    > > I have created a patch that allows you to turn it on and off in \pset.
    >
    > The patch unfortunately causes tests to fail:
    
    It doesn't seem like the originally proposed design here will be
    accepted. Perhaps a patch implementing what Tom proposed or some
    variant of it will get accepted, but the only patch we have is now
    more than 1 month old and has not been updated. I'm accordingly
    marking this CommitFest entry "Returned with Feedback." If Noboru
    Saito decides to update the patch with some new design, he can change
    the status and move the patch forward to a future CommitFest.
    
    -- 
    Robert Haas
    EDB: http://www.enterprisedb.com