Thread

1. Support for pg_receivexlog --post-segment command

   Feike Steenbergen <feikesteenbergen@gmail.com> — 2017-01-06T12:45:09Z

   Hi all,
   
   When reading through "Support for pg_receivexlog --format=plain|tar"[1], I
   came across a notion from Magnus Hagander that has crossed my mind a few
   times as well in the past years. As the feature proposed here is not
   directly related to that thread, I thought it best to start a new thread to
   discuss.
   
   > I have been talking to David about it a couple of times, and he agreed
   that it'd be useful to have a post-segment command. We haven't discussed it
   in much detail though. I'll add him to direct-cc here to see if he has any
   further input :)
   
   I'm coming across a few usecases where this would seem very useful. I'm
   looking to push finished segments to some api as soon as possible. Having
   the post-segment command would allow me to get there. I've tried the
   following approaches, none of them are very satisfying however:
   
   - periodic checking of new files (cron)
   - using inotify
   - tailing verbose pg_receivexlog output to see when a segment was switched
   
   > It could be that the best idea is to just notify some other process of
   what's happening. But making it an external command would give that a lot
   of flexibility. Of course, we need to be careful not to put ourselves back
   in the position we are in with archive_command, in that it's very difficult
   to write a good one.
   
   A signal for  would be good enough for my use case, I don't necessarily
   need all the bookkeeping to ensure the post-segment command was finished
   successfully. However I can see people expecting similar behaviour for the
   post-segment command as for the archive_command. If we would use an
   external command, does this also imply that we need some bookkeeping around
   which segments are ready and done, similar to what is done on the server in
   the archive_status directory?
   
   Thanks,
   
   Feike
   
   [1]
   https://www.postgresql.org/message-id/CAB7nPqTEVXjtH+fehcCbP791H71cfLN_p9rrd-h=yMJfshZQWQ@mail.gmail.com
   

2. Re: Support for pg_receivexlog --post-segment command

   Magnus Hagander <magnus@hagander.net> — 2017-01-06T12:50:56Z

   On Fri, Jan 6, 2017 at 1:45 PM, Feike Steenbergen <
   feikesteenbergen@gmail.com> wrote:
   
   > Hi all,
   >
   > When reading through "Support for pg_receivexlog --format=plain|tar"[1], I
   > came across a notion from Magnus Hagander that has crossed my mind a few
   > times as well in the past years. As the feature proposed here is not
   > directly related to that thread, I thought it best to start a new thread to
   > discuss.
   >
   > > I have been talking to David about it a couple of times, and he agreed
   > that it'd be useful to have a post-segment command. We haven't discussed it
   > in much detail though. I'll add him to direct-cc here to see if he has any
   > further input :)
   >
   > I'm coming across a few usecases where this would seem very useful. I'm
   > looking to push finished segments to some api as soon as possible. Having
   > the post-segment command would allow me to get there. I've tried the
   > following approaches, none of them are very satisfying however:
   >
   > - periodic checking of new files (cron)
   > - using inotify
   > - tailing verbose pg_receivexlog output to see when a segment was switched
   >
   
   Yeah, these are all kind of ugly. Or platform specific. Or both.
   
   You can make them work, but they feel hackish.
   
   
   > > It could be that the best idea is to just notify some other process of
   > what's happening. But making it an external command would give that a lot
   > of flexibility. Of course, we need to be careful not to put ourselves back
   > in the position we are in with archive_command, in that it's very difficult
   > to write a good one.
   >
   > A signal for  would be good enough for my use case, I don't necessarily
   > need all the bookkeeping to ensure the post-segment command was finished
   > successfully. However I can see people expecting similar behaviour for the
   > post-segment command as for the archive_command. If we would use an
   > external command, does this also imply that we need some bookkeeping around
   > which segments are ready and done, similar to what is done on the server in
   > the archive_status directory?
   >
   
   I'd really like to avoid having to re-implement that functionality in
   pg_receivexlog. Given the experiences we have with how difficult it is to
   get right in archive_command...
   
   I think we're better off clearly documenting that we don't care about it.
   And basically let the external command be responsible for that part.
   
   So for example, your typical backup manager would listen to this signal or
   whatever to react quickly. But it would *also* have some sort of fallback.
   For example, whenever it's triggered it also checks if there are any
   previous segments it missed (this would also cover the startup sequence).
   
   Since we never actually remove anything (unlike archive_command which has
   the integration with wal segment rotation), I think this can be done
   perfectly safe.
   
   Looking at the usecases where you have been doing it, are there any where
   this would not work?
   
   -- 
    Magnus Hagander
    Me: http://www.hagander.net/
    Work: http://www.redpill-linpro.com/
   

3. Re: Support for pg_receivexlog --post-segment command

   Feike Steenbergen <feikesteenbergen@gmail.com> — 2017-01-06T13:09:48Z

   On 6 January 2017 at 13:50, Magnus Hagander <magnus@hagander.net> wrote:
   > I think we're better off clearly documenting that we don't care about it.
   And basically let the external command be responsible for that part.
   
   > So for example, your typical backup manager would listen to this signal
   or whatever to react quickly. But it would *also* have some sort of
   fallback. For example, whenever it's triggered it also checks if there are
   any previous segments it missed (this would also cover the startup
   sequence).
   
   For me this works fine. I just want to ensure that if there is any work to
   be done, my backup manager will do the work quickly. My implementation
   might be very simply a process that checks every n seconds or when
   signalled.
   
   > Since we never actually remove anything (unlike archive_command which has
   the integration with wal segment rotation), I think this can be done
   perfectly safe.
   >
   > Looking at the usecases where you have been doing it, are there any where
   this would not work?
   
   This would work for all usecases I've come across.
   

4. Re: Support for pg_receivexlog --post-segment command

   David Steele <david@pgmasters.net> — 2017-01-06T13:30:35Z

   On 1/6/17 8:09 AM, Feike Steenbergen wrote:
   > On 6 January 2017 at 13:50, Magnus Hagander <magnus@hagander.net
   > <mailto:magnus@hagander.net>> wrote:
   >> I think we're better off clearly documenting that we don't care about
   > it. And basically let the external command be responsible for that part.
   >
   >> So for example, your typical backup manager would listen to this
   > signal or whatever to react quickly. But it would *also* have some sort
   > of fallback. For example, whenever it's triggered it also checks if
   > there are any previous segments it missed (this would also cover the
   > startup sequence).
   
   I'm fine with the backup manager doing all the work of keeping track of 
   what has been compressed, moved to archive, etc.  No need to reinvent 
   the wheel here.
   
   For my part I still prefer an actual command to be executed so it will 
   start/restart the archiver if it is not already running or died.  This 
   reduces the number of processes that I need to ensure are running.
   
   If the consensus is that a signal is better then I'll make that work.  I 
   will say this raises the bar on what is required to write a good archive 
   command and we already know it is quite a difficult task.
   
   > For me this works fine. I just want to ensure that if there is any work
   > to be done, my backup manager will do the work quickly. My
   > implementation might be very simply a process that checks every n
   > seconds or when signalled.
   >
   >> Since we never actually remove anything (unlike archive_command which
   > has the integration with wal segment rotation), I think this can be done
   > perfectly safe.
   >>
   >> Looking at the usecases where you have been doing it, are there any
   > where this would not work?
   >
   > This would work for all usecases I've come across.
   
   Agreed.
   
   -- 
   -David
   david@pgmasters.net
   
   
   

5. Re: Support for pg_receivexlog --post-segment command

   Magnus Hagander <magnus@hagander.net> — 2017-01-06T13:37:57Z

   On Fri, Jan 6, 2017 at 2:30 PM, David Steele <david@pgmasters.net> wrote:
   
   > On 1/6/17 8:09 AM, Feike Steenbergen wrote:
   >
   >> On 6 January 2017 at 13:50, Magnus Hagander <magnus@hagander.net
   >> <mailto:magnus@hagander.net>> wrote:
   >>
   >>> I think we're better off clearly documenting that we don't care about
   >>>
   >> it. And basically let the external command be responsible for that part.
   >>
   >> So for example, your typical backup manager would listen to this
   >>>
   >> signal or whatever to react quickly. But it would *also* have some sort
   >> of fallback. For example, whenever it's triggered it also checks if
   >> there are any previous segments it missed (this would also cover the
   >> startup sequence).
   >>
   >
   > I'm fine with the backup manager doing all the work of keeping track of
   > what has been compressed, moved to archive, etc.  No need to reinvent the
   > wheel here.
   >
   > For my part I still prefer an actual command to be executed so it will
   > start/restart the archiver if it is not already running or died.  This
   > reduces the number of processes that I need to ensure are running.
   >
   > If the consensus is that a signal is better then I'll make that work.  I
   > will say this raises the bar on what is required to write a good archive
   > command and we already know it is quite a difficult task.
   
   
   I like the idea of a command as well, for flexibility. If you want a
   signal, you can write a trivial command that sends the signal... Maximum
   flexibility, as long as we don't create a lot of caveats for users.
   
   -- 
    Magnus Hagander
    Me: http://www.hagander.net/
    Work: http://www.redpill-linpro.com/
   

6. Re: Support for pg_receivexlog --post-segment command

   Feike Steenbergen <feikesteenbergen@gmail.com> — 2017-01-06T13:49:37Z

   On Fri, Jan 6, 2017 at 2:30 PM, David Steele <david@pgmasters.net> wrote:
   > For my part I still prefer an actual command to be executed so it will
   start/restart the archiver if it is not already running or died.  This
   reduces the number of processes that I need to ensure are running.
   >
   > If the consensus is that a signal is better then I'll make that work.  I
   will say this raises the bar on what is required to write a good archive
   command and we already know it is quite a difficult task.
   
   On 6 January 2017 at 14:37, Magnus Hagander <magnus@hagander.net> wrote:
   > I like the idea of a command as well, for flexibility. If you want a
   signal, you can write a trivial command that sends the signal... Maximum
   flexibility, as long as we don't create a lot of caveats for users.
   
   Agreed, I think it is also easier to understand the mechanism (instead of a
   signal), and would allow for some reuse of already existing scripts.
   
   If we do use a full command (vs a signal), I propose we do also offer the
   %p and %f placeholders for the command.
   

7. Re: Support for pg_receivexlog --post-segment command

   David Steele <david@pgmasters.net> — 2017-01-06T13:55:35Z

   On 1/6/17 8:49 AM, Feike Steenbergen wrote:
   >
   > On Fri, Jan 6, 2017 at 2:30 PM, David Steele <david@pgmasters.net
   > <mailto:david@pgmasters.net>> wrote:
   >> For my part I still prefer an actual command to be executed so it will
   > start/restart the archiver if it is not already running or died.  This
   > reduces the number of processes that I need to ensure are running.
   >>
   >> If the consensus is that a signal is better then I'll make that work.
   > I will say this raises the bar on what is required to write a good
   > archive command and we already know it is quite a difficult task.
   >
   > On 6 January 2017 at 14:37, Magnus Hagander <magnus@hagander.net
   > <mailto:magnus@hagander.net>> wrote:
   >> I like the idea of a command as well, for flexibility. If you want a
   > signal, you can write a trivial command that sends the signal... Maximum
   > flexibility, as long as we don't create a lot of caveats for users.
   >
   > Agreed, I think it is also easier to understand the mechanism (instead
   > of a signal), and would allow for some reuse of already existing scripts.
   >
   > If we do use a full command (vs a signal), I propose we do also offer
   > the %p and %f placeholders for the command.
   
   Agreed.  It shouldn't be that hard and could be very useful.  If nothing 
   else it will eliminate the need to configure path to the pg_receivexlog 
   queue in the archiver.
   
   -- 
   -David
   david@pgmasters.net
   
   
   

8. Re: Support for pg_receivexlog --post-segment command

   Magnus Hagander <magnus@hagander.net> — 2017-01-06T14:42:22Z

   On Fri, Jan 6, 2017 at 2:55 PM, David Steele <david@pgmasters.net> wrote:
   
   > On 1/6/17 8:49 AM, Feike Steenbergen wrote:
   >
   >>
   >> On Fri, Jan 6, 2017 at 2:30 PM, David Steele <david@pgmasters.net
   >> <mailto:david@pgmasters.net>> wrote:
   >>
   >>> For my part I still prefer an actual command to be executed so it will
   >>>
   >> start/restart the archiver if it is not already running or died.  This
   >> reduces the number of processes that I need to ensure are running.
   >>
   >>>
   >>> If the consensus is that a signal is better then I'll make that work.
   >>>
   >> I will say this raises the bar on what is required to write a good
   >> archive command and we already know it is quite a difficult task.
   >>
   >> On 6 January 2017 at 14:37, Magnus Hagander <magnus@hagander.net
   >> <mailto:magnus@hagander.net>> wrote:
   >>
   >>> I like the idea of a command as well, for flexibility. If you want a
   >>>
   >> signal, you can write a trivial command that sends the signal... Maximum
   >> flexibility, as long as we don't create a lot of caveats for users.
   >>
   >> Agreed, I think it is also easier to understand the mechanism (instead
   >> of a signal), and would allow for some reuse of already existing scripts.
   >>
   >> If we do use a full command (vs a signal), I propose we do also offer
   >> the %p and %f placeholders for the command.
   >>
   >
   > Agreed.  It shouldn't be that hard and could be very useful.  If nothing
   > else it will eliminate the need to configure path to the pg_receivexlog
   > queue in the archiver.
   
   
   Is there actual value in providing both %p and %f? It's not like it's
   really hard to do, but since the path will be specified on the same
   commandline, you could just put it in the command?
   
   E.g.
   pg_receivexlog -D /blah/foo --whatever /some/where/myscript %p
   if you want the path in it could just be written
   pg_receivexlog -D /blah/foo --whatever /some/where/myscript /blah/foo/%f
   
   And you have that path in the *same place* already? (unless you want to
   teach the script about it, in which case you want just %f anyway)
   
   -- 
    Magnus Hagander
    Me: http://www.hagander.net/
    Work: http://www.redpill-linpro.com/
   

9. Re: Support for pg_receivexlog --post-segment command

   Feike Steenbergen <feikesteenbergen@gmail.com> — 2017-01-06T15:15:29Z

   On 6 January 2017 at 15:42, Magnus Hagander <magnus@hagander.net> wrote:
   
   > Is there actual value in providing both %p and %f? It's not like it's
   really hard to do, but since the path will be specified on the same
   commandline, you could just put it in the command?
   
   As %f can be determined from %p I don't mind that much. However, having a
   single static --whatever command may be very useful for configuration
   management or backup tools that want to use static commands.
   
   The other reason why I'd offer both is to have some uniformity with
   archive_command, possibly allowing some reuse of code.
   

10. Re: Support for pg_receivexlog --post-segment command

    David Steele <david@pgmasters.net> — 2017-01-06T15:18:00Z

    On 1/6/17 10:15 AM, Feike Steenbergen wrote:
    >
    > On 6 January 2017 at 15:42, Magnus Hagander <magnus@hagander.net
    > <mailto:magnus@hagander.net>> wrote:
    >
    >> Is there actual value in providing both %p and %f? It's not like it's
    > really hard to do, but since the path will be specified on the same
    > commandline, you could just put it in the command?
    >
    > As %f can be determined from %p I don't mind that much. However, having
    > a single static --whatever command may be very useful for configuration
    > management or backup tools that want to use static commands.
    >
    > The other reason why I'd offer both is to have some uniformity with
    > archive_command, possibly allowing some reuse of code.
    
    Agreed.  Currently pgBackRest only accepts %p for archive_command 
    because it already knows where $PGDATA is.  Of course I can change that, 
    but I think it makes sense to keep command options uniform anyway.
    
    -- 
    -David
    david@pgmasters.net