Thread

1. pg_retainxlog for inclusion in 9.3?

   Magnus Hagander <magnus@hagander.net> — 2013-01-01T15:10:32Z

   So, it turns out the reason I got no feedback on this tool, was that I
   forgot both to email about and to actually push the code to github :O
   So this is actually code that's almost half a year old and that I was
   supposed to submit for the first or second commitfest. Oops.
   
   So, the tool and a README for it right now can be found at
   https://github.com/mhagander/pg_retainxlog for the time being.
   
   The idea behind the tool is to plug a hole in the case when
   pg_receivexlog is used for log archiving, which is that you still need
   a "blocking" archive_command in order to make sure that files aren't
   recycled on the master. So for 9.2 you can do this with an
   archive_command that checks if the file has appeared properly on the
   slave - but that usually means you're back at requiring ssh
   connectivity between the machines, for example. Even though this
   information is actually avialable on the master...
   
   This can also be of use to pure replication scenarios, where you don't
   know how to tune wal_keep_segments, but using actual live feedback
   instead of guessing.
   
   When pg_retainxlog is used as an archive_command, it will check the
   pg_stat_replication view instead of checking the slave. It will then
   only return ok once the requested logfile has been replicated to the
   slave. By default it will look for a replication client named
   pg_receivexlog, but it supports overriding the query with anything -
   so you can say things like "needs to have arrived on at least two
   replication slaves before we consider it archived". Or if used instead
   of wal_keep_segmnets, needs to have arrived at *all* replication
   slaves.
   
   Is this a tool that people would like to see included in the general
   toolchain? If so, I'll reformat it to work in the general build
   environment and submit it for the last commitfest.
   
   (comments on the code itself are of course also welcome)
   
   --
    Magnus Hagander
    Me: http://www.hagander.net/
    Work: http://www.redpill-linpro.com/
   
   
   

2. Re: pg_retainxlog for inclusion in 9.3?

   Hannu Krosing <hannu@2ndquadrant.com> — 2013-01-01T16:07:22Z

   On 01/01/2013 04:10 PM, Magnus Hagander wrote:
   > So, it turns out the reason I got no feedback on this tool, was that I
   > forgot both to email about and to actually push the code to github :O
   > So this is actually code that's almost half a year old and that I was
   > supposed to submit for the first or second commitfest. Oops.
   >
   > So, the tool and a README for it right now can be found at
   > https://github.com/mhagander/pg_retainxlog for the time being.
   >
   > The idea behind the tool is to plug a hole in the case when
   > pg_receivexlog is used for log archiving, which is that you still need
   > a "blocking" archive_command in order to make sure that files aren't
   > recycled on the master. So for 9.2 you can do this with an
   > archive_command that checks if the file has appeared properly on the
   > slave - but that usually means you're back at requiring ssh
   > connectivity between the machines, for example. Even though this
   > information is actually avialable on the master...
   >
   > This can also be of use to pure replication scenarios, where you don't
   > know how to tune wal_keep_segments, but using actual live feedback
   > instead of guessing.
   >
   > When pg_retainxlog is used as an archive_command, it will check the
   > pg_stat_replication view instead of checking the slave. It will then
   > only return ok once the requested logfile has been replicated to the
   > slave. By default it will look for a replication client named
   > pg_receivexlog, but it supports overriding the query with anything -
   > so you can say things like "needs to have arrived on at least two
   > replication slaves before we consider it archived". Or if used instead
   > of wal_keep_segmnets, needs to have arrived at *all* replication
   > slaves.
   >
   > Is this a tool that people would like to see included in the general
   > toolchain? If so, I'll reformat it to work in the general build
   > environment and submit it for the last commitfest.
   >
   
   +1
   
   ------------------------
   Hannu Krosing
   
   
   

3. Re: pg_retainxlog for inclusion in 9.3?

   Dimitri Fontaine <dimitri@2ndquadrant.fr> — 2013-01-01T16:52:57Z

   Magnus Hagander <magnus@hagander.net> writes:
   > Is this a tool that people would like to see included in the general
   > toolchain? If so, I'll reformat it to work in the general build
   > environment and submit it for the last commitfest.
   
   Please do.
   
   Regards,
   -- 
   Dimitri Fontaine
   http://2ndQuadrant.fr     PostgreSQL : Expertise, Formation et Support
   
   
   

4. Re: pg_retainxlog for inclusion in 9.3?

   Robert Haas <robertmhaas@gmail.com> — 2013-01-03T14:13:30Z

   On Tue, Jan 1, 2013 at 10:10 AM, Magnus Hagander <magnus@hagander.net> wrote:
   > So, it turns out the reason I got no feedback on this tool, was that I
   > forgot both to email about and to actually push the code to github :O
   > So this is actually code that's almost half a year old and that I was
   > supposed to submit for the first or second commitfest. Oops.
   >
   > So, the tool and a README for it right now can be found at
   > https://github.com/mhagander/pg_retainxlog for the time being.
   >
   > The idea behind the tool is to plug a hole in the case when
   > pg_receivexlog is used for log archiving, which is that you still need
   > a "blocking" archive_command in order to make sure that files aren't
   > recycled on the master. So for 9.2 you can do this with an
   > archive_command that checks if the file has appeared properly on the
   > slave - but that usually means you're back at requiring ssh
   > connectivity between the machines, for example. Even though this
   > information is actually avialable on the master...
   >
   > This can also be of use to pure replication scenarios, where you don't
   > know how to tune wal_keep_segments, but using actual live feedback
   > instead of guessing.
   >
   > When pg_retainxlog is used as an archive_command, it will check the
   > pg_stat_replication view instead of checking the slave. It will then
   > only return ok once the requested logfile has been replicated to the
   > slave. By default it will look for a replication client named
   > pg_receivexlog, but it supports overriding the query with anything -
   > so you can say things like "needs to have arrived on at least two
   > replication slaves before we consider it archived". Or if used instead
   > of wal_keep_segmnets, needs to have arrived at *all* replication
   > slaves.
   >
   > Is this a tool that people would like to see included in the general
   > toolchain? If so, I'll reformat it to work in the general build
   > environment and submit it for the last commitfest.
   >
   > (comments on the code itself are of course also welcome)
   
   I would tend to vote for putting this in contrib rather than src/bin.
   But apart from that I have no objection to the idea.
   
   -- 
   Robert Haas
   EnterpriseDB: http://www.enterprisedb.com
   The Enterprise PostgreSQL Company
   
   
   

5. Re: pg_retainxlog for inclusion in 9.3?

   Magnus Hagander <magnus@hagander.net> — 2013-01-03T16:32:19Z

   On Thu, Jan 3, 2013 at 3:13 PM, Robert Haas <robertmhaas@gmail.com> wrote:
   > On Tue, Jan 1, 2013 at 10:10 AM, Magnus Hagander <magnus@hagander.net> wrote:
   >> So, it turns out the reason I got no feedback on this tool, was that I
   >> forgot both to email about and to actually push the code to github :O
   >> So this is actually code that's almost half a year old and that I was
   >> supposed to submit for the first or second commitfest. Oops.
   >>
   >> So, the tool and a README for it right now can be found at
   >> https://github.com/mhagander/pg_retainxlog for the time being.
   >>
   >> The idea behind the tool is to plug a hole in the case when
   >> pg_receivexlog is used for log archiving, which is that you still need
   >> a "blocking" archive_command in order to make sure that files aren't
   >> recycled on the master. So for 9.2 you can do this with an
   >> archive_command that checks if the file has appeared properly on the
   >> slave - but that usually means you're back at requiring ssh
   >> connectivity between the machines, for example. Even though this
   >> information is actually avialable on the master...
   >>
   >> This can also be of use to pure replication scenarios, where you don't
   >> know how to tune wal_keep_segments, but using actual live feedback
   >> instead of guessing.
   >>
   >> When pg_retainxlog is used as an archive_command, it will check the
   >> pg_stat_replication view instead of checking the slave. It will then
   >> only return ok once the requested logfile has been replicated to the
   >> slave. By default it will look for a replication client named
   >> pg_receivexlog, but it supports overriding the query with anything -
   >> so you can say things like "needs to have arrived on at least two
   >> replication slaves before we consider it archived". Or if used instead
   >> of wal_keep_segmnets, needs to have arrived at *all* replication
   >> slaves.
   >>
   >> Is this a tool that people would like to see included in the general
   >> toolchain? If so, I'll reformat it to work in the general build
   >> environment and submit it for the last commitfest.
   >>
   >> (comments on the code itself are of course also welcome)
   >
   > I would tend to vote for putting this in contrib rather than src/bin.
   > But apart from that I have no objection to the idea.
   
   Any particular reason? It goes pretty tightly together with
   pg_receivexlog, which is why I'd prefer putting it alongside that one.
   But if you have a good argument against it, I can change my mind :)
   
   --
    Magnus Hagander
    Me: http://www.hagander.net/
    Work: http://www.redpill-linpro.com/
   
   
   

6. Re: pg_retainxlog for inclusion in 9.3?

   Robert Haas <robertmhaas@gmail.com> — 2013-01-03T17:30:39Z

   On Thu, Jan 3, 2013 at 11:32 AM, Magnus Hagander <magnus@hagander.net> wrote:
   > Any particular reason? It goes pretty tightly together with
   > pg_receivexlog, which is why I'd prefer putting it alongside that one.
   > But if you have a good argument against it, I can change my mind :)
   
   Mostly that it seems like a hack, and I suspect we may come up with a
   better way to do this in the future.
   
   -- 
   Robert Haas
   EnterpriseDB: http://www.enterprisedb.com
   The Enterprise PostgreSQL Company
   
   
   

7. Re: pg_retainxlog for inclusion in 9.3?

   Peter Eisentraut <peter_e@gmx.net> — 2013-01-04T18:13:59Z

   On 1/3/13 12:30 PM, Robert Haas wrote:
   > On Thu, Jan 3, 2013 at 11:32 AM, Magnus Hagander <magnus@hagander.net> wrote:
   >> Any particular reason? It goes pretty tightly together with
   >> pg_receivexlog, which is why I'd prefer putting it alongside that one.
   >> But if you have a good argument against it, I can change my mind :)
   > 
   > Mostly that it seems like a hack, and I suspect we may come up with a
   > better way to do this in the future.
   
   It does seem like a hack.  Couldn't this be implemented with a backend
   switch instead?
   
   Also, as a small practical matter, since this is a server-side program
   (since it's being used as archive_command), we shouldn't put it into the
   pg_basebackup directory, because that would blur the lines about what to
   install where, in particular for the translations.
   
   
   
   
   

8. Re: pg_retainxlog for inclusion in 9.3?

   Dimitri Fontaine <dimitri@2ndquadrant.fr> — 2013-01-04T21:55:34Z

   Robert Haas <robertmhaas@gmail.com> writes:
   > Mostly that it seems like a hack, and I suspect we may come up with a
   > better way to do this in the future.
   
   Do you have the specs of such better way? Would it be a problem to have
   both pg_retainxlog and the new way?
   
   -- 
   Dimitri Fontaine
   http://2ndQuadrant.fr     PostgreSQL : Expertise, Formation et Support
   
   
   

9. Re: pg_retainxlog for inclusion in 9.3?

   Magnus Hagander <magnus@hagander.net> — 2013-01-05T14:11:26Z

   On Fri, Jan 4, 2013 at 7:13 PM, Peter Eisentraut <peter_e@gmx.net> wrote:
   > On 1/3/13 12:30 PM, Robert Haas wrote:
   >> On Thu, Jan 3, 2013 at 11:32 AM, Magnus Hagander <magnus@hagander.net> wrote:
   >>> Any particular reason? It goes pretty tightly together with
   >>> pg_receivexlog, which is why I'd prefer putting it alongside that one.
   >>> But if you have a good argument against it, I can change my mind :)
   >>
   >> Mostly that it seems like a hack, and I suspect we may come up with a
   >> better way to do this in the future.
   >
   > It does seem like a hack.  Couldn't this be implemented with a backend
   > switch instead?
   
   It definitely is a bit of a hack.
   
   I assume by backend switch you mean guc, right? If so, no, not easily
   so. Because it's the archiver process that does the deleting. And this
   process does not have access to a "full backend interface", e.g. the
   ability to run a query. We could make it look at the same data that's
   currently shown in pg_stat_replicatoin through shared memory, but thta
   would *only* work in the very most simple cases (e.g. a single
   pg_receivexlog and no other replication). The ability to run a custom
   SQL query is going to be necessary for anything a bit more advanced.
   
   
   > Also, as a small practical matter, since this is a server-side program
   > (since it's being used as archive_command), we shouldn't put it into the
   > pg_basebackup directory, because that would blur the lines about what to
   > install where, in particular for the translations.
   
   Good argument. That along with the being a hack, and the comment from
   Robert, means that maybe contrib/ is a better place for it, yes.
   
   --
    Magnus Hagander
    Me: http://www.hagander.net/
    Work: http://www.redpill-linpro.com/
   
   
   

10. Re: pg_retainxlog for inclusion in 9.3?

    Phil Sorber <phil@omniti.com> — 2013-01-05T18:34:02Z

    On Tue, Jan 1, 2013 at 10:10 AM, Magnus Hagander <magnus@hagander.net> wrote:
    > So, it turns out the reason I got no feedback on this tool, was that I
    > forgot both to email about and to actually push the code to github :O
    > So this is actually code that's almost half a year old and that I was
    > supposed to submit for the first or second commitfest. Oops.
    >
    > So, the tool and a README for it right now can be found at
    > https://github.com/mhagander/pg_retainxlog for the time being.
    >
    > The idea behind the tool is to plug a hole in the case when
    > pg_receivexlog is used for log archiving, which is that you still need
    > a "blocking" archive_command in order to make sure that files aren't
    > recycled on the master. So for 9.2 you can do this with an
    > archive_command that checks if the file has appeared properly on the
    > slave - but that usually means you're back at requiring ssh
    > connectivity between the machines, for example. Even though this
    > information is actually avialable on the master...
    >
    > This can also be of use to pure replication scenarios, where you don't
    > know how to tune wal_keep_segments, but using actual live feedback
    > instead of guessing.
    >
    > When pg_retainxlog is used as an archive_command, it will check the
    > pg_stat_replication view instead of checking the slave. It will then
    > only return ok once the requested logfile has been replicated to the
    > slave. By default it will look for a replication client named
    > pg_receivexlog, but it supports overriding the query with anything -
    > so you can say things like "needs to have arrived on at least two
    > replication slaves before we consider it archived". Or if used instead
    > of wal_keep_segmnets, needs to have arrived at *all* replication
    > slaves.
    >
    > Is this a tool that people would like to see included in the general
    > toolchain? If so, I'll reformat it to work in the general build
    > environment and submit it for the last commitfest.
    >
    > (comments on the code itself are of course also welcome)
    >
    > --
    >  Magnus Hagander
    >  Me: http://www.hagander.net/
    >  Work: http://www.redpill-linpro.com/
    >
    >
    > --
    > Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
    > To make changes to your subscription:
    > http://www.postgresql.org/mailpref/pgsql-hackers
    
    +1 to this concept, however it may be implemented.
    
    
    

11. Re: pg_retainxlog for inclusion in 9.3?

    Magnus Hagander <magnus@hagander.net> — 2013-01-14T10:03:19Z

    On Sat, Jan 5, 2013 at 3:11 PM, Magnus Hagander <magnus@hagander.net> wrote:
    > On Fri, Jan 4, 2013 at 7:13 PM, Peter Eisentraut <peter_e@gmx.net> wrote:
    >> On 1/3/13 12:30 PM, Robert Haas wrote:
    >>> On Thu, Jan 3, 2013 at 11:32 AM, Magnus Hagander <magnus@hagander.net> wrote:
    >>>> Any particular reason? It goes pretty tightly together with
    >>>> pg_receivexlog, which is why I'd prefer putting it alongside that one.
    >>>> But if you have a good argument against it, I can change my mind :)
    >>>
    >>> Mostly that it seems like a hack, and I suspect we may come up with a
    >>> better way to do this in the future.
    >>
    >> It does seem like a hack.  Couldn't this be implemented with a backend
    >> switch instead?
    >
    > It definitely is a bit of a hack.
    >
    > I assume by backend switch you mean guc, right? If so, no, not easily
    > so. Because it's the archiver process that does the deleting. And this
    > process does not have access to a "full backend interface", e.g. the
    > ability to run a query. We could make it look at the same data that's
    > currently shown in pg_stat_replicatoin through shared memory, but thta
    > would *only* work in the very most simple cases (e.g. a single
    > pg_receivexlog and no other replication). The ability to run a custom
    > SQL query is going to be necessary for anything a bit more advanced.
    >
    >
    >> Also, as a small practical matter, since this is a server-side program
    >> (since it's being used as archive_command), we shouldn't put it into the
    >> pg_basebackup directory, because that would blur the lines about what to
    >> install where, in particular for the translations.
    >
    > Good argument. That along with the being a hack, and the comment from
    > Robert, means that maybe contrib/ is a better place for it, yes.
    
    Here's a version for inclusion in /contrib.
    
    --
     Magnus Hagander
     Me: http://www.hagander.net/
     Work: http://www.redpill-linpro.com/
    

12. Re: pg_retainxlog for inclusion in 9.3?

    Robert Haas <robertmhaas@gmail.com> — 2013-01-14T16:56:19Z

    On Fri, Jan 4, 2013 at 4:55 PM, Dimitri Fontaine <dimitri@2ndquadrant.fr> wrote:
    > Robert Haas <robertmhaas@gmail.com> writes:
    >> Mostly that it seems like a hack, and I suspect we may come up with a
    >> better way to do this in the future.
    >
    > Do you have the specs of such better way? Would it be a problem to have
    > both pg_retainxlog and the new way?
    
    Well, I think in the long term we are likely to want the master to
    have some kind of ability to track the positions of its slaves, even
    when they are disconnected.  And, optionally, to retain the WAL that
    they need, again even when they are disconnected.  If such an ability
    materializes, this will be moot (even as I think that pg_standby is
    now largely moot, at least for new installations, now that we have
    standby_mode=on).
    
    -- 
    Robert Haas
    EnterpriseDB: http://www.enterprisedb.com
    The Enterprise PostgreSQL Company
    
    
    

13. Re: pg_retainxlog for inclusion in 9.3?

    Magnus Hagander <magnus@hagander.net> — 2013-01-15T07:51:36Z

    On Mon, Jan 14, 2013 at 5:56 PM, Robert Haas <robertmhaas@gmail.com> wrote:
    > On Fri, Jan 4, 2013 at 4:55 PM, Dimitri Fontaine <dimitri@2ndquadrant.fr> wrote:
    >> Robert Haas <robertmhaas@gmail.com> writes:
    >>> Mostly that it seems like a hack, and I suspect we may come up with a
    >>> better way to do this in the future.
    >>
    >> Do you have the specs of such better way? Would it be a problem to have
    >> both pg_retainxlog and the new way?
    >
    > Well, I think in the long term we are likely to want the master to
    > have some kind of ability to track the positions of its slaves, even
    > when they are disconnected.  And, optionally, to retain the WAL that
    > they need, again even when they are disconnected.  If such an ability
    > materializes, this will be moot (even as I think that pg_standby is
    > now largely moot, at least for new installations, now that we have
    > standby_mode=on).
    
    I agree. But just as we had pg_standby for quite a while before we got
    standby_mode=on, I believe we should have pg_retainxlog (or something
    like it) until we have something more integrated.
    
    --
     Magnus Hagander
     Me: http://www.hagander.net/
     Work: http://www.redpill-linpro.com/
    
    
    

14. Re: pg_retainxlog for inclusion in 9.3?

    Robert Haas <robertmhaas@gmail.com> — 2013-01-15T18:54:33Z

    On Tue, Jan 15, 2013 at 2:51 AM, Magnus Hagander <magnus@hagander.net> wrote:
    > On Mon, Jan 14, 2013 at 5:56 PM, Robert Haas <robertmhaas@gmail.com> wrote:
    >> On Fri, Jan 4, 2013 at 4:55 PM, Dimitri Fontaine <dimitri@2ndquadrant.fr> wrote:
    >>> Robert Haas <robertmhaas@gmail.com> writes:
    >>>> Mostly that it seems like a hack, and I suspect we may come up with a
    >>>> better way to do this in the future.
    >>>
    >>> Do you have the specs of such better way? Would it be a problem to have
    >>> both pg_retainxlog and the new way?
    >>
    >> Well, I think in the long term we are likely to want the master to
    >> have some kind of ability to track the positions of its slaves, even
    >> when they are disconnected.  And, optionally, to retain the WAL that
    >> they need, again even when they are disconnected.  If such an ability
    >> materializes, this will be moot (even as I think that pg_standby is
    >> now largely moot, at least for new installations, now that we have
    >> standby_mode=on).
    >
    > I agree. But just as we had pg_standby for quite a while before we got
    > standby_mode=on, I believe we should have pg_retainxlog (or something
    > like it) until we have something more integrated.
    
    Yep, not disagreeing.
    
    -- 
    Robert Haas
    EnterpriseDB: http://www.enterprisedb.com
    The Enterprise PostgreSQL Company
    
    
    

15. Re: pg_retainxlog for inclusion in 9.3?

    Peter Eisentraut <peter_e@gmx.net> — 2013-01-24T17:04:40Z

    After reviewing this, it appears to me that this is really just a very
    verbose version of
    
    archive_command = 'sleep $initialsleep; while test $(psql -AtX -c "select pg_xlogfile_name(something) < $$%f$$ collate \"C\";")	= t; sleep $sleep; done'
    
    I think it might be better to just document this as an example.  I don't
    quite see the overhead of maintaining another tool justified.
    
    
    
    

16. Re: pg_retainxlog for inclusion in 9.3?

    Magnus Hagander <magnus@hagander.net> — 2013-01-24T17:12:39Z

    On Thu, Jan 24, 2013 at 6:04 PM, Peter Eisentraut <peter_e@gmx.net> wrote:
    > After reviewing this, it appears to me that this is really just a very
    > verbose version of
    >
    > archive_command = 'sleep $initialsleep; while test $(psql -AtX -c "select pg_xlogfile_name(something) < $$%f$$ collate \"C\";") = t; sleep $sleep; done'
    >
    > I think it might be better to just document this as an example.  I don't
    > quite see the overhead of maintaining another tool justified.
    
    Well, obviously I don't entirely agree ;)
    
    Yes, it's a convenience command. Like pg_standby was. And like many
    other commands that we maintain as part of *core*, such as createuser,
    vacuumdb, etc. Those can all be done with an even *simpler* command
    than the one you suggest above. So I don't see that as an argument why
    it wouldn't be useful.
    
    Also, the command you suggest above does not work on Windows. You can
    probably write a .BAT file to do it for you, but I'm pretty sure it's
    impossible to do it as an archive_command there.
    
    --
     Magnus Hagander
     Me: http://www.hagander.net/
     Work: http://www.redpill-linpro.com/
    
    
    

17. Re: pg_retainxlog for inclusion in 9.3?

    Tom Lane <tgl@sss.pgh.pa.us> — 2013-01-24T17:25:18Z

    Magnus Hagander <magnus@hagander.net> writes:
    > On Thu, Jan 24, 2013 at 6:04 PM, Peter Eisentraut <peter_e@gmx.net> wrote:
    >> I think it might be better to just document this as an example.  I don't
    >> quite see the overhead of maintaining another tool justified.
    
    > Well, obviously I don't entirely agree ;)
    
    > Yes, it's a convenience command. Like pg_standby was. And like many
    > other commands that we maintain as part of *core*, such as createuser,
    > vacuumdb, etc. Those can all be done with an even *simpler* command
    > than the one you suggest above. So I don't see that as an argument why
    > it wouldn't be useful.
    
    We've discussed removing a lot of those tools, too.  Not breaking
    backwards compatibility is probably the only reason they're still there.
    
    In the case at hand, I seem to recall from upthread that we expect
    this'd be obsolete in a release or two.  If that's true then I think
    a para or two of documentation is a better idea than a tool we'll be
    essentially condemned to keep maintaining forever.
    
    > Also, the command you suggest above does not work on Windows. You can
    > probably write a .BAT file to do it for you, but I'm pretty sure it's
    > impossible to do it as an archive_command there.
    
    Perhaps we could whip up such a .BAT file and put it in the docs?
    
    			regards, tom lane
    
    
    

18. Re: pg_retainxlog for inclusion in 9.3?

    Magnus Hagander <magnus@hagander.net> — 2013-01-24T17:28:28Z

    On Thu, Jan 24, 2013 at 6:25 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
    > Magnus Hagander <magnus@hagander.net> writes:
    >> On Thu, Jan 24, 2013 at 6:04 PM, Peter Eisentraut <peter_e@gmx.net> wrote:
    >>> I think it might be better to just document this as an example.  I don't
    >>> quite see the overhead of maintaining another tool justified.
    >
    >> Well, obviously I don't entirely agree ;)
    >
    >> Yes, it's a convenience command. Like pg_standby was. And like many
    >> other commands that we maintain as part of *core*, such as createuser,
    >> vacuumdb, etc. Those can all be done with an even *simpler* command
    >> than the one you suggest above. So I don't see that as an argument why
    >> it wouldn't be useful.
    >
    > We've discussed removing a lot of those tools, too.  Not breaking
    > backwards compatibility is probably the only reason they're still there.
    >
    > In the case at hand, I seem to recall from upthread that we expect
    > this'd be obsolete in a release or two.  If that's true then I think
    > a para or two of documentation is a better idea than a tool we'll be
    > essentially condemned to keep maintaining forever.
    
    Not really sure there is such an expectation - any more than there was
    such an expectation when we initially put pg_standby in there. It
    would be *possible* to do it, certainly. But it's not like we have an
    actual plan. And AFAIK the stuff that was discussed upthread was a
    simplified version of it - not the full flexibility.
    
    That said, it's certainly a point that we'd have to maintain it. But I
    don't see why we'd have to maintain it beyond the point where we
    included the same functionality in core, if we did.
    
    
    >> Also, the command you suggest above does not work on Windows. You can
    >> probably write a .BAT file to do it for you, but I'm pretty sure it's
    >> impossible to do it as an archive_command there.
    >
    > Perhaps we could whip up such a .BAT file and put it in the docs?
    
    That would probably work, yes.
    
    
    --
     Magnus Hagander
     Me: http://www.hagander.net/
     Work: http://www.redpill-linpro.com/
    
    
    

19. Re: pg_retainxlog for inclusion in 9.3?

    Fujii Masao <masao.fujii@gmail.com> — 2013-01-25T17:21:00Z

    On Sat, Jan 5, 2013 at 11:11 PM, Magnus Hagander <magnus@hagander.net> wrote:
    > On Fri, Jan 4, 2013 at 7:13 PM, Peter Eisentraut <peter_e@gmx.net> wrote:
    >> On 1/3/13 12:30 PM, Robert Haas wrote:
    >>> On Thu, Jan 3, 2013 at 11:32 AM, Magnus Hagander <magnus@hagander.net> wrote:
    >>>> Any particular reason? It goes pretty tightly together with
    >>>> pg_receivexlog, which is why I'd prefer putting it alongside that one.
    >>>> But if you have a good argument against it, I can change my mind :)
    >>>
    >>> Mostly that it seems like a hack, and I suspect we may come up with a
    >>> better way to do this in the future.
    >>
    >> It does seem like a hack.  Couldn't this be implemented with a backend
    >> switch instead?
    >
    > It definitely is a bit of a hack.
    >
    > I assume by backend switch you mean guc, right? If so, no, not easily
    > so. Because it's the archiver process that does the deleting.
    
    The process which deletes the old WAL files is the checkpointer. The
    checkpointer can access to the shared memory and know the location
    of the WAL record which has been already replicated to the standby.
    ISTM it's not difficult to implement the logic which pg_retainxlog provides
    into the checkpointer. How about just changing the checkpointer so
    that it checks whether the WAL file to delete has been already not
    only archived but also replicated if GUC flag is enabled?
    
    Regards,
    
    -- 
    Fujii Masao
    
    
    

20. Re: pg_retainxlog for inclusion in 9.3?

    Andres Freund <andres@2ndquadrant.com> — 2013-01-25T17:24:19Z

    On 2013-01-26 02:21:00 +0900, Fujii Masao wrote:
    > On Sat, Jan 5, 2013 at 11:11 PM, Magnus Hagander <magnus@hagander.net> wrote:
    > > On Fri, Jan 4, 2013 at 7:13 PM, Peter Eisentraut <peter_e@gmx.net> wrote:
    > >> On 1/3/13 12:30 PM, Robert Haas wrote:
    > >>> On Thu, Jan 3, 2013 at 11:32 AM, Magnus Hagander <magnus@hagander.net> wrote:
    > >>>> Any particular reason? It goes pretty tightly together with
    > >>>> pg_receivexlog, which is why I'd prefer putting it alongside that one.
    > >>>> But if you have a good argument against it, I can change my mind :)
    > >>>
    > >>> Mostly that it seems like a hack, and I suspect we may come up with a
    > >>> better way to do this in the future.
    > >>
    > >> It does seem like a hack.  Couldn't this be implemented with a backend
    > >> switch instead?
    > >
    > > It definitely is a bit of a hack.
    > >
    > > I assume by backend switch you mean guc, right? If so, no, not easily
    > > so. Because it's the archiver process that does the deleting.
    >
    > The process which deletes the old WAL files is the checkpointer. The
    > checkpointer can access to the shared memory and know the location
    > of the WAL record which has been already replicated to the standby.
    > ISTM it's not difficult to implement the logic which pg_retainxlog provides
    > into the checkpointer. How about just changing the checkpointer so
    > that it checks whether the WAL file to delete has been already not
    > only archived but also replicated if GUC flag is enabled?
    
    The problem with that is that to implement it robustly we would need
    persistent state about the replicas.
    
    Greetings,
    
    Andres Freund
    
    --
     Andres Freund	                   http://www.2ndQuadrant.com/
     PostgreSQL Development, 24x7 Support, Training & Services
    
    
    

21. Re: pg_retainxlog for inclusion in 9.3?

    Peter Eisentraut <peter_e@gmx.net> — 2013-01-25T21:23:21Z

    On 1/25/13 12:24 PM, Andres Freund wrote:
    > On 2013-01-26 02:21:00 +0900, Fujii Masao wrote:
    >> The process which deletes the old WAL files is the checkpointer. The
    >> checkpointer can access to the shared memory and know the location
    >> of the WAL record which has been already replicated to the standby.
    >> ISTM it's not difficult to implement the logic which pg_retainxlog provides
    >> into the checkpointer. How about just changing the checkpointer so
    >> that it checks whether the WAL file to delete has been already not
    >> only archived but also replicated if GUC flag is enabled?
    
    That makes sense.
    
    > The problem with that is that to implement it robustly we would need
    > persistent state about the replicas.
    
    Well, pg_retainxlog kind of handwaves around that.  If you use it in the
    default mode, it assumes that the pg_receivexlog entries in
    pg_stat_replication are that state.  And then it says, if you use other
    kinds of clients, you need to keep track of that state yourself.  But
    that seems to assume that pg_receivexlog never disconnects (thus losing
    its entries from pg_stat_replication).  (pg_receivexlog is designed to
    automatically reconnect on connection loss, so this possibility was
    obviously thought about.)
    
    So it seems to me this just doesn't work (this way).