Thread

1. Big 7.4 items

   Bruce Momjian <pgman@candle.pha.pa.us> — 2002-12-13T06:22:27Z

   I wanted to outline some of the big items we are looking at for 7.4:
   
   Win32 Port:
   
   	Katie Ward and Jan are working on contributing their Win32
   	port for 7.4.  They plan to have a patch available by the end of
   	December.
   
   Point-In-Time Recovery (PITR)
   
   	J. R. Nield did a PITR patch late in 7.3 development, and Patrick
   	MacDonald from Red Hat is working on merging it into CVS and
   	adding any missing pieces.  Patrick, do you have an ETA on that?
   
   Replication
   
   	I have talked to Darren Johnson and I believe 7.4 is the time to
   	merge the Postgres-R source tree into our main CVS.  Most of the
   	replication code will be in its own directory, with only minor
   	changes to our existing tree.  They have single-master
   	replication working now, so we may have that feature in some
   	capacity for 7.4.  I know others are working on replication
   	solutions.  This is probably the time to decide for certain if
   	this is the direction we want to go for replication.  Most who
   	have have studied Postgres-R feel it is the most promising
   	multi-master replication solution for reliably networked hosts.
   
   Comments?
   
   -- 
     Bruce Momjian                        |  http://candle.pha.pa.us
     pgman@candle.pha.pa.us               |  (610) 359-1001
     +  If your life is a hard drive,     |  13 Roberts Road
     +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073
   
   

2. Re: Big 7.4 items

   Shridhar Daithankar <shridhar_daithankar@persistent.co.in> — 2002-12-13T07:05:05Z

   On 13 Dec 2002 at 1:22, Bruce Momjian wrote:
   > Replication
   > 
   > 	I have talked to Darren Johnson and I believe 7.4 is the time to
   > 	merge the Postgres-R source tree into our main CVS.  Most of the
   > 	replication code will be in its own directory, with only minor
   > 	changes to our existing tree.  They have single-master
   > 	replication working now, so we may have that feature in some
   > 	capacity for 7.4.  I know others are working on replication
   > 	solutions.  This is probably the time to decide for certain if
   > 	this is the direction we want to go for replication.  Most who
   > 	have have studied Postgres-R feel it is the most promising
   > 	multi-master replication solution for reliably networked hosts.
   > 
   > Comments?
   
   Some.
   
   1) What kind of replication are we looking at? log file replay/syncnronous etc. 
   If it is real time, like usogres( I hope I am in line with things here), that 
   would be real good .Choice is always good..
   
   2 If we are going to have replication, can we have built in load balancing? Is 
   it a good idea to have it in postgresql or a separate application would be way 
   to go?
   
   And where are nested transactions?
   
   
   
   Bye
    Shridhar
   
   --
   Booker's Law:	An ounce of application is worth a ton of abstraction.
   
   
   

3. Re: Big 7.4 items

   Hannu Krosing <hannu@tm.ee> — 2002-12-13T10:53:14Z

   On Fri, 2002-12-13 at 06:22, Bruce Momjian wrote:
   > I wanted to outline some of the big items we are looking at for 7.4:
   > Point-In-Time Recovery (PITR)
   > 
   > 	J. R. Nield did a PITR patch late in 7.3 development, and Patrick
   > 	MacDonald from Red Hat is working on merging it into CVS and
   > 	adding any missing pieces.  Patrick, do you have an ETA on that?
   
   How hard would it be to extend PITR for master-slave (hot backup)
   repliaction, which should then amount to continuously shipping logs to
   slave and doing nonstop PITR there :)
   
   It will never be usable for multi-master replication, but somehow it
   feels that for master-slave replication simple log replay would be most
   simple and robust solution.
   
   -- 
   Hannu Krosing <hannu@tm.ee>
   
   

4. Re: Big 7.4 items

   Mike Mascari <mascarm@mascari.com> — 2002-12-13T12:55:25Z

   Bruce Momjian wrote:
   > I wanted to outline some of the big items we are looking at for 7.4:
   > 
   > Win32 Port:
   > 
   > 	Katie Ward and Jan are working on contributing their Win32
   > 	port for 7.4.  They plan to have a patch available by the end of
   > 	December.
   > 
   > Point-In-Time Recovery (PITR)
   > 
   > 	J. R. Nield did a PITR patch late in 7.3 development, and Patrick
   > 	MacDonald from Red Hat is working on merging it into CVS and
   > 	adding any missing pieces.  Patrick, do you have an ETA on that?
   > 
   > Replication
   > 
   > 	I have talked to Darren Johnson and I believe 7.4 is the time to
   > 	merge the Postgres-R source tree into our main CVS.  Most of the
   > 	replication code will be in its own directory, with only minor
   > 	changes to our existing tree.  They have single-master
   > 	replication working now, so we may have that feature in some
   > 	capacity for 7.4.  I know others are working on replication
   > 	solutions.  This is probably the time to decide for certain if
   > 	this is the direction we want to go for replication.  Most who
   > 	have have studied Postgres-R feel it is the most promising
   > 	multi-master replication solution for reliably networked hosts.
   > 
   > Comments?
   
   What about distributed TX support:
   
   http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&oe=UTF-8&threadm=20021106111554.69ae1dcd.pgsql%40snaga.org&rnum=2&prev=/groups%3Fq%3DNAGAYASU%2BSatoshi%26ie%3DUTF-8%26oe%3DUTF-8%26hl%3Den
   
   Mike Mascari
   mascarm@mascari.com
   
   
   

5. Re: Big 7.4 items

   Joe Conway <mail@joeconway.com> — 2002-12-13T15:08:54Z

   Bruce Momjian wrote:
   > Win32 Port:
   > 
   > 	Katie Ward and Jan are working on contributing their Win32
   > 	port for 7.4.  They plan to have a patch available by the end of
   > 	December.
   
   I have .Net Studio available to me, so if you need help in merging or testing 
   or whatever, let me know.
   
   > Point-In-Time Recovery (PITR)
   > 
   > 	J. R. Nield did a PITR patch late in 7.3 development, and Patrick
   > 	MacDonald from Red Hat is working on merging it into CVS and
   > 	adding any missing pieces.  Patrick, do you have an ETA on that?
   
   As Hannu asked (and related to your question below), is there any thought of 
   extending this to allow simple log based replication? In many important 
   scenarios that would be more than adequate, and simpler to set up.
   
   > Replication
   > 
   > 	I have talked to Darren Johnson and I believe 7.4 is the time to
   > 	merge the Postgres-R source tree into our main CVS.  Most of the
   > 	replication code will be in its own directory, with only minor
   > 	changes to our existing tree.  They have single-master
   > 	replication working now, so we may have that feature in some
   > 	capacity for 7.4.  I know others are working on replication
   > 	solutions.  This is probably the time to decide for certain if
   > 	this is the direction we want to go for replication.  Most who
   > 	have have studied Postgres-R feel it is the most promising
   > 	multi-master replication solution for reliably networked hosts.
   
   I'd question if we would want the one-and-only builtin replication method to 
   be dependent on an external communication library (Spread). I would like to 
   see Postgres-R merged, but I'd also like to see a simple log-based option.
   
   > Comments?
   > 
   
   I'd also second Mike Mascari's question -- whatever happened to the person 
   working on two-phase commit? Is that likely to be done for 7.4? Did he ever 
   send in a patch?
   
   Joe
   
   
   

6. Re: Big 7.4 items

   Bruce Momjian <pgman@candle.pha.pa.us> — 2002-12-13T16:51:12Z

   Shridhar Daithankar wrote:
   > And where are nested transactions?
   
   I didn't mention nested transactions because it didn't seem to be a
   _big_ item like the others.
   
   -- 
     Bruce Momjian                        |  http://candle.pha.pa.us
     pgman@candle.pha.pa.us               |  (610) 359-1001
     +  If your life is a hard drive,     |  13 Roberts Road
     +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073
   
   

7. Re: Big 7.4 items

   Haris Peco <snpe@snpe.co.yu> — 2002-12-13T17:05:25Z

   On Friday 13 December 2002 17:51, Bruce Momjian wrote:
   > Shridhar Daithankar wrote:
   > > And where are nested transactions?
   >
   > I didn't mention nested transactions because it didn't seem to be a
   > _big_ item like the others.
   
   This is big item
   
   regards
   Haris Peco
   
   

8. Re: Big 7.4 items

   Bruce Momjian <pgman@candle.pha.pa.us> — 2002-12-13T17:31:40Z

   Shridhar Daithankar wrote:
   > 1) What kind of replication are we looking at? log file
   > replay/synchronous etc.  If it is real time, like usogres( I
   > hope I am in line with things here), that would be real good.
   > Choice is always good.
   
   Good.  This is the discussion we need.  Let me quote the TODO list
   replication section first:
   
       * Add replication of distributed databases [replication]
           o automatic failover
           o load balancing
           o master/slave replication
           o multi-master replication
           o partition data across servers
           o sample implementation in contrib/rserv
           o queries across databases or servers (two-phase commit)
           o allow replication over unreliable or non-persistent links
           o http://gborg.postgresql.org/project/pgreplication/projdisplay.php
   
   OK, the first thing is that there isn't any one replication solution
   that will behave optimally in all situations.  
   
   Now, let me describe Postgres-R and then the other replication
   solutions.  Postgres-R is multi-master, meaning you can send SELECT and
   UPDATE/DELETE queries to any of the servers in the cluster, and get the
   same result.  It is also synchronous, meaning it doesn't update the
   local copy until it is sure the other nodes agree to the change. It
   allows failover, because if one node goes down, the others keep going.
   
   Now, let me contrast:
   
   rserv and dbmirror do master/slave.  There is no mechanism to allow you
   to do updates on the slave, and have them propagate to the master.  You
   can, however, send SELECT queries to the slave, and in fact that's how
   usogres does load balancing.
   
   Two-phase commit is probably the most popular commercial replication
   solution.  While it works for multi-master, it suffers from poor
   performance and doesn't handle cases where one node disappears very
   well.
   
   Another replication need is for asynchronous replication, most
   traditionally for traveling salesmen who need to update their databases
   periodically.  The only solution I know for that is PeerDirect's
   PostgreSQL commercial offering at http://www.peerdirect.com.  It is
   possible PITR may help with this, but we need to handle propagating
   changes made by the salesmen back up into the server, and to do that, we
   will need a mechanism to handle conflicts that occur when two people
   update the same records.  This is always a problem for asynchronous
   replication.
   
   > 2 If we are going to have replication, can we have built in load
   > balancing? Is it a good idea to have it in postgresql or a
   > separate application would be way to go?
   
   Well, because Postgres-R is multi-master, it has automatic load
   balancing.  You can have your users point to whatever node you want. 
   You can implement this "pointing" by using dns IP address cycling, or
   have a router that auto-load balances, though you would need to keep a
   db session on the same node, of course.
   
   So, in summary, I think we will eventually have two directions for
   replication.  One is Postgres-R for multi-master, synchronous
   replication, and PITR, for asynchronous replication.  I don't think
   there is any value to use PITR for synchronous replication because by
   definition, you don't _store_ the changes for later use because it is
   synchronous.  In synchronous, you communicate your changes to all the
   nodes involved, then commit them.
   
   I will describe the use of 'spread' and the Postgres-R internal issues
   in my next email.
   
   --
     Bruce Momjian                        |  http://candle.pha.pa.us
     pgman@candle.pha.pa.us               |  (610) 359-1001
     +  If your life is a hard drive,     |  13 Roberts Road
     +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073
   
   

9. Re: Big 7.4 items

   Bruce Momjian <pgman@candle.pha.pa.us> — 2002-12-13T17:32:52Z

   Hannu Krosing wrote:
   > On Fri, 2002-12-13 at 06:22, Bruce Momjian wrote:
   > > I wanted to outline some of the big items we are looking at for 7.4:
   > > Point-In-Time Recovery (PITR)
   > > 
   > > 	J. R. Nield did a PITR patch late in 7.3 development, and Patrick
   > > 	MacDonald from Red Hat is working on merging it into CVS and
   > > 	adding any missing pieces.  Patrick, do you have an ETA on that?
   > 
   > How hard would it be to extend PITR for master-slave (hot backup)
   > repliaction, which should then amount to continuously shipping logs to
   > slave and doing nonstop PITR there :)
   > 
   > It will never be usable for multi-master replication, but somehow it
   > feels that for master-slave replication simple log replay would be most
   > simple and robust solution.
   
   Exactly.  See my previous email.  We eventually have two replication
   solutions:  one, Postgres-R for multi-master, and PITR used for for
   asynchonous master/slave.
   
   -- 
     Bruce Momjian                        |  http://candle.pha.pa.us
     pgman@candle.pha.pa.us               |  (610) 359-1001
     +  If your life is a hard drive,     |  13 Roberts Road
     +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073
   
   

10. Re: Big 7.4 items

    Bruce Momjian <pgman@candle.pha.pa.us> — 2002-12-13T17:34:37Z

    Mike Mascari wrote:
    > What about distributed TX support:
    > 
    > http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&oe=UTF-8&threadm=20021106111554.69ae1dcd.pgsql%40snaga.org&rnum=2&prev=/groups%3Fq%3DNAGAYASU%2BSatoshi%26ie%3DUTF-8%26oe%3DUTF-8%26hl%3Den
    
    OK, yes, that is Satoshi's 2-phase commit implementation.  I will
    address 2-phase commit vs Postgres-R in my next email about spread.
    
    
    -- 
      Bruce Momjian                        |  http://candle.pha.pa.us
      pgman@candle.pha.pa.us               |  (610) 359-1001
      +  If your life is a hard drive,     |  13 Roberts Road
      +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073
    
    

11. Re: Big 7.4 items

    Mike Mascari <mascarm@mascari.com> — 2002-12-13T17:57:25Z

    Okay. But please keep in mind that a 2-phase commit implementation is used for more than just replication. Any distributed TX will require a 2PC protocol. As an example, for the DBLINK implementation to ultimately be transaction safe (at least amongst multiple PostgreSQL installations), the players in the distributed transaction must all be participants in a 2PC exchange. And a participant whose communications link is dropped needs to be able to recover by asking the coordinator whether or not to complete or abort the distributed TX. I am 100% ignorant of the distributed TX standard Tom referenced earlier, but I'd guess there might be an assumption of 2PC support in the implementation. In other words, I think we still need 2PC, regardless of the method of replication. And if  Satoshi Nagayasu has an implementation ready, why not investigate its possibilities?
    
    Mike Mascari
    mascarm@mascari.com
    
    ----- Original Message ----- 
    From: "Bruce Momjian" <pgman@candle.pha.pa.us>
    
    
    > Mike Mascari wrote:
    > > What about distributed TX support:
    
    > OK, yes, that is Satoshi's 2-phase commit implementation.  I will
    > address 2-phase commit vs Postgres-R in my next email about spread.
    
    
    
    

12. Re: Big 7.4 items

    Bruce Momjian <pgman@candle.pha.pa.us> — 2002-12-13T18:20:09Z

    Joe Conway wrote:
    > Bruce Momjian wrote:
    > > Win32 Port:
    > > 
    > > 	Katie Ward and Jan are working on contributing their Win32
    > > 	port for 7.4.  They plan to have a patch available by the end of
    > > 	December.
    > 
    > I have .Net Studio available to me, so if you need help in merging or testing 
    > or whatever, let me know.
    
    OK, Jan, let him know how he can help.
    
    > > Point-In-Time Recovery (PITR)
    > > 
    > > 	J. R. Nield did a PITR patch late in 7.3 development, and Patrick
    > > 	MacDonald from Red Hat is working on merging it into CVS and
    > > 	adding any missing pieces.  Patrick, do you have an ETA on that?
    > 
    > As Hannu asked (and related to your question below), is there any thought of 
    > extending this to allow simple log based replication? In many important 
    > scenarios that would be more than adequate, and simpler to set up.
    
    Yes, see previous email.
    
    > I'd question if we would want the one-and-only builtin replication method to 
    > be dependent on an external communication library (Spread). I would like to 
    > see Postgres-R merged, but I'd also like to see a simple log-based option.
    
    OK, let me reiterate I think we will have two replication solutions in
    the end --- on Postgres-R for multi-master/synchronous, and PITR for
    master/slave asynchronous replication.
    
    Let me address the Spread issue and two-phase commit.  (Spread is an
    open source piece of software used by Postgres-R.)
    
    In two-phase commit, when one node is about to commit, it gets a lock
    from all the other nodes, does its commit, then releases the lock.  (Of
    course, this is simplified.)  It is called two-phase because it says to
    all the other nodes "I am about to do something, is that OK?", then when
    gets all OK's, it does the commit and says "I did the commit".
    
    Postgres-R uses a different mechanism.  This method is shown on page 22
    and 24 and following of:
    
    	ftp://gborg.postgresql.org/pub/pgreplication/stable/PostgreSQLReplication.pdf.gz
    
    The basic difference is that Spread groups all the write sets into a
    queue who's ordering is the same on all the nodes.  Instead of asking
    for approval for a commit, a node puts its commit in the Spread queue,
    and then waits for its own commit to come back in the queue, meaning all
    the other nodes saw its commit too.
    
    The only tricky part is that while reading the other node's write sets
    before its own arrives, it has to check to see if any of these conflict
    with its own write set. If it conflicts, it has to assume the earlier
    write set succeeded and its own failed.  It also has to check the write
    set stream and apply only those changes that don't conflict.
    
    As stated before in Postgres-R discussion, this mechanism hinges on
    being able to determine which write sets conflict because there is no
    explicit "I aborted", only a stream of write sets, and each node has to
    accept the non-conflicting ones and reject the conflicting ones.
    
    > I'd also second Mike Mascari's question -- whatever happened to the person 
    > working on two-phase commit? Is that likely to be done for 7.4? Did he ever 
    > send in a patch?
    
    I have not seen a patch from him, but it is very likely he could have
    one for 7.4.  This is why it is good we discuss this now and figure out
    where we want to go for 7.4 so we can get started.
    
    -- 
      Bruce Momjian                        |  http://candle.pha.pa.us
      pgman@candle.pha.pa.us               |  (610) 359-1001
      +  If your life is a hard drive,     |  13 Roberts Road
      +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073
    
    

13. Re: Big 7.4 items

    Bruce Momjian <pgman@candle.pha.pa.us> — 2002-12-13T18:22:53Z

    Mike Mascari wrote:
    > Okay. But please keep in mind that a 2-phase commit implementation
    > is used for more than just replication. Any distributed TX will
    > require a 2PC protocol. As an example, for the DBLINK implementation
    > to ultimately be transaction safe (at least amongst multiple
    > PostgreSQL installations), the players in the distributed
    > transaction must all be participants in a 2PC exchange. And a
    > participant whose communications link is dropped needs to be
    > able to recover by asking the coordinator whether or not to
    > complete or abort the distributed TX. I am 100% ignorant of the
    > distributed TX standard Tom referenced earlier, but I'd guess
    > there might be an assumption of 2PC support in the implementation.
    > In other words, I think we still need 2PC, regardless of the
    > method of replication. And if  Satoshi Nagayasu has an implementation
    > ready, why not investigate its possibilities?
    
    This is a good point.  I don't want to push Postgres-R as our solution. 
    Rather, I have looked at both and like Postgres-R, but others need to
    look at both and decide so we are all in agreement when we move forward.
    
    --
      Bruce Momjian                        |  http://candle.pha.pa.us
      pgman@candle.pha.pa.us               |  (610) 359-1001
      +  If your life is a hard drive,     |  13 Roberts Road
      +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073
    
    

14. Re: Big 7.4 items

    Jan Wieck <janwieck@yahoo.com> — 2002-12-13T18:36:59Z

    Bruce Momjian wrote:
    
    > OK, the first thing is that there isn't any one replication solution
    > that will behave optimally in all situations.
    
    Right
    
    > Now, let me describe Postgres-R and then the other replication
    > solutions.  Postgres-R is multi-master, meaning you can send SELECT and
    > UPDATE/DELETE queries to any of the servers in the cluster, and get the
    > same result.  It is also synchronous, meaning it doesn't update the
    > local copy until it is sure the other nodes agree to the change. It
    > allows failover, because if one node goes down, the others keep going.
    
    Wrong
    
    It is asynchronous without the need of 2 phase commit. It is group
    communication based and requires the group communication system to
    guarantee total order. The tricky part is, that the local transaction
    must be on hold until the own commit message comes back without a prior
    lock conflict by a replication transaction. If such a lock confict
    occurs, the replication transaction wins and the local transaction rolls
    back.
    
    > 
    > Now, let me contrast:
    > 
    > rserv and dbmirror do master/slave.  There is no mechanism to allow you
    > to do updates on the slave, and have them propagate to the master.  You
    > can, however, send SELECT queries to the slave, and in fact that's how
    > usogres does load balancing.
    
    But you cannot use the result of such a SELECT to update anything. So
    you can only phase out complete read only transaction to the slaves.
    Requires support from the application since the load balancing system
    cannot know automatically what will be a read only transaction and what
    not.
    
    > 
    > Two-phase commit is probably the most popular commercial replication
    > solution.  While it works for multi-master, it suffers from poor
    > performance and doesn't handle cases where one node disappears very
    > well.
    > 
    > Another replication need is for asynchronous replication, most
    > traditionally for traveling salesmen who need to update their databases
    > periodically.  The only solution I know for that is PeerDirect's
    > PostgreSQL commercial offering at http://www.peerdirect.com.  It is
    > possible PITR may help with this, but we need to handle propagating
    > changes made by the salesmen back up into the server, and to do that, we
    > will need a mechanism to handle conflicts that occur when two people
    > update the same records.  This is always a problem for asynchronous
    > replication.
    
    PITR doesn't help here at all, since PeerDirect's replication is trigger
    and control table based. What makes our replication system unique is
    that it works bidirectional in a heterogenious world.
    
    > I will describe the use of 'spread' and the Postgres-R internal issues
    > in my next email.
    
    The last time i was playing with spread (that was at Great Bridge in
    Norfolk), it was IMHO useless (for Postgres-R) because it sometimes
    dropped messages when the network load got too high. This occured
    without any indication, no error, nothing. This is not exactly what I
    understand as total order. I hope they have made some substantial
    progress on that.
    
    
    Jan
    
    -- 
    #======================================================================#
    # It's easier to get forgiveness for being wrong than for being right. #
    # Let's break this rule - forgive me.                                  #
    #================================================== JanWieck@Yahoo.com #
    
    

15. Re: Big 7.4 items

    Mike Mascari <mascarm@mascari.com> — 2002-12-13T18:51:17Z

    ----- Original Message ----- 
    From: "Bruce Momjian" <pgman@candle.pha.pa.us>
    
    > Mike Mascari wrote:
    > > Okay. But please keep in mind that a 2-phase commit implementation
    > > is used for more than just replication. 
    > 
    > This is a good point.  I don't want to push Postgres-R as our solution. 
    > Rather, I have looked at both and like Postgres-R, but others need to
    > look at both and decide so we are all in agreement when we move forward.
    > 
    
    After having read your post regarding Spread, I see that it is an alternative to 2PC as a distributed TX protocol. If I understand you correctly, a DBLINK implementation built atop Spread would also be possible. Correct? The question then is, do other RDBMS expose a 2PC implementation which could not then be leveraged at a later time? For example imagine:
    
    1. 7.4 includes a native 2PC protocol with:
    
    CREATE DATABASE LINK accounting
    CONNECT TO accounting.acme.com:5432
    IDENTIFIED BY mascarm/mascarm;
    
    SELECT *
    FROM employees@accounting;
    
    INSERT INTO employees@accounting
    VALUES (1, 'Mike', 'Mascari');
    
    That would be great, allowing PostgreSQL servers running in different departments to participate in a distributed tx. 
    
    2. 7.5 includes a DBLINK which supports PostgreSQL participating in a heterogenous distributed transaction (with say, an Oracle database):
    
    CREATE DATABASE LINK finance
    CONNECT TO <oracle names entry>
    IDENTIFIED BY mascarm/mascarm
    USING INTERFACE 'pg2oracle.so';
    
    INSERT INTO employees@finance
    VALUES (1, 'Mike', 'Mascari');
    
    I guess I'm basically asking:
    
    1) Is it necessary to *choose* between support for 2PC and Spread (Postgres-R) or can't we have both? Spread for Replication, 2PC for non-replicating distributed TX?
    
    2) Do major SQL DBMS vendors which support distributed options expose a callable interface into a 2PC protocol that would allow PostgreSQL to participate? I could check on this...
    
    3) Are there any standards (besides ODBC, which, the last time I looked just had COMMIT/ABORT APIs), that have been defined and adopted by the industry for distributed tx?
    
    Again, I'd guess most people want:
    
    1) High performance Master/Master replication *and* (r.e. Postgres-R)
    2) Ability to participate in distrubuted tx's (r.e. 2PC?)
    
    Mike Mascari
    mascarm@mascari.com
    
    
    
    
    

16. Re: Big 7.4 items

    Neil Conway <neilc@samurai.com> — 2002-12-13T19:01:30Z

    On Fri, 2002-12-13 at 13:20, Bruce Momjian wrote:
    > Let me address the Spread issue and two-phase commit.  (Spread is an
    > open source piece of software used by Postgres-R.)
    
    Note that while Spread is open source in the sense that "the source is
    available", it's license is significantly more restrictive than
    PostgreSQL's:
    
        http://www.spread.org/license/
    
    Just FYI...
    
    Cheers,
    
    Neil
    -- 
    Neil Conway <neilc@samurai.com> || PGP Key ID: DB3C29FC
    
    
    
    
    

17. Re: Big 7.4 items

    Bruce Momjian <pgman@candle.pha.pa.us> — 2002-12-13T19:09:16Z

    Neil Conway wrote:
    > On Fri, 2002-12-13 at 13:20, Bruce Momjian wrote:
    > > Let me address the Spread issue and two-phase commit.  (Spread is an
    > > open source piece of software used by Postgres-R.)
    > 
    > Note that while Spread is open source in the sense that "the source is
    > available", it's license is significantly more restrictive than
    > PostgreSQL's:
    > 
    >     http://www.spread.org/license/
    > 
    
    Interesting.  It looks like a modified version of the old BSD license
    where you are required to mention you are using Spread.  I believe we
    can get that reduced.  (I think Darren already addressed this with
    them.) We certainly are not going to accept software that requires all
    PostgreSQL user sites to mention Spread.
    
    The whole "mention" aspect of the old BSD license was pretty ambiguous,
    and I assume this is similar.
    
    -- 
      Bruce Momjian                        |  http://candle.pha.pa.us
      pgman@candle.pha.pa.us               |  (610) 359-1001
      +  If your life is a hard drive,     |  13 Roberts Road
      +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073
    
    

18. Re: Big 7.4 items

    Mike Mascari <mascarm@mascari.com> — 2002-12-13T19:34:54Z

    I wrote:
    > 
    > I guess I'm basically asking:
    > 
    > 1) Is it necessary to *choose* between support for 2PC and Spread (Postgres-R) or can't we have both? Spread for Replication, 2PC for non-replicating distributed TX?
    > 
    > 2) Do major SQL DBMS vendors which support distributed options expose a callable interface into a 2PC protocol that would allow PostgreSQL to participate? I could check on this...
    >
    > 3) Are there any standards (besides ODBC, which, the last time I looked just had COMMIT/ABORT APIs), that have been defined and adopted by the industry for distributed tx?
    
    Answer:
    
    The Open Group's Open/XA C193 specificiation for API for distributed transactions:
    
    http://www.opengroup.org/public/pubs/catalog/c193.htm
    
    I couldn't find any draft copies on the web, but a good description at the Sybase site:
    
    http://manuals.sybase.com/onlinebooks/group-xs/xsg1111e/xatuxedo/@ebt-link;pt=61?target=%25N%13_446_START_RESTART_N%25
    
    The standard is 2PC based.
    
    Mike Mascari
    mascarm@mascari.com
    
    
    
    

19. Re: Big 7.4 items

    Jan Wieck <janwieck@yahoo.com> — 2002-12-13T19:45:43Z

    Bruce Momjian wrote:
    > 
    > Joe Conway wrote:
    > > Bruce Momjian wrote:
    > > > Win32 Port:
    > > >
    > > >     Katie Ward and Jan are working on contributing their Win32
    > > >     port for 7.4.  They plan to have a patch available by the end of
    > > >     December.
    > >
    > > I have .Net Studio available to me, so if you need help in merging or testing
    > > or whatever, let me know.
    > 
    > OK, Jan, let him know how he can help.
    
    My current plan is to comb out the Win32 port only from what we've done
    all together against 7.2.1. The result should be a clean patch that
    applied against 7.2.1 builds a native windows port.
    
    From there, this patch must be lifted up to 7.4.
    
    I have the original context diff now down from 160,000 lines to 80,000
    lines. I think I will have the clean diff against 7.2.1 somewhere next
    week. That would IMHO be a good time for Tom to start complaining so
    that we can work in the required changes during the 7.4 lifting. ;-)
    
    
    Jan
    
    -- 
    #======================================================================#
    # It's easier to get forgiveness for being wrong than for being right. #
    # Let's break this rule - forgive me.                                  #
    #================================================== JanWieck@Yahoo.com #
    
    

20. Re: Big 7.4 items

    Bruce Momjian <pgman@candle.pha.pa.us> — 2002-12-13T21:09:44Z

    Jan Wieck wrote:
    > Bruce Momjian wrote:
    > 
    > > OK, the first thing is that there isn't any one replication solution
    > > that will behave optimally in all situations.
    > 
    > Right
    > 
    > > Now, let me describe Postgres-R and then the other replication
    > > solutions.  Postgres-R is multi-master, meaning you can send SELECT and
    > > UPDATE/DELETE queries to any of the servers in the cluster, and get the
    > > same result.  It is also synchronous, meaning it doesn't update the
    > > local copy until it is sure the other nodes agree to the change. It
    > > allows failover, because if one node goes down, the others keep going.
    > 
    > Wrong
    > 
    > It is asynchronous without the need of 2 phase commit. It is group
    
    Well, Darren's PDF at:
    
    	ftp://gborg.postgresql.org/pub/pgreplication/stable/PostgreSQLReplication.pdf.gz
    
    calls Postgres-R "Type: Embedded, Peer-to-Peer, Sync".  I don't know
    enough about replication so I will let you fight it out with him.  ;-)
    
    > communication based and requires the group communication system to
    > guarantee total order. The tricky part is, that the local transaction
    > must be on hold until the own commit message comes back without a prior
    > lock conflict by a replication transaction. If such a lock confict
    > occurs, the replication transaction wins and the local transaction rolls
    > back.
    
    Yep, that's the tricky part.
    
    > > 
    > > Now, let me contrast:
    > > 
    > > rserv and dbmirror do master/slave.  There is no mechanism to allow you
    > > to do updates on the slave, and have them propagate to the master.  You
    > > can, however, send SELECT queries to the slave, and in fact that's how
    > > usogres does load balancing.
    > 
    > But you cannot use the result of such a SELECT to update anything. So
    > you can only phase out complete read only transaction to the slaves.
    > Requires support from the application since the load balancing system
    > cannot know automatically what will be a read only transaction and what
    > not.
    
    Good point.  It has to be a read-only session because you can't jump
    nodes during a session.  That definately limits its usefulness.
    
    > > Two-phase commit is probably the most popular commercial replication
    > > solution.  While it works for multi-master, it suffers from poor
    > > performance and doesn't handle cases where one node disappears very
    > > well.
    > > 
    > > Another replication need is for asynchronous replication, most
    > > traditionally for traveling salesmen who need to update their databases
    > > periodically.  The only solution I know for that is PeerDirect's
    > > PostgreSQL commercial offering at http://www.peerdirect.com.  It is
    > > possible PITR may help with this, but we need to handle propagating
    > > changes made by the salesmen back up into the server, and to do that, we
    > > will need a mechanism to handle conflicts that occur when two people
    > > update the same records.  This is always a problem for asynchronous
    > > replication.
    > 
    > PITR doesn't help here at all, since PeerDirect's replication is trigger
    > and control table based. What makes our replication system unique is
    > that it works bidirectional in a heterogenious world.
    
    I was only suggesting that PITR _may_ help as an archive method for the
    changes.  PeerDirect stores those changes in a table?
    
    > > I will describe the use of 'spread' and the Postgres-R internal issues
    > > in my next email.
    > 
    > The last time i was playing with spread (that was at Great Bridge in
    > Norfolk), it was IMHO useless (for Postgres-R) because it sometimes
    > dropped messages when the network load got too high. This occured
    > without any indication, no error, nothing. This is not exactly what I
    > understand as total order. I hope they have made some substantial
    > progress on that.
    
    That's a serious problem, clearly.  Hopefully it is either fixed or it
    will get fixed.  We can't use it without reliability.
    
    -- 
      Bruce Momjian                        |  http://candle.pha.pa.us
      pgman@candle.pha.pa.us               |  (610) 359-1001
      +  If your life is a hard drive,     |  13 Roberts Road
      +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073
    
    

21. Re: Big 7.4 items

    Neil Conway <neilc@samurai.com> — 2002-12-13T23:38:33Z

    On Fri, 2002-12-13 at 13:36, Jan Wieck wrote:
    > But you cannot use the result of such a SELECT to update anything. So
    > you can only phase out complete read only transaction to the slaves.
    > Requires support from the application since the load balancing system
    > cannot know automatically what will be a read only transaction and what
    > not.
    
    Interesting -- SQL contains the concept of "read only" and "read write"
    transactions (the default is RW). If we implemented that (which
    shouldn't be too difficult[1]), it might make differentiating between
    classes of transactions a little easier. Client applications would still
    need to be modified, but not nearly as much.
    
    Does this sound like it's worth doing?
    
    [1] -- AFAICS, the only tricky implementation detail is deciding exactly
    which database operations are "writes". Does nextval() count, for
    example?
    
    Cheers,
    
    Neil
    -- 
    Neil Conway <neilc@samurai.com> || PGP Key ID: DB3C29FC
    
    
    
    
    

22. Re: Big 7.4 items

    Bruce Momjian <pgman@candle.pha.pa.us> — 2002-12-14T00:48:09Z

    Neil Conway wrote:
    > On Fri, 2002-12-13 at 13:36, Jan Wieck wrote:
    > > But you cannot use the result of such a SELECT to update anything. So
    > > you can only phase out complete read only transaction to the slaves.
    > > Requires support from the application since the load balancing system
    > > cannot know automatically what will be a read only transaction and what
    > > not.
    > 
    > Interesting -- SQL contains the concept of "read only" and "read write"
    > transactions (the default is RW). If we implemented that (which
    > shouldn't be too difficult[1]), it might make differentiating between
    > classes of transactions a little easier. Client applications would still
    > need to be modified, but not nearly as much.
    > 
    > Does this sound like it's worth doing?
    > 
    > [1] -- AFAICS, the only tricky implementation detail is deciding exactly
    > which database operations are "writes". Does nextval() count, for
    > example?
    
    You can't migrate a session between nodes, so the entire _session_ has
    to be read-only.
    
    -- 
      Bruce Momjian                        |  http://candle.pha.pa.us
      pgman@candle.pha.pa.us               |  (610) 359-1001
      +  If your life is a hard drive,     |  13 Roberts Road
      +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073
    
    

23. Re: Big 7.4 items

    Christopher Kings-Lynne <chriskl@familyhealth.com.au> — 2002-12-14T01:20:16Z

    > This is a good point.  I don't want to push Postgres-R as our solution.
    > Rather, I have looked at both and like Postgres-R, but others need to
    > look at both and decide so we are all in agreement when we move forward.
    
    I think in either way, it's clear that they need to be in the main CVS, in
    order for it to get up to speed.
    
    Chris
    
    
    
    

24. Re: Big 7.4 items

    Neil Conway <neilc@samurai.com> — 2002-12-14T04:50:09Z

    On Fri, 2002-12-13 at 20:20, Christopher Kings-Lynne wrote:
    > > This is a good point.  I don't want to push Postgres-R as our solution.
    > > Rather, I have looked at both and like Postgres-R, but others need to
    > > look at both and decide so we are all in agreement when we move forward.
    > 
    > I think in either way, it's clear that they need to be in the main CVS, in
    > order for it to get up to speed.
    
    Why's that?
    
    Cheers,
    
    Neil
    -- 
    Neil Conway <neilc@samurai.com> || PGP Key ID: DB3C29FC
    
    
    
    
    

25. Re: Big 7.4 items

    Shridhar Daithankar <shridhar_daithankar@persistent.co.in> — 2002-12-14T05:39:30Z

    On Friday 13 December 2002 11:01 pm, you wrote:
    > Good.  This is the discussion we need.  Let me quote the TODO list
    > replication section first:
    >
    >     * Add replication of distributed databases [replication]
    >         o automatic failover
    
    Very good. We need that for HA.
    
    >         o load balancing
    
    >         o master/slave replication
    >         o multi-master replication
    >         o partition data across servers
    
    I am interested in this for multitude of reasons. Scalability is obviously one 
    of them. But I was just wondering about some things(After going thr. all the 
    messages on this).
    
    Once we have partitioning and replication, that effectively means database 
    cache can span multiple machines and no longer restricted by shared memory. 
    So will it work on mosix now? Just a thought.
    
    
    > OK, the first thing is that there isn't any one replication solution
    > that will behave optimally in all situations.
    >
    > Now, let me describe Postgres-R and then the other replication
    > solutions.  Postgres-R is multi-master, meaning you can send SELECT and
    > UPDATE/DELETE queries to any of the servers in the cluster, and get the
    > same result.  It is also synchronous, meaning it doesn't update the
    > local copy until it is sure the other nodes agree to the change. It
    > allows failover, because if one node goes down, the others keep going.
    >
    > Now, let me contrast:
    >
    > rserv and dbmirror do master/slave.  There is no mechanism to allow you
    > to do updates on the slave, and have them propagate to the master.  You
    > can, however, send SELECT queries to the slave, and in fact that's how
    > usogres does load balancing.
    
    Seems like mirroring v/s striping to me. Can we have both combined in either 
    fashion just like RAID.
    
    Most importantly will it be able to resize the cluster on the fly? Are we 
    looking at network management of database like Oracle does. (OK the tools are 
    unwarranted in many situation but it has to offer it).
    
    Most importantly I would like to see this thing easy to setup from a one 
    point-of-administration view.
    
    Something like declare a cluster of these n1 machines as database partitions 
    and have these another n2 machine do a slave sync with them for handling 
    loads. If these kind of command-line options are there, adding easy tools on 
    top of them should be a pop.
    
    And please, in place database upgrades. Otherwise it will be a huge pain to 
    maintain such a cluster over long period of times.
    
    
    > Another replication need is for asynchronous replication, most
    > traditionally for traveling salesmen who need to update their databases
    > periodically.  The only solution I know for that is PeerDirect's
    > PostgreSQL commercial offering at http://www.peerdirect.com.  It is
    > possible PITR may help with this, but we need to handle propagating
    > changes made by the salesmen back up into the server, and to do that, we
    > will need a mechanism to handle conflicts that occur when two people
    > update the same records.  This is always a problem for asynchronous
    > replication.
    
    We need not offer entire asynchronous replication all at once. We can have 
    levels of asynchronous replication like read only(Syncing essentially) and 
    Read-write. Even if we get slave sync only at first, that will be huge plus. 
    
    > > 2 If we are going to have replication, can we have built in load
    > > balancing? Is it a good idea to have it in postgresql or a
    > > separate application would be way to go?
    >
    > Well, because Postgres-R is multi-master, it has automatic load
    > balancing.  You can have your users point to whatever node you want.
    > You can implement this "pointing" by using dns IP address cycling, or
    > have a router that auto-load balances, though you would need to keep a
    > db session on the same node, of course.
    
    Umm. W.r.t above point i.e. combining data partitioning and slave-sync, will 
    it take a partitioned cluster as a single server or that cluster can take 
    care of itself in such situattions?
    
    >
    > So, in summary, I think we will eventually have two directions for
    > replication.  One is Postgres-R for multi-master, synchronous
    > replication, and PITR, for asynchronous replication.  I don't think
    
    I would put that as two options rather than directions. We need to be able to 
    deploy them both if required.
    
    Imagine postgresql running over 500 machine cluster..;-)
    
     Shridhar
    
    

26. Re: Big 7.4 items

    Justin Clift <justin@postgresql.org> — 2002-12-14T07:02:49Z

    Bruce Momjian wrote:
    > Joe Conway wrote:
    <snip>
    >>>Point-In-Time Recovery (PITR)
    >>>
    >>>	J. R. Nield did a PITR patch late in 7.3 development, and Patrick
    >>>	MacDonald from Red Hat is working on merging it into CVS and
    >>>	adding any missing pieces.  Patrick, do you have an ETA on that?
    >>
    >>As Hannu asked (and related to your question below), is there any thought of 
    >>extending this to allow simple log based replication? In many important 
    >>scenarios that would be more than adequate, and simpler to set up.
    <snip>
    
    For PITR-log-based-replication, how much data would be required to be pushed out to each slave system in order to bring 
    it up to date?
    
    I'm having visions of a 16MB WAL file being pushed out to slave systems in order to update them with a few rows of data...
    
    :-/
    
    Regards and best wishes,
    
    Justin Clift
    
    -- 
    "My grandfather once told me that there are two kinds of people: those
    who work and those who take the credit. He told me to try to be in the
    first group; there was less competition there."
    - Indira Gandhi
    
    
    

27. Re: Big 7.4 items

    Shridhar Daithankar <shridhar_daithankar@persistent.co.in> — 2002-12-14T07:34:43Z

    On 14 Dec 2002 at 18:02, Justin Clift wrote:
    > For PITR-log-based-replication, how much data would be required to be pushed out to each slave system in order to bring 
    > it up to date?
    > 
    > I'm having visions of a 16MB WAL file being pushed out to slave systems in order to update them with a few rows of data...
    
    I was under impression that data is pushed to slave after a checkpoint is 
    complete. i.e. 16MB of WAL file has recycled.
    
    Conversely a slave would contain accurate data upto last WAL checkpoint.
    
    I think tunable WAL size should be of some help in such scenario. Otherwise the 
    system designer has to use async.  replication. for granularity upto a 
    transaction.
    
    
    Bye
     Shridhar
    
    --
    Conference, n.:	A special meeting in which the boss gathers subordinates to 
    hear	what they have to say, so long as it doesn't conflict with what	he's 
    already decided to do.
    
    
    

28. Re: Big 7.4 items

    Christopher Kings-Lynne <chriskl@familyhealth.com.au> — 2002-12-14T09:53:03Z

    > > I think in either way, it's clear that they need to be in the main CVS, in
    > > order for it to get up to speed.
    >
    > Why's that?
    
    Because until replication is in CVS, it won't be used, tested and improved
    and developed as fast...
    
    Chris
    
    
    
    

29. Re: Big 7.4 items

    Greg Copeland <greg@copelandconsulting.net> — 2002-12-16T13:56:25Z

    On Fri, 2002-12-13 at 04:53, Hannu Krosing wrote:
    > On Fri, 2002-12-13 at 06:22, Bruce Momjian wrote:
    > > I wanted to outline some of the big items we are looking at for 7.4:
    > > Point-In-Time Recovery (PITR)
    > > 
    > > 	J. R. Nield did a PITR patch late in 7.3 development, and Patrick
    > > 	MacDonald from Red Hat is working on merging it into CVS and
    > > 	adding any missing pieces.  Patrick, do you have an ETA on that?
    > 
    > How hard would it be to extend PITR for master-slave (hot backup)
    > repliaction, which should then amount to continuously shipping logs to
    > slave and doing nonstop PITR there :)
    > 
    > It will never be usable for multi-master replication, but somehow it
    > feels that for master-slave replication simple log replay would be most
    > simple and robust solution.
    
    I'm curious, what would be the recovery strategy for PITR master-slave
    replication should the master fail (assuming hot fail over/backup)?  A
    simple dump/restore?  Are there/is there any facilities in PorstgreSQL
    for PITR archival which prevents PITR logs from be recycled (or perhaps,
    simply archived off)?  What about PITR streaming to networked and/or
    removable media?
    
    -- 
    Greg Copeland <greg@copelandconsulting.net>
    Copeland Computer Consulting
    
    
    

30. Re: Big 7.4 items

    Shridhar Daithankar <shridhar_daithankar@persistent.co.in> — 2002-12-16T14:02:58Z

    On Monday 16 December 2002 07:26 pm, you wrote:
    > I'm curious, what would be the recovery strategy for PITR master-slave
    > replication should the master fail (assuming hot fail over/backup)?  A
    > simple dump/restore?  Are there/is there any facilities in PorstgreSQL
    > for PITR archival which prevents PITR logs from be recycled (or perhaps,
    > simply archived off)?  What about PITR streaming to networked and/or
    > removable media?
    
    In asynchrounous replication, WAL log records are fed to anoter host, which 
    replays those transactions to sync the data. This way it does not matter if 
    WAL log is recycled as it is already replicated someplace else..
    
    HTH
    
     Shridhar
    
    

31. Re: Big 7.4 items

    Greg Copeland <greg@copelandconsulting.net> — 2002-12-16T14:13:13Z

    I must of miscommunicated here as you're describing PITR replication. 
    I'm asking about a master failing and the slaving picking up.  Now, some
    n-time later, how do you recover your master system to be back in sync
    with the slave.  Obviously, I'm assuming some level of manual recovery. 
    I'm wondering what the general approach would be.
    
    Consider that on the slave which is now the active server (master dead),
    it's possible that the slave's PITR's will be recycled before the master
    can come back up.  As such, unless there is a, an archival process for
    PITR or b, a method of streaming PITR's off for archival, the odds of
    using PITR to recover the master (resync if you will) seem greatly
    reduced as you will be unable to replay PITR on the master for
    synchronization.
    
    Greg
    
    
    
    On Mon, 2002-12-16 at 08:02, Shridhar Daithankar wrote:
    > On Monday 16 December 2002 07:26 pm, you wrote:
    > > I'm curious, what would be the recovery strategy for PITR master-slave
    > > replication should the master fail (assuming hot fail over/backup)?  A
    > > simple dump/restore?  Are there/is there any facilities in PorstgreSQL
    > > for PITR archival which prevents PITR logs from be recycled (or perhaps,
    > > simply archived off)?  What about PITR streaming to networked and/or
    > > removable media?
    > 
    > In asynchrounous replication, WAL log records are fed to anoter host, which 
    > replays those transactions to sync the data. This way it does not matter if 
    > WAL log is recycled as it is already replicated someplace else..
    > 
    > HTH
    > 
    >  Shridhar
    > 
    > ---------------------------(end of broadcast)---------------------------
    > TIP 4: Don't 'kill -9' the postmaster
    -- 
    Greg Copeland <greg@copelandconsulting.net>
    Copeland Computer Consulting
    
    
    

32. Re: Big 7.4 items

    Shridhar Daithankar <shridhar_daithankar@persistent.co.in> — 2002-12-16T14:20:09Z

    On Monday 16 December 2002 07:43 pm, you wrote:
    > Consider that on the slave which is now the active server (master dead),
    > it's possible that the slave's PITR's will be recycled before the master
    > can come back up.  As such, unless there is a, an archival process for
    > PITR or b, a method of streaming PITR's off for archival, the odds of
    > using PITR to recover the master (resync if you will) seem greatly
    > reduced as you will be unable to replay PITR on the master for
    > synchronization.
    
    I agree. Since we are talking about features in future release, I think it 
    should be added to TODO if not already there.
    
    I don't know about WAL numbering but AFAIU, it increments and old files are 
    removed once there are enough WAL files as specified in posgresql.conf. IIRC 
    there are some perl based replication project exist already which use this 
    feature.
    
     Shridhar
    
    
    

33. Re: Big 7.4 items

    Greg Copeland <greg@copelandconsulting.net> — 2002-12-16T14:37:53Z

    On Mon, 2002-12-16 at 08:20, Shridhar Daithankar wrote:
    > On Monday 16 December 2002 07:43 pm, you wrote:
    > > Consider that on the slave which is now the active server (master dead),
    > > it's possible that the slave's PITR's will be recycled before the master
    > > can come back up.  As such, unless there is a, an archival process for
    > > PITR or b, a method of streaming PITR's off for archival, the odds of
    > > using PITR to recover the master (resync if you will) seem greatly
    > > reduced as you will be unable to replay PITR on the master for
    > > synchronization.
    > 
    > I agree. Since we are talking about features in future release, I think it 
    > should be added to TODO if not already there.
    > 
    > I don't know about WAL numbering but AFAIU, it increments and old files are 
    > removed once there are enough WAL files as specified in posgresql.conf. IIRC 
    > there are some perl based replication project exist already which use this 
    > feature.
    > 
    
    The problem with this is that most people, AFAICT, are going to size WAL
    based on their performance/sizing requirements and not based on
    theoretical estimates which someone might make to allow for a window of
    failure.  That is, I don't believe increasing the number of WAL's is
    going to satisfactorily address the issue.
    
    
    -- 
    Greg Copeland <greg@copelandconsulting.net>
    Copeland Computer Consulting
    
    
    

34. Re: Big 7.4 items

    Shridhar Daithankar <shridhar_daithankar@persistent.co.in> — 2002-12-16T14:42:29Z

    On Monday 16 December 2002 08:07 pm, you wrote:
    > On Mon, 2002-12-16 at 08:20, Shridhar Daithankar wrote:
    > > I don't know about WAL numbering but AFAIU, it increments and old files
    > > are removed once there are enough WAL files as specified in
    > > posgresql.conf. IIRC there are some perl based replication project exist
    > > already which use this feature.
    >
    > The problem with this is that most people, AFAICT, are going to size WAL
    > based on their performance/sizing requirements and not based on
    > theoretical estimates which someone might make to allow for a window of
    > failure.  That is, I don't believe increasing the number of WAL's is
    > going to satisfactorily address the issue.
    
    Sorry for not being clear. When I said, WAL numbering, I meant WAL naming 
    conventions where numbers are used to mark WAL files. 
    
    It is not number of WAL files. It is entirely upto the installation and IIRC, 
    even in replication project(Sorry I forgot the exact name), you can set 
    number of WAL files that it can have.
    
     Shridhar
    
    
    

35. Re: Big 7.4 items

    Bruce Momjian <pgman@candle.pha.pa.us> — 2002-12-16T15:51:46Z

    Shridhar Daithankar wrote:
    > On Monday 16 December 2002 08:07 pm, you wrote:
    > > On Mon, 2002-12-16 at 08:20, Shridhar Daithankar wrote:
    > > > I don't know about WAL numbering but AFAIU, it increments and old files
    > > > are removed once there are enough WAL files as specified in
    > > > posgresql.conf. IIRC there are some perl based replication project exist
    > > > already which use this feature.
    > >
    > > The problem with this is that most people, AFAICT, are going to size WAL
    > > based on their performance/sizing requirements and not based on
    > > theoretical estimates which someone might make to allow for a window of
    > > failure.  That is, I don't believe increasing the number of WAL's is
    > > going to satisfactorily address the issue.
    > 
    > Sorry for not being clear. When I said, WAL numbering, I meant WAL naming 
    > conventions where numbers are used to mark WAL files. 
    > 
    > It is not number of WAL files. It is entirely upto the installation and IIRC, 
    > even in replication project(Sorry I forgot the exact name), you can set 
    > number of WAL files that it can have.
    
    Basically, PITR is going to have a way to archive off a log of database
    changes, either from WAL or from somewhere else.  At some point, there
    is going to have to be administrative action which says, "I have a
    master down for three days.  I am going to have to save my PITR logs for
    that period."  So, PITR will probably be used for recovery of a failed
    master, and such recover is going to have to require some administrative
    action _if_ the automatic expiration of PITR logs is shorter than the
    duration the master is down.
    
    -- 
      Bruce Momjian                        |  http://candle.pha.pa.us
      pgman@candle.pha.pa.us               |  (610) 359-1001
      +  If your life is a hard drive,     |  13 Roberts Road
      +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073
    
    

36. Re: Big 7.4 items

    Patrick Macdonald <patrickm@redhat.com> — 2002-12-16T17:47:28Z

    Bruce Momjian wrote:
    > 
    > I wanted to outline some of the big items we are looking at for 7.4:
    > 
    > [snip]
    >
    > Point-In-Time Recovery (PITR)
    > 
    >         J. R. Nield did a PITR patch late in 7.3 development, and Patrick
    >         MacDonald from Red Hat is working on merging it into CVS and
    >         adding any missing pieces.  Patrick, do you have an ETA on that?
    
    Neil Conway and I will be working on this starting the beginning
    of January.  By the middle of January, we hope to have a handle on
    an ETA.
    
    Cheers,
    Patrick
    --
    Patrick Macdonald
    Red Hat Database Development
    
    

37. Re: Big 7.4 items

    Bruce Momjian <pgman@candle.pha.pa.us> — 2002-12-16T18:38:37Z

    Patrick Macdonald wrote:
    > Bruce Momjian wrote:
    > > 
    > > I wanted to outline some of the big items we are looking at for 7.4:
    > > 
    > > [snip]
    > >
    > > Point-In-Time Recovery (PITR)
    > > 
    > >         J. R. Nield did a PITR patch late in 7.3 development, and Patrick
    > >         MacDonald from Red Hat is working on merging it into CVS and
    > >         adding any missing pieces.  Patrick, do you have an ETA on that?
    > 
    > Neil Conway and I will be working on this starting the beginning
    > of January.  By the middle of January, we hope to have a handle on
    > an ETA.
    
    Ewe, that is later than I was hoping.  I have put J.R's PITR patch up
    at:
    
    	ftp://candle.pha.pa.us/pub/postgresql/PITR_20020822_02.gz
    
    (I have tried to contact J.R. several times over the past few months,
    with no reply.)
    
    J.R felt it was ready to go.  I would like to have an evaluation of the
    patch to know what it does and what is missing.  I would like that
    sooner rather than later because:
    
    	o  I want to avoid too much code drift
    	o  I don't want to find there are major pieces missing and to 
    	   not have enough time to implement them in 7.4
    	o  It is a big feature so I would like sufficient testing before beta
    
    OK, I just talked to Patrick on the phone, and he says Neil Conway is
    working on merging the code into 7.3, and adding missing pieces like
    logging table creation.  So, it seems PITR is moving forward.  Neil, can
    you comment on where you are with this, and what still needs to be done?
    Do we need to start looking at log archiving options?  How is the PITR
    log contents different from the WAL log contents, except of course no
    pre-write page images?
    
    If we need to discuss things, perhaps we can do it now and get folks
    working on other pieces, or at least thinking about them.
    
    -- 
      Bruce Momjian                        |  http://candle.pha.pa.us
      pgman@candle.pha.pa.us               |  (610) 359-1001
      +  If your life is a hard drive,     |  13 Roberts Road
      +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073
    
    

38. Re: Big 7.4 items

    Neil Conway <neilc@samurai.com> — 2002-12-16T19:46:19Z

    On Mon, 2002-12-16 at 13:38, Bruce Momjian wrote:
    > OK, I just talked to Patrick on the phone, and he says Neil Conway is
    > working on merging the code into 7.3, and adding missing pieces like
    > logging table creation.  So, it seems PITR is moving forward.  Neil, can
    > you comment on where you are with this, and what still needs to be done?
    
    Well, I should have a preliminary merge of the old PITR patch with CVS
    HEAD done by Wednesday or Thursday. It took me a while to merge because
    (a) I've got final exams at university at the moment (b) I had to merge
    most of it by hand, as much of the diff is a single hunk (!), for some
    reason.
    
    As for the status of the code, I haven't really had a chance to evaluate
    it; as Patrick noted, I think we should be able to give you an ETA by
    the middle of January or so (I'll be offline starting Thursday until the
    first week of January).
    
    Cheers,
    
    Neil
    -- 
    Neil Conway <neilc@samurai.com> || PGP Key ID: DB3C29FC
    
    
    
    
    

39. Re: Big 7.4 items

    Patrick Macdonald <patrickm@redhat.com> — 2002-12-16T19:50:27Z

    Bruce Momjian wrote:
    > 
    > Patrick Macdonald wrote:
    > > Bruce Momjian wrote:
    > > >
    > > > I wanted to outline some of the big items we are looking at for 7.4:
    > > >
    > > > [snip]
    > > >
    > > > Point-In-Time Recovery (PITR)
    > > >
    > > >         J. R. Nield did a PITR patch late in 7.3 development, and Patrick
    > > >         MacDonald from Red Hat is working on merging it into CVS and
    > > >         adding any missing pieces.  Patrick, do you have an ETA on that?
    > >
    > > Neil Conway and I will be working on this starting the beginning
    > > of January.  By the middle of January, we hope to have a handle on
    > > an ETA.
    > 
    > Ewe, that is later than I was hoping.  I have put J.R's PITR patch up
    > at:
    > 
    >         ftp://candle.pha.pa.us/pub/postgresql/PITR_20020822_02.gz
    > 
    > (I have tried to contact J.R. several times over the past few months,
    > with no reply.)
    > 
    > J.R felt it was ready to go.  I would like to have an evaluation of the
    > patch to know what it does and what is missing.  I would like that
    > sooner rather than later because:
    > 
    >         o  I want to avoid too much code drift
    >         o  I don't want to find there are major pieces missing and to
    >            not have enough time to implement them in 7.4
    >         o  It is a big feature so I would like sufficient testing before beta
    > 
    > OK, I just talked to Patrick on the phone, and he says Neil Conway is
    > working on merging the code into 7.3, and adding missing pieces like
    > logging table creation.  So, it seems PITR is moving forward.
    
    Well, sort of.  I stated that Neil was already working on merging the
    patch into the CVS tip.  I also mentioned that there are missing 
    pieces but have no idea if Neil is currently working on them.
    
    Cheers,
    Patrick
    --
    Patrick Macdonald
    Red Hat Database Development
    
    

40. Re: Big 7.4 items

    Bruce Momjian <pgman@candle.pha.pa.us> — 2002-12-16T19:52:04Z

    Patrick Macdonald wrote:
    > > OK, I just talked to Patrick on the phone, and he says Neil Conway is
    > > working on merging the code into 7.3, and adding missing pieces like
    > > logging table creation.  So, it seems PITR is moving forward.
    > 
    > Well, sort of.  I stated that Neil was already working on merging the
    > patch into the CVS tip.  I also mentioned that there are missing 
    > pieces but have no idea if Neil is currently working on them.
    
    Oh, OK.  What I would like to do is find out what actually needs to be
    done so we can get folks started on it.  If we can get a 7.3 merge,
    maybe we should get it into CVS and then list the items needing
    attention and folks can submit patches to implement those.
    
    -- 
      Bruce Momjian                        |  http://candle.pha.pa.us
      pgman@candle.pha.pa.us               |  (610) 359-1001
      +  If your life is a hard drive,     |  13 Roberts Road
      +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073
    
    

41. Re: Big 7.4 items

    Janardhan <jana-reddy@mediaring.com.sg> — 2002-12-17T02:35:49Z

    The  file  "ftp://candle.pha.pa.us/pub/postgresql/PITR_20020822_02.gz" 
    do not
    have read permissions to copy. please provide the read permissions to copy.
    
    Regards
    jana
    
    >Patrick Macdonald wrote:
    >  
    >
    >>Bruce Momjian wrote:
    >>    
    >>
    >>>I wanted to outline some of the big items we are looking at for 7.4:
    >>>
    >>>[snip]
    >>>
    >>>Point-In-Time Recovery (PITR)
    >>>
    >>>        J. R. Nield did a PITR patch late in 7.3 development, and Patrick
    >>>        MacDonald from Red Hat is working on merging it into CVS and
    >>>        adding any missing pieces.  Patrick, do you have an ETA on that?
    >>>      
    >>>
    >>Neil Conway and I will be working on this starting the beginning
    >>of January.  By the middle of January, we hope to have a handle on
    >>an ETA.
    >>    
    >>
    >
    >Ewe, that is later than I was hoping.  I have put J.R's PITR patch up
    >at:
    >
    >	ftp://candle.pha.pa.us/pub/postgresql/PITR_20020822_02.gz
    >
    >(I have tried to contact J.R. several times over the past few months,
    >with no reply.)
    >
    >J.R felt it was ready to go.  I would like to have an evaluation of the
    >patch to know what it does and what is missing.  I would like that
    >sooner rather than later because:
    >
    >	o  I want to avoid too much code drift
    >	o  I don't want to find there are major pieces missing and to 
    >	   not have enough time to implement them in 7.4
    >	o  It is a big feature so I would like sufficient testing before beta
    >
    >OK, I just talked to Patrick on the phone, and he says Neil Conway is
    >working on merging the code into 7.3, and adding missing pieces like
    >logging table creation.  So, it seems PITR is moving forward.  Neil, can
    >you comment on where you are with this, and what still needs to be done?
    >Do we need to start looking at log archiving options?  How is the PITR
    >log contents different from the WAL log contents, except of course no
    >pre-write page images?
    >
    >If we need to discuss things, perhaps we can do it now and get folks
    >working on other pieces, or at least thinking about them.
    >
    >  
    >
    
    

42. Re: Big 7.4 items

    Bruce Momjian <pgman@candle.pha.pa.us> — 2002-12-17T03:26:54Z

    Oops, sorry.  Permissions fixed.
    
    ---------------------------------------------------------------------------
    
    Janardhan wrote:
    > The  file  "ftp://candle.pha.pa.us/pub/postgresql/PITR_20020822_02.gz" 
    > do not
    > have read permissions to copy. please provide the read permissions to copy.
    > 
    > Regards
    > jana
    > 
    > >Patrick Macdonald wrote:
    > >  
    > >
    > >>Bruce Momjian wrote:
    > >>    
    > >>
    > >>>I wanted to outline some of the big items we are looking at for 7.4:
    > >>>
    > >>>[snip]
    > >>>
    > >>>Point-In-Time Recovery (PITR)
    > >>>
    > >>>        J. R. Nield did a PITR patch late in 7.3 development, and Patrick
    > >>>        MacDonald from Red Hat is working on merging it into CVS and
    > >>>        adding any missing pieces.  Patrick, do you have an ETA on that?
    > >>>      
    > >>>
    > >>Neil Conway and I will be working on this starting the beginning
    > >>of January.  By the middle of January, we hope to have a handle on
    > >>an ETA.
    > >>    
    > >>
    > >
    > >Ewe, that is later than I was hoping.  I have put J.R's PITR patch up
    > >at:
    > >
    > >	ftp://candle.pha.pa.us/pub/postgresql/PITR_20020822_02.gz
    > >
    > >(I have tried to contact J.R. several times over the past few months,
    > >with no reply.)
    > >
    > >J.R felt it was ready to go.  I would like to have an evaluation of the
    > >patch to know what it does and what is missing.  I would like that
    > >sooner rather than later because:
    > >
    > >	o  I want to avoid too much code drift
    > >	o  I don't want to find there are major pieces missing and to 
    > >	   not have enough time to implement them in 7.4
    > >	o  It is a big feature so I would like sufficient testing before beta
    > >
    > >OK, I just talked to Patrick on the phone, and he says Neil Conway is
    > >working on merging the code into 7.3, and adding missing pieces like
    > >logging table creation.  So, it seems PITR is moving forward.  Neil, can
    > >you comment on where you are with this, and what still needs to be done?
    > >Do we need to start looking at log archiving options?  How is the PITR
    > >log contents different from the WAL log contents, except of course no
    > >pre-write page images?
    > >
    > >If we need to discuss things, perhaps we can do it now and get folks
    > >working on other pieces, or at least thinking about them.
    > >
    > >  
    > >
    > 
    
    -- 
      Bruce Momjian                        |  http://candle.pha.pa.us
      pgman@candle.pha.pa.us               |  (610) 359-1001
      +  If your life is a hard drive,     |  13 Roberts Road
      +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073
    
    

43. Re: Big 7.4 items

    Thomas F.O'Connell <tfo@monsterlabs.com> — 2002-12-17T17:15:41Z

    So if this gets added to the 7.3 branch, will there be documentation 
    accompanying it?
    
    -tfo
    
    In article <200212161838.gBGIcb717436@candle.pha.pa.us>,
     pgman@candle.pha.pa.us (Bruce Momjian) wrote:
    
    > OK, I just talked to Patrick on the phone, and he says Neil Conway is
    > working on merging the code into 7.3, and adding missing pieces like
    > logging table creation.  So, it seems PITR is moving forward.
    
    

44. Re: Big 7.4 items

    Bruce Momjian <pgman@candle.pha.pa.us> — 2002-12-17T17:21:19Z

    I meant he is merging it into HEAD, not the 7.3 CVS.  Sorry for the
    confusion.
    
    ---------------------------------------------------------------------------
    
    Thomas O'Connell wrote:
    > So if this gets added to the 7.3 branch, will there be documentation 
    > accompanying it?
    > 
    > -tfo
    > 
    > In article <200212161838.gBGIcb717436@candle.pha.pa.us>,
    >  pgman@candle.pha.pa.us (Bruce Momjian) wrote:
    > 
    > > OK, I just talked to Patrick on the phone, and he says Neil Conway is
    > > working on merging the code into 7.3, and adding missing pieces like
    > > logging table creation.  So, it seems PITR is moving forward.
    > 
    > ---------------------------(end of broadcast)---------------------------
    > TIP 4: Don't 'kill -9' the postmaster
    > 
    
    -- 
      Bruce Momjian                        |  http://candle.pha.pa.us
      pgman@candle.pha.pa.us               |  (610) 359-1001
      +  If your life is a hard drive,     |  13 Roberts Road
      +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073