Horton

Re: [PATCH 10/16] Introduce the concept that wal has a 'origin' node

Andres Freund <andres@2ndquadrant.com>

View thread
From: Andres Freund <andres@2ndquadrant.com>
To: pgsql-hackers@postgresql.org
Cc: Tom Lane <tgl@sss.pgh.pa.us>, Daniel Farina <daniel@heroku.com>, Simon Riggs <simon@2ndquadrant.com>
Date: 2012-06-19T14:45:42Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Don't waste the last segment of each 4GB logical log file.

  2. Stamp HEAD as 9.3devel.

  3. Wake WALSender to reduce data loss at failover for async commit.

  4. Make the visibility map crash-safe.

On Tuesday, June 19, 2012 04:30:59 PM Tom Lane wrote:
> Andres Freund <andres@2ndquadrant.com> writes:
> > On Tuesday, June 19, 2012 04:17:01 PM Tom Lane wrote:
> >> ...  (If you are thinking
> >> of something sufficiently high-level that merging could possibly work,
> >> then it's not WAL, and we shouldn't be trying to make the WAL
> >> representation cater for it.)
> > 
> > The idea is that if youre replaying changes on node A originating from
> > node B you set the origin to *B* in the wal records that are generated
> > during that. So when B, in a bidirectional setup, replays the changes
> > that A has made it can simply ignore all changes which originated on
> > itself.
> 
> This is most certainly not possible at the level of WAL. 
Huh? This isn't used during normal crash-recovery replay. The information is 
used when decoding the wal into logical changes and applying those. Its just a 
common piece of information thats needed for a large number of records. 

Alternatively it could be added to all the records that need it, but that 
would smear the necessary logic - which is currently trivial - over more of 
the backend. And it would increase the actual size of wal which this one did 
not.

> As I said above, we shouldn't be trying to shoehorn high level logical-
> replication commands into WAL streams.  No good can come of confusing those
> concepts.
Its not doing anything high-level in there? All that patch does is embedding 
one single piece of information in previously unused space.

I can follow the argument that you do not want *any* logical information in 
the wal. But as I said in the patchset-introducing email: I don't really see 
an alternative. Otherwise we would just duplicate all the locking/scalability 
issues of xlog as well as the amount of writes.
This is, besides logging some more informations when wal_level = logical in 
some particular records (HEAP_UPDATE|DELETE and ensuring fpw's don't remove 
the record data in HEAP_(INSERT|UPDATE|DELETE) in patch 07/16 the only change 
that I really forsee being needed for doing the logical stuff.

Do you really see this as such a big problem?

Andres
-- 
 Andres Freund	                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services