Horton

Re: [PATCH 06/16] Add support for a generic wal reading facility dubbed XLogReader

Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>

View thread
From: Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
To: Andres Freund <andres@2ndquadrant.com>
Cc: pgsql-hackers@postgresql.org
Date: 2012-06-14T21:19:00Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Don't waste the last segment of each 4GB logical log file.

  2. Stamp HEAD as 9.3devel.

  3. Wake WALSender to reduce data loss at failover for async commit.

  4. Make the visibility map crash-safe.

On 13.06.2012 14:28, Andres Freund wrote:
> Features:
> - streaming reading/writing
> - filtering
> - reassembly of records
>
> Reusing the ReadRecord infrastructure in situations where the code that wants
> to do so is not tightly integrated into xlog.c is rather hard and would require
> changes to rather integral parts of the recovery code which doesn't seem to be
> a good idea.

It would be nice refactor ReadRecord and its subroutines out of xlog.c. 
That file has grown over the years to be really huge, and separating the 
code to read WAL sounds like it should be a pretty natural split. I 
don't want to duplicate all the WAL reading code, so we really should 
find a way to reuse that. I'd suggest rewriting ReadRecord into a thin 
wrapper that just calls the new xlogreader code.

> Missing:
> - "compressing" the stream when removing uninteresting records
> - writing out correct CRCs
> - validating CRCs
> - separating reader/writer

- comments.

At a quick glance, I couldn't figure out how this works. There seems to 
be some callback functions? If you want to read an xlog stream using 
this facility, what do you do? Can this be used for writing WAL, as well 
as reading? If so, what do you need the write support for?

-- 
   Heikki Linnakangas
   EnterpriseDB   http://www.enterprisedb.com