[PATCH 11/14] Introduce wal decoding via catalog timetravel
Andres Freund <andres@2ndquadrant.com>
From: Andres Freund <andres@2ndquadrant.com>
To: pgsql-hackers@postgresql.org
Date: 2012-11-15T01:17:11Z
Lists: pgsql-hackers
Attachments
- 0011-Introduce-wal-decoding-via-catalog-timetravel.patch (text/x-patch) patch 0011
This introduces several things: * 'reorderbuffer' module which reassembles transactions from a stream of interspersed changes * 'snapbuilder' which builds catalog snapshots so that tuples from wal can be understood * logging more data into wal to facilitate logical decoding * wal decoding into an reorderbuffer * shared library output plugins with 5 callbacks * init * begin * change * commit * walsender infrastructur to stream out changes and to keep the global xmin low enough * INIT_LOGICAL_REPLICATION $plugin; waits till a consistent snapshot is built and returns * initial LSN * replication slot identifier * id of a pg_export() style snapshot * START_LOGICAL_REPLICATION $id $lsn; streams out changes * uses named output plugins for output specification Todo: * testing infrastructure (isolationtester) * persistence/spilling to disk of built snapshots, longrunning transactions * user docs * more frequent lowering of xmins * more docs about the internals * support for user declared catalog tables * actual exporting of initial pg_export snapshots after INIT_LOGICAL_REPLICATION * own shared memory segment instead of piggybacking on walsender's * nicer interface between snapbuild.c, reorderbuffer.c, decode.c and the outside. * more frequent xl_running_xid's so xmin can be upped more frequently * add STOP_LOGICAL_REPLICATION $id --- src/backend/access/heap/heapam.c | 280 +++++- src/backend/access/transam/xlog.c | 1 + src/backend/catalog/index.c | 74 ++ src/backend/replication/Makefile | 2 + src/backend/replication/logical/Makefile | 19 + src/backend/replication/logical/decode.c | 496 ++++++++++ src/backend/replication/logical/logicalfuncs.c | 247 +++++ src/backend/replication/logical/reorderbuffer.c | 1156 +++++++++++++++++++++++ src/backend/replication/logical/snapbuild.c | 1144 ++++++++++++++++++++++ src/backend/replication/repl_gram.y | 32 +- src/backend/replication/repl_scanner.l | 2 + src/backend/replication/walsender.c | 566 ++++++++++- src/backend/storage/ipc/procarray.c | 23 + src/backend/storage/ipc/standby.c | 8 +- src/backend/utils/cache/inval.c | 2 +- src/backend/utils/cache/relcache.c | 3 +- src/backend/utils/misc/guc.c | 11 + src/backend/utils/time/tqual.c | 249 +++++ src/bin/pg_controldata/pg_controldata.c | 2 + src/include/access/heapam_xlog.h | 23 + src/include/access/transam.h | 5 + src/include/access/xlog.h | 3 +- src/include/catalog/index.h | 4 + src/include/nodes/nodes.h | 2 + src/include/nodes/replnodes.h | 22 + src/include/replication/decode.h | 21 + src/include/replication/logicalfuncs.h | 44 + src/include/replication/output_plugin.h | 76 ++ src/include/replication/reorderbuffer.h | 284 ++++++ src/include/replication/snapbuild.h | 128 +++ src/include/replication/walsender.h | 1 + src/include/replication/walsender_private.h | 34 +- src/include/storage/itemptr.h | 3 + src/include/storage/sinval.h | 2 + src/include/utils/tqual.h | 31 +- 35 files changed, 4966 insertions(+), 34 deletions(-) create mode 100644 src/backend/replication/logical/Makefile create mode 100644 src/backend/replication/logical/decode.c create mode 100644 src/backend/replication/logical/logicalfuncs.c create mode 100644 src/backend/replication/logical/reorderbuffer.c create mode 100644 src/backend/replication/logical/snapbuild.c create mode 100644 src/include/replication/decode.h create mode 100644 src/include/replication/logicalfuncs.h create mode 100644 src/include/replication/output_plugin.h create mode 100644 src/include/replication/reorderbuffer.h create mode 100644 src/include/replication/snapbuild.h
Commits
Same data as JSON:
GET /api/v1/messages/:b64id/commits
the thread's linked commits as JSON, with link sources.
API reference →
-
Background worker processes
- da07a1e85651 9.3.0 cited
-
Rearrange storage of data in xl_running_xacts.
- 5c11725867ac 9.3.0 cited
-
Basic binary heap implementation.
- 7a2fe9bd0371 9.3.0 cited
-
Embedded list interface
- a66ee69add6e 9.3.0 cited
-
Refactor xlog.c to create src/backend/postmaster/startup.c
- 9aceb6ab3c20 9.2.0 cited