Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
Amit Kapila <amit.kapila16@gmail.com>
Commits
GET /api/v1/messages/:b64id/commits
the thread's linked commits as JSON, with link sources.
API reference →
-
Tighten the concurrent abort check during decoding.
- 2ce353fc1902 14.0 landed
-
Improve hash_create()'s API for some added robustness.
- b3817f5f7746 14.0 landed
-
Use HASH_BLOBS for xidhash.
- a1b8aa1e4eec 14.0 landed
-
Fix initialization of RelationSyncEntry for streaming transactions.
- 69bd60672af6 14.0 landed
-
Remove unused function declaration in logicalproto.h.
- ddd5f6d2609b 14.0 landed
-
Add additional tests to test streaming of in-progress transactions.
- 58b5ae9d62bd 14.0 landed
-
Fix inline marking introduced in commit 464824323e.
- ac15b499f7f9 14.0 landed
-
Add support for streaming to built-in logical replication.
- 464824323e57 14.0 landed
-
Fix the SharedFileSetUnregister API.
- 4ab77697f67a 14.0 landed
-
Fix comment in procarray.c
- 77c7267c37f7 14.0 cited
-
Suppress compiler warning in non-cassert builds.
- e942af7b8261 14.0 cited
-
Extend the BufFile interface.
- 808e13b282ef 14.0 landed
-
Mark a few logical decoding related variables with PGDLLIMPORT.
- b48cac3b10a0 14.0 landed
-
Implement streaming mode in ReorderBuffer.
- 7259736a6e5b 14.0 landed
-
Extend the logical decoding output plugin API with stream methods.
- 45fdc9738b36 14.0 landed
-
WAL Log invalidations at command end with wal_level=logical.
- c55040ccd017 14.0 landed
-
Immediately WAL-log subtransaction and top-level XID association.
- 0bead9af484c 14.0 landed
-
Allow logical replication to transfer data in binary format.
- 9de77b545313 14.0 cited
-
Only superuser can set sslcert/sslkey in postgres_fdw user mappings
- cebf9d6e6ee1 13.0 cited
-
Track statistics for spilling of changes from ReorderBuffer.
- 9290ad198b15 13.0 landed
-
Add logical_decoding_work_mem to limit ReorderBuffer memory usage.
- cec2edfa7859 13.0 landed
-
logical decoding: process ASSIGNMENT during snapshot build
- bac2fae05c77 13.0 cited
-
Emit invalidations to standby for transactions without xid.
- c6ff84b06a68 9.6.0 cited
On Mon, Jun 29, 2020 at 4:24 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Mon, Jun 22, 2020 at 4:30 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> >
> > Other than above tests, can we somehow verify that the invalidations
> > generated at commit time are the same as what we do with this patch?
> > We have verified with individual commands but it would be great if we
> > can verify for the regression tests.
>
> I have verified this using a few random test cases. For verifying
> this I have made some temporary code changes with an assert as shown
> below. Basically, on DecodeCommit we call
> ReorderBufferAddInvalidations function only for an assert checking.
>
> -void
> ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
> XLogRecPtr
> lsn, Size nmsgs,
> -
> SharedInvalidationMessage *msgs)
> +
> SharedInvalidationMessage *msgs, bool commit)
> {
> ReorderBufferTXN *txn;
>
> txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
> -
> + if (commit)
> + {
> + Assert(txn->ninvalidations == nmsgs);
> + return;
> + }
>
> The result is that for a normal local test it works fine. But with
> regression suit, it hit an assert at many places because if the
> rollback of the subtransaction is involved then at commit time
> invalidation messages those are not logged whereas with command time
> invalidation those are logged.
>
Yeah, somehow, we need to ignore rollback to savepoint tests and
verify for others.
> As of now, I have only put assert on the count, if we need to verify
> the exact messages then we might need to somehow categories the
> invalidation messages because the ordering of the messages will not be
> the same. For testing this we will have to arrange them by category
> i.e relcahce, catcache and then we can compare them.
>
Can't we do this by verifying that each message at commit time exists
in the list of invalidation messages we have collected via processing
XLOG_XACT_INVALIDATIONS?
One additional question on patch
v30-0003-Extend-the-output-plugin-API-with-stream-methods:
+static void
+stream_commit_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn)
{
..
..
+ state.report_location = apply_lsn;
..
..
+ ctx->write_location = apply_lsn;
..
}
Can't we name the last parameter as 'commit_lsn' as that is how
documentation in the patch spells it and it sounds more appropriate?
Also, is there a reason for assigning report_location and
write_location differently than what we do in commit_cb_wrapper?
Basically, assign those as txn->final_lsn and txn->end_lsn
respectively.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com