Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

Dilip Kumar <dilipbalaut@gmail.com>

From: Dilip Kumar <dilipbalaut@gmail.com>

To: vignesh C <vignesh21@gmail.com>

Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>, Amit Kapila <amit.kapila16@gmail.com>, a.kondratov@postgrespro.ru, Peter Eisentraut <peter.eisentraut@2ndquadrant.com>, PostgreSQL Hackers <pgsql-hackers@postgresql.org>

Date: 2019-10-30T07:48:59Z

Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

Tighten the concurrent abort check during decoding.
- 2ce353fc1902 14.0 landed
Improve hash_create()'s API for some added robustness.
- b3817f5f7746 14.0 landed
Use HASH_BLOBS for xidhash.
- a1b8aa1e4eec 14.0 landed
Fix initialization of RelationSyncEntry for streaming transactions.
- 69bd60672af6 14.0 landed
Remove unused function declaration in logicalproto.h.
- ddd5f6d2609b 14.0 landed
Add additional tests to test streaming of in-progress transactions.
- 58b5ae9d62bd 14.0 landed
Fix inline marking introduced in commit 464824323e.
- ac15b499f7f9 14.0 landed
Add support for streaming to built-in logical replication.
- 464824323e57 14.0 landed
Fix the SharedFileSetUnregister API.
- 4ab77697f67a 14.0 landed
Fix comment in procarray.c
- 77c7267c37f7 14.0 cited
Suppress compiler warning in non-cassert builds.
- e942af7b8261 14.0 cited
Extend the BufFile interface.
- 808e13b282ef 14.0 landed
Mark a few logical decoding related variables with PGDLLIMPORT.
- b48cac3b10a0 14.0 landed
Implement streaming mode in ReorderBuffer.
- 7259736a6e5b 14.0 landed
Extend the logical decoding output plugin API with stream methods.
- 45fdc9738b36 14.0 landed
WAL Log invalidations at command end with wal_level=logical.
- c55040ccd017 14.0 landed
Immediately WAL-log subtransaction and top-level XID association.
- 0bead9af484c 14.0 landed
Allow logical replication to transfer data in binary format.
- 9de77b545313 14.0 cited
Only superuser can set sslcert/sslkey in postgres_fdw user mappings
- cebf9d6e6ee1 13.0 cited
Track statistics for spilling of changes from ReorderBuffer.
- 9290ad198b15 13.0 landed
Add logical_decoding_work_mem to limit ReorderBuffer memory usage.
- cec2edfa7859 13.0 landed
logical decoding: process ASSIGNMENT during snapshot build
- bac2fae05c77 13.0 cited
Emit invalidations to standby for transactions without xid.
- c6ff84b06a68 9.6.0 cited

On Wed, Oct 30, 2019 at 9:38 AM vignesh C <vignesh21@gmail.com> wrote:
>
I have noticed one more problem in the logic of setting the logical
decoding work mem from the create subscription command.  Suppose in
subscription command we don't give the work mem then it sends the
garbage value to the walsender and the walsender overwrite its value
with the garbage value.  After investigating a bit I have found the
reason for the same.

@@ -406,6 +406,9 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
  appendStringInfo(&cmd, "proto_version '%u'",
  options->proto.logical.proto_version);

+ appendStringInfo(&cmd, ", work_mem '%d'",
+ options->proto.logical.work_mem);

I think the problem is we are unconditionally sending the work_mem as
part of the CREATE REPLICATION SLOT, without checking whether it's
valid or not.

--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -71,6 +71,7 @@ GetSubscription(Oid subid, bool missing_ok)
  sub->name = pstrdup(NameStr(subform->subname));
  sub->owner = subform->subowner;
  sub->enabled = subform->subenabled;
+ sub->workmem = subform->subworkmem;

Another problem is that there is no handling if the subform->subworkmem is NULL.

-- 
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com