Horton

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

Amit Kapila <amit.kapila16@gmail.com>

View thread
From: Amit Kapila <amit.kapila16@gmail.com>
To: Dilip Kumar <dilipbalaut@gmail.com>
Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>, Peter Eisentraut <peter.eisentraut@2ndquadrant.com>, PostgreSQL Hackers <pgsql-hackers@postgresql.org>
Date: 2019-11-15T10:49:27Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Tighten the concurrent abort check during decoding.

  2. Improve hash_create()'s API for some added robustness.

  3. Use HASH_BLOBS for xidhash.

  4. Fix initialization of RelationSyncEntry for streaming transactions.

  5. Remove unused function declaration in logicalproto.h.

  6. Add additional tests to test streaming of in-progress transactions.

  7. Fix inline marking introduced in commit 464824323e.

  8. Add support for streaming to built-in logical replication.

  9. Fix the SharedFileSetUnregister API.

  10. Fix comment in procarray.c

  11. Suppress compiler warning in non-cassert builds.

  12. Extend the BufFile interface.

  13. Mark a few logical decoding related variables with PGDLLIMPORT.

  14. Implement streaming mode in ReorderBuffer.

  15. Extend the logical decoding output plugin API with stream methods.

  16. WAL Log invalidations at command end with wal_level=logical.

  17. Immediately WAL-log subtransaction and top-level XID association.

  18. Allow logical replication to transfer data in binary format.

  19. Only superuser can set sslcert/sslkey in postgres_fdw user mappings

  20. Track statistics for spilling of changes from ReorderBuffer.

  21. Add logical_decoding_work_mem to limit ReorderBuffer memory usage.

  22. logical decoding: process ASSIGNMENT during snapshot build

  23. Emit invalidations to standby for transactions without xid.

On Fri, Nov 15, 2019 at 4:01 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Fri, Nov 15, 2019 at 3:50 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> >
> > Few other comments on this patch:
> > 1.
> > + case REORDER_BUFFER_CHANGE_INVALIDATION:
> > +
> > + /*
> > + * Execute the invalidation message locally.
> > + *
> > + * XXX Do we need to care about relcacheInitFileInval and
> > + * the other fields added to ReorderBufferChange, or just
> > + * about the message itself?
> > + */
> > + LocalExecuteInvalidationMessage(&change->data.inval.msg);
> > + break;
> >
> > Here, why are we executing messages individually?  Can't we just
> > follow what we do in DecodeCommit which is to record the invalidations
> > in ReorderBufferTXN as we encounter them and then allow them to
> > execute on each REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID.  Is there a
> > reason why we don't do ReorderBufferXidSetCatalogChanges when we
> > receive any invalidation message?
> IMHO, the reason is that in DecodeCommit, we get all the invalidation
> at one time so, at REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID, we don't
> know which invalidation message to execute so for being safe we have
> to execute all.  But, since we are logging all invalidation
> individually, we exactly know at this stage which cache to invalidate.
> So it is better to only invalidate required cache not all.
>

In that case, invalidations can be processed multiple times, the first
time when these individual WAL logs for invalidation are processed and
then later at commit time when we accumulate all invalidation messages
and then execute them for REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID.
Can we avoid to execute invalidations from other places after this
patch which also includes executing them as part of XLOG_INVALIDATIONS
processing?

-- 
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com