Horton

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

Dilip Kumar <dilipbalaut@gmail.com>

View thread
From: Dilip Kumar <dilipbalaut@gmail.com>
To: Amit Kapila <amit.kapila16@gmail.com>
Cc: Erik Rijkers <er@xs4all.nl>, Kuntal Ghosh <kuntalghosh.2007@gmail.com>, Tomas Vondra <tomas.vondra@2ndquadrant.com>, Michael Paquier <michael@paquier.xyz>, Peter Eisentraut <peter.eisentraut@2ndquadrant.com>, PostgreSQL Hackers <pgsql-hackers@postgresql.org>
Date: 2020-06-07T11:37:47Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Tighten the concurrent abort check during decoding.

  2. Improve hash_create()'s API for some added robustness.

  3. Use HASH_BLOBS for xidhash.

  4. Fix initialization of RelationSyncEntry for streaming transactions.

  5. Remove unused function declaration in logicalproto.h.

  6. Add additional tests to test streaming of in-progress transactions.

  7. Fix inline marking introduced in commit 464824323e.

  8. Add support for streaming to built-in logical replication.

  9. Fix the SharedFileSetUnregister API.

  10. Fix comment in procarray.c

  11. Suppress compiler warning in non-cassert builds.

  12. Extend the BufFile interface.

  13. Mark a few logical decoding related variables with PGDLLIMPORT.

  14. Implement streaming mode in ReorderBuffer.

  15. Extend the logical decoding output plugin API with stream methods.

  16. WAL Log invalidations at command end with wal_level=logical.

  17. Immediately WAL-log subtransaction and top-level XID association.

  18. Allow logical replication to transfer data in binary format.

  19. Only superuser can set sslcert/sslkey in postgres_fdw user mappings

  20. Track statistics for spilling of changes from ReorderBuffer.

  21. Add logical_decoding_work_mem to limit ReorderBuffer memory usage.

  22. logical decoding: process ASSIGNMENT during snapshot build

  23. Emit invalidations to standby for transactions without xid.

On Fri, Jun 5, 2020 at 11:37 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Fri, May 29, 2020 at 8:31 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > Apart from this one more fix in 0005,  basically, CheckLiveXid was
> > never reset, so I have fixed that as well.
> >
>
> I have made a number of modifications in the 0001 patch and attached
> is the result.  I have changed/added comments, done some cosmetic
> cleanup, and ran pgindent.  The most notable change is to remove the
> below code change:
> DecodeXactOp()
> {
> ..
> - * However, it's critical to process XLOG_XACT_ASSIGNMENT records even
> + * However, it's critical to process records with subxid assignment even
>   * when the snapshot is being built: it is possible to get later records
>   * that require subxids to be properly assigned.
>   */
>   if (SnapBuildCurrentState(builder) < SNAPBUILD_FULL_SNAPSHOT &&
> - info != XLOG_XACT_ASSIGNMENT)
> + !TransactionIdIsValid(XLogRecGetTopXid(r)))
> ..
> }
>
> I have not only removed the change done by the patch but the check
> related to XLOG_XACT_ASSIGNMENT as well.  That check has been added by
> commit bac2fae05c to ensure that we process XLOG_XACT_ASSIGNMENT even
> if snapshot state is not SNAPBUILD_FULL_SNAPSHOT.  Now, with this
> patch that is not required because we are making the subtransaction
> and top-level transaction much earlier than this.  I have verified
> that it doesn't reopen the bug by running the test provided in the
> original report [1].
>
> Let me know what you think of the changes?  If you find them okay,
> then feel to include them in the next patch-set.
>
> [1] - https://www.postgresql.org/message-id/CAONYFtOv%2BEr1p3WAuwUsy1zsCFrSYvpHLhapC_fMD-zNaRWxYg%40mail.gmail.com

Thanks for the patch, I will review it and include it in my next version.

-- 
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com