Horton

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

Amit Kapila <amit.kapila16@gmail.com>

View thread
From: Amit Kapila <amit.kapila16@gmail.com>
To: Dilip Kumar <dilipbalaut@gmail.com>
Cc: Erik Rijkers <er@xs4all.nl>, Kuntal Ghosh <kuntalghosh.2007@gmail.com>, Tomas Vondra <tomas.vondra@2ndquadrant.com>, Michael Paquier <michael@paquier.xyz>, Peter Eisentraut <peter.eisentraut@2ndquadrant.com>, PostgreSQL Hackers <pgsql-hackers@postgresql.org>
Date: 2020-07-09T11:15:22Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Tighten the concurrent abort check during decoding.

  2. Improve hash_create()'s API for some added robustness.

  3. Use HASH_BLOBS for xidhash.

  4. Fix initialization of RelationSyncEntry for streaming transactions.

  5. Remove unused function declaration in logicalproto.h.

  6. Add additional tests to test streaming of in-progress transactions.

  7. Fix inline marking introduced in commit 464824323e.

  8. Add support for streaming to built-in logical replication.

  9. Fix the SharedFileSetUnregister API.

  10. Fix comment in procarray.c

  11. Suppress compiler warning in non-cassert builds.

  12. Extend the BufFile interface.

  13. Mark a few logical decoding related variables with PGDLLIMPORT.

  14. Implement streaming mode in ReorderBuffer.

  15. Extend the logical decoding output plugin API with stream methods.

  16. WAL Log invalidations at command end with wal_level=logical.

  17. Immediately WAL-log subtransaction and top-level XID association.

  18. Allow logical replication to transfer data in binary format.

  19. Only superuser can set sslcert/sslkey in postgres_fdw user mappings

  20. Track statistics for spilling of changes from ReorderBuffer.

  21. Add logical_decoding_work_mem to limit ReorderBuffer memory usage.

  22. logical decoding: process ASSIGNMENT during snapshot build

  23. Emit invalidations to standby for transactions without xid.

On Thu, Jul 9, 2020 at 2:41 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Wed, Jul 8, 2020 at 3:32 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
>
> Only replying to the replication origin point, other comment looks
> fine to me so I will work on those.
>
> > Replication Origins
> > ------------------------------
> > I think we also need to conclude on origins related discussion [1].
> > As far as I can see, the origin_id can be sent with the first startup
> > message. The origin_lsn and origin_commit can be sent with the last
> > start of streaming commit if we want but not sure if that is of use.
> > If we need to send it earlier then we need to record it with other WAL
> > records.  The point is that those are set with
> > pg_replication_origin_xact_setup but not sure how and when that
> > function is called.
>
> pg_replication_origin_xact_setup is exposed function so this will
> allow a user to set an origin for their session so that all the
> operation done from that session will be marked by that origin id.
>

Hmm, I think that can be done by pg_replication_origin_session_setup.

> And the clear use case for this is to avoid sending such transactions
> by suing FilterByOrigin.   But I am not sure about the point that we
> discussed at [1] that what is the use of the origin and origin_lsn we
> send at pgoutput_begin_txn.
>

I could see the use of 'origin' with FilterByOrigin but not sure how
origin_lsn can be used?

>  The other alternative is that we can ignore that
> > for now and once the usage is clear we can enhance it.  What do you
> > think?
>
> That seems like a sensible option to me.
>

I have responded to that another thread.  Let us see if someone
responds to it.  Feel free to add if you have some points related to
that thread.

-- 
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com