Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

Dilip Kumar <dilipbalaut@gmail.com>

From: Dilip Kumar <dilipbalaut@gmail.com>

To: Amit Kapila <amit.kapila16@gmail.com>

Cc: Jeff Janes <jeff.janes@gmail.com>, Erik Rijkers <er@xs4all.nl>, Kuntal Ghosh <kuntalghosh.2007@gmail.com>, Tomas Vondra <tomas.vondra@2ndquadrant.com>, Michael Paquier <michael@paquier.xyz>, Peter Eisentraut <peter.eisentraut@2ndquadrant.com>, PostgreSQL Hackers <pgsql-hackers@postgresql.org>

Date: 2020-08-28T08:48:05Z

Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

Tighten the concurrent abort check during decoding.
- 2ce353fc1902 14.0 landed
Improve hash_create()'s API for some added robustness.
- b3817f5f7746 14.0 landed
Use HASH_BLOBS for xidhash.
- a1b8aa1e4eec 14.0 landed
Fix initialization of RelationSyncEntry for streaming transactions.
- 69bd60672af6 14.0 landed
Remove unused function declaration in logicalproto.h.
- ddd5f6d2609b 14.0 landed
Add additional tests to test streaming of in-progress transactions.
- 58b5ae9d62bd 14.0 landed
Fix inline marking introduced in commit 464824323e.
- ac15b499f7f9 14.0 landed
Add support for streaming to built-in logical replication.
- 464824323e57 14.0 landed
Fix the SharedFileSetUnregister API.
- 4ab77697f67a 14.0 landed
Fix comment in procarray.c
- 77c7267c37f7 14.0 cited
Suppress compiler warning in non-cassert builds.
- e942af7b8261 14.0 cited
Extend the BufFile interface.
- 808e13b282ef 14.0 landed
Mark a few logical decoding related variables with PGDLLIMPORT.
- b48cac3b10a0 14.0 landed
Implement streaming mode in ReorderBuffer.
- 7259736a6e5b 14.0 landed
Extend the logical decoding output plugin API with stream methods.
- 45fdc9738b36 14.0 landed
WAL Log invalidations at command end with wal_level=logical.
- c55040ccd017 14.0 landed
Immediately WAL-log subtransaction and top-level XID association.
- 0bead9af484c 14.0 landed
Allow logical replication to transfer data in binary format.
- 9de77b545313 14.0 cited
Only superuser can set sslcert/sslkey in postgres_fdw user mappings
- cebf9d6e6ee1 13.0 cited
Track statistics for spilling of changes from ReorderBuffer.
- 9290ad198b15 13.0 landed
Add logical_decoding_work_mem to limit ReorderBuffer memory usage.
- cec2edfa7859 13.0 landed
logical decoding: process ASSIGNMENT during snapshot build
- bac2fae05c77 13.0 cited
Emit invalidations to standby for transactions without xid.
- c6ff84b06a68 9.6.0 cited

Attachments

tap_test_for_out_of_order_subxact_abort.patch (application/octet-stream) patch

On Thu, Aug 27, 2020 at 11:16 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, Aug 26, 2020 at 11:22 PM Jeff Janes <jeff.janes@gmail.com> wrote:
> >
> >
> > On Tue, Aug 25, 2020 at 8:58 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> >>
> >>  I am planning
> >> to push the first patch (v53-0001-Extend-the-BufFile-interface) in
> >> this series tomorrow unless you have any comments on the same.
> >
> >
> >
> > I'm getting compiler warnings now, src/backend/storage/file/sharedfileset.c line 288 needs to be:
> >
> > bool        found PG_USED_FOR_ASSERTS_ONLY = false;
> >
>
> Thanks for the report. Tom Lane has already fixed this [1].
>
> [1] - https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=e942af7b8261cd8070d0eeaf518dbc1a664859fd

As discussed, I have added a another test case for covering the out of
order subtransaction rollback scenario.

-- 
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com