Horton
Menu

RE: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5

Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>

View thread
From: "Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com>
To: 'Masahiko Sawada' <sawada.mshk@gmail.com>, Amit Kapila <amit.kapila16@gmail.com>
Cc: Duncan Sands <duncan.sands@deepbluecap.com>, "pgsql-bugs@lists.postgresql.org" <pgsql-bugs@lists.postgresql.org>
Date: 2025-05-28T12:27:51Z
Lists: pgsql-bugs

Attachments

Dear Sawada-san, Amit,

> > It can impact the performance for large transactions with fewer
> > invalidations, especially the ones which has spilled changes because
> > it needs to traverse the entire list of changes again at the end.
> 
> Agreed.
> 
> > The
> > other idea would be to add new member(s) in ReorderBufferTXN to
> > receive distributed invalidations. For adding the new member in
> > ReorderBufferTXN: (a) in HEAD, it should be okay, (b) for
> > backbranches, we may be able to add at the end, but we should check if
> > there are any extensions using sizeof(ReorderBufferTxn) and if they
> > are using what we need to do.
> 
> If we can make sure that that change won't break the existing
> extensions, I think this would be the most reasonable solution.

Based on the discussion, I created PoC for master/PG17. Please see attached.
The basic idea is to introduce the new queue which only contains distributed inval
messages. Contents are consumed at end of transactions. I feel some of codes can
be re-used so that internal functions are introduced. At least, it could pass
regression tests and workloads discussed here.

Best regards,
Hayato Kuroda
FUJITSU LIMITED

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Fix cache-dependent test failures in logical decoding.

  2. Fix re-distributing previously distributed invalidation messages during logical decoding.

  3. Fix data loss in logical replication.