RE: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
From: "Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com>
To: 'Amit Kapila' <amit.kapila16@gmail.com>, Duncan Sands <duncan.sands@deepbluecap.com>
Cc: "pgsql-bugs@lists.postgresql.org" <pgsql-bugs@lists.postgresql.org>, Sawada Masahiko <sawada.mshk@gmail.com>
Date: 2025-05-21T11:48:12Z
Lists: pgsql-bugs
Attachments
- PG17-0001-Avoid-distributing-invalidation-messages-several-tim.patch (application/octet-stream) patch 0001
Dear hackers, > I think the problem here is that when we are distributing > invalidations to a concurrent transaction, in addition to queuing the > invalidations as a change, we also copy the distributed invalidations > along with the original transaction's invalidations via repalloc in > ReorderBufferAddInvalidations. So, when there are many in-progress > transactions, each would try to copy all its accumulated invalidations > to the remaining in-progress transactions. This could lead to such an > increase in allocation request size. However, after queuing the > change, we don't need to copy it along with the original transaction's > invalidations. This is because the copy is only required when we don't > process any changes in cases like ReorderBufferForget(). I have > analyzed all such cases, and my analysis is as follows: Based on the analysis, I created a PoC which avoids the repalloc(). Invalidation messages distributed by SnapBuildDistributeSnapshotAndInval() are skipped to add in the list, just queued - repalloc can be skipped. Also, the function distributes messages only in the list, so received messages won't be sent again. Now a patch for PG17 is created for testing purpose. Duncan, can you apply this and confirms whether the issue can be solved? Best regards, Hayato Kuroda FUJITSU LIMITED
Commits
Same data as JSON:
GET /api/v1/messages/:b64id/commits
the thread's linked commits as JSON, with link sources.
API reference →
-
Fix cache-dependent test failures in logical decoding.
- 87819f766f37 13.22 landed
-
Fix re-distributing previously distributed invalidation messages during logical decoding.
- d87d07b7ad3b 18.0 landed
- 45c357e0e85d 17.6 landed
- b2ae077205e1 16.10 landed
- fc0fb77c550f 15.14 landed
- 983b3636259b 14.19 landed
- 1230be12f086 13.22 landed
-
Fix data loss in logical replication.
- 247ee94150b6 13.21 cited
- 4909b38af034 18.0 cited