035_standby_logical_decoding unbounded hang
Noah Misch <noah@leadboat.com>
From: Noah Misch <noah@leadboat.com>
To: pgsql-hackers@lists.postgresql.org
Cc: Tom Lane <tgl@sss.pgh.pa.us>
Date: 2024-02-11T01:02:27Z
Lists: pgsql-hackers
Attachments
- repro-standby-slot-test-race-v1.patch (text/plain) patch v1
- standby-slot-test-1-timeout-v1.patch (text/plain) patch v1
- standby-slot-test-2-race-v1.patch (text/plain) patch v1
Coincidentally, one of my buildfarm animals hanged several weeks in a different test, 035_standby_logical_decoding.pl. A LOG_SNAPSHOT_INTERVAL_MS reduction was part of making it reproducible: On Fri, Feb 02, 2024 at 04:01:45PM -0800, Noah Misch wrote: > On Fri, Feb 02, 2024 at 02:30:03PM -0800, Noah Misch wrote: > > On Fri, Feb 02, 2024 at 05:07:14PM -0500, Tom Lane wrote: > > > If you look at the buildfarm's failures page and filter down to > > > just subscriptionCheck failures, what you find is that all of the > > > last 6 such failures are in 031_column_list.pl: > > https://www.postgresql.org/message-id/flat/16d6d9cc-f97d-0b34-be65-425183ed3721%40gmail.com > > reported a replacement BgWriterDelay value reproducing it. > > Correction: the recipe changes LOG_SNAPSHOT_INTERVAL_MS in addition to > BgWriterDelay. I'm reusing this thread just in case there's overlap with the 031_column_list.pl cause and fix. The 035_standby_logical_decoding.pl hang is a race condition arising from an event sequence like this: - Test script sends CREATE SUBSCRIPTION to subscriber, which loses the CPU. - Test script calls pg_log_standby_snapshot() on primary. Emits XLOG_RUNNING_XACTS. - checkpoint_timeout makes a primary checkpoint finish. Emits XLOG_RUNNING_XACTS. - bgwriter executes LOG_SNAPSHOT_INTERVAL_MS logic. Emits XLOG_RUNNING_XACTS. - CREATE SUBSCRIPTION wakes up and sends CREATE_REPLICATION_SLOT to standby. Other test code already has a solution for this, so the attached patches add a timeout and copy the existing solution. I'm also attaching the hack that makes it 100% reproducible.
Commits
-
Back-patch test modifications that were done as part of b6df0798a5.
- 22cecaddf7c8 16.3 landed
- b5abeb751419 15.7 landed
-
Fix the intermittent buildfarm failures in 031_column_list.
- b6df0798a5e2 17.0 landed
-
Fix test race between primary XLOG_RUNNING_XACTS and standby logical slot.
- c59a97313ba3 16.3 landed
- 0e162810df76 17.0 landed
-
Bound waits in 035_standby_logical_decoding.pl.
- f024746484b9 16.3 landed
- 4791f87f34bd 17.0 landed