FSM Corruption (was: Could not read block at end of the relation)

Ronan Dunklau <ronan.dunklau@aiven.io>

From: Ronan Dunklau <ronan.dunklau@aiven.io>
To: pgsql-bugs <pgsql-bugs@lists.postgresql.org>
Date: 2024-03-01T08:56:51Z
Lists: pgsql-bugs

Attachments

Le mardi 27 février 2024, 11:34:14 CET Ronan Dunklau a écrit :
> I suspected the FSM could be corrupted in some way but taking a look at it
> just after the errors have been triggered, the offending (non
> existing)blocks are just not present in the FSM either.

I think I may have missed something on my first look. On other affected 
clusters, the FSM is definitely corrupted.  So it looks like we have an FSM 
corruption bug on our hands.

The occurence of this bug happening makes it hard to reproduce,  but it's 
definitely frequent enough we witnessed it on a dozen PostgreSQL clusters. 

In our case, we need to repair the FSM. The instructions on the wiki do work, 
but maybe we should add something like the attached patch (modeled after the 
same feature in pg_visibility) to make it possible to repair the FSM 
corruption online. What do you think about it ?

The investigation of the corruption is still ongoing.


Best regards,

--
Ronan Dunklau

Commits

  1. freespace: Don't return blocks past the end of the main fork.

  2. Fix WAL-logging of FSM and VM truncation.

  3. Extend relations multiple blocks at a time to improve scalability.