Horton

Thread

Commits

  1. Fix race condition in COMMIT PREPARED causing orphaned 2PC files

  1. Fix orphaned 2pc file which may casue instance restart failed

    ChengWen Wu <drec.wu@foxmail.com> — 2024-09-08T05:01:37Z

    Hi all,&nbsp; &nbsp; &nbsp;I found that there is a race condition between two global transaction, which may cause instance
restart failed with error 'could not access status of transaction xxx","Could not open file ""pg_xact/xxx"": No such file or directory'.


&nbsp; &nbsp; &nbsp;The scenery to reproduce the problem is:
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;1. gxact1 is doing `FinishPreparedTransaction` and checkpoint
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;is issued, so gxact1 will generate a 2pc file.
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;2. then gxact1 was removed from `TwoPhaseState-&gt;prepXacts` and
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;its state memory was returned to freelist.
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;3. but just before gxact1 remove its 2pc file, gxact2 is issued,
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;gxact2 will reuse the same state memory of gxact1 and will
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;reset `gxact-&gt;ondisk` to false.
&nbsp; &nbsp; &nbsp; &nbsp; 4. gxact1 continue and found that `gxact-&gt;ondisk` is false, it won't
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; remove its 2pc file. This file is orphaned.


&nbsp; &nbsp; If gxact1's local xid is not frozen, the startup process will remove
the orphaned 2pc file. However, if the xid's corresponding clog file is
truncated by `vacuum`, the startup process will raise error 'could not
access status of transaction xxx', due to it could not found the
transaction's status file in dir `pg_xact`.



&nbsp; &nbsp;The potential fix is attached.

  2. Re: Fix orphaned 2pc file which may casue instance restart failed

    Michael Paquier <michael@paquier.xyz> — 2024-09-11T09:21:37Z

    On Sun, Sep 08, 2024 at 01:01:37PM +0800, 清浅 wrote:
> Hi all,  I found that there is a race condition
> between two global transaction, which may cause instance restart
> failed with error 'could not access status of transaction
> xxx","Could not open file ""pg_xact/xxx"": No such file or
> directory'.
> 
> 
>   The scenery to reproduce the problem is:
>     1. gxact1 is doing `FinishPreparedTransaction` and checkpoint
>       is issued, so gxact1 will generate a 2pc file.
>     2. then gxact1 was removed from `TwoPhaseState-&gt;prepXacts` and
>       its state memory was returned to freelist.
>     3. but just before gxact1 remove its 2pc file, gxact2 is issued,
>       gxact2 will reuse the same state memory of gxact1 and will
>       reset `gxact-&gt;ondisk` to false.
>     4. gxact1 continue and found that `gxact-&gt;ondisk` is false, it won't
>       remove its 2pc file. This file is orphaned.
> 
>   If gxact1's local xid is not frozen, the startup process will remove
> the orphaned 2pc file. However, if the xid's corresponding clog file is
> truncated by `vacuum`, the startup process will raise error 'could not
> access status of transaction xxx', due to it could not found the
> transaction's status file in dir `pg_xact`.

Hmm.  I've not seen that in the field.  Let me check that..
--
Michael

  3. Re: Fix orphaned 2pc file which may casue instance restart failed

    Michael Paquier <michael@paquier.xyz> — 2024-10-01T06:49:53Z

    On Wed, Sep 11, 2024 at 06:21:37PM +0900, Michael Paquier wrote:
>>   If gxact1's local xid is not frozen, the startup process will remove
>> the orphaned 2pc file. However, if the xid's corresponding clog file is
>> truncated by `vacuum`, the startup process will raise error 'could not
>> access status of transaction xxx', due to it could not found the
>> transaction's status file in dir `pg_xact`.
> 
> Hmm.  I've not seen that in the field.  Let me check that..

Ah, yes, that's clearly broken.  The origin of the problem was
effe7d9552dd, which was from me, so backpatched all the way down after
adding a comment explaining the reason why we'd better read the value
before the 2PC state lock is released.
--
Michael