Thread

MarkBufferDirty Assert held LW_EXCLUSIVE lock fail when ginFinishSplit

feichanghong <feichanghong@qq.com> — 2024-01-22T08:04:01Z

Hi,


Environmental information is as follows:
PostgreSQL Version: 17, branch: master, commit: abb0b4fc
Operating system: centos 7, Kernel version: 5.10.13
Client: psql


After the instance crash recovery, I encountered an assertion failure with
MarkBufferDirty when inserting data into a gin index. The specific stack trace
is as follows:
```
#0  0x00007feb10d19277 in raise () from /lib64/libc.so.6
#1  0x00007feb10d1a968 in abort () from /lib64/libc.so.6
#2  0x00000000009a4184 in ExceptionalCondition (
    conditionName=conditionName@entry=0xb44e38 
    "LWLockHeldByMeInMode(BufferDescriptorGetContentLock(bufHdr), LW_EXCLUSIVE)",
    fileName=fileName@entry=0xb45594 "bufmgr.c",
    lineNumber=lineNumber@entry=2216) at assert.c:66
#3  0x0000000000840da0 in MarkBufferDirty (buffer=buffer@entry=865) at bufmgr.c:2216
#4  0x000000000052f953 in ginPlaceToPage (btree=0x7fff6a3316b0, stack=0x1e5cbd0,
    insertdata=0x1e31ce8, updateblkno=626, childbuf=865, buildStats=0x0)
    at ginbtree.c:407
#5  0x000000000053084b in ginFinishSplit (btree=0x7fff6a3316b0, stack=0x1e5bac8,
    freestack=false, buildStats=0x0) at ginbtree.c:738
#6  0x000000000053103f in ginFindLeafPage (btree=btree@entry=0x7fff6a3316b0,
    searchMode=searchMode@entry=false, rootConflictCheck=rootConflictCheck@entry=false)
    at ginbtree.c:111
#7  0x000000000053b850 in ginEntryInsert (ginstate=ginstate@entry=0x1e1f708, attnum=1,
    key=31597248, category=0 '\000', items=0x1e222d8, nitem=1, buildStats=0x0)
    at gininsert.c:195
#8  0x00000000005373f2 in ginInsertCleanup (ginstate=ginstate@entry=0x1e1f708,
    full_clean=full_clean@entry=false, fill_fsm=fill_fsm@entry=true,
    forceCleanup=forceCleanup@entry=false, stats=stats@entry=0x0) at ginfast.c:930
#9  0x0000000000537f2a in ginHeapTupleFastInsert (ginstate=ginstate@entry=0x1e1f708,
    collector=collector@entry=0x7fff6a331ab0) at ginfast.c:471
```


In the function ginFindLeafPage, if we encounter a page marked with
GIN_INCOMPLETE_SPLIT, we call ginFinishSplit to finish the incomplete split and
remove the GIN_INCOMPLETE_SPLIT flag from that page. ginFinishSplit requires
that "On entry, stack->buffer is exclusively locked," as explained in its
comments.


The function ginFindLeafPage determines the type of lock held by ginTraverseLock:
leaf pages hold GIN_EXCLUSIVE, and internal page hold GIN_SHARE. If ginFindLeafPage
traverses to an internal page with an incomplete split, it will only hold a
GIN_SHARE lock, which does not meet the requirements of ginFinishSplit. If a
crash occurs when an internal page is split, but no downlink is inserted in the
parent page, this problem may occur.


I injected the following code into the ginbtree.c file to trigger a crash
during the splitting of an internal page:
```
--- a/src/backend/access/gin/ginbtree.c
+++ b/src/backend/access/gin/ginbtree.c
@@ -621,6 +621,12 @@ ginPlaceToPage(GinBtree btree, GinBtreeStack *stack,
                 PageSetLSN(BufferGetPage(lbuffer), recptr);
             if (BufferIsValid(childbuf))
                 PageSetLSN(childpage, recptr);
+
+            if (stack->parent != NULL && !GinPageIsLeaf(newlpage))
+            {
+                XLogFlush(recptr);
+                elog(PANIC, "internal page split for block: %u", stack->blkno);
+            }
         }
         END_CRIT_SECTION();
```


With the modifications above, the problem can be reproduced with the following
steps:
```
alter system set autovacuum to off;
alter system set gin_pending_list_limit to 64;
select pg_reload_conf();
create table t (a text);
create extension btree_gin;
create index on t USING gin (a);
-- PANIC crash
insert into t select generate_series(1, 100000);


-- Assert fail
insert into t select 1;
```


I reviewed all places where ginFinishSplit is called, and only in two instances
in ginFindLeafPage might it be possible to not hold an exclusive lock on
stack->buffer.


The patch I provided in the attachment has been verified to fix the issue
mentioned above.
However, it may not be perfect: passing GIN_EXCLUSIVE to ginStepRight might
affect performance. Do we need to add a parameter to ginStepRight, and within
the function ginStepRight, elevate the lock level for an incomplete split page?


Looking forward to suggestions from developers!




Best Regards,
Fei Changhong
Alibaba Cloud Computing Ltd.