Horton

Re: Deadlock in XLogInsert at AIX

Heikki Linnakangas <hlinnaka@iki.fi>

View thread View on postgr.es
From: Heikki Linnakangas <hlinnaka@iki.fi>
To: Konstantin Knizhnik <k.knizhnik@postgrespro.ru>, PostgreSQL Hackers <pgsql-hackers@postgresql.org>
Date: 2017-02-01T08:17:23Z
Lists: pgsql-hackers 
On 01/31/2017 05:03 PM, Konstantin Knizhnik wrote:
> One more assertion failure:
>
>
> ExceptionalCondition(conditionName = "!(OldPageRqstPtr <=
> XLogCtl->InitializedUpTo)", errorType = "FailedAssertion", fileName =
> "xlog.c", lineNumber = 1887), line 54 in "assert.c"
>
> (dbx) p OldPageRqstPtr
> 153551667200
> (dbx) p XLogCtl->InitializedUpTo
> 153551667200
> (dbx) p InitializedUpTo
> 153551659008
>
> I slightly modify xlog.c code - store value of XLogCtl->InitializedUpTo
> in local variable:
>
>
>   1870         LWLockAcquire(WALBufMappingLock, LW_EXCLUSIVE);
>   1871
>   1872         /*
>   1873          * Now that we have the lock, check if someone
> initialized the page
>   1874          * already.
>   1875          */
>   1876         while (upto >= XLogCtl->InitializedUpTo || opportunistic)
>   1877         {
>   1878                 XLogRecPtr InitializedUpTo =
> XLogCtl->InitializedUpTo;
>   1879                 nextidx = XLogRecPtrToBufIdx(InitializedUpTo);
>   1880
>   1881                 /*
>   1882                  * Get ending-offset of the buffer page we need
> to replace (this may
>   1883                  * be zero if the buffer hasn't been used yet).
> Fall through if it's
>   1884                  * already written out.
>   1885                  */
>   1886                 OldPageRqstPtr = XLogCtl->xlblocks[nextidx];
>   1887                 Assert(OldPageRqstPtr <= XLogCtl->InitializedUpTo);
>
>
> And, as you can see,  XLogCtl->InitializedUpTo is not equal to saved
> value InitializedUpTo.
> But we are under exclusive WALBufMappingLock and InitializedUpTo is
> updated only under this lock.
> So it means that LW-locks doesn't work!

Yeah, so it seems. XLogCtl->InitializeUpTo is quite clearly protected by 
the WALBufMappingLock. All references to it (after StartupXLog) happen 
while holding the lock.

Can you get the assembly output of the AdvanceXLInsertBuffer() function? 
I wonder if the compiler is rearranging things so that 
XLogCtl->InitializedUpTo is fetched before the LWLockAcquire call. Or 
should there be a memory barrier instruction somewhere in LWLockAcquire?

- Heikki

Commits

  1. For all ppc compilers, implement compare_exchange and fetch_add with asm.

  2. For PowerPC instruction "addi", use constraint "b".

  3. For all ppc compilers, implement pg_atomic_fetch_add_ with inline asm.

  4. Replace xlc __fetch_and_add() with inline asm.

  5. Test pg_atomic_fetch_add_ with variable addend and 16-bit edge cases.