Horton

Re: hung backends stuck in spinlock heavy endless loop

Andres Freund <andres@2ndquadrant.com>

View thread View on postgr.es
From: Andres Freund <andres@2ndquadrant.com>
To: Merlin Moncure <mmoncure@gmail.com>
Cc: PostgreSQL-development <pgsql-hackers@postgresql.org>
Date: 2015-01-13T22:33:30Z
Lists: pgsql-hackers 
Hi,

On 2015-01-13 16:29:51 -0600, Merlin Moncure wrote:
> On my workstation today (running vanilla 9.4.0) I was testing some new
> code that does aggressive parallel loading to a couple of tables. It
> ran ok several dozen times and froze up with no external trigger.
> There were at most 8 active backends that were stuck (the loader is
> threaded to a cap) -- each query typically resolves in a few seconds
> but they were hung for 30 minutes+.

Interesting.

> Had to do restart immediate as
> backends were not responding to cancel...but I snapped a 'perf top'
> before I did so.  The results were interesting so I'm posting them
> here.  So far I have not been able to reproduce...FYI

Can you compile postgres with -fno-omit-frame-pointer? Then, when this
happens the next time, you can take a perf record -g, which will tell us
which lock the contention is at.

>  61.03%  postgres                     [.] s_lock
>  13.56%  postgres                     [.] LWLockRelease
>  10.11%  postgres                     [.] LWLockAcquire

That profile looks like it might end up being helped by the lwlock
and/or freelist changes in 9.5.

Greetings,

Andres Freund

-- 
 Andres Freund	                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Commits

  1. Improve hash_create's API for selecting simple-binary-key hash functions.

  2. Fix WAL-logging of B-tree "unlink halfdead page" operation.

  3. Fix backup-block numbering in redo of b-tree split.

  4. Cleanup of new b-tree page deletion code.

  5. Fix bug in the new B-tree incomplete-split code.

  6. Fix bug in new B-tree page deletion code.

  7. Fix rm_desc routine of b-tree page delete records.

  8. Make the handling of interrupted B-tree page splits more robust.

  9. Fix race condition in B-tree page deletion.