Horton

Re: hung backends stuck in spinlock heavy endless loop

Merlin Moncure <mmoncure@gmail.com>

View thread View on postgr.es
From: Merlin Moncure <mmoncure@gmail.com>
To: Andres Freund <andres@2ndquadrant.com>
Cc: Tom Lane <tgl@sss.pgh.pa.us>, Peter Geoghegan <pg@heroku.com>, PostgreSQL-development <pgsql-hackers@postgresql.org>
Date: 2015-01-14T15:47:19Z
Lists: pgsql-hackers 
On Wed, Jan 14, 2015 at 9:30 AM, Andres Freund <andres@2ndquadrant.com> wrote:
> If you gdb in, and type 'fin' a couple times, to wait till the function
> finishes, is there actually any progress? I'm wondering whether it's
> just many catalog accesses + contention, or some other
> problem. Alternatively set a breakpoint on ScanPgRelation() or so and
> see how often it's hit.

well, i restarted the database, forgetting my looper was running which
immediately spun up and it got stuck again with a similar profile
(lots of cpu in spinlock):

Samples: 3K of event 'cycles', Event count (approx.): 2695723228
 31.16%  postgres            [.] s_lock
 22.32%  postgres            [.] tas
 12.13%  postgres            [.] tas
  5.93%  postgres            [.] spin_delay
  5.69%  postgres            [.] LWLockRelease
  3.75%  postgres            [.] LWLockAcquireCommon
  3.61%  perf                [.] 0x00000000000526c4
  2.51%  postgres            [.] FunctionCall2Coll
  1.48%  libc-2.17.so        [.] 0x000000000016a132

> If you gdb in, and type 'fin' a couple times,
(gdb) fin
Run till exit from #0  0x00007ff4c63f7a97 in semop () from
/lib/x86_64-linux-gnu/libc.so.6
0x00000000006de073 in PGSemaphoreLock ()
(gdb) fin
Run till exit from #0  0x00000000006de073 in PGSemaphoreLock ()

It returned once.  Second time, it didn't at least so far (minute or so).

>> I can send the code off-list if you guys think it'd help.
>
> Might be interesting.

sent (off-list).

merlin

Commits

  1. Improve hash_create's API for selecting simple-binary-key hash functions.

  2. Fix WAL-logging of B-tree "unlink halfdead page" operation.

  3. Fix backup-block numbering in redo of b-tree split.

  4. Cleanup of new b-tree page deletion code.

  5. Fix bug in the new B-tree incomplete-split code.

  6. Fix bug in new B-tree page deletion code.

  7. Fix rm_desc routine of b-tree page delete records.

  8. Make the handling of interrupted B-tree page splits more robust.

  9. Fix race condition in B-tree page deletion.