Horton

Re: hung backends stuck in spinlock heavy endless loop

Andres Freund <andres@2ndquadrant.com>

View thread View on postgr.es
From: Andres Freund <andres@2ndquadrant.com>
To: Merlin Moncure <mmoncure@gmail.com>
Cc: Tom Lane <tgl@sss.pgh.pa.us>, Peter Geoghegan <pg@heroku.com>, PostgreSQL-development <pgsql-hackers@postgresql.org>
Date: 2015-01-14T15:49:59Z
Lists: pgsql-hackers 
On 2015-01-14 09:47:19 -0600, Merlin Moncure wrote:
> On Wed, Jan 14, 2015 at 9:30 AM, Andres Freund <andres@2ndquadrant.com> wrote:
> > If you gdb in, and type 'fin' a couple times, to wait till the function
> > finishes, is there actually any progress? I'm wondering whether it's
> > just many catalog accesses + contention, or some other
> > problem. Alternatively set a breakpoint on ScanPgRelation() or so and
> > see how often it's hit.
> 
> well, i restarted the database, forgetting my looper was running which
> immediately spun up and it got stuck again with a similar profile
> (lots of cpu in spinlock):
> 
> Samples: 3K of event 'cycles', Event count (approx.): 2695723228
>  31.16%  postgres            [.] s_lock
>  22.32%  postgres            [.] tas
>  12.13%  postgres            [.] tas
>   5.93%  postgres            [.] spin_delay
>   5.69%  postgres            [.] LWLockRelease
>   3.75%  postgres            [.] LWLockAcquireCommon
>   3.61%  perf                [.] 0x00000000000526c4
>   2.51%  postgres            [.] FunctionCall2Coll
>   1.48%  libc-2.17.so        [.] 0x000000000016a132
> 
> > If you gdb in, and type 'fin' a couple times,
> (gdb) fin
> Run till exit from #0  0x00007ff4c63f7a97 in semop () from
> /lib/x86_64-linux-gnu/libc.so.6
> 0x00000000006de073 in PGSemaphoreLock ()
> (gdb) fin
> Run till exit from #0  0x00000000006de073 in PGSemaphoreLock ()
> 
> It returned once.  Second time, it didn't at least so far (minute or so).

Hm, that's autovac though, not the normal user backends that actually do
stuff, right? If you could additionally check those, it'd be great.

Greetings,

Andres Freund

-- 
 Andres Freund	                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Commits

  1. Improve hash_create's API for selecting simple-binary-key hash functions.

  2. Fix WAL-logging of B-tree "unlink halfdead page" operation.

  3. Fix backup-block numbering in redo of b-tree split.

  4. Cleanup of new b-tree page deletion code.

  5. Fix bug in the new B-tree incomplete-split code.

  6. Fix bug in new B-tree page deletion code.

  7. Fix rm_desc routine of b-tree page delete records.

  8. Make the handling of interrupted B-tree page splits more robust.

  9. Fix race condition in B-tree page deletion.