Re: hung backends stuck in spinlock heavy endless loop

On Tue, Jan 13, 2015 at 5:21 PM, Andres Freund <andres@2ndquadrant.com> wrote:
> On 2015-01-13 15:17:15 -0800, Peter Geoghegan wrote:
>> I'm inclined to think that this is a livelock, and so the problem
>> isn't evident from the structure of the B-Tree, but it can't hurt to
>> check.
>
> My guess is rather that it's contention on the freelist lock via
> StrategyGetBuffer's. I've seen profiles like this due to exactly that
> before - and it fits to parallel loading quite well.

I think I've got it to pop again. s_lock is only showing 35%
(increasing very slowly if at all) but performance is mostly halted.
Frame pointer is compiled out.  perf report attached.

merlin

 35.82%  postgres                            [.] s_lock
 23.71%  postgres                            [.] tas
 14.01%  postgres                            [.] tas
  6.82%  postgres                            [.] spin_delay
  5.93%  postgres                            [.] LWLockRelease
  4.36%  postgres                            [.] LWLockAcquireCommon
  2.34%  postgres                            [.] FunctionCall2Coll
  1.79%  postgres                            [.] _bt_compare
  0.80%  postgres                            [.] LockBuffer
  0.52%  postgres                            [.] btoidcmp
  0.49%  postgres                            [.] ReleaseAndReadBuffer
  0.47%  postgres                            [.] _bt_moveright
  0.36%  postgres                            [.] _bt_checkpage
  0.36%  postgres                            [.] _bt_relandgetbuf
  0.20%  perf                                [.] 0x000000000004312a
  0.19%  postgres                            [.] LWLockAcquire
  0.13%  libv8.so                            [.] 0x00000000001bbbe0
  0.11%  libc-2.17.so                        [.] 0x0000000000151134
  0.09%  libwebkit.so                        [.] 0x000000000106ccb7
  0.05%  libgdk_pixbuf-2.0.so.0.2800.1       [.] 0x00000000000139c7
  0.04%  Xorg                                [.] 0x00000000000efb00
  0.03%  libglib-2.0.so.0.3800.1             [.] 0x00000000000876a2
  0.03%  [kernel]                            [k] __ticket_spin_lock
  0.02%  perf-12966.map                      [.] 0x00000625db944621
  0.02%  libskia.so                          [.] 0x000000000021d15f
  0.02%  libcairo.so.2.11200.16              [.] 0x000000000002440b
  0.02%  libpthread-2.17.so                  [.]
__pthread_mutex_unlock_usercnt
  0.02%  [kernel]                            [k]
copy_user_enhanced_fast_string
  0.02%  radeon_drv.so                       [.] 0x0000000000045002
  0.02%  libpthread-2.17.so                  [.] pthread_mutex_lock
  0.02%  [kernel]                            [k] fget_light
  0.01%  [kernel]                            [k] __schedule
  0.01%  libexa.so                           [.] 0x0000000000006e1d
  0.01%  libgdk-x11-2.0.so.0.2400.20         [.] 0x000000000002f0b0
  0.01%  libvte.so.9.2800.2                  [.] 0x0000000000039028
  0.01%  [radeon]                            [k] r100_mm_rreg
  0.01%  [kernel]                            [k] xhci_irq
  0.01%  [kernel]                            [k] native_write_msr_safe
  0.01%  [kernel]                            [k]
update_cfs_rq_blocked_load
  0.01%  libglib-2.0.so.0.3800.1             [.] g_hash_table_lookup
  0.01%  libgobject-2.0.so.0.3800.1          [.]
g_type_check_instance_is_a
Press '?' for help on key bindings