Horton

Re: hung backends stuck in spinlock heavy endless loop

Merlin Moncure <mmoncure@gmail.com>

View thread View on postgr.es
From: Merlin Moncure <mmoncure@gmail.com>
To: Peter Geoghegan <pg@heroku.com>
Cc: Andres Freund <andres@2ndquadrant.com>, Tom Lane <tgl@sss.pgh.pa.us>, PostgreSQL-development <pgsql-hackers@postgresql.org>
Date: 2015-01-15T00:26:08Z
Lists: pgsql-hackers 
On Wed, Jan 14, 2015 at 5:39 PM, Peter Geoghegan <pg@heroku.com> wrote:
> On Wed, Jan 14, 2015 at 3:38 PM, Merlin Moncure <mmoncure@gmail.com> wrote:
>> (gdb) print  BufferGetBlockNumber(buf)
>> $15 = 9
>>
>> ..and it stays 9, continuing several times having set breakpoint.
>
>
> And the index involved? I'm pretty sure that this in an internal page, no?

The index is the oid index on pg_class.  Some more info:

*) temp table churn is fairly high.  Several dozen get spawned and
destroted at the start of a replication run, all at once, due to some
dodgy coding via dblink.  During the replication run, the temp table
churn rate drops.

*)  running btreecheck, I see:
cds2=# select bt_index_verify('pg_class_oid_index');
NOTICE:  page 7 of index "pg_class_oid_index" is deleted
NOTICE:  page 10 of index "pg_class_oid_index" is deleted
NOTICE:  page 12 of index "pg_class_oid_index" is deleted
 bt_index_verify
─────────────────


cds2=# select bt_leftright_verify('pg_class_oid_index');
WARNING:  left link/right link pair don't comport at level 0, block 9,
last: 2, current left: 4
WARNING:  left link/right link pair don't comport at level 0, block 9,
last: 9, current left: 4
WARNING:  left link/right link pair don't comport at level 0, block 9,
last: 9, current left: 4
WARNING:  left link/right link pair don't comport at level 0, block 9,
last: 9, current left: 4
WARNING:  left link/right link pair don't comport at level 0, block 9,
last: 9, current left: 4
[repeat infinity until cancel]

which looks like the index is corrupted?  ISTM _bt_moveright is
hanging because it's trying to move from block 9 to block 9 and so
loops forever.

merlin

Commits

  1. Improve hash_create's API for selecting simple-binary-key hash functions.

  2. Fix WAL-logging of B-tree "unlink halfdead page" operation.

  3. Fix backup-block numbering in redo of b-tree split.

  4. Cleanup of new b-tree page deletion code.

  5. Fix bug in the new B-tree incomplete-split code.

  6. Fix bug in new B-tree page deletion code.

  7. Fix rm_desc routine of b-tree page delete records.

  8. Make the handling of interrupted B-tree page splits more robust.

  9. Fix race condition in B-tree page deletion.