Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic
Justin Pryzby <pryzby@telsasoft.com>
From: Justin Pryzby <pryzby@telsasoft.com>
To: Peter Geoghegan <pg@bowt.ie>
Cc: pgsql-hackers@lists.postgresql.org, Matthias van de Meent <boekewurm+postgres@gmail.com>, Masahiko Sawada <sawada.mshk@gmail.com>, Andres Freund <andres@anarazel.de>, Tom Lane <tgl@sss.pgh.pa.us>
Date: 2021-06-08T11:03:29Z
Lists: pgsql-hackers
Attachments
- dump_block.page.gz (application/gzip)
On Sun, Jun 06, 2021 at 11:00:38AM -0700, Peter Geoghegan wrote:
> On Sun, Jun 6, 2021 at 9:35 AM Justin Pryzby <pryzby@telsasoft.com> wrote:
> > I'll leave the instance running for a little bit before restarting (or kill-9)
> > in case someone requests more info.
>
> How about dumping the page image out, and sharing it with the list?
> This procedure should work fine from gdb:
>
> https://wiki.postgresql.org/wiki/Getting_a_stack_trace_of_a_running_PostgreSQL_backend_on_Linux/BSD#Dumping_a_page_image_from_within_GDB
> I suggest that you dump the "page" pointer inside lazy_scan_prune(). I
> imagine that you have the instance already stuck in an infinite loop,
> so what we'll probably see from the page image is the page after the
> first prune and another no-progress prune.
The cluster was again rejecting with "too many clients already".
I was able to open a shell this time, but it immediately froze when I tried to
tab complete "pg_stat_acti"...
I was able to dump the page image, though - attached. I can send you its
"data" privately, if desirable. I'll also try to step through this.
postgres=# SELECT lp, lp_off, lp_flags, lp_len, t_xmin, t_xmax, t_field3, t_ctid, t_infomask2, t_infomask, t_hoff, t_bits, t_oid FROM heap_page_items(pg_read_binary_file('/tmp/dump_block.page'));
lp | lp_off | lp_flags | lp_len | t_xmin | t_xmax | t_field3 | t_ctid | t_infomask2 | t_infomask | t_hoff | t_bits | t_oid
----+--------+----------+--------+-----------+-----------+----------+--------+-------------+------------+--------+----------------------------------+-------
1 | 1320 | 1 | 259 | 926025112 | 0 | 0 | (1,1) | 32799 | 10499 | 32 | 11111111111111111111111000100000 |
2 | 8088 | 1 | 104 | 926018702 | 0 | 0 | (1,2) | 32799 | 10497 | 32 | 11111111111111111111100000000000 |
3 | 0 | 0 | 0 | | | | | | | | |
4 | 7904 | 1 | 179 | 926018702 | 0 | 0 | (1,4) | 32799 | 10499 | 32 | 11111111111111111111111000100000 |
5 | 7728 | 1 | 176 | 926018702 | 0 | 0 | (1,5) | 32799 | 10499 | 32 | 11111111111111111111111000100000 |
6 | 7464 | 1 | 259 | 926014884 | 926025112 | 0 | (1,1) | 49183 | 9475 | 32 | 11111111111111111111111000100000 |
7 | 2 | 2 | 0 | | | | | | | | |
8 | 4 | 2 | 0 | | | | | | | | |
9 | 19 | 2 | 0 | | | | | | | | |
10 | 0 | 0 | 0 | | | | | | | | |
11 | 20 | 2 | 0 | | | | | | | | |
12 | 5792 | 1 | 1666 | 926014887 | 0 | 0 | (1,12) | 31 | 10499 | 32 | 11111111111111111111101000100000 |
13 | 5 | 2 | 0 | | | | | | | | |
14 | 3912 | 1 | 1880 | 925994211 | 0 | 0 | (1,14) | 31 | 10499 | 32 | 11111111111111111111110100110000 |
15 | 0 | 3 | 0 | | | | | | | | |
16 | 18 | 2 | 0 | | | | | | | | |
17 | 0 | 3 | 0 | | | | | | | | |
18 | 1936 | 1 | 1976 | 926013259 | 0 | 0 | (1,18) | 32799 | 10499 | 32 | 11111111111111111111110100110000 |
19 | 1760 | 1 | 176 | 926014887 | 0 | 0 | (1,19) | 32799 | 10499 | 32 | 11111111111111111111111000100000 |
20 | 1584 | 1 | 176 | 926014889 | 0 | 0 | (1,20) | 32799 | 10499 | 32 | 11111111111111111111111000100000 |
21 | 6 | 2 | 0 | | | | | | | | |
22 | 0 | 3 | 0 | | | | | | | | |
23 | 0 | 3 | 0 | | | | | | | | |
24 | 0 | 3 | 0 | | | | | | | | |
25 | 0 | 3 | 0 | | | | | | | | |
26 | 0 | 3 | 0 | | | | | | | | |
27 | 0 | 3 | 0 | | | | | | | | |
28 | 0 | 3 | 0 | | | | | | | | |
(28 rows)
No great surprise that it's again in pg_statistic.
#0 GetPrivateRefCountEntry (buffer=buffer@entry=411, do_move=do_move@entry=false) at bufmgr.c:313
#1 0x00000000007ecb4f in GetPrivateRefCount (buffer=411) at bufmgr.c:398
#2 BufferGetBlockNumber (buffer=buffer@entry=411) at bufmgr.c:2762
#3 0x00000000004fa0f3 in heap_prune_chain (prstate=0x7fff7e4a9180, rootoffnum=7, buffer=411) at pruneheap.c:625
#4 heap_page_prune (relation=relation@entry=0x7fe636faed28, buffer=buffer@entry=411, vistest=vistest@entry=0xe7bcc0 <GlobalVisCatalogRels>, old_snap_xmin=old_snap_xmin@entry=0, old_snap_ts=old_snap_ts@entry=0,
report_stats=report_stats@entry=false, off_loc=<optimized out>, off_loc@entry=0x12b433c) at pruneheap.c:278
#5 0x00000000004fd9bf in lazy_scan_prune (vacrel=vacrel@entry=0x12b42d0, buf=buf@entry=411, blkno=blkno@entry=1, page=page@entry=0x2aaaab54be00 "J\f", vistest=vistest@entry=0xe7bcc0 <GlobalVisCatalogRels>,
prunestate=prunestate@entry=0x7fff7e4aae40) at vacuumlazy.c:1712
#6 0x0000000000500263 in lazy_scan_heap (aggressive=<optimized out>, params=0x12ce89c, vacrel=<optimized out>) at vacuumlazy.c:1347
#7 heap_vacuum_rel (rel=0x7fe636faed28, params=0x12ce89c, bstrategy=<optimized out>) at vacuumlazy.c:612
#8 0x000000000067418a in table_relation_vacuum (bstrategy=<optimized out>, params=0x12ce89c, rel=0x7fe636faed28) at ../../../src/include/access/tableam.h:1678
#9 vacuum_rel (relid=2619, relation=<optimized out>, params=params@entry=0x12ce89c) at vacuum.c:2001
--
Justin
Commits
-
Deduplicate choice of horizon for a relation procarray.c.
- 3d0a4636aa4c 14.0 landed
- d9d8aa9bb9aa 15.0 landed
-
Use correct horizon when vacuuming catalog relations.
- 5a1e1d83022b 14.0 landed
-
Truncate line pointer array during VACUUM.
- 3c3b8a4b2689 14.0 cited
-
Remove tupgone special case from vacuumlazy.c.
- 8523492d4e34 14.0 cited
-
Refactor lazy_scan_heap() loop.
- 7ab96cf6b312 14.0 cited
-
snapshot scalability: Don't compute global horizons while building snapshots.
- dc7420c2c927 14.0 cited