Horton

Thread

Commits

  1. Fix inconsistent out-of-memory error reporting in dsa.c.

  2. Fix race in dsm_unpin_segment() when handles are reused.

  3. Fix race in dsm_attach() when handles are reused.

  4. Fix rare dsa_allocate() failures due to freepage.c corruption.

  5. Add WL_EXIT_ON_PM_DEATH pseudo-event.

  1. REL_11_STABLE: dsm.c - cannot unpin a segment that is not pinned

    Justin Pryzby <pryzby@telsasoft.com> — 2019-02-16T02:38:54Z

    I saw this error once last week while stress testing to reproduce earlier bugs,
but tentatively thought it was a downstream symptom of those bugs (since
fixed), and now wanted to check that #15585 and others were no longer
reproducible.  Unfortunately I got this error while running same test case [2]
as for previous bug ('could not attach').

2019-02-14 23:40:41.611 MST [32287] ERROR:  cannot unpin a segment that is not pinned

On commit faf132449c0cafd31fe9f14bbf29ca0318a89058 (REL_11_STABLE including
both of last week's post-11.2 DSA patches), I reproduced twice, once within
~2.5 hours, once within 30min.

I'm not able to reproduce on master running overnight and now 16+hours.

See also:

pg11.1: dsa_area could not attach to segment - resolved by commit 6c0fb941892519ad6b8873e99c4001404fb9a128
  [1] https://www.postgresql.org/message-id/20181231221734.GB25379%40telsasoft.com
pg11.1: dsa_area could not attach to segment:
  [2] https://www.postgresql.org/message-id/20190211040132.GV31721%40telsasoft.com
BUG #15585: infinite DynamicSharedMemoryControlLock waiting in parallel query
  [3] https://www.postgresql.org/message-id/flat/15585-324ff6a93a18da46%40postgresql.org
dsa_allocate() faliure - resolved by commit 7215efdc005e694ec93678a6203dbfc714d12809 (also doesn't affect master)
  [4] https://www.postgresql.org/message-id/flat/CAMAYy4%2Bw3NTBM5JLWFi8twhWK4%3Dk_5L4nV5%2BbYDSPu8r4b97Zg%40mail.gmail.com

#0  0x00007f96a589e277 in raise () from /lib64/libc.so.6
#1  0x00007f96a589f968 in abort () from /lib64/libc.so.6
#2  0x000000000088b6f7 in errfinish (dummy=dummy@entry=0) at elog.c:555
#3  0x000000000088eee8 in elog_finish (elevel=elevel@entry=22, fmt=fmt@entry=0xa52cb0 "cannot unpin a segment that is not pinned") at elog.c:1376
#4  0x00000000007578b2 in dsm_unpin_segment (handle=1780672242) at dsm.c:914
#5  0x00000000008aee15 in dsa_release_in_place (place=0x7f96a6991640) at dsa.c:618
#6  0x00000000007571a9 in dsm_detach (seg=0x2470f78) at dsm.c:731
#7  0x0000000000509233 in DestroyParallelContext (pcxt=0x24c18c0) at parallel.c:900
#8  0x000000000062db65 in ExecParallelCleanup (pei=0x25aacf8) at execParallel.c:1154
#9  0x0000000000640588 in ExecShutdownGather (node=node@entry=0x2549b60) at nodeGather.c:406
#10 0x0000000000630208 in ExecShutdownNode (node=0x2549b60) at execProcnode.c:767
#11 0x000000000067724f in planstate_tree_walker (planstate=planstate@entry=0x2549a48, walker=walker@entry=0x6301c0 <ExecShutdownNode>, context=context@entry=0x0) at nodeFuncs.c:3739
#12 0x00000000006301dd in ExecShutdownNode (node=0x2549a48) at execProcnode.c:749
#13 0x000000000067724f in planstate_tree_walker (planstate=planstate@entry=0x25495d0, walker=walker@entry=0x6301c0 <ExecShutdownNode>, context=context@entry=0x0) at nodeFuncs.c:3739
#14 0x00000000006301dd in ExecShutdownNode (node=0x25495d0) at execProcnode.c:749
#15 0x000000000067724f in planstate_tree_walker (planstate=planstate@entry=0x25494b8, walker=walker@entry=0x6301c0 <ExecShutdownNode>, context=context@entry=0x0) at nodeFuncs.c:3739
#16 0x00000000006301dd in ExecShutdownNode (node=0x25494b8) at execProcnode.c:749
#17 0x000000000067724f in planstate_tree_walker (planstate=planstate@entry=0x25492a8, walker=walker@entry=0x6301c0 <ExecShutdownNode>, context=context@entry=0x0) at nodeFuncs.c:3739
#18 0x00000000006301dd in ExecShutdownNode (node=node@entry=0x25492a8) at execProcnode.c:749
#19 0x0000000000628fbd in ExecutePlan (execute_once=<optimized out>, dest=0xd96e60 <donothingDR>, direction=<optimized out>, numberTuples=0, sendTuples=true, operation=CMD_SELECT, use_parallel_mode=<optimized out>, 
    planstate=0x25492a8, estate=0x2549038) at execMain.c:1787
#20 standard_ExecutorRun (queryDesc=0x2576990, direction=<optimized out>, count=0, execute_once=<optimized out>) at execMain.c:364
#21 0x00000000005c635f in ExplainOnePlan (plannedstmt=plannedstmt@entry=0x2579820, into=into@entry=0x0, es=es@entry=0x2529170, 
    queryString=queryString@entry=0x243f348 "explain analyze SELECT colcld.child c, parent p, array_agg(colpar.attname::text ORDER BY colpar.attnum) cols, array_agg(format_type(colpar.atttypid, colpar.atttypmod) ORDER BY colpar.attnum) AS types "..., params=params@entry=0x0, queryEnv=queryEnv@entry=0x0, planduration=planduration@entry=0x7fff1048afa0) at explain.c:535
#22 0x00000000005c665f in ExplainOneQuery (query=<optimized out>, cursorOptions=<optimized out>, into=0x0, es=0x2529170, 
    queryString=0x243f348 "explain analyze SELECT colcld.child c, parent p, array_agg(colpar.attname::text ORDER BY colpar.attnum) cols, array_agg(format_type(colpar.atttypid, colpar.atttypmod) ORDER BY colpar.attnum) AS types "..., params=0x0, queryEnv=0x0) at explain.c:371
#23 0x00000000005c6bbe in ExplainQuery (pstate=pstate@entry=0x2461bd8, stmt=stmt@entry=0x24f87a8, 
    queryString=queryString@entry=0x243f348 "explain analyze SELECT colcld.child c, parent p, array_agg(colpar.attname::text ORDER BY colpar.attnum) cols, array_agg(format_type(colpar.atttypid, colpar.atttypmod) ORDER BY colpar.attnum) AS types "..., params=params@entry=0x0, queryEnv=queryEnv@entry=0x0, dest=dest@entry=0x2461b40) at explain.c:254
#24 0x0000000000782a5d in standard_ProcessUtility (pstmt=0x24f8930, 
    queryString=0x243f348 "explain analyze SELECT colcld.child c, parent p, array_agg(colpar.attname::text ORDER BY colpar.attnum) cols, array_agg(format_type(colpar.atttypid, colpar.atttypmod) ORDER BY colpar.attnum) AS types "..., context=PROCESS_UTILITY_TOPLEVEL, params=0x0, queryEnv=0x0, dest=0x2461b40, completionTag=0x7fff1048b160 "") at utility.c:675
#25 0x000000000077fcc6 in PortalRunUtility (portal=0x24a4bd8, pstmt=0x24f8930, isTopLevel=<optimized out>, setHoldSnapshot=<optimized out>, dest=0x2461b40, completionTag=0x7fff1048b160 "") at pquery.c:1178
#26 0x0000000000780b03 in FillPortalStore (portal=portal@entry=0x24a4bd8, isTopLevel=isTopLevel@entry=true) at pquery.c:1038
#27 0x0000000000781638 in PortalRun (portal=portal@entry=0x24a4bd8, count=count@entry=9223372036854775807, isTopLevel=isTopLevel@entry=true, run_once=run_once@entry=true, dest=dest@entry=0x2512490, 
#28 0x000000000077d27f in exec_simple_query (
    query_string=0x243f348 "explain analyze SELECT colcld.child c, parent p, array_agg(colpar.attname::text ORDER BY colpar.attnum) cols, array_agg(format_type(colpar.atttypid, colpar.atttypmod) ORDER BY colpar.attnum) AS types "...) at postgres.c:1145
#29 0x000000000077e5b2 in PostgresMain (argc=<optimized out>, argv=argv@entry=0x24691b8, dbname=0x24690b0 "postgres", username=<optimized out>) at postgres.c:4182
#30 0x000000000047cdf3 in BackendRun (port=0x24613a0) at postmaster.c:4361
#31 BackendStartup (port=0x24613a0) at postmaster.c:4033
#32 ServerLoop () at postmaster.c:1706
#33 0x0000000000706449 in PostmasterMain (argc=argc@entry=20, argv=argv@entry=0x2439ba0) at postmaster.c:1379
#34 0x000000000047ddd1 in main (argc=20, argv=0x2439ba0) at main.c:228

  2. Re: REL_11_STABLE: dsm.c - cannot unpin a segment that is not pinned

    Thomas Munro <thomas.munro@enterprisedb.com> — 2019-02-16T04:08:01Z

    On Sat, Feb 16, 2019 at 3:38 PM Justin Pryzby <pryzby@telsasoft.com> wrote:
> I saw this error once last week while stress testing to reproduce earlier bugs,
> but tentatively thought it was a downstream symptom of those bugs (since
> fixed), and now wanted to check that #15585 and others were no longer
> reproducible.  Unfortunately I got this error while running same test case [2]
> as for previous bug ('could not attach').
>
> 2019-02-14 23:40:41.611 MST [32287] ERROR:  cannot unpin a segment that is not pinned
>
> On commit faf132449c0cafd31fe9f14bbf29ca0318a89058 (REL_11_STABLE including
> both of last week's post-11.2 DSA patches), I reproduced twice, once within
> ~2.5 hours, once within 30min.
>
> I'm not able to reproduce on master running overnight and now 16+hours.

Oh, I think I know why: dsm_unpin_segment() containt another variant
of the race fixed by 6c0fb941 (that was for dsm_attach() being
confused by segments with the same handle that are concurrently going
away, but dsm_unpin_segment() does a handle lookup too, so it can be
confused by the same phenomenon).  Untested, but the fix is probably:

diff --git a/src/backend/storage/ipc/dsm.c b/src/backend/storage/ipc/dsm.c
index cfbebeb31d..23ccc59f13 100644
--- a/src/backend/storage/ipc/dsm.c
+++ b/src/backend/storage/ipc/dsm.c
@@ -844,8 +844,8 @@ dsm_unpin_segment(dsm_handle handle)
        LWLockAcquire(DynamicSharedMemoryControlLock, LW_EXCLUSIVE);
        for (i = 0; i < dsm_control->nitems; ++i)
        {
-               /* Skip unused slots. */
-               if (dsm_control->item[i].refcnt == 0)
+               /* Skip unused slots and segments that are
concurrently going away. */
+               if (dsm_control->item[i].refcnt <= 1)
                        continue;

                /* If we've found our handle, we can stop searching. */

-- 
Thomas Munro
http://www.enterprisedb.com

  3. Re: REL_11_STABLE: dsm.c - cannot unpin a segment that is not pinned

    Justin Pryzby <pryzby@telsasoft.com> — 2019-02-17T19:41:45Z

    On Sat, Feb 16, 2019 at 09:16:01PM +1300, Thomas Munro wrote:
> On Sat, Feb 16, 2019 at 5:31 PM Justin Pryzby <pryzby@telsasoft.com> wrote:
> > Thanks, will leave it spinning overnight.

No errors in ~36 hours (126 CPU-hrs), so that seems to work.  Thanks.

> > Do you know if any of the others should also be changed ?
> 
> Good question, let me double check...
> 
> > $ grep 'refcnt == 0' src/backend/storage/ipc/dsm.c
> >                 if (refcnt == 0)
> 
> That's dsm_cleanup_using_control_segment() and runs when starting up
> before any workers can be running to clean up after a preceding crash,
> so it's OK (if it's 1, meaning we crashed while that slot was going
> away, we'll try to destroy it again, which is correct).  Good.
> 
> >                 if (dsm_control->item[i].refcnt == 0)
> 
> That's dsm_postmaster_shutdown(), similar but at shutdown time, run by
> the postmaster, and it errs on the side of trying to destroy.  Good.
> 
> >                 if (dsm_control->item[i].refcnt == 0)
> 
> That's dsm_create(), and it's looking specifically for a free slot,
> and that's 0 only, it'll step over used/active (refcnt > 1) and
> used/going-away (refcnt == 1).  Good.

  4. Re: REL_11_STABLE: dsm.c - cannot unpin a segment that is not pinned

    Justin Pryzby <pryzby@telsasoft.com> — 2019-02-17T20:07:00Z

    On Sun, Feb 17, 2019 at 01:41:45PM -0600, Justin Pryzby wrote:
> On Sat, Feb 16, 2019 at 09:16:01PM +1300, Thomas Munro wrote:
> > On Sat, Feb 16, 2019 at 5:31 PM Justin Pryzby <pryzby@telsasoft.com> wrote:
> > > Thanks, will leave it spinning overnight.
> 
> No errors in ~36 hours (126 CPU-hrs), so that seems to work.  Thanks.

Actually...

On killing the postmaster having completed this stress test, one of the
backends was left running and didn't die on its own.  It did die gracefully
when I killed the backend or the client.

I was able to repeat the result, on first try, but took numerous attempts to
repeat the 2nd and 3rd time to save pg_stat_activity.

Is there some issue regarding dsm_postmaster_shutdown ?

[pryzbyj@database postgresql]$ ps -wwf --ppid 31656
UID        PID  PPID  C STIME TTY          TIME CMD
pryzbyj   4512 31656  1 13:00 ?        00:00:00 postgres: pryzbyj postgres [local] EXPLAIN
pryzbyj  31657 31656  0 12:59 ?        00:00:00 postgres: logger   
pryzbyj  31659 31656  0 12:59 ?        00:00:00 postgres: checkpointer   
pryzbyj  31662 31656  0 12:59 ?        00:00:00 postgres: stats collector   
pryzbyj  31785 31656  0 12:59 ?        00:00:00 postgres: pryzbyj postgres [local] idle

datid            | 13285
datname          | postgres
pid              | 4512
usesysid         | 10
usename          | pryzbyj
application_name | psql
client_addr      | 
client_hostname  | 
client_port      | -1
backend_start    | 2019-02-17 13:00:50.79285-07
xact_start       | 2019-02-17 13:00:50.797711-07
query_start      | 2019-02-17 13:00:50.797711-07
state_change     | 2019-02-17 13:00:50.797713-07
wait_event_type  | IPC
wait_event       | ExecuteGather
state            | active
backend_xid      | 
backend_xmin     | 1569
query            | explain analyze SELECT colcld.child c, parent p, array_agg(colpar.attname::text ORDER BY colpar.attnum) cols, array_agg(format_type(colpar.atttypid, colpar.atttypmod) ORDER BY colpar.attnum) AS typ
es FROM queued_alters qa JOIN pg_attribute colpar ON to_regclass(qa.parent)=colpar.attrelid AND colpar.attnum>0 AND NOT colpar.attisdropped JOIN (SELECT *, attrelid::regclass::text AS child FROM pg_attribute) colcld 
ON to_regclass(qa.child) =colcld.attrelid AND colcld.attnum>0 AND NOT colcld.attisdropped WHERE colcld.attname=colpar.attname AND colpar.atttypid!=colcld.atttypid GROUP BY 1,2 ORDER BY parent LIKE 'unused%', regexp_r
eplace(colcld.child, '.*_((([0-9]{4}_[0-9]{2})_[0-9]{2})|(([0-9]{6})([0-9]{2})?))$', '\3\5') DESC, regexp_replace(colcld.child, '.*_', '') DESC LIMIT 1
backend_type     | client backend

#0  0x00007fe131637163 in __epoll_wait_nocancel () from /lib64/libc.so.6
#1  0x0000000000758d26 in WaitEventSetWaitBlock (nevents=1, occurred_events=0x7ffde16775a0, cur_timeout=-1, set=0x7fe132640e50) at latch.c:1048
#2  WaitEventSetWait (set=set@entry=0x7fe132640e50, timeout=timeout@entry=-1, occurred_events=occurred_events@entry=0x7ffde16775a0, nevents=nevents@entry=1, wait_event_info=wait_event_info@entry=134217731)
    at latch.c:1000
#3  0x00000000007591c2 in WaitLatchOrSocket (latch=0x7fe12a7591b4, wakeEvents=wakeEvents@entry=1, sock=sock@entry=-1, timeout=-1, timeout@entry=0, wait_event_info=wait_event_info@entry=134217731) at latch.c:385
#4  0x00000000007592a0 in WaitLatch (latch=<optimized out>, wakeEvents=wakeEvents@entry=1, timeout=timeout@entry=0, wait_event_info=wait_event_info@entry=134217731) at latch.c:339
#5  0x00000000006401e2 in gather_readnext (gatherstate=<optimized out>) at nodeGather.c:367
#6  gather_getnext (gatherstate=0x2af1f70) at nodeGather.c:256
#7  ExecGather (pstate=0x2af1f70) at nodeGather.c:207
#8  0x0000000000630188 in ExecProcNodeInstr (node=0x2af1f70) at execProcnode.c:461
#9  0x0000000000653506 in ExecProcNode (node=0x2af1f70) at ../../../src/include/executor/executor.h:247
#10 ExecSort (pstate=0x2af1e58) at nodeSort.c:107
#11 0x0000000000630188 in ExecProcNodeInstr (node=0x2af1e58) at execProcnode.c:461
#12 0x0000000000638a89 in ExecProcNode (node=0x2af1e58) at ../../../src/include/executor/executor.h:247
#13 fetch_input_tuple (aggstate=aggstate@entry=0x2af19e0) at nodeAgg.c:406
#14 0x000000000063a6b0 in agg_retrieve_direct (aggstate=0x2af19e0) at nodeAgg.c:1740
#15 ExecAgg (pstate=0x2af19e0) at nodeAgg.c:1555
#16 0x0000000000630188 in ExecProcNodeInstr (node=0x2af19e0) at execProcnode.c:461
#17 0x0000000000653506 in ExecProcNode (node=0x2af19e0) at ../../../src/include/executor/executor.h:247
#18 ExecSort (pstate=0x2af18c8) at nodeSort.c:107
#19 0x0000000000630188 in ExecProcNodeInstr (node=0x2af18c8) at execProcnode.c:461
#20 0x00000000006498e1 in ExecProcNode (node=0x2af18c8) at ../../../src/include/executor/executor.h:247
#21 ExecLimit (pstate=0x2af16b8) at nodeLimit.c:95
#22 0x0000000000630188 in ExecProcNodeInstr (node=0x2af16b8) at execProcnode.c:461
#23 0x0000000000628eda in ExecProcNode (node=0x2af16b8) at ../../../src/include/executor/executor.h:247
#24 ExecutePlan (execute_once=<optimized out>, dest=0xd96e60 <donothingDR>, direction=<optimized out>, numberTuples=0, sendTuples=true, operation=CMD_SELECT, use_parallel_mode=<optimized out>, planstate=0x2af16b8, 
    estate=0x2af1448) at execMain.c:1723
#25 standard_ExecutorRun (queryDesc=0x2b1eda0, direction=<optimized out>, count=0, execute_once=<optimized out>) at execMain.c:364
#26 0x00000000005c635f in ExplainOnePlan (plannedstmt=plannedstmt@entry=0x2b21c30, into=into@entry=0x0, es=es@entry=0x2ad1580, 
    queryString=queryString@entry=0x29e7348 "explain analyze SELECT colcld.child c, parent p, array_agg(colpar.attname::text ORDER BY colpar.attnum) cols, array_agg(format_type(colpar.atttypid, colpar.atttypmod) ORDER BY colpar.attnum) AS types "..., params=params@entry=0x0, queryEnv=queryEnv@entry=0x0, planduration=planduration@entry=0x7ffde1677970) at explain.c:535
#27 0x00000000005c665f in ExplainOneQuery (query=<optimized out>, cursorOptions=<optimized out>, into=0x0, es=0x2ad1580, 
    queryString=0x29e7348 "explain analyze SELECT colcld.child c, parent p, array_agg(colpar.attname::text ORDER BY colpar.attnum) cols, array_agg(format_type(colpar.atttypid, colpar.atttypmod) ORDER BY colpar.attnum) AS types "..., params=0x0, queryEnv=0x0) at explain.c:371
#28 0x00000000005c6bbe in ExplainQuery (pstate=pstate@entry=0x2a09bd8, stmt=stmt@entry=0x2aa0bb8, 
    queryString=queryString@entry=0x29e7348 "explain analyze SELECT colcld.child c, parent p, array_agg(colpar.attname::text ORDER BY colpar.attnum) cols, array_agg(format_type(colpar.atttypid, colpar.atttypmod) ORDER BY colpar.attnum) AS types "..., params=params@entry=0x0, queryEnv=queryEnv@entry=0x0, dest=dest@entry=0x2a09b40) at explain.c:254
#29 0x0000000000782a1d in standard_ProcessUtility (pstmt=0x2aa0d40,

  5. Re: REL_11_STABLE: dsm.c - cannot unpin a segment that is not pinned

    Thomas Munro <thomas.munro@enterprisedb.com> — 2019-02-17T20:26:53Z

    On Mon, Feb 18, 2019 at 9:07 AM Justin Pryzby <pryzby@telsasoft.com> wrote:
> On Sun, Feb 17, 2019 at 01:41:45PM -0600, Justin Pryzby wrote:
> > On Sat, Feb 16, 2019 at 09:16:01PM +1300, Thomas Munro wrote:
> > > On Sat, Feb 16, 2019 at 5:31 PM Justin Pryzby <pryzby@telsasoft.com> wrote:
> > > > Thanks, will leave it spinning overnight.
> >
> > No errors in ~36 hours (126 CPU-hrs), so that seems to work.  Thanks.

Great news.  I will commit that.

> Actually...
>
> On killing the postmaster having completed this stress test, one of the
> backends was left running and didn't die on its own.  It did die gracefully
> when I killed the backend or the client.
>
> I was able to repeat the result, on first try, but took numerous attempts to
> repeat the 2nd and 3rd time to save pg_stat_activity.
>
> Is there some issue regarding dsm_postmaster_shutdown ?

Huh.  What exactly do you mean by "killing the postmaster"?  If you
mean SIGKILL or something, one problem with 11 is that
gather_readnext() doesn't respond to postmaster death.  I fixed that
(and every similar place) in master with commit cfdf4dc4fc9, like so:

-                       WaitLatch(MyLatch, WL_LATCH_SET, 0,
WAIT_EVENT_EXECUTE_GATHER);
+                       (void) WaitLatch(MyLatch, WL_LATCH_SET |
WL_EXIT_ON_PM_DEATH, 0,
+
WAIT_EVENT_EXECUTE_GATHER);

-- 
Thomas Munro
http://www.enterprisedb.com

  6. Re: REL_11_STABLE: dsm.c - cannot unpin a segment that is not pinned

    Justin Pryzby <pryzby@telsasoft.com> — 2019-02-17T20:35:37Z

    On Mon, Feb 18, 2019 at 09:26:53AM +1300, Thomas Munro wrote:
> On Mon, Feb 18, 2019 at 9:07 AM Justin Pryzby <pryzby@telsasoft.com> wrote:
> > Actually...
> >
> > On killing the postmaster having completed this stress test, one of the
> > backends was left running and didn't die on its own.  It did die gracefully
> > when I killed the backend or the client.
> >
> > I was able to repeat the result, on first try, but took numerous attempts to
> > repeat the 2nd and 3rd time to save pg_stat_activity.
> >
> > Is there some issue regarding dsm_postmaster_shutdown ?
> 
> Huh.  What exactly do you mean by "killing the postmaster"?  If you
> mean SIGKILL or something, one problem with 11 is that

I mean unqualified /bin/kill which is kill -TERM (-15 in linux).

I gather (pun acknowledged but not intended) you mean "one problem with PG v11"
and not "one problem with kill -11" (which is what I first thought, although I
was somehow confusing kill -9, and since I don't know why anyone would ever
want to manually send SIGSEGV).

I think you're suggesting that's a known issue with v11, so nothing to do.

Thanks,
Justin

  7. Re: REL_11_STABLE: dsm.c - cannot unpin a segment that is not pinned

    Thomas Munro <thomas.munro@enterprisedb.com> — 2019-02-17T21:26:12Z

    On Mon, Feb 18, 2019 at 9:35 AM Justin Pryzby <pryzby@telsasoft.com> wrote:
> On Mon, Feb 18, 2019 at 09:26:53AM +1300, Thomas Munro wrote:
> > Huh.  What exactly do you mean by "killing the postmaster"?  If you
> > mean SIGKILL or something, one problem with 11 is that
>
> I mean unqualified /bin/kill which is kill -TERM (-15 in linux).
>
> I gather (pun acknowledged but not intended) you mean "one problem with PG v11"
> and not "one problem with kill -11" (which is what I first thought, although I
> was somehow confusing kill -9, and since I don't know why anyone would ever
> want to manually send SIGSEGV).
>
> I think you're suggesting that's a known issue with v11, so nothing to do.

Yeah.  I suppose we should probably consider back-patching a fix for that.

I've pushed the second DSM fix.  Here is a summary of the errors you
(and others) have reported, for the benefit of people searching the
archives.  I will give the master commit IDs, but the fixes should be
in 11.3 and 10.8.

1.  "dsa_allocate could not find %zu free pages": freepage.c, fixed in 7215efdc.
2.  "dsa_area could not attach to segment": dsm.c, fixed in 6c0fb941.
3.  "cannot unpin a segment that is not pinned": dsm.c, fixed in 0b55aaac.

That resolves all the bugs I'm currently aware of in this area.
(There is still the question of whether we should change the way DSA
cleans up to avoid a self-deadlock on recursive error, but that needs
some more thought.)

-- 
Thomas Munro
http://www.enterprisedb.com

  8. Re: REL_11_STABLE: dsm.c - cannot unpin a segment that is not pinned

    Justin Pryzby <pryzby@telsasoft.com> — 2019-02-17T22:36:37Z

    On Mon, Feb 18, 2019 at 09:26:53AM +1300, Thomas Munro wrote:
> Huh.  What exactly do you mean by "killing the postmaster"?  If you
> mean SIGKILL or something, one problem with 11 is that
> gather_readnext() doesn't respond to postmaster death.  I fixed that
> (and every similar place) in master with commit cfdf4dc4fc9, like so:

On Mon, Feb 18, 2019 at 10:26:12AM +1300, Thomas Munro wrote:
> Yeah.  I suppose we should probably consider back-patching a fix for that.

It hasn't been an issue for us, but that seems like a restart hazard.  Who
knows what all the distros initscripts do, how thin a layer they are around
pg_ctl or kill, but you risk waiting indefinitely for postmaster and its gather
backend/s to die, all the while rejecting new clients with 'the database system
is shutting down'.

+1

Justin

  9. Re: REL_11_STABLE: dsm.c - cannot unpin a segment that is not pinned

    Thomas Munro <thomas.munro@enterprisedb.com> — 2019-02-18T00:24:50Z

    On Mon, Feb 18, 2019 at 10:26 AM Thomas Munro
<thomas.munro@enterprisedb.com> wrote:
> 1.  "dsa_allocate could not find %zu free pages": freepage.c, fixed in 7215efdc.
> 2.  "dsa_area could not attach to segment": dsm.c, fixed in 6c0fb941.
> 3.  "cannot unpin a segment that is not pinned": dsm.c, fixed in 0b55aaac.
>
> That resolves all the bugs I'm currently aware of in this area.

Bleugh.  After that optimistic statement, Justin reminded me of a
one-off segfault report hiding over on the pgsql-performance list[1].
That was buried in a bunch of discussion of bug #1 in the above list,
which is now fixed.  However, the segfault report was never directly
addressed.

After thinking really hard and doubling down on coffee, I think I know
how it happened -- actually it's something I have mentioned once
before as a minor defect, but I hadn't connected all the dots.  Bug #1
above was occurring occasionally in Jakub's workload, and there is a
path through the code that would result in plain old allocation
failure instead of raising FATAL error #1, and a place where failure
to allocate can return InvalidDsaPointer instead of raising an "out of
memory" error so that calling code could finish up eating a null
pointer that it wasn't expecting (that's user controllable, using the
DSA_ALLOC_NO_OOM flag, but that one place didn't respect it).  Here is
a patch that fixes those problems.

[1] https://postgr.es/m/CAJk1zg3ZXhDsFg7tQGJ3ZD6N9dp%2BQ1_DU2N3%3Ds3Ywb-u6Lhc5A%40mail.gmail.com

-- 
Thomas Munro
http://www.enterprisedb.com