Horton

Re: "could not reattach to shared memory" on buildfarm member dory

Noah Misch <noah@leadboat.com>

View thread View on postgr.es
From: Noah Misch <noah@leadboat.com>
To: Heath Lord <heath.lord@crunchydata.com>, Stephen Frost <sfrost@snowman.net>, Tom Lane <tgl@sss.pgh.pa.us>
Cc: Andres Freund <andres@anarazel.de>, Thomas Munro <thomas.munro@enterprisedb.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2018-12-03T05:35:06Z
Lists: pgsql-hackers

Attachments

On Tue, Sep 25, 2018 at 08:05:12AM -0700, Noah Misch wrote:
> On Mon, Sep 24, 2018 at 01:53:05PM -0400, Tom Lane wrote:
> > Overall, I agree that neither of these approaches are exactly attractive.
> > We're paying a heck of a lot of performance or complexity to solve a
> > problem that shouldn't even be there, and that we don't understand well.
> > In particular, the theory that some privileged code is injecting a thread
> > into every new process doesn't square with my results at
> > https://www.postgresql.org/message-id/15345.1525145612%40sss.pgh.pa.us
> > 
> > I think our best course of action at this point is to do nothing until
> > we have a clearer understanding of what's actually happening on dory.
> > Perhaps such understanding will yield an idea for a less painful fix.
> 
> I see.

Could one of you having a dory login use
https://live.sysinternals.com/Procmon.exe to capture process events during
backend startup?  The ideal would be one capture where startup failed reattach
and another where it succeeded, but having the successful run alone would be a
good start.  The procedure is roughly this:

- Install PostgreSQL w/ debug symbols.
- Start a postmaster.
- procmon /nomonitor
- procmon "Filter" menu -> Enable Advanced Output
- Ctrl-l, add filter for "Process Name" is "postgres.exe"
- Ctrl-e (starts collecting data)
- psql (leave it running)
- After ~60s, Ctrl-e again in procmon (stops collecting data)
- File -> Save -> PML
- File -> Save -> XML, include stack traces, resolve stack symbols
- Compress the PML and XML files, and mail them here


I'm attaching the data from a system not having the problem.  On this system,
backend startup sees six thread creations:

1. main thread
2. thread created before postgres.exe has control
3. thread created before postgres.exe has control
4. thread created before postgres.exe has control
5. in pgwin32_signal_initialize()
6. in src\backend\port\win32\timer.c:setitimer()

Threads 2-4 exit exactly 30s after creation.  If we fail to reattach to shared
memory, we'll exit before reaching code to start 5 or 6.  It would be quite
interesting if dory makes a different number of threads or if threads 2-4 live
some duration other than 30s.  It would also be interesting if dory has "Load
Image" events after postgres.exe code has started running.  This unaffected
system loads mswsock.dll during read_inheritable_socket().

Thanks,
nm

Commits

  1. Avoid "could not reattach" by providing space for concurrent allocation.

  2. Assert that pgwin32_signal_initialize() has been called early enough.

  3. Remove investigative code for can't-reattach-to-shared-memory errors.

  4. Does it help to wait before reattaching?

  5. Map and unmap the shared memory block before risking VirtualFree.

  6. Further effort at preventing memory map dump from affecting the results.

  7. Remove Windows module-list-dumping code.

  8. Dump full memory maps around failing Windows reattach code.

  9. Get still more info about Windows can't-reattach-to-shared-memory errors.

  10. Get more info about Windows can't-reattach-to-shared-memory errors.

  11. Try to get some info about Windows can't-reattach-to-shared-memory errors.