Re: "could not reattach to shared memory" on buildfarm member dory

Heath Lord <heath.lord@crunchydata.com>

From: Heath Lord <heath.lord@crunchydata.com>
To: Noah Misch <noah@leadboat.com>
Cc: Stephen Frost <sfrost@snowman.net>, Andres Freund <andres@anarazel.de>, Thomas Munro <thomas.munro@enterprisedb.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>, Tom Lane <tgl@sss.pgh.pa.us>, Joseph Ayers <joseph.ayers@crunchydata.com>
Date: 2019-01-29T16:28:56Z
Lists: pgsql-hackers
On Thu, Jan 17, 2019 at 3:27 AM Noah Misch <noah@leadboat.com> wrote:

> On Sun, Dec 02, 2018 at 09:35:06PM -0800, Noah Misch wrote:
> > On Tue, Sep 25, 2018 at 08:05:12AM -0700, Noah Misch wrote:
> > > On Mon, Sep 24, 2018 at 01:53:05PM -0400, Tom Lane wrote:
> > > > Overall, I agree that neither of these approaches are exactly
> attractive.
> > > > We're paying a heck of a lot of performance or complexity to solve a
> > > > problem that shouldn't even be there, and that we don't understand
> well.
> > > > In particular, the theory that some privileged code is injecting a
> thread
> > > > into every new process doesn't square with my results at
> > > >
> https://www.postgresql.org/message-id/15345.1525145612%40sss.pgh.pa.us
> > > >
> > > > I think our best course of action at this point is to do nothing
> until
> > > > we have a clearer understanding of what's actually happening on dory.
> > > > Perhaps such understanding will yield an idea for a less painful fix.
> > >
> > > I see.
> >
> > Could one of you having a dory login use
> > https://live.sysinternals.com/Procmon.exe to capture process events
> during
> > backend startup?
>
> Ping.
>
> > The ideal would be one capture where startup failed reattach
> > and another where it succeeded, but having the successful run alone
> would be a
> > good start.  The procedure is roughly this:
> >
> > - Install PostgreSQL w/ debug symbols.
> > - Start a postmaster.
> > - procmon /nomonitor
> > - procmon "Filter" menu -> Enable Advanced Output
> > - Ctrl-l, add filter for "Process Name" is "postgres.exe"
> > - Ctrl-e (starts collecting data)
> > - psql (leave it running)
> > - After ~60s, Ctrl-e again in procmon (stops collecting data)
> > - File -> Save -> PML
> > - File -> Save -> XML, include stack traces, resolve stack symbols
> > - Compress the PML and XML files, and mail them here
>

Noah,
   I apologize for not responding earlier, but we attempting to capture the
information you requested.  However, where would you like us to pull the
install for PostgreSQL with the debug symbols in it?  We are aware of the
BigSQL and EDB installers, but are unsure if these contain the debug
symbols.  Currently this machine has nothing on it other than the necessary
dependencies to build postgres for the community.  If you could please give
us some more information on what you would like us to install to gather
this information to help debug the issue we are seeing we would really
appreciate it.  Also, if there is any additional information on what we
have installed on the server or how we are configured please just let us
know and we will get you that as soon as possible.  Thank you again for
your interest in this issue.

-Heath

Commits

  1. Avoid "could not reattach" by providing space for concurrent allocation.

  2. Assert that pgwin32_signal_initialize() has been called early enough.

  3. Remove investigative code for can't-reattach-to-shared-memory errors.

  4. Does it help to wait before reattaching?

  5. Map and unmap the shared memory block before risking VirtualFree.

  6. Further effort at preventing memory map dump from affecting the results.

  7. Remove Windows module-list-dumping code.

  8. Dump full memory maps around failing Windows reattach code.

  9. Get still more info about Windows can't-reattach-to-shared-memory errors.

  10. Get more info about Windows can't-reattach-to-shared-memory errors.

  11. Try to get some info about Windows can't-reattach-to-shared-memory errors.