Horton

Re: "could not reattach to shared memory" on buildfarm member dory

Noah Misch <noah@leadboat.com>

View thread View on postgr.es
From: Noah Misch <noah@leadboat.com>
To: Heath Lord <heath.lord@crunchydata.com>
Cc: Stephen Frost <sfrost@snowman.net>, Andres Freund <andres@anarazel.de>, Thomas Munro <thomas.munro@enterprisedb.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>, Tom Lane <tgl@sss.pgh.pa.us>, Joseph Ayers <joseph.ayers@crunchydata.com>
Date: 2019-01-31T05:04:25Z
Lists: pgsql-hackers 
On Tue, Jan 29, 2019 at 11:28:56AM -0500, Heath Lord wrote:
> On Thu, Jan 17, 2019 at 3:27 AM Noah Misch <noah@leadboat.com> wrote:
> > On Sun, Dec 02, 2018 at 09:35:06PM -0800, Noah Misch wrote:
> > > Could one of you having a dory login use
> > > https://live.sysinternals.com/Procmon.exe to capture process events
> > during
> > > backend startup?
> > > The ideal would be one capture where startup failed reattach
> > > and another where it succeeded, but having the successful run alone
> > would be a
> > > good start.  The procedure is roughly this:
> > >
> > > - Install PostgreSQL w/ debug symbols.
> > > - Start a postmaster.
> > > - procmon /nomonitor
> > > - procmon "Filter" menu -> Enable Advanced Output
> > > - Ctrl-l, add filter for "Process Name" is "postgres.exe"
> > > - Ctrl-e (starts collecting data)
> > > - psql (leave it running)
> > > - After ~60s, Ctrl-e again in procmon (stops collecting data)
> > > - File -> Save -> PML
> > > - File -> Save -> XML, include stack traces, resolve stack symbols
> > > - Compress the PML and XML files, and mail them here
> 
>    I apologize for not responding earlier, but we attempting to capture the
> information you requested.  However, where would you like us to pull the
> install for PostgreSQL with the debug symbols in it?  We are aware of the
> BigSQL and EDB installers, but are unsure if these contain the debug
> symbols.

Please use a locally-built PostgreSQL as similar as possible to what dory's
buildfarm runs produce, except building with debug symbols.  A
potentially-convenient way to achieve this would be to add CONFIG=>'Debug' to
build_env of your buildfarm configuration, then run the buildfarm client with
--keepall.  By the end of the run, $buildroot/$branch/inst will contain an
installation.  (I haven't actually tested that.)  You can copy that installation
to any convenient place (e.g. your home directory) and use that copy for the
remaining steps.

> Currently this machine has nothing on it other than the necessary
> dependencies to build postgres for the community.  If you could please give
> us some more information on what you would like us to install to gather
> this information to help debug the issue we are seeing we would really
> appreciate it.

At the moment, I think the only other thing you'll need is to download
https://live.sysinternals.com/Procmon.exe.

> Also, if there is any additional information on what we
> have installed on the server or how we are configured please just let us
> know and we will get you that as soon as possible.

Thanks.  I don't have specific questions about that at this time, but feel free
to share any lists or config dumps that seem relevant.

Commits

  1. Avoid "could not reattach" by providing space for concurrent allocation.

  2. Assert that pgwin32_signal_initialize() has been called early enough.

  3. Remove investigative code for can't-reattach-to-shared-memory errors.

  4. Does it help to wait before reattaching?

  5. Map and unmap the shared memory block before risking VirtualFree.

  6. Further effort at preventing memory map dump from affecting the results.

  7. Remove Windows module-list-dumping code.

  8. Dump full memory maps around failing Windows reattach code.

  9. Get still more info about Windows can't-reattach-to-shared-memory errors.

  10. Get more info about Windows can't-reattach-to-shared-memory errors.

  11. Try to get some info about Windows can't-reattach-to-shared-memory errors.