Horton

Re: "could not reattach to shared memory" on buildfarm member dory

Noah Misch <noah@leadboat.com>

View thread View on postgr.es
From: Noah Misch <noah@leadboat.com>
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Heath Lord <heath.lord@crunchydata.com>, Thomas Munro <thomas.munro@enterprisedb.com>, Stephen Frost <sfrost@snowman.net>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2018-05-01T02:59:14Z
Lists: pgsql-hackers 
On Mon, Apr 30, 2018 at 08:01:40PM -0400, Tom Lane wrote:
> It's clear from dory's results that something is causing a 4MB chunk
> of memory to get reserved in the process's address space, sometimes.
> It might happen during the main MapViewOfFileEx call, or during the
> preceding VirtualFree, or with my map/unmap dance in place, it might
> happen during that.  Frequently it doesn't happen at all, at least not
> before the point where we've successfully done MapViewOfFileEx.  But
> if it does happen, and the chunk happens to get put in a spot that
> overlaps where we want to put the shmem block, kaboom.
> 
> What seems like a plausible theory at this point is that the apparent
> asynchronicity is due to the allocation being triggered by a different
> thread, and the fact that our added monitoring code seems to make the
> failure more likely can be explained by that code changing the timing.
> But what thread could it be?  It doesn't really look to me like either
> the signal thread or the timer thread could eat 4MB.  syslogger.c
> also spawns a thread, on Windows, but AFAICS that's not being used in
> this test configuration.  Maybe the reason dory is showing the problem
> is something or other is spawning a thread we don't even know about?

Likely some privileged daemon is creating a thread in every new process.  (On
Windows, it's not unusual for one process to create a thread in another
process.)  We don't have good control over that.

> I'm at a loss for a reasonable way to fix it
> for real.  Is there a way to seize control of a Windows process so that
> there are no other running threads?

I think not.

> Any other ideas?

PostgreSQL could retry the whole process creation, analogous to
internal_forkexec() retries.  Have the failed process exit after recording the
fact that it couldn't attach.  Make the postmaster notice and spawn a
replacement.  Give up after 100 failed attempts.

Commits

  1. Avoid "could not reattach" by providing space for concurrent allocation.

  2. Assert that pgwin32_signal_initialize() has been called early enough.

  3. Remove investigative code for can't-reattach-to-shared-memory errors.

  4. Does it help to wait before reattaching?

  5. Map and unmap the shared memory block before risking VirtualFree.

  6. Further effort at preventing memory map dump from affecting the results.

  7. Remove Windows module-list-dumping code.

  8. Dump full memory maps around failing Windows reattach code.

  9. Get still more info about Windows can't-reattach-to-shared-memory errors.

  10. Get more info about Windows can't-reattach-to-shared-memory errors.

  11. Try to get some info about Windows can't-reattach-to-shared-memory errors.