Horton

Re: "could not reattach to shared memory" on buildfarm member dory

Tom Lane <tgl@sss.pgh.pa.us>

View thread View on postgr.es
From: Tom Lane <tgl@sss.pgh.pa.us>
To: Noah Misch <noah@leadboat.com>
Cc: Heath Lord <heath.lord@crunchydata.com>, Thomas Munro <thomas.munro@enterprisedb.com>, Stephen Frost <sfrost@snowman.net>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2018-05-01T00:01:40Z
Lists: pgsql-hackers 
I wrote:
> The solution I was thinking about last night was to have
> PGSharedMemoryReAttach call MapViewOfFileEx to map the shared memory
> segment at an unspecified address, then unmap it, then call VirtualFree,
> and finally call MapViewOfFileEx with the real target address.  The idea
> here is to get these various DLLs to set up any memory allocation pools
> they're going to set up before we risk doing VirtualFree.  I am not,
> at this point, convinced this will fix it :-( ... but I'm not sure what
> else to try.

So the answer is that that doesn't help at all.

It's clear from dory's results that something is causing a 4MB chunk
of memory to get reserved in the process's address space, sometimes.
It might happen during the main MapViewOfFileEx call, or during the
preceding VirtualFree, or with my map/unmap dance in place, it might
happen during that.  Frequently it doesn't happen at all, at least not
before the point where we've successfully done MapViewOfFileEx.  But
if it does happen, and the chunk happens to get put in a spot that
overlaps where we want to put the shmem block, kaboom.

What seems like a plausible theory at this point is that the apparent
asynchronicity is due to the allocation being triggered by a different
thread, and the fact that our added monitoring code seems to make the
failure more likely can be explained by that code changing the timing.
But what thread could it be?  It doesn't really look to me like either
the signal thread or the timer thread could eat 4MB.  syslogger.c
also spawns a thread, on Windows, but AFAICS that's not being used in
this test configuration.  Maybe the reason dory is showing the problem
is something or other is spawning a thread we don't even know about?

I'm going to go put a 1-sec sleep into the beginning of
PGSharedMemoryReAttach and see if that changes anything.  If I'm right
that this is being triggered by another thread, that should allow the
other thread to do its thing (at least most of the time) so that the
failure rate ought to go way down.

Even if that does happen, I'm at a loss for a reasonable way to fix it
for real.  Is there a way to seize control of a Windows process so that
there are no other running threads?  Any other ideas?

			regards, tom lane

Commits

  1. Avoid "could not reattach" by providing space for concurrent allocation.

  2. Assert that pgwin32_signal_initialize() has been called early enough.

  3. Remove investigative code for can't-reattach-to-shared-memory errors.

  4. Does it help to wait before reattaching?

  5. Map and unmap the shared memory block before risking VirtualFree.

  6. Further effort at preventing memory map dump from affecting the results.

  7. Remove Windows module-list-dumping code.

  8. Dump full memory maps around failing Windows reattach code.

  9. Get still more info about Windows can't-reattach-to-shared-memory errors.

  10. Get more info about Windows can't-reattach-to-shared-memory errors.

  11. Try to get some info about Windows can't-reattach-to-shared-memory errors.