Horton

Thread

Commits

  1. Check for STATUS_DELETE_PENDING on Windows.

  2. Fix our Windows stat() emulation to handle file sizes > 4GB.

  3. Attempt to handle pending-delete files on Windows

  1. stat() vs ERROR_DELETE_PENDING, round N + 1

    Thomas Munro <thomas.munro@gmail.com> — 2021-09-01T22:10:41Z

    Hi,

Back 2017, Michael and Magus apparently fixed a bug report[1] about
failing basebackups on Windows due to its concurrent file access
semantics:

commit 9951741bbeb3ec37ca50e9ce3df1808c931ff6a6
Author: Magnus Hagander <magnus@hagander.net>
Date:   Wed Jan 4 10:48:30 2017 +0100

    Attempt to handle pending-delete files on Windows

I think this has been re-broken by:

commit bed90759fcbcd72d4d06969eebab81e47326f9a2
Author: Tom Lane <tgl@sss.pgh.pa.us>
Date:   Fri Oct 9 16:20:12 2020 -0400

    Fix our Windows stat() emulation to handle file sizes > 4GB.

There's code in there that appears to understand the
ERROR_PENDING_DELETE stuff, but it seems to be too late, as this bit
will fail with ERROR_ACCESS_DENIED first:

        /* fast not-exists check */
        if (GetFileAttributes(name) == INVALID_FILE_ATTRIBUTES)
        {
                _dosmaperr(GetLastError());
                return -1;
        }

... and if you comment that out, then the CreateFile() call will fail
and we'll return before we get to the code that purports to grok
pending deletes.  I don't really understand that code, but I can
report that it's not reached.

This came up because in our work on AIO, we have extra io worker
processes that might have file handles open even in a single session
scenario like 010_pg_basebackup.pl, so we make these types of problems
more likely to hit (hence also my CF entry to fix a related problem in
DROP TABLESPACE).  But that's just chance: I assume basebackup could
fail for anyone in 14 for the same reason due to any other backend
that hasn't processed a sinval to close the file yet.

Perhaps we need some combination of the old way (that apparently knew
how to detect pending deletes) and the new way (that knows about large
files)?

[1] https://www.postgresql.org/message-id/flat/20160712083220.1426.58667%40wrigleys.postgresql.org

  2. Re: stat() vs ERROR_DELETE_PENDING, round N + 1

    Thomas Munro <thomas.munro@gmail.com> — 2021-09-02T10:28:09Z

    On Thu, Sep 2, 2021 at 10:10 AM Thomas Munro <thomas.munro@gmail.com> wrote:
> Perhaps we need some combination of the old way (that apparently knew
> how to detect pending deletes) and the new way (that knows about large
> files)?

I tried that, but as far as I can tell, the old approach didn't really
work either :-(

A disruptive solution that works in my tests: we could reuse the
global barrier proposed in CF #2962.  If you see EACCES, ask every
backend to close all vfds at their next CFI() and wait for them all to
finish, and then retry.  If you get EACCES again it really means
EACCES, but you'll very probably get ENOENT.

The cheapest solution would be to assume EACCES really means ENOENT,
but that seems unacceptably incorrect.

I suspect it might be possible to use underdocumented/unstable NtXXX()
interfaces to get at the information, but I don't know much about
that.

Is there another way that is cheap, correct and documented?

  3. Re: stat() vs ERROR_DELETE_PENDING, round N + 1

    Tom Lane <tgl@sss.pgh.pa.us> — 2021-09-02T10:31:51Z

    Thomas Munro <thomas.munro@gmail.com> writes:
> A disruptive solution that works in my tests: we could reuse the
> global barrier proposed in CF #2962.  If you see EACCES, ask every
> backend to close all vfds at their next CFI() and wait for them all to
> finish, and then retry.  If you get EACCES again it really means
> EACCES, but you'll very probably get ENOENT.

That seems quite horrid :-(.  But if it works, doesn't that mean that
somewhere we are opening a problematic file without the correct
sharing flags?

			regards, tom lane

  4. Re: stat() vs ERROR_DELETE_PENDING, round N + 1

    Thomas Munro <thomas.munro@gmail.com> — 2021-09-02T10:51:06Z

    On Thu, Sep 2, 2021 at 10:31 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Thomas Munro <thomas.munro@gmail.com> writes:
> > A disruptive solution that works in my tests: we could reuse the
> > global barrier proposed in CF #2962.  If you see EACCES, ask every
> > backend to close all vfds at their next CFI() and wait for them all to
> > finish, and then retry.  If you get EACCES again it really means
> > EACCES, but you'll very probably get ENOENT.
>
> That seems quite horrid :-(.  But if it works, doesn't that mean that
> somewhere we are opening a problematic file without the correct
> sharing flags?

I'm no expert, but not AFAICS.  We managed to delete the file while
some other backend had it open, which FILE_SHARE_DELETE allowed.  We
just can't open it or create a new file with the same name until it's
really gone (all handles closed).

  5. Re: stat() vs ERROR_DELETE_PENDING, round N + 1

    Tom Lane <tgl@sss.pgh.pa.us> — 2021-09-02T11:12:41Z

    Thomas Munro <thomas.munro@gmail.com> writes:
> On Thu, Sep 2, 2021 at 10:31 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> That seems quite horrid :-(.  But if it works, doesn't that mean that
>> somewhere we are opening a problematic file without the correct
>> sharing flags?

> I'm no expert, but not AFAICS.  We managed to delete the file while
> some other backend had it open, which FILE_SHARE_DELETE allowed.  We
> just can't open it or create a new file with the same name until it's
> really gone (all handles closed).

Right, but we shouldn't ever need to access such a file --- we couldn't do
so on Unix, certainly.  So for the open() case, it's sufficient to return
ENOENT, and the problem is only to make sure that that's what we return if
the underlying error is ERROR_DELETE_PENDING.

It's harder if the desire is to create a new file of the same name.
I'm inclined to think that the best answer might be "if it hurts,
don't do that".  We should not have such a case for ordinary relation
files or WAL files, but maybe there are individual other cases where
some redesign is indicated?

			regards, tom lane

  6. Re: stat() vs ERROR_DELETE_PENDING, round N + 1

    Thomas Munro <thomas.munro@gmail.com> — 2021-09-02T12:44:01Z

    On Thu, Sep 2, 2021 at 11:12 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Thomas Munro <thomas.munro@gmail.com> writes:
> > I'm no expert, but not AFAICS.  We managed to delete the file while
> > some other backend had it open, which FILE_SHARE_DELETE allowed.  We
> > just can't open it or create a new file with the same name until it's
> > really gone (all handles closed).
>
> Right, but we shouldn't ever need to access such a file --- we couldn't do
> so on Unix, certainly.  So for the open() case, it's sufficient to return
> ENOENT, and the problem is only to make sure that that's what we return if
> the underlying error is ERROR_DELETE_PENDING.

Yeah.  The problem is that it still shows up in directory listings
AFAIK, so something like basebackup.c sees it, and even if it didn't,
it reads the directory, and then stats the files, and then opens the
files at different times.  The non-public API way to ask for the real
reason after such a failure would apparently be to call
NtFileCreate(), which can return STATUS_DELETE_PENDING.  I don't know
what complications that might involve, but I see now that we have code
that digs such non-public APIs out of ntdll.dll already (for long dead
OS versions only).  Hmm.

(Another thing you can't do is delete the directory that contains such
a file, which is a problem for DROP TABLESPACE and the reason I
developed the global barrier thing.)

> It's harder if the desire is to create a new file of the same name.
> I'm inclined to think that the best answer might be "if it hurts,
> don't do that".  We should not have such a case for ordinary relation
> files or WAL files, but maybe there are individual other cases where
> some redesign is indicated?

I guess GetNewRelFileNode()’s dilemma branch is an example; it'd
probably be nicer to users to treat a pending-deleted file as a
collision.

        if (access(rpath, F_OK) == 0)
        {
            /* definite collision */
            collides = true;
        }
        else
        {
            /*
             * Here we have a little bit of a dilemma: if errno is something
             * other than ENOENT, should we declare a collision and loop? In
             * practice it seems best to go ahead regardless of the errno.  If
             * there is a colliding file we will get an smgr failure when we
             * attempt to create the new relation file.
             */
            collides = false;
        }

  7. Re: stat() vs ERROR_DELETE_PENDING, round N + 1

    Thomas Munro <thomas.munro@gmail.com> — 2021-09-02T13:01:43Z

    On Fri, Sep 3, 2021 at 12:44 AM Thomas Munro <thomas.munro@gmail.com> wrote:
> NtFileCreate()

Erm, that's spelled NtCreateFile.  I see Michael mentioned this
before[1]; I don't think it's only available in kernel mode though,
the docs[2] say "This function is the user-mode equivalent to the
ZwCreateFile function", and other open source user space stuff is
using it.  It's explicitly internal and subject to change though,
hence my desire to avoid it.

[1] https://www.postgresql.org/message-id/flat/a9c76882-27c7-9c92-7843-21d5521b70a9%40postgrespro.ru
[2] https://docs.microsoft.com/en-us/windows/win32/api/winternl/nf-winternl-ntcreatefile

  8. Re: stat() vs ERROR_DELETE_PENDING, round N + 1

    Kyotaro Horiguchi <horikyota.ntt@gmail.com> — 2021-09-03T02:01:03Z

    At Fri, 3 Sep 2021 01:01:43 +1200, Thomas Munro <thomas.munro@gmail.com> wrote in 
> On Fri, Sep 3, 2021 at 12:44 AM Thomas Munro <thomas.munro@gmail.com> wrote:
> > NtFileCreate()
> 
> Erm, that's spelled NtCreateFile.  I see Michael mentioned this
> before[1]; I don't think it's only available in kernel mode though,
> the docs[2] say "This function is the user-mode equivalent to the
> ZwCreateFile function", and other open source user space stuff is
> using it.  It's explicitly internal and subject to change though,
> hence my desire to avoid it.
> 
> [1] https://www.postgresql.org/message-id/flat/a9c76882-27c7-9c92-7843-21d5521b70a9%40postgrespro.ru
> [2] https://docs.microsoft.com/en-us/windows/win32/api/winternl/nf-winternl-ntcreatefile

Might be stupid, if a delete-pending'ed file can obstruct something,
couldn't we change unlink on Windows to rename to a temporary random
name then remove it?  We do something like it explicitly while WAL
file removal. (It may cause degradation on bulk file deletion, and we
may need further fix so that such being-deleted files are excluded
while running a directory scan, though..)

However, looking [1], with that strategy there may be a case where
such "deleted" files may be left alone forever, though.


[1] https://www.postgresql.org/message-id/002101d79fc2%24c96dff60%245c49fe20%24%40gmail.com

regards.

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

  9. Re: stat() vs ERROR_DELETE_PENDING, round N + 1

    Thomas Munro <thomas.munro@gmail.com> — 2021-09-05T13:32:55Z

    On Fri, Sep 3, 2021 at 2:01 PM Kyotaro Horiguchi
<horikyota.ntt@gmail.com> wrote:
> Might be stupid, if a delete-pending'ed file can obstruct something,
> couldn't we change unlink on Windows to rename to a temporary random
> name then remove it?  We do something like it explicitly while WAL
> file removal. (It may cause degradation on bulk file deletion, and we
> may need further fix so that such being-deleted files are excluded
> while running a directory scan, though..)
>
> However, looking [1], with that strategy there may be a case where
> such "deleted" files may be left alone forever, though.

It's a good idea.  I tested it and it certainly does fix the
basebackup problem I've seen (experimental patch attached).  But,
yeah, I'm also a bit worried that that path could be fragile and need
special handling in lots of places.

I also tried writing a new open() wrapper using the lower level
NtCreateFile() interface, and then an updated stat() wrapper built on
top of that.  As a non-Windows person, getting that to (mostly) work
involved a fair amount of suffering.  I can share that if someone is
interested, but while learning about that family of interfaces, I
realised we could keep the existing Win32-based code, but also
retrieve the NT status, leading to a very small change (experimental
patch attached).

The best idea is probably to set FILE_DISPOSITION_DELETE |
FILE_DISPOSITION_POSIX_SEMANTICS before unlinking.  This appears to be
a silver bullet, but isn't available on ancient Windows releases that
we support, or file systems other than local NTFS.  So maybe we need a
combination of that + STATUS_DELETE_PENDING as shown in the attached.
I'll look into that next.

  10. Re: stat() vs ERROR_DELETE_PENDING, round N + 1

    Andres Freund <andres@anarazel.de> — 2021-09-05T21:44:37Z

    Hi,

On 2021-09-06 01:32:55 +1200, Thomas Munro wrote:
> It's a good idea.  I tested it and it certainly does fix the
> basebackup problem I've seen (experimental patch attached).  But,
> yeah, I'm also a bit worried that that path could be fragile and need
> special handling in lots of places.

It's also expensive-ish.


> I also tried writing a new open() wrapper using the lower level
> NtCreateFile() interface, and then an updated stat() wrapper built on
> top of that.  As a non-Windows person, getting that to (mostly) work
> involved a fair amount of suffering.  I can share that if someone is
> interested, but while learning about that family of interfaces, I
> realised we could keep the existing Win32-based code, but also
> retrieve the NT status, leading to a very small change (experimental
> patch attached).

Is it guaranteed, or at least reliable, that the status we fetch with
RtlGetLastNtStatus is actually from the underlying filesystem operation,
rather than some other work that happens during the win32->nt translation?
E.g. a memory allocation or such? Presumably most of such work happens before
the actual nt "syscall", but ...


> The best idea is probably to set FILE_DISPOSITION_DELETE |
> FILE_DISPOSITION_POSIX_SEMANTICS before unlinking.  This appears to be
> a silver bullet, but isn't available on ancient Windows releases that
> we support, or file systems other than local NTFS.  So maybe we need a
> combination of that + STATUS_DELETE_PENDING as shown in the attached.
> I'll look into that next.

When was that introduced?

I'd be ok to only fix these bugs on e.g. Win10, Win Server 2019, Win Server
2016 or such. I don't think we need to support OSs that the vendor doesn't
support - and I wouldn't count "only security fixes" as support in this
context.
                        main            extended
Windows 10              Oct 14, 2025
Windows Server 2019     Jan 9, 2024     Jan 9, 2029
Windows Server 2016     Jan 11, 2022    Jan 12, 2027
Windows 7               Jan 13, 2015    Jan 14, 2020
Windows Vista           Apr 10, 2012    Apr 11, 2017


One absurd detail here is that the deault behaviour changed sometime in
Windows 10's lifetime:
https://stackoverflow.com/questions/60424732/did-the-behaviour-of-deleted-files-open-with-fileshare-delete-change-on-windows

"The behavior changed in recent releases of Windows 10 -- without notice
AFAIK. DeleteFileW now tries to use POSIX semantics if the filesystem supports
it. NTFS does."


>  #ifndef FRONTEND
> -	Assert(pgwin32_signal_event != NULL);	/* small chance of pg_usleep() */
> +	/* XXX When called by stat very early on, this fails! */
> +	//Assert(pgwin32_signal_event != NULL);	/* small chance of pg_usleep() */
>  #endif

Perhaps we should move the win32 signal initialization into StartupHacks()?
There's some tension around it using ereport(), and MemoryContextInit() only
being called a tad later, but that seems resolvable.


> +	 * Our open wrapper will report STATUS_DELETE_PENDING as ENOENT.  We pass
> +	 * in a special private flag to say that it's _pgstat64() calling, to
> +	 * activate a mode that allows directories to be opened for limited
> +	 * purposes.
> +	 *
> +	 * XXX Think about fd pressure, since we're opening an fd?
>  	 */

If I understand https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/getmaxstdio?view=msvc-160
etc correctly, it looks like there is. But only at the point we do
_open_osfhandle(). So perhaps we should a pgwin32_open() version returning a
handle and make pgwin32_open() a wrapper around that?


Greetings,

Andres Freund

  11. Re: stat() vs ERROR_DELETE_PENDING, round N + 1

    Tom Lane <tgl@sss.pgh.pa.us> — 2021-09-05T21:55:00Z

    Andres Freund <andres@anarazel.de> writes:
> On 2021-09-06 01:32:55 +1200, Thomas Munro wrote:
>> The best idea is probably to set FILE_DISPOSITION_DELETE |
>> FILE_DISPOSITION_POSIX_SEMANTICS before unlinking.  This appears to be
>> a silver bullet, but isn't available on ancient Windows releases that
>> we support, or file systems other than local NTFS.  So maybe we need a
>> combination of that + STATUS_DELETE_PENDING as shown in the attached.
>> I'll look into that next.

> When was that introduced?

Googling says that it was introduced in Win10, although in RS2 (version
1703, general availability in 4/2017) not the initial release.

> I'd be ok to only fix these bugs on e.g. Win10, Win Server 2019, Win Server
> 2016 or such.

Yeah, particularly if the fix is trivial on the newer systems and
incredibly complicated otherwise.  Between the effort needed and
the risk of introducing new bugs, I'm really not excited about
an invasive fix for this.

			regards, tom lane

  12. Re: stat() vs ERROR_DELETE_PENDING, round N + 1

    Thomas Munro <thomas.munro@gmail.com> — 2021-09-05T22:22:34Z

    On Mon, Sep 6, 2021 at 9:55 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Andres Freund <andres@anarazel.de> writes:
> > I'd be ok to only fix these bugs on e.g. Win10, Win Server 2019, Win Server
> > 2016 or such.
>
> Yeah, particularly if the fix is trivial on the newer systems and
> incredibly complicated otherwise.  Between the effort needed and
> the risk of introducing new bugs, I'm really not excited about
> an invasive fix for this.

If DeleteFile() is automatically using
FILE_DISPOSITION_POSIX_SEMANTICS by default when possible on recent
releases as per the SO link that Andres posted above ("18363.657
definitely has the new behavior"), then that's great news and maybe we
shouldn't even bother to try to request that mode ourselves explicitly
(eg in some kind of unlink wrapper).  Then we'd need just one
accomodation for older systems and non-NTFS systems, not two, and I
currently think that should be the short and sweet approach shown in
0001-Handle-STATUS_DELETE_PENDING-on-Windows.patch, with some tidying
and adjustments per feedback.

  13. Re: stat() vs ERROR_DELETE_PENDING, round N + 1

    Michael Paquier <michael@paquier.xyz> — 2021-09-06T06:36:33Z

    > If DeleteFile() is automatically using
> FILE_DISPOSITION_POSIX_SEMANTICS by default when possible on recent
> releases as per the SO link that Andres posted above ("18363.657
> definitely has the new behavior"), then that's great news and maybe we
> shouldn't even bother to try to request that mode ourselves explicitly
> (eg in some kind of unlink wrapper).  Then we'd need just one
> accomodation for older systems and non-NTFS systems, not two, and I
> currently think that should be the short and sweet approach shown in
> 0001-Handle-STATUS_DELETE_PENDING-on-Windows.patch, with some tidying
> and adjustments per feedback.

Having a non-invasive fix for this long-standing issue would be really
great, even if that means reducing the scope of systems where this can
be fixed.

The last time I poked at the bear (54fb8c7d), there was a test posted
by Alexander Lakhin that was really useful in making sure that
concurrency is correctly handled when a file is unlinked:
https://www.postgresql.org/message-id/c3427edf-d7c0-ff57-90f6-b5de3bb62709@gmail.com

It worked with VS but not on MinGW.  How does your patch react to this
test?
--
Michael

  14. Re: stat() vs ERROR_DELETE_PENDING, round N + 1

    Thomas Munro <thomas.munro@gmail.com> — 2021-09-06T11:45:41Z

    On Mon, Sep 6, 2021 at 9:44 AM Andres Freund <andres@anarazel.de> wrote:
> Is it guaranteed, or at least reliable, that the status we fetch with
> RtlGetLastNtStatus is actually from the underlying filesystem operation,
> rather than some other work that happens during the win32->nt translation?
> E.g. a memory allocation or such? Presumably most of such work happens before
> the actual nt "syscall", but ...

I don't know.  I know at least that it's thread-local, so that's
something.  I guess it's plausible that CreateFile() might want to
free a temporary buffer that it used for conversion to NT pathname
format, and whatever code it uses to do that might clobber the NT
status.  Nothing like that seems to happen in common cases though, and
I guess it would also be clobbered on success.  Frustrating.

Alright then, here also is the version that bypasses CreateFile() and
goes straight to NtCreateFile().  This way, the status can't possibly
be clobbered before we see it, but maybe there are other risks due to
using a much wider set of unstable ntdll interfaces...

Both versions pass all tests on CI, including the basebackup one in a
scenario where an unlinked file has an open descriptor, but still need
a bit more tidying.

> "The behavior changed in recent releases of Windows 10 -- without notice
> AFAIK. DeleteFileW now tries to use POSIX semantics if the filesystem supports
> it. NTFS does."

Nice find.  I wonder if this applies also to rename()...

> >  #ifndef FRONTEND
> > -     Assert(pgwin32_signal_event != NULL);   /* small chance of pg_usleep() */
> > +     /* XXX When called by stat very early on, this fails! */
> > +     //Assert(pgwin32_signal_event != NULL); /* small chance of pg_usleep() */
> >  #endif
>
> Perhaps we should move the win32 signal initialization into StartupHacks()?
> There's some tension around it using ereport(), and MemoryContextInit() only
> being called a tad later, but that seems resolvable.

The dependencies among open(), pg_usleep(),
pgwin32_signal_initialize() and read_backend_variables() are not very
nice.  I don't have a fix for that yet.

> > +      * XXX Think about fd pressure, since we're opening an fd?

> If I understand https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/getmaxstdio?view=msvc-160
> etc correctly, it looks like there is. But only at the point we do
> _open_osfhandle(). So perhaps we should a pgwin32_open() version returning a
> handle and make pgwin32_open() a wrapper around that?

Yeah.  Done, in both variants.

I haven't tried it, but I suspect the difference between stat() and
lstat() could be handled with FILE_OPEN_REPARSE_POINT (as
NtCreateFile() calls it) or FILE_FLAG_OPEN_REPARSE_POINT (as
CreateFile() calls it).

  15. Re: stat() vs ERROR_DELETE_PENDING, round N + 1

    Thomas Munro <thomas.munro@gmail.com> — 2021-09-06T13:04:46Z

    On Mon, Sep 6, 2021 at 6:36 PM Michael Paquier <michael@paquier.xyz> wrote:
> Having a non-invasive fix for this long-standing issue would be really
> great, even if that means reducing the scope of systems where this can
> be fixed.

I hope those patches fix at least the basebackup problem on all
relevant OS versions, until the better POSIX thing is everywhere (they
can't fix all related problems though, since zombie files still stop
you creating new ones with the same name or deleting the containing
directory).   I didn't try to find out how far back those APIs go, but
they look ancient/fundamental and widely used by other software...
But do they qualify as non-invasive?

> The last time I poked at the bear (54fb8c7d), there was a test posted
> by Alexander Lakhin that was really useful in making sure that
> concurrency is correctly handled when a file is unlinked:
> https://www.postgresql.org/message-id/c3427edf-d7c0-ff57-90f6-b5de3bb62709@gmail.com

Thanks.  It's a confusing topic with many inconclusive threads.

> It worked with VS but not on MinGW.  How does your patch react to this
> test?

Thanks.  Adding Alexander in CC in case he has ideas/feedback.

  16. Re: stat() vs ERROR_DELETE_PENDING, round N + 1

    Alexander Law <exclusion@gmail.com> — 2021-09-07T06:00:01Z

    06.09.2021 16:04, Thomas Munro wrote:
> On Mon, Sep 6, 2021 at 6:36 PM Michael Paquier <michael@paquier.xyz> wrote:
>> The last time I poked at the bear (54fb8c7d), there was a test posted
>> by Alexander Lakhin that was really useful in making sure that
>> concurrency is correctly handled when a file is unlinked:
>> https://www.postgresql.org/message-id/c3427edf-d7c0-ff57-90f6-b5de3bb62709@gmail.com
The new approach looks very promising. Knowing that the file is really
in the DELETE_PENDING state simplifies a lot.
I've tested the patch v2_0001_Check... with my demo tests [1] and [2],
and it definitely works.

[1]
https://www.postgresql.org/message-id/e5179494-715e-f8a3-266b-0cf52adac8f4%40gmail.com
[2]
https://www.postgresql.org/message-id/c3427edf-d7c0-ff57-90f6-b5de3bb62709@gmail.com

Best regards,
Alexander

  17. Re: stat() vs ERROR_DELETE_PENDING, round N + 1

    Michael Paquier <michael@paquier.xyz> — 2021-09-07T06:05:19Z

    On Tue, Sep 07, 2021 at 09:00:01AM +0300, Alexander Lakhin wrote:
> The new approach looks very promising. Knowing that the file is really
> in the DELETE_PENDING state simplifies a lot.
> I've tested the patch v2_0001_Check... with my demo tests [1] and [2],
> and it definitely works.

Oho, nice.  Just to be sure.  You are referring to
v2-0001-Check*.patch posted here, right?
https://www.postgresql.org/message-id/CA+hUKGKj3p+2AciBGacCf_cXE0JLCYevWHexvOpK6uL1+V-zag@mail.gmail.com
--
Michael

  18. Re: stat() vs ERROR_DELETE_PENDING, round N + 1

    Alexander Law <exclusion@gmail.com> — 2021-09-07T07:00:00Z

    Hello Michael,
07.09.2021 09:05, Michael Paquier wrote:
> On Tue, Sep 07, 2021 at 09:00:01AM +0300, Alexander Lakhin wrote:
>> The new approach looks very promising. Knowing that the file is really
>> in the DELETE_PENDING state simplifies a lot.
>> I've tested the patch v2_0001_Check... with my demo tests [1] and [2],
>> and it definitely works.
> Oho, nice.  Just to be sure.  You are referring to
> v2-0001-Check*.patch posted here, right?
> https://www.postgresql.org/message-id/CA+hUKGKj3p+2AciBGacCf_cXE0JLCYevWHexvOpK6uL1+V-zag@mail.gmail.com
Yes, i've tested that one, on the master branch (my tests needed a minor
modification due to PostgresNode changes).

Best regards,
Alexander

  19. Re: stat() vs ERROR_DELETE_PENDING, round N + 1

    Thomas Munro <thomas.munro@gmail.com> — 2021-09-10T05:04:09Z

    On Tue, Sep 7, 2021 at 7:00 PM Alexander Lakhin <exclusion@gmail.com> wrote:
> 07.09.2021 09:05, Michael Paquier wrote:
> > On Tue, Sep 07, 2021 at 09:00:01AM +0300, Alexander Lakhin wrote:
> >> The new approach looks very promising. Knowing that the file is really
> >> in the DELETE_PENDING state simplifies a lot.
> >> I've tested the patch v2_0001_Check... with my demo tests [1] and [2],
> >> and it definitely works.

> > Oho, nice.  Just to be sure.  You are referring to
> > v2-0001-Check*.patch posted here, right?
> > https://www.postgresql.org/message-id/CA+hUKGKj3p+2AciBGacCf_cXE0JLCYevWHexvOpK6uL1+V-zag@mail.gmail.com

> Yes, i've tested that one, on the master branch (my tests needed a minor
> modification due to PostgresNode changes).

Thanks very much!

Time to tidy up some loose ends.  There are a couple of judgement
calls involved.  Here's what Andres and I came up with in an off-list
chat.  Any different suggestions?

1.  I abandoned the "direct NtCreateFile()" version for now.  I guess
using more and wider unstable interfaces might expose us to greater
risk of silent API/behavior changes or have subtle bugs.  If we ever
have a concrete reason to believe that RtlGetLastNtStatus() is not
reliable here, we could reconsider.

2.  I dropped the assertion that the signal event has been created
before the first call to the open() wrapper.  Instead, I taught
pg_usleep() to fall back to plain old SleepEx() if the signal stuff
isn't up yet.  Other solutions are possible of course, but it struck
me as a bad idea to place initialisation ordering constraints on very
basic facilities like open() and stat().

I should point out explicitly that with this patch, stat() benefits
from open()'s tolerance for sharing violations, as a side effect.
That is, it'll retry for a short time in the hope that whoever opened
our file without allowing sharing will soon go away.  I don't know how
useful that bandaid loop really is in practice, but I don't see why
we'd want that for open() and not stat(), so this change seems good to
me on consistency grounds at the very least.

3.  We fixed the warnings about macro redefinition with #define
UMDF_USING_NTSTATUS and #include <ntstatus.h> in win32_port.h.  (Other
ideas considered: (1) Andres reported that it also works to move the
#include to ~12 files that need things from it, ie things that were
suppressed from windows.h by that macro and must now be had from
ntstatus.h, but the files you have to change are probably different in
back branches if we decide to do that, (2) I tried defining that macro
locally in files that need it, *before* including c.h/postgres.h, and
then locally include ntstatus.h afterwards, but that seems to violate
project style and generally seems weird.)

Another thing to point out explicitly is that I added a new file
src/port/win32ntdll.c, which is responsible for fishing out the NT
function pointers.  It was useful to be able to do that in the
abandoned NtCreateFile() variant because it needed three of them and I
could reduce boiler-plate noise with a static array of function names
to loop over.  In this version the array has just one element, but I'd
still rather centralise this stuff in one place and make it easy to
add any more of these that we eventually find a need for.

BTW, I also plan to help Victor get his "POSIX semantics" patch[1]
into the tree (and extend it to cover more ops).  That should make
these problems go away in a more complete way IIUC, but doesn't work
everywhere (not sure if we have any build farm animals where it
doesn't work, if so it might be nice to change that), so it's
complementary to this patch.  (My earlier idea that that stuff would
magically start happening for free on all relevant systems some time
soon has faded.)

[1] https://www.postgresql.org/message-id/flat/a529b660-da15-5b62-21a0-9936768210fd%40postgrespro.ru

  20. Re: stat() vs ERROR_DELETE_PENDING, round N + 1

    Thomas Munro <thomas.munro@gmail.com> — 2021-09-23T02:57:39Z

    On Fri, Sep 10, 2021 at 5:04 PM Thomas Munro <thomas.munro@gmail.com> wrote:
> On Tue, Sep 7, 2021 at 7:00 PM Alexander Lakhin <exclusion@gmail.com> wrote:
> > 07.09.2021 09:05, Michael Paquier wrote:
> > > On Tue, Sep 07, 2021 at 09:00:01AM +0300, Alexander Lakhin wrote:
> > >> The new approach looks very promising. Knowing that the file is really
> > >> in the DELETE_PENDING state simplifies a lot.
> > >> I've tested the patch v2_0001_Check... with my demo tests [1] and [2],
> > >> and it definitely works.

Since our handling of that stuff never really worked the way we wanted
(or if it did, then Windows' behaviour changed, possibly well over a
decade ago, from what I could dig up), this isn't an open item
candidate for 14 after all, it's a pre-existing condition.  So I
propose to push this fix to master only soon, and then let it stew
there for a little while to see how the buildfarm Windows variants and
the Windows hacker community testing on master react.  If it looks
good, we can back-patch it a bit later, perhaps some more convenient
time WRT the release.

I added a CF entry to see if anyone else wants to review it and get CI.

One small detail I'd like to draw attention to is this bit, which
differs slightly from the [non-working] previous attempts by mapping
to two different errors:

+         * If there's no O_CREAT flag, then we'll pretend the file is
+         * invisible.  With O_CREAT, we have no choice but to report that
+         * there's a file in the way (which wouldn't happen on Unix).

...

+            if (fileFlags & O_CREAT)
+                err = ERROR_FILE_EXISTS;
+            else
+                err = ERROR_FILE_NOT_FOUND;

  21. Re: stat() vs ERROR_DELETE_PENDING, round N + 1

    Juan José Santamaría Flecha <juanjo.santamaria@gmail.com> — 2021-09-23T09:05:47Z

    On Thu, Sep 23, 2021 at 4:58 AM Thomas Munro <thomas.munro@gmail.com> wrote:

>
> One small detail I'd like to draw attention to is this bit, which
> differs slightly from the [non-working] previous attempts by mapping
> to two different errors:
>
> +         * If there's no O_CREAT flag, then we'll pretend the file is
> +         * invisible.  With O_CREAT, we have no choice but to report that
> +         * there's a file in the way (which wouldn't happen on Unix).
>
> ...
>
> +            if (fileFlags & O_CREAT)
> +                err = ERROR_FILE_EXISTS;
> +            else
> +                err = ERROR_FILE_NOT_FOUND;
>

When GetTempFileName() finds a duplicated file name and the file is pending
for deletion, it fails with an "ERROR_ACCESS_DENIED" error code. That may
describe the situation better than "ERROR_FILE_EXISTS".

Regards,

Juan José Santamaría Flecha

  22. Re: stat() vs ERROR_DELETE_PENDING, round N + 1

    Thomas Munro <thomas.munro@gmail.com> — 2021-09-28T00:49:32Z

    On Thu, Sep 23, 2021 at 9:13 PM Juan José Santamaría Flecha
<juanjo.santamaria@gmail.com> wrote:
> On Thu, Sep 23, 2021 at 4:58 AM Thomas Munro <thomas.munro@gmail.com> wrote:
>> One small detail I'd like to draw attention to is this bit, which
>> differs slightly from the [non-working] previous attempts by mapping
>> to two different errors:
>>
>> +         * If there's no O_CREAT flag, then we'll pretend the file is
>> +         * invisible.  With O_CREAT, we have no choice but to report that
>> +         * there's a file in the way (which wouldn't happen on Unix).
>>
>> ...
>>
>> +            if (fileFlags & O_CREAT)
>> +                err = ERROR_FILE_EXISTS;
>> +            else
>> +                err = ERROR_FILE_NOT_FOUND;
>
>
> When GetTempFileName() finds a duplicated file name and the file is pending for deletion, it fails with an "ERROR_ACCESS_DENIED" error code. That may describe the situation better than "ERROR_FILE_EXISTS".

Thanks for looking.  Why do you think that's better?  I assume that's
just the usual NT->Win32 error conversion at work.

The only case I can think of so far in our tree where you'd notice
this change of errno for the O_CREAT case is relfilenode creation[1],
and there it's just a case of printing a different message.  Trying to
create a relfilenode that exists already in delete-pending state will
fail, but with this change we'll log the %m string for EEXIST instead
of EACCES (what you see today) or ENOENT (which seems nonsensical, "I
can't create your file because it doesn't exist", and what you'd get
with this patch if I didn't have the special case for O_CREAT).  So I
think that's pretty arguably an improvement.

As for how likely you are to reach that case... hmm, I don't know what
access() returns for a file in delete-pending state.  The docs say
"The function returns -1 if the named file does not exist or does not
have the given mode", so perhaps it returns 0 for such a case, in
which case we'll consider it a collision and keep searching for
another free relfilenode.  If that's the case, it's probably really
really unlikely you'll reach the case described in the previous
paragraph, so it probably doesn't matter much.

Do we have any other code paths where this finer point could cause
problems?  Looking around at code that handles EEXIST, most of it is
directory creation (unaffected by this patch), and then
src/port/mkdtemp.c for which this change is appropriate (it implements
POSIX mkdtemp(), which shouldn't report EACCES to its caller if
something it tries bumps into a delete-pending file, it should see
EEXIST and try a new name, which I think it will do with this patch,
through its call to open(O_CREAT | O_EXCL)).

[1] https://www.postgresql.org/message-id/flat/CA%2BhUKGJz_pZTF9mckn6XgSv69%2BjGwdgLkxZ6b3NWGLBCVjqUZA%40mail.gmail.com

  23. Re: stat() vs ERROR_DELETE_PENDING, round N + 1

    Juan José Santamaría Flecha <juanjo.santamaria@gmail.com> — 2021-09-29T10:26:25Z

    On Tue, Sep 28, 2021 at 2:50 AM Thomas Munro <thomas.munro@gmail.com> wrote:

> On Thu, Sep 23, 2021 at 9:13 PM Juan José Santamaría Flecha
> <juanjo.santamaria@gmail.com> wrote:
> > When GetTempFileName() finds a duplicated file name and the file is
> pending for deletion, it fails with an "ERROR_ACCESS_DENIED" error code.
> That may describe the situation better than "ERROR_FILE_EXISTS".
>
> Thanks for looking.  Why do you think that's better?  I assume that's
> just the usual NT->Win32 error conversion at work.
>
> When a function returns an error caused by accessing a file
in DELETE_PENDING you should expect an EACCES. Nonetheless, if we can
emulate a POSIX behaviour by mapping it to EEXIST, that works for me. I
also consider that having the logic for DELETE_PENDING in a single function
is an improvement.

Regards,

Juan José Santamaría Flecha

  24. Re: stat() vs ERROR_DELETE_PENDING, round N + 1

    Daniel Gustafsson <daniel@yesql.se> — 2021-11-03T11:02:53Z

    This patch doesn't compile on Windows according to Appveyor, seemingly because
of a syntax error in the new win32ntdll.h file, but the MSVC logs are hard on
the eye so it might be unrelated.

--
Daniel Gustafsson		https://vmware.com/

  25. Re: stat() vs ERROR_DELETE_PENDING, round N + 1

    Daniel Gustafsson <daniel@yesql.se> — 2021-12-01T14:11:53Z

    > On 3 Nov 2021, at 12:02, Daniel Gustafsson <daniel@yesql.se> wrote:
> 
> This patch doesn't compile on Windows according to Appveyor, seemingly because
> of a syntax error in the new win32ntdll.h file, but the MSVC logs are hard on
> the eye so it might be unrelated.

As the thread has stalled with a patch that doesn't apply, I'm marking this
patch Returned with Feedback.  Please feel free to resubmit when a new patch is
ready.

--
Daniel Gustafsson		https://vmware.com/

  26. Re: stat() vs ERROR_DELETE_PENDING, round N + 1

    Thomas Munro <thomas.munro@gmail.com> — 2021-12-04T05:18:32Z

    On Thu, Dec 2, 2021 at 3:11 AM Daniel Gustafsson <daniel@yesql.se> wrote:
> > On 3 Nov 2021, at 12:02, Daniel Gustafsson <daniel@yesql.se> wrote:
> >
> > This patch doesn't compile on Windows according to Appveyor, seemingly because
> > of a syntax error in the new win32ntdll.h file, but the MSVC logs are hard on
> > the eye so it might be unrelated.
>
> As the thread has stalled with a patch that doesn't apply, I'm marking this
> patch Returned with Feedback.  Please feel free to resubmit when a new patch is
> ready.

I think this was broken by WIN32_LEAN_AND_MEAN (and since gained a
merge conflict, but that's easy to fix).  I'll try to figure out the
right system header hacks to unbreak it...

  27. Re: stat() vs ERROR_DELETE_PENDING, round N + 1

    Thomas Munro <thomas.munro@gmail.com> — 2021-12-06T08:17:54Z

    On Sat, Dec 4, 2021 at 6:18 PM Thomas Munro <thomas.munro@gmail.com> wrote:
> On Thu, Dec 2, 2021 at 3:11 AM Daniel Gustafsson <daniel@yesql.se> wrote:
> > > This patch doesn't compile on Windows according to Appveyor, seemingly because
> > > of a syntax error in the new win32ntdll.h file, but the MSVC logs are hard on
> > > the eye so it might be unrelated.

> I think this was broken by WIN32_LEAN_AND_MEAN (and since gained a
> merge conflict, but that's easy to fix).  I'll try to figure out the
> right system header hacks to unbreak it...

Short version: It needed <winternl.h>.

Long version:  Where Unix shares headers between user space and kernel
with #ifdef _KERNEL, today I learned that Windows seems to have two
universes of headers, with some stuff defined in both places.  You
can't cross the streams.  I had already defined UMDF_USING_NTSTATUS,
which tells <windows.h> that you're planning to include <ntstatus.h>,
to avoid a bunch of double-definitions (the other approach I'd found
on the 'net was to #define and #undef WIN32_NO_STATUS in the right
places), but when WIN32_LEAN_AND_MEAN was added, that combination lost
the definition of NTSTATUS, which is needed by various macros like
WAIT_OBJECT_0 (it's used in casts).  It's supposed to come from
<ntdef.h>, but if you include that directly you get more double
definitions of other random stuff.  Eventually I learned that
<winternl.h> fixes that.  No doubt this is eroding the gains made by
WIN32_LEAN_AND_MEAN, but I don't see how to avoid it until we do the
work to stop including <windows.h> in win32_port.h.  Well, I do know
one way...  I noticed that <bcrypt.h> just defines NTSTATUS itself if
it sees that <ntdef.h> hasn't been included (by testing its include
guard).  I tried that and it worked, but it seems pretty ugly and not
something that we should be doing.

  28. Re: stat() vs ERROR_DELETE_PENDING, round N + 1

    Thomas Munro <thomas.munro@gmail.com> — 2021-12-09T08:16:57Z

    On Mon, Dec 6, 2021 at 9:17 PM Thomas Munro <thomas.munro@gmail.com> wrote:
> On Sat, Dec 4, 2021 at 6:18 PM Thomas Munro <thomas.munro@gmail.com> wrote:
> > I think this was broken by WIN32_LEAN_AND_MEAN (and since gained a
> > merge conflict, but that's easy to fix).  I'll try to figure out the
> > right system header hacks to unbreak it...
>
> Short version: It needed <winternl.h>.

Slightly improvement: now I include <winternl.h> only from
src/port/open.c and src/port/win32ntdll.c, so I avoid the extra
include for the other ~1500 translation units.  That requires a small
extra step to work, see comment in win32ntdll.h.  I checked that this
still cross-compiles OK under mingw on Linux.  This is the version
that I'm planning to push to master only tomorrow if there are no
objections.

  29. Re: stat() vs ERROR_DELETE_PENDING, round N + 1

    Thomas Munro <thomas.munro@gmail.com> — 2021-12-10T03:25:07Z

    On Thu, Dec 9, 2021 at 9:16 PM Thomas Munro <thomas.munro@gmail.com> wrote:
> Slightly improvement: now I include <winternl.h> only from
> src/port/open.c and src/port/win32ntdll.c, so I avoid the extra
> include for the other ~1500 translation units.  That requires a small
> extra step to work, see comment in win32ntdll.h.  I checked that this
> still cross-compiles OK under mingw on Linux.  This is the version
> that I'm planning to push to master only tomorrow if there are no
> objections.

Done.  I'll keep an eye on the build farm.