Horton

Re: Windows pg_basebackup unable to create >2GB pg_wal.tar tarballs ("could not close file: Invalid argument" when creating pg_wal.tar of size ~ 2^31 bytes)

Davinder Singh <davinder.singh@enterprisedb.com>

View thread View on postgr.es
From: Davinder Singh <davinder.singh@enterprisedb.com>
To: Thomas Munro <thomas.munro@gmail.com>
Cc: Robert Haas <robertmhaas@gmail.com>, Jakub Wartak <jakub.wartak@enterprisedb.com>, PostgreSQL Hackers <pgsql-hackers@postgresql.org>, Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: 2024-12-07T07:42:59Z
Lists: pgsql-hackers

Attachments

On Fri, Dec 6, 2024 at 11:31 AM Thomas Munro <thomas.munro@gmail.com> wrote:

> Some better new:
>
> 1.  _chsize_s does in fact seem to work in an msvcrt.dll build.
> Thanks to Andres for testing that for me on Windows with a standalone
> 2 line program on ucrt and msvcrt runtimes.
>
> 2.  MinGW always has _chsize_s, but it does more or less what I had
> been proposing already if it can't find the function by the Windows
> thing like dlsym, so there's no point in writing another thing like that.
> The test was with a 3G file though so it was the real function.
>
>
> https://github.com/mingw-w64/mingw-w64/blob/master/mingw-w64-crt/secapi/_chsize_s.c
>
> 3.  After a long time trying various ways to make it tidy and getting
> incomprehensible results I eventually realised that precompiled
> headers were causing strange results on MinGW/Meson builds (on CI),
> due to fighting over macros with this guy:
>
>
> https://github.com/mingw-w64/mingw-w64/blob/master/mingw-w64-headers/crt/unistd.h
>
> I found it easiest to completely stop it from getting in our way
> completely, with the attached.  Thoughts, anyone?
>
> Davinder, Jakub, are you in a position repro the issue and make
> versions of the patch for the 13, 15 and master branches on top of
> that?
>

Yes, Thomas, the issue is reproducible.
After applying your patch and replacing off_t with pgoff_t, the problem has
been resolved. I have attached the patches for the 13, 15, and master.
These patches include only the changes related to pgoff_t, intended for use
with the v3-0001 patch--


Regards,
Davinder.

Commits

  1. Fix off_t overflow in pg_basebackup on Windows.

  2. Provide 64-bit ftruncate() and lseek() on Windows.

  3. Require ucrt if using MinGW.