Horton

Re: Windows pg_basebackup unable to create >2GB pg_wal.tar tarballs ("could not close file: Invalid argument" when creating pg_wal.tar of size ~ 2^31 bytes)

Jakub Wartak <jakub.wartak@enterprisedb.com>

View thread View on postgr.es
From: Jakub Wartak <jakub.wartak@enterprisedb.com>
To: Thomas Munro <thomas.munro@gmail.com>
Cc: PostgreSQL Hackers <pgsql-hackers@postgresql.org>, Davinder Singh <davinder.singh@enterprisedb.com>, Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: 2024-11-22T09:55:15Z
Lists: pgsql-hackers

Attachments

Hi Thomas!

On Thu, Nov 21, 2024 at 2:38 PM Thomas Munro <thomas.munro@gmail.com> wrote:

> On Thu, Nov 21, 2024 at 11:44 PM Jakub Wartak
> <jakub.wartak@enterprisedb.com> wrote:
> > This literally looks like something like off_t/size_t would be limited
> to 2^31 somewhere.
>
> off_t is 32 bits on Windows.  I'd be quite suspicious of the
> arithmetic involving 'currpos'.  What happens if you change all the
> off_t in walmethods.c/.h to pgoff_t?  The lseek() is suspicious too,
> and might need to be redirected to _lseeki64().
>

Thank you for responding and suggesting this. First one was not enough, on
REL_15_STABLE with just pgoff_t i still got the same error:

C:\compiledpg\bin>pg_basebackup.exe -U postgres -D "c:\backup6" -F t -P -X
stream -c fast --compress=none --create-slot --slot=slot10
Password:
pg_basebackup: error: could not close file "0000000100000008000000AE":
Invalid argument
pg_basebackup: error: background process terminated unexpectedly

but with attached _lseeki64 dirty patch I've got success and resulting
tarball greater than 2^31 too:

C:\compiledpg\bin>pg_basebackup.exe -U postgres -D "c:\backup7" -F t -P -X
stream -c fast --compress=none --create-slot --slot=slot11
Password:
18134035/18134035 kB (100%), 1/1 tablespace
C:\compiledpg\bin>
C:\compiledpg\bin>dir c:\backup7\pg_wal.tar
[..]
11/22/2024  10:37 AM     4,026,778,112 pg_wal.tar

PoC patch is attached and I'll register CFentry to see how the tests will
go (all of this MSVC/MinGW stuff is frightening to me, of course this will
fail on non-Win32...).

-J.

Commits

  1. Fix off_t overflow in pg_basebackup on Windows.

  2. Provide 64-bit ftruncate() and lseek() on Windows.

  3. Require ucrt if using MinGW.