Re: logical decoding : exceeded maxAllocatedDescs for .spill files

Thomas Munro <thomas.munro@gmail.com>

From: Thomas Munro <thomas.munro@gmail.com>

To: Juan José Santamaría Flecha <juanjo.santamaria@gmail.com>

Cc: Amit Khandekar <amitdkhan.pg@gmail.com>, Amit Kapila <amit.kapila16@gmail.com>, Andres Freund <andres@anarazel.de>, Tom Lane <tgl@sss.pgh.pa.us>, Robert Haas <robertmhaas@gmail.com>, Tomas Vondra <tomas.vondra@2ndquadrant.com>, Alvaro Herrera from 2ndQuadrant <alvherre@alvh.no-ip.org>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>

Date: 2019-11-19T18:58:16Z

Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

When a TAP file has non-zero exit status, retain temporary directories.
- 048c7ccd7d6d 9.6.17 landed
- d8efc5900f7c 10.12 landed
- 887657d183fc 11.7 landed
- 78a26c3edd85 12.2 landed
- bf989aaf3561 13.0 landed
Fix running out of file descriptors for spill files.
- 1ad47e8757bb 9.4.26 landed
- a6f4f407ada0 9.5.21 landed
- 27b5f48c79f7 10.12 landed
- 3e3a79735235 11.7 landed
- f8a6d8e71b17 12.2 landed
- d20703805383 13.0 landed
- ba5b4e506489 9.6.17 landed
Track statistics for spilling of changes from ReorderBuffer.
- 9290ad198b15 13.0 cited
Handle ReadFile() EOF correctly on Windows.
- 2189f49c420f 12.2 landed
- 6969deeb8d39 13.0 landed
Add logical_decoding_work_mem to limit ReorderBuffer memory usage.
- cec2edfa7859 13.0 cited
Generational memory allocator
- a4ccc1cef5a0 11.0 cited
Support retaining data dirs on successful TAP tests
- 90627cf98a8e 11.0 cited

On Wed, Nov 20, 2019 at 1:14 AM Juan José Santamaría Flecha
<juanjo.santamaria@gmail.com> wrote:
> On Tue, Nov 19, 2019 at 12:49 PM Thomas Munro <thomas.munro@gmail.com> wrote:
>> On Wed, Nov 20, 2019 at 12:28 AM Amit Khandekar <amitdkhan.pg@gmail.com> wrote:
>> > On Windows, it is documented that ReadFile() (which is called by
>> > pg_pread) will return false on EOF but only when the file is open for
>> > asynchronous reads/writes. But here we are just dealing with usual
>> > synchronous reads. So pg_pread() code should indeed return 0 on EOF on
>> > Windows. Not yet able to figure out how FileRead() managed to return
>> > this error on Windows. But from your symptoms, it does look like
>> > pg_pread()=>ReadFile() returned false (despite doing asynchronous
>> > reads), and so _dosmaperr() gets called, and then it does not find the
>> > eof error in doserrors[], so the "unrecognized win32 error code"
>> > message is printed. May have to dig up more on this.
>>
>> Hmm.  See also this report:
>>
>> https://www.postgresql.org/message-id/flat/CABuU89MfEvJE%3DWif%2BHk7SCqjSOF4rhgwJWW6aR3hjojpGqFbjQ%40mail.gmail.com
>>
>
> The files from pgwin32_open() are open for synchronous access, while pg_pread() uses the asynchronous functionality to offset the read. Under these circunstances, a read past EOF will return ERROR_HANDLE_EOF (38), as explained in:

Oh, thanks.

> https://devblogs.microsoft.com/oldnewthing/20150121-00/?p=44863

!?!

Amit, since it looks like you are Windows-enabled and have a repro,
would you mind confirming that this fixes the problem?

--- a/src/port/pread.c
+++ b/src/port/pread.c
@@ -41,6 +41,9 @@ pg_pread(int fd, void *buf, size_t size, off_t offset)
        overlapped.Offset = offset;
        if (!ReadFile(handle, buf, size, &result, &overlapped))
        {
+               if (GetLastError() == ERROR_HANDLE_EOF)
+                       return 0;
+
                _dosmaperr(GetLastError());
                return -1;
        }