Horton

Re: logical decoding : exceeded maxAllocatedDescs for .spill files

Juan José Santamaría Flecha <juanjo.santamaria@gmail.com>

View thread
From: Juan José Santamaría Flecha <juanjo.santamaria@gmail.com>
To: Thomas Munro <thomas.munro@gmail.com>
Cc: Amit Khandekar <amitdkhan.pg@gmail.com>, Amit Kapila <amit.kapila16@gmail.com>, Andres Freund <andres@anarazel.de>, Tom Lane <tgl@sss.pgh.pa.us>, Robert Haas <robertmhaas@gmail.com>, Tomas Vondra <tomas.vondra@2ndquadrant.com>, Alvaro Herrera from 2ndQuadrant <alvherre@alvh.no-ip.org>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2019-11-19T12:14:22Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. When a TAP file has non-zero exit status, retain temporary directories.

  2. Fix running out of file descriptors for spill files.

  3. Track statistics for spilling of changes from ReorderBuffer.

  4. Handle ReadFile() EOF correctly on Windows.

  5. Add logical_decoding_work_mem to limit ReorderBuffer memory usage.

  6. Generational memory allocator

  7. Support retaining data dirs on successful TAP tests

On Tue, Nov 19, 2019 at 12:49 PM Thomas Munro <thomas.munro@gmail.com>
wrote:

> On Wed, Nov 20, 2019 at 12:28 AM Amit Khandekar <amitdkhan.pg@gmail.com>
> wrote:
> > On Tue, 19 Nov 2019 at 14:07, Amit Kapila <amit.kapila16@gmail.com>
> wrote:
> > > No, I got this before applying the patch.  However, after applying the
> > > patch, I got below error in the same test:
> > >
> > > postgres=# SELECT 1 from
> > > pg_logical_slot_get_changes('regression_slot', NULL,NULL) LIMIT 1;
> > > ERROR:  could not read from reorderbuffer spill file: Invalid argument
> > >
> > > It seems to me that FileRead API used in the patch can return value <
> > > 0 on  EOF.  See the API usage in BufFileLoadBuffer.  I got this error
> > > on a windows machine and in the server log the message was "LOG:
> > > unrecognized win32 error code: 38" which indicates "Reached the end of
> > > the file."
> >
> > On Windows, it is documented that ReadFile() (which is called by
> > pg_pread) will return false on EOF but only when the file is open for
> > asynchronous reads/writes. But here we are just dealing with usual
> > synchronous reads. So pg_pread() code should indeed return 0 on EOF on
> > Windows. Not yet able to figure out how FileRead() managed to return
> > this error on Windows. But from your symptoms, it does look like
> > pg_pread()=>ReadFile() returned false (despite doing asynchronous
> > reads), and so _dosmaperr() gets called, and then it does not find the
> > eof error in doserrors[], so the "unrecognized win32 error code"
> > message is printed. May have to dig up more on this.
>
> Hmm.  See also this report:
>
>
> https://www.postgresql.org/message-id/flat/CABuU89MfEvJE%3DWif%2BHk7SCqjSOF4rhgwJWW6aR3hjojpGqFbjQ%40mail.gmail.com
>
>
The files from pgwin32_open() are open for synchronous access, while
pg_pread() uses the asynchronous functionality to offset the read. Under
these circunstances, a read past EOF will return ERROR_HANDLE_EOF (38), as
explained in:

https://devblogs.microsoft.com/oldnewthing/20150121-00/?p=44863


Regards,

Juan José Santamaría Flecha