Horton

Re: logical decoding : exceeded maxAllocatedDescs for .spill files

Amit Kapila <amit.kapila16@gmail.com>

View thread
From: Amit Kapila <amit.kapila16@gmail.com>
To: Amit Khandekar <amitdkhan.pg@gmail.com>
Cc: Alvaro Herrera from 2ndQuadrant <alvherre@alvh.no-ip.org>, Andres Freund <andres@anarazel.de>, Juan José Santamaría Flecha <juanjo.santamaria@gmail.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>, Robert Haas <robertmhaas@gmail.com>, Thomas Munro <thomas.munro@gmail.com>, Tom Lane <tgl@sss.pgh.pa.us>, Tomas Vondra <tomas.vondra@2ndquadrant.com>
Date: 2019-12-05T10:50:18Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. When a TAP file has non-zero exit status, retain temporary directories.

  2. Fix running out of file descriptors for spill files.

  3. Track statistics for spilling of changes from ReorderBuffer.

  4. Handle ReadFile() EOF correctly on Windows.

  5. Add logical_decoding_work_mem to limit ReorderBuffer memory usage.

  6. Generational memory allocator

  7. Support retaining data dirs on successful TAP tests

On Tue, Dec 3, 2019 at 11:10 AM Amit Khandekar <amitdkhan.pg@gmail.com> wrote:
>
> On Wed, 27 Nov 2019 at 14:16, Amit Khandekar <amitdkhan.pg@gmail.com> wrote:
> > What I found was : We do attempt to close the opened vfds in the
> > PG_CATCH block. In ReorderBufferCommit(), ReorderBufferIterTXNFinish
> > is called both in PG_TRY and PG_CATCH. This closes all the opened
> > vfds. But the issue is : if the ereport() occurs inside
> > ReorderBufferIterTXNInit(), then iterstate is still NULL. So in
> > PG_CATCH, ReorderBufferIterTXNFinish() is not called, so the vfds in
> > state->entries[] remain open.
> >
> > We can have &iterstate passed to ReorderBufferIterTXNInit() as another
> > argument, and initialize it first thing inside the function. This way,
> > it will never be NULL. But need to be careful about the possibility of
> > having a iterstate in a half-cooked state, so cleanup might use some
> > uninitialized handles. Will work on it. At least, we can make sure the
> > iterstate->entries handle doesn't have junk values.
>
> Done as stated above; attached v3 patch. I have verified that the file
> handles do get closed in PG_CATCH block via
> ReorderBufferIterTXNFinish().
>

I couldn't reproduce the original problem (on HEAD) reported with the
test case in the patch.  So, I can't verify the fix.  I think it is
because of recent commits cec2edfa7859279f36d2374770ca920c59c73dd8 and
9290ad198b15d6b986b855d2a58d087a54777e87.  It seems you need to either
change the value of logical_decoding_work_mem or change the test in
some way so that the original problem can be reproduced.

-- 
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com