Horton

Re: logical decoding : exceeded maxAllocatedDescs for .spill files

Amit Kapila <amit.kapila16@gmail.com>

View thread
From: Amit Kapila <amit.kapila16@gmail.com>
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Noah Misch <noah@leadboat.com>, Amit Khandekar <amitdkhan.pg@gmail.com>, Alvaro Herrera from 2ndQuadrant <alvherre@alvh.no-ip.org>, Andres Freund <andres@anarazel.de>, Juan José Santamaría Flecha <juanjo.santamaria@gmail.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>, Robert Haas <robertmhaas@gmail.com>, Thomas Munro <thomas.munro@gmail.com>, Tomas Vondra <tomas.vondra@2ndquadrant.com>
Date: 2020-01-12T02:48:08Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. When a TAP file has non-zero exit status, retain temporary directories.

  2. Fix running out of file descriptors for spill files.

  3. Track statistics for spilling of changes from ReorderBuffer.

  4. Handle ReadFile() EOF correctly on Windows.

  5. Add logical_decoding_work_mem to limit ReorderBuffer memory usage.

  6. Generational memory allocator

  7. Support retaining data dirs on successful TAP tests

On Sat, Jan 11, 2020 at 11:16 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>
> Amit Kapila <amit.kapila16@gmail.com> writes:
> > On Fri, Jan 10, 2020 at 9:31 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >> ... So, we have the below
> >> options:
> >> (a) remove this test entirely from all branches and once we found the
> >> memory leak problem in back-branches, then consider adding it again
> >> without max_files_per_process restriction.
> >> (b) keep this test without max_files_per_process restriction till v11
> >> and once the memory leak issue in v10 is found, we can back-patch to
> >> v10 as well.
>
> > I am planning to go with option (a) and attached are patches to revert
> > the entire test on HEAD and back branches.  I am planning to commit
> > these by Tuesday unless someone has a better idea.
>
> Makes sense to me.  We've certainly found out something interesting
> from this test, but not what it was expecting to find ;-).  I think
> that there could be scope for two sorts of successor tests:
>
> * I still like my idea of directly constraining max_safe_fds through
> some sort of debug option.  But to my mind, we want to run the entire
> regression suite with that restriction, not just one small test.
>

Good idea.

> * The seeming bug in v10 suggests that we aren't testing large enough
> logical-decoding cases, or at least aren't noticing leaks in that
> area.  I'm not sure what a good design is for testing that.  I'm not
> thrilled with just using a larger (and slower) test case, but it's
> not clear to me how else to attack it.
>

It is not clear to me either at this stage, but I think we can decide
that after chasing the issue in v10.  My current plan is to revert
this test and make a note of the memory leak problem found (probably
track in Older Bugs section of PostgreSQL 12 Open Items).  I think
once we found the issue id v10, we might be in a better position to
decide if the test on the lines of the current test would make sense
or we need something else.

-- 
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com