Re: logical decoding : exceeded maxAllocatedDescs for .spill files

Amit Khandekar <amitdkhan.pg@gmail.com>

From: Amit Khandekar <amitdkhan.pg@gmail.com>

To: Amit Kapila <amit.kapila16@gmail.com>

Cc: Andres Freund <andres@anarazel.de>, Tom Lane <tgl@sss.pgh.pa.us>, Thomas Munro <thomas.munro@gmail.com>, Robert Haas <robertmhaas@gmail.com>, Tomas Vondra <tomas.vondra@2ndquadrant.com>, Alvaro Herrera from 2ndQuadrant <alvherre@alvh.no-ip.org>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>

Date: 2019-11-18T12:59:43Z

Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

When a TAP file has non-zero exit status, retain temporary directories.
- 048c7ccd7d6d 9.6.17 landed
- d8efc5900f7c 10.12 landed
- 887657d183fc 11.7 landed
- 78a26c3edd85 12.2 landed
- bf989aaf3561 13.0 landed
Fix running out of file descriptors for spill files.
- 1ad47e8757bb 9.4.26 landed
- a6f4f407ada0 9.5.21 landed
- 27b5f48c79f7 10.12 landed
- 3e3a79735235 11.7 landed
- f8a6d8e71b17 12.2 landed
- d20703805383 13.0 landed
- ba5b4e506489 9.6.17 landed
Track statistics for spilling of changes from ReorderBuffer.
- 9290ad198b15 13.0 cited
Handle ReadFile() EOF correctly on Windows.
- 2189f49c420f 12.2 landed
- 6969deeb8d39 13.0 landed
Add logical_decoding_work_mem to limit ReorderBuffer memory usage.
- cec2edfa7859 13.0 cited
Generational memory allocator
- a4ccc1cef5a0 11.0 cited
Support retaining data dirs on successful TAP tests
- 90627cf98a8e 11.0 cited

On Mon, 18 Nov 2019 at 17:52, Amit Kapila <amit.kapila16@gmail.com> wrote:
> I have one more question regarding this patch.   It seems to me that
> the files opened via OpenTransientFile or OpenTemporaryFile are
> automatically closed at transaction end(abort), but that doesn't seem
> to be the case for files opened with PathNameOpenFile.  See
> AtEOXact_Files and AtEOSubXact_Files.  So, now with the change
> proposed by this patch, don't we need to deal it in some other way?

For the API's that use VFDs (like PathNameOpenFile), the files opened
are always recorded in the VfdCache array. So it is not required to do
the cleanup at (sub)transaction end, because the kernel fds get closed
dynamically in ReleaseLruFiles() whenever they reach max_safe_fds
limit. So if a transaction aborts, the fds might remain open, but
those will get cleaned up whenever we require more fds, through
ReleaseLruFiles(). Whereas, for files opened through
OpenTransientFile(), VfdCache is not involved, so this needs
transaction end cleanup.

-- 
Thanks,
-Amit Khandekar
EnterpriseDB Corporation
The Postgres Database Company