Re: logical decoding : exceeded maxAllocatedDescs for .spill files

Robert Haas <robertmhaas@gmail.com>

From: Robert Haas <robertmhaas@gmail.com>

To: Tom Lane <tgl@sss.pgh.pa.us>

Cc: Amit Khandekar <amitdkhan.pg@gmail.com>, Tomas Vondra <tomas.vondra@2ndquadrant.com>, Alvaro Herrera from 2ndQuadrant <alvherre@alvh.no-ip.org>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>

Date: 2019-09-13T16:31:38Z

Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

When a TAP file has non-zero exit status, retain temporary directories.
- 048c7ccd7d6d 9.6.17 landed
- d8efc5900f7c 10.12 landed
- 887657d183fc 11.7 landed
- 78a26c3edd85 12.2 landed
- bf989aaf3561 13.0 landed
Fix running out of file descriptors for spill files.
- 1ad47e8757bb 9.4.26 landed
- a6f4f407ada0 9.5.21 landed
- 27b5f48c79f7 10.12 landed
- 3e3a79735235 11.7 landed
- f8a6d8e71b17 12.2 landed
- d20703805383 13.0 landed
- ba5b4e506489 9.6.17 landed
Track statistics for spilling of changes from ReorderBuffer.
- 9290ad198b15 13.0 cited
Handle ReadFile() EOF correctly on Windows.
- 2189f49c420f 12.2 landed
- 6969deeb8d39 13.0 landed
Add logical_decoding_work_mem to limit ReorderBuffer memory usage.
- cec2edfa7859 13.0 cited
Generational memory allocator
- a4ccc1cef5a0 11.0 cited
Support retaining data dirs on successful TAP tests
- 90627cf98a8e 11.0 cited

On Fri, Sep 13, 2019 at 12:14 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Amit Khandekar <amitdkhan.pg@gmail.com> writes:
> > You mean tracking excess kernel fds right ? Yeah, we can use VFDs so
> > that excess fds are automatically closed. But Alvaro seems to be
> > talking in context of tracking of file seek position. VFD  does not
> > have a mechanism to track file offsets if one of the vfd cached file
> > is closed and reopened.
>
> Hm.  It used to, but somebody got rid of that on the theory that
> we could use pread/pwrite instead.  I'm inclined to think that that
> was the right tradeoff, but it'd mean that getting logical decoding
> to adhere to the VFD API requires extra work to track file position
> on the caller side.

Oops.  I forgot that we'd removed that.

> Again, though, the advice that's been given here is that we should
> fix logical decoding to use the VFD API as it stands, not change
> that API.  I concur with that.

A reasonable position.  So I guess logical decoding has to track the
file position itself, but perhaps use the VFD layer for managing FD
pooling.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company