Re: logical decoding : exceeded maxAllocatedDescs for .spill files

Tomas Vondra <tomas.vondra@2ndquadrant.com>

From: Tomas Vondra <tomas.vondra@2ndquadrant.com>

To: Andres Freund <andres@anarazel.de>

Cc: Alvaro Herrera <alvherre@2ndquadrant.com>, Amit Khandekar <amitdkhan.pg@gmail.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>

Date: 2019-09-13T14:57:34Z

Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

When a TAP file has non-zero exit status, retain temporary directories.
- 048c7ccd7d6d 9.6.17 landed
- d8efc5900f7c 10.12 landed
- 887657d183fc 11.7 landed
- 78a26c3edd85 12.2 landed
- bf989aaf3561 13.0 landed
Fix running out of file descriptors for spill files.
- 1ad47e8757bb 9.4.26 landed
- a6f4f407ada0 9.5.21 landed
- 27b5f48c79f7 10.12 landed
- 3e3a79735235 11.7 landed
- f8a6d8e71b17 12.2 landed
- d20703805383 13.0 landed
- ba5b4e506489 9.6.17 landed
Track statistics for spilling of changes from ReorderBuffer.
- 9290ad198b15 13.0 cited
Handle ReadFile() EOF correctly on Windows.
- 2189f49c420f 12.2 landed
- 6969deeb8d39 13.0 landed
Add logical_decoding_work_mem to limit ReorderBuffer memory usage.
- cec2edfa7859 13.0 cited
Generational memory allocator
- a4ccc1cef5a0 11.0 cited
Support retaining data dirs on successful TAP tests
- 90627cf98a8e 11.0 cited

On Thu, Sep 12, 2019 at 11:34:01AM -0700, Andres Freund wrote:
>Hi,
>
>On 2019-09-12 09:57:55 -0300, Alvaro Herrera wrote:
>> On 2019-Sep-12, Tomas Vondra wrote:
>>
>> > On Wed, Sep 11, 2019 at 09:51:40AM -0300, Alvaro Herrera from 2ndQuadrant wrote:
>> > > On 2019-Sep-11, Amit Khandekar wrote:
>>
>> > > I think doing this all the time would make restore very slow -- there's a
>> > > reason we keep the files open, after all.
>> >
>> > How much slower? It certainly will have a hit, but maybe it's negligible
>> > compared to all the other stuff happening in this code?
>
>I'd expect it to be significant.
>
>
>> > As a sidenote - in the other thread about streaming, one of the patches
>> > does change how we log subxact assignments. In the end, this allows using
>> > just a single file for the top-level transaction, instead of having one
>> > file per subxact. That would also solve this.
>
>Uhm, how is rollback to savepoint going to be handled in that case? I
>don't think it's great to just retain space for all rolled back
>subtransactions.
>

The code can just do ftruncate() to the position of the subtransaction.
That's what the patch [1] does.

[1] https://commitfest.postgresql.org/24/1927/


regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services