Re: logical decoding : exceeded maxAllocatedDescs for .spill files

Juan José Santamaría Flecha <juanjo.santamaria@gmail.com>

From: Juan José Santamaría Flecha <juanjo.santamaria@gmail.com>

To: Amit Khandekar <amitdkhan.pg@gmail.com>

Cc: Amit Kapila <amit.kapila16@gmail.com>, Andres Freund <andres@anarazel.de>, Tom Lane <tgl@sss.pgh.pa.us>, Thomas Munro <thomas.munro@gmail.com>, Robert Haas <robertmhaas@gmail.com>, Tomas Vondra <tomas.vondra@2ndquadrant.com>, Alvaro Herrera from 2ndQuadrant <alvherre@alvh.no-ip.org>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>

Date: 2019-11-20T12:11:26Z

Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

When a TAP file has non-zero exit status, retain temporary directories.
- 048c7ccd7d6d 9.6.17 landed
- d8efc5900f7c 10.12 landed
- 887657d183fc 11.7 landed
- 78a26c3edd85 12.2 landed
- bf989aaf3561 13.0 landed
Fix running out of file descriptors for spill files.
- 1ad47e8757bb 9.4.26 landed
- a6f4f407ada0 9.5.21 landed
- 27b5f48c79f7 10.12 landed
- 3e3a79735235 11.7 landed
- f8a6d8e71b17 12.2 landed
- d20703805383 13.0 landed
- ba5b4e506489 9.6.17 landed
Track statistics for spilling of changes from ReorderBuffer.
- 9290ad198b15 13.0 cited
Handle ReadFile() EOF correctly on Windows.
- 2189f49c420f 12.2 landed
- 6969deeb8d39 13.0 landed
Add logical_decoding_work_mem to limit ReorderBuffer memory usage.
- cec2edfa7859 13.0 cited
Generational memory allocator
- a4ccc1cef5a0 11.0 cited
Support retaining data dirs on successful TAP tests
- 90627cf98a8e 11.0 cited

On Wed, Nov 20, 2019 at 9:48 AM Amit Khandekar <amitdkhan.pg@gmail.com>
wrote:

> On Wed, 20 Nov 2019 at 13:10, Amit Kapila <amit.kapila16@gmail.com> wrote:
> > See comment in pgunlink() "We need to loop because even though
> > PostgreSQL uses flags that allow unlink while the file is open, other
> > applications might have the file
> > open without those flags.".  Can you once see if there is any flag
> > that you have missed to pass to allow this?
>
> > If there is nothing we
> > can do about it, then we might need to use some different API or maybe
> > define a new API that can handle this.
>
> There were objections against modifying the vfd api only for this
> replication-related use-case. Having a new API will require all the
> changes required to enable the virtual FDs feature that we need from
> vfd. If nothing works out from the FILE_SHARE_DELETE thing, I am
> thinking, we can use VFD, plus we can keep track of per-subtransaction
> vfd handles, and do something similar to AtEOSubXact_Files().
>
>
The comment about "other applications might have the file open without
those flags." is surely due to systems working with an antivirus touching
Postgres files.

I was not able to reproduce the Permission denied error with current HEAD,
until I opened another CMD inside the "pg_replslot/regression_slot" folder.
This will be problematic, is the deletion of the folder actually needed?

Regards,

Juan José Santamaría Flecha