Horton

Re: pg11+: pg_ls_*dir LIMIT 1: temporary files .. not closed at end-of-transaction

Tom Lane <tgl@sss.pgh.pa.us>

View thread View on postgr.es
From: Tom Lane <tgl@sss.pgh.pa.us>
To: Justin Pryzby <pryzby@telsasoft.com>
Cc: pgsql-hackers@postgresql.org, Fabien COELHO <coelho@cri.ensmp.fr>, Alvaro Herrera <alvherre@2ndquadrant.com>, David Steele <david@pgmasters.net>, "Bossart, Nathan" <bossartn@amazon.com>, Thomas Munro <thomas.munro@gmail.com>
Date: 2020-03-08T19:40:09Z
Lists: pgsql-hackers 
Justin Pryzby <pryzby@telsasoft.com> writes:
> On Sun, Mar 08, 2020 at 02:37:49PM -0400, Tom Lane wrote:
>> I guess we ought to change that function to use returns-a-tuplestore
>> protocol instead of thinking it can hold a directory open across calls.
>> It's not hard to think of use-cases where the existing behavior would
>> cause issues worse than a nanny-ish WARNING, especially on platforms
>> with tight "ulimit -n" limits.

> Thanks for the analysis.

> Do you mean it should enumerate all files during the initial SRF call, or use
> something other than the SRF_* macros ?

It has to enumerate all the files during the first call.  I suppose it
could do that and then still hand back the results one-at-a-time, but
there seems little point compared to filling a tuplestore immediately.
So probably the SRF_ macros are useless here.

Another possible solution is to register an exprstate-shutdown hook to
ensure the resource is cleaned up, but I'm not very happy with that
because it does nothing to prevent the hazard of overrunning the
available resources if you have several of these active at once.

I've just finished scanning the source code and concluding that all
of these functions are similarly broken:

pg_ls_dir
pg_ls_dir_files
pg_tablespace_databases
pg_logdir_ls_internal
pg_timezone_names
pgrowlocks

The first five risk leaking an open directory, the last risks leaking
an active tablescan and open relation.

I don't see anything in the documentation (either funcapi.h or
xfunc.sgml) warning that the function might not get run to completion,
either ...

			regards, tom lane

Commits

  1. Teach pg_ls_dir_files() to ignore ENOENT failures from stat().

  2. Remove useless pfree()s at the ends of various ValuePerCall SRFs.

  3. Avoid holding a directory FD open across assorted SRF calls.

  4. Document pg_ls_*dir hiding of directories and special files

  5. Avoid holding a directory FD open across pg_ls_dir_files() calls.

  6. Adjust nodeFunctionscan.c to reset transient memory context between calls