Horton

Re: pg11+: pg_ls_*dir LIMIT 1: temporary files .. not closed at end-of-transaction

Justin Pryzby <pryzby@telsasoft.com>

View thread View on postgr.es
From: Justin Pryzby <pryzby@telsasoft.com>
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: pgsql-hackers@postgresql.org, Fabien COELHO <coelho@cri.ensmp.fr>, Alvaro Herrera <alvherre@2ndquadrant.com>, David Steele <david@pgmasters.net>, "Bossart, Nathan" <bossartn@amazon.com>, Thomas Munro <thomas.munro@gmail.com>
Date: 2020-03-12T12:11:56Z
Lists: pgsql-hackers

Attachments

On Wed, Mar 11, 2020 at 03:32:38PM -0400, Tom Lane wrote:
> > I patched this one to see what it looks like and to allow /hopefully/ moving
> > forward one way or another with the pg_ls_tmpfile() patch set (or at least
> > avoid trying to do anything there which is too inconsistent with this fix).
> 
> I reviewed this, added some test cases, and pushed it, so that we can see

Thanks, tests were on my periphery..

|    In passing, fix bogus error report for stat() failure: it was
|    whining about the directory when it should be fingering the
|    individual file.  Doubtless a copy-and-paste error.

Thanks again ; that was my 0001 patch on the other thread.  No rebase conflict
even ;)
https://www.postgresql.org/message-id/20191228101650.GG12890%40telsasoft.com

> Do you want to have a go at that?

First draft attached.  Note that I handled pg_ls_dir, even though I'm proposing
on the other thread to collapse/merge/meld it with pg_ls_dir_files [0].
Possibly that's a bad idea with tuplestore, due to returning a scalar vs a row
and needing to conditionally call CreateTemplateTupleDesc vs
get_call_result_type.  I'll rebase that patch later today.

I didn't write test cases yet.  Also didn't look for functions not on your
list.

I noticed this doesn't actually do anything, but kept it for now...except in
pg_ls_dir error case:

src/include/utils/tuplestore.h:/* tuplestore_donestoring() used to be required, but is no longer used */
src/include/utils/tuplestore.h:#define tuplestore_donestoring(state)    ((void) 0)

I found a few documentation bits that I think aren't relevant, but could
possibly be misread to encourage the bad coding practice.  This is about *sql*
functions:

|37.5.8. SQL Functions Returning Sets
|When an SQL function is declared as returning SETOF sometype, the function's
|final query is executed TO COMPLETION, and each row it outputs is returned as
|an element of the result set.
|...
|Set-returning functions in the select list are always evaluated as though they
|are on the inside of a nested-loop join with the rest of the FROM clause, so
|that the function(s) are run TO COMPLETION before the next row from the FROM
|clause is considered.

-- 
Justin

[0] https://www.postgresql.org/message-id/20200310183037.GA29065%40telsasoft.com
v9-0008-generalize-pg_ls_dir_files-and-retire-pg_ls_dir.patch

Commits

  1. Teach pg_ls_dir_files() to ignore ENOENT failures from stat().

  2. Remove useless pfree()s at the ends of various ValuePerCall SRFs.

  3. Avoid holding a directory FD open across assorted SRF calls.

  4. Document pg_ls_*dir hiding of directories and special files

  5. Avoid holding a directory FD open across pg_ls_dir_files() calls.

  6. Adjust nodeFunctionscan.c to reset transient memory context between calls