Horton

Thread

Commits

  1. Add pg_ls_tmpdir function

  1. pg_ls_tmpdir()

    Nathan Bossart <bossartn@amazon.com> — 2018-09-12T21:07:25Z

    Hi hackers,

Attached is a patch to add a pg_ls_tmpdir() function that lists the
contents of a specified tablespace's pgsql_tmp directory.  This is
very similar to the existing pg_ls_logdir() and pg_ls_waldir()
functions.

By providing more visibility into the temporary file directories,
users can more easily track queries that are consuming a lot of disk
space for temporary files.  This function already provides enough
information to calculate the total temporary file usage per PID, but
it might be worthwhile to create a system view that does this, too.

I am submitting this patch for consideration in commitfest 2018-11.

Nathan

  2. Re: pg_ls_tmpdir()

    Laurenz Albe <laurenz.albe@cybertec.at> — 2018-09-26T20:36:03Z

    Bossart, Nathan wrote:
> Attached is a patch to add a pg_ls_tmpdir() function that lists the
> contents of a specified tablespace's pgsql_tmp directory.  This is
> very similar to the existing pg_ls_logdir() and pg_ls_waldir()
> functions.
> 
> By providing more visibility into the temporary file directories,
> users can more easily track queries that are consuming a lot of disk
> space for temporary files.  This function already provides enough
> information to calculate the total temporary file usage per PID, but
> it might be worthwhile to create a system view that does this, too.
> 
> I am submitting this patch for consideration in commitfest 2018-11.

The patch applies, builds without warning and passes "make check-world".

Since pgsql_tmp is only created when the first temp file is written,
calling the function on a newly initdb'ed data directory fails with

ERROR:  could not open directory "base/pgsql_tmp": No such file or directory

This should be fixed; it shouldn't be an error.

Calling the function with a non-existing tablespace OID produces:

ERROR:  could not open directory "pg_tblspc/1665/PG_12_201809121/pgsql_tmp": No such file or directory

Wouldn't it be better to have a check if the tablespace exists?

About the code:
The catversion bump shouldn't be part of the patch, it will
rot too quickly.  The committer is supposed to add it.


I think the idea to have such a function is fine, but I have two doubts:

1. Do we really need two functions, one without input argument
   to list the default tablespace?
   I think that anybody who uses with such a function whould
   be able to feed the OID of "pg_default".

2. There already are functions "pg_ls_logdir" and "pg_ls_waldir",
   and I can imagine new requests, e.g. pg_ls_datafiles() to list the
   data files in a database directory.

   It may make sense to have a generic function like

      pg_ls_dir(dirname text, tablespace oid)

   instead.  But maybe that's taking it too far...

I'll set the patch to "waiting for author".

Yours,
Laurenz Albe

  3. Re: pg_ls_tmpdir(); AND, Function for listing archive_status directory

    Justin Pryzby <pryzby@telsasoft.com> — 2018-10-01T15:37:37Z

    On Wed, Sep 26, 2018 at 10:36:03PM +0200, Laurenz Albe wrote:
> Bossart, Nathan wrote:
> > Attached is a patch to add a pg_ls_tmpdir() function that lists the
> > contents of a specified tablespace's pgsql_tmp directory.  This is
> > very similar to the existing pg_ls_logdir() and pg_ls_waldir()
> > functions.

On Sun, Sep 30, 2018 at 10:59:20PM +0200, Christoph Moench-Tegeder wrote:
> while setting up monitoring for a new PostgreSQL instance, I noticed that
> there's no build-in way for a pg_monitor role to check the contents of
> the archive_status directory. We got pg_ls_waldir() in 10, but that
> only lists pg_wal - not it's (sic) subdirectory. 

>    It may make sense to have a generic function like
>       pg_ls_dir(dirname text, tablespace oid)
>    instead.  But maybe that's taking it too far...

I see Cristoph has another pg_ls_* function, which suggests that Laurenz idea
is good, and a generic pg_ls function is desirable.

Justin

  4. Re: pg_ls_tmpdir()

    Andres Freund <andres@anarazel.de> — 2018-10-01T15:45:07Z

    Hi,

On 2018-09-26 22:36:03 +0200, Laurenz Albe wrote:
> 2. There already are functions "pg_ls_logdir" and "pg_ls_waldir",
>    and I can imagine new requests, e.g. pg_ls_datafiles() to list the
>    data files in a database directory.
> 
>    It may make sense to have a generic function like
> 
>       pg_ls_dir(dirname text, tablespace oid)
> 
>    instead.  But maybe that's taking it too far...

There's a generic pg_ls_dir() already - I'm now sure why a generic
function would take the tablespace oid however?

Greetings,

Andres Freund

  5. Re: pg_ls_tmpdir()

    Nathan Bossart <bossartn@amazon.com> — 2018-10-01T15:50:21Z

    Hi,

On 9/26/18, 3:36 PM, "Laurenz Albe" <laurenz.albe@cybertec.at> wrote:
> The patch applies, builds without warning and passes "make check-world".

Thanks for the review!

> Since pgsql_tmp is only created when the first temp file is written,
> calling the function on a newly initdb'ed data directory fails with
>
> ERROR:  could not open directory "base/pgsql_tmp": No such file or directory
>
> This should be fixed; it shouldn't be an error.

Done.

> Calling the function with a non-existing tablespace OID produces:
>
> ERROR:  could not open directory "pg_tblspc/1665/PG_12_201809121/pgsql_tmp": No such file or directory
>
> Wouldn't it be better to have a check if the tablespace exists?

Done.

> About the code:
> The catversion bump shouldn't be part of the patch, it will
> rot too quickly.  The committer is supposed to add it.

Removed.

> I think the idea to have such a function is fine, but I have two doubts:
>
> 1. Do we really need two functions, one without input argument
>    to list the default tablespace?
>    I think that anybody who uses with such a function whould
>    be able to feed the OID of "pg_default".

That seems reasonable to me.  I've removed the second version of the
function.

> 2. There already are functions "pg_ls_logdir" and "pg_ls_waldir",
>    and I can imagine new requests, e.g. pg_ls_datafiles() to list the
>    data files in a database directory.
>
>    It may make sense to have a generic function like
>
>       pg_ls_dir(dirname text, tablespace oid)
>
>    instead.  But maybe that's taking it too far...

There is an existing pg_ls_dir() function that appears to be available
only to superusers.  IMO it's also nice to break out specific "safe"
directories like this for other roles (e.g. pg_monitor).

Nathan

  6. Re: pg_ls_tmpdir()

    Christoph Berg <myon@debian.org> — 2018-10-02T10:31:41Z

    Re: Bossart, Nathan 2018-10-01 <69FD7E51-2B13-41FD-9438-17395C73F5BF@amazon.com>
> > 1. Do we really need two functions, one without input argument
> >    to list the default tablespace?
> >    I think that anybody who uses with such a function whould
> >    be able to feed the OID of "pg_default".
> 
> That seems reasonable to me.  I've removed the second version of the
> function.

You could make that one argument have a DEFAULT value that makes it
act on pg_default.

Christoph

  7. Re: pg_ls_tmpdir()

    Laurenz Albe <laurenz.albe@cybertec.at> — 2018-10-02T12:00:02Z

    Christoph Berg wrote:
> Re: Bossart, Nathan 2018-10-01 <69FD7E51-2B13-41FD-9438-17395C73F5BF@amazon.com>
> > > 1. Do we really need two functions, one without input argument
> > >     to list the default tablespace?
> > >     I think that anybody who uses with such a function whould
> > >     be able to feed the OID of "pg_default".
> > 
> > That seems reasonable to me.  I've removed the second version of the
> > function.
> 
> You could make that one argument have a DEFAULT value that makes it
> act on pg_default.

I looked at that, and I don't think you can add a DEFAULT for
system functions installed during bootstrap.
At least I failed in the attempt.

Yours,
Laurenz Albe

  8. Re: pg_ls_tmpdir()

    Andrew Dunstan <andrew.dunstan@2ndquadrant.com> — 2018-10-02T12:21:30Z

    
On 10/02/2018 08:00 AM, Laurenz Albe wrote:
> Christoph Berg wrote:
>> Re: Bossart, Nathan 2018-10-01 <69FD7E51-2B13-41FD-9438-17395C73F5BF@amazon.com>
>>>> 1. Do we really need two functions, one without input argument
>>>>      to list the default tablespace?
>>>>      I think that anybody who uses with such a function whould
>>>>      be able to feed the OID of "pg_default".
>>> That seems reasonable to me.  I've removed the second version of the
>>> function.
>> You could make that one argument have a DEFAULT value that makes it
>> act on pg_default.
> I looked at that, and I don't think you can add a DEFAULT for
> system functions installed during bootstrap.
> At least I failed in the attempt.


See the bottom of src/backend/catalog/system_views.sql starting around 
line 1010.

cheers

andrew


-- 
Andrew Dunstan                https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  9. Re: pg_ls_tmpdir()

    Nathan Bossart <bossartn@amazon.com> — 2018-10-02T16:32:58Z

    On 10/2/18, 7:22 AM, "Andrew Dunstan" <andrew.dunstan@2ndquadrant.com> wrote:
> On 10/02/2018 08:00 AM, Laurenz Albe wrote:
>> Christoph Berg wrote:
>>> Re: Bossart, Nathan 2018-10-01 <69FD7E51-2B13-41FD-9438-17395C73F5BF@amazon.com>
>>>>> 1. Do we really need two functions, one without input argument
>>>>>      to list the default tablespace?
>>>>>      I think that anybody who uses with such a function whould
>>>>>      be able to feed the OID of "pg_default".
>>>> That seems reasonable to me.  I've removed the second version of the
>>>> function.
>>> You could make that one argument have a DEFAULT value that makes it
>>> act on pg_default.
>> I looked at that, and I don't think you can add a DEFAULT for
>> system functions installed during bootstrap.
>> At least I failed in the attempt.
>
>
> See the bottom of src/backend/catalog/system_views.sql starting around 
> line 1010.

AFAICT the cleanest way to do this in system_views.sql is to hard-code
the pg_default tablespace OID in the DEFAULT expression.  So, it might
be best to use the two function approach if we want pg_ls_tmpdir() to
default to the pg_default tablespace.

Nathan

  10. Re: pg_ls_tmpdir()

    Tom Lane <tgl@sss.pgh.pa.us> — 2018-10-02T16:46:42Z

    "Bossart, Nathan" <bossartn@amazon.com> writes:
> On 10/2/18, 7:22 AM, "Andrew Dunstan" <andrew.dunstan@2ndquadrant.com> wrote:
>> See the bottom of src/backend/catalog/system_views.sql starting around 
>> line 1010.

> AFAICT the cleanest way to do this in system_views.sql is to hard-code
> the pg_default tablespace OID in the DEFAULT expression.  So, it might
> be best to use the two function approach if we want pg_ls_tmpdir() to
> default to the pg_default tablespace.

That would be pretty bletcherous, so +1 for just making two C functions.

			regards, tom lane

  11. Re: pg_ls_tmpdir()

    Nathan Bossart <bossartn@amazon.com> — 2018-10-02T19:01:30Z

    On 10/2/18, 11:46 AM, "Tom Lane" <tgl@sss.pgh.pa.us> wrote:
> "Bossart, Nathan" <bossartn@amazon.com> writes:
>> On 10/2/18, 7:22 AM, "Andrew Dunstan" <andrew.dunstan@2ndquadrant.com> wrote:
>>> See the bottom of src/backend/catalog/system_views.sql starting around 
>>> line 1010.
>
>> AFAICT the cleanest way to do this in system_views.sql is to hard-code
>> the pg_default tablespace OID in the DEFAULT expression.  So, it might
>> be best to use the two function approach if we want pg_ls_tmpdir() to
>> default to the pg_default tablespace.
>
> That would be pretty bletcherous, so +1 for just making two C functions.

Alright, here's an updated patch.

Nathan

  12. Re: pg_ls_tmpdir()

    Laurenz Albe <laurenz.albe@cybertec.at> — 2018-10-04T12:23:34Z

    Bossart, Nathan wrote:
> >> AFAICT the cleanest way to do this in system_views.sql is to hard-code
> >> the pg_default tablespace OID in the DEFAULT expression.  So, it might
> >> be best to use the two function approach if we want pg_ls_tmpdir() to
> >> default to the pg_default tablespace.
> >
> > That would be pretty bletcherous, so +1 for just making two C functions.
> 
> Alright, here's an updated patch.

Looks, good; marking as "ready for committer".

Yours,
Laurenz Albe

  13. Re: pg_ls_tmpdir()

    Michael Paquier <michael@paquier.xyz> — 2018-10-05T00:31:05Z

    On Thu, Oct 04, 2018 at 02:23:34PM +0200, Laurenz Albe wrote:
> Bossart, Nathan wrote:
>> Alright, here's an updated patch.
> 
> Looks, good; marking as "ready for committer".

Like Tom, I think it is less dirty to use the two-function approach.
Committed this way with a catalog version bump.
--
Michael

  14. Re: pg_ls_tmpdir()

    Nathan Bossart <bossartn@amazon.com> — 2018-10-05T01:14:46Z

    On 10/4/18, 7:31 PM, "Michael Paquier" <michael@paquier.xyz> wrote:
> Committed this way with a catalog version bump.

Thanks!

Nathan