Horton

Thread

Commits

  1. Fix (some of) pltcl memory usage

  1. Lack of Sanity Checking in file 'pctcl.c' for PostgreSQL 9.4.x

    Bill Parker <wp02855@gmail.com> — 2015-06-11T19:22:41Z

    ============================================================================
                        POSTGRESQL BUG REPORT TEMPLATE
============================================================================

Your name               : Bill Parker
Your email address      : wp02855 at gmail dot com

System Configuration:
---------------------
  Architecture (example: Intel Pentium)         :  x86/x86-64/AMD

  Operating System (example: Linux 2.4.18)      :  Linux 3.11.6-4

  PostgreSQL version (example: PostgreSQL 9.4.3):  PostgreSQL 9.4.x

  Compiler used (example: gcc 3.3.5)            :  gcc version 4.8.1

Please enter a FULL description of your problem:
------------------------------------------------

Hello All,

   In reviewing some code, in directory 'postgresql-9.4.3/src/pl/tcl',
file 'pltcl.c', there are several instances where calls to malloc()
are made, but no check for a return value of NULL is made, which
would indicate failure.   Additionally, it appears when malloc()
returns NULL, previously allocated memory in function 'perm_fmgr_info'
is not released, which could lead to memory leaks (even though the
comment at the top says 'this routine is a crock' :)

If you know how this problem might be fixed, list the solution below:
---------------------------------------------------------------------

The patch file below addresses these issues:

--- pltcl.c.orig        2015-06-11 08:41:24.316077095 -0700
+++ pltcl.c     2015-06-11 08:48:49.186617853 -0700
@@ -2136,11 +2136,28 @@
         * Allocate the new querydesc structure
         ************************************************************/
        qdesc = (pltcl_query_desc *) malloc(sizeof(pltcl_query_desc));
+       if (qdesc == NULL)
+           ereport(ERROR, ((errcode(ERRCODE_OUT_OF_MEMORY), errmsg("out of
memory")));
        snprintf(qdesc->qname, sizeof(qdesc->qname), "%p", qdesc);
        qdesc->nargs = nargs;
        qdesc->argtypes = (Oid *) malloc(nargs * sizeof(Oid));
+       if (qdesc->argtypes == NULL) {
+           free(qdesc);
+           ereport(ERROR, ((errcode(ERRCODE_OUT_OF_MEMORY), errmsg("out of
memory")));
+       }
        qdesc->arginfuncs = (FmgrInfo *) malloc(nargs * sizeof(FmgrInfo));
+       if (qdesc->arginfuncs == NULL) {
+           free(qdesc->argtypes);
+           free(qdesc);
+           ereport(ERROR, ((errcode(ERRCODE_OUT_OF_MEMORY), errmsg("out of
memory")));
+       }
        qdesc->argtypioparams = (Oid *) malloc(nargs * sizeof(Oid));
+       if (qdesc->argtypioparams == NULL) {
+           free(qdesc->inargfuncs);
+           free(qdesc->argtypes);
+           free(qdesc);
+       }
+           ereport(ERROR, ((errcode(ERRCODE_OUT_OF_MEMORY), errmsg("out of
memory")));

        /************************************************************
         * Execute the prepare inside a sub-transaction, so we can cope with

Please feel free to review and comment on the above patch file...

I am attaching the patch file to this bug report

Bill Parker (wp02855 at gmail dot com)

  2. Re: Lack of Sanity Checking in file 'pctcl.c' for PostgreSQL 9.4.x

    Michael Paquier <michael.paquier@gmail.com> — 2015-06-12T07:05:44Z

    On Fri, Jun 12, 2015 at 4:22 AM, Bill Parker wrote:
>    In reviewing some code, in directory 'postgresql-9.4.3/src/pl/tcl',
> file 'pltcl.c', there are several instances where calls to malloc()
> are made, but no check for a return value of NULL is made, which
> would indicate failure.   Additionally, it appears when malloc()
> returns NULL, previously allocated memory in function 'perm_fmgr_info'
> is not released, which could lead to memory leaks (even though the
> comment at the top says 'this routine is a crock' :)
>
> If you know how this problem might be fixed, list the solution below:
> Please feel free to review and comment on the above patch file...

Oh, nice catch again.

> I am attaching the patch file to this bug report

By the way, your patch does not compile properly and is not in-line
with the project's code format. See the updated patch attached ;)
-- 
Michael

  3. Re: Lack of Sanity Checking in file 'pctcl.c' for PostgreSQL 9.4.x

    Alvaro Herrera <alvherre@2ndquadrant.com> — 2015-06-12T19:47:17Z

    Michael Paquier wrote:
> On Fri, Jun 12, 2015 at 4:22 AM, Bill Parker wrote:
> >    In reviewing some code, in directory 'postgresql-9.4.3/src/pl/tcl',
> > file 'pltcl.c', there are several instances where calls to malloc()
> > are made, but no check for a return value of NULL is made, which
> > would indicate failure.   Additionally, it appears when malloc()
> > returns NULL, previously allocated memory in function 'perm_fmgr_info'
> > is not released, which could lead to memory leaks (even though the
> > comment at the top says 'this routine is a crock' :)
> >
> > If you know how this problem might be fixed, list the solution below:
> > Please feel free to review and comment on the above patch file...
> 
> Oh, nice catch again.
> 
> > I am attaching the patch file to this bug report
> 
> By the way, your patch does not compile properly and is not in-line
> with the project's code format. See the updated patch attached ;)

... or the conventions for allocating memory.  Why not just use palloc()?


-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  4. Re: Lack of Sanity Checking in file 'pctcl.c' for PostgreSQL 9.4.x

    Tom Lane <tgl@sss.pgh.pa.us> — 2015-06-13T03:41:51Z

    Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> Michael Paquier wrote:
>> By the way, your patch does not compile properly and is not in-line
>> with the project's code format. See the updated patch attached ;)

> ... or the conventions for allocating memory.  Why not just use palloc()?

That's hardly the fault of the proposed patch.  But yeah, it seems like
much the best fix here is to get rid of the malloc (and strdup) calls in
this code in favor of using the palloc infrastructure.  Even the calls
that *do* have manual failure checks are not compliant with our usual
coding standards.

			regards, tom lane

  5. Re: Lack of Sanity Checking in file 'pctcl.c' for PostgreSQL 9.4.x

    Michael Paquier <michael.paquier@gmail.com> — 2015-06-13T11:41:25Z

    On Sat, Jun 13, 2015 at 12:41 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Alvaro Herrera <alvherre@2ndquadrant.com> writes:
>> Michael Paquier wrote:
>>> By the way, your patch does not compile properly and is not in-line
>>> with the project's code format. See the updated patch attached ;)
>
>> ... or the conventions for allocating memory.  Why not just use palloc()?
>
> That's hardly the fault of the proposed patch.  But yeah, it seems like
> much the best fix here is to get rid of the malloc (and strdup) calls in
> this code in favor of using the palloc infrastructure.  Even the calls
> that *do* have manual failure checks are not compliant with our usual
> coding standards.

Hm. Regarding the code path mentioned by Bill something like the patch
attached is enough with a memory context for the query description.
Now, perhaps we could do more efforts with prodesc as well, see for
example compile_pltcl_function for pltcl and similarly for plperl.
Thoughts?
-- 
Michael

  6. Re: Lack of Sanity Checking in file 'pctcl.c' for PostgreSQL 9.4.x

    Alvaro Herrera <alvherre@2ndquadrant.com> — 2015-07-18T15:54:21Z

    Michael Paquier wrote:
> On Sat, Jun 13, 2015 at 12:41 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> > Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> >> Michael Paquier wrote:
> >>> By the way, your patch does not compile properly and is not in-line
> >>> with the project's code format. See the updated patch attached ;)
> >
> >> ... or the conventions for allocating memory.  Why not just use palloc()?
> >
> > That's hardly the fault of the proposed patch.  But yeah, it seems like
> > much the best fix here is to get rid of the malloc (and strdup) calls in
> > this code in favor of using the palloc infrastructure.  Even the calls
> > that *do* have manual failure checks are not compliant with our usual
> > coding standards.
> 
> Hm. Regarding the code path mentioned by Bill something like the patch
> attached is enough with a memory context for the query description.

Right.  Note this no longer needs the individual pfree() when aborting,
because the MemoryContextDelete would remove the whole thing.  (Don't
bother resubmitting; I have already fixed it.)  Will push shortly.

> Now, perhaps we could do more efforts with prodesc as well, see for
> example compile_pltcl_function for pltcl and similarly for plperl.
> Thoughts?

Right.  It would simplify the code: create a memory context child of
TopTransactionContext, then compile the function, and if successful,
then MemoryContextSetParent to some longer-lived context.  When the
function is invalidated, it's sufficient to delete the context and
create a new one.  Creating the context as child of
TopTransactionContext allows you to avoid an explicit
MemoryContextDelete() in the elog(ERROR) cases while compiling.

I do wonder how many users this code has ...

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  7. Re: Lack of Sanity Checking in file 'pctcl.c' for PostgreSQL 9.4.x

    Alvaro Herrera <alvherre@2ndquadrant.com> — 2015-07-18T20:19:15Z

    Alvaro Herrera wrote:

> > Now, perhaps we could do more efforts with prodesc as well, see for
> > example compile_pltcl_function for pltcl and similarly for plperl.
> > Thoughts?
> 
> Right.  It would simplify the code: create a memory context child of
> TopTransactionContext, then compile the function, and if successful,
> then MemoryContextSetParent to some longer-lived context.  When the
> function is invalidated, it's sufficient to delete the context and
> create a new one.  Creating the context as child of
> TopTransactionContext allows you to avoid an explicit
> MemoryContextDelete() in the elog(ERROR) cases while compiling.

With some additional effort, we could get rid of perm_fmgr_info, at
least in pltcl.  (That hack was introduced in a3ed622b63b and
7748e9e7e5a back in 2001 and we never actually fixed it ...)

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  8. Re: Lack of Sanity Checking in file 'pctcl.c' for PostgreSQL 9.4.x

    Michael Paquier <michael.paquier@gmail.com> — 2015-07-19T15:20:09Z

    On Sun, Jul 19, 2015 at 5:19 AM, Alvaro Herrera
<alvherre@2ndquadrant.com> wrote:
> Alvaro Herrera wrote:
>
>> > Now, perhaps we could do more efforts with prodesc as well, see for
>> > example compile_pltcl_function for pltcl and similarly for plperl.
>> > Thoughts?
>>
>> Right.  It would simplify the code: create a memory context child of
>> TopTransactionContext, then compile the function, and if successful,
>> then MemoryContextSetParent to some longer-lived context.  When the
>> function is invalidated, it's sufficient to delete the context and
>> create a new one.  Creating the context as child of
>> TopTransactionContext allows you to avoid an explicit
>> MemoryContextDelete() in the elog(ERROR) cases while compiling.
>
> With some additional effort, we could get rid of perm_fmgr_info, at
> least in pltcl.  (That hack was introduced in a3ed622b63b and
> 7748e9e7e5a back in 2001 and we never actually fixed it ...)

Yes it seems so, even for plperl and plpython there seem to be some
room for improvement at quick glance... This looks like a master-only
change to me though (and I am sure we are on the same page). For
back-branches something like the previous patch is definitely safer.
-- 
Michael

  9. Re: Lack of Sanity Checking in file 'pctcl.c' for PostgreSQL 9.4.x

    Alvaro Herrera <alvherre@2ndquadrant.com> — 2015-07-19T15:44:49Z

    Michael Paquier wrote:
> On Sun, Jul 19, 2015 at 5:19 AM, Alvaro Herrera
> <alvherre@2ndquadrant.com> wrote:

> > With some additional effort, we could get rid of perm_fmgr_info, at
> > least in pltcl.  (That hack was introduced in a3ed622b63b and
> > 7748e9e7e5a back in 2001 and we never actually fixed it ...)
> 
> Yes it seems so, even for plperl and plpython there seem to be some
> room for improvement at quick glance... This looks like a master-only
> change to me though (and I am sure we are on the same page). For
> back-branches something like the previous patch is definitely safer.

Yes, agreed on both counts.  I will push the patch when I have a moment.

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  10. Re: Lack of Sanity Checking in file 'pctcl.c' for PostgreSQL 9.4.x

    Alvaro Herrera <alvherre@2ndquadrant.com> — 2015-07-20T12:29:28Z

    Michael Paquier wrote:

> > With some additional effort, we could get rid of perm_fmgr_info, at
> > least in pltcl.  (That hack was introduced in a3ed622b63b and
> > 7748e9e7e5a back in 2001 and we never actually fixed it ...)
> 
> Yes it seems so, even for plperl and plpython there seem to be some
> room for improvement at quick glance... This looks like a master-only
> change to me though (and I am sure we are on the same page). For
> back-branches something like the previous patch is definitely safer.

Pushed.

All in all, the whole business of memory allocation in pltcl is pretty
nasty.  If people is really interested in pltcl, that should probably be
fixed to avoid memory leaks.  For one thing, the query cache hash is
global to the interpreter, not local to the function; whenever a
function is recompiled, the old SPI_prepared plans linger forever in the
interpreter (for the life of the session).  It would be nicer if those
were tied to the function lifetime.  For this, the hashtable would have
to be moved to the query desc struct.  pltcl_spi_prepare would have to
know the currently executing function desc struct ...

Maybe Bill is interested enough to write a patch?

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  11. Re: Lack of Sanity Checking in file 'pctcl.c' for PostgreSQL 9.4.x

    Michael Paquier <michael.paquier@gmail.com> — 2015-07-22T07:57:47Z

    On Mon, Jul 20, 2015 at 9:29 PM, Alvaro Herrera
<alvherre@2ndquadrant.com> wrote:
> Michael Paquier wrote:
>
>> > With some additional effort, we could get rid of perm_fmgr_info, at
>> > least in pltcl.  (That hack was introduced in a3ed622b63b and
>> > 7748e9e7e5a back in 2001 and we never actually fixed it ...)
>>
>> Yes it seems so, even for plperl and plpython there seem to be some
>> room for improvement at quick glance... This looks like a master-only
>> change to me though (and I am sure we are on the same page). For
>> back-branches something like the previous patch is definitely safer.
>
> Pushed.
>
> All in all, the whole business of memory allocation in pltcl is pretty
> nasty.  If people is really interested in pltcl, that should probably be
> fixed to avoid memory leaks.  For one thing, the query cache hash is
> global to the interpreter, not local to the function; whenever a
> function is recompiled, the old SPI_prepared plans linger forever in the
> interpreter (for the life of the session).  It would be nicer if those
> were tied to the function lifetime.  For this, the hashtable would have
> to be moved to the query desc struct.  pltcl_spi_prepare would have to
> know the currently executing function desc struct ...

If this is done, the same can be directly applied to plperl. For
plpython, we are going to visibly need a memory context initialized at
some point and rely on it for allocations instead of TopMemoryContext
when doing the various allocations... Do you think it is worth it in
this case?

For what it's worth, I have been playing for a couple of minutes with
the code of pltcl and plperl to replace the remaining malloc calls by
a memory context and remove the calls to perm_func_ctx. This does not
fix the memory leaks of pltcl when functions are recompiled by this is
working somewhat nicely with plperl by relying on free_plperl_function
and check-world passes. So attached they are. This seem to improve a
bit things to me. Thoughts?
-- 
Michael