Horton

Thread

Commits

  1. Invent start_proc parameters for PL/Tcl.

  2. Remove PL/Tcl's "module" facility.

  3. Remove all of the libpgtcl and pgtclsh files, including all references to

  1. I propose killing PL/Tcl's "modules" infrastructure

    Tom Lane <tgl@sss.pgh.pa.us> — 2017-02-25T18:14:07Z

    Over in
https://www.postgresql.org/message-id/alpine.DEB.2.11.1702251701030.3920@Sandal.Woodpecker
it's pointed out that pltcl_loadmod was never updated for the switch
to standard_conforming_strings (and the patch proposed there doesn't
begin to cover all the places that would need fixed for that).

This means that the "modules" functionality is entirely broken in any
installation that's got standard_conforming_strings turned on, which has
been the default since 9.1 and was possible long before that.

The fact that nobody has noticed seems to me to be clear proof that no one
is using this feature in the field.

Now, we could try to fix this bug, and add the regression test coverage
that the code clearly lacks, and upgrade the documentation about it from
its currently very sad state.  But I think the right answer is just to
remove the feature altogether.  It's evidently not being used, and it's
kind of insecure by design, and it would not be that hard for someone
to provide equivalent functionality entirely in userland if they really
wanted it.

Comments?

			regards, tom lane

  2. Re: I propose killing PL/Tcl's "modules" infrastructure

    Andrew Dunstan <andrew.dunstan@2ndquadrant.com> — 2017-02-25T18:37:26Z

    
On 02/25/2017 01:14 PM, Tom Lane wrote:
> Over in
> https://www.postgresql.org/message-id/alpine.DEB.2.11.1702251701030.3920@Sandal.Woodpecker
> it's pointed out that pltcl_loadmod was never updated for the switch
> to standard_conforming_strings (and the patch proposed there doesn't
> begin to cover all the places that would need fixed for that).
>
> This means that the "modules" functionality is entirely broken in any
> installation that's got standard_conforming_strings turned on, which has
> been the default since 9.1 and was possible long before that.
>
> The fact that nobody has noticed seems to me to be clear proof that no one
> is using this feature in the field.
>
> Now, we could try to fix this bug, and add the regression test coverage
> that the code clearly lacks, and upgrade the documentation about it from
> its currently very sad state.  But I think the right answer is just to
> remove the feature altogether.  It's evidently not being used, and it's
> kind of insecure by design, and it would not be that hard for someone
> to provide equivalent functionality entirely in userland if they really
> wanted it.
>
> Comments?
>
> 			


In PLv8 we added a parameter plv8.start_proc that names a parameterless
function that's executed when plv8 is first called in each session. It
can be used quite easily to implement something like a modules
infrastructure - in fact I have used it to good effect for exactly that.
Maybe something similar for pltcl would be a good thing.

cheers

andrew

-- 
Andrew Dunstan                https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  3. Re: I propose killing PL/Tcl's "modules" infrastructure

    Joshua D. Drake <jd@commandprompt.com> — 2017-02-25T18:39:14Z

    On 02/25/2017 10:14 AM, Tom Lane wrote:

> Now, we could try to fix this bug, and add the regression test coverage
> that the code clearly lacks, and upgrade the documentation about it from
> its currently very sad state.  But I think the right answer is just to
> remove the feature altogether.  It's evidently not being used, and it's
> kind of insecure by design, and it would not be that hard for someone
> to provide equivalent functionality entirely in userland if they really
> wanted it.

I don't see a reason to keep pl/tcl in core at all so ripping out the 
functionality seems the least disruptive and perhaps even a deprecation 
of the PL (at least from a core perspective) in v10.

JD



-- 
Command Prompt, Inc.                  http://the.postgres.company/
                         +1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Everyone appreciates your honesty, until you are honest with them.
Unless otherwise stated, opinions are my own.

  4. Re: I propose killing PL/Tcl's "modules" infrastructure

    Tom Lane <tgl@sss.pgh.pa.us> — 2017-02-25T18:44:48Z

    Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes:
> On 02/25/2017 01:14 PM, Tom Lane wrote:
>> Now, we could try to fix this bug, and add the regression test coverage
>> that the code clearly lacks, and upgrade the documentation about it from
>> its currently very sad state.  But I think the right answer is just to
>> remove the feature altogether.  It's evidently not being used, and it's
>> kind of insecure by design, and it would not be that hard for someone
>> to provide equivalent functionality entirely in userland if they really
>> wanted it.

> In PLv8 we added a parameter plv8.start_proc that names a parameterless
> function that's executed when plv8 is first called in each session. It
> can be used quite easily to implement something like a modules
> infrastructure - in fact I have used it to good effect for exactly that.
> Maybe something similar for pltcl would be a good thing.

Yeah, the only part that's even a bit hard to replicate in userland is
initializing the autoloading mechanism in each session.  It would be
cleaner to provide a feature similar to what you describe that could
be used for that purpose as well as others.  However, where does the
"parameterless function" come from?  Is it a regular PLv8 (or for this
purpose PL/Tcl) function expected to be present in pg_proc?

			regards, tom lane

  5. Re: I propose killing PL/Tcl's "modules" infrastructure

    Tom Lane <tgl@sss.pgh.pa.us> — 2017-02-25T18:46:53Z

    "Joshua D. Drake" <jd@commandprompt.com> writes:
> I don't see a reason to keep pl/tcl in core at all so ripping out the 
> functionality seems the least disruptive and perhaps even a deprecation 
> of the PL (at least from a core perspective) in v10.

I'm not in any hurry to remove or deprecate PL/Tcl as a whole.  It's
gotten more love in the past year than it got in the previous ten,
so evidently there are still people out there who use it.  But they
don't seem to be using this particular part of it.

			regards, tom lane

  6. Re: I propose killing PL/Tcl's "modules" infrastructure

    Andrew Dunstan <andrew.dunstan@2ndquadrant.com> — 2017-02-25T18:55:26Z

    
On 02/25/2017 01:44 PM, Tom Lane wrote:
> Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes:
>> On 02/25/2017 01:14 PM, Tom Lane wrote:
>>> Now, we could try to fix this bug, and add the regression test coverage
>>> that the code clearly lacks, and upgrade the documentation about it from
>>> its currently very sad state.  But I think the right answer is just to
>>> remove the feature altogether.  It's evidently not being used, and it's
>>> kind of insecure by design, and it would not be that hard for someone
>>> to provide equivalent functionality entirely in userland if they really
>>> wanted it.
>> In PLv8 we added a parameter plv8.start_proc that names a parameterless
>> function that's executed when plv8 is first called in each session. It
>> can be used quite easily to implement something like a modules
>> infrastructure - in fact I have used it to good effect for exactly that.
>> Maybe something similar for pltcl would be a good thing.
> Yeah, the only part that's even a bit hard to replicate in userland is
> initializing the autoloading mechanism in each session.  It would be
> cleaner to provide a feature similar to what you describe that could
> be used for that purpose as well as others.  However, where does the
> "parameterless function" come from?  Is it a regular PLv8 (or for this
> purpose PL/Tcl) function expected to be present in pg_proc?
>
> 			


Yes, it's a regular PLv8 function.Here's an example. It presupposes that
there is a table called plv8_modules (modname text, code text,
load_on_start boolean).

    CREATE OR REPLACE FUNCTION public.plv8_startup()
     RETURNS void
     LANGUAGE plv8
    AS $function$
      if (typeof plv8_loaded_modules == 'undefined')
        plv8_loaded_modules = {};
      load_module = function(modname)
      {
        if (plv8_loaded_modules[modname])
            return;
        var rows = plv8.execute("SELECT code from plv8_modules " +
                                " where modname = $1", [modname]);
        for (var r = 0; r < rows.length; r++)
        {
            var code = rows[r].code;
            eval("(function() { " + code + "})")();
            // plv8.elog(NOTICE,"loaded module " + modname);
            plv8_loaded_modules[modname] = 1;
        }
           
      };
      reload_module = function(modname)
      {
        var rows = plv8.execute("SELECT code from plv8_modules " +
                                " where modname = $1", [modname]);
        for (var r = 0; r < rows.length; r++)
        {
            var code = rows[r].code;
            eval("(function() { " + code + "})")();
            // plv8.elog(NOTICE,"loaded module " + modname);
            plv8_loaded_modules[modname] = 1;
        }

      };

      var rows = plv8.execute("SELECT modname, code from plv8_modules
    where load_on_start");
      for (var r = 0; r < rows.length; r++)
      {
        var modname = rows[r].modname;
        if (plv8_loaded_modules[modname])
            continue;
        var code = rows[r].code;
        eval("(function() { " + code + "})")();
        plv8_loaded_modules[modname] = 1;
        // plv8.elog(NOTICE,"loaded module " + modname);
      };
    $function$;


cheers

andrew

-- 
Andrew Dunstan                https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  7. Re: I propose killing PL/Tcl's "modules" infrastructure

    Tom Lane <tgl@sss.pgh.pa.us> — 2017-02-25T19:21:25Z

    Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes:
> On 02/25/2017 01:44 PM, Tom Lane wrote:
>> Yeah, the only part that's even a bit hard to replicate in userland is
>> initializing the autoloading mechanism in each session.  It would be
>> cleaner to provide a feature similar to what you describe that could
>> be used for that purpose as well as others.  However, where does the
>> "parameterless function" come from?  Is it a regular PLv8 (or for this
>> purpose PL/Tcl) function expected to be present in pg_proc?

> Yes, it's a regular PLv8 function.

OK ... how do you handle security considerations?  Can the GUC be set
at any time/by anybody?  What determines whether you have permissions
to call the particular function?

			regards, tom lane

  8. Re: I propose killing PL/Tcl's "modules" infrastructure

    Andrew Dunstan <andrew.dunstan@2ndquadrant.com> — 2017-02-25T20:11:12Z

    
On 02/25/2017 02:21 PM, Tom Lane wrote:
> Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes:
>> On 02/25/2017 01:44 PM, Tom Lane wrote:
>>> Yeah, the only part that's even a bit hard to replicate in userland is
>>> initializing the autoloading mechanism in each session.  It would be
>>> cleaner to provide a feature similar to what you describe that could
>>> be used for that purpose as well as others.  However, where does the
>>> "parameterless function" come from?  Is it a regular PLv8 (or for this
>>> purpose PL/Tcl) function expected to be present in pg_proc?
>> Yes, it's a regular PLv8 function.
> OK ... how do you handle security considerations?  Can the GUC be set
> at any time/by anybody?  What determines whether you have permissions
> to call the particular function?
>
> 			


It can be set by anyone, IIRC. Maybe it should be SUSET only, I don't
know. It's executed as the session owner. Execute permission on the
function are determined the same way as for any function. It's an
ordinary function call. The only difference is in how the call gets
triggered.

cheers

andrew


-- 
Andrew Dunstan                https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  9. Re: I propose killing PL/Tcl's "modules" infrastructure

    Tom Lane <tgl@sss.pgh.pa.us> — 2017-02-26T18:09:31Z

    I wrote:
> Now, we could try to fix this bug, and add the regression test coverage
> that the code clearly lacks, and upgrade the documentation about it from
> its currently very sad state.  But I think the right answer is just to
> remove the feature altogether.

BTW, I tried to poke into what it would take to write some regression test
coverage, and immediately hit a show-stopper:

$ pltcl_loadmod --help
can't find package Pgtcl
    while executing
"package require Pgtcl"
    (file "/home/postgres/testversion/bin/pltcl_loadmod" line 10)

That is, these scripts depend on the old Tcl client library, which
we removed from our core distro in 2004 (cf commit 41fa9e9ba).
So we don't even have a way of creating self-contained tests for them.

At this point I think there's no question that src/pl/tcl/modules/
needs to go away.  There might be some argument for retaining the
"autoload the unknown module" startup behavior in pltcl proper, but
I think that Andrew Dunstan's proposal of calling an initialization
function is a far cleaner solution.

			regards, tom lane

  10. Re: I propose killing PL/Tcl's "modules" infrastructure

    Tom Lane <tgl@sss.pgh.pa.us> — 2017-02-26T19:54:42Z

    Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes:
> [ we should borrow plv8's start_proc idea for pltcl ]

So after thinking about this for awhile, I propose the following
concrete spec for replacing pltcl's autoload-unknown behavior:

* Invent two GUCs, pltcl.start_proc and pltclu.start_proc, which default
to empty strings but can be set to the name (possibly schema-qualified)
of a parameterless function that must be of the corresponding language.
When so set, the specified function is called once just after creation of
any pltcl or pltclu interpreter.

* The function is called as the current SQL user, who must have
permissions to call it.  (This decision is more or less forced by
the fact that pltcl interpreters are per-userid; we want whatever
initialization gets done to be done in the new interpreter, and it
would be very weird and probably a security hole if we weren't
running as the same SQL userid that owns the interpreter.)

* Pre-call error conditions (no such function, wrong language, or no
permissions) result in a WARNING but the original operation continues.
(Making these WARNING not ERROR is possibly debatable, but it looks
like that's what plv8 does.)

* If the function itself throws an error, we do a transaction abort,
but the Tcl interpreter remains in existence and is deemed usable
for later operations.  So a failing start_proc can't lock you out
of pltcl operations altogether.

* I'm not terribly comfortable about what the permissions levels of the
GUCs ought to be.  The call permissions check means that you can't use
either GUC to call a function you couldn't have called anyway.  However
there's a separate risk of trojan-horse execution, analogous to what a
blackhat can get by controlling the search_path GUC setting used by a
SECURITY DEFINER function: the function might intend to invoke some pltcl
function, but you can get it to invoke some other pltcl function in
addition to that.  I think this means we had better make pltclu.start_proc
be SUSET, but from a convenience standpoint it'd be nice if
pltcl.start_proc were just USERSET.  An argument in favor of that is that
we don't restrict search_path which is just as dangerous; but on the other
hand, existing code should be expected to know that it needs to beware of
search_path, while it wouldn't know that start_proc needs to be locked
down.  Maybe we'd better make them both SUSET.

Comments?

			regards, tom lane

  11. Re: I propose killing PL/Tcl's "modules" infrastructure

    Robert Haas <robertmhaas@gmail.com> — 2017-02-27T06:53:04Z

    On Mon, Feb 27, 2017 at 1:24 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> * I'm not terribly comfortable about what the permissions levels of the
> GUCs ought to be.  The call permissions check means that you can't use
> either GUC to call a function you couldn't have called anyway.  However
> there's a separate risk of trojan-horse execution, analogous to what a
> blackhat can get by controlling the search_path GUC setting used by a
> SECURITY DEFINER function: the function might intend to invoke some pltcl
> function, but you can get it to invoke some other pltcl function in
> addition to that.  I think this means we had better make pltclu.start_proc
> be SUSET, but from a convenience standpoint it'd be nice if
> pltcl.start_proc were just USERSET.  An argument in favor of that is that
> we don't restrict search_path which is just as dangerous; but on the other
> hand, existing code should be expected to know that it needs to beware of
> search_path, while it wouldn't know that start_proc needs to be locked
> down.  Maybe we'd better make them both SUSET.

Making them SUSET sounds like a usability fail to me.  I'm not sure
how bad the security risks of NOT making them SUSET are, but I think
if we find that SUSET is required for safety then we've squeezed most
of the value out of the feature.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  12. Re: I propose killing PL/Tcl's "modules" infrastructure

    Tom Lane <tgl@sss.pgh.pa.us> — 2017-02-27T12:48:13Z

    Robert Haas <robertmhaas@gmail.com> writes:
> On Mon, Feb 27, 2017 at 1:24 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> * I'm not terribly comfortable about what the permissions levels of the
>> GUCs ought to be. ... Maybe we'd better make them both SUSET.

> Making them SUSET sounds like a usability fail to me.  I'm not sure
> how bad the security risks of NOT making them SUSET are, but I think
> if we find that SUSET is required for safety then we've squeezed most
> of the value out of the feature.

Well, the feature it's replacing (autoload an "unknown" module) had to be
squeezed down to being effectively superuser-only, so we're not really
losing anything compared to where we are now.  And the more I think about
it, the less I think we can introduce a new security-critical GUC and just
leave it as USERSET.

			regards, tom lane

  13. Re: I propose killing PL/Tcl's "modules" infrastructure

    Andrew Dunstan <andrew.dunstan@2ndquadrant.com> — 2017-02-27T14:36:56Z

    
On 02/26/2017 02:54 PM, Tom Lane wrote:
> * I'm not terribly comfortable about what the permissions levels of the
> GUCs ought to be.  The call permissions check means that you can't use
> either GUC to call a function you couldn't have called anyway.  However
> there's a separate risk of trojan-horse execution, analogous to what a
> blackhat can get by controlling the search_path GUC setting used by a
> SECURITY DEFINER function: the function might intend to invoke some pltcl
> function, but you can get it to invoke some other pltcl function in
> addition to that.  I think this means we had better make pltclu.start_proc
> be SUSET, but from a convenience standpoint it'd be nice if
> pltcl.start_proc were just USERSET.  An argument in favor of that is that
> we don't restrict search_path which is just as dangerous; but on the other
> hand, existing code should be expected to know that it needs to beware of
> search_path, while it wouldn't know that start_proc needs to be locked
> down.  Maybe we'd better make them both SUSET.
>

plperl's on_plperl_init and on_plperlu_init settings are both SUSET.

In practice with PLv8 this is usually set in the config file in my
experience.

cheers

andrew


-- 
Andrew Dunstan                https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  14. Re: I propose killing PL/Tcl's "modules" infrastructure

    Tom Lane <tgl@sss.pgh.pa.us> — 2017-02-27T20:42:26Z

    Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes:
> On 02/26/2017 02:54 PM, Tom Lane wrote:
>> * I'm not terribly comfortable about what the permissions levels of the
>> GUCs ought to be.

> plperl's on_plperl_init and on_plperlu_init settings are both SUSET.
> In practice with PLv8 this is usually set in the config file in my
> experience.

Ah, I'd forgotten about that precedent.  Being consistent with that seems
like a good thing --- and I agree with your point that this would likely
usually be set in postgresql.conf anyway, making the issue rather moot.

I noticed also that the precedent of plperl is that if the init code
fails, we give up on use of that interpreter, and try again to run
the init code if plperl is used again.  This is different from what
I had in my draft spec but it probably is a better definition; without
it, people might find themselves running in Tcl interpreters that do
not behave as intended.

In sum, then, PFA a patch that adds these GUCs.  They're still function
names but otherwise the details are closer to what plperl does.

			regards, tom lane

  15. Re: I propose killing PL/Tcl's "modules" infrastructure

    Jim Nasby <jim.nasby@bluetreble.com> — 2017-03-01T22:31:36Z

    On 2/27/17 2:42 PM, Tom Lane wrote:
> + SET pltcl.start_proc = 'no_such_function';
> + select tcl_int4add(1, 2);
> + ERROR:  function no_such_function() does not exist

Can the error message be more explicit somehow? Otherwise people will be 
quite confused as to where no_such_function() is coming from.

<begin creature-feep>
BTW, I'd think this functionality would be valuable for every PL. Maybe 
it's worth adding formal support for it to pg_language et all and leave 
it up to each language to decide whether it's supported or not? Multiple 
init functions might be useful too, similar to how we support multiple 
hook functions (though presumably a field of regproc[] is a better way 
to handle that...)

I'm also wondering if there'd be value to supporting code that runs on 
each function invocation.
-- 
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com
855-TREBLE2 (855-873-2532)

  16. Re: I propose killing PL/Tcl's "modules" infrastructure

    Tom Lane <tgl@sss.pgh.pa.us> — 2017-03-02T17:59:58Z

    Jim Nasby <Jim.Nasby@BlueTreble.com> writes:
> On 2/27/17 2:42 PM, Tom Lane wrote:
>> + SET pltcl.start_proc = 'no_such_function';
>> + select tcl_int4add(1, 2);
>> + ERROR:  function no_such_function() does not exist

> Can the error message be more explicit somehow? Otherwise people will be 
> quite confused as to where no_such_function() is coming from.

After thinking about that for awhile, it seemed like the most useful thing
to do is to set up an errcontext callback that will be active throughout
execution of the start_proc.  That will cover both setup failures like
the above, and errors occurring within the start_proc, which could be
equally confusing if you think they apply to the function you initially
tried to call.  v2 patch attached that does it like that.

> <begin creature-feep>
> BTW, I'd think this functionality would be valuable for every PL.

Maybe for some.  I see no value in putting anything about it in
pg_language though.  I don't see that we could share any useful amount of
mechanism, and it won't necessarily look the same in every language ---
plperl for instance prefers code fragments over procedures.

In any case, I'm not going there in this patch.

			regards, tom lane

  17. Re: I propose killing PL/Tcl's "modules" infrastructure

    Robert Haas <robertmhaas@gmail.com> — 2017-03-04T08:24:11Z

    On Thu, Mar 2, 2017 at 11:29 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> After thinking about that for awhile, it seemed like the most useful thing
> to do is to set up an errcontext callback that will be active throughout
> execution of the start_proc.  That will cover both setup failures like
> the above, and errors occurring within the start_proc, which could be
> equally confusing if you think they apply to the function you initially
> tried to call.  v2 patch attached that does it like that.

+1 for that approach.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company