Horton

Thread

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Move WAL sequence code into its own file

  2. Add ExplainState argument to pg_plan_query() and planner().

  3. Don't include access/htup_details.h in executor/tuptable.h

  4. Refactor to avoid code duplication in transformPLAssignStmt.

  5. Avoid including commands/dbcommands.h in so many places

  6. Restrict psql meta-commands in plain-text dumps.

  7. Split func.sgml into more manageable pieces

  8. Fix squashing algorithm for query texts

  9. EXPLAIN: Always use two fractional digits for row counts.

  10. Preliminary refactoring of plpgsql expression construction.

  11. plpgsql: pure parser and reentrant scanner

  12. Add some sanity checks in executor for query ID reporting

  13. Fix misleading error message context

  14. Add macros for looping through a List without a ListCell.

  1. proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2017-10-26T07:21:24Z

    Hi,

I propose a  new database object - a variable. The variable is persistent
object, that holds unshared session based not transactional in memory value
of any type. Like variables in any other languages. The persistence is
required for possibility to do static checks, but can be limited to session
- the variables can be temporal.

My proposal is related to session variables from Sybase, MSSQL or MySQL
(based on prefix usage @ or @@), or package variables from Oracle (access
is controlled by scope), or schema variables from DB2. Any design is coming
from different sources, traditions and has some advantages or
disadvantages. The base of my proposal is usage schema variables as session
variables for stored procedures. It should to help to people who try to
port complex projects to PostgreSQL from other databases.

The Sybase  (T-SQL) design is good for interactive work, but it is weak for
usage in stored procedures - the static check is not possible. Is not
possible to set some access rights on variables.

The ADA design (used on Oracle) based on scope is great, but our
environment is not nested. And we should to support other PL than PLpgSQL
more strongly.

There is not too much other possibilities - the variable that should be
accessed from different PL, different procedures (in time) should to live
somewhere over PL, and there is the schema only.

The variable can be created by CREATE statement:

CREATE VARIABLE public.myvar AS integer;
CREATE VARIABLE myschema.myvar AS mytype;

CREATE [TEMP] VARIABLE [IF NOT EXISTS] name AS type
  [ DEFAULT expression ] [[NOT] NULL]
  [ ON TRANSACTION END { RESET | DROP } ]
  [ { VOLATILE | STABLE } ];

It is dropped by command DROP VARIABLE  [ IF EXISTS] varname.

The access rights is controlled by usual access rights - by commands
GRANT/REVOKE. The possible rights are: READ, WRITE

The variables can be modified by SQL command SET (this is taken from
standard, and it natural)

SET varname = expression;

Unfortunately we use the SET command for different purpose. But I am
thinking so we can solve it with few tricks. The first is moving our GUC to
pg_catalog schema. We can control the strictness of SET command. In one
variant, we can detect custom GUC and allow it, in another we can disallow
a custom GUC and allow only schema variables. A new command LET can be
alternative.

The variables should be used in queries implicitly (without JOIN)

SELECT varname;

The SEARCH_PATH is used, when varname is located. The variables can be used
everywhere where query parameters are allowed.

I hope so this proposal is good enough and simple.

Comments, notes?

regards

Pavel

  2. Re: proposal: schema variables

    Nico Williams <nico@cryptonector.com> — 2017-10-26T22:07:33Z

    On Thu, Oct 26, 2017 at 09:21:24AM +0200, Pavel Stehule wrote:
> Comments, notes?

I like it.

I would further like to move all of postgresql.conf into the database,
as much as possible, as well as pg_ident.conf and pg_hba.conf.

Variables like current_user have a sort of nesting context
functionality: calling a SECURITY DEFINER function "pushes" a new value
onto current_user, then when the function returns the new value of
current_user is "popped" and the previous value restored.

It might be nice to be able to generalize this.

Questions that then arise:

 - can one see up the stack?
 - are there permissions issues with seeing up the stack?

I recently posted proposing a feature such that SECURITY DEFINER
functions could observe the _caller_'s current_user.

Nico
--

  3. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2017-10-27T05:08:43Z

    Hi

2017-10-27 0:07 GMT+02:00 Nico Williams <nico@cryptonector.com>:

> On Thu, Oct 26, 2017 at 09:21:24AM +0200, Pavel Stehule wrote:
> > Comments, notes?
>
> I like it.
>
> I would further like to move all of postgresql.conf into the database,
> as much as possible, as well as pg_ident.conf and pg_hba.conf.
>
> Variables like current_user have a sort of nesting context
> functionality: calling a SECURITY DEFINER function "pushes" a new value
> onto current_user, then when the function returns the new value of
> current_user is "popped" and the previous value restored.
>

My proposal doesn't expecting with nesting, because there is only one scope
- schema / session - but I don't think so it is necessary

current_user is a function - it is based on parser magic in Postgres. The
origin from Oracle uses the feature of ADA language. When function has no
parameters then parenthesis are optional. So current_user, current_time are
functions current_user(), current_time().


> It might be nice to be able to generalize this.
>
> Questions that then arise:
>
>  - can one see up the stack?
>  - are there permissions issues with seeing up the stack?
>

these variables are pined to schema - so there is not any relation to
stack. It is like global variables.

Theoretically we can introduce "functional" variables, where the value is
based on immediate evaluation of expression. It can be very similar to
current current_user.

>
>
> I recently posted proposing a feature such that SECURITY DEFINER
> functions could observe the _caller_'s current_user.
>

your use case is good example - this proposed feature doesn't depend on
stack, depends on security context (security context stack) what is super
set of call stack

Regards

Pavel



> Nico
> --
>

  4. Re: proposal: schema variables

    Tatsuo Ishii <ishii@sraoss.co.jp> — 2017-10-27T05:30:11Z

    > Hi,
> 
> I propose a  new database object - a variable. The variable is persistent
> object, that holds unshared session based not transactional in memory value
> of any type. Like variables in any other languages. The persistence is
> required for possibility to do static checks, but can be limited to session
> - the variables can be temporal.
> 
> My proposal is related to session variables from Sybase, MSSQL or MySQL
> (based on prefix usage @ or @@), or package variables from Oracle (access
> is controlled by scope), or schema variables from DB2. Any design is coming
> from different sources, traditions and has some advantages or
> disadvantages. The base of my proposal is usage schema variables as session
> variables for stored procedures. It should to help to people who try to
> port complex projects to PostgreSQL from other databases.
> 
> The Sybase  (T-SQL) design is good for interactive work, but it is weak for
> usage in stored procedures - the static check is not possible. Is not
> possible to set some access rights on variables.
> 
> The ADA design (used on Oracle) based on scope is great, but our
> environment is not nested. And we should to support other PL than PLpgSQL
> more strongly.
> 
> There is not too much other possibilities - the variable that should be
> accessed from different PL, different procedures (in time) should to live
> somewhere over PL, and there is the schema only.
> 
> The variable can be created by CREATE statement:
> 
> CREATE VARIABLE public.myvar AS integer;
> CREATE VARIABLE myschema.myvar AS mytype;
> 
> CREATE [TEMP] VARIABLE [IF NOT EXISTS] name AS type
>   [ DEFAULT expression ] [[NOT] NULL]
>   [ ON TRANSACTION END { RESET | DROP } ]
>   [ { VOLATILE | STABLE } ];
> 
> It is dropped by command DROP VARIABLE  [ IF EXISTS] varname.
> 
> The access rights is controlled by usual access rights - by commands
> GRANT/REVOKE. The possible rights are: READ, WRITE
> 
> The variables can be modified by SQL command SET (this is taken from
> standard, and it natural)
> 
> SET varname = expression;
> 
> Unfortunately we use the SET command for different purpose. But I am
> thinking so we can solve it with few tricks. The first is moving our GUC to
> pg_catalog schema. We can control the strictness of SET command. In one
> variant, we can detect custom GUC and allow it, in another we can disallow
> a custom GUC and allow only schema variables. A new command LET can be
> alternative.
> 
> The variables should be used in queries implicitly (without JOIN)
> 
> SELECT varname;
> 
> The SEARCH_PATH is used, when varname is located. The variables can be used
> everywhere where query parameters are allowed.
> 
> I hope so this proposal is good enough and simple.
> 
> Comments, notes?

Just q quick follow up. Looks like a greate feature!

Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp

  5. Re: proposal: schema variables

    Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com> — 2017-10-27T05:47:29Z

    From: pgsql-hackers-owner@postgresql.org
> [mailto:pgsql-hackers-owner@postgresql.org] On Behalf Of Pavel Stehule
> I propose a  new database object - a variable. The variable is persistent
> object, that holds unshared session based not transactional in memory value
> of any type. Like variables in any other languages. The persistence is
> required for possibility to do static checks, but can be limited to session
> - the variables can be temporal.
> 
> 
> My proposal is related to session variables from Sybase, MSSQL or MySQL
> (based on prefix usage @ or @@), or package variables from Oracle (access
> is controlled by scope), or schema variables from DB2. Any design is coming
> from different sources, traditions and has some advantages or disadvantages.
> The base of my proposal is usage schema variables as session variables for
> stored procedures. It should to help to people who try to port complex
> projects to PostgreSQL from other databases.

Very interesting.  I hope I could join the review and testing.

How do you think this would contribute to easing the port of Oracle PL/SQL procedures?  Would the combination of orafce and this feature promote auto-translation of PL/SQL procedures?  I'm curious what will be the major road blocks after adding the schema variable.

Regards
Takayuki Tsunakawa

  6. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2017-10-27T06:16:56Z

    2017-10-27 7:47 GMT+02:00 Tsunakawa, Takayuki <
tsunakawa.takay@jp.fujitsu.com>:

> From: pgsql-hackers-owner@postgresql.org
> > [mailto:pgsql-hackers-owner@postgresql.org] On Behalf Of Pavel Stehule
> > I propose a  new database object - a variable. The variable is persistent
> > object, that holds unshared session based not transactional in memory
> value
> > of any type. Like variables in any other languages. The persistence is
> > required for possibility to do static checks, but can be limited to
> session
> > - the variables can be temporal.
> >
> >
> > My proposal is related to session variables from Sybase, MSSQL or MySQL
> > (based on prefix usage @ or @@), or package variables from Oracle (access
> > is controlled by scope), or schema variables from DB2. Any design is
> coming
> > from different sources, traditions and has some advantages or
> disadvantages.
> > The base of my proposal is usage schema variables as session variables
> for
> > stored procedures. It should to help to people who try to port complex
> > projects to PostgreSQL from other databases.
>
> Very interesting.  I hope I could join the review and testing.
>

you are welcome. I wrote a prototype last year based on envelope functions.
But the integration must be much more close to SQL to be some clear benefit
of this feature. So there is lot of work. I hope so I have a prototype
after this winter. It is my plan for winter.


>
> How do you think this would contribute to easing the port of Oracle PL/SQL
> procedures?  Would the combination of orafce and this feature promote
> auto-translation of PL/SQL procedures?  I'm curious what will be the major
> road blocks after adding the schema variable.
>

It depends on creativity of PL/SQL developers. Usual .. 80% application is
possible to migrate with current GUC - some work does ora2pg. But GUC is
little bit slower (not too important) and is not simple possibility to
secure it.

So work with variables will be similar like GUC, but significantly more
natural (not necessary to build wrap functions). It should be much better
when value is of some composite type. The migrations will need some
inteligence still, but less work and code will be more readable and cleaner.

I talked already about "schema pined" functions (schema private/public
objects) - but I didn't think about it more deeply. There can be special
access right to schema variables, the pined schema can be preferred before
search_path. With this feature the schema will have very similar behave
like Oracle Modules. Using different words - we can implement scope access
rights based on schemas. But it is far horizon. What is important -
proposal doesn't block any future enhancing in this case, and is consistent
with current state. In future you can work with schema private functions,
tables, variables, sequences. So variables are nothing special.

Regards

Pavel


Regards
> Takayuki Tsunakawa
>
>
>

  7. Re: proposal: schema variables

    Gilles Darold <gilles.darold@dalibo.com> — 2017-10-27T13:38:40Z

    Le 26/10/2017 à 09:21, Pavel Stehule a écrit :
> Hi,
>
> I propose a  new database object - a variable. The variable is
> persistent object, that holds unshared session based not transactional
> in memory value of any type. Like variables in any other languages.
> The persistence is required for possibility to do static checks, but
> can be limited to session - the variables can be temporal.
>
> My proposal is related to session variables from Sybase, MSSQL or
> MySQL (based on prefix usage @ or @@), or package variables from
> Oracle (access is controlled by scope), or schema variables from DB2.
> Any design is coming from different sources, traditions and has some
> advantages or disadvantages. The base of my proposal is usage schema
> variables as session variables for stored procedures. It should to
> help to people who try to port complex projects to PostgreSQL from
> other databases.
>
> The Sybase  (T-SQL) design is good for interactive work, but it is
> weak for usage in stored procedures - the static check is not
> possible. Is not possible to set some access rights on variables.
>
> The ADA design (used on Oracle) based on scope is great, but our
> environment is not nested. And we should to support other PL than
> PLpgSQL more strongly.
>
> There is not too much other possibilities - the variable that should
> be accessed from different PL, different procedures (in time) should
> to live somewhere over PL, and there is the schema only.
>
> The variable can be created by CREATE statement:
>
> CREATE VARIABLE public.myvar AS integer;
> CREATE VARIABLE myschema.myvar AS mytype;
>
> CREATE [TEMP] VARIABLE [IF NOT EXISTS] name AS type
>   [ DEFAULT expression ] [[NOT] NULL]
>   [ ON TRANSACTION END { RESET | DROP } ]
>   [ { VOLATILE | STABLE } ];
>
> It is dropped by command DROP VARIABLE  [ IF EXISTS] varname.
>
> The access rights is controlled by usual access rights - by commands
> GRANT/REVOKE. The possible rights are: READ, WRITE
>
> The variables can be modified by SQL command SET (this is taken from
> standard, and it natural)
>
> SET varname = expression;
>
> Unfortunately we use the SET command for different purpose. But I am
> thinking so we can solve it with few tricks. The first is moving our
> GUC to pg_catalog schema. We can control the strictness of SET
> command. In one variant, we can detect custom GUC and allow it, in
> another we can disallow a custom GUC and allow only schema variables.
> A new command LET can be alternative.
>
> The variables should be used in queries implicitly (without JOIN)
>
> SELECT varname;
>
> The SEARCH_PATH is used, when varname is located. The variables can be
> used everywhere where query parameters are allowed.
>
> I hope so this proposal is good enough and simple.
>
> Comments, notes?
>
> regards
>
> Pavel
>
>

Great feature that will help for migration. How will you handle CONSTANT
declaration? With Oracle it is possible to declare a constant as follow:


  varname     CONSTANT INTEGER    := 500;


for a variable that can't be changed. Do you plan to add a CONSTANT or
READONLY keyword or do you want use GRANT on the object to deal with
this case?


Regards

-- 
Gilles Darold
Consultant PostgreSQL
http://dalibo.com - http://dalibo.org

  8. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2017-10-27T14:09:36Z

    2017-10-27 15:38 GMT+02:00 Gilles Darold <gilles.darold@dalibo.com>:

> Le 26/10/2017 à 09:21, Pavel Stehule a écrit :
> > Hi,
> >
> > I propose a  new database object - a variable. The variable is
> > persistent object, that holds unshared session based not transactional
> > in memory value of any type. Like variables in any other languages.
> > The persistence is required for possibility to do static checks, but
> > can be limited to session - the variables can be temporal.
> >
> > My proposal is related to session variables from Sybase, MSSQL or
> > MySQL (based on prefix usage @ or @@), or package variables from
> > Oracle (access is controlled by scope), or schema variables from DB2.
> > Any design is coming from different sources, traditions and has some
> > advantages or disadvantages. The base of my proposal is usage schema
> > variables as session variables for stored procedures. It should to
> > help to people who try to port complex projects to PostgreSQL from
> > other databases.
> >
> > The Sybase  (T-SQL) design is good for interactive work, but it is
> > weak for usage in stored procedures - the static check is not
> > possible. Is not possible to set some access rights on variables.
> >
> > The ADA design (used on Oracle) based on scope is great, but our
> > environment is not nested. And we should to support other PL than
> > PLpgSQL more strongly.
> >
> > There is not too much other possibilities - the variable that should
> > be accessed from different PL, different procedures (in time) should
> > to live somewhere over PL, and there is the schema only.
> >
> > The variable can be created by CREATE statement:
> >
> > CREATE VARIABLE public.myvar AS integer;
> > CREATE VARIABLE myschema.myvar AS mytype;
> >
> > CREATE [TEMP] VARIABLE [IF NOT EXISTS] name AS type
> >   [ DEFAULT expression ] [[NOT] NULL]
> >   [ ON TRANSACTION END { RESET | DROP } ]
> >   [ { VOLATILE | STABLE } ];
> >
> > It is dropped by command DROP VARIABLE  [ IF EXISTS] varname.
> >
> > The access rights is controlled by usual access rights - by commands
> > GRANT/REVOKE. The possible rights are: READ, WRITE
> >
> > The variables can be modified by SQL command SET (this is taken from
> > standard, and it natural)
> >
> > SET varname = expression;
> >
> > Unfortunately we use the SET command for different purpose. But I am
> > thinking so we can solve it with few tricks. The first is moving our
> > GUC to pg_catalog schema. We can control the strictness of SET
> > command. In one variant, we can detect custom GUC and allow it, in
> > another we can disallow a custom GUC and allow only schema variables.
> > A new command LET can be alternative.
> >
> > The variables should be used in queries implicitly (without JOIN)
> >
> > SELECT varname;
> >
> > The SEARCH_PATH is used, when varname is located. The variables can be
> > used everywhere where query parameters are allowed.
> >
> > I hope so this proposal is good enough and simple.
> >
> > Comments, notes?
> >
> > regards
> >
> > Pavel
> >
> >
>
> Great feature that will help for migration. How will you handle CONSTANT
> declaration? With Oracle it is possible to declare a constant as follow:
>
>
>   varname     CONSTANT INTEGER    := 500;
>
>
> for a variable that can't be changed. Do you plan to add a CONSTANT or
> READONLY keyword or do you want use GRANT on the object to deal with
> this case?
>

Plpgsql  declaration supports CONSTANT

I forgot it. Thank you

Pavel



>
> Regards
>
> --
> Gilles Darold
> Consultant PostgreSQL
> http://dalibo.com - http://dalibo.org
>
>
>
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers
>

  9. Re: proposal: schema variables

    Chris Travers <chris.travers@adjust.com> — 2017-10-28T14:24:43Z

    On Thu, Oct 26, 2017 at 9:21 AM, Pavel Stehule <pavel.stehule@gmail.com>
wrote:

> Hi,
>
> I propose a  new database object - a variable. The variable is persistent
> object, that holds unshared session based not transactional in memory value
> of any type. Like variables in any other languages. The persistence is
> required for possibility to do static checks, but can be limited to session
> - the variables can be temporal.
>
> My proposal is related to session variables from Sybase, MSSQL or MySQL
> (based on prefix usage @ or @@), or package variables from Oracle (access
> is controlled by scope), or schema variables from DB2. Any design is coming
> from different sources, traditions and has some advantages or
> disadvantages. The base of my proposal is usage schema variables as session
> variables for stored procedures. It should to help to people who try to
> port complex projects to PostgreSQL from other databases.
>
> The Sybase  (T-SQL) design is good for interactive work, but it is weak
> for usage in stored procedures - the static check is not possible. Is not
> possible to set some access rights on variables.
>
> The ADA design (used on Oracle) based on scope is great, but our
> environment is not nested. And we should to support other PL than PLpgSQL
> more strongly.
>
> There is not too much other possibilities - the variable that should be
> accessed from different PL, different procedures (in time) should to live
> somewhere over PL, and there is the schema only.
>
> The variable can be created by CREATE statement:
>
> CREATE VARIABLE public.myvar AS integer;
> CREATE VARIABLE myschema.myvar AS mytype;
>
> CREATE [TEMP] VARIABLE [IF NOT EXISTS] name AS type
>   [ DEFAULT expression ] [[NOT] NULL]
>   [ ON TRANSACTION END { RESET | DROP } ]
>   [ { VOLATILE | STABLE } ];
>
> It is dropped by command DROP VARIABLE  [ IF EXISTS] varname.
>
> The access rights is controlled by usual access rights - by commands
> GRANT/REVOKE. The possible rights are: READ, WRITE
>
> The variables can be modified by SQL command SET (this is taken from
> standard, and it natural)
>
> SET varname = expression;
>
> Unfortunately we use the SET command for different purpose. But I am
> thinking so we can solve it with few tricks. The first is moving our GUC to
> pg_catalog schema. We can control the strictness of SET command. In one
> variant, we can detect custom GUC and allow it, in another we can disallow
> a custom GUC and allow only schema variables. A new command LET can be
> alternative.
>
> The variables should be used in queries implicitly (without JOIN)
>
> SELECT varname;
>
> The SEARCH_PATH is used, when varname is located. The variables can be
> used everywhere where query parameters are allowed.
>
> I hope so this proposal is good enough and simple.
>
> Comments, notes?
>


I have a question on this.  Since one can issue set commands on arbitrary
settings (and later ALTER database/role/system on settings you have created
in the current session) I am wondering how much overlap there is between a
sort of extended GUC with custom settings and variables.

Maybe it would be simpler to treat variables and GUC settings to be similar
and see what can be done to extend GUC in this way?

I mean if instead we allowed restricting SET to known settings then we
could have a CREATE SETTING command which would behave like this and then
use SET the same way across both.

In essence I am wondering if this really needs to be as separate from GUC
as you are proposing.

If done this way then:

1.  You could issue grant or revoke on GUC settings, allowing some users
but not others to set things like work_mem for their queries
2.  You could specify allowed types in custom settings.
3.  In a subsequent stage you might be able to SELECT .... INTO
setting_name FROM ....;  allowing access to setting writes based on queries.



> regards
>
> Pavel
>
>
>


-- 
Best Regards,
Chris Travers
Database Administrator

Tel: +49 162 9037 210 | Skype: einhverfr | www.adjust.com
Saarbrücker Straße 37a, 10405 Berlin

  10. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2017-10-28T14:56:49Z

    Hi

2017-10-28 16:24 GMT+02:00 Chris Travers <chris.travers@adjust.com>:

>
>
> On Thu, Oct 26, 2017 at 9:21 AM, Pavel Stehule <pavel.stehule@gmail.com>
> wrote:
>
>> Hi,
>>
>> I propose a  new database object - a variable. The variable is persistent
>> object, that holds unshared session based not transactional in memory value
>> of any type. Like variables in any other languages. The persistence is
>> required for possibility to do static checks, but can be limited to session
>> - the variables can be temporal.
>>
>> My proposal is related to session variables from Sybase, MSSQL or MySQL
>> (based on prefix usage @ or @@), or package variables from Oracle (access
>> is controlled by scope), or schema variables from DB2. Any design is coming
>> from different sources, traditions and has some advantages or
>> disadvantages. The base of my proposal is usage schema variables as session
>> variables for stored procedures. It should to help to people who try to
>> port complex projects to PostgreSQL from other databases.
>>
>> The Sybase  (T-SQL) design is good for interactive work, but it is weak
>> for usage in stored procedures - the static check is not possible. Is not
>> possible to set some access rights on variables.
>>
>> The ADA design (used on Oracle) based on scope is great, but our
>> environment is not nested. And we should to support other PL than PLpgSQL
>> more strongly.
>>
>> There is not too much other possibilities - the variable that should be
>> accessed from different PL, different procedures (in time) should to live
>> somewhere over PL, and there is the schema only.
>>
>> The variable can be created by CREATE statement:
>>
>> CREATE VARIABLE public.myvar AS integer;
>> CREATE VARIABLE myschema.myvar AS mytype;
>>
>> CREATE [TEMP] VARIABLE [IF NOT EXISTS] name AS type
>>   [ DEFAULT expression ] [[NOT] NULL]
>>   [ ON TRANSACTION END { RESET | DROP } ]
>>   [ { VOLATILE | STABLE } ];
>>
>> It is dropped by command DROP VARIABLE  [ IF EXISTS] varname.
>>
>> The access rights is controlled by usual access rights - by commands
>> GRANT/REVOKE. The possible rights are: READ, WRITE
>>
>> The variables can be modified by SQL command SET (this is taken from
>> standard, and it natural)
>>
>> SET varname = expression;
>>
>> Unfortunately we use the SET command for different purpose. But I am
>> thinking so we can solve it with few tricks. The first is moving our GUC to
>> pg_catalog schema. We can control the strictness of SET command. In one
>> variant, we can detect custom GUC and allow it, in another we can disallow
>> a custom GUC and allow only schema variables. A new command LET can be
>> alternative.
>>
>> The variables should be used in queries implicitly (without JOIN)
>>
>> SELECT varname;
>>
>> The SEARCH_PATH is used, when varname is located. The variables can be
>> used everywhere where query parameters are allowed.
>>
>> I hope so this proposal is good enough and simple.
>>
>> Comments, notes?
>>
>
>
> I have a question on this.  Since one can issue set commands on arbitrary
> settings (and later ALTER database/role/system on settings you have created
> in the current session) I am wondering how much overlap there is between a
> sort of extended GUC with custom settings and variables.
>
> Maybe it would be simpler to treat variables and GUC settings to be
> similar and see what can be done to extend GUC in this way?
>
> I mean if instead we allowed restricting SET to known settings then we
> could have a CREATE SETTING command which would behave like this and then
> use SET the same way across both.
>
> In essence I am wondering if this really needs to be as separate from GUC
> as you are proposing.
>
> If done this way then:
>
> 1.  You could issue grant or revoke on GUC settings, allowing some users
> but not others to set things like work_mem for their queries
> 2.  You could specify allowed types in custom settings.
> 3.  In a subsequent stage you might be able to SELECT .... INTO
> setting_name FROM ....;  allowing access to setting writes based on queries.
>
>
The creating database objects and necessary infrastructure is the most
simple task of this project. I'll be more happy if there are zero
intersection because variables and GUC are designed for different purposes.
But due SET keyword the intersection there is.

When I thinking about it, I have only one, but important reason, why I
prefer design new type of database object -the GUC are stack based with
different default granularity - global, database, user, session, function.
This can be unwanted behave for variables - it can be source of hard to
detected bugs. I afraid so this behave can be too messy for usage as
variables.

@1 I have not clean opinion about it - not sure if rights are good enough -
probably some user limits can be more practical - but can be hard to choose
result when some user limits and GUC will be against
@2 With variables typed custom GUC are not necessary
@3 Why you need it? It is possible with set_config function now.

Regards

Pavel




>
>
>> regards
>>
>> Pavel
>>
>>
>>
>
>
> --
> Best Regards,
> Chris Travers
> Database Administrator
>
> Tel: +49 162 9037 210 <+49%20162%209037210> | Skype: einhverfr |
> www.adjust.com
> Saarbrücker Straße 37a, 10405 Berlin
> <https://maps.google.com/?q=Saarbr%C3%BCcker+Stra%C3%9Fe+37a,+10405+Berlin&entry=gmail&source=g>
>
>

  11. Re: proposal: schema variables

    Chris Travers <chris.travers@adjust.com> — 2017-10-29T08:51:55Z

    On Sat, Oct 28, 2017 at 4:56 PM, Pavel Stehule <pavel.stehule@gmail.com>
wrote:

>
>>
> The creating database objects and necessary infrastructure is the most
> simple task of this project. I'll be more happy if there are zero
> intersection because variables and GUC are designed for different purposes.
> But due SET keyword the intersection there is.
>
> When I thinking about it, I have only one, but important reason, why I
> prefer design new type of database object -the GUC are stack based with
> different default granularity - global, database, user, session, function.
> This can be unwanted behave for variables - it can be source of hard to
> detected bugs. I afraid so this behave can be too messy for usage as
> variables.
>
> @1 I have not clean opinion about it - not sure if rights are good enough
> - probably some user limits can be more practical - but can be hard to
> choose result when some user limits and GUC will be against
>

I was mostly thinking that users can probably set things like work_mem and
possibly this might be a problem.


> @2 With variables typed custom GUC are not necessary
>

I don't know about that.  For example with the geoip2lookup extension it is
nice that you could set the preferred language for translation on a per
user basis or the mmdb path on a per-db basis.


> @3 Why you need it? It is possible with set_config function now.
>

Yeah you could do it safely with set_config and a CTE, but suppose I have:

with a (Id, value) as (values (1::Int, 'foo'), (2, 'bar'), (3, 'baz'))
SELECT set_config('custom_val', value) from a where id = 2;

What is the result out of this?  I would *expect* that this would probably
run set_config 3 times and filter the output.


>
> Regards
>
> Pavel
>
>
>
>
>>
>>
>>> regards
>>>
>>> Pavel
>>>
>>>
>>>
>>
>>
>> --
>> Best Regards,
>> Chris Travers
>> Database Administrator
>>
>> Tel: +49 162 9037 210 <+49%20162%209037210> | Skype: einhverfr |
>> www.adjust.com
>> Saarbrücker Straße 37a, 10405 Berlin
>> <https://maps.google.com/?q=Saarbr%C3%BCcker+Stra%C3%9Fe+37a,+10405+Berlin&entry=gmail&source=g>
>>
>>
>


-- 
Best Regards,
Chris Travers
Database Administrator

Tel: +49 162 9037 210 | Skype: einhverfr | www.adjust.com
Saarbrücker Straße 37a, 10405 Berlin

  12. Re: proposal: schema variables

    Hannu Krosing <hannu.krosing@2ndquadrant.com> — 2017-10-29T10:47:58Z

    but you can always do

with a (id, value) as (
  values (1, 'foo'), (2, 'bar'), (3, 'baz')
)
select set_config('custom.value',(select value from a where id = 2),true);

if you are worried about the evaluation order

On 29 October 2017 at 09:51, Chris Travers <chris.travers@adjust.com> wrote:

>
>
> On Sat, Oct 28, 2017 at 4:56 PM, Pavel Stehule <pavel.stehule@gmail.com>
> wrote:
>
>>
>>>
>> The creating database objects and necessary infrastructure is the most
>> simple task of this project. I'll be more happy if there are zero
>> intersection because variables and GUC are designed for different purposes.
>> But due SET keyword the intersection there is.
>>
>> When I thinking about it, I have only one, but important reason, why I
>> prefer design new type of database object -the GUC are stack based with
>> different default granularity - global, database, user, session, function.
>> This can be unwanted behave for variables - it can be source of hard to
>> detected bugs. I afraid so this behave can be too messy for usage as
>> variables.
>>
>> @1 I have not clean opinion about it - not sure if rights are good enough
>> - probably some user limits can be more practical - but can be hard to
>> choose result when some user limits and GUC will be against
>>
>
> I was mostly thinking that users can probably set things like work_mem and
> possibly this might be a problem.
>
>
>> @2 With variables typed custom GUC are not necessary
>>
>
> I don't know about that.  For example with the geoip2lookup extension it
> is nice that you could set the preferred language for translation on a per
> user basis or the mmdb path on a per-db basis.
>
>
>> @3 Why you need it? It is possible with set_config function now.
>>
>
> Yeah you could do it safely with set_config and a CTE, but suppose I have:
>
> with a (Id, value) as (values (1::Int, 'foo'), (2, 'bar'), (3, 'baz'))
> SELECT set_config('custom_val', value) from a where id = 2;
>
> What is the result out of this?  I would *expect* that this would probably
> run set_config 3 times and filter the output.
>
>
>>
>> Regards
>>
>> Pavel
>>
>>
>>
>>
>>>
>>>
>>>> regards
>>>>
>>>> Pavel
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Best Regards,
>>> Chris Travers
>>> Database Administrator
>>>
>>> Tel: +49 162 9037 210 <+49%20162%209037210> | Skype: einhverfr |
>>> www.adjust.com
>>> Saarbrücker Straße 37a, 10405 Berlin
>>> <https://maps.google.com/?q=Saarbr%C3%BCcker+Stra%C3%9Fe+37a,+10405+Berlin&entry=gmail&source=g>
>>>
>>>
>>
>
>
> --
> Best Regards,
> Chris Travers
> Database Administrator
>
> Tel: +49 162 9037 210 <+49%20162%209037210> | Skype: einhverfr |
> www.adjust.com
> Saarbrücker Straße 37a, 10405 Berlin
> <https://maps.google.com/?q=Saarbr%C3%BCcker+Stra%C3%9Fe+37a,+10405+Berlin&entry=gmail&source=g>
>
>

  13. Re: proposal: schema variables

    Serge Rielau <serge@rielau.com> — 2017-10-30T21:42:40Z

    Pavel,

I wouldn't put in the DROP option.
Or at least not in that form of syntax.

By convention CREATE persists DDL and makes object definitions visible
across sessions.
DECLARE defines session private objects which cannot collide with other
sessions.

If you want variables with a short lifetime that get dropped at the end of
the transaction that by definition would imply a session private object. So
it ought to be DECLARE'd.

As far as I can see PG has been following this practice so far.

Cheers
Serge Rielau
Salesforce.com



--
Sent from: http://www.postgresql-archive.org/PostgreSQL-hackers-f1928748.html

  14. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2017-10-31T20:33:35Z

    Hi

2017-10-30 22:42 GMT+01:00 srielau <serge@rielau.com>:

> Pavel,
>
> I wouldn't put in the DROP option.
> Or at least not in that form of syntax.
>
> By convention CREATE persists DDL and makes object definitions visible
> across sessions.
> DECLARE defines session private objects which cannot collide with other
> sessions.
>
> If you want variables with a short lifetime that get dropped at the end of
> the transaction that by definition would imply a session private object. So
> it ought to be DECLARE'd.
>
> As far as I can see PG has been following this practice so far.
>

I am thinking so there is little bit overlap between DECLARE and CREATE
TEMP VARIABLE command. With DECLARE command, you are usually has not any
control when variable will be destroyed. For CREATE TEMP xxxx is DROP IF
EXISTS, but it should not be used.

It should be very similar to our current temporary tables, that are created
in session related temp schema.

I can imagine, so DECLARE command will be introduced as short cut for
CREATE TEMP VARIABLE, but in this moment I would not to open this topic. I
afraid of bikeshedding and I hope so CREATE TEMP VAR is anough.

Regards

Pavel


> Cheers
> Serge Rielau
> Salesforce.com
>
>
>
> --
> Sent from: http://www.postgresql-archive.org/PostgreSQL-hackers-
> f1928748.html
>
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers
>

  15. Re: proposal: schema variables

    Serge Rielau <serge@rielau.com> — 2017-10-31T21:08:18Z

    Pavel, I can imagine, so DECLARE command will be introduced as short cut 
for CREATE TEMP VARIABLE, but in this moment I would not to open this 
topic. I afraid of bikeshedding and I hope so CREATE TEMP VAR is anough. 
Language is important because language stays. You choice of syntax will 
outlive your code and possibly yourself.
My 2 cents Serge

  16. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2017-10-31T21:10:43Z

    2017-10-31 22:08 GMT+01:00 Serge Rielau <serge@rielau.com>:

> Pavel,
>
> I can imagine, so DECLARE command will be introduced as short cut for
> CREATE TEMP VARIABLE, but in this moment I would not to open this topic. I
> afraid of bikeshedding and I hope so CREATE TEMP VAR is anough.
>
> Language is important because language stays.
> You choice of syntax will outlive your code and possibly yourself.
>

sure. But in this moment I don't see difference between DECLARE VARIABLE
and CREATE TEMP VARIABLE different than "TEMP" keyword.

Regards

Pavel


> My 2 cents
> Serge
>

  17. Re: proposal: schema variables

    Serge Rielau <serge@rielau.com> — 2017-10-31T21:28:37Z

    Pavel,

There is no
DECLARE TEMP CURSOR
or
DECLARE TEMP variable in PLpgSQL
and
CREATE TEMP TABLE has a different meaning from what I understand you
envision for variables.

But maybe I'm mistaken. Your original post did not describe the entire
syntax:
CREATE [TEMP] VARIABLE [IF NOT EXISTS] name AS type 
  [ DEFAULT expression ] [[NOT] NULL]
  [ ON TRANSACTION END { RESET | DROP } ]
  [ { VOLATILE | STABLE } ];

Especially the TEMP is not spelled out and how its presence affects or
doesn't ON TRANSACTION END.
So may be if you elaborate I understand where you are coming from.

 



--
Sent from: http://www.postgresql-archive.org/PostgreSQL-hackers-f1928748.html

  18. Re: proposal: schema variables

    Gilles Darold <gilles.darold@dalibo.com> — 2017-10-31T22:36:53Z

    Le 31/10/2017 à 22:28, srielau a écrit :
> Pavel,
>
> There is no
> DECLARE TEMP CURSOR
> or
> DECLARE TEMP variable in PLpgSQL
> and
> CREATE TEMP TABLE has a different meaning from what I understand you
> envision for variables.
>
> But maybe I'm mistaken. Your original post did not describe the entire
> syntax:
> CREATE [TEMP] VARIABLE [IF NOT EXISTS] name AS type 
>   [ DEFAULT expression ] [[NOT] NULL]
>   [ ON TRANSACTION END { RESET | DROP } ]
>   [ { VOLATILE | STABLE } ];
>
> Especially the TEMP is not spelled out and how its presence affects or
> doesn't ON TRANSACTION END.
> So may be if you elaborate I understand where you are coming from.

I think that the TEMP keyword can be removed. If I understand well the
default scope for variable is the session, every transaction in a
session will see the same value. For the transaction level, probably the
reason of the TEMP keyword, I think the [ ON TRANSACTION END { RESET |
DROP } ] will allow to restrict the scope to this transaction level
without needing the TEMP keyword. When a variable is created in a
transaction, it is temporary if "ON TRANSACTION END DROP" is used
otherwise it will persist after the transaction end. I guess that this
is the same as using TEMP keyword?


-- 
Gilles Darold
Consultant PostgreSQL
http://dalibo.com - http://dalibo.org

  19. Re: proposal: schema variables

    Gilles Darold <gilles.darold@dalibo.com> — 2017-10-31T23:02:26Z

    Le 31/10/2017 à 23:36, Gilles Darold a écrit :
> Le 31/10/2017 à 22:28, srielau a écrit :
>> Pavel,
>>
>> There is no
>> DECLARE TEMP CURSOR
>> or
>> DECLARE TEMP variable in PLpgSQL
>> and
>> CREATE TEMP TABLE has a different meaning from what I understand you
>> envision for variables.
>>
>> But maybe I'm mistaken. Your original post did not describe the entire
>> syntax:
>> CREATE [TEMP] VARIABLE [IF NOT EXISTS] name AS type 
>>   [ DEFAULT expression ] [[NOT] NULL]
>>   [ ON TRANSACTION END { RESET | DROP } ]
>>   [ { VOLATILE | STABLE } ];
>>
>> Especially the TEMP is not spelled out and how its presence affects or
>> doesn't ON TRANSACTION END.
>> So may be if you elaborate I understand where you are coming from.
> I think that the TEMP keyword can be removed. If I understand well the
> default scope for variable is the session, every transaction in a
> session will see the same value. For the transaction level, probably the
> reason of the TEMP keyword, I think the [ ON TRANSACTION END { RESET |
> DROP } ] will allow to restrict the scope to this transaction level
> without needing the TEMP keyword. When a variable is created in a
> transaction, it is temporary if "ON TRANSACTION END DROP" is used
> otherwise it will persist after the transaction end. I guess that this
> is the same as using TEMP keyword?

I forgot to say that in the last case the DECLARE statement can be used
so I don't see the reason of this kind of "temporary" variables.

Maybe the variable object like used in DB2 and defined in document :
https://www.ibm.com/support/knowledgecenter/en/SSEPEK_11.0.0/sqlref/src/tpc/db2z_sql_createvariable.html
could be enough to cover our needs.


-- 
Gilles Darold
Consultant PostgreSQL
http://dalibo.com - http://dalibo.org

  20. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2017-11-01T04:15:58Z

    2017-10-31 22:28 GMT+01:00 srielau <serge@rielau.com>:

> Pavel,
>
> There is no
> DECLARE TEMP CURSOR
> or
> DECLARE TEMP variable in PLpgSQL
> and
>

sure .. DECLARE TEMP has no sense, I talked about similarity DECLARE and
CREATE TEMP


CREATE TEMP TABLE has a different meaning from what I understand you
> envision for variables.
>
> But maybe I'm mistaken. Your original post did not describe the entire
> syntax:
> CREATE [TEMP] VARIABLE [IF NOT EXISTS] name AS type
>   [ DEFAULT expression ] [[NOT] NULL]
>   [ ON TRANSACTION END { RESET | DROP } ]
>   [ { VOLATILE | STABLE } ];
>
> Especially the TEMP is not spelled out and how its presence affects or
> doesn't ON TRANSACTION END.
> So may be if you elaborate I understand where you are coming from.
>

TEMP has same functionality (and implementation) like our temp tables - so
at session end the temp variables are destroyed, but it can be assigned to
transaction.



>
>
>
>
> --
> Sent from: http://www.postgresql-archive.org/PostgreSQL-hackers-
> f1928748.html
>
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers
>

  21. Re: proposal: schema variables

    Serge Rielau <serge@rielau.com> — 2017-11-01T05:07:59Z

    " Although the syntax of CREATE TEMPORARY TABLE resembles that of the SQL standard, the effect is not the same. In the standard, temporary tables are defined just once and automatically exist (starting with empty contents) in every session that needs them. PostgreSQL instead requires each session to issue its own CREATE TEMPORARY TABLE command for each temporary table to be used. This allows different sessions to use the same temporary table name for different purposes, whereas the standard's approach constrains all instances of a given temporary table name to have the same table structure.”
Yeah, that’s a DECLAREd table in my book. No wonder we didn’t link up.
Cheers Serge

  22. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2017-11-01T05:56:03Z

    2017-11-01 6:07 GMT+01:00 Serge Rielau <serge@rielau.com>:

> "Although the syntax of CREATE TEMPORARY TABLE resembles that of the SQL
> standard, the effect is not the same. In the standard, temporary tables are
> defined just once and automatically exist (starting with empty contents) in
> every session that needs them. PostgreSQL instead requires each session
> to issue its own CREATE TEMPORARY TABLE command for each temporary table
> to be used. This allows different sessions to use the same temporary table
> name for different purposes, whereas the standard's approach constrains all
> instances of a given temporary table name to have the same table structure.”
> Yeah, that’s a DECLAREd table in my book. No wonder we didn’t link up.
>

This is known discussion about local / global temp tables in PostgresSQL.
And ToDo point: implementation of global temp tables in Postgres.

This temporary behave is marginal part of proposal - so I can to remove it
from proposal - and later open discussion about CREATE TEMPORARY VARIABLE
versus DECLARE VARIABLE

Regards

Pavel

Serge

>

  23. Re: proposal: schema variables

    Mark Dilger <hornschnorter@gmail.com> — 2017-11-01T18:03:28Z

    > Comments, notes?

How would variables behave on transaction rollback?

CREATE TEMP VARIABLE myvar;
SET myvar := 1;
BEGIN;
SET myvar := 2;
COMMIT;
BEGIN;
SET myvar := 3;
ROLLBACK;
SELECT myvar;

How would variables behave when modified in a procedure
that aborts rather than returning cleanly?


mark

  24. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2017-11-01T19:19:48Z

    2017-11-01 19:03 GMT+01:00 Mark Dilger <hornschnorter@gmail.com>:

>
> > Comments, notes?
>
> How would variables behave on transaction rollback?
>
> CREATE TEMP VARIABLE myvar;
> SET myvar := 1;
> BEGIN;
> SET myvar := 2;
> COMMIT;
> BEGIN;
> SET myvar := 3;
> ROLLBACK;
> SELECT myvar;
>
> How would variables behave when modified in a procedure
> that aborts rather than returning cleanly?
>
>
The result is 3

When you create variable like you did, then there are not any relation
between variable content and transactions. Almost every where session -
package - schema variables are untransactional. It can be changed, but with
negative impact on performance - so I propose relative simply solution -
reset to default on rollback, when variables was changed in transaction -
but it is not default behave.

Variables are variables like you know from PlpgSQL. But the holder is not
the plpgsql function. The holder is a schema in this case. The variable
(meta) is permanent. The content of variable is session based
untransactional.

Regards

Pavel


> mark
>

  25. Re: proposal: schema variables

    Gilles Darold <gilles.darold@dalibo.com> — 2017-11-01T22:13:47Z

    Le 01/11/2017 à 05:15, Pavel Stehule a écrit :
>
>
> 2017-10-31 22:28 GMT+01:00 srielau <serge@rielau.com
> <mailto:serge@rielau.com>>:
>
>     Pavel,
>
>     There is no
>     DECLARE TEMP CURSOR
>     or
>     DECLARE TEMP variable in PLpgSQL
>     and
>
>
> sure .. DECLARE TEMP has no sense, I talked about similarity DECLARE
> and CREATE TEMP
>
>
>     CREATE TEMP TABLE has a different meaning from what I understand you
>     envision for variables.
>
>     But maybe I'm mistaken. Your original post did not describe the entire
>     syntax:
>     CREATE [TEMP] VARIABLE [IF NOT EXISTS] name AS type
>       [ DEFAULT expression ] [[NOT] NULL]
>       [ ON TRANSACTION END { RESET | DROP } ]
>       [ { VOLATILE | STABLE } ];
>
>     Especially the TEMP is not spelled out and how its presence affects or
>     doesn't ON TRANSACTION END.
>     So may be if you elaborate I understand where you are coming from.
>
>
> TEMP has same functionality (and implementation) like our temp tables
> - so at session end the temp variables are destroyed, but it can be
> assigned to transaction.

Oh ok, I understand thanks for the precision.

-- 
Gilles Darold
Consultant PostgreSQL
http://dalibo.com - http://dalibo.org

  26. Re: proposal: schema variables

    Robert Haas <robertmhaas@gmail.com> — 2017-11-02T12:35:54Z

    On Thu, Oct 26, 2017 at 12:51 PM, Pavel Stehule <pavel.stehule@gmail.com> wrote:
> The variables can be modified by SQL command SET (this is taken from
> standard, and it natural)
>
> SET varname = expression;

Overloading SET to handle both variables and GUCs seems likely to
create problems, possibly including security problems.  For example,
maybe a security-definer function could leave behind variables to
trick the calling code into failing to set GUCs that it intended to
set.  Or maybe creating a variable at the wrong time will just break
things randomly.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  27. Re: proposal: schema variables

    Craig Ringer <craig@2ndquadrant.com> — 2017-11-02T15:07:55Z

    On 26 October 2017 at 15:21, Pavel Stehule <pavel.stehule@gmail.com> wrote:
> Hi,
>
> I propose a  new database object - a variable.

Didn't we have a pretty long discussion about this already in

Yeah.

https://www.postgresql.org/message-id/flat/CAMsr%2BYF0G8_FehQyFS8gSfnEer9OPsMOvpfniDJOVGQzJzHzsw%40mail.gmail.com#CAMsr+YF0G8_FehQyFS8gSfnEer9OPsMOvpfniDJOVGQzJzHzsw@mail.gmail.com

It'd be nice if you summarised any outcomes from that and addressed
it, rather than taking this as a new topic.

-- 
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

  28. Re: proposal: schema variables

    Nico Williams <nico@cryptonector.com> — 2017-11-02T15:35:07Z

    On Thu, Nov 02, 2017 at 06:05:54PM +0530, Robert Haas wrote:
> On Thu, Oct 26, 2017 at 12:51 PM, Pavel Stehule <pavel.stehule@gmail.com> wrote:
> > The variables can be modified by SQL command SET (this is taken from
> > standard, and it natural)
> >
> > SET varname = expression;
> 
> Overloading SET to handle both variables and GUCs seems likely to
> create problems, possibly including security problems.  For example,
> maybe a security-definer function could leave behind variables to
> trick the calling code into failing to set GUCs that it intended to
> set.  Or maybe creating a variable at the wrong time will just break
> things randomly.

That's already true of GUCs, since there are no access controls on
set_config()/current_setting().

Presumably "schema variables" would really just be GUC-like and not at
all like lexically scoped variables.  And also subject to access
controls, thus an overall improvement on set_config()/current_setting().

With access controls, GUCs could become schema variables, and settings
from postgresql.conf could move into the database itself (which I think
would be nice).

Nico
--

  29. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2017-11-02T15:40:36Z

    2017-11-02 16:35 GMT+01:00 Nico Williams <nico@cryptonector.com>:

> On Thu, Nov 02, 2017 at 06:05:54PM +0530, Robert Haas wrote:
> > On Thu, Oct 26, 2017 at 12:51 PM, Pavel Stehule <pavel.stehule@gmail.com>
> wrote:
> > > The variables can be modified by SQL command SET (this is taken from
> > > standard, and it natural)
> > >
> > > SET varname = expression;
> >
> > Overloading SET to handle both variables and GUCs seems likely to
> > create problems, possibly including security problems.  For example,
> > maybe a security-definer function could leave behind variables to
> > trick the calling code into failing to set GUCs that it intended to
> > set.  Or maybe creating a variable at the wrong time will just break
> > things randomly.
>
> That's already true of GUCs, since there are no access controls on
> set_config()/current_setting().
>
> Presumably "schema variables" would really just be GUC-like and not at
> all like lexically scoped variables.  And also subject to access
> controls, thus an overall improvement on set_config()/current_setting().
>
> With access controls, GUCs could become schema variables, and settings
> from postgresql.conf could move into the database itself (which I think
> would be nice).
>

I am sorry, but I don't plan it. the behave of GUC is too different than
behave of variables. But I am planning so system GUC can be "moved" to
pg_catalog to be possibility to specify any object exactly.

Regards

Pavel

>
> Nico
> --
>

  30. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2017-11-02T15:42:20Z

    2017-11-02 16:07 GMT+01:00 Craig Ringer <craig@2ndquadrant.com>:

> On 26 October 2017 at 15:21, Pavel Stehule <pavel.stehule@gmail.com>
> wrote:
> > Hi,
> >
> > I propose a  new database object - a variable.
>
> Didn't we have a pretty long discussion about this already in
>
> Yeah.
>
> https://www.postgresql.org/message-id/flat/CAMsr%2BYF0G8_
> FehQyFS8gSfnEer9OPsMOvpfniDJOVGQzJzHzsw%40mail.gmail.com#CAMsr+YF0G8_
> FehQyFS8gSfnEer9OPsMOvpfniDJOVGQzJzHzsw@mail.gmail.com
>
> It'd be nice if you summarised any outcomes from that and addressed
> it, rather than taking this as a new topic.
>

I am sorry. This thread follow mentioned and I started with small
recapitulation.

Regards

Pavel


> --
>  Craig Ringer                   http://www.2ndQuadrant.com/
>  PostgreSQL Development, 24x7 Support, Training & Services
>

  31. Re: proposal: schema variables

    Tom Lane <tgl@sss.pgh.pa.us> — 2017-11-02T15:48:44Z

    Nico Williams <nico@cryptonector.com> writes:
> With access controls, GUCs could become schema variables, and settings
> from postgresql.conf could move into the database itself (which I think
> would be nice).

People re-propose some variant of that every so often, but it never works,
because it ignores the fact that some of the GUCs' values are needed
before you can access system catalogs at all, or in places where relying
on system catalog access would be a bad idea.

Sure, we could have two completely different configuration mechanisms
so that some of the variables could be "inside the database", but that
doesn't seem like a net improvement to me.  The point of the Grand Unified
Configuration mechanism was to be unified, after all.

I'm on board with having a totally different mechanism for session
variables.  The fact that people have been abusing GUC to store
user-defined variables doesn't make it a good way to do that.

			regards, tom lane

  32. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2017-11-02T15:49:20Z

    2017-11-02 13:35 GMT+01:00 Robert Haas <robertmhaas@gmail.com>:

> On Thu, Oct 26, 2017 at 12:51 PM, Pavel Stehule <pavel.stehule@gmail.com>
> wrote:
> > The variables can be modified by SQL command SET (this is taken from
> > standard, and it natural)
> >
> > SET varname = expression;
>
> Overloading SET to handle both variables and GUCs seems likely to
> create problems, possibly including security problems.  For example,
> maybe a security-definer function could leave behind variables to
> trick the calling code into failing to set GUCs that it intended to
> set.  Or maybe creating a variable at the wrong time will just break
> things randomly.
>

The syntax CREATE OR REPLACE FUNCTION xxx $$ ... $$ SET GUC=, ... is always
related only to GUC. So there should not be any security risk.

It is another reason why GUC and variables should be separated.

I know so there is risk of possibility of collision. There are two
possibilities

a) use different keyword - but it is out of SQL/PSM and out of another
databases.

b) detect possible collision and raise error when assignment is ambiguous.
I am thinking about similar solution used in plpgsql, where is a
possibility of collision between SQL identifier and plpgsql variable.

Regards

Pavel




>
> --
> Robert Haas
> EnterpriseDB: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company
>

  33. Re: proposal: schema variables

    Robert Haas <robertmhaas@gmail.com> — 2017-11-02T17:21:54Z

    On Thu, Nov 2, 2017 at 9:05 PM, Nico Williams <nico@cryptonector.com> wrote:
>> Overloading SET to handle both variables and GUCs seems likely to
>> create problems, possibly including security problems.  For example,
>> maybe a security-definer function could leave behind variables to
>> trick the calling code into failing to set GUCs that it intended to
>> set.  Or maybe creating a variable at the wrong time will just break
>> things randomly.
>
> That's already true of GUCs, since there are no access controls on
> set_config()/current_setting().

No, it isn't.  Right now, SET always refers to a GUC, never a
variable, so there's no possibility of getting confused about whether
it's intending to change a GUC or an eponymous variable.  Once you
make SET able to change either one of two different kinds of objects,
then that possibility does exist.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  34. Re: proposal: schema variables

    Nico Williams <nico@cryptonector.com> — 2017-11-02T18:52:39Z

    On Thu, Nov 02, 2017 at 11:48:44AM -0400, Tom Lane wrote:
> Nico Williams <nico@cryptonector.com> writes:
> > With access controls, GUCs could become schema variables, and settings
> > from postgresql.conf could move into the database itself (which I think
> > would be nice).
> 
> People re-propose some variant of that every so often, but it never works,
> because it ignores the fact that some of the GUCs' values are needed
> before you can access system catalogs at all, or in places where relying
> on system catalog access would be a bad idea.

ISTM that it should be possible to break the chicken-egg issue by having
the config variables stored in such a way that knowing only the pgdata
directory path should suffice to find them.  That's effectively the case
already in that postgresql.conf is found... there.

One could do probably this as a PoC entirely as a SQL-coded VIEW that
reads and writes (via the adminpack module's pg_catalog.pg_file_write())
postgresql.conf (without preserving comments, or with some rules
regarding comments so that they are effectively attached to params).

Nico
--

  35. Re: proposal: schema variables

    Chris Travers <chris.travers@adjust.com> — 2017-11-03T12:58:02Z

    Some thoughts on this.

On Thu, Nov 2, 2017 at 4:48 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

> Nico Williams <nico@cryptonector.com> writes:
> > With access controls, GUCs could become schema variables, and settings
> > from postgresql.conf could move into the database itself (which I think
> > would be nice).
>
> People re-propose some variant of that every so often, but it never works,
> because it ignores the fact that some of the GUCs' values are needed
> before you can access system catalogs at all, or in places where relying
> on system catalog access would be a bad idea.
>

I think the basic point one should get here is that no matter the
unification, you still have some things in the db and some things out.

I would rather look at how the GUC could be improved on a functional/use
case level before we look at the question of a technical solution.

 One major use case today would be restricting how high various users can
set something like work_mem or the like.  As it stands, there isn't really
a way to control this with any granularity.  So some of the proposals
regarding granting access to a session variable would be very handy in
granting access to a GUC variable.

>
> Sure, we could have two completely different configuration mechanisms
> so that some of the variables could be "inside the database", but that
> doesn't seem like a net improvement to me.  The point of the Grand Unified
> Configuration mechanism was to be unified, after all.
>

+1

>
> I'm on board with having a totally different mechanism for session
> variables.  The fact that people have been abusing GUC to store
> user-defined variables doesn't make it a good way to do that.
>

What about having a more clunky syntax as:

SET VARIABLE foo='bar';

Perhaps one can have a short form of:

SET VAR foo = 'bar';

vs

SET foo = 'bar'; -- GUC



>
>                         regards, tom lane
>
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers
>



-- 
Best Regards,
Chris Travers
Database Administrator

Tel: +49 162 9037 210 | Skype: einhverfr | www.adjust.com
Saarbrücker Straße 37a, 10405 Berlin

  36. Re: proposal: schema variables

    Pavel Golub <pavel@microolap.com> — 2017-11-13T12:15:00Z

    Hello, Pavel.

You wrote:

PS> Hi,

PS> I propose a  new database object - a variable. The variable is
PS> persistent object, that holds unshared session based not
PS> transactional in memory value of any type. Like variables in any
PS> other languages. The persistence is required for possibility to do
PS> static checks, but can be limited to session - the variables can be temporal.

Great idea.

PS> My proposal is related to session variables from Sybase, MSSQL or
PS> MySQL (based on prefix usage @ or @@), or package variables from
PS> Oracle (access is controlled by scope), or schema variables from
PS> DB2. Any design is coming from different sources, traditions and
PS> has some advantages or disadvantages. The base of my proposal is
PS> usage schema variables as session variables for stored procedures.
PS> It should to help to people who try to port complex projects to PostgreSQL from other databases.

PS> The Sybase  (T-SQL) design is good for interactive work, but it
PS> is weak for usage in stored procedures - the static check is not
PS> possible. Is not possible to set some access rights on variables.

PS> The ADA design (used on Oracle) based on scope is great, but our
PS> environment is not nested. And we should to support other PL than PLpgSQL more strongly.

PS> There is not too much other possibilities - the variable that
PS> should be accessed from different PL, different procedures (in
PS> time) should to live somewhere over PL, and there is the schema only.

PS> The variable can be created by CREATE statement:

PS> CREATE VARIABLE public.myvar AS integer;
PS> CREATE VARIABLE myschema.myvar AS mytype;

PS> CREATE [TEMP] VARIABLE [IF NOT EXISTS] name AS type 
PS>   [ DEFAULT expression ] [[NOT] NULL]
PS>   [ ON TRANSACTION END { RESET | DROP } ]
PS>   [ { VOLATILE | STABLE } ];


PS> It is dropped by command DROP VARIABLE  [ IF EXISTS] varname.

PS> The access rights is controlled by usual access rights - by
PS> commands GRANT/REVOKE. The possible rights are: READ, WRITE

PS> The variables can be modified by SQL command SET (this is taken from standard, and it natural)

PS> SET varname = expression;

I propose LET keyword for this to distinguish GUC from variables, e.g.

LET varname = expression;

PS> Unfortunately we use the SET command for different purpose. But I
PS> am thinking so we can solve it with few tricks. The first is
PS> moving our GUC to pg_catalog schema. We can control the strictness
PS> of SET command. In one variant, we can detect custom GUC and allow
PS> it, in another we can disallow a custom GUC and allow only schema
PS> variables. A new command LET can be alternative.



PS> The variables should be used in queries implicitly (without JOIN)


PS> SELECT varname;


PS> The SEARCH_PATH is used, when varname is located. The variables
PS> can be used everywhere where query parameters are allowed. 



PS> I hope so this proposal is good enough and simple.


PS> Comments, notes?


PS> regards


PS> Pavel






-- 
With best wishes,
 Pavel                          mailto:pavel@gf.microolap.com

  37. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2017-11-13T12:30:58Z

    Hi

2017-11-13 13:15 GMT+01:00 Pavel Golub <pavel@microolap.com>:

> Hello, Pavel.
>
> You wrote:
>
> PS> Hi,
>
> PS> I propose a  new database object - a variable. The variable is
> PS> persistent object, that holds unshared session based not
> PS> transactional in memory value of any type. Like variables in any
> PS> other languages. The persistence is required for possibility to do
> PS> static checks, but can be limited to session - the variables can be
> temporal.
>
> Great idea.
>
> PS> My proposal is related to session variables from Sybase, MSSQL or
> PS> MySQL (based on prefix usage @ or @@), or package variables from
> PS> Oracle (access is controlled by scope), or schema variables from
> PS> DB2. Any design is coming from different sources, traditions and
> PS> has some advantages or disadvantages. The base of my proposal is
> PS> usage schema variables as session variables for stored procedures.
> PS> It should to help to people who try to port complex projects to
> PostgreSQL from other databases.
>
> PS> The Sybase  (T-SQL) design is good for interactive work, but it
> PS> is weak for usage in stored procedures - the static check is not
> PS> possible. Is not possible to set some access rights on variables.
>
> PS> The ADA design (used on Oracle) based on scope is great, but our
> PS> environment is not nested. And we should to support other PL than
> PLpgSQL more strongly.
>
> PS> There is not too much other possibilities - the variable that
> PS> should be accessed from different PL, different procedures (in
> PS> time) should to live somewhere over PL, and there is the schema only.
>
> PS> The variable can be created by CREATE statement:
>
> PS> CREATE VARIABLE public.myvar AS integer;
> PS> CREATE VARIABLE myschema.myvar AS mytype;
>
> PS> CREATE [TEMP] VARIABLE [IF NOT EXISTS] name AS type
> PS>   [ DEFAULT expression ] [[NOT] NULL]
> PS>   [ ON TRANSACTION END { RESET | DROP } ]
> PS>   [ { VOLATILE | STABLE } ];
>
>
> PS> It is dropped by command DROP VARIABLE  [ IF EXISTS] varname.
>
> PS> The access rights is controlled by usual access rights - by
> PS> commands GRANT/REVOKE. The possible rights are: READ, WRITE
>
> PS> The variables can be modified by SQL command SET (this is taken from
> standard, and it natural)
>
> PS> SET varname = expression;
>
> I propose LET keyword for this to distinguish GUC from variables, e.g.
>
> LET varname = expression;
>

 It is one possible variant. I plan to implement more variants and then
choose one.

Regards

Pavel

>
> PS> Unfortunately we use the SET command for different purpose. But I
> PS> am thinking so we can solve it with few tricks. The first is
> PS> moving our GUC to pg_catalog schema. We can control the strictness
> PS> of SET command. In one variant, we can detect custom GUC and allow
> PS> it, in another we can disallow a custom GUC and allow only schema
> PS> variables. A new command LET can be alternative.
>
>
>
> PS> The variables should be used in queries implicitly (without JOIN)
>
>
> PS> SELECT varname;
>
>
> PS> The SEARCH_PATH is used, when varname is located. The variables
> PS> can be used everywhere where query parameters are allowed.
>
>
>
> PS> I hope so this proposal is good enough and simple.
>
>
> PS> Comments, notes?
>
>
> PS> regards
>
>
> PS> Pavel
>
>
>
>
>
>
> --
> With best wishes,
>  Pavel                          mailto:pavel@gf.microolap.com
>
>

  38. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2018-02-02T22:06:44Z

    Hi

I wrote proof concept of schema variables. The patch is not nice, but the
functionality is almost complete (for scalars only) and can be good enough
for playing with this concept.

I recap a goals (the order is random):

1. feature like PL/SQL package variables (with similar content life cycle)
2. available from any PL used by PostgreSQL, data can be shared between
different PL
3. possibility to store short life data in fast secured storage
4. possibility to pass parameters and results to/from anonymous blocks
5. session variables with possibility to process static code check
6. multiple API available from different environments - SQL commands, SQL
functions, internal functions
7. data are stored in binary form

Example:

CREATE VARIABLE public.foo AS integer;

LET foo = 10 + 20;

DO $$
declare x int = random() * 1000;
BEGIN
  RAISE NOTICE '%', foo;
  LET foo = x + 100;
END;
$$;

SELECT public.foo + 10;
SELECT * FROM data WHERE col = foo;

All implemented features are described by regress tests

Interesting note - it is running without any modification of plpgsql code.

Regards

Pavel

  39. Re: [HACKERS] proposal: schema variables

    David G. Johnston <david.g.johnston@gmail.com> — 2018-02-03T00:48:05Z

     ​I've done a non-compilation documentation review, the diff from the poc
patch and the diff from master are attached.

Comments are inter-twined in the patch in xml comment format; though I
reiterate (some of?) them below.

On Fri, Feb 2, 2018 at 3:06 PM, Pavel Stehule <pavel.stehule@gmail.com>
wrote:

> Hi
>
> I wrote proof concept of schema variables. The patch is not nice, but the
> functionality is almost complete (for scalars only) and can be good enough
> for playing with this concept.
>
> I recap a goals (the order is random):
>
> 1. feature like PL/SQL package variables (with similar content life cycle)
> 2. available from any PL used by PostgreSQL, data can be shared between
> different PL
> 3. possibility to store short life data in fast secured storage
>

​The generic use of the word secure here bothers me.  I'm taking it to be
"protected by grant/revoke"-based privileges; plus session-locality.

4. possibility to pass parameters and results to/from anonymous blocks
> 5. session variables with possibility to process static code check
>

What does "process static code check" means here?​


> 6. multiple API available from different environments - SQL commands, SQL
> functions, internal functions
>

I made the public aspect of this explicit in the CREATE VARIABLE doc
(though as noted below it probably belongs in section II)
​

> 7. data are stored in binary form
>

Thoughts during my review:

There is, for me, a cognitive dissonance between "schema variable" and
"variable value" - I'm partial to the later.  Since we use "setting" for
GUCs the term variable here hopefully wouldn't cause ambiguity...

I've noticed that we don't seem to have or enforce any policy on how to
communicate "SQL standards compatibility" to the user...

We are missing the ability to alter ownership (or at least its
undocumented), and if that brings into existing ALTER VARIABLE we should
probably add ALTER TYPE TO new_type USING (cast) for completeness.

Its left for the reader to presume that because these are schema
"relations" that namespace resolution via search_path works the same as any
other relation.

I think I've answered my own question regarding DISCARD in that "variables"
discards values while if TEMP is in effect all temp variables are dropped.

Examples abound though it doesn't feel like too much: but saying "The usage
is very simple:" before giving the example in the function section seems to
be outside of our general style.  A better preamble than "An example:"
would be nice but the example is so simple I could not think of anything
worth writing.

Its worth considering how both:

https://www.postgresql.org/docs/10/static/ddl.html
and
https://www.postgresql.org/docs/10/static/queries.html

could be updated to incorporate the broad picture of schema variables, with
examples, and leave the reference (SQL and functions) sections mainly
relegated to syntax and reminders.

A moderate number of lines changed are for typos and minor grammar edits.

David J.

  40. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2018-02-03T06:58:33Z

    Hi

2018-02-03 1:48 GMT+01:00 David G. Johnston <david.g.johnston@gmail.com>:

> ​I've done a non-compilation documentation review, the diff from the poc
> patch and the diff from master are attached.
>
> Comments are inter-twined in the patch in xml comment format; though I
> reiterate (some of?) them below.
>
> On Fri, Feb 2, 2018 at 3:06 PM, Pavel Stehule <pavel.stehule@gmail.com>
> wrote:
>
>> Hi
>>
>> I wrote proof concept of schema variables. The patch is not nice, but the
>> functionality is almost complete (for scalars only) and can be good enough
>> for playing with this concept.
>>
>> I recap a goals (the order is random):
>>
>> 1. feature like PL/SQL package variables (with similar content life cycle)
>> 2. available from any PL used by PostgreSQL, data can be shared between
>> different PL
>> 3. possibility to store short life data in fast secured storage
>>
>
> ​The generic use of the word secure here bothers me.  I'm taking it to be
> "protected by grant/revoke"-based privileges; plus session-locality.
>

I have not a problem with any other formulation.


>
> 4. possibility to pass parameters and results to/from anonymous blocks
>> 5. session variables with possibility to process static code check
>>
>
> What does "process static code check" means here?​
>

It mean the possibility to check validity of code without code execution.
You can use plpgsql_check for example.


>
>
>> 6. multiple API available from different environments - SQL commands, SQL
>> functions, internal functions
>>
>
> I made the public aspect of this explicit in the CREATE VARIABLE doc
> (though as noted below it probably belongs in section II)
> ​
>
>> 7. data are stored in binary form
>>
>
> Thoughts during my review:
>
> There is, for me, a cognitive dissonance between "schema variable" and
> "variable value" - I'm partial to the later.  Since we use "setting" for
> GUCs the term variable here hopefully wouldn't cause ambiguity...
>

The "schema" is important in this case. 1) it is a analogy to "package
variable", 2) not necessary, but probably often it will be used together
with PLpgSQL. There are variables too. "Session variables" doesn't well
specify the implementation. The session variables can be GUC, psql client
variables or some custom implementation in Postgres or package variables in
Oracle.


> I've noticed that we don't seem to have or enforce any policy on how to
> communicate "SQL standards compatibility" to the user...
>
> We are missing the ability to alter ownership (or at least its
> undocumented), and if that brings into existing ALTER VARIABLE we should
> probably add ALTER TYPE TO new_type USING (cast) for completeness.
>

good note. I didn't test it. I am not sure, what variants of ALTER should
be supported. Type of variables is interface. Probably we can allow to add
new field, but change type or remove field can break other object. So it
can be prohibited like we doesn't support ALTER on views. ALTERing is
another and pretty complex topic, and I don't think it is necessary to
solve it now. This feature can be valuable without ALTER support, and
nothing block later ALTER VARIABLE implementation.

This design allows lot of interesting features (that can be implemented
step by step)

1. support for default expression
2. support for constraints and maybe triggers
3. reset on transaction end
4. initialization of session start - via default expression or triggers it
can be way how to start code on session start.


>
> Its left for the reader to presume that because these are schema
> "relations" that namespace resolution via search_path works the same as any
> other relation.
>
> I think I've answered my own question regarding DISCARD in that
> "variables" discards values while if TEMP is in effect all temp variables
> are dropped.
>

DISCARD should to remove TEMP variables and should to remove content of all
variables.


>
> Examples abound though it doesn't feel like too much: but saying "The
> usage is very simple:" before giving the example in the function section
> seems to be outside of our general style.  A better preamble than "An
> example:" would be nice but the example is so simple I could not think of
> anything worth writing.
>

This doc is just design frame. I invite any enhancing because this feature
can be difficult for some people, because mix persistent object with
temporal/session content - and term "variable" can be used in relation
algebra in different semantic. It is natural for people with stored
procedures experience - mainly with Oracle, but for any other can be little
bit difficult. I believe so there should be more practical examples -
related to RLS for example.



>
> Its worth considering how both:
>
> https://www.postgresql.org/docs/10/static/ddl.html
> and
> https://www.postgresql.org/docs/10/static/queries.html
>
> could be updated to incorporate the broad picture of schema variables,
> with examples, and leave the reference (SQL and functions) sections mainly
> relegated to syntax and reminders.
>
> A moderate number of lines changed are for typos and minor grammar edits.
>

Thank you very much

Regards

Pavel


> David J.
>
>

  41. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2018-02-07T06:34:44Z

    Hi

updated patch with your changes in documentation and pg_dump (initial)
support

Main issue of this patch is storage. We can reuse local buffers used for
temp tables. But it does allocation by 8KB and it creates temp files for
every object. That is too big overhead. Storing just in session memory is
too simple - then there should be lot of new code used, when variable will
be dropped.

I have ideas how to allow work with mix of scalar and composite types - so
it will be next step of this prototype.

Regards

Pavel


>

  42. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2018-03-08T18:00:36Z

    Hi

2018-02-07 7:34 GMT+01:00 Pavel Stehule <pavel.stehule@gmail.com>:

> Hi
>
> updated patch with your changes in documentation and pg_dump (initial)
> support
>
> Main issue of this patch is storage. We can reuse local buffers used for
> temp tables. But it does allocation by 8KB and it creates temp files for
> every object. That is too big overhead. Storing just in session memory is
> too simple - then there should be lot of new code used, when variable will
> be dropped.
>
> I have ideas how to allow work with mix of scalar and composite types - so
> it will be next step of this prototype.
>
> Regards
>
> Pavel
>

new update - rebased, + some initial support for composite values on right
side and custom types, arrays are supported too.

omega=# CREATE VARIABLE xx AS (a int, b numeric);
CREATE VARIABLE
omega=# LET xx = (10, 20)::xx;
LET
omega=# SELECT xx;
+---------+
|   xx    |
+---------+
| (10,20) |
+---------+
(1 row)

omega=# SELECT xx.a + xx.b;
+----------+
| ?column? |
+----------+
|       30 |
+----------+
(1 row)

omega=# \d xx
schema variable "public.xx"
+--------+---------+
| Column |  Type   |
+--------+---------+
| a      | integer |
| b      | numeric |
+--------+---------+


Regards

Pavel

  43. Re: [HACKERS] proposal: schema variables

    Pavel Luzanov <p.luzanov@postgrespro.ru> — 2018-03-12T06:49:29Z

    Hi,

I plan to make usability and feature test review in several days.

Is there any chances that it will work on replicas?
Such possibility is very helpful in generating reports.
Now, LET command produces an error:

ERROR:  cannot execute LET in a read-only transaction

But if we say that variables are non-transactional ?

-----
Pavel Luzanov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

On 08.03.2018 21:00, Pavel Stehule wrote:
> Hi
>
> 2018-02-07 7:34 GMT+01:00 Pavel Stehule <pavel.stehule@gmail.com 
> <mailto:pavel.stehule@gmail.com>>:
>
>     Hi
>
>     updated patch with your changes in documentation and pg_dump
>     (initial) support
>
>     Main issue of this patch is storage. We can reuse local buffers
>     used for temp tables. But it does allocation by 8KB and it creates
>     temp files for every object. That is too big overhead. Storing
>     just in session memory is too simple - then there should be lot of
>     new code used, when variable will be dropped.
>
>     I have ideas how to allow work with mix of scalar and composite
>     types - so it will be next step of this prototype.
>
>     Regards
>
>     Pavel
>
>
> new update - rebased, + some initial support for composite values on 
> right side and custom types, arrays are supported too.
>
> omega=# CREATE VARIABLE xx AS (a int, b numeric);
> CREATE VARIABLE
> omega=# LET xx = (10, 20)::xx;
> LET
> omega=# SELECT xx;
> +---------+
> |   xx    |
> +---------+
> | (10,20) |
> +---------+
> (1 row)
>
> omega=# SELECT xx.a + xx.b;
> +----------+
> | ?column? |
> +----------+
> |       30 |
> +----------+
> (1 row)
>
> omega=# \d xx
> schema variable "public.xx"
> +--------+---------+
> | Column |  Type   |
> +--------+---------+
> | a      | integer |
> | b      | numeric |
> +--------+---------+
>
>
> Regards
>
> Pavel
>
>

  44. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2018-03-12T06:54:26Z

    2018-03-12 7:49 GMT+01:00 Pavel Luzanov <p.luzanov@postgrespro.ru>:

> Hi,
>
> I plan to make usability and feature test review in several days.
>
> Is there any chances that it will work on replicas?
> Such possibility is very helpful in generating reports.
> Now, LET command produces an error:
>
> ERROR:  cannot execute LET in a read-only transaction
>
>

> But if we say that variables are non-transactional ?
>

sure, it should to work. Now, I am try to solve a issues on concept level -
the LET code is based on DML code base, so probably there is check for rw
transactions. But it is useless for LET command.

Regards

Pavel


>
> -----
> Pavel Luzanov
> Postgres Professional: http://www.postgrespro.com
> The Russian Postgres Company
>
> On 08.03.2018 21:00, Pavel Stehule wrote:
>
> Hi
>
> 2018-02-07 7:34 GMT+01:00 Pavel Stehule <pavel.stehule@gmail.com>:
>
>> Hi
>>
>> updated patch with your changes in documentation and pg_dump (initial)
>> support
>>
>> Main issue of this patch is storage. We can reuse local buffers used for
>> temp tables. But it does allocation by 8KB and it creates temp files for
>> every object. That is too big overhead. Storing just in session memory is
>> too simple - then there should be lot of new code used, when variable will
>> be dropped.
>>
>> I have ideas how to allow work with mix of scalar and composite types -
>> so it will be next step of this prototype.
>>
>> Regards
>>
>> Pavel
>>
>
> new update - rebased, + some initial support for composite values on right
> side and custom types, arrays are supported too.
>
> omega=# CREATE VARIABLE xx AS (a int, b numeric);
> CREATE VARIABLE
> omega=# LET xx = (10, 20)::xx;
> LET
> omega=# SELECT xx;
> +---------+
> |   xx    |
> +---------+
> | (10,20) |
> +---------+
> (1 row)
>
> omega=# SELECT xx.a + xx.b;
> +----------+
> | ?column? |
> +----------+
> |       30 |
> +----------+
> (1 row)
>
> omega=# \d xx
> schema variable "public.xx"
> +--------+---------+
> | Column |  Type   |
> +--------+---------+
> | a      | integer |
> | b      | numeric |
> +--------+---------+
>
>
> Regards
>
> Pavel
>
>
>
>

  45. Re: [HACKERS] proposal: schema variables

    Pavel Luzanov <p.luzanov@postgrespro.ru> — 2018-03-12T15:38:39Z

    On 12.03.2018 09:54, Pavel Stehule wrote:
>
> 2018-03-12 7:49 GMT+01:00 Pavel Luzanov <p.luzanov@postgrespro.ru 
> <mailto:p.luzanov@postgrespro.ru>>:
>
>
>     Is there any chances that it will work on replicas?
>
> ...
>
> sure, it should to work. Now, I am try to solve a issues on concept 
> level - the LET code is based on DML code base, so probably there is 
> check for rw transactions. But it is useless for LET command.

Very, very good!

As I understand, the work on this patch now in progress and it not in 
commitfest.
Please explain what features of schema variables I can review now.

 From first post of this thread the syntax of the CREATE VARIABLE command:
CREATE [TEMP] VARIABLE [IF NOT EXISTS] name AS type
   [ DEFAULT expression ] [[NOT] NULL]
   [ ON TRANSACTION END { RESET | DROP } ]
   [ { VOLATILE | STABLE } ];

But in psql I see only:
\h create variable
Command:     CREATE VARIABLE
Description: define a new permissioned typed schema variable
Syntax:
CREATE VARIABLE [ IF NOT EXISTS ] name [ AS ] data_type ]

I can include DEFAULT clause in CREATE VARIABLE command, but the value 
not used:
postgres=# create variable i int default 0;
CREATE VARIABLE
postgres=# select i;
  i
---

(1 row)

postgres=# \d+ i
  schema variable "public.i"
  Column |  Type   | Storage
--------+---------+---------
  i      | integer | plain


BTW, I found an error in handling of table aliases:

postgres=# create variable x text;
CREATE VARIABLE
postgres=# select * from pg_class AS x where x.relname = 'x';
ERROR:  type text is not composite

It thinks that x.relname is an attribute of x variable instead of an 
alias for pg_class table.


-----
Pavel Luzanov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

  46. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2018-03-12T16:13:40Z

    2018-03-12 16:38 GMT+01:00 Pavel Luzanov <p.luzanov@postgrespro.ru>:

>
> On 12.03.2018 09:54, Pavel Stehule wrote:
>
>
> 2018-03-12 7:49 GMT+01:00 Pavel Luzanov <p.luzanov@postgrespro.ru>:
>
>>
>> Is there any chances that it will work on replicas?
>>
> ...
>
> sure, it should to work. Now, I am try to solve a issues on concept level
> - the LET code is based on DML code base, so probably there is check for rw
> transactions. But it is useless for LET command.
>
>
> Very, very good!
>
> As I understand, the work on this patch now in progress and it not in
> commitfest.
> Please explain what features of schema variables I can review now.
>
> From first post of this thread the syntax of the CREATE VARIABLE command:
> CREATE [TEMP] VARIABLE [IF NOT EXISTS] name AS type
>   [ DEFAULT expression ] [[NOT] NULL]
>   [ ON TRANSACTION END { RESET | DROP } ]
>   [ { VOLATILE | STABLE } ];
>

Now, it is too early for review - it is in development. Some features are
not implemented yet - DEFAULTs, ON TRANSACTION END .., others has not sense
(what I know now VOLATILE, STABLE). Schema variables are passed as
parameters to query, so the behave is like any other params - it is STABLE
only.


>
> But in psql I see only:
> \h create variable
> Command:     CREATE VARIABLE
> Description: define a new permissioned typed schema variable
> Syntax:
> CREATE VARIABLE [ IF NOT EXISTS ] name [ AS ] data_type ]
>
> I can include DEFAULT clause in CREATE VARIABLE command, but the value not
> used:
> postgres=# create variable i int default 0;
> CREATE VARIABLE
> postgres=# select i;
>  i
> ---
>
> (1 row)
>
> postgres=# \d+ i
>  schema variable "public.i"
>  Column |  Type   | Storage
> --------+---------+---------
>  i      | integer | plain
>
>
defaults are not implemented yet


>
> BTW, I found an error in handling of table aliases:
>
> postgres=# create variable x text;
> CREATE VARIABLE
> postgres=# select * from pg_class AS x where x.relname = 'x';
> ERROR:  type text is not composite
>
> It thinks that x.relname is an attribute of x variable instead of an alias
> for pg_class table.
>
>
It is not well handled collision. This should be detected and prohibited.
In this case, because x is scalar, then x.xx has not sense, and then it
should not be handled like variable. So the current design is not too
practical - it generates more collisions than it is necessary and still,
there are some errors.

Now, there is one important question - storage - Postgres stores all
objects to files - only memory storage is not designed yet. This is part,
where I need a help.

Regards

Pavel


>
> -----
> Pavel Luzanov
> Postgres Professional: http://www.postgrespro.com
> The Russian Postgres Company
>
>

  47. Re: proposal: schema variables

    Pavel Luzanov <p.luzanov@postgrespro.ru> — 2018-03-13T09:54:30Z

    Pavel Stehule wrote
> Now, there is one important question - storage - Postgres stores all
> objects to files - only memory storage is not designed yet. This is part,
> where I need a help.

O, I do not feel confident in such questions.
May be some ideas you can get from extension with similar functionality:
https://github.com/postgrespro/pg_variables

-----
Pavel Luzanov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company



--
Sent from: http://www.postgresql-archive.org/PostgreSQL-hackers-f1928748.html

  48. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2018-03-13T18:44:01Z

    2018-03-13 10:54 GMT+01:00 Pavel Luzanov <p.luzanov@postgrespro.ru>:

> Pavel Stehule wrote
> > Now, there is one important question - storage - Postgres stores all
> > objects to files - only memory storage is not designed yet. This is part,
> > where I need a help.
>
> O, I do not feel confident in such questions.
> May be some ideas you can get from extension with similar functionality:
> https://github.com/postgrespro/pg_variables


Unfortunately not - it doesn't implement this functionality

Regards

Pavel


>
>
> -----
> Pavel Luzanov
> Postgres Professional: http://www.postgrespro.com
> The Russian Postgres Company
>
>
>
> --
> Sent from: http://www.postgresql-archive.org/PostgreSQL-hackers-
> f1928748.html
>
>

  49. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2018-03-20T17:38:47Z

    Hi

I am sending new update. The code is less ugly, and the current
functionality is +/- final for first stage. It should be good enough for
playing and testing this concept.

What is supported:

1. scalar, composite and array variables
2. composite can be defined on place or some composite type can be used
3. variable, or any field of variable, can have defined default value
4. variable is database object - the access rights are required
5. the values are stored in binary form with defined typmod

An usage is very simple:

postgres=# create variable foo as numeric default 0;
CREATE VARIABLE
postgres=# select foo;
┌─────┐
│ foo │
╞═════╡
│   0 │
└─────┘
(1 row)

postgres=# let foo = pi();
LET
postgres=# select foo;
┌──────────────────┐
│       foo        │
╞══════════════════╡
│ 3.14159265358979 │
└──────────────────┘
(1 row)

postgres=# create variable boo as (x numeric default 0, y numeric default
0);
CREATE VARIABLE
postgres=# let boo.x = 100;
LET
postgres=# select boo;
┌─────────┐
│   boo   │
╞═════════╡
│ (100,0) │
└─────────┘
(1 row)

postgres=# select boo.x;
┌─────┐
│  x  │
╞═════╡
│ 100 │
└─────┘
(1 row)

Please try it.

Regards

Pavel

  50. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2018-03-21T05:24:16Z

    2018-03-20 18:38 GMT+01:00 Pavel Stehule <pavel.stehule@gmail.com>:

> Hi
>
> I am sending new update. The code is less ugly, and the current
> functionality is +/- final for first stage. It should be good enough for
> playing and testing this concept.
>
> What is supported:
>
> 1. scalar, composite and array variables
> 2. composite can be defined on place or some composite type can be used
> 3. variable, or any field of variable, can have defined default value
> 4. variable is database object - the access rights are required
> 5. the values are stored in binary form with defined typmod
>
> An usage is very simple:
>
> postgres=# create variable foo as numeric default 0;
> CREATE VARIABLE
> postgres=# select foo;
> ┌─────┐
> │ foo │
> ╞═════╡
> │   0 │
> └─────┘
> (1 row)
>
> postgres=# let foo = pi();
> LET
> postgres=# select foo;
> ┌──────────────────┐
> │       foo        │
> ╞══════════════════╡
> │ 3.14159265358979 │
> └──────────────────┘
> (1 row)
>
> postgres=# create variable boo as (x numeric default 0, y numeric default
> 0);
> CREATE VARIABLE
> postgres=# let boo.x = 100;
> LET
> postgres=# select boo;
> ┌─────────┐
> │   boo   │
> ╞═════════╡
> │ (100,0) │
> └─────────┘
> (1 row)
>
> postgres=# select boo.x;
> ┌─────┐
> │  x  │
> ╞═════╡
> │ 100 │
> └─────┘
> (1 row)
>
> Please try it.
>

small fix - support for SQL functions


>
> Regards
>
> Pavel
>

  51. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2018-03-23T05:37:58Z

    2018-03-21 6:24 GMT+01:00 Pavel Stehule <pavel.stehule@gmail.com>:

>
>
> 2018-03-20 18:38 GMT+01:00 Pavel Stehule <pavel.stehule@gmail.com>:
>
>> Hi
>>
>> I am sending new update. The code is less ugly, and the current
>> functionality is +/- final for first stage. It should be good enough for
>> playing and testing this concept.
>>
>> What is supported:
>>
>> 1. scalar, composite and array variables
>> 2. composite can be defined on place or some composite type can be used
>> 3. variable, or any field of variable, can have defined default value
>> 4. variable is database object - the access rights are required
>> 5. the values are stored in binary form with defined typmod
>>
>> An usage is very simple:
>>
>> postgres=# create variable foo as numeric default 0;
>> CREATE VARIABLE
>> postgres=# select foo;
>> ┌─────┐
>> │ foo │
>> ╞═════╡
>> │   0 │
>> └─────┘
>> (1 row)
>>
>> postgres=# let foo = pi();
>> LET
>> postgres=# select foo;
>> ┌──────────────────┐
>> │       foo        │
>> ╞══════════════════╡
>> │ 3.14159265358979 │
>> └──────────────────┘
>> (1 row)
>>
>> postgres=# create variable boo as (x numeric default 0, y numeric default
>> 0);
>> CREATE VARIABLE
>> postgres=# let boo.x = 100;
>> LET
>> postgres=# select boo;
>> ┌─────────┐
>> │   boo   │
>> ╞═════════╡
>> │ (100,0) │
>> └─────────┘
>> (1 row)
>>
>> postgres=# select boo.x;
>> ┌─────┐
>> │  x  │
>> ╞═════╡
>> │ 100 │
>> └─────┘
>> (1 row)
>>
>> Please try it.
>>
>
> small fix - support for SQL functions
>
>

the patch is in commit fest list https://commitfest.postgresql.org/18/1608/

Regards

Pavel


>
>> Regards
>>
>> Pavel
>>
>
>

  52. Re: [HACKERS] proposal: schema variables

    Artur Zakirov <a.zakirov@postgrespro.ru> — 2018-04-17T14:14:12Z

    Hello Pavel,

On Thu, Oct 26, 2017 at 09:21:24AM +0200, Pavel Stehule wrote:
> I hope so this proposal is good enough and simple.
> 
> Comments, notes?

As I understood variables are stored in pg_class table. Did you consider
storing variables in a special catalog table? It can be named as
pg_variable.

pg_variable table requires more code of course, but it would fix the issues:
- pg_class has a lot attributes which are not related with variables,
  and I think variables don't need many of them
- in a future variables might want to have some additional attributes
  which are not needed for relations, you can easily add them to
  pg_variable

What do you think?

-- 
Arthur Zakirov
Postgres Professional: http://www.postgrespro.com
Russian Postgres Company

  53. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2018-04-17T16:28:19Z

    Hi

2018-04-17 16:14 GMT+02:00 Arthur Zakirov <a.zakirov@postgrespro.ru>:

> Hello Pavel,
>
> On Thu, Oct 26, 2017 at 09:21:24AM +0200, Pavel Stehule wrote:
> > I hope so this proposal is good enough and simple.
> >
> > Comments, notes?
>
> As I understood variables are stored in pg_class table. Did you consider
> storing variables in a special catalog table? It can be named as
> pg_variable.
>
> pg_variable table requires more code of course, but it would fix the
> issues:
> - pg_class has a lot attributes which are not related with variables,
>   and I think variables don't need many of them
> - in a future variables might want to have some additional attributes
>   which are not needed for relations, you can easily add them to
>   pg_variable
>
> What do you think?
>

I though about it, and I am inclined to prefer pg_class instead separate
tables.

It true, so there are lot of "unused" attributes for this purpose, but
there is lot of shared attributes, and lot of shared code. Semantically, I
see variables in family of sequences, tables, indexes, views. Now, it
shares code, and I hope in next steps more code can be shared -
constraints, triggers.

There are two objective arguments for using pg_class:

1. unique name in schema - it reduces risk of collisions
2. sharing lot of code

So in this case I don't see well benefits of separate table.

Regards

Pavel



>
> --
> Arthur Zakirov
> Postgres Professional: http://www.postgrespro.com
> Russian Postgres Company
>

  54. Re: [HACKERS] proposal: schema variables

    Artur Zakirov <a.zakirov@postgrespro.ru> — 2018-04-18T11:37:11Z

    On Tue, Apr 17, 2018 at 06:28:19PM +0200, Pavel Stehule wrote:
> I though about it, and I am inclined to prefer pg_class instead separate
> tables.
> 
> It true, so there are lot of "unused" attributes for this purpose, but
> there is lot of shared attributes, and lot of shared code. Semantically, I
> see variables in family of sequences, tables, indexes, views. Now, it
> shares code, and I hope in next steps more code can be shared -
> constraints, triggers.
> 
> There are two objective arguments for using pg_class:
> 
> 1. unique name in schema - it reduces risk of collisions
> 2. sharing lot of code
> 
> So in this case I don't see well benefits of separate table.

Understood. I haven't strong opinion here though. But I thought that
pg_class approach may limit extensibility of variables.

BTW:
- there is unitialized variable 'j' in pg_dump.c:15422
- in tab-complete.c:1268 initialization needs extra NULL before
  &Query_for_list_of_variables

Also I think makeRangeVarForTargetOfSchemaVariable() has non friendly
argument names 'field1', 'field2', 'field2'.

-- 
Arthur Zakirov
Postgres Professional: http://www.postgrespro.com
Russian Postgres Company

  55. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2018-04-18T11:54:44Z

    Hi

I am sending rebased patch

2018-04-18 13:37 GMT+02:00 Arthur Zakirov <a.zakirov@postgrespro.ru>:

> On Tue, Apr 17, 2018 at 06:28:19PM +0200, Pavel Stehule wrote:
> > I though about it, and I am inclined to prefer pg_class instead separate
> > tables.
> >
> > It true, so there are lot of "unused" attributes for this purpose, but
> > there is lot of shared attributes, and lot of shared code. Semantically,
> I
> > see variables in family of sequences, tables, indexes, views. Now, it
> > shares code, and I hope in next steps more code can be shared -
> > constraints, triggers.
> >
> > There are two objective arguments for using pg_class:
> >
> > 1. unique name in schema - it reduces risk of collisions
> > 2. sharing lot of code
> >
> > So in this case I don't see well benefits of separate table.
>
> Understood. I haven't strong opinion here though. But I thought that
> pg_class approach may limit extensibility of variables.
>

I didn't touch limit (I don't know if there will be some issue - still is
far to upstream). This is technology, but workable, demo. I though so some
users had a problem to imagine what is persistent variable in my view.

But almost all code and tests can be used for final version - only storage
implementation is nothing more than workaround.


>
> BTW:
> - there is unitialized variable 'j' in pg_dump.c:15422
> - in tab-complete.c:1268 initialization needs extra NULL before
>   &Query_for_list_of_variables
>

I found it too today when I did rebase. But thank you for report.


>
> Also I think makeRangeVarForTargetOfSchemaVariable() has non friendly
> argument names 'field1', 'field2', 'field2'.
>

yes, I hadn't better names :(

In this routine I am doing diagnostic what semantic has sense for current
values. the field1, field2 can be schema.variable or variable.field. So
when I used semantic names: schema, varname, fieldname, then it was more
messy (for me).

Regards

Pavel


>
> --
> Arthur Zakirov
> Postgres Professional: http://www.postgrespro.com
> Russian Postgres Company
>

  56. Re: [HACKERS] proposal: schema variables

    Robert Haas <robertmhaas@gmail.com> — 2018-04-20T15:32:07Z

    On Tue, Apr 17, 2018 at 12:28 PM, Pavel Stehule <pavel.stehule@gmail.com> wrote:
> It true, so there are lot of "unused" attributes for this purpose, but there
> is lot of shared attributes, and lot of shared code. Semantically, I see
> variables in family of sequences, tables, indexes, views. Now, it shares
> code, and I hope in next steps more code can be shared - constraints,
> triggers.

I dunno, it seems awfully different to me.  There's only one "column",
right?  What code is really shared here?  Are constraints and triggers
even desirable feature for variables?  What would be the use case?

I think stuffing this into pg_class is pretty strange.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  57. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2018-04-20T17:45:06Z

    2018-04-20 17:32 GMT+02:00 Robert Haas <robertmhaas@gmail.com>:

> On Tue, Apr 17, 2018 at 12:28 PM, Pavel Stehule <pavel.stehule@gmail.com>
> wrote:
> > It true, so there are lot of "unused" attributes for this purpose, but
> there
> > is lot of shared attributes, and lot of shared code. Semantically, I see
> > variables in family of sequences, tables, indexes, views. Now, it shares
> > code, and I hope in next steps more code can be shared - constraints,
> > triggers.
>
> I dunno, it seems awfully different to me.  There's only one "column",
> right?  What code is really shared here?  Are constraints and triggers
> even desirable feature for variables?  What would be the use case?
>

The schema variable can hold composite value. The patch allows to use any
composite type or adhoc composite values

DECLARE x AS compositetype;
DECLARE x AS (a int, b int, c int);

Constraints are clear, no.

Triggers are strange maybe, but why not - it can be used like enhanced
constraints, can be used for some value calculations, ..


> I think stuffing this into pg_class is pretty strange.
>

It will be if variable is just scalar value without any possibilities. But
then there is only low benefit

The access rights implementation is shared with other from pg_class too.

Regards

Pavel


>
> --
> Robert Haas
> EnterpriseDB: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company
>

  58. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2018-04-29T12:34:41Z

    Hi

I am sending rebased patch to master

Regards

Pavel

  59. Re: proposal: schema variables

    Fabrízio de Royes Mello <fabriziomello@gmail.com> — 2018-04-30T17:28:31Z

    The following review has been posted through the commitfest application:
make installcheck-world:  tested, passed
Implements feature:       not tested
Spec compliant:           not tested
Documentation:            not tested

1) There are some errors applying the patch against the current master:

fabrizio@macanudo:/d/postgresql (master) 
$ git apply /tmp/schema-variables-poc-180429-01-diff
/tmp/schema-variables-poc-180429-01-diff:2305: trailing whitespace.
 
/tmp/schema-variables-poc-180429-01-diff:2317: indent with spaces.
    We can support UPDATE and SELECT commands on variables.
/tmp/schema-variables-poc-180429-01-diff:2319: indent with spaces.
    possible syntaxes:
/tmp/schema-variables-poc-180429-01-diff:2321: indent with spaces.
        -- there can be a analogy with record functions
/tmp/schema-variables-poc-180429-01-diff:2322: indent with spaces.
        SELECT varname;
warning: squelched 14 whitespace errors
warning: 19 lines add whitespace errors.


2) There are some warnings when during build process

schemavar.c:383:18: warning: expression which evaluates to zero treated as a null pointer constant of type 'HeapTuple' (aka 'struct HeapTupleData *')
      [-Wnon-literal-null-conversion]
                HeapTuple       tp = InvalidOid;
                                     ^~~~~~~~~~
../../../src/include/postgres_ext.h:36:21: note: expanded from macro 'InvalidOid'
#define InvalidOid              ((Oid) 0)
                                ^~~~~~~~~
1 warning generated.
tab-complete.c:1268:21: warning: initialization from incompatible pointer type [-Wincompatible-pointer-types]
  {"VARIABLE", NULL, &Query_for_list_of_variables},
                     ^
tab-complete.c:1268:21: note: (near initialization for ‘words_after_create[48].vquery’)
llvmjit_expr.c: In function ‘llvm_compile_expr’:
llvmjit_expr.c:253:3: warning: enumeration value ‘EEOP_PARAM_SCHEMA_VARIABLE’ not handled in switch [-Wswitch]
   switch (opcode)
   ^

  60. Re: [HACKERS] proposal: schema variables

    Peter Eisentraut <peter.eisentraut@2ndquadrant.com> — 2018-05-01T01:56:45Z

    On 4/20/18 13:45, Pavel Stehule wrote:
>     I dunno, it seems awfully different to me.  There's only one "column",
>     right?  What code is really shared here?  Are constraints and triggers
>     even desirable feature for variables?  What would be the use case?
> 
> 
> The schema variable can hold composite value. The patch allows to use
> any composite type or adhoc composite values
> 
> DECLARE x AS compositetype;
> DECLARE x AS (a int, b int, c int);

I'm not sure that this anonymous composite type thing is such a good
idea.  Such a variable will then be incompatible with anything else,
because it's of a different type.

In any case, I find that a weak argument for storing this in pg_class.
You could just as well create these pg_class entries implicitly and link
them from "pg_variable", same as composite types have a main entry in
pg_type and additional stuff in pg_class.

>     I think stuffing this into pg_class is pretty strange.
> 
> It will be if variable is just scalar value without any possibilities.
> But then there is only low benefit
> 
> The access rights implementation is shared with other from pg_class too.

In DB2, the privileges for variables are named READ and WRITE.  That
would make more sense to me than reusing the privilege names for tables.

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  61. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2018-05-01T03:11:27Z

    Hi

2018-05-01 3:56 GMT+02:00 Peter Eisentraut <peter.eisentraut@2ndquadrant.com
>:

> On 4/20/18 13:45, Pavel Stehule wrote:
> >     I dunno, it seems awfully different to me.  There's only one
> "column",
> >     right?  What code is really shared here?  Are constraints and
> triggers
> >     even desirable feature for variables?  What would be the use case?
> >
> >
> > The schema variable can hold composite value. The patch allows to use
> > any composite type or adhoc composite values
> >
> > DECLARE x AS compositetype;
> > DECLARE x AS (a int, b int, c int);
>
> I'm not sure that this anonymous composite type thing is such a good
> idea.  Such a variable will then be incompatible with anything else,
> because it's of a different type.
>

Using anonymous composite type variable is just shortcut for situations
when mentioned feature is not a problem. These variables are global, so
there can be only one variable of some specific composite type, and
incompatibility with others is not a issue.

This feature can be interesting for short live temp variables - these
variables can be used for parametrization of anonymous block.

But this feature is not significant, and can be removed from patch.


> In any case, I find that a weak argument for storing this in pg_class.
> You could just as well create these pg_class entries implicitly and link
> them from "pg_variable", same as composite types have a main entry in
> pg_type and additional stuff in pg_class.
>
> >     I think stuffing this into pg_class is pretty strange.
> >
> > It will be if variable is just scalar value without any possibilities.
> > But then there is only low benefit
> >
> > The access rights implementation is shared with other from pg_class too.
>
> In DB2, the privileges for variables are named READ and WRITE.  That
> would make more sense to me than reusing the privilege names for tables.
>
>
good idea

Regards

Pavel

> --
> Peter Eisentraut              http://www.2ndQuadrant.com/
> PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
>

  62. Re: [HACKERS] proposal: schema variables

    Gilles Darold <gilles.darold@dalibo.com> — 2018-06-27T10:21:16Z

    Hi,

I'm reviewing the patch as it was flagged in the current commit fest. 
Here are my feedback:

  - The patch need to be rebased due to changes in file 
src/sgml/catalogs.sgml

  - Some compilation warning must be fixed:

    analyze.c: In function ‘transformLetStmt’:
    analyze.c:1568:17: warning: variable ‘rte’ set but not used
    [-Wunused-but-set-variable]
       RangeTblEntry *rte;
                      ^~~
    tab-complete.c:1268:21: warning: initialization from incompatible
    pointer type [-Wincompatible-pointer-types]
       {"VARIABLE", NULL, &Query_for_list_of_variables},

    In the last warning a NULL is missing, should be written:
    {"VARIABLE", NULL, NULL, &Query_for_list_of_variables},


  - How about Peter's suggestion?:
     "In DB2, the privileges for variables are named READ and WRITE. 
That would make more sense to me than reusing the privilege names for 
tables.
     The patch use SELECT and UPDATE which make sense too for SELECT but 
less for UPDATE.

  - The implementation of "ALTER VARIABLE varname SET SCHEMA 
schema_name;" is missing

  - ALTER VARIABLE var1 OWNER TO gilles; ok but not documented and 
missing in regression test

  - ALTER VARIABLE var1 RENAME TO var2; ok but not documented and 
missing in regression test

More generally I think that some comments must be rewritten, especially 
those talking about a PoC. In documentation there is HTML comments that 
can be removed.

Comment at end of file src/backend/commands/schemavar.c generate some 
"indent with spaces" errors with git apply but perhaps the comment can 
be entirely removed or undocumented details moved to the right place.

Otherwise all regression tests passed without issue and especially your 
new regression tests about schema variables.

I have a patch rebased, let me known if you want me to post the new diff.


-- 
Gilles Darold
Consultant PostgreSQL
http://dalibo.com - http://dalibo.org

  63. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2018-06-27T11:22:10Z

    Hi

2018-06-27 12:21 GMT+02:00 Gilles Darold <gilles.darold@dalibo.com>:

> Hi,
>
> I'm reviewing the patch as it was flagged in the current commit fest. Here
> are my feedback:
>
>  - The patch need to be rebased due to changes in file
> src/sgml/catalogs.sgml
>
>  - Some compilation warning must be fixed:
>
> analyze.c: In function ‘transformLetStmt’:
> analyze.c:1568:17: warning: variable ‘rte’ set but not used
> [-Wunused-but-set-variable]
>   RangeTblEntry *rte;
>                  ^~~
> tab-complete.c:1268:21: warning: initialization from incompatible pointer
> type [-Wincompatible-pointer-types]
>   {"VARIABLE", NULL, &Query_for_list_of_variables},
>
> In the last warning a NULL is missing, should be written: {"VARIABLE",
> NULL, NULL, &Query_for_list_of_variables},
>
>
>  - How about Peter's suggestion?:
>     "In DB2, the privileges for variables are named READ and WRITE. That
> would make more sense to me than reusing the privilege names for tables.
>
    The patch use SELECT and UPDATE which make sense too for SELECT but
> less for UPDATE.
>
>  - The implementation of "ALTER VARIABLE varname SET SCHEMA schema_name;"
> is missing
>
>  - ALTER VARIABLE var1 OWNER TO gilles; ok but not documented and missing
> in regression test
>
>  - ALTER VARIABLE var1 RENAME TO var2; ok but not documented and missing
> in regression test
>
> More generally I think that some comments must be rewritten, especially
> those talking about a PoC. In documentation there is HTML comments that can
> be removed.
>
> Comment at end of file src/backend/commands/schemavar.c generate some
> "indent with spaces" errors with git apply but perhaps the comment can be
> entirely removed or undocumented details moved to the right place.
>
> Otherwise all regression tests passed without issue and especially your
> new regression tests about schema variables.
>
> I have a patch rebased, let me known if you want me to post the new diff.
>

I plan significant refactoring of this patch for next commitfest. There was
anotherstrong Peter's and Robert comments

1. The schema variables should to have own system table
2. The composite schema variables should to use explicitly defined
composite type
3. The memory management is not nice - transactional drop table with
content is implemented ugly.

I hope, so I can start on these issues next month.

Thank you for review - I'll recheck ALTER commands.

Regards

Pavel

>
> --
> Gilles Darold
> Consultant PostgreSQLhttp://dalibo.com - http://dalibo.org
>
>

  64. Re: [HACKERS] proposal: schema variables

    Gilles Darold <gilles.darold@dalibo.com> — 2018-06-27T17:15:54Z

    Le 27/06/2018 à 13:22, Pavel Stehule a écrit :
> Hi
>
> 2018-06-27 12:21 GMT+02:00 Gilles Darold <gilles.darold@dalibo.com 
> <mailto:gilles.darold@dalibo.com>>:
>
>     Hi,
>
>     I'm reviewing the patch as it was flagged in the current commit
>     fest. Here are my feedback:
>
>      - The patch need to be rebased due to changes in file
>     src/sgml/catalogs.sgml
>
>      - Some compilation warning must be fixed:
>
>         analyze.c: In function ‘transformLetStmt’:
>         analyze.c:1568:17: warning: variable ‘rte’ set but not used
>         [-Wunused-but-set-variable]
>           RangeTblEntry *rte;
>                          ^~~
>         tab-complete.c:1268:21: warning: initialization from
>         incompatible pointer type [-Wincompatible-pointer-types]
>           {"VARIABLE", NULL, &Query_for_list_of_variables},
>
>         In the last warning a NULL is missing, should be written:
>         {"VARIABLE", NULL, NULL, &Query_for_list_of_variables},
>
>
>      - How about Peter's suggestion?:
>         "In DB2, the privileges for variables are named READ and
>     WRITE. That would make more sense to me than reusing the privilege
>     names for tables.
>
>         The patch use SELECT and UPDATE which make sense too for
>     SELECT but less for UPDATE.
>
>      - The implementation of "ALTER VARIABLE varname SET SCHEMA
>     schema_name;" is missing
>
>      - ALTER VARIABLE var1 OWNER TO gilles; ok but not documented and
>     missing in regression test
>
>      - ALTER VARIABLE var1 RENAME TO var2; ok but not documented and
>     missing in regression test
>
>     More generally I think that some comments must be rewritten,
>     especially those talking about a PoC. In documentation there is
>     HTML comments that can be removed.
>
>     Comment at end of file src/backend/commands/schemavar.c generate
>     some "indent with spaces" errors with git apply but perhaps the
>     comment can be entirely removed or undocumented details moved to
>     the right place.
>
>     Otherwise all regression tests passed without issue and especially
>     your new regression tests about schema variables.
>
>     I have a patch rebased, let me known if you want me to post the
>     new diff.
>
>
> I plan significant refactoring of this patch for next commitfest. 
> There was anotherstrong Peter's and Robert comments
>
> 1. The schema variables should to have own system table
> 2. The composite schema variables should to use explicitly defined 
> composite type
> 3. The memory management is not nice - transactional drop table with 
> content is implemented ugly.
>
> I hope, so I can start on these issues next month.
>
> Thank you for review - I'll recheck ALTER commands.
>
>
>     Otherwise all regression tests passed without issue and especially
>     your new regression tests about schema variables.
>
>     I have a patch rebased, let me known if you want me to post the
>     new diff.
>
>
> I plan significant refactoring of this patch for next commitfest. 
> There was anotherstrong Peter's and Robert c
> Regards


Ok Pavel, I've changed the status to "Waiting for authors" so that no 
one will make an other review until you send a new patch.


-- 
Gilles Darold
Consultant PostgreSQL
http://dalibo.com - http://dalibo.org

  65. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2018-06-27T17:17:05Z

    2018-06-27 19:15 GMT+02:00 Gilles Darold <gilles.darold@dalibo.com>:

> Le 27/06/2018 à 13:22, Pavel Stehule a écrit :
>
> Hi
>
> 2018-06-27 12:21 GMT+02:00 Gilles Darold <gilles.darold@dalibo.com>:
>
>> Hi,
>>
>> I'm reviewing the patch as it was flagged in the current commit fest.
>> Here are my feedback:
>>
>>  - The patch need to be rebased due to changes in file
>> src/sgml/catalogs.sgml
>>
>>  - Some compilation warning must be fixed:
>>
>> analyze.c: In function ‘transformLetStmt’:
>> analyze.c:1568:17: warning: variable ‘rte’ set but not used
>> [-Wunused-but-set-variable]
>>   RangeTblEntry *rte;
>>                  ^~~
>> tab-complete.c:1268:21: warning: initialization from incompatible pointer
>> type [-Wincompatible-pointer-types]
>>   {"VARIABLE", NULL, &Query_for_list_of_variables},
>>
>> In the last warning a NULL is missing, should be written: {"VARIABLE",
>> NULL, NULL, &Query_for_list_of_variables},
>>
>>
>>  - How about Peter's suggestion?:
>>     "In DB2, the privileges for variables are named READ and WRITE. That
>> would make more sense to me than reusing the privilege names for tables.
>>
>     The patch use SELECT and UPDATE which make sense too for SELECT but
>> less for UPDATE.
>>
>>  - The implementation of "ALTER VARIABLE varname SET SCHEMA schema_name;"
>> is missing
>>
>>  - ALTER VARIABLE var1 OWNER TO gilles; ok but not documented and missing
>> in regression test
>>
>>  - ALTER VARIABLE var1 RENAME TO var2; ok but not documented and missing
>> in regression test
>>
>> More generally I think that some comments must be rewritten, especially
>> those talking about a PoC. In documentation there is HTML comments that can
>> be removed.
>>
>> Comment at end of file src/backend/commands/schemavar.c generate some
>> "indent with spaces" errors with git apply but perhaps the comment can be
>> entirely removed or undocumented details moved to the right place.
>>
>> Otherwise all regression tests passed without issue and especially your
>> new regression tests about schema variables.
>>
>> I have a patch rebased, let me known if you want me to post the new diff.
>>
>
> I plan significant refactoring of this patch for next commitfest. There
> was anotherstrong Peter's and Robert comments
>
> 1. The schema variables should to have own system table
> 2. The composite schema variables should to use explicitly defined
> composite type
> 3. The memory management is not nice - transactional drop table with
> content is implemented ugly.
>
> I hope, so I can start on these issues next month.
>
> Thank you for review - I'll recheck ALTER commands.
>
>>
>> Otherwise all regression tests passed without issue and especially your
>> new regression tests about schema variables.
>>
>> I have a patch rebased, let me known if you want me to post the new diff.
>>
>
> I plan significant refactoring of this patch for next commitfest. There
> was anotherstrong Peter's and Robert c
> Regards
>
>
> Ok Pavel, I've changed the status to "Waiting for authors" so that no one
> will make an other review until you send a new patch.
>
sure

Thank you

Pavel


>
> --
> Gilles Darold
> Consultant PostgreSQLhttp://dalibo.com - http://dalibo.org
>
>

  66. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2018-08-08T20:29:29Z

    2018-06-27 19:15 GMT+02:00 Gilles Darold <gilles.darold@dalibo.com>:

> Le 27/06/2018 à 13:22, Pavel Stehule a écrit :
>
> Hi
>
> 2018-06-27 12:21 GMT+02:00 Gilles Darold <gilles.darold@dalibo.com>:
>
>> Hi,
>>
>> I'm reviewing the patch as it was flagged in the current commit fest.
>> Here are my feedback:
>>
>>  - The patch need to be rebased due to changes in file
>> src/sgml/catalogs.sgml
>>
>>  - Some compilation warning must be fixed:
>>
>> analyze.c: In function ‘transformLetStmt’:
>> analyze.c:1568:17: warning: variable ‘rte’ set but not used
>> [-Wunused-but-set-variable]
>>   RangeTblEntry *rte;
>>                  ^~~
>> tab-complete.c:1268:21: warning: initialization from incompatible pointer
>> type [-Wincompatible-pointer-types]
>>   {"VARIABLE", NULL, &Query_for_list_of_variables},
>>
>> In the last warning a NULL is missing, should be written: {"VARIABLE",
>> NULL, NULL, &Query_for_list_of_variables},
>>
>>
>>  - How about Peter's suggestion?:
>>     "In DB2, the privileges for variables are named READ and WRITE. That
>> would make more sense to me than reusing the privilege names for tables.
>>
>     The patch use SELECT and UPDATE which make sense too for SELECT but
>> less for UPDATE.
>>
>>  - The implementation of "ALTER VARIABLE varname SET SCHEMA schema_name;"
>> is missing
>>
>>  - ALTER VARIABLE var1 OWNER TO gilles; ok but not documented and missing
>> in regression test
>>
>>  - ALTER VARIABLE var1 RENAME TO var2; ok but not documented and missing
>> in regression test
>>
>> More generally I think that some comments must be rewritten, especially
>> those talking about a PoC. In documentation there is HTML comments that can
>> be removed.
>>
>> Comment at end of file src/backend/commands/schemavar.c generate some
>> "indent with spaces" errors with git apply but perhaps the comment can be
>> entirely removed or undocumented details moved to the right place.
>>
>> Otherwise all regression tests passed without issue and especially your
>> new regression tests about schema variables.
>>
>> I have a patch rebased, let me known if you want me to post the new diff.
>>
>
> I plan significant refactoring of this patch for next commitfest. There
> was anotherstrong Peter's and Robert comments
>
> 1. The schema variables should to have own system table
> 2. The composite schema variables should to use explicitly defined
> composite type
> 3. The memory management is not nice - transactional drop table with
> content is implemented ugly.
>
> I hope, so I can start on these issues next month.
>
> Thank you for review - I'll recheck ALTER commands.
>
>>
>> Otherwise all regression tests passed without issue and especially your
>> new regression tests about schema variables.
>>
>> I have a patch rebased, let me known if you want me to post the new diff.
>>
>
> I plan significant refactoring of this patch for next commitfest. There
> was anotherstrong Peter's and Robert c
> Regards
>
>
> Ok Pavel, I've changed the status to "Waiting for authors" so that no one
> will make an other review until you send a new patch.
>
I am sending a new update of this feature. The functionality is +/- same
like previous patch, but a implementation is based on own catalog table.

I removed functions for manipulation with schema variables. These functions
can be added later simply. Now If we hold these functions, then we should
to solve often collision inside pg_proc.

Changes:

* own catalog - pg_variable
* the rights are renamed - READ|WRITE
* the code is cleaner

Regards

Pavel

>
> --
> Gilles Darold
> Consultant PostgreSQLhttp://dalibo.com - http://dalibo.org
>
>

  67. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2018-08-08T20:35:28Z

    removed forgotten file

Regards

Pavel

  68. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2018-08-11T05:39:49Z

    Hi

I am sending updated patch. It should to solve almost all Giles's and
Peter's objections.

I am not happy so executor access values of variables directly. It is most
simple implementation - and I hope so it is good enough, but now the access
to variables is too volatile. But it is works good enough for usability
testing.

I am thinking about some cache of used variables in ExprContext, so the
variable in one ExprContext will look like stable - more like PLpgSQL
variables.

Regards

Pavel

  69. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2018-08-11T18:46:34Z

    2018-08-11 7:39 GMT+02:00 Pavel Stehule <pavel.stehule@gmail.com>:

> Hi
>
> I am sending updated patch. It should to solve almost all Giles's and
> Peter's objections.
>
> I am not happy so executor access values of variables directly. It is most
> simple implementation - and I hope so it is good enough, but now the access
> to variables is too volatile. But it is works good enough for usability
> testing.
>
> I am thinking about some cache of used variables in ExprContext, so the
> variable in one ExprContext will look like stable - more like PLpgSQL
> variables.
>

I wrote EState based schema variable values cache, so now the variables in
queries are stable (like PARAM_EXTERN) and can be used for optimization.

Regards

Pavel


>
> Regards
>
> Pavel
>

  70. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2018-08-12T05:35:33Z

    Hi

2018-08-11 20:46 GMT+02:00 Pavel Stehule <pavel.stehule@gmail.com>:

>
>
> 2018-08-11 7:39 GMT+02:00 Pavel Stehule <pavel.stehule@gmail.com>:
>
>> Hi
>>
>> I am sending updated patch. It should to solve almost all Giles's and
>> Peter's objections.
>>
>> I am not happy so executor access values of variables directly. It is
>> most simple implementation - and I hope so it is good enough, but now the
>> access to variables is too volatile. But it is works good enough for
>> usability testing.
>>
>> I am thinking about some cache of used variables in ExprContext, so the
>> variable in one ExprContext will look like stable - more like PLpgSQL
>> variables.
>>
>
> I wrote EState based schema variable values cache, so now the variables in
> queries are stable (like PARAM_EXTERN) and can be used for optimization.
>

new update - after cleaning

Regards

Pavel


> Regards
>
> Pavel
>
>
>>
>> Regards
>>
>> Pavel
>>
>
>

  71. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2018-08-14T14:38:56Z

    Hi

I wrote missing collation support

Regards

Pavel

  72. Re: [HACKERS] proposal: schema variables

    Fabien COELHO <coelho@cri.ensmp.fr> — 2018-08-21T17:55:57Z

    Hello Pavel,

AFAICR, I had an objection on such new objects when you first proposed 
something similar in October 2016.

Namely, if session variables are not transactional, they cannot be used to 
implement security related auditing features which were advertised as the 
motivating use case: an the audit check may fail on a commit because of a 
differed constraint, but the variable would keep its "okay" value unduly, 
which would create a latent security issue, the audit check having failed 
but the variable saying the opposite.

So my point was that they should be transactional by default, although I 
would be ok with an option for having a voluntary non transactional 
version.

Is this issue addressed somehow with this version?

-- 
Fabien.

  73. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2018-08-21T18:48:25Z

    Hi Fabien

Dne út 21. 8. 2018 19:56 uživatel Fabien COELHO <coelho@cri.ensmp.fr>
napsal:

>
> Hello Pavel,
>
> AFAICR, I had an objection on such new objects when you first proposed
> something similar in October 2016.
>
> Namely, if session variables are not transactional, they cannot be used to
> implement security related auditing features which were advertised as the
> motivating use case: an the audit check may fail on a commit because of a
> differed constraint, but the variable would keep its "okay" value unduly,
> which would create a latent security issue, the audit check having failed
> but the variable saying the opposite.
>
> So my point was that they should be transactional by default, although I
> would be ok with an option for having a voluntary non transactional
> version.
>
> Is this issue addressed somehow with this ?



1. I respect your opinion, but I dont agree with it. Oracle, db2 has
similar or very similar feature non transactional, and I didnt find any
requests to change it.

2. the prototype implementation was based on relclass items, and some
transactional behave was possible. Peter E. had objections to this design
and proposed own catalog table. I did it. Now, the transactional behave is
harder to implement, although it is not impossible. This patch is not small
now, so I didnt implement it. I have a strong opinion so default behave
have to be non transactional.

Transactional variables significantly increases complexity of this patch,
now is simple, because we can reset variable on drop variable command.
Maybe I miss some simply implementation, but I spent on it more than few
days. Still, any cooperation are welcome.

Regards

Pavel




> --
> Fabien.
>

  74. Re: [HACKERS] proposal: schema variables

    Fabien COELHO <coelho@cri.ensmp.fr> — 2018-08-22T07:00:45Z

    Hello Pavel,

>> AFAICR, I had an objection on such new objects when you first proposed
>> something similar in October 2016.
>>
>> Namely, if session variables are not transactional, they cannot be used to
>> implement security related auditing features which were advertised as the
>> motivating use case: an the audit check may fail on a commit because of a
>> differed constraint, but the variable would keep its "okay" value unduly,
>> which would create a latent security issue, the audit check having failed
>> but the variable saying the opposite.
>>
>> So my point was that they should be transactional by default, although I
>> would be ok with an option for having a voluntary non transactional
>> version.
>>
>> Is this issue addressed somehow with this ?
>
>
> 1. I respect your opinion, but I dont agree with it. Oracle, db2 has
> similar or very similar feature non transactional, and I didnt find any
> requests to change it.

The argument of authority that "X does it like that" is not a valid answer 
to my technical objection about security implications of this feature.

> 2. the prototype implementation was based on relclass items, and some
> transactional behave was possible. Peter E. had objections to this design
> and proposed own catalog table. I did it. Now, the transactional behave is
> harder to implement, although it is not impossible. This patch is not small
> now, so I didnt implement it.

"It is harder to implement" does not look like a valid answer either.

> I have a strong opinion so default behave have to be non transactional.

The fact that you have a "strong opinion" does not really answer my 
objection. Moreover, I said that I would be ok with a non transactional 
option, provided that a default transactional is available.

> Transactional variables significantly increases complexity of this patch,
> now is simple, because we can reset variable on drop variable command.
> Maybe I miss some simply implementation, but I spent on it more than few
> days. Still, any cooperation are welcome.

"It is simpler to implement this way" is not an answer either, especially 
as you said that it could have been on point 2.


As I do not see any clear answer to my objection about security 
implications, I understand that it is not addressed by this patch.


At the bare minimum, if this feature ever made it as is, I think that a 
clear caveat must be included in the documentation about not using it for 
any security-related purpose.

Also, I'm not really sure how useful such a non-transactional object can 
be for other purposes: the user should take into account that the 
transaction may fail and the value of the session variable be inconsistent 
as a result. Sometimes it may not matter, but if it matters there is no 
easy way around the fact.

-- 
Fabien.

  75. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2018-08-23T05:35:22Z

    2018-08-22 9:00 GMT+02:00 Fabien COELHO <coelho@cri.ensmp.fr>:

>
> Hello Pavel,
>
> AFAICR, I had an objection on such new objects when you first proposed
>>> something similar in October 2016.
>>>
>>> Namely, if session variables are not transactional, they cannot be used
>>> to
>>> implement security related auditing features which were advertised as the
>>> motivating use case: an the audit check may fail on a commit because of a
>>> differed constraint, but the variable would keep its "okay" value unduly,
>>> which would create a latent security issue, the audit check having failed
>>> but the variable saying the opposite.
>>>
>>> So my point was that they should be transactional by default, although I
>>> would be ok with an option for having a voluntary non transactional
>>> version.
>>>
>>> Is this issue addressed somehow with this ?
>>>
>>
>>
>> 1. I respect your opinion, but I dont agree with it. Oracle, db2 has
>> similar or very similar feature non transactional, and I didnt find any
>> requests to change it.
>>
>
> The argument of authority that "X does it like that" is not a valid answer
> to my technical objection about security implications of this feature.
>
> 2. the prototype implementation was based on relclass items, and some
>> transactional behave was possible. Peter E. had objections to this design
>> and proposed own catalog table. I did it. Now, the transactional behave is
>> harder to implement, although it is not impossible. This patch is not
>> small
>> now, so I didnt implement it.
>>
>
> "It is harder to implement" does not look like a valid answer either.
>
> I have a strong opinion so default behave have to be non transactional.
>>
>
> The fact that you have a "strong opinion" does not really answer my
> objection. Moreover, I said that I would be ok with a non transactional
> option, provided that a default transactional is available.
>
> Transactional variables significantly increases complexity of this patch,
>> now is simple, because we can reset variable on drop variable command.
>> Maybe I miss some simply implementation, but I spent on it more than few
>> days. Still, any cooperation are welcome.
>>
>
> "It is simpler to implement this way" is not an answer either, especially
> as you said that it could have been on point 2.
>
>
> As I do not see any clear answer to my objection about security
> implications, I understand that it is not addressed by this patch.
>
>
> At the bare minimum, if this feature ever made it as is, I think that a
> clear caveat must be included in the documentation about not using it for
> any security-related purpose.
>
> Also, I'm not really sure how useful such a non-transactional object can
> be for other purposes: the user should take into account that the
> transaction may fail and the value of the session variable be inconsistent
> as a result. Sometimes it may not matter, but if it matters there is no
> easy way around the fact.
>

I agree, so it should be well documented to be clear, what is possible,
what not, and to be correct expectations.

This feature has two (three) purposes

1. global variables for PL language
2. holding some session based informations, that can be used in security
definer functions.
3. Because it is not transactional, then it allows write operation on read
only hot stand by instances.

It is not transactional safe, but it is secure in sense a possibility to
set a access rights. I understand, so some patterns are not possible, but
when you need hold some keys per session, then this simply solution can be
good enough. The variables are clean after session end.

I think it is possible for some more complex patterns, but then developer
should be smarter, and should to enocode state result to content of
variable. There is strong benefit - read write access to variables is very
cheap and fast.

I invite any patch to doc (or everywhere) with explanation and about
possible risks.

Regards

Pavel





> --
> Fabien.
>

  76. Re: [HACKERS] proposal: schema variables

    Fabien COELHO <coelho@cri.ensmp.fr> — 2018-08-23T08:17:28Z

    Hello Pavel,

> 2. holding some session based informations, that can be used in security
> definer functions.

Hmmm, I see our disagreement. My point is that this feature is *NOT* fit 
for security-related uses because if the transaction fails the variable 
would keep the value it had if the transaction had not failed...

> 3. Because it is not transactional, then it allows write operation on read

> It is not transactional safe, but it is secure in sense a possibility to
> set a access rights.

This is a misleading play on words. It is secure wrt to access right, but 
unsecure wrt security purposes which is the only point for having such a 
feature in the first place.

> I understand, so some patterns are not possible, but when you need hold 
> some keys per session, then this simply solution can be good enough.

Security vs "good enough in some cases" looks bad to me.

> I think it is possible for some more complex patterns,

I'm not sure of any pattern which would be correct wrt security if it 
depends on the success of a transaction.

> but then developer should be smarter, and should to enocode state result 
> to content of variable.

I do not see how the developer can be smarter if they need a transactional 
for security but they do not have it.

> There is strong benefit - read write access to variables is very cheap 
> and fast.

I'd say that PostgreSQL is about "ACID & security" first, not "cheap & 
fast" first.

> I invite any patch to doc (or everywhere) with explanation and about
> possible risks.

Hmmm... You are the one proposing the feature...

Here is something, thanks for adjusting it to the syntax you are proposing 
and inserting it where appropriate. Possibly in the corresponding CREATE 
doc?

"""
<caution>
<par>
Beware that session variables are not transactional.
This is a concern in a security context where the variable must be set to
some trusted value depending on the success of the transaction:
if the transaction fails, the variable keeps its trusted value unduly.
</par>

<par>
For instance, the following pattern does NOT work:

<programlisting>
CREATE USER auditer;
SET ROLE auditer;
CREATE SESSION VARIABLE is_audited BOOLEAN DEFAULT FALSE ...;
-- ensure that only "auditer" can write "is_audited":
REVOKE ... ON SESSION VARIABLE is_audited FROM ...;

-- create an audit function
CREATE FUNCTION audit_session(...) SECURITY DEFINER AS $$
   -- record the session and checks in some place...
   -- then tell it was done:
   LET is_audited = TRUE;
$$;

-- the intention is that other security definier functions can check that
-- the session is audited by checking on "is_audited", eg:
CREATE FUNCTION only_for_audited(...) SECURITY DEFINER AS $$
   IF NOT is_audited THEN RAISE "security error";
   -- do protected stuff here.
$$;
</programlisting>

The above pattern can be attacked with the following approach:
<programlisting>
BEGIN;
SELECT audit_session(...);
-- success, "is_audited" is set...
ROLLBACK;
-- the audit login has been reverted, but "is_audited" retains its value.

-- any subsequent operation believes wrongly that the session is audited,
-- but its logging has really been removed by the ROLLBACK.

-- ok but should not:
SELECT only_for_audited(...);
</programlisting>
</par>
</caution>
"""


For the record, I'm "-1" on this feature as proposed, for what it's worth, 
because of the misleading security implications. This feature would just 
help people have their security wrong.


-- 
Fabien.

  77. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2018-08-23T08:44:10Z

    2018-08-23 10:17 GMT+02:00 Fabien COELHO <coelho@cri.ensmp.fr>:

>
> Hello Pavel,
>
> 2. holding some session based informations, that can be used in security
>> definer functions.
>>
>
> Hmmm, I see our disagreement. My point is that this feature is *NOT* fit
> for security-related uses because if the transaction fails the variable
> would keep the value it had if the transaction had not failed...
>
> 3. Because it is not transactional, then it allows write operation on read
>>
>
> It is not transactional safe, but it is secure in sense a possibility to
>> set a access rights.
>>
>
> This is a misleading play on words. It is secure wrt to access right, but
> unsecure wrt security purposes which is the only point for having such a
> feature in the first place.
>
> I understand, so some patterns are not possible, but when you need hold
>> some keys per session, then this simply solution can be good enough.
>>
>
> Security vs "good enough in some cases" looks bad to me.
>

We don't find a agreement, because you are concentrated on transation, me
on session. And we have different expectations.


> I think it is possible for some more complex patterns,
>>
>
> I'm not sure of any pattern which would be correct wrt security if it
> depends on the success of a transaction.
>
> but then developer should be smarter, and should to enocode state result
>> to content of variable.
>>
>
> I do not see how the developer can be smarter if they need a transactional
> for security but they do not have it.
>
> There is strong benefit - read write access to variables is very cheap and
>> fast.
>>
>
> I'd say that PostgreSQL is about "ACID & security" first, not "cheap &
> fast" first.
>
> I invite any patch to doc (or everywhere) with explanation and about
>> possible risks.
>>
>
> Hmmm... You are the one proposing the feature...
>
> Here is something, thanks for adjusting it to the syntax you are proposing
> and inserting it where appropriate. Possibly in the corresponding CREATE
> doc?
>
> """
> <caution>
> <par>
> Beware that session variables are not transactional.
> This is a concern in a security context where the variable must be set to
> some trusted value depending on the success of the transaction:
> if the transaction fails, the variable keeps its trusted value unduly.
> </par>
>
> <par>
> For instance, the following pattern does NOT work:
>
> <programlisting>
> CREATE USER auditer;
> SET ROLE auditer;
> CREATE SESSION VARIABLE is_audited BOOLEAN DEFAULT FALSE ...;
> -- ensure that only "auditer" can write "is_audited":
> REVOKE ... ON SESSION VARIABLE is_audited FROM ...;
>
> -- create an audit function
> CREATE FUNCTION audit_session(...) SECURITY DEFINER AS $$
>   -- record the session and checks in some place...
>   -- then tell it was done:
>   LET is_audited = TRUE;
> $$;
>
> -- the intention is that other security definier functions can check that
> -- the session is audited by checking on "is_audited", eg:
> CREATE FUNCTION only_for_audited(...) SECURITY DEFINER AS $$
>   IF NOT is_audited THEN RAISE "security error";
>   -- do protected stuff here.
> $$;
> </programlisting>
>
> The above pattern can be attacked with the following approach:
> <programlisting>
> BEGIN;
> SELECT audit_session(...);
> -- success, "is_audited" is set...
> ROLLBACK;
> -- the audit login has been reverted, but "is_audited" retains its value.
>
> -- any subsequent operation believes wrongly that the session is audited,
> -- but its logging has really been removed by the ROLLBACK.
>
> -- ok but should not:
> SELECT only_for_audited(...);
> </programlisting>
> </par>
> </caution>
> """
>
>
It is good example of not supported pattern. It is not designed for this.
I'll merge this doc.

Note: I am not sure, if I have all relations to described issue, but if I
understand well, then solution can be reset on transaction end, maybe reset
on rollback. This is solvable, I'll look how it is complex.


>
> For the record, I'm "-1" on this feature as proposed, for what it's worth,
> because of the misleading security implications. This feature would just
> help people have their security wrong.
>

I respect your opinion - and I hope so integration of your proposed doc is
good warning for users that would to use not transactional variable like
transactional source.

Regards

Pavel


>
> --
> Fabien.
>
>

  78. Re: [HACKERS] proposal: schema variables

    Fabien COELHO <coelho@cri.ensmp.fr> — 2018-08-23T09:46:32Z

    >> Security vs "good enough in some cases" looks bad to me.
>
> We don't find a agreement, because you are concentrated on transation, 
> me on session. And we have different expectations.

I do not understand your point, as usual. I raise a factual issue about 
security, and you do not answer how this can be solved with your proposal, 
but appeal to argument of authority and declare your "strong opinion".

I do not see any intrinsic opposition between having session objects and 
transactions. Nothing prevents a session object to be transactional beyond 
your willingness that it should not be.

Now, I do expect all PostgreSQL features to be security-wise, whatever 
their scope.

I do not think that security should be traded for "cheap & fast", esp as 
the sole use case for a feature is a security pattern that cannot be 
implemented securely with it. This appears to me as a huge contradiction, 
hence my opposition against this feature as proposed.

The good news is that I'm a nobody: if a committer is happy with your 
patch, it will get committed, you do not need my approval.

-- 
Fabien.

  79. Re: [HACKERS] proposal: schema variables

    Pavel Luzanov <p.luzanov@postgrespro.ru> — 2018-08-23T12:39:07Z

    On 23.08.2018 12:46, Fabien COELHO wrote:
> I do not understand your point, as usual. I raise a factual issue 
> about security, and you do not answer how this can be solved with your 
> proposal, but appeal to argument of authority and declare your "strong 
> opinion".
>
> I do not see any intrinsic opposition between having session objects 
> and transactions. Nothing prevents a session object to be 
> transactional beyond your willingness that it should not be.
>
> Now, I do expect all PostgreSQL features to be security-wise, whatever 
> their scope.
>
> I do not think that security should be traded for "cheap & fast", esp 
> as the sole use case for a feature is a security pattern that cannot 
> be implemented securely with it. This appears to me as a huge 
> contradiction, hence my opposition against this feature as proposed.

I can't to agree with your position.

Consider this example.
I want to record some inappropriate user actions to audit table and 
rollback transaction.
But aborting transaction will also abort record to audit table.
So, do not use tables, becouse they have security implications.

This is very similar to your approach.

Schema variables is a very needed and important feature, but for others 
purposes.

-----
Pavel Luzanov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

  80. Re: [HACKERS] proposal: schema variables

    Fabien COELHO <coelho@cri.ensmp.fr> — 2018-08-29T18:10:02Z

    Hello Pavel L.

>> I do not understand your point, as usual. I raise a factual issue about 
>> security, and you do not answer how this can be solved with your proposal, 
>> but appeal to argument of authority and declare your "strong opinion".
>> 
>> I do not see any intrinsic opposition between having session objects and 
>> transactions. Nothing prevents a session object to be transactional beyond 
>> your willingness that it should not be.
>> 
>> Now, I do expect all PostgreSQL features to be security-wise, whatever 
>> their scope.
>> 
>> I do not think that security should be traded for "cheap & fast", esp as 
>> the sole use case for a feature is a security pattern that cannot be 
>> implemented securely with it. This appears to me as a huge contradiction, 
>> hence my opposition against this feature as proposed.
>
> I can't to agree with your position.
>
> Consider this example. I want to record some inappropriate user actions 
> to audit table and rollback transaction. But aborting transaction will 
> also abort record to audit table. So, do not use tables, becouse they 
> have security implications.

Indeed, you cannot record a transaction failure from a transaction.

> This is very similar to your approach.

I understand that your point is that some use case could require a non 
transactional session variable. I'm not sure of how the use case would go 
on though, because once the "attacker" disconnects, the session variable 
disappears, so it does not record that there was a problem.

Anyway, I'm not against having session variables per se. I'm argumenting 
that there is a good case to have them transactional by default, and 
possibly an option to have them non transactional if this is really needed 
by some use case to provide.

The only use case put forward by Pavel S. is the security audit one 
where a session variable stores that audit checks have been performed, 
which AFAICS cannot be implemented securely with the proposed non 
transactional session variables.

-- 
Fabien.

  81. Re: [HACKERS] proposal: schema variables

    Dean Rasheed <dean.a.rasheed@gmail.com> — 2018-09-04T07:21:07Z

    AFAICS this patch does nothing to consider parallel safety -- that is,
as things stand, a variable is allowed in a query that may be
parallelised, but its value is not copied to workers, leading to
incorrect results. For example:

create table foo(a int);
insert into foo select * from generate_series(1,1000000);
create variable zero int;
let zero = 0;

explain (costs off) select count(*) from foo where a%10 = zero;

                  QUERY PLAN
-----------------------------------------------
 Finalize Aggregate
   ->  Gather
         Workers Planned: 2
         ->  Partial Aggregate
               ->  Parallel Seq Scan on foo
                     Filter: ((a % 10) = zero)
(6 rows)

select count(*) from foo where a%10 = zero;

 count
-------
 38037    -- Different random result each time, should be 100,000
(1 row)

Thoughts?

Regards,
Dean

  82. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2018-09-04T13:00:51Z

    Hi

2018-09-04 9:21 GMT+02:00 Dean Rasheed <dean.a.rasheed@gmail.com>:

> AFAICS this patch does nothing to consider parallel safety -- that is,
> as things stand, a variable is allowed in a query that may be
> parallelised, but its value is not copied to workers, leading to
> incorrect results. For example:
>
> create table foo(a int);
> insert into foo select * from generate_series(1,1000000);
> create variable zero int;
> let zero = 0;
>
> explain (costs off) select count(*) from foo where a%10 = zero;
>
>                   QUERY PLAN
> -----------------------------------------------
>  Finalize Aggregate
>    ->  Gather
>          Workers Planned: 2
>          ->  Partial Aggregate
>                ->  Parallel Seq Scan on foo
>                      Filter: ((a % 10) = zero)
> (6 rows)
>
> select count(*) from foo where a%10 = zero;
>
>  count
> -------
>  38037    -- Different random result each time, should be 100,000
> (1 row)
>
> Thoughts?
>

The query use copy of values of variables now - but unfortunately, these
values are not passed to workers.  Should be fixed.

Thank you for test case.

Pavel


> Regards,
> Dean
>

  83. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2018-09-06T08:30:04Z

    Hi

here is updated patch - I wrote some transactional support

I am not sure how these new features are understandable and if these
features does it better or not.

There are possibility to reset to default value when

a) any transaction is finished - the scope of value is limited by
transaction

CREATE VARIABLE foo int ON TRANSACTION END RESET;

b) when transaction finished by rollback

CREATE VARIABLE foo int ON ROLLBACK RESET

Now, when I am thinking about it, the @b is simple, but not too practical -
when some fails, then we lost a value (any transaction inside session can
fails). The @a has sense - the behave is global value (what is not possible
in Postgres now), but this value is destroyed by any unhandled exceptions,
and it cleaned on transaction end. The @b is just for information and for
discussion, but I'll remove it - because it is obscure.

The open question is syntax. PostgreSQL has already ON COMMIT xxx . It is
little bit unclean, because it has semantic "on transaction end", but if I
didn't implement @b, then ON COMMIT syntax can be used.

Regards

Pavel

  84. Re: [HACKERS] proposal: schema variables

    Fabien COELHO <coelho@cri.ensmp.fr> — 2018-09-07T12:34:04Z

    Hello Pavel,

> here is updated patch - I wrote some transactional support
>
> I am not sure how these new features are understandable and if these
> features does it better or not.

> There are possibility to reset to default value when
>
> a) any transaction is finished - the scope of value is limited by
> transaction
>
> CREATE VARIABLE foo int ON TRANSACTION END RESET;

With this option I understand that it is a "within a transactionnal" 
variable, i.e. when the transaction ends, whether commit or rollback, the 
variable is reset to a default variable. It is not really a "session" 
variable anymore, each transaction has its own value.

  -- begin session
  -- foo has default value, eg NULL
  BEGIN;
     LET foo = 1;
  COMMIT/ROLLBACK;
  -- foo has default value again, NULL

> b) when transaction finished by rollback
>
> CREATE VARIABLE foo int ON ROLLBACK RESET

That is a little bit safer and you are back to a SESSION-scope variable, 
which is reset to the default value if the (any) transaction fails?

   -- begin session
   -- foo has default value, eg NULL
   BEGIN;
     LET foo = 1;
   COMMIT;
   -- foo has value 1
   BEGIN;
     -- foo has value 1...
   ROLLBACK;
   -- foo has value NULL

c) A more logical (from a transactional point of view - but not necessary 
simple to implement, I do not know) feature/variant would be to reset the 
value to the one it had at the beginning of the transaction, which is not 
necessarily the default.

   -- begin session
   -- foo has default value, eg NULL
   BEGIN;
     LET foo = 1;
   COMMIT;
   -- foo has value 1
   BEGIN;
     LET foo = 2; (*)
     -- foo has value 2
   ROLLBACK;
   -- foo has value 1 back, change (*) has been reverted

> Now, when I am thinking about it, the @b is simple, but not too practical -
> when some fails, then we lost a value (any transaction inside session can
> fails).

Indeed.

> The @a has sense - the behave is global value (what is not possible
> in Postgres now), but this value is destroyed by any unhandled exceptions,
> and it cleaned on transaction end. The @b is just for information and for
> discussion, but I'll remove it - because it is obscure.

Indeed.

> The open question is syntax. PostgreSQL has already ON COMMIT xxx . It is
> little bit unclean, because it has semantic "on transaction end", but if I
> didn't implement @b, then ON COMMIT syntax can be used.

I was more arguing on the third (c) option, i.e. on rollback the value is 
reverted to its value at the beginning of the rollbacked transaction.

At the minimum, ISTM that option (b) is enough to implement the audit 
pattern, but it would mean that any session which has a rollback, for any 
reason (deadlock, serialization...), would have to be reinitialized, which 
would be a drawback.

The to options could be non-transactional session variables "ON ROLLBACK 
DO NOT RESET/DO NOTHING", and somehow transactional session variables "ON 
ROLLBACK RESET TO DEFAULT" (b) or "ON ROLLBACK RESET TO INITIAL" (c).

-- 
Fabien.

  85. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2018-09-07T14:28:00Z

    2018-09-07 14:34 GMT+02:00 Fabien COELHO <coelho@cri.ensmp.fr>:

>
> Hello Pavel,
>
> here is updated patch - I wrote some transactional support
>>
>> I am not sure how these new features are understandable and if these
>> features does it better or not.
>>
>
> There are possibility to reset to default value when
>>
>> a) any transaction is finished - the scope of value is limited by
>> transaction
>>
>> CREATE VARIABLE foo int ON TRANSACTION END RESET;
>>
>
> With this option I understand that it is a "within a transactionnal"
> variable, i.e. when the transaction ends, whether commit or rollback, the
> variable is reset to a default variable. It is not really a "session"
> variable anymore, each transaction has its own value.
>

yes, the correct name should be "schema variable with transaction scope". I
think it can be useful like short life global variable. These variables can
works like transaction caches.


>  -- begin session
>  -- foo has default value, eg NULL
>  BEGIN;
>     LET foo = 1;
>  COMMIT/ROLLBACK;
>  -- foo has default value again, NULL
>
> b) when transaction finished by rollback
>>
>> CREATE VARIABLE foo int ON ROLLBACK RESET
>>
>
> That is a little bit safer and you are back to a SESSION-scope variable,
> which is reset to the default value if the (any) transaction fails?
>
>   -- begin session
>   -- foo has default value, eg NULL
>   BEGIN;
>     LET foo = 1;
>   COMMIT;
>   -- foo has value 1
>   BEGIN;
>     -- foo has value 1...
>   ROLLBACK;
>   -- foo has value NULL
>
> c) A more logical (from a transactional point of view - but not necessary
> simple to implement, I do not know) feature/variant would be to reset the
> value to the one it had at the beginning of the transaction, which is not
> necessarily the default.
>
>   -- begin session
>   -- foo has default value, eg NULL
>   BEGIN;
>     LET foo = 1;
>   COMMIT;
>   -- foo has value 1
>   BEGIN;
>     LET foo = 2; (*)
>     -- foo has value 2
>   ROLLBACK;
>   -- foo has value 1 back, change (*) has been reverted
>
> Now, when I am thinking about it, the @b is simple, but not too practical -
>> when some fails, then we lost a value (any transaction inside session can
>> fails).
>>
>
> Indeed.
>
> The @a has sense - the behave is global value (what is not possible
>> in Postgres now), but this value is destroyed by any unhandled exceptions,
>> and it cleaned on transaction end. The @b is just for information and for
>> discussion, but I'll remove it - because it is obscure.
>>
>
> Indeed.
>
> The open question is syntax. PostgreSQL has already ON COMMIT xxx . It is
>> little bit unclean, because it has semantic "on transaction end", but if I
>> didn't implement @b, then ON COMMIT syntax can be used.
>>
>
> I was more arguing on the third (c) option, i.e. on rollback the value is
> reverted to its value at the beginning of the rollbacked transaction.
>

> At the minimum, ISTM that option (b) is enough to implement the audit
> pattern, but it would mean that any session which has a rollback, for any
> reason (deadlock, serialization...), would have to be reinitialized, which
> would be a drawback.
>
> The to options could be non-transactional session variables "ON ROLLBACK
> DO NOT RESET/DO NOTHING", and somehow transactional session variables "ON
> ROLLBACK RESET TO DEFAULT" (b) or "ON ROLLBACK RESET TO INITIAL" (c).
>

@b is hardly understandable for not trained people, because any rollback in
session does reset. But people expecting @c, or some near @c.

I understand so you talked about @c. Now I think so it is possible to
implement, but it is not trivial. The transactional behave have to
calculate not only with transactions, but with SAVEPOINTS and ROLLBACK TO
savepoints. On second hand, the implementation will be relatively compact.

I'll hold it in my memory, but there are harder issues (support for
parallelism).

Regards

Pavel



> --
> Fabien.
>
>

  86. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2018-09-14T21:31:16Z

    út 4. 9. 2018 v 9:21 odesílatel Dean Rasheed <dean.a.rasheed@gmail.com>
napsal:

> AFAICS this patch does nothing to consider parallel safety -- that is,
> as things stand, a variable is allowed in a query that may be
> parallelised, but its value is not copied to workers, leading to
> incorrect results. For example:
>
> create table foo(a int);
> insert into foo select * from generate_series(1,1000000);
> create variable zero int;
> let zero = 0;
>
> explain (costs off) select count(*) from foo where a%10 = zero;
>
>                   QUERY PLAN
> -----------------------------------------------
>  Finalize Aggregate
>    ->  Gather
>          Workers Planned: 2
>          ->  Partial Aggregate
>                ->  Parallel Seq Scan on foo
>                      Filter: ((a % 10) = zero)
> (6 rows)
>
> select count(*) from foo where a%10 = zero;
>
>  count
> -------
>  38037    -- Different random result each time, should be 100,000
> (1 row)
>
> Thoughts?
>

This issue should be fixed in attached patch (and more others).

The code is more cleaner now, there are more tests, and documentation is
mostly complete. I am sorry - my English is not good.
New features:

o ON COMMIT DROP and ON TRANSACTION END RESET -- remove temp variable on
commit, reset variable on transaction end (commit, rollback)
o LET var = DEFAULT -- reset specified variable

Regards

Pavel


> Regards,
> Dean
>

  87. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2018-09-15T16:06:59Z

    > The code is more cleaner now, there are more tests, and documentation is
> mostly complete. I am sorry - my English is not good.
> New features:
>
> o ON COMMIT DROP and ON TRANSACTION END RESET -- remove temp variable on
> commit, reset variable on transaction end (commit, rollback)
> o LET var = DEFAULT -- reset specified variable
>
>
fix some forgotten warnings and dependency issue
few more tests

Regards

Pavel


> Regards
>
> Pavel
>
>
>> Regards,
>> Dean
>>
>

  88. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2018-09-17T19:46:01Z

    so 15. 9. 2018 v 18:06 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
napsal:

>
>
>
>
>> The code is more cleaner now, there are more tests, and documentation is
>> mostly complete. I am sorry - my English is not good.
>> New features:
>>
>> o ON COMMIT DROP and ON TRANSACTION END RESET -- remove temp variable on
>> commit, reset variable on transaction end (commit, rollback)
>> o LET var = DEFAULT -- reset specified variable
>>
>>
> fix some forgotten warnings and dependency issue
> few more tests
>
>
new update:

o support NOT NULL check
o implementation limited transaction variables - these variables doesn't
respects subtransactions(this is much more complex), drop variable drops
content although the drop can be reverted (maybe this limit will be
removed).

CREATE TRANSACTION VARIABLE fx AS int;

LET fx = 10;
BEGIN
  LET fx = 20;
ROLLBACK;

SELECT fx;

Regards

Pavel


> Regards
>
> Pavel
>
>
>> Regards
>>
>> Pavel
>>
>>
>>> Regards,
>>> Dean
>>>
>>

  89. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2018-09-19T08:30:31Z

    Hi

new update:

I fixed pg_restore, and I cleaned a code related to transaction processing

There should be a full functionality now.

Regards

Pavel

  90. Re: [HACKERS] proposal: schema variables

    Artur Zakirov <a.zakirov@postgrespro.ru> — 2018-09-19T11:23:06Z

    Hello,

On Wed, Sep 19, 2018 at 10:30:31AM +0200, Pavel Stehule wrote:
> Hi
> 
> new update:
> 
> I fixed pg_restore, and I cleaned a code related to transaction processing
> 
> There should be a full functionality now.

I reviewed a little bit the patch. I have a few comments.

> <title><structname>pg_views</structname> Columns</title>

I think there is a typo here. It should be "pg_variable".

> GRANT { READ | WRITE | ALL [ PRIVILEGES ] }

Shouldn't we use here GRANT { SELECT | LET | ... } syntax for the
constistency. Same for REVOKE. I'm not experienced syntax developer
though. But we use SELECT and LET commands when working with variables.
So we should GRANT and REVOKE priveleges for this commands.

> [ { ON COMMIT DROP | ON TRANSACTION END RESET } ]

I think we may join them and have the syntax { ON COMMIT DROP | RESET }
to get more simpler syntax. If we create a variable with ON COMMIT
DROP, PostgreSQL will drop it regardless of whether transaction was
committed or rollbacked:

=# ...
=# begin;
=# create variable int1 int on commit drop;
=# rollback;
=# -- There is no variable int1

CREATE TABLE syntax has similar options [1]. ON COMMIT controls
the behaviour of temporary tables at the end a transaction block,
whether it was committed or rollbacked. But I'm not sure is this a good
example of precedence.

> -	ONCOMMIT_DROP				/* ON COMMIT DROP */
> +	ONCOMMIT_DROP,				/* ON COMMIT DROP */
> } OnCommitAction;

There is the extra comma here after ONCOMMIT_DROP.

1 - https://www.postgresql.org/docs/current/static/sql-createtable.html

-- 
Arthur Zakirov
Postgres Professional: http://www.postgrespro.com
Russian Postgres Company

  91. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2018-09-19T12:08:04Z

    Hi
st 19. 9. 2018 v 13:23 odesílatel Arthur Zakirov <a.zakirov@postgrespro.ru>
napsal:

> Hello,
>
> On Wed, Sep 19, 2018 at 10:30:31AM +0200, Pavel Stehule wrote:
> > Hi
> >
> > new update:
> >
> > I fixed pg_restore, and I cleaned a code related to transaction
> processing
> >
> > There should be a full functionality now.
>
> I reviewed a little bit the patch. I have a few comments.
>
> > <title><structname>pg_views</structname> Columns</title>
>
> I think there is a typo here. It should be "pg_variable".
>

I'll fix it.

>
> > GRANT { READ | WRITE | ALL [ PRIVILEGES ] }
>
> Shouldn't we use here GRANT { SELECT | LET | ... } syntax for the
> constistency. Same for REVOKE. I'm not experienced syntax developer
> though. But we use SELECT and LET commands when working with variables.
> So we should GRANT and REVOKE priveleges for this commands.
>

I understand to your proposal, - and I have not strong opinion. Originally
I proposed {SELECT|UPDATE), but some people prefer {READ|WRITE}. Now I
prefer Peter's proposal (what is implemented now) - READ|WRITE, because it
is very illustrative - and the mentioned difference is good because the
variables are not tables (by default), are not persistent, so different
rights are good for me. I see "GRANT LET" like very low clear, because
nobody knows what LET command does. Unfortunately we cannot to use standard
"SET" command, because it is used in Postgres for different purpose.
READ|WRITE are totally clear, and for user it is another signal so
variables are different than tables (so it is not one row table).

I prefer current state, but if common opinion will be different, I have not
problem to change it.


> > [ { ON COMMIT DROP | ON TRANSACTION END RESET } ]
>
> I think we may join them and have the syntax { ON COMMIT DROP | RESET }
> to get more simpler syntax. If we create a variable with ON COMMIT
> DROP, PostgreSQL will drop it regardless of whether transaction was
> committed or rollbacked:
>

I though about it too. I'll try to explain my idea. Originally I was
surprised so postgres uses "ON COMMIT" syntax, but in documentation is used
term "at transaction end". But it has some sense. ON COMMIT DROP is allowed
only for temporary tables and ON COMMIT DELETE ROWS is allowed for tables.
With these clauses the PostgreSQL is more aggressive in cleaning. It
doesn't need to calculate with rollback, because the rollback does cleaning
by self. So syntax "ON COMMIT" is fully correct it is related only for
commit event. It has not sense on rollback event (and doesn't change a
behave on rollback event).

The content of variables is not transactional (by default). It is not
destroyed by rollback. So I have to calculate with rollback too. So the
most correct syntax should be "ON COMMIT ON ROLLBACK RESET" what is little
bit messy and I used "ON TRANSACTION END". It should be signal, so this
event is effective on rollback event and it is valid for not transaction
variable. This logic is not valid to transactional variables, where ON
COMMIT RESET has sense. But this behave is not default and then I prefer
more generic syntax.

Again I have not a problem to change it, but I am thinking so current
design is logically correct.


> =# ...
> =# begin;
> =# create variable int1 int on commit drop;
> =# rollback;
> =# -- There is no variable int1
>
>
PostgreSQL catalog is transactional (where the metadata is stored), so when
I am working with metadata, then I use ON COMMIT syntax, because the behave
of  ON ROLLBACK cannot be changed.

So I see two different cases - work with catalog (what is transactional)
and work with variable value, what is (like other variables in programming
languages) not transactional. "ON TRANSACTION END RESET" means - does reset
on any transaction end.

I hope so I explained it cleanly - if not, please, ask.

CREATE TABLE syntax has similar options [1]. ON COMMIT controls
> the behaviour of temporary tables at the end a transaction block,
> whether it was committed or rollbacked. But I'm not sure is this a good
> example of precedence.
>
> > -     ONCOMMIT_DROP                           /* ON COMMIT DROP */
> > +     ONCOMMIT_DROP,                          /* ON COMMIT DROP */
> > } OnCommitAction;
>
> There is the extra comma here after ONCOMMIT_DROP.
>

I'll fix it.

Regards

Pavel

>
> 1 - https://www.postgresql.org/docs/current/static/sql-createtable.html
>
> --
> Arthur Zakirov
> Postgres Professional: http://www.postgrespro.com
> Russian Postgres Company
>

  92. Re: [HACKERS] proposal: schema variables

    Artur Zakirov <a.zakirov@postgrespro.ru> — 2018-09-19T12:53:25Z

    On Wed, Sep 19, 2018 at 02:08:04PM +0200, Pavel Stehule wrote:
> Unfortunately we cannot to use standard
> "SET" command, because it is used in Postgres for different purpose.
> READ|WRITE are totally clear, and for user it is another signal so
> variables are different than tables (so it is not one row table).
> 
> I prefer current state, but if common opinion will be different, I have not
> problem to change it.

I see. I grepped the thread before writhing this but somehow missed the
discussion.

> The content of variables is not transactional (by default). It is not
> destroyed by rollback. So I have to calculate with rollback too. So the
> most correct syntax should be "ON COMMIT ON ROLLBACK RESET" what is little
> bit messy and I used "ON TRANSACTION END". It should be signal, so this
> event is effective on rollback event and it is valid for not transaction
> variable. This logic is not valid to transactional variables, where ON
> COMMIT RESET has sense. But this behave is not default and then I prefer
> more generic syntax.
> ...
> So I see two different cases - work with catalog (what is transactional)
> and work with variable value, what is (like other variables in programming
> languages) not transactional. "ON TRANSACTION END RESET" means - does reset
> on any transaction end.
> 
> I hope so I explained it cleanly - if not, please, ask.

I understood what you mean, thank you. I thought that
{ ON COMMIT DROP | ON TRANSACTION END RESET } parameters are used only
for transactional variables in the first place. But is there any sense
in using this parameters with non-transactional variables? That is when
we create non-transactional variable we don't want that the variable
will rollback or reset its value after the end of a transaction.

-- 
Arthur Zakirov
Postgres Professional: http://www.postgrespro.com
Russian Postgres Company

  93. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2018-09-19T14:36:40Z

    st 19. 9. 2018 v 14:53 odesílatel Arthur Zakirov <a.zakirov@postgrespro.ru>
napsal:

> On Wed, Sep 19, 2018 at 02:08:04PM +0200, Pavel Stehule wrote:
> > Unfortunately we cannot to use standard
> > "SET" command, because it is used in Postgres for different purpose.
> > READ|WRITE are totally clear, and for user it is another signal so
> > variables are different than tables (so it is not one row table).
> >
> > I prefer current state, but if common opinion will be different, I have
> not
> > problem to change it.
>
> I see. I grepped the thread before writhing this but somehow missed the
> discussion.
>
> > The content of variables is not transactional (by default). It is not
> > destroyed by rollback. So I have to calculate with rollback too. So the
> > most correct syntax should be "ON COMMIT ON ROLLBACK RESET" what is
> little
> > bit messy and I used "ON TRANSACTION END". It should be signal, so this
> > event is effective on rollback event and it is valid for not transaction
> > variable. This logic is not valid to transactional variables, where ON
> > COMMIT RESET has sense. But this behave is not default and then I prefer
> > more generic syntax.
> > ...
> > So I see two different cases - work with catalog (what is transactional)
> > and work with variable value, what is (like other variables in
> programming
> > languages) not transactional. "ON TRANSACTION END RESET" means - does
> reset
> > on any transaction end.
> >
> > I hope so I explained it cleanly - if not, please, ask.
>
> I understood what you mean, thank you. I thought that
> { ON COMMIT DROP | ON TRANSACTION END RESET } parameters are used only
> for transactional variables in the first place. But is there any sense
> in using this parameters with non-transactional variables? That is when
> we create non-transactional variable we don't want that the variable
> will rollback or reset its value after the end of a transaction.
>

ON COMMIT DROP is used only for temp variables (transaction or not
transaction). The purpose is same like for tables. Sometimes you can to
have object with shorter life than is session.

ON TRANSACTION END RESET has sense mainly for not transaction variables. I
see two use cases.

1. protect some sensitive data - on transaction end guaranteed reset and
cleaning on end transaction. So you can be sure, so variable is not
initialized (has default value), or you are inside transaction.

2. automatic initialization - ON TRANSACTION END RESET ensure so variable
is in init state for any transaction.

Both cases has sense for transaction or not transaction variables.

I am thinking so transaction life time for content has sense. Is cheaper to
reset variable than drop it (what ON COMMIT DROP does)

What do you think?

Pavel



> --
> Arthur Zakirov
> Postgres Professional: http://www.postgrespro.com
> Russian Postgres Company
>

  94. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2018-09-20T09:08:57Z

    Hi

st 19. 9. 2018 v 13:23 odesílatel Arthur Zakirov <a.zakirov@postgrespro.ru>
napsal:

> Hello,
>
> On Wed, Sep 19, 2018 at 10:30:31AM +0200, Pavel Stehule wrote:
> > Hi
> >
> > new update:
> >
> > I fixed pg_restore, and I cleaned a code related to transaction
> processing
> >
> > There should be a full functionality now.
>
> I reviewed a little bit the patch. I have a few comments.
>
> > <title><structname>pg_views</structname> Columns</title>
>
> I think there is a typo here. It should be "pg_variable".
>

fixed


> > -     ONCOMMIT_DROP                           /* ON COMMIT DROP */
> > +     ONCOMMIT_DROP,                          /* ON COMMIT DROP */
> > } OnCommitAction;
>
> There is the extra comma here after ONCOMMIT_DROP.
>

fixed

Thank you for comments

attached updated patch




> 1 - https://www.postgresql.org/docs/current/static/sql-createtable.html
>
> --
> Arthur Zakirov
> Postgres Professional: http://www.postgrespro.com
> Russian Postgres Company
>

  95. Re: [HACKERS] proposal: schema variables

    Artur Zakirov <a.zakirov@postgrespro.ru> — 2018-09-21T19:46:18Z

    On Wed, Sep 19, 2018 at 04:36:40PM +0200, Pavel Stehule wrote:
> ON COMMIT DROP is used only for temp variables (transaction or not
> transaction). The purpose is same like for tables. Sometimes you can to
> have object with shorter life than is session.
> 
> ON TRANSACTION END RESET has sense mainly for not transaction variables. I
> see two use cases.
> 
> 1. protect some sensitive data - on transaction end guaranteed reset and
> cleaning on end transaction. So you can be sure, so variable is not
> initialized (has default value), or you are inside transaction.
> 
> 2. automatic initialization - ON TRANSACTION END RESET ensure so variable
> is in init state for any transaction.
> 
> Both cases has sense for transaction or not transaction variables.
> 
> I am thinking so transaction life time for content has sense. Is cheaper to
> reset variable than drop it (what ON COMMIT DROP does)
> 
> What do you think?

Thanks, I understood the cases.

But I think there is more sense to use these options only with transactional
variables. It is more consistent and simple for me.

As a summary, it is 1 voice vs 1 voice :) So it is better to leave the
syntax as is without changes for now.

-- 
Arthur Zakirov
Postgres Professional: http://www.postgrespro.com
Russian Postgres Company

  96. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2018-09-22T03:35:45Z

    pá 21. 9. 2018 v 21:46 odesílatel Arthur Zakirov <a.zakirov@postgrespro.ru>
napsal:

> On Wed, Sep 19, 2018 at 04:36:40PM +0200, Pavel Stehule wrote:
> > ON COMMIT DROP is used only for temp variables (transaction or not
> > transaction). The purpose is same like for tables. Sometimes you can to
> > have object with shorter life than is session.
> >
> > ON TRANSACTION END RESET has sense mainly for not transaction variables.
> I
> > see two use cases.
> >
> > 1. protect some sensitive data - on transaction end guaranteed reset and
> > cleaning on end transaction. So you can be sure, so variable is not
> > initialized (has default value), or you are inside transaction.
> >
> > 2. automatic initialization - ON TRANSACTION END RESET ensure so variable
> > is in init state for any transaction.
> >
> > Both cases has sense for transaction or not transaction variables.
> >
> > I am thinking so transaction life time for content has sense. Is cheaper
> to
> > reset variable than drop it (what ON COMMIT DROP does)
> >
> > What do you think?
>
> Thanks, I understood the cases.
>
> But I think there is more sense to use these options only with
> transactional
> variables. It is more consistent and simple for me.
>

I agree so it can be hard to imagine - and if I return back to start
discussion about schema variables - it can be hard because it joins some
concepts - variables has persistent transactional metadata, but the content
is not transactional.

I don't think so the variability is a issue in this case. There is a lot of
examples, so lot of combinations are possible - global temp tables and
package variables (Oracle), local temp tables and local variables
(Postgres), session variables and memory tables (MSSQL). Any combination of
feature has cases where can be very practical and useful.

ON TRANSACTION END RESET can be useful, because we have not a session event
triggers (and in this moment I am not sure if it is necessary and practical
- their usage can be very fragile). But some work can do ON xxx clauses,
that should not to have negative impact on performance or fragility.

ON TRANSACTION END RESET ensure cleaned and initialized to default value
for any transaction. Other possibility is ON COMMAND END RESET (but I would
not to implement it now), ...


> As a summary, it is 1 voice vs 1 voice :) So it is better to leave the
> syntax as is without changes for now.
>

:) now is enough time to think about syntax. Some features can be removed
and returned back later, where this concept will be more absorbed.

Regards

Pavel


>
> --
> Arthur Zakirov
> Postgres Professional: http://www.postgrespro.com
> Russian Postgres Company
>

  97. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2018-09-22T06:00:04Z

    Hi

rebased against yesterday changes in tab-complete.c

Regards

Pavel

  98. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2018-09-29T08:34:40Z

    so 22. 9. 2018 v 8:00 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
napsal:

> Hi
>
> rebased against yesterday changes in tab-complete.c
>

rebased against last changes in master

Regards

Pavel



> Regards
>
> Pavel
>

  99. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2018-09-29T22:19:23Z

    so 29. 9. 2018 v 10:34 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
napsal:

>
>
> so 22. 9. 2018 v 8:00 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
> napsal:
>
>> Hi
>>
>> rebased against yesterday changes in tab-complete.c
>>
>
> rebased against last changes in master
>

+ using content of schema variable for estimation
+ subtransaction support

I hope so now, there are almost complete functionality. Please, check it.

Regards

Pavel


>
> Regards
>
> Pavel
>
>
>
>> Regards
>>
>> Pavel
>>
>

  100. Re: [HACKERS] proposal: schema variables

    Thomas Munro <thomas.munro@enterprisedb.com> — 2018-10-02T23:01:22Z

    On Sun, Sep 30, 2018 at 11:20 AM Pavel Stehule <pavel.stehule@gmail.com> wrote:
> I hope so now, there are almost complete functionality. Please, check it.

Hi Pavel,

FYI there is a regression test failure on Windows:

plpgsql ... FAILED

*** 4071,4077 ****
end;
$$ language plpgsql;
select stacked_diagnostics_test();
- NOTICE: sqlstate: 22012, message: division by zero, context:
[PL/pgSQL function zero_divide() line 4 at RETURN <- SQL statement
"SELECT zero_divide()" <- PL/pgSQL function stacked_diagnostics_test()
line 6 at PERFORM]
+ NOTICE: sqlstate: 42702, message: column reference "v" is ambiguous,
context: [PL/pgSQL function zero_divide() line 4 at RETURN <- SQL
statement "SELECT zero_divide()" <- PL/pgSQL function
stacked_diagnostics_test() line 6 at PERFORM]

https://ci.appveyor.com/project/postgresql-cfbot/postgresql/build/1.0.15234

-- 
Thomas Munro
http://www.enterprisedb.com

  101. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2018-10-05T01:34:14Z

    st 3. 10. 2018 v 1:01 odesílatel Thomas Munro <thomas.munro@enterprisedb.com>
napsal:

> On Sun, Sep 30, 2018 at 11:20 AM Pavel Stehule <pavel.stehule@gmail.com>
> wrote:
> > I hope so now, there are almost complete functionality. Please, check it.
>
> Hi Pavel,
>
> FYI there is a regression test failure on Windows:
>
> plpgsql ... FAILED
>
> *** 4071,4077 ****
> end;
> $$ language plpgsql;
> select stacked_diagnostics_test();
> - NOTICE: sqlstate: 22012, message: division by zero, context:
> [PL/pgSQL function zero_divide() line 4 at RETURN <- SQL statement
> "SELECT zero_divide()" <- PL/pgSQL function stacked_diagnostics_test()
> line 6 at PERFORM]
> + NOTICE: sqlstate: 42702, message: column reference "v" is ambiguous,
> context: [PL/pgSQL function zero_divide() line 4 at RETURN <- SQL
> statement "SELECT zero_divide()" <- PL/pgSQL function
> stacked_diagnostics_test() line 6 at PERFORM]
>
> https://ci.appveyor.com/project/postgresql-cfbot/postgresql/build/1.0.15234


please, check attached patch

Thank you for report

Pavel


>
> --
> Thomas Munro
> http://www.enterprisedb.com
>

  102. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2018-10-07T17:13:44Z

    Hi

ne 30. 9. 2018 v 0:19 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
napsal:

>
>
> so 29. 9. 2018 v 10:34 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
> napsal:
>
>>
>>
>> so 22. 9. 2018 v 8:00 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
>> napsal:
>>
>>> Hi
>>>
>>> rebased against yesterday changes in tab-complete.c
>>>
>>
>> rebased against last changes in master
>>
>
> + using content of schema variable for estimation
> + subtransaction support
>
> I hope so now, there are almost complete functionality. Please, check it.
>

new update

minor white space issue
one more regress test and 2 pg_dump tests

Regards

Pavel


>
> Regards
>
> Pavel
>
>
>>
>> Regards
>>
>> Pavel
>>
>>
>>
>>> Regards
>>>
>>> Pavel
>>>
>>

  103. Re: [HACKERS] proposal: schema variables

    Erik Rijkers <er@xs4all.nl> — 2018-10-23T12:50:18Z

    > [schema-variables-20181007-01.patch.gz]

Hi,

I tried to test your schema-variables patch but got stuck here instead 
(after applying succesfully on top of e6f5d1acc):

make[2]: *** No rule to make target 
'../../../src/include/catalog/pg_variable.h', needed by 'bki-stamp'.  
Stop.
make[1]: *** [submake-catalog-headers] Error 2
make[1]: *** Waiting for unfinished jobs....
make: *** [submake-generated-headers] Error 2
Makefile:141: recipe for target 'submake-catalog-headers' failed
src/Makefile.global:370: recipe for target 'submake-generated-headers' 
failed


thanks,

Erik Rijkers

  104. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2018-10-23T18:05:07Z

    Hi

út 23. 10. 2018 v 14:50 odesílatel Erik Rijkers <er@xs4all.nl> napsal:

> > [schema-variables-20181007-01.patch.gz]
>
> Hi,
>
> I tried to test your schema-variables patch but got stuck here instead
> (after applying succesfully on top of e6f5d1acc):
>
> make[2]: *** No rule to make target
> '../../../src/include/catalog/pg_variable.h', needed by 'bki-stamp'.
> Stop.
> make[1]: *** [submake-catalog-headers] Error 2
> make[1]: *** Waiting for unfinished jobs....
> make: *** [submake-generated-headers] Error 2
> Makefile:141: recipe for target 'submake-catalog-headers' failed
> src/Makefile.global:370: recipe for target 'submake-generated-headers'
> failed
>
>
Unfortunately previous patch was completely broken. I am sorry

Please, check this patch.

Regards

Pavel


> thanks,
>
> Erik Rijkers
>

  105. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2018-11-21T07:24:50Z

    Hi

just rebase


Regards

Pavel

  106. Re: [HACKERS] proposal: schema variables

    Dmitry Dolgov <9erthalion6@gmail.com> — 2018-11-30T23:17:21Z

    > On Wed, Nov 21, 2018 at 8:25 AM Pavel Stehule <pavel.stehule@gmail.com> wrote:
>
> just rebase

Thanks for working on this patch.

I'm a bit confused, but cfbot again says that there are some conflicts.
Probably they are the minor one, from src/bin/psql/help.c

For now I'm moving it to the next CF.

  107. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2018-12-01T06:32:41Z

    so 1. 12. 2018 v 0:16 odesílatel Dmitry Dolgov <9erthalion6@gmail.com>
napsal:

> > On Wed, Nov 21, 2018 at 8:25 AM Pavel Stehule <pavel.stehule@gmail.com>
> wrote:
> >
> > just rebase
>
> Thanks for working on this patch.
>
> I'm a bit confused, but cfbot again says that there are some conflicts.
> Probably they are the minor one, from src/bin/psql/help.c
>

rebased again

Regards

Pavel


> For now I'm moving it to the next CF.
>

  108. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2018-12-31T13:23:44Z

    st 21. 11. 2018 v 8:24 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
napsal:

> Hi
>
> just rebase
>
>
rebase

Regards

Pavel


>
> Regards
>
> Pavel
>

  109. Re: [HACKERS] proposal: schema variables

    Erik Rijkers <er@xs4all.nl> — 2018-12-31T15:40:24Z

    On 2018-12-31 14:23, Pavel Stehule wrote:
> st 21. 11. 2018 v 8:24 odesílatel Pavel Stehule 
> <pavel.stehule@gmail.com>

> [schema-variables-20181231-01.patch.gz]

Hi Pavel,

I gave this a quick try-out with the script I had from previous 
versions,
and found these two errors:

------------
drop schema if exists schema1 cascade;
create schema if not exists schema1;
drop variable if exists schema1.myvar1;  --> error 49
create variable schema1.myvar1 as text ;
select schema1.myvar1;
let schema1.myvar1 = 'variable value ""';
select schema1.myvar1;
alter variable schema1.myvar1 rename to myvar2;
select schema1.myvar2;
create variable schema1.myvar1 as text ;
let schema1.myvar1 = 'variable value ""';
select schema1.myvar1;
alter variable schema1.myvar1 rename to myvar2; --> error 4287
select schema1.myvar2;
------------


The above, ran with   psql -qXa  gives the following output:

drop schema if exists schema1 cascade;
create schema if not exists schema1;
drop variable if exists schema1.myvar1;  --> error 49
ERROR:  unrecognized object type: 49
create variable schema1.myvar1 as text ;
select schema1.myvar1;
  myvar1
--------

(1 row)

let schema1.myvar1 = 'variable value ""';
select schema1.myvar1;
       myvar1
-------------------
  variable value ""
(1 row)

alter variable schema1.myvar1 rename to myvar2;
select schema1.myvar2;
       myvar2
-------------------
  variable value ""
(1 row)

create variable schema1.myvar1 as text ;
let schema1.myvar1 = 'variable value ""';
select schema1.myvar1;
       myvar1
-------------------
  variable value ""
(1 row)

alter variable schema1.myvar1 rename to myvar2; --> error 4287
ERROR:  unsupported object class 4287
select schema1.myvar2;
       myvar2
-------------------
  variable value ""
(1 row)



thanks,


Erik Rijkers

  110. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2018-12-31T17:33:30Z

    Hi

po 31. 12. 2018 v 16:40 odesílatel Erik Rijkers <er@xs4all.nl> napsal:

> On 2018-12-31 14:23, Pavel Stehule wrote:
> > st 21. 11. 2018 v 8:24 odesílatel Pavel Stehule
> > <pavel.stehule@gmail.com>
>
> > [schema-variables-20181231-01.patch.gz]
>
> Hi Pavel,
>
> I gave this a quick try-out with the script I had from previous
> versions,
> and found these two errors:
>
> ------------
> drop schema if exists schema1 cascade;
> create schema if not exists schema1;
> drop variable if exists schema1.myvar1;  --> error 49
> create variable schema1.myvar1 as text ;
> select schema1.myvar1;
> let schema1.myvar1 = 'variable value ""';
> select schema1.myvar1;
> alter variable schema1.myvar1 rename to myvar2;
> select schema1.myvar2;
> create variable schema1.myvar1 as text ;
> let schema1.myvar1 = 'variable value ""';
> select schema1.myvar1;
> alter variable schema1.myvar1 rename to myvar2; --> error 4287
> select schema1.myvar2;
> ------------
>
>
> The above, ran with   psql -qXa  gives the following output:
>
> drop schema if exists schema1 cascade;
> create schema if not exists schema1;
> drop variable if exists schema1.myvar1;  --> error 49
> ERROR:  unrecognized object type: 49
> create variable schema1.myvar1 as text ;
> select schema1.myvar1;
>   myvar1
> --------
>
> (1 row)
>
> let schema1.myvar1 = 'variable value ""';
> select schema1.myvar1;
>        myvar1
> -------------------
>   variable value ""
> (1 row)
>
> alter variable schema1.myvar1 rename to myvar2;
> select schema1.myvar2;
>        myvar2
> -------------------
>   variable value ""
> (1 row)
>
> create variable schema1.myvar1 as text ;
> let schema1.myvar1 = 'variable value ""';
> select schema1.myvar1;
>        myvar1
> -------------------
>   variable value ""
> (1 row)
>
> alter variable schema1.myvar1 rename to myvar2; --> error 4287
> ERROR:  unsupported object class 4287
> select schema1.myvar2;
>        myvar2
> -------------------
>   variable value ""
> (1 row)
>
>
Should be fixed now.

Thank you for report

Pavel


>
> thanks,
>
>
> Erik Rijkers
>
>
>

  111. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2019-01-22T19:32:46Z

    Hi

fresh rebased patch, no other changes

Pavel

  112. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2019-01-30T16:34:24Z

    Hi

just rebase

Regards

Pavel

  113. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2019-01-31T11:49:27Z

    Hi

just rebase

regards

Pavel

  114. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2019-03-03T20:27:13Z

    Hi

čt 31. 1. 2019 v 12:49 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
napsal:

> Hi
>
> just rebase
>
> regards
>
> Pavel
>

rebase and fix compilation due changes related pg_dump

Regards

Pavel

  115. Re: Re: [HACKERS] proposal: schema variables

    David Steele <david@pgmasters.net> — 2019-03-07T06:52:16Z

    On 3/3/19 10:27 PM, Pavel Stehule wrote:
> 
> rebase and fix compilation due changes related pg_dump

This patch hasn't receive any review in a while and I'm not sure if 
that's because nobody is interested or the reviewers think it does not 
need any more review.

It seems to me that this patch as implemented does not quite satisfy any 
one.

I think we need to hear something from the reviewers soon or I'll push 
this patch to PG13 as Andres recommends [1].

-- 
-David
david@pgmasters.net

[1] 
https://www.postgresql.org/message-id/20190216054526.zss2cufdxfeudr4i%40alap3.anarazel.de

  116. Re: Re: [HACKERS] proposal: schema variables

    Fabien COELHO <coelho@cri.ensmp.fr> — 2019-03-07T08:10:47Z

    Hello David,

> This patch hasn't receive any review in a while and I'm not sure if that's 
> because nobody is interested or the reviewers think it does not need any more 
> review.
>
> It seems to me that this patch as implemented does not quite satisfy any one.
>
> I think we need to hear something from the reviewers soon or I'll push this 
> patch to PG13 as Andres recommends [1].

I have discussed the feature extensively with Pavel on the initial thread.

My strong opinion based on the underlying use case is that it that such 
session variables should be transactional by default, and Pavel strong 
opinion is that they should not, to be closer to Oracle comparable 
feature.

According to the documentation, the current implementation does provide a 
transactional feature. However, it is not the default behavior, so I'm in 
disagreement on a key feature, although I do really appreciate that Pavel
implemented the transactional behavior.

Otherwise, ISTM that they could be named "SESSION VARIABLE" because the 
variable only exists in memory, in a session, and we could thing of adding 
other kind of variables later on.

I do intend to review it in depth when it is transactional by default.

Anyway, the patch is non trivial and very large, so targetting v12 now is 
indeed out of reach.

-- 
Fabien.

  117. Re: Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2019-03-07T08:32:25Z

    čt 7. 3. 2019 v 9:10 odesílatel Fabien COELHO <coelho@cri.ensmp.fr> napsal:

>
> Hello David,
>
> > This patch hasn't receive any review in a while and I'm not sure if
> that's
> > because nobody is interested or the reviewers think it does not need any
> more
> > review.
> >
> > It seems to me that this patch as implemented does not quite satisfy any
> one.
> >
> > I think we need to hear something from the reviewers soon or I'll push
> this
> > patch to PG13 as Andres recommends [1].
>
> I have discussed the feature extensively with Pavel on the initial thread.
>
> My strong opinion based on the underlying use case is that it that such
> session variables should be transactional by default, and Pavel strong
> opinion is that they should not, to be closer to Oracle comparable
> feature.
>
> According to the documentation, the current implementation does provide a
> transactional feature. However, it is not the default behavior, so I'm in
> disagreement on a key feature, although I do really appreciate that Pavel
> implemented the transactional behavior.
>
> Otherwise, ISTM that they could be named "SESSION VARIABLE" because the
> variable only exists in memory, in a session, and we could thing of adding
> other kind of variables later on.
>
> I do intend to review it in depth when it is transactional by default.
>

I am sorry. I cannot to support this request. Variables are not
transactional. My opinion is strong in this part.

I would not to repeat this discussion from start. I am sorry.

Regards

Pavel


> Anyway, the patch is non trivial and very large, so targetting v12 now is
> indeed out of reach.
>
> --
> Fabien.
>
>

  118. Re: Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2019-03-07T08:37:58Z

    Hi


>> My strong opinion based on the underlying use case is that it that such
>> session variables should be transactional by default, and Pavel strong
>> opinion is that they should not, to be closer to Oracle comparable
>> feature.
>
>
It is closer to any known database Oracle, DB2, Firebird, MSSQL, MySQL,

Regards

Pavel

  119. Re: [HACKERS] proposal: schema variables

    David Steele <david@pgmasters.net> — 2019-03-07T09:26:06Z

    On 3/7/19 10:10 AM, Fabien COELHO wrote:
> 
> Anyway, the patch is non trivial and very large, so targetting v12 now 
> is indeed out of reach.

Agreed.  I have set the target version to PG13.

Regards,
-- 
-David
david@pgmasters.net

  120. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2019-03-24T05:57:10Z

    Hi

rebase against current master

Regards

Pavel

  121. Re: [HACKERS] proposal: schema variables

    Erik Rijkers <er@xs4all.nl> — 2019-03-24T09:25:38Z

    On 2019-03-24 06:57, Pavel Stehule wrote:
> Hi
> 
> rebase against current master
> 

I ran into this:

(schema 'varschema2' does not exist):

drop variable varschema2.testv cascade;
ERROR:  schema "varschema2" does not exist
create variable if not exists testv as text;
server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.
connection to server was lost


(both statements are needed to force the crash)


thanks,

Erik Rijkers

  122. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2019-03-24T09:32:53Z

    ne 24. 3. 2019 v 10:25 odesílatel Erik Rijkers <er@xs4all.nl> napsal:

> On 2019-03-24 06:57, Pavel Stehule wrote:
> > Hi
> >
> > rebase against current master
> >
>
> I ran into this:
>
> (schema 'varschema2' does not exist):
>
> drop variable varschema2.testv cascade;
> ERROR:  schema "varschema2" does not exist
> create variable if not exists testv as text;
> server closed the connection unexpectedly
>         This probably means the server terminated abnormally
>         before or while processing the request.
> connection to server was lost
>
>
> (both statements are needed to force the crash)
>

I cannot to reproduce it.

please, try compilation with "make distclean"; configure ..

or if the problem persists, please send test case, or backtrace

Regards

Pavel

>
>
> thanks,
>
> Erik Rijkers
>
>
>

  123. Re: [HACKERS] proposal: schema variables

    Erik Rijkers <er@xs4all.nl> — 2019-03-25T19:40:15Z

    On 2019-03-24 10:32, Pavel Stehule wrote:
> ne 24. 3. 2019 v 10:25 odesílatel Erik Rijkers <er@xs4all.nl> napsal:
> 
>> On 2019-03-24 06:57, Pavel Stehule wrote:
>> > Hi
>> >
>> > rebase against current master
>> 
>> I ran into this:
>> 
>> (schema 'varschema2' does not exist):
>> 
>> drop variable varschema2.testv cascade;
>> ERROR:  schema "varschema2" does not exist
>> create variable if not exists testv as text;
>> server closed the connection unexpectedly
>>         This probably means the server terminated abnormally
>>         before or while processing the request.
>> connection to server was lost
>> 
>> 
>> (both statements are needed to force the crash)
>> 
> 
> I cannot to reproduce it.
>  [backtrace and stuff]

Sorry, I don't have the wherewithal to get more info but I have repeated 
this now on 4 different machines (debian jessie/stretch; centos).

I did notice that sometimes those two offending lines
"
   drop variable varschema2.testv cascade;
   create variable if not exists testv as text;
"
have to be repeated a few times (never more than 4 or 5 times) before 
the crash occurs (signal 11: Segmentation fault).


Erik Rijkers

  124. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2019-03-26T05:40:22Z

    Hi

ne 24. 3. 2019 v 6:57 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
napsal:

> Hi
>
> rebase against current master
>


fixed issue IF NOT EXISTS & related regress tests

Regards

Pavel


> Regards
>
> Pavel
>

  125. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2019-03-26T05:41:04Z

    po 25. 3. 2019 v 20:40 odesílatel Erik Rijkers <er@xs4all.nl> napsal:

> On 2019-03-24 10:32, Pavel Stehule wrote:
> > ne 24. 3. 2019 v 10:25 odesílatel Erik Rijkers <er@xs4all.nl> napsal:
> >
> >> On 2019-03-24 06:57, Pavel Stehule wrote:
> >> > Hi
> >> >
> >> > rebase against current master
> >>
> >> I ran into this:
> >>
> >> (schema 'varschema2' does not exist):
> >>
> >> drop variable varschema2.testv cascade;
> >> ERROR:  schema "varschema2" does not exist
> >> create variable if not exists testv as text;
> >> server closed the connection unexpectedly
> >>         This probably means the server terminated abnormally
> >>         before or while processing the request.
> >> connection to server was lost
> >>
> >>
> >> (both statements are needed to force the crash)
> >>
> >
> > I cannot to reproduce it.
> >  [backtrace and stuff]
>
> Sorry, I don't have the wherewithal to get more info but I have repeated
> this now on 4 different machines (debian jessie/stretch; centos).
>
> I did notice that sometimes those two offending lines
> "
>    drop variable varschema2.testv cascade;
>    create variable if not exists testv as text;
> "
> have to be repeated a few times (never more than 4 or 5 times) before
> the crash occurs (signal 11: Segmentation fault).
>

Should be fixed now.

Thank you for report

Pavel


>
> Erik Rijkers
>
>
>

  126. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2019-04-02T18:02:12Z

    út 26. 3. 2019 v 6:40 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
napsal:

> Hi
>
> ne 24. 3. 2019 v 6:57 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
> napsal:
>
>> Hi
>>
>> rebase against current master
>>
>
>
> fixed issue IF NOT EXISTS & related regress tests
>

another rebase

Regards

Pavel


> Regards
>
> Pavel
>
>
>> Regards
>>
>> Pavel
>>
>

  127. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2019-04-19T06:32:25Z

    fresh rebase

Regards

Pavel

  128. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2019-05-09T04:34:20Z

    Hi

rebased patch

Regards

Pavel

  129. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2019-05-24T17:12:13Z

    Hi

čt 9. 5. 2019 v 6:34 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
napsal:

> Hi
>
> rebased patch
>

rebase after pgindent

Regards

Pavel

>
> Regards
>
> Pavel
>
>
>

  130. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2019-06-30T03:10:08Z

    pá 24. 5. 2019 v 19:12 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
napsal:

> Hi
>
> čt 9. 5. 2019 v 6:34 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
> napsal:
>
>> Hi
>>
>> rebased patch
>>
>
> rebase after pgindent
>

fresh rebase

Regards

Pavel


> Regards
>
> Pavel
>
>>
>> Regards
>>
>> Pavel
>>
>>
>>

  131. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2019-07-16T12:50:49Z

    Hi

ne 30. 6. 2019 v 5:10 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
napsal:

>
>
> pá 24. 5. 2019 v 19:12 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
> napsal:
>
>> Hi
>>
>> čt 9. 5. 2019 v 6:34 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
>> napsal:
>>
>>> Hi
>>>
>>> rebased patch
>>>
>>
>> rebase after pgindent
>>
>
> fresh rebase
>

just rebase again

Regards

Pavel



> Regards
>
> Pavel
>
>
>> Regards
>>
>> Pavel
>>
>>>
>>> Regards
>>>
>>> Pavel
>>>
>>>
>>>

  132. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2019-08-10T07:10:14Z

    Hi

just rebase

Regards

Pavel

  133. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2019-10-04T04:12:41Z

    Hi

so 10. 8. 2019 v 9:10 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
napsal:

> Hi
>
> just rebase
>

fresh rebase

Regards

Pavel


> Regards
>
> Pavel
>

  134. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2019-10-10T09:41:10Z

    Hi

minor change - replace heap_tuple_fetch_attr by detoast_external_attr.

Regards

Pavel

pá 4. 10. 2019 v 6:12 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
napsal:

> Hi
>
> so 10. 8. 2019 v 9:10 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
> napsal:
>
>> Hi
>>
>> just rebase
>>
>
> fresh rebase
>
> Regards
>
> Pavel
>
>
>> Regards
>>
>> Pavel
>>
>

  135. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2019-11-03T16:27:17Z

    čt 10. 10. 2019 v 11:41 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
napsal:

> Hi
>
> minor change - replace heap_tuple_fetch_attr by detoast_external_attr.
>
>
similar update - heap_open, heap_close was replaced by table_open,
table_close

Regards

Pavel

  136. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2019-11-18T18:47:58Z

    ne 3. 11. 2019 v 17:27 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
napsal:

>
>
> čt 10. 10. 2019 v 11:41 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
> napsal:
>
>> Hi
>>
>> minor change - replace heap_tuple_fetch_attr by detoast_external_attr.
>>
>>
> similar update - heap_open, heap_close was replaced by table_open,
> table_close
>

fresh rebase

Regards

Pavel


> Regards
>
> Pavel
>

  137. Re: [HACKERS] proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2019-12-14T21:43:38Z

    po 18. 11. 2019 v 19:47 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
napsal:

>
>
> ne 3. 11. 2019 v 17:27 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
> napsal:
>
>>
>>
>> čt 10. 10. 2019 v 11:41 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
>> napsal:
>>
>>> Hi
>>>
>>> minor change - replace heap_tuple_fetch_attr by detoast_external_attr.
>>>
>>>
>> similar update - heap_open, heap_close was replaced by table_open,
>> table_close
>>
>
> fresh rebase
>

only rebase

Regards

Pavel


> Regards
>
> Pavel
>
>
>> Regards
>>
>> Pavel
>>
>

  138. Re: proposal: schema variables

    Philippe BEAUDOIN <phb07@apra.asso.fr> — 2019-12-22T12:03:10Z

    The following review has been posted through the commitfest application:
make installcheck-world:  tested, passed
Implements feature:       tested, failed
Spec compliant:           not tested
Documentation:            tested, failed

Hi Pavel,

First of all, I would like to congratulate you for this great work. This patch is really cool. The lack of package variables is sometimes a blocking issue for Oracle to Postgres migrations, because the usual emulation with GUC is sometimes not enough, in particular when there are security concerns or when the database is used in a public cloud.

As I look forward to having this patch commited, I decided to spend some time to participate to the review, although I am not a C specialist and I have not a good knowledge of the Postgres internals. Here is my report.

A) Installation

The patch applies correctly and the compilation is fine. The "make check" doesn't report any issue.

B) Basic usage

I tried some simple schema variables use cases. No problem.

C) The interface

The SQL changes look good to me.

However, in the CREATE VARIABLE command, I would replace the "TRANSACTION" word by "TRANSACTIONAL".

I have also tried to replace this word by a ON ROLLBACK clause at the end of the statement, like for ON COMMIT, but I have not found a satisfying wording to propose.

D) Behaviour

I am ok with variables not being transactional by default. That's the most simple, the most efficient, it emulates the package variables of other RDBMS and it will probably fit the most common use cases.

Note that I am not strongly opposed to having by default transactional variables. But I don't know whether this change would be a great work. We would have at least to find another keyword in the CREATE VARIABLE statement. Something like "NON-TRANSACTIONAL VARIABLE" ?

It is possible to create a NOT NULL variable without DEFAULT. When trying to read the variable before a LET statement, one gets an error massage saying that the NULL value is not allowed (and the documentation is clear about this case). Just for the records, I wondered whether it wouldn't be better to forbid a NOT NULL variable creation that wouldn't have a DEFAULT value. But finally, I think this behaviour provides a good way to force the variable initialisation before its use. So let's keep it as is.

E) ACL and Rights

I played a little bit with the GRANT and REVOKE statements. 

I have got an error (Issue 1). The following statement chain:
  create variable public.sv1 int;
  grant read on variable sv1 to other_user;
  drop owned by other_user;
reports : ERROR:  unexpected object class 4287

I then tried to use DEFAULT PRIVILEGES. Despite this is not documented, I successfuly performed:
  alter default privileges in schema public grant read on variables to simple_user;
  alter default privileges in schema public grant write on variables to simple_user;

When variables are then created, the grants are properly given.
And the psql \ddp command perfectly returns:
             Default access privileges
  Owner   | Schema | Type |    Access privileges    
----------+--------+------+-------------------------
 postgres | public |      | simple_user=SW/postgres
(1 row)

So the ALTER DEFAULT PRIVILEGES documentation chapter has to reflect this new syntax (Issue 2).

BTW, in the ACL, the READ privilege is represented by a S letter. A comment in the source reports that the R letter was used in the past for rule privilege. Looking at the postgres sources, I see that this privilege on rules has been suppressed  in 8.2, so 13 years ago. As this R letter would be a so much better choice, I wonder whether it couldn't be reused now for this new purpose. Is it important to keep this letter frozen ?

F) Extension

I then created an extension, whose installation script creates a schema variable and functions that use it. The schema variable is correctly linked to the extension, so that dropping the extension drops the variable.

But there is an issue when dumping the database (Issue 3). The script generated by pg_dump includes the CREATE EXTENSION statement as expected but also a redundant CREATE VARIABLE statement for the variable that belongs to the extension. As a result, one of course gets an error at restore time.

G) Row Level Security

I did a test activating RLS on a table and creating a POLICY that references a schema variable in its USING and WITH CHECK clauses. Everything worked fine.

H) psql

A \dV meta-command displays all the created variables.
I would change a little bit the provided view. More precisely I would:
- rename "Constraint" into "Is nullable" and report it as a boolean
- rename "Special behave" into "Is transactional" and report it as a boolean
- change the order of columns so to have:
Schema | Name | Type | Is nullable | Default | Owner | Is transactional | Transaction end action
"Is nullable" being aside "Default"

I) Performance

I just quickly looked at the performance, and didn't notice any issue.

About variables read performance, I have noticed that:
  select sum(1) from generate_series(1,10000000);
and
  select sum(sv1) from generate_series(1,10000000);
have similar response times.

About planning, a condition with a variable used as a constant is indexable, as if it were a literal.

J) Documentation

There are some wordings to improve in the documentation. But I am not the best person to give advice about english language ;-).

However, aside the already mentionned lack of changes in the ALTER DEFAULT PRIVILEGES chapter, I also noticed :
- line 50 of the patch, the sentence "(hidden attribute; must be explicitly selected)" looks false as the oid column of pg_variable is displayed, as for other tables of the catalog;
- at several places, the word "behave" should be replaced by "behaviour"
- line 433, a get_schema_variable() function is mentionned; is it a function that can really be called by users ?

May be it would be interesting to also add a chapter in the Section V of the documentation, in order to more globally present the schema variables concept, aside the new or the modified statements.

K) Coding

I am not able to appreciate the way the feature has been coded. So I let this for other reviewers ;-)


To conclude, again, thanks a lot for this feature !
And if I may add this. I dream of an additional feature: adding a SHARED clause to the CREATE VARIABLE statement in order to be able to create memory spaces that could be shared by all connections on the database and accessible in SQL and PL, under the protection of ACL. But that's another story ;-)

Best regards. Philippe.

The new status of this patch is: Waiting on Author

  139. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2019-12-22T18:50:22Z

    Hi

ne 22. 12. 2019 v 13:04 odesílatel Philippe BEAUDOIN <phb07@apra.asso.fr>
napsal:

> The following review has been posted through the commitfest application:
> make installcheck-world:  tested, passed
> Implements feature:       tested, failed
> Spec compliant:           not tested
> Documentation:            tested, failed
>
> Hi Pavel,
>
> First of all, I would like to congratulate you for this great work. This
> patch is really cool. The lack of package variables is sometimes a blocking
> issue for Oracle to Postgres migrations, because the usual emulation with
> GUC is sometimes not enough, in particular when there are security concerns
> or when the database is used in a public cloud.
>
> As I look forward to having this patch commited, I decided to spend some
> time to participate to the review, although I am not a C specialist and I
> have not a good knowledge of the Postgres internals. Here is my report.
>
> A) Installation
>
> The patch applies correctly and the compilation is fine. The "make check"
> doesn't report any issue.
>
> B) Basic usage
>
> I tried some simple schema variables use cases. No problem.
>
> C) The interface
>
> The SQL changes look good to me.
>
> However, in the CREATE VARIABLE command, I would replace the "TRANSACTION"
> word by "TRANSACTIONAL".
>

There is not technical problem - the problem is in introduction new keyword
"transactional" that is near to "transaction". I am not sure if it is
practical to have two "similar" keyword and how much the CREATE statement
has to use correct English grammar.

I am not native speaker, so I am not able to see how bad is using
"TRANSACTION" instead "TRANSACTIONAL" in this context. So I see a risk to
have two important (it is not syntactic sugar) similar keywords.

Just I afraid so using TRANSACTIONAL instead just TRANSACTION is not too
user friendly. I have not strong opinion about this - and the
implementation is easy, but I am not feel comfortable with introduction
this keyword.


> I have also tried to replace this word by a ON ROLLBACK clause at the end
> of the statement, like for ON COMMIT, but I have not found a satisfying
> wording to propose.
>



>
> D) Behaviour
>
> I am ok with variables not being transactional by default. That's the most
> simple, the most efficient, it emulates the package variables of other
> RDBMS and it will probably fit the most common use cases.
>
> Note that I am not strongly opposed to having by default transactional
> variables. But I don't know whether this change would be a great work. We
> would have at least to find another keyword in the CREATE VARIABLE
> statement. Something like "NON-TRANSACTIONAL VARIABLE" ?
>

Variables almost everywhere (global user settings - GUC is only one planet
exception) are non transactional by default. I don't see any reason
introduce new different design than is wide used.


> It is possible to create a NOT NULL variable without DEFAULT. When trying
> to read the variable before a LET statement, one gets an error massage
> saying that the NULL value is not allowed (and the documentation is clear
> about this case). Just for the records, I wondered whether it wouldn't be
> better to forbid a NOT NULL variable creation that wouldn't have a DEFAULT
> value. But finally, I think this behaviour provides a good way to force the
> variable initialisation before its use. So let's keep it as is.
>

This is a question - and there are two possibilities

postgres=# do $$
declare x int not null;
begin
  raise notice '%', x;
end;
$$ ;
ERROR:  variable "x" must have a default value, since it's declared NOT NULL
LINE 2: declare x int not null;
                      ^

PLpgSQL requires it. But there is not a possibility to enforce future
setting.

So I know so behave of schema variables is little bit different, but I
think so this difference has interesting use case. You can check if the
variable was modified somewhere or not.


> E) ACL and Rights
>
> I played a little bit with the GRANT and REVOKE statements.
>
> I have got an error (Issue 1). The following statement chain:
>   create variable public.sv1 int;
>   grant read on variable sv1 to other_user;
>   drop owned by other_user;
> reports : ERROR:  unexpected object class 4287
>

this is bug and should be fixed


> I then tried to use DEFAULT PRIVILEGES. Despite this is not documented, I
> successfuly performed:
>   alter default privileges in schema public grant read on variables to
> simple_user;
>   alter default privileges in schema public grant write on variables to
> simple_user;
>
> When variables are then created, the grants are properly given.
> And the psql \ddp command perfectly returns:
>              Default access privileges
>   Owner   | Schema | Type |    Access privileges
> ----------+--------+------+-------------------------
>  postgres | public |      | simple_user=SW/postgres
> (1 row)
>
> So the ALTER DEFAULT PRIVILEGES documentation chapter has to reflect this
> new syntax (Issue 2).
>
> BTW, in the ACL, the READ privilege is represented by a S letter. A
> comment in the source reports that the R letter was used in the past for
> rule privilege. Looking at the postgres sources, I see that this privilege
> on rules has been suppressed  in 8.2, so 13 years ago. As this R letter
> would be a so much better choice, I wonder whether it couldn't be reused
> now for this new purpose. Is it important to keep this letter frozen ?
>

I have not a idea why it is. I'll recheck it - but in this moment I prefer
a consistency with existing ACL - it can be in future as one block if it
will be necessary for somebody.


>
> F) Extension
>
> I then created an extension, whose installation script creates a schema
> variable and functions that use it. The schema variable is correctly linked
> to the extension, so that dropping the extension drops the variable.
>
> But there is an issue when dumping the database (Issue 3). The script
> generated by pg_dump includes the CREATE EXTENSION statement as expected
> but also a redundant CREATE VARIABLE statement for the variable that
> belongs to the extension. As a result, one of course gets an error at
> restore time.
>

It is bug and should be fixed


> G) Row Level Security
>
> I did a test activating RLS on a table and creating a POLICY that
> references a schema variable in its USING and WITH CHECK clauses.
> Everything worked fine.
>
> H) psql
>
> A \dV meta-command displays all the created variables.
> I would change a little bit the provided view. More precisely I would:
> - rename "Constraint" into "Is nullable" and report it as a boolean
> - rename "Special behave" into "Is transactional" and report it as a
> boolean
> - change the order of columns so to have:
> Schema | Name | Type | Is nullable | Default | Owner | Is transactional |
> Transaction end action
> "Is nullable" being aside "Default"
>

ok


> I) Performance
>
> I just quickly looked at the performance, and didn't notice any issue.
>
> About variables read performance, I have noticed that:
>   select sum(1) from generate_series(1,10000000);
> and
>   select sum(sv1) from generate_series(1,10000000);
> have similar response times.
>

> About planning, a condition with a variable used as a constant is
> indexable, as if it were a literal.
>
> J) Documentation
>
> There are some wordings to improve in the documentation. But I am not the
> best person to give advice about english language ;-).
>
> However, aside the already mentionned lack of changes in the ALTER DEFAULT
> PRIVILEGES chapter, I also noticed :
> - line 50 of the patch, the sentence "(hidden attribute; must be
> explicitly selected)" looks false as the oid column of pg_variable is
> displayed, as for other tables of the catalog;
> - at several places, the word "behave" should be replaced by "behaviour"
> - line 433, a get_schema_variable() function is mentionned; is it a
> function that can really be called by users ?
>
> May be it would be interesting to also add a chapter in the Section V of
> the documentation, in order to more globally present the schema variables
> concept, aside the new or the modified statements.
>
> K) Coding
>
> I am not able to appreciate the way the feature has been coded. So I let
> this for other reviewers ;-)
>
>
> To conclude, again, thanks a lot for this feature !
> And if I may add this. I dream of an additional feature: adding a SHARED
> clause to the CREATE VARIABLE statement in order to be able to create
> memory spaces that could be shared by all connections on the database and
> accessible in SQL and PL, under the protection of ACL. But that's another
> story ;-)
>

sure, it is another story :-).

Thank you for review - I'll try to fix bugs this week.

Pavel


> Best regards. Philippe.
>
> The new status of this patch is: Waiting on Author
>

  140. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2019-12-25T21:45:49Z

    Hi

ne 22. 12. 2019 v 13:04 odesílatel Philippe BEAUDOIN <phb07@apra.asso.fr>
napsal:

> The following review has been posted through the commitfest application:
> make installcheck-world:  tested, passed
> Implements feature:       tested, failed
> Spec compliant:           not tested
> Documentation:            tested, failed
>
> Hi Pavel,
>
> First of all, I would like to congratulate you for this great work. This
> patch is really cool. The lack of package variables is sometimes a blocking
> issue for Oracle to Postgres migrations, because the usual emulation with
> GUC is sometimes not enough, in particular when there are security concerns
> or when the database is used in a public cloud.
>
> As I look forward to having this patch commited, I decided to spend some
> time to participate to the review, although I am not a C specialist and I
> have not a good knowledge of the Postgres internals. Here is my report.
>
> A) Installation
>
> The patch applies correctly and the compilation is fine. The "make check"
> doesn't report any issue.
>
> B) Basic usage
>
> I tried some simple schema variables use cases. No problem.
>
> C) The interface
>
> The SQL changes look good to me.
>
> However, in the CREATE VARIABLE command, I would replace the "TRANSACTION"
> word by "TRANSACTIONAL".
>
> I have also tried to replace this word by a ON ROLLBACK clause at the end
> of the statement, like for ON COMMIT, but I have not found a satisfying
> wording to propose.
>

I propose compromise solution - I introduced new not reserved keyword
"TRANSACTIONAL". User can use TRANSACTION or TRANSACTIONAL. It is similar
relation like "TEMP" or "TEMPORAL"


>
> D) Behaviour
>
> I am ok with variables not being transactional by default. That's the most
> simple, the most efficient, it emulates the package variables of other
> RDBMS and it will probably fit the most common use cases.
>
> Note that I am not strongly opposed to having by default transactional
> variables. But I don't know whether this change would be a great work. We
> would have at least to find another keyword in the CREATE VARIABLE
> statement. Something like "NON-TRANSACTIONAL VARIABLE" ?
>
> It is possible to create a NOT NULL variable without DEFAULT. When trying
> to read the variable before a LET statement, one gets an error massage
> saying that the NULL value is not allowed (and the documentation is clear
> about this case). Just for the records, I wondered whether it wouldn't be
> better to forbid a NOT NULL variable creation that wouldn't have a DEFAULT
> value. But finally, I think this behaviour provides a good way to force the
> variable initialisation before its use. So let's keep it as is.
>
> E) ACL and Rights
>
> I played a little bit with the GRANT and REVOKE statements.
>
> I have got an error (Issue 1). The following statement chain:
>   create variable public.sv1 int;
>   grant read on variable sv1 to other_user;
>   drop owned by other_user;
> reports : ERROR:  unexpected object class 4287
>

should be fixed


> I then tried to use DEFAULT PRIVILEGES. Despite this is not documented, I
> successfuly performed:
>   alter default privileges in schema public grant read on variables to
> simple_user;
>   alter default privileges in schema public grant write on variables to
> simple_user;
>

should be fixed


> When variables are then created, the grants are properly given.
> And the psql \ddp command perfectly returns:
>              Default access privileges
>   Owner   | Schema | Type |    Access privileges
> ----------+--------+------+-------------------------
>  postgres | public |      | simple_user=SW/postgres
> (1 row)
>
> So the ALTER DEFAULT PRIVILEGES documentation chapter has to reflect this
> new syntax (Issue 2).
>
> BTW, in the ACL, the READ privilege is represented by a S letter. A
> comment in the source reports that the R letter was used in the past for
> rule privilege. Looking at the postgres sources, I see that this privilege
> on rules has been suppressed  in 8.2, so 13 years ago. As this R letter
> would be a so much better choice, I wonder whether it couldn't be reused
> now for this new purpose. Is it important to keep this letter frozen ?
>

I use ACL_READ constant in my patch. The value of ACL_READ is defined
elsewhere. So the changing from S to R should be done by separate patch and
by separate discussion.


> F) Extension
>
> I then created an extension, whose installation script creates a schema
> variable and functions that use it. The schema variable is correctly linked
> to the extension, so that dropping the extension drops the variable.
>
> But there is an issue when dumping the database (Issue 3). The script
> generated by pg_dump includes the CREATE EXTENSION statement as expected
> but also a redundant CREATE VARIABLE statement for the variable that
> belongs to the extension. As a result, one of course gets an error at
> restore time.
>

should be fixed now


> G) Row Level Security
>
> I did a test activating RLS on a table and creating a POLICY that
> references a schema variable in its USING and WITH CHECK clauses.
> Everything worked fine.
>
> H) psql
>
> A \dV meta-command displays all the created variables.
> I would change a little bit the provided view. More precisely I would:
> - rename "Constraint" into "Is nullable" and report it as a boolean
> - rename "Special behave" into "Is transactional" and report it as a
> boolean
> - change the order of columns so to have:
> Schema | Name | Type | Is nullable | Default | Owner | Is transactional |
> Transaction end action
> "Is nullable" being aside "Default"
>

I implemented your proposal


> I) Performance
>
> I just quickly looked at the performance, and didn't notice any issue.
>
> About variables read performance, I have noticed that:
>   select sum(1) from generate_series(1,10000000);
> and
>   select sum(sv1) from generate_series(1,10000000);
> have similar response times.
>
> About planning, a condition with a variable used as a constant is
> indexable, as if it were a literal.
>
> J) Documentation
>
> There are some wordings to improve in the documentation. But I am not the
> best person to give advice about english language ;-).
>
> However, aside the already mentionned lack of changes in the ALTER DEFAULT
> PRIVILEGES chapter, I also noticed :
> - line 50 of the patch, the sentence "(hidden attribute; must be
> explicitly selected)" looks false as the oid column of pg_variable is
> displayed, as for other tables of the catalog;
> - at several places, the word "behave" should be replaced by "behaviour"
> - line 433, a get_schema_variable() function is mentionned; is it a
> function that can really be called by users ?
>

should be fixed


> May be it would be interesting to also add a chapter in the Section V of
> the documentation, in order to more globally present the schema variables
> concept, aside the new or the modified statements.
>

We can finalize documentation little bit later, when will be clear what
related functionality is implemented.

updated patch attached





> K) Coding
>
> I am not able to appreciate the way the feature has been coded. So I let
> this for other reviewers ;-)
>
>
> To conclude, again, thanks a lot for this feature !
> And if I may add this. I dream of an additional feature: adding a SHARED
> clause to the CREATE VARIABLE statement in order to be able to create
> memory spaces that could be shared by all connections on the database and
> accessible in SQL and PL, under the protection of ACL. But that's another
> story ;-)
>
> Best regards. Philippe.
>

Thank you very much for review

>
> The new status of this patch is: Waiting on Author
>

  141. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2019-12-26T18:13:41Z

    Hi

fresh rebase

Regards

Pavel

  142. Re: proposal: schema variables

    Philippe BEAUDOIN <phb07@apra.asso.fr> — 2019-12-30T16:26:10Z

    The following review has been posted through the commitfest application:
make installcheck-world:  tested, passed
Implements feature:       tested, passed
Spec compliant:           not tested
Documentation:            tested, failed

Hi Pavel,

I have tested the latest version of your patch.
Both issues I reported are now fixed. And you largely applied my proposals. That's great !

I have also spent some time to review more closely the documentation. I will send you a direct mail with an attached file for some minor comments on this topic.

Except these documentation remarks to come, I haven't any other issue or suggestion to report.
Note that I have not closely looked at the C code itself. But may be some other reviewers have already done that job.
If yes, my feeling is that the patch could soon be set as "Ready for commiter".

Best regards. Philippe.

The new status of this patch is: Waiting on Author

  143. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2019-12-30T20:05:40Z

    Hi

po 30. 12. 2019 v 17:27 odesílatel Philippe BEAUDOIN <phb07@apra.asso.fr>
napsal:

> The following review has been posted through the commitfest application:
> make installcheck-world:  tested, passed
> Implements feature:       tested, passed
> Spec compliant:           not tested
> Documentation:            tested, failed
>
> Hi Pavel,
>
> I have tested the latest version of your patch.
> Both issues I reported are now fixed. And you largely applied my
> proposals. That's great !
>
> I have also spent some time to review more closely the documentation. I
> will send you a direct mail with an attached file for some minor comments
> on this topic.
>
> Except these documentation remarks to come, I haven't any other issue or
> suggestion to report.
> Note that I have not closely looked at the C code itself. But may be some
> other reviewers have already done that job.
> If yes, my feeling is that the patch could soon be set as "Ready for
> commiter".
>
> Best regards. Philippe.
>
> The new status of this patch is: Waiting on Author
>

Thank you very much for your comments, and notes. Updated patch attached.

Regards

Pavel

  144. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2020-01-17T21:10:35Z

    Hi

po 30. 12. 2019 v 21:05 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
napsal:

> Hi
>
> po 30. 12. 2019 v 17:27 odesílatel Philippe BEAUDOIN <phb07@apra.asso.fr>
> napsal:
>
>> The following review has been posted through the commitfest application:
>> make installcheck-world:  tested, passed
>> Implements feature:       tested, passed
>> Spec compliant:           not tested
>> Documentation:            tested, failed
>>
>> Hi Pavel,
>>
>> I have tested the latest version of your patch.
>> Both issues I reported are now fixed. And you largely applied my
>> proposals. That's great !
>>
>> I have also spent some time to review more closely the documentation. I
>> will send you a direct mail with an attached file for some minor comments
>> on this topic.
>>
>> Except these documentation remarks to come, I haven't any other issue or
>> suggestion to report.
>> Note that I have not closely looked at the C code itself. But may be some
>> other reviewers have already done that job.
>> If yes, my feeling is that the patch could soon be set as "Ready for
>> commiter".
>>
>> Best regards. Philippe.
>>
>> The new status of this patch is: Waiting on Author
>>
>
> Thank you very much for your comments, and notes. Updated patch attached.
>

rebase



> Regards
>
> Pavel
>
>

  145. Re: proposal: schema variables

    Tomas Vondra <tomas.vondra@2ndquadrant.com> — 2020-01-21T23:41:41Z

    Hi,

I did a quick review of this patch today, so let me share a couple of
comments.

Firstly, the patch is pretty large - it has ~270kB. Not the largest
patch ever, but large. I think breaking the patch into smaller pieces
would significantly improve the chnce of it getting committed.

Is it possible to break the patch into smaller pieces that could be
applied incrementally? For example, we could move the "transactional"
behavior into a separate patch, but it's not clear to me how much code
would that actually move to that second patch. Any other parts that
could be moved to a separate patch?

I see one of the main contention points was a rather long discussion
about transactional vs. non-transactional behavior. I agree with Pavel's
position that the non-transactional behavior should be the default,
simply because it's better aligned with what the other databases are
doing (and supporting migrations seems like one of the main use cases
for this feature).

I do understand it may not be suitable for some other use cases,
mentioned by Fabien, but IMHO it's fine to require explicit
specification of transactional behavior. Well, we can't have both as
default, and I don't think there's an obvious reason why it should be
the other way around.

Now, a bunch of comments about the code (some of that nitpicking):


1) This hunk in doc/src/sgml/catalogs.sgml still talks about "table
creation" instead of schema creation:

      <row>
       <entry><structfield>vartypmod</structfield></entry>
       <entry><type>int4</type></entry>
       <entry></entry>
       <entry>
        <structfield>vartypmod</structfield> records type-specific data
        supplied at table creation time (for example, the maximum
        length of a <type>varchar</type> column).  It is passed to
        type-specific input functions and length coercion functions.
        The value will generally be -1 for types that do not need <structfield>vartypmod</structfield>.
       </entry>
      </row>


2) This hunk in doc/src/sgml/ref/alter_default_privileges.sgml uses
"role_name" instead of "variable_name"

     GRANT { READ | WRITE | ALL [ PRIVILEGES ] }
     ON VARIABLES
     TO { [ GROUP ] <replaceable class="parameter">role_name</replaceable> | PUBLIC } [, ...] [ WITH GRANT OPTION ]


3) I find the syntax in create_variable.sgml a bit too over-complicated:

<synopsis>
CREATE [ { TEMPORARY | TEMP } ] [ { TRANSACTIONAL | TRANSACTION } ] VARIABLE [ IF NOT EXISTS ] <replaceable class="parameter">name</replaceable> [ AS ] <replaceable class="parameter">data_type</replaceable> ] [ COLLATE <replaceable class="parameter">collation</replaceable> ]
     [ NOT NULL ] [ DEFAULT <replaceable class="parameter">default_expr</replaceable> ] [ { ON COMMIT DROP | ON { TRANSACTIONAL | TRANSACTION } END RESET } ]
</synopsis>

Do we really need both TRANSACTION and TRANSACTIONAL? Why not just one
that we already have in the grammar (i.e. TRANSACTION)?


4) I think we should rename schemavariable.h to schema_variable.h.


5) objectaddress.c has extra line after 'break;' in one switch.


6) The comment is wrong:

     /*
      * Find the ObjectAddress for a type or domain
      */
     static ObjectAddress
     get_object_address_variable(List *object, bool missing_ok)


7) I think the code/comments are really inconsistent in talking about
"variables" and "schema variables". For example in objectaddress.c we do
these two things:

         case OCLASS_VARIABLE:
             appendStringInfoString(&buffer, "schema variable");
             break;

vs.

         case DEFACLOBJ_VARIABLE:
             appendStringInfoString(&buffer,
                                    " on variables");
             break;

That's going to be confusing for people.


8) I'm rather confused by CMD_PLAN_UTILITY, which seems to be defined
merely to support LET. I'm not sure why that's necessary (Why wouldn't
CMD_UTILITY be sufficient?).

Having to add conditions checking for CMD_PLAN_UTILITY to various places
in planner.c is rather annoying, and I wonder how likely it's this will
unnecessarily break external code in extensions etc.


9) This comment in planner.c seems obsolete (not updated to reflect
addition of the CMD_PLAN_UTILITY check):

   /*
    * If this is an INSERT/UPDATE/DELETE, and we're not being called from
    * inheritance_planner, add the ModifyTable node.
    */
   if (parse->commandType != CMD_SELECT && parse->commandType != CMD_PLAN_UTILITY && !inheritance_update)


10) I kinda wonder what happens when a function is used in a WHERE
condition, but it depends on a variable and alsu mutates it on each
call ...


11) I think Query->hasSchemaVariable (in analyze.c) should be renamed to
hasSchemaVariables (which reflects the other fields referring to things
like window functions etc.)


12) I find it rather suspicious that we make decisions in utility.c
solely based on commandType (whether it's CMD_UTILITY or not). IMO
it's pretty strange/ugly that T_LetStmt can be both CMD_UTILITY and
CMD_PLAN_UTILITY:

     case T_LetStmt:
         {
             if (pstmt->commandType == CMD_UTILITY)
                 doLetStmtReset(pstmt);
             else
             {
                 Assert(pstmt->commandType == CMD_PLAN_UTILITY);
                 doLetStmtEval(pstmt, params, queryEnv, queryString);
             }

             if (completionTag)
                 strcpy(completionTag, "LET");
         }
         break;


13) Not sure why we moved DO_TABLE in addBoundaryDependencies
(pg_dump.c), seems unnecessary:

      case DO_CONVERSION:
-    case DO_TABLE:
+    case DO_VARIABLE:
      case DO_ATTRDEF:
+    case DO_TABLE:
      case DO_PROCLANG:


14) namespace.c defines VariableIsVisible twice:

     extern bool VariableIsVisible(Oid relid);
     ...
     extern bool VariableIsVisible(Oid varid);


15) I'd say lookup_variable and identify_variable should use camelcase
just like the other functions in the same file.


regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  146. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2020-01-24T05:08:45Z

    st 22. 1. 2020 v 0:41 odesílatel Tomas Vondra <tomas.vondra@2ndquadrant.com>
napsal:

> Hi,
>
> I did a quick review of this patch today, so let me share a couple of
> comments.
>
> Firstly, the patch is pretty large - it has ~270kB. Not the largest
> patch ever, but large. I think breaking the patch into smaller pieces
> would significantly improve the chnce of it getting committed.
>
> Is it possible to break the patch into smaller pieces that could be
> applied incrementally? For example, we could move the "transactional"
> behavior into a separate patch, but it's not clear to me how much code
> would that actually move to that second patch. Any other parts that
> could be moved to a separate patch?
>

I am sending two patches - 0001 - schema variables, 0002 - transactional
variables

>
> I see one of the main contention points was a rather long discussion
> about transactional vs. non-transactional behavior. I agree with Pavel's
> position that the non-transactional behavior should be the default,
> simply because it's better aligned with what the other databases are
> doing (and supporting migrations seems like one of the main use cases
> for this feature).
>
> I do understand it may not be suitable for some other use cases,
> mentioned by Fabien, but IMHO it's fine to require explicit
> specification of transactional behavior. Well, we can't have both as
> default, and I don't think there's an obvious reason why it should be
> the other way around.
>
> Now, a bunch of comments about the code (some of that nitpicking):
>
>
> 1) This hunk in doc/src/sgml/catalogs.sgml still talks about "table
> creation" instead of schema creation:
>
>       <row>
>        <entry><structfield>vartypmod</structfield></entry>
>        <entry><type>int4</type></entry>
>        <entry></entry>
>        <entry>
>         <structfield>vartypmod</structfield> records type-specific data
>         supplied at table creation time (for example, the maximum
>         length of a <type>varchar</type> column).  It is passed to
>         type-specific input functions and length coercion functions.
>         The value will generally be -1 for types that do not need
> <structfield>vartypmod</structfield>.
>        </entry>
>       </row>
>

fixed


>
> 2) This hunk in doc/src/sgml/ref/alter_default_privileges.sgml uses
> "role_name" instead of "variable_name"
>
>      GRANT { READ | WRITE | ALL [ PRIVILEGES ] }
>      ON VARIABLES
>      TO { [ GROUP ] <replaceable class="parameter">role_name</replaceable>
> | PUBLIC } [, ...] [ WITH GRANT OPTION ]
>

I think so this is correct


>
> 3) I find the syntax in create_variable.sgml a bit too over-complicated:
>
> <synopsis>
> CREATE [ { TEMPORARY | TEMP } ] [ { TRANSACTIONAL | TRANSACTION } ]
> VARIABLE [ IF NOT EXISTS ] <replaceable
> class="parameter">name</replaceable> [ AS ] <replaceable
> class="parameter">data_type</replaceable> ] [ COLLATE <replaceable
> class="parameter">collation</replaceable> ]
>      [ NOT NULL ] [ DEFAULT <replaceable
> class="parameter">default_expr</replaceable> ] [ { ON COMMIT DROP | ON {
> TRANSACTIONAL | TRANSACTION } END RESET } ]
> </synopsis>
>
> Do we really need both TRANSACTION and TRANSACTIONAL? Why not just one
> that we already have in the grammar (i.e. TRANSACTION)?
>

It was a Philippe's wish - the implementation is simple, and it is similar
like TEMP, TEMPORARY. I have not any opinion about it.


>
> 4) I think we should rename schemavariable.h to schema_variable.h.
>

done


>
> 5) objectaddress.c has extra line after 'break;' in one switch.
>

fixed


>
> 6) The comment is wrong:
>
>      /*
>       * Find the ObjectAddress for a type or domain
>       */
>      static ObjectAddress
>      get_object_address_variable(List *object, bool missing_ok)
>

fixed


>
> 7) I think the code/comments are really inconsistent in talking about
> "variables" and "schema variables". For example in objectaddress.c we do
> these two things:
>
>          case OCLASS_VARIABLE:
>              appendStringInfoString(&buffer, "schema variable");
>              break;
>
> vs.
>
>          case DEFACLOBJ_VARIABLE:
>              appendStringInfoString(&buffer,
>                                     " on variables");
>              break;
>
> That's going to be confusing for people.
>
>
fixed


>
> 8) I'm rather confused by CMD_PLAN_UTILITY, which seems to be defined
> merely to support LET. I'm not sure why that's necessary (Why wouldn't
> CMD_UTILITY be sufficient?).
>

Currently out utility statements cannot to hold a execution plan, and
cannot be prepared.

so this enhancing is motivated mainly by performance reasons. I would to
allow any SELECT query there, not just expressions only (see a limits of
CALL statement)



> Having to add conditions checking for CMD_PLAN_UTILITY to various places
> in planner.c is rather annoying, and I wonder how likely it's this will
> unnecessarily break external code in extensions etc.
>
>
> 9) This comment in planner.c seems obsolete (not updated to reflect
> addition of the CMD_PLAN_UTILITY check):
>
>    /*
>     * If this is an INSERT/UPDATE/DELETE, and we're not being called from
>     * inheritance_planner, add the ModifyTable node.
>     */
>    if (parse->commandType != CMD_SELECT && parse->commandType !=
> CMD_PLAN_UTILITY && !inheritance_update)
>

"If this is an INSERT/UPDATE/DELETE," is related to parse->commandType !=
CMD_SELECT && parse->commandType != CMD_PLAN_UTILITY

>
>
> 10) I kinda wonder what happens when a function is used in a WHERE
> condition, but it depends on a variable and alsu mutates it on each
> call ...
>
>
> 11) I think Query->hasSchemaVariable (in analyze.c) should be renamed to
> hasSchemaVariables (which reflects the other fields referring to things
> like window functions etc.)
>
>
done


> 12) I find it rather suspicious that we make decisions in utility.c
> solely based on commandType (whether it's CMD_UTILITY or not). IMO
> it's pretty strange/ugly that T_LetStmt can be both CMD_UTILITY and
> CMD_PLAN_UTILITY:
>
>      case T_LetStmt:
>          {
>              if (pstmt->commandType == CMD_UTILITY)
>                  doLetStmtReset(pstmt);
>              else
>              {
>                  Assert(pstmt->commandType == CMD_PLAN_UTILITY);
>                  doLetStmtEval(pstmt, params, queryEnv, queryString);
>              }
>
>              if (completionTag)
>                  strcpy(completionTag, "LET");
>          }
>          break;
>
>
> 13) Not sure why we moved DO_TABLE in addBoundaryDependencies
> (pg_dump.c), seems unnecessary:
>
>       case DO_CONVERSION:
> -    case DO_TABLE:
> +    case DO_VARIABLE:
>       case DO_ATTRDEF:
> +    case DO_TABLE:
>       case DO_PROCLANG:
>

fixed


>
> 14) namespace.c defines VariableIsVisible twice:
>
>      extern bool VariableIsVisible(Oid relid);
>      ...
>      extern bool VariableIsVisible(Oid varid);
>
>
fixed


> 15) I'd say lookup_variable and identify_variable should use camelcase
> just like the other functions in the same file.
>

fixed

Regards

Pavel


>
> regards
>
> --
> Tomas Vondra                  http://www.2ndQuadrant.com
> PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
>

  147. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2020-01-26T17:26:16Z

    >
>
> 12) I find it rather suspicious that we make decisions in utility.c
> solely based on commandType (whether it's CMD_UTILITY or not). IMO
> it's pretty strange/ugly that T_LetStmt can be both CMD_UTILITY and
> CMD_PLAN_UTILITY:
>
>      case T_LetStmt:
>          {
>              if (pstmt->commandType == CMD_UTILITY)
>                  doLetStmtReset(pstmt);
>              else
>              {
>                  Assert(pstmt->commandType == CMD_PLAN_UTILITY);
>                  doLetStmtEval(pstmt, params, queryEnv, queryString);
>              }
>
>              if (completionTag)
>                  strcpy(completionTag, "LET");
>          }
>          break;
>
>
>
It looks strange, but it has sense, because the LET stmt supports reset to
default value.

I can write

1. LET var = DEFAULT;
2. LET var = (query);

In first case I have not any query, that I can assign, and in this case the
LET statement is really only UTILITY.

I did comment there

Regards

Pavel




>
>
> regards
>
> --
> Tomas Vondra                  http://www.2ndQuadrant.com
> PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
>

  148. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2020-02-07T16:09:47Z

    Hi

rebase

Regards

Pavel

  149. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2020-02-10T18:47:35Z

    pá 7. 2. 2020 v 17:09 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
napsal:

> Hi
>
> rebase
>
> Regards
>
> Pavel
>

Hi

another rebase, fix \dV statement (for 0001 patch)

Regards

Pavel

  150. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2020-02-15T08:05:01Z

    po 10. 2. 2020 v 19:47 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
napsal:

>
>
> pá 7. 2. 2020 v 17:09 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
> napsal:
>
>> Hi
>>
>> rebase
>>
>> Regards
>>
>> Pavel
>>
>
> Hi
>
> another rebase, fix \dV statement (for 0001 patch)
>

rebase

Pavel


> Regards
>
> Pavel
>

  151. Re: proposal: schema variables

    DUVAL REMI <remi.duval@cheops.fr> — 2020-02-26T14:53:55Z

    The following review has been posted through the commitfest application:
make installcheck-world:  not tested
Implements feature:       tested, passed
Spec compliant:           tested, failed
Documentation:            tested, failed

Hello Pavel

First thanks for working on this patch cause it might be really helpful for those of us trying to migrate PL code between RDBMs.

I tried your patch for migrating an Oracle package body to PL/pgSQL after also testing a solution using set_config and current_setting (which works but I'm not really satisfied by this workaround solution).

So I compiled latest postgres sources from github on Linux (redhat 7.7) using only your patch number 1 (I did not try the second part of the patch).

For my use-case it's working great, performances are excellent (compared to other solution for porting "package variables").
I did not test all the features involved by the patch (especially ALTER variable).

I have some feedback however :

1) Failure when using pg_dump 13 on a 12.1 database

When exporting a 12.1 database using pg_dump from the latest development sources I have an error regarding variables export 

[pg12@TST-LINUX-PG-03 ~]$ /opt/postgres12/pg12/bin/pg_dump -h localhost -p 5432 -U postgres -f dump_pg12.sql database1
pg_dump: error: query failed: ERROR:  relation "pg_variable" does not exist
LINE 1: ...og.pg_get_expr(v.vardefexpr,0) as vardefexpr FROM pg_variabl...
                                                             ^
pg_dump: error: query was: SELECT v.tableoid, v.oid, v.varname, v.vareoxaction, v.varnamespace, 
(SELECT rolname FROM pg_catalog.pg_roles WHERE oid = varowner) AS rolname
, (SELECT pg_catalog.array_agg(acl ORDER BY row_n) FROM (SELECT acl, row_n 
FROM pg_catalog.unnest(coalesce(v.varacl,pg_catalog.acldefault('V',v.varowner)))
 WITH ORDINALITY AS perm(acl,row_n) 
 WHERE NOT EXISTS ( SELECT 1 FROM pg_catalog.unnest(coalesce(pip.initprivs,pg_catalog.acldefault('V',v.varowner))) AS init(init_acl) 
                    WHERE acl = init_acl)) as foo) as varacl, ...:

I think that it should have worked anyway cause the documentation states :
https://www.postgresql.org/docs/current/upgrading.html
"It is recommended that you use the pg_dump and pg_dumpall programs from the newer version of PostgreSQL, to take advantage of enhancements that might have been made in these programs." (that's what I did here)

I think there should be a way to avoid dumping the variable if they don't exist, should'nt it ?

2) Displaying the variables + completion
I created 2 variables using :
CREATE VARIABLE my_pkg.g_dat_deb varchar(11);
CREATE VARIABLE my_pkg.g_dat_fin varchar(11);
When I try to display them, I can only see them when prefixing by the schema :
bdd13=> \dV
"Did not find any schema variables."
bdd13=> \dV my_pkg.*
                                               List of variables
   Schema   |      Name      |         Type          | Is nullable | Default | Owner | Transactional end action
------------+----------------+-----------------------+-------------+---------+-------+--------------------------
 my_pkg| g_dat_deb   | character varying(11) | t           |         | myowner   |
 my_pkg| g_dat_fin     | character varying(11) | t           |         | myowner   |
(3 rows)

bdd13=> \dV my_pkg
Did not find any schema variable named "my_pck".
NB : Using this template, functions are returned, maybe variables should also be listed ? (here by querying on "my_pkg%")
cts_get13=> \dV my_p [TAB]
=> completion using [TAB] key is not working

Is this normal that I cannot see all the variables when not specifying any schema ?
Also the completion works for functions, but not for variable.
That's just some bonus but it might be good to have it.

I think the way variables are listed using \dV should match with \df for querying functions

3) Any way to define CONSTANTs ?
We already talked a bit about this subject and also Gilles Darold introduces it in this mailing-list topic but I'd like to insist on it.
I think it would be nice to have a way to say that a variable should not be changed once defined.
Maybe it's hard to implement and can be implemented later, but I just want to know if this concern is open.


Otherwise the documentation looks good to me.

Regards

Rémi

  152. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2020-02-26T20:40:02Z

    st 26. 2. 2020 v 15:54 odesílatel remi duval <remi.duval@cheops.fr> napsal:

> The following review has been posted through the commitfest application:
> make installcheck-world:  not tested
> Implements feature:       tested, passed
> Spec compliant:           tested, failed
> Documentation:            tested, failed
>
> Hello Pavel
>
> First thanks for working on this patch cause it might be really helpful
> for those of us trying to migrate PL code between RDBMs.
>
> I tried your patch for migrating an Oracle package body to PL/pgSQL after
> also testing a solution using set_config and current_setting (which works
> but I'm not really satisfied by this workaround solution).
>
> So I compiled latest postgres sources from github on Linux (redhat 7.7)
> using only your patch number 1 (I did not try the second part of the patch).
>
> For my use-case it's working great, performances are excellent (compared
> to other solution for porting "package variables").
> I did not test all the features involved by the patch (especially ALTER
> variable).
>

ALTER VARIABLE is not implemented yet


> I have some feedback however :
>
> 1) Failure when using pg_dump 13 on a 12.1 database
>
> When exporting a 12.1 database using pg_dump from the latest development
> sources I have an error regarding variables export
>
> [pg12@TST-LINUX-PG-03 ~]$ /opt/postgres12/pg12/bin/pg_dump -h localhost
> -p 5432 -U postgres -f dump_pg12.sql database1
> pg_dump: error: query failed: ERROR:  relation "pg_variable" does not exist
> LINE 1: ...og.pg_get_expr(v.vardefexpr,0) as vardefexpr FROM pg_variabl...
>                                                              ^
> pg_dump: error: query was: SELECT v.tableoid, v.oid, v.varname,
> v.vareoxaction, v.varnamespace,
> (SELECT rolname FROM pg_catalog.pg_roles WHERE oid = varowner) AS rolname
> , (SELECT pg_catalog.array_agg(acl ORDER BY row_n) FROM (SELECT acl, row_n
> FROM
> pg_catalog.unnest(coalesce(v.varacl,pg_catalog.acldefault('V',v.varowner)))
>  WITH ORDINALITY AS perm(acl,row_n)
>  WHERE NOT EXISTS ( SELECT 1 FROM
> pg_catalog.unnest(coalesce(pip.initprivs,pg_catalog.acldefault('V',v.varowner)))
> AS init(init_acl)
>                     WHERE acl = init_acl)) as foo) as varacl, ...:
>
> I think that it should have worked anyway cause the documentation states :
> https://www.postgresql.org/docs/current/upgrading.html
> "It is recommended that you use the pg_dump and pg_dumpall programs from
> the newer version of PostgreSQL, to take advantage of enhancements that
> might have been made in these programs." (that's what I did here)
>
> I think there should be a way to avoid dumping the variable if they don't
> exist, should'nt it ?
>

There was a protection against dump 11, but now it should be Postgres 12.
Fixed.


>
> 2) Displaying the variables + completion
> I created 2 variables using :
> CREATE VARIABLE my_pkg.g_dat_deb varchar(11);
> CREATE VARIABLE my_pkg.g_dat_fin varchar(11);
> When I try to display them, I can only see them when prefixing by the
> schema :
> bdd13=> \dV
> "Did not find any schema variables."
> bdd13=> \dV my_pkg.*
>                                                List of variables
>    Schema   |      Name      |         Type          | Is nullable |
> Default | Owner | Transactional end action
>
> ------------+----------------+-----------------------+-------------+---------+-------+--------------------------
>  my_pkg| g_dat_deb   | character varying(11) | t           |         |
> myowner   |
>  my_pkg| g_dat_fin     | character varying(11) | t           |         |
> myowner   |
> (3 rows)
>

it is ok - it depends on SEARCH_PATH value


> bdd13=> \dV my_pkg
> Did not find any schema variable named "my_pck".
> NB : Using this template, functions are returned, maybe variables should
> also be listed ? (here by querying on "my_pkg%")
> cts_get13=> \dV my_p [TAB]
> => completion using [TAB] key is not working
>
> Is this normal that I cannot see all the variables when not specifying any
> schema ?
> Also the completion works for functions, but not for variable.
> That's just some bonus but it might be good to have it.
>
> I think the way variables are listed using \dV should match with \df for
> querying functions
>

fixed


> 3) Any way to define CONSTANTs ?
> We already talked a bit about this subject and also Gilles Darold
> introduces it in this mailing-list topic but I'd like to insist on it.
> I think it would be nice to have a way to say that a variable should not
> be changed once defined.
> Maybe it's hard to implement and can be implemented later, but I just want
> to know if this concern is open.
>

This topic is open. I tried to play with it. The problem is syntax. When I
try to reproduce syntax from PLpgSQL, then I need to introduce new reserved
keyword. So my initial idea was wrong.
We need to open discussion about implementable syntax. For this moment you
can use a workaround - any schema variable without WRITE right is constant.
Implementation is easy. Design of syntax is harder.

please see attached patch

Regards

Pavel


>
> Otherwise the documentation looks good to me.
>
> Regards
>
> Rémi

  153. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2020-02-27T14:37:48Z

    Hi


> 3) Any way to define CONSTANTs ?
> We already talked a bit about this subject and also Gilles Darold
> introduces it in this mailing-list topic but I'd like to insist on it.
> I think it would be nice to have a way to say that a variable should not
> be changed once defined.
> Maybe it's hard to implement and can be implemented later, but I just want
> to know if this concern is open.
>

I played little bit with it and I didn't find any nice solution, but maybe
I found the solution. I had ideas about some variants, but almost all time
I had a problem with parser's shifts because all potential keywords are not
reserved.

last variant, but maybe best is using keyword WITH

So the syntax can looks like

CREATE [ TEMP ] VARIABLE varname [ AS ] type [ NOT NULL ] [ DEFAULT
expression ] [ WITH [ OPTIONS ] '(' ... ')' ] ]

What do you think about this syntax? It doesn't need any new keyword, and
it easy to enhance it.

CREATE VARIABLE foo AS int DEFAULT 10 WITH OPTIONS ( CONSTANT);

?

Regards

Pavel

  154. RE: proposal: schema variables

    DUVAL REMI <remi.duval@cheops.fr> — 2020-02-27T14:59:12Z

    Hello Pavel.

That looks pretty good to me !

I’m adding Philippe Beaudoin who was also interested in this topic.

Recap : We were looking for a way to separate variable from constants in the “Schema Variables” proposition from Pavel.
Pavel was saying that there are some limitations regarding the keywords we can use, as the community don’t want to introduce too much new keywords in Postgres SQL (PL/pgSQL is a different list of keywords).
“CONSTANT” is not a keyword in SQL for Now (though it is one in PL/pgSQL).
Pavel’s syntax allow to use it as a keyword in the “WITH OPTIONS” clause that is already supported.
… I think it’s a good idea.

The list of keywords is defined in : postgresql\src\include\parser\kwlist.h

Pavel, I saw that in DB2, those variables are called “Global Variables”, is it something we can consider changing, or do you prefer to keep using the “Schema Variable” name ?


De : Pavel Stehule [mailto:pavel.stehule@gmail.com]
Envoyé : jeudi 27 février 2020 15:38
À : DUVAL REMI <REMI.DUVAL@CHEOPS.FR>
Cc : PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Objet : Re: proposal: schema variables


Hi


3) Any way to define CONSTANTs ?
We already talked a bit about this subject and also Gilles Darold introduces it in this mailing-list topic but I'd like to insist on it.
I think it would be nice to have a way to say that a variable should not be changed once defined.
Maybe it's hard to implement and can be implemented later, but I just want to know if this concern is open.

I played little bit with it and I didn't find any nice solution, but maybe I found the solution. I had ideas about some variants, but almost all time I had a problem with parser's shifts because all potential keywords are not reserved.

last variant, but maybe best is using keyword WITH

So the syntax can looks like

CREATE [ TEMP ] VARIABLE varname [ AS ] type [ NOT NULL ] [ DEFAULT expression ] [ WITH [ OPTIONS ] '(' ... ')' ] ]

What do you think about this syntax? It doesn't need any new keyword, and it easy to enhance it.

CREATE VARIABLE foo AS int DEFAULT 10 WITH OPTIONS ( CONSTANT);

?

Regards

Pavel

  155. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2020-02-27T15:09:44Z

    čt 27. 2. 2020 v 15:59 odesílatel DUVAL REMI <REMI.DUVAL@cheops.fr> napsal:

> Hello Pavel.
>
>
>
> That looks pretty good to me !
>
>
>
> I’m adding Philippe Beaudoin who was also interested in this topic.
>
>
>
> Recap : We were looking for a way to separate variable from constants in
> the “Schema Variables” proposition from Pavel.
>
> Pavel was saying that there are some limitations regarding the keywords we
> can use, as the community don’t want to introduce too much new keywords in
> Postgres SQL (PL/pgSQL is a different list of keywords).
>
> “CONSTANT” is not a keyword in SQL for Now (though it is one in PL/pgSQL).
>
> Pavel’s syntax allow to use it as a keyword in the “WITH OPTIONS” clause
> that is already supported.
>
> … I think it’s a good idea.
>
>
>
> The list of keywords is defined in : postgresql\src\include\parser\kwlist.h
>
>
>
> Pavel, I saw that in DB2, those variables are called “Global Variables”,
> is it something we can consider changing, or do you prefer to keep using
> the “Schema Variable” name ?
>

It is most hard question. Global variables has sense, but when we will use
it in plpgsql, then this name can be little bit confusing. Personally I
prefer "schema variable" although my opinion is not too strong. This name
more signalize so this is more generic, more database related than some
special kind of plpgsql variables. Now, I think so maybe is better to use
schema variables, because there is different syntax then global temp
tables. Variables are global by design. So in this moment I prefer the name
"schema variables". It can be used as global variables in plpgsql, but it
is one case.

Pavel



>
>
>
> *De :* Pavel Stehule [mailto:pavel.stehule@gmail.com]
> *Envoyé :* jeudi 27 février 2020 15:38
> *À :* DUVAL REMI <REMI.DUVAL@CHEOPS.FR>
> *Cc :* PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
> *Objet :* Re: proposal: schema variables
>
>
>
>
>
> Hi
>
>
>
>
> 3) Any way to define CONSTANTs ?
> We already talked a bit about this subject and also Gilles Darold
> introduces it in this mailing-list topic but I'd like to insist on it.
> I think it would be nice to have a way to say that a variable should not
> be changed once defined.
> Maybe it's hard to implement and can be implemented later, but I just want
> to know if this concern is open.
>
>
>
> I played little bit with it and I didn't find any nice solution, but maybe
> I found the solution. I had ideas about some variants, but almost all time
> I had a problem with parser's shifts because all potential keywords are not
> reserved.
>
>
>
> last variant, but maybe best is using keyword WITH
>
>
>
> So the syntax can looks like
>
>
>
> CREATE [ TEMP ] VARIABLE varname [ AS ] type [ NOT NULL ] [ DEFAULT
> expression ] [ WITH [ OPTIONS ] '(' ... ')' ] ]
>
>
>
> What do you think about this syntax? It doesn't need any new keyword, and
> it easy to enhance it.
>
>
>
> CREATE VARIABLE foo AS int DEFAULT 10 WITH OPTIONS ( CONSTANT);
>
>
>
> ?
>
>
>
> Regards
>
>
>
> Pavel
>
>
>
>
>

  156. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2020-02-28T15:30:28Z

    čt 27. 2. 2020 v 15:37 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
napsal:

>
> Hi
>
>
>> 3) Any way to define CONSTANTs ?
>> We already talked a bit about this subject and also Gilles Darold
>> introduces it in this mailing-list topic but I'd like to insist on it.
>> I think it would be nice to have a way to say that a variable should not
>> be changed once defined.
>> Maybe it's hard to implement and can be implemented later, but I just
>> want to know if this concern is open.
>>
>
> I played little bit with it and I didn't find any nice solution, but maybe
> I found the solution. I had ideas about some variants, but almost all time
> I had a problem with parser's shifts because all potential keywords are not
> reserved.
>
> last variant, but maybe best is using keyword WITH
>
> So the syntax can looks like
>
> CREATE [ TEMP ] VARIABLE varname [ AS ] type [ NOT NULL ] [ DEFAULT
> expression ] [ WITH [ OPTIONS ] '(' ... ')' ] ]
>
> What do you think about this syntax? It doesn't need any new keyword, and
> it easy to enhance it.
>
> CREATE VARIABLE foo AS int DEFAULT 10 WITH OPTIONS ( CONSTANT);
>

After some more thinking and because in other patch I support syntax CREATE
TRANSACTION VARIABLE ... I change my opinion and implemented support for
syntax CREATE IMMUTABLE VARIABLE for define constants.

See attached patch

Regards

Pavel


>
> ?
>
> Regards
>
> Pavel
>
>
>

  157. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2020-02-29T09:09:47Z

    pá 28. 2. 2020 v 16:30 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
napsal:

>
>
> čt 27. 2. 2020 v 15:37 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
> napsal:
>
>>
>> Hi
>>
>>
>>> 3) Any way to define CONSTANTs ?
>>> We already talked a bit about this subject and also Gilles Darold
>>> introduces it in this mailing-list topic but I'd like to insist on it.
>>> I think it would be nice to have a way to say that a variable should not
>>> be changed once defined.
>>> Maybe it's hard to implement and can be implemented later, but I just
>>> want to know if this concern is open.
>>>
>>
>> I played little bit with it and I didn't find any nice solution, but
>> maybe I found the solution. I had ideas about some variants, but almost all
>> time I had a problem with parser's shifts because all potential keywords
>> are not reserved.
>>
>> last variant, but maybe best is using keyword WITH
>>
>> So the syntax can looks like
>>
>> CREATE [ TEMP ] VARIABLE varname [ AS ] type [ NOT NULL ] [ DEFAULT
>> expression ] [ WITH [ OPTIONS ] '(' ... ')' ] ]
>>
>> What do you think about this syntax? It doesn't need any new keyword, and
>> it easy to enhance it.
>>
>> CREATE VARIABLE foo AS int DEFAULT 10 WITH OPTIONS ( CONSTANT);
>>
>
> After some more thinking and because in other patch I support syntax
> CREATE TRANSACTION VARIABLE ... I change my opinion and implemented support
> for
> syntax CREATE IMMUTABLE VARIABLE for define constants.
>

second try to fix pg_dump

Regards

Pavel


>
> See attached patch
>
> Regards
>
> Pavel
>
>
>>
>> ?
>>
>> Regards
>>
>> Pavel
>>
>>
>>

  158. Re: proposal: schema variables

    Asif Rehman <asifr.rehman@gmail.com> — 2020-03-05T14:10:49Z

    On Sat, Feb 29, 2020 at 2:10 PM Pavel Stehule <pavel.stehule@gmail.com>
wrote:

>
>
> pá 28. 2. 2020 v 16:30 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
> napsal:
>
>>
>>
>> čt 27. 2. 2020 v 15:37 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
>> napsal:
>>
>>>
>>> Hi
>>>
>>>
>>>> 3) Any way to define CONSTANTs ?
>>>> We already talked a bit about this subject and also Gilles Darold
>>>> introduces it in this mailing-list topic but I'd like to insist on it.
>>>> I think it would be nice to have a way to say that a variable should
>>>> not be changed once defined.
>>>> Maybe it's hard to implement and can be implemented later, but I just
>>>> want to know if this concern is open.
>>>>
>>>
>>> I played little bit with it and I didn't find any nice solution, but
>>> maybe I found the solution. I had ideas about some variants, but almost all
>>> time I had a problem with parser's shifts because all potential keywords
>>> are not reserved.
>>>
>>> last variant, but maybe best is using keyword WITH
>>>
>>> So the syntax can looks like
>>>
>>> CREATE [ TEMP ] VARIABLE varname [ AS ] type [ NOT NULL ] [ DEFAULT
>>> expression ] [ WITH [ OPTIONS ] '(' ... ')' ] ]
>>>
>>> What do you think about this syntax? It doesn't need any new keyword,
>>> and it easy to enhance it.
>>>
>>> CREATE VARIABLE foo AS int DEFAULT 10 WITH OPTIONS ( CONSTANT);
>>>
>>
>> After some more thinking and because in other patch I support syntax
>> CREATE TRANSACTION VARIABLE ... I change my opinion and implemented support
>> for
>> syntax CREATE IMMUTABLE VARIABLE for define constants.
>>
>
> second try to fix pg_dump
>
> Regards
>
> Pavel
>
>
>>
>> See attached patch
>>
>> Regards
>>
>> Pavel
>>
>>
>>>
>>> ?
>>>
>>> Regards
>>>
>>> Pavel
>>>
>>>
>>>
Hi Pavel,

I have been reviewing the latest patch (schema-variables-20200229.patch.gz)
and here are few comments:

1- There is a compilation error, when compiled with --with-llvm enabled on
CentOS 7.

llvmjit_expr.c: In function ‘llvm_compile_expr’:
llvmjit_expr.c:1090:5: warning: initialization from incompatible pointer
type [enabled by default]
     build_EvalXFunc(b, mod, "ExecEvalParamVariable",
     ^
llvmjit_expr.c:1090:5: warning: (near initialization for ‘(anonymous)[0]’)
[enabled by default]
llvmjit_expr.c:1090:5: warning: initialization from incompatible pointer
type [enabled by default]
llvmjit_expr.c:1090:5: warning: (near initialization for ‘(anonymous)[0]’)
[enabled by default]
llvmjit_expr.c:1090:5: warning: initialization from incompatible pointer
type [enabled by default]
llvmjit_expr.c:1090:5: warning: (near initialization for ‘(anonymous)[0]’)
[enabled by default]
llvmjit_expr.c:1090:5: warning: passing argument 5 of ‘build_EvalXFuncInt’
from incompatible pointer type [enabled by default]
llvmjit_expr.c:60:21: note: expected ‘struct ExprEvalStep *’ but argument
is of type ‘LLVMValueRef’
 static LLVMValueRef build_EvalXFuncInt(LLVMBuilderRef b, LLVMModuleRef mod,
                     ^
llvmjit_expr.c:1092:29: error: ‘i’ undeclared (first use in this function)
     LLVMBuildBr(b, opblocks[i + 1]);
                             ^
llvmjit_expr.c:1092:29: note: each undeclared identifier is reported only
once for each function it appears in
make[2]: *** [llvmjit_expr.o] Error 1



After looking into it, it turns out that:
- parameter order was incorrect in build_EvalXFunc()
- LLVMBuildBr() is using the undeclared variable 'i' whereas it should be
using 'opno'.


2- Similarly, If the default expression is referencing a function or object,
dependency should be marked, so if the function is not dropped silently.
otherwise, a cache lookup error will come.

postgres=# create or replace function foofunc() returns timestamp as $$
begin return now(); end; $$ language plpgsql;
CREATE FUNCTION
postgres=# create schema test;
CREATE SCHEMA
postgres=# create variable test.v1 as timestamp default foofunc();
CREATE VARIABLE
postgres=# drop function foofunc();
DROP FUNCTION
postgres=# select test.v1;
ERROR:  cache lookup failed for function 16437



3- Variable DEFAULT expression is apparently being evaluated at the time of
first access. whereas I think that It should be at the time of variable
creation. consider the following example:

postgres=# create variable test.v2 as timestamp default now();
CREATE VARIABLE
postgres=# select now();
              now
-------------------------------
 2020-03-05 12:13:29.775373+00
(1 row)
postgres=# select test.v2;
             v2
----------------------------
 2020-03-05 12:13:37.192317 -- I was expecting this to be earlier than the
above timestamp.
(1 row)

postgres=# select test.v2;
             v2
----------------------------
 2020-03-05 12:13:37.192317
(1 row)
postgres=# let test.v2 = default;
LET
postgres=# select test.v2;
             v2
----------------------------
 2020-03-05 12:14:07.538615
(1 row)


To continue my testing of the patch I made few fixes for the above-mentioned
comments. The patch for those changes is attached if it could be of any use.

--
Asif Rehman
Highgo Software (Canada/China/Pakistan)
URL : www.highgo.ca

  159. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2020-03-05T17:54:29Z

    čt 5. 3. 2020 v 15:11 odesílatel Asif Rehman <asifr.rehman@gmail.com>
napsal:

>
>
> On Sat, Feb 29, 2020 at 2:10 PM Pavel Stehule <pavel.stehule@gmail.com>
> wrote:
>
>>
>>
>> pá 28. 2. 2020 v 16:30 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
>> napsal:
>>
>>>
>>>
>>> čt 27. 2. 2020 v 15:37 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
>>> napsal:
>>>
>>>>
>>>> Hi
>>>>
>>>>
>>>>> 3) Any way to define CONSTANTs ?
>>>>> We already talked a bit about this subject and also Gilles Darold
>>>>> introduces it in this mailing-list topic but I'd like to insist on it.
>>>>> I think it would be nice to have a way to say that a variable should
>>>>> not be changed once defined.
>>>>> Maybe it's hard to implement and can be implemented later, but I just
>>>>> want to know if this concern is open.
>>>>>
>>>>
>>>> I played little bit with it and I didn't find any nice solution, but
>>>> maybe I found the solution. I had ideas about some variants, but almost all
>>>> time I had a problem with parser's shifts because all potential keywords
>>>> are not reserved.
>>>>
>>>> last variant, but maybe best is using keyword WITH
>>>>
>>>> So the syntax can looks like
>>>>
>>>> CREATE [ TEMP ] VARIABLE varname [ AS ] type [ NOT NULL ] [ DEFAULT
>>>> expression ] [ WITH [ OPTIONS ] '(' ... ')' ] ]
>>>>
>>>> What do you think about this syntax? It doesn't need any new keyword,
>>>> and it easy to enhance it.
>>>>
>>>> CREATE VARIABLE foo AS int DEFAULT 10 WITH OPTIONS ( CONSTANT);
>>>>
>>>
>>> After some more thinking and because in other patch I support syntax
>>> CREATE TRANSACTION VARIABLE ... I change my opinion and implemented support
>>> for
>>> syntax CREATE IMMUTABLE VARIABLE for define constants.
>>>
>>
>> second try to fix pg_dump
>>
>> Regards
>>
>> Pavel
>>
>>
>>>
>>> See attached patch
>>>
>>> Regards
>>>
>>> Pavel
>>>
>>>
>>>>
>>>> ?
>>>>
>>>> Regards
>>>>
>>>> Pavel
>>>>
>>>>
>>>>
> Hi Pavel,
>
> I have been reviewing the latest patch (schema-variables-20200229.patch.gz)
> and here are few comments:
>
> 1- There is a compilation error, when compiled with --with-llvm enabled on
> CentOS 7.
>
> llvmjit_expr.c: In function ‘llvm_compile_expr’:
> llvmjit_expr.c:1090:5: warning: initialization from incompatible pointer
> type [enabled by default]
>      build_EvalXFunc(b, mod, "ExecEvalParamVariable",
>      ^
> llvmjit_expr.c:1090:5: warning: (near initialization for ‘(anonymous)[0]’)
> [enabled by default]
> llvmjit_expr.c:1090:5: warning: initialization from incompatible pointer
> type [enabled by default]
> llvmjit_expr.c:1090:5: warning: (near initialization for ‘(anonymous)[0]’)
> [enabled by default]
> llvmjit_expr.c:1090:5: warning: initialization from incompatible pointer
> type [enabled by default]
> llvmjit_expr.c:1090:5: warning: (near initialization for ‘(anonymous)[0]’)
> [enabled by default]
> llvmjit_expr.c:1090:5: warning: passing argument 5 of ‘build_EvalXFuncInt’
> from incompatible pointer type [enabled by default]
> llvmjit_expr.c:60:21: note: expected ‘struct ExprEvalStep *’ but argument
> is of type ‘LLVMValueRef’
>  static LLVMValueRef build_EvalXFuncInt(LLVMBuilderRef b, LLVMModuleRef
> mod,
>                      ^
> llvmjit_expr.c:1092:29: error: ‘i’ undeclared (first use in this function)
>      LLVMBuildBr(b, opblocks[i + 1]);
>                              ^
> llvmjit_expr.c:1092:29: note: each undeclared identifier is reported only
> once for each function it appears in
> make[2]: *** [llvmjit_expr.o] Error 1
>
>
>
> After looking into it, it turns out that:
> - parameter order was incorrect in build_EvalXFunc()
> - LLVMBuildBr() is using the undeclared variable 'i' whereas it should be
> using 'opno'.
>
>
> 2- Similarly, If the default expression is referencing a function or
> object,
> dependency should be marked, so if the function is not dropped silently.
> otherwise, a cache lookup error will come.
>
> postgres=# create or replace function foofunc() returns timestamp as $$
> begin return now(); end; $$ language plpgsql;
> CREATE FUNCTION
> postgres=# create schema test;
> CREATE SCHEMA
> postgres=# create variable test.v1 as timestamp default foofunc();
> CREATE VARIABLE
> postgres=# drop function foofunc();
> DROP FUNCTION
> postgres=# select test.v1;
> ERROR:  cache lookup failed for function 16437
>
>
Thank you for this analyze and patches. I merged them to attached patch



>
> 3- Variable DEFAULT expression is apparently being evaluated at the time of
> first access. whereas I think that It should be at the time of variable
> creation. consider the following example:
>
> postgres=# create variable test.v2 as timestamp default now();
> CREATE VARIABLE
> postgres=# select now();
>               now
> -------------------------------
>  2020-03-05 12:13:29.775373+00
> (1 row)
> postgres=# select test.v2;
>              v2
> ----------------------------
>  2020-03-05 12:13:37.192317 -- I was expecting this to be earlier than
> the above timestamp.
> (1 row)
>
> postgres=# select test.v2;
>              v2
> ----------------------------
>  2020-03-05 12:13:37.192317
> (1 row)
> postgres=# let test.v2 = default;
> LET
> postgres=# select test.v2;
>              v2
> ----------------------------
>  2020-03-05 12:14:07.538615
> (1 row)
>
>
This is expected and wanted - same behave has plpgsql variables.

CREATE OR REPLACE FUNCTION public.foo()
 RETURNS void
 LANGUAGE plpgsql
AS $function$
declare x timestamp default current_timestamp;
begin
  raise notice '%', x;
end;
$function$

postgres=# select foo();
NOTICE:  2020-03-05 18:49:12.465054
┌─────┐
│ foo │
╞═════╡
│     │
└─────┘
(1 row)

postgres=# select foo();
NOTICE:  2020-03-05 18:49:13.255197
┌─────┐
│ foo │
╞═════╡
│     │
└─────┘
(1 row)

You can use

CREATE VARIABLE cuser AS text DEFAULT session_user;

Has not any sense to use a value from creating time

And a analogy with CREATE TABLE

CREATE TABLE fooo(a timestamp DEFAULT current_timestamp) -- there is not a
create time timestamp


I fixed buggy behave of IMMUTABLE variables

Regards

Pavel



>
> To continue my testing of the patch I made few fixes for the
> above-mentioned
> comments. The patch for those changes is attached if it could be of any
> use.
>
> --
> Asif Rehman
> Highgo Software (Canada/China/Pakistan)
> URL : www.highgo.ca
>
>

  160. RE: proposal: schema variables

    DUVAL REMI <remi.duval@cheops.fr> — 2020-03-06T15:44:14Z

    Hello Pavel

I tested your patch again and I think things are better now, close to perfect for me.


1)      Patch review


-          We can define CONSTANTs with CREATE IMMUTABLE VARIABLE … I’m really pleased with this

-          The previous bug I mentioned to you by private mail (see detail below) has been fixed and a non-regression test case has been added for it.

-          I’m now able to export a 12.1 database using pg_dump from the latest git HEAD (internal version 130000).

NB: the condition is “if internal_version < 130000 => don’t try to export any schema variable”.

Also I was able to test a use case for a complex Oracle package I migrated to PostgreSQL (it has a total of 194 variables and constants in it !).
The Oracle package has been migrated to a PostgreSQL schema with one routine per Oracle subprogram.
I’m able to run my business test case using schema variables on those routines and it’s giving me the expected result.

NB: Previous bug was
database1=> CREATE VARIABLE T0_var AS char(14);
CREATE VARIABLE
database1=> CREATE IMMUTABLE VARIABLE Taille_var AS numeric DEFAULT 14;
CREATE VARIABLE
database1=> LET T0_var = LPAD('999', trunc(Taille_var::NUMERIC)::INTEGER, '0');
ERROR:  schema variable "taille_var" is declared IMMUTABLE
database1=> LET T0_var = LPAD('999', trunc(Taille_var::NUMERIC)::INTEGER, '0');
ERROR:  variable "taille_var" has not valid content

ð  It’s now fixed !


2)      Regarding documentation

It’s pretty good except some small details :

-          sql-createvariable.html => IMMUTABLE parameter : The value of the variable cannot be changed. => I think an article is needed here (the)

-          sql-createvariable.html => ON COMMIT DROP : The ON COMMIT DROP clause specifies the bahaviour of      temporary => behaviour
Also there are “tabs” between words (here between “of” and “temporary”), making the paragraph look strange.

-          sql-createvariable.html => Maybe we should mention that the IMMUTABLE parameter (CREATE IMMUTABLE VARIABLE …) is the way to define global CONSTANTs, because people will look for a way to create global constants in the documentation and it would be good if they can find the CONSTANT word in it.
Also maybe it’s worth mentioning that “schema variables” relates to “global variables” (for the same purpose of people searching in the documentation).

-          sql-altervariable.html => sentence “These restrictions enforce that altering the owner doesn't do anything you couldn't do by dropping and recreating the variable.“ => not sure I understand this one. Isn’t it “does not do anything you could do” instead of “you couln’t do” .. but maybe it’s me

Otherwise, this is a really nice feature and I’m looking forward to have it in PostgreSQL.

Thanks a lot

Duval Rémi

De : Pavel Stehule [mailto:pavel.stehule@gmail.com]
Envoyé : jeudi 5 mars 2020 18:54
À : Asif Rehman <asifr.rehman@gmail.com>
Cc : DUVAL REMI <REMI.DUVAL@CHEOPS.FR>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Objet : Re: proposal: schema variables



čt 5. 3. 2020 v 15:11 odesílatel Asif Rehman <asifr.rehman@gmail.com<mailto:asifr.rehman@gmail.com>> napsal:


On Sat, Feb 29, 2020 at 2:10 PM Pavel Stehule <pavel.stehule@gmail.com<mailto:pavel.stehule@gmail.com>> wrote:


pá 28. 2. 2020 v 16:30 odesílatel Pavel Stehule <pavel.stehule@gmail.com<mailto:pavel.stehule@gmail.com>> napsal:


čt 27. 2. 2020 v 15:37 odesílatel Pavel Stehule <pavel.stehule@gmail.com<mailto:pavel.stehule@gmail.com>> napsal:

Hi


3) Any way to define CONSTANTs ?
We already talked a bit about this subject and also Gilles Darold introduces it in this mailing-list topic but I'd like to insist on it.
I think it would be nice to have a way to say that a variable should not be changed once defined.
Maybe it's hard to implement and can be implemented later, but I just want to know if this concern is open.

I played little bit with it and I didn't find any nice solution, but maybe I found the solution. I had ideas about some variants, but almost all time I had a problem with parser's shifts because all potential keywords are not reserved.

last variant, but maybe best is using keyword WITH

So the syntax can looks like

CREATE [ TEMP ] VARIABLE varname [ AS ] type [ NOT NULL ] [ DEFAULT expression ] [ WITH [ OPTIONS ] '(' ... ')' ] ]

What do you think about this syntax? It doesn't need any new keyword, and it easy to enhance it.

CREATE VARIABLE foo AS int DEFAULT 10 WITH OPTIONS ( CONSTANT);

After some more thinking and because in other patch I support syntax CREATE TRANSACTION VARIABLE ... I change my opinion and implemented support for
syntax CREATE IMMUTABLE VARIABLE for define constants.

second try to fix pg_dump

Regards

Pavel


See attached patch

Regards

Pavel


?

Regards

Pavel



Hi Pavel,

I have been reviewing the latest patch (schema-variables-20200229.patch.gz)
and here are few comments:

1- There is a compilation error, when compiled with --with-llvm enabled on
CentOS 7.

llvmjit_expr.c: In function ‘llvm_compile_expr’:
llvmjit_expr.c:1090:5: warning: initialization from incompatible pointer type [enabled by default]
     build_EvalXFunc(b, mod, "ExecEvalParamVariable",
     ^
llvmjit_expr.c:1090:5: warning: (near initialization for ‘(anonymous)[0]’) [enabled by default]
llvmjit_expr.c:1090:5: warning: initialization from incompatible pointer type [enabled by default]
llvmjit_expr.c:1090:5: warning: (near initialization for ‘(anonymous)[0]’) [enabled by default]
llvmjit_expr.c:1090:5: warning: initialization from incompatible pointer type [enabled by default]
llvmjit_expr.c:1090:5: warning: (near initialization for ‘(anonymous)[0]’) [enabled by default]
llvmjit_expr.c:1090:5: warning: passing argument 5 of ‘build_EvalXFuncInt’ from incompatible pointer type [enabled by default]
llvmjit_expr.c:60:21: note: expected ‘struct ExprEvalStep *’ but argument is of type ‘LLVMValueRef’
 static LLVMValueRef build_EvalXFuncInt(LLVMBuilderRef b, LLVMModuleRef mod,
                     ^
llvmjit_expr.c:1092:29: error: ‘i’ undeclared (first use in this function)
     LLVMBuildBr(b, opblocks[i + 1]);
                             ^
llvmjit_expr.c:1092:29: note: each undeclared identifier is reported only once for each function it appears in
make[2]: *** [llvmjit_expr.o] Error 1


After looking into it, it turns out that:
- parameter order was incorrect in build_EvalXFunc()
- LLVMBuildBr() is using the undeclared variable 'i' whereas it should be
using 'opno'.


2- Similarly, If the default expression is referencing a function or object,
dependency should be marked, so if the function is not dropped silently.
otherwise, a cache lookup error will come.

postgres=# create or replace function foofunc() returns timestamp as $$ begin return now(); end; $$ language plpgsql;
CREATE FUNCTION
postgres=# create schema test;
CREATE SCHEMA
postgres=# create variable test.v1 as timestamp default foofunc();
CREATE VARIABLE
postgres=# drop function foofunc();
DROP FUNCTION
postgres=# select test.v1;
ERROR:  cache lookup failed for function 16437

Thank you for this analyze and patches. I merged them to attached patch




3- Variable DEFAULT expression is apparently being evaluated at the time of
first access. whereas I think that It should be at the time of variable
creation. consider the following example:

postgres=# create variable test.v2 as timestamp default now();
CREATE VARIABLE
postgres=# select now();
              now
-------------------------------
 2020-03-05 12:13:29.775373+00
(1 row)
postgres=# select test.v2;
             v2
----------------------------
 2020-03-05 12:13:37.192317 -- I was expecting this to be earlier than the above timestamp.
(1 row)

postgres=# select test.v2;
             v2
----------------------------
 2020-03-05 12:13:37.192317
(1 row)
postgres=# let test.v2 = default;
LET
postgres=# select test.v2;
             v2
----------------------------
 2020-03-05 12:14:07.538615
(1 row)

This is expected and wanted - same behave has plpgsql variables.

CREATE OR REPLACE FUNCTION public.foo()
 RETURNS void
 LANGUAGE plpgsql
AS $function$
declare x timestamp default current_timestamp;
begin
  raise notice '%', x;
end;
$function$

postgres=# select foo();
NOTICE:  2020-03-05 18:49:12.465054
┌─────┐
│ foo │
╞═════╡
│     │
└─────┘
(1 row)

postgres=# select foo();
NOTICE:  2020-03-05 18:49:13.255197
┌─────┐
│ foo │
╞═════╡
│     │
└─────┘
(1 row)

You can use

CREATE VARIABLE cuser AS text DEFAULT session_user;

Has not any sense to use a value from creating time

And a analogy with CREATE TABLE

CREATE TABLE fooo(a timestamp DEFAULT current_timestamp) -- there is not a create time timestamp


I fixed buggy behave of IMMUTABLE variables

Regards

Pavel



To continue my testing of the patch I made few fixes for the above-mentioned
comments. The patch for those changes is attached if it could be of any use.

--
Asif Rehman
Highgo Software (Canada/China/Pakistan)
URL : www.highgo.ca<http://www.highgo.ca>

  161. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2020-03-06T18:54:35Z

    pá 6. 3. 2020 v 16:44 odesílatel DUVAL REMI <REMI.DUVAL@cheops.fr> napsal:

> Hello Pavel
>
>
>
> I tested your patch again and I think things are better now, close to
> perfect for me.
>
>
>
> *1)      **Patch review*
>
>
>
> -          We can define CONSTANTs with CREATE IMMUTABLE VARIABLE … I’m
> really pleased with this
>
> -          The previous bug I mentioned to you by private mail (see
> detail below) has been fixed and a non-regression test case has been added
> for it.
>
> -          I’m now able to export a 12.1 database using pg_dump from the
> latest git HEAD (internal version 130000).
>
> NB: the condition is “if internal_version < 130000 => don’t try to export
> any schema variable”.
>
>
>
> Also I was able to test a use case for a complex Oracle package I migrated
> to PostgreSQL (it has a total of 194 variables and constants in it !).
>
> The Oracle package has been migrated to a PostgreSQL schema with one
> routine per Oracle subprogram.
>
> I’m able to run my business test case using schema variables on those
> routines and it’s giving me the expected result.
>
>
>
> NB: Previous bug was
>
> database1=> CREATE VARIABLE T0_var AS char(14);
> CREATE VARIABLE
> database1=> CREATE IMMUTABLE VARIABLE Taille_var AS numeric DEFAULT 14;
> CREATE VARIABLE
> database1=> LET T0_var = LPAD('999', trunc(Taille_var::NUMERIC)::INTEGER,
> '0');
> *ERROR:  schema variable "taille_var" is declared IMMUTABLE*
> database1=> LET T0_var = LPAD('999', trunc(Taille_var::NUMERIC)::INTEGER,
> '0');
> *ERROR:  variable "taille_var" has not valid content*
>
> ð  It’s now fixed !
>
>
>
> *2)      **Regarding documentation *
>
>
>
> It’s pretty good except some small details :
>
> -          sql-createvariable.html => IMMUTABLE parameter : The value of
> *the* variable cannot be changed. => I think an article is needed here
> (the)
>

fixed

-          sql-createvariable.html => ON COMMIT DROP : The ON COMMIT DROP
> clause specifies the bahaviour of      temporary => behaviour
> Also there are “tabs” between words (here between “of” and “temporary”),
> making the paragraph look strange.
>

fixed

> -          sql-createvariable.html => Maybe we should mention that the
> IMMUTABLE parameter (CREATE IMMUTABLE VARIABLE …) is the way to define
> global CONSTANTs, because people will look for a way to create global
> constants in the documentation and it would be good if they can find the
> CONSTANT word in it.
> Also maybe it’s worth mentioning that “schema variables” relates to
> “global variables” (for the same purpose of people searching in the
> documentation).
>
probably it should be somewhere
https://www.postgresql.org/docs/current/plpgsql-porting.html

I wrote note there


> -          sql-altervariable.html => sentence “These restrictions enforce
> that altering the owner doesn't do anything you couldn't do by dropping and
> recreating the variable.“ => not sure I understand this one. Isn’t it
> “does not do anything you could do” instead of “you couln’t do” .. but
> maybe it’s me
>
This sentence is used more times (from alter_view0

To alter the owner, you must also be a direct or indirect member of the new
   owning role, and that role must have <literal>CREATE</literal> privilege
on
   the view's schema.  (These restrictions enforce that altering the owner
   doesn't do anything you couldn't do by dropping and recreating the view.
   However, a superuser can alter ownership of any view anyway.)


>
>
> Otherwise, this is a really nice feature and I’m looking forward to have
> it in PostgreSQL.
>

Thank you very much

updated patch attached



>
> Thanks a lot
>
>
>
> Duval Rémi
>
>
>
> *De :* Pavel Stehule [mailto:pavel.stehule@gmail.com]
> *Envoyé :* jeudi 5 mars 2020 18:54
> *À :* Asif Rehman <asifr.rehman@gmail.com>
> *Cc :* DUVAL REMI <REMI.DUVAL@CHEOPS.FR>; PostgreSQL Hackers <
> pgsql-hackers@lists.postgresql.org>
> *Objet :* Re: proposal: schema variables
>
>
>
>
>
>
>
> čt 5. 3. 2020 v 15:11 odesílatel Asif Rehman <asifr.rehman@gmail.com>
> napsal:
>
>
>
>
>
> On Sat, Feb 29, 2020 at 2:10 PM Pavel Stehule <pavel.stehule@gmail.com>
> wrote:
>
>
>
>
>
> pá 28. 2. 2020 v 16:30 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
> napsal:
>
>
>
>
>
> čt 27. 2. 2020 v 15:37 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
> napsal:
>
>
>
> Hi
>
>
>
>
> 3) Any way to define CONSTANTs ?
> We already talked a bit about this subject and also Gilles Darold
> introduces it in this mailing-list topic but I'd like to insist on it.
> I think it would be nice to have a way to say that a variable should not
> be changed once defined.
> Maybe it's hard to implement and can be implemented later, but I just want
> to know if this concern is open.
>
>
>
> I played little bit with it and I didn't find any nice solution, but maybe
> I found the solution. I had ideas about some variants, but almost all time
> I had a problem with parser's shifts because all potential keywords are not
> reserved.
>
>
>
> last variant, but maybe best is using keyword WITH
>
>
>
> So the syntax can looks like
>
>
>
> CREATE [ TEMP ] VARIABLE varname [ AS ] type [ NOT NULL ] [ DEFAULT
> expression ] [ WITH [ OPTIONS ] '(' ... ')' ] ]
>
>
>
> What do you think about this syntax? It doesn't need any new keyword, and
> it easy to enhance it.
>
>
>
> CREATE VARIABLE foo AS int DEFAULT 10 WITH OPTIONS ( CONSTANT);
>
>
>
> After some more thinking and because in other patch I support syntax
> CREATE TRANSACTION VARIABLE ... I change my opinion and implemented support
> for
>
> syntax CREATE IMMUTABLE VARIABLE for define constants.
>
>
>
> second try to fix pg_dump
>
>
>
> Regards
>
>
>
> Pavel
>
>
>
>
>
> See attached patch
>
>
>
> Regards
>
>
>
> Pavel
>
>
>
>
>
> ?
>
>
>
> Regards
>
>
>
> Pavel
>
>
>
>
>
>
>
> Hi Pavel,
>
>
>
> I have been reviewing the latest patch (schema-variables-20200229.patch.gz)
>
> and here are few comments:
>
>
>
> 1- There is a compilation error, when compiled with --with-llvm enabled on
>
> CentOS 7.
>
>
>
> llvmjit_expr.c: In function ‘llvm_compile_expr’:
>
> llvmjit_expr.c:1090:5: warning: initialization from incompatible pointer
> type [enabled by default]
>
>      build_EvalXFunc(b, mod, "ExecEvalParamVariable",
>
>      ^
>
> llvmjit_expr.c:1090:5: warning: (near initialization for ‘(anonymous)[0]’)
> [enabled by default]
>
> llvmjit_expr.c:1090:5: warning: initialization from incompatible pointer
> type [enabled by default]
>
> llvmjit_expr.c:1090:5: warning: (near initialization for ‘(anonymous)[0]’)
> [enabled by default]
>
> llvmjit_expr.c:1090:5: warning: initialization from incompatible pointer
> type [enabled by default]
>
> llvmjit_expr.c:1090:5: warning: (near initialization for ‘(anonymous)[0]’)
> [enabled by default]
>
> llvmjit_expr.c:1090:5: warning: passing argument 5 of ‘build_EvalXFuncInt’
> from incompatible pointer type [enabled by default]
>
> llvmjit_expr.c:60:21: note: expected ‘struct ExprEvalStep *’ but argument
> is of type ‘LLVMValueRef’
>
>  static LLVMValueRef build_EvalXFuncInt(LLVMBuilderRef b, LLVMModuleRef
> mod,
>
>                      ^
>
> llvmjit_expr.c:1092:29: error: ‘i’ undeclared (first use in this function)
>
>      LLVMBuildBr(b, opblocks[i + 1]);
>
>                              ^
>
> llvmjit_expr.c:1092:29: note: each undeclared identifier is reported only
> once for each function it appears in
>
> make[2]: *** [llvmjit_expr.o] Error 1
>
>
>
>
>
> After looking into it, it turns out that:
>
> - parameter order was incorrect in build_EvalXFunc()
>
> - LLVMBuildBr() is using the undeclared variable 'i' whereas it should be
>
> using 'opno'.
>
>
>
>
>
> 2- Similarly, If the default expression is referencing a function or
> object,
>
> dependency should be marked, so if the function is not dropped silently.
>
> otherwise, a cache lookup error will come.
>
>
>
> postgres=# create or replace function foofunc() returns timestamp as $$
> begin return now(); end; $$ language plpgsql;
>
> CREATE FUNCTION
>
> postgres=# create schema test;
>
> CREATE SCHEMA
>
> postgres=# create variable test.v1 as timestamp default foofunc();
>
> CREATE VARIABLE
>
> postgres=# drop function foofunc();
>
> DROP FUNCTION
>
> postgres=# select test.v1;
>
> ERROR:  cache lookup failed for function 16437
>
>
>
> Thank you for this analyze and patches. I merged them to attached patch
>
>
>
>
>
>
>
>
>
> 3- Variable DEFAULT expression is apparently being evaluated at the time of
>
> first access. whereas I think that It should be at the time of variable
>
> creation. consider the following example:
>
>
>
> postgres=# create variable test.v2 as timestamp default now();
>
> CREATE VARIABLE
>
> postgres=# select now();
>
>               now
>
> -------------------------------
>
>  2020-03-05 12:13:29.775373+00
>
> (1 row)
>
> postgres=# select test.v2;
>
>              v2
>
> ----------------------------
>
>  2020-03-05 12:13:37.192317 -- I was expecting this to be earlier than the
> above timestamp.
>
> (1 row)
>
>
>
> postgres=# select test.v2;
>
>              v2
>
> ----------------------------
>
>  2020-03-05 12:13:37.192317
>
> (1 row)
>
> postgres=# let test.v2 = default;
>
> LET
>
> postgres=# select test.v2;
>
>              v2
>
> ----------------------------
>
>  2020-03-05 12:14:07.538615
>
> (1 row)
>
>
>
> This is expected and wanted - same behave has plpgsql variables.
>
>
>
> CREATE OR REPLACE FUNCTION public.foo()
>  RETURNS void
>  LANGUAGE plpgsql
> AS $function$
> declare x timestamp default current_timestamp;
> begin
>   raise notice '%', x;
> end;
> $function$
>
>
>
> postgres=# select foo();
> NOTICE:  2020-03-05 18:49:12.465054
> ┌─────┐
> │ foo │
> ╞═════╡
> │     │
> └─────┘
> (1 row)
>
> postgres=# select foo();
> NOTICE:  2020-03-05 18:49:13.255197
> ┌─────┐
> │ foo │
> ╞═════╡
> │     │
> └─────┘
> (1 row)
>
>
>
> You can use
>
>
>
> CREATE VARIABLE cuser AS text DEFAULT session_user;
>
>
>
> Has not any sense to use a value from creating time
>
>
>
> And a analogy with CREATE TABLE
>
>
>
> CREATE TABLE fooo(a timestamp DEFAULT current_timestamp) -- there is not a
> create time timestamp
>
>
>
>
>
> I fixed buggy behave of IMMUTABLE variables
>
>
>
> Regards
>
>
>
> Pavel
>
>
>
>
>
>
>
> To continue my testing of the patch I made few fixes for the
> above-mentioned
>
> comments. The patch for those changes is attached if it could be of any
> use.
>
>
>
> --
>
> Asif Rehman
>
> Highgo Software (Canada/China/Pakistan)
> URL : www.highgo.ca
>
>

  162. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2020-03-07T21:15:15Z

    Hi

I fixed the some ugly parts of this patch - now the LET x = DEFAULT, LET x
= NULL is processed by more usual way.
Statement LET is doesn't switch between STMT_UTILITY and STMT_PLAN_UTILITY
like before. It is only STMT_PLAN_UTILITY statement.

Regards

Pavel

  163. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2020-03-08T18:12:07Z

    so 7. 3. 2020 v 22:15 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
napsal:

> Hi
>
> I fixed the some ugly parts of this patch - now the LET x = DEFAULT, LET x
> = NULL is processed by more usual way.
> Statement LET is doesn't switch between STMT_UTILITY and STMT_PLAN_UTILITY
> like before. It is only STMT_PLAN_UTILITY statement.
>

I did some cleaning - I mainly replaced CMD_PLAN_UTILITY by CMD_LET because
there is not another similar statement in queue.

Regards

Pavel


> Regards
>
> Pavel
>

  164. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2020-03-13T18:44:33Z

    Hi

ne 8. 3. 2020 v 19:12 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
napsal:

>
>
> so 7. 3. 2020 v 22:15 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
> napsal:
>
>> Hi
>>
>> I fixed the some ugly parts of this patch - now the LET x = DEFAULT, LET
>> x = NULL is processed by more usual way.
>> Statement LET is doesn't switch between STMT_UTILITY and
>> STMT_PLAN_UTILITY like before. It is only STMT_PLAN_UTILITY statement.
>>
>
> I did some cleaning - I mainly replaced CMD_PLAN_UTILITY by CMD_LET
> because there is not another similar statement in queue.
>

 another cleaning - I repleaced CMD_LET by CMD_SELECT_UTILITY. This name is
more illustrative, and little bit cleaned code.

CMD_SELECT_UTILITY is mix of CMD_SELECT and CMD_UTILITY. It allows PREPARE
and parametrizations like CMD_SELECT. But execution is like CMD_UTILITY by
own utility routine with explicit dest receiver setting. This design
doesn't need to introduce new executor and planner nodes (like ModifyTable
and ModifyTablePath).

Regards

Pavel



> Regards
>
> Pavel
>
>
>> Regards
>>
>> Pavel
>>
>

  165. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2020-03-18T05:58:30Z

    Hi

fresh patch - rebase and fix issue reported by Remi - broken usage CREATE
VARIABLE inside PLpgSQL

Regards

Pavel

  166. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2020-03-20T07:18:45Z

    čt 19. 3. 2020 v 10:43 odesílatel DUVAL REMI <REMI.DUVAL@cheops.fr> napsal:

> Hello
>
>
>
> I played around with the ALTER VARIABLE statement to make sure it’s OK and
> it seems fine to me.
>
>
>
> Another last thing before commiting.
>
>
>
> I agree with Thomas Vondra, that this part might/should be simplified :
>
>
>
> [ { ON COMMIT DROP | ON { TRANSACTIONAL | TRANSACTION } END RESET } ]
>
>
>
> I would only allow “ON TRANSACTION END RESET”.
>
> I think we don’t need both here.
>
> Philippe Beaudoin was indeed talking about the TRANSACTIONAL keyword, but
> that would have make sense (and I think that’s what he meant) , if you
> could do something like “CREATE [NON] TRANSACTIONAL VARIABLE …”.
>
> But here I don’t think that the ON TRANSACTIONAL END RESET has any sense
> in English, and it only complicates the syntax.
>
>
>
> Maybe Thomas Vondra (if it’s him) would be more inclined to commit the
> patch if it has this more simple syntax has he requested.
>
>
>
> What do you think ?
>

I removed TRANSACTIONAL from this clause, see attachement change.diff

Updated patch attached.

I hope it would be the last touch, making it fully ready for a commiter.
>

Thank you very much for review and testing

Pavel

>

  167. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2020-03-20T08:28:22Z

    pá 20. 3. 2020 v 8:18 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
napsal:

>
>
> čt 19. 3. 2020 v 10:43 odesílatel DUVAL REMI <REMI.DUVAL@cheops.fr>
> napsal:
>
>> Hello
>>
>>
>>
>> I played around with the ALTER VARIABLE statement to make sure it’s OK
>> and it seems fine to me.
>>
>>
>>
>> Another last thing before commiting.
>>
>>
>>
>> I agree with Thomas Vondra, that this part might/should be simplified :
>>
>>
>>
>> [ { ON COMMIT DROP | ON { TRANSACTIONAL | TRANSACTION } END RESET } ]
>>
>>
>>
>> I would only allow “ON TRANSACTION END RESET”.
>>
>> I think we don’t need both here.
>>
>> Philippe Beaudoin was indeed talking about the TRANSACTIONAL keyword, but
>> that would have make sense (and I think that’s what he meant) , if you
>> could do something like “CREATE [NON] TRANSACTIONAL VARIABLE …”.
>>
>> But here I don’t think that the ON TRANSACTIONAL END RESET has any sense
>> in English, and it only complicates the syntax.
>>
>>
>>
>> Maybe Thomas Vondra (if it’s him) would be more inclined to commit the
>> patch if it has this more simple syntax has he requested.
>>
>>
>>
>> What do you think ?
>>
>
> I removed TRANSACTIONAL from this clause, see attachement change.diff
>
> Updated patch attached.
>
> I hope it would be the last touch, making it fully ready for a commiter.
>>
>
> Thank you very much for review and testing
>

documentation fix



> Pavel
>
>>

  168. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2020-03-22T07:40:33Z

    Hi

rebase

Regards

Pavel

  169. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2020-04-10T17:30:47Z

    Hi

rebase

Regards

Pavel

ne 22. 3. 2020 v 8:40 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
napsal:

> Hi
>
> rebase
>
> Regards
>
> Pavel
>

  170. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2020-05-21T10:10:15Z

    Hi

just rebase without any other changes

Regards

Pavel

  171. Re: proposal: schema variables

    Amit Kapila <amit.kapila16@gmail.com> — 2020-05-21T11:34:44Z

    On Thu, May 21, 2020 at 3:41 PM Pavel Stehule <pavel.stehule@gmail.com> wrote:
>
> Hi
>
> just rebase without any other changes
>

You seem to forget attaching the rebased patch.

-- 
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

  172. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2020-05-21T12:49:57Z

    čt 21. 5. 2020 v 13:34 odesílatel Amit Kapila <amit.kapila16@gmail.com>
napsal:

> On Thu, May 21, 2020 at 3:41 PM Pavel Stehule <pavel.stehule@gmail.com>
> wrote:
> >
> > Hi
> >
> > just rebase without any other changes
> >
>
> You seem to forget attaching the rebased patch.
>

I am sorry

attached.

Pavel


> --
> With Regards,
> Amit Kapila.
> EnterpriseDB: http://www.enterprisedb.com
>

  173. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2020-07-05T13:33:34Z

    čt 21. 5. 2020 v 14:49 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
napsal:

>
>
> čt 21. 5. 2020 v 13:34 odesílatel Amit Kapila <amit.kapila16@gmail.com>
> napsal:
>
>> On Thu, May 21, 2020 at 3:41 PM Pavel Stehule <pavel.stehule@gmail.com>
>> wrote:
>> >
>> > Hi
>> >
>> > just rebase without any other changes
>> >
>>
>> You seem to forget attaching the rebased patch.
>>
>
> I am sorry
>
> attached.
>

fresh rebase

Regards

Pavel


> Pavel
>
>
>> --
>> With Regards,
>> Amit Kapila.
>> EnterpriseDB: http://www.enterprisedb.com
>>
>

  174. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2020-07-06T08:17:07Z

    ne 5. 7. 2020 v 15:33 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
napsal:

>
>
> čt 21. 5. 2020 v 14:49 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
> napsal:
>
>>
>>
>> čt 21. 5. 2020 v 13:34 odesílatel Amit Kapila <amit.kapila16@gmail.com>
>> napsal:
>>
>>> On Thu, May 21, 2020 at 3:41 PM Pavel Stehule <pavel.stehule@gmail.com>
>>> wrote:
>>> >
>>> > Hi
>>> >
>>> > just rebase without any other changes
>>> >
>>>
>>> You seem to forget attaching the rebased patch.
>>>
>>
>> I am sorry
>>
>> attached.
>>
>
> fresh rebase
>

fix test build

Regards

Pavel


> Regards
>
> Pavel
>
>
>> Pavel
>>
>>
>>> --
>>> With Regards,
>>> Amit Kapila.
>>> EnterpriseDB: http://www.enterprisedb.com
>>>
>>

  175. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2020-07-11T04:44:24Z

    po 6. 7. 2020 v 10:17 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
napsal:

>
>
> ne 5. 7. 2020 v 15:33 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
> napsal:
>
>>
>>
>> čt 21. 5. 2020 v 14:49 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
>> napsal:
>>
>>>
>>>
>>> čt 21. 5. 2020 v 13:34 odesílatel Amit Kapila <amit.kapila16@gmail.com>
>>> napsal:
>>>
>>>> On Thu, May 21, 2020 at 3:41 PM Pavel Stehule <pavel.stehule@gmail.com>
>>>> wrote:
>>>> >
>>>> > Hi
>>>> >
>>>> > just rebase without any other changes
>>>> >
>>>>
>>>> You seem to forget attaching the rebased patch.
>>>>
>>>
>>> I am sorry
>>>
>>> attached.
>>>
>>
>> fresh rebase
>>
>
> fix test build
>

rebase

Pavel


> Regards
>
> Pavel
>
>
>> Regards
>>
>> Pavel
>>
>>
>>> Pavel
>>>
>>>
>>>> --
>>>> With Regards,
>>>> Amit Kapila.
>>>> EnterpriseDB: http://www.enterprisedb.com
>>>>
>>>

  176. Re: proposal: schema variables

    Michael Paquier <michael@paquier.xyz> — 2020-09-24T03:56:37Z

    On Sat, Jul 11, 2020 at 06:44:24AM +0200, Pavel Stehule wrote:
> rebase

Per the CF bot, this needs an extra rebase as it does not apply
anymore.  This has not attracted much the attention of committers as
well.
--
Michael

  177. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2020-09-24T03:58:55Z

    čt 24. 9. 2020 v 5:56 odesílatel Michael Paquier <michael@paquier.xyz>
napsal:

> On Sat, Jul 11, 2020 at 06:44:24AM +0200, Pavel Stehule wrote:
> > rebase
>
> Per the CF bot, this needs an extra rebase as it does not apply
> anymore.  This has not attracted much the attention of committers as
> well.
>

I'll fix it today

--
> Michael
>

  178. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2020-09-24T18:49:50Z

    čt 24. 9. 2020 v 5:58 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
napsal:

>
>
> čt 24. 9. 2020 v 5:56 odesílatel Michael Paquier <michael@paquier.xyz>
> napsal:
>
>> On Sat, Jul 11, 2020 at 06:44:24AM +0200, Pavel Stehule wrote:
>> > rebase
>>
>> Per the CF bot, this needs an extra rebase as it does not apply
>> anymore.  This has not attracted much the attention of committers as
>> well.
>>
>
> I'll fix it today
>

fixed patch attached

Regards

Pavel


> --
>> Michael
>>
>

  179. Re: proposal: schema variables

    Michael Paquier <michael@paquier.xyz> — 2020-10-01T03:38:24Z

    On Thu, Sep 24, 2020 at 08:49:50PM +0200, Pavel Stehule wrote:
> fixed patch attached

It looks like there are again conflicts within setrefs.c.
--
Michael

  180. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2020-10-01T05:08:38Z

    čt 1. 10. 2020 v 5:38 odesílatel Michael Paquier <michael@paquier.xyz>
napsal:

> On Thu, Sep 24, 2020 at 08:49:50PM +0200, Pavel Stehule wrote:
> > fixed patch attached
>
> It looks like there are again conflicts within setrefs.c.
>

fresh patch

Regards

Pavel

--
> Michael
>

  181. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2020-11-10T18:45:26Z

    čt 1. 10. 2020 v 7:08 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
napsal:

>
>
> čt 1. 10. 2020 v 5:38 odesílatel Michael Paquier <michael@paquier.xyz>
> napsal:
>
>> On Thu, Sep 24, 2020 at 08:49:50PM +0200, Pavel Stehule wrote:
>> > fixed patch attached
>>
>> It looks like there are again conflicts within setrefs.c.
>>
>
> fresh patch
>

rebase



> Regards
>
> Pavel
>
> --
>> Michael
>>
>

  182. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2020-12-19T06:57:35Z

    Hi

only rebase

Regards

Pavel

  183. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2020-12-26T04:52:44Z

    so 19. 12. 2020 v 7:57 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
napsal:

> Hi
>
> only rebase
>

rebase and comments fixes

Regards

Pavel




> Regards
>
> Pavel
>

  184. Re: proposal: schema variables

    Erik Rijkers <er@xs4all.nl> — 2020-12-26T06:18:21Z

    On 2020-12-26 05:52, Pavel Stehule wrote:
> so 19. 12. 2020 v 7:57 odesílatel Pavel Stehule 
> <pavel.stehule@gmail.com>
> napsal:
> [schema-variables-20201222.patch.gz (~]
> 
>> Hi
>> 
>> only rebase
>> 
> 
> rebase and comments fixes
> 

Hi Pavel,

This file is the exact same as the file you sent Tuesday. Is it a 
mistake?

  185. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2020-12-26T06:23:58Z

    so 26. 12. 2020 v 7:18 odesílatel Erik Rijkers <er@xs4all.nl> napsal:

> On 2020-12-26 05:52, Pavel Stehule wrote:
> > so 19. 12. 2020 v 7:57 odesílatel Pavel Stehule
> > <pavel.stehule@gmail.com>
> > napsal:
> > [schema-variables-20201222.patch.gz (~]
> >
> >> Hi
> >>
> >> only rebase
> >>
> >
> > rebase and comments fixes
> >
>
> Hi Pavel,
>
> This file is the exact same as the file you sent Tuesday. Is it a
> mistake?
>

It is the same. Unfortunately,  it was sent in an isolated thread, and
commitfest applications didn't find the latest version. I tried to attach
new thread to the commitfest application, but without success.

  186. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2021-01-01T08:45:13Z

    Hi

fresh rebase

Regards

Pavel

  187. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2021-01-08T06:20:50Z

    Hi

just rebase

Regards

Pavel

  188. Re: proposal: schema variables

    Erik Rijkers <er@xs4all.nl> — 2021-01-08T17:54:55Z

    On 2021-01-08 07:20, Pavel Stehule wrote:
> Hi
> 
> just rebase
> 
> [schema-variables-20200108.patch]

Hey Pavel,

My gcc 8.3.0 compile says:
(on debian 10/Buster)

utility.c: In function ‘CreateCommandTag’:
utility.c:2332:8: warning: this statement may fall through 
[-Wimplicit-fallthrough=]
     tag = CMDTAG_SELECT;
     ~~~~^~~~~~~~~~~~~~~
utility.c:2334:3: note: here
    case T_LetStmt:
    ^~~~


compile, check, check-world, runs without further problem.

I also changed a few typos/improvements in the documentation, see 
attached.

One thing I wasn't sure of: I have assumed that
   ON TRANSACTIONAL END RESET

should be
   ON TRANSACTION END RESET

and changed it accordingly, please double-check.


Erik Rijkers

  189. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2021-01-10T16:54:48Z

    pá 8. 1. 2021 v 18:54 odesílatel Erik Rijkers <er@xs4all.nl> napsal:

> On 2021-01-08 07:20, Pavel Stehule wrote:
> > Hi
> >
> > just rebase
> >
> > [schema-variables-20200108.patch]
>
> Hey Pavel,
>
> My gcc 8.3.0 compile says:
> (on debian 10/Buster)
>
> utility.c: In function ‘CreateCommandTag’:
> utility.c:2332:8: warning: this statement may fall through
> [-Wimplicit-fallthrough=]
>      tag = CMDTAG_SELECT;
>      ~~~~^~~~~~~~~~~~~~~
> utility.c:2334:3: note: here
>     case T_LetStmt:
>     ^~~~
>

yes, there was an error from the last rebase. Fixed


>
> compile, check, check-world, runs without further problem.
>
> I also changed a few typos/improvements in the documentation, see
> attached.
>
> One thing I wasn't sure of: I have assumed that
>    ON TRANSACTIONAL END RESET
>
> should be
>    ON TRANSACTION END RESET
>
> and changed it accordingly, please double-check.
>

It looks well, I merged these changes to patch. Thank you very much for
these corectures

Updated patch attached

Regards

Pavel


>
> Erik Rijkers
>

  190. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2021-01-14T06:35:29Z

    Hi

rebase

Regards

Pavel

  191. Re: proposal: schema variables

    Erik Rijkers <er@xs4all.nl> — 2021-01-14T09:24:27Z

    On 2021-01-14 07:35, Pavel Stehule wrote:
> [schema-variables-20210114.patch.gz]


Build is fine. My (small) list of tests run OK.

I did notice a few more documentation peculiarities:


'The PostgreSQL has schema variables'  should be
'PostgreSQL has schema variables'


A link to the LET command should be added to the 'See Also' of the 
CREATE VARIABLE, ALTER VARIABLE, and DROP VARIABLE pages. (After all, 
the LET command is the most interesting)
Similarly, an ALTER VARIABLE link should be added to the 'See Also' 
section of LET.


Somehow, the sgml in the doc files causes too large spacing in the html, 
example:
I copy from the LET html:
    'if that is defined.      If no explicit'
    (6 spaces between 'defined.' and 'If')
Can you have a look?  Sorry - I can't find the cause.  It occurs on a 
few more places in the newly added sgml/html.  The unwanted spaces are 
visible also in the pdf.
(firefox 78.6.1, debian)


Thanks,

Erik Rijkers

  192. Re: proposal: schema variables

    Josef Šimánek <josef.simanek@gmail.com> — 2021-01-14T10:31:33Z

    I did some testing locally. All runs smoothly, compiled without warning.

Later on (once merged) it would be nice to write down a documentation
page for the whole feature as a set next to documented individual
commands.
It took me a few moments to understand how this works.

I was looking how to create variable with non default value in one
command, but if I understand it correctly, that's not the purpose.
Variable lives in a schema with default value, but you can set it per
session via LET.

Thus "CREATE VARIABLE" statement should not be usually part of
"classic" queries, but it should be threatened more like TABLE or
other DDL statements.

On the other side LET is there to use the variable in "classic" queries.

Does that make sense? Feel free to ping me if any help with
documentation would be needed. I can try to prepare an initial
variables guide once I'll ensure I do  understand this feature well.

PS: Now it is clear to me why it is called "schema variables", not
"session variables".

čt 14. 1. 2021 v 7:36 odesílatel Pavel Stehule <pavel.stehule@gmail.com> napsal:
>
> Hi
>
> rebase
>
> Regards
>
> Pavel
>

  193. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2021-01-18T09:50:25Z

    čt 14. 1. 2021 v 10:24 odesílatel Erik Rijkers <er@xs4all.nl> napsal:

> On 2021-01-14 07:35, Pavel Stehule wrote:
> > [schema-variables-20210114.patch.gz]
>
>
> Build is fine. My (small) list of tests run OK.
>
> I did notice a few more documentation peculiarities:
>
>
> 'The PostgreSQL has schema variables'  should be
> 'PostgreSQL has schema variables'
>
>
fixed


> A link to the LET command should be added to the 'See Also' of the
> CREATE VARIABLE, ALTER VARIABLE, and DROP VARIABLE pages. (After all,
> the LET command is the most interesting)
> Similarly, an ALTER VARIABLE link should be added to the 'See Also'
> section of LET.
>

fixed


>
> Somehow, the sgml in the doc files causes too large spacing in the html,
> example:
> I copy from the LET html:
>     'if that is defined.      If no explicit'
>     (6 spaces between 'defined.' and 'If')
> Can you have a look?  Sorry - I can't find the cause.  It occurs on a
> few more places in the newly added sgml/html.  The unwanted spaces are
> visible also in the pdf.
>

Should be fixed in the last version. Probably there were some problems with
invisible white chars.

Thank you for check

Pavel



> (firefox 78.6.1, debian)
>
>
> Thanks,
>
> Erik Rijkers
>
>
>

  194. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2021-01-18T09:57:51Z

    Hi

čt 14. 1. 2021 v 11:31 odesílatel Josef Šimánek <josef.simanek@gmail.com>
napsal:

> I did some testing locally. All runs smoothly, compiled without warning.
>
> Later on (once merged) it would be nice to write down a documentation
> page for the whole feature as a set next to documented individual
> commands.
> It took me a few moments to understand how this works.
>
> I was looking how to create variable with non default value in one
> command, but if I understand it correctly, that's not the purpose.
> Variable lives in a schema with default value, but you can set it per
> session via LET.
>
> Thus "CREATE VARIABLE" statement should not be usually part of
> "classic" queries, but it should be threatened more like TABLE or
> other DDL statements.
>
> On the other side LET is there to use the variable in "classic" queries.
>
> Does that make sense? Feel free to ping me if any help with
> documentation would be needed. I can try to prepare an initial
> variables guide once I'll ensure I do  understand this feature well.
>

I invite any help with doc. Maybe there can be page in section advanced
features

https://www.postgresql.org/docs/current/tutorial-advanced.html

Regards

Pavel


>
> PS: Now it is clear to me why it is called "schema variables", not
> "session variables".
>
> čt 14. 1. 2021 v 7:36 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
> napsal:
> >
> > Hi
> >
> > rebase
> >
> > Regards
> >
> > Pavel
> >
>

  195. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2021-01-18T09:59:27Z

    po 18. 1. 2021 v 10:50 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
napsal:

>
>
> čt 14. 1. 2021 v 10:24 odesílatel Erik Rijkers <er@xs4all.nl> napsal:
>
>> On 2021-01-14 07:35, Pavel Stehule wrote:
>> > [schema-variables-20210114.patch.gz]
>>
>>
>> Build is fine. My (small) list of tests run OK.
>>
>> I did notice a few more documentation peculiarities:
>>
>>
>> 'The PostgreSQL has schema variables'  should be
>> 'PostgreSQL has schema variables'
>>
>>
> fixed
>
>
>> A link to the LET command should be added to the 'See Also' of the
>> CREATE VARIABLE, ALTER VARIABLE, and DROP VARIABLE pages. (After all,
>> the LET command is the most interesting)
>> Similarly, an ALTER VARIABLE link should be added to the 'See Also'
>> section of LET.
>>
>
> fixed
>
>
>>
>> Somehow, the sgml in the doc files causes too large spacing in the html,
>> example:
>> I copy from the LET html:
>>     'if that is defined.      If no explicit'
>>     (6 spaces between 'defined.' and 'If')
>> Can you have a look?  Sorry - I can't find the cause.  It occurs on a
>> few more places in the newly added sgml/html.  The unwanted spaces are
>> visible also in the pdf.
>>
>
> Should be fixed in the last version. Probably there were some problems
> with invisible white chars.
>
> Thank you for check
>
> Pavel
>
>
>
>> (firefox 78.6.1, debian)
>>
>
and here is the patch

Regards

Pavel


>>
>> Thanks,
>>
>> Erik Rijkers
>>
>>
>>

  196. Re: proposal: schema variables

    Erik Rijkers <er@xs4all.nl> — 2021-01-18T14:24:15Z

    On 2021-01-18 10:59, Pavel Stehule wrote:
>> 
> and here is the patch
> [schema-variables-20200118.patch.gz ]


One small thing:

The drop variable synopsis is:

DROP VARIABLE [ IF EXISTS ] name [, ...] [ CASCADE | RESTRICT ]

In that text following it, 'RESTRICT' is not documented. When used it 
does not give an error but I don't see how it 'works'.


Erik

  197. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2021-01-18T18:17:06Z

    po 18. 1. 2021 v 15:24 odesílatel Erik Rijkers <er@xs4all.nl> napsal:

> On 2021-01-18 10:59, Pavel Stehule wrote:
> >>
> > and here is the patch
> > [schema-variables-20200118.patch.gz ]
>
>
> One small thing:
>
> The drop variable synopsis is:
>
> DROP VARIABLE [ IF EXISTS ] name [, ...] [ CASCADE | RESTRICT ]
>
> In that text following it, 'RESTRICT' is not documented. When used it
> does not give an error but I don't see how it 'works'.
>

should be fixed now. Thank you for check

Regards

Pavel



>
> Erik
>
>
>
>

  198. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2021-01-23T09:50:29Z

    Hi

only rebase

Regards

Pavel

  199. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2021-02-02T08:43:57Z

    Hi

rebase and set PK for pg_variable table

Regards

Pavel

  200. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2021-02-16T17:46:13Z

    Hi

út 2. 2. 2021 v 9:43 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
napsal:

> Hi
>
> rebase and set PK for pg_variable table
>

rebase

Pavel


> Regards
>
> Pavel
>

  201. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2021-03-01T07:50:28Z

    út 16. 2. 2021 v 18:46 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
napsal:

> Hi
>
> út 2. 2. 2021 v 9:43 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
> napsal:
>
>> Hi
>>
>> rebase and set PK for pg_variable table
>>
>
> rebase
>

rebase



> Pavel
>
>
>> Regards
>>
>> Pavel
>>
>

  202. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2021-03-13T06:01:13Z

    Hi

fresh rebase

Pavel

  203. Re: proposal: schema variables - doc

    Erik Rijkers <er@xs4all.nl> — 2021-03-17T12:05:11Z

    > On 2021.03.13. 07:01 Pavel Stehule <pavel.stehule@gmail.com> wrote:
> Hi
> fresh rebase
> [schema-variables-20210313.patch.gz]


Hi Pavel,

I notice that the phrase 'schema variable' is not in the index at the end ('bookindex.html').  Not good.

It is also not in the index at the front of the manual - also not good.

Maybe these two (front and back index) can be added?


If a user searches the pdf, the first occurrence he finds is at:

  43.13.2.4. Global variables and constants
  (in itself that occurrence/mention is all right, but is should not be the first find, I think)

(I think there was in earlier versions of the patch an entry in the 'contents', i.e., at the front of the manual).  I think it would be good to have it in the front-index, pointing to either LET or CREATE VARIABLE, or maybe even to a small introductory paragraph somewhere else (again, I seem to remember that there was one in an earlier patch version).


Of the new commands that this patch brings, 'LET' is the most immediately illuminating for a user (even when a CREATE VARIABLE has to be done first.  There is an entry 'LET' in the index (good), but it would be better if that with LET-entry too the phrase 'schema variable' occurred.  (I don't know if that's possible)


Then, in the CREATE VARIABLE paragraphs it says
   'Changing a schema variable is non-transactional by default.'

I think that, unless there exists a mode where schema vars can be made transactional, 'by default' should be deleted (and there is no such 'transactional mode' for schema variables, is there?).  The 'Description' also has such a 'By default' which is better removed for the same reason.


In the CREATE VARIABLE page the example is:

CREATE VARIABLE var1 AS integer;
SELECT var1;

I suggest to make that

CREATE VARIABLE var1 AS date;
LET var1 = (select current_date);
SELECT var1;

So that the example immediately shows an application of functionality.


Thanks,

Erik Rijkers













> 
> Pavel

  204. Re: proposal: schema variables - doc

    Pavel Stehule <pavel.stehule@gmail.com> — 2021-03-22T09:47:07Z

    Hi

st 17. 3. 2021 v 13:05 odesílatel Erik Rijkers <er@xs4all.nl> napsal:

>
> > On 2021.03.13. 07:01 Pavel Stehule <pavel.stehule@gmail.com> wrote:
> > Hi
> > fresh rebase
> > [schema-variables-20210313.patch.gz]
>
>
> Hi Pavel,
>
> I notice that the phrase 'schema variable' is not in the index at the end
> ('bookindex.html').  Not good.
>
> It is also not in the index at the front of the manual - also not good.
>
> Maybe these two (front and back index) can be added?
>

I inserted new indexterm "schema variable", and now this part of
bookindex.html looks like:

schema variablealtering, ALTER VARIABLEchanging, LETdefining, CREATE
VARIABLEdescription, Descriptionremoving, DROP VARIABLE



>
> If a user searches the pdf, the first occurrence he finds is at:
>
>   43.13.2.4. Global variables and constants
>   (in itself that occurrence/mention is all right, but is should not be
> the first find, I think)
>
> (I think there was in earlier versions of the patch an entry in the
> 'contents', i.e., at the front of the manual).  I think it would be good to
> have it in the front-index, pointing to either LET or CREATE VARIABLE, or
> maybe even to a small introductory paragraph somewhere else (again, I seem
> to remember that there was one in an earlier patch version).
>


I wrote new section to "advanced features" about schema variables


>
>
> Of the new commands that this patch brings, 'LET' is the most immediately
> illuminating for a user (even when a CREATE VARIABLE has to be done first.
> There is an entry 'LET' in the index (good), but it would be better if that
> with LET-entry too the phrase 'schema variable' occurred.  (I don't know if
> that's possible)
>
>
> Then, in the CREATE VARIABLE paragraphs it says
>    'Changing a schema variable is non-transactional by default.'
>
> I think that, unless there exists a mode where schema vars can be made
> transactional, 'by default' should be deleted (and there is no such
> 'transactional mode' for schema variables, is there?).  The 'Description'
> also has such a 'By default' which is better removed for the same reason.
>

fixed


>
> In the CREATE VARIABLE page the example is:
>
> CREATE VARIABLE var1 AS integer;
> SELECT var1;
>
> I suggest to make that
>
> CREATE VARIABLE var1 AS date;
> LET var1 = (select current_date);
> SELECT var1;
>
> So that the example immediately shows an application of functionality.
>

done

Thank you for the documentation review.

Updated patch attached

Regards

Pavel



>
> Thanks,
>
> Erik Rijkers
>
>
>
>
>
>
>
>
>
>
>
>
>
> >
> > Pavel
>

  205. Re: proposal: schema variables - doc

    Pavel Stehule <pavel.stehule@gmail.com> — 2021-03-25T08:05:43Z

    Hi



po 22. 3. 2021 v 10:47 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
napsal:

> Hi
>
> st 17. 3. 2021 v 13:05 odesílatel Erik Rijkers <er@xs4all.nl> napsal:
>
>>
>> > On 2021.03.13. 07:01 Pavel Stehule <pavel.stehule@gmail.com> wrote:
>> > Hi
>> > fresh rebase
>> > [schema-variables-20210313.patch.gz]
>>
>>
>> Hi Pavel,
>>
>> I notice that the phrase 'schema variable' is not in the index at the end
>> ('bookindex.html').  Not good.
>>
>> It is also not in the index at the front of the manual - also not good.
>>
>> Maybe these two (front and back index) can be added?
>>
>
> I inserted new indexterm "schema variable", and now this part of
> bookindex.html looks like:
>
> schema variablealtering, ALTER VARIABLEchanging, LETdefining, CREATE
> VARIABLEdescription, Descriptionremoving, DROP VARIABLE
>
>
>
>>
>> If a user searches the pdf, the first occurrence he finds is at:
>>
>>   43.13.2.4. Global variables and constants
>>   (in itself that occurrence/mention is all right, but is should not be
>> the first find, I think)
>>
>> (I think there was in earlier versions of the patch an entry in the
>> 'contents', i.e., at the front of the manual).  I think it would be good to
>> have it in the front-index, pointing to either LET or CREATE VARIABLE, or
>> maybe even to a small introductory paragraph somewhere else (again, I seem
>> to remember that there was one in an earlier patch version).
>>
>
>
> I wrote new section to "advanced features" about schema variables
>
>
>>
>>
>> Of the new commands that this patch brings, 'LET' is the most immediately
>> illuminating for a user (even when a CREATE VARIABLE has to be done first.
>> There is an entry 'LET' in the index (good), but it would be better if that
>> with LET-entry too the phrase 'schema variable' occurred.  (I don't know if
>> that's possible)
>>
>>
>> Then, in the CREATE VARIABLE paragraphs it says
>>    'Changing a schema variable is non-transactional by default.'
>>
>> I think that, unless there exists a mode where schema vars can be made
>> transactional, 'by default' should be deleted (and there is no such
>> 'transactional mode' for schema variables, is there?).  The 'Description'
>> also has such a 'By default' which is better removed for the same reason.
>>
>
> fixed
>
>
>>
>> In the CREATE VARIABLE page the example is:
>>
>> CREATE VARIABLE var1 AS integer;
>> SELECT var1;
>>
>> I suggest to make that
>>
>> CREATE VARIABLE var1 AS date;
>> LET var1 = (select current_date);
>> SELECT var1;
>>
>> So that the example immediately shows an application of functionality.
>>
>
> done
>
> Thank you for the documentation review.
>
> Updated patch attached
>
> Regards
>
> Pavel
>
>
fresh update with merged Eric's changes in documentation

Regards

Pavel


>
>>
>> Thanks,
>>
>> Erik Rijkers
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> >
>> > Pavel
>>
>

  206. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2021-04-04T08:30:01Z

    Hi

so 13. 3. 2021 v 7:01 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
napsal:

> Hi
>
> fresh rebase
>

only rebase

Regards

Pavel


>
> Pavel
>

  207. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2021-04-10T06:58:41Z

    Hi

I am sending a strongly updated patch for schema variables.

I rewrote an execution of a LET statement. In the previous implementation I
hacked STMT_SELECT. Now, I introduced a new statement STMT_LET, and I
implemented a new executor node SetVariable. Now I think this
implementation is much cleaner. Implementation with own executor node
reduces necessary work on PL side - and allows the LET statement to be
prepared - what is important from a security view.

I'll try to write a second implementation based on a cleaner implementation
like utility command too. I expect so this version will be more simple, but
utility commands cannot be prepared, and probably, there should be special
support for any PL. I hope a cleaner implementation can help to move this
patch.

We can choose one variant in the next step and this variant can be
finalized.

Notes, comments?

Regards

Pavel

  208. Re: proposal: schema variables

    Laurenz Albe <laurenz.albe@cybertec.at> — 2024-07-19T11:41:00Z

    On Sat, 2021-04-10 at 08:58 +0200, Pavel Stehule wrote:
> I am sending a strongly updated patch for schema variables.
> 
> I rewrote an execution of a LET statement. In the previous implementation I hacked
> STMT_SELECT. Now, I introduced a new statement STMT_LET, and I implemented a new
> executor node SetVariable. Now I think this implementation is much cleaner.
> Implementation with own executor node reduces necessary work on PL side - and allows
> the LET statement to be prepared - what is important from a security view.
> 
> I'll try to write a second implementation based on a cleaner implementation like
> utility command too. I expect so this version will be more simple, but utility
> commands cannot be prepared, and probably, there should be special support for
> any PL. I hope a cleaner implementation can help to move this patch.
> 
> We can choose one variant in the next step and this variant can be finalized.
> 
> Notes, comments?

Thank you!

I tried to give the patch a spin, but it doesn't apply any more,
and there are too many conflicts for me to fix manually.

So I had a look at the documentation:

> --- a/doc/src/sgml/advanced.sgml
> +++ b/doc/src/sgml/advanced.sgml

> +   <para>
> +    The value of a schema variable is local to the current session. Retrieving
> +    a variable's value returns either a NULL or a default value, unless its value
> +    is set to something else in the current session with a LET command. The content
> +    of a variable is not transactional. This is the same as in regular variables
> +    in PL languages.
> +   </para>
> +
> +   <para>
> +    Schema variables are retrieved by the <command>SELECT</command> SQL command.
> +    Their value is set with the <command>LET</command> SQL command.
> +    While schema variables share properties with tables, their value cannot be updated
> +    with an <command>UPDATE</command> command.

"PL languages" -> "procedural languages".  Perhaps a link to the "procedural Languages"
chapter would be a good idea.
I don't think we should say "regular" variables: are there irregular variables?

My feeling is that "SQL statement <command>XY</command>" is better than
"<command>XY</command> SQL command".

I think the last sentence should go.  The properties they share with tables are
that they live in a schema and can be used with SELECT.
Also, it is not necessary to mention that they cannot be UPDATEd.  They cannot
be TRUNCATEd or CALLed either, so why mention UPDATE specifically?

> --- a/doc/src/sgml/catalogs.sgml
> +++ b/doc/src/sgml/catalogs.sgml

> +     <row>
> +      <entry><structfield>varisnotnull</structfield></entry>
> +      <entry><type>boolean</type></entry>
> +      <entry></entry>
> +      <entry>
> +       True if the schema variable doesn't allow null value. The default value is false.
> +      </entry>
> +     </row>

I think the attribute should be called "varnotnull", similar to "attnotnull".
This attribute determines whether the variable is NOT NULL or not, not about
its current setting.

There is a plural missing: "doesn't allow null valueS".

> +     <row>
> +      <entry><structfield>vareoxaction</structfield></entry>
> +      <entry><type>char</type></entry>
> +      <entry></entry>
> +      <entry>
> +       <literal>n</literal> = no action, <literal>d</literal> = drop the variable,
> +       <literal>r</literal> = reset the variable to its default value.
> +      </entry>
> +     </row>

Perhaps the name "varxactendaction" would be better.

A descriptive sentence is missing.

> --- /dev/null
> +++ b/doc/src/sgml/ref/create_variable.sgml

> +  <para>
> +   The value of a schema variable is local to the current session. Retrieving
> +   a variable's value returns either a NULL or a default value, unless its value
> +   is set to something else in the current session with a LET command. The content
> +   of a variable is not transactional. This is the same as in regular variables in PL languages.
> +  </para>

"regular variables in PL languages" -> "variables in procedural languages"

> +  <para>
> +   Schema variables are retrieved by the <command>SELECT</command> SQL command.
> +   Their value is set with the <command>LET</command> SQL command.
> +   While schema variables share properties with tables, their value cannot be updated
> +   with an <command>UPDATE</command> command.
> +  </para>

That's just a literal copy from the tutorial section.  I have the same comments
as there.

> +   <varlistentry>
> +    <term><literal>NOT NULL</literal></term>
> +    <listitem>
> +     <para>
> +      The <literal>NOT NULL</literal> clause forbids to set the variable to
> +      a null value. A variable created as NOT NULL and without an explicitly
> +      declared default value cannot be read until it is initialized by a LET
> +      command. This obliges the user to explicitly initialize the variable
> +      content before reading it.
> +     </para>
> +    </listitem>
> +   </varlistentry>

What is the reason for that behavior?  I'd have expected that a NOT NULL
variable needs a default value.

> --- /dev/null
> +++ b/doc/src/sgml/ref/let.sgml

> +   <varlistentry>
> +    <term><literal>sql_expression</literal></term>
> +    <listitem>
> +     <para>
> +      An SQL expression. The result is cast into the schema variable's type.
> +     </para>
> +    </listitem>
> +   </varlistentry>

Always, even if there is no assignment or implicit cast?

I see no such wording fir INSERT or UPDATE, so if only assignment casts are used,
the second sentence should be removed.

> --- a/doc/src/sgml/ref/pg_restore.sgml
> +++ b/doc/src/sgml/ref/pg_restore.sgml

> +     <varlistentry>
> +      <term><option>-A <replaceable class="parameter">schema_variable</replaceable></option></term>
> +      <term><option>--variable=<replaceable class="parameter">schema_variable</replaceable></option></term>
> +      <listitem>
> +       <para>
> +        Restore a named schema variable only.  Multiple schema variables may be specified with
> +        multiple <option>-A</option> switches.
> +       </para>
> +      </listitem>
> +     </varlistentry>

Do we need that?  We have no such option for functions and other non-relations.

And if we really want such an option for "pg_restore", why not for "pg_dump"?

Yours,
Laurenz Albe

  209. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-07-20T19:48:43Z

    pá 19. 7. 2024 v 13:41 odesílatel Laurenz Albe <laurenz.albe@cybertec.at>
napsal:

> On Sat, 2021-04-10 at 08:58 +0200, Pavel Stehule wrote:
> > I am sending a strongly updated patch for schema variables.
> >
> > I rewrote an execution of a LET statement. In the previous
> implementation I hacked
> > STMT_SELECT. Now, I introduced a new statement STMT_LET, and I
> implemented a new
> > executor node SetVariable. Now I think this implementation is much
> cleaner.
> > Implementation with own executor node reduces necessary work on PL side
> - and allows
> > the LET statement to be prepared - what is important from a security
> view.
> >
> > I'll try to write a second implementation based on a cleaner
> implementation like
> > utility command too. I expect so this version will be more simple, but
> utility
> > commands cannot be prepared, and probably, there should be special
> support for
> > any PL. I hope a cleaner implementation can help to move this patch.
> >
> > We can choose one variant in the next step and this variant can be
> finalized.
> >
> > Notes, comments?
>
> Thank you!
>
> I tried to give the patch a spin, but it doesn't apply any more,
> and there are too many conflicts for me to fix manually.
>
>
>
just fresh rebase

I'll reply to your comments tomorrow.

 Regards

Pavel

  210. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-07-22T06:37:22Z

    pá 19. 7. 2024 v 13:41 odesílatel Laurenz Albe <laurenz.albe@cybertec.at>
napsal:

> On Sat, 2021-04-10 at 08:58 +0200, Pavel Stehule wrote:
> > I am sending a strongly updated patch for schema variables.
> >
> > I rewrote an execution of a LET statement. In the previous
> implementation I hacked
> > STMT_SELECT. Now, I introduced a new statement STMT_LET, and I
> implemented a new
> > executor node SetVariable. Now I think this implementation is much
> cleaner.
> > Implementation with own executor node reduces necessary work on PL side
> - and allows
> > the LET statement to be prepared - what is important from a security
> view.
> >
> > I'll try to write a second implementation based on a cleaner
> implementation like
> > utility command too. I expect so this version will be more simple, but
> utility
> > commands cannot be prepared, and probably, there should be special
> support for
> > any PL. I hope a cleaner implementation can help to move this patch.
> >
> > We can choose one variant in the next step and this variant can be
> finalized.
> >
> > Notes, comments?
>
> Thank you!
>
> I tried to give the patch a spin, but it doesn't apply any more,
> and there are too many conflicts for me to fix manually.
>
> So I had a look at the documentation:
>
> > --- a/doc/src/sgml/advanced.sgml
> > +++ b/doc/src/sgml/advanced.sgml
>
> > +   <para>
> > +    The value of a schema variable is local to the current session.
> Retrieving
> > +    a variable's value returns either a NULL or a default value, unless
> its value
> > +    is set to something else in the current session with a LET command.
> The content
> > +    of a variable is not transactional. This is the same as in regular
> variables
> > +    in PL languages.
> > +   </para>
> > +
> > +   <para>
> > +    Schema variables are retrieved by the <command>SELECT</command> SQL
> command.
> > +    Their value is set with the <command>LET</command> SQL command.
> > +    While schema variables share properties with tables, their value
> cannot be updated
> > +    with an <command>UPDATE</command> command.
>
> "PL languages" -> "procedural languages".  Perhaps a link to the
> "procedural Languages"
> chapter would be a good idea.
> I don't think we should say "regular" variables: are there irregular
> variables?
>
> My feeling is that "SQL statement <command>XY</command>" is better than
> "<command>XY</command> SQL command".
>

probably, you are reading an old version of this patch. I cannot find these
sentences.


>
> I think the last sentence should go.  The properties they share with
> tables are
> that they live in a schema and can be used with SELECT.
> Also, it is not necessary to mention that they cannot be UPDATEd.  They
> cannot
> be TRUNCATEd or CALLed either, so why mention UPDATE specifically?
>
> > --- a/doc/src/sgml/catalogs.sgml
> > +++ b/doc/src/sgml/catalogs.sgml
>
> > +     <row>
> > +      <entry><structfield>varisnotnull</structfield></entry>
> > +      <entry><type>boolean</type></entry>
> > +      <entry></entry>
> > +      <entry>
> > +       True if the schema variable doesn't allow null value. The
> default value is false.
> > +      </entry>
> > +     </row>
>
> I think the attribute should be called "varnotnull", similar to
> "attnotnull".
> This attribute determines whether the variable is NOT NULL or not, not
> about
> its current setting.
>
> There is a plural missing: "doesn't allow null valueS".
>

changed


>
> > +     <row>
> > +      <entry><structfield>vareoxaction</structfield></entry>
> > +      <entry><type>char</type></entry>
> > +      <entry></entry>
> > +      <entry>
> > +       <literal>n</literal> = no action, <literal>d</literal> = drop
> the variable,
> > +       <literal>r</literal> = reset the variable to its default value.
> > +      </entry>
> > +     </row>
>
> Perhaps the name "varxactendaction" would be better.
>
> A descriptive sentence is missing.
>

I renamed field, recent version looks like

     <row>
      <entry role="catalog_table_entry"><para role="column_definition">
       <structfield>varxactendaction</structfield> <type>char</type>
      </para>
      <para>
       Action performed at end of transaction:
       <literal>n</literal> = no action, <literal>d</literal> = drop the
variable,
       <literal>r</literal> = reset the variable to its default value.
      </para></entry>
     </row>


>
> > --- /dev/null
> > +++ b/doc/src/sgml/ref/create_variable.sgml
>
> > +  <para>
> > +   The value of a schema variable is local to the current session.
> Retrieving
> > +   a variable's value returns either a NULL or a default value, unless
> its value
> > +   is set to something else in the current session with a LET command.
> The content
> > +   of a variable is not transactional. This is the same as in regular
> variables in PL languages.
> > +  </para>
>
> "regular variables in PL languages" -> "variables in procedural languages"
>

fixed


>
> > +  <para>
> > +   Schema variables are retrieved by the <command>SELECT</command> SQL
> command.
> > +   Their value is set with the <command>LET</command> SQL command.
> > +   While schema variables share properties with tables, their value
> cannot be updated
> > +   with an <command>UPDATE</command> command.
> > +  </para>
>
> That's just a literal copy from the tutorial section.  I have the same
> comments
> as there.
>

fixed


>
> > +   <varlistentry>
> > +    <term><literal>NOT NULL</literal></term>
> > +    <listitem>
> > +     <para>
> > +      The <literal>NOT NULL</literal> clause forbids to set the
> variable to
> > +      a null value. A variable created as NOT NULL and without an
> explicitly
> > +      declared default value cannot be read until it is initialized by
> a LET
> > +      command. This obliges the user to explicitly initialize the
> variable
> > +      content before reading it.
> > +     </para>
> > +    </listitem>
> > +   </varlistentry>
>
> What is the reason for that behavior?  I'd have expected that a NOT NULL
> variable needs a default value.
>

changed - now, the default is required when variable is NOT NULL



>
> > --- /dev/null
> > +++ b/doc/src/sgml/ref/let.sgml
>
> > +   <varlistentry>
> > +    <term><literal>sql_expression</literal></term>
> > +    <listitem>
> > +     <para>
> > +      An SQL expression. The result is cast into the schema variable's
> type.
> > +     </para>
> > +    </listitem>
> > +   </varlistentry>
>
> Always, even if there is no assignment or implicit cast?
>

It uses implicit cast in COERCION_ASSIGNMENT context. coerce_to_target_type
is used always

This part of doc currently looks

    <listitem>
     <para>
      An SQL expression (can be subquery in parenthesis). The result must
      be of castable to the same data type as the session variable (in
      implicit or assignment context).
     </para>
    </listitem>





> I see no such wording fir INSERT or UPDATE, so if only assignment casts
> are used,
> the second sentence should be removed.
>
> > --- a/doc/src/sgml/ref/pg_restore.sgml
> > +++ b/doc/src/sgml/ref/pg_restore.sgml
>
> > +     <varlistentry>
> > +      <term><option>-A <replaceable
> class="parameter">schema_variable</replaceable></option></term>
> > +      <term><option>--variable=<replaceable
> class="parameter">schema_variable</replaceable></option></term>
> > +      <listitem>
> > +       <para>
> > +        Restore a named schema variable only.  Multiple schema
> variables may be specified with
> > +        multiple <option>-A</option> switches.
> > +       </para>
> > +      </listitem>
> > +     </varlistentry>
>
> Do we need that?  We have no such option for functions and other
> non-relations.
>

It is designed to be consistent with others. pg_restore supports functions
-P, triggers -T

>
> And if we really want such an option for "pg_restore", why not for
> "pg_dump"?
>

I have no strong opinion about it, I think so it is consistent with other
non-relations, but it is not important.

I moved this feature to a separate patch. It can be committed optionaly or
later.

pg_restore has options -P, -T, and pg_dump does not have these options.
These options (functionality) can be implemented in pg_dump too, but
unfortunately -T is used for different purposes (exclude table).

Regards

Pavel




>
> Yours,
> Laurenz Albe
>

  211. Re: proposal: schema variables

    Laurenz Albe <laurenz.albe@cybertec.at> — 2024-07-22T08:23:39Z

    Thanks for the updated patch and the fixes!

On Mon, 2024-07-22 at 08:37 +0200, Pavel Stehule wrote:
> > > --- a/doc/src/sgml/ref/pg_restore.sgml
> > > +++ b/doc/src/sgml/ref/pg_restore.sgml
> > 
> > > +     <varlistentry>
> > > +      <term><option>-A <replaceable class="parameter">schema_variable</replaceable></option></term>
> > > +      <term><option>--variable=<replaceable class="parameter">schema_variable</replaceable></option></term>
> > > +      <listitem>
> > > +       <para>
> > > +        Restore a named schema variable only.  Multiple schema variables may be specified with
> > > +        multiple <option>-A</option> switches.
> > > +       </para>
> > > +      </listitem>
> > > +     </varlistentry>
> > 
> > Do we need that?  We have no such option for functions and other non-relations.
> 
> It is designed to be consistent with others. pg_restore supports functions -P, triggers -T 
> > 
> > And if we really want such an option for "pg_restore", why not for "pg_dump"?
> > 
> 
> I have no strong opinion about it, I think so it is consistent with other non-relations, but it is not important.
> 
> I moved this feature to a separate patch. It can be committed optionaly or later.
> 
> pg_restore has options -P, -T, and pg_dump does not have these options. These options (functionality) can
> be implemented in pg_dump too, but unfortunately -T is used for different purposes (exclude table).

Ah!  I didn't realize that -P and -T are the same.  So no objections, although I'm
not sure if anyone will ever want to restore a single variable from a backup.

Yours,
Laurenz Albe

  212. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-07-22T08:55:30Z

    po 22. 7. 2024 v 10:23 odesílatel Laurenz Albe <laurenz.albe@cybertec.at>
napsal:

> Thanks for the updated patch and the fixes!
>
> On Mon, 2024-07-22 at 08:37 +0200, Pavel Stehule wrote:
> > > > --- a/doc/src/sgml/ref/pg_restore.sgml
> > > > +++ b/doc/src/sgml/ref/pg_restore.sgml
> > >
> > > > +     <varlistentry>
> > > > +      <term><option>-A <replaceable
> class="parameter">schema_variable</replaceable></option></term>
> > > > +      <term><option>--variable=<replaceable
> class="parameter">schema_variable</replaceable></option></term>
> > > > +      <listitem>
> > > > +       <para>
> > > > +        Restore a named schema variable only.  Multiple schema
> variables may be specified with
> > > > +        multiple <option>-A</option> switches.
> > > > +       </para>
> > > > +      </listitem>
> > > > +     </varlistentry>
> > >
> > > Do we need that?  We have no such option for functions and other
> non-relations.
> >
> > It is designed to be consistent with others. pg_restore supports
> functions -P, triggers -T
> > >
> > > And if we really want such an option for "pg_restore", why not for
> "pg_dump"?
> > >
> >
> > I have no strong opinion about it, I think so it is consistent with
> other non-relations, but it is not important.
> >
> > I moved this feature to a separate patch. It can be committed optionaly
> or later.
> >
> > pg_restore has options -P, -T, and pg_dump does not have these options.
> These options (functionality) can
> > be implemented in pg_dump too, but unfortunately -T is used for
> different purposes (exclude table).
>
> Ah!  I didn't realize that -P and -T are the same.  So no objections,
> although I'm
> not sure if anyone will ever want to restore a single variable from a
> backup.
>

I wrote it mainly for completeness and symmetry. I can imagine only one use
case - possibility to offline trace the changes  of schema, but who knows.
The cost is just an occupation of 'A' char. Maybe it doesn't need a short
option, and a long option can be good enough.

Pavel



> Yours,
> Laurenz Albe
>

  213. Re: proposal: schema variables

    Laurenz Albe <laurenz.albe@cybertec.at> — 2024-07-23T14:34:44Z

    Thanks for the fixes and the new patch set!
I think that this would be a very valuable feature!

This is a very incomplete review after playing with the patch set for a while.

Some bugs and oddities I have found:

"psql" support:

  \? gives

    \dV     [PATTERN]      list variables

  I think that should say "schema variables" to distinguish them from
  psql variables.

  Running \dV when connected to an older server gives

    ERROR:  relation "pg_catalog.pg_variable" does not exist
    LINE 16: FROM pg_catalog.pg_variable v
                  ^

  I think it would be better not to run the query and show a response like

    session variables don't exist in server version 16

The LET statement:

    CREATE VARIABLE testvar AS int4multirange[];
    LET testvar = '{\{[2\,7]\,[11\,13]\}}';
    ERROR:  variable "laurenz.testvar" is of type int4multirange[], but expression is of type text
    LINE 1: LET testvar = '{\{[2\,7]\,[11\,13]\}}';
                          ^
    HINT:  You will need to rewrite or cast the expression.

  Sure, I can add an explicit type cast, but I think that the type should
  be determined by the type of the variable.  The right-hand side should be
  treated as "unknown", and the type input function should be used.

Parameter session_variables_ambiguity_warning:

    --- a/src/backend/utils/misc/guc_tables.c
    +++ b/src/backend/utils/misc/guc_tables.c
    @@ -1544,6 +1544,16 @@ struct config_bool ConfigureNamesBool[] =
            false,
            NULL, NULL, NULL
        },
    +   {
    +       {"session_variables_ambiguity_warning", PGC_USERSET, DEVELOPER_OPTIONS,
    +           gettext_noop("Raise warning when reference to a session variable is ambiguous."),
    +           NULL,
    +           GUC_NOT_IN_SAMPLE
    +       },
    +       &session_variables_ambiguity_warning,
    +       false,
    +       NULL, NULL, NULL
    +   },

  I think the short_desc should be "Raise a warning" (with the indefinite article).

  DEVELOPER_OPTIONS is the wrong category.  We normally use that for parameters
  that are only relevant for PostgreSQL hackers.  I think it should be
  CLIENT_CONN_OTHER.

CREATE VARIABLE command:

    CREATE VARIABLE str AS text NOT NULL;
    ERROR:  session variable must have a default value, since it's declared NOT NULL

  Perhaps this error message would be better:

    session variables declared as NOT NULL must have a default value

  This is buggy:

    CREATE VARIABLE str AS text NOT NULL DEFAULT NULL;

  Ugh.

    SELECT str;
    ERROR:  null value is not allowed for NOT NULL session variable "laurenz.str"
    DETAIL:  The result of DEFAULT expression is NULL.

  Perhaps that is a leftover from the previous coding, but I think there need be
  no check upon SELECT.  It should be enough to check during CREATE VARIABLE and
  LET.

pg_dump support:

  The attempt to dump a database with an older version leads to

    pg_dump: error: query failed: ERROR:  relation "pg_catalog.pg_variable" does not exist
    LINE 14: FROM pg_catalog.pg_variable v
                  ^

  Dumping variables must be conditional on the server version.

IMMUTABLE variables:

    +   <varlistentry id="sql-createvariable-immutable">
    +    <term><literal>IMMUTABLE</literal></term>
    +    <listitem>
    +     <para>
    +      The assigned value of the session variable can not be changed.
    +      Only if the session variable doesn't have a default value, a single
    +      initialization is allowed using the <command>LET</command> command. Once
    +      done, no further change is allowed until end of transaction
    +      if the session variable was created with clause <literal>ON TRANSACTION
    +      END RESET</literal>, or until reset of all session variables by
    +      <command>DISCARD VARIABLES</command>, or until reset of all session
    +      objects by command <command>DISCARD ALL</command>.
    +     </para>
    +    </listitem>
    +   </varlistentry>

  I can see the usefulness of IMMUTABLE variables, but I am surprised that
  they are reset by DISCARD.  What is the use case you have in mind?
  The use case I can envision is an application that sets a value right after
  authentication, for use with row-level security.  But then it would be harmful
  if the user could reset the variable with DISCARD.

Documentation:

    --- a/doc/src/sgml/ddl.sgml
    +++ b/doc/src/sgml/ddl.sgml

    +   <para>
    +    The session variables can be shadowed by column references in a query. When
    +    a query contains identifiers or qualified identifiers that could be used as
    +    both a session variable identifiers and as column identifier, then the
    +    column identifier is preferred every time. Warnings can be emitted when
    +    this situation happens by enabling configuration parameter <xref
    +    linkend="guc-session-variables-ambiguity-warning"/>. User can explicitly
    +    qualify the source object by syntax <literal>table.column</literal> or
    +    <literal>variable.column</literal>.
    +   </para>

  I think you mean <literal>schema.variable</literal>, not <literal>variable.column</literal>.

Yours,
Laurenz Albe

  214. Re: proposal: schema variables

    Laurenz Albe <laurenz.albe@cybertec.at> — 2024-07-23T21:41:28Z

    On Tue, 2024-07-23 at 16:34 +0200, Laurenz Albe wrote:
> CREATE VARIABLE command:
> 
>   This is buggy:
> 
>     CREATE VARIABLE str AS text NOT NULL DEFAULT NULL;
> 
>   Ugh.
> 
>     SELECT str;
>     ERROR:  null value is not allowed for NOT NULL session variable "laurenz.str"
>     DETAIL:  The result of DEFAULT expression is NULL.
> 
>   Perhaps that is a leftover from the previous coding, but I think there need be
>   no check upon SELECT.  It should be enough to check during CREATE VARIABLE and
>   LET.

I'm having second thoughts about that.

I was thinking of a variable like of a table column, but there is a fundamental
difference: there is a clear moment when a tuple is added (INSERT or UPDATE),
which is the point where a column can be checked for NULL values.

A variable can be SELECTed without having been LET before, in which case it
has the default value.  But there is no way to test the default value before
the variable is SELECTed.  So while DEFAULT NULL for a non-nullable variable
seems weird, it is no worse than DEFAULT somefunc() for a function that returns
NULL.

So perhaps the behavior I complained about above is actually the right one.
In the view of that, it doesn't seem necessary to enforce a DEFAULT value for
a NOT NULL variable: NOT NULL might just as well mean "you have to LET it before
you can SELECT it".

> IMMUTABLE variables:
> 
>     +   <varlistentry id="sql-createvariable-immutable">
>     +    <term><literal>IMMUTABLE</literal></term>
>     +    <listitem>
>     +     <para>
>     +      The assigned value of the session variable can not be changed.
>     +      Only if the session variable doesn't have a default value, a single
>     +      initialization is allowed using the <command>LET</command> command. Once
>     +      done, no further change is allowed until end of transaction
>     +      if the session variable was created with clause <literal>ON TRANSACTION
>     +      END RESET</literal>, or until reset of all session variables by
>     +      <command>DISCARD VARIABLES</command>, or until reset of all session
>     +      objects by command <command>DISCARD ALL</command>.
>     +     </para>
>     +    </listitem>
>     +   </varlistentry>
> 
>   I can see the usefulness of IMMUTABLE variables, but I am surprised that
>   they are reset by DISCARD.  What is the use case you have in mind?
>   The use case I can envision is an application that sets a value right after
>   authentication, for use with row-level security.  But then it would be harmful
>   if the user could reset the variable with DISCARD.

I'm beginning to be uncertain about that as well.  You might want to use a
connection pool, and you LET the variable when you take it out of the pool.
When the session is returned to the pool, variables get DISCARDed.

Sure, a user can call DISCARD, but only if he or she is in an interactive session.

So perhaps it is good as it is.

Yours,
Laurenz Albe

  215. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-07-24T15:19:55Z

    Hi

út 23. 7. 2024 v 16:34 odesílatel Laurenz Albe <laurenz.albe@cybertec.at>
napsal:

> Thanks for the fixes and the new patch set!
> I think that this would be a very valuable feature!
>
>
Thank you :-)


> This is a very incomplete review after playing with the patch set for a
> while.
>
> Some bugs and oddities I have found:
>
> "psql" support:
>
>   \? gives
>
>     \dV     [PATTERN]      list variables
>
>   I think that should say "schema variables" to distinguish them from
>   psql variables.
>

enhanced to "list session variables"


>
>   Running \dV when connected to an older server gives
>
>     ERROR:  relation "pg_catalog.pg_variable" does not exist
>     LINE 16: FROM pg_catalog.pg_variable v
>                   ^
>
>   I think it would be better not to run the query and show a response like
>
>     session variables don't exist in server version 16
>

there is standardized error message - and I used now

<-->if (pset.sversion < 180000)
<-->{
<--><-->char<--><-->sverbuf[32];

<--><-->pg_log_error("The server (version %s) does not support session
variables.",
<--><--><--><--><--> formatPGVersionNumber(pset.sversion, false,
<--><--><--><--><--><--><--><--><--><-->   sverbuf, sizeof(sverbuf)));
<--><-->return true;
<-->}


>
> The LET statement:
>
>     CREATE VARIABLE testvar AS int4multirange[];
>     LET testvar = '{\{[2\,7]\,[11\,13]\}}';
>     ERROR:  variable "laurenz.testvar" is of type int4multirange[], but
> expression is of type text
>     LINE 1: LET testvar = '{\{[2\,7]\,[11\,13]\}}';
>                           ^
>     HINT:  You will need to rewrite or cast the expression.
>
>   Sure, I can add an explicit type cast, but I think that the type should
>   be determined by the type of the variable.  The right-hand side should be
>   treated as "unknown", and the type input function should be used.
>

fixed - it needed set pstate->p_resolve_unknowns = false;

I used your example in regress test


> Parameter session_variables_ambiguity_warning:
>
>     --- a/src/backend/utils/misc/guc_tables.c
>     +++ b/src/backend/utils/misc/guc_tables.c
>     @@ -1544,6 +1544,16 @@ struct config_bool ConfigureNamesBool[] =
>             false,
>             NULL, NULL, NULL
>         },
>     +   {
>     +       {"session_variables_ambiguity_warning", PGC_USERSET,
> DEVELOPER_OPTIONS,
>     +           gettext_noop("Raise warning when reference to a session
> variable is ambiguous."),
>     +           NULL,
>     +           GUC_NOT_IN_SAMPLE
>     +       },
>     +       &session_variables_ambiguity_warning,
>     +       false,
>     +       NULL, NULL, NULL
>     +   },
>
>   I think the short_desc should be "Raise a warning" (with the indefinite
> article).
>
>   DEVELOPER_OPTIONS is the wrong category.  We normally use that for
> parameters
>   that are only relevant for PostgreSQL hackers.  I think it should be
>   CLIENT_CONN_OTHER.
>

 changed


> CREATE VARIABLE command:
>
>     CREATE VARIABLE str AS text NOT NULL;
>     ERROR:  session variable must have a default value, since it's
> declared NOT NULL
>
>   Perhaps this error message would be better:
>
>     session variables declared as NOT NULL must have a default value
>

changed


>
>   This is buggy:
>
>     CREATE VARIABLE str AS text NOT NULL DEFAULT NULL;
>
>   Ugh.
>
>     SELECT str;
>     ERROR:  null value is not allowed for NOT NULL session variable
> "laurenz.str"
>     DETAIL:  The result of DEFAULT expression is NULL.
>
>   Perhaps that is a leftover from the previous coding, but I think there
> need be
>   no check upon SELECT.  It should be enough to check during CREATE
> VARIABLE and
>   LET.
>

I think it is correct. When you use SELECT str, then DEFAULT expression is
executed, and then the result is assigned to a variable, and there is NOT
NULL guard, which raises an exception. The variable is not initialized when
you run DDL, but it is initialized when you first read or write from/to the
variable. The DEFAULT expression is not evaluated in DDL time. In this
case, I can detect the problem in DDL time because the result is
transformed to NULL node, but generally there can be SQL non immutable
function, and then I need to wait until the DEFAULT expression will be
evaluated - and it is time of first reading. Unfortunately, there is not an
available check if some expression is NULL, that I can use in DDL time, but
I have it in plpgsql_check.

You can have a function

CREATE OR REPLACE FUNCTION foo()
RETURNS int AS $$
BEGIN
  RETURN NULL;
END:
$$ LANGUAGE plpgsql;

the function is not marked as IMMUTABLE

I can

CREATE VARIABLE v AS int NOT NULL DEFAULT foo();

In DDL time I know nothing about result of foo()

When I run SELECT v; then foo is executed, and because it is in
contradiction with the NOT NULL clause, it fails. And I think it is correct.

It is consistent with PL/pgSQL. I can write

(2024-07-24 11:57:33) postgres=# CREATE OR REPLACE FUNCTION fx()
postgres-# RETURNS void AS $$
postgres$# DECLARE v int NOT NULL DEFAULT NULL;
postgres$# BEGIN
postgres$# END;
postgres$# $$ LANGUAGE plpgsql;
CREATE FUNCTION
(2024-07-24 12:06:54) postgres=# SELECT fx();
ERROR:  null value cannot be assigned to variable "v" declared NOT NULL
CONTEXT:  PL/pgSQL function fx() line 2 during statement block local
variable initialization

DDL is ok, and SELECT fails.

I think so DDL should not to evaluate DEFAULT expressions - the result from
this are issues described in discussion about usage 'now'::timestamp


>
> pg_dump support:
>
>   The attempt to dump a database with an older version leads to
>
>     pg_dump: error: query failed: ERROR:  relation
> "pg_catalog.pg_variable" does not exist
>     LINE 14: FROM pg_catalog.pg_variable v
>                   ^

  Dumping variables must be conditional on the server version.
>


>
fixed, there was guard but for pg 17



>
> IMMUTABLE variables:
>
>     +   <varlistentry id="sql-createvariable-immutable">
>     +    <term><literal>IMMUTABLE</literal></term>
>     +    <listitem>
>     +     <para>
>     +      The assigned value of the session variable can not be changed.
>     +      Only if the session variable doesn't have a default value, a
> single
>     +      initialization is allowed using the <command>LET</command>
> command. Once
>     +      done, no further change is allowed until end of transaction
>     +      if the session variable was created with clause <literal>ON
> TRANSACTION
>     +      END RESET</literal>, or until reset of all session variables by
>     +      <command>DISCARD VARIABLES</command>, or until reset of all
> session
>     +      objects by command <command>DISCARD ALL</command>.
>     +     </para>
>     +    </listitem>
>     +   </varlistentry>
>
>   I can see the usefulness of IMMUTABLE variables, but I am surprised that
>   they are reset by DISCARD.  What is the use case you have in mind?
>   The use case I can envision is an application that sets a value right
> after
>   authentication, for use with row-level security.  But then it would be
> harmful
>   if the user could reset the variable with DISCARD.
>

Primary I think about IMMUTABLE variables like about some form of cache.
This cache is protected against unwanted repeated write - and can help with
detection of this situation.
It is not a security tool primarily - there are access rights for this
purpose. The scope of this cache can be different - sometimes session,
sometimes transaction. When you use a pooler like pgbouncer, then the
lifetime is ended by command DISCARD ALL. Moreover, after DISCARD ALL, the
session should be in default state, so then any session variable should be
not initialized. And there is an implementation reason too. When I handle
command DISCARD VARIABLES, I just reset related memory contexts. It is
quick and safe.

I can imagine more strict behaviour - like impossibility to discard context
or necessity to have default immutable expressions. But then this clause is
not useful for connection pooling with transaction mode :-/. So current
design is a compromise - if somebody wants strict behaviour, then can set
default immutable expressions (by self). And then there will not be a
problem with DISCARD VARIABLES.

But because it supports a possibility ON TRANSACTION END RESET, then
theoretically it can be used in transaction mode too without possibility to
be reset by DISCARD ALL.

At the end there is just one question - the DISCARD VARIABLES should
support some exceptions, or the immutability of variables should have some
exceptions? I can imagine the surprise so after DISCARD ALL, the immutable
variable lost a value, but the same surprise can be in the moment after
DISCARD ALL when the LET immutablevar statement fails. Then we have to
choose a priority - DISCARD x IMMUTABLE.

I prefer to have a stronger DISCARD VARIABLES command without exceptions
(for semantic, for simple implementation), but I am open to discussion. If
we want to support the IMMUTABLE clause, then we have to reply to the
mentioned question. What do you think about it?



>
> Documentation:
> RestoreArchive
>     --- a/doc/src/sgml/ddl.sgml
>     +++ b/doc/src/sgml/ddl.sgml
>
>     +   <para>
>     +    The session variables can be shadowed by column references in a
> query. When
>     +    a query contains identifiers or qualified identifiers that could
> be used as
>     +    both a session variable identifiers and as column identifier,
> then the
>     +    column identifier is preferred every time. Warnings can be
> emitted when
>     +    this situation happens by enabling configuration parameter <xref
>     +    linkend="guc-session-variables-ambiguity-warning"/>. User can
> explicitly
>     +    qualify the source object by syntax
> <literal>table.column</literal> or
>     +    <literal>variable.column</literal>.
>     +   </para>
>
>   I think you mean <literal>schema.variable</literal>, not
> <literal>variable.column</literal>.
>

fixed




> Yours,
> Laurenz Albe
>

  216. Re: proposal: schema variables

    Laurenz Albe <laurenz.albe@cybertec.at> — 2024-07-24T17:02:11Z

    On Wed, 2024-07-24 at 17:19 +0200, Pavel Stehule wrote:
> >   This is buggy:
> > 
> >     CREATE VARIABLE str AS text NOT NULL DEFAULT NULL;
> > 
> >   Ugh.
> > 
> >     SELECT str;
> >     ERROR:  null value is not allowed for NOT NULL session variable "laurenz.str"
> >     DETAIL:  The result of DEFAULT expression is NULL.
> > 
> >   Perhaps that is a leftover from the previous coding, but I think there need be
> >   no check upon SELECT.  It should be enough to check during CREATE VARIABLE and
> >   LET.
> 
> I think it is correct. When you use SELECT str, then DEFAULT expression is
> executed, and then the result is assigned to a variable, and there is NOT NULL
> guard, which raises an exception. The variable is not initialized when you run
> DDL, but it is initialized when you first read or write from/to the variable.
> The DEFAULT expression is not evaluated in DDL time. In this case, I can detect
> the problem in DDL time because the result is transformed to NULL node, but
> generally there can be SQL non immutable function, and then I need to wait until
> the DEFAULT expression will be evaluated - and it is time of first reading.
> Unfortunately, there is not an available check if some expression is NULL,
> that I can use in DDL time, but I have it in plpgsql_check.

That makes sense to me.

In that case, I think we can drop the requirement that NOT NULL variables
need a default clause.

> 
> >   I can see the usefulness of IMMUTABLE variables, but I am surprised that
> >   they are reset by DISCARD.  What is the use case you have in mind?
> >   The use case I can envision is an application that sets a value right after
> >   authentication, for use with row-level security.  But then it would be harmful
> >   if the user could reset the variable with DISCARD.
> 
> Primary I think about IMMUTABLE variables like about some form of cache.
> This cache is protected against unwanted repeated write - and can help with
> detection of this situation.

We can leave it as it is.  The IMMUTABLE feature need not go into the first
release, so that can be discussed some more later.

Thanks for the new patch set; I'll look at it soon.

Yours,
Laurenz Albe

  217. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-07-24T18:01:01Z

    st 24. 7. 2024 v 19:02 odesílatel Laurenz Albe <laurenz.albe@cybertec.at>
napsal:

> On Wed, 2024-07-24 at 17:19 +0200, Pavel Stehule wrote:
> > >   This is buggy:
> > >
> > >     CREATE VARIABLE str AS text NOT NULL DEFAULT NULL;
> > >
> > >   Ugh.
> > >
> > >     SELECT str;
> > >     ERROR:  null value is not allowed for NOT NULL session variable
> "laurenz.str"
> > >     DETAIL:  The result of DEFAULT expression is NULL.
> > >
> > >   Perhaps that is a leftover from the previous coding, but I think
> there need be
> > >   no check upon SELECT.  It should be enough to check during CREATE
> VARIABLE and
> > >   LET.
> >
> > I think it is correct. When you use SELECT str, then DEFAULT expression
> is
> > executed, and then the result is assigned to a variable, and there is
> NOT NULL
> > guard, which raises an exception. The variable is not initialized when
> you run
> > DDL, but it is initialized when you first read or write from/to the
> variable.
> > The DEFAULT expression is not evaluated in DDL time. In this case, I can
> detect
> > the problem in DDL time because the result is transformed to NULL node,
> but
> > generally there can be SQL non immutable function, and then I need to
> wait until
> > the DEFAULT expression will be evaluated - and it is time of first
> reading.
> > Unfortunately, there is not an available check if some expression is
> NULL,
> > that I can use in DDL time, but I have it in plpgsql_check.
>
> That makes sense to me.
>
> In that case, I think we can drop the requirement that NOT NULL variables
> need a default clause.
>

removed


>
> >
> > >   I can see the usefulness of IMMUTABLE variables, but I am surprised
> that
> > >   they are reset by DISCARD.  What is the use case you have in mind?
> > >   The use case I can envision is an application that sets a value
> right after
> > >   authentication, for use with row-level security.  But then it would
> be harmful
> > >   if the user could reset the variable with DISCARD.
> >
> > Primary I think about IMMUTABLE variables like about some form of cache.
> > This cache is protected against unwanted repeated write - and can help
> with
> > detection of this situation.
>
> We can leave it as it is.  The IMMUTABLE feature need not go into the first
> release, so that can be discussed some more later.
>
> Thanks for the new patch set; I'll look at it soon.
>

Thank you very much

Regards

Pavel



>
> Yours,
> Laurenz Albe
>

  218. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-07-24T19:03:15Z

    út 23. 7. 2024 v 23:41 odesílatel Laurenz Albe <laurenz.albe@cybertec.at>
napsal:

> On Tue, 2024-07-23 at 16:34 +0200, Laurenz Albe wrote:
> > CREATE VARIABLE command:
> >
> >   This is buggy:
> >
> >     CREATE VARIABLE str AS text NOT NULL DEFAULT NULL;
> >
> >   Ugh.
> >
> >     SELECT str;
> >     ERROR:  null value is not allowed for NOT NULL session variable
> "laurenz.str"
> >     DETAIL:  The result of DEFAULT expression is NULL.
> >
> >   Perhaps that is a leftover from the previous coding, but I think there
> need be
> >   no check upon SELECT.  It should be enough to check during CREATE
> VARIABLE and
> >   LET.
>
> I'm having second thoughts about that.
>
> I was thinking of a variable like of a table column, but there is a
> fundamental
> difference: there is a clear moment when a tuple is added (INSERT or
> UPDATE),
> which is the point where a column can be checked for NULL values.
>
> A variable can be SELECTed without having been LET before, in which case it
> has the default value.  But there is no way to test the default value
> before
> the variable is SELECTed.  So while DEFAULT NULL for a non-nullable
> variable
> seems weird, it is no worse than DEFAULT somefunc() for a function that
> returns
> NULL.
>
> So perhaps the behavior I complained about above is actually the right one.
> In the view of that, it doesn't seem necessary to enforce a DEFAULT value
> for
> a NOT NULL variable: NOT NULL might just as well mean "you have to LET it
> before
> you can SELECT it".
>

exactly


>
> > IMMUTABLE variables:
> >
> >     +   <varlistentry id="sql-createvariable-immutable">
> >     +    <term><literal>IMMUTABLE</literal></term>
> >     +    <listitem>
> >     +     <para>
> >     +      The assigned value of the session variable can not be changed.
> >     +      Only if the session variable doesn't have a default value, a
> single
> >     +      initialization is allowed using the <command>LET</command>
> command. Once
> >     +      done, no further change is allowed until end of transaction
> >     +      if the session variable was created with clause <literal>ON
> TRANSACTION
> >     +      END RESET</literal>, or until reset of all session variables
> by
> >     +      <command>DISCARD VARIABLES</command>, or until reset of all
> session
> >     +      objects by command <command>DISCARD ALL</command>.
> >     +     </para>
> >     +    </listitem>
> >     +   </varlistentry>
> >
> >   I can see the usefulness of IMMUTABLE variables, but I am surprised
> that
> >   they are reset by DISCARD.  What is the use case you have in mind?
> >   The use case I can envision is an application that sets a value right
> after
> >   authentication, for use with row-level security.  But then it would be
> harmful
> >   if the user could reset the variable with DISCARD.
>
> I'm beginning to be uncertain about that as well.  You might want to use a
> connection pool, and you LET the variable when you take it out of the pool.
> When the session is returned to the pool, variables get DISCARDed.
>
> Sure, a user can call DISCARD, but only if he or she is in an interactive
> session.
>
> So perhaps it is good as it is.
>

I think this design should work. There are a lot of scenarios, where
session variables can be used well, and sure, there will be scenarios where
it doesn't work well, but now, I think it is a good balance between
usability, complexity and code complexity. There are a lot of lines, but
the code is almost very simple.

Regards

Pavel


>
> Yours,
> Laurenz Albe
>

  219. Re: proposal: schema variables

    Laurenz Albe <laurenz.albe@cybertec.at> — 2024-07-25T13:52:47Z

    Thanks for the updated patch set.

I found a problem in 0019-transactional-variables.patch:

--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -9851,6 +9851,17 @@ SCRAM-SHA-256$<replaceable>&lt;iteration count&gt;</replaceable>:<replaceable>&l
       </para></entry>
      </row>
 
+     <row>
+      <entry><structfield>varistransact</structfield></entry>
+      <entry><type>boolean</type></entry>
+      <entry></entry>
+      <entry>
+       True, when the variable is "transactional". In case of transaction
+       rollback, transactional variables are reset to their content at the
+       transaction start. The default value is false.
+      </entry>
+     </row>

That's messed up; it should be

     <row>
      <entry role="catalog_table_entry"><para role="column_definition">
       <structfield>varistransact</structfield> <type>boolean</type>
      </para>
      <para>
       True, when the variable is <quote>transactional</quote>. In the case
       of a transaction rollback, transactional variables are reset to the
       value they had when the transaction started. The default value is
       <literal>false</literal>.
      </para></entry>
     </row>

I have started reading through the first patch, and so far I have only found
language problems.

I wonder how I should go about this.  At first, I intended to send an edited
version of the first patch, but as later patches depend on earlier patches,
that would mess up the patch set.

I can send my suggested modifications in text, but then you have to copy and
paste them all, which is cumbersome.

What would be best for you?


Thinking further, I wondered about the order of patches.
If some committer eventually takes mercy on this patch set, I expect that
only a part of the functionality will go in as a first step.
Does the order of the patches in the patch set match such a process?

I'd guess that temporary session variables or ON TRANSACTION END RESET
would be things that can be committed later on, but PL/pgSQL support
should be in the first commit.

What is your approach to that?

Yours,
Laurenz Albe

  220. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-07-25T16:32:24Z

    Hi

čt 25. 7. 2024 v 15:52 odesílatel Laurenz Albe <laurenz.albe@cybertec.at>
napsal:

> Thanks for the updated patch set.
>
> I found a problem in 0019-transactional-variables.patch:
>
> --- a/doc/src/sgml/catalogs.sgml
> +++ b/doc/src/sgml/catalogs.sgml
> @@ -9851,6 +9851,17 @@ SCRAM-SHA-256$<replaceable>&lt;iteration
> count&gt;</replaceable>:<replaceable>&l
>        </para></entry>
>       </row>
>
> +     <row>
> +      <entry><structfield>varistransact</structfield></entry>
> +      <entry><type>boolean</type></entry>
> +      <entry></entry>
> +      <entry>
> +       True, when the variable is "transactional". In case of transaction
> +       rollback, transactional variables are reset to their content at the
> +       transaction start. The default value is false.
> +      </entry>
> +     </row>
>
> That's messed up; it should be
>
>      <row>
>       <entry role="catalog_table_entry"><para role="column_definition">
>        <structfield>varistransact</structfield> <type>boolean</type>
>       </para>
>       <para>
>        True, when the variable is <quote>transactional</quote>. In the case
>        of a transaction rollback, transactional variables are reset to the
>        value they had when the transaction started. The default value is
>        <literal>false</literal>.
>       </para></entry>
>      </row>
>
>
changed


> I have started reading through the first patch, and so far I have only
> found
> language problems.
>
> I wonder how I should go about this.  At first, I intended to send an
> edited
> version of the first patch, but as later patches depend on earlier patches,
> that would mess up the patch set.
>
> I can send my suggested modifications in text, but then you have to copy
> and
> paste them all, which is cumbersome.
>
> What would be best for you?
>

send me it in any form - probably the diff of patches are best. I'll do
necessary fixes.

There are not good tool for maintaining chronologically incremental
patchset


>
> Thinking further, I wondered about the order of patches.
> If some committer eventually takes mercy on this patch set, I expect that
> only a part of the functionality will go in as a first step.
> Does the order of the patches in the patch set match such a process?
>

yes

we talk so first seven patches are mandatory - others are optional and
introduce new functionality or increase performance



>
> I'd guess that temporary session variables or ON TRANSACTION END RESET
> would be things that can be committed later on, but PL/pgSQL support
> should be in the first commit.
>

The plpgsql is supported by a second patch, because variables are
accessible by queries. But patch 16 introduces the
PLpgSQL LET command, and then allows us to evaluate an expression as a
simple expression, which is significantly faster.


> What is your approach to that?
>

The first six, seven patches are fundamental.  For others we can talk about
priorities. I see performance related patches to be more important, but
they are a little bit more technically difficult or maybe related to a not
too cool area now (like PL/pgSQL - I like it, but almost all people's
stored procedures aren't used). Implementation of new features is almost
done by simple patches. And again for somebody temp variables are not
important (usage temp variables in PLpgSQL is +/- zero), but for others can
be very important (it can be very interesting for writing psql scripts,
because it allows parametrization of DO commands, and it is cheaper than
temp tables for passing result).

So others patches have some order, because there has to be some order, but
it is not final or immutable. I can imagine throwing these patches to
different patchset. On the second hand, the size of these patches is less
than the first two, and I think it is interesting, because from my
perspective, these patches cover this area completely. But again, the order
of these patches is important for some dependencies in files (regress
tests), but it doesn't draw the priority of implemented features. Current
state is the result of my work or work of other peoples that did review of
these patches. I am open to changing this order, if somebody would do it.

Regards

Pavel



> Yours,
> Laurenz Albe
>

  221. Re: proposal: schema variables

    Laurenz Albe <laurenz.albe@cybertec.at> — 2024-07-27T14:19:23Z

    I went through the v20240724-2-0001 patch and mostly changed some documentation
and the comments.  Attached is my updated version.

Comments:

> --- a/src/backend/catalog/namespace.c
> +++ b/src/backend/catalog/namespace.c
> [...]
> +bool
> +VariableIsVisible(Oid varid)
> [...]
> +   /*
> +    * Quick check: if it ain't in the path at all, it ain't visible. Items in
> +    * the system namespace are surely in the path and so we needn't even do
> +    * list_member_oid() for them.
> +    */
> +   varnamespace = varform->varnamespace;
> +   if (varnamespace != PG_CATALOG_NAMESPACE &&
> +       !list_member_oid(activeSearchPath, varnamespace))
> +       visible = false;
> +   else

I cannot imagine that we'll ever have session variables in "pg_catalog".
Did you put that there as defensive coding?

> --- a/src/backend/catalog/namespace.c
> +++ b/src/backend/catalog/namespace.c
> [...]
> +Datum
> +pg_variable_is_visible(PG_FUNCTION_ARGS)

That function should get documented in functions-info.html#FUNCTIONS-INFO-SCHEMA-TABLE
I have added an entry in the attached patch.

> --- /dev/null
> +++ b/doc/src/sgml/ref/create_variable.sgml
> [...]
> +  <note>
> +   <para>
> +    Inside a query or an expression, the session variable can be shadowed by
> +    column or by routine's variable or routine argument. Such collisions of
> +    identifiers can be resolved by using qualified identifiers. Session variables
> +    can use schema name, columns can use table aliases, routine variables
> +    can use block labels, and routine arguments can use the routine name.
> +   </para>
> +  </note>
> + </refsect1>

I have slightly rewritten the text and moved it to doc/src/sgml/ddl.sgml.
I felt that to be a better place for generic information about session variable's
behavior.  Also, the single paragraph in doc/src/sgml/ddl.sgml felt lonely.
I left the note in CREATE VARIABLE, but it is now just a link to ddl.sgml.

> --- a/src/bin/pg_dump/pg_dump.c
> +++ b/src/bin/pg_dump/pg_dump.c
> [...]
> +void
> +getVariables(Archive *fout)
> +{
> [...]
> +   for (i = 0; i < ntups; i++)
> +   {
> [...]
> +       /* Decide whether we want to dump it */
> +       selectDumpableObject(&(varinfo[i].dobj), fout);
> +
> +       /* Do not try to dump ACL if no ACL exists. */
> +       if (!PQgetisnull(res, i, i_varacl))
> +           varinfo[i].dobj.components |= DUMP_COMPONENT_ACL;
> +
> +       if (strlen(varinfo[i].rolname) == 0)
> +           pg_log_warning("owner of variable \"%s\" appears to be invalid",
> +                          varinfo[i].dobj.name);
> +
> +       /* Decide whether we want to dump it */
> +       selectDumpableObject(&(varinfo[i].dobj), fout);
> +
> +       vtype = findTypeByOid(varinfo[i].vartype);
> +       addObjectDependency(&varinfo[i].dobj, vtype->dobj.dumpId);
> +   }

One of the two selectDumpableObject() calls seems redundant.
I removed the first one; I hope that's OK.

> --- a/src/bin/psql/tab-complete.c
> +++ b/src/bin/psql/tab-complete.c
> [...]
> @@ -4421,7 +4449,7 @@ psql_completion(const char *text, int start, int end)
>  
>  /* PREPARE xx AS */
>     else if (Matches("PREPARE", MatchAny, "AS"))
> -       COMPLETE_WITH("SELECT", "UPDATE", "INSERT INTO", "DELETE FROM");
> +       COMPLETE_WITH("SELECT", "UPDATE", "INSERT INTO", "DELETE FROM", "LET");

I guess that belongs in the second patch, but I didn't change it.
Since the feature cannot be committed without LET, it doesn't really matter in
which of the two patches it is.

> --- a/src/test/regress/expected/psql.out
> +++ b/src/test/regress/expected/psql.out
> [...]
> +\dV regression|mydb.public.var
> +cross-database references are not implemented: regression|mydb.public.var

That's a weird test - why the pipe?  Is there something I don't get?
There is already another test for cross-database references.

> +\dV "no.such.database"."no.such.schema"."no.such.variable"
> +cross-database references are not implemented: "no.such.database"."no.such.schema"."no.such.variable"

And another one.  Do we need so many tests for that?


I hope I didn't create too many conflicts with the rest of the patch set.

I plan to review the other patches too, but I'll go on vacations for three weeks
in a week, and fall promises to be busy.  So it might be a while.

Yours,
Laurenz Albe

  222. Re: proposal: schema variables

    Laurenz Albe <laurenz.albe@cybertec.at> — 2024-07-29T09:56:46Z

    On Sat, 2024-07-27 at 16:19 +0200, Laurenz Albe wrote:
> I went through the v20240724-2-0001 patch and mostly changed some documentation
> and the comments.  Attached is my updated version.

Sorry; I forgot the new files.  Here is an updated replacement for your first patch.

I'll start working on the second patch now.

Yours,
Laurenz Albe

  223. Re: proposal: schema variables

    Laurenz Albe <laurenz.albe@cybertec.at> — 2024-07-30T19:46:46Z

    This is my review of the second patch in the series.

Again, I mostly changed code comments and documentation.

Noteworthy remarks:

- A general reminder: single line comments should start with a lower case
  letter and have to period in the end:

  /* this is a typical single line comment */

- Variable as parameter:

  CREATE VARIABLE var AS date;
  LET var = current_date;
  PREPARE stmt(date) AS SELECT $1;
  EXECUTE stmt(var);
  ERROR:  paramid of PARAM_VARIABLE param is out of range

  Is that working as intended?  I don't understand the error message.

  Perhaps there is a bug in src/backend/executor/execExpr.c.

- IdentifyVariable

  > --- a/src/backend/catalog/namespace.c
  > +++ b/src/backend/catalog/namespace.c
  > [...]
  > +/*
  > + * IdentifyVariable - try to find variable identified by list of names.
  > [...]

  The function comment doesn't make clear to me what the function does.
  Perhaps that is so confusing because the function seems to serve several
  purposes (during parsing, in ANALYZE, and to identify shadowed variables
  in a later patch).

  I rewrote the function comment, but didn't touch the code or code comments.

  Perhaps part of the reason why this function is so complicated is that
  you support things like this:

    CREATE SCHEMA sch;
    CREATE TYPE cp AS (a integer, b integer);
    CREATE VARIABLE sch.v AS cp;
    LET sch.v = (1, 2);
    SELECT sch.v.b;
     b 
    ═══
     2
    (1 row)

    test=# SELECT test.sch.v.b;
     b 
    ═══
     2
    (1 row)

  We don't support that for tables:

    CREATE TABLE tab (c cp);
    INSERT INTO tab VALUES (ROW(42, 43));
    SELECT tab.c.b FROM tab;
    ERROR:  cross-database references are not implemented: tab.c.b

  You have to use extra parentheses:

    SELECT (tab.c).b FROM tab;
     b  
    ════
     43
    (1 row)

  I'd say that this should be the same for session variables.
  What do you think?

  Code comments:

  - There are lots of variables declared at the beginning, but they are only
    used in code blocks further down.  For clarity, you should declare a
    variable in the code block where it is used.

  - +                   /*
    +                    * In this case, "a" is used as catalog name - check it.
    +                    */
    +                   if (strcmp(a, get_database_name(MyDatabaseId)) != 0)
    +                   {
    +                       if (!noerror)
    +                           ereport(ERROR,
    +                                   (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
    +                                    errmsg("cross-database references are not implemented: %s",
    +                                           NameListToString(names))));
    +                   }
    +                   else
    +                   {

    There needn't be an "else", since the ereport() will never return.
    Less indentation is better.

- src/backend/commands/session_variable.c

  There is one comment that confuses me and that I did not edit:

  > +Datum
  > +GetSessionVariable(Oid varid, bool *isNull, Oid *typid)
  > +{
  > +   SVariable   svar;
  > +
  > +   svar = get_session_variable(varid);
  > +
  > +   /*
  > +    * Although svar is freshly validated in this point, the svar->is_valid can
  > +    * be false, due possible accepting invalidation message inside domain
  > +    * check. Now, the validation is done after lock, that can also accept
  > +    * invalidation message, so validation should be trustful.
  > +    *
  > +    * For now, we don't need to repeat validation. Only svar should be valid
  > +    * pointer.
  > +    */
  > +   Assert(svar);
  > +
  > +   *typid = svar->typid;
  > +
  > +   return copy_session_variable_value(svar, isNull);
  > +}

- src/backend/executor/execMain.c

  > @@ -196,6 +198,51 @@ standard_ExecutorStart(QueryDesc *queryDesc, int eflags)
  >     Assert(queryDesc->sourceText != NULL);
  >     estate->es_sourceText = queryDesc->sourceText;
  >
  > +   /*
  > +    * The executor doesn't work with session variables directly. Values of
  > +    * related session variables are copied to dedicated array, and this array
  > +    * is passed to executor.
  > +    */

  Why?  Is that a performance measure?  The comment should explain that.

- parallel safety

  > --- a/src/backend/optimizer/util/clauses.c
  > +++ b/src/backend/optimizer/util/clauses.c
  > @@ -935,6 +936,13 @@ max_parallel_hazard_walker(Node *node, max_parallel_hazard_context *context)
  >         if (param->paramkind == PARAM_EXTERN)
  >             return false;
  >
  > +       /* We doesn't support passing session variables to workers */
  > +       if (param->paramkind == PARAM_VARIABLE)
  > +       {
  > +           if (max_parallel_hazard_test(PROPARALLEL_RESTRICTED, context))
  > +               return true;
  > +       }

  Even if a later patch alleviates that restriction, this patch should document that
  session variables imply "parallel restricted".  I have added that to doc/src/sgml/parallel.sgml.

Attached are the first two patches with my edits (the first patch is unchanged;
I just add it again to keep the cfbot happy.

I hope to get to review the other patches at some later time.

Yours,
Laurenz Albe

  224. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-07-30T20:56:19Z

    Hi

so 27. 7. 2024 v 16:19 odesílatel Laurenz Albe <laurenz.albe@cybertec.at>
napsal:

> I went through the v20240724-2-0001 patch and mostly changed some
> documentation
> and the comments.  Attached is my updated version.
>
> Comments:
>
> > --- a/src/backend/catalog/namespace.c
> > +++ b/src/backend/catalog/namespace.c
> > [...]
> > +bool
> > +VariableIsVisible(Oid varid)
> > [...]
> > +   /*
> > +    * Quick check: if it ain't in the path at all, it ain't visible.
> Items in
> > +    * the system namespace are surely in the path and so we needn't
> even do
> > +    * list_member_oid() for them.
> > +    */
> > +   varnamespace = varform->varnamespace;
> > +   if (varnamespace != PG_CATALOG_NAMESPACE &&
> > +       !list_member_oid(activeSearchPath, varnamespace))
> > +       visible = false;
> > +   else
>
> I cannot imagine that we'll ever have session variables in "pg_catalog".
> Did you put that there as defensive coding?
>

No, I just used a pattern for relations. I removed the test against
PG_CATALOG_NAMESPACE and wrote comment

<-->/*
<--> * Quick check: if it ain't in the path at all, it ain't visible. We
<--> * don't expect usage of session variables in the system namespace.
<--> */




>
> > --- a/src/backend/catalog/namespace.c
> > +++ b/src/backend/catalog/namespace.c
> > [...]
> > +Datum
> > +pg_variable_is_visible(PG_FUNCTION_ARGS)
>
> That function should get documented in
> functions-info.html#FUNCTIONS-INFO-SCHEMA-TABLE
> I have added an entry in the attached patch.
>

merged


>
> > --- /dev/null
> > +++ b/doc/src/sgml/ref/create_variable.sgml
> > [...]
> > +  <note>
> > +   <para>
> > +    Inside a query or an expression, the session variable can be
> shadowed by
> > +    column or by routine's variable or routine argument. Such
> collisions of
> > +    identifiers can be resolved by using qualified identifiers. Session
> variables
> > +    can use schema name, columns can use table aliases, routine
> variables
> > +    can use block labels, and routine arguments can use the routine
> name.
> > +   </para>
> > +  </note>
> > + </refsect1>
>
> I have slightly rewritten the text and moved it to doc/src/sgml/ddl.sgml.
> I felt that to be a better place for generic information about session
> variable's
> behavior.  Also, the single paragraph in doc/src/sgml/ddl.sgml felt lonely.
> I left the note in CREATE VARIABLE, but it is now just a link to ddl.sgml.
>

merged


>
> > --- a/src/bin/pg_dump/pg_dump.c
> > +++ b/src/bin/pg_dump/pg_dump.c
> > [...]
> > +void
> > +getVariables(Archive *fout)
> > +{
> > [...]
> > +   for (i = 0; i < ntups; i++)
> > +   {
> > [...]
> > +       /* Decide whether we want to dump it */
> > +       selectDumpableObject(&(varinfo[i].dobj), fout);
> > +
> > +       /* Do not try to dump ACL if no ACL exists. */
> > +       if (!PQgetisnull(res, i, i_varacl))
> > +           varinfo[i].dobj.components |= DUMP_COMPONENT_ACL;
> > +
> > +       if (strlen(varinfo[i].rolname) == 0)
> > +           pg_log_warning("owner of variable \"%s\" appears to be
> invalid",
> > +                          varinfo[i].dobj.name);
> > +
> > +       /* Decide whether we want to dump it */
> > +       selectDumpableObject(&(varinfo[i].dobj), fout);
> > +
> > +       vtype = findTypeByOid(varinfo[i].vartype);
> > +       addObjectDependency(&varinfo[i].dobj, vtype->dobj.dumpId);
> > +   }
>
> One of the two selectDumpableObject() calls seems redundant.
> I removed the first one; I hope that's OK.
>

sure


>
> > --- a/src/bin/psql/tab-complete.c
> > +++ b/src/bin/psql/tab-complete.c
> > [...]
> > @@ -4421,7 +4449,7 @@ psql_completion(const char *text, int start, int
> end)
> >
> >  /* PREPARE xx AS */
> >     else if (Matches("PREPARE", MatchAny, "AS"))
> > -       COMPLETE_WITH("SELECT", "UPDATE", "INSERT INTO", "DELETE FROM");
> > +       COMPLETE_WITH("SELECT", "UPDATE", "INSERT INTO", "DELETE FROM",
> "LET");
>
> I guess that belongs in the second patch, but I didn't change it.
> Since the feature cannot be committed without LET, it doesn't really
> matter in
> which of the two patches it is.
>
> > --- a/src/test/regress/expected/psql.out
> > +++ b/src/test/regress/expected/psql.out
> > [...]
> > +\dV regression|mydb.public.var
> > +cross-database references are not implemented:
> regression|mydb.public.var
>

> That's a weird test - why the pipe?  Is there something I don't get?
>

This test is not related to pipe, but usage of invalid multipart names

These tests look weird, but when you look at the psql.out file, then it is
consistent with others.

There is already another test for cross-database references.
>
> > +\dV "no.such.database"."no.such.schema"."no.such.variable"
> > +cross-database references are not implemented:
> "no.such.database"."no.such.schema"."no.such.variable"
>
> And another one.  Do we need so many tests for that?
>

It is crash psql parser test - these tests are designed like \dt or \di


>
>
> I hope I didn't create too many conflicts with the rest of the patch set.
>
> I plan to review the other patches too, but I'll go on vacations for three
> weeks
> in a week, and fall promises to be busy.  So it might be a while.
>

I am sending patches with merged your changes (related to first patch)

Regards

Pavel



>
> Yours,
> Laurenz Albe
>

  225. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-07-31T06:41:50Z

    út 30. 7. 2024 v 21:46 odesílatel Laurenz Albe <laurenz.albe@cybertec.at>
napsal:

> This is my review of the second patch in the series.
>
> Again, I mostly changed code comments and documentation.
>
> Noteworthy remarks:
>
> - A general reminder: single line comments should start with a lower case
>   letter and have to period in the end:
>
>   /* this is a typical single line comment */
>
> - Variable as parameter:
>
>   CREATE VARIABLE var AS date;
>   LET var = current_date;
>   PREPARE stmt(date) AS SELECT $1;
>   EXECUTE stmt(var);
>   ERROR:  paramid of PARAM_VARIABLE param is out of range
>
>   Is that working as intended?  I don't understand the error message.
>
>   Perhaps there is a bug in src/backend/executor/execExpr.c.
>
> - IdentifyVariable
>
>   > --- a/src/backend/catalog/namespace.c
>   > +++ b/src/backend/catalog/namespace.c
>   > [...]
>   > +/*
>   > + * IdentifyVariable - try to find variable identified by list of
> names.
>   > [...]
>
>   The function comment doesn't make clear to me what the function does.
>   Perhaps that is so confusing because the function seems to serve several
>   purposes (during parsing, in ANALYZE, and to identify shadowed variables
>   in a later patch).
>
>   I rewrote the function comment, but didn't touch the code or code
> comments.
>
>   Perhaps part of the reason why this function is so complicated is that
>   you support things like this:
>
>     CREATE SCHEMA sch;
>     CREATE TYPE cp AS (a integer, b integer);
>     CREATE VARIABLE sch.v AS cp;
>     LET sch.v = (1, 2);
>     SELECT sch.v.b;
>      b
>     ═══
>      2
>     (1 row)
>
>     test=# SELECT test.sch.v.b;
>      b
>     ═══
>      2
>     (1 row)
>
>   We don't support that for tables:
>
>     CREATE TABLE tab (c cp);
>     INSERT INTO tab VALUES (ROW(42, 43));
>     SELECT tab.c.b FROM tab;
>     ERROR:  cross-database references are not implemented: tab.c.b
>
>   You have to use extra parentheses:
>
>     SELECT (tab.c).b FROM tab;
>      b
>     ════
>      43
>     (1 row)
>
>   I'd say that this should be the same for session variables.
>   What do you think?
>
>   Code comments:
>
>   - There are lots of variables declared at the beginning, but they are
> only
>     used in code blocks further down.  For clarity, you should declare a
>     variable in the code block where it is used.
>
>   - +                   /*
>     +                    * In this case, "a" is used as catalog name -
> check it.
>     +                    */
>     +                   if (strcmp(a, get_database_name(MyDatabaseId)) !=
> 0)
>     +                   {
>     +                       if (!noerror)
>     +                           ereport(ERROR,
>     +
>  (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
>     +                                    errmsg("cross-database references
> are not implemented: %s",
>     +                                           NameListToString(names))));
>     +                   }
>     +                   else
>     +                   {
>
>     There needn't be an "else", since the ereport() will never return.
>     Less indentation is better.
>
> - src/backend/commands/session_variable.c
>
>   There is one comment that confuses me and that I did not edit:
>
>   > +Datum
>   > +GetSessionVariable(Oid varid, bool *isNull, Oid *typid)
>   > +{
>   > +   SVariable   svar;
>   > +
>   > +   svar = get_session_variable(varid);
>   > +
>   > +   /*
>   > +    * Although svar is freshly validated in this point, the
> svar->is_valid can
>   > +    * be false, due possible accepting invalidation message inside
> domain
>   > +    * check. Now, the validation is done after lock, that can also
> accept
>   > +    * invalidation message, so validation should be trustful.
>   > +    *
>   > +    * For now, we don't need to repeat validation. Only svar should
> be valid
>   > +    * pointer.
>   > +    */
>   > +   Assert(svar);
>   > +
>   > +   *typid = svar->typid;
>   > +
>   > +   return copy_session_variable_value(svar, isNull);
>   > +}
>
> - src/backend/executor/execMain.c
>
>   > @@ -196,6 +198,51 @@ standard_ExecutorStart(QueryDesc *queryDesc, int
> eflags)
>   >     Assert(queryDesc->sourceText != NULL);
>   >     estate->es_sourceText = queryDesc->sourceText;
>   >
>   > +   /*
>   > +    * The executor doesn't work with session variables directly.
> Values of
>   > +    * related session variables are copied to dedicated array, and
> this array
>   > +    * is passed to executor.
>   > +    */
>
>   Why?  Is that a performance measure?  The comment should explain that.
>
> - parallel safety
>
>   > --- a/src/backend/optimizer/util/clauses.c
>   > +++ b/src/backend/optimizer/util/clauses.c
>   > @@ -935,6 +936,13 @@ max_parallel_hazard_walker(Node *node,
> max_parallel_hazard_context *context)
>   >         if (param->paramkind == PARAM_EXTERN)
>   >             return false;
>   >
>   > +       /* We doesn't support passing session variables to workers */
>   > +       if (param->paramkind == PARAM_VARIABLE)
>   > +       {
>   > +           if (max_parallel_hazard_test(PROPARALLEL_RESTRICTED,
> context))
>   > +               return true;
>   > +       }
>
>   Even if a later patch alleviates that restriction, this patch should
> document that
>   session variables imply "parallel restricted".  I have added that to
> doc/src/sgml/parallel.sgml.
>
> Attached are the first two patches with my edits (the first patch is
> unchanged;
> I just add it again to keep the cfbot happy.
>

Probably you didn't attach new files - the second patch is not complete. Or
you didn't make changes there?

Regards

Pavel



>
> I hope to get to review the other patches at some later time.
>
> Yours,
> Laurenz Albe
>

  226. Re: proposal: schema variables

    Laurenz Albe <laurenz.albe@cybertec.at> — 2024-07-31T06:57:31Z

    On Wed, 2024-07-31 at 08:41 +0200, Pavel Stehule wrote:
> Probably you didn't attach new files - the second patch is not complete. Or you didn't make changes there?

Hm.  What is missing?

Yours,
Laurenz Albe

  227. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-07-31T07:04:08Z

    st 31. 7. 2024 v 8:57 odesílatel Laurenz Albe <laurenz.albe@cybertec.at>
napsal:

> On Wed, 2024-07-31 at 08:41 +0200, Pavel Stehule wrote:
> > Probably you didn't attach new files - the second patch is not complete.
> Or you didn't make changes there?
>
> Hm.  What is missing?
>

let.sgml,
session_variable.c
svariable_receiver.c
session_variable.h
...

Regards

Pavel




>
> Yours,
> Laurenz Albe
>

  228. Re: proposal: schema variables

    Laurenz Albe <laurenz.albe@cybertec.at> — 2024-07-31T09:45:00Z

    On Wed, 2024-07-31 at 09:04 +0200, Pavel Stehule wrote:
> st 31. 7. 2024 v 8:57 odesílatel Laurenz Albe <laurenz.albe@cybertec.at> napsal:
> > On Wed, 2024-07-31 at 08:41 +0200, Pavel Stehule wrote:
> > > Probably you didn't attach new files - the second patch is not complete. Or you didn't make changes there?
> > 
> > Hm.  What is missing?
> 
> let.sgml,
> session_variable.c
> svariable_receiver.c 
> session_variable.h
> ...

Argh, I forgit the new files, sorry.

The attached patches should be complete.

Yours,
Laurenz Albe

  229. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-07-31T22:05:54Z

    Hi

st 31. 7. 2024 v 11:45 odesílatel Laurenz Albe <laurenz.albe@cybertec.at>
napsal:

> On Wed, 2024-07-31 at 09:04 +0200, Pavel Stehule wrote:
> > st 31. 7. 2024 v 8:57 odesílatel Laurenz Albe <laurenz.albe@cybertec.at>
> napsal:
> > > On Wed, 2024-07-31 at 08:41 +0200, Pavel Stehule wrote:
> > > > Probably you didn't attach new files - the second patch is not
> complete. Or you didn't make changes there?
> > >
> > > Hm.  What is missing?
> >
> > let.sgml,
> > session_variable.c
> > svariable_receiver.c
> > session_variable.h
> > ...
>
> Argh, I forgit the new files, sorry.
>
> The attached patches should be complete.
>

merged Laurenz's patches

Regards

Pavel


>
> Yours,
> Laurenz Albe
>

  230. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-08-01T06:12:13Z

    Hi

fresh rebase + fix format in doc

Regards

Pavel

  231. Re: proposal: schema variables

    Erik Rijkers <er@xs4all.nl> — 2024-08-01T07:45:34Z

    doc-build fails: a slash tumbled backwards in doc/src/sgml:

$ grep 'xref.*.\\>'  *.sgml
plpgsql.sgml:     (see <xref linkend="ddl-session-variables"\>).

That should simply be a forward slash, of course.

Thanks,

Erik Rijkers




Op 8/1/24 om 08:12 schreef Pavel Stehule:
> Hi
> 
> fresh rebase + fix format in doc
> 
> Regards
> 
> Pavel
>

  232. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-08-01T07:48:16Z

    čt 1. 8. 2024 v 9:45 odesílatel Erik Rijkers <er@xs4all.nl> napsal:

> doc-build fails: a slash tumbled backwards in doc/src/sgml:
>
> $ grep 'xref.*.\\>'  *.sgml
> plpgsql.sgml:     (see <xref linkend="ddl-session-variables"\>).
>
> That should simply be a forward slash, of course.
>

should be fixed in my today patchset



>
> Thanks,
>
> Erik Rijkers
>
>
>
>
> Op 8/1/24 om 08:12 schreef Pavel Stehule:
> > Hi
> >
> > fresh rebase + fix format in doc
> >
> > Regards
> >
> > Pavel
> >
>

  233. Re: proposal: schema variables

    Laurenz Albe <laurenz.albe@cybertec.at> — 2024-08-01T11:22:01Z

    On Thu, 2024-08-01 at 08:12 +0200, Pavel Stehule wrote:
> fresh rebase + fix format in doc

Thanks!

I'm curious, but too lazy to build the patch now, so I'm asking:
what did you do about this error?

> CREATE VARIABLE var AS date;
> LET var = current_date;
> PREPARE stmt(date) AS SELECT $1;
> EXECUTE stmt(var);
> ERROR:  paramid of PARAM_VARIABLE param is out of range

Or does a later patch take care of that?

Yours,
Laurenz Albe

  234. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-08-01T11:24:58Z

    čt 1. 8. 2024 v 13:22 odesílatel Laurenz Albe <laurenz.albe@cybertec.at>
napsal:

> On Thu, 2024-08-01 at 08:12 +0200, Pavel Stehule wrote:
> > fresh rebase + fix format in doc
>
> Thanks!
>
> I'm curious, but too lazy to build the patch now, so I'm asking:
> what did you do about this error?
>

I try to investigate this issue now.

The patchset is just merging of your work


> > CREATE VARIABLE var AS date;
> > LET var = current_date;
> > PREPARE stmt(date) AS SELECT $1;
> > EXECUTE stmt(var);
> > ERROR:  paramid of PARAM_VARIABLE param is out of range
>


>
> Or does a later patch take care of that?
>

This is a clear bug, and I have to fix it. I hope so I'll do this today



>
> Yours,
> Laurenz Albe
>

  235. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-08-01T22:02:27Z

    čt 1. 8. 2024 v 13:22 odesílatel Laurenz Albe <laurenz.albe@cybertec.at>
napsal:

> On Thu, 2024-08-01 at 08:12 +0200, Pavel Stehule wrote:
> > fresh rebase + fix format in doc
>
> Thanks!
>
> I'm curious, but too lazy to build the patch now, so I'm asking:
> what did you do about this error?
>
> > CREATE VARIABLE var AS date;
> > LET var = current_date;
> > PREPARE stmt(date) AS SELECT $1;
> > EXECUTE stmt(var);
> > ERROR:  paramid of PARAM_VARIABLE param is out of range
>
> Or does a later patch take care of that?
>

It is working with patch "allow read an value of session variable directly
from  expression executor"

It looks very similar to evaluation of parameters of CALL statements.
Without the mentioned patch, the CALL statements don't allow variables as
an argument. I blocked usage of session variables as EXECUTE parameters
until direct access to session variables in expression evaluation is
supported.

Regards

Pavel






> Yours,
> Laurenz Albe
>

  236. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-08-03T05:40:00Z

    Hi

fix build plpgsql

Regards

Pavel

  237. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-08-07T10:00:14Z

    Hi

rebase + usage hash_seq_init_with_hash_value

Attention - regress tests fails due problem with
hash_seq_init_with_hash_value

Regards

Pavel

  238. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-08-08T05:17:00Z

    Hi

fresh rebase + revert usage of hash_seq_init_with_hash_value

regards

Pavel

  239. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-08-15T05:55:15Z

    út 30. 7. 2024 v 21:46 odesílatel Laurenz Albe <laurenz.albe@cybertec.at>
napsal:

> This is my review of the second patch in the series.
>
> Again, I mostly changed code comments and documentation.
>
> Noteworthy remarks:
>
> - A general reminder: single line comments should start with a lower case
>   letter and have to period in the end:
>

Should it be "have not to period in the end" ?


>
>   /* this is a typical single line comment */
>

I fixed almost all without parts related to psql and executor - there
almost all current comments break the mentioned rule. So I use the style
used in these files.  I can fix it (if you like it) - or it can be fixed
generally in a separated patch set.



> - Variable as parameter:
>
>   CREATE VARIABLE var AS date;
>   LET var = current_date;
>   PREPARE stmt(date) AS SELECT $1;
>   EXECUTE stmt(var);
>   ERROR:  paramid of PARAM_VARIABLE param is out of range
>
>   Is that working as intended?  I don't understand the error message.
>

fixed in 0002 patch (variables cannot be used as EXECUTE argument) and 0014
(enable usage variables as an argument of EXECUTE)


>
>   Perhaps there is a bug in src/backend/executor/execExpr.c.
>
> - IdentifyVariable
>
>   > --- a/src/backend/catalog/namespace.c
>   > +++ b/src/backend/catalog/namespace.c
>   > [...]
>   > +/*
>   > + * IdentifyVariable - try to find variable identified by list of
> names.
>   > [...]
>
>   The function comment doesn't make clear to me what the function does.
>   Perhaps that is so confusing because the function seems to serve several
>   purposes (during parsing, in ANALYZE, and to identify shadowed variables
>   in a later patch).
>
>   I rewrote the function comment, but didn't touch the code or code
> comments.
>

merged


>
>   Perhaps part of the reason why this function is so complicated is that
>   you support things like this:
>
>     CREATE SCHEMA sch;
>     CREATE TYPE cp AS (a integer, b integer);
>     CREATE VARIABLE sch.v AS cp;
>     LET sch.v = (1, 2);
>     SELECT sch.v.b;
>      b
>     ═══
>      2
>     (1 row)
>
>     test=# SELECT test.sch.v.b;
>      b
>     ═══
>      2
>     (1 row)
>
>   We don't support that for tables:
>
>     CREATE TABLE tab (c cp);
>     INSERT INTO tab VALUES (ROW(42, 43));
>     SELECT tab.c.b FROM tab;
>     ERROR:  cross-database references are not implemented: tab.c.b
>
>   You have to use extra parentheses:
>
>     SELECT (tab.c).b FROM tab;
>      b
>     ════
>      43
>     (1 row)
>
>   I'd say that this should be the same for session variables.
>   What do you think?
>

I prefer the current state, but I don't have a very strong opinion about
it. It can save 115 lines of almost trivial code, but I think the user
experience can be much worse.  Using composite types in tables is not too
common a pattern (and when I read some recommendations for Oracle, it is an
antipattern), but usage of composite variables is common (it can hold a
row). Moreover, we talked long about possible identifier's collisions, and
the pattern schema.variable is very good protection against possible
collisions. Probably the pattern catalog.schema.variable.field is not too
interesting but the support has 50 lines.

More, the plpgsql supports pattern label.variable.field, so can be little
bit unfriendly if session variables doesn't support "similar" pattern

create type t as (x int, y int);
do $$
<<lab1>>declare v t;
begin
  v := (20,20); raise notice '%', lab1.v.x;
end $$;
NOTICE:  20
DO



>   Code comments:
>
>   - There are lots of variables declared at the beginning, but they are
> only
>     used in code blocks further down.  For clarity, you should declare a
>     variable in the code block where it is used.
>

moved


>
>   - +                   /*
>     +                    * In this case, "a" is used as catalog name -
> check it.
>     +                    */
>     +                   if (strcmp(a, get_database_name(MyDatabaseId)) !=
> 0)
>     +                   {
>     +                       if (!noerror)
>     +                           ereport(ERROR,
>     +
>  (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
>     +                                    errmsg("cross-database references
> are not implemented: %s",
>     +                                           NameListToString(names))));
>     +                   }
>     +                   else
>     +                   {
>
>     There needn't be an "else", since the ereport() will never return.
>     Less indentation is better.
>
>
done



> - src/backend/commands/session_variable.c
>
>   There is one comment that confuses me and that I did not edit:
>
>   > +Datum
>   > +GetSessionVariable(Oid varid, bool *isNull, Oid *typid)
>   > +{
>   > +   SVariable   svar;
>   > +
>   > +   svar = get_session_variable(varid);
>   > +
>   > +   /*
>   > +    * Although svar is freshly validated in this point, the
> svar->is_valid can
>   > +    * be false, due possible accepting invalidation message inside
> domain
>   > +    * check. Now, the validation is done after lock, that can also
> accept
>   > +    * invalidation message, so validation should be trustful.
>   > +    *
>   > +    * For now, we don't need to repeat validation. Only svar should
> be valid
>   > +    * pointer.
>   > +    */
>

This comment is related to assertions. Before I had there
`Assert(svar->is_valid)`, because I expected it. But it was not always
true. And although it is true, we don't need to validate a variable,
because at this moment, the variable should be locked, and then we can
return content safely.


>   > +   Assert(svar);
>   > +
>   > +   *typid = svar->typid;
>   > +
>   > +   return copy_session_variable_value(svar, isNull);
>   > +}
>
>


> - src/backend/executor/execMain.c
>
>   > @@ -196,6 +198,51 @@ standard_ExecutorStart(QueryDesc *queryDesc, int
> eflags)
>   >     Assert(queryDesc->sourceText != NULL);
>   >     estate->es_sourceText = queryDesc->sourceText;
>   >
>   > +   /*
>   > +    * The executor doesn't work with session variables directly.
> Values of
>   > +    * related session variables are copied to dedicated array, and
> this array
>   > +    * is passed to executor.
>   > +    */
>
>   Why?  Is that a performance measure?  The comment should explain that.
>

Session variables are volatile internally - some function that uses LET
statements can be called more times inside a query. This behavior is not
consistent with plpgsql's variables or external parameters. And if we want
to support parallel queries, then volatile session variables can be pretty
messy (and unusable). If we want the same behaviour for queries with or
without parallel support, then the session variables should be "stable"
(like stable functions). Simple implementation is using "snapshot" of
values of used session variables when query is started. This "snapshot" is
immutable, so we don't need to make a copy more times.

I changed this comment

<-->/*
<--> * The executor doesn't work with session variables directly. Values of
<--> * related session variables are copied to a dedicated array, and this
array
<--> * is passed to the executor. This array is stable "snapshot" of values
of used
<--> * session variables. There are three benefits of this strategy:
<--> *
<--> * - consistency with external parameters and plpgsql variables,
<--> *
<--> * - session variables can be parallel safe,
<--> *
<--> * - we don't need make fresh copy for any read of session variable
<--> *    (this is necessary because the internally the session variable can
<--> *    be changed inside query execution time, and then a reference to
<--> *    previously returned value can be corrupted).
<--> */



> - parallel safety
>
>   > --- a/src/backend/optimizer/util/clauses.c
>   > +++ b/src/backend/optimizer/util/clauses.c
>   > @@ -935,6 +936,13 @@ max_parallel_hazard_walker(Node *node,
> max_parallel_hazard_context *context)
>   >         if (param->paramkind == PARAM_EXTERN)
>   >             return false;
>   >
>   > +       /* We doesn't support passing session variables to workers */
>   > +       if (param->paramkind == PARAM_VARIABLE)
>   > +       {
>   > +           if (max_parallel_hazard_test(PROPARALLEL_RESTRICTED,
> context))
>   > +               return true;
>   > +       }
>
>   Even if a later patch alleviates that restriction, this patch should
> document that
>   session variables imply "parallel restricted".  I have added that to
> doc/src/sgml/parallel.sgml.
>

merged (and removed in 0015)


>
> Attached are the first two patches with my edits (the first patch is
> unchanged;
> I just add it again to keep the cfbot happy.
>
> I hope to get to review the other patches at some later time.
>
> Yours,
> Laurenz Albe
>

Attached new patchset

Regards

Pavel

  240. Re: proposal: schema variables

    Laurenz Albe <laurenz.albe@cybertec.at> — 2024-08-27T06:15:42Z

    On Thu, 2024-08-15 at 07:55 +0200, Pavel Stehule wrote:
> út 30. 7. 2024 v 21:46 odesílatel Laurenz Albe <laurenz.albe@cybertec.at> napsal:
> > - A general reminder: single line comments should start with a lower case
> >   letter and have to period in the end:
> 
> Should it be "have not to period in the end" ?

I made a typo.  I mean "have *no* period in the end".

> I fixed almost all without parts related to psql and executor - there almost all
> current comments break the mentioned rule. So I use the style used in these files.
> I can fix it (if you like it) - or it can be fixed generally in a separated patch set.

Thanks.  I also noticed that a lot of existing comments break that rule,
so I think that it is fine if you stick with the way that the surrounding
code does it.

> > - Variable as parameter:
> > 
> >   CREATE VARIABLE var AS date;
> >   LET var = current_date;
> >   PREPARE stmt(date) AS SELECT $1;
> >   EXECUTE stmt(var);
> >   ERROR:  paramid of PARAM_VARIABLE param is out of range
> > 
> >   Is that working as intended?  I don't understand the error message.
> 
> fixed in 0002 patch (variables cannot be used as EXECUTE argument) and 0014 (enable usage variables as an argument of EXECUTE)

Thanks.

> >   > --- a/src/backend/catalog/namespace.c
> >   > +++ b/src/backend/catalog/namespace.c
> >   > [...]
> >   > +/*
> >   > + * IdentifyVariable - try to find variable identified by list of names.
> >   > [...]
> > 
> >   Perhaps part of the reason why this function is so complicated is that
> >   you support things like this:
> > 
> >     CREATE SCHEMA sch;
> >     CREATE TYPE cp AS (a integer, b integer);
> >     CREATE VARIABLE sch.v AS cp;
> >     LET sch.v = (1, 2);
> >     SELECT sch.v.b;
> >      b 
> >     ═══
> >      2
> >     (1 row)
> > 
> >     test=# SELECT test.sch.v.b;
> >      b 
> >     ═══
> >      2
> >     (1 row)
> > 
> >   We don't support that for tables:
> > 
> >     CREATE TABLE tab (c cp);
> >     INSERT INTO tab VALUES (ROW(42, 43));
> >     SELECT tab.c.b FROM tab;
> >     ERROR:  cross-database references are not implemented: tab.c.b
> > 
> >   You have to use extra parentheses:
> > 
> >     SELECT (tab.c).b FROM tab;
> >      b  
> >     ════
> >      43
> >     (1 row)
> > 
> >   I'd say that this should be the same for session variables.
> >   What do you think?
> 
> I prefer the current state, but I don't have a very strong opinion about it.
> It can save 115 lines of almost trivial code, but I think the user experience
> can be much worse.  Using composite types in tables is not too common a
> pattern (and when I read some recommendations for Oracle, it is an antipattern),
> but usage of composite variables is common (it can hold a row). Moreover,
> we talked long about possible identifier's collisions, and the pattern
> schema.variable is very good protection against possible collisions.
> Probably the pattern catalog.schema.variable.field is not too interesting
> but the support has 50 lines.
> 
> More, the plpgsql supports pattern label.variable.field, so can be little bit
> unfriendly if session variables doesn't support "similar" pattern

I see your point, and I'm fine with leaving it as it is.

> 
> 
> > - src/backend/commands/session_variable.c
> > 
> >   There is one comment that confuses me and that I did not edit:
> > 
> >   > +Datum
> >   > +GetSessionVariable(Oid varid, bool *isNull, Oid *typid)
> >   > +{
> >   > +   SVariable   svar;
> >   > +
> >   > +   svar = get_session_variable(varid);
> >   > +
> >   > +   /*
> >   > +    * Although svar is freshly validated in this point, the svar->is_valid can
> >   > +    * be false, due possible accepting invalidation message inside domain
> >   > +    * check. Now, the validation is done after lock, that can also accept
> >   > +    * invalidation message, so validation should be trustful.
> >   > +    *
> >   > +    * For now, we don't need to repeat validation. Only svar should be valid
> >   > +    * pointer.
> >   > +    */
> 
> This comment is related to assertions. Before I had there `Assert(svar->is_valid)`,
> because I expected it. But it was not always true. And although it is true,
> we don't need to validate a variable, because at this moment, the variable
> should be locked, and then we can return content safely.

I guess my main problem is the word "trustful".  I don't recognize that word.
Perhaps you can reword the comment along the lines of your above explanation.

> 
> > - src/backend/executor/execMain.c
> > 
> >   > @@ -196,6 +198,51 @@ standard_ExecutorStart(QueryDesc *queryDesc, int eflags)
> >   >     Assert(queryDesc->sourceText != NULL);
> >   >     estate->es_sourceText = queryDesc->sourceText;
> >   >
> >   > +   /*
> >   > +    * The executor doesn't work with session variables directly. Values of
> >   > +    * related session variables are copied to dedicated array, and this array
> >   > +    * is passed to executor.
> >   > +    */
> > 
> >   Why?  Is that a performance measure?  The comment should explain that.
> 
> Session variables are volatile internally - some function that uses LET statements
> can be called more times inside a query. This behavior is not consistent with
> plpgsql's variables or external parameters. And if we want to support parallel
> queries, then volatile session variables can be pretty messy (and unusable).
> If we want the same behaviour for queries with or without parallel support,
> then the session variables should be "stable" (like stable functions).
> Simple implementation is using "snapshot" of values of used session variables
> when query is started. This "snapshot" is immutable, so we don't need to make
> a copy more times.
> 
> I changed this comment

Thanks.

> > - parallel safety
> > 
> >   > --- a/src/backend/optimizer/util/clauses.c
> >   > +++ b/src/backend/optimizer/util/clauses.c
> >   > @@ -935,6 +936,13 @@ max_parallel_hazard_walker(Node *node, max_parallel_hazard_context *context)
> >   >         if (param->paramkind == PARAM_EXTERN)
> >   >             return false;
> >   >
> >   > +       /* We doesn't support passing session variables to workers */
> >   > +       if (param->paramkind == PARAM_VARIABLE)
> >   > +       {
> >   > +           if (max_parallel_hazard_test(PROPARALLEL_RESTRICTED, context))
> >   > +               return true;
> >   > +       }
> > 
> >   Even if a later patch alleviates that restriction, this patch should document that
> >   session variables imply "parallel restricted".  I have added that to doc/src/sgml/parallel.sgml.
> > 
> 
> merged (and removed in 0015)

Thanks.


I hope I can do some more review at some point in the future...

I sincerely hope that this patch set gets merged at some point.
The big obstacle is that it is so large.  That's probably because it is pretty
feature-complete (but, as we have found, not totally free of bugs).

Judging from the amount of time I put into my review so far, I guess that this
patch set would keep a committer busy for several weeks.  Perhaps the only way to
get this done would be to make you a committer...

Yours,
Laurenz Albe

  241. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-08-27T06:52:27Z

    Hi

út 27. 8. 2024 v 8:15 odesílatel Laurenz Albe <laurenz.albe@cybertec.at>
napsal:

> On Thu, 2024-08-15 at 07:55 +0200, Pavel Stehule wrote:
> > út 30. 7. 2024 v 21:46 odesílatel Laurenz Albe <laurenz.albe@cybertec.at>
> napsal:
> > > - A general reminder: single line comments should start with a lower
> case
> > >   letter and have to period in the end:
> >
> > Should it be "have not to period in the end" ?
>
> I made a typo.  I mean "have *no* period in the end".
>
> > I fixed almost all without parts related to psql and executor - there
> almost all
> > current comments break the mentioned rule. So I use the style used in
> these files.
> > I can fix it (if you like it) - or it can be fixed generally in a
> separated patch set.
>
> Thanks.  I also noticed that a lot of existing comments break that rule,
> so I think that it is fine if you stick with the way that the surrounding
> code does it.
>
> > > - Variable as parameter:
> > >
> > >   CREATE VARIABLE var AS date;
> > >   LET var = current_date;
> > >   PREPARE stmt(date) AS SELECT $1;
> > >   EXECUTE stmt(var);
> > >   ERROR:  paramid of PARAM_VARIABLE param is out of range
> > >
> > >   Is that working as intended?  I don't understand the error message.
> >
> > fixed in 0002 patch (variables cannot be used as EXECUTE argument) and
> 0014 (enable usage variables as an argument of EXECUTE)
>
> Thanks.
>
> > >   > --- a/src/backend/catalog/namespace.c
> > >   > +++ b/src/backend/catalog/namespace.c
> > >   > [...]
> > >   > +/*
> > >   > + * IdentifyVariable - try to find variable identified by list of
> names.
> > >   > [...]
> > >
> > >   Perhaps part of the reason why this function is so complicated is
> that
> > >   you support things like this:
> > >
> > >     CREATE SCHEMA sch;
> > >     CREATE TYPE cp AS (a integer, b integer);
> > >     CREATE VARIABLE sch.v AS cp;
> > >     LET sch.v = (1, 2);
> > >     SELECT sch.v.b;
> > >      b
> > >     ═══
> > >      2
> > >     (1 row)
> > >
> > >     test=# SELECT test.sch.v.b;
> > >      b
> > >     ═══
> > >      2
> > >     (1 row)
> > >
> > >   We don't support that for tables:
> > >
> > >     CREATE TABLE tab (c cp);
> > >     INSERT INTO tab VALUES (ROW(42, 43));
> > >     SELECT tab.c.b FROM tab;
> > >     ERROR:  cross-database references are not implemented: tab.c.b
> > >
> > >   You have to use extra parentheses:
> > >
> > >     SELECT (tab.c).b FROM tab;
> > >      b
> > >     ════
> > >      43
> > >     (1 row)
> > >
> > >   I'd say that this should be the same for session variables.
> > >   What do you think?
> >
> > I prefer the current state, but I don't have a very strong opinion about
> it.
> > It can save 115 lines of almost trivial code, but I think the user
> experience
> > can be much worse.  Using composite types in tables is not too common a
> > pattern (and when I read some recommendations for Oracle, it is an
> antipattern),
> > but usage of composite variables is common (it can hold a row). Moreover,
> > we talked long about possible identifier's collisions, and the pattern
> > schema.variable is very good protection against possible collisions.
> > Probably the pattern catalog.schema.variable.field is not too interesting
> > but the support has 50 lines.
> >
> > More, the plpgsql supports pattern label.variable.field, so can be
> little bit
> > unfriendly if session variables doesn't support "similar" pattern
>
> I see your point, and I'm fine with leaving it as it is.
>
> >
> >
> > > - src/backend/commands/session_variable.c
> > >
> > >   There is one comment that confuses me and that I did not edit:
> > >
> > >   > +Datum
> > >   > +GetSessionVariable(Oid varid, bool *isNull, Oid *typid)
> > >   > +{
> > >   > +   SVariable   svar;
> > >   > +
> > >   > +   svar = get_session_variable(varid);
> > >   > +
> > >   > +   /*
> > >   > +    * Although svar is freshly validated in this point, the
> svar->is_valid can
> > >   > +    * be false, due possible accepting invalidation message
> inside domain
> > >   > +    * check. Now, the validation is done after lock, that can
> also accept
> > >   > +    * invalidation message, so validation should be trustful.
> > >   > +    *
> > >   > +    * For now, we don't need to repeat validation. Only svar
> should be valid
> > >   > +    * pointer.
> > >   > +    */
> >
> > This comment is related to assertions. Before I had there
> `Assert(svar->is_valid)`,
> > because I expected it. But it was not always true. And although it is
> true,
> > we don't need to validate a variable, because at this moment, the
> variable
> > should be locked, and then we can return content safely.
>
> I guess my main problem is the word "trustful".  I don't recognize that
> word.
> Perhaps you can reword the comment along the lines of your above
> explanation.
>

I'll try to change it


>
> >
> > > - src/backend/executor/execMain.c
> > >
> > >   > @@ -196,6 +198,51 @@ standard_ExecutorStart(QueryDesc *queryDesc,
> int eflags)
> > >   >     Assert(queryDesc->sourceText != NULL);
> > >   >     estate->es_sourceText = queryDesc->sourceText;
> > >   >
> > >   > +   /*
> > >   > +    * The executor doesn't work with session variables directly.
> Values of
> > >   > +    * related session variables are copied to dedicated array,
> and this array
> > >   > +    * is passed to executor.
> > >   > +    */
> > >
> > >   Why?  Is that a performance measure?  The comment should explain
> that.
> >
> > Session variables are volatile internally - some function that uses LET
> statements
> > can be called more times inside a query. This behavior is not consistent
> with
> > plpgsql's variables or external parameters. And if we want to support
> parallel
> > queries, then volatile session variables can be pretty messy (and
> unusable).
> > If we want the same behaviour for queries with or without parallel
> support,
> > then the session variables should be "stable" (like stable functions).
> > Simple implementation is using "snapshot" of values of used session
> variables
> > when query is started. This "snapshot" is immutable, so we don't need to
> make
> > a copy more times.
> >
> > I changed this comment
>
> Thanks.
>
> > > - parallel safety
> > >
> > >   > --- a/src/backend/optimizer/util/clauses.c
> > >   > +++ b/src/backend/optimizer/util/clauses.c
> > >   > @@ -935,6 +936,13 @@ max_parallel_hazard_walker(Node *node,
> max_parallel_hazard_context *context)
> > >   >         if (param->paramkind == PARAM_EXTERN)
> > >   >             return false;
> > >   >
> > >   > +       /* We doesn't support passing session variables to workers
> */
> > >   > +       if (param->paramkind == PARAM_VARIABLE)
> > >   > +       {
> > >   > +           if (max_parallel_hazard_test(PROPARALLEL_RESTRICTED,
> context))
> > >   > +               return true;
> > >   > +       }
> > >
> > >   Even if a later patch alleviates that restriction, this patch should
> document that
> > >   session variables imply "parallel restricted".  I have added that to
> doc/src/sgml/parallel.sgml.
> > >
> >
> > merged (and removed in 0015)
>
> Thanks.
>
>
> I hope I can do some more review at some point in the future...
>
> I sincerely hope that this patch set gets merged at some point.
> The big obstacle is that it is so large.  That's probably because it is
> pretty
> feature-complete (but, as we have found, not totally free of bugs).
>

Any feature that builds its own system catalog cannot be short. The first
two patches have 300KB, like all others and just basic support for reading
and writing. All other features have 300KB. On second hand almost all code
is simple, and there are no changes in critical parts of Postgres. The size
and complexity of JSONPath is level higher. I can throw 200KB from another
300KB patch set which can be better for review, but it can be harder to
maintain this patch set. I'll try it, and I'll send a reduced version.


> Judging from the amount of time I put into my review so far, I guess that
> this
> patch set would keep a committer busy for several weeks.  Perhaps the only
> way to
> get this done would be to make you a committer...
>

:-)

Unfortunately, I am very bad at languages, I had a lot of problems in basic
school just with Czech lang, Russian and English was more terrible) - so I
very appreciate your work on this patch.

Thank you very much

Pavel


> Yours,
> Laurenz Albe
>

  242. Re: proposal: schema variables

    Laurenz Albe <laurenz.albe@cybertec.at> — 2024-08-27T14:52:00Z

    time""On Tue, 2024-08-27 at 08:52 +0200, Pavel Stehule wrote:
> I can throw 200KB from another 300KB patch set which can be better for review, but it
> can be harder to maintain this patch set. I'll try it, and I'll send a reduced version.

That was not a criticism, and I think the way you split up the patch set right now
is as good as it probably gets.  Ideally, one could say something like "we need at least
patch 1 to 4, 5 and 6 are optional, but desirable, all others can easily be deferred
to a later time".

Yours,
Laurenz Albe

  243. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-08-29T17:33:46Z

    út 27. 8. 2024 v 8:52 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
napsal:

> Hi
>
> út 27. 8. 2024 v 8:15 odesílatel Laurenz Albe <laurenz.albe@cybertec.at>
> napsal:
>
>> On Thu, 2024-08-15 at 07:55 +0200, Pavel Stehule wrote:
>> > út 30. 7. 2024 v 21:46 odesílatel Laurenz Albe <
>> laurenz.albe@cybertec.at> napsal:
>> > > - A general reminder: single line comments should start with a lower
>> case
>> > >   letter and have to period in the end:
>> >
>> > Should it be "have not to period in the end" ?
>>
>> I made a typo.  I mean "have *no* period in the end".
>>
>> > I fixed almost all without parts related to psql and executor - there
>> almost all
>> > current comments break the mentioned rule. So I use the style used in
>> these files.
>> > I can fix it (if you like it) - or it can be fixed generally in a
>> separated patch set.
>>
>> Thanks.  I also noticed that a lot of existing comments break that rule,
>> so I think that it is fine if you stick with the way that the surrounding
>> code does it.
>>
>> > > - Variable as parameter:
>> > >
>> > >   CREATE VARIABLE var AS date;
>> > >   LET var = current_date;
>> > >   PREPARE stmt(date) AS SELECT $1;
>> > >   EXECUTE stmt(var);
>> > >   ERROR:  paramid of PARAM_VARIABLE param is out of range
>> > >
>> > >   Is that working as intended?  I don't understand the error message.
>> >
>> > fixed in 0002 patch (variables cannot be used as EXECUTE argument) and
>> 0014 (enable usage variables as an argument of EXECUTE)
>>
>> Thanks.
>>
>> > >   > --- a/src/backend/catalog/namespace.c
>> > >   > +++ b/src/backend/catalog/namespace.c
>> > >   > [...]
>> > >   > +/*
>> > >   > + * IdentifyVariable - try to find variable identified by list of
>> names.
>> > >   > [...]
>> > >
>> > >   Perhaps part of the reason why this function is so complicated is
>> that
>> > >   you support things like this:
>> > >
>> > >     CREATE SCHEMA sch;
>> > >     CREATE TYPE cp AS (a integer, b integer);
>> > >     CREATE VARIABLE sch.v AS cp;
>> > >     LET sch.v = (1, 2);
>> > >     SELECT sch.v.b;
>> > >      b
>> > >     ═══
>> > >      2
>> > >     (1 row)
>> > >
>> > >     test=# SELECT test.sch.v.b;
>> > >      b
>> > >     ═══
>> > >      2
>> > >     (1 row)
>> > >
>> > >   We don't support that for tables:
>> > >
>> > >     CREATE TABLE tab (c cp);
>> > >     INSERT INTO tab VALUES (ROW(42, 43));
>> > >     SELECT tab.c.b FROM tab;
>> > >     ERROR:  cross-database references are not implemented: tab.c.b
>> > >
>> > >   You have to use extra parentheses:
>> > >
>> > >     SELECT (tab.c).b FROM tab;
>> > >      b
>> > >     ════
>> > >      43
>> > >     (1 row)
>> > >
>> > >   I'd say that this should be the same for session variables.
>> > >   What do you think?
>> >
>> > I prefer the current state, but I don't have a very strong opinion
>> about it.
>> > It can save 115 lines of almost trivial code, but I think the user
>> experience
>> > can be much worse.  Using composite types in tables is not too common a
>> > pattern (and when I read some recommendations for Oracle, it is an
>> antipattern),
>> > but usage of composite variables is common (it can hold a row).
>> Moreover,
>> > we talked long about possible identifier's collisions, and the pattern
>> > schema.variable is very good protection against possible collisions.
>> > Probably the pattern catalog.schema.variable.field is not too
>> interesting
>> > but the support has 50 lines.
>> >
>> > More, the plpgsql supports pattern label.variable.field, so can be
>> little bit
>> > unfriendly if session variables doesn't support "similar" pattern
>>
>> I see your point, and I'm fine with leaving it as it is.
>>
>> >
>> >
>> > > - src/backend/commands/session_variable.c
>> > >
>> > >   There is one comment that confuses me and that I did not edit:
>> > >
>> > >   > +Datum
>> > >   > +GetSessionVariable(Oid varid, bool *isNull, Oid *typid)
>> > >   > +{
>> > >   > +   SVariable   svar;
>> > >   > +
>> > >   > +   svar = get_session_variable(varid);
>> > >   > +
>> > >   > +   /*
>> > >   > +    * Although svar is freshly validated in this point, the
>> svar->is_valid can
>> > >   > +    * be false, due possible accepting invalidation message
>> inside domain
>> > >   > +    * check. Now, the validation is done after lock, that can
>> also accept
>> > >   > +    * invalidation message, so validation should be trustful.
>> > >   > +    *
>> > >   > +    * For now, we don't need to repeat validation. Only svar
>> should be valid
>> > >   > +    * pointer.
>> > >   > +    */
>> >
>> > This comment is related to assertions. Before I had there
>> `Assert(svar->is_valid)`,
>> > because I expected it. But it was not always true. And although it is
>> true,
>> > we don't need to validate a variable, because at this moment, the
>> variable
>> > should be locked, and then we can return content safely.
>>
>> I guess my main problem is the word "trustful".  I don't recognize that
>> word.
>> Perhaps you can reword the comment along the lines of your above
>> explanation.
>>
>
> I'll try to change it
>

is this better

<-->/*
<--> * Although svar is freshly validated in this point, the svar->is_valid
can
<--> * be false, due possible accepting invalidation message inside domain
<--> * check. But now, the variable, and all dependencies are locked, so we
<--> * don't need to repeat validation.
<--> */


>
>
>>
>> >
>> > > - src/backend/executor/execMain.c
>> > >
>> > >   > @@ -196,6 +198,51 @@ standard_ExecutorStart(QueryDesc *queryDesc,
>> int eflags)
>> > >   >     Assert(queryDesc->sourceText != NULL);
>> > >   >     estate->es_sourceText = queryDesc->sourceText;
>> > >   >
>> > >   > +   /*
>> > >   > +    * The executor doesn't work with session variables directly.
>> Values of
>> > >   > +    * related session variables are copied to dedicated array,
>> and this array
>> > >   > +    * is passed to executor.
>> > >   > +    */
>> > >
>> > >   Why?  Is that a performance measure?  The comment should explain
>> that.
>> >
>> > Session variables are volatile internally - some function that uses LET
>> statements
>> > can be called more times inside a query. This behavior is not
>> consistent with
>> > plpgsql's variables or external parameters. And if we want to support
>> parallel
>> > queries, then volatile session variables can be pretty messy (and
>> unusable).
>> > If we want the same behaviour for queries with or without parallel
>> support,
>> > then the session variables should be "stable" (like stable functions).
>> > Simple implementation is using "snapshot" of values of used session
>> variables
>> > when query is started. This "snapshot" is immutable, so we don't need
>> to make
>> > a copy more times.
>> >
>> > I changed this comment
>>
>> Thanks.
>>
>> > > - parallel safety
>> > >
>> > >   > --- a/src/backend/optimizer/util/clauses.c
>> > >   > +++ b/src/backend/optimizer/util/clauses.c
>> > >   > @@ -935,6 +936,13 @@ max_parallel_hazard_walker(Node *node,
>> max_parallel_hazard_context *context)
>> > >   >         if (param->paramkind == PARAM_EXTERN)
>> > >   >             return false;
>> > >   >
>> > >   > +       /* We doesn't support passing session variables to
>> workers */
>> > >   > +       if (param->paramkind == PARAM_VARIABLE)
>> > >   > +       {
>> > >   > +           if (max_parallel_hazard_test(PROPARALLEL_RESTRICTED,
>> context))
>> > >   > +               return true;
>> > >   > +       }
>> > >
>> > >   Even if a later patch alleviates that restriction, this patch
>> should document that
>> > >   session variables imply "parallel restricted".  I have added that
>> to doc/src/sgml/parallel.sgml.
>> > >
>> >
>> > merged (and removed in 0015)
>>
>> Thanks.
>>
>>
>> I hope I can do some more review at some point in the future...
>>
>> I sincerely hope that this patch set gets merged at some point.
>> The big obstacle is that it is so large.  That's probably because it is
>> pretty
>> feature-complete (but, as we have found, not totally free of bugs).
>>
>
> Any feature that builds its own system catalog cannot be short. The first
> two patches have 300KB, like all others and just basic support for reading
> and writing. All other features have 300KB. On second hand almost all code
> is simple, and there are no changes in critical parts of Postgres. The size
> and complexity of JSONPath is level higher. I can throw 200KB from another
> 300KB patch set which can be better for review, but it can be harder to
> maintain this patch set. I'll try it, and I'll send a reduced version.
>
>
>> Judging from the amount of time I put into my review so far, I guess that
>> this
>> patch set would keep a committer busy for several weeks.  Perhaps the
>> only way to
>> get this done would be to make you a committer...
>>
>
> :-)
>
> Unfortunately, I am very bad at languages, I had a lot of problems in
> basic school just with Czech lang, Russian and English was more terrible) -
> so I very appreciate your work on this patch.
>
> Thank you very much
>
> Pavel
>


Regards

Pavel


>
>
>> Yours,
>> Laurenz Albe
>>
>

  244. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-08-29T18:17:06Z

    Hi

út 27. 8. 2024 v 16:52 odesílatel Laurenz Albe <laurenz.albe@cybertec.at>
napsal:

> time""On Tue, 2024-08-27 at 08:52 +0200, Pavel Stehule wrote:
> > I can throw 200KB from another 300KB patch set which can be better for
> review, but it
> > can be harder to maintain this patch set. I'll try it, and I'll send a
> reduced version.
>
> That was not a criticism, and I think the way you split up the patch set
> right now
> is as good as it probably gets.  Ideally, one could say something like "we
> need at least
> patch 1 to 4, 5 and 6 are optional, but desirable, all others can easily
> be deferred
> to a later time".
>

It was mentioned here more times (I thought).

1..4 are fundamental
5..6 optional (6 are just tests)

others can be deferred (5-6 can be deferred too). Without support of
temporary session variables, it  is not too probable to repeatedly CREATE,
DROP the same variable in one session, so memory usage can be similar to
today's workarounds, but against workarounds, session variables use types
and access rights. Note - plpgsql doesn't try to delete compiled code from
cache immediately too - so the behaviour implemented in 1..4 is "similar"
to plpgsql func cache

14 .. allow you to use session variables as arguments of CALL or EXECUTE
statement, and variables can be used in plpgsql simple expr.
15 .. variables don't block parallelism
16 .. the result of plpgsql simple expr can be assigned to session variables
17 .. expr with session variable can be inlined in SQL

14-17 are performance related

7 - was requested by some people - some functionality can be possibly
replaced by plpgsql_check.
It has only 40 rows - it just raise warning or error when some conditions
are true

Regards

Pavel



>
> Yours,
> Laurenz Albe
>

  245. Re: proposal: schema variables

    Laurenz Albe <laurenz.albe@cybertec.at> — 2024-09-02T14:00:51Z

    On Thu, 2024-08-29 at 19:33 +0200, Pavel Stehule wrote:
> > > > >   > +   /*
> > > > >   > +    * Although svar is freshly validated in this point, the svar->is_valid can
> > > > >   > +    * be false, due possible accepting invalidation message inside domain
> > > > >   > +    * check. Now, the validation is done after lock, that can also accept
> > > > >   > +    * invalidation message, so validation should be trustful.
> > > > >   > +    *
> > > > >   > +    * For now, we don't need to repeat validation. Only svar should be valid
> > > > >   > +    * pointer.
> > > > >   > +    */
> > > > 
> > > > This comment is related to assertions. Before I had there `Assert(svar->is_valid)`,
> > > > because I expected it. But it was not always true. And although it is true,
> > > > we don't need to validate a variable, because at this moment, the variable
> > > > should be locked, and then we can return content safely.
> > > 
> > > I guess my main problem is the word "trustful".  I don't recognize that word.
> > > Perhaps you can reword the comment along the lines of your above explanation.
> > 
> > 
> > I'll try to change it
> 
> is this better
> 
> <-->/*
> <--> * Although svar is freshly validated in this point, the svar->is_valid can
> <--> * be false, due possible accepting invalidation message inside domain
> <--> * check. But now, the variable, and all dependencies are locked, so we
> <--> * don't need to repeat validation.
> <--> */

Much better.

Here is an improved version:

  Although "svar" is freshly validated in this point, svar->is_valid can
  be false, if an invalidation message ws processed during the domain check.
  But the variable and all its dependencies are locked now, so we don't need
  to repeat the validation.

Yours,
Laurenz Albe

  246. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-09-03T11:41:31Z

    Hi

po 2. 9. 2024 v 16:00 odesílatel Laurenz Albe <laurenz.albe@cybertec.at>
napsal:

> On Thu, 2024-08-29 at 19:33 +0200, Pavel Stehule wrote:
> > > > > >   > +   /*
> > > > > >   > +    * Although svar is freshly validated in this point, the
> svar->is_valid can
> > > > > >   > +    * be false, due possible accepting invalidation message
> inside domain
> > > > > >   > +    * check. Now, the validation is done after lock, that
> can also accept
> > > > > >   > +    * invalidation message, so validation should be
> trustful.
> > > > > >   > +    *
> > > > > >   > +    * For now, we don't need to repeat validation. Only
> svar should be valid
> > > > > >   > +    * pointer.
> > > > > >   > +    */
> > > > >
> > > > > This comment is related to assertions. Before I had there
> `Assert(svar->is_valid)`,
> > > > > because I expected it. But it was not always true. And although it
> is true,
> > > > > we don't need to validate a variable, because at this moment, the
> variable
> > > > > should be locked, and then we can return content safely.
> > > >
> > > > I guess my main problem is the word "trustful".  I don't recognize
> that word.
> > > > Perhaps you can reword the comment along the lines of your above
> explanation.
> > >
> > >
> > > I'll try to change it
> >
> > is this better
> >
> > <-->/*
> > <--> * Although svar is freshly validated in this point, the
> svar->is_valid can
> > <--> * be false, due possible accepting invalidation message inside
> domain
> > <--> * check. But now, the variable, and all dependencies are locked, so
> we
> > <--> * don't need to repeat validation.
> > <--> */
>
> Much better.
>
> Here is an improved version:
>
>   Although "svar" is freshly validated in this point, svar->is_valid can
>   be false, if an invalidation message ws processed during the domain
> check.
>   But the variable and all its dependencies are locked now, so we don't
> need
>   to repeat the validation.
>
>

merged

thank you

Regards

Pavel


> Yours,
> Laurenz Albe
>

  247. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-09-12T05:15:34Z

    Hi

fresh rebase

regards


Pavel

  248. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-09-18T06:21:44Z

    Hi

fresh rebase

regards

Pavel

  249. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-09-22T08:43:22Z

    Hi

fix assertion failure after
https://github.com/postgres/postgres/commit/24f5205948093a96edf8213294b3d585ac3fe1fb

the analyze routine for LetStmt was fixed

<-->/*
<--> * The query is executed as utility command by nested executor call.
<--> * Assigned queryId is required in this case.
<--> */
<-->if (IsQueryIdEnabled())

Regards

Pavel

  250. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-10-09T04:24:56Z

    Hi

fresh rebase

Regards

Pavel

  251. Re: proposal: schema variables

    Laurenz Albe <laurenz.albe@cybertec.at> — 2024-10-23T04:11:08Z

    I have gone over patch 3 from the set and worked on the comments.

Apart from that, I have modified your patch as follows:

> +/*
> + * pg_session_variables - designed for testing
> + *
> + * This is a function designed for testing and debugging.  It returns the
> + * content of sessionvars as-is, and can therefore display entries about
> + * session variables that were dropped but for which this backend didn't
> + * process the shared invalidations yet.
> + */
> +Datum
> +pg_session_variables(PG_FUNCTION_ARGS)
> +{
> +#define NUM_PG_SESSION_VARIABLES_ATTS 8
> +
> +	elog(DEBUG1, "pg_session_variables start");

I don't think that message is necessary, particularly with DEBUG1.
I have removed this message and the "end" message as well.

> +		while ((svar = (SVariable) hash_seq_search(&status)) != NULL)
> +		{
> +			Datum		values[NUM_PG_SESSION_VARIABLES_ATTS];
> +			bool		nulls[NUM_PG_SESSION_VARIABLES_ATTS];
> +			HeapTuple	tp;
> +			bool		var_is_valid = false;
> +
> +			memset(values, 0, sizeof(values));
> +			memset(nulls, 0, sizeof(nulls));

Instead of explicitly zeroing out the arrays, I have used an empty initializer
in the definition, like

  bool		nulls[NUM_PG_SESSION_VARIABLES_ATTS] = {};

That should have the same effect.
If you don't like that, I have no real problem with your original code.

> +			values[0] = ObjectIdGetDatum(svar->varid);
> +			values[3] = ObjectIdGetDatum(svar->typid);

You are using the type ID without checking if it exists in the catalog.
I think that is a bug.

There is a dependency between the variable and the type, but if a concurrent
session drops both the variable and the data type, the non-existing type ID
would still show up in the function output.
Even worse, the OID could have been reused for a different type since.

I am attaching just patch number 3 and leave you to adapt the patch set,
but I don't think any of the other patches should be affected.

Yours,
Laurenz Albe

  252. Re: proposal: schema variables

    Laurenz Albe <laurenz.albe@cybertec.at> — 2024-10-24T08:29:02Z

    ... and here is a review for patch 4

I didn't change any code, just added the odd article to a comment.

While running the regression tests with "make installcheck", I noticed two problems:

  --- /home/laurenz/postgresql/src/test/regress/expected/session_variables.out    2024-10-24 11:14:06.717663613 +0300
  +++ /home/laurenz/postgresql/src/test/regress/results/session_variables.out 2024-10-24 11:15:37.999286228 +0300
  @@ -30,6 +30,7 @@
   GRANT ALL ON SCHEMA svartest TO regress_variable_owner;
   CREATE VARIABLE svartest.var1 AS int;
   CREATE ROLE regress_variable_reader;
  +ERROR:  role "regress_variable_reader" already exists

I suggest that patch 0001 should drop role "regress_variable_reader" again.

  @@ -107,7 +108,7 @@
   CONTEXT:  SQL function "sqlfx" statement 1
   SELECT plpgsqlfx(20);
   ERROR:  permission denied for session variable var1
  -CONTEXT:  SQL expression "$1 + var1"
  +CONTEXT:  PL/pgSQL expression "$1 + var1"

That looks like bit rot from your commit 4af123ad45.

Yours,
Laurenz Albe

  253. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-10-25T05:21:34Z

    st 23. 10. 2024 v 6:11 odesílatel Laurenz Albe <laurenz.albe@cybertec.at>
napsal:

> I have gone over patch 3 from the set and worked on the comments.
>
> Apart from that, I have modified your patch as follows:
>
> > +/*
> > + * pg_session_variables - designed for testing
> > + *
> > + * This is a function designed for testing and debugging.  It returns
> the
> > + * content of sessionvars as-is, and can therefore display entries about
> > + * session variables that were dropped but for which this backend didn't
> > + * process the shared invalidations yet.
> > + */
> > +Datum
> > +pg_session_variables(PG_FUNCTION_ARGS)
> > +{
> > +#define NUM_PG_SESSION_VARIABLES_ATTS 8
> > +
> > +     elog(DEBUG1, "pg_session_variables start");
>
> I don't think that message is necessary, particularly with DEBUG1.
> I have removed this message and the "end" message as well.
>

removed


>
> > +             while ((svar = (SVariable) hash_seq_search(&status)) !=
> NULL)
> > +             {
> > +                     Datum
>  values[NUM_PG_SESSION_VARIABLES_ATTS];
> > +                     bool
> nulls[NUM_PG_SESSION_VARIABLES_ATTS];
> > +                     HeapTuple       tp;
> > +                     bool            var_is_valid = false;
> > +
> > +                     memset(values, 0, sizeof(values));
> > +                     memset(nulls, 0, sizeof(nulls));
>
> Instead of explicitly zeroing out the arrays, I have used an empty
> initializer
> in the definition, like
>
>   bool          nulls[NUM_PG_SESSION_VARIABLES_ATTS] = {};
>
> That should have the same effect.
> If you don't like that, I have no real problem with your original code.
>

I prefer the original way - minimally it is a common pattern. I didn't find
any usage of `= {} ` in code


> > +                     values[0] = ObjectIdGetDatum(svar->varid);
> > +                     values[3] = ObjectIdGetDatum(svar->typid);
>
> You are using the type ID without checking if it exists in the catalog.
> I think that is a bug.
>

The original idea was using typid as hint identification of deleted
variables. The possibility that this id will not be consistent for the
current catalogue was expected. And it
is a reason why the result type is just Oid and not regtype. Without it,
pg_session_variables shows just empty rows (except oid) for dropped not yet
purged variables.


>
> There is a dependency between the variable and the type, but if a
> concurrent
> session drops both the variable and the data type, the non-existing type ID
> would still show up in the function output.
> Even worse, the OID could have been reused for a different type since.
>

I agree so usage `select typid::regtype, ... from pg_session_variables(.. `

can be dangerous, but it is the reason why we have there typname column.

Showing typid has some information value, but I don't think it is
absolutely necessary. I see some possible changes:

1. no change
2. remove typid column
3. show typid only when variable is valid, and using regtype as output
type, remove typname

What do you prefer?

I'll send the attachment with merging changes for patch 4

Regards

Pavel


>
> I am attaching just patch number 3 and leave you to adapt the patch set,
> but I don't think any of the other patches should be affected.
>

The comments from your patch are merged



>
> Yours,
> Laurenz Albe
>

  254. Re: proposal: schema variables

    Laurenz Albe <laurenz.albe@cybertec.at> — 2024-10-25T07:41:40Z

    On Fri, 2024-10-25 at 07:21 +0200, Pavel Stehule wrote:
> > > +     elog(DEBUG1, "pg_session_variables start");
> > 
> > I don't think that message is necessary, particularly with DEBUG1.
> > I have removed this message and the "end" message as well.
> 
> removed

Thanks.

> > > +                     memset(values, 0, sizeof(values));
> > > +                     memset(nulls, 0, sizeof(nulls));
> > 
> > Instead of explicitly zeroing out the arrays, I have used an empty initializer
> > in the definition, like
> > 
> >   bool          nulls[NUM_PG_SESSION_VARIABLES_ATTS] = {};
> > 
> > That should have the same effect.
> > If you don't like that, I have no real problem with your original code.
> 
> I prefer the original way - minimally it is a common pattern. I didn't find any usage of `= {} ` in code

That's alright by me.


> > > +                     values[0] = ObjectIdGetDatum(svar->varid);
> > > +                     values[3] = ObjectIdGetDatum(svar->typid);
> > 
> > You are using the type ID without checking if it exists in the catalog.
> > I think that is a bug.
> 
> The original idea was using typid as hint identification of deleted variables. The possibility
> that this id will not be consistent for the current catalogue was expected. And it
> is a reason why the result type is just Oid and not regtype. Without it, pg_session_variables
> shows just empty rows (except oid) for dropped not yet purged variables.

I see your point.  It is for testing and debugging only.

> 
> owing typid has some information value, but I don't think it is absolutely necessary. I see some possible changes:
> 
> 1. no change
> 2. remove typid column
> 3. show typid only when variable is valid, and using regtype as output type, remove typname
> 
> What do you prefer?

I'd say leave it as it is.  I agree that it is not dangerous, and if it is intentional that
non-existing type IDs might be displayed, I have no problem with it.

Yours,
Laurenz Albe

  255. Re: proposal: schema variables

    James Pang <jamespang886@gmail.com> — 2024-10-25T07:58:39Z

    Yes, a lot new coming sessions running some "select" and sql
parsing/planning there, including some partition tables in the query. but
there were other sessions DML on these tables at the same time too

Laurenz Albe <laurenz.albe@cybertec.at> 於 2024年7月19日週五 下午7:41寫道：

> On Sat, 2021-04-10 at 08:58 +0200, Pavel Stehule wrote:
> > I am sending a strongly updated patch for schema variables.
> >
> > I rewrote an execution of a LET statement. In the previous
> implementation I hacked
> > STMT_SELECT. Now, I introduced a new statement STMT_LET, and I
> implemented a new
> > executor node SetVariable. Now I think this implementation is much
> cleaner.
> > Implementation with own executor node reduces necessary work on PL side
> - and allows
> > the LET statement to be prepared - what is important from a security
> view.
> >
> > I'll try to write a second implementation based on a cleaner
> implementation like
> > utility command too. I expect so this version will be more simple, but
> utility
> > commands cannot be prepared, and probably, there should be special
> support for
> > any PL. I hope a cleaner implementation can help to move this patch.
> >
> > We can choose one variant in the next step and this variant can be
> finalized.
> >
> > Notes, comments?
>
> Thank you!
>
> I tried to give the patch a spin, but it doesn't apply any more,
> and there are too many conflicts for me to fix manually.
>
> So I had a look at the documentation:
>
> > --- a/doc/src/sgml/advanced.sgml
> > +++ b/doc/src/sgml/advanced.sgml
>
> > +   <para>
> > +    The value of a schema variable is local to the current session.
> Retrieving
> > +    a variable's value returns either a NULL or a default value, unless
> its value
> > +    is set to something else in the current session with a LET command.
> The content
> > +    of a variable is not transactional. This is the same as in regular
> variables
> > +    in PL languages.
> > +   </para>
> > +
> > +   <para>
> > +    Schema variables are retrieved by the <command>SELECT</command> SQL
> command.
> > +    Their value is set with the <command>LET</command> SQL command.
> > +    While schema variables share properties with tables, their value
> cannot be updated
> > +    with an <command>UPDATE</command> command.
>
> "PL languages" -> "procedural languages".  Perhaps a link to the
> "procedural Languages"
> chapter would be a good idea.
> I don't think we should say "regular" variables: are there irregular
> variables?
>
> My feeling is that "SQL statement <command>XY</command>" is better than
> "<command>XY</command> SQL command".
>
> I think the last sentence should go.  The properties they share with
> tables are
> that they live in a schema and can be used with SELECT.
> Also, it is not necessary to mention that they cannot be UPDATEd.  They
> cannot
> be TRUNCATEd or CALLed either, so why mention UPDATE specifically?
>
> > --- a/doc/src/sgml/catalogs.sgml
> > +++ b/doc/src/sgml/catalogs.sgml
>
> > +     <row>
> > +      <entry><structfield>varisnotnull</structfield></entry>
> > +      <entry><type>boolean</type></entry>
> > +      <entry></entry>
> > +      <entry>
> > +       True if the schema variable doesn't allow null value. The
> default value is false.
> > +      </entry>
> > +     </row>
>
> I think the attribute should be called "varnotnull", similar to
> "attnotnull".
> This attribute determines whether the variable is NOT NULL or not, not
> about
> its current setting.
>
> There is a plural missing: "doesn't allow null valueS".
>
> > +     <row>
> > +      <entry><structfield>vareoxaction</structfield></entry>
> > +      <entry><type>char</type></entry>
> > +      <entry></entry>
> > +      <entry>
> > +       <literal>n</literal> = no action, <literal>d</literal> = drop
> the variable,
> > +       <literal>r</literal> = reset the variable to its default value.
> > +      </entry>
> > +     </row>
>
> Perhaps the name "varxactendaction" would be better.
>
> A descriptive sentence is missing.
>
> > --- /dev/null
> > +++ b/doc/src/sgml/ref/create_variable.sgml
>
> > +  <para>
> > +   The value of a schema variable is local to the current session.
> Retrieving
> > +   a variable's value returns either a NULL or a default value, unless
> its value
> > +   is set to something else in the current session with a LET command.
> The content
> > +   of a variable is not transactional. This is the same as in regular
> variables in PL languages.
> > +  </para>
>
> "regular variables in PL languages" -> "variables in procedural languages"
>
> > +  <para>
> > +   Schema variables are retrieved by the <command>SELECT</command> SQL
> command.
> > +   Their value is set with the <command>LET</command> SQL command.
> > +   While schema variables share properties with tables, their value
> cannot be updated
> > +   with an <command>UPDATE</command> command.
> > +  </para>
>
> That's just a literal copy from the tutorial section.  I have the same
> comments
> as there.
>
> > +   <varlistentry>
> > +    <term><literal>NOT NULL</literal></term>
> > +    <listitem>
> > +     <para>
> > +      The <literal>NOT NULL</literal> clause forbids to set the
> variable to
> > +      a null value. A variable created as NOT NULL and without an
> explicitly
> > +      declared default value cannot be read until it is initialized by
> a LET
> > +      command. This obliges the user to explicitly initialize the
> variable
> > +      content before reading it.
> > +     </para>
> > +    </listitem>
> > +   </varlistentry>
>
> What is the reason for that behavior?  I'd have expected that a NOT NULL
> variable needs a default value.
>
> > --- /dev/null
> > +++ b/doc/src/sgml/ref/let.sgml
>
> > +   <varlistentry>
> > +    <term><literal>sql_expression</literal></term>
> > +    <listitem>
> > +     <para>
> > +      An SQL expression. The result is cast into the schema variable's
> type.
> > +     </para>
> > +    </listitem>
> > +   </varlistentry>
>
> Always, even if there is no assignment or implicit cast?
>
> I see no such wording fir INSERT or UPDATE, so if only assignment casts
> are used,
> the second sentence should be removed.
>
> > --- a/doc/src/sgml/ref/pg_restore.sgml
> > +++ b/doc/src/sgml/ref/pg_restore.sgml
>
> > +     <varlistentry>
> > +      <term><option>-A <replaceable
> class="parameter">schema_variable</replaceable></option></term>
> > +      <term><option>--variable=<replaceable
> class="parameter">schema_variable</replaceable></option></term>
> > +      <listitem>
> > +       <para>
> > +        Restore a named schema variable only.  Multiple schema
> variables may be specified with
> > +        multiple <option>-A</option> switches.
> > +       </para>
> > +      </listitem>
> > +     </varlistentry>
>
> Do we need that?  We have no such option for functions and other
> non-relations.
>
> And if we really want such an option for "pg_restore", why not for
> "pg_dump"?
>
> Yours,
> Laurenz Albe
>
>
>

  256. Re: proposal: schema variables

    James Pang <jamespang886@gmail.com> — 2024-10-25T07:59:26Z

    sorry, I sent to wrong email. please ignore.

James Pang <jamespang886@gmail.com> 於 2024年10月25日週五 下午3:58寫道：

> Yes, a lot new coming sessions running some "select" and sql
> parsing/planning there, including some partition tables in the query. but
> there were other sessions DML on these tables at the same time too
>
> Laurenz Albe <laurenz.albe@cybertec.at> 於 2024年7月19日週五 下午7:41寫道：
>
>> On Sat, 2021-04-10 at 08:58 +0200, Pavel Stehule wrote:
>> > I am sending a strongly updated patch for schema variables.
>> >
>> > I rewrote an execution of a LET statement. In the previous
>> implementation I hacked
>> > STMT_SELECT. Now, I introduced a new statement STMT_LET, and I
>> implemented a new
>> > executor node SetVariable. Now I think this implementation is much
>> cleaner.
>> > Implementation with own executor node reduces necessary work on PL side
>> - and allows
>> > the LET statement to be prepared - what is important from a security
>> view.
>> >
>> > I'll try to write a second implementation based on a cleaner
>> implementation like
>> > utility command too. I expect so this version will be more simple, but
>> utility
>> > commands cannot be prepared, and probably, there should be special
>> support for
>> > any PL. I hope a cleaner implementation can help to move this patch.
>> >
>> > We can choose one variant in the next step and this variant can be
>> finalized.
>> >
>> > Notes, comments?
>>
>> Thank you!
>>
>> I tried to give the patch a spin, but it doesn't apply any more,
>> and there are too many conflicts for me to fix manually.
>>
>> So I had a look at the documentation:
>>
>> > --- a/doc/src/sgml/advanced.sgml
>> > +++ b/doc/src/sgml/advanced.sgml
>>
>> > +   <para>
>> > +    The value of a schema variable is local to the current session.
>> Retrieving
>> > +    a variable's value returns either a NULL or a default value,
>> unless its value
>> > +    is set to something else in the current session with a LET
>> command. The content
>> > +    of a variable is not transactional. This is the same as in regular
>> variables
>> > +    in PL languages.
>> > +   </para>
>> > +
>> > +   <para>
>> > +    Schema variables are retrieved by the <command>SELECT</command>
>> SQL command.
>> > +    Their value is set with the <command>LET</command> SQL command.
>> > +    While schema variables share properties with tables, their value
>> cannot be updated
>> > +    with an <command>UPDATE</command> command.
>>
>> "PL languages" -> "procedural languages".  Perhaps a link to the
>> "procedural Languages"
>> chapter would be a good idea.
>> I don't think we should say "regular" variables: are there irregular
>> variables?
>>
>> My feeling is that "SQL statement <command>XY</command>" is better than
>> "<command>XY</command> SQL command".
>>
>> I think the last sentence should go.  The properties they share with
>> tables are
>> that they live in a schema and can be used with SELECT.
>> Also, it is not necessary to mention that they cannot be UPDATEd.  They
>> cannot
>> be TRUNCATEd or CALLed either, so why mention UPDATE specifically?
>>
>> > --- a/doc/src/sgml/catalogs.sgml
>> > +++ b/doc/src/sgml/catalogs.sgml
>>
>> > +     <row>
>> > +      <entry><structfield>varisnotnull</structfield></entry>
>> > +      <entry><type>boolean</type></entry>
>> > +      <entry></entry>
>> > +      <entry>
>> > +       True if the schema variable doesn't allow null value. The
>> default value is false.
>> > +      </entry>
>> > +     </row>
>>
>> I think the attribute should be called "varnotnull", similar to
>> "attnotnull".
>> This attribute determines whether the variable is NOT NULL or not, not
>> about
>> its current setting.
>>
>> There is a plural missing: "doesn't allow null valueS".
>>
>> > +     <row>
>> > +      <entry><structfield>vareoxaction</structfield></entry>
>> > +      <entry><type>char</type></entry>
>> > +      <entry></entry>
>> > +      <entry>
>> > +       <literal>n</literal> = no action, <literal>d</literal> = drop
>> the variable,
>> > +       <literal>r</literal> = reset the variable to its default value.
>> > +      </entry>
>> > +     </row>
>>
>> Perhaps the name "varxactendaction" would be better.
>>
>> A descriptive sentence is missing.
>>
>> > --- /dev/null
>> > +++ b/doc/src/sgml/ref/create_variable.sgml
>>
>> > +  <para>
>> > +   The value of a schema variable is local to the current session.
>> Retrieving
>> > +   a variable's value returns either a NULL or a default value, unless
>> its value
>> > +   is set to something else in the current session with a LET command.
>> The content
>> > +   of a variable is not transactional. This is the same as in regular
>> variables in PL languages.
>> > +  </para>
>>
>> "regular variables in PL languages" -> "variables in procedural languages"
>>
>> > +  <para>
>> > +   Schema variables are retrieved by the <command>SELECT</command> SQL
>> command.
>> > +   Their value is set with the <command>LET</command> SQL command.
>> > +   While schema variables share properties with tables, their value
>> cannot be updated
>> > +   with an <command>UPDATE</command> command.
>> > +  </para>
>>
>> That's just a literal copy from the tutorial section.  I have the same
>> comments
>> as there.
>>
>> > +   <varlistentry>
>> > +    <term><literal>NOT NULL</literal></term>
>> > +    <listitem>
>> > +     <para>
>> > +      The <literal>NOT NULL</literal> clause forbids to set the
>> variable to
>> > +      a null value. A variable created as NOT NULL and without an
>> explicitly
>> > +      declared default value cannot be read until it is initialized by
>> a LET
>> > +      command. This obliges the user to explicitly initialize the
>> variable
>> > +      content before reading it.
>> > +     </para>
>> > +    </listitem>
>> > +   </varlistentry>
>>
>> What is the reason for that behavior?  I'd have expected that a NOT NULL
>> variable needs a default value.
>>
>> > --- /dev/null
>> > +++ b/doc/src/sgml/ref/let.sgml
>>
>> > +   <varlistentry>
>> > +    <term><literal>sql_expression</literal></term>
>> > +    <listitem>
>> > +     <para>
>> > +      An SQL expression. The result is cast into the schema variable's
>> type.
>> > +     </para>
>> > +    </listitem>
>> > +   </varlistentry>
>>
>> Always, even if there is no assignment or implicit cast?
>>
>> I see no such wording fir INSERT or UPDATE, so if only assignment casts
>> are used,
>> the second sentence should be removed.
>>
>> > --- a/doc/src/sgml/ref/pg_restore.sgml
>> > +++ b/doc/src/sgml/ref/pg_restore.sgml
>>
>> > +     <varlistentry>
>> > +      <term><option>-A <replaceable
>> class="parameter">schema_variable</replaceable></option></term>
>> > +      <term><option>--variable=<replaceable
>> class="parameter">schema_variable</replaceable></option></term>
>> > +      <listitem>
>> > +       <para>
>> > +        Restore a named schema variable only.  Multiple schema
>> variables may be specified with
>> > +        multiple <option>-A</option> switches.
>> > +       </para>
>> > +      </listitem>
>> > +     </varlistentry>
>>
>> Do we need that?  We have no such option for functions and other
>> non-relations.
>>
>> And if we really want such an option for "pg_restore", why not for
>> "pg_dump"?
>>
>> Yours,
>> Laurenz Albe
>>
>>
>>

  257. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-10-25T20:38:01Z

    Hi

čt 24. 10. 2024 v 10:29 odesílatel Laurenz Albe <laurenz.albe@cybertec.at>
napsal:

> ... and here is a review for patch 4
>
> I didn't change any code, just added the odd article to a comment.
>
> While running the regression tests with "make installcheck", I noticed two
> problems:
>
>   ---
> /home/laurenz/postgresql/src/test/regress/expected/session_variables.out
> 2024-10-24 11:14:06.717663613 +0300
>   +++
> /home/laurenz/postgresql/src/test/regress/results/session_variables.out
> 2024-10-24 11:15:37.999286228 +0300
>   @@ -30,6 +30,7 @@
>    GRANT ALL ON SCHEMA svartest TO regress_variable_owner;
>    CREATE VARIABLE svartest.var1 AS int;
>    CREATE ROLE regress_variable_reader;
>   +ERROR:  role "regress_variable_reader" already exists
>

> I suggest that patch 0001 should drop role "regress_variable_reader" again.
>

I did it,


>
>   @@ -107,7 +108,7 @@
>    CONTEXT:  SQL function "sqlfx" statement 1
>    SELECT plpgsqlfx(20);
>    ERROR:  permission denied for session variable var1
>   -CONTEXT:  SQL expression "$1 + var1"
>   +CONTEXT:  PL/pgSQL expression "$1 + var1"
>
> That looks like bit rot from your commit 4af123ad45.
>

fixed

merged your changes of comments





>
> Yours,
> Laurenz Albe
>

  258. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-10-28T07:01:26Z

    Hi

fresh rebase

Regards

Pavel

  259. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-10-29T07:16:42Z

    Hi

again, necessary rebase

Regards

Pavel

  260. Re: proposal: schema variables

    Laurenz Albe <laurenz.albe@cybertec.at> — 2024-11-02T05:46:12Z

    On Tue, 2024-10-29 at 08:16 +0100, Pavel Stehule wrote:
> again, necessary rebase

I have started looking at patch 5, and I have some questions and comments.

- The commit message is headed "memory cleaning after DROP VARIABLE", but
  the rest of the commit message speaks of sinval messages.  These two
  things are independent, aren't they?  And both lead to the need to validate
  the variables, right?

  Then this code comment would for example be wrong:

     /* true after accepted sinval message */
     static bool needs_validation = false;

  It also becomes "true" after DROP VARIABLE, right?
  I am happy to fix the comment, but I want to understand the patch first.

- I see that the patch adds cleanup of invalid session variable to each
  COMMIT.  Is that a good idea?  I'd expect that it is good enough to clean
  up whenever session variables are accessed.
  Calling remove_invalid_session_variables() during each COMMIT will affect
  all transactions, and I don't see the benefit.

  Also, do we need to call it during pg_session_variables()?

Yours,
Laurenz Albe

  261. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-11-02T07:36:30Z

    so 2. 11. 2024 v 6:46 odesílatel Laurenz Albe <laurenz.albe@cybertec.at>
napsal:

> On Tue, 2024-10-29 at 08:16 +0100, Pavel Stehule wrote:
> > again, necessary rebase
>
> I have started looking at patch 5, and I have some questions and comments.
>
> - The commit message is headed "memory cleaning after DROP VARIABLE", but
>   the rest of the commit message speaks of sinval messages.  These two
>   things are independent, aren't they?  And both lead to the need to
> validate
>   the variables, right?
>

Maybe it can be formulated differently, but it is true. There are a lot of
sinval messages, but in this case
only sinval messages related to DROP VARIABLE are interesting.


>   Then this code comment would for example be wrong:
>
>      /* true after accepted sinval message */
>      static bool needs_validation = false;
>
>   It also becomes "true" after DROP VARIABLE, right?
>   I am happy to fix the comment, but I want to understand the patch first.
>

sinval message can be raised by any operation over the pg_variable table.

 <-><-->if (hashvalue == 0 || svar->hashvalue == hashvalue)
 <-><-->{
 <-><--><-->svar->is_valid = false;
+<-><--><-->needs_validation = true;
+<-><-->}
+<->}

When I execute DROP VARIABLE, then the hash value is specified, but the
hash can be zero for some massive cleaning, and there are other events that
can send sinval message. I think an ANALYZE does this. So the comment /*
true after accepted sinval message */ is more accurate than /* true after
DROP VARIABLE */.


>
> - I see that the patch adds cleanup of invalid session variable to each
>   COMMIT.  Is that a good idea?  I'd expect that it is good enough to clean
>   up whenever session variables are accessed.
>   Calling remove_invalid_session_variables() during each COMMIT will affect
>   all transactions, and I don't see the benefit.
>

If I remember well, there were two reasons why I did it.

1. Minimize the unwanted surprises for users that will check memory usage -
So if you drop the variables, then the allocated space is released in
possibly near time. The rule - allocated space is released, when in the
next transaction you use any session variable looks a little bit crazy
(although I think so there will not be real significant difference in
functionality). Correct me, if I am wrong, but I don't remember any memory
(or resource) cleaning in Postgres, that is delayed to second transactions.
I agree, there is overhead of cleaning, but this can be very fast when the
user doesn't use session variables, because the hash table with session
variables is not initialized. I can imagine some usage some hooks there as
alternative

2. The main reason why it is implemented is implementation of temporal
variables with RESET or DROP on transaction end. Related code should be
triggered at commit time, it cannot be delayed.


>   Also, do we need to call it during pg_session_variables()?
>

I think it can be removed. Originally pg_session_variables showed only
valid variables, but it is not true now.

Regards

Pavel


>
> Yours,
> Laurenz Albe
>

  262. Re: proposal: schema variables

    Laurenz Albe <laurenz.albe@cybertec.at> — 2024-11-04T09:24:09Z

    On Sat, 2024-11-02 at 08:36 +0100, Pavel Stehule wrote:
> so 2. 11. 2024 v 6:46 odesílatel Laurenz Albe <laurenz.albe@cybertec.at> napsal:
> > - The commit message is headed "memory cleaning after DROP VARIABLE", but
> >   the rest of the commit message speaks of sinval messages.  These two
> >   things are independent, aren't they?  And both lead to the need to validate
> >   the variables, right?
> 
> Maybe it can be formulated differently, but it is true. There are a lot of sinval messages, but in this case
> only sinval messages related to DROP VARIABLE are interesting.

Okay...

> >   Then this code comment would for example be wrong:
> > 
> >      /* true after accepted sinval message */
> >      static bool needs_validation = false;
> > 
> >   It also becomes "true" after DROP VARIABLE, right?
> >   I am happy to fix the comment, but I want to understand the patch first.
> > 
> 
> sinval message can be raised by any operation over the pg_variable table.

I set a watchpoint on "needs_validation", and when I run DROP VARIABLE, it is called
directly from the command:

Hardware watchpoint 2: needs_validation

Old value = false
New value = true
0x00000000006ad44c in SessionVariableDropPostprocess (varid=<optimized out>) at session_variable.c:169
169				svar->drop_lxid = MyProc->vxid.lxid;
(gdb) where
#0  0x00000000006ad44c in SessionVariableDropPostprocess (varid=<optimized out>) at session_variable.c:169
#1  0x0000000000625dec in DropVariableById (varid=<optimized out>) at pg_variable.c:259
#2  0x00000000005fec57 in doDeletion (object=0x4ac85e8, flags=0) at dependency.c:1450
...
#8  0x00000000008bb7e0 in standard_ProcessUtility (pstmt=0x49ac700, queryString=0x49abbb0 "DROP VARIABLE a;", readOnlyTree=<optimized out>, 
    context=PROCESS_UTILITY_TOPLEVEL, params=0x0, queryEnv=0x0, dest=0x49acac0, qc=0x7ffe19d2db60) at utility.c:1076
...
#12 0x00000000008b6742 in exec_simple_query (query_string=0x49abbb0 "DROP VARIABLE a;") at postgres.c:1283

Later, there is a sinval message and "pg_variable_cache_callback()" is called, which sets
"needs_validation" again.

Perhaps my confustion is as follows: if DROP VARIABLE sends an invalidation message,
and the handler sets "needs_validation", why is it necessary to set "needs_validation"
in SessionVariableDropPostprocess() too?

If we didn't set "needs_validation" in SessionVariableDropPostprocess(), the comment
would be true.

> > - I see that the patch adds cleanup of invalid session variable to each
> >   COMMIT.  Is that a good idea?  I'd expect that it is good enough to clean
> >   up whenever session variables are accessed.
> >   Calling remove_invalid_session_variables() during each COMMIT will affect
> >   all transactions, and I don't see the benefit.
> 
> If I remember well, there were two reasons why I did it.
> 
> 1. Minimize the unwanted surprises for users that will check memory usage - So if you
>    drop the variables, then the allocated space is released in possibly near time.
>    The rule - allocated space is released, when in the next transaction you use any
>    session variable looks a little bit crazy (although I think so there will not be real
>    significant difference in functionality).
>    Correct me, if I am wrong, but I don't remember any memory (or resource) cleaning
>    in Postgres, that is delayed to second transactions. I agree, there is overhead of cleaning,
>    but this can be very fast when the user doesn't use session variables, because the hash table
>    with session variables is not initialized. I can imagine some usage some hooks there as alternative

I don't think that is a good enough reason.

Memory used by an idle backend is not totally predictable for other reasons as well:
- the catalog cache
- memory that PostgreSQL freed, but that is kept in the malloc arena so that
  it can be reused for the next malloc() call

I believe that the disadvantage of keeping some memory allocated across transaction
is not as bad as the pain of going through all variables on each COMMIT.
If you have set one or two session variables and run a lot of statements in
autocommit mode, that is an unnecessary overhead.

> 2. The main reason why it is implemented is implementation of temporal variables
>    with RESET or DROP on transaction end. Related code should be triggered at
>    commit time, it cannot be delayed.

That makes sense.

If I remember right, temporary variables are an optional feature.
So I suggest that you move AtPreEOXact_SessionVariables() to the patch that adds
temporary session variables.

> >   Also, do we need to call it during pg_session_variables()?
> 
> I think it can be removed. Originally pg_session_variables showed only valid variables, but it is not true now.

Right.

I'll wait for your reaction to my above suggestions before I start working on the comments.

Yours,
Laurenz Albe

  263. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-11-06T18:24:20Z

    Hi

only fresh rebase

Regards

Pavel

  264. Re: proposal: schema variables

    Dmitry Dolgov <9erthalion6@gmail.com> — 2024-11-10T15:24:19Z

    Hi folks,

Thanks for continuing this work. As a side note, I would like to mention
how strange the situation in this CF item is. If I understand correctly,
there are two hackers threads discussing the same patch, with recent
patches posted in both of them. This is obviously confusing, e.g. the
main concern from another thread, about names clashing, wasn't even
mentioned in this one. Is it possible to reconcile development in one
thread?

  265. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-11-10T16:19:09Z

    ne 10. 11. 2024 v 16:24 odesílatel Dmitry Dolgov <9erthalion6@gmail.com>
napsal:

> Hi folks,
>
> Thanks for continuing this work. As a side note, I would like to mention
> how strange the situation in this CF item is. If I understand correctly,
> there are two hackers threads discussing the same patch, with recent
> patches posted in both of them. This is obviously confusing, e.g. the
> main concern from another thread, about names clashing, wasn't even
> mentioned in this one. Is it possible to reconcile development in one
> thread?
>

This is probably my error. I don't try to organize threads, just I'll try
to reply in the thread where I got a question.

Regards

Pavel

  266. Re: proposal: schema variables

    Dmitry Dolgov <9erthalion6@gmail.com> — 2024-11-10T17:41:07Z

    > On Sun, Nov 10, 2024 at 05:19:09PM GMT, Pavel Stehule wrote:
> ne 10. 11. 2024 v 16:24 odesílatel Dmitry Dolgov <9erthalion6@gmail.com>
> napsal:
>
> > Hi folks,
> >
> > Thanks for continuing this work. As a side note, I would like to mention
> > how strange the situation in this CF item is. If I understand correctly,
> > there are two hackers threads discussing the same patch, with recent
> > patches posted in both of them. This is obviously confusing, e.g. the
> > main concern from another thread, about names clashing, wasn't even
> > mentioned in this one. Is it possible to reconcile development in one
> > thread?
> >
>
> This is probably my error. I don't try to organize threads, just I'll try
> to reply in the thread where I got a question.

It's fine. I just would appreciate some clarity about which patch to
look at. To confirm, the series in this thread contains everything from
the other one, plus some improvements, right?

  267. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-11-10T17:51:40Z

    ne 10. 11. 2024 v 17:19 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
napsal:

>
>
> ne 10. 11. 2024 v 16:24 odesílatel Dmitry Dolgov <9erthalion6@gmail.com>
> napsal:
>
>> Hi folks,
>>
>> Thanks for continuing this work. As a side note, I would like to mention
>> how strange the situation in this CF item is. If I understand correctly,
>> there are two hackers threads discussing the same patch, with recent
>> patches posted in both of them. This is obviously confusing, e.g. the
>> main concern from another thread, about names clashing, wasn't even
>> mentioned in this one. Is it possible to reconcile development in one
>> thread?
>>
>
> This is probably my error. I don't try to organize threads, just I'll try
> to reply in the thread where I got a question.
>

I thought a lot of time about better solutions for identifier collisions
and I really don't think so there is some consistent user friendly syntax.
Personally I think there is an easy already implemented solution -
convention - just use a dedicated schema for variables and this schema
should not be in the search path. Or use secondary convention - like using
prefix "__" for session variables. Common convention is using "_" for
PLpgSQL variables. I searched how this issue is solved in other databases,
or in standard, and I found nothing special. The Oracle and SQL/PSM has a
concept of visibility - the variables are not visible outside packages or
modules, but Postgres has nothing similar. It can be emulated by a
dedicated schema without inserting a search path, but it is less strong.

I think we can introduce an alternative syntax, that will not be user
friendly or readable friendly, but it can be without collisions - or can
decrease possible risks.

It is nothing new - SQL does it with old, "new" syntax of inner joins, or
in Postgres we can

where salary < 40000

or

where pg_catalog.int4lt(salary, 40000);


or some like we use for operators OPERATOR(*schema*.*operatorname*)

So introducing VARIABLE(schema.variablename) syntax as an alternative
syntax for accessing variables I really like. I strongly prefer to use this
as only alternative (secondary) syntax, because I don't think it is
friendly syntax or writing friendly, but it is safe, and I can imagine
tools that can replace generic syntax to this special, or that detects
generic syntax and shows some warning. Then users can choose what they
prefer. Two syntaxes - generic and special can be good enough for all - and
this can be perfectly consistent with current Postgres.

Regards

Pavel


> Regards
>
> Pavel
>
>

  268. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-11-10T18:04:34Z

    ne 10. 11. 2024 v 18:41 odesílatel Dmitry Dolgov <9erthalion6@gmail.com>
napsal:

> > On Sun, Nov 10, 2024 at 05:19:09PM GMT, Pavel Stehule wrote:
> > ne 10. 11. 2024 v 16:24 odesílatel Dmitry Dolgov <9erthalion6@gmail.com>
> > napsal:
> >
> > > Hi folks,
> > >
> > > Thanks for continuing this work. As a side note, I would like to
> mention
> > > how strange the situation in this CF item is. If I understand
> correctly,
> > > there are two hackers threads discussing the same patch, with recent
> > > patches posted in both of them. This is obviously confusing, e.g. the
> > > main concern from another thread, about names clashing, wasn't even
> > > mentioned in this one. Is it possible to reconcile development in one
> > > thread?
> > >
> >
> > This is probably my error. I don't try to organize threads, just I'll try
> > to reply in the thread where I got a question.
>
> It's fine. I just would appreciate some clarity about which patch to
> look at. To confirm, the series in this thread contains everything from
> the other one, plus some improvements, right?
>

I don't remember any change that can be visible for users in this thread.
Laurens does very precious code review (thank you again) - there are bigger
changes in comments, and less changes in code - some parts of code are
moved between patches, some lines were redundant and removed, some lines
were artefacts of some git work and were cleaned. Some redundant tests were
removed. There is no new code.

Regards

Pavel

  269. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-11-10T18:09:04Z

    ne 10. 11. 2024 v 18:51 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
napsal:

>
>
> ne 10. 11. 2024 v 17:19 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
> napsal:
>
>>
>>
>> ne 10. 11. 2024 v 16:24 odesílatel Dmitry Dolgov <9erthalion6@gmail.com>
>> napsal:
>>
>>> Hi folks,
>>>
>>> Thanks for continuing this work. As a side note, I would like to mention
>>> how strange the situation in this CF item is. If I understand correctly,
>>> there are two hackers threads discussing the same patch, with recent
>>> patches posted in both of them. This is obviously confusing, e.g. the
>>> main concern from another thread, about names clashing, wasn't even
>>> mentioned in this one. Is it possible to reconcile development in one
>>> thread?
>>>
>>
>> This is probably my error. I don't try to organize threads, just I'll try
>> to reply in the thread where I got a question.
>>
>
> I thought a lot of time about better solutions for identifier collisions
> and I really don't think so there is some consistent user friendly syntax.
> Personally I think there is an easy already implemented solution -
> convention - just use a dedicated schema for variables and this schema
> should not be in the search path. Or use secondary convention - like using
> prefix "__" for session variables. Common convention is using "_" for
> PLpgSQL variables. I searched how this issue is solved in other databases,
> or in standard, and I found nothing special. The Oracle and SQL/PSM has a
> concept of visibility - the variables are not visible outside packages or
> modules, but Postgres has nothing similar. It can be emulated by a
> dedicated schema without inserting a search path, but it is less strong.
>

There can be more collisions in Oracle, because the functions without
arguments don't need parentheses. Postgres is safer, because this syntax is
not allowed.


>
> I think we can introduce an alternative syntax, that will not be user
> friendly or readable friendly, but it can be without collisions - or can
> decrease possible risks.
>
> It is nothing new - SQL does it with old, "new" syntax of inner joins, or
> in Postgres we can
>
> where salary < 40000
>
> or
>
> where pg_catalog.int4lt(salary, 40000);
>
>
> or some like we use for operators OPERATOR(*schema*.*operatorname*)
>
> So introducing VARIABLE(schema.variablename) syntax as an alternative
> syntax for accessing variables I really like. I strongly prefer to use this
> as only alternative (secondary) syntax, because I don't think it is
> friendly syntax or writing friendly, but it is safe, and I can imagine
> tools that can replace generic syntax to this special, or that detects
> generic syntax and shows some warning. Then users can choose what they
> prefer. Two syntaxes - generic and special can be good enough for all - and
> this can be perfectly consistent with current Postgres.
>
> Regards
>
> Pavel
>
>
>> Regards
>>
>> Pavel
>>
>>
>

  270. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-11-10T22:41:31Z

    Hi

po 4. 11. 2024 v 10:24 odesílatel Laurenz Albe <laurenz.albe@cybertec.at>
napsal:

> On Sat, 2024-11-02 at 08:36 +0100, Pavel Stehule wrote:
> > so 2. 11. 2024 v 6:46 odesílatel Laurenz Albe <laurenz.albe@cybertec.at>
> napsal:
> > > - The commit message is headed "memory cleaning after DROP VARIABLE",
> but
> > >   the rest of the commit message speaks of sinval messages.  These two
> > >   things are independent, aren't they?  And both lead to the need to
> validate
> > >   the variables, right?
> >
> > Maybe it can be formulated differently, but it is true. There are a lot
> of sinval messages, but in this case
> > only sinval messages related to DROP VARIABLE are interesting.
>
> Okay...
>
> > >   Then this code comment would for example be wrong:
> > >
> > >      /* true after accepted sinval message */
> > >      static bool needs_validation = false;
> > >
> > >   It also becomes "true" after DROP VARIABLE, right?
> > >   I am happy to fix the comment, but I want to understand the patch
> first.
> > >
> >
> > sinval message can be raised by any operation over the pg_variable table.
>
> I set a watchpoint on "needs_validation", and when I run DROP VARIABLE, it
> is called
> directly from the command:
>
> Hardware watchpoint 2: needs_validation
>
> Old value = false
> New value = true
> 0x00000000006ad44c in SessionVariableDropPostprocess (varid=<optimized
> out>) at session_variable.c:169
> 169                             svar->drop_lxid = MyProc->vxid.lxid;
> (gdb) where
> #0  0x00000000006ad44c in SessionVariableDropPostprocess (varid=<optimized
> out>) at session_variable.c:169
> #1  0x0000000000625dec in DropVariableById (varid=<optimized out>) at
> pg_variable.c:259
> #2  0x00000000005fec57 in doDeletion (object=0x4ac85e8, flags=0) at
> dependency.c:1450
> ...
> #8  0x00000000008bb7e0 in standard_ProcessUtility (pstmt=0x49ac700,
> queryString=0x49abbb0 "DROP VARIABLE a;", readOnlyTree=<optimized out>,
>     context=PROCESS_UTILITY_TOPLEVEL, params=0x0, queryEnv=0x0,
> dest=0x49acac0, qc=0x7ffe19d2db60) at utility.c:1076
> ...
> #12 0x00000000008b6742 in exec_simple_query (query_string=0x49abbb0 "DROP
> VARIABLE a;") at postgres.c:1283
>
> Later, there is a sinval message and "pg_variable_cache_callback()" is
> called, which sets
> "needs_validation" again.
>
> Perhaps my confustion is as follows: if DROP VARIABLE sends an
> invalidation message,
> and the handler sets "needs_validation", why is it necessary to set
> "needs_validation"
> in SessionVariableDropPostprocess() too?
>
> If we didn't set "needs_validation" in SessionVariableDropPostprocess(),
> the comment
> would be true.
>

I thought about it, and it  is redundant. I removed it. All tests passed
still.



> > > - I see that the patch adds cleanup of invalid session variable to each
> > >   COMMIT.  Is that a good idea?  I'd expect that it is good enough to
> clean
> > >   up whenever session variables are accessed.
> > >   Calling remove_invalid_session_variables() during each COMMIT will
> affect
> > >   all transactions, and I don't see the benefit.
> >
> > If I remember well, there were two reasons why I did it.
> >
> > 1. Minimize the unwanted surprises for users that will check memory
> usage - So if you
> >    drop the variables, then the allocated space is released in possibly
> near time.
> >    The rule - allocated space is released, when in the next transaction
> you use any
> >    session variable looks a little bit crazy (although I think so there
> will not be real
> >    significant difference in functionality).
> >    Correct me, if I am wrong, but I don't remember any memory (or
> resource) cleaning
> >    in Postgres, that is delayed to second transactions. I agree, there
> is overhead of cleaning,
> >    but this can be very fast when the user doesn't use session
> variables, because the hash table
> >    with session variables is not initialized. I can imagine some usage
> some hooks there as alternative
>
> I don't think that is a good enough reason.
>
> Memory used by an idle backend is not totally predictable for other
> reasons as well:
> - the catalog cache
> - memory that PostgreSQL freed, but that is kept in the malloc arena so
> that
>   it can be reused for the next malloc() call
>
> I believe that the disadvantage of keeping some memory allocated across
> transaction
> is not as bad as the pain of going through all variables on each COMMIT.
> If you have set one or two session variables and run a lot of statements in
> autocommit mode, that is an unnecessary overhead.
>
> > 2. The main reason why it is implemented is implementation of temporal
> variables
> >    with RESET or DROP on transaction end. Related code should be
> triggered at
> >    commit time, it cannot be delayed.
>
> That makes sense.
>
> If I remember right, temporary variables are an optional feature.
> So I suggest that you move AtPreEOXact_SessionVariables() to the patch
> that adds
> temporary session variables.
>

moved


>
> > >   Also, do we need to call it during pg_session_variables()?
> >
> > I think it can be removed. Originally pg_session_variables showed only
> valid variables, but it is not true now.
>

removed


>
> Right.
>
> I'll wait for your reaction to my above suggestions before I start working
> on the comments.
>

new patch set attached

Regards

Pavel


>
> Yours,
> Laurenz Albe
>

  271. Re: proposal: schema variables

    Laurenz Albe <laurenz.albe@cybertec.at> — 2024-11-13T14:24:02Z

    Thanks for the updated patch set.

Here is my review of patch 0005:

> --- a/src/backend/access/transam/xact.c
> +++ b/src/backend/access/transam/xact.c
> +#include "commands/session_variable.h"

You probably forgot to move that to the patch for temporary variables.
I did that.

> --- a/src/backend/commands/session_variable.c
> +++ b/src/backend/commands/session_variable.c
> @@ -83,6 +92,19 @@ static HTAB *sessionvars = NULL; /* hash table for session variables */
>  
>  static MemoryContext SVariableMemoryContext = NULL;
>  
> +/* true after accepted sinval message */
> +static bool needs_validation = false;
> +
> +/*
> + * The content of session variables is not removed immediately. When it
> + * is possible we do this at the transaction end. But when the transaction failed,
> + * we cannot do it, because we lost access to the system catalog. So we
> + * try to do it in the next transaction before any get or set of any session
> + * variable. We don't want to repeat this opening cleaning in transaction,
> + * So we store the id of the transaction where opening validation was done.
> + */
> +static LocalTransactionId validated_lxid = InvalidLocalTransactionId;

I have moved the reference to the transaction end to the temporary variable
patch.

I have gone over the comments in patch 0005 and 0006.
I hope I got everything right.  Attached is an updated patch set.

Yours,
Laurenz Albe

  272. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-11-13T15:06:37Z

    st 13. 11. 2024 v 15:24 odesílatel Laurenz Albe <laurenz.albe@cybertec.at>
napsal:

> Thanks for the updated patch set.
>
> Here is my review of patch 0005:
>
> > --- a/src/backend/access/transam/xact.c
> > +++ b/src/backend/access/transam/xact.c
> > +#include "commands/session_variable.h"
>
> You probably forgot to move that to the patch for temporary variables.
> I did that.
>

+1


> > --- a/src/backend/commands/session_variable.c
> > +++ b/src/backend/commands/session_variable.c
> > @@ -83,6 +92,19 @@ static HTAB *sessionvars = NULL; /* hash table for
> session variables */
> >
> >  static MemoryContext SVariableMemoryContext = NULL;
> >
> > +/* true after accepted sinval message */
> > +static bool needs_validation = false;
> > +
> > +/*
> > + * The content of session variables is not removed immediately. When it
> > + * is possible we do this at the transaction end. But when the
> transaction failed,
> > + * we cannot do it, because we lost access to the system catalog. So we
> > + * try to do it in the next transaction before any get or set of any
> session
> > + * variable. We don't want to repeat this opening cleaning in
> transaction,
> > + * So we store the id of the transaction where opening validation was
> done.
> > + */
> > +static LocalTransactionId validated_lxid = InvalidLocalTransactionId;
>
> I have moved the reference to the transaction end to the temporary variable
> patch.
>

+1


> I have gone over the comments in patch 0005 and 0006.
> I hope I got everything right.  Attached is an updated patch set.
>

Thank you

Pavel


>
> Yours,
> Laurenz Albe
>

  273. Re: proposal: schema variables

    Dmitry Dolgov <9erthalion6@gmail.com> — 2024-11-13T16:34:59Z

    > On Sun, Nov 10, 2024 at 06:51:40PM GMT, Pavel Stehule wrote:
> ne 10. 11. 2024 v 17:19 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
> napsal:
> I thought a lot of time about better solutions for identifier collisions
> and I really don't think so there is some consistent user friendly syntax.
> Personally I think there is an easy already implemented solution -
> convention - just use a dedicated schema for variables and this schema
> should not be in the search path. Or use secondary convention - like using
> prefix "__" for session variables. Common convention is using "_" for
> PLpgSQL variables. I searched how this issue is solved in other databases,
> or in standard, and I found nothing special. The Oracle and SQL/PSM has a
> concept of visibility - the variables are not visible outside packages or
> modules, but Postgres has nothing similar. It can be emulated by a
> dedicated schema without inserting a search path, but it is less strong.
>
> I think we can introduce an alternative syntax, that will not be user
> friendly or readable friendly, but it can be without collisions - or can
> decrease possible risks.
>
> It is nothing new - SQL does it with old, "new" syntax of inner joins, or
> in Postgres we can
>
> where salary < 40000
>
> or
>
> where pg_catalog.int4lt(salary, 40000);
>
>
> or some like we use for operators OPERATOR(*schema*.*operatorname*)
>
> So introducing VARIABLE(schema.variablename) syntax as an alternative
> syntax for accessing variables I really like. I strongly prefer to use this
> as only alternative (secondary) syntax, because I don't think it is
> friendly syntax or writing friendly, but it is safe, and I can imagine
> tools that can replace generic syntax to this special, or that detects
> generic syntax and shows some warning. Then users can choose what they
> prefer. Two syntaxes - generic and special can be good enough for all - and
> this can be perfectly consistent with current Postgres.

As far as I recall, last time this topic was discussed in hackers, two
options were proposed: the one with VARIABLE(name), what you mention
here; and another one with adding variables to the FROM clause. The
VARIABLE(...) syntax didn't get much negative feedback, so I guess why
not -- if you find it fitting, it would be interesting to see the
implementation.

I'm afraid it should not be just an alternative syntax, but the only one
allowed, because otherwise I don't see how scenarious like "drop a
column with the same name" could be avoided. As in the previous thread:

    -- we've got a variable b at the same time
    SELECT a, b FROM table1;

Then dropping the column b, but everything still works beause the
variable b got silently picked up. But if it would be required to say
VARIABLE(b), then all fine.

And to make sure we're on the same page, could you post couple of
examples from curretly existing tests in the patch, how are they going
to look like with this proposal?

About adding variables to the FROM clause. Looks like this option was
quite popular, and you've mentioned some technical challenges
implementing that. If you'd like to go with another approach, it would
be great to elaborate on that -- maybe even with a PoC, to make a
convincing point here.

  274. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-11-13T18:18:40Z

    st 13. 11. 2024 v 17:35 odesílatel Dmitry Dolgov <9erthalion6@gmail.com>
napsal:

> > On Sun, Nov 10, 2024 at 06:51:40PM GMT, Pavel Stehule wrote:
> > ne 10. 11. 2024 v 17:19 odesílatel Pavel Stehule <
> pavel.stehule@gmail.com>
> > napsal:
> > I thought a lot of time about better solutions for identifier collisions
> > and I really don't think so there is some consistent user friendly
> syntax.
> > Personally I think there is an easy already implemented solution -
> > convention - just use a dedicated schema for variables and this schema
> > should not be in the search path. Or use secondary convention - like
> using
> > prefix "__" for session variables. Common convention is using "_" for
> > PLpgSQL variables. I searched how this issue is solved in other
> databases,
> > or in standard, and I found nothing special. The Oracle and SQL/PSM has a
> > concept of visibility - the variables are not visible outside packages or
> > modules, but Postgres has nothing similar. It can be emulated by a
> > dedicated schema without inserting a search path, but it is less strong.
> >
> > I think we can introduce an alternative syntax, that will not be user
> > friendly or readable friendly, but it can be without collisions - or can
> > decrease possible risks.
> >
> > It is nothing new - SQL does it with old, "new" syntax of inner joins, or
> > in Postgres we can
> >
> > where salary < 40000
> >
> > or
> >
> > where pg_catalog.int4lt(salary, 40000);
> >
> >
> > or some like we use for operators OPERATOR(*schema*.*operatorname*)
> >
> > So introducing VARIABLE(schema.variablename) syntax as an alternative
> > syntax for accessing variables I really like. I strongly prefer to use
> this
> > as only alternative (secondary) syntax, because I don't think it is
> > friendly syntax or writing friendly, but it is safe, and I can imagine
> > tools that can replace generic syntax to this special, or that detects
> > generic syntax and shows some warning. Then users can choose what they
> > prefer. Two syntaxes - generic and special can be good enough for all -
> and
> > this can be perfectly consistent with current Postgres.
>
> As far as I recall, last time this topic was discussed in hackers, two
> options were proposed: the one with VARIABLE(name), what you mention
> here; and another one with adding variables to the FROM clause. The
> VARIABLE(...) syntax didn't get much negative feedback, so I guess why
> not -- if you find it fitting, it would be interesting to see the
> implementation.
>
> I'm afraid it should not be just an alternative syntax, but the only one
> allowed, because otherwise I don't see how scenarious like "drop a
> column with the same name" could be avoided. As in the previous thread:
>
>     -- we've got a variable b at the same time
>     SELECT a, b FROM table1;
>

I am sorry, but I am in very strong opposition  against this idea. Nobody
did reply to my questions, that can change my opinion.

1. This introduces possible inconsistency between LET syntax and SELECT
syntax.
What will be the syntax of LET?

LET var = var FROM var

PLpgSQL does something, and it is really strange, and source of some
unwanted bugs. See https://commitfest.postgresql.org/50/5044/

With current design I can support

LET var = expr with wars

or

LET var = (QUERY with vars)

It is perfectly consistent. The expressions will be expressions.

2. I don't know of any product in the world that introduced the same
requirement. So this syntax will be proprietary (SQL/PSM it really doesn't
require) and shock for any users that know other databases. Proprietary
syntax in this area far from syntaxes of other products is hell. Try to
explain to users the working with OUT variables of Postgres's procedures
and functions. And there is some deeper logic.

3. There is a simple solution - convention. Use your own schema like vars,
and use session variables in this schema, When this schema will not be on
the search path, then there is not a collision.
Variables living in schema. Nobody without CREATE right can create it. So
it is safe. Or use prefix in __ for variables in the search path.

4. this requirement introduces syntax inconsistency between plpgsql
variables and session variables - which breaks one goal of the patch -
introducing some global variables for plpgsql (and for all PL).

5. Using more variables and FROM clauses decrease readability of FROM clause

SELECT v1, v2, a, b, c FROM t1, v1, v2, t2, ...

6. Usually composite variables don't want to unpack. When you should use
FROM clause, then composite variables will be unpacked. Then all fields can
be possibly in collision with all other column name

Example

CREATE TYPE t1 AS (id int, name varchar)
CREATE TABLE tab(id int, name varchar)
CREATE VARIABLE var1 AS t1;

SELECT id, name, foo(var1) FROM tab, var1;

Now I have a collision in columns id, name, and everywhere I need to use
aliases. Without necessity to use var in FROM clause, I can just write

SELECT id, name, foo(var) FROM tab

and there is not any risk of collision




> Then dropping the column b, but everything still works beause the
> variable b got silently picked up. But if it would be required to say
> VARIABLE(b), then all fine.
>

but same risk you have any time in plpgsql - all time. I don't remember any
bug report related to this issue.


>
> And to make sure we're on the same page, could you post couple of
> examples from curretly existing tests in the patch, how are they going
> to look like with this proposal?
>
> About adding variables to the FROM clause. Looks like this option was
> quite popular, and you've mentioned some technical challenges
> implementing that. If you'd like to go with another approach, it would
> be great to elaborate on that -- maybe even with a PoC, to make a
> convincing point here.
>

There is not any problem with implementation. I see the main problem with
usability, and I really don't want to implement some like LET var = var
FROM var; I am sorry
It fixes one issue, but it increases possible collisions - so the variables
will be unusable.

Regards

Pavel

  275. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-11-14T07:41:26Z

    st 13. 11. 2024 v 17:35 odesílatel Dmitry Dolgov <9erthalion6@gmail.com>
napsal:

> > On Sun, Nov 10, 2024 at 06:51:40PM GMT, Pavel Stehule wrote:
> > ne 10. 11. 2024 v 17:19 odesílatel Pavel Stehule <
> pavel.stehule@gmail.com>
> > napsal:
> > I thought a lot of time about better solutions for identifier collisions
> > and I really don't think so there is some consistent user friendly
> syntax.
> > Personally I think there is an easy already implemented solution -
> > convention - just use a dedicated schema for variables and this schema
> > should not be in the search path. Or use secondary convention - like
> using
> > prefix "__" for session variables. Common convention is using "_" for
> > PLpgSQL variables. I searched how this issue is solved in other
> databases,
> > or in standard, and I found nothing special. The Oracle and SQL/PSM has a
> > concept of visibility - the variables are not visible outside packages or
> > modules, but Postgres has nothing similar. It can be emulated by a
> > dedicated schema without inserting a search path, but it is less strong.
> >
> > I think we can introduce an alternative syntax, that will not be user
> > friendly or readable friendly, but it can be without collisions - or can
> > decrease possible risks.
> >
> > It is nothing new - SQL does it with old, "new" syntax of inner joins, or
> > in Postgres we can
> >
> > where salary < 40000
> >
> > or
> >
> > where pg_catalog.int4lt(salary, 40000);
> >
> >
> > or some like we use for operators OPERATOR(*schema*.*operatorname*)
> >
> > So introducing VARIABLE(schema.variablename) syntax as an alternative
> > syntax for accessing variables I really like. I strongly prefer to use
> this
> > as only alternative (secondary) syntax, because I don't think it is
> > friendly syntax or writing friendly, but it is safe, and I can imagine
> > tools that can replace generic syntax to this special, or that detects
> > generic syntax and shows some warning. Then users can choose what they
> > prefer. Two syntaxes - generic and special can be good enough for all -
> and
> > this can be perfectly consistent with current Postgres.
>
> As far as I recall, last time this topic was discussed in hackers, two
> options were proposed: the one with VARIABLE(name), what you mention
> here; and another one with adding variables to the FROM clause. The
> VARIABLE(...) syntax didn't get much negative feedback, so I guess why
> not -- if you find it fitting, it would be interesting to see the
> implementation.
>
> I'm afraid it should not be just an alternative syntax, but the only one
> allowed, because otherwise I don't see how scenarious like "drop a
> column with the same name" could be avoided. As in the previous thread:
>
>     -- we've got a variable b at the same time
>     SELECT a, b FROM table1;
>
> Then dropping the column b, but everything still works beause the
> variable b got silently picked up. But if it would be required to say
> VARIABLE(b), then all fine.
>

In this scenario you will get  a warning related to variable shadowing
(before you drop a column).

I think this issue can be partially similar to creating two equally named
tables in different schemas (both schemas are in search path). When you
drop one table, the query will work, but the result is different. It is the
same issue. The SQL has no concept of shadowing and on the base line it is
not necessary. But when you integrate SQL with some procedural code then
you should solve this issue (or accept). This issue is real, and it is in
every procedural enhancement of SQL that I know with the same syntax.  On
the other hand I doubt this is a real issue. The changes of system
catalogue are tested before production - so probably you will read a
warning about a shadowed variable, and probably you will get different
results, because variable b has the same value for all rows, and probably
will have different value than column b. I can imagine the necessity of
disabling this warning on production systems. Shadowing by self is not an
issue, probably, but it is a signal of code quality problems.

But this scenario is real, and then it is a question if the warning about
shadowed variables should be only optional and if it can be disabled. Maybe
not. Generally the shadowing is a strange concept - it is safeguard against
serious issues, but it should not be used generally and everywhere the
developer should rename the conflict identifiers.

Regards

Pavel


> And to make sure we're on the same page, could you post couple of
> examples from curretly existing tests in the patch, how are they going
> to look like with this proposal?
>
> About adding variables to the FROM clause. Looks like this option was
> quite popular, and you've mentioned some technical challenges
> implementing that. If you'd like to go with another approach, it would
> be great to elaborate on that -- maybe even with a PoC, to make a
> convincing point here.
>

  276. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-11-15T04:45:58Z

    čt 14. 11. 2024 v 8:41 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
napsal:

>
>
> st 13. 11. 2024 v 17:35 odesílatel Dmitry Dolgov <9erthalion6@gmail.com>
> napsal:
>
>> > On Sun, Nov 10, 2024 at 06:51:40PM GMT, Pavel Stehule wrote:
>> > ne 10. 11. 2024 v 17:19 odesílatel Pavel Stehule <
>> pavel.stehule@gmail.com>
>> > napsal:
>> > I thought a lot of time about better solutions for identifier collisions
>> > and I really don't think so there is some consistent user friendly
>> syntax.
>> > Personally I think there is an easy already implemented solution -
>> > convention - just use a dedicated schema for variables and this schema
>> > should not be in the search path. Or use secondary convention - like
>> using
>> > prefix "__" for session variables. Common convention is using "_" for
>> > PLpgSQL variables. I searched how this issue is solved in other
>> databases,
>> > or in standard, and I found nothing special. The Oracle and SQL/PSM has
>> a
>> > concept of visibility - the variables are not visible outside packages
>> or
>> > modules, but Postgres has nothing similar. It can be emulated by a
>> > dedicated schema without inserting a search path, but it is less strong.
>> >
>> > I think we can introduce an alternative syntax, that will not be user
>> > friendly or readable friendly, but it can be without collisions - or can
>> > decrease possible risks.
>> >
>> > It is nothing new - SQL does it with old, "new" syntax of inner joins,
>> or
>> > in Postgres we can
>> >
>> > where salary < 40000
>> >
>> > or
>> >
>> > where pg_catalog.int4lt(salary, 40000);
>> >
>> >
>> > or some like we use for operators OPERATOR(*schema*.*operatorname*)
>> >
>> > So introducing VARIABLE(schema.variablename) syntax as an alternative
>> > syntax for accessing variables I really like. I strongly prefer to use
>> this
>> > as only alternative (secondary) syntax, because I don't think it is
>> > friendly syntax or writing friendly, but it is safe, and I can imagine
>> > tools that can replace generic syntax to this special, or that detects
>> > generic syntax and shows some warning. Then users can choose what they
>> > prefer. Two syntaxes - generic and special can be good enough for all -
>> and
>> > this can be perfectly consistent with current Postgres.
>>
>> As far as I recall, last time this topic was discussed in hackers, two
>> options were proposed: the one with VARIABLE(name), what you mention
>> here; and another one with adding variables to the FROM clause. The
>> VARIABLE(...) syntax didn't get much negative feedback, so I guess why
>> not -- if you find it fitting, it would be interesting to see the
>> implementation.
>>
>> I'm afraid it should not be just an alternative syntax, but the only one
>> allowed, because otherwise I don't see how scenarious like "drop a
>> column with the same name" could be avoided. As in the previous thread:
>>
>>     -- we've got a variable b at the same time
>>     SELECT a, b FROM table1;
>>
>> Then dropping the column b, but everything still works beause the
>> variable b got silently picked up. But if it would be required to say
>> VARIABLE(b), then all fine.
>>
>
> In this scenario you will get  a warning related to variable shadowing
> (before you drop a column).
>
> I think this issue can be partially similar to creating two equally named
> tables in different schemas (both schemas are in search path). When you
> drop one table, the query will work, but the result is different. It is the
> same issue. The SQL has no concept of shadowing and on the base line it is
> not necessary. But when you integrate SQL with some procedural code then
> you should solve this issue (or accept). This issue is real, and it is in
> every procedural enhancement of SQL that I know with the same syntax.  On
> the other hand I doubt this is a real issue. The changes of system
> catalogue are tested before production - so probably you will read a
> warning about a shadowed variable, and probably you will get different
> results, because variable b has the same value for all rows, and probably
> will have different value than column b. I can imagine the necessity of
> disabling this warning on production systems. Shadowing by self is not an
> issue, probably, but it is a signal of code quality problems.
>
> But this scenario is real, and then it is a question if the warning about
> shadowed variables should be only optional and if it can be disabled. Maybe
> not. Generally the shadowing is a strange concept - it is safeguard against
> serious issues, but it should not be used generally and everywhere the
> developer should rename the conflict identifiers.
>

There can be another example against usage of the FROM clause for
variables. Because it solves just one special case, but others are not
covered.

Theoretically, variables can have the same names as tables. The table
overshadows the variable, so it can work. But when somebody drops the
variable, then the query still can work. So requirement of usage variable
in FROM clause protects us just against drop column, but not against
dropping table. In Postgres the dropping table is possibly risky due
search_path (that introduces shadowing concept) without introduction
variables. There is a possibility of this issue, but how common is this
issue?

Regards

Pavel



>
> Regards
>
> Pavel
>
>
>> And to make sure we're on the same page, could you post couple of
>> examples from curretly existing tests in the patch, how are they going
>> to look like with this proposal?
>>
>> About adding variables to the FROM clause. Looks like this option was
>> quite popular, and you've mentioned some technical challenges
>> implementing that. If you'd like to go with another approach, it would
>> be great to elaborate on that -- maybe even with a PoC, to make a
>> convincing point here.
>>
>

  277. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-11-16T06:10:31Z

    st 13. 11. 2024 v 17:35 odesílatel Dmitry Dolgov <9erthalion6@gmail.com>
napsal:

> > On Sun, Nov 10, 2024 at 06:51:40PM GMT, Pavel Stehule wrote:
> > ne 10. 11. 2024 v 17:19 odesílatel Pavel Stehule <
> pavel.stehule@gmail.com>
> > napsal:
> > I thought a lot of time about better solutions for identifier collisions
> > and I really don't think so there is some consistent user friendly
> syntax.
> > Personally I think there is an easy already implemented solution -
> > convention - just use a dedicated schema for variables and this schema
> > should not be in the search path. Or use secondary convention - like
> using
> > prefix "__" for session variables. Common convention is using "_" for
> > PLpgSQL variables. I searched how this issue is solved in other
> databases,
> > or in standard, and I found nothing special. The Oracle and SQL/PSM has a
> > concept of visibility - the variables are not visible outside packages or
> > modules, but Postgres has nothing similar. It can be emulated by a
> > dedicated schema without inserting a search path, but it is less strong.
> >
> > I think we can introduce an alternative syntax, that will not be user
> > friendly or readable friendly, but it can be without collisions - or can
> > decrease possible risks.
> >
> > It is nothing new - SQL does it with old, "new" syntax of inner joins, or
> > in Postgres we can
> >
> > where salary < 40000
> >
> > or
> >
> > where pg_catalog.int4lt(salary, 40000);
> >
> >
> > or some like we use for operators OPERATOR(*schema*.*operatorname*)
> >
> > So introducing VARIABLE(schema.variablename) syntax as an alternative
> > syntax for accessing variables I really like. I strongly prefer to use
> this
> > as only alternative (secondary) syntax, because I don't think it is
> > friendly syntax or writing friendly, but it is safe, and I can imagine
> > tools that can replace generic syntax to this special, or that detects
> > generic syntax and shows some warning. Then users can choose what they
> > prefer. Two syntaxes - generic and special can be good enough for all -
> and
> > this can be perfectly consistent with current Postgres.
>
> As far as I recall, last time this topic was discussed in hackers, two
> options were proposed: the one with VARIABLE(name), what you mention
> here; and another one with adding variables to the FROM clause. The
> VARIABLE(...) syntax didn't get much negative feedback, so I guess why
> not -- if you find it fitting, it would be interesting to see the
> implementation.
>
> I'm afraid it should not be just an alternative syntax, but the only one
> allowed, because otherwise I don't see how scenarious like "drop a
> column with the same name" could be avoided. As in the previous thread:
>
>     -- we've got a variable b at the same time
>     SELECT a, b FROM table1;
>
> Then dropping the column b, but everything still works beause the
> variable b got silently picked up. But if it would be required to say
> VARIABLE(b), then all fine.
>
> And to make sure we're on the same page, could you post couple of
> examples from curretly existing tests in the patch, how are they going
> to look like with this proposal?
>

What do you think about the following design?  I can implement a warning
"variable_usage_guard" when the variable is accessed without using
VARIABLE() syntax. We can discuss later if this warning can be enabled by
default or not. There I am open to any variant.

So for variable public.a and table public.foo(a, b)

I can write

LET a = 10; -- there is not possible collision
LET a = a + 1; -- there is not possible collision, no warning

SELECT a, b FROM foo; -- there is a collision - and warning "variable a is
shadowed"
SELECT VARIABLE(a), b FROM foo; -- no collision, no warning

After ALTER TABLE foo DROP COLUMN a;

SELECT a, b FROM foo; -- possible warning "the usage in variable without
safe syntax",
SELECT VARIABLE(a), b FROM foo; -- no warning

I think this design can be good for all. variable_usage_guard can be
enabled by default. If somebody uses conventions for collision protection,
then he can safely disable it.

Comments, notes?

Regards

Pavel


> About adding variables to the FROM clause. Looks like this option was
> quite popular, and you've mentioned some technical challenges
> implementing that. If you'd like to go with another approach, it would
> be great to elaborate on that -- maybe even with a PoC, to make a
> convincing point here.
>

  278. Re: proposal: schema variables

    Dmitry Dolgov <9erthalion6@gmail.com> — 2024-11-16T14:27:30Z

    > On Sat, Nov 16, 2024 at 07:10:31AM GMT, Pavel Stehule wrote:

Sorry, got distracted. Let me try to answer step by step.

> > As far as I recall, last time this topic was discussed in hackers, two
> > options were proposed: the one with VARIABLE(name), what you mention
> > here; and another one with adding variables to the FROM clause. The
> > VARIABLE(...) syntax didn't get much negative feedback, so I guess why
> > not -- if you find it fitting, it would be interesting to see the
> > implementation.
> >
> > I'm afraid it should not be just an alternative syntax, but the only one
> > allowed, because otherwise I don't see how scenarious like "drop a
> > column with the same name" could be avoided. As in the previous thread:
> >
> >     -- we've got a variable b at the same time
> >     SELECT a, b FROM table1;
> >
>
> I am sorry, but I am in very strong opposition  against this idea. Nobody
> did reply to my questions, that can change my opinion.

From your reply it's not quite clear, are you opposed to have a mandatory
VARIABLE syntax, or having variables in the FROM clause? My main proposal was
about the former, but the points that are following seems to talk about the
latter. I think it's fine to reject the idea about the FROM clause, as long as
you got some reasonable arguments.

> > Then dropping the column b, but everything still works beause the
> > variable b got silently picked up. But if it would be required to say
> > VARIABLE(b), then all fine.
>
> but same risk you have any time in plpgsql - all time. I don't remember any
> bug report related to this issue.

Which exactly scenario about plpgsql do you have in mind? Just have tried to
declare a variable inside a plpgsql function with the same name as a table
column, and got an error about an ambiguous reference.

> Theoretically, variables can have the same names as tables. The table
> overshadows the variable, so it can work. But when somebody drops the
> variable, then the query still can work. So requirement of usage variable
> in FROM clause protects us just against drop column, but not against
> dropping table. In Postgres the dropping table is possibly risky due
> search_path (that introduces shadowing concept) without introduction
> variables. There is a possibility of this issue, but how common is this
> issue?

This sounds to me like an argument against allowing name clashing between
variables and tables. It makes even more sense, since session variables are in
many ways similar to tables.

> I think this issue can be partially similar to creating two equally named
> tables in different schemas (both schemas are in search path). When you
> drop one table, the query will work, but the result is different. It is the
> same issue. The SQL has no concept of shadowing and on the base line it is
> not necessary.

The point is that most of users are aware about schemas and search path
dangers. But to me such a precedent is not an excuse to introduce a new feature
with similar traps, which folks would have to learn by making mistakes. Judging
from the feedback to this patch over time, I've got an impression that lots of
people are also not fans of that.

> > Then dropping the column b, but everything still works beause the
> > variable b got silently picked up. But if it would be required to say
> > VARIABLE(b), then all fine.
> >
>
> In this scenario you will get  a warning related to variable shadowing
> (before you drop a column).
>
> [...]
>
> What do you think about the following design?  I can implement a warning
> "variable_usage_guard" when the variable is accessed without using
> VARIABLE() syntax. We can discuss later if this warning can be enabled by
> default or not. There I am open to any variant.

I don't follow what are you winning by that? In the context of problem above
(i.e. dropping a column), such a warning is functionally equivalend to a
warning about shadowing.

The problem is that it doesn't sound very appealing to have a feature, which
requires some extra efforts to be used in a right way (e.g. put everything into
a special vars schema, or keep an eye on logs). Most certainly there are such
bits in PostgreSQL today, with all the best practices, crowd wisdom, etc. But
the bar for new features in this sense is much higher, you can see it from the
feedback to this patch. Thus I believe it makes sense, from purely tactical
reasons, to not try to convince half of the community to lower the bar, but
instead try to modify the feature to make it more acceptable, even if some
parts you might not like.

Btw, could you repeat, what was exactly your argument against mandatory
VARIABLE() syntax? It's somehow scattered across many replies, would be great
to summarize it in a couple of phrases.

> Shadowing by self is not an issue, probably, but it is a signal of code
> quality problems.

Agree, but I'm afraid code quality of an average application using PostgreSQL
is quite low, so here we are.

As a side note, I've recently caught myself thinking "it would be cool to have
session variables here". The use case was preparing a policy for RLS, based on
some session-level data set by an application. This session-level data is of a
composite data type, so simple current_setting is cumbersome to use, and a
temporary table will be dropped at the end, taking the policy with it due to
the recorded dependency between them. Thus a session variable of some composite
type sounds like a good fit.

  279. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-11-16T14:34:58Z

    Hi


> As a side note, I've recently caught myself thinking "it would be cool to
> have
> session variables here". The use case was preparing a policy for RLS,
> based on
> some session-level data set by an application. This session-level data is
> of a
> composite data type, so simple current_setting is cumbersome to use, and a
> temporary table will be dropped at the end, taking the policy with it due
> to
> the recorded dependency between them. Thus a session variable of some
> composite
> type sounds like a good fit.
>

yes, RLS support is one mentioned use case, and strong reason the access
rights are implemented there.

Regards

Pavel

  280. Re: proposal: schema variables

    Wolfgang Walther <walther@technowledgy.de> — 2024-11-16T14:56:53Z

    Dmitry Dolgov:
> This sounds to me like an argument against allowing name clashing between
> variables and tables. It makes even more sense, since session variables are in
> many ways similar to tables.

+1

My mental model of a session variable is similar to a single-row, 
optionally global temporary, table.

Is there any substantial difference that I am not aware of?

Best,

Wolfgang

  281. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-11-16T15:36:19Z

    so 16. 11. 2024 v 15:56 odesílatel Wolfgang Walther <walther@technowledgy.de>
napsal:

> Dmitry Dolgov:
> > This sounds to me like an argument against allowing name clashing between
> > variables and tables. It makes even more sense, since session variables
> are in
> > many ways similar to tables.
>
> +1
>

It doesn't help too much, because the unique tuple (schema, name), and
there is a search path.

Secondly, the pg_class is not good enough for description of scalar
variables, and enhancing pg_class for scalar variables can be messy.



>
> My mental model of a session variable is similar to a single-row,
> optionally global temporary, table.
>
> Is there any substantial difference that I am not aware of?
>

What I know, the variables are used as query parameters, not as relations -
Oracle, DB2, MSSQL, MySQL, ...

semantically, yes - it is a global temporary object, but  it can be scalar
or composite value - it is not row.

(global (temp)) table can hold 0, 1 or more rows (and rows are always
composite of 0..n fields). The variable holds a value of some type.
Proposed session variables are like plpgsql variables (only with different
scope). In Postgres there is a difference between a scalar variable and
composite variable with one field.

Regards

Pavel


> Best,
>
> Wolfgang
>

  282. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-11-16T15:41:10Z

    Hi

rebase - based on Laurenz's patch set

Regards

Pavel

  283. Re: proposal: schema variables

    Wolfgang Walther <walther@technowledgy.de> — 2024-11-16T17:13:38Z

    Pavel Stehule:
> (global (temp)) table can hold 0, 1 or more rows (and rows are always 
> composite of 0..n fields). The variable holds a value of some type. 
> Proposed session variables are like plpgsql variables (only with 
> different scope). In Postgres there is a difference between a scalar 
> variable and composite variable with one field. 

I can store composite values in table columns, too. A table column can 
either be scalar or composite in that sense.

So, maybe rephrase: Single-row, single-column (global (temp)) table = 
variable. One "cell" of that table.

For me, the major difference between a variable and a table is, that the 
table has 0...n rows and 0...m columns, while the variable has *exactly* 
one in both cases, not 0 either.

I must put tables into FROM, why not those nice mini-tables called 
variables as well? Because they are potentially scalar, you say!

But: I can already put functions returning scalar values into FROM:

   SELECT * FROM format('hello');

The function returns a plain string only.

I don't know. This just "fits" for me.

Or to put it differently: I don't really care whether I have to write 
"(SELECT variable FROM variable)" instead of just "variable". I don't 
want session variables for the syntax, I want session variables, because 
they are **so much better** than custom GUCs.

Best,

Wolfgang

  284. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-11-16T22:07:29Z

    so 16. 11. 2024 v 18:13 odesílatel Wolfgang Walther <walther@technowledgy.de>
napsal:

> Pavel Stehule:
> > (global (temp)) table can hold 0, 1 or more rows (and rows are always
> > composite of 0..n fields). The variable holds a value of some type.
> > Proposed session variables are like plpgsql variables (only with
> > different scope). In Postgres there is a difference between a scalar
> > variable and composite variable with one field.
>
> I can store composite values in table columns, too. A table column can
> either be scalar or composite in that sense.
>
> So, maybe rephrase: Single-row, single-column (global (temp)) table =
> variable. One "cell" of that table.
>

the tables are tables and variables are variables. For tables you have
INSERT, UPDATE, DELETE commands, for variables you have a LET command.

and scalar is not a single column composite.

The session variables can in some particular use cases replace global temp
tables, but this is not the goal. I would like to see global temp tables in
Postgres too. Maybe session variables prepare a field for this, because
some people better understand global temp objects. But again my proposal is
not related to global temp tables. This is a different feature.


>
> For me, the major difference between a variable and a table is, that the
> table has 0...n rows and 0...m columns, while the variable has *exactly*
> one in both cases, not 0 either.
>
> I must put tables into FROM, why not those nice mini-tables called
> variables as well? Because they are potentially scalar, you say!
>
> But: I can already put functions returning scalar values into FROM:
>
>    SELECT * FROM format('hello');
>
> The function returns a plain string only.
>
> I don't know. This just "fits" for me.
>

There are more issues - one - when you use some composite in FROM clause,
then you expect an unpacked result. But there are a lot of uses, when
unpackaging is not wanted. There is a syntax for this but it is really not
intuitive and not well readable.


>
> Or to put it differently: I don't really care whether I have to write
> "(SELECT variable FROM variable)" instead of just "variable". I don't
> want session variables for the syntax, I want session variables, because
> they are **so much better** than custom GUCs.
>

session variables are better than GUC for the proposed purpose.  But it
should look like variables. The software should respect standards or common
typical usage when it is possible. If we introduce fully proprietary
design, then it will be hell for all people who know any other databases.
And I don't see a strong benefit from this syntax. It solves just one case,
it doesn't solve other possible issues, and it introduces another possible
risk.

Regards

Pavel


> Best,
>
> Wolfgang
>
>

  285. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-11-16T22:49:42Z

    so 16. 11. 2024 v 15:27 odesílatel Dmitry Dolgov <9erthalion6@gmail.com>
napsal:

> > On Sat, Nov 16, 2024 at 07:10:31AM GMT, Pavel Stehule wrote:
>
> Sorry, got distracted. Let me try to answer step by step.
>
> > > As far as I recall, last time this topic was discussed in hackers, two
> > > options were proposed: the one with VARIABLE(name), what you mention
> > > here; and another one with adding variables to the FROM clause. The
> > > VARIABLE(...) syntax didn't get much negative feedback, so I guess why
> > > not -- if you find it fitting, it would be interesting to see the
> > > implementation.
> > >
> > > I'm afraid it should not be just an alternative syntax, but the only
> one
> > > allowed, because otherwise I don't see how scenarious like "drop a
> > > column with the same name" could be avoided. As in the previous thread:
> > >
> > >     -- we've got a variable b at the same time
> > >     SELECT a, b FROM table1;
> > >
> >
> > I am sorry, but I am in very strong opposition  against this idea. Nobody
> > did reply to my questions, that can change my opinion.
>
> From your reply it's not quite clear, are you opposed to have a mandatory
> VARIABLE syntax, or having variables in the FROM clause? My main proposal
> was
> about the former, but the points that are following seems to talk about the
> latter. I think it's fine to reject the idea about the FROM clause, as
> long as
> you got some reasonable arguments.
>

I am against a requirement to specify a variable in the FROM clause.


>
> > > Then dropping the column b, but everything still works beause the
> > > variable b got silently picked up. But if it would be required to say
> > > VARIABLE(b), then all fine.
> >
> > but same risk you have any time in plpgsql - all time. I don't remember
> any
> > bug report related to this issue.
>
> Which exactly scenario about plpgsql do you have in mind? Just have tried
> to
> declare a variable inside a plpgsql function with the same name as a table
> column, and got an error about an ambiguous reference.
>

Until you execute the query, you cannot know if there is a conflict or not.
So you can change table structure and you can change the procedure's code,
and there can be an invisible conflict until execution and query
evaluation.  The conflict between PL/pgSQL and SQL raises an error. The
conflict between session variables and SQL raises warnings. The issue is
detected.



>
> > Theoretically, variables can have the same names as tables. The table
> > overshadows the variable, so it can work. But when somebody drops the
> > variable, then the query still can work. So requirement of usage variable
> > in FROM clause protects us just against drop column, but not against
> > dropping table. In Postgres the dropping table is possibly risky due
> > search_path (that introduces shadowing concept) without introduction
> > variables. There is a possibility of this issue, but how common is this
> > issue?
>
> This sounds to me like an argument against allowing name clashing between
> variables and tables. It makes even more sense, since session variables
> are in
> many ways similar to tables.
>
>
It doesn't help too much. It can fix just one issue. But you can have
tables with the same name in different schemas inside schemas from
search_path. Unique table names solve nothing.


> > I think this issue can be partially similar to creating two equally named
> > tables in different schemas (both schemas are in search path). When you
> > drop one table, the query will work, but the result is different. It is
> the
> > same issue. The SQL has no concept of shadowing and on the base line it
> is
> > not necessary.
>
> The point is that most of users are aware about schemas and search path
> dangers. But to me such a precedent is not an excuse to introduce a new
> feature
> with similar traps, which folks would have to learn by making mistakes.
> Judging
> from the feedback to this patch over time, I've got an impression that
> lots of
> people are also not fans of that.
>

Unfortunately - I don't believe so there is some syntax without traps. You
can check all implementations in other databases. These designs are very
different, and all have some issues and all have some limits. It is native
- you are trying to join the procedural and functional world.

I understand the risks. These risks are there. But there is no silver
bullet - all proposed designs fixed just one case, and not others, and then
I don't see a strong enough benefit to introduce design that is far from
common usage. Maybe I have a different experience, because I am a man from
the stored procedures area, and the risk of collisions is a known issue
well solved by common conventions and in postgres by
plpgsql.variable_conflict setting. The proposed patch set has very similar
functionality. I think the introduction of VARIABLE(xx) syntax and safe
syntax guard warning the usage of variables can be safe in how it is
possible. But still I want to allow "short" "usual" usage to people who use
a safe convention. There is no risk when you use a safe prefix or safe
schema.



>
> > > Then dropping the column b, but everything still works beause the
> > > variable b got silently picked up. But if it would be required to say
> > > VARIABLE(b), then all fine.
> > >
> >
> > In this scenario you will get  a warning related to variable shadowing
> > (before you drop a column).
> >
> > [...]
> >
> > What do you think about the following design?  I can implement a warning
> > "variable_usage_guard" when the variable is accessed without using
> > VARIABLE() syntax. We can discuss later if this warning can be enabled by
> > default or not. There I am open to any variant.
>
> I don't follow what are you winning by that? In the context of problem
> above
> (i.e. dropping a column), such a warning is functionally equivalend to a
> warning about shadowing.
>
> The problem is that it doesn't sound very appealing to have a feature,
> which
> requires some extra efforts to be used in a right way (e.g. put everything
> into
> a special vars schema, or keep an eye on logs). Most certainly there are
> such
> bits in PostgreSQL today, with all the best practices, crowd wisdom, etc.
> But
> the bar for new features in this sense is much higher, you can see it from
> the
> feedback to this patch. Thus I believe it makes sense, from purely tactical
> reasons, to not try to convince half of the community to lower the bar, but
> instead try to modify the feature to make it more acceptable, even if some
> parts you might not like.
>
> Btw, could you repeat, what was exactly your argument against mandatory
> VARIABLE() syntax? It's somehow scattered across many replies, would be
> great
> to summarize it in a couple of phrases.
>
> > Shadowing by self is not an issue, probably, but it is a signal of code
> > quality problems.
>
> Agree, but I'm afraid code quality of an average application using
> PostgreSQL
> is quite low, so here we are.
>
> As a side note, I've recently caught myself thinking "it would be cool to
> have
> session variables here". The use case was preparing a policy for RLS,
> based on
> some session-level data set by an application. This session-level data is
> of a
> composite data type, so simple current_setting is cumbersome to use, and a
> temporary table will be dropped at the end, taking the policy with it due
> to
> the recorded dependency between them. Thus a session variable of some
> composite
> type sounds like a good fit.
>

  286. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-11-17T04:41:00Z

    so 16. 11. 2024 v 23:49 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
napsal:

>
>
> so 16. 11. 2024 v 15:27 odesílatel Dmitry Dolgov <9erthalion6@gmail.com>
> napsal:
>
>> > On Sat, Nov 16, 2024 at 07:10:31AM GMT, Pavel Stehule wrote:
>>
>> Sorry, got distracted. Let me try to answer step by step.
>>
>> > > As far as I recall, last time this topic was discussed in hackers, two
>> > > options were proposed: the one with VARIABLE(name), what you mention
>> > > here; and another one with adding variables to the FROM clause. The
>> > > VARIABLE(...) syntax didn't get much negative feedback, so I guess why
>> > > not -- if you find it fitting, it would be interesting to see the
>> > > implementation.
>> > >
>> > > I'm afraid it should not be just an alternative syntax, but the only
>> one
>> > > allowed, because otherwise I don't see how scenarious like "drop a
>> > > column with the same name" could be avoided. As in the previous
>> thread:
>> > >
>> > >     -- we've got a variable b at the same time
>> > >     SELECT a, b FROM table1;
>> > >
>> >
>> > I am sorry, but I am in very strong opposition  against this idea.
>> Nobody
>> > did reply to my questions, that can change my opinion.
>>
>> From your reply it's not quite clear, are you opposed to have a mandatory
>> VARIABLE syntax, or having variables in the FROM clause? My main proposal
>> was
>> about the former, but the points that are following seems to talk about
>> the
>> latter. I think it's fine to reject the idea about the FROM clause, as
>> long as
>> you got some reasonable arguments.
>>
>
> I am against a requirement to specify a variable in the FROM clause.
>
>
>>
>> > > Then dropping the column b, but everything still works beause the
>> > > variable b got silently picked up. But if it would be required to say
>> > > VARIABLE(b), then all fine.
>> >
>> > but same risk you have any time in plpgsql - all time. I don't remember
>> any
>> > bug report related to this issue.
>>
>> Which exactly scenario about plpgsql do you have in mind? Just have tried
>> to
>> declare a variable inside a plpgsql function with the same name as a table
>> column, and got an error about an ambiguous reference.
>>
>
> Until you execute the query, you cannot know if there is a conflict or
> not. So you can change table structure and you can change the procedure's
> code, and there can be an invisible conflict until execution and query
> evaluation.  The conflict between PL/pgSQL and SQL raises an error. The
> conflict between session variables and SQL raises warnings. The issue is
> detected.
>
>
>
>>
>> > Theoretically, variables can have the same names as tables. The table
>> > overshadows the variable, so it can work. But when somebody drops the
>> > variable, then the query still can work. So requirement of usage
>> variable
>> > in FROM clause protects us just against drop column, but not against
>> > dropping table. In Postgres the dropping table is possibly risky due
>> > search_path (that introduces shadowing concept) without introduction
>> > variables. There is a possibility of this issue, but how common is this
>> > issue?
>>
>> This sounds to me like an argument against allowing name clashing between
>> variables and tables. It makes even more sense, since session variables
>> are in
>> many ways similar to tables.
>>
>>
> It doesn't help too much. It can fix just one issue. But you can have
> tables with the same name in different schemas inside schemas from
> search_path. Unique table names solve nothing.
>

the combination of pg_class and pg_attribute cannot describe scalar
variables (without hacks). Then you need to  enhance pg_class, which can be
confusing. And on the second hand almost all columns in pg_class have no
sense for variables. And when variables and tables are in different tables,
you cannot ensure a unique name. Variables are similar to tables only in
possibility to hold a value. That is all. But variables don't store data to
file, don't store data in pages, don't allow usage of other storages or
formats, and don't support foreign storage. The similarity between
variables and tables is like the similarity between horses and cars. Both
can help with moving.


>
>> > I think this issue can be partially similar to creating two equally
>> named
>> > tables in different schemas (both schemas are in search path). When you
>> > drop one table, the query will work, but the result is different. It is
>> the
>> > same issue. The SQL has no concept of shadowing and on the base line it
>> is
>> > not necessary.
>>
>> The point is that most of users are aware about schemas and search path
>> dangers. But to me such a precedent is not an excuse to introduce a new
>> feature
>> with similar traps, which folks would have to learn by making mistakes.
>> Judging
>> from the feedback to this patch over time, I've got an impression that
>> lots of
>> people are also not fans of that.
>>
>
> Unfortunately - I don't believe so there is some syntax without traps. You
> can check all implementations in other databases. These designs are very
> different, and all have some issues and all have some limits. It is native
> - you are trying to join the procedural and functional world.
>
> I understand the risks. These risks are there. But there is no silver
> bullet - all proposed designs fixed just one case, and not others, and then
> I don't see a strong enough benefit to introduce design that is far from
> common usage. Maybe I have a different experience, because I am a man from
> the stored procedures area, and the risk of collisions is a known issue
> well solved by common conventions and in postgres by
> plpgsql.variable_conflict setting. The proposed patch set has very similar
> functionality. I think the introduction of VARIABLE(xx) syntax and safe
> syntax guard warning the usage of variables can be safe in how it is
> possible. But still I want to allow "short" "usual" usage to people who use
> a safe convention. There is no risk when you use a safe prefix or safe
> schema.
>
>
>
>>
>> > > Then dropping the column b, but everything still works beause the
>> > > variable b got silently picked up. But if it would be required to say
>> > > VARIABLE(b), then all fine.
>> > >
>> >
>> > In this scenario you will get  a warning related to variable shadowing
>> > (before you drop a column).
>> >
>> > [...]
>> >
>> > What do you think about the following design?  I can implement a warning
>> > "variable_usage_guard" when the variable is accessed without using
>> > VARIABLE() syntax. We can discuss later if this warning can be enabled
>> by
>> > default or not. There I am open to any variant.
>>
>> I don't follow what are you winning by that? In the context of problem
>> above
>> (i.e. dropping a column), such a warning is functionally equivalend to a
>> warning about shadowing.
>>
>> The problem is that it doesn't sound very appealing to have a feature,
>> which
>> requires some extra efforts to be used in a right way (e.g. put
>> everything into
>> a special vars schema, or keep an eye on logs). Most certainly there are
>> such
>> bits in PostgreSQL today, with all the best practices, crowd wisdom, etc.
>> But
>> the bar for new features in this sense is much higher, you can see it
>> from the
>> feedback to this patch. Thus I believe it makes sense, from purely
>> tactical
>> reasons, to not try to convince half of the community to lower the bar,
>> but
>> instead try to modify the feature to make it more acceptable, even if some
>> parts you might not like.
>>
>> Btw, could you repeat, what was exactly your argument against mandatory
>> VARIABLE() syntax? It's somehow scattered across many replies, would be
>> great
>> to summarize it in a couple of phrases.
>>
>> > Shadowing by self is not an issue, probably, but it is a signal of code
>> > quality problems.
>>
>> Agree, but I'm afraid code quality of an average application using
>> PostgreSQL
>> is quite low, so here we are.
>>
>> As a side note, I've recently caught myself thinking "it would be cool to
>> have
>> session variables here". The use case was preparing a policy for RLS,
>> based on
>> some session-level data set by an application. This session-level data is
>> of a
>> composite data type, so simple current_setting is cumbersome to use, and a
>> temporary table will be dropped at the end, taking the policy with it due
>> to
>> the recorded dependency between them. Thus a session variable of some
>> composite
>> type sounds like a good fit.
>>
>

  287. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-11-17T04:53:17Z

    so 16. 11. 2024 v 23:07 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
napsal:

>
>
> so 16. 11. 2024 v 18:13 odesílatel Wolfgang Walther <
> walther@technowledgy.de> napsal:
>
>> Pavel Stehule:
>> > (global (temp)) table can hold 0, 1 or more rows (and rows are always
>> > composite of 0..n fields). The variable holds a value of some type.
>> > Proposed session variables are like plpgsql variables (only with
>> > different scope). In Postgres there is a difference between a scalar
>> > variable and composite variable with one field.
>>
>> I can store composite values in table columns, too. A table column can
>> either be scalar or composite in that sense.
>>
>> So, maybe rephrase: Single-row, single-column (global (temp)) table =
>> variable. One "cell" of that table.
>>
>
> the tables are tables and variables are variables. For tables you have
> INSERT, UPDATE, DELETE commands, for variables you have a LET command.
>
> and scalar is not a single column composite.
>

example

assignment to scalar versus single composite

LET a  = 10
LET a.a = 10 or LET a = ROW(10)

Single column tables can be the result of some ALTERS - or sometimes you
can use a table type. But for example, plpgsql, very strongly differs
between scalar and composite variables. So introducing the "new" concept -
single field composite is scalar introduces strong inconsistency there.

Regards

Pavel



>
> The session variables can in some particular use cases replace global temp
> tables, but this is not the goal. I would like to see global temp tables in
> Postgres too. Maybe session variables prepare a field for this, because
> some people better understand global temp objects. But again my proposal is
> not related to global temp tables. This is a different feature.
>
>
>>
>> For me, the major difference between a variable and a table is, that the
>> table has 0...n rows and 0...m columns, while the variable has *exactly*
>> one in both cases, not 0 either.
>>
>> I must put tables into FROM, why not those nice mini-tables called
>> variables as well? Because they are potentially scalar, you say!
>>
>> But: I can already put functions returning scalar values into FROM:
>>
>>    SELECT * FROM format('hello');
>>
>> The function returns a plain string only.
>>
>> I don't know. This just "fits" for me.
>>
>
> There are more issues - one - when you use some composite in FROM clause,
> then you expect an unpacked result. But there are a lot of uses, when
> unpackaging is not wanted. There is a syntax for this but it is really not
> intuitive and not well readable.
>
>
>>
>> Or to put it differently: I don't really care whether I have to write
>> "(SELECT variable FROM variable)" instead of just "variable". I don't
>> want session variables for the syntax, I want session variables, because
>> they are **so much better** than custom GUCs.
>>
>
> session variables are better than GUC for the proposed purpose.  But it
> should look like variables. The software should respect standards or common
> typical usage when it is possible. If we introduce fully proprietary
> design, then it will be hell for all people who know any other databases.
> And I don't see a strong benefit from this syntax. It solves just one case,
> it doesn't solve other possible issues, and it introduces another possible
> risk.
>
> Regards
>
> Pavel
>
>
>> Best,
>>
>> Wolfgang
>>
>>

  288. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-11-19T19:14:01Z

    Hi

I wrote POC of VARIABLE(varname) syntax support

here is a copy from regress test

SET session_variables_ambiguity_warning TO on;
SET session_variables_use_fence_warning_guard TO on;
SET search_path TO 'public';
CREATE VARIABLE a AS int;
LET a = 10;
CREATE TABLE test_table(a int, b int);
INSERT INTO test_table VALUES(20, 20);
-- no warning
SELECT a;
 a..
----
 10
(1 row)

-- warning variable is shadowed
SELECT a, b FROM test_table;
WARNING:  session variable "a" is shadowed
LINE 1: SELECT a, b FROM test_table;
               ^
DETAIL:  Session variables can be shadowed by columns, routine's variables
and routine's arguments with the same name.
 a  | b..
----+----
 20 | 20
(1 row)

-- no warning
SELECT variable(a) FROM test_table;
 a..
----
 10
(1 row)

ALTER TABLE test_table DROP COLUMN a;
-- warning - variable fence is not used
SELECT a, b FROM test_table;
WARNING:  session variable "a" is not used inside variable fence
LINE 1: SELECT a, b FROM test_table;
               ^
DETAIL:  The collision of session variable' names and column names is
possible.
 a  | b..
----+----
 10 | 20
(1 row)

-- no warning
SELECT variable(a), b FROM test_table;
 a  | b..
----+----
 10 | 20
(1 row)

DROP VARIABLE a;
DROP TABLE test_table;
SET session_variables_ambiguity_warning TO DEFAULT;
SET session_variables_use_fence_warning_guard TO DEFAULT;
SET search_path TO DEFAULT;

Regards

Pavel

  289. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-11-19T21:30:00Z

    Hi

testing farms reports some issue related to introduction new parser node. I
increased verbosity

Regards

Pavel

  290. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-11-20T07:25:53Z

    Hi

fix missing support for VariableFence in raw_expression_tree_walker_impl

Regards

Pavel

  291. Re: proposal: schema variables

    Marcos Pegoraro <marcos@f10.com.br> — 2024-11-20T13:29:10Z

    Em ter., 19 de nov. de 2024 às 16:15, Pavel Stehule <pavel.stehule@gmail.com>
escreveu:

> I wrote POC of VARIABLE(varname) syntax support
>

Not related with POC of VARIABLE but seeing your patches ...

Wouldn't it be better to use just one syntax and message for what to do ON
COMMIT ?

When creating a new variable you use
CREATE VARIABLE ... ON COMMIT DROP | ON TRANSACTION END RESET

On PSQL \dV+ you show
Transactional end action

Maybe all them could be just ON COMMIT
CREATE VARIABLE ... [ON COMMIT {NO ACTION, DROP, RESET}] and \dV+ just "on
commit" on title column

regards
Marcos

  292. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-11-20T13:52:16Z

    Hi

st 20. 11. 2024 v 14:29 odesílatel Marcos Pegoraro <marcos@f10.com.br>
napsal:

> Em ter., 19 de nov. de 2024 às 16:15, Pavel Stehule <
> pavel.stehule@gmail.com> escreveu:
>
>> I wrote POC of VARIABLE(varname) syntax support
>>
>
> Not related with POC of VARIABLE but seeing your patches ...
>
> Wouldn't it be better to use just one syntax and message for what to do ON
> COMMIT ?
>
> When creating a new variable you use
> CREATE VARIABLE ... ON COMMIT DROP | ON TRANSACTION END RESET
>
> On PSQL \dV+ you show
> Transactional end action
>
> Maybe all them could be just ON COMMIT
> CREATE VARIABLE ... [ON COMMIT {NO ACTION, DROP, RESET}] and \dV+ just "on
> commit" on title column
>

ON COMMIT DROP is related to temporary objects. In this case, you don't
need to think about ROLLBACK, because in this case, the temp objects are
removed implicitly.

ON TRANSACTION END RESET can be used for non temporary objects too. So this
is a little bit of a different feature. But the reset is executed if the
transaction is ended by ROLLBACK too. So using a syntax just ON COMMIT can
be a little bit messy. TRANSACTION END is more intuitive, I think. If I
remember there was a proposal ON COMMIT OR ROLLBACK, but I think
TRANSACTION END is better and more intuitive, and better describes what is
implemented. I can imagine to support clauses ON COMMIT RESET or ON
ROLLBACK RESET that can be used independently, but for this time, I don't
want to increase a complexity now - reset is just at transaction end
without dependency if the transaction was committed or rollbacked.

Regards

Pavel


> regards
> Marcos
>

  293. Re: proposal: schema variables

    Marcos Pegoraro <marcos@f10.com.br> — 2024-11-20T14:14:55Z

    Em qua., 20 de nov. de 2024 às 10:52, Pavel Stehule <pavel.stehule@gmail.com>
escreveu:

> COMMIT can be a little bit messy. TRANSACTION END is more intuitive, I
> think.
>
>>
Exactly to be not messy I would just ON COMMIT for all, and DOCs can
explain that this option is ignored for temp objects and do the same at the
end of transaction, independently if commited or rolled back

  294. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-11-20T14:57:23Z

    st 20. 11. 2024 v 15:15 odesílatel Marcos Pegoraro <marcos@f10.com.br>
napsal:

> Em qua., 20 de nov. de 2024 às 10:52, Pavel Stehule <
> pavel.stehule@gmail.com> escreveu:
>
>> COMMIT can be a little bit messy. TRANSACTION END is more intuitive, I
>> think.
>>
>>>
> Exactly to be not messy I would just ON COMMIT for all, and DOCs can
> explain that this option is ignored for temp objects and do the same at the
> end of transaction, independently if commited or rolled back
>

I feel it differently - when I read ON COMMIT, then I expect really just a
COMMIT event, not ROLLBACK.  Attention - the metadata about variables are
transactional, but the variables are not transactional (by default).

But this feeling can be subjective. The objective argument against using ON
COMMIT like ON TRANSACTION END is fact, so we lost a possibility for a more
precious setting.

I can imagine scenarios with ON COMMIT RESET - and variable can hold some
last value from transaction, or ON ROLLBACK RESET - and variable can hold
some computed value from successfully ended transaction - last inserted id.

In this case, I don't see a problem to reopen a discussion about syntax or
postpone this feature. I think the possibility to reset variables at some
specified time can be an interesting feature (that can increase safety,
because the application doesn't need to solve initial setup),  but from the
list of implemented features for session variables, this is not too far
from the end. If TRANSACTION END is not intuitive - with exactly the same
functionality can be RESET AT TRANSACTION START - so the ON TRANSACTION END
can be transformed to ON BEGIN RESET, and this syntax can be maybe better,
because it signalize, for every transaction, the variable will be
initialized to default. What do you think? Can be ON BEGIN RESET acceptable
for you.

Regards

Pavel

  295. Re: proposal: schema variables

    Dmitry Dolgov <9erthalion6@gmail.com> — 2024-11-20T20:13:13Z

    > On Tue, Nov 19, 2024 at 08:14:01PM +0100, Pavel Stehule wrote:
> Hi
>
> I wrote POC of VARIABLE(varname) syntax support

Thanks, the results look good. I'm still opposed the idea of having a
warning, and think it has to be an error -- but it's my subjective
opinion. Lets see if there are more votes on that topic.

  296. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-11-21T04:07:09Z

    st 20. 11. 2024 v 21:14 odesílatel Dmitry Dolgov <9erthalion6@gmail.com>
napsal:

> > On Tue, Nov 19, 2024 at 08:14:01PM +0100, Pavel Stehule wrote:
> > Hi
> >
> > I wrote POC of VARIABLE(varname) syntax support
>
> Thanks, the results look good. I'm still opposed the idea of having a
> warning, and think it has to be an error -- but it's my subjective
> opinion. Lets see if there are more votes on that topic.
>

The error breaks the possibility to use variables (session variables) like
Oracle's package variables easily. It increases effort for transformation
or porting because you should identify variables inside queries and you
should wrap it to fence.  On the other hand, extensions that can read a
query after transformation can easily detect unwrapped variables and they
can raise an error. It can be very easy to implement this check to
plpgsql_check for example or like plpgsql.extra_check.

In my ideal world, the shadowing warning should be enabled by default, and
an unfencing warning disabled by default. But I have not a problem with
activating both warnings by default. I think warnings  are enough, because
if there is some risk then a shadowing warning is activated. And my
experience is more positive about warnings, people read it.

Regards

Pavel

  297. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-11-26T06:21:49Z

    Hi

fix after  96a81c1

Regards

Pavel

  298. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-11-27T18:14:12Z

    Hi

rebase + doc for variable fence + move variable fence patch after
GUC-session_variables_ambiguity_warning

Regards

Pavel

  299. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-12-05T06:51:24Z

    Hi

only rebase

Regards

Pavel

  300. Re: proposal: schema variables

    jian he <jian.universality@gmail.com> — 2024-12-07T02:13:26Z

    On Thu, Dec 5, 2024 at 2:52 PM Pavel Stehule <pavel.stehule@gmail.com> wrote:
>
> Hi
>
> only rebase

hi.
disclaimer, i *only* applied
v20241205-0001-Enhancing-catalog-for-support-session-variables-and-.patch.

create variable v2 as text;
alter variable v2 rename to v2;
ERROR:  session variable "v2" already exists in schema "public"

the above is  coverage tests for report_namespace_conflict:
        case VariableRelationId:
            Assert(OidIsValid(nspOid));
            msgfmt = gettext_noop("session variable \"%s\" already
exists in schema \"%s\"");
            break;

create type t1 as (a int, b int);
CREATE VARIABLE var1 AS t1;
alter type t1 drop attribute a;
should "alter type t1 drop attribute a;" not allowed?


GetCommandLogLevel also needs to deal with  case T_CreateSessionVarStmt?


there are no regress tests for the change we made in
find_composite_type_dependencies?
It looks like it will be reachable for sure.


CreateVariable, print out error position:
    if (get_typtype(typid) == TYPTYPE_PSEUDO)
        ereport(ERROR,
                (errcode(ERRCODE_WRONG_OBJECT_TYPE),
                 errmsg("session variable cannot be pseudo-type %s",
                        format_type_be(typid)),
                 parser_errposition(pstate, stmt->typeName->location)));

Alvaro Herrera told me actually, you don't need the extra parentheses
around errcode.
so you can:
    if (get_typtype(typid) == TYPTYPE_PSEUDO)
        ereport(ERROR,
                errcode(ERRCODE_WRONG_OBJECT_TYPE),
                errmsg("session variable cannot be pseudo-type %s",
                        format_type_be(typid)),
                parser_errposition(pstate, stmt->typeName->location))


pg_variable_is_visible seems to have done twice the system cache call.
maybe follow through with the pg_table_is_visible, pg_type_is_visible
code pattern.


IN src/bin/psql/describe.c
+ appendPQExpBufferStr(&buf, "\nWHERE true\n");
this is not needed?
------------------------------------------------
some of the `foreach` can change to foreach_oid, foreach_node
see: https://git.postgresql.org/cgit/postgresql.git/commit/?id=14dd0f27d7cd56ffae9ecdbe324965073d01a9ff
------------------------------------------------
doc/src/sgml/ref/create_variable.sgml
<programlisting>
CREATE VARIABLE var1 AS date;
LET var1 = current_date;
SELECT var1;
</programlisting>

v20241205-0001-Enhancing-catalog-for-support-session-variables-and-.patch
alone cannot do `LET var1 = current_date;`, `SELECT var1;`
maybe the following patch can do it. but that makes
we cannot commit
v20241205-0001-Enhancing-catalog-for-support-session-variables-and-.patch
alone.
------------------------------------------------
since CREATE VARIABLE is an extension to standard, so in create_schema.sgml
Compatibility section,
do we need to mention CREATE SCHEMA CREATE VARIABLE is an extension
from standard
?
-----------------------------------------------

/*
 * Drop variable by OID, and register the needed session variable
 * cleanup.
 */
void
DropVariableById(Oid varid)
i am not sure of the meaning of "register the needed session variable cleanup".

  301. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-12-08T18:32:40Z

    Hi

so 7. 12. 2024 v 3:13 odesílatel jian he <jian.universality@gmail.com>
napsal:

> On Thu, Dec 5, 2024 at 2:52 PM Pavel Stehule <pavel.stehule@gmail.com>
> wrote:
> >
> > Hi
> >
> > only rebase
>
> hi.
> disclaimer, i *only* applied
> v20241205-0001-Enhancing-catalog-for-support-session-variables-and-.patch.
>
> create variable v2 as text;
> alter variable v2 rename to v2;
> ERROR:  session variable "v2" already exists in schema "public"
>
> the above is  coverage tests for report_namespace_conflict:
>         case VariableRelationId:
>             Assert(OidIsValid(nspOid));
>             msgfmt = gettext_noop("session variable \"%s\" already
> exists in schema \"%s\"");
>             break;
>

I don't understand what is an issue?

This is consistent with other database object

(2024-12-07 18:25:29) postgres=# create table foo(a int);
CREATE TABLE
(2024-12-07 18:25:35) postgres=# alter table foo rename
a           COLUMN      CONSTRAINT  TO
(2024-12-07 18:25:35) postgres=# alter table foo rename to foo;
ERROR:  relation "foo" already exists



>
> create type t1 as (a int, b int);
> CREATE VARIABLE var1 AS t1;
> alter type t1 drop attribute a;
> should "alter type t1 drop attribute a;" not allowed?
>

you can add a new attribute, or you can drop the attribute. You cannot
change the type of the attribute.


>
> GetCommandLogLevel also needs to deal with  case T_CreateSessionVarStmt?
>

yes - it should be there - I fixed this + regress test


>
> there are no regress tests for the change we made in
> find_composite_type_dependencies?
> It looks like it will be reachable for sure.
>

It was tested in patch02, where the correct work with variable's values.

But I wrote few tests, that better describe this functionality and I added
to patch01

+SET log_statement TO ddl;
 -- should be ok
 CREATE VARIABLE var1 AS int;
 -- should fail, pseudotypes are not allowed
@@ -15,6 +16,36 @@ CREATE VARIABLE var1 AS int;
 ERROR:  session variable "var1" already exists
 -- should be ok
 DROP VARIABLE IF EXISTS var1;
+-- the variable can use composite types
+CREATE TABLE t1 (a int, b int);
+CREATE VARIABLE var1 AS t1;
+-- should fail
+DROP TABLE t1;
+ERROR:  cannot drop table t1 because other objects depend on it
+DETAIL:  session variable var1 depends on type t1
+HINT:  Use DROP ... CASCADE to drop the dependent objects too.
+-- should be ok
+ALTER TABLE t1 ADD COLUMN c int;
+-- should fail
+ALTER TABLE t1 ALTER COLUMN b TYPE numeric;
+ERROR:  cannot alter table "t1" because session variable "public.var1"
uses it
+DROP VARIABLE var1;
+DROP TABLE t1;
+CREATE TYPE t1 AS (a int, b int);
+CREATE VARIABLE var1 AS t1;
+-- should fail
+DROP TYPE t1;
+ERROR:  cannot drop type t1 because other objects depend on it
+DETAIL:  session variable var1 depends on type t1
+HINT:  Use DROP ... CASCADE to drop the dependent objects too.
+-- should be ok
+ALTER TYPE t1 ADD ATTRIBUTE c int;
+-- should fail
+ALTER TYPE t1 ALTER ATTRIBUTE b  TYPE numeric;
+ERROR:  cannot alter type "t1" because session variable "public.var1" uses
it
+DROP VARIABLE var1;
+DROP TYPE t1;
+SET log_statement TO default;



>
>
> CreateVariable, print out error position:
>     if (get_typtype(typid) == TYPTYPE_PSEUDO)
>         ereport(ERROR,
>                 (errcode(ERRCODE_WRONG_OBJECT_TYPE),
>                  errmsg("session variable cannot be pseudo-type %s",
>                         format_type_be(typid)),
>                  parser_errposition(pstate, stmt->typeName->location)));
>
> Alvaro Herrera told me actually, you don't need the extra parentheses
> around errcode.
> so you can:
>     if (get_typtype(typid) == TYPTYPE_PSEUDO)
>         ereport(ERROR,
>                 errcode(ERRCODE_WRONG_OBJECT_TYPE),
>                 errmsg("session variable cannot be pseudo-type %s",
>                         format_type_be(typid)),
>                 parser_errposition(pstate, stmt->typeName->location))
>

yes, it is not necessary, but it is used almost everywhere in Postgres
source code. So I prefer
to be consistent with usual ereport notation. It can be fixed later, but it
is used 7136 times in pg code.
Probably this needs a wide discussion. Without extra parenthesis the code
looks little bit nicer,
but this should be changed by a dedicated patch everywhere.



>
> pg_variable_is_visible seems to have done twice the system cache call.
> maybe follow through with the pg_table_is_visible, pg_type_is_visible
> code pattern.
>
>
done


>
> IN src/bin/psql/describe.c
> + appendPQExpBufferStr(&buf, "\nWHERE true\n");
> this is not needed?
>

Yes, it is a little bit messy, but it is necessary due to the usage of a
ValidateSQLNamePattern - it uses the "AND" operator, and
then there should be some expression before. I think that in this way, the
code is most simple. For objects based on
pg_class table is common usage "\nWHERE prokind = \"x\"\n" with the next
enhancement of filtering. But for variables
There is nothing like filtering inside the table pg_variable, so "WHERE
true\n" is the best analogy. See listConversions()

------------------------------------------------
> some of the `foreach` can change to foreach_oid, foreach_node
> see:
> https://git.postgresql.org/cgit/postgresql.git/commit/?id=14dd0f27d7cd56ffae9ecdbe324965073d01a9ff


done


>
> ------------------------------------------------
> doc/src/sgml/ref/create_variable.sgml
> <programlisting>
> CREATE VARIABLE var1 AS date;
> LET var1 = current_date;
> SELECT var1;
> </programlisting>
>
> v20241205-0001-Enhancing-catalog-for-support-session-variables-and-.patch
> alone cannot do `LET var1 = current_date;`, `SELECT var1;`
> maybe the following patch can do it. but that makes
> we cannot commit
> v20241205-0001-Enhancing-catalog-for-support-session-variables-and-.patch
> alone.
>

moved.

The patches 01 and 02 should be committed together to carry some valuable
functionality.



> ------------------------------------------------
> since CREATE VARIABLE is an extension to standard, so in create_schema.sgml
> Compatibility section,
> do we need to mention CREATE SCHEMA CREATE VARIABLE is an extension
> from standard
> ?
>

done


> -----------------------------------------------
>
> /*
>  * Drop variable by OID, and register the needed session variable
>  * cleanup.
>  */
> void
> DropVariableById(Oid varid)
> i am not sure of the meaning of "register the needed session variable
> cleanup".
>

Without context it is messy. It is related to functionality introduced in
patch [PATCH 05/21] memory cleaning after DROP VARIABLE

So I moved text ", and register the needed session variable
 * cleanup." there

Thank you very much for review

updated patchset attached

Regards

Pavel

  302. Re: proposal: schema variables

    jian he <jian.universality@gmail.com> — 2024-12-09T06:16:04Z

    On Mon, Dec 9, 2024 at 2:33 AM Pavel Stehule <pavel.stehule@gmail.com> wrote:
>
> Hi
>
again. only applied
v20241208-0001-Enhancing-catalog-for-support-session-variables-and-.patch.

/* we want SessionVariableCreatePostprocess to see the catalog changes. */
0001 doesn't have SessionVariableCreatePostprocess,
so this comment is wrong in the context of 0001.

typo:
As above, but if the variable isn't found and is_mussing is not NULL
is_mussing should be is_missing.

----------------------------------------------
issue with  grant.sgml and revoke.sgml.

* there are no regress tests for WITH GRANT OPTION but it seems to work;
there are no REVOKE CASCADE tests. the following tests show
REVOKE CASCADE works.

create variable v1 as int;
GRANT select on VARIABLE v1 to alice with grant option;
set session authorization alice;
GRANT select on VARIABLE v1 to bob with grant option;
reset session authorization;

select varacl from pg_variable where varname  = 'v1';
--output
{jian=rw/jian,alice=r*/jian,bob=r*/alice}
revoke all privileges on variable v1 from alice cascade;
select varacl from pg_variable where varname  = 'v1';
--output
 {jian=rw/jian}

revoke GRANT OPTION FOR all privileges on variable v1 from alice cascade;
also works.

* in revoke.sgml and grant.sgml.
if you look closely, " | ALL VARIABLES IN SCHEMA schema_name [, ...] }" is wrong
because there is no left-curly-bracket, but there is a right-curly-bracket.

* in revoke.sgml.
<phrase>where <replaceable
class="parameter">role_specification</replaceable> can be:</phrase>
    [ GROUP ] <replaceable class="parameter">role_name</replaceable>
  | PUBLIC
  | CURRENT_ROLE
  | CURRENT_USER
  | SESSION_USER
should be at the end of the synopsis section?
otherwise it looks weird, maybe we can put the REVOKE VARIABLE code upper.
grant.sgml changes position looks fine to me.


*  <para>
   The <command>GRANT</command> command has two basic variants: one
   that grants privileges on a database object (table, column, view,
   foreign table, sequence, database, foreign-data wrapper, foreign server,
   function, procedure, procedural language, large object, configuration
   parameter, schema, tablespace, or type), and one that grants
   membership in a role.  These variants are similar in many ways, but
   they are different enough to be described separately.
  </para>
this <para> in grant.sgml needs to also mention "variable"?

*   <para>
    Privileges on databases, tablespaces, schemas, languages, and
    configuration parameters are
    <productname>PostgreSQL</productname> extensions.
   </para>
this <para> in grant.sgml needs to also mention "variable"?

----------------------------------------------
*
+   <para>
+    Inside a query or an expression, a session variable can be
+    <quote>shadowed</quote> by a column with the same name.  Similarly, the
+    name of a function or procedure argument or a PL/pgSQL variable (see
PL/pgSQL should decorated as <application>PL/pgSQL</application>

* we already support \dV and \dV+ in
v20241208-0001-Enhancing-catalog-for-support-session-variables-and-.patch
so we should update doc/src/sgml/ref/psql-ref.sgml also.
I briefly searched \dV in
v20241208-0002-Storage-for-session-variables-and-SQL-interface.patch,
no entry.


* in doc/src/sgml/ddl.sgml
<table id="privilege-abbrevs-table"> and <table
id="privileges-summary-table"> need to change for variable?
<varlistentry id="ddl-priv-select">, <varlistentry
id="ddl-priv-update"> need to change for variable?

it's kind of tricky. if we only apply
v20241208-0001-Enhancing-catalog-for-support-session-variables-and-.patch
we can not  SELECT or  UPDATE variable.
so how are we going to mention these privileges for variable?


 * we can add some tests for EVENT TRIGGER test,
since event trigger support CREATE|DROP variable. atually, I think
there is a bug.

create variable v1 as int;
CREATE OR REPLACE FUNCTION event_trigger_report_dropped()
RETURNS event_trigger
LANGUAGE plpgsql
AS $$
DECLARE r record;
BEGIN
FOR r IN SELECT * from pg_event_trigger_dropped_objects()
LOOP
IF NOT r.normal AND NOT r.original THEN
    CONTINUE;
END IF;
RAISE NOTICE 'NORMAL: orig=% normal=% istemp=% type=% identity=% name=% args=%',
    r.original, r.normal, r.is_temporary, r.object_type,
    r.object_identity, r.address_names, r.address_args;
END LOOP;
END; $$;

CREATE EVENT TRIGGER regress_event_trigger_report_dropped ON sql_drop
WHEN TAG IN ('DROP VARIABLE')
EXECUTE PROCEDURE event_trigger_report_dropped();

--output:
src9=# drop variable v1;
NOTICE:  test_event_trigger: ddl_command_start DROP VARIABLE
NOTICE:  NORMAL: orig=t normal=f istemp=f type=session variable
identity=public.v1 name={public,$} args={}
DROP VARIABLE

should i expect
NOTICE:  NORMAL: orig=t normal=f istemp=f type=session variable
identity=public.v1 name={public,$} args={}
to be
NOTICE:  NORMAL: orig=t normal=f istemp=f type=session variable
identity=public.v1 name={public,v1} args={}
?

  303. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-12-09T16:54:26Z

    Hi

st 20. 11. 2024 v 21:14 odesílatel Dmitry Dolgov <9erthalion6@gmail.com>
napsal:

> > On Tue, Nov 19, 2024 at 08:14:01PM +0100, Pavel Stehule wrote:
> > Hi
> >
> > I wrote POC of VARIABLE(varname) syntax support
>
> Thanks, the results look good. I'm still opposed the idea of having a
> warning, and think it has to be an error -- but it's my subjective
> opinion. Lets see if there are more votes on that topic.
>

Maybe the warning of usage of unfenced variables can be changed (enhanced)
to some different mechanism that can be more restrictive (and safer), but I
think it can still be user friendly.

My idea is based on assumption so users with knowledge of stored procedures
know  variables and related risks (and know tools how to minimize risks),
and for other people the risk is higher and we should force usage of
variable fences. I think we can force usage of variable fences at query
runtime, when query is not executed from the SPI environment. This
behaviour can be enabled by default, but can be optionally disabled.

CREATE VARIABLE s.x AS int; -- allowed when user has create right on schema
s
CREATE VIEW v1 AS SELECT x; -- no problem, the check is dynamic
(execution), not static
CREATE VIEW v2 AS SELECT VARIABLE(x); -- no problem

SELECT x; -- fails on access to unfenced variable
SELECT * FROM v1; -- fails on access to unfenced variable
SELECT * FROM v2; -- ok

but inside pl, this check will not be active, and then with default setting
I can write an code like

LET var = 10; -- fencing is not allowed there, and there is not any
collision
DO $$
BEGIN
  RAISE NOTICE 'var=%', var;
  RAISE NOTICE 'var=%', (SELECT * FROM v1); --is ok here too
END;
$$;

Outside PL the fencing can be required, inside PL the fencing can be
optional. With this proposal we can limit the possible risk usage of
unfenced variables only in PL context, and the behaviour can be very
similar to PL/SQL or SQL/PSM. This check is only a runtime check, so it has
no impact on any DDL statement. It doesn't change the syntax or behavior,
so it can be implemented subsequently - it is just a safeguard against
unwanted usage of variables in an environment, where users have no
possibility to use variables already. I can imagine that this check
"allow_unfenced_variables" can have three values (everywhere, pl, nowhere)
and the default can be pl. The results of queries don't depend on the
current setting of this check. For all values for all possible queries and
situations, the result is the same (when it is finished). But sometimes,
the check can break the execution - in similar meaning like access rights.
All previous proposed warnings can be unchanged.

Comments, notes?

Regards

Pavel

  304. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-12-09T23:20:38Z

    po 9. 12. 2024 v 7:16 odesílatel jian he <jian.universality@gmail.com>
napsal:

> On Mon, Dec 9, 2024 at 2:33 AM Pavel Stehule <pavel.stehule@gmail.com>
> wrote:
> >
> > Hi
> >
> again. only applied
> v20241208-0001-Enhancing-catalog-for-support-session-variables-and-.patch.
>
> /* we want SessionVariableCreatePostprocess to see the catalog changes. */
> 0001 doesn't have SessionVariableCreatePostprocess,
> so this comment is wrong in the context of 0001.
>

moved to patch 11


>
> typo:
> As above, but if the variable isn't found and is_mussing is not NULL
> is_mussing should be is_missing.
>
>
fixed


> ----------------------------------------------
> issue with  grant.sgml and revoke.sgml.
>
> * there are no regress tests for WITH GRANT OPTION but it seems to work;
> there are no REVOKE CASCADE tests. the following tests show
> REVOKE CASCADE works.
>
> create variable v1 as int;
> GRANT select on VARIABLE v1 to alice with grant option;
> set session authorization alice;
> GRANT select on VARIABLE v1 to bob with grant option;
> reset session authorization;
>
> select varacl from pg_variable where varname  = 'v1';
> --output
> {jian=rw/jian,alice=r*/jian,bob=r*/alice}
> revoke all privileges on variable v1 from alice cascade;
> select varacl from pg_variable where varname  = 'v1';
> --output
>  {jian=rw/jian}
>
> revoke GRANT OPTION FOR all privileges on variable v1 from alice cascade;
> also works.
>

these features uses generic code, so I didn't wrote regress test, but I
don't see any
reason why don't use your examples in regress tests.


>
> * in revoke.sgml and grant.sgml.
> if you look closely, " | ALL VARIABLES IN SCHEMA schema_name [, ...] }" is
> wrong
> because there is no left-curly-bracket, but there is a right-curly-bracket.
>

fixed


>
> * in revoke.sgml.
> <phrase>where <replaceable
> class="parameter">role_specification</replaceable> can be:</phrase>
>     [ GROUP ] <replaceable class="parameter">role_name</replaceable>
>   | PUBLIC
>   | CURRENT_ROLE
>   | CURRENT_USER
>   | SESSION_USER
> should be at the end of the synopsis section?
> otherwise it looks weird, maybe we can put the REVOKE VARIABLE code upper.
> grant.sgml changes position looks fine to me.
>

it was wrong, REVOKE VARIABLE should be moved up

fixed


>
>
> *  <para>
>    The <command>GRANT</command> command has two basic variants: one
>    that grants privileges on a database object (table, column, view,
>    foreign table, sequence, database, foreign-data wrapper, foreign server,
>    function, procedure, procedural language, large object, configuration
>    parameter, schema, tablespace, or type), and one that grants
>    membership in a role.  These variants are similar in many ways, but
>    they are different enough to be described separately.
>   </para>
> this <para> in grant.sgml needs to also mention "variable"?
>

done


>
> *   <para>
>     Privileges on databases, tablespaces, schemas, languages, and
>     configuration parameters are
>     <productname>PostgreSQL</productname> extensions.
>    </para>
> this <para> in grant.sgml needs to also mention "variable"?
>
>
done


> * we already support \dV and \dV+ in
> v20241208-0001-Enhancing-catalog-for-support-session-variables-and-.patch
> so we should update doc/src/sgml/ref/psql-ref.sgml also.
> I briefly searched \dV in
> v20241208-0002-Storage-for-session-variables-and-SQL-interface.patch,
> no entry.
>

done

----------------------------------------------
> *
> +   <para>
> +    Inside a query or an expression, a session variable can be
> +    <quote>shadowed</quote> by a column with the same name.  Similarly,
> the
> +    name of a function or procedure argument or a PL/pgSQL variable (see
> PL/pgSQL should decorated as <application>PL/pgSQL</application>
>
>
moved this para to patch02


>
>
> * in doc/src/sgml/ddl.sgml
> <table id="privilege-abbrevs-table"> and <table
> id="privileges-summary-table"> need to change for variable?
> <varlistentry id="ddl-priv-select">, <varlistentry
> id="ddl-priv-update"> need to change for variable?
>
> it's kind of tricky. if we only apply
> v20241208-0001-Enhancing-catalog-for-support-session-variables-and-.patch
> we can not  SELECT or  UPDATE variable.
> so how are we going to mention these privileges for variable?
>

I did it to 01 patch.

The line between 01 and 02 patch can be fuzzy sometimes little bit

Just note: applying only the 01 patch does not make any sense.  These
patches (01 and 02) are separated just for review
and testing, but for any usage both patches should be committed or none.



>
>  * we can add some tests for EVENT TRIGGER test,
> since event trigger support CREATE|DROP variable. atually, I think
> there is a bug.
>
> create variable v1 as int;
> CREATE OR REPLACE FUNCTION event_trigger_report_dropped()
> RETURNS event_trigger
> LANGUAGE plpgsql
> AS $$
> DECLARE r record;
> BEGIN
> FOR r IN SELECT * from pg_event_trigger_dropped_objects()
> LOOP
> IF NOT r.normal AND NOT r.original THEN
>     CONTINUE;
> END IF;
> RAISE NOTICE 'NORMAL: orig=% normal=% istemp=% type=% identity=% name=%
> args=%',
>     r.original, r.normal, r.is_temporary, r.object_type,
>     r.object_identity, r.address_names, r.address_args;
> END LOOP;
> END; $$;
>
> CREATE EVENT TRIGGER regress_event_trigger_report_dropped ON sql_drop
> WHEN TAG IN ('DROP VARIABLE')
> EXECUTE PROCEDURE event_trigger_report_dropped();
>
> --output:
> src9=# drop variable v1;
> NOTICE:  test_event_trigger: ddl_command_start DROP VARIABLE
> NOTICE:  NORMAL: orig=t normal=f istemp=f type=session variable
> identity=public.v1 name={public,$} args={}
> DROP VARIABLE
>
> should i expect
> NOTICE:  NORMAL: orig=t normal=f istemp=f type=session variable
> identity=public.v1 name={public,$} args={}
> to be
> NOTICE:  NORMAL: orig=t normal=f istemp=f type=session variable
> identity=public.v1 name={public,v1} args={}
> ?
>

fixed - there was missing pstrdup(varname) in the related part in
getObjectIdentityParts

I used your example in regress test

new patchset attached with necessary rebase

Regards

Pavel

  305. Re: proposal: schema variables

    jian he <jian.universality@gmail.com> — 2024-12-10T03:32:37Z

    hi.

GRANT|REVOKE  ALL VARIABLES IN SCHEMA schema_name [, ...] }
seems to work.
might be better to add tests.

also src/bin/psql/tab-complete.in.c
COMPLETE_WITH_SCHEMA_QUERY_PLUS(Query_for_list_of_grantables,
we can also add "ALL VARIABLES IN SCHEMA "

also need change this <para> in grant.sgml:
  <para>
   There is also an option to grant privileges on all objects of the same
   type within one or more schemas.  This functionality is currently supported
   only for tables, sequences, functions, and procedures.  <literal>ALL
   TABLES</literal> also affects views and foreign tables, just like the
   specific-object <command>GRANT</command> command.  <literal>ALL
   FUNCTIONS</literal> also affects aggregate and window functions, but not
   procedures, again just like the specific-object <command>GRANT</command>
   command.  Use <literal>ALL ROUTINES</literal> to include procedures.
  </para>

revoke.sgml, we should use <replaceable
class="parameter">role_specification</replaceable>?
so it will become like:

REVOKE [ GRANT OPTION FOR ]
    { { SELECT | UPDATE } [, ...] | ALL [ PRIVILEGES ] }
    ON { VARIABLE variable_name [, ...]
       | ALL VARIABLES IN SCHEMA schema_name [, ...] }
    FROM role_specification [, ...]

maybe also add
[ GRANTED BY role_specification ]
but I didn't test "REVOKE [ GRANTED BY role_specification ]".

Speaking of acl tests,
similar to has_table_privilege I am afraid we need to have a function
like has_variable_privilege for acl tests.
has_table_privilege has 6 function signatures. so there will be more code.
------------------------------------------------------
doc/src/sgml/ref/create_variable.sgml
<synopsis> section:
CREATE VARIABLE [ IF NOT EXISTS ] name [ AS ] data_type ] [ COLLATE collation ]

redundant right square bracket after "data_type".

  306. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-12-11T20:51:43Z

    Hi

út 10. 12. 2024 v 4:32 odesílatel jian he <jian.universality@gmail.com>
napsal:

> hi.
>
> GRANT|REVOKE  ALL VARIABLES IN SCHEMA schema_name [, ...] }
> seems to work.
> might be better to add tests.
>

done


>
> also src/bin/psql/tab-complete.in.c
> COMPLETE_WITH_SCHEMA_QUERY_PLUS(Query_for_list_of_grantables,
> we can also add "ALL VARIABLES IN SCHEMA "
>

done


>
> also need change this <para> in grant.sgml:
>   <para>
>    There is also an option to grant privileges on all objects of the same
>    type within one or more schemas.  This functionality is currently
> supported
>    only for tables, sequences, functions, and procedures.  <literal>ALL
>    TABLES</literal> also affects views and foreign tables, just like the
>    specific-object <command>GRANT</command> command.  <literal>ALL
>    FUNCTIONS</literal> also affects aggregate and window functions, but not
>    procedures, again just like the specific-object <command>GRANT</command>
>    command.  Use <literal>ALL ROUTINES</literal> to include procedures.
>   </para>
>

done


>
> revoke.sgml, we should use <replaceable
> class="parameter">role_specification</replaceable>?
> so it will become like:
>
> REVOKE [ GRANT OPTION FOR ]
>     { { SELECT | UPDATE } [, ...] | ALL [ PRIVILEGES ] }
>     ON { VARIABLE variable_name [, ...]
>        | ALL VARIABLES IN SCHEMA schema_name [, ...] }
>     FROM role_specification [, ...]
>

done


>
> maybe also add
> [ GRANTED BY role_specification ]
> but I didn't test "REVOKE [ GRANTED BY role_specification ]".
>

It i working, so I enhanced doc and regress tests


>
> Speaking of acl tests,
> similar to has_table_privilege I am afraid we need to have a function
> like has_variable_privilege for acl tests.
> has_table_privilege has 6 function signatures. so there will be more code.
>

ok, I wrote these functions





> ------------------------------------------------------
> doc/src/sgml/ref/create_variable.sgml
> <synopsis> section:
> CREATE VARIABLE [ IF NOT EXISTS ] name [ AS ] data_type ] [ COLLATE
> collation ]
>
> redundant right square bracket after "data_type".
>

fixed

Regards

Pavel

  307. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-12-14T15:40:42Z

    Hi

po 9. 12. 2024 v 17:54 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
napsal:

> Hi
>
> st 20. 11. 2024 v 21:14 odesílatel Dmitry Dolgov <9erthalion6@gmail.com>
> napsal:
>
>> > On Tue, Nov 19, 2024 at 08:14:01PM +0100, Pavel Stehule wrote:
>> > Hi
>> >
>> > I wrote POC of VARIABLE(varname) syntax support
>>
>> Thanks, the results look good. I'm still opposed the idea of having a
>> warning, and think it has to be an error -- but it's my subjective
>> opinion. Lets see if there are more votes on that topic.
>>
>
> Maybe the warning of usage of unfenced variables can be changed (enhanced)
> to some different mechanism that can be more restrictive (and safer), but I
> think it can still be user friendly.
>
> My idea is based on assumption so users with knowledge of stored
> procedures know  variables and related risks (and know tools how to
> minimize risks), and for other people the risk is higher and we should
> force usage of variable fences. I think we can force usage of variable
> fences at query runtime, when query is not executed from the SPI
> environment. This behaviour can be enabled by default, but can be
> optionally disabled.
>
> CREATE VARIABLE s.x AS int; -- allowed when user has create right on
> schema s
> CREATE VIEW v1 AS SELECT x; -- no problem, the check is dynamic
> (execution), not static
> CREATE VIEW v2 AS SELECT VARIABLE(x); -- no problem
>
> SELECT x; -- fails on access to unfenced variable
> SELECT * FROM v1; -- fails on access to unfenced variable
> SELECT * FROM v2; -- ok
>
> but inside pl, this check will not be active, and then with default
> setting I can write an code like
>
> LET var = 10; -- fencing is not allowed there, and there is not any
> collision
> DO $$
> BEGIN
>   RAISE NOTICE 'var=%', var;
>   RAISE NOTICE 'var=%', (SELECT * FROM v1); --is ok here too
> END;
> $$;
>
> Outside PL the fencing can be required, inside PL the fencing can be
> optional. With this proposal we can limit the possible risk usage of
> unfenced variables only in PL context, and the behaviour can be very
> similar to PL/SQL or SQL/PSM. This check is only a runtime check, so it has
> no impact on any DDL statement. It doesn't change the syntax or behavior,
> so it can be implemented subsequently - it is just a safeguard against
> unwanted usage of variables in an environment, where users have no
> possibility to use variables already. I can imagine that this check
> "allow_unfenced_variables" can have three values (everywhere, pl, nowhere)
> and the default can be pl. The results of queries don't depend on the
> current setting of this check. For all values for all possible queries and
> situations, the result is the same (when it is finished). But sometimes,
> the check can break the execution - in similar meaning like access rights.
> All previous proposed warnings can be unchanged.
>
>
here is a implementation with dynamic variable fence usage guard (depends
on context)

(2024-12-14 16:34:13) postgres=# set
session_variables_use_fence_context_guard to nospi ;
SET
(2024-12-14 16:34:25) postgres=# create variable xx as int;
CREATE VARIABLE
(2024-12-14 16:34:32) postgres=# select xx;
ERROR:  session variable "public.xx" is not used inside variable fence
DETAIL:  There is a risk of unwanted usage of session variable.
HINT:  Use variable fence "VARIABLE(varname) for access to variable".
(2024-12-14 16:34:38) postgres=# let xx = 20;
LET
(2024-12-14 16:34:42) postgres=# select variable(xx);
┌────┐
│ xx │
╞════╡
│ 20 │
└────┘
(1 row)

(2024-12-14 16:34:48) postgres=# do $$
postgres$# begin
postgres$#   raise notice '%', xx;
postgres$# end;
postgres$# $$;
NOTICE:  20
DO


Regards

Pavel





> Comments, notes?
>
> Regards
>
> Pavel
>
>
>
>
>

  308. Re: proposal: schema variables

    jian he <jian.universality@gmail.com> — 2024-12-18T03:00:01Z

    hi.

/*
 * has_session_variable_privilege variants
 *        These are all named "has_session_variable_privilege" at the SQL level.
 *        They take various combinations of variable name, variable OID,
 *        user name, user OID, or implicit user = current_user.
 *
 *        The result is a boolean value: true if user has the indicated
 *        privilege, false if not.  The variants that take a relation OID
 *        return NULL if the OID doesn't exist.
 */
/*
 * has_session_variable_privilege_name_name
 *        Check user privileges on a session variable given
 *        name username, text sessin variable name, and text priv name.
 */
"The variants that take a relation OID return NULL if the OID doesn't exist."
should it be
"The variants that take an OID type return NULL if the OID doesn't exist."
?

typo, "sessin" should be "session".
----------------<<<>>>>-------------------
  <sect1 id="ddl-session-variables">
   <title>Session Variables</title>
only mentioned that "Session variables themselves are persistent, but their
values are neither persistent nor shared (like the content of temporary tables).
"
I feel like this sentence is not that explicit. we actually want to say
"Once a session exits, the variable value is reset to NULL, one
session cannot see another session variable value."

+    <para>
+     A persistent database object that holds a value in session memory.  This
+     value is private to each session and is released when the session ends.
+     Read or write access to session variables is controlled by privileges,
+     similar to other database objects.
+    </para>
i do like this description in glossary.sgml.
maybe we can copy it and put it to ddl.sgml "<sect1 id="ddl-session-variables">
----------------<<<>>>>-------------------
REVOKE [ GRANT OPTION FOR ]
    { { SELECT | UPDATE } [, ...] | ALL [ PRIVILEGES ] }
    ON { VARIABLE <replaceable>variable_name</replaceable> [, ...]
       | ALL VARIABLES IN SCHEMA <replaceable
class="parameter">schema_name</replaceable> [, ...] }
    FROM { [ GROUP ] <replaceable
class="parameter">role_specification</replaceable> | PUBLIC } [, ...]
    [ GRANTED BY <replaceable
class="parameter">role_specification</replaceable> ]
    [ CASCADE | RESTRICT ]
revoke, seems still not right.
since with this, we can say:
REVOKE ALL PRIVILEGES ON VARIABLE v1 FROM group group alice CASCADE;

i think the correct one should be:
REVOKE [ GRANT OPTION FOR ]
    { { SELECT | UPDATE } [, ...] | ALL [ PRIVILEGES ] }
    ON { VARIABLE <replaceable>variable_name</replaceable> [, ...]
       | ALL VARIABLES IN SCHEMA <replaceable
class="parameter">schema_name</replaceable> [, ...] }
    FROM <replaceable class="parameter">role_specification</replaceable> [, ...]
    [ GRANTED BY <replaceable
class="parameter">role_specification</replaceable> ]
    [ CASCADE | RESTRICT ]

----------------<<<>>>>-------------------
<programlisting>
CREATE VARIABLE public.current_user_id AS integer;
GRANT READ ON VARIABLE public.current_user_id TO PUBLIC;
LET current_user_id = (SELECT id FROM users WHERE usename = session_user);
SELECT current_user_id;
</programlisting>
"GRANT READ" should be "GRANT SELECT".
----------------<<<>>>>-------------------
doc/src/sgml/ref/alter_default_privileges.sgml
GRANT { SELECT | UPDATE | ALL [ PRIVILEGES ] }
    ON VARIABLES
    TO { [ GROUP ] <replaceable
class="parameter">role_name</replaceable> | PUBLIC } [, ...] [ WITH
GRANT OPTION ]
the above part is wrong?
should be:
GRANT { { SELECT | UPDATE } [,...]
    | ALL [ PRIVILEGES ] }
    ON VARIABLES
    TO { [ GROUP ] <replaceable
class="parameter">role_name</replaceable> | PUBLIC } [, ...] [ WITH
GRANT OPTION ]

since we can:
ALTER DEFAULT PRIVILEGES
FOR ROLE alice
IN SCHEMA svartest
GRANT SELECT, UPDATE ON VARIABLES TO bob;
----------------<<<>>>>-----------------------------
CREATE VARIABLE IF NOT EXISTS v2 AS comp;
grant update on variable v2 to alice;
set role alice;
LET v2.a  = 12; --acl permission error
LET v2.b = 12; --acl permission error
LET v2 = (11,12); --ok.

not sure this is the desired behavior, for composite type variables, you are
allowed to change all the values, but you are not allowed to update the field
value of the composite.  The following are normal table test update cases.

create type comp as (a int, b  int);
create table t2(a comp);
insert into t2 select '(11,12)';
grant update (a ) on t2 to alice;
set role alice;
update t2 set a.a = 13; --ok
update t2 set a.b = 13; --ok
update t2 set a = '(11,13)'; --ok
----------------<<<>>>>-----------------------------
domain seems to have an issue.

CREATE domain d1 AS int;
CREATE VARIABLE var1 AS d1;
let var1 = 3;
--this should fail?.
alter domain d1 add check (value <> 3);
select var1;
ERROR:  value for domain d1 violates check constraint "d1_check"
----------------<<<>>>>-----------------------------
doc/src/sgml/ref/alter_variable.sgml
<title>Parameters</title> section, the order should
be: name, new_owner, new_name, new_schema?

I am beginning to look around 0002.

  309. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-12-19T07:25:51Z

    st 18. 12. 2024 v 4:00 odesílatel jian he <jian.universality@gmail.com>
napsal:

> hi.
>
> /*
>  * has_session_variable_privilege variants
>  *        These are all named "has_session_variable_privilege" at the SQL
> level.
>  *        They take various combinations of variable name, variable OID,
>  *        user name, user OID, or implicit user = current_user.
>  *
>  *        The result is a boolean value: true if user has the indicated
>  *        privilege, false if not.  The variants that take a relation OID
>  *        return NULL if the OID doesn't exist.
>  */
> /*
>  * has_session_variable_privilege_name_name
>  *        Check user privileges on a session variable given
>  *        name username, text sessin variable name, and text priv name.
>  */
> "The variants that take a relation OID return NULL if the OID doesn't
> exist."
> should it be
> "The variants that take an OID type return NULL if the OID doesn't exist."
> ?
>

yes, this comment was wrong, and I fixed it

 *<><-->The result is a boolean value: true if user has the indicated
 *<><-->privilege, false if not, or NULL if session variable doesn't
 *<><-->exists.


>
> typo, "sessin" should be "session".
>

fixed


> ----------------<<<>>>>-------------------
>   <sect1 id="ddl-session-variables">
>    <title>Session Variables</title>
> only mentioned that "Session variables themselves are persistent, but their
> values are neither persistent nor shared (like the content of temporary
> tables).
> "
> I feel like this sentence is not that explicit. we actually want to say
> "Once a session exits, the variable value is reset to NULL, one
> session cannot see another session variable value."
>

This is not fully true. I wrote new paragraph there

   <para>
    The session variable holds a value in session memory.  This value is
private
    to each session and is released when the session ends.
   </para>


>
> +    <para>
> +     A persistent database object that holds a value in session memory.
> This
> +     value is private to each session and is released when the session
> ends.
> +     Read or write access to session variables is controlled by
> privileges,
> +     similar to other database objects.
> +    </para>
> i do like this description in glossary.sgml.
> maybe we can copy it and put it to ddl.sgml "<sect1
> id="ddl-session-variables">
>

ok - I did it

----------------<<<>>>>-------------------
> REVOKE [ GRANT OPTION FOR ]
>     { { SELECT | UPDATE } [, ...] | ALL [ PRIVILEGES ] }
>     ON { VARIABLE <replaceable>variable_name</replaceable> [, ...]
>        | ALL VARIABLES IN SCHEMA <replaceable
> class="parameter">schema_name</replaceable> [, ...] }
>     FROM { [ GROUP ] <replaceable
> class="parameter">role_specification</replaceable> | PUBLIC } [, ...]
>     [ GRANTED BY <replaceable
> class="parameter">role_specification</replaceable> ]
>     [ CASCADE | RESTRICT ]
> revoke, seems still not right.
> since with this, we can say:
> REVOKE ALL PRIVILEGES ON VARIABLE v1 FROM group group alice CASCADE;
>
> i think the correct one should be:
> REVOKE [ GRANT OPTION FOR ]
>     { { SELECT | UPDATE } [, ...] | ALL [ PRIVILEGES ] }
>     ON { VARIABLE <replaceable>variable_name</replaceable> [, ...]
>        | ALL VARIABLES IN SCHEMA <replaceable
> class="parameter">schema_name</replaceable> [, ...] }
>     FROM <replaceable class="parameter">role_specification</replaceable>
> [, ...]
>     [ GRANTED BY <replaceable
> class="parameter">role_specification</replaceable> ]
>     [ CASCADE | RESTRICT ]
>

fixed


>
> ----------------<<<>>>>-------------------
> <programlisting>
> CREATE VARIABLE public.current_user_id AS integer;
> GRANT READ ON VARIABLE public.current_user_id TO PUBLIC;
> LET current_user_id = (SELECT id FROM users WHERE usename = session_user);
> SELECT current_user_id;
> </programlisting>
> "GRANT READ" should be "GRANT SELECT".
>

fixed - note it is from second patch


> ----------------<<<>>>>-------------------
> doc/src/sgml/ref/alter_default_privileges.sgml
> GRANT { SELECT | UPDATE | ALL [ PRIVILEGES ] }
>     ON VARIABLES
>     TO { [ GROUP ] <replaceable
> class="parameter">role_name</replaceable> | PUBLIC } [, ...] [ WITH
> GRANT OPTION ]
> the above part is wrong?
> should be:
> GRANT { { SELECT | UPDATE } [,...]
>     | ALL [ PRIVILEGES ] }
>     ON VARIABLES
>     TO { [ GROUP ] <replaceable
> class="parameter">role_name</replaceable> | PUBLIC } [, ...] [ WITH
> GRANT OPTION ]
>
> since we can:
> ALTER DEFAULT PRIVILEGES
> FOR ROLE alice
> IN SCHEMA svartest
> GRANT SELECT, UPDATE ON VARIABLES TO bob;
>

fixed


> ----------------<<<>>>>-----------------------------
> CREATE VARIABLE IF NOT EXISTS v2 AS comp;
> grant update on variable v2 to alice;
> set role alice;
> LET v2.a  = 12; --acl permission error
> LET v2.b = 12; --acl permission error
> LET v2 = (11,12); --ok.
>



>
> not sure this is the desired behavior, for composite type variables, you
> are
> allowed to change all the values, but you are not allowed to update the
> field
> value of the composite.  The following are normal table test update cases.
>
> create type comp as (a int, b  int);
> create table t2(a comp);
> insert into t2 select '(11,12)';
> grant update (a ) on t2 to alice;
> set role alice;
> update t2 set a.a = 13; --ok
> update t2 set a.b = 13; --ok
> update t2 set a = '(11,13)'; --ok
>

I think this is a bug, but I need more time for investigation. For field
update you need to read the content
the variable, but you are missing SELECT right on the variable, and then
the LET fails. Unfortunately
this is done inside the executor, so it is harder to fix it.



> ----------------<<<>>>>-----------------------------
> domain seems to have an issue.
>
> CREATE domain d1 AS int;
> CREATE VARIABLE var1 AS d1;
> let var1 = 3;
> --this should fail?.
> alter domain d1 add check (value <> 3);
> select var1;
> ERROR:  value for domain d1 violates check constraint "d1_check"
>

I fixed it

CREATE DOMAIN testvar_domain AS int;
CREATE VARIABLE var1 AS testvar_domain;

(2024-12-18 21:21:15) postgres=# ALTER DOMAIN testvar_domain ADD
CHECK(value <> 100);
ERROR:  cannot alter domain "testvar_domain" because session variable
"public.var1" uses it

Unfortunately I cannot force constraint check validation in other sessions,
so the most safe solution for now is
restriction of this ALTER when domain is used by some variable.  I wrote
regress tests for this.
Note: looks so validation of domain check constraints doesn't work for
temporary tables (what is expected,
not sure if it is documented).

----------------<<<>>>>-----------------------------
> doc/src/sgml/ref/alter_variable.sgml
> <title>Parameters</title> section, the order should
> be: name, new_owner, new_name, new_schema?
>

changed


>
> I am beginning to look around 0002.
>

Thank you very much

Regards

Pavel

  310. Re: Re: proposal: schema variables

    jian he <jian.universality@gmail.com> — 2024-12-20T07:57:33Z

    hi.
review is based on
v20241219-0002-Storage-for-session-variables-and-SQL-interface.patch
and
v20241219-0001-Enhancing-catalog-for-support-session-variables-and-.patch.

in doc/src/sgml/catalogs.sgml

<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>defaclobjtype</structfield> <type>char</type>
</para>
<para>
Type of object this entry is for:
<literal>r</literal> = relation (table, view),
<literal>S</literal> = sequence,
<literal>f</literal> = function,
<literal>T</literal> = type,
<literal>n</literal> = schema
</para></entry>
</row>
this need updated for session variable?

psql meta command \ddp
src/bin/psql/describe.c listDefaultACLs
also need to change.
----------------<<>>-------------------------
+-- show variable
+SELECT public.svar;
+SELECT public.svar.c;
+SELECT (public.svar).*;
+
+-- the variable is shadowed, raise error
+SELECT public.svar.c FROM public.svar;
+
+-- can be fixed by alias
+SELECT public.svar.c FROM public.svar x

The above tests are duplicated in session_variables.sql.
----------------<<>>-------------------------
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -49,7 +49,7 @@ typedef struct PlannedStmt

     NodeTag        type;

-    CmdType        commandType;    /*
select|insert|update|delete|merge|utility */
+    CmdType        commandType;    /*
select|let|insert|update|delete|merge|utility */

since we don't have CMD_LET CmdType.
so this comment change is not necessary?
----------------<<>>-------------------------
the following are simple tests that triggers makeParamSessionVariable error
messages. we don't have related tests, we can add it on session_variables.sql.

create type t1 as (a int, b int);
CREATE VARIABLE v1 text;
CREATE VARIABLE v2 as t1;
select v1.a;
select v2.c;

also
select v2.c;
ERROR:  could not identify column "c" in variable
LINE 1: select v2.c;
               ^
the error message is no good. i think we want:
ERROR:  could not identify column "c" in  variable "public.v1"
----------------<<>>-------------------------
+    /*
+     * Check for duplicates. Note that this does not really prevent
+     * duplicates, it's here just to provide nicer error message in common
+     * case. The real protection is the unique key on the catalog.
+     */
+    if (SearchSysCacheExists2(VARIABLENAMENSP,
+                              PointerGetDatum(varName),
+                              ObjectIdGetDatum(varNamespace)))
+    {
+    }
I am confused by these comments. i think the SearchSysCacheExists2
do actually prevent duplicates.
I noticed that publication_add_relation also has similar comments.
----------------<<>>-------------------------
typedef struct LetStmt
{
    NodeTag        type;
    List       *target;            /* target variable */
    Node       *query;            /* source expression */
    int            location;
} LetStmt;
here, location should be a type of ParseLoc.
----------------<<>>-------------------------
in 0001, 0002, function SetSessionVariableWithSecurityCheck
never being used.
----------------<<>>-------------------------
+/*
+ * transformLetStmt -
+ *      transform an Let Statement
+ */
+static Query *
+transformLetStmt(ParseState *pstate, LetStmt *stmt)
...
+    /*
+     * Save target session variable ID.  This value is used multiple times: by
+     * the query rewriter (for getting related defexpr), by planner (for
+     * acquiring an AccessShareLock on target variable), and by the executor
+     * (we need to know the target variable ID).
+     */
+    query->resultVariable = varid;

"defexpr", do you mean "default expression"?
Generally letsmt is like: "let variable = (SelectStmt)"
is there nothing related to the DEFAULT expression?
"(we need to know the target variable ID)." in ExecuteLetStmt that is true.
but I commented out the following lines, the regress test still
succeeded.

extract_query_dependencies_walker
/* record dependency on the target variable of a LET command */
// if (OidIsValid(query->resultVariable))
//     record_plan_variable_dependency(context, query->resultVariable);

ScanQueryForLocks
// /* process session variables */
// if (OidIsValid(parsetree->resultVariable))
// {
//     if (acquire)
//         LockDatabaseObject(VariableRelationId, parsetree->resultVariable,
//                            0, AccessShareLock);
//     else
//         UnlockDatabaseObject(VariableRelationId, parsetree->resultVariable,
//                              0, AccessShareLock);
// }
----------------<<>>-------------------------
in transformLetStmt, we already did ACL privilege check,
we don't need do it again at ExecuteLetStmt?

  311. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-12-20T12:53:20Z

    Hi


----------------<<<>>>>-----------------------------
>> CREATE VARIABLE IF NOT EXISTS v2 AS comp;
>> grant update on variable v2 to alice;
>> set role alice;
>> LET v2.a  = 12; --acl permission error
>> LET v2.b = 12; --acl permission error
>> LET v2 = (11,12); --ok.
>>
>
>
>
>>
>> not sure this is the desired behavior, for composite type variables, you
>> are
>> allowed to change all the values, but you are not allowed to update the
>> field
>> value of the composite.  The following are normal table test update cases.
>>
>> create type comp as (a int, b  int);
>> create table t2(a comp);
>> insert into t2 select '(11,12)';
>> grant update (a ) on t2 to alice;
>> set role alice;
>> update t2 set a.a = 13; --ok
>> update t2 set a.b = 13; --ok
>> update t2 set a = '(11,13)'; --ok
>>
>
> I think this is a bug, but I need more time for investigation. For field
> update you need to read the content
> the variable, but you are missing SELECT right on the variable, and then
> the LET fails. Unfortunately
> this is done inside the executor, so it is harder to fix it.
>
>
I fixed this issue - the change is done in Patch 02

Reards

Pavel



>
>

  312. Re: Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-12-20T22:00:12Z

    Hi

pá 20. 12. 2024 v 8:58 odesílatel jian he <jian.universality@gmail.com>
napsal:

> hi.
> review is based on
> v20241219-0002-Storage-for-session-variables-and-SQL-interface.patch
> and
> v20241219-0001-Enhancing-catalog-for-support-session-variables-and-.patch.
>
> in doc/src/sgml/catalogs.sgml
>
> <row>
> <entry role="catalog_table_entry"><para role="column_definition">
> <structfield>defaclobjtype</structfield> <type>char</type>
> </para>
> <para>
> Type of object this entry is for:
> <literal>r</literal> = relation (table, view),
> <literal>S</literal> = sequence,
> <literal>f</literal> = function,
> <literal>T</literal> = type,
> <literal>n</literal> = schema
> </para></entry>
> </row>
> this need updated for session variable?
>

yes, fixed


>
> psql meta command \ddp
> src/bin/psql/describe.c listDefaultACLs
> also need to change.
>

fixed


> ----------------<<>>-------------------------
> +-- show variable
> +SELECT public.svar;
> +SELECT public.svar.c;
> +SELECT (public.svar).*;
> +
> +-- the variable is shadowed, raise error
> +SELECT public.svar.c FROM public.svar;
> +
> +-- can be fixed by alias
> +SELECT public.svar.c FROM public.svar x
>
> The above tests are duplicated in session_variables.sql.
>

It looks like duplicated test, but it isn't - look on session_variables.out

+-- the variable is shadowed, raise error
+SELECT public.svar.c FROM public.svar;
+ERROR:  column svar.c does not exist
+LINE 1: SELECT public.svar.c FROM public.svar;
+               ^
+-- can be fixed by alias
+SELECT public.svar.c FROM public.svar x;
+  c..
+-----
+ 300
+(1 row)



> ----------------<<>>-------------------------
> --- a/src/include/nodes/plannodes.h
> +++ b/src/include/nodes/plannodes.h
> @@ -49,7 +49,7 @@ typedef struct PlannedStmt
>
>      NodeTag        type;
>
> -    CmdType        commandType;    /*
> select|insert|update|delete|merge|utility */
> +    CmdType        commandType;    /*
> select|let|insert|update|delete|merge|utility */
>
> since we don't have CMD_LET CmdType.
> so this comment change is not necessary?
>

true, removed


> ----------------<<>>-------------------------
> the following are simple tests that triggers makeParamSessionVariable error
> messages. we don't have related tests, we can add it on
> session_variables.sql.
>
> create type t1 as (a int, b int);
> CREATE VARIABLE v1 text;
> CREATE VARIABLE v2 as t1;
> select v1.a;
> select v2.c;
>
>
done


> also
> select v2.c;
> ERROR:  could not identify column "c" in variable
> LINE 1: select v2.c;
>                ^
> the error message is no good. i think we want:
> ERROR:  could not identify column "c" in  variable "public.v1"
>

done

----------------<<>>-------------------------
> +    /*
> +     * Check for duplicates. Note that this does not really prevent
> +     * duplicates, it's here just to provide nicer error message in common
> +     * case. The real protection is the unique key on the catalog.
> +     */
> +    if (SearchSysCacheExists2(VARIABLENAMENSP,
> +                              PointerGetDatum(varName),
> +                              ObjectIdGetDatum(varNamespace)))
> +    {
> +    }
> I am confused by these comments. i think the SearchSysCacheExists2
> do actually prevent duplicates.
> I noticed that publication_add_relation also has similar comments.
>

RowExclusiveLock doesn't block concurrent inserts. So theoretically, two
sessions can try
to create variables with the same name and same schema. Using syscache
cache cannot
protect against this scenario. The real protection is only a unique index.
It is same like

IF NOT EXISTS(SELECT * FROM foo WHERE id = 10) THEN
  INSERT INTO foo(id) VALUES(10)
END IF;

This fragment doesn't protect us against duplicit id in table foo. The real
protection can
be done only by unique index, but when you use this fragment, then you can
reduce
error messages like `unique index violation` and that can benefit in some
scenarios.

I think this comment is correct, but enhancig is welcome.



> ----------------<<>>-------------------------
> typedef struct LetStmt
> {
>     NodeTag        type;
>     List       *target;            /* target variable */
>     Node       *query;            /* source expression */
>     int            location;
> } LetStmt;
> here, location should be a type of ParseLoc.
>

changed


> ----------------<<>>-------------------------
> in 0001, 0002, function SetSessionVariableWithSecurityCheck
> never being used.
>

moved to patch 18 plpgsql-implementation-for-LET-statement

> ----------------<<>>-------------------------
> +/*
> + * transformLetStmt -
> + *      transform an Let Statement
> + */
> +static Query *
> +transformLetStmt(ParseState *pstate, LetStmt *stmt)
> ...
> +    /*
> +     * Save target session variable ID.  This value is used multiple
> times: by
> +     * the query rewriter (for getting related defexpr), by planner (for
> +     * acquiring an AccessShareLock on target variable), and by the
> executor
> +     * (we need to know the target variable ID).
> +     */
> +    query->resultVariable = varid;
>
> "defexpr", do you mean "default expression"?
> Generally letsmt is like: "let variable = (SelectStmt)"
> is there nothing related to the DEFAULT expression?
> "(we need to know the target variable ID)." in ExecuteLetStmt that is true.
> but I commented out the following lines, the regress test still
> succeeded.
>

This comment is partially obsolete - early implementations of LET
statements supports syntax LET var = DEFAULT.
I removed this feature, because the implementation was not trivial and the
benefit is not too high.

I changed this comment to

<-->/*
<--> * Save target session variable ID. It is used later for
<--> * acquiring an AccessShareLock on target variable, setting
<--> * plan dependency and finally for creating VariableDestReceiver.
<--> */



>
> extract_query_dependencies_walker
> /* record dependency on the target variable of a LET command */
> // if (OidIsValid(query->resultVariable))
> //     record_plan_variable_dependency(context, query->resultVariable);
>

I checked regress tests, and plan invalidation is tested there, but not for
target variable.
I enhanced this test to check the invalidation of plans when dropped
variable is used
as target.

SET plan_cache_mode TO force_generic_plan;

LET var1 = 3.14;
SELECT plpgsqlfx();
LET var1 = 3.14 * 2;
SELECT plpgsqlfx();

SELECT plpgsqlfx2(10.0);
SELECT var2;

DROP VARIABLE var1;
DROP VARIABLE var2;

-- dependency (plan invalidation) should work
CREATE VARIABLE var1 AS numeric;
CREATE VARIABLE var2 AS numeric;

LET var1 = 3.14 * 3;
SELECT plpgsqlfx();
LET var1 = 3.14 * 4;
SELECT plpgsqlfx();

SELECT plpgsqlfx2(10.0);
SELECT var2;

DROP VARIABLE var1;
DROP VARIABLE var2;
DROP FUNCTION plpgsqlfx();
DROP FUNCTION plpgsqlfx2();

SET plan_cache_mode TO DEFAULT;





>
> ScanQueryForLocks
> // /* process session variables */
> // if (OidIsValid(parsetree->resultVariable))
> // {
> //     if (acquire)
> //         LockDatabaseObject(VariableRelationId,
> parsetree->resultVariable,
> //                            0, AccessShareLock);
> //     else
> //         UnlockDatabaseObject(VariableRelationId,
> parsetree->resultVariable,
> //                              0, AccessShareLock);
> // }
>

This is protection against dropping an session variable when it is used. I
think so this is tested by isolation tester
in Patch 05




> ----------------<<>>-------------------------
> in transformLetStmt, we already did ACL privilege check,
> we don't need do it again at ExecuteLetStmt?
>

It is redundant, but surely it should be checked in the executor stage. I
removed this check from transformLetStmt.

Regards

Pavel

  313. Re: Re: proposal: schema variables

    jian he <jian.universality@gmail.com> — 2024-12-27T15:19:29Z

    hi.

+ if (!OidIsValid(varid))
+ AcceptInvalidationMessages();
+ else if (OidIsValid(varid))
+ LockDatabaseObject(VariableRelationId, varid, 0, AccessShareLock);

we can change it to
+ if (!OidIsValid(varid))
+ AcceptInvalidationMessages();
+ else
+ LockDatabaseObject(VariableRelationId, varid, 0, AccessShareLock);

inval_count didn't explain at all, other places did actually explain it.
Can we add some text explaining inval_count? (i didn't fully understand
this part, that is why i am asking..)

seems IdentifyVariable all these three ereport(ERROR...) don't have
regress tests,
i think we should have it. Am I missing something?

create variable v2 as int;
let v2.a = 1;
ERROR:  type "integer" of target session variable "public.v2" is not a
composite type
LINE 1: let v2.a = 1;
            ^
the error messages look weird.
IMO, it should either be
"type of session variable "public.v2" is not a composite type"
or
"session variable "public.v2" don't have attribute "a"

also, the above can be as a regress test for:
+ if (attrname && !is_rowtype)
+ ereport(ERROR,
+ (errcode(ERRCODE_WRONG_OBJECT_TYPE),
+ errmsg("type \"%s\" of target session variable \"%s.%s\" is not a
composite type",
+ format_type_be(typid),
+ get_namespace_name(get_session_variable_namespace(varid)),
+ get_session_variable_name(varid)),
+ parser_errposition(pstate, stmt->location)));
since we don't have coverage tests for it.


+ if (coerced_expr == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("variable \"%s.%s\" is of type %s,"
+ " but expression is of type %s",
+ get_namespace_name(get_session_variable_namespace(varid)),
+ get_session_variable_name(varid),
+ format_type_be(typid),
+ format_type_be(exprtypid)),
+ errhint("You will need to rewrite or cast the expression."),
+ parser_errposition(pstate, exprLocation((Node *) expr))));

generally, errmsg(...) should put it into one line for better grep-ability
so we can change it to:
+errmsg("variable \"%s.%s\" is of type %s, but expression is of type %s"...)

also no coverage tests?
simple test case for it:
create variable v2 as int;
let v2 = '1'::jsonb;

---------------<<<>>>--------------
+let_target:
+ ColId opt_indirection
+ {
+ $$ = list_make1(makeString($1));
+ if ($2)
+  $$ = list_concat($$,
+   check_indirection($2, yyscanner));
+ }
if you look closely, it seems the indentation level is wrong in
line "$$ = list_concat($$,"?

---------------<<<>>>--------------
i did some refactoring in session_variables.sql for privilege check.
make the tests more neat, please check attached.

  314. Re: Re: proposal: schema variables

    jian he <jian.universality@gmail.com> — 2024-12-28T10:34:27Z

    hi.

src9=# select 'XLogRecPtr'::regtype;
ERROR:  type "xlogrecptr" does not exist
LINE 1: select 'XLogRecPtr'::regtype;
               ^
so
+ <structfield>varcreate_lsn</structfield> <type>XLogRecPtr</type>
should be
<structfield>varcreate_lsn</structfield> <type>pg_lsn</type>
?

also
+     <row>
+      <entry role="catalog_table_entry"><para role="column_definition">
+       <structfield>varcreate_lsn</structfield> <type>XLogRecPtr</type>
+      </para>
+      <para>
+       LSN of the transaction where the variable was created.
+       <structfield>varcreate_lsn</structfield> and
+       <structfield>oid</structfield> together form the all-time unique
+       identifier (<structfield>oid</structfield> alone is not enough, since
+       object identifiers can get reused).
+      </para></entry>
+     </row>
+
we have "pg_variable_oid_index" PRIMARY KEY, btree (oid)
for table pg_variable.
so I am confused by saying the column "oid" itself is not enough to
prove unique.

in let.sgml
<term><literal>session_variable</literal></term>
should be
<term><replaceable class="parameter">session_variable</replaceable></term>

<term><literal>sql_expression</literal></term>
should be
<term><replaceable class="parameter">sql_expression</replaceable></term>

  315. Re: Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-12-28T15:32:17Z

    Hi

pá 27. 12. 2024 v 16:20 odesílatel jian he <jian.universality@gmail.com>
napsal:

> hi.
>
> + if (!OidIsValid(varid))
> + AcceptInvalidationMessages();
> + else if (OidIsValid(varid))
> + LockDatabaseObject(VariableRelationId, varid, 0, AccessShareLock);
>
> we can change it to
> + if (!OidIsValid(varid))
> + AcceptInvalidationMessages();
> + else
> + LockDatabaseObject(VariableRelationId, varid, 0, AccessShareLock);
>

done


>
> inval_count didn't explain at all, other places did actually explain it.
> Can we add some text explaining inval_count? (i didn't fully understand
> this part, that is why i am asking..)
>

done


>
> seems IdentifyVariable all these three ereport(ERROR...) don't have
> regress tests,
> i think we should have it. Am I missing something?
>

done

>
> create variable v2 as int;
> let v2.a = 1;
> ERROR:  type "integer" of target session variable "public.v2" is not a
> composite type
> LINE 1: let v2.a = 1;
>             ^
> the error messages look weird.
> IMO, it should either be
> "type of session variable "public.v2" is not a composite type"
> or
> "session variable "public.v2" don't have attribute "a"
>

changed

I did inspiration from transformAssignmentIndirection now

(2024-12-28 16:07:57) postgres=# let x.a = 20;
ERROR:  cannot assign to field "a" of session variable "public.x" because
its type integer is not a composite type
LINE 1: let x.a = 20;
            ^

>
> also, the above can be as a regress test for:
> + if (attrname && !is_rowtype)
> + ereport(ERROR,
> + (errcode(ERRCODE_WRONG_OBJECT_TYPE),
> + errmsg("type \"%s\" of target session variable \"%s.%s\" is not a
> composite type",
> + format_type_be(typid),
> + get_namespace_name(get_session_variable_namespace(varid)),
> + get_session_variable_name(varid)),
> + parser_errposition(pstate, stmt->location)));
> since we don't have coverage tests for it.
>
>
done


>
> + if (coerced_expr == NULL)
> + ereport(ERROR,
> + (errcode(ERRCODE_DATATYPE_MISMATCH),
> + errmsg("variable \"%s.%s\" is of type %s,"
> + " but expression is of type %s",
> + get_namespace_name(get_session_variable_namespace(varid)),
> + get_session_variable_name(varid),
> + format_type_be(typid),
> + format_type_be(exprtypid)),
> + errhint("You will need to rewrite or cast the expression."),
> + parser_errposition(pstate, exprLocation((Node *) expr))));
>
> generally, errmsg(...) should put it into one line for better grep-ability
> so we can change it to:
> +errmsg("variable \"%s.%s\" is of type %s, but expression is of type
> %s"...)
>

done


>
> also no coverage tests?
> simple test case for it:
> create variable v2 as int;
> let v2 = '1'::jsonb;
>

done


>
> ---------------<<<>>>--------------
> +let_target:
> + ColId opt_indirection
> + {
> + $$ = list_make1(makeString($1));
> + if ($2)
> +  $$ = list_concat($$,
> +   check_indirection($2, yyscanner));
> + }
> if you look closely, it seems the indentation level is wrong in
> line "$$ = list_concat($$,"?
>

let_target is not needed as separate rule now, so I removed it and little
bit cleaned (really only little bit)


>
> ---------------<<<>>>--------------
> i did some refactoring in session_variables.sql for privilege check.
> make the tests more neat, please check attached.
>

merged with minor changes in comment's formatting

I'll send the patch set with next mail

Best regards

Pavel

  316. Re: Re: proposal: schema variables

    jian he <jian.universality@gmail.com> — 2024-12-28T16:46:28Z

    hi.

+ if (stmt->collClause)
+ collation = LookupCollation(pstate,
+ stmt->collClause->collname,
+ stmt->collClause->location);
+ else
+ collation = typcollation;;

two semi-colon. should be only one.
------------------<<>>>---------------
+ /* locks the variable with an AccessShareLock */
+ varid = IdentifyVariable(names, &attrname, &not_unique, false);
+ if (not_unique)
+ ereport(ERROR,
+ (errcode(ERRCODE_AMBIGUOUS_PARAMETER),
+ errmsg("target \"%s\" of LET command is ambiguous",
+ NameListToString(names)),
+ parser_errposition(pstate, stmt->location)));

the following are tests for the above "LET command is ambiguous" error message.

create schema test;
CREATE TYPE test AS (test int);
CREATE variable test.test as test;
set search_path to test;
let test.test = 1;
------------------<<>>>---------------
+ else
+ {
+ /* the last field of list can be star too */
+ Assert(IsA(field2, A_Star));
+
+ /*
+ * In this case, the field1 should be variable name. But
+ * direct unboxing of composite session variables is not
+ * supported now, and then we don't need to try lookup
+ * related variable.
+ *
+ * Unboxing is supported by syntax (var).*
+ */
+ return InvalidOid;
+ }
I don't fully understand the above comments,
add
`elog(INFO, "%s:%d called", __FILE__, __LINE__); ` within the ELSE branch.
Then I found out the ELSE branch doesn't have coverage tests.

------------------<<>>>---------------
+ /*
+ * a.b.c can mean catalog.schema.variable or
+ * schema.variable.field.
....
+ /*
+ * a.b can mean "schema"."variable" or "variable"."field".
+ * Check both variants, and returns InvalidOid with
+ * not_unique flag, when both interpretations are
+ * possible.
+ */
here, we use the word "field", but the function IdentifyVariable above
comment, we use
word "attribute", for consistency's sake, we should use "attribute"
instead of "field"

+/* -----
+ * IdentifyVariable - try to find a variable from a list of identifiers
+ *
+ * Returns the OID of the variable found, or InvalidOid.
+ *
+ * "names" is a list of up to four identifiers; possible meanings are:
+ * - variable  (searched on the search_path)
+ * - schema.variable
+ * - variable.attribute  (searched on the search_path)
+ * - schema.variable.attribute
+ * - database.schema.variable
+ * - database.schema.variable.attribute

------------------<<>>>---------------

  317. Re: Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-12-28T17:29:26Z

    Hi

so 28. 12. 2024 v 11:35 odesílatel jian he <jian.universality@gmail.com>
napsal:

> hi.
>
> src9=# select 'XLogRecPtr'::regtype;
> ERROR:  type "xlogrecptr" does not exist
> LINE 1: select 'XLogRecPtr'::regtype;
>                ^
> so
> + <structfield>varcreate_lsn</structfield> <type>XLogRecPtr</type>
> should be
> <structfield>varcreate_lsn</structfield> <type>pg_lsn</type>
> ?
>

done


>
> also
> +     <row>
> +      <entry role="catalog_table_entry"><para role="column_definition">
> +       <structfield>varcreate_lsn</structfield> <type>XLogRecPtr</type>
> +      </para>
> +      <para>
> +       LSN of the transaction where the variable was created.
> +       <structfield>varcreate_lsn</structfield> and
> +       <structfield>oid</structfield> together form the all-time unique
> +       identifier (<structfield>oid</structfield> alone is not enough,
> since
> +       object identifiers can get reused).
> +      </para></entry>
> +     </row>
> +
> we have "pg_variable_oid_index" PRIMARY KEY, btree (oid)
> for table pg_variable.
> so I am confused by saying the column "oid" itself is not enough to
> prove unique.
>

The session variable is stored in memory until the end of the session.
Theoretically, some sessions with used session variables can be opened for
a very long time without any activity - so it is not possible to process
sinval message. Other sessions can drop and create a lot of session
variables (this is very possible with temporary session variables). Oid in
Postgres can overflow, and postgres can reuse used oid of dropped objects
(oid is only 32bit integer). And after some time, the session with the used
variable can be activated, and the session variable can be used. Before
usage the session variable is rechecked against pg_variable, and
theoretically the variable with the same oid can be there (although it is a
different variable with possibly different type). The implementation should
protect against this scenario. The stored value must not be used in this
case - the usage of old value is dangerous - the calculations with the
variable can lose sense or can crash postgres. LSN is forever unique - it
is 64bit integer - so it is our safeguard so we can detect obsolete values
(stored in memory) although there are variables with the same oid.

oid in pg_variable ensures a unique identifier for any session variable in
one moment. Compound key [oid, varcreate_lsn] is a unique identifier for
ever.



> in let.sgml
> <term><literal>session_variable</literal></term>
> should be
> <term><replaceable class="parameter">session_variable</replaceable></term>
>

done


>
> <term><literal>sql_expression</literal></term>
> should be
> <term><replaceable class="parameter">sql_expression</replaceable></term>
>

done

  318. Re: Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-12-28T21:49:59Z

    Hi

so 28. 12. 2024 v 17:47 odesílatel jian he <jian.universality@gmail.com>
napsal:

> hi.
>
> + if (stmt->collClause)
> + collation = LookupCollation(pstate,
> + stmt->collClause->collname,
> + stmt->collClause->location);
> + else
> + collation = typcollation;;
>
> two semi-colon. should be only one.
>

removed


> ------------------<<>>>---------------
> + /* locks the variable with an AccessShareLock */
> + varid = IdentifyVariable(names, &attrname, &not_unique, false);
> + if (not_unique)
> + ereport(ERROR,
> + (errcode(ERRCODE_AMBIGUOUS_PARAMETER),
> + errmsg("target \"%s\" of LET command is ambiguous",
> + NameListToString(names)),
> + parser_errposition(pstate, stmt->location)));
>
> the following are tests for the above "LET command is ambiguous" error
> message.
>
> create schema test;
> CREATE TYPE test AS (test int);
> CREATE variable test.test as test;
> set search_path to test;
> let test.test = 1;
>

 done

------------------<<>>>---------------
> + else
> + {
> + /* the last field of list can be star too */
> + Assert(IsA(field2, A_Star));
> +
> + /*
> + * In this case, the field1 should be variable name. But
> + * direct unboxing of composite session variables is not
> + * supported now, and then we don't need to try lookup
> + * related variable.
> + *
> + * Unboxing is supported by syntax (var).*
> + */
> + return InvalidOid;
> + }
> I don't fully understand the above comments,
>

The parser allows only two syntaxes - identifier.identifier or
identifier.star. Second
syntax is not supported by session variables, and then I didn't try to
search for the variable.
Some users can be confused by similar syntaxes identifier.* and
(identifier).* Only
second syntax is composite unboxing, and only second syntax is supported for
session variables.

Maybe the note about unboxing is messy there?

add
> `elog(INFO, "%s:%d called", __FILE__, __LINE__); ` within the ELSE branch.
> Then I found out the ELSE branch doesn't have coverage tests.
>

I don't understand this comment? I don't use elog(INFO anywhere



>
> ------------------<<>>>---------------
> + /*
> + * a.b.c can mean catalog.schema.variable or
> + * schema.variable.field.
> ....
> + /*
> + * a.b can mean "schema"."variable" or "variable"."field".
> + * Check both variants, and returns InvalidOid with
> + * not_unique flag, when both interpretations are
> + * possible.
> + */
> here, we use the word "field", but the function IdentifyVariable above
> comment, we use
> word "attribute", for consistency's sake, we should use "attribute"
> instead of "field"
>

done


>
> +/* -----
> + * IdentifyVariable - try to find a variable from a list of identifiers
> + *
> + * Returns the OID of the variable found, or InvalidOid.
> + *
> + * "names" is a list of up to four identifiers; possible meanings are:
> + * - variable  (searched on the search_path)
> + * - schema.variable
> + * - variable.attribute  (searched on the search_path)
> + * - schema.variable.attribute
> + * - database.schema.variable
> + * - database.schema.variable.attribute
>
> ------------------<<>>>---------------
>

updated patchset assigned

Thank you very much for review

Regards

Pavel

  319. Re: Re: proposal: schema variables

    jian he <jian.universality@gmail.com> — 2024-12-29T02:48:51Z

    On Sun, Dec 29, 2024 at 5:50 AM Pavel Stehule <pavel.stehule@gmail.com> wrote:
>
> Hi
>
>
>> ------------------<<>>>---------------
>> + else
>> + {
>> + /* the last field of list can be star too */
>> + Assert(IsA(field2, A_Star));
>> +
>> + /*
>> + * In this case, the field1 should be variable name. But
>> + * direct unboxing of composite session variables is not
>> + * supported now, and then we don't need to try lookup
>> + * related variable.
>> + *
>> + * Unboxing is supported by syntax (var).*
>> + */
>> + return InvalidOid;
>> + }
>> I don't fully understand the above comments,
>
>
> The parser allows only two syntaxes - identifier.identifier or identifier.star. Second
> syntax is not supported by session variables, and then I didn't try to search for the variable.
> Some users can be confused by similar syntaxes identifier.* and (identifier).* Only
> second syntax is composite unboxing, and only second syntax is supported for
> session variables.
>
> Maybe the note about unboxing is messy there?
>
>> add
>> `elog(INFO, "%s:%d called", __FILE__, __LINE__); ` within the ELSE branch.
>> Then I found out the ELSE branch doesn't have coverage tests.
>
>
> I don't understand this comment? I don't use elog(INFO anywhere
>
>

sorry for confusion, i mean,
i added " elog(INFO", the regress test is still successful,
therefore it means the above ELSE branch code doesn't have coverage tests.

  320. Re: Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-12-29T06:24:55Z

    Hi


>> + {
>> + /* the last field of list can be star too */
>> + Assert(IsA(field2, A_Star));
>> +
>> + /*
>> + * In this case, the field1 should be variable name. But
>> + * direct unboxing of composite session variables is not
>> + * supported now, and then we don't need to try lookup
>> + * related variable.
>> + *
>> + * Unboxing is supported by syntax (var).*
>> + */
>> + return InvalidOid;
>> + }
>> I don't fully understand the above comments,
>>
>
> The parser allows only two syntaxes - identifier.identifier or
> identifier.star. Second
> syntax is not supported by session variables, and then I didn't try to
> search for the variable.
> Some users can be confused by similar syntaxes identifier.* and
> (identifier).* Only
> second syntax is composite unboxing, and only second syntax is supported
> for
> session variables.
>
>
I changed this comment to

<--><--><--><--><-->/*
<--><--><--><--><--> * The syntax ident.* is used only by table aliases,
<--><--><--><--><--> * and then this identifier cannot be a reference to
<--><--><--><--><--> * session variable.
<--><--><--><--><--> */

  321. Re: Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2024-12-29T08:42:11Z

    Hi

ne 29. 12. 2024 v 3:49 odesílatel jian he <jian.universality@gmail.com>
napsal:

> On Sun, Dec 29, 2024 at 5:50 AM Pavel Stehule <pavel.stehule@gmail.com>
> wrote:
> >
> > Hi
> >
> >
> >> ------------------<<>>>---------------
> >> + else
> >> + {
> >> + /* the last field of list can be star too */
> >> + Assert(IsA(field2, A_Star));
> >> +
> >> + /*
> >> + * In this case, the field1 should be variable name. But
> >> + * direct unboxing of composite session variables is not
> >> + * supported now, and then we don't need to try lookup
> >> + * related variable.
> >> + *
> >> + * Unboxing is supported by syntax (var).*
> >> + */
> >> + return InvalidOid;
> >> + }
> >> I don't fully understand the above comments,
> >
> >
> > The parser allows only two syntaxes - identifier.identifier or
> identifier.star. Second
> > syntax is not supported by session variables, and then I didn't try to
> search for the variable.
> > Some users can be confused by similar syntaxes identifier.* and
> (identifier).* Only
> > second syntax is composite unboxing, and only second syntax is supported
> for
> > session variables.
> >
> > Maybe the note about unboxing is messy there?
> >
> >> add
> >> `elog(INFO, "%s:%d called", __FILE__, __LINE__); ` within the ELSE
> branch.
> >> Then I found out the ELSE branch doesn't have coverage tests.
> >
> >
> > I don't understand this comment? I don't use elog(INFO anywhere
> >
> >
>
> sorry for confusion, i mean,
> i added " elog(INFO", the regress test is still successful,
> therefore it means the above ELSE branch code doesn't have coverage tests.
>

yes, force this case can be little bit tricky

a) the variable should not be shadowed,
or session_variables_ambiguity_warning should be on
b) the transformColumnRef should be executed and the star symbol should be
in the list - it is possible for aggregate functions

SELECT count(v.*) FROM foo

I wrote requested regress tests

Regards

Pavel

  322. Re: Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-01-02T06:46:06Z

    Hi

fresh rebase + update year in copyright lines

Regards

Pavel

  323. Re: Re: proposal: schema variables

    jian he <jian.universality@gmail.com> — 2025-01-03T07:18:22Z

    hi.

in the function svariableStartupReceiver all these "ereport(ERROR"
cannot happen,
since transformLetStmt already did all the heavy work.
base on https://www.postgresql.org/docs/current/error-message-reporting.html
all these "ereport(ERROR," in the svariableStartupReceiver can be
simplified as "elog(ERROR,"
or Assert.


After standard_ExecutorStart->InitPlan, queryDesc.tupDesc will not
include attr->attisdropped is true scarenio.
In standard_ExecutorStart, I added the following code then ran the
regress test again to prove my point.

standard_ExecutorStart
    /*
     * Initialize the plan state tree
     */
    InitPlan(queryDesc, eflags);
    for (int i = 0; i < queryDesc->tupDesc->natts; i++)
    {
        Form_pg_attribute attr = TupleDescAttr(queryDesc->tupDesc, i);
        if (attr->attisdropped)
        {
            elog(INFO, "some attribute is dropped queryDesc->operation
is %d", queryDesc->operation);
        }
    }
    MemoryContextSwitchTo(oldcontext);
-------------------------
svariableStartupReceiver parameter typeinfo is from queryDesc->tupDesc
So I think svariableStartupReceiver, typeinfo->natts will always equal one.
therefore SVariableState.slot_offset is not necessary.

overall, i think svariableStartupReceiver can be simplified as the following:

static void
svariableStartupReceiver(DestReceiver *self, int operation, TupleDesc typeinfo)
{
    SVariableState *myState = (SVariableState *) self;
    int            natts = typeinfo->natts;
    Form_pg_attribute attr;
    LOCKTAG        locktag PG_USED_FOR_ASSERTS_ONLY;
    Assert(myState->pub.mydest == DestVariable);
    Assert(OidIsValid(myState->varid));
    Assert(SearchSysCacheExists1(VARIABLEOID, myState->varid));
#ifdef USE_ASSERT_CHECKING
    SET_LOCKTAG_OBJECT(locktag,
                       MyDatabaseId,
                       VariableRelationId,
                       myState->varid,
                       0);
    Assert(LockHeldByMe(&locktag, AccessShareLock, false));
#endif
    Assert(natts == 1);
    attr = TupleDescAttr(typeinfo, 0);
    myState->need_detoast = attr->attlen == -1;
    myState->rows = 0;
}

I've attached the file containing the changes I mentioned earlier.

-------------------------<<>>>-------------------------------
Overall, 0001 and 0002 the doc looks good to me now.
The following are only some minor issues I came up with.

In Table 5.1. ACL Privilege Abbreviations
<table id="privilege-abbrevs-table">
<title><acronym>ACL</acronym> Privilege Abbreviations</title>

<literal>VARIABLE</literal> (occurred 3 times)
should be
<literal>SESSION VARIABLE</literal>
?

doc/src/sgml/glossary.sgml
I want to do minor tweak. from
    <para>
     A persistent database object that holds a value in session memory.  This
     value is private to each session and is released when the session ends.
     Read or write access to session variables is controlled by privileges,
     similar to other database objects.
    </para>
to
    <para>
     A persistent database object that holds a value in session memory.  This
     value is private to each session and is reset to default value
(null) when the session ends.
     Read or write access to session variables is controlled by access
privileges,
     similar to other database objects.
    </para>

in let.sgml.
  <para>
   Example:
<programlisting>
CREATE VARIABLE myvar AS integer;
LET myvar = 10;
LET myvar = (SELECT sum(val) FROM tab);
</programlisting>
  </para>

it should be
 <refsect1>
  <title>Examples</title>
...your example code
 </refsect1>

  324. Re: Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-01-03T22:59:30Z

    Hi

pá 3. 1. 2025 v 8:18 odesílatel jian he <jian.universality@gmail.com>
napsal:

> hi.
>
> in the function svariableStartupReceiver all these "ereport(ERROR"
> cannot happen,
> since transformLetStmt already did all the heavy work.
> base on
> https://www.postgresql.org/docs/current/error-message-reporting.html
> all these "ereport(ERROR," in the svariableStartupReceiver can be
> simplified as "elog(ERROR,"
> or Assert.
>
>
> After standard_ExecutorStart->InitPlan, queryDesc.tupDesc will not
> include attr->attisdropped is true scarenio.
> In standard_ExecutorStart, I added the following code then ran the
> regress test again to prove my point.
>
> standard_ExecutorStart
>     /*
>      * Initialize the plan state tree
>      */
>     InitPlan(queryDesc, eflags);
>     for (int i = 0; i < queryDesc->tupDesc->natts; i++)
>     {
>         Form_pg_attribute attr = TupleDescAttr(queryDesc->tupDesc, i);
>         if (attr->attisdropped)
>         {
>             elog(INFO, "some attribute is dropped queryDesc->operation
> is %d", queryDesc->operation);
>         }
>     }
>     MemoryContextSwitchTo(oldcontext);
> -------------------------
> svariableStartupReceiver parameter typeinfo is from queryDesc->tupDesc
> So I think svariableStartupReceiver, typeinfo->natts will always equal one.
> therefore SVariableState.slot_offset is not necessary.
>

It is true. The syntax allows just `expression` or `(subquery)`, and  all
dirty work does execution of subquery

>
> overall, i think svariableStartupReceiver can be simplified as the
> following:
>
> static void
> svariableStartupReceiver(DestReceiver *self, int operation, TupleDesc
> typeinfo)
> {
>     SVariableState *myState = (SVariableState *) self;
>     int            natts = typeinfo->natts;
>     Form_pg_attribute attr;
>     LOCKTAG        locktag PG_USED_FOR_ASSERTS_ONLY;
>     Assert(myState->pub.mydest == DestVariable);
>     Assert(OidIsValid(myState->varid));
>     Assert(SearchSysCacheExists1(VARIABLEOID, myState->varid));
> #ifdef USE_ASSERT_CHECKING
>     SET_LOCKTAG_OBJECT(locktag,
>                        MyDatabaseId,
>                        VariableRelationId,
>                        myState->varid,
>                        0);
>     Assert(LockHeldByMe(&locktag, AccessShareLock, false));
> #endif
>     Assert(natts == 1);
>     attr = TupleDescAttr(typeinfo, 0);
>     myState->need_detoast = attr->attlen == -1;
>     myState->rows = 0;
> }
>

done + regress tests


>
> I've attached the file containing the changes I mentioned earlier.
>
> -------------------------<<>>>-------------------------------
> Overall, 0001 and 0002 the doc looks good to me now.
> The following are only some minor issues I came up with.
>
> In Table 5.1. ACL Privilege Abbreviations
> <table id="privilege-abbrevs-table">
> <title><acronym>ACL</acronym> Privilege Abbreviations</title>
>
> <literal>VARIABLE</literal> (occurred 3 times)
> should be
> <literal>SESSION VARIABLE</literal>
> ?
>

changed


>
> doc/src/sgml/glossary.sgml
> I want to do minor tweak. from
>     <para>
>      A persistent database object that holds a value in session memory.
> This
>      value is private to each session and is released when the session
> ends.
>      Read or write access to session variables is controlled by privileges,
>      similar to other database objects.
>     </para>
> to
>     <para>
>      A persistent database object that holds a value in session memory.
> This
>      value is private to each session and is reset to default value
> (null) when the session ends.
>      Read or write access to session variables is controlled by access
> privileges,
>      similar to other database objects.
>     </para>
>

I think the sentence  "and is reset to default value (null) when the
session ends" is a little bit misguided. The memory is
really released. I changed it to

     A persistent database object that holds a value in session memory.
This
     value is private to each session and is released when the session ends.
     The default value of the session variable is null.  Read or write
access
     to session variables is controlled by privileges, similar to other
database
     objects.



> in let.sgml.
>   <para>
>    Example:
> <programlisting>
> CREATE VARIABLE myvar AS integer;
> LET myvar = 10;
> LET myvar = (SELECT sum(val) FROM tab);
> </programlisting>
>   </para>
>
> it should be
>  <refsect1>
>   <title>Examples</title>
> ...your example code
>  </refsect1>
>

done

Regards

Pavel

  325. Re: Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-01-04T05:36:56Z

    Hi

fix typo

Regards

Pavel

  326. Re: Re: proposal: schema variables

    jian he <jian.universality@gmail.com> — 2025-01-05T04:52:15Z

    diff --git a/src/include/nodes/primnodes.h b/src/include/nodes/primnodes.h
index 9c2957eb546..624858db301 100644
--- a/src/include/nodes/primnodes.h
+++ b/src/include/nodes/primnodes.h
@@ -361,6 +361,9 @@ typedef struct Const
  * of the `paramid' field contain the SubLink's subLinkId, and
  * the low-order 16 bits contain the column number.  (This type
  * of Param is also converted to PARAM_EXEC during planning.)
+ *
+ * PARAM_VARIABLE:  The parameter is a reference to a session variable
+ * (paramid holds the variable's OID).
  */
 typedef enum ParamKind
 {
@@ -368,6 +371,7 @@ typedef enum ParamKind
  PARAM_EXEC,
  PARAM_SUBLINK,
  PARAM_MULTIEXPR,
+ PARAM_VARIABLE,
 } ParamKind;

 typedef struct Param
@@ -380,6 +384,10 @@ typedef struct Param
  int32 paramtypmod pg_node_attr(query_jumble_ignore);
  /* OID of collation, or InvalidOid if none */
  Oid paramcollid pg_node_attr(query_jumble_ignore);
+ /* OID of session variable if it is used */
+ Oid paramvarid;
+ /* true when param is used like base node of assignment indirection */
+ bool parambasenode;
  /* token location, or -1 if unknown */
  ParseLoc location;
 } Param;

comment should be "(paramvarid holds the variable's OID)"
also
+ /* true when param is used like base node of assignment indirection */
is kind of hard to understand.
param->parambasenode = true;
only happens in transformLetStmt so we can change it to
+ /* true when param is used in part of the LET statement.*/


--- a/src/include/executor/execdesc.h
+++ b/src/include/executor/execdesc.h
@@ -51,6 +51,10 @@ typedef struct QueryDesc
  /* This field is set by ExecutePlan */
  bool already_executed; /* true if previously executed */

+ /* reference to session variables buffer */
+ int num_session_variables;
+ SessionVariableValue *session_variables;
+
  /* This is always set NULL by the core system, but plugins can change it */
  struct Instrumentation *totaltime; /* total time spent in ExecutorRun */
 } QueryDesc;
QueryDesc->num_session_variables and
QueryDesc->session_variables are never used in 0001 and 0002.
it may be used in later patches,
but at least 0001 and 0002, we don't need to change
struct QueryDesc?


contrib/postgres_fdw/deparse.c
There are some cases of "case T_Param:"
do we need to do something on it?
also on src/backend/nodes/queryjumblefuncs.c, one
appearance of "case T_Param:". (but it seems no need change here)



CREATE VARIABLE v1 AS int;
CREATE VARIABLE v2 AS int;
SELECT pg_stat_statements_reset() IS NOT NULL AS t;
select count(*) from tenk1 where unique1 = v1;
select count(*) from tenk1 where unique1 = v2;

should the last two queries be normalized as one query in pg_stat_statements?
if so, then maybe in typedef struct Param
we need change to:
Oid            paramvarid pg_node_attr(query_jumble_ignore);
but then
let v1 = v1+1;
let v1 = v2+1;
will be normalized as one query.

  327. Re: Re: proposal: schema variables

    jian he <jian.universality@gmail.com> — 2025-01-05T16:10:37Z

    + /*
+ * The arguments of EXECUTE are evaluated by a direct expression
+ * executor call.  This mode doesn't support session variables yet.
+ * It will be enabled later.
+ */
+ if (pstate->p_hasSessionVariables)
+ elog(ERROR, "session variable cannot be used as an argument");

it should be:
    /*
     * The arguments of CALL statement are evaluated by a direct expression
     * executor call.  This path is unsupported yet, so block it.
     */
    if (pstate->p_hasSessionVariables)
        ereport(ERROR,
                errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
                errmsg("session variable cannot be used as an argument"));


similarly, EvaluateParams we can change it to
    /*
     * The arguments of EXECUTE are evaluated by a direct expression
     * executor call.  This mode doesn't support session variables yet.
     * It will be enabled later.
     */
    if (pstate->p_hasSessionVariables)
        ereport(ERROR,
                errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
                errmsg("session variable cannot be used as an argument"));

in src/backend/executor/execExpr.c
we don't need
+#include "catalog/pg_variable.h"
?

  328. Re: Re: proposal: schema variables

    jian he <jian.universality@gmail.com> — 2025-01-06T07:58:36Z

    comment out the changes in
src/backend/utils/cache/plancache.c

    // /* process session variables */
    // if (OidIsValid(parsetree->resultVariable))
    // {
    //     if (acquire)
    //         LockDatabaseObject(VariableRelationId, parsetree->resultVariable,
    //                            0, AccessShareLock);
    //     else
    //         UnlockDatabaseObject(VariableRelationId,
parsetree->resultVariable,
    //                              0, AccessShareLock);
    // }

    // else if (IsA(node, Param))
    // {
    //     Param       *p = (Param *) node;
    //     if (p->paramkind == PARAM_VARIABLE)
    //     {
    //         if (acquire)
    //             LockDatabaseObject(VariableRelationId, p->paramvarid,
    //                                0, AccessShareLock);
    //         else
    //             UnlockDatabaseObject(VariableRelationId, p->paramvarid,
    //                                  0, AccessShareLock);
    //     }
    // }
the regress tests are still successful, that means these code changes
don't have related tests.


SetSessionVariable(Oid varid, Datum value, bool isNull)
{
    create_sessionvars_hashtables();
    svar = (SVariable) hash_search(sessionvars, &varid, HASH_ENTER, &found);
    if (!found)
        setup_session_variable(svar, varid);
    /* if this fails, it won't change the stored value */
    set_session_variable(svar, value, isNull);
}
after set_session_variable,
we want to make sure that svar->is_valid is true,
svar->value = value and  svar->isnull= isNull.
Based on this, I've simplified the function set_session_variable,
refer v1-0001-minor-refactoring-set_session_variable.no-cfbot


we use PlannerGlobal
{
   Oid            basenodeSessionVarid;
   Bitmapset *checkSelectPermVarids;
}
to solve the self-assigned corner case SELECT privilege.
(let v1.a =v1.a; in this case, we need have SELECT priv for v1.a
but let v1.a = 1, we don't need SELECT priv for v1.a).

i found out these two field value(information) most case is the same
as PlannerGlobal.sessionVariables;
I came up with another solution, introduce a bool (Query.is_Variable_assigned),
and get rid of PlannerGlobal.basenodeSessionVarid,
PlannerGlobal.checkSelectPermVarids.
not sure it make sense to you, refer
v1-0002-refactoring-LET-statement-self-assign-privileg.no-cfbot

  329. Re: Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-01-06T08:39:39Z

    Hi

ne 5. 1. 2025 v 5:52 odesílatel jian he <jian.universality@gmail.com>
napsal:

> diff --git a/src/include/nodes/primnodes.h b/src/include/nodes/primnodes.h
> index 9c2957eb546..624858db301 100644
> --- a/src/include/nodes/primnodes.h
> +++ b/src/include/nodes/primnodes.h
> @@ -361,6 +361,9 @@ typedef struct Const
>   * of the `paramid' field contain the SubLink's subLinkId, and
>   * the low-order 16 bits contain the column number.  (This type
>   * of Param is also converted to PARAM_EXEC during planning.)
> + *
> + * PARAM_VARIABLE:  The parameter is a reference to a session variable
> + * (paramid holds the variable's OID).
>   */
>  typedef enum ParamKind
>  {
> @@ -368,6 +371,7 @@ typedef enum ParamKind
>   PARAM_EXEC,
>   PARAM_SUBLINK,
>   PARAM_MULTIEXPR,
> + PARAM_VARIABLE,
>  } ParamKind;
>
>  typedef struct Param
> @@ -380,6 +384,10 @@ typedef struct Param
>   int32 paramtypmod pg_node_attr(query_jumble_ignore);
>   /* OID of collation, or InvalidOid if none */
>   Oid paramcollid pg_node_attr(query_jumble_ignore);
> + /* OID of session variable if it is used */
> + Oid paramvarid;
> + /* true when param is used like base node of assignment indirection */
> + bool parambasenode;
>   /* token location, or -1 if unknown */
>   ParseLoc location;
>  } Param;
>
> comment should be "(paramvarid holds the variable's OID)"
> also
> + /* true when param is used like base node of assignment indirection */
> is kind of hard to understand.
> param->parambasenode = true;
> only happens in transformLetStmt so we can change it to
> + /* true when param is used in part of the LET statement.*/
>

I don't think the proposed change of comment is better, but the text
can be extended.

<-->/*
<--> * true if param is used as base node of assignment indirection
<--> * (when target of LET statement is an array field or an record field).
<--> * For this param we do not check SELECT access right, because this
<--> * param is used just for execution of UPDATE operation.
<--> */



>
>
> --- a/src/include/executor/execdesc.h
> +++ b/src/include/executor/execdesc.h
> @@ -51,6 +51,10 @@ typedef struct QueryDesc
>   /* This field is set by ExecutePlan */
>   bool already_executed; /* true if previously executed */
>
> + /* reference to session variables buffer */
> + int num_session_variables;
> + SessionVariableValue *session_variables;
> +
>   /* This is always set NULL by the core system, but plugins can change it
> */
>   struct Instrumentation *totaltime; /* total time spent in ExecutorRun */
>  } QueryDesc;
> QueryDesc->num_session_variables and
> QueryDesc->session_variables are never used in 0001 and 0002.
> it may be used in later patches,
> but at least 0001 and 0002, we don't need to change
> struct QueryDesc?
>
>
moved to patch 17


> contrib/postgres_fdw/deparse.c
> There are some cases of "case T_Param:"
> do we need to do something on it?
>

I think so session variables cannot be executed remotely, and then do
nothing there it is correct.


> also on src/backend/nodes/queryjumblefuncs.c, one
> appearance of "case T_Param:". (but it seems no need change here)
>

CREATE VARIABLE v1 AS int;
> CREATE VARIABLE v2 AS int;
> SELECT pg_stat_statements_reset() IS NOT NULL AS t;
> select count(*) from tenk1 where unique1 = v1;
> select count(*) from tenk1 where unique1 = v2;
>
> should the last two queries be normalized as one query in
> pg_stat_statements?
> if so, then maybe in typedef struct Param
> we need change to:
> Oid            paramvarid pg_node_attr(query_jumble_ignore);
>

changed

In this case for this moment we can try to be consistent with plpgsql
variables.   So I did it

but then
> let v1 = v1+1;
> let v1 = v2+1;
> will be normalized as one query.
>

yes, but this case is not too important, because at the end (I hope), this
statement inside plpgsql will be executed as
simple expression, and then it will not be processed by pg_stat_statements

  330. Re: Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-01-06T10:01:17Z

    ne 5. 1. 2025 v 17:11 odesílatel jian he <jian.universality@gmail.com>
napsal:

> + /*
> + * The arguments of EXECUTE are evaluated by a direct expression
> + * executor call.  This mode doesn't support session variables yet.
> + * It will be enabled later.
> + */
> + if (pstate->p_hasSessionVariables)
> + elog(ERROR, "session variable cannot be used as an argument");
>
> it should be:
>     /*
>      * The arguments of CALL statement are evaluated by a direct expression
>      * executor call.  This path is unsupported yet, so block it.
>      */
>     if (pstate->p_hasSessionVariables)
>         ereport(ERROR,
>                 errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
>                 errmsg("session variable cannot be used as an argument"));
>
> done


>
> similarly, EvaluateParams we can change it to
>     /*
>      * The arguments of EXECUTE are evaluated by a direct expression
>      * executor call.  This mode doesn't support session variables yet.
>      * It will be enabled later.
>      */
>     if (pstate->p_hasSessionVariables)
>         ereport(ERROR,
>                 errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
>                 errmsg("session variable cannot be used as an argument"));
>

done


>
> in src/backend/executor/execExpr.c
> we don't need
> +#include "catalog/pg_variable.h"
> ?
>

moved to patch 16

  331. Re: Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-01-06T12:21:09Z

    Hi

po 6. 1. 2025 v 8:59 odesílatel jian he <jian.universality@gmail.com>
napsal:

> comment out the changes in
> src/backend/utils/cache/plancache.c
>
>     // /* process session variables */
>     // if (OidIsValid(parsetree->resultVariable))
>     // {
>     //     if (acquire)
>     //         LockDatabaseObject(VariableRelationId,
> parsetree->resultVariable,
>     //                            0, AccessShareLock);
>     //     else
>     //         UnlockDatabaseObject(VariableRelationId,
> parsetree->resultVariable,
>     //                              0, AccessShareLock);
>     // }
>
>     // else if (IsA(node, Param))
>     // {
>     //     Param       *p = (Param *) node;
>     //     if (p->paramkind == PARAM_VARIABLE)
>     //     {
>     //         if (acquire)
>     //             LockDatabaseObject(VariableRelationId, p->paramvarid,
>     //                                0, AccessShareLock);
>     //         else
>     //             UnlockDatabaseObject(VariableRelationId, p->paramvarid,
>     //                                  0, AccessShareLock);
>     //     }
>     // }
> the regress tests are still successful, that means these code changes
> don't have related tests.
>

the locking is tested by isolation test in patch05


>
>
> SetSessionVariable(Oid varid, Datum value, bool isNull)
> {
>     create_sessionvars_hashtables();
>     svar = (SVariable) hash_search(sessionvars, &varid, HASH_ENTER,
> &found);
>     if (!found)
>         setup_session_variable(svar, varid);
>     /* if this fails, it won't change the stored value */
>     set_session_variable(svar, value, isNull);
> }
> after set_session_variable,
> we want to make sure that svar->is_valid is true,
> svar->value = value and  svar->isnull= isNull.
> Based on this, I've simplified the function set_session_variable,
> refer v1-0001-minor-refactoring-set_session_variable.no-cfbot
>

Unfortunately, I don't think it is possible. See comments

/*
 * Assign a new value to the session variable.  It is copied to
 * SVariableMemoryContext if necessary.
 *
 * **** If any error happens, the existing value won't be modified. ****
 */

So the variable cannot be released before memory for new content is
allocated

The current code is a little bit more complex, but significantly reduces
the risk so the variable will hold unexpected value.

The current code ensures stability of LET command - when it is successful,
then value is changed, or when it is not successful, then variable is not
changed.

Proposed code breaks it. There is risk of palloc exception, and that means,
after unsuccessful LET the variable can be initialized. So there are three
ending states, not just two.

pfree theoretically can raise an exception, but in this case it should be
code our design error (using wrong allocator)


>
> we use PlannerGlobal
> {
>    Oid            basenodeSessionVarid;
>    Bitmapset *checkSelectPermVarids;
> }
> to solve the self-assigned corner case SELECT privilege.
> (let v1.a =v1.a; in this case, we need have SELECT priv for v1.a
> but let v1.a = 1, we don't need SELECT priv for v1.a).
>
> i found out these two field value(information) most case is the same
> as PlannerGlobal.sessionVariables;
> I came up with another solution, introduce a bool
> (Query.is_Variable_assigned),
> and get rid of PlannerGlobal.basenodeSessionVarid,
> PlannerGlobal.checkSelectPermVarids.
> not sure it make sense to you, refer
> v1-0002-refactoring-LET-statement-self-assign-privileg.no-cfbot
>

It is true, so bitmapset checkSelectPermVarids contains almost all the same
data like sessionVariables.
But checkSelectPermVarids allows fast checking if a variable is used
already, and sessionVariables list ensures necessary order.

My implementation needs more memory (one bitmapset), but I think it is very
simple and less error prone (doesn't depend on iteration order).

In this case and this moment I prefer my bitmapset based solution. It can
be optimized maybe later - I thought about a dedicated item in
sessionVariables for the basenode parameter.
This should be a clear winner for value passed types, but for varlena types
(for long varlena types), it can force passing content of some session
variable twice.

Regards

Pavel

  332. Re: Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-01-06T19:10:12Z

    Hi


>
>>
>> we use PlannerGlobal
>> {
>>    Oid            basenodeSessionVarid;
>>    Bitmapset *checkSelectPermVarids;
>> }
>> to solve the self-assigned corner case SELECT privilege.
>> (let v1.a =v1.a; in this case, we need have SELECT priv for v1.a
>> but let v1.a = 1, we don't need SELECT priv for v1.a).
>>
>> i found out these two field value(information) most case is the same
>> as PlannerGlobal.sessionVariables;
>> I came up with another solution, introduce a bool
>> (Query.is_Variable_assigned),
>> and get rid of PlannerGlobal.basenodeSessionVarid,
>> PlannerGlobal.checkSelectPermVarids.
>> not sure it make sense to you, refer
>> v1-0002-refactoring-LET-statement-self-assign-privileg.no-cfbot
>>
>
> It is true, so bitmapset checkSelectPermVarids contains almost all the
> same data like sessionVariables.
> But checkSelectPermVarids allows fast checking if a variable is used
> already, and sessionVariables list ensures necessary order.
>
> My implementation needs more memory (one bitmapset), but I think it is
> very simple and less error prone (doesn't depend on iteration order).
>
> In this case and this moment I prefer my bitmapset based solution. It can
> be optimized maybe later - I thought about a dedicated item in
> sessionVariables for the basenode parameter.
> This should be a clear winner for value passed types, but for varlena
> types (for long varlena types), it can force passing content of some
> session variable twice.
>

I found so storing oid to bitmapset can have big memory overhead. Bitmapset
with one oid value like 16000 has about 2kB. So instead storing varid, I
try to store paramid. paramid starts by zero.
See patch 03. Now the overhead of the usage of bitmapset for calculation of
the variable excluded from SELECT acl check has tens bytes overhead.

Regards

Pavel


>
> Regards
>
> Pavel
>
>
>
>
>
>

  333. Re: Re: proposal: schema variables

    jian he <jian.universality@gmail.com> — 2025-01-07T09:07:22Z

    hi.

some minor issues.
'transformMergeStmt also needs
"qry->hasSessionVariables = pstate->p_hasSessionVariables;"
?


<function>acldefault</function> in doc/src/sgml/func.sgml
Need an update for SESSION VARIABLE?

Table 9.70. Access Privilege Inquiry Functions
sorting order: has_session_variable_privilege should after has_server_privilege?


Similar to src/test/regress/sql/event_trigger.sql,
we can use the following query to test four functions in
Table 9.79. Object Information and Addressing Functions
(9.27.5. Object Information and Addressing Functions)
for session variables.

SELECT
  e.varname,
  pg_describe_object('pg_variable'::regclass, e.oid, 0) as descr,
  b.type, b.object_names, b.object_args,
  pg_identify_object(a.classid, a.objid, a.objsubid) as ident
FROM pg_variable as e,
LATERAL pg_identify_object_as_address('pg_variable'::regclass, e.oid, 0) as b,
LATERAL pg_get_object_address(b.type, b.object_names, b.object_args) as a;


in function transformColumnRef
if (expr_kind_allows_session_variables(pstate->p_expr_kind))
{
}
can change to
if (!node &&
    !(relname && crerr == CRERR_NO_COLUMN) &&
    expr_kind_allows_session_variables(pstate->p_expr_kind))
{
}
can reduce the function call of expr_kind_allows_session_variables,
also not lose readability.


typedef struct SVariableData says:
    /*
     * Stored value and type description can be outdated when we receive a
     * sinval message.  We then have to check if the stored data are still
     * trustworthy.
     */
    bool        is_valid;

CREATE VARIABLE var3 AS int;
LET var3 = 10;
GRANT SELECT ON VARIABLE var3 TO regress_noowner;
I don't understand why the last statement GRANT SELECT needs to call
pg_variable_cache_callback?
and it will invalidate the original var3.


+/*
+ * Returns a copy of the value of the session variable (in the current memory
+ * context).  The caller is responsible for permission checks.
+ */
+Datum
+GetSessionVariable(Oid varid, bool *isNull, Oid *typid)
+{
+ SVariable svar;
+
+ svar = get_session_variable(varid);
+
+ /*
+ * Although "svar" is freshly validated in this point, svar->is_valid can
+ * be false, if an invalidation message ws processed during the domain check.
+ * But the variable and all its dependencies are locked now, so we don't need
+ * to repeat the validation.
+ */
+ Assert(svar);
+
+ *typid = svar->typid;
+
+ return copy_session_variable_value(svar, isNull);
+}
typo, "ws processed" should be "was processed".
also the Assert is not helpful? since svar is NULL,
get_session_variable would have segfault.
also SVariable svar; is not initialized to NULL, so generally it will
not be NULL by default.
also the comments seem to imply that before
copy_session_variable_value, svar->is_valid can be false.
if svar->is_valid is false, then why do copy_session_variable_value.

  334. Re: Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-01-07T21:21:38Z

    Hi

út 7. 1. 2025 v 10:07 odesílatel jian he <jian.universality@gmail.com>
napsal:

> hi.
>
> some minor issues.
> 'transformMergeStmt also needs
> "qry->hasSessionVariables = pstate->p_hasSessionVariables;"
> ?
>
>
yes, done



>
> <function>acldefault</function> in doc/src/sgml/func.sgml
> Need an update for SESSION VARIABLE?
>
> Table 9.70. Access Privilege Inquiry Functions
> sorting order: has_session_variable_privilege should after
> has_server_privilege?
>
>
swapped


>
> Similar to src/test/regress/sql/event_trigger.sql,
> we can use the following query to test four functions in
> Table 9.79. Object Information and Addressing Functions
> (9.27.5. Object Information and Addressing Functions)
> for session variables.
>
> SELECT
>   e.varname,
>   pg_describe_object('pg_variable'::regclass, e.oid, 0) as descr,
>   b.type, b.object_names, b.object_args,
>   pg_identify_object(a.classid, a.objid, a.objsubid) as ident
> FROM pg_variable as e,
> LATERAL pg_identify_object_as_address('pg_variable'::regclass, e.oid, 0)
> as b,
> LATERAL pg_get_object_address(b.type, b.object_names, b.object_args) as a;
>
>
done



>
> in function transformColumnRef
> if (expr_kind_allows_session_variables(pstate->p_expr_kind))
> {
> }
> can change to
> if (!node &&
>     !(relname && crerr == CRERR_NO_COLUMN) &&
>     expr_kind_allows_session_variables(pstate->p_expr_kind))
> {
> }
> can reduce the function call of expr_kind_allows_session_variables,
> also not lose readability.
>

In this case, I prefer current code. It is more accented so the code is
related to places where variables are allowed.
+ following patches like GUC-session_variables_ambiguity_warning or
variable-fence-syntax~upport-and-variable-fence-usa
are less invasive and much more readable.

My opinion about it is not extra strong, and surely it can be subjective. I
think the current form is quite well readable
and the proposed change is not significantly better.



>
>
> typedef struct SVariableData says:
>     /*
>      * Stored value and type description can be outdated when we receive a
>      * sinval message.  We then have to check if the stored data are still
>      * trustworthy.
>      */
>     bool        is_valid;
>
> CREATE VARIABLE var3 AS int;
> LET var3 = 10;
> GRANT SELECT ON VARIABLE var3 TO regress_noowner;
> I don't understand why the last statement GRANT SELECT needs to call
> pg_variable_cache_callback?
> and it will invalidate the original var3.
>

Reason is simple - you did the change of pg_variable. Unfortunately, the
system of system cache invalidation is sometimes too simple and too
aggressive.
Inside pg_variable_cache_callback I cannot identify if a related row in
pg_variable was removed (by another session) or just updated.
Theoretically I can miss the information of what row was touched too.
Important note - invalidation in this case doesn't means so variable or the
value of variable should be thrown. It just means, so before any usage of
the current variable's value, we should recheck our cached data against
fresh syscache. Cache invalidation is relative common in Postgres, but
validation is very quick - just the check of varcreate_lsn


> +/*
> + * Returns a copy of the value of the session variable (in the current
> memory
> + * context).  The caller is responsible for permission checks.
> + */
> +Datum
> +GetSessionVariable(Oid varid, bool *isNull, Oid *typid)
> +{
> + SVariable svar;
> +
> + svar = get_session_variable(varid);
> +
> + /*
> + * Although "svar" is freshly validated in this point, svar->is_valid can
> + * be false, if an invalidation message ws processed during the domain
> check.
> + * But the variable and all its dependencies are locked now, so we don't
> need
> + * to repeat the validation.
> + */
> + Assert(svar);
> +
> + *typid = svar->typid;
> +
> + return copy_session_variable_value(svar, isNull);
> +}
> typo, "ws processed" should be "was processed".
>

fixed


> also the Assert is not helpful? since svar is NULL,
> get_session_variable would have segfault.
> also SVariable svar; is not initialized to NULL, so generally it will
> not be NULL by default.
>

assert removed


> also the comments seem to imply that before
> copy_session_variable_value, svar->is_valid can be false.
> if svar->is_valid is false, then why do copy_session_variable_value.
>

When the flag is_valid is true, then we are sure, so the stored value is
valid. When this flag is false, then we are not sure, so this value is
valid. But inside one statement,
and if the variable was locked, then we don't need to repeat validation (if
we did it immediately before), because we are sure, so the related system
catalog should be valid, because it is locked, and then we can safely
return the value. The flag is_valid can be changed any time when we are
using catalog cache - in this case when we do domain check. So although it
can look
weird after execution get_session_variable we can return the correct value,
but the flag is_valid can be false. It doesn't mean, so the current value
is wrong. It means, so the next command
should repeat validation.

In the attached version I rewrote the check if we can or cannot to ACL
SELECT check for variable used as base node of assignment indirection. I
significantly simplified the code - it is based on fact so we know varid of
the base node variable. It should be the same session variable like
resultVariable. Then I don't need to use bitmapset of used session
variables. Just I can check if this variable is used by param that is not
basenode param.

Regards

Pavel

  335. Re: Re: proposal: schema variables

    jian he <jian.universality@gmail.com> — 2025-01-08T09:31:11Z

    hi.

you forgot change
<function>acldefault</function>
should add
'V' for <literal>SESSION VARIABLE</literal>


in doc/src/sgml/ddl.sgml
<sect1 id="ddl-session-variables">
maybe some examples (<programlisting>) of session variables being
shadowed would be great.

because doc/src/sgml/ref/create_variable.sgml said
  <note>
   <para>
    Session variables can be <quote>shadowed</quote> by other identifiers.
    For details, see <xref linkend="ddl-session-variables"/>.
   </para>
  </note>
If I click the link, then in ddl.sgml, there is also a bunch of text
saying that variable will be shadowed
in some situations, but there are no simple examples to demonstrate that.


"Create an date session variable <literal>var1</literal>:"
maybe it should be rephrased as
"Create a session variable <literal>var1</literal> as date data type?"
(i am not native english speaker)

-----------------------------------------------------------------------
--- a/src/include/nodes/primnodes.h
+++ b/src/include/nodes/primnodes.h
@@ -379,7 +379,8 @@ typedef struct Param
        Expr            xpr;
        ParamKind       paramkind;              /* kind of parameter.
See above */
        int                     paramid;                /* numeric ID
for parameter */
-       Oid                     paramtype;              /* pg_type OID
of parameter's datatype */
+       /* pg_type OID of parameter's datatype */
+       Oid                     paramtype
pg_node_attr(query_jumble_ignore);

I think we need the above to make
select v2;
select v1;
normalized as one query.
-----------------------------------------------------------------------
when we create a new table, we do something like this:
DefineRelation
heap_create_with_catalog
GetNewRelFileNumber
GetNewOidWithIndex

relation Oid uniqueness and variable uniqueness is the same thing.
If variable oid uniqueness problem ever reached,
at that moment, we should care more about relation oid uniqueness problem?

and in GetNewRelFileNumber, we have comments:
""
 * As with GetNewOidWithIndex(), there is some theoretical risk of a race
 * condition, but it doesn't seem worth worrying about.
"""
also comments in GetNewOidWithIndex
"""
 * Note that we are effectively assuming that the table has a relatively small
 * number of entries (much less than 2^32) and there aren't very long runs of
 * consecutive existing OIDs.  This is a mostly reasonable assumption for
 * system catalogs.
"""
that means pg_catalog.pg_variable.varcreate_lsn is not really necessary?
-----------------------------------------------------------------------
I think the latest patch for LET self assign ACL SELECT is not correct,
previously I also tried the same idea.

test demo.
CREATE TYPE vartest_t1 AS (a int, b int);
CREATE VARIABLE var1 AS vartest_t1;
CREATE ROLE regress_var_test_role;
GRANT UPDATE ON VARIABLE var1 TO regress_var_test_role;
GRANT select ON table tenk1 TO regress_var_test_role;
SET ROLE TO regress_var_test_role;

--both should fail
LET var1.a = var1.a + 10;
LET var1.a = (select * from (select count(*) from tenk1 where unique1
= var1.a + 10));
--------------------------------------------------------

  336. Re: Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-01-08T16:33:31Z

    Hi

st 8. 1. 2025 v 10:31 odesílatel jian he <jian.universality@gmail.com>
napsal:

> hi.
>
> you forgot change
> <function>acldefault</function>
> should add
> 'V' for <literal>SESSION VARIABLE</literal>
>

done


>
>
> in doc/src/sgml/ddl.sgml
> <sect1 id="ddl-session-variables">
> maybe some examples (<programlisting>) of session variables being
> shadowed would be great.
>
> because doc/src/sgml/ref/create_variable.sgml said
>   <note>
>    <para>
>     Session variables can be <quote>shadowed</quote> by other identifiers.
>     For details, see <xref linkend="ddl-session-variables"/>.
>    </para>
>   </note>
> If I click the link, then in ddl.sgml, there is also a bunch of text
> saying that variable will be shadowed
> in some situations, but there are no simple examples to demonstrate that.
>

done

>
>
> "Create an date session variable <literal>var1</literal>:"
> maybe it should be rephrased as
> "Create a session variable <literal>var1</literal> as date data type?"
> (i am not native english speaker)
>

changed to

Create a session variable <literal>var1</literal> of data type date


>
> -----------------------------------------------------------------------
> --- a/src/include/nodes/primnodes.h
> +++ b/src/include/nodes/primnodes.h
> @@ -379,7 +379,8 @@ typedef struct Param
>         Expr            xpr;
>         ParamKind       paramkind;              /* kind of parameter.
> See above */
>         int                     paramid;                /* numeric ID
> for parameter */
> -       Oid                     paramtype;              /* pg_type OID
> of parameter's datatype */
> +       /* pg_type OID of parameter's datatype */
> +       Oid                     paramtype
> pg_node_attr(query_jumble_ignore);
>
> I think we need the above to make
> select v2;
> select v1;
> normalized as one query.
>

Why do you think so? paramtype is PARAM_VARIABLE every time for all session
variables.

If we will ignore paramtype, then we cannot to decide SELECT v1 and SELECT
$1


-----------------------------------------------------------------------
> when we create a new table, we do something like this:
> DefineRelation
> heap_create_with_catalog
> GetNewRelFileNumber
> GetNewOidWithIndex
>
> relation Oid uniqueness and variable uniqueness is the same thing.
> If variable oid uniqueness problem ever reached,
> at that moment, we should care more about relation oid uniqueness problem?
>
> and in GetNewRelFileNumber, we have comments:
> ""
>  * As with GetNewOidWithIndex(), there is some theoretical risk of a race
>  * condition, but it doesn't seem worth worrying about.
> """
> also comments in GetNewOidWithIndex
> """
>  * Note that we are effectively assuming that the table has a relatively
> small
>  * number of entries (much less than 2^32) and there aren't very long runs
> of
>  * consecutive existing OIDs.  This is a mostly reasonable assumption for
>  * system catalogs.
> """
> that means pg_catalog.pg_variable.varcreate_lsn is not really necessary?
>

I don't think so.  This is a different problem, and different use case (I
think)

oids will always be unique. Any system table has a unique index on the oid
column. It is true for session variables too.

The values of session variables are always in memory. The values of other
database objects are primary in files. Postgres routines never hold
database objects in memory longer than one command or one transaction. And
in this time, the consistency of memory is protected by locks. When the
system drops some database objects, then it drops related files too.
Nothing similar is for session variables. It is a database object that is
forever only in memory, I cannot verify the content against some file.
varcreate_lsn is used instead.

You can imagine scenario:

session A:

CREATE VARIABLE v1 AS int;
-- variable v1 with oid 1 is created

session B:
LET v1 = 100;
PREPARE p AS SELECT v1;

session A:

repeat bin n times:
  DROP VARIABLE v1;
  CREATE VARIABLE v1 AS numeric;

session b:
there is possibility so there will be variable with varid 1

EXECUTE p --> CRASH

The plan of p will be replaced (due dependency), but without possibility to
check against varcreate_lsn we will use wrong value - possibly with wrong
type, surely with wrong content because related variables were dropped a
long time ago. With varcreate_lsn I can detect that the stored value is
obsolete (although it has the same id).






-----------------------------------------------------------------------
> I think the latest patch for LET self assign ACL SELECT is not correct,
> previously I also tried the same idea.
>
> test demo.
> CREATE TYPE vartest_t1 AS (a int, b int);
> CREATE VARIABLE var1 AS vartest_t1;
> CREATE ROLE regress_var_test_role;
> GRANT UPDATE ON VARIABLE var1 TO regress_var_test_role;
> GRANT select ON table tenk1 TO regress_var_test_role;
> SET ROLE TO regress_var_test_role;
>
> --both should fail
> LET var1.a = var1.a + 10;
> LET var1.a = (select * from (select count(*) from tenk1 where unique1
> = var1.a + 10));
>

fixed + regress test

> --------------------------------------------------------
>

In today's version I changed GetSessionVariable API little bit. Now it
returns only value and *isnull. GetSessionVariableWithTypeCheck is
completely removed. Instead I add
GetSessionVariableWithTypeid, but it is introduced in patch
allow-parallel-execution-queries-with-session-variab.patch. It is not
necessary before. I removed the field varid from structure
SessionVariableValue (execnodes.h) because it was not used everywhere.

Regards

Pavel

  337. Re: Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-01-08T19:00:28Z

    Hi

fix after 7b27f5fd36cb3270e8ac25aefd73b552663d1392

Regards

Pavel

  338. Re: Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-01-09T05:57:11Z

    Hi

minor change - remove one unwanted whitechar

Regards

Pavel

  339. Re: Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-01-15T07:28:13Z

    Hi

necessary rebase

Regards

Pavel

  340. Re: Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-01-17T07:18:20Z

    Hi

fix oid collision

Regards

Pavel

  341. Re: Re: proposal: schema variables

    Bruce Momjian <bruce@momjian.us> — 2025-01-17T13:41:51Z

    On Fri, Jan 17, 2025 at 08:18:20AM +0100, Pavel Stehule wrote:
> Hi
> 
> fix oid collision

What is the purpose of continually posting this patch to the email
lists?

-- 
  Bruce Momjian  <bruce@momjian.us>        https://momjian.us
  EDB                                      https://enterprisedb.com

  Do not let urgent matters crowd out time for investment in the future.

  342. Re: Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-01-17T13:48:29Z

    Hi

pá 17. 1. 2025 v 14:41 odesílatel Bruce Momjian <bruce@momjian.us> napsal:

> On Fri, Jan 17, 2025 at 08:18:20AM +0100, Pavel Stehule wrote:
> > Hi
> >
> > fix oid collision
>
> What is the purpose of continually posting this patch to the email
> lists?
>

The people still do a review, so I am fixing this patch. I am fixing it
immediately, when I detect an issue.

I can reduce the frequency once per week.

Regards

Pavel


> --
>   Bruce Momjian  <bruce@momjian.us>        https://momjian.us
>   EDB                                      https://enterprisedb.com
>
>   Do not let urgent matters crowd out time for investment in the future.
>
>
>

  343. Re: Re: proposal: schema variables

    Bruce Momjian <bruce@momjian.us> — 2025-01-17T14:16:08Z

    On Fri, Jan 17, 2025 at 02:48:29PM +0100, Pavel Stehule wrote:
> Hi
> 
> pá 17. 1. 2025 v 14:41 odesílatel Bruce Momjian <bruce@momjian.us> napsal:
> 
>     On Fri, Jan 17, 2025 at 08:18:20AM +0100, Pavel Stehule wrote:
>     > Hi
>     >
>     > fix oid collision
> 
>     What is the purpose of continually posting this patch to the email
>     lists?
> 
> 
> The people still do a review, so I am fixing this patch. I am fixing it
> immediately, when I detect an issue.
> 
> I can reduce the frequency once per week.

Is this really something we are considering applying, since it has been
around for years?  I am unclear on that and we had better know if we are
going to continue reviewing this.

-- 
  Bruce Momjian  <bruce@momjian.us>        https://momjian.us
  EDB                                      https://enterprisedb.com

  Do not let urgent matters crowd out time for investment in the future.

  344. Re: Re: proposal: schema variables

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2025-01-17T14:47:28Z

    On 2025-Jan-17, Bruce Momjian wrote:

> Is this really something we are considering applying, since it has been
> around for years?  I am unclear on that and we had better know if we are
> going to continue reviewing this.

The fact that the patch has been around for years doesn't automatically
mean it's a bad idea.

I have proposed that we discuss this patch at fosdem developer's meeting
next month, precisely to seek consensus on whether this patch is
something we want or not.  My view is that this is a feature that has
been requested by users for years, so IMO we want this or something
similar.

I wonder if the reason that committers stay away from it is that
reviewing it fully (and thus taking responsibility for it) seems such a
daunting task.  I might be wrong, but I think this may be the largest
patch since FTS.

-- 
Álvaro Herrera               48°01'N 7°57'E  —  https://www.EnterpriseDB.com/
Tom: There seems to be something broken here.
Teodor: I'm in sackcloth and ashes...  Fixed.
                               http://postgr.es/m/482D1632.8010507@sigaev.ru

  345. Fwd: Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-01-17T15:32:07Z

    Hi

pá 17. 1. 2025 v 15:39 odesílatel Bruce Momjian <bruce@momjian.us> napsal:

> On Fri, Jan 17, 2025 at 03:28:55PM +0100, Pavel Stehule wrote:
> > Dne pá 17. 1. 2025 15:16 uživatel Bruce Momjian <bruce@momjian.us>
> napsal:
> >     Is this really something we are considering applying, since it has
> been
> >     around for years?  I am unclear on that and we had better know if we
> are
> >     going to continue reviewing this.
> >
> > I hope so it is possible, minimally in some basic form. And i think so
> there
> > was real good  progres of quality in last three months.
>
> I am not asking if it is improving.  I am asking if it is a desired
> feature;  see:
>
>         https://wiki.postgresql.org/wiki/Todo#Development_Process
>         Desirability -> Design -> Implement -> Test -> Review -> Commit
>
> I am asking if we have had the Desirability discussion, and its outcome,
> because if we can't agree on its Desirability, the other stages are
> useless.
>

This discussion was around 2017 when I wrote a proposal and I hadn't a
feeling so we don't write this feature.
Big discussion was related to whether variables should be transactional or
not. Next the patch was stalled from
two reasons: a) there was not necessary infrastructure for utility
commands, b) I searched for ways to ensure
the validity of the content of variables. I found a good solution at the
end of  2022. It is true, so time has changed
from this time, and Postgres and people are different. In this time the
migration from Oracle was stronger
topic.

If you read all the discussion, you can find more times the sentence so
this can be a good feature (not from me).
Surely the session variables can be implemented differently - minimally
there are four different implementations
mssql, db2, mysql and oracle, and there can be unfinished discussion about
which way is better or if the session
variables are necessary. Yes, we can live without it - we are living
without it, but emulation by GUC is not secure,
so some scenarios are not possible, and others are breakneck with
emulation.

I understand the question if we need it is open still and every time. This
feature is interesting for people who
a) use stored procedures
b) use RLS

Both these groups are not the majority of users. But these people are here.

Btw - EDB supports Oracle way, and Postgres Pro uses extension for
emulation. So there is a real request
for this feature. Common solution for Postgres is using GUC. But there is
no possibility to set access
rights so the workaround cannot be secured.

There is one stronger argument for session variables - we are missing
global temporary tables. It is a real
limit and more times I found users with bloated pg_class, pg_attributes due
using temp tables. I don't believe
so we can have a global temp table - it is a significantly more difficult
task than session variables. At the end
session variables are trivial against global temp tables, and can replace
global temp tables in some use cases.
And the solution can be nicer, cleaner, safer than with a workaround based
on GUC.

Regards

Pavel






>
> --
>   Bruce Momjian  <bruce@momjian.us>        https://momjian.us
>   EDB                                      https://enterprisedb.com
>
>   Do not let urgent matters crowd out time for investment in the future.
>
>
>

  346. Re: Fwd: Re: proposal: schema variables

    Bruce Momjian <bruce@momjian.us> — 2025-01-17T15:35:48Z

    On Fri, Jan 17, 2025 at 04:32:07PM +0100, Pavel Stehule wrote:
> This discussion was around 2017 when I wrote a proposal and I hadn't a feeling

2017 is seven years ago so it would be good to get current feedback on
the desirability of this feature.

> There is one stronger argument for session variables - we are missing global
> temporary tables. It is a real
> limit and more times I found users with bloated pg_class, pg_attributes due
> using temp tables. I don't believe
> so we can have a global temp table - it is a significantly more difficult task
> than session variables. At the end
> session variables are trivial against global temp tables, and can replace
> global temp tables in some use cases.
> And the solution can be nicer, cleaner, safer than with a workaround based on
> GUC.

So this feature would be like global GUC variables, with permission
control?

-- 
  Bruce Momjian  <bruce@momjian.us>        https://momjian.us
  EDB                                      https://enterprisedb.com

  Do not let urgent matters crowd out time for investment in the future.

  347. Re: Fwd: Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-01-17T15:55:07Z

    pá 17. 1. 2025 v 16:35 odesílatel Bruce Momjian <bruce@momjian.us> napsal:

> On Fri, Jan 17, 2025 at 04:32:07PM +0100, Pavel Stehule wrote:
> > This discussion was around 2017 when I wrote a proposal and I hadn't a
> feeling
>
> 2017 is seven years ago so it would be good to get current feedback on
> the desirability of this feature.
>
> > There is one stronger argument for session variables - we are missing
> global
> > temporary tables. It is a real
> > limit and more times I found users with bloated pg_class, pg_attributes
> due
> > using temp tables. I don't believe
> > so we can have a global temp table - it is a significantly more
> difficult task
> > than session variables. At the end
> > session variables are trivial against global temp tables, and can replace
> > global temp tables in some use cases.
> > And the solution can be nicer, cleaner, safer than with a workaround
> based on
> > GUC.
>
> So this feature would be like global GUC variables, with permission
> control?
>

+ types and domain type check - holds data in binary form - there are not
conversions binary, text
+ it is declared - so less space for misuse is there. Custom GUC are
absolutely tolerant
+ it is a fully database object, only owner can alter it,  and event
triggers are supported, sinval
+ possibility to set mutability, default value

Regards

Pavel



>
> --
>   Bruce Momjian  <bruce@momjian.us>        https://momjian.us
>   EDB                                      https://enterprisedb.com
>
>   Do not let urgent matters crowd out time for investment in the future.
>
>
>

  348. Re: Fwd: Re: proposal: schema variables

    Bruce Momjian <bruce@momjian.us> — 2025-01-17T16:01:41Z

    On Fri, Jan 17, 2025 at 04:55:07PM +0100, Pavel Stehule wrote:
> pá 17. 1. 2025 v 16:35 odesílatel Bruce Momjian <bruce@momjian.us> napsal:
> 
>     So this feature would be like global GUC variables, with permission
>     control?
> 
> + types and domain type check - holds data in binary form - there are not
> conversions binary, text
> + it is declared - so less space for misuse is there. Custom GUC are absolutely
> tolerant
> + it is a fully database object, only owner can alter it,  and event triggers
> are supported, sinval
> + possibility to set mutability, default value

Okay, good summary.  Now, can people give feedback that they would want
this committed to PostgreSQL?

-- 
  Bruce Momjian  <bruce@momjian.us>        https://momjian.us
  EDB                                      https://enterprisedb.com

  Do not let urgent matters crowd out time for investment in the future.

  349. Re: Fwd: Re: proposal: schema variables

    Julien Rouhaud <rjuju123@gmail.com> — 2025-01-17T16:10:47Z

    On Fri, Jan 17, 2025 at 11:01:41AM -0500, Bruce Momjian wrote:
> On Fri, Jan 17, 2025 at 04:55:07PM +0100, Pavel Stehule wrote:
> > pá 17. 1. 2025 v 16:35 odesílatel Bruce Momjian <bruce@momjian.us> napsal:
> > 
> >     So this feature would be like global GUC variables, with permission
> >     control?
> > 
> > + types and domain type check - holds data in binary form - there are not
> > conversions binary, text
> > + it is declared - so less space for misuse is there. Custom GUC are absolutely
> > tolerant
> > + it is a fully database object, only owner can alter it,  and event triggers
> > are supported, sinval
> > + possibility to set mutability, default value
> 
> Okay, good summary.  Now, can people give feedback that they would want
> this committed to PostgreSQL?

I unsurprisingly would really like to see it committed, for all the reasons
Pavel mentioned.  It would have helped me on various projects many times.

  350. Re: Fwd: Re: proposal: schema variables

    Laurenz Albe <laurenz.albe@cybertec.at> — 2025-01-17T16:43:13Z

    On Fri, 2025-01-17 at 11:01 -0500, Bruce Momjian wrote:
> On Fri, Jan 17, 2025 at 04:55:07PM +0100, Pavel Stehule wrote:
> > pá 17. 1. 2025 v 16:35 odesílatel Bruce Momjian <bruce@momjian.us> napsal:
> > 
> >     So this feature would be like global GUC variables, with permission
> >     control?
> > 
> > + types and domain type check - holds data in binary form - there are not
> > conversions binary, text
> > + it is declared - so less space for misuse is there. Custom GUC are absolutely
> > tolerant
> > + it is a fully database object, only owner can alter it,  and event triggers
> > are supported, sinval
> > + possibility to set mutability, default value
> 
> Okay, good summary.  Now, can people give feedback that they would want
> this committed to PostgreSQL?

I would like to see this committed too, or at least relevant parts of it.

It addresses the perennial problem of people putting state into placeholder
GUCs to pass information between the application and the database
(SET myapp.application_id = 'user_laurenz').

Also, it cann pass information between the code in DO statements and
the surrounding SQL code.

Yours,
Laurenz Albe

  351. Re: Fwd: Re: proposal: schema variables

    Dmitry Dolgov <9erthalion6@gmail.com> — 2025-01-17T17:47:52Z

    > On Fri, Jan 17, 2025 at 11:01:41AM GMT, Bruce Momjian wrote:
> On Fri, Jan 17, 2025 at 04:55:07PM +0100, Pavel Stehule wrote:
> > pá 17. 1. 2025 v 16:35 odesílatel Bruce Momjian <bruce@momjian.us> napsal:
> >
> >     So this feature would be like global GUC variables, with permission
> >     control?
> >
> > + types and domain type check - holds data in binary form - there are not
> > conversions binary, text
> > + it is declared - so less space for misuse is there. Custom GUC are absolutely
> > tolerant
> > + it is a fully database object, only owner can alter it,  and event triggers
> > are supported, sinval
> > + possibility to set mutability, default value
>
> Okay, good summary.  Now, can people give feedback that they would want
> this committed to PostgreSQL?

+1 into the bucket "want committed" from me as well. Throughout the
review process I've stumbled upon a few cases in my own projects, where
it would be useful.

  352. Re: Fwd: Re: proposal: schema variables

    Wolfgang Walther <walther@technowledgy.de> — 2025-01-17T20:20:03Z

    Bruce Momjian:
> Now, can people give feedback that they would want
> this committed to PostgreSQL?

 From a user's perspective: Yes!

I've been waiting for this for a long time and I really hope this can go 
through, eventually.

Best,

Wolfgang

  353. Re: Fwd: Re: proposal: schema variables

    Marcos Pegoraro <marcos@f10.com.br> — 2025-01-17T21:30:40Z

    >
> > pá 17. 1. 2025 v 16:35 odesílatel Bruce Momjian <bruce@momjian.us>
> napsal:
>
> Okay, good summary.  Now, can people give feedback that they would want
> this committed to PostgreSQL?
>

I would love to have this functionality as soon as possible.
I already mentioned to Pavel that he did something very big, and
consequently difficult for anyone to commit, because these patches change a
lot of things in a lot of places.
Of course we want as much as possible, but who knows if a first, leaner
version was committed, with just session variables, so nothing related to
schemas, catalogs, grants,  dumps, etc, just a variable in memory, only
that. It would be really cool, and certainly easier for a committer to
agree with the code. And if the first commit occurs, others can follow.

Remember that some people don't use PSQL, so they don't have \set
Even \set variables are not typed, so CREATE VARIABLE T AS
DOMAIN_NUMBER_BETWEEN_1_AND_5 is really cool.

regards
Marcos

  354. Re: Fwd: Re: proposal: schema variables

    Marcos Pegoraro <marcos@f10.com.br> — 2025-01-17T21:54:17Z

    Em sex., 17 de jan. de 2025 às 13:01, Bruce Momjian <bruce@momjian.us>
escreveu:

> Okay, good summary.  Now, can people give feedback that they would want
> this committed to PostgreSQL?


I would love to have this functionality as soon as possible.
I already mentioned to Pavel that he did something very big, and
consequently difficult for anyone to commit, because these patches change a
lot of things in a lot of places.
Of course we want as much as possible, but who knows if a first, leaner
version was committed, with just session variables, so nothing related to
schemas, catalogs, grants,  dumps, etc, just a variable in memory, only
that. It would be really cool, and certainly easier for a committer to
agree with the code. And if the first commit occurs, others can follow.

Remember that some people don't use PSQL, so they don't have \set
Even \set variables are not typed, so CREATE VARIABLE T AS
DOMAIN_NUMBER_BETWEEN_1_AND_5 is really cool.

regards
Marcos

  355. Re: Fwd: Re: proposal: schema variables

    Gilles Darold <gilles@darold.net> — 2025-01-18T07:24:04Z

    Le 17/01/2025 à 19:01, Bruce Momjian a écrit :
> On Fri, Jan 17, 2025 at 04:55:07PM +0100, Pavel Stehule wrote:
>> pá 17. 1. 2025 v 16:35 odesílatel Bruce Momjian <bruce@momjian.us> napsal:
>>
>>      So this feature would be like global GUC variables, with permission
>>      control?
>>
>> + types and domain type check - holds data in binary form - there are not
>> conversions binary, text
>> + it is declared - so less space for misuse is there. Custom GUC are absolutely
>> tolerant
>> + it is a fully database object, only owner can alter it,  and event triggers
>> are supported, sinval
>> + possibility to set mutability, default value
> Okay, good summary.  Now, can people give feedback that they would want
> this committed to PostgreSQL?
>

For me, this is a must have. Not only because of the huge amount of time 
that we will save in migration to PostgreSQL (i know that this is not an 
argument) but because the only way we have to emulate such feature is to 
use custom configuration directives or a PL language that allows global 
variable. I have question almost every week on how to do that and the 
only answer I have is do this ugly pl/pgsql coding. It will be nice to 
have a feature to do that. Thanks Pavel and all involved in this 
implementation.

-- 
Gilles Darold
http://www.darold.net/

  356. Re: Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-01-20T08:26:11Z

    Hi

po 20. 1. 2025 v 8:56 odesílatel Álvaro Herrera <alvherre@alvh.no-ip.org>
napsal:

> On 2025-Jan-17, Bruce Momjian wrote:
>
> > Is this really something we are considering applying, since it has been
> > around for years?  I am unclear on that and we had better know if we are
> > going to continue reviewing this.
>
> The fact that the patch has been around for years doesn't automatically
> mean it's a bad idea.
>
> I have proposed that we discuss this patch at fosdem developer's meeting
> next month, precisely to seek consensus on whether this patch is
> something we want or not.  My view is that this is a feature that has
> been requested by users for years, so IMO we want this or something
> similar.
>
> I wonder if the reason that committers stay away from it is that
> reviewing it fully (and thus taking responsibility for it) seems such a
> daunting task.  I might be wrong, but I think this may be the largest
> patch since FTS.
>

This patch is huge, but I think it is not comparable with parallel
processing support or with replication support.

It doesn't introduce new processes or new data structures or does important
changes in planner, and I think so almost all code is very simple.

In early versions of this patch, there was a complex part to ensure
validity of content in memory. But it was fully removed and replaced
just by comparing with create_lsn.

Regards

Pavel


> --
> Álvaro Herrera               48°01'N 7°57'E  —
> https://www.EnterpriseDB.com/
> Tom: There seems to be something broken here.
> Teodor: I'm in sackcloth and ashes...  Fixed.
>
> http://postgr.es/m/482D1632.8010507@sigaev.ru
>

  357. Re: Re: proposal: schema variables

    Bruce Momjian <bruce@momjian.us> — 2025-01-20T20:15:43Z

    On Fri, Jan 17, 2025 at 03:47:28PM +0100, Álvaro Herrera wrote:
> On 2025-Jan-17, Bruce Momjian wrote:
> 
> > Is this really something we are considering applying, since it has been
> > around for years?  I am unclear on that and we had better know if we are
> > going to continue reviewing this.
> 
> The fact that the patch has been around for years doesn't automatically
> mean it's a bad idea.

Yes, I think we passed the Desirability criteria with the feedback on
this thread, but it is now a question of whether the code complexity
justifies the feature.  I saw a few people saying they want _some_ parts
of the patch, which opens the suggestion that even people who want the
patch are seeing parts of the patch that are too much.  I have seen this
patch circling around, and I think it needs a step a back for analysis.

> I have proposed that we discuss this patch at fosdem developer's meeting
> next month, precisely to seek consensus on whether this patch is
> something we want or not.  My view is that this is a feature that has
> been requested by users for years, so IMO we want this or something
> similar.

Yes, the meeting review is a very good idea.

> I wonder if the reason that committers stay away from it is that
> reviewing it fully (and thus taking responsibility for it) seems such a
> daunting task.  I might be wrong, but I think this may be the largest
> patch since FTS.

I think we have to identify a committer who is willing to consider
application of this patch before the patch can move forward.

-- 
  Bruce Momjian  <bruce@momjian.us>        https://momjian.us
  EDB                                      https://enterprisedb.com

  Do not let urgent matters crowd out time for investment in the future.

  358. Re: Re: proposal: schema variables

    Laurenz Albe <laurenz.albe@cybertec.at> — 2025-01-20T20:40:34Z

    On Mon, 2025-01-20 at 15:15 -0500, Bruce Momjian wrote:
> Yes, I think we passed the Desirability criteria with the feedback on
> this thread, but it is now a question of whether the code complexity
> justifies the feature.  I saw a few people saying they want _some_ parts
> of the patch, which opens the suggestion that even people who want the
> patch are seeing parts of the patch that are too much.  I have seen this
> patch circling around, and I think it needs a step a back for analysis.

I'd say that patches 1 to 6 from the patch set are essential.

Yours,
Laurenz Albe

  359. Re: Re: proposal: schema variables

    jian he <jian.universality@gmail.com> — 2025-02-06T14:49:00Z

    hi.

git diff --check
shows there is some white space there.

Adding hack mailing list discussion links in each patch would be great.
Others said that the first 6 patches are essential, maybe we can only
post the first 6 patches.
so the test CI result would be more reliable. also people would not
feel intimidated by
the bigger patchset.


create domain dnn as int not null;
CREATE VARIABLE var3 AS dnn;
alter domain dnn drop not null;
alter domain dnn set not null;
ERROR:  cannot alter domain "dnn" because session variable "public.var3" uses it

This is not great.
we allow a constraint, then we drop it, now we cannot re add it again.
0001 and 0002 are simple, but the size is still large.
maybe we can not support the domain in the first 2 patches.

also
CREATE VARIABLE var3 AS dnn;
let var3 = 11;
create view v1 as select var3;
select * from v1;
you can see reconnect to session then
ERROR:  domain dnn does not allow null values
this is not ok?


also
create table t(var3 int);
CREATE POLICY p1r ON t AS RESTRICTIVE TO alice USING (var3 <> var3);
create table t1(a int);
CREATE POLICY p2 ON t1 AS RESTRICTIVE TO alice USING (a <> var3);
p1r is so confusing. there is no way to understand the intention.
p2 should also not be allowed, since var3 value is volatile,
session reconnection will change the value.


src/bin/pg_dump/pg_dump.h
/*
 * The VariableInfo struct is used to represent session variables
 */
typedef struct _VariableInfo
{
    DumpableObject dobj;
    DumpableAcl dacl;
    Oid            vartype;
    char       *vartypname;
    char       *varacl;
    char       *rvaracl;
    char       *initvaracl;
    char       *initrvaracl;
    Oid            varcollation;
    const char *rolname;        /* name of owner, or empty string */
} VariableInfo;
these fields (varacl, rvaracl, initvaracl, initrvaracl) were never being used.
we can remove them.

        CollInfo   *coll;
        coll = findCollationByOid(varcollation);
        if (coll)
            appendPQExpBuffer(query, " COLLATE %s",
                              fmtQualifiedDumpable(coll));
here, it should be
```CollInfo   *coll = NULL;```?


I don't understand the changes made in getAdditionalACLs.
I thought pg_init_privs had nothing to do with the session variable.


minor issue in getVariables.
    query = createPQExpBuffer();
    resetPQExpBuffer(query);
no need to use resetPQExpBuffer here.


create type ab as (a int, b text);
create type abc as (a ab, c text);
create type abcd as (a abc, d text);
CREATE VARIABLE var2 AS abcd;

select var2.a.c;
ERROR:  cross-database references are not implemented: var2.a.c

Is this error what we expected? I am not 100% sure.
--------------------another contrived corner case.-----------------------
create type pg_variable as (
  oid oid, vartype oid, varcreate_lsn pg_lsn,
  varname name, varnamespace oid, varowner oid,
  vartypmod int, varcollation oid, varacl aclitem[]);
create variable pg_variable as pg_variable;
let pg_variable = row (18041, 10116, '0/25137B0','pg_variable', 2200,
10,-1,0, NULL)::pg_variable;

select pg_variable.oid from pg_variable where pg_variable.oid = pg_variable.oid;
this query, the WHERE clause, it's really hard to distinguish session
variable or column reference.
I am not sure if this is fine or not.

  360. Re: Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-02-07T07:24:55Z

    Hi

čt 6. 2. 2025 v 15:49 odesílatel jian he <jian.universality@gmail.com>
napsal:

> hi.
>
> git diff --check
> shows there is some white space there.
>

yes, it was few times in documentation

fixed


>
> Adding hack mailing list discussion links in each patch would be great.
>

Unfortunately, it is not separated. The organization of patches is not
mapped to discussion, because originally there was one big patch that was
divided
and reorganized. So there was no separate discussion about catalog or
syntax. It was a pelmel and bikeshedding (lot of bikeshedding).


> Others said that the first 6 patches are essential, maybe we can only
> post the first 6 patches.
> so the test CI result would be more reliable. also people would not
> feel intimidated by
> the bigger patchset.
>
>
I understand, but I am not sure if it is a good idea. I would not repeat a
lot of bikeshedding like variables should be immutable, transactional, ...
etc..

Maybe I can repeat some higher level description of this patchset

Really essential are just patches 01-02. Just with these patches the user
gets almost all necessary and required functionality - possibility to work
with declared variables with access rights.
I don't think these patches can be reduced - maybe access rights can be
separated, but if you check source code, the part related to access rights
is few lines (and access rights cannot be supported in next steps due
possible compatibility issues). These patches are almost very simple -
mainly 01 - it is mechanical work related to maintaining a new database
class - some code, and a lot of tests.

Patches 03-05 are related to memory cleaning when variables are dropped. It
is important from a technical quality perspective, but until temporary
variables are supported the variables are not usually dropped frequently,
and then real impact for usage is not extra big. DISCARD VARIABLES can be
important for applications that use connection pooling, but this patch is
very simple again.
Patch05 is not long but does difficult work - sinval processing, cache
invalidation, contains isolation tests. It is probably the most difficult
part of this patchset. If we accept a small memory leak after the dropped
variable, then we can live without this patch (not forever - but I can
imagine passing this patch to the next major release). This is a necessary
prerequisite for temporary variables.

Patch 06 contains just plpgsql regress tests - nothing special or difficult.

Patches 07-09 can be important from user friendly usage perspective -
detection of shaded variables can help with unwanted shadowing, variable
fences can strongly reduce any potential risks of introduction of session
variables. I think variable fences can be an interesting safety tool,
concept - these patches are not small (+/- 20kB), but the code is a minimal
part of these patches. I think it can be very practical and pragmatic to
support variable fences from the start by requiring it outside SPI by
default. From my perspective, it is more important than variable shadowing
detection.

Patches 10-11 EXPLAIN, PREPARE support for LET statements are interesting
for consistency with other statements. Nothing special, nothing difficult,
nothing extra important, but interesting for consistent behaviour of all
PostgreSQL commands.

Patches 12-15 - temp variables, immutable variables, default values
support, reset at transaction end introduces a very interesting feature
that allows to do some special work with session variables and increases a
comfort for work with session variables. These patches can be postponed or
applied independently. There is not fully agreement on the syntax RESET AT
TRANSACTION END, but this is just one patch from this set, and other
patches don't depend on this patch.

Patches 16-19 are important from performance perspective and significantly
increases the performance of queries that contain session variables or the
performance of LET statements for some special cases (like simple
expressions from PLpgSQL, or parallel query execution). These patches have
zero impact on functionality, but significant impact on performance. These
patches are separated because of some deeper changes, and can be postponed,
so they are removed from patch 02. These patches are independent of others
- and can be applied later .. it is typical for PLpgSQL so the performance
was enhanced step by step from relatively bad to really good now.

Patch 20 enhances error messages to support variables - nothing special,
nothing important - these messages are not fully correct now against
plpgsql variables, and nobody protests.

Patch 21 supports transactional behaviour to session variables (the content
can be transactional). It is an unusual feature, other databases don't have
similar functionality, but for some cases can be nice, and can introduce
mutable transactional storage on read only replicas.

Patch 22 introduces the pg_dump option - short simple patch just for
consistency with already supported options of pg_dump. Nothing important,
nothing complex.

So really essential are patches 01, 02, 04. These patch holds almost all
valuable functionality that the people want
03, 05 are important for technical consistency - developers doesn't like
memory leaks after dropped objects (but I think real impact will be small
or +/- nothing)
08, 09 can be interesting for the possibility to force some good (safe)
style of usage from the beginning.

Others are interesting, important, but can be postponed or applied
independently. Some one can prefer functionality, someone can prefer
performance. For performance testing patches 16-19 can be interesting,
important (the performance in plpgsql is very significantly different if
the query is executed by SPI or directly - on second hand, plpgsql is not
used by majority users now). But they are separated because it is about 25%
of (01+02), and the code there is not trivial like in 01 and 02.

For domain behaviour check and other issues I need little bit more time

Regards

Pavel









>
> create domain dnn as int not null;
> CREATE VARIABLE var3 AS dnn;
> alter domain dnn drop not null;
> alter domain dnn set not null;
> ERROR:  cannot alter domain "dnn" because session variable "public.var3"
> uses it
>
> This is not great.
> we allow a constraint, then we drop it, now we cannot re add it again.
> 0001 and 0002 are simple, but the size is still large.
> maybe we can not support the domain in the first 2 patches.
>
> also
> CREATE VARIABLE var3 AS dnn;
> let var3 = 11;
> create view v1 as select var3;
> select * from v1;
> you can see reconnect to session then
> ERROR:  domain dnn does not allow null values
> this is not ok?
>
>
> also
> create table t(var3 int);
> CREATE POLICY p1r ON t AS RESTRICTIVE TO alice USING (var3 <> var3);
> create table t1(a int);
> CREATE POLICY p2 ON t1 AS RESTRICTIVE TO alice USING (a <> var3);
> p1r is so confusing. there is no way to understand the intention.
> p2 should also not be allowed, since var3 value is volatile,
> session reconnection will change the value.
>
>
> src/bin/pg_dump/pg_dump.h
> /*
>  * The VariableInfo struct is used to represent session variables
>  */
> typedef struct _VariableInfo
> {
>     DumpableObject dobj;
>     DumpableAcl dacl;
>     Oid            vartype;
>     char       *vartypname;
>     char       *varacl;
>     char       *rvaracl;
>     char       *initvaracl;
>     char       *initrvaracl;
>     Oid            varcollation;
>     const char *rolname;        /* name of owner, or empty string */
> } VariableInfo;
> these fields (varacl, rvaracl, initvaracl, initrvaracl) were never being
> used.
> we can remove them.
>
>         CollInfo   *coll;
>         coll = findCollationByOid(varcollation);
>         if (coll)
>             appendPQExpBuffer(query, " COLLATE %s",
>                               fmtQualifiedDumpable(coll));
> here, it should be
> ```CollInfo   *coll = NULL;```?
>
>
> I don't understand the changes made in getAdditionalACLs.
> I thought pg_init_privs had nothing to do with the session variable.
>
>
> minor issue in getVariables.
>     query = createPQExpBuffer();
>     resetPQExpBuffer(query);
> no need to use resetPQExpBuffer here.
>
>
> create type ab as (a int, b text);
> create type abc as (a ab, c text);
> create type abcd as (a abc, d text);
> CREATE VARIABLE var2 AS abcd;
>
> select var2.a.c;
> ERROR:  cross-database references are not implemented: var2.a.c
>
> Is this error what we expected? I am not 100% sure.
> --------------------another contrived corner case.-----------------------
> create type pg_variable as (
>   oid oid, vartype oid, varcreate_lsn pg_lsn,
>   varname name, varnamespace oid, varowner oid,
>   vartypmod int, varcollation oid, varacl aclitem[]);
> create variable pg_variable as pg_variable;
> let pg_variable = row (18041, 10116, '0/25137B0','pg_variable', 2200,
> 10,-1,0, NULL)::pg_variable;
>
> select pg_variable.oid from pg_variable where pg_variable.oid =
> pg_variable.oid;
> this query, the WHERE clause, it's really hard to distinguish session
> variable or column reference.
> I am not sure if this is fine or not.
>

  361. Re: Re: proposal: schema variables

    jian he <jian.universality@gmail.com> — 2025-02-07T13:14:09Z

    On Fri, Feb 7, 2025 at 3:25 PM Pavel Stehule <pavel.stehule@gmail.com> wrote:
>

Hi
The following review is based on v20250117.

pg_dump order seems not right.
CREATE FUNCTION public.test11(text) RETURNS text
    LANGUAGE sql
    AS $$select v4 $$;
CREATE VARIABLE public.v4 AS text;

first dump function then variable. restore would fail.
we should first dump variables then function.
probably placed it right after CREATE DOMAIN/CREATE TYPE


drop table if exists t3;
create variable v4 as text;
let v4 = 'hello';
CREATE TABLE t3 (a timestamp, v4 text);
INSERT INTO t3 SELECT i FROM generate_series('2020-01-01'::timestamp,
                                             '2020-12-31'::timestamp,
                                             '10 minute'::interval) s(i);
ANALYZE t3;
create or replace function test11(text) returns text as $$select v4 $$
language sql;
CREATE STATISTICS s4 (ndistinct) ON test11(v4), test11(v4 || 'h') FROM t3;
this "CREATE STATISTICS s4..." should error out?


any objects built on top of functions that use variables should be
marked as volatile.
and we should also consider the implications of it.

  362. Re: Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-02-09T20:56:54Z

    Hi


> create domain dnn as int not null;
> CREATE VARIABLE var3 AS dnn;
> alter domain dnn drop not null;
> alter domain dnn set not null;
> ERROR:  cannot alter domain "dnn" because session variable "public.var3"
> uses it
>

> This is not great.
> we allow a constraint, then we drop it, now we cannot re add it again.
>

I think this is correct. And I am afraid so we need to choose from two bad
things. There is not possible any other way

The data of session variables are not shared, so we cannot recheck if data
are valid against new constraints or not.

Then there was two possibilities (and not any other)
a) disallow add new constraints
b) ignore possible inconsistency

I implemented @a (can be easily removed), for example temp tables uses @b

you can try - in one session do modification of domain
and in second session do just


create temp table t (a dnn);
-- after drop not null
insert into t values(null);
-- you can successfully set not null to dnn
insert into t values(null); -- now fails
but select * from t -- and the t has null in a


So unfortunately there is no good solution - you can prefer consistency
(and stop altering) or you can allow altering (and stop consistency).

Usage of variables disallow type altering already - so disallowing altering
domain looks more correct - but I am open to any discussion. I can change
the behaviour the same way like temp tables.
In this case I don't see any reason be extra dogmatic - after any update or
reset, the value will be corrected for new constraints, and what is
important - although the value will not be fully consistent for domain
type, it is still fully consistent for base type, so there is not risk of
crashes - and after disconnect of all sessions, the all inconsistent values
will be lost equally like data of temp tables. Now, a more restrictive
variant is implemented - so it is safe, and future change should not have
compatibility issues.


> 0001 and 0002 are simple, but the size is still large.
> maybe we can not support the domain in the first 2 patches.
>

Removing domain support decrease the size about few kilobytes


> also
> CREATE VARIABLE var3 AS dnn;
> let var3 = 11;
> create view v1 as select var3;
> select * from v1;
> you can see reconnect to session then
> ERROR:  domain dnn does not allow null values
> this is not ok?
>

this is ok

the content of session variables is not shared. The variable in the second
session is initialized to null, and the null is disallowed.

The variable is initialized when it is used for the first time in the
session. And it is initialized to default value - without default value,
it is initialized to NULL (possibility to set default does different
patch). So when you use a view for the first time, then the variable is
initialized
and it fails.

It is consistent with plpgsql

(2025-02-09 20:23:14) postgres=#
do $$
declare x dnn;
begin
end
$$;
ERROR:  domain dnn does not allow null values
CONTEXT:  PL/pgSQL function inline_code_block line 2 during statement block
local variable initialization

There is no raised syntax error or some compilation error. The raised error
is just runtime initialization error like session variables


>
>
> also
> create table t(var3 int);
> CREATE POLICY p1r ON t AS RESTRICTIVE TO alice USING (var3 <> var3);
> create table t1(a int);
> CREATE POLICY p2 ON t1 AS RESTRICTIVE TO alice USING (a <> var3);
> p1r is so confusing. there is no way to understand the intention.
> p2 should also not be allowed, since var3 value is volatile,
> session reconnection will change the value.
>

One of the main targets of session variables is using RLS.

Postgres allows to

`CREATE POLICY p2 ON t1 AS RESTRICTIVE TO pavel USING (a <>
current_user::regrole::int)`

and I don't need reconnect - just I can use `set role to ...` or I can use
security definer function

Inside the queries, the session variables are stable, because the values
are passed as copy when the query is started.

This can be changed by wrapping a volatile function, but on the top level,
the variables are stable (we talked about implementation of really volatile
variables), but it is not implemented.

create variable v int;

CREATE OR REPLACE FUNCTION public.fx()
 RETURNS integer
 LANGUAGE plpgsql
AS $function$
begin
  let v = v + 1;
  return v;
end;
$function$

(2025-02-09 20:41:34) postgres=# select v, fx() from generate_series(1,10);
┌───┬────┐
│ v │ fx │
╞═══╪════╡
│ 5 │  6 │
│ 5 │  7 │
│ 5 │  8 │
│ 5 │  9 │
│ 5 │ 10 │
│ 5 │ 11 │
│ 5 │ 12 │
│ 5 │ 13 │
│ 5 │ 14 │
│ 5 │ 15 │
└───┴────┘
(10 rows)

The current behave is similar to usage plpgsql variables in queries - the
values are copied and are stable for the query

(2025-02-09 20:49:15) postgres=# do $$
declare
  r record;
  v int default 0;
begin
  for r in select v, i from generate_series(1,3) g(i)
  loop
    v := v + 1;
    raise notice '%', r;
  end loop;
end$$;
NOTICE:  (0,1)
NOTICE:  (0,2)
NOTICE:  (0,3)
DO


>
>
> src/bin/pg_dump/pg_dump.h
> /*
>  * The VariableInfo struct is used to represent session variables
>  */
> typedef struct _VariableInfo
> {
>     DumpableObject dobj;
>     DumpableAcl dacl;
>     Oid            vartype;
>     char       *vartypname;
>     char       *varacl;
>     char       *rvaracl;
>     char       *initvaracl;
>     char       *initrvaracl;
>     Oid            varcollation;
>     const char *rolname;        /* name of owner, or empty string */
> } VariableInfo;
> these fields (varacl, rvaracl, initvaracl, initrvaracl) were never being
> used.
> we can remove them.
>

removed


>
>         CollInfo   *coll;
>         coll = findCollationByOid(varcollation);
>         if (coll)
>             appendPQExpBuffer(query, " COLLATE %s",
>                               fmtQualifiedDumpable(coll));
> here, it should be
> ```CollInfo   *coll = NULL;```?
>

No, why? The coll value is set by findCollationByOid every time, so
initialization is useless.


>
> I don't understand the changes made in getAdditionalACLs.
> I thought pg_init_privs had nothing to do with the session variable.
>

yes, it is useless

removed


>
>
> minor issue in getVariables.
>     query = createPQExpBuffer();
>     resetPQExpBuffer(query);
> no need to use resetPQExpBuffer here.
>

fixed


>
>
> create type ab as (a int, b text);
> create type abc as (a ab, c text);
> create type abcd as (a abc, d text);
> CREATE VARIABLE var2 AS abcd;
>
> select var2.a.c;
> ERROR:  cross-database references are not implemented: var2.a.c
>
> Is this error what we expected? I am not 100% sure.
>

Unfortunately, it is expected. Postgres parser doesn't support direct
access to nested composite types. It can be designed differently, but it is
implemented
to be consistent with current postgres behavior.

create type ab as (a int, b text);
create type abc as (a ab, c text);
create type abcd as (a abc, d text);
create table foo(a abc);

(2025-02-09 06:53:36) postgres=# select foo.a.a from  foo;
ERROR:  cross-database references are not implemented: foo.a.a

you should to use parenthesis

(2025-02-09 06:53:42) postgres=# select (foo.a).a from  foo;
┌───┐
│ a │
╞═══╡
└───┘
(0 rows)



--------------------another contrived corner case.-----------------------
> create type pg_variable as (
>   oid oid, vartype oid, varcreate_lsn pg_lsn,
>   varname name, varnamespace oid, varowner oid,
>   vartypmod int, varcollation oid, varacl aclitem[]);
> create variable pg_variable as pg_variable;
> let pg_variable = row (18041, 10116, '0/25137B0','pg_variable', 2200,
> 10,-1,0, NULL)::pg_variable;
>
> select pg_variable.oid from pg_variable where pg_variable.oid =
> pg_variable.oid;
> this query, the WHERE clause, it's really hard to distinguish session
> variable or column reference.
> I am not sure if this is fine or not.
>

Yes, the basic rule is - don't name the variable exactly like the table or
like the column.  Better use a dedicated schema that should not be on the
search path.

There are some prepared tools already

create variable pg_variable as pg_variable;


* patch 07

set session_variables_ambiguity_warning to on;

(2025-02-09 21:09:22) postgres=# select pg_variable.oid from pg_variable
where pg_variable.oid = pg_variable.oid;
WARNING:  session variable "pg_variable.oid" is shadowed
LINE 1: select pg_variable.oid from pg_variable where pg_variable.oi...
               ^
DETAIL:  Session variables can be shadowed by columns, routine's variables
and routine's arguments with the same name.
WARNING:  session variable "pg_variable.oid" is shadowed
LINE 1: select pg_variable.oid from pg_variable where pg_variable.oi...
                                                      ^
DETAIL:  Session variables can be shadowed by columns, routine's variables
and routine's arguments with the same name.
WARNING:  session variable "pg_variable.oid" is shadowed
LINE 1: ...able.oid from pg_variable where pg_variable.oid = pg_variabl...
                                                             ^
DETAIL:  Session variables can be shadowed by columns, routine's variables
and routine's arguments with the same name.
┌───────┐
│  oid  │
╞═══════╡
│ 16389 │
└───────┘
(1 row)


* patch 08 - variable fencing - it helps with forcing of usage of some
identifier like the variable

(2025-02-09 21:11:58) postgres=# set session_variables_ambiguity_warning to
off;
SET
(2025-02-09 21:12:05) postgres=# let pg_variable.oid = 16389;
LET
(2025-02-09 21:12:07) postgres=# select pg_variable.oid from pg_variable
where pg_variable.oid = variable(pg_variable).oid;
┌───────┐
│  oid  │
╞═══════╡
│ 16389 │
└───────┘
(1 row)

(2025-02-09 21:12:10) postgres=# select pg_variable.oid from pg_variable
where pg_variable.oid = variable(pg_variable.oid);
┌───────┐
│  oid  │
╞═══════╡
│ 16389 │
└───────┘
(1 row)

* patch 09 - possibility to force usage of variable fencing in some
contexts - it can raise an error when the session variable was used,
but without fencing (in some contexts). Now the contextes are "none, all,
nonspi", but probably better (and more intuitive) will be (and +/- equal)
"none, all, nested".
patch 09 implements some protection against unwanted usage of the session
variable.

I think these tools are enough for safe working with session variables. But
(similarly to PL/pgSQL or PL/SQL) the basic rule is usage of names that are
not in possible collisions.

I don't want to force dedicated schema or dedicated syntax - because it can
be a problem for porting from other databases. But common style for
plpgsql or plsql is using some form of hungarian notation or using prefixes
like _ for local variable and __ for global variable - or using schema name
like package name
schema_name.varname

regards

Pavel

  363. Re: Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-02-11T07:11:29Z

    Hi

pá 7. 2. 2025 v 14:14 odesílatel jian he <jian.universality@gmail.com>
napsal:

> On Fri, Feb 7, 2025 at 3:25 PM Pavel Stehule <pavel.stehule@gmail.com>
> wrote:
> >
>
> Hi
> The following review is based on v20250117.
>
> pg_dump order seems not right.
> CREATE FUNCTION public.test11(text) RETURNS text
>     LANGUAGE sql
>     AS $$select v4 $$;
> CREATE VARIABLE public.v4 AS text;
>
> first dump function then variable. restore would fail.
> we should first dump variables then function.
> probably placed it right after CREATE DOMAIN/CREATE TYPE
>

I cannot repeat this issue. The import should to work, because dump
contains `SET check_function_bodies = false;`
The order is designed to support default values, and the default value can
be a function, so the variables should be
dumped after functions.

I tested the new syntax too. And you can see, the order for new syntax is
changed due dependencies

SET check_function_bodies = false;
SET xmloption = content;
SET client_min_messages = warning;
SET row_security = off;

--
-- Name: fx1(); Type: FUNCTION; Schema: public; Owner: pavel
--

CREATE FUNCTION public.fx1() RETURNS integer
    LANGUAGE sql
    AS $$ select v1 $$;


ALTER FUNCTION public.fx1() OWNER TO pavel;

--
-- Name: v1; Type: VARIABLE; Schema: public; Owner: pavel
--

CREATE VARIABLE public.v1 AS integer;


ALTER VARIABLE public.v1 OWNER TO pavel;

--
-- Name: fx3(); Type: FUNCTION; Schema: public; Owner: pavel
--

CREATE FUNCTION public.fx3() RETURNS integer
    LANGUAGE sql
    RETURN public.v1;


ALTER FUNCTION public.fx3() OWNER TO pavel;



>
>
> drop table if exists t3;
> create variable v4 as text;
> let v4 = 'hello';
> CREATE TABLE t3 (a timestamp, v4 text);
> INSERT INTO t3 SELECT i FROM generate_series('2020-01-01'::timestamp,
>                                              '2020-12-31'::timestamp,
>                                              '10 minute'::interval) s(i);
> ANALYZE t3;
> create or replace function test11(text) returns text as $$select v4 $$
> language sql;
> CREATE STATISTICS s4 (ndistinct) ON test11(v4), test11(v4 || 'h') FROM t3;
> this "CREATE STATISTICS s4..." should error out?
>
>
> any objects built on top of functions that use variables should be
> marked as volatile.
> and we should also consider the implications of it.
>

There is not any request so expression of statistics should be immutable,
although it makes sense (for me).

(2025-02-11 07:48:28) postgres=# create table t4(a int, b int);
CREATE TABLE
(2025-02-11 07:52:32) postgres=# create statistics s5 (ndistinct) on a, (b
* (random()*10)::int) from t4;
CREATE STATISTICS

The access to variables in the query is stable (when it is not wrapped by
volatile functions - because variables
are passed as query parameters to the queries.

I think it is working correctly

(2025-02-11 07:54:58) postgres=# create or replace function fx20() returns
int as $$ let x = x + 1; select x $$ language sql;
CREATE FUNCTION
(2025-02-11 07:55:49) postgres=# let x = 0;
LET
(2025-02-11 07:55:57) postgres=# select x, fx20(), i from
generate_series(1,3) g(i);
┌───┬──────┬───┐
│ x │ fx20 │ i │
╞═══╪══════╪═══╡
│ 0 │    1 │ 1 │
│ 0 │    2 │ 2 │
│ 0 │    3 │ 3 │
└───┴──────┴───┘
(3 rows)

I rewrote the patch 09 - the forsing usage of variable fences
(VARIABLE(varname)). There are two possible concepts:

1. controlling visibility of session variables - the variables without
fencing are visible somewhere, inside fencing are visible everywhere
2. forcing usage of variable fence syntax and raising an error when the
variable is used without fence.

Originally I implemented @2, but the disadvantage can be a lot of errors (a
lot of warnings related to shadowed variables), so this is not a good
default due to the possibility of a lot of writes to log. On the other
hand, there are mystic hidden variable issues. Concept @1 can be more
simple for gentle introducing of variables. And I can believe it can be a
good safe default - the variables are implicitly visible only inside stored
procedures, outside stored procedures, the variable fencing should be used,
and without variable fence - just the variable is not accessible (so there
is not an issue with shadowing warning). The code change between @1 and @2
concepts are just the change of two lines of code. The concept @1 is a
little bit similar to search path (where using variable fencing is like
using a qualified name). So I think it is closer to current postgres
concepts.

Regards

Pavel

  364. Re: Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-02-12T04:43:24Z

    Hi

rebase after a654af21ae522cc8e867e052defd16f76ffaef2e

Regards

Pavel

  365. Re: Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-02-16T09:13:38Z

    Hi

I removed a patch 09 - possibility to set implicit visibility of session
variables. This patch was a replacement of patch - possibility to check the
usage of variable fences.

This specific feature can be implemented in many ways, and at this moment
is useless to try to implement it.

Regards

Pavel

  366. Re: Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-02-20T20:22:15Z

    Hi

forced rebase

Regards

Pavel

  367. Re: Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-03-01T07:23:24Z

    Hi

fix regress tests after 95dbd827f2edc4d10bebd7e840a0bd6782cf69b7

Regards

Pavel

  368. Re: Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-03-17T18:32:59Z

    Hi

I was asked for sending a reduced patchset

Regards

Pavel

  369. Re: Re: proposal: schema variables

    Marcos Pegoraro <marcos@f10.com.br> — 2025-03-17T20:52:38Z

    Em seg., 17 de mar. de 2025 às 15:33, Pavel Stehule <pavel.stehule@gmail.com>
escreveu:

> I was asked for sending a reduced patchset
>

Would be good to explain what this reduced patchset is.
Complete patch contains this and that
Reduced patch contains only this.

Regards
Marcos

  370. Re: Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-03-18T05:46:09Z

    Hi

po 17. 3. 2025 v 21:53 odesílatel Marcos Pegoraro <marcos@f10.com.br>
napsal:

> Em seg., 17 de mar. de 2025 às 15:33, Pavel Stehule <
> pavel.stehule@gmail.com> escreveu:
>
>> I was asked for sending a reduced patchset
>>
>
> Would be good to explain what this reduced patchset is.
> Complete patch contains this and that
> Reduced patch contains only this.
>

Reduced patch contains:

* possibility to create and drop session variables (support for catalog
pg_variable and related operations)
* possibility to set the session variable (command LET) and possibility to
use session variable in the query
* access right support
* support for DISCARD command
* memory cleaning at transaction end when the variable is dropped
* warning when variable is shadowed by column
* introduction of variable's fences - syntax VARIABLE(varname)

Complete patch contains plus

* LET can be described by EXPLAIN
* LET can be prepared statement
* temporary session variables
* RESET at transaction end
* implementation of DEFAULT value for the variable
* implementation IMMUTABLE and NOT NULL clauses for the variable
* variable can be used as an argument of CALL statement (and doesn't block
simple evaluation in plpgsql)
* used variable doesn't block with parallel execution
* LET from plpgsql can use simple expression evaluation (performance
optimization for PL/pgSQL)
* variables doesn't block inlining SQL functions
* fix message "column doesn't exist" to "column or variable doesn't exist"
* support for transactional variables (content can be transactional)
* possibility to specify the name of restored variable for pg_restore

Regards

Pavel





>
> Regards
> Marcos
>

  371. Re: Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-04-02T06:46:48Z

    Hi

necessary rebase

Regards

Pavel

  372. Re: Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-04-05T10:33:21Z

    Hi

only rebase

Regards

Pavel

  373. Re: Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-05-15T06:48:37Z

    Hi

fresh rebase

Regards

Pavel

  374. Re: Re: proposal: schema variables

    Bruce Momjian <bruce@momjian.us> — 2025-05-20T14:56:29Z

    On Thu, May 15, 2025 at 08:48:37AM +0200, Pavel Stehule wrote:
> Hi
> 
> fresh rebase

I will again ask why this patch set is being reposted when there is no
plan to apply it to git master?

-- 
  Bruce Momjian  <bruce@momjian.us>        https://momjian.us
  EDB                                      https://enterprisedb.com

  Do not let urgent matters crowd out time for investment in the future.

  375. Re: Re: proposal: schema variables

    Marcos Pegoraro <marcos@f10.com.br> — 2025-05-20T16:33:18Z

    Em ter., 20 de mai. de 2025 às 11:56, Bruce Momjian <bruce@momjian.us>
escreveu:

> I will again ask why this patch set is being reposted when there is no
> plan to apply it to git master


Too bad. I would love to have this functionality, from the user's point of
view there are problems where it would solve them wonderfully. I don't know
technically of what prevents it from being natively on core, but it would
be great, it would definitely be.

regards
Marcos

  376. Re: Re: proposal: schema variables

    Bruce Momjian <bruce@momjian.us> — 2025-05-20T16:39:19Z

    On Tue, May 20, 2025 at 01:33:18PM -0300, Marcos Pegoraro wrote:
> Em ter., 20 de mai. de 2025 às 11:56, Bruce Momjian <bruce@momjian.us>
> escreveu:
> 
>     I will again ask why this patch set is being reposted when there is no
>     plan to apply it to git master
> 
> Too bad. I would love to have this functionality, from the user's point of view
> there are problems where it would solve them wonderfully. I don't know
> technically of what prevents it from being natively on core, but it would be
> great, it would definitely be.

My only point is that we should only be using email lists for work that
is being actively worked on to be added to community Postgres.  There
has been talk of a trimmed-down version of this being applied, but I
don't see any work in that direction.

This patch should be moved to a separate location where perhaps people
can subscribe to updates when they are posted, perhaps github.

-- 
  Bruce Momjian  <bruce@momjian.us>        https://momjian.us
  EDB                                      https://enterprisedb.com

  Do not let urgent matters crowd out time for investment in the future.

  377. Re: proposal: schema variables

    Daniel Gustafsson <daniel@yesql.se> — 2025-05-20T18:47:36Z

    > On 20 May 2025, at 18:39, Bruce Momjian <bruce@momjian.us> wrote:
> 
> On Tue, May 20, 2025 at 01:33:18PM -0300, Marcos Pegoraro wrote:
>> Em ter., 20 de mai. de 2025 às 11:56, Bruce Momjian <bruce@momjian.us>
>> escreveu:
>> 
>>    I will again ask why this patch set is being reposted when there is no
>>    plan to apply it to git master
>> 
>> Too bad. I would love to have this functionality, from the user's point of view
>> there are problems where it would solve them wonderfully. I don't know
>> technically of what prevents it from being natively on core, but it would be
>> great, it would definitely be.
> 
> My only point is that we should only be using email lists for work that
> is being actively worked on to be added to community Postgres.  There
> has been talk of a trimmed-down version of this being applied, but I
> don't see any work in that direction.
> 
> This patch should be moved to a separate location where perhaps people
> can subscribe to updates when they are posted, perhaps github.

As a project with no roadmap governed by open forum consensus I don't think we
have any right to tell community members what they can or cannot work on here,
any technical discussion which conforms with our published policies should be
welcome.  If Pavel want's to continue rebasing his patchset here then he has,
IMHO, every right to do so.

Whether or not a committer will show interest at some point is another thing,
but we are seeing a very good role-model for taking responsibility for ones
work here at the very least =)

--
Daniel Gustafsson

  378. Re: proposal: schema variables

    Bruce Momjian <bruce@momjian.us> — 2025-05-20T20:28:01Z

    On Tue, May 20, 2025 at 08:47:36PM +0200, Daniel Gustafsson wrote:
> > On 20 May 2025, at 18:39, Bruce Momjian <bruce@momjian.us> wrote:
> > My only point is that we should only be using email lists for work that
> > is being actively worked on to be added to community Postgres.  There
> > has been talk of a trimmed-down version of this being applied, but I
> > don't see any work in that direction.
> > 
> > This patch should be moved to a separate location where perhaps people
> > can subscribe to updates when they are posted, perhaps github.
> 
> As a project with no roadmap governed by open forum consensus I don't think we
> have any right to tell community members what they can or cannot work on here,
> any technical discussion which conforms with our published policies should be
> welcome.  If Pavel want's to continue rebasing his patchset here then he has,
> IMHO, every right to do so.
> 
> Whether or not a committer will show interest at some point is another thing,
> but we are seeing a very good role-model for taking responsibility for ones
> work here at the very least =)

Well, we do have a right, e.g., we would not allow someone to repeatedly
post patches for a Postgres extension we don't manage, or the jdbc
driver.  I also don't think we would allow someone to continue posting
patches for a feature we have decided to reject, and I think we have
decided to reject the patch in in its current form.  I think we might
accept a trimmed-down version, but I don't see the patch moving in that
direction.

Now, of course, if I am the only one who feels this way, I can suppress
these emails on my end.

-- 
  Bruce Momjian  <bruce@momjian.us>        https://momjian.us
  EDB                                      https://enterprisedb.com

  Do not let urgent matters crowd out time for investment in the future.

  379. Re: Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-05-20T20:28:31Z

    Hi

út 20. 5. 2025 v 18:39 odesílatel Bruce Momjian <bruce@momjian.us> napsal:

> On Tue, May 20, 2025 at 01:33:18PM -0300, Marcos Pegoraro wrote:
> > Em ter., 20 de mai. de 2025 às 11:56, Bruce Momjian <bruce@momjian.us>
> > escreveu:
> >
> >     I will again ask why this patch set is being reposted when there is
> no
> >     plan to apply it to git master
> >
> > Too bad. I would love to have this functionality, from the user's point
> of view
> > there are problems where it would solve them wonderfully. I don't know
> > technically of what prevents it from being natively on core, but it
> would be
> > great, it would definitely be.
>
> My only point is that we should only be using email lists for work that
> is being actively worked on to be added to community Postgres.  There
> has been talk of a trimmed-down version of this being applied, but I
> don't see any work in that direction.
>

I sent a reduced version a few months ago - from 21 patches to 8 (and it
can be reduced to six if we postpone tools for detection ambiguity).
The timing was not perfect - the focus was and it is concentrated to finish
pg18.

I am very sorry if this topic and patches bother anyone. I am afraid if I
close it to some personal github, it will not be visible, and I am sure this
feature is missing in Postgres. Today we have few workarounds. Some
workarounds are not available everywhere, some workarounds cannot
be used for security. With integrated solutions some scenarios can be done
more easily, more secure, faster, more comfortable.  It is true, so
mentioned scenarios are not "hot" today. Stored procedures or RLS or
migration procedures from other databases are not extra common. But
who uses it, then he misses session variables.

This topic is difficult, because there is no common solution. SQL/PSM is
almost dead. T-SQL (and MySQL) design is weak and cannot be used for
security.
Oracle's design is joined with just one environment. And although almost
all widely used databases have supported session variables for decades, no
one design
is perfect. Proposed design is not perfect too (it introduces possible
ambiguity) , but I think it can support most wanted use cases (can be
enhanced in future),
and it is consistent with Postgres. There are more ways to reduce risk of
unwanted ambiguity to zero. But it increases the size of the patch.

Regards

Pavel







>
> This patch should be moved to a separate location where perhaps people
> can subscribe to updates when they are posted, perhaps github.
>
> --
>   Bruce Momjian  <bruce@momjian.us>        https://momjian.us
>   EDB                                      https://enterprisedb.com
>
>   Do not let urgent matters crowd out time for investment in the future.
>

  380. Re: proposal: schema variables

    Laurenz Albe <laurenz.albe@cybertec.at> — 2025-05-20T20:36:54Z

    On Tue, 2025-05-20 at 16:28 -0400, Bruce Momjian wrote:
> On Tue, May 20, 2025 at 08:47:36PM +0200, Daniel Gustafsson wrote:
> > > On 20 May 2025, at 18:39, Bruce Momjian <bruce@momjian.us> wrote:
> > > My only point is that we should only be using email lists for work that
> > > is being actively worked on to be added to community Postgres.  There
> > > has been talk of a trimmed-down version of this being applied, but I
> > > don't see any work in that direction.
> > > 
> > > This patch should be moved to a separate location where perhaps people
> > > can subscribe to updates when they are posted, perhaps github.
> > 
> > As a project with no roadmap governed by open forum consensus I don't think we
> > have any right to tell community members what they can or cannot work on here,
> > any technical discussion which conforms with our published policies should be
> > welcome.  If Pavel want's to continue rebasing his patchset here then he has,
> > IMHO, every right to do so.
> > 
> > Whether or not a committer will show interest at some point is another thing,
> > but we are seeing a very good role-model for taking responsibility for ones
> > work here at the very least =)
> 
> Well, we do have a right, e.g., we would not allow someone to repeatedly
> post patches for a Postgres extension we don't manage, or the jdbc
> driver.  I also don't think we would allow someone to continue posting
> patches for a feature we have decided to reject, and I think we have
> decided to reject the patch in in its current form.  I think we might
> accept a trimmed-down version, but I don't see the patch moving in that
> direction.
> 
> Now, of course, if I am the only one who feels this way, I can suppress
> these emails on my end.

In my opinion, this patch set is adding something that would be valuable to
have in core.

If no committer intends to pick it up and commit it, I think the proper
action would be to step up and reject the patch set, not complain about the
insistence of the author.

Yours,
Laurenz Albe

  381. Re: proposal: schema variables

    Bruce Momjian <bruce@momjian.us> — 2025-05-20T21:06:59Z

    On Tue, May 20, 2025 at 10:36:54PM +0200, Laurenz Albe wrote:
> On Tue, 2025-05-20 at 16:28 -0400, Bruce Momjian wrote:
> > Well, we do have a right, e.g., we would not allow someone to repeatedly
> > post patches for a Postgres extension we don't manage, or the jdbc
> > driver.  I also don't think we would allow someone to continue posting
> > patches for a feature we have decided to reject, and I think we have
> > decided to reject the patch in in its current form.  I think we might
> > accept a trimmed-down version, but I don't see the patch moving in that
> > direction.
> > 
> > Now, of course, if I am the only one who feels this way, I can suppress
> > these emails on my end.
> 
> In my opinion, this patch set is adding something that would be valuable to
> have in core.
> 
> If no committer intends to pick it up and commit it, I think the proper
> action would be to step up and reject the patch set, not complain about the
> insistence of the author.

Are you saying I should not complain until we have officially rejected
the patch set?  If we officially reject it, the patch author would no
longer post it?

-- 
  Bruce Momjian  <bruce@momjian.us>        https://momjian.us
  EDB                                      https://enterprisedb.com

  Do not let urgent matters crowd out time for investment in the future.

  382. Re: Re: proposal: schema variables

    Bruce Momjian <bruce@momjian.us> — 2025-05-20T21:10:09Z

    On Tue, May 20, 2025 at 10:28:31PM +0200, Pavel Stehule wrote:
> út 20. 5. 2025 v 18:39 odesílatel Bruce Momjian <bruce@momjian.us> napsal:
> I sent a reduced version a few months ago - from 21 patches to 8 (and it can be
> reduced to six if we postpone tools for detection ambiguity).
> The timing was not perfect - the focus was and it is concentrated to finish
> pg18.

It was not clear to me that this patch set was being reduced to make it
more likely it would be accepted?  I thought it was the same patch set
since 20??.

> I am very sorry if this topic and patches bother anyone. I am afraid if I close
> it to some personal github, it will not be visible, and I am sure this
> feature is missing in Postgres. Today we have few workarounds. Some workarounds
> are not available everywhere, some workarounds cannot
> be used for security. With integrated solutions some scenarios can be done more
> easily, more secure, faster, more comfortable.  It is true, so
> mentioned scenarios are not "hot" today. Stored procedures or RLS or migration
> procedures from other databases are not extra common. But
> who uses it, then he misses session variables.

Understood.  If people feel it is progressing toward acceptance, I
certainly withdraw my objection and apologize.

-- 
  Bruce Momjian  <bruce@momjian.us>        https://momjian.us
  EDB                                      https://enterprisedb.com

  Do not let urgent matters crowd out time for investment in the future.

  383. Re: Re: proposal: schema variables

    Michael Paquier <michael@paquier.xyz> — 2025-05-21T00:21:42Z

    On Tue, May 20, 2025 at 10:28:31PM +0200, Pavel Stehule wrote:
> This topic is difficult, because there is no common solution. SQL/PSM is
> almost dead. T-SQL (and MySQL) design is weak and cannot be used for
> security.
> Oracle's design is joined with just one environment. And although almost
> all widely used databases have supported session variables for decades, no
> one design
> is perfect. Proposed design is not perfect too (it introduces possible
> ambiguity) , but I think it can support most wanted use cases (can be
> enhanced in future),
> and it is consistent with Postgres. There are more ways to reduce risk of
> unwanted ambiguity to zero. But it increases the size of the patch.

One thing that I keep hearing about this feature is that this would be
really helpful for migration from Oracle to PostgreSQL, helping a lot
with rewrites of pl/pgsql functions.

There is one page on the wiki about private variables, dating back to
2016:
https://wiki.postgresql.org/wiki/CREATE_PRIVATE_VARIABLE

Perhaps it would help to summarize a bit the pros and cons of existing
implementations to drive which implementation would be the most suited
on a wiki page?  The way standards are defined is by overwriting
existing standards, and we don't have one in the SQL specification.
It's hard to say if there will be one at some point, but if the main
SQL products around the world have one, it pretty much is a point in
favor of having a standard.

Another possible angle that could be taken is to invest in a proposal
for the SQL committee to consider, forcing an actual standard that
PostgreSQL could rely on rather than having one behavior implemented
to have it standard-incompatible a few years after.  And luckily, we
have Vik Fearing and Peter Eisentraut in this community who invest
time and resources in this area.

FWIW, I do agree with the opinion that if you want to propose rebased
versions of this patch set on a periodic basis, you are free to do so.
This is the core concept of an open community.  In terms of my
committer time, I'm pretty much already booked in terms of features
planned for the next release, so I guess that I won't be able to
invest time on this thread.  Just saying.

I know that this patch set has been discussed at FOSDEM at some point,
so others may be able to comment more about that.  That's just one
opinion among many others.
--
Michael

  384. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-05-21T05:15:27Z

    út 20. 5. 2025 v 23:07 odesílatel Bruce Momjian <bruce@momjian.us> napsal:

> On Tue, May 20, 2025 at 10:36:54PM +0200, Laurenz Albe wrote:
> > On Tue, 2025-05-20 at 16:28 -0400, Bruce Momjian wrote:
> > > Well, we do have a right, e.g., we would not allow someone to
> repeatedly
> > > post patches for a Postgres extension we don't manage, or the jdbc
> > > driver.  I also don't think we would allow someone to continue posting
> > > patches for a feature we have decided to reject, and I think we have
> > > decided to reject the patch in in its current form.  I think we might
> > > accept a trimmed-down version, but I don't see the patch moving in that
> > > direction.
> > >
> > > Now, of course, if I am the only one who feels this way, I can suppress
> > > these emails on my end.
> >
> > In my opinion, this patch set is adding something that would be valuable
> to
> > have in core.
> >
> > If no committer intends to pick it up and commit it, I think the proper
> > action would be to step up and reject the patch set, not complain about
> the
> > insistence of the author.
>
> Are you saying I should not complain until we have officially rejected
> the patch set?  If we officially reject it, the patch author would no
> longer post it?
>

I'll respect committers. I really don't want to worry people in the
community. It is not my way, and I am sorry.

I think this is an important feature - for some group of developers, and
then I push my energy and time for this.
On the other hand, I accept that there is a lot of work that is important
for a wider group of users. So it is. Unfortunately,
without parser's hooks this feature cannot be implemented as an extension.
But parser's hooks was proposed,
and rejected, so there is no other possibility, how to do it. More -
implementation of this feature as an extension is not
best way.

regards

Pavel








>
> --
>   Bruce Momjian  <bruce@momjian.us>        https://momjian.us
>   EDB                                      https://enterprisedb.com
>
>   Do not let urgent matters crowd out time for investment in the future.
>

  385. Re: Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-05-21T05:49:34Z

    st 21. 5. 2025 v 2:21 odesílatel Michael Paquier <michael@paquier.xyz>
napsal:

> On Tue, May 20, 2025 at 10:28:31PM +0200, Pavel Stehule wrote:
> > This topic is difficult, because there is no common solution. SQL/PSM is
> > almost dead. T-SQL (and MySQL) design is weak and cannot be used for
> > security.
> > Oracle's design is joined with just one environment. And although almost
> > all widely used databases have supported session variables for decades,
> no
> > one design
> > is perfect. Proposed design is not perfect too (it introduces possible
> > ambiguity) , but I think it can support most wanted use cases (can be
> > enhanced in future),
> > and it is consistent with Postgres. There are more ways to reduce risk of
> > unwanted ambiguity to zero. But it increases the size of the patch.
>
> One thing that I keep hearing about this feature is that this would be
> really helpful for migration from Oracle to PostgreSQL, helping a lot
> with rewrites of pl/pgsql functions.
>
> There is one page on the wiki about private variables, dating back to
> 2016:
> https://wiki.postgresql.org/wiki/CREATE_PRIVATE_VARIABLE
>
>
I wrote mail

https://www.postgresql.org/message-id/CAFj8pRB8kdWQCdN2X1_63c58+07Oy4Z+ruDK_xPTUP+Pe8R2Pw@mail.gmail.com

and there is another wiki page
https://wiki.postgresql.org/wiki/Variable_Design



> Perhaps it would help to summarize a bit the pros and cons of existing
> implementations to drive which implementation would be the most suited
> on a wiki page?  The way standards are defined is by overwriting
> existing standards, and we don't have one in the SQL specification.
> It's hard to say if there will be one at some point, but if the main
> SQL products around the world have one, it pretty much is a point in
> favor of having a standard.
>

Although it is  maybe a peccant idea - I can imagine two different
implementations of server side session variables with different syntaxes
(and different advantages and disadvantages, and different use cases). The
implementations are not going against, but we should to accept
fact, so one feature is implemented twice. We should choose just one, that
will be implemented first. Proposed helps with migration from
PL/SQL.


>
> Another possible angle that could be taken is to invest in a proposal
> for the SQL committee to consider, forcing an actual standard that
> PostgreSQL could rely on rather than having one behavior implemented
> to have it standard-incompatible a few years after.  And luckily, we
> have Vik Fearing and Peter Eisentraut in this community who invest
> time and resources in this area.
>

Theoretically the proposed design is a subset of implementation from DB2 -
I designed it without knowledge of this DB2 feature. But without
introduction of concept of modules (that is partially redundant to
schemas), this design is very natural and I am very sure, so there is not
another way, how to design it. We can ask Peter or Vik about real
possibilities in this area. I have not any information from this area, just
I haven't seen the changes in SQL/PSM for decades, so I didn't think  about
it.



> FWIW, I do agree with the opinion that if you want to propose rebased
> versions of this patch set on a periodic basis, you are free to do so.
> This is the core concept of an open community.  In terms of my
> committer time, I'm pretty much already booked in terms of features
> planned for the next release, so I guess that I won't be able to
> invest time on this thread.  Just saying.
>

thank you for an info


> I know that this patch set has been discussed at FOSDEM at some point,
> so others may be able to comment more about that.  That's just one
> opinion among many others.
> --
> Michael
>

  386. Re: proposal: schema variables

    Laurenz Albe <laurenz.albe@cybertec.at> — 2025-05-21T06:27:13Z

    On Tue, 2025-05-20 at 17:06 -0400, Bruce Momjian wrote:
> f no committer intends to pick it up and commit it, I think the proper
> > action would be to step up and reject the patch set, not complain about the
> > insistence of the author.
> 
> Are you saying I should not complain until we have officially rejected
> the patch set?  If we officially reject it, the patch author would no
> longer post it?

No, you are free to complain, just as Pavel is free to keep his patch
set from rotting.  What I am trying to say is that rejecting the patch
set is the most effective way to complain.

The current state is that Pavel keeps the patch set alive, and occasionally
people review it or parts of it.  But no committer is accepting or rejecting
it, so the patch set remains in limbo.  This annoys you, and it is certainly
not a happy experience for the author.  Rejection is not nice, but at least
it would make it easier for Pavel to move on and spend his energy elsewhere.

Yours,
Laurenz Albe

  387. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-05-21T06:36:45Z

    st 21. 5. 2025 v 8:27 odesílatel Laurenz Albe <laurenz.albe@cybertec.at>
napsal:

> On Tue, 2025-05-20 at 17:06 -0400, Bruce Momjian wrote:
> > f no committer intends to pick it up and commit it, I think the proper
> > > action would be to step up and reject the patch set, not complain
> about the
> > > insistence of the author.
> >
> > Are you saying I should not complain until we have officially rejected
> > the patch set?  If we officially reject it, the patch author would no
> > longer post it?
>
> No, you are free to complain, just as Pavel is free to keep his patch
> set from rotting.  What I am trying to say is that rejecting the patch
> set is the most effective way to complain.
>
> The current state is that Pavel keeps the patch set alive, and occasionally
> people review it or parts of it.  But no committer is accepting or
> rejecting
> it, so the patch set remains in limbo.  This annoys you, and it is
> certainly
> not a happy experience for the author.  Rejection is not nice, but at least
> it would make it easier for Pavel to move on and spend his energy
> elsewhere.
>

Rejection is not nice, but it can be, and if it will be, I hope, there will
be more cleaner
status if we want this feature or not, and if possibly want this feature,
what we expect.

This feature can be designed differently in respect to some priorities. But
without known
priorities, any patch, proposal can end in the same state.

Regards

Pavel



>
> Yours,
> Laurenz Albe
>

  388. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-05-21T07:12:54Z

    Hi

út 19. 11. 2024 v 20:14 odesílatel Pavel Stehule <pavel.stehule@gmail.com>
napsal:

> Hi
>
> I wrote POC of VARIABLE(varname) syntax support
>
> here is a copy from regress test
>
> SET session_variables_ambiguity_warning TO on;
> SET session_variables_use_fence_warning_guard TO on;
> SET search_path TO 'public';
> CREATE VARIABLE a AS int;
> LET a = 10;
> CREATE TABLE test_table(a int, b int);
> INSERT INTO test_table VALUES(20, 20);
> -- no warning
> SELECT a;
>  a..
> ----
>  10
> (1 row)
>
> -- warning variable is shadowed
> SELECT a, b FROM test_table;
> WARNING:  session variable "a" is shadowed
> LINE 1: SELECT a, b FROM test_table;
>                ^
> DETAIL:  Session variables can be shadowed by columns, routine's variables
> and routine's arguments with the same name.
>  a  | b..
> ----+----
>  20 | 20
> (1 row)
>
> -- no warning
> SELECT variable(a) FROM test_table;
>  a..
> ----
>  10
> (1 row)
>
> ALTER TABLE test_table DROP COLUMN a;
> -- warning - variable fence is not used
> SELECT a, b FROM test_table;
> WARNING:  session variable "a" is not used inside variable fence
> LINE 1: SELECT a, b FROM test_table;
>                ^
> DETAIL:  The collision of session variable' names and column names is
> possible.
>  a  | b..
> ----+----
>  10 | 20
> (1 row)
>
> -- no warning
> SELECT variable(a), b FROM test_table;
>  a  | b..
> ----+----
>  10 | 20
> (1 row)
>
> DROP VARIABLE a;
> DROP TABLE test_table;
> SET session_variables_ambiguity_warning TO DEFAULT;
> SET session_variables_use_fence_warning_guard TO DEFAULT;
> SET search_path TO DEFAULT;
>
>
Last discussion is related to reducing the size of the session variable
patch set.

I have an idea to use variable's fencing more aggressively from the start,
and then we can reduce it in future. This should not break issues with
compatibility and doesn't need some like version flags.

The real problem of proposed session variables is possible collisions
between session variables identifiers and table or columns identifiers. I
designed some tools to minimize the risk of unwanted collisions, but these
tools increase the size of code and don't reduce the complexity of the
patch and tests. The proposed change probably doesn't reduce a lot of code,
but can reduce some tests, and mainly possible risk of some unwanted impact
- at the end it can be less work for reviewers and less stress for
committers - and the implementation can be divided to allone workable
following steps.

Step 1
=====

So the main change is the hard requirement for usage variable's fence
everywhere where collisions are possible - and then in the first step, the
collisions will not be possible, and then we don't need it to solve, and we
don't need to test it.

CREATE VARIABLE public.foo AS int;
LET foo = 10;
SELECT VARIABLE(foo);

DO $$
BEGIN
  RAISE NOTICE '% %', VARIABLE(foo), VARIABLE(public.foo);
END;
$$;

Step 2
=====
Necessity of usage variable fencing in PL/pgSQL can be a problem for
migration from PL/SQL. But this can be solved separately by using SPI
params hooks - similar to how PL/pgSQL works with PL/pgSQL variables. In
this step we can push optimization for fast execution of the LET statement
or optimization of usage variables in queries.

After this step will be possible:

DO $$
BEGIN
  RAISE NOTICE '% %', foo, VARIABLE(public.foo);
END;
$$;

SELECT VARIABLE(foo);

No other visible change in this step. WIth this step the people who do
migration form Oracle and PL/pgSQL developers will be very happy. They
don't need more. There can be collisions, but the collisions can be limited
just to PL/pgSQL scope, and we can use already implemented mechanisms.

Step 3
=====
We can talk in future about less requirement of usage variable fencing in
queries. This needs to introduce some form of detection collisions and how
they should be solved (outside PL/pgSQL).
We can talk about other features like temporal, default values,
transactional, etc ...

This proposal doesn't reduce lines of code, but significantly reduces
possible impacts of introducing session variables to other parts of SQL.
Moreover, it allows us to separate some
 work and related discussion into separate blocks - any block can be
implemented in different major pg releases.

I think a lot of users will be very happy just with step 1 and step 2, and
anything else can be discussed in future.

Is this plan acceptable?

Regards

Pavel





> Regards
>
> Pavel
>

  389. Re: proposal: schema variables

    Bruce Momjian <bruce@momjian.us> — 2025-05-21T20:41:38Z

    On Wed, May 21, 2025 at 07:15:27AM +0200, Pavel Stehule wrote:
> út 20. 5. 2025 v 23:07 odesílatel Bruce Momjian <bruce@momjian.us> napsal:
>     > If no committer intends to pick it up and commit it, I think the proper
>     > action would be to step up and reject the patch set, not complain about
>     the
>     > insistence of the author.
> 
>     Are you saying I should not complain until we have officially rejected
>     the patch set?  If we officially reject it, the patch author would no
>     longer post it?
> 
> I'll respect committers. I really don't want to worry people in the community.
> It is not my way, and I am sorry.

I realize I am being the bad guy by asking these questions, but I don't
think it is good for the project to get distracted with a feature that
isn't progressing, and it is unpleasant for an author to keep working on
something with no clear direction from the community.  I am happy to
learn that progress is being made.

I see this feature being first proposed in 2012:

	https://www.postgresql.org/message-id/flat/CAFj8pRDdk_4E8HiffbVOfk97iR%2BSLFoZpRT4D2nTE89YU-hQrg%40mail.gmail.com

and the first question was:

	I don't really see what we can do with this that we can't do
	without this.

Now, I think we have answered that question, and gotten closer to seeing
the complexities of adding this feature.

I am asking that, given its age, we more clearly direct this patch,
either toward completion or rejection.

> I think this is an important feature - for some group of developers, and then I
> push my energy and time for this.
> On the other hand, I accept that there is a lot of work that is important for a
> wider group of users. So it is. Unfortunately,
> without parser's hooks this feature cannot be implemented as an extension. But
> parser's hooks was proposed,
> and rejected, so there is no other possibility, how to do it. More -
> implementation of this feature as an extension is not
> best way.

I will reply to this in my next email.

-- 
  Bruce Momjian  <bruce@momjian.us>        https://momjian.us
  EDB                                      https://enterprisedb.com

  Do not let urgent matters crowd out time for investment in the future.

  390. Re: proposal: schema variables

    Bruce Momjian <bruce@momjian.us> — 2025-05-21T21:22:15Z

    On Wed, May 21, 2025 at 09:12:54AM +0200, Pavel Stehule wrote:
> Last discussion is related to reducing the size of the session variable patch
> set.
> 
> I have an idea to use variable's fencing more aggressively from the start, and
> then we can reduce it in future. This should not break issues with
> compatibility and doesn't need some like version flags.
> 
> The real problem of proposed session variables is possible collisions between
> session variables identifiers and table or columns identifiers. I designed some
> tools to minimize the risk of unwanted collisions, but these tools increase the
> size of code and don't reduce the complexity of the patch and tests. The
> proposed change probably doesn't reduce a lot of code, but can reduce some
> tests, and mainly possible risk of some unwanted impact - at the end it can be
> less work for reviewers and less stress for committers - and the implementation
> can be divided to allone workable following steps.

Yes, I remember the discussions about how the creation of server
variables could break existing queries.  Our scoping rules are already
complex, so adding another scope would add a lot of complexity.

> Step 1
> =====
> 
> So the main change is the hard requirement for usage variable's fence
> everywhere where collisions are possible - and then in the first step, the
> collisions will not be possible, and then we don't need it to solve, and we
> don't need to test it.
> 
> CREATE VARIABLE public.foo AS int;
> LET foo = 10;
> SELECT VARIABLE(foo);

Yes, I can see how adding fencing like VARIABLE() would simplify things.

> Step 2
> =====
> Necessity of usage variable fencing in PL/pgSQL can be a problem for migration
> from PL/SQL. But this can be solved separately by using SPI params hooks -
> similar to how PL/pgSQL works with PL/pgSQL variables. In this step we can push
> optimization for fast execution of the LET statement or optimization of usage
> variables in queries.

Yes, there is already going to be migration requirements in moving from
PL/SQL to PL/pgSQL, so the requirement to add VARIABLE() seems minimal.

> After this step will be possible:
> 
> DO $$
> BEGIN
>   RAISE NOTICE '% %', foo, VARIABLE(public.foo);
> END;
> $$;
> 
> SELECT VARIABLE(foo);
> 
> No other visible change in this step. WIth this step the people who do
> migration form Oracle and PL/pgSQL developers will be very happy. They don't
> need more. There can be collisions, but the collisions can be limited just to
> PL/pgSQL scope, and we can use already implemented mechanisms.
> 
> Step 3
> =====
> We can talk in future about less requirement of usage variable fencing in
> queries. This needs to introduce some form of detection collisions and how they
> should be solved (outside PL/pgSQL).
> We can talk about other features like temporal, default values, transactional,
> etc ...

I feel that if we haven't found a good solution to this in 13 years, we
should assume it is unsolvable and just accept an imperfect solution.

-- 
  Bruce Momjian  <bruce@momjian.us>        https://momjian.us
  EDB                                      https://enterprisedb.com

  Do not let urgent matters crowd out time for investment in the future.

  391. Re: Re: proposal: schema variables

    Bruce Momjian <bruce@momjian.us> — 2025-05-21T21:23:36Z

    On Wed, May 21, 2025 at 07:49:34AM +0200, Pavel Stehule wrote:
> st 21. 5. 2025 v 2:21 odesílatel Michael Paquier <michael@paquier.xyz> napsal:
>     One thing that I keep hearing about this feature is that this would be
>     really helpful for migration from Oracle to PostgreSQL, helping a lot
>     with rewrites of pl/pgsql functions.
> 
>     There is one page on the wiki about private variables, dating back to
>     2016:
>     https://wiki.postgresql.org/wiki/CREATE_PRIVATE_VARIABLE
> 
> I wrote mail 
> 
> https://www.postgresql.org/message-id/
> CAFj8pRB8kdWQCdN2X1_63c58+07Oy4Z+ruDK_xPTUP+Pe8R2Pw@mail.gmail.com
> 
> and there is another wiki page https://wiki.postgresql.org/wiki/Variable_Design

Yes, these URLs are very helpful, thanks.

-- 
  Bruce Momjian  <bruce@momjian.us>        https://momjian.us
  EDB                                      https://enterprisedb.com

  Do not let urgent matters crowd out time for investment in the future.

  392. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-06-03T07:26:54Z

    Hi

st 21. 5. 2025 v 23:22 odesílatel Bruce Momjian <bruce@momjian.us> napsal:

> On Wed, May 21, 2025 at 09:12:54AM +0200, Pavel Stehule wrote:
> > Last discussion is related to reducing the size of the session variable
> patch
> > set.
> >
> > I have an idea to use variable's fencing more aggressively from the
> start, and
> > then we can reduce it in future. This should not break issues with
> > compatibility and doesn't need some like version flags.
> >
> > The real problem of proposed session variables is possible collisions
> between
> > session variables identifiers and table or columns identifiers. I
> designed some
> > tools to minimize the risk of unwanted collisions, but these tools
> increase the
> > size of code and don't reduce the complexity of the patch and tests. The
> > proposed change probably doesn't reduce a lot of code, but can reduce
> some
> > tests, and mainly possible risk of some unwanted impact - at the end it
> can be
> > less work for reviewers and less stress for committers - and the
> implementation
> > can be divided to allone workable following steps.
>
> Yes, I remember the discussions about how the creation of server
> variables could break existing queries.  Our scoping rules are already
> complex, so adding another scope would add a lot of complexity.
>
> > Step 1
> > =====
> >
> > So the main change is the hard requirement for usage variable's fence
> > everywhere where collisions are possible - and then in the first step,
> the
> > collisions will not be possible, and then we don't need it to solve, and
> we
> > don't need to test it.
> >
> > CREATE VARIABLE public.foo AS int;
> > LET foo = 10;
> > SELECT VARIABLE(foo);
>
> Yes, I can see how adding fencing like VARIABLE() would simplify things.
>
> > Step 2
> > =====
> > Necessity of usage variable fencing in PL/pgSQL can be a problem for
> migration
> > from PL/SQL. But this can be solved separately by using SPI params hooks
> -
> > similar to how PL/pgSQL works with PL/pgSQL variables. In this step we
> can push
> > optimization for fast execution of the LET statement or optimization of
> usage
> > variables in queries.
>
> Yes, there is already going to be migration requirements in moving from
> PL/SQL to PL/pgSQL, so the requirement to add VARIABLE() seems minimal.
>
> > After this step will be possible:
> >
> > DO $$
> > BEGIN
> >   RAISE NOTICE '% %', foo, VARIABLE(public.foo);
> > END;
> > $$;
> >
> > SELECT VARIABLE(foo);
> >
> > No other visible change in this step. WIth this step the people who do
> > migration form Oracle and PL/pgSQL developers will be very happy. They
> don't
> > need more. There can be collisions, but the collisions can be limited
> just to
> > PL/pgSQL scope, and we can use already implemented mechanisms.
> >
> > Step 3
> > =====
> > We can talk in future about less requirement of usage variable fencing in
> > queries. This needs to introduce some form of detection collisions and
> how they
> > should be solved (outside PL/pgSQL).
> > We can talk about other features like temporal, default values,
> transactional,
> > etc ...
>
> I feel that if we haven't found a good solution to this in 13 years, we
> should assume it is unsolvable and just accept an imperfect solution.
>

I am sending an reorganized and modified patch set with implementation of
session variables

Important changes:

1. session variables fence is required everywhere except target of LET
command. Then there
   are no possible collisions of session variables with column or plpgsql
variables identifiers. The code
   related to collision detection or collision solving is removed.

2. I little bit reduced the size of patch because I didn't introduce
EXPR_KIND_LET_TARGET and
    EXPR_KIND_ASSING_TARGET contextes.

3. There are not other changes in the code

4. First two patches (originally 170KB and 160KB) are divided to some few
patches with max lengths (81KB and 53KB). The patches
   are incremental still (and possibly tested incrementally) with more
cleaner  coverage

0001-introduce-new-class-catalog-pg_variable.patch - new catalog
0002-CREATE-DROP-ALTER-VARIABLE.patch - DDL commands
0003-GRANT-REVOKE-variable.patch - ACL SELECT, UPDATE
0004-support-of-session-variables-for-psql.patch - psql tab complete and \dV
0005-support-of-session-variables-for-pg_dump.patch - pg_dump can backup
session variables
0006-session-variable-fences-parsing.patch - parser support for syntax
VARIABLE(varname)
0007-local-HASHTAB-for-currently-used-session-variables-a.patch - local
memory based storage for values of session variables
0008-collect-session-variables-used-in-plan-and-assign-pa.patch - planner
support for reading session variables from query
0009-fill-an-auxiliary-buffer-with-values-of-session-vari.patch - executor
support for reading session variables from query
0010-svariableReceiver.patch - DestReceiver used by LET command
0011-LET-command-assign-a-result-of-expression-to-the-ses.patch - LET
command implementation, parsing, execution
0012-function-pg_session_variables-for-cleaning-tests.patch - debug
function - returns a list of session variables from local memory storage
0013-DISCARD-VARIABLES.patch - throw memory used by session variables
0014-memory-cleaning-after-DROP-VARIABLE.patch - clean related memory after
DROP VARIABLE
0015-plpgsql-tests.patch - special tests for plpgsql, plan cache,
identifier collision, validity of memory used by session variables

The new total size of patches is 377KB, the size of previous patches was
400KB. The code is less than 1/3  of this (I think less than 2000 lines).
Almost all code is very simple without impact on other parts of Postgres.
Maybe an exception is memory-cleaning-after-DROP-VARIABLE
patch - but still there is no impact to other parts of Postgres.

Now there is a little bit overhead of git format-patch format. I
refactorized tests to separate files acl, ddl, dml. U use longer unique
identifiers,
so the execution of the tests should be more stable and less sensitive for
parallel tests execution.
I removed a few explicitly redundant tests.

This is the first step of implementing session variables in Postgres. It
contains only basic implementation of session variables. The lot of
proposed features (in this thread) can be implemented in the future without
compatibility issues. Their implementation is postponed now,
because the patch set is large enough now.

Although this is just a subset of proposed (discussed) functionality, I
think it can be valuable for people that need session variables.

This patchset doesn't contains:

1. Using session variables without variable fences inside plpgsql (then the
usage will be the same as PL/pgSQL or PL/SQL variables)
2. Performance optimization of usage of session variables (mostly for
plpgsql)
3. Implementation of EXPLAIN LET, PREPARE LET commands
4. Implementation of temporary session variables
5. Possibility to set default value for session variable (now it is NULL)
6. Immutable session variables support
7. Implementation of transactional behave of content of session variables
8. Special option --variable for pg_dump

Please, do a review if you can.

Comments, notes?

Regards

Pavel





>
> --
>   Bruce Momjian  <bruce@momjian.us>        https://momjian.us
>   EDB                                      https://enterprisedb.com
>
>   Do not let urgent matters crowd out time for investment in the future.
>

  393. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-06-03T11:43:21Z

    Hi

fix badly named role - needs prefix regress_

CREATE ROLE svartest_dml_read_role;
+WARNING:  roles created by regression test cases should have names
starting with "regress_"
 CREATE OR REPLACE FUNCTION svartest_dml.func_secdef()
 RETURNS int AS $$

  394. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-06-04T20:22:06Z

    Hi

I found a small problem in the pg_session_variables function. I expected
syscache to be flushed for recent catalog changes
before any usage of this function. But it is not always true. When I used
the DCATCACHE_FORCE_RELEASE build option, I got
different results from isolation tests. Fix is simple - just
call AcceptInvalidationMessages(); before touching syscache in this
function.

Note: pg_session_variables is a debug function - it shows entries of a hash
table with session variables used in the current session.
The result's attribute with information if the entry is related to a valid
or to invalid value (related tuple in pg_variable exists or not) was not
correct.

Regards

Pavel

  395. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-06-10T14:25:51Z

    Hi

I run pgindent - the changes are mostly only in comments

Regards

Pavel

  396. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-06-13T04:53:20Z

    Hi

rebase after 0f65f3eec478db8ac4f217a608b4478fed023bac

Reards

Pavel

  397. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-06-25T17:33:32Z

    Hi

* fresh rebase
+ new small test of usage LET from extended query protocol (test of usage
of parameters)
+ few notes in patch 11 commit message about reasons why LET keyword was
used instead SET for assign command

Regards

Pavel

  398. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-07-11T19:55:40Z

    Hi

just fresh rebase

Regards

Pavel

  399. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-07-22T05:53:26Z

    Hi

rebase after e2debb6

Regards

Pavel

  400. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-07-23T15:25:06Z

    Hi

fresh rebase - testing farms reports failure

Regards

Pavel

  401. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-08-05T05:30:18Z

    Hi

fresh rebase after 4e23c9ef65accde7eb3e56aa28d50ae5cf79b64b

Regards

Pavel

  402. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-08-13T19:24:09Z

    Hi

fix regress tests after 71ea0d6795438f95f4ee6e35867058c44b270d51

Regards

Pavel

  403. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-08-29T07:03:30Z

    Hi

rebase after 325fc0ab14d11fc87da594857ffbb6636affe7c0

Regards

Pavel

  404. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-09-13T09:28:56Z

    Hi

minor change (after private talk with Jim Jones)

renaming function `pg_session_variables` to
`pg_get_session_variables_memory`
renaming out parameter of this function `removed` to `dropped`
+ regress test plpgsql - plpgsql routines has not strong dependency to
session variables (same dependency like to other database objects)

Regards

Pavel

  405. Re: proposal: schema variables

    Jim Jones <jim.jones@uni-muenster.de> — 2025-09-15T08:21:33Z

    Hi

On 9/13/25 11:28, Pavel Stehule wrote:
> 
> minor change (after private talk with Jim Jones)
> 

After another pass, there are a few additional tests that should be
included in this patch:


== Additional tests for GRANT x ON VARIABLE ==
==============================================

We wanna make sure the proper error message is raised.

postgres=# GRANT INSERT ON VARIABLE var TO jim;
ERROR:  invalid privilege type INSERT for session variable

postgres=# GRANT DELETE ON VARIABLE var TO jim;
ERROR:  invalid privilege type DELETE for session variable


== Tests for ALTER DEFAULT PRIVILEGES ==
========================================

I couldn't find regression tests for ALTER DEFAULT PRIVILEGES ... if I
haven't just missed them, I think they'd be a nice addition.

postgres=# ALTER DEFAULT PRIVILEGES IN SCHEMA s GRANT UPDATE ON
VARIABLES TO jim;
ALTER DEFAULT PRIVILEGES

postgres=# ALTER DEFAULT PRIVILEGES IN SCHEMA s GRANT INSERT ON
VARIABLES TO jim;
ERROR:  invalid privilege type INSERT for session variable

postgres=# ALTER DEFAULT PRIVILEGES IN SCHEMA s GRANT DELETE ON
VARIABLES TO jim;
ERROR:  invalid privilege type DELETE for session variable

postgres=# ALTER DEFAULT PRIVILEGES IN SCHEMA s GRANT SELECT ON
VARIABLES TO jim;
ALTER DEFAULT PRIVILEGES


Best, Jim

  406. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-09-15T16:55:44Z

    Hi

po 15. 9. 2025 v 10:21 odesílatel Jim Jones <jim.jones@uni-muenster.de>
napsal:

> Hi
>
> On 9/13/25 11:28, Pavel Stehule wrote:
> >
> > minor change (after private talk with Jim Jones)
> >
>
> After another pass, there are a few additional tests that should be
> included in this patch:
>
>
> == Additional tests for GRANT x ON VARIABLE ==
> ==============================================
>
> We wanna make sure the proper error message is raised.
>
> postgres=# GRANT INSERT ON VARIABLE var TO jim;
> ERROR:  invalid privilege type INSERT for session variable
>
> postgres=# GRANT DELETE ON VARIABLE var TO jim;
> ERROR:  invalid privilege type DELETE for session variable
>
>
> == Tests for ALTER DEFAULT PRIVILEGES ==
> ========================================
>
> I couldn't find regression tests for ALTER DEFAULT PRIVILEGES ... if I
> haven't just missed them, I think they'd be a nice addition.
>
> postgres=# ALTER DEFAULT PRIVILEGES IN SCHEMA s GRANT UPDATE ON
> VARIABLES TO jim;
> ALTER DEFAULT PRIVILEGES
>
> postgres=# ALTER DEFAULT PRIVILEGES IN SCHEMA s GRANT INSERT ON
> VARIABLES TO jim;
> ERROR:  invalid privilege type INSERT for session variable
>
> postgres=# ALTER DEFAULT PRIVILEGES IN SCHEMA s GRANT DELETE ON
> VARIABLES TO jim;
> ERROR:  invalid privilege type DELETE for session variable
>
> postgres=# ALTER DEFAULT PRIVILEGES IN SCHEMA s GRANT SELECT ON
> VARIABLES TO jim;
> ALTER DEFAULT PRIVILEGES
>

done

Regards

Pavel


>
>
> Best, Jim
>

  407. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-09-29T10:13:25Z

    Hi

fresh rebase

Regards

Pavel

  408. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-10-06T05:54:09Z

    Hi

fresh rebase after 1a8b5b11e48a8fb086228542d1d4b379f23bdc1e

I wrote (with Jim Jones's big help) notes to proposed implementation of
session variables and to session variables topics generally.

https://wiki.postgresql.org/wiki/Implementation_of_declarative_catalog_session_variables

please, read it, and ask if there is something unclear or messy

Regards

Pavel

  409. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-10-12T04:57:56Z

    Hi

fresh rebase and fix after c83ac02ec7309edb7561eee93895c31a54b93d3d

Regards

Pavel

  410. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-10-30T16:27:12Z

    Hi

fresh rebase

Regards

Pavel

  411. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-11-02T20:40:46Z

    Hí

I rewrote the transformLetStmt routine. Now, some checks and
transformations are shared with VariableFence transformation and with
PLAssignStmt transformation (after
https://github.com/postgres/postgres/commit/b0fb2c6aa5a485e28210e13ae5536c1231b1261f
)

Although I removed redundant code, the patch 0011 is longer (4KB/58KB),
because I needed to modify another lines. But now, the code is much more
consistent (and more robust).

Regards

Pavel

  412. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-11-10T06:40:30Z

    Hi

fresh rebase

Regards

Pavel

  413. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-11-24T19:59:48Z

    Hi

I was asked to write the most reduced implementation of session variables.
I thought about possible reductions and there is one - catalog. We
can implement temporary session variables. In this case the catalog is not
necessary. After reduction the total size of the patchset is 134KB (doc +
code + tests). The size of the previous patchset was 390KB.

Unfortunately - without catalog some the dependency check is not available,
and then features that depends on dependencies are not available:

1. no catalog, no dependencies - only buildin types can be used (no custom
types, no domains),
2. no catalog, no dependencies - variables cannot be used in objects that
depend on some objects (views, sql functions),
3. simplified syntax - only scalar types can be used (no arrays, no
composites),
4. no catalog, no access rights, only owner check (variables can be used
just by their creator (owner)),
5. no catalog, no plan cache invalidation support (instead type check
before any usage),
6. no catalog, only temp variables are supported - schema cannot be
specified,
7. without direct access from the expression executor - variables cannot be
a parameter of CALL statement
8. session variables block parallel execution
9. no catalog, no object address - no event triggers for session variables
10. no catalog - DDL for session variables (CREATE TEMP VARIABLE, DROP
VARIABLE) are not transactional. Without catalog, implementation of
transactional DDL can be very difficult.

Although there are lot of strong limits, I think so implemented feature can
be still useful (parametrization of anonymous block and secure
session scope storage)

CREATE TEMP VARIABLE x AS int;
CREATE TEMP VARIABLE y AS int;
LET x = 100;
SELECT VARIABLE(x);
DO $$
BEGIN
  RAISE NOTICE '%', VARIABLE(x);
  LET y = VARIABLE(x) + 10;
 END $$;
SELECT VARIABLE(y);
DROP VARIABLE x;
DROP VARIABLE y;

This simple and reduced implementation doesn't block continuous development
in different directions (without compatibility breaks).

When I worked on this reduced version I found the importance of catalog
based implementation. It is almost impossible to implement variables with
life scope longer than transactions without a catalog. There is a strong
necessity of using dependency mechanisms. With catalog, the plan cache
invalidation can be used (instead typecheck). It is more robust.

Comments, notes?

Regards

Pavel

  414. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-11-24T21:19:10Z

    Hi

I am sorry. I forgot to add let.sgml file to patch

Regards

Pavel

  415. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-11-25T04:43:04Z

    Hi

I am sending a link to our wiki with general description of implementations
of session variables in mayor RDBMS

https://wiki.postgresql.org/wiki/Implementation_of_declarative_catalog_session_variables

Regards

Pavel

  416. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-11-30T06:01:42Z

    Hi

only rebase

Regards

Pavel

  417. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-12-03T04:27:04Z

    Hi

fresh rebase after a87987cafca683e9076c424f99bae117211a83a4

regards

Pavel

  418. Re: proposal: schema variables

    Jim Jones <jim.jones@uni-muenster.de> — 2025-12-03T13:44:16Z

    Hi Pavel

On 03/12/2025 05:27, Pavel Stehule wrote:
> Hi
> 
> fresh rebase after a87987cafca683e9076c424f99bae117211a83a4


I'm going through the patch again and have a few initial comments.

== Memory Management ==

DROP VARIABLE seems to be leaking memory:

postgres=# CREATE TEMPORARY VARIABLE var AS text;
CREATE VARIABLE
postgres=# LET var = repeat('🐘', 100000000);
LET
postgres=# SELECT pg_size_pretty(used_bytes)
FROM pg_backend_memory_contexts
WHERE name = 'session variables';
 pg_size_pretty
----------------
 381 MB
(1 row)

postgres=# DROP VARIABLE var;
DROP VARIABLE
postgres=# SELECT pg_size_pretty(used_bytes)
FROM pg_backend_memory_contexts
WHERE name = 'session variables';
 pg_size_pretty
----------------
 381 MB
(1 row)


If we simply set the variable to NULL it works as expected:


postgres=# LET var = repeat('🐘', 100000000);
LET
postgres=# SELECT pg_size_pretty(used_bytes)
FROM pg_backend_memory_contexts
WHERE name = 'session variables';
 pg_size_pretty
----------------
 381 MB
(1 row)

postgres=# LET var = NULL;
LET
postgres=# SELECT pg_size_pretty(used_bytes)
FROM pg_backend_memory_contexts
WHERE name = 'session variables';
 pg_size_pretty
----------------
 240 bytes
(1 row)

Most likely you forgot to pfree "svar->value" at DropVariableByName(), e.g.

void
DropVariableByName(char *varname)
{

...

if (!svar->typbyval && !svar->isnull)
	pfree(DatumGetPointer(svar->value));

(void) hash_search(sessionvars,
		   varname,
		   HASH_REMOVE,
		   NULL);

}


== TAB completion ==

Why suggest CREATE VARIABLE (non-temporary) if it is not supported?

postgres=# CREATE V<TAB>
VARIABLE  VIEW
postgres=# CREATE VARIABLE var AS int;
ERROR:  only temporal session variables are supported

It would be nice to have tab completion for DROP VARIABLE and LET as well:

postgres=# DROP VARIABLE <TAB>

postgres=# LET <TAB>



== Missing IF EXISTS in DROP VARIABLE ==

DROP VARIABLE IF EXISTS var;
ERROR:  syntax error at or near "EXISTS"
LINE 1: DROP VARIABLE IF EXISTS var;

               ^


Best, Jim

  419. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-12-05T06:50:25Z

    Hi

st 3. 12. 2025 v 14:44 odesílatel Jim Jones <jim.jones@uni-muenster.de>
napsal:

> Hi Pavel
>
> On 03/12/2025 05:27, Pavel Stehule wrote:
> > Hi
> >
> > fresh rebase after a87987cafca683e9076c424f99bae117211a83a4
>
>
> I'm going through the patch again and have a few initial comments.
>


>
> == Memory Management ==
>
> DROP VARIABLE seems to be leaking memory:
>
> postgres=# CREATE TEMPORARY VARIABLE var AS text;
> CREATE VARIABLE
> postgres=# LET var = repeat('🐘', 100000000);
> LET
> postgres=# SELECT pg_size_pretty(used_bytes)
> FROM pg_backend_memory_contexts
> WHERE name = 'session variables';
>  pg_size_pretty
> ----------------
>  381 MB
> (1 row)
>
> postgres=# DROP VARIABLE var;
> DROP VARIABLE
> postgres=# SELECT pg_size_pretty(used_bytes)
> FROM pg_backend_memory_contexts
> WHERE name = 'session variables';
>  pg_size_pretty
> ----------------
>  381 MB
> (1 row)
>
>
> If we simply set the variable to NULL it works as expected:
>
>
> postgres=# LET var = repeat('🐘', 100000000);
> LET
> postgres=# SELECT pg_size_pretty(used_bytes)
> FROM pg_backend_memory_contexts
> WHERE name = 'session variables';
>  pg_size_pretty
> ----------------
>  381 MB
> (1 row)
>
> postgres=# LET var = NULL;
> LET
> postgres=# SELECT pg_size_pretty(used_bytes)
> FROM pg_backend_memory_contexts
> WHERE name = 'session variables';
>  pg_size_pretty
> ----------------
>  240 bytes
> (1 row)
>
> Most likely you forgot to pfree "svar->value" at DropVariableByName(), e.g.
>
> void
> DropVariableByName(char *varname)
> {
>
> ...
>
> if (!svar->typbyval && !svar->isnull)
>         pfree(DatumGetPointer(svar->value));
>
> (void) hash_search(sessionvars,
>                    varname,
>                    HASH_REMOVE,
>                    NULL);
>
> }
>
>
yes, there was a bug, fixed


>
> == TAB completion ==
>
> Why suggest CREATE VARIABLE (non-temporary) if it is not supported?
>
> postgres=# CREATE V<TAB>
> VARIABLE  VIEW
> postgres=# CREATE VARIABLE var AS int;
> ERROR:  only temporal session variables are supported
>
> It would be nice to have tab completion for DROP VARIABLE and LET as well:
>
> postgres=# DROP VARIABLE <TAB>
>
> postgres=# LET <TAB>
>
>
>
> == Missing IF EXISTS in DROP VARIABLE ==
>
> DROP VARIABLE IF EXISTS var;
> ERROR:  syntax error at or near "EXISTS"
> LINE 1: DROP VARIABLE IF EXISTS var;
>
>                ^
>

Both mentioned issues are related to the declared target of this patchset -
maximal reduction of the size.

But It not difficult to support these requested features - see the patches
0008 and 0009

Best regards

Pavel



>
>
> Best, Jim
>
>
>
>

  420. Re: proposal: schema variables

    Jim Jones <jim.jones@uni-muenster.de> — 2025-12-06T11:29:48Z

    
On 05/12/2025 07:50, Pavel Stehule wrote:
> yes, there was a bug, fixed

> Both mentioned issues are related to the declared target of this
> patchset - maximal reduction of the size.

Nice, the memory is now being freed after a DROP VARIABLE and the tab
completion for LET and DROP VARIABLE works:

postgres=# CREATE TEMPORARY VARIABLE var AS text;
CREATE VARIABLE
postgres=# LET <TAB>
var  x
postgres=# LET var = repeat('🐘', 200000000);
LET
postgres=# SELECT pg_size_pretty(used_bytes)
FROM pg_backend_memory_contexts
WHERE name = 'session variables';
 pg_size_pretty
----------------
 763 MB
(1 row)

postgres=# DROP VARIABLE <TAB>
var  x
postgres=# DROP VARIABLE var;
DROP VARIABLE
postgres=# SELECT pg_size_pretty(used_bytes)
FROM pg_backend_memory_contexts
WHERE name = 'session variables';
 pg_size_pretty
----------------
 240 bytes
(1 row)


-- DROP VARIABLE IF EXISTS also works:

postgres=# DROP VARIABLE IF EXISTS x;
DROP VARIABLE


Some comments and a few minor issues:

== session_variables_ddl.sql ==

1) duplicate tests

...
DROP VARIABLE IF EXISTS x;
DROP VARIABLE IF EXISTS x;
...

2) Typos in some comments "should to fail" > "should fail"

== Error messages ==

3) It is not possible to create a VIEW that depends on a session
variable, which makes perfect sense.

postgres=# CREATE VIEW v AS SELECT variable(var);
ERROR:  session variable "var" cannot be referenced in a persistent object

The error message is clear, but in case of TEMPORARY VIEWS it gets a bit
misleading, since a TEMPORARY VIEW is not a persistent object:

postgres=# CREATE TEMPORARY VIEW tv AS SELECT variable(var);
ERROR:  session variable "var" cannot be referenced in a persistent object

Perhaps something more generic? For instance:

errmsg("session variable \"%s\" cannot be referenced in catalog
objects", param->paramvarname)

== ddl.sgml ==

4) There are invalid examples

-- No schema qualified VARIABLE is supported:

CREATE VARIABLE public.current_user_id AS integer;

-- Only TEMPORARY VARIABLES are supported:
CREATE VARIABLE var1 AS date;

5) The term "variable fence" is introduced and emphasised, but not
described.

6) There is a slight repetition regarding the variable's isolation

"This value is private to each session .."
"The value of a session variable is local to the current session"

I would write something along these lines:

"Session variables are temporary database objects that can hold a value.
A session variable can be created using the CREATE VARIABLE command and
can only be accessed by its owner. The value of a session variable is
stored in session memory and is private to each session. It is
automatically released when the session ends.

In a query, a session variable can only be referenced using the special
<literal>VARIABLE(varname)</literal> syntax. This avoids any risk of
collision between variable names and column names.

You set the value of a session variable with the <command>LET</command>
statement and retrieve it with <command>SELECT</command>:

<programlisting>
CREATE TEMPORARY VARIABLE var1 AS date;
LET var1 = current_date;
SELECT VARIABLE(var1);
    var1
------------
 2025-12-06
(1 row)
</programlisting>

By default, retrieving a session variable returns
<literal>NULL</literal> unless it has been set in the current session
using the <command>LET</command> command. Session variables are not
transactional: changes to their values persist even if the transaction
is rolled back, similar to variables in procedural languages."

== let.sgml ==

7) Invalid example (missing TEMP/TEMPORARY)

CREATE VARIABLE myvar AS integer;

8) Typo in the Synopsis (TEMPORAL should be TEMPORARY):

CREATE { TEMP | TEMPORAL } VARIABLE  [ IF NOT EXISTS ] name [ AS ] data_type

9) In the description it says "The CREATE VARIABLE command creates a
temporal session variable.", but isn't the command now CREATE
TEMP/TEMPORARY VARIABLE? Is it ok to remove the TEMPORARY in the
description?

10) The description includes also info regarding SELECT and LET. Since
this page is about CREATE TEMPORARY VARIABLE, I guess it is out of place?


Thanks!

Best, Jim

  421. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-12-08T08:15:50Z

    Hi

so 6. 12. 2025 v 12:29 odesílatel Jim Jones <jim.jones@uni-muenster.de>
napsal:

>
>
> On 05/12/2025 07:50, Pavel Stehule wrote:
> > yes, there was a bug, fixed
>
> > Both mentioned issues are related to the declared target of this
> > patchset - maximal reduction of the size.
>
> Nice, the memory is now being freed after a DROP VARIABLE and the tab
> completion for LET and DROP VARIABLE works:
>
> postgres=# CREATE TEMPORARY VARIABLE var AS text;
> CREATE VARIABLE
> postgres=# LET <TAB>
> var  x
> postgres=# LET var = repeat('🐘', 200000000);
> LET
> postgres=# SELECT pg_size_pretty(used_bytes)
> FROM pg_backend_memory_contexts
> WHERE name = 'session variables';
>  pg_size_pretty
> ----------------
>  763 MB
> (1 row)
>
> postgres=# DROP VARIABLE <TAB>
> var  x
> postgres=# DROP VARIABLE var;
> DROP VARIABLE
> postgres=# SELECT pg_size_pretty(used_bytes)
> FROM pg_backend_memory_contexts
> WHERE name = 'session variables';
>  pg_size_pretty
> ----------------
>  240 bytes
> (1 row)
>
>
> -- DROP VARIABLE IF EXISTS also works:
>
> postgres=# DROP VARIABLE IF EXISTS x;
> DROP VARIABLE
>
>
> Some comments and a few minor issues:
>
> == session_variables_ddl.sql ==
>
> 1) duplicate tests
>
> ...
> DROP VARIABLE IF EXISTS x;
> DROP VARIABLE IF EXISTS x;
> ...
>

fixed


>
> 2) Typos in some comments "should to fail" > "should fail"
>

fixed


>
> == Error messages ==
>
> 3) It is not possible to create a VIEW that depends on a session
> variable, which makes perfect sense.
>
> postgres=# CREATE VIEW v AS SELECT variable(var);
> ERROR:  session variable "var" cannot be referenced in a persistent object
>
> The error message is clear, but in case of TEMPORARY VIEWS it gets a bit
> misleading, since a TEMPORARY VIEW is not a persistent object:
>
> postgres=# CREATE TEMPORARY VIEW tv AS SELECT variable(var);
> ERROR:  session variable "var" cannot be referenced in a persistent object
>
> Perhaps something more generic? For instance:
>
> errmsg("session variable \"%s\" cannot be referenced in catalog
> objects", param->paramvarname)
>

changed like you proposed


>
> == ddl.sgml ==
>
> 4) There are invalid examples
>
> -- No schema qualified VARIABLE is supported:
>
> CREATE VARIABLE public.current_user_id AS integer;
>
> -- Only TEMPORARY VARIABLES are supported:
> CREATE VARIABLE var1 AS date;
>

fixed


>
> 5) The term "variable fence" is introduced and emphasised, but not
> described.
>

?? There is already

+++ b/doc/src/sgml/ddl.sgml
@@ -5676,6 +5676,17 @@ EXPLAIN SELECT count(*) FROM measurement WHERE
logdate &gt;= DATE '2008-01-01';
     The session variable holds value in session memory.  This value is
private
     to each session and is released when the session ends.
    </para>
+
+   <para>
+    In an query the session variable can be used only inside
+    <firstterm>variable fence</firstterm>. This is special syntax for
+    session variable identifier, and can be used only for session variable
+    identifier. The special syntax for accessing session variables removes
+    risk of collisions between variable identifiers and column names.
+<programlisting>
+SELECT VARIABLE(current_user_id);
+</programlisting>
+   </para>
   </sect1>



>
> 6) There is a slight repetition regarding the variable's isolation
>
> "This value is private to each session .."
> "The value of a session variable is local to the current session"
>
> I would write something along these lines:
>
> "Session variables are temporary database objects that can hold a value.
> A session variable can be created using the CREATE VARIABLE command and
> can only be accessed by its owner. The value of a session variable is
> stored in session memory and is private to each session. It is
> automatically released when the session ends.
>

done

>
> In a query, a session variable can only be referenced using the special
> <literal>VARIABLE(varname)</literal> syntax. This avoids any risk of
> collision between variable names and column names.
>

done


>
> You set the value of a session variable with the <command>LET</command>
> statement and retrieve it with <command>SELECT</command>:
>
> <programlisting>
> CREATE TEMPORARY VARIABLE var1 AS date;
> LET var1 = current_date;
> SELECT VARIABLE(var1);
>     var1
> ------------
>  2025-12-06
> (1 row)
> </programlisting>
>
> By default, retrieving a session variable returns
> <literal>NULL</literal> unless it has been set in the current session
> using the <command>LET</command> command. Session variables are not
> transactional: changes to their values persist even if the transaction
> is rolled back, similar to variables in procedural languages."
>

done


>
> == let.sgml ==
>
> 7) Invalid example (missing TEMP/TEMPORARY)
>
> CREATE VARIABLE myvar AS integer;
>

fixed


>
> 8) Typo in the Synopsis (TEMPORAL should be TEMPORARY):
>
> CREATE { TEMP | TEMPORAL } VARIABLE  [ IF NOT EXISTS ] name [ AS ]
> data_type
>

fixed


>
> 9) In the description it says "The CREATE VARIABLE command creates a
> temporal session variable.", but isn't the command now CREATE
> TEMP/TEMPORARY VARIABLE? Is it ok to remove the TEMPORARY in the
> description?
>

Although the TEMP clause in CREATE VARIABLE is mandatory now, I prefer to
look on this
like some temporary limit - so I don't libe to rename CREATE VARIABLE to
CREATE TEMP VARIABLE.

I changed this part to

  <para>
   The <command>CREATE VARIABLE</command> command creates a session
   variable. Currently only temporary session variables are supported,
   and then the keyword <literal>TEMPORARY</literal> is required.
  </para>


>
> 10) The description includes also info regarding SELECT and LET. Since
> this page is about CREATE TEMPORARY VARIABLE, I guess it is out of place?
>
>
I am sorry, I don't understand this point. Can you describe it?


For the current patchset I wrote initial support for transactional DDL for
session variables - patch 0010 and 0011.
Now, the DDL is blocked in read only transactions, parallel worker and
inside recovery.

Regards

Pavel




>
> Thanks!
>
> Best, Jim
>

  422. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-12-08T13:57:51Z

    Hi

I found a "native" way to disallow a usage of custom types - it is the same
as checking used by virtual columns - and virtual columns is another
feature where custom types are not allowed.

Regards

Pavel

  423. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-12-09T05:51:43Z

    Hi

I fixed two issues

- broken regress test - test of blocking usage of domains
- typo in ruleutils.c s/VARIABLES/VARIABLE/

Best regards

Pavel

  424. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-12-12T14:46:33Z

    Hi

- rebase
- palloc replaced by palloc_object, and palloc_array

Regards

Pavel

  425. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-12-15T04:42:34Z

    Hi

only rebase

Regards

Pavel

  426. Re: proposal: schema variables

    Pavel Stehule <pavel.stehule@gmail.com> — 2025-12-27T07:52:28Z

    Hi

only rebase

Regards

Pavel