Horton

Thread

Commits

  1. Fix crash when using partition bound expressions

  2. Allow generalized expression syntax for partition bounds

  1. BUG #15668: Server crash in transformPartitionRangeBounds

    The Post Office <noreply@postgresql.org> — 2019-03-05T09:05:55Z

    The following bug has been logged on the website:

Bug reference:      15668
Logged by:          Alexander Lakhin
Email address:      exclusion@gmail.com
PostgreSQL version: Unsupported/Unknown
Operating system:   Ubuntu 18.04
Description:        

The following query:
CREATE TABLE range_parted (a int) PARTITION BY RANGE (a);
CREATE TABLE rp_part PARTITION OF range_parted FOR VALUES FROM
(unknown.unknown) TO (1);

crashes server (on the master branch) with the stack trace:
Core was generated by `postgres: law regression [local] CREATE TABLE        
                        '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000560ab19ea0bc in transformPartitionRangeBounds
(pstate=pstate@entry=0x560ab3290da8, blist=<optimized out>, 
    parent=parent@entry=0x7f7846a6bea8) at parse_utilcmd.c:3754
3754                            if (strcmp("minvalue", cname) == 0)
(gdb) bt
#0  0x0000560ab19ea0bc in transformPartitionRangeBounds
(pstate=pstate@entry=0x560ab3290da8, blist=<optimized out>, 
    parent=parent@entry=0x7f7846a6bea8) at parse_utilcmd.c:3754
#1  0x0000560ab19eea30 in transformPartitionBound
(pstate=pstate@entry=0x560ab3290da8, 
    parent=parent@entry=0x7f7846a6bea8, spec=0x560ab31fd448) at
parse_utilcmd.c:3706
#2  0x0000560ab1a4436f in DefineRelation (stmt=stmt@entry=0x560ab3295b30,
relkind=relkind@entry=114 'r', ownerId=10, 
    ownerId@entry=0, typaddress=typaddress@entry=0x0, 
    queryString=queryString@entry=0x560ab31da3b0 "CREATE TABLE rp_part
PARTITION OF range_parted FOR VALUES FROM (unknown.unknown) TO (1);") at
tablecmds.c:881
#3  0x0000560ab1bba086 in ProcessUtilitySlow
(pstate=pstate@entry=0x560ab3295a20, pstmt=pstmt@entry=0x560ab31db4b0, 
    queryString=queryString@entry=0x560ab31da3b0 "CREATE TABLE rp_part
PARTITION OF range_parted FOR VALUES FROM (unknown.unknown) TO (1);",
context=context@entry=PROCESS_UTILITY_TOPLEVEL, params=params@entry=0x0, 
    queryEnv=queryEnv@entry=0x0, completionTag=0x7ffcc3c17120 "",
dest=0x560ab31db590) at utility.c:1003
#4  0x0000560ab1bb8b60 in standard_ProcessUtility (pstmt=0x560ab31db4b0, 
    queryString=0x560ab31da3b0 "CREATE TABLE rp_part PARTITION OF
range_parted FOR VALUES FROM (unknown.unknown) TO (1);",
context=PROCESS_UTILITY_TOPLEVEL, params=0x0, queryEnv=0x0,
dest=0x560ab31db590, completionTag=0x7ffcc3c17120 "")
    at utility.c:923
#5  0x0000560ab1bb6022 in PortalRunUtility (portal=0x560ab32415e0,
pstmt=0x560ab31db4b0, isTopLevel=<optimized out>, 
    setHoldSnapshot=<optimized out>, dest=<optimized out>,
completionTag=0x7ffcc3c17120 "") at pquery.c:1175
#6  0x0000560ab1bb6ae0 in PortalRunMulti
(portal=portal@entry=0x560ab32415e0, isTopLevel=isTopLevel@entry=true, 
    setHoldSnapshot=setHoldSnapshot@entry=false,
dest=dest@entry=0x560ab31db590, altdest=altdest@entry=0x560ab31db590, 
    completionTag=completionTag@entry=0x7ffcc3c17120 "") at pquery.c:1328
#7  0x0000560ab1bb76da in PortalRun (portal=portal@entry=0x560ab32415e0,
count=count@entry=9223372036854775807, 
    isTopLevel=isTopLevel@entry=true, run_once=run_once@entry=true,
dest=dest@entry=0x560ab31db590, 
    altdest=altdest@entry=0x560ab31db590, completionTag=0x7ffcc3c17120 "")
at pquery.c:796
#8  0x0000560ab1bb3582 in exec_simple_query (
    query_string=0x560ab31da3b0 "CREATE TABLE rp_part PARTITION OF
range_parted FOR VALUES FROM (unknown.unknown) TO (1);") at
postgres.c:1215
#9  0x0000560ab1bb54ee in PostgresMain (argc=<optimized out>,
argv=argv@entry=0x560ab32056a0, dbname=<optimized out>, 
    username=<optimized out>) at postgres.c:4256
#10 0x0000560ab1b42e00 in BackendRun (port=0x560ab31fdde0,
port=0x560ab31fdde0) at postmaster.c:4382
#11 BackendStartup (port=0x560ab31fdde0) at postmaster.c:4073
#12 ServerLoop () at postmaster.c:1703
#13 0x0000560ab1b43eb9 in PostmasterMain (argc=3, argv=0x560ab31d4a10) at
postmaster.c:1376
#14 0x0000560ab18e33a1 in main (argc=3, argv=0x560ab31d4a10) at main.c:228

  2. Re: BUG #15668: Server crash in transformPartitionRangeBounds

    Amit Langote <amitlangote09@gmail.com> — 2019-03-05T14:04:17Z

    Hi,

(cc'ing -hackers and Peter E)

On Tue, Mar 5, 2019 at 8:02 PM PG Bug reporting form
<noreply@postgresql.org> wrote:
>
> The following bug has been logged on the website:
>
> Bug reference:      15668
> Logged by:          Alexander Lakhin
> Email address:      exclusion@gmail.com
> PostgreSQL version: Unsupported/Unknown
> Operating system:   Ubuntu 18.04
> Description:
>
> The following query:
> CREATE TABLE range_parted (a int) PARTITION BY RANGE (a);
> CREATE TABLE rp_part PARTITION OF range_parted FOR VALUES FROM
> (unknown.unknown) TO (1);
>
> crashes server (on the master branch) with the stack trace:
> Core was generated by `postgres: law regression [local] CREATE TABLE
>                         '.
> Program terminated with signal SIGSEGV, Segmentation fault.
> #0  0x0000560ab19ea0bc in transformPartitionRangeBounds
> (pstate=pstate@entry=0x560ab3290da8, blist=<optimized out>,
>     parent=parent@entry=0x7f7846a6bea8) at parse_utilcmd.c:3754
> 3754                            if (strcmp("minvalue", cname) == 0)

Thanks for the report.  Seems to be a bug of the following commit in
HEAD, of which I was one of the authors:

commit 7c079d7417a8f2d4bf5144732e2f85117db9214f
Author: Peter Eisentraut <peter@eisentraut.org>
Date:   Fri Jan 25 11:27:59 2019 +0100

    Allow generalized expression syntax for partition bounds

That seems to be caused by some over-optimistic coding in
transformPartitionRangeBounds.  Following will crash too.

CREATE TABLE rp_part PARTITION OF range_parted FOR VALUES FROM
(a.a.a.a.a.a.a.a.a.a.a.a) TO (1);

If I try the list partitioning syntax, it doesn't crash but gives the
following error:

create table lparted1 partition of lparted for values in (a.a.a.a.a.a);
ERROR:  improper qualified name (too many dotted names): a.a.a.a.a.a
LINE 1: ...able lparted1 partition of lparted for values in (a.a.a.a.a....
                                                             ^
Maybe we should error out as follows in
transformPartitionRangeBounds(), although that means we'll get
different error message than when using list partitioning syntax:

@@ -3749,6 +3749,12 @@ transformPartitionRangeBounds(ParseState
*pstate, List *blist,
            if (list_length(cref->fields) == 1 &&
                IsA(linitial(cref->fields), String))
                cname = strVal(linitial(cref->fields));
+           else
+               ereport(ERROR,
+                       (errcode(ERRCODE_INVALID_OBJECT_DEFINITION),
+                        errmsg("invalid expression for range bound"),
+                        parser_errposition(pstate,
+                                           exprLocation((Node *) expr))));

            Assert(cname != NULL);
            if (strcmp("minvalue", cname) == 0)

Thanks,
Amit

  3. Re: BUG #15668: Server crash in transformPartitionRangeBounds

    Michael Paquier <michael@paquier.xyz> — 2019-03-06T06:48:25Z

    On Tue, Mar 05, 2019 at 11:04:17PM +0900, Amit Langote wrote:
> Maybe we should error out as follows in
> transformPartitionRangeBounds(), although that means we'll get
> different error message than when using list partitioning syntax:

Hm.  I don't think that this is a good idea as you could lose some
information for the expression transformation handling, and the error
handling becomes inconsistent depending on the partition bound
strategy.  It seems to me that if we cannot extract any special value
from the ColumnRef expression generated, then we ought to let
transformPartitionBoundValue() and particularly transformExprRecurse()
do the analysis work and complain if needed:
=# CREATE TABLE rp_part PARTITION OF range_parted FOR VALUES FROM
(unknown.unknown) TO (1);
ERROR:  42P01: missing FROM-clause entry for table "unknown"
LINE 1: ...p_part PARTITION OF range_parted FOR VALUES FROM
(unknown.un...
=# CREATE TABLE rp_part PARTITION OF range_parted FOR VALUES FROM
(a.a.a.a.a.a.a.a.a.a.a.a) TO (1);
ERROR:  42601: improper qualified name (too many dotted names):
a.a.a.a.a.a.a.a.a.a.a.a
LINE 1: ...p_part PARTITION OF range_parted FOR VALUES FROM
(a.a.a.a.a....

What about something like the attached instead?  Minus the test
cases which should go to create_table.sql of course.
--
Michael

  4. Re: BUG #15668: Server crash in transformPartitionRangeBounds

    Amit Langote <langote_amit_f8@lab.ntt.co.jp> — 2019-03-06T07:00:42Z

    Hi,

On 2019/03/06 15:48, Michael Paquier wrote:
> On Tue, Mar 05, 2019 at 11:04:17PM +0900, Amit Langote wrote:
>> Maybe we should error out as follows in
>> transformPartitionRangeBounds(), although that means we'll get
>> different error message than when using list partitioning syntax:
> 
> Hm.  I don't think that this is a good idea as you could lose some
> information for the expression transformation handling, and the error
> handling becomes inconsistent depending on the partition bound
> strategy.  It seems to me that if we cannot extract any special value
> from the ColumnRef expression generated, then we ought to let
> transformPartitionBoundValue() and particularly transformExprRecurse()
> do the analysis work and complain if needed:
> =# CREATE TABLE rp_part PARTITION OF range_parted FOR VALUES FROM
> (unknown.unknown) TO (1);
> ERROR:  42P01: missing FROM-clause entry for table "unknown"
> LINE 1: ...p_part PARTITION OF range_parted FOR VALUES FROM
> (unknown.un...
> =# CREATE TABLE rp_part PARTITION OF range_parted FOR VALUES FROM
> (a.a.a.a.a.a.a.a.a.a.a.a) TO (1);
> ERROR:  42601: improper qualified name (too many dotted names):
> a.a.a.a.a.a.a.a.a.a.a.a
> LINE 1: ...p_part PARTITION OF range_parted FOR VALUES FROM
> (a.a.a.a.a....
> 
> What about something like the attached instead?  Minus the test
> cases which should go to create_table.sql of course.

Thanks for looking at this.  Your patch seems better, because it allows us
to keep the error message consistent with the message one would get with
list-partitioned syntax.

Thanks,
Amit

  5. Re: BUG #15668: Server crash in transformPartitionRangeBounds

    Michael Paquier <michael@paquier.xyz> — 2019-03-06T08:27:30Z

    On Wed, Mar 06, 2019 at 04:00:42PM +0900, Amit Langote wrote:
> Thanks for looking at this.  Your patch seems better, because it allows us
> to keep the error message consistent with the message one would get with
> list-partitioned syntax.

Thanks for confirming.  I think that it would be nice as well to add
more test coverage for such error patterns with all the strategies.
It would be good to fix that first, so I can take care of that.

Now I don't really find the error "missing FROM-clause entry for
table" quite convincing when this is applied to a partition bound when
using a column defined in the relation.  Adding more error classes in
the set of CRERR_NO_RTE would be perhaps nice, still I am not sure how
elegant it could be made when looking at expressions for partition
bounds.
--
Michael

  6. Re: BUG #15668: Server crash in transformPartitionRangeBounds

    Amit Langote <langote_amit_f8@lab.ntt.co.jp> — 2019-03-11T06:44:39Z

    Hi,

On 2019/03/06 17:27, Michael Paquier wrote:
> On Wed, Mar 06, 2019 at 04:00:42PM +0900, Amit Langote wrote:
>> Thanks for looking at this.  Your patch seems better, because it allows us
>> to keep the error message consistent with the message one would get with
>> list-partitioned syntax.
> 
> Thanks for confirming.  I think that it would be nice as well to add
> more test coverage for such error patterns with all the strategies.
> It would be good to fix that first, so I can take care of that.

I've added some tests to your patch.  Also improved the comments a bit.

I noticed another issue with the code -- it's using strcmp() to compare
specified string against "minvalue" and "maxvalue", which causes the
following silly error:

create table q2 partition of q for values from ("MINVALUE") to (maxvalue);
ERROR:  column "MINVALUE" does not exist
LINE 1: create table q2 partition of q for values from ("MINVALUE") ...

It should be using pg_strncasecmp().

> Now I don't really find the error "missing FROM-clause entry for
> table" quite convincing when this is applied to a partition bound when
> using a column defined in the relation.  Adding more error classes in
> the set of CRERR_NO_RTE would be perhaps nice, still I am not sure how
> elegant it could be made when looking at expressions for partition
> bounds.

Note that this is not just a problem for partition bounds.  You can see it
with default expressions too.

create table foo (a int default (aa.a));
ERROR:  missing FROM-clause entry for table "aa"
LINE 1: create table foo (a int default (aa.a));

create table foo (a int default (a.a.aa.a.a.a.a.aa));
ERROR:  improper qualified name (too many dotted names): a.a.aa.a.a.a.a.aa
LINE 1: create table foo (a int default (a.a.aa.a.a.a.a.aa));

We could make the error message more meaningful depending on the context,
but maybe it'd better be pursue it as a separate project.

Thanks,
Amit

  7. Re: BUG #15668: Server crash in transformPartitionRangeBounds

    Michael Paquier <michael@paquier.xyz> — 2019-03-11T07:21:14Z

    On Mon, Mar 11, 2019 at 03:44:39PM +0900, Amit Langote wrote:
> We could make the error message more meaningful depending on the context,
> but maybe it'd better be pursue it as a separate project.

Yeah, I noticed that stuff when working on it this afternoon.  The
error message does not completely feel right even in your produced
tests.  Out of curiosity I have been working on this thing myself,
and it is possible to have a context-related message.  Please see
attached, that's in my opinion less confusing, and of course
debatable.  Still this approach does not feel completely right either
as that means hijacking the code path which generates a generic
message for missing RTEs. :(
--
Michael

  8. Re: BUG #15668: Server crash in transformPartitionRangeBounds

    Robert Haas <robertmhaas@gmail.com> — 2019-03-12T15:13:14Z

    On Mon, Mar 11, 2019 at 2:45 AM Amit Langote
<Langote_Amit_f8@lab.ntt.co.jp> wrote:
> I noticed another issue with the code -- it's using strcmp() to compare
> specified string against "minvalue" and "maxvalue", which causes the
> following silly error:
>
> create table q2 partition of q for values from ("MINVALUE") to (maxvalue);
> ERROR:  column "MINVALUE" does not exist
> LINE 1: create table q2 partition of q for values from ("MINVALUE") ...
>
> It should be using pg_strncasecmp().

Uh, why?  Generally, an unquoted keyword is equivalent to a quoted
lowercase version of that same keyword, not anything else.  Like
CREATE TABLE "foo" = CREATE TABLE FOO <> CREATE TABLE "FOO".

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  9. Re: BUG #15668: Server crash in transformPartitionRangeBounds

    Tom Lane <tgl@sss.pgh.pa.us> — 2019-03-12T16:35:20Z

    Robert Haas <robertmhaas@gmail.com> writes:
> On Mon, Mar 11, 2019 at 2:45 AM Amit Langote
> <Langote_Amit_f8@lab.ntt.co.jp> wrote:
>> I noticed another issue with the code -- it's using strcmp() to compare
>> specified string against "minvalue" and "maxvalue", which causes the
>> following silly error:
>> 
>> create table q2 partition of q for values from ("MINVALUE") to (maxvalue);
>> ERROR:  column "MINVALUE" does not exist
>> LINE 1: create table q2 partition of q for values from ("MINVALUE") ...
>> 
>> It should be using pg_strncasecmp().

> Uh, why?  Generally, an unquoted keyword is equivalent to a quoted
> lowercase version of that same keyword, not anything else.  Like
> CREATE TABLE "foo" = CREATE TABLE FOO <> CREATE TABLE "FOO".

Yeah.  The behavior shown above is entirely correct, and accepting the
statement would be flat out wrong; it would cause trouble if somebody
created a table containing multiple case-variations of MINVALUE.

			regards, tom lane

  10. Re: BUG #15668: Server crash in transformPartitionRangeBounds

    Amit Langote <langote_amit_f8@lab.ntt.co.jp> — 2019-03-13T04:08:01Z

    On 2019/03/13 1:35, Tom Lane wrote:
> Robert Haas <robertmhaas@gmail.com> writes:
>> On Mon, Mar 11, 2019 at 2:45 AM Amit Langote
>> <Langote_Amit_f8@lab.ntt.co.jp> wrote:
>>> I noticed another issue with the code -- it's using strcmp() to compare
>>> specified string against "minvalue" and "maxvalue", which causes the
>>> following silly error:
>>>
>>> create table q2 partition of q for values from ("MINVALUE") to (maxvalue);
>>> ERROR:  column "MINVALUE" does not exist
>>> LINE 1: create table q2 partition of q for values from ("MINVALUE") ...
>>>
>>> It should be using pg_strncasecmp().
> 
>> Uh, why?  Generally, an unquoted keyword is equivalent to a quoted
>> lowercase version of that same keyword, not anything else.  Like
>> CREATE TABLE "foo" = CREATE TABLE FOO <> CREATE TABLE "FOO".

OK.  Perhaps, I reacted too strongly to encountering the following
behavior with HEAD:

create table p1 partition of p for values from ("minValue") to (1);
ERROR:  column "minValue" does not exist

but,

create table p1 partition of p for values from ("minvalue") to (1);
\d p1
                 Table "public.p1"
 Column │  Type   │ Collation │ Nullable │ Default
────────┼─────────┼───────────┼──────────┼─────────
 a      │ integer │           │          │
Partition of: p FOR VALUES FROM (MINVALUE) TO (1)

But as you and Tom have pointed out, maybe it's normal.

> Yeah.  The behavior shown above is entirely correct, and accepting the
> statement would be flat out wrong; it would cause trouble if somebody
> created a table containing multiple case-variations of MINVALUE.

Sorry, I didn't understand this last part.  Different case-variations will
all be interpreted as a minvalue (negative infinity) range bound and
flagged if the resulting range bound constraint would be invalid.

Did you mean something like the following:

create table p1 partition of ... from ("minValue") to ("MINVALUE");


which using pg_strncasecmp() comparisons gives:

create table p1 partition of p for values from ("minValue") to ("MINVALUE");
ERROR:  empty range bound specified for partition "p1"
DETAIL:  Specified lower bound (MINVALUE) is greater than or equal to
upper bound (MINVALUE).

which is same as the behavior with unquoted keyword syntax:

create table p1 partition of p for values from (minValue) to (MINVALUE);
ERROR:  empty range bound specified for partition "p1"
DETAIL:  Specified lower bound (MINVALUE) is greater than or equal to
upper bound (MINVALUE).

whereas quoted identifier syntax on HEAD gives:

create table p1 partition of p for values from ("minValue") to ("MINVALUE");
ERROR:  column "minValue" does not exist
LINE 1: create table p1 partition of p for values from ("minValue") ...

However, as you guys said, HEAD is behaving sanely.

Thanks,
Amit

  11. Re: BUG #15668: Server crash in transformPartitionRangeBounds

    Amit Langote <langote_amit_f8@lab.ntt.co.jp> — 2019-03-13T05:15:24Z

    On 2019/03/11 16:21, Michael Paquier wrote:
> On Mon, Mar 11, 2019 at 03:44:39PM +0900, Amit Langote wrote:
>> We could make the error message more meaningful depending on the context,
>> but maybe it'd better be pursue it as a separate project.
> 
> Yeah, I noticed that stuff when working on it this afternoon.  The
> error message does not completely feel right even in your produced
> tests.  Out of curiosity I have been working on this thing myself,
> and it is possible to have a context-related message.  Please see
> attached, that's in my opinion less confusing, and of course
> debatable.  Still this approach does not feel completely right either
> as that means hijacking the code path which generates a generic
> message for missing RTEs. :(

@@ -3259,6 +3259,9 @@ errorMissingRTE(ParseState *pstate, RangeVar
+	 *
+	 * Also, in the context of parsing a partition bound, produce a more
+	 * helpful error message.
 	 */
 	if (rte && rte->alias &&
 		strcmp(rte->eref->aliasname, relation->relname) != 0 &&


-	if (rte)
+	if (pstate->p_expr_kind == EXPR_KIND_PARTITION_BOUND)
+		ereport(ERROR,
+				(errcode(ERRCODE_UNDEFINED_TABLE),
+				 errmsg("invalid reference in partition bound expression for table
\"%s\"",
+						relation->relname)));
+	else if (rte)

Hmm, it seems odd to me that it's OK for default expressions to emit the
"missing RTE" error, whereas partition bound expressions would emit this
special error message?

create table foo (a int default (bar.a));
ERROR:  missing FROM-clause entry for table "bar"

Thanks,
Amit

  12. Re: BUG #15668: Server crash in transformPartitionRangeBounds

    Amit Langote <langote_amit_f8@lab.ntt.co.jp> — 2019-03-13T06:17:47Z

    On 2019/03/13 14:15, Amit Langote wrote:
> On 2019/03/11 16:21, Michael Paquier wrote:
>> On Mon, Mar 11, 2019 at 03:44:39PM +0900, Amit Langote wrote:
>>> We could make the error message more meaningful depending on the context,
>>> but maybe it'd better be pursue it as a separate project.
>>
>> Yeah, I noticed that stuff when working on it this afternoon.  The
>> error message does not completely feel right even in your produced
>> tests.  Out of curiosity I have been working on this thing myself,
>> and it is possible to have a context-related message.  Please see
>> attached, that's in my opinion less confusing, and of course
>> debatable.  Still this approach does not feel completely right either
>> as that means hijacking the code path which generates a generic
>> message for missing RTEs. :(
> 
> @@ -3259,6 +3259,9 @@ errorMissingRTE(ParseState *pstate, RangeVar
> +	 *
> +	 * Also, in the context of parsing a partition bound, produce a more
> +	 * helpful error message.
>  	 */
>  	if (rte && rte->alias &&
>  		strcmp(rte->eref->aliasname, relation->relname) != 0 &&
> 
> 
> -	if (rte)
> +	if (pstate->p_expr_kind == EXPR_KIND_PARTITION_BOUND)
> +		ereport(ERROR,
> +				(errcode(ERRCODE_UNDEFINED_TABLE),
> +				 errmsg("invalid reference in partition bound expression for table
> \"%s\"",
> +						relation->relname)));
> +	else if (rte)
> 
> Hmm, it seems odd to me that it's OK for default expressions to emit the
> "missing RTE" error, whereas partition bound expressions would emit this
> special error message?
> 
> create table foo (a int default (bar.a));
> ERROR:  missing FROM-clause entry for table "bar"

Looking into this a bit more, I wonder if it would be a good idea to to
have one of those error-emitting switch (pstate->p_expr_kind) blocks in
transformColumnRef(), as shown in the attached patch?

For example, the following error is emitted by one of such blocks that's
in transformSubLink():

create table foo (a int default (select * from non_existent_table));
ERROR:  cannot use subquery in DEFAULT expression

However, if we decide to go with this approach, we will start getting
different error messages from what HEAD gives in certain cases.  With the
patch, you will get the following error when trying to use an aggregate
function in for DEFAULT expression:

create table foo (a int default (avg(foo.a)));
ERROR:  cannot use column reference in DEFAULT expression

but on HEAD, you get:

create table foo (a int default (avg(foo.a)));
ERROR:  aggregate functions are not allowed in DEFAULT expressions

That's because, on HEAD, transformAggregateCall() (or something that calls
it) first calls transformColumnRef() to resolve 'foo.a', which checks that
foo.a is a valid column reference, but it doesn't concern itself with the
fact that the bigger expression it's part of is being used for DEFAULT
expression.  It's only after 'foo.a' has been resolved as a valid column
reference that check_agglevels_and_constraints(), via
transformAggregateCall, emits an error that the overall expression is
invalid to use as a DEFAULT expression.  With patches, error will be
emitted even before resolving 'foo.a'.

While transformAggregateCall() works like that, transformSubLink(), which
I mentioned in the first example, doesn't bother to analyze the query
first (select * from non_existent_table) to notice that the referenced
table doesn't exist.  If it had bothered to analyze the query first, we
would've most likely gotten an error from errorMissingRTE(), not what we
get today.  So, there are certainly some inconsistencies even today in how
these errors are emitted.

Thanks,
Amit

  13. Re: BUG #15668: Server crash in transformPartitionRangeBounds

    Michael Paquier <michael@paquier.xyz> — 2019-03-14T04:23:08Z

    On Wed, Mar 13, 2019 at 03:17:47PM +0900, Amit Langote wrote:
> but on HEAD, you get:
> 
> create table foo (a int default (avg(foo.a)));
> ERROR:  aggregate functions are not allowed in DEFAULT expressions

I actually think that what you propose here makes more sense than what
HEAD does because the most inner expression gets evaluated first.
This for example generates the same error as on HEAD:
=# create table foo (a int default (avg(1)));
ERROR:  42803: aggregate functions are not allowed in DEFAULT expressions
--
Michael

  14. Re: BUG #15668: Server crash in transformPartitionRangeBounds

    Michael Paquier <michael@paquier.xyz> — 2019-03-20T02:07:31Z

    On Thu, Mar 14, 2019 at 01:23:08PM +0900, Michael Paquier wrote:
> I actually think that what you propose here makes more sense than what
> HEAD does because the most inner expression gets evaluated first.
> This for example generates the same error as on HEAD:
> =# create table foo (a int default (avg(1)));
> ERROR:  42803: aggregate functions are not allowed in DEFAULT expressions

I have been working on that, and in the case of a non-existing column
the patch would generate the following error on HEAD:
ERROR:  42703: column "non_existent" does not exist
But with the patch we get that:
ERROR:  cannot use column reference in DEFAULT expression

Still I think that this looks right as we should not have any direct
column reference anyway, and it keeps the code more simple.  So I have
added more tests to cover those grounds.

The docs of CREATE TABLE are actually wrong, right?  It mentions that
"subqueries and cross-references to other columns in the current table
are not allowed", but cookDefault() rejects any kind of column
references anyway for default expressions, including references to the
column which uses the default expression (or we talk about generated
columns here still if I recall Peter Eisentraunt's patch correctly
generated columns don't allow references to the column using the
expression itself, which is logic by the way).

+    * transformExpr() should have already rejected column references,
+    * subqueries, aggregates, window functions, and SRFs, based on the
+    * EXPR_KIND_ for a default expression.
     */
I would have added an assertion here, perhaps an elog().  Same remark
for cookDefault().  The attached patch has an assertion.

 CREATE TABLE part_bogus_expr_fail PARTITION OF list_parted FOR VALUES
 IN (sum(a));
-ERROR:  aggregate functions are not allowed in partition bound
+ERROR:  cannot use column reference in partition bound expression
It would be nice to also test the case where an aggregate is
forbidden, so I have added a test with sum(1) instead of a column
reference.

We never actually tested in the tree the case of subqueries and SRFs
used in default expressions, so added.

The last patch you sent did not fix the original problem of the
thread.  That was intentional from your side I guess to show your
point, still we are touching the same area of the code so I propose to
fix everything together, and to improve the test coverage for list and
range strategies.  In order to achieve that, I have merged my previous 
proposal into your patch, and added more tests.  The new tests for the
range strategy reproduce the crash.  The result is attached.

What do you think?
--
Michael

  15. Re: BUG #15668: Server crash in transformPartitionRangeBounds

    Amit Langote <langote_amit_f8@lab.ntt.co.jp> — 2019-03-20T09:07:23Z

    Hi,

On 2019/03/20 11:07, Michael Paquier wrote:
> On Thu, Mar 14, 2019 at 01:23:08PM +0900, Michael Paquier wrote:
>> I actually think that what you propose here makes more sense than what
>> HEAD does because the most inner expression gets evaluated first.
>> This for example generates the same error as on HEAD:
>> =# create table foo (a int default (avg(1)));
>> ERROR:  42803: aggregate functions are not allowed in DEFAULT expressions
> 
> I have been working on that,

Thanks a lot.

> and in the case of a non-existing column
> the patch would generate the following error on HEAD:
> ERROR:  42703: column "non_existent" does not exist
> But with the patch we get that:
> ERROR:  cannot use column reference in DEFAULT expression
> 
> Still I think that this looks right as we should not have any direct
> column reference anyway, and it keeps the code more simple.  So I have
> added more tests to cover those grounds.

I agree that we should error out immediately, once we encounter an (sub-)
expression that's not supported by a given feature (default, partition
bounds, etc.)

Actually, doesn't that mean we should error out because of the aggregate
in the following example:

create table foo (a int default (avg(a));

because we can notice the aggregate before we look into its arguments.
Maybe, we should move the error-checking switch to a point before checking
the arguments?  That looks slightly more drastic change to make though.

> The docs of CREATE TABLE are actually wrong, right?  It mentions that
> "subqueries and cross-references to other columns in the current table
> are not allowed", but cookDefault() rejects any kind of column
> references anyway for default expressions, including references to the
> column which uses the default expression (or we talk about generated
> columns here still if I recall Peter Eisentraunt's patch correctly
> generated columns don't allow references to the column using the
> expression itself, which is logic by the way).

Yeah, the documentation in your patch looks correct at a first glance.

> +    * transformExpr() should have already rejected column references,
> +    * subqueries, aggregates, window functions, and SRFs, based on the
> +    * EXPR_KIND_ for a default expression.
>      */
> I would have added an assertion here, perhaps an elog().  Same remark
> for cookDefault().  The attached patch has an assertion.

+1

>  CREATE TABLE part_bogus_expr_fail PARTITION OF list_parted FOR VALUES
>  IN (sum(a));
> -ERROR:  aggregate functions are not allowed in partition bound
> +ERROR:  cannot use column reference in partition bound expression
> It would be nice to also test the case where an aggregate is
> forbidden, so I have added a test with sum(1) instead of a column
> reference.

As I said above, maybe we should try to rearrange things so that we get
the former error in both cases.

> We never actually tested in the tree the case of subqueries and SRFs
> used in default expressions, so added.
> 
> The last patch you sent did not fix the original problem of the
> thread.  That was intentional from your side I guess to show your
> point,

Yeah, that's right.

> still we are touching the same area of the code so I propose to
> fix everything together, and to improve the test coverage for list and
> range strategies.  In order to achieve that, I have merged my previous 
> proposal into your patch, and added more tests.  The new tests for the
> range strategy reproduce the crash.  The result is attached.

We may want to fix the crash first.  It might be better to hear other
opinions before doing something about the error messages.

Thanks,
Amit

  16. Re: BUG #15668: Server crash in transformPartitionRangeBounds

    Michael Paquier <michael@paquier.xyz> — 2019-03-20T09:17:27Z

    On Wed, Mar 20, 2019 at 06:07:23PM +0900, Amit Langote wrote:
> because we can notice the aggregate before we look into its arguments.
> Maybe, we should move the error-checking switch to a point before checking
> the arguments?  That looks slightly more drastic change to make though.

Yeah, I think that it would be more invasive because the parsing logic
looks at the column references first.  This way of doing things is
also something which is logic in its own way as the most internal
portions of an expression get checked first, so I quite like that way
of doing things.  And that's just consistent with the point of view of
the parsing check order.

> We may want to fix the crash first.  It might be better to hear other
> opinions before doing something about the error messages.

The thing is that in order to keep the tests for the crash, we finish
with the inintuitive RTE-related errors, so it is also inconsistent to
not group things..
--
Michael

  17. Re: BUG #15668: Server crash in transformPartitionRangeBounds

    Michael Paquier <michael@paquier.xyz> — 2019-03-22T05:09:57Z

    On Wed, Mar 20, 2019 at 06:17:27PM +0900, Michael Paquier wrote:
> The thing is that in order to keep the tests for the crash, we finish
> with the inintuitive RTE-related errors, so it is also inconsistent to
> not group things..

As I have seen no feedback from others regarding the changes in error
messages depending on the parsing context, so I have been looking at
splitting the fix for the crash and changing the error messages, and
attached is the result of the split (minus the commit messages).  The
first patch fixes the crash, and includes test cases to cover the
crash as well as extra cases for list and range strategies with
partition bounds.  Some of the error messages are confusing, but that
fixes the issue.  This is not the most elegant thing without the
second patch, but well that could be worse.

The second patch adds better error context for the different error
messages, and includes tests for default expressions, which we could
discuss in a separate thread.  So I am not proposing to commit that
without more feedback.
--
Michael

  18. Re: BUG #15668: Server crash in transformPartitionRangeBounds

    Amit Langote <langote_amit_f8@lab.ntt.co.jp> — 2019-03-22T05:49:42Z

    Hi,

Thanks for splitting.  It makes sense, because, as you know, the bug that
causes the crash is a separate problem from unintuitive error messages
which result from the way in which we parse expressions.

On 2019/03/22 14:09, Michael Paquier wrote:
> On Wed, Mar 20, 2019 at 06:17:27PM +0900, Michael Paquier wrote:
>> The thing is that in order to keep the tests for the crash, we finish
>> with the inintuitive RTE-related errors, so it is also inconsistent to
>> not group things..
> 
> As I have seen no feedback from others regarding the changes in error
> messages depending on the parsing context, so I have been looking at
> splitting the fix for the crash and changing the error messages, and
> attached is the result of the split (minus the commit messages).  The
> first patch fixes the crash, and includes test cases to cover the
> crash as well as extra cases for list and range strategies with
> partition bounds.  Some of the error messages are confusing, but that
> fixes the issue.  This is not the most elegant thing without the
> second patch, but well that could be worse.

A comment on this one:

+	if (cname == NULL)
+	{
+		/*
+		 * No field names have been found, meaning that there
+		 * is not much to do with special value handling.  Instead
+		 * let the expression transformation handle any errors and
+		 * limitations.
+		 */

This comment sounds a bit misleading.  The code above this "did" find
field names, but there were too many.  What this comment should mention is
that parsing didn't return a single field name, which is the format that
the code below this can do something useful with.  I had proposed that
upthread, but maybe that feedback got lost in the discussion about other
related issues.

I had proposed this:

+      /*
+       * There should be a single field named "minvalue" or "maxvalue".
+       */
      if (list_length(cref->fields) == 1 &&
          IsA(linitial(cref->fields), String))
          cname = strVal(linitial(cref->fields));

-     Assert(cname != NULL);
-     if (strcmp("minvalue", cname) == 0)
+     if (cname == NULL)
+     {
+         /*
+          * ColumnRef is not in the desired single-field-name form; for
+          * consistency, let transformExpr() report the error rather
+          * than doing it ourselves.
+          */
+     }

Maybe that could use few more tweaks, but hope I've made my point.

+CREATE TABLE part_bogus_expr_fail PARTITION OF range_parted
+  FOR VALUES FROM (sum(a)) TO ('2019-01-01');
+ERROR:  function sum(date) does not exist
+LINE 2:   FOR VALUES FROM (sum(a)) TO ('2019-01-01');

Maybe, we should add to this patch only the tests relevant to the cases
that would lead to crash without this patch.

Tests regarding error messages fine tuning can be added in the other patch.

> The second patch adds better error context for the different error
> messages, and includes tests for default expressions, which we could
> discuss in a separate thread.  So I am not proposing to commit that
> without more feedback.

A separate thread will definitely attract more attention, at least in due
time. :)

Thanks,
Amit

  19. Re: BUG #15668: Server crash in transformPartitionRangeBounds

    Michael Paquier <michael@paquier.xyz> — 2019-03-26T01:15:32Z

    On Fri, Mar 22, 2019 at 02:49:42PM +0900, Amit Langote wrote:
> This comment sounds a bit misleading.  The code above this "did" find
> field names, but there were too many.  What this comment should mention is
> that parsing didn't return a single field name, which is the format that
> the code below this can do something useful with.  I had proposed that
> upthread, but maybe that feedback got lost in the discussion about other
> related issues.

True.  I was reviewing that stuff yesterday and I have not been able
to finish wrapping it.

> +CREATE TABLE part_bogus_expr_fail PARTITION OF range_parted
> +  FOR VALUES FROM (sum(a)) TO ('2019-01-01');
> +ERROR:  function sum(date) does not exist
> +LINE 2:   FOR VALUES FROM (sum(a)) TO ('2019-01-01');
> 
> Maybe, we should add to this patch only the tests relevant to the cases
> that would lead to crash without this patch.

Done as you suggested, with a minimal set enough to trigger the crash,
still the error message is rather misleading as you would expect :)

> A separate thread will definitely attract more attention, at least in due
> time. :)

Sure.  For now I have committed a lighter version of 0001, with
tweaked comments based on your suggestion, as well as a minimum set of
test cases.  I have added on the way some tests for range partitions
which have been missing from the start, and improved the existing set
by removing the original "a.a" references, and switching to use
max(date) for range partitions to bump correctly on the aggregate
error.  I am just updating the second patch now and I'll begin a new
thread soon.
--
Michael

  20. Re: BUG #15668: Server crash in transformPartitionRangeBounds

    Amit Langote <langote_amit_f8@lab.ntt.co.jp> — 2019-03-26T01:24:52Z

    Hi,

On 2019/03/26 10:15, Michael Paquier wrote:
> Done as you suggested, with a minimal set enough to trigger the crash,
> still the error message is rather misleading as you would expect :)

Thanks for committing.

>> A separate thread will definitely attract more attention, at least in due
>> time. :)
> 
> Sure.  For now I have committed a lighter version of 0001, with
> tweaked comments based on your suggestion, as well as a minimum set of
> test cases.  I have added on the way some tests for range partitions
> which have been missing from the start, and improved the existing set
> by removing the original "a.a" references, and switching to use
> max(date) for range partitions to bump correctly on the aggregate
> error.  I am just updating the second patch now and I'll begin a new
> thread soon.

Thanks.

Regards,
Amit