Horton

Thread

Commits

  1. Fix maintenance hazards caused by ill-considered use of default: cases.

  2. Fix handling of extended statistics during ALTER COLUMN TYPE.

  3. Standardize terminology for pg_statistic_ext entries.

  4. Complete tab completion for DROP STATISTICS

  5. getObjectDescription: support extended statistics

  6. Fix dependencies for extended statistics objects.

  7. Change CREATE STATISTICS syntax

  8. Add pg_dump tests for CREATE STATISTICS

  9. pg_dump/t/002: append terminating semicolon to SQL commands

  10. Improve pg_dump regression tests and code coverage

  1. WITH clause in CREATE STATISTICS

    Alvaro Herrera <alvherre@2ndquadrant.com> — 2017-04-20T21:24:26Z

    Simon just pointed out that having the WITH clause appear in the middle
of the CREATE STATISTICS command looks odd; apparently somebody else
already complained on list about the same.  Other commands put the WITH
clause at the end, so perhaps we should do likewise in the new command.

Here's a patch to implement that.  I verified that if I change
qualified_name to qualified_name_list, bison does not complain about
conflicts, so this new syntax should support extension to multiple
relations without a problem.

Discuss.

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  2. Re: WITH clause in CREATE STATISTICS

    Tom Lane <tgl@sss.pgh.pa.us> — 2017-04-20T22:13:17Z

    Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> Simon just pointed out that having the WITH clause appear in the middle
> of the CREATE STATISTICS command looks odd; apparently somebody else
> already complained on list about the same.  Other commands put the WITH
> clause at the end, so perhaps we should do likewise in the new command.

> Here's a patch to implement that.  I verified that if I change
> qualified_name to qualified_name_list, bison does not complain about
> conflicts, so this new syntax should support extension to multiple
> relations without a problem.

Yeah, WITH is fully reserved, so as long as the clause looks like
WITH ( stuff... ) you're pretty much gonna be able to drop it
wherever you want.

> Discuss.

+1 for WITH at the end; the existing syntax looks weird to me too.

			regards, tom lane

  3. Re: WITH clause in CREATE STATISTICS

    Tomas Vondra <tomas.vondra@2ndquadrant.com> — 2017-04-21T00:21:41Z

    On 04/21/2017 12:13 AM, Tom Lane wrote:
> Alvaro Herrera <alvherre@2ndquadrant.com> writes:
>> Simon just pointed out that having the WITH clause appear in the middle
>> of the CREATE STATISTICS command looks odd; apparently somebody else
>> already complained on list about the same.  Other commands put the WITH
>> clause at the end, so perhaps we should do likewise in the new command.
>
>> Here's a patch to implement that.  I verified that if I change
>> qualified_name to qualified_name_list, bison does not complain about
>> conflicts, so this new syntax should support extension to multiple
>> relations without a problem.
>
> Yeah, WITH is fully reserved, so as long as the clause looks like
> WITH ( stuff... ) you're pretty much gonna be able to drop it
> wherever you want.
>
>> Discuss.
>
> +1 for WITH at the end; the existing syntax looks weird to me too.
>

-1 from me

I like the current syntax more, and  WHERE ... WITH seems a bit weird to 
me. But more importantly, one thing Dean probably considered when 
proposing the current syntax was that we may add support for partial 
statistics, pretty much like partial indexes. And we don't allow WITH at 
the end (after WHERE) for indexes:

test=# create index on t (a) where a < 100 with (fillfactor=10);
ERROR:  syntax error at or near "with"
LINE 1: create index on t (a) where a < 100 with (fillfactor=10);
                                             ^
test=# create index on t (a) with (fillfactor=10) where a < 100;


regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  4. Re: WITH clause in CREATE STATISTICS

    Tels <nospam-pg-abuse@bloodgate.com> — 2017-04-21T10:30:26Z

    Moin,

On Thu, April 20, 2017 8:21 pm, Tomas Vondra wrote:
> On 04/21/2017 12:13 AM, Tom Lane wrote:
>> Alvaro Herrera <alvherre@2ndquadrant.com> writes:
>>> Simon just pointed out that having the WITH clause appear in the middle
>>> of the CREATE STATISTICS command looks odd; apparently somebody else
>>> already complained on list about the same.  Other commands put the WITH
>>> clause at the end, so perhaps we should do likewise in the new command.
>>
>>> Here's a patch to implement that.  I verified that if I change
>>> qualified_name to qualified_name_list, bison does not complain about
>>> conflicts, so this new syntax should support extension to multiple
>>> relations without a problem.
>>
>> Yeah, WITH is fully reserved, so as long as the clause looks like
>> WITH ( stuff... ) you're pretty much gonna be able to drop it
>> wherever you want.
>>
>>> Discuss.
>>
>> +1 for WITH at the end; the existing syntax looks weird to me too.
>>
>
> -1 from me
>
> I like the current syntax more, and  WHERE ... WITH seems a bit weird to
> me. But more importantly, one thing Dean probably considered when
> proposing the current syntax was that we may add support for partial
> statistics, pretty much like partial indexes. And we don't allow WITH at
> the end (after WHERE) for indexes:
>
> test=# create index on t (a) where a < 100 with (fillfactor=10);
> ERROR:  syntax error at or near "with"
> LINE 1: create index on t (a) where a < 100 with (fillfactor=10);
>                                              ^
> test=# create index on t (a) with (fillfactor=10) where a < 100;

While I'm not sure about where to put the WITH, so to speak, I do favour
consistency.

So I'm inclinded to keep the syntax like for the "create index".

More importantly however, I'd rather see the syntax on the "ON (column)
FROM relname" to be changed to match the existing examples. (Already wrote
this to Simon, not sure if my email made it to the list)

So instead:

 CREATE STATISTICS stats_name ON (columns) FROM relname

I'd rather see:

 CREATE STATISTICS stats_name ON table(col);

as this both mirrors CREATE INDEX and foreign keys with REFERENCES. It
could also be extended to both more columns, expressions or other tables
like so:

 CREATE STATISTICS stats ON t1(col1, col2 / 2), t2 (a,b);

and even:

 CREATE STATISTICS stats ON t1(col1, col2 / 2), t2 (a,b) WITH (options)
WHERE t2.a > 4;

This looks easy to remember, since it compares to:

 CREATE INDEX idx_name ON t2(a,b) WITH (options) WHERE t2.a > 4;

Or am I'm missing something?

Regards,

Tels

  5. Re: WITH clause in CREATE STATISTICS

    David Rowley <david.rowley@2ndquadrant.com> — 2017-04-21T11:04:38Z

    On 21 April 2017 at 22:30, Tels <nospam-pg-abuse@bloodgate.com> wrote:
> I'd rather see:
>
>  CREATE STATISTICS stats_name ON table(col);
>
> as this both mirrors CREATE INDEX and foreign keys with REFERENCES. It
> could also be extended to both more columns, expressions or other tables
> like so:
>
>  CREATE STATISTICS stats ON t1(col1, col2 / 2), t2 (a,b);
>
> and even:
>
>  CREATE STATISTICS stats ON t1(col1, col2 / 2), t2 (a,b) WITH (options)
> WHERE t2.a > 4;

How would you express a join condition with that syntax?

> This looks easy to remember, since it compares to:
>
>  CREATE INDEX idx_name ON t2(a,b) WITH (options) WHERE t2.a > 4;
>
> Or am I'm missing something?

Sadly yes, you are, and it's not the first time.

I seem to remember mentioning this to you already in [1].

Please, can you read over [2], it mentions exactly what you're
proposing and why it's not any good.

[1] https://www.postgresql.org/message-id/CAKJS1f9HMeT+7adicEaU8heOMpOB5pKkCVYZLiEZje3DVutVPw@mail.gmail.com
[2] https://www.postgresql.org/message-id/CAEZATCUtGR+U5+QTwjHhe9rLG2nguEysHQ5NaqcK=VbJ78VQFA@mail.gmail.com

-- 
 David Rowley                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

  6. Re: WITH clause in CREATE STATISTICS

    Tels <nospam-pg-abuse@bloodgate.com> — 2017-04-21T13:52:05Z

    Moin,

On Fri, April 21, 2017 7:04 am, David Rowley wrote:
> On 21 April 2017 at 22:30, Tels <nospam-pg-abuse@bloodgate.com> wrote:
>> I'd rather see:
>>
>>  CREATE STATISTICS stats_name ON table(col);
>>
>> as this both mirrors CREATE INDEX and foreign keys with REFERENCES. It
>> could also be extended to both more columns, expressions or other tables
>> like so:
>>
>>  CREATE STATISTICS stats ON t1(col1, col2 / 2), t2 (a,b);
>>
>> and even:
>>
>>  CREATE STATISTICS stats ON t1(col1, col2 / 2), t2 (a,b) WITH (options)
>> WHERE t2.a > 4;
>
> How would you express a join condition with that syntax?
>
>> This looks easy to remember, since it compares to:
>>
>>  CREATE INDEX idx_name ON t2(a,b) WITH (options) WHERE t2.a > 4;
>>
>> Or am I'm missing something?
>
> Sadly yes, you are, and it's not the first time.
>
> I seem to remember mentioning this to you already in [1].
>
> Please, can you read over [2], it mentions exactly what you're
> proposing and why it's not any good.
>
> [1]
> https://www.postgresql.org/message-id/CAKJS1f9HMeT+7adicEaU8heOMpOB5pKkCVYZLiEZje3DVutVPw@mail.gmail.com
> [2]
> https://www.postgresql.org/message-id/CAEZATCUtGR+U5+QTwjHhe9rLG2nguEysHQ5NaqcK=VbJ78VQFA@mail.gmail.com

Ah, ok, thank you, somehow I missed your answer the first time. So, just
ignore me :)

Best wishes,

Tels

  7. Re: WITH clause in CREATE STATISTICS

    Dean Rasheed <dean.a.rasheed@gmail.com> — 2017-04-22T08:08:56Z

    On 21 April 2017 at 01:21, Tomas Vondra <tomas.vondra@2ndquadrant.com> wrote:
> On 04/21/2017 12:13 AM, Tom Lane wrote:
>>
>> Alvaro Herrera <alvherre@2ndquadrant.com> writes:
>>>
>>> Simon just pointed out that having the WITH clause appear in the middle
>>> of the CREATE STATISTICS command looks odd; apparently somebody else
>>> already complained on list about the same.  Other commands put the WITH
>>> clause at the end, so perhaps we should do likewise in the new command.
>>
>> +1 for WITH at the end; the existing syntax looks weird to me too.
>
> -1 from me
>

Yeah, I'm still marginally in favour of the current syntax because
it's a bit more consistent with the CREATE VIEW syntax which puts the
WITH (options) clause before the query, and assuming that multi-table
and partial statistics support do get added in the future, the thing
that the statistics get created on is going to look more like a query.

OTOH, a few people have now commented that the syntax looks weird, and
not just because of the placement of the WITH clause. I don't have any
better ideas, but it's not too late to change if anyone else does...

Regards,
Dean

  8. Re: WITH clause in CREATE STATISTICS

    Simon Riggs <simon@2ndquadrant.com> — 2017-04-22T09:30:32Z

    On 21 April 2017 at 01:21, Tomas Vondra <tomas.vondra@2ndquadrant.com> wrote:
> On 04/21/2017 12:13 AM, Tom Lane wrote:
>>
>> Alvaro Herrera <alvherre@2ndquadrant.com> writes:
>>>
>>> Simon just pointed out that having the WITH clause appear in the middle
>>> of the CREATE STATISTICS command looks odd; apparently somebody else
>>> already complained on list about the same.  Other commands put the WITH
>>> clause at the end, so perhaps we should do likewise in the new command.
>>
>>
>>> Here's a patch to implement that.  I verified that if I change
>>> qualified_name to qualified_name_list, bison does not complain about
>>> conflicts, so this new syntax should support extension to multiple
>>> relations without a problem.
>>
>>
>> Yeah, WITH is fully reserved, so as long as the clause looks like
>> WITH ( stuff... ) you're pretty much gonna be able to drop it
>> wherever you want.
>>
>>> Discuss.
>>
>>
>> +1 for WITH at the end; the existing syntax looks weird to me too.
>>
>
> -1 from me
>
> I like the current syntax more, and  WHERE ... WITH seems a bit weird to me.
> But more importantly, one thing Dean probably considered when proposing the
> current syntax was that we may add support for partial statistics, pretty
> much like partial indexes. And we don't allow WITH at the end (after WHERE)
> for indexes:
>
> test=# create index on t (a) where a < 100 with (fillfactor=10);
> ERROR:  syntax error at or near "with"
> LINE 1: create index on t (a) where a < 100 with (fillfactor=10);
>                                             ^
> test=# create index on t (a) with (fillfactor=10) where a < 100;

OK, didn't know about WHERE clause; makes sense.

Currently WITH is supported in two places, which feels definitely
wrong. The WITH clause is used elsewhere to provide optional
parameters, and if there are none present it is optional.

OK, so lets try...

1.
No keyword at all, just list of statistics we store (i.e. just lose the WITH)
CREATE STATISTICS s1 (dependencies, ndistinct) ON (a, b) FROM t1 WHERE
partial-stuff;

and if there are options, use the WITH for the optional parameters like this
CREATE STATISTICS s1 (dependencies, ndistinct) WITH (options) ON (a,
b) FROM t1 WHERE partial-stuff;

2.
USING keyword, no brackets
CREATE STATISTICS s1 USING (dependencies, ndistinct) ON (a, b) FROM t1
WHERE partial-stuff;

and if there are options, use the WITH for the optional parameters like this
CREATE STATISTICS s1 USING (dependencies, ndistinct) WITH (options) ON
(a, b) FROM t1 WHERE partial-stuff;


I think I like (2)

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  9. Re: WITH clause in CREATE STATISTICS

    Pavel Stehule <pavel.stehule@gmail.com> — 2017-04-22T10:29:54Z

    2017-04-22 11:30 GMT+02:00 Simon Riggs <simon@2ndquadrant.com>:

> On 21 April 2017 at 01:21, Tomas Vondra <tomas.vondra@2ndquadrant.com>
> wrote:
> > On 04/21/2017 12:13 AM, Tom Lane wrote:
> >>
> >> Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> >>>
> >>> Simon just pointed out that having the WITH clause appear in the middle
> >>> of the CREATE STATISTICS command looks odd; apparently somebody else
> >>> already complained on list about the same.  Other commands put the WITH
> >>> clause at the end, so perhaps we should do likewise in the new command.
> >>
> >>
> >>> Here's a patch to implement that.  I verified that if I change
> >>> qualified_name to qualified_name_list, bison does not complain about
> >>> conflicts, so this new syntax should support extension to multiple
> >>> relations without a problem.
> >>
> >>
> >> Yeah, WITH is fully reserved, so as long as the clause looks like
> >> WITH ( stuff... ) you're pretty much gonna be able to drop it
> >> wherever you want.
> >>
> >>> Discuss.
> >>
> >>
> >> +1 for WITH at the end; the existing syntax looks weird to me too.
> >>
> >
> > -1 from me
> >
> > I like the current syntax more, and  WHERE ... WITH seems a bit weird to
> me.
> > But more importantly, one thing Dean probably considered when proposing
> the
> > current syntax was that we may add support for partial statistics, pretty
> > much like partial indexes. And we don't allow WITH at the end (after
> WHERE)
> > for indexes:
> >
> > test=# create index on t (a) where a < 100 with (fillfactor=10);
> > ERROR:  syntax error at or near "with"
> > LINE 1: create index on t (a) where a < 100 with (fillfactor=10);
> >                                             ^
> > test=# create index on t (a) with (fillfactor=10) where a < 100;
>
> OK, didn't know about WHERE clause; makes sense.
>
> Currently WITH is supported in two places, which feels definitely
> wrong. The WITH clause is used elsewhere to provide optional
> parameters, and if there are none present it is optional.
>
> OK, so lets try...
>
> 1.
> No keyword at all, just list of statistics we store (i.e. just lose the
> WITH)
> CREATE STATISTICS s1 (dependencies, ndistinct) ON (a, b) FROM t1 WHERE
> partial-stuff;
>
> and if there are options, use the WITH for the optional parameters like
> this
> CREATE STATISTICS s1 (dependencies, ndistinct) WITH (options) ON (a,
> b) FROM t1 WHERE partial-stuff;
>
> 2.
> USING keyword, no brackets
> CREATE STATISTICS s1 USING (dependencies, ndistinct) ON (a, b) FROM t1
> WHERE partial-stuff;
>
> and if there are options, use the WITH for the optional parameters like
> this
> CREATE STATISTICS s1 USING (dependencies, ndistinct) WITH (options) ON
> (a, b) FROM t1 WHERE partial-stuff;
>
>
> I think I like (2)
>

+1

Regards

Pavel


>
> --
> Simon Riggs                http://www.2ndQuadrant.com/
> PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
>
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers
>

  10. Re: WITH clause in CREATE STATISTICS

    David Rowley <david.rowley@2ndquadrant.com> — 2017-04-22T10:39:32Z

    On 22 April 2017 at 21:30, Simon Riggs <simon@2ndquadrant.com> wrote:
> CREATE STATISTICS s1 USING (dependencies, ndistinct) ON (a, b) FROM t1
> WHERE partial-stuff;

+1

Seems much more CREATE INDEX like, and that's pretty good given that
most of the complaints so far were about it bearing enough resemblance
to CREATE INDEX

-- 
 David Rowley                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

  11. Re: WITH clause in CREATE STATISTICS

    Alvaro Herrera <alvherre@2ndquadrant.com> — 2017-05-03T15:18:01Z

    Simon Riggs wrote:

> 2.
> USING keyword, no brackets
> CREATE STATISTICS s1 USING (dependencies, ndistinct) ON (a, b) FROM t1
> WHERE partial-stuff;
> 
> and if there are options, use the WITH for the optional parameters like this
> CREATE STATISTICS s1 USING (dependencies, ndistinct) WITH (options) ON
> (a, b) FROM t1 WHERE partial-stuff;
> 
> 
> I think I like (2)

OK, sounds sensible.

Note that the USING list is also optional -- if you don't specify it, we
default to creating all stat types.  Also note that we currently don't
have any option other than stat types, so the WITH would always be empty
-- in other words we should remove it until we implement some option.

(I can readily see two possible options to implement for pg11, so the
omission of the WITH clause would be temporary:
 1. sample size to use instead of the per-column values
 2. whether to forcibly collect stats for all column for this stat
 object even if the column has gotten a SET STATISTICS 0
Surely there can be others.)

Patch attached that adds the USING clause replacing the WITH clause,
which is also optional and only accepts statistic types (it doesn't
accept "foo = OFF" anymore, as it seems pointless, but I'm open to
accepting it if people care about it.)

(This patch removes WITH, but I verified that bison accepts having both.
The second attached reversed patch is what I used for removal.)

In the meantime, I noticed that pg_dump support for extstats is not
covered, which I'll go fix next.

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  12. Re: WITH clause in CREATE STATISTICS

    Alvaro Herrera <alvherre@2ndquadrant.com> — 2017-05-03T17:27:46Z

    Alvaro Herrera wrote:

> In the meantime, I noticed that pg_dump support for extstats is not
> covered, which I'll go fix next.

Here I add one, which seems to work for me.

Considering that Stephen missed a terminating semicolon for test with
create_order 96 (the last one prior to my commit) in commit
31a8b77a9244, I propose that we change whatever is concatenating those
strings append a terminating semicolon.  (Surely we don't care about two
tests stanzas writing a single SQL command by omitting the semicolon
terminator.)

I wonder if there's any rationale to the create_order numbers.  Surely
we only care for objects that depend on others.

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  13. Re: WITH clause in CREATE STATISTICS

    Stephen Frost <sfrost@snowman.net> — 2017-05-03T17:41:48Z

    Alvaro,

* Alvaro Herrera (alvherre@2ndquadrant.com) wrote:
> Alvaro Herrera wrote:
> 
> > In the meantime, I noticed that pg_dump support for extstats is not
> > covered, which I'll go fix next.
> 
> Here I add one, which seems to work for me.
> 
> Considering that Stephen missed a terminating semicolon for test with
> create_order 96 (the last one prior to my commit) in commit
> 31a8b77a9244, I propose that we change whatever is concatenating those
> strings append a terminating semicolon.  (Surely we don't care about two
> tests stanzas writing a single SQL command by omitting the semicolon
> terminator.)

Whoops, sorry about that.  Yes, we could pretty easily add that.  The
create SQL is built up at the bottom of 002_pg_dump.pl:

$create_sql .= $tests{$test}->{create_sql};

> I wonder if there's any rationale to the create_order numbers.  Surely
> we only care for objects that depend on others.

Yes, it was just a way to manage those dependencies.  If there's value
in doing something more complicated then we could certainly do that, but
I'm not sure why it would be necessary to add that complexity.

Thanks!

Stephen

  14. Re: WITH clause in CREATE STATISTICS

    Alvaro Herrera <alvherre@2ndquadrant.com> — 2017-05-03T18:33:42Z

    Alvaro Herrera wrote:
> Simon Riggs wrote:
> 
> > 2.
> > USING keyword, no brackets
> > CREATE STATISTICS s1 USING (dependencies, ndistinct) ON (a, b) FROM t1
> > WHERE partial-stuff;
> > 
> > and if there are options, use the WITH for the optional parameters like this
> > CREATE STATISTICS s1 USING (dependencies, ndistinct) WITH (options) ON
> > (a, b) FROM t1 WHERE partial-stuff;
> > 
> > 
> > I think I like (2)
> 
> OK, sounds sensible.

Yawn.  So, we have not achieved the stated goal which was to get rid of
the ugly clause in the middle of the command; moreover we have gained
*yet another* clause in the middle of the command.  Is this really an
improvement?  We're trading this
    CREATE STATISTICS s1 WITH (dependencies, ndistinct, options) ON (a, b) FROM t1 WHERE partial-stuff;
for this:
    CREATE STATISTICS s1 USING (dependencies, ndistinct) WITH (options) ON (a, b) FROM t1 WHERE partial-stuff;

Can we decide by a show of hands, please, whether we're really on board
with this change?

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  15. Re: WITH clause in CREATE STATISTICS

    Alvaro Herrera <alvherre@2ndquadrant.com> — 2017-05-03T18:57:00Z

    Stephen Frost wrote:

> > Here I add one, which seems to work for me.

Pushed it -- I also added another one which specifies options, to test
WITH handling in ruleutils.

> > Considering that Stephen missed a terminating semicolon for test with
> > create_order 96 (the last one prior to my commit) in commit
> > 31a8b77a9244, I propose that we change whatever is concatenating those
> > strings append a terminating semicolon.  (Surely we don't care about two
> > tests stanzas writing a single SQL command by omitting the semicolon
> > terminator.)
> 
> Whoops, sorry about that.  Yes, we could pretty easily add that.  The
> create SQL is built up at the bottom of 002_pg_dump.pl:
> 
> $create_sql .= $tests{$test}->{create_sql};

Okay, I added it.

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  16. Re: WITH clause in CREATE STATISTICS

    Tom Lane <tgl@sss.pgh.pa.us> — 2017-05-03T20:42:56Z

    Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> Yawn.  So, we have not achieved the stated goal which was to get rid of
> the ugly clause in the middle of the command; moreover we have gained
> *yet another* clause in the middle of the command.  Is this really an
> improvement?  We're trading this
>     CREATE STATISTICS s1 WITH (dependencies, ndistinct, options) ON (a, b) FROM t1 WHERE partial-stuff;
> for this:
>     CREATE STATISTICS s1 USING (dependencies, ndistinct) WITH (options) ON (a, b) FROM t1 WHERE partial-stuff;

> Can we decide by a show of hands, please, whether we're really on board
> with this change?

That seems totally messy :-(

I can't see any good reason why "WITH (options)" can't be at the end of
the query.  WITH is a fully reserved word, there is not going to be any
danger of a parse problem from having it follow the FROM or WHERE clauses.
And the end is where we have other instances of "WITH (options)".

It also seems like we don't need to have *both* fully-reserved keywords
introducing each clause *and* parentheses around the lists.  Maybe
dropping the parens around the stats-types list and the column-names
list would help to declutter?  (But I'd keep parens around the WITH
options, for consistency with other statements.)

One other point is that as long as we've got reserved keywords introducing
each clause, there isn't actually an implementation reason why we couldn't
accept the clauses in any order.  Not sure I want to document it that way,
but it might not be a bad thing if the grammar was forgiving about whether
you write the USING or ON part first ...

			regards, tom lane

  17. Re: WITH clause in CREATE STATISTICS

    Andrew Dunstan <andrew.dunstan@2ndquadrant.com> — 2017-05-03T21:31:55Z

    
On 05/03/2017 04:42 PM, Tom Lane wrote:
> Alvaro Herrera <alvherre@2ndquadrant.com> writes:
>> Yawn.  So, we have not achieved the stated goal which was to get rid of
>> the ugly clause in the middle of the command; moreover we have gained
>> *yet another* clause in the middle of the command.  Is this really an
>> improvement?  We're trading this
>>     CREATE STATISTICS s1 WITH (dependencies, ndistinct, options) ON (a, b) FROM t1 WHERE partial-stuff;
>> for this:
>>     CREATE STATISTICS s1 USING (dependencies, ndistinct) WITH (options) ON (a, b) FROM t1 WHERE partial-stuff;
>> Can we decide by a show of hands, please, whether we're really on board
>> with this change?
> That seems totally messy :-(
>
> I can't see any good reason why "WITH (options)" can't be at the end of
> the query.  WITH is a fully reserved word, there is not going to be any
> danger of a parse problem from having it follow the FROM or WHERE clauses.
> And the end is where we have other instances of "WITH (options)".
>
> It also seems like we don't need to have *both* fully-reserved keywords
> introducing each clause *and* parentheses around the lists.  Maybe
> dropping the parens around the stats-types list and the column-names
> list would help to declutter?  (But I'd keep parens around the WITH
> options, for consistency with other statements.)
>
> One other point is that as long as we've got reserved keywords introducing
> each clause, there isn't actually an implementation reason why we couldn't
> accept the clauses in any order.  Not sure I want to document it that way,
> but it might not be a bad thing if the grammar was forgiving about whether
> you write the USING or ON part first ...
>
> 			


+1 for allowing arbitrary order of clauses. I would document it with the
USING clause at the end, and have that be what psql supports and pg_dump
produces. Since there are no WITH options now we should leave that out
until it's required.

cheers

andrew

-- 
Andrew Dunstan                https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  18. Re: WITH clause in CREATE STATISTICS

    Alvaro Herrera <alvherre@2ndquadrant.com> — 2017-05-03T21:36:55Z

    Andrew Dunstan wrote:

> On 05/03/2017 04:42 PM, Tom Lane wrote:

> > One other point is that as long as we've got reserved keywords introducing
> > each clause, there isn't actually an implementation reason why we couldn't
> > accept the clauses in any order.  Not sure I want to document it that way,
> > but it might not be a bad thing if the grammar was forgiving about whether
> > you write the USING or ON part first ...
> 
> +1 for allowing arbitrary order of clauses. I would document it with the
> USING clause at the end, and have that be what psql supports and pg_dump
> produces. Since there are no WITH options now we should leave that out
> until it's required.

Ok, sounds good to me.  Unless there are objections I'm going to have a
shot at implementing this.  Thanks for the discussion.

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  19. Re: WITH clause in CREATE STATISTICS

    Tomas Vondra <tomas.vondra@2ndquadrant.com> — 2017-05-03T22:12:20Z

    
On 5/3/17 11:36 PM, Alvaro Herrera wrote:
> Andrew Dunstan wrote:
> 
>> On 05/03/2017 04:42 PM, Tom Lane wrote:
> 
>>> One other point is that as long as we've got reserved keywords introducing
>>> each clause, there isn't actually an implementation reason why we couldn't
>>> accept the clauses in any order.  Not sure I want to document it that way,
>>> but it might not be a bad thing if the grammar was forgiving about whether
>>> you write the USING or ON part first ...
>>
>> +1 for allowing arbitrary order of clauses. I would document it with the
>> USING clause at the end, and have that be what psql supports and pg_dump
>> produces. Since there are no WITH options now we should leave that out
>> until it's required.
> 
> Ok, sounds good to me.  Unless there are objections I'm going to have a
> shot at implementing this.  Thanks for the discussion.
> 

Works for me. Do you also plan to remove the parentheses for the USING 
clause?

regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  20. Re: WITH clause in CREATE STATISTICS

    Simon Riggs <simon@2ndquadrant.com> — 2017-05-04T05:28:26Z

    On 3 May 2017 at 23:31, Andrew Dunstan <andrew.dunstan@2ndquadrant.com> wrote:

>> It also seems like we don't need to have *both* fully-reserved keywords
>> introducing each clause *and* parentheses around the lists.  Maybe
>> dropping the parens around the stats-types list and the column-names
>> list would help to declutter?  (But I'd keep parens around the WITH
>> options, for consistency with other statements.)

+1

>> One other point is that as long as we've got reserved keywords introducing
>> each clause, there isn't actually an implementation reason why we couldn't
>> accept the clauses in any order.  Not sure I want to document it that way,
>> but it might not be a bad thing if the grammar was forgiving about whether
>> you write the USING or ON part first ...
>
> +1 for allowing arbitrary order of clauses.

+1

> I would document it with the
> USING clause at the end, and have that be what psql supports and pg_dump
> produces. Since there are no WITH options now we should leave that out
> until it's required.

Let's record the target syntax in parser comments so we can just slot
things in when needed later, without rediscussion.

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  21. Re: WITH clause in CREATE STATISTICS

    Alvaro Herrera <alvherre@2ndquadrant.com> — 2017-05-04T18:48:33Z

    Here's a patch implementing this idea.  From gram.y's comment, the
support syntax is now:

  /*****************************************************************************
   *
   *        QUERY :
!  *                CREATE STATISTICS stats_name [(stat types)] arguments
! 
!  *            where 'arguments' can be one or more of:
!  *                    { ON (columns)
!  *                      | FROM relations
!  *                      | WITH (options)
!  *                      | WHERE expression }

Note that I removed the USING keyword in the stat types list, and also
made it mandatory that that list appears immediately after the new stats
name.  This should make it possible to have USING in the relation list
(the FROM clause), if we allow explicit multiple relations with join
syntax there.  The other options can appear in any order.

Also, both WITH and WHERE are accepted by the grammar, but immediately
throw "feature not implemented" error at parse time.

I was on the fence about adding copy/equal/out support for the new
StatisticArgument node; it seems pointless because that node does not
leave gram.y anyway.

Unless there are objections, I'll push this tomorrow.

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  22. Re: WITH clause in CREATE STATISTICS

    Tom Lane <tgl@sss.pgh.pa.us> — 2017-05-04T21:13:37Z

    Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> Here's a patch implementing this idea.  From gram.y's comment, the
> support syntax is now:

>   /*****************************************************************************
>    *
>    *        QUERY :
> !  *                CREATE STATISTICS stats_name [(stat types)] arguments
> ! 
> !  *            where 'arguments' can be one or more of:
> !  *                    { ON (columns)
> !  *                      | FROM relations
> !  *                      | WITH (options)
> !  *                      | WHERE expression }

> Note that I removed the USING keyword in the stat types list, and also
> made it mandatory that that list appears immediately after the new stats
> name.  This should make it possible to have USING in the relation list
> (the FROM clause), if we allow explicit multiple relations with join
> syntax there.  The other options can appear in any order.

Hmm ... I'm not sure that I buy that particular argument.  If you're
concerned that the grammar could not handle "FROM x JOIN y USING (z)",
wouldn't it also have a problem with "FROM x JOIN y ON (z)"?

It might work anyway, since the grammar should know whether ON or USING
is needed to complete the JOIN clause.  But I think you'd better check
whether the complete join syntax works there, even if we're not going
to support it now.

I'm not against what you've done here, because I had no love for USING
in this context anyway; it conveys approximately nothing to the mind
about what is in the list it's introducing.  But I'm concerned whether
we're boxing ourselves in by using ON.

Actually, "ON" doesn't seem all that mnemonic either.  Maybe "FOR"
would be a good substitute, if it turns out that "ON" has a problem?

			regards, tom lane

  23. Re: WITH clause in CREATE STATISTICS

    Sven R. Kunze <srkunze@mail.de> — 2017-05-04T22:16:40Z

    On 04.05.2017 23:13, Tom Lane wrote:
> I'm not against what you've done here, because I had no love for USING
> in this context anyway; it conveys approximately nothing to the mind
> about what is in the list it's introducing.  But I'm concerned whether
> we're boxing ourselves in by using ON.
>
> Actually, "ON" doesn't seem all that mnemonic either.  Maybe "FOR"
> would be a good substitute, if it turns out that "ON" has a problem?

The whole syntax reminds me of a regular SELECT clause. So, SELECT?


Also considering the most generic form of statistic support mentioned in 
[1], one could even thing about allowing aggregates, windowing functions 
etc, aka the full SELECT clause in the future.


Sven


[1] 
https://www.postgresql.org/message-id/CAEZATCUtGR+U5+QTwjHhe9rLG2nguEysHQ5NaqcK=VbJ78VQFA@mail.gmail.com

  24. Re: WITH clause in CREATE STATISTICS

    Alvaro Herrera <alvherre@2ndquadrant.com> — 2017-05-11T17:08:38Z

    Tom Lane wrote:

> Hmm ... I'm not sure that I buy that particular argument.  If you're
> concerned that the grammar could not handle "FROM x JOIN y USING (z)",
> wouldn't it also have a problem with "FROM x JOIN y ON (z)"?
> 
> It might work anyway, since the grammar should know whether ON or USING
> is needed to complete the JOIN clause.  But I think you'd better check
> whether the complete join syntax works there, even if we're not going
> to support it now.

Tomas spent some time trying to shoehorn the whole join syntax into the
FROM clause, but stopped once he realized that the joined_table
production uses table_ref, which allow things like TABLESAMPLE, SRFs,
LATERAL, etc which presumably we don't want to accept in CREATE STATS.
I didn't look into it any further.  But because of the other
considerations, I did end up changing the ON to FOR.

So the attached is the final version which I intend to push shortly.

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  25. Re: WITH clause in CREATE STATISTICS

    Tom Lane <tgl@sss.pgh.pa.us> — 2017-05-11T17:19:07Z

    Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> Tom Lane wrote:
>> Hmm ... I'm not sure that I buy that particular argument.  If you're
>> concerned that the grammar could not handle "FROM x JOIN y USING (z)",
>> wouldn't it also have a problem with "FROM x JOIN y ON (z)"?

> Tomas spent some time trying to shoehorn the whole join syntax into the
> FROM clause, but stopped once he realized that the joined_table
> production uses table_ref, which allow things like TABLESAMPLE, SRFs,
> LATERAL, etc which presumably we don't want to accept in CREATE STATS.
> I didn't look into it any further.  But because of the other
> considerations, I did end up changing the ON to FOR.

Have you thought further about the upthread suggestion to just borrow
SELECT's syntax lock stock and barrel?  That is, it'd look something
like

CREATE STATISTICS name [(list of stats types)] expression-list FROM ...
    [ WHERE ... ] [ WITH (options) ]

This would certainly support any FROM stuff we might want later.
Yeah, you'd need to reject any features you weren't supporting at
execution, but doing that would allow issuing a friendlier message than
"syntax error" anyway.

If you don't have the cycles for that, I'd be willing to look into it.

			regards, tom lane

  26. Re: WITH clause in CREATE STATISTICS

    Alvaro Herrera <alvherre@2ndquadrant.com> — 2017-05-11T17:22:07Z

    Tom Lane wrote:

> Have you thought further about the upthread suggestion to just borrow
> SELECT's syntax lock stock and barrel?  That is, it'd look something
> like
> 
> CREATE STATISTICS name [(list of stats types)] expression-list FROM ...
>     [ WHERE ... ] [ WITH (options) ]
> 
> This would certainly support any FROM stuff we might want later.
> Yeah, you'd need to reject any features you weren't supporting at
> execution, but doing that would allow issuing a friendlier message than
> "syntax error" anyway.

Hmm, no, I hadn't looked at this angle.

> If you don't have the cycles for that, I'd be willing to look into it.

I'll give it a try today, and pass the baton if I'm unable to.

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  27. Re: WITH clause in CREATE STATISTICS

    Alvaro Herrera <alvherre@2ndquadrant.com> — 2017-05-11T19:32:36Z

    Tom Lane wrote:

> Have you thought further about the upthread suggestion to just borrow
> SELECT's syntax lock stock and barrel?  That is, it'd look something
> like
> 
> CREATE STATISTICS name [(list of stats types)] expression-list FROM ...
>     [ WHERE ... ] [ WITH (options) ]
> 
> This would certainly support any FROM stuff we might want later.
> Yeah, you'd need to reject any features you weren't supporting at
> execution, but doing that would allow issuing a friendlier message than
> "syntax error" anyway.
> 
> If you don't have the cycles for that, I'd be willing to look into it.

Bison seems to like the productions below.  Is this what you had in
mind?  These mostly mimic joined_table and table_ref, stripping out the
rules that we don't need.

Note that I had to put back the FOR keyword in there; without the FOR,
that is "CREATE STATS any_name opt_name_list expr_list FROM" there was
one shift/reduce conflict, which I tried to solve by splitting
opt_name_list using two rules (one with "'(' name_list ')'" and one
without), but that gave rise to two reduce/reduce conflicts, and I didn't
see any way to deal with them.

(Note that I ended up realizing that stats_type_list is unnecessary
since we can use name_list.)


CreateStatsStmt:
			CREATE opt_if_not_exists STATISTICS any_name
			opt_name_list FOR expr_list FROM stats_joined_table
				{
					CreateStatsStmt *n = makeNode(CreateStatsStmt);
					n->defnames = $4;
					n->stat_types = $6;
					n->if_not_exists = $2;

					stmt->relation = $9;
					stmt->keys = $7;
					$$ = (Node *)n;
				}
			;

stats_joined_table:
		  '(' stats_joined_table ')'
				{
				}
			| stats_table_ref CROSS JOIN stats_table_ref
				{
				}
			| stats_table_ref join_type JOIN stats_table_ref join_qual
				{
				}
			| stats_table_ref JOIN stats_table_ref join_qual
				{
				}
			| stats_table_ref NATURAL join_type JOIN stats_table_ref
				{
				}
			| stats_table_ref NATURAL JOIN table_ref
				{
				}
		;

stats_table_ref:
			relation_expr opt_alias_clause
				{
				}
			| stats_joined_table
				{
				}
			| '(' stats_joined_table ')' alias_clause
				{
				}
		;



-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  28. Re: WITH clause in CREATE STATISTICS

    Tom Lane <tgl@sss.pgh.pa.us> — 2017-05-11T19:50:19Z

    Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> Tom Lane wrote:
>> Have you thought further about the upthread suggestion to just borrow
>> SELECT's syntax lock stock and barrel?

> Bison seems to like the productions below.  Is this what you had in
> mind?  These mostly mimic joined_table and table_ref, stripping out the
> rules that we don't need.

I'd suggest just using the from_list production and then complaining
at runtime if what you get is too complicated.  Otherwise, you have
to maintain a duplicate set of productions, and you're going to be
unable to throw anything more informative than "syntax error" when
somebody tries to exceed the implementation limits.

> Note that I had to put back the FOR keyword in there; without the FOR,
> that is "CREATE STATS any_name opt_name_list expr_list FROM" there was
> one shift/reduce conflict, which I tried to solve by splitting
> opt_name_list using two rules (one with "'(' name_list ')'" and one
> without), but that gave rise to two reduce/reduce conflicts, and I didn't
> see any way to deal with them.

That's fine.  I was going to suggest using ON after the stats type list
just because it seemed to read better.  FOR is about as good.

> (Note that I ended up realizing that stats_type_list is unnecessary
> since we can use name_list.)

Check.

			regards, tom lane

  29. Re: WITH clause in CREATE STATISTICS

    Alvaro Herrera <alvherre@2ndquadrant.com> — 2017-05-11T22:13:30Z

    Tom Lane wrote:
> Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> > Tom Lane wrote:
> >> Have you thought further about the upthread suggestion to just borrow
> >> SELECT's syntax lock stock and barrel?
> 
> > Bison seems to like the productions below.  Is this what you had in
> > mind?  These mostly mimic joined_table and table_ref, stripping out the
> > rules that we don't need.
> 
> I'd suggest just using the from_list production and then complaining
> at runtime if what you get is too complicated.  Otherwise, you have
> to maintain a duplicate set of productions, and you're going to be
> unable to throw anything more informative than "syntax error" when
> somebody tries to exceed the implementation limits.

Hmm, yeah, makes sense.  Here's a patch for this approach.  I ended up
using on ON again for the list of columns.  I suppose the checks in
CreateStatistics() could still be improved, but I'd go ahead and push
this tomorrow morning and we can hammer those details later on, if it's
still needed.  Better avoid shipping beta with outdated grammar ...

BTW the new castNode() family of macros don't work with Value nodes
(because the tags are different depending on what's stored, but each
type does not have its own struct.  Oh well.)

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  30. Re: WITH clause in CREATE STATISTICS

    Tom Lane <tgl@sss.pgh.pa.us> — 2017-05-11T22:37:12Z

    Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> BTW the new castNode() family of macros don't work with Value nodes
> (because the tags are different depending on what's stored, but each
> type does not have its own struct.  Oh well.)

Yeah.  Value nodes are pretty much of a wart --- it's not clear to me
that they add anything compared to just having a separate node type
for each of the possible subclasses.  It's never bothered anyone enough
to try to replace them, though.

I'll take a look at the patch later.

			regards, tom lane

  31. Re: WITH clause in CREATE STATISTICS

    Tom Lane <tgl@sss.pgh.pa.us> — 2017-05-12T02:46:45Z

    Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> Hmm, yeah, makes sense.  Here's a patch for this approach.

I did some code review on this patch.  The attached update makes the
following hopefully-uncontroversial changes:

* fix inconsistencies in the set of fields in CreateStatsStmt

* get rid of struct CreateStatsArgument, which seems to be a leftover

* tighten up parse checks in CreateStatistics(), and improve comments

* add missing check for ownership of target table; the documentation
  says this is checked, and surely it is a security bug that it's not

* adjust regression test to compensate for the above, because it
  fails otherwise

* change end of regression test to suppress cascade drop messages;
  it's astonishing that this test hasn't caused intermittent buildfarm
  failures due to unstable drop order

I did not review the rest of the regression test changes, nor the
tab-complete changes.  I can however report that DROP STATISTICS <tab>
doesn't successfully complete any stats object names.

Also, while reading the docs changes, I got rather ill from the
inconsistent and not very grammatical handling of "statistics" as a
noun, particularly the inability to decide from one sentence to the next
if that is singular or plural.  In the attached, I fixed this in the
ref/*_statistics.sgml files by always saying "statistics object" instead.
If people agree that that reads better, we should make an effort to
propagate that terminology elsewhere in the docs, and into error messages,
objectaddress.c output, etc.

Although I've not done anything about it here, I'm not happy about the
handling of dependencies for stats objects.  I do not think that cloning
RemoveStatistics as RemoveStatisticsExt is sane at all.  The former is
meant to deal with cleaning up pg_statistic rows that we know will be
there, so there's no real need to expend pg_depend overhead to track them.
For objects that are only loosely connected, the right thing is to use
the dependency system; in particular, this design has no real chance of
working well with cross-table stats.  Also, it's really silly to have
*both* this hard-wired mechanism and a pg_depend entry; the latter is
surely redundant if you have the former.  IMO we should revert
RemoveStatisticsExt and instead handle things by making stats objects
auto-dependent on the individual column(s) they reference (not the whole
table).

I'm also of the opinion that having an AUTO dependency, rather than
a NORMAL dependency, on the stats object's schema is the wrong semantics.
There isn't any other case where you can drop a non-empty schema without
saying CASCADE, and I'm mystified why this case should act that way.

Lastly, I tried the example given in the CREATE STATISTICS reference page,
and it doesn't seem to work.  Without the stats object, the two queries
are both estimated at one matching row, whereas they really produce 100
and 0 rows respectively.  With the stats object, they seem to both get
estimated at ~100 rows, which is a considerable improvement for one case
but a considerable disimprovement for the other.  If this is not broken,
I'd like to know why not.  What's the point of an expensive extended-
stats mechanism if it can't get this right?

			regards, tom lane

  32. Re: WITH clause in CREATE STATISTICS

    Tomas Vondra <tomas.vondra@2ndquadrant.com> — 2017-05-12T11:13:27Z

    
On 5/12/17 4:46 AM, Tom Lane wrote:
> Alvaro Herrera <alvherre@2ndquadrant.com> writes:
>> Hmm, yeah, makes sense.  Here's a patch for this approach.
> 
> ...
> 
> Also, while reading the docs changes, I got rather ill from the
> inconsistent and not very grammatical handling of "statistics" as a
> noun, particularly the inability to decide from one sentence to the next
> if that is singular or plural.  In the attached, I fixed this in the
> ref/*_statistics.sgml files by always saying "statistics object" instead.
> If people agree that that reads better, we should make an effort to
> propagate that terminology elsewhere in the docs, and into error messages,
> objectaddress.c output, etc.
> 

I'm fine with the 'statistics object' wording. I've been struggling with 
this bit while working on the patch, and I agree it sounds better and 
getting it consistent is definitely worthwhile.

 >
> Although I've not done anything about it here, I'm not happy about the
> handling of dependencies for stats objects.  I do not think that cloning
> RemoveStatistics as RemoveStatisticsExt is sane at all.  The former is
> meant to deal with cleaning up pg_statistic rows that we know will be
> there, so there's no real need to expend pg_depend overhead to track them.
> For objects that are only loosely connected, the right thing is to use
> the dependency system; in particular, this design has no real chance of
> working well with cross-table stats.  Also, it's really silly to have
> *both* this hard-wired mechanism and a pg_depend entry; the latter is
> surely redundant if you have the former.  IMO we should revert
> RemoveStatisticsExt and instead handle things by making stats objects
> auto-dependent on the individual column(s) they reference (not the whole
> table).
> 
> I'm also of the opinion that having an AUTO dependency, rather than
> a NORMAL dependency, on the stats object's schema is the wrong semantics.
> There isn't any other case where you can drop a non-empty schema without
> saying CASCADE, and I'm mystified why this case should act that way.
> 

Yeah, it's a bit frankensteinian ...

 >
> Lastly, I tried the example given in the CREATE STATISTICS reference page,
> and it doesn't seem to work.  Without the stats object, the two queries
> are both estimated at one matching row, whereas they really produce 100
> and 0 rows respectively.  With the stats object, they seem to both get
> estimated at ~100 rows, which is a considerable improvement for one case
> but a considerable disimprovement for the other.  If this is not broken,
> I'd like to know why not.  What's the point of an expensive extended-
> stats mechanism if it can't get this right?
> 

I assume you're talking about the functional dependencies and in that 
case that's expected behavior, because f. dependencies do assume the 
conditions are "consistent" with the functional dependencies.

This is an inherent limitation of functional dependencies, and perhaps a 
price for the simplicity of this statistics type. If your application 
executes queries that are likely not consistent with this, don't use 
functional dependencies.

The simplicity is why dependencies were implemented first, mostly to 
introduce all the infrastructure. The other statistics types (MCV, 
histograms) don't have this limitation, but those did not make it into pg10.


regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  33. Re: WITH clause in CREATE STATISTICS

    Tom Lane <tgl@sss.pgh.pa.us> — 2017-05-12T14:05:27Z

    Tomas Vondra <tomas.vondra@2ndquadrant.com> writes:
> On 5/12/17 4:46 AM, Tom Lane wrote:
>> If people agree that that reads better, we should make an effort to
>> propagate that terminology elsewhere in the docs, and into error messages,
>> objectaddress.c output, etc.

> I'm fine with the 'statistics object' wording. I've been struggling with 
> this bit while working on the patch, and I agree it sounds better and 
> getting it consistent is definitely worthwhile.

OK.  We can work on that as a followon patch.

>> Although I've not done anything about it here, I'm not happy about the
>> handling of dependencies for stats objects.

> Yeah, it's a bit frankensteinian ...

I'm prepared to create a fix for that, but it'd be easier to commit the
current patch first, to avoid merge conflicts.

>> Lastly, I tried the example given in the CREATE STATISTICS reference page,
>> and it doesn't seem to work.

> I assume you're talking about the functional dependencies and in that 
> case that's expected behavior, because f. dependencies do assume the 
> conditions are "consistent" with the functional dependencies.

Hm.  OK, but then that example is pretty misleading, because it leads
the reader to suppose that the planner can tell the difference between
the selectivities of the two queries.  Maybe what's lacking is an
explanation of how you'd use this statistics type.

			regards, tom lane

  34. Re: WITH clause in CREATE STATISTICS

    Alvaro Herrera <alvherre@2ndquadrant.com> — 2017-05-12T14:43:22Z

    Tom Lane wrote:
> Tomas Vondra <tomas.vondra@2ndquadrant.com> writes:

> >> Although I've not done anything about it here, I'm not happy about the
> >> handling of dependencies for stats objects.
> 
> > Yeah, it's a bit frankensteinian ...
> 
> I'm prepared to create a fix for that, but it'd be easier to commit the
> current patch first, to avoid merge conflicts.

It seems we're mostly in agreement regarding the parts I was touching.
Do you want to push your version of the patch?

> >> Lastly, I tried the example given in the CREATE STATISTICS reference page,
> >> and it doesn't seem to work.
> 
> > I assume you're talking about the functional dependencies and in that 
> > case that's expected behavior, because f. dependencies do assume the 
> > conditions are "consistent" with the functional dependencies.
> 
> Hm.  OK, but then that example is pretty misleading, because it leads
> the reader to suppose that the planner can tell the difference between
> the selectivities of the two queries.  Maybe what's lacking is an
> explanation of how you'd use this statistics type.

I think we should remove the complex example from that page, and instead
refer the reader to chapters 14 and 69.  The CREATE STATISTICS page can
just give some overview of what the syntax looks like, and all
explanations of complex topics such as "what does it mean for a query to
be consistent with the functional dependencies" can be given at length
elsewhere.

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  35. Re: WITH clause in CREATE STATISTICS

    Tom Lane <tgl@sss.pgh.pa.us> — 2017-05-12T15:06:26Z

    Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> Tom Lane wrote:
>> I'm prepared to create a fix for that, but it'd be easier to commit the
>> current patch first, to avoid merge conflicts.

> It seems we're mostly in agreement regarding the parts I was touching.
> Do you want to push your version of the patch?

It's still almost all your work, so I figured you should push it.

>> Hm.  OK, but then that example is pretty misleading, because it leads
>> the reader to suppose that the planner can tell the difference between
>> the selectivities of the two queries.  Maybe what's lacking is an
>> explanation of how you'd use this statistics type.

> I think we should remove the complex example from that page, and instead
> refer the reader to chapters 14 and 69.

Seems reasonable.  If we did add enough material there to be a standalone
description, it would be duplicative probably.  However, that does put
the onus on the other chapters to offer a complete definition, rather
than leaving details to the man page as we do in many other cases.

			regards, tom lane

  36. Re: WITH clause in CREATE STATISTICS

    Alvaro Herrera <alvherre@2ndquadrant.com> — 2017-05-12T15:28:46Z

    Tom Lane wrote:
> Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> > Tom Lane wrote:
> >> I'm prepared to create a fix for that, but it'd be easier to commit the
> >> current patch first, to avoid merge conflicts.
> 
> > It seems we're mostly in agreement regarding the parts I was touching.
> > Do you want to push your version of the patch?
> 
> It's still almost all your work, so I figured you should push it.

OK, I'll do that shortly then.

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  37. Tab-completing DROP STATISTICS

    Alvaro Herrera <alvherre@2ndquadrant.com> — 2017-05-12T19:52:50Z

    Tom Lane wrote:

> I did not review the rest of the regression test changes, nor the
> tab-complete changes.  I can however report that DROP STATISTICS <tab>
> doesn't successfully complete any stats object names.

The attached patch fixes this problem.

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  38. Re: WITH clause in CREATE STATISTICS

    Tom Lane <tgl@sss.pgh.pa.us> — 2017-05-12T20:36:30Z

    I wrote:
> Tomas Vondra <tomas.vondra@2ndquadrant.com> writes:
>> On 5/12/17 4:46 AM, Tom Lane wrote:
>>> Although I've not done anything about it here, I'm not happy about the
>>> handling of dependencies for stats objects.

>> Yeah, it's a bit frankensteinian ...

> I'm prepared to create a fix for that, but it'd be easier to commit the
> current patch first, to avoid merge conflicts.

Done.  I noted that we weren't making an owner dependency either :-(.

Also, we might want to think about letting stats objects be extension
members sometime.  As things stand, if an extension script does CREATE
STATISTICS, the stats object will just be "loose" rather than bound
into the extension.

We still have docs and nomenclature issues to work on.  Anyone want
to take point on those?

			regards, tom lane

  39. Re: WITH clause in CREATE STATISTICS

    Alvaro Herrera <alvherre@2ndquadrant.com> — 2017-05-12T20:48:01Z

    Tom Lane wrote:

> Although I've not done anything about it here, I'm not happy about the
> handling of dependencies for stats objects.  I do not think that cloning
> RemoveStatistics as RemoveStatisticsExt is sane at all.  The former is
> meant to deal with cleaning up pg_statistic rows that we know will be
> there, so there's no real need to expend pg_depend overhead to track them.
> For objects that are only loosely connected, the right thing is to use
> the dependency system; in particular, this design has no real chance of
> working well with cross-table stats.  Also, it's really silly to have
> *both* this hard-wired mechanism and a pg_depend entry; the latter is
> surely redundant if you have the former.  IMO we should revert
> RemoveStatisticsExt and instead handle things by making stats objects
> auto-dependent on the individual column(s) they reference (not the whole
> table).
> 
> I'm also of the opinion that having an AUTO dependency, rather than
> a NORMAL dependency, on the stats object's schema is the wrong semantics.
> There isn't any other case where you can drop a non-empty schema without
> saying CASCADE, and I'm mystified why this case should act that way.

Here are two patches regarding handling of dependencies.  The first one
implements your first suggestion: add a NORMAL dependency on each
column, and do away with RemoveStatisticsExt.  This works well and
should uncontroversial.

If we only do this, then DROP TABLE needs to say CASCADE if there's a
statistics object in the table.  This seems pointless to me, so the
second patch throws in an additional dependency on the table as a whole,
AUTO this time, so that if the table is dropped, the statistics goes
away without requiring cascade.  There is no point in forcing CASCADE
for this case, ISTM.  This works well too, but I admit there might be
resistance to doing it.  OTOH this is how CREATE INDEX works.

(I considered what would happen with a stats object covering multiple
tables.  I think it's reasonable to drop the stats too in that case,
under the rationale that the object is no longer useful.  Not really
sure about this.)

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  40. Re: WITH clause in CREATE STATISTICS

    Tom Lane <tgl@sss.pgh.pa.us> — 2017-05-12T21:20:04Z

    Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> Tom Lane wrote:
>> Although I've not done anything about it here, I'm not happy about the
>> handling of dependencies for stats objects.

> Here are two patches regarding handling of dependencies.

Oh, sorry --- I already pushed something about this.

> The first one
> implements your first suggestion: add a NORMAL dependency on each
> column, and do away with RemoveStatisticsExt.  This works well and
> should uncontroversial.

No, I wanted an AUTO dependency there, for exactly the reason you
mention:

> If we only do this, then DROP TABLE needs to say CASCADE if there's a
> statistics object in the table.

I don't think we really want that, do we?  A stats object can't be of
any value if the underlying table is gone.  Perhaps that calculus
would change for cross-table stats but I don't see why.

> This seems pointless to me, so the
> second patch throws in an additional dependency on the table as a whole,
> AUTO this time, so that if the table is dropped, the statistics goes
> away without requiring cascade.  There is no point in forcing CASCADE
> for this case, ISTM.  This works well too, but I admit there might be
> resistance to doing it.  OTOH this is how CREATE INDEX works.

Uh, no it isn't.  Indexes have auto dependencies on the individual
columns they index, and *not* on the table as a whole.

			regards, tom lane

  41. Re: WITH clause in CREATE STATISTICS

    Alvaro Herrera <alvherre@2ndquadrant.com> — 2017-05-12T21:30:10Z

    Tom Lane wrote:
> Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> > Tom Lane wrote:
> >> Although I've not done anything about it here, I'm not happy about the
> >> handling of dependencies for stats objects.
> 
> > Here are two patches regarding handling of dependencies.
> 
> Oh, sorry --- I already pushed something about this.

That's fine, thanks.

I'm glad that this thread got you interested in look over the extended
statistics stuff.  Thanks for the input.

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  42. Re: WITH clause in CREATE STATISTICS

    Tom Lane <tgl@sss.pgh.pa.us> — 2017-05-12T21:35:56Z

    Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> Tom Lane wrote:
>> Oh, sorry --- I already pushed something about this.

> That's fine, thanks.

Oh, I see your patch also fixes missing code in getObjectDescription().
We need that.  Is there a decent way to get the compiler to warn about
that oversight?

			regards, tom lane

  43. Re: WITH clause in CREATE STATISTICS

    Alvaro Herrera <alvherre@2ndquadrant.com> — 2017-05-12T21:44:26Z

    Tom Lane wrote:
> Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> > Tom Lane wrote:
> >> Oh, sorry --- I already pushed something about this.
> 
> > That's fine, thanks.
> 
> Oh, I see your patch also fixes missing code in getObjectDescription().
> We need that.  Is there a decent way to get the compiler to warn about
> that oversight?

We could remove the default clause.  That results in the expected
warning, and no others (i.e. the switch was already complete):

/pgsql/source/master/src/backend/catalog/objectaddress.c:2657:2: warning: enumeration value 'OCLASS_STATISTIC_EXT' not handled in switch [-Wswitch]

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  44. Re: WITH clause in CREATE STATISTICS

    Tom Lane <tgl@sss.pgh.pa.us> — 2017-05-12T21:54:40Z

    Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> Tom Lane wrote:
>> Oh, I see your patch also fixes missing code in getObjectDescription().
>> We need that.  Is there a decent way to get the compiler to warn about
>> that oversight?

> We could remove the default clause.  That results in the expected
> warning, and no others (i.e. the switch was already complete):

> /pgsql/source/master/src/backend/catalog/objectaddress.c:2657:2: warning: enumeration value 'OCLASS_STATISTIC_EXT' not handled in switch [-Wswitch]

Hm.  That would behave less than desirably if getObjectClass() could
return a value that wasn't a member of the enum, but it's hard to
credit that happening.  I guess I'd vote for removing the default:
case from all of the places that have "switch (getObjectClass(object))",
as forgetting to add an entry seems like a much larger hazard.

			regards, tom lane

  45. Re: WITH clause in CREATE STATISTICS

    Alvaro Herrera <alvherre@2ndquadrant.com> — 2017-05-12T22:10:10Z

    Tom Lane wrote:
> Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> > Tom Lane wrote:
> >> Oh, I see your patch also fixes missing code in getObjectDescription().
> >> We need that.  Is there a decent way to get the compiler to warn about
> >> that oversight?
> 
> > We could remove the default clause.  That results in the expected
> > warning, and no others (i.e. the switch was already complete):
> 
> > /pgsql/source/master/src/backend/catalog/objectaddress.c:2657:2: warning: enumeration value 'OCLASS_STATISTIC_EXT' not handled in switch [-Wswitch]
> 
> Hm.  That would behave less than desirably if getObjectClass() could
> return a value that wasn't a member of the enum, but it's hard to
> credit that happening.  I guess I'd vote for removing the default:
> case from all of the places that have "switch (getObjectClass(object))",
> as forgetting to add an entry seems like a much larger hazard.

Ignoring the one in alter.c's AlterObjectNamespace_oid, which only
handles a small subset of the enum values, that gives the following
warnings:

/pgsql/source/master/src/backend/catalog/dependency.c: In function 'doDeletion':
/pgsql/source/master/src/backend/catalog/dependency.c:1106:2: warning: enumeration value 'OCLASS_ROLE' not handled in switch [-Wswitch]
  switch (getObjectClass(object))
  ^
/pgsql/source/master/src/backend/catalog/dependency.c:1106:2: warning: enumeration value 'OCLASS_DATABASE' not handled in switch [-Wswitch]
/pgsql/source/master/src/backend/catalog/dependency.c:1106:2: warning: enumeration value 'OCLASS_TBLSPACE' not handled in switch [-Wswitch]

/pgsql/source/master/src/backend/catalog/dependency.c:1106:2: warning: enumeration value 'OCLASS_SUBSCRIPTION' not handled in switch [-Wswitch]

/pgsql/source/master/src/backend/commands/tablecmds.c: In function 'ATExecAlterColumnType':
/pgsql/source/master/src/backend/commands/tablecmds.c:9112:3: warning: enumeration value 'OCLASS_AM' not handled in switch [-Wswitch]
   switch (getObjectClass(&foundObject))
   ^
/pgsql/source/master/src/backend/commands/tablecmds.c:9112:3: warning: enumeration value 'OCLASS_STATISTIC_EXT' not handled in switch [-Wswitch]
/pgsql/source/master/src/backend/commands/tablecmds.c:9112:3: warning: enumeration value 'OCLASS_EVENT_TRIGGER' not handled in switch [-Wswitch]
/pgsql/source/master/src/backend/commands/tablecmds.c:9112:3: warning: enumeration value 'OCLASS_PUBLICATION' not handled in switch [-Wswitch]
/pgsql/source/master/src/backend/commands/tablecmds.c:9112:3: warning: enumeration value 'OCLASS_PUBLICATION_REL' not handled in switch [-Wswitch]
/pgsql/source/master/src/backend/commands/tablecmds.c:9112:3: warning: enumeration value 'OCLASS_SUBSCRIPTION' not handled in switch [-Wswitch]
/pgsql/source/master/src/backend/commands/tablecmds.c:9112:3: warning: enumeration value 'OCLASS_TRANSFORM' not handled in switch [-Wswitch]

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  46. Re: WITH clause in CREATE STATISTICS

    Tom Lane <tgl@sss.pgh.pa.us> — 2017-05-13T00:03:04Z

    Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> Tom Lane wrote:
>> Hm.  That would behave less than desirably if getObjectClass() could
>> return a value that wasn't a member of the enum, but it's hard to
>> credit that happening.  I guess I'd vote for removing the default:
>> case from all of the places that have "switch (getObjectClass(object))",
>> as forgetting to add an entry seems like a much larger hazard.

> Ignoring the one in alter.c's AlterObjectNamespace_oid, which only
> handles a small subset of the enum values, that gives the following
> warnings:

> /pgsql/source/master/src/backend/catalog/dependency.c: In function 'doDeletion':
> /pgsql/source/master/src/backend/catalog/dependency.c:1106:2: warning: enumeration value 'OCLASS_ROLE' not handled in switch [-Wswitch]
>   switch (getObjectClass(object))
>   ^
> /pgsql/source/master/src/backend/catalog/dependency.c:1106:2: warning: enumeration value 'OCLASS_DATABASE' not handled in switch [-Wswitch]
> /pgsql/source/master/src/backend/catalog/dependency.c:1106:2: warning: enumeration value 'OCLASS_TBLSPACE' not handled in switch [-Wswitch]

> /pgsql/source/master/src/backend/catalog/dependency.c:1106:2: warning: enumeration value 'OCLASS_SUBSCRIPTION' not handled in switch [-Wswitch]

Hm.  I think it's intentional that shared objects are not handled there,
but I wonder whether the lack of SUBSCRIPTION represents a bug.
Anyway I think it would be fine to add those to the switch as explicit
error cases.


> /pgsql/source/master/src/backend/commands/tablecmds.c: In function 'ATExecAlterColumnType':
> /pgsql/source/master/src/backend/commands/tablecmds.c:9112:3: warning: enumeration value 'OCLASS_AM' not handled in switch [-Wswitch]
>    switch (getObjectClass(&foundObject))
>    ^
> /pgsql/source/master/src/backend/commands/tablecmds.c:9112:3: warning: enumeration value 'OCLASS_STATISTIC_EXT' not handled in switch [-Wswitch]
> /pgsql/source/master/src/backend/commands/tablecmds.c:9112:3: warning: enumeration value 'OCLASS_EVENT_TRIGGER' not handled in switch [-Wswitch]
> /pgsql/source/master/src/backend/commands/tablecmds.c:9112:3: warning: enumeration value 'OCLASS_PUBLICATION' not handled in switch [-Wswitch]
> /pgsql/source/master/src/backend/commands/tablecmds.c:9112:3: warning: enumeration value 'OCLASS_PUBLICATION_REL' not handled in switch [-Wswitch]
> /pgsql/source/master/src/backend/commands/tablecmds.c:9112:3: warning: enumeration value 'OCLASS_SUBSCRIPTION' not handled in switch [-Wswitch]
> /pgsql/source/master/src/backend/commands/tablecmds.c:9112:3: warning: enumeration value 'OCLASS_TRANSFORM' not handled in switch [-Wswitch]

All of those ought to go into the category that prints
"unexpected object depending on column", I think; certainly
"unrecognized object class" is not a better report, and the fact that
we've evidently not touched this switch in the last few additions of
object types provides fodder for the idea that the default: is unhelpful.
(Not to mention that not having OCLASS_STATISTIC_EXT here is an outright
bug as of today...)  So losing the default: case here seems good to me.

Alternatively, we could drop the explicit listing of oclasses we're not
expecting to see, and let the default: case be "unexpected object
depending on column".  Treating the default separately is only of value if
we'd expect getObjectDescription to fail, but there's no good reason for
this code to be passing judgment on whether that might happen.

What do we want this to do about statistics objects?  We could either
throw a feature-not-implemented error or try to update the stats
object for the new column type.

			regards, tom lane