Re: POC: GROUP BY optimization

jian he <jian.universality@gmail.com>

From: jian he <jian.universality@gmail.com>

To: Alexander Korotkov <aekorotkov@gmail.com>

Cc: Andrei Lepikhov <a.lepikhov@postgrespro.ru>, Tom Lane <tgl@sss.pgh.pa.us>, Richard Guo <guofenglinux@gmail.com>, Pavel Borisov <pashkin.elfe@gmail.com>, vignesh C <vignesh21@gmail.com>, PostgreSQL Developers <pgsql-hackers@lists.postgresql.org>, Tomas Vondra <tomas.vondra@enterprisedb.com>, Teodor Sigaev <teodor@sigaev.ru>, David Rowley <dgrowleyml@gmail.com>, "a.rybakina" <a.rybakina@postgrespro.ru>

Date: 2024-04-19T10:44:44Z

Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

Restore preprocess_groupclause()
- 505c008ca37c 17.0 landed
Rename PathKeyInfo to GroupByOrdering
- 0c1af2c35c7b 17.0 landed
Add invariants check to get_useful_group_keys_orderings()
- 91143c03d4ca 17.0 landed
Fix asymmetry in setting EquivalenceClass.ec_sortref
- 199012a3d844 17.0 landed
Multiple revisions to the GROUP BY reordering tests
- 874d817baa16 17.0 landed
Get rid of pg_class usage in SJE regression tests
- e1b7fde418f2 17.0 landed
Rename index "abc" in aggregates.sql
- b91f91870828 17.0 landed
Explore alternative orderings of group-by pathkeys during optimization.
- 0452b461bc40 17.0 landed
Generalize the common code of adding sort before processing of grouping
- 7ab80ac1caf9 17.0 landed
Fix out-dated comment in preprocess_groupclause()
- f6c70b81802a 15.0 landed
- 78a9af1a2764 16.0 landed
Force parallelism in partition_aggregate
- 2fe6b2a806f2 16.0 landed
- 01474f56981a 15.0 landed
Optimize order of GROUP BY keys
- db0d67db2401 15.0 landed

On Thu, Apr 18, 2024 at 6:58 PM Alexander Korotkov <aekorotkov@gmail.com> wrote:
>
> Thank you for the fixes you've proposed.  I didn't look much into
> details yet, but I think the main concern Tom expressed in [1] is
> whether the feature is reasonable at all.  I think at this stage the
> most important thing is to come up with convincing examples showing
> how huge performance benefits it could cause.  I will return to this
> later today and will try to provide some convincing examples.
>

hi.
I found a case where it improved performance.

+-- GROUP BY optimization by reorder columns
+CREATE TABLE btg AS SELECT
+ i % 100 AS x,
+ i % 100 AS y,
+ 'abc' || i % 10 AS z,
+ i AS w
+FROM generate_series(1,10000) AS i;
+CREATE INDEX abc ON btg(x,y);
+ANALYZE btg;
+
I change
+FROM generate_series(1,10000) AS i;
to
+ FROM generate_series(1, 1e6) AS i;

Then I found out about these 2 queries performance improved a lot.
A: explain(analyze) SELECT count(*) FROM btg GROUP BY w, x, y, z ORDER
BY y, x \watch i=0.1 c=10
B: explain(analyze) SELECT count(*) FROM btg GROUP BY w, x, z, y ORDER
BY y, x, z, w \watch i=0.1 c=10

set (enable_seqscan,enable_hashagg) from on to off:
queryA execution time from 1533.013 ms to 533.430 ms
queryB execution time from 1996.817 ms to 497.020 ms