Re: POC, WIP: OR-clause support for indexes

Andrei Lepikhov <a.lepikhov@postgrespro.ru>

From: Andrei Lepikhov <a.lepikhov@postgrespro.ru>

To: Alexander Korotkov <aekorotkov@gmail.com>

Cc: jian he <jian.universality@gmail.com>, Alena Rybakina <a.rybakina@postgrespro.ru>, Robert Haas <robertmhaas@gmail.com>, pgsql-hackers@postgresql.org, Peter Geoghegan <pg@bowt.ie>, "Finnerty, Jim" <jfinnert@amazon.com>, Marcos Pegoraro <marcos@f10.com.br>, teodor@sigaev.ru, Tomas Vondra <tomas.vondra@enterprisedb.com>, Peter Eisentraut <peter@eisentraut.org>, Ranier Vilela <ranier.vf@gmail.com>

Date: 2024-03-19T05:16:59Z

Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

Make group_similar_or_args() reorder clause list as little as possible
- 775a06d44c04 18.0 landed
Allow usage of match_orclause_to_indexcol() for joins
- 627d63419e22 18.0 landed
Skip not SOAP-supported indexes while transforming an OR clause into SAOP
- 5bba0546eecb 18.0 landed
Remove the wrong assertion from match_orclause_to_indexcol()
- d4d11940df94 18.0 landed
Teach bitmap path generation about transforming OR-clauses to SAOP's
- ae4569161a27 18.0 landed
Transform OR-clauses to SAOP's during index matching
- d4378c0005e6 18.0 landed
Fix the value of or_to_any_transform_limit in postgresql.conf.sample
- 2af75e117478 17.0 landed
Transform OR clauses to ANY expression
- 72bd38cc99a1 17.0 landed
MergeAttributes code deduplication
- 64444ce071f6 17.0 cited
SEARCH and CYCLE clauses
- 3696a600e229 14.0 cited
Improve estimation of OR clauses using extended statistics.
- 25a9e54d2db3 14.0 cited
Teach btree to handle ScalarArrayOpExpr quals natively.
- 9e8da0f75731 9.2.0 cited
Revise collation derivation method and expression-tree representation.
- b310b6e31ce5 9.1.0 cited
Instead of trying to force WHERE clauses into CNF or DNF normal form,
- 9888192fb773 8.0.0 cited

Attachments

v22-1-0002-Teach-generate_bitmap_or_paths-to-build-BitmapOr-pat.patch (text/plain) patch v22-0002

On 14/3/2024 16:31, Alexander Korotkov wrote:
> On Wed, Mar 13, 2024 at 2:16 PM Andrei Lepikhov 
> As you can see this case is not related to partial indexes.  Just no 
> index selective for the whole query.  However, splitting scan by the OR 
> qual lets use a combination of two selective indexes.
I have rewritten the 0002-* patch according to your concern. A candidate 
and some thoughts are attached.
As I see, we have a problem here: expanding each array and trying to 
apply an element to each index can result in a lengthy planning stage.
Also, an index scan with the SAOP may potentially be more effective than 
with the list of OR clauses.
Originally, the transformation's purpose was to reduce a query's 
complexity and the number of optimization ways to speed up planning and 
(sometimes) execution. Here, we reduce planning complexity only in the 
case of an array size larger than MAX_SAOP_ARRAY_SIZE.
Maybe we can fall back to the previous version of the second patch, 
keeping in mind that someone who wants to get maximum profit from the 
BitmapOr scan of multiple indexes can just disable this optimization, 
enabling deep search of the most optimal scanning way?
As a compromise solution, I propose adding one more option to the 
previous version: if an element doesn't fit any partial index, try to 
cover it with a plain index.
In this case, we still do not guarantee the most optimal fit of elements 
to the set of indexes, but we speed up planning. Does that make sense?

-- 
regards,
Andrei Lepikhov
Postgres Professional