Thread

Commits

  1. Consider startup cost as a figure of merit for partial paths.

  2. Fix add_partial_path interaction with disabled_nodes

  3. Support parallel joins, and make related improvements.

  1. Consider low startup cost in add_partial_path

    James Coleman <jtc331@gmail.com> — 2019-09-27T18:24:10Z

    Over in the incremental sort patch discussion we found [1] a case
    where a higher cost plan ends up being chosen because a low startup
    cost partial path is ignored in favor of a lower total cost partial
    path and a limit is a applied on top of that which would normal favor
    the lower startup cost plan.
    
    45be99f8cd5d606086e0a458c9c72910ba8a613d originally added
    `add_partial_path` with the comment:
    
    > Neither do we need to consider startup costs:
    > parallelism is only used for plans that will be run to completion.
    > Therefore, this routine is much simpler than add_path: it needs to
    > consider only pathkeys and total cost.
    
    I'm not entirely sure if that is still true or not--I can't easily
    come up with a scenario in which it's not, but I also can't come up
    with an inherent reason why such a scenario cannot exist.
    
    We could just continue to include this change as part of the
    incremental sort patch itself, but it seemed worth it to me to break
    it out for some more targeted discussion, and also include Robert as
    the initial author of add_partial_path in the hopes that maybe we
    could retrieve some almost 4-year-old memories on why this was
    inherently true then, and maybe that would shed some light on whether
    it's still inherently true.
    
    I've attached a patch (by Tomas Vondra, also cc'd) to consider startup
    cost in add_partial_path, but should we apply the patch we'll also
    likely need to apply the same kind of change to
    add_partial_path_precheck.
    
    James Coleman
    
    [1]: https://www.postgresql.org/message-id/20190720132244.3vgg2uynfpxh3me5%40development
    
  2. Re: Consider low startup cost in add_partial_path

    Robert Haas <robertmhaas@gmail.com> — 2019-09-28T04:16:05Z

    On Fri, Sep 27, 2019 at 2:24 PM James Coleman <jtc331@gmail.com> wrote:
    > Over in the incremental sort patch discussion we found [1] a case
    > where a higher cost plan ends up being chosen because a low startup
    > cost partial path is ignored in favor of a lower total cost partial
    > path and a limit is a applied on top of that which would normal favor
    > the lower startup cost plan.
    >
    > 45be99f8cd5d606086e0a458c9c72910ba8a613d originally added
    > `add_partial_path` with the comment:
    >
    > > Neither do we need to consider startup costs:
    > > parallelism is only used for plans that will be run to completion.
    > > Therefore, this routine is much simpler than add_path: it needs to
    > > consider only pathkeys and total cost.
    >
    > I'm not entirely sure if that is still true or not--I can't easily
    > come up with a scenario in which it's not, but I also can't come up
    > with an inherent reason why such a scenario cannot exist.
    
    I think I just didn't think carefully about the Limit case.
    
    -- 
    Robert Haas
    EnterpriseDB: http://www.enterprisedb.com
    The Enterprise PostgreSQL Company
    
    
    
    
  3. Re: Consider low startup cost in add_partial_path

    Tomas Vondra <tomas.vondra@2ndquadrant.com> — 2019-09-28T22:37:41Z

    On Sat, Sep 28, 2019 at 12:16:05AM -0400, Robert Haas wrote:
    >On Fri, Sep 27, 2019 at 2:24 PM James Coleman <jtc331@gmail.com> wrote:
    >> Over in the incremental sort patch discussion we found [1] a case
    >> where a higher cost plan ends up being chosen because a low startup
    >> cost partial path is ignored in favor of a lower total cost partial
    >> path and a limit is a applied on top of that which would normal favor
    >> the lower startup cost plan.
    >>
    >> 45be99f8cd5d606086e0a458c9c72910ba8a613d originally added
    >> `add_partial_path` with the comment:
    >>
    >> > Neither do we need to consider startup costs:
    >> > parallelism is only used for plans that will be run to completion.
    >> > Therefore, this routine is much simpler than add_path: it needs to
    >> > consider only pathkeys and total cost.
    >>
    >> I'm not entirely sure if that is still true or not--I can't easily
    >> come up with a scenario in which it's not, but I also can't come up
    >> with an inherent reason why such a scenario cannot exist.
    >
    >I think I just didn't think carefully about the Limit case.
    >
    
    Thanks! In that case I suggest we treat it as a separate patch/fix,
    independent of the incremental sort patch. I don't want to bury it in
    that patch series, it's already pretty large.
    
    regards
    
    -- 
    Tomas Vondra                  http://www.2ndQuadrant.com
    PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
    
    
    
    
  4. Re: Consider low startup cost in add_partial_path

    James Coleman <jtc331@gmail.com> — 2019-09-28T23:21:33Z

    On Saturday, September 28, 2019, Tomas Vondra <tomas.vondra@2ndquadrant.com>
    wrote:
    
    > On Sat, Sep 28, 2019 at 12:16:05AM -0400, Robert Haas wrote:
    >
    >> On Fri, Sep 27, 2019 at 2:24 PM James Coleman <jtc331@gmail.com> wrote:
    >>
    >>> Over in the incremental sort patch discussion we found [1] a case
    >>> where a higher cost plan ends up being chosen because a low startup
    >>> cost partial path is ignored in favor of a lower total cost partial
    >>> path and a limit is a applied on top of that which would normal favor
    >>> the lower startup cost plan.
    >>>
    >>> 45be99f8cd5d606086e0a458c9c72910ba8a613d originally added
    >>> `add_partial_path` with the comment:
    >>>
    >>> > Neither do we need to consider startup costs:
    >>> > parallelism is only used for plans that will be run to completion.
    >>> > Therefore, this routine is much simpler than add_path: it needs to
    >>> > consider only pathkeys and total cost.
    >>>
    >>> I'm not entirely sure if that is still true or not--I can't easily
    >>> come up with a scenario in which it's not, but I also can't come up
    >>> with an inherent reason why such a scenario cannot exist.
    >>>
    >>
    >> I think I just didn't think carefully about the Limit case.
    >>
    >>
    > Thanks! In that case I suggest we treat it as a separate patch/fix,
    > independent of the incremental sort patch. I don't want to bury it in
    > that patch series, it's already pretty large.
    >
    
    Now the trick is to figure out a way to demonstrate it in test :)
    
    Basically we need:
    Path A: Can short circuit with LIMIT but has high total cost
    Path B: Can’t short circuit with LIMIT but has lower total cost
    
    (Both must be parallel aware of course.)
    
    Maybe ordering in B can be a sort node and A can be an index scan (perhaps
    with very high random page cost?) and force choosing a parallel plan?
    
    I’m trying to describe this to jog my thoughts (not in front of my laptop
    right now so can’t try it out).
    
    Any other ideas?
    
    James
    
  5. Re: Consider low startup cost in add_partial_path

    James Coleman <jtc331@gmail.com> — 2019-10-02T14:22:17Z

    On Sat, Sep 28, 2019 at 7:21 PM James Coleman <jtc331@gmail.com> wrote:
    > Now the trick is to figure out a way to demonstrate it in test :)
    >
    > Basically we need:
    > Path A: Can short circuit with LIMIT but has high total cost
    > Path B: Can’t short circuit with LIMIT but has lower total cost
    >
    > (Both must be parallel aware of course.)
    
    I'm adding one requirement, or clarifying it anyway: the above paths
    must be partial paths, and can't just apply at the top level of the
    parallel part of the plan. I.e., the lower startup cost has to matter
    at a subtree of the parallel portion of the plan.
    
    > Maybe ordering in B can be a sort node and A can be an index scan (perhaps with very high random page cost?) and force choosing a parallel plan?
    >
    > I’m trying to describe this to jog my thoughts (not in front of my laptop right now so can’t try it out).
    >
    > Any other ideas?
    
    I've been playing with this a good bit, and I'm struggling to come up
    with a test case. Because the issue only manifests in a subtree of the
    parallel portion of the plan, a scan on a single relation won't do.
    Merge join seems like a good area to look at because it requires
    ordering, and that ordering can be either the result of an index scan
    (short-circuit-able) or an explicit sort (not short-circuit-able). But
    I've been unable to make that result in any different plans with
    either 2 or 3 relations joined together, ordered, and a limit applied.
    
    In all cases I've been starting with:
    
    set enable_hashjoin = off;
    set enable_nestloop = off;
    set max_parallel_workers_per_gather = 4;
    set min_parallel_index_scan_size = 0;
    set min_parallel_table_scan_size = 0;
    set parallel_setup_cost = 0;
    set parallel_tuple_cost = 0;
    
    I've also tried various combinations of random_page_cost,
    cpu_index_tuple_cost, cpu_tuple_cost.
    
    Interestingly I've noticed plans joining two relations that look like:
    
     Limit
       ->  Merge Join
             Merge Cond: (t1.pk = t2.pk)
             ->  Gather Merge
                   Workers Planned: 4
                   ->  Parallel Index Scan using t_pkey on t t1
             ->  Gather Merge
                   Workers Planned: 4
                   ->  Parallel Index Scan using t_pkey on t t2
    
    Where I would have expected a Gather Merge above a parallelized merge
    join. Is that reasonable to expect?
    
    If there doesn't seem to be an obvious way to reproduce the issue
    currently, but we know we have a reproduction example along with
    incremental sort, what is the path forward for this? Is it reasonable
    to try to commit it anyway knowing that it's a "correct" change and
    been demonstrated elsewhere?
    
    James
    
    
    
    
  6. Re: Consider low startup cost in add_partial_path

    Robert Haas <robertmhaas@gmail.com> — 2019-10-04T12:36:44Z

    On Wed, Oct 2, 2019 at 10:22 AM James Coleman <jtc331@gmail.com> wrote:
    > In all cases I've been starting with:
    >
    > set enable_hashjoin = off;
    > set enable_nestloop = off;
    > set max_parallel_workers_per_gather = 4;
    > set min_parallel_index_scan_size = 0;
    > set min_parallel_table_scan_size = 0;
    > set parallel_setup_cost = 0;
    > set parallel_tuple_cost = 0;
    >
    > I've also tried various combinations of random_page_cost,
    > cpu_index_tuple_cost, cpu_tuple_cost.
    >
    > Interestingly I've noticed plans joining two relations that look like:
    >
    >  Limit
    >    ->  Merge Join
    >          Merge Cond: (t1.pk = t2.pk)
    >          ->  Gather Merge
    >                Workers Planned: 4
    >                ->  Parallel Index Scan using t_pkey on t t1
    >          ->  Gather Merge
    >                Workers Planned: 4
    >                ->  Parallel Index Scan using t_pkey on t t2
    >
    > Where I would have expected a Gather Merge above a parallelized merge
    > join. Is that reasonable to expect?
    
    Well, you told the planner that parallel_setup_cost = 0, so starting
    workers is free. And you told the planner that parallel_tuple_cost =
    0, so shipping tuples from the worker to the leader is also free. So
    it is unclear why it should prefer a single Gather Merge over two
    Gather Merges: after all, the Gather Merge is free!
    
    If you use give those things some positive cost, even if it's smaller
    than the default, you'll probably get a saner-looking plan choice.
    
    -- 
    Robert Haas
    EnterpriseDB: http://www.enterprisedb.com
    The Enterprise PostgreSQL Company
    
    
    
    
  7. Re: Consider low startup cost in add_partial_path

    James Coleman <jtc331@gmail.com> — 2019-10-24T18:38:33Z

    On Fri, Oct 4, 2019 at 8:36 AM Robert Haas <robertmhaas@gmail.com> wrote:
    >
    > On Wed, Oct 2, 2019 at 10:22 AM James Coleman <jtc331@gmail.com> wrote:
    > > In all cases I've been starting with:
    > >
    > > set enable_hashjoin = off;
    > > set enable_nestloop = off;
    > > set max_parallel_workers_per_gather = 4;
    > > set min_parallel_index_scan_size = 0;
    > > set min_parallel_table_scan_size = 0;
    > > set parallel_setup_cost = 0;
    > > set parallel_tuple_cost = 0;
    > >
    > > I've also tried various combinations of random_page_cost,
    > > cpu_index_tuple_cost, cpu_tuple_cost.
    > >
    > > Interestingly I've noticed plans joining two relations that look like:
    > >
    > >  Limit
    > >    ->  Merge Join
    > >          Merge Cond: (t1.pk = t2.pk)
    > >          ->  Gather Merge
    > >                Workers Planned: 4
    > >                ->  Parallel Index Scan using t_pkey on t t1
    > >          ->  Gather Merge
    > >                Workers Planned: 4
    > >                ->  Parallel Index Scan using t_pkey on t t2
    > >
    > > Where I would have expected a Gather Merge above a parallelized merge
    > > join. Is that reasonable to expect?
    >
    > Well, you told the planner that parallel_setup_cost = 0, so starting
    > workers is free. And you told the planner that parallel_tuple_cost =
    > 0, so shipping tuples from the worker to the leader is also free. So
    > it is unclear why it should prefer a single Gather Merge over two
    > Gather Merges: after all, the Gather Merge is free!
    >
    > If you use give those things some positive cost, even if it's smaller
    > than the default, you'll probably get a saner-looking plan choice.
    
    That makes sense.
    
    Right now I currently see trying to get this a separate test feels a
    bit like a distraction.
    
    Given there doesn't seem to be an obvious way to reproduce the issue
    currently, but we know we have a reproduction example along with
    incremental sort, what is the path forward for this? Is it reasonable
    to try to commit it anyway knowing that it's a "correct" change and
    been demonstrated elsewhere?
    
    James
    
    
    
    
  8. Re: Consider low startup cost in add_partial_path

    Tomas Vondra <tomas.vondra@2ndquadrant.com> — 2020-04-05T14:14:49Z

    Hi,
    
    For the record, here is the relevant part of the Incremental Sort patch
    series, updating add_partial_path and add_partial_path_precheck to also
    consider startup cost.
    
    The changes in the first two patches are pretty straight-forward, plus
    there's a proposed optimization in the precheck function to only run
    compare_pathkeys if entirely necessary. I'm currently evaluating those
    changes and I'll post the results to the incremental sort thread.
    
    
    regards
    
    -- 
    Tomas Vondra                  http://www.2ndQuadrant.com
    PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
    
  9. Re: Consider low startup cost in add_partial_path

    Robert Haas <robertmhaas@gmail.com> — 2026-02-06T21:06:04Z

    [ reviving an old thread ]
    
    On Sun, Apr 5, 2020 at 10:14 AM Tomas Vondra
    <tomas.vondra@2ndquadrant.com> wrote:
    > For the record, here is the relevant part of the Incremental Sort patch
    > series, updating add_partial_path and add_partial_path_precheck to also
    > consider startup cost.
    >
    > The changes in the first two patches are pretty straight-forward, plus
    > there's a proposed optimization in the precheck function to only run
    > compare_pathkeys if entirely necessary. I'm currently evaluating those
    > changes and I'll post the results to the incremental sort thread.
    
    I rediscovered this problem while testing pg_plan_advice, and I think
    we should commit something to fix it. Consider the following query
    from the regression tests:
    
    explain (costs off) select a,b,sum(c) from t group by 1,2 order by
    1,2,3 limit 1;
    
    With the settings in use at the time the plan is executed, the plan
    has a cost of 68.11. If you rerun the same query with
    enable_seqscan=off, the estimated cost drops to 65.28. This is
    obviously a bit of a surprising outcome, since we should be
    considering a subset of the original possibilities and should
    therefore not be able to achieve a better outcome. What happens is
    that the planner is deciding between Finalize GroupAggregate => Gather
    Merge => Partial GroupAggregate => Incremental Sort => Parallel Index
    Scan and GroupAggregate => Gather Merge => Incremental Sort =>
    Parallel Index Scan. When we're computing partial paths for
    partially_grouped_rel, we normally construct two: one that does
    Incremental Sort => Parallel Index Scan and another that does Sort =>
    Parallel Seq Scan. Because add_partial_path() doesn't consider startup
    costs, the second path causes the first one to be discarded. Once we
    add the Gather Merge node, we do consider startup costs (since it's
    not a partial path any more) and now the high startup cost of Sort =>
    Parallel Seq Scan makes it lose out. Instead, we abandon the idea of
    two-step aggregation altogether and pick a plan that involves one step
    aggregation. To me, this is a pretty clear illustration that the
    comments that I wrote here are just wrong, the reasoning is wrong, and
    what we're doing doesn't make a whole lot of sense. We're throwing
    away paths that would turn out to be the winning option if we kept
    them on the basis of a specious argument that startup cost doesn't
    matter.
    
    So, I started working on Tomas's patches from April 5, 2020. They
    don't apply any more, so I had to rebase them. That resulted in a
    bunch of regression test failures, and some of those plan changes
    didn't make a whole lot of sense. At that point, I realized that when
    I added the disabled_nodes field to the Path structure, I failed to
    properly adjust the logic in add_partial_path() for the fact that we
    now keep path lists sorted first by number of disabled nodes and then
    by total cost. After fixing that, there were still two tests failing.
    One was the query above, where the plan improves. In the other case, a
    Hash Left Join became a Hash Right Join, and the reason that happened
    is because the two paths had exactly equivalent total costs, but the
    Hash Right Join has a cheaper startup cost. Arguably that's an
    improvement, too, but it defeated the purpose of the test case, so
    that will need to be adjusted. On further study, I found yet another
    bug, which is that add_partial_path_precheck() *also* wasn't properly
    adjusted to deal with the disabled_nodes field. In other words,
    everything that is wrong here is 100% my fault: some of it I did wrong
    when I added parallel query, and the rest of it I did wrong when I
    added disabled_nodes.
    
    So attached are a couple of patches to try to improve things. 0001
    fixes the lack of a disabled_nodes test in add_partial_path. 0002
    rewrites teaches add_partial_path about startup cost, and also
    rewrites add_partial_path_precheck to do the cost comparisons in the
    same manner as we do in compare_path_costs_fuzzily. This fixes both
    the failure of that function to consider disabled_nodes, and also the
    failure of that function to consider startup_cost. These are formally
    two separate bugs, but I do not really relish the idea of fixing them
    separately, because it means rewriting add_partial_path_precheck()
    twice, and I think the ending state will be the same as what I have
    here, and the intermediate state will be of no use to anyone and
    possibly buggy as well. The only argument I can see for trying to
    separate the two fixes to add_partial_path_precheck() is of one of the
    following two things is true:
    
    (1) We want to back-patch the disable_cost portion of the fix. This
    has the possibility of destabilizing plans in released branches, so I
    assume this is not going to be very appealing.
    
    (2) We don't actually want the changes to consider startup cost. In
    that case, this will definitely need to be changed. There's certainly
    an argument that considering the startup cost increases the number of
    paths we retain and therefore increases planning cost, but I feel like
    that argument should lose out to the counter-argument that what we're
    doing now is based on the fiction that the startup cost *can't* matter
    in the case of a partial path, which we now know is not true: James
    Coleman found that originally when working on Incremental Sort, and we
    now have a regression test query that demonstrates it. If we're going
    to skip considering startup cost in certain cases to reduce path
    explosion, it should be based on a principled design choice, which
    doesn't appear to be the case here. Rather, it appears to be based on
    vintage-2016 Robert writing some stuff that wasn't true, and then we
    kinda just rolled with it.
    
    --
    Robert Haas
    EDB: http://www.enterprisedb.com
    
  10. Re: Consider low startup cost in add_partial_path

    Richard Guo <guofenglinux@gmail.com> — 2026-02-17T07:44:39Z

    On Sat, Feb 7, 2026 at 6:06 AM Robert Haas <robertmhaas@gmail.com> wrote:
    > I rediscovered this problem while testing pg_plan_advice, and I think
    > we should commit something to fix it.
    
    Yeah, I agree that we should consider startup cost for partial paths,
    given that cost_gather (and cost_gather_merge) accounts for the
    subpath's startup_cost when calculating the cost of the Gather path.
    
    The changes LGTM.  I noticed one comment that may need to be updated
    accordingly:
    
     *    As in add_path, the partial_pathlist is kept sorted with the cheapest
     *    total path in front. ...
    
    This is not true anymore.  The partial_pathlist is now sorted by
    disabled_nodes and then by cheapest total cost.
    
    - Richard
    
    
    
    
  11. Re: Consider low startup cost in add_partial_path

    Robert Haas <robertmhaas@gmail.com> — 2026-02-19T19:29:52Z

    On Tue, Feb 17, 2026 at 2:44 AM Richard Guo <guofenglinux@gmail.com> wrote:
    > Yeah, I agree that we should consider startup cost for partial paths,
    > given that cost_gather (and cost_gather_merge) accounts for the
    > subpath's startup_cost when calculating the cost of the Gather path.
    >
    > The changes LGTM.  I noticed one comment that may need to be updated
    > accordingly:
    >
    >  *    As in add_path, the partial_pathlist is kept sorted with the cheapest
    >  *    total path in front. ...
    >
    > This is not true anymore.  The partial_pathlist is now sorted by
    > disabled_nodes and then by cheapest total cost.
    
    Thanks. I have committed 0001 after adjusting that comment.
    
    Here's the remaining patch once again. No comments on this one?
    
    -- 
    Robert Haas
    EDB: http://www.enterprisedb.com
    
  12. Re: Consider low startup cost in add_partial_path

    Richard Guo <guofenglinux@gmail.com> — 2026-02-25T06:35:05Z

    On Fri, Feb 20, 2026 at 4:30 AM Robert Haas <robertmhaas@gmail.com> wrote:
    > Thanks. I have committed 0001 after adjusting that comment.
    >
    > Here's the remaining patch once again. No comments on this one?
    
    (Sorry I somehow lost track of this thread.)
    
    I noticed that in this patch, add_partial_path_precheck duplicates the
    entire logic of compare_path_costs_fuzzily.  I wonder if it would be
    better to avoid this code duplication.
    
    I'm thinking that maybe we could extract the core math of
    compare_path_costs_fuzzily into an internal helper function that
    operates on cost values (and disabled_nodes) rather than on Path
    structures.  We could then call this helper directly from within
    add_partial_path_precheck.
    
    That said, I'm not quite sure if that refactoring is worth the churn,
    or if there is a better way to do that.
    
    Otherwise, the patch looks good to me.
    
    - Richard
    
    
    
    
  13. Re: Consider low startup cost in add_partial_path

    Robert Haas <robertmhaas@gmail.com> — 2026-03-04T22:04:23Z

    On Wed, Feb 25, 2026 at 1:35 AM Richard Guo <guofenglinux@gmail.com> wrote:
    > I noticed that in this patch, add_partial_path_precheck duplicates the
    > entire logic of compare_path_costs_fuzzily.  I wonder if it would be
    > better to avoid this code duplication.
    >
    > I'm thinking that maybe we could extract the core math of
    > compare_path_costs_fuzzily into an internal helper function that
    > operates on cost values (and disabled_nodes) rather than on Path
    > structures.  We could then call this helper directly from within
    > add_partial_path_precheck.
    
    Maybe, but it would be a ten-parameter function.
    
    > Otherwise, the patch looks good to me.
    
    Turns out that version had a few problems:
    
    1. It didn't handle the case where costcmp was COSTS_BETTER{1|2} but
    keyscmp was PATHKEYS_EQUAL correctly. It had logic like this: if
    (costcmp == COSTS_BETTER1) { if (keyscmp == PATHKEYS_BETTER1)
    remove_old = true; }. This says that if the new path is better on cost
    and the new path is also better on pathkeys, throw away the old path.
    That's too weak. It should instead say if (costcmp == COSTS_BETTER1) {
    if (keyscmp != PATHKEYS_BETTER2) remove_old = true; }. That is, if the
    new path is better cost and is EQUAL OR BETTER on pathkeys, throw away
    the old path. As a result, v56 would have kept too many paths.
    
    2. It failed to preserve an equivalent of the (old_path->total_cost >
    new_path->total_cost * 1.0000000001) test. It now tests
    compare_path_costs_fuzzily(new_path, old_path, 1.0000000001) ==
    COSTS_BETTER1).
    
    3. It wasn't pgindent-clean.
    
    Fixing the first of these problems results in one additional
    regression test change. The query is:
    
    explain (costs off) select count(*)
      from tenk1 t1
      join tenk1 t2 on t1.unique1 = t2.unique2
      join tenk1 t3 on t2.unique1 = t3.unique1
       order by count(*);
    
    The sides of the t1-t2 join get swapped: the driving table is chosen
    to be t2 rather than t1. Without that fix, we keep both
    hash_join(t1,t2) and hash_join(t2,t1), The former has a total cost
    that is lower by a very small amount and therefore ends up first in
    the partial pathlist and so wins. With that fix, we see that
    hash_join(t2,t1) has an essentially equal total cost and a much lower
    startup cost, so we keep only that one. In other words, this change
    seems to be correct.
    
    Here's v57.
    
    -- 
    Robert Haas
    EDB: http://www.enterprisedb.com
    
  14. Re: Consider low startup cost in add_partial_path

    Robert Haas <robertmhaas@gmail.com> — 2026-03-05T18:17:07Z

    On Wed, Mar 4, 2026 at 5:04 PM Robert Haas <robertmhaas@gmail.com> wrote:
    > Here's v57.
    
    I've instrumented the length of the partial pathlist across all
    RelOptInfo objects in a run of the core regression test suite. Here's
    the result:
    
    without the patch: 1->9343 2->5859 3->38
    with the v57 patch: 1->9306 2->5877 3->51 4->4 5->2
    
    That seems very reasonable to me. It means that, across the test
    suite, we retain 0.3% more paths than before, and the percentage of
    partial pathlists of length 1 drops from 61.31% to 61.06%. That is a
    very modest impact, and it has the upside that in some cases we might
    get better plans, which is worth a lot.
    
    In fact, looking through the regression test changes in the patch, the
    changes to incremental_sort.out look like improvements: in the first
    hunk, we now push a PartialAggregate beneath Gather Merge, allowing
    more work to be done in parallel. In the second hunk, we now put the
    cheaper scan on the inner side of a Parallel Hash Join instead of the
    more expensive one. Since the inner side is the build table, the total
    cost of that side becomes startup cost for hash join, so this change
    is theoretically better. The total cost of the Parallel Hash Join
    remains unchanged, while its startup cost drops from 612.32 to 264.32.
    In this instance, that doesn't improve the startup or total cost for
    the plan as a whole, but I believe that's just a consequence of the
    fact that this particular query happens to be putting all of the rows
    into a single group. In other cases, it might help.
    
    The case in join_hash.out is similar: it's testing a join between two
    tables which, at present, are exactly the same, so even the tiniest
    perturbation in the costing formulas can upset the plan and cause it
    not to test the right thing any more. In the patch, I change it so
    that one side of the join has 2 rows and the other side has 3 rows,
    and it then puts the 2-row side on the inner side of the parallel hash
    join for the same reasons discussed in the previous paragraph. This
    should work just as before but be more stable with these changes.
    
    I feel like I've done all the due diligence that I can think of here
    to be sure that this is correct and doesn't cause a path explosion or
    plan regressions, so I plan to go ahead and commit this unless there
    are objections RSN.
    
    Thanks,
    
    -- 
    Robert Haas
    EDB: http://www.enterprisedb.com
    
    
    
    
  15. Re: Consider low startup cost in add_partial_path

    Robert Haas <robertmhaas@gmail.com> — 2026-03-09T13:08:08Z

    On Thu, Mar 5, 2026 at 1:17 PM Robert Haas <robertmhaas@gmail.com> wrote:
    > I feel like I've done all the due diligence that I can think of here
    > to be sure that this is correct and doesn't cause a path explosion or
    > plan regressions, so I plan to go ahead and commit this unless there
    > are objections RSN.
    
    Committed.
    
    -- 
    Robert Haas
    EDB: http://www.enterprisedb.com