Horton
Menu

Thread

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. pg_plan_advice: Fix another unique-semijoin bug.

  2. pg_plan_advice: Export feedback-related definitions.

  3. pg_plan_advice: Fix a bug when a subquery is pruned away entirely.

  4. pg_plan_advice: Add alternatives test to Makefile.

  5. pg_plan_advice: Handle non-repeatable TABLESAMPLE scans.

  6. pg_stash_advice: Allow stashed advice to be persisted to disk.

  7. Add pg_stash_advice contrib module.

  8. pg_plan_advice: Avoid assertion failure with partitionwise aggregate.

  9. pg_plan_advice: Invent DO_NOT_SCAN(relation_identifier).

  10. Add an alternative_plan_name field to PlannerInfo.

  11. pg_plan_advice: Refactor to invent pgpa_planner_info

  12. Respect disabled_nodes in fix_alternative_subplan.

  13. get_memoize_path: Don't exit quickly when PGS_NESTLOOP_PLAIN is unset.

  14. test_plan_advice: Set TAP test priority 50 in meson.build.

  15. pg_plan_advice: Avoid a crash under GEQO.

  16. Test pg_plan_advice using a new test_plan_advice module.

  17. pg_plan_advice: Always install pg_plan_advice.h, and in the right place

  18. pg_plan_advice: Fix failures to accept identifier keywords.

  19. Add pg_plan_advice contrib module.

  20. Allow extensions to mark an individual index as disabled.

  21. Replace get_relation_info_hook with build_simple_rel_hook.

  22. Store information about Append node consolidation in the final plan.

  23. Store information about elided nodes in the final plan.

  24. Store information about range-table flattening in the final plan.

  25. Pass cursorOptions to planner_setup_hook.

  26. Fix PGS_CONSIDER_NONPARTIAL interaction with Materialize nodes.

  27. Fix mistakes in commit 4020b370f214315b8c10430301898ac21658143f

  28. Allow for plugin control over path generation strategies.

  29. Update some comments for fasthash

  30. Allow passing a pointer to GetNamedDSMSegment()'s init callback.

  31. Don't reset the pathlist of partitioned joinrels.

  32. Treat number of disabled nodes in a path as a separate cost metric.

  1. pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2025-10-30T14:00:05Z

    As I have mentioned on previous threads, for the past while I have
been working on planner extensibility. I've posted some extensibility
patches previously, and got a few of them committed in
Sepember/October with Tom's help, but I think the time has come a
patch which actually makes use of that infrastructure as well as some
further infrastructure that I'm also including in this posting.[1] The
final patch in this series adds a new contrib module called
pg_plan_advice. Very briefly, what pg_plan_advice knows how to do is
process a plan and emits a (potentially long) long text string in a
special-purpose mini-language that describes a bunch of key planning
decisions, such as the join order, selected join methods, types of
scans used to access individual tables, and where and how
partitionwise join and parallelism were used. You can then set
pg_plan_advice.advice to that string to get a future attempt to plan
the same query to reproduce those decisions, or (maybe a better idea)
you can trim that string down to constrain some decisions (e.g. the
join order) but not others (e.g. the join methods), or (if you want to
make your life more exciting) you can edit that advice string and
thereby attempt to coerce the planner into planning the query the way
you think best. There is a README that explains the design philosophy
and thinking in a lot more detail, which is a good place to start if
you're curious, and I implore you to read it if you're interested, and
*especially* if you're thinking of flaming me.

But that doesn't mean that you *shouldn't* flame me. There are a
remarkable number of things that someone could legitimately be unhappy
about in this patch set. First, any form of user control over the
planner tends to be a lightning rod for criticism around here. I've
come to believe that's the wrong way of thinking about it: we can want
to improve the planner over the long term and *also* want to have
tools available to work around problems with it in the short term.
Further, we should not imagine that we're going to solve problems that
have stumped other successful database projects any time in the
foreseeable future; no product will ever get 100% of cases right, and
you don't need to get to very obscure cases before other products
throw up their hands just as we do. But, second, even if you're OK
with the idea of some kind of user control over the planner, you could
very well be of the opinion that what I've implemented here is utter
crap. I've certainly had to make a ton of very opinionated decisions
to get to this point, and you are entitled to hate them. Of course, I
did have *reasons* for making the decisions, so if your operating
theory as to why I did something is that I'm a stupid moron, perhaps
consider an alternative explanation or two as well. Finally, even if
you're OK with the concept and feel that I've made some basically
reasonable design decisions, you might notice that the code is full of
bugs, needs a lot of cleanup, is missing features, lacks
documentation, and a bunch of other stuff. In that judgement, you
would be absolutely correct. I'm not posting it here because I'm
hoping to get it committed in November -- or at least, not THIS
November.  What I would like to do is getting some design feedback on
the preliminary patches, which I think will be more possible if
reviewers also have the main pg_plan_advice to look at as a way of
understanding why the exist, and also some feedback on the
pg_plan_advice patch itself.

Now I do want to caveat the statement that I am looking for feedback
just a little bit. I imagine that there will be some people reading
this who are already imagining how great life will be when they put
this into production, and begin complaining about either (1) features
that it's missing or (2) things that they don't like about the design
of the advice mini-language. What I'd ask you to keep in mind is that
you will not be able to put this into production unless and until
something gets committed, and getting this committed is probably going
to be super-hard even if you don't creep the scope, so maybe don't do
that, especially if you haven't read the README yet to understand what
the scope is actually intended to be. The details of the advice
mini-language are certainly open to negotiation; of everything, that
would be one of the easier things to change. However, keep in mind
that there are probably LOTS AND LOTS of people who all have their own
opinions about what decisions I should have made when designing that
mini-language, and an outcome where you personally get everything you
want and everyone who disagrees is out of luck is unlikely. In other
words, constructive suggestions for improvement are welcome, but
please think twice before turning this into a bikeshedding nightmare.
Now is the time to talk about whether I've got the overall design
somewhat correct moreso than whether I've spelled everything the way
you happen to prefer.[2]

I want to mention that, beyond the fact that I'm sure some people will
want to use something like this (with more feature and a lot fewer
bugs) in production, it seems to be super-useful for testing. We have
a lot of regression test cases that try to coerce the planner to do a
particular thing by manipulating enable_* GUCs, and I've spent a lot
of time trying to do similar things by hand, either for regression
test coverage or just private testing. This facility, even with all of
the bugs and limitations that it currently has, is exponentially more
powerful than frobbing enable_* GUCs. Once you get the hang of the
advice mini-language, you can very quickly experiment with all sorts
of plan shapes in ways that are currently very hard to do, and thereby
find out how expensive the planner thinks those things are and which
ones it thinks are even legal. So I see this as not only something
that people might find useful for in production deployments, but also
something that can potentially be really useful to advance PostgreSQL
development.

Which brings me to the question of where this code ought to go if it
goes anywhere at all. I decided to propose pg_plan_advice as a contrib
module rather than a part of core because I had to make a WHOLE lot of
opinionated design decisions just to get to the point of having
something that I could post and hopefully get feedback on. I figured
that all of those opinionated decisions would be a bit less
unpalatable if they were mostly encapsulated in a contrib module, with
the potential for some future patch author to write a different
contrib module that adopted different solutions to all of those
problems. But what I've also come to realize is that there's so much
infrastructure here that leaving the next person to reinvent it may
not be all that appealing. Query jumbling is a previous case where we
initially thought that different people might want to do different
things, but eventually realized that most people really just wanted
some solution that they didn't have to think too hard about. Likewise,
in this patch, the relation identifier system described in the README
is the only thing of its kind, to my knowledge, and any system that
wants to accomplish something similar to what pg_plan_advice does
would need a system like that. pg_hint_plan doesn't have something
like that, because pg_hint_plan is just trying to do hints. This is
trying to do round-trip-safe plan stability, where the system will
tell you how to refer unambiguously to a certain part of the query in
a way that will work correctly on every single query regardless of how
it's structured or how many times it refers to the same tables or to
different tables using the same aliases. If we say that we're never
going to put any of that infrastructure in core, then anyone who wants
to write a module to control the planner is going to need to start by
either (a) reinventing something similar, (b) cloning all the relevant
code, or (c) just giving up on the idea of unambiguous references to
parts of a query. None of those seem like great options, so now I'm
less sure whether contrib is actually the right place for this code,
but that's where I have put it for now. Feedback welcome, on this and
everything else.

Perhaps more than any other patch I've ever written, I know I'm
playing with fire here just by putting this out on the list, but I'm
nevertheless hopeful that something good can come of it, and I hope we
can have a constructive discussion about what that thing should be. I
think there is unquestionably is a lot of demand for the ability to
influence the planner in some form, but there is a lot of room for
debate about what exactly that should mean in practice. While I
personally am pretty happy with the direction of the code I've
written, modulo the large amount of not-yet-completed bug fixing and
cleanup, there's certainly plenty of room for other people to feel
differently, and finding out what other people think is, of course,
the whole point of posting things publicly before committing them --
or in this case, before even finishing them.[3] If you're interested
it contributing to the conversation, I urge you to start with the
following things: (1) the README in the final patch; (2) the
regression test examples in the final patch, which give a good sense
of what it actually looks like to use this; and (3) the earlier
patches, which show the minimum amount of core infrastructure that I
think we need in order to make something like this workable (ideas on
how to further reduce that footprint are very welcome).

Thanks,

--
Robert Haas
EDB: http://www.enterprisedb.com

[1] All of the earlier patches have been posted previously in some
form, but the commit messages have been rewritten for clarity, and the
"Allow for plugin control over path generation strategies" patch has
been heavily rewritten since it was last posted; the earlier versions
turned out to have substantial inadequacies.

[2] This is not to say that proposal to modify or improve the syntax
are unwelcome, but the bigger obstacle to getting something committed
here is probably reaching some agreement on the internal details. Any
changes to src/include/optimizer or src/backend/optimizer need careful
scrutiny from a design perspective. Also, keep in mind that the syntax
needs to fit what we can actually do: a proposal to change the syntax
to something that implies semantics we can't implement is a dead
letter.

[3] Note, however, that a proposal to achieve the same or similar
goals by different means is more welcome than a proposal that I should
have done some other project entirely. I've already put a lot of work
into these goals and hope to achieve them, at least to some degree,
before I start working toward something else.

  2. Re: pg_plan_advice

    Jakub Wartak <jakub.wartak@enterprisedb.com> — 2025-10-31T09:58:59Z

    On Thu, Oct 30, 2025 at 3:00 PM Robert Haas <robertmhaas@gmail.com> wrote:

[..over 400kB of attachments, yay]

Thank You for working on this!

My gcc-13 was nitpicking a little bit (see
compilation_warnings_v1.txt), so attached is just a tiny diff to fix
some of those issues. After that, clang-20 run was clean too.

> First, any form of user control over the planner tends to be a lightning rod for criticism around here.

I do not know where this is coming from, but everybody I've talked to
was saying this is needed to handle real enterprise databases and
applications. I just really love it, how one could precisely adjust
the plan with this even with the presence of heavy aliasing:

postgres=# explain (plan_advice, costs off) SELECT * FROM (select *
from t1 a join t2 b using (id)) a, t2 b, t3 c WHERE a.id = b.id and
b.id = c.id;
                     QUERY PLAN
-----------------------------------------------------
 Merge Join
   Merge Cond: (a.id = c.id)
   ->  Merge Join
         Merge Cond: (a.id = b.id)
         ->  Index Scan using t1_pkey on t1 a
         ->  Index Scan using t2_pkey on t2 b
   ->  Sort
         Sort Key: c.id
         ->  Seq Scan on t3 c
 Supplied Plan Advice:
   SEQ_SCAN(ble5) /* not matched */
 Generated Plan Advice:
   JOIN_ORDER(a#2 b#2 c)
   MERGE_JOIN_PLAIN(b#2 c)
   SEQ_SCAN(c)
   INDEX_SCAN(a#2 public.t1_pkey )
   NO_GATHER(c a#2 b#2)
(17 rows)

postgres=# set pg_plan_advice.advice = 'SEQ_SCAN(b#2)';
SET
postgres=# explain (plan_advice, costs off) SELECT * FROM (select *
from t1 a join t2 b using (id)) a, t2 b, t3 c WHERE a.id = b.id and
b.id = c.id;
                     QUERY PLAN
----------------------------------------------------
 Hash Join
   Hash Cond: (b.id = a.id)
   ->  Seq Scan on t2 b
   ->  Hash
         ->  Merge Join
               Merge Cond: (a.id = c.id)
               ->  Index Scan using t1_pkey on t1 a
               ->  Sort
                     Sort Key: c.id
                     ->  Seq Scan on t3 c
 Supplied Plan Advice:
   SEQ_SCAN(b#2) /* matched */
 Generated Plan Advice:
   JOIN_ORDER(b#2 (a#2 c))
   MERGE_JOIN_PLAIN(c)
   HASH_JOIN(c)
   SEQ_SCAN(b#2 c)
   INDEX_SCAN(a#2 public.t1_pkey)
   NO_GATHER(c a#2 b#2)

To attract a little attention to the $thread, the only bigger design
(usability) question that keeps ringing in my head is how we are going
to bind it to specific queries without even issuing any SETs(or ALTER
USER) in the far future in the grand scheme of things. The discussed
query id (hash), full query text comparison, maybe even strstr(query ,
"partial hit") or regex all seem to be kind too limited in terms of
what crazy ORMs can come up with (each query will be potentially
slightly different, but if optimizer reference points are stable that
should nail it good enough, but just enabling it for the very specific
set of queries and not the others [with same aliases] is some major
challenge).

Due to this, at some point I was even thinking about some hashes for
every plan node (including hashes of subplans), e.g.:
 Merge Join // hash(MERGE_JOIN_PLAIN(b#2) + ';' somehashval1 + ';'+
somehahsval2 ) => somehashval3
   Merge Cond: (a.id = c.id)
   ->  Merge Join
         Merge Cond: (a.id = b.id)
         ->  Index Scan using t1_pkey on t1 a // hash(INDEX_SCAN(a#2
public.t1_pkey)) => somehashval1
         ->  Index Scan using t2_pkey on t2 b // hash(INDEX_SCAN(b#2
public.t2_pkey)) => somehashval2

and then having a way to use `somehashval3` (let's say it's SHA1) as a
way to activate the necessary advice. Something like having a way to
express it using plan_advice.on_subplanhashes_plan_advice =
'somehashval3: SEQ_SCAN(b#2)'. This would have the benefit of being
able to override multiple similiar SQL queries in one go rather than
collecting all possible query_ids, but probably it's stupid,
heavyweight, but that would be my dream ;)

-J.

  3. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2025-10-31T12:51:34Z

    On Fri, Oct 31, 2025 at 5:59 AM Jakub Wartak
<jakub.wartak@enterprisedb.com> wrote:
> > First, any form of user control over the planner tends to be a lightning rod for criticism around here.
>
> I do not know where this is coming from, but everybody I've talked to
> was saying this is needed to handle real enterprise databases and
> applications. I just really love it, how one could precisely adjust
> the plan with this even with the presence of heavy aliasing:

Thanks for the kind words.

I'll respond to the points about compiler warnings later.

> To attract a little attention to the $thread, the only bigger design
> (usability) question that keeps ringing in my head is how we are going
> to bind it to specific queries without even issuing any SETs(or ALTER
> USER) in the far future in the grand scheme of things. The discussed
> query id (hash), full query text comparison, maybe even strstr(query ,
> "partial hit") or regex all seem to be kind too limited in terms of
> what crazy ORMs can come up with (each query will be potentially
> slightly different, but if optimizer reference points are stable that
> should nail it good enough, but just enabling it for the very specific
> set of queries and not the others [with same aliases] is some major
> challenge).

Yeah, I haven't really dealt with this problem yet.

> Due to this, at some point I was even thinking about some hashes for
> every plan node (including hashes of subplans),
[...]
>
> and then having a way to use `somehashval3` (let's say it's SHA1) as a
> way to activate the necessary advice. Something like having a way to

This doesn't make sense to me, because it seems circular. We can't use
anything in the plan to choose which advice string to use, because the
purpose of the advice string is to influence the choice of plan. In
other words, our choice of what advice string to use has to be based
on the properties of the query, not the plan. We can implement
anything we want to do in terms of exactly how that works: we can use
the query ID, or the query text, or the query node tree.
Hypothetically, we could call out to a user-defined function and pass
the query text or the query node tree as an argument and let it do
whatever it wants to decide on an advice string. The practical problem
here is computational cost -- any computation that gets performed for
every single query is going to have to be pretty cheap to avoid
creating a performance problem. That's why I thought matching on query
ID or exact matching on query text would likely be the most practical
approaches, aside from the obvious alternative of setting and
resetting pg_plan_advice.advice manually. But I haven't really
explored this area too much yet, because I need to get all the basics
working first.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  4. Re: pg_plan_advice

    Alastair Turner <minion@decodable.me> — 2025-10-31T21:17:41Z

    On Fri, 31 Oct 2025, 12:51 Robert Haas, <robertmhaas@gmail.com> wrote:

> On Fri, Oct 31, 2025 at 5:59 AM Jakub Wartak
> <jakub.wartak@enterprisedb.com> wrote:
> > > First, any form of user control over the planner tends to be a
> lightning rod for criticism around here.
> >
> > I do not know where this is coming from, but everybody I've talked to
> > was saying this is needed to handle real enterprise databases and
> > applications. I just really love it, how one could precisely adjust
> > the plan with this even with the presence of heavy aliasing:
>

I really like the functionality of the current patch as well, even though I
am suspicious of user control over the planner. By giving concise, precise
control over a plan, this allows people who believe they can out-plan the
planner to test their alternative, and possibly fail.

Whatever other UIs and integrations you build as you develop this towards
you goal, please keep what's currently there user accessible. Not only for
testing code, but also for testing users' belief that they know better.

Alastair

  5. Re: pg_plan_advice

    Hannu Krosing <hannuk@google.com> — 2025-11-01T16:10:08Z

    This weas recently shared in LinkedIn
https://www.vldb.org/pvldb/vol18/p5126-bress.pdf

For example it says that 31% of all queries are metadata queries, 78%
have LIMIT, 20% of queries have 10+ joins, with 0.52% exceeding 100
joins. , 12% of expressions have depths between 11-100 levels, some
exceeding 100. These deeply nested conditions create optimization
challenges benchmarks don't capture.etc.

This reinforces my belief thet we either should have some kind of
two-level optimization, where most queries are handled quickly but
with something to trigger a more elaborate optimisation and
investigation workflow.

Or alternatively we could just have an extra layer before the query is
sent to the database which deals with unwinding the product of
excessively stupid query generators (usually, but not always, some BI
tools :) )


On Fri, Oct 31, 2025 at 10:18 PM Alastair Turner <minion@decodable.me> wrote:
>
>
> On Fri, 31 Oct 2025, 12:51 Robert Haas, <robertmhaas@gmail.com> wrote:
>>
>> On Fri, Oct 31, 2025 at 5:59 AM Jakub Wartak
>> <jakub.wartak@enterprisedb.com> wrote:
>> > > First, any form of user control over the planner tends to be a lightning rod for criticism around here.
>> >
>> > I do not know where this is coming from, but everybody I've talked to
>> > was saying this is needed to handle real enterprise databases and
>> > applications. I just really love it, how one could precisely adjust
>> > the plan with this even with the presence of heavy aliasing:
>
>
> I really like the functionality of the current patch as well, even though I am suspicious of user control over the planner. By giving concise, precise control over a plan, this allows people who believe they can out-plan the planner to test their alternative, and possibly fail.
>
> Whatever other UIs and integrations you build as you develop this towards you goal, please keep what's currently there user accessible. Not only for testing code, but also for testing users' belief that they know better.
>
> Alastair

  6. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2025-11-03T16:18:08Z

    On Fri, Oct 31, 2025 at 5:17 PM Alastair Turner <minion@decodable.me> wrote:
> I really like the functionality of the current patch as well, even though I am suspicious of user control over the planner. By giving concise, precise control over a plan, this allows people who believe they can out-plan the planner to test their alternative, and possibly fail.

Indeed. The downside of letting people control anything is that they
may leverage that control to do something bad. However, I think it is
unlikely that very many people would prefer to write an entire query
plan by hand. If you wanted to do that, why would you being using
PostgreSQL in the first place? Furthermore, if somebody does try to do
that, I expect that they will find it frustrating and difficult: the
planner considers a large number of options even for simple queries
and an absolutely vast number of options for more difficult queries,
and a human being trying possibilities one by one is only ever going
to consider a tiny fraction of those possibilities. The ideal
possibility often won't be in that small subset of the search space,
and the user will be wasting their time. If that were the design goal
of this feature, I don't think it would be worth having.

But it isn't. As I say in the README, what I consider the principal
use case is reproducing plans that you know to have worked well in the
past. Sometimes, the planner is correct for a while and then it's
wrong later. We don't need to accept the proposition that users can
out-plan the planner. We only need to accept that they can tell good
plans from bad plans better than the planner. That is a low bar to
clear. The planner never finds out what happens when the plans that it
generates are actually executed, but users do. If they are
sufficiently experienced, they can make reasonable judgements about
whether the plan they're currently getting is one they'd like to
continue getting. Of course, they may make wrong judgements even then,
because they lack knowledge or experience or just make a mistake, but
it's not a farcically unreasonable thing to do. I've basically never
wanted to write my own query plan from scratch, but I've certainly
looked at many plans over the years and judged them to be great, or
terrible, or good for now but risky in the long-term; and I'm probably
not the only human being on the planet capable of making such
judgements with some degree of competence.

> Whatever other UIs and integrations you build as you develop this towards you goal, please keep what's currently there user accessible. Not only for testing code, but also for testing users' belief that they know better.

And this is also a good point. Knowledgeable and experienced users can
look at a plan that the planner generated, feel like it's bad, and
wonder why the planner picked it. You can try to figure that out by,
for example, setting enable_SOMETHING = false and re-running EXPLAIN,
but since there aren't that many such knobs relevant to any given
query, and since changing any of those knobs can affect large swathes
of the query and not just the part you're trying to understand better,
it can actually be really difficult to understand why the planner
thought that something was the best option. Sometimes you can't even
tell whether the planner thinks that the plan you expected to be
chosen is *impossible* or just *more expensive*, which is always one
of the things that I'm keen to find out when something weird is
happening. This can make answering that question a great deal easier.
If some important index is not getting used, you can say "no, really,
I want to see what happens with this query when you plan it with that
index" -- and then it either gives you a plan that does use that
index, and you can see how much more expensive it is and why, or it
still doesn't give you a plan using that index, and you know that the
index is inapplicable to the query or unusable in general for some
reason. You don't necessarily have it as a goal to coerce the planner
in production; your goal may very well be to find out why your belief
that you know better is incorrect.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  7. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2025-11-03T16:41:35Z

    On Sat, Nov 1, 2025 at 12:10 PM Hannu Krosing <hannuk@google.com> wrote:
> This reinforces my belief thet we either should have some kind of
> two-level optimization, where most queries are handled quickly but
> with something to trigger a more elaborate optimisation and
> investigation workflow.
>
> Or alternatively we could just have an extra layer before the query is
> sent to the database which deals with unwinding the product of
> excessively stupid query generators (usually, but not always, some BI
> tools :) )

I'd like to keep the focus of this thread on the patches that I'm
proposing, rather than other ideas for improving the planner. I
actually agree with you that at least the first of these things might
be a very good idea, but that would be an entirely separate project
from these patches, and I feel a lot more qualified to do this project
than that one.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  8. Re: pg_plan_advice

    John Naylor <johncnaylorls@gmail.com> — 2025-11-04T11:47:48Z

    On Thu, Oct 30, 2025 at 9:00 PM Robert Haas <robertmhaas@gmail.com> wrote:
> First, any form of user control over the
> planner tends to be a lightning rod for criticism around here. I've
> come to believe that's the wrong way of thinking about it: we can want
> to improve the planner over the long term and *also* want to have
> tools available to work around problems with it in the short term.

The most frustrating real-world incidents I've had were in the course
of customers planning a major version upgrade, or worse, after
upgrading and finding that a 5 minute query now takes 5 hours. I
mention this to emphasize that workarounds will be needed also to deal
with rare unintended effects that arise from our very attempts to
improve the planner.

> Further, we should not imagine that we're going to solve problems that
> have stumped other successful database projects any time in the
> foreseeable future; no product will ever get 100% of cases right, and
> you don't need to get to very obscure cases before other products
> throw up their hands just as we do.

Right.

> it seems to be super-useful for testing. We have
> a lot of regression test cases that try to coerce the planner to do a
> particular thing by manipulating enable_* GUCs, and I've spent a lot
> of time trying to do similar things by hand, either for regression
> test coverage or just private testing. This facility, even with all of
> the bugs and limitations that it currently has, is exponentially more
> powerful than frobbing enable_* GUCs. Once you get the hang of the
> advice mini-language, you can very quickly experiment with all sorts
> of plan shapes in ways that are currently very hard to do, and thereby
> find out how expensive the planner thinks those things are and which
> ones it thinks are even legal. So I see this as not only something
> that people might find useful for in production deployments, but also
> something that can potentially be really useful to advance PostgreSQL
> development.

That sounds very useful as well.

--
John Naylor
Amazon Web Services

  9. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2025-11-04T19:54:44Z

    On Fri, Oct 31, 2025 at 5:59 AM Jakub Wartak
<jakub.wartak@enterprisedb.com> wrote:
> My gcc-13 was nitpicking a little bit (see
> compilation_warnings_v1.txt), so attached is just a tiny diff to fix
> some of those issues. After that, clang-20 run was clean too.

Here's v2. Change log:

- Attempted to fix the compiler warnings. I didn't add elog() before
pg_unreachable() as you suggested; instead, I added a dummy return
afterwards. Let's see if that works. Also, I decided after reading the
comment for list_truncate() that what I'd done there was not going to
be acceptable, so I rewrote the code slightly. It now copies the list
when adding to it, instead of relying on the ability to use
list_truncate() to recreate the prior tstate.

- Deleted the SQL-callable pg_parse_advice function and related code.
That was useful to me early in development but I don't think anyone
will need it at this point; if you want to test whether an advice
string can be parsed, just try setting pg_plan_advice.advice.

- Fixed a couple of dumb bugs in pgpa_trove.c.

- Added a few more regression test scenarios.

- Fixed a couple of typos/thinkos.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  10. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2025-11-06T16:45:47Z

    Here's v3. I've attempted to fix some more things that cfbot didn't
like, one of which was an actual bug in 0005, and I also fixed a
stupid few bugs in pgpa_collector.c and added a few more tests.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  11. Re: pg_plan_advice

    Matheus Alcantara <matheusssilv97@gmail.com> — 2025-11-17T14:42:45Z

    Hi

On Thu Nov 6, 2025 at 1:45 PM -03, Robert Haas wrote:
> Here's v3. I've attempted to fix some more things that cfbot didn't
> like, one of which was an actual bug in 0005, and I also fixed a
> stupid few bugs in pgpa_collector.c and added a few more tests.
>
I've spent some time playing with these patches. I still don't have to
much comments on the syntax yet but I've noticed a small bug or perhaps
I'm missing something?

When I run CREATE EXTENSION pg_plan_advice I'm able to use the
EXPLAIN(plan_advice) but if try to open another connection, with the
extension already previously created, I'm unable to use once I drop and
re-create the extension.

tpch=# create extension pg_plan_advice;
ERROR:  extension "pg_plan_advice" already exists
tpch=# explain(plan_advice) select 1;
ERROR:  unrecognized EXPLAIN option "plan_advice"
LINE 1: explain(plan_advice) select 1;
                ^
tpch=# drop extension pg_plan_advice ;
DROP EXTENSION
tpch=# create extension pg_plan_advice;
CREATE EXTENSION
tpch=# explain(plan_advice) select 1;
                QUERY PLAN
------------------------------------------
 Result  (cost=0.00..0.01 rows=1 width=4)
 Generated Plan Advice:
   NO_GATHER("*RESULT*")


And thanks for working on this. I think that this can be a very useful
feature for both users and for postgres hackers, +1 for the idea.

-- 
Matheus Alcantara
EDB: http://www.enterprisedb.com

  12. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2025-11-17T15:09:36Z

    On Mon, Nov 17, 2025 at 9:42 AM Matheus Alcantara
<matheusssilv97@gmail.com> wrote:
> I've spent some time playing with these patches. I still don't have to
> much comments on the syntax yet but I've noticed a small bug or perhaps
> I'm missing something?

Cool, thanks for looking. I am guessing that the paucity of feedback
thus far is partly because there's a lot of stuff to absorb -- though
the main point at this stage is really to get some opinions on the
planner infrastructure/hooks, which don't necessarily require full
understanding of (never mind agreement with) the design of
pg_plan_advice itself.

> When I run CREATE EXTENSION pg_plan_advice I'm able to use the
> EXPLAIN(plan_advice) but if try to open another connection, with the
> extension already previously created, I'm unable to use once I drop and
> re-create the extension.

This is just an idiosyncrasy of PostgreSQL's extension framework.
Whether or not EXPLAIN (PLAN_ADVICE) works depends on whether the
shared module has been loaded, not whether the extension has been
created. The purpose of CREATE EXTENSION is to put SQL objects, such
as function definitions, into the database, but there's no SQL
required to enable EXPLAIN (PLAN_ADVICE) -- or for setting the
pg_plan_advice.advice GUC. However, running CREATE EXTENSION to
establish the function definitions will incidentally load the shared
module into that particular session.

Therefore, the best way to use this module is to add pg_plan_advice to
shared_preload_libraries. Alternatively, you can use
session_preload_libraries or run LOAD in an individual session. If you
don't care about the collector interface, that's really all you need.
If you do care about the collector interface, then in addition you
will need to run CREATE EXTENSION, so that the SQL functions needed to
access it are available.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  13. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2025-11-18T16:19:24Z

    Here's v4. This version has some bug fixes and test case changes to
0005 and 0006, with the goal of getting CI to pass cleanly (which it
now does for me, but let's see if cfbot agrees).

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  14. Re: pg_plan_advice

    Dian Fay <di@nmfay.com> — 2025-11-23T00:43:40Z

    On Tue Nov 18, 2025 at 11:19 AM EST, Robert Haas wrote:
> Here's v4. This version has some bug fixes and test case changes to
> 0005 and 0006, with the goal of getting CI to pass cleanly (which it
> now does for me, but let's see if cfbot agrees).

Thanks for working on this, Robert! I think the design seems solid (and
very powerful) from a user perspective. I was curious what would happen
with row-level security interactions so I tried it out on a toy example
I put together a while back. I found one case where scan advice fails on
an intentionally naive/bad policy implementation, but I'm not sure why
and it seems like the kind of weird corner case that might be useful to
reason about. See attached for the setup script, then:

set pg_plan_advice.advice = 'BITMAP_HEAP_SCAN(item public.item_tags_idx)';
set item_reader.allowed_tags = '{alpha,beta}';
set role item_reader;

explain (plan_advice, analyze, verbose, costs, timing)
select * from item
where value ilike 'a%' and tags && array[1];

Seq Scan on public.item  (cost=0.00..41777312.00 rows=54961 width=67) (actual time=2.947..8603.333 rows=6762.00 loops=1)
 Disabled: true
 Output: item.id, item.value, item.tags
 Filter: (EXISTS(SubPlan exists_1) AND (item.value ~~* 'a%'::text) AND (item.tags && '{1}'::integer[]))
 Rows Removed by Filter: 993238
 Buffers: shared hit=1012312
 SubPlan exists_1
   ->  Seq Scan on public.tag  (cost=0.00..41.75 rows=1 width=0) (actual time=0.008..0.008 rows=0.21 loops=1000000)
         Filter: ((current_setting('item_reader.allowed_tags'::text) IS NOT NULL) AND ((current_setting('item_reader.allowed_tags'::text))::text[] @> ARRAY[tag.name]) AND (item.tags @> ARRAY[tag.id]))
         Rows Removed by Filter: 18
         Buffers: shared hit=1000000
Planning Time: 1.168 ms
Supplied Plan Advice:
 BITMAP_HEAP_SCAN(item public.item_tags_idx) /* matched, failed */
Generated Plan Advice:
 SEQ_SCAN(item tag@exists_1)
 NO_GATHER(item tag@exists_1)
Execution Time: 8603.615 ms

Since the policies don't contain any execution boundaries, all the quals
should be going into a single bucket for planning if I understand the
process correctly. The bitmap heap scan should be a candidate given the
`tags &&` predicate (and indeed if I switch to a privileged role, the
advice matches successfully without any policies in the mix), but gdb
shows the walker bouncing out of pgpa_walker_contains_scan without any
candidate scans for the BITMAP_HEAP_SCAN strategy.

I do want to avoid getting bikesheddy about the advice language so I'll
forbear from syntax discussion, but one design thought with lower-level
implications did occur to me as I was playing with this: it might be
useful in some situations to influence the planner _away_ from known
worse paths while leaving it room to decide on the best other option. I
think the work you did in path management should make this pretty
straightforward for join and scan strategies, since it looks like you've
basically made the enable_* gucs a runtime-configurable bitmask (which
seems like a perfectly reasonable approach to my "have done some source
diving but not an internals hacker" eyes), and could disable one as
easily as forcing one.

"Don't use this one index" sounds more fiddly to implement, but also
less valuable since in that case you probably already know which other
index it should be using.

  15. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2025-11-24T16:14:27Z

    On Sat, Nov 22, 2025 at 7:43 PM Dian Fay <di@nmfay.com> wrote:
> Thanks for working on this, Robert!

Thanks for looking at it! I was hoping for a bit more in the way of
responses by now, honestly.

> Since the policies don't contain any execution boundaries, all the quals
> should be going into a single bucket for planning if I understand the
> process correctly. The bitmap heap scan should be a candidate given the
> `tags &&` predicate (and indeed if I switch to a privileged role, the
> advice matches successfully without any policies in the mix), but gdb
> shows the walker bouncing out of pgpa_walker_contains_scan without any
> candidate scans for the BITMAP_HEAP_SCAN strategy.

I can understand why it seems that way, but when I try setting
enable_seqscan=false instead of using pg_plan_advice, I get exactly
the same result. I think this is actually a great example both of why
this is actually a very powerful tool and also why it has the
potential to be really confusing. The power comes from the fact that
you can find out whether the planner thinks that the thing you want to
do is even possible. In this case, that's easy anyway because the
example is simple enough, but sometimes you can't set
enable_seqscan=false or similar because it would change too many other
things in the plan at that same time and you wouldn't be able to
compare. In those situations, this figures to be useful. However, all
this can do is tell you that the answer to the question "is this a
possible plan shape?" is "no". It cannot tell you why, and you may
easily find the result counterintuitive.

And honestly, this is one of the things I'm worried about if we go
forward with this, that we'll get a ton of people who think it doesn't
work because it doesn't force the planner to do things which the
planner rejects on non-cost considerations. We're going to need really
good documentation to explain to people that if you use this to try to
force a plan and you can't, that's not a bug, that's the planner
telling you that that plan shape is not able to be considered for some
reason. That won't keep people from complaining about things that
aren't really bugs, but at least it will mean that there's a link we
can give them to explain why the way they're thinking about it is
incorrect. However, that will just beg the next question of WHY the
planner doesn't think a certain plan can be considered, and honestly,
I've found over the years that I often need to resort to the source
code to answer those kinds of questions. People who are not good at
reading C source code are not going to like that answer very much, but
I still think it's better if they know THAT the planner thinks the
plan shape is impossible even if we can't tell them WHY the planner
thinks that the plan shape is impossible. We probably will want to
document at least some of the common reasons why this happens, to cut
down on getting the same questions over and over again.

In this particular case, I think the problem is that the user-supplied
qual item.tags @> ARRAY[id] is not leakproof and therefore must be
tested after the security qual. There's no way to use a Bitmap Heap
Scan without reversing the order of those tests.

> I do want to avoid getting bikesheddy about the advice language so I'll
> forbear from syntax discussion, but one design thought with lower-level
> implications did occur to me as I was playing with this: it might be
> useful in some situations to influence the planner _away_ from known
> worse paths while leaving it room to decide on the best other option. I
> think the work you did in path management should make this pretty
> straightforward for join and scan strategies, since it looks like you've
> basically made the enable_* gucs a runtime-configurable bitmask (which
> seems like a perfectly reasonable approach to my "have done some source
> diving but not an internals hacker" eyes), and could disable one as
> easily as forcing one.

I mostly agree. Saying not to use a sequential scan on a certain
table, or not to use a particular index, or not to use a particular
join method seem like things that would be potentially useful, and
they would be straightforward generalizations of what the code already
does. For me, that would principally be a way to understand better why
the planner chose what it did. I often wonder what the planner's
second choice would have been, but I don't just want the plan with the
second-cheapest overall cost, because that will be something just
trivially different. I want the cheapest plan that excludes some key
element of the current plan, so I can see a meaningfully different
alternative.

That said, I don't see this being a general thing that would make
sense across all of the tags that pg_plan_advice supports. For
example, NO_JOIN_ORDER() sounds hard to implement and largely useless.

The main reason I haven't done this is that I want to keep the focus
on plan stability, or said differently, on things that can properly
round-trip. You should be able to run a query with EXPLAIN
(PLAN_ADVICE), then set pg_plan_advice.advice to the resulting string,
rerun the query, and get the same plan with all of the advice
successfully matching. Since EXPLAIN (PLAN_ADVICE) would never emit
these proposed negative tags, we'd need to think a little bit harder
about how that stuff should be tested. That's not necessarily a big
deal or anything, but I didn't think it was an essential element of
the initial scope, so I left it out. I'm happy to add it in at some
point, or for someone else to do so, but not until this much is
working well.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  16. Re: pg_plan_advice

    Dian Fay <di@nmfay.com> — 2025-11-30T03:16:44Z

    On Mon Nov 24, 2025 at 11:14 AM EST, Robert Haas wrote:
> On Sat, Nov 22, 2025 at 7:43 PM Dian Fay <di@nmfay.com> wrote:
>> Since the policies don't contain any execution boundaries, all the quals
>> should be going into a single bucket for planning if I understand the
>> process correctly. The bitmap heap scan should be a candidate given the
>> `tags &&` predicate (and indeed if I switch to a privileged role, the
>> advice matches successfully without any policies in the mix), but gdb
>> shows the walker bouncing out of pgpa_walker_contains_scan without any
>> candidate scans for the BITMAP_HEAP_SCAN strategy.
>
> In this particular case, I think the problem is that the user-supplied
> qual item.tags @> ARRAY[id] is not leakproof and therefore must be
> tested after the security qual. There's no way to use a Bitmap Heap
> Scan without reversing the order of those tests.

Right, I keep forgetting the functions underneath those array operators
aren't leakproof. Thanks for digging.

> And honestly, this is one of the things I'm worried about if we go
> forward with this, that we'll get a ton of people who think it doesn't
> work because it doesn't force the planner to do things which the
> planner rejects on non-cost considerations. We're going to need really
> good documentation to explain to people that if you use this to try to
> force a plan and you can't, that's not a bug, that's the planner
> telling you that that plan shape is not able to be considered for some
> reason.

Once we're closer to consensus on pg_plan_advice or something like it
landing, I'm interested in helping out on this end of things!

  17. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2025-12-05T19:57:09Z

    On Sat, Nov 29, 2025 at 10:17 PM Dian Fay <di@nmfay.com> wrote:
> Once we're closer to consensus on pg_plan_advice or something like it
> landing, I'm interested in helping out on this end of things!

Thanks!

014f9a831a320666bf2195949f41710f970c54ad removes the need for what was
previously 0004, so here is a new patch series with that dropped, to
avoid confusing cfbot or human reviewers.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  18. Re: pg_plan_advice

    Greg Burd <greg@burd.me> — 2025-12-08T20:39:25Z

    On Fri, Dec 5, 2025, at 2:57 PM, Robert Haas wrote:
> 014f9a831a320666bf2195949f41710f970c54ad removes the need for what was
> previously 0004, so here is a new patch series with that dropped, to
> avoid confusing cfbot or human reviewers.
>
> -- 
> Robert Haas
> EDB: http://www.enterprisedb.com

Hey Robert,

Thanks for working on this!  I think the idea has merit, I hope it lands sometime soon.

I've worked on extending PostgreSQL's planner for a JSONB-like document system. The new query shapes frequently caused mysterious planner issues. Having the ability to:

1. Dump detailed planner decisions for comparison during development
2. Record planner choices from customer databases to reproduce their issues
3. Apply fixed plans to specific queries as a quick customer workaround while addressing root causes in later releases

...would have been invaluable.

Yes, there's danger here ("with great power comes great responsibility"), but I see this as providing more information to make better decisions when working with the black art of planner logic.

ORM Query Variation Challenge

Jakob's point about "crazy ORMs" is important. ORMs generate queries with minor variations that should ideally match the same plan advice. I need to study the plan advice matching logic more deeply to understand how it handles query variations.

This reminded me of Erlang/Elixir's "parse transforms" - compiled code forms an AST that registered transforms can modify via pattern matching. The concept might be relevant here: pattern-matching portions of query ASTs to apply advice despite syntactic variations. I'd need to think more about whether this intersects well with the current design or if it's impractical, but it's worth exploring.

> Attachments:
> * v5-0001-Store-information-about-range-table-flattening-in.patch

contrib/pg_overexplain/pg_overexplain.c

+               /* Advance to next SubRTInfo, if it's time. */
+               if (lc_subrtinfo != NULL)
+               {
+                       next_rtinfo = lfirst(lc_subrtinfo);
+                       if (rti > next_rtinfo->rtoffset)

Should the test be >= not >? Unless I am I reading this wrong, when rti == rtoffset, that's the first entry of the new subplan's range table. That would mean that the current logic skips displaying the subplan name for the first RTE of each subplan.

in src/include/nodes/plannodes.h there is:

+typedef struct SubPlanRTInfo
+{
+       NodeTag         type;
+       const char *plan_name;
+       Index           rtoffset;
+       bool            dummy;
+} SubPlanRTInfo;

This is where I get confused, if rtoffset is an Index, then the comparison (above) in pg_overexplain uses rti > next_rtinfo->rtoffset where rti starts at 1. If rtoffset is 0 for the first subplan, the logic might be off-by-one, no?

> This commit teaches pg_overexplain'e RANGE_TABLE option to make use
Minor nit in the commit message, "pg_overexplain'e" should be "pg_overexplain's"

> * v5-0002-Store-information-about-elided-nodes-in-the-final.patch

+/*
+ * Record some details about a node removed from the plan during setrefs
+ * procesing, for the benefit of code trying to reconstruct planner decisions
+ * from examination of the final plan tree.
+ */

Nit, "procesing" should be "processing"

> * v5-0003-Store-information-about-Append-node-consolidation.patch

src/backend/optimizer/path/allpaths.c

/* Now consider each interesting sort ordering */
foreach(lcp, all_child_pathkeys)
{
        List       *subpaths = NIL;
        bool            subpaths_valid = true;
+       List       *subpath_cars = NIL;
        List       *startup_subpaths = NIL;
        bool            startup_subpaths_valid = true;
+       List       *startup_subpath_cars = NIL;
        List       *partial_subpaths = NIL;
+       List       *partial_subpath_cars = NIL;
        List       *pa_partial_subpaths = NIL;
        List       *pa_nonpartial_subpaths = NIL;
+       List       *pa_subpath_cars = NIL;

I find "cars" a bit cryptic (albeit clever), I think I've decoded it properly and it stands for "child_append_relid_sets", correct?  Could you add a comment or use a clearer name like subpath_child_relids or consolidated_relid_sets?


+accumulate_append_subpath(Path *path, List **subpaths, List **special_subpaths,
+                                                 List **child_append_relid_sets)
 {
        if (IsA(path, AppendPath))
        {
@@ -2219,6 +2256,8 @@ accumulate_append_subpath(Path *path, List **subpaths, List **special_subpaths)
                if (!apath->path.parallel_aware || apath->first_partial_path == 0)
                {
                        *subpaths = list_concat(*subpaths, apath->subpaths);
+                       *child_append_relid_sets =
+                               lappend(*child_append_relid_sets, path->parent->relids);

Is it possible that when pulling up multiple subpaths from an AppendPath, only ONE relid set is added to child_append_relid_sets, but MULTIPLE paths are added to subpaths?  If so, that would break the correspondence between the lists which would be bad, right?

src/include/nodes/pathnodes.h
+ * Whenever accumulate_append_subpath() allows us to consolidate multiple
+ * levels of Append paths are consolidated down to one, we store the RTI
+ * sets for the omitted paths in child_append_relid_sets. This is not necessary
+ * for planning or execution; we do it for the benefit of code that wants
+ * to inspect the final plan and understand how it came to be.

Minor: "paths are consolidated" is redundant, should be "paths consolidated" or "allows us to consolidate".

> * v5-0004-Allow-for-plugin-control-over-path-generation-str.patch

src/backend/optimizer/path/costsize.c
+       else
+               enable_mask |= PGS_CONSIDER_NONPARTIAL;

-       path->disabled_nodes = enable_seqscan ? 0 : 1;
+       path->disabled_nodes =
+               (baserel->pgs_mask & enable_mask) == enable_mask ? 0 : 1;

When parallel_workers > 0 the path is partial and doesn't need PGS_CONSIDER_NONPARTIAL. But if parallel_workers == 0, it's non-partial and DOES need it, right?  Would this mean that non-partial paths can be disabled even when the scan type itself (e.g., PGS_SEQSCAN) is enabled?  Intentional?

> * v5-0005-WIP-Add-pg_plan_advice-contrib-module.patch

It seems this is still WIP with a solid start, I'm not going to dig too much into it. :)

Keep it up, best.

-greg

  19. Re: pg_plan_advice

    Jacob Champion <jacob.champion@enterprisedb.com> — 2025-12-09T01:18:50Z

    Hello,

On Fri, Dec 5, 2025 at 11:57 AM Robert Haas <robertmhaas@gmail.com> wrote:
> 014f9a831a320666bf2195949f41710f970c54ad removes the need for what was
> previously 0004, so here is a new patch series with that dropped, to
> avoid confusing cfbot or human reviewers.

I really like this idea! Telling the planner, "if you need to make a
decision for [this thing], choose [this way]," seems to be a really
nice way of sidestepping many of the concerns with "user control".

I've started an attempt to throw a fuzzer at this, because I'm pretty
useless when it comes to planner/optimizer review. I don't really know
what the overall fuzzing strategy is going to be, given the multiple
complicated inputs that have to be constructed and somehow correlated
with each other, but I'll try to start small and expand:

a) fuzz the parser first, because it's easy and we can get interesting inputs
b) fuzz the AST utilities, seeded with "successful" corpus members from a)
c) stare really hard at the corpus of b) and figure out how to
usefully mutate a PlannedStmt with it
d) use c) to fuzz pgpa_plan_walker, then pgpa_output_advice, then...?

I'm in the middle of an implementation of b) now, and it noticed the
following code (which probably bodes well for the fuzzer itself!):

>        if (rid->partnsp == NULL)
>            result = psprintf("%s/%s", result,
>                              quote_identifier(rid->partnsp));

I assume that should be quote_identifier(rid->partrel)?

= Other Notes =

GCC 11 complains about the following code in pgpa_collect_advice():

>        dsa_area   *area = pg_plan_advice_dsa_area();
>        dsa_pointer ca_pointer;
>
>        pgpa_make_collected_advice(userid, dbid, queryId, now,
>                                   query_string, advice_string, area,
>                                   &ca_pointer);
>        pgpa_store_shared_advice(ca_pointer);

It doesn't know that area is guaranteed to be non-NULL, so it can't
prove that ca_pointer is initialized.

(GCC also complains about unique_nonjoin_rtekind() not initializing
the rtekind, but I think that's because of a bug [1].)

--Jacob

[1] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107838

  20. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2025-12-09T19:34:43Z

    On Mon, Dec 8, 2025 at 3:39 PM Greg Burd <greg@burd.me> wrote:
> Thanks for working on this!  I think the idea has merit, I hope it lands sometime soon.

Thanks. I think it needs a good deal more review first, but I
appreciate the support.

> > Attachments:
> > * v5-0001-Store-information-about-range-table-flattening-in.patch
>
> contrib/pg_overexplain/pg_overexplain.c
>
> +               /* Advance to next SubRTInfo, if it's time. */
> +               if (lc_subrtinfo != NULL)
> +               {
> +                       next_rtinfo = lfirst(lc_subrtinfo);
> +                       if (rti > next_rtinfo->rtoffset)
>
> Should the test be >= not >? Unless I am I reading this wrong, when rti == rtoffset, that's the first entry of the new subplan's range table. That would mean that the current logic skips displaying the subplan name for the first RTE of each subplan.

I don't think so. I think I actually had it that way at one point, and
I believe I found that it was wrong. RTIs are 1-based, so the smallest
per-subquery RTI is 1. rtoffset is the amount that must be added to
the per-subquery RTI to get a "flat" RTI that can be used to index
into the final range table. But if you find that theoretical argument
unconvincing, by all means please test it and see what happens!

> > This commit teaches pg_overexplain'e RANGE_TABLE option to make use
> Minor nit in the commit message, "pg_overexplain'e" should be "pg_overexplain's"

Thanks, fixed in my local branch.

> > * v5-0002-Store-information-about-elided-nodes-in-the-final.patch
>
> +/*
> + * Record some details about a node removed from the plan during setrefs
> + * procesing, for the benefit of code trying to reconstruct planner decisions
> + * from examination of the final plan tree.
> + */
>
> Nit, "procesing" should be "processing"

Thanks, fixed in my local branch.

> > * v5-0003-Store-information-about-Append-node-consolidation.patch
>
> src/backend/optimizer/path/allpaths.c
>
> /* Now consider each interesting sort ordering */
> foreach(lcp, all_child_pathkeys)
> {
>         List       *subpaths = NIL;
>         bool            subpaths_valid = true;
> +       List       *subpath_cars = NIL;
>         List       *startup_subpaths = NIL;
>         bool            startup_subpaths_valid = true;
> +       List       *startup_subpath_cars = NIL;
>         List       *partial_subpaths = NIL;
> +       List       *partial_subpath_cars = NIL;
>         List       *pa_partial_subpaths = NIL;
>         List       *pa_nonpartial_subpaths = NIL;
> +       List       *pa_subpath_cars = NIL;
>
> I find "cars" a bit cryptic (albeit clever), I think I've decoded it properly and it stands for "child_append_relid_sets", correct?  Could you add a comment or use a clearer name like subpath_child_relids or consolidated_relid_sets?

I certainly admit that this is a bit too clever. I am not entirely
sure how to make it less clever. There needs to be a
child-append-relid-sets list corresponding to every current and future
subpath list, and the names of some of those subpath lists are already
quite long, so whatever naming convention we choose for the "cars"
lists had better not add too much more length to the variable name. I
felt like someone looking at this might initially be confused by what
"cars" meant, but then I thought that they would probably look at how
the variable was used and see that it was for example being passed as
the second argument to get_singleton_append_subpath(), which is named
child_append_relid_sets, or being passed to create_append_path or
create_merge_append_path, which also use that naming. I figured that
this would clear up the confusion pretty quickly. I could certainly
add a comment above this block of variable assignments saying
something like "for each list of paths, we must also maintain a list
of child append relid sets, etc. etc." but I worried that this would
create as much confusion as it solved, i.e. somebody reading the code
would be going: why is this comment here? Is it trying to tell me that
there's something weirder going on than what is anyway obvious?

If I get more opinions that some clarification is needed here, I'm
happy to change it, especially if those opinions agree with each other
on exactly what to change, but I think for now I'll leave it as it is.

> +accumulate_append_subpath(Path *path, List **subpaths, List **special_subpaths,
> +                                                 List **child_append_relid_sets)
>  {
>         if (IsA(path, AppendPath))
>         {
> @@ -2219,6 +2256,8 @@ accumulate_append_subpath(Path *path, List **subpaths, List **special_subpaths)
>                 if (!apath->path.parallel_aware || apath->first_partial_path == 0)
>                 {
>                         *subpaths = list_concat(*subpaths, apath->subpaths);
> +                       *child_append_relid_sets =
> +                               lappend(*child_append_relid_sets, path->parent->relids);
>
> Is it possible that when pulling up multiple subpaths from an AppendPath, only ONE relid set is added to child_append_relid_sets, but MULTIPLE paths are added to subpaths?  If so, that would break the correspondence between the lists which would be bad, right?

That would indeed be bad, but I'm not clear on how you think it could
happen. Can you clarify?

> src/include/nodes/pathnodes.h
> + * Whenever accumulate_append_subpath() allows us to consolidate multiple
> + * levels of Append paths are consolidated down to one, we store the RTI
> + * sets for the omitted paths in child_append_relid_sets. This is not necessary
> + * for planning or execution; we do it for the benefit of code that wants
> + * to inspect the final plan and understand how it came to be.
>
> Minor: "paths are consolidated" is redundant, should be "paths consolidated" or "allows us to consolidate".

Thanks, fixed in my local branch.

> > * v5-0004-Allow-for-plugin-control-over-path-generation-str.patch
>
> src/backend/optimizer/path/costsize.c
> +       else
> +               enable_mask |= PGS_CONSIDER_NONPARTIAL;
>
> -       path->disabled_nodes = enable_seqscan ? 0 : 1;
> +       path->disabled_nodes =
> +               (baserel->pgs_mask & enable_mask) == enable_mask ? 0 : 1;
>
> When parallel_workers > 0 the path is partial and doesn't need PGS_CONSIDER_NONPARTIAL. But if parallel_workers == 0, it's non-partial and DOES need it, right?  Would this mean that non-partial paths can be disabled even when the scan type itself (e.g., PGS_SEQSCAN) is enabled?  Intentional?

See this comment:

 * Finally, unsetting PGS_CONSIDER_NONPARTIAL disables all non-partial paths
 * except those that use Gather or Gather Merge. In most other cases, a
 * plugin can nudge the planner toward a particular strategy by disabling
 * all of the others, but that doesn't work here: unsetting PGS_SEQSCAN,
 * for instance, would disable both partial and non-partial sequential scans.

> It seems this is still WIP with a solid start, I'm not going to dig too much into it. :)
>
> Keep it up, best.

Thanks for the review so far!


--
Robert Haas
EDB: http://www.enterprisedb.com

  21. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2025-12-09T19:45:47Z

    On Mon, Dec 8, 2025 at 8:19 PM Jacob Champion
<jacob.champion@enterprisedb.com> wrote:
> I really like this idea! Telling the planner, "if you need to make a
> decision for [this thing], choose [this way]," seems to be a really
> nice way of sidestepping many of the concerns with "user control".
>
> I've started an attempt to throw a fuzzer at this, because I'm pretty
> useless when it comes to planner/optimizer review. I don't really know
> what the overall fuzzing strategy is going to be, given the multiple
> complicated inputs that have to be constructed and somehow correlated
> with each other, but I'll try to start small and expand:
>
> a) fuzz the parser first, because it's easy and we can get interesting inputs
> b) fuzz the AST utilities, seeded with "successful" corpus members from a)
> c) stare really hard at the corpus of b) and figure out how to
> usefully mutate a PlannedStmt with it
> d) use c) to fuzz pgpa_plan_walker, then pgpa_output_advice, then...?

Cool. I'm bad at fuzzing, but I think fuzzing by someone who is good
at it is very promising for this kind of patch.

> I'm in the middle of an implementation of b) now, and it noticed the
> following code (which probably bodes well for the fuzzer itself!):
>
> >        if (rid->partnsp == NULL)
> >            result = psprintf("%s/%s", result,
> >                              quote_identifier(rid->partnsp));
>
> I assume that should be quote_identifier(rid->partrel)?

Yes, thanks. Fixed locally. By the way, if your fuzzer can also
produces some things to add contrib/pg_plan_advice/sql for cases like
this, that would be quite helpful. Ideally I would have caught this
with a manually-written test case, but obviously that didn't happen.

> = Other Notes =
>
> GCC 11 complains about the following code in pgpa_collect_advice():
>
> >        dsa_area   *area = pg_plan_advice_dsa_area();
> >        dsa_pointer ca_pointer;
> >
> >        pgpa_make_collected_advice(userid, dbid, queryId, now,
> >                                   query_string, advice_string, area,
> >                                   &ca_pointer);
> >        pgpa_store_shared_advice(ca_pointer);
>
> It doesn't know that area is guaranteed to be non-NULL, so it can't
> prove that ca_pointer is initialized.

I don't know what to do about that. I can understand why it might be
unable to prove that, but I don't see an obvious way to change the
code that would make life easier. I could add Assert(area != NULL)
before the call to pgpa_make_collected_advice() if that helps.

> (GCC also complains about unique_nonjoin_rtekind() not initializing
> the rtekind, but I think that's because of a bug [1].)

This one could be fixed with a dummy initialization, if needed.

--
Robert Haas
EDB: http://www.enterprisedb.com

  22. Re: pg_plan_advice

    amit <amitlangote09@gmail.com> — 2025-12-10T11:20:38Z

    Hi Robert,

On Thu, Oct 30, 2025 at 11:00 PM Robert Haas <robertmhaas@gmail.com> wrote:
> As I have mentioned on previous threads, for the past while I have
> been working on planner extensibility. I've posted some extensibility
> patches previously, and got a few of them committed in
> Sepember/October with Tom's help, but I think the time has come a
> patch which actually makes use of that infrastructure as well as some
> further infrastructure that I'm also including in this posting.[1] The
> final patch in this series adds a new contrib module called
> pg_plan_advice. Very briefly, what pg_plan_advice knows how to do is
> process a plan and emits a (potentially long) long text string in a
> special-purpose mini-language that describes a bunch of key planning
> decisions, such as the join order, selected join methods, types of
> scans used to access individual tables, and where and how
> partitionwise join and parallelism were used. You can then set
> pg_plan_advice.advice to that string to get a future attempt to plan
> the same query to reproduce those decisions, or (maybe a better idea)
> you can trim that string down to constrain some decisions (e.g. the
> join order) but not others (e.g. the join methods), or (if you want to
> make your life more exciting) you can edit that advice string and
> thereby attempt to coerce the planner into planning the query the way
> you think best. There is a README that explains the design philosophy
> and thinking in a lot more detail, which is a good place to start if
> you're curious, and I implore you to read it if you're interested, and
> *especially* if you're thinking of flaming me.

Thanks for posting this.  Looks very interesting to me.

These are just high-level comments after browsing the patches and
reading some bits like pgpa_identifier to get myself familiarized with
the project.  I like that the key concept here is plan stability
rather than plan control, because that framing makes it easier to
treat this as infrastructure instead of policy.

> I want to mention that, beyond the fact that I'm sure some people will
> want to use something like this (with more feature and a lot fewer
> bugs) in production, it seems to be super-useful for testing. We have
> a lot of regression test cases that try to coerce the planner to do a
> particular thing by manipulating enable_* GUCs, and I've spent a lot
> of time trying to do similar things by hand, either for regression
> test coverage or just private testing. This facility, even with all of
> the bugs and limitations that it currently has, is exponentially more
> powerful than frobbing enable_* GUCs. Once you get the hang of the
> advice mini-language, you can very quickly experiment with all sorts
> of plan shapes in ways that are currently very hard to do, and thereby
> find out how expensive the planner thinks those things are and which
> ones it thinks are even legal. So I see this as not only something
> that people might find useful for in production deployments, but also
> something that can potentially be really useful to advance PostgreSQL
> development.

+1, the testing benefits make this worthwhile.

> Which brings me to the question of where this code ought to go if it
> goes anywhere at all. I decided to propose pg_plan_advice as a contrib
> module rather than a part of core because I had to make a WHOLE lot of
> opinionated design decisions just to get to the point of having
> something that I could post and hopefully get feedback on. I figured
> that all of those opinionated decisions would be a bit less
> unpalatable if they were mostly encapsulated in a contrib module, with
> the potential for some future patch author to write a different
> contrib module that adopted different solutions to all of those
> problems. But what I've also come to realize is that there's so much
> infrastructure here that leaving the next person to reinvent it may
> not be all that appealing. Query jumbling is a previous case where we
> initially thought that different people might want to do different
> things, but eventually realized that most people really just wanted
> some solution that they didn't have to think too hard about. Likewise,
> in this patch, the relation identifier system described in the README
> is the only thing of its kind, to my knowledge, and any system that
> wants to accomplish something similar to what pg_plan_advice does
> would need a system like that. pg_hint_plan doesn't have something
> like that, because pg_hint_plan is just trying to do hints. This is
> trying to do round-trip-safe plan stability, where the system will
> tell you how to refer unambiguously to a certain part of the query in
> a way that will work correctly on every single query regardless of how
> it's structured or how many times it refers to the same tables or to
> different tables using the same aliases. If we say that we're never
> going to put any of that infrastructure in core, then anyone who wants
> to write a module to control the planner is going to need to start by
> either (a) reinventing something similar, (b) cloning all the relevant
> code, or (c) just giving up on the idea of unambiguous references to
> parts of a query. None of those seem like great options, so now I'm
> less sure whether contrib is actually the right place for this code,
> but that's where I have put it for now. Feedback welcome, on this and
> everything else.

On the relation identifier system: IMHO this part doesn't seem as
opinionated as the advice mini-language. The requirements pretty much
dictate the design -- you need alias names and occurrence counters to
handle self-joins, partition fields for partitioned tables, and a
string representation to survive dump/restore. There doesn't seem to
be much flexibility in that.

Given that, it seems more practical to put this in core from the
start. Extensions that might want to build plan-advice-like
functionality shouldn’t have to clone this logic and wait another
release for something that’s already well-defined and deterministic.
The mini-language is opinionated and belongs in contrib, but the
identifier infrastructure just solves a fundamental problem cleanly.

On the infrastructure patches (0001-0005): these look sensible. The
range table flattening info, elided node tracking, and append node
consolidation preserve information that's currently lost -- there's
some additional overhead to track this, but it's fixed per-relation
per-subquery, which seems reasonable.  The path generation hooks
(0005) are a clear improvement: moving from global enable_* GUCs to
per-RelOptInfo pgs_mask gives extensions the granularity they need for
relation-specific and join-specific decisions. Yes, you need C code to
use them, but you'd need to write C code to do something of value in
this area anyway, and the hooks give you control that GUCs can't
provide.

Overall, I'm supportive of getting these committed once they're ready.
contrib/pg_plan_advice is a compelling proof-of-concept for why these
hooks are needed.

I'll try to post more specific comments once I've read this some more.

--
Thanks, Amit Langote

  23. Re: pg_plan_advice

    Jakub Wartak <jakub.wartak@enterprisedb.com> — 2025-12-10T11:43:39Z

    On Fri, Dec 5, 2025 at 8:57 PM Robert Haas <robertmhaas@gmail.com> wrote:
[..]
> 014f9a831a320666bf2195949f41710f970c54ad removes the need for what was
> previously 0004, so here is a new patch series with that dropped, to
> avoid confusing cfbot or human reviewers.

Quick-question regarding cross-interactions of the extensions: would
it be possible for auto_explain to have something like
auto_explain.log_custom_options='PLAN_ADVICES' so that it could be
dumping the advice of the queries involved . I can see there is
ApplyExtensionExplainOption() and that would have to probably be used
by auto_explain(?) Or is there any other better way or perhaps it
somehow is against some design or it's just outside of initial scope?
This would solve two problems:
a) sometimes explaining manually (psql) is simply not realistic as it
is being run by app only
b) auto_explain could log nested queries and could print plan advices
along the way, which can be very painful process otherwise
(reverse-engineering how the optimizer would name things  in more
complex queries run from inside PLPGSQL functions)

BTW, some feedback: the plan advices (plan fixing) seems to work fine
for nested queries inside PLPGSQL, and also I've discovered (?) that
one can do even today with patchset the following:
   alter function blah(bigint) set pg_plan_advice.advice =
'NESTED_LOOP_MATERIALIZE(b)';
which seems to be pretty cool, because it allows more targeted fixes
without even having capability of fixing plans for specific query_id
(as discussed earlier).

For the generation part, the only remaining thing is how it integrates
with partitions (especially the ones being dynamically created/dropped
over time). Right now one needs to keep the advice(s) in sync after
altering the partitions, but it could be expected that some form of
regexp/partition-templating would be built into pg_plan_advices
instead. Anyway, I think this one should go into documentation just as
known-limitations for now.

While scratching my head on how to prove that this is not crashing
I've also checked below ones (TLDR all ok):
1. PG_TEST_INITDB_EXTRA_OPTS="-c
shared_preload_libraries='pg_plan_advice'"  meson test  # It was clean
2. PG_TEST_INITDB_EXTRA_OPTS="-c
shared_preload_libraries='pg_plan_advice'" PGOPTIONS="-c
pg_plan_advice.advice=NESTED_LOOP_MATERIALIZE(certainlynotused)" meson
test # This had several failures, but all is OK: it's just some of
them had to additional (expected) text inside regression.diffs:
NESTED_LOOP_MATERIALIZE(certainlynotused) /* not matched */
3. PG_TEST_INITDB_EXTRA_OPTS="-c
shared_preload_libraries='pg_plan_advice' -c
pg_plan_advice.shared_collection_limit=42"  meson test # It was clean
too

-J.

  24. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2025-12-10T13:33:11Z

    On Wed, Dec 10, 2025 at 6:43 AM Jakub Wartak
<jakub.wartak@enterprisedb.com> wrote:
> Quick-question regarding cross-interactions of the extensions: would
> it be possible for auto_explain to have something like
> auto_explain.log_custom_options='PLAN_ADVICES' so that it could be
> dumping the advice of the queries involved

Yes, I had the same idea. I think the tricky part here is that an
option can have an argument. Most options will probably have Boolean
arguments, but there are existing in-core counterexamples, such as
FORMAT. We should try to figure out the GUC in such a way that it can
be used either to set a Boolean option by just specifying it, or that
it can be used to set an option to a value by writing both. Maybe it's
fine if the GUC value is just a comma-separated list of entries, and
each entry can either be an option name or an option name followed by
a space followed by an option value, i.e. if FORMAT were custom, then
you could write auto_explain.log_custom_options='format xml,
plan_advice' or auto_explain.log_custom_options='plan_advice true,
range_table false' and have sensible things happen. In fact, very
possibly the GUC should just accept any options whether in-core or
out-of-core and not distinguish, so it would be more like
auto_explain.log_options.

> BTW, some feedback: the plan advices (plan fixing) seems to work fine
> for nested queries inside PLPGSQL, and also I've discovered (?) that
> one can do even today with patchset the following:
>    alter function blah(bigint) set pg_plan_advice.advice =
> 'NESTED_LOOP_MATERIALIZE(b)';
> which seems to be pretty cool, because it allows more targeted fixes
> without even having capability of fixing plans for specific query_id
> (as discussed earlier).

Yes, this is a big advantage of reusing the GUC machinery for this
purpose (but see the thread on "[PATCH] Allow complex data for GUC
extra").

> For the generation part, the only remaining thing is how it integrates
> with partitions (especially the ones being dynamically created/dropped
> over time). Right now one needs to keep the advice(s) in sync after
> altering the partitions, but it could be expected that some form of
> regexp/partition-templating would be built into pg_plan_advices
> instead. Anyway, I think this one should go into documentation just as
> known-limitations for now.

Right. I don't think trying to address this at this stage makes sense.
To maintain my sanity, I want to focus for now only on things that
round-trip: that is, we can generate it, and then we can accept that
same stuff. If we're using a parallel plan for every partition e.g.
they are all sequential scans or all index scans, we could generate
SEQ_SCAN(foo/*) or similar and then we could accept that. But figuring
that out would take a bunch of additional infrastructure that I don't
have the time or energy to create right this minute, and I don't see
it as anywhere close to essential for v1. Some other problems here:

1. What happens when a small number of partitions are different? The
code puts quite a bit of energy into detecting conflicting advice, and
honestly probably should put even more, and you might say, well, if
there's just one partition that used an index scan, then I still want
the advice to read SEQ_SCAN(foo/*) INDEX_SCAN(foo/foo23 foo23_a_idx)
and not signal a conflict, but that's slightly unprincipled.

2. INDEX_SCAN() specifications and similar will tend not to be
different for every partition because the index names will be
different for every partition. You might want something that says "for
each partition of foo, use the index on that partition that is a child
of this index on the parent".

Long run, there's a lot of things that can be added to this to make it
more concise (and more expressive, too). Another similar idea is to
have something like NO_GATHER_UNLESS_I_SAID_SO() so that a
non-parallel query doesn't have to do NO_GATHER(every single relation
including all the partitions). I'm pretty sure this is a valuable
idea, but, again, it's not essential for v1.

> While scratching my head on how to prove that this is not crashing
> I've also checked below ones (TLDR all ok):
> 1. PG_TEST_INITDB_EXTRA_OPTS="-c
> shared_preload_libraries='pg_plan_advice'"  meson test  # It was clean
> 2. PG_TEST_INITDB_EXTRA_OPTS="-c
> shared_preload_libraries='pg_plan_advice'" PGOPTIONS="-c
> pg_plan_advice.advice=NESTED_LOOP_MATERIALIZE(certainlynotused)" meson
> test # This had several failures, but all is OK: it's just some of
> them had to additional (expected) text inside regression.diffs:
> NESTED_LOOP_MATERIALIZE(certainlynotused) /* not matched */
> 3. PG_TEST_INITDB_EXTRA_OPTS="-c
> shared_preload_libraries='pg_plan_advice' -c
> pg_plan_advice.shared_collection_limit=42"  meson test # It was clean
> too

You can set pg_plan_advice.always_explain_supplied_advice=false to
clean up some of the noise here. This kind of testing is why I
invented that option. I think that in production, we REALLY REALLY
want any supplied advice to show up in the EXPLAIN plan even if the
user did not specify the PLAN_ADVICE option to EXPLAIN. Otherwise,
understanding what is going on with an EXPLAIN plan that a
hypothetical customer sends to a hypothetical PostgreSQL expert who
has to support said hypothetical customer will be a miserable
experience. But for testing purposes, it's nice to be able to shut it
off so you don't get random regression diffs.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  25. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2025-12-10T14:54:13Z

    On Wed, Dec 10, 2025 at 6:20 AM Amit Langote <amitlangote09@gmail.com> wrote:
> These are just high-level comments after browsing the patches and
> reading some bits like pgpa_identifier to get myself familiarized with
> the project.  I like that the key concept here is plan stability
> rather than plan control, because that framing makes it easier to
> treat this as infrastructure instead of policy.

Thanks, I agree. I'm sure people will use this for plan control, but
if you start with that, then it's really unclear what things you
should allow to be controlled and what things not. Defining the focus
as plan stability makes round-trip safety a priority and the scope of
what you can request is what the planner could have generated had the
costing come out just so. There's still some definitional questions at
the margin, but IMHO it's much less fuzzy.

> On the relation identifier system: IMHO this part doesn't seem as
> opinionated as the advice mini-language. The requirements pretty much
> dictate the design -- you need alias names and occurrence counters to
> handle self-joins, partition fields for partitioned tables, and a
> string representation to survive dump/restore. There doesn't seem to
> be much flexibility in that.

Right. There's some flexibility. For instance, you could handle
partitions using occurrence numbers, which would actually save a bunch
of code, but that seems obviously worse in terms of user experience.
Also, you could if you wanted key it off of the name of the table
rather than the relation alias used for the table. I think that's also
worse but possibly it's debatable. You could change the order of the
pieces in the representation; e.g. maybe plan_name should come first
rather than last; or you could change the separator characters. But,
honestly, none of that strikes me as sufficient grounds to want
multiple implementations. If the choices I've made don't seem good to
other people, then we should just change them and hopefully find
something everybody can live with. It's a bit like the way that
extension SQL scripts use "--" as a separator: maybe not everybody
agrees that this is the absolutely most elegant choice, but nobody's
proposing a a second version of the extension mechanism just to do
something different.

> Given that, it seems more practical to put this in core from the
> start. Extensions that might want to build plan-advice-like
> functionality shouldn’t have to clone this logic and wait another
> release for something that’s already well-defined and deterministic.
> The mini-language is opinionated and belongs in contrib, but the
> identifier infrastructure just solves a fundamental problem cleanly.

It's not quite as easy to make a sharp distinction between these
things as someone might hope. Note that the lexer and parser handle
the whole mini-language, which includes parsing the relation
identifiers. That doesn't of course mean that the code to *generate*
relation identifiers couldn't be in core, and I actually had it that
way at one point, but it's not very much code and I wasn't too
impressed with how that turned out. It seemed to couple the core code
to the extension more tightly than necessary for not much real
benefit.

But that's not to say I disagree with you categorically. Suppose we
decided (and I'm not saying we should) to start showing relation
identifiers in EXPLAIN output instead of identifying things in EXPLAIN
output as we do today. Maybe we even decide to show elided subqueries
and similar as first-class parts of the EXPLAIN output, also using
relation identifier syntax. That would be a pretty significant change,
and would destabilize a WHOLE LOT of regression test outputs, but then
relation identifiers become a first-class PostgreSQL concept that
everyone who looks at EXPLAIN output will encounter and, probably,
come to understand. Then obviously the relation identifier generation
code needs to be in core, and that makes total sense because we're
actually using it for something, and arguably we've made life easier
for everyone who wants to use pg_plan_advice in the future because
they're already familiar with the identifier convention. The downside
is everyone has to get used to the new EXPLAIN output even if they
don't care about pg_plan_advice or hate it with a fiery passion.

So my point here is that there are things we can decide to do to make
some or all of this "core," but IMHO it's not just as simple as saying
"this is in, that's out". It's more about deciding what the end state
ought to look like, and how integrated this stuff ought to be into the
fabric of PostgreSQL. I started with the minimal level of integration:
little pieces of core infrastructure, all used by a giant extension.
Now we need to either decide that's where we want to settle, or decide
to push to some greater or lesser degree toward more integration.

> On the infrastructure patches (0001-0005): these look sensible. The
> range table flattening info, elided node tracking, and append node
> consolidation preserve information that's currently lost -- there's
> some additional overhead to track this, but it's fixed per-relation
> per-subquery, which seems reasonable.  The path generation hooks
> (0005) are a clear improvement: moving from global enable_* GUCs to
> per-RelOptInfo pgs_mask gives extensions the granularity they need for
> relation-specific and join-specific decisions. Yes, you need C code to
> use them, but you'd need to write C code to do something of value in
> this area anyway, and the hooks give you control that GUCs can't
> provide.
>
> Overall, I'm supportive of getting these committed once they're ready.
> contrib/pg_plan_advice is a compelling proof-of-concept for why these
> hooks are needed.

Great. I don't think there's anything terribly controversial in
0001-0004. I think the comments and so on might need improving and
there could be little mini-bugs or whatever, but basically I think
they work and I don't anticipate any major problems. However, I'd want
at least one other person to do a detailed review before committing
anything. 0005 might be a little more controversial. There's some
design choices to dislike (though I believe I've made them for good
reason) and there's a question of whether it's as complete as we want.
It might be fine to commit it the way it is and just adjust it later
if we find that something ought to be different, but it's also
possible that we should think harder about some of the choices or hold
off for a bit while other parts of this effort move forward. I'm happy
to hear opinions on the best strategy here.

> I'll try to post more specific comments once I've read this some more.

Thanks for the review so far, and that sounds great!

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  26. Re: pg_plan_advice

    Corey Huinker <corey.huinker@gmail.com> — 2025-12-10T21:09:24Z

    On Wed, Dec 10, 2025 at 9:54 AM Robert Haas <robertmhaas@gmail.com> wrote:

> On Wed, Dec 10, 2025 at 6:20 AM Amit Langote <amitlangote09@gmail.com>
> wrote:
> > These are just high-level comments after browsing the patches and
> > reading some bits like pgpa_identifier to get myself familiarized with
> > the project.  I like that the key concept here is plan stability
> > rather than plan control, because that framing makes it easier to
> > treat this as infrastructure instead of policy.
>
> Thanks, I agree. I'm sure people will use this for plan control, but
> if you start with that, then it's really unclear what things you
> should allow to be controlled and what things not. Defining the focus
> as plan stability makes round-trip safety a priority and the scope of
> what you can request is what the planner could have generated had the
> costing come out just so. There's still some definitional questions at
> the margin, but IMHO it's much less fuzzy.
>

I couldn't have said this any better than Amit did. In my experience, lack
of a plan stability feature is far and away the most cited reason for not
porting to PostgreSQL. They want query plan stability first and foremost.
The amount of plan tweaking they do is actually pretty minimal, once they
get good-enough performance during user acceptance they want to encase
those query plans in amber because that's what the customer signed-off on.
After that, they're happy to scan the performance trendlines, and only make
tweaks when it's worth a change request.

But that's not to say I disagree with you categorically. Suppose we
> decided (and I'm not saying we should) to start showing relation
> identifiers in EXPLAIN output instead of identifying things in EXPLAIN
> output as we do today. Maybe we even decide to show elided subqueries
> and similar as first-class parts of the EXPLAIN output, also using
> relation identifier syntax. That would be a pretty significant change,
> and would destabilize a WHOLE LOT of regression test outputs, but then
> relation identifiers become a first-class PostgreSQL concept that
> everyone who looks at EXPLAIN output will encounter and, probably,
> come to understand.


I think the change would be worth the destabilization, because it makes it
so much easier to talk about complex query plans. Additionally, it would
make it reasonable to programmatically extract portions of a plan, allowing
for much more fine-grained regression tests regarding plans.

Showing the elided subqueries would be a huge benefit, outlining the
benefits that the planner is giving you "for free".


> > On the infrastructure patches (0001-0005): these look sensible. The
> > range table flattening info, elided node tracking, and append node
>

One thing I am curious about is that by tracking the elided nodes, would it
make more sense in the long run to have the initial post-naming plan tree
be immutable, and generate a separate copy minus the elided parts?

  27. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2025-12-10T21:29:50Z

    On Wed, Dec 10, 2025 at 4:09 PM Corey Huinker <corey.huinker@gmail.com> wrote:
> I think the change would be worth the destabilization, because it makes it so much easier to talk about complex query plans. Additionally, it would make it reasonable to programmatically extract portions of a plan, allowing for much more fine-grained regression tests regarding plans.

I'll wait for more votes before thinking about doing anything about
this, because I have my doubts about whether the consensus will
actually go in favor of such a large change. Or maybe someone else
would like to try mocking it up (even if somewhat imperfectly) so we
can all see just how large an impact it makes.

>> > On the infrastructure patches (0001-0005): these look sensible. The
>> > range table flattening info, elided node tracking, and append node
>
> One thing I am curious about is that by tracking the elided nodes, would it make more sense in the long run to have the initial post-naming plan tree be immutable, and generate a separate copy minus the elided parts?

Probably not. Having two entire copies of the plan tree would be
pretty expensive. I think that we've bet on the right idea, namely,
that the primary consumer of plan trees should be the executor, and
the primary goal should be to create plan trees that make the executor
run fast. I believe the right approach is basically what we do today:
you're allowed to put things into the plan that aren't technically
necessary for execution, if they're useful for instrumentation and
observability purposes and they don't add an unreasonable amount of
overhead. These patches basically just extend that existing principle
to a few new things.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  28. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2025-12-11T15:09:47Z

    On Fri, Dec 5, 2025 at 2:57 PM Robert Haas <robertmhaas@gmail.com> wrote:
> 014f9a831a320666bf2195949f41710f970c54ad removes the need for what was
> previously 0004, so here is a new patch series with that dropped, to
> avoid confusing cfbot or human reviewers.

Here's v6, with minor improvements over v5.

0001: Unchanged.

0002, 0003: Unchanged except for typo fixes pointed out by reviewers.

0004: I've improved the hook placement, which was previously such as
to make correct unique-semijoin handling impossible, and I improved
the associated comment about how to use the hook, based on experience
trying to actually do so.

0005: Fixed a small bug related to unique-semijoin handling (other
problems remain). Tidied things up to avoid producing non-actionable
NO_GATHER() advice in a number of cases, per some off-list feedback
from Ajaykumar Pal.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  29. Re: pg_plan_advice

    Jacob Champion <jacob.champion@enterprisedb.com> — 2025-12-12T01:11:09Z

    On Tue, Dec 9, 2025 at 11:46 AM Robert Haas <robertmhaas@gmail.com> wrote:
> By the way, if your fuzzer can also
> produces some things to add contrib/pg_plan_advice/sql for cases like
> this, that would be quite helpful. Ideally I would have caught this
> with a manually-written test case, but obviously that didn't happen.

Sure! (They'll need to be golfed down.) Here are three entries that
hit the crash, each on its own line:

> join_order(qoe((nested_l oindex_scanp_plain))se(nested_loop_plain)nested_loo/_pseq_scanlain)
> join_order(qoe((nested_loop_plain))se(nested_loop_plain)nesemij/insted_loop_plain)
> gather(gather(gar(g/ther0))gtaher(gathethga))

Something the fuzzer really likes is zero-length identifiers ("").
Maybe that's by design, but I thought I'd mention it since the
standard lexer doesn't allow that and syntax.sql doesn't exercise it.

> > It doesn't know that area is guaranteed to be non-NULL, so it can't
> > prove that ca_pointer is initialized.
>
> I don't know what to do about that. I can understand why it might be
> unable to prove that, but I don't see an obvious way to change the
> code that would make life easier. I could add Assert(area != NULL)
> before the call to pgpa_make_collected_advice() if that helps.

With USE_ASSERT_CHECKING, that should help, but I'm not sure if it
does without. (I could have sworn there was a conversation about that
at some point but I can't remember any of the keywords.) Could also
just make a dummy assignment. Or tag pg_plan_advice_dsa_area() with
__attribute__((returns_nonnull)), but that's more portability work.

--Jacob

  30. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2025-12-12T17:36:17Z

    On Thu, Dec 11, 2025 at 8:11 PM Jacob Champion
<jacob.champion@enterprisedb.com> wrote:
> Sure! (They'll need to be golfed down.) Here are three entries that
> hit the crash, each on its own line:
>
> > join_order(qoe((nested_l oindex_scanp_plain))se(nested_loop_plain)nested_loo/_pseq_scanlain)
> > join_order(qoe((nested_loop_plain))se(nested_loop_plain)nesemij/insted_loop_plain)
> > gather(gather(gar(g/ther0))gtaher(gathethga))

At least for me, setting pg_plan_advice.advice to any of these strings
does not provoke a crash. What I discovered after a bit of
experimentation is that you get the crash if you (a) set the string to
something like this and then (b) run an EXPLAIN. Turns out, I already
had a test in syntax.sql that is sufficient to provoke the crash, so,
locally, I've added 'EXPLAIN SELECT 1' after each test case in
syntax.sql that is expected to successfully alter the value of the
GUC.

> Something the fuzzer really likes is zero-length identifiers ("").
> Maybe that's by design, but I thought I'd mention it since the
> standard lexer doesn't allow that and syntax.sql doesn't exercise it.

That's not by design. I've added a matching error check locally.

> > > It doesn't know that area is guaranteed to be non-NULL, so it can't
> > > prove that ca_pointer is initialized.
> >
> > I don't know what to do about that. I can understand why it might be
> > unable to prove that, but I don't see an obvious way to change the
> > code that would make life easier. I could add Assert(area != NULL)
> > before the call to pgpa_make_collected_advice() if that helps.
>
> With USE_ASSERT_CHECKING, that should help, but I'm not sure if it
> does without. (I could have sworn there was a conversation about that
> at some point but I can't remember any of the keywords.) Could also
> just make a dummy assignment. Or tag pg_plan_advice_dsa_area() with
> __attribute__((returns_nonnull)), but that's more portability work.

As in initialize ca_pointer to InvalidDsaPointer?

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  31. Re: pg_plan_advice

    Jacob Champion <jacob.champion@enterprisedb.com> — 2025-12-12T18:09:44Z

    On Fri, Dec 12, 2025 at 9:36 AM Robert Haas <robertmhaas@gmail.com> wrote:
> At least for me, setting pg_plan_advice.advice to any of these strings
> does not provoke a crash. What I discovered after a bit of
> experimentation is that you get the crash if you (a) set the string to
> something like this and then (b) run an EXPLAIN.

Makes sense (this fuzzer was exercising pgpa_format_advice_target()).

> > With USE_ASSERT_CHECKING, that should help, but I'm not sure if it
> > does without. (I could have sworn there was a conversation about that
> > at some point but I can't remember any of the keywords.) Could also
> > just make a dummy assignment. Or tag pg_plan_advice_dsa_area() with
> > __attribute__((returns_nonnull)), but that's more portability work.
>
> As in initialize ca_pointer to InvalidDsaPointer?

Yeah.

Next bit of fuzzer feedback: I need the following diff in
pgpa_trove_add_to_hash() to avoid a crash when the hashtable starts to
fill up:

>     element = pgpa_trove_entry_insert(hash, key, &found);
> +   if (!found)
> +       element->indexes = NULL;
>     element->indexes = bms_add_member(element->indexes, index);

The advice string that triggered this is horrific, but I can send it
to you offline if you're morbidly curious. (I can spend time to
minimize it or I can get more fuzzer coverage, and I'd rather do the
latter right now :D)

--Jacob

  32. Re: pg_plan_advice

    Ajay Pal <ajay.pal.k@gmail.com> — 2025-12-15T06:30:43Z

    During further testing of the plan_advice patch's latest version, I
observed that the following query is generating a no_gather plan. This
specific plan structure is not being accepted by the query planner.

postgres=*# set local pg_plan_advice.advice='NO_GATHER("*RESULT*")';
SET
postgres=*# explain ( plan_advice) SELECT
CAST('99999999999999999999999999999999999999999999999999' AS NUMERIC);
                QUERY PLAN
-------------------------------------------
 Result  (cost=0.00..0.01 rows=1 width=32)
 Supplied Plan Advice:
   NO_GATHER("*RESULT*") /* not matched */
 Generated Plan Advice:
   NO_GATHER("*RESULT*")
(5 rows)

Thanks
Ajay

On Fri, Dec 12, 2025 at 11:40 PM Jacob Champion
<jacob.champion@enterprisedb.com> wrote:
>
> On Fri, Dec 12, 2025 at 9:36 AM Robert Haas <robertmhaas@gmail.com> wrote:
> > At least for me, setting pg_plan_advice.advice to any of these strings
> > does not provoke a crash. What I discovered after a bit of
> > experimentation is that you get the crash if you (a) set the string to
> > something like this and then (b) run an EXPLAIN.
>
> Makes sense (this fuzzer was exercising pgpa_format_advice_target()).
>
> > > With USE_ASSERT_CHECKING, that should help, but I'm not sure if it
> > > does without. (I could have sworn there was a conversation about that
> > > at some point but I can't remember any of the keywords.) Could also
> > > just make a dummy assignment. Or tag pg_plan_advice_dsa_area() with
> > > __attribute__((returns_nonnull)), but that's more portability work.
> >
> > As in initialize ca_pointer to InvalidDsaPointer?
>
> Yeah.
>
> Next bit of fuzzer feedback: I need the following diff in
> pgpa_trove_add_to_hash() to avoid a crash when the hashtable starts to
> fill up:
>
> >     element = pgpa_trove_entry_insert(hash, key, &found);
> > +   if (!found)
> > +       element->indexes = NULL;
> >     element->indexes = bms_add_member(element->indexes, index);
>
> The advice string that triggered this is horrific, but I can send it
> to you offline if you're morbidly curious. (I can spend time to
> minimize it or I can get more fuzzer coverage, and I'd rather do the
> latter right now :D)
>
> --Jacob
>
>

  33. Re: pg_plan_advice

    Jacob Champion <jacob.champion@enterprisedb.com> — 2025-12-15T16:37:43Z

    On Fri, Dec 12, 2025 at 10:09 AM Jacob Champion
<jacob.champion@enterprisedb.com> wrote:
> Next bit of fuzzer feedback:

And another bit, but this time I was able to minimize into a
regression case, attached.

This comment in pgpa_identifier_matches_target() seems to be incorrect:

>     /*
>      * The identifier must specify a schema, but the target may leave the
>      * schema NULL to match anything.
>      */

But I don't know whether that's because the assumption itself is
wrong, or because a layer above hasn't filtered something out before
getting to this point.

--Jacob

  34. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2025-12-15T20:06:08Z

    Here's v7.

In 0001, I removed "const" from a node's struct declaration, because
Tom gave me some feedback to avoid that on another recent patch, and I
noticed I had done it here also. 0002, 0003, and 0004 are unchanged.

In 0005:

- Refactored the code to avoid issuing SEMIJOIN_NON_UNIQUE() advice in
cases where uniqueness wasn't actually considered.
- Adjusted the code not to issue NO_GATHER() advice for non-relation
RTEs. (This is the issue reported by Ajay Pal in a recent message to
this thread, which was also mentioned in an XXX in the code.)
- Reject zero-length delimited identifiers, per Jacob's email.
- Properly initialize element->indexes in pgpa_trove_add_to_hash, per
Jacob'e email.
- Add gather((d d/d.d)) test case, per Jacob, and fix the related bug
in pgpa_identifier_matches_target, per Jacob's email.
- Add EXPLAIN SELECT 1 after various test cases in syntax.sql, to
improve test coverage, per analysis of why the existing test case
didn't catch a bug previously reported by Jacob.
- Added a dummy initialization to pgpa_collector.c to placate nervous
compilers, per discussion with Jacob.

I think this mostly catches me up on responding to issues reported
here, although there is one thing reported to me off-list that I
haven't dealt with yet. If there's anything reported on thread that
isn't addressed here, let me know.

Thanks,

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  35. Re: pg_plan_advice

    Jakub Wartak <jakub.wartak@enterprisedb.com> — 2025-12-17T10:12:40Z

    On Mon, Dec 15, 2025 at 9:06 PM Robert Haas <robertmhaas@gmail.com> wrote:
>
> Here's v7.
[..]

OK, so I've tested today from Your's branch directly, so I hope that
was also v7. Given the following q20 query:

SELECT s_name, s_address
FROM supplier, nation
WHERE s_suppkey in
    (SELECT ps_suppkey
     FROM partsupp
     WHERE ps_partkey in
         (SELECT p_partkey
          FROM part
          WHERE p_name LIKE 'forest%' )
       AND ps_availqty >
         (SELECT 0.5 * sum(l_quantity)
          FROM lineitem
          WHERE l_partkey = ps_partkey
            AND l_suppkey = ps_suppkey
            AND l_shipdate >= DATE '1994-01-01'
            AND l_shipdate < DATE '1994-01-01' + INTERVAL '1' year ) )
  AND s_nationkey = n_nationkey
  AND n_name = 'CANADA'
ORDER BY s_name;

in normal conditions (w/o advice) the above query generates:

 Sort  (cost=1010985030.44..1010985030.59 rows=61 width=51)
   Sort Key: supplier.s_name
   ->  Nested Loop  (cost=0.42..1010985028.63 rows=61 width=51)
         Join Filter: (nation.n_nationkey = supplier.s_nationkey)
         ->  Seq Scan on nation  (cost=0.00..1.31 rows=1 width=4)
               Filter: (n_name = 'CANADA'::bpchar)
         ->  Nested Loop Semi Join  (cost=0.42..1010985008.29
rows=1522 width=55)
               Join Filter: (partsupp.ps_suppkey = supplier.s_suppkey)
               ->  Seq Scan on supplier  (cost=0.00..249.30 rows=7730 width=59)
               ->  Materialize  (cost=0.42..1010755994.57 rows=1973 width=4)
                     ->  Nested Loop  (cost=0.42..1010755984.71
rows=1973 width=4)
                           ->  Seq Scan on part  (cost=0.00..4842.25
rows=1469 width=4)
                                 Filter: ((p_name)::text ~~ 'forest%'::text)
                           ->  Index Scan using pk_partsupp on
partsupp  (cost=0.42..688053.87 rows=1 width=8)
                                 Index Cond: (ps_partkey = part.p_partkey)
                                 Filter: ((ps_availqty)::numeric >
(SubPlan expr_1))
                                 SubPlan expr_1
                                   ->  Aggregate
(cost=172009.42..172009.44 rows=1 width=32)
                                         ->  Seq Scan on lineitem
(cost=0.00..172009.42 rows=1 width=5)
                                               Filter: ((l_shipdate >=
'1994-01-01'::date) AND (l_shipdate < '1995-01-01 00:00:00'::timestamp
without time zone) AND (l_partkey = partsupp.ps_partkey) AND
(l_suppkey = partsupp.ps_suppkey))


 Generated Plan Advice:
   JOIN_ORDER(nation (supplier (part partsupp)))
   NESTED_LOOP_PLAIN(partsupp partsupp) <--- [X]
   NESTED_LOOP_MATERIALIZE(partsupp)
   SEQ_SCAN(nation supplier part lineitem@expr_1)
   INDEX_SCAN(partsupp public.pk_partsupp)
   SEMIJOIN_NON_UNIQUE((partsupp part))
   NO_GATHER(supplier nation partsupp part lineitem@expr_1)

Please see the - I think it's confusing? -
NESTED_LOOP_MATERIALIZE(partsupp partsupp) - that's 2x the same
string? This causes it to turn into below plan -- I've marked the
problem with [X]

 Sort  (cost=50035755.50..50035755.66 rows=61 width=51)
   Sort Key: supplier.s_name
   ->  Nested Loop  (cost=12562154.32..50035753.70 rows=61 width=51)
         Join Filter: (nation.n_nationkey = supplier.s_nationkey)
         ->  Seq Scan on nation  (cost=0.00..1.31 rows=1 width=4)
               Filter: (n_name = 'CANADA'::bpchar)
         ->  Nested Loop Semi Join  (cost=12562154.32..50035733.36
rows=1522 width=55)
             [X] -- missing Join Filter here
               ->  Seq Scan on supplier  (cost=0.00..249.30 rows=7730 width=59)
               [X] -- HJ instead of Materialize+Nested Loop below:
               ->  Hash Join  (cost=12562154.32..12567002.09 rows=1 width=4)
                     Hash Cond: (part.p_partkey = partsupp.ps_partkey)
                     ->  Seq Scan on part  (cost=0.00..4842.25
rows=1469 width=4)
                           Filter: ((p_name)::text ~~ 'forest%'::text)
                     ->  Hash  (cost=12562154.02..12562154.02 rows=24 width=8)
                           ->  Index Scan using pk_partsupp on
partsupp  (cost=0.42..12562154.02 rows=24 width=8)
                                 [X] -- wrong Index Cond below
(suppkey instead of partkey)
                                 Index Cond: (ps_suppkey = supplier.s_suppkey)
                                 Filter: ((ps_availqty)::numeric >
(SubPlan expr_1))
                                 SubPlan expr_1
                                   ->  Aggregate
(cost=172009.42..172009.44 rows=1 width=32)
                                         ->  Seq Scan on lineitem
(cost=0.00..172009.42 rows=1 width=5)
                                               Filter: ((l_shipdate >=
'1994-01-01'::date) AND (l_shipdate < '1995-01-01 00:00:00'::timestamp
without time zone) AND (l_partkey = partsupp.ps_partkey) AND
(l_suppkey = partsupp.ps_suppkey))

Supplied Plan Advice:
   SEQ_SCAN(nation) /* matched */
   SEQ_SCAN(supplier) /* matched */
   SEQ_SCAN(part) /* matched */
   SEQ_SCAN(lineitem@expr_1) /* matched */
   INDEX_SCAN(partsupp public.pk_partsupp) /* matched */
   JOIN_ORDER(nation (supplier (part partsupp))) /* matched, conflicting */
   NESTED_LOOP_PLAIN(partsupp) /* matched, conflicting */
   NESTED_LOOP_PLAIN(partsupp) /* matched, conflicting */
   NESTED_LOOP_MATERIALIZE(partsupp) /* matched, conflicting, failed */
   SEMIJOIN_NON_UNIQUE((partsupp part)) /* matched, conflicting */
   NO_GATHER(supplier) /* matched */
   NO_GATHER(nation) /* matched */
   NO_GATHER(partsupp) /* matched */
   NO_GATHER(part) /* matched */
   NO_GATHER(lineitem@expr_1) /* matched */

So the difference is basically between:
    set pg_plan_advice.advice = '[..] NESTED_LOOP_PLAIN(partsupp
partsupp) NESTED_LOOP_MATERIALIZE(partsupp) [..]';
which causes wrong plan and outcome:
    NESTED_LOOP_MATERIALIZE(partsupp) /* matched, conflicting, failed */

and apparently proper advice like below which has better yield:
    set pg_plan_advice.advice = '[..] NESTED_LOOP_PLAIN(part partsupp)
NESTED_LOOP_MATERIALIZE(partsupp) [..]';
which is not generated , but caused good plan, however it also prints:
   NESTED_LOOP_PLAIN(part) /* matched, conflicting, failed */
   NESTED_LOOP_MATERIALIZE(partsupp) /* matched, conflicting */
but that seems "failed" there, seems to be untrue?

Another idea is perhaps, we could have some elog(WARNING) - but not
Asserts() - in assert-only enabled build that could alert us in case
of duplicated entries being detected for the same ops in
pg_plan_advice_explain_feedback()?

-J.

  36. Re: pg_plan_advice

    Jakub Wartak <jakub.wartak@enterprisedb.com> — 2025-12-17T13:44:02Z

    On Wed, Dec 17, 2025 at 11:12 AM Jakub Wartak
<jakub.wartak@enterprisedb.com> wrote:
>
> On Mon, Dec 15, 2025 at 9:06 PM Robert Haas <robertmhaas@gmail.com> wrote:
> >
> > Here's v7.
> [..]
>[..q20..]

OK, now for the q10:

 Sort
   Sort Key: (sum((lineitem.l_extendedprice * ('1'::numeric -
lineitem.l_discount)))) DESC
   ->  Finalize GroupAggregate
         Group Key: customer.c_custkey, nation.n_name
         ->  Gather Merge
               Workers Planned: 2
               ->  Partial GroupAggregate
                     Group Key: customer.c_custkey, nation.n_name
                     ->  Sort
                           Sort Key: customer.c_custkey, nation.n_name
                           ->  Hash Join
                                 Hash Cond: (customer.c_nationkey =
nation.n_nationkey)
                                 ->  Parallel Hash Join
                                       Hash Cond: (orders.o_custkey =
customer.c_custkey)
                                       ->  Nested Loop
                                             ->  Parallel Seq Scan on orders
                                                   Filter:
((o_orderdate >= '1993-10-01'::date) AND (o_orderdate < '1994-01-01
00:00:00'::timestamp without time zone))
                                             ->  Index Scan using
lineitem_l_orderkey_idx_l_returnflag on lineitem
                                                   Index Cond:
(l_orderkey = orders.o_orderkey)
                                       ->  Parallel Hash
                                             ->  Parallel Seq Scan on customer
                                 ->  Hash
                                       ->  Seq Scan on nation
 Generated Plan Advice:
   JOIN_ORDER(orders lineitem customer nation)
   NESTED_LOOP_PLAIN(lineitem)
   HASH_JOIN(customer nation)
   SEQ_SCAN(orders customer nation)
   INDEX_SCAN(lineitem public.lineitem_l_orderkey_idx_l_returnflag)
   GATHER_MERGE((customer orders lineitem nation))

but when set the advice it generates wrong NL instead of expected
Parallel HJ (so another way to fix is to simply disable PQ, yuck),
but:

 Sort
   Sort Key: (sum((lineitem.l_extendedprice * ('1'::numeric -
lineitem.l_discount)))) DESC
   ->  Finalize GroupAggregate
         Group Key: customer.c_custkey, nation.n_name
         ->  Gather Merge
               Workers Planned: 2
               ->  Partial GroupAggregate
                     Group Key: customer.c_custkey, nation.n_name
                     ->  Sort
                           Sort Key: customer.c_custkey, nation.n_name
                           ->  Nested Loop
                                 ->  Hash Join
                                       Hash Cond:
(customer.c_nationkey = nation.n_nationkey)
                                       ->  Parallel Hash Join
                                             Hash Cond:
(orders.o_custkey = customer.c_custkey)
                                             ->  Parallel Seq Scan on orders
                                                   Filter:
((o_orderdate >= '1993-10-01'::date) AND (o_orderdate < '1994-01-01
00:00:00'::timestamp without time zone))
                                             ->  Parallel Hash
                                                   ->  Parallel Seq
Scan on customer
                                       ->  Hash
                                             ->  Seq Scan on nation
                                 ->  Index Scan using
lineitem_l_orderkey_idx_l_returnflag on lineitem
                                       Index Cond: (l_orderkey =
orders.o_orderkey)
 Supplied Plan Advice:
   SEQ_SCAN(orders) /* matched */
   SEQ_SCAN(customer) /* matched */
   SEQ_SCAN(nation) /* matched */
   INDEX_SCAN(lineitem public.lineitem_l_orderkey_idx_l_returnflag) /*
matched */
   JOIN_ORDER(orders lineitem customer nation) /* matched,
conflicting, failed */
   NESTED_LOOP_PLAIN(lineitem) /* matched, conflicting */
   HASH_JOIN(customer) /* matched, conflicting */
   HASH_JOIN(nation) /* matched, conflicting */
   GATHER_MERGE((customer orders lineitem nation)) /* matched */

So to me it looks like in Generated Plan Advice we:
- have proper HASH_JOIN(customer nation)
- but it somehow forgot to include "HASH_JOIN(orders)" to cover for
that Parallel Hash Join on (orders.o_custkey = customer.c_custkey)
with input from NL. After adding that manually, it achieves the same
input plan properly.

Please let me know if I'm wrong, I was kind of thinking Parallel is
not fully supported, but README/tests seem to state otherwise.

-J.

  37. Re: pg_plan_advice

    Jakub Wartak <jakub.wartak@enterprisedb.com> — 2025-12-18T12:27:27Z

    On Wed, Dec 17, 2025 at 2:44 PM Jakub Wartak
<jakub.wartak@enterprisedb.com> wrote:
>
> On Wed, Dec 17, 2025 at 11:12 AM Jakub Wartak
> <jakub.wartak@enterprisedb.com> wrote:
> >
> > On Mon, Dec 15, 2025 at 9:06 PM Robert Haas <robertmhaas@gmail.com> wrote:
> > >
> > > Here's v7.
> > [..]
> >[..q20..]
>
> OK, now for the q10:

Hi, this is a follow-up just to the q10.

> So to me it looks like in Generated Plan Advice we:
> - have proper HASH_JOIN(customer nation)
> - but it somehow forgot to include "HASH_JOIN(orders)" to cover for
> that Parallel Hash Join on (orders.o_custkey = customer.c_custkey)
> with input from NL. After adding that manually, it achieves the same
> input plan properly.
[..]

Well, it's quite a ride with the Q10 and I partially wrong with above:

0. The reported earlier wrong missing "HASH_JOIN(orders customer)" -
that part was okay
1. The Incremental Sort is being used in the original plan, but is
still IS not reflected in the generated advice.
2a. I've noticed Memoize/Index Scan was not being respected for "nation"
2b. Seq scan for nation was being done for "nation"

So total modification list, I've ended up doing (+ for adding , - for removing):

+ HASH_JOIN(orders customer) -- from earlier reply
+ NESTED_LOOP_MEMOIZE(nation)
+ INDEX_SCAN(nation public.pk_nation)
- HASH_JOIN(customer nation) -- as it was we were having NL() in org plan
SEQ_SCAN(orders customer nation) ==> SEQ_SCAN(orders customer)

In full the best shape seems to be Q10 with pg_plan_advice.advice =
'HASH_JOIN(orders customer) JOIN_ORDER(orders lineitem customer
nation)    NESTED_LOOP_PLAIN(lineitem)    SEQ_SCAN(orders customer)
INDEX_SCAN(lineitem public.lineitem_l_orderkey_idx_l_returnflag)
GATHER_MERGE((customer orders lineitem nation))
NESTED_LOOP_MEMOIZE(nation)';

which yields:
 Sort
   Sort Key: (sum((lineitem.l_extendedprice * ('1'::numeric -
lineitem.l_discount)))) DESC
   ->  GroupAggregate
         Group Key: customer.c_custkey, nation.n_name
         ->  Gather Merge
               Workers Planned: 2
               ->  Sort
                     Sort Key: customer.c_custkey, nation.n_name
                     ->  Nested Loop
                           ->  Parallel Hash Join
                                 Hash Cond: (orders.o_custkey =
customer.c_custkey)
                                 ->  Nested Loop
                                       ->  Parallel Seq Scan on orders
                                             Filter: ((o_orderdate >=
'1993-10-01'::date) AND (o_orderdate < '1994-01-01
00:00:00'::timestamp without time zone))
                                       ->  Index Scan using
lineitem_l_orderkey_idx_l_returnflag on lineitem
                                             Index Cond: (l_orderkey =
orders.o_orderkey)
                                 ->  Parallel Hash
                                       ->  Parallel Seq Scan on customer
                           ->  Memoize
                                 Cache Key: customer.c_nationkey
                                 Cache Mode: logical
                                 ->  Index Scan using pk_nation on nation
                                       Index Cond: (n_nationkey =
customer.c_nationkey)

but that Incremental Sort *is* still missing. In original plan we are doing
   Incremental Sort (Sort Key: customer.c_custkey, nation.n_name,
Presorted Key: customer.c_custkey)
   <-- .... Sort(Sort Key: customer.c_custkey)

However, even with my overrides I haven't found an immediately obvious
way to force it to use Incremental Sort on a specific field, so it
just sorts on two at once. Maybe it's something that should be
expressed through GATHER_MERGE()?, but that's not obvious how and
where. In terms of raw performance , it seems to be very similiar
(98ms +/- 8ms even between those two).

-J.

  38. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2025-12-18T13:36:44Z

    On Wed, Dec 17, 2025 at 5:12 AM Jakub Wartak
<jakub.wartak@enterprisedb.com> wrote:
>  Sort  (cost=1010985030.44..1010985030.59 rows=61 width=51)
>    Sort Key: supplier.s_name
>    ->  Nested Loop  (cost=0.42..1010985028.63 rows=61 width=51)
>          Join Filter: (nation.n_nationkey = supplier.s_nationkey)
>          ->  Seq Scan on nation  (cost=0.00..1.31 rows=1 width=4)
>                Filter: (n_name = 'CANADA'::bpchar)
>          ->  Nested Loop Semi Join  (cost=0.42..1010985008.29
> rows=1522 width=55)
>                Join Filter: (partsupp.ps_suppkey = supplier.s_suppkey)
>                ->  Seq Scan on supplier  (cost=0.00..249.30 rows=7730 width=59)
>                ->  Materialize  (cost=0.42..1010755994.57 rows=1973 width=4)
>                      ->  Nested Loop  (cost=0.42..1010755984.71
> rows=1973 width=4)
>                            ->  Seq Scan on part  (cost=0.00..4842.25
> rows=1469 width=4)
>                                  Filter: ((p_name)::text ~~ 'forest%'::text)
>                            ->  Index Scan using pk_partsupp on
> partsupp  (cost=0.42..688053.87 rows=1 width=8)
>                                  Index Cond: (ps_partkey = part.p_partkey)
>                                  Filter: ((ps_availqty)::numeric >
> (SubPlan expr_1))
>                                  SubPlan expr_1
>                                    ->  Aggregate
> (cost=172009.42..172009.44 rows=1 width=32)
>                                          ->  Seq Scan on lineitem
> (cost=0.00..172009.42 rows=1 width=5)
>                                                Filter: ((l_shipdate >=
> '1994-01-01'::date) AND (l_shipdate < '1995-01-01 00:00:00'::timestamp
> without time zone) AND (l_partkey = partsupp.ps_partkey) AND
> (l_suppkey = partsupp.ps_suppkey))
>
>
>  Generated Plan Advice:
>    JOIN_ORDER(nation (supplier (part partsupp)))
>    NESTED_LOOP_PLAIN(partsupp partsupp) <--- [X]
>    NESTED_LOOP_MATERIALIZE(partsupp)
>    SEQ_SCAN(nation supplier part lineitem@expr_1)
>    INDEX_SCAN(partsupp public.pk_partsupp)
>    SEMIJOIN_NON_UNIQUE((partsupp part))
>    NO_GATHER(supplier nation partsupp part lineitem@expr_1)

Yeah, that's not right. There are three nested loops here, so we
should have three pieces of nested loop advice.
NESTED_LOOP_MATERIALIZE(partsupp) covers the innermost nested loop.
The other two are NESTED_LOOP_PLAIN, but the advice should cover all
the tables on the inner side of the join. I think it should read:

NESTED_LOOP_PLAIN((part partsupp) (supplier part partsupp))

Ordering isn't significant here, so NESTED_LOOP_PLAIN((part supplier
partsupp) (partsupp part)) would be logically equivalent. Doesn't
matter exactly what we output here, but it shouldn't be just partsupp.

> and apparently proper advice like below which has better yield:
>     set pg_plan_advice.advice = '[..] NESTED_LOOP_PLAIN(part partsupp)

This isn't quite what you want, because this says that part should be
on the outer side of a NESTED_LOOP_PLAIN by itself and partsupp should
also be on the outer side of a NESTED_LOOP_PLAIN by itself. You need
the extra set of parentheses to indicate that the join product of
those two tables should be on the outer side of a NESTED_LOOP_PLAIN,
rather than each table individually.

What must be happening here is that either pgpa_join.c (maybe with
complicity from pgpa_walker.c) is not populating the
pgpa_plan_walker_context's join_strategies[JSTRAT_NESTED_LOOP_PLAIN]
member correctly, or else pgpa_output.c is not serializing it to text
correctly. I suspect the former is a more likely but I'm not sure
exactly what's happening.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  39. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2025-12-18T20:39:13Z

    On Wed, Dec 17, 2025 at 8:44 AM Jakub Wartak
<jakub.wartak@enterprisedb.com> wrote:
> OK, now for the q10:
>
>  Sort
>    Sort Key: (sum((lineitem.l_extendedprice * ('1'::numeric -
> lineitem.l_discount)))) DESC
>    ->  Finalize GroupAggregate
>          Group Key: customer.c_custkey, nation.n_name
>          ->  Gather Merge
>                Workers Planned: 2
>                ->  Partial GroupAggregate
>                      Group Key: customer.c_custkey, nation.n_name
>                      ->  Sort
>                            Sort Key: customer.c_custkey, nation.n_name
>                            ->  Hash Join
>                                  Hash Cond: (customer.c_nationkey =
> nation.n_nationkey)
>                                  ->  Parallel Hash Join
>                                        Hash Cond: (orders.o_custkey =
> customer.c_custkey)
>                                        ->  Nested Loop
>                                              ->  Parallel Seq Scan on orders
>                                                    Filter:
> ((o_orderdate >= '1993-10-01'::date) AND (o_orderdate < '1994-01-01
> 00:00:00'::timestamp without time zone))
>                                              ->  Index Scan using
> lineitem_l_orderkey_idx_l_returnflag on lineitem
>                                                    Index Cond:
> (l_orderkey = orders.o_orderkey)
>                                        ->  Parallel Hash
>                                              ->  Parallel Seq Scan on customer
>                                  ->  Hash
>                                        ->  Seq Scan on nation
>  Generated Plan Advice:
>    JOIN_ORDER(orders lineitem customer nation)
>    NESTED_LOOP_PLAIN(lineitem)
>    HASH_JOIN(customer nation)
>    SEQ_SCAN(orders customer nation)
>    INDEX_SCAN(lineitem public.lineitem_l_orderkey_idx_l_returnflag)
>    GATHER_MERGE((customer orders lineitem nation))

This looks correct to me.

> but when set the advice it generates wrong NL instead of expected
> Parallel HJ (so another way to fix is to simply disable PQ, yuck),
> but:

This is obviously bad. I'm not quite sure what happened here, but my
guess is that something prevented the JOIN_ORDER advice from being
applied cleanly and then everything went downhill from there. I wonder
if JOIN_ORDER doesn't interact properly with incremental sorts --
that's a situation for which I don't think I have existing test
coverage.

> So to me it looks like in Generated Plan Advice we:
> - have proper HASH_JOIN(customer nation)
> - but it somehow forgot to include "HASH_JOIN(orders)" to cover for
> that Parallel Hash Join on (orders.o_custkey = customer.c_custkey)
> with input from NL. After adding that manually, it achieves the same
> input plan properly.

The first table in the JOIN_ORDER() specification isn't supposed to
have a join method specification, because the join method specifier
says what appears on the inner, i.e. second, arm of the join.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  40. Re: pg_plan_advice

    Haibo Yan <tristan.yim@gmail.com> — 2025-12-29T23:33:53Z

    Hi Robert,

Thank you very much for your work on the pg_plan_advice patch series. It is
an impressive and substantial contribution, and it seems like a meaningful
step forward toward addressing long-standing query plan stability issues in
PostgreSQL.

While reviewing the v7 patches, I noticed a few points that I wanted to
raise for discussion:
1. GEQO interaction (patch 4):
Since GEQO relies on randomized search, is there a risk that the optimizer
may fail to explore the specific join order or path that is being enforced
by the advice mask? In that case, could this lead to failures such as
inability to construct the required join relation or excessive planning
time if the desired path is not sampled?
2. Parallel query serialization (patches 1–3):
Several new fields (subrtinfos, elidedNodes, child_append_relid_sets) are
added to PlannedStmt, but I did not see corresponding changes in outfuncs.c
/ readfuncs.c. Without serialization support, parallel workers executing
subplans or Append nodes may not receive this metadata. Is this handled
elsewhere, or is it something still pending?
3. Alias handling when generating advice (patch 5):
In pgpa_output_relation_name, the advice string is generated using
get_rel_name(relid), which resolves to the underlying table name rather
than the RTE alias. In self-join cases this could be ambiguous (e.g.,
my_table vs my_table). Would it be more appropriate to use the RTE alias
when available?
4. Minor typo (patch 4):
In src/include/nodes/relation.h, parititonwise appears to be a typo and
should likely be partitionwise.

I hope these comments are helpful, and I apologize in advance if any of
this is already addressed elsewhere in the series.

Best regards,
Haibo

On Mon, Dec 15, 2025 at 12:06 PM Robert Haas <robertmhaas@gmail.com> wrote:

> Here's v7.
>
> In 0001, I removed "const" from a node's struct declaration, because
> Tom gave me some feedback to avoid that on another recent patch, and I
> noticed I had done it here also. 0002, 0003, and 0004 are unchanged.
>
> In 0005:
>
> - Refactored the code to avoid issuing SEMIJOIN_NON_UNIQUE() advice in
> cases where uniqueness wasn't actually considered.
> - Adjusted the code not to issue NO_GATHER() advice for non-relation
> RTEs. (This is the issue reported by Ajay Pal in a recent message to
> this thread, which was also mentioned in an XXX in the code.)
> - Reject zero-length delimited identifiers, per Jacob's email.
> - Properly initialize element->indexes in pgpa_trove_add_to_hash, per
> Jacob'e email.
> - Add gather((d d/d.d)) test case, per Jacob, and fix the related bug
> in pgpa_identifier_matches_target, per Jacob's email.
> - Add EXPLAIN SELECT 1 after various test cases in syntax.sql, to
> improve test coverage, per analysis of why the existing test case
> didn't catch a bug previously reported by Jacob.
> - Added a dummy initialization to pgpa_collector.c to placate nervous
> compilers, per discussion with Jacob.
>
> I think this mostly catches me up on responding to issues reported
> here, although there is one thing reported to me off-list that I
> haven't dealt with yet. If there's anything reported on thread that
> isn't addressed here, let me know.
>
> Thanks,
>
> --
> Robert Haas
> EDB: http://www.enterprisedb.com
>

  41. Re: pg_plan_advice

    Lukas Fittl <lukas@fittl.com> — 2025-12-30T01:15:18Z

    On Mon, Dec 15, 2025 at 12:06 PM Robert Haas <robertmhaas@gmail.com> wrote:
> Here's v7.

I'm excited about this patch series, and in an effort to help land the
infrastructure, here is a review of 0001 - 0003 to start:

For 0001, I'm not sure the following comment is correct:

> /* When recursing = true, it's an unplanned or dummy subquery. */
> rtinfo->dummy = recursing;

Later in that function we only recurse if its a dummy subquery - in the
case of an unplanned subquery (rel->subroot == NULL)
add_rtes_to_flat_rtable won't be called again (instead the relation RTEs
are directly added to the finalrtable). Maybe we can
clarify that comment as "When recursing = true, it's a dummy subquery or
its children.".

From my medium-level understanding of the planner, I don't think the lack
of tracking unplanned subqueries
in subrtinfos is a problem, at least for the pg_plan_advice type use cases.

---

For 0002:

It might be helpful to clarify in a comment that ElidedNode's plan_node_id
represents the surviving node, not that of the elided node.

I also noticed that this currently doesn't support cases where multiple
nodes are elided, e.g. with multi-level table partitioning:

CREATE TABLE pt (l1 date, l2 text) PARTITION BY RANGE (l1);
CREATE TABLE pt_202512 PARTITION OF pt FOR VALUES FROM ('2025-12-01') TO
('2026-01-01') PARTITION BY LIST (l2);
CREATE TABLE pt_202512_TEST PARTITION OF pt_202512 FOR VALUES IN ('TEST');

EXPLAIN (RANGE_TABLE) SELECT * FROM pt WHERE l1 = '2025-12-15' AND l2 =
'TEST';

                            QUERY PLAN
-------------------------------------------------------------------
 Seq Scan on pt_202512_test pt  (cost=0.00..29.05 rows=1 width=36)
   Filter: ((l1 = '2025-12-15'::date) AND (l2 = 'TEST'::text))
   Scan RTI: 3
   Elided Node Type: Append
   Elided Node RTIs: 1    <=== This is missing RTI 2
 RTI 1 (relation, inherited, in-from-clause):
   Relation: pt
 RTI 2 (relation, inherited, in-from-clause):
   Relation: pt_202512
 RTI 3 (relation, in-from-clause):
   Relation: pt_202512_test
 Unprunable RTIs: 1 2 3

In a quick test, adding child_append_relid_sets (from 0003) to the relids
being passed to record_elided_node fixes
that. Presumably the case of partitionwise join relids doesn't matter,
because that would prevent it being elided.

---

For 0003:

I also find the "cars" variable suffix a bit hard to understand, but not
sure a comment next to the variables is that useful.
Separately, the noise generated by all the additional "_cars" variables
isn't great.

I wonder a little bit if we couldn't introduce a better abstraction here,
e.g. a struct "AppendPathInput" that contains the
two related lists, and gets populated by
accumulate_append_subpath/get_singleton_append_subpath and then
passed to create_append_path as a single argument.

---

Note that 0005 needs a rebase,
since 48d4a1423d2e92d10077365532d92e059ba2eb2e changed the
GetNamedDSMSegment API.

You may also want to move the CF entry to the PG19-4 commitfest so CFbot
runs again.

Thanks,
Lukas

-- 
Lukas Fittl

  42. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-01-06T19:36:03Z

    On Mon, Dec 29, 2025 at 8:15 PM Lukas Fittl <lukas@fittl.com> wrote:
> For 0001, I'm not sure the following comment is correct:
>
> > /* When recursing = true, it's an unplanned or dummy subquery. */
> > rtinfo->dummy = recursing;
>
> Later in that function we only recurse if its a dummy subquery - in the case of an unplanned subquery (rel->subroot == NULL)
> add_rtes_to_flat_rtable won't be called again (instead the relation RTEs are directly added to the finalrtable). Maybe we can
> clarify that comment as "When recursing = true, it's a dummy subquery or its children.".

Presumably, a child of an unplanned or dummy subquery will also be
unplanned or dummy, so I'm not sure I understand the need to clarify
here.

> From my medium-level understanding of the planner, I don't think the lack of tracking unplanned subqueries
> in subrtinfos is a problem, at least for the pg_plan_advice type use cases.

I don't think so, either. I believe that anything that falls into this
category is something that is not actually going to be reflected in
the final plan tree, but we can't lose track of it completely because
it can matter for purposes like locking or invalidation. Since plan
advice only targets things that appear in the final plan tree, it
shouldn't care. If we did want to care, e.g. to emit advice like
WE_ARE_EXPECTING_THIS_TO_BE_DUMMY(whatever_table), we'd need more than
an rtoffset per subquery; we'd have to map each RTI individually,
because some RTIs are tossed completely for in the "dummy" case,
meaning that the rtoffset isn't constant for the whole subquery.
AFAICT, this is not an issue because we need not care about the dummy
subqueries at all. The only reason I included the SubPlanRTInfo at all
for this case is that the previous SubPlanRTInfo might be for a
non-dummy subquery, and some code might want to look at the next entry
in the list to see where the portion of the range table belonging to
that previous subqueries ends. This lets you do that.

> For 0002:
>
> It might be helpful to clarify in a comment that ElidedNode's plan_node_id represents the surviving node, not that of the elided node.

Good point. I'll add this comment:

+ *
+ * plan_node_id is that of the surviving plan node, the sole child of the
+ * one which was elided.

> I also noticed that this currently doesn't support cases where multiple nodes are elided, e.g. with multi-level table partitioning:
>
> CREATE TABLE pt (l1 date, l2 text) PARTITION BY RANGE (l1);
> CREATE TABLE pt_202512 PARTITION OF pt FOR VALUES FROM ('2025-12-01') TO ('2026-01-01') PARTITION BY LIST (l2);
> CREATE TABLE pt_202512_TEST PARTITION OF pt_202512 FOR VALUES IN ('TEST');
>
> EXPLAIN (RANGE_TABLE) SELECT * FROM pt WHERE l1 = '2025-12-15' AND l2 = 'TEST';
>
>                             QUERY PLAN
> -------------------------------------------------------------------
>  Seq Scan on pt_202512_test pt  (cost=0.00..29.05 rows=1 width=36)
>    Filter: ((l1 = '2025-12-15'::date) AND (l2 = 'TEST'::text))
>    Scan RTI: 3
>    Elided Node Type: Append
>    Elided Node RTIs: 1    <=== This is missing RTI 2
>  RTI 1 (relation, inherited, in-from-clause):
>    Relation: pt
>  RTI 2 (relation, inherited, in-from-clause):
>    Relation: pt_202512
>  RTI 3 (relation, in-from-clause):
>    Relation: pt_202512_test
>  Unprunable RTIs: 1 2 3
>
> In a quick test, adding child_append_relid_sets (from 0003) to the relids being passed to record_elided_node fixes
> that. Presumably the case of partitionwise join relids doesn't matter, because that would prevent it being elided.

I'm not really sure there's a problem here. We definitely do not want
to end up with something like "Elided Node RTIs: 1 2". What I've found
experimentally is that it's often important to preserve relid sets,
but you need to preserve them as sets, not individually. So there
could be an argument that we somehow want to preserve both {1} and {2}
here, but that's not equivalent to {1,2}, which looks like a
partitionwise join between relid 1 and relid 2. But it isn't
especially clear to me that we actually need to preserve RTI 2 here.
One reason why preserving RTIs is important is so that as we descend a
join tree, we can find the RTIs that the optimizer thought it was
joining, but that only requires finding RTI 1, not RTI 2. Another
reason why preserving RTIs is important is so that we can use relation
identifiers to describe planning decisions made with respect to those
RTIs, but that doesn't apply here because partition expansion just
always happens.

Of course, it's quite possible that there are reasons unrelated to
this patch set why this information would be good to preserve, but if
we want to do it, we're going to have to adjust the data
representation somehow. We'd either need to give the ElidedNode a
"cars" representation instead of a single RTI set, or we'd need to
have some separate way of representing this. I hesitate a little bit
to design something without a use case in mind, but maybe you have
one?

> For 0003:
>
> I also find the "cars" variable suffix a bit hard to understand, but not sure a comment next to the variables is that useful.
> Separately, the noise generated by all the additional "_cars" variables isn't great.
>
> I wonder a little bit if we couldn't introduce a better abstraction here, e.g. a struct "AppendPathInput" that contains the
> two related lists, and gets populated by accumulate_append_subpath/get_singleton_append_subpath and then
> passed to create_append_path as a single argument.

I spent some time thinking about this day and haven't been quite able
to come up with something that I like. The problem is that
pa_partial_subpaths and pa_nonpartial_subpaths share a single
child_append_relid_sets variable, namely pa_subpath_cars, and
accumulate_append_subpaths gets called with that as the last argument
and different things for the previous two. One thing I tried was
making the AppendPathInput struct contain three lists rather than two,
but then accumulate_append_subpath() needs an argument that makes it
work in one of three different modes:

Mode 1: normal -- add everything to the "normal" list
Mode 2: building parallel-aware append with partial path -- add things
to the "normal" list except for parallel-aware appends which need to
be split between the normal and special lists
Mode 3: building parallel-aware append with non-partial path -- add
things to the "special" list

I also tried splitting up accumulate_append_subpath() into two
functions, thinking that maybe I could segregate the parallel-append
handling from the more normal cases. This seems somewhat appealing in
the sense that having accumulate_append_subpath() hold a bunch of
extra logic that only one call site needs isn't very nice, but
changing it doesn't really seem to help with the problem that we have
two subpath lists sharing one cars list in this case. I'll try to find
some more time to think about this, but if you have any ideas
meanwhile, I'd be happy to hear them.

> Note that 0005 needs a rebase, since 48d4a1423d2e92d10077365532d92e059ba2eb2e changed the GetNamedDSMSegment API.

I'll fix this. I don't love the way that commit made the callback and
the callback arg non-consecutive arguments.

> You may also want to move the CF entry to the PG19-4 commitfest so CFbot runs again.

It seems that I cannot move it to 19-4. I moved it to 19-Final.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  43. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-01-06T19:50:46Z

    On Mon, Dec 29, 2025 at 6:34 PM Haibo Yan <tristan.yim@gmail.com> wrote:
> 1. GEQO interaction (patch 4):
> Since GEQO relies on randomized search, is there a risk that the optimizer may fail to explore the specific join order or path that is being enforced by the advice mask? In that case, could this lead to failures such as inability to construct the required join relation or excessive planning time if the desired path is not sampled?

The interaction of this feature with GEQO definitely needs more study.
If you have some time to work on this, I think testing and reporting
results would be quite useful. However, I don't think we should ever
get planner failure, and I'm doubtful about excessive planning time as
well. The effect of plan advice is to disable some paths just as if
enable_<whatever> were set to false, so if you provide very specific
advice while planning with GEQO, I think you might just end up with a
disabled path that doesn't account for the advice. However, this
should be checked, and I haven't gotten there yet. I'll add an XXX to
the README to make sure this doesn't get forgotten.

> 2. Parallel query serialization (patches 1–3):
> Several new fields (subrtinfos, elidedNodes, child_append_relid_sets) are added to PlannedStmt, but I did not see corresponding changes in outfuncs.c / readfuncs.c. Without serialization support, parallel workers executing subplans or Append nodes may not receive this metadata. Is this handled elsewhere, or is it something still pending?

I believe that gen_node_support.pl should take care of this
automatically unless the node type is flagged as
pg_node_attr(custom_read_write).

> 3. Alias handling when generating advice (patch 5):
> In pgpa_output_relation_name, the advice string is generated using get_rel_name(relid), which resolves to the underlying table name rather than the RTE alias. In self-join cases this could be ambiguous (e.g., my_table vs my_table). Would it be more appropriate to use the RTE alias when available?

No. That function is only used for indexes.

> 4. Minor typo (patch 4):
> In src/include/nodes/relation.h, parititonwise appears to be a typo and should likely be partitionwise.

Will fix, thanks.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  44. Re: pg_plan_advice

    Lukas Fittl <lukas@fittl.com> — 2026-01-07T07:04:19Z

    On Mon, Dec 29, 2025 at 5:15 PM Lukas Fittl <lukas@fittl.com> wrote:

> On Mon, Dec 15, 2025 at 12:06 PM Robert Haas <robertmhaas@gmail.com>
> wrote:
> > Here's v7.
>
> I'm excited about this patch series, and in an effort to help land the
> infrastructure, here is a review of 0001 - 0003 to start:
>

For the review of 0004, I decided to spend a few days to test if the plan
generation strategy logic will
work for pg_hint_plan, as an existing extension in the ecosystem that is
widely used, but functions today
by virtue of modifying planner GUCs whilst the planner is running.

First of all, as is expected, the extension completely stops working with
0004 in place - because we now
only read the GUCs at planner start, the mechanism of modifying them in the
middle doesn't work. I don't
think we can avoid this.

That said, good news: After a bunch of iterations, I get a clean pass on
the pg_hint_plan regression tests,
whilst completely dropping its copying of core code and hackish re-run of
set_plain_rel_pathlist. See [0]
for a draft PR (on my own fork of pg_hint_plan) with individual patches
that explain some regression test
differences.

Adding Michael in CC, since he's been thankfully maintaining pg_hint_plan
over the years, and I think if
0004 gets merged that should significantly reduce the maintenance burden,
independently of what happens
with pg_plan_advice - so his input would be useful here.

The biggest change in the regression test output was due to how the
"Parallel" hint worked in pg_hint_plan
(basically it was setting parallel_*_cost to zero, and then messed with the
gucs that factor into
compute_parallel_worker) -- I think the only sensible thing to do is to
change that in pg_hint_plan, and
instead rely on rejecting non-partial paths with PGS_CONSIDER_NONPARTIAL if
"hard" enforcement of
parallelism is requested. That caused some minor plan changes, but I think
they can still be argued to be
matching the user's intent of "make a scan involving this relation
parallel".

There were two bugs in 0004 that I had to fix to make this work:

In cost_index, we are checking "path->path.parallel_workers == 0", but
parallel_workers only gets
set later in the function, causing the PGS_CONSIDER_NONPARTIAL mask to not
be applied. Replacing
this with checking the "partial_path" argument instead makes it work.

In cost_samplescan, we set the PGS_CONSIDER_NONPARTIAL mask if its a
non-partial path, but that
causes Sample Scans to always be disabled when
setting PGS_CONSIDER_NONPARTIAL on the
relation. I think we could simply drop that check, since we never generate
partial sample scan paths.

Otherwise 0004 looks good to me, and the mechanism of working with mask
values felt natural to me,
especially in contrast with existing ways to achieve something similar. I
did not test partition wise joins,
since pg_hint_plan doesn't cover them today.

[0]: https://github.com/lfittl/pg_hint_plan/pull/1

Thanks,
Lukas

-- 
Lukas Fittl

  45. Re: pg_plan_advice

    Haibo Yan <tristan.yim@gmail.com> — 2026-01-07T22:46:54Z

    >> 1. GEQO interaction (patch 4):
>> Since GEQO relies on randomized search, is there a risk that the
optimizer may fail to explore the specific join order or path that is being
enforced by the advice mask? In that case, could this lead to failures such
as inability to construct the required join relation or excessive planning
time if the desired path is not sampled?

> The interaction of this feature with GEQO definitely needs more study.
> If you have some time to work on this, I think testing and reporting
> results would be quite useful. However, I don't think we should ever
> get planner failure, and I'm doubtful about excessive planning time as
> well. The effect of plan advice is to disable some paths just as if
> enable_<whatever> were set to false, so if you provide very specific
> advice while planning with GEQO, I think you might just end up with a
> disabled path that doesn't account for the advice. However, this
> should be checked, and I haven't gotten there yet. I'll add an XXX to
> the README to make sure this doesn't get forgotten.

I conducted extensive tests today using randomized advice strings to
challenge pg_plan_advice under GEQO pressure. The results strongly support
your hypothesis: for standard Left-Deep trees (which GEQO natively
supports), the interaction is stable and efficient.

I executed a stress test involving 100,000 iterations (100 random join
structures x 1000 random seeds). The planning time remained low, and no
planning failures occurred for valid topology advice.

Observation on Bushy Plans: I did identify one anomaly regarding "Bushy
Plans" (e.g., ((t1 t2) (t3 t4))). Since PostgreSQL's GEQO implementation is
strictly Left-Deep and cannot generate Bushy trees, if a user manually
forges a Bushy Plan advice:

It does not cause a planner crash (e.g., "failed to construct join
relation").

Instead, the planner seems to silently ignore the structural constraint of
the advice and falls back to a path GEQO can actually find.

I believe this behavior is acceptable because pg_plan_advice is intended to
stabilize plans that the optimizer can generate. Since GEQO cannot generate
Bushy plans, users should not be supplying them.

Script
-----------------------------------------------------------------------------------
/* * GEQO Stress Test for pg_plan_advice
 * -----------------------------------
 * Methodology:
 * 1. Generates 100 random "Left-Deep" join topologies (t1 joining t2..t100
in random orders).
 * 2. This simulates valid advice that GEQO is capable of producing.
 * 3. For each topology, runs 1000 iterations with random GEQO seeds.
 * 4. Measures success rate and planning time overhead.
 */
DO $$
DECLARE
    v_jo       TEXT;
    v_jo_rest  TEXT;
    v_nl       TEXT;
    v_scan     TEXT;
    v_ng       TEXT;
    v_adv      TEXT;
    v_sql      TEXT;
    v_seed     FLOAT;
    v_ok       INT := 0;
    v_err      INT := 0;
    v_msg      TEXT;
    k          INT;
    i          INT;
    j          INT;
    v_ts1      timestamp;
    v_ts2      timestamp;
    v_cur_ms   numeric;
    v_total_ms numeric := 0;
    v_max_ms   numeric := 0;
BEGIN
    -- Pre-generate static parts of the advice to save time
    SELECT string_agg('t'||n, ' ' ORDER BY n) INTO v_nl   FROM
generate_series(2,100) n;
    SELECT string_agg('t'||n, ' ' ORDER BY n) INTO v_scan FROM
generate_series(1,100) n;
    SELECT string_agg('t'||n, ' ' ORDER BY n) INTO v_ng   FROM
generate_series(1,100) n;

    -- Construct the SQL: t1 JOIN t2 JOIN t3 ... JOIN t100
    v_sql := 'EXPLAIN (COSTS OFF) SELECT count(*) FROM t1';
    FOR j IN 2..100 LOOP
        v_sql := v_sql || ' JOIN t' || j || ' ON t1.id=t' || j || '.id';
    END LOOP;

    -- Configure GEQO for stress testing (force it ON, low effort/pool)
    PERFORM set_config('geqo', 'on', false);
    PERFORM set_config('geqo_threshold', '12', false);
    PERFORM set_config('geqo_effort', '1', false);
    PERFORM set_config('geqo_pool_size', '0', false);

    RAISE NOTICE 'Starting Stress Test: 100 Outer Loops (Random Plans) x
1000 Inner Loops (Random Seeds)...';

    -- Outer Loop: Generate 100 different valid Advice structures
    FOR k IN 1..100 LOOP
        -- Randomize the join order of t2..t100 to simulate different
Left-Deep trees
        SELECT string_agg('t'||n, ' ' ORDER BY random()) INTO v_jo_rest
FROM generate_series(2,100) n;
        v_jo := 't1 ' || v_jo_rest;

        v_adv := 'JOIN_ORDER(' || v_jo || ') ' ||
                 'NESTED_LOOP_PLAIN(' || v_nl || ') ' ||
                 'SEQ_SCAN(' || v_scan || ') ' ||
                 'NO_GATHER(' || v_ng || ')';

        PERFORM set_config('pg_plan_advice.advice', v_adv, false);

        -- Inner Loop: Test the specific advice against 1000 random GEQO
seeds
        FOR i IN 1..1000 LOOP
            v_seed := random();
            PERFORM set_config('geqo_seed', v_seed::text, false);

            BEGIN
                v_ts1 := clock_timestamp();

                EXECUTE v_sql;

                v_ts2 := clock_timestamp();

                v_cur_ms := EXTRACT(EPOCH FROM (v_ts2 - v_ts1)) * 1000;
                v_total_ms := v_total_ms + v_cur_ms;

                IF v_cur_ms > v_max_ms THEN
                    v_max_ms := v_cur_ms;
                END IF;

                v_ok := v_ok + 1;

            EXCEPTION WHEN OTHERS THEN
                GET STACKED DIAGNOSTICS v_msg = MESSAGE_TEXT;
                v_err := v_err + 1;
                RAISE WARNING 'Outer % / Inner % Crashed! Seed: %, Err: %',
k, i, v_seed, v_msg;
            END;
        END LOOP;

        RAISE NOTICE 'Batch %/100 completed.', k;
    END LOOP;

    RAISE NOTICE '---------------------------';
    RAISE NOTICE 'Total Scenarios: 100,000';
    RAISE NOTICE 'Success:         %', v_ok;
    RAISE NOTICE 'Failed:          %', v_err;
    RAISE NOTICE 'Total Time:      % ms', round(v_total_ms, 2);
    RAISE NOTICE 'Avg Time:        % ms', round(v_total_ms / (v_ok + v_err
+ 0.0001), 2);
    RAISE NOTICE 'Max Time:        % ms', round(v_max_ms, 2);
    RAISE NOTICE '---------------------------';

    IF v_err > 0 THEN
        RAISE NOTICE 'CONCLUSION: Conflict found.';
    ELSE
        RAISE NOTICE 'CONCLUSION: No errors found.';
    END IF;
END $$;
-----------------------------------------------------------------------------------


On Tue, Jan 6, 2026 at 11:50 AM Robert Haas <robertmhaas@gmail.com> wrote:

> On Mon, Dec 29, 2025 at 6:34 PM Haibo Yan <tristan.yim@gmail.com> wrote:
> > 1. GEQO interaction (patch 4):
> > Since GEQO relies on randomized search, is there a risk that the
> optimizer may fail to explore the specific join order or path that is being
> enforced by the advice mask? In that case, could this lead to failures such
> as inability to construct the required join relation or excessive planning
> time if the desired path is not sampled?
>
> The interaction of this feature with GEQO definitely needs more study.
> If you have some time to work on this, I think testing and reporting
> results would be quite useful. However, I don't think we should ever
> get planner failure, and I'm doubtful about excessive planning time as
> well. The effect of plan advice is to disable some paths just as if
> enable_<whatever> were set to false, so if you provide very specific
> advice while planning with GEQO, I think you might just end up with a
> disabled path that doesn't account for the advice. However, this
> should be checked, and I haven't gotten there yet. I'll add an XXX to
> the README to make sure this doesn't get forgotten.
>
> > 2. Parallel query serialization (patches 1–3):
> > Several new fields (subrtinfos, elidedNodes, child_append_relid_sets)
> are added to PlannedStmt, but I did not see corresponding changes in
> outfuncs.c / readfuncs.c. Without serialization support, parallel workers
> executing subplans or Append nodes may not receive this metadata. Is this
> handled elsewhere, or is it something still pending?
>
> I believe that gen_node_support.pl should take care of this
> automatically unless the node type is flagged as
> pg_node_attr(custom_read_write).
>
> > 3. Alias handling when generating advice (patch 5):
> > In pgpa_output_relation_name, the advice string is generated using
> get_rel_name(relid), which resolves to the underlying table name rather
> than the RTE alias. In self-join cases this could be ambiguous (e.g.,
> my_table vs my_table). Would it be more appropriate to use the RTE alias
> when available?
>
> No. That function is only used for indexes.
>
> > 4. Minor typo (patch 4):
> > In src/include/nodes/relation.h, parititonwise appears to be a typo and
> should likely be partitionwise.
>
> Will fix, thanks.
>
> --
> Robert Haas
> EDB: http://www.enterprisedb.com
>

  46. Re: pg_plan_advice

    Jacob Champion <jacob.champion@enterprisedb.com> — 2026-01-07T23:11:24Z

    On Mon, Dec 15, 2025 at 12:06 PM Robert Haas <robertmhaas@gmail.com> wrote:
> - Add gather((d d/d.d)) test case, per Jacob, and fix the related bug
> in pgpa_identifier_matches_target, per Jacob's email.

I think this fix affected the ability to omit the partition schema in
advice strings. Attached is a quick test that shows it on my machine,
plus an attempted fix that mashes together the v6 and v7 approaches.
(I have diffs in my generated plan advice compared to what's in
v7-0005, so I'm not sure if my .out file is correct.)

--Jacob

  47. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-01-08T16:07:31Z

    On Wed, Jan 7, 2026 at 2:04 AM Lukas Fittl <lukas@fittl.com> wrote:
> That said, good news: After a bunch of iterations, I get a clean pass on the pg_hint_plan regression tests,
> whilst completely dropping its copying of core code and hackish re-run of set_plain_rel_pathlist. See [0]
> for a draft PR (on my own fork of pg_hint_plan) with individual patches that explain some regression test
> differences.

That sounds AWESOME.

> The biggest change in the regression test output was due to how the "Parallel" hint worked in pg_hint_plan
> (basically it was setting parallel_*_cost to zero, and then messed with the gucs that factor into
> compute_parallel_worker) -- I think the only sensible thing to do is to change that in pg_hint_plan, and
> instead rely on rejecting non-partial paths with PGS_CONSIDER_NONPARTIAL if "hard" enforcement of
> parallelism is requested. That caused some minor plan changes, but I think they can still be argued to be
> matching the user's intent of "make a scan involving this relation parallel".

Cool. I'm sort of curious what changed, but maybe it's not important
enough to spend time discussing right now.

> There were two bugs in 0004 that I had to fix to make this work:
>
> In cost_index, we are checking "path->path.parallel_workers == 0", but parallel_workers only gets
> set later in the function, causing the PGS_CONSIDER_NONPARTIAL mask to not be applied. Replacing
> this with checking the "partial_path" argument instead makes it work.

I agree that this is a bug. I'm thinking this might be the appropriate fix:

     enable_mask = (indexonly ? PGS_INDEXONLYSCAN : PGS_INDEXSCAN)
-        | (path->path.parallel_workers == 0 ? PGS_CONSIDER_NONPARTIAL : 0);
+        | (partial_path ? 0 : PGS_CONSIDER_NONPARTIAL);

> In cost_samplescan, we set the PGS_CONSIDER_NONPARTIAL mask if its a non-partial path, but that
> causes Sample Scans to always be disabled when setting PGS_CONSIDER_NONPARTIAL on the
> relation. I think we could simply drop that check, since we never generate partial sample scan paths.

This one is less obvious to me. I mean, if PGS_CONSIDER_NONPARTIAL
lets us consider non-partial plans, and a sample scan is a non-partial
plan, then shouldn't the flag need to be set in order for us to
consider it? If not, maybe we need to rethink the name or the
semantics of that bit in some way.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  48. Re: pg_plan_advice

    Lukas Fittl <lukas@fittl.com> — 2026-01-08T16:30:42Z

    On Thu, Jan 8, 2026 at 8:07 AM Robert Haas <robertmhaas@gmail.com> wrote:
> > The biggest change in the regression test output was due to how the "Parallel" hint worked in pg_hint_plan
> > (basically it was setting parallel_*_cost to zero, and then messed with the gucs that factor into
> > compute_parallel_worker) -- I think the only sensible thing to do is to change that in pg_hint_plan, and
> > instead rely on rejecting non-partial paths with PGS_CONSIDER_NONPARTIAL if "hard" enforcement of
> > parallelism is requested. That caused some minor plan changes, but I think they can still be argued to be
> > matching the user's intent of "make a scan involving this relation parallel".
>
> Cool. I'm sort of curious what changed, but maybe it's not important
> enough to spend time discussing right now.

In my assessment there were roughly four kinds of changes on the
regression test outputs for pg_hint_plan:

1) The order of operations (e.g. whats the inner/outer rel in a
parallel plan) - I think that's probably caused by the previous zero
cost confusing the planner
2) The placement of the Gather node in the case of joins (its now on
top of the join in more cases, vs below it) - I think that's also
caused by the previous zero cost logic
3) Parallelism wasn't enforced before, but is enforced now (e.g.
sequential scans on empty relations didn't get a Gather node
previously)
4) Worker count changed because rel_parallel_workers is still limited
by max_parallel_workers_per_gather, and so my patched version now sets
that for the whole query to the highest of the workers requested in
Parallel hints (vs before it was able to keep that to just what a plan
node needed)

> > There were two bugs in 0004 that I had to fix to make this work:
> >
> > In cost_index, we are checking "path->path.parallel_workers == 0", but parallel_workers only gets
> > set later in the function, causing the PGS_CONSIDER_NONPARTIAL mask to not be applied. Replacing
> > this with checking the "partial_path" argument instead makes it work.
>
> I agree that this is a bug. I'm thinking this might be the appropriate fix:
>
>      enable_mask = (indexonly ? PGS_INDEXONLYSCAN : PGS_INDEXSCAN)
> -        | (path->path.parallel_workers == 0 ? PGS_CONSIDER_NONPARTIAL : 0);
> +        | (partial_path ? 0 : PGS_CONSIDER_NONPARTIAL);

Yup, that's exactly the change I had locally that worked as expected.

> > In cost_samplescan, we set the PGS_CONSIDER_NONPARTIAL mask if its a non-partial path, but that
> > causes Sample Scans to always be disabled when setting PGS_CONSIDER_NONPARTIAL on the
> > relation. I think we could simply drop that check, since we never generate partial sample scan paths.
>
> This one is less obvious to me. I mean, if PGS_CONSIDER_NONPARTIAL
> lets us consider non-partial plans, and a sample scan is a non-partial
> plan, then shouldn't the flag need to be set in order for us to
> consider it? If not, maybe we need to rethink the name or the
> semantics of that bit in some way.

Yeah, I would agree with you that is inconsistent with the flag's name
- but on the flip side, its difficult for the caller to conditionally
set the flag (which you'd have to do to avoid a "Disabled" showing in
the plan), since we're setting it on the RelOptInfo (do we know if the
scan is a sample scan at that point?).

I wonder if its worth considering to invert the flag, i.e.
"PGS_PREFER_PARTIAL" - that way its clear that in case of paths that
can't be partial, we use the non-partial version without a penalty.

PS: I realized my prior mails were not plain text and had bad word
wrap (thanks Gmail-based workflow!) - hopefully this one is better :)

Thanks,
Lukas

-- 
Lukas Fittl

  49. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-01-08T16:37:51Z

    On Thu, Jan 8, 2026 at 11:31 AM Lukas Fittl <lukas@fittl.com> wrote:
> Yeah, I would agree with you that is inconsistent with the flag's name
> - but on the flip side, its difficult for the caller to conditionally
> set the flag (which you'd have to do to avoid a "Disabled" showing in
> the plan), since we're setting it on the RelOptInfo (do we know if the
> scan is a sample scan at that point?).

How about checking rte->tablesample, as set_rel_pathlist does?

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  50. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-01-08T18:21:35Z

    On Wed, Jan 7, 2026 at 6:11 PM Jacob Champion
<jacob.champion@enterprisedb.com> wrote:
> On Mon, Dec 15, 2025 at 12:06 PM Robert Haas <robertmhaas@gmail.com> wrote:
> > - Add gather((d d/d.d)) test case, per Jacob, and fix the related bug
> > in pgpa_identifier_matches_target, per Jacob's email.
>
> I think this fix affected the ability to omit the partition schema in
> advice strings. Attached is a quick test that shows it on my machine,
> plus an attempted fix that mashes together the v6 and v7 approaches.
> (I have diffs in my generated plan advice compared to what's in
> v7-0005, so I'm not sure if my .out file is correct.)

Right, so I also am seeing some instability in the plans for some of
the test cases. I've attempted to address that in v8 by making the
three partitioned tables have unequal row counts. I also adapted your
test case and adopted your code fix. Unfortunately, I haven't been
able to respond to everyone's reports about v7 yet, partly because of
Christmas vacation, partly because of the number of reports, and
partly because, uh, I'm slow I guess. But here's the list of things
that have changed:

0001, 0003: No changes.

0002: Comment update.

0004: Comment update, bug fix to cost_index() per comment from Lukas.

- Added an XXX to the README to highlight the need for more GEQO investigation.
- Regression test expected output changes.
- Make partitionwise_join test tables different sizes in the hopes of
stabilizing test results.
- Add test case for forcing partitionwise join order with and without
a schema specification.
- Add new GUC pg_plan_advice.trace_mask because I've written similar
debugging code too many times already.
- Replace reference to inner_beneath_any_gather[i] with
inner_beneath_any_gather[k] per comment from Jakub.
- Add logic to pgpa_planner_apply_join_path_advice taking into account
that PARTITIONWISE() implicitly constrains the join order.
- Rebased.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  51. Re: pg_plan_advice

    Lukas Fittl <lukas@fittl.com> — 2026-01-08T19:13:43Z

    On Thu, Jan 8, 2026 at 10:22 AM Robert Haas <robertmhaas@gmail.com> wrote:
> 0004: Comment update, bug fix to cost_index() per comment from Lukas.

Thanks! I've tested this and this works as expected on current master
with the updated pg_hint_plan code, and checking for tablesample in
the code that sets the mask.

On Thu, Jan 8, 2026 at 8:38 AM Robert Haas <robertmhaas@gmail.com> wrote:
> On Thu, Jan 8, 2026 at 11:31 AM Lukas Fittl <lukas@fittl.com> wrote:
> > Yeah, I would agree with you that is inconsistent with the flag's name
> > - but on the flip side, its difficult for the caller to conditionally
> > set the flag (which you'd have to do to avoid a "Disabled" showing in
> > the plan), since we're setting it on the RelOptInfo (do we know if the
> > scan is a sample scan at that point?).
>
> How about checking rte->tablesample, as set_rel_pathlist does?

Yeah, that works - its a bit inconvenient for two reasons, but I don't
think that warrants a redesign:

1) get_relation_info_hook doesn't get a RangeTblEntry passed (like
set_rel_pathlist_hook), but that's solvable by looking it up via
simple_rte_array
2) It requires maintaining a special case in the logic that says "make
it parallel", vs the planner that is authoritative (i.e. if we add
more special cases in the future, each extension will have to be
updated to reflect that)

Thanks,
Lukas

-- 
Lukas Fittl

  52. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-01-08T19:23:48Z

    On Thu, Jan 8, 2026 at 2:14 PM Lukas Fittl <lukas@fittl.com> wrote:
> On Thu, Jan 8, 2026 at 10:22 AM Robert Haas <robertmhaas@gmail.com> wrote:
> > 0004: Comment update, bug fix to cost_index() per comment from Lukas.
>
> Thanks! I've tested this and this works as expected on current master
> with the updated pg_hint_plan code, and checking for tablesample in
> the code that sets the mask.

Nice!

> > How about checking rte->tablesample, as set_rel_pathlist does?
>
> Yeah, that works - its a bit inconvenient for two reasons, but I don't
> think that warrants a redesign:
>
> 1) get_relation_info_hook doesn't get a RangeTblEntry passed (like
> set_rel_pathlist_hook), but that's solvable by looking it up via
> simple_rte_array

That's a pretty normal thing to have to do in this kind of code, IMHO.

> 2) It requires maintaining a special case in the logic that says "make
> it parallel", vs the planner that is authoritative (i.e. if we add
> more special cases in the future, each extension will have to be
> updated to reflect that)

IMHO, this is a policy decision by the extension and so the logic
belongs in the extension. pg_plan_advice has no similar exception,
because it made a different policy decision.

Overall, I feel that your experiment here is a pretty compelling
argument for committing 0004 (the pgs_mask stuff). I'm not sure it's
quite time to do that yet, because maybe there are still some design
changes we want to consider for it, or maybe somebody else wants to
review first. But even if that patch did nothing other than get rid of
lots of complexity and copy-paste in pg_hint_plan, that would be
enough to justify this infrastructure. The fact that basically works
as intended for both pg_plan_advice and pg_hint_plan, despite them
having no common code, is a really good sign, IMHO.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  53. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-01-08T19:37:26Z

    On Thu, Jan 8, 2026 at 11:31 AM Lukas Fittl <lukas@fittl.com> wrote:
> In my assessment there were roughly four kinds of changes on the
> regression test outputs for pg_hint_plan:
>
> 1) The order of operations (e.g. whats the inner/outer rel in a
> parallel plan) - I think that's probably caused by the previous zero
> cost confusing the planner
> 2) The placement of the Gather node in the case of joins (its now on
> top of the join in more cases, vs below it) - I think that's also
> caused by the previous zero cost logic
> 3) Parallelism wasn't enforced before, but is enforced now (e.g.
> sequential scans on empty relations didn't get a Gather node
> previously)
> 4) Worker count changed because rel_parallel_workers is still limited
> by max_parallel_workers_per_gather, and so my patched version now sets
> that for the whole query to the highest of the workers requested in
> Parallel hints (vs before it was able to keep that to just what a plan
> node needed)

Ideally, if you say "hey, I want to use parallel query here," you
would get the best plan that the planner knows how to construct that
happens to use parallelism. #2 sounds to me like you weren't really
getting that, because you were making parallelism on cost rather than
disabling non-parallel paths. #3 also sounds that way, because you
asked for parallelism and didn't get it. So I would tend to view those
changes as improvements.

The others are a little trickier. I don't think I quite understand
what this patch changed with respect to #4. I'm guessing that maybe
what's happening here is that this isn't really due to anything in the
patch set, but is rather due to relying on pgs_mask rather than
copy-pasting lots of code, which maybe incidentally removed the
ability to tweak something that pg_hint_plan was previously tweaking.
Maybe you want to think about writing a patch to go with 0004 that
addresses that gap specifically?

I'm actually kind of surprised that you didn't run into a similar
problem with the Rows() hint, for which 0004 also doesn't provide
infrastructure. If there's no problem, cool, but if there's a problem
there you haven't detected yet, maybe we should try to plug that gap,
too.

I don't quite know what to say about #1. If your theory about it being
due to the zero costs is correct, that's another example of the
current implementation not really being able to achieve the ideal
behavior, and the patch making it better. If there's something else
going on there, then I don't know.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  54. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-01-08T19:53:14Z

    On Wed, Jan 7, 2026 at 5:47 PM Haibo Yan <tristan.yim@gmail.com> wrote:
> Instead, the planner seems to silently ignore the structural constraint of the advice and falls back to a path GEQO can actually find.

Does the plan end up disabled in that case?

> I believe this behavior is acceptable because pg_plan_advice is intended to stabilize plans that the optimizer can generate. Since GEQO cannot generate Bushy plans, users should not be supplying them.

Right, I agree. A core principal here is that you can only nudge the
planner towards a plan it would have considered anyway. In the case of
GEQO, there is some randomness to which plans are considered. Your
advice will only be reliably take into account if it applies to
elements that must be part of the final plan. For instance, if you
advise the use of a sequential scan or an index scan, that should
work, because that relation has to be scanned somehow. Advice on a
join method should almost always work, since it can apply to any
non-leading table. Of course, you also won't be able to advise an
infeasible join method, but that would be true without GEQO, too.
Advice on the join order is going to be iffy when using GEQO -- if a
compatible join order is highly likely to be considered, e.g. because
you specify something like JOIN_ORDER(just_one) that only sets the
driving table -- then it'll probably work, but if you give a complete
join order specification, it probably won't. If you want to avoid
that, you can adjust geqo_threshold.

Thanks for looking into this.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  55. Re: pg_plan_advice

    Lukas Fittl <lukas@fittl.com> — 2026-01-08T20:47:05Z

    On Thu, Jan 8, 2026 at 11:37 AM Robert Haas <robertmhaas@gmail.com> wrote:
> Ideally, if you say "hey, I want to use parallel query here," you
> would get the best plan that the planner knows how to construct that
> happens to use parallelism. #2 sounds to me like you weren't really
> getting that, because you were making parallelism on cost rather than
> disabling non-parallel paths. #3 also sounds that way, because you
> asked for parallelism and didn't get it. So I would tend to view those
> changes as improvements.

Agreed from my perspective.

> The others are a little trickier. I don't think I quite understand
> what this patch changed with respect to #4. I'm guessing that maybe
> what's happening here is that this isn't really due to anything in the
> patch set, but is rather due to relying on pgs_mask rather than
> copy-pasting lots of code, which maybe incidentally removed the
> ability to tweak something that pg_hint_plan was previously tweaking.
> Maybe you want to think about writing a patch to go with 0004 that
> addresses that gap specifically?

Yeah, I don't think this needs to be reviewed in detail here now, but
I do think there is a potential to improve worker count overrides in
core code. I agree this can be done as a follow-up to 0004 - and I
don't have a good sense yet for how infrastructure for this could look
like.

pg_hint_plan has some existing logic to copy parallel worker counts
into other paths (e.g. for joins), and I kept that for now in that
draft patch [0] to reduce further test output differences.

> I'm actually kind of surprised that you didn't run into a similar
> problem with the Rows() hint, for which 0004 also doesn't provide
> infrastructure. If there's no problem, cool, but if there's a problem
> there you haven't detected yet, maybe we should try to plug that gap,
> too.

Yeah, 0004 doesn't provide infrastructure for that directly. However,
it does add joinrel_setup_hook which enables modifying joinrel->rows
at the right time without copying core code. Previously pg_hint_plan
modified the row estimates for joins through copied core code [1], but
0004 lets it affect join rels through the new hook [2].

> I don't quite know what to say about #1. If your theory about it being
> due to the zero costs is correct, that's another example of the
> current implementation not really being able to achieve the ideal
> behavior, and the patch making it better. If there's something else
> going on there, then I don't know.

Yeah, I don't think we need to worry about this in the context of
applying 0004, and its something that Michael or other pg_hint_plan
maintainers can assess when updating it.

Its worth noting for clarity, whilst 0004 requires extensions that
modify certain GUCs during the planning process to instead use the PGS
mask, max_parallel_workers_per_gather is not one of these GUCs. I was
driven to also do the parallel hint changes because I wanted to prove
that pg_hint_plan can now function without any copying of core code,
but technically all the parallel regression test changes can be
avoided when keeping the prior zero parallel_*_cost mechanism +
modifying max_parallel_workers_per_gather during planning as before,
whilst still updating the scan/join hints to use the PGS mask logic.

Thanks,
Lukas

[0]: https://github.com/lfittl/pg_hint_plan/blob/postgres-19-with-plan-generation-strategies/pg_hint_plan.c#L4602
[1]: https://github.com/ossc-db/pg_hint_plan/blob/master/make_join_rel.c#L126
[2]: https://github.com/lfittl/pg_hint_plan/blob/postgres-19-with-plan-generation-strategies/pg_hint_plan.c#L4372

-- 
Lukas Fittl

  56. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-01-08T20:58:54Z

    On Thu, Dec 18, 2025 at 8:36 AM Robert Haas <robertmhaas@gmail.com> wrote:
> What must be happening here is that either pgpa_join.c (maybe with
> complicity from pgpa_walker.c) is not populating the
> pgpa_plan_walker_context's join_strategies[JSTRAT_NESTED_LOOP_PLAIN]
> member correctly, or else pgpa_output.c is not serializing it to text
> correctly. I suspect the former is a more likely but I'm not sure
> exactly what's happening.

I think I see the problem: pgpa_process_unrolled_join() returns a set
called "all_relids" but it only returns the union of the inner relid
sets, not including the outer relid set. In your example, we want to
get:

NESTED_LOOP_PLAIN((part partsupp) (supplier part partsupp))

But the join order is:

JOIN_ORDER(nation (supplier (part partsupp)))

So every table is the outer table of some unrolled join, except for
the innermost table, which is partsupp. So all the others get omitted
from the output, and we get the output you saw:

NESTED_LOOP_PLAIN(partsupp partsupp)

Proposed fix attached.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  57. Re: pg_plan_advice

    Haibo Yan <tristan.yim@gmail.com> — 2026-01-08T22:52:29Z

    > Does the plan end up disabled in that case?

I understand that except for join order advice which might not be
honored(due to GEQO's randomness), other forms of advice—like scan types or
join methods—remain effective since those operations are inevitable parts
of the plan regardless of the specific tree structure.
Thanks for the explanation regarding the design philosophy. It clarifies
that the primary goal is stabilizing plans within the optimizer's valid
search space rather than forcing impossible paths. I will keep the
geqo_threshold adjustment in mind if strict structure enforcement is ever
needed.
Regards
Haibo

On Thu, Jan 8, 2026 at 11:53 AM Robert Haas <robertmhaas@gmail.com> wrote:

> On Wed, Jan 7, 2026 at 5:47 PM Haibo Yan <tristan.yim@gmail.com> wrote:
> > Instead, the planner seems to silently ignore the structural constraint
> of the advice and falls back to a path GEQO can actually find.
>
> Does the plan end up disabled in that case?
>
> > I believe this behavior is acceptable because pg_plan_advice is intended
> to stabilize plans that the optimizer can generate. Since GEQO cannot
> generate Bushy plans, users should not be supplying them.
>
> Right, I agree. A core principal here is that you can only nudge the
> planner towards a plan it would have considered anyway. In the case of
> GEQO, there is some randomness to which plans are considered. Your
> advice will only be reliably take into account if it applies to
> elements that must be part of the final plan. For instance, if you
> advise the use of a sequential scan or an index scan, that should
> work, because that relation has to be scanned somehow. Advice on a
> join method should almost always work, since it can apply to any
> non-leading table. Of course, you also won't be able to advise an
> infeasible join method, but that would be true without GEQO, too.
> Advice on the join order is going to be iffy when using GEQO -- if a
> compatible join order is highly likely to be considered, e.g. because
> you specify something like JOIN_ORDER(just_one) that only sets the
> driving table -- then it'll probably work, but if you give a complete
> join order specification, it probably won't. If you want to avoid
> that, you can adjust geqo_threshold.
>
> Thanks for looking into this.
>
> --
> Robert Haas
> EDB: http://www.enterprisedb.com
>

  58. Re: pg_plan_advice

    Michael Paquier <michael@paquier.xyz> — 2026-01-08T23:29:38Z

    On Thu, Jan 08, 2026 at 11:07:31AM -0500, Robert Haas wrote:
> On Wed, Jan 7, 2026 at 2:04 AM Lukas Fittl <lukas@fittl.com> wrote:
>> That said, good news: After a bunch of iterations, I get a clean
>> pass on the pg_hint_plan regression tests, whilst completely
>> dropping its copying of core code and hackish re-run of
>> set_plain_rel_pathlist. See [0] for a draft PR (on my own fork of
>> pg_hint_plan) with individual patches that explain some regression
>> test differences.
> 
> That sounds AWESOME.

So you are telling me that I can commit code that deletes code.  Count
me in.  The project has some merge requests that I've been holding on
a bit due to what's happening here and because I did not really look
at the internals that have changed.  It's great to see that you have
begun an investigation, Lukas.

>> The biggest change in the regression test output was due to how the
>> "Parallel" hint worked in pg_hint_plan (basically it was setting
>> parallel_*_cost to zero, and then messed with the gucs that factor
>> into compute_parallel_worker) -- I think the only sensible thing to
>> do is to change that in pg_hint_plan, and instead rely on rejecting
>> non-partial paths with PGS_CONSIDER_NONPARTIAL if "hard"
>> enforcement of parallelism is requested. That caused some minor
>> plan changes, but I think they can still be argued to be matching
>> the user's intent of "make a scan involving this relation
>> parallel".
> 
> Cool. I'm sort of curious what changed, but maybe it's not important
> enough to spend time discussing right now.

I suspect that this is going to be an incremental integration process,
and it smells to me that it is going to require more than one major
release before being able to remove the whole set of hacks that
pg_hint_plan has been using, particularly with the GUCs, the costing
and the forced update of the backend routines which is a ugly
historical hack.  Saying that, I would need to look at the plan
outputs to be sure, perhaps we would be OK even with slight changes.
These happen every year, because the plans tested are complex enough
that some of the sub-paths are changed, but the hints still work
properly.  This year for v19 we have at least the changes in the
expression names.
--
Michael

  59. Re: pg_plan_advice

    Jakub Wartak <jakub.wartak@enterprisedb.com> — 2026-01-09T13:52:59Z

    On Thu, Jan 8, 2026 at 7:21 PM Robert Haas <robertmhaas@gmail.com> wrote:
[..]
> I've attempted to address that in v8 [..snip]
[..]

The full TPC-H queries set still reported some issues with v8 (for
q20/with wrong plan after using advice and two "partially matched" for
q9 and q2). However after applying that tiny patch from [1] it makes
all of the problems go away and for the TPC-H query set there are no
failures anymore (yay!). By failure I mean a different plan when using
advice and/or any partial match or failure to apply advice.

So, I've switched to more realistic test for each TPC-H query:
1) ensure we have valid stats, gather plan, save advices
2) clear stats using pg_clear_relation_stats() , apply advice and
check if we have exact same plan

The only thing this hav revealed is what appears to be some tiny
problem with placing GroupAggregates or am I wrong or is that known
limitation? (The original plan shows "Partial GroupAggregate" while
the one using advice is not aware of the need to use it; yet
contrib/pg_plan_advices/README in "Future Work" indicates it is out of
scope for now, right?)

--- /tmp/plan
+++ /tmp/planadviced
Sort
   Sort Key: (sum((lineitem.l_extendedprice * ('1'::numeric -
lineitem.l_discount)))) DESC
-  ->  Finalize GroupAggregate
+  ->  GroupAggregate
         Group Key: nation.n_name
         ->  Gather Merge
               Workers Planned: 2
-              ->  Partial GroupAggregate
-                    Group Key: nation.n_name
-                    ->  Sort
-                          Sort Key: nation.n_name
-                          ->  Hash Join
-                                Hash Cond: ((lineitem.l_suppkey =
supplier.s_suppkey) AND (customer.c_nationkey = supplier.s_nationkey))
+              ->  Sort
+                    Sort Key: nation.n_name
+                    ->  Hash Join
+                          Hash Cond: ((lineitem.l_suppkey =
supplier.s_suppkey) AND (customer.c_nationkey = supplier.s_nationkey))

-J.

[1] - https://www.postgresql.org/message-id/CA%2BTgmoZzBkd1BG8qusicUjme0kZuT8konQM_rcr0gMXs-TpK7A%40mail.gmail.com

  60. Re: pg_plan_advice

    Lukas Fittl <lukas@fittl.com> — 2026-01-11T20:20:32Z

    On Tue, Jan 6, 2026 at 11:36 AM Robert Haas <robertmhaas@gmail.com> wrote:
>
> On Mon, Dec 29, 2025 at 8:15 PM Lukas Fittl <lukas@fittl.com> wrote:
> > For 0001, I'm not sure the following comment is correct:
> >
> > > /* When recursing = true, it's an unplanned or dummy subquery. */
> > > rtinfo->dummy = recursing;
> >
> > Later in that function we only recurse if its a dummy subquery - in the case of an unplanned subquery (rel->subroot == NULL)
> > add_rtes_to_flat_rtable won't be called again (instead the relation RTEs are directly added to the finalrtable). Maybe we can
> > clarify that comment as "When recursing = true, it's a dummy subquery or its children.".
>
> Presumably, a child of an unplanned or dummy subquery will also be
> unplanned or dummy, so I'm not sure I understand the need to clarify
> here.

I think I was more trying to argue that unplanned subqueries are not
actually being considered here, since the recursing flag will never be
true for an unplanned subquery. The "or its children" part was more to
capture my understanding, and seems fine to omit too.

> > I also noticed that this currently doesn't support cases where multiple nodes are elided, e.g. with multi-level table partitioning:
>
> ...
> I'm not really sure there's a problem here. We definitely do not want
> to end up with something like "Elided Node RTIs: 1 2". What I've found
> experimentally is that it's often important to preserve relid sets,
> but you need to preserve them as sets, not individually. I hesitate a little bit
> to design something without a use case in mind, but maybe you have
> one?

It just seemed inconsistent to me, but I think I follow your argument
as to why just adding it to the set isn't correct. I don't have a
particular use case beyond advice/hint application in mind, so if this
works in your assessment and is not an oversight, that sounds good to
me.

>
> > For 0003:
> >
> > I also find the "cars" variable suffix a bit hard to understand, but not sure a comment next to the variables is that useful.
> > Separately, the noise generated by all the additional "_cars" variables isn't great.
> >
> > I wonder a little bit if we couldn't introduce a better abstraction here, e.g. a struct "AppendPathInput" that contains the
> > two related lists, and gets populated by accumulate_append_subpath/get_singleton_append_subpath and then
> > passed to create_append_path as a single argument.
>
> I spent some time thinking about this day and haven't been quite able
> to come up with something that I like. The problem is that
> pa_partial_subpaths and pa_nonpartial_subpaths share a single
> child_append_relid_sets variable, namely pa_subpath_cars, and
> accumulate_append_subpaths gets called with that as the last argument
> and different things for the previous two. One thing I tried was
> making the AppendPathInput struct contain three lists rather than two,
> but then accumulate_append_subpath() needs an argument that makes it
> work in one of three different modes:
>
> Mode 1: normal -- add everything to the "normal" list
> Mode 2: building parallel-aware append with partial path -- add things
> to the "normal" list except for parallel-aware appends which need to
> be split between the normal and special lists
> Mode 3: building parallel-aware append with non-partial path -- add
> things to the "special" list
>

Yeah, the difference in these modes makes this a bit challenging.

I wonder a bit if we shouldn't instead focus on this being about the
inputs to create_append_path (and the 4 different variants of calling
it in add_paths_to_append_rel), and make sure we group some of them
together in a struct, but still pass the individual fields of that
struct to accumulate_append_subpaths.

I've sketched out what I mean in the attached (once as a patch on top
of v8, and then again as a separate patch that's combined with
v8/0003). That makes add_paths_to_append_rel easier to understand (to
me at least), at a slight increase in complexity in cases where we
call create_append_path without passing child_append_relid_sets or
partial subpaths.

Thanks,
Lukas

-- 
Lukas Fittl

  61. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-01-12T13:10:00Z

    On Fri, Jan 9, 2026 at 8:53 AM Jakub Wartak
<jakub.wartak@enterprisedb.com> wrote:
> The only thing this hav revealed is what appears to be some tiny
> problem with placing GroupAggregates or am I wrong or is that known
> limitation? (The original plan shows "Partial GroupAggregate" while
> the one using advice is not aware of the need to use it; yet
> contrib/pg_plan_advices/README in "Future Work" indicates it is out of
> scope for now, right?)

This is great news -- and yes, that's out of scope for now.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  62. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-01-12T14:50:03Z

    On Thu, Jan 8, 2026 at 6:29 PM Michael Paquier <michael@paquier.xyz> wrote:
> So you are telling me that I can commit code that deletes code.  Count
> me in.  The project has some merge requests that I've been holding on
> a bit due to what's happening here and because I did not really look
> at the internals that have changed.  It's great to see that you have
> begun an investigation, Lukas.

:-)

> I suspect that this is going to be an incremental integration process,
> and it smells to me that it is going to require more than one major
> release before being able to remove the whole set of hacks that
> pg_hint_plan has been using, particularly with the GUCs, the costing
> and the forced update of the backend routines which is a ugly
> historical hack.  Saying that, I would need to look at the plan
> outputs to be sure, perhaps we would be OK even with slight changes.
> These happen every year, because the plans tested are complex enough
> that some of the sub-paths are changed, but the hints still work
> properly.  This year for v19 we have at least the changes in the
> expression names.

I think it's quite possible that you could do a full rip and replace
with some study of what needs to be added on top of 0004. The core
idea of 0004 is that it adds a field called pgs_mask in several of
places, which allows you to set a bitmask to control which operations
the planner will consider to be enabled. You can set this via some
existing hooks, and it also adds a  a new hook (joinrel_setup_hook)
which is called near the start of add_paths_to_joinrel(). So, you can
set pgs_mask in PlannerGlobal to control planning for the entire
operation, RelOptInfo to control it for a particular rel, or
JoinPathExtraData to set it for a particular joinrel and a particular
choice of outer and inner rel. I am pretty well convinced that this is
a good model: instead of duplicating a bunch of planner code, as
pg_hint_plan currently does, just have a way to tell the existing
planner code what you want it to do.

Now, one fly in the ointment is that pgs_mask is just a mask -- that
is, it gives us space to store 64 related Booleans, but nothing else.
So if we want to store an integer, like a number of parallel workers,
we need a separate field for that. But if that integer can reasonably
be set at the same levels that are possible for pgs_mask, it's really
easy: just look at the places where 0004 adds a pgs_mask field, and
add an integer field as well. There is obviously some limit to the
number of fields we can reasonably add to PlannerGlobal, RelOptInfo,
and JoinPathExtraData, but if it starts to become too much, we could
bundle some or all of them up in a struct. The patch has already
figured out how to get the mask to propagate down to the places where
low-level planning decisions are being made, so it seems worth trying
to piggy-back on that for anything else that we need.

Now, maybe that won't work out for some reason. But on the other hand,
maybe it will work out. I think it's possible that for the price of a
quite small patch adding another field or a couple of fields on top of
pgs_mask and in the same places, you could get rid of a lot more code.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  63. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-01-12T17:13:32Z

    On Sun, Jan 11, 2026 at 3:21 PM Lukas Fittl <lukas@fittl.com> wrote:
> I wonder a bit if we shouldn't instead focus on this being about the
> inputs to create_append_path (and the 4 different variants of calling
> it in add_paths_to_append_rel), and make sure we group some of them
> together in a struct, but still pass the individual fields of that
> struct to accumulate_append_subpaths.
>
> I've sketched out what I mean in the attached

This is a good idea, IMHO. It didn't compile and I made some other
cosmetic fixes, but I like the direction.

Here's v9. Changes:

* Updated all patches with reviewers to date and discussion link. I
have listed the same reviewers for all of 0001-0004, but the list for
0005 is different, based on my reading of the emails on this thread.
If someone feels that I've made the wrong choices about listing them,
please let me know.
* Incorporated Lukas's idea into 0003 and listed him as a co-author.
* Some comment improvements to 0004.

In 0005:

* Disallow JOIN_ORDER() without targets, per feedback from Ajay Pal.
* pgindent
* Incorporate the fix to pgpa_process_unrolled_join() that I
previously posted separately, and update the associated comments.

I now feel that I've had enough substantial feedback on 0003 and 0004
that it wouldn't be crazy to think about committing those, unless
further review turns up problems or something. I think I need more
substantial review on 0001 and 0002 before I could consider moving
forward with those. And 0005 needs a lot more review, and also a lot
more work from my side. Given that, where do I go from here? I don't
think it makes sense to commit 0002 or 0003 without 0001, but 0004 is
mostly independent of 0001-0003. So, I think that if more feedback on
0001 and 0002 is forthcoming soon, I'll probably just wait a bit and
hope to commit 0001-0004 in the current order. If not, I'll rearrange
the series to move 0004 to the front, and plan to commit that first.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  64. Re: pg_plan_advice

    John Naylor <johncnaylorls@gmail.com> — 2026-01-13T11:38:51Z

    On Tue, Jan 13, 2026 at 12:14 AM Robert Haas <robertmhaas@gmail.com> wrote:
> Here's v9. Changes:

This is perhaps the least interesting part of 0005, but since I
committed this API, I thought I'd chime in:

+ /* alias_name may not be NULL */
+ sp_len = fasthash_accum_cstring(&hs, key.alias_name);
+
+ /* partition_name and plan_name, however, can be NULL */
+ if (key.partition_name != NULL)
+ sp_len += fasthash_accum_cstring(&hs, key.partition_name);
+ if (key.plan_name != NULL)
+ sp_len += fasthash_accum_cstring(&hs, key.plan_name);

It looks like it would be helpful if fasthash_accum_cstring just
returned zero when given a NULL string, as in the attached. We could
also do something like add a large number to the hash, but I'm not
sure that's necessary.

+ /*
+ * hashfn_unstable.h recommends using string length as tweak. It's not
+ * clear to me what to do if there are multiple strings, so for now I'm
+ * just using the total of all of the lengths.
+ */
+ return fasthash_final32(&hs, sp_len);

Sounds reasonable, so the patch also documents that.

--
John Naylor
Amazon Web Services

  65. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-01-13T15:09:14Z

    On Tue, Jan 13, 2026 at 6:39 AM John Naylor <johncnaylorls@gmail.com> wrote:
> It looks like it would be helpful if fasthash_accum_cstring just
> returned zero when given a NULL string, as in the attached. We could
> also do something like add a large number to the hash, but I'm not
> sure that's necessary.

I think that pg_plan_advice's requirement are unusual here, so I would
suggest not adding a branch to fasthash_accum_cstring. If this were a
requirement that almost every caller had, then it would make sense to
pay the cost in the common function. But there will probably be a
small performance cost to this, hash function are often called in hot
paths, and pg_plan_advice is, I think, unusual, so I don't really like
doing it given thoe facts.

> + /*
> + * hashfn_unstable.h recommends using string length as tweak. It's not
> + * clear to me what to do if there are multiple strings, so for now I'm
> + * just using the total of all of the lengths.
> + */
> + return fasthash_final32(&hs, sp_len);
>
> Sounds reasonable, so the patch also documents that.

Some kind of comment change here seems useful to me. I wonder whether
it should be generalized even more than this statement. I also wonder
if this is really the optimal strategy. But I definitely agree that
clarifying this in whatever way makes sense is a good idea.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  66. Re: pg_plan_advice

    Jacob Champion <jacob.champion@enterprisedb.com> — 2026-01-13T18:48:36Z

    On Mon, Dec 8, 2025 at 5:18 PM Jacob Champion
<jacob.champion@enterprisedb.com> wrote:
> a) fuzz the parser first, because it's easy and we can get interesting inputs
> b) fuzz the AST utilities, seeded with "successful" corpus members from a)
> c) stare really hard at the corpus of b) and figure out how to
> usefully mutate a PlannedStmt with it

Got stuck a bit at (c). The first two fit very well with my preferred
fuzzer setup, where I mock the world and fuzz the heck out of a tiny
corner of it. But a "mutated plan" would 1) take a lot of time for me
to design and 2) probably be counterproductive if I start chasing
impossible plans.

So I've inverted it, so that the server calls libfuzzer midquery,
instead of libfuzzer driving the code under test. That gives me *real*
plans that I can then hit with a bunch of garbage advice -- but it's
more than an order of magnitude slower, unfortunately, so I have to
seed it with the output of a+b before it gets anywhere, and then I
cannot minimize the corpus (which fills up rapidly with unoptimized
inputs) because libfuzzer isn't driving. I feel like there is
considerable room for improvement here... but I could spend a bunch of
time finding it that is then not spent fuzzing.

--

The first thing found with the new architecture is this:

    -- note that f is not a partitioned table
    SET pg_plan_advice.advice = 'join_order(f/e (f d))';
    EXPLAIN (COSTS OFF, PLAN_ADVICE)
        SELECT * FROM gt_fact f JOIN gt_dim d ON f.dim_id = d.id;
    ERROR: cannot determine RTI for advice target

Test, and a quick guess at expected output, attached.

--Jacob

  67. Re: pg_plan_advice

    Jakub Wartak <jakub.wartak@enterprisedb.com> — 2026-01-14T11:02:11Z

    On Mon, Jan 12, 2026 at 6:13 PM Robert Haas <robertmhaas@gmail.com> wrote:
[..]
> Here's v9. Changes:

OK, so I was thinking v9 is going to be pretty slick ride, however got
some issues inside 0005 :

1) with cassert/debug builds (meson setup build --prefix=/usr/pgsql19
--buildtype=debug -Dcassert=true) I've started getting nonalways
non-deterministic failures for TPC-H Q4 and Q8. That was a somewhat
self-dissolving error (happened fresh after data load when the testing
suite was launched rapidly afterwards), so I've tracked it down to the
autoanalyze gathering stats. So if load the the whole suite of data,
do not run analyze and stay with autovacuum=off (to avoid autoanalyze)
and run the testing query suite, it identified this failure to force
NL instead of HJ in q8 but also *uncovered* runtime error in q4 that
happens with no stats

a) q4.sql (please see attached file for repro). More or less: right
after import I get a hard failure if the earlier recommended advice is
enabled (smells like a bug to me: we shouldn't get any errors even if
advice is bad). This can be solved by ANALYZE, but brought up back by
truncating pg_statistics
ERROR:  unique semijoin found for relids (b 3) but not observed during planning
STATEMENT:  explain (timing off, costs off, settings off, memory off)

a) q8.sql (please see attached file for demo). It is even more
bizarre, happens right after import , fixed by ANALYZE, but even
TRUNCATING pg_statistic doesnt bring back the problem. Pinpointed that
additional pg_clear_relation_stats() triggers the problem back.

2) Somewhat in default buildtype debugoptimized (plain "meson setup
build --prefix=/usr/pgsql19") I'm getting crashes with v9 in
contrib/pg_plan_advice/pgpa_planner.c:pgpa_join_path_setup line 460,
full stack trace attached.

2026-01-14 10:54:04.718 CET [97138] LOG:  client backend (PID 97408)
was terminated by signal 11: Segmentation fault
2026-01-14 10:54:04.718 CET [97138] DETAIL:  Failed process was
running: explain (timing off, costs off, settings off, memory off)
        SELECT
            s_name,
            s_address
        [..]
            AND s_nationkey = n_nationkey
            AND n_name = 'CANADA'
        ORDER BY
            s_name;
2026-01-14 10:54:04.718 CET [97138] LOG:  terminating any other active
server processes

To me it looks like "pps" is NULL and hits "if
(pps->generate_advice_string)" because
GetPlannerGlobalExtensionState() returns NULL because
root->glob->extension_state_allocated is 0 (while planner_extension_id
is also 0). Crash is only happening for q20 and q4, till I've tried
the below fixup which seems to solve it (?) - it's just based on the
fact that all other uses of GetPlannerGlobalExtensionState() seem to
check for NULL:

-               if (pps->generate_advice_string)
+               if (pps != NULL && pps->generate_advice_string)

3) Also so I went ahead runnning the full suite (without and with
ANALYZE statistics) with asan, so with CFLAGS="-O2 -g -ggdb
-fno-sanitize-recover=all -fsanitize=address" and
ASAN_OPTIONS=detect_leaks=0:abort_on_error=1:print_stacktrace=1:disable_coredump=0:strict_string_checks=1:check_initialization_order=1:strict_init_order=1:detect_stack_use_after_return=0
(the last option seem to be critical to avoid hitting max_stack_depth
issues on my gcc - XXX marker here). So it did catch previous issue,
e.g:

[..all queries running fine..]
q19.sql
q1.sql
q20.sql
AddressSanitizer:DEADLYSIGNAL
=================================================================
==153072==ERROR: AddressSanitizer: SEGV on unknown address
0x000000000008 (pc 0x7de84f7822cc bp 0x7ffda327e420 sp 0x7ffda327e130
T0)
==153072==The signal is caused by a READ memory access.
==153072==Hint: address points to the zero page.
    #0 0x7de84f7822cc in pgpa_join_path_setup
../contrib/pg_plan_advice/pgpa_planner.c:460
    #1 0x64bff4d5966d in add_paths_to_joinrel
../src/backend/optimizer/path/joinpath.c:180
    #2 0x64bff4d60a8a in populate_joinrel_with_paths
../src/backend/optimizer/path/joinrels.c:1197
    #3 0x64bff4d63377 in make_join_rel
../src/backend/optimizer/path/joinrels.c:774
[..]

but with the above fixup all seems to clean in multiple scenarios
(stats, no stats) and including basic "ninja test".

3b) XXX - marker:I was looking for a solution and apparently cfbot
farm has those options, so they should be testing it anyway. And this
brings me to a fact, that it maybe could be detected by cfbot, however
the $thread is not registered so cfbot had no chance to see what's
more there? (I'm mainly thinking about any cross-platform issues, if
any).

-J.

  68. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-01-14T22:11:11Z

    On Wed, Jan 14, 2026 at 6:02 AM Jakub Wartak
<jakub.wartak@enterprisedb.com> wrote:
> a) q4.sql (please see attached file for repro). More or less: right
> after import I get a hard failure if the earlier recommended advice is
> enabled (smells like a bug to me: we shouldn't get any errors even if
> advice is bad). This can be solved by ANALYZE, but brought up back by
> truncating pg_statistics
> ERROR:  unique semijoin found for relids (b 3) but not observed during planning
> STATEMENT:  explain (timing off, costs off, settings off, memory off)

Hmm, so the plan tree walker thinks that we did a semijoin between
lineitem and orders by making lineitem unique on the join column and
then performing a regular join. That appears to be correct. But
pgpa_join_path_setup never created a pgpa_join_path_setup for that
possibility, or created one that doesn't actually match up properly to
what was found in the plan tree. Can you check whether a
pgpa_sj_unique_rel gets created in pgpa_join_path_setup, and with what
contents?

> a) q8.sql (please see attached file for demo). It is even more
> bizarre, happens right after import , fixed by ANALYZE, but even
> TRUNCATING pg_statistic doesnt bring back the problem. Pinpointed that
> additional pg_clear_relation_stats() triggers the problem back.

I found this one. I now think that
pgpa_planner_apply_join_path_advice() shouldn't added anything to
jo_permit_indexes when a join method hint implicitly permits a join
order. I simplified your test case to this:

set pg_plan_advice.advice = 'JOIN_ORDER(n1 region customer)
NESTED_LOOP_PLAIN(region)';
explain (costs off, plan_advice)
    SELECT
        n1.n_name AS nation
    FROM
        customer,
        nation n1,
        region
    WHERE
        c_nationkey = n1.n_nationkey
        AND n1.n_regionkey = r_regionkey
        AND r_name = 'AMERICA';

What was happening here is that when we considered a join between
{customer, nation} and region, pgpa_planner_apply_join_path_advice()
said, well, according to the JOIN_ORDER advice, this join order is not
allowed, which is correct. And, according to the NESTED_LOOP_PLAIN
advice, this join order is allowed, which is also correct, because
NESTED_LOOP_PLAIN(region) denies join orders where region is the
driving table, since those would make it impossible to respect the
advice, and this join order doesn't do that. Then, it concludes that
because one piece of advice says the join order is OK and the other
says it isn't, the advice conflicts. This is where I think it's going
off the rails: the NESTED_LOOP_PLAIN() advice should only be allowed
to act as a negative constraint, not a positive one. So what I did is:

diff --git a/contrib/pg_plan_advice/pgpa_planner.c
b/contrib/pg_plan_advice/pgpa_planner.c
index 13f81e9b063..95dd71deb84 100644
--- a/contrib/pg_plan_advice/pgpa_planner.c
+++ b/contrib/pg_plan_advice/pgpa_planner.c
@@ -982,7 +982,6 @@ pgpa_planner_apply_join_path_advice(JoinType
jointype, uint64 *pgs_mask_p,
                 jo_deny_indexes = bms_add_member(jo_deny_indexes, i);
             else if (restrict_method)
             {
-                jo_permit_indexes = bms_add_member(jo_permit_indexes, i);
                 jm_indexes = bms_add_member(jm_indexes, i);
                 if (join_mask != 0 && join_mask != my_join_mask)
                     jm_conflict = true;
@@ -1038,8 +1037,6 @@ pgpa_planner_apply_join_path_advice(JoinType
jointype, uint64 *pgs_mask_p,
                 }
                 else if (advice_unique != jt_unique)
                     jo_deny_indexes = bms_add_member(jo_deny_indexes, i);
-                else
-                    jo_permit_indexes = bms_add_member(jo_permit_indexes, i);
             }
             continue;
         }

> To me it looks like "pps" is NULL and hits "if
> (pps->generate_advice_string)" because
> GetPlannerGlobalExtensionState() returns NULL because
> root->glob->extension_state_allocated is 0 (while planner_extension_id
> is also 0). Crash is only happening for q20 and q4, till I've tried
> the below fixup which seems to solve it (?) - it's just based on the
> fact that all other uses of GetPlannerGlobalExtensionState() seem to
> check for NULL:
>
> -               if (pps->generate_advice_string)
> +               if (pps != NULL && pps->generate_advice_string)

Agreed, will incorporate that fix.

> 3b) XXX - marker:I was looking for a solution and apparently cfbot
> farm has those options, so they should be testing it anyway. And this
> brings me to a fact, that it maybe could be detected by cfbot, however
> the $thread is not registered so cfbot had no chance to see what's
> more there? (I'm mainly thinking about any cross-platform issues, if
> any).

I mean, there is https://commitfest.postgresql.org/patch/6184/

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  69. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-01-14T22:23:21Z

    On Tue, Jan 13, 2026 at 1:48 PM Jacob Champion
<jacob.champion@enterprisedb.com> wrote:
> The first thing found with the new architecture is this:
>
>     -- note that f is not a partitioned table
>     SET pg_plan_advice.advice = 'join_order(f/e (f d))';
>     EXPLAIN (COSTS OFF, PLAN_ADVICE)
>         SELECT * FROM gt_fact f JOIN gt_dim d ON f.dim_id = d.id;
>     ERROR: cannot determine RTI for advice target
>
> Test, and a quick guess at expected output, attached.

Thanks. There are two separate bugs here. One is that
pgpa_walker_get_rti() is completely wrong-headed in thinking that only
system-generated advice should reach that function, and therefore that
it doesn't need to deal with 0 return values from
pgpa_compute_rti_from_identifier(). I've deleted pgpa_walker_get_rti()
and made the code that called it instead call
pgpa_compute_rti_from_identifier() and deal with 0 return values. That
revealed a second bug, which is that it thought that the
join_order(f/e (f d)) advice was fully matched, despite f/e not
existing in the query. That turns out to be because
pgpa_join_order_permits_join() was doing entry->flags |=
PGPA_TE_MATCH_FULL even when processing a sublist -- so the fact that
it found (f d) in the query made it think that it had matched the
entire join order specification, when in reality it had only matched
the entirety of a sublist. With that fixed, plan_advice.advice =
'join_order(f/d1 (d1 d2))' produces this:

+ Nested Loop
+   Disabled: true
+   Join Filter: ((d1.id = f.dim1_id) AND (d2.id = f.dim2_id))
+   ->  Nested Loop
+         ->  Seq Scan on jo_dim1 d1
+               Filter: (val1 = 1)
+         ->  Materialize
+               ->  Seq Scan on jo_dim2 d2
+                     Filter: (val2 = 1)
+   ->  Seq Scan on jo_fact f
+ Supplied Plan Advice:
+   JOIN_ORDER(f/d1 (d1 d2)) /* partially matched */
+ Generated Plan Advice:
+   JOIN_ORDER(d1 d2 f)
+   NESTED_LOOP_PLAIN(f)
+   NESTED_LOOP_MATERIALIZE(d2)
+   SEQ_SCAN(d1 d2 f)
+   NO_GATHER(f d1 d2)

This is because we don't have a global view of whether the join order
is valid. The planner works up from the bottom of the plan tree and
sees that joining f to d1 or d2 first contradicts the (d1 d2) portion
of the JOIN_ORDER advice, so the first join that gets done is the
d1-d2 join, which is not disabled. Joining the result to f also
contradicts the JOIN_ORDER advice, but there's no alternative to
consider so the planner picks the disabled path as the only option.

I was hoping to get a new version of the patch set with fixes for
these issues out today, but I've run out of day, so I'll have to come
back to that, hopefully tomorrow.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  70. Re: pg_plan_advice

    Jakub Wartak <jakub.wartak@enterprisedb.com> — 2026-01-15T12:28:30Z

    On Wed, Jan 14, 2026 at 11:11 PM Robert Haas <robertmhaas@gmail.com> wrote:

Hi Robert,

> On Wed, Jan 14, 2026 at 6:02 AM Jakub Wartak
> <jakub.wartak@enterprisedb.com> wrote:
> > a) q4.sql (please see attached file for repro). More or less: right
> > after import I get a hard failure if the earlier recommended advice is
> > enabled (smells like a bug to me: we shouldn't get any errors even if
> > advice is bad). This can be solved by ANALYZE, but brought up back by
> > truncating pg_statistics
> > ERROR:  unique semijoin found for relids (b 3) but not observed during planning
> > STATEMENT:  explain (timing off, costs off, settings off, memory off)
>
> Hmm, so the plan tree walker thinks that we did a semijoin between
> lineitem and orders by making lineitem unique on the join column and
> then performing a regular join. That appears to be correct. But
> pgpa_join_path_setup never created a pgpa_join_path_setup for that
> possibility, or created one that doesn't actually match up properly to
> what was found in the plan tree. Can you check whether a
> pgpa_sj_unique_rel gets created in pgpa_join_path_setup, and with what
> contents?

OK, so today, on just barebone v9 (even without any fixes from this $subthread),
I couldn't get it to reproduce right out of the box right on the fresh
cluster. It
appears to another missing piece of the puzzle was to have
max_parallel_workers_per_gather=0  (in addition to TRUNCATING pg_statistic),
because otherwise it did not want to generate advice out of the box that would
generate this specific ERROR.

Maybe I'm big rookie here (OR just dumb), but it took me some time to realize
why we emit SEMIJOIN_UNIQUE() there, clearly the plan without parallelism
has "Nested Loop", not like "Nested Loop Semi Join"
(with max_parallel_workers_per_gather = 2). Yet it emits that and somehow
later the query feature walker->query_features[PGPAQF_SEMIJOIN_UNIQUE]
also is there, so that "unique semijoin found for.." error could be thrown.

As per VERBOSE explain, one can spot this SemiJoin transformation is being
applied (Nested Loop/Inner Unique: true), however sj_unique_rtis is empty (?!):

[..]
NOTICE:  jointype=outer pps_NULL?=0
NOTICE:  added SEMIJOIN_UNIQUE
NOTICE:  pgpa_plan_walker: walking over SEMIJOIN_UNIQUE features: 3,
sj_unique_rtis=<> sj_unique_rels=<>
ERROR:  unique semijoin found for relids (b 3) but not observed during planning

Only *after* this, I've realized how hard all of that is reading comments nearby
`typedef struct pgpa_sj_unique_rel`.

Anyway it appears that pgpa_plan_walker()/pgpa_planner_walker() is not
having proper
input information to begin with about SJs? It looks there is just one
single place that
sets pps->sj_unique_rels (lappend() in pgpa_join_path_setup()), but
that's code path is
only being launched when requesting explain is asking for advice:

explain (costs off, plan_advice) SELECT
[..]
NOTICE:  jointype=inner pps_NULL?=0
NOTICE:  found=0
NOTICE:  not a duplicate, appending "(b 3)" to pps->sj_unique_rels
NOTICE:  jointype=outer pps_NULL?=0
NOTICE:  found=true! (ur->plan_name=(null) bms_ur_relids=3)
NOTICE:  found=1
NOTICE:  jointype=inner pps_NULL?=0
NOTICE:  found=true! (ur->plan_name=(null) bms_ur_relids=3)
NOTICE:  found=1
NOTICE:  jointype=outer pps_NULL?=0
NOTICE:  found=true! (ur->plan_name=(null) bms_ur_relids=3)
NOTICE:  found=1
NOTICE:  added SEMIJOIN_UNIQUE
WARNING:  could not dump unrecognized node type: 0 // ignore?
NOTICE:  pgpa_plan_walker: walking over SEMIJOIN_UNIQUE features: 3,
sj_unique_rtis=((b 3)) sj_unique_rels=({})
(+ no error!)

while with basic EXPLAIN (and advices planner/advises touching SJ
transforms), I'm getting:

dbt3=# explain (costs off) SELECT
[..]
NOTICE:  jointype=inner pps_NULL?=0
NOTICE:  jointype=outer pps_NULL?=0
NOTICE:  jointype=inner pps_NULL?=0
NOTICE:  jointype=outer pps_NULL?=0
NOTICE:  added SEMIJOIN_UNIQUE
NOTICE:  pgpa_plan_walker: walking over SEMIJOIN_UNIQUE features: 3,
sj_unique_rtis=<> sj_unique_rels=<>
ERROR:  unique semijoin found for relids (b 3) but not observed during planning

So we have started v9 with:
  if (pps->generate_advice_string) { -- but that's wrong due to
potential crash in -02 builds + asan complaints

we fixed that above bug with:
  if (pps != NULL && pps->generate_advice_string) {  -- but that's
wrong due to not initializing SJ for normal explains

so we end up doing simply this?
  if (pps != NULL) {

The last one seems to pass all my tests (with already provided fixup
from yesterday), but I'm absolutely not sure if that's the proper way to
address that).

> > a) q8.sql (please see attached file for demo). It is even more
> > bizarre, happens right after import , fixed by ANALYZE, but even
> > TRUNCATING pg_statistic doesnt bring back the problem. Pinpointed that
> > additional pg_clear_relation_stats() triggers the problem back.
>
> I found this one. I now think that
> pgpa_planner_apply_join_path_advice() shouldn't added anything to
> jo_permit_indexes when a join method hint implicitly permits a join
> order. I simplified your test case to this:
>
> set pg_plan_advice.advice = 'JOIN_ORDER(n1 region customer)
> NESTED_LOOP_PLAIN(region)';
> explain (costs off, plan_advice)
>     SELECT
>         n1.n_name AS nation
>     FROM
>         customer,
>         nation n1,
>         region
>     WHERE
>         c_nationkey = n1.n_nationkey
>         AND n1.n_regionkey = r_regionkey
>         AND r_name = 'AMERICA';
>
> What was happening here is that when we considered a join between
> {customer, nation} and region, pgpa_planner_apply_join_path_advice()
> said, well, according to the JOIN_ORDER advice, this join order is not
> allowed, which is correct. And, according to the NESTED_LOOP_PLAIN
> advice, this join order is allowed, which is also correct, because
> NESTED_LOOP_PLAIN(region) denies join orders where region is the
> driving table, since those would make it impossible to respect the
> advice, and this join order doesn't do that. Then, it concludes that
> because one piece of advice says the join order is OK and the other
> says it isn't, the advice conflicts. This is where I think it's going
> off the rails: the NESTED_LOOP_PLAIN() advice should only be allowed
> to act as a negative constraint, not a positive one. So what I did is:

Yes! 3 three lines patches seems to help (and causes no other problems to
best of my knowledge). The simplified test case results seem to be only
changing like below, but it really fixes the Q8 NL->HJ.

  Supplied Plan Advice:
-   JOIN_ORDER(n1 region customer) /* matched, conflicting */
-   NESTED_LOOP_PLAIN(region) /* matched, conflicting */
+   JOIN_ORDER(n1 region customer) /* matched */
+   NESTED_LOOP_PLAIN(region) /* matched */

BTW: I have found also that it fixes another (not yet here disclosed bug,
because I've found it just today :): when running without stats, and
with enable_nestloop=OFF (globally) Q5 was failing too due some
sequence of HJ/Parallel HJ
and slightly different Hash Cond) - nvm, wIth this patch it does NOT misbehave.

Maybe it would be good to include that into tests inside 0005, for that
small tiny query above?

> > 3b) XXX - marker:I was looking for a solution and apparently cfbot
> > farm has those options, so they should be testing it anyway. And this
> > brings me to a fact, that it maybe could be detected by cfbot, however
> > the $thread is not registered so cfbot had no chance to see what's
> > more there? (I'm mainly thinking about any cross-platform issues, if
> > any).
>
> I mean, there is https://commitfest.postgresql.org/patch/6184/

Whoops, mea culpa, I was looking for PG-4 commitfest for some reason
(so I should
be looking on https://cfbot.cputube.org/next.html not just under "/"
[main] one).

-J.

  71. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-01-15T14:40:49Z

    On Thu, Jan 15, 2026 at 7:28 AM Jakub Wartak
<jakub.wartak@enterprisedb.com> wrote:
> The last one seems to pass all my tests (with already provided fixup
> from yesterday), but I'm absolutely not sure if that's the proper way to
> address that).

Thanks. I don't think that's the right fix but the analysis was very
helpful to me in understanding the problem. I think the issue is that
the previous code was confusing "need to generate an advice string"
with "need to walk the plan tree". The latter is a superset of the
former, because we also need to walk the plan tree to generate advice
feedback.

So here's v10. 0001-0004 are unchanged, with the exception that in
0003, I have adjusted accumulate_append_subpath() to incorporate the
absorbed AppendPath's child_append_relid_sets into the surviving
AppendPath's child_append_relid_sets, instead of only the absorbed
AppendPath's relids proper. I don't think this makes any practical
difference to pg_plan_advice, but I might be wrong, and the old way
seems like an obvious oversight. 0005 has a bunch of small fixes,
thanks to all the review comments:

- Fixed mis-spelled Reviewed-by header for Ajay Pal.
- Added Reviewed-by header for John Naylor.
- Added modified version of the test case proposed by Jacob.
- Changed a "break" to "continue" in pgpa_occurrence_number. I don't
think this had any practical impact but it was inconsistent and
contradicted the comment.
- Added a new walk_plan_tree flag to pgpa_planner_state to avoid
getting confused about whether to build pgpa_sj_unique_rel objects.
- Don't let implicit join order constraints arising from join methord
or semijoin uniqueness advice to add to jo_permit_indexes.
- Don't consider join order advice fully matched if we only fully
matched a sublist.
- Remove pgpa_walker_get_rti and properly handle 0 return values from
pgpa_compute_rti_from_identifier instead.

I'm very appreciative to everyone for all the testing and reports
about 0005; I still do need some substantive code review particularly
of 0001.

Thanks,

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  72. Re: pg_plan_advice

    Jacob Champion <jacob.champion@enterprisedb.com> — 2026-01-16T18:14:19Z

    On Wed, Jan 14, 2026 at 2:23 PM Robert Haas <robertmhaas@gmail.com> wrote:
> With that fixed, plan_advice.advice =
> 'join_order(f/d1 (d1 d2))' produces this:

Nice! With v10, the next crash comes from pgpa_walker_would_advise()
(from a code branch that has its own copy of the "cannot determine RTI
for advice target" error, so I assume it's a similar issue?).

Reproducing query:

   SET pg_plan_advice.advice = 'gather( ( ( orders ) ) )';
   EXPLAIN (COSTS OFF, PLAN_ADVICE)
     SELECT o_year FROM (
       SELECT extract(year FROM o_orderdate) AS o_year FROM orders
   );

results in

    TRAP: failed Assert("child_target->ttype ==
PGPA_TARGET_IDENTIFIER"), File:
"../contrib/pg_plan_advice/pgpa_walker.c", Line: 679, PID: 451047

--

I'm going to change my fuzzing focus over to Jakub's TPC-H schema for
a bit, because it doesn't require the corpus to adapt to new
identifiers for each new query, and I can fuzz the interesting queries
he finds directly. :D

--Jacob

  73. Re: pg_plan_advice

    Jakub Wartak <jakub.wartak@enterprisedb.com> — 2026-01-19T10:53:38Z

    On Thu, Jan 15, 2026 at 3:41 PM Robert Haas <robertmhaas@gmail.com> wrote:
>
[..]
>
> So here's v10.
[..]
> I'm very appreciative to everyone for all the testing and reports
> about 0005; I still do need some substantive code review particularly
> of 0001.

Hi,

1. With v10 all my minimal TPC-H checks are OK (both with stats/without stats,
   parallel and non-parallel).

2. I couldn't find any glaring issue during code review of v10-000[124]. But I
   have some questions:
   a) v10-0001 - any example producing such a dummy subplan? (whatever
I've tried I
      cannot come up with one)
   b) v10-0001 - maybe we could add a comment nearby "dummy" struct
member to look
      on pgpa_plan_walker() on example how to use it, but that's part of v5 and
      contrib...
   c) In v10-0004, maybe in pathnodes.h we could use typedef enum rather than
      list of #defines? (see attached)

3. Yes, I could too also repro Jacob's and get the same failure, so it's real:
   TRAP: failed Assert("child_target->ttype == PGPA_TARGET_IDENTIFIER"),
   File: "../contrib/pg_plan_advice/pgpa_walker.c", Line: 679, PID: 32344

4. Some raw perf numbers on non-assert builds (please ignore +/- 3%
jumps), it just hurts
   in one scenario where oq2 drops like 9% of juice (quite expected, it's not
   an issue to be, just posting full results)

tps                           oq1  oq2    oq3  oq4
master                        41   14745  439  435
master+v10-000[1-4]           42   15055  439  432
master+v10full                41   14734  429  437
master+v10full+loaded         42   15014  442  438
master+v10full+loaded+advice  41   13481  424  439

(same but in percentages)
%tps_to_master                oq1  oq2    oq3  oq4
master                        100  100    100  100
master+v10-000[1-4]           102  102    100  99
master+v10full                100  100    98   100
master+v10full+loaded         102  102    101  101
master+v10full+loaded+advice  100  91     97   101


Some explanation:
* oq => my shortcut for Optimizer stress Query (to disambiguate from
TPC-H Queries)
* master+v10full+loaded - shared_preloaded_libraries was set to
  have pg_plan_advice
* master+v10full+loaded+advice - as above, but with system-wide GUC set
  to lengthy and irrelevant (as none of the queries used such aliases)
     JOIN_ORDER(x1 x2 x3 x4 x5 x6 x7 x8 x9 x10 x11)
     NESTED_LOOP_PLAIN(x2 x3 x4 x5 x6 x7 x8 x9 x10 x11)
     SEQ_SCAN(x1 x2 x3 x4 x5 x6 x7 x8 x9 x10 x11)
     NO_GATHER(x1 x2 x3 x4 x5 x6 x7 x8 x9 x10 x11)
  The idea was to see how that impacts oq1..4 while not using those.

So out of curiosity the oq2 on 1 CPU core behavior looks like below:
- no advices --> ~1000 TPS
- enabled pg_plan_advice.advice to lengthy, but unrelated thing and it
gets ~890TPS
- in both cases (empty and set) the bottleneck seems to in palloc0, but
    empty plan_advice: it's more like palloc0() <- newNode() <-
create_index_path()
    <- build_index_paths()
    with plan_advice set: palloc0() <- newNode() <- create_nestloop_path() ..
- so if anything people should not put something there blindly, but just SET
  and RESET afterwards (unless we get pinning of SQL plan id to advices) as
  this might have cost in high-TPS scenarios.

-- details about suite for benchmarking:
SELECT 'CREATE TABLE t' || g || ' (id int primary key, val int)'
FROM generate_series(1, 11) g;
\gexec
-- 1k parts
CREATE TABLE tstresspart (id int, val text) PARTITION BY RANGE (id);
SELECT 'CREATE TABLE tpart' || g || ' PARTITION OF tstresspart FOR
VALUES FROM ('
|| g*10 || ') TO (' || (g+1)*10 || ')' FROM generate_series(1, 1000) g;
\gexec

-- oq1, obtakes ~500ms, below GEQO threshold
EXPLAIN SELECT * FROM t1, t2, t3, t4, t5, t6, t7, t8, t9, t10, t11
WHERE t1.id = t2.id AND t2.id = t3.id AND t3.id = t4.id
  AND t4.id = t5.id AND t5.id = t6.id AND t6.id = t7.id
  AND t7.id = t8.id AND t8.id = t9.id AND t9.id = t10.id
  AND t10.id = t11.id;

-- oq2, hit nested subqueries hard
EXPLAIN SELECT * FROM t1
    WHERE id IN (SELECT id FROM t2
    WHERE id IN (SELECT id FROM t3
    WHERE id IN (SELECT id FROM t4
    WHERE id IN (SELECT id FROM t5
    WHERE id IN (SELECT id FROM t6
    WHERE id IN (SELECT id FROM t7
    WHERE id IN (SELECT id FROM t8
    WHERE id IN (SELECT id FROM t9
    WHERE id IN (SELECT id FROM t10
    WHERE id IN (SELECT id FROM t11))))))))))
OR id IN (SELECT val FROM t1);

-- oq3, part stress test, no part pruning
EXPLAIN SELECT * FROM tstresspart WHERE id = (SELECT (random()*1000));

-- oq4, stress test IN/VALUES
perl -e 'print "SELECT * FROM t1 WHERE id IN ("; for(1..40000)
{ print "$_"; print "," if $_ != 40000 }; print ");"' > oq4.sql

-J.

  74. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-01-19T18:59:28Z

    On Fri, Jan 16, 2026 at 1:14 PM Jacob Champion
<jacob.champion@enterprisedb.com> wrote:
> Nice! With v10, the next crash comes from pgpa_walker_would_advise()
> (from a code branch that has its own copy of the "cannot determine RTI
> for advice target" error, so I assume it's a similar issue?).

Actually, GATHER(((x))) should be rejected as invalid syntax. I
believe this is the correct fix:

-generic_sublist: '(' generic_target_list ')'
+generic_sublist: '(' simple_target_list ')'

If you say GATHER((a b c)), that means "put a Gather node on top of
the join between a, b, and c". If you say GATHER(a b c), that means
"put a Gather node on top of each of a, b, and c individually" i.e.
create a plan with three completely separate Gather nodes. GATHER(((a
b c))) is meaningless. The only plan advice type where we need
arbitrarily deep nesting is JOIN_ORDER(), because there we use
sublists to denote bushy or right-deep joins. Otherwise, we should be
limited to no sublists at all for things like SEQ_SCAN and NO_GATHER,
and to a single level for most other advice tags.

Fixup patch attached. Thanks VERY much for the continued testing.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  75. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-01-19T19:59:57Z

    On Mon, Jan 19, 2026 at 5:53 AM Jakub Wartak
<jakub.wartak@enterprisedb.com> wrote:
> 2. I couldn't find any glaring issue during code review of v10-000[124]. But I
>    have some questions:
>    a) v10-0001 - any example producing such a dummy subplan? (whatever
> I've tried I
>       cannot come up with one)

If you just add something like this, you can see lots of examples from
the regression tests:

         /* When recursing = true, it's an unplanned or dummy subquery. */
         rtinfo->dummy = recursing;
+        if (rtinfo->dummy)
+            elog(WARNING, "hey look i'm a dummy");

Here's a stripped down example that doesn't require the regression database:

EXPLAIN SELECT * FROM random() UNION SELECT * FROM random() WHERE false;

>    b) v10-0001 - maybe we could add a comment nearby "dummy" struct
> member to look
>       on pgpa_plan_walker() on example how to use it, but that's part of v5 and
>       contrib...

Maybe. I think people should know how to grep for stuff like that, and
I don't want to introduce forward dependencies.

>    c) In v10-0004, maybe in pathnodes.h we could use typedef enum rather than
>       list of #defines? (see attached)

I personally hate that style and I think Andres loves it. Whee!

> 3. Yes, I could too also repro Jacob's and get the same failure, so it's real:
>    TRAP: failed Assert("child_target->ttype == PGPA_TARGET_IDENTIFIER"),
>    File: "../contrib/pg_plan_advice/pgpa_walker.c", Line: 679, PID: 32344

I have responded to that separately.

> 4. Some raw perf numbers on non-assert builds (please ignore +/- 3%
> jumps), it just hurts
>    in one scenario where oq2 drops like 9% of juice (quite expected, it's not
>    an issue to be, just posting full results)
>
> tps                           oq1  oq2    oq3  oq4
> master                        41   14745  439  435
> master+v10-000[1-4]           42   15055  439  432
> master+v10full                41   14734  429  437
> master+v10full+loaded         42   15014  442  438
> master+v10full+loaded+advice  41   13481  424  439
>
> (same but in percentages)
> %tps_to_master                oq1  oq2    oq3  oq4
> master                        100  100    100  100
> master+v10-000[1-4]           102  102    100  99
> master+v10full                100  100    98   100
> master+v10full+loaded         102  102    101  101
> master+v10full+loaded+advice  100  91     97   101

I think these numbers look pretty good. I mean, there is obviously
room for improvement. We should look at where the CPU cycles are going
in the oq2 case and try to optimize. But even without that, it's not
terrible, IMHO.

> So out of curiosity the oq2 on 1 CPU core behavior looks like below:
> - no advices --> ~1000 TPS
> - enabled pg_plan_advice.advice to lengthy, but unrelated thing and it
> gets ~890TPS

I'm not sure exactly where the CPU cycles are going here, but one
known problem is that we have to re-parse the advice string for every
query. This thread discusses the challenges of creating some
infrastructure that would allow us to avoid that:

http://postgr.es/m/f87504a6-9dfd-4467-89de-84232cb54f72@gmail.com

Maybe I should start thinking about other ways to avoid that overhead,
because that thread doesn't seem to be progressing much, but maybe the
reparsing isn't even the main problem.

> - in both cases (empty and set) the bottleneck seems to in palloc0, but
>     empty plan_advice: it's more like palloc0() <- newNode() <-
> create_index_path()
>     <- build_index_paths()
>     with plan_advice set: palloc0() <- newNode() <- create_nestloop_path() ..

I've also seen some palloc-related issues with the patch -- it has to
build some data structures and that does palloc stuff -- but there
shouldn't really be any difference in the code paths you show here.
That's just core code, which should be doing the same thing either way
if the advice is not relevant.

> - so if anything people should not put something there blindly, but just SET
>   and RESET afterwards (unless we get pinning of SQL plan id to advices) as
>   this might have cost in high-TPS scenarios.

Yes, I think that's definitely a potential issue. I'd like the
overhead of this module to be as low as possible, but it's bound to
have at least some overhead, and people will have to decide whether
it's worth it.

--
Robert Haas
EDB: http://www.enterprisedb.com

  76. Re: pg_plan_advice

    Jakub Wartak <jakub.wartak@enterprisedb.com> — 2026-01-20T09:19:48Z

    On Mon, Jan 19, 2026 at 9:00 PM Robert Haas <robertmhaas@gmail.com> wrote:
>
> On Mon, Jan 19, 2026 at 5:53 AM Jakub Wartak
> <jakub.wartak@enterprisedb.com> wrote:
> >    a) v10-0001 - any example producing such a dummy subplan? (whatever
> > I've tried I cannot come up with one)
[..]
> EXPLAIN SELECT * FROM random() UNION SELECT * FROM random() WHERE false;

Oh well, that was easy, thanks! Now I see `RTI 3 (function,
in-from-clause): /  Subplan: setop_2 (dummy)`

I don't have any further insights on v10-[124] other than mentioned earlier.

> >    c) In v10-0004, maybe in pathnodes.h we could use typedef enum rather than
> >       list of #defines? (see attached)
>
> I personally hate that style and I think Andres loves it. Whee!

Oh, ok, nvm, but while two of You we are at this, vim or emacs ? ;)
/me ducks & covers

> > 4. Some raw perf numbers on non-assert builds (please ignore +/- 3%
> > jumps), it just hurts
> >    in one scenario where oq2 drops like 9% of juice (quite expected, it's not
> >    an issue to be, just posting full results)
> >
> > tps                           oq1  oq2    oq3  oq4
> > master                        41   14745  439  435
> > master+v10-000[1-4]           42   15055  439  432
> > master+v10full                41   14734  429  437
> > master+v10full+loaded         42   15014  442  438
> > master+v10full+loaded+advice  41   13481  424  439
> >
> > (same but in percentages)
> > %tps_to_master                oq1  oq2    oq3  oq4
> > master                        100  100    100  100
> > master+v10-000[1-4]           102  102    100  99
> > master+v10full                100  100    98   100
> > master+v10full+loaded         102  102    101  101
> > master+v10full+loaded+advice  100  91     97   101
>
> I think these numbers look pretty good. I mean, there is obviously
> room for improvement. We should look at where the CPU cycles are going
> in the oq2 case and try to optimize. But even without that, it's not
> terrible, IMHO.
>
> > So out of curiosity the oq2 on 1 CPU core behavior looks like below:
> > - no advices --> ~1000 TPS
> > - enabled pg_plan_advice.advice to lengthy, but unrelated thing and it
> > gets ~890TPS
>
> I'm not sure exactly where the CPU cycles are going here, but one
> known problem is that we have to re-parse the advice string for every
> query. This thread discusses the challenges of creating some
> infrastructure that would allow us to avoid that:
>
> http://postgr.es/m/f87504a6-9dfd-4467-89de-84232cb54f72@gmail.com
>
> Maybe I should start thinking about other ways to avoid that overhead,

Meh...

> because that thread doesn't seem to be progressing much, but maybe the
> reparsing isn't even the main problem.
> > - in both cases (empty and set) the bottleneck seems to in palloc0, but
> >     empty plan_advice: it's more like palloc0() <- newNode() <-
> > create_index_path()
> >     <- build_index_paths()
> >     with plan_advice set: palloc0() <- newNode() <- create_nestloop_path() ..
>
> I've also seen some palloc-related issues with the patch -- it has to
> build some data structures and that does palloc stuff -- but there
> shouldn't really be any difference in the code paths you show here.
> That's just core code, which should be doing the same thing either way
> if the advice is not relevant.

Yeah, in both it looks like memory allocation and lots of newNode()
called , quite expected.

> > - so if anything people should not put something there blindly, but just SET
> >   and RESET afterwards (unless we get pinning of SQL plan id to advices) as
> >   this might have cost in high-TPS scenarios.
>
> Yes, I think that's definitely a potential issue. I'd like the
> overhead of this module to be as low as possible, but it's bound to
> have at least some overhead, and people will have to decide whether
> it's worth it.

I think we should simply ignore, and maybe later just note the fact this is
not free with a single sentence in some docs for 0005. I was just curious of the
impact and this was measured using pure EXPLAIN (so without query execution to
measure impact of non-empty pg_plan_advice), I'm assuming that in
properly managed
systems execution part will always dominate the workload anyway and
one should be
using prepared statements anyway.

-J.

  77. Re: pg_plan_advice

    Ajay Pal <ajay.pal.k@gmail.com> — 2026-01-21T11:51:57Z

    Subsequent testing revealed that UNION operations involving constants
which enforce empty subplans result in the generated partition-wise
plan not being recognized by the planner.
Below is the query and output for more details.

CREATE TABLE a_test (id int, category text) PARTITION BY LIST (category);
CREATE TABLE a_active PARTITION OF a_test FOR VALUES IN ('active');
CREATE TABLE a_retired PARTITION OF a_test FOR VALUES IN ('retired');

postgres=# set pg_plan_advice.advice='PARTITIONWISE(unnamed_subquery)';
SET
postgres=# EXPLAIN (PLAN_ADVICE)
SELECT * FROM a_active WHERE id = 1
UNION ALL
SELECT * FROM a_active WHERE id = 2 AND 1=0; -- Constant false forces
                        QUERY PLAN
----------------------------------------------------------
 Seq Scan on a_active  (cost=0.00..25.88 rows=6 width=36)
   Filter: (id = 1)
 Supplied Plan Advice:
   PARTITIONWISE(unnamed_subquery) /* not matched */
 Generated Plan Advice:
   SEQ_SCAN(a_active@unnamed_subquery)
   PARTITIONWISE(unnamed_subquery)
   NO_GATHER(a_active@unnamed_subquery)
(8 rows)

Thanks
Ajay

On Tue, Jan 20, 2026 at 2:50 PM Jakub Wartak
<jakub.wartak@enterprisedb.com> wrote:
>
> On Mon, Jan 19, 2026 at 9:00 PM Robert Haas <robertmhaas@gmail.com> wrote:
> >
> > On Mon, Jan 19, 2026 at 5:53 AM Jakub Wartak
> > <jakub.wartak@enterprisedb.com> wrote:
> > >    a) v10-0001 - any example producing such a dummy subplan? (whatever
> > > I've tried I cannot come up with one)
> [..]
> > EXPLAIN SELECT * FROM random() UNION SELECT * FROM random() WHERE false;
>
> Oh well, that was easy, thanks! Now I see `RTI 3 (function,
> in-from-clause): /  Subplan: setop_2 (dummy)`
>
> I don't have any further insights on v10-[124] other than mentioned earlier.
>
> > >    c) In v10-0004, maybe in pathnodes.h we could use typedef enum rather than
> > >       list of #defines? (see attached)
> >
> > I personally hate that style and I think Andres loves it. Whee!
>
> Oh, ok, nvm, but while two of You we are at this, vim or emacs ? ;)
> /me ducks & covers
>
> > > 4. Some raw perf numbers on non-assert builds (please ignore +/- 3%
> > > jumps), it just hurts
> > >    in one scenario where oq2 drops like 9% of juice (quite expected, it's not
> > >    an issue to be, just posting full results)
> > >
> > > tps                           oq1  oq2    oq3  oq4
> > > master                        41   14745  439  435
> > > master+v10-000[1-4]           42   15055  439  432
> > > master+v10full                41   14734  429  437
> > > master+v10full+loaded         42   15014  442  438
> > > master+v10full+loaded+advice  41   13481  424  439
> > >
> > > (same but in percentages)
> > > %tps_to_master                oq1  oq2    oq3  oq4
> > > master                        100  100    100  100
> > > master+v10-000[1-4]           102  102    100  99
> > > master+v10full                100  100    98   100
> > > master+v10full+loaded         102  102    101  101
> > > master+v10full+loaded+advice  100  91     97   101
> >
> > I think these numbers look pretty good. I mean, there is obviously
> > room for improvement. We should look at where the CPU cycles are going
> > in the oq2 case and try to optimize. But even without that, it's not
> > terrible, IMHO.
> >
> > > So out of curiosity the oq2 on 1 CPU core behavior looks like below:
> > > - no advices --> ~1000 TPS
> > > - enabled pg_plan_advice.advice to lengthy, but unrelated thing and it
> > > gets ~890TPS
> >
> > I'm not sure exactly where the CPU cycles are going here, but one
> > known problem is that we have to re-parse the advice string for every
> > query. This thread discusses the challenges of creating some
> > infrastructure that would allow us to avoid that:
> >
> > http://postgr.es/m/f87504a6-9dfd-4467-89de-84232cb54f72@gmail.com
> >
> > Maybe I should start thinking about other ways to avoid that overhead,
>
> Meh...
>
> > because that thread doesn't seem to be progressing much, but maybe the
> > reparsing isn't even the main problem.
> > > - in both cases (empty and set) the bottleneck seems to in palloc0, but
> > >     empty plan_advice: it's more like palloc0() <- newNode() <-
> > > create_index_path()
> > >     <- build_index_paths()
> > >     with plan_advice set: palloc0() <- newNode() <- create_nestloop_path() ..
> >
> > I've also seen some palloc-related issues with the patch -- it has to
> > build some data structures and that does palloc stuff -- but there
> > shouldn't really be any difference in the code paths you show here.
> > That's just core code, which should be doing the same thing either way
> > if the advice is not relevant.
>
> Yeah, in both it looks like memory allocation and lots of newNode()
> called , quite expected.
>
> > > - so if anything people should not put something there blindly, but just SET
> > >   and RESET afterwards (unless we get pinning of SQL plan id to advices) as
> > >   this might have cost in high-TPS scenarios.
> >
> > Yes, I think that's definitely a potential issue. I'd like the
> > overhead of this module to be as low as possible, but it's bound to
> > have at least some overhead, and people will have to decide whether
> > it's worth it.
>
> I think we should simply ignore, and maybe later just note the fact this is
> not free with a single sentence in some docs for 0005. I was just curious of the
> impact and this was measured using pure EXPLAIN (so without query execution to
> measure impact of non-empty pg_plan_advice), I'm assuming that in
> properly managed
> systems execution part will always dominate the workload anyway and
> one should be
> using prepared statements anyway.
>
> -J.
>
>

  78. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-01-21T19:26:02Z

    On Wed, Jan 21, 2026 at 6:52 AM Ajay Pal <ajay.pal.k@gmail.com> wrote:
> Subsequent testing revealed that UNION operations involving constants
> which enforce empty subplans result in the generated partition-wise
> plan not being recognized by the planner.

Good catch, thank you. Should be fixed in this version.

Changes in v11:

- What was previously 0004 has been promoted to 0001. This is the
pgs_mask stuff to allow extensions to control the planner more easily,
which seems separately committable based on reviews to this point.
- Fixed a bug found by Jacob wherein GATHER() and some other tags
permitted multiple levels of nested sublists instead of, as intended,
only a single one.
- Added a test for that case, too.
- Fixed JOIN_ORDER(justonerel) so that it correctly reports "matched"
instead of incorrectly reporting "partially matched" even when
justonerel was part of a join problem.
- Add tests for specifying a prefix of the join order list.
- Removed an XXX comment from the join order tests that now seems bogus to me.
- Fixed a bug in pg_plan_advice_explain_text_multiline that caused it
to display nothing in text format when the label was empty, instead of
when the value was empty.
- Add COSTS OFF to various tests in syntax.sql. I initially omitted
this because I thought it wouldn't be necessary, but I should know
better.
- Add a test for EXPLAIN (PLAN_ADVICE) in non-text format.
- Add tests for SEMIJOIN_UNIQUE() and SEMIJOIN_NON_UNIQUE().
- Rewrite a bunch of buggy logic related to SEMIJOIN_UNIQUE() and
SEMIJOIN_NON_UNIQUE().
- Show the join type in pg_plan_advice.trace_mask output.
- Per this email from Ajay, apply the unique_nonjoin_rtekind() test to
elided (Merge)Append nodes as we do to non-elided ones.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  79. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-01-26T16:07:55Z

    Here is v12.

The big change in this version is that I've added extensive SGML
documentation for v0005. If the README was a little too low-level for
you, this might work better. If you'd like to view it without
downloading the patch set, I've put it up here:

https://robertmhaas.github.io/postgresql-static/html-pgpa-v12/pgplanadvice.html

Aside from that:

* Added a new GUC pg_plan_advice.always_store_advice_details. Without
that, you can't generate advice or see feedback on supplied advice
when using prepared queries, because we don't know at plan time that
it's right to incur the overhead of generating that stuff, and most of
the time it won't be.
* Revoked privileges on pg_clear_collected_shared_advice() as I had
already done on pg_get_collected_shared_advice().
* Removed a bogus elog(ERROR) in pgpa_walker_would_advise() in favor
of returning 0. I think somebody, likely Jakub, pointed this out
earlier, but I didn't quite absorb what I was being told until I
rediscovered the problem.
* Added a bunch more tests. I think the test coverage is getting
pretty decent now, but it could still use some tests targeting more
complex scenarios and corner cases. If you are curious about the
coverage report, see here:

https://robertmhaas.github.io/postgresql-static/coveragereport-pgpa-v12/contrib/pg_plan_advice/index.html

The low number for pgpa_scanner.l is basically bogus, but I don't know
of a way to make it not bogus. The low number for pgpa_ast.c is due to
a bunch of things related to bitmap scans not being right, which at
this point is, I think, the largest outstanding issue with the patch.
It's probably more interesting to look into ways of covering a few
more lines from pgpa_planner.c and pgpa_walker.c, which is where a lot
of the complexity in this code lives. Also, it would be nice to have
coverage of foreign scan cases, but I'm not quite sure what I need to
do to create tests for this module that also depend on postgres_fdw.
Any tips appreciated.

--
Robert Haas
EDB: http://www.enterprisedb.com

  80. Re: pg_plan_advice

    Ajay Pal <ajay.pal.k@gmail.com> — 2026-01-27T07:49:38Z

    Hi,

with v12 patch, found below observations,

#1 Grouped Hash Join, This forces the join of dim1 and dim2 to happen
first, and then places that resulting set on the inner side of a Hash
Join against fact.
but the planner partially matches the generated advice.

CREATE TABLE fact (f_id int, d1_id int, d2_id int, d3_id int);
CREATE TABLE dim1 (id int PRIMARY KEY, val text);
CREATE TABLE dim2 (id int PRIMARY KEY, val text);
CREATE TABLE dim3 (id int PRIMARY KEY, val text);

INSERT INTO fact SELECT g, g%10, g%10, g%10 FROM generate_series(1, 10000) g;
INSERT INTO dim1 SELECT g, 'd1-'||g FROM generate_series(0, 9) g;
INSERT INTO dim2 SELECT g, 'd2-'||g FROM generate_series(0, 9) g;
INSERT INTO dim3 SELECT g, 'd3-'||g FROM generate_series(0, 9) g;
ANALYZE fact, dim1, dim2, dim3;

-- We want (dim1 JOIN dim2) to be the inner side of a Hash Join
SET LOCAL pg_plan_advice.advice = 'HASH_JOIN((dim1 dim2))';

postgres=*# EXPLAIN (COSTS OFF, PLAN_ADVICE)
SELECT * FROM fact
                  JOIN dim1 ON fact.d1_id = dim1.id
                  JOIN dim2 ON fact.d2_id = dim2.id;
                        QUERY PLAN
-----------------------------------------------------------
 Nested Loop
   Disabled: true
   ->  Nested Loop
         Disabled: true
         ->  Seq Scan on fact
         ->  Index Scan using dim1_pkey on dim1
               Index Cond: (id = fact.d1_id)
   ->  Index Scan using dim2_pkey on dim2
         Index Cond: (id = fact.d2_id)
 Supplied Plan Advice:
   HASH_JOIN((dim1 dim2)) /* partially matched */
 Generated Plan Advice:
   JOIN_ORDER(fact dim1 dim2)
   NESTED_LOOP_PLAIN(dim1 dim2)
   SEQ_SCAN(fact)
   INDEX_SCAN(dim1 public.dim1_pkey dim2 public.dim2_pkey)
   NO_GATHER(fact dim1 dim2)
(17 rows)

#2 Multiple Instances of Same Table in Subqueries, here target the
second instance of dim1 inside the subquery 'sq'. both seq_scan and
index_scan advices are not matching.

SET LOCAL pg_plan_advice.advice = 'SEQ_SCAN(dim1#2@sq)
INDEX_SCAN(dim1@sq dim1_pkey)';

postgres=*# EXPLAIN (COSTS OFF, PLAN_ADVICE)
SELECT * FROM fact
JOIN (
    SELECT a.id FROM dim1 a
    JOIN dim1 b ON a.id = b.id
    OFFSET 0
) sq ON fact.d1_id = sq.id;
                    QUERY PLAN
---------------------------------------------------
 Hash Join
   Hash Cond: (fact.d1_id = b.id)
   ->  Seq Scan on fact
   ->  Hash
         ->  Seq Scan on dim1 b
 Supplied Plan Advice:
   SEQ_SCAN(dim1#2@sq) /* not matched */
   INDEX_SCAN(dim1@sq dim1_pkey) /* not matched */
 Generated Plan Advice:
   JOIN_ORDER(fact sq)
   HASH_JOIN(sq)
   SEQ_SCAN(b@sq fact)
   NO_GATHER(fact b@sq)
(13 rows)

Thanks
Ajay

On Mon, Jan 26, 2026 at 9:38 PM Robert Haas <robertmhaas@gmail.com> wrote:
>
> Here is v12.
>
> The big change in this version is that I've added extensive SGML
> documentation for v0005. If the README was a little too low-level for
> you, this might work better. If you'd like to view it without
> downloading the patch set, I've put it up here:
>
> https://robertmhaas.github.io/postgresql-static/html-pgpa-v12/pgplanadvice.html
>
> Aside from that:
>
> * Added a new GUC pg_plan_advice.always_store_advice_details. Without
> that, you can't generate advice or see feedback on supplied advice
> when using prepared queries, because we don't know at plan time that
> it's right to incur the overhead of generating that stuff, and most of
> the time it won't be.
> * Revoked privileges on pg_clear_collected_shared_advice() as I had
> already done on pg_get_collected_shared_advice().
> * Removed a bogus elog(ERROR) in pgpa_walker_would_advise() in favor
> of returning 0. I think somebody, likely Jakub, pointed this out
> earlier, but I didn't quite absorb what I was being told until I
> rediscovered the problem.
> * Added a bunch more tests. I think the test coverage is getting
> pretty decent now, but it could still use some tests targeting more
> complex scenarios and corner cases. If you are curious about the
> coverage report, see here:
>
> https://robertmhaas.github.io/postgresql-static/coveragereport-pgpa-v12/contrib/pg_plan_advice/index.html
>
> The low number for pgpa_scanner.l is basically bogus, but I don't know
> of a way to make it not bogus. The low number for pgpa_ast.c is due to
> a bunch of things related to bitmap scans not being right, which at
> this point is, I think, the largest outstanding issue with the patch.
> It's probably more interesting to look into ways of covering a few
> more lines from pgpa_planner.c and pgpa_walker.c, which is where a lot
> of the complexity in this code lives. Also, it would be nice to have
> coverage of foreign scan cases, but I'm not quite sure what I need to
> do to create tests for this module that also depend on postgres_fdw.
> Any tips appreciated.
>
> --
> Robert Haas
> EDB: http://www.enterprisedb.com

  81. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-01-27T12:32:11Z

    On Tue, Jan 27, 2026 at 2:49 AM Ajay Pal <ajay.pal.k@gmail.com> wrote:
> #1 Grouped Hash Join, This forces the join of dim1 and dim2 to happen
> first, and then places that resulting set on the inner side of a Hash
> Join against fact.
> but the planner partially matches the generated advice.
>
> -- We want (dim1 JOIN dim2) to be the inner side of a Hash Join
> SET LOCAL pg_plan_advice.advice = 'HASH_JOIN((dim1 dim2))';
>
> postgres=*# EXPLAIN (COSTS OFF, PLAN_ADVICE)
> SELECT * FROM fact
>                   JOIN dim1 ON fact.d1_id = dim1.id
>                   JOIN dim2 ON fact.d2_id = dim2.id;
>                         QUERY PLAN
> -----------------------------------------------------------
>  Nested Loop
>    Disabled: true
>    ->  Nested Loop
>          Disabled: true
>          ->  Seq Scan on fact
>          ->  Index Scan using dim1_pkey on dim1
>                Index Cond: (id = fact.d1_id)
>    ->  Index Scan using dim2_pkey on dim2
>          Index Cond: (id = fact.d2_id)
>  Supplied Plan Advice:
>    HASH_JOIN((dim1 dim2)) /* partially matched */
>  Generated Plan Advice:
>    JOIN_ORDER(fact dim1 dim2)
>    NESTED_LOOP_PLAIN(dim1 dim2)
>    SEQ_SCAN(fact)
>    INDEX_SCAN(dim1 public.dim1_pkey dim2 public.dim2_pkey)
>    NO_GATHER(fact dim1 dim2)
> (17 rows)

Thanks for the report, but this is actually correct behavior. There's
no join clause between dim1 and dim2, so the planner doesn't consider
a dim1-dim2 join. This is a good example of the phenomenon described
in the documentation: you can't force the planner to create an
arbitrary plan that it wouldn't otherwise have considered. I might
tweak the documentation wording a little to try  to mention that this
is another way "partially matched" can happen, but there's no bug
here.

> #2 Multiple Instances of Same Table in Subqueries, here target the
> second instance of dim1 inside the subquery 'sq'. both seq_scan and
> index_scan advices are not matching.
>
> SET LOCAL pg_plan_advice.advice = 'SEQ_SCAN(dim1#2@sq)
> INDEX_SCAN(dim1@sq dim1_pkey)';
>
> postgres=*# EXPLAIN (COSTS OFF, PLAN_ADVICE)
> SELECT * FROM fact
> JOIN (
>     SELECT a.id FROM dim1 a
>     JOIN dim1 b ON a.id = b.id
>     OFFSET 0
> ) sq ON fact.d1_id = sq.id;
>                     QUERY PLAN
> ---------------------------------------------------
>  Hash Join
>    Hash Cond: (fact.d1_id = b.id)
>    ->  Seq Scan on fact
>    ->  Hash
>          ->  Seq Scan on dim1 b
>  Supplied Plan Advice:
>    SEQ_SCAN(dim1#2@sq) /* not matched */
>    INDEX_SCAN(dim1@sq dim1_pkey) /* not matched */
>  Generated Plan Advice:
>    JOIN_ORDER(fact sq)
>    HASH_JOIN(sq)
>    SEQ_SCAN(b@sq fact)
>    NO_GATHER(fact b@sq)
> (13 rows)

I'm not sure what why you expected this to work. You can see what the
correct relation identifiers are from the generated plan advice, and
you've used something else, so it doesn't match. It's documented in
both the SGML documentation and the README that relation identifiers
are based on the relation alias, not the relation name.

In general, this seems like a good to reiterate that this is first and
foremost a plan stability feature. More than anything, these examples
show that if you try to write your own plan advice from scratch to
force a novel plan that the planner has never produced itself, you may
not have much luck. If you do want to try to produce a novel plan, you
should at least look at the generated plan advice and adapt it instead
of starting from scratch. And if you find, when trying to produce a
novel plan, that it doesn't work, you need to consider the possibility
that this is because the optimizer did not ever consider that plan,
and that is why pg_plan_advice is unable to induce the planner to
prefer it. That's not to say there can't be any remaining bugs in
pg_plan_advice; there probably are. But it also is absolutely not a
"write your own plan and do anything you like" feature.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  82. Re: pg_plan_advice

    Ajay Pal <ajay.pal.k@gmail.com> — 2026-01-27T12:48:46Z

    Thank you Robert for clarification.

On Tue, Jan 27, 2026 at 6:02 PM Robert Haas <robertmhaas@gmail.com> wrote:
>
> On Tue, Jan 27, 2026 at 2:49 AM Ajay Pal <ajay.pal.k@gmail.com> wrote:
> > #1 Grouped Hash Join, This forces the join of dim1 and dim2 to happen
> > first, and then places that resulting set on the inner side of a Hash
> > Join against fact.
> > but the planner partially matches the generated advice.
> >
> > -- We want (dim1 JOIN dim2) to be the inner side of a Hash Join
> > SET LOCAL pg_plan_advice.advice = 'HASH_JOIN((dim1 dim2))';
> >
> > postgres=*# EXPLAIN (COSTS OFF, PLAN_ADVICE)
> > SELECT * FROM fact
> >                   JOIN dim1 ON fact.d1_id = dim1.id
> >                   JOIN dim2 ON fact.d2_id = dim2.id;
> >                         QUERY PLAN
> > -----------------------------------------------------------
> >  Nested Loop
> >    Disabled: true
> >    ->  Nested Loop
> >          Disabled: true
> >          ->  Seq Scan on fact
> >          ->  Index Scan using dim1_pkey on dim1
> >                Index Cond: (id = fact.d1_id)
> >    ->  Index Scan using dim2_pkey on dim2
> >          Index Cond: (id = fact.d2_id)
> >  Supplied Plan Advice:
> >    HASH_JOIN((dim1 dim2)) /* partially matched */
> >  Generated Plan Advice:
> >    JOIN_ORDER(fact dim1 dim2)
> >    NESTED_LOOP_PLAIN(dim1 dim2)
> >    SEQ_SCAN(fact)
> >    INDEX_SCAN(dim1 public.dim1_pkey dim2 public.dim2_pkey)
> >    NO_GATHER(fact dim1 dim2)
> > (17 rows)
>
> Thanks for the report, but this is actually correct behavior. There's
> no join clause between dim1 and dim2, so the planner doesn't consider
> a dim1-dim2 join. This is a good example of the phenomenon described
> in the documentation: you can't force the planner to create an
> arbitrary plan that it wouldn't otherwise have considered. I might
> tweak the documentation wording a little to try  to mention that this
> is another way "partially matched" can happen, but there's no bug
> here.
>
> > #2 Multiple Instances of Same Table in Subqueries, here target the
> > second instance of dim1 inside the subquery 'sq'. both seq_scan and
> > index_scan advices are not matching.
> >
> > SET LOCAL pg_plan_advice.advice = 'SEQ_SCAN(dim1#2@sq)
> > INDEX_SCAN(dim1@sq dim1_pkey)';
> >
> > postgres=*# EXPLAIN (COSTS OFF, PLAN_ADVICE)
> > SELECT * FROM fact
> > JOIN (
> >     SELECT a.id FROM dim1 a
> >     JOIN dim1 b ON a.id = b.id
> >     OFFSET 0
> > ) sq ON fact.d1_id = sq.id;
> >                     QUERY PLAN
> > ---------------------------------------------------
> >  Hash Join
> >    Hash Cond: (fact.d1_id = b.id)
> >    ->  Seq Scan on fact
> >    ->  Hash
> >          ->  Seq Scan on dim1 b
> >  Supplied Plan Advice:
> >    SEQ_SCAN(dim1#2@sq) /* not matched */
> >    INDEX_SCAN(dim1@sq dim1_pkey) /* not matched */
> >  Generated Plan Advice:
> >    JOIN_ORDER(fact sq)
> >    HASH_JOIN(sq)
> >    SEQ_SCAN(b@sq fact)
> >    NO_GATHER(fact b@sq)
> > (13 rows)
>
> I'm not sure what why you expected this to work. You can see what the
> correct relation identifiers are from the generated plan advice, and
> you've used something else, so it doesn't match. It's documented in
> both the SGML documentation and the README that relation identifiers
> are based on the relation alias, not the relation name.
>
> In general, this seems like a good to reiterate that this is first and
> foremost a plan stability feature. More than anything, these examples
> show that if you try to write your own plan advice from scratch to
> force a novel plan that the planner has never produced itself, you may
> not have much luck. If you do want to try to produce a novel plan, you
> should at least look at the generated plan advice and adapt it instead
> of starting from scratch. And if you find, when trying to produce a
> novel plan, that it doesn't work, you need to consider the possibility
> that this is because the optimizer did not ever consider that plan,
> and that is why pg_plan_advice is unable to induce the planner to
> prefer it. That's not to say there can't be any remaining bugs in
> pg_plan_advice; there probably are. But it also is absolutely not a
> "write your own plan and do anything you like" feature.
>
> --
> Robert Haas
> EDB: http://www.enterprisedb.com

  83. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-01-28T17:13:39Z

    On Mon, Jan 12, 2026 at 12:13 PM Robert Haas <robertmhaas@gmail.com> wrote:
> If not, I'll rearrange
> the series to move 0004 to the front, and plan to commit that first.

That rearrangement got done in v11, and I've now committed that patch
after fixing a few typos that I found. Cautiously, woohoo, but let's
see if anything breaks.

Here's v13. Aside from dropping the now-committed patch, the other big
change here is that I've fixed the way bitmap heap scans work by
reducing scope. Previously, you could write things like
BITMAP_HEAP_SCAN(foo some_foo_idx) or BITMAP_HEAP_SCAN(foo
&&(foo_a_idx foo_b_idx)) to try to compel a particular choice of index
with a bitmap heap scan; however, it didn't actually work. I've now
removed that, and now all of the arguments to BITMAP_HEAP_SCAN() are
relation identifiers, and it specifies only that they should use some
kind of bitmap heap scan. I'm not sure that this is the right thing to
do, but it might be, for a couple of reasons:

1. The decision as to what should be included in a bitmap path is not
made by the general costing machinery, but by choose_bitmap_and(),
which uses its own special algorithm. A general principle of
pg_plan_advice is that it doesn't let you force the optimizer to
consider options that are rejected for reasons other than cost. This
is a grey area. choose_bitmap_and() considers cost, but it also has
other heuristics to limit the search space. We could allow overriding
the former part of the heuristic but not the latter, but that seems
complicated, and might also make this whole thing even more confusing
to use, since it would be really unclear why you could force some
index combinations and not others. We could also allow forcing any
combination, but that needs a lot of thought, since it deviates from
the general design principle and might open a king-sized can of worms.
Point being, the idea that BITMAP_HEAP_SCAN() has no business allowing
an index specification in the first place is not completely without
merit.

2. The only situations in which we consider multiple actual bitmap
paths (vs. potential bitmap paths, as discussed in the previous point)
is when there are possibly-useful parameterizations that we don't want
to ignore. However, I studied what happens in the core regression
tests, and it seems like we don't really have that many test cases
where just forcing some kind of bitmap heap scan wouldn't be good
enough. To use a parameterized bitmap heap scan path, we have to be
under the inner side of a nested loop with the parameterized rel on
the other side, so if the advice produces any other join order or join
method, then the parameterized paths won't even be considered and
there's only one path to choose from. However, it is true that we
might make the wrong decision about which bitmap path to use on the
inner side of a parameterized nested loop. In the future, that could
be addressed either by adding control over choice of parameterization
or by re-adding something like what I had in earlier patch versions
where you can specify particular indexes. But. I don't think we need
to decide right now which of those things we might ultimately want to
do.

(Another interesting point is that in a healthy number of cases where
we consider both parameterized and unparameterized bitmap paths, we
consider the same indexes or sets of indexes in both cases.)

I've gone ahead and removed the "WIP" designation from the
pg_plan_advice patch in this version. There are still a few areas that
need some more investigation, and I'm sure there are still bugs, but I
feel like the bitmap scan thing was the last really big area where
there was a huge problem staring any potential reviewer right in the
face. The remaining XXX comments are things where review comments
would actually be pretty helpful -- not to tell me that I have an XXX
comment, but to suggest what the resolution might be. Of course, I'll
continue looking into that on my own, as well.

Thanks to all who have reviewed so far, and please keep it coming. I
am especially in need of more code review at this point.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  84. Re: pg_plan_advice

    Zsolt Parragi <zsolt.parragi@percona.com> — 2026-01-28T23:40:49Z

    Hello

Just noticed this in the committed patch, it doesn't seem intentional:
(the last line wasn't part of the patch, probably an accidental leftover)

src/backend/optimizer/ptah/costsize.c:1462

        if (path->parallel_workers == 0)
                enable_mask |= PGS_CONSIDER_NONPARTIAL;
        path->disabled_nodes =
                (baserel->pgs_mask & enable_mask) != enable_mask ? 1 : 0;
        path->disabled_nodes = 0;

  85. Re: pg_plan_advice

    Tom Lane <tgl@sss.pgh.pa.us> — 2026-01-29T02:41:18Z

    Robert Haas <robertmhaas@gmail.com> writes:
> That rearrangement got done in v11, and I've now committed that patch
> after fixing a few typos that I found. Cautiously, woohoo, but let's
> see if anything breaks.

skink's not happy:

=2162940== VALGRINDERROR-BEGIN
==2162940== Conditional jump or move depends on uninitialised value(s)
==2162940==    at 0x434D366: cost_material (costsize.c:2593)
==2162940==    by 0x436E7E9: materialize_finished_plan (createplan.c:6531)
==2162940==    by 0x4383054: standard_planner (planner.c:533)
==2162940==    by 0x4383541: planner (planner.c:324)
==2162940==    by 0x44C50B4: pg_plan_query (postgres.c:905)
==2162940==    by 0x4237444: standard_ExplainOneQuery (explain.c:354)
==2162940==    by 0x42375DF: ExplainOneQuery (explain.c:310)
==2162940==    by 0x4237A7A: ExplainOneUtility (explain.c:458)
==2162940==    by 0x42375C3: ExplainOneQuery (explain.c:301)
==2162940==    by 0x4237745: ExplainQuery (explain.c:224)
==2162940==    by 0x44CC834: standard_ProcessUtility (utility.c:868)
==2162940==    by 0x44CD09D: ProcessUtility (utility.c:525)
==2162940==  Uninitialised value was created by a stack allocation
==2162940==    at 0x436E76A: materialize_finished_plan (createplan.c:6506)
==2162940== 
==2162940== VALGRINDERROR-END

			regards, tom lane

  86. Re: pg_plan_advice

    David G. Johnston <david.g.johnston@gmail.com> — 2026-01-29T04:58:26Z

    On Wed, Jan 28, 2026 at 10:14 AM Robert Haas <robertmhaas@gmail.com> wrote:

> On Mon, Jan 12, 2026 at 12:13 PM Robert Haas <robertmhaas@gmail.com>
> wrote:
> > If not, I'll rearrange
> > the series to move 0004 to the front, and plan to commit that first.
>
> That rearrangement got done in v11, and I've now committed that patch
> after fixing a few typos that I found. Cautiously, woohoo, but let's
> see if anything breaks.
>
> Here's v13.
>

Documentation review patch attached.

Overall it flowed well.  Most of the changes are just typos or similar.

I found the idea of seeing "Disabled: true" to not make immediate sense.
I've left some comments near that section and plan to come back to it.

The comment about collector memory usage was duplicated more-or-less.  I
kept the first, less loud, instance.

Removed the few uses of "we" in favor of pg_plan_advice being the actor.

David J.

  87. Re: pg_plan_advice

    Lukas Fittl <lukas@fittl.com> — 2026-01-29T05:44:26Z

    > Robert Haas <robertmhaas@gmail.com> writes:
> > That rearrangement got done in v11, and I've now committed that patch
> > after fixing a few typos that I found. Cautiously, woohoo, but let's
> > see if anything breaks.

Yay!

On Wed, Jan 28, 2026 at 6:41 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> skink's not happy:
>
> =2162940== VALGRINDERROR-BEGIN
> ==2162940== Conditional jump or move depends on uninitialised value(s)
> ==2162940==    at 0x434D366: cost_material (costsize.c:2593)
> ==2162940==    by 0x436E7E9: materialize_finished_plan (createplan.c:6531)
> ==2162940==    by 0x4383054: standard_planner (planner.c:533)
> ...
> ==2162940==    by 0x44CD09D: ProcessUtility (utility.c:525)
> ==2162940==  Uninitialised value was created by a stack allocation
> ==2162940==    at 0x436E76A: materialize_finished_plan (createplan.c:6506)
>

From a quick look, I think that's a missing initialization of
path->parallel_workers when cost_material gets called with the dummy
path created in materialize_finished_plan.

The indirect relationship between those functions doesn't seem great
(i.e. anything that gets read in cost_material has to be initialized
in materialize_finished_plan). What if we just zero initialize the
dummy Path structure instead, like in the attached?

Thanks,
Lukas

--
Lukas Fittl

  88. Re: pg_plan_advice

    Richard Guo <guofenglinux@gmail.com> — 2026-01-29T06:44:24Z

    > On Wed, Jan 28, 2026 at 6:41 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> > skink's not happy:
> >
> > =2162940== VALGRINDERROR-BEGIN
> > ==2162940== Conditional jump or move depends on uninitialised value(s)
> > ==2162940==    at 0x434D366: cost_material (costsize.c:2593)
> > ==2162940==    by 0x436E7E9: materialize_finished_plan (createplan.c:6531)
> > ==2162940==    by 0x4383054: standard_planner (planner.c:533)
> > ...
> > ==2162940==    by 0x44CD09D: ProcessUtility (utility.c:525)
> > ==2162940==  Uninitialised value was created by a stack allocation
> > ==2162940==    at 0x436E76A: materialize_finished_plan (createplan.c:6506)

FWIW, the following related code in reparameterize_path() seems unsafe
to me:

    spath = reparameterize_path(root, spath,
                                required_outer,
                                loop_count);
    enabled =
        (mpath->path.disabled_nodes <= spath->disabled_nodes);
    if (spath == NULL)
        return NULL;
    return (Path *) create_material_path(rel, spath, enabled);

I think we should access spath->disabled_nodes until after we have
verified that spath is not NULL.

Also, it's not quite clear to me why create_material_path (and
cost_material) requires the "enabled" parameter.  I couldn't find any
comments on this, so it might be worth adding some comments here.

- Richard

  89. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-01-29T13:33:13Z

    On Wed, Jan 28, 2026 at 6:41 PM Zsolt Parragi <zsolt.parragi@percona.com> wrote:
> Just noticed this in the committed patch, it doesn't seem intentional:
> (the last line wasn't part of the patch, probably an accidental leftover)
>
> src/backend/optimizer/ptah/costsize.c:1462
>
>         if (path->parallel_workers == 0)
>                 enable_mask |= PGS_CONSIDER_NONPARTIAL;
>         path->disabled_nodes =
>                 (baserel->pgs_mask & enable_mask) != enable_mask ? 1 : 0;
>         path->disabled_nodes = 0;

Good catch. I have included a fix for this in
4020b370f214315b8c10430301898ac21658143f.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  90. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-01-29T13:34:25Z

    On Wed, Jan 28, 2026 at 9:41 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> skink's not happy:

Thanks for the report. 4020b370f214315b8c10430301898ac21658143f
includes an attempt at a fix.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  91. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-01-29T13:35:43Z

    On Thu, Jan 29, 2026 at 12:45 AM Lukas Fittl <lukas@fittl.com> wrote:
> The indirect relationship between those functions doesn't seem great
> (i.e. anything that gets read in cost_material has to be initialized
> in materialize_finished_plan). What if we just zero initialize the
> dummy Path structure instead, like in the attached?

This might be the right answer, but I didn't want to add more memory
zeroing than needed without discussion, so I have just pushed a
minimal fix for now. Thanks for the analysis.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  92. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-01-29T14:10:11Z

    On Thu, Jan 29, 2026 at 1:44 AM Richard Guo <guofenglinux@gmail.com> wrote:
> I think we should access spath->disabled_nodes until after we have
> verified that spath is not NULL.

Good catch. I have included a fix for this in
4020b370f214315b8c10430301898ac21658143f.

> Also, it's not quite clear to me why create_material_path (and
> cost_material) requires the "enabled" parameter.  I couldn't find any
> comments on this, so it might be worth adding some comments here.

I think cost_material() got an enabled argument because of
materialize_finished_plan(). Most of the time, plan nodes are created
from paths, and we want to use the parent's pgs_mask to determine
whether the chosen strategy (e.g. materialization) is enabled.
However, materialize_finished_plan() creates a plan directly, so
there's no RelOptInfo or JoinPathExtraData whose pgs_mask we can
consult, and so we have to fall back on consulting enable_material
directly. Once that parameter got added to cost_material(), it made
sense to me to add it to create_material_path() as well.

There are probably other ways I could have done this. For instance, we
could make create_material_path() pass true to cost_material(), and
not have its own "enabled" argument. That would mean that
reparameterize_path() would have to unconditionally pass true. I'm not
sure if that would be a problem. Another idea is to make
cost_material() consult enable_material itself if path->parent is
NULL, so that it wholly encapsulates the calculation locally.

One thing that is a little different about materialization vs. some
other operations is that it's used for multiple purposes. Sometimes we
use it because it's required for correctness, and other times we do it
to reduce cost. And, it can be used to affect cost in different
scenarios. In the Nested Loop case, PGS_NESTLOOP_MATERIALIZE should
control whether we are willing to materialize, but that doesn't apply
to other cases, like build_subplan() or create_mergejoin_plan().
Moreover, these latter cases have not yet been converted to use Paths
-- hopefully they will be converted someday, but maybe not soon. All
of this makes things a little more complicated for materialization vs.
something like a nested loop or a sequential scan, where the logic to
create that thing is all in one place and paths are in use.

But that is not to say that I'm deeply invested in cost_material()
having an "enabled" argument -- that seems like a detail to me that
could go either way. If it seems objectionable, we could change it,
and I think we could probably do so in more than one way and without
really affecting the behavior. Alternatively, we could leave it the
way it is and add a comment clarifying whatever portion of the above
you think is worth documenting in the source. It seemed like a
sufficiently minor detail to me that I did not bother, but that might
have been a mistake.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  93. Re: pg_plan_advice

    Alexandra Wang <alexandra.wang.oss@gmail.com> — 2026-02-02T19:36:47Z

    Hi Robert,

On Wed, Jan 28, 2026 at 9:14 AM Robert Haas <robertmhaas@gmail.com> wrote:

> Thanks to all who have reviewed so far, and please keep it coming. I
> am especially in need of more code review at this point.
>

Thanks for the patches! I’ve reviewed 0001 - 0003 so far; here are my
comments.

0001:
The code looks good to me. However, I feel a bit uneasy about not
seeing a test case for the additional subplan origin display added in
pg_overexplain. Maybe we could add the following test cases to
exercise that code:

-- should show "Subplan: sub"
EXPLAIN (RANGE_TABLE, COSTS OFF)
SELECT * FROM vegetables v,
       (SELECT * FROM vegetables WHERE genus = 'daucus' OFFSET 0) sub;

-- should show "Subplan: unnamed_subquery"
EXPLAIN (RANGE_TABLE, COSTS OFF)
SELECT * FROM vegetables v,
       (SELECT * FROM vegetables WHERE genus = 'daucus' OFFSET 0);

0002:
Looks good to me.

0003:
I see code like this:

@@ -2232,6 +2251,11 @@ accumulate_append_subpath(Path *path, List
**subpaths, List **special_subpaths)
  if (!apath->path.parallel_aware || apath->first_partial_path == 0)
  {
  *subpaths = list_concat(*subpaths, apath->subpaths);
+ *child_append_relid_sets =
+ lappend(*child_append_relid_sets, path->parent->relids);
+ *child_append_relid_sets =
+ list_concat(*child_append_relid_sets,
+ apath->child_append_relid_sets);

in accumulate_append_subpath(), but in get_singleton_append_subpath()
there are only calls to lappend() and no list_concat(). Is that
intentional? Do we also want to concatenate the newly pulled up
child_append_relid_sets with the existing ones in
get_singleton_append_subpath()?

In add_paths_to_append_rel():

@@ -1785,13 +1790,16 @@ add_paths_to_append_rel(PlannerInfo *root,
RelOptInfo *rel,
        {
            Path       *path = (Path *) lfirst(l);
            AppendPath *appendpath;
+           AppendPathInput append = {0};
+
+           append.partial_subpaths = list_make1(path);
+           append.child_append_relid_sets = list_make1(rel->relids);

Could you help me understand why we need to populate
append.child_append_relid_sets here? I don’t see this child rel being
pulled up at this point.

0004:
I’ve only read through the README and documentation so far; I’ll
continue reviewing the code in 0004.

Best,
Alex

-- 
Alexandra Wang
EDB: https://www.enterprisedb.com

  94. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-02-07T16:25:57Z

    Thanks for the review!

On Mon, Feb 2, 2026 at 2:37 PM Alexandra Wang
<alexandra.wang.oss@gmail.com> wrote:
> 0001:
> The code looks good to me. However, I feel a bit uneasy about not
> seeing a test case for the additional subplan origin display added in
> pg_overexplain. Maybe we could add the following test cases to
> exercise that code:

Done.

> 0002:
> Looks good to me.

Cool.

> 0003:
> in accumulate_append_subpath(), but in get_singleton_append_subpath()
> there are only calls to lappend() and no list_concat(). Is that
> intentional? Do we also want to concatenate the newly pulled up
> child_append_relid_sets with the existing ones in
> get_singleton_append_subpath()?

Oh, good catch! Adjusted.

> In add_paths_to_append_rel():
>
> @@ -1785,13 +1790,16 @@ add_paths_to_append_rel(PlannerInfo *root, RelOptInfo *rel,
>         {
>             Path       *path = (Path *) lfirst(l);
>             AppendPath *appendpath;
> +           AppendPathInput append = {0};
> +
> +           append.partial_subpaths = list_make1(path);
> +           append.child_append_relid_sets = list_make1(rel->relids);
>
> Could you help me understand why we need to populate
> append.child_append_relid_sets here? I don’t see this child rel being
> pulled up at this point.

Oops, good point. Adjusted this, too.

> 0004:
> I’ve only read through the README and documentation so far; I’ll
> continue reviewing the code in 0004.

Thanks!

I plan to post an updated patch set shortly.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  95. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-02-07T16:44:39Z

    On Thu, Jan 29, 2026 at 9:10 AM Robert Haas <robertmhaas@gmail.com> wrote:
> I think cost_material() got an enabled argument because of
> materialize_finished_plan(). Most of the time, plan nodes are created
> from paths, and we want to use the parent's pgs_mask to determine
> whether the chosen strategy (e.g. materialization) is enabled.
> However, materialize_finished_plan() creates a plan directly, so
> there's no RelOptInfo or JoinPathExtraData whose pgs_mask we can
> consult, and so we have to fall back on consulting enable_material
> directly. Once that parameter got added to cost_material(), it made
> sense to me to add it to create_material_path() as well.

If this explanation seems a little weak, it's because it was. Here's a
better explanation: the patch added a per-rel flag
PGS_NESTLOOP_MATERIALIZE that enables the use of a Nested Loop join
with an inner Materialize node. When we construct such a path, the
"parent" point for the materialize path points to the rel that is on
the inner side of the join, not the joinrel itself. But the pgs_mask
flags of the joinrel and its inner side could be different. Therefore,
the Materialize node can't just look at path->parent->pgs_mask to
decide whether to mark the node as disabled. So, when I wrote the
patch originally, I added an enabled flag here to make sure that we
pass down the information about whether the join method was enabled at
the joinrel level, since the joinrel's pgs_mask is not otherwise
available to cost_material(). Later on, I discovered the need for
PGS_CONSIDER_NONPARTIAL, but by then I had forgotten why that
"enabled" argument existed and pushed the logic to handle
PGS_CONSIDER_NONPARTIAL into cost_material(). So the logic as
currently committed is buggy in the case where the joinrel and the
innerrel have differing pgs_mask values. match_unsorted_outer() and
consider_parallel_nestloop() have the intention of generating
Materialize nodes only when materialization is enabled, so that the
Materialize nodes are never marked disabled. But with the patch as
committed, a non-partial Nested Loop with inner Materialize can
disable the Materialize node if the inner rel's pgs_mask has
PGS_CONSIDER_NONPARTIAL unset.

The right fix is to make all the decisions about whether the
Materialize nod should be created, and whether it should be enabled,
in the caller, and have none of that logic in cost_material(). I will
post a new patch set shortly, which will include a patch to rectify
this issue.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  96. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-02-07T16:46:17Z

    On Wed, Jan 28, 2026 at 11:59 PM David G. Johnston
<david.g.johnston@gmail.com> wrote:
> Documentation review patch attached.

Thanks for the review. I think I agree with some of these changes but
not all of them, but I haven't quite had time to look through them
yet, so the next version of the patch set will not include any of
them. However, I will come back around to this next week and look
through all of your comments.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  97. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-02-07T17:37:56Z

    Here is a new patch set (v14).

I finally got time to implement an idea I've been thinking about for a
while, which is to run the regression tests planning every query
twice. The first time, we generate plan advice. The second time, we
replan using that advice to guide planning. If everything in the world
is wonderful, this shouldn't cause the regression tests to fail. Of
course, it did, so this patch set includes a bunch of changes to both
make this kind of testing possible and to fix the issues thus
revealed. Here's a rundown of the problems I discovered, some of which
were problems in pg_plan_advice, some of which are problems created by
my recently committed patches, and a few of which are long-standing
planner problems that just haven't been very apparent up until now.

0. I couldn't actually write the test code to do the above because
planner_setup_hook doesn't get cursorOptions as an argument, so having
the planner setup hook call back into planner() wasn't easy to do
correctly. I've added a new preparatory patch (0002) to fix it. I plan
to proceed with this patch quickly unless there are objections, since
planner_setup_hook is brand new and there seems to be no downside to
fixing this quickly.

1. A bunch of tests failed because they use debug_parallel_query=on.
That introduced a single-copy Gather node into the plan, which caused
pg_plan_advice to emit GATHER() advice. When the queries are then
replanned, they use a real Gather node, not a single-copy one. I've
updated pg_plan_advice to disregard single-copy Gather nodes.

2. A bunch of tests failed because pg_plan_advice was assuming, in
some places, that it was the only thing adjusting pgs_mask. But, a lot
of our regression tests run under enable_SOMETHING=false. I adjusted
pg_plan_advice so that it only ever *clears* bits from pgs_mask
instead of sometimes being willing to set them. This means that the
effects of enable_SOMETHING=false and pg_plan_advice are additive,
rather than (as in previous versions) letting pg_plan_advice sometimes
override the GUC value for no particular reason.

3. One test failed because cost_material() tries to check
PGS_CONSIDER_NONPARTIAL instead of relying on the caller to set the
"enabled" flag properly. See
http://postgr.es/m/CA+TgmoawzvCoZAwFS85tE5+c8vBkqgcS8ZstQ_ohjXQ9wGT9sw@mail.gmail.com
or patch 0001 for details and a fix. I plan to commit this one
relatively quickly, too, barring objections.

4. One test failed because setting enable_indexonlyscan=false, or
clearing the equivalent pgs_mask bit, can cause a Bitmap Heap Scan not
to be used. I worked around this in pg_plan_advice, but I think we
should consider doing something about it in the core planner. See
http://postgr.es/m/CA+TgmoZJAH4vA0q3T0t+CnuCXvhXvg+ZcBqs8s_LUJ0D12QhBA@mail.gmail.com
for all the details of what's happening here.

5. One test failed because pg_plan_advice was forcing the use of a
particular index by deleting all the others from rel->indexlist in its
get_relation_info_hook. This is currently the only method we have for
a loadable module to force the use of an index, but the problem is
that it makes the index invisible for all purposes, and this
particular test relies on self-join elimination. The query can't try
to force the use of index #1 without hiding the existence of index #2
from the SJE code, and then we're in trouble. I've also verified that
deleting elements from the index list can break left join removal. So
I think we need a way to tell the planner that we don't want a certain
index to be scanned without telling it that the index doesn't exist at
all. Therefore, I've revived my previous patch to add a 'disabled'
flag to IndexOptInfo. At the time I last proposed that, Tom said it
wasn't needed because we could just delete from the index list, and I
didn't have any reason why we couldn't just do that, so I made
pg_plan_advice work that way. But now I do have a reason, so brought
that patch back as 0006 and adjusted pg_plan_advice to use it.

6. One tests failed because add_partial_path() doesn't consider
startup cost as a figure of merit. There's a comment there written by
me at the time I was introducing parallel query, back in 2016, which
claims it isn't necessary, but by 2020, while looking at incremental
sort, James Coleman had figured out that this was incorrect. In brief,
what happens here is that the query plan doesn't use a sequential
scan, but disabling sequential scans changes the final plan, and in
fact the new plan has a lower cost than the old one. For full details
and proposed patches, see
http://postgr.es/m/CA+TgmobRufbUSksBoxytGJS1P+mQY4rWctCk-d0iAUO6-k9Wrg@mail.gmail.com
-- I've also incorporated those patches into this patch set as 0008
and 0009, so that the new testing framework added in 0010 will
actually pass.

So here's an overview of the new patch set.

0001. New patch: fix PGS_CONSIDER_NONPARTIAL interaction with
Materialize nodes. See point (3) above.
0002. New patch: pass cursorOptions to planner_setup_hook. See point (0) above.
0003-0005. These were 0001-0003 in the previous patch set, and I have
included the changes suggested by Alexandra Wang in her review.
0006. Revived patch from an old thread: Allow extensions to mark an
individual index as disabled. See point (5) above.
0007. pg_plan_advice. This was 0004 in the previous patch set. This
has a few updates:
- It has been adjusted for the new 0001, 0002, and 0006 patches.
- The issues mentioned in points (1), (2), and (4) above have been fixed.
- There's now a system for other plugins to add "advisors", which is a
way for that other module to supply an advice string that can be used
during query planning. Here, I've done that to enable the testing
described above, but it could also be used by a module that wants to
do on-the-fly advice injection.
- Minor cosmetic improvements, mostly reordering some functions in
pgpa_planner.c for (IMHO) better clarity.
0008-0009. My proposed patches to address add_partial_path's failure
to consider startup costs and related bugs I found while
investigating. See point (6) above.
0010. New patch: add src/test/modules/test_plan_advice, implementing
the testing procedure described above.

I think there's quite a bit more that can be done with this new
testing methodology to find additional issues here, and I plan to
pursue that. But I'm fairly happy with the results of this initial
round of testing. On the other hand, it flushed out a bunch of issues
that seem worth fixing. But it also didn't break a hundred things for
a hundred different reasons, which seems good, too.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  98. Re: pg_plan_advice

    Alexandra Wang <alexandra.wang.oss@gmail.com> — 2026-02-09T15:55:17Z

    Hi Robert,

On Sat, Feb 7, 2026 at 9:38 AM Robert Haas <robertmhaas@gmail.com> wrote:
> Here is a new patch set (v14).

Thanks for the patches! 0003 - 0005 look good to me.

Thanks,
Alex
-- 
Alexandra Wang
EDB: https://www.enterprisedb.com

  99. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-02-10T23:03:34Z

    On Mon, Feb 9, 2026 at 10:55 AM Alexandra Wang
<alexandra.wang.oss@gmail.com> wrote:
> On Sat, Feb 7, 2026 at 9:38 AM Robert Haas <robertmhaas@gmail.com> wrote:
> > Here is a new patch set (v14).
>
> Thanks for the patches! 0003 - 0005 look good to me.

I have committed those, as well as 0001 and 0002. Here's v15. The main
patch is now 0002, and has the following changes since the last
version:

- Added a new GUC pg_plan_advice.feedback_warnings, disabled by
default, which can be set to true to produce a warning about plan
advice strings that aren't fully working. (Previously, you had to use
EXPLAIN to get this information.)

- Use get_namespace_name_or_temp, rather than get_name_namespace,
consistently. One use of the latter function crept in, breaking
INDEX_SCAN and INDEX_ONLY_SCAN advice for temporary tables.

- Fix a problem in pgpa_scan.c that could cause spurious NO_GATHER
advice to be generated in certain situations, such as when joins were
proven empty.

- Fix a logic error in the handling of JOIN_ORDER advice that could
cause it to be marked as conflicting with PARTITIONWISE advice when
that was not in reality the case.

- Incorporate documentation corrections from David G. Johnston. I
didn't take all of his suggestions, but I took many of them, sometimes
with some additional wordsmithing on my part.

- Remove a stray comment.

Also a reminder that 0003 and 0004 (previously 0008 and 0009) don't
properly belong to this thread, but I've included them here because
otherwise the tests in the last patch don't pass. See
http://postgr.es/m/CA+TgmobRufbUSksBoxytGJS1P+mQY4rWctCk-d0iAUO6-k9Wrg@mail.gmail.com
for discussion of those patches.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  100. Re: pg_plan_advice

    Ajay Pal <ajay.pal.k@gmail.com> — 2026-02-12T11:41:25Z

    Hi,

pg_plan_advice failed to match JOIN_ORDER advice because the genetic
algorithm never attempts the specific join path requested.

Test SQL:

LOAD 'pg_plan_advice';
SET pg_plan_advice.always_explain_supplied_advice = on;

-- Create enough tables to trigger GEQO (default threshold is 12)
CREATE TABLE t1 (id int); CREATE TABLE t2 (id int); CREATE TABLE t3 (id int);
CREATE TABLE t4 (id int); CREATE TABLE t5 (id int); CREATE TABLE t6 (id int);
CREATE TABLE t7 (id int); CREATE TABLE t8 (id int); CREATE TABLE t9 (id int);
CREATE TABLE t10 (id int); CREATE TABLE t11 (id int); CREATE TABLE t12 (id int);
CREATE TABLE t13 (id int);

-- 1. Force GEQO on
SET geqo = on;
SET geqo_threshold = 12;

-- 2. Run a massive join. Verify if advice is generated.
EXPLAIN (COSTS OFF, PLAN_ADVICE)
SELECT * FROM t1, t2, t3, t4, t5, t6, t7, t8, t9, t10, t11, t12, t13
WHERE t1.id = t2.id AND t2.id = t3.id AND t3.id = t4.id AND t4.id = t5.id
  AND t5.id = t6.id AND t6.id = t7.id AND t7.id = t8.id AND t8.id = t9.id
  AND t9.id = t10.id AND t10.id = t11.id AND t11.id = t12.id AND
t12.id = t13.id;

--3. SET pg_plan_advice.advice = 'JOIN_ORDER(t13 (t5 (t12 (t1 (t6 (t9
(t11 (t10 (t2 (t7 (t4 (t8 t3))))))))))))';
--4. Run Query again

Supplied Plan Advice:
   JOIN_ORDER(t13 (t5 (t12 (t1 (t6 (t9 (t11 (t10 (t2 (t7 (t4 (t8
t3)))))))))))) /* matched, failed */
 Generated Plan Advice:
   JOIN_ORDER(t13 (t5 (t12 (t8 t9 t1 t10 t3 t4 t6 t7 t2 t11))))
   NESTED_LOOP_PLAIN(t9 t1 t10 t3 t4 t6 t7 t2 t11)
   HASH_JOIN((t1 t2 t3 t4 t6 t7 t8 t9 t10 t11) (t1 t2 t3 t4 t6 t7 t8 t9 t10 t11
    t12) (t1 t2 t3 t4 t5 t6 t7 t8 t9 t10 t11 t12))
   SEQ_SCAN(t13 t5 t12 t8 t9 t1 t10 t3 t4 t6 t7 t2 t11)
   NO_GATHER(t1 t2 t3 t4 t5 t6 t7 t8 t9 t10 t11 t12 t13)

Thanks
Ajay

On Wed, Feb 11, 2026 at 4:36 AM Robert Haas <robertmhaas@gmail.com> wrote:
>
> On Mon, Feb 9, 2026 at 10:55 AM Alexandra Wang
> <alexandra.wang.oss@gmail.com> wrote:
> > On Sat, Feb 7, 2026 at 9:38 AM Robert Haas <robertmhaas@gmail.com> wrote:
> > > Here is a new patch set (v14).
> >
> > Thanks for the patches! 0003 - 0005 look good to me.
>
> I have committed those, as well as 0001 and 0002. Here's v15. The main
> patch is now 0002, and has the following changes since the last
> version:
>
> - Added a new GUC pg_plan_advice.feedback_warnings, disabled by
> default, which can be set to true to produce a warning about plan
> advice strings that aren't fully working. (Previously, you had to use
> EXPLAIN to get this information.)
>
> - Use get_namespace_name_or_temp, rather than get_name_namespace,
> consistently. One use of the latter function crept in, breaking
> INDEX_SCAN and INDEX_ONLY_SCAN advice for temporary tables.
>
> - Fix a problem in pgpa_scan.c that could cause spurious NO_GATHER
> advice to be generated in certain situations, such as when joins were
> proven empty.
>
> - Fix a logic error in the handling of JOIN_ORDER advice that could
> cause it to be marked as conflicting with PARTITIONWISE advice when
> that was not in reality the case.
>
> - Incorporate documentation corrections from David G. Johnston. I
> didn't take all of his suggestions, but I took many of them, sometimes
> with some additional wordsmithing on my part.
>
> - Remove a stray comment.
>
> Also a reminder that 0003 and 0004 (previously 0008 and 0009) don't
> properly belong to this thread, but I've included them here because
> otherwise the tests in the last patch don't pass. See
> http://postgr.es/m/CA+TgmobRufbUSksBoxytGJS1P+mQY4rWctCk-d0iAUO6-k9Wrg@mail.gmail.com
> for discussion of those patches.
>
> --
> Robert Haas
> EDB: http://www.enterprisedb.com

  101. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-02-12T13:08:16Z

    On Thu, Feb 12, 2026 at 6:41 AM Ajay Pal <ajay.pal.k@gmail.com> wrote:
> pg_plan_advice failed to match JOIN_ORDER advice because the genetic
> algorithm never attempts the specific join path requested.

Seems expected. It is bad if using GEQO results in a crash or if the
advice cause the expected outcome when the path is considered, but if
the randomness of GEQO causes it not to consider the path the user
wants, then the user either needs to stop using GEQO, or use
less-strict plan advice, or just understand that this kind of outcome
is a possibility.

Thanks,

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  102. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-02-12T19:00:22Z

    On Thu, Feb 12, 2026 at 8:08 AM Robert Haas <robertmhaas@gmail.com> wrote:
> On Thu, Feb 12, 2026 at 6:41 AM Ajay Pal <ajay.pal.k@gmail.com> wrote:
> > pg_plan_advice failed to match JOIN_ORDER advice because the genetic
> > algorithm never attempts the specific join path requested.
>
> Seems expected. It is bad if using GEQO results in a crash or if the
> advice cause the expected outcome when the path is considered, but if

sorry, this should have said "doesn't cause"

> the randomness of GEQO causes it not to consider the path the user
> wants, then the user either needs to stop using GEQO, or use
> less-strict plan advice, or just understand that this kind of outcome
> is a possibility.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  103. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-02-18T19:57:58Z

    On Tue, Feb 10, 2026 at 6:03 PM Robert Haas <robertmhaas@gmail.com> wrote:
> I have committed those, as well as 0001 and 0002. Here's v15.

Here's v16. As a note, my biggest need right now is for some code
review. Testing help is, of course, also still very welcome.

What I have done in this version is run the test_plan_advice machinery
with the pg_plan_advice.feedback_warnings=on and try to fix as many of
the resulting failures as I could. The biggest revelation that I had
as a result of doing that is that it's really unfortunate for this
patch that get_relation_info_hook is only called for RTE_RELATION
baserels. I was somewhat aware of this problem before, but I thought
that it might be a non-critical thing that could be fixed at some
indefinite future point, maybe years in the future. However, this
testing convinced me that this could not really be postponed if I
wanted the patch set to actually work as intended. It's true that
there's nothing to control in terms of scan method for something like
a subquery scan, but you can want to control the placement of or
prevent the occurrence of Gather or Gather Merge nodes in join
problems that involve non-relation RTEs, and there's really no way to
do that with the existing hook placement.

But it seems like there's a pretty easy solution, which is to move the
hook up from get_relation_info to its sole caller, build_simple_rel.
Patch 0002 implements this and has a commit message explaining how
code that currently uses get_relation_info_hook can be modified to
work with the replacement, buld_simple_rel_hook.

The rest of the patch set is as before, except for the main patch, now
0003, which has the following additional changes:

- It has been updated to use build_simple_rel_hook rather than
get_relation_info_hook.

- I have corrected some order of operations problems in
pgpa_walk_recursively(). The previous coding code deliver incorrect
results when an elided SubqueryScan should have been considered the
only RTI-bearing plan node beneath a Gather or Gather Merge.

- I have adjusted pgpa_planner_apply_joinrel_advice() to fix a problem
where GATHER() or GATHER_MERGE() advice could be falsely reported as
conflicting with PARTITIONWISE() advice.

- I have adjusted pgpa_plan_walker() and pgpa_walk_recursively() to
fix a problem with Gather or Gather Merge nodes together with
partitionwise aggregation, where the generated plan advice was
sometimes incorrect.

- I have done some further work on the documentation. Specifically, I
added a list of known limitations, or at least the start of one; I
expanded the discussion of what "matched" and "partially matched"
mean; I expanded the discussion of how NO_GATHER works; and I and
changed some references from "table" to "relation".

- Comment improvements.

As you might or might not guess, a lot of these bug fixes and
documentation updates were indirect consequences of the replacement of
get_relation_info_hook with buld_simple_rel_hook. Once I made that
change, it fixed some problems, but revealed others which I then had
to fix or document. However, some of these fixes are just cleaning up
other things revealed by running test_plan_advice with
pg_plan_advice.feedback_warnings=on.

Even though I have been running test_plan_advice locally with
pg_plan_advice.feedback_warnings=on, I have not enabled that in this
version of the patch set. That's because that still causes 1 test
failure, due to self-join elimination doing something that perhaps it
shouldn't. More about that later, after I've further analyzed it.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  104. Re: pg_plan_advice

    Alexandra Wang <alexandra.wang.oss@gmail.com> — 2026-02-19T06:52:57Z

    Hi Robert,

On Tue, Feb 10, 2026 at 3:03 PM Robert Haas <robertmhaas@gmail.com> wrote:

> I have committed those, as well as 0001 and 0002. Here's v15. The main
> patch is now 0002, and has the following changes since the last
> version:
>

I'm about halfway through reviewing 0002 and only now finally
understand what 0001 is doing. 0001 looks good to me.

This took me a bit to work through. I'll send more feedback by the end
of this week.

Best,
Alex

-- 
Alexandra Wang
EDB: https://www.enterprisedb.com

  105. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-02-19T19:46:38Z

    On Thu, Feb 19, 2026 at 1:53 AM Alexandra Wang
<alexandra.wang.oss@gmail.com> wrote:
> I'm about halfway through reviewing 0002 and only now finally
> understand what 0001 is doing. 0001 looks good to me.

Great!

> This took me a bit to work through. I'll send more feedback by the end
> of this week.

Yeah, it's a huge patch. Thanks for having a look through it. I look
forward to your feedback.

Here's v17. I have committed what was previously 0004 (which Richard
reviewed on another thread) so that is dropped from this version. In
addition, this version fixes the last issue that was causing failures
running test_plan_advice with pg_plan_advice.feedback_warnings=on. In
my previous email, I said that I thought the failure in question might
be caused by some undesirable coding in the self-join elimination
code, but that turned out to be incorrect. Instead, the problem was
that pga_identifier.c was using planenr_rt_fetch() in some places, and
it needs to use rt_fetch() everywhere. That's because
planner_rt_fetch() consults simple_rte_array, which both self-join
elimination and join removal can mutate, while rt_fetch() uses the
Query's rtable directly, and that never changes. Since we're trying to
create stable identifiers, "never changes" is what we want. In
addition to changing that, I also cleaned up a few XXX comments.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  106. Re: pg_plan_advice

    Alexandra Wang <alexandra.wang.oss@gmail.com> — 2026-02-24T01:10:02Z

    Hi Robert,

I've been reviewing the v17-0003 patch and learned a lot along the
way.  The design is well thought out and the implementation looks
solid.  I only have a few minor comments and questions.

While reading the patch, I noticed several places where we infer
planner intent from currently observed plan shapes (e.g., Gather
projection, Agg used for semijoin uniqueness, Append elision, etc.),
which I quote below:

In pgpa_decompose_join():
> /*
>  * Can we see a Result node here, to project above a Gather? So far I've
>  * found no example that behaves that way; rather, the Gather or Gather
>  * Merge is made to project. Hence, don't test is_result_node_with_child()
>  * at this point.
>  */

In pgpa_descend_any_unique():
> else if (IsA(*plan, Agg))
> {
> /*
> * If this is a simple Agg node, then assume it's here to implement
> * semijoin uniqueness. Otherwise, assume it's completing an eager
> * aggregation or partitionwise aggregation operation that began at a
> * higher level of the plan tree.
> *
> * (Note that when we're using an Agg node for uniqueness, there's no
> * need for any case other than AGGSPLIT_SIMPLE, because there's no
> * aggregated column being * computed. However, the fact that
> * AGGSPLIT_SIMPLE is in use doesn't prove that this Agg is here for
> * the semijoin uniqueness. Maybe we should adjust an Agg node to
> * carry a "purpose" field so that code like this can be more certain
> * of its analysis.)
> */
> descend = true;
> sjunique = (((Agg *) *plan)->aggsplit == AGGSPLIT_SIMPLE);
> }

In pgpa_walk_recursively():
>  * Exception: We disregard any single_copy Gather nodes. These are created
>  * by debug_parallel_query, and having them affect the plan advice is
>  * counterproductive, as the result will be to advise the use of a real
>  * Gather node, rather than a single copy one.
>  */
> if (IsA(plan, Gather) && !((Gather *) plan)->single_copy)
> {
> active_query_features =
> lappend(list_copy(active_query_features),
> pgpa_add_feature(walker, PGPAQF_GATHER, plan));
> beneath_any_gather = true;
> }

In pgpa_build_scan():
> /*
>  * If setrefs processing elided an Append or MergeAppend node that had
>  * only one surviving child, it might be a partitionwise operation,
>  * but then this is either a setop over subqueries, or a partitionwise
>  * operation (which might be a scan or a join in reality, but here we
>  * don't care about the distinction and consider it simply a scan).
>  *
>  * A setop over subqueries, or a trivial SubQueryScan that was elided,
>  * is an "ordinary" scan i.e. one for which we need to generate advice
>  * because the planner has not made any meaningful choice.
>  */
> if ((nodetype == T_Append || nodetype == T_MergeAppend) &&
> unique_nonjoin_rtekind(relids,
>   walker->pstmt->rtable) == RTE_RELATION)
> strategy = PGPA_SCAN_PARTITIONWISE;
> else
> strategy = PGPA_SCAN_ORDINARY;

All of the interpretations above look reasonable, but they made me
wonder how modules like this detect planner-behavior drift over time
if we add new plan node types, add or change plan shapes, or even a
new debug GUC.

I realize this isn't specific to this patch, but I'm curious whether
something like monitoring a possible set of parent-child plan node
patterns has ever been considered, to make structural changes easier
to notice when those invariants stop holding. I don't think this is
actionable for this patch, but I'm just recording the thought here
while reading.

---

I ran the pg_plan_advice module tests as well as test_plan_advice, and
everything passed locally.

I also noticed that several tests in pg_plan_advice are not listed in
the Makefile. Specifically, they are semijoin.sql, syntax.sql,
prepared.sql, and local_collector.sql. I ran them manually, and they
all passed.

---

In pgpa_walk_recursively():
> /*
>  * Check the future_query_features list to see whether this was previously
>  * identified as a plan node that needs to be treated as a query feature.
>  * We must do this before handling elided nodes, because if there's an
>  * elided node associated with a future query feature, the RTIs associated
>  * with the elided node should be the only ones attributed to the query
>  * feature.
>  */
> foreach_ptr(pgpa_query_feature, qf, walker->future_query_features)
> {
> if (qf->plan == plan)
> {
> active_query_features = list_copy(active_query_features);
> active_query_features = lappend(active_query_features, qf);
> walker->future_query_features =
> list_delete_ptr(walker->future_query_features, plan);
> break;
> }
> }

Should this be deleting "qf" instead of "plan"? Right now the list is
not emptied after the plan tree walk as intended.

---

In pgpa_decompose_join():
> if (found_any_outer_gather &&
> ...
> if (found_any_inner_gather &&

"found_any_{outer,inner}_gather" should be dereferenced.

---

In pgpa_trim_shared_advice():
> /* Don't leave stale pointers around. */
> memset(&chunk_array[remaining_chunk_count], 0,
>   sizeof(pgpa_shared_advice_chunk *)
>   * (total_chunk_count - remaining_chunk_count));

Should the element size be "sizeof(dsa_pointer)", consistent with the
preceding memmove()?

---

In pg_plan_advice_advice_check_hook():
> if (error != NULL)
> {
> GUC_check_errdetail("Could not parse advice: %s", error); //
> return false;
> }
>
> MemoryContextSwitchTo(oldcontext);
> MemoryContextDelete(tmpcontext);
>
> return true;

If an error occurs, do we leak the memory context? Should we also
switch and delete memory context before returning false?

---

In pg_get_collected_local_advice(), we skip rows when:

> if (!member_can_set_role(userid, ca->userid))
> continue;

Do we need to do the same check for pg_get_collected_shared_advice()?

---

In pgpa_planner_append_feedback(), "StringInfoData buf;" is unused.

---

The "ExplainState *explain_state" field and the "MemoryContext
trove_cxt" field in "struct pgpa_planner_state" are both unused.

Best,
Alex

-- 
Alexandra Wang
EDB: https://www.enterprisedb.com

  107. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-02-26T13:55:56Z

    Thanks, Alex, for the review.

On Mon, Feb 23, 2026 at 8:10 PM Alexandra Wang
<alexandra.wang.oss@gmail.com> wrote:
> While reading the patch, I noticed several places where we infer
> planner intent from currently observed plan shapes (e.g., Gather
> projection, Agg used for semijoin uniqueness, Append elision, etc.),
> which I quote below:
[....]
> All of the interpretations above look reasonable, but they made me
> wonder how modules like this detect planner-behavior drift over time
> if we add new plan node types, add or change plan shapes, or even a
> new debug GUC.

In my personal opinion, this is one of the really key issues for this
patch and deserves more discussion, but it feels like it's been very
difficult to get any senior hackers to pay attention to this patch.
I'm not sure whether that's because it's a huge patch, or they just
don't want to touch it with a ten foot pole, or they're just busy with
their own work, but it's a little hard as the patch author to know how
reasonable the decisions you've made are until you hear what other
people think of them.

In terms of keeping things working, I think that test_plan_advice has
a pretty good chance of catching future breakage. If somebody adds a
new planner optimization that breaks the plan tree walk that
pg_plan_advice does, they should also be adding test cases for it. If
test_plan_advice can't generate plan advice any more after that
change, the patch will cause a test failure there, or if it can
generate plan advice but that plan advice doesn't properly apply to
planning, again a test failure will result. But as to whether
test_plan_advice is a good enough test or whether more things need to
be added for people to feel comfortable, well, that's a question.
Also, there's the issue that even if test_plan_advice is good enough,
or can be made so, that still potentially burdens future patch authors
with the task of updating pg_plan_advice for whatever they do. That's
not unique to this patch, of course; any patch that expands the
capabilities of PostgreSQL in a new direction has that issue to some
degree. To take a few examples from my own past work, consider wait
events or parallel query. But there's still a judgement call to made
about whether the patch adds too much burden for what we get out of
it. Personally, I think that some kind of user control over the
planner is one of the really big things that experienced PostgreSQL
users want, and therefore I'm inclined to believe it's worth it. But
most people are in favor of their own patches, so that's not really
surprising.

> I realize this isn't specific to this patch, but I'm curious whether
> something like monitoring a possible set of parent-child plan node
> patterns has ever been considered, to make structural changes easier
> to notice when those invariants stop holding. I don't think this is
> actionable for this patch, but I'm just recording the thought here
> while reading.

One thing I've been thinking about is that we might want to consider
annotating some planner nodes with a little more information so that
pg_plan_advice doesn't have to infer quite as much. For example, if we
annotated Agg nodes with their purpose (regular agg, eager agg,
partitionwise agg, semijoin uniqueness) and their relid set,
pg_plan_advice would have an easier time figuring out what's going on
and could throw errors if unexpected things happen instead of
delivering wrong results, possibly silently. I have avoided doing that
kind of thing so far because I wanted to keep the core changes as
small as possible, but there's an argument that I've taken too much
risk there. One of the other things that bugs me about this patch is
that if it turns out that there are things in the advice generation
machinery that are not fully reliable, it may not be possible to fix
those without an ABI break. That is, we can fix any problem we might
have by having the core planner publish more information, but we've
been trying really hard lately to avoid changing structure definitions
or function signatures in minor releases, and I can imagine that it
wouldn't be that hard for pg_plan_advice to be shown to have some
problem that can't be fixed any other way. So what happens if I commit
this and then such a bug shows up? Do we compromise on ABI stability,
or do we just leave the bug unfixed for the life time of that release
and fix it in the next major release, or what? I suppose it's hard to
know in the abstract, but this is another area where getting some
opinions from other committers would be awfully useful.

> I also noticed that several tests in pg_plan_advice are not listed in
> the Makefile. Specifically, they are semijoin.sql, syntax.sql,
> prepared.sql, and local_collector.sql. I ran them manually, and they
> all passed.

Yeah, thanks, I actually also found this myself while doing some
refactoring, shortly before you sent this email. Will fix.

> In pgpa_walk_recursively():
> > /*
> >  * Check the future_query_features list to see whether this was previously
> >  * identified as a plan node that needs to be treated as a query feature.
> >  * We must do this before handling elided nodes, because if there's an
> >  * elided node associated with a future query feature, the RTIs associated
> >  * with the elided node should be the only ones attributed to the query
> >  * feature.
> >  */
> > foreach_ptr(pgpa_query_feature, qf, walker->future_query_features)
> > {
> > if (qf->plan == plan)
> > {
> > active_query_features = list_copy(active_query_features);
> > active_query_features = lappend(active_query_features, qf);
> > walker->future_query_features =
> > list_delete_ptr(walker->future_query_features, plan);
> > break;
> > }
> > }
>
> Should this be deleting "qf" instead of "plan"? Right now the list is
> not emptied after the plan tree walk as intended.

Yeah, I think you're right. I don't think that mistake breaks
anything, it just means the list grows needlessly long, but I'll fix
it.

> In pgpa_decompose_join():
> > if (found_any_outer_gather &&
> > ...
> > if (found_any_inner_gather &&
>
> "found_any_{outer,inner}_gather" should be dereferenced.

Yikes, good catch.

> In pgpa_trim_shared_advice():
> > /* Don't leave stale pointers around. */
> > memset(&chunk_array[remaining_chunk_count], 0,
> >   sizeof(pgpa_shared_advice_chunk *)
> >   * (total_chunk_count - remaining_chunk_count));
>
> Should the element size be "sizeof(dsa_pointer)", consistent with the
> preceding memmove()?

Yes, thanks.

> In pg_plan_advice_advice_check_hook():
> > if (error != NULL)
> > {
> > GUC_check_errdetail("Could not parse advice: %s", error); //
> > return false;
> > }
> >
> > MemoryContextSwitchTo(oldcontext);
> > MemoryContextDelete(tmpcontext);
> >
> > return true;
>
> If an error occurs, do we leak the memory context? Should we also
> switch and delete memory context before returning false?

I think this should be mostly harmless. The parent context, at least
in most cases, shouldn't be anything that long-lived, and when it's
destroyed, it will take this context with it. I think there are some
cases where the parent context can be longer-lived, but they shouldn't
happen often enough for this to become a significant issue. On the
other hand, there's also no reason that I can see not to tighten this
up. I changed it like this:

    if (error != NULL)
        GUC_check_errdetail("Could not parse advice: %s", error);

    MemoryContextSwitchTo(oldcontext);
    MemoryContextDelete(tmpcontext);

    return (error == NULL);

> In pg_get_collected_local_advice(), we skip rows when:
>
> > if (!member_can_set_role(userid, ca->userid))
> > continue;
>
> Do we need to do the same check for pg_get_collected_shared_advice()?

Hmm, yeah, probably a good idea.

> In pgpa_planner_append_feedback(), "StringInfoData buf;" is unused.
...
> The "ExplainState *explain_state" field and the "MemoryContext
> trove_cxt" field in "struct pgpa_planner_state" are both unused.

Thanks, deleted this things.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  108. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-02-27T22:46:21Z

    On Thu, Feb 26, 2026 at 8:55 AM Robert Haas <robertmhaas@gmail.com> wrote:
> Thanks, Alex, for the review.

Here's v18. In addition to fixing the problems pointed out by Alex,
there are a couple of significant changes in this version.

First, I realized that it might be confusing to have the collector
interface as part of pg_plan_advice, because for most of what
pg_plan_advice does, you didn't need the extension, but for that part,
you did. So, I broke that part out into its own extension, now called
pg_collect_advice, and put it into a separate patch. I think this is
conceptually cleaner: pg_plan_advice is just the core functionality of
generating and enforcing advice, and anything else is a separate
extension that logically sits on top of that core functionality. This
also has the advantage that you can decide to make the core
functionality available without any chance of somebody getting access
to the collector, if desired.

Second, I also added a third contrib module called pg_stash_advice.
This uses the same hook that I previously added for test_plan_advice,
but unlike that module, this one's not just a test. It lets you set up
an "advice stash" which is basically a query_id->advice_string hash
table. If you then set pg_stash_advice.stash_name to the name of your
advice stash, it will do a lookup into that hash table every time a
query is planned and, if the query ID is found, it will do the
planning with the corresponding advice string. What I think is
particularly cool about this is that it shows that you can really use
that hook to apply advice on the fly in any way you want. I suspect
that query ID matching will be suitable for a lot of use cases, but
you could have a similar module that matches on query text or does
anything else that you want as long as an advice string pops out at
the end. It shows that the core pg_plan_advice infrastructure is
pluggable. So this is both something that I think a lot of people will
find useful all on its own, and also a design pattern that people can
copy and adapt.

Finally, I did a lot of minor cleanups. I originally planned to try to
include a full list in this email, but as the number of fixes got
larger, I eventually realized that would get incredibly tedious, even
for me. An awful lot of what got fixed was just straightforward typos,
but there were also some comments where the wording was garbled and
didn't really make sense, or where I thought the comment should be
longer or shorter than it actually was, or where I used the wrong data
type in the code but in a way that didn't actually break anything.
There are a few fixes to the code, but they're all very minor stuff
that I'm not sure has any real-world consequences, although it might
so it's good to fix it, but the overwhelming bulk of it is cosmetic.

Thanks,

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  109. Re: pg_plan_advice

    David G. Johnston <david.g.johnston@gmail.com> — 2026-02-28T01:16:10Z

    On Fri, Feb 27, 2026 at 3:46 PM Robert Haas <robertmhaas@gmail.com> wrote:

> On Thu, Feb 26, 2026 at 8:55 AM Robert Haas <robertmhaas@gmail.com> wrote:
> > Thanks, Alex, for the review.
>
> Here's v18. In addition to fixing the problems pointed out by Alex,
> there are a couple of significant changes in this version.
>
>
I have a mind to walk through the readmes and sgmls but its going to be in
chunks.  Here's one for the readme for pg_plan_advice with a couple of
preliminary sgml changes.

David J.

  110. Re: pg_plan_advice

    David G. Johnston <david.g.johnston@gmail.com> — 2026-03-02T04:10:52Z

    On Fri, Feb 27, 2026 at 6:16 PM David G. Johnston <
david.g.johnston@gmail.com> wrote:

> On Fri, Feb 27, 2026 at 3:46 PM Robert Haas <robertmhaas@gmail.com> wrote:
>
>> On Thu, Feb 26, 2026 at 8:55 AM Robert Haas <robertmhaas@gmail.com>
>> wrote:
>> > Thanks, Alex, for the review.
>>
>> Here's v18. In addition to fixing the problems pointed out by Alex,
>> there are a couple of significant changes in this version.
>>
>>
> I have a mind to walk through the readmes and sgmls but its going to be in
> chunks.  Here's one for the readme for pg_plan_advice with a couple of
> preliminary sgml changes.
>
>
0003 sgml focus with some readme.

There is an inconsistency between readme and sgml regarding the "join
(strategy|method) advice" label.

The wording for partitionwise is better in the readme than the sgml.

I did make some bulkier suggestions - they do not contain proper markup.

There may be some repeated suggestions from my previous review - I didn't
try to match up what you did and did not take in.

I re-ordered semijoin to be alphabetical - which also had the benefit of
matching the layout of the paragraph.  Flipping the order of "former" and
"latter" is quite intentional.

I defined what "successfully enforced" means in the emit warning GUC.  That
was my unresearched guess after reading how "failed" behaves.

I found "negative join order constraint" challenging to parse.  I tried to
word it more like what is done in the readme.

I don't know if this conflicts with my previous diff of the same patch.  A
couple of overlap spots possibly but they were largely independent (readme
then, sgml now).

David J.

  111. Re: pg_plan_advice

    David G. Johnston <david.g.johnston@gmail.com> — 2026-03-02T22:09:12Z

    On Sun, Mar 1, 2026 at 9:10 PM David G. Johnston <david.g.johnston@gmail.com>
wrote:

> On Fri, Feb 27, 2026 at 6:16 PM David G. Johnston <
> david.g.johnston@gmail.com> wrote:
>
>> On Fri, Feb 27, 2026 at 3:46 PM Robert Haas <robertmhaas@gmail.com>
>> wrote:
>>
>>> On Thu, Feb 26, 2026 at 8:55 AM Robert Haas <robertmhaas@gmail.com>
>>> wrote:
>>> > Thanks, Alex, for the review.
>>>
>>> Here's v18. In addition to fixing the problems pointed out by Alex,
>>> there are a couple of significant changes in this version.
>>>
>>>
>> I have a mind to walk through the readmes and sgmls but its going to be
>> in chunks.  Here's one for the readme for pg_plan_advice with a couple of
>> preliminary sgml changes.
>>
>>
> 0003 sgml focus with some readme.
>
>
And now 0004 sgml (no readme):

My OCD wants these named pg_advice_{plan,collect,stash} so they sort
together.

Strongly thinking using "entries" throughout makes more sense than "query
texts and advice string" - it is shorter and more inclusive since the
actual stored info covers IDs and timestamp.

I made one swap where shared was being mentioned before local.

I added some unresearched answers to open questions I had at the end of the
main section.  Namely, pertaining to advice feedback output and capturing
explain plans themselves.

David J.

  112. Re: pg_plan_advice

    David G. Johnston <david.g.johnston@gmail.com> — 2026-03-02T23:11:17Z

    On Mon, Mar 2, 2026 at 3:09 PM David G. Johnston <david.g.johnston@gmail.com>
wrote:

> On Sun, Mar 1, 2026 at 9:10 PM David G. Johnston <
> david.g.johnston@gmail.com> wrote:
>
>> On Fri, Feb 27, 2026 at 6:16 PM David G. Johnston <
>> david.g.johnston@gmail.com> wrote:
>>
>>> On Fri, Feb 27, 2026 at 3:46 PM Robert Haas <robertmhaas@gmail.com>
>>> wrote:
>>>
>>>> On Thu, Feb 26, 2026 at 8:55 AM Robert Haas <robertmhaas@gmail.com>
>>>> wrote:
>>>> > Thanks, Alex, for the review.
>>>>
>>>> Here's v18. In addition to fixing the problems pointed out by Alex,
>>>> there are a couple of significant changes in this version.
>>>>
>>>>
>>> I have a mind to walk through the readmes and sgmls but its going to be
>>> in chunks.  Here's one for the readme for pg_plan_advice with a couple of
>>> preliminary sgml changes.
>>>
>>>
>> 0003 sgml focus with some readme.
>>
>>
> And now 0004 sgml (no readme):
>
>
Lastly, 0007 sgml (stash)

Placed entry in correct position on contrib page.

Expanded a bit on the security aspect comment.  I suppose there is some
indirect exposure via decisions being made implying table sizes or
records-per-FK...I left that unmentioned.

Added some explicit limitations to user-supplied values.

I would personally like to see "pg_set_stashed_advice" returning something
besides void.  I would usually go for text, but maybe in the interest of
i18n an integer would suffice.  +1 if a row was added, -1 if a row is
removed, or 0 for an update.  That would necessitate any other no-op being
an error.  Presently, supplying NULL for the query_id is not an error -
that seems like an oversight.  Same goes for supplying NULL for
stash_name.  If both of those cases produce errors (leaving NULL
advice_string being a remove indicator) the integer return seems like it
should work just fine.

Sorta feels like this module would appreciate advice strings having a
comment feature - so instead of just leaving an empty string behind saying
"we know, no advice needed" - it could contain actual content that isn't
applied advice.  Given the complexity of planning, comments seem warranted
in the advice themselves in any case.

Feels like this module needs export and import functions, especially given
the intro paragraph about the contents being volatile.  Or maybe a psql
script example producing a dynamic script file involving \gexec.

David J.

  113. Re: pg_plan_advice (now with transparent SQL plan performance overrides - pg_stash_advice)

    Jakub Wartak <jakub.wartak@enterprisedb.com> — 2026-03-03T11:42:13Z

    On Fri, Feb 27, 2026 at 11:46 PM Robert Haas <robertmhaas@gmail.com> wrote:
>
> On Thu, Feb 26, 2026 at 8:55 AM Robert Haas <robertmhaas@gmail.com> wrote:
> Here's v18. [..]
[..]
> Second, I also added a third contrib module called pg_stash_advice.
> This uses the same hook that I previously added for test_plan_advice,
> but unlike that module, this one's not just a test. It lets you set up
> an "advice stash" which is basically a query_id->advice_string hash
> table. If you then set pg_stash_advice.stash_name to the name of your
> advice stash, it will do a lookup into that hash table every time a
> query is planned and, if the query ID is found, it will do the
> planning with the corresponding advice string. What I think is
> particularly cool about this is that it shows that you can really use
> that hook to apply advice on the fly in any way you want. I suspect
> that query ID matching will be suitable for a lot of use cases, but
> you could have a similar module that matches on query text or does
> anything else that you want as long as an advice string pops out at
> the end. It shows that the core pg_plan_advice infrastructure is
> pluggable. So this is both something that I think a lot of people will
> find useful all on its own, and also a design pattern that people can
> copy and adapt.

Hi Robert, I'm glad that you posted this. I've thought this is so important that
I've sligthly added more descriptive subject as this is NEW and easily to miss
out as it was burried in the second paragraph and most folks probably would miss
that this $thread is now includes code for transparent SQL plan
overrides (most known
as baselines or SQL plan management) - it has many names, but I havent come
across "stash" as one of them, so people could miss it too easily.

1. First thought is that I found it quite surprising, we now have 3 modules
and it's might cause confusion and lack of consistency:
- 3 can to be in shared_preload_libraries (pg_stash_advice,
pg_plan_advice, pg_collecti_advice)
- 2 others can use CREATE EXTENSION (pg_stash_advice, pg_collect_advice), but
  "create extension pg_plan_advice;" fails
so maybe they all should behave the same as people (including me) won't read
the docs and just blindly add it here and there and issue CREATE EXTENSION,
but it's going to be hard to remember for which ones (?) So we need more
consistency?

2. Should pgca always duplicate entries like that? (each call gets new entry?)
Shouldn't it just update collection_time for identical calls of
userid/dbid/queryid?

postgres=# select * from pg_get_collected_shared_advice();
 id | userid | dbid |       queryid       |        collection_time
   |                      query                      |
 advice
----+--------+------+---------------------+-------------------------------+-------------------------------------------------+-------------------------------------------
  0 |     10 |    5 | 1069089066624131207 | 2026-03-03
09:47:40.322294+01 | select * from pg_get_collected_shared_advice(); |
NO_GATHER(pg_get_collected_shared_advice)
(1 row)

postgres=# select * from pg_get_collected_shared_advice();
 id | userid | dbid |       queryid       |        collection_time
   |                      query                      |
 advice
----+--------+------+---------------------+-------------------------------+-------------------------------------------------+-------------------------------------------
  0 |     10 |    5 | 1069089066624131207 | 2026-03-03
09:47:40.322294+01 | select * from pg_get_collected_shared_advice(); |
NO_GATHER(pg_get_collected_shared_advice)
  1 |     10 |    5 | 1069089066624131207 | 2026-03-03
09:47:41.983973+01 | select * from pg_get_collected_shared_advice(); |
NO_GATHER(pg_get_collected_shared_advice)
(2 rows)

It floods like that for everything, most visible with couple of independent
starts of pgbench -M prepared. Maybe I'm wrong , but I don't think it worked
like that earlier before refactor?

3. The good news is that I have managed to finally overrride SQL plans
completley transparently using set of those modules for online pgbench runs.
However I found couple of issues.

3a. Because query_id each time will be different for every query even
in standard
pgbench mode, I've managed to achieve it only using prepared statements.
Realistically we'll need some way of modular matching not just on query_id OR
query_id should be changed how it being calculcated or maybe by some other means
(argument is: not all users/apps use prepared statements):
- at least some minimal query jumbling, part of me belives that e.g. code from
  62d712ecfd940 / pgss's generate_normalized_query() should be more reusable
  across other extensions, and then why not just use it here?
- maybe it should: ignore uppercase vs lowercase, removal of extra whitespaces
- maybe it should: ignore schema names (in multi-tenant shops)
- maybe it should: remove SQL comments (/*+  xxx */) or newlines (as there are
  client-side libraries that annotate SQL queries by putting comments to locate
  e.g. app source code location/origin)
- maybe we even should have some regexp

I'm not buying argument that it will make something slower, because :
a) this is on-demand loaded module for those who want it
b) it exists purely to avoid way bigger performance problems in the first place

3b. When performing manual testing using PREPARE s1 / EXECUTE s1, the SQL plans
are effective out of the box. Below I'm intentionally downgrading runtime
performance and that works:

postgres=# PREPARE p1 (int) AS UPDATE pgbench_accounts SET abalance =
abalance + $1 WHERE aid = $2;
PREPARE
postgres=# \timing on
Timing is on.
postgres=# EXECUTE p1(42, 42);
UPDATE 1
Time: 8.241 ms

-- making it slow, queryid from explain (verbose), same session:
postgres=# select pg_set_stashed_advice('abc456',
-9041336337128051785, 'SEQ_SCAN(pgbench_accounts)');
 pg_set_stashed_advice
-----------------------

(1 row)

Time: 0.995 ms
postgres=# show pg_stash_advice.stash_name ;
 pg_stash_advice.stash_name
----------------------------
 abc456
(1 row)

Time: 0.366 ms
-- OK, that's effective (up from 8ms)
postgres=# EXECUTE p1(42, 42);
UPDATE 1
Time: 448.415 ms

3c. However for pgbench -M prepared, such online plan alterations are
strangley not effective. Even creating new stash under different name, setting
GUC, and pg_reloading_conf is not effective too for hundreths of seconds of
pgbench.

3d. ... however restarting restarting pgbench helps and the session
changes plan (and thus
performance characteristics). When using manual way of reproducing
this more like:

sess1=# PREPARE p1 (int) AS UPDATE pgbench_accounts SET abalance =
abalance + $1 WHERE aid = $2;
PREPARE
sess1=# EXECUTE p1(42, 42);
UPDATE 1
sess1=# \timing on
Timing is on.
sess1=# EXECUTE p1(42, 42);
UPDATE 1
Time: 8.823 ms

-- now from 2nd session
sess2=# select pg_set_stashed_advice('abc456', -9041336337128051785,
'SEQ_SCAN(pgbench_accounts)');

-- .. it was NOT effective, and still fast, but it should be slow:
sess1=# EXECUTE p1(42, 42);
UPDATE 1
Time: 8.781 ms
postgres=#

However doing it from sess2 sometimes, makes it often faster effective:
postgres=# select pg_create_advice_stash('xyz123');
postgres=# select pg_set_stashed_advice('xyz123',
-9041336337128051785, 'SEQ_SCAN(pgbench_accounts)');
postgres=# alter system set pg_stash_advice.stash_name TO 'xyz123';
postgres=# select pg_reload_conf();

but apparently that was still not solving the pgbench case problem.
So it doesn't seem to be deterministic when the new plan is applied (?)

3e. As this was pretty concerning, I've repeated the pgbench -M prepared
excercise and I've figured out that I could achieve effect that I wanted (
boucing between fast <-> slow immediatley) by injecting call via gdb on that
backend to InvalidateSystemCaches(). Kind of brute-force, but only
then it worked
instantly the pgbench backend started using new "stash" immediatley). Question
should or should not it immediatley effective and have you got any idea why
there is such difference in behaviour between pgbench and psql? Should we
investigate it further? I think, we should?

4. The familiy of pg_*stash*() functions could return some ::bool result instead
of void. Like true? (it's usage "feeling" that leaves one wondering if
the command
was effective or not, e.g. to get consistency with let's say pg_reload_conf())

5. QQ: will pg_stash_advice persist the stashes one day?

6. Any idea for better name than 'stash' ? :) It's some new term that is for
sure not wildly recognized. Some other used name across industry: plan
stability,
advice freeze, plan freeze, plan force, baselines, plan management, force plan,
force paths, SQL plan optimization profiles, query store (MSSQL).

7. I saw you have written that in docs to be careful about memory use, but
wouldn't be it safer if that maximum memory for pgca (when collecting in shared
with almost infinite limite) would be still subject to like let's say 5% of s_b
(or any other number here)?

8. I'm wondering if we should apply standard PostgreSQL case insensitivity
rule (e.g. like for relations) for those stashes? On one front we ignore case
sensitivty for objects, one another this is GUC so perhaps it is OK (I
feel it is
ok, but I wanted to ask).

9. If IsQueryIdEnabled() is false (even after trying out to use 'auto')
shouldn't this module raise warning when pg_stash_advice.stash_name != NULL?

> [..altered sequence],
[..]
> First I realized that it might be confusing to have the collector
> interface as part of pg_plan_advice, because for most of what
> pg_plan_advice does, you didn't need the extension, but for that part,
> you did. So, I broke that part out into its own extension, now called
> pg_collect_advice, [..]

10. I'm was here mainly for v18-0007 (pg_stash_advice), but this still looks
like some small bug to me (minmax matching in v18-0003):

create table t1  as select * from generate_series(1, 100000) as id;
create unique index t1_pk on t1 (id);
analyze t1;

-- OK "matched"
postgres=# set pg_plan_advice.advice to 'INDEX_ONLY_SCAN(t1@minmax_1
public.t1_pk)';
SET
postgres=# explain (plan_advice, costs off) select max(id) from t1;
                        QUERY PLAN
-----------------------------------------------------------
 Result
   Replaces: MinMaxAggregate
   InitPlan minmax_1
     ->  Limit
           ->  Index Only Scan Backward using t1_pk on t1
                 Index Cond: (id IS NOT NULL)
 Supplied Plan Advice:
   INDEX_ONLY_SCAN(t1@minmax_1 public.t1_pk) /* matched */
 Generated Plan Advice:
   INDEX_ONLY_SCAN(t1@minmax_1 public.t1_pk)
   NO_GATHER(t1@minmax_1)
(11 rows)

Manual SET, wont work and it is failed (which is OK)

postgres=# set pg_plan_advice.advice to 'SEQ_SCAN(t1)';
SET
postgres=# explain (plan_advice, costs off) select max(id) from t1;
                        QUERY PLAN
----------------------------------------------------------
 Result
   Replaces: MinMaxAggregate
   InitPlan minmax_1
     ->  Limit
           ->  Index Only Scan Backward using t1_pk on t1
                 Index Cond: (id IS NOT NULL)
 Supplied Plan Advice:
   SEQ_SCAN(t1) /* matched, failed */
 Generated Plan Advice:
   INDEX_ONLY_SCAN(t1@minmax_1 public.t1_pk)
   NO_GATHER(t1@minmax_1)

However with below SEQ_SCAN is applied/matched, but marked as failed
(so bug?):

postgres=# set pg_plan_advice.advice to 'SEQ_SCAN(t1@minmax_1)';
SET
postgres=# explain (plan_advice, costs off) select max(id) from t1;
                  QUERY PLAN
-----------------------------------------------
 Aggregate
   ->  Seq Scan on t1
 Supplied Plan Advice:
   SEQ_SCAN(t1@minmax_1) /* matched, failed */
 Generated Plan Advice:
   SEQ_SCAN(t1)
   NO_GATHER(t1)

-J.

  114. Re: pg_plan_advice (now with transparent SQL plan performance overrides - pg_stash_advice)

    Robert Haas <robertmhaas@gmail.com> — 2026-03-03T18:55:47Z

    On Tue, Mar 3, 2026 at 6:42 AM Jakub Wartak
<jakub.wartak@enterprisedb.com> wrote:
> 1. First thought is that I found it quite surprising, we now have 3 modules
> and it's might cause confusion and lack of consistency:
> - 3 can to be in shared_preload_libraries (pg_stash_advice,
> pg_plan_advice, pg_collecti_advice)
> - 2 others can use CREATE EXTENSION (pg_stash_advice, pg_collect_advice), but
>   "create extension pg_plan_advice;" fails
> so maybe they all should behave the same as people (including me) won't read
> the docs and just blindly add it here and there and issue CREATE EXTENSION,
> but it's going to be hard to remember for which ones (?) So we need more
> consistency?

That's a possible point of view, but the flip side is that then it's
all-or-nothing. I think pg_plan_advice is a toolkit that bridge as a
rather large gulf between the theoretical power to walk plan trees and
use pgs_mask to modify behavior and the practicalities of actually
doing so. I think we would be well-advised to think of pg_plan_advice
as a core of functionality that, some day, we might even think of
moving into core, and pg_collect_advice and pg_stash_advice as things
that can be built around that core. I think it's important to have
demonstrations available that show that pg_plan_advice is not a
"sealed hood" that you must accept exactly as it is, but rather a
toolkit that you can do a variety of things with. pg_collect_advice
and pg_stash_advice are examples of what you can do, but many
variations on those same themes are possible.

> 2. Should pgca always duplicate entries like that? (each call gets new entry?)
> Shouldn't it just update collection_time for identical calls of
> userid/dbid/queryid?

To make it do that, we would need a completely different way of
storing entries, in order to avoid a linear scan of all collected data
for each new query. It would be a completely different module. I think
we probably want to have something like that, but it isn't this. See
previous point.

> It floods like that for everything, most visible with couple of independent
> starts of pgbench -M prepared. Maybe I'm wrong , but I don't think it worked
> like that earlier before refactor?

The functionality has not changed since I first posted this. It's only
had bug fixes and now some refactoring.

> 3a. Because query_id each time will be different for every query even
> in standard
> pgbench mode, I've managed to achieve it only using prepared statements.
> Realistically we'll need some way of modular matching not just on query_id OR
> query_id should be changed how it being calculcated or maybe by some other means
> (argument is: not all users/apps use prepared statements):

Sure. See once again the first point in this email, about how this is
a toolkit. You can write your own module and use
pg_plan_advice_add_advisor to add your own hook that does the matching
any way it likes and returns any string it wants. Maybe someone will
add that functionality to pg_stash_advice at some point in the future,
or maybe somebody will add a separate module that does it, either in
contrib or on PGXN or wherever they want to publish it. There are lots
of nice things here that people could want to have, but none of that
is going to happen in time for PostgreSQL 19. The very most we're
going to get is what I've already posted.

It's way too late to think about reinventing query jumbling or even
designing a new way to do query matching. Feature freeze is in just
over a month. The question is whether it's reasonable to commit what
I've already got, not whether it's reasonable to start a whole new
subproject.

> 3c. However for pgbench -M prepared, such online plan alterations are
> strangley not effective.
[....]
> 3d. ... however restarting restarting pgbench helps and the session
> changes plan (and thus
> performance characteristics).
> 3e. As this was pretty concerning, I've repeated the pgbench -M prepared
> excercise and I've figured out that I could achieve effect that I wanted (
> boucing between fast <-> slow immediatley) by injecting call via gdb on that
> backend to InvalidateSystemCaches(). Kind of brute-force, but only
> then it worked

One thing to keep in mind is that the whole point of -M prepared is to
avoid replanning. My guess is that what you're seeing here is that
only replan when something invalidates the plan, which seems more like
a feature than a bug, although possibly somebody is going to want a
way to force an invalidation when the stashed advice changes. That's a
pretty big hammer, though. Does this behavior go away if you use -M
extended?

> 4. The familiy of pg_*stash*() functions could return some ::bool result instead
> of void. Like true? (it's usage "feeling" that leaves one wondering if
> the command
> was effective or not, e.g. to get consistency with let's say pg_reload_conf())

I'm up for changing the return value to something more informative if
there's an informative thing to return, but if there is no information
being returned, I stand by the choice of void as most appropriate.

> 5. QQ: will pg_stash_advice persist the stashes one day?

I have no current plan to implement that, but who knows?

> 6. Any idea for better name than 'stash' ? :) It's some new term that is for
> sure not wildly recognized. Some other used name across industry: plan
> stability,
> advice freeze, plan freeze, plan force, baselines, plan management, force plan,
> force paths, SQL plan optimization profiles, query store (MSSQL).

I'm up to change this if there's a consensus on what it should be
called, but that will require more than 1 vote. I picked stash because
I wanted to capture the idea of sticking something in a place where
you could grab it easily. Personally, my main critique of that name is
that this particular module matches by query ID and you could instead
match by $WHATEVER, so what would you call the next module that also
stashes advice strings but keyed in some slightly different way?
(Likewise, what do you call the next alternative to pg_collect_advice
that uses different collection criteria or storage?)

Honestly, I was hoping and sort of expecting to get a lot more naming
feedback back when I first posted this. Don't call it advice, call it
hints or guidance or tweaks! Don't write HASH_JOIN(foo), write
JOIN_METHOD(foo, HASH_JOIN)! And so forth. On the one hand, the fact
that I didn't get that back then has made this less work, and I'm
still pretty happy with my choices. Furthermore, it's still not too
late to do some renaming if we agree on what that should look like. At
the same time, I don't particularly want to get into a fun and
exciting game of design-by-committee-under-time-pressure. As a
Demotivators poster said many years ago, "none of us is as dumb as all
of us." I freely admit that some of my naming and some of my design
decisions are almost certainly suboptimal, but at the same time, it
would be easy to underestimate the amount of thought that I've put
into some of the naming, so I'd only like to change it if we have a
pretty clear consensus that any given counter-proposal is better than
what I did. I especially do not want to go change it and then have to
change it all again because somebody else shows up with a new opinion.

> 7. I saw you have written that in docs to be careful about memory use, but
> wouldn't be it safer if that maximum memory for pgca (when collecting in shared
> with almost infinite limite) would be still subject to like let's say 5% of s_b
> (or any other number here)?

It depends on what you mean by safer. One thing that is unsafe is
running the computer out of memory. However, another thing that is
unsafe is getting the wrong answer. I felt that limiting the number of
queries was a good compromise. If you limit it to a certain amount of
memory usage, then (1) it's much harder to actually figure out whether
we're over budget and how much we need to free to be under budget and
(2) I suspect it's also harder to be sure whether you've set the limit
high enough to not lose any of the data you intended to retain. I
think that a fixed limit like 5% of shared_buffers would be, bluntly,
completely nuts. There is no reason to suppose that the amount of
memory someone is willing to use to store advice has anything to do
with the size of shared_buffers. If you have 100 sessions running
pgbench and you enabled the local collector with that limit in all of
them, you would be using 500% of shared_buffers in advice-storage
memory and that would probably take down the server. On the other
hand, if you're using the shared collector with shared_buffers=2GB,
that is only 10MB, which might easily be too small to store all the
data you care about even on a test system.

The reason I set the limits in terms of number of queries was that (1)
it's easy for the code to figure out whether it's over the limit and
easy for it to reduce utilization to the limit and (2) I thought that
people using this were likely to have a better idea how many queries
they wanted to collect than how many MB/GB they wanted to collect. Of
course, (1) could be solved with enough effort and (2) could be
wrong-headed on my part, but if somebody wanted this changed, it would
have been nice to know that a few months ago when I first posted this
rather than now.

> 8. I'm wondering if we should apply standard PostgreSQL case insensitivity
> rule (e.g. like for relations) for those stashes? On one front we ignore case
> sensitivty for objects, one another this is GUC so perhaps it is OK (I
> feel it is
> ok, but I wanted to ask).

Yeah, I'm not sure. I wondered about this, too.

> 9. If IsQueryIdEnabled() is false (even after trying out to use 'auto')
> shouldn't this module raise warning when pg_stash_advice.stash_name != NULL?

I think the bar for printing a warning on every single query is
extremely high. Doing that just because of a questionable
configuration setting doesn't seem like a good idea.

> 10. I'm was here mainly for v18-0007 (pg_stash_advice), but this still looks
> like some small bug to me (minmax matching in v18-0003):
>
> create table t1  as select * from generate_series(1, 100000) as id;
> create unique index t1_pk on t1 (id);
> analyze t1;
>
> -- OK "matched"
> postgres=# set pg_plan_advice.advice to 'INDEX_ONLY_SCAN(t1@minmax_1
> public.t1_pk)';
> SET
> postgres=# explain (plan_advice, costs off) select max(id) from t1;
>                         QUERY PLAN
> -----------------------------------------------------------
>  Result
>    Replaces: MinMaxAggregate
>    InitPlan minmax_1
>      ->  Limit
>            ->  Index Only Scan Backward using t1_pk on t1
>                  Index Cond: (id IS NOT NULL)
>  Supplied Plan Advice:
>    INDEX_ONLY_SCAN(t1@minmax_1 public.t1_pk) /* matched */
>  Generated Plan Advice:
>    INDEX_ONLY_SCAN(t1@minmax_1 public.t1_pk)
>    NO_GATHER(t1@minmax_1)
> (11 rows)
>
> Manual SET, wont work and it is failed (which is OK)
>
> postgres=# set pg_plan_advice.advice to 'SEQ_SCAN(t1)';
> SET
> postgres=# explain (plan_advice, costs off) select max(id) from t1;
>                         QUERY PLAN
> ----------------------------------------------------------
>  Result
>    Replaces: MinMaxAggregate
>    InitPlan minmax_1
>      ->  Limit
>            ->  Index Only Scan Backward using t1_pk on t1
>                  Index Cond: (id IS NOT NULL)
>  Supplied Plan Advice:
>    SEQ_SCAN(t1) /* matched, failed */
>  Generated Plan Advice:
>    INDEX_ONLY_SCAN(t1@minmax_1 public.t1_pk)
>    NO_GATHER(t1@minmax_1)
>
> However with below SEQ_SCAN is applied/matched, but marked as failed
> (so bug?):
>
> postgres=# set pg_plan_advice.advice to 'SEQ_SCAN(t1@minmax_1)';
> SET
> postgres=# explain (plan_advice, costs off) select max(id) from t1;
>                   QUERY PLAN
> -----------------------------------------------
>  Aggregate
>    ->  Seq Scan on t1
>  Supplied Plan Advice:
>    SEQ_SCAN(t1@minmax_1) /* matched, failed */
>  Generated Plan Advice:
>    SEQ_SCAN(t1)
>    NO_GATHER(t1)

I think this is just out of scope for now. It's documented that we
don't have a way of controlling aggregation behavior at present. Here
again, we can do more things in future releases, but it's too late to
add more to the scope for this release. There's still plenty of time
to fix bugs, but this isn't a bug. We'd need a whole lot of new
machinery to prevent this sort of thing, including new core hooks and
new syntax. Here again, we have the freedom to decide that I chose the
scope wrongly and that this is therefore not shippable as is, but I do
not think there is time to significantly expand the scope at this
point.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  115. Re: pg_plan_advice (now with transparent SQL plan performance overrides - pg_stash_advice)

    David G. Johnston <david.g.johnston@gmail.com> — 2026-03-03T19:28:16Z

    On Tuesday, March 3, 2026, Robert Haas <robertmhaas@gmail.com> wrote:

> >
> > However with below SEQ_SCAN is applied/matched, but marked as failed
> > (so bug?):
> >
> > postgres=# set pg_plan_advice.advice to 'SEQ_SCAN(t1@minmax_1)';
> > SET
> > postgres=# explain (plan_advice, costs off) select max(id) from t1;
> >                   QUERY PLAN
> > -----------------------------------------------
> >  Aggregate
> >    ->  Seq Scan on t1
> >  Supplied Plan Advice:
> >    SEQ_SCAN(t1@minmax_1) /* matched, failed */
> >  Generated Plan Advice:
> >    SEQ_SCAN(t1)
> >    NO_GATHER(t1)
>
> I think this is just out of scope for now. It's documented that we
> don't have a way of controlling aggregation behavior at present. Here
> again, we can do more things in future releases, but it's too late to
> add more to the scope for this release. There's still plenty of time
> to fix bugs, but this isn't a bug. We'd need a whole lot of new
> machinery to prevent this sort of thing, including new core hooks and
> new syntax. Here again, we have the freedom to decide that I chose the
> scope wrongly and that this is therefore not shippable as is, but I do
> not think there is time to significantly expand the scope at this
> point.
>
>
I mostly get why specifying an index that doesn't exist as part of advice,
alongside a target relation, produces "matched" along with "inapplicable"
and "failed".

INDEX_SCAN(f no_such_index) /* matched, inapplicable, failed */

But less understandable is why a failure to match a subplan-qualified
target produces "matched" when the subplan doesn't appear.

SEQ_SCAN(t1@minmax_1) /* matched, failed */

Maybe we need to do something like:

relname - matches anywhere in the plan tree
relname@somewhere - only looks at "somewhere" for matches; absence of
somewhere results in "not matched" (the expected feedback for the
advice/query combination above).

This would necessitate needing some way to specify "top-level" after the
"@" symbol - leaving it blank would suffice.  I haven't worked through
"directly at the named subplan" versus "the named subplan and any
descendants"...

If keeping the status quo the existing behavior should be documented. The
existing wording for not matched; "or it may occur if the relevant portion
of the query was not planned," seems to be the one that covers this case.

David J.

  116. Re: pg_plan_advice (now with transparent SQL plan performance overrides - pg_stash_advice)

    Robert Haas <robertmhaas@gmail.com> — 2026-03-04T03:36:13Z

    On Tue, Mar 3, 2026 at 2:28 PM David G. Johnston
<david.g.johnston@gmail.com> wrote:
> I mostly get why specifying an index that doesn't exist as part of advice, alongside a target relation, produces "matched" along with "inapplicable" and "failed".
>
> INDEX_SCAN(f no_such_index) /* matched, inapplicable, failed */
>
> But less understandable is why a failure to match a subplan-qualified target produces "matched" when the subplan doesn't appear.
>
> SEQ_SCAN(t1@minmax_1) /* matched, failed */

Because there's not a way to control aggregation behavior at present,
you can't directly conrol whether t1 or t1@minmax_1 appears in the
winning plan. This case is "matched" because we saw t1@minmax_1 get
planned, but it's "failed" because the non-minmax strategy then won on
cost.

Aside: This is actually an instance of a very common problem into
which I have already invested an enormous amount of time and energy.
Suppose that for some operation X (which might be a scan or a join or
any other kind of thing) you tell the planner that you want it to use
strategy S. If strategy S is not what was going to be chosen anyway,
you've just increased the cost of X. Unsurprisingly, the planner's
response to that is very often to choose not to perform X at all.
Otherwise, it ends up looking like the planner has just made an
end-run around the advice. For example, HASH_JOIN(X) says "put X on
the inner side of a hash join". Obviously, this means that when we see
a join with X as the inner rel, we should disallow all join strategies
other than hash join. Less obviously, it also means that when we see a
join with X as the outer rel, we shouldn't allow *any* join
strategies. If you don't do that, you get a problem very similar to
what you're complaining about here. The difference is that join
control is supported by the patch set, and so I have put in the work
to avoid instances of this problem that occur in that case, and
aggregate control is not, so I haven't.

> Maybe we need to do something like:
>
> relname - matches anywhere in the plan tree
> relname@somewhere - only looks at "somewhere" for matches; absence of somewhere results in "not matched" (the expected feedback for the advice/query combination above).

This would not work in general; "anywhere in the plan tree" is too
broad a scope, and it would be easy to construct an example where it
falsely matches an unrelated part of the query. I think that the
solution here probably looks more like letting the user write
AGGREGATE_PLAIN(something) or AGGREGATE_MINMAX(something), but to make
it really work, we would need to figure out what the "something"
should be, and also remember to account for partitionwise aggregation,
eager aggregation, partial aggregation, hashed aggregation, sorted
aggregation, plain aggregation, and mixed aggregation, some of which
can be combined with some of the others but not all of the others, and
then we would need to add design and implement hooks in core to allow
that advice to be enforced, and then we would need to get
pg_plan_advice to properly accept the syntax and enforce it and also
be able to generate the syntax from a finished plan tree, and then
document all that and add tests. I think that work is worth doing,
but, again, I think it would be a much better idea to spend the next
few weeks trying to figure out whether it makes sense to commit what
I've got, and perform any necessarily stabilization of that
functionality, than to keep saying "hey, maybe you should radically
expand the scope." It's probably a six-month project to get all that
working, and we don't have that time for v19.

> If keeping the status quo the existing behavior should be documented. The existing wording for not matched; "or it may occur if the relevant portion of the query was not planned," seems to be the one that covers this case.

That's definitely relevant here, although these specific examples seem
to have more to do with "failed" than "not matched".

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  117. Re: pg_plan_advice (now with transparent SQL plan performance overrides - pg_stash_advice)

    David G. Johnston <david.g.johnston@gmail.com> — 2026-03-04T04:11:38Z

    On Tuesday, March 3, 2026, Robert Haas <robertmhaas@gmail.com> wrote:

>
> >
> > SEQ_SCAN(t1@minmax_1) /* matched, failed */
>
> Because there's not a way to control aggregation behavior at present,
> you can't directly conrol whether t1 or t1@minmax_1 appears in the
> winning plan.


Ok, that’s what I was missing here, it saw the subplan in its options but
the winning plan didn’t include it.  So “matched/failed” may produce a plan
where the target having been matched isn’t actually visible to the user.

Maybe add a note like this to pg_plan_advice:

Generated advice is produced to a high level of specificity without knowing
what limitations the advice interpreter has in applying that advice.
Therefore, generated advice targets may later fail for no other reason than
cost-based decisions resulted in the originally chosen plan to no longer be
chosen.  The planner will still likely see the original target on a now
losing plan and thus the advice feedback will report matched even when the
winning plan does not include the specific target.

David J.

  118. Re: pg_plan_advice (now with transparent SQL plan performance overrides - pg_stash_advice)

    Jakub Wartak <jakub.wartak@enterprisedb.com> — 2026-03-04T09:50:40Z

    On Tue, Mar 3, 2026 at 7:56 PM Robert Haas <robertmhaas@gmail.com> wrote:

Hi Robert,

> On Tue, Mar 3, 2026 at 6:42 AM Jakub Wartak
> <jakub.wartak@enterprisedb.com> wrote:
> > 1. First thought is that I found it quite surprising, we now have 3 modules
> > and it's might cause confusion and lack of consistency:
> > - 3 can to be in shared_preload_libraries (pg_stash_advice,
> > pg_plan_advice, pg_collecti_advice)
> > - 2 others can use CREATE EXTENSION (pg_stash_advice, pg_collect_advice), but
> >   "create extension pg_plan_advice;" fails
> > so maybe they all should behave the same as people (including me) won't read
> > the docs and just blindly add it here and there and issue CREATE EXTENSION,
> > but it's going to be hard to remember for which ones (?) So we need more
> > consistency?
>
> That's a possible point of view, but the flip side is that then it's
> all-or-nothing. I think pg_plan_advice is a toolkit that bridge as a
> rather large gulf between the theoretical power to walk plan trees and
> use pgs_mask to modify behavior and the practicalities of actually
> doing so. I think we would be well-advised to think of pg_plan_advice
> as a core of functionality that, some day, we might even think of
> moving into core, and pg_collect_advice and pg_stash_advice as things
> that can be built around that core. I think it's important to have
> demonstrations available that show that pg_plan_advice is not a
> "sealed hood" that you must accept exactly as it is, but rather a
> toolkit that you can do a variety of things with. pg_collect_advice
> and pg_stash_advice are examples of what you can do, but many
> variations on those same themes are possible.

This is micro-thing, feel free to ignore, but well I was after something much
more easy: `CREATE EXTENSION pg_plan_advice`to be no-op without any failure
even if it doesnt provide any views or fucntions right now (so empty
share/extension/pg_plan_advice.control and -1.0.sql) or at least some dummy
function, just so that CREATE EXTENSION pg_plan_advice wouldn't fail.
This is nothing technical, it's just sharp rough edge for user (but
technically sound), that 2 are deployed with CREATE EXTENSION but 3rd one is
not. I just think that if we put 3 into shared_preload_libraries then I won't
have to think for which ones to exec CREATE EXTENSION (I would just blindly do
it for all and the error wouldn't make somebody unhappy that something is
potentially not working because CREATE EXTENSION is not for it) - it's pure
user-focused usability feedback.

> > 2. Should pgca always duplicate entries like that? (each call gets new entry?)
> > Shouldn't it just update collection_time for identical calls of
> > userid/dbid/queryid?
>
> To make it do that, we would need a completely different way of
> storing entries, in order to avoid a linear scan of all collected data
> for each new query. It would be a completely different module. I think
> we probably want to have something like that, but it isn't this. See
> previous point.
>
[..]
> The functionality has not changed since I first posted this. It's only
> had bug fixes and now some refactoring.

OK, I misremembered it then.

> > 3c. However for pgbench -M prepared, such online plan alterations are
> > strangley not effective.
> [....]
> > 3d. ... however restarting restarting pgbench helps and the session
> > changes plan (and thus
> > performance characteristics).
> > 3e. As this was pretty concerning, I've repeated the pgbench -M prepared
> > excercise and I've figured out that I could achieve effect that I wanted (
> > boucing between fast <-> slow immediatley) by injecting call via gdb on that
> > backend to InvalidateSystemCaches(). Kind of brute-force, but only
> > then it worked
>
> One thing to keep in mind is that the whole point of -M prepared is to
> avoid replanning. My guess is that what you're seeing here is that
> only replan when something invalidates the plan, which seems more like
> a feature than a bug, although possibly somebody is going to want a
> way to force an invalidation when the stashed advice changes. That's a
> pretty big hammer, though. Does this behavior go away if you use -M
> extended?

Yes with -M extended it is instant. I have found also that with
-M prepared I can do simple one-time `analyze pgbench_accounts` (when changing
SEQ_SCAN <-> INDEX_SCAN for that table) and that is also enough for
the backend to immediatley see (and react to) to what's in the active
configured stash even for future changes without further ANALYZEs.
Not sure if pg_stash_advice needs a function to flush-force all backends,
so the plans are 100% effecitve, as apparently we seem to have ANALYZE
already, but it is not that obvious that one might want to use it.

If there would be such function to gurantee, we probably wouldn't see
complaints like 'I have done this and session still is using old plan'.

>
> > 5. QQ: will pg_stash_advice persist the stashes one day?
>
> I have no current plan to implement that, but who knows?

OK, so perhaps docs for pg_create_advice_stash() and pg_set_stashed_advice()
should mention those 'stashes' are not persistent across restarts. Without
this I can already hear scream of some users from the future that they
applied advice, it fixed problem and after some time it disappeared (In
other RDBMS it is persistent, so users coming from there might have such
expectations).

Also related, we could also mention that this information is not replicated to
standbys as this is in-memory only.

> > 6. Any idea for better name than 'stash' ? :) It's some new term that is for
> > sure not wildly recognized. Some other used name across industry: plan
> > stability,
> > advice freeze, plan freeze, plan force, baselines, plan management, force plan,
> > force paths, SQL plan optimization profiles, query store (MSSQL).
>
> I'm up to change this if there's a consensus on what it should be
> called, but that will require more than 1 vote. I picked stash because
> I wanted to capture the idea of sticking something in a place where
> you could grab it easily. Personally, my main critique of that name is
> that this particular module matches by query ID and you could instead
> match by $WHATEVER, so what would you call the next module that also
> stashes advice strings but keyed in some slightly different way?
> (Likewise, what do you call the next alternative to pg_collect_advice
> that uses different collection criteria or storage?)
>
> Honestly, I was hoping and sort of expecting to get a lot more naming
> feedback back when I first posted this. Don't call it advice, call it
> hints or guidance or tweaks! Don't write HASH_JOIN(foo), write
> JOIN_METHOD(foo, HASH_JOIN)! And so forth. On the one hand, the fact
> that I didn't get that back then has made this less work, and I'm
> still pretty happy with my choices. Furthermore, it's still not too
> late to do some renaming if we agree on what that should look like. At
> the same time, I don't particularly want to get into a fun and
> exciting game of design-by-committee-under-time-pressure. As a
> Demotivators poster said many years ago, "none of us is as dumb as all
> of us." I freely admit that some of my naming and some of my design
> decisions are almost certainly suboptimal, but at the same time, it
> would be easy to underestimate the amount of thought that I've put
> into some of the naming, so I'd only like to change it if we have a
> pretty clear consensus that any given counter-proposal is better than
> what I did. I especially do not want to go change it and then have to
> change it all again because somebody else shows up with a new opinion.

Well IMHO all the rest naming in pretty great shape and I think that
pg_[collect|plan]_advice are great names too. It's just that `stash`
keyword doesn't ring a bell to me at all that `pg_stash_advice` is
related in any way to online/transparent/runtime plan modification and
can be used to alter plans for other backends. Something like
`pg_deploy_advice` or `pg_apply_advice` would be more in line with the
other two, but perhaps it's just me..

> > 7. I saw you have written that in docs to be careful about memory use, but
> > wouldn't be it safer if that maximum memory for pgca (when collecting in shared
> > with almost infinite limite) would be still subject to like let's say 5% of s_b
> > (or any other number here)?
>
> It depends on what you mean by safer. One thing that is unsafe is
> running the computer out of memory. However, another thing that is
> unsafe is getting the wrong answer. I felt that limiting the number of
> queries was a good compromise. If you limit it to a certain amount of
> memory usage, then (1) it's much harder to actually figure out whether
> we're over budget and how much we need to free to be under budget and
> (2) I suspect it's also harder to be sure whether you've set the limit
> high enough to not lose any of the data you intended to retain. I
> think that a fixed limit like 5% of shared_buffers would be, bluntly,
> completely nuts. There is no reason to suppose that the amount of
> memory someone is willing to use to store advice has anything to do
> with the size of shared_buffers. If you have 100 sessions running
> pgbench and you enabled the local collector with that limit in all of
> them, you would be using 500% of shared_buffers in advice-storage
> memory and that would probably take down the server. On the other
> hand, if you're using the shared collector with shared_buffers=2GB,
> that is only 10MB, which might easily be too small to store all the
> data you care about even on a test system.
>
> The reason I set the limits in terms of number of queries was that (1)
> it's easy for the code to figure out whether it's over the limit and
> easy for it to reduce utilization to the limit and (2) I thought that
> people using this were likely to have a better idea how many queries
> they wanted to collect than how many MB/GB they wanted to collect. Of
> course, (1) could be solved with enough effort and (2) could be
> wrong-headed on my part, but if somebody wanted this changed, it would
> have been nice to know that a few months ago when I first posted this
> rather than now.

I havent reported it earlier, well because there was little sense playing with
collection implementation much earlier if we didn't have pinning of the plans
itself (and the whole debate query_id also seemed pointless back then). Well
anyway, it is much like work_mem danger, but work_mem is about well specific
amount of memory (* N plan nodes) so it's just easier to put safer value there.

How about if we would just measure (estimate) with some small table number of
entries vs memory used and put that into the docs?, so users are wary that
they shouldn't just blidnly set it to high value? E.g. with
1000 local limit I get ~280kB for pg_collect_advice context and with 1000000
local limit I've got it to ~50MB and stopped looking further (it was still
growing). Itself that's not terrible but higher values with lots of backends
might cause huge memory pressure (OOMs).

Or maybe other idea: is there is possibility of making GUCs like
local_collection_limits/local_collector settable only using
SET/SET LOCAL, but not global? I mean what's the point of having being
able to collect locally system-wide when realistically I cannot
pull it back from backend-local memory? (and this removes the danger
of multple backends goind wild with memory together).

Below maybe useful for others - if they trying this, if testing performance
please ensure you do not have Casserts as with Casserts enabling higher
local_collection_limits are barley usable:

progress: 137.0 s, 1516.9 tps, lat 0.659 ms stddev 0.174, 0 failed
progress: 138.0 s, 1469.1 tps, lat 0.680 ms stddev 0.174, 0 failed
progress: 139.0 s, 1050.0 tps, lat 0.952 ms stddev 0.206, 0 failed
progress: 140.0 s, 922.0 tps, lat 1.085 ms stddev 0.194, 0 failed
[..]
progress: 143.0 s, 664.0 tps, lat 1.504 ms stddev 0.377, 0 failed
progress: 144.0 s, 595.0 tps, lat 1.681 ms stddev 0.386, 0 failed
progress: 145.0 s, 531.0 tps, lat 1.883 ms stddev 0.402, 0 failed
[..]
progress: 159.0 s, 260.0 tps, lat 3.840 ms stddev 0.607, 0 failed
progress: 160.0 s, 271.0 tps, lat 3.702 ms stddev 0.447, 0 failed

resetting it back to 0 or disabling local collector and reloading won't
fix it, backend needs to re-establish connection. Even with just 10000
local collection limit it just gets down from ~1500 tps to 900 tps.
It's seems the impact on CPU coming to be from exec_simple_query()
-> finish_xact_command() -> MemoryContextCheck() -> AllocSetCheck(),
so memory context validation that have literally nothing to do with
with this patch (other than it using a lot of memory in those scenarios)

> > 9. If IsQueryIdEnabled() is false (even after trying out to use 'auto')
> > shouldn't this module raise warning when pg_stash_advice.stash_name != NULL?
>
> I think the bar for printing a warning on every single query is
> extremely high. Doing that just because of a questionable
> configuration setting doesn't seem like a good idea.

Why on every single query? I'm thinking that this should bail out in pgca&pgsa
during initialization of those modules. What's the point of those modules
if queryid is always 0? (I'm assuming somebody has compue_query_id=off and
still loaded those modules).

> > 10. I'm was here mainly for v18-0007 (pg_stash_advice), but this still looks
> > like some small bug to me (minmax matching in v18-0003):
> >
[..]
>
> I think this is just out of scope for now. It's documented that we
> don't have a way of controlling aggregation behavior at present. [..]

Oh ok, I've completley missed "pgplanadvice-limitations" SGML section about
aggregates, so sure - fair limitation.

BTW: the good news is that I haven't seen a single crash when throwing
wild stuff on it or having strange ideas at pg_stash_advice usage.

-J.

  119. Re: pg_plan_advice (now with transparent SQL plan performance overrides - pg_stash_advice)

    Robert Haas <robertmhaas@gmail.com> — 2026-03-04T13:37:33Z

    On Tue, Mar 3, 2026 at 11:11 PM David G. Johnston
<david.g.johnston@gmail.com> wrote:
> Ok, that’s what I was missing here, it saw the subplan in its options but the winning plan didn’t include it.  So “matched/failed” may produce a plan where the target having been matched isn’t actually visible to the user.
>
> Maybe add a note like this to pg_plan_advice:
>
> Generated advice is produced to a high level of specificity without knowing what limitations the advice interpreter has in applying that advice.  Therefore, generated advice targets may later fail for no other reason than cost-based decisions resulted in the originally chosen plan to no longer be chosen.  The planner will still likely see the original target on a now losing plan and thus the advice feedback will report matched even when the winning plan does not include the specific target.

There is doubtless lots of room for improvement in the documentation,
but this specific text doesn't seem like a good idea, because it's not
true in general. The whole point of test_plan_advice is to ensure that
generated advice targets *don't* fail, and with all the patches
applied, I get a clean run where every single advice target generated
by the main regression test suite can be successfully applied back to
the plan that generated it. I think the real issue here is that Jakub
has found a case where fiddling with the scan method causes the
optimal aggregation method to change, and the resulting weird behavior
stems from the fact that aggregation control is not supported. What
you want to be able to do is nail down the aggregation behavior first,
and then it would be clear whether to provide scan advice for t1 or
for t1@minmax_1.

But I just don't see what the big deal is here. One thing you can do
is provide advice for both t1 and t1@minmax_1 and then the advice will
be followed for whichever one appears in the final plan. The other
will be marked as failed, but you can decide to just ignore that.
Possibly you could even indirectly control whether the minmax path is
selected by advising an inefficient strategy for the one you don't
want to be be picked and a great strategy for the one you do want to
be picked. Also, I've personally never run into a real-world case
where the planner made a bad decision about whether to use a minmaxagg
or a regular agg, which is why all of my development time and effort
went into scans and joins and related topics where I do regularly see
things go wrong.

So at the risk of repeating myself: It is not in general true that you
should expect your advice to randomly not work, but that happens in
this case because aggregation control is not supported, which is a
documented limitation.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  120. Re: pg_plan_advice (now with transparent SQL plan performance overrides - pg_stash_advice)

    Robert Haas <robertmhaas@gmail.com> — 2026-03-04T14:17:25Z

    On Wed, Mar 4, 2026 at 4:50 AM Jakub Wartak
<jakub.wartak@enterprisedb.com> wrote:
> This is micro-thing, feel free to ignore, but well I was after something much
> more easy: `CREATE EXTENSION pg_plan_advice`to be no-op without any failure
> even if it doesnt provide any views or fucntions right now (so empty
> share/extension/pg_plan_advice.control and -1.0.sql) or at least some dummy
> function, just so that CREATE EXTENSION pg_plan_advice wouldn't fail.
> This is nothing technical, it's just sharp rough edge for user (but
> technically sound), that 2 are deployed with CREATE EXTENSION but 3rd one is
> not. I just think that if we put 3 into shared_preload_libraries then I won't
> have to think for which ones to exec CREATE EXTENSION (I would just blindly do
> it for all and the error wouldn't make somebody unhappy that something is
> potentially not working because CREATE EXTENSION is not for it) - it's pure
> user-focused usability feedback.

Absolutely not. We don't do that for other contrib modules that don't
require SQL definitions (e.g. auth_delay, auto_explain,
basebackup_to_shell, passwordcheck) and I don't think we should start
now. I agree that it can be confusing for users that there is a
distinction between extensions and loadable modules, but the right
solution to that problem is to educate the users. I think people who
do know how things are supposed to work would find it quite irritating
to be handed an extension that doesn't actually install anything.
Also, I have some hope that at some point in the future, we might
decide to ingest pg_plan_advice into core while keeping the other
things as contrib modules, and if that ever happens, it will be a lot
easier if it's only a loadable module and has no extension associated
with it.

> Yes with -M extended it is instant. I have found also that with
> -M prepared I can do simple one-time `analyze pgbench_accounts` (when changing
> SEQ_SCAN <-> INDEX_SCAN for that table) and that is also enough for
> the backend to immediatley see (and react to) to what's in the active
> configured stash even for future changes without further ANALYZEs.
> Not sure if pg_stash_advice needs a function to flush-force all backends,
> so the plans are 100% effecitve, as apparently we seem to have ANALYZE
> already, but it is not that obvious that one might want to use it.
>
> If there would be such function to gurantee, we probably wouldn't see
> complaints like 'I have done this and session still is using old plan'.

True, but causing a system-wide cache invalidation can also create
quite a performance hit. I'm not quite sure what the best thing to do
here is. I can see an argument that changing the advice for a certain
query ought to invalidated stored plans for that query, but I don't
think our invalidation infrastructure is capable of doing anything
that targeted. Just invalidating everything seems pretty heavy-handed,
but maybe it will turn out to be the right answer. I think we should
wait for more people to play with this before deciding anything.

> > > 5. QQ: will pg_stash_advice persist the stashes one day?
> >
> > I have no current plan to implement that, but who knows?
>
> OK, so perhaps docs for pg_create_advice_stash() and pg_set_stashed_advice()
> should mention those 'stashes' are not persistent across restarts. Without
> this I can already hear scream of some users from the future that they
> applied advice, it fixed problem and after some time it disappeared (In
> other RDBMS it is persistent, so users coming from there might have such
> expectations).

I mean, there is already text about this in the very first paragraph
of the pg_stash_advice documentation. Perhaps you're saying you think
that information needs to be mentioned in multiple places in that
documentation, or in a different place than where it's currently
mentioned, but I'm slightly suspicious that you might not have
actually read what I already wrote.

> Well IMHO all the rest naming in pretty great shape and I think that
> pg_[collect|plan]_advice are great names too. It's just that `stash`
> keyword doesn't ring a bell to me at all that `pg_stash_advice` is
> related in any way to online/transparent/runtime plan modification and
> can be used to alter plans for other backends. Something like
> `pg_deploy_advice` or `pg_apply_advice` would be more in line with the
> other two, but perhaps it's just me..

I don't think it's just you. I originally named this pg_auto_advice.
However, the naming problem that then ran into was: what do you call
the containers that actually store the advice? I ended up calling them
"advice stores". But I didn't like that very much, because now the
name of the module (pg_auto_advice) and the name of the objects that
it creates (advice stores) are not obviously related. Moreover, I had
an unpleasant hunch that people weren't going to like the idea of
using "store" as a noun. So I tried to think of a word that I could
use for both the name of the extension and the name of the objects,
and stash is what I came up with. Your suggestions here are in much
the same vein as my original idea, but I would argue that they also
have the same problem: we're not going to call the named advice
containers "advice deploys" or "advice applies". Now, perhaps there's
an argument that those names don't have to match, but I think it makes
it a lot less confusing that they do.

> How about if we would just measure (estimate) with some small table number of
> entries vs memory used and put that into the docs?, so users are wary that
> they shouldn't just blidnly set it to high value? E.g. with
> 1000 local limit I get ~280kB for pg_collect_advice context and with 1000000
> local limit I've got it to ~50MB and stopped looking further (it was still
> growing). Itself that's not terrible but higher values with lots of backends
> might cause huge memory pressure (OOMs).

It depends a lot on the length of the query strings and the advice
strings. I wondered whether there was some point in having a setting
to collect only the advice string and not the query string, but in the
end I feel like pg_collect_advice is very much 1.0 software. It works,
but it's fairly primitive. It definitely shouldn't be confused with
industrial-strength, battle-grade, Teflon-hardened code. The problem
from my point of view is that not only are we short on time for this
release, but I do not feel like I know what the requirements for
something better really are. I think your suggestions so far --
deduplication, better memory control -- are very reasonable ideas, but
I think it's hard to know what is really important until more people
get their hands on this, try it out, and then (probably) complain
about it. I think between the people on this thread, or even between
you and I, we could easily come up with a list of 30, maybe even 50,
possible improvements to pg_collect_advice, but I am not at all
confident we'd correctly guess which 3-5 of those were going to be
most important to users. I think we just need to wait for more data
before deciding how to evolve this.

> Or maybe other idea: is there is possibility of making GUCs like
> local_collection_limits/local_collector settable only using
> SET/SET LOCAL, but not global? I mean what's the point of having being
> able to collect locally system-wide when realistically I cannot
> pull it back from backend-local memory? (and this removes the danger
> of multple backends goind wild with memory together).

The GUC infrastructure doesn't support this.

> resetting it back to 0 or disabling local collector and reloading won't
> fix it, backend needs to re-establish connection. Even with just 10000
> local collection limit it just gets down from ~1500 tps to 900 tps.
> It's seems the impact on CPU coming to be from exec_simple_query()
> -> finish_xact_command() -> MemoryContextCheck() -> AllocSetCheck(),
> so memory context validation that have literally nothing to do with
> with this patch (other than it using a lot of memory in those scenarios)

Sounds like you are running a debug build with asserts enabled, which
you probably don't want to do if you're trying to benchmark.

> > > 9. If IsQueryIdEnabled() is false (even after trying out to use 'auto')
> > > shouldn't this module raise warning when pg_stash_advice.stash_name != NULL?
> >
> > I think the bar for printing a warning on every single query is
> > extremely high. Doing that just because of a questionable
> > configuration setting doesn't seem like a good idea.
>
> Why on every single query? I'm thinking that this should bail out in pgca&pgsa
> during initialization of those modules. What's the point of those modules
> if queryid is always 0? (I'm assuming somebody has compue_query_id=off and
> still loaded those modules).

That seems like a tremendous overreaction, given that (1) it might
cause the server to fail to start and (2) compute_query_id can be
changed at any time.

> BTW: the good news is that I haven't seen a single crash when throwing
> wild stuff on it or having strange ideas at pg_stash_advice usage.

I don't think there are too many crashes left (famous last words).
Here's a list of things I'm currently most worried about,
approximately in order starting from the most worrying:

* Maybe the advice-generation stuff doesn't correctly analyze all
possible plan trees, esp. cases not covered by our core regression
tests.
* Maybe the stuff that uses DSM isn't careful enough and can therefore
cause server-lifespan memory leaks in some scenario.
* Maybe I haven't got the security model right and some aspect of what
I've done there is CVE-worthy.
* Maybe advice application is broken in some cases in a way that can't
be fixed without additional core changes.

I'd be very grateful for review targeting any of these areas.

Thanks,

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  121. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-03-04T15:17:37Z

    On Fri, Feb 27, 2026 at 8:16 PM David G. Johnston
<david.g.johnston@gmail.com> wrote:
> I have a mind to walk through the readmes and sgmls but its going to be in chunks.  Here's one for the readme for pg_plan_advice with a couple of preliminary sgml changes.

While I'm grateful for the feedback, I feel like you tend to suggest a
lot of edits that seem like they're just substituting your
idiosyncratic preferences for mine e.g. writing "types of scan" vs.
"scan types," or writing "additional, separate join problems" vs.
"independent join problems" or "judiciously" vs. "conservatively". I
don't really consider these to be improvements, nor do I necessarily
think they're worse, but I just don't see the point in litigating this
kind of stuff. If I've written something that is legitimately unclear,
or factually incorrect, or there's a spelling or punctuation mistake,
I'm happy to correct that kind of stuff, but I don't really want to go
through and replace a bunch of words that I liked with a bunch of
synonyms that you picked.

Also, when you just provide a diff like this, it's not that clear to
me why you're suggesting particular changes, which makes it hard to
decide whether I agree with them. And in a lot of cases I don't.
Looking at some particular examples:

+isn't going to work any more. That's expected. It should be resilient to
+changes in the statistics, including any CREATE STATISTICS related changes.

This is broadly true but seems a bit obvious to mention in a README.
If we were going to mention it, I'd think it would go in the
user-facing documentation. But I don't quite see why we should mention
it at all. If plan advice couldn't override changes caused by
statistics, what would even be the point of it? Also, it's not
categorically true in all situations, because as discussed elsewhere,
we have limitations like lack of control over aggregation strategy.

 Tags such as NESTED_LOOP_PLAIN specify the method that should be used to
-perform a certain join. More specifically, NESTED_LOOP_PLAIN(x (y z)) says
+perform a certain join - with the target appearing directly on the inner side
+of the join list first. Thus, NESTED_LOOP_PLAIN(x (y z)) says
 that the plan should put the relation whose identifier is "x" on the inner
 side of a plain nested loop (one without materialization or memoization)
 and that it should also put a join between the relation whose identifier is

This seems like you're adding a second explanation of what the
paragraph already goes onto say, except that the existing explanation
is more precise and detailed.

-useless in practice. It gives the planner too much freedom to do things that
+problematic in practice. It gives the planner too much freedom to do
things that

I mean, I stand by the word I picked. I don't want to weaken it.

-This means that if advice can say that a certain optimization or technique
-should be used, it should also be able to say that the optimization or
-technique should not be used. We should never assume that the absence of an
-instruction to do a certain thing means that it should not be done; all
-instructions must be explicit.
+In other words, advice tags must define whether they encourage or discourage
+certain optimizations or techniques. (NO_GATHER is an example of the latter.
+There is no generic "NOT" syntax, e.g., NOT(HASH_JOIN(dim2 dim4).))

My text explains an important design principle that future hackers
must keep in mind when modifying this system to avoid breaking
everything. Your replacement text just describes how it works today.
Considering that this is a README for hackers, I think that's much
worse.

-  advice" mini-language. It is intended to allow stabilization of plan choices
+  advice" domain specific language (DSL). It is intended to allow
stabilization of plan choices

There's a debate to be had about whether it's better to say
mini-language or domain specific language here, but it's hard for me
to decide which is better if all you provide is a diff replacing A
with B. I definitely think it's worse to write (DSL) here. There is no
point in defining an acronym if we're never going to use it anywhere.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  122. Re: pg_plan_advice

    David G. Johnston <david.g.johnston@gmail.com> — 2026-03-04T15:44:42Z

    On Wed, Mar 4, 2026 at 8:17 AM Robert Haas <robertmhaas@gmail.com> wrote:

> On Fri, Feb 27, 2026 at 8:16 PM David G. Johnston
> <david.g.johnston@gmail.com> wrote:
> > I have a mind to walk through the readmes and sgmls but its going to be
> in chunks.  Here's one for the readme for pg_plan_advice with a couple of
> preliminary sgml changes.
>
> While I'm grateful for the feedback, I feel like you tend to suggest a
> lot of edits that seem like they're just substituting your
> idiosyncratic preferences for mine


Yeah, some of these end up being mostly stylistic.  Though I do try to
limit them to ones where I see inconsistency or the style I'm reading just
doesn't resonate with me.  I usually point out the ones that are IMO
material, versus just something that tripped me up while I was reading, but
failed to do so here.

I do need to work in a way to better annotate/comment on the why of these.
Any suggestions for a better flow or feedback format?  Inline comments
wrapped in sgml comments?  Or just copy the diff into the email body and
inline comment there - leaving the original diff attachment as-is?


> -  advice" mini-language. It is intended to allow stabilization of plan
> choices
> +  advice" domain specific language (DSL). It is intended to allow
> stabilization of plan choices
>
> There's a debate to be had about whether it's better to say
> mini-language or domain specific language here, but it's hard for me
> to decide which is better if all you provide is a diff replacing A
> with B. I definitely think it's worse to write (DSL) here. There is no
> point in defining an acronym if we're never going to use it anywhere.
>
>
This was truly just a "have you considered using this terminology instead"
kind of prompt.  The acronym would have been useful when going an replacing
the other uses of mini-language that I left alone since I hadn't myself
decided which one was better.

I didn't do my usual email recap on this first patch which is my bad.  I
corrected that with the others.

David J.

  123. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-03-04T16:20:45Z

    On Wed, Mar 4, 2026 at 10:45 AM David G. Johnston
<david.g.johnston@gmail.com> wrote:
> I do need to work in a way to better annotate/comment on the why of these.  Any suggestions for a better flow or feedback format?  Inline comments wrapped in sgml comments?  Or just copy the diff into the email body and inline comment there - leaving the original diff attachment as-is?

My suggestion is to break these fixes up into three categories: clear
errors, stylistic suggestions, substantive concerns.

Clear errors can be handled by just sending a patch that fixes all the
clear errors without changing anything else. If I intended to write
"applied" and I actually typed "aplied," it's fine to bundle that with
10 other mistakes and submit them without commentary.

For substantive concerns, I think it's most helpful to just quote the
patch hunk in the body of your email and say what your concern is. If
you have suggested wording feel free to suggest that as well, but I'd
focus more on saying what the problem is rather than jumping to the
solution. At least some of these are cases where what I wrote wasn't
sufficient for you to understand the patch, which a very fair issue to
raise, but if you try to write your own wording, the fact that you
don't understand the patch makes it hard for you to write quality
documentation for it. If you say "I read where you said X and I tried
Y and it seemed like the wrong thing happened, so either the
documentation sucks or the code is buggy," now we're having a
worthwhile conversation. If you just change the documentation based on
your understanding of the results of an undisclosed experiment, I feel
like the chances of the result being an improvement are not great.

Stylistic concerns are the most complicated. If your concern is
something like "this sentence is hard to understand," I'd class that
as a substantive concern and treat it the same way. Beyond that, I'm
not really sure. Honestly, I think we may just have different
stylistic preferences, because my experience so far reading your
proposed documentation patches is that I tend to agree with relatively
little of what you want to do. I believe, though, that other
committers feel differently about it and find your proposed changes
quite helpful. So I'm not sure exactly what to recommend here, but
perhaps take a lighter touch when it's my patch? I'm OK with some
friendly suggestions that I can accept or reject, but going through a
huge list of minor wording tweaks is as much work as going through a
substantial review of the code itself, but for a lot less benefit,
especially if they all look like random changes that I don't
understand why you're proposing.

Hope that makes sense and isn't too harsh.

Thanks,

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  124. Re: pg_plan_advice

    David G. Johnston <david.g.johnston@gmail.com> — 2026-03-06T14:46:24Z

    On Wed, Mar 4, 2026 at 9:20 AM Robert Haas <robertmhaas@gmail.com> wrote:

> On Wed, Mar 4, 2026 at 10:45 AM David G. Johnston
> <david.g.johnston@gmail.com> wrote:
> > I do need to work in a way to better annotate/comment on the why of
> these.  Any suggestions for a better flow or feedback format?  Inline
> comments wrapped in sgml comments?  Or just copy the diff into the email
> body and inline comment there - leaving the original diff attachment as-is?
>
> My suggestion is to break these fixes up into three categories: clear
> errors, stylistic suggestions, substantive concerns.
>
>
Thank you for putting in the time to respond.  That was quite helpful.
I've tweaked my tooling to help me remember to do this going forward.

David J.

  125. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-03-10T14:55:26Z

    On Fri, Mar 6, 2026 at 9:47 AM David G. Johnston
<david.g.johnston@gmail.com> wrote:
> Thank you for putting in the time to respond.  That was quite helpful.  I've tweaked my tooling to help me remember to do this going forward.

Thanks. Here's v19. I've incorporated a bunch of your changes, some
that were fixes for clear errors and a few of the more stylistic cases
where I decided that I agreed with you, and I've also fixed some
similar things upon which you did not remark. Separately, I've also
committed the patches that were previously 0001, 0002, and 0005, so
all the preparatory stuff is done now, and this version just contains
the substantive commits, for which I am still in need of review,
especially for 0004. I have also made a minor code adjustment:
pg_plan_advice.trace_mask now prints the subplan name except for the
toplevel "subplan".

Here are a few comments on some of the changes you made that I did not adopt:

- In the README, "we" are the PostgreSQL developers and "the user" is
the person using the module. In the documentation, "you" is the user.
If there are deviations from this idea, we should fix them, but it
seems OK that the two documents have different rules, inasmuch as they
address different audiences.

- For the same reason, I chose not to copy the note about
enable_partitionwise_join=off into the README, as that is not a core
concept for developers but a likely error for users.  We also don't
really want to start duplicating content too much; arguably, we have
too much of that already.

- I don't really think we need to document the exact rules for
argument to, e.g., the pg_stash_advice functions. I think that makes
the documentation ponderous without adding any real utility. At most,
I would mention in some centralized place what the rules for stash
names are. Separately, there is the question of whether the current
naming rules are the right ones.

- In a lot of places, especially in the README, I just disagreed with
your choice of what to emphasize. For instance, I thought my longer
explanation of how we must not infer guidance from the absence of
advice was better than your shorter one, because it's such a critical
point for future developers touching this code to understand. On the
other hand, saying that advice should be stable in the face of
statistics changes was redundant: if it doesn't even do that much,
then what would even be the point?

- I chose to retain the use of the term "core planner" in the SGML
documentation rather than your suggestion of deleting the word "core".
I do not love the wording I've chosen here, but I don't love your
change, either. It seems to me that there is a risk of people being
confused about the distinction between the planning-related logic in
src/backend/optimizer and the planning-related object in
pg_plan_advice itself. I included the word in core for emphasis.
Whether this is the right idea is debatable, of course.

Thanks,

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  126. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-03-12T17:15:39Z

    On Tue, Mar 10, 2026 at 10:55 AM Robert Haas <robertmhaas@gmail.com> wrote:
> Thanks. Here's v19.

I've now committed the main patch. I think that I'm not likely to get
too much more feedback on that in this release cycle, and I judge that
more people will be sad if it doesn't get shipped this cycle than if
it does. That might turn out to be catastrophically wrong, but if many
problem reports quickly begin to emerge, it can always be reverted.
I'm hopeful my testing has been thorough enough to avoid that, but
there's a big difference between lab-testing and real world usage, so
we'll see.

I'm still hoping to get some more feedback on the remaining patches,
which are much smaller and conceptually simpler. While there is no
time to redesign them at this point in the release cycle, there is
still the opportunity to fix bugs, or decide that they're too
half-baked to ship. So here is v20 with just those patches. Of course,
post-commit review of the main patch is also very welcome.

Thanks,

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  127. Re: pg_plan_advice

    Alexandra Wang <alexandra.wang.oss@gmail.com> — 2026-03-12T23:45:49Z

    Hi Robert,

On Thu, Mar 12, 2026 at 10:16 AM Robert Haas <robertmhaas@gmail.com> wrote:
> I'm still hoping to get some more feedback on the remaining patches,
> which are much smaller and conceptually simpler. While there is no
> time to redesign them at this point in the release cycle, there is
> still the opportunity to fix bugs, or decide that they're too
> half-baked to ship. So here is v20 with just those patches. Of course,
> post-commit review of the main patch is also very welcome.

Thanks for the patches!

I've run the meson tests for all three modules, and they all pass on
my laptop.

For the make tests, I had to add

+EXTRA_INSTALL = contrib/pg_plan_advice

in contrib/pg_collect_advice/Makefile in order for "make check" for
that module to run.

I don't really have a sense of how others feel about including these
modules, so I can't speak to that. Personally, though, I very much
like the test_plan_advice module because it gives me peace of mind,
and I feel it should accompany the already committed pg_plan_advice
module.

I reviewed the code and have a few minor comments:

0001:
The "make check" issue mentioned above.

0002:
Looks good to me.

0003:
There is a typo in the commit message:
"pg_stash.advice_stash" should be "pg_stash_advice.stash_name".

> stash->pgsa_stash_id = pgsa_state->next_stash_id++;

I doubt there will be more than 2 billion stashes in the system, but
if in any case we reach that number, we don't handle int overflow.
Should we set a limit on how many stashes can be stored?

Nit:
find_defelem_by_defname() is defined in all three modules, and also in
pg_plan_advice. Knowing it is very small, would it make sense to
extern the one in pg_plan_advice and reuse it?

Best,
Alex

-- 
Alexandra Wang
EDB: https://www.enterprisedb.com

  128. Re: pg_plan_advice

    Lukas Fittl <lukas@fittl.com> — 2026-03-13T08:38:25Z

    Hi Robert,

On Thu, Mar 12, 2026 at 10:15 AM Robert Haas <robertmhaas@gmail.com> wrote:
> I've now committed the main patch. I think that I'm not likely to get
> too much more feedback on that in this release cycle, and I judge that
> more people will be sad if it doesn't get shipped this cycle than if
> it does. That might turn out to be catastrophically wrong, but if many
> problem reports quickly begin to emerge, it can always be reverted.

I think its good to get this in, and not do it in the last minute, but
just to be clear on my part, I've reviewed earlier versions, but I
have not reviewed the latest code to the extent I would have liked
before it was committed, due to competing priorities on my end. That
said, its a good forcing function, so let me do my part to help shake
out any bugs that remain.

From a code review perspective, I found some minor issues:

- Identifiers that are named like advice types cause an error (so I
can't name my table "hash_join")
- pgpa_destroy_join_unroller does not free the inner_unrollers values
(which I think goes against the intent of cleaning up the allocations
right away)
- pgpa_parser uses pgpa_yyerror, but that function doesn't use
elog(ERROR) like the main parser does - I think it'd be more
consistent to use YYABORT explicitly, e.g. how we do it in cubeparse
- pgpa_scanner accepts integers with underscores, but incorrectly uses
a simple strtoint on them (which would fail), instead of pg_strtoint32
/ pg_strtoint32_safe
- pgpa_walk_recursively uses "0" for the boolean value being passed to
a recursive call in one case (should be "false")

Attached nocfbot-0001 that addresses these.

From a usability perspective, I do wonder about two things when it
comes to users specifying advice directly (which will happen in
practice, e.g. when debugging plan problems):

1) The lack of being able to target scan advice (e.g. SEQ_SCAN) to a
partition parent is frustrating - I believe we discussed partitioning
earlier in the thread in terms of gathering+applying it, but I do
wonder if we shouldn't at least allow users to specify a partitioned
table/partitioned index, instead of only the children. See attached
nocfbot-0002 for an idea what would be enough, I think.

2) I find the join strategy advice hard to use - pg_hint_plan has
hints (e.g. HashJoin) that allow saying "use a hash join when joining
these two rels in any order", vs pg_plan_advice requires setting a
specific outer rel, which only makes sense when you want to fully
specify every aspect of the plan. I suspect users who directly write
advice will struggle with specifying join strategy advice as it is
right now. We could consider having a different syntax for saying "I
want a hash join when these two rels are joined, but I don't care
about the order", e.g. "USE_HASH_JOIN(a b)". If you think that's
worthwhile I'd be happy to take a stab at a patch.

> I'm still hoping to get some more feedback on the remaining patches,
> which are much smaller and conceptually simpler. While there is no
> time to redesign them at this point in the release cycle, there is
> still the opportunity to fix bugs, or decide that they're too
> half-baked to ship. So here is v20 with just those patches. Of course,
> post-commit review of the main patch is also very welcome.

For v20-0001, from a quick conceptual review:

I find the two separate GUC mechanisms for local backend vs shared
memory a bit confusing as a user (which one should I be using?).
Enabling the shared memory mechanism on a system-wide basis seems like
it'd likely have too high overhead anyway for production systems?
(specifically worried about the advice capturing for each plan, though
I haven't benchmarked it)

I wonder if we shouldn't keep this simpler for now, and e.g. only do
the backend local version to start - we could iterate a bit on
system-wide collection out-of-core, e.g. I'm considering teaching
pg_stat_plans to optionally collect plan advice the first time it sees
a plan ID (plan advice is roughly a superset of what we consider a
plan ID there), and then we could come back to this for PG20.

For v20-0002:

I think we need a form of this in tree, because otherwise
pg_plan_advice breaks as planner logic changes. This of course puts a
burden on other hackers making changes, but I think that's what we're
signing up for with having pg_plan_advice in contrib - and maybe we're
overestimating how often that'll actually be a problem.

To help assess impact, I did a quick test run and looked at three
not-yet-committed patches in the commitfest that affect planner logic
([0], [1] and [2]), to see if they'd require pg_plan_advice changes
(master with v20-0002 applied). Maybe I picked the wrong patches, but
at least with those no pg_plan_advice changes were needed with the
test_plan_advice test enabled.

On the code itself:
- Is there a reason you're setting
"pg_plan_advice.always_store_advice_details" to true, instead of using
pg_plan_advice_request_advice_generation?
- I wonder if we could somehow detect advice not matching? Right now
that'd be silently ignored, i.e. you'd only get a test failure when we
generate the wrong advice that causes a plan change in the regression
tests.

For v20-0003, initial thoughts:

I think getting at least a basic version of this in would be good, as
a server-wide way to set advice for queries can help people get out of
a problem when Postgres behaves badly - and we know from pg_hint_plan
(which has a hint table) that this can be useful even without doing
any kind of parameter sniffing/etc to be smart about different
parameters for the same query.

The name "stash" feels a bit confusing as an end-user facing term.
Maybe something like "pg_apply_advice", or "pg_query_advice" would be
better? (though I kind of wish we could tie it more closely to "plan
advice", but e.g. "pg_plan_advice_apply" feels too lengthy)

---

Thanks,
Lukas

[0]: https://commitfest.postgresql.org/patch/5487/ ("Pull-up subquery
if INNER JOIN-ON contains refs to upper-query")
[1]: https://commitfest.postgresql.org/patch/5738/ ("Improve hash
join's handling of tuples with null join keys")
[2]: https://commitfest.postgresql.org/patch/6315/ ("Enable
partitionwise join for partition keys wrapped by RelabelType")


--
Lukas Fittl

  129. Re: pg_plan_advice

    Alexander Lakhin <exclusion@gmail.com> — 2026-03-14T12:00:00Z

    Hello Robert,

12.03.2026 19:15, Robert Haas wrote:
> I've now committed the main patch. I think that I'm not likely to get
> too much more feedback on that in this release cycle, and I judge that
> more people will be sad if it doesn't get shipped this cycle than if
> it does. That might turn out to be catastrophically wrong, but if many
> problem reports quickly begin to emerge, it can always be reverted.
> I'm hopeful my testing has been thorough enough to avoid that, but
> there's a big difference between lab-testing and real world usage, so
> we'll see.

I've found a crash inside pgpa_join_path_setup(), reproduced with:
echo "geqo_threshold = 2" >/tmp/extra.config
TEMP_CONFIG=/tmp/extra.config make -s check -C contrib/pg_plan_advice/

Program terminated with signal SIGSEGV, Segmentation fault.
(gdb) bt
#0  0x000070820b96d215 in pgpa_join_path_setup (root=0x5ef5a84682b8, joinrel=0x5ef5a8497f10, outerrel=0x5ef5a8472b48,
     innerrel=0x5ef5a84baeb8, jointype=JOIN_UNIQUE_INNER, extra=0x7fff08823490) at pgpa_planner.c:602
#1  0x00005ef57df9742b in add_paths_to_joinrel (root=0x5ef5a84682b8, joinrel=0x5ef5a8497f10, outerrel=0x5ef5a8472b48,
     innerrel=0x5ef5a84baeb8, jointype=JOIN_UNIQUE_INNER, sjinfo=0x5ef5a8473c00, restrictlist=0x5ef5a8498450) at 
joinpath.c:178
#2  0x00005ef57df9d178 in populate_joinrel_with_paths (root=0x5ef5a84682b8, rel1=0x5ef5a8472b48, rel2=0x5ef5a84731f0,
     joinrel=0x5ef5a8497f10, sjinfo=0x5ef5a8473c00, restrictlist=0x5ef5a8498450) at joinrels.c:1197
#3  0x00005ef57df9c6ab in make_join_rel (root=0x5ef5a84682b8, rel1=0x5ef5a8472b48, rel2=0x5ef5a84731f0) at joinrels.c:774
#4  0x00005ef57df700be in merge_clump (root=0x5ef5a84682b8, clumps=0x5ef5a8497e60, new_clump=0x5ef5a8497eb0, num_gene=2,
     force=false) at geqo_eval.c:259
#5  0x00005ef57df6ff3e in gimme_tree (root=0x5ef5a84682b8, tour=0x5ef5a84ba688, num_gene=2) at geqo_eval.c:198
#6  0x00005ef57df6fe12 in geqo_eval (root=0x5ef5a84682b8, tour=0x5ef5a84ba688, num_gene=2) at geqo_eval.c:101
#7  0x00005ef57df708fa in random_init_pool (root=0x5ef5a84682b8, pool=0x5ef5a84c3988) at geqo_pool.c:108
#8  0x00005ef57df70439 in geqo (root=0x5ef5a84682b8, number_of_rels=2, initial_rels=0x5ef5a84ba1f8) at geqo_main.c:127
#9  0x00005ef57df77aa3 in make_rel_from_joinlist (root=0x5ef5a84682b8, joinlist=0x5ef5a8473b40) at allpaths.c:3902
#10 0x00005ef57df715e2 in make_one_rel (root=0x5ef5a84682b8, joinlist=0x5ef5a8473b40) at allpaths.c:240
#11 0x00005ef57dfbede4 in query_planner (root=0x5ef5a84682b8, qp_callback=0x5ef57dfc5ae9 <standard_qp_callback>,
     qp_extra=0x7fff08823af0) at planmain.c:297
...

Could please look at this?

Best regards,
Alexander

  130. Re: pg_plan_advice

    Zsolt Parragi <zsolt.parragi@percona.com> — 2026-03-14T15:06:04Z

    Hello

I noticed a difference between installed headers with make and meson,
which is caused by pg_plan_advice.h. It is completely missing from the
make build, and installs to the wrong location with meson.

Please see the attached patch that fixes this.

  131. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-03-16T16:52:55Z

    On Sat, Mar 14, 2026 at 11:06 AM Zsolt Parragi
<zsolt.parragi@percona.com> wrote:
> I noticed a difference between installed headers with make and meson,
> which is caused by pg_plan_advice.h. It is completely missing from the
> make build, and installs to the wrong location with meson.
>
> Please see the attached patch that fixes this.

Thanks. The changes to the Makefile seem to mirror what is done in
contrib/isn/Makefile, but I'm not so sure about the meson.build
changes. sepgsql uses dir_data / 'contrib' rather than
dir_include_server. src/pl/pl{perl,pgsql,python} use
dir_include_server, but they also live in src/pl, not contrib. I don't
think I understand what the underlying principal is supposed to be
here. If you or anyone else knows, please enlighten me.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  132. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-03-16T17:11:16Z

    On Sat, Mar 14, 2026 at 8:00 AM Alexander Lakhin <exclusion@gmail.com> wrote:
> I've found a crash inside pgpa_join_path_setup(), reproduced with:
> echo "geqo_threshold = 2" >/tmp/extra.config
> TEMP_CONFIG=/tmp/extra.config make -s check -C contrib/pg_plan_advice/
> ...
>
> Could please look at this?

Thanks. Proposed fix attached.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  133. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-03-16T17:34:59Z

    On Thu, Mar 12, 2026 at 7:46 PM Alexandra Wang
<alexandra.wang.oss@gmail.com> wrote:
> For the make tests, I had to add
>
> +EXTRA_INSTALL = contrib/pg_plan_advice

Thanks, will fix.

> There is a typo in the commit message:
> "pg_stash.advice_stash" should be "pg_stash_advice.stash_name".

Thanks, will fix.

> I doubt there will be more than 2 billion stashes in the system, but
> if in any case we reach that number, we don't handle int overflow.
> Should we set a limit on how many stashes can be stored?

I think the thing to do here is just change the stash ID to a 64-bit
value. I'll do that in the next version. We assume in numerous places
that a 64-bit integer never overflows (e.g. LSNs) which is pretty fair
considering how long it would take for that to happen.

> Nit:
> find_defelem_by_defname() is defined in all three modules, and also in
> pg_plan_advice. Knowing it is very small, would it make sense to
> extern the one in pg_plan_advice and reuse it?

I think if we want to avoid duplicating this function, we should
actually put it someplace in core, e.g. expose it via defrem.h and put
the code in commands/define.c. Any user of extendplan.h is likely to
need a function like this, but we shouldn't put it there because there
could be other use cases as well.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  134. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-03-16T20:51:09Z

    On Fri, Mar 13, 2026 at 4:39 AM Lukas Fittl <lukas@fittl.com> wrote:
> From a code review perspective, I found some minor issues:
>
> - Identifiers that are named like advice types cause an error (so I
> can't name my table "hash_join")

Thanks, I extracted and committed your fix for that issue.

> - pgpa_destroy_join_unroller does not free the inner_unrollers values
> (which I think goes against the intent of cleaning up the allocations
> right away)

Yeah, that makes sense.

> - pgpa_parser uses pgpa_yyerror, but that function doesn't use
> elog(ERROR) like the main parser does - I think it'd be more
> consistent to use YYABORT explicitly, e.g. how we do it in cubeparse

Perhaps, but I think this needs more thought, because you haven't done
anything about the calls in pgpa_scanner.l. I think right now, the
operating principle is that parsing continues after an error and that
we mustn't crash as a result, though the resulting tree will probably
be invalid for practical use. If we're going to change that to stop on
the spot, I think we should probably do it for both lexing and
parsing, and think about whether that leads to any other changes or
simplificatons.

> - pgpa_scanner accepts integers with underscores, but incorrectly uses
> a simple strtoint on them (which would fail), instead of pg_strtoint32
> / pg_strtoint32_safe

OK.

> - pgpa_walk_recursively uses "0" for the boolean value being passed to
> a recursive call in one case (should be "false")

OK.

> From a usability perspective, I do wonder about two things when it
> comes to users specifying advice directly (which will happen in
> practice, e.g. when debugging plan problems):
>
> 1) The lack of being able to target scan advice (e.g. SEQ_SCAN) to a
> partition parent is frustrating - I believe we discussed partitioning
> earlier in the thread in terms of gathering+applying it, but I do
> wonder if we shouldn't at least allow users to specify a partitioned
> table/partitioned index, instead of only the children. See attached
> nocfbot-0002 for an idea what would be enough, I think.

I'm not on board with this without a lot more study. I've been down
this road before, and it can easily end in tears. Examining the
partition structure on the fly can have a performance cost, and it
might even have security ramifications or at least bugs if there are
concurrent modifications to the partitioning structure happening.
Also, the test_plan_advice framework doesn't do much to tell us
whether this actually works. Also, I understand the frustration and
I'm sure we'll want to introduce various forms of wildcards, but I
think there will be a lot of opinions about that should actually look
like. One can, as you've done here, follow index links from child to
parent. One can do a wildcard match on the index name. One could want
to specify an index on a particular column rather than a specific
name, to survive index renamings. I wouldn't be surprised if there are
other ideas, too. Three weeks before feature freeze is not the time to
be taking an opinionated position on what the best answers will
ultimately turn out to be. It's easy to write a tool that will spit
out matching index advice for all indexes involving in a partitioning
hierarchy, and I think that's what people should do for now. Or,
perhaps they should use the generated advice from actual plans instead
of writing hand-crafted advice.

We always have the option to add more to this in the future, but
taking things out is not half so easy.

> 2) I find the join strategy advice hard to use - pg_hint_plan has
> hints (e.g. HashJoin) that allow saying "use a hash join when joining
> these two rels in any order", vs pg_plan_advice requires setting a
> specific outer rel, which only makes sense when you want to fully
> specify every aspect of the plan. I suspect users who directly write
> advice will struggle with specifying join strategy advice as it is
> right now. We could consider having a different syntax for saying "I
> want a hash join when these two rels are joined, but I don't care
> about the order", e.g. "USE_HASH_JOIN(a b)". If you think that's
> worthwhile I'd be happy to take a stab at a patch.

I'd be inclined to classify that as a design error in pg_hint_plan,
but maybe I'm just not understanding something. Under what
circumstances would you know that you wanted two tables to be joined
via a hash join but not care which one was on which side of the join?
The rels on the two sides are treated very differently. One of them
needs to be small enough to fit in the hash table and should be one
where repeated index lookups aren't better (else, you should be
advising a nested loop and an index scan). The other can be basically
anything. What I know (in a situation where I might write some advice
manually) is that I want a certain table to be the one that goes into
the hash table, not that there should be a hash join someplace in the
general vicinity of one of the tables.

Also, there's a definitional question here. What exactly does
USE_HASH_JOIN(a b) mean? Possible definitions:

1. "a" must be joined directly to "b" without any other tables on
either side, and a hash join must be used to perform that join.
2. either "a" must appear on the inner side of a hash join with "b"
somewhere on the other side, possibly accompanied by other tables, or
the reverse
3. the plan must contain at least one hash join where "a" and "b"
appear on opposite sides of the join

Suppose I have a fact table f and three dimension tables d1, d2, and
d3. If I wrote USE_HASH_JOIN(f d1) USE_HASH_JOIN(f d2) USE_HASH_JOIN(f
d3), what happens? Under definition 1, the advice fails, because f
must be first joined to either d1, d2, or d3, and whichever one is
chosen, the other advice can't now be satisfied. Under definition 2, I
will definitely end up with hash joins all the way through, but it's
possible that the driving table will be one of the dimension tables,
which can be first joined to f and then to the other tables. Under
definition 3, I'm not even guaranteed to end up with all hash joins: I
can join d1, d2, and d3 to each other in any way I like, and then hash
join the result of that to f in either direction, and the rule is
satisfied for all three advice items.

None of those possibilities sound right. I argue that definitions 1
and 3 produce such absurd results in this scenario that they're not
even worth any further consideration, but I can see someone arguing
that definition 2 doesn't sound so bad. After all, you could still fix
it to achieve the probably-expected outcome by also writing
JOIN_ORDER(f). But that only works here because we know what we want
the driving table to me. Suppose alternatively that we have two large
tables B1 and B2 and a small table S, and we've figured out that the
planner tends to like to use the index on table S when it really ought
to be using a hash join, but we trust its judgement as to how to join
B1 and B2. With HASH_JOIN() as I've implemented it, that's easy: just
write HASH_JOIN(S). With your USE_HASH_JOIN, how do I do that exactly?
If I write USE_HASH_JOIN(B1 S), definition 2 permits a plan like this:

Hash Join
  -> Nested Loop
    -> Seq Scan on B2
    -> Index Scan on S
  -> Hash
    -> Seq Scan on B1

To me, that looks a heck of a lot like my USE_HASH_JOIN() locution
just didn't do anything. It certainly didn't do what I intended it to
do, and the only way I can make sure that something like this doesn't
happen is to *also* constrain the join order. If I'm willing to decide
which of B1 and B2 should be the driving table, then I can write
JOIN_ORDER(B1 B2 S) or JOIN_ORDER(B2 B1 S) and now everything is
fixed. But this seems 100% backwards given your stated goal: you want
to be able to constraint the join method without constraining the join
order. As I see it, the problem here is that a symmetric USE_HASH_JOIN
directive either uses something like definition 1 which is too tight a
constraint, or definition 2 or 3 which are extremely weak constraints
that essentially allow the planner to satisfy the constraint using
some other part of the plan tree.

Now, of course, I got to pick these examples, so I picked examples
that prove my point. Maybe there are examples where a "one side or the
other" constraint actually works better. But I don't know what those
examples are. When I've experimented this kind of thing, I've found
that I never get the results that I want because the planner just does
something stupid that technically satisfies the constraint but is
nothing like what I actually meant. If you know of examples where my
definitions suck and the "one side or the other" definition produces
great results, I'd love to hear about them ... although I would have
loved to hear about them even more 4.5 months ago when I first posted
this patch set and already had the phrase "useless in practice" in the
README on exactly this topic. This is exactly why I put the patches up
for design review before they were fully baked.

> For v20-0001, from a quick conceptual review:
>
> I find the two separate GUC mechanisms for local backend vs shared
> memory a bit confusing as a user (which one should I be using?).
> Enabling the shared memory mechanism on a system-wide basis seems like
> it'd likely have too high overhead anyway for production systems?
> (specifically worried about the advice capturing for each plan, though
> I haven't benchmarked it)
>
> I wonder if we shouldn't keep this simpler for now, and e.g. only do
> the backend local version to start - we could iterate a bit on
> system-wide collection out-of-core, e.g. I'm considering teaching
> pg_stat_plans to optionally collect plan advice the first time it sees
> a plan ID (plan advice is roughly a superset of what we consider a
> plan ID there), and then we could come back to this for PG20.

The shared version is rather useful for testing, though. That's
actually why I created it initially: turn on the shared collector, run
the regression tests, and then use SQL to look through the collector
results for interesting things. You can't do that with the local
collector.

> To help assess impact, I did a quick test run and looked at three
> not-yet-committed patches in the commitfest that affect planner logic
> ([0], [1] and [2]), to see if they'd require pg_plan_advice changes
> (master with v20-0002 applied). Maybe I picked the wrong patches, but
> at least with those no pg_plan_advice changes were needed with the
> test_plan_advice test enabled.

Nice.

> On the code itself:
> - Is there a reason you're setting
> "pg_plan_advice.always_store_advice_details" to true, instead of using
> pg_plan_advice_request_advice_generation?
> - I wonder if we could somehow detect advice not matching? Right now
> that'd be silently ignored, i.e. you'd only get a test failure when we
> generate the wrong advice that causes a plan change in the regression
> tests.

I think it already does more than what you seem to be thinking:
test_plan_advice checks the advice feedback, too. However, it's also
true that even more could be done. The code proposed here checks that
all advice is /* matched */ without being /* failed */ or /*
inapplicable */, but I have code locally that checks the reverse,
namely that every decision that, during the re-plan, every decision
could have been constrained by advice actually was. I felt it was a
bit too late to add that to what I was submitting for v19, but that
decision could certainly be revisited. Taking it even further, we
could do structural comparisons of the before-and-after plans, or we
could search for disabled nodes aside from what default_pgs_mask
should imply. I'm not very confident that those things are worth the
code they would take to implement, though. The existing checks found a
lot of bugs, but I think whatever is still wrong is probably not
super-likely to be found by additional cross-checks. That could be
wrong; it's just a hunch.

> For v20-0003, initial thoughts:
>
> I think getting at least a basic version of this in would be good, as
> a server-wide way to set advice for queries can help people get out of
> a problem when Postgres behaves badly - and we know from pg_hint_plan
> (which has a hint table) that this can be useful even without doing
> any kind of parameter sniffing/etc to be smart about different
> parameters for the same query.

Yep.

> The name "stash" feels a bit confusing as an end-user facing term.
> Maybe something like "pg_apply_advice", or "pg_query_advice" would be
> better? (though I kind of wish we could tie it more closely to "plan
> advice", but e.g. "pg_plan_advice_apply" feels too lengthy)

I've heard that other people also find "stash" not as intuitive as it
could be, and I'm open to changing it. However, whatever we call this
has to make its relationship to pg_plan_advice clear. If we have
pg_plan_advice and pg_query_advice, which one is the core module and
which one is automatically supplying advice? You can't tell from the
name, which I think will be confusing. In the long run, I suspect we
will end up with a moderately-large pile of tools for either capturing
advice or automatically supplying it, and we should think about how
we'd like all those to get named. Right now, I suppose we might end up
with something like this:

pg_plan_advice: The core. There can only be one.

different ways of capturing advice strings: pg_collect_advice,
pg_capture_advice, pg_save_advice, pg_baseline_advice, ....

different ways of supplying advice strings: pg_stash_advice,
pg_auto_advice, pg_lookup_advice, pg_store_advice,
pg_advice_from_query_comment, ...

I don't love this, because the pattern I see developing here is that
module names will either be very long or they'll just take a word from
an existing module name (like "collect") and replace it with a synonym
(like "capture"). That's not going to produce anything very mnemonic,
so it would be nice to do better. But I think the route to doing
better has to be to get more specific with the naming, not less. Like,
if we renamed pg_stash_advice to
pg_lookup_in_memory_advice_by_query_id, then the name tells you
EXACTLY what it does, which is great. Unfortunately, the name is also
very long, which is why I didn't go that route.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  135. Re: pg_plan_advice

    Lukas Fittl <lukas@fittl.com> — 2026-03-16T23:25:05Z

    On Mon, Mar 16, 2026 at 1:51 PM Robert Haas <robertmhaas@gmail.com> wrote:
> On Fri, Mar 13, 2026 at 4:39 AM Lukas Fittl <lukas@fittl.com> wrote:
> > - pgpa_parser uses pgpa_yyerror, but that function doesn't use
> > elog(ERROR) like the main parser does - I think it'd be more
> > consistent to use YYABORT explicitly, e.g. how we do it in cubeparse
>
> Perhaps, but I think this needs more thought, because you haven't done
> anything about the calls in pgpa_scanner.l. I think right now, the
> operating principle is that parsing continues after an error and that
> we mustn't crash as a result, though the resulting tree will probably
> be invalid for practical use. If we're going to change that to stop on
> the spot, I think we should probably do it for both lexing and
> parsing, and think about whether that leads to any other changes or
> simplificatons.

Fair - I don't think there is a practical impact from not doing the
YYABORT calls, since as you say care is taken to not crash in that
case.

> > From a usability perspective, I do wonder about two things when it
> > comes to users specifying advice directly (which will happen in
> > practice, e.g. when debugging plan problems):
> >
> > 1) The lack of being able to target scan advice (e.g. SEQ_SCAN) to a
> > partition parent is frustrating - I believe we discussed partitioning
> > earlier in the thread in terms of gathering+applying it, but I do
> > wonder if we shouldn't at least allow users to specify a partitioned
> > table/partitioned index, instead of only the children. See attached
> > nocfbot-0002 for an idea what would be enough, I think.
>
> I'm not on board with this without a lot more study. I've been down
> this road before, and it can easily end in tears. Examining the
> partition structure on the fly can have a performance cost, and it
> might even have security ramifications or at least bugs if there are
> concurrent modifications to the partitioning structure happening.
> Also, the test_plan_advice framework doesn't do much to tell us
> whether this actually works. Also, I understand the frustration and
> I'm sure we'll want to introduce various forms of wildcards, but I
> think there will be a lot of opinions about that should actually look
> like. One can, as you've done here, follow index links from child to
> parent. One can do a wildcard match on the index name. One could want
> to specify an index on a particular column rather than a specific
> name, to survive index renamings. I wouldn't be surprised if there are
> other ideas, too. Three weeks before feature freeze is not the time to
> be taking an opinionated position on what the best answers will
> ultimately turn out to be. It's easy to write a tool that will spit
> out matching index advice for all indexes involving in a partitioning
> hierarchy, and I think that's what people should do for now.

That's fair. I would like us to do something about this in the PG20
release cycle - for my part, I think its reasonable to follow the
declarative partitioning parent-child relationship for indexes if
present - assuming we can sort out the performance/etc. aspects of
that.

For 19 I think we might want to consider calling this out more
explicitly in the documentation under the "Scan Method Advice"
paragraph, i.e. that one cannot specify partition parent table names
(at least not ones that have no data of their own) and instead one has
to specify the partitions individually. Otherwise I think users will
just be confused by the Append node that says "Disabled: true" and the
advice that didn't match.

> > 2) I find the join strategy advice hard to use - pg_hint_plan has
> > hints (e.g. HashJoin) that allow saying "use a hash join when joining
> > these two rels in any order", vs pg_plan_advice requires setting a
> > specific outer rel, which only makes sense when you want to fully
> > specify every aspect of the plan. I suspect users who directly write
> > advice will struggle with specifying join strategy advice as it is
> > right now. We could consider having a different syntax for saying "I
> > want a hash join when these two rels are joined, but I don't care
> > about the order", e.g. "USE_HASH_JOIN(a b)". If you think that's
> > worthwhile I'd be happy to take a stab at a patch.
>
> I'd be inclined to classify that as a design error in pg_hint_plan,
> but maybe I'm just not understanding something. Under what
> circumstances would you know that you wanted two tables to be joined
> via a hash join but not care which one was on which side of the join?

I think the common case would be someone sees the planner picked a
Nested Loop, and instead wants to see the plan that prefers a Hash
Join (or Merge Join), e.g. to understand costing differences. That's
how I usually use pg_hint_plan, to understand what the alternate plan
looked like that the planner didn't pick, but where costs were close.
The top-level "enable_nestloop = off" often tends to not work that
well for complex plans, hence my historic use of pg_hint_plan's
HASHJOIN/MERGEJOIN (or NO_NESTLOOP) hints for this purpose.

> Also, there's a definitional question here. What exactly does
> USE_HASH_JOIN(a b) mean? Possible definitions:
>
> ...
>
> Now, of course, I got to pick these examples, so I picked examples
> that prove my point. Maybe there are examples where a "one side or the
> other" constraint actually works better. But I don't know what those
> examples are. When I've experimented this kind of thing, I've found
> that I never get the results that I want because the planner just does
> something stupid that technically satisfies the constraint but is
> nothing like what I actually meant. If you know of examples where my
> definitions suck and the "one side or the other" definition produces
> great results, I'd love to hear about them ...

Thanks for the detailed work through - I think I see your
implementation choice for this more clearly now. I've also re-read the
documentation section on join methods and I think that is clear enough
in terms of how it works.

I don't think a change here is necessary. I think for the use case I
described I will just resort to testing both variants (i.e. being more
specific which shape of plan I want), which I think aligns with the
goals of pg_plan_advice as compared to pg_hint_plan.

Later in the release cycle I'll see if I can put together a community
resource that compares pg_hint_plan to pg_plan_advice, and where they
differ. I suspect many end users will have similar questions, and
whilst I don't think explaining the differences belongs in the regular
Postgres docs, it could fit the wiki as a cheatsheet of sorts.

> although I would have
> loved to hear about them even more 4.5 months ago when I first posted
> this patch set and already had the phrase "useless in practice" in the
> README on exactly this topic. This is exactly why I put the patches up
> for design review before they were fully baked.

Understood - I'll admit I mainly looked at the high-level join logic
before (and the join hooks in detail when doing the pg_hint_plan
testing) but had not fully understood how you dealt with join
hierarchies / specifying them in the advice. I had previously looked
at examples where multiple tables were listed assuming it worked like
hint plan, but that's not the case.

>
> > For v20-0001, from a quick conceptual review:
> >
> > I find the two separate GUC mechanisms for local backend vs shared
> > memory a bit confusing as a user (which one should I be using?).
> > Enabling the shared memory mechanism on a system-wide basis seems like
> > it'd likely have too high overhead anyway for production systems?
> > (specifically worried about the advice capturing for each plan, though
> > I haven't benchmarked it)
> >
> > I wonder if we shouldn't keep this simpler for now, and e.g. only do
> > the backend local version to start - we could iterate a bit on
> > system-wide collection out-of-core, e.g. I'm considering teaching
> > pg_stat_plans to optionally collect plan advice the first time it sees
> > a plan ID (plan advice is roughly a superset of what we consider a
> > plan ID there), and then we could come back to this for PG20.
>
> The shared version is rather useful for testing, though. That's
> actually why I created it initially: turn on the shared collector, run
> the regression tests, and then use SQL to look through the collector
> results for interesting things. You can't do that with the local
> collector.

Right - I can see the usefulness for testing, but I worry that people
use it on production systems and then experience unexpected
performance issues. That said, we could address that with a warning in
the docs noting its not intended for production use.

Out of time for today to think through naming for 0003 more, but I'll
see that I find more time this week.

Thanks,
Lukas

-- 
Lukas Fittl

  136. Re: pg_plan_advice

    Zsolt Parragi <zsolt.parragi@percona.com> — 2026-03-17T05:06:08Z

    > Thanks. The changes to the Makefile seem to mirror what is done in
> contrib/isn/Makefile, but I'm not so sure about the meson.build
> changes. sepgsql uses dir_data / 'contrib' rather than
> dir_include_server. src/pl/pl{perl,pgsql,python} use
> dir_include_server, but they also live in src/pl, not contrib. I don't
> think I understand what the underlying principal is supposed to be
> here. If you or anyone else knows, please enlighten me.

PGXS defines it as:

#   HEADERS_$(MODULE) -- files to install into
#     $(includedir_server)/$MODULEDIR/$MODULE; the value of $MODULE must be
#     listed in MODULES or MODULE_big

where

#   MODULEDIR -- subdirectory of $PREFIX/share into which DATA and DOCS files
#     should be installed (if not set, default is "extension" if EXTENSION
#     is set, or "contrib" if not)

And I mirrored that in meson.

Data seems to be wrong for headers, as that's

#   DATA -- random files to install into $PREFIX/share/$MODULEDIR

> sepgsql uses dir_data / 'contrib'

Also, sepgsql installs an sql file, not an include.

  137. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-03-17T13:44:34Z

    On Mon, Mar 16, 2026 at 7:25 PM Lukas Fittl <lukas@fittl.com> wrote:
> That's fair. I would like us to do something about this in the PG20
> release cycle - for my part, I think its reasonable to follow the
> declarative partitioning parent-child relationship for indexes if
> present - assuming we can sort out the performance/etc. aspects of
> that.

I wonder if instead of doing this as you did it, we should try to make
partition expansion happen around expand_inherited_rtentry time. So
maybe you can write something like PARTITIONED_SEQ_SCAN(x) or
PARTITIONED_INDEX_SCAN(x y) and, internally, that gets replaced with
one SEQ_SCAN(..) or INDEX_SCAN(...) entry per child before the core
planning logic engages. And, during the plan walk, we could do the
reverse transformation: if we computed matching advice items for every
partition, consolidate them down to just one before constructing the
final advice string. If we did that, the whole thing would be
symmetric and we'd have a certain amount of automatic test coverage,
plus we'd shorten a lot of automatically written advice strings
considerably. Maybe this is not better than the more on-the-fly way
that you did it, but I think it's worth some study.

While I'm on the subject, there are some other opportunities for
brevity that I have not pursued for this release. In particular,
NO_GATHER(...) often seems quite tedious to me. We could introduce a
wildcard, like "*", that just means everything, so you could write
NO_GATHER(*) for non-parallel queries. However, that seems like it's
actually not great, because as soon as you have a single Gather or
Gather Merge node in the plan, you're back to needing to write all the
other ones out. So another idea is to let you write something like
ONLY_EXPLICIT_GATHER() as a no-argument incantation that means that
everything that isn't mentioned as a GATHER() or GATHER_MERGE() target
should be considered NO_GATHER(). Or you could call that something
like NO_GATHER_OTHERS() or whatever. Or maybe there could be some
general default facility that lets you say what should happen when
nothing is specified, like DEFAULTS(NO_GATHER), but right now the
number of things that you could apply that to would be quite limited.

> For 19 I think we might want to consider calling this out more
> explicitly in the documentation under the "Scan Method Advice"
> paragraph, i.e. that one cannot specify partition parent table names
> (at least not ones that have no data of their own) and instead one has
> to specify the partitions individually. Otherwise I think users will
> just be confused by the Append node that says "Disabled: true" and the
> advice that didn't match.

We can consider that, but I think the bigger picture here is that
writing advice strings by hand is hard, and that's why EXPLAIN
(PLAN_ADVICE) and pg_collect_advice exist -- to give you a starting
point. I fear that if we try to enumerate a lot of specific examples
of ways in which people might be confused in the documentation, they
won't read it, and they'll still be confused. I think the primary
focus of the documentation should be to get people to use the advice
generation facilities as their main way to discover how to use the
system, and then pointing out specific things that may still be
confusing where it makes sense is also good to do. For example, in a
case like this, if you sit down and write INDEX_SCAN(partitioned_table
partitioned_index), yeah that's not going to work, and you're going to
be confused (potentially). But that's not really what you're supposed
to do. You're supposed to start by running EXPLAIN (PLAN_ADVICE) on
the query whose plan you're trying to manipulate, and if you do that,
you'll see that the generated advice shows a separate INDEX_SCAN() or
SEQ_SCAN() item for each child table, and then it's sorta obvious what
you're supposed to be doing. You may very well not like that -- I
think many, many people are going to complain about it -- but you'll
understand what is possible with the system that we have. Now that is
not to say that I think you're wrong about documenting this, and I've
certainly tried to document some other instances of things that I
found confusing even as the author of the system. However, there's
also a lot of cases that I haven't tried to document because it's just
too much useless, abstract information. On this particular point, if
there's a nice plan to fit this into the documentation that doesn't
feel like a jarring topic shift or a long detour into minor details,
I'm fine with it, but I don't think it's worth a lot of contortions to
fit this specific thing in considering how many other things there
are.

> I think the common case would be someone sees the planner picked a
> Nested Loop, and instead wants to see the plan that prefers a Hash
> Join (or Merge Join), e.g. to understand costing differences. That's
> how I usually use pg_hint_plan, to understand what the alternate plan
> looked like that the planner didn't pick, but where costs were close.
> The top-level "enable_nestloop = off" often tends to not work that
> well for complex plans, hence my historic use of pg_hint_plan's
> HASHJOIN/MERGEJOIN (or NO_NESTLOOP) hints for this purpose.

It's interesting to me that this works well for you even in complex
plans. For example, let's say I have something like this (omitting
sorts):

Merge Join
-> Nested Loop
  -> Hash Join
    -> Seq Scan on A
    -> Hash
      -> Seq Scan on B
  -> Index Scan on C
-> Nested Loop
  -> Seq Scan on D
  -> Index Scan on E

Now, what do you write here to get rid of the nested loop involving C?
Your example before of USE_HASH_JOIN(x y) seemed to imply that you
mentioned two tables, but this is a 3-table join, so are you
mentioning all three tables in this case, like USE_HASH_JOIN(A B C)?
Or are you mentioning just C, or A and C, or what? In pg_plan_advice,
it's just HASH_JOIN(C). I'm unclear what the right answer is in
pg_hint_plan. The documentation says that HashJoin(table table [...])
forces hash join for the joins on the tables specified, but I don't
know whether that means that the whole thing functions as a single
constraint (i.e. the N-way join product of all mentioned tables should
be computed using a HashJoin at the uppermost level) or as a
constraint per table (every mentioned table should be involved in a
hash join). If it's the former, then you could write HashJoin(A B C)
in this case, but that wouldn't preclude switching the join order so
that the A-C join is done first using a Nested Loop and then the join
to B is done afterward as a HashJoin, which is probably not what you
wanted. If it's the latter, then HashJoin(C) is probably good enough,
although not necessarily. If there are join clauses connecting B and
C, the planner could try to cheat by doing a hash join between B and C
first and then doing a Nested Loop join to A afterwards, which is also
probably not what you wanted, but typically there won't be such join
clauses so it will work out. Also, if the HashJoin(C) just means there
has to be a hash join somewhere above C, rather than that C has to be
on onside or the other of a hash join by itself, then the planner
could also cheat by switching the outer merge join to a hash join, but
I'm guessing that probably isn't what it means. But if that's the
case, then what would you write you did want to replace the outer
merge join with a hash join? Both sides have more than one table, so
if HashJoin(x) means x has to appear alone on one side of the join,
there would be no way to get what you want here. I wish the actual
behavior of pg_hint_plan were better-documented here, and I'd
appreciate an explanation of how you use it in a case like this and
what actually happens.

But all that having been said, I do think there's space for softer
constraints than what pg_plan_advice currently offers. For example,
there's currently no way to say "I'd like an index scan but I don't
care which index you use". I've been thinking we could invent
ANY_INDEX_SCAN() and ANY_INDEX_ONLY_SCAN() for that purpose at some
point, or maybe people would prefer negative constraints instead, like
NO_SEQ_SCAN(). And maybe your idea of either-way join method
constraints falls in that category too. It's easiest for me to imagine
someone wanting that for merge join, where the two sides are treated
more nearly symmetrically. But I think we would need to nail down
exactly what the semantics of that are. Given that we've got a
sublists available as a tool, we could define a "symmetric join method
request" to take two arguments where the first is the relation
identifiers, or a list of all the relation identifiers, that appear on
one side of the join, and the same for the other side of the join. So
in the example above, if you wanted to replace the nested loop with a
hash join in one direction or the other, you could write
FLIPFLOPPY_HASH_JOIN((A B) C), and if you wanted to replace the outer
merge join, you could write FLIPFLOPPY_HASH_JOIN((A B C) (D E)). I'm
not altogether convinced that's better than just writing HASH_JOIN(C)
or HASH_JOIN((D E)), and there's some definitely user and code
complexity to supporting both methods, but maybe it will turn out to
be the right thing.

(It also occurs to me that the proposed semantics of
FLIPFLOPPY_HASH_JOIN are actually both stronger and weaker than the
existing HASH_JOIN directive. It's weaker in that the sides of the
join can be switched, but it's stronger in that it constrains what has
to be on both sides of the join, whereas HASH_JOIN does not do that.
Of course, as is hopefully clear by now, this is not the only possible
set of semantics that we could choose to implement here: things that
seem simple when you think about 2-table cases are often not simple at
all when scaled up to more complex situations. More than anything
else, I want whatever we implement to be extremely well-defined, with
absolutely no room for debate about what a given advice tag does or
whether a certain plan complies with a certain advice item.)

> Later in the release cycle I'll see if I can put together a community
> resource that compares pg_hint_plan to pg_plan_advice, and where they
> differ. I suspect many end users will have similar questions, and
> whilst I don't think explaining the differences belongs in the regular
> Postgres docs, it could fit the wiki as a cheatsheet of sorts.

Yeah, that sounds nice.

> Right - I can see the usefulness for testing, but I worry that people
> use it on production systems and then experience unexpected
> performance issues. That said, we could address that with a warning in
> the docs noting its not intended for production use.

I think it depends a lot on what you mean by "production use". I think
it's definitely true that pg_collect_advice is not intended for
continuous advice collection. If you try to use it that way, you'll
either start throwing away entries you wanted to keep (if the
collection limit is low) or you'll blow out memory (if the collection
limit is high). But it's completely reasonable to enable it on a
production server in a controlled way, to collect all the queries and
advice strings for one representative transaction of each type (or
even 10 or 50 representative transactions of each type). So I feel
like this is about understanding how it's intended to be used, and the
answer is definitely not "just like pg_stat_statements!".

BTW, I wonder if it would be worth considering, obviously for next
release cycle rather than this one, extending pg_stat_statements to
have the ability to grab plan advice as well, rather than building the
query normalization and deduplication features that pg_stat_statements
already has into pg_collect_advice.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  138. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-03-17T17:44:06Z

    On Tue, Mar 17, 2026 at 1:06 AM Zsolt Parragi <zsolt.parragi@percona.com> wrote:
> PGXS defines it as:
>
> #   HEADERS_$(MODULE) -- files to install into
> #     $(includedir_server)/$MODULEDIR/$MODULE; the value of $MODULE must be
> #     listed in MODULES or MODULE_big
>
> where
>
> #   MODULEDIR -- subdirectory of $PREFIX/share into which DATA and DOCS files
> #     should be installed (if not set, default is "extension" if EXTENSION
> #     is set, or "contrib" if not)
>
> And I mirrored that in meson.

Right, OK. This all seems rather confusing and a bit under-documented,
but after looking it over I think you've got it correct, so committed.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  139. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-03-17T21:45:52Z

    On Thu, Mar 12, 2026 at 1:15 PM Robert Haas <robertmhaas@gmail.com> wrote:
> I'm still hoping to get some more feedback on the remaining patches,
> which are much smaller and conceptually simpler. While there is no
> time to redesign them at this point in the release cycle, there is
> still the opportunity to fix bugs, or decide that they're too
> half-baked to ship. So here is v20 with just those patches. Of course,
> post-commit review of the main patch is also very welcome.

I've now committed test_plan_advice, since it seems crazy to me to
have pg_plan_advice in the tree without it and reviewers evidently
agree at least with the concept test_plan_advice is something we
should have. Here are the remaining two patches, as v21.

Separately, I also committed a fix for the GEQO crash that Alexander
Lakhin found. The patch I proposed on list was missing a bms_copy() --
as proposed, it fixed the crash but was still wrong. I added that and
committed it.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  140. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-03-18T17:07:54Z

    Hi,

Coverity complained about the fact that pgpa_planner_feedback_warning
does not do anything to free the memory used by flagbuf or detailbuf,
and Tom asked whether that failure could amount to a significant leak.
The best way to use a lot of memory here is to have a very long advice
string that doesn't do anything useful, so I created this test case:

load 'pg_plan_advice';
select set_config('pg_plan_advice.advice', v.v, false) from (select
string_agg('SEQ_SCAN(hogehogehoge' || g || ')', ' ') v from
generate_series(1,10000) g) v;
set pg_plan_advice.feedback_warnings = true;
select 1;

While this is not the worst case imaginable, it's pretty bad. There's
probably no use case for having 10000 advice items in your advice
string even if they were all valid, and here none of them are valid
for the final query (select 1). So this produces "WARNING:  supplied
plan advice was not enforced" with a 10000 line detail message where
all the lines look like this, but with different numbers:

advice SEQ_SCAN(hogehogehoge7348) feedback is "not matched"

Failure to free the buffers costs us just over 1MB of memory, which is
not wildly disproportionate given that the message itself is over half
a MB long, so I'm not sure that freeing it is all that useful here. I
don't think the context we use for planning normally sticks around for
all that long. If I'm wrong about that and it does, then we should
probably wrap our own shorter-lived context around the stuff this
module is doing rather than trying to free allocations individually.

But this does raise two points which are perhaps worthy of some
further consideration:

1. Maybe we should limit the length of the detail message. In some
other cases, like reportDependentObjects, we limit the detail message
to the first 100 problems found, and then say at the end of that
detail message how many more problems there were afterwards. We could
do that here, too. I'm not 100% sure it's worth the code, but then
again it's not much code.

2. The way the current code works is to transform the advice feedback
into a Node-tree representation which is stashed in the PlannedStmt's
extension_state, and then also passes that Node-tree representation to
pgpa_planner_feedback_warning, which generates the actual warning.
That Node-tree representation is currently used by EXPLAIN to display
the advice feedback, which I think for most people will be a nicer
interface than enabling pg_plan_advice.feedback_warnings, and it can
also be used by other extensions. For instance, we could extend
pg_stash_advice so that it looks at the advice feedback and updates
the shared store, so that users can monitor whether their stashed
advice is doing what they hoped it would.

However, in a case like this, that Node tree is actually quite large:
about 16MB. I guess that's because pgpa_planner_append_feedback() has
to do multiple allocations for every item of feedback: a C string,
DefElem, an Integer, plus whatever lappend() charges us to add to a
List. We could save that memory by adding code here to optimize for
the case where we need to generate warnings but we don't need the Node
tree for anything else. I'm inclined not to do that, because (A) I
don't think temporarily using 16MB when the user specifies 10,000
items of bogus advice is really that bad and (B) complicating the code
has its own costs, such as maybe introducing more bugs. However, maybe
someone else sees it differently.

Another idea is to try to find a more economic Node representation.
For instance, we could jam the flags into the DefElem's location
field, instead of building a separate Integer node, or we could build
the advice feedback as a giant binary blob and wrap it in a varlena
and a Const node and leave consumers to make sense of that as best
they're able, or we could invent a new Node type that's just to the
perfect thing to hold a C string and an integer. I'm inclined to think
that the first two are too hacky and the third is too special-purpose,
but again someone else might see it otherwise.

Thoughts?

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  141. Re: pg_plan_advice

    Lukas Fittl <lukas@fittl.com> — 2026-03-18T18:59:46Z

    On Wed, Mar 18, 2026 at 10:08 AM Robert Haas <robertmhaas@gmail.com> wrote:
> But this does raise two points which are perhaps worthy of some
> further consideration:
>
> 1. Maybe we should limit the length of the detail message. In some
> other cases, like reportDependentObjects, we limit the detail message
> to the first 100 problems found, and then say at the end of that
> detail message how many more problems there were afterwards. We could
> do that here, too. I'm not 100% sure it's worth the code, but then
> again it's not much code.

I think we have similar problems elsewhere in Postgres where a large
user input causes an even larger log output - e.g. a case I'm familiar
with are complicated queries with long IN list inputs and their
associated EXPLAIN plans being logged by auto_explain - I recently had
a case where someone reported an OOM due to auto_explain trying to log
a > 100 MB sized query plan.

I think its actually less a problem with plan advice, since presumably
we won't have ORMs generating plan advice, and even if we do it'd be
per-table - so I think its unlikely a genuine use case would use 1000s
of advice tags.

That said, I also don't think super long long messages are actually
helpful. I do wonder if we should have a more coarse GUC that limits
DETAIL lines of any kind to a maximum length (e.g. 100 kB) across the
board instead of special casing every emitter.

> 2. The way the current code works is to transform the advice feedback
> into a Node-tree representation which is stashed in the PlannedStmt's
> extension_state, and then also passes that Node-tree representation to
> pgpa_planner_feedback_warning, which generates the actual warning.
> That Node-tree representation is currently used by EXPLAIN to display
> the advice feedback, which I think for most people will be a nicer
> interface than enabling pg_plan_advice.feedback_warnings, and it can
> also be used by other extensions. For instance, we could extend
> pg_stash_advice so that it looks at the advice feedback and updates
> the shared store, so that users can monitor whether their stashed
> advice is doing what they hoped it would.

Yeah, I think the ability for other extensions to retrieve this is
pretty important - whether in pg_stash_advice, or any other kind of
plan management extension that wants to know the outcome of applied
advice.

>
> However, in a case like this, that Node tree is actually quite large:
> about 16MB. I guess that's because pgpa_planner_append_feedback() has
> to do multiple allocations for every item of feedback: a C string,
> DefElem, an Integer, plus whatever lappend() charges us to add to a
> List. We could save that memory by adding code here to optimize for
> the case where we need to generate warnings but we don't need the Node
> tree for anything else. I'm inclined not to do that, because (A) I
> don't think temporarily using 16MB when the user specifies 10,000
> items of bogus advice is really that bad and (B) complicating the code
> has its own costs, such as maybe introducing more bugs. However, maybe
> someone else sees it differently.
>
> Another idea is to try to find a more economic Node representation.
> For instance, we could jam the flags into the DefElem's location
> field, instead of building a separate Integer node, or we could build
> the advice feedback as a giant binary blob and wrap it in a varlena
> and a Const node and leave consumers to make sense of that as best
> they're able, or we could invent a new Node type that's just to the
> perfect thing to hold a C string and an integer. I'm inclined to think
> that the first two are too hacky and the third is too special-purpose,
> but again someone else might see it otherwise.

Yeah, I feel like we're definitely constrained here by the fact that
advice tags are defined by a contrib module vs in-core - if they were
in-core we could just add a dedicated node type for them. I don't
think inventing a specialized binary format only defined in the
contrib module makes sense.

Two other ideas:

1) What if we return the utilized advice string as a separate DefElem
with a list of strings, and then the feedback just has to reference an
index into that list? (though I suppose that doesn't actually save
memory, now that I think that through -- unless we assume the caller
already has the advice string, but I don't think we can rely on that)

2) We could consider having separate DefElems for the different flag
types (i.e. "feedback_failed", "feedback_match_full", etc), and then a
list of strings attached to each - that'd save the nested DefElem and
the Integer node

Thanks,
Lukas

-- 
Lukas Fittl

  142. Re: pg_plan_advice

    Tom Lane <tgl@sss.pgh.pa.us> — 2026-03-18T20:44:34Z

    Don't know if you noticed yet, but avocet has shown [1] that one
pg_plan_advice test case is unstable under debug_discard_caches = 1:

diff -U3 /home/buildfarm/avocet/buildroot/HEAD/pgsql.build/contrib/pg_plan_advice/expected/prepared.out /home/buildfarm/avocet/buildroot/HEAD/pgsql.build/contrib/pg_plan_advice/results/prepared.out
--- /home/buildfarm/avocet/buildroot/HEAD/pgsql.build/contrib/pg_plan_advice/expected/prepared.out	2026-03-16 11:04:23.453026641 +0100
+++ /home/buildfarm/avocet/buildroot/HEAD/pgsql.build/contrib/pg_plan_advice/results/prepared.out	2026-03-17 20:42:16.262657929 +0100
@@ -57,11 +57,13 @@
 -- Prepared, but always_store_advice_details = true, so should show feedback.
 PREPARE pt4 AS SELECT * FROM ptab;
 EXPLAIN (COSTS OFF, PLAN_ADVICE) EXECUTE pt2;
-       QUERY PLAN       
-------------------------
+           QUERY PLAN           
+--------------------------------
  Seq Scan on ptab
+ Supplied Plan Advice:
+   SEQ_SCAN(ptab) /* matched */
  Generated Plan Advice:
    SEQ_SCAN(ptab)
    NO_GATHER(ptab)
-(4 rows)
+(6 rows)

I can reproduce this pretty trivially by adding
SET debug_discard_caches = 1;
at the top of contrib/pg_plan_advice/sql/prepared.sql.

It looks like the appearance of "Supplied Plan Advice:" depends
on whether the prepared query's plan got regenerated or not.
I'm not sure if this represents a bug (ie undesirable behavior) or
it's just that the test is being insufficiently careful about
being reproducible.

			regards, tom lane

[1] https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=avocet&dt=2026-03-16%2010%3A03%3A01

  143. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-03-18T22:19:31Z

    On Wed, Mar 18, 2026 at 4:44 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Don't know if you noticed yet, but avocet has shown [1] that one
> pg_plan_advice test case is unstable under debug_discard_caches = 1:

I had not. Thanks for the pointer.

> It looks like the appearance of "Supplied Plan Advice:" depends
> on whether the prepared query's plan got regenerated or not.
> I'm not sure if this represents a bug (ie undesirable behavior) or
> it's just that the test is being insufficiently careful about
> being reproducible.

Well, that's embarrassing: it's a copy-and-paste error. The test
prepares and executes pt1, then prepares and executes pt2, then
prepares pt3 and executes pt1, then prepares pt4 and execute p2. pt3
and pt4 are never used for anything. Also there's a related typo in a
comment. See attached.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  144. Re: pg_plan_advice

    Tom Lane <tgl@sss.pgh.pa.us> — 2026-03-18T22:26:42Z

    Robert Haas <robertmhaas@gmail.com> writes:
> On Wed, Mar 18, 2026 at 4:44 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> It looks like the appearance of "Supplied Plan Advice:" depends
>> on whether the prepared query's plan got regenerated or not.
>> I'm not sure if this represents a bug (ie undesirable behavior) or
>> it's just that the test is being insufficiently careful about
>> being reproducible.

> Well, that's embarrassing: it's a copy-and-paste error. The test
> prepares and executes pt1, then prepares and executes pt2, then
> prepares pt3 and executes pt1, then prepares pt4 and execute p2. pt3
> and pt4 are never used for anything. Also there's a related typo in a
> comment. See attached.

Ah ... so the observed behavior is because if pt2 does get replanned,
that happens with a different always_store_advice_details setting
than it used originally?

			regards, tom lane

  145. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-03-18T22:34:29Z

    On Wed, Mar 18, 2026 at 6:26 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> > Well, that's embarrassing: it's a copy-and-paste error. The test
> > prepares and executes pt1, then prepares and executes pt2, then
> > prepares pt3 and executes pt1, then prepares pt4 and execute p2. pt3
> > and pt4 are never used for anything. Also there's a related typo in a
> > comment. See attached.
>
> Ah ... so the observed behavior is because if pt2 does get replanned,
> that happens with a different always_store_advice_details setting
> than it used originally?

Close, but not quite. always_store_advice_details is set for the
even-numbered tests, so it's the same for pt2 and pt4. But
pg_plan_advice.advice is empty for the the first half of the file and
set for the second half of the file, so it's empty when pt2 is
prepared and contains SEQ_SCAN(ptab) when pt4 is prepared. That
accounts for the difference. avocet is actually doing the right thing,
and the rest of the buildfarm is doing the wrong thing for lack of
replanning.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  146. Re: pg_plan_advice

    Tom Lane <tgl@sss.pgh.pa.us> — 2026-03-18T22:43:54Z

    Robert Haas <robertmhaas@gmail.com> writes:
> On Wed, Mar 18, 2026 at 6:26 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> Ah ... so the observed behavior is because if pt2 does get replanned,
>> that happens with a different always_store_advice_details setting
>> than it used originally?

> Close, but not quite. always_store_advice_details is set for the
> even-numbered tests, so it's the same for pt2 and pt4. But
> pg_plan_advice.advice is empty for the the first half of the file and
> set for the second half of the file, so it's empty when pt2 is
> prepared and contains SEQ_SCAN(ptab) when pt4 is prepared. That
> accounts for the difference. avocet is actually doing the right thing,
> and the rest of the buildfarm is doing the wrong thing for lack of
> replanning.

Well, avocet is producing the output you want, but I think the rest
are behaving correctly given the way the script is written.

Anyway, I confirm that the patched output is stable with
debug_discard_caches = 1, so LGTM.

			regards, tom lane

  147. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-03-18T22:57:59Z

    On Wed, Mar 18, 2026 at 6:43 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Well, avocet is producing the output you want, but I think the rest
> are behaving correctly given the way the script is written.

Yeah, this goes back to my "that's embarrassing" comment: obviously, I
really did not proofread this well at all.

> Anyway, I confirm that the patched output is stable with
> debug_discard_caches = 1, so LGTM.

Yeah, I also tested that here. Glad it works that way for you also.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  148. Re: pg_plan_advice

    Tom Lane <tgl@sss.pgh.pa.us> — 2026-03-19T16:46:47Z

    Robert Haas <robertmhaas@gmail.com> writes:
> I've now committed test_plan_advice, since it seems crazy to me to
> have pg_plan_advice in the tree without it and reviewers evidently
> agree at least with the concept test_plan_advice is something we
> should have. Here are the remaining two patches, as v21.

I just realized that this new test module has added yet another
execution of the entire core regression test suite ... and to add
insult to injury, it runs the planner twice for each plannable
query therein.  I think this is an outrageous abuse of our buildfarm
and urgently request that you find some less climate-destroying
way of getting reasonable test coverage of pg_plan_advice.

As an example of what this has done to our slower buildfarm members,
a few days ago my own animal longfin could run the BF script in 32
minutes and change, if not too much recompilation was needed.  Now its
minimum cycle time seems to be 35-plus minutes, or about 10% slower.
I don't think pg_plan_advice deserves that large a fraction of the
project's resources forevermore.

I don't see any good reason why you couldn't get adequate coverage of
pg_plan_advice with a few dozen queries.  Certainly all the DDL
testing that happens in the core regression tests isn't adding
anything here.

I would dig into why grison and schnauzer are failing this test,
except that I don't agree that we should be running it in the
first place.

			regards, tom lane

  149. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-03-19T16:54:43Z

    On Wed, Mar 18, 2026 at 3:00 PM Lukas Fittl <lukas@fittl.com> wrote:
> I think we have similar problems elsewhere in Postgres where a large
> user input causes an even larger log output - e.g. a case I'm familiar
> with are complicated queries with long IN list inputs and their
> associated EXPLAIN plans being logged by auto_explain - I recently had
> a case where someone reported an OOM due to auto_explain trying to log
> a > 100 MB sized query plan.
>
> I think its actually less a problem with plan advice, since presumably
> we won't have ORMs generating plan advice, and even if we do it'd be
> per-table - so I think its unlikely a genuine use case would use 1000s
> of advice tags.
>
> That said, I also don't think super long long messages are actually
> helpful. I do wonder if we should have a more coarse GUC that limits
> DETAIL lines of any kind to a maximum length (e.g. 100 kB) across the
> board instead of special casing every emitter.

I think it would be difficult for generic code to do something
sensible when the message is really long. I mean, it could just cut it
off after N lines, but then you have no idea how much more there would
have been, and you probably want to tell the user something about
that. You could add a completely generic message to the end like "plus
10525 more lines of output that were truncated for display," but
that's pretty unsatisfying. If you want to show something contextually
appropriate, the implementation needs to be separate for each case
even if the limit is common. Anyway, the question here is not about
such a generic mechanism, but about whether somebody wants to argue
for sticking a limit of 100 on feedback messages on the theory that
log spam is bad, or whether it's fine as-is either because (a) the
likelihood of a significant number of people hitting that limit is
thought to be too low to worry about or (b) the likelihood of someone
wanting all of those messages (e.g. for machine-parsing) is thought to
be high enough that a limit is actually worse than no limit. I do not
really have a horse in the race, so if nobody else has a strong
opinion, I'm going to leave it alone for now and consider changing it
if a strong opinion materializes later.

> 1) What if we return the utilized advice string as a separate DefElem
> with a list of strings, and then the feedback just has to reference an
> index into that list? (though I suppose that doesn't actually save
> memory, now that I think that through -- unless we assume the caller
> already has the advice string, but I don't think we can rely on that)

I think that if we're using Integer nodes, it's just never going to be
very economical. We could use an IntList, which I believe would be
better, but there are a few complications that make me not like this
idea very much. One, what the feedback is actually about is a
reconstruction of a particular advice item into string form, not the
original string, e.g. if you input SEQ_SCAN(foo bar), the feedback
will be on SEQ_SCAN(foo) and SEQ_SCAN(bar). Two, even if you ignore
that, this would leave consumers of the data with the problem of
finding the end of the advice item unless you stored both starting and
ending indexes, which would have its own costs.

> 2) We could consider having separate DefElems for the different flag
> types (i.e. "feedback_failed", "feedback_match_full", etc), and then a
> list of strings attached to each - that'd save the nested DefElem and
> the Integer node

But it would also very often duplicate a bunch of the strings, which
seems likely to work out to a loss more often than not. You could
avoid that by have a list of strings per unique flag combination, but
that would be extra work to compute and I think it would be less
convenient for consumers. One user-visible consequence would be that
the advice feedback in EXPLAIN output would have much less to do with
the original order of the advice string.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  150. Re: pg_plan_advice

    Tom Lane <tgl@sss.pgh.pa.us> — 2026-03-19T17:02:18Z

    Robert Haas <robertmhaas@gmail.com> writes:
> ... Anyway, the question here is not about
> such a generic mechanism, but about whether somebody wants to argue
> for sticking a limit of 100 on feedback messages on the theory that
> log spam is bad, or whether it's fine as-is either because (a) the
> likelihood of a significant number of people hitting that limit is
> thought to be too low to worry about or (b) the likelihood of someone
> wanting all of those messages (e.g. for machine-parsing) is thought to
> be high enough that a limit is actually worse than no limit.

FWIW, I'm inclined to think that people won't actually have this
turned on unless they want to see that output, and then they'll
probably want to see all of it.  So I'm inclined to leave it as-is
for now.  If field experience teaches differently, we can always
improve it later.

			regards, tom lane

  151. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-03-19T17:17:04Z

    On Thu, Mar 19, 2026 at 12:46 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> I just realized that this new test module has added yet another
> execution of the entire core regression test suite ... and to add
> insult to injury, it runs the planner twice for each plannable
> query therein.  I think this is an outrageous abuse of our buildfarm
> and urgently request that you find some less climate-destroying
> way of getting reasonable test coverage of pg_plan_advice.

I don't think just nuking this is a reasonable option. During the
development of pg_plan_advice, test_plan_advice was the single most
valuable testing tool. It was not close. Manual review, both by me and
by others, found bugs; fuzz testing by Jacob Champion found bugs; AI
code review found bugs. test_plan_advice was an order of magnitude
more effective than all of those things put together. In fact, I'd say
maybe two orders of magnitude. Pretty much every significant logical
error was something that required a very specific query shape to find,
and the regression tests are by far the best repository of weird query
shapes that we have.

I think without something like test_plan_advice, the chances of us
noticing if future planner changes break pg_plan_advice are near zero,
and with it, they're quite good -- because we're not only testing the
queries that exist in the regression test suite now, but we're testing
future queries that are added to the regression test suite by people
who are hacking on the planner. Those added queries presumably create
plan shapes that are relevant to the code that they're changing,
whereas a fixed pg_plan_advice-only works if people think to add
something to it, which they very likely won't, and even if they do, I
have no confidence that they'll know what things to add and what not.
I certainly didn't know which queries in the regression tests were
going to break under test_plan_advice until I tried it.

One thing we might be able to do here to save on cycles is combine
test_plan_advice with some existing run of the regression tests. We
seem to run them to test sepgsql, to test pg_upgrade, and in
027_stream_regress.pl. I don't think we can piggyback on sepgsql here
because there are too many cases where it just won't be built or
tested, but we could possibly piggyback on one of the other runs, or
on the main regression tests. If that's not viable, another option
would be to have it run on only some buildfarm members rather than all
of them.  But if the tests aren't run by default, then I fear people
will experience quite a bit frustration the first time something
breaks only after they commit.

Before I forget, another idea that might help is to see if we can
tweak meson.build to start running this particular test earlier. I
thought about that during development, but I didn't actually do it. If
the issue is that test being last to finish, that could help. If the
issue is total resource consumption, it won't help with that.

> I would dig into why grison and schnauzer are failing this test,
> except that I don't agree that we should be running it in the
> first place.

I'll go have a look.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  152. Re: pg_plan_advice

    Matheus Alcantara <matheusssilv97@gmail.com> — 2026-03-19T20:12:32Z

    On 19/03/26 14:17, Robert Haas wrote:
> Before I forget, another idea that might help is to see if we can
> tweak meson.build to start running this particular test earlier. I
> thought about that during development, but I didn't actually do it. If
> the issue is that test being last to finish, that could help. If the
> issue is total resource consumption, it won't help with that.
> 
We can add a 'priority' for the test:

   'tap': {
     'tests': [
       't/001_replan_regress.pl',
     ],
+   'test_kwargs': {'priority': 50},
   },



Even if it's not help with resource consumption I think that it still 
worth adding. It reduces from ~5m to ~4m on my machine.

--
Matheus Alcantara
EDB: https://www.enterprisedb.com

  153. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-03-19T20:38:23Z

    On Thu, Mar 19, 2026 at 1:17 PM Robert Haas <robertmhaas@gmail.com> wrote:
> > I would dig into why grison and schnauzer are failing this test,
> > except that I don't agree that we should be running it in the
> > first place.
>
> I'll go have a look.

grison failed like this:

 CREATE INDEX ON vaccluster(wrap_do_analyze(i));
+ERROR:  indexes on virtual generated columns are not supported

This is a surprising diff, because either that command is trying to
create an index on a virtual generated column, or it's not, and it
either is or isn't for every machine in the buildfarm and under
test_plan_advice or not. The problem might be that the first
TupleDescAttr(...) == ATTRIBUTE_GENERATED_VIRTUAL test in DefineIndex
can be reached with attno == 0, which seems like it will then access
memory that is not part of the TupleDesc. If that memory happens to
contain a 'v' in the right spot, this will happen. (Is it not viable
to have TupleDescAttr() assert that the second argument is >= 0?)

skink has a failure that looks like this:

+WARNING:  supplied plan advice was not enforced
+DETAIL:  advice NESTED_LOOP_MEMOIZE(nt) feedback is "matched, failed"

I think this is caused by a minor bug in the pgs_mask infrastructure.
get_memoize_path() exits quickly when outer_path->parent->rows < 2, on
the theory that the resulting path must lose on cost. But that
presumes that we could do a plain nested loop instead, i.e. that
PGS_NESTLOOP_PLAIN is set. And it might not be. Before the pgs_mask
stuff, that case couldn't happen: enable_nestloop=off disabled all
nested loops, and enable_memoize=off disabled only the memoized
version, but there wasn't any way to disable only the non-memoized
version (which, of course, was part of the point of the whole thing).
I think the fix is as attached.

Unfortunately, the other failures look like they are pointing to a
rather more serious problem. schnauzer's got a failure that looks like
this:

+WARNING:  supplied plan advice was not enforced
+DETAIL:  advice SEQ_SCAN(pg_trigger@exists_1) feedback is "matched, failed"
+advice NO_GATHER(pg_trigger@exists_1) feedback is "matched, failed"

And an older run on skink has a failure that looks like this:

+WARNING:  supplied plan advice was not enforced
+DETAIL:  advice SEQ_SCAN(pg_trigger@exists_6) feedback is "matched, failed"
+advice NO_GATHER(pg_trigger@exists_6) feedback is "matched, failed"

What these failures have in common is that both of them involve
selecting from information_schema.views. What "matched, failed" means
is that we saw the advice target during planning and tried to
influence the plan, but then observed that the final plan doesn't
respect the supplied advice. The query for information_schema.views
involves three subplans, so the fact that exists_6 is mentioned here
is a strong hint that the AlternativeSubPlan machinery is in play
here. The query is planned once, and one of the two alternative
subplans is chosen, generating advice for that plan. Upon replanning,
the other plan is chosen, so the subplan for which we have plan advice
doesn't appear in the query at all, leading to this. Now, you might
wonder how that's possible, considering that we're planning the same
query twice in a row with advice that isn't supposed to change
anything. My guess is that it's possible because these machines are
slow and other tests are running concurrently. If those other sessions
execute DDL, they can send sinval messages, which can cause the second
planning cycle to see different statistics than the first one. That
then means the plan can change in any way except for what the advice
system already knows how to control, and choice of AlternativeSubPlan
is not in that set of things. I think I actually saw a failure similar
to this once or twice locally during development, but that was back
when the code had a lot of bugs, and I assumed that the failure was
caused by some transient bug in whatever changes I was hacking on at
the time, or some other bug that I fixed later, rather than being a
real issue. I think the reason it doesn't happen very often is because
the statistics have to change enough at just the right moment and even
on slower buildfarm machines, most of the time, they don't.

It's not really clear to me exactly where to place the blame for this
category of failure. One view is that tests are being run in parallel
and I didn't think about that, and therefore this is a defect in the
test methodology that needs to be rectified somehow (hopefully not by
throwing it out). We might be able to fix that by running the test
suite serially rather than in parallel, although I expect that since
you (Tom) are already unhappy with the time this takes, that will
probably go over like a lead balloon. Another angle is to blame this
on the decision to assign different plan names to the different
subroots that we use for the alternative subplans. If we used
exists_1, exists_2, and exists_3 twice each instead of
exists_1..exists_6, this wouldn't happen, though that idea seems
questionable on theoretical grounds. A third possible take is that not
including choice-of-alternative-subplan in the initial scope was an
error.

As of this moment, I'm not really sure which way to jump. I need to
think further about what to do about this one. We can continue the
discussion about reducing the cost at the same time; again, I am
definitely not saying that it isn't legitimate to be concerned about
the CPU cycles expended running these tests, but those CPU cycles have
found three separate problems in two days, which is not nothing.

Separately, I am now 100% convinced that I need to go revise the
pg_collect_advice patch, because that adds yet another run of the core
regression tests, but for much less possibility of any real gain. I'll
go get rid of that and figure out what, if anything, to replace it
with.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  154. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-03-19T21:33:37Z

    On Thu, Mar 19, 2026 at 4:12 PM Matheus Alcantara
<matheusssilv97@gmail.com> wrote:
> We can add a 'priority' for the test:
>
>    'tap': {
>      'tests': [
>        't/001_replan_regress.pl',
>      ],
> +   'test_kwargs': {'priority': 50},
>    },
>
> Even if it's not help with resource consumption I think that it still
> worth adding. It reduces from ~5m to ~4m on my machine.

I tested this out here. Without this change, 'meson test' takes
2:53-2:55 for me. After making the change, I got times from 2:53-2:56,
so basically no change. But I suspect your proposal here is still the
right thing to do. I wondered if it should actually do what
src/test/regress/meson.build does:

    'test_kwargs': {
      'priority': 50,
      'timeout': 1000,
    },

 ...but it seems as though the timeout for TAP tests is already 1000s,
so maybe we don't need to change anything. Or maybe the recent
"timedout" errors in the buildfarm are a sign that 1000s isn't long
enough for this more-intensive run:

https://buildfarm.postgresql.org/cgi-bin/show_failures.pl?max_days=3&stage=timedout&filter=Submit

If so, that would be sad. On my local machine, which is a ~3 yo
MacBook, running just the "regress" test suite takes ~11.1 s, and
running just the "test_plan_advice" suite takes ~12s, so I admit that
I'm slightly confused about why this is having such a big impact for
you and Tom. Obviously I'm not running with expensive options like
debug_discard_caches or Valgrind enabled, but presumably you're not
doing that locally either and it still shaves a minute off the runtime
for you. What exactly is different, I'm not entirely sure.

Anyway, I think I should still go make your suggested change, unless
somebody objects. We may change more later, but if this provides some
relief to some people for now, it seems worth doing.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  155. Re: pg_plan_advice

    Daniel Gustafsson <daniel@yesql.se> — 2026-03-19T22:02:28Z

    > On 19 Mar 2026, at 21:38, Robert Haas <robertmhaas@gmail.com> wrote:

> We can continue the
> discussion about reducing the cost at the same time; again, I am
> definitely not saying that it isn't legitimate to be concerned about
> the CPU cycles expended running these tests, but those CPU cycles have
> found three separate problems in two days, which is not nothing.

With that in mind it could perhaps be worth to have this as a v19 open and
keep the more exhaustive test for a bit to see if more things shake loose?

--
Daniel Gustafsson

  156. Re: pg_plan_advice

    Tom Lane <tgl@sss.pgh.pa.us> — 2026-03-19T22:03:14Z

    Robert Haas <robertmhaas@gmail.com> writes:
> .... Or maybe the recent
> "timedout" errors in the buildfarm are a sign that 1000s isn't long
> enough for this more-intensive run:

> https://buildfarm.postgresql.org/cgi-bin/show_failures.pl?max_days=3&stage=timedout&filter=Submit

I think those are all mostly-unrelated.  icarus has never yet been
stable enough to complete a BF run, and morepork and schnauzer's
wait_timeout parameters were set to 7200 (2 hours) which the test run
had organically grown to exceed.  Maybe test_plan_advice helped push
them over the edge, but they were going to need a larger setting
sometime soon anyway.  (And I see Mikael's fixed that.)

> If so, that would be sad. On my local machine, which is a ~3 yo
> MacBook, running just the "regress" test suite takes ~11.1 s, and
> running just the "test_plan_advice" suite takes ~12s, so I admit that
> I'm slightly confused about why this is having such a big impact for
> you and Tom.

Please note that I was citing the runtime of a much slower machine
(longfin is a 2018 mac mini).  But in any case, what I was griping
about was the additional cost added to a buildfarm run; I don't see
that test_plan_advice is a lot slower than the main regression tests.
It's just that those are already a significant investment, and we
just iterated them another time.

			regards, tom lane

  157. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-03-19T22:19:17Z

    On Thu, Mar 19, 2026 at 6:03 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Please note that I was citing the runtime of a much slower machine
> (longfin is a 2018 mac mini).  But in any case, what I was griping
> about was the additional cost added to a buildfarm run; I don't see
> that test_plan_advice is a lot slower than the main regression tests.
> It's just that those are already a significant investment, and we
> just iterated them another time.

Right, and there's definitely plenty of worthless crap in there that
isn't adding any value. For example, every \dWHATEVER command in the
regression test is running basically the same queries every time, and
after the first time we're probably not learning anything new. And all
the DDL commands that are part of the regression tests are fairly
useless here. The grison failure is actually triggered by a DDL
command, but I think that might just be luck rather than the
test_plan_advice module is doing anything to systematically increase
the likelihood of such findings. But I don't know how we can separate
the wheat from the chaff. Obviously a lot of the DDL and \d commands
in the tests are either setup for queries that we should care about
testing, or verification that those queries did what they were
supposed to do. If we split the main regression test suite up, so that
we had one test suite for planner-related stuff, another for DDL, and
another for data types, or something like that, then test_plan_advice
would probably mostly just need to run on the first of those. But that
kind of split seems like a lot of work.

Do you think the idea of piggybacking the test_plan_advice run onto
another run that we're already doing has any potential? That would
reduce the incremental cost quite a lot, I think.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  158. Re: pg_plan_advice

    Tom Lane <tgl@sss.pgh.pa.us> — 2026-03-19T22:43:48Z

    Robert Haas <robertmhaas@gmail.com> writes:
> Do you think the idea of piggybacking the test_plan_advice run onto
> another run that we're already doing has any potential? That would
> reduce the incremental cost quite a lot, I think.

It would, but it's conceptually ugly and it might make it much harder
to detangle the cause of a failure, so I don't care for it much.

I don't have any great ideas here.  Your point about the test having
helped to find a lot of bugs is compelling, and so is the fact that
it's seemingly exposing more issues we've not understood yet.
Maybe we can eventually buy back the cycles by not running it by
default, but clearly now is not the time for that.

			regards, tom lane

  159. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-03-19T23:11:18Z

    On Thu, Mar 19, 2026 at 6:43 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> It would, but it's conceptually ugly and it might make it much harder
> to detangle the cause of a failure, so I don't care for it much.

It does carry that risk. *Typically* failures are going to be a
WARNING message complaining about something related to advice, so the
chance of confusion is perhaps not as high as it would be in some
other cases -- but the grison failure is a counterexample. I'm
somewhat inclined to discount that particular counterexample because
the bug is entirely unrelated to test_plan_advice or pg_plan_advice,
so I am not sure it really would have mattered if we hadn't known that
test_plan_advice was what precipitated it. But there might be other
cases where that isn't so.

> I don't have any great ideas here.  Your point about the test having
> helped to find a lot of bugs is compelling, and so is the fact that
> it's seemingly exposing more issues we've not understood yet.
> Maybe we can eventually buy back the cycles by not running it by
> default, but clearly now is not the time for that.

OK, thanks. To be honest, my biggest fear here is not that the test
doesn't have enough value, but that it has a little too much value,
i.e. that we're going to find that future planner improvements require
pg_plan_advice adjustments more often than we're all comfortable with.
Hopefully that fear is unjustified, but we're not going to know for a
while.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  160. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-03-20T16:06:40Z

    On Thu, Mar 19, 2026 at 1:02 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> FWIW, I'm inclined to think that people won't actually have this
> turned on unless they want to see that output, and then they'll
> probably want to see all of it.  So I'm inclined to leave it as-is
> for now.  If field experience teaches differently, we can always
> improve it later.

Thanks for chiming in. I'll leave it as-is for the time being. My
guess is that this will be fairly low on the list of things people
complain about, but I'll be very happy if proven wrong, since this one
is easily changed.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  161. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-03-21T13:13:59Z

    On Thu, Mar 19, 2026 at 4:38 PM Robert Haas <robertmhaas@gmail.com> wrote:
> schnauzer's got a failure that looks like this:
>
> +WARNING:  supplied plan advice was not enforced
> +DETAIL:  advice SEQ_SCAN(pg_trigger@exists_1) feedback is "matched, failed"
> +advice NO_GATHER(pg_trigger@exists_1) feedback is "matched, failed"
>
> And an older run on skink has a failure that looks like this:
>
> +WARNING:  supplied plan advice was not enforced
> +DETAIL:  advice SEQ_SCAN(pg_trigger@exists_6) feedback is "matched, failed"
> +advice NO_GATHER(pg_trigger@exists_6) feedback is "matched, failed"

I spent a bunch of time looking into this. I don't have a definitive
answer to the question of what to do about it yet, but I wanted to
write down some initial thoughts.

First, I think that I broke fix_alternative_subplan when I added
disabled_nodes. Before, when disabling things just affected the cost,
the logic here would have taken what was disabled into account in
picking between alternatives. Now it doesn't, because I didn't add a
disabled_nodes field to SubPlan. You can see that it breaking with
this test case:

CREATE TABLE t1 (a int);
CREATE TABLE t2 (a int);
CREATE INDEX ON t2(a);
INSERT INTO t1 SELECT generate_series(1, 1000);
INSERT INTO t2 SELECT generate_series(1, 100000);
ANALYZE;

EXPLAIN (VERBOSE, COSTS ON)
SELECT * FROM t1
WHERE EXISTS (SELECT 1 FROM t2 WHERE t2.a = t1.a) OR t1.a < 0;

SET enable_seqscan = off;
SET enable_indexonlyscan = off;

EXPLAIN (VERBOSE, COSTS ON)
SELECT * FROM t1
WHERE EXISTS (SELECT 1 FROM t2 WHERE t2.a = t1.a) OR t1.a < 0;

With the currently-committed code, you get a hashed SubPlan both
times, and it's just disabled in the second case. But there's a
perfectly good non-hashed variant that is not disabled: scan t2 using
a parameterized Index Scan. So that sucks. I have a small patch to fix
this, which I'll post later. I don't think we can or should do
anything about this in released branches, but we should fix it in
master regardless of what happens to pg_plan_advice or
test_plan_advice.

Second, in terms of actually fixing the problem, I think the issue
here is that the scope I chose for pg_plan_advice doesn't quite fit
the goal of making test_plan_advice the canonical way of testing this
code. In order to keep this project manageable in size, I decided
that, for the first version, pg_plan_advice wasn't going to try to
control anything above the level of scan/join planning, so for example
we don't care what kind of aggregation the planner chooses. That
should be OK for test_plan_advice, because test_plan_advice works by
checking if all the advice applied cleanly, and since we didn't emit
any advice about the aggregation method in the first place, there
can't be any problem with applying it later. In other words, if the
aggregation method chosen does flip between consecutive planning
cycles, it should not cause a test_plan_advice test failure. This same
general principle applies to a bunch of other cases too, like set
operation planning: if there's more than one way to do it, the planner
can change its mind and test_plan_advice should not care. However,
there's an important exception: if something changes above the level
of scan/join planning that affects what rels are involved in scan/join
planning, then a plan change will cause a test_plan_advice failure.

The failures above are of that type: the way the AlternativeSubPlan
machinery works is that the query gets copied and replanned, and each
plan has a separate plan name. So the final plan has either
pg_trigger@exists_5 or pg_trigger@exists_6 in it, but never both.
There's one other case that I think is similar, which is the
MinMaxAggPath stuff: if we choose the MinMaxAggPath, then the final
plan will have something like t1@minmax_1 where it otherwise would
have had just t1, or perhaps t1@minmax_1 instead of t1@something_else.
These cases have something in common, which is that they are the only
cases where we make a new PlannerInfo to try replanning part of the
query in a second way. That's the pattern that causes breakage here. I
would be remiss if I did not mention that Jakub Wartak was poking at
me about the MinMaxAggPath case a while back, but I dismissed it as
out of scope, which was accurate, but that's because I didn't think at
the time that it would destabilize test_plan_advice. Now, I think it
could, although I don't think we have seen any such failures in the
buildfarm yet. Perhaps a concurrent statistics change is more likely
to flip the hashed/non-hashed SubPlan decision than the choice of
whether to use MinMaxAggPath.

One approach that I considered here is to try to unify the "sibling"
relations. If the final plan is bound to contain either
pg_trigger@exists_5 or pg_trigger@exists_6, maybe the advice shouldn't
actually think there are two separate things. Maybe instead of
subqueries exists_1 ... exists_6 we should end up with subqueries
exists_1 ... exists_3, with each name used twice. That amounts to
deciding that the patch to give each subplan a unique name got it
wrong. While this idea has some appeal, ultimately I think it's a
loser, because addressing the problem this way wouldn't actually fix
the test_plan_advice instability we're currently seeing, or at least
not necessarily. For example, in the test case earlier in this email,
INDEX_SCAN(t2@whatever) can only be applied if the non-hashed subplan
is chosen, because we won't consider a plan index scan to even be a
possibility for a full table scan. An index-only scan is considered,
but not a plain index scan. This means that even if we flattened the
rels in each pair of siblings together and called each pair by the
same name, we could still see failures to apply advice cleanly.
Moreover, that's assuming that optimizations like self-join
elimination, left join removal, and partition pruning always apply in
the same way to both copies, which doesn't sound like a safe
assumption at all.

A second approach is to try to control the initial conditions of the
two planning cycles better. Possibly just running the tests serially
instead of in parallel would get that done, but that seems too slow to
consider and, even if we did it anyway, I'm not all that sure that an
autoanalyze or autovacuum in the background couldn't mess us up. Or,
alternatively, if we could keep the second planning cycle from seeing
different statistics than the first, I think that would do it, but I
don't think there's a feasible way of doing that.

So I'm left with the idea that to get test_plan_advice to be fully
stable on these slower machines, it will probably be necessary to make
it control which AlternativeSubPlan is chosen and whether a
MinMaxAggPath is chosen or not. I have some ideas about how to
accomplish that in a reasonably elegant fashion without adding too
much new machinery, but I need to spend some more time validating
those ideas before committing to a precise course of action. More
soon.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  162. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-03-24T20:47:12Z

    On Thu, Mar 19, 2026 at 4:38 PM Robert Haas <robertmhaas@gmail.com> wrote:
> skink has a failure that looks like this:
>
> +WARNING:  supplied plan advice was not enforced
> +DETAIL:  advice NESTED_LOOP_MEMOIZE(nt) feedback is "matched, failed"
>
> I think this is caused by a minor bug in the pgs_mask infrastructure.
> get_memoize_path() exits quickly when outer_path->parent->rows < 2, on
> the theory that the resulting path must lose on cost. But that
> presumes that we could do a plain nested loop instead, i.e. that
> PGS_NESTLOOP_PLAIN is set. And it might not be. Before the pgs_mask
> stuff, that case couldn't happen: enable_nestloop=off disabled all
> nested loops, and enable_memoize=off disabled only the memoized
> version, but there wasn't any way to disable only the non-memoized
> version (which, of course, was part of the point of the whole thing).
> I think the fix is as attached.

The new test in that version was exactly backwards. I have corrected
that issue and committed this.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  163. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-03-24T21:09:51Z

    On Sat, Mar 21, 2026 at 9:13 AM Robert Haas <robertmhaas@gmail.com> wrote:
> So I'm left with the idea that to get test_plan_advice to be fully
> stable on these slower machines, it will probably be necessary to make
> it control which AlternativeSubPlan is chosen and whether a
> MinMaxAggPath is chosen or not. I have some ideas about how to
> accomplish that in a reasonably elegant fashion without adding too
> much new machinery, but I need to spend some more time validating
> those ideas before committing to a precise course of action. More
> soon.

Here is v22. There are four new patches.

0001 adds a disabled_nodes fields to SubPlan, to fix the bug that I
identified in the email to which this is a reply.

0002-0004 are an attempt to fix the remaining buildfarm failures not
already addressed (or attempted to be addressed, anyway) by other
commits. The basic idea, implemented by 0004, is to add a
DO_NOT_SCAN() advice tag. This advice is generated when we consider a
MinMaxAggPath or a hashed SubPlan. In either case, all relations which
are part of the non-selected alternative are marked DO_NOT_SCAN(),
which works like scan type advice but disables every possible scan
type rather than still allowing exactly one of them. Unless I've
missed something, this should be sufficient to make pg_plan_advice
stabilize which of two alternative SubPlans we pick and whether or not
a min/max aggregate is chosen. 0002 does some preliminary refactoring
to provide a more centralized way of tracking per-PlannerInfo details
within pg_plan_advice. 0003 makes the necessary change to
src/backend/optimizer, which consists of adding an alternative_root
field to each PlannerInfo and setting it appropriately. 0004 then
updates pg_plan_advice to implement DO_NOT_SCAN().

0005 is the pg_collect_advice module from previous versions of the
patch set. The main change here is that I completely rewrote the TAP
test, which previously was running the entire regression test suite
yet another time. That's been replaced with something that is much
faster and much better targeted at properly testing the shared advice
collector. Aside from that, I added one more check for
InvalidDsaPointer where the code was previously lacking one.

0006 is the pg_stash_advice module from previous versions of the patch
set. I have adjusted this to be much safer against permanent DSA
leaks. It now uses dshash_find_or_insert_extended instead of relying
on the ability to dshash_find a just-inserted entry without error. It
now also holds an LWLock while inserting or updating an entry in the
dshash table, for reasons explained in the comments. On the other
hand, it no longer unnecessarily holds the LWLock in exclusive mode
when looking up advice strings for automatic application, which was a
rather silly mistake in the previous version. A few additional tests
have been added. Alphabetization in contrib/Makefile has been fixed.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  164. Re: pg_plan_advice

    Lukas Fittl <lukas@fittl.com> — 2026-03-25T23:59:06Z

    Hi Robert,

On Tue, Mar 24, 2026 at 2:10 PM Robert Haas <robertmhaas@gmail.com> wrote:
>
> On Sat, Mar 21, 2026 at 9:13 AM Robert Haas <robertmhaas@gmail.com> wrote:
> > So I'm left with the idea that to get test_plan_advice to be fully
> > stable on these slower machines, it will probably be necessary to make
> > it control which AlternativeSubPlan is chosen and whether a
> > MinMaxAggPath is chosen or not. I have some ideas about how to
> > accomplish that in a reasonably elegant fashion without adding too
> > much new machinery, but I need to spend some more time validating
> > those ideas before committing to a precise course of action. More
> > soon.
>
> Here is v22. There are four new patches.
>
> 0001 adds a disabled_nodes fields to SubPlan, to fix the bug that I
> identified in the email to which this is a reply.

This looks good, and is consistent with how it would have worked
before the introduction of disabled nodes (since costs would have just
been very high and thus discourage a subplan with many disabled
nodes).

Only nit is that the commit hash reference in the commit message
doesn't seem right, I think you probably meant
e22253467942fdb100087787c3e1e3a8620c54b2

> 0002-0004 are an attempt to fix the remaining buildfarm failures not
> already addressed (or attempted to be addressed, anyway) by other
> commits. The basic idea, implemented by 0004, is to add a
> DO_NOT_SCAN() advice tag. This advice is generated when we consider a
> MinMaxAggPath or a hashed SubPlan. In either case, all relations which
> are part of the non-selected alternative are marked DO_NOT_SCAN(),
> which works like scan type advice but disables every possible scan
> type rather than still allowing exactly one of them. Unless I've
> missed something, this should be sufficient to make pg_plan_advice
> stabilize which of two alternative SubPlans we pick and whether or not
> a min/max aggregate is chosen. 0002 does some preliminary refactoring
> to provide a more centralized way of tracking per-PlannerInfo details
> within pg_plan_advice. 0003 makes the necessary change to
> src/backend/optimizer, which consists of adding an alternative_root
> field to each PlannerInfo and setting it appropriately. 0004 then
> updates pg_plan_advice to implement DO_NOT_SCAN().

For 0002:

I think that overall looks like a good refactoring, with two minor notes:

> diff --git a/contrib/pg_plan_advice/pgpa_planner.c b/contrib/pg_plan_advice/pgpa_planner.c
> index fee88904760..70139ff42be 100644
> --- a/contrib/pg_plan_advice/pgpa_planner.c
> +++ b/contrib/pg_plan_advice/pgpa_planner.c
> ...
> @@ -2017,34 +1949,64 @@ pgpa_planner_feedback_warning(List *feedback)
>                  errdetail("%s", detailbuf.data));
>  }
>
> -#ifdef USE_ASSERT_CHECKING
> -
>  /*
> - * Fast hash function for a key consisting of an RTI and plan name.
> + * Get or create the pgpa_planner_info for the subroot with the given
> + * plan_name.
>  */
> -static uint32
> -pgpa_ri_checker_hash_key(pgpa_ri_checker_key key)
> +static pgpa_planner_info *
> +pgpa_planner_get_proot(pgpa_planner_state *pps, PlannerInfo *root)
>  {

I'd word that as "Get or create the pgpa_planner_info for the given
PlannerInfo and its associated plan_name", since you're not passing a
plan_name as the argument.

> @@ -2053,19 +2015,34 @@ pgpa_ri_checker_save(pgpa_planner_state *pps, PlannerInfo *root,
>                      RelOptInfo *rel)
> {
> ...
> +   if (proot->rid_array_size <= rel->relid)
> +   {
> +           int                     new_size = Max(proot->rid_array_size, 8);
> +
> +           while (new_size < rel->relid)
> +                   new_size *= 2;

This could use pg_nextpower2_32 on the rel->relid instead of the
manual while loop.

---

For 0003:

I wonder if "original_root" wouldn't be more correct here as a name
(instead of "alternative_root"), since if I follow the implementation
correctly, you are adding a pointer on each alternative root, back to
the original root that the alternative was copied from.

I also wonder if maybe we should be more narrow in what we keep here.
It seems 0004 mainly needs the original plan name, so maybe its better
if we just keep that for targeting purposes, vs a full pointer to the
PlannerInfo. The planner makes an effort to zap unused subplans at the
end of set_plan_references, and I think this new field would then be
the only pointer to those unused subplans. If we decided to add an
early free there at some point (instead of just making them a NULL
entry in the list), that'd break pg_plan_advice.

---

For 0004:

> +    /*
> +     * If the corresponding PlannerInfo has an alternative_root, then this is
> +     * the plan name from that PlannerInfo; otherwise, it is the same as
> +     * plan_name.
> +     *
> +     * is_alternative_plan is set to true for every pgpa_planner_info that
> +     * shares an alternative_plan_name with at least one other, and to false
> +     * otherwise.
> +     */
> +    char       *alternative_plan_name;
> +    bool        is_alternative_plan;

Per the earlier note, I think using "original_plan_name" would make
more sense here, because it'll be the name the alternatives are based
on. I also think "has_alternative_plan" is more clear for the boolean,
since it'll be set on the info for the original info as well, if I
understand correctly.

Otherwise 0004 looks like a reasonable compromise for now. I feel like
we can find better ways of doing this over time, and there are parts
I'm not excited about (e.g. the targeting feels a bit brittle when it
comes to anything that'd cause generated alternative subplan names to
change), but I think it works for now.

> 0005 is the pg_collect_advice module from previous versions of the
> patch set. The main change here is that I completely rewrote the TAP
> test, which previously was running the entire regression test suite
> yet another time. That's been replaced with something that is much
> faster and much better targeted at properly testing the shared advice
> collector. Aside from that, I added one more check for
> InvalidDsaPointer where the code was previously lacking one.

From doing an initial code review, I feel like the interface.c vs
collector.c split in this module is confusing - e.g. I would have
expected SQL callable functions like "pg_get_collected_shared_advice"
to be part of the interface. It also makes reading the code confusing,
e.g. I just looked for where pg_collect_advice_get_mcxt is defined,
and I had to jump from collector.c to interface.c - maybe its better
if that's just in one file, since it'd still be slightly under 1000
lines anyway?

From a design perspective, I'm -1 on storing of full query text
strings in shared memory when the shared collector mode. With large
query texts and without an aggregate MB size limit that's an
expressway into OOM land, even if you used a low value like 100
entries max, because ORMs are just really good at creating large query
texts unexpectedly. I'm also skeptical whether that's a good idea for
the local collection mode, but it'd be less problematic there.
Overall, I think this needs to rely on queryid instead and not store
query texts. I would not make queryid optional, but instead enable it
automatically - which fits together with pg_stash_advice taking it as
input.

I realize not having query texts reduces its effectiveness (since you
don't see which parameters produced which plan advice), but it still
helps surface which different advice strings where seen for which
query IDs, letting you identify if you're getting a mix of bad and
good plans. And I'm just really worried people will enable this on
production in shared collection mode and take down their system.

> 0006 is the pg_stash_advice module from previous versions of the patch
> set. I have adjusted this to be much safer against permanent DSA
> leaks. It now uses dshash_find_or_insert_extended instead of relying
> on the ability to dshash_find a just-inserted entry without error. It
> now also holds an LWLock while inserting or updating an entry in the
> dshash table, for reasons explained in the comments. On the other
> hand, it no longer unnecessarily holds the LWLock in exclusive mode
> when looking up advice strings for automatic application, which was a
> rather silly mistake in the previous version. A few additional tests
> have been added. Alphabetization in contrib/Makefile has been fixed.

From a design perspective, I'm worried about the fact that we lose the
stashed advice on a restart. e.g. imagine a DBA using pg_stash_advice
to pin a query that sometimes picks a bad plan to the good plan, but
then their cloud provider applies a security update overnight.
Suddenly the database is slow because the bad plans are being picked
again.

pg_hint_plan's solution to this (the "hints" table [0]) uses an actual
table managed by the extension - but I suspect that doesn't fit the
picture, since it'd be per database, etc. It does have the benefit of
being restart safe though, and being copied to replicas.

I wonder if we could find a way to dump and restore the advice stash
information via a file, so its at least crash and restart safe?

Thanks,
Lukas

[0]: https://github.com/ossc-db/pg_hint_plan/blob/master/docs/hint_table.md

-- 
Lukas Fittl

  165. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-03-26T13:55:53Z

    On Wed, Mar 25, 2026 at 7:59 PM Lukas Fittl <lukas@fittl.com> wrote:
> > 0001 adds a disabled_nodes fields to SubPlan, to fix the bug that I
> > identified in the email to which this is a reply.
>
> This looks good, and is consistent with how it would have worked
> before the introduction of disabled nodes (since costs would have just
> been very high and thus discourage a subplan with many disabled
> nodes).
>
> Only nit is that the commit hash reference in the commit message
> doesn't seem right, I think you probably meant
> e22253467942fdb100087787c3e1e3a8620c54b2

Whoops. Obviously got the wrong thing stuck in my cut and paste buffer
when I was writing that. Thanks for checking it. I'm going to go ahead
and commit this, because I'm pretty confident that it's correct, and
the rest of these patches are not going to fix the buildfarm
instability without it, and I'm pretty sure multiple committers are
pretty tired of seeing these test_plan_advice failures already.

> For 0002:
>
> I think that overall looks like a good refactoring, with two minor notes:
>
> I'd word that as "Get or create the pgpa_planner_info for the given
> PlannerInfo and its associated plan_name", since you're not passing a
> plan_name as the argument.

Right, the comment isn't quite correct. I don't think your rewording
is quite right either, though, because there's really no reason to
mention plan_name here at all. I'll adjust it.

> > @@ -2053,19 +2015,34 @@ pgpa_ri_checker_save(pgpa_planner_state *pps, PlannerInfo *root,
> >                      RelOptInfo *rel)
> > {
> > ...
> > +   if (proot->rid_array_size <= rel->relid)
> > +   {
> > +           int                     new_size = Max(proot->rid_array_size, 8);
> > +
> > +           while (new_size < rel->relid)
> > +                   new_size *= 2;
>
> This could use pg_nextpower2_32 on the rel->relid instead of the
> manual while loop.

OK, will change that also, and then also commit this one.

> For 0003:
>
> I wonder if "original_root" wouldn't be more correct here as a name
> (instead of "alternative_root"), since if I follow the implementation
> correctly, you are adding a pointer on each alternative root, back to
> the original root that the alternative was copied from.

To me, that seems less clear. Someone might think that the original
root means the toplevel PlannerInfo, or that whatever PlannerInfo we
had around when we created the current PlannerInfo will be the
original_root. But in fact we are only using this in a much more
narrow situation, namely, when we're creating a new PlannerInfo as a
way to consider an alternative implementation of the same portion of
the query. That is, I think alternative conveys a sibling
relationship, and original doesn't necessarily do so.

> I also wonder if maybe we should be more narrow in what we keep here.
> It seems 0004 mainly needs the original plan name, so maybe its better
> if we just keep that for targeting purposes, vs a full pointer to the
> PlannerInfo. The planner makes an effort to zap unused subplans at the
> end of set_plan_references, and I think this new field would then be
> the only pointer to those unused subplans. If we decided to add an
> early free there at some point (instead of just making them a NULL
> entry in the list), that'd break pg_plan_advice.

The dangling pointers are a good point; I agree that's bad. However,
I'd be more inclined to fix it by nulling out the alternative_root
pointers at the end of set_plan_references. I think that would just be
the case where root->isAltSubplan[ndx] && root->isUsedSubplan[ndx].
The reason I'm reluctant to just store the name is that there's not an
easy way to find a PlannerInfo by name. I originally proposed an
"allroots" list in PlannerGlobal, but we went with subplanNames on
Tom's suggestion. I subsequently realized that this kind of stinks for
code that is trying to use this infrastructure for anything, for
exactly this reason, but Tom never responded and I never pressed the
issue. But I think we're boxing ourselves into a corner if we just
keep storing names that can't be looked up everywhere. It doesn't
matter for the issue before us, so maybe doing as you say here is the
right idea just so we can move forward, but I think we're probably
kidding ourselves a little bit.

> From a design perspective, I'm -1 on storing of full query text
> strings in shared memory when the shared collector mode. With large
> query texts and without an aggregate MB size limit that's an
> expressway into OOM land, even if you used a low value like 100
> entries max, because ORMs are just really good at creating large query
> texts unexpectedly. I'm also skeptical whether that's a good idea for
> the local collection mode, but it'd be less problematic there.
> Overall, I think this needs to rely on queryid instead and not store
> query texts. I would not make queryid optional, but instead enable it
> automatically - which fits together with pg_stash_advice taking it as
> input.
>
> I realize not having query texts reduces its effectiveness (since you
> don't see which parameters produced which plan advice), but it still
> helps surface which different advice strings where seen for which
> query IDs, letting you identify if you're getting a mix of bad and
> good plans. And I'm just really worried people will enable this on
> production in shared collection mode and take down their system.

I fully admit that pg_collect_advice is crude, but I don't think
ripping out some portion of the limited functionality that it has is
going to get us where we want to be. If it hadn't collected the query
strings, it would have been useless for the purpose for which I
originally wrote it. We could add a GUC for a length limit, perhaps,
but I think the real feature that this needs to be used in the way
that you seem to want to use it is deduplication, and as I said
earlier, I think we should consider adding the advice collection logic
to pg_stat_statements rather than building an alternative version of
that module with overlapping functionality. If you think this is a
sufficiently large foot-gun that we shouldn't ship it, then I'll just
drop this patch for this cycle and we can revisit what to do for next
cycle. It's not critical, and with more time we can talk through
possible approaches and have time to code something up in a
responsible way. There's just no time to make big design changes right
now. If some small adjustment (like adding a GUC to limit the length
of the query string we're willing to collect) elevates it to
acceptability then I'm happy to do that, but otherwise we should just
revisit the topic for next cycle.

> From a design perspective, I'm worried about the fact that we lose the
> stashed advice on a restart. e.g. imagine a DBA using pg_stash_advice
> to pin a query that sometimes picks a bad plan to the good plan, but
> then their cloud provider applies a security update overnight.
> Suddenly the database is slow because the bad plans are being picked
> again.
>
> pg_hint_plan's solution to this (the "hints" table [0]) uses an actual
> table managed by the extension - but I suspect that doesn't fit the
> picture, since it'd be per database, etc. It does have the benefit of
> being restart safe though, and being copied to replicas.
>
> I wonder if we could find a way to dump and restore the advice stash
> information via a file, so its at least crash and restart safe?

I think if we want to ship this extension in this release, then the
only alternative is to tell users they have to put in place a manual
process for this. That is not great, but the original version of
pg_prewarm had the exact same issue, and I don't think anyone thought
that made it useless. Indeed, pginfcore still has nothing comparable
to autoprewarm, and I'm not saying that as a way of throwing shade on
pgfincore.

What I'm concerned about with this module is completely different: I'm
wondering whether the approach it takes to using DSA is OK, whether
the security model is adequate, and that sort of thing. Expanding the
scope is 100% off the table in my book. Feature freeze is in less than
two weeks.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  166. Re: pg_plan_advice

    Matheus Alcantara <matheusssilv97@gmail.com> — 2026-03-26T14:30:40Z

    On 26/03/26 10:55, Robert Haas wrote:
>> I realize not having query texts reduces its effectiveness (since you
>> don't see which parameters produced which plan advice), but it still
>> helps surface which different advice strings where seen for which
>> query IDs, letting you identify if you're getting a mix of bad and
>> good plans. And I'm just really worried people will enable this on
>> production in shared collection mode and take down their system.
> 
> I fully admit that pg_collect_advice is crude, but I don't think
> ripping out some portion of the limited functionality that it has is
> going to get us where we want to be. If it hadn't collected the query
> strings, it would have been useless for the purpose for which I
> originally wrote it. We could add a GUC for a length limit, perhaps,
> but I think the real feature that this needs to be used in the way
> that you seem to want to use it is deduplication, and as I said
> earlier, I think we should consider adding the advice collection logic
> to pg_stat_statements rather than building an alternative version of
> that module with overlapping functionality.
>

I also think that we should consider adding the advice string on 
pg_stat_statements. It seems to make more sense to me IMHO.

Adding support for auto_explain to explain(plan_advice, ...) (or any 
other custom explain option from loadable modules) would help or make 
sense here? I have been thinking about this for a while.


--
Matheus Alcantara
EDB: https://www.enterprisedb.com

  167. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-03-26T14:37:15Z

    On Thu, Mar 26, 2026 at 10:30 AM Matheus Alcantara
<matheusssilv97@gmail.com> wrote:
> Adding support for auto_explain to explain(plan_advice, ...) (or any
> other custom explain option from loadable modules) would help or make
> sense here? I have been thinking about this for a while.

I think that some generic support for custom explain options in
auto_explain is a good idea, but if you use that method to collect
advice strings, you're going to have quite a bit of log-filtering work
to do to get anything useful out of it. That might be fine for some
people, and it's certainly better than nothing, but I think eventually
we want a cleaner way. But still, +many for upgrading auto_explain
with this feature.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  168. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-03-26T17:20:21Z

    On Thu, Mar 26, 2026 at 9:55 AM Robert Haas <robertmhaas@gmail.com> wrote:
> Whoops. Obviously got the wrong thing stuck in my cut and paste buffer
> when I was writing that. Thanks for checking it. I'm going to go ahead
> and commit this, because I'm pretty confident that it's correct, and
> the rest of these patches are not going to fix the buildfarm
> instability without it, and I'm pretty sure multiple committers are
> pretty tired of seeing these test_plan_advice failures already.

Done.

> Right, the comment isn't quite correct. I don't think your rewording
> is quite right either, though, because there's really no reason to
> mention plan_name here at all. I'll adjust it.

Done and committed, after also adjusting the memory context handling
to avoid re-breaking GEQO.

> The dangling pointers are a good point; I agree that's bad. However,
> I'd be more inclined to fix it by nulling out the alternative_root
> pointers at the end of set_plan_references. I think that would just be
> the case where root->isAltSubplan[ndx] && root->isUsedSubplan[ndx].
> The reason I'm reluctant to just store the name is that there's not an
> easy way to find a PlannerInfo by name. I originally proposed an
> "allroots" list in PlannerGlobal, but we went with subplanNames on
> Tom's suggestion. I subsequently realized that this kind of stinks for
> code that is trying to use this infrastructure for anything, for
> exactly this reason, but Tom never responded and I never pressed the
> issue. But I think we're boxing ourselves into a corner if we just
> keep storing names that can't be looked up everywhere. It doesn't
> matter for the issue before us, so maybe doing as you say here is the
> right idea just so we can move forward, but I think we're probably
> kidding ourselves a little bit.

Here's a new version, where I've replaced alternative_root by
alternative_plan_name, serving the same function.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  169. Re: pg_plan_advice

    Lukas Fittl <lukas@fittl.com> — 2026-03-26T19:51:29Z

    Hi Robert,

On Thu, Mar 26, 2026 at 10:20 AM Robert Haas <robertmhaas@gmail.com> wrote:
> > The dangling pointers are a good point; I agree that's bad. However,
> > I'd be more inclined to fix it by nulling out the alternative_root
> > pointers at the end of set_plan_references. I think that would just be
> > the case where root->isAltSubplan[ndx] && root->isUsedSubplan[ndx].
> > The reason I'm reluctant to just store the name is that there's not an
> > easy way to find a PlannerInfo by name. I originally proposed an
> > "allroots" list in PlannerGlobal, but we went with subplanNames on
> > Tom's suggestion. I subsequently realized that this kind of stinks for
> > code that is trying to use this infrastructure for anything, for
> > exactly this reason, but Tom never responded and I never pressed the
> > issue. But I think we're boxing ourselves into a corner if we just
> > keep storing names that can't be looked up everywhere. It doesn't
> > matter for the issue before us, so maybe doing as you say here is the
> > right idea just so we can move forward, but I think we're probably
> > kidding ourselves a little bit.
>
> Here's a new version, where I've replaced alternative_root by
> alternative_plan_name, serving the same function.

Great, I think that's better for now, and if we have a broader use
case in the future we can always adjust this to be the full
PlannerInfo.

That said, reflecting on the change, I wonder if its odd that we're
copying a string pointer instead of making an actual string copy. I
think that's probably okay in practice?

I'm still 50/50 on the naming here, since we have the alternative sub
plan that has an "alternative plan name" that's not that of the
alternative itself, but rather the base plan that was utilized. But I
see your concern regarding the naming being confusing in terms of what
the "original" or "base" would actually refer to. I've also considered
whether something like "alternative_plan_group" could make sense
(since all alternative sub plans will have the same value), but maybe
that conveys too much intent on what this is used for.

That said, I think for now, to get the buildfarm happy again, v23/0001
seems good.

v23/0002 also looks good.

Thanks,
Lukas

--
Lukas Fittl

  170. Re: pg_plan_advice

    Mark Dilger <mark.dilger@enterprisedb.com> — 2026-03-26T23:20:49Z

    Hi Robert,

Thanks for the pg_plan_advice patches!  I've tried hard to attack them, to
get
them to fail in some catastrophic way.  You seem to have hardened the code
well.  I found only one concern for you to consider, a kind of memory leak
ratchet:

Once the system reaches memory pressure where:
  - The 8192-byte DSA size class is exhausted (needs a new DSM segment, OS
refuses)
  - Smaller size classes still have free space in existing superblocks

Then every single query that triggers advice collection will:
  1. Successfully allocate an advice entry from existing free space
  2. Enter store_shared_advice, hit the same chunk boundary
  3. Fail to allocate the chunk
  4. Leak the advice entry
  5. Reduce remaining free space in the small size classes

This continues until the small size classes are also exhausted, at which
point
make_collected_advice itself fails (the DSA area has been consumed by leaked
entries). The ratchet is self-reinforcing: each failure guarantees the next
failure while consuming more resources, assuming nobody else is freeing
memory simultaneously.

I looked for situations where something inside postgres would keep retrying
after the OOM, but the most likely I think is just an application that
treats
OOM as a transient error and keeps retrying.

See the make_collected_advice() call in pg_collect_advice_save(); the point
where DSA memory is allocated but not yet linked into any data structure.
Everything downstream from here (the four dsa_allocate0 calls inside
store_shared_advice) can fail and leak it.

On Thu, Mar 26, 2026 at 10:20 AM Robert Haas <robertmhaas@gmail.com> wrote:

> On Thu, Mar 26, 2026 at 9:55 AM Robert Haas <robertmhaas@gmail.com> wrote:
> > Whoops. Obviously got the wrong thing stuck in my cut and paste buffer
> > when I was writing that. Thanks for checking it. I'm going to go ahead
> > and commit this, because I'm pretty confident that it's correct, and
> > the rest of these patches are not going to fix the buildfarm
> > instability without it, and I'm pretty sure multiple committers are
> > pretty tired of seeing these test_plan_advice failures already.
>
> Done.
>
> > Right, the comment isn't quite correct. I don't think your rewording
> > is quite right either, though, because there's really no reason to
> > mention plan_name here at all. I'll adjust it.
>
> Done and committed, after also adjusting the memory context handling
> to avoid re-breaking GEQO.
>
> > The dangling pointers are a good point; I agree that's bad. However,
> > I'd be more inclined to fix it by nulling out the alternative_root
> > pointers at the end of set_plan_references. I think that would just be
> > the case where root->isAltSubplan[ndx] && root->isUsedSubplan[ndx].
> > The reason I'm reluctant to just store the name is that there's not an
> > easy way to find a PlannerInfo by name. I originally proposed an
> > "allroots" list in PlannerGlobal, but we went with subplanNames on
> > Tom's suggestion. I subsequently realized that this kind of stinks for
> > code that is trying to use this infrastructure for anything, for
> > exactly this reason, but Tom never responded and I never pressed the
> > issue. But I think we're boxing ourselves into a corner if we just
> > keep storing names that can't be looked up everywhere. It doesn't
> > matter for the issue before us, so maybe doing as you say here is the
> > right idea just so we can move forward, but I think we're probably
> > kidding ourselves a little bit.
>
> Here's a new version, where I've replaced alternative_root by
> alternative_plan_name, serving the same function.
>
> --
> Robert Haas
> EDB: http://www.enterprisedb.com
>


-- 

*Mark Dilger*

  171. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-03-26T23:25:57Z

    On Thu, Mar 26, 2026 at 3:52 PM Lukas Fittl <lukas@fittl.com> wrote:
> That said, reflecting on the change, I wonder if its odd that we're
> copying a string pointer instead of making an actual string copy. I
> think that's probably okay in practice?

Normally, all of planning happens in the same memory context. Under
GEQO, joins are planned in shorter-lived contexts that are reset, but
I don't think a new PlannerInfo can get created in one of those
short-lived contexts. At any rate, there's no point in having two
copies of the same string in the same memory context.

> I'm still 50/50 on the naming here, since we have the alternative sub
> plan that has an "alternative plan name" that's not that of the
> alternative itself, but rather the base plan that was utilized. But I
> see your concern regarding the naming being confusing in terms of what
> the "original" or "base" would actually refer to. I've also considered
> whether something like "alternative_plan_group" could make sense
> (since all alternative sub plans will have the same value), but maybe
> that conveys too much intent on what this is used for.

Let's go with this for now, and if a consensus emerges that I got it
wrong, we can always change it.

> That said, I think for now, to get the buildfarm happy again, v23/0001
> seems good.
>
> v23/0002 also looks good.

Thanks. Committed. Nothing has obviously broken so far, BUT even
machines like skink that were failing weren't failing on every run, so
it may be a while before we get a clear view of the situation --
unless of course this didn't fix it or even made things worse, in
which case we might find out a lot faster. Hopefully not, but then I
thought this was going to work the first time.

In the meanwhile, we should try to make a decision on what to do about
pg_collect_advice and pg_stash_advice.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  172. Re: pg_plan_advice

    Alexander Lakhin <exclusion@gmail.com> — 2026-03-27T06:00:00Z

    Hello Robert.

27.03.2026 01:25, Robert Haas wrote:
> Thanks. Committed. Nothing has obviously broken so far, BUT even
> machines like skink that were failing weren't failing on every run, so
> it may be a while before we get a clear view of the situation --
> unless of course this didn't fix it or even made things worse, in
> which case we might find out a lot faster. Hopefully not, but then I
> thought this was going to work the first time.

I could not reproduce recent failures from skink and morepork so far, but
I found that changing some GUCs can trigger similar warnings, e.g.:
echo "join_collapse_limit = 1000" >/tmp/extra.config
TEMP_CONFIG=/tmp/extra.config make -s check -C src/test/modules/test_plan_advice/
--- .../src/test/regress/expected/identity.out       2026-02-13 06:15:55.871368851 +0200
+++ .../src/test/modules/test_plan_advice/tmp_check/results/identity.out 2026-03-27 06:50:11.123234266 +0200
@@ -10,6 +10,8 @@
  ALTER TABLE itest3 ALTER COLUMN a ADD GENERATED ALWAYS AS IDENTITY;  -- error
  ERROR:  column "a" of relation "itest3" is already an identity column
  SELECT table_name, column_name, column_default, is_nullable, is_identity, identity_generation, identity_start, 
identity_increment, identity_maximum, identity_minimum, identity_cycle FROM information_schema.columns WHERE table_name 
LIKE 'itest_' ORDER BY 1, 2;
+WARNING:  supplied plan advice was not enforced
+DETAIL:  advice JOIN_ORDER(nc (co nco (c a)) t (bt nbt) ad nt (dep seq)) feedback is "matched, failed"
...

echo "geqo_threshold = 8" >/tmp/extra.config
TEMP_CONFIG=/tmp/extra.config make -s check -C src/test/modules/test_plan_advice/
--- .../src/test/regress/expected/create_function_sql.out 2026-02-13 06:15:55.865368936 +0200
+++ .../src/test/modules/test_plan_advice/tmp_check/results/create_function_sql.out 2026-03-27 06:53:44.235504942 +0200
@@ -482,6 +482,9 @@
      FROM information_schema.parameters JOIN information_schema.routines USING (specific_schema, specific_name)
      WHERE routine_schema = 'temp_func_test' AND routine_name ~ '^functest_is_'
      ORDER BY 1, 2;
+WARNING:  supplied plan advice was not enforced
+DETAIL:  advice JOIN_ORDER(n (ss (p (t#2 nt#2) l)) t nt) feedback is "matched, failed"
+advice NESTED_LOOP_PLAIN((ss p l t#2 nt#2)) feedback is "matched, failed"
   routine_name  | ordinal_position | parameter_name | parameter_default
...

and an assertion failure with:
enable_parallel_append = off
enable_partitionwise_aggregate = on
cpu_tuple_cost = 1000

TRAP: failed Assert("relids != NULL"), File: "pgpa_scan.c", Line: 248, PID: 1956762
...
2026-03-27 07:51:02.562 EET postmaster[1956525] LOG:  client backend (PID 1956762) was terminated by signal 6: Aborted
2026-03-27 07:51:02.562 EET postmaster[1956525] DETAIL:  Failed process was running: SELECT COUNT(*) FROM 
stats_import.part_parent;

Best regards,
Alexander

  173. Re: pg_plan_advice

    Jakub Wartak <jakub.wartak@enterprisedb.com> — 2026-03-27T08:00:03Z

    On Thu, Mar 26, 2026 at 6:20 PM Robert Haas <robertmhaas@gmail.com> wrote:
>[..v23]

0003: please be the judge here, as I'm not sure. Isn't there some too high
concurrency hit in pg_get_collected_shared_advice? If I do
pgbench -M extended -c 12 -j 12 -P 1 -S:

    progress: 59.0 s, 191008.4 tps, lat 0.063 ms stddev 0.200, 0 failed
    progress: 60.0 s, 197571.2 tps, lat 0.061 ms stddev 0.026, 0 failed
    progress: 61.0 s, 189825.5 tps, lat 0.063 ms stddev 0.208, 0 failed
    progress: 62.0 s, 197082.4 tps, lat 0.061 ms stddev 0.027, 0 failed
    progress: 63.0 s, 69345.9 tps, lat 0.173 ms stddev 1.651, 0 failed
    progress: 64.0 s, 47243.6 tps, lat 0.251 ms stddev 2.128, 0 failed
    progress: 65.0 s, 48211.6 tps, lat 0.247 ms stddev 2.156, 0 failed

there is visible collapse from 190k to 48k tps was due to constant flood
of artificial calls of: select count(*) from pg_get_collected_shared_advice();

The code does LW_SHARED there over potentially lots of of tuplestore_putvalues()
calls. However any other backend does pgca_planner_shutdown()->
pg_collect_advice_save()->store_shared_advice() which is trying to grab
LW_EXCLUSIVE lock, so everything might be be blocked across whole cluster? (I
mean for the duration of tuplestore entry and that seems to even talk about
"tape"/"disk", so to me it looks like prolonged I/O operations for temp might
impact CPU-only planning stuff?)

Maybe it is possible to buffer those reads under LW_SHARED into
backend-only (private)
memory and later just fill tuplestore later to avoid such hazard? (but the
obvious problem is how much memory we can have and how big shared area can
become). Or maybe after some time simply release it and sleep and re-take it?

0004: question, why in the pg_get_advice_stashes() the second call to
dshash_seq_init() nearby "Emit results" is done with exclusive=true , but
apparently only reads it?

-J.

  174. Re: pg_plan_advice

    Lukas Fittl <lukas@fittl.com> — 2026-03-27T08:31:11Z

    On Fri, Mar 27, 2026 at 1:00 AM Jakub Wartak
<jakub.wartak@enterprisedb.com> wrote:
>
> On Thu, Mar 26, 2026 at 6:20 PM Robert Haas <robertmhaas@gmail.com> wrote:
> >[..v23]
>
> 0003: please be the judge here, as I'm not sure. Isn't there some too high
> concurrency hit in pg_get_collected_shared_advice? If I do

I've been thinking more about 0003 (pg_collect_advice) today, and I'm
getting increasingly skeptical that we should try to get that into 19.
It just feels like there is more design work to be done here, and I
don't see the pressing need to have this in place.

Instead, I wonder if we should just add a "debug_log_plan_advice"
setting to pg_plan_advice, that always logs the plan advice when
enabled. Basically like "always_store_advice_details", but emit a log
line in addition to doing the work. That could either be enabled on a
single session with a sufficiently high client_min_messages to consume
it directly, or written to the log for a few minutes when trying to
capture production activity (on small production systems where the
logging overhead is acceptable).

I don't see a log-based approach be less useful than the shared memory
approach, because I think our aggregation design here is not right yet
(and doesn't scale to production traffic), and so we might as well
have the community try out some things with the log output instead.

For the other one 0004 (pg_stash_advice) I feel its worth trying to
get it in, if we can figure out the remaining issues. I'll try to do
another pass on that tomorrow after some sleep.

Thanks,
Lukas

-- 
Lukas Fittl

  175. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-03-27T12:55:41Z

    On Fri, Mar 27, 2026 at 2:00 AM Alexander Lakhin <exclusion@gmail.com> wrote:
> I could not reproduce recent failures from skink and morepork so far, but
> I found that changing some GUCs can trigger similar warnings, e.g.:

Thanks for the reports.

> echo "geqo_threshold = 8" >/tmp/extra.config

Failures here are expected. When planning is done via GEQO, not all
join orders are explored, and pg_plan_advice can only constrain the
join order from among the options considered by the planner. So, with
GEQO + test_plan_advice, any given test is going to pass if the second
round of planning considers the join order chosen by the first, and
fail otherwise.

This could be improved at some point in the future. For example,
somebody could add hooks to GEQO so that pg_plan_advice can cause it
to generate only candidates which are consistent with the supplied
advice. In practice, I'm not sure this is going to be a good use of
time. I suspect the energy would be better invested in improving GEQO
or coming up with a more useful replacement. The gap that exists here
doesn't mean that you can't use pg_plan_advice with GEQO; it only
means that you are going to have a bad time using them together if you
provide *complete* (or nearly-complete) plan advice.

> echo "join_collapse_limit = 1000" >/tmp/extra.config

The cause here actually seems to be GEQO once again. Raising the
join_collapse_limit causes some join problems to get bigger, which has
the result that they then use GEQO. At least for me, if I also bump up
geqo_threshold, the failures go away.

> and an assertion failure with:
> enable_parallel_append = off
> enable_partitionwise_aggregate = on
> cpu_tuple_cost = 1000
>
> TRAP: failed Assert("relids != NULL"), File: "pgpa_scan.c", Line: 248, PID: 1956762

Obviously, this one's a bug. I think the attached should fix it.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  176. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-03-27T13:08:30Z

    On Fri, Mar 27, 2026 at 4:00 AM Jakub Wartak
<jakub.wartak@enterprisedb.com> wrote:
> there is visible collapse from 190k to 48k tps was due to constant flood
> of artificial calls of: select count(*) from pg_get_collected_shared_advice();
>
> The code does LW_SHARED there over potentially lots of of tuplestore_putvalues()
> calls. However any other backend does pgca_planner_shutdown()->
> pg_collect_advice_save()->store_shared_advice() which is trying to grab
> LW_EXCLUSIVE lock, so everything might be be blocked across whole cluster? (I
> mean for the duration of tuplestore entry and that seems to even talk about
> "tape"/"disk", so to me it looks like prolonged I/O operations for temp might
> impact CPU-only planning stuff?)

Yeah ... I mean, I don't know what you want here.  If you fetch very
large quantities of data under a shared lock while concurrent activity
is trying to add data under an exclusive lock, that's going to be
slow. Now, as you say, there are ways to improve this. However, I
don't feel like running pg_get_collected_shared_advice() in a tight
loop is a normal use case. Normally you would turn it on, run a bunch
of queries, and then run that once at the end. Even that could hit
some issues because every session will be fighting to insert into the
hash table, but here you've made it much worse in a way that I would
say is artificial.

> 0004: question, why in the pg_get_advice_stashes() the second call to
> dshash_seq_init() nearby "Emit results" is done with exclusive=true , but
> apparently only reads it?

Good question. Actually, couldn't both of those loops use a shared lock only?

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  177. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-03-27T14:20:30Z

    On Thu, Mar 26, 2026 at 7:21 PM Mark Dilger
<mark.dilger@enterprisedb.com> wrote:
> Then every single query that triggers advice collection will:
>   1. Successfully allocate an advice entry from existing free space
>   2. Enter store_shared_advice, hit the same chunk boundary
>   3. Fail to allocate the chunk
>   4. Leak the advice entry
>   5. Reduce remaining free space in the small size classes

Yeah, that's a leak. I just got through trying to harden
pg_stash_advice so that kind of thing can't happen, but I failed to
realize that pg_collect_advice has a version of the same issue.

Thanks,

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  178. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-03-27T15:44:15Z

    On Fri, Mar 27, 2026 at 4:31 AM Lukas Fittl <lukas@fittl.com> wrote:
> I've been thinking more about 0003 (pg_collect_advice) today, and I'm
> getting increasingly skeptical that we should try to get that into 19.
> It just feels like there is more design work to be done here, and I
> don't see the pressing need to have this in place.
>
> Instead, I wonder if we should just add a "debug_log_plan_advice"
> setting to pg_plan_advice, that always logs the plan advice when
> enabled. Basically like "always_store_advice_details", but emit a log
> line in addition to doing the work. That could either be enabled on a
> single session with a sufficiently high client_min_messages to consume
> it directly, or written to the log for a few minutes when trying to
> capture production activity (on small production systems where the
> logging overhead is acceptable).

Sure, we could do something like that. It means that people have to do
log parsing to get the advice out cleanly, but it avoids the concerns
about memory utilization, and it's simple to code.

> For the other one 0004 (pg_stash_advice) I feel its worth trying to
> get it in, if we can figure out the remaining issues. I'll try to do
> another pass on that tomorrow after some sleep.

Let me just talk a little about the way that I see the space of
possible designs here. Let's set aside what we do or do not have time
to code for a moment and just think about what makes some kind of
sense in theory. I believe we can divide this up along a few different
axes:

1. Where is advice stored for on-the-fly retrieval? Possible answers
include: in shared memory, in local memory, in a table, on disk. But
realistically, I doubt that "in a table" is a realistic option. Even
if we hard-coded a direct index lookup i.e. without going through the
query planner, I think this would be a fairly expensive thing to do
for every query, and if this is going to be usable on production
systems, it has to be fast. I am absolutely certain that "on disk" is
not a realistic option. Local memory is an option. It has the
advantage of making server-lifespan memory leaks impossible, and the
further advantage of avoiding contention between different backends,
since each backend would have its own copy of the data. The big
disadvantage is memory consumption. If we think the advice store is
going to be small, then that might be fine, but somebody is
potentially going to have thousands of advice strings stored,
duplicating that across hundreds (or, gulp, thousands) of backends is
pretty bad.

2. What provision do we have for durability? Possible answers include:
in a table, on disk, nothing. I went with nothing, partly on the
theory that it gives users more flexibility. We don't really care
where they store their query IDs and advice strings, as long as they
have a way to feed them into the mechanism. But of course I was also
trying to save myself implementation complexity. There are some tricky
things about a table: as implemented, the advice stores are
cluster-wide objects, but tables are database objects. If we're
supposed to automatically load advice strings from a table that might
be in another database into an in-memory store, well, we can't. That
could be fixed by scoping stores to a specific database, which would
be inconvenient only for users who have the same schema in multiple
databases and would want to share stashes across DBs, which is
probably not common. A disk file is also an option. It requires
inventing some kind of custom format that we can generate and parse,
which has some complexity, but reading from a table has some
complexity too; I'm not sure which is messier.

3. How do we limit memory usage? One possible approach involves
limiting the size of the hash table by entries or by memory
consumption, but I don't think that's too valuable if that's all we
do, because presumably all that's going to do is annoy people who hit
the limit. It could make sense if we switched to a design where the
superuser creates the stash, assigns it a memory limit, and then
there's a way to give permission to use that stash to some particular
user who is bound by the memory limit. In that kind of design, the
person setting aside the memory has different and superior privileges
to the person using it, so the cap serves a real purpose. A
complicating factor here is that dshash doesn't seem to be well set up
to enforce memory limits. You can do it if each dshash uses a separate
DSA, but that's clunky, too. A completely different direction is to
treat the in-memory component of the system as a cache, backed by a
table or file from which cold entries are retrieved as needed. The
problem with this - and I think it's a pretty big problem - is that
performance will probably fall off a cliff as soon as you overflow the
cache. I mean, it might not, if most of the requests can be satisfied
from cache and a small percentage get fetched from cold storage, but
if it's a case where a uniformly-accessed working set is 10% larger
than the cache, the cache hit ratio will go down the tubes and so will
performance.

4. How do we match advice strings to queries? The query ID is the
obvious answer, but you could also think of having an option to match
on query text, for cases when query IDs collide. You could do
something like store a tag in the query string or in a GUC and look
that up, instead of the query ID, but then I'm not sure why you don't
just store the advice string instead of the tag, and then you don't
need a hash table lookup anymore. You could do some kind of pattern
matching on the query string rather than using the query ID, but that
feels like it would be hard to get right, and also probably more
expensive. There are probably other options here but I don't know what
they are. I doubt that it makes sense from a performance standpoint to
delegate out to a function written in a high-level language, and if
you want to delegate to C code then you can just forget about
pg_stash_advice and just use the add_advisor hook directly. I really
feel like I'm probably missing some possible techniques, here. I
wonder if other people will come up with some clever ideas.

5. What should the security model be? Right now it's as simple as can
be: the superuser gets to decide who can use the mechanism. But also,
not to be overlooked, an individual session can always opt out of
automatic advice application by clearing the stash_name GUC. It
shouldn't be possible to force wrong answers on another session even
if you can impose arbitrary plan_advice, but there is a good chance
that you could find a way to make their session catastrophically slow,
so it's good that users can opt themselves out of the mechanism.
Nonetheless, if we really want to have mutually untrusting users be
able to use this facility, then stashes should have owners, and you
should only be able to access a stash whose owner has authorized you
to do so. This is all a bit awkward because there's no way for an
extension to integrate with the built-in permissions mechanisms for
handling e.g. dropped users, and in-memory objects are a poor fit
anyway. Also, all of this is bound to have a performance cost: if
every access to a stash involves permissions checking, that's going to
add a possibly-significant amount of overhead to a code path that
might be very hot. And, in many environments, that permissions check
would be a complete waste of cycles. Things could maybe be simplified
by deciding that stash access can't be delegated: a stash has one and
only one owner, and it can affect only that user and not any other.
That is unlike what we do for built-in objects, but it simplifies the
permissions check to strcmp(), which is super-cheap compared to
catalog lookups. All in all, I don't really know which way to jump
here: what I've got right now looks almost stupidly primitive, and I
suspect it is, but adding complexity along what seem to be the obvious
lines isn't an obvious win, either.

6. How do we tell users when there's a problem? Right now, the only
available answer is to set pg_plan_advice.feedback_warnings, which I
don't think is unreasonable. If you're using advice as intended, i.e.
only for particular queries that really need it, then it really
shouldn't be generating any output unless something's gone wrong, but
I also don't think everyone is going to want this information going to
the main log file. Somehow percolating the advice feedback back to the
advisor that provided it -- pg_stash_advice or whatever -- feels like
a thing that is probably worth doing. pg_stash_advice could then
summarize the feedback and provide reports: hey, look, of the 100
queries for which you stashed advice, 97 of them always showed /*
matched */ for every item of advice, but the last 3 had varying
numbers of other results. Being able to query that via SQL seems like
it would be quite useful. I tend to think that it's almost a given
that we're going to eventually want something like this, but the
details aren't entirely clear to me. It could for example be developed
as a separate extension that can work for any advisor, rather than
being coupled to pg_stash_advice specifically -- but I'm also not
saying that's better, and I think it might be worse. Figuring out
exactly what we want to do here is a lot less critical than the items
mentioned above because, no matter what we do about any of those
things, this can always be added afterwards if and when desired. But,
I'm mentioning it for completeness.

If there are other design axes we should be talking about, I'm keen to
hear them. This is just what came to mind off-hand. Of course, I'm
also interested in everyone's views on what the right decisions are
from among the available options (or other options they may have in
mind).

Thanks,

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  179. Re: pg_plan_advice

    Lukas Fittl <lukas@fittl.com> — 2026-03-29T18:58:39Z

    Hi Robert,

On Fri, Mar 27, 2026 at 8:44 AM Robert Haas <robertmhaas@gmail.com> wrote:
>
> On Fri, Mar 27, 2026 at 4:31 AM Lukas Fittl <lukas@fittl.com> wrote:
> > For the other one 0004 (pg_stash_advice) I feel its worth trying to
> > get it in, if we can figure out the remaining issues. I'll try to do
> > another pass on that tomorrow after some sleep.
>
> Let me just talk a little about the way that I see the space of
> possible designs here. Let's set aside what we do or do not have time
> to code for a moment and just think about what makes some kind of
> sense in theory. I believe we can divide this up along a few different
> axes:
>
> 1. Where is advice stored for on-the-fly retrieval? Possible answers
> include: in shared memory, in local memory, in a table, on disk. But
> realistically, I doubt that "in a table" is a realistic option. Even
> if we hard-coded a direct index lookup i.e. without going through the
> query planner, I think this would be a fairly expensive thing to do
> for every query, and if this is going to be usable on production
> systems, it has to be fast. I am absolutely certain that "on disk" is
> not a realistic option. Local memory is an option. It has the
> advantage of making server-lifespan memory leaks impossible, and the
> further advantage of avoiding contention between different backends,
> since each backend would have its own copy of the data. The big
> disadvantage is memory consumption. If we think the advice store is
> going to be small, then that might be fine, but somebody is
> potentially going to have thousands of advice strings stored,
> duplicating that across hundreds (or, gulp, thousands) of backends is
> pretty bad.

I've been pondering this particular question for the last few days
(because pg_hint_plan uses a table, so in a sense the question is, why
not do that), and I've come to the conclusion that I think your choice
of shared memory seems reasonable, especially in combination with the
stash GUC being a way to control which stash gets used.

I don't think local memory makes as much sense, because realistically
one would want their advice applied to all backends, and whilst we
could invent some synchronization mechanism, that seems more brittle
than managing shared memory.

The other problem of using a table (if we were to go that route),
besides performance overhead, is that advice would have to be tied to
individual databases where the extension was created - and for
multi-tenant use cases where you have a one-customer-per-database
setup, it'd basically be unusable. I think the combination of shared
memory and the GUC mechanism would work really well for that since you
could pick the advice stash to use for a given database using ALTER
DATABASE, without repeating the advice definitions.

It is worth noting in such a use case, one could still have a problem
with queryids being different between databases, but as of 787514b30bb
("Use relation name instead of OID in query jumbling for
RangeTblEntry") that should no longer be the case, if the
schemas/queries match.

>
> 2. What provision do we have for durability? Possible answers include:
> in a table, on disk, nothing. I went with nothing, partly on the
> theory that it gives users more flexibility. We don't really care
> where they store their query IDs and advice strings, as long as they
> have a way to feed them into the mechanism. But of course I was also
> trying to save myself implementation complexity. There are some tricky
> things about a table: as implemented, the advice stores are
> cluster-wide objects, but tables are database objects. If we're
> supposed to automatically load advice strings from a table that might
> be in another database into an in-memory store, well, we can't. That
> could be fixed by scoping stores to a specific database, which would
> be inconvenient only for users who have the same schema in multiple
> databases and would want to share stashes across DBs, which is
> probably not common. A disk file is also an option. It requires
> inventing some kind of custom format that we can generate and parse,
> which has some complexity, but reading from a table has some
> complexity too; I'm not sure which is messier.

I think a simple disk file is the way to go, similar to how
autoprewarm works with its "autoprewarm.blocks" file. Its a bit
awkward that that just sits in the main data directory, but since
pg_prewarm already does it today, I think its okay to have another
contrib module do the same. As noted I'm mainly worried about restarts
that the user didn't control, causing advice that was set to be lost.

I've attached a patch of how that could look like on top of your v23,
that copies the modified stash information to a
"pg_stash_advice.entries" file, and loads it after restarts.

> 3. How do we limit memory usage? One possible approach involves
> limiting the size of the hash table by entries or by memory
> consumption, but I don't think that's too valuable if that's all we
> do, because presumably all that's going to do is annoy people who hit
> the limit. It could make sense if we switched to a design where the
> superuser creates the stash, assigns it a memory limit, and then
> there's a way to give permission to use that stash to some particular
> user who is bound by the memory limit. In that kind of design, the
> person setting aside the memory has different and superior privileges
> to the person using it, so the cap serves a real purpose. A
> complicating factor here is that dshash doesn't seem to be well set up
> to enforce memory limits. You can do it if each dshash uses a separate
> DSA, but that's clunky, too. A completely different direction is to
> treat the in-memory component of the system as a cache, backed by a
> table or file from which cold entries are retrieved as needed. The
> problem with this - and I think it's a pretty big problem - is that
> performance will probably fall off a cliff as soon as you overflow the
> cache. I mean, it might not, if most of the requests can be satisfied
> from cache and a small percentage get fetched from cold storage, but
> if it's a case where a uniformly-accessed working set is 10% larger
> than the cache, the cache hit ratio will go down the tubes and so will
> performance.

Because the number of entries here is controlled by the user (i.e. its
not a function of the workload, but a function of how much advice you
as a user have set), I'm much less worried about memory usage, as long
as we document it clearly how to measure the amount of memory used.

We could also consider having a parameter that sets a maximum
size/number of entries, and require you to remove entries if that is
exceeded.

I agree on your concerns that a hybrid design (i.e. one that falls
back to on-disk) has a performance cliff that will be unexpected. I
don't think we need to solve for this use case of having that many
advice strings for now, as I think the main utility of pg_stash_advice
is to fix a handful of badly behaving queries through manual
intervention, not to automatically fix thousands of them.

> 4. How do we match advice strings to queries? The query ID is the
> obvious answer, but you could also think of having an option to match
> on query text, for cases when query IDs collide. You could do
> something like store a tag in the query string or in a GUC and look
> that up, instead of the query ID, but then I'm not sure why you don't
> just store the advice string instead of the tag, and then you don't
> need a hash table lookup anymore. You could do some kind of pattern
> matching on the query string rather than using the query ID, but that
> feels like it would be hard to get right, and also probably more
> expensive. There are probably other options here but I don't know what
> they are. I doubt that it makes sense from a performance standpoint to
> delegate out to a function written in a high-level language, and if
> you want to delegate to C code then you can just forget about
> pg_stash_advice and just use the add_advisor hook directly. I really
> feel like I'm probably missing some possible techniques, here. I
> wonder if other people will come up with some clever ideas.

I think for what makes sense in tree at this point, simple queryid
matching is the way to go.

I'm sure there are better ways to do matching of queries (i.e. make it
dependent on parameters in some way, be smart about significant table
size changes, etc), but we can let the community iterate on that out
of tree, since they can just build an extension like pg_stash_advice
that use the functions provided by pg_plan_advice.

> 5. What should the security model be? Right now it's as simple as can
> be: the superuser gets to decide who can use the mechanism. But also,
> not to be overlooked, an individual session can always opt out of
> automatic advice application by clearing the stash_name GUC. It
> shouldn't be possible to force wrong answers on another session even
> if you can impose arbitrary plan_advice, but there is a good chance
> that you could find a way to make their session catastrophically slow,
> so it's good that users can opt themselves out of the mechanism.
> Nonetheless, if we really want to have mutually untrusting users be
> able to use this facility, then stashes should have owners, and you
> should only be able to access a stash whose owner has authorized you
> to do so. This is all a bit awkward because there's no way for an
> extension to integrate with the built-in permissions mechanisms for
> handling e.g. dropped users, and in-memory objects are a poor fit
> anyway. Also, all of this is bound to have a performance cost: if
> every access to a stash involves permissions checking, that's going to
> add a possibly-significant amount of overhead to a code path that
> might be very hot. And, in many environments, that permissions check
> would be a complete waste of cycles. Things could maybe be simplified
> by deciding that stash access can't be delegated: a stash has one and
> only one owner, and it can affect only that user and not any other.
> That is unlike what we do for built-in objects, but it simplifies the
> permissions check to strcmp(), which is super-cheap compared to
> catalog lookups. All in all, I don't really know which way to jump
> here: what I've got right now looks almost stupidly primitive, and I
> suspect it is, but adding complexity along what seem to be the obvious
> lines isn't an obvious win, either.

I think the current trade-off is probably okay in terms of requiring a
superuser or its equivalent to grant access to create stash entries,
and allow unprivileged users to opt out of applied advice.

In practice for a good amount of our user base these days the question
will be "Does my cloud provider give me access to create stash
entries", so its maybe worth thinking about if we could also allow
pg_maintain to manage entries by default?

> 6. How do we tell users when there's a problem? Right now, the only
> available answer is to set pg_plan_advice.feedback_warnings, which I
> don't think is unreasonable. If you're using advice as intended, i.e.
> only for particular queries that really need it, then it really
> shouldn't be generating any output unless something's gone wrong, but
> I also don't think everyone is going to want this information going to
> the main log file. Somehow percolating the advice feedback back to the
> advisor that provided it -- pg_stash_advice or whatever -- feels like
> a thing that is probably worth doing. pg_stash_advice could then
> summarize the feedback and provide reports: hey, look, of the 100
> queries for which you stashed advice, 97 of them always showed /*
> matched */ for every item of advice, but the last 3 had varying
> numbers of other results. Being able to query that via SQL seems like
> it would be quite useful. I tend to think that it's almost a given
> that we're going to eventually want something like this, but the
> details aren't entirely clear to me. It could for example be developed
> as a separate extension that can work for any advisor, rather than
> being coupled to pg_stash_advice specifically -- but I'm also not
> saying that's better, and I think it might be worse. Figuring out
> exactly what we want to do here is a lot less critical than the items
> mentioned above because, no matter what we do about any of those
> things, this can always be added afterwards if and when desired. But,
> I'm mentioning it for completeness.

I think figuring out a feedback mechanism would be a good thing, but
we could definitely add that in a later release without a major design
change of pg_stash_advice, I think. And as you note,
pg_plan_advice.feedback_warnings exists as a mechanism today to
validate whether advice was applied as expected.

> If there are other design axes we should be talking about, I'm keen to
> hear them. This is just what came to mind off-hand. Of course, I'm
> also interested in everyone's views on what the right decisions are
> from among the available options (or other options they may have in
> mind).

I can't think of other angles at this point, at least not in the
context of what it makes sense to do in-tree for this release.

Thanks,
Lukas

-- 
Lukas Fittl

  180. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-03-30T14:53:30Z

    Hi,

Thanks for taking the time to respond. My reading of your comments is
that we are in overall agreement on the design, with the possible
exception of persisting data cross restarts. I will write more about
that topic below; but I think if that's the only design disagreement
we have, it makes sense to go forward with committing the patch that I
have, and we can continue to discuss whether we want to add something
related to persistence. The only reason not to do that would be if
there were a consensus that the lack of a persistence framework was
such a critical defect that we shouldn't ship this at all without
that, but I don't agree with that idea and I think it would be a
pretty strong position for someone to take.

On Sun, Mar 29, 2026 at 2:59 PM Lukas Fittl <lukas@fittl.com> wrote:
> I think a simple disk file is the way to go, similar to how
> autoprewarm works with its "autoprewarm.blocks" file. Its a bit
> awkward that that just sits in the main data directory, but since
> pg_prewarm already does it today, I think its okay to have another
> contrib module do the same. As noted I'm mainly worried about restarts
> that the user didn't control, causing advice that was set to be lost.
>
> I've attached a patch of how that could look like on top of your v23,
> that copies the modified stash information to a
> "pg_stash_advice.entries" file, and loads it after restarts.

I'll be honest: I don't like this design much at all, but I do see the
practical advantages of it, and we have done similar things elsewhere,
in particular in autoprewarm. Before I get to the specifics of your
patch, let me complain about some things that I don't like at the
design level. We lose a lot by directing data through a bespoke
mechanism rather than handling it as table data. There are no
checksums, so we have less protection against corruption. There is no
write-ahead logging, so data does not make it to standbys, which is
more of a potential issue for pg_stash_advice than it is for
autoprewarm. All the code to read and write the file is specific to
this contrib module, so it can have its own bugs separate from every
similar module's bugs. The data can't easily be examined and
manipulated from SQL as table data can. It's just a messy one-off that
solves a practical problem but is otherwise not very nice. Of course,
sometimes such messy one-offs are the right answer.

In terms of the patch itself, the concurrency situation here seems
noticeably worse than with autoprewarm. In that case, there's only one
authorized writer at a time, tracked via pid_using_dumpfile. But in
this case, it seems like multiple backends could be writing to the
temporary dump file at the same time, which could result in a
corrupted file that doesn't reload properly. Your code also has a race
condition when reloading the data: the first arriving backend tries to
reload the flat file, but any other backends that arrive while that's
in progress see no stashed advice, and if the load fails for some
reason, it's never retried, and the first modification to the
in-memory state will clobber the file. autoprewarm has this issue to
some extent as well, but that's more OK there because recreating the
contents of shared buffers is only an approximate good, but people
probably don't want their stashed advice to disappear out from under
them if it was billed as persistent. That said, I'm not entirely
opposed to a design where there's a small window where the advice
stash is empty after a restart, because avoiding that means that it
has to be safe to do the reload of saved advice from the middle of a
query planning cycle, which is probably true with a flat-file design
but wouldn't be true with a table. Still, I don't know whether the
current behavior is deliberate or accidental.

I also feel a bit uncomfortable with the idea of rewriting the entire
file on every single change. If the hypothesis that this is only for
adjusting the behavior of a small number of critical queries is
correct, then it won't matter, but if people start using this for lots
of queries, it's potentially painful. Neither autoprewarm nor
pg_stat_statements does that. pg_stat_statements reads data only at
postmaster startup and writes data only at postmaster shutdown, so it
simply accepts loss of incremental changes in case of a crash, but
that also means it doesn't read and write the file repeatedly.
autoprewarm writes the file periodically from a background worker so
that the on-disk state doesn't drift too far out of sync with what's
in memory, without promising perfect durability. Both of those
placements have the further advantage that the reading and writing of
the file is not being done "in medias res," which does seem to have
certain advantages from a robustness perspective. For example, without
necessarily endorsing this design, suppose you added a background
worker and there are GUCs to configure the database that it connects
to and the query it executes to restore advice stashes. Or,
alternatively, a background worker that still uses a flat file. Either
way, that opens up design ideas like: when you see that the in-memory
stashes are not yet reloaded, you can decide to wait up to X seconds
for that to happen and then proceed anyway if it hasn't happened by
then. I'm not saying that is the right idea here necessarily, but it's
an option, whereas what you've done doesn't lend itself to that sort
of idea.

One other note is that fscanf() ending in a newline could eat up
whitespace at the start of the following line. Since a stash name can
begin with whitespace, that could be an issue.

> Because the number of entries here is controlled by the user (i.e. its
> not a function of the workload, but a function of how much advice you
> as a user have set), I'm much less worried about memory usage, as long
> as we document it clearly how to measure the amount of memory used.

The module doesn't have a built-in way to do that right now. Are you
thinking we would document that pg_get_dsm_registry_allocations() can
be used?

> In practice for a good amount of our user base these days the question
> will be "Does my cloud provider give me access to create stash
> entries", so its maybe worth thinking about if we could also allow
> pg_maintain to manage entries by default?

Wouldn't it make more sense for the cloud provider to grant execute
permissions on these functions to pg_maintain if they are so inclined?
This is a brand-new facility, so I think we had better be conservative
in terms of default permissions.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  181. Re: pg_plan_advice

    Lukas Fittl <lukas@fittl.com> — 2026-04-01T02:25:04Z

    On Mon, Mar 30, 2026 at 7:53 AM Robert Haas <robertmhaas@gmail.com> wrote:
>
> Hi,
>
> Thanks for taking the time to respond. My reading of your comments is
> that we are in overall agreement on the design, with the possible
> exception of persisting data cross restarts. I will write more about
> that topic below; but I think if that's the only design disagreement
> we have, it makes sense to go forward with committing the patch that I
> have, and we can continue to discuss whether we want to add something
> related to persistence. The only reason not to do that would be if
> there were a consensus that the lack of a persistence framework was
> such a critical defect that we shouldn't ship this at all without
> that, but I don't agree with that idea and I think it would be a
> pretty strong position for someone to take.

Yeah, I think my position is that having a solution to persistence
would be very good, but if that's not doable for this release, I think
we have a potential way forward in future releases, at least when it
comes to being restart-safe.

That said, I still think it'll make a big difference in practice to be
restart safe right away, if we can make it happen.

>
> On Sun, Mar 29, 2026 at 2:59 PM Lukas Fittl <lukas@fittl.com> wrote:
> > I think a simple disk file is the way to go, similar to how
> > autoprewarm works with its "autoprewarm.blocks" file. Its a bit
> > awkward that that just sits in the main data directory, but since
> > pg_prewarm already does it today, I think its okay to have another
> > contrib module do the same. As noted I'm mainly worried about restarts
> > that the user didn't control, causing advice that was set to be lost.
> >
> > I've attached a patch of how that could look like on top of your v23,
> > that copies the modified stash information to a
> > "pg_stash_advice.entries" file, and loads it after restarts.
>
> I'll be honest: I don't like this design much at all, but I do see the
> practical advantages of it, and we have done similar things elsewhere,
> in particular in autoprewarm. Before I get to the specifics of your
> patch, let me complain about some things that I don't like at the
> design level. We lose a lot by directing data through a bespoke
> mechanism rather than handling it as table data. There are no
> checksums, so we have less protection against corruption. There is no
> write-ahead logging, so data does not make it to standbys, which is
> more of a potential issue for pg_stash_advice than it is for
> autoprewarm. All the code to read and write the file is specific to
> this contrib module, so it can have its own bugs separate from every
> similar module's bugs. The data can't easily be examined and
> manipulated from SQL as table data can. It's just a messy one-off that
> solves a practical problem but is otherwise not very nice. Of course,
> sometimes such messy one-offs are the right answer.

I think if we wanted a table, we should make it a table - but I think
the fact that we want this to be low-overhead for running queries to
examine whether there is anything for them to apply, would require
some kind of cache in front of it, and that gets complicated pretty
quickly.

For the file-based direction, just for reference, I'm attaching an
updated version of that (on top of Robert's earlier v23), that
utilizes a background worker to write out the dump file as needed, at
most every 60 seconds. It also reworks some of the output logic to do
better memory management, and uses a TSV file format that can be
easily read again.

To be clear, I think its okay to go ahead with merging pg_stash_advice
without that and make it a best effort to get the file saving in too -
but I think with the current design in this patch represents a
reasonable solution to what we can do in terms of persistence across
restarts in either 19 or 20.

>
> > Because the number of entries here is controlled by the user (i.e. its
> > not a function of the workload, but a function of how much advice you
> > as a user have set), I'm much less worried about memory usage, as long
> > as we document it clearly how to measure the amount of memory used.
>
> The module doesn't have a built-in way to do that right now. Are you
> thinking we would document that pg_get_dsm_registry_allocations() can
> be used?

Yeah, for example. Alternatively we could provide a function/view that
lists all advice across all stashes, so you can more easily see the
result size of that and estimate what the in-memory use is. But
pointing to pg_get_dsm_registry_allocations seems easier.

> > In practice for a good amount of our user base these days the question
> > will be "Does my cloud provider give me access to create stash
> > entries", so its maybe worth thinking about if we could also allow
> > pg_maintain to manage entries by default?
>
> Wouldn't it make more sense for the cloud provider to grant execute
> permissions on these functions to pg_maintain if they are so inclined?
> This is a brand-new facility, so I think we had better be conservative
> in terms of default permissions.

I guess. I'm always worried that providers get that wrong and forget
to give end users the permissions - but I suppose end users can
complain to their providers if that's the case.

I've done another look over pg_set_stashed_advice and I think its in
good shape. The only trailing thought I have is that we could consider
running a fuzzer against the pg_set_advice function in particular,
just to see if anything pops up (beyond having the ability to make a
very large memory allocation through a large advice string, which is
maybe a problem?).

Thanks,
Lukas

-- 
Lukas Fittl

  182. Re: pg_plan_advice

    Lukas Fittl <lukas@fittl.com> — 2026-04-01T06:33:50Z

    On Tue, Mar 31, 2026 at 7:25 PM Lukas Fittl <lukas@fittl.com> wrote:
>
> On Mon, Mar 30, 2026 at 7:53 AM Robert Haas <robertmhaas@gmail.com> wrote:
> >
> >
> > The module doesn't have a built-in way to do that right now. Are you
> > thinking we would document that pg_get_dsm_registry_allocations() can
> > be used?
>
> Yeah, for example. Alternatively we could provide a function/view that
> lists all advice across all stashes, so you can more easily see the
> result size of that and estimate what the in-memory use is. But
> pointing to pg_get_dsm_registry_allocations seems easier.

Actually, that won't work in practice with the code as of v23 -
pg_get_dsm_registry_allocations() always returns the fixed 64 byte
allocation from GetNamedDSMSegment, but is oblivious to the individual
DSA allocations (even after adding hundreds of entries):

SELECT * FROM pg_get_dsm_registry_allocations();

      name       |  type   | size
-----------------+---------+------
 pg_stash_advice | segment |   64
(1 row)

Is there a reason you didn't use GetNamedDSA / GetNamedDSHash for the
other allocations? (which we have as of fe07100e82b09)

With the adjustments in the attached patch, this gets reported as expected:

SELECT * FROM pg_get_dsm_registry_allocations();

         name          |  type   |   size
-----------------------+---------+-----------
 pg_stash_advice       | segment |        24
 pg_stash_advice_stash | hash    |   1048576
 pg_stash_advice_dsa   | area    | 803209216
 pg_stash_advice_entry | hash    |   1048576
(4 rows)

>
> > > In practice for a good amount of our user base these days the question
> > > will be "Does my cloud provider give me access to create stash
> > > entries", so its maybe worth thinking about if we could also allow
> > > pg_maintain to manage entries by default?
> >
> > Wouldn't it make more sense for the cloud provider to grant execute
> > permissions on these functions to pg_maintain if they are so inclined?
> > This is a brand-new facility, so I think we had better be conservative
> > in terms of default permissions.
>
> I guess. I'm always worried that providers get that wrong and forget
> to give end users the permissions - but I suppose end users can
> complain to their providers if that's the case.
>
> I've done another look over pg_set_stashed_advice and I think its in
> good shape. The only trailing thought I have is that we could consider
> running a fuzzer against the pg_set_advice function in particular,
> just to see if anything pops up (beyond having the ability to make a
> very large memory allocation through a large advice string, which is
> maybe a problem?).

Obviously I meant "I've done another look over pg_stash_advice and I
think its in good shape".

I've done some basic fuzzing with the pg_set_stashed_advice function,
including concurrently setting advice, and that didn't yield any
surprises.

Thanks,
Lukas

-- 
Lukas Fittl

  183. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-04-02T16:15:09Z

    On Tue, Mar 31, 2026 at 10:25 PM Lukas Fittl <lukas@fittl.com> wrote:
> To be clear, I think its okay to go ahead with merging pg_stash_advice
> without that and make it a best effort to get the file saving in too -
> but I think with the current design in this patch represents a
> reasonable solution to what we can do in terms of persistence across
> restarts in either 19 or 20.

Somewhat against my better judgement, I have attempted to put your
patch into committable shape. In the process, I rewrote pretty much
the whole thing.  So here's v24, also dropping pg_collect_advice.

0001 is the pg_stash_advice patch from v23, but with a number of
changes motivated by your desire to add persistence. I split the code
into two files, because I felt that file was starting to get a little
bit large, and I didn't want to just add a whole bunch more stuff to
it. For the most part, this is just code movement, but I did make a
couple of substantive changes. First, I restricted stash names to
alphanumeric characters and underscores, basically looking like
identifier names. This is partly because I got thinking about the
escaping requirements for the persistence file, but it's also because
I realized that letting somebody name their stashes with spaces or
non-printable characters in the name or a bunch of random punctuation
was probably more confusing than useful. Second,
pgsa_set_advice_string() was previously taking a lock itself, but most
of its sister functions require the caller to do that; I changed it to
match. Third, lock is also now held when calling
pgsa_clear_advice_string(), which may not be entirely necessary, but
it seems safer and shouldn't cost anything meaningful.

0002 adds persistence. Here's a list of changes from your version:

- I changed the GUC name pg_stash_advice.save to
pg_stash_advice.persist, since it controls both whether advice is
saved automatically and also whether it's loaded automatically at
startup time.

- I added a GUC pg_stash_advice.persist_interval, so that the interval
between writes can be configured.

- Instead of a dump_requested flag, I added a pg_atomic_uint64
change_count. This avoids needing to take &pgsa_state->lock in
LW_EXCLUSIVE mode. Even leaving that aside, I don't think a Boolean is
adequate here. Your patch cleared the flag before dumping, but that
means if the act of dumping fails, you forget that it needed to be
done. If you instead clear the flag after dumping, then you don't
realize you need to do it again if any concurrent changes happen.

- I set the restart time to BGW_DEFAULT_RESTART_INTERVAL rather than
0. Restarting the worker in a tight loop is a bad plan.

- I added a function pg_start_stash_advice_worker(). You could do
something like add pg_stash_advice to session_preload_libraries, start
using it, and use this to kick off the worker. Then eventually you can
restart the server with pg_stash_advice moved to
shared_preload_libraries.

- As you had coded it, any interrupt that jostled the worker would
trigger an immediate write-out if one was pending. That behavior seems
hard to explain and document, so I made it work more like autoprewarm,
which always respects the configured interval even in case of
interrupts.

- I added a mechanism to prevent the user from manually manipulating
stashes or stash entries when persistence is enabled but before the
dump file has been reloaded. Without this, reloading the dump file
could error out if, for example, the user already managed to recreate
a stash with the same name as one that exists in the dump file.

- As you had coded it, data is inserted into the dynamic shared memory
structures as it's read from the disk file. I felt that could produce
rather odd behavior, especially in view of the lack of the lockout
mechanism mentioned in the previous point. We might partially process
the dump file and then die with some data loaded and other data not
loaded. Other backends could see the partial results. While the
lockout mechanism by itself is sufficient to prevent that, I felt
uncomfortable about relying on that completely. It means we start
consuming shared memory even before we know whether there's an error,
and continue to consume it after we've died from an error, and it also
means we have a very hard dependency on the lockout mechanism, which
really only works if there's only ever one load and save file and
loading only happens at startup time. I felt it was better to slurp
all the data into memory first, parse it completely, and then start
making changes to shared memory only if we don't find any problems, so
I made it work that way. We replace the tabs and newlines that end
fields and lines with \0 on the fly so that we can just point into
that buffer, instead of having to pstrdup() anything. (Note that, even
if we stuck with your approach of something based on pg_get_line(), it
would probably be better to use one of the other variants, e.g.
pg_get_line_buf(), to avoid allocating new buffers constantly.)

- I completely reworked the string escaping. Your pgsa_escape_string()
had a bug where the strpbrk call didn't check for \r. Also, I didn't
like the behavior of just ignoring a backslash when it was followed by
end of string or something unexpected; I felt those should be errors.
Given the decision to slurp the whole file, as mentioned in the
previous point, it also made sense to do the unescaping in place, so
that we didn't need to allocate additional memory. I particularly
didn't like the decision to sometimes allocate memory and sometimes
not. While it was economical in a sense, it meant that the memory
consumption could be very different depending on how many entries
needed escaping.

- I completely reworked the error reporting. Now, if we hit an error
parsing the file (or doing anything else), we just signal an ERROR,
and we rely on the fact that the postmaster will restart us. It's an
explicit goal not to apply incremental changes when the file overall
is not valid, which also means that we are only concerned about
reporting the first error that we detect, which also seems good for
avoiding log spam. On the other hand, the error reports are more
specific and detailed, and now all follow the same general pattern:
"syntax error in file \"%s\" line %u: problem description here". (I
was also not entirely happy with the fact that you could potentially
call fprintf() lots of times before finally reporting an error. While
you did save and restore errno around the calls to FreeFile() and
unlink(), I think it makes the code hard to reason about; e.g. what if
a later fprintf call hit a different error than an earlier one?)

- I felt it was a bit odd to install a zero length file, so I made the
persistence mechanism remove the existing file if there's nothing to
persist when it goes to write out. I am not totally sure this was the
right call.

- I added a message when saving entries symmetrical to the one you had
for loading entries, and also some DEBUG1/2 messages in case someone
needs more details.

- I added a TAP test. This isn't as comprehensive as it could be -- in
particular, it doesn't cover all the possible error cases that occur
when trying to reload data from disk. I could add that, but it would
mean stopping and restarting the server a bunch of times, and I wasn't
sure that it was a good idea to add that much overhead for a few lines
of code coverage.

- Your documentation changes still said that the stash data would be
"restored when the first session attaches after a server restart" but
that doesn't sound like something that a user will understand, and
also wasn't what actually happened since you added the background
worker. I rewrote this.

There's almost none of your code remaining at this point, but I listed
you as a co-author for 0002. I think a case could be made for Author
or for no listing. Let me know if you have an opinion. And, please
review!

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  184. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-04-02T16:44:32Z

    On Wed, Apr 1, 2026 at 2:34 AM Lukas Fittl <lukas@fittl.com> wrote:
> Is there a reason you didn't use GetNamedDSA / GetNamedDSHash for the
> other allocations? (which we have as of fe07100e82b09)

I'm under the impression that GetNamedDSA and GetNamedDSHash exist for
the purposes of making it easy for extensions to coordinate with each
other across backends, rather than being something you're supposed to
use to improve observability. I think it's potentially good for there
to be a way to see the size of every DSA that exists in the system,
but this clearly isn't that, because none of the code in src/backend
uses it when creating DSAs. You might argue that DSAs for short-lived
things like parallel query or parallel VACUUM don't need to be tracked
like this (which seems arguable), but they are also used for
long-lived contexts in the logical replication launcher, by
LISTEN/NOTIFY, by the shared-memory statistics collector, and in
GetSessionDsmHandle(), and those places don't use GetNamedDSA()
either.

Architecturally, I don't like the idea of replacing "having a pointer
to an object" with "being able to look up that object by name". I
think it's good design that pg_stash_advice creates one structure that
serves as a sort of root and then hangs everything else off of that. I
admit that leaves me not knowing quite what to do about the problem of
knowing how much memory it's using, though. Adding a function just to
return size information seems a little clunky, but it might be the
right idea: it could for example return a count of stashes, a count of
entries, the total length of the entries, and the allocated size of
the DSA.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  185. Re: pg_plan_advice

    Tom Lane <tgl@sss.pgh.pa.us> — 2026-04-02T23:43:33Z

    My animal sifaka just showed an all-new type of test_plan_advice
failure [1]:

diff -U3 /Users/buildfarm/bf-data/HEAD/pgsql.build/src/test/regress/expected/limit.out /Users/buildfarm/bf-data/HEAD/pgsql.build/src/test/modules/test_plan_advice/tmp_check/results/limit.out
--- /Users/buildfarm/bf-data/HEAD/pgsql.build/src/test/regress/expected/limit.out	2026-04-02 12:35:13
+++ /Users/buildfarm/bf-data/HEAD/pgsql.build/src/test/modules/test_plan_advice/tmp_check/results/limit.out	2026-04-02 12:49:59
@@ -5,6 +5,8 @@
 SELECT ''::text AS two, unique1, unique2, stringu1
 		FROM onek WHERE unique1 > 50
 		ORDER BY unique1 LIMIT 2;
+WARNING:  supplied plan advice was not enforced
+DETAIL:  advice INDEX_SCAN(onek public.onek_unique1) feedback is "matched, inapplicable, failed"
  two | unique1 | unique2 | stringu1 
 -----+---------+---------+----------
      |      51 |      76 | ZBAAAA
=== EOF ===
[12:50:02.062](11.620s) not ok 1 - regression tests pass

This is unlike the other cases we've been looking at: no sub-selects,
no GEQO, not even any joins.  There is pretty much only one plausible
plan for that query, so how did it fail?

After looking around, I think the likely explanation is that the
concurrently-run alter_table.sql test feels free to mess with the set
of indexes on onek.  It doesn't drop onek_unique1, but it does
momentarily rename it:

ALTER INDEX onek_unique1 RENAME TO attmp_onek_unique1;
ALTER INDEX attmp_onek_unique1 RENAME TO onek_unique1;

I've not looked closely at pg_plan_advice, but if it matches indexes
by name then it seems there's a window here where the advice would
fail to apply.  Also, further down we find

ALTER TABLE onek ADD CONSTRAINT onek_unique1_constraint UNIQUE (unique1);
ALTER INDEX onek_unique1_constraint RENAME TO onek_unique1_constraint_foo;
ALTER TABLE onek DROP CONSTRAINT onek_unique1_constraint_foo;

which means there's a window there where there are two plausible
indexes to choose.  Will test_plan_advice cope if the transient one
is chosen?

We could imagine dodging this problem either by having alter_table.sql
test some purpose-built table instead of a shared one, or by having it
do these hacks inside transactions so that other sessions can't see
the intermediate states.  But I'm quite resistant to that answer,
because I think part of the point here is to ensure that concurrent
DDL doesn't misbehave.  (Which it doesn't: these test fragments have
been there since 2018 and 2012 respectively, and not caused issues
AFAIK.)  Preventing our tests from exercising concurrent DDL in order
to satisfy test_plan_advice is not a good plan IMO.  There's also the
prospect of a long tail of whack-a-mole as we locate other places in
the tests where this sort of thing happens occasionally.

So I'm not sure what to do here, but we have a problem.

			regards, tom lane

[1] https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=sifaka&dt=2026-04-02%2016%3A35%3A13

  186. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-04-03T02:08:51Z

    Thanks for the report.

On Thu, Apr 2, 2026 at 7:43 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> After looking around, I think the likely explanation is that the
> concurrently-run alter_table.sql test feels free to mess with the set
> of indexes on onek.  It doesn't drop onek_unique1, but it does
> momentarily rename it:
>
> ALTER INDEX onek_unique1 RENAME TO attmp_onek_unique1;
> ALTER INDEX attmp_onek_unique1 RENAME TO onek_unique1;

Yeah, the fact that it said /* inapplicable */ strongly supports this
theory. There's only two ways that can happen, and an index not
existing with the expected name is one of them (and the only one
that's possible in a query involving only a single table).

> I've not looked closely at pg_plan_advice, but if it matches indexes
> by name then it seems there's a window here where the advice would
> fail to apply.  Also, further down we find
>
> ALTER TABLE onek ADD CONSTRAINT onek_unique1_constraint UNIQUE (unique1);
> ALTER INDEX onek_unique1_constraint RENAME TO onek_unique1_constraint_foo;
> ALTER TABLE onek DROP CONSTRAINT onek_unique1_constraint_foo;
>
> which means there's a window there where there are two plausible
> indexes to choose.  Will test_plan_advice cope if the transient one
> is chosen?

It's going to be unhappy if the second planning cycle is incapable of
choosing the same index that the first planning cycle did. I have to
admit that it didn't occur to me that our regression tests would do
something like this. I figured they had to be operating on mostly
independent objects or things would already be broken, but I failed to
consider the possibility that there could be concurrent DDL of a sort
that wouldn't affect the normal running of the regression tests but
would affect pg_plan_advice. Or at least, if I did ever consider it, I
stopped considering it when test_plan_advice appeared to be passing.

> So I'm not sure what to do here, but we have a problem.

I'm not sure, either, and I agree that we have a problem. I'll give it
some more thought tomorrow.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  187. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-04-03T02:15:36Z

    On Thu, Apr 2, 2026 at 12:15 PM Robert Haas <robertmhaas@gmail.com> wrote:
> So here's v24, also dropping pg_collect_advice.

That version didn't actually pass CI. Here's v25.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  188. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-04-03T17:13:47Z

    On Thu, Apr 2, 2026 at 10:08 PM Robert Haas <robertmhaas@gmail.com> wrote:
> I'm not sure, either, and I agree that we have a problem. I'll give it
> some more thought tomorrow.

OK, here are my thoughts.

I don't believe it's viable to change pg_plan_advice in such a way
that it won't run into trouble in cases like this.  Somebody could
argue that the choice of INDEX_SCAN(table_name index_name) was bad
design, and that I should have done something like
INDEX_SCAN(table_name indexed_columns) instead, and that might be
true. There might also be an argument that we should have both things
with different spellings, and that might also very well be true. But
we don't really know that changing that design decision would fully
stabilize test_plan_advice. The regression tests can do anything they
like, as long as they reliably pass. It now seems optimistic to me to
suppose that an index with a different name is the only current or
future issue we'll ever have. I mean, if the table were small enough
not to care about whether an index scan or a sequential scan is used,
you could concurrently drop the one and only index altogether, and
what's test_plan_advice supposed to do about that?

So, I argue that there are three possible categories of solutions
here: (1) don't let the problem cases happen in the first place, (2)
detect that a problem has happened, or (3) give up on
test_plan_advice.

In category (1), the simplest idea would be (1a) to run the tests
serially. That would probably involve running them much less often,
like in one of the CI builds but not all of them or something like
that. Another idea that I had is to (1b) try to take stronger locks on
the relations involved to prevent concurrent DDL on them, like a
ShareUpdateExclusiveLock, or (1c) some kind of bespoke interlock
specific to test_plan_advice. I think that might cause random breakage
of other types, though. Another idea in this category is to try to
make the main regression tests "pg_plan_advice clean". I know Tom
already expressed opposition to that idea, but here me out: we could
(1d) have a separate test suite that still does stuff like this, so we
don't lose test coverage, and move some stuff there. Or, instead of
completely separating it, we could (1e) have two schedule files, one
of which includes all the tests and a second of which includes only
the tests that are test_plan_advice-clean. Although my theory that the
main regression tests couldn't have multiple different sessions
simultaneously doing DDL on the same objects has been proven wrong,
I'd still be willing to bet that it's a minority position. Of course,
as Tom pointed out, there could be a "long tail" of failures here, but
maybe we could create some throwaway infrastructure to help figure
that out. For example if we're mostly worried about tables, we could
have each backend accumulate a list of table OIDs that it touched and
spit that out into the log file when it exits. That wouldn't be
committable code, but it would be enough to let us run the regression
tests with that once and see what overlaps exist. I bet there's very
low risk of newly-added tests adding more such cases: the ones that we
have are probably ancient. Of course, maybe I'm wrong about that, too,
but it's a theory that we can discuss.

In category (2), what if, (2a) whenever we see advice feedback that
we'd otherwise print, we try replanning the query a THIRD time without
any supplied advice? If we generate different advice than we did the
first time we planned it, then we know for sure that something is
unstable, and we can decide not to complain about whatever went wrong.
This isn't completely guaranteed to work, though: what if concurrent
DDL changes something between planning cycle 1 and planning cycle 2
and then changes it back before planning cycle 3? But maybe it would
be acceptable to make a rule that the main regression test case
shouldn't do that, and adjust cases that currently do to work
otherwise. If we're not willing to make any rules at all to prevent
the main regression test suite from sabotaging test_plan_advice, then
it's probably doomed. And, I think there's a reasonable argument that
insisting that the main regression test suite absolutely has to change
the definition of an object in a way that test_plan_advice will care
about and then change it back to exactly the initial state while in a
concurrent session some other backend is running queries against that
object is tantamount to legislating deliberate sabotage. But that
said, this proposal has some other imperfections as well. In
particular, a bug that caused the third planning cycle to always
produce different results than the first would hide all future
problems that test_plan_advice might have caught, which is pretty sad.
Another variant of the same basic idea is to (2b) just detect when
we've seen any shared invalidations between the start of the first
planning cycle and the end of the second, and go "never mind, don't
complain even if we saw a problem". The problem with this idea is
that, as in the previous proposal, it might make the tests too
insensitive to real issues. But I wonder if this might be fixable.
Maybe we could (2c) make test_plan_advice take planner_hook and wrap a
loop around the problem: it just keeps replanning the query via
standard_planner (which would eventually reach
test_plan_advice_advisor) until no sinval messages are absorbed
between the start and end of planner, which I think we could detect
using SharedInvalidMessageCounter, or until some retry limit is
exhausted and we error out. I'd need to try this and see how well it
works out in practice, and how often the retry is actually hit, but it
seems like it might be somewhat viable.

In category (3), the most blunt option is obviously just (3a) throw
test_plan_advice away, which I think is probably dooming
pg_plan_advice to getting silently broken in the future. I don't
really have any other ideas in this category except for (1a) already
mentioned, which is sort of a hybrid solution.

My current thought is to do some research into (1e) and (2c).
Specifically, for (1e), I want to try to figure out if this is the
only case of this type or if there are lots of others, since that
seems likely to have a pretty large bearing on what is realistic here.
And for (2c), I think I just want to try it out and see if it seems at
all feasible. Probably obviously, this is not going to happen before
next week, but I hope that the frequency of buildfarm failures is now
low enough that this isn't a critical issue. If that's wrong, let me
know, but from my point of view, even if we eventually chose (3a),
having a good a sense as possible of what the potential failure modes
are here would help to design the next solution, and AFAIK this is the
first failure we've seen since the DO_NOT_SCAN stuff went in.

(In fact, I had a little bit of trouble finding this in the BF results
even knowing it was there: filtering by test_plan_advice failures
doesn't find anything recent. sifaka's failure shows up as
TestModulesCheck-en_US.UTF-8, but frustratingly, the names for the
stage logs don't seem to quite match the name of what failed. There is
testmodules-install-check-C and
testmodules-install-check-en_US.UTF-8, but those have "install" in the
name and are punctuated differently, so it's not instantly clear that
it's the same thing. Anyway, I do see it in there now, but what I'm
saying is that if there have been other failures that are related to
this, it's possible I have missed them due to stuff like this, so it's
helpful that you (Tom) pointed this one out.)

Tom, would welcome your thoughts, if you have any, and anyone else's
thoughts as well. If none, I'll proceed as described above and update
when I know more.

Thanks,

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  189. Re: pg_plan_advice

    Tom Lane <tgl@sss.pgh.pa.us> — 2026-04-03T18:20:32Z

    Robert Haas <robertmhaas@gmail.com> writes:
> (In fact, I had a little bit of trouble finding this in the BF results
> even knowing it was there: filtering by test_plan_advice failures
> doesn't find anything recent. sifaka's failure shows up as
> TestModulesCheck-en_US.UTF-8, but frustratingly, the names for the
> stage logs don't seem to quite match the name of what failed. There is
> testmodules-install-check-C and
> testmodules-install-check-en_US.UTF-8, but those have "install" in the
> name and are punctuated differently, so it's not instantly clear that
> it's the same thing. Anyway, I do see it in there now, but what I'm
> saying is that if there have been other failures that are related to
> this, it's possible I have missed them due to stuff like this, so it's
> helpful that you (Tom) pointed this one out.)

I grepped the buildfarm database for 'supplied plan advice' and got
no other hits since 6455e55b0 went in.  That's not a huge sample
size of course, but probably several hundred runs so far.  If there's
another message wording I should check for, let me know.

> Tom, would welcome your thoughts, if you have any, and anyone else's
> thoughts as well. If none, I'll proceed as described above and update
> when I know more.

I don't like anything in category 1 except (1a) run the test scripts
serially for test_plan_advice.  As I said before, I am strongly
against allowing test_plan_advice to constrain what our tests do.

Another idea in category 2, which I think is a bit different from
any option you listed, is to repeat the "plan without advice, then
again with advice, see if it matches" process up to maybe 5-ish times
before declaring failure.  If it works any one time, then write off
the previous failures as being induced by concurrent activity.
Unlike what you mentioned, this isn't dependent on sinval checks,
which I think are next door to useless in the context of the
regression tests: there's a constant storm of sinval activity going
on, to the point where you might as well figure "check for sinval
arrival" is constant "true".

However, eyeing the calendar, I think the only options that are likely
to be stabilizable before feature freeze are (1a) run the test scripts
serially for test_plan_advice or (3a) throw test_plan_advice away.
I know you don't want to do (3a) and I understand why not.  How much
will (1a) slow things down?

			regards, tom lane

  190. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-04-04T00:14:49Z

    On Fri, Apr 3, 2026 at 2:20 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> However, eyeing the calendar, I think the only options that are likely
> to be stabilizable before feature freeze are (1a) run the test scripts
> serially for test_plan_advice or (3a) throw test_plan_advice away.
> I know you don't want to do (3a) and I understand why not.  How much
> will (1a) slow things down?

I don't know. For me, the speed of the regression tests is rarely a
bottleneck, and they run on my machine in about 12 seconds. But on
slow buildfarm machines, I'm guessing it's going to extend the runtime
significantly. But I also feel like if we've only seen one buildfarm
failure since the last round of stabilization, it might not be a
catastrophe if nothing further is done before feature freeze. In fact,
I think it might be *good*. Given the apparently-low failure rate that
we now have, it feels to me like we might want to run like this for a
month or even or two or three to get a clearer feeling for whether the
failure you saw is the only one or whether, perhaps, there are others.
Or even just how often this one happens. I mean, I'm also not that
opposed to having it made serial now if you really think that's
better. But what concerns me is I feel like we might inconvenience a
lot of people who really care about the tests running fast while at
the same time eliminating our ability to gather any more information
about the problem.

I mean, there is possibly an argument that we don't really need to
gather any more information about the problem; it does seem like we
understand what is going on here, and if we had a great, simple fix I
would probably just apply it and be done with it. But I also don't
quite understand why you're in such a rush. If we still feel like
running the tests serially is the best solution in a month, can't we
just do it then?

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  191. Re: pg_plan_advice

    Tom Lane <tgl@sss.pgh.pa.us> — 2026-04-04T03:14:46Z

    Robert Haas <robertmhaas@gmail.com> writes:
> ... But I also feel like if we've only seen one buildfarm
> failure since the last round of stabilization, it might not be a
> catastrophe if nothing further is done before feature freeze. In fact,
> I think it might be *good*. Given the apparently-low failure rate that
> we now have, it feels to me like we might want to run like this for a
> month or even or two or three to get a clearer feeling for whether the
> failure you saw is the only one or whether, perhaps, there are others.
> Or even just how often this one happens.

Reasonable point.

> I mean, there is possibly an argument that we don't really need to
> gather any more information about the problem; it does seem like we
> understand what is going on here, and if we had a great, simple fix I
> would probably just apply it and be done with it. But I also don't
> quite understand why you're in such a rush. If we still feel like
> running the tests serially is the best solution in a month, can't we
> just do it then?

The terms that I'm thinking in are "how much redesign will we accept
post-feature-freeze, in either pg_plan_advice or test_plan_advice,
before choosing to revert those modules entirely for v19?".  I think
that running those tests serially is a sufficiently low-risk option
that it'd be okay to put it in post-freeze, even very long after.
I'm not sure that any of the other group-1 or group-2 options you
suggested would be okay post-freeze.  (Of course, ultimately that'd
be the RMT's decision not mine.)

I believe that we probably will need to do something in this
area before v19 release.  If we're willing to commit to it being
"run the tests serially", then sure we can wait awhile before
actually doing that.  Maybe we'll even think of a better idea
... but what we can do about this post-freeze seems pretty
constrained to me.

			regards, tom lane

  192. Re: pg_plan_advice

    Lukas Fittl <lukas@fittl.com> — 2026-04-04T08:11:25Z

    On Thu, Apr 2, 2026 at 7:15 PM Robert Haas <robertmhaas@gmail.com> wrote:
>
> On Thu, Apr 2, 2026 at 12:15 PM Robert Haas <robertmhaas@gmail.com> wrote:
> > So here's v24, also dropping pg_collect_advice.
>
> That version didn't actually pass CI. Here's v25.

I've reviewed the pg_stash_advice code and documentation, and I think
this is overall sound.  As I mentioned previously, I think its a very
important addition to make pg_plan_advice work for practical problems
end users encounter.

To me this looks good to go, with three minor notes below. For context
I've spent a few hours today going through the code manually, and
doing testing.

And thank you for the detailed notes in your earlier email, and
reworking this. Regarding authorship, I'm happy to be listed as
co-author on the persistence part if you want to keep that in the
commit message. Overall I'm also willing to put in work during the
remaining cycle to test/review/address issues in pg_plan_advice or
pg_stash_advice, so we can hopefully sort out the other items being
discussed.

For 0001:

> diff --git a/doc/src/sgml/pgstashadvice.sgml b/doc/src/sgml/pgstashadvice.sgml
> new file mode 100644
> index 00000000000..ec60552a447
> --- /dev/null
> +++ b/doc/src/sgml/pgstashadvice.sgml
> @@ -0,0 +1,216 @@
> ...
> +   <varlistentry>
> +    <term>
> +     <function>pg_create_advice_stash(stash_name text) returns void</function>
> +     <indexterm>
> +      <primary>pg_create_advice_stash</primary>
> +     </indexterm>
> +    </term>
> +
> +    <listitem>
> +     <para>
> +      Creates a new, empty advice stash with the given name.
> +     </para>
> +    </listitem>
> +   </varlistentry>
> +

I think we should document the restrictions on advice names here (i.e.
they must be alphanumeric or contain an underscore, not start with a
digit, and maximum NAMEDATLEN).

For 0002:

> diff --git a/contrib/pg_stash_advice/pg_stash_advice.c b/contrib/pg_stash_advice/pg_stash_advice.c
> index 15e7adf849b..1858c6a135a 100644
> --- a/contrib/pg_stash_advice/pg_stash_advice.c
> +++ b/contrib/pg_stash_advice/pg_stash_advice.c
> ...
> @@ -464,6 +522,43 @@ pgsa_drop_stash(char *stash_name)
>         }
>     }
>     dshash_seq_term(&iterator);
> +
> +    /* Bump change count. */
> +    pg_atomic_add_fetch_u64(&pgsa_state->change_count, 1);
> +}
> +
> +/*
> + * Remove all stashes and entries from shared memory.
> + *
> + * This is intended to be called before reloading from a dump file, so that
> + * a failed previous attempt doesn't leave stale data behind.
> + */
> +void
> +pgsa_reset_all_stashes(void)
> +{

I think this might be good to expose on the SQL level as well - in
case someone accidentally created a lot of stashes it could be tedious
to remove them all, e.g. if they wanted to clear all the memory after
an experiment.

> diff --git a/contrib/pg_stash_advice/stashpersist.c b/contrib/pg_stash_advice/stashpersist.c
> new file mode 100644
> index 00000000000..da96ee0d803
> --- /dev/null
> +++ b/contrib/pg_stash_advice/stashpersist.c
>...
> +            /* Parse the query ID. */
> +            errno = 0;
> +            queryId = strtoll(queryid_str, &endptr, 10);
> +            if (*endptr != '\0' || errno != 0 || queryid_str == endptr)
> +                ereport(ERROR,
> +                        (errcode(ERRCODE_DATA_CORRUPTED),
> +                         errmsg("syntax error in file \"%s\" line %u: invalid query ID \"%s\"",
> +                                PGSA_DUMP_FILE, lineno, queryid_str)));
> +

It might be worth adding a queryId == 0 check here, since we won't
check it later, and its helpful to avoid unpredictable behavior just
in case someone decided to mess with the file manually.

Thanks,
Lukas

-- 
Lukas Fittl

  193. Re: pg_plan_advice

    Andrei Lepikhov <lepihov@gmail.com> — 2026-04-04T09:34:35Z

    On 4/4/26 05:14, Tom Lane wrote:
> Robert Haas <robertmhaas@gmail.com> writes:
> The terms that I'm thinking in are "how much redesign will we accept
> post-feature-freeze, in either pg_plan_advice or test_plan_advice,
> before choosing to revert those modules entirely for v19?".  I think
> that running those tests serially is a sufficiently low-risk option
> that it'd be okay to put it in post-freeze, even very long after.
> I'm not sure that any of the other group-1 or group-2 options you
> suggested would be okay post-freeze.  (Of course, ultimately that'd
> be the RMT's decision not mine.)
> 
> I believe that we probably will need to do something in this
> area before v19 release.  If we're willing to commit to it being
> "run the tests serially", then sure we can wait awhile before
> actually doing that.  Maybe we'll even think of a better idea
> ... but what we can do about this post-freeze seems pretty
> constrained to me.

As you work on the code, please keep the pg_plan_advice issue [1] in 
mind. I came across it while designing the optimisation in [2]. Even if 
[2] is not added to the Postgres core, this still looks like a valid 
query plan and may be proposed by an extension. So, the hinting module 
should avoid conflicts with other extensions, just as pg_hint_plan does.

[1] pg_plan_advice fails when NestLoop outer side is Sort over FunctionScan
https://www.postgresql.org/message-id/78dd9572-7569-4025-984d-e07d7f381b6e@gmail.com
[2] Try a presorted outer path when referenced by an ORDER BY prefix
https://www.postgresql.org/message-id/19a9265c-c441-4a43-bc0d-dac533438da0%40gmail.com

-- 
regards, Andrei Lepikhov,
pgEdge

  194. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-04-04T18:42:19Z

    On Sat, Apr 4, 2026 at 5:34 AM Andrei Lepikhov <lepihov@gmail.com> wrote:
> As you work on the code, please keep the pg_plan_advice issue [1] in
> mind. I came across it while designing the optimisation in [2]. Even if
> [2] is not added to the Postgres core, this still looks like a valid
> query plan and may be proposed by an extension. So, the hinting module
> should avoid conflicts with other extensions, just as pg_hint_plan does.
>
> [1] pg_plan_advice fails when NestLoop outer side is Sort over FunctionScan
> https://www.postgresql.org/message-id/78dd9572-7569-4025-984d-e07d7f381b6e@gmail.com
> [2] Try a presorted outer path when referenced by an ORDER BY prefix
> https://www.postgresql.org/message-id/19a9265c-c441-4a43-bc0d-dac533438da0%40gmail.com

I'll take a look at that issue when I have a free moment. We certainly
cannot promise in general that pg_plan_advice will be able to make
sense of plans that PostgreSQL's own planner does not produce; that
would require magical code. But there might be something that can be
done to ameliorate this particular instance.

By the way, I'm really glad you hit that error. That particular error
check is there precisely to find plans that pg_plan_advice isn't able
to understand, and it sounds like it is doing its job as intended.
Having problems isn't great, but knowing that you have problems is a
lot better than still having them but not knowing about it.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  195. Re: pg_plan_advice

    Andrei Lepikhov <lepihov@gmail.com> — 2026-04-04T21:02:37Z

    On 4/4/26 20:42, Robert Haas wrote:
> On Sat, Apr 4, 2026 at 5:34 AM Andrei Lepikhov <lepihov@gmail.com> wrote:
> By the way, I'm really glad you hit that error. That particular error
> check is there precisely to find plans that pg_plan_advice isn't able
> to understand, and it sounds like it is doing its job as intended.
> Having problems isn't great, but knowing that you have problems is a
> lot better than still having them but not knowing about it.
That’s exactly what concerns me. I see it as a potential design flaw if 
the extension has to make assumptions about possible plan configurations.
I’m not sure how it works in detail, of course. However, when I designed 
Postgres replanning in the past, and made similar core changes to what 
you’ve done for pg_plan_advice, this kind of problem couldn’t have 
happened. So, I think it’s worth questioning the current approach and 
looking for other options.

-- 
regards, Andrei Lepikhov,
pgEdge

  196. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-04-04T22:52:21Z

    On Sat, Apr 4, 2026 at 5:02 PM Andrei Lepikhov <lepihov@gmail.com> wrote:
> That’s exactly what concerns me. I see it as a potential design flaw if
> the extension has to make assumptions about possible plan configurations.
> I’m not sure how it works in detail, of course. However, when I designed
> Postgres replanning in the past, and made similar core changes to what
> you’ve done for pg_plan_advice, this kind of problem couldn’t have
> happened. So, I think it’s worth questioning the current approach and
> looking for other options.

I mean, any plan stability feature is intrinsically tied to a
particular planner. Nobody thinks you can use Aurora Postgres's Query
Plan Management feature with MySQL or DB2 or Oracle. Those products
obviously have to have their own features for plan stability. The same
is true here. There's more overlap because you're creating the plan
out of the same basic building blocks rather than an entirely
different set of things, but if you assemble them in a way that
PostgreSQL doesn't, then some things may not work. pg_plan_advice is
one of those things; the executor is another. Of course, I don't think
anybody here is keen to break stuff for no good reason, which is why I
will take a look at the report you posted. But fundamentally, it's the
same issue. If somebody uses a plugin that replaces large parts of the
plan with a CustomScan, pg_plan_advice isn't going to work with that,
either: how could it possibly? Maybe there could be some way to make
pg_plan_advice pluggable so that if extensions fiddle with the planner
they can also do matching fiddling with pg_plan_advice if they're so
inclined, but having it "just work" would require magic.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  197. Re: pg_plan_advice

    Andrei Lepikhov <lepihov@gmail.com> — 2026-04-05T07:57:13Z

    On 5/4/26 00:52, Robert Haas wrote:
> On Sat, Apr 4, 2026 at 5:02 PM Andrei Lepikhov <lepihov@gmail.com> wrote:
>> That’s exactly what concerns me. I see it as a potential design flaw if
>> the extension has to make assumptions about possible plan configurations.
>> I’m not sure how it works in detail, of course. However, when I designed
>> Postgres replanning in the past, and made similar core changes to what
>> you’ve done for pg_plan_advice, this kind of problem couldn’t have
>> happened. So, I think it’s worth questioning the current approach and
>> looking for other options.
> 
> I mean, any plan stability feature is intrinsically tied to a
> particular planner. Nobody thinks you can use Aurora Postgres's Query
> Plan Management feature with MySQL or DB2 or Oracle. Those products

I don’t expect any Postgres extension to work in DB2.

These optimisations are simple. Here, I provided the optimiser with one 
extra path that it skipped itself just to reduce computational overhead 
- nice in the general case, but not ok in analytics. This extension of 
planning scope allowed the optimiser to build JOIN over the Sort 
operator, which didn’t change the main logic at all. I followed the 
usual cost-based model and used add_path.

Another optimisation improves Memoize so it can run on top of SubPlan 
when the cost model predicts many repeated parameter values. One more 
extension uses MergeJoin estimation on the required values of its inputs 
to determine how many tuples are needed from each input, which adds 
kinda 'soft' LIMIT emerged from the plan structure ... The Append node 
serves as the backbone of any partitioning or sharding setup, but 
contributors often overlook it, and we use multiple extra optimisations 
here too.

There’s a lot to say about branched out-of-core optimisations 
infrastructure, but it’s clear that supporting analytical workloads 
means adding extra features. Developers usually stick to standard 
Postgres practices, cost model and routines providing the planner with 
alternatives without forcing any 'magical' paths. So, they expect 
built-in extensions not to interfere with their code by design.

Looking back at the pg_plan_advice development cycle, I don’t see many 
discussions about the design. It seems unusual given how complex the 
planner's structure is. It makes sense to follow the typical way and let 
it serve out of the contrib for some time and see if it works well.

Introducing such a module into the core would effectively cancel 
alternative solutions, as seen with PGSS. Therefore, it is important to 
ensure the code is well-designed before proceeding. Do you agree?

-- 
regards, Andrei Lepikhov,
pgEdge

  198. Re: pg_plan_advice

    Alexander Lakhin <exclusion@gmail.com> — 2026-04-05T08:00:00Z

    Hello Robert,

I and SQLsmith have discovered one more anomaly (reproduced starting from
e0e4c132e):
load 'test_plan_advice';
select object_type from
  (select object_type from information_schema.element_types limit 1),
  lateral
  (select sum(1) over (partition by a) from generate_series(1, 2) g(a) where false);

triggers an internal error:
ERROR:  XX000: no rtoffset for plan unnamed_subquery
LOCATION:  pgpa_plan_walker, pgpa_walker.c:110

Could you please have a look?

Best regards,
Alexander

  199. Re: pg_plan_advice

    Alexander Lakhin <exclusion@gmail.com> — 2026-04-05T12:00:00Z

    05.04.2026 11:00, Alexander Lakhin wrote:
> I and SQLsmith have discovered one more anomaly (reproduced starting from
> e0e4c132e):
> load 'test_plan_advice';
> select object_type from
>  (select object_type from information_schema.element_types limit 1),
>  lateral
>  (select sum(1) over (partition by a) from generate_series(1, 2) g(a) where false);
>
> triggers an internal error:
> ERROR:  XX000: no rtoffset for plan unnamed_subquery
> LOCATION:  pgpa_plan_walker, pgpa_walker.c:110

And another error, which might be interesting to you:
CREATE EXTENSION tsm_system_time;
CREATE TABLE t(i int);
SELECT 1 FROM (SELECT i FROM t TABLESAMPLE system_time (1000)), LATERAL (SELECT i LIMIT 1);

ERROR:  XX000: plan node has no RTIs: 378
LOCATION:  pgpa_build_scan, pgpa_scan.c:200

Best regards,
Alexander

  200. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-04-06T12:47:30Z

    On Sun, Apr 5, 2026 at 3:57 AM Andrei Lepikhov <lepihov@gmail.com> wrote:
> Looking back at the pg_plan_advice development cycle, I don’t see many
> discussions about the design. It seems unusual given how complex the
> planner's structure is. It makes sense to follow the typical way and let
> it serve out of the contrib for some time and see if it works well.
>
> Introducing such a module into the core would effectively cancel
> alternative solutions, as seen with PGSS. Therefore, it is important to
> ensure the code is well-designed before proceeding. Do you agree?

I don't know how anyone could disagree with the idea that PostgreSQL
code should be well-designed, but that doesn't mean that I agree that
your particular design criticism is fair, and I definitely don't.

As for the amount of design discussion on the mailing list, I was
disappointed in that, too. In addition to posting to the list, I
privately asked numerous people to help review and test. Some did, but
on the whole, I was expecting a more vigorous debate and a lot of
people telling me what an idiot I am. Instead, the most common
feedback I got was some form of "can you ship it right now, please?".
That probably has less to do with the design being good (although I
believe that it is) or my code being good (although I hope that it is)
than with people just really wanting PostgreSQL to have something of
this sort. So I am somewhat afraid that this will turn out to have
more problems than anyone has noticed so far, and maybe for reasons
that will feel dumb in hindsight. But on March 12th, I asked myself
whether more people were going to be unhappy if I committed
pg_plan_advice this release cycle or if I didn't, and my educated
guess was the latter, so I committed it. If that turns out to have
been the wrong call, then I apologize to the whole community in
advance.

But I do not apologize for the fact that pg_plan_advice tries to
interpret plan trees -- which I personally think is one of the best
design decisions I have ever made while hacking on PostgreSQL -- or
that it can't interpret the variant ones that your extension produces.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  201. Re: pg_plan_advice

    Andrei Lepikhov <lepihov@gmail.com> — 2026-04-06T13:22:04Z

    On 06/04/2026 14:47, Robert Haas wrote:
> On Sun, Apr 5, 2026 at 3:57 AM Andrei Lepikhov <lepihov@gmail.com> wrote:
>> Looking back at the pg_plan_advice development cycle, I don’t see many
>> discussions about the design. It seems unusual given how complex the
>> planner's structure is. It makes sense to follow the typical way and let
>> it serve out of the contrib for some time and see if it works well.
> But I do not apologize for the fact that pg_plan_advice tries to
> interpret plan trees -- which I personally think is one of the best
> design decisions I have ever made while hacking on PostgreSQL -- or
> that it can't interpret the variant ones that your extension produces.

I challenge solely the design of the extension, not interested in holy 
wars on the hinting approach.
Postgres modules that use hooks are second-class citizens because the 
core hooks were never designed to let an extension module be as 
effective as the core code. It's probably OK, considering safety and 
maintainability concerns.
But this extension effectively makes alternative modules third-class 
citizens (not sure such a term exists in English) - people prioritise 
contrib modules over any others. And they definitely will use this one. 
So, I envision complaints about conflicting extensions in the near 
future - think about Citus or TimescaleDB optimisations, for example.
It would be better to introduce such a code at the beginning of the 
development cycle, not right before the code freeze. At least we would 
discuss its design without rushing.

-- 
regards, Andrei Lepikhov,
pgEdge

  202. Re: pg_plan_advice

    Andres Freund <andres@anarazel.de> — 2026-04-06T13:56:23Z

    Hi,

On 2026-04-04 23:02:37 +0200, Andrei Lepikhov wrote:
> On 4/4/26 20:42, Robert Haas wrote:
> > On Sat, Apr 4, 2026 at 5:34 AM Andrei Lepikhov <lepihov@gmail.com> wrote:
> > By the way, I'm really glad you hit that error. That particular error
> > check is there precisely to find plans that pg_plan_advice isn't able
> > to understand, and it sounds like it is doing its job as intended.
> > Having problems isn't great, but knowing that you have problems is a
> > lot better than still having them but not knowing about it.
> That’s exactly what concerns me. I see it as a potential design flaw if the
> extension has to make assumptions about possible plan configurations.
> I’m not sure how it works in detail, of course. However, when I designed
> Postgres replanning in the past, and made similar core changes to what
> you’ve done for pg_plan_advice, this kind of problem couldn’t have happened.
> So, I think it’s worth questioning the current approach and looking for
> other options.

You're making sweeping high-level demands, implying they're easy ("when I
designed ... this kind of problem couldn’t have happened"), without any
concrete technical suggestions for how to actually achieve that.  In very
strong language.  Your high level demand, that somehow plan shape influencing
code should just work regardless of what crazy thing extensions have done
seems ... not entirely realistic, to put it very kindly.

I suggest you rethink your approach of engaging with others.

Greetings,

Andres Freund

  203. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-04-06T14:01:52Z

    On Mon, Apr 6, 2026 at 9:22 AM Andrei Lepikhov <lepihov@gmail.com> wrote:
> So, I envision complaints about conflicting extensions in the near
> future - think about Citus or TimescaleDB optimisations, for example.

Definitely possible.

> It would be better to introduce such a code at the beginning of the
> development cycle, not right before the code freeze. At least we would
> discuss its design without rushing.

Yes, the timing is not ideal. However, I posted the patch on October
30th and committed the main patch on March 12th. I think that's a
reasonable length of time to wait for people to provide feedback.
During that time, the only person who provided information on how this
will interact with out-of-core extensions was Lukas Fittl, who came to
the conclusion that the pgs_mask infrastructure will be reusable by
pg_hint_plan and will result in that module being simpler and
involving less code duplication. Other extension authors could have
provided feedback during that time as well, but none did, even after I
posted to my blog to try to raise the visibility of this project. As
far as I can tell, most extension developers don't pay much attention
to core development until after we ship a beta. Had I waited until
July to commit, I think there's a chance that it would have simply
resulted in me getting whatever feedback I'm going to get next summer
rather than this summer. At least this way, the issues will hopefully
be fresh in my mind when the feedback arrives.

Of course, you also seem to be assuming that whatever feedback I get
will be negative, and it may well be. But, there is also some tiny
possibility that I have done a good job and that people will like it.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  204. Re: pg_plan_advice

    Peter Geoghegan <pg@bowt.ie> — 2026-04-06T14:14:33Z

    On Mon, Apr 6, 2026 at 9:56 AM Andres Freund <andres@anarazel.de> wrote:
> You're making sweeping high-level demands, implying they're easy ("when I
> designed ... this kind of problem couldn’t have happened"), without any
> concrete technical suggestions for how to actually achieve that.  In very
> strong language.  Your high level demand, that somehow plan shape influencing
> code should just work regardless of what crazy thing extensions have done
> seems ... not entirely realistic, to put it very kindly.
>
> I suggest you rethink your approach of engaging with others.

+1

-- 
Peter Geoghegan

  205. Re: pg_plan_advice

    Andrei Lepikhov <lepihov@gmail.com> — 2026-04-06T15:11:15Z

    On 06/04/2026 15:56, Andres Freund wrote:
> You're making sweeping high-level demands, implying they're easy ("when I
> designed ... this kind of problem couldn’t have happened"), without any
> concrete technical suggestions for how to actually achieve that.  In very
> strong language.  Your high level demand, that somehow plan shape influencing
> code should just work regardless of what crazy thing extensions have done
> seems ... not entirely realistic, to put it very kindly.
Sorry about that.

I haven't had much practice with English. Sometimes, things I wouldn't 
normally say in technical discussions in my native language come out 
here. As well as part of the meaning definitely lost in translation.
The actual reason was to highlight that quite closely related features 
exist in the Postgres world (not only pg_hint_plan). Even if we can’t 
expose the code of enterprise forks, it worth to discuss alternative 
design ideas.

-- 
regards, Andrei Lepikhov,
pgEdge

  206. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-04-06T19:52:51Z

    On Sun, Apr 5, 2026 at 4:00 AM Alexander Lakhin <exclusion@gmail.com> wrote:
> I and SQLsmith have discovered one more anomaly (reproduced starting from
> e0e4c132e):
> load 'test_plan_advice';
> select object_type from
>   (select object_type from information_schema.element_types limit 1),
>   lateral
>   (select sum(1) over (partition by a) from generate_series(1, 2) g(a) where false);
>
> triggers an internal error:
> ERROR:  XX000: no rtoffset for plan unnamed_subquery
> LOCATION:  pgpa_plan_walker, pgpa_walker.c:110
>
> Could you please have a look?

Thanks for the report. What seems to be happening here is that the
whole query is replaced by a single Result node, since the join must
be empty. But that means that unnamed_subquery doesn't make it into
the final plan tree, and then pgpa_plan_walker() is sad about not
finding it. Normally it wouldn't care, but apparently this query
involves at least one semijoin someplace that the planner considered
converting into a regular join with one side made unique, so
pgpa_plan_walker() has an entry in sj_unique_rels and then wants to
adjust that entry for the final, flattened range table, and it can't.
I'm inclined to think that the fix is just:

-            elog(ERROR, "no rtoffset for plan %s", proot->plan_name);
+            continue;

...plus a comment update, but I want to spend some time mulling over
whether that might break anything else before I go do it.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  207. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-04-06T20:15:21Z

    On Sun, Apr 5, 2026 at 8:00 AM Alexander Lakhin <exclusion@gmail.com> wrote:
> And another error, which might be interesting to you:
> CREATE EXTENSION tsm_system_time;
> CREATE TABLE t(i int);
> SELECT 1 FROM (SELECT i FROM t TABLESAMPLE system_time (1000)), LATERAL (SELECT i LIMIT 1);
>
> ERROR:  XX000: plan node has no RTIs: 378
> LOCATION:  pgpa_build_scan, pgpa_scan.c:200

Thanks also for this report. The plan looks like this:

 Nested Loop  (cost=0.00..154.75 rows=2550 width=4)
   ->  Materialize  (cost=0.00..78.25 rows=2550 width=4)
         ->  Sample Scan on t  (cost=0.00..65.50 rows=2550 width=4)
               Sampling: system_time ('1000'::double precision)
   ->  Limit  (cost=0.00..0.01 rows=1 width=4)
         ->  Result  (cost=0.00..0.01 rows=1 width=4)

And it's unhappy because it's expecting the Materialize node to be the
RTI-bearing node. In a turn of events that will probably shock nobody
here, I also didn't quite realize that a Materialize node could get
inserted here. It's kind of a problem, too, because what if the sides
of the join were switched? Then we'd have a Nested Loop with an inner
Materialize node and would conclude that the strategy was
PGS_NESTLOOP_MATERIALIZE, when in reality it would be
PGS_NESTLOOP_PLAIN plus a Materialize node inserted at the scan level,
so the generated advice would be incorrect. I guess the fix is
probably to view a Materialize node on top of a Sample Scan for a
!repeatable_across_scans tsmhandler as part of the scan, which is kind
of annoying but probably doable. Not for the first time, I really wish
we stored an RTI set in every plan node, or (maybe more economically)
had some kind of enum in key plan nodes indicating why the node was
inserted. Right now, pg_plan_advice does a lot of reading the tea
leaves, which is great in that it avoids bloating Plan trees with
additional metadata, but a little scary in terms of being able to be
certain that one will get the right answer reliably.

I'll work on a fix.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  208. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-04-07T14:17:12Z

    On Sat, Apr 4, 2026 at 4:12 AM Lukas Fittl <lukas@fittl.com> wrote:
> I think we should document the restrictions on advice names here (i.e.
> they must be alphanumeric or contain an underscore, not start with a
> digit, and maximum NAMEDATLEN).

I committed 0001 without this change. Please feel free to propose a
clean-up patch that adds this. I wasn't certain where the best place
to add it was, or what the wording ought to be exactly.

> > +/*
> > + * Remove all stashes and entries from shared memory.
> > + *
> > + * This is intended to be called before reloading from a dump file, so that
> > + * a failed previous attempt doesn't leave stale data behind.
> > + */
> > +void
> > +pgsa_reset_all_stashes(void)
> > +{
>
> I think this might be good to expose on the SQL level as well - in
> case someone accidentally created a lot of stashes it could be tedious
> to remove them all, e.g. if they wanted to clear all the memory after
> an experiment.

From SQL, you can just do select pg_drop_advice_stash(stash_name) from
pg_get_advice_stashes(), which seems good enough to me. If there's a
compelling reason to have more than that, we can think about it, but I
don't particularly see one, and having fewer exposed entrypoints is
better, ceteris paribus. One disadvantage of that approach is that if
it fails due to some concurrency issue, then you might end up with
some stashes dropped and others not, but I don't see this as being
such a frequent operation that it should really cause an issue.

> > diff --git a/contrib/pg_stash_advice/stashpersist.c b/contrib/pg_stash_advice/stashpersist.c
> > new file mode 100644
> > index 00000000000..da96ee0d803
> > --- /dev/null
> > +++ b/contrib/pg_stash_advice/stashpersist.c
> >...
> > +            /* Parse the query ID. */
> > +            errno = 0;
> > +            queryId = strtoll(queryid_str, &endptr, 10);
> > +            if (*endptr != '\0' || errno != 0 || queryid_str == endptr)
> > +                ereport(ERROR,
> > +                        (errcode(ERRCODE_DATA_CORRUPTED),
> > +                         errmsg("syntax error in file \"%s\" line %u: invalid query ID \"%s\"",
> > +                                PGSA_DUMP_FILE, lineno, queryid_str)));
> > +
>
> It might be worth adding a queryId == 0 check here, since we won't
> check it later, and its helpful to avoid unpredictable behavior just
> in case someone decided to mess with the file manually.

Good point. Added that and committed 0002. I also changed the
write-to-the-persist file path to take only shared locks instead of
exclusive, since the latter seems unnecessary.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  209. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-04-07T21:21:19Z

    On Fri, Apr 3, 2026 at 11:14 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> I believe that we probably will need to do something in this
> area before v19 release.  If we're willing to commit to it being
> "run the tests serially", then sure we can wait awhile before
> actually doing that.  Maybe we'll even think of a better idea
> ... but what we can do about this post-freeze seems pretty
> constrained to me.

Here's a new patch set. All of these patches are new, but I'm
continuing to increment the same version number sequence.

0001 and 0002 implement the "retry a few times" idea for avoiding
test_plan_advice failures. I argue that (a) these are reasonable
post-commit stabilization that should not be blocked by feature freeze
and (b) most people here will be happier with a solution like this
that will normally cost very little than they will be with switching
test_plan_advice to executing serially. The RMT can decide whether it
agrees. The other question here is whether it's really a good idea to
apply this now considering that we've seen only one failure so far. I
think it's probably a good idea to do something like this before
release, so that we hopefully reduce the false positive rate from the
test to something much closer to zero, but I think we've still had
only the one failure, and I'm really interested in knowing how close
the failure rate is to zero already. The RMT may have an opinion on
how long to wait before doing something like this, too.

0003 fixes the problem with tablesample scans that Alexander Lakhin
reported. The bug occurs when a tablesample handler does not set
repeatable_across_scans and the resulting Sample Scan appears below a
join. The test case provided by Alexander shows the Sample Scan on the
inner side of the join, but it's also possible to construct a case
where it occurs on the outer side of the join. This commit adds tests
for both cases.

0004 fixes an oversight in commit
6455e55b0da47255f332a96f005ba0dd1c7176c2, which failed to add a new
pg_regress test to the pg_plan_advice Makefile.

0005 fixes the other issue that Alexander Lakhin recently reported,
which manifested as ERROR:  no rtoffset for plan unnamed_subquery when
trying to generate advice. That turns out to occur when a subquery is
proven empty and that subquery contains a semijoin that could have
been implemented by making one side unique. I chose a different fix
than what I mentioned in my response to Alexander's email. There was
already code that handles the case where a SubPlanRTInfo exists and is
marked dummy, and this fix extends that handling to the case where no
SubPlanRTInfo exists at all, which seems better than treating those
two cases in separate parts of the code.

I don't think that anyone will argue that 0003-0005 are things we
can't or shouldn't fix after feature freeze, and I plan to apply those
fixes shortly after feature freeze, unless there are objections or
better ideas. I could rush them in before that, too, but I don't think
what the tree needs are more people trying to commit all at once right
now.

Thanks,

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  210. Re: pg_plan_advice

    Tom Lane <tgl@sss.pgh.pa.us> — 2026-04-07T22:05:47Z

    Robert Haas <robertmhaas@gmail.com> writes:
> 0001 and 0002 implement the "retry a few times" idea for avoiding
> test_plan_advice failures. I argue that (a) these are reasonable
> post-commit stabilization that should not be blocked by feature freeze
> and (b) most people here will be happier with a solution like this
> that will normally cost very little than they will be with switching
> test_plan_advice to executing serially. The RMT can decide whether it
> agrees.

I'm not on the RMT, but I agree this is a nicer solution.
(I didn't read these patches in detail, but in a quick once-over
they seemed plausible.)

> The other question here is whether it's really a good idea to
> apply this now considering that we've seen only one failure so far. I
> think it's probably a good idea to do something like this before
> release, so that we hopefully reduce the false positive rate from the
> test to something much closer to zero, but I think we've still had
> only the one failure, and I'm really interested in knowing how close
> the failure rate is to zero already. The RMT may have an opinion on
> how long to wait before doing something like this, too.

No strong opinion about that.  Certainly waiting a couple of weeks
to gather more data seems reasonable.

			regards, tom lane

  211. Re: pg_plan_advice

    Nathan Bossart <nathandbossart@gmail.com> — 2026-04-08T14:49:01Z

    On Tue, Apr 07, 2026 at 06:05:47PM -0400, Tom Lane wrote:
> Robert Haas <robertmhaas@gmail.com> writes:
>> 0001 and 0002 implement the "retry a few times" idea for avoiding
>> test_plan_advice failures. I argue that (a) these are reasonable
>> post-commit stabilization that should not be blocked by feature freeze
>> and (b) most people here will be happier with a solution like this
>> that will normally cost very little than they will be with switching
>> test_plan_advice to executing serially. The RMT can decide whether it
>> agrees.
> 
> I'm not on the RMT, but I agree this is a nicer solution.
> (I didn't read these patches in detail, but in a quick once-over
> they seemed plausible.)
> 
>> The other question here is whether it's really a good idea to
>> apply this now considering that we've seen only one failure so far. I
>> think it's probably a good idea to do something like this before
>> release, so that we hopefully reduce the false positive rate from the
>> test to something much closer to zero, but I think we've still had
>> only the one failure, and I'm really interested in knowing how close
>> the failure rate is to zero already. The RMT may have an opinion on
>> how long to wait before doing something like this, too.
> 
> No strong opinion about that.  Certainly waiting a couple of weeks
> to gather more data seems reasonable.

I am only 1/3 of the RMT, but I am fine with the plan as stated.

-- 
nathan

  212. Re: pg_plan_advice

    Melanie Plageman <melanieplageman@gmail.com> — 2026-04-08T16:18:11Z

    On Wed, Apr 8, 2026 at 10:49 AM Nathan Bossart <nathandbossart@gmail.com> wrote:
>
> On Tue, Apr 07, 2026 at 06:05:47PM -0400, Tom Lane wrote:
> > Robert Haas <robertmhaas@gmail.com> writes:
> >
> >> The other question here is whether it's really a good idea to
> >> apply this now considering that we've seen only one failure so far. I
> >> think it's probably a good idea to do something like this before
> >> release, so that we hopefully reduce the false positive rate from the
> >> test to something much closer to zero, but I think we've still had
> >> only the one failure, and I'm really interested in knowing how close
> >> the failure rate is to zero already. The RMT may have an opinion on
> >> how long to wait before doing something like this, too.
> >
> > No strong opinion about that.  Certainly waiting a couple of weeks
> > to gather more data seems reasonable.
>
> I am only 1/3 of the RMT, but I am fine with the plan as stated.

I agree with waiting a few weeks to continue catching bugs.

As for 0001/0002 and the retry approach: if that's the best way to
avoid spurious test failures, I'm fine with it. I haven't reviewed the
code in detail and don't have an alternative to suggest. I'm
definitely against running anything serially.

As for the other ideas and suggestions so far:

I don't see a way to split up the regression test suite that wouldn't
make it harder to figure out where to add tests in the future. The
whole point is to avoid regressing pg_plan_advice when new things are
added to the planner, and that works because people don't have to
think about a pg_plan_advice -- their new test queries automatically
get coverage.

I do think there needs to be a way to run this in CI, but it doesn't
have to be on by default.

For the buildfarm, I don't have a strong opinion about whether to
limit it to some animals or some runs. Running on only some animals is
easier to reason about when you see a failure (i.e. that animal runs
with test_plan_advice, so it might be that), but running it once a day
or once a week on all animals gives broader coverage. That said, the
kind of coverage you gain from timing differences across animals --
catching races and transient issues -- may be less relevant for
test_plan_advice than for other tests.

- Melanie

  213. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-04-13T16:01:25Z

    On Wed, Apr 8, 2026 at 12:18 PM Melanie Plageman
<melanieplageman@gmail.com> wrote:
> > > No strong opinion about that.  Certainly waiting a couple of weeks
> > > to gather more data seems reasonable.
> >
> > I am only 1/3 of the RMT, but I am fine with the plan as stated.
>
> I agree with waiting a few weeks to continue catching bugs.

Sounds like we have a consensus. I have committed the three bug-fix
patches (unrelated to the retry-loop stuff) plus the preparatory
refactoring patch for the retry-loop patch. That renames a few
identifiers, so it seemed best to get it out of the way sooner rather
than later. I'll hold off on the main retry-loop patch for now. So far
I haven't seen any other buildfarm failures that look related to this
issue, so either I've missed some (which is certainly possible) or the
chances of failure are very low.

Thanks,

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  214. Re: pg_plan_advice

    Alexander Lakhin <exclusion@gmail.com> — 2026-04-14T18:00:01Z

    13.04.2026 19:01, Robert Haas wrote:
> Sounds like we have a consensus. I have committed the three bug-fix
> patches (unrelated to the retry-loop stuff) ...

Thanks again for committing these fixes, Robert! With all the fixes in
place, I and SQLsmith have reached another error:
CREATE TABLE t1(a int);
CREATE TABLE t2(b int);

SELECT 1 FROM t1 WHERE EXISTS
   (SELECT 1 FROM
     (SELECT 1 FROM
       (SELECT 1) LEFT JOIN t2 ON true),
     t2 WHERE a = b);

ERROR:  XX000: unique semijoin found for relids (b 3 5 7) but not observed during planning
LOCATION:  pgpa_plan_walker, pgpa_walker.c:153

Could you please have a look?

Best regards,
Alexander

  215. Re: pg_plan_advice

    Tender Wang <tndrwang@gmail.com> — 2026-04-15T10:30:01Z

    Alexander Lakhin <exclusion@gmail.com> 于2026年4月15日周三 02:00写道：
>
> 13.04.2026 19:01, Robert Haas wrote:
>
> Sounds like we have a consensus. I have committed the three bug-fix
> patches (unrelated to the retry-loop stuff) ...
>
>
> Thanks again for committing these fixes, Robert! With all the fixes in
> place, I and SQLsmith have reached another error:
> CREATE TABLE t1(a int);
> CREATE TABLE t2(b int);
>
> SELECT 1 FROM t1 WHERE EXISTS
>   (SELECT 1 FROM
>     (SELECT 1 FROM
>       (SELECT 1) LEFT JOIN t2 ON true),
>     t2 WHERE a = b);
>
> ERROR:  XX000: unique semijoin found for relids (b 3 5 7) but not observed during planning
> LOCATION:  pgpa_plan_walker, pgpa_walker.c:153
>
> Could you please have a look?
>
I did some research, and  the sj_unique_rtis contains {3,5,6,7}.
You can see that 6 is in the set. How 6 is added into the uniquerel->relids.
After deconstruct_jointree(), the joinlist is as follow:
(gdb) call nodeToString(joinlist)
$1 = 0x1ef1238 "({RANGETBLREF :rtindex 1} {RANGETBLREF :rtindex 7}
{RANGETBLREF :rtindex 5} {RANGETBLREF :rtindex 3})"
You can see that no 6 in the list.
The 6 is added when processing (7, 5), in make_join_rel(), we have below logic:

   /*
    * Add outer join relid(s) to form the canonical relids.  Any added outer
    * joins besides sjinfo itself are appended to pushed_down_joins.
    */
   joinrelids = add_outer_joins_to_relids(root, joinrelids, sjinfo,
                                 &pushed_down_joins);

In this case, 6 was added to the joinrelids.
When processing {1}, {3,5,6,7}, the {3,5,6,7} is the uniquerel, so in
the pgpa_join_path_setup(),
the {3,5,6,7} was appended to proot->sj_unique_rels.

In the plan_showdown phase, in pgpa_qf_add_plan_rtis(), we can add 7,
5, and 3 to qf->relids.
It seems difficult to add "6" to qf->relids when walking through the
plan tree.(Maybe have an easy way, I don't know too much
pg_plan_advice related code).

-- 
Thanks,
Tender Wang

  216. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-04-15T19:46:58Z

    On Wed, Apr 15, 2026 at 6:30 AM Tender Wang <tndrwang@gmail.com> wrote:
> In the plan_showdown phase, in pgpa_qf_add_plan_rtis(), we can add 7,
> 5, and 3 to qf->relids.
> It seems difficult to add "6" to qf->relids when walking through the
> plan tree.(Maybe have an easy way, I don't know too much
> pg_plan_advice related code).

Thanks for looking through this.  sj_unique_rtis is actually not set
from the plan tree walk, but based on the calls to
pgpa_join_path_setup that occur during planning, so it makes sense
that the join RTI crept in there. I'm guessing that this is another
place that needs a call to pgpa_filter_out_join_relids -- I've had a
few of those bugs already.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  217. Re: pg_plan_advice

    Tender Wang <tndrwang@gmail.com> — 2026-04-16T01:45:05Z

    Robert Haas <robertmhaas@gmail.com> 于2026年4月16日周四 03:47写道：
>
> On Wed, Apr 15, 2026 at 6:30 AM Tender Wang <tndrwang@gmail.com> wrote:
> > In the plan_showdown phase, in pgpa_qf_add_plan_rtis(), we can add 7,
> > 5, and 3 to qf->relids.
> > It seems difficult to add "6" to qf->relids when walking through the
> > plan tree.(Maybe have an easy way, I don't know too much
> > pg_plan_advice related code).
>
> Thanks for looking through this.  sj_unique_rtis is actually not set
> from the plan tree walk, but based on the calls to
> pgpa_join_path_setup that occur during planning, so it makes sense
> that the join RTI crept in there. I'm guessing that this is another
> place that needs a call to pgpa_filter_out_join_relids -- I've had a
> few of those bugs already.
I try a quick fix as follow:
diff --git a/contrib/pg_plan_advice/pgpa_planner.c
b/contrib/pg_plan_advice/pgpa_planner.c
index 72ef3230abc..971f301e950 100644
--- a/contrib/pg_plan_advice/pgpa_planner.c
+++ b/contrib/pg_plan_advice/pgpa_planner.c
@@ -541,6 +541,7 @@ pgpa_join_path_setup(PlannerInfo *root, RelOptInfo *joinrel,
        {
                pgpa_planner_state *pps;
                RelOptInfo *uniquerel;
+               Bitmapset *relids;

                uniquerel = jointype == JOIN_UNIQUE_OUTER ? outerrel : innerrel;
                pps = GetPlannerGlobalExtensionState(root->glob,
planner_extension_id);
@@ -562,8 +563,11 @@ pgpa_join_path_setup(PlannerInfo *root,
RelOptInfo *joinrel,
                        oldcontext = MemoryContextSwitchTo(pps->mcxt);
                        proot = pgpa_planner_get_proot(pps, root);
                        if (!list_member(proot->sj_unique_rels,
uniquerel->relids))
+                       {
+                               relids =
pgpa_filter_out_join_relids(uniquerel->relids, root->parse->rtable);
                                proot->sj_unique_rels =
lappend(proot->sj_unique_rels,
-
                         bms_copy(uniquerel->relids));
+
                         bms_copy(relids));
+                       }
                        MemoryContextSwitchTo(oldcontext);
                }
        }

postgres=# LOAD 'pg_plan_advice';
LOAD
postgres=# EXPLAIN (COSTS OFF, PLAN_ADVICE)SELECT 1 FROM t1 WHERE EXISTS
   (SELECT 1 FROM
     (SELECT 1 FROM
       (SELECT 1) LEFT JOIN t2 ON true),
     t2 WHERE a = b);
                    QUERY PLAN
---------------------------------------------------
 Hash Join
   Hash Cond: (t1.a = t2.b)
   ->  Seq Scan on t1
   ->  Hash
         ->  HashAggregate
               Group Key: t2.b
               ->  Nested Loop
                     ->  Nested Loop Left Join
                           ->  Result
                           ->  Seq Scan on t2 t2_1
                     ->  Materialize
                           ->  Seq Scan on t2
 Generated Plan Advice:
   JOIN_ORDER(t1 ("*RESULT*" t2#2 t2))
   NESTED_LOOP_PLAIN(t2#2)
   NESTED_LOOP_MATERIALIZE(t2)
   HASH_JOIN((t2 t2#2 "*RESULT*"))
   SEQ_SCAN(t1 t2#2 t2)
   SEMIJOIN_UNIQUE((t2 t2#2 "*RESULT*"))
   NO_GATHER(t1 t2 t2#2 "*RESULT*")
(20 rows)




-- 
Thanks,
Tender Wang

  218. Re: pg_plan_advice

    Robert Haas <robertmhaas@gmail.com> — 2026-04-17T19:00:05Z

    On Wed, Apr 15, 2026 at 9:45 PM Tender Wang <tndrwang@gmail.com> wrote:
> I try a quick fix as follow:

Thanks, but that's not quite correct: it filters out the unique relids
only after testing the list, and also copies the list an extra time
unnecessarily. I've pushed a fix that I believe to be correct, with a
test case.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  219. Re: pg_plan_advice

    Michael Paquier <michael@paquier.xyz> — 2026-06-10T00:21:25Z

    On Mon, Apr 06, 2026 at 10:01:52AM -0400, Robert Haas wrote:
> On Mon, Apr 6, 2026 at 9:22 AM Andrei Lepikhov <lepihov@gmail.com> wrote:
>> It would be better to introduce such a code at the beginning of the
>> development cycle, not right before the code freeze. At least we would
>> discuss its design without rushing.
>
> Yes, the timing is not ideal. However, I posted the patch on October
> 30th and committed the main patch on March 12th. I think that's a
> reasonable length of time to wait for people to provide feedback.

(Speaking with the pg_hint_plan kind-of-maintainer hat on.)
The timing is fine IMO.  In terms of integration with new APIs of
upstream, there is really nothing one can do until we are at least in
feature freeze.  Trying to work around APIs that have been committed
in the tree, which may be tuned after the initial commit, is just a
loss of time.  Things may get adjusted during beta, but the waves are
much weaker to deal with.

> During that time, the only person who provided information on how this
> will interact with out-of-core extensions was Lukas Fittl, who came to
> the conclusion that the pgs_mask infrastructure will be reusable by
> pg_hint_plan and will result in that module being simpler and
> involving less code duplication. Other extension authors could have
> provided feedback during that time as well, but none did, even after I
> posted to my blog to try to raise the visibility of this project. As
> far as I can tell, most extension developers don't pay much attention
> to core development until after we ship a beta. Had I waited until
> July to commit, I think there's a chance that it would have simply
> resulted in me getting whatever feedback I'm going to get next summer
> rather than this summer. At least this way, the issues will hopefully
> be fresh in my mind when the feedback arrives.

I have an answer to this one, in the shape of the following commits in
pg_hint_plan:
https://github.com/ossc-db/pg_hint_plan/commit/e42246a82589001de2f08255d3b4d984fb134d38
https://github.com/ossc-db/pg_hint_plan/commit/75b3d0142d2a8ea0e3d656e1c95ea3fdd6e8f082
https://github.com/ossc-db/pg_hint_plan/commit/5d386d3ecb832d3ea205d1e42e305cafefbefc76

The first commit is the most relevant one, and on a number basis I
finish with that, where I have been able to basically remove *all* the
historical hacks of the model in terms of plugs it added in the
planner:
 17 files changed, 1486 insertions(+), 3820 deletions(-)

At the end, I am particularly happy with the way things are regarding
the new join_path_setup and joinrel_setup hooks, that have removed
most of the bloat.

One thing that has caused me quite a bit of headache was parallel
hints.  At the end, I have followed Lukas suggestion to remove the old
path regeneration logic that was based on an enforcement of the GUCs
and switched to the PGS logic.  This is coming with some breakages in
the module, but these are actually super minor compared to the
accumulation of weird historical behavior that we had in it:
- When specifying only a JOIN hint (without leading), we now let the
planner decide the inner/outer order depending on the cost it sees,
not the order of the clauses.  That can always be enforced with a
Leading hint, which is the same thing as the JOIN_ORDER hint in
pg_plan_advice.
- Some slight changes in the way parallel hints are propagated to
child relations, due to build_simple_rel_hook().  We cannot really
avoid that, both behaviors are debatable, edge enough that I don't
worry much in terms of plan instabilities after a major release.  We
have some degree of that for each major release, users care *a lot*
about plan stability across minor releases, work around these after
major upgrades.

I strongly suspect that all these things are just going to be noise.
The regression test suite has basically no changes.

There are still gaps between pg_plan_advice and pg_hint_plan, and the
maintenance of the latter is now muuuuuch easier (still need to
maintain some versions for the stable branches).  The end game for me
would be to close the gap and merge both things together, then drop
pg_hint_plan.  I'll try to find some victi^D^D^D^D^D resources to do
some of the leg work to do the gap here (not planning to do that
myself), for some patches to-be-proposed in v20.  We have row hints,
parallel worker hints (aka RelOptInfo.rel_parallel_workers), memoize
hints, SET hints that could be added to contrib/pg_plan_advice.  There
are also hints that negate scan behaviors.  The negation hints are
not that popular, I think, but that may worth considering.

As of today on HEAD, 60%-ish of the remaining code relates to the
custom hint string parsing and feeding into the various hint
structures.  40%-ish of the code comes from the hooks and the internal
routines used by the hooks, for something like 5k lines of code.  This
is a difference between night and day.  (There may be more
simplifications doable in the code, planning an extra round of checks
during beta.)

In short, thanks for the work you have done in the v19 release cycle
in this area, Robert and others.
--
Michael