Horton

Thread

Commits

  1. Propagate lateral-reference information to indirect descendant relations.

  2. Expand partitioned table RTEs level by level, without flattening.

  1. Bogus lateral-reference-propagation logic in create_lateral_join_info

    Tom Lane <tgl@sss.pgh.pa.us> — 2019-02-05T21:56:52Z

    While poking at bug #15613 (in which FDWs are failing to mark created
Paths with correct outer-reference sets), I thought it'd be a good idea
to add some asserts to Path creation saying that every Path should be
parameterized by at least whatever the relation's required LATERAL
references are.  I did that as per the added assertions in relnode.c
below.  I didn't expect any failures in the existing regression tests,
since we know those don't exercise bug #15613, but darn if the addition
to get_appendrel_parampathinfo didn't blow up in the core tests.

A bit of excavation later, it turns out that this is a bug since day 1
in create_lateral_join_info.  It needs to propagate lateral_relids
and related fields from appendrel parents to children, but it was not,
in the original code, accounting for the possibility of grandchildren.
Those need to get the lateral fields propagated from their topmost
ancestor too, but they weren't.  This leads to having an intermediate
appendrel that is marked with some lateral_relids when its children
are not, causing add_paths_to_append_rel to believe that unparameterized
paths can be built, triggering the new assertion.  I'm not sure if there
are any worse consequences; the regression test case that's triggering
the Assert seems to work otherwise, so we might be accidentally failing
to fail.  But it's not supposed to be like that.

Commit 0a480502b hacked this code up to deal with grandchildren for
the case of partitioned tables, but the regression test that's falling
over involves nested UNION ALL subqueries, which are not that.  Rather
than add another RTE-kind exception, though, I think we ought to rewrite
it completely and get rid of the nested loops in favor of one traversal
of the append_rel_list.  This does require assuming that the
append_rel_list has ancestor entries before descendant entries, but
that's okay because of the way the list is built.  (I note that 0a480502b
is effectively assuming that ancestors appear before children in the RTE
list, which is no safer an assumption.)

Also, I'd really like to know why I had to put in the exception seen
in the loop for AppendRelInfos that do not point to a valid parent.
It seems to me that that is almost certainly working around a bug in
the partitioning logic.  (Without it, the partition_prune regression test
crashes.)  Or would somebody like to own up to having created that state
of affairs intentionally?  If so why?

Anyway, I propose to commit and back-patch the initsplan.c part of the
attached.  The added asserts in relnode.c should probably not go in
until we have a bug #15613 fix that will prevent postgres_fdw from
triggering them, so I'll do that later.

			regards, tom lane

  2. Re: Bogus lateral-reference-propagation logic in create_lateral_join_info

    David Rowley <david.rowley@2ndquadrant.com> — 2019-02-06T01:26:30Z

    On Wed, 6 Feb 2019 at 10:57, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Also, I'd really like to know why I had to put in the exception seen
> in the loop for AppendRelInfos that do not point to a valid parent.
> It seems to me that that is almost certainly working around a bug in
> the partitioning logic.  (Without it, the partition_prune regression test
> crashes.)  Or would somebody like to own up to having created that state
> of affairs intentionally?  If so why?

Sounds pretty strange to me.   What exactly do you mean by not
pointing to a valid parent? Do you mean the parent_relid index is not
a valid RelOptInfo?

Can you point to the regression test that's doing this?

-- 
 David Rowley                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

  3. Re: Bogus lateral-reference-propagation logic in create_lateral_join_info

    Tom Lane <tgl@sss.pgh.pa.us> — 2019-02-06T03:11:53Z

    David Rowley <david.rowley@2ndquadrant.com> writes:
> On Wed, 6 Feb 2019 at 10:57, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> Also, I'd really like to know why I had to put in the exception seen
>> in the loop for AppendRelInfos that do not point to a valid parent.
>> It seems to me that that is almost certainly working around a bug in
>> the partitioning logic.  (Without it, the partition_prune regression test
>> crashes.)  Or would somebody like to own up to having created that state
>> of affairs intentionally?  If so why?

> Sounds pretty strange to me.   What exactly do you mean by not
> pointing to a valid parent? Do you mean the parent_relid index is not
> a valid RelOptInfo?

Exactly --- the parent_relid index does not have a RelOptInfo in
simple_rel_array[].

> Can you point to the regression test that's doing this?

Yeah, I misspoke above, it's in partition_join not partition_prune,
specifically

DELETE FROM prt1_l
WHERE EXISTS (
  SELECT 1
    FROM int4_tbl,
         LATERAL (SELECT int4_tbl.f1 FROM int8_tbl LIMIT 2) ss
    WHERE prt1_l.c IS NULL);

I didn't run this totally to bottom yet, but what seems to be
happening is that inheritance_planner is creating a partition-specific
subplan for the DELETE, and it's allowing AppendRelInfos from the
parent query to propagate into the subquery even though they have
nothing to do with that subquery.

We could just decide that it's okay for code dealing with the subquery
to ignore the irrelevant AppendRelInfos (which is basically what my
draft patch did), but I find that to be an uncomfortable answer: it
seems *way* too likely to result in code that can mask real bugs.
I'd be happier fixing things so that inheritance_planner doesn't
propagate anything into the subquery that doesn't make sense in the
subquery's context.  But perhaps that's unreasonably hard?  Not enough
data yet.

			regards, tom lane

  4. Re: Bogus lateral-reference-propagation logic in create_lateral_join_info

    Amit Langote <langote_amit_f8@lab.ntt.co.jp> — 2019-02-06T09:59:27Z

    On 2019/02/06 12:11, Tom Lane wrote:
> Yeah, I misspoke above, it's in partition_join not partition_prune,
> specifically
> 
> DELETE FROM prt1_l
> WHERE EXISTS (
>   SELECT 1
>     FROM int4_tbl,
>          LATERAL (SELECT int4_tbl.f1 FROM int8_tbl LIMIT 2) ss
>     WHERE prt1_l.c IS NULL);
> 
> I didn't run this totally to bottom yet, but what seems to be
> happening is that inheritance_planner is creating a partition-specific
> subplan for the DELETE, and it's allowing AppendRelInfos from the
> parent query to propagate into the subquery even though they have
> nothing to do with that subquery.
>
> We could just decide that it's okay for code dealing with the subquery
> to ignore the irrelevant AppendRelInfos (which is basically what my
> draft patch did), but I find that to be an uncomfortable answer: it
> seems *way* too likely to result in code that can mask real bugs.
> I'd be happier fixing things so that inheritance_planner doesn't
> propagate anything into the subquery that doesn't make sense in the
> subquery's context.  But perhaps that's unreasonably hard?  Not enough
> data yet.

The target-relation specific entries in the append_rel_list of the
original PlannerInfo are *only* for inheritance_planner to use, so it
seems OK to ignore them during child target planning in any way possible,
which in your patch's case is by ignoring the AppendRelInfos whose
parent_relid fetches a NULL base rel.  The reason for them to be NULL, as
might be clear to you, is that reference to the parent target relation in
the child query's jointree gets replaced by the reference to child target
relation.  Maybe we could document that in the comment, instead of this
done by your patch:

+         * Apparently append_rel_list can contain bogus parent rels nowadays
+         *
+        if (parentrel == NULL)
+            continue;*/

Note that a RelOptInfo is never built for the *original* target relation
of the query in the inheritance case (only children, including parent in
its role as child in the regular inheritance case), so there's no
possibility of LATERAL info (or anything that's initialized by
query_planner for that matter) ever being associated with that relation.

Thanks,
Amit

  5. Re: Bogus lateral-reference-propagation logic in create_lateral_join_info

    Tom Lane <tgl@sss.pgh.pa.us> — 2019-02-06T16:23:22Z

    Amit Langote <Langote_Amit_f8@lab.ntt.co.jp> writes:
> On 2019/02/06 12:11, Tom Lane wrote:
>> I didn't run this totally to bottom yet, but what seems to be
>> happening is that inheritance_planner is creating a partition-specific
>> subplan for the DELETE, and it's allowing AppendRelInfos from the
>> parent query to propagate into the subquery even though they have
>> nothing to do with that subquery.
>> 
>> We could just decide that it's okay for code dealing with the subquery
>> to ignore the irrelevant AppendRelInfos (which is basically what my
>> draft patch did), but I find that to be an uncomfortable answer: it
>> seems *way* too likely to result in code that can mask real bugs.
>> I'd be happier fixing things so that inheritance_planner doesn't
>> propagate anything into the subquery that doesn't make sense in the
>> subquery's context.  But perhaps that's unreasonably hard?  Not enough
>> data yet.

> The target-relation specific entries in the append_rel_list of the
> original PlannerInfo are *only* for inheritance_planner to use, so it
> seems OK to ignore them during child target planning in any way possible,
> which in your patch's case is by ignoring the AppendRelInfos whose
> parent_relid fetches a NULL base rel.

I experimented with having inheritance_planner remove AppendRelInfos that
aren't relevant to the current child query.  While it's quite easy to get
rid of everything at or above the current child, as per the attached,
that's not enough to stop initsplan.c from seeing irrelevant entries:
there might still be some linking siblings of the current child to their
children, or descendants of those.  So I think we'll have to put up with
using a test-for-NULL hack in create_lateral_join_info.  I'll adjust the
comment to explain why we need it.

I'm posting the attached mainly because I'm wondering if we should
apply it despite it not being able to remove every irrelevant entry.
In many cases (particularly with wide partition trees) it'd greatly
reduce the length of the append_rel_list passed down to each
subquery, and maybe that'd save enough cycles to make it worth doing
just on performance grounds.  I've not attempted to measure though.

			regards, tom lane