Horton

Re: Partition-wise join for join between (declaratively) partitioned tables

Robert Haas <robertmhaas@gmail.com>

View thread View on postgr.es
From: Robert Haas <robertmhaas@gmail.com>
To: Ashutosh Bapat <ashutosh.bapat@enterprisedb.com>
Cc: Amit Langote <Langote_Amit_f8@lab.ntt.co.jp>, Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp>, Thomas Munro <thomas.munro@enterprisedb.com>, Rafia Sabih <rafia.sabih@enterprisedb.com>, Rajkumar Raghuwanshi <rajkumar.raghuwanshi@enterprisedb.com>, pgsql-hackers <pgsql-hackers@postgresql.org>
Date: 2017-10-06T19:34:05Z
Lists: pgsql-hackers 
On Fri, Oct 6, 2017 at 3:07 PM, Ashutosh Bapat
<ashutosh.bapat@enterprisedb.com> wrote:
> On Fri, Oct 6, 2017 at 8:45 PM, Robert Haas <robertmhaas@gmail.com> wrote:
>> I think this is very good work and I'm excited about the feature.  Now
>> I'll wait to see whether the buildfarm, or Tom, yell at me for
>> whatever problems this may still have...
>
> Buildfarm animal prion turned red. Before going into that failure,
> good news is that the other animals are green. So the plans are
> stable.
>
> prion runs the regression with -DRELCACHE_FORCE_RELEASE, which
> destroys a relcache entry as soon as its reference count drops down to
> 0. This destroys everything that's there in corresponding relcache
> entry including partition key information and partition descriptor
> information. find_partition_scheme() and set_relation_partition_info()
> both assume that the relcache information will survive as long as the
> relation lock is held. They do not copy the relevant partitioning
> information but just copy the pointers. That assumption is wrong.
> Because of -DRELCACHE_FORCE_RELEASE, as soon as refcount drops to
> zero, the data in partition scheme and partition bounds goes invalid
> and various checks to see if partition wise join is possible fail.
> That causes partition_join test to fail on prion. But I think, the bug
> could cause crash as well.
>
> The fix is to copy the relevant partitioning information from relcache
> into PartitionSchemeData and RelOptInfo. Here's a quick patch with
> that fix.

Committed.  I hope that makes things less red rather than more,
because I'm going to be AFK for a few hours anyway.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Commits

  1. Add test for partitionwise join involving default partition.

  2. Rewrite the code that applies scan/join targets to paths.

  3. Fix code related to partitioning schemes for dropped columns.

  4. Copy information from the relcache instead of pointing to it.

  5. Basic partition-wise join functionality.

  6. Associate partitioning information with each RelOptInfo.

  7. Expand partitioned table RTEs level by level, without flattening.

  8. Set partitioned_rels appropriately when UNION ALL is used.

  9. Remove dedicated B-tree root-split record types.

  10. Assorted preparatory refactoring for partition-wise join.

  11. Teach adjust_appendrel_attrs(_multilevel) to do multiple translations.

  12. Avoid unnecessary single-child Append nodes.

  13. Revisit handling of UNION ALL subqueries with non-Var output columns.