Horton

Re: speeding up planning with partitions

Amit Langote <langote_amit_f8@lab.ntt.co.jp>

View thread View on postgr.es
From: Amit Langote <Langote_Amit_f8@lab.ntt.co.jp>
To: jesper.pedersen@redhat.com
Cc: "Imai, Yoshikazu" <imai.yoshikazu@jp.fujitsu.com>, "'Amit Langote'" <amitlangote09@gmail.com>, Tom Lane <tgl@sss.pgh.pa.us>, David Rowley <david.rowley@2ndquadrant.com>, Alvaro Herrera <alvherre@2ndquadrant.com>, Justin Pryzby <pryzby@telsasoft.com>, Pg Hackers <pgsql-hackers@postgresql.org>
Date: 2019-03-06T02:00:03Z
Lists: pgsql-hackers 
On 2019/03/06 0:57, Jesper Pedersen wrote:
> On 3/5/19 5:24 AM, Amit Langote wrote:
>> Attached an updated version.  This incorporates fixes for both Jesper's
>> and Imai-san's review.  I haven't been able to pin down the bug (or
>> whatever) that makes throughput go down as the partition count increases,
>> when tested with a --enable-cassert build.
>>
> 
> Thanks !
> 
> I'm seeing the throughput going down as well, but are you sure it isn't
> just the extra calls of MemoryContextCheck you are seeing ? A flamegraph
> diff highlights that area -- sent offline.
> 
> A non cassert build shows the same profile for 64 and 1024 partitions.

Thanks for testing.

I now see what's happening here.  I was aware that it's MemoryContextCheck
overhead but didn't quite understand why the time it takes should increase
with the number of partitions increasing, especially given that the patch
makes it so that only one partition is accessed if that's what the query
needs to do.  What I had forgotten however is that MemoryContextCheck
checks *all* contexts in every transaction, including the "partition
descriptor" context which stores a partitioned table's PartitionDesc.
PartitionDesc gets bigger as the number of partitions increase, so does
the time to check the context it's allocated in.

So, the decrease in throughput in cassert build as the number of
partitions increases is not due to any fault of this patch series as I had
suspected.  Phew!

Thanks,
Amit

Commits

  1. Clean up handling of constraint_exclusion and enable_partition_pruning.

  2. Add test case exercising formerly-unreached code in inheritance_planner.

  3. Speed up planning when partitions can be pruned at plan time.

  4. Avoid crash in partitionwise join planning under GEQO.

  5. Avoid passing query tlist around separately from root->processed_tlist.

  6. Build "other rels" of appendrel baserels in a separate step.

  7. Get rid of duplicate child RTE for a partitioned table.

  8. Rearrange make_partitionedrel_pruneinfo to avoid work when we can't prune.

  9. Don't copy PartitionBoundInfo in set_relation_partition_info.

  10. Move building of child base quals out into a new function

  11. Reorganize planner code moved in b60c39759908

  12. Move inheritance expansion code into its own file

  13. Fix inherited UPDATE/DELETE with UNION ALL subqueries.

  14. Rearrange planner to save the whole PlannerInfo (subroot) for a subquery.