Horton

Thread

Commits

  1. Fix relation descriptor leak.

  1. Re: adding partitioned tables to publications

    Zhao Rui <875941708@qq.com> — 2020-12-30T14:15:24Z

    The first file of Amit's patch can not only re-range the code, but also fix a hidden bug.
To make it easy to see, I attach another patch.
"RelationIdGetRelation" will increase ref on owner-&gt;relrefarr, without "RelationClose", the owner-&gt;relrefarr will enlarge and re-hash.
When the capacity of owner-&gt;relrefarr is over than 10 million, enlarge and re-hash takes serial hours. And what's worse, increase ref will also take minutes, as the hash collision resolution is based on looking up an array in order.
When we want to publish 10 billion data under one partition table, it takes serial days up to increase ref, enlarge and re-hash, and CPU is always 99%. 
After applying my patch, 10 billion will be published in 10 minutes.





------------------ Original ------------------
From: &nbsp;"Amit Langote";<amitlangote09@gmail.com&gt;;
Send time:&nbsp;Friday, Apr 17, 2020 10:58 PM
To:&nbsp;"Peter Eisentraut"<peter.eisentraut@2ndquadrant.com&gt;; 
Cc:&nbsp;"Petr Jelinek"<petr@2ndquadrant.com&gt;; "Rafia Sabih"<rafia.pghackers@gmail.com&gt;; "PostgreSQL-development"<pgsql-hackers@postgresql.org&gt;; 
Subject: &nbsp;Re: adding partitioned tables to publications



On Fri, Apr 17, 2020 at 10:23 PM Peter Eisentraut
<peter.eisentraut@2ndquadrant.com&gt; wrote:
&gt; On 2020-04-09 09:28, Amit Langote wrote:
&gt; &gt; While figuring this out, I thought the nearby code could be rearranged
&gt; &gt; a bit, especially to de-duplicate the code.&nbsp; Also, I think
&gt; &gt; get_rel_sync_entry() may be a better place to set the map, rather than
&gt; &gt; maybe_send_schema().&nbsp; Thoughts?
&gt;
&gt; because I didn't really have an opinion on that at the time, but if you
&gt; still want it considered or have any open thoughts on this thread,
&gt; please resend or explain.

Sure, thanks for taking care of the bug.

Rebased the code rearrangement patch.&nbsp; Also resending the patch to fix
TAP tests for improving coverage as described in:
https://www.postgresql.org/message-id/CA%2BHiwqFyydvQ5g%3Dqa54UM%2BXjm77BdhX-nM4dXQkNOgH%3DzvDjoA%40mail.gmail.com

To summarize:
1. Missing coverage for a couple of related blocks in
apply_handle_tuple_routing()
2. Missing coverage report for the code in pgoutput.c added by 83fd4532

-- 
Amit Langote
EnterpriseDB: http://www.enterprisedb.com

  2. Re: adding partitioned tables to publications

    Amit Kapila <amit.kapila16@gmail.com> — 2021-01-11T11:42:45Z

    On Wed, Dec 30, 2020 at 8:03 PM 赵锐 <875941708@qq.com> wrote:
>
> The first file of Amit's patch can not only re-range the code, but also fix a hidden bug.
> To make it easy to see, I attach another patch.
> "RelationIdGetRelation" will increase ref on owner->relrefarr, without "RelationClose", the owner->relrefarr will enlarge and re-hash.
> When the capacity of owner->relrefarr is over than 10 million, enlarge and re-hash takes serial hours. And what's worse, increase ref will also take minutes, as the hash collision resolution is based on looking up an array in order.
> When we want to publish 10 billion data under one partition table, it takes serial days up to increase ref, enlarge and re-hash, and CPU is always 99%.
> After applying my patch, 10 billion will be published in 10 minutes.
>

It is a clear relation descriptor leak. The proposed fix seems correct
to me. The patch wasn't getting applied to HEAD. So, I have prepared
the separate patches for HEAD and 13. There are minor modifications in
the patch like I have used RelationIsValid before closing the
relation. I have not added any test because I see that there is
already a test in src/test/subscription/t/013_partition.

Kindly let me know your English name so that I can give you credit as
a co-author?

-- 
With Regards,
Amit Kapila.

  3. Re: adding partitioned tables to publications

    Zhao Rui <875941708@qq.com> — 2021-01-11T12:03:34Z

    Thanks for your reply. The patch is exactly what I want.
My English name is Mark Zhao, which should be the current email name.


Thanks,
Mark Zhao




------------------&nbsp;Original&nbsp;------------------
From: &nbsp;"Amit Kapila";<amit.kapila16@gmail.com&gt;;
Send time:&nbsp;Monday, Jan 11, 2021 8:12 PM
To:&nbsp;"赵锐"<875941708@qq.com&gt;; 
Cc:&nbsp;"Peter Eisentraut"<peter.eisentraut@2ndquadrant.com&gt;; "Amit Langote"<amitlangote09@gmail.com&gt;; "Petr Jelinek"<petr@2ndquadrant.com&gt;; "Rafia Sabih"<rafia.pghackers@gmail.com&gt;; "PostgreSQL-development"<pgsql-hackers@postgresql.org&gt;; 
Subject: &nbsp;Re: adding partitioned tables to publications



On Wed, Dec 30, 2020 at 8:03 PM 赵锐 <875941708@qq.com&gt; wrote:
&gt;
&gt; The first file of Amit's patch can not only re-range the code, but also fix a hidden bug.
&gt; To make it easy to see, I attach another patch.
&gt; "RelationIdGetRelation" will increase ref on owner-&gt;relrefarr, without "RelationClose", the owner-&gt;relrefarr will enlarge and re-hash.
&gt; When the capacity of owner-&gt;relrefarr is over than 10 million, enlarge and re-hash takes serial hours. And what's worse, increase ref will also take minutes, as the hash collision resolution is based on looking up an array in order.
&gt; When we want to publish 10 billion data under one partition table, it takes serial days up to increase ref, enlarge and re-hash, and CPU is always 99%.
&gt; After applying my patch, 10 billion will be published in 10 minutes.
&gt;

It is a clear relation descriptor leak. The proposed fix seems correct
to me. The patch wasn't getting applied to HEAD. So, I have prepared
the separate patches for HEAD and 13. There are minor modifications in
the patch like I have used RelationIsValid before closing the
relation. I have not added any test because I see that there is
already a test in src/test/subscription/t/013_partition.

Kindly let me know your English name so that I can give you credit as
a co-author?

-- 
With Regards,
Amit Kapila.

  4. Re: adding partitioned tables to publications

    Amit Kapila <amit.kapila16@gmail.com> — 2021-01-12T08:11:37Z

    On Mon, Jan 11, 2021 at 5:44 PM Mark Zhao <875941708@qq.com> wrote:
>
> Thanks for your reply. The patch is exactly what I want.
> My English name is Mark Zhao, which should be the current email name.
>

Pushed the fix.

-- 
With Regards,
Amit Kapila.

  5. Re: adding partitioned tables to publications

    Amit Langote <amitlangote09@gmail.com> — 2021-01-13T07:19:09Z

    On Tue, Jan 12, 2021 at 5:09 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, Jan 11, 2021 at 5:44 PM Mark Zhao <875941708@qq.com> wrote:
> >
> > Thanks for your reply. The patch is exactly what I want.
> > My English name is Mark Zhao, which should be the current email name.
> >
>
> Pushed the fix.

Thanks Amit and Mark.  I hadn't realized at the time that the relation
descriptor leak was occurring, but good to see it tested and fixed.

-- 
Amit Langote
EDB: http://www.enterprisedb.com