Horton

Thread

Commits

  1. Fix memory leak in SnapBuildSerialize.

  1. SnapBuildSerialize function forgot pfree variable ondisk_c

    范孝剑 (康贤) <funnyxj.fxj@alibaba-inc.com> — 2020-11-05T07:35:50Z

    Hi all,
    When I tested logical decoding, I found that walsender process memory usage grow very high, by debugging,  I found SnapBuildSerialize palloc memory for variable ondisk_c, but don't pfree it at last.
So when master LogStandbySnapshot() too frequently, walsender memory will grow very high and OOM finally.

  2. Re: SnapBuildSerialize function forgot pfree variable ondisk_c

    Amit Kapila <amit.kapila16@gmail.com> — 2020-11-06T03:22:19Z

    On Thu, Nov 5, 2020 at 2:06 PM 范孝剑(康贤) <funnyxj.fxj@alibaba-inc.com> wrote:
>
> Hi all,
>     When I tested logical decoding, I found that walsender process memory usage grow very high, by debugging,  I found SnapBuildSerialize palloc memory for variable ondisk_c, but don't pfree it at last.
>

By looking at code, it is clear that it is good to free the memory
allocated for variable ondisk_c.

> So when master LogStandbySnapshot() too frequently, walsender memory will grow very high and OOM finally.
>

Is there any particular scenario where you are seeing this behavior?
Do you have any reproducible test case? Have you confirmed that after
freeing that memory your problem is solved? It is not clear to me why
other users of Logical Replication are not facing this issue?

-- 
With Regards,
Amit Kapila.

  3. 回复：SnapBuildSerialize function forgot pfree variable ondisk_c

    范孝剑 (康贤) <funnyxj.fxj@alibaba-inc.com> — 2020-11-13T06:06:20Z

    If we create logical slot frequently, every time when creating a logical slot, it will generate a LogStandbySnapshot wal record.

Session A:
do language plpgsql $$
declare
  v_text text := 'a';
begin
  for i in 1..290000 loop
    execute $_$select pg_create_logical_replication_slot('logical_slot4', 'test_decoding')$_$;
    execute $_$select pg_drop_replication_slot('logical_slot4')$_$;
  end loop;
exception when others then
  raise notice 'execute  failed.';
end;$$;

Session B:
pg_recvlogical -d postgres --start -S test -f test.log





------------------------------------------------------------------
发件人：Amit Kapila <amit.kapila16@gmail.com>
发送时间：2020年11月6日(星期五) 11:21
收件人：范孝剑(康贤) <funnyxj.fxj@alibaba-inc.com>
抄　送：pgsql-bugs <pgsql-bugs@lists.postgresql.org>
主　题：Re: SnapBuildSerialize function forgot pfree variable ondisk_c

On Thu, Nov 5, 2020 at 2:06 PM 范孝剑(康贤) <funnyxj.fxj@alibaba-inc.com> wrote:
>
> Hi all,
>     When I tested logical decoding, I found that walsender process memory usage grow very high, by debugging,  I found SnapBuildSerialize palloc memory for variable ondisk_c, but don't pfree it at last.
>

By looking at code, it is clear that it is good to free the memory
allocated for variable ondisk_c.

> So when master LogStandbySnapshot() too frequently, walsender memory will grow very high and OOM finally.
>

Is there any particular scenario where you are seeing this behavior?
Do you have any reproducible test case? Have you confirmed that after
freeing that memory your problem is solved? It is not clear to me why
other users of Logical Replication are not facing this issue?

-- 
With Regards,
Amit Kapila.

  4. Re: SnapBuildSerialize function forgot pfree variable ondisk_c

    Amit Kapila <amit.kapila16@gmail.com> — 2021-01-12T13:42:34Z

    On Fri, Nov 13, 2020 at 11:36 AM 范孝剑(康贤) <funnyxj.fxj@alibaba-inc.com> wrote:
>
> If we create logical slot frequently, every time when creating a logical slot, it will generate a LogStandbySnapshot wal record.
>
> Session A:
> do language plpgsql $$
> declare
>   v_text text := 'a';
> begin
>   for i in 1..290000 loop
>     execute $_$select pg_create_logical_replication_slot('logical_slot4', 'test_decoding')$_$;
>     execute $_$select pg_drop_replication_slot('logical_slot4')$_$;
>   end loop;
> exception when others then
>   raise notice 'execute  failed.';
> end;
> $$;
>
> Session B:
> pg_recvlogical -d postgres --start -S test -f test.log
>

Thanks for sharing the test. I am able to reproduce the issue and the
attached fixes it for me. This needs to be backpatched till 9.5 where
it was introduced. I am planning to push this tomorrow.

-- 
With Regards,
Amit Kapila.

  5. Re: SnapBuildSerialize function forgot pfree variable ondisk_c

    Amit Kapila <amit.kapila16@gmail.com> — 2021-01-13T05:40:26Z

    On Tue, Jan 12, 2021 at 7:12 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Fri, Nov 13, 2020 at 11:36 AM 范孝剑(康贤) <funnyxj.fxj@alibaba-inc.com> wrote:
> >
> > If we create logical slot frequently, every time when creating a logical slot, it will generate a LogStandbySnapshot wal record.
> >
> > Session A:
> > do language plpgsql $$
> > declare
> >   v_text text := 'a';
> > begin
> >   for i in 1..290000 loop
> >     execute $_$select pg_create_logical_replication_slot('logical_slot4', 'test_decoding')$_$;
> >     execute $_$select pg_drop_replication_slot('logical_slot4')$_$;
> >   end loop;
> > exception when others then
> >   raise notice 'execute  failed.';
> > end;
> > $$;
> >
> > Session B:
> > pg_recvlogical -d postgres --start -S test -f test.log
> >
>
> Thanks for sharing the test. I am able to reproduce the issue and the
> attached fixes it for me. This needs to be backpatched till 9.5 where
> it was introduced. I am planning to push this tomorrow.
>

Pushed!

-- 
With Regards,
Amit Kapila.