Horton

Re: Decoding speculative insert with toast leaks memory

Amit Kapila <amit.kapila16@gmail.com>

View thread View on postgr.es
From: Amit Kapila <amit.kapila16@gmail.com>
To: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com>
Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>, Andres Freund <andres@anarazel.de>, Petr Jelinek <petr.jelinek@enterprisedb.com>, Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
Date: 2021-05-27T03:32:18Z
Lists: pgsql-hackers 
On Thu, Mar 25, 2021 at 11:04 AM Ashutosh Bapat
<ashutosh.bapat.oss@gmail.com> wrote:
>
> Hi All,
> We saw OOM in a system where WAL sender consumed Gigabttes of memory
> which was never released. Upon investigation, we found out that there
> were many ReorderBufferToastHash memory contexts linked to
> ReorderBuffer context, together consuming gigs of memory. They were
> running INSERT ... ON CONFLICT .. among other things. A similar report
> at [1]
>
..
>
> but by then we might have reused the toast_hash and thus can not be
> destroyed. But that isn't the problem since the reused toast_hash will
> be destroyed eventually.
>
> It's only when the change next to speculative insert is something
> other than INSERT/UPDATE/DELETE that we have to worry about a
> speculative insert that was never confirmed. So may be for those
> cases, we check whether specinsert != null and destroy toast_hash if
> it exists.
>

Can we consider the possibility to destroy the toast_hash in
ReorderBufferCleanupTXN/ReorderBufferTruncateTXN? It will delay the
clean up of memory till the end of stream or txn but there won't be
any memory leak.

-- 
With Regards,
Amit Kapila.

Commits

  1. Fix ABI break introduced by commit 4daa140a2f.

  2. Fix valgrind issue in pgoutput.c.

  3. Fix decoding of speculative aborts.

  4. pgoutput: Fix memory leak due to RelationSyncEntry.map.

  5. Fix initialization of RelationSyncEntry for streaming transactions.

  6. Implement streaming mode in ReorderBuffer.