Re: Lots of memory allocated when reassigning Large Objects

Tom Lane <tgl@sss.pgh.pa.us>

From: Tom Lane <tgl@sss.pgh.pa.us>
To: Guillaume Lelarge <guillaume@lelarge.info>
Cc: Justin Pryzby <pryzby@telsasoft.com>, PostgreSQL Developers <pgsql-hackers@postgresql.org>
Date: 2021-11-29T19:39:14Z
Lists: pgsql-hackers

Attachments

Guillaume Lelarge <guillaume@lelarge.info> writes:
> I've tried Justin's patch but it didn't help with my memory allocation
> issue. FWIW, I attach the patch I used in v14.

[ looks closer ... ]  Ah, that patch is a bit buggy: it fails to do the
right thing in the cases where the loop does a "continue".  The attached
revision seems to behave properly.

I still see a small leakage, which I think is due to accumulation of
pending sinval messages for the catalog updates.  I'm curious whether
that's big enough to be a problem for Guillaume's use case.  (We've
speculated before about bounding the memory used for pending sinval
in favor of just issuing a cache reset when the list would be too
big.  But nobody's done anything about it, suggesting that people
seldom have a problem in practice.)

>> DROP OWNED BY likely has similar issues.

> Didn't try it, but it wouldn't be a surprise.

I tried just changing the REASSIGN to a DROP in Justin's example,
and immediately hit

ERROR:  out of shared memory
HINT:  You might need to increase max_locks_per_transaction.

thanks to the per-object locks we try to acquire.  So I'm not
sure that the DROP case can reach an interesting amount of
local memory leaked before it runs out of lock-table space.

			regards, tom lane

Commits

  1. Avoid leaking memory during large-scale REASSIGN OWNED BY operations.

  2. Reduce memory consumption for pending invalidation messages.