Re: Out of Memory errors are frustrating as heck!

Tomas Vondra <tomas.vondra@2ndquadrant.com>

From: Tomas Vondra <tomas.vondra@2ndquadrant.com>
To: Gunther <raj@gusw.net>
Cc: Tom Lane <tgl@sss.pgh.pa.us>, Justin Pryzby <pryzby@telsasoft.com>, Jeff Janes <jeff.janes@gmail.com>, pgsql-performance@lists.postgresql.org, Alvaro Herrera <alvherre@2ndquadrant.com>
Date: 2019-10-06T21:06:18Z
Lists: pgsql-performance
On Sat, Aug 24, 2019 at 11:40:09AM -0400, Gunther wrote:
>Thanks Tom, yes I'd say it's using a lot of memory, but wouldn't call 
>it "leak" as it doesn't grow during the 30 min or so that this query 
>runs. It explodes to 4GB and then stays flat until done.
>

Well, the memory context stats you've shared however show this:

total: 1345345736 bytes in 1209 blocks; 4529600 free (270 chunks); 1340816136 used

That's only ~1.3GB, and ~1.1GB of that is the expression context. So
when you say 4GB, when does that happen and can you share stats showing
state at that point?

>Yes, and this time the query is super complicated with many joins and 
>tables involved. The query plan has 100 lines. Not easy to share for 
>reproduce and I have my issue under control by adding some swap just 
>in case. The swap space was never actually used.
>

Still, without the query plan we can hardly do any guesses about what
might be the issue.


regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services 



Commits

  1. Consider BufFiles when adjusting hashjoin parameters

  2. Allocate hash join files in a separate memory context