Re: Out of Memory errors are frustrating as heck!

Tom Lane <tgl@sss.pgh.pa.us>

From: Tom Lane <tgl@sss.pgh.pa.us>
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>
Cc: Justin Pryzby <pryzby@telsasoft.com>, Gunther <raj@gusw.net>, pgsql-performance@lists.postgresql.org, Jeff Janes <jeff.janes@gmail.com>, Alvaro Herrera <alvherre@2ndquadrant.com>
Date: 2019-04-20T20:46:03Z
Lists: pgsql-performance
Tomas Vondra <tomas.vondra@2ndquadrant.com> writes:
> I think it's really a matter of underestimate, which convinces the planner
> to hash the larger table. In this case, the table is 42GB, so it's
> possible it actually works as expected. With work_mem = 4MB I've seen 32k
> batches, and that's not that far off, I'd day. Maybe there are more common
> values, but it does not seem like a very contrived data set.

Maybe we just need to account for the per-batch buffers while estimating
the amount of memory used during planning.  That would force this case
into a mergejoin instead, given that work_mem is set so small.

			regards, tom lane



Commits

  1. Consider BufFiles when adjusting hashjoin parameters

  2. Allocate hash join files in a separate memory context