Re: Out of Memory errors are frustrating as heck!

Justin Pryzby <pryzby@telsasoft.com>

From: Justin Pryzby <pryzby@telsasoft.com>
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>
Cc: Gunther <raj@gusw.net>, Tom Lane <tgl@sss.pgh.pa.us>, Alvaro Herrera <alvherre@2ndquadrant.com>, Jeff Janes <jeff.janes@gmail.com>, pgsql-performance@lists.postgresql.org
Date: 2019-04-19T21:01:17Z
Lists: pgsql-performance
On Wed, Apr 17, 2019 at 11:52:44PM -0400, Gunther wrote:
> Hi guys. I don't want to be pushy, but I found it strange that after so much

Were you able to reproduce the issue in some minimized way ?  Like after
joining fewer tables or changing to join with fewer join conditions ?

On Thu, Apr 18, 2019 at 05:21:28PM +0200, Tomas Vondra wrote:
> As for the issue - I think the current hypothesis is that the data
> distribution is skewed in some strange way, triggering some unexpected
> behavior in hash join. That seems plausible, but it's really hard to
> investigate without knowing anything about the data distribution :-(
> 
> It would be possible to do at least one of these two things:
> 
> (a) export pg_stats info about distribution of the join keys

For starts, could you send the MCVs, maybe with some variation on this query ?
https://wiki.postgresql.org/wiki/Slow_Query_Questions#Statistics:_n_distinct.2C_MCV.2C_histogram

Justin



Commits

  1. Consider BufFiles when adjusting hashjoin parameters

  2. Allocate hash join files in a separate memory context