Horton

Re: tuplesort memory usage: grow_memtuples

Robert Haas <robertmhaas@gmail.com>

View thread
From: Robert Haas <robertmhaas@gmail.com>
To: Peter Geoghegan <peter@2ndquadrant.com>
Cc: Simon Riggs <simon@2ndquadrant.com>, Jeff Janes <jeff.janes@gmail.com>, pgsql-hackers <pgsql-hackers@postgresql.org>, Greg S <stark@mit.edu>
Date: 2012-11-15T19:16:34Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Incorporate a couple of recent tuplesort.c improvements into tuplestore.c.

On Thu, Nov 15, 2012 at 2:13 PM, Peter Geoghegan <peter@2ndquadrant.com> wrote:
> On 15 November 2012 18:13, Robert Haas <robertmhaas@gmail.com> wrote:
>> Ah, yeah.  I wondered in passing about that but forgot to follow up on
>> it.  The problem specifically is that the intermediate result
>> memtupsize * newmemtuples might overflow.  I believe that the old
>> memtupsize can never be more than 2^26 bytes, because the allocation
>> limit is 1GB and each SortTuple is 16 bytes.
>
> Do you mean the intermediate result of memtupsize * allowedMem?

Yes.

> Oh,
> yeah, that could overflow rather easily on a platform where long is
> only 32-bit.

Or even 64-bit, with really large values of work_mem.

> We're multiplying the entire current allocation size of
> the array by the maximum length. I guess the fact that you didn't spot
> it made me overconfident. :-)

Ha!

So what's next here?  Do you want to work on these issue some more?
Or does Jeff?  I'd like to see this go in, but I'm not sure I have the
bandwidth to do the legwork myself.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company