Horton

Re: allocation limit for encoding conversion

Tom Lane <tgl@sss.pgh.pa.us>

View thread View on postgr.es
From: Tom Lane <tgl@sss.pgh.pa.us>
To: Andres Freund <andres@anarazel.de>
Cc: Alvaro Herrera <alvherre@2ndquadrant.com>, pgsql-hackers@lists.postgresql.org
Date: 2019-09-24T22:09:28Z
Lists: pgsql-hackers 
Andres Freund <andres@anarazel.de> writes:
> On 2019-09-24 16:19:41 -0400, Tom Lane wrote:
>> Andres Freund <andres@anarazel.de> writes:
>>> It's probably too big a hammer for this specific case, but I think at
>>> some point we ought to stop using fixed size allocations for this kind
>>> of work. Instead we should use something roughly like our StringInfo,
>>> except that when exceeding the current size limit, the overflowing data
>>> is stored in a separate allocation. And only once we actually need the
>>> data in a consecutive form, we allocate memory that's large enough to
>>> store the all the separate allocations in their entirety.

>> That sounds pretty messy :-(.

> I don't think it's too bad - except for now allowing the .data member of
> such a 'chunked' StringInfo to be directly accessible, it'd be just
> about the same interface as the current stringinfo.

I dunno.  What you're describing would be a whole lotta work, and it'd
break a user-visible API, and no amount of finagling is going to prevent
it from making conversions somewhat slower, and the cases where it matters
to not preallocate a surely-large-enough buffer are really few and far
between.  I have to think that we have better ways to spend our time.

			regards, tom lane

Commits

  1. Avoid unnecessary out-of-memory errors during encoding conversion.

  2. Allow repalloc() to give back space when a large chunk is downsized.