Horton

allocation limit for encoding conversion

Alvaro Herrera <alvherre@2ndquadrant.com>

View thread View on postgr.es
From: Alvaro Herrera <alvherre@2ndquadrant.com>
To: pgsql-hackers@lists.postgresql.org
Date: 2019-08-16T18:14:18Z
Lists: pgsql-hackers

Attachments

Somebody ran into issues when generating large XML output (upwards of
256 MB) and then sending via a connection with a different
client_encoding.  This occurs because we pessimistically allocate 4x as
much memory as the string needs, and we run into the 1GB palloc
limitation.  ISTM we can do better now by using huge allocations, as per
the preliminary attached patch (which probably needs an updated overflow
check rather than have it removed altogether); but at least it is able
to process this query, which it wasn't without the patch:

select query_to_xml(
    'select a, cash_words(a::text::money) from generate_series(0, 2000000) a',
    true, false, '');

-- 
Álvaro Herrera

Commits

  1. Avoid unnecessary out-of-memory errors during encoding conversion.

  2. Allow repalloc() to give back space when a large chunk is downsized.