Re: Vacuum: allow usage of more than 1GB of work mem

On 13/07/18 01:39, Andrew Dunstan wrote:
> On 07/12/2018 06:34 PM, Alvaro Herrera wrote:
>> On 2018-Jul-12, Andrew Dunstan wrote:
>>
>>> I fully understand. I think this needs to go back to "Waiting on Author".
>> Why?  Heikki's patch applies fine and passes the regression tests.
> 
> Well, I understood Claudio was going to do some more work (see
> upthread).

Claudio raised a good point, that doing small pallocs leads to 
fragmentation, and in particular, it might mean that we can't give back 
the memory to the OS. The default glibc malloc() implementation has a 
threshold of 4 or 32 MB or something like that - allocations larger than 
the threshold are mmap()'d, and can always be returned to the OS. I 
think a simple solution to that is to allocate larger chunks, something 
like 32-64 MB at a time, and carve out the allocations for the nodes 
from those chunks. That's pretty straightforward, because we don't need 
to worry about freeing the nodes in retail. Keep track of the current 
half-filled chunk, and allocate a new one when it fills up.

He also wanted to refactor the iterator API, to return one ItemPointer 
at a time. I don't think that's necessary, the current iterator API is 
more convenient for the callers, but I don't feel strongly about that.

Anything else?

> If we're going to go with Heikki's patch then do we need to
> change the author, or add him as an author?

Let's list both of us. At least in the commit message, doesn't matter 
much what the commitfest app says.

- Heikki