Horton

Re: Memory-Bounded Hash Aggregation

Heikki Linnakangas <hlinnaka@iki.fi>

View thread View on postgr.es
From: Heikki Linnakangas <hlinnaka@iki.fi>
To: Jeff Davis <pgsql@j-davis.com>, Peter Geoghegan <pg@bowt.ie>
Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>, Taylor Vesely <tvesely@pivotal.io>, Adam Lee <ali@pivotal.io>, Melanie Plageman <mplageman@pivotal.io>, PostgreSQL Hackers <pgsql-hackers@postgresql.org>
Date: 2020-02-06T08:01:36Z
Lists: pgsql-hackers 
On 05/02/2020 21:56, Jeff Davis wrote:
> On Tue, 2020-02-04 at 18:10 +0200, Heikki Linnakangas wrote:
>> I'd love to change the LogicalTape API so that you could allocate
>> and
>> free tapes more freely. I wrote a patch to do that, as part of
>> replacing
>> tuplesort.c's polyphase algorithm with a simpler one (see [1]), but
>> I
>> never got around to committing it. Maybe the time is ripe to do that
>> now?
> 
> It's interesting that you wrote a patch to pause the tapes a while ago.
> Did it just fall through the cracks or was there a problem with it?
> 
> Is pause/resume functionality required, or is it good enough that
> rewinding a tape frees the buffer, to be lazily allocated later?

It wasn't strictly required for what I was hacking on then. IIRC it 
would have saved some memory during sorting, but Peter G felt that it 
wasn't worth the trouble, because he made some other changes around the 
same time, which made it less important 
(https://www.postgresql.org/message-id/CAM3SWZS0nwOPoJQHvxugA9kKPzky2QC2348TTWdSStZOkke5tg%40mail.gmail.com). 
I dropped the ball on both patches then, but I still think they would be 
worthwhile.

- Heikki

Commits

  1. Fix costing for disk-based hash aggregation.

  2. Fixes for Disk-based Hash Aggregation.

  3. Disk-based Hash Aggregation.

  4. Extend ExecBuildAggTrans() to support a NULL pointer check.

  5. Save calculated transitionSpace in Agg node.

  6. Minor refactor of nodeAgg.c.

  7. Logical Tape Set: lazily allocate read buffer.

  8. Introduce TupleHashTableHash() and LookupTupleHashEntryHash().

  9. Logical Tape Set: use min heap for freelist.

  10. Fix comments in execGrouping.c

  11. Avoid integer overflow while sifting-up a heap in tuplesort.c.

  12. Make the overflow guards in ExecChooseHashTableSize be more protective.

  13. Improve performance of our private version of qsort. Per recent testing,