Horton

Re: [HACKERS] [POC] Faster processing at Gather node

Rafia Sabih <rafia.sabih@enterprisedb.com>

View thread View on postgr.es
From: Rafia Sabih <rafia.sabih@enterprisedb.com>
To: Robert Haas <robertmhaas@gmail.com>
Cc: Amit Kapila <amit.kapila16@gmail.com>, Andres Freund <andres@anarazel.de>, PostgreSQL Developers <pgsql-hackers@postgresql.org>
Date: 2017-12-05T13:02:52Z
Lists: pgsql-hackers

Attachments

On Mon, Dec 4, 2017 at 9:20 PM, Robert Haas <robertmhaas@gmail.com> wrote:

> On Sun, Dec 3, 2017 at 10:30 PM, Amit Kapila <amit.kapila16@gmail.com>
> wrote:
> > I thought there are some cases (though less) where we want to Shutdown
> > the nodes (ExecShutdownNode) earlier and release the resources sooner.
> > However, if you are not completely sure about this change, then we can
> > leave it as it.  Thanks for sharing your thoughts.
>
> OK, thanks.  I committed that patch, after first running 100 million
> tuples through a Gather over and over again to test for leaks.
> Hopefully I haven't missed anything here, but it looks like it's
> solid.  Here once again are the remaining patches.  While the
> already-committed patches are nice, these two are the ones that
>

Hi,
I was spending sometime in verifying this memory-leak patch for
gather-merge case and I too found it good. In the query I tried, around 10
million tuples were passed through gather-merge. On analysing the output of
top it looks acceptable memory usage and it gets freed once the query is
completed. Since, I was trying on my local system only, I tested for upto 8
workers and didn't find any memory leaks for the queries I tried.
One may find the attached file for the test-case.

-- 
Regards,
Rafia Sabih
EnterpriseDB: http://www.enterprisedb.com/

Commits

  1. C comment: fix wording about shared memory message queue

  2. shm_mq: Have the receiver set the sender's less frequently.

  3. shm_mq: Reduce spinlock usage.

  4. Remove use of byte-masking macros in record_image_cmp

  5. Remove memory leak protection from Gather and Gather Merge nodes.

  6. Avoid projecting tuples unnecessarily in Gather and Gather Merge.

  7. Tweak use of ExecContextForcesOids by Gather (Merge).

  8. Pass eflags down to parallel workers.

  9. Disable installcheck tests for test_session_hooks

  10. First-draft release notes for 10.1.

  11. Remove TupleDesc remapping logic from tqueue.c.

  12. Avoid memory leaks when a GatherMerge node is rescanned.

  13. Code review for nodeGatherMerge.c.

  14. Add a Gather Merge executor node.

  15. Push scan/join target list beneath Gather when possible.