Horton

Re: [HACKERS] [POC] Faster processing at Gather node

Ants Aasma <ants.aasma@eesti.ee>

View thread View on postgr.es
From: Ants Aasma <ants.aasma@eesti.ee>
To: Robert Haas <robertmhaas@gmail.com>
Cc: Amit Kapila <amit.kapila16@gmail.com>, Rafia Sabih <rafia.sabih@enterprisedb.com>, Andres Freund <andres@anarazel.de>, PostgreSQL Developers <pgsql-hackers@postgresql.org>
Date: 2017-11-16T15:23:12Z
Lists: pgsql-hackers 
On Thu, Nov 16, 2017 at 6:42 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> The problem here is that we have no idea how big the queue needs to
> be.  The workers will always be happy to generate tuples faster than
> the leader can read them, if that's possible, but it will only
> sometimes help performance to let them do so.   I think in most cases
> we'll end up allocating the local queue - because the workers can
> generate faster than the leader can read - but only occasionally will
> it make anything faster.

For the Gather Merge driven by Parallel Index Scan case it seems to me
that the correct queue size is one that can store two index pages
worth of tuples. Additional space will always help buffer any
performance variations, but there should be a step function somewhere
around 1+1/n_workers pages. I wonder if the queue could be dynamically
sized based on the driving scan. With some limits of course as parent
nodes to the parallel index scan can increase the row count by
arbitrary amounts.

Regards,
Ants Aasma
--
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26, A-2700 Wiener Neustadt
Web: http://www.postgresql-support.de, http://www.cybertec.at

Commits

  1. C comment: fix wording about shared memory message queue

  2. shm_mq: Have the receiver set the sender's less frequently.

  3. shm_mq: Reduce spinlock usage.

  4. Remove use of byte-masking macros in record_image_cmp

  5. Remove memory leak protection from Gather and Gather Merge nodes.

  6. Avoid projecting tuples unnecessarily in Gather and Gather Merge.

  7. Tweak use of ExecContextForcesOids by Gather (Merge).

  8. Pass eflags down to parallel workers.

  9. Disable installcheck tests for test_session_hooks

  10. First-draft release notes for 10.1.

  11. Remove TupleDesc remapping logic from tqueue.c.

  12. Avoid memory leaks when a GatherMerge node is rescanned.

  13. Code review for nodeGatherMerge.c.

  14. Add a Gather Merge executor node.

  15. Push scan/join target list beneath Gather when possible.