Horton

Re: pgsql: Add parallel-aware hash joins.

Andres Freund <andres@anarazel.de>

View thread View on postgr.es
From: Andres Freund <andres@anarazel.de>
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Alvaro Herrera <alvherre@alvh.no-ip.org>, Robert Haas <robertmhaas@gmail.com>, Thomas Munro <thomas.munro@enterprisedb.com>, "pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Date: 2018-01-24T21:07:07Z
Lists: pgsql-hackers 
Hi,

On 2018-01-24 15:58:16 -0500, Tom Lane wrote:
> Yeah.  We already have topo sort code in pg_dump, maybe we could push that
> into someplace like src/common or src/fe_utils?  Although pg_dump hasn't
> got any need for edge weights, so maybe sharing code isn't worth it.

I suspect it may be more work to share than worth it, but either way, it
shouldn't be too hard.  Hm, isn't dbObjectTypePriority kinda an edge
weight? Seems like we properly could implement it as that.


> We could flush the existing schedule files and use a simple format like
> 	testname: list of earlier tests it depends on
> (I guess there would be more properties than just the dependencies,
> but still not hard to parse.)

Yea, I think there'd need to be a few more. There's some tests that use
multiple connections, and I suspect it'll be useful to have "implicit"
ordering dependencies for a few test, like a "barrier". Otherwise
e.g. the tablespace test will be annoying to order.


> > If we keep the timings from an earlier
> > run somwhere, we can use the timing of runs as edge weights, making the
> > schedule better.
> 
> I think we could just use constant values hand-coded in the schedule file.
> It might occasionally be worth updating them, but realistically it's not
> going to matter that they be very accurate.  Probably weights like 1, 2,
> and 3 would be plenty ;-)

The reason I like the idea of using prior tests as scheduling input is
that the slowness actually depends a lot on the type of machine its run
on, and more importantly on things like valgrind, CCA, fsync=on/off,
jit=on/off (far most expensive tests is e.g. the recursion test in
errors.sql :)).

Greetings,

Andres Freund

Commits

  1. Update obsolete sentence in README.parallel.

  2. Rewrite ConditionVariableBroadcast() to avoid live-lock.

  3. Tweak parallel hash join test case in hopes of improving stability.

  4. Rename pg_rewind's copy_file_range() to avoid conflict with new linux syscall.

  5. Fix some minor errors in new PHJ code.

  6. Fix EXPLAIN ANALYZE output for Parallel Hash.

  7. Fix rare assertion failure in parallel hash join.

  8. Cancel CV sleep during subtransaction abort.

  9. Add parallel-aware hash joins.

  10. Fix EXPLAIN ANALYZE of hash join when the leader doesn't participate.

  11. Add some regression tests that exercise hash join code.