Horton

Re: POC: Sharing record typmods between backends

Thomas Munro <thomas.munro@enterprisedb.com>

View thread View on postgr.es
From: Thomas Munro <thomas.munro@enterprisedb.com>
To: Andres Freund <andres@anarazel.de>
Cc: Robert Haas <robertmhaas@gmail.com>, Dilip Kumar <dilipbalaut@gmail.com>, Pg Hackers <pgsql-hackers@postgresql.org>
Date: 2017-07-10T09:39:09Z
Lists: pgsql-hackers

Attachments

On Thu, Jun 1, 2017 at 6:29 AM, Andres Freund <andres@anarazel.de> wrote:
> On May 31, 2017 11:28:18 AM PDT, Robert Haas <robertmhaas@gmail.com> wrote:
>>> On 2017-05-31 13:27:28 -0400, Dilip Kumar wrote:
[ ... various discussion in support of using DHT ... ]

Ok, good.

Here's a new version that introduces a per-session DSM segment to hold
the shared record typmod registry (and maybe more things later).  The
per-session segment is created the first time you run a parallel query
(though there is handling for failure to allocate that allows the
parallel query to continue with no workers) and lives until your
leader backend exits.  When parallel workers start up, they see its
handle in the per-query segment and attach to it, which puts
typcache.c into write-through cache mode so their idea of record
typmods stays in sync with the leader (and each other).

I also noticed that I could delete even more of tqueue.c than before:
it doesn't seem to have any remaining reason to need to know the
TupleDesc.

One way to test this code is to apply just
0003-rip-out-tqueue-remapping-v3.patch and then try the example from
the first message in this thread to see it break, and then try again
with the other two patches applied.  By adding debugging trace you can
see that the worker pushes a bunch of TupleDescs into shmem, they get
pulled out by the leader when it sees the tuples, and then on a second
invocation the (new) worker can reuse them: it finds matches already
in shmem from the first invocation.

I used a DSM segment with a TOC and a DSA area inside that, like the
existing per-query DSM segment, but obviously you could spin it
various different ways.  One example: just have a DSA area and make a
new kind of TOC thing that deals in dsa_pointers.  Better ideas?

I believe combo CIDs should also go in there, to enable parallel
write, but I'm not 100% sure: that's neither per-session nor per-query
data, that's per-transaction.  So perhaps the per-session DSM could
hold a per-session DSA and a per-transaction DSA, where the latter is
reset for each transaction, just like TopTransactionContext (though
dsa.c doesn't have a 'reset thyself' function currently).  That seems
like a good place to store a shared combo CID hash table using DHT.
Thoughts?

-- 
Thomas Munro
http://www.enterprisedb.com

Commits

  1. Remove TupleDesc remapping logic from tqueue.c.

  2. Add support for coordinating record typmods among parallel workers.

  3. Improve division of labor between execParallel.c and nodeGather[Merge].c.

  4. Add minimal regression test for blessed record type transfer.

  5. Consolidate the function pointer types used by dshash.c.

  6. Fix unlikely shared memory leak after failure in dshash_create().

  7. Refactor typcache.c's record typmod hash table.

  8. Add a hash_combine function for mixing hash values.

  9. Backpatch introduction of TupleDescAttr(tupdesc, i).

  10. Partially flatten struct tupleDesc so that it can be used in DSM.

  11. Change tupledesc->attrs[n] to TupleDescAttr(tupledesc, n).