Re: Explain buffers wrong counter with parallel plans

Jonathan Katz <jonathan.katz@excoventures.com>

From: "Jonathan S. Katz" <jonathan.katz@excoventures.com>
To: Amit Kapila <amit.kapila16@gmail.com>
Cc: Robert Haas <robertmhaas@gmail.com>, Alvaro Herrera <alvherre@2ndquadrant.com>, Adrien Nayrat <adrien.nayrat@anayrat.info>, Andres Freund <andres@anarazel.de>, PostgreSQL mailing lists <pgsql-hackers@postgresql.org>
Date: 2018-07-27T17:42:45Z
Lists: pgsql-hackers
> On Jul 27, 2018, at 8:31 AM, Amit Kapila <amit.kapila16@gmail.com> wrote:
> 
> On Thu, Jul 26, 2018 at 7:31 PM, Jonathan S. Katz
> <jonathan.katz@excoventures.com> wrote:
>> 
>>> On Jul 25, 2018, at 10:25 PM, Amit Kapila <amit.kapila16@gmail.com> wrote:
>>> 
>>> 
>>> You mean to say the number (Buffers: shared read=442478) in HEAD,
>>> right?  If so, yeah, I am also wondering why the results of the patch
>>> are different in HEAD and 11beta2.  So, if I read correctly, the
>>> numbers in 11beta2 appears correct, but on HEAD it is not correct, it
>>> should have divided the buffers read by loops.
> 
> I want to correct myself here, the numbers on HEAD are correct, but
> not on PG11beta2.  Is there any chance that either you forgot to apply
> the patch or maybe it is not using correct binaries in case of
> 11beta2.

I need to correct myself too as I was unclear - that was an *unpatched*
11beta2, sorry for the confusion.

>>> I will figure that
>>> out, but this is just to clarify that both of us are seeing the
>>> results in the same way.
>> 
>> I’ll try it again but patch it against 11beta2 and see what results I get.
>> 
>>> 
>>>> and there appears to be a performance hit.
>>>> 
>>> 
>>> Do you mean to say the performance of the same query in 11beta2 and
>>> HEAD or something else?
>> 
>> Correct. But per your advice let me try running it against a patched
>> version of 11beta2 and see what happens.
>> 
> 
> Yeah, that would be better.  Today, I have tried the patch on both
> Head and PG11 and I am getting same and correct results.

I have applied the the patch to PG11beta2 and tested. I received
similar numbers to to the patched HEAD tests, e.g:

    => EXPLAIN (analyze,buffers,timing off,costs off) SELECT count(*) FROM t1;
                                    QUERY PLAN
    --------------------------------------------------------------------------
     Finalize Aggregate (actual rows=1 loops=1)
       Buffers: shared hit=224 read=442254
       ->  Gather (actual rows=7 loops=1)
             Workers Planned: 6
             Workers Launched: 6
             Buffers: shared hit=224 read=442254
             ->  Partial Aggregate (actual rows=1 loops=7)
                   Buffers: shared hit=224 read=442254
                   ->  Parallel Seq Scan on t1 (actual rows=14285714 loops=7)
                         Buffers: shared hit=224 read=442254
     Planning Time: 0.104 ms
     Execution Time: 5308.140 ms


I ran the tests a few more times and I understand why the execution
times vary (with an explanation from Andres) so I am satisfied.

Thanks for working on this!

Jonathan

Commits

  1. Adjust comment atop ExecShutdownNode.

  2. Prohibit shutting down resources if there is a possibility of back up.

  3. Fix buffer usage stats for parallel nodes.

  4. Match the buffer usage tracking for leader and worker backends.

  5. Improve division of labor between execParallel.c and nodeGather[Merge].c.