Thread

1. Plan weirdness. A sort produces more rows than the node beneath it

   Dane Foster <studdugie@gmail.com> — 2023-08-04T14:59:19Z

   Hello,
   
   I'm trying to understand a bit of weirdness in a plan output. There is a
   sort node above a sequential scan node where the scan node produces 26,026
   rows yet the sort node above it produces 42,995,408. How is it possible to
   sort more data than you received?
   
   https://explain.dalibo.com/plan/1ee665h69f92chc5
   
   The PostgreSQL version is 14.2 running on Amazon's RDS. Thanks.
   
   
   Dane
   

2. Re: Plan weirdness. A sort produces more rows than the node beneath it

   Jeff Janes <jeff.janes@gmail.com> — 2023-08-04T15:07:19Z

   On Fri, Aug 4, 2023 at 11:00 AM Dane Foster <studdugie@gmail.com> wrote:
   
   > Hello,
   >
   > I'm trying to understand a bit of weirdness in a plan output. There is a
   > sort node above a sequential scan node where the scan node produces 26,026
   > rows yet the sort node above it produces 42,995,408. How is it possible
   > to sort more data than you received?
   >
   
   This is normal for a merge join.  For every tie in the first input, the
   qualifying part of the 2nd input must be rescanned, and the rows are
   tallied again (in the sort node) each time they are rescanned.
   
   Cheers,
   
   Jeff
   
   >
   

3. Re: Plan weirdness. A sort produces more rows than the node beneath it

   Dane Foster <studdugie@gmail.com> — 2023-08-04T15:09:09Z

   Thanks for the explanation.
   
   
   Dane
   
   
   On Fri, Aug 4, 2023 at 11:07 AM Jeff Janes <jeff.janes@gmail.com> wrote:
   
   > On Fri, Aug 4, 2023 at 11:00 AM Dane Foster <studdugie@gmail.com> wrote:
   >
   >> Hello,
   >>
   >> I'm trying to understand a bit of weirdness in a plan output. There is a
   >> sort node above a sequential scan node where the scan node produces 26,026
   >> rows yet the sort node above it produces 42,995,408. How is it possible
   >> to sort more data than you received?
   >>
   >
   > This is normal for a merge join.  For every tie in the first input, the
   > qualifying part of the 2nd input must be rescanned, and the rows are
   > tallied again (in the sort node) each time they are rescanned.
   >
   > Cheers,
   >
   > Jeff
   >
   >>
   

4. Re: Plan weirdness. A sort produces more rows than the node beneath it

   Tom Lane <tgl@sss.pgh.pa.us> — 2023-08-04T15:10:21Z

   Dane Foster <studdugie@gmail.com> writes:
   > I'm trying to understand a bit of weirdness in a plan output. There is a
   > sort node above a sequential scan node where the scan node produces 26,026
   > rows yet the sort node above it produces 42,995,408. How is it possible to
   > sort more data than you received?
   
   If the sort is the inner input to a merge join, this could reflect
   mark-and-restore rescanning of the sort's output.  Are there a
   whole lot of duplicate keys on the merge's other side?
   
   			regards, tom lane
   
   
   
   

5. Re: Plan weirdness. A sort produces more rows than the node beneath it

   Dane Foster <studdugie@gmail.com> — 2023-08-04T15:15:40Z

   > If the sort is the inner input to a merge join, this could reflect
   > mark-and-restore rescanning of the sort's output.  Are there a
   > whole lot of duplicate keys on the merge's other side?
   
   Yes. The course_id column's values repeat a LOT on the merge side.
   
   Dane
   
   
   On Fri, Aug 4, 2023 at 11:10 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
   
   > Dane Foster <studdugie@gmail.com> writes:
   > > I'm trying to understand a bit of weirdness in a plan output. There is a
   > > sort node above a sequential scan node where the scan node produces
   > 26,026
   > > rows yet the sort node above it produces 42,995,408. How is it possible
   > to
   > > sort more data than you received?
   >
   > If the sort is the inner input to a merge join, this could reflect
   > mark-and-restore rescanning of the sort's output.  Are there a
   > whole lot of duplicate keys on the merge's other side?
   >
   >                         regards, tom lane
   >
   

6. Re: Plan weirdness. A sort produces more rows than the node beneath it

   Tom Lane <tgl@sss.pgh.pa.us> — 2023-08-04T15:31:57Z

   Dane Foster <studdugie@gmail.com> writes:
   >> If the sort is the inner input to a merge join, this could reflect
   >> mark-and-restore rescanning of the sort's output.  Are there a
   >> whole lot of duplicate keys on the merge's other side?
   
   > Yes. The course_id column's values repeat a LOT on the merge side.
   
   Hmm.  The planner should avoid using a merge join if it knows that
   to be true.  Maybe analyze'ing that table would prompt it to use
   some other join method?
   
   			regards, tom lane
   
   
   
   

7. Re: Plan weirdness. A sort produces more rows than the node beneath it

   Dane Foster <studdugie@gmail.com> — 2023-08-04T16:41:28Z

   >
   > Hmm.  The planner should avoid using a merge join if it knows that
   > to be true.  Maybe analyze'ing that table would prompt it to use
   > some other join method?
   
   
   
   The planner has updated stats on the table and wants to use a nested loop:
   
   https://explain.dalibo.com/plan/3814d5356cc82528
   
   
   But the nested loop version is around 8 seconds slower so I forced the
   issue. But thanks to this conversation I now understand what's happening
   with the row count. This understanding helped make the nested loops' plan
   easier to understand. Unfortunately, there doesn't seem to be any hope for
   the merge join variant in terms of being easily understood. The uninitiated
   sees a scan node and its parent sort node and their brain defaults to
   thinking: the sort node will produce the same number of rows as the node
   feeding it.
   
   
   Cheers,
   
   Dane