Horton

Re: [HACKERS] [PATCH] Incremental sort

Alexander Korotkov <a.korotkov@postgrespro.ru>

View thread View on postgr.es
From: Alexander Korotkov <a.korotkov@postgrespro.ru>
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>
Cc: Antonin Houska <ah@cybertec.at>, pgsql-hackers <pgsql-hackers@postgresql.org>
Date: 2018-03-05T22:07:18Z
Lists: pgsql-hackers

Attachments

Hi!

Thank you for reviewing this patch!
Revised version is attached.

On Mon, Mar 5, 2018 at 1:19 PM, Tomas Vondra <tomas.vondra@2ndquadrant.com>
wrote:

> I have started reviewing the patch and doing some testing, and I have
> pretty quickly ran into a segfault. Attached is a simple reproducer and
> an backtrace. AFAICS the bug seems to be somewhere in the tuplesort
> changes, likely resetting a memory context too soon or something like
> that. I haven't investigated it further, but it matches my hunch that
> tuplesort is likely where the bugs will be.
>

Right.  Incremental sort patch introduces maincontext of memory which
is persistent between incremental sort groups.  But mergeruns()
reallocates memtuples in sortcontext which is cleared by tuplesort_reset().
Fixed in the revised patch.

Otherwise the patch seems fairly complete. A couple of minor things that
> I noticed while eyeballing the changes in a diff editor.
>
>
> 1) On a couple of places the new code has this comment
>
>     /* even when not parallel-aware */
>
> while all the immediately preceding blocks use
>
>     /* even when not parallel-aware, for EXPLAIN ANALYZE */
>
> I suggest using the same comment, otherwise it kinda suggests it's not
> because of EXPLAIN ANALYZE.
>

Right, fixed.  I also found that incremental sort shoudn't support
DSM reinitialization similarly to regular sort.  Fixes in the revised patch.

2) I think the purpose of sampleSlot should be explicitly documented
> (and I'm not sure "sample" is a good term here, as is suggest some sort
> of sampling (for example nodeAgg uses grp_firstTuple).
>

Yes, "sample" isn't a good term here.  However, "first" isn't really
correct,
because we can skip some tuples from beginning of the group in
order to not form groups too frequently.  I'd rather name it "pivot" tuple
slot.


> 3) skipCols/SkipKeyData seems a bit strange too, I think. I'd use
> PresortedKeyData or something like that.
>

Good point, renamed.

4) In cmpSortSkipCols, when checking if the columns changed, the patch
> does this:
>
>     n = ((IncrementalSort *) node->ss.ps.plan)->skipCols;
>
>     for (i = 0; i < n; i++)
>     {
>         ... check i-th key ...
>     }
>
> My hunch is that checking the keys from the last one, i.e.
>
>     for (i = (n-1); i >= 0; i--)
>     {
>         ....
>     }
>
> would be faster. The reasoning is that with "ORDER BY a,b" the column
> "b" changes more often. But I've been unable to test this because of the
> segfault crashes.
>

Agreed.

5) The changes from
>
>     if (pathkeys_contained_in(...))
>
> to
>
>     n = pathkeys_common(pathkeys, subpath->pathkeys);
>
>
>     if (n == 0)
>
> seem rather inconvenient to me, as it makes the code unnecessarily
> verbose. I wonder if there's a better way to deal with this.
>

I would rather say, that it changes from

    if (pathkeys_contained_in(...))

to

    n = pathkeys_common(pathkeys, subpath->pathkeys);

    if (n == list_length(pathkeys))

I've introduced pathkeys_common_contained_in() which returns the same
result as pathkeys_contained_in(), but sets number of common pathkeys
to the last argument.  It simplifies code a little bit. The name, probably,
could be improved.

------
Alexander Korotkov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

Commits

  1. Further adjustments to Hashagg EXPLAIN ANALYZE output

  2. Rework EXPLAIN format for incremental sort

  3. Fix typos and improve incremental sort comments

  4. Stabilize incremental_sort tests

  5. Minor improvements in Incremental Sort explain

  6. Consider Incremental Sort paths at additional places

  7. Fix representation of SORT_TYPE_STILL_IN_PROGRESS.

  8. Fix failures in incremental_sort due to number of workers

  9. Fix show_incremental_sort_info with force_parallel_mode

  10. Implement Incremental Sort

  11. Fix handling of "Subplans Removed" field in EXPLAIN output.

  12. Fix EXPLAIN (SETTINGS) to follow policy about when to print empty fields.

  13. Ensure plpgsql result tuples have the right composite type marking.

  14. Propagate sort instrumentation from workers back to leader.

  15. Make new regression test case parallel-safe, and improve its output.

  16. Push limit through subqueries to underlying sort, where possible.

  17. Fix inappropriate printing of never-measured times in EXPLAIN.

  18. Fix some infelicities in EXPLAIN output for parallel query plans.