Horton

Re: Adding skip scan (including MDAM style range skip scan) to nbtree

Peter Geoghegan <pg@bowt.ie>

View thread
From: Peter Geoghegan <pg@bowt.ie>
To: Tomas Vondra <tomas@vondra.me>
Cc: Matthias van de Meent <boekewurm+postgres@gmail.com>, Mark Dilger <mark.dilger@enterprisedb.com>, Heikki Linnakangas <hlinnaka@iki.fi>, pgsql-hackers@lists.postgresql.org
Date: 2025-05-11T16:07:56Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. nbtree: Always set skipScan flag on rescan.

  2. meson: Build numeric.c with -ftree-vectorize.

  3. Fix "variable not found in subplan target lists" in semijoin de-duplication.

  4. Revert "nbtree: Remove useless row compare arg."

  5. nbtree: Remove useless row compare arg.

  6. Prevent premature nbtree array advancement.

  7. nbtree: tighten up array recheck rules.

  8. Avoid treating nonrequired nbtree keys as required.

  9. Adjust overstrong nbtree skip array assertion.

  10. Make NULL tuple values always advance skip arrays.

  11. Avoid extra index searches through preprocessing.

  12. Improve nbtree skip scan primitive scan scheduling.

  13. Further optimize nbtree search scan key comparisons.

  14. Add nbtree skip scan optimization.

  15. Improve nbtree array primitive scan scheduling.

  16. nbtree: Make BTMaxItemSize into object-like macro.

  17. Show index search count in EXPLAIN ANALYZE, take 2.

  18. Make parallel nbtree index scans use an LWLock.

  19. Show index search count in EXPLAIN ANALYZE.

  20. Avoid nbtree parallel scan currPos confusion.

  21. nbtree: Remove useless 'strat' local variable.

  22. Normalize nbtree truncated high key array behavior.

  23. Refactor handling of nbtree array redundancies.

  24. Fix nbtree pgstats accounting with parallel scans.

  25. Avoid parallel nbtree index scan hangs with SAOPs.

  26. Show Parallel Bitmap Heap Scan worker stats in EXPLAIN ANALYZE

  27. Enhance nbtree ScalarArrayOp execution.

  28. Skip checking of scan keys required for directional scan in B-tree

  29. Instead of using a numberOfRequiredKeys count to distinguish required

On Sat, May 10, 2025 at 10:59 AM Tomas Vondra <tomas@vondra.me> wrote:
> But doesn't it also highlight how fragile this memory allocation is? The
> skip scan patch didn't do anything wrong - it just added a couple
> fields, using a little bit more memory. I think we understand allocating
> more memory may need more time, but we expect the effect to be somewhat
> proportional. Which doesn't seem to be the case here.
>
> Many other patches add fields somewhere, it seems like bad luck the skip
> scan happened to trigger this behavior. It's quite likely other patches
> ran into the same issue, except that no one noticed. Maybe the skip scan
> did that in much hotter code, not sure.

But what did the skip scan commit (specifically commit 92fe23d9,
without any of the follow-up commits) change about memory allocation,
that might be at issue with your test case? You said that that commit
"just added a couple fields". What specific fields are you talking
about, that were added by commit 92fe23d9?

I already speculated that the issue might be tied to the addition of a
new support routine (skip support), but the experiment we ran to try
to validate that theory disproved it. What else is there?

Again, commit 92fe23d9 didn't make BTScanOpaqueData any larger
(follow-up commit 8a510275 added a new BTScanOpaqueData.skipScan bool
field, but that didn't even affect sizeof BTScanOpaqueData, since the
field fit into what was previously just alignment padding space).
AFAICT nothing that seems like it might be relevant changed, apart
from the addition of the new support routine, which was ruled out
already.

-- 
Peter Geoghegan