Horton

Re: BitmapHeapScan streaming read user and prelim refactoring

Andres Freund <andres@anarazel.de>

View thread View on postgr.es
From: Andres Freund <andres@anarazel.de>
To: Alexander Lakhin <exclusion@gmail.com>
Cc: Melanie Plageman <melanieplageman@gmail.com>, Thomas Munro <thomas.munro@gmail.com>, Tomas Vondra <tomas@vondra.me>, Nazir Bilal Yavuz <byavuz81@gmail.com>, Dilip Kumar <dilipbalaut@gmail.com>, Heikki Linnakangas <hlinnaka@iki.fi>, Pg Hackers <pgsql-hackers@postgresql.org>
Date: 2025-03-22T20:14:11Z
Lists: pgsql-hackers 
Hi,

On 2025-03-22 22:00:00 +0200, Alexander Lakhin wrote:
> 15.03.2025 16:43, Melanie Plageman wrote:
> > On Thu, Mar 13, 2025 at 5:41 PM Melanie Plageman
> > <melanieplageman@gmail.com> wrote:
> > > Overall, I feel pretty good about merging this once Thomas merges the
> > > read stream patches.
> > This was committed in 944e81bf99db2b5b70b, 2b73a8cd33b745c, and
> > c3953226a07527a1e2.
> > I've marked it as committed in the CF app.
> 
> It looks like that change made the bitmapops test unstable (on slow animals?): [1], [2], [3].
> I've reproduced such test failures when running
> TESTS=$(printf 'bitmapops %.0s' `seq 50`) make -s check-tests
> under Valgrind:
> ...
> ok 17        - bitmapops                               52719 ms
> not ok 18    - bitmapops                               57566 ms
> ok 19        - bitmapops                               60179 ms
> ok 20        - bitmapops                               32927 ms
> ok 21        - bitmapops                               45127 ms
> ok 22        - bitmapops                               42924 ms
> ok 23        - bitmapops                               61035 ms
> ok 24        - bitmapops                               56316 ms
> ok 25        - bitmapops                               52874 ms
> not ok 26    - bitmapops                               67468 ms
> ok 27        - bitmapops                               55605 ms
> ok 28        - bitmapops                               24021 ms
> ...
> 
> diff -U3 /home/vagrant/postgresql/src/test/regress/expected/bitmapops.out
> /home/vagrant/postgresql/src/test/regress/results/bitmapops.out
> --- /home/vagrant/postgresql/src/test/regress/expected/bitmapops.out 2025-03-16 01:37:52.716885600 -0700
> +++ /home/vagrant/postgresql/src/test/regress/results/bitmapops.out 2025-03-22 03:47:54.014702406 -0700
> @@ -24,14 +24,14 @@
>  SELECT count(*) FROM bmscantest WHERE a = 1 AND b = 1;
>   count
>  -------
> -    23
> +    18
>  (1 row)

Is it possible that this is the bug reported here:
https://postgr.es/m/873c33c5-ef9e-41f6-80b2-2f5e11869f1c%40garret.ru

I.e. that the all-visible logic in bitmap heapscans is fundamentally broken?

I can reproduce different results on a fast machien by just putting a VACUUM
FREEZE bmscantest after the CREATE INDEXes in bitmapops.sql. After I disable
the all-visible logic in bitmapheap_stream_read_next(), I can't observe such a
difference anymore.

Greetings,

Andres Freund

Commits

  1. Fix bitmapheapscan incorrect recheck of NULL tuples

  2. Increase default maintenance_io_concurrency to 16

  3. Separate TBM[Shared|Private]Iterator and TBMIterateResult

  4. Improve read_stream.c advice for dense streams.

  5. Increase default effective_io_concurrency to 16

  6. Delay extraction of TIDBitmap per page offsets

  7. Add lossy indicator to TBMIterateResult

  8. Move BitmapTableScan per-scan setup into a helper

  9. Add and use BitmapHeapScanDescData struct

  10. Fix bitmap table scan crash on iterator release

  11. Bitmap Table Scans use unified TBMIterator

  12. Add common interface for TBMIterators

  13. Make table_scan_bitmap_next_block() async-friendly

  14. Move EXPLAIN counter increment to heapam_scan_bitmap_next_block

  15. Refactor tidstore.c iterator buffering.

  16. BitmapHeapScan: Remove incorrect assert and reset field

  17. Change BitmapAdjustPrefetchIterator to accept BlockNumber

  18. BitmapHeapScan: Use correct recheck flag for skip_fetch

  19. BitmapHeapScan: Push skip_fetch optimization into table AM

  20. BitmapHeapScan: postpone setting can_skip_fetch

  21. BitmapHeapScan: begin scan after bitmap creation

  22. Fix EXPLAIN Bitmap heap scan to count pages with no visible tuples

  23. Remove redundant snapshot copying from parallel leader to workers

  24. Remove some obsolete smgrcloseall() calls.

  25. Remove the "snapshot too old" feature.

  26. Compute XID horizon for page level index vacuum on primary.