Horton

Re: BitmapHeapScan streaming read user and prelim refactoring

Alexander Law <exclusion@gmail.com>

View thread View on postgr.es
From: Alexander Lakhin <exclusion@gmail.com>
To: Melanie Plageman <melanieplageman@gmail.com>, Thomas Munro <thomas.munro@gmail.com>
Cc: Tomas Vondra <tomas@vondra.me>, Nazir Bilal Yavuz <byavuz81@gmail.com>, Dilip Kumar <dilipbalaut@gmail.com>, Heikki Linnakangas <hlinnaka@iki.fi>, Andres Freund <andres@anarazel.de>, Pg Hackers <pgsql-hackers@postgresql.org>
Date: 2025-03-22T20:00:00Z
Lists: pgsql-hackers 
Hello Melanie,

15.03.2025 16:43, Melanie Plageman wrote:
> On Thu, Mar 13, 2025 at 5:41 PM Melanie Plageman
> <melanieplageman@gmail.com> wrote:
>> Overall, I feel pretty good about merging this once Thomas merges the
>> read stream patches.
> This was committed in 944e81bf99db2b5b70b, 2b73a8cd33b745c, and
> c3953226a07527a1e2.
> I've marked it as committed in the CF app.

It looks like that change made the bitmapops test unstable (on slow animals?): [1], [2], [3].
I've reproduced such test failures when running
TESTS=$(printf 'bitmapops %.0s' `seq 50`) make -s check-tests
under Valgrind:
...
ok 17        - bitmapops                               52719 ms
not ok 18    - bitmapops                               57566 ms
ok 19        - bitmapops                               60179 ms
ok 20        - bitmapops                               32927 ms
ok 21        - bitmapops                               45127 ms
ok 22        - bitmapops                               42924 ms
ok 23        - bitmapops                               61035 ms
ok 24        - bitmapops                               56316 ms
ok 25        - bitmapops                               52874 ms
not ok 26    - bitmapops                               67468 ms
ok 27        - bitmapops                               55605 ms
ok 28        - bitmapops                               24021 ms
...

diff -U3 /home/vagrant/postgresql/src/test/regress/expected/bitmapops.out 
/home/vagrant/postgresql/src/test/regress/results/bitmapops.out
--- /home/vagrant/postgresql/src/test/regress/expected/bitmapops.out 2025-03-16 01:37:52.716885600 -0700
+++ /home/vagrant/postgresql/src/test/regress/results/bitmapops.out 2025-03-22 03:47:54.014702406 -0700
@@ -24,14 +24,14 @@
  SELECT count(*) FROM bmscantest WHERE a = 1 AND b = 1;
   count
  -------
-    23
+    18
  (1 row)

  -- Test bitmap-or.
  SELECT count(*) FROM bmscantest WHERE a = 1 OR b = 1;
   count
  -------
-  2485
+  1044
  (1 row)

  -- clean up
diff -U3 /home/vagrant/postgresql/src/test/regress/expected/bitmapops.out 
/home/vagrant/postgresql/src/test/regress/results/bitmapops.out
--- /home/vagrant/postgresql/src/test/regress/expected/bitmapops.out 2025-03-16 01:37:52.716885600 -0700
+++ /home/vagrant/postgresql/src/test/regress/results/bitmapops.out 2025-03-22 03:54:53.129549597 -0700
@@ -31,7 +31,7 @@
  SELECT count(*) FROM bmscantest WHERE a = 1 OR b = 1;
   count
  -------
-  2485
+  1044
  (1 row)

git bisect for this deviation has pointed at 2b73a8cd3.

[1] https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=skink&dt=2025-03-16%2010%3A34%3A17
[2] https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=flaviventris&dt=2025-03-17%2020%3A07%3A43
[3] https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=skink&dt=2025-03-17%2001%3A16%3A02

Best regards,
Alexander Lakhin
Neon (https://neon.tech)

Commits

  1. Fix bitmapheapscan incorrect recheck of NULL tuples

  2. Increase default maintenance_io_concurrency to 16

  3. Separate TBM[Shared|Private]Iterator and TBMIterateResult

  4. Improve read_stream.c advice for dense streams.

  5. Increase default effective_io_concurrency to 16

  6. Delay extraction of TIDBitmap per page offsets

  7. Add lossy indicator to TBMIterateResult

  8. Move BitmapTableScan per-scan setup into a helper

  9. Add and use BitmapHeapScanDescData struct

  10. Fix bitmap table scan crash on iterator release

  11. Bitmap Table Scans use unified TBMIterator

  12. Add common interface for TBMIterators

  13. Make table_scan_bitmap_next_block() async-friendly

  14. Move EXPLAIN counter increment to heapam_scan_bitmap_next_block

  15. Refactor tidstore.c iterator buffering.

  16. BitmapHeapScan: Remove incorrect assert and reset field

  17. Change BitmapAdjustPrefetchIterator to accept BlockNumber

  18. BitmapHeapScan: Use correct recheck flag for skip_fetch

  19. BitmapHeapScan: Push skip_fetch optimization into table AM

  20. BitmapHeapScan: postpone setting can_skip_fetch

  21. BitmapHeapScan: begin scan after bitmap creation

  22. Fix EXPLAIN Bitmap heap scan to count pages with no visible tuples

  23. Remove redundant snapshot copying from parallel leader to workers

  24. Remove some obsolete smgrcloseall() calls.

  25. Remove the "snapshot too old" feature.

  26. Compute XID horizon for page level index vacuum on primary.