Fix netmask handling in inet_minmax_multi_ops

Tomas Vondra <tomas.vondra@postgresql.org>

Commit: 0c7726c2827e3ff685c460acd757a4b0c7ee09f7
Author: Tomas Vondra <tomas.vondra@postgresql.org>
Date: 2023-03-20T09:20:35Z
Releases: 15.3
Fix netmask handling in inet_minmax_multi_ops

When calculating distance in brin_minmax_multi_distance_inet(), the
netmask was applied incorrectly. This results in (seemingly) incorrect
ordering of values, triggering an assert.

For builds without asserts this is mostly harmless - we may merge other
ranges, possibly resulting in slightly less efficient index. But it's
still correct and the greedy algorithm doesn't guarantee optimality
anyway.

Backpatch to 14, where minmax-multi indexes were introduced.

Reported by Dmitry Dolgov, investigation and fix by me.

Reported-by: Dmitry Dolgov
Backpatch-through: 14
Discussion: https://postgr.es/m/17774-c6f3e36dd4471e67@postgresql.org

Files

Discussion