Fix netmask handling in inet_minmax_multi_ops

Tomas Vondra <tomas.vondra@postgresql.org>

Commit: 6a78a42fea0c457dfee2a5ac5431df02342eafd2
Author: Tomas Vondra <tomas.vondra@postgresql.org>
Date: 2023-03-20T09:16:54Z
Releases: 14.8
Fix netmask handling in inet_minmax_multi_ops

When calculating distance in brin_minmax_multi_distance_inet(), the
netmask was applied incorrectly. This results in (seemingly) incorrect
ordering of values, triggering an assert.

For builds without asserts this is mostly harmless - we may merge other
ranges, possibly resulting in slightly less efficient index. But it's
still correct and the greedy algorithm doesn't guarantee optimality
anyway.

Backpatch to 14, where minmax-multi indexes were introduced.

Reported by Dmitry Dolgov, investigation and fix by me.

Reported-by: Dmitry Dolgov
Backpatch-through: 14
Discussion: https://postgr.es/m/17774-c6f3e36dd4471e67@postgresql.org

Files

Discussion