Re: Fw:Re: Fw: ltree_compare in contrib/ltree/ltree_op.c overflows int32 on deep ltree comparisons, returning the wrong sign

On 13/06/2026 09:12, Ayush Tiwari wrote:
> This looks like a classic case of integer overflow that's
> happening in ltree_compare function in ltree_op.c.
> 
> return (al->len - bl->len) * 10 * (an + 1);
> return res * 10 * (an + 1);
> return (a->numlevel - b->numlevel) * 10 * (an + 1);
> 
> I think the calculation should be done as int64, something of this sort:
> 
>      int64 v = (int64) (al->len - bl->len) * 10 * (an + 1);
>      if (v > PG_INT32_MAX) return PG_INT32_MAX;
>      if (v < PG_INT32_MIN) return PG_INT32_MIN;
>      return (int) v;
> 
> And needed to adjust the ltree_penalty function too.
> 
> Attached is a draft patch for this, I guess we can add a helper
> function too for the above conversion.

Yeah, that works. However, I note that the multiplication is only really 
needed by the ltree_penalty() caller. All the other callers just check 
if the return value is less than, equal, or greater than zero. It feels 
a little silly to do all that work of multiplication and clamping for 
those callers. And for ltree_penalty(), the caller actually converts the 
return value to a float, so clamping it to int32 range feels a little 
silly for that too. So I propose the attached, which splits the 
ltree_compare() function into two variants: one for ltree_penalty() that 
returns a float, and one for others that don't care about the 
"magnitude". It duplicates a little code, but I think it's easier to 
reason about. What do you think?

- Heikki