Re: [PATCH] Fix old thinko in formula to compute sweight in numeric_sqrt().

Joel Jacobson <joel@compiler.org>

From: "Joel Jacobson" <joel@compiler.org>
To: "Dean Rasheed" <dean.a.rasheed@gmail.com>
Cc: pgsql-hackers <pgsql-hackers@postgresql.org>
Date: 2023-01-31T21:59:10Z
Lists: pgsql-hackers

Attachments

On Tue, Jan 31, 2023, at 20:25, Dean Rasheed wrote:
> That seems a bit wordy, given the context of this comment. I think
> it's sufficient to just give the formula, and note that it simplifies
> when DEC_DIGITS is even (not just 4):
>
>     /*
>      * Assume the input was normalized, so arg.weight is accurate.  The result
>      * then has at least sweight = floor(arg.weight * DEC_DIGITS / 2 + 1)
>      * digits before the decimal point.  When DEC_DIGITS is even, we can save
>      * a few cycles, since the division is exact and there is no need to
>      *  round down.
>      */
> #if DEC_DIGITS == ((DEC_DIGITS / 2) * 2)
>     sweight = arg.weight * DEC_DIGITS / 2 + 1;
> #else
>     if (arg.weight >= 0)
>         sweight = arg.weight * DEC_DIGITS / 2 + 1;
>     else
>         sweight = 1 - (1 - arg.weight * DEC_DIGITS) / 2;
> #endif

Nice, you managed to simplify it even further.
I think the comment and the code now are crystal clear together.

I've tested it successfully, test report attached. In summary:
DEC_DIGITS=1 now produce 16 sig. figs. in the range sqrt(2e-31) .. sqrt(2e+32), which before had a mix of 17 and 18 sig. figs in the result.
DEC_DIGTIS=2 now produce 16 sig. figs. in the range sqrt(2e-31) .. sqrt(2e+31), which before always had 17 sig. figs in the result.
DEC_DIGITS=4 is unchanged.

Exact tested patch attached, code copy/pasted verbatim from your email.

Test 

Commits

  1. Clarify the choice of rscale in numeric_sqrt().

  2. Ensure that numeric.c compiles with other NBASE values.