Horton
Menu

Re: speed up verifying UTF-8

Heikki Linnakangas <hlinnaka@iki.fi>

View thread
From: Heikki Linnakangas <hlinnaka@iki.fi>
To: John Naylor <john.naylor@enterprisedb.com>
Cc: Greg Stark <stark@mit.edu>, Amit Khandekar <amitdkhan.pg@gmail.com>, pgsql-hackers <pgsql-hackers@postgresql.org>
Date: 2021-06-03T19:16:04Z
Lists: pgsql-hackers 
On 03/06/2021 22:10, John Naylor wrote:
> On Thu, Jun 3, 2021 at 3:08 PM Heikki Linnakangas <hlinnaka@iki.fi 
> <mailto:hlinnaka@iki.fi>> wrote:
>  >                 x1 = half1 + UINT64CONST(0x7f7f7f7f7f7f7f7f);
>  >                 x2 = half2 + UINT64CONST(0x7f7f7f7f7f7f7f7f);
>  >
>  >                 /* then check that the high bit is set in each byte. */
>  >                 x = (x1 | x2);
>  >                 x &= UINT64CONST(0x8080808080808080);
>  >                 if (x != UINT64CONST(0x8080808080808080))
>  >                         return 0;
> 
> That seems right, I'll try that and update the patch. (Forgot to attach 
> earlier anyway)

Ugh, actually that has the same issue as before. If one of the bytes is 
in one half is zero, but not in the other half, this fail to detect it. 
Sorry for the noise..

- Heikki

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Simplify coding style of is_valid_ascii()

  2. Apply auto-vectorization to the inner loop of numeric multiplication.