Re: [POC] verifying UTF-8 using SIMD instructions

John Naylor <john.naylor@enterprisedb.com>

From: John Naylor <john.naylor@enterprisedb.com>
To: Heikki Linnakangas <hlinnaka@iki.fi>
Cc: pgsql-hackers <pgsql-hackers@postgresql.org>
Date: 2021-02-24T16:25:49Z
Lists: pgsql-hackers

Attachments

The cfbot reported a build failure on Windows because of the use of binary
literals. I've turned those into hex for v6, so let's see how far it gets
now.

I also decided to leave out the patch that adds an ascii fast path to
non-UTF-8 encodings. That would really require more testing than I have
time for.

As before, 0001 is v4 of Heikk's noError conversion patch, whose
regressions tests I build upon.

0002 has no ascii fast path in the fallback implementation. 0003 and 0004
add it back in using 8- and 16-byte strides, respectively. That will make
it easier to test on non-Intel platforms, so we can decide which way to go
here. Also did a round of editing the comments in the SSE4.2 file.

I ran the multibyte conversion regression test found in the message below,
and it passed. That doesn't test UTF-8 explicitly, but all conversions
round-trip through UTF-8, so it does get some coverage.

https://www.postgresql.org/message-id/b9e3167f-f84b-7aa4-5738-be578a4db924%40iki.fi
-- 
John Naylor
EDB: http://www.enterprisedb.com

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →
  1. Simplify coding style of is_valid_ascii()

  2. Apply auto-vectorization to the inner loop of numeric multiplication.