Re: [POC] verifying UTF-8 using SIMD instructions
John Naylor <john.naylor@enterprisedb.com>
From: John Naylor <john.naylor@enterprisedb.com>
To: Heikki Linnakangas <hlinnaka@iki.fi>
Cc: pgsql-hackers <pgsql-hackers@postgresql.org>
Date: 2021-02-24T16:25:49Z
Lists: pgsql-hackers
Attachments
- v6-0001-Add-noError-argument-to-encoding-conversion-funct.patch (application/octet-stream) patch v6-0001
- v6-0002-Use-SSE-4-for-verifying-UTF-8-text.patch (application/octet-stream) patch v6-0002
- v6-0003-Add-an-ASCII-fast-path-to-the-fallback-UTF-8-vali.patch (application/octet-stream) patch v6-0003
- v6-0004-Widen-the-ASCII-fast-path-stride-in-the-fallback-.patch (application/octet-stream) patch v6-0004
The cfbot reported a build failure on Windows because of the use of binary literals. I've turned those into hex for v6, so let's see how far it gets now. I also decided to leave out the patch that adds an ascii fast path to non-UTF-8 encodings. That would really require more testing than I have time for. As before, 0001 is v4 of Heikk's noError conversion patch, whose regressions tests I build upon. 0002 has no ascii fast path in the fallback implementation. 0003 and 0004 add it back in using 8- and 16-byte strides, respectively. That will make it easier to test on non-Intel platforms, so we can decide which way to go here. Also did a round of editing the comments in the SSE4.2 file. I ran the multibyte conversion regression test found in the message below, and it passed. That doesn't test UTF-8 explicitly, but all conversions round-trip through UTF-8, so it does get some coverage. https://www.postgresql.org/message-id/b9e3167f-f84b-7aa4-5738-be578a4db924%40iki.fi -- John Naylor EDB: http://www.enterprisedb.com
Commits
Same data as JSON:
GET /api/v1/messages/:b64id/commits
the thread's linked commits as JSON, with link sources.
API reference →
-
Simplify coding style of is_valid_ascii()
- 6b41a1579bd3 16.0 landed
-
Apply auto-vectorization to the inner loop of numeric multiplication.
- 88709176236c 14.0 cited