Horton
Menu

Re: speed up verifying UTF-8

John Naylor <john.naylor@enterprisedb.com>

View thread
From: John Naylor <john.naylor@enterprisedb.com>
To: Vladimir Sitnikov <sitnikov.vladimir@gmail.com>
Cc: pgsql-hackers <pgsql-hackers@postgresql.org>, Amit Khandekar <amitdkhan.pg@gmail.com>, Heikki Linnakangas <hlinnaka@iki.fi>, Thomas Munro <thomas.munro@gmail.com>, Greg Stark <stark@mit.edu>
Date: 2021-08-26T15:35:54Z
Lists: pgsql-hackers

Attachments

I wrote:

> Naively, the shift-based DFA requires 64-bit integers to encode the
transitions, but I recently came across an idea from Dougall Johnson of
using the Z3 SMT solver to pack the transitions into 32-bit integers [1].
That halves the size of the transition table for free. I adapted that
effort to the existing conventions in v22 and arrived at the attached
python script.
> [...]
> I'll include something like the attached text file diff in the next
patch. Some comments are now outdated, but this is good enough for
demonstration.

Attached is v23 incorporating the 32-bit transition table, with the
necessary comment adjustments.

--
John Naylor
EDB: http://www.enterprisedb.com

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Simplify coding style of is_valid_ascii()

  2. Apply auto-vectorization to the inner loop of numeric multiplication.