Horton
Menu

Re: speed up verifying UTF-8

John Naylor <john.naylor@enterprisedb.com>

View thread
From: John Naylor <john.naylor@enterprisedb.com>
To: Vladimir Sitnikov <sitnikov.vladimir@gmail.com>
Cc: pgsql-hackers <pgsql-hackers@postgresql.org>, Amit Khandekar <amitdkhan.pg@gmail.com>, Heikki Linnakangas <hlinnaka@iki.fi>, Thomas Munro <thomas.munro@gmail.com>, Greg Stark <stark@mit.edu>
Date: 2021-07-28T18:12:11Z
Lists: pgsql-hackers

Attachments

I wrote:

> On Mon, Jul 26, 2021 at 7:55 AM Vladimir Sitnikov <
sitnikov.vladimir@gmail.com> wrote:
> >
> > >+ utf8_advance(s, state, len);
> > >+
> > >+ /*
> > >+ * If we saw an error during the loop, let the caller handle it. We
treat
> > >+ * all other states as success.
> > >+ */
> > >+ if (state == ERR)
> > >+ return 0;
> >
> > Did you mean state = utf8_advance(s, state, len); there? (reassign
state variable)
>
> Yep, that's a bug, thanks for catching!

Fixed in v21, with a regression test added. Also, utf8_advance() now
directly changes state by a passed pointer rather than returning a value.
Some cosmetic changes:

s/valid_bytes/non_error_bytes/ since the former is kind of misleading now.

Some other var name and symbol changes. In my first DFA experiment, ASC
conflicted with the parser or scanner somehow, but it doesn't here, so it's
clearer to use this.

Rewrote a lot of comments about the state machine and regression tests.
--
John Naylor
EDB: http://www.enterprisedb.com

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Simplify coding style of is_valid_ascii()

  2. Apply auto-vectorization to the inner loop of numeric multiplication.