Cryptohash OpenSSL error queue in FIPS enabled builds

Daniel Gustafsson <daniel@yesql.se>

From: Daniel Gustafsson <daniel@yesql.se>
To: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2022-04-22T14:56:38Z
Lists: pgsql-hackers

Attachments

In trying out an OpenSSL 3.1 build with FIPS enabled I realized that our
cryptohash code had a small issue.  Calling a banned cipher generated two
different error messages interleaved:

  postgres=# select md5('foo');
  ERROR:  could not compute MD5 hash: unsupported
  postgres=# select md5('foo');
  ERROR:  could not compute MD5 hash: initialization error

It turns out that OpenSSL places two errors in the queue for this operation,
and we only consume one without clearing the queue in between, so we grab an
error from the previous run.

Consuming all (both) errors and creating a concatenated string seems overkill
as it would alter the API from a const error string to something that needs
freeing etc (also, very few OpenSSL consumers actually drain the queue, OpenSSL
themselves don't).  Skimming the OpenSSL code I was unable to find another
example of two errors generated.  The attached calls ERR_clear_error() as how
we do in libpq in order to avoid consuming earlier errors.

--
Daniel Gustafsson		https://vmware.com/

Commits

  1. pgcrypto: remove questionmark from error message

  2. pgcrypto: report init errors as PXE_CIPHER_INIT

  3. Clear the OpenSSL error queue before cryptohash operations