scram-sha-256 broken with FIPS and OpenSSL 1.0.2

Michael Paquier <michael@paquier.xyz>

From: Michael Paquier <michael@paquier.xyz>
To: Postgres hackers <pgsql-hackers@lists.postgresql.org>
Cc: Daniel Gustafsson <daniel@yesql.se>
Date: 2020-09-24T02:53:14Z
Lists: pgsql-hackers

Attachments

Hi all,

Enabling FIPS with OpenSSL 1.0.2 causes direct calls to the SHAXXX
routines to fail:
"Low level API call to digest SHA256 forbidden in fips mode"

This got discussed back in 2018, but I never got back to it:
https://www.postgresql.org/message-id/20180911030250.GA27115@paquier.xyz

One thing I did not like back in the past patch was that we did not
handle failures if one of OpenSSL's call failed, but this can easily
be handled by using a trick similar to jsonapi.c to fail hard if that
happens.

It is worth noting that the low-level SHA routines are not recommended
for years in OpenSSL, and that these have been officially marked as
deprecated in 3.0.0.  So, while the changes in sha2.h don't make this
stuff back-patchable per the ABI breakage it introduces, switching 
sha2_openssl.c to use EVP is a better move in the long term, even if
that means that SCRAM+FIPS would not work with PG 10~13, so the
attached is something for HEAD, even if this would be possible to do
in older releases as the routines used in the attached are available
in versions of OpenSSL older than 1.0.1.

Any thoughts?
--
Michael

Commits

  1. Change SHA2 implementation based on OpenSSL to use EVP digest routines

  2. Move SHA2 routines to a new generic API layer for crypto hashes

  3. Use OpenSSL EVP API for symmetric encryption in pgcrypto.