Re: OpenSSL 3.0.0 compatibility

> On 11 Mar 2021, at 11:03, Peter Eisentraut <peter.eisentraut@enterprisedb.com> wrote:

> The ssl tests fail with a small error message difference that must have been introduced recently, because I think this was never reported before:

This was mentioned by Michael on 26/11, it was earlier in the 3.0.0 cycle
reported as "nested asn.1 error".  Waiting for 3.0.0 to go beta or GA before
changing saves us from having to change again should they update, but tests
will fail for anyone testing out new OpenSSL versions.

> The pgcrypto tests fail all over the place.  Some of these failures are quite likely because of the disabled legacy provider, but some appear to be due to bad error handling.

The below ones are likely due to the provider not being loaded, but as you say
they are probably cases of poor error handling as they fail in various places
while probably being due to the same root cause:

+ERROR:  encrypt error: Key was too big

+ERROR:  encrypt error: Cipher cannot be initialized ?

+ERROR:  Wrong key or corrupt data

Then there are a few where we get padding back where we really should have
ended up with the "Cipher cannot be initialized" error since DES is in the
legacy provider:

 select decrypt_iv(decode('50735067b073bb93', 'hex'), '0123456', 'abcd', 'des');
- decrypt_iv
-------------
- foo
+            decrypt_iv
+----------------------------------
+ \177\177\177\177\177\177\177\177
 (1 row)

> Then I tried enabling the legacy provider in openssl.cnf.  This caused pg_strong_random() to fail, which causes initdb to fail, like this:

Huh?  Enabling the legacy provider shouldn't affect the CRNG.  I see no such
failure, and haven't in any alpha version tested.  How did you change the
openssl config?  If one can break pg_strong_random with a config change then we
need defensive coding to cope with that.

> So, we knew pgcrypto was in trouble with openssl 3.0.0, but can someone else get its tests to pass with some kind of openssl.cnf configuration?

If I enable the legacy provider in openssl.cnf like this:

[openssl_init]
providers = provider_sect

[provider_sect]
default = default_sect
legacy = legacy_sect

[default_sect]
activate = 1

[legacy_sect]
activate = 1

.. and apply the padding changes as proposed in a patch upthread like this (these
work for all OpenSSL versions I've tested, and I'm rather more puzzled as to
why we got away with not having them in the past):

diff --git a/contrib/pgcrypto/openssl.c b/contrib/pgcrypto/openssl.c
index ed8e74a2b9..e236b0d79c 100644
--- a/contrib/pgcrypto/openssl.c
+++ b/contrib/pgcrypto/openssl.c
@@ -379,6 +379,8 @@ gen_ossl_decrypt(PX_Cipher *c, const uint8 *data, unsigned dlen,
        {
                if (!EVP_DecryptInit_ex(od->evp_ctx, od->evp_ciph, NULL, NULL, NULL))
                        return PXE_CIPHER_INIT;
+               if (!EVP_CIPHER_CTX_set_padding(od->evp_ctx, 0))
+                       return PXE_CIPHER_INIT;
                if (!EVP_CIPHER_CTX_set_key_length(od->evp_ctx, od->klen))
                        return PXE_CIPHER_INIT;
                if (!EVP_DecryptInit_ex(od->evp_ctx, NULL, NULL, od->key, od->iv))
@@ -403,6 +405,8 @@ gen_ossl_encrypt(PX_Cipher *c, const uint8 *data, unsigned dlen,
        {
                if (!EVP_EncryptInit_ex(od->evp_ctx, od->evp_ciph, NULL, NULL, NULL))
                        return PXE_CIPHER_INIT;
+               if (!EVP_CIPHER_CTX_set_padding(od->evp_ctx, 0))
+                       return PXE_CIPHER_INIT;
                if (!EVP_CIPHER_CTX_set_key_length(od->evp_ctx, od->klen))
                        return PXE_CIPHER_INIT;
                if (!EVP_EncryptInit_ex(od->evp_ctx, NULL, NULL, od->key, od->iv))

.. then all the pgcrypto tests pass for me as well as "make check", with a single
SSL test failing on the above mentioned PEM lib error message.

Did you build OpenSSL with anything non-standard?

--
Daniel Gustafsson		https://vmware.com/