Horton

Thread

Commits

  1. pgcrypto: Make it possible to disable built-in crypto

  2. pgcrypto: Add function to check FIPS mode

  3. citext: Allow tests to pass in OpenSSL FIPS mode

  4. pgcrypto: Remove non-OpenSSL support

  1. Replace current implementations in crypt() and gen_salt() to OpenSSL

    Koshi Shibagaki (Fujitsu) <shibagaki.koshi@fujitsu.com> — 2024-02-15T12:42:26Z

    Hi
This is Shibagaki.

When FIPS mode is enabled, some encryption algorithms cannot be used.
Since PostgreSQL15, pgcrypto requires OpenSSL[1], digest() and other functions
also follow this policy.

However, crypt() and gen_salt() do not use OpenSSL as mentioned in [2].
Therefore, if we run crypt() and gen_salt() on a machine with FIPS mode enabled,
they are not affected by FIPS mode. This means we can use encryption algorithms 
disallowed in FIPS.

I would like to change the proprietary implementations of crypt() and gen_salt()
to use OpenSSL API.
If it's not a problem, I am going to create a patch, but if you have a better 
approach, please let me know.

Thank you


[1] https://github.com/postgres/postgres/commit/db7d1a7b0530e8cbd045744e1c75b0e63fb6916f
[2] https://peter.eisentraut.org/blog/2023/12/05/postgresql-and-fips-mode

crypt() and gen_salt() are performed on in example below.

/////

-- OS RHEL8.6

$openssl version
OpenSSL 1.1.1k  FIPS 25 Mar 2021

$fips-mode-setup --check
FIPS mode is enabled.

$./pgsql17/bin/psql
psql (17devel)
Type "help" for help.

postgres=# SHOW server_version;
 server_version 
----------------
 17devel
(1 row)

postgres=# SELECT digest('data','md5');
ERROR:  Cannot use "md5": Cipher cannot be initialized

postgres=# SELECT crypt('new password',gen_salt('md5')); -- md5 is not available when fips mode is turned on. This is a normal behavior
ERROR:  crypt(3) returned NULL

postgres=# SELECT crypt('new password',gen_salt('des')); -- however, des is avalable. This may break a FIPS rule
     crypt    
---------------
 32REGk7H6dSnE
(1 row)

/////

FYI - OpenSSL itself cannot use DES algorithm while encrypting files. This is an expected behavior.

-----------------------------------------------
Fujitsu Limited
Shibagaki Koshi
shibagaki.koshi@fujitsu.com

  2. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Peter Eisentraut <peter@eisentraut.org> — 2024-02-15T15:49:45Z

    On 15.02.24 13:42, Koshi Shibagaki (Fujitsu) wrote:
> However, crypt() and gen_salt() do not use OpenSSL as mentioned in [2].
> Therefore, if we run crypt() and gen_salt() on a machine with FIPS mode enabled,
> they are not affected by FIPS mode. This means we can use encryption algorithms
> disallowed in FIPS.
> 
> I would like to change the proprietary implementations of crypt() and gen_salt()
> to use OpenSSL API.
> If it's not a problem, I am going to create a patch, but if you have a better
> approach, please let me know.

The problems are:

1. All the block ciphers currently supported by crypt() and gen_salt() 
are not FIPS-compliant.

2. The crypt() and gen_salt() methods built on top of them (modes of 
operation, kind of) are not FIPS-compliant.

3. The implementations (crypt-blowfish.c, crypt-des.c, etc.) are not 
structured in a way that OpenSSL calls can easily be patched in.

So if you want FIPS-compliant cryptography, these interfaces look like a 
dead end.  I don't know if there are any modern equivalents of these 
functions that we should be supplying instead.

  3. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Daniel Gustafsson <daniel@yesql.se> — 2024-02-16T09:16:37Z

    > On 15 Feb 2024, at 16:49, Peter Eisentraut <peter@eisentraut.org> wrote:

> 1. All the block ciphers currently supported by crypt() and gen_salt() are not FIPS-compliant.
> 
> 2. The crypt() and gen_salt() methods built on top of them (modes of operation, kind of) are not FIPS-compliant.

I wonder if it's worth trying to make pgcrypto disallow non-FIPS compliant
ciphers when the compiled against OpenSSL is running with FIPS mode enabled, or
raise a WARNING when used?  It seems rather unlikely that someone running
OpenSSL with FIPS=yes want to use our DES cipher without there being an error
or misconfiguration somewhere.

Something like the below untested pseudocode.

diff --git a/contrib/pgcrypto/pgcrypto.c b/contrib/pgcrypto/pgcrypto.c
index 96447c5757..3d4391ebe1 100644
--- a/contrib/pgcrypto/pgcrypto.c
+++ b/contrib/pgcrypto/pgcrypto.c
@@ -187,6 +187,14 @@ pg_crypt(PG_FUNCTION_ARGS)
 			   *resbuf;
 	text	   *res;
 
+#if defined FIPS_mode
+	if (FIPS_mode())
+#else
+	if (EVP_default_properties_is_fips_enabled(OSSL_LIB_CTX_get0_global_default()))
+#endif
+		ereport(ERROR,
+				(errmsg("not available when using OpenSSL in FIPS mode")));
+
 	buf0 = text_to_cstring(arg0);
 	buf1 = text_to_cstring(arg1);

Greenplum implemented similar functionality but with a GUC, fips_mode=<bool>.
The problem with that is that it gives the illusion that enabling such a GUC
gives any guarantees about FIPS which isn't really the case since postgres
isn't FIPS certified.

--
Daniel Gustafsson

  4. RE: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Koshi Shibagaki (Fujitsu) <shibagaki.koshi@fujitsu.com> — 2024-02-16T11:32:44Z

    Dear Peter

Thanks for the replying

> 1. All the block ciphers currently supported by crypt() and gen_salt() are not
> FIPS-compliant.
>
> 2. The crypt() and gen_salt() methods built on top of them (modes of operation,
> kind of) are not FIPS-compliant.
> 
> 3. The implementations (crypt-blowfish.c, crypt-des.c, etc.) are not structured
> in a way that OpenSSL calls can easily be patched in.

Indeed, all the algorithm could not be used in FIPS and huge engineering might 
be needed for the replacement. If the benefit is smaller than the cost, we 
should consider another way - e.g., prohibit to call these functions in FIPS 
mode as in the pseudocode Daniel sent. Replacing OpenSSL is a way, the objective
is to eliminate the user's error in choosing an encryption algorithm.


-----------------------------------------------
Fujitsu Limited
Shibagaki Koshi
shibagaki.koshi@fujitsu.com

  5. RE: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Koshi Shibagaki (Fujitsu) <shibagaki.koshi@fujitsu.com> — 2024-02-16T11:35:41Z

    Dear Daniel

Thanks for your reply.

> I wonder if it's worth trying to make pgcrypto disallow non-FIPS compliant
> ciphers when the compiled against OpenSSL is running with FIPS mode
> enabled, or raise a WARNING when used?  It seems rather unlikely that
> someone running OpenSSL with FIPS=yes want to use our DES cipher without
> there being an error or misconfiguration somewhere.

Indeed, users do not use non-FIPS compliant ciphers in crypt() and gen_salt() 
such as DES with FIPS mode enabled.
However, can we reduce human error by having these functions make the judgment 
as to whether ciphers can or cannot be used?

If pgcrypto checks if FIPS enabled or not as in the pseudocode, it is easier to 
achieve than replacing to OpenSSL.
Currently, OpenSSL internally determines if it is in FIPS mode or not, but would
it be a problem to have PostgreSQL take on that role?

-----------------------------------------------
Fujitsu Limited
Shibagaki Koshi
shibagaki.koshi@fujitsu.com

  6. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Joe Conway <mail@joeconway.com> — 2024-02-16T11:56:13Z

    On 2/16/24 04:16, Daniel Gustafsson wrote:
>> On 15 Feb 2024, at 16:49, Peter Eisentraut <peter@eisentraut.org> wrote:
> 
>> 1. All the block ciphers currently supported by crypt() and gen_salt() are not FIPS-compliant.
>> 
>> 2. The crypt() and gen_salt() methods built on top of them (modes of operation, kind of) are not FIPS-compliant.
> 
> I wonder if it's worth trying to make pgcrypto disallow non-FIPS compliant
> ciphers when the compiled against OpenSSL is running with FIPS mode enabled, or
> raise a WARNING when used?  It seems rather unlikely that someone running
> OpenSSL with FIPS=yes want to use our DES cipher without there being an error
> or misconfiguration somewhere.
> 
> Something like the below untested pseudocode.
> 
> diff --git a/contrib/pgcrypto/pgcrypto.c b/contrib/pgcrypto/pgcrypto.c
> index 96447c5757..3d4391ebe1 100644
> --- a/contrib/pgcrypto/pgcrypto.c
> +++ b/contrib/pgcrypto/pgcrypto.c
> @@ -187,6 +187,14 @@ pg_crypt(PG_FUNCTION_ARGS)
>   			   *resbuf;
>   	text	   *res;
>   
> +#if defined FIPS_mode
> +	if (FIPS_mode())
> +#else
> +	if (EVP_default_properties_is_fips_enabled(OSSL_LIB_CTX_get0_global_default()))
> +#endif
> +		ereport(ERROR,
> +				(errmsg("not available when using OpenSSL in FIPS mode")));

Makes sense +1

-- 
Joe Conway
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  7. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Peter Eisentraut <peter@eisentraut.org> — 2024-02-16T12:57:56Z

    On 16.02.24 10:16, Daniel Gustafsson wrote:
>> 2. The crypt() and gen_salt() methods built on top of them (modes of operation, kind of) are not FIPS-compliant.
> I wonder if it's worth trying to make pgcrypto disallow non-FIPS compliant
> ciphers when the compiled against OpenSSL is running with FIPS mode enabled, or
> raise a WARNING when used?  It seems rather unlikely that someone running
> OpenSSL with FIPS=yes want to use our DES cipher without there being an error
> or misconfiguration somewhere.

I wonder on what level this kind of check would be done.  For example, 
the password hashing done for SCRAM is not FIPS-compliant either, but 
surely we don't want to disallow that.  Maybe this should be done on the 
level of block ciphers.  So if someone wanted to add a "crypt-aes" 
module, that would then continue to work.

  8. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Daniel Gustafsson <daniel@yesql.se> — 2024-02-16T13:30:38Z

    > On 16 Feb 2024, at 13:57, Peter Eisentraut <peter@eisentraut.org> wrote:
> 
> On 16.02.24 10:16, Daniel Gustafsson wrote:
>>> 2. The crypt() and gen_salt() methods built on top of them (modes of operation, kind of) are not FIPS-compliant.
>> I wonder if it's worth trying to make pgcrypto disallow non-FIPS compliant
>> ciphers when the compiled against OpenSSL is running with FIPS mode enabled, or
>> raise a WARNING when used?  It seems rather unlikely that someone running
>> OpenSSL with FIPS=yes want to use our DES cipher without there being an error
>> or misconfiguration somewhere.
> 
> I wonder on what level this kind of check would be done.  For example, the password hashing done for SCRAM is not FIPS-compliant either, but surely we don't want to disallow that.

Can you elaborate?  When building with OpenSSL all SCRAM hashing will use the
OpenSSL implementation of pg_hmac and pg_cryptohash, so it would be subject to
OpenSSL FIPS configuration no?

> Maybe this should be done on the level of block ciphers.  So if someone wanted to add a "crypt-aes" module, that would then continue to work.

That's a fair point, we can check individual ciphers.  I'll hack up a version
doing this.

--
Daniel Gustafsson

  9. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Peter Eisentraut <peter@eisentraut.org> — 2024-02-16T14:49:01Z

    On 16.02.24 14:30, Daniel Gustafsson wrote:
>> On 16 Feb 2024, at 13:57, Peter Eisentraut <peter@eisentraut.org> wrote:
>>
>> On 16.02.24 10:16, Daniel Gustafsson wrote:
>>>> 2. The crypt() and gen_salt() methods built on top of them (modes of operation, kind of) are not FIPS-compliant.
>>> I wonder if it's worth trying to make pgcrypto disallow non-FIPS compliant
>>> ciphers when the compiled against OpenSSL is running with FIPS mode enabled, or
>>> raise a WARNING when used?  It seems rather unlikely that someone running
>>> OpenSSL with FIPS=yes want to use our DES cipher without there being an error
>>> or misconfiguration somewhere.
>>
>> I wonder on what level this kind of check would be done.  For example, the password hashing done for SCRAM is not FIPS-compliant either, but surely we don't want to disallow that.
> 
> Can you elaborate?  When building with OpenSSL all SCRAM hashing will use the
> OpenSSL implementation of pg_hmac and pg_cryptohash, so it would be subject to
> OpenSSL FIPS configuration no?

Yes, but the overall methods of composing all this into secrets and 
protocol messages etc. are not covered by FIPS.

>> Maybe this should be done on the level of block ciphers.  So if someone wanted to add a "crypt-aes" module, that would then continue to work.
> 
> That's a fair point, we can check individual ciphers.  I'll hack up a version
> doing this.

Like, if we did a "crypt-aes", would that be FIPS-compliant?  I don't know.

  10. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Daniel Gustafsson <daniel@yesql.se> — 2024-02-16T15:09:59Z

    > On 16 Feb 2024, at 15:49, Peter Eisentraut <peter@eisentraut.org> wrote:

> Like, if we did a "crypt-aes", would that be FIPS-compliant?  I don't know.

If I remember my FIPS correct: Only if it used a FIPS certified implementation,
like the one in OpenSSL when the fips provider has been loaded.  The cipher
must be allowed *and* the implementation must be certified.

--
Daniel Gustafsson

  11. RE: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Koshi Shibagaki (Fujitsu) <shibagaki.koshi@fujitsu.com> — 2024-02-20T09:56:27Z

    Let me confirm the discussion in threads. I think there are two topics.
1. prohibit the use of ciphers disallowed in FIPS mode at the level of block 
cipher (crypt-bf, etc...) in crypt() and gen_salt()
2. adding new "crypt-aes" module.

If this is correct, I would like to make a patch for the first topic, as I think
I can handle it. 
Daniel, please let me know if you have been making a patch based on the idea.


Also, I think the second one should be discussed in a separate thread, so could 
you split it into a separate thread?

Thank you

-----------------------------------------------
Fujitsu Limited
Shibagaki Koshi
shibagaki.koshi@fujitsu.com

  12. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Daniel Gustafsson <daniel@yesql.se> — 2024-02-20T10:09:46Z

    > On 20 Feb 2024, at 10:56, Koshi Shibagaki (Fujitsu) <shibagaki.koshi@fujitsu.com> wrote:

> Let me confirm the discussion in threads. I think there are two topics.
> 1. prohibit the use of ciphers disallowed in FIPS mode at the level of block 
> cipher (crypt-bf, etc...) in crypt() and gen_salt()

That level might be overkill given that any cipher not in the FIPS certfied
module mustn't be used, but it's also not the wrong place to put it IMHO.

> 2. adding new "crypt-aes" module.

I think this was a hypothetical scenario and not a concrete proposal.

> If this is correct, I would like to make a patch for the first topic, as I think
> I can handle it. 
> Daniel, please let me know if you have been making a patch based on the idea.

I haven't yet started on that so feel free to take a stab at it, I'd be happy
to review it.  Note that there are different API's for doing this in OpenSSL
1.0.2 and OpenSSL 3.x, so a solution must take both into consideration.

--
Daniel Gustafsson

  13. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Peter Eisentraut <peter@eisentraut.org> — 2024-02-20T11:18:57Z

    On 20.02.24 11:09, Daniel Gustafsson wrote:
>> On 20 Feb 2024, at 10:56, Koshi Shibagaki (Fujitsu) <shibagaki.koshi@fujitsu.com> wrote:
> 
>> Let me confirm the discussion in threads. I think there are two topics.
>> 1. prohibit the use of ciphers disallowed in FIPS mode at the level of block
>> cipher (crypt-bf, etc...) in crypt() and gen_salt()
> 
> That level might be overkill given that any cipher not in the FIPS certfied
> module mustn't be used, but it's also not the wrong place to put it IMHO.

I think we are going about this the wrong way.  It doesn't make sense to 
ask OpenSSL what a piece of code that doesn't use OpenSSL should do. 
(And would that even give a sensible answer?  Like, you can configure 
OpenSSL to load the fips module, but you can also load the legacy module 
alongside it(??).)  And as you say, even if this code supported modern 
block ciphers, it wouldn't be FIPS compliant.

I think there are several less weird ways to address this:

* Just document it.

* Make a pgcrypto-level GUC setting.

* Split out these functions into a separate extension.

* Deprecate these functions.

Or some combination of these.

  14. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Robert Haas <robertmhaas@gmail.com> — 2024-02-20T11:27:02Z

    On Tue, Feb 20, 2024 at 4:49 PM Peter Eisentraut <peter@eisentraut.org> wrote:
> I think there are several less weird ways to address this:
>
> * Just document it.
>
> * Make a pgcrypto-level GUC setting.
>
> * Split out these functions into a separate extension.
>
> * Deprecate these functions.
>
> Or some combination of these.

I don't think the first two of these proposals help anything. AIUI,
FIPS mode is supposed to be a system wide toggle that affects
everything on the machine. The third one might help if you can be
compliant by just choosing not to install that extension, and the
fourth one solves the problem by sledgehammer.

Does Linux provide some way of asking whether "fips=1" was specified
at kernel boot time?

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  15. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Daniel Gustafsson <daniel@yesql.se> — 2024-02-20T11:39:37Z

    > On 20 Feb 2024, at 12:27, Robert Haas <robertmhaas@gmail.com> wrote:
> 
> On Tue, Feb 20, 2024 at 4:49 PM Peter Eisentraut <peter@eisentraut.org> wrote:
>> I think there are several less weird ways to address this:
>> 
>> * Just document it.
>> 
>> * Make a pgcrypto-level GUC setting.
>> 
>> * Split out these functions into a separate extension.
>> 
>> * Deprecate these functions.
>> 
>> Or some combination of these.
> 
> I don't think the first two of these proposals help anything. AIUI,
> FIPS mode is supposed to be a system wide toggle that affects
> everything on the machine. The third one might help if you can be
> compliant by just choosing not to install that extension, and the
> fourth one solves the problem by sledgehammer.

A fifth option is to throw away our in-tree implementations and use the OpenSSL
API's for everything, which is where this thread started.  If the effort to
payoff ratio is palatable to anyone then patches are for sure welcome.

> Does Linux provide some way of asking whether "fips=1" was specified
> at kernel boot time?

There is a crypto.fips_enabled sysctl but I have no idea how portable that is
across distributions etc.

--
Daniel Gustafsson

  16. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Daniel Gustafsson <daniel@yesql.se> — 2024-02-20T11:51:24Z

    > On 20 Feb 2024, at 12:18, Peter Eisentraut <peter@eisentraut.org> wrote:

> I think we are going about this the wrong way.  It doesn't make sense to ask OpenSSL what a piece of code that doesn't use OpenSSL should do.

Given that pgcrypto cannot be built without OpenSSL, and ideally we should be
using the OpenSSL implementations for everything, I don't think it's too far
fetched.

--
Daniel Gustafsson

  17. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Robert Haas <robertmhaas@gmail.com> — 2024-02-20T12:24:49Z

    On Tue, Feb 20, 2024 at 5:09 PM Daniel Gustafsson <daniel@yesql.se> wrote:
> A fifth option is to throw away our in-tree implementations and use the OpenSSL
> API's for everything, which is where this thread started.  If the effort to
> payoff ratio is palatable to anyone then patches are for sure welcome.

That generally seems fine, although I'm fuzzy on what our policy
actually is. We have fallback implementations for some things and not
others, IIRC.

> > Does Linux provide some way of asking whether "fips=1" was specified
> > at kernel boot time?
>
> There is a crypto.fips_enabled sysctl but I have no idea how portable that is
> across distributions etc.

My guess would be that it's pretty portable, but my guesses about
Linux might not be very good. Still, if we wanted to go this route, it
probably wouldn't be too hard to figure out how portable this is.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  18. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Peter Eisentraut <peter@eisentraut.org> — 2024-02-20T12:34:04Z

    On 20.02.24 12:27, Robert Haas wrote:
> I don't think the first two of these proposals help anything. AIUI,
> FIPS mode is supposed to be a system wide toggle that affects
> everything on the machine. The third one might help if you can be
> compliant by just choosing not to install that extension, and the
> fourth one solves the problem by sledgehammer.
> 
> Does Linux provide some way of asking whether "fips=1" was specified
> at kernel boot time?

What you are describing only happens on Red Hat systems, I think.  They 
have built additional integration around this, which is great.  But 
that's not something you can rely on being the case on all systems, not 
even all Linux systems.

  19. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Daniel Gustafsson <daniel@yesql.se> — 2024-02-20T12:35:02Z

    > On 20 Feb 2024, at 13:24, Robert Haas <robertmhaas@gmail.com> wrote:
> 
> On Tue, Feb 20, 2024 at 5:09 PM Daniel Gustafsson <daniel@yesql.se> wrote:
>> A fifth option is to throw away our in-tree implementations and use the OpenSSL
>> API's for everything, which is where this thread started.  If the effort to
>> payoff ratio is palatable to anyone then patches are for sure welcome.
> 
> That generally seems fine, although I'm fuzzy on what our policy
> actually is. We have fallback implementations for some things and not
> others, IIRC.

I'm not sure there is a well-formed policy, but IIRC the idea with cryptohash
was to provide in-core functionality iff OpenSSL isn't used, and only use the
OpenSSL implementations if it is.  Since pgcrypto cannot be built without
OpenSSL (since db7d1a7b0530e8cbd045744e1c75b0e63fb6916f) I don't think it's a
problem to continue the work from that commit and replace more with OpenSSL
implementations.

--
Daniel Gustafsson

  20. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Peter Eisentraut <peter@eisentraut.org> — 2024-02-20T12:40:27Z

    On 20.02.24 12:39, Daniel Gustafsson wrote:
> A fifth option is to throw away our in-tree implementations and use the OpenSSL
> API's for everything, which is where this thread started.  If the effort to
> payoff ratio is palatable to anyone then patches are for sure welcome.

The problem is that, as I understand it, these crypt routines are not 
designed in a way that you can just plug in a crypto library underneath. 
  Effectively, the definition of what, say, blowfish crypt does, is 
whatever is in that source file, and transitively, whatever OpenBSD 
does.  (Fun question: Does OpenBSD care about FIPS?)  Of course, you 
could reimplement the same algorithms independently, using OpenSSL or 
whatever.  But I don't think this will really improve the state of the 
world in aggregate, because to a large degree we are relying on the 
upstream to keep these implementations maintained, and if we rewrite 
them, we become the upstream.

  21. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Daniel Gustafsson <daniel@yesql.se> — 2024-02-20T14:52:36Z

    > On 20 Feb 2024, at 13:40, Peter Eisentraut <peter@eisentraut.org> wrote:
> 
> On 20.02.24 12:39, Daniel Gustafsson wrote:
>> A fifth option is to throw away our in-tree implementations and use the OpenSSL
>> API's for everything, which is where this thread started.  If the effort to
>> payoff ratio is palatable to anyone then patches are for sure welcome.
> 
> The problem is that, as I understand it, these crypt routines are not designed in a way that you can just plug in a crypto library underneath.  Effectively, the definition of what, say, blowfish crypt does, is whatever is in that source file, and transitively, whatever OpenBSD does.  

I don't disagree, but if the OP is willing to take a stab at it then..

> (Fun question: Does OpenBSD care about FIPS?)

No, LibreSSL ripped out FIPS support early on.

>  Of course, you could reimplement the same algorithms independently, using OpenSSL or whatever.  But I don't think this will really improve the state of the world in aggregate, because to a large degree we are relying on the upstream to keep these implementations maintained, and if we rewrite them, we become the upstream.

As a sidenote, we are already trailing behind upstream on this, the patch in
[0] sits on my TODO, but given the lack of complaints over the years it's not
been bumped to the top.

--
Daniel Gustafsson

[0] https://www.postgresql.org/message-id/flat/CAA-7PziyARoKi_9e2xdC75RJ068XPVk1CHDDdscu2BGrPuW9TQ%40mail.gmail.com#b20783dd6c72e95a8a0f6464d1228ed5

  22. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Joe Conway <mail@joeconway.com> — 2024-10-26T18:10:29Z

    >> On 20 Feb 2024, at 13:40, Peter Eisentraut <peter@eisentraut.org> wrote:
>> On 20.02.24 12:39, Daniel Gustafsson wrote:
>>> A fifth option is to throw away our in-tree implementations and use the OpenSSL
>>> API's for everything, which is where this thread started.  If the effort to
>>> payoff ratio is palatable to anyone then patches are for sure welcome.
>> The problem is that, as I understand it, these crypt routines are
>> not designed in a way that you can just plug in a crypto library
>> underneath.  Effectively, the definition of what, say, blowfish
>> crypt does, is whatever is in that source file, and transitively,
>> whatever OpenBSD does.


Someone asked me about this issue a few days ago which led me back to 
this unresolved thread.


> diff --git a/contrib/pgcrypto/pgcrypto.c b/contrib/pgcrypto/pgcrypto.c
> index 96447c5757..3d4391ebe1 100644
> --- a/contrib/pgcrypto/pgcrypto.c
> +++ b/contrib/pgcrypto/pgcrypto.c
> @@ -187,6 +187,14 @@ pg_crypt(PG_FUNCTION_ARGS)
>  			   *resbuf;
>  	text	   *res;
>  
> +#if defined FIPS_mode
> +	if (FIPS_mode())
> +#else
> +	if (EVP_default_properties_is_fips_enabled(OSSL_LIB_CTX_get0_global_default()))
> +#endif
> +		ereport(ERROR,
> +				(errmsg("not available when using OpenSSL in FIPS mode")));
> +
>  	buf0 = text_to_cstring(arg0);
>  	buf1 = text_to_cstring(arg1);
> 
> 
> Greenplum implemented similar functionality but with a GUC, fips_mode=<bool>.
> The problem with that is that it gives the illusion that enabling such a GUC
> gives any guarantees about FIPS which isn't really the case since postgres
> isn't FIPS certified.


Rather than depend on figuring out if we are in FIPS_mode in a portable 
way, I think the GUC is simpler and sufficient. Why not do that and just 
use a better name, e.g. legacy_crypto_enabled or something similar 
(bike-shedding welcomed) as in the attached.

-- 
Joe Conway
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  23. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Daniel Gustafsson <daniel@yesql.se> — 2024-10-29T09:57:28Z

    > On 26 Oct 2024, at 20:10, Joe Conway <mail@joeconway.com> wrote:

> Rather than depend on figuring out if we are in FIPS_mode in a portable way, I think the GUC is simpler and sufficient. Why not do that and just use a better name, e.g. legacy_crypto_enabled or something similar (bike-shedding welcomed) as in the attached.

I'm not very enthusiastic about adding a GUC to match a system property like
that for the same reason why we avoid GUCs with transitive dependencies.

Re-reading the thread and thinking about I think the best solution would be to
split these functions off into their own extension.

--
Daniel Gustafsson

  24. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Joe Conway <mail@joeconway.com> — 2024-10-29T12:53:10Z

    On 10/29/24 05:57, Daniel Gustafsson wrote:
>> On 26 Oct 2024, at 20:10, Joe Conway <mail@joeconway.com> wrote:
> 
>> Rather than depend on figuring out if we are in FIPS_mode in a portable way, I think the GUC is simpler and sufficient. Why not do that and just use a better name, e.g. legacy_crypto_enabled or something similar (bike-shedding welcomed) as in the attached.
> 
> I'm not very enthusiastic about adding a GUC to match a system property like
> that for the same reason why we avoid GUCs with transitive dependencies.
> 
> Re-reading the thread and thinking about I think the best solution would be to
> split these functions off into their own extension.


Seems like that would be an issue for backward comparability and upgrades.

-- 
Joe Conway
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  25. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Daniel Gustafsson <daniel@yesql.se> — 2024-10-29T14:08:42Z

    > On 29 Oct 2024, at 13:53, Joe Conway <mail@joeconway.com> wrote:
> 
> On 10/29/24 05:57, Daniel Gustafsson wrote:
>>> On 26 Oct 2024, at 20:10, Joe Conway <mail@joeconway.com> wrote:
>>> Rather than depend on figuring out if we are in FIPS_mode in a portable way, I think the GUC is simpler and sufficient. Why not do that and just use a better name, e.g. legacy_crypto_enabled or something similar (bike-shedding welcomed) as in the attached.
>> I'm not very enthusiastic about adding a GUC to match a system property like
>> that for the same reason why we avoid GUCs with transitive dependencies.
>> Re-reading the thread and thinking about I think the best solution would be to
>> split these functions off into their own extension.
> 
> Seems like that would be an issue for backward comparability and upgrades.

That's undoubtedly a downside of this proposal which the GUC proposal doesn't have.
--
Daniel Gustafsson

  26. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Jacob Champion <jacob.champion@enterprisedb.com> — 2024-10-29T16:52:36Z

    On Sat, Oct 26, 2024 at 11:10 AM Joe Conway <mail@joeconway.com> wrote:
> Rather than depend on figuring out if we are in FIPS_mode in a portable
> way, I think the GUC is simpler and sufficient. Why not do that and just
> use a better name, e.g. legacy_crypto_enabled or something similar
> (bike-shedding welcomed) as in the attached.

While it's definitely simpler, I read the original request as "forbid
non-FIPS crypto if the system tells us to", and I don't think this
proposal does that.

--Jacob

  27. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Joe Conway <mail@joeconway.com> — 2024-10-29T17:34:08Z

    On 10/29/24 12:52, Jacob Champion wrote:
> On Sat, Oct 26, 2024 at 11:10 AM Joe Conway <mail@joeconway.com> wrote:
>> Rather than depend on figuring out if we are in FIPS_mode in a portable
>> way, I think the GUC is simpler and sufficient. Why not do that and just
>> use a better name, e.g. legacy_crypto_enabled or something similar
>> (bike-shedding welcomed) as in the attached.
> 
> While it's definitely simpler, I read the original request as "forbid
> non-FIPS crypto if the system tells us to", and I don't think this
> proposal does that.


No, you just have to be able to configure the system in such a way that 
the non-FIPs crypto is not available. This is entirely sufficient for that.


-- 
Joe Conway
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  28. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Joe Conway <mail@joeconway.com> — 2024-11-19T17:30:39Z

    On 10/29/24 10:08, Daniel Gustafsson wrote:
>> On 29 Oct 2024, at 13:53, Joe Conway <mail@joeconway.com> wrote:
>> On 10/29/24 05:57, Daniel Gustafsson wrote:
>>>> On 26 Oct 2024, at 20:10, Joe Conway <mail@joeconway.com> wrote:

>>>> Rather than depend on figuring out if we are in FIPS_mode in a
>>>> portable way, I think the GUC is simpler and sufficient. Why
>>>> not do that and just use a better name, e.g.
>>>> legacy_crypto_enabled or something similar (bike-shedding
>>>> welcomed) as in the attached.

>>> I'm not very enthusiastic about adding a GUC to match a system property like
>>> that for the same reason why we avoid GUCs with transitive dependencies.
>>> Re-reading the thread and thinking about I think the best solution would be to
>>> split these functions off into their own extension.

>> Seems like that would be an issue for backward comparability and upgrades.

> That's undoubtedly a downside of this proposal which the GUC proposal doesn't have.

Any other opinions out there?

-- 
Joe Conway
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  29. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Robert Haas <robertmhaas@gmail.com> — 2024-11-19T23:12:34Z

    On Tue, Nov 19, 2024 at 12:30 PM Joe Conway <mail@joeconway.com> wrote:
> Any other opinions out there?

Why should we accept your patch (which adds a legacy_crypto_enabled
GUC) instead of adopting the approach originally proposed (i.e. use
the OpenSSL version of the functions)? It seems to me that the two
proposals accomplish the same thing, except that one of them also adds
a GUC.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  30. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Daniel Gustafsson <daniel@yesql.se> — 2024-11-20T17:14:44Z

    > On 19 Nov 2024, at 18:30, Joe Conway <mail@joeconway.com> wrote:

> Any other opinions out there?

Couldn't installations who would be satisfied with a GUC gate revoke privileges
from the relevant functions already today and achieve almost the same result?

--
Daniel Gustafsson

  31. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Joe Conway <mail@joeconway.com> — 2024-11-21T19:06:11Z

    On 11/19/24 18:12, Robert Haas wrote:
> On Tue, Nov 19, 2024 at 12:30 PM Joe Conway <mail@joeconway.com> wrote:
>> Any other opinions out there?
> 
> Why should we accept your patch (which adds a legacy_crypto_enabled
> GUC) instead of adopting the approach originally proposed (i.e. use
> the OpenSSL version of the functions)?


Because that idea was rejected earlier in the thread by multiple people 
other than me?

   ¯\_(ツ)_/¯


-- 
Joe Conway
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  32. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Joe Conway <mail@joeconway.com> — 2024-11-21T19:11:24Z

    On 11/20/24 12:14, Daniel Gustafsson wrote:
>> On 19 Nov 2024, at 18:30, Joe Conway <mail@joeconway.com> wrote:
> 
>> Any other opinions out there?
> 
> Couldn't installations who would be satisfied with a GUC gate revoke privileges
> from the relevant functions already today and achieve almost the same result?

I think that would qualify as a "mitigation" but not "FIPS compliant".

When the OS is made FIPS compliant, for example, you run something on 
the command line and then you need to reboot (RHEL at least). I believe 
that is considered configuration for FIPS.

A postmaster GUC (requiring restart) would be a way to configure 
Postgres to eliminate these two non-FIPS functions that could not be 
undone without another restart (similar to the OS example above).

-- 
Joe Conway
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  33. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Robert Haas <robertmhaas@gmail.com> — 2024-11-21T20:43:58Z

    On Thu, Nov 21, 2024 at 2:06 PM Joe Conway <mail@joeconway.com> wrote:
> Because that idea was rejected earlier in the thread by multiple people
> other than me?
>
>    ¯\_(ツ)_/¯

I actually don't see that in the thread anywhere. Which messages are
you talking about?

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  34. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Joe Conway <mail@joeconway.com> — 2024-11-21T21:39:58Z

    On 11/21/24 15:43, Robert Haas wrote:
> On Thu, Nov 21, 2024 at 2:06 PM Joe Conway <mail@joeconway.com> wrote:
>> Because that idea was rejected earlier in the thread by multiple people
>> other than me?
>>
>>    ¯\_(ツ)_/¯
> 
> I actually don't see that in the thread anywhere. Which messages are
> you talking about?


Well there is this:
> From: 	Peter Eisentraut <peter(at)eisentraut(dot)org>

> On 15.02.24 13:42, Koshi Shibagaki (Fujitsu) wrote:
>> However, crypt() and gen_salt() do not use OpenSSL as mentioned in [2].
>> Therefore, if we run crypt() and gen_salt() on a machine with FIPS mode enabled,
>> they are not affected by FIPS mode. This means we can use encryption algorithms
>> disallowed in FIPS.
>> 
>> I would like to change the proprietary implementations of crypt() and gen_salt()
>> to use OpenSSL API.
>> If it's not a problem, I am going to create a patch, but if you have a better
>> approach, please let me know.
> 
> The problems are:
> 
> 1. All the block ciphers currently supported by crypt() and gen_salt() 
> are not FIPS-compliant.
> 
> 2. The crypt() and gen_salt() methods built on top of them (modes of 
> operation, kind of) are not FIPS-compliant.
> 
> 3. The implementations (crypt-blowfish.c, crypt-des.c, etc.) are not 
> structured in a way that OpenSSL calls can easily be patched in.
> 
> So if you want FIPS-compliant cryptography, these interfaces look like a 
> dead end.  I don't know if there are any modern equivalents of these 
> functions that we should be supplying instead.

and this

> From: 	Peter Eisentraut <peter(at)eisentraut(dot)org>
> 
> On 20.02.24 12:39, Daniel Gustafsson wrote:
>> A fifth option is to throw away our in-tree implementations and use the OpenSSL
>> API's for everything, which is where this thread started.  If the effort to
>> payoff ratio is palatable to anyone then patches are for sure welcome.
> 
> 
> The problem is that, as I understand it, these crypt routines are not 
> designed in a way that you can just plug in a crypto library underneath. 
>   Effectively, the definition of what, say, blowfish crypt does, is 
> whatever is in that source file, and transitively, whatever OpenBSD 
> does.  (Fun question: Does OpenBSD care about FIPS?)  Of course, you 
> could reimplement the same algorithms independently, using OpenSSL or 
> whatever.  But I don't think this will really improve the state of the 
> world in aggregate, because to a large degree we are relying on the 
> upstream to keep these implementations maintained, and if we rewrite 
> them, we become the upstream.

And Daniel proposed this:

> From: 	Daniel Gustafsson <daniel(at)yesql(dot)se>

>> On 15 Feb 2024, at 16:49, Peter Eisentraut <peter(at)eisentraut(dot)org> wrote:
> 
>> 1. All the block ciphers currently supported by crypt() and gen_salt() are not FIPS-compliant.
>> 
>> 2. The crypt() and gen_salt() methods built on top of them (modes of operation, kind of) are not FIPS-compliant.
> 
> I wonder if it's worth trying to make pgcrypto disallow non-FIPS compliant
> ciphers when the compiled against OpenSSL is running with FIPS mode enabled, or
> raise a WARNING when used?  It seems rather unlikely that someone running
> OpenSSL with FIPS=yes want to use our DES cipher without there being an error
> or misconfiguration somewhere.

and then there is this:

> From: 	"Koshi Shibagaki (Fujitsu)" <shibagaki(dot)koshi(at)fujitsu(dot)com>

> Dear Daniel
> 
> Thanks for your reply.
> 
>> I wonder if it's worth trying to make pgcrypto disallow non-FIPS compliant
>> ciphers when the compiled against OpenSSL is running with FIPS mode
>> enabled, or raise a WARNING when used?  It seems rather unlikely that
>> someone running OpenSSL with FIPS=yes want to use our DES cipher without
>> there being an error or misconfiguration somewhere.
> 
> Indeed, users do not use non-FIPS compliant ciphers in crypt() and gen_salt() 
> such as DES with FIPS mode enabled.
> However, can we reduce human error by having these functions make the judgment 
> as to whether ciphers can or cannot be used?
> 
> If pgcrypto checks if FIPS enabled or not as in the pseudocode, it is easier to 
> achieve than replacing to OpenSSL.
> Currently, OpenSSL internally determines if it is in FIPS mode or not, but would
> it be a problem to have PostgreSQL take on that role?


I mean, perhaps I am misreading and/or interpreting all of that 
differently to you, but from my reading of the entire thread there was 
clearly no consensus to using openssl to provide those two functions.

Frankly I don't care which solution is picked as long as we pick 
something that allows users of pgcrypto to be FIPS complaint, and we 
don't drag this out past pg18 feature freeze because there are too many 
opinions and no one is willing to take a stand.

So the patch I posted is my attempt to take a stand. If you have a 
better patch you would like to propose to fix this problem, please do.

-- 
Joe Conway
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  35. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Daniel Gustafsson <daniel@yesql.se> — 2024-11-22T14:11:55Z

    > On 21 Nov 2024, at 22:39, Joe Conway <mail@joeconway.com> wrote:

> I mean, perhaps I am misreading and/or interpreting all of that differently to you, but from my reading of the entire thread there was clearly no consensus to using openssl to provide those two functions.

My interpretation (or perhaps, my opinion) is that it would be ideal to
reimplement these functions using OpenSSL *if possible* but the cost/benefit
ratio is probably tilted such that it will never happen.

> [..] we don't drag this out past pg18 feature freeze

Agreed.

> If you have a better patch you would like to propose to fix this problem,
> please do.

I'm still not thrilled about having a transitive dependency GUC, so attached is
a (very lightly tested POC) version of your patch which expands it from boolean
to enum with on/off/fips; the fips value being "disable if openssl is in fips
mode, else enable".  I'm not sure if that's better, but at least it gives users
a way to control the FIPS mode setting in one place and have crypto consumers
follow the set value (or they can explicitly turn it off if they just want them
disabled even without FIPS).

--
Daniel Gustafsson

  36. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Robert Haas <robertmhaas@gmail.com> — 2024-11-22T17:09:10Z

    On Thu, Nov 21, 2024 at 4:39 PM Joe Conway <mail@joeconway.com> wrote:
> I mean, perhaps I am misreading and/or interpreting all of that
> differently to you, but from my reading of the entire thread there was
> clearly no consensus to using openssl to provide those two functions.

OK, I see the problem now. I don't interpret those messages as
opposing the idea of making this use OpenSSL, but they do say it would
be hard to implement, which is a problem.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  37. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Joe Conway <mail@joeconway.com> — 2024-11-23T16:13:37Z

    On 11/22/24 09:11, Daniel Gustafsson wrote:
>> On 21 Nov 2024, at 22:39, Joe Conway <mail@joeconway.com> wrote:
> 
>> I mean, perhaps I am misreading and/or interpreting all of that differently to you, but from my reading of the entire thread there was clearly no consensus to using openssl to provide those two functions.
> 
> My interpretation (or perhaps, my opinion) is that it would be ideal to
> reimplement these functions using OpenSSL *if possible* but the cost/benefit
> ratio is probably tilted such that it will never happen.
> 
>> [..] we don't drag this out past pg18 feature freeze
> 
> Agreed.
> 
>> If you have a better patch you would like to propose to fix this problem,
>> please do.
> 
> I'm still not thrilled about having a transitive dependency GUC, so attached is
> a (very lightly tested POC) version of your patch which expands it from boolean
> to enum with on/off/fips; the fips value being "disable if openssl is in fips
> mode, else enable".  I'm not sure if that's better, but at least it gives users
> a way to control the FIPS mode setting in one place and have crypto consumers
> follow the set value (or they can explicitly turn it off if they just want them
> disabled even without FIPS).

Works for me.

I do wonder if the GUC should be PGC_POSTMASTER (as I had suggested it 
ought to be in an earlier post) rather than PGC_SUSET (which was the way 
my posted patch had it). But perhaps PGC_SUSET is sufficient, and it 
makes testing easier.

One other question this spawned -- do we document the minimum supported 
version of OpenSSL anywhere? I remembered it had recently been 
increased, but could only find confirmation in the git logs that 1.1.1 
was now the minimum.

-- 
Joe Conway
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  38. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Daniel Gustafsson <daniel@yesql.se> — 2024-11-24T13:48:14Z

    > On 23 Nov 2024, at 17:13, Joe Conway <mail@joeconway.com> wrote:

> I do wonder if the GUC should be PGC_POSTMASTER (as I had suggested it ought to be in an earlier post) rather than PGC_SUSET (which was the way my posted patch had it). But perhaps PGC_SUSET is sufficient, and it makes testing easier.

I copied PGC_SUSET from your patch, since I think it seems sufficient for this.

> One other question this spawned -- do we document the minimum supported version of OpenSSL anywhere? I remembered it had recently been increased, but could only find confirmation in the git logs that 1.1.1 was now the minimum.

It's documented under installation requirements in the docs, where the 17 docs
currently state 1.0.2 as the minimum:

	https://www.postgresql.org/docs/devel/install-requirements.html

--
Daniel Gustafsson

  39. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Daniel Gustafsson <daniel@yesql.se> — 2024-12-03T13:59:25Z

    > On 24 Nov 2024, at 14:48, Daniel Gustafsson <daniel@yesql.se> wrote:

Any other opinions or should we proceed with the proposed GUC?

--
Daniel Gustafsson

  40. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Joe Conway <mail@joeconway.com> — 2024-12-03T14:04:32Z

    On 12/3/24 08:59, Daniel Gustafsson wrote:
>> On 24 Nov 2024, at 14:48, Daniel Gustafsson <daniel@yesql.se> wrote:
> 
> Any other opinions or should we proceed with the proposed GUC?


I'm good with it. Did you want to commit or would you rather I do it?

-- 
Joe Conway
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  41. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Daniel Gustafsson <daniel@yesql.se> — 2024-12-03T21:56:29Z

    > On 3 Dec 2024, at 15:04, Joe Conway <mail@joeconway.com> wrote:
> 
> On 12/3/24 08:59, Daniel Gustafsson wrote:
>>> On 24 Nov 2024, at 14:48, Daniel Gustafsson <daniel@yesql.se> wrote:
>> Any other opinions or should we proceed with the proposed GUC?
> 
> I'm good with it. Did you want to commit or would you rather I do it?

No worries, I can make it happen.  ssnce there has been no objections since
this patch was posted I'll get it commmitted shortly.

--
Daniel Gustafsson

  42. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Daniel Gustafsson <daniel@yesql.se> — 2024-12-04T13:54:17Z

    > On 3 Dec 2024, at 22:56, Daniel Gustafsson <daniel@yesql.se> wrote:
> 
>> On 3 Dec 2024, at 15:04, Joe Conway <mail@joeconway.com> wrote:
>> 
>> On 12/3/24 08:59, Daniel Gustafsson wrote:
>>>> On 24 Nov 2024, at 14:48, Daniel Gustafsson <daniel@yesql.se> wrote:
>>> Any other opinions or should we proceed with the proposed GUC?
>> 
>> I'm good with it. Did you want to commit or would you rather I do it?
> 
> No worries, I can make it happen.  ssnce there has been no objections since
> this patch was posted I'll get it commmitted shortly.

Looking over this again I realized it's a bit silly to fall back on FIPS_mode()
when EVP_default_properties_is_fips_enabled isn't available since that would
only be OpenSSL versions before 3.0 (and since we don't support 1.0.2 then no
such version can have FIPS).  Sharing back a v3 which is what I think we should
go with.

--
Daniel Gustafsson

  43. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Robert Haas <robertmhaas@gmail.com> — 2024-12-04T14:28:06Z

    On Wed, Dec 4, 2024 at 8:54 AM Daniel Gustafsson <daniel@yesql.se> wrote:
> Looking over this again I realized it's a bit silly to fall back on FIPS_mode()
> when EVP_default_properties_is_fips_enabled isn't available since that would
> only be OpenSSL versions before 3.0 (and since we don't support 1.0.2 then no
> such version can have FIPS).  Sharing back a v3 which is what I think we should
> go with.

The comment suggests to me that if the user happened to be using
OpenSSL 1.1.1 and CheckLegacyCryptoMode() was called, the expected
outcome would be an error, but it will just return.

Am I confused?

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  44. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Daniel Gustafsson <daniel@yesql.se> — 2024-12-04T14:33:46Z

    > On 4 Dec 2024, at 15:28, Robert Haas <robertmhaas@gmail.com> wrote:
> 
> On Wed, Dec 4, 2024 at 8:54 AM Daniel Gustafsson <daniel@yesql.se> wrote:
>> Looking over this again I realized it's a bit silly to fall back on FIPS_mode()
>> when EVP_default_properties_is_fips_enabled isn't available since that would
>> only be OpenSSL versions before 3.0 (and since we don't support 1.0.2 then no
>> such version can have FIPS).  Sharing back a v3 which is what I think we should
>> go with.
> 
> The comment suggests to me that if the user happened to be using
> OpenSSL 1.1.1 and CheckLegacyCryptoMode() was called, the expected
> outcome would be an error, but it will just return.

I think I know what you mean, but just to be clear so I know what to reword,
the comment in the code or the above quoted email?

If the GUC is set to fips it will mimic the OpenSSL setting (disallow when
OpenSSL is in FIPS mode and allow when OpenSSL isn't in FIPS mode), and thus
allow internal crypto since OpenSSL 1.1.1 cannot operate in FIPS mode.  If the
GUC is set to on or off it will allow or disallow built-in crypto without
considering the OpenSSL state.

--
Daniel Gustafsson

  45. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Robert Haas <robertmhaas@gmail.com> — 2024-12-04T14:39:41Z

    On Wed, Dec 4, 2024 at 9:33 AM Daniel Gustafsson <daniel@yesql.se> wrote:
> > The comment suggests to me that if the user happened to be using
> > OpenSSL 1.1.1 and CheckLegacyCryptoMode() was called, the expected
> > outcome would be an error, but it will just return.
>
> I think I know what you mean, but just to be clear so I know what to reword,
> the comment in the code or the above quoted email?
>
> If the GUC is set to fips it will mimic the OpenSSL setting (disallow when
> OpenSSL is in FIPS mode and allow when OpenSSL isn't in FIPS mode), and thus
> allow internal crypto since OpenSSL 1.1.1 cannot operate in FIPS mode.  If the
> GUC is set to on or off it will allow or disallow built-in crypto without
> considering the OpenSSL state.

Never mind, I was being stupid.

*wanders off to find a paper bag*

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  46. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Joe Conway <mail@joeconway.com> — 2024-12-04T14:40:20Z

    On 12/4/24 09:33, Daniel Gustafsson wrote:
> since OpenSSL 1.1.1 cannot operate in FIPS mode.

I don't think that is correct. The RHEL 8 openssl which was FIPS 140-2 
validated is 1.1.1k. See:

https://csrc.nist.gov/CSRC/media/projects/cryptographic-module-validation-program/documents/security-policies/140sp4642.pdf

-- 
Joe Conway
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  47. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Daniel Gustafsson <daniel@yesql.se> — 2024-12-04T14:45:51Z

    > On 4 Dec 2024, at 15:40, Joe Conway <mail@joeconway.com> wrote:
> 
> On 12/4/24 09:33, Daniel Gustafsson wrote:
>> since OpenSSL 1.1.1 cannot operate in FIPS mode.
> 
> I don't think that is correct. The RHEL 8 openssl which was FIPS 140-2 validated is 1.1.1k. See:
> 
> https://csrc.nist.gov/CSRC/media/projects/cryptographic-module-validation-program/documents/security-policies/140sp4642.pdf

Does RHEL publish the source of their fork somewhere?  In OpenSSL 1.1.1 the
code for FIPS_mode is:

int FIPS_mode(void)
{
    /* This version of the library does not support FIPS mode. */
    return 0;
}

Do you know if RHEL patched OpenSSL to allow FIPS_mode() to return other than 0
or if that function is useless regardless?

--
Daniel Gustafsson

  48. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Joe Conway <mail@joeconway.com> — 2024-12-04T14:52:52Z

    On 12/4/24 09:45, Daniel Gustafsson wrote:
>> On 4 Dec 2024, at 15:40, Joe Conway <mail@joeconway.com> wrote:
>> 
>> On 12/4/24 09:33, Daniel Gustafsson wrote:
>>> since OpenSSL 1.1.1 cannot operate in FIPS mode.
>> 
>> I don't think that is correct. The RHEL 8 openssl which was FIPS 140-2 validated is 1.1.1k. See:
>> 
>> https://csrc.nist.gov/CSRC/media/projects/cryptographic-module-validation-program/documents/security-policies/140sp4642.pdf
> 
> Does RHEL publish the source of their fork somewhere?  In OpenSSL 1.1.1 the
> code for FIPS_mode is:
> 
> int FIPS_mode(void)
> {
>      /* This version of the library does not support FIPS mode. */
>      return 0;
> }
> 
> Do you know if RHEL patched OpenSSL to allow FIPS_mode() to return other than 0
> or if that function is useless regardless?

Yes the RHEL and OpenSUSE rpms for openssl are heavily patched for the 
FIPS versions, as is the Ubuntu one. It has been a while but last time I 
looked at all of this they were all using very similar patches to allow 
the "system wide" FIPS mode rather than depending on the app to 
explicitly go into FIPS_mode().

I can look for links, but investigating it involved (for example) 
installing the source rpm and then wading through hundreds of patches in 
the SOURCE directory.

-- 
Joe Conway
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  49. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Daniel Gustafsson <daniel@yesql.se> — 2024-12-04T15:01:06Z

    > On 4 Dec 2024, at 15:52, Joe Conway <mail@joeconway.com> wrote:
> 
> On 12/4/24 09:45, Daniel Gustafsson wrote:
>>> On 4 Dec 2024, at 15:40, Joe Conway <mail@joeconway.com> wrote:
>>> On 12/4/24 09:33, Daniel Gustafsson wrote:
>>>> since OpenSSL 1.1.1 cannot operate in FIPS mode.
>>> I don't think that is correct. The RHEL 8 openssl which was FIPS 140-2 validated is 1.1.1k. See:
>>> https://csrc.nist.gov/CSRC/media/projects/cryptographic-module-validation-program/documents/security-policies/140sp4642.pdf
>> Does RHEL publish the source of their fork somewhere?  In OpenSSL 1.1.1 the
>> code for FIPS_mode is:
>> int FIPS_mode(void)
>> {
>>     /* This version of the library does not support FIPS mode. */
>>     return 0;
>> }
>> Do you know if RHEL patched OpenSSL to allow FIPS_mode() to return other than 0
>> or if that function is useless regardless?
> 
> Yes the RHEL and OpenSUSE rpms for openssl are heavily patched for the FIPS versions, as is the Ubuntu one. It has been a while but last time I looked at all of this they were all using very similar patches to allow the "system wide" FIPS mode rather than depending on the app to explicitly go into FIPS_mode().
> 
> I can look for links, but investigating it involved (for example) installing the source rpm and then wading through hundreds of patches in the SOURCE directory.

While I dislike not having a "follow-the-lib" setting on the GUC and rely on
the transitive dependency, maybe that's the only option if we can't reliably
detect the operating mode.  Sigh, as if OpenSSL wasn't messy enough even
without vendor patches =)

--
Daniel Gustafsson

  50. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Joe Conway <mail@joeconway.com> — 2024-12-04T15:57:11Z

    On 12/4/24 10:01, Daniel Gustafsson wrote:
>> On 4 Dec 2024, at 15:52, Joe Conway <mail@joeconway.com> wrote:
>> 
>> On 12/4/24 09:45, Daniel Gustafsson wrote:
>>>> On 4 Dec 2024, at 15:40, Joe Conway <mail@joeconway.com> wrote:
>>>> On 12/4/24 09:33, Daniel Gustafsson wrote:
>>>>> since OpenSSL 1.1.1 cannot operate in FIPS mode.
>>>> I don't think that is correct. The RHEL 8 openssl which was FIPS 140-2 validated is 1.1.1k. See:
>>>> https://csrc.nist.gov/CSRC/media/projects/cryptographic-module-validation-program/documents/security-policies/140sp4642.pdf
>>> Does RHEL publish the source of their fork somewhere?  In OpenSSL 1.1.1 the
>>> code for FIPS_mode is:
>>> int FIPS_mode(void)
>>> {
>>>     /* This version of the library does not support FIPS mode. */
>>>     return 0;
>>> }
>>> Do you know if RHEL patched OpenSSL to allow FIPS_mode() to return other than 0
>>> or if that function is useless regardless?
>> 
>> Yes the RHEL and OpenSUSE rpms for openssl are heavily patched for the FIPS versions, as is the Ubuntu one. It has been a while but last time I looked at all of this they were all using very similar patches to allow the "system wide" FIPS mode rather than depending on the app to explicitly go into FIPS_mode().
>> 
>> I can look for links, but investigating it involved (for example) installing the source rpm and then wading through hundreds of patches in the SOURCE directory.

I can send you the source RPM for openssl 1.1.1c which was an earlier 
FIPS validated version, but the main FIPS patch contains:

8<-------------
diff -up openssl-1.1.1b/crypto/o_fips.c.fips openssl-1.1.1b/crypto/o_fips.c
--- openssl-1.1.1b/crypto/o_fips.c.fips	2019-02-26 15:15:30.000000000 +0100
+++ openssl-1.1.1b/crypto/o_fips.c	2019-02-28 11:30:06.817745466 +0100
@@ -8,17 +8,28 @@
   */

  #include "internal/cryptlib.h"
+#include "internal/fips_int.h"

  int FIPS_mode(void)
  {
+#ifdef OPENSSL_FIPS
+    return FIPS_module_mode();
+#else
      /* This version of the library does not support FIPS mode. */
      return 0;
+#endif
  }
8<-------------

> While I dislike not having a "follow-the-lib" setting on the GUC and rely on
> the transitive dependency, maybe that's the only option if we can't reliably
> detect the operating mode.  Sigh, as if OpenSSL wasn't messy enough even
> without vendor patches =)

Yep, that was my concern. I believe the RHEL, OpenSUSE, and Ubuntu 
solutions are very similar, but they may be different enough that it 
will be painful to reliably detect FIPS_mode. The RHEL and SUSE source 
RPMs were findable. I think to get the Ubuntu FIPS deb I had to pay for 
a subscription. But as I said it has been a while so I don't remember 
exactly (like 6+ years).

-- 
Joe Conway
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  51. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Joe Conway <mail@joeconway.com> — 2024-12-04T16:01:38Z

    On 12/4/24 10:57, Joe Conway wrote:
> On 12/4/24 10:01, Daniel Gustafsson wrote:
>>> On 4 Dec 2024, at 15:52, Joe Conway <mail@joeconway.com> wrote:
>>> 
>>> On 12/4/24 09:45, Daniel Gustafsson wrote:
>>>>> On 4 Dec 2024, at 15:40, Joe Conway <mail@joeconway.com> wrote:
>>>>> On 12/4/24 09:33, Daniel Gustafsson wrote:
>>>>>> since OpenSSL 1.1.1 cannot operate in FIPS mode.
>>>>> I don't think that is correct. The RHEL 8 openssl which was FIPS 140-2 validated is 1.1.1k. See:
>>>>> https://csrc.nist.gov/CSRC/media/projects/cryptographic-module-validation-program/documents/security-policies/140sp4642.pdf
>>>> Does RHEL publish the source of their fork somewhere?  In OpenSSL 1.1.1 the
>>>> code for FIPS_mode is:
>>>> int FIPS_mode(void)
>>>> {
>>>>     /* This version of the library does not support FIPS mode. */
>>>>     return 0;
>>>> }
>>>> Do you know if RHEL patched OpenSSL to allow FIPS_mode() to return other than 0
>>>> or if that function is useless regardless?
>>> 
>>> Yes the RHEL and OpenSUSE rpms for openssl are heavily patched for the FIPS versions, as is the Ubuntu one. It has been a while but last time I looked at all of this they were all using very similar patches to allow the "system wide" FIPS mode rather than depending on the app to explicitly go into FIPS_mode().
>>> 
>>> I can look for links, but investigating it involved (for example) installing the source rpm and then wading through hundreds of patches in the SOURCE directory.
> 
> I can send you the source RPM for openssl 1.1.1c which was an earlier
> FIPS validated version, but the main FIPS patch contains:
> 
> 8<-------------
> diff -up openssl-1.1.1b/crypto/o_fips.c.fips openssl-1.1.1b/crypto/o_fips.c
> --- openssl-1.1.1b/crypto/o_fips.c.fips	2019-02-26 15:15:30.000000000 +0100
> +++ openssl-1.1.1b/crypto/o_fips.c	2019-02-28 11:30:06.817745466 +0100
> @@ -8,17 +8,28 @@
>     */
> 
>    #include "internal/cryptlib.h"
> +#include "internal/fips_int.h"
> 
>    int FIPS_mode(void)
>    {
> +#ifdef OPENSSL_FIPS
> +    return FIPS_module_mode();
> +#else
>        /* This version of the library does not support FIPS mode. */
>        return 0;
> +#endif
>    }
> 8<-------------

FWIW, here is a link to a 1.1.1k source RPM:
  https://yum.oracle.com/repo/OracleLinux/OL8/baseos/latest/x86_64/getPackageSource/openssl-1.1.1k-4.el8.src.rpm

-- 
Joe Conway
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  52. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Daniel Gustafsson <daniel@yesql.se> — 2024-12-09T12:23:58Z

    > On 4 Dec 2024, at 16:57, Joe Conway <mail@joeconway.com> wrote:

> I can send you the source RPM for openssl 1.1.1c which was an earlier FIPS validated version, but the main FIPS patch contains:

AFAICT the forks of 1.1.1 which offer FIPS certification all patch the common
OpenSSL API FIPS_mode() rather than invent a new one, so the earlier approach
should work fine. PFA an updated version which I propose we go ahead with.

--
Daniel Gustafsson

  53. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Joe Conway <mail@joeconway.com> — 2024-12-09T14:11:11Z

    On 12/9/24 07:23, Daniel Gustafsson wrote:
>> On 4 Dec 2024, at 16:57, Joe Conway <mail@joeconway.com> wrote:
> 
>> I can send you the source RPM for openssl 1.1.1c which was an earlier FIPS validated version, but the main FIPS patch contains:
> 
> AFAICT the forks of 1.1.1 which offer FIPS certification all patch the common
> OpenSSL API FIPS_mode() rather than invent a new one, so the earlier approach
> should work fine. PFA an updated version which I propose we go ahead with.

That sounds correct from my memory of it.

I have not done any actual testing (yet), but on quick scan this part 
looks suspicious:
8<-------------------
+_PG_init(void)
+{
+	DefineCustomEnumVariable("pgcrypto.legacy_crypto_enabled",
+							 "Sets if builtin crypto functions are enabled.",
+							 "Yes enables builtin crypto, No unconditionally disables and 
OpenSSL "
+							 "will disable if OpenSSL is in FIPS mode",
+							 &legacy_crypto_enabled,
8<-------------------

Rather than:
  "Yes enables builtin crypto, No unconditionally disables and OpenSSL "
                                                               ^^^^^^^
  "will disable if OpenSSL is in FIPS mode"

I think that should say:
  "Yes enables builtin crypto, No unconditionally disables and fips "
                                                               ^^^^
  "will disable if OpenSSL is in FIPS mode"

-- 
Joe Conway
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  54. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Daniel Gustafsson <daniel@yesql.se> — 2024-12-09T21:37:58Z

    > On 9 Dec 2024, at 15:11, Joe Conway <mail@joeconway.com> wrote:
> 
> On 12/9/24 07:23, Daniel Gustafsson wrote:
>>> On 4 Dec 2024, at 16:57, Joe Conway <mail@joeconway.com> wrote:
>>> I can send you the source RPM for openssl 1.1.1c which was an earlier FIPS validated version, but the main FIPS patch contains:
>> AFAICT the forks of 1.1.1 which offer FIPS certification all patch the common
>> OpenSSL API FIPS_mode() rather than invent a new one, so the earlier approach
>> should work fine. PFA an updated version which I propose we go ahead with.
> 
> That sounds correct from my memory of it.
> 
> I have not done any actual testing (yet), but on quick scan this part looks suspicious:

Not only suspicious but plain wrong, fixed in the attached, thanks!

--
Daniel Gustafsson

  55. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Peter Eisentraut <peter@eisentraut.org> — 2024-12-12T11:07:39Z

    On 09.12.24 22:37, Daniel Gustafsson wrote:
>> On 9 Dec 2024, at 15:11, Joe Conway <mail@joeconway.com> wrote:
>>
>> On 12/9/24 07:23, Daniel Gustafsson wrote:
>>>> On 4 Dec 2024, at 16:57, Joe Conway <mail@joeconway.com> wrote:
>>>> I can send you the source RPM for openssl 1.1.1c which was an earlier FIPS validated version, but the main FIPS patch contains:
>>> AFAICT the forks of 1.1.1 which offer FIPS certification all patch the common
>>> OpenSSL API FIPS_mode() rather than invent a new one, so the earlier approach
>>> should work fine. PFA an updated version which I propose we go ahead with.
>>
>> That sounds correct from my memory of it.
>>
>> I have not done any actual testing (yet), but on quick scan this part looks suspicious:
> 
> Not only suspicious but plain wrong, fixed in the attached, thanks!

I think these function names are wrong:

+      <varname>pgcrypto.legacy_crypto_enabled</varname> determines if the
+      built in legacy crypto functions <literal>pg_gen_salt</literal>,
+      <literal>pg_gen_salt_rounds</literal>, and 
<literal>pg_crypt</literal>
+      are available for use.

Those are the C-level functions.  The SQL-level functions are called 
gen_salt and crypt.

  56. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Daniel Gustafsson <daniel@yesql.se> — 2024-12-12T11:58:48Z

    
> On 12 Dec 2024, at 12:07, Peter Eisentraut <peter@eisentraut.org> wrote:
> 
> On 09.12.24 22:37, Daniel Gustafsson wrote:
>>> On 9 Dec 2024, at 15:11, Joe Conway <mail@joeconway.com> wrote:
>>> 
>>> On 12/9/24 07:23, Daniel Gustafsson wrote:
>>>>> On 4 Dec 2024, at 16:57, Joe Conway <mail@joeconway.com> wrote:
>>>>> I can send you the source RPM for openssl 1.1.1c which was an earlier FIPS validated version, but the main FIPS patch contains:
>>>> AFAICT the forks of 1.1.1 which offer FIPS certification all patch the common
>>>> OpenSSL API FIPS_mode() rather than invent a new one, so the earlier approach
>>>> should work fine. PFA an updated version which I propose we go ahead with.
>>> 
>>> That sounds correct from my memory of it.
>>> 
>>> I have not done any actual testing (yet), but on quick scan this part looks suspicious:
>> Not only suspicious but plain wrong, fixed in the attached, thanks!
> 
> I think these function names are wrong:
> 
> +      <varname>pgcrypto.legacy_crypto_enabled</varname> determines if the
> +      built in legacy crypto functions <literal>pg_gen_salt</literal>,
> +      <literal>pg_gen_salt_rounds</literal>, and <literal>pg_crypt</literal>
> +      are available for use.
> 
> Those are the C-level functions.  The SQL-level functions are called gen_salt and crypt.

Nice catch, they have been copied around since v1 of the patch without anyone
noticing.  Fixed, and I also added parentheses to match the docs page in
question (<function>crypt()</function> instead of <function>crypt</function>).

--
Daniel Gustafsson

  57. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Daniel Gustafsson <daniel@yesql.se> — 2025-01-13T14:18:37Z

    Circling back to wrap up this thread I retested with, and without, OpenSSL FIPS
and tweaked the docs and code a little to ensure it detects the right function
to use.  The attached is what I propose we go ahead with.

--
Daniel Gustafsson

  58. RE: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com> — 2025-01-14T12:12:50Z

    Dear Daniel,

Thanks for working on the project. I have few cosmetic comments.

```
+      built in legacy crypto functions <literal>gen_salt()</literal>,
```

According to other lines, `<literal>gen_salt()</literal>` should be `<function>gen_salt()</function>`.

```
+      <literal>pg_gen_salt_rounds()</literal>, and <literal>crypt()</literal>
```

Similar with [1], `pg_gen_salt_rounds` is not an SQL function.
I think we do not have to mention the function because it's just another implementation of gen_salt().
Also, use <function> instead of <literal>.

```
+void
+_PG_init(void)
+{
+	DefineCustomEnumVariable("pgcrypto.legacy_crypto_enabled",
+							 "Sets if builtin crypto functions are enabled.",
+							 "\"on\" enables builtin crypto, \"off\" unconditionally disables and \"fips\" "
+							 "will disable builtin crypto if OpenSSL is in FIPS mode",
+							 &legacy_crypto_enabled,
+							 LGC_ON,
+							 legacy_crypto_options,
+							 PGC_SUSET,
+							 0,
+							 NULL,
+							 NULL,
+							 NULL);
+}
```

I think we must call MarkGUCPrefixReserved() to catch the mis-spell.

Also: I'm not sure whether we should bump the version of pgcrypto. It should be done when
the API is changed, but the patch does not do. Thought?

[1]: https://www.postgresql.org/message-id/1f32ff67-255d-4c0c-8433-c8c721842aa3%40eisentraut.org

Best regards,
Hayato Kuroda
FUJITSU LIMITED

  59. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Daniel Gustafsson <daniel@yesql.se> — 2025-01-15T14:24:52Z

    > On 14 Jan 2025, at 13:12, Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com> wrote:

> Similar with [1], `pg_gen_salt_rounds` is not an SQL function.
> I think we do not have to mention the function because it's just another implementation of gen_salt().
> Also, use <function> instead of <literal>.

Fixed.

> I think we must call MarkGUCPrefixReserved() to catch the mis-spell.

Good point, fixed.

> Also: I'm not sure whether we should bump the version of pgcrypto. It should be done when
> the API is changed, but the patch does not do. Thought?

I don't think this constitutes a change which warrants a version bump so I've
left that out for now.

The attached includes a rename from "legacy_crypto" to "builtin_crypto".  While
legacy might apply now, there is work ongoing to modernize the algorithms
supported by crypt [0] so legacy might not be applicable soon (this GUC would
however still be relevant as the proposed code isn't FIPS certified).  Builtin
seems like a more future-proof choice in terms of naming.

--
Daniel Gustafsson

[0] c763235a2757e2f5f9e3e27268b9028349cef659.camel@oopsware.de

  60. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Joe Conway <mail@joeconway.com> — 2025-01-18T17:32:10Z

    On 1/15/25 09:24, Daniel Gustafsson wrote:
>> On 14 Jan 2025, at 13:12, Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com> wrote:
> 
>> Similar with [1], `pg_gen_salt_rounds` is not an SQL function.
>> I think we do not have to mention the function because it's just another implementation of gen_salt().
>> Also, use <function> instead of <literal>.
> 
> Fixed.
> 
>> I think we must call MarkGUCPrefixReserved() to catch the mis-spell.
> 
> Good point, fixed.
> 
>> Also: I'm not sure whether we should bump the version of pgcrypto. It should be done when
>> the API is changed, but the patch does not do. Thought?
> 
> I don't think this constitutes a change which warrants a version bump so I've
> left that out for now.
> 
> The attached includes a rename from "legacy_crypto" to "builtin_crypto".  While
> legacy might apply now, there is work ongoing to modernize the algorithms
> supported by crypt [0] so legacy might not be applicable soon (this GUC would
> however still be relevant as the proposed code isn't FIPS certified).  Builtin
> seems like a more future-proof choice in terms of naming.


I wonder if it makes any sense to test the "fips" value of the GUC here:
8<----------------
--- a/contrib/pgcrypto/expected/crypt-des.out
+++ b/contrib/pgcrypto/expected/crypt-des.out
@@ -28,4 +28,11 @@ FROM ctest;
   t
  (1 row)

+-- check disabling of built in crypto functions
+SET pgcrypto.builtin_crypto_enabled = off;
+UPDATE ctest SET salt = gen_salt('des');
+ERROR:  use of built-in crypto functions is disabled
+UPDATE ctest SET res = crypt(data, salt);
+ERROR:  use of built-in crypto functions is disabled
+RESET pgcrypto.builtin_crypto_enabled;
  DROP TABLE ctest;
8<----------------

The usual case would be "fips disabled", in which case it ought to work 
same as "on".

Maybe we could document that the test should fail if fips is enabled?

FWIW I have not tested at all on a fips enabled machine. I will see 
about doing that...

-- 
Joe Conway
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  61. RE: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Koshi Shibagaki (Fujitsu) <shibagaki.koshi@fujitsu.com> — 2025-01-20T00:26:01Z

    Thank you for moving this discussion forward.

> Maybe we could document that the test should fail if fips is enabled?
> 
> FWIW I have not tested at all on a fips enabled machine. I will see about doing
> that...
I tested all on a fips enabled machine and test failed.
Since all tests have been made to run even with FIPS enabled in PostgreSQL 17, 
it would be ideal for this test to follow suit.
Following the modification that made the citext tests compatible with FIPS mode[1], 
how about make multiple expected output?

diff --git a/contrib/pgcrypto/expected/crypt-des_1.out b/contrib/pgcrypto/expected/crypt-des_1.out
new file mode 100644
index 0000000000..f8106b0ee2
--- /dev/null
+++ b/contrib/pgcrypto/expected/crypt-des_1.out
@@ -0,0 +1,44 @@
+--
+-- crypt() and gen_salt(): crypt-des
+--
+SELECT crypt('', 'NB');
+     crypt     
+---------------
+ NBPx/38Y48kHg
+(1 row)
+
+SELECT crypt('foox', 'NB');
+     crypt     
+---------------
+ NB53EGGqrrb5E
+(1 row)
+
+-- We are supposed to pass in a 2-character salt.
+-- error since salt is too short:
+SELECT crypt('password', 'a');
+ERROR:  invalid salt
+CREATE TABLE ctest (data text, res text, salt text);
+INSERT INTO ctest VALUES ('password', '', '');
+UPDATE ctest SET salt = gen_salt('des');
+UPDATE ctest SET res = crypt(data, salt);
+SELECT res = crypt(data, res) AS "worked"
+FROM ctest;
+ worked 
+--------
+ t
+(1 row)
+
+-- check disabling of built in crypto functions
+SET pgcrypto.builtin_crypto_enabled = off;
+UPDATE ctest SET salt = gen_salt('des');
+ERROR:  use of built-in crypto functions is disabled
+UPDATE ctest SET res = crypt(data, salt);
+ERROR:  use of built-in crypto functions is disabled
+RESET pgcrypto.builtin_crypto_enabled;
+SET pgcrypto.builtin_crypto_enabled = fips;
+UPDATE ctest SET salt = gen_salt('des');
+ERROR:  use of non-FIPS certified crypto not allowed when OpenSSL is in FIPS mode
+UPDATE ctest SET res = crypt(data, salt);
+ERROR:  use of non-FIPS certified crypto not allowed when OpenSSL is in FIPS mode
+RESET pgcrypto.builtin_crypto_enabled;
+DROP TABLE ctest;


[1] https://github.com/postgres/postgres/commit/3c551ebede46194237f82062b54b92e474b5c743




Koshi Shibagaki
FUJITSU LIMITED
https://www.fujitsu.com/

  62. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Daniel Gustafsson <daniel@yesql.se> — 2025-01-20T07:54:04Z

    > On 20 Jan 2025, at 01:26, Koshi Shibagaki (Fujitsu) <shibagaki.koshi@fujitsu.com> wrote:
> 
> Thank you for moving this discussion forward.
> 
>> Maybe we could document that the test should fail if fips is enabled?
>> 
>> FWIW I have not tested at all on a fips enabled machine. I will see about doing
>> that...
> I tested all on a fips enabled machine and test failed.
> Since all tests have been made to run even with FIPS enabled in PostgreSQL 17, 
> it would be ideal for this test to follow suit.
> Following the modification that made the citext tests compatible with FIPS mode[1], 
> how about make multiple expected output?

Agreed, I'll roll that into the next version of the patch.

--
Daniel Gustafsson

  63. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2025-01-20T12:00:31Z

    On 2025-Jan-15, Daniel Gustafsson wrote:

> > On 14 Jan 2025, at 13:12, Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com> wrote:

> > Also: I'm not sure whether we should bump the version of pgcrypto.
> > It should be done when the API is changed, but the patch does not
> > do. Thought?
> 
> I don't think this constitutes a change which warrants a version bump
> so I've left that out for now.

Extension versions need to be changed only when the SQL definition of
the module is modified.  If the patch does not require a .sql script
that would run with ALTER EXTENSION UPDATE, then you should not modify
the extension version.

-- 
Álvaro Herrera         PostgreSQL Developer  —  https://www.EnterpriseDB.com/
<Schwern> It does it in a really, really complicated way
<crab> why does it need to be complicated?
<Schwern> Because it's MakeMaker.

  64. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Daniel Gustafsson <daniel@yesql.se> — 2025-01-21T11:39:27Z

    > On 20 Jan 2025, at 01:26, Koshi Shibagaki (Fujitsu) <shibagaki.koshi@fujitsu.com> wrote:
> 
> Thank you for moving this discussion forward.
> 
>> Maybe we could document that the test should fail if fips is enabled?
>> 
>> FWIW I have not tested at all on a fips enabled machine. I will see about doing
>> that...
> I tested all on a fips enabled machine and test failed.

Did the patch as posted fail, or did it fail when you changed the GUC to follow
the fips mode?  I assume it's the latter since the code in question doesn't
care about FIPS at all (hence this patch).  Re-testing it again against OpenSSL
3.4 with FIPS enabled as well as disabled I can't reproduce any failure.

> Since all tests have been made to run even with FIPS enabled in PostgreSQL 17, 
> it would be ideal for this test to follow suit.

The work which was done was to ensure that the tests passes regardless of if
FIPS is enabled or not, they were not designed to test FIPS.

After thinking about I don't think we need an alternative output file since it
won't add any testing:

> +SET pgcrypto.builtin_crypto_enabled = fips;
> +UPDATE ctest SET salt = gen_salt('des');
> +ERROR:  use of non-FIPS certified crypto not allowed when OpenSSL is in FIPS mode
> +UPDATE ctest SET res = crypt(data, salt);

If we add such an alternative output we also need the other case where FIPS is
disabled and the functions work, which means we add no test coverage at all as
both options are allowed to pass.

--
Daniel Gustafsson

  65. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Joe Conway <mail@joeconway.com> — 2025-01-21T17:51:35Z

    On 1/21/25 06:39, Daniel Gustafsson wrote:
>> On 20 Jan 2025, at 01:26, Koshi Shibagaki (Fujitsu) <shibagaki.koshi@fujitsu.com> wrote:
>> 
>> Thank you for moving this discussion forward.
>> 
>>> Maybe we could document that the test should fail if fips is enabled?
>>> 
>>> FWIW I have not tested at all on a fips enabled machine. I will see about doing
>>> that...
>> I tested all on a fips enabled machine and test failed.
> 
> Did the patch as posted fail, or did it fail when you changed the GUC to follow
> the fips mode?  I assume it's the latter since the code in question doesn't
> care about FIPS at all (hence this patch).  Re-testing it again against OpenSSL
> 3.4 with FIPS enabled as well as disabled I can't reproduce any failure.
> 
>> Since all tests have been made to run even with FIPS enabled in PostgreSQL 17, 
>> it would be ideal for this test to follow suit.
> 
> The work which was done was to ensure that the tests passes regardless of if
> FIPS is enabled or not, they were not designed to test FIPS.
> 
> After thinking about I don't think we need an alternative output file since it
> won't add any testing:
> 
>> +SET pgcrypto.builtin_crypto_enabled = fips;
>> +UPDATE ctest SET salt = gen_salt('des');
>> +ERROR:  use of non-FIPS certified crypto not allowed when OpenSSL is in FIPS mode
>> +UPDATE ctest SET res = crypt(data, salt);
> 
> If we add such an alternative output we also need the other case where FIPS is
> disabled and the functions work, which means we add no test coverage at all as
> both options are allowed to pass.

I was thinking the same -- perhaps just an SQL comment that says 
something like: "-- expected to fail when in fips mode"
or similar.

I also wonder if it is worthwhile to expose a SQL callable function to 
indicate whether the backend is in fips mode or not. I think that would 
be a useful addition, but I guess one could derive the answer based on 
whether these functions work or not when "builtin_crypto_enabled = fips"

-- 
Joe Conway
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  66. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Daniel Gustafsson <daniel@yesql.se> — 2025-01-21T20:47:08Z

    > On 21 Jan 2025, at 18:51, Joe Conway <mail@joeconway.com> wrote:
> On 1/21/25 06:39, Daniel Gustafsson wrote:

>> If we add such an alternative output we also need the other case where FIPS is
>> disabled and the functions work, which means we add no test coverage at all as
>> both options are allowed to pass.
> 
> I was thinking the same -- perhaps just an SQL comment that says something like: "-- expected to fail when in fips mode"
> or similar.

But it isn't actually expected to fail when in FIPS mode since the test sets
the GUC to off.  It would only fail (due to different error message) if the GUC
in the testfile was changed and I don't we need to cater for manual alterings
of the testdata.

> I also wonder if it is worthwhile to expose a SQL callable function to indicate whether the backend is in fips mode or not. I think that would be a useful addition, but I guess one could derive the answer based on whether these functions work or not when "builtin_crypto_enabled = fips"

It could indeed be useful, but I doubt we can make it portable to check for
anything but the state of OpenSSL.  If the operating system has a FIPS mode
then we won't capture that.  That might not be a problem since if the OS is in
FIPS mode then OpenSSL most likely will be too but we can't guarantee it.
Definitely worth thinking about, I can see it being useful especially in DBaaS
environments to ensure that the libraries are in the expected state.

--
Daniel Gustafsson

  67. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Tom Lane <tgl@sss.pgh.pa.us> — 2025-01-21T20:59:19Z

    Daniel Gustafsson <daniel@yesql.se> writes:
> It could indeed be useful, but I doubt we can make it portable to check for
> anything but the state of OpenSSL.  If the operating system has a FIPS mode
> then we won't capture that.  That might not be a problem since if the OS is in
> FIPS mode then OpenSSL most likely will be too but we can't guarantee it.

Not our problem, I think.  The OS vendor would have to have fallen
down on the job quite badly to offer an OS-level "FIPS mode" while
shipping an OpenSSL that doesn't honor that.  It would not be optional
for OpenSSL to be in that mode if the OS is.

(If we end up inventing a FIPS-mode flag, I would fully expect
interested vendors to patch our code to force it on when the
OS-level flag is set, which is exactly what they will have done
to OpenSSL.  We should design our behavior with that in mind.)

			regards, tom lane

  68. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Daniel Gustafsson <daniel@yesql.se> — 2025-01-21T21:05:09Z

    > On 21 Jan 2025, at 21:59, Tom Lane <tgl@sss.pgh.pa.us> wrote:

> (If we end up inventing a FIPS-mode flag, I would fully expect
> interested vendors to patch our code to force it on when the
> OS-level flag is set, which is exactly what they will have done
> to OpenSSL.  We should design our behavior with that in mind.)

This patch is essentially a FIPS-mode flag as it's designed to block the
built-in non-certified code in pgcrypto which ensures that OpenSSL is used for
all crypto operations.  When setting this GUC to "fips" it will match the
OpenSSL setting, disable built-in crypto when OpenSSL has FIPS enabled and
allow it when OpenSSL has FIPS disabled.  Setting it to off will disable
built-in crypto regardless of FIPS mode in OpenSSL.

--
Daniel Gustafsson

  69. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Joe Conway <mail@joeconway.com> — 2025-01-21T21:13:26Z

    On 1/21/25 15:59, Tom Lane wrote:
> Daniel Gustafsson <daniel@yesql.se> writes:
>> It could indeed be useful, but I doubt we can make it portable to check for
>> anything but the state of OpenSSL.  If the operating system has a FIPS mode
>> then we won't capture that.  That might not be a problem since if the OS is in
>> FIPS mode then OpenSSL most likely will be too but we can't guarantee it.
> 
> Not our problem, I think.  The OS vendor would have to have fallen
> down on the job quite badly to offer an OS-level "FIPS mode" while
> shipping an OpenSSL that doesn't honor that.  It would not be optional
> for OpenSSL to be in that mode if the OS is.
> 
> (If we end up inventing a FIPS-mode flag, I would fully expect
> interested vendors to patch our code to force it on when the
> OS-level flag is set, which is exactly what they will have done
> to OpenSSL.  We should design our behavior with that in mind.)


I think this is a non-issue. Every implementation I have seen, the 
OS-level enabling of FIPS mode is just a way to ensure openssl is 
automatically put into FIPS mode when the library is loaded, just as if 
(and not depending on) the application had invoked FIPS mode manually. 
All matters here is that the loaded openssl thinks it is in FIPS mode.

I think that could be done with a subset of the proposed 
CheckBuiltinCryptoMode() function. E.g. something like (completely 
untested):

8<----------------------
+ /*
+  * CheckFIPSMode
+  *
+  * Function to determine if OpenSSL is operating in FIPS mode
+  */
+ int
+ CheckFIPSMode(void)
+ {
+ 	int			fips_enabled;
+
+ 	/*
+ 	 * EVP_default_properties_is_fips_enabled was added in OpenSSL 3.0, 
before
+ 	 * that FIPS_mode() was used to test for FIPS being enabled.  The last
+ 	 * upstream OpenSSL version before 3.0 which supported FIPS was 
1.0.2, but
+ 	 * there are forks of 1.1.1 which are FIPS certified so we still need to
+ 	 * test with FIPS_mode() even though we don't support 1.0.2.
+ 	 */
+ 	fips_enabled =
+ #if OPENSSL_VERSION_NUMBER >= 0x30000000L
+ 		EVP_default_properties_is_fips_enabled(NULL);
+ #else
+ 		FIPS_mode();
+ #endif
+
+ 	return fips_enabled;
+ }
8<-----------------

The we could call CheckFIPSMode() from CheckBuiltinCryptoMode() as well 
as from a SQL-level wrapper.

-- 
Joe Conway
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  70. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Daniel Gustafsson <daniel@yesql.se> — 2025-01-21T22:57:55Z

    > On 21 Jan 2025, at 22:13, Joe Conway <mail@joeconway.com> wrote:

> I think this is a non-issue. Every implementation I have seen, the OS-level enabling of FIPS mode is just a way to ensure openssl is automatically put into FIPS mode when the library is loaded, just as if (and not depending on) the application had invoked FIPS mode manually. All matters here is that the loaded openssl thinks it is in FIPS mode.

Good point.  The attached v9 adds a 0001 which expose a SQL function (along
with version bump and docs) for returning the FIPS mode, and 0002 is the
previous patch except it use the function from 0001.

--
Daniel Gustafsson

  71. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Joe Conway <mail@joeconway.com> — 2025-01-22T18:59:56Z

    On 1/21/25 17:57, Daniel Gustafsson wrote:
>> On 21 Jan 2025, at 22:13, Joe Conway <mail@joeconway.com> wrote:
> 
>> I think this is a non-issue. Every implementation I have seen, the OS-level enabling of FIPS mode is just a way to ensure openssl is automatically put into FIPS mode when the library is loaded, just as if (and not depending on) the application had invoked FIPS mode manually. All matters here is that the loaded openssl thinks it is in FIPS mode.
> 
> Good point.  The attached v9 adds a 0001 which expose a SQL function (along
> with version bump and docs) for returning the FIPS mode, and 0002 is the
> previous patch except it use the function from 0001.


I found it necessary to add:
   #include <openssl/crypto.h>
in
   contrib/pgcrypto/openssl.c
to avoid a symbol not defined warning.

Otherwise looks good here:
8<---------------
SET pgcrypto.builtin_crypto_enabled = on;
select crypt('secret data', gen_salt('des'));
      crypt
---------------
  OCAkiP03AAbPA
(1 row)

SET pgcrypto.builtin_crypto_enabled = off;
select crypt('secret data', gen_salt('des'));
ERROR:  use of built-in crypto functions is disabled

SET pgcrypto.builtin_crypto_enabled = fips;
select crypt('secret data', gen_salt('des'));
ERROR:  use of non-FIPS certified crypto not allowed when OpenSSL is in 
FIPS mode

select fips_mode();
  fips_mode
-----------
  t
(1 row)
8<---------------

Although come to think of it, probably:
   "use of non-FIPS certified crypto"
                    ^^^^^^^^^
should rather say:
   "use of non-FIPS validated crypto"
                    ^^^^^^^^^

FWIW, I tested with non-FIPS (OpenSSL 3.0.13 30 Jan 2024) on Linux Mint 
22.1 and FIPS (aws-lc [1][2]) on Amazon Linux 2023.

[1] https://github.com/aws/aws-lc/tree/fips-2022-11-02
[2] 
https://csrc.nist.gov/CSRC/media/projects/cryptographic-module-validation-program/documents/security-policies/140sp4759.pdf

-- 
Joe Conway
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  72. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Daniel Gustafsson <daniel@yesql.se> — 2025-01-22T22:49:45Z

    > On 22 Jan 2025, at 19:59, Joe Conway <mail@joeconway.com> wrote:

> I found it necessary to add:
>  #include <openssl/crypto.h>
> in
>  contrib/pgcrypto/openssl.c
> to avoid a symbol not defined warning.

Makes sense, it doesn't reproduce in my tree but reading OpenSSL code it seems
very plausible (and clearly happens in your environment).

> Although come to think of it, probably:
>  "use of non-FIPS certified crypto"
>                   ^^^^^^^^^
> should rather say:
>  "use of non-FIPS validated crypto"
>                   ^^^^^^^^^

That's probably better yes.  I was under the impression that the terminology
used was "FIPS certified" but reading the OpenSSL and FIPS documentation they
too use "FIPS validated" so I've switched to that as per your comment.

> FWIW, I tested with non-FIPS (OpenSSL 3.0.13 30 Jan 2024) on Linux Mint 22.1 and FIPS (aws-lc [1][2]) on Amazon Linux 2023.

Thanks.  My testing has been with a range of plain upstream OpenSSL trees from
1.1.1 to 3.4 (compiled on macOS).

Rebased v10 with the above fixed attached.

--
Daniel Gustafsson

  73. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Joe Conway <mail@joeconway.com> — 2025-01-23T20:42:53Z

    On 1/22/25 17:49, Daniel Gustafsson wrote:
>> On 22 Jan 2025, at 19:59, Joe Conway <mail@joeconway.com> wrote:
> 
>> I found it necessary to add:
>>  #include <openssl/crypto.h>
>> in
>>  contrib/pgcrypto/openssl.c
>> to avoid a symbol not defined warning.
> 
> Makes sense, it doesn't reproduce in my tree but reading OpenSSL code it seems
> very plausible (and clearly happens in your environment).
> 
>> Although come to think of it, probably:
>>  "use of non-FIPS certified crypto"
>>                   ^^^^^^^^^
>> should rather say:
>>  "use of non-FIPS validated crypto"
>>                   ^^^^^^^^^
> 
> That's probably better yes.  I was under the impression that the terminology
> used was "FIPS certified" but reading the OpenSSL and FIPS documentation they
> too use "FIPS validated" so I've switched to that as per your comment.
> 
>> FWIW, I tested with non-FIPS (OpenSSL 3.0.13 30 Jan 2024) on Linux Mint 22.1 and FIPS (aws-lc [1][2]) on Amazon Linux 2023.
> 
> Thanks.  My testing has been with a range of plain upstream OpenSSL trees from
> 1.1.1 to 3.4 (compiled on macOS).
> 
> Rebased v10 with the above fixed attached.


LGTM

-- 
Joe Conway
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  74. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Daniel Gustafsson <daniel@yesql.se> — 2025-01-24T13:55:40Z

    > On 23 Jan 2025, at 21:42, Joe Conway <mail@joeconway.com> wrote:

> LGTM

After staring at it a bit more and fixing a few small details I committed this.

--
Daniel Gustafsson

  75. Re: Replace current implementations in crypt() and gen_salt() to OpenSSL

    Joe Conway <mail@joeconway.com> — 2025-01-24T13:59:03Z

    On 1/24/25 08:55, Daniel Gustafsson wrote:
>> On 23 Jan 2025, at 21:42, Joe Conway <mail@joeconway.com> wrote:
> 
>> LGTM
> 
> After staring at it a bit more and fixing a few small details I committed this.

\o/

Thanks for getting this over the finish line!

-- 
Joe Conway
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com