0002-Support-post-quantum-signature-algorithms-in-SCRAM-c.patch

From 469d7b478aa8c1f05003411d023b96486221f8c4 Mon Sep 17 00:00:00 2001
From: Filip Janus <fjanus@redhat.com>
Date: Tue, 14 Oct 2025 10:52:47 +0200
Subject: [PATCH 2/2] Support post-quantum signature algorithms in SCRAM
 channel binding

When using post-quantum signature algorithms (e.g., ML-DSA/Dilithium)
in server certificates, SCRAM channel binding with tls-server-end-point
would fail with "could not find digest for NID UNDEF" error.

The issue occurs because post-quantum algorithms like ML-DSA don't have
a traditional separate digest algorithm that can be queried via
EVP_get_digestbynid(). Unlike traditional algorithms (e.g., RSA-SHA256),
where the hash function is a separate step, ML-DSA uses SHAKE256
internally as an integral part of the signature algorithm.

This commit adds a fallback mechanism:
- When EVP_get_digestbynid() returns NULL, use X509_get_signature_nid()
  to verify the certificate has a valid signature algorithm
- If valid, use SHA-256 for certificate hashing as recommended for
  unknown algorithms

While RFC 5929 doesn't clearly define the behavior for signature
algorithms without separate digest mappings, SHA-256 is a reasonable
choice as it is widely used, secure for modern standards, and matches
the 256-bit security level of algorithms like ML-DSA.

This change allows PostgreSQL to work with post-quantum cryptography
without requiring channel_binding=disable.
---
 src/backend/libpq/be-secure-openssl.c    | 32 ++++++++++++++++++++++--
 src/interfaces/libpq/fe-secure-openssl.c | 32 +++++++++++++++++++++---
 2 files changed, 59 insertions(+), 5 deletions(-)

diff --git a/src/backend/libpq/be-secure-openssl.c b/src/backend/libpq/be-secure-openssl.c
index c8b63ef8249..21ba0c09353 100644
--- a/src/backend/libpq/be-secure-openssl.c
+++ b/src/backend/libpq/be-secure-openssl.c
@@ -1621,8 +1621,36 @@ be_tls_get_certificate_hash(Port *port, size_t *len)
 		default:
 			algo_type = EVP_get_digestbynid(algo_nid);
 			if (algo_type == NULL)
-				elog(ERROR, "could not find digest for NID %s",
-					 OBJ_nid2sn(algo_nid));
+			{
+				/*
+				 * EVP_get_digestbynid() returned NULL. This can happen for:
+				 * 1. Post-quantum algorithms (ML-DSA, Falcon) that don't have
+				 *    a traditional digest mapping
+				 * 2. Invalid/corrupted certificates
+				 *
+				 * Use X509_get_signature_nid() to check if this is a valid
+				 * signature algorithm. If valid, use SHA-256 for hashing.
+				 */
+				int sig_nid = X509_get_signature_nid(server_cert);
+				
+				if (sig_nid != NID_undef && sig_nid > 0)
+				{
+					/*
+					 * Valid signature algorithm without digest mapping (likely
+					 * post-quantum). RFC 5929 doesn't clearly define this case.
+					 * We use SHA-256 as it's widely used, reasonably secure for
+					 * modern standards, and matches the security level of ML-DSA
+					 * which internally uses a 256-bit hash algorithm (SHAKE256).
+					 */
+					algo_type = EVP_sha256();
+				}
+				else
+				{
+					/* Invalid or corrupted certificate */
+					elog(ERROR, "could not find digest for NID %s",
+						 OBJ_nid2sn(algo_nid));
+				}
+			}
 			break;
 	}
 
diff --git a/src/interfaces/libpq/fe-secure-openssl.c b/src/interfaces/libpq/fe-secure-openssl.c
index 51dd7b9fec0..a3a35e171ff 100644
--- a/src/interfaces/libpq/fe-secure-openssl.c
+++ b/src/interfaces/libpq/fe-secure-openssl.c
@@ -385,9 +385,35 @@ pgtls_get_peer_certificate_hash(PGconn *conn, size_t *len)
 			algo_type = EVP_get_digestbynid(algo_nid);
 			if (algo_type == NULL)
 			{
-				libpq_append_conn_error(conn, "could not find digest for NID %s",
-										OBJ_nid2sn(algo_nid));
-				return NULL;
+				/*
+				 * EVP_get_digestbynid() returned NULL. This can happen for:
+				 * 1. Post-quantum algorithms (ML-DSA, Falcon) that don't have
+				 *    a traditional digest mapping
+				 * 2. Invalid/corrupted certificates
+				 *
+				 * Use X509_get_signature_nid() to check if this is a valid
+				 * signature algorithm. If valid, use SHA-256 for hashing.
+				 */
+				int sig_nid = X509_get_signature_nid(peer_cert);
+				
+				if (sig_nid != NID_undef && sig_nid > 0)
+				{
+					/*
+					 * Valid signature algorithm without digest mapping (likely
+					 * post-quantum). RFC 5929 doesn't clearly define this case.
+					 * We use SHA-256 as it's widely used, reasonably secure for
+					 * modern standards, and matches the security level of ML-DSA
+					 * which internally uses a 256-bit hash algorithm (SHAKE256).
+					 */
+					algo_type = EVP_sha256();
+				}
+				else
+				{
+					/* Invalid or corrupted certificate */
+					libpq_append_conn_error(conn, "could not find digest for NID %s",
+											OBJ_nid2sn(algo_nid));
+					return NULL;
+				}
 			}
 			break;
 	}
-- 
2.39.5 (Apple Git-154)