Thread

1. psql v16.3 successfully connects via TLSv1.3 proxy, but psql v16.4 says "tlsv1 alert no application protocol"

   Markus KARG <markus@headcrashing.eu> — 2024-12-25T17:05:29Z

   Summary: The TLS behavior of psql changed between 16.3 and 16.4 as shown 
   below. If this is a bug, I kindly ask to fix it. If this is intended 
   behavior, I kindly as for a link with instructions how to work around 
   the problem.
   
   
   Server:
   
   I am running the official PostgreSQL 17.2 Docker Container 
   (https://hub.docker.com/layers/library/postgres/17.2/images/sha256-c063081175f45f4a3a5ac03c234e060e67618ebe75b49e2a7ffb79f8357bd1e6) 
   proxied by a TLSv1.3 proxy (official Traefik 3.2.3 Docker Container 
   https://hub.docker.com/layers/library/traefik/v3.2.3/images/sha256-06966a9ba1747ad724a490b8f27df1434c64e8eee5d681df03c4761c9653f62c). 
   Traefik utilizes ACME with Let's Encrypt to produce the TLS certificate.
   
   I have neither reconfigured TLS in any other way, nor have I manually 
   provided TLS certificates to neither client nor server.
   
   
   Client:
   
   Using the official PostgresSQL Docker Container (16.3 vs 16.4+), I am 
   asking psql to connect to my server. While psql 16.3 and earlier 
   versions successfully connect via the TLS proxy to the PostgreSQL 
   server, psql 16.4 and later versions fail doing so:
   
   root@hetzner-2:~# docker run -it postgres:16.3 psql "host=headcrashing.eu port=5432 
   dbname=postgres user=postgres password=... sslmode=require"
   psql (16.3 (Debian 16.3-1.pgdg120+1), server 17.2 (Debian 17.2-1.pgdg120+1))
   WARNING: psql major version 16, server major version 17.
             Some psql features might not work.
   SSL connection (protocol: TLSv1.3, cipher: TLS_AES_128_GCM_SHA256, compression: off)
   Type"help" for help.
   
   postgres=# \q
   root@hetzner-2:~# docker run -it postgres:16.4 psql "host=headcrashing.eu port=5432 
   dbname=postgres user=postgres password=... sslmode=require"
   psql: error: connection to server at"headcrashing.eu" (49.13.53.107), port 5432 failed: SSL error: tlsv1 alert no application protocol
   
   
   Public Test Environment
   
   Feel free to connect to my personal PostgreSQL 17 instance running at 
   |postgres.headcrashing.eu:5432| (TLS required).
   
   
   With kind regards
   
   -Markus
   

2. Re: psql v16.3 successfully connects via TLSv1.3 proxy, but psql v16.4 says "tlsv1 alert no application protocol"

   Tom Lane <tgl@sss.pgh.pa.us> — 2024-12-25T17:39:22Z

   Markus KARG <markus@headcrashing.eu> writes:
   > Summary: The TLS behavior of psql changed between 16.3 and 16.4 as shown 
   > below.
   
   Have you tried the current release (16.6)?
   
   			regards, tom lane
   
   
   
   

3. Re: psql v16.3 successfully connects via TLSv1.3 proxy, but psql v16.4 says "tlsv1 alert no application protocol"

   Markus KARG <markus@headcrashing.eu> — 2024-12-25T17:45:06Z

   > Markus KARG<markus@headcrashing.eu> writes:
   >> Summary: The TLS behavior of psql changed between 16.3 and 16.4 as shown
   >> below.
   > Have you tried the current release (16.6)?
   >
   > 			regards, tom lane
   
   
   Yes, but psql 16.6 fails in the same way:
   
   root@hetzner-2:~# docker run -it postgres:16.6 psql 
   "host=headcrashing.eu port=5432 dbname=postgres user=postgres 
   password=HeadcrashingPostgres2022 sslmode=require"
   psql: error: connection to server at "headcrashing.eu" (49.13.53.107), 
   port 5432 failed: SSL error: tlsv1 alert no application protocol
   
   -Markus
   

4. Re: psql v16.3 successfully connects via TLSv1.3 proxy, but psql v16.4 says "tlsv1 alert no application protocol"

   Heikki Linnakangas <hlinnaka@iki.fi> — 2024-12-25T17:52:55Z

   On 25/12/2024 19:05, Markus KARG wrote:
   > I am running the official PostgreSQL 17.2 Docker Container (https:// 
   > hub.docker.com/layers/library/postgres/17.2/images/sha256- 
   > c063081175f45f4a3a5ac03c234e060e67618ebe75b49e2a7ffb79f8357bd1e6) 
   > proxied by a TLSv1.3 proxy (official Traefik 3.2.3 Docker Container 
   > https://hub.docker.com/layers/library/traefik/v3.2.3/images/ 
   > sha256-06966a9ba1747ad724a490b8f27df1434c64e8eee5d681df03c4761c9653f62c). Traefik utilizes ACME with Let's Encrypt to produce the TLS certificate.
   
   In v17, libpq requests the ALPN extension in the TLS handshake. Looks 
   like the proxy doesn't know about the "postgresql" ALPN protocol, and 
   rejects the connection.
   
   I guess Traefik needs some configuration changes to tell it that the 
   "postgresql" protocol is expected. Or code changes.
   
   > Using the official PostgresSQL Docker Container (16.3 vs 16.4+), I am 
   > asking psql to connect to my server. While psql 16.3 and earlier 
   > versions successfully connect via the TLS proxy to the PostgreSQL 
   > server, psql 16.4 and later versions fail doing so:
   > 
   > root@hetzner-2:~# docker run -it postgres:16.3 psql "host=headcrashing.eu port=5432 
   > dbname=postgres user=postgres password=... sslmode=require"
   > psql (16.3 (Debian 16.3-1.pgdg120+1), server 17.2 (Debian 17.2-1.pgdg120+1))
   > WARNING: psql major version 16, server major version 17.
   >           Some psql features might not work.
   > SSL connection (protocol: TLSv1.3, cipher: TLS_AES_128_GCM_SHA256, compression: off)
   > Type"help" for help.
   > 
   > postgres=# \q
   > root@hetzner-2:~# docker run -it postgres:16.4 psql "host=headcrashing.eu port=5432 
   > dbname=postgres user=postgres password=... sslmode=require"
   > psql: error: connection to server at"headcrashing.eu" (49.13.53.107), port 5432 failed: SSL error: tlsv1 alert no application protocol
   
   There were no changes between 16.3 and 16.4 to explain this. When I test 
   that with v16 client that I built from sources, I don't get that error.
   
   The error message suggests that you're actually using libpq v17. And 
   indeed I get that error when connecting with v17 client. Perhaps the 
   postgres:16.4 docker image was built with v17 libpq?
   
   -- 
   Heikki Linnakangas
   Neon (https://neon.tech)
   
   
   
   
   

5. Re: psql v16.3 successfully connects via TLSv1.3 proxy, but psql v16.4 says "tlsv1 alert no application protocol"

   Markus KARG <markus@headcrashing.eu> — 2024-12-25T18:22:19Z

   > On 25/12/2024 19:05, Markus KARG wrote:
   >> I am running the official PostgreSQL 17.2 Docker Container (https:// 
   >> hub.docker.com/layers/library/postgres/17.2/images/sha256- 
   >> c063081175f45f4a3a5ac03c234e060e67618ebe75b49e2a7ffb79f8357bd1e6) 
   >> proxied by a TLSv1.3 proxy (official Traefik 3.2.3 Docker Container 
   >> https://hub.docker.com/layers/library/traefik/v3.2.3/images/ 
   >> sha256-06966a9ba1747ad724a490b8f27df1434c64e8eee5d681df03c4761c9653f62c). 
   >> Traefik utilizes ACME with Let's Encrypt to produce the TLS certificate.
   >
   > In v17, libpq requests the ALPN extension in the TLS handshake. Looks 
   > like the proxy doesn't know about the "postgresql" ALPN protocol, and 
   > rejects the connection.
   >
   > I guess Traefik needs some configuration changes to tell it that the 
   > "postgresql" protocol is expected. Or code changes.
   
   Traefik does NOT REJECT the connection (if it would, the error message 
   from psql would be different).
   
   Traefik is "postgres-aware" already since 3.0.0, while I am running 3.2.3.
   
   Note that psql v16.3 works fine but psql v16.4 is not, so a change 
   introduced by v17 this CANNOT be the cause of the current problem.
   
   >> Using the official PostgresSQL Docker Container (16.3 vs 16.4+), I am 
   >> asking psql to connect to my server. While psql 16.3 and earlier 
   >> versions successfully connect via the TLS proxy to the PostgreSQL 
   >> server, psql 16.4 and later versions fail doing so:
   >>
   >> root@hetzner-2:~# docker run -it postgres:16.3 psql 
   >> "host=headcrashing.eu port=5432 dbname=postgres user=postgres 
   >> password=... sslmode=require"
   >> psql (16.3 (Debian 16.3-1.pgdg120+1), server 17.2 (Debian 
   >> 17.2-1.pgdg120+1))
   >> WARNING: psql major version 16, server major version 17.
   >>           Some psql features might not work.
   >> SSL connection (protocol: TLSv1.3, cipher: TLS_AES_128_GCM_SHA256, 
   >> compression: off)
   >> Type"help" for help.
   >>
   >> postgres=# \q
   >> root@hetzner-2:~# docker run -it postgres:16.4 psql 
   >> "host=headcrashing.eu port=5432 dbname=postgres user=postgres 
   >> password=... sslmode=require"
   >> psql: error: connection to server at"headcrashing.eu" (49.13.53.107), 
   >> port 5432 failed: SSL error: tlsv1 alert no application protocol
   >
   > There were no changes between 16.3 and 16.4 to explain this. When I 
   > test that with v16 client that I built from sources, I don't get that 
   > error.
   >
   > The error message suggests that you're actually using libpq v17. And 
   > indeed I get that error when connecting with v17 client. Perhaps the 
   > postgres:16.4 docker image was built with v17 libpq?
   I am using the original, pre-built container images found on Docker Hub 
   and have NOT built them on my own. I am not a PostgresSQL committer 
   either. So I cannot answer your question.
   
   
   
   

6. Re: psql v16.3 successfully connects via TLSv1.3 proxy, but psql v16.4 says "tlsv1 alert no application protocol"

   Markus KARG <markus@headcrashing.eu> — 2024-12-25T18:52:30Z

   >> On 25/12/2024 19:05, Markus KARG wrote:
   >>> I am running the official PostgreSQL 17.2 Docker Container (https:// 
   >>> hub.docker.com/layers/library/postgres/17.2/images/sha256- 
   >>> c063081175f45f4a3a5ac03c234e060e67618ebe75b49e2a7ffb79f8357bd1e6) 
   >>> proxied by a TLSv1.3 proxy (official Traefik 3.2.3 Docker Container 
   >>> https://hub.docker.com/layers/library/traefik/v3.2.3/images/ 
   >>> sha256-06966a9ba1747ad724a490b8f27df1434c64e8eee5d681df03c4761c9653f62c). 
   >>> Traefik utilizes ACME with Let's Encrypt to produce the TLS 
   >>> certificate.
   >>
   >> In v17, libpq requests the ALPN extension in the TLS handshake. Looks 
   >> like the proxy doesn't know about the "postgresql" ALPN protocol, and 
   >> rejects the connection.
   >>
   >> I guess Traefik needs some configuration changes to tell it that the 
   >> "postgresql" protocol is expected. Or code changes.
   
   Following this theory, sslnegotiation=postgres used with psql v17.x 
   should solve the problem, as it falls back to the same mechanism 
   supported in v16.3, correct? Unfortunately it fails with the same error 
   message:
   
   root@hetzner-2:~# docker run -it postgres:17.2 psql 
   "host=headcrashing.eu port=5432 dbname=postgres user=postgres 
   password=... sslmode=require sslnegotiation=postgres"
   
   psql: error: connection to server at "headcrashing.eu" (49.13.53.107), 
   port 5432 failed: SSL error: tlsv1 alert no application protocol
   
   The same happens when using sslnegotication=direct.
   
   Regarding your question: I have tried "sslnegotiation=postgres" and 
   "ssqlnegotiation=direct" with the original official PostgreSQL Docker 
   Container found on Docker Hub. Using v16.3 psql says it doesn't know the 
   option "sslnegotiation", and 16.4...17 say "tlsv1 alert no application 
   protocol". Does PROOFS your claim that the original images on Docker Hub 
   are using libpq v17 even for psql v16.4+!
   
   -Markus
   
   -Markus
   
   
   
   
   

7. Re: psql v16.3 successfully connects via TLSv1.3 proxy, but psql v16.4 says "tlsv1 alert no application protocol"

   Tom Lane <tgl@sss.pgh.pa.us> — 2024-12-25T19:37:00Z

   Markus KARG <markus@headcrashing.eu> writes:
   > Following this theory, sslnegotiation=postgres used with psql v17.x 
   > should solve the problem, as it falls back to the same mechanism 
   > supported in v16.3, correct?
   
   No, sorry, it looks like we insert the ALPN extension into the SSL
   request packet regardless of that.  Maybe there should have been
   a way to suppress that, but v17 libpq doesn't provide one.
   
   > Regarding your question: I have tried "sslnegotiation=postgres" and 
   > "ssqlnegotiation=direct" with the original official PostgreSQL Docker 
   > Container found on Docker Hub.
   
   Let's clarify something here: there is nothing "official" about either
   that docker container or anything else you might find on DockerHub.
   The Postgres community does not produce any such packaging.  You'd
   have to discussion the inclusion of v17 libpq with whoever did build
   that container.  I know that Debian thinks it's a good idea to use
   latest libpq with older servers, and this builder might be following
   their lead.  (The timing would be about right, since v17.0 came out
   about the same time as 16.4.)  This example does show that there
   are pitfalls in that policy.
   
   			regards, tom lane
   
   
   
   

8. Re: psql v16.3 successfully connects via TLSv1.3 proxy, but psql v16.4 says "tlsv1 alert no application protocol"

   Markus KARG <markus@headcrashing.eu> — 2024-12-25T22:47:15Z

   > Markus KARG<markus@headcrashing.eu> writes:
   >> Following this theory, sslnegotiation=postgres used with psql v17.x
   >> should solve the problem, as it falls back to the same mechanism
   >> supported in v16.3, correct?
   > No, sorry, it looks like we insert the ALPN extension into the SSL
   > request packet regardless of that.  Maybe there should have been
   > a way to suppress that, but v17 libpq doesn't provide one.
   
   Thank you, Tom, you saved my day! I have explicitly added the name 
   `postgresql` to the ALPN configuration on Traefik, and immediately psql 
   v16.4+ (including psql v17) worked like a charm!
   
   As a reference for others, in `traefik-dynamic.yaml` add the follwing to 
   make it work (as long as you patch the `default` TLS options, take care 
   to keep all those protocols contained; alternative path a per-router tls 
   config instead to ONLY have postgresql in that list):
   
        ## Dynamic configuration
        tls:
          options:
            default:
              alpnProtocols:
                - acme-tls/1
                - http/1.1
                - h2
                - postgresql
   
   >> Regarding your question: I have tried "sslnegotiation=postgres" and
   >> "ssqlnegotiation=direct" with the original official PostgreSQL Docker
   >> Container found on Docker Hub.
   > Let's clarify something here: there is nothing "official" about either
   > that docker container or anything else you might find on DockerHub.
   > The Postgres community does not produce any such packaging.  You'd
   > have to discussion the inclusion of v17 libpq with whoever did build
   > that container.  I know that Debian thinks it's a good idea to use
   > latest libpq with older servers, and this builder might be following
   > their lead.  (The timing would be about right, since v17.0 came out
   > about the same time as 16.4.)  This example does show that there
   > are pitfalls in that policy.
   
   Thanks a lot for this really essential information! In fact, the authors 
   of that official docker image (they really call it that way) call 
   themselves "The PostgreSQL Docker Community". This is totally 
   misleading, as one understands that to be "The Docker Community withing 
   the PostgreSQL Community", which it (apparently) is not. Instead, it 
   seems that it actually is "The PostgreSQL Community at Docker, Inc.", 
   people totally unrelated to the "real" PostgreSQL Community!
   
   Thanks a lot, Tom, for your kind help. You provided the best Christmas 
   present this year! :-)