Failing SSL connection due to weird interaction with openssl

Lars Kanis <lars@greiz-reinsdorf.de>

From: Lars Kanis <lars@greiz-reinsdorf.de>
To: pgsql-hackers@postgresql.org
Date: 2012-10-23T08:09:40Z
Lists: pgsql-hackers

Attachments

While investigating a ruby-pg issue [1], we noticed that a libpq SSL 
connection can fail, if the running application uses OpenSSL for other 
work, too. Root cause is the thread local error queue of OpenSSL, that 
is used to transmit textual error messages to the application after a 
failed crypto operation. In case that the application leaves errors on 
the queue, the communication to the PostgreSQL server can fail with a 
message left from the previous failed OpenSSL operation, in particular 
when using non-blocking operations on the socket. This issue with 
openssl is quite old now - see [3].

For [1] it turned out that the issue is subdivided into these three parts:
1. the ruby-openssl binding does not clear the thread local error queue 
of OpenSSL after a certificate verify
2. OpenSSL makes use of a shared error queue for different crypto contexts.
3. libpq does not ensure a cleared error queue when doing SSL_* calls

To 1: Remaining messages on the error queue can generally lead to 
failing operations, later on. I'd talk to the ruby-openssl developers, 
to discuss how we can avoid any remaining messages on the queue.

To 2: SSL_get_error() inspects the shared error queue under some 
conditions. It's maybe poor API design, but it's documented behaviour 
[2]. So we certainly have to get along with it.

To 3: To make libpq independent to a previous error state, the error 
queue might be cleared with a call to ERR_clear_error() prior 
SSL_connect/read/write as in the attached trivial patch. This would make 
libpq robust against other uses of openssl within the application.

What do you think about clearing the OpenSSL error queue in libpq in 
that way?

[1] 
https://bitbucket.org/ged/ruby-pg/issue/142/async_exec-over-ssl-connection-can-fail-on
[2] http://www.openssl.org/docs/ssl/SSL_get_error.html
[3] 
http://www.educatedguesswork.org/movabletype/archives/2005/03/curse_you_opens.html