Thread

Disable OpenSSL compression

Albe Laurenz <laurenz.albe@wien.gv.at> — 2011-11-08T13:59:01Z

I ran into a performance problem described in this thread:

http://archives.postgresql.org/pgsql-performance/2011-10/msg00249.php
continued here:
http://archives.postgresql.org/pgsql-performance/2011-11/msg00045.php

OpenSSL compresses data by default, and that causes a
performance penalty of 100% and more, at least when
SELECTing larger bytea objects.
The backend process becomes CPU bound.

From OpenSSL version 1.0.0. on, compression can be
disabled.  The attached patch does that, and with that
patch I see dramatic performance improvements:

Unpatched:

samples %       image name         symbol name
6754    83.7861 libz.so.1.2.3      /lib64/libz.so.1.2.3
618      7.6665 libcrypto.so.1.0.0 /usr/lib64/libcrypto.so.1.0.0
534      6.6245 postgres           hex_encode
95       1.1785 libc-2.12.so       memcpy

Patched:

samples %       image name         symbol name
751     50.1670 libcrypto.so.1.0.0 /usr/lib64/libcrypto.so.1.0.0
594     39.6794 postgres           hex_encode
83       5.5444 libc-2.12.so       memcpy

(the test case is selecting one 27 MB bytea in text
mode over a localhost connection)

Are there any objections to this?

It is possible that this could cause a performance
regression for people who SELECT lots of compressible
data over really slow network connections, but is that
a realistic scenario?

If there are concerns about that, maybe a GUC variable like
ssl_compression (defaulting to off) would be a solution.

Yours,
Laurenz Albe

Re: Disable OpenSSL compression

Magnus Hagander <magnus@hagander.net> — 2011-11-08T14:09:11Z

On Tue, Nov 8, 2011 at 14:59, Albe Laurenz <laurenz.albe@wien.gv.at> wrote:
> I ran into a performance problem described in this thread:
>
> http://archives.postgresql.org/pgsql-performance/2011-10/msg00249.php
> continued here:
> http://archives.postgresql.org/pgsql-performance/2011-11/msg00045.php
>
> OpenSSL compresses data by default, and that causes a
> performance penalty of 100% and more, at least when
> SELECTing larger bytea objects.
> The backend process becomes CPU bound.
>
> From OpenSSL version 1.0.0. on, compression can be
> disabled.  The attached patch does that, and with that
> patch I see dramatic performance improvements:
>
> Unpatched:
>
> samples %       image name         symbol name
> 6754    83.7861 libz.so.1.2.3      /lib64/libz.so.1.2.3
> 618      7.6665 libcrypto.so.1.0.0 /usr/lib64/libcrypto.so.1.0.0
> 534      6.6245 postgres           hex_encode
> 95       1.1785 libc-2.12.so       memcpy
>
> Patched:
>
> samples %       image name         symbol name
> 751     50.1670 libcrypto.so.1.0.0 /usr/lib64/libcrypto.so.1.0.0
> 594     39.6794 postgres           hex_encode
> 83       5.5444 libc-2.12.so       memcpy
>
> (the test case is selecting one 27 MB bytea in text
> mode over a localhost connection)
>
> Are there any objections to this?

This should probably  be made an option.

And doesn't it, at least in a lot of cases, make more sense to control
this from the client side? It might typically be good to use
comopression if you are connecting over a slow link such as mobile or
satellite. And typically the client knows that, not the server. So
either client, or pg_hba driven, perhaps?


> It is possible that this could cause a performance
> regression for people who SELECT lots of compressible
> data over really slow network connections, but is that
> a realistic scenario?

Turning it off unconditionally can certainly create such a regression.
I don't think it's at all unrealstic.


-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

Re: Disable OpenSSL compression

Martin Pihlak <martin.pihlak@gmail.com> — 2011-11-08T14:23:14Z

On 11/08/2011 03:59 PM, Albe Laurenz wrote:
> If there are concerns about that, maybe a GUC variable like
> ssl_compression (defaulting to off) would be a solution.
> 

I'd vote for a libpq connect option instead. Something like
sslcompress=yes|no accompanied by PGSSLCOMPRESS environment
variable. And defaulting to "yes", as not to break any
backward compatibilty. For instance we expect SSL to provide
compression, wouldn't even use it without it.

Regards,
Martin

Re: Disable OpenSSL compression

Marko Kreen <markokr@gmail.com> — 2011-11-08T14:25:05Z

On Tue, Nov 8, 2011 at 3:59 PM, Albe Laurenz <laurenz.albe@wien.gv.at> wrote:
> It is possible that this could cause a performance
> regression for people who SELECT lots of compressible
> data over really slow network connections, but is that
> a realistic scenario?

Yes, it's a realistic scenario.  Please make it a option.

Also, high-security links may prefer compression.

-- 
marko

Re: Disable OpenSSL compression

Tom Lane <tgl@sss.pgh.pa.us> — 2011-11-08T14:34:00Z

Marko Kreen <markokr@gmail.com> writes:
> On Tue, Nov 8, 2011 at 3:59 PM, Albe Laurenz <laurenz.albe@wien.gv.at> wrote:
>> It is possible that this could cause a performance
>> regression for people who SELECT lots of compressible
>> data over really slow network connections, but is that
>> a realistic scenario?

> Yes, it's a realistic scenario.  Please make it a option.

I distinctly recall us getting bashed a few years ago because there
wasn't any convenient way to turn SSL compression *on*.  Now that SSL
finally does the sane thing by default, you want to turn it off?

The fact of the matter is that in most situations where you want SSL,
ie links across insecure WANs, compression is a win.  Testing a local
connection, as you seem to have done, is just about 100% irrelevant to
performance in the real world.

There might be some argument for providing a client option to disable
compression, but it should not be forced, and it shouldn't even be the
default.  But before adding YA connection option, I'd want to see some
evidence that it's useful over non-local connections.

			regards, tom lane

Re: Disable OpenSSL compression

Dave Page <dpage@pgadmin.org> — 2011-11-08T14:41:02Z

On Tue, Nov 8, 2011 at 2:34 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>
> The fact of the matter is that in most situations where you want SSL,
> ie links across insecure WANs, compression is a win.  Testing a local
> connection, as you seem to have done, is just about 100% irrelevant to
> performance in the real world.

I would disagree with that. Deployments in the cloud may have fast,
but untrustworthy network connections.

-- 
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: Disable OpenSSL compression

Andrew Dunstan <andrew@dunslane.net> — 2011-11-08T14:45:30Z


On 11/08/2011 09:34 AM, Tom Lane wrote:
> Marko Kreen<markokr@gmail.com>  writes:
>> On Tue, Nov 8, 2011 at 3:59 PM, Albe Laurenz<laurenz.albe@wien.gv.at>  wrote:
>>> It is possible that this could cause a performance
>>> regression for people who SELECT lots of compressible
>>> data over really slow network connections, but is that
>>> a realistic scenario?
>> Yes, it's a realistic scenario.  Please make it a option.
> I distinctly recall us getting bashed a few years ago because there
> wasn't any convenient way to turn SSL compression *on*.  Now that SSL
> finally does the sane thing by default, you want to turn it off?
>
> The fact of the matter is that in most situations where you want SSL,
> ie links across insecure WANs, compression is a win.  Testing a local
> connection, as you seem to have done, is just about 100% irrelevant to
> performance in the real world.
>
> There might be some argument for providing a client option to disable
> compression, but it should not be forced, and it shouldn't even be the
> default.  But before adding YA connection option, I'd want to see some
> evidence that it's useful over non-local connections.


I can certainly conceive of situations where one wants SSL on a high 
speed/bandwidth network. I don't think we should assume that all or even 
most real world SSL use will be across slow networks.

Here's another data point: 
<http://journal.paul.querna.org/articles/2011/04/05/openssl-memory-use/>


cheers

andrew

Re: Disable OpenSSL compression

Tom Lane <tgl@sss.pgh.pa.us> — 2011-11-08T14:58:03Z

Andrew Dunstan <andrew@dunslane.net> writes:
> On 11/08/2011 09:34 AM, Tom Lane wrote:
>> There might be some argument for providing a client option to disable
>> compression, but it should not be forced, and it shouldn't even be the
>> default.  But before adding YA connection option, I'd want to see some
>> evidence that it's useful over non-local connections.

> I can certainly conceive of situations where one wants SSL on a high 
> speed/bandwidth network. I don't think we should assume that all or even 
> most real world SSL use will be across slow networks.

Even for that use-case, I don't believe that testing on a local loopback
connection should be considered representative.

			regards, tom lane

Re: Disable OpenSSL compression

Robert Haas <robertmhaas@gmail.com> — 2011-11-08T15:03:13Z

On Tue, Nov 8, 2011 at 9:58 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Andrew Dunstan <andrew@dunslane.net> writes:
>> On 11/08/2011 09:34 AM, Tom Lane wrote:
>>> There might be some argument for providing a client option to disable
>>> compression, but it should not be forced, and it shouldn't even be the
>>> default.  But before adding YA connection option, I'd want to see some
>>> evidence that it's useful over non-local connections.
>
>> I can certainly conceive of situations where one wants SSL on a high
>> speed/bandwidth network. I don't think we should assume that all or even
>> most real world SSL use will be across slow networks.
>
> Even for that use-case, I don't believe that testing on a local loopback
> connection should be considered representative.

Probably not, but I think we ought to provide the option to disable
compression for those who want to do that.  I also agree with you that
we should leave the default as-is.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: Disable OpenSSL compression

Marko Kreen <markokr@gmail.com> — 2011-11-08T15:05:46Z

On Tue, Nov 8, 2011 at 4:34 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Marko Kreen <markokr@gmail.com> writes:
>> On Tue, Nov 8, 2011 at 3:59 PM, Albe Laurenz <laurenz.albe@wien.gv.at> wrote:
>>> It is possible that this could cause a performance
>>> regression for people who SELECT lots of compressible
>>> data over really slow network connections, but is that
>>> a realistic scenario?
>
>> Yes, it's a realistic scenario.  Please make it a option.
>
> I distinctly recall us getting bashed a few years ago because there
> wasn't any convenient way to turn SSL compression *on*.  Now that SSL
> finally does the sane thing by default, you want to turn it off?
>
> The fact of the matter is that in most situations where you want SSL,
> ie links across insecure WANs, compression is a win.  Testing a local
> connection, as you seem to have done, is just about 100% irrelevant to
> performance in the real world.
>
> There might be some argument for providing a client option to disable
> compression, but it should not be forced, and it shouldn't even be the
> default.  But before adding YA connection option, I'd want to see some
> evidence that it's useful over non-local connections.

+1 for keeping current default.

But I can imagine scenarios where having option to turn compression
off could be useful:
- when minimal latency is required
- when "normal" latency is required, but data is big
- when serving big non-compressible blobs - zlib can be very slow
- when serving lots of connections and want
  to minimize unnecessary cpu and memory load

Depending on how zlib is used by openssl, some of
them may not happen in practice.

-- 
marko

Re: Disable OpenSSL compression

Albe Laurenz <laurenz.albe@wien.gv.at> — 2011-11-08T15:19:02Z

Tom Lane wrote:
> I distinctly recall us getting bashed a few years ago because there
> wasn't any convenient way to turn SSL compression *on*.  Now that SSL
> finally does the sane thing by default, you want to turn it off?
> 
> The fact of the matter is that in most situations where you want SSL,
> ie links across insecure WANs, compression is a win.  Testing a local
> connection, as you seem to have done, is just about 100% irrelevant to
> performance in the real world.

Maybe that's paranoia, but we use SSL via the company's LAN to keep
potentially sensitive data from crossing the network unencrypted.

> There might be some argument for providing a client option to disable
> compression, but it should not be forced, and it shouldn't even be the
> default.  But before adding YA connection option, I'd want to see some
> evidence that it's useful over non-local connections.

I will try to provide test results via remote connection; I thought
that localhost was a good enough simulation for a situation where
you are not network bound.

I agree with you that a client option would make more sense.
The big problem I personally have with that is that it only works
if you use libpq. When using the JDBC driver or Npgsql, a client
option wouldn't help me at all.

Yours,
Laurenz Albe

Re: Disable OpenSSL compression

Albe Laurenz <laurenz.albe@wien.gv.at> — 2011-11-08T15:59:02Z

Tom Lane wrote:
> There might be some argument for providing a client option to disable
> compression, but it should not be forced, and it shouldn't even be the
> default.  But before adding YA connection option, I'd want to see some
> evidence that it's useful over non-local connections.

Here are numbers from a test via LAN.
The client machine has OpenSSL 0.9.8e, the server OpenSSL 1.0.0.

The client command run was

echo 'select ...' | time psql "host=..." -o /dev/null

and \timing was turned on in .psqlrc

In addition to the oprofile data I collected three times:
- the duration as shown in the server log
- the duration as shown by \timing
- the duration of the psql command as measured by "time"

Without patch:

duration: 5730.996 ms (log), 5975.093 ms (\timing), 22.87 s (time)

samples  %        image name               symbol name
4428     80.2029  libz.so.1.2.3            /lib64/libz.so.1.2.3
559      10.1250  postgres                 hex_encode
361       6.5387  libcrypto.so.1.0.0       /usr/lib64/libcrypto.so.1.0.0
83        1.5034  libc-2.12.so             memcpy

With patch:

duration: 3001.009 ms (log), 3243.690 ms (\timing), 20.27 s (time)

samples  %        image name               symbol name
1072     58.0401  libcrypto.so.1.0.0       /usr/lib64/libcrypto.so.1.0.0
587      31.7813  postgres                 hex_encode
105       5.6849  libc-2.12.so             memcpy


I think this makes a good case for disabling compression.

Yours,
Laurenz Albe

Re: Disable OpenSSL compression

Kenneth Marshall <ktm@rice.edu> — 2011-11-08T16:03:13Z

On Tue, Nov 08, 2011 at 04:59:02PM +0100, Albe Laurenz wrote:
> Tom Lane wrote:
> > There might be some argument for providing a client option to disable
> > compression, but it should not be forced, and it shouldn't even be the
> > default.  But before adding YA connection option, I'd want to see some
> > evidence that it's useful over non-local connections.
> 
> Here are numbers from a test via LAN.
> The client machine has OpenSSL 0.9.8e, the server OpenSSL 1.0.0.
> 
> The client command run was
> 
> echo 'select ...' | time psql "host=..." -o /dev/null
> 
> and \timing was turned on in .psqlrc
> 
> In addition to the oprofile data I collected three times:
> - the duration as shown in the server log
> - the duration as shown by \timing
> - the duration of the psql command as measured by "time"
> 
> Without patch:
> 
> duration: 5730.996 ms (log), 5975.093 ms (\timing), 22.87 s (time)
> 
> samples  %        image name               symbol name
> 4428     80.2029  libz.so.1.2.3            /lib64/libz.so.1.2.3
> 559      10.1250  postgres                 hex_encode
> 361       6.5387  libcrypto.so.1.0.0       /usr/lib64/libcrypto.so.1.0.0
> 83        1.5034  libc-2.12.so             memcpy
> 
> With patch:
> 
> duration: 3001.009 ms (log), 3243.690 ms (\timing), 20.27 s (time)
> 
> samples  %        image name               symbol name
> 1072     58.0401  libcrypto.so.1.0.0       /usr/lib64/libcrypto.so.1.0.0
> 587      31.7813  postgres                 hex_encode
> 105       5.6849  libc-2.12.so             memcpy
> 
> 
> I think this makes a good case for disabling compression.
> 
> Yours,
> Laurenz Albe

Certainly a good case for providing the option to disable compression.

Regards,
Ken

Re: Disable OpenSSL compression

Kenneth Marshall <ktm@rice.edu> — 2011-11-08T16:06:12Z

On Tue, Nov 08, 2011 at 04:19:02PM +0100, Albe Laurenz wrote:
> Tom Lane wrote:
> > I distinctly recall us getting bashed a few years ago because there
> > wasn't any convenient way to turn SSL compression *on*.  Now that SSL
> > finally does the sane thing by default, you want to turn it off?
> > 
> > The fact of the matter is that in most situations where you want SSL,
> > ie links across insecure WANs, compression is a win.  Testing a local
> > connection, as you seem to have done, is just about 100% irrelevant to
> > performance in the real world.
> 
> Maybe that's paranoia, but we use SSL via the company's LAN to keep
> potentially sensitive data from crossing the network unencrypted.
> 
> > There might be some argument for providing a client option to disable
> > compression, but it should not be forced, and it shouldn't even be the
> > default.  But before adding YA connection option, I'd want to see some
> > evidence that it's useful over non-local connections.
> 
> I will try to provide test results via remote connection; I thought
> that localhost was a good enough simulation for a situation where
> you are not network bound.
> 
> I agree with you that a client option would make more sense.
> The big problem I personally have with that is that it only works
> if you use libpq. When using the JDBC driver or Npgsql, a client
> option wouldn't help me at all.
> 
> Yours,
> Laurenz Albe
> 

I think that JDBC and Npgsql should also support disabling compression.

Regards,
Ken

Re: Disable OpenSSL compression

Christopher Browne <cbbrowne@gmail.com> — 2011-11-08T16:50:17Z

On Tue, Nov 8, 2011 at 11:06 AM, ktm@rice.edu <ktm@rice.edu> wrote:
> I think that JDBC and Npgsql should also support disabling compression.

That's the *real* problem here...

You're quite right that if we allow controlling this on the libpq
side, it is surely desirable to allow controlling this via JDBC,
Npgsql, and other mechanisms people may have around.  (There are
native protocol implementations for Common Lisp and Go, for instance.
They may not be particularly important, )

Unfortunately, each protocol implementation is independent, which
really is the nature of the beast, which includes:
a) The code of the implementation,
b) Release of the implementation,
c) Packaging of releases into software distributions.

With that series of complications, I wonder if maybe the right place
to control this is pg_hba.conf.  That has the merit of centralizing
control in such a way that it would apply commonly to
libpq/JDBC/Npgsql/..., though with the demerit that the control does
not take place on the client side, which is desirable.

I wonder how many SSL parameters there are which would be worth trying
to have available.  I expect we'd benefit from looking at all the
relevant ones at once, so as to not have the problem of hacking "one
more" into place and perhaps doing it a bit differently each time.
-- 
When confronted by a difficult problem, solve it by reducing it to the
question, "How would the Lone Ranger handle this?"

Re: Disable OpenSSL compression

Jeroen Vermeulen <jtv@xs4all.nl> — 2011-11-08T17:20:29Z

On 2011-11-08 22:59, Albe Laurenz wrote:

> In addition to the oprofile data I collected three times:
> - the duration as shown in the server log
> - the duration as shown by \timing
> - the duration of the psql command as measured by "time"

[...]

> I think this makes a good case for disabling compression.

It's a few data points, but is it enough to make a good case?  As I 
understand it, compression can save time not only on transport but also 
on the amount of data that needs to go through encryption -- probably 
depending on choice of cypher, hardware support, machine word width, 
compilation details etc.  Would it make sense to run a wider experiment, 
e.g. in the buld farm?

Another reason why I believe compression is often used with encryption 
is to maximize information content per byte of data: harder to guess, 
harder to crack.  Would that matter?

Jeroen

Re: Disable OpenSSL compression

Tom Lane <tgl@sss.pgh.pa.us> — 2011-11-08T17:39:05Z

Jeroen Vermeulen <jtv@xs4all.nl> writes:
> Another reason why I believe compression is often used with encryption 
> is to maximize information content per byte of data: harder to guess, 
> harder to crack.  Would that matter?

Yes, it would.  There's a reason why the OpenSSL default is what it is.

			regards, tom lane

Re: Disable OpenSSL compression

Stephen Frost <sfrost@snowman.net> — 2011-11-08T18:46:46Z

Tom,

* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> The fact of the matter is that in most situations where you want SSL,
> ie links across insecure WANs, compression is a win.  Testing a local
> connection, as you seem to have done, is just about 100% irrelevant to
> performance in the real world.

I'm mystified by the idea that SSL shouldn't be used on local networks.
If the only things talking to the database are other servers on
physically secure networks, perhaps, but when you've got databases
exposed (even through firewalls) to client networks (which are in the
same building), and any data that's even remotely sensetive, you should
be using SSL or IPSEC.  The chances of eaves-dropping on a typiacal WAN
physical/dedicated link (not over the Internet..) are actually much less
than some disgruntled employee spoofing the local switches to monitor
someone else's traffic.  For starters, you're going to need some pretty
specialized gear to eavesdrop on a T1 or similar link and once it's past
the last mile and into the fibre network...  Well, there's some folks
who can manage that, but it's not very many.

> There might be some argument for providing a client option to disable
> compression, but it should not be forced, and it shouldn't even be the
> default.  But before adding YA connection option, I'd want to see some
> evidence that it's useful over non-local connections.

I agree that it should be an option and that it should be on by default.
It's going to typically be a win.

	Thanks,

		Stephen

Re: Disable OpenSSL compression

Albe Laurenz <laurenz.albe@wien.gv.at> — 2011-11-09T11:27:20Z

Christopher Browne wrote:
>> I think that JDBC and Npgsql should also support disabling
compression.
>
> That's the *real* problem here...
> 
> You're quite right that if we allow controlling this on the libpq
> side, it is surely desirable to allow controlling this via JDBC,
> Npgsql, and other mechanisms people may have around.
[...]
> With that series of complications, I wonder if maybe the right place
> to control this is pg_hba.conf.

I think that wouldn't work, because to query pg_hba.conf, you have to
know user and database, which come from the client side.
But the SSL negotiation takes place earlier, namely when the
connection is established.

> I wonder how many SSL parameters there are which would be worth trying
> to have available.  I expect we'd benefit from looking at all the
> relevant ones at once, so as to not have the problem of hacking "one
> more" into place and perhaps doing it a bit differently each time.

Sure, if anybody can think of any.  A quick look at
"man SSL_CTX_set_options" didn't show me any, but then OpenSSL's
documentation is very bad (the page does not even mention
SSL_OP_NO_COMPRESSION) and I am no SSL expert.

Is the following proposal acceptable:

- Add a GUC ssl_compression, defaulting to "on".
- Add a client option "sslcompression" and an environment variable
  PGSSLCOMPRESSION, defaulting to "1".

Compression will be disabled if either side refuses.

That way you can control the setting per client, but you can also
force it on clients that do not use libpq if you want.

Yours,
Laurenz Albe

Re: Disable OpenSSL compression

Magnus Hagander <magnus@hagander.net> — 2011-11-09T18:07:36Z

On Wednesday, November 9, 2011, Albe Laurenz wrote:

> Christopher Browne wrote:
> >> I think that JDBC and Npgsql should also support disabling
> compression.
> >
> > That's the *real* problem here...
> >
> > You're quite right that if we allow controlling this on the libpq
> > side, it is surely desirable to allow controlling this via JDBC,
> > Npgsql, and other mechanisms people may have around.
> [...]
> > With that series of complications, I wonder if maybe the right place
> > to control this is pg_hba.conf.
>
> I think that wouldn't work, because to query pg_hba.conf, you have to
> know user and database, which come from the client side.
> But the SSL negotiation takes place earlier, namely when the
> connection is established.
>

Oh, right, that's going to be a problem doing it there.


>
> > I wonder how many SSL parameters there are which would be worth trying
> > to have available.  I expect we'd benefit from looking at all the
> > relevant ones at once, so as to not have the problem of hacking "one
> > more" into place and perhaps doing it a bit differently each time.
>
> Sure, if anybody can think of any.  A quick look at
> "man SSL_CTX_set_options" didn't show me any, but then OpenSSL's
> documentation is very bad (the page does not even mention
> SSL_OP_NO_COMPRESSION) and I am no SSL expert.
>

Oh yeah, their docs are. Um. Yeah, let's leave it at that.

I think the other one is to control which encryption options are available
- but we already have a guc for that.


Is the following proposal acceptable:
>
> - Add a GUC ssl_compression, defaulting to "on".
> - Add a client option "sslcompression" and an environment variable
>  PGSSLCOMPRESSION, defaulting to "1".
>

Seems like the reasonable thing, yes.


Compression will be disabled if either side refuses.
>

I assume OpenSSL takes care of this for us, right? We just have to set the
flags on the connection?




-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

Re: Disable OpenSSL compression

Tom Lane <tgl@sss.pgh.pa.us> — 2011-11-09T22:26:14Z

Magnus Hagander <magnus@hagander.net> writes:
> On Wednesday, November 9, 2011, Albe Laurenz wrote:
>> Is the following proposal acceptable:
>> 
>> - Add a GUC ssl_compression, defaulting to "on".
>> - Add a client option "sslcompression" and an environment variable
>> PGSSLCOMPRESSION, defaulting to "1".

> Seems like the reasonable thing, yes.

A GUC is entirely, completely, 100% the wrong answer.  It has no way to
deal with the fact that some clients may need compression and others
not.

It should be a client option, full stop.  The fact that that will be
more work to implement does not make "kluge it at the server" the right
answer.

			regards, tom lane

Re: Disable OpenSSL compression

Noah Misch <noah@leadboat.com> — 2011-11-10T02:29:19Z

On Wed, Nov 09, 2011 at 05:26:14PM -0500, Tom Lane wrote:
> Magnus Hagander <magnus@hagander.net> writes:
> > On Wednesday, November 9, 2011, Albe Laurenz wrote:
> >> Is the following proposal acceptable:
> >> 
> >> - Add a GUC ssl_compression, defaulting to "on".
> >> - Add a client option "sslcompression" and an environment variable
> >> PGSSLCOMPRESSION, defaulting to "1".
> 
> > Seems like the reasonable thing, yes.
> 
> A GUC is entirely, completely, 100% the wrong answer.  It has no way to
> deal with the fact that some clients may need compression and others
> not.
> 
> It should be a client option, full stop.  The fact that that will be
> more work to implement does not make "kluge it at the server" the right
> answer.

Assuming the GUC is PGC_BACKEND or better, what is the distinction beyond the
cosmetic "sslcompression=off" vs. "options='-c ssl_compression=off'"?

Does OpenSSL respect changes to this setting during a connection's lifetime?  If
so, we could offer a PGC_USERSET GUC, enabling the client to even alternate use
of compression within a single session.

Re: Disable OpenSSL compression

Albe Laurenz <laurenz.albe@wien.gv.at> — 2011-11-10T08:42:25Z

Magnus Hagander wrote:
>> Is the following proposal acceptable:
>>
>> - Add a GUC ssl_compression, defaulting to "on".
>> - Add a client option "sslcompression" and an environment variable
>>   PGSSLCOMPRESSION, defaulting to "1".
> 
> Seems like the reasonable thing, yes.
> 
>> Compression will be disabled if either side refuses.
> 
> I assume OpenSSL takes care of this for us, right? We just have to set
the flags on the connection?

Right.

Yours,
Laurenz Albe

Re: Disable OpenSSL compression

Andrew Dunstan <andrew@dunslane.net> — 2011-11-10T14:22:51Z

On 11/08/2011 12:39 PM, Tom Lane wrote:
> Jeroen Vermeulen<jtv@xs4all.nl>  writes:
>> Another reason why I believe compression is often used with encryption
>> is to maximize information content per byte of data: harder to guess,
>> harder to crack.  Would that matter?
> Yes, it would.  There's a reason why the OpenSSL default is what it is.
>
> 			

An interesting data point on this is that RedHat's nss_compat_ossl 
package doesn't support SSL compression at all 
<http://fedoraproject.org/wiki/Nss_compat_ossl>, and it's supposed to be 
a path to FIPS 140 compliance: 
<http://fedoraproject.org/wiki/FedoraCryptoConsolidation>. The latter 
URL, incidentally, contains a lot of good information, and lays out many 
of the reasons why I'd like to see us support NSS as an alternative to 
OpenSSL, notwithstanding the supposed dirtiness of its API. I imagine 
this would be of interest to commercial Postgres vendors also.

cheers

andrew

Re: Disable OpenSSL compression

Magnus Hagander <magnus@hagander.net> — 2011-11-10T14:47:57Z

On Thursday, November 10, 2011, Andrew Dunstan wrote:

>
>
> On 11/08/2011 12:39 PM, Tom Lane wrote:
>
>> Jeroen Vermeulen<jtv@xs4all.nl>  writes:
>>
>>> Another reason why I believe compression is often used with encryption
>>> is to maximize information content per byte of data: harder to guess,
>>> harder to crack.  Would that matter?
>>>
>> Yes, it would.  There's a reason why the OpenSSL default is what it is.
>>
>>
>>
>
>
> An interesting data point on this is that RedHat's nss_compat_ossl package
> doesn't support SSL compression at all <http://fedoraproject.org/**
> wiki/Nss_compat_ossl <http://fedoraproject.org/wiki/Nss_compat_ossl>>,
> and it's supposed to be a path to FIPS 140 compliance: <
> http://fedoraproject.org/**wiki/FedoraCryptoConsolidation<http://fedoraproject.org/wiki/FedoraCryptoConsolidation>
> **>. The latter URL, incidentally, contains a lot of good information,
> and lays out many of the reasons why I'd like to see us support NSS as an
> alternative to OpenSSL, notwithstanding the supposed dirtiness of its API.
> I imagine this would be of interest to commercial Postgres vendors also.


Interesting points. I hadn't really considered it from the FIPS perspective.

I thought the main idea was that if we want to support another one it's
probably going to be GnuTLS because that one offers key-file-compatibility
with OpenSSL, which NSS doesnät.

//Magnus



-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

Re: Disable OpenSSL compression

Albe Laurenz <laurenz.albe@wien.gv.at> — 2011-11-10T15:18:41Z

Tom Lane wrote:
>>> Is the following proposal acceptable:
>>>
>>> - Add a GUC ssl_compression, defaulting to "on".
>>> - Add a client option "sslcompression" and an environment variable
>>> PGSSLCOMPRESSION, defaulting to "1".

> A GUC is entirely, completely, 100% the wrong answer.  It has no way
to
> deal with the fact that some clients may need compression and others
> not.

If you leave the GUC at its default value, you can control compression
on the client side.

You can force a certain SSL cipher on the client, why not a compression
setting?

> It should be a client option, full stop.  The fact that that will be
> more work to implement does not make "kluge it at the server" the
right
> answer.

I could go and try to convince Npgsql and JDBC to accept patches to
do that on the client side, but that would be more effort than I
want to invest.  But then there's still closed source software like
Devart dotConnect...

In my environment it would make sense to control the setting on the
server side, because all our database clients connect via LAN, and
network bandwidth is not the bottleneck in our database applications.

Yours,
Laurenz Albe

Re: Disable OpenSSL compression

Tom Lane <tgl@sss.pgh.pa.us> — 2011-11-10T16:01:54Z

"Albe Laurenz" <laurenz.albe@wien.gv.at> writes:
> Tom Lane wrote:
>> A GUC is entirely, completely, 100% the wrong answer.  It has no way
>> to deal with the fact that some clients may need compression and others
>> not.

> You can force a certain SSL cipher on the client, why not a compression
> setting?

To my mind, the argument for the ssl_cipher setting is to allow the DBA
to enforce a site-wide security policy to the effect of "we consider
only these ciphers strong enough for production use".  It's a pretty
weak argument (especially since the setting is not cognizant of where
the connection is coming from), but at least it's an argument.

There's no comparable security argument for an ssl_compression setting.
If anything, a security-minded DBA might wish to insist on compression
being *on*, but you aren't proposing to give him control in that
direction (and AFAICT openssl doesn't offer a force-on flag for it).

But in any case, my objection is that there's no adequate use-case
for this GUC, because it's much more sensible to set it from the client
side.  We have too many GUCs already --- Josh B regularly goes on the
warpath looking for ones we can remove.  This one should never get in
there to start with.

> I could go and try to convince Npgsql and JDBC to accept patches to
> do that on the client side, but that would be more effort than I
> want to invest.  But then there's still closed source software like
> Devart dotConnect...

This argument reads as nothing except "I'm too lazy to solve it right,
so I want you to accept a wrong solution".

			regards, tom lane

Re: Disable OpenSSL compression

Robert Haas <robertmhaas@gmail.com> — 2011-11-10T16:09:34Z

On Thu, Nov 10, 2011 at 11:01 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> But in any case, my objection is that there's no adequate use-case
> for this GUC, because it's much more sensible to set it from the client
> side.  We have too many GUCs already --- Josh B regularly goes on the
> warpath looking for ones we can remove.  This one should never get in
> there to start with.

Of course, we also have no shortage of connection parameters.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: Disable OpenSSL compression

Marko Kreen <markokr@gmail.com> — 2011-11-10T16:24:20Z

On Thu, Nov 10, 2011 at 5:18 PM, Albe Laurenz <laurenz.albe@wien.gv.at> wrote:
> I could go and try to convince Npgsql and JDBC to accept patches to
> do that on the client side, but that would be more effort than I
> want to invest.  But then there's still closed source software like
> Devart dotConnect...

Are you certain Java/C# even support SSL compression?
Quick grep over some old jdk code I had around
did not find it...

In any case, the connection libraries do not need to match
such optional features.

-- 
marko

Re: Disable OpenSSL compression

Bruce Momjian <bruce@momjian.us> — 2011-11-10T21:53:53Z

Tom Lane wrote:
> "Albe Laurenz" <laurenz.albe@wien.gv.at> writes:
> > Tom Lane wrote:
> >> A GUC is entirely, completely, 100% the wrong answer.  It has no way
> >> to deal with the fact that some clients may need compression and others
> >> not.
> 
> > You can force a certain SSL cipher on the client, why not a compression
> > setting?
> 
> To my mind, the argument for the ssl_cipher setting is to allow the DBA
> to enforce a site-wide security policy to the effect of "we consider
> only these ciphers strong enough for production use".  It's a pretty
> weak argument (especially since the setting is not cognizant of where
> the connection is coming from), but at least it's an argument.
> 
> There's no comparable security argument for an ssl_compression setting.
> If anything, a security-minded DBA might wish to insist on compression
> being *on*, but you aren't proposing to give him control in that
> direction (and AFAICT openssl doesn't offer a force-on flag for it).
> 
> But in any case, my objection is that there's no adequate use-case
> for this GUC, because it's much more sensible to set it from the client
> side.  We have too many GUCs already --- Josh B regularly goes on the
> warpath looking for ones we can remove.  This one should never get in
> there to start with.

How is the compression connection parameter set?  It seems odd for it to
be compiled into the application because the application could be run on
different networks.  I don't know of any way to inject connection
options from outside the application like libpq's PGOPTIONS.  Would we
add a libpq environment variable for this, like PGUSER?

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + It's impossible for everything to be true. +

Re: Disable OpenSSL compression

Tom Lane <tgl@sss.pgh.pa.us> — 2011-11-10T22:12:43Z

Bruce Momjian <bruce@momjian.us> writes:
> How is the compression connection parameter set?  It seems odd for it to
> be compiled into the application because the application could be run on
> different networks.  I don't know of any way to inject connection
> options from outside the application like libpq's PGOPTIONS.

Huh?  You put it in the connection string, typically.  This is not
different from how you'd specify sslmode to start with.

			regards, tom lane

Re: Disable OpenSSL compression

Bruce Momjian <bruce@momjian.us> — 2011-11-10T22:39:18Z

Tom Lane wrote:
> Bruce Momjian <bruce@momjian.us> writes:
> > How is the compression connection parameter set?  It seems odd for it to
> > be compiled into the application because the application could be run on
> > different networks.  I don't know of any way to inject connection
> > options from outside the application like libpq's PGOPTIONS.
> 
> Huh?  You put it in the connection string, typically.  This is not
> different from how you'd specify sslmode to start with.

Well, you are saying the client is more flexible, but if the client is a
binary, it isn't flexible without an environment variable to control it.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + It's impossible for everything to be true. +

Re: Disable OpenSSL compression

Tom Lane <tgl@sss.pgh.pa.us> — 2011-11-10T22:45:10Z

Bruce Momjian <bruce@momjian.us> writes:
> Tom Lane wrote:
>> Huh?  You put it in the connection string, typically.  This is not
>> different from how you'd specify sslmode to start with.

> Well, you are saying the client is more flexible, but if the client is a
> binary, it isn't flexible without an environment variable to control it.

As long as the client can take a connection string, it's perfectly
flexible.  If it can't, this is just one more reason why it should
be fixed to do so.

			regards, tom lane

Re: Disable OpenSSL compression

Albe Laurenz <laurenz.albe@wien.gv.at> — 2011-11-11T15:32:04Z

Tom Lane wrote:
> But in any case, my objection is that there's no adequate use-case
> for this GUC, because it's much more sensible to set it from the
client
> side.  We have too many GUCs already --- Josh B regularly goes on the
> warpath looking for ones we can remove.  This one should never get in
> there to start with.

I agree that it is sensible to have the setting on the client,
and that there should not be too many GUCs.

>> I could go and try to convince Npgsql and JDBC to accept patches to
>> do that on the client side, but that would be more effort than I
>> want to invest.  But then there's still closed source software like
>> Devart dotConnect...
> 
> This argument reads as nothing except "I'm too lazy to solve it right,
> so I want you to accept a wrong solution".

In a way, yes, except that I think that "wrong" is exaggerated.
As DBA I like to have an option to control things from the server
end -- if that's laziness, so be it.

So, should I forget about the GUC or is anybody going to back me?

I'd still be willing to write a patch for a client-only solution.

Yours,
Laurenz Albe

Re: Disable OpenSSL compression

Bruce Momjian <bruce@momjian.us> — 2011-11-11T20:11:42Z

Albe Laurenz wrote:
> Tom Lane wrote:
> > But in any case, my objection is that there's no adequate use-case
> > for this GUC, because it's much more sensible to set it from the
> client
> > side.  We have too many GUCs already --- Josh B regularly goes on the
> > warpath looking for ones we can remove.  This one should never get in
> > there to start with.
> 
> I agree that it is sensible to have the setting on the client,
> and that there should not be too many GUCs.
> 
> >> I could go and try to convince Npgsql and JDBC to accept patches to
> >> do that on the client side, but that would be more effort than I
> >> want to invest.  But then there's still closed source software like
> >> Devart dotConnect...
> > 
> > This argument reads as nothing except "I'm too lazy to solve it right,
> > so I want you to accept a wrong solution".
> 
> In a way, yes, except that I think that "wrong" is exaggerated.
> As DBA I like to have an option to control things from the server
> end -- if that's laziness, so be it.
> 
> So, should I forget about the GUC or is anybody going to back me?
> 
> I'd still be willing to write a patch for a client-only solution.

Agreed.  There is clearly a win in turning off SSL compression for
certain workloads, and if people think the client is the right location,
then let's do it there.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + It's impossible for everything to be true. +

Re: Disable OpenSSL compression

Albe Laurenz <laurenz.albe@wien.gv.at> — 2011-11-15T14:39:18Z

Bruce Momjian wrote:
>> I'd still be willing to write a patch for a client-only solution.
> 
> Agreed.  There is clearly a win in turning off SSL compression for
> certain workloads, and if people think the client is the right
location,
> then let's do it there.

Here it is. I'll add it to the November commitfest.

Yours,
Laurenz Albe

Re: Disable OpenSSL compression

Albe Laurenz <laurenz.albe@wien.gv.at> — 2011-11-17T09:11:45Z

I wrote:
> Here it is. I'll add it to the November commitfest.

Here is the second version.

I realized that it is better to set the option on the SSL object
and not on the SSL context so that it is possible to change it
per connection.

I also improved the documentation.

Yours,
Laurenz Albe

Re: Disable OpenSSL compression

Magnus Hagander <magnus@hagander.net> — 2011-11-28T12:21:02Z

On Thu, Nov 17, 2011 at 10:11, Albe Laurenz <laurenz.albe@wien.gv.at> wrote:
> I wrote:
>> Here it is. I'll add it to the November commitfest.
>
> Here is the second version.
>
> I realized that it is better to set the option on the SSL object
> and not on the SSL context so that it is possible to change it
> per connection.
>
> I also improved the documentation.

Thanks, applied.

I changed the documentation around a bit further - I think it's easier
to read now.

-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/