Thread

1. Unnecessary connection overhead due copy-on-write (mainly openssl)

   Andres Freund <andres@anarazel.de> — 2025-06-05T19:58:32Z

   Hi,
   
   Looking at [1] I, again, noticed that a decent portion of our connection
   overhead is due to openssl's atexit handler.
   
   On my older workstation (with a few noisy things running):
   
   c=16;pgbench -n -M prepared -c$c -j$c -P1 -T10 -f <(echo 'select') -C
   -> 3057 TPS
   
   If I change the exit() in proc_exit() to a _exit():
   -> 3633 TPS
   
   The reason for this difference is that by default openssl registers an atexit
   handler that frees a lot of memory that was initialized in postmaster. That in
   turn triggers page-faults due to the relevant pages now differing in child
   processes. Which a) isn't cheap b) causes contention with postmaster, since
   those datastructures are shared.
   
   
   It's possible to tell openssl to not register an atexit handler, see [2]:
   
   > OPENSSL_INIT_NO_ATEXIT
   >   By default OpenSSL will attempt to clean itself up when the process exits via
   >   an "atexit" handler. Using this option suppresses that behaviour. This means
   >   that the application will have to clean up OpenSSL explicitly using
   >   OPENSSL_cleanup().
   
   One slight difficulty is that we initialize openssl somewhat indirectly, via
   PostmasterMain()->InitProcessGlobals()->pg_prng_strong_seed() which then, if
   built with openssl support, triggers initialization within RAND_status().
   
   
   The quick hack of putting
   
   #ifdef USE_OPENSSL
   	OPENSSL_init_crypto(OPENSSL_INIT_NO_ATEXIT, NULL);
   #endif
   
   at the start of PostmasterMain() gets the connection speed up a fair bit:
   -> 3449 TPS
   
   
   The reason this isn't as good as using _exit is that there are other libraries
   with (effectively) atexit handlers. In particular ICU pulls in libstdc++,
   which in turn seems to have a lot of destructors for global objects that
   aren't cheap.
   
   If I build without ICU support, the connection rate with exit() (and the
   openssl "fix") is
   -> 3863 TPS
   and if I use _exit() it is
   -> 3900 TPS
   
   I.e. at that point the remaining atexit handlers only play a small role.
   
   I don't know if there's a decent solution for the nontrivial overhead due to
   ICU -> libstdc++'s atexit handlers.
   
   
   
   There are a few related issues where we ourselves to blame. The most prominent
   one is that we go around and delete PostmasterContext in child processes. That
   however doesn't really save memory, as the memory is still needed in
   postmaster, we just end up causing page faults that trigger copy-on-write.
   
   If I just comment out the MemoryContextDelete in PostgresMain() I see
   connection rates improve from
   -> 3891 TPS
   to
   -> 4004 TPS
   
   
   If I build a much more minimal postgres, disabling all optional dependencies
   other than openssl I see a significant improvement, just due fewer mmaps for
   the libraries:
   -> 4865 TPS
   
   Further disabling openssl and zlib interestingly does not help, interestingly.
   
   
   Greetings,
   
   Andres Freund
   
   [1] https://postgr.es/m/CAFbpF8OA44_UG%2BRYJcWH9WjF7E3GA6gka3gvH6nsrSnEe9H0NA%40mail.gmail.com
   [2] https://docs.openssl.org/3.1/man3/OPENSSL_init_crypto/#name
   
   
   
   

2. Re: Unnecessary connection overhead due copy-on-write (mainly openssl)

   Peter Eisentraut <peter@eisentraut.org> — 2025-06-06T11:56:08Z

   On 05.06.25 21:58, Andres Freund wrote:
   > The reason for this difference is that by default openssl registers an atexit
   > handler that frees a lot of memory that was initialized in postmaster. That in
   > turn triggers page-faults due to the relevant pages now differing in child
   > processes. Which a) isn't cheap b) causes contention with postmaster, since
   > those datastructures are shared.
   > 
   > 
   > It's possible to tell openssl to not register an atexit handler, see [2]:
   > 
   >> OPENSSL_INIT_NO_ATEXIT
   >>    By default OpenSSL will attempt to clean itself up when the process exits via
   >>    an "atexit" handler. Using this option suppresses that behaviour. This means
   >>    that the application will have to clean up OpenSSL explicitly using
   >>    OPENSSL_cleanup().
   
   It seems weird to me that openssl spends so much effort tidying up its 
   memory allocations just before exiting.  We could just skip that. 
   Looking through the code of OPENSSL_cleanup(), there might be one or two 
   cases of log or trace files that get flushed during cleanup, so it's not 
   an absolute no-brainer to skip all the cleanup.
   
   
   
   
   

3. Re: Unnecessary connection overhead due copy-on-write (mainly openssl)

   Robert Haas <robertmhaas@gmail.com> — 2025-06-06T13:04:04Z

   On Thu, Jun 5, 2025 at 3:58 PM Andres Freund <andres@anarazel.de> wrote:
   > There are a few related issues where we ourselves to blame. The most prominent
   > one is that we go around and delete PostmasterContext in child processes. That
   > however doesn't really save memory, as the memory is still needed in
   > postmaster, we just end up causing page faults that trigger copy-on-write.
   
   If we're not going to bother deleting PostmasterContext, we could also
   skip creating it in the first place. After all, if the storage isn't
   actually freed, then we won't know whether things are leaking into
   that context that actually do get used in child processes, so there's
   really no point.
   
   The current structure amounts to a design decision that at some point
   in time the postmaster might allocate an amount of memory that we need
   to free in child processes, whether or not that's actually true
   currently. Not deleting it any more -- or not having it any more -- is
   deciding that it shouldn't ever allocate a significant amount of
   memory.
   
   I don't know whether that's a good bet, but I wouldn't be surprised. I
   think we've talked about wanting to move some things that the
   postmaster currently does to a separate process, whether for
   multi-threading or other reasons. But, if we do take the position that
   the postmaster shouldn't allocate a significant amount of stuff, we
   might want to add some checks someplace to prove that it doesn't.
   Otherwise, it might get broken by some future patch without anybody
   noticing.
   
   (For clarity, I'm not attempting to insist on anything here, just
   sharing a few thoughts that come to mind.)
   
   -- 
   Robert Haas
   EDB: http://www.enterprisedb.com
   
   
   
   

4. Re: Unnecessary connection overhead due copy-on-write (mainly openssl)

   Jacob Champion <jacob.champion@enterprisedb.com> — 2025-06-06T15:41:20Z

   On Fri, Jun 6, 2025 at 4:56 AM Peter Eisentraut <peter@eisentraut.org> wrote:
   > It seems weird to me that openssl spends so much effort tidying up its
   > memory allocations just before exiting.  We could just skip that.
   > Looking through the code of OPENSSL_cleanup(), there might be one or two
   > cases of log or trace files that get flushed during cleanup, so it's not
   > an absolute no-brainer to skip all the cleanup.
   
   I guess I'd be concerned that a hardware crypto provider might need
   good-faith cleanup to work well. I understand they can't rely on
   atexit in general, but there would be a big difference between "you
   might have to clean up after a crash" and "every single connection
   litters the hardware with unused stuff".
   
   But that's pure FUD and guesswork; I have no examples to point to, so
   there might not be any providers that need that.
   
   --Jacob
   
   
   
   

5. Re: Unnecessary connection overhead due copy-on-write (mainly openssl)

   Christoph Berg <myon@debian.org> — 2025-06-06T15:53:59Z

   Re: Jacob Champion
   > I guess I'd be concerned that a hardware crypto provider might need
   > good-faith cleanup to work well.
   
   Hopefully not in every single backend.
   
   Christoph
   
   
   
   

6. Re: Unnecessary connection overhead due copy-on-write (mainly openssl)

   Nico Williams <nico@cryptonector.com> — 2025-06-06T16:25:46Z

   On Fri, Jun 06, 2025 at 08:41:20AM -0700, Jacob Champion wrote:
   > I guess I'd be concerned that a hardware crypto provider might need
   > good-faith cleanup to work well. I understand they can't rely on
   > atexit in general, but there would be a big difference between "you
   > might have to clean up after a crash" and "every single connection
   > litters the hardware with unused stuff".
   
   I'd expect all subsystems to recover cleanly from unclean shutdowns.  I
   know, that's a lot to expect, but nowadays pretty much all filesystems
   used in production do, for example.
   
   > But that's pure FUD and guesswork; I have no examples to point to, so
   > there might not be any providers that need that.
   
   I doubt that PG w/ OpenSSL in any configuration maintains stateful
   interactions with HW cryptographic providers.
   
   
   
   

7. Re: Unnecessary connection overhead due copy-on-write (mainly openssl)

   Andres Freund <andres@anarazel.de> — 2025-06-06T16:37:40Z

   Hi,
   
   On 2025-06-06 08:41:20 -0700, Jacob Champion wrote:
   > On Fri, Jun 6, 2025 at 4:56 AM Peter Eisentraut <peter@eisentraut.org> wrote:
   > > It seems weird to me that openssl spends so much effort tidying up its
   > > memory allocations just before exiting.  We could just skip that.
   > > Looking through the code of OPENSSL_cleanup(), there might be one or two
   > > cases of log or trace files that get flushed during cleanup, so it's not
   > > an absolute no-brainer to skip all the cleanup.
   > 
   > I guess I'd be concerned that a hardware crypto provider might need
   > good-faith cleanup to work well. I understand they can't rely on
   > atexit in general, but there would be a big difference between "you
   > might have to clean up after a crash" and "every single connection
   > litters the hardware with unused stuff".
   
   It's not just crashes, e.g. the startup packet timeout is also handled by
   _exit() - and it can be triggered remotely. ISTM that if crypto providers
   can't handle _exit(), we have a bigger problem.
   
   Alternatively we could try deferring more of openssl's initialization to
   outside of postmaster - but that doesn't seem particularly realistic.
   
   Greetings,
   
   Andres Freund
   
   
   
   

8. Re: Unnecessary connection overhead due copy-on-write (mainly openssl)

   Jacob Champion <jacob.champion@enterprisedb.com> — 2025-06-06T18:58:38Z

   On Fri, Jun 6, 2025 at 9:25 AM Nico Williams <nico@cryptonector.com> wrote:
   > I'd expect all subsystems to recover cleanly from unclean shutdowns.  I
   > know, that's a lot to expect, but nowadays pretty much all filesystems
   > used in production do, for example.
   
   I guess, but if we stop cleaning up entirely, we will suddenly be
   stressing those code paths... But maybe that's a community service? :)
   
   I realize I'm making an argument from fear and ignorance. Maybe that
   ecosystem is very healthy. I'm just imagining the following
   conversation:
   
   DBA: we upgraded our server and our HSM is freaking out after a few
   thousand connections; what gives?
   us: oh, we stopped cleaning up after ourselves for performance! tell
   your vendor to fix their drivers!
   DBA: hahahaha
   
   [1] is a description of the kind of problem I'm worried about. (It's
   not 1:1 applicable to this situation, I just think we might start
   seeing those sorts of bug reports.)
   
   > I doubt that PG w/ OpenSSL in any configuration maintains stateful
   > interactions with HW cryptographic providers.
   
   (Why? From looking over the Cryptoki/PKCS#11 stuff, for example, isn't
   a lot of that API stateful?)
   
   --Jacob
   
   [1] https://github.com/OpenSC/libp11/issues/228#issuecomment-402941378
   
   
   
   

9. Re: Unnecessary connection overhead due copy-on-write (mainly openssl)

   Jacob Champion <jacob.champion@enterprisedb.com> — 2025-06-06T18:58:46Z

   On Fri, Jun 6, 2025 at 9:37 AM Andres Freund <andres@anarazel.de> wrote:
   > It's not just crashes, e.g. the startup packet timeout is also handled by
   > _exit() - and it can be triggered remotely.
   
   Fair point...
   
   > ISTM that if crypto providers
   > can't handle _exit(), we have a bigger problem.
   
   ...so I guess I need to figure out whether we have a bigger problem. I
   hope we don't.
   
   Note that OpenSSL seems to be interested in removing the atexit()
   handling altogether, and requiring applications to manually call the
   cleanup function, in 4.0. [1]
   
   --Jacob
   
   [1] https://github.com/openssl/openssl/issues/22508
   
   
   
   

10. Re: Unnecessary connection overhead due copy-on-write (mainly openssl)

    Nico Williams <nico@cryptonector.com> — 2025-06-06T20:18:50Z

    On Fri, Jun 06, 2025 at 11:58:38AM -0700, Jacob Champion wrote:
    > > I'd expect all subsystems to recover cleanly from unclean shutdowns.  I
    > > know, that's a lot to expect, but nowadays pretty much all filesystems
    > > used in production do, for example.
    > 
    > I guess, but if we stop cleaning up entirely, we will suddenly be
    > stressing those code paths... But maybe that's a community service? :)
    
    The latter.
    
    > I realize I'm making an argument from fear and ignorance. Maybe that
    > ecosystem is very healthy. I'm just imagining the following
    > conversation:
    > 
    > DBA: we upgraded our server and our HSM is freaking out after a few
    > thousand connections; what gives?
    > us: oh, we stopped cleaning up after ourselves for performance! tell
    > your vendor to fix their drivers!
    > DBA: hahahaha
    
    TPMs for example have a concept of session.  You can have up to 64 open
    sessions, and if you use the TPM resource manager and you're accessing
    it through a file descriptor then the RM will just clean up when you
    exit.  Though if you're accessing the raw TPM directly then fail to
    flush sessions then yes, you'll eventually be unable to create new ones.
    
    However no one will be using a discrete or firmware TPM for TLS server
    certificate private key usage: discrete TPMs are way way too slow for
    that, and firmware TPMs are... also way too slow.  You wouldn't bother
    with a software TPM for this unless it's for privilege separation.
    
    Anyways, if you were using a TPM then the user's startup scripts, or
    postgres itself could just flush all sessions and be done.
    
    Other types of hardware cryptographic providers also tend to have a
    notion of "session", and they all tend to have relatively paltry limits,
    which means that the software side that calls them will generally need
    to be prepared to a) close its own sessions eagerly (at the cost of
    extra overhead on the next operation), and b) recover from running out
    of sessions (by flushing others at the cost of causing those that were
    live to need retries).
    
    But anyways, IIUC the OpenSSL engine interface is itself stateless and I
    would expect providers to auto-recover.  And anyways I expect no one
    uses PG with HW cryptographic providers to perform TLS server
    signatures.  Instead the best current practice would be to use
    short-lived server certificates with software keys and longer-lived
    credentials in hardware with which to fetch new short-lived credentials
    with software keys.  The kinds of HSMs that can do high rates of
    signatures are neither cheap nor commonly used, and those do tend to
    have higher session limits, and again you can recover from running out
    of sessions by flushing extant sessions.
    
    > > I doubt that PG w/ OpenSSL in any configuration maintains stateful
    > > interactions with HW cryptographic providers.
    > 
    > (Why? From looking over the Cryptoki/PKCS#11 stuff, for example, isn't
    > a lot of that API stateful?)
    
    PKCS#11 is stateful, yes (it has session handles), but there are
    generally low limits on how many sessions you can keep open, therefore
    high pressure to close them soon, therefore the inference is that that
    must be what actually happens at the rather high cost of having to set
    up new sessions often.  That inference could be wrong, but then as you
    note you'd be doing the community a service by testing it and making it
    true in the future.
    
    Nico
    -- 
    
    
    
    

11. Re: Unnecessary connection overhead due copy-on-write (mainly openssl)

    Jacob Champion <jacob.champion@enterprisedb.com> — 2025-06-09T16:40:04Z

    On Fri, Jun 6, 2025 at 1:18 PM Nico Williams <nico@cryptonector.com> wrote:
    > However no one will be using a discrete or firmware TPM for TLS server
    > certificate private key usage: discrete TPMs are way way too slow for
    > that, and firmware TPMs are... also way too slow.  You wouldn't bother
    > with a software TPM for this unless it's for privilege separation.
    
    There are other cryptographic things users could be doing on the
    server side, too, via extensions, or even other library dependencies
    that themselves rely on OpenSSL.
    
    But in any case, what you've written seems reasonable to me, and I
    don't have any reason to gum up the works, so I'll pipe down. (I've
    asked around internally to see if there are any concerns, too; I'm
    happy to share if I find anything.)
    
    Thanks!
    --Jacob