Thread
-
Enquiry about TDE with PgSQL
Ashish Mukherjee <ashish.mukherjee@gmail.com> — 2025-10-13T15:55:27Z
Hello, Thank you to the group members for the responses to my previous questions about TDE problems with PgSQL 17. I would like to enquire that based on the anecdotal experience of group members, which TDE solution works best for PgSQL 17 databases. The scenario I have is of a large number of tables (15-20K) and some tables with 100M tuples each. The total database size is 4TB. I understand there may be variations and specific testing is required, however high-level pointers and feedback would be valuable. Regards, Ashish
-
Re: Enquiry about TDE with PgSQL
Greg Sabino Mullane <htamfids@gmail.com> — 2025-10-16T22:04:49Z
> > I would like to enquire that based on the anecdotal experience of group > members, which TDE solution works best for PgSQL 17 databases. Generally speaking, there is no "best". People use whatever vendor they happen to already use. Your best solution is to avoid TDE altogether. If you really need encryption at rest, have the OS do it. That works well (transparently, even), is very battle-tested, and has minimal performance impact. TDE, on the other hand, is a very complex and difficult thing to add into Postgres. Currently it means you are using a forked version of Postgres and are incurring overhead every time you read or write to disk. The scenario I have is of a large number of tables (15-20K) and some > tables with 100M tuples each. The total database size is 4TB. The size and number of tables does not really matter. How often you write WAL, and how often things move in and out of shared buffers is what matters. Cheers, Greg
-
Re: Enquiry about TDE with PgSQL
Ron Johnson <ronljohnsonjr@gmail.com> — 2025-10-17T04:49:06Z
On Thu, Oct 16, 2025 at 6:05 PM Greg Sabino Mullane <htamfids@gmail.com> wrote: > I would like to enquire that based on the anecdotal experience of group >> members, which TDE solution works best for PgSQL 17 databases. > > > Generally speaking, there is no "best". People use whatever vendor they > happen to already use. Your best solution is to avoid TDE altogether. If > you really need encryption at rest, have the OS do it. That works well > (transparently, even), is very battle-tested, and has minimal performance > impact. > But filesystem encryption still means that validly logged-in users see the unencrypted data. That's great for a laptop that might get stolen, or for drives that are discarded without being wiped, but are no protection against hackers who want to exfiltrate your data. (Neither protect against ransomware, but that's a different problem.) > TDE, on the other hand, is a very complex and difficult thing to add > into Postgres. > TDE was added to SQL Server, with (to us, at least) minimally-noticed overhead. Oracle has it, too, but I don't know the details. The bottom line is that requirements for TDE are escalating, whether you like it or not, as Yet Another Layer Of Defense against hackers exfiltrating data, and then threatening to leak it to the public. -- Death to <Redacted>, and butter sauce. Don't boil me, I'm still alive. <Redacted> lobster!
-
Re: Enquiry about TDE with PgSQL
Laurenz Albe <laurenz.albe@cybertec.at> — 2025-10-17T07:01:52Z
On Fri, 2025-10-17 at 00:49 -0400, Ron Johnson wrote: > On Thu, Oct 16, 2025 at 6:05 PM Greg Sabino Mullane <htamfids@gmail.com> wrote: > > > > TDE, on the other hand, is a very complex and difficult thing to add into Postgres. > > TDE was added to SQL Server, with (to us, at least) minimally-noticed overhead. > Oracle has it, too, but I don't know the details. > > The bottom line is that requirements for TDE are escalating, whether you like it or > not, as Yet Another Layer Of Defense against hackers exfiltrating data, and then > threatening to leak it to the public. Bruce Momjian has interesting things to say about that in https://compiledconversations.com/6/ (unfortunately I don't remember where exactly in this 84 minute piece). It is a feature that users want (or need to comply with whatever they feel they have to comply with). On the other hand, it has very limited technical or security value, which hampers its acceptance into core. Yours, Laurenz Albe
-
Re: Enquiry about TDE with PgSQL
Ron Johnson <ronljohnsonjr@gmail.com> — 2025-10-17T12:47:40Z
On Fri, Oct 17, 2025 at 3:01 AM Laurenz Albe <laurenz.albe@cybertec.at> wrote: > On Fri, 2025-10-17 at 00:49 -0400, Ron Johnson wrote: > > On Thu, Oct 16, 2025 at 6:05 PM Greg Sabino Mullane <htamfids@gmail.com> > wrote: > > > > > > TDE, on the other hand, is a very complex and difficult thing to add > into Postgres. > > > > TDE was added to SQL Server, with (to us, at least) minimally-noticed > overhead. > > Oracle has it, too, but I don't know the details. > > > > The bottom line is that requirements for TDE are escalating, whether you > like it or > > not, as Yet Another Layer Of Defense against hackers exfiltrating data, > and then > > threatening to leak it to the public. > > Bruce Momjian has interesting things to say about that in > https://compiledconversations.com/6/ (unfortunately I don't remember where > exactly in this 84 minute piece). > > It is a feature that users want (or need to comply with whatever they feel > they have to comply with). On the other hand, it has very limited > technical > or security value, which hampers its acceptance into core. > I gave you a reason: "Yet Another Layer Of Defense against hackers exfiltrating data". It's the same reason PgBackRest encrypts backups. -- Death to <Redacted>, and butter sauce. Don't boil me, I'm still alive. <Redacted> lobster!
-
Re: Enquiry about TDE with PgSQL
Greg Sabino Mullane <htamfids@gmail.com> — 2025-10-17T13:12:42Z
On Fri, Oct 17, 2025 at 12:49 AM Ron Johnson <ronljohnsonjr@gmail.com> wrote: But filesystem encryption still means that validly logged-in users see the > unencrypted data. That's great for a laptop that might get stolen, or for > drives that are discarded without being wiped, but are no protection > against hackers who want to exfiltrate your data. I stand by my recommendation. If someone is logged in and has access to your data directory (e.g. is root or postgres user), then they also have the TDE key or some easy way to bypass it. TDE was added to SQL Server, with (to us, at least) minimally-noticed > overhead. Oracle has it, too, but I don't know the details. > The bottom line is that requirements for TDE are escalating, whether you > like it or not I'm not arguing against putting TDE in Postgres - indeed, I am all for that. But it's a very tricky thing to do technically, with minimal benefits other than "checking the box" of some security requirements document. The bottom line is that requirements for TDE are escalating, whether you > like it or not, as Yet Another Layer Of Defense against hackers > exfiltrating data, and then threatening to leak it to the public. > I'd love to see a real-world example where TDE would have saved someone but disk encryption could not. -- Cheers, Greg -- Crunchy Data - https://www.crunchydata.com Enterprise Postgres Software Products & Tech Support
-
Re: Enquiry about TDE with PgSQL
rainer@ultra-secure.de — 2025-10-17T14:08:14Z
Am 2025-10-17 15:12, schrieb Greg Sabino Mullane: > On Fri, Oct 17, 2025 at 12:49 AM Ron Johnson > <ronljohnsonjr@gmail.com> wrote: > >> But filesystem encryption still means that validly logged-in users >> see the unencrypted data. That's great for a laptop that might get >> stolen, or for drives that are discarded without being wiped, but >> are no protection against hackers who want to exfiltrate your data. > > I stand by my recommendation. If someone is logged in and has access > to your data directory (e.g. is root or postgres user), then they also > have the TDE key or some easy way to bypass it. > >> TDE was added to SQL Server, with (to us, at least) >> minimally-noticed overhead. Oracle has it, too, but I don't know >> the details. >> The bottom line is that requirements for TDE are escalating, whether >> you like it or not > > I'm not arguing against putting TDE in Postgres - indeed, I am all for > that. But it's a very tricky thing to do technically, with minimal > benefits other than "checking the box" of some security requirements > document. > >> The bottom line is that requirements for TDE are escalating, whether >> you like it or not, as Yet Another Layer Of Defense against hackers >> exfiltrating data, and then threatening to leak it to the public. > > I'd love to see a real-world example where TDE would have saved > someone but disk encryption could not. At least with Oracle and TDE, the rman backups are encrypted, too. So, you cannot just download the dumps from a file-share and import them somewhere else (or if you get hold of the tapes, use those to restore the files). It's not really about protecting the files from a hacker but from a person inside the organization who wants to sell the data. TDE really only (IMHO) makes sense when you also have a HSM. Now, I've seen a case where a customer had Oracle + Dataguard + TDE + HSM. The thing is, you rarely interact with the HSM normally. So, it's getting a bit hazy in that department when you have to add a new secret...plus, while Oracle and the HSM vendor will happily license you the feature, they will happily point their fingers at each other in case a problem arrives. In this case, the HSM was wiped clear of all the keys in there, in an attempt to add a new key. Oracle closed the wallets and the database was offline. Dang! Now, you had fix figures worth of hardware doing exactly nothing. Due to incredibly lucky circumstances, it was possible to recover the database without data loss. So, with TDE, you really need to be sure you actually know what you're doing.
-
Re: Enquiry about TDE with PgSQL
Bruce Momjian <bruce@momjian.us> — 2025-10-30T19:59:58Z
On Fri, Oct 17, 2025 at 09:01:52AM +0200, Laurenz Albe wrote: > On Fri, 2025-10-17 at 00:49 -0400, Ron Johnson wrote: > > On Thu, Oct 16, 2025 at 6:05 PM Greg Sabino Mullane <htamfids@gmail.com> wrote: > > > > > > TDE, on the other hand, is a very complex and difficult thing to add into Postgres. > > > > TDE was added to SQL Server, with (to us, at least) minimally-noticed overhead. > > Oracle has it, too, but I don't know the details. > > > > The bottom line is that requirements for TDE are escalating, whether you like it or > > not, as Yet Another Layer Of Defense against hackers exfiltrating data, and then > > threatening to leak it to the public. > > Bruce Momjian has interesting things to say about that in > https://compiledconversations.com/6/ (unfortunately I don't remember where > exactly in this 84 minute piece). Here is my most recent blog about TDE: https://momjian.us/main/blogs/pgblog/2025.html#February_22_2025 -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Do not let urgent matters crowd out time for investment in the future.
-
Re: Enquiry about TDE with PgSQL
Kai Wagner <kai.wagner@percona.com> — 2025-10-31T14:01:48Z
As I personally believe, there is no real way around TDE in the future, either by extensibility of the core (start with the storage manager and move your way on from there), to make an extension possible, or by directly adding it to the core, there are more reasons coming or are already on their way. With the PCI DSS v4.1 standard, one key rule to comply with is, that "If PAN is stored, it must be rendered unreadable". Of course there are other ways, like tokenization, hashing etc. but this regulation is pushing towards at rest encryption in the long run, and not only disk encryption. We can dislike it, but we are already seeing the need coming from large industries and companies that they cannot work around this anymore, as the auditors doing the checkboxes do not really care about "good alternatives", as they do not even technically understand what this is about. They do compare postgres simply against other already in use databases at these orgs (MySQL or MongoDB), and as such, we are currently the only one that cannot be used in such a use case, at least not without the willingness of the auditor to make it happen. On Thu, Oct 30, 2025 at 9:00 PM Bruce Momjian <bruce@momjian.us> wrote: > On Fri, Oct 17, 2025 at 09:01:52AM +0200, Laurenz Albe wrote: > > On Fri, 2025-10-17 at 00:49 -0400, Ron Johnson wrote: > > > On Thu, Oct 16, 2025 at 6:05 PM Greg Sabino Mullane < > htamfids@gmail.com> wrote: > > > > > > > > TDE, on the other hand, is a very complex and difficult thing to add > into Postgres. > > > > > > TDE was added to SQL Server, with (to us, at least) minimally-noticed > overhead. > > > Oracle has it, too, but I don't know the details. > > > > > > The bottom line is that requirements for TDE are escalating, whether > you like it or > > > not, as Yet Another Layer Of Defense against hackers exfiltrating > data, and then > > > threatening to leak it to the public. > > > > Bruce Momjian has interesting things to say about that in > > > https://url.avanan.click/v2/r01/___https://compiledconversations.com/6/___.YXAzOnBlcmNvbmE6YTpnOjMxNTMyOGQ0MzIyODAzOTU2M2E4OWUxZjNjZmMyYTRlOjc6MDJjZDplMmZkOGI3NTExNjAzNDI4YzZlZTZjMDQwNjU1YWQyZTVlYzU4NmQ4NjMzYzQxZGVjNzUxMGM5MmM0YThkM2M5OnA6VDpO > (unfortunately I don't remember where > > exactly in this 84 minute piece). > > Here is my most recent blog about TDE: > > > https://url.avanan.click/v2/r01/___https://momjian.us/main/blogs/pgblog/2025.html%23February_22_2025___.YXAzOnBlcmNvbmE6YTpnOjMxNTMyOGQ0MzIyODAzOTU2M2E4OWUxZjNjZmMyYTRlOjc6ODI4Nzo4OTUwMTcwNDljNjA0OGYxNzU3MDhlMjhiNDIwZjNiNzNjYTZmNWJjZmM2MmNmNWJkMGFhNTllOTMzNjA2Y2EyOnA6VDpO > > -- > Bruce Momjian <bruce@momjian.us> > https://url.avanan.click/v2/r01/___https://momjian.us___.YXAzOnBlcmNvbmE6YTpnOjMxNTMyOGQ0MzIyODAzOTU2M2E4OWUxZjNjZmMyYTRlOjc6ZTBkOTozZDU5MmRlNGI0YTU5ZmIxM2UzNmE1NTgzY2U1YjBjNmZlZWMwNmEyNzBhYjdlYTlhNDhlZTU4MGVjMDQ4MTk5OnA6VDpO > EDB > https://url.avanan.click/v2/r01/___https://enterprisedb.com___.YXAzOnBlcmNvbmE6YTpnOjMxNTMyOGQ0MzIyODAzOTU2M2E4OWUxZjNjZmMyYTRlOjc6ZWFlNjoyYWE0NWVmY2EwZTBhNGM3Y2Q2NzQwNDQ5NmM5OGMwODkxNDUxYzY2YmI4NWZhNzM0NmUwZjI1Mzg4NzE4ZDhhOnA6VDpO > > Do not let urgent matters crowd out time for investment in the future. > > >
-
Re: Enquiry about TDE with PgSQL
Fernando Laudares Camargos <fernando.laudares.camargos@gmail.com> — 2025-10-31T14:41:40Z
Hi Ashish, If you are looking for "native" TDE in PostgreSQL, you won't find it in the PostgreSQL community distribution. If you search for "postgresql tde", you will find a list of vendors that provide TDE functions in their own distributions of PostgreSQL, such as: - https://www.cybertec-postgresql.com/en/products/cybertec-postgresql-enterprise-edition-pgee/ - https://www.enterprisedb.com/products/database - https://docs.percona.com/pg-tde/ What's important to understand is that if you migrate from PostgreSQL community to one of these distributions and encrypt your tables using the respective solution offered by the distribution, moving back to PostgreSQL Community will require a logical backup (pg_dump) and restore. Disclaimer: I work for Percona. On Mon, Oct 13, 2025 at 12:55 PM Ashish Mukherjee < ashish.mukherjee@gmail.com> wrote: > Hello, > > Thank you to the group members for the responses to my previous questions > about TDE problems with PgSQL 17. > > I would like to enquire that based on the anecdotal experience of group > members, which TDE solution works best for PgSQL 17 databases. The scenario > I have is of a large number of tables (15-20K) and some tables with 100M > tuples each. The total database size is 4TB. > > I understand there may be variations and specific testing is required, > however high-level pointers and feedback would be valuable. > > Regards, > Ashish >
-
Re: Enquiry about TDE with PgSQL
Bruce Momjian <bruce@momjian.us> — 2025-10-31T14:54:16Z
On Fri, Oct 31, 2025 at 03:01:48PM +0100, Kai Wagner wrote: > As I personally believe, there is no real way around TDE in the future, either > by extensibility of the core (start with the storage manager and move your way > on from there), to make an extension possible, or by directly adding it to the > core, there are more reasons coming or are already on their way. > > With the PCI DSS v4.1 standard, one key rule to comply with is, that "If PAN is Uh, I think you mean the 4.0.1 standard, which became active on January 1, 2025. I am surprised this is only being mentioned now: https://blog.pcisecuritystandards.org/just-published-pci-dss-v4-0-1 When will PCI DSS v4.0 be retired? As with all new versions of PCI DSS, there will be a period where both the current and updated version will be active at the same time. PCI DSS --> v4.0 will be retired on 31 December 2024. After that point, PCI DSS --> v4.0.1 will be the only active version of the standard supported by PCI SSC. While it was active on Jan 1, it became effective on March 31, 2025: Does PCI DSS v4.0.1 change the 31 March 2025 effective date for the new requirements? No. This limited revision does not impact the effective date of these new requirements. > stored, it must be rendered unreadable". Of course there are other ways, like > tokenization, hashing etc. but this regulation is pushing towards at rest > encryption in the long run, and not only disk encryption. We can dislike it, > but we are already seeing the need coming from large industries and companies > that they cannot work around this anymore, as the auditors doing the checkboxes > do not really care about "good alternatives", as they do not even technically > understand what this is about. They do compare postgres simply against other > already in use databases at these orgs (MySQL or MongoDB), and as such, we are > currently the only one that cannot be used in such a use case, at least not > without the willingness of the auditor to make it happen. I see the new specification that disk-level encryption is insufficient, starting on page 93 (page 97 in the PDF URL): https://www.middlebury.edu/sites/default/files/2025-01/PCI-DSS-v4_0_1.pdf#page=97 --> 3.5.1.2 If disk-level or partition-level encryption (rather than file-, column-, or field-level database encryption) is used to render PAN unreadable, it is implemented only as follows: • On removable electronic media OR • If used for non-removable electronic media, PAN is also --> rendered unreadable via another mechanism that meets Requirement 3.5.1. ... While disk or partition encryption may still be present on these types of devices, it cannot be the only mechanism used to protect PAN stored on those systems. Any stored PAN must also be rendered unreadable per Requirement 3.5.1—for example, through truncation or a data-level encryption mechanism. Full disk encryption helps to protect data in the event of physical loss of a disk and therefore its use is appropriate only for removable electronic media storage devices. Purpose Disk-level and partition-level encryption typically encrypts the entire disk or partition using the same key, with all data automatically decrypted when the system runs or when an authorized --> user requests it. For this reason, disk-level encryption is not --> appropriate to protect stored PAN on computers, laptops, servers, storage arrays, or any other system that provides transparent decryption upon user authentication. PAN is: https://www.middlebury.edu/sites/default/files/2025-01/PCI-DSS-v4_0_1.pdf#page=391 PAN Acronym for “primary account number.” Unique payment card number (credit, debit, or prepaid cards, etc.) that identifies the issuer and the cardholder account. So it seems we have somewhat of a stand-off, with the Postgres project questioning the value of TDE and the PCI writers doubling-down on specifying disk-level encryption as insufficient. The biggest possible downside of this standoff is that enterprises that need to meet PCI compliance specifications are forced to use specialized versions of Postgres or Postgres extensions that support TDE. The fact that it has been seven months since PCI 4.0.1 was effective, with little to no discussion or movement on adding TDE to community Postgres means to me that we are unlikely to see TDE added to community Postgres anytime soon. I have a small hope that adding compression to the writing of temporary files will reduce the code changes needed to encrypt temporary files, thus reducing the amount of TDE code changes needed. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Do not let urgent matters crowd out time for investment in the future.
-
Re: Enquiry about TDE with PgSQL
Adrian Klaver <adrian.klaver@aklaver.com> — 2025-10-31T15:21:18Z
On 10/31/25 07:54, Bruce Momjian wrote: > On Fri, Oct 31, 2025 at 03:01:48PM +0100, Kai Wagner wrote: >> With the PCI DSS v4.1 standard, one key rule to comply with is, that "If PAN is > > Uh, I think you mean the 4.0.1 standard, which became active on January > 1, 2025. I am surprised this is only being mentioned now: > So it seems we have somewhat of a stand-off, with the Postgres project > questioning the value of TDE and the PCI writers doubling-down on > specifying disk-level encryption as insufficient. Yeah, what I would like to know is how many of the data breaches actually grab directly from the storage versus getting it through the database or other software above the storage? It seems to me social engineering plays a bigger role in this. -- Adrian Klaver adrian.klaver@aklaver.com
-
Re: Enquiry about TDE with PgSQL
Greg Sabino Mullane <htamfids@gmail.com> — 2025-10-31T15:25:04Z
On Fri, Oct 31, 2025 at 10:54 AM Bruce Momjian <bruce@momjian.us> wrote: > Disk-level and partition-level encryption typically encrypts > the entire disk or partition using the same key, with all data > automatically decrypted when the system runs or when an authorized > --> user requests it. For this reason, disk-level encryption is not > --> appropriate to protect stored PAN on computers, laptops, servers, > storage arrays, or any other system that provides transparent > decryption upon user authentication. > Hmm, I read this a few times but still not sure what the technical objection is. Yes, the entire disk is encrypted with the same key, but why is that insufficient to protect things? Anyone care to guess what they are thinking here? The biggest possible downside of this standoff is that enterprises that > need to meet PCI compliance specifications are forced to use specialized > versions of Postgres or Postgres extensions that support TDE. > Not always a downside for the companies selling those specialized versions though. Cheers, Greg -- Crunchy Data - https://www.crunchydata.com Enterprise Postgres Software Products & Tech Support
-
Re: Enquiry about TDE with PgSQL
Bruce Momjian <bruce@momjian.us> — 2025-10-31T15:34:30Z
On Fri, Oct 31, 2025 at 11:25:04AM -0400, Greg Sabino Mullane wrote: > On Fri, Oct 31, 2025 at 10:54 AM Bruce Momjian <bruce@momjian.us> wrote: > > Disk-level and partition-level encryption typically encrypts > the entire disk or partition using the same key, with all data > automatically decrypted when the system runs or when an authorized > --> user requests it. For this reason, disk-level encryption is not > --> appropriate to protect stored PAN on computers, laptops, servers, > storage arrays, or any other system that provides transparent > decryption upon user authentication. > > > Hmm, I read this a few times but still not sure what the technical objection > is. Yes, the entire disk is encrypted with the same key, but why is that > insufficient to protect things? Anyone care to guess what they are thinking > here? This is more an argument for security using layers. With storage encryption, the file system as visible is unencrypted, and backups of that file system can be unencrypted. Community Postgres relies on file system permissions to secure the data directory. You can argue that once file system permissions are bypassed, security is impossible, and agree with that, but some feel the extra step needed to pull the Postgres encryption key from memory is a security feature. And for backups, the Postgres encryption key might not be in memory. Of course, forcing encrypted backups is a solution, but an extra step. > The biggest possible downside of this standoff is that enterprises that > need to meet PCI compliance specifications are forced to use specialized > versions of Postgres or Postgres extensions that support TDE. > > Not always a downside for the companies selling those specialized versions > though. Yes, no question they are happy. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Do not let urgent matters crowd out time for investment in the future.
-
Re: Enquiry about TDE with PgSQL
Ron Johnson <ronljohnsonjr@gmail.com> — 2025-10-31T15:34:45Z
On Fri, Oct 31, 2025 at 11:25 AM Greg Sabino Mullane <htamfids@gmail.com> wrote: > On Fri, Oct 31, 2025 at 10:54 AM Bruce Momjian <bruce@momjian.us> wrote: > >> Disk-level and partition-level encryption typically encrypts >> the entire disk or partition using the same key, with all data >> automatically decrypted when the system runs or when an authorized >> --> user requests it. For this reason, disk-level encryption is not >> --> appropriate to protect stored PAN on computers, laptops, servers, >> storage arrays, or any other system that provides transparent >> decryption upon user authentication. >> > > Hmm, I read this a few times but still not sure what the technical > objection is. Yes, the entire disk is encrypted with the same key, but why > is that insufficient to protect things? Anyone care to guess what they are > thinking here? > Networking. Who breaks into a DC and steals a rack of disks or SSDs? Very, very few evil-doers. Who hacks into networks and exfiltrates data over the wire? Many hackers. -- Death to <Redacted>, and butter sauce. Don't boil me, I'm still alive. <Redacted> lobster!
-
Re: Enquiry about TDE with PgSQL
Adrian Klaver <adrian.klaver@aklaver.com> — 2025-10-31T15:37:44Z
On 10/31/25 08:25, Greg Sabino Mullane wrote: > On Fri, Oct 31, 2025 at 10:54 AM Bruce Momjian <bruce@momjian.us > <mailto:bruce@momjian.us>> wrote: > > Disk-level and partition-level encryption typically encrypts > the entire disk or partition using the same key, with all data > automatically decrypted when the system runs or when an > authorized > --> user requests it. For this reason, disk-level encryption is not > --> appropriate to protect stored PAN on computers, laptops, > servers, > storage arrays, or any other system that provides transparent > decryption upon user authentication. > > > Hmm, I read this a few times but still not sure what the technical > objection is. Yes, the entire disk is encrypted with the same key, but > why is that insufficient to protect things? Anyone care to guess what > they are thinking here? My best guess, is that the more the storage encryption is fragmented by different keys the less likely all the disk could be decrypted by a single action. The weak link is '... other system that provides transparent decryption upon user authentication.'. At some point the data needs to be decrypted for end user use. That means the point of attack is moved to the user and storage encryption does not really matter. > > The biggest possible downside of this standoff is that enterprises > that need to meet PCI compliance specifications are forced to use > specialized versions of Postgres or Postgres extensions that support > TDE. > > > Not always a downside for the companies selling those specialized > versions though. > > Cheers, > Greg > > -- > Crunchy Data - https://www.crunchydata.com <https://www.crunchydata.com> > Enterprise Postgres Software Products & Tech Support > -- Adrian Klaver adrian.klaver@aklaver.com
-
Re: Enquiry about TDE with PgSQL
Laurenz Albe <laurenz.albe@cybertec.at> — 2025-10-31T16:40:31Z
On Fri, 2025-10-31 at 08:21 -0700, Adrian Klaver wrote: > Yeah, what I would like to know is how many of the data breaches > actually grab directly from the storage versus getting it through the > database or other software above the storage? It seems to me social > engineering plays a bigger role in this. This is not about actual security considerations, it is about checkboxes. Consequently, rational arguments are missing the point. Yours, Laurenz Albe
-
Re: Enquiry about TDE with PgSQL
Bruce Momjian <bruce@momjian.us> — 2025-10-31T16:49:25Z
On Fri, Oct 31, 2025 at 05:40:31PM +0100, Laurenz Albe wrote: > On Fri, 2025-10-31 at 08:21 -0700, Adrian Klaver wrote: > > Yeah, what I would like to know is how many of the data breaches > > actually grab directly from the storage versus getting it through the > > database or other software above the storage? It seems to me social > > engineering plays a bigger role in this. > > This is not about actual security considerations, it is about checkboxes. > Consequently, rational arguments are missing the point. I think the big question is that, now with the effective PCI spec disallowing only storage-level encryption, can we, as a project, continue to reject in-core TDE because it is a check-box item. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Do not let urgent matters crowd out time for investment in the future.
-
Re: Enquiry about TDE with PgSQL
Adrian Klaver <adrian.klaver@aklaver.com> — 2025-10-31T17:01:07Z
On 10/31/25 09:40, Laurenz Albe wrote: > On Fri, 2025-10-31 at 08:21 -0700, Adrian Klaver wrote: >> Yeah, what I would like to know is how many of the data breaches >> actually grab directly from the storage versus getting it through the >> database or other software above the storage? It seems to me social >> engineering plays a bigger role in this. > > This is not about actual security considerations, it is about checkboxes. > Consequently, rational arguments are missing the point. Alright, been there. Years ago I used to drive a delivery truck for wholesale greenhouse and one of my chores was to go to a remote greenhouse we operated and pickup/deliver plants. There was a whole process for securing the key that you used to open the entry door. I pointed out that the greenhouse walls where two layers of plastic inflated by an air blower and then I proceeded to pull out my pocket knife as an example of a 'universal' key. The door key process stayed because it made people feel the greenhouse contents where safe. FYI, things did get stolen though that was because folks left them outside and the thieves did not have to bother with a knife. > > Yours, > Laurenz Albe -- Adrian Klaver adrian.klaver@aklaver.com
-
Re: Enquiry about TDE with PgSQL
Christophe Pettus <xof@thebuild.com> — 2025-10-31T17:04:35Z
> On Oct 31, 2025, at 08:21, Adrian Klaver <adrian.klaver@aklaver.com> wrote: > Yeah, what I would like to know is how many of the data breaches actually grab directly from the storage versus getting it through the database or other software above the storage? Essentially zero. PCI, like a lot of data security standards, are a magpie's assemblage of things that the authors have heard about all of which sound "secure" to them. However, since these particular magpies have machine guns (metaphorically) and can do serious damage to businesses, we must play along with the masquerade.
-
Re: Enquiry about TDE with PgSQL
Bruce Momjian <bruce@momjian.us> — 2025-10-31T17:06:21Z
On Fri, Oct 31, 2025 at 10:04:35AM -0700, Christophe Pettus wrote: > > > > On Oct 31, 2025, at 08:21, Adrian Klaver <adrian.klaver@aklaver.com> > > wrote: Yeah, what I would like to know is how many of the data > > breaches actually grab directly from the storage versus getting it > > through the database or other software above the storage? > > Essentially zero. > > PCI, like a lot of data security standards, are a magpie's assemblage > of things that the authors have heard about all of which sound > "secure" to them. However, since these particular magpies have > machine guns (metaphorically) and can do serious damage to businesses, > we must play along with the masquerade. Yes, we have been avoiding the masquerade for years. The question is can we continue. From the lack of discussion since April 1, 2025, it seems the answer is yes. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Do not let urgent matters crowd out time for investment in the future.
-
RE: Enquiry about TDE with PgSQL
Clay Jackson (cjackson) <clay.jackson@quest.com> — 2025-10-31T17:32:23Z
Pardo me for jumping in here - but would filesystem level encryption possibly meet your requirements? Clay Jackson Database Solutions Sales Engineer clay.jackson@quest.com office 949-754-1203 mobile 425-802-9603 -----Original Message----- From: Bruce Momjian <bruce@momjian.us> Sent: Friday, October 31, 2025 10:06 AM To: Christophe Pettus <xof@thebuild.com> Cc: Adrian Klaver <adrian.klaver@aklaver.com>; Kai Wagner <kai.wagner@percona.com>; Laurenz Albe <laurenz.albe@cybertec.at>; Ron Johnson <ronljohnsonjr@gmail.com>; pgsql-general <pgsql-general@postgresql.org> Subject: Re: Enquiry about TDE with PgSQL CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. On Fri, Oct 31, 2025 at 10:04:35AM -0700, Christophe Pettus wrote: > > > > On Oct 31, 2025, at 08:21, Adrian Klaver <adrian.klaver@aklaver.com> > > wrote: Yeah, what I would like to know is how many of the data > > breaches actually grab directly from the storage versus getting it > > through the database or other software above the storage? > > Essentially zero. > > PCI, like a lot of data security standards, are a magpie's assemblage > of things that the authors have heard about all of which sound > "secure" to them. However, since these particular magpies have > machine guns (metaphorically) and can do serious damage to businesses, > we must play along with the masquerade. Yes, we have been avoiding the masquerade for years. The question is can we continue. From the lack of discussion since April 1, 2025, it seems the answer is yes. -- Bruce Momjian <bruce@momjian.us> https://momjian.us/ EDB https://enterprisedb.com/ Do not let urgent matters crowd out time for investment in the future.
-
Re: Enquiry about TDE with PgSQL
Álvaro Herrera <alvherre@kurilemu.de> — 2025-10-31T17:33:54Z
On 2025-Oct-31, Bruce Momjian wrote: > Yes, we have been avoiding the masquerade for years. The question is > can we continue. From the lack of discussion since April 1, 2025, it > seems the answer is yes. Maybe, but I think the only reason for this is that some companies are implementing it locally in their forks or whatever. I bet there are many prospective customers that we (the open source Postgres project) are not reaching because of lack of certifiability in this area. Can we continue to ignore it? My impression is that that strategy will continue to work, perhaps indefinitely. Is it a good idea? Of that I am not so sure. -- Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/ "Once again, thank you and all of the developers for your hard work on PostgreSQL. This is by far the most pleasant management experience of any database I've worked on." (Dan Harris) http://archives.postgresql.org/pgsql-performance/2006-04/msg00247.php
-
Re: Enquiry about TDE with PgSQL
Christophe Pettus <xof@thebuild.com> — 2025-10-31T17:40:56Z
> On Oct 31, 2025, at 10:32, Clay Jackson (cjackson) <Clay.Jackson@quest.com> wrote: > > Pardo me for jumping in here - but would filesystem level encryption possibly meet your requirements? If we're talking about PCI DSS, the answer is: Yes, but. Filesystem-level encryption is acceptable IF the encryption keys (or other passwords used to unlock them) are separate from the user access controls to the host that has the encrypted volume attached. You have to go through a second step of decrypting the volume (or making it available for decrypted reads) separate from just mounting it.
-
Re: Enquiry about TDE with PgSQL
Bruce Momjian <bruce@momjian.us> — 2025-10-31T18:22:22Z
On Fri, Oct 31, 2025 at 06:33:54PM +0100, Álvaro Herrera wrote: > On 2025-Oct-31, Bruce Momjian wrote: > > > Yes, we have been avoiding the masquerade for years. The question is > > can we continue. From the lack of discussion since April 1, 2025, it > > seems the answer is yes. > > Maybe, but I think the only reason for this is that some companies are > implementing it locally in their forks or whatever. I bet there are > many prospective customers that we (the open source Postgres project) > are not reaching because of lack of certifiability in this area. > > Can we continue to ignore it? My impression is that that strategy will > continue to work, perhaps indefinitely. Is it a good idea? Of that I > am not so sure. Agreed. Just to state the obvious, I have never heard of any Postgres support company discouraging the community from implementing TDE. In fact, I have heard them strongly encourage it. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Do not let urgent matters crowd out time for investment in the future.
-
Re: Enquiry about TDE with PgSQL
Kai Wagner <kai.wagner@percona.com> — 2025-10-31T20:04:32Z
On Fri, Oct 31, 2025 at 7:22 PM Bruce Momjian <bruce@momjian.us> wrote: > On Fri, Oct 31, 2025 at 06:33:54PM +0100, Álvaro Herrera wrote: > > On 2025-Oct-31, Bruce Momjian wrote: > > > > > Yes, we have been avoiding the masquerade for years. The question is > > > can we continue. From the lack of discussion since April 1, 2025, it > > > seems the answer is yes. > I think this assumption can be considered a false positive. The main reason this hasn't surfaced yet is that it first takes some time to adjust, and more importantly, there are the downstream forks with the necessary changes that are already in use or continue to be sold. So why stop doing this? > > > > Maybe, but I think the only reason for this is that some companies are > > implementing it locally in their forks or whatever. I bet there are > > many prospective customers that we (the open source Postgres project) > > are not reaching because of lack of certifiability in this area. > > > > Can we continue to ignore it? My impression is that that strategy will > > continue to work, perhaps indefinitely. Is it a good idea? Of that I > > am not so sure. > > Agreed. Just to state the obvious, I have never heard of any Postgres > support company discouraging the community from implementing TDE. In > fact, I have heard them strongly encourage it. > I don't think, as stated initially, that we can continue to ignore this any longer. As a project, we are losing out on a significant number of users who are willing to use fully open-source solutions, but are held back due to this requirement. We had numerous conversations over the last few years, exactly about this fact, and people went with MySQL, Mongo, or others - not because of "does this technically make sense to us as engineers, but because they couldn't fulfill their internal requirements". As Laurenz already stated very well: "rational arguments are missing the point". It's not news that we also tried a way of implementing it. What I would like to achieve here is a group of interested people who can actually make a call on how this is envisioned to work. Do we handle everything in core directly, or do we make all necessary parts extensible? This approach may be more efficient in the long run, as it also enables a variety of other use cases. This is the conversation I would like to have. We're absolutely happy and willing to spend as much time as needed to implement a solution that works directly with PostgreSQL, so we no longer ignore huge parts of the industry, which will only get worse over the next few years, as more and more standards are to follow. Once we lose this user base, we all know how long it will take to regain any of them. I don't think we want, nor should we, to go down that route, as it would harm us as a project in the long run. We have been on a rise for many years, but if we want to stay there and continue to do so, some "checkbox" or "security requirements" need to be implemented, despite "technical arguments," as otherwise some of the largest companies worldwide cannot consider using postgres. > > -- > Bruce Momjian <bruce@momjian.us> > https://url.avanan.click/v2/r01/___https://momjian.us___.YXAzOnBlcmNvbmE6YTpnOmEyOTY2NTBjZWViOWUxZGMyMWI2ZDQwNGQ2NjZjNWQ1Ojc6MGE3MDo5YjgwNjIzYzk1NzI1OGRhZmNiYjJmYjFjNjI4NjFmOTI2NDM0YzM3Y2NhODZmNDE2YmEyY2UyMmM4ZDkxY2FmOnA6VDpO > EDB > https://url.avanan.click/v2/r01/___https://enterprisedb.com___.YXAzOnBlcmNvbmE6YTpnOmEyOTY2NTBjZWViOWUxZGMyMWI2ZDQwNGQ2NjZjNWQ1Ojc6NWUxZDo5Y2FiZWYwNzc1YzlhNTY0MGY2ZGM0MjNmMWNjMTY1ZmJlZGNlNmZlMmI4ZjE2ZGY2Y2RmYWEwZjM5NTUzYjY4OnA6VDpO > > Do not let urgent matters crowd out time for investment in the future. >
-
Re: Enquiry about TDE with PgSQL
Bruce Momjian <bruce@momjian.us> — 2025-10-31T20:53:57Z
On Fri, Oct 31, 2025 at 09:04:32PM +0100, Kai Wagner wrote: > On Fri, Oct 31, 2025 at 7:22 PM Bruce Momjian <bruce@momjian.us> wrote: > > On Fri, Oct 31, 2025 at 06:33:54PM +0100, Álvaro Herrera wrote: > > On 2025-Oct-31, Bruce Momjian wrote: > > > > > Yes, we have been avoiding the masquerade for years. The question is > > > can we continue. From the lack of discussion since April 1, 2025, it > > > seems the answer is yes. > > I think this assumption can be considered a false positive. The main reason > this hasn't surfaced yet is that it first takes some time to adjust, and more > importantly, there are the downstream forks with the necessary changes that are > already in use or continue to be sold. So why stop doing this? Keep in mind this is coming up seven months after the standard became effective, and it is being brought up by someone from Percona, and not from an end-user. I would have thought we would have had more end users complaining. > I don't think, as stated initially, that we can continue to ignore this any > longer. As a project, we are losing out on a significant number of users who > are willing to use fully open-source solutions, but are held back due to this > requirement. We had numerous conversations over the last few years, exactly > about this fact, and people went with MySQL, Mongo, or others - not because of > "does this technically make sense to us as engineers, but because they couldn't > fulfill their internal requirements". As Laurenz already stated very well: > "rational arguments are missing the point". > > It's not news that we also tried a way of implementing it. What I would like to > achieve here is a group of interested people who can actually make a call on > how this is envisioned to work. Do we handle everything in core directly, or do > we make all necessary parts extensible? This approach may be more efficient in We created a group several years ago, got pretty far, but ended up stopping for reasons I stated in my blog. I am not excited about doing this again unless there is a clear change of community opinion, which I have not seen. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Do not let urgent matters crowd out time for investment in the future.
-
Re: Enquiry about TDE with PgSQL
Ron Johnson <ronljohnsonjr@gmail.com> — 2025-10-31T21:19:57Z
On Fri, Oct 31, 2025 at 4:53 PM Bruce Momjian <bruce@momjian.us> wrote: > On Fri, Oct 31, 2025 at 09:04:32PM +0100, Kai Wagner wrote: > > On Fri, Oct 31, 2025 at 7:22 PM Bruce Momjian <bruce@momjian.us> wrote: > > > > On Fri, Oct 31, 2025 at 06:33:54PM +0100, Álvaro Herrera wrote: > > > On 2025-Oct-31, Bruce Momjian wrote: > > > > > > > Yes, we have been avoiding the masquerade for years. The > question is > > > > can we continue. From the lack of discussion since April 1, > 2025, it > > > > seems the answer is yes. > > > > I think this assumption can be considered a false positive. The main > reason > > this hasn't surfaced yet is that it first takes some time to adjust, and > more > > importantly, there are the downstream forks with the necessary changes > that are > > already in use or continue to be sold. So why stop doing this? > > Keep in mind this is coming up seven months after the standard became > effective, and it is being brought up by someone from Percona, and not > from an end-user. I would have thought we would have had more end users > complaining. > > > I don't think, as stated initially, that we can continue to ignore this > any > > longer. As a project, we are losing out on a significant number of users > who > > are willing to use fully open-source solutions, but are held back due to > this > > requirement. We had numerous conversations over the last few years, > exactly > > about this fact, and people went with MySQL, Mongo, or others - not > because of > > "does this technically make sense to us as engineers, but because they > couldn't > > fulfill their internal requirements". As Laurenz already stated very > well: > > "rational arguments are missing the point". > > > > It's not news that we also tried a way of implementing it. What I would > like to > > achieve here is a group of interested people who can actually make a > call on > > how this is envisioned to work. Do we handle everything in core > directly, or do > > we make all necessary parts extensible? This approach may be more > efficient in > > We created a group several years ago, got pretty far, but ended up > stopping for reasons I stated in my blog. I am not excited about doing > this again unless there is a clear change of community opinion, which I > have not seen. We're not complaining because we need TDE sooner than later, and PGDG's opinions on TDE are well known. EDB and Percona are there to take our money... -- Death to <Redacted>, and butter sauce. Don't boil me, I'm still alive. <Redacted> lobster!
-
Re: Enquiry about TDE with PgSQL
Bruce Momjian <bruce@momjian.us> — 2025-10-31T22:31:10Z
On Fri, Oct 31, 2025 at 05:19:57PM -0400, Ron Johnson wrote: > On Fri, Oct 31, 2025 at 4:53 PM Bruce Momjian <bruce@momjian.us> wrote: > On Fri, Oct 31, 2025 at 09:04:32PM +0100, Kai Wagner wrote: > We created a group several years ago, got pretty far, but ended up > stopping for reasons I stated in my blog. I am not excited about doing > this again unless there is a clear change of community opinion, which I > have not seen. > > We're not complaining because we need TDE sooner than later, and > PGDG's opinions on TDE are well known. > > EDB and Percona are there to take our money... I understand, but historically that is not the way things normally progress. I am thinking of Oracle hints. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Do not let urgent matters crowd out time for investment in the future.
-
Re: Enquiry about TDE with PgSQL
Markus Wanner <markus@bluegap.ch> — 2025-10-31T23:39:00Z
It's always entertaining to read PCI DSS... In the "guidance and purpose" column of page 95, the standard reads: Disk-level encryption typically encrypts the entire disk or partition using the same key, with all data automatically decrypted when the system runs or when an authorized user requests it. Granted, there would be benefits from a more fine grained encryption, e.g. where you'd encrypt PAN data with a different key. I guess that would be called column encryption. But I don't think any version of TDE implements it that way. The best I've seen is per tablespace. Further: Many disk-encryption solutions [..] perform the appropriate cryptographic transformations without any special action by the user other than supplying a password or passphrase at system start-up or at the beginning of a session. Well, isn't that exactly how TDE also works? It loads the key at start-up (of the database system, not the operating system, but that's equivalent for dedicated database servers) and decrypts "without any special action by the user". Otherwise, it would not be very transparent. This provides no protection from a malicious individual that has already managed to gain access to a valid user account I see a point here in possible separation of concerns. With a sysadmin being able to manage the system, but not having (direct) access to the data. That's more of an obscurity rather than hard security, though (as the key likely is somewhere in memory fully accessible for a malicious sysadmin). Still potentially beneficial in combination with all the auditing etc... So, my conclusions: * some potential process level security, no scientific security gains * most (if not all) existing TDE products on the market don't actually satisfy the stated purpose (or differ enough from disk-level encryption) * these look like unnecessary hurdles pushed into the standard by companies trying to shake off competitors who miss these (somewhat dubious) features * as an OSS project, it's still pointless to fight the standard and much more feasible to just implement that darn feature. Best Regards, Markus On 10/31/25 15:54, Bruce Momjian wrote: > On Fri, Oct 31, 2025 at 03:01:48PM +0100, Kai Wagner wrote: >> As I personally believe, there is no real way around TDE in the future, either >> by extensibility of the core (start with the storage manager and move your way >> on from there), to make an extension possible, or by directly adding it to the >> core, there are more reasons coming or are already on their way. >> >> With the PCI DSS v4.1 standard, one key rule to comply with is, that "If PAN is > > Uh, I think you mean the 4.0.1 standard, which became active on January > 1, 2025. I am surprised this is only being mentioned now: > > https://blog.pcisecuritystandards.org/just-published-pci-dss-v4-0-1 > > When will PCI DSS v4.0 be retired? > > As with all new versions of PCI DSS, there will be a period where both > the current and updated version will be active at the same time. PCI DSS > --> v4.0 will be retired on 31 December 2024. After that point, PCI DSS > --> v4.0.1 will be the only active version of the standard supported by PCI > SSC. > > While it was active on Jan 1, it became effective on March 31, 2025: > > Does PCI DSS v4.0.1 change the 31 March 2025 effective date for the new > requirements? > > No. This limited revision does not impact the effective date of these > new requirements. > >> stored, it must be rendered unreadable". Of course there are other ways, like >> tokenization, hashing etc. but this regulation is pushing towards at rest >> encryption in the long run, and not only disk encryption. We can dislike it, >> but we are already seeing the need coming from large industries and companies >> that they cannot work around this anymore, as the auditors doing the checkboxes >> do not really care about "good alternatives", as they do not even technically >> understand what this is about. They do compare postgres simply against other >> already in use databases at these orgs (MySQL or MongoDB), and as such, we are >> currently the only one that cannot be used in such a use case, at least not >> without the willingness of the auditor to make it happen. > > I see the new specification that disk-level encryption is insufficient, > starting on page 93 (page 97 in the PDF URL): > > https://www.middlebury.edu/sites/default/files/2025-01/PCI-DSS-v4_0_1.pdf#page=97 > > --> 3.5.1.2 If disk-level or partition-level encryption (rather than > file-, column-, or field-level database encryption) is used to > render PAN unreadable, it is implemented only as follows: > > • On removable electronic media > > OR > > • If used for non-removable electronic media, PAN is also > --> rendered unreadable via another mechanism that meets Requirement > 3.5.1. > > ... > > While disk or partition encryption may still be present on these > types of devices, it cannot be the only mechanism used to protect > PAN stored on those systems. Any stored PAN must also be rendered > unreadable per Requirement 3.5.1—for example, through truncation > or a data-level encryption mechanism. Full disk encryption helps > to protect data in the event of physical loss of a disk and > therefore its use is appropriate only for removable electronic > media storage devices. > > Purpose > > Disk-level and partition-level encryption typically encrypts > the entire disk or partition using the same key, with all data > automatically decrypted when the system runs or when an authorized > --> user requests it. For this reason, disk-level encryption is not > --> appropriate to protect stored PAN on computers, laptops, servers, > storage arrays, or any other system that provides transparent > decryption upon user authentication. > > PAN is: > > https://www.middlebury.edu/sites/default/files/2025-01/PCI-DSS-v4_0_1.pdf#page=391 > > PAN Acronym for “primary account number.” Unique payment card > number (credit, debit, or prepaid cards, etc.) that identifies > the issuer and the cardholder account. > > So it seems we have somewhat of a stand-off, with the Postgres project > questioning the value of TDE and the PCI writers doubling-down on > specifying disk-level encryption as insufficient. > > The biggest possible downside of this standoff is that enterprises that > need to meet PCI compliance specifications are forced to use specialized > versions of Postgres or Postgres extensions that support TDE. > > The fact that it has been seven months since PCI 4.0.1 was effective, > with little to no discussion or movement on adding TDE to community > Postgres means to me that we are unlikely to see TDE added to community > Postgres anytime soon. I have a small hope that adding compression to > the writing of temporary files will reduce the code changes needed to > encrypt temporary files, thus reducing the amount of TDE code changes > needed. > -
RE: Enquiry about TDE with PgSQL
Clay Jackson (cjackson) <clay.jackson@quest.com> — 2025-11-01T00:00:35Z
Speaking for myself, not Quest, as an "interested observer", I think Markus summarized it REALLY well. Unfortunately, like a lot of "overloaded" terms/standards, "TDE", whatever it means, has become a "checkbox" item. Clay Jackson Database Solutions Sales Engineer clay.jackson@quest.com office 949-754-1203 mobile 425-802-9603 -----Original Message----- From: Markus Wanner <markus@bluegap.ch> Sent: Friday, October 31, 2025 4:39 PM To: Bruce Momjian <bruce@momjian.us>; Kai Wagner <kai.wagner@percona.com> Cc: Laurenz Albe <laurenz.albe@cybertec.at>; Ron Johnson <ronljohnsonjr@gmail.com>; pgsql-general <pgsql-general@postgresql.org> Subject: Re: Enquiry about TDE with PgSQL CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. It's always entertaining to read PCI DSS... In the "guidance and purpose" column of page 95, the standard reads: Disk-level encryption typically encrypts the entire disk or partition using the same key, with all data automatically decrypted when the system runs or when an authorized user requests it. Granted, there would be benefits from a more fine grained encryption, e.g. where you'd encrypt PAN data with a different key. I guess that would be called column encryption. But I don't think any version of TDE implements it that way. The best I've seen is per tablespace. Further: Many disk-encryption solutions [..] perform the appropriate cryptographic transformations without any special action by the user other than supplying a password or passphrase at system start-up or at the beginning of a session. Well, isn't that exactly how TDE also works? It loads the key at start-up (of the database system, not the operating system, but that's equivalent for dedicated database servers) and decrypts "without any special action by the user". Otherwise, it would not be very transparent. This provides no protection from a malicious individual that has already managed to gain access to a valid user account I see a point here in possible separation of concerns. With a sysadmin being able to manage the system, but not having (direct) access to the data. That's more of an obscurity rather than hard security, though (as the key likely is somewhere in memory fully accessible for a malicious sysadmin). Still potentially beneficial in combination with all the auditing etc... So, my conclusions: * some potential process level security, no scientific security gains * most (if not all) existing TDE products on the market don't actually satisfy the stated purpose (or differ enough from disk-level encryption) * these look like unnecessary hurdles pushed into the standard by companies trying to shake off competitors who miss these (somewhat dubious) features * as an OSS project, it's still pointless to fight the standard and much more feasible to just implement that darn feature. Best Regards, Markus On 10/31/25 15:54, Bruce Momjian wrote: > On Fri, Oct 31, 2025 at 03:01:48PM +0100, Kai Wagner wrote: >> As I personally believe, there is no real way around TDE in the >> future, either by extensibility of the core (start with the storage >> manager and move your way on from there), to make an extension >> possible, or by directly adding it to the core, there are more reasons coming or are already on their way. >> >> With the PCI DSS v4.1 standard, one key rule to comply with is, that >> "If PAN is > > Uh, I think you mean the 4.0.1 standard, which became active on > January 1, 2025. I am surprised this is only being mentioned now: > > > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fblog > .pcisecuritystandards.org%2Fjust-published-pci-dss-v4-0-1&data=05%7C02 > %7Cclay.jackson%40quest.com%7C85d5352e5f094dbc847908de18d6b603%7C91c36 > 9b51c9e439c989c1867ec606603%7C0%7C0%7C638975507657957194%7CUnknown%7CT > WFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiI > sIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=i9KeBmcLhsLkzK%2F > 9gCmvx7%2FXLXdxbwg5zrfS99ISt8o%3D&reserved=0 > > When will PCI DSS v4.0 be retired? > > As with all new versions of PCI DSS, there will be a period where both > the current and updated version will be active at the same time. > PCI DSS > --> v4.0 will be retired on 31 December 2024. After that point, PCI DSS > --> v4.0.1 will be the only active version of the standard supported > --> by PCI > SSC. > > While it was active on Jan 1, it became effective on March 31, 2025: > > Does PCI DSS v4.0.1 change the 31 March 2025 effective date for the new > requirements? > > No. This limited revision does not impact the effective date of these > new requirements. > >> stored, it must be rendered unreadable". Of course there are other >> ways, like tokenization, hashing etc. but this regulation is pushing >> towards at rest encryption in the long run, and not only disk >> encryption. We can dislike it, but we are already seeing the need >> coming from large industries and companies that they cannot work >> around this anymore, as the auditors doing the checkboxes do not >> really care about "good alternatives", as they do not even >> technically understand what this is about. They do compare postgres >> simply against other already in use databases at these orgs (MySQL or >> MongoDB), and as such, we are currently the only one that cannot be used in such a use case, at least not without the willingness of the auditor to make it happen. > > I see the new specification that disk-level encryption is > insufficient, starting on page 93 (page 97 in the PDF URL): > > > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww. > middlebury.edu%2Fsites%2Fdefault%2Ffiles%2F2025-01%2FPCI-DSS-v4_0_1.pd > f%23page%3D97&data=05%7C02%7Cclay.jackson%40quest.com%7C85d5352e5f094d > bc847908de18d6b603%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C638975 > 507657992323%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwL > jAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C% > 7C&sdata=mb5D5ywMtVHX%2BSGw5on4KK6aA7SxDq76QLCK%2Bi5K9Pc%3D&reserved=0 > > --> 3.5.1.2 If disk-level or partition-level encryption (rather than > file-, column-, or field-level database encryption) is used to > render PAN unreadable, it is implemented only as follows: > > . On removable electronic media > > OR > > . If used for non-removable electronic media, PAN is also > --> rendered unreadable via another mechanism that meets Requirement > 3.5.1. > > ... > > While disk or partition encryption may still be present on these > types of devices, it cannot be the only mechanism used to protect > PAN stored on those systems. Any stored PAN must also be rendered > unreadable per Requirement 3.5.1-for example, through truncation > or a data-level encryption mechanism. Full disk encryption helps > to protect data in the event of physical loss of a disk and > therefore its use is appropriate only for removable electronic > media storage devices. > > Purpose > > Disk-level and partition-level encryption typically encrypts > the entire disk or partition using the same key, with all data > automatically decrypted when the system runs or when an > authorized > --> user requests it. For this reason, disk-level encryption is not > --> appropriate to protect stored PAN on computers, laptops, > --> servers, > storage arrays, or any other system that provides transparent > decryption upon user authentication. > > PAN is: > > > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww. > middlebury.edu%2Fsites%2Fdefault%2Ffiles%2F2025-01%2FPCI-DSS-v4_0_1.pd > f%23page%3D391&data=05%7C02%7Cclay.jackson%40quest.com%7C85d5352e5f094 > dbc847908de18d6b603%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C63897 > 5507658014907%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIw > LjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C > %7C&sdata=v8S4Nnt%2Bh0a%2BjVfuoNk3BYjbM5vMtsdpcCXSFVX38mk%3D&reserved= > 0 > > PAN Acronym for "primary account number." Unique payment card > number (credit, debit, or prepaid cards, etc.) that identifies > the issuer and the cardholder account. > > So it seems we have somewhat of a stand-off, with the Postgres project > questioning the value of TDE and the PCI writers doubling-down on > specifying disk-level encryption as insufficient. > > The biggest possible downside of this standoff is that enterprises > that need to meet PCI compliance specifications are forced to use > specialized versions of Postgres or Postgres extensions that support TDE. > > The fact that it has been seven months since PCI 4.0.1 was effective, > with little to no discussion or movement on adding TDE to community > Postgres means to me that we are unlikely to see TDE added to > community Postgres anytime soon. I have a small hope that adding > compression to the writing of temporary files will reduce the code > changes needed to encrypt temporary files, thus reducing the amount of > TDE code changes needed. > -
Re: Enquiry about TDE with PgSQL
Christophe Pettus <xof@thebuild.com> — 2025-11-01T00:16:09Z
On Oct 31, 2025, at 07:54, Bruce Momjian <bruce@momjian.us> wrote: > So it seems we have somewhat of a stand-off, with the Postgres project > questioning the value of TDE and the PCI writers doubling-down on > specifying disk-level encryption as insufficient. PCI definitely exhibits a preference away from disk-level encryption, although it doesn't prohibit it: you have to make sure that simply mounting the disk doesn't decrypt it. Their concern is that if user credentials are compromised, and an attacker then has to do something else in order to see the plaintext. This kind of implies TDE, although they don't use that term. Now, the road forks here: 1. If a customer wants TDE and isn't interested in hearing about other solutions, then TDE is only thing that will meet that goal. 2. The PCI spec doesn't specifically offer up TDE as an alternative to disk-level encryption, though. It exhibits a strong preference for column-level encryption of sensitive data, which doesn't require TDE. In some ways, there's no real point of discussion. You can comply with PCI without TDE (I would argue that, in fact, you are in a better position with column-level encryption), but if the organization wants TDE, then the technical arguments rarely matter.
-
Re: Enquiry about TDE with PgSQL
Bruce Momjian <bruce@momjian.us> — 2025-11-01T00:21:04Z
On Fri, Oct 31, 2025 at 05:16:09PM -0700, Christophe Pettus wrote: > On Oct 31, 2025, at 07:54, Bruce Momjian <bruce@momjian.us> wrote: > > So it seems we have somewhat of a stand-off, with the Postgres > > project questioning the value of TDE and the PCI writers > > doubling-down on specifying disk-level encryption as insufficient. > > PCI definitely exhibits a preference away from disk-level encryption, > although it doesn't prohibit it: you have to make sure that simply > mounting the disk doesn't decrypt it. Their concern is that if > user credentials are compromised, and an attacker then has to do > something else in order to see the plaintext. This kind of implies > TDE, although they don't use that term. > > Now, the road forks here: > > 1. If a customer wants TDE and isn't interested in hearing about other > solutions, then TDE is only thing that will meet that goal. > > 2. The PCI spec doesn't specifically offer up TDE as an alternative to > disk-level encryption, though. It exhibits a strong preference for > column-level encryption of sensitive data, which doesn't require TDE. > > In some ways, there's no real point of discussion. You can comply > with PCI without TDE (I would argue that, in fact, you are in a better > position with column-level encryption), but if the organization wants > TDE, then the technical arguments rarely matter. I think column-level encryption, on the client side, actually does improve security and is preferable to file system level TDE, and I think many here feel the same way. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Do not let urgent matters crowd out time for investment in the future.
-
RE: Enquiry about TDE with PgSQL
Clay Jackson (cjackson) <clay.jackson@quest.com> — 2025-11-01T00:24:25Z
I can't disagree - but the question them becomes, as Markus and other have pointed out; would that allow a customer/user to check the "Encryption" box for PCI or any other "compliance review" Clay Jackson Database Solutions Sales Engineer clay.jackson@quest.com office 949-754-1203 mobile 425-802-9603 -----Original Message----- From: Bruce Momjian <bruce@momjian.us> Sent: Friday, October 31, 2025 5:21 PM To: Christophe Pettus <xof@thebuild.com> Cc: pgsql-general <pgsql-general@postgresql.org>; Kai Wagner <kai.wagner@percona.com>; Laurenz Albe <laurenz.albe@cybertec.at>; Ron Johnson <ronljohnsonjr@gmail.com> Subject: Re: Enquiry about TDE with PgSQL CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. On Fri, Oct 31, 2025 at 05:16:09PM -0700, Christophe Pettus wrote: > On Oct 31, 2025, at 07:54, Bruce Momjian <bruce@momjian.us> wrote: > > So it seems we have somewhat of a stand-off, with the Postgres > > project questioning the value of TDE and the PCI writers > > doubling-down on specifying disk-level encryption as insufficient. > > PCI definitely exhibits a preference away from disk-level encryption, > although it doesn't prohibit it: you have to make sure that simply > mounting the disk doesn't decrypt it. Their concern is that if user > credentials are compromised, and an attacker then has to do something > else in order to see the plaintext. This kind of implies TDE, > although they don't use that term. > > Now, the road forks here: > > 1. If a customer wants TDE and isn't interested in hearing about other > solutions, then TDE is only thing that will meet that goal. > > 2. The PCI spec doesn't specifically offer up TDE as an alternative to > disk-level encryption, though. It exhibits a strong preference for > column-level encryption of sensitive data, which doesn't require TDE. > > In some ways, there's no real point of discussion. You can comply > with PCI without TDE (I would argue that, in fact, you are in a better > position with column-level encryption), but if the organization wants > TDE, then the technical arguments rarely matter. I think column-level encryption, on the client side, actually does improve security and is preferable to file system level TDE, and I think many here feel the same way. -- Bruce Momjian <bruce@momjian.us> https://momjian.us/ EDB https://enterprisedb.com/ Do not let urgent matters crowd out time for investment in the future.
-
Re: Enquiry about TDE with PgSQL
Bruce Momjian <bruce@momjian.us> — 2025-11-01T00:33:49Z
On Sat, Nov 1, 2025 at 12:24:25AM +0000, Clay Jackson (cjackson) wrote: > I can't disagree - but the question them becomes, as Markus and > other have pointed out; would that allow a customer/user to check the > "Encryption" box for PCI or any other "compliance review" I think so. It says storage encryption is insufficient, but it doesn't say client-side column-level encryption is insufficient. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Do not let urgent matters crowd out time for investment in the future.
-
Re: Enquiry about TDE with PgSQL
Christophe Pettus <xof@thebuild.com> — 2025-11-01T01:32:17Z
> On Oct 31, 2025, at 17:21, Bruce Momjian <bruce@momjian.us> wrote: > > I think column-level encryption, on the client side, actually does > improve security and is preferable to file system level TDE, and I think > many here feel the same way. Absolutely. Unfortunately, too many IT security policies are basically a grab-bag of things that someone has read that all claimed to be "best practice," and the degree to which they can be educated on the topic is variable.
-
Re: Enquiry about TDE with PgSQL
Christophe Pettus <xof@thebuild.com> — 2025-11-01T01:35:52Z
On Oct 31, 2025, at 17:24, Clay Jackson (cjackson) <Clay.Jackson@quest.com> wrote: > > I can't disagree - but the question them becomes, as Markus and other have pointed out; would that allow a customer/user to check the "Encryption" box for PCI or any other "compliance review" The answer is: it depends (doesn't it always?). Doing secure column-level encryption meets the PCI standard, and a competent PCI auditor will know that. However, TDE has this cache as being "the way one does it," and if the organization is that way, it's hard to move them off of it. As a sign of how the PCI world views TDE, at least one of the major credit card associations does not use it, and they have literally everyone's credit card number, with expiration date and CVV, sitting on their disks.
-
Re: Enquiry about TDE with PgSQL
Chris Travers <chris.travers@gmail.com> — 2025-11-01T04:18:43Z
I maintain that the way forward is to get TDE in core. Perhaps someone could pick up the previous patches and try to push them again Best Wishes, Chris Travers On Sat, Nov 1, 2025, 8:36 AM Christophe Pettus <xof@thebuild.com> wrote: > On Oct 31, 2025, at 17:24, Clay Jackson (cjackson) <Clay.Jackson@quest.com> > wrote: > > > > I can't disagree - but the question them becomes, as Markus and other > have pointed out; would that allow a customer/user to check the > "Encryption" box for PCI or any other "compliance review" > > The answer is: it depends (doesn't it always?). Doing secure column-level > encryption meets the PCI standard, and a competent PCI auditor will know > that. However, TDE has this cache as being "the way one does it," and if > the organization is that way, it's hard to move them off of it. > > As a sign of how the PCI world views TDE, at least one of the major credit > card associations does not use it, and they have literally everyone's > credit card number, with expiration date and CVV, sitting on their disks. > >
-
Re: Enquiry about TDE with PgSQL
Kai Wagner <kai.wagner@percona.com> — 2025-11-01T07:34:57Z
On Sat, Nov 1, 2025 at 5:19 AM Chris Travers <chris.travers@gmail.com> wrote: > I maintain that the way forward is to get TDE in core. Perhaps someone > could pick up the previous patches and try to push them again > I wholeheartedly agree, as in this thread we are trying to do the same thing again, that has already happened all they years before. We lose ourselves in technical reasons, wondering why this makes no sense and how it could be achieved differently, but we forget that we live in a vacuum and bubble here. The auditor, most of the time (as I've seen many times), has no knowledge of these technical aspects. It's a box to check, with a simple 'yes' or 'no'. They don't even wanna hear any "but this also satisfies it, as this isn't clearly stated and worded in the standard". This doesn't get us anywhere anymore; they will not put their checkbox there if there is no simple answer to it. @Bruce Momjian <bruce@momjian.us> I totally understand your frustration from previous times and also your point of view, that's absolutely valid, no doubt about that. The time has changed over the course of the last 5+ years, and maybe it is time to reconsider. Just because it didn't succeed last time doesn't mean we have to end up in the same spot this time. We discussed it at length, and I am committed to supporting and making happen what's necessary to get TDE fully functional with postgres directly. The way of the implementation is a different question. Who from the former times, or maybe even now, being interested in the topic, would be open for a TDE group, to technically discuss options, possibilities etc. that we can POC on and share for further feedback?! > > Best Wishes, > Chris Travers > > > On Sat, Nov 1, 2025, 8:36 AM Christophe Pettus <xof@thebuild.com> wrote: > >> On Oct 31, 2025, at 17:24, Clay Jackson (cjackson) < >> Clay.Jackson@quest.com> wrote: >> > >> > I can't disagree - but the question them becomes, as Markus and other >> have pointed out; would that allow a customer/user to check the >> "Encryption" box for PCI or any other "compliance review" >> >> The answer is: it depends (doesn't it always?). Doing secure >> column-level encryption meets the PCI standard, and a competent PCI auditor >> will know that. However, TDE has this cache as being "the way one does >> it," and if the organization is that way, it's hard to move them off of it. >> >> As a sign of how the PCI world views TDE, at least one of the major >> credit card associations does not use it, and they have literally >> everyone's credit card number, with expiration date and CVV, sitting on >> their disks. >> >>
-
Re: Enquiry about TDE with PgSQL
Bruce Momjian <bruce@momjian.us> — 2025-11-01T13:58:24Z
On Sat, Nov 1, 2025 at 08:34:57AM +0100, Kai Wagner wrote: > I wholeheartedly agree, as in this thread we are trying to do the same thing > again, that has already happened all they years before. We lose ourselves in > technical reasons, wondering why this makes no sense and how it could be > achieved differently, but we forget that we live in a vacuum and bubble here. > The auditor, most of the time (as I've seen many times), has no knowledge of > these technical aspects. It's a box to check, with a simple 'yes' or 'no'. They > don't even wanna hear any "but this also satisfies it, as this isn't clearly > stated and worded in the standard". This doesn't get us anywhere anymore; they > will not put their checkbox there if there is no simple answer to it. > > @Bruce Momjian I totally understand your frustration from previous times and > also your point of view, that's absolutely valid, no doubt about that. The time > has changed over the course of the last 5+ years, and maybe it is time to > reconsider. Just because it didn't succeed last time doesn't mean we have to > end up in the same spot this time. We discussed it at length, and I am > committed to supporting and making happen what's necessary to get TDE fully > functional with postgres directly. The way of the implementation is a different > question. Who from the former times, or maybe even now, being interested in the > topic, would be open for a TDE group, to technically discuss options, > possibilities etc. that we can POC on and share for further feedback?! Without the calculus of feature value vs code overhead changing, talking isn't going to accomplish anything. My patch was already very small for shared buffers, and the WAL code would have been small, but the code weight of encrypting temporary files wasn't justified by the community, and I couldn't argue against that conclusion. Crunchy had someone working on restructuring temporary file writes to make it easier, but they were unsuccessful. Maybe the posted temporary file compression patch will help reduce the code weight. If you want to move forward with TDE without waiting to see if the temporary file compression patch will reduce the TDE code impact, you need to dig into how the community does feature calculus and how that calculus can be changed --- this is not something technology can fix. Companies are willing to add the code weight because it is a sale for them, and customers are willing to pay to meet the check box --- that calculus just doesn't work in the community. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Do not let urgent matters crowd out time for investment in the future.
-
Re: Enquiry about TDE with PgSQL
Adrian Klaver <adrian.klaver@aklaver.com> — 2025-11-01T15:32:17Z
On 11/1/25 06:58, Bruce Momjian wrote: > On Sat, Nov 1, 2025 at 08:34:57AM +0100, Kai Wagner wrote: > If you want to move forward with TDE without waiting to see if the > temporary file compression patch will reduce the TDE code impact, you > need to dig into how the community does feature calculus and how that > calculus can be changed --- this is not something technology can fix. > > Companies are willing to add the code weight because it is a sale for > them, and customers are willing to pay to meet the check box --- that > calculus just doesn't work in the community. > That is not true: https://docs.percona.com/pg-tde/index.html and https://www.cybertec-postgresql.com/en/products/postgresql-transparent-data-encryption/ The community does provide it, just not the version of Postgres released here: https://www.postgresql.org/download/ -- Adrian Klaver adrian.klaver@aklaver.com
-
Re: Enquiry about TDE with PgSQL
Greg Sabino Mullane <htamfids@gmail.com> — 2025-11-01T18:34:36Z
On Sat, Nov 1, 2025 at 11:32 AM Adrian Klaver <adrian.klaver@aklaver.com> wrote: > The community does provide it, just not the version of Postgres released > here: > I'm not sure of your point here. Companies have forked versions / patches, yes, but not sure how that equates to "community-provided" Cheers, Greg -- Crunchy Data - https://www.crunchydata.com Enterprise Postgres Software Products & Tech Support
-
Re: Enquiry about TDE with PgSQL
Ken Marshall <ktm@rice.edu> — 2025-11-01T18:54:56Z
+1 from me for having TDE in-core or available as an extension The security auditors that I have worked with have been increasingly unwilling to actual evaluate the merits of an implementation or perhaps no longer have the knowledge or skills needed. This is a needed checkbox to allow PostgreSQL to be deployed in those environments. Regards, Ken
-
Re: Enquiry about TDE with PgSQL
rainer@ultra-secure.de — 2025-11-01T20:07:01Z
> Am 01.11.2025 um 19:54 schrieb Ken Marshall <ktm@rice.edu>: > > +1 from me for having TDE in-core or available as an extension > > The security auditors that I have worked with have been increasingly > unwilling to actual evaluate the merits of an implementation or perhaps > no longer have the knowledge or skills needed. This is a needed > checkbox to allow PostgreSQL to be deployed in those environments. > > Do you actually have HSMs with your TDE (assuming you use it elsewhere? We run, for a customer, an Oracle DataGuard configuration with TDE with a HSM. We have a support-contract with a 3rd party company that helps us with the more obscure problems on Oracle that we don’t encounter every day and they told us of all their clients (banks, insurance companies), we are the only ones with TDE. They loath working with it ;-) There’s apparently another non-disclosed customer that uses it. It may be that a lot of people now use „cloud HSMs“ - but I’m a bit of a purist for these kinds of things in that I believe that unless you own the hardware (HSMs are usually tamper-proof enough so you can deploy them in 3rd-party datacenters that aren’t your own), you don’t really control the keys. In our case, the databases are backed up with rman to an NFS share that is provided by a virtualized linux server - the severs itself are hardware. If you don’t have TDE, your backups aren’t encrypted and they end up on the veeam server like everything else, where an admin could copy them somewhere else and potentially take them elsewhere. With the HSM, we don’t actually know the secret to decrypt the data (there may be a way to get it, I don’t know). We know the secret to unseal the wallet (that sits on the HSM, I believe) so that the database actually mounts and starts. It’s pretty bullet-proof (I believe there’s techniques to prevent sniffing out the secret from RAM and HSMs usually implement those in their client software). In fact, it’s so bullet-proof that should you lose the keys on the HSM, your data is gone if you have no other backups or backups of the HSM. If the amount of data is small enough, you can GPG encrypt a „normal“ full dump - but that become unfeasible as database size grows.
-
Re: Enquiry about TDE with PgSQL
Adrian Klaver <adrian.klaver@aklaver.com> — 2025-11-01T21:15:43Z
On 11/1/25 11:34, Greg Sabino Mullane wrote: > On Sat, Nov 1, 2025 at 11:32 AM Adrian Klaver <adrian.klaver@aklaver.com > <mailto:adrian.klaver@aklaver.com>> wrote: > > The community does provide it, just not the version of Postgres > released here: > > > I'm not sure of your point here. Companies have forked versions / > patches, yes, but not sure how that equates to "community-provided" Well this gets in to a pet peeve of my with regard to OSS and the definition of 'community', it seems to adjust in size of coverage depending on the argument. For a recent thread on -www that gets into this, as concerns contributors, see: https://www.postgresql.org/message-id/aJ9ur1eSnk5pp9aD%40msg.df7cb.de To get to said contributors list you go: https://www.postgresql.org/community/contributors/ where you see folks recognized for their work outside the core code. Examples of the expanded 'commmunity': 1) From the 'community' site you can go to: https://www.postgresql.org/download/ where that packaging is done outside of core. 2) Again from postgresql.org: https://www.postgresql.org/docs/18/external-extensions.html "PostgreSQL is designed to be easily extensible. For this reason, extensions loaded into the database can function just like features that are built in. The contrib/ directory shipped with the source code contains several extensions, which are described in Appendix F. Other extensions are developed independently, like PostGIS. Even PostgreSQL replication solutions can be developed externally. For example, Slony-I is a popular primary/standby replication solution that is developed independently from the core project." 3) Of course there is: https://www.postgresql.org/community/ "PostgreSQL is well-supported by its active community." To me that means extensions provided by members of the 'community' are 'community-provided'? > > Cheers, > Greg > > -- > Crunchy Data - https://www.crunchydata.com <https://www.crunchydata.com> > Enterprise Postgres Software Products & Tech Support > -- Adrian Klaver adrian.klaver@aklaver.com
-
Re: Enquiry about TDE with PgSQL
Bruce Momjian <bruce@momjian.us> — 2025-11-01T23:42:39Z
On Sat, Nov 1, 2025 at 02:15:43PM -0700, Adrian Klaver wrote: > 3) Of course there is: > > https://www.postgresql.org/community/ > > "PostgreSQL is well-supported by its active community." > > > To me that means extensions provided by members of the 'community' are > 'community-provided'? Yes, this point was so prone to confusion I decided not to reply to the original email. ;-) -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Do not let urgent matters crowd out time for investment in the future.
-
Re: Enquiry about TDE with PgSQL
Kai Wagner <kai.wagner@percona.com> — 2025-11-02T10:14:58Z
On Sat, Nov 1, 2025 at 2:58 PM Bruce Momjian <bruce@momjian.us> wrote: > If you want to move forward with TDE without waiting to see if the > temporary file compression patch will reduce the TDE code impact, you > need to dig into how the community does feature calculus and how that > calculus can be changed --- this is not something technology can fix. > I fully agree here, and as I stated above, I don't question this at all, as this is the strength of the diverse and spread community. Looking at this thread alone, and the multiple different "users" popping up, that see the need and would like to see TDE in core or in an extension drives my thinking as it makes sense to start thinking and looking into the temp file compression already right now, if and how this could make our code changes easier, or if additional extensibility should be directly part of the first patch, so you can enable it through its extension. Either way, it should be considered and kept in mind now, not after the fact, or we will continue to spin this wheel. And this thinking and willingness alone, to actually be open to that idea, requires some upfront discussion, so you at least know you are welcome to work on it without wasting effort, because no one will ever want to merge. (and yes, of course everyone can work on whatever they want at any point in time, but people might prefer working on something that actually benefits the project and is welcome, rather than ending in >/dev/null) Given your experience Bruce, can you offer us some advice on how you would approach it currently? What do you think makes the most sense, and how should we proceed with collaboration to at least see a small change in making this happen in the future? > > Companies are willing to add the code weight because it is a sale for > them, and customers are willing to pay to meet the check box --- that > calculus just doesn't work in the community. > And this should always be at the forefront, as this makes the project exactly as strong as it is today. > > -- > Bruce Momjian <bruce@momjian.us> > https://url.avanan.click/v2/r01/___https://momjian.us___.YXAzOnBlcmNvbmE6YTpnOmZhM2VjN2JlNjcyOTZmMGQwMjEzOGM1NWU5OTA5NzM0Ojc6ZDgxOTo3NzQ0MzQwZjhmODA4YmY1ZjFiZmFlMzdlNmE4MzUxYzc0NGU0NTIwMjA5YWU1Mjk2NjQ5YjFiZjk0NDI3MDNjOnA6VDpO > EDB > https://url.avanan.click/v2/r01/___https://enterprisedb.com___.YXAzOnBlcmNvbmE6YTpnOmZhM2VjN2JlNjcyOTZmMGQwMjEzOGM1NWU5OTA5NzM0Ojc6MjZhZTplMDhhZWQ5Nzg3YmVjODI4MDQ5YmU2ODc1MGExMTVmMmRmOTVlOTk2N2FmM2Y1N2VkN2VmYTZhYWMyZDQ1MzE3OnA6VDpO > > Do not let urgent matters crowd out time for investment in the future. >
-
Re: Enquiry about TDE with PgSQL
Bruce Momjian <bruce@momjian.us> — 2025-11-03T15:08:15Z
On Sat, Nov 1, 2025 at 09:07:01PM +0100, Rainer Duffner wrote: > Do you actually have HSMs with your TDE (assuming you use it > elsewhere? We run, for a customer, an Oracle DataGuard configuration > with TDE with a HSM. There were some interesting ideas in this email I want to reply to. > We have a support-contract with a 3rd party company that helps us > with the more obscure problems on Oracle that we don’t encounter > every day and they told us of all their clients (banks, insurance > companies), we are the only ones with TDE. They loath working with it > ;-) Is it the Oracle API they don't like, that Postgres can improve upon, or something fundamental they don't like, or don't see the value in? > There’s apparently another non-disclosed customer that uses it. > > It may be that a lot of people now use „cloud HSMs“ - but I’m > a bit of a purist for these kinds of things in that I believe that > unless you own the hardware (HSMs are usually tamper-proof enough so > you can deploy them in 3rd-party datacenters that aren’t your own), > you don’t really control the keys. > > In our case, the databases are backed up with rman to an NFS share > that is provided by a virtualized linux server - the severs itself are > hardware. > > If you don’t have TDE, your backups aren’t encrypted and they end > up on the veeam server like everything else, where an admin could copy > them somewhere else and potentially take them elsewhere. > > With the HSM, we don’t actually know the secret to decrypt the data > (there may be a way to get it, I don’t know). We know the secret > to unseal the wallet (that sits on the HSM, I believe) so that the > database actually mounts and starts. > > It’s pretty bullet-proof (I believe there’s techniques to prevent > sniffing out the secret from RAM and HSMs usually implement those in > their client software). In fact, it’s so bullet-proof that should > you lose the keys on the HSM, your data is gone if you have no other > backups or backups of the HSM. As far as I know, there are two ways to generate the data encryption key. One is for the HSM to generate it, and then only the HSM knows it. The other method is to create the encryption key on a USB memory stick, copy the key into the HSM, and then remove the USB memory stick and store it in a secure location like a safe. The second method seems like a better option to me. Oh, and make a second copy of the USB memory stick. > If the amount of data is small enough, you can GPG encrypt a > „normal“ full dump - but that become unfeasible as database size > grows. This is a good point. For TDE storage, the data is encrypted once on write, and then can be backed up as many times as needed without re-encryption. With storage-level encryption, the data has to be encrypted for every backup. However, considering how much TLS is used, I assumed the encryption overhead would be minimal compared to the transfer overhead. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Do not let urgent matters crowd out time for investment in the future.
-
Re: Enquiry about TDE with PgSQL
rainer@ultra-secure.de — 2025-11-03T15:39:45Z
Am 2025-11-03 16:08, schrieb Bruce Momjian: > Is it the Oracle API they don't like, that Postgres can improve upon, > or > something fundamental they don't like, or don't see the value in? I am not sure. It just complicates everything. Documentation isn't thin, it's skeletal. And of course, actual support from the HSM-vendor for this use-case is non-existent. Same for Oracle. They'll both point at each other. Who'd have thought that. > As far as I know, there are two ways to generate the data encryption > key. One is for the HSM to generate it, and then only the HSM knows > it. > The other method is to create the encryption key on a USB memory stick, > copy the key into the HSM, and then remove the USB memory stick and > store it in a secure location like a safe. The second method seems > like > a better option to me. Oh, and make a second copy of the USB memory > stick. The keys are generated on the HSM. There's HSM client you've got to install that manages the communication to the HSM. The HSM should be backed up, too. Which is only possible by connecting physically to it with a notebook and inserting an USB stick. Which begs the question: where do you source an USB stick with the same trust-level as the 20k-a-pop HSM?
-
Re: Enquiry about TDE with PgSQL
Bruce Momjian <bruce@momjian.us> — 2025-11-03T16:01:06Z
On Mon, Nov 3, 2025 at 04:39:45PM +0100, rainer@ultra-secure.de wrote: > Am 2025-11-03 16:08, schrieb Bruce Momjian: > > > Is it the Oracle API they don't like, that Postgres can improve upon, or > > something fundamental they don't like, or don't see the value in? > > > I am not sure. > > It just complicates everything. > Documentation isn't thin, it's skeletal. Okay, these are things we can improve on. It think the API of my final posted patch had a pretty simple API, but pushing that API out to external tools will add complexity I didn't implement, and that complexity could be a reason to reject TDE. > And of course, actual support from the HSM-vendor for this use-case is > non-existent. > Same for Oracle. Yes, my patch used shell scripts --- not sure if that is good or bad. I will admit that companies are better at integrating with external vendors, particulary hardware vendors. There is an organization mismatch betwween the community and companies, and the community basically forces companies to intract on community terms --- companies are a more natural iteraction for other companies. > > As far as I know, there are two ways to generate the data encryption > > key. One is for the HSM to generate it, and then only the HSM knows it. > > The other method is to create the encryption key on a USB memory stick, > > copy the key into the HSM, and then remove the USB memory stick and > > store it in a secure location like a safe. The second method seems like > > a better option to me. Oh, and make a second copy of the USB memory > > stick. > > > The keys are generated on the HSM. > There's HSM client you've got to install that manages the communication to > the HSM. > > The HSM should be backed up, too. Which is only possible by connecting > physically to it with a notebook and inserting an USB stick. The problem is that if anything happens with the HSM, you are stuck. The HSM adds an additional risk. > Which begs the question: where do you source an USB stick with the same > trust-level as the 20k-a-pop HSM? I don't know. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Do not let urgent matters crowd out time for investment in the future.
-
Re: Enquiry about TDE with PgSQL
Adrian Klaver <adrian.klaver@aklaver.com> — 2025-11-03T16:20:21Z
On 11/3/25 08:01, Bruce Momjian wrote: > On Mon, Nov 3, 2025 at 04:39:45PM +0100, rainer@ultra-secure.de wrote: >> Am 2025-11-03 16:08, schrieb Bruce Momjian: > I will admit that companies are better at integrating with external > vendors, particulary hardware vendors. There is an organization > mismatch betwween the community and companies, and the community > basically forces companies to intract on community terms --- companies > are a more natural iteraction for other companies. Again a distinction without a difference. If you go here: https://www.postgresql.org/community/contributors/ and look at the names under Core Team and Major Contributors, out of 59 names only 6 do not list a company name. Companies live at the heart of the community. Hence the the policy of no company having no more then 50%(?) of the Core team. -- Adrian Klaver adrian.klaver@aklaver.com
-
Re: Enquiry about TDE with PgSQL
Bruce Momjian <bruce@momjian.us> — 2025-11-03T16:56:25Z
On Sun, Nov 2, 2025 at 11:14:58AM +0100, Kai Wagner wrote: > I fully agree here, and as I stated above, I don't question this at all, as > this is the strength of the diverse and spread community. Looking at this > thread alone, and the multiple different "users" popping up, that see the need > and would like to see TDE in core or in an extension drives my thinking as it > makes sense to start thinking and looking into the temp file compression > already right now, if and how this could make our code changes easier, or if > additional extensibility should be directly part of the first patch, so you can > enable it through its extension. Either way, it should be considered and kept > in mind now, not after the fact, or we will continue to spin this wheel. And > this thinking and willingness alone, to actually be open to that idea, requires > some upfront discussion, so you at least know you are welcome to work on it > without wasting effort, because no one will ever want to merge. (and yes, of > course everyone can work on whatever they want at any point in time, but people > might prefer working on something that actually benefits the project and is > welcome, rather than ending in >/dev/null) > > Given your experience Bruce, can you offer us some advice on how you would > approach it currently? What do you think makes the most sense, and how should > we proceed with collaboration to at least see a small change in making this > happen in the future? Good questions. I think we have to decide if the community wants TDE, and if so how much code change will the community accept to get it. And then we have to decide if this happens in an extension, like Percona's, or in the core code. The problem with the Percona extension is it seems like it was developed mostly/all by Percona employees, meaning development was driven/steered by Percona, and there was insufficient feedback from the community for it to be polished enough to be a general community solution. (I fortunately attended all the Percona TDE talks at the Riga conference.) The community needs to get involved with that extension if it is to become a community-polished solution. An extension does allow for fewer core code changes, but as I stated in a previous email, the changes needed for command-line tools, external tools, and replication to work in a TDE environment could make an extension-based-TDE API too complex to be useful. As far as how to move forward, I think we need to look at the temporary file compression patch, see if that will help TDE code impact, and then decide, with that patch, if the TDE core code impact is acceptable. If not, we then need to look at an extension, what hooks are needed, and then decide if the final result have a simple enough API to be useful. (Do we start from scratch or use Percona's?) If the API is complex, and could lead to corruption/data loss due to encryption, that solution is unacceptable. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Do not let urgent matters crowd out time for investment in the future.
-
Re: Enquiry about TDE with PgSQL
Bruce Momjian <bruce@momjian.us> — 2025-11-03T16:59:47Z
On Mon, Nov 3, 2025 at 08:20:21AM -0800, Adrian Klaver wrote: > On 11/3/25 08:01, Bruce Momjian wrote: > > On Mon, Nov 3, 2025 at 04:39:45PM +0100, rainer@ultra-secure.de wrote: > > > Am 2025-11-03 16:08, schrieb Bruce Momjian: > > > I will admit that companies are better at integrating with external > > vendors, particulary hardware vendors. There is an organization > > mismatch between the community and companies, and the community > > basically forces companies to intract on community terms --- companies > > are a more natural iteraction for other companies. > > Again a distinction without a difference. > > If you go here: > > https://www.postgresql.org/community/contributors/ > > and look at the names under Core Team and Major Contributors, out of 59 > names only 6 do not list a company name. Companies live at the heart of the > community. Hence the the policy of no company having no more then 50%(?) of > the Core team. The issue is that interacting with external companies is rarely done by the community, so if you need to get community software to interact with external software, the external software vendor has to get involved with the community, and for non-Postgres-focused external companies, that is a big hurdle. Companies have many employees who can make it their job to get the integration working, as has happened with the many company-controlled Postgres TDE solutions. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Do not let urgent matters crowd out time for investment in the future.
-
Re: Enquiry about TDE with PgSQL
Adrian Klaver <adrian.klaver@aklaver.com> — 2025-11-03T18:07:50Z
On 11/3/25 8:59 AM, Bruce Momjian wrote: > On Mon, Nov 3, 2025 at 08:20:21AM -0800, Adrian Klaver wrote: >> On 11/3/25 08:01, Bruce Momjian wrote: >>> On Mon, Nov 3, 2025 at 04:39:45PM +0100, rainer@ultra-secure.de wrote: >>>> Am 2025-11-03 16:08, schrieb Bruce Momjian: >> >> Again a distinction without a difference. >> >> If you go here: >> >> https://www.postgresql.org/community/contributors/ >> >> and look at the names under Core Team and Major Contributors, out of 59 >> names only 6 do not list a company name. Companies live at the heart of the >> community. Hence the the policy of no company having no more then 50%(?) of >> the Core team. > > The issue is that interacting with external companies is rarely done by > the community, so if you need to get community software to interact with > external software, the external software vendor has to get involved with > the community, and for non-Postgres-focused external companies, that is > a big hurdle. Companies have many employees who can make it their job > to get the integration working, as has happened with the many > company-controlled Postgres TDE solutions. To be specific it is a customer service issue. So why is this page: https://www.postgresql.org/support/professional_support/ "Commercial support is available from many different companies providing professional services to the PostgreSQL community." not a solution done by the community? -- Adrian Klaver adrian.klaver@aklaver.com
-
Re: Enquiry about TDE with PgSQL
Bruce Momjian <bruce@momjian.us> — 2025-11-03T18:14:18Z
On Mon, Nov 3, 2025 at 10:07:50AM -0800, Adrian Klaver wrote: > On 11/3/25 8:59 AM, Bruce Momjian wrote: > > On Mon, Nov 3, 2025 at 08:20:21AM -0800, Adrian Klaver wrote: > > > On 11/3/25 08:01, Bruce Momjian wrote: > > > > On Mon, Nov 3, 2025 at 04:39:45PM +0100, rainer@ultra-secure.de wrote: > > > > > Am 2025-11-03 16:08, schrieb Bruce Momjian: > > > > > > > Again a distinction without a difference. > > > > > > If you go here: > > > > > > https://www.postgresql.org/community/contributors/ > > > > > > and look at the names under Core Team and Major Contributors, out of 59 > > > names only 6 do not list a company name. Companies live at the heart of the > > > community. Hence the the policy of no company having no more then 50%(?) of > > > the Core team. > > > > The issue is that interacting with external companies is rarely done by > > the community, so if you need to get community software to interact with > > external software, the external software vendor has to get involved with > > the community, and for non-Postgres-focused external companies, that is > > a big hurdle. Companies have many employees who can make it their job > > to get the integration working, as has happened with the many > > company-controlled Postgres TDE solutions. > > To be specific it is a customer service issue. > > So why is this page: > > https://www.postgresql.org/support/professional_support/ > > "Commercial support is available from many different companies providing > professional services to the PostgreSQL community." > > not a solution done by the community? I am not sure why you are not understanding. To get Postgres to interoperate with company product X, you need communication between the Postgres community and the company producing product X. The community rarely interacts directly with companies, though it does interact with employees of Postgres-focused companies. The smaller the company and the less the company relies on Postgres, the less likely the company will interact with the community on the community's terms. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Do not let urgent matters crowd out time for investment in the future.
-
Re: Enquiry about TDE with PgSQL
Laurenz Albe <laurenz.albe@cybertec.at> — 2025-11-03T18:30:01Z
On Mon, 2025-11-03 at 16:39 +0100, rainer@ultra-secure.de wrote: > The HSM should be backed up, too. Which is only possible by connecting > physically to it with a notebook and inserting an USB stick. > > Which begs the question: where do you source an USB stick with the same > trust-level as the 20k-a-pop HSM? I'd say that you don't need a very secure USB stick. You just put the USB stick in a very secure safe that only two very trustworthy people can open together. Yours, Laurenz Albe
-
Re: Enquiry about TDE with PgSQL
Laurenz Albe <laurenz.albe@cybertec.at> — 2025-11-03T18:42:06Z
On Mon, 2025-11-03 at 11:56 -0500, Bruce Momjian wrote: > The problem with the Percona extension is it seems like it was developed > mostly/all by Percona employees, meaning development was driven/steered > by Percona, and there was insufficient feedback from the community for > it to be polished enough to be a general community solution. Reading a Percona blog, it looks like you need a modified server to get to encrypt WAL, and they probably have no support for encrypting temporary files. So I'd say that TDE can probably not be a pure extension. Perhaps somebody from Percona can confirm. But I don't think it's a shortage of implementations for TDE that is the problem. Since you say that encrypting the temp files is the biggest hurdle for community acceptance, what about a first version that does not encrypt temp files? For one, that will be good for encrypted backups (which is one of the good use cases for TDE), and then you could argue that temp files are not data *at rest*, so data-at-rest-encryption does not apply to them. Rome wasn't built in a day, and neither were parallel query or declarative partitioning. Yours, Laurenz Albe
-
Re: Enquiry about TDE with PgSQL
Adrian Klaver <adrian.klaver@aklaver.com> — 2025-11-03T18:46:30Z
On 11/3/25 10:14, Bruce Momjian wrote: > On Mon, Nov 3, 2025 at 10:07:50AM -0800, Adrian Klaver wrote: >> >> To be specific it is a customer service issue. >> >> So why is this page: >> >> https://www.postgresql.org/support/professional_support/ >> >> "Commercial support is available from many different companies providing >> professional services to the PostgreSQL community." >> >> not a solution done by the community? > > I am not sure why you are not understanding. To get Postgres to > interoperate with company product X, you need communication between the > Postgres community and the company producing product X. The community > rarely interacts directly with companies, though it does interact with > employees of Postgres-focused companies. The smaller the company and > the less the company relies on Postgres, the less likely the company > will interact with the community on the community's terms. > I do understand it comes down to two possibilities: 1) Community is what you download here: https://www.postgresql.org/ftp/source/ and anything above that is third party involvement and nothing to do with the community. Or 2) As the postgresql.org mentions in many places that I have previously cited community includes layers above the source, extending to commercial support. If 1) is correct then yes there is a community communication issue. If 2) is correct there is not, as you can go here: https://www.postgresql.org/support/professional_support/ and find support. -- Adrian Klaver adrian.klaver@aklaver.com
-
Re: Enquiry about TDE with PgSQL
Bruce Momjian <bruce@momjian.us> — 2025-11-04T02:05:54Z
On Mon, Nov 3, 2025 at 07:42:06PM +0100, Laurenz Albe wrote: > On Mon, 2025-11-03 at 11:56 -0500, Bruce Momjian wrote: > > The problem with the Percona extension is it seems like it was developed > > mostly/all by Percona employees, meaning development was driven/steered > > by Percona, and there was insufficient feedback from the community for > > it to be polished enough to be a general community solution. > > Reading a Percona blog, it looks like you need a modified server to get > to encrypt WAL, and they probably have no support for encrypting > temporary files. So I'd say that TDE can probably not be a pure extension. > Perhaps somebody from Percona can confirm. Yes, the server has to be modified because the hooks they need don't exist in the community source code. They also have encryption control on the table level, which I frankly think will never work long-term because the storage API doesn't have enough table-level detail, so I think they are considering tablespace-level or cluster-level encryption. > But I don't think it's a shortage of implementations for TDE that is the > problem. > > Since you say that encrypting the temp files is the biggest hurdle for > community acceptance, what about a first version that does not encrypt > temp files? For one, that will be good for encrypted backups (which is > one of the good use cases for TDE), and then you could argue that temp > files are not data *at rest*, so data-at-rest-encryption does not apply > to them. Rome wasn't built in a day, and neither were parallel query > or declarative partitioning. Uh, people will say that if the solution is not 100% secure in its coverage, it is much less useful and therefore not worth it. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Do not let urgent matters crowd out time for investment in the future.
-
Re: Enquiry about TDE with PgSQL
Adrian Klaver <adrian.klaver@aklaver.com> — 2025-11-04T05:23:33Z
On 11/3/25 18:05, Bruce Momjian wrote: > On Mon, Nov 3, 2025 at 07:42:06PM +0100, Laurenz Albe wrote: >> On Mon, 2025-11-03 at 11:56 -0500, Bruce Momjian wrote: >>> The problem with the Percona extension is it seems like it was developed >>> mostly/all by Percona employees, meaning development was driven/steered >>> by Percona, and there was insufficient feedback from the community for >>> it to be polished enough to be a general community solution. > > Uh, people will say that if the solution is not 100% secure in its > coverage, it is much less useful and therefore not worth it. > In a different realm you have been there before and it worked out: https://momjian.us/main/blogs/pgblog/2010.html#May_12_2010 -- Adrian Klaver adrian.klaver@aklaver.com
-
Re: Enquiry about TDE with PgSQL
Laurenz Albe <laurenz.albe@cybertec.at> — 2025-11-04T05:40:26Z
On Mon, 2025-11-03 at 21:05 -0500, Bruce Momjian wrote: > On Mon, Nov 3, 2025 at 07:42:06PM +0100, Laurenz Albe wrote: > > > Since you say that encrypting the temp files is the biggest hurdle for > > community acceptance, what about a first version that does not encrypt > > temp files? For one, that will be good for encrypted backups (which is > > one of the good use cases for TDE), and then you could argue that temp > > files are not data *at rest*, so data-at-rest-encryption does not apply > > to them. Rome wasn't built in a day, and neither were parallel query > > or declarative partitioning. > > Uh, people will say that if the solution is not 100% secure in its > coverage, it is much less useful and therefore not worth it. Some people will doubtless say that. Others will consider the checkbox requirement satisfied and use it. Yet others will consider a mislaid backup their biggest problem and will consider TDE a technically useful solution. 9.6, which introduced parallel query, only supported it for sequential scans, which was much less useful than what we have today. I for one wouldn't consider an implementation of TDE with some features missing "not worth it". If anything, I consider the marginal security improvement that TDE as a whole provides not worth it. But I am sold on the claim that having TDE would promote the adoption of PostgreSQL. I am curious what others think. Yours, Laurenz Albe
-
Re: Enquiry about TDE with PgSQL
Álvaro Herrera <alvherre@kurilemu.de> — 2025-11-04T12:12:07Z
On 2025-Nov-04, Laurenz Albe wrote: > 9.6, which introduced parallel query, only supported it for sequential > scans, which was much less useful than what we have today. I for one > wouldn't consider an implementation of TDE with some features missing > "not worth it". We call this incremental development, and we've accepted that it might be the only way to get complex features developed. We did it for partitioning, parallel query, AIO, and lots of other things. IMO it's a good strategy, even if we sometimes have to backtrack. -- Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/ "No tengo por qué estar de acuerdo con lo que pienso" (Carlos Caszeli) -
RE: Enquiry about TDE with PgSQL
Clay Jackson (cjackson) <clay.jackson@quest.com> — 2025-11-04T15:15:31Z
Again, speaking for myself only and not officially for Quest. " Uh, people will say that if the solution is not 100% secure in its coverage, it is much less useful and therefore not worth it." I would assert that NO system is "100% secure" given enough money and resources. I think the more important point of this discussion is how users of PostgreSQL can "check the box" for compliance with whatever security and encryption "standards" are "required" in their environment, and/or mitigate the risks of not being "compliant". Clay Jackson -----Original Message----- From: Bruce Momjian <bruce@momjian.us> Sent: Monday, November 3, 2025 6:06 PM To: Laurenz Albe <laurenz.albe@cybertec.at> Cc: Kai Wagner <kai.wagner@percona.com>; Chris Travers <chris.travers@gmail.com>; Christophe Pettus <xof@thebuild.com>; Clay Jackson (cjackson) <Clay.Jackson@quest.com>; pgsql-general <pgsql-general@postgresql.org>; Ron Johnson <ronljohnsonjr@gmail.com> Subject: Re: Enquiry about TDE with PgSQL CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. On Mon, Nov 3, 2025 at 07:42:06PM +0100, Laurenz Albe wrote: > On Mon, 2025-11-03 at 11:56 -0500, Bruce Momjian wrote: > > The problem with the Percona extension is it seems like it was > > developed mostly/all by Percona employees, meaning development was > > driven/steered by Percona, and there was insufficient feedback from > > the community for it to be polished enough to be a general community solution. > > Reading a Percona blog, it looks like you need a modified server to > get to encrypt WAL, and they probably have no support for encrypting > temporary files. So I'd say that TDE can probably not be a pure extension. > Perhaps somebody from Percona can confirm. Yes, the server has to be modified because the hooks they need don't exist in the community source code. They also have encryption control on the table level, which I frankly think will never work long-term because the storage API doesn't have enough table-level detail, so I think they are considering tablespace-level or cluster-level encryption. > But I don't think it's a shortage of implementations for TDE that is > the problem. > > Since you say that encrypting the temp files is the biggest hurdle for > community acceptance, what about a first version that does not encrypt > temp files? For one, that will be good for encrypted backups (which > is one of the good use cases for TDE), and then you could argue that > temp files are not data *at rest*, so data-at-rest-encryption does not > apply to them. Rome wasn't built in a day, and neither were parallel > query or declarative partitioning. Uh, people will say that if the solution is not 100% secure in its coverage, it is much less useful and therefore not worth it. -- Bruce Momjian <bruce@momjian.us> https://momjian.us/ EDB https://enterprisedb.com/ Do not let urgent matters crowd out time for investment in the future.
-
Re: Enquiry about TDE with PgSQL
Jan Wieremjewicz <jan.wieremjewicz@percona.com> — 2025-11-06T09:29:04Z
Just wanted to throw in my two cents, or actually, two points, on TDE: 1. Extensions seem like the right place for most of the TDE logic. Encryption usually depends on organization specific or even proprietary security systems. Think of features like KMS integrations that need custom handling and are often owned by Security, Compliance, or IT rather than DBAs, so it is hard to convince DBAs to stick to a "supported" one. That kind of setup is difficult to generalize into PostgreSQL core, and extensions give the flexibility to make it work. The same goes for cloud KMS or HSM certification integrations, which add another layer of complexity that is easier to manage outside core. Also, by relying mainly on new hooks, we can keep changes to the core minimal and make TDE as non-intrusive as possible for those who do not need it. 2. About the "single-vendor open source" comment... That was never the goal when we kicked off pg_tde. Sure, Percona funded the initial work, but we would really like to see this become a broader community effort. We saw a gap that enterprise users were facing and tried to fill it based on what we had learned from other databases, but the more people get involved, the better this can become. On Tue, Nov 4, 2025 at 3:06 AM Bruce Momjian <bruce@momjian.us> wrote: > On Mon, Nov 3, 2025 at 07:42:06PM +0100, Laurenz Albe wrote: > > On Mon, 2025-11-03 at 11:56 -0500, Bruce Momjian wrote: > > > The problem with the Percona extension is it seems like it was > developed > > > mostly/all by Percona employees, meaning development was driven/steered > > > by Percona, and there was insufficient feedback from the community for > > > it to be polished enough to be a general community solution. > > > > Reading a Percona blog, it looks like you need a modified server to get > > to encrypt WAL, and they probably have no support for encrypting > > temporary files. So I'd say that TDE can probably not be a pure > extension. > > Perhaps somebody from Percona can confirm. > > Yes, the server has to be modified because the hooks they need don't > exist in the community source code. They also have encryption control > on the table level, which I frankly think will never work long-term > because the storage API doesn't have enough table-level detail, so I > think they are considering tablespace-level or cluster-level encryption. > > > But I don't think it's a shortage of implementations for TDE that is the > > problem. > > > > Since you say that encrypting the temp files is the biggest hurdle for > > community acceptance, what about a first version that does not encrypt > > temp files? For one, that will be good for encrypted backups (which is > > one of the good use cases for TDE), and then you could argue that temp > > files are not data *at rest*, so data-at-rest-encryption does not apply > > to them. Rome wasn't built in a day, and neither were parallel query > > or declarative partitioning. > > Uh, people will say that if the solution is not 100% secure in its > coverage, it is much less useful and therefore not worth it. > > -- > Bruce Momjian <bruce@momjian.us> > https://url.avanan.click/v2/r01/___https://momjian.us___.YXAzOnBlcmNvbmE6YTpnOjI1Nzk5MzJmMmQ0NDFlMGE0NGRhMjc4NDMwYWVkZDZlOjc6MTFhOToxZDRhZjAyODZkMjQ3MzlhM2EwMWUzNmFkZGM0ZTkyYmQ3MjE0NWVlY2VjZDRmYTFkMWM2NTUxOTUwYjQ1OGY5OnA6VDpO > EDB > https://url.avanan.click/v2/r01/___https://enterprisedb.com___.YXAzOnBlcmNvbmE6YTpnOjI1Nzk5MzJmMmQ0NDFlMGE0NGRhMjc4NDMwYWVkZDZlOjc6ZGM0YTphNjQ1MzQyOWEyNzBiNzYyYTc5NjJlN2NlYjU1ZTM5YmZkYWZjMTYyNzViN2IzNWQzOTU4N2QxZTIwNjcxMDdlOnA6VDpO > > Do not let urgent matters crowd out time for investment in the future. > > > -- <https://www.percona.com/> Jan Wieremjewicz Sr. Product Manager, Percona jan.wieremjewicz@percona.com <email@percona.com> CET (GMT+1) Databases Run Better With Percona
-
Re: Enquiry about TDE with PgSQL
Bruce Momjian <bruce@momjian.us> — 2025-11-08T03:25:41Z
On Mon, Nov 3, 2025 at 09:23:33PM -0800, Adrian Klaver wrote: > On 11/3/25 18:05, Bruce Momjian wrote: > > On Mon, Nov 3, 2025 at 07:42:06PM +0100, Laurenz Albe wrote: > > > On Mon, 2025-11-03 at 11:56 -0500, Bruce Momjian wrote: > > > > The problem with the Percona extension is it seems like it was developed > > > > mostly/all by Percona employees, meaning development was driven/steered > > > > by Percona, and there was insufficient feedback from the community for > > > > it to be polished enough to be a general community solution. > > > > > Uh, people will say that if the solution is not 100% secure in its > > coverage, it is much less useful and therefore not worth it. > > > > In a different realm you have been there before and it worked out: > > https://momjian.us/main/blogs/pgblog/2010.html#May_12_2010 Yes, pg_upgrade was a hard one too, and it took a long time be accepted, so there is hope. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Do not let urgent matters crowd out time for investment in the future.