is there a way to automate deduplication of strings?
Chris Papademetrious <christopher.papademetrious@synopsys.com>
From: Chris Papademetrious <Christopher.Papademetrious@synopsys.com>
To: "pgsql-novice@lists.postgresql.org" <pgsql-novice@lists.postgresql.org>
Date: 2025-12-27T12:36:20Z
Lists: pgsql-novice
Hello everyone! First time poster here. I have a question about deduplicating text strings stored in a database. I am aware of the pattern of creating a separate table for unique values, then referencing those values by key. But this requires some transactional complexity for storage and retrieval, along with cleanup of no-longer-referenced values over time. And, this complexity grows with the number of primary-table columns that use this indirection. I would only use this for (1) seldom-referenced columns that (2) have a high rate of duplication and (3) have an average string length that makes deduplication worthwhile. Are there any native or extension-based methods to simplify this in Postgres? I searched and came up empty, but maybe I'm not searching with the right terms. Thanks! * Chris