Thread

1. Experimenting with wider Unicode storage

   Thomas Munro <thomas.munro@gmail.com> — 2026-04-15T13:43:26Z

   Hi,
   
   We only allow one character encoding per database.  The SQL standard
   and most comparable RDBMSs are more flexible, though the details vary.
   In most, you can attach CHARACTER SET to column definitions, a whole
   CREATE TABLE (non-standard), CREATE DOMAIN, CREATE SCHEMA and probably
   more places.  In some, encoding is implied by COLLATE instead
   (non-standard).  When different encodings meet, text is transcoded,
   which works but is bad for performance.
   
   Our single database encoding also has some restrictions: it must
   encode ASCII as ASCII, and any byte that is part of a multibyte
   sequence must not look like ASCII, since many code paths require that
   and fixing that is hard.  That excludes a few encodings that people
   want.
   
   That's all OK by now, as modern information systems use Unicode
   everywhere.  There is one practical problem that causes people to
   complain[2] about PostgreSQL though: about half of the global
   population uses a language that arbitrarily gained a byte per
   character by switching to UTF-8 compared to various legacy encodings
   or GB18030.  That's storage and RAM that you have to pay for up front
   and forever (space, I/O[3]).
   
   I wondered about inventing new PostgreSQL-backend-compatible encodings
   that swizzle bits around to make CJK/I languages fit, but I kept
   coming back to Unicode.
   
   In many systems there is a special way to use UTF-16, which gets you
   back to around two bytes per character for Chinese, Japanese, Korean
   and probably Indian languages.  In MySQL/MariaDB, you can use various
   encodings of UTF-16 as a CHARACTER SET with the normal text types, and
   in Oracle, DB2, SQL Server/Sybase you can use a separate NVARCHAR type
   for UTF-16, alongside the regular text types whose encodings are
   controlled by CHARACTER SET or COLLATE.  N* types are shorthand for
   the standardese "NATIONAL <TYPE>", a string in an unspecified special
   encoding, for which they all chose some kind of UTF-16.
   
   At first I thought that with first class extension types as one of our
   superpowers, we might be able to do that with an extension, but the
   problem there is that text is so tangled up with locales, and I
   certainly didn't want to convert all over the place.  So I tried to
   hack up a minimal demonstration of what a separate UTF-16 text type
   might look like as a core data type.
   
   You can save a lot of space if you have a separate "utf16" type
   ("national text"?), but only for East and South Asian languages:
   
   + language | octets  | delta |                     string
   +----------+---------+-------+-------------------------------------------------
   + English  | 45→90   | +100% | In a hole in the ground there lived a hobbit.
   + Spanish  | 44→86   |  +95% | En un agujero en el suelo, vivía un hobbit.
   + Russian  | 59→66   |  +12% | В норе под землей жил-был хоббит.
   + Arabic   | 57→64   |  +12% | كان يعيش هوبيت في حفرة في الأرض.
   + Hebrew   | 43→48   |  +12% | בתוך חור באדמה חי הוביט.
   + Greek    | 74→82   |  +11% | Σε μια τρύπα στο έδαφος ζούσε ένα χόμπιτ.
   + Korean   | 55→46   |  -16% | 땅속 어느 구멍에 한 호빗이 살고 있었다.
   + Hindi    | 111→86  |  -23% | जमीन में बने एक गड्ढे में एक हॉबिट रहता था।
   + Tamil    | 146→108 |  -26% | அந்த நிலத்தில் ஒரு துளையில் ஒரு ஹாபிட்
   வசித்து வந்தது.
   + Chinese  | 51→34   |  -33% | 在地下一个洞里，住着一个霍比特人。
   + Japanese | 66→44   |  -33% | 穴のなかに、ひとりのホビットが暮らしていた。
   
   It's actually -33% in Korean too without spaces, or if you use
   double-width spaces, a more common stylistic choice IIUC (this Korean
   string happens to use ASCII spaces).  The same sort of thing explains
   why Russian, Arabic, Hewbrew and Greek lose ~12% instead of breaking
   even: they share space and some punctuation with ASCII.
   
   The attached is highly exploratory concept code to try the idea out
   and see if experts in CJK computing, defenders of the type system and
   others think it might be worth exploring further.  Some of the
   technical challenges and observations I spotted along the way:
   
   * "text" (etc) and "utf16" need to be comparable incrementally without
   conversion
   * N^2 explosions in cross-type support function definitions must
   surely be avoided
   * the solution to that is surely generic programming, but PostgreSQL
   is written in ye olde C
   * therefore, this POC patch is big on macros as poor-man's C++
   * to support fast paths like memcmp()-based ucs_basic comparison you
   need big endian UTF-16
   * to stuff UTF-16 into varlena you need to tolerate unaligned access
   * by a happy coincidence, ICU supports big endian, unaligned UTF-16
   * Windows libc locales might in theory allow UTF-16, but that'd have
   to be native endian, which I didn't implement
   * other systems actually allow both endiannesses as encodings or
   subtypes, at the user's option
   * they also allow you to control whether surrogates are allowed
   
   In this patch you can see some string iterator concepts that I have
   been hacking on for an entirely different purpose, namely trying to
   figure out how to make our multibyte string support go faster (and
   also be safer) by hoisting all the character-at-a-time loops out to
   specialisations in string handling functions.  That's not shown here.
   Neither is any kind of silent transcoding that plagues other systems
   that do this kind of thing:
   
   * the "utf16" type is only allowed to contain text that is a ASCII,
   LATIN1 or Unicode, depending on the database encoding
   * for UTF8, it's all of Unicode, for which mb_iterator can trivally
   produce UTF-16 or UTF-32
   * for LATIN1, that becomes trivial casting since LATIN1 is a strict
   subset of Unicode
   * for anything else, utf16 only allows ASCII characters, as a degraded
   mode just to allow the tests to pass
   
   In other words, so far utf16 is not allowed to represent anything that
   "text" could not represent and convert trivially.  That was originally
   a decision to make a quick proof-of-concept plausible, but maybe it's
   even a good idea...
   
   I have no doubt that there are lots of complicated problems that I
   haven't met yet, when you add more types.  What I was trying to
   explore here was whether you can exclude most of them by providing
   enough conversion-free (incremental) cross-type support.  This is
   something SQL Server DBAs talk about: mixtures of NVARCHAR and VARCHAR
   columns and index befuddle the planner and introduce hidden execution
   costs if you're not careful.  It struck me that with the above
   restriction you could perhaps keep all strings cheaply and
   incrementally comparable.  There are still some things you can't do:
   
   * if you're comparing "text" with "utf16" then you lose the
   length-based not-equal fast path
   * that's a big deal if it means detoasting
   
   Sharing for discussion only.  It has enough working to support
   converting individual columns text->utf16 and use btree indexes.  Many
   more text functions would need to be converted to generic form, and
   many more support functions would be needed for full functionality.
   
   (Individual Indian languages could in theory be compacted even further
   to single-byte ISCII.  ISCII is in general infeasible as a server
   encoding because it has stateful shifts between many scripts, but
   single-script variants of ISCII as supported on some Unixen would in
   theory be plausible.  Since India has so many languages and scripts
   and information systems often need to support all of them, I am
   reliably informed that UTF-8 reigns supreme there despite taking 3
   bytes to represent the tiny 6 bit (?) character set of any individual
   language like Devanagari (Hindi etc).  GNU/Linux doesn't even support
   ISCII, so that idea is basically DOA.)
   
   [1] https://www.postgresql.org/docs/current/infoschema-character-sets.html
   [2] https://www.postgresql.org/message-id/flat/ME2PR01MB2532E72B514DC46ED0E10F798A0C0%40ME2PR01MB2532.ausprd01.prod.outlook.com#7d490f97a3df6dfef61e485161e72e06
   

2. Re: Experimenting with wider Unicode storage

   Henson Choi <assam258@gmail.com> — 2026-04-16T01:23:32Z

   Hi Thomas,
   
   Thank you for sharing this very interesting and creative approach.
   Encoding is indeed a crucial factor in capacity planning and
   performance benchmarking — I find this direction quite compelling.
   
   I'm currently working on a few other things, so my responses may not
   always be quick, but I wanted to let you know I'm genuinely
   interested in following this work.
   
   As it happens, I'm currently collaborating with Ishii-san — who, as
   you know, is one of the original architects of multibyte/CJK support
   in PostgreSQL — on Row Pattern Recognition; that might also be a
   thread worth keeping an eye on.
   
   It also strikes me that this is a topic worth considering in the
   context of the rapid growth of SNS and AI-generated data. The
   pervasive use of emoji — which cannot be represented in legacy
   encodings like EUC-KR at all — is in fact accelerating the migration
   toward Unicode in Korea and other Asian markets. This makes the
   storage efficiency of Unicode for CJK characters an increasingly
   practical concern, not just a theoretical one.
   
   I'd like to take some time to analyze the current situation around
   character encoding in Korea — where both EUC-KR legacy systems and
   UTF-8 coexist in complex ways — review the patches you've attached,
   and then share some thoughts and feedback.
   
   Best regards,
   Henson
   

3. Re: Experimenting with wider Unicode storage

   Henson Choi <assam258@gmail.com> — 2026-04-21T01:16:26Z

   Hi Thomas,
   
   
   Thank you again for sharing this exploration, and for including
   Korean in your experiment table.  Rather than comment on the
   patch itself, let me offer a ground-level report on where Korean
   encoding reality sits in April 2026, because the picture has
   shifted enough that I think it is worth entering into the record
   before this thread accumulates momentum on motivations that may
   no longer fully hold on this side of the region.
   
   
   UTF-8 has already won in Korea, largely by inertia rather than
   active choice.  Public web statistics put .kr sites at roughly
   96% UTF-8 with a small EUC-KR residual of about 4% [1] —
   noticeably higher than the ~1% Shift-JIS residual on .jp [2],
   but steadily shrinking.  The mechanism is mundane: modern Linux
   distributions default to UTF-8 locales, PostgreSQL's initdb
   inherits that, and every new cluster is therefore UTF-8 from
   birth.  The remaining legacy installations are not "haven't
   migrated yet" — they are "have decided not to migrate," which is
   a different and much slower population.
   
   
   A clarification that often trips people up: in Korean practice,
   "EUC-KR" is the label written down and CP949 is what actually
   moves on the wire.  Microsoft's UHC has been the Windows default
   for decades, and the MIME label has simply stuck.  The historical
   stack goes KS X 1001 (완성형, 2,350 syllables) → EUC-KR → CP949
   (11,172 syllables) → UTF-8.  PostgreSQL's strict EUC_KR decoder
   rejects the bytes CP949 adds, which occasionally causes real
   incidents when Windows-exchanged files are loaded.  For any
   design choice about "Korean legacy support", this matters — what
   needs supporting is usually CP949, not EUC-KR proper.
   
   
   Server encoding and client encoding are also routinely split.  A
   common Korean deployment pattern is a PostgreSQL cluster with
   UTF-8 as server encoding, while legacy Windows / Delphi / C++ /
   older Java clients connect with client_encoding set to EUC-KR or
   CP949 and let PostgreSQL transcode at the wire boundary.  Many
   systems that look like "EUC-KR systems" from the outside are
   actually UTF-8 storage with an EUC-KR wire.  The storage-layer
   share of legacy is therefore probably smaller still than the
   3.8% web figure would suggest.
   
   
   On the Korean row of your table landing at -16% under UTF-16:
   that is structural, not noise.  Modern Korean writing mandates
   word-space separation (unlike Chinese and Japanese), has
   effectively abandoned hanja since the 1990s, and freely
   interleaves ASCII acronyms (IT, AI, CEO).  As a result Korean
   carries the highest ASCII share among CJK languages, and UTF-16
   pays for each ASCII position (one byte → two) in exactly the
   range where the Hangul savings are meant to come from.  Columns
   without spaces — names, titles, addresses — could approach -33%,
   but general prose cannot.  Those same short columns are, however,
   exactly where the compression angle I return to further below
   captures the equivalent saving without a new data type.
   
   
   Storage pressure, to the extent modern operators feel it at all,
   has largely migrated to other layers.  Memory and disk have both
   followed exponential price/volume curves, and the CPU cost of
   text comparison has disappeared inside other costs — network,
   storage I/O, planning, JIT — to the point of invisibility in
   profiler output.  For OLTP, the 2-vs-3-byte difference on Korean
   columns does not feel meaningful on modern hardware.  For bulk
   scans where byte counts still do matter, the industry answer has
   already been columnar + zstd, which routinely reaches 90%+
   compression on natural-language text and flattens the
   CJK-vs-Latin ratio to irrelevance.  Embedded and edge are not
   PostgreSQL's primary target, and archival sits in zstd territory
   too.  The domains that historically motivated "we must narrow
   CJK storage" have either moved outside the PostgreSQL shape or
   been absorbed by general-purpose compression.
   
   
   Meanwhile the cultural arrow points toward more Unicode, not
   less.  KakaoTalk (which saturates domestic messaging), Naver
   comments, Instagram captions, and YouTube normalise emoji in
   everyday prose, while AI-generated Korean text contributes
   middle dots, em dashes, and curly quotes at a scale that was
   not present a few years ago.  The share of non-EUC-KR content
   in everyday Korean prose is, informally, rising steadily.  Each
   emoji is four UTF-8 bytes and is unrepresentable in any legacy
   encoding at all.
   A partial-coverage alternative looks increasingly awkward against
   that trend.
   
   
   Korean upstream feedback on encoding has also been notably quiet
   despite a very active de-Oracle migration wave in the late 2010s.
   I suspect this silence is not apathy but absence of a felt
   problem — most of the community has simply moved on.
   
   
   I should be careful here.  The "Korean side needs narrower CJK
   storage" argument was genuinely strong around 2010, and I
   remember when it motivated serious engineering time.  It is much
   weaker in 2026: UTF-8 has won by default, legacy survivors are
   confined to wire protocols and specific applications, OLTP does
   not feel the byte cost, and bulk scan is already handled
   elsewhere.  I raise this not to dismiss the technical work — the
   patch shows real craft and the exploration is interesting on its
   own terms.  But if the cover-letter motivation rests partly on
   "this will help East Asian users, including Korea," I wanted you
   to have a ground-level report: for Korean users specifically, the
   pressure may no longer be strong enough to justify the complexity
   described.  The calculus may well differ in Japanese or Chinese
   markets — that is not for me to say.
   
   
   One broader question, then, that I wanted to put to you: there
   are three distinct axes on which utf16 could be pursued — as a
   server character set, as a data type, or as a compression angle.
   The character-set direction runs straight into the "continuation
   byte must not look like ASCII" rule, as you already noted, and
   is therefore effectively closed on PostgreSQL.  The data-type
   direction is the current patch, which carries substantial
   catalogue and operator surface, while the storage wins mostly
   accrue on wider values — where columnar + zstd is already doing
   the work.  What still seems genuinely unaddressed in practice is
   the short-value regime: word-sized strings such as names,
   titles, cities, and tags, which fall below the TOAST compression
   threshold and therefore never see a compressor at all.  Would
   framing this as "a compression method effective on word-sized
   values" be a more productive angle than either of the other two?
   The storage outcome could be similar with much less surface area
   to maintain.
   
   
   A fair counter on memory, before I go on: disk pressure has
   clearly migrated elsewhere, but shared_buffers and work_mem
   remain finite, and compression primarily addresses the disk
   side.  A data-type approach that goes far enough to shrink the
   in-memory representation — modifying every string function
   along the way — tends to become a degraded form of a new
   character set: doing most of the character-set work without the
   character-set slot in PostgreSQL's encoding machinery, which as
   above is closed.  None of the three axes therefore cleanly
   solves the in-memory case; for truly memory-bound CJK workloads
   the honest answer is probably just more RAM.
   
   
   One concrete instantiation of that compression angle, if Korean
   capacity specifically is the example that matters: take CP949
   (which is what actually circulates under the EUC-KR label) as a
   compression base and, for any character CP949 cannot represent,
   spell it inline as a readable textual escape such as \u2603 or
   U+2603 rather than a binary marker byte.  Native Korean text
   then stays at two bytes per Hangul, emoji and modern Unicode
   remain fully representable (at a modest cost per occurrence),
   the in-memory representation stays plain UTF-8, and the on-disk
   byte stream stays entirely within ASCII + CP949 — no new marker
   byte, no collision with existing code paths that scan for raw
   ASCII bytes.  If the source text itself contains sequences that
   look like the escape syntax (for instance documentation quoting
   \u-style literals), a simple doubling rule disambiguates them;
   such cases are vanishingly rare in Korean business data.  This
   targets exactly the short-value regime above, with far less
   surface than a new data type.
   
   
   For tighter byte density, one could go further by devising a
   dedicated binary-level encoding, or by wiring zstd's external
   dictionary feature into the column-compression path with a
   pre-trained per-language dictionary — but either of those paths
   carries its own implementation and operational costs.
   
   
   Should you nonetheless decide to press on with utf16 as a data
   type, I am willing to take the patch through a proper review; I
   have already applied it on top of master and confirmed that the
   regression tests pass, so the mechanical footing is in place.
   
   
   [1] https://w3techs.com/technologies/segmentation/tld-kr-/character_encoding
   [2] https://w3techs.com/technologies/segmentation/tld-jp-/character_encoding
   
   
   Best regards,
   Henson
   
   >
   

4. Re: Experimenting with wider Unicode storage

   Thomas Munro <thomas.munro@gmail.com> — 2026-04-30T00:40:52Z

   On Tue, Apr 21, 2026 at 1:16 PM Henson Choi <assam258@gmail.com> wrote:
   > Thank you again for sharing this exploration, and for including
   > Korean in your experiment table.  Rather than comment on the
   > patch itself, let me offer a ground-level report on where Korean
   > encoding reality sits in April 2026, because the picture has
   > shifted enough that I think it is worth entering into the record
   > before this thread accumulates momentum on motivations that may
   > no longer fully hold on this side of the region.
   
   Hi Henson,
   
   Thank you for this thoughtful and broad feedback, which provided a lot
   of useful context.  I appreciated all of it, and have responses to a
   couple of the most actionable paragraphs:
   
   > One broader question, then, that I wanted to put to you: there
   > are three distinct axes on which utf16 could be pursued — as a
   > server character set, as a data type, or as a compression angle.
   > The character-set direction runs straight into the "continuation
   > byte must not look like ASCII" rule, as you already noted, and
   > is therefore effectively closed on PostgreSQL.  The data-type
   > direction is the current patch, which carries substantial
   > catalogue and operator surface, while the storage wins mostly
   > accrue on wider values — where columnar + zstd is already doing
   > the work.  What still seems genuinely unaddressed in practice is
   > the short-value regime: word-sized strings such as names,
   > titles, cities, and tags, which fall below the TOAST compression
   > threshold and therefore never see a compressor at all.  Would
   > framing this as "a compression method effective on word-sized
   > values" be a more productive angle than either of the other two?
   > The storage outcome could be similar with much less surface area
   > to maintain.
   
   Yeah, that is an interesting angle that I hadn't considered, at least
   not with that framing.  There are even a couple of Unicode standards
   that might apply here, and that I believe some other systems are
   using:
   
   https://en.wikipedia.org/wiki/Standard_Compression_Scheme_for_Unicode
   https://en.wikipedia.org/wiki/Binary_Ordered_Compression_for_Unicode
   https://www.unicode.org/notes/tn6/
   
   BOCU-1 maintains binary codepoint order and reports typical
   English/French as no size change compared to UTF-8,
   Greek/Russian/Arabic/Hebrew as -40%, Hindi as -60% (this makes sense:
   it's almost a generalised ISCII, so you get down to one byte per
   character in any given Indian language), Japanese as -40% and
   Chinese/Korean as -25% (Japanese presumably wins with kana sequences).
   
   One of the ideas already mentioned in comments in the experimental
   patch was that the iterator abstraction could allow for incremental
   decompression, and I suppose there might be a way to expand BOCU-1 or
   similar to UTF-8 incrementally in that layer.  I haven't looked into
   that seriously though; so far I had only been thinking of that as a
   way of generalising some open coded special cases that appear in a few
   places to avoid detoasting.  ICU might also be able to consume it
   incrementally, IDK.
   
   zstd etc can clearly compress much more than that, as you say, but
   then you have to deal with dictionary problems and it's hard to do
   that for small values in a row-oriented system, as you say.  BOCU-1 is
   dictionary-free, so you read it in direct byte order with only a tiny
   state in a register or two, which seems to be potentially along the
   lines you're suggesting.  Food for thought.
   
   > A fair counter on memory, before I go on: disk pressure has
   > clearly migrated elsewhere, but shared_buffers and work_mem
   > remain finite, and compression primarily addresses the disk
   > side.  A data-type approach that goes far enough to shrink the
   > in-memory representation — modifying every string function
   > along the way — tends to become a degraded form of a new
   > character set: doing most of the character-set work without the
   > character-set slot in PostgreSQL's encoding machinery, which as
   > above is closed.  None of the three axes therefore cleanly
   > solves the in-memory case; for truly memory-bound CJK workloads
   > the honest answer is probably just more RAM.
   
   Yeah.  It's an annoying set of constraints that led me to consider
   this, while surveying text handling choices made in lots of database
   systems.  Of course it wouldn't be my preference to introduce a new
   type, but I couldn't see how how else to fit it in, and since I was
   already investigating "modifying every string function along the way"
   for other reasons, I wanted to explore what it would take to do that
   generically enough to handle something as different as this while
   remaining maintainable...
   
   BTW here is the link that I forgot to add to the bottom of my earlier
   email as reference [3], which is a blog from when SQL Server
   introduced the *opposite* thing: UTF-8 support (like Windows itself,
   in 2019).  Previously they had only legacy single/multi-byte encodings
   in VARCHAR and UTF-16 in NVARCHAR, so there they were discussing this
   tradeoff in reverse, ie space savings for some languages, but reported
   25% increase in disk I/O for CJK databases moved to UTF-8.  (I don't
   immediately know why SCSU didn't fix that.)
   
   https://techcommunity.microsoft.com/blog/sqlserver/introducing-utf-8-support-for-sql-server/734928
   
   > Should you nonetheless decide to press on with utf16 as a data
   > type, I am willing to take the patch through a proper review; I
   > have already applied it on top of master and confirmed that the
   > regression tests pass, so the mechanical footing is in place.
   
   Thanks.  I'm not planning to do more with the "separate UTF-16 type"
   concept at this stage, based on your feedback so far.  I am still
   working on a couple of text/encoding refactoring prototypes with other
   goals, and will try to think about that "special Unicode compression"
   angle while doing so.