Horton

Re: [HACKERS] Custom compression methods

Robert Haas <robertmhaas@gmail.com>

View thread View on postgr.es
From: Robert Haas <robertmhaas@gmail.com>
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>
Cc: Alvaro Herrera <alvherre@alvh.no-ip.org>, Ildus Kurbangaliev <i.kurbangaliev@postgrespro.ru>, "pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Date: 2017-12-14T15:25:34Z
Lists: pgsql-hackers 
On Wed, Dec 13, 2017 at 1:34 PM, Tomas Vondra
<tomas.vondra@2ndquadrant.com> wrote:
> Wouldn't that require some universal compression level, shared by all
> supported compression algorithms? I don't think there is such thing.
>
> Defining it should not be extremely difficult, although I'm sure there
> will be some cumbersome cases. For example what if an algorithm "a"
> supports compression levels 0-10, and algorithm "b" only supports 0-3?
>
> You may define 11 "universal" compression levels, and map the four
> levels for "b" to that (how). But then everyone has to understand how
> that "universal" mapping is defined.

What we could do is use the "namespace" feature of reloptions to
distinguish options for the column itself from options for the
compression algorithm.  Currently namespaces are used only to allow
you to configure toast.whatever = somevalue, but we could let you say
pglz.something = somevalue or lz4.whatever = somevalue.  Or maybe, to
avoid confusion -- what happens if somebody invents a compression
method called toast? -- we should do it as compress.lz4.whatever =
somevalue.  I think this takes us a bit far afield from the core
purpose of this patch and should be a separate patch at a later time,
but I think it would be cool.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Commits

  1. docs: Update TOAST storage docs for configurable compression.

  2. Further tweaking of pg_dump's handling of default_toast_compression.

  3. Fix interaction of TOAST compression with expression indexes.

  4. Tidy up more loose ends related to configurable TOAST compression.

  5. Short-circuit slice requests that are for more than the object's size.

  6. Mostly-cosmetic adjustments of TOAST-related macros.

  7. Remove useless configure probe for <lz4/lz4.h>.

  8. Error on invalid TOAST compression in CREATE or ALTER TABLE.

  9. docs: Fix omissions related to configurable TOAST compression.

  10. More code cleanup for configurable TOAST compression.

  11. Bring configure support for LZ4 up to snuff.

  12. Make compression.sql regression test independent of default.

  13. Use valid compression method in brin_form_tuple

  14. Fix up pg_dump's handling of per-attribute compression options.

  15. Allow configurable LZ4 TOAST compression.

  16. Fix inconsistencies in the code

  17. Mostly-cosmetic improvements in memory chunk header alignment coding.

  18. Allow numeric to use a more compact, 2-byte header in many cases.