Horton

Re: [HACKERS] Custom compression methods

Alvaro Herrera <alvherre@alvh.no-ip.org>

View thread View on postgr.es
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>
Cc: Robert Haas <robertmhaas@gmail.com>, Ildus Kurbangaliev <i.kurbangaliev@postgrespro.ru>, "pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Date: 2017-12-13T16:55:01Z
Lists: pgsql-hackers 
Tomas Vondra wrote:

> On 12/13/2017 01:54 AM, Robert Haas wrote:

> > 3. Compression is only applied to large-ish values.  If you are just
> > making the data type representation more compact, you probably want to
> > apply the new representation to all values.  If you are compressing in
> > the sense that the original data gets smaller but harder to interpret,
> > then you probably only want to apply the technique where the value is
> > already pretty wide, and maybe respect the user's configured storage
> > attributes.  TOAST knows about some of that kind of stuff.
> 
> Good point. One such parameter that I really miss is compression level.
> I can imagine tuning it through CREATE COMPRESSION METHOD, but it does
> not seem quite possible with compression happening in a datatype.

Hmm, actually isn't that the sort of thing that you would tweak using a
column-level option instead of a compression method?
  ALTER TABLE ALTER COLUMN SET (compression_level=123)
The only thing we need for this is to make tuptoaster.c aware of the
need to check for a parameter.

> > I don't think TOAST needs to be entirely transparent for the
> > datatypes.  We've already dipped our toe in the water by allowing some
> > operations on "short" varlenas, and there's really nothing to prevent
> > a given datatype from going further.  The OID problem you mentioned
> > would presumably be solved by hard-coding the OIDs for any built-in,
> > privileged compression methods.
> 
> Stupid question, but what do you mean by "short" varlenas?

Those are varlenas with 1-byte header rather than the standard 4-byte
header.

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Commits

  1. docs: Update TOAST storage docs for configurable compression.

  2. Further tweaking of pg_dump's handling of default_toast_compression.

  3. Fix interaction of TOAST compression with expression indexes.

  4. Tidy up more loose ends related to configurable TOAST compression.

  5. Short-circuit slice requests that are for more than the object's size.

  6. Mostly-cosmetic adjustments of TOAST-related macros.

  7. Remove useless configure probe for <lz4/lz4.h>.

  8. Error on invalid TOAST compression in CREATE or ALTER TABLE.

  9. docs: Fix omissions related to configurable TOAST compression.

  10. More code cleanup for configurable TOAST compression.

  11. Bring configure support for LZ4 up to snuff.

  12. Make compression.sql regression test independent of default.

  13. Use valid compression method in brin_form_tuple

  14. Fix up pg_dump's handling of per-attribute compression options.

  15. Allow configurable LZ4 TOAST compression.

  16. Fix inconsistencies in the code

  17. Mostly-cosmetic improvements in memory chunk header alignment coding.

  18. Allow numeric to use a more compact, 2-byte header in many cases.