Horton

Re: [HACKERS] Custom compression methods

Alvaro Herrera <alvherre@alvh.no-ip.org>

View thread View on postgr.es
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>
Cc: Ildus Kurbangaliev <i.kurbangaliev@postgrespro.ru>, pgsql-hackers@postgresql.org, Ildar Musin <i.musin@postgrespro.ru>
Date: 2017-12-01T19:38:42Z
Lists: pgsql-hackers 
Tomas Vondra wrote:

> On 11/30/2017 09:51 PM, Alvaro Herrera wrote:

> > Just passing by, but wouldn't this fit in the ACCESS METHOD group of
> > commands?  So this could be simplified down to
> > CREATE ACCESS METHOD ts1 TYPE COMPRESSION
> > we have that for indexes and there are patches flying for heap storage,
> > sequences, etc.
> 
> I think that would conflate two very different concepts. In my mind,
> access methods define how rows are stored.

In mine, they define how things are accessed (i.e. more general than
what you're thinking).  We *currently* use them to store rows [in
indexes], but there is no reason why we couldn't expand that.

So we group access methods in "types"; the current type we have is for
indexes, and methods in that type define how are indexes accessed.  This
new type would indicate how would values be compressed.  I disagree that
there is no parallel there.

I'm trying to avoid pointless proliferation of narrowly defined DDL
commands.

> Furthermore, the "TYPE" in CREATE COMPRESSION method was meant to
> restrict the compression algorithm to a particular data type (so, if it
> relies on tsvector, you can't apply it to text columns).

Yes, of course.  I'm saying that the "datatype" property of a
compression access method would be declared somewhere else, not in the
TYPE clause of the CREATE ACCESS METHOD command.  Perhaps it makes sense
to declare that a certain compression access method is good only for a
certain data type, and then you can put that in the options clause,
"CREATE ACCESS METHOD hyperz TYPE COMPRESSION WITH (type = tsvector)".
But many compression access methods would be general in nature and so
could be used for many datatypes (say, snappy).

To me it makes sense to say "let's create this method which is for data
compression" (CREATE ACCESS METHOD hyperz TYPE COMPRESSION) followed by
either "let's use this new compression method for the type tsvector"
(ALTER TYPE tsvector SET COMPRESSION hyperz) or "let's use this new
compression method for the column tc" (ALTER TABLE ALTER COLUMN tc SET
COMPRESSION hyperz).

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Commits

  1. docs: Update TOAST storage docs for configurable compression.

  2. Further tweaking of pg_dump's handling of default_toast_compression.

  3. Fix interaction of TOAST compression with expression indexes.

  4. Tidy up more loose ends related to configurable TOAST compression.

  5. Short-circuit slice requests that are for more than the object's size.

  6. Mostly-cosmetic adjustments of TOAST-related macros.

  7. Remove useless configure probe for <lz4/lz4.h>.

  8. Error on invalid TOAST compression in CREATE or ALTER TABLE.

  9. docs: Fix omissions related to configurable TOAST compression.

  10. More code cleanup for configurable TOAST compression.

  11. Bring configure support for LZ4 up to snuff.

  12. Make compression.sql regression test independent of default.

  13. Use valid compression method in brin_form_tuple

  14. Fix up pg_dump's handling of per-attribute compression options.

  15. Allow configurable LZ4 TOAST compression.

  16. Fix inconsistencies in the code

  17. Mostly-cosmetic improvements in memory chunk header alignment coding.

  18. Allow numeric to use a more compact, 2-byte header in many cases.