Horton

Re: [HACKERS] Custom compression methods

Alexander Korotkov <a.korotkov@postgrespro.ru>

View thread View on postgr.es
From: Alexander Korotkov <a.korotkov@postgrespro.ru>
To: Konstantin Knizhnik <k.knizhnik@postgrespro.ru>
Cc: Ildus Kurbangaliev <i.kurbangaliev@postgrespro.ru>, Ildar Musin <i.musin@postgrespro.ru>, Robert Haas <robertmhaas@gmail.com>, Tomas Vondra <tomas.vondra@2ndquadrant.com>, Alvaro Herrera <alvherre@2ndquadrant.com>, Евгений Шишкин <itparanoia@gmail.com>, Andres Freund <andres@anarazel.de>, Oleg Bartunov <obartunov@gmail.com>, Craig Ringer <craig@2ndquadrant.com>, Peter Eisentraut <peter.eisentraut@2ndquadrant.com>, PostgreSQL Hackers <pgsql-hackers@postgresql.org>, Chapman Flack <chap@anastigmatix.net>
Date: 2018-04-24T11:05:20Z
Lists: pgsql-hackers 
On Mon, Apr 23, 2018 at 7:34 PM, Konstantin Knizhnik <
k.knizhnik@postgrespro.ru> wrote:

> IMHO end-user do not have skills and time to create their own compression
> algorithms. And without knowledge of specific of particular data set,
> it is very hard to implement something more efficient than universal
> compression library.
> But if you think that it is not a right place and time to discuss it, I do
> not insist.
>

For sure, end-users wouldn't implement own compression algorithms.
In the same way as end-users wouldn't implement custom datatypes,
operator classes, procedural language handlers etc.  But those are
useful extension mechanisms which pass test of time.  And extension
developers use them.


> But in any case, I think that it will be useful to provide some more
> examples of custom compression API usage.
> From my point of view the most useful will be integration with zstd.
> But if it is possible to find some example of data-specific compression
> algorithms which show better results than universal compression,
> it will be even more impressive.
>

Yes, this patch definitely lacks of good usage example.  That may
lead to some misunderstanding of its purpose.  Good use-cases
should be shown before we can consider committing this.  I think
Ildus should try to implement at least custom dictionary compression
method where dictionary is specified by user in parameters.

------
Alexander Korotkov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

Commits

  1. docs: Update TOAST storage docs for configurable compression.

  2. Further tweaking of pg_dump's handling of default_toast_compression.

  3. Fix interaction of TOAST compression with expression indexes.

  4. Tidy up more loose ends related to configurable TOAST compression.

  5. Short-circuit slice requests that are for more than the object's size.

  6. Mostly-cosmetic adjustments of TOAST-related macros.

  7. Remove useless configure probe for <lz4/lz4.h>.

  8. Error on invalid TOAST compression in CREATE or ALTER TABLE.

  9. docs: Fix omissions related to configurable TOAST compression.

  10. More code cleanup for configurable TOAST compression.

  11. Bring configure support for LZ4 up to snuff.

  12. Make compression.sql regression test independent of default.

  13. Use valid compression method in brin_form_tuple

  14. Fix up pg_dump's handling of per-attribute compression options.

  15. Allow configurable LZ4 TOAST compression.

  16. Fix inconsistencies in the code

  17. Mostly-cosmetic improvements in memory chunk header alignment coding.

  18. Allow numeric to use a more compact, 2-byte header in many cases.