Horton

Re: [HACKERS] Custom compression methods

Robert Haas <robertmhaas@gmail.com>

View thread View on postgr.es
From: Robert Haas <robertmhaas@gmail.com>
To: Konstantin Knizhnik <k.knizhnik@postgrespro.ru>
Cc: Alexander Korotkov <a.korotkov@postgrespro.ru>, Ildus Kurbangaliev <i.kurbangaliev@postgrespro.ru>, Ildar Musin <i.musin@postgrespro.ru>, Tomas Vondra <tomas.vondra@2ndquadrant.com>, Alvaro Herrera <alvherre@2ndquadrant.com>, Евгений Шишкин <itparanoia@gmail.com>, Andres Freund <andres@anarazel.de>, Oleg Bartunov <obartunov@gmail.com>, Craig Ringer <craig@2ndquadrant.com>, Peter Eisentraut <peter.eisentraut@2ndquadrant.com>, PostgreSQL Hackers <pgsql-hackers@postgresql.org>, Chapman Flack <chap@anastigmatix.net>
Date: 2018-06-19T14:57:37Z
Lists: pgsql-hackers 
On Mon, Apr 23, 2018 at 12:34 PM, Konstantin Knizhnik
<k.knizhnik@postgrespro.ru> wrote:
> May be. But in any cases, there are several direction where compression can
> be used:
> - custom compression algorithms
> - libpq compression
> - page level compression
> ...
>
> and  them should be somehow finally "married" with each other.

I agree that we should try to avoid multiplying the number of
compression-related APIs.  Ideally there should be one API for
registering a compression algorithms, and then there can be different
methods of selecting that compression algorithm depending on the
purpose for which it will be used.  For instance, you could select a
column compression format using some variant of ALTER TABLE ... ALTER
COLUMN, but you would obviously use some other method to select the
WAL compression format.  However, it's a little unclear to me how we
would actually make the idea of a single API work.  For column
compression, we need everything to be accessible through the catalogs.
For something like WAL compression, we need it to be completely
independent of the catalogs.  Those things are opposites, so a single
API can't have both properties.  Maybe there can be some pieces
shared, but as much as I'd like it to be otherwise, it doesn't seem
possible to share it completely.

I also agree with Ildus and Alexander that we cannot and should not
try to solve every problem in one patch.  Rather, we should just think
ahead, so that we make as much of what goes into this patch reusable
in the future as we can.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Commits

  1. docs: Update TOAST storage docs for configurable compression.

  2. Further tweaking of pg_dump's handling of default_toast_compression.

  3. Fix interaction of TOAST compression with expression indexes.

  4. Tidy up more loose ends related to configurable TOAST compression.

  5. Short-circuit slice requests that are for more than the object's size.

  6. Mostly-cosmetic adjustments of TOAST-related macros.

  7. Remove useless configure probe for <lz4/lz4.h>.

  8. Error on invalid TOAST compression in CREATE or ALTER TABLE.

  9. docs: Fix omissions related to configurable TOAST compression.

  10. More code cleanup for configurable TOAST compression.

  11. Bring configure support for LZ4 up to snuff.

  12. Make compression.sql regression test independent of default.

  13. Use valid compression method in brin_form_tuple

  14. Fix up pg_dump's handling of per-attribute compression options.

  15. Allow configurable LZ4 TOAST compression.

  16. Fix inconsistencies in the code

  17. Mostly-cosmetic improvements in memory chunk header alignment coding.

  18. Allow numeric to use a more compact, 2-byte header in many cases.