Horton

Re: [HACKERS] Custom compression methods

Justin Pryzby <pryzby@telsasoft.com>

View thread View on postgr.es
From: Justin Pryzby <pryzby@telsasoft.com>
To: Andres Freund <andres@anarazel.de>
Cc: Robert Haas <robertmhaas@gmail.com>, Dilip Kumar <dilipbalaut@gmail.com>, Tomas Vondra <tomas.vondra@2ndquadrant.com>, Alexander Korotkov <a.korotkov@postgrespro.ru>, David Steele <david@pgmasters.net>, Ildus Kurbangaliev <i.kurbangaliev@gmail.com>, Dmitry Dolgov <9erthalion6@gmail.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2021-03-16T05:48:19Z
Lists: pgsql-hackers 
I'm a minor contributor now to a couple bits of this patch set, but I can
answer a couple of these points.

On Mon, Mar 15, 2021 at 03:58:35PM -0700, Andres Freund wrote:
> Comments about 0003:
> - why is HIDE_TOAST_COMPRESSION useful? Doesn't quite seem to be
>   comparable to HIDE_TABLEAM?

That was my idea and implementation.
It's because until 3 weeks ago, the patchset supported a "plugable compression
API" like CREATE ACCESS METHOD, a suggestion from Alvaro to avoid making a new
table and everything involved just for a few rows).  Now, the patch is limited
to lz4, and the "pluggable compression APIs" isn't included in the latest
patchsets.

> Comments about 0005:
> - I'm personally not really convinced tracking the compression type in
>   pg_attribute the way you do is really worth it (. Especially given
>   that it's right now only about new rows anyway.  Seems like it'd be
>   easier to just treat it as a default for new rows, and dispense with
>   all the logic around mismatching compression types etc?

I made the half-serious suggestion to make it a per-relation relopt.
That would allow implementing pg_dump --no-toast-compression, to allow
restoring a dump from a server with LZ4 tables to a server --without-lz4.
Similar to --no-tablespaces.

That would also avoid adding a compression column in \d (which avoids the need
for HIDE_TOAST_COMPRESSION).

> > I'm open to being convinced that we don't need to do either of these
> > things, and that the cost of iterating over all varlenas in the tuple
> > is not so bad as to preclude doing things as you have them here. But,
> > I'm afraid it's going to be too expensive.
>
> I mean, I would just define several of those places away by not caring
> about tuples in a different compressino formation ending up in a
> table...

If I understand you right, this is because it's desirable to allow 1) migrating
existing data from pglz to lz4; 2) also allow moving away from lz4, if need be.

-- 
Justin

Commits

  1. docs: Update TOAST storage docs for configurable compression.

  2. Further tweaking of pg_dump's handling of default_toast_compression.

  3. Fix interaction of TOAST compression with expression indexes.

  4. Tidy up more loose ends related to configurable TOAST compression.

  5. Short-circuit slice requests that are for more than the object's size.

  6. Mostly-cosmetic adjustments of TOAST-related macros.

  7. Remove useless configure probe for <lz4/lz4.h>.

  8. Error on invalid TOAST compression in CREATE or ALTER TABLE.

  9. docs: Fix omissions related to configurable TOAST compression.

  10. More code cleanup for configurable TOAST compression.

  11. Bring configure support for LZ4 up to snuff.

  12. Make compression.sql regression test independent of default.

  13. Use valid compression method in brin_form_tuple

  14. Fix up pg_dump's handling of per-attribute compression options.

  15. Allow configurable LZ4 TOAST compression.

  16. Fix inconsistencies in the code

  17. Mostly-cosmetic improvements in memory chunk header alignment coding.

  18. Allow numeric to use a more compact, 2-byte header in many cases.