Horton

Re: [HACKERS] Custom compression methods

Dilip Kumar <dilipbalaut@gmail.com>

View thread View on postgr.es
From: Dilip Kumar <dilipbalaut@gmail.com>
To: Andres Freund <andres@anarazel.de>
Cc: Robert Haas <robertmhaas@gmail.com>, Justin Pryzby <pryzby@telsasoft.com>, Tomas Vondra <tomas.vondra@2ndquadrant.com>, Alexander Korotkov <a.korotkov@postgrespro.ru>, David Steele <david@pgmasters.net>, Ildus Kurbangaliev <i.kurbangaliev@gmail.com>, Dmitry Dolgov <9erthalion6@gmail.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2021-03-16T05:46:08Z
Lists: pgsql-hackers 
On Tue, Mar 16, 2021 at 4:28 AM Andres Freund <andres@anarazel.de> wrote:

Replying to some of the comments..

> - Is nodeModifyTable.c really the right place for the logic around
>   CompareCompressionMethodAndDecompress()?  And is doing it in every
>   place that does "user initiated" inserts really the right way? Why
>   isn't this done on the tuptoasting level?

I think if we do in tuptoasting level then it will be even costlier
because in nodeModifyTable.c at least many time we will get the
virtual tuple e.g. if a user is directly inserting the tuple but once
we go down to tuptoasting level by then we will always get the
HeapTuple and we will have to deform in every case where tupdesc has
any varlena because we don't have any flag in the tuple header to tell
us whether there are any compressed data or not. In the below
thread[1] we have considered these 2 approaches and basically, in
unrelated paths like pg_bench, we did not see any performance
regression with any of those approaches.

[1] https://www.postgresql.org/message-id/CAFiTN-vcbfy5ScKVUp16c1N_wzP0RL6EkPBAg_Jm3eDK0ftO5Q%40mail.gmail.com

> > I'm open to being convinced that we don't need to do either of these
> > things, and that the cost of iterating over all varlenas in the tuple
> > is not so bad as to preclude doing things as you have them here. But,
> > I'm afraid it's going to be too expensive.
>
> I mean, I would just define several of those places away by not caring
> about tuples in a different compressino formation ending up in a
> table...

I am just wondering that why we don't need to process in case of
storage change, I mean if the target table has the attribute storage
as external and if there are some compressed data coming from the
source table then we will be inserting those compressed data as it is
in the target attribute without externalizing.  Maybe it is done to
avoid such performance impacts?  Well, we can do the same for the
compression also and just provide some mechanism to recompress maybe
in vacuum full/cluster.

--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

Commits

  1. docs: Update TOAST storage docs for configurable compression.

  2. Further tweaking of pg_dump's handling of default_toast_compression.

  3. Fix interaction of TOAST compression with expression indexes.

  4. Tidy up more loose ends related to configurable TOAST compression.

  5. Short-circuit slice requests that are for more than the object's size.

  6. Mostly-cosmetic adjustments of TOAST-related macros.

  7. Remove useless configure probe for <lz4/lz4.h>.

  8. Error on invalid TOAST compression in CREATE or ALTER TABLE.

  9. docs: Fix omissions related to configurable TOAST compression.

  10. More code cleanup for configurable TOAST compression.

  11. Bring configure support for LZ4 up to snuff.

  12. Make compression.sql regression test independent of default.

  13. Use valid compression method in brin_form_tuple

  14. Fix up pg_dump's handling of per-attribute compression options.

  15. Allow configurable LZ4 TOAST compression.

  16. Fix inconsistencies in the code

  17. Mostly-cosmetic improvements in memory chunk header alignment coding.

  18. Allow numeric to use a more compact, 2-byte header in many cases.