Horton

Re: [HACKERS] Custom compression methods

Dilip Kumar <dilipbalaut@gmail.com>

View thread View on postgr.es
From: Dilip Kumar <dilipbalaut@gmail.com>
To: Justin Pryzby <pryzby@telsasoft.com>
Cc: Robert Haas <robertmhaas@gmail.com>, Tomas Vondra <tomas.vondra@2ndquadrant.com>, Alexander Korotkov <a.korotkov@postgrespro.ru>, David Steele <david@pgmasters.net>, Ildus Kurbangaliev <i.kurbangaliev@gmail.com>, Dmitry Dolgov <9erthalion6@gmail.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2021-03-02T08:01:15Z
Lists: pgsql-hackers 
On Mon, Mar 1, 2021 at 8:53 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> Now, I think the only pending thing is related to the expandedrecord,
> basically, currently, we have detoasted the compressed filed only in
> expanded_record_set_field_internal function.  I am still not
> completely sure that for the built-in types do we need to do something
> for expanded_record_set_tuple and expanded_record_set_field or not,  I
> mean in these functions do we only expand the external to survive the
> COMMIT/ROLLBACK or do we also expand it send it to some target table
> like we do in expanded_record_set_field_internal.

I have done further analysis of the compressed field in the
expandedrecord. My observation is that only in
expanded_record_set_field_internal we unconditionally pass true and
only when it is called from ER_get_flat_size.  In all the other
functions (expanded_record_set_tuple and expanded_record_set_fields)
we only pass expand_external to true if estate->atomic is not set.
And, the estate->atomic is set to false only if we are executing the
anonymous block from a transaction block (there might be another way
to have estate->atomic as false).  But the point is that the
flattening in these two functions are conditional which means we can
not use these expanded records to form some kind of row, otherwise, we
can not have the conditional flattening based on the way how the PL
block is being executed, so I think this proves Robert's point that we
are expanding this only for surviving the commit/rollback inside the
PL block.  That means for the built-in types, decompression in
expanded_record_set_field_internal should be sufficient and that was
already done in my latest version of patch v29-0001.

--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

Commits

  1. docs: Update TOAST storage docs for configurable compression.

  2. Further tweaking of pg_dump's handling of default_toast_compression.

  3. Fix interaction of TOAST compression with expression indexes.

  4. Tidy up more loose ends related to configurable TOAST compression.

  5. Short-circuit slice requests that are for more than the object's size.

  6. Mostly-cosmetic adjustments of TOAST-related macros.

  7. Remove useless configure probe for <lz4/lz4.h>.

  8. Error on invalid TOAST compression in CREATE or ALTER TABLE.

  9. docs: Fix omissions related to configurable TOAST compression.

  10. More code cleanup for configurable TOAST compression.

  11. Bring configure support for LZ4 up to snuff.

  12. Make compression.sql regression test independent of default.

  13. Use valid compression method in brin_form_tuple

  14. Fix up pg_dump's handling of per-attribute compression options.

  15. Allow configurable LZ4 TOAST compression.

  16. Fix inconsistencies in the code

  17. Mostly-cosmetic improvements in memory chunk header alignment coding.

  18. Allow numeric to use a more compact, 2-byte header in many cases.