Horton

Re: [HACKERS] Custom compression methods

Dilip Kumar <dilipbalaut@gmail.com>

View thread View on postgr.es
From: Dilip Kumar <dilipbalaut@gmail.com>
To: Robert Haas <robertmhaas@gmail.com>
Cc: Justin Pryzby <pryzby@telsasoft.com>, Tomas Vondra <tomas.vondra@2ndquadrant.com>, Alexander Korotkov <a.korotkov@postgrespro.ru>, David Steele <david@pgmasters.net>, Ildus Kurbangaliev <i.kurbangaliev@gmail.com>, Dmitry Dolgov <9erthalion6@gmail.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2021-02-27T02:14:18Z
Lists: pgsql-hackers 
On Fri, Feb 26, 2021 at 8:10 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Sun, Feb 21, 2021 at 5:33 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
>
> Based on offlist discussion with Robert, I have done further analysis
> of the composite type data.  So the Idea is that I have analyzed all
> the callers of
> HeapTupleGetDatum and HeapTupleHeaderGetDatum and divide them into two
> category 1) Callers which are forming the tuple from values that can
> not have compressed/external data.
> 2) Callers which can have external/compressed data

I just realized that there is one more function
"heap_copy_tuple_as_datum" which is flattening the tuple based on the
HeapTupleHasExternal check, so I think I will have to analyze the
caller of this function as well and need to do a similar analysis,
although there are just a few callers for this.  And, I think the fix
in ExecEvalConvertRowtype is wrong, we will have to do something for
the compressed type here as well.  I am not sure what is the best way
to fix it because we are directly getting the input tuple so we can
not put an optimization of dettoasting before forming the tuple.  We
might detoast in execute_attr_map_tuple, when the source and target
row types are different because we are anyway deforming and processing
each filed in that function but the problem is execute_attr_map_tuple
is used at multiple places but for that, we can make another version
of this function which actually detoast along with conversion and use
that in ExecEvalConvertRowtype.  But if there is no tuple conversion
needed then we directly use heap_copy_tuple_as_datum and in that case,
there is no deforming at all so maybe, in this case, we can not do
anything but I think ExecEvalConvertRowtype should not be the very
common path.

--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

Commits

  1. docs: Update TOAST storage docs for configurable compression.

  2. Further tweaking of pg_dump's handling of default_toast_compression.

  3. Fix interaction of TOAST compression with expression indexes.

  4. Tidy up more loose ends related to configurable TOAST compression.

  5. Short-circuit slice requests that are for more than the object's size.

  6. Mostly-cosmetic adjustments of TOAST-related macros.

  7. Remove useless configure probe for <lz4/lz4.h>.

  8. Error on invalid TOAST compression in CREATE or ALTER TABLE.

  9. docs: Fix omissions related to configurable TOAST compression.

  10. More code cleanup for configurable TOAST compression.

  11. Bring configure support for LZ4 up to snuff.

  12. Make compression.sql regression test independent of default.

  13. Use valid compression method in brin_form_tuple

  14. Fix up pg_dump's handling of per-attribute compression options.

  15. Allow configurable LZ4 TOAST compression.

  16. Fix inconsistencies in the code

  17. Mostly-cosmetic improvements in memory chunk header alignment coding.

  18. Allow numeric to use a more compact, 2-byte header in many cases.