Re: [HACKERS] Custom compression methods

Dilip Kumar <dilipbalaut@gmail.com>

From: Dilip Kumar <dilipbalaut@gmail.com>

To: Robert Haas <robertmhaas@gmail.com>

Cc: Andres Freund <andres@anarazel.de>, Justin Pryzby <pryzby@telsasoft.com>, Tomas Vondra <tomas.vondra@2ndquadrant.com>, Alexander Korotkov <a.korotkov@postgrespro.ru>, David Steele <david@pgmasters.net>, Ildus Kurbangaliev <i.kurbangaliev@gmail.com>, Dmitry Dolgov <9erthalion6@gmail.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>

Date: 2021-03-17T11:40:44Z

Lists: pgsql-hackers

Attachments

v36-0002-Expand-the-external-data-before-forming-the-tupl.patch (text/x-patch) patch v36-0002
v36-0004-Add-default_toast_compression-GUC.patch (text/x-patch) patch v36-0004
v36-0001-Get-datum-from-tuple-which-doesn-t-contain-exter.patch (text/x-patch) patch v36-0001
v36-0003-Built-in-compression-method.patch (text/x-patch) patch v36-0003
v36-0005-Alter-table-set-compression.patch (text/x-patch) patch v36-0005
v36-0006-default-to-with-lz4.patch (text/x-patch) patch v36-0006

On Wed, Mar 17, 2021 at 1:01 AM Robert Haas <robertmhaas@gmail.com> wrote:
>
Please find the updated version of the patch set.
Changes:
0001:
- Added comment for PG_RETURN_HEAPTUPLEHEADER_RAW().
- Added assert in heap_copy_tuple_as_raw_datum() that there should be
no external data in tuple

0002:
- Wrapper over heap_form_tuple and used in ExecEvalRow() and
ExecEvalFieldStoreForm()

0003:
1. Used 2 bits of varatt_external.va_extsize to store compression method
2. Removed logic of compressing.
3. Some other minor cleanup

0004: some minor cleanup
0005: Removed code of recompress from ATRewriteTuple, now only
recompress happens in Vacuum FULL and CLUSTER. So that there is at
least some way to recompress.  Also, here we don't need to take the
overhead of deforming the tuple so it should not affect performance
also in unrelated cases.  We will have to pay the penalty if we need
to decompress a lot of attributes.
0006: removed as suggested by Robert
0007: now 0006, only for CFbot

-- I still have the HIDE_TOAST_COMPRESSION code in the patch set do we
think we need to remove?  I don't mind keeping it as it is because we
can avoid changes in unrelated regression tests.

-- 
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

Commits

docs: Update TOAST storage docs for configurable compression.
- e8c435a824e1 14.0 landed
Further tweaking of pg_dump's handling of default_toast_compression.
- 54bb91c30e39 14.0 landed
Fix interaction of TOAST compression with expression indexes.
- 5db1fd7823a1 14.0 landed
Tidy up more loose ends related to configurable TOAST compression.
- e5595de03ec6 14.0 landed
Short-circuit slice requests that are for more than the object's size.
- 063dd37ebc76 14.0 landed
Mostly-cosmetic adjustments of TOAST-related macros.
- aeb1631ed207 14.0 landed
Remove useless configure probe for <lz4/lz4.h>.
- 2c75f8a612b2 14.0 landed
Error on invalid TOAST compression in CREATE or ALTER TABLE.
- a4d5284a10b5 14.0 landed
docs: Fix omissions related to configurable TOAST compression.
- 24f0e395ac58 14.0 landed
More code cleanup for configurable TOAST compression.
- 226e2be3876d 14.0 landed
Bring configure support for LZ4 up to snuff.
- 4d399a6fbeb7 14.0 landed
Make compression.sql regression test independent of default.
- fd1ac9a54896 14.0 landed
Use valid compression method in brin_form_tuple
- 882b2cdc08c4 14.0 landed
Fix up pg_dump's handling of per-attribute compression options.
- aa25d1089ac0 14.0 landed
Allow configurable LZ4 TOAST compression.
- bbe0a81db69b 14.0 landed
Fix inconsistencies in the code
- 6b8548964bcc 13.0 cited
Mostly-cosmetic improvements in memory chunk header alignment coding.
- f65d21b25808 11.0 cited
Allow numeric to use a more compact, 2-byte header in many cases.
- 145343534c15 9.1.0 cited