Horton

Re: [HACKERS] Custom compression methods

Dilip Kumar <dilipbalaut@gmail.com>

View thread View on postgr.es
From: Dilip Kumar <dilipbalaut@gmail.com>
To: Justin Pryzby <pryzby@telsasoft.com>
Cc: Robert Haas <robertmhaas@gmail.com>, Tomas Vondra <tomas.vondra@2ndquadrant.com>, Alexander Korotkov <a.korotkov@postgrespro.ru>, David Steele <david@pgmasters.net>, Ildus Kurbangaliev <i.kurbangaliev@gmail.com>, Dmitry Dolgov <9erthalion6@gmail.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2021-03-08T10:02:39Z
Lists: pgsql-hackers

Attachments

On Sun, Mar 7, 2021 at 1:27 AM Justin Pryzby <pryzby@telsasoft.com> wrote:
>
> On Sat, Mar 06, 2021 at 08:59:16PM +0530, Dilip Kumar wrote:
> > - Alter table set compression, will not rewrite the old data, so only
> > the new tuple will be compressed with the new compression method.
> > - No preserve.
>
> In this patch, SET default_toast_compression=lz4 "works" even if without-lz4,
> but then CREATE TABLE fails.  You should either allow table creation (as
> above), or check in check_default_toast_compression() if lz4 is enabled.
> Its comment about "catalog access" is incorrect now.

As of now I have made GUC behavior similar to the CREATE TABLE, in
both case it will throw an error if it is not compiled with lz4
method.

>
> +                   if (strcmp(def->compression, newdef->compression))
> != 0
>
> + * NULL for non varlena type or the uncompressed data.
> remove "the"
>
> +        * InvalidOid for the plain/external storage otherwise default
> remove "the"
>
> +      behavior is to exclude compression methods, resulting in the columns
> remove "the"
>
>
> + attcompression and attstorage for the respective index attribute if ... the respective input values are
> say "and/or attstorage"
>
> + If this variable is set to <literal>true</literal>, column's [...]
> I wrote this, but I guess it should say: columns'
>
> I think you could also say either of these:
>  .. column compression method details are not displayed.
>  .. details of column compression are not displayed.

I have fixed the above comments,  and also some other minor fixup.

So now only pending point is,  how do we handle the upgrade when you
are upgrading from --with-lz4 to --without-lz4 binary and a couple of
options discussed here are
a) Should we allow table creation with lz4 even if it is compiled
--without-lz4?  In case of xml we always allow table creation even if
it is compiled --wthout-libxml
b) Instead of allowing this always, only allow during binary upgrade.

With this we will be able to make binary upgrades successful.
However, if users have to access the data compiled with the lz4 then
then need to enable the lz4 support.  But I think that is true with
any case for example with a hot standby setup if the user configures
the standby without lz4 then also for accessing the lz4 compressed
data user needs to enable the lz4 support.

--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

Commits

  1. docs: Update TOAST storage docs for configurable compression.

  2. Further tweaking of pg_dump's handling of default_toast_compression.

  3. Fix interaction of TOAST compression with expression indexes.

  4. Tidy up more loose ends related to configurable TOAST compression.

  5. Short-circuit slice requests that are for more than the object's size.

  6. Mostly-cosmetic adjustments of TOAST-related macros.

  7. Remove useless configure probe for <lz4/lz4.h>.

  8. Error on invalid TOAST compression in CREATE or ALTER TABLE.

  9. docs: Fix omissions related to configurable TOAST compression.

  10. More code cleanup for configurable TOAST compression.

  11. Bring configure support for LZ4 up to snuff.

  12. Make compression.sql regression test independent of default.

  13. Use valid compression method in brin_form_tuple

  14. Fix up pg_dump's handling of per-attribute compression options.

  15. Allow configurable LZ4 TOAST compression.

  16. Fix inconsistencies in the code

  17. Mostly-cosmetic improvements in memory chunk header alignment coding.

  18. Allow numeric to use a more compact, 2-byte header in many cases.