Horton

Re: [HACKERS] Custom compression methods

Justin Pryzby <pryzby@telsasoft.com>

View thread View on postgr.es
From: Justin Pryzby <pryzby@telsasoft.com>
To: Dilip Kumar <dilipbalaut@gmail.com>
Cc: Robert Haas <robertmhaas@gmail.com>, Tomas Vondra <tomas.vondra@2ndquadrant.com>, Alexander Korotkov <a.korotkov@postgrespro.ru>, David Steele <david@pgmasters.net>, Ildus Kurbangaliev <i.kurbangaliev@gmail.com>, Dmitry Dolgov <9erthalion6@gmail.com>, pgsql-hackers@lists.postgresql.org
Date: 2021-03-06T19:57:51Z
Lists: pgsql-hackers 
On Sat, Mar 06, 2021 at 08:59:16PM +0530, Dilip Kumar wrote:
> - Alter table set compression, will not rewrite the old data, so only
> the new tuple will be compressed with the new compression method.
> - No preserve.

+1, this simplifies things.  If someone *wants* to rewrite the table, they can
VACUUM FULL, CLUSTER, or dump+restore.

I checked that it's possible to do simple column manipulations on columns
written with-lz4 with binaries built without-lz4:
 - is null
 - texteq if length differs
 - explain analyze

If I pg_upgrade from an binary with-lz4 to one without-lz4, it fails
while restoring the schema, after running check, which is bad:
| pg_restore: error: could not execute query: ERROR:  not built with lz4 support
|CREATE TABLE "public"."a" (
|    "t" "text" COMPRESSION lz4,

For comparison, upgrading from binaries with-libxml to binaries without-libxml
actualy passes pg_upgrade.

It's arguable which behavior is desirable:
 - allow CREATE TABLE(..COMPRESSION lz4) during pg_upgrade;
 - allow CREATE TABLE(..COMPRESSION lz4) always.  This has the advantage that
   GetAttributeCompression() doesn't have conditional compilation.  This seems
   to be parallel to the libxml case - apparently, it's possible to create an
   XML column, but not insert into it.
 - abort pg_upgrade --check if the old cluster has lz4 and the new one doesn't,
   if there are any lz4 compressed columns.  This avoids the possibilty of
running an upgrade to binaries without lz4, starting a new cluster (which
leaves the old cluster unsafe to start if --link was used), and then the new
cluster may even appear to work, until an LZ4 column is accessed in a
nontrivial way.  It has the disadvantage that there's no obvious parallel in
pg_upgrade (checksums and xml are the closest?).  And the disadvantage that
some people might *want* the upgrade to succeed in that case to then recompile
with lz4 afterwards.

In this patch, SET default_toast_compression=lz4 "works" even if without-lz4,
but then CREATE TABLE fails.  You should either allow table creation (as
above), or check in check_default_toast_compression() if lz4 is enabled.
Its comment about "catalog access" is incorrect now.

Now, I wonder if default_toast_compression should be a GUC, or a reloption.
An obvious advantage of being a GUC is that regression tests are trivial with
make installcheck.

Some minor fixes:

+                   if (strcmp(def->compression, newdef->compression))
!= 0

+ * NULL for non varlena type or the uncompressed data.
remove "the"

+        * InvalidOid for the plain/external storage otherwise default
remove "the"

+      behavior is to exclude compression methods, resulting in the columns
remove "the"


+ attcompression and attstorage for the respective index attribute if ... the respective input values are
say "and/or attstorage"

+ If this variable is set to <literal>true</literal>, column's [...]
I wrote this, but I guess it should say: columns'

I think you could also say either of these:
 .. column compression method details are not displayed.
 .. details of column compression are not displayed.

-- 
Justin

Commits

  1. docs: Update TOAST storage docs for configurable compression.

  2. Further tweaking of pg_dump's handling of default_toast_compression.

  3. Fix interaction of TOAST compression with expression indexes.

  4. Tidy up more loose ends related to configurable TOAST compression.

  5. Short-circuit slice requests that are for more than the object's size.

  6. Mostly-cosmetic adjustments of TOAST-related macros.

  7. Remove useless configure probe for <lz4/lz4.h>.

  8. Error on invalid TOAST compression in CREATE or ALTER TABLE.

  9. docs: Fix omissions related to configurable TOAST compression.

  10. More code cleanup for configurable TOAST compression.

  11. Bring configure support for LZ4 up to snuff.

  12. Make compression.sql regression test independent of default.

  13. Use valid compression method in brin_form_tuple

  14. Fix up pg_dump's handling of per-attribute compression options.

  15. Allow configurable LZ4 TOAST compression.

  16. Fix inconsistencies in the code

  17. Mostly-cosmetic improvements in memory chunk header alignment coding.

  18. Allow numeric to use a more compact, 2-byte header in many cases.