Horton

Re: [HACKERS] Custom compression methods

Justin Pryzby <pryzby@telsasoft.com>

View thread View on postgr.es
From: Justin Pryzby <pryzby@telsasoft.com>
To: Dilip Kumar <dilipbalaut@gmail.com>
Cc: Robert Haas <robertmhaas@gmail.com>, Tomas Vondra <tomas.vondra@2ndquadrant.com>, Alexander Korotkov <a.korotkov@postgrespro.ru>, David Steele <david@pgmasters.net>, Ildus Kurbangaliev <i.kurbangaliev@gmail.com>, Dmitry Dolgov <9erthalion6@gmail.com>, pgsql-hackers@lists.postgresql.org
Date: 2021-03-11T21:24:05Z
Lists: pgsql-hackers 
On Thu, Mar 11, 2021 at 10:07:30AM +0530, Dilip Kumar wrote:
>  On Thu, Mar 11, 2021 at 8:50 AM Justin Pryzby <pryzby@telsasoft.com> wrote:
> >
> > Looking at v23-0002-alter-table-set-compression, ATRewriteTable() was calling
> > CompareCompressionMethodAndDecompress().
> 
> While changing the compression method user might be just interested
> to compress the future tuple with the new compression method but
> doesn't want to rewrite all the old tuple.  So IMHO without any option
> just force rewrite whenever changing the compression method doesn't look that
> great.

I mean to keep the current behavior where SET is only a catalog change.
But I'm comparing with earlier implementation.

Does your new patch avoid recompressing things if the compression is unchanged?

> > I think what's wanted here is that decompression should only happen when the
> > tuple uses a different compression than the column's currently set compression.
> > So there's no overhead in the usual case.  I guess CLUSTER and INSERT SELECT
> > should do the same.
> >
> > This is important to allow someone to get rid of LZ4 compression, if they want
> > to get rid of that dependency.
> >
> > But it's also really desirable for admins to be able to "migrate" existing
> > data.  People will want to test this, and I guess INSERT SELECT and CLUSTER are
> > the obvious ways.
> 
> For INSERT SELECT we were already doing in the older version so we can
> include that code here, we will also have to include the patches for
> decompressing data before forming the composite types because without
> that we can not ensure that lz4 does not exist anywhere in the table.
> Said that with that also we can not ensure that it doesn't exist anywhere
> in the system because it might exist in the WAL and if you do the crash
> recovery then might get those lz4 compressed data back.

I think this is no concern except for PITR, in which case it's working as
intended (if someone does a partial replay to an intermediate state where
tables were still LZ4).  If they replay to a point following the SET pglz +
CLUSTER, then LZ4 doesn't exist in the heap, and anything in the WAL is pretty
uninteresting.

My patch this morning compiled ok on mac, so you should include it.
|0005-f-2nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch
I mailed to Thomas offlist about getting pkg-config installed on the BSD CI
environment.

-- 
Justin

Commits

  1. docs: Update TOAST storage docs for configurable compression.

  2. Further tweaking of pg_dump's handling of default_toast_compression.

  3. Fix interaction of TOAST compression with expression indexes.

  4. Tidy up more loose ends related to configurable TOAST compression.

  5. Short-circuit slice requests that are for more than the object's size.

  6. Mostly-cosmetic adjustments of TOAST-related macros.

  7. Remove useless configure probe for <lz4/lz4.h>.

  8. Error on invalid TOAST compression in CREATE or ALTER TABLE.

  9. docs: Fix omissions related to configurable TOAST compression.

  10. More code cleanup for configurable TOAST compression.

  11. Bring configure support for LZ4 up to snuff.

  12. Make compression.sql regression test independent of default.

  13. Use valid compression method in brin_form_tuple

  14. Fix up pg_dump's handling of per-attribute compression options.

  15. Allow configurable LZ4 TOAST compression.

  16. Fix inconsistencies in the code

  17. Mostly-cosmetic improvements in memory chunk header alignment coding.

  18. Allow numeric to use a more compact, 2-byte header in many cases.