Horton

Re: [HACKERS] Custom compression methods

Dilip Kumar <dilipbalaut@gmail.com>

View thread View on postgr.es
From: Dilip Kumar <dilipbalaut@gmail.com>
To: Robert Haas <robertmhaas@gmail.com>
Cc: Justin Pryzby <pryzby@telsasoft.com>, Tomas Vondra <tomas.vondra@2ndquadrant.com>, Alexander Korotkov <a.korotkov@postgrespro.ru>, David Steele <david@pgmasters.net>, Ildus Kurbangaliev <i.kurbangaliev@gmail.com>, Dmitry Dolgov <9erthalion6@gmail.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2021-03-06T15:29:16Z
Lists: pgsql-hackers

Attachments

On Thu, Mar 4, 2021 at 4:03 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Thu, Mar 4, 2021 at 2:49 AM Robert Haas <robertmhaas@gmail.com> wrote:
> >
> > Hi,
> >
> > Does this patch need to do something about ExtractReplicaIdentity()?
> > If there are compressed fields in the tuple being built, can we rely
> > on the decompression engine being available at the time we need to do
> > something with the tuple?
>
> We log the replica identity tuple in the WAL so that later walsender
> can stream this to the subscriber, and before sending to the
> subscriber anyway we have detoast all the data.  Said that I think the
> problem you are worried about is not only with 'replica identity
> tuple' but it is with any tuple.  I mean we copy the compressed field
> as it is in WAL and suppose we copy some fields which are compressed
> with lz4 and then we restart the server with another binary that is
> compiled without lz4.  Now, the problem is the walsender can not
> decompress those

Based on the off list discussion with Robert, there are a couple of
problems which might be very difficult to handle when we support the
custom compression method using the access methods, the major problems
are 1)  compressed data inside composite type 2) Access method might
get dropped before walsender decode the compressed data.   I think the
first problem we still have is some solution although it may impact
performance in some cases i.e. extended record.  But the problem of
the compressed data inside the WAL is a bigger problem.  So as of now
we are planning to go ahead only with the built-in methods and if we
are only continuing with the built-in method so it doesn't make sense
to continue the access method infrastructure.  I have rewrote the
patches without using the access method for compression.   Changes in
the patches

- Removed complete dependency on the access method for compression
- While moving the tuple from one table to another table with
different compression method, no need to compare the compression
method and decompress.
- Alter table set compression, will not rewrite the old data, so only
the new tuple will be compressed with the new compression method.
- No preserve.

I feel the built-in method patch now looks cleaner and smaller than it
was before.

-- 
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

Commits

  1. docs: Update TOAST storage docs for configurable compression.

  2. Further tweaking of pg_dump's handling of default_toast_compression.

  3. Fix interaction of TOAST compression with expression indexes.

  4. Tidy up more loose ends related to configurable TOAST compression.

  5. Short-circuit slice requests that are for more than the object's size.

  6. Mostly-cosmetic adjustments of TOAST-related macros.

  7. Remove useless configure probe for <lz4/lz4.h>.

  8. Error on invalid TOAST compression in CREATE or ALTER TABLE.

  9. docs: Fix omissions related to configurable TOAST compression.

  10. More code cleanup for configurable TOAST compression.

  11. Bring configure support for LZ4 up to snuff.

  12. Make compression.sql regression test independent of default.

  13. Use valid compression method in brin_form_tuple

  14. Fix up pg_dump's handling of per-attribute compression options.

  15. Allow configurable LZ4 TOAST compression.

  16. Fix inconsistencies in the code

  17. Mostly-cosmetic improvements in memory chunk header alignment coding.

  18. Allow numeric to use a more compact, 2-byte header in many cases.