Horton

Re: [HACKERS] Custom compression methods

Dilip Kumar <dilipbalaut@gmail.com>

View thread View on postgr.es
From: Dilip Kumar <dilipbalaut@gmail.com>
To: Robert Haas <robertmhaas@gmail.com>
Cc: Justin Pryzby <pryzby@telsasoft.com>, Tomas Vondra <tomas.vondra@2ndquadrant.com>, Alexander Korotkov <a.korotkov@postgrespro.ru>, David Steele <david@pgmasters.net>, Ildus Kurbangaliev <i.kurbangaliev@gmail.com>, Dmitry Dolgov <9erthalion6@gmail.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2021-02-10T14:51:59Z
Lists: pgsql-hackers

Attachments

On Tue, Feb 9, 2021 at 6:14 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> > Basically, we are just checking whether the stored value is compressed
> > or not, but we are clearly ignoring the fact that it might be
> > compressed and stored externally on disk.  So basically if the value
> > is stored externally we can not know whether the external data were
> > compressed or not without fetching the values from the toast table,  I
> > think instead of fetching the complete data from toast we can just
> > fetch the header using 'toast_fetch_datum_slice'.
> >
> > Any other thoughts on this?
>
> I think I was partially wrong here.  Basically, there is a way to know
> whether the external data are compressed or not using
> VARATT_EXTERNAL_IS_COMPRESSED macro.  However, if it is compressed
> then we will have to fetch the toast slice of size
> toast_compress_header, to know the compression method.

I have fixed this issue, so now we will be able to detect the
compression method of the externalized compressed data as well.  I
have also added a test case for this. I have rebased other patches
also on top of this patch.  I have fixed the doc compilation issue in
patch 0004 raised by Justin.  I still could not figure out what is the
right place for inheriting the compression method related change in
the "ddl.sgml" so that is still there, any suggestions on that?

--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

Commits

  1. docs: Update TOAST storage docs for configurable compression.

  2. Further tweaking of pg_dump's handling of default_toast_compression.

  3. Fix interaction of TOAST compression with expression indexes.

  4. Tidy up more loose ends related to configurable TOAST compression.

  5. Short-circuit slice requests that are for more than the object's size.

  6. Mostly-cosmetic adjustments of TOAST-related macros.

  7. Remove useless configure probe for <lz4/lz4.h>.

  8. Error on invalid TOAST compression in CREATE or ALTER TABLE.

  9. docs: Fix omissions related to configurable TOAST compression.

  10. More code cleanup for configurable TOAST compression.

  11. Bring configure support for LZ4 up to snuff.

  12. Make compression.sql regression test independent of default.

  13. Use valid compression method in brin_form_tuple

  14. Fix up pg_dump's handling of per-attribute compression options.

  15. Allow configurable LZ4 TOAST compression.

  16. Fix inconsistencies in the code

  17. Mostly-cosmetic improvements in memory chunk header alignment coding.

  18. Allow numeric to use a more compact, 2-byte header in many cases.