Horton

Re: Optimize partial TOAST decompression

Andrey Borodin <x4mmm@yandex-team.ru>

View thread
From: Andrey Borodin <x4mmm@yandex-team.ru>
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>
Cc: Alvaro Herrera <alvherre@2ndquadrant.com>, Binguo Bao <djydewang@gmail.com>, Paul Ramsey <pramsey@cleverelephant.ca>, Pgsql Hackers <pgsql-hackers@postgresql.org>
Date: 2019-09-30T16:20:22Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Properly determine length for on-disk TOAST values

  2. Blind attempt to fix pglz_maximum_compressed_size

  3. Optimize partial TOAST decompression


> 30 сент. 2019 г., в 20:56, Tomas Vondra <tomas.vondra@2ndquadrant.com> написал(а):
> 
> I mean this:
> 
>   /*
>    * Use int64 to prevent overflow during calculation.
>    */
>   compressed_size = (int32) ((int64) rawsize * 9 + 8) / 8;
> 
> I'm not very familiar with pglz internals, but I'm a bit puzzled by
> this. My first instinct was to compare it to this:
> 
>   #define PGLZ_MAX_OUTPUT(_dlen)	((_dlen) + 4)
> 
> but clearly that's a very different (much simpler) formula. So why
> shouldn't pglz_maximum_compressed_size simply use this macro?

compressed_size accounts for possible increase of size during compression. pglz can consume up to 1 control byte for each 8 bytes of data in worst case.
Even if whole data is compressed well - there can be prefix compressed extremely ineffectively. Thus, if you are going to decompress rawsize bytes, you need at most compressed_size bytes of compressed input.