Horton

Re: Compressed TOAST Slicing

Andrey Borodin <x4mmm@yandex-team.ru>

View thread
From: Andrey Borodin <x4mmm@yandex-team.ru>
To: Paul Ramsey <pramsey@cleverelephant.ca>
Cc: Michael Paquier <michael@paquier.xyz>, Regina Obe <r@pcorp.us>, pgsql-hackers@lists.postgresql.org
Date: 2019-03-12T16:10:08Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Add support for partial TOAST decompression

  2. Remove remaining hard-wired OID references in the initial catalog data.

  3. Rephrase references to "time qualification".


> 12 марта 2019 г., в 19:40, Paul Ramsey <pramsey@cleverelephant.ca> написал(а):
> 
>> On Mar 11, 2019, at 10:42 PM, Michael Paquier <michael@paquier.xyz> wrote:
>> 
>> int32
>> pglz_decompress(const char *source, int32 slen, char *dest,
>> -                               int32 rawsize)
>> +                               int32 rawsize, bool is_slice)
> 
> The sanity check is just that both buffers are completely read they reach their respective ends. With a partial buffer on one side, that check just will definitionally not  happen when slicing (it’s not possible to know a priori what location in the compressed buffer corresponds to a location in the uncompressed one). I can ensure the old API still holds for pglz_decompress() and add a new pglz_decompress_slice() that takes the parameter, is that sufficient?

I think that providing two separate entry points for this functionality is better option.
The word "slice" is widely used for [start:end] slicing, not sure it's good word. But I'm not good in English.

Either way we could replace

if (dp != destend || sp != srcend)
	return -1;

with

if (dp != destend && sp != srcend)
	return -1;

and that's it. || defends from data corruption and some kind of programming mistakes, but actually that's not the purpose of data compression.

Best regards, Andrey Borodin.