Horton

Re: Compressed TOAST Slicing

Paul Ramsey <pramsey@cleverelephant.ca>

View thread
From: Paul Ramsey <pramsey@cleverelephant.ca>
To: Michael Paquier <michael@paquier.xyz>
Cc: Regina Obe <r@pcorp.us>, pgsql-hackers@lists.postgresql.org
Date: 2019-03-12T14:40:43Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Add support for partial TOAST decompression

  2. Remove remaining hard-wired OID references in the initial catalog data.

  3. Rephrase references to "time qualification".


> On Mar 11, 2019, at 10:42 PM, Michael Paquier <michael@paquier.xyz> wrote:
> 
> On Mon, Mar 11, 2019 at 08:38:56PM +0000, Regina Obe wrote:
>> I tested on windows mingw64 (as of a week ago) and confirmed the
>> patch applies cleanly and significantly faster for left, substr
>> tests than head. 
> 
> int32
> pglz_decompress(const char *source, int32 slen, char *dest,
> -                               int32 rawsize)
> +                               int32 rawsize, bool is_slice)
> The performance improvements are nice, but breaking a published API is
> less nice particularly since some work has been done to make pglz more
> plugabble (see 60838df9, guess how wrote that).  Could it be possible
> to rework this part please?  It's been some time since I touched this
> code, but it would be really nice if we don't have an extra parameter,
> and just not bypass the sanity checks at the end.  Using a parameter
> to bypass those checks may cause problems for future callers of it.

The sanity check is just that both buffers are completely read they reach their respective ends. With a partial buffer on one side, that check just will definitionally not  happen when slicing (it’s not possible to know a priori what location in the compressed buffer corresponds to a location in the uncompressed one). I can ensure the old API still holds for pglz_decompress() and add a new pglz_decompress_slice() that takes the parameter, is that sufficient?

P.