Horton

Re: Compressed TOAST Slicing

Paul Ramsey <pramsey@cleverelephant.ca>

View thread
From: Paul Ramsey <pramsey@cleverelephant.ca>
To: Stephen Frost <sfrost@snowman.net>
Cc: Robert Haas <robertmhaas@gmail.com>, Tom Lane <tgl@sss.pgh.pa.us>, Andres Freund <andres@anarazel.de>, Michael Paquier <michael@paquier.xyz>, Regina Obe <r@pcorp.us>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2019-03-19T23:36:27Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Add support for partial TOAST decompression

  2. Remove remaining hard-wired OID references in the initial catalog data.

  3. Rephrase references to "time qualification".


> On Mar 19, 2019, at 4:47 AM, Stephen Frost <sfrost@snowman.net> wrote:
> 
> Greetings,
> 
> * Paul Ramsey (pramsey@cleverelephant.ca) wrote:
>>> On Mar 18, 2019, at 7:34 AM, Robert Haas <robertmhaas@gmail.com> wrote:
>>> +1.  I think Paul had it right originally.
>> 
>> In that spirit, here is a “one pglz_decompress function, new parameter” version for commit.
> 
> Alright, I've been working through this and have made a few improvements
> (the big comment block at the top of pg_lzcompress.c needed updating,
> among a couple other minor things), but I was trying to wrap my head
> around this:
> 
> 
> Specifically, the two SET_VARSIZE() calls, do we really need both..?
> Are we sure that we're setting the length correctly there..?  Is there
> any cross-check we can do?

Well, we don’t need to do the two SET_VARSIZE() calls, but we *do* need to use rawsize in the call before the return, since we cannot be sure that the size of the uncompressed bit is as large as the requested slice (even though it will be 99 times out of 100)


> I have to admit that I find the new argument to pglz_decompress() a bit
> awkward to describe and document; if you have any thoughts as to how
> that could be improved, that'd be great.

The only thing I can see is loosening the integrity check in pglz_decompress which is a guardrail on something I’m not sure we ever hit. Instead of checking that both the src and dst buffers are fully used up, a test that at least one of them is used up should come up true in all error-free-happy cases.

P