Horton

Re: Compressed TOAST Slicing

Paul Ramsey <pramsey@cleverelephant.ca>

View thread
From: Paul Ramsey <pramsey@cleverelephant.ca>
To: Andrey Borodin <x4mmm@yandex-team.ru>
Cc: Владимир Лесков <vladimirlesk@yandex-team.ru>, Stephen Frost <sfrost@snowman.net>, Daniel Verite <daniel@manitou-mail.org>, Simon Riggs <simon@2ndquadrant.com>, rafia.sabih@enterprisedb.com, Tom Lane <tgl@sss.pgh.pa.us>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2019-04-09T17:12:56Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Add support for partial TOAST decompression

  2. Remove remaining hard-wired OID references in the initial catalog data.

  3. Rephrase references to "time qualification".

> On Apr 9, 2019, at 10:09 AM, Andrey Borodin <x4mmm@yandex-team.ru> wrote:
> 
> He advised me to use algorithm that splits copied regions into smaller non-overlapping subregions with exponentially increasing size.
> 
> while (off <= len)
> {
>    memcpy(dp, dp - off, off);
>    len -= off;
>    dp += off;
>    off *= 2;
> }
> memcpy(dp, dp - off, len);
> 
> On original Paul's test without patch of this thread this optimization gave about x2.5 speedup.
> I've composed more detailed tests[0] and tested against current master. Now it only gives 20%-25% of decompression speedup, but I think it is still useful.

Wow, well beyond slicing, just being able to decompress 25% faster is a win for pretty much any TOAST use case. I guess the $100 question is: portability? The whole reason for the old-skool code that’s there now was concerns about memcpy’ing overlapping addresses and Bad Things happening.

P.