Horton

Thread

  1. Improve compression speeds in pg_lzcompress.c

    Takeshi Yamamuro <yamamuro.takeshi@lab.ntt.co.jp> — 2013-01-07T07:29:58Z

    Hi, hackers,

The attached is a patch to improve compression speeds with loss of
compression ratios in backend/utils/adt/pg_lzcompress.c. Recent
modern compression techniques like google LZ4 and Snappy inspreid
me to write this patch. Thre are two points of my patch:

1. Skip at most 255 literals that might be incompressible
   during pattern matching for LZ compression.

2. Update a hash table every PGLZ_HASH_GAP literals.

A sequence of literals is typically mixed up with compressible parts
and incompressible ones. Then, IMHO that it is reasonable to skip
PGLZ_SKIP_SIZE literals every a match is not found. The skipped multiple
literals are just copied to the output buffer, so pglz_out_literal() is
re-written (and renamed pglz_out_literals) so as to copy multiple
bytes, not a single byte.

And also, the current implementation updates a hash table for every a single
literal. However, as the updates obviously eat much processor time, skipping
the updates dynamically improves compression speeds.

I've done quick comparison tests with a Xeon 5670 processor.
A sequence logs of Apache hadoop and TREC GOV2 web data were used
as test sets. The former is highly compressible (low entroy) and the
other is difficult to compress (high entropy).

*******************
                         Compression Speed (Ratio)
Apache hadoop logs:
gzip                            78.22MiB/s ( 5.31%)
bzip2                            3.34MiB/s ( 3.04%)
lz4                            939.45MiB/s ( 9.17%)
pg_lzcompress(original)         37.80MiB/s (11.76%)
pg_lzcompress(patch apaplied)   99.42MiB/s (14.19%)

TREC GOV2 web data:
gzip                            21.22MiB/s (32.66%)
bzip2                            8.61MiB/s (27.86%)
lz4                            250.98MiB/s (49.82%)
pg_lzcompress(original)         20.44MiB/s (50.09%)
pg_lzcompress(patch apaplied)   48.67MiB/s (61.87%)

*******************

Obviously, both the compression ratio and the speed in the current
implementation are inferior to those in gzip. And, my patch
loses gzip and bzip2 in view of compression ratios though, the
compression speed overcomes those in gzip and bzip2.

Anyway, the compression speed in lz4 is very fast, so in my
opinion, there is a room to improve the current implementation
in pg_lzcompress.

regards,
-- 
----
Takeshi Yamamuro
NTT Cyber Communications Laboratory Group
Software Innovation Center
(Open Source Software Center)
Tel: +81-3-5860-5057 Fax: +81-3-5463-5490
Mail:yamamuro.takeshi@lab.ntt.co.jp

  2. Re: Improve compression speeds in pg_lzcompress.c

    Simon Riggs <simon@2ndquadrant.com> — 2013-01-07T09:10:31Z

    On 7 January 2013 07:29, Takeshi Yamamuro
<yamamuro.takeshi@lab.ntt.co.jp> wrote:

> Anyway, the compression speed in lz4 is very fast, so in my
> opinion, there is a room to improve the current implementation
> in pg_lzcompress.

So why don't we use LZ4?

-- 
 Simon Riggs                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

  3. Re: Improve compression speeds in pg_lzcompress.c

    John R Pierce <pierce@hogranch.com> — 2013-01-07T09:19:59Z

    On 1/7/2013 1:10 AM, Simon Riggs wrote:
> On 7 January 2013 07:29, Takeshi Yamamuro
> <yamamuro.takeshi@lab.ntt.co.jp>  wrote:
>
>> >Anyway, the compression speed in lz4 is very fast, so in my
>> >opinion, there is a room to improve the current implementation
>> >in pg_lzcompress.
> So why don't we use LZ4?


what will changing compression formats do for compatability?

this is for the compressed data in pg_toast storage or something? will 
this break pg_upgrade style operations?

  4. Re: Improve compression speeds in pg_lzcompress.c

    Simon Riggs <simon@2ndquadrant.com> — 2013-01-07T09:57:58Z

    On 7 January 2013 09:19, John R Pierce <pierce@hogranch.com> wrote:
> On 1/7/2013 1:10 AM, Simon Riggs wrote:
>>
>> On 7 January 2013 07:29, Takeshi Yamamuro
>> <yamamuro.takeshi@lab.ntt.co.jp>  wrote:
>>
>>> >Anyway, the compression speed in lz4 is very fast, so in my
>>> >opinion, there is a room to improve the current implementation
>>> >in pg_lzcompress.
>>
>> So why don't we use LZ4?
>
> what will changing compression formats do for compatability?
>
> this is for the compressed data in pg_toast storage or something? will this
> break pg_upgrade style operations?

Anything that changes on-disk format would need to consider how to do
pg_upgrade. It's the major blocker in that area.

For this, it would be possible to have a new format and old format
coexist, but that will take more time to think through than we have
for this release, so this is a nice idea for further investigation in
9.4. Thanks for raising that point.

-- 
 Simon Riggs                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

  5. Re: Improve compression speeds in pg_lzcompress.c

    Andres Freund <andres@2ndquadrant.com> — 2013-01-07T10:05:24Z

    On 2013-01-07 09:57:58 +0000, Simon Riggs wrote:
> On 7 January 2013 09:19, John R Pierce <pierce@hogranch.com> wrote:
> > On 1/7/2013 1:10 AM, Simon Riggs wrote:
> >>
> >> On 7 January 2013 07:29, Takeshi Yamamuro
> >> <yamamuro.takeshi@lab.ntt.co.jp>  wrote:
> >>
> >>> >Anyway, the compression speed in lz4 is very fast, so in my
> >>> >opinion, there is a room to improve the current implementation
> >>> >in pg_lzcompress.
> >>
> >> So why don't we use LZ4?
> >
> > what will changing compression formats do for compatability?
> >
> > this is for the compressed data in pg_toast storage or something? will this
> > break pg_upgrade style operations?
>
> Anything that changes on-disk format would need to consider how to do
> pg_upgrade. It's the major blocker in that area.
>
> For this, it would be possible to have a new format and old format
> coexist, but that will take more time to think through than we have
> for this release, so this is a nice idea for further investigation in
> 9.4. Thanks for raising that point.

I think there should be enough bits available in the toast pointer to
indicate the type of compression. I seem to remember somebody even
posting a patch to that effect?
I agree that it's probably too late in the 9.3 cycle to start with this.

Greetings,

Andres Freund

--
 Andres Freund	                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

  6. Re: Improve compression speeds in pg_lzcompress.c

    John R Pierce <pierce@hogranch.com> — 2013-01-07T10:21:26Z

    On 1/7/2013 2:05 AM, Andres Freund wrote:
> I think there should be enough bits available in the toast pointer to
> indicate the type of compression. I seem to remember somebody even
> posting a patch to that effect?
> I agree that it's probably too late in the 9.3 cycle to start with this.

so an upgraded database would have old toasted values in the old 
compression format, and new toasted values in the new format in an 
existing table?      that's kind of ugly.

  7. Re: Improve compression speeds in pg_lzcompress.c

    Andres Freund <andres@2ndquadrant.com> — 2013-01-07T10:39:22Z

    On 2013-01-07 02:21:26 -0800, John R Pierce wrote:
> On 1/7/2013 2:05 AM, Andres Freund wrote:
> >I think there should be enough bits available in the toast pointer to
> >indicate the type of compression. I seem to remember somebody even
> >posting a patch to that effect?
> >I agree that it's probably too late in the 9.3 cycle to start with this.
>
> so an upgraded database would have old toasted values in the old compression
> format, and new toasted values in the new format in an existing table?
> that's kind of ugly.

Well, ISTM thats just life. What you prefer? Converting all toast values
during pg_upgrade kinda goes against the aim of quick upgrades.

Greetings,

Andres Freund

--
 Andres Freund	                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

  8. Re: Improve compression speeds in pg_lzcompress.c

    Greg Stark <stark@mit.edu> — 2013-01-07T13:36:33Z

    On Mon, Jan 7, 2013 at 10:21 AM, John R Pierce <pierce@hogranch.com> wrote:
> On 1/7/2013 2:05 AM, Andres Freund wrote:
>>
>> I think there should be enough bits available in the toast pointer to
>> indicate the type of compression. I seem to remember somebody even
>> posting a patch to that effect?
>> I agree that it's probably too late in the 9.3 cycle to start with this.
>
>
> so an upgraded database would have old toasted values in the old compression
> format, and new toasted values in the new format in an existing table?
> that's kind of ugly.

I haven't looked at the patch. It's not obvious to me from the
description that the output isn't backwards compatible. The way the LZ
toast compression works the output is self-describing. There are many
different outputs that would decompress to the same thing and the
compressing code can choose how hard to look for earlier matches and
when to just copy bytes wholesale but the decompression will work
regardless.


-- 
greg

  9. Re: Improve compression speeds in pg_lzcompress.c

    Simon Riggs <simon@2ndquadrant.com> — 2013-01-07T13:48:06Z

    On 7 January 2013 13:36, Greg Stark <stark@mit.edu> wrote:
> On Mon, Jan 7, 2013 at 10:21 AM, John R Pierce <pierce@hogranch.com> wrote:
>> On 1/7/2013 2:05 AM, Andres Freund wrote:
>>>
>>> I think there should be enough bits available in the toast pointer to
>>> indicate the type of compression. I seem to remember somebody even
>>> posting a patch to that effect?
>>> I agree that it's probably too late in the 9.3 cycle to start with this.
>>
>>
>> so an upgraded database would have old toasted values in the old compression
>> format, and new toasted values in the new format in an existing table?
>> that's kind of ugly.
>
> I haven't looked at the patch. It's not obvious to me from the
> description that the output isn't backwards compatible. The way the LZ
> toast compression works the output is self-describing. There are many
> different outputs that would decompress to the same thing and the
> compressing code can choose how hard to look for earlier matches and
> when to just copy bytes wholesale but the decompression will work
> regardless.

Good point, and a great reason to use this patch rather than LZ4 for 9.3

We could even have tuning parameters for toast compression, as long as
we keep the on disk format identical.

-- 
 Simon Riggs                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

  10. Re: Improve compression speeds in pg_lzcompress.c

    ktm@rice.edu <ktm@rice.edu> — 2013-01-07T13:52:09Z

    On Mon, Jan 07, 2013 at 09:10:31AM +0000, Simon Riggs wrote:
> On 7 January 2013 07:29, Takeshi Yamamuro
> <yamamuro.takeshi@lab.ntt.co.jp> wrote:
> 
> > Anyway, the compression speed in lz4 is very fast, so in my
> > opinion, there is a room to improve the current implementation
> > in pg_lzcompress.
> 
> So why don't we use LZ4?
> 
+1

Regards,
Ken

  11. Re: Improve compression speeds in pg_lzcompress.c

    ktm@rice.edu <ktm@rice.edu> — 2013-01-07T13:57:41Z

    On Mon, Jan 07, 2013 at 01:36:33PM +0000, Greg Stark wrote:
> On Mon, Jan 7, 2013 at 10:21 AM, John R Pierce <pierce@hogranch.com> wrote:
> > On 1/7/2013 2:05 AM, Andres Freund wrote:
> >>
> >> I think there should be enough bits available in the toast pointer to
> >> indicate the type of compression. I seem to remember somebody even
> >> posting a patch to that effect?
> >> I agree that it's probably too late in the 9.3 cycle to start with this.
> >
> >
> > so an upgraded database would have old toasted values in the old compression
> > format, and new toasted values in the new format in an existing table?
> > that's kind of ugly.
> 
> I haven't looked at the patch. It's not obvious to me from the
> description that the output isn't backwards compatible. The way the LZ
> toast compression works the output is self-describing. There are many
> different outputs that would decompress to the same thing and the
> compressing code can choose how hard to look for earlier matches and
> when to just copy bytes wholesale but the decompression will work
> regardless.
> 

I think this comment refers to the lz4 option. I do agree that the patch
that was posted to improve the current compression speed should be able
to be implemented to allow the current results to be decompressed as well.

Regards,
Ken

  12. Re: Improve compression speeds in pg_lzcompress.c

    Andres Freund <andres@2ndquadrant.com> — 2013-01-07T14:03:21Z

    Hi,

It seems worth rereading the thread around
http://archives.postgresql.org/message-id/CAAZKuFb59sABSa7gCG0vnVnGb-mJCUBBbrKiyPraNXHnis7KMw%40mail.gmail.com
for people wanting to work on this.

Greetings,

Andres Freund

-- 
 Andres Freund	                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

  13. Re: Improve compression speeds in pg_lzcompress.c

    Tom Lane <tgl@sss.pgh.pa.us> — 2013-01-07T16:16:07Z

    Takeshi Yamamuro <yamamuro.takeshi@lab.ntt.co.jp> writes:
> The attached is a patch to improve compression speeds with loss of
> compression ratios in backend/utils/adt/pg_lzcompress.c.

Why would that be a good tradeoff to make?  Larger stored values require
more I/O, which is likely to swamp any CPU savings in the compression
step.  Not to mention that a value once written may be read many times,
so the extra I/O cost could be multiplied many times over later on.

Another thing to keep in mind is that the compression area in general
is a minefield of patents.  We're fairly confident that pg_lzcompress
as-is doesn't fall foul of any, but any significant change there would
probably require more research.

			regards, tom lane

  14. Re: Improve compression speeds in pg_lzcompress.c

    Merlin Moncure <mmoncure@gmail.com> — 2013-01-07T20:18:57Z

    On Mon, Jan 7, 2013 at 10:16 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Takeshi Yamamuro <yamamuro.takeshi@lab.ntt.co.jp> writes:
>> The attached is a patch to improve compression speeds with loss of
>> compression ratios in backend/utils/adt/pg_lzcompress.c.
>
> Why would that be a good tradeoff to make?  Larger stored values require
> more I/O, which is likely to swamp any CPU savings in the compression
> step.  Not to mention that a value once written may be read many times,
> so the extra I/O cost could be multiplied many times over later on.

I disagree.  pg compression is so awful it's almost never a net win.
I turn it off.

> Another thing to keep in mind is that the compression area in general
> is a minefield of patents.  We're fairly confident that pg_lzcompress
> as-is doesn't fall foul of any, but any significant change there would
> probably require more research.

A minefield of *expired* patents.  Fast lz based compression is used
all over the place -- for example by the lucene.

lz4.

merlin

  15. Re: Improve compression speeds in pg_lzcompress.c

    Tom Lane <tgl@sss.pgh.pa.us> — 2013-01-07T20:41:00Z

    Merlin Moncure <mmoncure@gmail.com> writes:
> On Mon, Jan 7, 2013 at 10:16 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> Takeshi Yamamuro <yamamuro.takeshi@lab.ntt.co.jp> writes:
>>> The attached is a patch to improve compression speeds with loss of
>>> compression ratios in backend/utils/adt/pg_lzcompress.c.

>> Why would that be a good tradeoff to make?  Larger stored values require
>> more I/O, which is likely to swamp any CPU savings in the compression
>> step.  Not to mention that a value once written may be read many times,
>> so the extra I/O cost could be multiplied many times over later on.

> I disagree.  pg compression is so awful it's almost never a net win.
> I turn it off.

One report doesn't make it useless, but even if it is so on your data,
why would making it even less effective be a win?

>> Another thing to keep in mind is that the compression area in general
>> is a minefield of patents.  We're fairly confident that pg_lzcompress
>> as-is doesn't fall foul of any, but any significant change there would
>> probably require more research.

> A minefield of *expired* patents.  Fast lz based compression is used
> all over the place -- for example by the lucene.

The patents that had to be dodged for original LZ compression are gone,
true, but what's your evidence for saying that newer versions don't have
newer patents?

			regards, tom lane

  16. Re: Improve compression speeds in pg_lzcompress.c

    Robert Haas <robertmhaas@gmail.com> — 2013-01-07T21:07:51Z

    On Mon, Jan 7, 2013 at 11:16 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Why would that be a good tradeoff to make?  Larger stored values require
> more I/O, which is likely to swamp any CPU savings in the compression
> step.  Not to mention that a value once written may be read many times,
> so the extra I/O cost could be multiplied many times over later on.

I agree with this analysis, but I note that the test results show it
actually improving things along both parameters.

I'm not sure how general that result is.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  17. Re: Improve compression speeds in pg_lzcompress.c

    Tom Lane <tgl@sss.pgh.pa.us> — 2013-01-07T21:19:36Z

    Robert Haas <robertmhaas@gmail.com> writes:
> On Mon, Jan 7, 2013 at 11:16 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> Why would that be a good tradeoff to make?  Larger stored values require
>> more I/O, which is likely to swamp any CPU savings in the compression
>> step.  Not to mention that a value once written may be read many times,
>> so the extra I/O cost could be multiplied many times over later on.

> I agree with this analysis, but I note that the test results show it
> actually improving things along both parameters.

Hm ... one of us is reading those results backwards, then.

			regards, tom lane

  18. Re: Improve compression speeds in pg_lzcompress.c

    Merlin Moncure <mmoncure@gmail.com> — 2013-01-07T21:36:25Z

    On Mon, Jan 7, 2013 at 2:41 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Merlin Moncure <mmoncure@gmail.com> writes:
>> On Mon, Jan 7, 2013 at 10:16 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>> Takeshi Yamamuro <yamamuro.takeshi@lab.ntt.co.jp> writes:
>>>> The attached is a patch to improve compression speeds with loss of
>>>> compression ratios in backend/utils/adt/pg_lzcompress.c.
>
>>> Why would that be a good tradeoff to make?  Larger stored values require
>>> more I/O, which is likely to swamp any CPU savings in the compression
>>> step.  Not to mention that a value once written may be read many times,
>>> so the extra I/O cost could be multiplied many times over later on.
>
>> I disagree.  pg compression is so awful it's almost never a net win.
>> I turn it off.
>
> One report doesn't make it useless, but even if it is so on your data,
> why would making it even less effective be a win?

That's a fair point.  I'm neutral on the OP's proposal -- it's just
moving spots around the dog.  If we didn't have better options, maybe
offering options to tune what we have would be worth implementing...
but by your standard ISTM we can't even do *that*.

>>> Another thing to keep in mind is that the compression area in general
>>> is a minefield of patents.  We're fairly confident that pg_lzcompress
>>> as-is doesn't fall foul of any, but any significant change there would
>>> probably require more research.
>
>> A minefield of *expired* patents.  Fast lz based compression is used
>> all over the place -- for example by the lucene.
>
> The patents that had to be dodged for original LZ compression are gone,
> true, but what's your evidence for saying that newer versions don't have
> newer patents?

That's impossible (at least for a non-attorney) to do because the
patents are still flying (for example:
http://www.google.com/patents/US7650040).  That said, you've framed
the debate so that any improvement to postgres compression requires an
IP lawyer.  That immediately raises some questions:

*) why hold only compression type features in postgres to that
standard?  Patents get mentioned here and there in the context of
other features in the archives but only compression seems to require a
proven clean pedigree.   Why don't we require a patent search for
other interesting features? What evidence do *you* offer that lz4
violates any patents?

*) why is postgres the only FOSS project that cares about
patentability of say, lz4?  (google 'lz4 patent')

merlin

  19. Re: Improve compression speeds in pg_lzcompress.c

    Andrew Dunstan <andrew@dunslane.net> — 2013-01-07T21:44:50Z

    On 01/07/2013 04:19 PM, Tom Lane wrote:
> Robert Haas <robertmhaas@gmail.com> writes:
>> On Mon, Jan 7, 2013 at 11:16 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>> Why would that be a good tradeoff to make?  Larger stored values require
>>> more I/O, which is likely to swamp any CPU savings in the compression
>>> step.  Not to mention that a value once written may be read many times,
>>> so the extra I/O cost could be multiplied many times over later on.
>> I agree with this analysis, but I note that the test results show it
>> actually improving things along both parameters.
> Hm ... one of us is reading those results backwards, then.
>
> 			

I just went back and looked. Unless I'm misreading it he has about a 2.5 
times speed improvement but about a 20% worse compression result.

What would be interesting would be to see if the knobs he's tweaked 
could be tweaked a bit more to give us substantial speedup without 
significant space degradation.

cheers

andrew

  20. Re: Improve compression speeds in pg_lzcompress.c

    Robert Haas <robertmhaas@gmail.com> — 2013-01-07T22:00:39Z

    On Mon, Jan 7, 2013 at 4:19 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Hm ... one of us is reading those results backwards, then.

*looks*

It's me.

Sorry for the noise.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  21. Re: Improve compression speeds in pg_lzcompress.c

    Takeshi Yamamuro <yamamuro.takeshi@lab.ntt.co.jp> — 2013-01-08T09:04:24Z

    Hi,

 >>>> Why would that be a good tradeoff to make? Larger stored values 
require
 >>>> more I/O, which is likely to swamp any CPU savings in the compression
 >>>> step. Not to mention that a value once written may be read many times,
 >>>> so the extra I/O cost could be multiplied many times over later on.
 >>> I agree with this analysis, but I note that the test results show it
 >>> actually improving things along both parameters.
 >> Hm ... one of us is reading those results backwards, then.
I think that it's a parameter-tuning issue.
I added the two parameters, PGLZ_SKIP_SIZE and PGLZ_HASH_GAP, and
set PGLZ_SKIP_SIZE=3 and PGLZ_HASH_GAP=8 for the quick tests.
And also, I found that the performance in my patch was nearly
equal to that in the current implementation when
PGLZ_SKIP_SIZE=1 and PGLZ_HASH_GAP=1.

Apart from my patch, what I care is that the current one might
be much slow against I/O. For example, when compressing
and writing large values, compressing data (20-40MiB/s) might be
a dragger against writing data in disks (50-80MiB/s). Moreover,
IMHO modern (and very fast) I/O subsystems such as SSD make a
bigger issue in this case.

Then, I think it's worth keeping discussions to improve
compression stuffs for 9.4, or later.


 > Another thing to keep in mind is that the compression area in general
 > is a minefield of patents.  We're fairly confident that pg_lzcompress
 > as-is doesn't fall foul of any, but any significant change there would
 > probably require more research.
Agree, and we know ...
we need to have patent-free ideas to improve compression issues.
For example, pluggable compression IF, or something.


 > I just went back and looked. Unless I'm misreading it he has about a 2.5
 > times speed improvement but about a 20% worse compression result.
 >
 > What would be interesting would be to see if the knobs he's tweaked
 > could be tweaked a bit more to give us substantial speedup without
 > significant space degradation.
Yes, you're right, and these results highly depend
on data sets though.


regards,
-- 
----
Takeshi Yamamuro
NTT Cyber Communications Laboratory Group
Software Innovation Center
(Open Source Software Center)
Tel: +81-3-5860-5057 Fax: +81-3-5463-5490
Mail:yamamuro.takeshi@lab.ntt.co.jp

  22. Re: Improve compression speeds in pg_lzcompress.c

    Takeshi Yamamuro <yamamuro.takeshi@lab.ntt.co.jp> — 2013-01-08T09:08:13Z

    >> So why don't we use LZ4?
>>
> +1
Agree though, I think there're still patent issues there.

regards,
-- 
----
Takeshi Yamamuro
NTT Cyber Communications Laboratory Group
Software Innovation Center
(Open Source Software Center)
Tel: +81-3-5860-5057 Fax: +81-3-5463-5490
Mail:yamamuro.takeshi@lab.ntt.co.jp

  23. Re: Improve compression speeds in pg_lzcompress.c

    Takeshi Yamamuro <yamamuro.takeshi@lab.ntt.co.jp> — 2013-01-08T09:19:13Z

    Hi,

(2013/01/07 22:36), Greg Stark wrote:
> On Mon, Jan 7, 2013 at 10:21 AM, John R Pierce<pierce@hogranch.com>  wrote:
>> On 1/7/2013 2:05 AM, Andres Freund wrote:
>>>
>>> I think there should be enough bits available in the toast pointer to
>>> indicate the type of compression. I seem to remember somebody even
>>> posting a patch to that effect?
>>> I agree that it's probably too late in the 9.3 cycle to start with this.
>>
>>
>> so an upgraded database would have old toasted values in the old compression
>> format, and new toasted values in the new format in an existing table?
>> that's kind of ugly.
>
> I haven't looked at the patch. It's not obvious to me from the
> description that the output isn't backwards compatible. The way the LZ
> toast compression works the output is self-describing. There are many
> different outputs that would decompress to the same thing and the
> compressing code can choose how hard to look for earlier matches and
> when to just copy bytes wholesale but the decompression will work
> regardless.

My patch is not backwards compatible, so we need some features
to switch these old and new disk formats.

I think the discussion below is helpful in this use.
That is, PGLZ_Header is used as this purpose.
http://archives.postgresql.org/pgsql-hackers/2012-03/msg00971.php

regards,
-- 
----
Takeshi Yamamuro
NTT Cyber Communications Laboratory Group
Software Innovation Center
(Open Source Software Center)
Tel: +81-3-5860-5057 Fax: +81-3-5463-5490
Mail:yamamuro.takeshi@lab.ntt.co.jp

  24. Re: Improve compression speeds in pg_lzcompress.c

    Hannu Krosing <hannu@krosing.net> — 2013-01-08T11:24:32Z

    On 01/08/2013 10:19 AM, Takeshi Yamamuro wrote:
> Hi,
>
> (2013/01/07 22:36), Greg Stark wrote:
>> On Mon, Jan 7, 2013 at 10:21 AM, John R Pierce<pierce@hogranch.com>  
>> wrote:
>>> On 1/7/2013 2:05 AM, Andres Freund wrote:
>>>>
>>>> I think there should be enough bits available in the toast pointer to
>>>> indicate the type of compression. I seem to remember somebody even
>>>> posting a patch to that effect?
>>>> I agree that it's probably too late in the 9.3 cycle to start with 
>>>> this.
>>>
>>>
>>> so an upgraded database would have old toasted values in the old 
>>> compression
>>> format, and new toasted values in the new format in an existing table?
>>> that's kind of ugly.
>>
>> I haven't looked at the patch. It's not obvious to me from the
>> description that the output isn't backwards compatible. The way the LZ
>> toast compression works the output is self-describing. There are many
>> different outputs that would decompress to the same thing and the
>> compressing code can choose how hard to look for earlier matches and
>> when to just copy bytes wholesale but the decompression will work
>> regardless.
>
> My patch is not backwards compatible, so we need some features
> to switch these old and new disk formats.
Is it a feature of our compressed format that it is hard to make this
backwards compatible.

Only decompression should work anyway as we have not supported
physical compatibility in the other direction in our other tools.

That is, we don't have pg_downgrade :)

Hannu

>
> I think the discussion below is helpful in this use.
> That is, PGLZ_Header is used as this purpose.
> http://archives.postgresql.org/pgsql-hackers/2012-03/msg00971.php
>
> regards,

  25. Re: Improve compression speeds in pg_lzcompress.c

    Robert Haas <robertmhaas@gmail.com> — 2013-01-08T13:20:42Z

    On Tue, Jan 8, 2013 at 4:04 AM, Takeshi Yamamuro
<yamamuro.takeshi@lab.ntt.co.jp> wrote:
> Apart from my patch, what I care is that the current one might
> be much slow against I/O. For example, when compressing
> and writing large values, compressing data (20-40MiB/s) might be
> a dragger against writing data in disks (50-80MiB/s). Moreover,
> IMHO modern (and very fast) I/O subsystems such as SSD make a
> bigger issue in this case.

What about just turning compression off?

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  26. Re: Improve compression speeds in pg_lzcompress.c

    Claudio Freire <klaussfreire@gmail.com> — 2013-01-08T14:51:06Z

    On Tue, Jan 8, 2013 at 10:20 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> On Tue, Jan 8, 2013 at 4:04 AM, Takeshi Yamamuro
> <yamamuro.takeshi@lab.ntt.co.jp> wrote:
>> Apart from my patch, what I care is that the current one might
>> be much slow against I/O. For example, when compressing
>> and writing large values, compressing data (20-40MiB/s) might be
>> a dragger against writing data in disks (50-80MiB/s). Moreover,
>> IMHO modern (and very fast) I/O subsystems such as SSD make a
>> bigger issue in this case.
>
> What about just turning compression off?

I've been relying on compression for some big serialized blob fields
for some time now. I bet I'm not alone, lots of people save serialized
data to text fields. So rather than removing it, I'd just change the
default to off (if that was the decision).

However, it might be best to evaluate some of the modern fast
compression schemes like snappy/lz4 (250MB/s per core sounds pretty
good), and implement pluggable compression schemes instead. Snappy
wasn't designed for nothing, it was most likely because it was
necessary. Cassandra (just to name a system I'm familiar with) started
without compression, and then it was deemed necessary to the point
they invested considerable time into it. I've always found the fact
that pg does compression of toast tables quite forward-thinking, and
I'd say the feature has to remain there, extended and modernized,
maybe off by default, but there.

  27. Re: Improve compression speeds in pg_lzcompress.c

    Robert Haas <robertmhaas@gmail.com> — 2013-01-08T22:14:20Z

    On Tue, Jan 8, 2013 at 9:51 AM, Claudio Freire <klaussfreire@gmail.com> wrote:
> On Tue, Jan 8, 2013 at 10:20 AM, Robert Haas <robertmhaas@gmail.com> wrote:
>> On Tue, Jan 8, 2013 at 4:04 AM, Takeshi Yamamuro
>> <yamamuro.takeshi@lab.ntt.co.jp> wrote:
>>> Apart from my patch, what I care is that the current one might
>>> be much slow against I/O. For example, when compressing
>>> and writing large values, compressing data (20-40MiB/s) might be
>>> a dragger against writing data in disks (50-80MiB/s). Moreover,
>>> IMHO modern (and very fast) I/O subsystems such as SSD make a
>>> bigger issue in this case.
>>
>> What about just turning compression off?
>
> I've been relying on compression for some big serialized blob fields
> for some time now. I bet I'm not alone, lots of people save serialized
> data to text fields. So rather than removing it, I'd just change the
> default to off (if that was the decision).
>
> However, it might be best to evaluate some of the modern fast
> compression schemes like snappy/lz4 (250MB/s per core sounds pretty
> good), and implement pluggable compression schemes instead. Snappy
> wasn't designed for nothing, it was most likely because it was
> necessary. Cassandra (just to name a system I'm familiar with) started
> without compression, and then it was deemed necessary to the point
> they invested considerable time into it. I've always found the fact
> that pg does compression of toast tables quite forward-thinking, and
> I'd say the feature has to remain there, extended and modernized,
> maybe off by default, but there.

I'm not offering any opinion on whether we should have compression as
a general matter.  Maybe yes, maybe no, but my question was about the
OP's use case.  If he's willing to accept less efficient compression
in order to get faster compression, perhaps he should just not use
compression at all.

Personally, my biggest gripe about the way we do compression is that
it's easy to detoast the same object lots of times.  More generally,
our in-memory representation of user data values is pretty much a
mirror of our on-disk representation, even when that leads to excess
conversions.  Beyond what we do for TOAST, there's stuff like numeric
where not only toast but then post-process the results into yet
another internal form before performing any calculations - and then of
course we have to convert back before returning from the calculation
functions.  And for things like XML, JSON, and hstore we have to
repeatedly parse the string, every time someone wants to do anything
to do.  Of course, solving this is a very hard problem, and not
solving it isn't a reason not to have more compression options - but
more compression options will not solve the problems that I personally
have in this area, by and large.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  28. Re: Improve compression speeds in pg_lzcompress.c

    Benedikt Grundmann <bgrundmann@janestreet.com> — 2013-01-09T07:56:12Z

    > Personally, my biggest gripe about the way we do compression is that
> it's easy to detoast the same object lots of times.  More generally,
> our in-memory representation of user data values is pretty much a
> mirror of our on-disk representation, even when that leads to excess
> conversions.  Beyond what we do for TOAST, there's stuff like numeric
> where not only toast but then post-process the results into yet
> another internal form before performing any calculations - and then of
> course we have to convert back before returning from the calculation
> functions.  And for things like XML, JSON, and hstore we have to
> repeatedly parse the string, every time someone wants to do anything
> to do.  Of course, solving this is a very hard problem, and not
> solving it isn't a reason not to have more compression options - but
> more compression options will not solve the problems that I personally
> have in this area, by and large.
>
> At the risk of saying something totally obvious and stupid as I haven't
looked at the actual representation this sounds like a memoisation
problem.  In ocaml terms:

type 'a rep =
  | On_disk_rep     of Byte_sequence
  | In_memory_rep of 'a

type 'a t = 'a rep ref

let get_mem_rep t converter =
  match !t with
  | On_disk_rep seq ->
    let res = converter seq in
    t := In_memory_rep res;
    res
  | In_memory_rep x -> x
;;

... (if you need the other direction that it's straightforward too)...

Translating this into c is relatively straightforward if you have the
luxury of a fresh start
and don't have to be super efficient:

typedef enum { ON_DISK_REP, IN_MEMORY_REP } rep_kind_t;

type t = {
  rep_kind_t rep_kind;
  union {
    char *on_disk;
    void *in_memory;
  } rep;
};

void *get_mem_rep(t *t, void * (*converter)(char *)) {
  void *res;
  switch (t->rep_kind) {
     case ON_DISK_REP:
        res = converter(t->on_disk);
        t->rep.in_memory = res;
        t->rep_kind = IN_MEMORY_REP;
        return res;
     case IN_MEMORY_REP;
        return t->rep.in_memory;
  }
}

Now of course fitting this into the existing types and ensuring that there
is neither too early freeing of memory nor memory leaks or other bugs is
probably a nightmare and why you said that this is a hard problem.

Cheers,

Bene