Re: Improve compression speeds in pg_lzcompress.c

Hi,

 >>>> Why would that be a good tradeoff to make? Larger stored values 
require
 >>>> more I/O, which is likely to swamp any CPU savings in the compression
 >>>> step. Not to mention that a value once written may be read many times,
 >>>> so the extra I/O cost could be multiplied many times over later on.
 >>> I agree with this analysis, but I note that the test results show it
 >>> actually improving things along both parameters.
 >> Hm ... one of us is reading those results backwards, then.
I think that it's a parameter-tuning issue.
I added the two parameters, PGLZ_SKIP_SIZE and PGLZ_HASH_GAP, and
set PGLZ_SKIP_SIZE=3 and PGLZ_HASH_GAP=8 for the quick tests.
And also, I found that the performance in my patch was nearly
equal to that in the current implementation when
PGLZ_SKIP_SIZE=1 and PGLZ_HASH_GAP=1.

Apart from my patch, what I care is that the current one might
be much slow against I/O. For example, when compressing
and writing large values, compressing data (20-40MiB/s) might be
a dragger against writing data in disks (50-80MiB/s). Moreover,
IMHO modern (and very fast) I/O subsystems such as SSD make a
bigger issue in this case.

Then, I think it's worth keeping discussions to improve
compression stuffs for 9.4, or later.


 > Another thing to keep in mind is that the compression area in general
 > is a minefield of patents.  We're fairly confident that pg_lzcompress
 > as-is doesn't fall foul of any, but any significant change there would
 > probably require more research.
Agree, and we know ...
we need to have patent-free ideas to improve compression issues.
For example, pluggable compression IF, or something.


 > I just went back and looked. Unless I'm misreading it he has about a 2.5
 > times speed improvement but about a 20% worse compression result.
 >
 > What would be interesting would be to see if the knobs he's tweaked
 > could be tweaked a bit more to give us substantial speedup without
 > significant space degradation.
Yes, you're right, and these results highly depend
on data sets though.


regards,
-- 
----
Takeshi Yamamuro
NTT Cyber Communications Laboratory Group
Software Innovation Center
(Open Source Software Center)
Tel: +81-3-5860-5057 Fax: +81-3-5463-5490
Mail:yamamuro.takeshi@lab.ntt.co.jp