Horton

Re: [HACKERS] Custom compression methods

Andrew Dunstan <andrew@dunslane.net>

View thread View on postgr.es
From: Andrew Dunstan <andrew@dunslane.net>
To: Tom Lane <tgl@sss.pgh.pa.us>, Robert Haas <robertmhaas@gmail.com>
Cc: Justin Pryzby <pryzby@telsasoft.com>, Dilip Kumar <dilipbalaut@gmail.com>, Alvaro Herrera <alvherre@alvh.no-ip.org>, Andres Freund <andres@anarazel.de>, Tomas Vondra <tomas.vondra@2ndquadrant.com>, Alexander Korotkov <a.korotkov@postgrespro.ru>, David Steele <david@pgmasters.net>, Ildus Kurbangaliev <i.kurbangaliev@gmail.com>, Dmitry Dolgov <9erthalion6@gmail.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2021-03-28T20:48:29Z
Lists: pgsql-hackers 
On 3/24/21 12:45 PM, Tom Lane wrote:
> Robert Haas <robertmhaas@gmail.com> writes:
>> On Wed, Mar 24, 2021 at 11:42 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>> On reflection, though, I wonder if we've made pg_dump do the right
>>> thing anyway.  There is a strong case to be made for the idea that
>>> when dumping from a pre-14 server, it should emit
>>> SET default_toast_compression = 'pglz';
>>> rather than omitting any mention of the variable, which is what
>>> I made it do in aa25d1089.
>> But also ... aren't we just doing this to work around a test case that
>> isn't especially good in the first place? Counting the number of lines
>> in the diff between A and B is an extremely crude proxy for "they're
>> similar enough that we probably haven't broken anything."
> I wouldn't be proposing this if the xversion failures were the only
> reason; making them go away is just a nice side-effect.  The core
> point is that the charter of pg_dump is to reproduce the source
> database's state, and as things stand we're failing to ensure we
> do that.
>
> (But yeah, we really need a better way of making this check in
> the xversion tests.  I don't like the arbitrary "n lines of diff
> is probably OK" business one bit.)
>
> 			



Well, I ran this module for years privately and used to have a matrix of
the exact number of diff lines expected for each combination of source
and target branch. If I didn't get that exact number of lines I reported
an error on stderr. That was fine when we weren't reporting the results
on the server, and I just sent an email to -hackers if I found an error.
I kept this matrix by examining the diffs to make sure they were all
benign. That was a pretty laborious process. So I decided to try a
heuristic approach instead, and by trial and error came up with this
2000 lines measurement. When this appeared to be working and stable the
module was released into the wild for other buildfarm owners to deploy.

Nothing is hidden here - the diffs are reported, see for example
<https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=crake&dt=2021-03-28%2015%3A37%3A07&stg=xversion-upgrade-REL9_4_STABLE-HEAD>
What we're comparing here is target pg_dumpall against the original
source vs target pg_dumpall against the upgraded source.

If someone wants to come up with a better rule for detecting that
nothing has gone wrong, I'll be happy to implement it. I don't
particularly like the current rule either, it's there faute de mieux.


cheers


andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

Commits

  1. docs: Update TOAST storage docs for configurable compression.

  2. Further tweaking of pg_dump's handling of default_toast_compression.

  3. Fix interaction of TOAST compression with expression indexes.

  4. Tidy up more loose ends related to configurable TOAST compression.

  5. Short-circuit slice requests that are for more than the object's size.

  6. Mostly-cosmetic adjustments of TOAST-related macros.

  7. Remove useless configure probe for <lz4/lz4.h>.

  8. Error on invalid TOAST compression in CREATE or ALTER TABLE.

  9. docs: Fix omissions related to configurable TOAST compression.

  10. More code cleanup for configurable TOAST compression.

  11. Bring configure support for LZ4 up to snuff.

  12. Make compression.sql regression test independent of default.

  13. Use valid compression method in brin_form_tuple

  14. Fix up pg_dump's handling of per-attribute compression options.

  15. Allow configurable LZ4 TOAST compression.

  16. Fix inconsistencies in the code

  17. Mostly-cosmetic improvements in memory chunk header alignment coding.

  18. Allow numeric to use a more compact, 2-byte header in many cases.