Horton

Re: new heapcheck contrib module

Mark Dilger <mark.dilger@enterprisedb.com>

View thread View on postgr.es
From: Mark Dilger <mark.dilger@enterprisedb.com>
To: Peter Geoghegan <pg@bowt.ie>
Cc: Robert Haas <robertmhaas@gmail.com>, Tom Lane <tgl@sss.pgh.pa.us>, "Andrey M. Borodin" <x4mmm@yandex-team.ru>, Stephen Frost <sfrost@snowman.net>, Michael Paquier <michael@paquier.xyz>, Amul Sul <sulamul@gmail.com>, Dilip Kumar <dilipbalaut@gmail.com>, Andres Freund <andres@anarazel.de>, PostgreSQL-development <pgsql-hackers@postgresql.org>
Date: 2020-11-19T21:50:33Z
Lists: pgsql-hackers 

> On Nov 19, 2020, at 11:47 AM, Peter Geoghegan <pg@bowt.ie> wrote:
> 
>> I think in general you're worrying too much about the possibility of
>> this tool causing backend crashes. I think it's good that you wrote
>> the heapcheck code in a way that's hardened against that, and I think
>> we should try to harden other things as time permits. But I don't
>> think that the remote possibility of a crash due to the lack of such
>> hardening should dictate the design behavior of this tool. If the
>> crash possibilities are not remote, then I think the solution is to
>> fix them, rather than cutting out important checks.
> 
> I couldn't agree more.

Owing to how much run-time overhead it would entail, much of the backend code has not been, and probably will not be, hardened against corruption.  The amcheck code uses backend code for accessing heaps and indexes.  Only some of those uses can be preceded with sufficient safety checks to avoid stepping on landmines.  It makes sense to me to have a "don't run through minefields" option, and a "go ahead, run through minefields" option for pg_amcheck, given that users in differing situations will have differing business consequences to bringing down the server in question.

As an example that we've already looked at, checking the status of an xid against clog is a dangerous thing to do.  I wrote a patch to make it safer to query clog (0003) and a patch for pg_amcheck to use the safer interface (0004) and it looks unlikely either of those will ever be committed.  I doubt other backend hardening is any more likely to get committed.  It doesn't follow that if crash possibilities are not remote that we should therefore harden the backend.  The performance considerations of the backend are not well aligned with the safety considerations of this tool.  The backend code is written with the assumption of non-corrupt data, and this tool with the assumption of corrupt data, or at least a fair probability of corrupt data.  I don't see how any one-hardening-fits-all will ever work.

—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Commits

  1. Add pg_amcheck, a CLI for contrib/amcheck.

  2. Refactor and generalize the ParallelSlot machinery.

  3. Generalize parallel slot result handling.

  4. Move some code from src/bin/scripts to src/fe_utils to permit reuse.

  5. Factor pattern-construction logic out of processSQLNamePattern.

  6. Doc: clean up verify_heapam() documentation.

  7. Fix more portability issues in new amcheck code.

  8. Fix portability issues in new amcheck test.

  9. Try to avoid a compiler warning about using fxid uninitialized.

  10. Extend amcheck to check heap pages.

  11. Adjust walsender usage of xlogreader, simplify APIs

  12. Improve checking of child pages in contrib/amcheck.

  13. Sanitize line pointers within contrib/amcheck.

  14. Fix possible sorting error when aborting use of abbreviated keys.