Horton

Re: pg_amcheck contrib application

Robert Haas <robertmhaas@gmail.com>

View thread View on postgr.es
From: Robert Haas <robertmhaas@gmail.com>
To: Mark Dilger <mark.dilger@enterprisedb.com>
Cc: Tom Lane <tgl@sss.pgh.pa.us>, Noah Misch <noah@leadboat.com>, Peter Geoghegan <pg@bowt.ie>, Thomas Munro <thomas.munro@gmail.com>, "Andrey M. Borodin" <x4mmm@yandex-team.ru>, Stephen Frost <sfrost@snowman.net>, Michael Paquier <michael@paquier.xyz>, Amul Sul <sulamul@gmail.com>, Dilip Kumar <dilipbalaut@gmail.com>, Andres Freund <andres@anarazel.de>, PostgreSQL-development <pgsql-hackers@postgresql.org>
Date: 2021-04-09T20:51:53Z
Lists: pgsql-hackers 
On Fri, Apr 9, 2021 at 2:50 PM Mark Dilger <mark.dilger@enterprisedb.com> wrote:
> I think #4, above, requires some clarification.  If there are missing chunks, the very definition of how large we expect subsequent chunks to be is ill-defined.  I took a fairly conservative approach to avoid lots of bogus complaints about chunks that are of unexpected size.   Not all such complaints are removed, but enough are removed that I needed to add a final complaint at the end about the total size seen not matching the total size expected.

My instinct is to suppose that the size that we expect for future
chunks is independent of anything being wrong with previous chunks. So
if each chunk is supposed to be 2004 bytes (which probably isn't the
real number) and the value is 7000 bytes long, we expect chunks 0-2 to
be 2004 bytes each, chunk 3 to be 988 bytes, and chunk 4 and higher to
not exist. If chunk 1 happens to be missing or the wrong length or
whatever, our expectations for chunks 2 and 3 are utterly unchanged.

> Corruption #1:
>
>         UPDATE $toastname SET chunk_seq = chunk_seq + 1000
>
> Before:
>
> # heap table "postgres"."public"."test", block 0, offset 2, attribute 2:
> #     toast value 16445 chunk 0 has sequence number 1000, but expected sequence number 0
> # heap table "postgres"."public"."test", block 0, offset 2, attribute 2:
> #     toast value 16445 chunk 1 has sequence number 1001, but expected sequence number 1
> # heap table "postgres"."public"."test", block 0, offset 2, attribute 2:
> #     toast value 16445 chunk 2 has sequence number 1002, but expected sequence number 2
> # heap table "postgres"."public"."test", block 0, offset 2, attribute 2:
> #     toast value 16445 chunk 3 has sequence number 1003, but expected sequence number 3
> # heap table "postgres"."public"."test", block 0, offset 2, attribute 2:
> #     toast value 16445 chunk 4 has sequence number 1004, but expected sequence number 4
> # heap table "postgres"."public"."test", block 0, offset 2, attribute 2:
> #     toast value 16445 chunk 5 has sequence number 1005, but expected sequence number 5
>
> After:
>
> # heap table "postgres"."public"."test", block 0, offset 2, attribute 2:
> #     toast value 16445 missing chunks 0 through 999

Applying the above principle would lead to complaints that chunks 0-5
are missing, and 1000-1005 are extra.

> Corruption #2:
>
>         UPDATE $toastname SET chunk_seq = chunk_seq * 1000

Similarly here, except the extra chunk numbers are different.

> Corruption #3:
>
>         UPDATE $toastname SET chunk_id = (chunk_id::integer + 10000000)::oid WHERE chunk_seq = 3

And here we'd just get a complaint that chunk 3 is missing.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

Commits

  1. amcheck: Improve some confusing reports about TOAST problems.

  2. amcheck: Reword some messages and fix an alignment problem.

  3. amcheck: fix multiple problems with TOAST pointer validation

  4. amcheck: Remove duplicate XID/MXID bounds checks.

  5. amcheck: Fix verify_heapam's tuple visibility checking rules.

  6. nbtree VACUUM: Cope with buggy opclasses.

  7. Improve pg_amcheck's TAP test 003_check.pl.

  8. Fix a confusing amcheck corruption message.

  9. Doc: add note about how to run the pg_amcheck regression tests.

  10. In pg_amcheck tests, don't depend on perl's Q/q pack code.

  11. pg_amcheck: Keep trying to fix the tests.

  12. pg_amcheck: Try to fix still more test failures.

  13. Try to avoid apparent platform-dependency in IPC::Run

  14. Fix portability issues in pg_amcheck's 004_verify_heapam.pl.

  15. Try to fix compiler warnings.

  16. Add pg_amcheck, a CLI for contrib/amcheck.

  17. Refactor and generalize the ParallelSlot machinery.

  18. Remove old-style VACUUM FULL (which was known for a little while as