Horton

Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index.

Pavel Borisov <pashkin.elfe@gmail.com>

View thread
From: Pavel Borisov <pashkin.elfe@gmail.com>
To: Peter Geoghegan <pg@bowt.ie>
Cc: Postgres hackers <pgsql-hackers@lists.postgresql.org>
Date: 2021-03-03T08:08:00Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. amcheck: Optimize speed of checking for unique constraint violation

  2. amcheck: Report an error when the next page to a leaf is not a leaf

  3. amcheck: Don't load the right sibling page into BtreeCheckState

  4. amcheck: Refactoring the storage of the last visible entry

  5. Teach contrib/amcheck to check the unique constraint violation

  6. Add macros in hash and btree AMs to get the special area of their pages

>
> You're going to have to "couple" buffer locks in the style of
> _bt_check_unique() (as well as keeping a buffer lock on "the first
> leaf page a duplicate might be on" throughout) if you need the test to
> work reliably. But why bother with that? The tool doesn't have to be
> 100% perfect at detecting corruption (nothing can be), and it's rather
> unlikely that it will matter for this test. A simple test that doesn't
> handle cross-page duplicates is still going to be very effective.
>

Indeed at first, I did the test which doesn't bother checking duplicates
cross-page which I considered very rare, but then a customer sent me his
corrupted index where I found this rare thing which was not detectable by
amcheck and he was puzzled with the issue. Even rare inconsistencies can
appear when people handle huge amounts of data. So I did an update that
handles a wider class of errors. I don't suppose that cross page unique
check is expensive as it uses same things that are already used in amcheck
for cross-page checks.

Is it suitable if I omit suspected duplicates message in the very-very rare
case amcheck can not detect but leave cross-page checks?

>>I don't think that it's acceptable for your new check to raise a
>>WARNING instead of an ERROR.
It is not instead of an ERROR. If at least one violation is detected,
amcheck will output the final ERROR message. The purpose is not to stop
checking at the first violation. But I can make them reported in a current
amcheck style if it is necessary.

-- 
Best regards,
Pavel Borisov

Postgres Professional: http://postgrespro.com <http://www.postgrespro.com>