RE: Online checksums verification in the backend

Shinoda, Noriyoshi <noriyoshi.shinoda@hpe.com>

From: "Shinoda, Noriyoshi (PN Japan A&PS Delivery)" <noriyoshi.shinoda@hpe.com>
To: Michael Paquier <michael@paquier.xyz>, Julien Rouhaud <rjuju123@gmail.com>
Cc: Justin Pryzby <pryzby@telsasoft.com>, Masahiko Sawada <masahiko.sawada@2ndquadrant.com>, Robert Haas <robertmhaas@gmail.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>, Masahiko Sawada <sawada.mshk@gmail.com>
Date: 2020-10-28T23:50:35Z
Lists: pgsql-hackers

Attachments

Hi,

I have tested this great feature in the latest commit environment on Red Hat Enterprise Linux 7.8. I modified a few blocks in a relation file to raise a checksum error. When I executed the pg_relation_check_pages function, the backend terminated abnormally. The attached file is the operation log.

Regards,
Noriyoshi Shinoda

-----Original Message-----
From: Michael Paquier [mailto:michael@paquier.xyz] 
Sent: Wednesday, October 28, 2020 2:09 PM
To: Julien Rouhaud <rjuju123@gmail.com>
Cc: Justin Pryzby <pryzby@telsasoft.com>; Masahiko Sawada <masahiko.sawada@2ndquadrant.com>; Robert Haas <robertmhaas@gmail.com>; PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>; Masahiko Sawada <sawada.mshk@gmail.com>
Subject: Re: Online checksums verification in the backend

On Tue, Oct 27, 2020 at 07:47:19PM +0800, Julien Rouhaud wrote:
> I think it's also worth noting that the IOLock is now acquired just 
> before getting the buffer state, and released after the read (or after 
> finding that the buffer is dirty).  This is consistent with how it's 
> done elsewhere, so I'm fine.

Consistency is the point.  This API should be safe to use by design.
I have done some extra performance tests similar to what I did upthread, and this version showed similar numbers.

> Other than that I'm quite happy with the changes you made, thanks a lot!

Thanks for confirming.  I have gone through the whole set today, splitted the thing into two commits and applied them.  We had buildfarm member florican complain about a mistake in one of the
GetDatum() calls that I took care of already, and there is nothing else on my radar.
--
Michael

Commits

  1. Revert pg_relation_check_pages()

  2. Fix incorrect placement of pfree() in pg_relation_check_pages()

  3. Add pg_relation_check_pages() to check on-disk pages of a relation

  4. Add CheckBuffer() to check on-disk pages without shared buffer loading

  5. Extend amcheck to check heap pages.