Horton

Thread

Commits

  1. amcheck: Improve some confusing reports about TOAST problems.

  2. amcheck: Reword some messages and fix an alignment problem.

  3. amcheck: fix multiple problems with TOAST pointer validation

  4. amcheck: Remove duplicate XID/MXID bounds checks.

  5. amcheck: Fix verify_heapam's tuple visibility checking rules.

  6. nbtree VACUUM: Cope with buggy opclasses.

  7. Improve pg_amcheck's TAP test 003_check.pl.

  8. Fix a confusing amcheck corruption message.

  9. Doc: add note about how to run the pg_amcheck regression tests.

  10. In pg_amcheck tests, don't depend on perl's Q/q pack code.

  11. pg_amcheck: Keep trying to fix the tests.

  12. pg_amcheck: Try to fix still more test failures.

  13. Try to avoid apparent platform-dependency in IPC::Run

  14. Fix portability issues in pg_amcheck's 004_verify_heapam.pl.

  15. Try to fix compiler warnings.

  16. Add pg_amcheck, a CLI for contrib/amcheck.

  17. Refactor and generalize the ParallelSlot machinery.

  18. Remove old-style VACUUM FULL (which was known for a little while as

  1. pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-03-03T15:22:28Z

    New thread, was "Re: new heapcheck contrib module"


> On Mar 2, 2021, at 10:24 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> 
> On Tue, Mar 2, 2021 at 12:10 PM Mark Dilger
> <mark.dilger@enterprisedb.com> wrote:
>> On further reflection, I decided to implement these changes and not worry about the behavioral change.
> 
> Thanks.
> 
>> I skipped this part.  The initcmd argument is only handed to ParallelSlotsGetIdle().  Doing as you suggest would not really be simpler, it would just move that argument to ParallelSlotsSetup().  But I don't feel strongly about it, so I can move this, too, if you like.
>> 
>> I didn't do this either, and for the same reason.  It's just a parameter to ParallelSlotsGetIdle(), so nothing is really gained by moving it to ParallelSlotsSetup().
> 
> OK. I thought it was more natural to pass a bunch of arguments at
> setup time rather than passing a bunch of arguments at get-idle time,
> but I don't feel strongly enough about it to insist, and somebody else
> can always change it later if they decide I had the right idea.

When you originally proposed the idea, I thought that it would work out as a simpler interface to have it your way, but in terms of the interface it came out about the same.  Internally it is still simpler to do it your way, so since you seem to still like your way better, this next version has it that way.

>> Rather than the slots user tweak the slot's ConnParams, ParallelSlotsGetIdle() takes a dbname argument, and uses it as ConnParams->override_dbname.
> 
> OK, but you forgot to update the comments. ParallelSlotsGetIdle()
> still talks about a cparams argument that it no longer has.

Fixed.

> The usual idiom for sizing a memory allocation involving
> FLEXIBLE_ARRAY_MEMBER is something like offsetof(ParallelSlotArray,
> slots) + numslots * sizeof(ParallelSlot). Your version uses sizeof();
> don't.

Fixed.

> Other than that 0001 looks to me to be in pretty good shape now.




And your other review email, also moved to this new thread....

> On Mar 2, 2021, at 12:39 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> 
> On Tue, Mar 2, 2021 at 1:24 PM Robert Haas <robertmhaas@gmail.com> wrote:
>> Other than that 0001 looks to me to be in pretty good shape now.
> 
> Incidentally, we might want to move this to a new thread with a better
> subject line, since the current subject line really doesn't describe
> the uncommitted portion of the work. And create a new CF entry, too.

Moved here.

> Moving onto 0002:
> 
> The index checking options should really be called btree index
> checking options. I think I'd put the table options first, and the
> btree options second. Other kinds of indexes could follow some day. I
> would personally omit the short forms of --heapallindexed and
> --parent-check; I think we'll run out of option names too quickly if
> people add more kinds of checks.

Done.

While doing this, I also renamed some of the variables to more closely match the option name.  I think the code is clearer now.

> 
> Perhaps VerifyBtreeSlotHandler should emit a warning of some kind if
> PQntuples(res) != 0.

The functions bt_index_check and bt_index_parent_check are defined to return VOID, which results in PQntuples(res) == 1.  I added code to verify this condition, but it only serves to alert the user if the amcheck version is behaving in an unexpected way, perhaps due to a amcheck/pg_amcheck version mismatch.

> +               /*
> +                * Test that this function works, but for now we're
> not using the list
> +                * 'relations' that it builds.
> +                */
> +               conn = connectDatabase(&cparams, progname, opts.echo,
> false, true);
> 
> This comment appears to have nothing to do with the code, since
> connectDatabase() does not build a list of 'relations'.

True.  Removed.

> amcheck_sql seems to include paranoia, but do we need that if we're
> using a secure search path? Similarly for other SQL queries, e.g. in
> prepare_table_command.

I removed the OPERATOR(pg_catalog.=) paranoia.

> It might not be strictly necessary for the static functions in
> pg_amcheck.c to use_three completelyDifferent NamingConventions for
> its static functions.

The idea is that the functions that interoperate with parallel slots would follow its NamingConvention; those interoperating with patternToSQLRegex and PQExpBuffers would follow their namingConvention; and those not so interoperating would follow a less obnoxious naming_convention.  To my eye, that color codes the function names in a useful way.  To your eye, it just looks awful.  I've changed it to use just one naming_convention.

> should_processing_continue() is one semicolon over budget.

That's not the first time I've done that recently.  Removed.

> The initializer for opts puts a comma even after the last member
> initializer. Is that going to be portable to all compilers?

I don't know.  I learned to put commas at the end of lists back when I did mostly perl programming, as you get cleaner diffs when you add more stuff to the list later.  Whether I can get away with that in C using initializers I don't know.  I don't have a multiplicity of compilers to check.

I have removed the extra comma.

> +   for (failed = false, cell = opts.include.head; cell; cell = cell->next)
> 
> I think failed has to be false here, because it gets initialized at
> the top of the function. If we need to reinitialize it for some
> reason, I would prefer you do that on the previous line, separate from
> the for loop stuff.

It does have to be false there.  There is no need to reinitialize it.

> +   char       *dbrgx;          /* Database regexp parsed from pattern, or
> +                                * NULL */
> +   char       *nsprgx;         /* Schema regexp parsed from pattern, or NULL */
> +   char       *relrgx;         /* Relation regexp parsed from pattern, or
> +                                * NULL */
> +   bool        tblonly;        /* true if relrgx should only match tables */
> +   bool        idxonly;        /* true if relrgx should only match indexes */
> 
> Maybe: db_regex, nsp_regex, rel_regex, table_only, index_only?
> 
> Just because it seems theoretically possible that someone will see
> nsprgx and not immediately understand what it's supposed to mean, even
> if they know that nsp is a common abbreviation for namespace in
> PostgreSQL code, and even if they also know what a regular expression
> is.

Changed.  Along the way, I noticed that "tbl" and "idx" were being used in C/SQL both to mean ("table_only", "index_only") in some contexts and ("is_table", "is_index') in others, so I replaced all instances of "tbl" and "idx" with the unambiguous labels.

> Your four messages about there being nothing to check seem like they
> could be consolidated down to one: "nothing to check for pattern
> \"%s\"".

I anticipated your review comment, but I'm worried about the case that somebody runs

  pg_amcheck -t "foo" -i "foo"

and one of those matches and the other does not.  The message 'nothing to check for pattern "foo"' will be wrong (because there was something to check for it) and unhelpful (because it doesn't say which failed to match.)

> I would favor changing things so that once argument parsing is
> complete, we switch to reporting all errors that way. So in other
> words here, and everything that follows:
> 
> +               fprintf(stderr, "%s: no databases to check\n", progname);

Same concern about the output for

  pg_amcheck -t "foo" -i "foo" -d "foo"

You might think I'm being silly here, as database names, table names, and index names should in normal usage not be hard for the user to distinguish.  But consider

  pg_amcheck "mydb.myschema.mytable"

If it says, 'nothing to check for pattern "mydb.myschema.mytable"', you don't know if the database doesn't exist or if the table doesn't exist.

> 
> +        * ParallelSlots based event loop follows.
> 
> "Main event loop."

Changed.

> To me it would read slightly better to change each reference to
> "relations list" to "list of relations", but perhaps that is too
> nitpicky.

No harm picking those nits.  Changed.

> I think the two instances of goto finish could be avoided with not
> much work. At most a few things need to happen only if !failed, and
> maybe not even that, if you just said "break;" instead.

Good point.  Changed.

> + * Note: Heap relation corruption is returned by verify_heapam() without the
> + * use of raising errors, but running verify_heapam() on a corrupted table may
> 
> How about "Heap relation corruption() is reported by verify_heapam()
> via the result set, rather than an ERROR, ..."

Changed, though I assumed your parens for corruption() were not intended.



Ok, so now you've moved on to reviewing the regression tests....

> It seems mighty inefficient to have a whole bunch of consecutive calls
> to remove_relation_file() or corrupt_first_page() when every such call
> stops and restarts the database. I would guess these tests will run
> noticeably faster if you don't do that. Either the functions need to
> take a list of arguments, or the stop/start needs to be pulled up and
> done in the caller.

Changed.

> corrupt_first_page() could use a comment explaining what exactly we're
> overwriting, and in particular noting that we don't want to just
> clobber the LSN, but rather something where we can detect a wrong
> value.

Added comments that we're skipping past the PageHeader and overwriting garbage starting in the line pointers.

> There's a long list of calls to command_checks_all() in 003_check.pl
> that don't actually check anything but that the command failed, but
> run it with a bunch of different options. I don't understand the value
> of that, and suggest reducing the number of cases tested. If you want,
> you can have tests elsewhere that focus -- perhaps by using verbose
> mode -- on checking that the right tables are being checked.

This should be better in this next patch series.

> 
> This is not yet a full review of everything in this patch -- I haven't
> sorted through all of the tests yet, or all of the new query
> construction logic -- but to me this looks pretty close to
> committable.

Thanks for the review!

  2. Re: pg_amcheck contrib application

    Robert Haas <robertmhaas@gmail.com> — 2021-03-03T17:15:49Z

    On Wed, Mar 3, 2021 at 10:22 AM Mark Dilger
<mark.dilger@enterprisedb.com> wrote:
> > Your four messages about there being nothing to check seem like they
> > could be consolidated down to one: "nothing to check for pattern
> > \"%s\"".
>
> I anticipated your review comment, but I'm worried about the case that somebody runs
>
>   pg_amcheck -t "foo" -i "foo"
>
> and one of those matches and the other does not.  The message 'nothing to check for pattern "foo"' will be wrong (because there was something to check for it) and unhelpful (because it doesn't say which failed to match.)

Fair point.

> Changed, though I assumed your parens for corruption() were not intended.

Uh, yeah.

> Thanks for the review!

+        fprintf(stderr, "%s: no relations to check", progname);

Missing newline.

Generally, I would favor using pg_log_whatever as a way of reporting
messages starting when option parsing is complete. In other words,
starting here:

+        fprintf(stderr, "%s: no databases to check\n", progname);

I see no real advantage in having a bunch of these using
fprintf(stderr, ...), which to me seems most appropriate only for very
early failures.

Perhaps amcheck_sql could be spread across fewer lines, now that it
doesn't have so many decorations?

pg_basebackup uses -P as a short form for --progress, so maybe we
should match that here.

When I do "pg_amcheck --progress", it just says "259/259 (100%)" which
I don't find too clear. The corresponding pg_basebackup output is
"32332/32332 kB (100%), 1/1 tablespace" which has the advantage of
including units. I think if you just add the word "relations" to your
message it will be nicer.

When I do "pg_amcheck -s public" it tells me that there are no
relations to check in schemas for "public". I think "schemas matching"
would read better than "schemas for." Similar with the other messages.
When I try "pg_amcheck -t nessie" it tells me that there are no tables
to check for "nessie" but saying that there are no tables to check
matching "nessie" to me sounds more natural.

The code doesn't seem real clear on the difference between a database
name and a pattern. Consider:

createdb rhaas
createdb 'rh*s'
PGDATABASE='rh*s' pg_amcheck

It checks the rhaas database, which I venture to say is just plain wrong.

The error message when I exclude the only checkable database is not
very clear. "pg_amcheck -D rhaas"  says pg_amcheck: no checkable
database: "rhaas". Well, I get that there's no checkable database. But
as a user I have no idea what "rhaas" is. I can even get it to issue
this complaint more than once:

createdb q
createdb qq
pg_amcheck -D 'q*' q qq

Now it issues the "no checkable database" complaint twice, once for q
and once for qq. But if there's no checkable database, I only need to
know that once. Either the message is wrongly-worded, or it should
only be issued once and doesn't need to include the pattern. I think
it's the second one, but I could be wrong.

Using a pattern as the only or first argument doesn't work; i.e.
"pg_amcheck rhaas" works but "pg_amcheck rhaa?" fails because there is
no database with that exact literal name. This seems like another
instance of confusion between a literal database name and a database
name pattern. I'm not quite sure what the right solution is here. We
could give up on having database patterns altogether -- the comparable
issue does not arise for database and schema name patterns -- or the
maintenance database could default to something that's not going to be
a pattern, like "postgres," rather than being taken from a
command-line argument that is intended to be a pattern. Or some hybrid
approach e.g. -d options are patterns, but don't set the maintenance
database, while extra command line arguments are literal database
names, and thus are presumably OK to use as the maintenance DB. But
it's too weird IMHO to support patterns here and then have supplying
one inevitably fail unless you also specify --maintenance-db.

It's sorta annoying that there doesn't seem to be an easy way to find
out exactly what relations got checked as a result of whatever I did.
Perhaps pg_amcheck -v should print a line for each relation saying
that it's checking that relation; it's not actually that verbose as
things stand. If we thought that was overdoing it, we could set things
up so that multiple -v options keep increasing the verbosity level, so
that you can get this via pg_amcheck -vv. I submit that pg_amcheck -e
is not useful for this purpose because the queries, besides being
long, use the relation OIDs rather than the names, so it's not easy to
see what happened.

I think that something's not working in terms of schema exclusion. If
I create a brand-new database and then run "pg_amcheck -S pg_catalog
-S information_schema -S pg_toast" it still checks stuff. In fact it
seems to check the exact same amount of stuff that it checks if I run
it with no command-line options at all. In fact, if I run "pg_amcheck
-S '*'" that still checks everything. Unless I'm misunderstanding what
this option is supposed to do, the fact that a version of this patch
where this seemingly doesn't work at all escaped to the list suggests
that your testing has got some gaps.

I like the semantics of --no-toast-expansion and --no-index-expansion
as you now have them, but I find I don't really like the names. Could
I suggest --no-dependent-indexes and --no-dependent-toast?

I tried pg_amcheck --startblock=tsgsdg and got an error message
without a trailing newline. I tried --startblock=-525523 and got no
error. I tried --startblock=99999999999999999999999999 and got a
complaint that the value was out of bounds, but without a trailing
newline. Maybe there's an argument that the bounds don't need to be
checked, but surely there's no argument for checking one and not the
other. I haven't tried the corresponding cases with --endblock but you
should. I tried --startblock=2 --endblock=1 and got a complaint that
the ending block precedes the starting block, which is totally
reasonable (though I might say "start block" and "end block" rather
than using the -ing forms) but this message is prefixed with
"pg_amcheck: " whereas the messages about an altogether invalid
starting block where not so prefixed. Is there a reason not to make
this consistent?

I also tried using a random positive integer for startblock, and for
every relation I am told "ERROR:  starting block number must be
between 0 and <whatever>". That makes sense, because I used a big
number for the start block and I don't have any big relations, but it
makes for an absolute ton of output, because every verify_heapam query
is 11 lines long. This suggests a couple of possible improvements.
First, maybe we should only display the query that produced the error
in verbose mode. Second, maybe the verify_heapam() query should be
tightened up so that it doesn't stretch across quite so many lines. I
think the call to verify_heapam() could be spread across like 2 lines
rather than 7, which would improve readability. On a related note, I
wonder why we need every verify_heapam() call to join to pg_class and
pg_namespace just to fetch the schema and table name which,
presumably, we should or at least could already have. This kinda
relates to my comment earlier about making -v print a message per
relation so that we can see, in human-readable format, which relations
are getting checked. Right now, if you got an error checking just one
relation, how would you know which relation you got it from? Unless
the server happens to report that information in the message, you're
just in the dark, because pg_amcheck won't tell you.

The line "Read the description of the amcheck contrib module for
details" seems like it could be omitted. Perhaps the first line of the
help message could be changed to read "pg_amcheck uses amcheck to find
corruption in a PostgreSQL database." or something like that, instead.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  3. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-03-04T06:25:56Z

    
> On Mar 3, 2021, at 9:15 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> 
> On Wed, Mar 3, 2021 at 10:22 AM Mark Dilger
> <mark.dilger@enterprisedb.com> wrote:
>>> Your four messages about there being nothing to check seem like they
>>> could be consolidated down to one: "nothing to check for pattern
>>> \"%s\"".
>> 
>> I anticipated your review comment, but I'm worried about the case that somebody runs
>> 
>>  pg_amcheck -t "foo" -i "foo"
>> 
>> and one of those matches and the other does not.  The message 'nothing to check for pattern "foo"' will be wrong (because there was something to check for it) and unhelpful (because it doesn't say which failed to match.)
> 
> Fair point.
> 
>> Changed, though I assumed your parens for corruption() were not intended.
> 
> Uh, yeah.
> 
>> Thanks for the review!
> 
> +        fprintf(stderr, "%s: no relations to check", progname);
> 
> Missing newline.
> 
> Generally, I would favor using pg_log_whatever as a way of reporting
> messages starting when option parsing is complete. In other words,
> starting here:
> 
> +        fprintf(stderr, "%s: no databases to check\n", progname);
> 
> I see no real advantage in having a bunch of these using
> fprintf(stderr, ...), which to me seems most appropriate only for very
> early failures.

Ok, the newline issues should be fixed, and the use of pg_log_{error,warning,info} is now used more consistently.

> Perhaps amcheck_sql could be spread across fewer lines, now that it
> doesn't have so many decorations?

Done.

> pg_basebackup uses -P as a short form for --progress, so maybe we
> should match that here.

Done.

> When I do "pg_amcheck --progress", it just says "259/259 (100%)" which
> I don't find too clear. The corresponding pg_basebackup output is
> "32332/32332 kB (100%), 1/1 tablespace" which has the advantage of
> including units. I think if you just add the word "relations" to your
> message it will be nicer.

Done.  It now shows:

% pg_amcheck -P
259/259 relations (100%) 870/870 pages (100%)

As you go along, the percent of relations processed may not be equal to the percent of pages, though at the end they are both 100%.  The value of printing both can only be seen while things are underway.

> When I do "pg_amcheck -s public" it tells me that there are no
> relations to check in schemas for "public". I think "schemas matching"
> would read better than "schemas for." Similar with the other messages.
> When I try "pg_amcheck -t nessie" it tells me that there are no tables
> to check for "nessie" but saying that there are no tables to check
> matching "nessie" to me sounds more natural.

Done.

% pg_amcheck -s public
pg_amcheck: error: no relations to check in schemas matching "public"

> The code doesn't seem real clear on the difference between a database
> name and a pattern. Consider:
> 
> createdb rhaas
> createdb 'rh*s'
> PGDATABASE='rh*s' pg_amcheck
> 
> It checks the rhaas database, which I venture to say is just plain wrong.

This next version treats any arguments supplied with -d and -D as database patterns, and all others as database names.  Exclusion patterns (-D) only override inclusion patterns, not names. 

> The error message when I exclude the only checkable database is not
> very clear. "pg_amcheck -D rhaas"  says pg_amcheck: no checkable
> database: "rhaas". Well, I get that there's no checkable database. But
> as a user I have no idea what "rhaas" is. I can even get it to issue
> this complaint more than once:
> 
> createdb q
> createdb qq
> pg_amcheck -D 'q*' q qq
> 
> Now it issues the "no checkable database" complaint twice, once for q
> and once for qq. But if there's no checkable database, I only need to
> know that once. Either the message is wrongly-worded, or it should
> only be issued once and doesn't need to include the pattern. I think
> it's the second one, but I could be wrong.

I think this whole problem goes away with the change to how -D/-d work and don't interact with database names.  At least, I don't get any problems like the one you mention:

% PGDATABASE=postgres pg_amcheck -D postgres
pg_amcheck: warning: skipping database "postgres": amcheck is not installed
pg_amcheck: error: no relations to check

% PGDATABASE=mark.dilger pg_amcheck -D mark.dilger --progress
259/259 relations (100%) 870/870 pages (100%)

> Using a pattern as the only or first argument doesn't work; i.e.
> "pg_amcheck rhaas" works but "pg_amcheck rhaa?" fails because there is
> no database with that exact literal name. This seems like another
> instance of confusion between a literal database name and a database
> name pattern. I'm not quite sure what the right solution is here. We
> could give up on having database patterns altogether -- the comparable
> issue does not arise for database and schema name patterns -- or the
> maintenance database could default to something that's not going to be
> a pattern, like "postgres," rather than being taken from a
> command-line argument that is intended to be a pattern. Or some hybrid
> approach e.g. -d options are patterns, but don't set the maintenance
> database, while extra command line arguments are literal database
> names, and thus are presumably OK to use as the maintenance DB. But
> it's too weird IMHO to support patterns here and then have supplying
> one inevitably fail unless you also specify --maintenance-db.

Right.  I think the changes in this next version address all your concerns as stated, but here are some examples:

% pg_amcheck "mark.d*" --progress         
pg_amcheck: error: connection to server on socket "/tmp/.s.PGSQL.5432" failed: FATAL:  database "mark.d*" does not exist

% PGDATABASE=postgres pg_amcheck "mark.d*" --progress
pg_amcheck: error: connection to server on socket "/tmp/.s.PGSQL.5432" failed: FATAL:  database "mark.d*" does not exist

% PGDATABASE=postgres pg_amcheck -d "mark.d*" --progress
520/520 relations (100%) 1815/1815 pages (100%)

% pg_amcheck --all --maintenance-db="mark.d*" --progress
pg_amcheck: error: connection to server on socket "/tmp/.s.PGSQL.5432" failed: FATAL:  database "mark.d*" does not exist

% pg_amcheck --all -D="mark.d*" --progress
pg_amcheck: warning: skipping database "template1": amcheck is not installed
520/520 relations (100%) 1815/1815 pages (100%)

> It's sorta annoying that there doesn't seem to be an easy way to find
> out exactly what relations got checked as a result of whatever I did.
> Perhaps pg_amcheck -v should print a line for each relation saying
> that it's checking that relation; it's not actually that verbose as
> things stand. If we thought that was overdoing it, we could set things
> up so that multiple -v options keep increasing the verbosity level, so
> that you can get this via pg_amcheck -vv. I submit that pg_amcheck -e
> is not useful for this purpose because the queries, besides being
> long, use the relation OIDs rather than the names, so it's not easy to
> see what happened.

I added that, as shown here:

% pg_amcheck mark.dilger --table=pg_subscription --table=pg_publication -v  
pg_amcheck: in database "mark.dilger": using amcheck version "1.3" in schema "public"
pg_amcheck: checking btree index "mark.dilger"."pg_toast"."pg_toast_6100_index" (oid 4184) (1/1 page)
pg_amcheck: checking btree index "mark.dilger"."pg_catalog"."pg_publication_oid_index" (oid 6110) (1/1 page)
pg_amcheck: checking btree index "mark.dilger"."pg_catalog"."pg_publication_pubname_index" (oid 6111) (1/1 page)
pg_amcheck: checking btree index "mark.dilger"."pg_catalog"."pg_subscription_oid_index" (oid 6114) (1/1 page)
pg_amcheck: checking btree index "mark.dilger"."pg_catalog"."pg_subscription_subname_index" (oid 6115) (1/1 page)
pg_amcheck: checking table "mark.dilger"."pg_toast"."pg_toast_6100" (oid 4183) (0/0 pages)
pg_amcheck: checking table "mark.dilger"."pg_catalog"."pg_subscription" (oid 6100) (0/0 pages)
pg_amcheck: checking table "mark.dilger"."pg_catalog"."pg_publication" (oid 6104) (0/0 pages)

> I think that something's not working in terms of schema exclusion. If
> I create a brand-new database and then run "pg_amcheck -S pg_catalog
> -S information_schema -S pg_toast" it still checks stuff. In fact it
> seems to check the exact same amount of stuff that it checks if I run
> it with no command-line options at all. In fact, if I run "pg_amcheck
> -S '*'" that still checks everything. Unless I'm misunderstanding what
> this option is supposed to do, the fact that a version of this patch
> where this seemingly doesn't work at all escaped to the list suggests
> that your testing has got some gaps.

Good catch.  That works now, but beware that -S doesn't apply to excluding things brought in by toast or index expansion, so:

% pg_amcheck mark.dilger -S pg_catalog -S pg_toast --progress -v
pg_amcheck: in database "mark.dilger": using amcheck version "1.3" in schema "public"
 0/14 relations (0%)  0/90 pages (0%)                                      
pg_amcheck: checking table "mark.dilger"."public"."foo" (oid 16385) (45/45 pages)
pg_amcheck: checking btree index "mark.dilger"."public"."foo_idx" (oid 16388) (30/30 pages)
pg_amcheck: checking table "mark.dilger"."information_schema"."sql_features" (oid 13051) (8/8 pages)
pg_amcheck: checking btree index "mark.dilger"."pg_toast"."pg_toast_13051_index" (oid 13055) (1/1 page)
pg_amcheck: checking table "mark.dilger"."information_schema"."sql_implementation_info" (oid 13056) (1/1 page)
pg_amcheck: checking btree index "mark.dilger"."pg_toast"."pg_toast_13056_index" (oid 13060) (1/1 page)
pg_amcheck: checking table "mark.dilger"."information_schema"."sql_parts" (oid 13061) (1/1 page)
pg_amcheck: checking btree index "mark.dilger"."pg_toast"."pg_toast_13061_index" (oid 13065) (1/1 page)
pg_amcheck: checking table "mark.dilger"."information_schema"."sql_sizing" (oid 13066) (1/1 page)
pg_amcheck: checking btree index "mark.dilger"."pg_toast"."pg_toast_13066_index" (oid 13070) (1/1 page)
pg_amcheck: checking table "mark.dilger"."pg_toast"."pg_toast_13051" (oid 13054) (0/0 pages)
pg_amcheck: checking table "mark.dilger"."pg_toast"."pg_toast_13056" (oid 13059) (0/0 pages)
pg_amcheck: checking table "mark.dilger"."pg_toast"."pg_toast_13061" (oid 13064) (0/0 pages)
pg_amcheck: checking table "mark.dilger"."pg_toast"."pg_toast_13066" (oid 13069) (0/0 pages)
14/14 relations (100%) 90/90 pages (100%)

but

% pg_amcheck mark.dilger -S pg_catalog -S pg_toast -S information_schema --progress -v
pg_amcheck: in database "mark.dilger": using amcheck version "1.3" in schema "public"
0/2 relations (0%)  0/75 pages (0%)                                      
pg_amcheck: checking table "mark.dilger"."public"."foo" (oid 16385) (45/45 pages)
pg_amcheck: checking btree index "mark.dilger"."public"."foo_idx" (oid 16388) (30/30 pages)
2/2 relations (100%) 75/75 pages (100%)

The first one checks so much because the toast and indexes for tables in the "information_schema" are not excluded by -S, but:

% pg_amcheck mark.dilger -S pg_catalog -S pg_toast --progress --no-dependent-indexes --no-dependent-toast -v
pg_amcheck: in database "mark.dilger": using amcheck version "1.3" in schema "public"
0/5 relations (0%)  0/56 pages (0%)                                      
pg_amcheck: checking table "mark.dilger"."public"."foo" (oid 16385) (45/45 pages)
pg_amcheck: checking table "mark.dilger"."information_schema"."sql_features" (oid 13051) (8/8 pages)
pg_amcheck: checking table "mark.dilger"."information_schema"."sql_implementation_info" (oid 13056) (1/1 page)
pg_amcheck: checking table "mark.dilger"."information_schema"."sql_parts" (oid 13061) (1/1 page)
pg_amcheck: checking table "mark.dilger"."information_schema"."sql_sizing" (oid 13066) (1/1 page)
5/5 relations (100%) 56/56 pages (100%)

works as you might expect.

> I like the semantics of --no-toast-expansion and --no-index-expansion
> as you now have them, but I find I don't really like the names. Could
> I suggest --no-dependent-indexes and --no-dependent-toast?

Changed.

> I tried pg_amcheck --startblock=tsgsdg and got an error message
> without a trailing newline.

Fixed.

> I tried --startblock=-525523 and got no
> error.

Fixed.

> I tried --startblock=99999999999999999999999999 and got a
> complaint that the value was out of bounds, but without a trailing
> newline.

Fixed.

> Maybe there's an argument that the bounds don't need to be
> checked, but surely there's no argument for checking one and not the
> other.

It checks both now, and also for --endblock

> I haven't tried the corresponding cases with --endblock but you
> should. I tried --startblock=2 --endblock=1 and got a complaint that
> the ending block precedes the starting block, which is totally
> reasonable (though I might say "start block" and "end block" rather
> than using the -ing forms)

I think this is fixed up now.  There is an interaction with amcheck's verify_heapam(), where that function raises an error if the startblock or endblock arguments are out of bounds for the relation in question.  Rather than aborting the entire pg_amcheck run, it avoids passing inappropriate block ranges to verify_heapam() and outputs a warning, so:

% pg_amcheck mark.dilger -t foo -t pg_class --progress -v --startblock=35 --endblock=77
pg_amcheck: in database "mark.dilger": using amcheck version "1.3" in schema "public"
0/6 relations (0%)  0/55 pages (0%)                                      
pg_amcheck: checking table "mark.dilger"."public"."foo" (oid 16385) (10/45 pages)
pg_amcheck: warning: ignoring endblock option 77 beyond end of table "mark.dilger"."public"."foo"
pg_amcheck: checking btree index "mark.dilger"."public"."foo_idx" (oid 16388) (30/30 pages)
pg_amcheck: checking table "mark.dilger"."pg_catalog"."pg_class" (oid 1259) (0/13 pages)
pg_amcheck: warning: ignoring startblock option 35 beyond end of table "mark.dilger"."pg_catalog"."pg_class"
pg_amcheck: warning: ignoring endblock option 77 beyond end of table "mark.dilger"."pg_catalog"."pg_class"
pg_amcheck: checking btree index "mark.dilger"."pg_catalog"."pg_class_relname_nsp_index" (oid 2663) (6/6 pages)
pg_amcheck: checking btree index "mark.dilger"."pg_catalog"."pg_class_tblspc_relfilenode_index" (oid 3455) (5/5 pages)
pg_amcheck: checking btree index "mark.dilger"."pg_catalog"."pg_class_oid_index" (oid 2662) (4/4 pages)
6/6 relations (100%) 55/55 pages (100%) 

The way the (x/y pages) is printed takes into account that the [startblock..endblock] range may reduce the number of pages to check (x) to something less than the number of pages in the relation (y), but the reporting is a bit of a lie when the startblock is beyond the end of the table, as it doesn't get passed to verify_heapam and so the number of blocks checked may be more than the zero blocks reported. I think I might need to fix this up tomorrow, but I want to get what I have in this patch set posted tonight, so it's not fixed here.  Also, there are multiple ways of addressing this, and I'm having trouble deciding which way is best.  I can exclude the relation from being checked at all, or realize earlier that I'm not going to honor the startblock argument and compute the blocks to check correctly.  Thoughts?

> but this message is prefixed with
> "pg_amcheck: " whereas the messages about an altogether invalid
> starting block where not so prefixed. Is there a reason not to make
> this consistent?

That was a stray usage of pg_log_error where fprintf should have been used.  Fixed.

> I also tried using a random positive integer for startblock, and for
> every relation I am told "ERROR:  starting block number must be
> between 0 and <whatever>". That makes sense, because I used a big
> number for the start block and I don't have any big relations, but it
> makes for an absolute ton of output, because every verify_heapam query
> is 11 lines long.

This happens because the range was being passed down to verify_heapam.  It won't do that now.

> This suggests a couple of possible improvements.
> First, maybe we should only display the query that produced the error
> in verbose mode.

No longer relevant.

> Second, maybe the verify_heapam() query should be
> tightened up so that it doesn't stretch across quite so many lines.

Not a bad idea, but no longer relevant to the startblock/endblock issues.  Done.

> I
> think the call to verify_heapam() could be spread across like 2 lines
> rather than 7, which would improve readability.

Done.

> On a related note, I
> wonder why we need every verify_heapam() call to join to pg_class and
> pg_namespace just to fetch the schema and table name which,
> presumably, we should or at least could already have.

We didn't have it, but we do now, so that join is removed.

> This kinda
> relates to my comment earlier about making -v print a message per
> relation so that we can see, in human-readable format, which relations
> are getting checked.

Done.

> Right now, if you got an error checking just one
> relation, how would you know which relation you got it from? Unless
> the server happens to report that information in the message, you're
> just in the dark, because pg_amcheck won't tell you.

That information is now included in the query text, so you can see it in the error message along with the oid.

> 
> The line "Read the description of the amcheck contrib module for
> details" seems like it could be omitted. Perhaps the first line of the
> help message could be changed to read "pg_amcheck uses amcheck to find
> corruption in a PostgreSQL database." or something like that, instead.

Done.

  4. Re: pg_amcheck contrib application

    Robert Haas <robertmhaas@gmail.com> — 2021-03-04T15:29:12Z

    Most of these changes sound good. I'll go through the whole patch
again today, or as much of it as I can. But before I do that, I want
to comment on this point specifically.

On Thu, Mar 4, 2021 at 1:25 AM Mark Dilger <mark.dilger@enterprisedb.com> wrote:
> I think this is fixed up now.  There is an interaction with amcheck's verify_heapam(), where that function raises an error if the startblock or endblock arguments are out of bounds for the relation in question.  Rather than aborting the entire pg_amcheck run, it avoids passing inappropriate block ranges to verify_heapam() and outputs a warning, so:
>
> % pg_amcheck mark.dilger -t foo -t pg_class --progress -v --startblock=35 --endblock=77
> pg_amcheck: in database "mark.dilger": using amcheck version "1.3" in schema "public"
> 0/6 relations (0%)  0/55 pages (0%)
> pg_amcheck: checking table "mark.dilger"."public"."foo" (oid 16385) (10/45 pages)
> pg_amcheck: warning: ignoring endblock option 77 beyond end of table "mark.dilger"."public"."foo"
> pg_amcheck: checking btree index "mark.dilger"."public"."foo_idx" (oid 16388) (30/30 pages)
> pg_amcheck: checking table "mark.dilger"."pg_catalog"."pg_class" (oid 1259) (0/13 pages)
> pg_amcheck: warning: ignoring startblock option 35 beyond end of table "mark.dilger"."pg_catalog"."pg_class"
> pg_amcheck: warning: ignoring endblock option 77 beyond end of table "mark.dilger"."pg_catalog"."pg_class"
> pg_amcheck: checking btree index "mark.dilger"."pg_catalog"."pg_class_relname_nsp_index" (oid 2663) (6/6 pages)
> pg_amcheck: checking btree index "mark.dilger"."pg_catalog"."pg_class_tblspc_relfilenode_index" (oid 3455) (5/5 pages)
> pg_amcheck: checking btree index "mark.dilger"."pg_catalog"."pg_class_oid_index" (oid 2662) (4/4 pages)
> 6/6 relations (100%) 55/55 pages (100%)
>
> The way the (x/y pages) is printed takes into account that the [startblock..endblock] range may reduce the number of pages to check (x) to something less than the number of pages in the relation (y), but the reporting is a bit of a lie when the startblock is beyond the end of the table, as it doesn't get passed to verify_heapam and so the number of blocks checked may be more than the zero blocks reported. I think I might need to fix this up tomorrow, but I want to get what I have in this patch set posted tonight, so it's not fixed here.  Also, there are multiple ways of addressing this, and I'm having trouble deciding which way is best.  I can exclude the relation from being checked at all, or realize earlier that I'm not going to honor the startblock argument and compute the blocks to check correctly.  Thoughts?

I think this whole approach is pretty suspect because the number of
blocks in the relation can increase (by relation extension) or
decrease (by VACUUM or TRUNCATE) between the time when we query for
the list of target relations and the time we get around to executing
any queries against them. I think it's OK to use the number of
relation pages for progress reporting because progress reporting is
only approximate anyway, but I wouldn't print them out in the progress
messages, and I wouldn't try to fix up the startblock and endblock
arguments on the basis of how long you think that relation is going to
be. You seem to view the fact that the server reported the error as
the reason for the problem, but I don't agree. I think having the
server report the error here is right, and the problem is that the
error reporting sucked because it was long-winded and didn't
necessarily tell you which table had the problem.

There are a LOT of things that can go wrong when we go try to run
verify_heapam on a table. The table might have been dropped; in fact,
on a busy production system, such cases are likely to occur routinely
if DDL is common, which for many users it is. The system catalog
entries might be screwed up, so that the relation can't be opened.
There might be an unreadable page in the relation, either because the
OS reports an I/O error or something like that, or because checksum
verification fails. There are various other possibilities. We
shouldn't view such errors as low-level things that occur only in
fringe cases; this is a corruption-checking tool, and we should expect
that running it against messed-up databases will be common. We
shouldn't try to interpret the errors we get or make any big decisions
about them, but we should have a clear way of reporting them so that
the user can decide what to do.

Just as an experiment, I suggest creating a database with 100 tables
in it, each with 1 index, and then deleting a single pg_attribute
entry for 10 of the tables, and then running pg_amcheck. I think you
will get 20 errors -  one for each messed-up table and one for the
corresponding index. Maybe you'll get errors for the TOAST tables
checks too, if the tables have TOAST tables, although that seems like
it should be avoidable. Now, now matter what you do, the tool is going
to produce a lot of output here, because you have a lot of problems,
and that's OK. But how understandable is that output, and how concise
is it? If it says something like:

pg_amcheck: could not check "SCHEMA_NAME"."TABLE_NAME": ERROR: some
attributes are missing or something

...and that line is repeated 20 times, maybe with a context or detail
line for each one or something like that, then you have got a good UI.
If it's not clear which tables have the problem, you have got a bad
UI. If it dumps out 300 lines of output instead of 20 or 40, you have
a UI that is so verbose that usability is going to be somewhat
impaired, which is why I suggested only showing the query in verbose
mode.

BTW, another thing that might be interesting is to call
PQsetErrorVerbosity(conn, PQERRORS_VERBOSE) in verbose mode. It's
probably possible to contrive a case where the server error message is
something generic like "cache lookup failed for relation %u" which
occurs in a whole bunch of places in the source code, and being able
get the file and line number information can be really useful when
trying to track such things down.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  5. Re: pg_amcheck contrib application

    Robert Haas <robertmhaas@gmail.com> — 2021-03-04T17:27:54Z

    On Thu, Mar 4, 2021 at 10:29 AM Robert Haas <robertmhaas@gmail.com> wrote:
> Most of these changes sound good. I'll go through the whole patch
> again today, or as much of it as I can. But before I do that, I want
> to comment on this point specifically.

Just a thought - I don't feel strongly about this - but you want want
to consider storing your list of patterns in an array that gets
resized as necessary rather than a list. Then the pattern ID would
just be pattern_ptr - pattern_array, and finding the pattern by ID
would just be pattern_ptr = &pattern_array[pattern_id]. I don't think
there's a real efficiency issue here because the list of patterns is
almost always going to be short, and even if somebody decides to
provide a very long list of patterns (e.g. by using xargs) it's
probably still not that big a deal. A sufficiently obstinate user
running an operating system where argument lists can be extremely long
could probably make this the dominant cost by providing a gigantic
number of patterns that don't match anything, but such a person is
trying to prove a point, rather than accomplish anything useful, so I
don't care. But, the code might be more elegant the other way.

This patch increases the number of cases where we use ^ to assert that
exactly one of two things is true from 4 to 5. I think it might be
better to just write out (a && !b) || (b && !a), but there is some
precedent for the way you did it so perhaps it's fine.

The name prepare_table_commmand() is oddly non-parallel with
verify_heapam_slot_handler(). Seems better to call it either a table
throughout, or a heapam throughout. Actually I think I would prefer
"heap" to either of those, but I definitely think we shouldn't switch
terminology. Note that prepare_btree_command() doesn't have this
issue, since it matches verify_btree_slot_handler(). On a related
note, "c.relam = 2" is really a test for is_heap, not is_table. We
might have other table AMs in the future, but only one of those AMs
will be called heap, and only one will have OID 2.

You've got some weird round-tripping stuff where you sent literal
values to the server so that you can turn around and get them back
from the server. For example, you've got prepare_table_command()
select rel->nspname and rel->relname back from the server as literals,
which seems silly because we have to already have that information or
we couldn't ask the server to give it to us ... and if we already have
it, then why do we need to get it again? The reason it's like this
seems to be that after calling prepare_table_command(), we use
ParallelSlotSetHandler() to set verify_heapam_slot_handler() as the
callback, and we set sql.data as the callback, so we don't have access
to the RelationInfo object when we're handling the slot result. But
that's easy to fix: just store the sql as a field inside the
RelationInfo, and then pass a pointer to the whole RelationInfo to the
slot handler. Then you don't need to round-trip the table and schema
names; and you have the values available even if an error happens.

On a somewhat related note, I think it might make sense to have the
slot handlers try to free memory. It seems hard to make pg_amcheck
leak enough memory to matter, but I guess it's not entirely
implausible that someone could be checking let's say 10 million
relations. Freeing the query strings could probably prevent a half a
GB or so of accumulated memory usage under those circumstances. I
suppose freeing nspname and relname would save a bit more, but it's
hardly worth doing since they are a lot shorter and you've got to have
all that information in memory at once at some point anyway; similarly
with the RelationInfo structures, which have the further complexity of
being part of a linked list you might not want to corrupt. But you
don't need to have every query string in memory at the same time, just
as many as are running at one in time.

Also, maybe compile_relation_list_one_db() should keep the result set
around so that you don't need to pstrdup() the nspname and relname in
the first place. Right now, just before compile_relation_list_one_db()
calls PQclear() you have two copies of every nspname and relname
allocated. If you just kept the result sets around forever, the peak
memory usage would be lower than it is currently. If you really wanted
to get fancy you could arrange to free each result set when you've
finished that database, but that seems annoying to code and I'm pretty
sure it doesn't matter.

The CTEs called "include_raw" and "exclude_raw" which are used as part
of the query to construct a list of tables. The regexes are fished
through there, and the pattern IDs, which makes sense, but the raw
patterns are also fished through, and I don't see a reason for that.
We don't seem to need that for anything. The same seems to apply to
the query used to resolve database patterns.

I see that most of the queries have now been adjusted to be spread
across fewer lines, which is good, but please make sure to do that
everywhere. In particular, I notice that the bt_index_check calls are
still too spread out.

More in a bit, need to grab some lunch.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  6. Re: pg_amcheck contrib application

    Robert Haas <robertmhaas@gmail.com> — 2021-03-04T21:23:42Z

    On Thu, Mar 4, 2021 at 12:27 PM Robert Haas <robertmhaas@gmail.com> wrote:
> More in a bit, need to grab some lunch.

Moving on to the tests, in 003_check.pl, I think it would be slightly
better if relation_toast were to select ct.oid::regclass and then just
have the caller use that value directly. We'd certainly want to do
that if the name could contain any characters that might require
quoting. Here that's not possible, but I think we might as well use
the same technique anyway.

I'm not sure how far to go with it, but I think that you might want to
try to enhance the logging in some of the cases where the TAP tests
might fail. In particular, if either of these trip in the buildfarm,
it doesn't seem like it will be too easy to figure out why they
failed:

+    fail('Xid thresholds not as expected');
+        fail('Page layout differs from our expectations');

You might want to rephrase the message to incorporate the values that
triggered the failure, e.g. "datfrozenxid $datfrozenxid is not between
3 and $relfrozenxid", "expected (a,b) = (12345678,abcdefg) but got
($x,$y)", so that if the buildfarm happens to fail there's a shred of
hope that we might be able to guess the reason from the message. You
could also give some thought to whether there are any tests that can
be improved in similar ways. Test::More is nice in that when you run a
test with eq() or like() and it fails it will tell you about the input
values in the diagnostic, but if you do something like is($x < 4, ...)
instead of cmp_ok($x, '<', 4, ...) then you lose that. I'm not saying
you're doing that exact thing, just saying that looking through the
test code with an eye to finding things where you could output a
little more info about a potential failure might be a worthwhile
activity.

If it were me, I would get rid of ROWCOUNT and have a list of
closures, and then loop over the list and call each one e.g. my
@corruption = ( sub { ... }, sub { ... }, sub { ... }) or maybe
something like what I did with @scenario in
src/bin/pg_verifybackup/t/003_corruption.pl, but this is ultimately a
style preference and I think the way you actually did it is also
reasonable, and some people might find it more readable than the other
way.

The name int4_fickle_ops is positively delightful and I love having a
test case like this.

On the whole, I think these tests look quite solid. I am a little
concerned, as you may gather from the comment above, that they will
not survive contact with the buildfarm, because they will turn out to
be platform or OS-dependent in some way. However, I can see that
you've taken steps to avoid such dependencies, and maybe we'll be
lucky and those will work. Also, while I am suspicious something's
going to break, I don't know what it's going to be, so I can't suggest
any method to avoid it. I think we'll just have to keep an eye on the
buildfarm post-commit and see what crops up.

Turning to the documentation, I see that it is documented that a bare
command-line argument can be a connection string rather than a
database name. That sounds like a good plan, but when I try
'pg_amcheck sslmode=require' it does not work: FATAL:  database
"sslmode=require" does not exist. The argument to -e is also
documented to be a connection string, but that also seems not to work.
Some thought might need to be given to what exactly these connection
opens are supposed to mean. Like, do the connection options I set via
-e apply to all the connections I make, or just the one to the
maintenance database? How do I set connection options for connections
to databases whose names aren't specified explicitly but are
discovered by querying pg_database? Maybe instead of allowing these to
be a connection string, we should have a separate option that can be
used just for the purpose of setting connection options that then
apply to all connections. That seems a little bit oddly unlike other
tools, but if I want sslmode=verify-ca or something on all my
connections, there should be an easy way to get it.

The documentation makes many references to patterns, but does not
explain what a pattern is. I see that psql's documentation contains an
explanation, and pg_dump's documentation links to psql's
documentation. pg_amcheck should probably link to psql's
documentation, too.

In the documentation for -d, you say that "If -a --all is also
specified, -d --database does not additionally affect which databases
are checked." I suggest replacing "does not additionally affect which
databases are checked" with "has no effect."

In two places you say "without regard for" but I think it should be
"without regard to".

In the documentation for --no-strict-names you use "nor" where I think
it should say "or".

I kind of wonder whether we need --quiet. It seems like right now it
only does two things. One is to control complaints about ignoring the
startblock and endblock options, but I don't agree with that behavior
anyway. The other is control whether we complain about unmatched
patterns, but I think that could just be controlled --no-strict-names
i.e. normally an unmatched pattern results in a complaint and a
failure, but with --no-strict-names there is neither a complaint nor a
failure. Having a flag to control whether we get the message
separately from whether we get the failure doesn't seem helpful.

I don't think it's good to say "This is an alias for" in the
documentation of -i -I -t -T. I suggest instead saying "This is
similar to".

Instead of "Option BLAH takes precedence over..." I suggest "The BLAH
option takes precedence over..."

OK, that's it from me for this review pass.

Thanks,

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  7. Re: pg_amcheck contrib application

    Peter Geoghegan <pg@bowt.ie> — 2021-03-04T22:04:37Z

    On Thu, Mar 4, 2021 at 7:29 AM Robert Haas <robertmhaas@gmail.com> wrote:
> I think this whole approach is pretty suspect because the number of
> blocks in the relation can increase (by relation extension) or
> decrease (by VACUUM or TRUNCATE) between the time when we query for
> the list of target relations and the time we get around to executing
> any queries against them. I think it's OK to use the number of
> relation pages for progress reporting because progress reporting is
> only approximate anyway, but I wouldn't print them out in the progress
> messages, and I wouldn't try to fix up the startblock and endblock
> arguments on the basis of how long you think that relation is going to
> be.

I don't think that the struct AmcheckOptions block fields (e.g.,
startblock) should be of type 'long' -- that doesn't work well on
Windows, where 'long' is only 32-bit. To be fair we already do the
same thing elsewhere, but there is no reason to repeat those mistakes.
(I'm rather suspicious of 'long' in general.)

I think that you could use BlockNumber + strtoul() without breaking Windows.

> There are a LOT of things that can go wrong when we go try to run
> verify_heapam on a table. The table might have been dropped; in fact,
> on a busy production system, such cases are likely to occur routinely
> if DDL is common, which for many users it is. The system catalog
> entries might be screwed up, so that the relation can't be opened.
> There might be an unreadable page in the relation, either because the
> OS reports an I/O error or something like that, or because checksum
> verification fails. There are various other possibilities. We
> shouldn't view such errors as low-level things that occur only in
> fringe cases; this is a corruption-checking tool, and we should expect
> that running it against messed-up databases will be common. We
> shouldn't try to interpret the errors we get or make any big decisions
> about them, but we should have a clear way of reporting them so that
> the user can decide what to do.

I agree.

Your database is not supposed to be corrupt. Once your database has
become corrupt, all bets are off -- something happened that was
supposed to be impossible -- which seems like a good reason to be
modest about what we think we know.

The user should always see the unvarnished truth. pg_amcheck should
not presume to suppress errors from lower level code, except perhaps
in well-scoped special cases.

-- 
Peter Geoghegan

  8. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-03-04T22:39:04Z

    
> On Mar 4, 2021, at 2:04 PM, Peter Geoghegan <pg@bowt.ie> wrote:
> 
> On Thu, Mar 4, 2021 at 7:29 AM Robert Haas <robertmhaas@gmail.com> wrote:
>> I think this whole approach is pretty suspect because the number of
>> blocks in the relation can increase (by relation extension) or
>> decrease (by VACUUM or TRUNCATE) between the time when we query for
>> the list of target relations and the time we get around to executing
>> any queries against them. I think it's OK to use the number of
>> relation pages for progress reporting because progress reporting is
>> only approximate anyway, but I wouldn't print them out in the progress
>> messages, and I wouldn't try to fix up the startblock and endblock
>> arguments on the basis of how long you think that relation is going to
>> be.
> 
> I don't think that the struct AmcheckOptions block fields (e.g.,
> startblock) should be of type 'long' -- that doesn't work well on
> Windows, where 'long' is only 32-bit. To be fair we already do the
> same thing elsewhere, but there is no reason to repeat those mistakes.
> (I'm rather suspicious of 'long' in general.)
> 
> I think that you could use BlockNumber + strtoul() without breaking Windows.

Fair enough.

>> There are a LOT of things that can go wrong when we go try to run
>> verify_heapam on a table. The table might have been dropped; in fact,
>> on a busy production system, such cases are likely to occur routinely
>> if DDL is common, which for many users it is. The system catalog
>> entries might be screwed up, so that the relation can't be opened.
>> There might be an unreadable page in the relation, either because the
>> OS reports an I/O error or something like that, or because checksum
>> verification fails. There are various other possibilities. We
>> shouldn't view such errors as low-level things that occur only in
>> fringe cases; this is a corruption-checking tool, and we should expect
>> that running it against messed-up databases will be common. We
>> shouldn't try to interpret the errors we get or make any big decisions
>> about them, but we should have a clear way of reporting them so that
>> the user can decide what to do.
> 
> I agree.
> 
> Your database is not supposed to be corrupt. Once your database has
> become corrupt, all bets are off -- something happened that was
> supposed to be impossible -- which seems like a good reason to be
> modest about what we think we know.
> 
> The user should always see the unvarnished truth. pg_amcheck should
> not presume to suppress errors from lower level code, except perhaps
> in well-scoped special cases.

I think Robert mistook why I was doing that.  I was thinking about a different usage pattern.  If somebody thinks a subset of relations have been badly corrupted, but doesn't know which relations those might be, they might try to find them with pg_amcheck, but wanting to just check the first few blocks per relation in order to sample the relations.  So,

  pg_amcheck --startblock=0 --endblock=9 --no-dependent-indexes

or something like that.  I don't think it's very fun to have it error out for each relation that doesn't have at least ten blocks, nor is it fun to have those relations skipped by error'ing out before checking any blocks, as they might be the corrupt relations you are looking for.  But using --startblock and --endblock for this is not a natural fit, as evidenced by how I was trying to "fix things up" for the user, so I'll punt on this usage until some future version, when I might add a sampling option.

—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  9. Re: pg_amcheck contrib application

    Robert Haas <robertmhaas@gmail.com> — 2021-03-08T16:26:08Z

    On Thu, Mar 4, 2021 at 5:39 PM Mark Dilger <mark.dilger@enterprisedb.com> wrote:
> I think Robert mistook why I was doing that.  I was thinking about a different usage pattern.  If somebody thinks a subset of relations have been badly corrupted, but doesn't know which relations those might be, they might try to find them with pg_amcheck, but wanting to just check the first few blocks per relation in order to sample the relations.  So,
>
>   pg_amcheck --startblock=0 --endblock=9 --no-dependent-indexes
>
> or something like that.  I don't think it's very fun to have it error out for each relation that doesn't have at least ten blocks, nor is it fun to have those relations skipped by error'ing out before checking any blocks, as they might be the corrupt relations you are looking for.  But using --startblock and --endblock for this is not a natural fit, as evidenced by how I was trying to "fix things up" for the user, so I'll punt on this usage until some future version, when I might add a sampling option.

I admit I hadn't thought of that use case. I guess somebody could want
to do that, but it doesn't seem all that useful. Checking the first
up-to-ten blocks of every relation is not a very representative
sample, and it's not clear to me that sampling is a good idea even if
it were representative. What good is it to know that 10% of my
database is probably not corrupted?

On the other hand, people want to do all kinds of things that seem
strange to me, and this might be another one. But, if that's so, then
I think the right place to implement it is in amcheck itself, not
pg_amcheck. I think pg_amcheck should be, now and in the future, a
thin wrapper around the functionality provided by amcheck, just
providing target selection and parallel execution. If you put
something into pg_amcheck that figures out how long the relation is
and runs it on some of the blocks, that functionality is only
accessible to people who are accessing amcheck via pg_amcheck. If you
put it in amcheck itself and just expose it through pg_amcheck, then
it's accessible either way. It's probably cleaner and more performant
to do it that way, too.

So if you did add a sampling option in the future, that's the way I
would recommend doing it, but I think it is probably best not to go
there right now.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  10. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-03-08T20:30:16Z

    
> On Mar 8, 2021, at 8:26 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> 
> On Thu, Mar 4, 2021 at 5:39 PM Mark Dilger <mark.dilger@enterprisedb.com> wrote:
>> I think Robert mistook why I was doing that.  I was thinking about a different usage pattern.  If somebody thinks a subset of relations have been badly corrupted, but doesn't know which relations those might be, they might try to find them with pg_amcheck, but wanting to just check the first few blocks per relation in order to sample the relations.  So,
>> 
>>  pg_amcheck --startblock=0 --endblock=9 --no-dependent-indexes
>> 
>> or something like that.  I don't think it's very fun to have it error out for each relation that doesn't have at least ten blocks, nor is it fun to have those relations skipped by error'ing out before checking any blocks, as they might be the corrupt relations you are looking for.  But using --startblock and --endblock for this is not a natural fit, as evidenced by how I was trying to "fix things up" for the user, so I'll punt on this usage until some future version, when I might add a sampling option.
> 
> I admit I hadn't thought of that use case. I guess somebody could want
> to do that, but it doesn't seem all that useful. Checking the first
> up-to-ten blocks of every relation is not a very representative
> sample, and it's not clear to me that sampling is a good idea even if
> it were representative. What good is it to know that 10% of my
> database is probably not corrupted?


`cd $PGDATA; tar xfz my_csv_data.tgz` ctrl-C ctrl-C ctrl-C
`rm -rf $PGDATA` ctrl-C ctrl-C ctrl-C
`/my/stupid/backup/and/restore/script.sh` ctrl-C ctrl-C ctrl-C

# oh wow, i wonder if any relations got overwritten with csv file data, or had their relation files unlinked, or ...?

`pg_amcheck --jobs=8 --startblock=0 --endblock=10`

# ah, darn, it's spewing lots of irrelevant errors because some relations are too short

`pg_amcheck --jobs=8 --startblock=0 --endblock=0`

# ah, darn, it's still spewing lots of irrelevant errors because I have lots of indexes with zero blocks of data

`pg_amcheck --jobs=8`

# ah, darn, it's taking forever, because it's processing huge tables in their entirety

I agree this can be left to later, and the --startblock and --endblock options are the wrong way to do it.

—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  11. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-03-10T16:10:02Z

    Robert, Peter, in response to your review comments spanning multiple emails:

> On Mar 4, 2021, at 7:29 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> 
> Most of these changes sound good. I'll go through the whole patch
> again today, or as much of it as I can. But before I do that, I want
> to comment on this point specifically.
> 
> On Thu, Mar 4, 2021 at 1:25 AM Mark Dilger <mark.dilger@enterprisedb.com> wrote:
>> I think this is fixed up now.  There is an interaction with amcheck's verify_heapam(), where that function raises an error if the startblock or endblock arguments are out of bounds for the relation in question.  Rather than aborting the entire pg_amcheck run, it avoids passing inappropriate block ranges to verify_heapam() and outputs a warning, so:
>> 
>> % pg_amcheck mark.dilger -t foo -t pg_class --progress -v --startblock=35 --endblock=77
>> pg_amcheck: in database "mark.dilger": using amcheck version "1.3" in schema "public"
>> 0/6 relations (0%)  0/55 pages (0%)
>> pg_amcheck: checking table "mark.dilger"."public"."foo" (oid 16385) (10/45 pages)
>> pg_amcheck: warning: ignoring endblock option 77 beyond end of table "mark.dilger"."public"."foo"
>> pg_amcheck: checking btree index "mark.dilger"."public"."foo_idx" (oid 16388) (30/30 pages)
>> pg_amcheck: checking table "mark.dilger"."pg_catalog"."pg_class" (oid 1259) (0/13 pages)
>> pg_amcheck: warning: ignoring startblock option 35 beyond end of table "mark.dilger"."pg_catalog"."pg_class"
>> pg_amcheck: warning: ignoring endblock option 77 beyond end of table "mark.dilger"."pg_catalog"."pg_class"
>> pg_amcheck: checking btree index "mark.dilger"."pg_catalog"."pg_class_relname_nsp_index" (oid 2663) (6/6 pages)
>> pg_amcheck: checking btree index "mark.dilger"."pg_catalog"."pg_class_tblspc_relfilenode_index" (oid 3455) (5/5 pages)
>> pg_amcheck: checking btree index "mark.dilger"."pg_catalog"."pg_class_oid_index" (oid 2662) (4/4 pages)
>> 6/6 relations (100%) 55/55 pages (100%)
>> 
>> The way the (x/y pages) is printed takes into account that the [startblock..endblock] range may reduce the number of pages to check (x) to something less than the number of pages in the relation (y), but the reporting is a bit of a lie when the startblock is beyond the end of the table, as it doesn't get passed to verify_heapam and so the number of blocks checked may be more than the zero blocks reported. I think I might need to fix this up tomorrow, but I want to get what I have in this patch set posted tonight, so it's not fixed here.  Also, there are multiple ways of addressing this, and I'm having trouble deciding which way is best.  I can exclude the relation from being checked at all, or realize earlier that I'm not going to honor the startblock argument and compute the blocks to check correctly.  Thoughts?
> 
> I think this whole approach is pretty suspect because the number of
> blocks in the relation can increase (by relation extension) or
> decrease (by VACUUM or TRUNCATE) between the time when we query for
> the list of target relations and the time we get around to executing
> any queries against them. I think it's OK to use the number of
> relation pages for progress reporting because progress reporting is
> only approximate anyway,

Fair point.

> but I wouldn't print them out in the progress
> messages,

Removed. 

> and I wouldn't try to fix up the startblock and endblock
> arguments on the basis of how long you think that relation is going to
> be.

Yeah, in light of a new day, that seems like a bad idea to me, too.  Removed.

> You seem to view the fact that the server reported the error as
> the reason for the problem, but I don't agree. I think having the
> server report the error here is right, and the problem is that the
> error reporting sucked because it was long-winded and didn't
> necessarily tell you which table had the problem.

No, I was thinking about a different usage pattern, but I've answer that already elsewhere on this thread.

> There are a LOT of things that can go wrong when we go try to run
> verify_heapam on a table. The table might have been dropped; in fact,
> on a busy production system, such cases are likely to occur routinely
> if DDL is common, which for many users it is. The system catalog
> entries might be screwed up, so that the relation can't be opened.
> There might be an unreadable page in the relation, either because the
> OS reports an I/O error or something like that, or because checksum
> verification fails. There are various other possibilities. We
> shouldn't view such errors as low-level things that occur only in
> fringe cases; this is a corruption-checking tool, and we should expect
> that running it against messed-up databases will be common. We
> shouldn't try to interpret the errors we get or make any big decisions
> about them, but we should have a clear way of reporting them so that
> the user can decide what to do.

Once again, I think you are right and have removed the objectionable behavior, but....

The --startblock and --endblock options make the most sense when the user is only checking one table, like

  pg_amcheck --startblock=17 --endblock=19 --table=my_schema.my_corrupt_table

because the user likely has some knowledge about that table, perhaps from a prior run of pg_amcheck.  The --startblock and --endblock arguments are a bit strange when used globally, as relations don't all have the same number of blocks, so

  pg_amcheck --startblock=17 --endblock=19 mydb

will very likely emit lots of error messages for tables which don't have blocks in that range.  That's not entirely pg_amcheck's fault, as it just did what the user asked, but it also doesn't seem super helpful.  I'm not going to do anything about it in this release.

> Just as an experiment, I suggest creating a database with 100 tables
> in it, each with 1 index, and then deleting a single pg_attribute
> entry for 10 of the tables, and then running pg_amcheck. I think you
> will get 20 errors -  one for each messed-up table and one for the
> corresponding index. Maybe you'll get errors for the TOAST tables
> checks too, if the tables have TOAST tables, although that seems like
> it should be avoidable. Now, now matter what you do, the tool is going
> to produce a lot of output here, because you have a lot of problems,
> and that's OK. But how understandable is that output, and how concise
> is it? If it says something like:
> 
> pg_amcheck: could not check "SCHEMA_NAME"."TABLE_NAME": ERROR: some
> attributes are missing or something
> 
> ...and that line is repeated 20 times, maybe with a context or detail
> line for each one or something like that, then you have got a good UI.
> If it's not clear which tables have the problem, you have got a bad
> UI. If it dumps out 300 lines of output instead of 20 or 40, you have
> a UI that is so verbose that usability is going to be somewhat
> impaired, which is why I suggested only showing the query in verbose
> mode.

After running 'make installcheck', if I delete all entries from pg_class where relnamespace = 'pg_toast'::regclass, by running 'pg_amcheck regression', I get lines that look like this:

heap relation "regression"."public"."quad_poly_tbl":
    ERROR:  could not open relation with OID 17177
heap relation "regression"."public"."gin_test_tbl":
    ERROR:  could not open relation with OID 24793
heap relation "regression"."pg_catalog"."pg_depend":
    ERROR:  could not open relation with OID 8888
heap relation "regression"."public"."spgist_text_tbl":
    ERROR:  could not open relation with OID 25624

which seems ok.


If instead I delete pg_attribute entries, as you suggest above, I get rows like this:

heap relation "regression"."regress_rls_schema"."rls_tbl":
    ERROR:  catalog is missing 1 attribute(s) for relid 26467
heap relation "regression"."regress_rls_schema"."rls_tbl_force":
    ERROR:  catalog is missing 1 attribute(s) for relid 26474

which also seems ok.


If instead, I manually corrupt relation files belonging to the regression database, I get lines that look like this for corrupt heap relations:

relation "regression"."public"."functional_dependencies", block 28, offset 54, attribute 0
    attribute 0 with length 4294967295 ends at offset 50 beyond total tuple length 43
relation "regression"."public"."functional_dependencies", block 28, offset 55
    multitransaction ID is invalid
relation "regression"."public"."functional_dependencies", block 28, offset 57
    multitransaction ID is invalid

and for corrupt btree relations:

btree relation "regression"."public"."tenk1_unique1":
    ERROR:  high key invariant violated for index "tenk1_unique1"
    DETAIL:  Index tid=(1,38) points to heap tid=(70,26) page lsn=0/33A96D0.
btree relation "regression"."public"."tenk1_unique2":
    ERROR:  index tuple size does not equal lp_len in index "tenk1_unique2"
    DETAIL:  Index tid=(1,35) tuple size=4913 lp_len=16 page lsn=0/33DFD98.
    HINT:  This could be a torn page problem.
btree relation "regression"."public"."tenk1_thous_tenthous":
    ERROR:  index tuple size does not equal lp_len in index "tenk1_thous_tenthous"
    DETAIL:  Index tid=(1,36) tuple size=4402 lp_len=16 page lsn=0/34C0770.
    HINT:  This could be a torn page problem.

which likewise seems ok.


> BTW, another thing that might be interesting is to call
> PQsetErrorVerbosity(conn, PQERRORS_VERBOSE) in verbose mode. It's
> probably possible to contrive a case where the server error message is
> something generic like "cache lookup failed for relation %u" which
> occurs in a whole bunch of places in the source code, and being able
> get the file and line number information can be really useful when
> trying to track such things down.

Good idea.  I decided to also honor the --quiet flag

        if (opts.verbose)
            PQsetErrorVerbosity(free_slot->connection, PQERRORS_VERBOSE);
        else if (opts.quiet)
            PQsetErrorVerbosity(free_slot->connection, PQERRORS_TERSE);

> On Mar 4, 2021, at 2:04 PM, Peter Geoghegan <pg@bowt.ie> wrote:
> 
> I don't think that the struct AmcheckOptions block fields (e.g.,
> startblock) should be of type 'long' -- that doesn't work well on
> Windows, where 'long' is only 32-bit. To be fair we already do the
> same thing elsewhere, but there is no reason to repeat those mistakes.
> (I'm rather suspicious of 'long' in general.)
> 
> I think that you could use BlockNumber + strtoul() without breaking Windows.


Thanks for reviewing!

Good points.  I decided to use int64 instead of BlockNumber.  The option processing needs to give a sensible error message if the user gives a negative number for the argument, so unsigned types are a bad fit.

> On Thu, Mar 4, 2021 at 10:29 AM Robert Haas <robertmhaas@gmail.com> wrote:
>> Most of these changes sound good. I'll go through the whole patch
>> again today, or as much of it as I can. But before I do that, I want
>> to comment on this point specifically.
> 
> Just a thought - I don't feel strongly about this - but you want want
> to consider storing your list of patterns in an array that gets
> resized as necessary rather than a list. Then the pattern ID would
> just be pattern_ptr - pattern_array, and finding the pattern by ID
> would just be pattern_ptr = &pattern_array[pattern_id]. I don't think
> there's a real efficiency issue here because the list of patterns is
> almost always going to be short, and even if somebody decides to
> provide a very long list of patterns (e.g. by using xargs) it's
> probably still not that big a deal. A sufficiently obstinate user
> running an operating system where argument lists can be extremely long
> could probably make this the dominant cost by providing a gigantic
> number of patterns that don't match anything, but such a person is
> trying to prove a point, rather than accomplish anything useful, so I
> don't care. But, the code might be more elegant the other way.

Done.  I was not too motivated by the efficiency argument, but the code to look up patterns is cleaner when the pattern_id is an index into an array than when it is a field in a struct that has to be searched in a list.

> This patch increases the number of cases where we use ^ to assert that
> exactly one of two things is true from 4 to 5. I think it might be
> better to just write out (a && !b) || (b && !a), but there is some
> precedent for the way you did it so perhaps it's fine.

Your formulation takes longer for me to read and understand (by, perhaps, some milliseconds), but checking what C compilers guarantee to store in

  bool a = (i == j);
  bool b = (k == l);

I found it hard to be sure that some compiler wouldn't do weird things with that.  Two "true" values a and b could pass the (a ^ b) test if they represent "true" in two different bit patterns.  I don't really think there is a risk here in practice, but looking up the relevant C standards isn't quick for future readers of this code, so I went with your formulation.

> The name prepare_table_commmand() is oddly non-parallel with
> verify_heapam_slot_handler(). Seems better to call it either a table
> throughout, or a heapam throughout. Actually I think I would prefer
> "heap" to either of those, but I definitely think we shouldn't switch
> terminology. Note that prepare_btree_command() doesn't have this
> issue, since it matches verify_btree_slot_handler(). On a related
> note, "c.relam = 2" is really a test for is_heap, not is_table. We
> might have other table AMs in the future, but only one of those AMs
> will be called heap, and only one will have OID 2.

Changed to use "heap" in many places where "table" was used previously, and to use "btree" in many places where "index" was used previously.  The term "heapam" now only occurs as part of "verify_heapam", a function defined in contrib/amcheck and not changed here.

> You've got some weird round-tripping stuff where you sent literal
> values to the server so that you can turn around and get them back
> from the server. For example, you've got prepare_table_command()
> select rel->nspname and rel->relname back from the server as literals,
> which seems silly because we have to already have that information or
> we couldn't ask the server to give it to us ... and if we already have
> it, then why do we need to get it again? The reason it's like this
> seems to be that after calling prepare_table_command(), we use
> ParallelSlotSetHandler() to set verify_heapam_slot_handler() as the
> callback, and we set sql.data as the callback, so we don't have access
> to the RelationInfo object when we're handling the slot result. But
> that's easy to fix: just store the sql as a field inside the
> RelationInfo, and then pass a pointer to the whole RelationInfo to the
> slot handler. Then you don't need to round-trip the table and schema
> names; and you have the values available even if an error happens.

Changed.  I was doing that mostly so that people examining the server logs would have something more than the oid in the sql to suggest which table or index is being checked.

> On a somewhat related note, I think it might make sense to have the
> slot handlers try to free memory. It seems hard to make pg_amcheck
> leak enough memory to matter, but I guess it's not entirely
> implausible that someone could be checking let's say 10 million
> relations. Freeing the query strings could probably prevent a half a
> GB or so of accumulated memory usage under those circumstances. I
> suppose freeing nspname and relname would save a bit more, but it's
> hardly worth doing since they are a lot shorter and you've got to have
> all that information in memory at once at some point anyway; similarly
> with the RelationInfo structures, which have the further complexity of
> being part of a linked list you might not want to corrupt. But you
> don't need to have every query string in memory at the same time, just
> as many as are running at one in time.

Changed.

> Also, maybe compile_relation_list_one_db() should keep the result set
> around so that you don't need to pstrdup() the nspname and relname in
> the first place. Right now, just before compile_relation_list_one_db()
> calls PQclear() you have two copies of every nspname and relname
> allocated. If you just kept the result sets around forever, the peak
> memory usage would be lower than it is currently. If you really wanted
> to get fancy you could arrange to free each result set when you've
> finished that database, but that seems annoying to code and I'm pretty
> sure it doesn't matter.


Hmm.  When compile_relation_list_one_db() is processing the ith database out of N databases, all (nspname,relname) pairs are allocated for databases in [0..i], and additionally the result set for database i is in memory.  The result sets for [0..i-1] have already been freed.  Keeping around the result sets for all N databases seems more expensive, considering how much stuff is in struct pg_result, if N is large and the relations are spread across the databases rather than clumped together in the last one.

I think your proposal might be a win for some users and a loss for others.  Given that it is not a clear win, I don't care to implement it that way, as it takes more effort to remember which object owns which bit of memory.

I have added pfree()s to the handlers to free the nspname and relname when finished.  This does little to reduce the peak memory usage, though.

> The CTEs called "include_raw" and "exclude_raw" which are used as part
> of the query to construct a list of tables. The regexes are fished
> through there, and the pattern IDs, which makes sense, but the raw
> patterns are also fished through, and I don't see a reason for that.
> We don't seem to need that for anything. The same seems to apply to
> the query used to resolve database patterns.

Changed.

Both queries are changed to no longer have a "pat" column, and the "id" field (renamed as "pattern_id" for clarity) is used instead.

> I see that most of the queries have now been adjusted to be spread
> across fewer lines, which is good, but please make sure to do that
> everywhere. In particular, I notice that the bt_index_check calls are
> still too spread out.

When running` pg_amcheck --echo`, the queries for a table and index now print as:

SELECT blkno, offnum, attnum, msg FROM "public".verify_heapam(
relation := 33024, on_error_stop := false, check_toast := true, skip := 'none')
SELECT * FROM "public".bt_index_check(index := '33029'::regclass, heapallindexed := false)

Which is two lines per heap table, and just one line per btree index.

> On Thu, Mar 4, 2021 at 12:27 PM Robert Haas <robertmhaas@gmail.com> wrote:
>> More in a bit, need to grab some lunch.
> 
> Moving on to the tests, in 003_check.pl, I think it would be slightly
> better if relation_toast were to select ct.oid::regclass and then just
> have the caller use that value directly. We'd certainly want to do
> that if the name could contain any characters that might require
> quoting. Here that's not possible, but I think we might as well use
> the same technique anyway.

Using c.reltoastrelid::regclass, which is basically the same idea.

> I'm not sure how far to go with it, but I think that you might want to
> try to enhance the logging in some of the cases where the TAP tests
> might fail. In particular, if either of these trip in the buildfarm,
> it doesn't seem like it will be too easy to figure out why they
> failed:
> 
> +    fail('Xid thresholds not as expected');
> +        fail('Page layout differs from our expectations');

Ok, I've extended these messages with the extra debugging information.  I have also changed them to use 'plan skip_all', since what we are really talking about here is an inability for the test to properly exercise pg_amcheck, not an actual failure of pg_amcheck to function correctly.  This should save us some grief if the test isn't portable to all platforms in the build farm, though we'll have to check whether the skip messages are happening on any farm animals.

> You might want to rephrase the message to incorporate the values that
> triggered the failure, e.g. "datfrozenxid $datfrozenxid is not between
> 3 and $relfrozenxid", "expected (a,b) = (12345678,abcdefg) but got
> ($x,$y)", so that if the buildfarm happens to fail there's a shred of
> hope that we might be able to guess the reason from the message.

Added to the skip_all message.

> You
> could also give some thought to whether there are any tests that can
> be improved in similar ways. Test::More is nice in that when you run a
> test with eq() or like() and it fails it will tell you about the input
> values in the diagnostic, but if you do something like is($x < 4, ...)
> instead of cmp_ok($x, '<', 4, ...) then you lose that. I'm not saying
> you're doing that exact thing, just saying that looking through the
> test code with an eye to finding things where you could output a
> little more info about a potential failure might be a worthwhile
> activity.

I'm mostly using command_checks_all and command_fails_like.  The main annoyance is that when a pattern fails to match, you get a rather long error message.  I'm not sure that it's lacking information, though.

> If it were me, I would get rid of ROWCOUNT and have a list of
> closures, and then loop over the list and call each one e.g. my
> @corruption = ( sub { ... }, sub { ... }, sub { ... }) or maybe
> something like what I did with @scenario in
> src/bin/pg_verifybackup/t/003_corruption.pl, but this is ultimately a
> style preference and I think the way you actually did it is also
> reasonable, and some people might find it more readable than the other
> way.

Unchanged.  I think the closure idea is ok, but I am using the ROWCOUNT constant elsewhere (specifically, when inserting rows into the table) and using a constant for this helps keep the number of rows of data and the number of corruptions synchronized.

> The name int4_fickle_ops is positively delightful and I love having a
> test case like this.

I know you know this already, but for others reading this thread, the test using int4_fickle_ops is testing the kind of index corruption that might happen if you changed the sort order underlying an index, such as by updating collation definitions.  It was simpler to not muck around with collations in the test itself, but to achieve the sort order breakage this way.

> On the whole, I think these tests look quite solid. I am a little
> concerned, as you may gather from the comment above, that they will
> not survive contact with the buildfarm, because they will turn out to
> be platform or OS-dependent in some way. However, I can see that
> you've taken steps to avoid such dependencies, and maybe we'll be
> lucky and those will work. Also, while I am suspicious something's
> going to break, I don't know what it's going to be, so I can't suggest
> any method to avoid it. I think we'll just have to keep an eye on the
> buildfarm post-commit and see what crops up.

As I mentioned above, I've changed some failures to 'plan skip_all => reason', so that the build farm won't break if the tests aren't portable in ways I'm already thinking about.  We'll just see if it breaks for additional ways that I'm not thinking about.

> Turning to the documentation, I see that it is documented that a bare
> command-line argument can be a connection string rather than a
> database name. That sounds like a good plan, but when I try
> 'pg_amcheck sslmode=require' it does not work: FATAL:  database
> "sslmode=require" does not exist. The argument to -e is also
> documented to be a connection string, but that also seems not to work.
> Some thought might need to be given to what exactly these connection
> opens are supposed to mean. Like, do the connection options I set via
> -e apply to all the connections I make, or just the one to the
> maintenance database? How do I set connection options for connections
> to databases whose names aren't specified explicitly but are
> discovered by querying pg_database? Maybe instead of allowing these to
> be a connection string, we should have a separate option that can be
> used just for the purpose of setting connection options that then
> apply to all connections. That seems a little bit oddly unlike other
> tools, but if I want sslmode=verify-ca or something on all my
> connections, there should be an easy way to get it.

I'm not sure where you are getting the '-e' from.   That is the short form of --echo, and not what you are likely to want.  However, your larger point is valid.

I don't like the idea that pg_amcheck would handle these options in a way that is incompatible with reindexdb or vacuumdb.  I think pg_amcheck can have a superset of those tools' options, but it should not have options that are incompatible with those tools' options.  That way, if the extra options that pg_amcheck offers become popular, we can add support for them in those other tools.  But if the options are incompatible, we'd not be able to do that without breaking backward compatibility of those tools' interfaces, which we wouldn't want to do.

As such, I have solved the problem by reducing the number of dbname arguments you can provide on the command-line to just one.  (This does not limit the number of database *patterns* that you can supply.)  Those tools only allow one dbname on the command line, so this is not a regression of functionality from what those tools offer.  Only the single dbname argument, or single maintenance-db argument, can be a connection string.  The database patterns do not support that, nor would it make sense for them to do so.

All of the following should now work:

  pg_amcheck --all "port=5555 sslmode=require"

  pg_amcheck --maintenance-db="host=myhost port=5555 dbname=mydb sslmode=require" --all

  pg_amcheck -d foo -d bar -d baz mydb

  pg_amcheck -d foo -d bar -d baz "host=myhost dbname=mydb"

Note that using --all with a connection string is a pg_amcheck extension.  It doesn't currently work in reindexdb, which complains.

There is a strange case, `pg_amcheck --maintenance-db="port=5555 dbname=postgres" "port=5432 dbname=regression"`, which doesn't complain, despite there being nothing listening on port 5555.  This is because pg_amcheck completely ignores the maintenance-db argument in this instance, but I have not changed this behavior, because reindexdb does the same thing.

> The documentation makes many references to patterns, but does not
> explain what a pattern is. I see that psql's documentation contains an
> explanation, and pg_dump's documentation links to psql's
> documentation. pg_amcheck should probably link to psql's
> documentation, too.

A prior version of this patch had a reference to that, but no more.  Thanks for noticing.  I've put it back in.  There is some tension here between the desire to keep the docs concise and the desire to explain things better with examples, etc.  I'm not sure I've got that balance right, but I'm too close to the project to be the right person to make that call.  Does it seem ok?

> In the documentation for -d, you say that "If -a --all is also
> specified, -d --database does not additionally affect which databases
> are checked." I suggest replacing "does not additionally affect which
> databases are checked" with "has no effect."

Changed.

> In two places you say "without regard for" but I think it should be
> "without regard to".

Changed.

> In the documentation for --no-strict-names you use "nor" where I think
> it should say "or".

Changed.

> I kind of wonder whether we need --quiet. It seems like right now it
> only does two things. One is to control complaints about ignoring the
> startblock and endblock options, but I don't agree with that behavior
> anyway. The other is control whether we complain about unmatched
> patterns, but I think that could just be controlled --no-strict-names
> i.e. normally an unmatched pattern results in a complaint and a
> failure, but with --no-strict-names there is neither a complaint nor a
> failure. Having a flag to control whether we get the message
> separately from whether we get the failure doesn't seem helpful.

Hmm.  I think that having --quiet plus --no-strict-names suppress the warnings about unmatched patterns has some value.

Also, as discussed above, I also now decrease the PGVerbosity to PQERRORS_TERSE, which has additional value, I think.

But I don't feel strongly about this, and if you'd rather --quiet be removed, that's fine, too.  But I'll wait to hear back about that.

> I don't think it's good to say "This is an alias for" in the
> documentation of -i -I -t -T. I suggest instead saying "This is
> similar to".

Changed.

> Instead of "Option BLAH takes precedence over..." I suggest "The BLAH
> option takes precedence over..."

Changed.

  12. Re: pg_amcheck contrib application

    Robert Haas <robertmhaas@gmail.com> — 2021-03-10T20:28:39Z

    On Wed, Mar 10, 2021 at 11:10 AM Mark Dilger
<mark.dilger@enterprisedb.com> wrote:
> Once again, I think you are right and have removed the objectionable behavior, but....
>
> The --startblock and --endblock options make the most sense when the user is only checking one table, like
>
>   pg_amcheck --startblock=17 --endblock=19 --table=my_schema.my_corrupt_table
>
> because the user likely has some knowledge about that table, perhaps from a prior run of pg_amcheck.  The --startblock and --endblock arguments are a bit strange when used globally, as relations don't all have the same number of blocks, so
>
>   pg_amcheck --startblock=17 --endblock=19 mydb
>
> will very likely emit lots of error messages for tables which don't have blocks in that range.  That's not entirely pg_amcheck's fault, as it just did what the user asked, but it also doesn't seem super helpful.  I'm not going to do anything about it in this release.

+1 to all that. I tend toward the opinion that trying to make
--startblock and --endblock do anything useful in the context of
checking multiple relations is not really possible, and therefore we
just shouldn't put any effort into it. But if user feedback shows
otherwise, we can always do something about it later.

> After running 'make installcheck', if I delete all entries from pg_class where relnamespace = 'pg_toast'::regclass, by running 'pg_amcheck regression', I get lines that look like this:
>
> heap relation "regression"."public"."quad_poly_tbl":
>     ERROR:  could not open relation with OID 17177

In this here example, the first line ends in a colon.

> relation "regression"."public"."functional_dependencies", block 28, offset 54, attribute 0
>     attribute 0 with length 4294967295 ends at offset 50 beyond total tuple length 43

But this here one does not. Seems like it should be consistent.

The QUALIFIED_NAME_FIELDS macro doesn't seem to be used anywhere,
which is good, because macros with unbalanced parentheses are usually
not a good plan; and a macro that expands to a comma-separated list of
things is suspect too.

"invalid skip options\n" seems too plural.

With regard to your use of strtol() for --{start,end}block, telling
the user that their input is garbage seems pejorative, even though it
may be accurate. Compare:

[rhaas EDBAS]$ pg_dump -jdsgdsgd
pg_dump: error: invalid number of parallel jobs

In the message "relation end block argument precedes start block
argument\n", I think you could lose both instances of the word
"argument" and probably the word "relation" as well. I actually don't
know why all of these messages about start and end block mention
"relation". It's not like there is some other kind of
non-relation-related start block with which it could be confused.

The comment for run_command() explains some things about the cparams
argument, but those things are false. In fact the argument is unused.

Usual PostgreSQL practice when freeing memory in e.g.
verify_heap_slot_handler is to set the pointers to NULL as well. The
performance cost of this is trivial, and it makes debugging a lot
easier should somebody accidentally write code to access one of those
things after it's been freed.

The documentation says that -D "does exclude any database that was
listed explicitly as dbname on the command line, nor does it exclude
the database chosen in the absence of any dbname argument." The first
part of this makes complete sense to me, but I'm not sure about the
second part. If I type pg_amcheck --all -D 'r*', I think I'm expecting
that "rhaas" won't be checked. Likewise, if I say pg_amcheck -d
'bob*', I think I only want to check the bob-related databases and not
rhaas.

I suggest documenting --endblock as "Check table blocks up to and
including the specified ending block number. An error will occur if a
relation being checked has fewer than this number of blocks." And
similarly for --startblock: "Check table blocks beginning with the
specified block number. An error will occur, etc." Perhaps even
mention something like "This option is probably only useful when
checking a single table." Also, the documentation here isn't clear
that this affects only table checking, not index checking.

It appears that pg_amcheck sometimes makes dummy connections to the
database that don't do anything, e.g. pg_amcheck -t 'q*' resulted in:

2021-03-10 15:00:14.273 EST [95473] LOG:  connection received: host=[local]
2021-03-10 15:00:14.274 EST [95473] LOG:  connection authorized:
user=rhaas database=rhaas application_name=pg_amcheck
2021-03-10 15:00:14.286 EST [95473] LOG:  statement: SELECT
pg_catalog.set_config('search_path', '', false);
2021-03-10 15:00:14.290 EST [95464] DEBUG:  forked new backend,
pid=95474 socket=11
2021-03-10 15:00:14.291 EST [95464] DEBUG:  server process (PID 95473)
exited with exit code 0
2021-03-10 15:00:14.291 EST [95474] LOG:  connection received: host=[local]
2021-03-10 15:00:14.293 EST [95474] LOG:  connection authorized:
user=rhaas database=rhaas application_name=pg_amcheck
2021-03-10 15:00:14.296 EST [95474] LOG:  statement: SELECT
pg_catalog.set_config('search_path', '', false);
<...more queries from PID 95474...>
2021-03-10 15:00:14.321 EST [95464] DEBUG:  server process (PID 95474)
exited with exit code 0

It doesn't seem to make sense to connect to a database, set the search
path, exit, and then immediately reconnect to the same database.

This is slightly inconsistent:

pg_amcheck: checking heap table "rhaas"."public"."foo"
heap relation "rhaas"."public"."foo":
    ERROR:  XX000: catalog is missing 144 attribute(s) for relid 16392
    LOCATION:  RelationBuildTupleDesc, relcache.c:652
query was: SELECT blkno, offnum, attnum, msg FROM "public".verify_heapam(
relation := 16392, on_error_stop := false, check_toast := true, skip := 'none')

In line 1 it's a heap table, but in line 2 it's a heap relation.

That's all I've got.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  13. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-03-11T04:02:00Z

    
> On Mar 10, 2021, at 12:28 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> 
> On Wed, Mar 10, 2021 at 11:10 AM Mark Dilger
> <mark.dilger@enterprisedb.com> wrote:
>> Once again, I think you are right and have removed the objectionable behavior, but....
>> 
>> The --startblock and --endblock options make the most sense when the user is only checking one table, like
>> 
>>  pg_amcheck --startblock=17 --endblock=19 --table=my_schema.my_corrupt_table
>> 
>> because the user likely has some knowledge about that table, perhaps from a prior run of pg_amcheck.  The --startblock and --endblock arguments are a bit strange when used globally, as relations don't all have the same number of blocks, so
>> 
>>  pg_amcheck --startblock=17 --endblock=19 mydb
>> 
>> will very likely emit lots of error messages for tables which don't have blocks in that range.  That's not entirely pg_amcheck's fault, as it just did what the user asked, but it also doesn't seem super helpful.  I'm not going to do anything about it in this release.
> 
> +1 to all that. I tend toward the opinion that trying to make
> --startblock and --endblock do anything useful in the context of
> checking multiple relations is not really possible, and therefore we
> just shouldn't put any effort into it. But if user feedback shows
> otherwise, we can always do something about it later.
> 
>> After running 'make installcheck', if I delete all entries from pg_class where relnamespace = 'pg_toast'::regclass, by running 'pg_amcheck regression', I get lines that look like this:
>> 
>> heap relation "regression"."public"."quad_poly_tbl":
>>    ERROR:  could not open relation with OID 17177
> 
> In this here example, the first line ends in a colon.
> 
>> relation "regression"."public"."functional_dependencies", block 28, offset 54, attribute 0
>>    attribute 0 with length 4294967295 ends at offset 50 beyond total tuple length 43
> 
> But this here one does not. Seems like it should be consistent.

Good point.  It also seems inconsistent that in one it refers to a "relation" and in the other to a "heap relation", but they're both heap relations.  Changed to use "heap relation" both places, and to both use colons.

> 
> The QUALIFIED_NAME_FIELDS macro doesn't seem to be used anywhere,
> which is good, because macros with unbalanced parentheses are usually
> not a good plan; and a macro that expands to a comma-separated list of
> things is suspect too.

Yeah, that whole macro was supposed to be removed.  Looks like I somehow only removed the end of it, plus some functions that were using it.  Not sure how I fat fingered that in the editor, but I've removed the rest now.

> "invalid skip options\n" seems too plural.

Changed to something less plural.

> With regard to your use of strtol() for --{start,end}block, telling
> the user that their input is garbage seems pejorative, even though it
> may be accurate. Compare:
> 
> [rhaas EDBAS]$ pg_dump -jdsgdsgd
> pg_dump: error: invalid number of parallel jobs
> 
> In the message "relation end block argument precedes start block
> argument\n", I think you could lose both instances of the word
> "argument" and probably the word "relation" as well. I actually don't
> know why all of these messages about start and end block mention
> "relation". It's not like there is some other kind of
> non-relation-related start block with which it could be confused.

Changed.

> The comment for run_command() explains some things about the cparams
> argument, but those things are false. In fact the argument is unused.

Removed unused argument and associated comment.

> Usual PostgreSQL practice when freeing memory in e.g.
> verify_heap_slot_handler is to set the pointers to NULL as well. The
> performance cost of this is trivial, and it makes debugging a lot
> easier should somebody accidentally write code to access one of those
> things after it's been freed.

I had been doing that and removed it, anticipating a complaint about useless code.  Ok, I put it back.

> The documentation says that -D "does exclude any database that was
> listed explicitly as dbname on the command line, nor does it exclude
> the database chosen in the absence of any dbname argument." The first
> part of this makes complete sense to me, but I'm not sure about the
> second part. If I type pg_amcheck --all -D 'r*', I think I'm expecting
> that "rhaas" won't be checked. Likewise, if I say pg_amcheck -d
> 'bob*', I think I only want to check the bob-related databases and not
> rhaas.

I think it's a tricky definitional problem.  I'll argue the other side for the moment:

If you say `pg_amcheck bob`, I think it is fair to assume that "bob" gets checked.  If you say `pg_amcheck bob -d="b*" -D="bo*"`, it is fair to expect all databases starting with /b/ to be checked, except those starting with /bo/, except that since you *explicitly* asked for "bob", that "bob" gets checked.  We both agree on this point, I think.

If you say `pg_amcheck --maintenance-db=bob -d="b*" -D="bo*", you don't expect "bob" to get checked, even though it was explicitly stated.

If you are named "bob", and run `pg_amcheck`, you expect it to get your name "bob" from the environment, and check database "bob".  It's implicit rather than explicit, but that doesn't change what you expect to happen.  It's just a short-hand for saying `pg_amcheck bob`.

Saying that `pg_amcheck -d="b*" -D="bo*" should not check "bob" implies that the database being retrieved from the environment is acting like a maintenance-db.  But that's not how it is treated when you just say `pg_amcheck` with no arguments.  I think treating it as a maintenance-db in some situations but not in others is strangely non-orthogonal.

On the other hand, I would expect some users to come back with precisely your complaint, so I don't know how best to solve this.  

> I suggest documenting --endblock as "Check table blocks up to and
> including the specified ending block number. An error will occur if a
> relation being checked has fewer than this number of blocks." And
> similarly for --startblock: "Check table blocks beginning with the
> specified block number. An error will occur, etc." Perhaps even
> mention something like "This option is probably only useful when
> checking a single table." Also, the documentation here isn't clear
> that this affects only table checking, not index checking.

Changed.

> It appears that pg_amcheck sometimes makes dummy connections to the
> database that don't do anything, e.g. pg_amcheck -t 'q*' resulted in:
> 
> 2021-03-10 15:00:14.273 EST [95473] LOG:  connection received: host=[local]
> 2021-03-10 15:00:14.274 EST [95473] LOG:  connection authorized:
> user=rhaas database=rhaas application_name=pg_amcheck
> 2021-03-10 15:00:14.286 EST [95473] LOG:  statement: SELECT
> pg_catalog.set_config('search_path', '', false);
> 2021-03-10 15:00:14.290 EST [95464] DEBUG:  forked new backend,
> pid=95474 socket=11
> 2021-03-10 15:00:14.291 EST [95464] DEBUG:  server process (PID 95473)
> exited with exit code 0
> 2021-03-10 15:00:14.291 EST [95474] LOG:  connection received: host=[local]
> 2021-03-10 15:00:14.293 EST [95474] LOG:  connection authorized:
> user=rhaas database=rhaas application_name=pg_amcheck
> 2021-03-10 15:00:14.296 EST [95474] LOG:  statement: SELECT
> pg_catalog.set_config('search_path', '', false);
> <...more queries from PID 95474...>
> 2021-03-10 15:00:14.321 EST [95464] DEBUG:  server process (PID 95474)
> exited with exit code 0
> 
> It doesn't seem to make sense to connect to a database, set the search
> path, exit, and then immediately reconnect to the same database.

I think I've cleaned that up now.

> This is slightly inconsistent:
> 
> pg_amcheck: checking heap table "rhaas"."public"."foo"
> heap relation "rhaas"."public"."foo":
>    ERROR:  XX000: catalog is missing 144 attribute(s) for relid 16392
>    LOCATION:  RelationBuildTupleDesc, relcache.c:652
> query was: SELECT blkno, offnum, attnum, msg FROM "public".verify_heapam(
> relation := 16392, on_error_stop := false, check_toast := true, skip := 'none')
> 
> In line 1 it's a heap table, but in line 2 it's a heap relation.

Changed to use "heap table" consistently, and along those lines, to use "btree index" rather than "btree relation".

  14. Re: pg_amcheck contrib application

    Peter Eisentraut <peter.eisentraut@enterprisedb.com> — 2021-03-11T08:12:22Z

    (I'm still not a fan of adding more client-side tools whose sole task is 
to execute server-side functionality in a slightly filtered way, but it 
seems people are really interested in this, so ...)

I want to register, if we are going to add this, it ought to be in 
src/bin/.  If we think it's a useful tool, it should be there with all 
the other useful tools.

I realize there is a dependency on a module in contrib, and it's 
probably now not the time to re-debate reorganizing contrib.  But if we 
ever get to that, this program should be the prime example why the 
current organization is problematic, and we should be prepared to make 
the necessary moves then.

  15. Re: pg_amcheck contrib application

    x4mmm@yandex-team.ru — 2021-03-11T11:36:17Z

    
> 11 марта 2021 г., в 13:12, Peter Eisentraut <peter.eisentraut@enterprisedb.com> написал(а):
> 
> client-side tools whose sole task is to execute server-side functionality in a slightly filtered way

By the way, can we teach pg_amcheck to verify database without creating local PGDATA and using bare minimum of file system quota?

We can implement a way for a pg_amcheck to ask for some specific file, which will be downloaded by backup tool and streamed to pg_amcheck.
E.g. pg_amcheck could have a restore_file_command = 'backup-tool bring-my-file %backup_id %file_name' and probably list_files_command='backup-tool list-files %backup_id'. And pg_amcheck could then fetch bare minimum of what is needed.

I see that this is somewhat orthogonal idea, but from my POV interesting one.

Best regards, Andrey Borodin.

  16. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-03-11T15:14:49Z

    
> On Mar 11, 2021, at 12:12 AM, Peter Eisentraut <peter.eisentraut@enterprisedb.com> wrote:
> 
> (I'm still not a fan of adding more client-side tools whose sole task is to execute server-side functionality in a slightly filtered way, but it seems people are really interested in this, so ...)
> 
> I want to register, if we are going to add this, it ought to be in src/bin/.  If we think it's a useful tool, it should be there with all the other useful tools.

I considered putting it in src/bin/scripts where reindexdb and vacuumdb also live.  It seems most similar to those two tools.

> I realize there is a dependency on a module in contrib, and it's probably now not the time to re-debate reorganizing contrib.  But if we ever get to that, this program should be the prime example why the current organization is problematic, and we should be prepared to make the necessary moves then.

Before settling on contrib/pg_amcheck as the location, I checked whether any tools under src/bin had dependencies on a contrib module, and couldn't find any current examples.  (There seems to have been one in the past, though I forget which that was at the moment.)

I have no argument with changing the location of this tool before it gets committed, but I wonder if we should do that now, or wait until some future time when contrib gets reorganized?  I can't quite tell which you prefer from your comments above.

—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  17. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-03-11T15:30:13Z

    
> On Mar 11, 2021, at 3:36 AM, Andrey Borodin <x4mmm@yandex-team.ru> wrote:
> 
> 
> 
>> 11 марта 2021 г., в 13:12, Peter Eisentraut <peter.eisentraut@enterprisedb.com> написал(а):
>> 
>> client-side tools whose sole task is to execute server-side functionality in a slightly filtered way
> 
> By the way, can we teach pg_amcheck to verify database without creating local PGDATA and using bare minimum of file system quota?

pg_amcheck does not need a local data directory to check a remote database server, though it does need to connect to that server.  The local file system quota should not be a problem, as pg_amcheck does not download and save any data to disk.  I am uncertain if this answers your question.  If you are imagining pg_amcheck running on the same server as the database cluster, then of course running pg_amcheck puts a burden on the server to read all the relation files necessary, much as running queries over the same relations would do.

> We can implement a way for a pg_amcheck to ask for some specific file, which will be downloaded by backup tool and streamed to pg_amcheck.
> E.g. pg_amcheck could have a restore_file_command = 'backup-tool bring-my-file %backup_id %file_name' and probably list_files_command='backup-tool list-files %backup_id'. And pg_amcheck could then fetch bare minimum of what is needed.
> 
> I see that this is somewhat orthogonal idea, but from my POV interesting one.

pg_amcheck is not designed to detect corruption directly, but rather to open one or more connections to the database and execute sql queries which employ the contrib/amcheck sql functions.

—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  18. Re: pg_amcheck contrib application

    Robert Haas <robertmhaas@gmail.com> — 2021-03-11T15:46:12Z

    On Thu, Mar 11, 2021 at 3:12 AM Peter Eisentraut
<peter.eisentraut@enterprisedb.com> wrote:
> (I'm still not a fan of adding more client-side tools whose sole task is
> to execute server-side functionality in a slightly filtered way, but it
> seems people are really interested in this, so ...)
>
> I want to register, if we are going to add this, it ought to be in
> src/bin/.  If we think it's a useful tool, it should be there with all
> the other useful tools.

I think this provides a pretty massive gain in usability. If you
wanted to check all of your tables and btree indexes without this, or
worse yet some subset of them that satisfied certain criteria, it
would be a real nuisance. You don't want to run all of the check
commands in a single transaction, because that keeps snapshots open,
and there's a good chance you do want to use parallelism. Even if you
ignore all that, the output you're going to get from running the
queries individually in psql is not going to be easy to sort through,
whereas the tool is going to distill that down to what you really need
to know.

Perhaps we should try to think of some way that some of these tools
could be unified, since it does seem a bit silly to have reindexdb,
vacuumdb, and pg_amcheck all as separate commands basically doing the
same kind of thing but for different maintenance operations, but I
don't think getting rid of them entirely is the way - and I don't
think that unifying them is a v14 project.

I also had the thought that maybe this should go in src/bin, because I
think this is going to be awfully handy for a lot of people. However,
I don't think there's a rule that binaries can't go in contrib --
oid2name and vacuumlo are existing precedents. But I guess that's only
2 out of quite a large number of binaries that we ship, so maybe it's
best not to add to it, especially for a tool which I at least suspect
is going to get a lot more use than either of those.

Anyone else want to vote for or against moving this to src/bin?

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  19. Re: pg_amcheck contrib application

    Robert Haas <robertmhaas@gmail.com> — 2021-03-11T16:09:52Z

    On Wed, Mar 10, 2021 at 11:02 PM Mark Dilger
<mark.dilger@enterprisedb.com> wrote:
> > The documentation says that -D "does exclude any database that was
> > listed explicitly as dbname on the command line, nor does it exclude
> > the database chosen in the absence of any dbname argument." The first
> > part of this makes complete sense to me, but I'm not sure about the
> > second part. If I type pg_amcheck --all -D 'r*', I think I'm expecting
> > that "rhaas" won't be checked. Likewise, if I say pg_amcheck -d
> > 'bob*', I think I only want to check the bob-related databases and not
> > rhaas.
>
> I think it's a tricky definitional problem.  I'll argue the other side for the moment:
>
> If you say `pg_amcheck bob`, I think it is fair to assume that "bob" gets checked.  If you say `pg_amcheck bob -d="b*" -D="bo*"`, it is fair to expect all databases starting with /b/ to be checked, except those starting with /bo/, except that since you *explicitly* asked for "bob", that "bob" gets checked.  We both agree on this point, I think.

+1.

> If you say `pg_amcheck --maintenance-db=bob -d="b*" -D="bo*", you don't expect "bob" to get checked, even though it was explicitly stated.

I expect that specifying --maintenance-db has zero effect on what gets
checked. The only thing that should do is tell me which database to
use to get the list of databases that I am going to check, just in
case the default is unsuitable and will fail.

> If you are named "bob", and run `pg_amcheck`, you expect it to get your name "bob" from the environment, and check database "bob".  It's implicit rather than explicit, but that doesn't change what you expect to happen.  It's just a short-hand for saying `pg_amcheck bob`.

+1.

> Saying that `pg_amcheck -d="b*" -D="bo*" should not check "bob" implies that the database being retrieved from the environment is acting like a maintenance-db.  But that's not how it is treated when you just say `pg_amcheck` with no arguments.  I think treating it as a maintenance-db in some situations but not in others is strangely non-orthogonal.

I don't think I agree with this. A maintenance DB in my mind doesn't
mean "a database we're not actually checking," but rather "a database
that we're using to get a list of other databases."

TBH, I guess I actually don't know why we ever treat a bare
command-line argument as a maintenance DB. I probably wouldn't do
that. We should only need a maintenance DB if we need to query for a
list of database to check, and if the user has explicitly named the
database to check, then we do not need to do that... unless they've
also done something like -D or -d, but then the explicitly-specified
database name is playing a double role. It is both one of the
databases we will check, and also the database we will use to figure
out what other databases to check. I think that's why this seems
non-orthogonal.

Here's my proposal:

1. If there are options present which require querying for a list of
databases (e.g. --all, -d, -D) then use connectMaintenanceDatabase()
and go figure out what they mean. The cparams passed to that function
are only affected by the use of --maintenance-db, not by any bare
command line arguments. If there are no arguments present which
require querying for a list of databases, then --maintenance-db has no
effect.

2. If there is a bare command line argument, add the named database to
the list of databases to be checked. This might be empty if no
relevant options were specified in step 1, or if those options matched
nothing. It might be a noop if the named database was already selected
by the options mentioned in step 1.

3. If there were no options present which required querying for a list
of databases, and if there is also no bare command line argument, then
default to the checking whatever database we connect to by default.

With this approach, --maintenance-db only ever affects how we get the
list of databases to check, and a bare command-line argument only ever
specifies a database to be checked. That seems cleaner.

An alternate possibility would be to say that there should only ever
be EITHER a bare command-line argument OR options that require
querying for a list of databases OR neither BUT NOT both. Then it's
simple:

0. If you have both options which require querying for a list of
databases and also a bare database name, error and die.
1. As above.
2. As above except the only possibility is now increasing the list of
target databases from length 0 to length 1.
3. As above.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  20. Re: pg_amcheck contrib application

    x4mmm@yandex-team.ru — 2021-03-11T17:10:38Z

    
> 11 марта 2021 г., в 20:30, Mark Dilger <mark.dilger@enterprisedb.com> написал(а):
> 
> 
> pg_amcheck does not need a local data directory to check a remote database server, though it does need to connect to that server.
No, I mean it it would be great if we did not need to materialise whole DB anywhere. Let's say I have a backup of 10Tb cluster in S3. And don't have that clusters hardware anymore. I want to spawn tiny VM with few GiBs of RAM and storage no larger than biggest index within DB + WAL from start to end. And stream-check all backup, mark it safe and sleep well. It would be perfect if we could do backup verification at cost of corruption monitoring (and not vice versa, which is trivial).

Best regards, Andrey Borodin.

  21. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-03-11T17:42:22Z

    
> On Mar 11, 2021, at 9:10 AM, Andrey Borodin <x4mmm@yandex-team.ru> wrote:
> 
> 
> 
>> 11 марта 2021 г., в 20:30, Mark Dilger <mark.dilger@enterprisedb.com> написал(а):
>> 
>> 
>> pg_amcheck does not need a local data directory to check a remote database server, though it does need to connect to that server.
> No, I mean it it would be great if we did not need to materialise whole DB anywhere. Let's say I have a backup of 10Tb cluster in S3. And don't have that clusters hardware anymore. I want to spawn tiny VM with few GiBs of RAM and storage no larger than biggest index within DB + WAL from start to end. And stream-check all backup, mark it safe and sleep well. It would be perfect if we could do backup verification at cost of corruption monitoring (and not vice versa, which is trivial).

Thanks for clarifying.  I agree that would be useful.  I don't see any way to make that part of this project, but maybe after the v14 cycle you'll look over the code a propose a way forward for that?'

—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  22. Re: pg_amcheck contrib application

    Robert Haas <robertmhaas@gmail.com> — 2021-03-11T19:34:03Z

    On Wed, Mar 10, 2021 at 11:02 PM Mark Dilger
<mark.dilger@enterprisedb.com> wrote:
> [ new patches ]

Seems like this is mostly ready to commit now, modulo exactly what to
do about the maintenance DB stuff, and whether to move it to src/bin.
Since neither of those affects 0001, I went ahead and committed that
part.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  23. Re: pg_amcheck contrib application

    Robert Haas <robertmhaas@gmail.com> — 2021-03-11T21:59:00Z

    On Thu, Mar 11, 2021 at 11:09 AM Robert Haas <robertmhaas@gmail.com> wrote:
> An alternate possibility would be to say that there should only ever
> be EITHER a bare command-line argument OR options that require
> querying for a list of databases OR neither BUT NOT both. Then it's
> simple:
>
> 0. If you have both options which require querying for a list of
> databases and also a bare database name, error and die.
> 1. As above.
> 2. As above except the only possibility is now increasing the list of
> target databases from length 0 to length 1.
> 3. As above.

Here's a proposed incremental patch, applying on top of your last
version, that describes the above behavior, plus makes a lot of other
changes to the documentation that seemed like good ideas to me. Your
mileage may vary, but I think this version is substantially more
concise than what you have while basically containing the same
information.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  24. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-03-12T05:00:07Z

    
> On Mar 11, 2021, at 1:59 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> 
> On Thu, Mar 11, 2021 at 11:09 AM Robert Haas <robertmhaas@gmail.com> wrote:
>> An alternate possibility would be to say that there should only ever
>> be EITHER a bare command-line argument OR options that require
>> querying for a list of databases OR neither BUT NOT both. Then it's
>> simple:
>> 
>> 0. If you have both options which require querying for a list of
>> databases and also a bare database name, error and die.
>> 1. As above.
>> 2. As above except the only possibility is now increasing the list of
>> target databases from length 0 to length 1.
>> 3. As above.
> 
> Here's a proposed incremental patch, applying on top of your last
> version, that describes the above behavior, plus makes a lot of other
> changes to the documentation that seemed like good ideas to me. Your
> mileage may vary, but I think this version is substantially more
> concise than what you have while basically containing the same
> information.

Your proposal is used in this next version of the patch, along with a resolution to the solution to the -D option handling, discussed before, and a change to make --schema and --exclude-schema options accept "database.schema" patterns as well as "schema" patterns.  It previously only interpreted the parameter as a schema without treating embedded dots as separators, but that seems strangely inconsistent with the way all the other pattern options work, so I made it consistent.  (I think the previous behavior was defensible, but harder to explain and perhaps less intuitive.)

  25. Re: pg_amcheck contrib application

    Robert Haas <robertmhaas@gmail.com> — 2021-03-12T13:33:14Z

    On Fri, Mar 12, 2021 at 12:00 AM Mark Dilger
<mark.dilger@enterprisedb.com> wrote:
> Your proposal is used in this next version of the patch, along with a resolution to the solution to the -D option handling, discussed before, and a change to make --schema and --exclude-schema options accept "database.schema" patterns as well as "schema" patterns.  It previously only interpreted the parameter as a schema without treating embedded dots as separators, but that seems strangely inconsistent with the way all the other pattern options work, so I made it consistent.  (I think the previous behavior was defensible, but harder to explain and perhaps less intuitive.)

Well, OK. In that case I guess we need to patch the docs a little
more. Here's a patch documentation that revised behavior, and also
tidying up a few other things I noticed along the way.

Since nobody is saying we *shouldn't* move this to src/bin, I think
you may as well go put it there per Peter's suggestion.

Then I think it's time to get this committed and move on to the next thing.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  26. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-03-12T16:41:44Z

    
> On Mar 12, 2021, at 5:33 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> 
> Well, OK. In that case I guess we need to patch the docs a little
> more. Here's a patch documentation that revised behavior, and also
> tidying up a few other things I noticed along the way.
> 
> Since nobody is saying we *shouldn't* move this to src/bin, I think
> you may as well go put it there per Peter's suggestion.
> 
> Then I think it's time to get this committed and move on to the next thing.

In this next patch, your documentation patch has been applied, and the whole project has been relocated from contrib/pg_amcheck to src/bin/pg_amcheck.

  27. Re: pg_amcheck contrib application

    Robert Haas <robertmhaas@gmail.com> — 2021-03-12T18:10:03Z

    On Fri, Mar 12, 2021 at 11:41 AM Mark Dilger
<mark.dilger@enterprisedb.com> wrote:
> In this next patch, your documentation patch has been applied, and the whole project has been relocated from contrib/pg_amcheck to src/bin/pg_amcheck.

Committed that way with some small adjustments. Let's see what the
buildfarm thinks.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  28. Re: pg_amcheck contrib application

    Peter Geoghegan <pg@bowt.ie> — 2021-03-12T18:32:10Z

    On Fri, Mar 12, 2021 at 10:10 AM Robert Haas <robertmhaas@gmail.com> wrote:
> Committed that way with some small adjustments. Let's see what the
> buildfarm thinks.

Thank you both, Mark and Robert. This is excellent work!

-- 
Peter Geoghegan

  29. Re: pg_amcheck contrib application

    Peter Geoghegan <pg@bowt.ie> — 2021-03-12T18:35:32Z

    On Fri, Mar 12, 2021 at 10:32 AM Peter Geoghegan <pg@bowt.ie> wrote:
> Thank you both, Mark and Robert. This is excellent work!

FYI I see these compiler warnings just now:

pg_amcheck.c:1653:4: warning: ISO C90 forbids mixed declarations and
code [-Wdeclaration-after-statement]
 1653 |    DatabaseInfo *dat = (DatabaseInfo *)
pg_malloc0(sizeof(DatabaseInfo));
      |    ^~~~~~~~~~~~
pg_amcheck.c: In function ‘compile_relation_list_one_db’:
pg_amcheck.c:2060:9: warning: variable ‘is_btree’ set but not used
[-Wunused-but-set-variable]
 2060 |   bool  is_btree = false;
      |         ^~~~~~~~

Looks like this 'is_btree' variable should be PG_USED_FOR_ASSERTS_ONLY.

-- 
Peter Geoghegan

  30. Re: pg_amcheck contrib application

    Erik Rijkers <er@xs4all.nl> — 2021-03-12T19:05:03Z

    > On 2021.03.12. 19:10 Robert Haas <robertmhaas@gmail.com> wrote:
> 
>  
> On Fri, Mar 12, 2021 at 11:41 AM Mark Dilger
> <mark.dilger@enterprisedb.com> wrote:
> > In this next patch, your documentation patch has been applied, and the whole project has been relocated from contrib/pg_amcheck to src/bin/pg_amcheck.
> 
> Committed that way with some small adjustments. Let's see what the
> buildfarm thinks.
> 

Hi,

An output-formatting error, I think:

I ran pg_amcheck against a 1.5 GB table:

-- pg_amcheck --progress --on-error-stop --heapallindexed -vt myjsonfile100k

pg_amcheck: including database: "testdb"
pg_amcheck: in database "testdb": using amcheck version "1.3" in schema "public"
0/4 relations (0%)      0/187978 pages (0%)                                      
pg_amcheck: checking heap table "testdb"."public"."myjsonfile100k"
pg_amcheck: checking btree index "testdb"."public"."myjsonfile100k_pkey"
2/4 relations (50%) 187977/187978 pages (99%), (testdb                             )
pg_amcheck: checking btree index "testdb"."pg_toast"."pg_toast_26110_index"
3/4 relations (75%) 187978/187978 pages (100%), (testdb                             )
pg_amcheck: checking heap table "testdb"."pg_toast"."pg_toast_26110"
4/4 relations (100%) 187978/187978 pages (100%) 


I think there is a formatting glitch in lines like:

2/4 relations (50%) 187977/187978 pages (99%), (testdb                             )

I suppose that last part should show up trimmed as '(testdb)', right?

Thanks,

Erik Rijkers

  31. Re: pg_amcheck contrib application

    Robert Haas <robertmhaas@gmail.com> — 2021-03-12T19:24:01Z

    On Fri, Mar 12, 2021 at 2:05 PM <er@xs4all.nl> wrote:
> I think there is a formatting glitch in lines like:
>
> 2/4 relations (50%) 187977/187978 pages (99%), (testdb                             )
>
> I suppose that last part should show up trimmed as '(testdb)', right?

Actually I think this is intentional. The idea is that as the line is
rewritten we don't want the close-paren to move around. It's the same
thing pg_basebackup does with its progress reporting.

Now that is not to say that some other behavior might not be better,
just that Mark was copying something that already exists, probably
because he knows that I'm finnicky about consistency....

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  32. Re: pg_amcheck contrib application

    Robert Haas <robertmhaas@gmail.com> — 2021-03-12T19:31:36Z

    On Fri, Mar 12, 2021 at 1:35 PM Peter Geoghegan <pg@bowt.ie> wrote:
> On Fri, Mar 12, 2021 at 10:32 AM Peter Geoghegan <pg@bowt.ie> wrote:
> > Thank you both, Mark and Robert. This is excellent work!

Thanks.

> FYI I see these compiler warnings just now:
>
> pg_amcheck.c:1653:4: warning: ISO C90 forbids mixed declarations and
> code [-Wdeclaration-after-statement]
>  1653 |    DatabaseInfo *dat = (DatabaseInfo *)
> pg_malloc0(sizeof(DatabaseInfo));
>       |    ^~~~~~~~~~~~
> pg_amcheck.c: In function ‘compile_relation_list_one_db’:
> pg_amcheck.c:2060:9: warning: variable ‘is_btree’ set but not used
> [-Wunused-but-set-variable]
>  2060 |   bool  is_btree = false;
>       |         ^~~~~~~~
>
> Looks like this 'is_btree' variable should be PG_USED_FOR_ASSERTS_ONLY.

I'll commit something shortly to address these.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  33. Re: pg_amcheck contrib application

    Robert Haas <robertmhaas@gmail.com> — 2021-03-12T21:43:05Z

    On Fri, Mar 12, 2021 at 2:31 PM Robert Haas <robertmhaas@gmail.com> wrote:
> I'll commit something shortly to address these.

There are some interesting failures in the test cases on the
buildfarm. One of the tests ($offnum == 13) corrupts the TOAST pointer
with a garbage value, expecting to get the message "final toast chunk
number 0 differs from expected value 6". But on florican and maybe
other systems we instead get "final toast chunk number 0 differs from
expected value 5". That's because the value of TOAST_MAX_CHUNK_SIZE
depends on MAXIMUM_ALIGNOF. I think that on 4-byte alignment systems
it works out to 2000 and on 8-byte alignment systems it works out to
1996, and the value being stored is 10000 bytes, hence the problem.
The place where the calculation goes different seems to be in
MaximumBytesPerTuple(), where it uses MAXALIGN_DOWN() on a value that,
according to my calculations, will be 2038 on all platforms, but the
output of MAXALIGN_DOWN() will be 2032 or 2036 depending on the
platform. I think the solution to this is just to change the message
to match \d+ chunks instead of exactly 6. We should do that right away
to avoid having the buildfarm barf.

But, I also notice a couple of other things I think could be improved here:

1. amcheck is really reporting the complete absence of any TOAST rows
here due to a corrupted va_valueid. It could pick a better phrasing of
that message than "final toast chunk number 0 differs from expected
value XXX". I mean, there is no chunk 0. There are no chunks at all.

2. Using SSSSSSSSS as the perl unpack code for the varlena header is
not ideal, because it's really 2 1-byte fields followed by 4 4-byte
fields. So I think you should be using CCllLL, for two unsigned bytes
and then two signed 4-byte quantities and then two unsigned 4-byte
quantities. I think if you did that you'd be overwriting the
va_valueid with the *same* garbage value on every platform, which
would be better than different ones. Perhaps when we improve the
message as suggested in (1) this will become a live issue, since we
might choose to say something like "no TOAST entries for value %u".

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  34. Re: pg_amcheck contrib application

    Peter Geoghegan <pg@bowt.ie> — 2021-03-12T22:08:49Z

    On Fri, Mar 12, 2021 at 1:43 PM Robert Haas <robertmhaas@gmail.com> wrote:
> There are some interesting failures in the test cases on the
> buildfarm.

I wonder if Andrew Dunstan (now CC'd) could configure his crake
buildfarm member to run pg_amcheck with the most expensive and
thorough options on the master branch (plus all new major version
branches going forward).

That would give us some degree of amcheck test coverage in the back
branches right away. It might even detect cross-version
inconsistencies. Or even pg_upgrade bugs.

-- 
Peter Geoghegan

  35. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-03-12T22:24:18Z

    
> On Mar 12, 2021, at 1:43 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> 
> On Fri, Mar 12, 2021 at 2:31 PM Robert Haas <robertmhaas@gmail.com> wrote:
>> I'll commit something shortly to address these.
> 
> There are some interesting failures in the test cases on the
> buildfarm. One of the tests ($offnum == 13) corrupts the TOAST pointer
> with a garbage value, expecting to get the message "final toast chunk
> number 0 differs from expected value 6". But on florican and maybe
> other systems we instead get "final toast chunk number 0 differs from
> expected value 5". That's because the value of TOAST_MAX_CHUNK_SIZE
> depends on MAXIMUM_ALIGNOF. I think that on 4-byte alignment systems
> it works out to 2000 and on 8-byte alignment systems it works out to
> 1996, and the value being stored is 10000 bytes, hence the problem.
> The place where the calculation goes different seems to be in
> MaximumBytesPerTuple(), where it uses MAXALIGN_DOWN() on a value that,
> according to my calculations, will be 2038 on all platforms, but the
> output of MAXALIGN_DOWN() will be 2032 or 2036 depending on the
> platform. I think the solution to this is just to change the message
> to match \d+ chunks instead of exactly 6. We should do that right away
> to avoid having the buildfarm barf.
> 
> But, I also notice a couple of other things I think could be improved here:
> 
> 1. amcheck is really reporting the complete absence of any TOAST rows
> here due to a corrupted va_valueid. It could pick a better phrasing of
> that message than "final toast chunk number 0 differs from expected
> value XXX". I mean, there is no chunk 0. There are no chunks at all.
> 
> 2. Using SSSSSSSSS as the perl unpack code for the varlena header is
> not ideal, because it's really 2 1-byte fields followed by 4 4-byte
> fields. So I think you should be using CCllLL, for two unsigned bytes
> and then two signed 4-byte quantities and then two unsigned 4-byte
> quantities. I think if you did that you'd be overwriting the
> va_valueid with the *same* garbage value on every platform, which
> would be better than different ones. Perhaps when we improve the
> message as suggested in (1) this will become a live issue, since we
> might choose to say something like "no TOAST entries for value %u".
> 
> -- 
> Robert Haas
> EDB: http://www.enterprisedb.com


This does nothing to change the verbiage from contrib/amcheck, but it should address the problems discussed here in pg_amcheck's regression tests.

  36. Re: pg_amcheck contrib application

    Robert Haas <robertmhaas@gmail.com> — 2021-03-12T22:55:35Z

    On Fri, Mar 12, 2021 at 5:24 PM Mark Dilger
<mark.dilger@enterprisedb.com> wrote:
> This does nothing to change the verbiage from contrib/amcheck, but it should address the problems discussed here in pg_amcheck's regression tests.

Committed.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  37. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-03-12T23:24:32Z

    
> On Mar 12, 2021, at 2:55 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> 
> On Fri, Mar 12, 2021 at 5:24 PM Mark Dilger
> <mark.dilger@enterprisedb.com> wrote:
>> This does nothing to change the verbiage from contrib/amcheck, but it should address the problems discussed here in pg_amcheck's regression tests.
> 
> Committed.

Thanks.

There are two more, attached here.  The first deals with error message text which differs between farm animals, and the second deals with an apparent problem with IPC::Run shell expanding an asterisk on some platforms but not others.  That second one, if true, seems like a problem with scope beyond the pg_amcheck project, as TestLib::command_checks_all uses IPC::Run, and it would be desirable to have consistent behavior across platforms.

  38. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-03-12T23:32:03Z

    
> On Mar 12, 2021, at 11:24 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> 
> On Fri, Mar 12, 2021 at 2:05 PM <er@xs4all.nl> wrote:
>> I think there is a formatting glitch in lines like:
>> 
>> 2/4 relations (50%) 187977/187978 pages (99%), (testdb                             )
>> 
>> I suppose that last part should show up trimmed as '(testdb)', right?
> 
> Actually I think this is intentional. The idea is that as the line is
> rewritten we don't want the close-paren to move around. It's the same
> thing pg_basebackup does with its progress reporting.
> 
> Now that is not to say that some other behavior might not be better,
> just that Mark was copying something that already exists, probably
> because he knows that I'm finnicky about consistency....

I think Erik's test case only checked one database, which might be why it looked odd to him.  But consider:

  pg_amcheck -d foo -d bar -d myreallylongdatabasename -d myshortername -d baz --progress

The tool will respect your database ordering, and check foo, then bar, etc.  If you have --jobs greater than one, it will start checking some relations in bar before finishing all relations in foo, but with a fudge factor, pg_amcheck can report that the processing has moved on to database bar, etc.

You wouldn't want the parens to jump around when the long database names get processed.  So it keeps the parens in the same location, space pads shorter database names, and truncates overlong database names.


—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  39. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-03-13T01:04:09Z

    
> On Mar 12, 2021, at 3:24 PM, Mark Dilger <mark.dilger@enterprisedb.com> wrote:
> 
> and the second deals with an apparent problem with IPC::Run shell expanding an asterisk on some platforms but not others.  That second one, if true, seems like a problem with scope beyond the pg_amcheck project, as TestLib::command_checks_all uses IPC::Run, and it would be desirable to have consistent behavior across platforms.

The problem with IPC::Run appears to be real, though I might just need to wait longer for the farm animals to prove me wrong about that.  But there is a similar symptom caused by an unrelated problem, one entirely my fault and spotted by Robert.  Here is a patch:

  40. Re: pg_amcheck contrib application

    Robert Haas <robertmhaas@gmail.com> — 2021-03-13T01:16:22Z

    On Fri, Mar 12, 2021 at 8:04 PM Mark Dilger
<mark.dilger@enterprisedb.com> wrote:
> The problem with IPC::Run appears to be real, though I might just need to wait longer for the farm animals to prove me wrong about that.  But there is a similar symptom caused by an unrelated problem, one entirely my fault and spotted by Robert.  Here is a patch:

OK, I committed this too, along with the one I hadn't committed yet
from your previous email. Gah, tests are so annoying. :-)

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  41. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-03-13T04:41:50Z

    
> On Mar 12, 2021, at 5:16 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> 
> Gah, tests are so annoying. :-)

There is another problem of non-portable option ordering in the tests.

  42. Re: pg_amcheck contrib application

    Tom Lane <tgl@sss.pgh.pa.us> — 2021-03-13T04:53:23Z

    Mark Dilger <mark.dilger@enterprisedb.com> writes:
> There is another problem of non-portable option ordering in the tests.

Don't almost all of the following tests have the same issue?

			regards, tom lane

  43. Re: pg_amcheck contrib application

    Tom Lane <tgl@sss.pgh.pa.us> — 2021-03-13T04:56:12Z

    ... btw, prairiedog (which has a rather old Perl) has a
different complaint:

Invalid type 'q' in unpack at t/004_verify_heapam.pl line 104.

			regards, tom lane

  44. Re: pg_amcheck contrib application

    Tom Lane <tgl@sss.pgh.pa.us> — 2021-03-13T05:08:13Z

    I wrote:
> Don't almost all of the following tests have the same issue?

Ah, nevermind, I was looking at an older version of 003_check.pl.
I concur that 24189277f missed only one here.

Pushed your fix.

			regards, tom lane

  45. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-03-13T05:09:21Z

    
> On Mar 12, 2021, at 9:08 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> 
> I wrote:
>> Don't almost all of the following tests have the same issue?
> 
> Ah, nevermind, I was looking at an older version of 003_check.pl.
> I concur that 24189277f missed only one here.
> 
> Pushed your fix.
> 
> 			regards, tom lane

Thanks!  Was just responding to your other email, but now I don't have to send it.

Sorry for painting so many farm animals red this evening.

—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  46. Re: pg_amcheck contrib application

    Tom Lane <tgl@sss.pgh.pa.us> — 2021-03-13T05:27:30Z

    I wrote:
> ... btw, prairiedog (which has a rather old Perl) has a
> different complaint:
> Invalid type 'q' in unpack at t/004_verify_heapam.pl line 104.

Hmm ... "man perlfunc" on that system quoth

                   q   A signed quad (64-bit) value.
                   Q   An unsigned quad value.
                         (Quads are available only if your system supports 64-bit
                          integer values _and_ if Perl has been compiled to support those.
                          Causes a fatal error otherwise.)

It does not seem unreasonable for us to rely on Perl having that
in 2021, so I'll see about upgrading this perl installation.

(I suppose gaur will need it too, sigh.)

			regards, tom lane

  47. Re: pg_amcheck contrib application

    Tom Lane <tgl@sss.pgh.pa.us> — 2021-03-13T05:30:35Z

    Mark Dilger <mark.dilger@enterprisedb.com> writes:
> Sorry for painting so many farm animals red this evening.

Not to worry.  We go through this sort of fire drill regularly
when somebody pushes a batch of brand new test code.

			regards, tom lane

  48. Re: pg_amcheck contrib application

    Tom Lane <tgl@sss.pgh.pa.us> — 2021-03-13T06:07:15Z

    I wrote:
>> ... btw, prairiedog (which has a rather old Perl) has a
>> different complaint:
>> Invalid type 'q' in unpack at t/004_verify_heapam.pl line 104.

> Hmm ... "man perlfunc" on that system quoth
>                    q   A signed quad (64-bit) value.
>                    Q   An unsigned quad value.
>                          (Quads are available only if your system supports 64-bit
>                           integer values _and_ if Perl has been compiled to support those.
>                           Causes a fatal error otherwise.)
> It does not seem unreasonable for us to rely on Perl having that
> in 2021, so I'll see about upgrading this perl installation.

Hm, wait a minute: hoverfly is showing the same failure, even though
it claims to be running a 64-bit Perl.  Now I'm confused.

			regards, tom lane

  49. Re: pg_amcheck contrib application

    Noah Misch <noah@leadboat.com> — 2021-03-13T06:16:55Z

    On Sat, Mar 13, 2021 at 01:07:15AM -0500, Tom Lane wrote:
> I wrote:
> >> ... btw, prairiedog (which has a rather old Perl) has a
> >> different complaint:
> >> Invalid type 'q' in unpack at t/004_verify_heapam.pl line 104.
> 
> > Hmm ... "man perlfunc" on that system quoth
> >                    q   A signed quad (64-bit) value.
> >                    Q   An unsigned quad value.
> >                          (Quads are available only if your system supports 64-bit
> >                           integer values _and_ if Perl has been compiled to support those.
> >                           Causes a fatal error otherwise.)
> > It does not seem unreasonable for us to rely on Perl having that
> > in 2021, so I'll see about upgrading this perl installation.
> 
> Hm, wait a minute: hoverfly is showing the same failure, even though
> it claims to be running a 64-bit Perl.  Now I'm confused.

On that machine:

[nm@power8-aix 7:0 2021-03-13T06:09:08 ~ 0]$ /usr/bin/perl64 -e 'unpack "q", ""'
[nm@power8-aix 7:0 2021-03-13T06:09:10 ~ 0]$ /usr/bin/perl -e 'unpack "q", ""'
Invalid type 'q' in unpack at -e line 1.

hoverfly does configure with PERL=perl64.  /usr/bin/prove is from the 32-bit
Perl, so I suspect the TAP suites get 32-bit Perl that way.  (There's no
"prove64".)  This test should unpack the field as two 32-bit values, not a
64-bit value, to avoid requiring more from the Perl installation.

  50. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-03-13T06:18:32Z

    
> On Mar 12, 2021, at 10:16 PM, Noah Misch <noah@leadboat.com> wrote:
> 
> On Sat, Mar 13, 2021 at 01:07:15AM -0500, Tom Lane wrote:
>> I wrote:
>>>> ... btw, prairiedog (which has a rather old Perl) has a
>>>> different complaint:
>>>> Invalid type 'q' in unpack at t/004_verify_heapam.pl line 104.
>> 
>>> Hmm ... "man perlfunc" on that system quoth
>>>                   q   A signed quad (64-bit) value.
>>>                   Q   An unsigned quad value.
>>>                         (Quads are available only if your system supports 64-bit
>>>                          integer values _and_ if Perl has been compiled to support those.
>>>                          Causes a fatal error otherwise.)
>>> It does not seem unreasonable for us to rely on Perl having that
>>> in 2021, so I'll see about upgrading this perl installation.
>> 
>> Hm, wait a minute: hoverfly is showing the same failure, even though
>> it claims to be running a 64-bit Perl.  Now I'm confused.
> 
> On that machine:
> 
> [nm@power8-aix 7:0 2021-03-13T06:09:08 ~ 0]$ /usr/bin/perl64 -e 'unpack "q", ""'
> [nm@power8-aix 7:0 2021-03-13T06:09:10 ~ 0]$ /usr/bin/perl -e 'unpack "q", ""'
> Invalid type 'q' in unpack at -e line 1.
> 
> hoverfly does configure with PERL=perl64.  /usr/bin/prove is from the 32-bit
> Perl, so I suspect the TAP suites get 32-bit Perl that way.  (There's no
> "prove64".)  This test should unpack the field as two 32-bit values, not a
> 64-bit value, to avoid requiring more from the Perl installation.

I will post a modified test in a bit that avoids using Q/q.

—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  51. Re: pg_amcheck contrib application

    Tom Lane <tgl@sss.pgh.pa.us> — 2021-03-13T06:22:54Z

    Mark Dilger <mark.dilger@enterprisedb.com> writes:
> On Mar 12, 2021, at 10:16 PM, Noah Misch <noah@leadboat.com> wrote:
>> hoverfly does configure with PERL=perl64.  /usr/bin/prove is from the 32-bit
>> Perl, so I suspect the TAP suites get 32-bit Perl that way.  (There's no
>> "prove64".)

Oh, that's annoying.

>> This test should unpack the field as two 32-bit values, not a
>> 64-bit value, to avoid requiring more from the Perl installation.

> I will post a modified test in a bit that avoids using Q/q.

Coping with both endiannesses might be painful.

			regards, tom lane

  52. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-03-13T06:28:44Z

    
> On Mar 12, 2021, at 10:22 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> 
> Mark Dilger <mark.dilger@enterprisedb.com> writes:
>> On Mar 12, 2021, at 10:16 PM, Noah Misch <noah@leadboat.com> wrote:
>>> hoverfly does configure with PERL=perl64.  /usr/bin/prove is from the 32-bit
>>> Perl, so I suspect the TAP suites get 32-bit Perl that way.  (There's no
>>> "prove64".)
> 
> Oh, that's annoying.
> 
>>> This test should unpack the field as two 32-bit values, not a
>>> 64-bit value, to avoid requiring more from the Perl installation.
> 
>> I will post a modified test in a bit that avoids using Q/q.
> 
> Coping with both endiannesses might be painful.

Not too bad if the bigint value is zero, as both the low and high 32bits will be zero, regardless of endianness.  The question is whether that gives up too much in terms of what the test is trying to do.  I'm not sure that it does, but if you'd rather solve this by upgrading perl, that's ok by me. 

—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  53. Re: pg_amcheck contrib application

    Andres Freund <andres@anarazel.de> — 2021-03-13T06:30:39Z

    Hi,

On 2021-03-13 01:22:54 -0500, Tom Lane wrote:
> Mark Dilger <mark.dilger@enterprisedb.com> writes:
> > On Mar 12, 2021, at 10:16 PM, Noah Misch <noah@leadboat.com> wrote:
> >> hoverfly does configure with PERL=perl64.  /usr/bin/prove is from the 32-bit
> >> Perl, so I suspect the TAP suites get 32-bit Perl that way.  (There's no
> >> "prove64".)
> 
> Oh, that's annoying.

I suspect we could solve that by invoking changing our /usr/bin/prove
invocation to instead be PERL /usr/bin/prove? That might be a good thing
independent of this issue, because it's not unreasonable for a user to
expect that we'd actually use the perl installation they configured...

Although I do not know how prove determines the perl installation it's
going to use for the test scripts...

- Andres

  54. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-03-13T06:33:29Z

    
> On Mar 12, 2021, at 10:28 PM, Mark Dilger <mark.dilger@enterprisedb.com> wrote:
> 
> 
> 
>> On Mar 12, 2021, at 10:22 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> 
>> Mark Dilger <mark.dilger@enterprisedb.com> writes:
>>> On Mar 12, 2021, at 10:16 PM, Noah Misch <noah@leadboat.com> wrote:
>>>> hoverfly does configure with PERL=perl64.  /usr/bin/prove is from the 32-bit
>>>> Perl, so I suspect the TAP suites get 32-bit Perl that way.  (There's no
>>>> "prove64".)
>> 
>> Oh, that's annoying.
>> 
>>>> This test should unpack the field as two 32-bit values, not a
>>>> 64-bit value, to avoid requiring more from the Perl installation.
>> 
>>> I will post a modified test in a bit that avoids using Q/q.
>> 
>> Coping with both endiannesses might be painful.
> 
> Not too bad if the bigint value is zero, as both the low and high 32bits will be zero, regardless of endianness.  The question is whether that gives up too much in terms of what the test is trying to do.  I'm not sure that it does, but if you'd rather solve this by upgrading perl, that's ok by me. 


I'm not advocating that this be the solution, but if we don't fix up the perl end of it, then this test change might be used instead.

  55. Re: pg_amcheck contrib application

    Tom Lane <tgl@sss.pgh.pa.us> — 2021-03-13T06:36:11Z

    Mark Dilger <mark.dilger@enterprisedb.com> writes:
> On Mar 12, 2021, at 10:22 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> Coping with both endiannesses might be painful.

> Not too bad if the bigint value is zero, as both the low and high 32bits will be zero, regardless of endianness.  The question is whether that gives up too much in terms of what the test is trying to do.  I'm not sure that it does, but if you'd rather solve this by upgrading perl, that's ok by me. 

I don't mind updating the perl installations on prairiedog and gaur,
but Noah might have some difficulty with his AIX flotilla, as I believe
he's not sysadmin there.

You might think about using some symmetric-but-not-zero value,
0x01010101 or the like.

			regards, tom lane

  56. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-03-13T06:55:11Z

    
> On Mar 12, 2021, at 10:36 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> 
> Mark Dilger <mark.dilger@enterprisedb.com> writes:
>> On Mar 12, 2021, at 10:22 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>> Coping with both endiannesses might be painful.
> 
>> Not too bad if the bigint value is zero, as both the low and high 32bits will be zero, regardless of endianness.  The question is whether that gives up too much in terms of what the test is trying to do.  I'm not sure that it does, but if you'd rather solve this by upgrading perl, that's ok by me. 
> 
> I don't mind updating the perl installations on prairiedog and gaur,
> but Noah might have some difficulty with his AIX flotilla, as I believe
> he's not sysadmin there.
> 
> You might think about using some symmetric-but-not-zero value,
> 0x01010101 or the like.

I thought about that, but I'm not sure that it proves much more than just using zero.  The test doesn't really do much of interest with this value, and it doesn't seem worth complicating the test.  The idea originally was that perl's "q" pack code would make reading/writing a number such as 12345678 easy, but since it's not easy, this is easy.

—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  57. Re: pg_amcheck contrib application

    Tom Lane <tgl@sss.pgh.pa.us> — 2021-03-13T07:00:55Z

    Mark Dilger <mark.dilger@enterprisedb.com> writes:
> On Mar 12, 2021, at 10:36 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> You might think about using some symmetric-but-not-zero value,
>> 0x01010101 or the like.

> I thought about that, but I'm not sure that it proves much more than just using zero.

Perhaps not.  I haven't really looked at any of this code, so I'll defer
to Robert's judgment about whether this represents an interesting testing
issue.

			regards, tom lane

  58. Re: pg_amcheck contrib application

    Noah Misch <noah@leadboat.com> — 2021-03-13T07:19:13Z

    On Sat, Mar 13, 2021 at 01:36:11AM -0500, Tom Lane wrote:
> Mark Dilger <mark.dilger@enterprisedb.com> writes:
> > On Mar 12, 2021, at 10:22 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> >> Coping with both endiannesses might be painful.
> 
> > Not too bad if the bigint value is zero, as both the low and high 32bits will be zero, regardless of endianness.  The question is whether that gives up too much in terms of what the test is trying to do.  I'm not sure that it does, but if you'd rather solve this by upgrading perl, that's ok by me. 
> 
> I don't mind updating the perl installations on prairiedog and gaur,
> but Noah might have some difficulty with his AIX flotilla, as I believe
> he's not sysadmin there.

The AIX animals have Perl v5.28.1.  hoverfly, in particular, got a big update
package less than a month ago.  Hence, I doubt it's a question of applying
routine updates.  The puzzle would be to either (a) compile a 32-bit Perl that
handles unpack('q') or (b) try a PostgreSQL configuration like "./configure
... PROVE='perl64 /usr/bin/prove --'" to run the TAP suites under perl64.
(For hoverfly, it's enough to run "prove" under $PERL.  mandrill, however,
needs a 32-bit $PERL for plperl, regardless of what it needs for "prove".)
Future AIX packagers would face doing the same.

With v5-0001-pg_amcheck-continuing-to-fix-portability-problems.patch being so
self-contained, something like it is the way to go.

  59. Re: pg_amcheck contrib application

    Tom Lane <tgl@sss.pgh.pa.us> — 2021-03-13T07:36:39Z

    Noah Misch <noah@leadboat.com> writes:
> On Sat, Mar 13, 2021 at 01:36:11AM -0500, Tom Lane wrote:
>> I don't mind updating the perl installations on prairiedog and gaur,
>> but Noah might have some difficulty with his AIX flotilla, as I believe
>> he's not sysadmin there.

> The AIX animals have Perl v5.28.1.  hoverfly, in particular, got a big update
> package less than a month ago.  Hence, I doubt it's a question of applying
> routine updates.  The puzzle would be to either (a) compile a 32-bit Perl that
> handles unpack('q') or (b) try a PostgreSQL configuration like "./configure
> ... PROVE='perl64 /usr/bin/prove --'" to run the TAP suites under perl64.
> (For hoverfly, it's enough to run "prove" under $PERL.  mandrill, however,
> needs a 32-bit $PERL for plperl, regardless of what it needs for "prove".)
> Future AIX packagers would face doing the same.

Yeah.  prairiedog and gaur are both frankenstein systems: some of the
software components are years newer than the underlying OS.  So I don't
mind changing them further; in the end they're both in the buildfarm
for reasons of hardware diversity, not because they represent platforms
anyone would run production PG on.  On the other hand, those AIX animals
represent systems that are still considered production grade (no?), so
we have to be willing to adapt to them not vice versa.

> With v5-0001-pg_amcheck-continuing-to-fix-portability-problems.patch being so
> self-contained, something like it is the way to go.

I'm only concerned about whether the all-zero value causes any
significant degradation in test quality.  Probably Peter G. would
have the most informed opinion about that.

			regards, tom lane

  60. Re: pg_amcheck contrib application

    Robert Haas <robertmhaas@gmail.com> — 2021-03-13T13:49:50Z

    On Sat, Mar 13, 2021 at 1:55 AM Mark Dilger
<mark.dilger@enterprisedb.com> wrote:
> I thought about that, but I'm not sure that it proves much more than just using zero.  The test doesn't really do much of interest with this value, and it doesn't seem worth complicating the test.  The idea originally was that perl's "q" pack code would make reading/writing a number such as 12345678 easy, but since it's not easy, this is easy.

I think it would be good to use a non-zero value here. We're doing a
lot of poking into raw bytes here, and if something goes wrong, a zero
value is more likely to look like something normal than whatever else.
I suggest picking a value where all 8 bytes are the same, but not
zero, and ideally chosen so that they don't look much like any of the
surrounding bytes.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  61. Re: pg_amcheck contrib application

    Tom Lane <tgl@sss.pgh.pa.us> — 2021-03-13T14:50:54Z

    Robert Haas <robertmhaas@gmail.com> writes:
> I think it would be good to use a non-zero value here. We're doing a
> lot of poking into raw bytes here, and if something goes wrong, a zero
> value is more likely to look like something normal than whatever else.
> I suggest picking a value where all 8 bytes are the same, but not
> zero, and ideally chosen so that they don't look much like any of the
> surrounding bytes.

Actually, it seems like we can let pack/unpack deal with byte-swapping
within 32-bit words; what we lose by not using 'q' format is just the
ability to correctly swap the two 32-bit words.  Hence, any value in
which upper and lower halves are the same should work, say
0x1234567812345678.

			regards, tom lane

  62. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-03-13T15:20:00Z

    
> On Mar 13, 2021, at 6:50 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> 
> Robert Haas <robertmhaas@gmail.com> writes:
>> I think it would be good to use a non-zero value here. We're doing a
>> lot of poking into raw bytes here, and if something goes wrong, a zero
>> value is more likely to look like something normal than whatever else.
>> I suggest picking a value where all 8 bytes are the same, but not
>> zero, and ideally chosen so that they don't look much like any of the
>> surrounding bytes.
> 
> Actually, it seems like we can let pack/unpack deal with byte-swapping
> within 32-bit words; what we lose by not using 'q' format is just the
> ability to correctly swap the two 32-bit words.  Hence, any value in
> which upper and lower halves are the same should work, say
> 0x1234567812345678.
> 
> 			regards, tom lane

The heap tuple in question looks as follows, with ???????? in the spot we're debating:

Tuple Layout (values in hex):

t_xmin: 223
t_xmax: 0
t_field3: 0
bi_hi: 0
bi_lo: 0
ip_posid: 1
t_infomask2: 3
t_infomask: b06
t_hoff: 18
t_bits: 0
a_1: ????????
a_2: ????????
b_header: 11        # little-endian, will be 88 on big endian
b_body1: 61
b_body2: 62
b_body3: 63
b_body4: 64
b_body5: 65
b_body6: 66
b_body7: 67
c_va_header: 1
c_va_vartag: 12
c_va_rawsize: 2714
c_va_extsize: 2710

valueid and toastrelid are not shown, as they won't be stable.  Relying on t_xmin to be stable makes the test brittle, but fortunately that is separated from a_1 and a_2 far enough that we should not need to worry about it.

We want to use values that don't look like any of the others.  The complete set of nibbles in the values above is [012345678B], leaving the set [9ACDEF] unused.  The attached patch uses the value DEADF9F9 as it seems a little easier to remember than other permutations of those nibbles.

  63. Re: pg_amcheck contrib application

    Robert Haas <robertmhaas@gmail.com> — 2021-03-13T15:59:59Z

    On Sat, Mar 13, 2021 at 10:20 AM Mark Dilger
<mark.dilger@enterprisedb.com> wrote:
> We want to use values that don't look like any of the others.  The complete set of nibbles in the values above is [012345678B], leaving the set [9ACDEF] unused.  The attached patch uses the value DEADF9F9 as it seems a little easier to remember than other permutations of those nibbles.

OK, committed. The nibbles seem less relevant than the bytes as a
whole, but that's fine.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  64. Re: pg_amcheck contrib application

    Noah Misch <noah@leadboat.com> — 2021-03-13T18:46:12Z

    On Fri, Mar 12, 2021 at 05:04:09PM -0800, Mark Dilger wrote:
> > On Mar 12, 2021, at 3:24 PM, Mark Dilger <mark.dilger@enterprisedb.com> wrote:
> > 
> > and the second deals with an apparent problem with IPC::Run shell expanding an asterisk on some platforms but not others.  That second one, if true, seems like a problem with scope beyond the pg_amcheck project, as TestLib::command_checks_all uses IPC::Run, and it would be desirable to have consistent behavior across platforms.

> > One of pg_amcheck's regression tests was passing an asterisk through
> > TestLib's command_checks_all() command, which gets through to
> > pg_amcheck without difficulty on most platforms, but appears to get
> > shell expanded on Windows (jacana) and AIX (hoverfly).

For posterity, I can't reproduce this on hoverfly.  The suite fails the same
way at f371a4c and f371a4c^.  More-recently (commit 58f5749), the suite passes
even after reverting f371a4c.  Self-contained IPC::Run usage also does not
corroborate the theory:

[nm@power8-aix 8:0 2021-03-13T18:32:23 clean 0]$ perl -MIPC::Run -e 'IPC::Run::run "printf", "%s\n", "*"'
*
[nm@power8-aix 8:0 2021-03-13T18:32:29 clean 0]$ perl -MIPC::Run -e 'IPC::Run::run "sh", "-c", "printf %s\\\\n *"'
COPYRIGHT
GNUmakefile.in
HISTORY
Makefile
README
README.git
aclocal.m4
config
configure
configure.ac
contrib
doc
src

> there is a similar symptom caused by an unrelated problem

> Subject: [PATCH v3] Avoid use of non-portable option ordering in
>  command_checks_all().

On a glibc system, "env POSIXLY_CORRECT=1 make check ..." tests this.

  65. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-03-13T18:51:27Z

    
> On Mar 13, 2021, at 10:46 AM, Noah Misch <noah@leadboat.com> wrote:
> 
> On Fri, Mar 12, 2021 at 05:04:09PM -0800, Mark Dilger wrote:
>>> On Mar 12, 2021, at 3:24 PM, Mark Dilger <mark.dilger@enterprisedb.com> wrote:
>>> 
>>> and the second deals with an apparent problem with IPC::Run shell expanding an asterisk on some platforms but not others.  That second one, if true, seems like a problem with scope beyond the pg_amcheck project, as TestLib::command_checks_all uses IPC::Run, and it would be desirable to have consistent behavior across platforms.
> 
>>> One of pg_amcheck's regression tests was passing an asterisk through
>>> TestLib's command_checks_all() command, which gets through to
>>> pg_amcheck without difficulty on most platforms, but appears to get
>>> shell expanded on Windows (jacana) and AIX (hoverfly).
> 
> For posterity, I can't reproduce this on hoverfly.  The suite fails the same
> way at f371a4c and f371a4c^.  More-recently (commit 58f5749), the suite passes
> even after reverting f371a4c.  Self-contained IPC::Run usage also does not
> corroborate the theory:
> 
> [nm@power8-aix 8:0 2021-03-13T18:32:23 clean 0]$ perl -MIPC::Run -e 'IPC::Run::run "printf", "%s\n", "*"'
> *
> [nm@power8-aix 8:0 2021-03-13T18:32:29 clean 0]$ perl -MIPC::Run -e 'IPC::Run::run "sh", "-c", "printf %s\\\\n *"'
> COPYRIGHT
> GNUmakefile.in
> HISTORY
> Makefile
> README
> README.git
> aclocal.m4
> config
> configure
> configure.ac
> contrib
> doc
> src
> 
>> there is a similar symptom caused by an unrelated problem
> 
>> Subject: [PATCH v3] Avoid use of non-portable option ordering in
>> command_checks_all().
> 
> On a glibc system, "env POSIXLY_CORRECT=1 make check ..." tests this.

Thanks for investigating!

The reason I suspected that passing the '*' through IPC::Run had to do with the error that pg_amcheck gave.  It complained that too many arguments where being passed to it, and that the first such argument was "pg_amcheck.c".  There's no reason pg_amcheck should know it's source file name, nor that the regression test should know that, which suggested that the asterisk was being shell expanded within the src/bin/pg_amcheck/ directory and the file listing was being passed into pg_amcheck as arguments.

That theory may have been wrong, but it was the only theory I had at the time.  I don't have access to any of the machines where that happened, so it is hard for me to investigate further.

—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  66. Re: pg_amcheck contrib application

    Noah Misch <noah@leadboat.com> — 2021-03-13T19:18:41Z

    On Sat, Mar 13, 2021 at 10:51:27AM -0800, Mark Dilger wrote:
> > On Mar 13, 2021, at 10:46 AM, Noah Misch <noah@leadboat.com> wrote:
> > On Fri, Mar 12, 2021 at 05:04:09PM -0800, Mark Dilger wrote:
> >>> On Mar 12, 2021, at 3:24 PM, Mark Dilger <mark.dilger@enterprisedb.com> wrote:
> >>> and the second deals with an apparent problem with IPC::Run shell expanding an asterisk on some platforms but not others.  That second one, if true, seems like a problem with scope beyond the pg_amcheck project, as TestLib::command_checks_all uses IPC::Run, and it would be desirable to have consistent behavior across platforms.
> > 
> >>> One of pg_amcheck's regression tests was passing an asterisk through
> >>> TestLib's command_checks_all() command, which gets through to
> >>> pg_amcheck without difficulty on most platforms, but appears to get
> >>> shell expanded on Windows (jacana) and AIX (hoverfly).
> > 
> > For posterity, I can't reproduce this on hoverfly.  The suite fails the same
> > way at f371a4c and f371a4c^.  More-recently (commit 58f5749), the suite passes
> > even after reverting f371a4c.  Self-contained IPC::Run usage also does not
> > corroborate the theory:
> > 
> > [nm@power8-aix 8:0 2021-03-13T18:32:23 clean 0]$ perl -MIPC::Run -e 'IPC::Run::run "printf", "%s\n", "*"'
> > *
> > [nm@power8-aix 8:0 2021-03-13T18:32:29 clean 0]$ perl -MIPC::Run -e 'IPC::Run::run "sh", "-c", "printf %s\\\\n *"'
> > COPYRIGHT
> > GNUmakefile.in
> > HISTORY
> > Makefile
> > README
> > README.git
> > aclocal.m4
> > config
> > configure
> > configure.ac
> > contrib
> > doc
> > src

> The reason I suspected that passing the '*' through IPC::Run had to do with the error that pg_amcheck gave.  It complained that too many arguments where being passed to it, and that the first such argument was "pg_amcheck.c".  There's no reason pg_amcheck should know it's source file name, nor that the regression test should know that, which suggested that the asterisk was being shell expanded within the src/bin/pg_amcheck/ directory and the file listing was being passed into pg_amcheck as arguments.

I agree.  I can reproduce the problem on Windows.  Commit f371a4c fixed
http://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=fairywren&dt=2021-03-12%2020%3A12%3A44
and I see logs of that kind of failure only on fairywren and jacana.

  67. Re: pg_amcheck contrib application

    Tom Lane <tgl@sss.pgh.pa.us> — 2021-03-15T17:04:14Z

    Looks like we're not quite out of the woods, as hornet and tern are
still unhappy:

#   Failed test 'pg_amcheck excluding all corrupt schemas status (got 2 vs expected 0)'
#   at t/003_check.pl line 498.

#   Failed test 'pg_amcheck excluding all corrupt schemas stdout /(?^:^$)/'
#   at t/003_check.pl line 498.
#                   'heap table "db1"."pg_catalog"."pg_statistic", block 2, offset 1, attribute 27:
#     final toast chunk number 0 differs from expected value 1
# heap table "db1"."pg_catalog"."pg_statistic", block 2, offset 1, attribute 27:
#     toasted value for attribute 27 missing from toast table
# '
#     doesn't match '(?^:^$)'
# Looks like you failed 2 tests of 60.
[12:18:06] t/003_check.pl ........... 
Dubious, test returned 2 (wstat 512, 0x200)
Failed 2/60 subtests 

These animals have somewhat weird alignment properties: MAXALIGN is 8
but ALIGNOF_DOUBLE is only 4.  I speculate that that is affecting their
choices about whether an out-of-line TOAST value is needed, breaking
this test case.

			regards, tom lane

  68. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-03-15T18:11:17Z

    
> On Mar 15, 2021, at 10:04 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> 
> Looks like we're not quite out of the woods, as hornet and tern are
> still unhappy:
> 
> #   Failed test 'pg_amcheck excluding all corrupt schemas status (got 2 vs expected 0)'
> #   at t/003_check.pl line 498.
> 
> #   Failed test 'pg_amcheck excluding all corrupt schemas stdout /(?^:^$)/'
> #   at t/003_check.pl line 498.
> #                   'heap table "db1"."pg_catalog"."pg_statistic", block 2, offset 1, attribute 27:
> #     final toast chunk number 0 differs from expected value 1
> # heap table "db1"."pg_catalog"."pg_statistic", block 2, offset 1, attribute 27:
> #     toasted value for attribute 27 missing from toast table
> # '
> #     doesn't match '(?^:^$)'
> # Looks like you failed 2 tests of 60.
> [12:18:06] t/003_check.pl ........... 
> Dubious, test returned 2 (wstat 512, 0x200)
> Failed 2/60 subtests 
> 
> These animals have somewhat weird alignment properties: MAXALIGN is 8
> but ALIGNOF_DOUBLE is only 4.  I speculate that that is affecting their
> choices about whether an out-of-line TOAST value is needed, breaking
> this test case.

The pg_amcheck test case is not corrupting any pg_catalog tables, but contrib/amcheck/verify_heapam is complaining about a corruption in pg_catalog.pg_statistic.

The logic in verify_heapam only looks for a value in the toast table if the tuple it gets from the main table (in this case, from pg_statistic) has an attribute that claims to be toasted.  The error message we're seeing that you quoted above simply means that no entry exists in the toast table.  The bit about "final toast chunk number 0 differs from expected value 1" is super unhelpful, as what it is really saying is that there were no chunks found.  I should submit a patch to not print that message in cases where the attribute is missing from the toast table.

Is it possible that pg_statistic really is corrupt here, and that this is not a bug in pg_amcheck?  It's not like we've been checking for corruption in the build farm up till now.  I notice that this test, as well as test 005_opclass_damage.pl, neglects to turn off autovacuum for the test node.  So maybe the corruptions are getting propogated during autovacuum?  This is just a guess, but I will submit a patch that turns off autovacuum for the test node shortly.

—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  69. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-03-15T18:38:35Z

    
> On Mar 15, 2021, at 11:11 AM, Mark Dilger <mark.dilger@enterprisedb.com> wrote:
> 
> I will submit a patch that turns off autovacuum for the test node shortly.

  70. Re: pg_amcheck contrib application

    Tom Lane <tgl@sss.pgh.pa.us> — 2021-03-15T18:57:20Z

    Mark Dilger <mark.dilger@enterprisedb.com> writes:
> On Mar 15, 2021, at 10:04 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> These animals have somewhat weird alignment properties: MAXALIGN is 8
>> but ALIGNOF_DOUBLE is only 4.  I speculate that that is affecting their
>> choices about whether an out-of-line TOAST value is needed, breaking
>> this test case.

> The logic in verify_heapam only looks for a value in the toast table if
> the tuple it gets from the main table (in this case, from pg_statistic)
> has an attribute that claims to be toasted.  The error message we're
> seeing that you quoted above simply means that no entry exists in the
> toast table.

Yeah, that could be phrased better.  Do we have a strong enough lock on
the table under examination to be sure that autovacuum couldn't remove
a dead toast entry before we reach it?  But this would only be an
issue if we are trying to check validity of toasted fields within
known-dead tuples, which I would argue we shouldn't, since lock or
no lock there's no guarantee the toast entry is still there.

Not sure that I believe the theory that this is from bad luck of
concurrent autovacuum timing, though.  The fact that we're seeing
this on just those two animals suggests strongly to me that it's
architecture-correlated, instead.

			regards, tom lane

  71. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-03-15T19:20:01Z

    
> On Mar 15, 2021, at 11:57 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> 
> Do we have a strong enough lock on
> the table under examination to be sure that autovacuum couldn't remove
> a dead toast entry before we reach it?

The main table and the toast table are only locked with AccessShareLock.  Each page in the main table is locked with BUFFER_LOCK_SHARE.  Toast is not checked unless the tuple passes visibility checks verifying the tuple is not dead.

>  But this would only be an
> issue if we are trying to check validity of toasted fields within
> known-dead tuples, which I would argue we shouldn't, since lock or
> no lock there's no guarantee the toast entry is still there.

It does not intentionally check toasted fields within dead tuples.  If that is happening, it's a bug, possibly in the visibility function.  But I'm not seeing a specific reason to assume that is the issue.  If we still see the complaint on tern or hornet after committing the patch to turn off autovacuum, we will be able to rule out the theory that the toast was removed by autovacuum.

—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  72. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-03-15T19:35:52Z

    
> On Mar 15, 2021, at 11:57 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> 
> Yeah, that could be phrased better.

Attaching the 0001 patch submitted earlier, plus 0002 which fixes the confusing corruption message.

  73. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-03-16T02:10:37Z

    
> On Mar 15, 2021, at 11:57 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> 
> Not sure that I believe the theory that this is from bad luck of
> concurrent autovacuum timing, though.  The fact that we're seeing
> this on just those two animals suggests strongly to me that it's
> architecture-correlated, instead.

I find it a little hard to see how amcheck is tripping over a toasted value just in this one table, pg_statistic, and not in any of the others.  The error message says the problem is in attribute 27, which I believe means it is stavalues2.  The comment in the header for this catalog is intriguing:

    /*
     * Values in these arrays are values of the column's data type, or of some
     * related type such as an array element type.  We presently have to cheat
     * quite a bit to allow polymorphic arrays of this kind, but perhaps
     * someday it'll be a less bogus facility.
     */
    anyarray    stavalues1;
    anyarray    stavalues2;
    anyarray    stavalues3;
    anyarray    stavalues4;
    anyarray    stavalues5;

This is hard to duplicate in a test, because you can't normally create tables with pseudo-type columns.  However, if amcheck is walking the tuple and does not correctly update the offset with the length of attribute 26, it may try to read attribute 27 at the wrong offset, unsurprisingly leading to garbage, perhaps a garbage toast pointer.  The attached patch v7-0004 adds a check to verify_heapam to see if the va_toastrelid matches the expected toast table oid for the table we're reading.  That check almost certainly should have been included in the initial version of verify_heapam, so even if it does nothing to help us in this issue, it's good that it be committed, I think.

It is unfortunate that the failing test only runs pg_amcheck after creating numerous corruptions, as we can't know if pg_amcheck would have complained about pg_statistic before the corruptions were created in other tables, or if it only does so after.  The attached patch v7-0003 adds a call to pg_amcheck after all tables are created and populated, but before any corruptions are caused.  This should help narrow down what is happening, and doesn't hurt to leave in place long-term.

I don't immediately see anything wrong with how pg_statistic uses a pseudo-type, but it leads me to want to poke a bit more at pg_statistic on hornet and tern, though I don't have any regression tests specifically for doing so.

Tests v7-0001 and v7-0002 are just repeats of the tests posted previously.

  74. Re: pg_amcheck contrib application

    Noah Misch <noah@leadboat.com> — 2021-03-16T06:09:13Z

    On Mon, Mar 15, 2021 at 02:57:20PM -0400, Tom Lane wrote:
> Mark Dilger <mark.dilger@enterprisedb.com> writes:
> > On Mar 15, 2021, at 10:04 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> >> These animals have somewhat weird alignment properties: MAXALIGN is 8
> >> but ALIGNOF_DOUBLE is only 4.  I speculate that that is affecting their
> >> choices about whether an out-of-line TOAST value is needed, breaking
> >> this test case.

That machine also has awful performance for filesystem metadata operations,
like open(O_CREAT).  Its CPU and read()/write() performance are normal.

> > The logic in verify_heapam only looks for a value in the toast table if
> > the tuple it gets from the main table (in this case, from pg_statistic)
> > has an attribute that claims to be toasted.  The error message we're
> > seeing that you quoted above simply means that no entry exists in the
> > toast table.
> 
> Yeah, that could be phrased better.  Do we have a strong enough lock on
> the table under examination to be sure that autovacuum couldn't remove
> a dead toast entry before we reach it?  But this would only be an
> issue if we are trying to check validity of toasted fields within
> known-dead tuples, which I would argue we shouldn't, since lock or
> no lock there's no guarantee the toast entry is still there.
> 
> Not sure that I believe the theory that this is from bad luck of
> concurrent autovacuum timing, though.

With autovacuum_naptime=1s, on hornet, the failure reproduced twice in twelve
runs.  With v6-0001-Turning-off-autovacuum-during-corruption-tests.patch
applied, 196 runs all succeeded.

> The fact that we're seeing
> this on just those two animals suggests strongly to me that it's
> architecture-correlated, instead.

That is possible.

  75. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-03-16T15:45:28Z

    
> On Mar 15, 2021, at 11:09 PM, Noah Misch <noah@leadboat.com> wrote:
> 
>> Not sure that I believe the theory that this is from bad luck of
>> concurrent autovacuum timing, though.
> 
> With autovacuum_naptime=1s, on hornet, the failure reproduced twice in twelve
> runs.  With v6-0001-Turning-off-autovacuum-during-corruption-tests.patch
> applied, 196 runs all succeeded.
> 
>> The fact that we're seeing
>> this on just those two animals suggests strongly to me that it's
>> architecture-correlated, instead.
> 
> That is possible.

I think autovacuum simply triggers the bug, and is not the cause of the bug.  If I turn autovacuum off and instead do an ANALYZE in each test database rather than performing the corruptions, I get reports about problems in pg_statistic.  This is on my mac laptop.  This rules out the theory that autovacuum is propogating corruptions into pg_statistic, and also the theory that it is architecture dependent.

I'll investigate further.

—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  76. Re: pg_amcheck contrib application

    Tom Lane <tgl@sss.pgh.pa.us> — 2021-03-16T16:07:35Z

    Mark Dilger <mark.dilger@enterprisedb.com> writes:
> I think autovacuum simply triggers the bug, and is not the cause of the bug.  If I turn autovacuum off and instead do an ANALYZE in each test database rather than performing the corruptions, I get reports about problems in pg_statistic.  This is on my mac laptop.  This rules out the theory that autovacuum is propogating corruptions into pg_statistic, and also the theory that it is architecture dependent.

I wonder whether amcheck is confused by the declaration of those columns
as "anyarray".

			regards, tom lane

  77. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-03-16T16:30:00Z

    
> On Mar 16, 2021, at 9:07 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> 
> Mark Dilger <mark.dilger@enterprisedb.com> writes:
>> I think autovacuum simply triggers the bug, and is not the cause of the bug.  If I turn autovacuum off and instead do an ANALYZE in each test database rather than performing the corruptions, I get reports about problems in pg_statistic.  This is on my mac laptop.  This rules out the theory that autovacuum is propogating corruptions into pg_statistic, and also the theory that it is architecture dependent.
> 
> I wonder whether amcheck is confused by the declaration of those columns
> as "anyarray".

It uses attlen and attalign for the attribute, so that idea does make sense.  It gets that via TupleDescAttr(RelationGetDescr(rel), attnum).

—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  78. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-03-16T16:51:04Z

    
> On Mar 16, 2021, at 9:30 AM, Mark Dilger <mark.dilger@enterprisedb.com> wrote:
> 
> 
> 
>> On Mar 16, 2021, at 9:07 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> 
>> Mark Dilger <mark.dilger@enterprisedb.com> writes:
>>> I think autovacuum simply triggers the bug, and is not the cause of the bug.  If I turn autovacuum off and instead do an ANALYZE in each test database rather than performing the corruptions, I get reports about problems in pg_statistic.  This is on my mac laptop.  This rules out the theory that autovacuum is propogating corruptions into pg_statistic, and also the theory that it is architecture dependent.
>> 
>> I wonder whether amcheck is confused by the declaration of those columns
>> as "anyarray".
> 
> It uses attlen and attalign for the attribute, so that idea does make sense.  It gets that via TupleDescAttr(RelationGetDescr(rel), attnum).

Yeah, that looks related:

regression=# select attname, attlen, attnum, attalign from pg_attribute where attrelid = 2619 and attname like 'stavalue%';
  attname   | attlen | attnum | attalign 
------------+--------+--------+----------
 stavalues1 |     -1 |     27 | d
 stavalues2 |     -1 |     28 | d
 stavalues3 |     -1 |     29 | d
 stavalues4 |     -1 |     30 | d
 stavalues5 |     -1 |     31 | d
(5 rows)

It shows them all has having attalign = 'd', but for some array types the alignment will be 'i', not 'd'.  So it's lying a bit about the contents.  But I'm now confused why this caused problems on the two hosts where integer and double have the same alignment?  It seems like that would be the one place where the bug would not happen, not the one place where it does.

I'm attaching a test that reliably reproduces this for me:

  79. Re: pg_amcheck contrib application

    Robert Haas <robertmhaas@gmail.com> — 2021-03-16T16:56:40Z

    On Tue, Mar 16, 2021 at 12:51 PM Mark Dilger
<mark.dilger@enterprisedb.com> wrote:
> Yeah, that looks related:
>
> regression=# select attname, attlen, attnum, attalign from pg_attribute where attrelid = 2619 and attname like 'stavalue%';
>   attname   | attlen | attnum | attalign
> ------------+--------+--------+----------
>  stavalues1 |     -1 |     27 | d
>  stavalues2 |     -1 |     28 | d
>  stavalues3 |     -1 |     29 | d
>  stavalues4 |     -1 |     30 | d
>  stavalues5 |     -1 |     31 | d
> (5 rows)
>
> It shows them all has having attalign = 'd', but for some array types the alignment will be 'i', not 'd'.  So it's lying a bit about the contents.  But I'm now confused why this caused problems on the two hosts where integer and double have the same alignment?  It seems like that would be the one place where the bug would not happen, not the one place where it does.

Wait, so the value in the attalign column isn't the alignment we're
actually using? I can understand how we might generate tuples like
that, if we pass the actual type to construct_array(), but shouldn't
we then get garbage when we deform the tuple? I mean,
heap_deform_tuple() is going to get the alignment from the tuple
descriptor, which is a table property. It doesn't (and can't) know
what type of value is stored inside any of these fields for real,
right?

In other words, isn't this actually corruption, caused by a bug in our
code, and how have we not noticed it before now?

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  80. Re: pg_amcheck contrib application

    Tom Lane <tgl@sss.pgh.pa.us> — 2021-03-16T17:48:37Z

    Robert Haas <robertmhaas@gmail.com> writes:
> On Tue, Mar 16, 2021 at 12:51 PM Mark Dilger
> <mark.dilger@enterprisedb.com> wrote:
>> It shows them all has having attalign = 'd', but for some array types the alignment will be 'i', not 'd'.  So it's lying a bit about the contents.  But I'm now confused why this caused problems on the two hosts where integer and double have the same alignment?  It seems like that would be the one place where the bug would not happen, not the one place where it does.

> Wait, so the value in the attalign column isn't the alignment we're
> actually using? I can understand how we might generate tuples like
> that, if we pass the actual type to construct_array(), but shouldn't
> we then get garbage when we deform the tuple?

No.  What should be happening there is that some arrays in the column
get larger alignment than they actually need, but that shouldn't cause
a problem (and has not done so, AFAIK, in the decades that it's been
like this).  As you say, deforming the tuple is going to rely on the
table's tupdesc for alignment; it can't know what is in the data.

I'm not entirely sure what's going on, but I think coming at this
with the mindset that "amcheck has detected some corruption" is
just going to lead you astray.  Almost certainly, it's "amcheck
is incorrectly claiming corruption".  That might be from mis-decoding
a TOAST-referencing datum.  (Too bad the message doesn't report the
TOAST OID it probed for, so we can see if that's sane or not.)

			regards, tom lane

  81. Re: pg_amcheck contrib application

    Tom Lane <tgl@sss.pgh.pa.us> — 2021-03-16T18:01:47Z

    ... btw, I now see that tern and hornet are passing this test
at least as much as they're failing, proving that there's some
timing or random chance involved.  That doesn't completely
eliminate the idea that there may be an architecture component
to the issue, but it sure reduces its credibility.  I now
believe the theory that the triggering condition is an auto-analyze
happening at the right time, and populating pg_statistic with
some data that other runs don't see.

			regards, tom lane

  82. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-03-16T18:07:16Z

    
> On Mar 16, 2021, at 10:48 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> 
> Robert Haas <robertmhaas@gmail.com> writes:
>> On Tue, Mar 16, 2021 at 12:51 PM Mark Dilger
>> <mark.dilger@enterprisedb.com> wrote:
>>> It shows them all has having attalign = 'd', but for some array types the alignment will be 'i', not 'd'.  So it's lying a bit about the contents.  But I'm now confused why this caused problems on the two hosts where integer and double have the same alignment?  It seems like that would be the one place where the bug would not happen, not the one place where it does.
> 
>> Wait, so the value in the attalign column isn't the alignment we're
>> actually using? I can understand how we might generate tuples like
>> that, if we pass the actual type to construct_array(), but shouldn't
>> we then get garbage when we deform the tuple?
> 
> No.  What should be happening there is that some arrays in the column
> get larger alignment than they actually need, but that shouldn't cause
> a problem (and has not done so, AFAIK, in the decades that it's been
> like this).  As you say, deforming the tuple is going to rely on the
> table's tupdesc for alignment; it can't know what is in the data.
> 
> I'm not entirely sure what's going on, but I think coming at this
> with the mindset that "amcheck has detected some corruption" is
> just going to lead you astray.  Almost certainly, it's "amcheck
> is incorrectly claiming corruption".  That might be from mis-decoding
> a TOAST-referencing datum.  (Too bad the message doesn't report the
> TOAST OID it probed for, so we can see if that's sane or not.)

I've added that and now get the toast pointer's va_valueid in the message:

mark.dilger@laptop280-ma-us amcheck % pg_amcheck -t "pg_catalog.pg_statistic" postgres
heap table "postgres"."pg_catalog"."pg_statistic", block 4, offset 2, attribute 29:
    toasted value id 13227 for attribute 29 missing from toast table
heap table "postgres"."pg_catalog"."pg_statistic", block 4, offset 5, attribute 27:
    toasted value id 13228 for attribute 27 missing from toast table

diff --git a/contrib/amcheck/verify_heapam.c b/contrib/amcheck/verify_heapam.c
index 5ccae46375..a0be71bb7f 100644
--- a/contrib/amcheck/verify_heapam.c
+++ b/contrib/amcheck/verify_heapam.c
@@ -1111,8 +1111,8 @@ check_tuple_attribute(HeapCheckContext *ctx)
        }
        if (!found_toasttup)
                report_corruption(ctx,
-                                                 psprintf("toasted value for attribute %u missing from toast table",
-                                                                  ctx->attnum));
+                                                 psprintf("toasted value id %u for attribute %u missing from toast table",
+                                                                  toast_pointer.va_valueid, ctx->attnum));
        else if (ctx->chunkno != (ctx->endchunk + 1))
                report_corruption(ctx,
                                                  psprintf("final toast chunk number %u differs from expected value %u",

—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  83. Re: pg_amcheck contrib application

    Tom Lane <tgl@sss.pgh.pa.us> — 2021-03-16T18:22:34Z

    Mark Dilger <mark.dilger@enterprisedb.com> writes:
> On Mar 16, 2021, at 10:48 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> (Too bad the message doesn't report the
>> TOAST OID it probed for, so we can see if that's sane or not.)

> I've added that and now get the toast pointer's va_valueid in the message:

> heap table "postgres"."pg_catalog"."pg_statistic", block 4, offset 2, attribute 29:
>     toasted value id 13227 for attribute 29 missing from toast table
> heap table "postgres"."pg_catalog"."pg_statistic", block 4, offset 5, attribute 27:
>     toasted value id 13228 for attribute 27 missing from toast table

That's awfully interesting, because OIDs less than 16384 should
only be generated during initdb.  So what we seem to be looking at
here is pg_statistic entries that were made during the ANALYZE
done by initdb (cf. vacuum_db()), and then were deleted during
a later auto-analyze (in the buildfarm) or deliberate analyze
(in your reproducer).  But if they're deleted, why is amcheck
looking for them?

I'm circling back around to the idea that amcheck is trying to
validate TOAST references that are already dead, and it's getting
burnt because something-or-other has already removed the toast
rows, though not the referencing datums.  That's legal behavior
once the rows are marked dead.  Upthread it was claimed that
amcheck isn't doing that, but this looks like a smoking gun to me.

			regards, tom lane

  84. Re: pg_amcheck contrib application

    Robert Haas <robertmhaas@gmail.com> — 2021-03-16T18:24:23Z

    On Tue, Mar 16, 2021 at 1:48 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> No.  What should be happening there is that some arrays in the column
> get larger alignment than they actually need, but that shouldn't cause
> a problem (and has not done so, AFAIK, in the decades that it's been
> like this).  As you say, deforming the tuple is going to rely on the
> table's tupdesc for alignment; it can't know what is in the data.

OK, I don't understand this. attalign is 'd', which is already the
maximum possible, and even if it weren't, individual rows can't decide
to use a larger OR smaller alignment than expected without breaking
stuff.

In what context is it OK to just add extra alignment padding?

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  85. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-03-16T18:24:33Z

    
> On Mar 16, 2021, at 10:48 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> 
> I'm not entirely sure what's going on, but I think coming at this
> with the mindset that "amcheck has detected some corruption" is
> just going to lead you astray.  Almost certainly, it's "amcheck
> is incorrectly claiming corruption".  That might be from mis-decoding
> a TOAST-referencing datum.  (Too bad the message doesn't report the
> TOAST OID it probed for, so we can see if that's sane or not.)

CopyStatistics seems to just copy Form_pg_statistic without regard for who owns the toast.  Is this safe?  Looking at RemoveStatistics, I'm not sure that it is.  Anybody more familiar with this code care to give an opinion?

—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  86. Re: pg_amcheck contrib application

    Robert Haas <robertmhaas@gmail.com> — 2021-03-16T18:40:20Z

    On Tue, Mar 16, 2021 at 2:22 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> I'm circling back around to the idea that amcheck is trying to
> validate TOAST references that are already dead, and it's getting
> burnt because something-or-other has already removed the toast
> rows, though not the referencing datums.  That's legal behavior
> once the rows are marked dead.  Upthread it was claimed that
> amcheck isn't doing that, but this looks like a smoking gun to me.

I think this theory has some legs. From check_tuple_header_and_visibilty():

                else if (!(infomask & HEAP_XMAX_COMMITTED))
                        return true;            /*
HEAPTUPLE_DELETE_IN_PROGRESS or
                                                                 *
HEAPTUPLE_LIVE */
                else
                        return false;           /*
HEAPTUPLE_RECENTLY_DEAD or HEAPTUPLE_DEAD */
        }
        return true;                            /* not dead */
}

That first case looks wrong to me. Don't we need to call
get_xid_status() here, Mark? As coded, it seems that if the xmin is ok
and the xmax is marked committed, we consider the tuple checkable. The
comment says it must be HEAPTUPLE_DELETE_IN_PROGRESS or
HEAPTUPLE_LIVE, but it seems to me that if the actual answer is either
HEAPTUPLE_RECENTLY_DEAD or HEAPTUPLE_DEAD depending on whether the
xmax is all-visible. And in the second case the comment says it's
either HEAPTUPLE_RECENTLY_DEAD or HEAPTUPLE_DEAD, but I think in that
case it's either HEAPTUPLE_DELETE_IN_PROGRESS or
HEAPTUPLE_RECENTLY_DEAD or HEAPTUPLE_DEAD, depending on the XID
status.

Another thought here is that it's probably not wicked smart to be
relying on the hint bits to match the actual status of the tuple in
cases of corruption. Maybe we should be warning about tuples that are
have xmin or xmax flagged as committed or invalid when in fact clog
disagrees. That's not a particularly uncommon case, and it's hard to
check.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  87. Re: pg_amcheck contrib application

    Tom Lane <tgl@sss.pgh.pa.us> — 2021-03-16T18:45:13Z

    Robert Haas <robertmhaas@gmail.com> writes:
> On Tue, Mar 16, 2021 at 1:48 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> No.  What should be happening there is that some arrays in the column
>> get larger alignment than they actually need, but that shouldn't cause
>> a problem (and has not done so, AFAIK, in the decades that it's been
>> like this).  As you say, deforming the tuple is going to rely on the
>> table's tupdesc for alignment; it can't know what is in the data.

> OK, I don't understand this. attalign is 'd', which is already the
> maximum possible, and even if it weren't, individual rows can't decide
> to use a larger OR smaller alignment than expected without breaking
> stuff.

> In what context is it OK to just add extra alignment padding?

It's *not* extra, according to pg_statistic's tuple descriptor.
Both forming and deforming of pg_statistic tuples should honor
that and locate stavaluesX values on d-aligned boundaries.

It could be that a particular entry is of an array type that
only requires i-alignment.  But that doesn't break anything,
it just means we inserted more padding than an omniscient
implementation would do.

(I suppose in some parallel universe there could be a machine
where i-alignment is stricter than d-alignment, and then we'd
have trouble.)

			regards, tom lane

  88. Re: pg_amcheck contrib application

    Tom Lane <tgl@sss.pgh.pa.us> — 2021-03-16T18:52:22Z

    Mark Dilger <mark.dilger@enterprisedb.com> writes:
> CopyStatistics seems to just copy Form_pg_statistic without regard for
> who owns the toast.  Is this safe?

No less so than a ton of other places that insert values that might've
come from on-disk storage.  heap_toast_insert_or_update() is responsible
for dealing with the problem.  These days it looks like it's actually
toast_tuple_init() that takes care of dereferencing previously-toasted
values during an INSERT.

			regards, tom lane

  89. Re: pg_amcheck contrib application

    Robert Haas <robertmhaas@gmail.com> — 2021-03-16T19:09:48Z

    On Tue, Mar 16, 2021 at 2:45 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> > In what context is it OK to just add extra alignment padding?
>
> It's *not* extra, according to pg_statistic's tuple descriptor.
> Both forming and deforming of pg_statistic tuples should honor
> that and locate stavaluesX values on d-aligned boundaries.
>
> It could be that a particular entry is of an array type that
> only requires i-alignment.  But that doesn't break anything,
> it just means we inserted more padding than an omniscient
> implementation would do.

OK, yeah, I just misunderstood what you were saying.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  90. Re: pg_amcheck contrib application

    Andrew Dunstan <andrew@dunslane.net> — 2021-03-16T19:21:39Z

    On 3/13/21 1:30 AM, Andres Freund wrote:
> Hi,
>
> On 2021-03-13 01:22:54 -0500, Tom Lane wrote:
>> Mark Dilger <mark.dilger@enterprisedb.com> writes:
>>> On Mar 12, 2021, at 10:16 PM, Noah Misch <noah@leadboat.com> wrote:
>>>> hoverfly does configure with PERL=perl64.  /usr/bin/prove is from the 32-bit
>>>> Perl, so I suspect the TAP suites get 32-bit Perl that way.  (There's no
>>>> "prove64".)
>> Oh, that's annoying.
> I suspect we could solve that by invoking changing our /usr/bin/prove
> invocation to instead be PERL /usr/bin/prove? That might be a good thing
> independent of this issue, because it's not unreasonable for a user to
> expect that we'd actually use the perl installation they configured...
>
> Although I do not know how prove determines the perl installation it's
> going to use for the test scripts...
>


There's a very good chance this would break msys builds, which are
configured to build against a pure native (i.e. non-msys) perl sucj as
AS or Strawberry, but need to run msys perl for TAP tests, so it gets
the paths right.

(Don't get me started on the madness that can result from managing this.
I've lost weeks of my life to it ... If you add cygwin into the mix and
you're trying to coordinate builds among three buildfarm animals it's a
major pain.)


cheers


andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

  91. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-03-16T19:31:42Z

    
> On Mar 16, 2021, at 11:40 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> 
> On Tue, Mar 16, 2021 at 2:22 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> I'm circling back around to the idea that amcheck is trying to
>> validate TOAST references that are already dead, and it's getting
>> burnt because something-or-other has already removed the toast
>> rows, though not the referencing datums.  That's legal behavior
>> once the rows are marked dead.  Upthread it was claimed that
>> amcheck isn't doing that, but this looks like a smoking gun to me.
> 
> I think this theory has some legs. From check_tuple_header_and_visibilty():
> 
>                else if (!(infomask & HEAP_XMAX_COMMITTED))
>                        return true;            /*
> HEAPTUPLE_DELETE_IN_PROGRESS or
>                                                                 *
> HEAPTUPLE_LIVE */
>                else
>                        return false;           /*
> HEAPTUPLE_RECENTLY_DEAD or HEAPTUPLE_DEAD */
>        }
>        return true;                            /* not dead */
> }
> 
> That first case looks wrong to me. Don't we need to call
> get_xid_status() here, Mark? As coded, it seems that if the xmin is ok
> and the xmax is marked committed, we consider the tuple checkable. The
> comment says it must be HEAPTUPLE_DELETE_IN_PROGRESS or
> HEAPTUPLE_LIVE, but it seems to me that if the actual answer is either
> HEAPTUPLE_RECENTLY_DEAD or HEAPTUPLE_DEAD depending on whether the
> xmax is all-visible. And in the second case the comment says it's
> either HEAPTUPLE_RECENTLY_DEAD or HEAPTUPLE_DEAD, but I think in that
> case it's either HEAPTUPLE_DELETE_IN_PROGRESS or
> HEAPTUPLE_RECENTLY_DEAD or HEAPTUPLE_DEAD, depending on the XID
> status.
> 
> Another thought here is that it's probably not wicked smart to be
> relying on the hint bits to match the actual status of the tuple in
> cases of corruption. Maybe we should be warning about tuples that are
> have xmin or xmax flagged as committed or invalid when in fact clog
> disagrees. That's not a particularly uncommon case, and it's hard to
> check.

This code was not committed as part of the recent pg_amcheck work, but longer ago, and I'm having trouble reconstructing exactly why it was written that way.

Changing check_tuple_header_and_visibilty() fixes the regression test and also manual tests against the "regression" database that I've been using.  I'd like to ponder the changes a while longer before I post, but the fact that these changes fix the tests seems to add credibility to this theory.


—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  92. Re: pg_amcheck contrib application

    Robert Haas <robertmhaas@gmail.com> — 2021-03-16T19:52:09Z

    On Mon, Mar 15, 2021 at 10:10 PM Mark Dilger
<mark.dilger@enterprisedb.com> wrote:
> It is unfortunate that the failing test only runs pg_amcheck after creating numerous corruptions, as we can't know if pg_amcheck would have complained about pg_statistic before the corruptions were created in other tables, or if it only does so after.  The attached patch v7-0003 adds a call to pg_amcheck after all tables are created and populated, but before any corruptions are caused.  This should help narrow down what is happening, and doesn't hurt to leave in place long-term.
>
> I don't immediately see anything wrong with how pg_statistic uses a pseudo-type, but it leads me to want to poke a bit more at pg_statistic on hornet and tern, though I don't have any regression tests specifically for doing so.
>
> Tests v7-0001 and v7-0002 are just repeats of the tests posted previously.

Since we now know that shutting autovacuum off makes the problem go
away, I don't see a reason to commit 0001. We should fix pg_amcheck
instead, if, as presently seems to be the case, that's where the
problem is.

I just committed 0002.

I think 0003 is probably a good idea, but I haven't committed it yet.

As for 0004, it seems to me that we might want to do a little more
rewording of these messages and perhaps we should try to do it all at
once. Like, for example, your other change to print out the toast
value ID seems like a good idea, and could apply to any new messages
as well as some existing ones. Maybe there are also more fields in the
TOAST pointer for which we could add checks.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  93. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-03-18T04:00:39Z

    
> On Mar 16, 2021, at 12:52 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> 
> On Mon, Mar 15, 2021 at 10:10 PM Mark Dilger
> <mark.dilger@enterprisedb.com> wrote:
>> It is unfortunate that the failing test only runs pg_amcheck after creating numerous corruptions, as we can't know if pg_amcheck would have complained about pg_statistic before the corruptions were created in other tables, or if it only does so after.  The attached patch v7-0003 adds a call to pg_amcheck after all tables are created and populated, but before any corruptions are caused.  This should help narrow down what is happening, and doesn't hurt to leave in place long-term.
>> 
>> I don't immediately see anything wrong with how pg_statistic uses a pseudo-type, but it leads me to want to poke a bit more at pg_statistic on hornet and tern, though I don't have any regression tests specifically for doing so.
>> 
>> Tests v7-0001 and v7-0002 are just repeats of the tests posted previously.
> 
> Since we now know that shutting autovacuum off makes the problem go
> away, I don't see a reason to commit 0001. We should fix pg_amcheck
> instead, if, as presently seems to be the case, that's where the
> problem is.

If you get unlucky, autovacuum will process one of the tables that the test intentionally corrupted, with bad consequences, ultimately causing build farm intermittent test failures.  We could wait to see if this ever happens before fixing it, if you prefer.  I'm not sure what that buys us, though.

The right approach, I think, is to extend the contrib/amcheck tests to include regressing this particular case to see if it fails.  I've written that test and verified that it fails against the old code and passes against the new.

> I just committed 0002.

Thanks!

> I think 0003 is probably a good idea, but I haven't committed it yet.

It won't do anything for us in this particular case, but it might make debugging failures quicker in the future.

> As for 0004, it seems to me that we might want to do a little more
> rewording of these messages and perhaps we should try to do it all at
> once. Like, for example, your other change to print out the toast
> value ID seems like a good idea, and could apply to any new messages
> as well as some existing ones. Maybe there are also more fields in the
> TOAST pointer for which we could add checks.

Of the toast pointer fields:

    int32       va_rawsize;     /* Original data size (includes header) */
    int32       va_extsize;     /* External saved size (doesn't) */
    Oid         va_valueid;     /* Unique ID of value within TOAST table */
    Oid         va_toastrelid;  /* RelID of TOAST table containing it */

all seem worth getting as part of any toast error message, even if these fields themselves are not corrupt.  It just makes it easier to understand the context of the error you're looking at.  At first I tried putting these into each message, but it is very wordy to say things like "toast pointer with rawsize %u and extsize %u pointing at relation with oid %u" and such.  It made more sense to just add these four fields to the verify_heapam tuple format.  That saves putting them in the message text itself, and has the benefit that you could filter the rows coming from verify_heapam() for ones where valueid is or is not null, for example.  This changes the external interface of verify_heapam, but I didn't bother with a amcheck--1.3--1.4.sql because amcheck--1.2--1.3. sql was added as part of the v14 development work and has not yet been released.  My assumption is that I can just change it, rather than making a new upgrade file.

These patches fix the visibility rules and add extra toast checking.  Some of the previous patch material is not included, since it is not clear to me if you wanted to commit any of it.

  94. Re: pg_amcheck contrib application

    Tom Lane <tgl@sss.pgh.pa.us> — 2021-03-18T04:12:27Z

    Mark Dilger <mark.dilger@enterprisedb.com> writes:
>> On Mar 16, 2021, at 12:52 PM, Robert Haas <robertmhaas@gmail.com> wrote:
>> Since we now know that shutting autovacuum off makes the problem go
>> away, I don't see a reason to commit 0001. We should fix pg_amcheck
>> instead, if, as presently seems to be the case, that's where the
>> problem is.

> If you get unlucky, autovacuum will process one of the tables that the test intentionally corrupted, with bad consequences, ultimately causing build farm intermittent test failures.

Um, yeah, the test had better shut off autovacuum on any table that
it intentionally corrupts.

			regards, tom lane

  95. Re: pg_amcheck contrib application

    Robert Haas <robertmhaas@gmail.com> — 2021-03-23T19:05:47Z

    On Thu, Mar 18, 2021 at 12:12 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Mark Dilger <mark.dilger@enterprisedb.com> writes:
> >> On Mar 16, 2021, at 12:52 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> >> Since we now know that shutting autovacuum off makes the problem go
> >> away, I don't see a reason to commit 0001. We should fix pg_amcheck
> >> instead, if, as presently seems to be the case, that's where the
> >> problem is.
>
> > If you get unlucky, autovacuum will process one of the tables that the test intentionally corrupted, with bad consequences, ultimately causing build farm intermittent test failures.
>
> Um, yeah, the test had better shut off autovacuum on any table that
> it intentionally corrupts.

Right, good point. But... does that really apply to
005_opclass_damage.pl? I feel like with the kind of physical damage
we're doing in 003_check.pl, it makes a lot of sense to stop vacuum
from crashing headlong into that table. But, 005 is doing "logical"
damage rather than "physical" damage, and I don't see why autovacuum
should misbehave in that kind of case. In fact, the fact that
autovacuum can handle such cases is one of the selling points for the
whole design of vacuum, as opposed to, for example, retail index
lookups.

Pending resolution of that question, I've committed the change to
disable autovacuum in 003, and also Mark's changes to have it also run
pg_amcheck BEFORE corrupting anything, so the post-corruption tests
fail - say by finding the wrong kind of corruption - we can see
whether it was also failing before the corruption was even introduced.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  96. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-03-23T19:20:35Z

    
> On Mar 23, 2021, at 12:05 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> 
> 005 is doing "logical"
> damage rather than "physical" damage, and I don't see why autovacuum
> should misbehave in that kind of case. In fact, the fact that
> autovacuum can handle such cases is one of the selling points for the
> whole design of vacuum, as opposed to, for example, retail index
> lookups.

That is a good point.  Checking that autovacuum behaves sensibly despite sort order breakage sounds reasonable, but test 005 doesn't do that reliably, because it does nothing to make sure that autovacuum runs against the affected table during the short window when the affected table exists.  All the same, I don't see that turning autovacuum off is required.  If autovacuum is broken in this regard, we may get occasional, hard to reproduce build farm failures, but that would be more informative than no failures at all.

—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  97. Re: pg_amcheck contrib application

    Peter Geoghegan <pg@bowt.ie> — 2021-03-23T19:41:54Z

    On Tue, Mar 23, 2021 at 12:05 PM Robert Haas <robertmhaas@gmail.com> wrote:
> Right, good point. But... does that really apply to
> 005_opclass_damage.pl? I feel like with the kind of physical damage
> we're doing in 003_check.pl, it makes a lot of sense to stop vacuum
> from crashing headlong into that table. But, 005 is doing "logical"
> damage rather than "physical" damage, and I don't see why autovacuum
> should misbehave in that kind of case. In fact, the fact that
> autovacuum can handle such cases is one of the selling points for the
> whole design of vacuum, as opposed to, for example, retail index
> lookups.

FWIW that is only 99.9% true (contrary to what README.HOT says). This
is the case because nbtree page deletion will in fact search the tree
to find a downlink to the target page, which must be removed at the
same time -- see the call to _bt_search() made within nbtpage.c.

This is much less of a problem than you'd think, though, even with an
opclass that gives wrong answers all the time. Because it's also true
that _bt_getstackbuf() is remarkably tolerant when it actually goes to
locate the downlink -- because that happens via a linear search that
matches on downlink block number (it doesn't use the opclass for that
part). This means that we'll accidentally fail to fail if the page is
*somewhere* to the right in the "true" key space. Which probably means
that it has a greater than 50% chance of not failing with a 100%
broken opclass. Which probably makes our odds better with more
plausible levels of misbehavior (e.g. collation changes).

That being said, I should make _bt_lock_subtree_parent() return false
and back out of page deletion without raising an error in the case
where we really cannot locate a valid downlink. We really ought to
soldier on when that happens, since we'll do that for a bunch of other
reasons already. I believe that the only reason we throw an error
today is for parity with the page split case (the main
_bt_getstackbuf() call). But this isn't the same situation at all --
this is VACUUM.

I will make this change to HEAD soon, barring objections.

-- 
Peter Geoghegan

  98. Re: pg_amcheck contrib application

    Tom Lane <tgl@sss.pgh.pa.us> — 2021-03-23T19:44:48Z

    Peter Geoghegan <pg@bowt.ie> writes:
> That being said, I should make _bt_lock_subtree_parent() return false
> and back out of page deletion without raising an error in the case
> where we really cannot locate a valid downlink. We really ought to
> soldier on when that happens, since we'll do that for a bunch of other
> reasons already. I believe that the only reason we throw an error
> today is for parity with the page split case (the main
> _bt_getstackbuf() call). But this isn't the same situation at all --
> this is VACUUM.

> I will make this change to HEAD soon, barring objections.

+1.  Not deleting the upper page seems better than the alternatives.

			regards, tom lane

  99. Re: pg_amcheck contrib application

    Peter Geoghegan <pg@bowt.ie> — 2021-03-23T19:53:35Z

    On Tue, Mar 23, 2021 at 12:44 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> > I will make this change to HEAD soon, barring objections.
>
> +1.  Not deleting the upper page seems better than the alternatives.

FWIW it might also work that way as a holdover from the old page
deletion algorithm. These days we decide exactly which pages (leaf
page plus possible internal pages) are to be deleted as a whole up
front (these are a subtree, though usually just a degenerate
single-leaf-page subtree -- internal page deletions are rare).

One of the advantages of this design is that we verify practically all
of the work involved in deleting an entire subtree up-front, inside
_bt_lock_subtree_parent(). It's clearly safe to back out of it if it
looks dicey.

--
Peter Geoghegan

  100. Re: pg_amcheck contrib application

    Peter Geoghegan <pg@bowt.ie> — 2021-03-23T23:13:22Z

    On Tue, Mar 23, 2021 at 12:53 PM Peter Geoghegan <pg@bowt.ie> wrote:
> One of the advantages of this design is that we verify practically all
> of the work involved in deleting an entire subtree up-front, inside
> _bt_lock_subtree_parent(). It's clearly safe to back out of it if it
> looks dicey.

That's taken care of. I just pushed a commit that teaches
_bt_lock_subtree_parent() to press on.

-- 
Peter Geoghegan

  101. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-03-24T06:13:07Z

    
> On Mar 17, 2021, at 9:00 PM, Mark Dilger <mark.dilger@enterprisedb.com> wrote:
> 
> Of the toast pointer fields:
> 
>    int32       va_rawsize;     /* Original data size (includes header) */
>    int32       va_extsize;     /* External saved size (doesn't) */
>    Oid         va_valueid;     /* Unique ID of value within TOAST table */
>    Oid         va_toastrelid;  /* RelID of TOAST table containing it */
> 
> all seem worth getting as part of any toast error message, even if these fields themselves are not corrupt.  It just makes it easier to understand the context of the error you're looking at. At first I tried putting these into each message, but it is very wordy to say things like "toast pointer with rawsize %u and extsize %u pointing at relation with oid %u" and such.  It made more sense to just add these four fields to the verify_heapam tuple format.  That saves putting them in the message text itself, and has the benefit that you could filter the rows coming from verify_heapam() for ones where valueid is or is not null, for example.  This changes the external interface of verify_heapam, but I didn't bother with a amcheck--1.3--1.4.sql because amcheck--1.2--1.3. sql was added as part of the v14 development work and has not yet been released.  My assumption is that I can just change it, rather than making a new upgrade file.
> 
> These patches fix the visibility rules and add extra toast checking. 

These new patches address the same issues as v9 (which was never committed), and v10 (which was never even posted to this list), with some changes.

Rather than print out all four toast pointer fields for each toast failure, va_rawsize, va_extsize, and va_toastrelid are only mentioned in the corruption message if they are related to the specific corruption.  Otherwise, just the va_valueid is mentioned in the corruption message.

The visibility rules fix is different in v11, relying on a visibility check which more closely follows the implementation of HeapTupleSatisfiesVacuumHorizon.

  102. Re: pg_amcheck contrib application

    Robert Haas <robertmhaas@gmail.com> — 2021-03-24T13:12:01Z

    On Wed, Mar 24, 2021 at 2:13 AM Mark Dilger
<mark.dilger@enterprisedb.com> wrote:
> The visibility rules fix is different in v11, relying on a visibility check which more closely follows the implementation of HeapTupleSatisfiesVacuumHorizon.

Hmm. The header comment you wrote says "If a tuple might not be
visible to any running transaction, then we must not check it." But, I
don't find that statement very clear: does it mean "if there could be
even one transaction to which this tuple is not visible, we must not
check it"? Or does it mean "if the number of transactions that can see
this tuple could potentially be zero, then we must not check it"? I
don't think either of those is actually what we care about. I think
what we should be saying is "if the tuple could have been inserted by
a transaction that also added a column to the table, but which
ultimately did not commit, then the table's current TupleDesc might
differ from the one used to construct this tuple, so we must not check
it."

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  103. Re: pg_amcheck contrib application

    Robert Haas <robertmhaas@gmail.com> — 2021-03-24T14:43:22Z

    On Wed, Mar 24, 2021 at 9:12 AM Robert Haas <robertmhaas@gmail.com> wrote:
> On Wed, Mar 24, 2021 at 2:13 AM Mark Dilger
> <mark.dilger@enterprisedb.com> wrote:
> > The visibility rules fix is different in v11, relying on a visibility check which more closely follows the implementation of HeapTupleSatisfiesVacuumHorizon.
>
> Hmm. The header comment you wrote says "If a tuple might not be
> visible to any running transaction, then we must not check it." But, I
> don't find that statement very clear: does it mean "if there could be
> even one transaction to which this tuple is not visible, we must not
> check it"? Or does it mean "if the number of transactions that can see
> this tuple could potentially be zero, then we must not check it"? I
> don't think either of those is actually what we care about. I think
> what we should be saying is "if the tuple could have been inserted by
> a transaction that also added a column to the table, but which
> ultimately did not commit, then the table's current TupleDesc might
> differ from the one used to construct this tuple, so we must not check
> it."

Hit send too soon. And I was wrong, too. Wahoo. Thinking about the
buildfarm failure, I realized that there's a second danger here,
unrelated to the possibility of different TupleDescs, which we talked
about before: if the tuple is dead, we can't safely follow any TOAST
pointers, because the TOAST chunks might disappear at any time. So
technically we could split the return value up into something
three-way: if the inserted is known to have committed, we can check
the tuple itself, because the TupleDesc has to be reasonable. And, if
the tuple is known not to be dead already, and known not to be in a
state where it could become dead while we're doing stuff, we can
follow the TOAST pointer. I'm not sure whether it's worth trying to be
that fancy or not.

If we were only concerned about the mismatched-TupleDesc problem, this
function could return true in a lot more cases. Once we get to the
comment that says "Okay, the inserter committed..." we could just
return true. Similarly, the HEAP_MOVED_IN and HEAP_MOVED_OFF cases
could just skip all the interior test and return true, because if the
tuple is being moved, the original inserter has to have committed.
Conversely, however, the !HeapTupleHeaderXminCommitted ->
TransactionIdIsCurrentTransactionId case probably ought to always
return false. One could argue otherwise: if we're the inserter, then
the only in-progress transaction that might have changed the TupleDesc
is us, so we could just consider this case to be a true return value
also, regardless of what's going on with xmax. In that case, we're not
asking "did the inserter definitely commit?" but "are the inserter's
possible DDL changes definitely visible to us?" which might be an OK
definition too.

However, the could-the-TOAST-data-disappear problem is another story.
I don't see how we can answer that question correctly with the logic
you've got here, because you have no XID threshold. Consider the case
where we reach this code:

+       if (!(tuphdr->t_infomask & HEAP_XMAX_COMMITTED))
+       {
+               if
(TransactionIdIsInProgress(HeapTupleHeaderGetRawXmax(tuphdr)))
+                       return true;            /*
HEAPTUPLE_DELETE_IN_PROGRESS */
+               else if
(!TransactionIdDidCommit(HeapTupleHeaderGetRawXmax(tuphdr)))
+
+                       /*
+                        * Not in Progress, Not Committed, so either
Aborted or crashed
+                        */
+                       return true;            /* HEAPTUPLE_LIVE */
+
+               /* At this point the xmax is known committed */
+       }

If we reach the case where the code comment says
HEAPTUPLE_DELETE_IN_PROGRESS, we know that the tuple isn't dead right
now, and so the TOAST tuples aren't dead either. But, by the time we
go try to look at the TOAST tuples, they might have become dead and
been pruned away, because the deleting transaction can commit at any
time, and after that pruning can happen at any time. Our only
guarantee that that won't happen is if the deleting XID is new enough
that it's invisible to some snapshot that our backend has registered.
That's approximately why HeapTupleSatisfiesVacuumHorizon needs to set
*dead_after in this case and one other, and I think you have the same
requirement.

I just noticed that this whole thing has another, related problem:
check_tuple_header_and_visibilty() and check_tuple_attribute() are
called from within check_tuple(), which is called while we hold a
buffer lock on the heap page. We should not be going and doing complex
operations that might take their own buffer locks - like TOAST index
checks - while we're holding an lwlock. That's going to have to be
changed so that the TOAST pointer checking happens after
UnlockReleaseBuffer(); in other words, we'll need to remember the
TOAST pointers to go look up and actually look them up after
UnlockReleaseBuffer(). But, when we do that, then the HEAPTUPLE_LIVE
case above has the same race condition that is already present in the
HEAPTUPLE_DELETE_IN_PROGRESS case: after we release the buffer pin,
some other transaction might delete the tuple and the associated TOAST
tuples, and they might then commit, and the tuple might become dead
and get pruned away before we check the TOAST table.

On a related note, I notice that your latest patch removes all the
logic that complains about XIDs being out of bounds. I don't think
that's good. Those seem like important checks. They're important for
finding problems with the relation,  and I think we probably also need
them because of the XID-horizon issue mentioned above. One possible
way of looking at it is to say that the XID_BOUNDS_OK case has two
sub-cases: either the XID is within bounds and is one that cannot
become all-visible concurrently because it's not visible to all of our
backend's registered snapshots, or it's within bounds but does have
the possibility of becoming all-visible. In the former case, if it
appears as XMAX we can safely follow TOAST pointers found within the
tuple; in the latter case, we can't.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  104. Re: pg_amcheck contrib application

    Robert Haas <robertmhaas@gmail.com> — 2021-03-24T20:46:45Z

    Mark,

Here's a quick and very dirty sketch of what I think perhaps this
logic could look like. This is pretty much untested and it might be
buggy, but at least you can see whether we're thinking at all in the
same direction.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  105. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-03-29T17:45:04Z

    
> On Mar 24, 2021, at 1:46 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> 
> Mark,
> 
> Here's a quick and very dirty sketch of what I think perhaps this
> logic could look like. This is pretty much untested and it might be
> buggy, but at least you can see whether we're thinking at all in the
> same direction.

Thanks!  The attached patch addresses your comments here and in your prior email.  In particular, this patch changes the tuple visibility logic to not check tuples for which the inserting transaction aborted or is still in progress, and to not check toast for tuples deleted in transactions older than our transaction snapshot's xmin.  A list of toasted attributes which are safe to check is compiled per main table page during the scan of the page, then checked after the buffer lock on the main page is released.

In the perhaps unusual case where verify_heapam() is called in a transaction which has also added tuples to the table being checked, this patch's visibility logic chooses not to check such tuples.  I'm on the fence about this choice, and am mostly following your lead.  I like that this decision maintains the invariant that we never check tuples which have not yet been committed.

The patch includes a bit of refactoring.  In the old code, heap_check() performed clog bounds checking on xmin and xmax prior to calling check_tuple_header_and_visibilty(), but I think that's not such a great choice.  If the tuple header is garbled to have random bytes in the xmin and xmax fields, and we can detect that situation because other tuple header fields are garbled in detectable ways, I'd rather get a report about the header being garbled than a report about the xmin or xmax being out of bounds.  In the new code, the tuple header is checked first, then the visibility is checked, then the tuple is checked against the current relation description, then the tuple attributes are checked.  I think the layout is easier to follow, too.

  106. Re: pg_amcheck contrib application

    Robert Haas <robertmhaas@gmail.com> — 2021-03-29T20:06:45Z

    On Mon, Mar 29, 2021 at 1:45 PM Mark Dilger
<mark.dilger@enterprisedb.com> wrote:
> Thanks!  The attached patch addresses your comments here and in your prior email.  In particular, this patch changes the tuple visibility logic to not check tuples for which the inserting transaction aborted or is still in progress, and to not check toast for tuples deleted in transactions older than our transaction snapshot's xmin.  A list of toasted attributes which are safe to check is compiled per main table page during the scan of the page, then checked after the buffer lock on the main page is released.
>
> In the perhaps unusual case where verify_heapam() is called in a transaction which has also added tuples to the table being checked, this patch's visibility logic chooses not to check such tuples.  I'm on the fence about this choice, and am mostly following your lead.  I like that this decision maintains the invariant that we never check tuples which have not yet been committed.
>
> The patch includes a bit of refactoring.  In the old code, heap_check() performed clog bounds checking on xmin and xmax prior to calling check_tuple_header_and_visibilty(), but I think that's not such a great choice.  If the tuple header is garbled to have random bytes in the xmin and xmax fields, and we can detect that situation because other tuple header fields are garbled in detectable ways, I'd rather get a report about the header being garbled than a report about the xmin or xmax being out of bounds.  In the new code, the tuple header is checked first, then the visibility is checked, then the tuple is checked against the current relation description, then the tuple attributes are checked.  I think the layout is easier to follow, too.

Hmm, so this got ~10x bigger from my version. Could you perhaps
separate it out into a series of patches for easier review? Say, one
that just fixes the visibility logic, and then a second to avoid doing
the TOAST check with a buffer lock held, and then more than that if
there are other pieces that make sense to separate out?

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  107. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-03-29T23:16:44Z

    
> On Mar 29, 2021, at 1:06 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> 
> Hmm, so this got ~10x bigger from my version. Could you perhaps
> separate it out into a series of patches for easier review? Say, one
> that just fixes the visibility logic, and then a second to avoid doing
> the TOAST check with a buffer lock held, and then more than that if
> there are other pieces that make sense to separate out?

Sure, here are four patches which do the same as the single v12 patch did.

  108. Re: pg_amcheck contrib application

    Robert Haas <robertmhaas@gmail.com> — 2021-03-30T19:45:03Z

    On Mon, Mar 29, 2021 at 7:16 PM Mark Dilger
<mark.dilger@enterprisedb.com> wrote:
> Sure, here are four patches which do the same as the single v12 patch did.

Thanks. Here are some comments on 0003 and 0004:

When you posted v11, you said that "Rather than print out all four
toast pointer fields for each toast failure, va_rawsize, va_extsize,
and va_toastrelid are only mentioned in the corruption message if they
are related to the specific corruption.  Otherwise, just the
va_valueid is mentioned in the corruption message." I like that
principal; in fact, as you know, I suggested it. But, with the v13
patches applied, exactly zero of the callers to
report_toast_corruption() appear to be following it, because none of
them include the value ID. I think you need to revise the messages,
e.g. "toasted value for attribute %u missing from toast table" ->
"toast value %u not found in toast table"; "final toast chunk number
%u differs from expected value %u" -> "toast value %u was expected to
end at chunk %u, but ended at chunk %u"; "toast chunk sequence number
is null" -> "toast value %u has toast chunk with null sequence
number". In the first of those example cases, I think you need not
mention the attribute number because it's already there in its own
column.

On a related note, it doesn't look like you are actually checking
va_toastrelid here. Doing so seems like it would be a good idea. It
also seems like it would be good to check that the compressed size is
less than or equal to the uncompressed size.

I do not like the name tuple_is_volatile, because volatile has a
couple of meanings already, and this isn't one of them. A
SQL-callable function is volatile if it might return different outputs
given the same inputs, even within the same SQL statement. A C
variable is volatile if it might be magically modified in ways not
known to the compiler. I had suggested tuple_cannot_die_now, which is
closer to the mark. If you want to be even more precise, you could
talk about whether the tuple is potentially prunable (e.g.
tuple_can_be_pruned, which inverts the sense). That's really what
we're worried about: whether MVCC rules would permit the tuple to be
pruned after we release the buffer lock and before we check TOAST.

I would ideally prefer not to rename report_corruption(). The old name
is clearer, and changing it produces a bunch of churn that I'd rather
avoid. Perhaps the common helper function could be called
report_corruption_internal(), and the callers could be
report_corruption() and report_toast_corruption().

Regarding 0001 and 0002, I think the logic in 0002 looks a lot closer
to correct now, but I want to go through it in more detail. I think,
though, that you've made some of my comments worse. For example, I
wrote: "It should be impossible for xvac to still be running, since
we've removed all that code, but even if it were, it ought to be safe
to read the tuple, since the original inserter must have committed.
But, if the xvac transaction committed, this tuple (and its associated
TOAST tuples) could be pruned at any time." You changed that to read
"We don't bother comparing against safe_xmin because the VACUUM FULL
must have committed prior to an upgrade and can't still be running."
Your comment is shorter, which is a point in its favor, but what I was
trying to emphasize is that the logic would be correct EVEN IF we
again started to use HEAP_MOVED_OFF and HEAP_MOVED_IN again. Your
version makes it sound like the code would need to be revised in that
case. If that's true, then my comment was wrong, but I didn't think it
was true, or I wouldn't have written the comment in that way.

Also, and maybe this is a job for a separate patch, but then again
maybe not, I wonder if it's really a good idea for get_xid_status to
return both a XidBoundsViolation and an XidCommitStatus. It seems to
me (without checking all that carefully) that it might be better to
just flatten all of that into a single enum, because right now it
seems like you often end up with two consecutive switch statements
where, perhaps, just one would suffice.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  109. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-03-31T04:34:01Z

    
> On Mar 30, 2021, at 12:45 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> 
> On Mon, Mar 29, 2021 at 7:16 PM Mark Dilger
> <mark.dilger@enterprisedb.com> wrote:
>> Sure, here are four patches which do the same as the single v12 patch did.
> 
> Thanks. Here are some comments on 0003 and 0004:
> 
> When you posted v11, you said that "Rather than print out all four
> toast pointer fields for each toast failure, va_rawsize, va_extsize,
> and va_toastrelid are only mentioned in the corruption message if they
> are related to the specific corruption.  Otherwise, just the
> va_valueid is mentioned in the corruption message." I like that
> principal; in fact, as you know, I suggested it. But, with the v13
> patches applied, exactly zero of the callers to
> report_toast_corruption() appear to be following it, because none of
> them include the value ID. I think you need to revise the messages,
> e.g.

These changes got lost between v11 and v12.  I've put them back, as well as updating to use your language.

> "toasted value for attribute %u missing from toast table" ->
> "toast value %u not found in toast table";

Changed.

> "final toast chunk number
> %u differs from expected value %u" -> "toast value %u was expected to
> end at chunk %u, but ended at chunk %u";

Changed.

> "toast chunk sequence number
> is null" -> "toast value %u has toast chunk with null sequence
> number".

Changed.

> In the first of those example cases, I think you need not
> mention the attribute number because it's already there in its own
> column.

Correct.  I'd removed that but lost that work in v12.

> On a related note, it doesn't look like you are actually checking
> va_toastrelid here. Doing so seems like it would be a good idea. It
> also seems like it would be good to check that the compressed size is
> less than or equal to the uncompressed size.

Yeah, those checks were in v11 but got lost when I changed things for v12.  They are back in v14.

> I do not like the name tuple_is_volatile, because volatile has a
> couple of meanings already, and this isn't one of them. A
> SQL-callable function is volatile if it might return different outputs
> given the same inputs, even within the same SQL statement. A C
> variable is volatile if it might be magically modified in ways not
> known to the compiler. I had suggested tuple_cannot_die_now, which is
> closer to the mark. If you want to be even more precise, you could
> talk about whether the tuple is potentially prunable (e.g.
> tuple_can_be_pruned, which inverts the sense). That's really what
> we're worried about: whether MVCC rules would permit the tuple to be
> pruned after we release the buffer lock and before we check TOAST.

I used "tuple_can_be_pruned".  I didn't like "tuple_cannot_die_now", and still don't like that name, as it has several wrong interpretations.  One meaning of "cannot die now" is that it has become immortal.  Another is "cannot be deleted from the table".  

> I would ideally prefer not to rename report_corruption(). The old name
> is clearer, and changing it produces a bunch of churn that I'd rather
> avoid. Perhaps the common helper function could be called
> report_corruption_internal(), and the callers could be
> report_corruption() and report_toast_corruption().

Yes, hence the commit message in the previous patch set, "This patch can probably be left out if the committer believes it creates more git churn than it is worth."  I've removed this patch from this next patch set, and used the function names you suggest.

> Regarding 0001 and 0002, I think the logic in 0002 looks a lot closer
> to correct now, but I want to go through it in more detail. I think,
> though, that you've made some of my comments worse. For example, I
> wrote: "It should be impossible for xvac to still be running, since
> we've removed all that code, but even if it were, it ought to be safe
> to read the tuple, since the original inserter must have committed.
> But, if the xvac transaction committed, this tuple (and its associated
> TOAST tuples) could be pruned at any time." You changed that to read
> "We don't bother comparing against safe_xmin because the VACUUM FULL
> must have committed prior to an upgrade and can't still be running."
> Your comment is shorter, which is a point in its favor, but what I was
> trying to emphasize is that the logic would be correct EVEN IF we
> again started to use HEAP_MOVED_OFF and HEAP_MOVED_IN again. Your
> version makes it sound like the code would need to be revised in that
> case. If that's true, then my comment was wrong, but I didn't think it
> was true, or I wouldn't have written the comment in that way.

I think the logic would have to change if we brought back the old VACUUM FULL behavior.

I'm not looking at the old VACUUM FULL code, but my assumption is that if the xvac code were resurrected, then when a tuple is moved off by a VACUUM FULL, the old tuple and associated toast cannot be pruned until concurrent transactions end.  So, if amcheck is running more-or-less concurrently with the VACUUM FULL and has a snapshot xmin no newer than the xid of the VACUUM FULL's xid, it can check the toast associated with the moved off tuple after the VACUUM FULL commits.  If instead the VACUUM FULL xid was older than amcheck's xmin, then the toast is in danger of being vacuumed away.  So the logic in verify_heapam would need to change to think about this distinction.  We don't have to concern ourselves about that, because VACUUM FULL cannot be running, and so the xid for it must be older than our xmin, and hence the toast is unconditionally not safe to check.

I'm changing the comments back to how you had them, but I'd like to know why my reasoning is wrong.

> Also, and maybe this is a job for a separate patch, but then again
> maybe not, I wonder if it's really a good idea for get_xid_status to
> return both a XidBoundsViolation and an XidCommitStatus. It seems to
> me (without checking all that carefully) that it might be better to
> just flatten all of that into a single enum, because right now it
> seems like you often end up with two consecutive switch statements
> where, perhaps, just one would suffice.

get_xid_status was written to return XidBoundsViolation separately from returning by reference an XidCommitStatus because, if you pass null for the XidCommitStatus parameter, the function can return earlier without taking the XactTruncationLock and checking clog.  I think that design made a lot of sense at the time get_xid_status was written, but there are no longer any callers passing null, so the function never returns early.

I am hesitant to refactor get_xid_status as you suggest until we're sure no such callers who pass null are needed.  So perhaps your idea of having that change as a separate patch for after this patch series is done and committed is the right strategy.

Also, even now, there are some places where the returned XidBoundsViolation is used right away, but some other processing happens before the XidCommitStatus is finally used.  If they were one value in a merged enum, there would still be two switches at least in the location I'm thinking of.

  110. Re: pg_amcheck contrib application

    Robert Haas <robertmhaas@gmail.com> — 2021-03-31T17:11:55Z

    On Wed, Mar 31, 2021 at 12:34 AM Mark Dilger
<mark.dilger@enterprisedb.com> wrote:
> I'm not looking at the old VACUUM FULL code, but my assumption is that if the xvac code were resurrected, then when a tuple is moved off by a VACUUM FULL, the old tuple and associated toast cannot be pruned until concurrent transactions end.  So, if amcheck is running more-or-less concurrently with the VACUUM FULL and has a snapshot xmin no newer than the xid of the VACUUM FULL's xid, it can check the toast associated with the moved off tuple after the VACUUM FULL commits.  If instead the VACUUM FULL xid was older than amcheck's xmin, then the toast is in danger of being vacuumed away.  So the logic in verify_heapam would need to change to think about this distinction.  We don't have to concern ourselves about that, because VACUUM FULL cannot be running, and so the xid for it must be older than our xmin, and hence the toast is unconditionally not safe to check.
>
> I'm changing the comments back to how you had them, but I'd like to know why my reasoning is wrong.

Let's start by figuring out *whether* your reasoning is wrong. My
assumption was that old-style VACUUM FULL would move tuples without
retoasting. That is, if we decided to move a tuple from page 2 of the
main table to page 1, we would just write the tuple into page 1,
marking it moved-in, and on page 2 we would mark it moved-off. And
that we would not examine or follow any TOAST pointers at all, so
whatever TOAST entries existed would be shared between the two tuples.
One tuple or the other would eventually die, depending on whether xvac
went on to commit or abort, but either way the TOAST doesn't need
updating because there's always exactly 1 remaining tuple using
pointers to those TOAST values.

Your assumption seems to be the opposite, that the TOASTed values
would be retoasted as part of VF. If that is true, then your idea is
right.

Do you agree with this analysis? If so, we can check the code and find
out which way it actually worked.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  111. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-03-31T17:31:50Z

    
> On Mar 31, 2021, at 10:11 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> 
> On Wed, Mar 31, 2021 at 12:34 AM Mark Dilger
> <mark.dilger@enterprisedb.com> wrote:
>> I'm not looking at the old VACUUM FULL code, but my assumption is that if the xvac code were resurrected, then when a tuple is moved off by a VACUUM FULL, the old tuple and associated toast cannot be pruned until concurrent transactions end.  So, if amcheck is running more-or-less concurrently with the VACUUM FULL and has a snapshot xmin no newer than the xid of the VACUUM FULL's xid, it can check the toast associated with the moved off tuple after the VACUUM FULL commits.  If instead the VACUUM FULL xid was older than amcheck's xmin, then the toast is in danger of being vacuumed away.  So the logic in verify_heapam would need to change to think about this distinction.  We don't have to concern ourselves about that, because VACUUM FULL cannot be running, and so the xid for it must be older than our xmin, and hence the toast is unconditionally not safe to check.
>> 
>> I'm changing the comments back to how you had them, but I'd like to know why my reasoning is wrong.
> 
> Let's start by figuring out *whether* your reasoning is wrong. My
> assumption was that old-style VACUUM FULL would move tuples without
> retoasting. That is, if we decided to move a tuple from page 2 of the
> main table to page 1, we would just write the tuple into page 1,
> marking it moved-in, and on page 2 we would mark it moved-off. And
> that we would not examine or follow any TOAST pointers at all, so
> whatever TOAST entries existed would be shared between the two tuples.
> One tuple or the other would eventually die, depending on whether xvac
> went on to commit or abort, but either way the TOAST doesn't need
> updating because there's always exactly 1 remaining tuple using
> pointers to those TOAST values.
> 
> Your assumption seems to be the opposite, that the TOASTed values
> would be retoasted as part of VF. If that is true, then your idea is
> right.
> 
> Do you agree with this analysis? If so, we can check the code and find
> out which way it actually worked.

Actually, that makes a lot of sense without even looking at the old code.  I was implicitly assuming that the toast table was undergoing a VF also, and that the toast pointers in the main table tuples would be updated to point to the new location, so we'd be unable to follow the pointers to the old location without danger of the old location entries being vacuumed away.  But if the main table tuples get moved while keeping their toast pointers unaltered, then you don't have to worry about that, although you do have to worry that a VF of the main table doesn't help so much with toast table bloat.

We're only discussing this in order to craft the right comment for a bit of code with respect to a hypothetical situation in which VF gets resurrected, so I'm not sure this should be top priority, but I'm curious enough now to go read the old code....

—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  112. Re: pg_amcheck contrib application

    Robert Haas <robertmhaas@gmail.com> — 2021-03-31T17:41:11Z

    On Wed, Mar 31, 2021 at 1:31 PM Mark Dilger
<mark.dilger@enterprisedb.com> wrote:
> Actually, that makes a lot of sense without even looking at the old code.  I was implicitly assuming that the toast table was undergoing a VF also, and that the toast pointers in the main table tuples would be updated to point to the new location, so we'd be unable to follow the pointers to the old location without danger of the old location entries being vacuumed away.  But if the main table tuples get moved while keeping their toast pointers unaltered, then you don't have to worry about that, although you do have to worry that a VF of the main table doesn't help so much with toast table bloat.
>
> We're only discussing this in order to craft the right comment for a bit of code with respect to a hypothetical situation in which VF gets resurrected, so I'm not sure this should be top priority, but I'm curious enough now to go read the old code....

Right, well, we wouldn't be PostgreSQL hackers if we didn't spend lots
of time worrying about obscure details. Whether that's good software
engineering or mere pedantry is sometimes debatable.

I took a look at commit 0a469c87692d15a22eaa69d4b3a43dd8e278dd64,
which removed old-style VACUUM FULL, and AFAICS, it doesn't contain
any references to tuple deforming, varlena, HeapTupleHasExternal, or
anything else that would make me think it has the foggiest idea
whether the tuples it's moving around contain TOAST pointers, so I
think I had the right idea.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  113. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-03-31T17:44:55Z

    
> On Mar 31, 2021, at 10:31 AM, Mark Dilger <mark.dilger@enterprisedb.com> wrote:
> 
> 
> 
>> On Mar 31, 2021, at 10:11 AM, Robert Haas <robertmhaas@gmail.com> wrote:
>> 
>> On Wed, Mar 31, 2021 at 12:34 AM Mark Dilger
>> <mark.dilger@enterprisedb.com> wrote:
>>> I'm not looking at the old VACUUM FULL code, but my assumption is that if the xvac code were resurrected, then when a tuple is moved off by a VACUUM FULL, the old tuple and associated toast cannot be pruned until concurrent transactions end.  So, if amcheck is running more-or-less concurrently with the VACUUM FULL and has a snapshot xmin no newer than the xid of the VACUUM FULL's xid, it can check the toast associated with the moved off tuple after the VACUUM FULL commits.  If instead the VACUUM FULL xid was older than amcheck's xmin, then the toast is in danger of being vacuumed away.  So the logic in verify_heapam would need to change to think about this distinction.  We don't have to concern ourselves about that, because VACUUM FULL cannot be running, and so the xid for it must be older than our xmin, and hence the toast is unconditionally not safe to check.
>>> 
>>> I'm changing the comments back to how you had them, but I'd like to know why my reasoning is wrong.
>> 
>> Let's start by figuring out *whether* your reasoning is wrong. My
>> assumption was that old-style VACUUM FULL would move tuples without
>> retoasting. That is, if we decided to move a tuple from page 2 of the
>> main table to page 1, we would just write the tuple into page 1,
>> marking it moved-in, and on page 2 we would mark it moved-off. And
>> that we would not examine or follow any TOAST pointers at all, so
>> whatever TOAST entries existed would be shared between the two tuples.
>> One tuple or the other would eventually die, depending on whether xvac
>> went on to commit or abort, but either way the TOAST doesn't need
>> updating because there's always exactly 1 remaining tuple using
>> pointers to those TOAST values.
>> 
>> Your assumption seems to be the opposite, that the TOASTed values
>> would be retoasted as part of VF. If that is true, then your idea is
>> right.
>> 
>> Do you agree with this analysis? If so, we can check the code and find
>> out which way it actually worked.
> 
> Actually, that makes a lot of sense without even looking at the old code.  I was implicitly assuming that the toast table was undergoing a VF also, and that the toast pointers in the main table tuples would be updated to point to the new location, so we'd be unable to follow the pointers to the old location without danger of the old location entries being vacuumed away. But if the main table tuples get moved while keeping their toast pointers unaltered, then you don't have to worry about that, although you do have to worry that a VF of the main table doesn't help so much with toast table bloat.
> 
> We're only discussing this in order to craft the right comment for a bit of code with respect to a hypothetical situation in which VF gets resurrected, so I'm not sure this should be top priority, but I'm curious enough now to go read the old code....

Well, that's annoying.  The documentation of postgres 8.2 for vacuum full [1] says,

  Selects "full" vacuum, which may reclaim more space, but takes much longer and exclusively locks the table.

I read "exclusively locks" as meaning it takes an ExclusiveLock, but the code shows that it takes an AccessExclusiveLock.  I think the docs are pretty misleading here, though I understand that grammatically it is hard to say "accessively exclusively locks" or such.  But a part of my analysis was based on the reasoning that if VF only takes an ExclusiveLock, then there must be concurrent readers possible.  VF went away long enough ago that I had forgotten exactly how inconvenient it was.

[1] https://www.postgresql.org/docs/8.2/sql-vacuum.html

—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  114. Re: pg_amcheck contrib application

    Robert Haas <robertmhaas@gmail.com> — 2021-03-31T17:51:40Z

    On Wed, Mar 31, 2021 at 1:44 PM Mark Dilger
<mark.dilger@enterprisedb.com> wrote:
> I read "exclusively locks" as meaning it takes an ExclusiveLock, but the code shows that it takes an AccessExclusiveLock.  I think the docs are pretty misleading here, though I understand that grammatically it is hard to say "accessively exclusively locks" or such.  But a part of my analysis was based on the reasoning that if VF only takes an ExclusiveLock, then there must be concurrent readers possible.  VF went away long enough ago that I had forgotten exactly how inconvenient it was.

It kinda depends on what you mean by concurrent readers, because a
transaction that could start on Monday and acquire an XID, and then on
Tuesday you could run VACUUM FULL on relation "foo", and then on
Wednesday the transaction from before could get around to reading some
data from "foo". The two transactions are concurrent, in the sense
that the 3-day transaction was running before the VACUUM FULL, was
still running after VACUUM FULL, read the same pages that the VACUUM
FULL modified, and cares whether the XID from the VACUUM FULL
committed or aborted. But, it's not concurrent in the sense that you
never have a situation where the VACUUM FULL does some of its
modifications, then an overlapping transaction sees them, and then it
does the rest of them.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  115. Re: pg_amcheck contrib application

    Robert Haas <robertmhaas@gmail.com> — 2021-04-01T15:08:00Z

    On Wed, Mar 31, 2021 at 12:34 AM Mark Dilger
<mark.dilger@enterprisedb.com> wrote:
> These changes got lost between v11 and v12.  I've put them back, as well as updating to use your language.

Here's an updated patch that includes your 0001 and 0002 plus a bunch
of changes by me. I intend to commit this version unless somebody
spots a problem with it.

Here are the things I changed:

- I renamed tuple_can_be_pruned to tuple_could_be_pruned because I
think it does a better job suggesting that we're uncertain about what
will happen.

- I got rid of bool header_garbled and changed it to bool result,
inverting the sense, because it didn't seem useful to have a function
that ended with if (some_boolean) return false; return true when I
could end the function with return some_other_boolean.

- I removed all the one-word comments that said /* corrupt */ or /*
checkable */ because they seemed redundant.

- In the xmin_status section of check_tuple_visibility(), I got rid of
the xmin_status == XID_IS_CURRENT_XID and xmin_status ==
XID_IN_PROGRESS cases because they were redundant with the xmin_status
!= XID_COMMITTED case.

- If xmax is a multi but seems to be garbled, I changed it to return
true rather than false. The inserter is known to have committed by
that point, so I think it's OK to try to deform the tuple. We just
shouldn't try to check TOAST.

- I changed both switches over xmax_status to break in each case and
then return true after instead of returning true for each case. I
think this is clearer.

- I changed get_xid_status() to perform the TransactionIdIs... checks
in the same order that HeapTupleSatisfies...() does them. I believe
that it's incorrect to conclude that the transaction must be in
progress because it neither IsCurrentTransaction nor DidCommit nor
DidAbort, because all of those things will be false for a transaction
that is running at the time of a system crash. The correct rule is
that if it neither IsCurrentTransaction nor IsInProgress nor DidCommit
then it's aborted.

- I moved a few comments and rewrote some others, including some of
the ones that you took from my earlier draft patch. The thing is, the
comment needs to be adjusted based on where you put it. Like, I had a
comment that says"It should be impossible for xvac to still be
running, since we've removed all that code, but even if it were, it
ought to be safe to read the tuple, since the original inserter must
have committed.  But, if the xvac transaction committed, this tuple
(and its associated TOAST tuples) could be pruned at any time." which
in my version was right before a TransactionIdDidCommit() test, and
explains why that test is there and why the code does what it does as
a result. But in your version you've moved it to a place where we've
already tested that the transaction has committed, and more
importantly, where we've already tested that it's not still running.
Saying that it "should" be impossible for it not to be running when
we've *actually checked that* doesn't make nearly as much sense as it
does when we haven't checked that and aren't going to do so.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  116. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-04-01T16:32:44Z

    
> On Apr 1, 2021, at 8:08 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> 
> On Wed, Mar 31, 2021 at 12:34 AM Mark Dilger
> <mark.dilger@enterprisedb.com> wrote:
>> These changes got lost between v11 and v12.  I've put them back, as well as updating to use your language.
> 
> Here's an updated patch that includes your 0001 and 0002 plus a bunch
> of changes by me. I intend to commit this version unless somebody
> spots a problem with it.
> 
> Here are the things I changed:
> 
> - I renamed tuple_can_be_pruned to tuple_could_be_pruned because I
> think it does a better job suggesting that we're uncertain about what
> will happen.

+1

> - I got rid of bool header_garbled and changed it to bool result,
> inverting the sense, because it didn't seem useful to have a function
> that ended with if (some_boolean) return false; return true when I
> could end the function with return some_other_boolean.

+1

> - I removed all the one-word comments that said /* corrupt */ or /*
> checkable */ because they seemed redundant.

Ok.

> - In the xmin_status section of check_tuple_visibility(), I got rid of
> the xmin_status == XID_IS_CURRENT_XID and xmin_status ==
> XID_IN_PROGRESS cases because they were redundant with the xmin_status
> != XID_COMMITTED case.

Ok.

> - If xmax is a multi but seems to be garbled, I changed it to return
> true rather than false. The inserter is known to have committed by
> that point, so I think it's OK to try to deform the tuple. We just
> shouldn't try to check TOAST.

It is hard to know what to do when at least one tuple header field is corrupt.  You don't necesarily know which one it is.  For example, if HEAP_XMAX_IS_MULTI is set, we try to interpret the xmax as a mxid, and if it is out of bounds, we report it as corrupt.  But was the xmax corrupt?  Or was the HEAP_XMAX_IS_MULTI bit corrupt?  It's not clear.  I took the view that if either xmin or xmax appear to be corrupt when interpreted in light of the various tuple header bits, all we really know is that the set of fields/bits don't make sense as a whole, so we report corruption, don't trust any of them, and abort further checking of the tuple.  You have be burden of proof the other way around.  If the xmin appears fine, and xmax appears corrupt, then we only know that xmax is corrupt, so the tuple is checkable because according to the xmin it committed.

I don't think how you have it causes undue problems, since deforming the tuple when you shouldn't merely risks a bunch of extra not-so-helpful corruption messages.  And hey, maybe they're helpful to somebody clever enough to diagnose why that particular bit of noise was generated. 

> - I changed both switches over xmax_status to break in each case and
> then return true after instead of returning true for each case. I
> think this is clearer.

Ok.

> - I changed get_xid_status() to perform the TransactionIdIs... checks
> in the same order that HeapTupleSatisfies...() does them. I believe
> that it's incorrect to conclude that the transaction must be in
> progress because it neither IsCurrentTransaction nor DidCommit nor
> DidAbort, because all of those things will be false for a transaction
> that is running at the time of a system crash. The correct rule is
> that if it neither IsCurrentTransaction nor IsInProgress nor DidCommit
> then it's aborted.

Ok.

> - I moved a few comments and rewrote some others, including some of
> the ones that you took from my earlier draft patch. The thing is, the
> comment needs to be adjusted based on where you put it. Like, I had a
> comment that says"It should be impossible for xvac to still be
> running, since we've removed all that code, but even if it were, it
> ought to be safe to read the tuple, since the original inserter must
> have committed.  But, if the xvac transaction committed, this tuple
> (and its associated TOAST tuples) could be pruned at any time." which
> in my version was right before a TransactionIdDidCommit() test, and
> explains why that test is there and why the code does what it does as
> a result. But in your version you've moved it to a place where we've
> already tested that the transaction has committed, and more
> importantly, where we've already tested that it's not still running.
> Saying that it "should" be impossible for it not to be running when
> we've *actually checked that* doesn't make nearly as much sense as it
> does when we haven't checked that and aren't going to do so.


            * If xmin_status happens to be XID_IN_PROGRESS, then in theory                     
 
Did you mean to say XID_IS_CURRENT_XID here? 

/* xmax is an MXID, not an MXID. Sanity check it. */ 

Is it an MXID or isn't it?

—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  117. Re: pg_amcheck contrib application

    Robert Haas <robertmhaas@gmail.com> — 2021-04-01T16:56:09Z

    On Thu, Apr 1, 2021 at 12:32 PM Mark Dilger
<mark.dilger@enterprisedb.com> wrote:
> > - If xmax is a multi but seems to be garbled, I changed it to return
> > true rather than false. The inserter is known to have committed by
> > that point, so I think it's OK to try to deform the tuple. We just
> > shouldn't try to check TOAST.
>
> It is hard to know what to do when at least one tuple header field is corrupt.  You don't necesarily know which one it is.  For example, if HEAP_XMAX_IS_MULTI is set, we try to interpret the xmax as a mxid, and if it is out of bounds, we report it as corrupt.  But was the xmax corrupt?  Or was the HEAP_XMAX_IS_MULTI bit corrupt?  It's not clear.  I took the view that if either xmin or xmax appear to be corrupt when interpreted in light of the various tuple header bits, all we really know is that the set of fields/bits don't make sense as a whole, so we report corruption, don't trust any of them, and abort further checking of the tuple.  You have be burden of proof the other way around.  If the xmin appears fine, and xmax appears corrupt, then we only know that xmax is corrupt, so the tuple is checkable because according to the xmin it committed.

I agree that it's hard to be sure what's gone once we start finding
corrupted data, but deciding that maybe xmin didn't really commit
because we see that there's something wrong with xmax seems excessive
to me. I thought about a related case: if xmax is a bad multi but is
also hinted invalid, should we try to follow TOAST pointers? I think
that's hard to say, because we don't know whether (1) the invalid
marking is in error, (2) it's wrong to consider it a multi rather than
an XID, (3) the stored multi got overwritten with a garbage value, or
(4) the stored multi got removed before the tuple was frozen. Not
knowing which of those is the case, how are we supposed to decide
whether the TOAST tuples might have been (or be about to get) pruned?

But, in the case we're talking about here, I don't think it's a
particularly close decision. All we need to say is that if xmax or the
infomask bits pertaining to it are corrupted, we're still going to
suppose that xmin and the infomask bits pertaining to it, which are
all different bytes and bits, are OK. To me, the contrary decision,
namely that a bogus xmax means xmin was probably lying about the
transaction having been committed in the first place, seems like a
serious overreaction. As you say:

> I don't think how you have it causes undue problems, since deforming the tuple when you shouldn't merely risks a bunch of extra not-so-helpful corruption messages.  And hey, maybe they're helpful to somebody clever enough to diagnose why that particular bit of noise was generated.

I agree. The biggest risk here is that we might omit >0 complaints
when only 0 are justified. That will panic users. The possibility that
we might omit >x complaints when only x are justified, for some x>0,
is also a risk, but it's not nearly as bad, because there's definitely
something wrong, and it's just a question of what it is exactly. So we
have to be really conservative about saying that X is corruption if
there's any possibility that it might be fine. But once we've
complained about one thing, we can take a more balanced approach about
whether to risk issuing more complaints. The possibility that
suppressing the additional complaints might complicate resolution of
the issue also needs to be considered.

>             * If xmin_status happens to be XID_IN_PROGRESS, then in theory
>
> Did you mean to say XID_IS_CURRENT_XID here?

Yes, I did, thanks.

> /* xmax is an MXID, not an MXID. Sanity check it. */
>
> Is it an MXID or isn't it?

Good catch.

New patch attached.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  118. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-04-01T17:06:04Z

    
> On Apr 1, 2021, at 9:56 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> 
> On Thu, Apr 1, 2021 at 12:32 PM Mark Dilger
> <mark.dilger@enterprisedb.com> wrote:
>>> - If xmax is a multi but seems to be garbled, I changed it to return
>>> true rather than false. The inserter is known to have committed by
>>> that point, so I think it's OK to try to deform the tuple. We just
>>> shouldn't try to check TOAST.
>> 
>> It is hard to know what to do when at least one tuple header field is corrupt.  You don't necesarily know which one it is.  For example, if HEAP_XMAX_IS_MULTI is set, we try to interpret the xmax as a mxid, and if it is out of bounds, we report it as corrupt.  But was the xmax corrupt?  Or was the HEAP_XMAX_IS_MULTI bit corrupt?  It's not clear.  I took the view that if either xmin or xmax appear to be corrupt when interpreted in light of the various tuple header bits, all we really know is that the set of fields/bits don't make sense as a whole, so we report corruption, don't trust any of them, and abort further checking of the tuple.  You have be burden of proof the other way around.  If the xmin appears fine, and xmax appears corrupt, then we only know that xmax is corrupt, so the tuple is checkable because according to the xmin it committed.
> 
> I agree that it's hard to be sure what's gone once we start finding
> corrupted data, but deciding that maybe xmin didn't really commit
> because we see that there's something wrong with xmax seems excessive
> to me. I thought about a related case: if xmax is a bad multi but is
> also hinted invalid, should we try to follow TOAST pointers? I think
> that's hard to say, because we don't know whether (1) the invalid
> marking is in error, (2) it's wrong to consider it a multi rather than
> an XID, (3) the stored multi got overwritten with a garbage value, or
> (4) the stored multi got removed before the tuple was frozen. Not
> knowing which of those is the case, how are we supposed to decide
> whether the TOAST tuples might have been (or be about to get) pruned?
> 
> But, in the case we're talking about here, I don't think it's a
> particularly close decision. All we need to say is that if xmax or the
> infomask bits pertaining to it are corrupted, we're still going to
> suppose that xmin and the infomask bits pertaining to it, which are
> all different bytes and bits, are OK. To me, the contrary decision,
> namely that a bogus xmax means xmin was probably lying about the
> transaction having been committed in the first place, seems like a
> serious overreaction. As you say:
> 
>> I don't think how you have it causes undue problems, since deforming the tuple when you shouldn't merely risks a bunch of extra not-so-helpful corruption messages.  And hey, maybe they're helpful to somebody clever enough to diagnose why that particular bit of noise was generated.
> 
> I agree. The biggest risk here is that we might omit >0 complaints
> when only 0 are justified. That will panic users. The possibility that
> we might omit >x complaints when only x are justified, for some x>0,
> is also a risk, but it's not nearly as bad, because there's definitely
> something wrong, and it's just a question of what it is exactly. So we
> have to be really conservative about saying that X is corruption if
> there's any possibility that it might be fine. But once we've
> complained about one thing, we can take a more balanced approach about
> whether to risk issuing more complaints. The possibility that
> suppressing the additional complaints might complicate resolution of
> the issue also needs to be considered.

This all seems fine to me.  The main thing is that we don't go on to check the toast, which we don't.

>>            * If xmin_status happens to be XID_IN_PROGRESS, then in theory
>> 
>> Did you mean to say XID_IS_CURRENT_XID here?
> 
> Yes, I did, thanks.

Ouch.  You've got a typo:  s/XID_IN_CURRENT_XID/XID_IS_CURRENT_XID/

>> /* xmax is an MXID, not an MXID. Sanity check it. */
>> 
>> Is it an MXID or isn't it?
> 
> Good catch.
> 
> New patch attached.

Seems fine other than the typo.

—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  119. Re: pg_amcheck contrib application

    Robert Haas <robertmhaas@gmail.com> — 2021-04-01T17:20:34Z

    On Thu, Apr 1, 2021 at 1:06 PM Mark Dilger <mark.dilger@enterprisedb.com> wrote:
> Seems fine other than the typo.

OK, let's try that again.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  120. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-04-01T17:24:24Z

    
> On Apr 1, 2021, at 10:20 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> 
> On Thu, Apr 1, 2021 at 1:06 PM Mark Dilger <mark.dilger@enterprisedb.com> wrote:
>> Seems fine other than the typo.
> 
> OK, let's try that again.

Looks good!

—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  121. Re: pg_amcheck contrib application

    Robert Haas <robertmhaas@gmail.com> — 2021-04-01T17:41:50Z

    On Thu, Apr 1, 2021 at 1:24 PM Mark Dilger <mark.dilger@enterprisedb.com> wrote:
> > On Apr 1, 2021, at 10:20 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> > OK, let's try that again.
> Looks good!

OK, committed. We still need to deal with what you had as 0003
upthread, so I guess the next step is for me to spend some time
reviewing that one.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  122. Re: pg_amcheck contrib application

    Robert Haas <robertmhaas@gmail.com> — 2021-04-01T20:08:56Z

    On Thu, Apr 1, 2021 at 1:41 PM Robert Haas <robertmhaas@gmail.com> wrote:
> OK, committed. We still need to deal with what you had as 0003
> upthread, so I guess the next step is for me to spend some time
> reviewing that one.

I did this, and it was a bit depressing. It appears that we now have
duplicate checks for xmin and xmax being out of the valid range.
Somehow you have the removal of those duplicate checks in 0003, but
why in the world didn't you put them into one of the previous patches
and just make 0003 about fixing the
holding-buffer-lock-while-following-TOAST-pointers problem? (And, gah,
why did I not look carefully enough to notice that you hadn't done
that?)

Other than that, I notice a few other things:

- There are a total of two (2) calls in the current source code to
palloc0fast, and hundreds of calls to palloc0. So I think you should
forget about using the fast variant and just do what almost every
other caller does.

- If you want to make this code faster, a better idea would be to
avoid doing all of this allocate and free work and just allocate an
array that's guaranteed to be big enough, and then keep track of how
many elements of that array are actually in use.

- #ifdef DECOMPRESSION_CORRUPTION_CHECKING is not a useful way of
introducing such a feature. Either we do it for real and expose it via
SQL and pg_amcheck as an optional behavior, or we rip it out and
revisit it later. Given the nearness of feature freeze, my vote is for
the latter.

- I'd remove the USE_LZ4 bit, too. Let's not define the presence of
LZ4 data in a non-LZ4-enabled cluster as corruption. If we do, then
people will expect to be able to use this to find places where they
are dependent on LZ4 if they want to move away from it -- and if we
don't recurse into composite datums, that will not actually work.

- check_toast_tuple() has an odd and slightly unclear calling
convention for which there are no comments. I wonder if it would be
better to reverse things and make bool *error the return value and
what is now the return value into a pointer argument, but whatever we
do I think it needs a few words in the comments. We don't need to
slavishly explain every argument -- I think toasttup and ctx and tctx
are reasonably clear -- but this is not.

- To me it would be more logical to reverse the order of the
toast_pointer.va_toastrelid != ctx->rel->rd_rel->reltoastrelid and
VARATT_EXTERNAL_GET_EXTSIZE(toast_pointer) > toast_pointer.va_rawsize
- VARHDRSZ checks. Whether we're pointing at the correct relation
feels more fundamental.

- If we moved the toplevel foreach loop in check_toasted_attributes()
out to the caller, say renaming the function to just
check_toasted_attribute(), we'd save a level of indentation in that
whole function and probably add a tad of clarity, too. You wouldn't
feel the need to Assert(ctx.toasted_attributes == NIL) in the caller
if the caller had just done list_free(ctx->toasted_attributes);
ctx->toasted_attributes = NIL.

- Is there a reason we need a cross-check on both the number of chunks
and on the total size? It seems to me that we should check that each
individual chunk has the size we expect, and that the total number of
chunks is what we expect. The overall size is then necessarily
correct.

- Why are all the changes to the tests in this patch? What do they
have to do with getting the TOAST checks out from under the buffer
lock? I really need you to structure the patch series so that each
patch is about one topic and, equally, so that each topic is only
covered by one patch. Otherwise it's just way too confusing.

- I think some of these messages need a bit of word-smithing, too, but
we can leave that for when we're closer to being done with this.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  123. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-04-05T00:02:40Z

    
> On Apr 1, 2021, at 1:08 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> 
> 
> 
> - There are a total of two (2) calls in the current source code to
> palloc0fast, and hundreds of calls to palloc0. So I think you should
> forget about using the fast variant and just do what almost every
> other caller does.

Done.

> - If you want to make this code faster, a better idea would be to
> avoid doing all of this allocate and free work and just allocate an
> array that's guaranteed to be big enough, and then keep track of how
> many elements of that array are actually in use.

Sounds like premature optimization to me.  I only used palloc0fast because the argument is compile-time known.  I wasn't specifically attempting to speed anything up.

> - #ifdef DECOMPRESSION_CORRUPTION_CHECKING is not a useful way of
> introducing such a feature. Either we do it for real and expose it via
> SQL and pg_amcheck as an optional behavior, or we rip it out and
> revisit it later. Given the nearness of feature freeze, my vote is for
> the latter.
> 
> - I'd remove the USE_LZ4 bit, too. Let's not define the presence of
> LZ4 data in a non-LZ4-enabled cluster as corruption. If we do, then
> people will expect to be able to use this to find places where they
> are dependent on LZ4 if they want to move away from it -- and if we
> don't recurse into composite datums, that will not actually work.

Ok, I have removed this bit.  I also removed the part of the patch that introduced a new corruption check, decompressing the data to see if it decompresses without error.

> - check_toast_tuple() has an odd and slightly unclear calling
> convention for which there are no comments. I wonder if it would be
> better to reverse things and make bool *error the return value and
> what is now the return value into a pointer argument, but whatever we
> do I think it needs a few words in the comments. We don't need to
> slavishly explain every argument -- I think toasttup and ctx and tctx
> are reasonably clear -- but this is not.
...
> - Is there a reason we need a cross-check on both the number of chunks
> and on the total size? It seems to me that we should check that each
> individual chunk has the size we expect, and that the total number of
> chunks is what we expect. The overall size is then necessarily
> correct.

Good point.  I've removed the extra check on the total size, since it cannot be wrong if the checks on the individual chunk sizes were all correct.  This eliminates the need for the odd calling convention for check_toast_tuple(), so I've changed that to return void and not take any return-by-reference arguments.

> - To me it would be more logical to reverse the order of the
> toast_pointer.va_toastrelid != ctx->rel->rd_rel->reltoastrelid and
> VARATT_EXTERNAL_GET_EXTSIZE(toast_pointer) > toast_pointer.va_rawsize
> - VARHDRSZ checks. Whether we're pointing at the correct relation
> feels more fundamental.

Done.

> - If we moved the toplevel foreach loop in check_toasted_attributes()
> out to the caller, say renaming the function to just
> check_toasted_attribute(), we'd save a level of indentation in that
> whole function and probably add a tad of clarity, too. You wouldn't
> feel the need to Assert(ctx.toasted_attributes == NIL) in the caller
> if the caller had just done list_free(ctx->toasted_attributes);
> ctx->toasted_attributes = NIL.

You're right.  It looks nicer that way.  Changed.

> - Why are all the changes to the tests in this patch? What do they
> have to do with getting the TOAST checks out from under the buffer
> lock? I really need you to structure the patch series so that each
> patch is about one topic and, equally, so that each topic is only
> covered by one patch. Otherwise it's just way too confusing.

v18-0001 - Finishes work started in commit 3b6c1259f9 that was overlooked owing to how I had separated the changes in v17-0002 vs. v17-0003

v18-0002 - Postpones the toast checks for a page until after the main table page lock is released

v18-0003 - Improves the corruption messages in ways already discussed earlier in this thread.  Changes the tests to expect the new messages, but adds no new checks

v18-0004 - Adding corruption checks of toast pointers.  Extends the regression tests to cover the new checks.

> 
> - I think some of these messages need a bit of word-smithing, too, but
> we can leave that for when we're closer to being done with this.

Ok.

  124. Re: pg_amcheck contrib application

    Robert Haas <robertmhaas@gmail.com> — 2021-04-07T20:16:22Z

    On Sun, Apr 4, 2021 at 8:02 PM Mark Dilger <mark.dilger@enterprisedb.com> wrote:
> v18-0001 - Finishes work started in commit 3b6c1259f9 that was overlooked owing to how I had separated the changes in v17-0002 vs. v17-0003

Committed.

> v18-0002 - Postpones the toast checks for a page until after the main table page lock is released

Committed, but I changed list_free() to list_free_deep() to avoid a
memory leak, and I revised the commit message to mention the important
point that we need to avoid following TOAST pointers from
potentially-prunable tuples.

> v18-0003 - Improves the corruption messages in ways already discussed earlier in this thread.  Changes the tests to expect the new messages, but adds no new checks

Kibitizing your message wording:

"toast value %u chunk data is null" -> "toast value %u chunk %d has
null data". We can mention the chunk number this way.

"toast value %u corrupt extended chunk has invalid varlena header: %0x
(sequence number %d)" -> "toast value %u chunk %d has invalid varlena
header %0x". We can be more consistent about how we incorporate the
chunk number into the text, and we don't really need to include the
word corrupt, because all of these are corruption complaints, and I
think it looks better without the colon.

"toast value %u chunk sequence number %u does not match the expected
sequence number %u" -> "toast value %u contains chunk %d where chunk
%d was expected". Shorter. Uses %d for a sequence number instead of
%u, which I think is correct -- anyway we should have them all one way
or all the other. I think I'd rather ditch the "sequence number"
technology and just talk about "chunk %d" or whatever.

"toast value %u chunk sequence number %u exceeds the end chunk
sequence number %u" -> "toast value %u chunk %d follows last expected
chunk %d"

"toast value %u chunk size %u differs from the expected size %u" ->
"toast value %u chunk %d has size %u, but expected size %u"

Other complaints:

Your commit message fails to mention the addition of
VARATT_EXTERNAL_GET_POINTER, which is a significant change/bug fix
unrelated to message wording.

It feels like we have a non-minimal number of checks/messages for the
series of toast chunks. I think that if we find a chunk after the last
chunk we were expecting to find (curchunk > endchunk) and you also get
a message if we have the wrong number of chunks in total (chunkno !=
(endchunk + 1)). Now maybe I'm wrong, but if the first message
triggers, it seems like the second message must also trigger. Is that
wrong? If not, maybe we can get rid of the first one entirely? That's
such a small change I think we could include it in this same patch, if
it's a correct idea.

On a related note, as I think I said before, I still think we should
be rejiggering this so that we're not testing both the size of each
individual chunk and the total size, because that ought to be
redundant. That might be better done as a separate patch but I think
we should try to clean it up.

> v18-0004 - Adding corruption checks of toast pointers.  Extends the regression tests to cover the new checks.

I think we could check that the result of
VARATT_EXTERNAL_GET_COMPRESS_METHOD is one of the values we expect to
see.

Using AllocSizeIsValid() seems pretty vile. I know that MaxAllocSize
is 0x3FFFFFFF in no small part because that's the maximum length that
can be represented by a varlena, but I'm not sure it's a good idea to
couple the concepts so closely like this. Maybe we can just #define
VARLENA_SIZE_LIMIT in this file and use that, and a message that says
size %u exceeds limit %u.

I'm a little worried about whether the additional test cases are
Endian-dependent at all. I don't immediately know what might be wrong
with them, but I'm going to think about that some more later. Any
chance you have access to a Big-endian box where you can test this?

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  125. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-04-08T19:02:36Z

    
> On Apr 7, 2021, at 1:16 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> 
> On Sun, Apr 4, 2021 at 8:02 PM Mark Dilger <mark.dilger@enterprisedb.com> wrote:
>> v18-0001 - Finishes work started in commit 3b6c1259f9 that was overlooked owing to how I had separated the changes in v17-0002 vs. v17-0003
> 
> Committed.

Thank you.

>> v18-0002 - Postpones the toast checks for a page until after the main table page lock is released
> 
> Committed, but I changed list_free() to list_free_deep() to avoid a
> memory leak, and I revised the commit message to mention the important
> point that we need to avoid following TOAST pointers from
> potentially-prunable tuples.

Thank you, and yes, I agree with that change.

>> v18-0003 - Improves the corruption messages in ways already discussed earlier in this thread.  Changes the tests to expect the new messages, but adds no new checks
> 
> Kibitizing your message wording:
> 
> "toast value %u chunk data is null" -> "toast value %u chunk %d has
> null data". We can mention the chunk number this way.

Changed.

> "toast value %u corrupt extended chunk has invalid varlena header: %0x
> (sequence number %d)" -> "toast value %u chunk %d has invalid varlena
> header %0x". We can be more consistent about how we incorporate the
> chunk number into the text, and we don't really need to include the
> word corrupt, because all of these are corruption complaints, and I
> think it looks better without the colon.

Changed.

> "toast value %u chunk sequence number %u does not match the expected
> sequence number %u" -> "toast value %u contains chunk %d where chunk
> %d was expected". Shorter. Uses %d for a sequence number instead of
> %u, which I think is correct -- anyway we should have them all one way
> or all the other. I think I'd rather ditch the "sequence number"
> technology and just talk about "chunk %d" or whatever.

I don't agree with this one.  I do agree with changing the message, but not to the message you suggest.

Imagine a toasted attribute with 18 chunks numbered [0..17].  Then we update the toast to have only 6 chunks numbered [0..5] except we corruptly keep chunks numbered [12..17] in the toast table.  We'd rather see a report like this:

# heap table "postgres"."public"."test", block 0, offset 1, attribute 2:
#     toast value 16444 chunk 6 has sequence number 12, but expected sequence number 6
# heap table "postgres"."public"."test", block 0, offset 1, attribute 2:
#     toast value 16444 chunk 7 has sequence number 13, but expected sequence number 7
# heap table "postgres"."public"."test", block 0, offset 1, attribute 2:
#     toast value 16444 chunk 8 has sequence number 14, but expected sequence number 8
# heap table "postgres"."public"."test", block 0, offset 1, attribute 2:
#     toast value 16444 chunk 9 has sequence number 15, but expected sequence number 9
# heap table "postgres"."public"."test", block 0, offset 1, attribute 2:
#     toast value 16444 chunk 10 has sequence number 16, but expected sequence number 10
# heap table "postgres"."public"."test", block 0, offset 1, attribute 2:
#     toast value 16444 chunk 11 has sequence number 17, but expected sequence number 11
# heap table "postgres"."public"."test", block 0, offset 1, attribute 2:
#     toast value 16444 was expected to end at chunk 6, but ended at chunk 12

than one like this:

# heap table "postgres"."public"."test", block 0, offset 1, attribute 2:
#     toast value 16444 contains chunk 12 where chunk 6 was expected
# heap table "postgres"."public"."test", block 0, offset 1, attribute 2:
#     toast value 16444 contains chunk 13 where chunk 7 was expected
# heap table "postgres"."public"."test", block 0, offset 1, attribute 2:
#     toast value 16444 contains chunk 14 where chunk 8 was expected
# heap table "postgres"."public"."test", block 0, offset 1, attribute 2:
#     toast value 16444 contains chunk 15 where chunk 9 was expected
# heap table "postgres"."public"."test", block 0, offset 1, attribute 2:
#     toast value 16444 contains chunk 16 where chunk 10 was expected
# heap table "postgres"."public"."test", block 0, offset 1, attribute 2:
#     toast value 16444 contains chunk 17 where chunk 11 was expected
# heap table "postgres"."public"."test", block 0, offset 1, attribute 2:
#     toast value 16444 was expected to end at chunk 6, but ended at chunk 12

because saying the toast value ended at "chunk 12" after saying that it contains "chunk 17" is contradictory.  You need the distinction between the chunk number and the chunk sequence number, since in corrupt circumstances they may not be the same.

> "toast value %u chunk sequence number %u exceeds the end chunk
> sequence number %u" -> "toast value %u chunk %d follows last expected
> chunk %d"

Changed.

> "toast value %u chunk size %u differs from the expected size %u" ->
> "toast value %u chunk %d has size %u, but expected size %u"

Changed.

> Other complaints:
> 
> Your commit message fails to mention the addition of
> VARATT_EXTERNAL_GET_POINTER, which is a significant change/bug fix
> unrelated to message wording.

Right you are.

> It feels like we have a non-minimal number of checks/messages for the
> series of toast chunks. I think that if we find a chunk after the last
> chunk we were expecting to find (curchunk > endchunk) and you also get
> a message if we have the wrong number of chunks in total (chunkno !=
> (endchunk + 1)). Now maybe I'm wrong, but if the first message
> triggers, it seems like the second message must also trigger. Is that
> wrong? If not, maybe we can get rid of the first one entirely? That's
> such a small change I think we could include it in this same patch, if
> it's a correct idea.

Motivated by discussions we had off-list, I dug into this one.

Purely as manual testing, and not part of the patch, I hacked the backend a bit to allow direct modification of the toast table.  After corrupting the toast with the following bit of SQL:

	WITH chunk_limit AS (
		SELECT chunk_id, MAX(chunk_seq) AS maxseq
			FROM $toastname
			GROUP BY chunk_id)
		INSERT INTO $toastname (chunk_id, chunk_seq, chunk_data)
			(SELECT t.chunk_id,
					t.chunk_seq + cl.maxseq + CASE WHEN t.chunk_seq < 3 THEN 1 ELSE 7 END,
					t.chunk_data
				FROM $toastname t
				INNER JOIN chunk_limit cl
				ON t.chunk_id = cl.chunk_id)

pg_amcheck reports the following corruption messages:

# heap table "postgres"."public"."test", block 0, offset 1, attribute 2:
#     toast value 16444 chunk 6 follows last expected chunk 5
# heap table "postgres"."public"."test", block 0, offset 1, attribute 2:
#     toast value 16444 chunk 7 follows last expected chunk 5
# heap table "postgres"."public"."test", block 0, offset 1, attribute 2:
#     toast value 16444 chunk 8 follows last expected chunk 5
# heap table "postgres"."public"."test", block 0, offset 1, attribute 2:
#     toast value 16444 chunk 9 has sequence number 15, but expected sequence number 9
# heap table "postgres"."public"."test", block 0, offset 1, attribute 2:
#     toast value 16444 chunk 10 has sequence number 16, but expected sequence number 10
# heap table "postgres"."public"."test", block 0, offset 1, attribute 2:
#     toast value 16444 chunk 11 has sequence number 17, but expected sequence number 11
# heap table "postgres"."public"."test", block 0, offset 1, attribute 2:
#     toast value 16444 was expected to end at chunk 6, but ended at chunk 12

I think if we'd left out the first three messages, it would read strangely.  We would be complaining about three chunks with the wrong sequence number, then conclude that there were six extra chunks.  A sufficiently savvy user might deduce the presence of chunks 6, 7, and 8, but the problem is more obvious (to my eyes, at least) if we keep the first three messages.  This seems like a judgement call and not a clear argument either way, so if you still want me to change it, I guess I don't mind doing so.

> On a related note, as I think I said before, I still think we should
> be rejiggering this so that we're not testing both the size of each
> individual chunk and the total size, because that ought to be
> redundant. That might be better done as a separate patch but I think
> we should try to clean it up.

Can you point me to the exact check you are mentioning, and with which patch applied?  I don't see any examples of this after applying the v18-0003.

>> v18-0004 - Adding corruption checks of toast pointers.  Extends the regression tests to cover the new checks.
> 
> I think we could check that the result of
> VARATT_EXTERNAL_GET_COMPRESS_METHOD is one of the values we expect to
> see.

Yes.  I had that before, pulled it out along with other toast compression checks, but have put it back in for v19.

> Using AllocSizeIsValid() seems pretty vile. I know that MaxAllocSize
> is 0x3FFFFFFF in no small part because that's the maximum length that
> can be represented by a varlena, but I'm not sure it's a good idea to
> couple the concepts so closely like this. Maybe we can just #define
> VARLENA_SIZE_LIMIT in this file and use that, and a message that says
> size %u exceeds limit %u.

Changed.

> I'm a little worried about whether the additional test cases are
> Endian-dependent at all. I don't immediately know what might be wrong
> with them, but I'm going to think about that some more later. Any
> chance you have access to a Big-endian box where you can test this?

I don't have a Big-endian box, but I think one of them may be wrong now that you mention the issue:

# Corrupt column c's toast pointer va_extinfo field

The problem is that the 30-bit extsize and 2-bit cmid split is not being handled in the perl test, and I don't see an easy way to have perl's pack/unpack do that for us.  There isn't any requirement that each possible corruption we check actually be manifested in the regression tests.  The simplest solution is to remove this problematic test, so that's what I did.  The other two new tests corrupt c_va_toastrelid and c_va_rawsize, both of which are read/written using unpack/pack, so perl should handle the endianness for us (I hope).

If you'd rather not commit these two extra tests, you don't have to, as I've split them out into v19-0003.  But if you do commit them, it makes more sense to me to be one commit with 0002+0003 together, rather than separately.   Not committing the new tests just means that verify_heapam() is able to detect additional forms of corruption that we're not covering in the regression tests.  But that's already true for some other corruption types, such as detecting toast chunks with null sequence numbers.

  126. Re: pg_amcheck contrib application

    Robert Haas <robertmhaas@gmail.com> — 2021-04-08T20:05:02Z

    On Thu, Apr 8, 2021 at 3:02 PM Mark Dilger <mark.dilger@enterprisedb.com> wrote:
> Imagine a toasted attribute with 18 chunks numbered [0..17].  Then we update the toast to have only 6 chunks numbered [0..5] except we corruptly keep chunks numbered [12..17] in the toast table.  We'd rather see a report like this:
[ toast value NNN chunk NNN has sequence number NNN, but expected
sequence number NNN ]
> than one like this:
[ toast value NNN contains chunk NNN where chunk NNN was expected ]
>
> because saying the toast value ended at "chunk 12" after saying that it contains "chunk 17" is contradictory.  You need the distinction between the chunk number and the chunk sequence number, since in corrupt circumstances they may not be the same.

Hmm, I see your point, and that's a good example to illustrate it.
But, with that example in front of me, I am rather doubtful that
either of these is what users actually want. Consider the case where I
should have chunks 0..17 and chunk 1 is just plain gone. This, by the
way, seems like a pretty likely case to arise in practice, since all
we need is for a block to get truncated away or zeroed erroneously, or
for a tuple to get pruned that shouldn't. With either of the above
schemes, I guess we're going to get a message about every chunk from 2
to 17, complaining that they're all misnumbered. We might also get a
complaint that the last chunk is the wrong size, and that the total
number of chunks isn't right. What we really want is a single
complaint saying chunk 1 is missing.

Likewise, in your example, I sort of feel like what I really want,
rather than either of the above outputs, is to get some messages like
this:

toast value NNN contains unexpected extra chunk [12-17]

Both your phrasing for those messages and what I suggested make it
sound like the problem is that the chunk number is wrong. But that
doesn't seem like it's taking the right view of the situation. Chunks
12-17 shouldn't exist at all, and if they do, we should say that, e.g.
by complaining about something like "toast value 16444 chunk 12
follows last expected chunk 5"

In other words, I don't buy the idea that the user will accept the
idea that there's a chunk number and a chunk sequence number, and that
they should know the difference between those things and what each of
them are. They're entitled to imagine that there's just one thing, and
that we're going to tell them about value that are extra or missing.
The fact that we're not doing that seems like it's just a matter of
missing code. If we start the index scan and get chunk 4, we can
easily emit messages for chunks 0..3 right on the spot, declaring them
missing. Things do get a bit hairy if the index scan returns values
out of order: what if it gives us chunk_seq = 2 and then chunk_seq =
1? But I think we could handle that by just issuing a complaint in any
such case that "toast index returns chunks out of order for toast
value NNN" and stopping further checking of that toast value.

> Purely as manual testing, and not part of the patch, I hacked the backend a bit to allow direct modification of the toast table.  After corrupting the toast with the following bit of SQL:
>
>         WITH chunk_limit AS (
>                 SELECT chunk_id, MAX(chunk_seq) AS maxseq
>                         FROM $toastname
>                         GROUP BY chunk_id)
>                 INSERT INTO $toastname (chunk_id, chunk_seq, chunk_data)
>                         (SELECT t.chunk_id,
>                                         t.chunk_seq + cl.maxseq + CASE WHEN t.chunk_seq < 3 THEN 1 ELSE 7 END,
>                                         t.chunk_data
>                                 FROM $toastname t
>                                 INNER JOIN chunk_limit cl
>                                 ON t.chunk_id = cl.chunk_id)
>
> pg_amcheck reports the following corruption messages:
>
> # heap table "postgres"."public"."test", block 0, offset 1, attribute 2:
> #     toast value 16444 chunk 6 follows last expected chunk 5
> # heap table "postgres"."public"."test", block 0, offset 1, attribute 2:
> #     toast value 16444 chunk 7 follows last expected chunk 5
> # heap table "postgres"."public"."test", block 0, offset 1, attribute 2:
> #     toast value 16444 chunk 8 follows last expected chunk 5
> # heap table "postgres"."public"."test", block 0, offset 1, attribute 2:
> #     toast value 16444 chunk 9 has sequence number 15, but expected sequence number 9
> # heap table "postgres"."public"."test", block 0, offset 1, attribute 2:
> #     toast value 16444 chunk 10 has sequence number 16, but expected sequence number 10
> # heap table "postgres"."public"."test", block 0, offset 1, attribute 2:
> #     toast value 16444 chunk 11 has sequence number 17, but expected sequence number 11
> # heap table "postgres"."public"."test", block 0, offset 1, attribute 2:
> #     toast value 16444 was expected to end at chunk 6, but ended at chunk 12
>
> I think if we'd left out the first three messages, it would read strangely.  We would be complaining about three chunks with the wrong sequence number, then conclude that there were six extra chunks.  A sufficiently savvy user might deduce the presence of chunks 6, 7, and 8, but the problem is more obvious (to my eyes, at least) if we keep the first three messages.  This seems like a judgement call and not a clear argument either way, so if you still want me to change it, I guess I don't mind doing so.

I mean, looking at it, the question here is why it's not just using
the same message for all of them. The fact that the chunk numbers are
higher than 5 is the problem. The sequence numbers seem like just a
distraction.

> > On a related note, as I think I said before, I still think we should
> > be rejiggering this so that we're not testing both the size of each
> > individual chunk and the total size, because that ought to be
> > redundant. That might be better done as a separate patch but I think
> > we should try to clean it up.
>
> Can you point me to the exact check you are mentioning, and with which patch applied?  I don't see any examples of this after applying the v18-0003.

Hmm, my mistake, I think.

> > I'm a little worried about whether the additional test cases are
> > Endian-dependent at all. I don't immediately know what might be wrong
> > with them, but I'm going to think about that some more later. Any
> > chance you have access to a Big-endian box where you can test this?
>
> I don't have a Big-endian box, but I think one of them may be wrong now that you mention the issue:
>
> # Corrupt column c's toast pointer va_extinfo field
>
> The problem is that the 30-bit extsize and 2-bit cmid split is not being handled in the perl test, and I don't see an easy way to have perl's pack/unpack do that for us.  There isn't any requirement that each possible corruption we check actually be manifested in the regression tests.  The simplest solution is to remove this problematic test, so that's what I did.  The other two new tests corrupt c_va_toastrelid and c_va_rawsize, both of which are read/written using unpack/pack, so perl should handle the endianness for us (I hope).

I don't immediately see why this particular thing should be an issue.
The format of the varlena header itself is different on big-endian and
little-endian machines, which is why postgres.h has all this stuff
conditioned on WORDS_BIGENDIAN. But va_extinfo doesn't have any
similar treatment, so I'm not sure what could go wrong there, as long
as the 4-byte value as a whole is being packed and unpacked according
to the machine's endian-ness.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  127. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-04-08T21:21:28Z

    
> On Apr 8, 2021, at 1:05 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> 
> On Thu, Apr 8, 2021 at 3:02 PM Mark Dilger <mark.dilger@enterprisedb.com> wrote:
>> Imagine a toasted attribute with 18 chunks numbered [0..17].  Then we update the toast to have only 6 chunks numbered [0..5] except we corruptly keep chunks numbered [12..17] in the toast table.  We'd rather see a report like this:
> [ toast value NNN chunk NNN has sequence number NNN, but expected
> sequence number NNN ]
>> than one like this:
> [ toast value NNN contains chunk NNN where chunk NNN was expected ]
>> 
>> because saying the toast value ended at "chunk 12" after saying that it contains "chunk 17" is contradictory.  You need the distinction between the chunk number and the chunk sequence number, since in corrupt circumstances they may not be the same.
> 
> Hmm, I see your point, and that's a good example to illustrate it.
> But, with that example in front of me, I am rather doubtful that
> either of these is what users actually want. Consider the case where I
> should have chunks 0..17 and chunk 1 is just plain gone. This, by the
> way, seems like a pretty likely case to arise in practice, since all
> we need is for a block to get truncated away or zeroed erroneously, or
> for a tuple to get pruned that shouldn't. With either of the above
> schemes, I guess we're going to get a message about every chunk from 2
> to 17, complaining that they're all misnumbered. We might also get a
> complaint that the last chunk is the wrong size, and that the total
> number of chunks isn't right. What we really want is a single
> complaint saying chunk 1 is missing.
> 
> Likewise, in your example, I sort of feel like what I really want,
> rather than either of the above outputs, is to get some messages like
> this:
> 
> toast value NNN contains unexpected extra chunk [12-17]
> 
> Both your phrasing for those messages and what I suggested make it
> sound like the problem is that the chunk number is wrong. But that
> doesn't seem like it's taking the right view of the situation. Chunks
> 12-17 shouldn't exist at all, and if they do, we should say that, e.g.
> by complaining about something like "toast value 16444 chunk 12
> follows last expected chunk 5"
> 
> In other words, I don't buy the idea that the user will accept the
> idea that there's a chunk number and a chunk sequence number, and that
> they should know the difference between those things and what each of
> them are. They're entitled to imagine that there's just one thing, and
> that we're going to tell them about value that are extra or missing.
> The fact that we're not doing that seems like it's just a matter of
> missing code.

Somehow, we have to get enough information about chunk_seq discontinuity into the output that if the user forwards it to -hackers, or if the output comes from a buildfarm critter, that we have all the information to help diagnose what went wrong.

As a specific example, if the va_rawsize suggests 2 chunks, and we find 150 chunks all with contiguous chunk_seq values, that is different from a debugging point of view than if we find 150 chunks with chunk_seq values spread all over the [0..MAXINT] range.  We can't just tell the user that there were 148 extra chunks.  We also shouldn't phrase the error in terms of "extra chunks", since it might be the va_rawsize that is corrupt.

I agree that the current message output might be overly verbose in how it reports this information.  Conceptually, we want to store up information about the chunk issues and report them all at once, but that's hard to do in general, as the number of chunk_seq discontinuities might be quite large, much too large to fit reasonably into any one message.  Maybe we could report just the first N discontinuities rather than all of them, but if somebody wants to open a hex editor and walk through the toast table, they won't appreciate having the corruption information truncated like that.

All this leads me to believe that we should report the following:

1) If the total number of chunks retrieved differs from the expected number, report how many we expected vs. how many we got
2) If the chunk_seq numbers are discontiguous, report each discontiguity.
3) If the index scan returned chunks out of chunk_seq order, report that
4) If any chunk is not the expected size, report that

So, for your example of chunk 1 missing from chunks [0..17], we'd report that we got one fewer chunks than we expected, that the second chunk seen was discontiguous from the first chunk seen, that the final chunk seen was smaller than expected by M bytes, and that the total size was smaller than we expected by N bytes.  The third of those is somewhat misleading, since the final chunk was presumably the right size; we just weren't expecting to hit a partial chunk quite yet.  But I don't see how to make that better in the general case.

> If we start the index scan and get chunk 4, we can
> easily emit messages for chunks 0..3 right on the spot, declaring them
> missing. Things do get a bit hairy if the index scan returns values
> out of order: what if it gives us chunk_seq = 2 and then chunk_seq =
> 1? But I think we could handle that by just issuing a complaint in any
> such case that "toast index returns chunks out of order for toast
> value NNN" and stopping further checking of that toast value.
> 
>> Purely as manual testing, and not part of the patch, I hacked the backend a bit to allow direct modification of the toast table.  After corrupting the toast with the following bit of SQL:
>> 
>>        WITH chunk_limit AS (
>>                SELECT chunk_id, MAX(chunk_seq) AS maxseq
>>                        FROM $toastname
>>                        GROUP BY chunk_id)
>>                INSERT INTO $toastname (chunk_id, chunk_seq, chunk_data)
>>                        (SELECT t.chunk_id,
>>                                        t.chunk_seq + cl.maxseq + CASE WHEN t.chunk_seq < 3 THEN 1 ELSE 7 END,
>>                                        t.chunk_data
>>                                FROM $toastname t
>>                                INNER JOIN chunk_limit cl
>>                                ON t.chunk_id = cl.chunk_id)
>> 
>> pg_amcheck reports the following corruption messages:
>> 
>> # heap table "postgres"."public"."test", block 0, offset 1, attribute 2:
>> #     toast value 16444 chunk 6 follows last expected chunk 5
>> # heap table "postgres"."public"."test", block 0, offset 1, attribute 2:
>> #     toast value 16444 chunk 7 follows last expected chunk 5
>> # heap table "postgres"."public"."test", block 0, offset 1, attribute 2:
>> #     toast value 16444 chunk 8 follows last expected chunk 5
>> # heap table "postgres"."public"."test", block 0, offset 1, attribute 2:
>> #     toast value 16444 chunk 9 has sequence number 15, but expected sequence number 9
>> # heap table "postgres"."public"."test", block 0, offset 1, attribute 2:
>> #     toast value 16444 chunk 10 has sequence number 16, but expected sequence number 10
>> # heap table "postgres"."public"."test", block 0, offset 1, attribute 2:
>> #     toast value 16444 chunk 11 has sequence number 17, but expected sequence number 11
>> # heap table "postgres"."public"."test", block 0, offset 1, attribute 2:
>> #     toast value 16444 was expected to end at chunk 6, but ended at chunk 12
>> 
>> I think if we'd left out the first three messages, it would read strangely.  We would be complaining about three chunks with the wrong sequence number, then conclude that there were six extra chunks.  A sufficiently savvy user might deduce the presence of chunks 6, 7, and 8, but the problem is more obvious (to my eyes, at least) if we keep the first three messages.  This seems like a judgement call and not a clear argument either way, so if you still want me to change it, I guess I don't mind doing so.
> 
> I mean, looking at it, the question here is why it's not just using
> the same message for all of them. The fact that the chunk numbers are
> higher than 5 is the problem. The sequence numbers seem like just a
> distraction.

Again, I don't think we can reach that conclusion.  You are biasing the corruption reports in favor of believing the va_rawsize rather than believing the toast table.

>>> On a related note, as I think I said before, I still think we should
>>> be rejiggering this so that we're not testing both the size of each
>>> individual chunk and the total size, because that ought to be
>>> redundant. That might be better done as a separate patch but I think
>>> we should try to clean it up.
>> 
>> Can you point me to the exact check you are mentioning, and with which patch applied?  I don't see any examples of this after applying the v18-0003.
> 
> Hmm, my mistake, I think.
> 
>>> I'm a little worried about whether the additional test cases are
>>> Endian-dependent at all. I don't immediately know what might be wrong
>>> with them, but I'm going to think about that some more later. Any
>>> chance you have access to a Big-endian box where you can test this?
>> 
>> I don't have a Big-endian box, but I think one of them may be wrong now that you mention the issue:
>> 
>> # Corrupt column c's toast pointer va_extinfo field
>> 
>> The problem is that the 30-bit extsize and 2-bit cmid split is not being handled in the perl test, and I don't see an easy way to have perl's pack/unpack do that for us.  There isn't any requirement that each possible corruption we check actually be manifested in the regression tests.  The simplest solution is to remove this problematic test, so that's what I did.  The other two new tests corrupt c_va_toastrelid and c_va_rawsize, both of which are read/written using unpack/pack, so perl should handle the endianness for us (I hope).
> 
> I don't immediately see why this particular thing should be an issue.
> The format of the varlena header itself is different on big-endian and
> little-endian machines, which is why postgres.h has all this stuff
> conditioned on WORDS_BIGENDIAN. But va_extinfo doesn't have any
> similar treatment, so I'm not sure what could go wrong there, as long
> as the 4-byte value as a whole is being packed and unpacked according
> to the machine's endian-ness.

Good point.  Perhaps the test was ok after all.

—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  128. Re: pg_amcheck contrib application

    Robert Haas <robertmhaas@gmail.com> — 2021-04-08T22:11:17Z

    On Thu, Apr 8, 2021 at 5:21 PM Mark Dilger <mark.dilger@enterprisedb.com> wrote:
> All this leads me to believe that we should report the following:
>
> 1) If the total number of chunks retrieved differs from the expected number, report how many we expected vs. how many we got
> 2) If the chunk_seq numbers are discontiguous, report each discontiguity.
> 3) If the index scan returned chunks out of chunk_seq order, report that
> 4) If any chunk is not the expected size, report that
>
> So, for your example of chunk 1 missing from chunks [0..17], we'd report that we got one fewer chunks than we expected, that the second chunk seen was discontiguous from the first chunk seen, that the final chunk seen was smaller than expected by M bytes, and that the total size was smaller than we expected by N bytes.  The third of those is somewhat misleading, since the final chunk was presumably the right size; we just weren't expecting to hit a partial chunk quite yet.  But I don't see how to make that better in the general case.

Hmm, that might be OK. It seems like it's going to be a bit verbose in
simple cases like 1 missing chunk, but on the plus side, it avoids a
mountain of output if the raw size has been overwritten with a
gigantic bogus value. But, how is #2 different from #3? Those sound
like the same thing to me.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  129. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-04-08T22:51:41Z

    
> On Apr 8, 2021, at 3:11 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> 
> On Thu, Apr 8, 2021 at 5:21 PM Mark Dilger <mark.dilger@enterprisedb.com> wrote:
>> All this leads me to believe that we should report the following:
>> 
>> 1) If the total number of chunks retrieved differs from the expected number, report how many we expected vs. how many we got
>> 2) If the chunk_seq numbers are discontiguous, report each discontiguity.
>> 3) If the index scan returned chunks out of chunk_seq order, report that
>> 4) If any chunk is not the expected size, report that
>> 
>> So, for your example of chunk 1 missing from chunks [0..17], we'd report that we got one fewer chunks than we expected, that the second chunk seen was discontiguous from the first chunk seen, that the final chunk seen was smaller than expected by M bytes, and that the total size was smaller than we expected by N bytes.  The third of those is somewhat misleading, since the final chunk was presumably the right size; we just weren't expecting to hit a partial chunk quite yet.  But I don't see how to make that better in the general case.
> 
> Hmm, that might be OK. It seems like it's going to be a bit verbose in
> simple cases like 1 missing chunk, but on the plus side, it avoids a
> mountain of output if the raw size has been overwritten with a
> gigantic bogus value. But, how is #2 different from #3? Those sound
> like the same thing to me.

#2 is if chunk_seq goes up but skips numbers.  #3 is if chunk_seq ever goes down, meaning the index scan did something unexpected.

—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  130. Re: pg_amcheck contrib application

    Robert Haas <robertmhaas@gmail.com> — 2021-04-08T22:55:46Z

    On Thu, Apr 8, 2021 at 6:51 PM Mark Dilger <mark.dilger@enterprisedb.com> wrote:
> #2 is if chunk_seq goes up but skips numbers.  #3 is if chunk_seq ever goes down, meaning the index scan did something unexpected.

Yeah, sure. But I think we could probably treat those the same way.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  131. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-04-09T18:50:32Z

    
> On Apr 8, 2021, at 3:11 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> 
> On Thu, Apr 8, 2021 at 5:21 PM Mark Dilger <mark.dilger@enterprisedb.com> wrote:
>> All this leads me to believe that we should report the following:
>> 
>> 1) If the total number of chunks retrieved differs from the expected number, report how many we expected vs. how many we got
>> 2) If the chunk_seq numbers are discontiguous, report each discontiguity.
>> 3) If the index scan returned chunks out of chunk_seq order, report that
>> 4) If any chunk is not the expected size, report that
>> 
>> So, for your example of chunk 1 missing from chunks [0..17], we'd report that we got one fewer chunks than we expected, that the second chunk seen was discontiguous from the first chunk seen, that the final chunk seen was smaller than expected by M bytes, and that the total size was smaller than we expected by N bytes.  The third of those is somewhat misleading, since the final chunk was presumably the right size; we just weren't expecting to hit a partial chunk quite yet.  But I don't see how to make that better in the general case.
> 
> Hmm, that might be OK. It seems like it's going to be a bit verbose in
> simple cases like 1 missing chunk, but on the plus side, it avoids a
> mountain of output if the raw size has been overwritten with a
> gigantic bogus value. But, how is #2 different from #3? Those sound
> like the same thing to me.

I think #4, above, requires some clarification.  If there are missing chunks, the very definition of how large we expect subsequent chunks to be is ill-defined.  I took a fairly conservative approach to avoid lots of bogus complaints about chunks that are of unexpected size.   Not all such complaints are removed, but enough are removed that I needed to add a final complaint at the end about the total size seen not matching the total size expected.

Here are a set of corruptions with the corresponding corruption reports from before and from after the code changes.  The corruptions are *not* cumulative.

Honestly, I'm not totally convinced that these changes are improvements in all cases.  Let me know if you want further changes, or if you'd like to see other corruptions and their before and after results.


Corruption #1:

	UPDATE $toastname SET chunk_seq = chunk_seq + 1000

Before:

# heap table "postgres"."public"."test", block 0, offset 2, attribute 2:
#     toast value 16445 chunk 0 has sequence number 1000, but expected sequence number 0
# heap table "postgres"."public"."test", block 0, offset 2, attribute 2:
#     toast value 16445 chunk 1 has sequence number 1001, but expected sequence number 1
# heap table "postgres"."public"."test", block 0, offset 2, attribute 2:
#     toast value 16445 chunk 2 has sequence number 1002, but expected sequence number 2
# heap table "postgres"."public"."test", block 0, offset 2, attribute 2:
#     toast value 16445 chunk 3 has sequence number 1003, but expected sequence number 3
# heap table "postgres"."public"."test", block 0, offset 2, attribute 2:
#     toast value 16445 chunk 4 has sequence number 1004, but expected sequence number 4
# heap table "postgres"."public"."test", block 0, offset 2, attribute 2:
#     toast value 16445 chunk 5 has sequence number 1005, but expected sequence number 5

After:

# heap table "postgres"."public"."test", block 0, offset 2, attribute 2:
#     toast value 16445 missing chunks 0 through 999


Corruption #2:

	UPDATE $toastname SET chunk_seq = chunk_seq * 1000

Before:

# heap table "postgres"."public"."test", block 0, offset 2, attribute 2:
#     toast value 16445 chunk 1 has sequence number 1000, but expected sequence number 1
# heap table "postgres"."public"."test", block 0, offset 2, attribute 2:
#     toast value 16445 chunk 2 has sequence number 2000, but expected sequence number 2
# heap table "postgres"."public"."test", block 0, offset 2, attribute 2:
#     toast value 16445 chunk 3 has sequence number 3000, but expected sequence number 3
# heap table "postgres"."public"."test", block 0, offset 2, attribute 2:
#     toast value 16445 chunk 4 has sequence number 4000, but expected sequence number 4
# heap table "postgres"."public"."test", block 0, offset 2, attribute 2:
#     toast value 16445 chunk 5 has sequence number 5000, but expected sequence number 5

After:

# heap table "postgres"."public"."test", block 0, offset 2, attribute 2:
#     toast value 16445 missing chunks 1 through 999
# heap table "postgres"."public"."test", block 0, offset 2, attribute 2:
#     toast value 16445 missing chunks 1001 through 1999
# heap table "postgres"."public"."test", block 0, offset 2, attribute 2:
#     toast value 16445 missing chunks 2001 through 2999
# heap table "postgres"."public"."test", block 0, offset 2, attribute 2:
#     toast value 16445 missing chunks 3001 through 3999
# heap table "postgres"."public"."test", block 0, offset 2, attribute 2:
#     toast value 16445 missing chunks 4001 through 4999
# heap table "postgres"."public"."test", block 0, offset 3, attribute 2:


Corruption #3:

	UPDATE $toastname SET chunk_id = (chunk_id::integer + 10000000)::oid WHERE chunk_seq = 3

Before:

# heap table "postgres"."public"."test", block 0, offset 2, attribute 2:
#     toast value 16445 chunk 3 has sequence number 4, but expected sequence number 3
# heap table "postgres"."public"."test", block 0, offset 2, attribute 2:
#     toast value 16445 chunk 4 has sequence number 5, but expected sequence number 4
# heap table "postgres"."public"."test", block 0, offset 2, attribute 2:
#     toast value 16445 was expected to end at chunk 6, but ended at chunk 5

After:

# heap table "postgres"."public"."test", block 0, offset 2, attribute 2:
#     toast value 16445 missing chunk 3
# heap table "postgres"."public"."test", block 0, offset 2, attribute 2:
#     toast value 16445 chunk 4 has size 20, but expected size 1996
# heap table "postgres"."public"."test", block 0, offset 2, attribute 2:
#     toast value 16445 was expected to end at chunk 6, but ended at chunk 5
# heap table "postgres"."public"."test", block 0, offset 2, attribute 2:
#     toast value 16445 was expected to have size 10000, but had size 8004
# heap table "postgres"."public"."test", block 0, offset 3, attribute 2:



—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  132. Re: pg_amcheck contrib application

    Robert Haas <robertmhaas@gmail.com> — 2021-04-09T20:51:53Z

    On Fri, Apr 9, 2021 at 2:50 PM Mark Dilger <mark.dilger@enterprisedb.com> wrote:
> I think #4, above, requires some clarification.  If there are missing chunks, the very definition of how large we expect subsequent chunks to be is ill-defined.  I took a fairly conservative approach to avoid lots of bogus complaints about chunks that are of unexpected size.   Not all such complaints are removed, but enough are removed that I needed to add a final complaint at the end about the total size seen not matching the total size expected.

My instinct is to suppose that the size that we expect for future
chunks is independent of anything being wrong with previous chunks. So
if each chunk is supposed to be 2004 bytes (which probably isn't the
real number) and the value is 7000 bytes long, we expect chunks 0-2 to
be 2004 bytes each, chunk 3 to be 988 bytes, and chunk 4 and higher to
not exist. If chunk 1 happens to be missing or the wrong length or
whatever, our expectations for chunks 2 and 3 are utterly unchanged.

> Corruption #1:
>
>         UPDATE $toastname SET chunk_seq = chunk_seq + 1000
>
> Before:
>
> # heap table "postgres"."public"."test", block 0, offset 2, attribute 2:
> #     toast value 16445 chunk 0 has sequence number 1000, but expected sequence number 0
> # heap table "postgres"."public"."test", block 0, offset 2, attribute 2:
> #     toast value 16445 chunk 1 has sequence number 1001, but expected sequence number 1
> # heap table "postgres"."public"."test", block 0, offset 2, attribute 2:
> #     toast value 16445 chunk 2 has sequence number 1002, but expected sequence number 2
> # heap table "postgres"."public"."test", block 0, offset 2, attribute 2:
> #     toast value 16445 chunk 3 has sequence number 1003, but expected sequence number 3
> # heap table "postgres"."public"."test", block 0, offset 2, attribute 2:
> #     toast value 16445 chunk 4 has sequence number 1004, but expected sequence number 4
> # heap table "postgres"."public"."test", block 0, offset 2, attribute 2:
> #     toast value 16445 chunk 5 has sequence number 1005, but expected sequence number 5
>
> After:
>
> # heap table "postgres"."public"."test", block 0, offset 2, attribute 2:
> #     toast value 16445 missing chunks 0 through 999

Applying the above principle would lead to complaints that chunks 0-5
are missing, and 1000-1005 are extra.

> Corruption #2:
>
>         UPDATE $toastname SET chunk_seq = chunk_seq * 1000

Similarly here, except the extra chunk numbers are different.

> Corruption #3:
>
>         UPDATE $toastname SET chunk_id = (chunk_id::integer + 10000000)::oid WHERE chunk_seq = 3

And here we'd just get a complaint that chunk 3 is missing.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  133. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-04-13T03:06:09Z

    
> On Apr 9, 2021, at 1:51 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> 
> On Fri, Apr 9, 2021 at 2:50 PM Mark Dilger <mark.dilger@enterprisedb.com> wrote:
>> I think #4, above, requires some clarification.  If there are missing chunks, the very definition of how large we expect subsequent chunks to be is ill-defined.  I took a fairly conservative approach to avoid lots of bogus complaints about chunks that are of unexpected size.   Not all such complaints are removed, but enough are removed that I needed to add a final complaint at the end about the total size seen not matching the total size expected.
> 
> My instinct is to suppose that the size that we expect for future
> chunks is independent of anything being wrong with previous chunks. So
> if each chunk is supposed to be 2004 bytes (which probably isn't the
> real number) and the value is 7000 bytes long, we expect chunks 0-2 to
> be 2004 bytes each, chunk 3 to be 988 bytes, and chunk 4 and higher to
> not exist. If chunk 1 happens to be missing or the wrong length or
> whatever, our expectations for chunks 2 and 3 are utterly unchanged.

Fair enough.

>> Corruption #1:
>> 
>>        UPDATE $toastname SET chunk_seq = chunk_seq + 1000
>> 
>> Before:
>> 
>> # heap table "postgres"."public"."test", block 0, offset 2, attribute 2:
>> #     toast value 16445 chunk 0 has sequence number 1000, but expected sequence number 0
>> # heap table "postgres"."public"."test", block 0, offset 2, attribute 2:
>> #     toast value 16445 chunk 1 has sequence number 1001, but expected sequence number 1
>> # heap table "postgres"."public"."test", block 0, offset 2, attribute 2:
>> #     toast value 16445 chunk 2 has sequence number 1002, but expected sequence number 2
>> # heap table "postgres"."public"."test", block 0, offset 2, attribute 2:
>> #     toast value 16445 chunk 3 has sequence number 1003, but expected sequence number 3
>> # heap table "postgres"."public"."test", block 0, offset 2, attribute 2:
>> #     toast value 16445 chunk 4 has sequence number 1004, but expected sequence number 4
>> # heap table "postgres"."public"."test", block 0, offset 2, attribute 2:
>> #     toast value 16445 chunk 5 has sequence number 1005, but expected sequence number 5
>> 
>> After:
>> 
>> # heap table "postgres"."public"."test", block 0, offset 2, attribute 2:
>> #     toast value 16445 missing chunks 0 through 999
> 
> Applying the above principle would lead to complaints that chunks 0-5
> are missing, and 1000-1005 are extra.

That sounds right.  It now reports:

# heap table "postgres"."public"."test", block 0, offset 16, attribute 2:
#     toast value 16459 missing chunks 0 through 4 with expected size 1996 and chunk 5 with expected size 20
# heap table "postgres"."public"."test", block 0, offset 16, attribute 2:
#     toast value 16459 unexpected chunks 1000 through 1004 each with size 1996 followed by chunk 1005 with size 20


>> Corruption #2:
>> 
>>        UPDATE $toastname SET chunk_seq = chunk_seq * 1000
> 
> Similarly here, except the extra chunk numbers are different.

It now reports:

# heap table "postgres"."public"."test", block 0, offset 17, attribute 2:
#     toast value 16460 missing chunks 1 through 4 with expected size 1996 and chunk 5 with expected size 20
# heap table "postgres"."public"."test", block 0, offset 17, attribute 2:
#     toast value 16460 unexpected chunk 1000 with size 1996
# heap table "postgres"."public"."test", block 0, offset 17, attribute 2:
#     toast value 16460 unexpected chunk 2000 with size 1996
# heap table "postgres"."public"."test", block 0, offset 17, attribute 2:
#     toast value 16460 unexpected chunk 3000 with size 1996
# heap table "postgres"."public"."test", block 0, offset 17, attribute 2:
#     toast value 16460 unexpected chunk 4000 with size 1996
# heap table "postgres"."public"."test", block 0, offset 17, attribute 2:
#     toast value 16460 unexpected chunk 5000 with size 20

I don't see any good way in this case to report the extra chunks in one row, as in the general case there could be arbitrarily many of them, with the message text getting arbitrarily large if it reported the chunks as "chunks 1000, 2000, 3000, 4000, 5000, ...".  I don't expect this sort of corruption being particularly common, though, so I'm not too bothered about it.

> 
>> Corruption #3:
>> 
>>        UPDATE $toastname SET chunk_id = (chunk_id::integer + 10000000)::oid WHERE chunk_seq = 3
> 
> And here we'd just get a complaint that chunk 3 is missing.

It now reports:

# heap table "postgres"."public"."test", block 0, offset 18, attribute 2:
#     toast value 16461 missing chunk 3 with expected size 1996
# heap table "postgres"."public"."test", block 0, offset 18, attribute 2:
#     toast value 16461 was expected to end at chunk 6 with total size 10000, but ended at chunk 5 with total size 8004

It sounds like you weren't expecting the second of these reports.  I think it is valuable, especially when there are multiple missing chunks and multiple extraneous chunks, as it makes it easier for the user to reconcile the missing chunks against the extraneous chunks.

  134. Re: pg_amcheck contrib application

    Robert Haas <robertmhaas@gmail.com> — 2021-04-14T17:17:33Z

    On Mon, Apr 12, 2021 at 11:06 PM Mark Dilger
<mark.dilger@enterprisedb.com> wrote:
> It now reports:
>
> # heap table "postgres"."public"."test", block 0, offset 18, attribute 2:
> #     toast value 16461 missing chunk 3 with expected size 1996
> # heap table "postgres"."public"."test", block 0, offset 18, attribute 2:
> #     toast value 16461 was expected to end at chunk 6 with total size 10000, but ended at chunk 5 with total size 8004
>
> It sounds like you weren't expecting the second of these reports.  I think it is valuable, especially when there are multiple missing chunks and multiple extraneous chunks, as it makes it easier for the user to reconcile the missing chunks against the extraneous chunks.

I wasn't, but I'm not overwhelmingly opposed to it, either. I do think
I would be in favor of splitting this kind of thing up into two
messages:

#     toast value 16459 unexpected chunks 1000 through 1004 each with
size 1996 followed by chunk 1005 with size 20

We'll have fewer message variants and I don't think any real
regression in usability if we say:

#     toast value 16459 has unexpected chunks 1000 through 1004 each
with size 1996
#     toast value 16459 has unexpected chunk 1005 with size 20

(Notice that I also inserted "has" so that the sentence a verb. Or we
could "contains.")

I committed 0001.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  135. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-04-15T17:07:16Z

    
> On Apr 14, 2021, at 10:17 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> 
> On Mon, Apr 12, 2021 at 11:06 PM Mark Dilger
> <mark.dilger@enterprisedb.com> wrote:
>> It now reports:
>> 
>> # heap table "postgres"."public"."test", block 0, offset 18, attribute 2:
>> #     toast value 16461 missing chunk 3 with expected size 1996
>> # heap table "postgres"."public"."test", block 0, offset 18, attribute 2:
>> #     toast value 16461 was expected to end at chunk 6 with total size 10000, but ended at chunk 5 with total size 8004
>> 
>> It sounds like you weren't expecting the second of these reports.  I think it is valuable, especially when there are multiple missing chunks and multiple extraneous chunks, as it makes it easier for the user to reconcile the missing chunks against the extraneous chunks.
> 
> I wasn't, but I'm not overwhelmingly opposed to it, either. I do think
> I would be in favor of splitting this kind of thing up into two
> messages:
> 
> #     toast value 16459 unexpected chunks 1000 through 1004 each with
> size 1996 followed by chunk 1005 with size 20
> 
> We'll have fewer message variants and I don't think any real
> regression in usability if we say:
> 
> #     toast value 16459 has unexpected chunks 1000 through 1004 each
> with size 1996
> #     toast value 16459 has unexpected chunk 1005 with size 20

Changed.

> (Notice that I also inserted "has" so that the sentence a verb. Or we
> could "contains.")

I have added the verb "has" rather than "contains" because "has" is more consistent with the phrasing of other similar corruption reports.

  136. Re: pg_amcheck contrib application

    Robert Haas <robertmhaas@gmail.com> — 2021-04-19T19:50:52Z

    On Thu, Apr 15, 2021 at 1:07 PM Mark Dilger
<mark.dilger@enterprisedb.com> wrote:
> I have added the verb "has" rather than "contains" because "has" is more consistent with the phrasing of other similar corruption reports.

That makes sense.

I think it's odd that a range of extraneous chunks is collapsed into a
single report if the size of each chunk happens to be
TOAST_MAX_CHUNK_SIZE and not otherwise. Why not just remember the
first and last extraneous chunk and the size of each? If the next
chunk you see is the next one in sequence and the same size as all the
others, extend your notion of the sequence end by 1. Otherwise, report
the range accumulated so far. It seems to me that this wouldn't be any
more code than you have now, and might actually be less.

I think that report_missing_chunks() could probably just report the
range of missing chunks and not bother reporting how big they were
expected to be. But, if it is going to report how big they were
expected to be, I think it should have only 2 cases rather than 4:
either a range of missing chunks of equal size, or a single missing
chunk of some size. If, as I propose, it doesn't report the expected
size, then you still have just 2 cases: a range of missing chunks, or
a single missing chunk.

Somehow I have a hard time feeling confident that check_toast_tuple()
is going to do the right thing. The logic is really complex and hard
to understand. 'chunkno' is a counter that advances every time we move
to the next chunk, and 'curchunk' is the value we actually find in the
TOAST tuple. This terminology is not easy to understand. Most messages
now report 'curchunk', but some still report 'chunkno'. Why does
'chunkno' need to exist at all? AFAICS the combination of 'curchunk'
and 'tctx->last_chunk_seen' ought to be sufficient. I can see no
particular reason why what you're calling 'chunkno' needs to exist
even as a local variable, let alone be printed out. Either we haven't
yet validated that the chunk_id extracted from the tuple is non-null
and greater than the last chunk number we saw, in which case we can
just complain about it if we find it to be otherwise, or we have
already done that validation, in which case we should complain about
that value and not 'chunkno' in any subsequent messages.

The conditionals between where you set max_valid_prior_chunk and where
you set last_chunk_seen seem hard to understand, particularly the
bifurcated way that missing chunks are reported. Initial missing
chunks are detected by (chunkno == 0 && max_valid_prior_chunk >= 0)
and later missing chunks are detected by (chunkno > 0 &&
max_valid_prior_chunk > tctx->last_chunk_seen). I'm not sure if this
is correct; I find it hard to get my head around what
max_valid_prior_chunk is supposed to represent. But in any case I
think it can be written more simply. Just keep track of what chunk_id
we expect to extract from the next TOAST tuple. Initially it's 0.
Then:

if (chunkno < tctx->expected_chunkno)
{
   // toast value %u index scan returned chunk number %d when chunk %d
was expected
   // don't modify tctx->expected_chunkno here, just hope the next
thing matches our previous expectation
}
else
{
    if (chunkno > tctx->expected_chunkno)
        // chunks are missing from tctx->expected_chunkno through
Min(chunkno - 1, tctx->final_expected_chunk), provided that the latter
value is greater than or equal to the former
    tctx->expected_chunkno = chunkno + 1;
}

If you do this, you only need to report extraneous chunks when chunkno
> tctx->final_expected_chunk, since chunkno < 0 is guaranteed to
trigger the first of the two complaints shown above.

In check_tuple_attribute I suggest "bool valid = false" rather than
"bool invalid = true". I think it's easier to understand.

I object to check_toasted_attribute() using 'chunkno' in a message for
the same reasons as above in regards to check_toast_tuple() i.e. I
think it's a concept which should not exist.

I think this patch could possibly be split up into multiple patches.
There's some question in my mind whether it's getting too late to
commit any of this, since some of it looks suspiciously like new
features after feature freeze. However, I kind of hate to ship this
release without at least doing something about the chunkno vs.
curchunk stuff, which is even worse in the committed code than in your
patch, and which I think will confuse the heck out of users if those
messages actually fire for anyone.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  137. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-04-20T00:07:58Z

    
> On Apr 19, 2021, at 12:50 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> 
> On Thu, Apr 15, 2021 at 1:07 PM Mark Dilger
> <mark.dilger@enterprisedb.com> wrote:
>> I have added the verb "has" rather than "contains" because "has" is more consistent with the phrasing of other similar corruption reports.
> 
> That makes sense.
> 
> I think it's odd that a range of extraneous chunks is collapsed into a
> single report if the size of each chunk happens to be
> TOAST_MAX_CHUNK_SIZE and not otherwise. Why not just remember the
> first and last extraneous chunk and the size of each? If the next
> chunk you see is the next one in sequence and the same size as all the
> others, extend your notion of the sequence end by 1. Otherwise, report
> the range accumulated so far. It seems to me that this wouldn't be any
> more code than you have now, and might actually be less.

In all cases of uncorrupted toasted attributes, the sequence of N chunks that make up the attribute should be N-1 chunks of TOAST_MAX_CHUNK_SIZE ending with a single chunk of up to TOAST_MAX_CHUNK_SIZE.  I'd like to refer to such sequences as "reasonably sized" sequences to make conversation easier.

If the toast pointer's va_extsize field leads us to believe that we should find 10 reasonably sized chunks, but instead we find 30 reasonably sized chunks, we know something is corrupt.  We shouldn't automatically prejudice the user against the additional 20 chunks.  We didn't expect them, but maybe that's because va_extsize was corrupt and gave us a false expectation.  We're not pointing fingers one way or the other.

On the other hand, if we expect 10 chunks and find an additional 20 unreasonably sized chunks, we can and should point fingers at the extra 20 chunks.  Even if we somehow knew that va_extsize was also corrupt, we'd still be justified in saying the 20 unreasonably sized chunks are each, individually corrupt.

I tried to write the code to report one corruption message per corruption found.  There are some edge cases where this is a definitional challenge, so it's not easy to say that I've always achieved this goal, but I think i've done so where the definitions are clear.  As such, the only time I'd want to combine toast chunks into a single corruption message is when they are not in themselves necessarily *individually* corrupt.  That is why I wrote the code to use TOAST_MAX_CHUNK_SIZE rather than just storing up any series of equally sized chunks.

On a related note, when complaining about a sequence of toast chunks, often the sequence is something like [maximal, maximal, ..., maximal, partial], but sometimes it's just [maximal...maximal], sometimes just [maximal], and sometimes just [partial].  If I'm complaining about that entire sequence, I'd really like to do so in just one message, otherwise it looks like separate complaints.

I can certainly change the code to be how you are asking, but I'd first like to know that you really understood what I was doing here and why the reports read the way they do.

> I think that report_missing_chunks() could probably just report the
> range of missing chunks and not bother reporting how big they were
> expected to be. But, if it is going to report how big they were
> expected to be, I think it should have only 2 cases rather than 4:
> either a range of missing chunks of equal size, or a single missing
> chunk of some size. If, as I propose, it doesn't report the expected
> size, then you still have just 2 cases: a range of missing chunks, or
> a single missing chunk.

Right, this is the same as above.  I'm trying not to split a single corruption complaint into separate reports.

> Somehow I have a hard time feeling confident that check_toast_tuple()
> is going to do the right thing. The logic is really complex and hard
> to understand. 'chunkno' is a counter that advances every time we move
> to the next chunk, and 'curchunk' is the value we actually find in the
> TOAST tuple. This terminology is not easy to understand. Most messages
> now report 'curchunk', but some still report 'chunkno'. Why does
> 'chunkno' need to exist at all? AFAICS the combination of 'curchunk'
> and 'tctx->last_chunk_seen' ought to be sufficient. I can see no
> particular reason why what you're calling 'chunkno' needs to exist
> even as a local variable, let alone be printed out. Either we haven't
> yet validated that the chunk_id extracted from the tuple is non-null
> and greater than the last chunk number we saw, in which case we can
> just complain about it if we find it to be otherwise, or we have
> already done that validation, in which case we should complain about
> that value and not 'chunkno' in any subsequent messages.

If we use tctx->last_chunk_seen as you propose, I imagine we'd set that to -1 prior to the first call to check_toast_tuple().  In the first call, we'd extract the toast chunk_seq and store it in curchunk and verify that it's one greater than tctx->last_chunk_seen.  That all seems fine.

But under corrupt conditions, curchunk = DatumGetInt32(fastgetattr(toasttup, 2, hctx->toast_rel->rd_att, &isnull)) could return -1.  That's invalid, of course, but now we don't know what to do.  We're supposed to complain when we get the same chunk_seq from the index scan more than once in a row, but we don't know if the value in last_chunk_seen is a real value or just the dummy initial value.  Worse still, when we get the next toast tuple back and it has a chunk_seq of -2, we want to complain that the index is returning tuples in reverse order, but we can't, because we still don't know if the -1 in last_chunk_seen is legitimate or a dummy value because that state information isn't carried over from the previous call.

Using chunkno solves this problem.  If chunkno == 0, it means this is our first call, and tctx->last_chunk_seen is uninitialized.  Otherwise, this is not the first call, and tctx->last_chunk_seen really is the chunk_seq seen in the prior call.  There is no ambiguity.

I could probably change "int chunkno" to "bool is_first_call" or similar.  I had previously used chunkno in the corruption report about chunks whose chunk_seq is null.  The idea was that if you have 100 chunks and the 30th chunk is corruptly nulled out, you could say something like "toast value 178337 has toast chunk 30 with null sequence number",  but you had me change that to "toast value 178337 has toast chunk with null sequence number", so generation of that message no longer needs the chunkno.  I had kept chunkno around for the other purpose of knowing whether tctx->last_chunk_seen has been initialized yet, but a bool for that would now be sufficient.  In any event, though you disagree with me about this below, I think the caller of this code still needs to track chunkno.

> The conditionals between where you set max_valid_prior_chunk and where
> you set last_chunk_seen seem hard to understand, particularly the
> bifurcated way that missing chunks are reported. Initial missing
> chunks are detected by (chunkno == 0 && max_valid_prior_chunk >= 0)
> and later missing chunks are detected by (chunkno > 0 &&
> max_valid_prior_chunk > tctx->last_chunk_seen). I'm not sure if this
> is correct;

When we read a chunk_seq from a toast tuple, we need to determine if it indicates a gap in the chunk sequence, but we need to be careful. 

The (chunkno == 0) and (chunkno > 0) stuff is just distinguishing between the first call and all subsequent calls.

For illustrative purposes, imagine that we expect chunks [0..4].

On the first call, we expect chunk_seq = 0, but that's not what we actually complain about if we get chunk_seq = 15.  We complain about all missing expected chunks, namely [0..4], not [0..14].  We also don't complain yet about seeing extraneous chunk 15, because it might be the first in a series of contiguous extraneous chunks, and we want to wait and report those all at once when the sequence finishes.  Simply complaining at this point that we didn't expect to see chunk_seq 15 is the kind of behavior that we already have committed and are trying to fix because the corruption reports are not on point.

On subsequent calls, we expect chunk_seq = last_chunk_seen+1, but that's also not what we actually complain about if we get some other value for chunk_seq.  What we complain about are the missing and extraneous sequences, not the individual chunk that had an unexpected value.

> I find it hard to get my head around what
> max_valid_prior_chunk is supposed to represent. But in any case I
> think it can be written more simply. Just keep track of what chunk_id
> we expect to extract from the next TOAST tuple. Initially it's 0.
> Then:
> 
> if (chunkno < tctx->expected_chunkno)
> {
>   // toast value %u index scan returned chunk number %d when chunk %d
> was expected
>   // don't modify tctx->expected_chunkno here, just hope the next
> thing matches our previous expectation
> }
> else
> {
>    if (chunkno > tctx->expected_chunkno)
>        // chunks are missing from tctx->expected_chunkno through
> Min(chunkno - 1, tctx->final_expected_chunk), provided that the latter
> value is greater than or equal to the former
>    tctx->expected_chunkno = chunkno + 1;
> }
> 
> If you do this, you only need to report extraneous chunks when chunkno
>> tctx->final_expected_chunk, since chunkno < 0 is guaranteed to
> trigger the first of the two complaints shown above.

In the example above, if we're expecting chunks [0..4] and get chunk_seq = 5, the max_valid_prior_chunk is 4.  If we instead get chunk_seq = 6, the max_valid_prior_chunk is still 4, because chunk 5 is out of bounds.

> In check_tuple_attribute I suggest "bool valid = false" rather than
> "bool invalid = true". I think it's easier to understand.

Yeah, I had it that way and changed it, because I don't much like having the only use of a boolean be a negation.

    bool foo = false; ... if (!foo) { ... }

seems worse to me than

    bool foo = true; ... if (foo) { ... }

But you're looking at it more from the perspective of english grammar, where "invalid = false" reads as a double-negative.  That's fine.  I can change it back.

> I object to check_toasted_attribute() using 'chunkno' in a message for
> the same reasons as above in regards to check_toast_tuple() i.e. I
> think it's a concept which should not exist.

So if we expect 100 chunks, get chunks [0..19, 80..99], you'd have me write the message as "expected 100 chunks but sequence ended at chunk 99"?  I think that's odd.  It makes infinitely more sense to me to say "expected 100 chunks but sequence ended at chunk 40".  Actually, this is an argument against changing "int chunkno" to "bool is_first_call", as I alluded to above, because we have to keep the chunkno around anyway.

> I think this patch could possibly be split up into multiple patches.
> There's some question in my mind whether it's getting too late to
> commit any of this, since some of it looks suspiciously like new
> features after feature freeze. However, I kind of hate to ship this
> release without at least doing something about the chunkno vs.
> curchunk stuff, which is even worse in the committed code than in your
> patch, and which I think will confuse the heck out of users if those
> messages actually fire for anyone.

I'm in favor of cleaning up the committed code to have easier to understand output.  I don't really agree with any of your proposed changes to my patch, though, which is I think a first.

—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  138. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-04-22T23:28:12Z

    
> On Apr 19, 2021, at 5:07 PM, Mark Dilger <mark.dilger@enterprisedb.com> wrote:
> 
> 
> 
>> On Apr 19, 2021, at 12:50 PM, Robert Haas <robertmhaas@gmail.com> wrote:
>> 
>> On Thu, Apr 15, 2021 at 1:07 PM Mark Dilger
>> <mark.dilger@enterprisedb.com> wrote:
>>> I have added the verb "has" rather than "contains" because "has" is more consistent with the phrasing of other similar corruption reports.
>> 
>> That makes sense.

I have refactored the patch to address your other concerns.  Breaking the patch into multiple pieces didn't add any clarity, but refactoring portions of it made things simpler to read, I think, so here it is as one patch file.

  139. Re: pg_amcheck contrib application

    Robert Haas <robertmhaas@gmail.com> — 2021-04-23T17:28:50Z

    On Thu, Apr 22, 2021 at 7:28 PM Mark Dilger
<mark.dilger@enterprisedb.com> wrote:
> I have refactored the patch to address your other concerns.  Breaking the patch into multiple pieces didn't add any clarity, but refactoring portions of it made things simpler to read, I think, so here it is as one patch file.

I was hoping that this version was going to be smaller than the last
version, but instead it went from 300+ lines to 500+ lines.

The main thing I'm unhappy about in the status quo is the use of
chunkno in error messages. I have suggested several times making that
concept go away, because I think users will be confused. Here's a
minimal patch that does just that. It's 32 lines and results in a net
removal of 4 lines. It differs somewhat from my earlier suggestions,
because my priority here is to get reasonably understandable output
without needing a ton of code, and as I was working on this I found
that some of my earlier suggestions would have needed more code to
implement and I didn't think it bought enough to be worth it. It's
possible this is too simple, or that it's buggy, so let me know what
you think. But basically, I think what got committed before is
actually mostly fine and doesn't need major revision. It just needs
tidying up to avoid the confusing chunkno concept.

Now, the other thing we've talked about is adding a few more checks,
to verify for example that the toastrelid is what we expect, and I see
in your v22 you thought of a few other things. I think we can consider
those, possibly as things where we consider it tidying up loose ends
for v14, or else as improvements for v15. But I don't think that the
fairly large size of your patch comes primarily from additional
checks. I think it mostly comes from the code to produce error reports
getting a lot more complicated. I apologize if my comments have driven
that complexity, but they weren't intended to.

One tiny problem with the attached patch is that it does not make any
regression tests fail, which also makes it hard for me to tell if it
breaks anything, or if the existing code works. I don't know how
practical it is to do anything about that. Do you have a patch handy
that allows manual updates and deletes on TOAST tables, for manual
testing purposes?

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  140. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-04-23T17:31:16Z

    
> On Apr 23, 2021, at 10:28 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> 
> On Thu, Apr 22, 2021 at 7:28 PM Mark Dilger
> <mark.dilger@enterprisedb.com> wrote:
>> I have refactored the patch to address your other concerns.  Breaking the patch into multiple pieces didn't add any clarity, but refactoring portions of it made things simpler to read, I think, so here it is as one patch file.
> 
> I was hoping that this version was going to be smaller than the last
> version, but instead it went from 300+ lines to 500+ lines.
> 
> The main thing I'm unhappy about in the status quo is the use of
> chunkno in error messages. I have suggested several times making that
> concept go away, because I think users will be confused. Here's a
> minimal patch that does just that. It's 32 lines and results in a net
> removal of 4 lines. It differs somewhat from my earlier suggestions,
> because my priority here is to get reasonably understandable output
> without needing a ton of code, and as I was working on this I found
> that some of my earlier suggestions would have needed more code to
> implement and I didn't think it bought enough to be worth it. It's
> possible this is too simple, or that it's buggy, so let me know what
> you think. But basically, I think what got committed before is
> actually mostly fine and doesn't need major revision. It just needs
> tidying up to avoid the confusing chunkno concept.
> 
> Now, the other thing we've talked about is adding a few more checks,
> to verify for example that the toastrelid is what we expect, and I see
> in your v22 you thought of a few other things. I think we can consider
> those, possibly as things where we consider it tidying up loose ends
> for v14, or else as improvements for v15. But I don't think that the
> fairly large size of your patch comes primarily from additional
> checks. I think it mostly comes from the code to produce error reports
> getting a lot more complicated. I apologize if my comments have driven
> that complexity, but they weren't intended to.
> 
> One tiny problem with the attached patch is that it does not make any
> regression tests fail, which also makes it hard for me to tell if it
> breaks anything, or if the existing code works. I don't know how
> practical it is to do anything about that. Do you have a patch handy
> that allows manual updates and deletes on TOAST tables, for manual
> testing purposes?

Yes, I haven't been posting that with the patch because, but I will test your patch and see what differs.

—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  141. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-04-23T18:05:08Z

    
> On Apr 23, 2021, at 10:31 AM, Mark Dilger <mark.dilger@enterprisedb.com> wrote:
> 
> I will test your patch and see what differs.

Here are the differences between master and you patch:

UPDATE $toastname SET chunk_seq = chunk_seq + 1000 WHERE chunk_id = $value_id_to_corrupt

-                       qr/${header}toast value 16459 chunk 0 has sequence number 1000, but expected sequence number 0/,
-                       qr/${header}toast value 16459 chunk 1 has sequence number 1001, but expected sequence number 1/,
-                       qr/${header}toast value 16459 chunk 2 has sequence number 1002, but expected sequence number 2/,
-                       qr/${header}toast value 16459 chunk 3 has sequence number 1003, but expected sequence number 3/,
-                       qr/${header}toast value 16459 chunk 4 has sequence number 1004, but expected sequence number 4/,
-                       qr/${header}toast value 16459 chunk 5 has sequence number 1005, but expected sequence number 5/;
+               qr/${header}toast value 16459 index scan returned chunk 1000 when expecting chunk 0/,
+               qr/${header}toast value 16459 chunk 1000 follows last expected chunk 5/,
+               qr/${header}toast value 16459 chunk 1001 follows last expected chunk 5/,
+               qr/${header}toast value 16459 chunk 1002 follows last expected chunk 5/,
+               qr/${header}toast value 16459 chunk 1003 follows last expected chunk 5/,
+               qr/${header}toast value 16459 chunk 1004 follows last expected chunk 5/,
+               qr/${header}toast value 16459 chunk 1005 follows last expected chunk 5/;

UPDATE $toastname SET chunk_seq = chunk_seq * 1000 WHERE chunk_id = $value_id_to_corrupt

-                       qr/${header}toast value $value_id_to_corrupt chunk 1 has sequence number 1000, but expected sequence number 1/,
-                       qr/${header}toast value $value_id_to_corrupt chunk 2 has sequence number 2000, but expected sequence number 2/,
-                       qr/${header}toast value $value_id_to_corrupt chunk 3 has sequence number 3000, but expected sequence number 3/,
-                       qr/${header}toast value $value_id_to_corrupt chunk 4 has sequence number 4000, but expected sequence number 4/,
-                       qr/${header}toast value $value_id_to_corrupt chunk 5 has sequence number 5000, but expected sequence number 5/;
-
+               qr/${header}toast value 16460 index scan returned chunk 1000 when expecting chunk 1/,
+               qr/${header}toast value 16460 chunk 1000 follows last expected chunk 5/,
+               qr/${header}toast value 16460 index scan returned chunk 2000 when expecting chunk 1001/,
+               qr/${header}toast value 16460 chunk 2000 follows last expected chunk 5/,
+               qr/${header}toast value 16460 index scan returned chunk 3000 when expecting chunk 2001/,
+               qr/${header}toast value 16460 chunk 3000 follows last expected chunk 5/,
+               qr/${header}toast value 16460 index scan returned chunk 4000 when expecting chunk 3001/,
+               qr/${header}toast value 16460 chunk 4000 follows last expected chunk 5/,
+               qr/${header}toast value 16460 index scan returned chunk 5000 when expecting chunk 4001/,
+               qr/${header}toast value 16460 chunk 5000 follows last expected chunk 5/;

INSERT INTO $toastname (chunk_id, chunk_seq, chunk_data)
    (SELECT chunk_id,
            10*chunk_seq + 1000,
            chunk_data
        FROM $toastname
        WHERE chunk_id = $value_id_to_corrupt)

-                       qr/${header}toast value $value_id_to_corrupt chunk 6 has sequence number 1000, but expected sequence number 6/,
-                       qr/${header}toast value $value_id_to_corrupt chunk 7 has sequence number 1010, but expected sequence number 7/,
-                       qr/${header}toast value $value_id_to_corrupt chunk 8 has sequence number 1020, but expected sequence number 8/,
-                       qr/${header}toast value $value_id_to_corrupt chunk 9 has sequence number 1030, but expected sequence number 9/,
-                       qr/${header}toast value $value_id_to_corrupt chunk 10 has sequence number 1040, but expected sequence number 10/,
-                       qr/${header}toast value $value_id_to_corrupt chunk 11 has sequence number 1050, but expected sequence number 11/,
-                       qr/${header}toast value $value_id_to_corrupt was expected to end at chunk 6, but ended at chunk 12/;
+              qr/${header}toast value $value_id_to_corrupt index scan returned chunk 1000 when expecting chunk 6/,
+              qr/${header}toast value $value_id_to_corrupt chunk 1000 follows last expected chunk 5/,
+              qr/${header}toast value $value_id_to_corrupt index scan returned chunk 1010 when expecting chunk 1001/,
+              qr/${header}toast value $value_id_to_corrupt chunk 1010 follows last expected chunk 5/,
+              qr/${header}toast value $value_id_to_corrupt index scan returned chunk 1020 when expecting chunk 1011/,
+              qr/${header}toast value $value_id_to_corrupt chunk 1020 follows last expected chunk 5/,
+              qr/${header}toast value $value_id_to_corrupt index scan returned chunk 1030 when expecting chunk 1021/,
+              qr/${header}toast value $value_id_to_corrupt chunk 1030 follows last expected chunk 5/,
+              qr/${header}toast value $value_id_to_corrupt index scan returned chunk 1040 when expecting chunk 1031/,
+              qr/${header}toast value $value_id_to_corrupt chunk 1040 follows last expected chunk 5/,
+              qr/${header}toast value $value_id_to_corrupt index scan returned chunk 1050 when expecting chunk 1041/,
+              qr/${header}toast value $value_id_to_corrupt chunk 1050 follows last expected chunk 5/;


—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  142. Re: pg_amcheck contrib application

    Robert Haas <robertmhaas@gmail.com> — 2021-04-23T18:10:14Z

    On Fri, Apr 23, 2021 at 2:05 PM Mark Dilger
<mark.dilger@enterprisedb.com> wrote:
> Here are the differences between master and you patch:

Thanks. Those messages look reasonable to me.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  143. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-04-23T18:15:19Z

    
> On Apr 23, 2021, at 11:05 AM, Mark Dilger <mark.dilger@enterprisedb.com> wrote:
> 
> Here are the differences between master and you patch:

Another difference I should probably mention is that a bunch of unrelated tests are failing with errors like:

    toast value 13465 chunk 0 has size 1995, but expected size 1996

which leads me to suspect your changes to how the size is calculated.

—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  144. Re: pg_amcheck contrib application

    Robert Haas <robertmhaas@gmail.com> — 2021-04-23T18:29:08Z

    On Fri, Apr 23, 2021 at 2:15 PM Mark Dilger
<mark.dilger@enterprisedb.com> wrote:
> Another difference I should probably mention is that a bunch of unrelated tests are failing with errors like:
>
>     toast value 13465 chunk 0 has size 1995, but expected size 1996
>
> which leads me to suspect your changes to how the size is calculated.

That seems like a pretty reasonable suspicion, but I can't see the problem:

-       expected_size = curchunk < endchunk ? TOAST_MAX_CHUNK_SIZE
-               : VARATT_EXTERNAL_GET_EXTSIZE(ta->toast_pointer) -
(endchunk * TOAST_MAX_CHUNK_SIZE);
+       expected_size = chunk_seq < last_chunk_seq ? TOAST_MAX_CHUNK_SIZE
+               : extsize % TOAST_MAX_CHUNK_SIZE;

What's different?

1. The variables are renamed.

2. It uses a new variable extsize instead of recomputing
VARATT_EXTERNAL_GET_EXTSIZE(ta->toast_pointer), but I think that
should have the same value.

3. I used modulo arithmetic (%) instead of subtracting endchunk *
TOAST_MAX_CHUNK_SIZE.

Is TOAST_MAX_CHUNK_SIZE 1996? How long a value did you insert?

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  145. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-04-23T18:32:39Z

    
> On Apr 23, 2021, at 11:29 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> 
> On Fri, Apr 23, 2021 at 2:15 PM Mark Dilger
> <mark.dilger@enterprisedb.com> wrote:
>> Another difference I should probably mention is that a bunch of unrelated tests are failing with errors like:
>> 
>>    toast value 13465 chunk 0 has size 1995, but expected size 1996
>> 
>> which leads me to suspect your changes to how the size is calculated.
> 
> That seems like a pretty reasonable suspicion, but I can't see the problem:
> 
> -       expected_size = curchunk < endchunk ? TOAST_MAX_CHUNK_SIZE
> -               : VARATT_EXTERNAL_GET_EXTSIZE(ta->toast_pointer) -
> (endchunk * TOAST_MAX_CHUNK_SIZE);
> +       expected_size = chunk_seq < last_chunk_seq ? TOAST_MAX_CHUNK_SIZE
> +               : extsize % TOAST_MAX_CHUNK_SIZE;
> 
> What's different?
> 
> 1. The variables are renamed.
> 
> 2. It uses a new variable extsize instead of recomputing
> VARATT_EXTERNAL_GET_EXTSIZE(ta->toast_pointer), but I think that
> should have the same value.
> 
> 3. I used modulo arithmetic (%) instead of subtracting endchunk *
> TOAST_MAX_CHUNK_SIZE.
> 
> Is TOAST_MAX_CHUNK_SIZE 1996? How long a value did you insert?

On  my laptop, yes, 1996 is TOAST_MAX_CHUNK_SIZE.

I'm not inserting anything.  These failures come from just regular tests that I have not changed.  I just applied your patch and ran `make check-world` and these fail in src/bin/pg_amcheck 

—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  146. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-04-23T18:36:29Z

    
> On Apr 23, 2021, at 11:29 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> 
> +       expected_size = chunk_seq < last_chunk_seq ? TOAST_MAX_CHUNK_SIZE
> +               : extsize % TOAST_MAX_CHUNK_SIZE;
> 
> What's different?

for one thing, if a sequence of chunks happens to fit perfectly, the final chunk will have size TOAST_MAX_CHUNK_SIZE, but you're expecting no larger than one less than that, given how modulo arithmetic works.

—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  147. Re: pg_amcheck contrib application

    Robert Haas <robertmhaas@gmail.com> — 2021-04-23T20:31:36Z

    On Fri, Apr 23, 2021 at 2:36 PM Mark Dilger
<mark.dilger@enterprisedb.com> wrote:
> > What's different?
>
> for one thing, if a sequence of chunks happens to fit perfectly, the final chunk will have size TOAST_MAX_CHUNK_SIZE, but you're expecting no larger than one less than that, given how modulo arithmetic works.

Good point.

Perhaps something like this, closer to the way you had it?

       expected_size = chunk_seq < last_chunk_seq ? TOAST_MAX_CHUNK_SIZE
               : extsize - (last_chunk_seq * TOAST_MAX_CHUNK_SIZE);

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  148. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-04-23T22:01:32Z

    
> On Apr 23, 2021, at 1:31 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> 
> Perhaps something like this, closer to the way you had it?
> 
>       expected_size = chunk_seq < last_chunk_seq ? TOAST_MAX_CHUNK_SIZE
>               : extsize - (last_chunk_seq * TOAST_MAX_CHUNK_SIZE);

It still suffers the same failures.  I'll try to post something that accomplishes the changes to the reports that you are looking for.

—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  149. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-04-26T17:52:34Z

    
> On Apr 23, 2021, at 3:01 PM, Mark Dilger <mark.dilger@enterprisedb.com> wrote:
> 
> I'll try to post something that accomplishes the changes to the reports that you are looking for.

The attached patch changes amcheck corruption reports as discussed upthread.  This patch is submitted for the v14 development cycle as a bug fix, per your complaint that the committed code generates reports sufficiently confusing to a user as to constitute a bug.

All other code refactoring and additional checks discussed upthread are reserved for the v15 development cycle and are not included here.

The minimal patch (not attached) that does not rename any variables is 135 lines.  Your patch was 159 lines.  The patch (attached) which includes your variable renaming is 174 lines.

  150. Re: pg_amcheck contrib application

    Robert Haas <robertmhaas@gmail.com> — 2021-04-30T16:39:23Z

    On Mon, Apr 26, 2021 at 1:52 PM Mark Dilger
<mark.dilger@enterprisedb.com> wrote:
> The attached patch changes amcheck corruption reports as discussed upthread.  This patch is submitted for the v14 development cycle as a bug fix, per your complaint that the committed code generates reports sufficiently confusing to a user as to constitute a bug.
>
> All other code refactoring and additional checks discussed upthread are reserved for the v15 development cycle and are not included here.
>
> The minimal patch (not attached) that does not rename any variables is 135 lines.  Your patch was 159 lines.  The patch (attached) which includes your variable renaming is 174 lines.

Hi,

I have compared this against my version. I found the following differences:

1. This version passes last_chunk_seq rather than extsize to
check_toast_tuple(). But this results in having to call
VARATT_EXTERNAL_GET_EXTSIZE() inside that function. I thought it was
nicer to do that in the caller, so that we don't do it twice.

2. You fixed some out-of-date comments.

3. You move the test for an unexpected chunk sequence further down in
the function. I don't see the point; I had put it by the related null
check, and still think that's better. You also deleted my comment /*
Either the TOAST index is corrupt, or we don't have all chunks. */
which I would have preferred to keep.

4. You don't return if chunk_seq > last_chunk_seq. That seems wrong,
because we cannot compute a sensible expected size in that case. I
think your code will subtract a larger value from a smaller one and,
this being unsigned arithmetic, say that the expected chunk size is
something gigantic. Returning and not issuing that complaint at all
seems better.

5. You fixed the incorrect formula I had introduced for the expected
size of the last chunk.

6. You changed the variable name in check_toasted_attribute() from
expected_chunkno to chunkno, and initialized it later in the function
instead of at declaration time. I don't find this to be an
improvement; including the word "expected" seems to me to be
substantially clearer. But I think I should have gone with
expected_chunk_seq for better consistency.

7. You restored the message "toast value %u was expected to end at
chunk %d, but ended at chunk %d" which my version deleted. I deleted
that message because I thought it was redundant, but I guess it's not:
there's nothing else to complain if the sequence of chunks ends early.
I think we should change the test from != to < though, because if it's
>, then we must have already complained about unexpected chunks. Also,
I think the message is actually wrong, because even though you renamed
the variable, it still ends up being the expected next chunkno rather
than the last chunkno we actually saw.

PFA my counter-proposal based on the above analysis.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  151. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-04-30T18:31:04Z

    
> On Apr 30, 2021, at 9:39 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> 
> On Mon, Apr 26, 2021 at 1:52 PM Mark Dilger
> <mark.dilger@enterprisedb.com> wrote:
>> The attached patch changes amcheck corruption reports as discussed upthread.  This patch is submitted for the v14 development cycle as a bug fix, per your complaint that the committed code generates reports sufficiently confusing to a user as to constitute a bug.
>> 
>> All other code refactoring and additional checks discussed upthread are reserved for the v15 development cycle and are not included here.
>> 
>> The minimal patch (not attached) that does not rename any variables is 135 lines.  Your patch was 159 lines.  The patch (attached) which includes your variable renaming is 174 lines.
> 
> Hi,
> 
> I have compared this against my version. I found the following differences:

Just to be clear, I did not use your patch v1 as the starting point.  I took the code as committed to master as the starting point, used your corruption report verbiage changes and at least some of your variable naming choices, but did not use the rest, in large part because it didn't work.  It caused corruption messages to be reported against tables that have no corruption.  For that matter, your v2 patch doesn't work either, and in the same way.  To wit:

  heap table "postgres"."pg_catalog"."pg_rewrite", block 6, offset 4, attribute 7:
     toast value 13461 chunk 0 has size 1995, but expected size 1996

I think there is something wrong with the way you are trying to calculate and use extsize, because I'm not corrupting pg_catalog.pg_rewrite.  You can get these same results by applying your patch to master, building, and running 'make check' from src/bin/pg_amcheck/


> 1. This version passes last_chunk_seq rather than extsize to
> check_toast_tuple(). But this results in having to call
> VARATT_EXTERNAL_GET_EXTSIZE() inside that function. I thought it was
> nicer to do that in the caller, so that we don't do it twice.

I don't see that VARATT_EXTERNAL_GET_EXTSIZE() is worth too much concern, given that is just a struct access and a bit mask.  You are avoiding calculating that twice, but at the expense of calculating last_chunk_seq twice, which involves division.  I don't think the division can be optimized as a mere bit shift, since TOAST_MAX_CHUNK_SIZE is not in general a power of two.  (For example, on my laptop it is 1996.)

I don't say this to nitpick at the performance one way vs. the other.  I doubt it makes any real difference.  I'm just confused why you want to change this particular thing right now, given that it is not a bug.

> 2. You fixed some out-of-date comments.

Yes, because they were wrong.  That's on me.  I failed to update them in a prior patch.

> 3. You move the test for an unexpected chunk sequence further down in
> the function. I don't see the point;

Relative to your patch, perhaps.  Relative to master, no tests have been moved.

> I had put it by the related null
> check, and still think that's better. You also deleted my comment /*
> Either the TOAST index is corrupt, or we don't have all chunks. */
> which I would have preferred to keep.

That's fine.  I didn't mean to remove it.  I was just taking a minimalist approach to constructing the patch.

> 4. You don't return if chunk_seq > last_chunk_seq. That seems wrong,
> because we cannot compute a sensible expected size in that case. I
> think your code will subtract a larger value from a smaller one and,
> this being unsigned arithmetic, say that the expected chunk size is
> something gigantic.

Your conclusion is probably right, but I think your analysis is based on a misreading of what "last_chunk_seq" means.  It's not the last one seen, but the last one expected.  (Should we rename the variable to avoid confusion?)  It won't compute a gigantic size.  Rather, it will expect *every* chunk with chunk_seq >= last_chunk_seq to have whatever size is appropriate for the last chunk. 

> Returning and not issuing that complaint at all
> seems better.

That might be best.  I had been resisting that because I don't want the extraneous chunks to be reported without chunk size information.  When debugging corrupted toast, it may be interesting to know the size of the extraneous chunks.  If there are 1000 extra chunks, somebody might want to see the sizes of them.

> 5. You fixed the incorrect formula I had introduced for the expected
> size of the last chunk.

Not really.  I just didn't introduce any change in that area.

> 6. You changed the variable name in check_toasted_attribute() from
> expected_chunkno to chunkno, and initialized it later in the function
> instead of at declaration time. I don't find this to be an
> improvement;

I think I just left the variable name and its initialization unchanged.

> including the word "expected" seems to me to be
> substantially clearer. But I think I should have gone with
> expected_chunk_seq for better consistency.

I agree that is a better name.

> 7. You restored the message "toast value %u was expected to end at
> chunk %d, but ended at chunk %d" which my version deleted. I deleted
> that message because I thought it was redundant, but I guess it's not:
> there's nothing else to complain if the sequence of chunks ends early.
> I think we should change the test from != to < though, because if it's
>> , then we must have already complained about unexpected chunks.

We can do it that way if you like.  I considered that and had trouble deciding if that made things less clear to users who might be less familiar with the structure of toasted attributes.  If some of the attributes have that message and others don't, they might conclude that only some of the attributes ended at the wrong chunk and fail to make the inference that to you or me is obvious.

>> Also,
> I think the message is actually wrong, because even though you renamed
> the variable, it still ends up being the expected next chunkno rather
> than the last chunkno we actually saw.

If we have seen any chunks, the variable is holding the expected next chunk seq, which is one greater than the last chunk seq we saw.

If we expect chunks 0..3 and see chunk 0 but not chunk 1, it will complain ..."expected to end at chunk 4, but ended at chunk 1".  This is clearly by design and not merely a bug, though I tend to agree with you that this is a strange wording choice.  I can't remember exactly when and how we decided to word the message this way, but it has annoyed me for a while, and I assumed it was something you suggested a while back, because I don't recall doing it.  Either way, since you seem to also be bothered by this, I agree we should change it.

> PFA my counter-proposal based on the above analysis.




—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  152. Re: pg_amcheck contrib application

    Robert Haas <robertmhaas@gmail.com> — 2021-04-30T18:56:45Z

    On Fri, Apr 30, 2021 at 2:31 PM Mark Dilger
<mark.dilger@enterprisedb.com> wrote:
> Just to be clear, I did not use your patch v1 as the starting point.

I thought that might be the case, but I was trying to understand what
you didn't like about my version, and comparing them seemed like a way
to figure that out.

> I took the code as committed to master as the starting point, used your corruption report verbiage changes and at least some of your variable naming choices, but did not use the rest, in large part because it didn't work.  It caused corruption messages to be reported against tables that have no corruption.  For that matter, your v2 patch doesn't work either, and in the same way.  To wit:
>
>   heap table "postgres"."pg_catalog"."pg_rewrite", block 6, offset 4, attribute 7:
>      toast value 13461 chunk 0 has size 1995, but expected size 1996
>
> I think there is something wrong with the way you are trying to calculate and use extsize, because I'm not corrupting pg_catalog.pg_rewrite.  You can get these same results by applying your patch to master, building, and running 'make check' from src/bin/pg_amcheck/

Argh, OK, I didn't realize. Should be fixed in this version.

> > 4. You don't return if chunk_seq > last_chunk_seq. That seems wrong,
> > because we cannot compute a sensible expected size in that case. I
> > think your code will subtract a larger value from a smaller one and,
> > this being unsigned arithmetic, say that the expected chunk size is
> > something gigantic.
>
> Your conclusion is probably right, but I think your analysis is based on a misreading of what "last_chunk_seq" means.  It's not the last one seen, but the last one expected.  (Should we rename the variable to avoid confusion?)  It won't compute a gigantic size.  Rather, it will expect *every* chunk with chunk_seq >= last_chunk_seq to have whatever size is appropriate for the last chunk.

I realize it's the last one expected. That's the point: we don't have
any expectation for the sizes of chunks higher than the last one we
expected to see. If the value is 2000 bytes and the chunk size is 1996
bytes, we expect chunk 0 to be 1996 bytes and chunk 1 to be 4 bytes.
If not, we can complain. But it makes no sense to complain about chunk
2 being of a size we don't expect. We don't expect it to exist in the
first place, so we have no notion of what size it ought to be.

> If we have seen any chunks, the variable is holding the expected next chunk seq, which is one greater than the last chunk seq we saw.
>
> If we expect chunks 0..3 and see chunk 0 but not chunk 1, it will complain ..."expected to end at chunk 4, but ended at chunk 1".  This is clearly by design and not merely a bug, though I tend to agree with you that this is a strange wording choice.  I can't remember exactly when and how we decided to word the message this way, but it has annoyed me for a while, and I assumed it was something you suggested a while back, because I don't recall doing it.  Either way, since you seem to also be bothered by this, I agree we should change it.

Can you review this version?

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  153. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-04-30T19:21:09Z

    
> On Apr 30, 2021, at 11:56 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> 
> On Fri, Apr 30, 2021 at 2:31 PM Mark Dilger
> <mark.dilger@enterprisedb.com> wrote:
>> Just to be clear, I did not use your patch v1 as the starting point.
> 
> I thought that might be the case, but I was trying to understand what
> you didn't like about my version, and comparing them seemed like a way
> to figure that out.
> 
>> I took the code as committed to master as the starting point, used your corruption report verbiage changes and at least some of your variable naming choices, but did not use the rest, in large part because it didn't work.  It caused corruption messages to be reported against tables that have no corruption.  For that matter, your v2 patch doesn't work either, and in the same way.  To wit:
>> 
>>  heap table "postgres"."pg_catalog"."pg_rewrite", block 6, offset 4, attribute 7:
>>     toast value 13461 chunk 0 has size 1995, but expected size 1996
>> 
>> I think there is something wrong with the way you are trying to calculate and use extsize, because I'm not corrupting pg_catalog.pg_rewrite.  You can get these same results by applying your patch to master, building, and running 'make check' from src/bin/pg_amcheck/
> 
> Argh, OK, I didn't realize. Should be fixed in this version.
> 
>>> 4. You don't return if chunk_seq > last_chunk_seq. That seems wrong,
>>> because we cannot compute a sensible expected size in that case. I
>>> think your code will subtract a larger value from a smaller one and,
>>> this being unsigned arithmetic, say that the expected chunk size is
>>> something gigantic.
>> 
>> Your conclusion is probably right, but I think your analysis is based on a misreading of what "last_chunk_seq" means.  It's not the last one seen, but the last one expected.  (Should we rename the variable to avoid confusion?)  It won't compute a gigantic size.  Rather, it will expect *every* chunk with chunk_seq >= last_chunk_seq to have whatever size is appropriate for the last chunk.
> 
> I realize it's the last one expected. That's the point: we don't have
> any expectation for the sizes of chunks higher than the last one we
> expected to see. If the value is 2000 bytes and the chunk size is 1996
> bytes, we expect chunk 0 to be 1996 bytes and chunk 1 to be 4 bytes.
> If not, we can complain. But it makes no sense to complain about chunk
> 2 being of a size we don't expect. We don't expect it to exist in the
> first place, so we have no notion of what size it ought to be.
> 
>> If we have seen any chunks, the variable is holding the expected next chunk seq, which is one greater than the last chunk seq we saw.
>> 
>> If we expect chunks 0..3 and see chunk 0 but not chunk 1, it will complain ..."expected to end at chunk 4, but ended at chunk 1".  This is clearly by design and not merely a bug, though I tend to agree with you that this is a strange wording choice.  I can't remember exactly when and how we decided to word the message this way, but it has annoyed me for a while, and I assumed it was something you suggested a while back, because I don't recall doing it.  Either way, since you seem to also be bothered by this, I agree we should change it.
> 
> Can you review this version?
> 
> -- 
> Robert Haas
> EDB: http://www.enterprisedb.com
> <simply-remove-chunkno-concept-v3.patch>

As requested off-list, here are NOT FOR COMMIT, WIP patches for testing only.

The first patch allows toast tables to be updated and adds regression tests of corrupted toasted attributes.  I never quite got deletes from toast tables to work, and there are probably other gotchas still lurking even with inserts and updates, but it limps along well enough for testing pg_amcheck.

The second patch updates the expected output of pg_amcheck to match the verbiage that you suggested upthread.

  154. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-04-30T19:26:36Z

    
> On Apr 30, 2021, at 11:56 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> 
> Can you review this version?

It looks mostly good to me.  There is a off-by-one error introduced with:

-   else if (chunkno != (endchunk + 1))
+   else if (expected_chunk_seq < last_chunk_seq)

I think that needs to be

+  else if (expected_chunk_seq <= last_chunk_seq)

because otherwise it won't complain if the only missing chunk is the very last one.

—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  155. Re: pg_amcheck contrib application

    Robert Haas <robertmhaas@gmail.com> — 2021-04-30T19:29:40Z

    On Fri, Apr 30, 2021 at 3:26 PM Mark Dilger
<mark.dilger@enterprisedb.com> wrote:
> It looks mostly good to me.  There is a off-by-one error introduced with:
>
> -   else if (chunkno != (endchunk + 1))
> +   else if (expected_chunk_seq < last_chunk_seq)
>
> I think that needs to be
>
> +  else if (expected_chunk_seq <= last_chunk_seq)
>
> because otherwise it won't complain if the only missing chunk is the very last one.

OK, how about this version?

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  156. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-04-30T19:41:08Z

    
> On Apr 30, 2021, at 12:29 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> 
> OK, how about this version?

I think that's committable.

The only nitpick might be

-                               psprintf("toast value %u was expected to end at chunk %d, but ended at chunk %d",
+                               psprintf("toast value %u index scan ended early while expecting chunk %d of %d",

When reporting to users about positions within a zero-based indexing scheme, what does "while expecting chunk 3 of 4" mean?  Is it talking about the last chunk from the set [0..3] which has cardinality 4, or does it mean the next-to-last chunk from [0..4] which ends with chunk 4, or what?  The prior language isn't any more clear than what you have here, so I have no objection to committing this, but the prior language was probably as goofy as it was because it was trying to deal with this issue.

Thoughts?

—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  157. Re: pg_amcheck contrib application

    Robert Haas <robertmhaas@gmail.com> — 2021-04-30T19:47:39Z

    On Fri, Apr 30, 2021 at 3:41 PM Mark Dilger
<mark.dilger@enterprisedb.com> wrote:
> I think that's committable.
>
> The only nitpick might be
>
> -                               psprintf("toast value %u was expected to end at chunk %d, but ended at chunk %d",
> +                               psprintf("toast value %u index scan ended early while expecting chunk %d of %d",
>
> When reporting to users about positions within a zero-based indexing scheme, what does "while expecting chunk 3 of 4" mean?  Is it talking about the last chunk from the set [0..3] which has cardinality 4, or does it mean the next-to-last chunk from [0..4] which ends with chunk 4, or what?  The prior language isn't any more clear than what you have here, so I have no objection to committing this, but the prior language was probably as goofy as it was because it was trying to deal with this issue.

Hmm, I think that might need adjustment, actually. What I was trying
to do is compensate for the fact that what we now have is the next
chunk_seq value we expect, not the last one we saw, nor the total
number of chunks we've seen regardless of what chunk_seq they had. But
I thought it would be too confusing to just give the chunk number we
were expecting and not say anything about how many chunks we thought
there would be in total. So maybe what I should do is change it to
something like this:

toast value %u was expected to end at chunk %d, but ended while
expecting chunk %d

i.e. same as the currently-committed code, except for changing "ended
at" to "ended while expecting."

--
Robert Haas
EDB: http://www.enterprisedb.com

  158. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-04-30T20:04:24Z

    
> On Apr 30, 2021, at 12:47 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> 
> Hmm, I think that might need adjustment, actually. What I was trying
> to do is compensate for the fact that what we now have is the next
> chunk_seq value we expect, not the last one we saw, nor the total
> number of chunks we've seen regardless of what chunk_seq they had. But
> I thought it would be too confusing to just give the chunk number we
> were expecting and not say anything about how many chunks we thought
> there would be in total. So maybe what I should do is change it to
> something like this:
> 
> toast value %u was expected to end at chunk %d, but ended while
> expecting chunk %d
> 
> i.e. same as the currently-committed code, except for changing "ended
> at" to "ended while expecting."

I find the grammar of this new formulation anomalous for hard to articulate reasons not quite the same as but akin to mismatched verb aspect.

—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  159. Re: pg_amcheck contrib application

    Mark Dilger <mark.dilger@enterprisedb.com> — 2021-04-30T20:26:49Z

    
> On Apr 30, 2021, at 1:04 PM, Mark Dilger <mark.dilger@enterprisedb.com> wrote:
> 
>> toast value %u was expected to end at chunk %d, but ended while
>> expecting chunk %d
>> 
>> i.e. same as the currently-committed code, except for changing "ended
>> at" to "ended while expecting."
> 
> I find the grammar of this new formulation anomalous for hard to articulate reasons not quite the same as but akin to mismatched verb aspect.

After further reflection, no other verbiage seems any better.  I'd say go ahead and commit it this way.

—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  160. Re: pg_amcheck contrib application

    Robert Haas <robertmhaas@gmail.com> — 2021-04-30T21:07:51Z

    On Fri, Apr 30, 2021 at 4:26 PM Mark Dilger
<mark.dilger@enterprisedb.com> wrote:
> After further reflection, no other verbiage seems any better.  I'd say go ahead and commit it this way.

OK. I'll plan to do that on Monday, barring objections.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  161. Re: pg_amcheck contrib application

    Robert Haas <robertmhaas@gmail.com> — 2021-05-03T16:34:08Z

    On Fri, Apr 30, 2021 at 5:07 PM Robert Haas <robertmhaas@gmail.com> wrote:
> On Fri, Apr 30, 2021 at 4:26 PM Mark Dilger
> <mark.dilger@enterprisedb.com> wrote:
> > After further reflection, no other verbiage seems any better.  I'd say go ahead and commit it this way.
>
> OK. I'll plan to do that on Monday, barring objections.

Done now.

-- 
Robert Haas
EDB: http://www.enterprisedb.com