Horton

Thread

Commits

  1. Adjust Valgrind macro usage to protect chunk headers

  2. Fix typo in memutils_memorychunk.h

  3. Avoid reference to nonexistent array element in ExecInitAgg().

  1. An oversight in ExecInitAgg for grouping sets

    Richard Guo <guofenglinux@gmail.com> — 2022-12-22T06:02:42Z

    I happened to notice $subject.  It happens when we build eqfunctions for
each grouping set.

     /* for each grouping set */
     for (int k = 0; k < phasedata->numsets; k++)
     {
         int         length = phasedata->gset_lengths[k];

         if (phasedata->eqfunctions[length - 1] != NULL)
             continue;

         phasedata->eqfunctions[length - 1] =
             execTuplesMatchPrepare(scanDesc,
                                    length,
                                    aggnode->grpColIdx,
                                    aggnode->grpOperators,
                                    aggnode->grpCollations,
                                    (PlanState *) aggstate);
     }

If it is an empty grouping set, its length will be zero, and accessing
phasedata->eqfunctions[length - 1] is not right.

I think we can just skip building the eqfunctions for empty grouping
set.

--- a/src/backend/executor/nodeAgg.c
+++ b/src/backend/executor/nodeAgg.c
@@ -3494,6 +3494,10 @@ ExecInitAgg(Agg *node, EState *estate, int eflags)
                {
                    int         length = phasedata->gset_lengths[k];

+                   /* skip empty grouping set */
+                   if (length == 0)
+                       continue;
+
                    if (phasedata->eqfunctions[length - 1] != NULL)
                        continue;

Thanks
Richard

  2. Re: An oversight in ExecInitAgg for grouping sets

    Richard Guo <guofenglinux@gmail.com> — 2022-12-27T07:12:17Z

    On Thu, Dec 22, 2022 at 2:02 PM Richard Guo <guofenglinux@gmail.com> wrote:

> I happened to notice $subject.  It happens when we build eqfunctions for
> each grouping set.
>
>      /* for each grouping set */
>      for (int k = 0; k < phasedata->numsets; k++)
>      {
>          int         length = phasedata->gset_lengths[k];
>
>          if (phasedata->eqfunctions[length - 1] != NULL)
>              continue;
>
>          phasedata->eqfunctions[length - 1] =
>              execTuplesMatchPrepare(scanDesc,
>                                     length,
>                                     aggnode->grpColIdx,
>                                     aggnode->grpOperators,
>                                     aggnode->grpCollations,
>                                     (PlanState *) aggstate);
>      }
>
> If it is an empty grouping set, its length will be zero, and accessing
> phasedata->eqfunctions[length - 1] is not right.
>
> I think we can just skip building the eqfunctions for empty grouping
> set.
>

Attached is a trivial patch for the fix.

Thanks
Richard

  3. Re: An oversight in ExecInitAgg for grouping sets

    Tom Lane <tgl@sss.pgh.pa.us> — 2023-01-02T21:25:19Z

    Richard Guo <guofenglinux@gmail.com> writes:
> On Thu, Dec 22, 2022 at 2:02 PM Richard Guo <guofenglinux@gmail.com> wrote:
>> If it is an empty grouping set, its length will be zero, and accessing
>> phasedata->eqfunctions[length - 1] is not right.

> Attached is a trivial patch for the fix.

Agreed, that's a latent bug.  It's only latent because the word just
before a palloc chunk will never be zero, either in our historical
palloc code or in v16's shiny new implementation.  Nonetheless it
is a horrible idea for ExecInitAgg to depend on that fact, so I
pushed your fix.

The thing that I find really distressing here is that it's been
like this for years and none of our automated testing caught it.
You'd have expected valgrind testing to do so ... but it does not,
because we've never marked that word NOACCESS.  Maybe we should
rethink that?  It'd require making mcxt.c do some valgrind flag
manipulations so it could access the hdrmask when appropriate.

			regards, tom lane

  4. Re: An oversight in ExecInitAgg for grouping sets

    Richard Guo <guofenglinux@gmail.com> — 2023-01-03T09:20:14Z

    On Tue, Jan 3, 2023 at 5:25 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:

> Agreed, that's a latent bug.  It's only latent because the word just
> before a palloc chunk will never be zero, either in our historical
> palloc code or in v16's shiny new implementation.  Nonetheless it
> is a horrible idea for ExecInitAgg to depend on that fact, so I
> pushed your fix.


Thanks for pushing it!


> The thing that I find really distressing here is that it's been
> like this for years and none of our automated testing caught it.
> You'd have expected valgrind testing to do so ... but it does not,
> because we've never marked that word NOACCESS.  Maybe we should
> rethink that?  It'd require making mcxt.c do some valgrind flag
> manipulations so it could access the hdrmask when appropriate.


Yeah, maybe we can do that.  It's true that it requires some additional
work to access hdrmask, as in the new implementation the palloc'd chunk
is always prefixed by hdrmask.

BTW, I noticed a typo in the comment of memorychunk.h.

--- a/src/include/utils/memutils_memorychunk.h
+++ b/src/include/utils/memutils_memorychunk.h
@@ -5,9 +5,9 @@
  *       MemoryContexts may use as a header for chunks of memory they
allocate.
  *
  * MemoryChunk provides a lightweight header that a MemoryContext can use
to
- * store a reference back to the block the which the given chunk is
allocated
- * on and also an additional 30-bits to store another value such as the
size
- * of the allocated chunk.
+ * store a reference back to the block which the given chunk is allocated
on
+ * and also an additional 30-bits to store another value such as the size
of
+ * the allocated chunk.

Thanks
Richard

  5. Re: An oversight in ExecInitAgg for grouping sets

    David Rowley <dgrowleyml@gmail.com> — 2023-01-04T22:17:48Z

    On Tue, 3 Jan 2023 at 10:25, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> The thing that I find really distressing here is that it's been
> like this for years and none of our automated testing caught it.
> You'd have expected valgrind testing to do so ... but it does not,
> because we've never marked that word NOACCESS.  Maybe we should
> rethink that?  It'd require making mcxt.c do some valgrind flag
> manipulations so it could access the hdrmask when appropriate.

Yeah, that probably could have been improved during the recent change.
Here's a patch for it.

I'm just doing a final Valgrind run on it now to check for errors.

David

  6. Re: An oversight in ExecInitAgg for grouping sets

    Richard Guo <guofenglinux@gmail.com> — 2023-01-05T07:06:21Z

    On Thu, Jan 5, 2023 at 6:18 AM David Rowley <dgrowleyml@gmail.com> wrote:

> On Tue, 3 Jan 2023 at 10:25, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> > The thing that I find really distressing here is that it's been
> > like this for years and none of our automated testing caught it.
> > You'd have expected valgrind testing to do so ... but it does not,
> > because we've never marked that word NOACCESS.  Maybe we should
> > rethink that?  It'd require making mcxt.c do some valgrind flag
> > manipulations so it could access the hdrmask when appropriate.
>
> Yeah, that probably could have been improved during the recent change.
> Here's a patch for it.


Thanks for the patch.  With it Valgrind is able to catch the invalid
read discussed in the initial email of this thread.

 VALGRINDERROR-BEGIN
 Invalid read of size 8
    at 0x4DB056: ExecInitAgg
    by 0x4C486A: ExecInitNode
    by 0x4B92B7: InitPlan
    by 0x4B81D7: standard_ExecutorStart
    by 0x4B7F1B: ExecutorStart

I reviewed this patch and have some comments.

It seems that for MemoryContextMethods in alignedalloc.c the access to
the chunk header is not wrapped by VALGRIND_MAKE_MEM_DEFINED and
VALGRIND_MAKE_MEM_NOACCESS.  Should we do that?

In GenerationFree, I think the VALGRIND_MAKE_MEM_DEFINED should be moved
to the start of this function, before we call MemoryChunkIsExternal.

In SlabFree, we should call MemoryChunkGetBlock after the call of
VALGRIND_MAKE_MEM_DEFINED, just like how we do in SlabRealloc.

In AllocSetStats, we have a call of MemoryChunkGetValue in Assert.  I
think we should wrap it with VALGRIND_MAKE_MEM_DEFINED and
VALGRIND_MAKE_MEM_NOACCESS.

Thanks
Richard

  7. Re: An oversight in ExecInitAgg for grouping sets

    David Rowley <dgrowleyml@gmail.com> — 2023-01-09T09:21:05Z

    On Thu, 5 Jan 2023 at 20:06, Richard Guo <guofenglinux@gmail.com> wrote:
> I reviewed this patch and have some comments.

Thanks for looking at this. I think I've fixed all the issues you mentioned.

One extra thing I noticed was that I had to add a new
VALGRIND_MAKE_MEM_DEFINED in AllocSetAlloc when grabbing an item off
the freelist. I didn't quite manage to figure out why that's needed as
when we do AllocSetFree() we don't mark the pfree'd memory with
NOACCESS, and it also looks like AllocSetReset() sets the keeper
block's memory to NOACCESS, but that function also clears the
freelists too, so the freelist chunk is not coming from a recently
reset context.

I might need to spend a bit more time on this to see if I can figure
out why this is happening.  On the other hand, maybe we should just
mark pfree'd memory as NOACCESS as that might find another class of
issues.

David

  8. Re: An oversight in ExecInitAgg for grouping sets

    Richard Guo <guofenglinux@gmail.com> — 2023-03-20T06:18:15Z

    On Mon, Jan 9, 2023 at 5:21 PM David Rowley <dgrowleyml@gmail.com> wrote:

> On Thu, 5 Jan 2023 at 20:06, Richard Guo <guofenglinux@gmail.com> wrote:
> > I reviewed this patch and have some comments.
>
> Thanks for looking at this. I think I've fixed all the issues you
> mentioned.
>
> One extra thing I noticed was that I had to add a new
> VALGRIND_MAKE_MEM_DEFINED in AllocSetAlloc when grabbing an item off
> the freelist. I didn't quite manage to figure out why that's needed as
> when we do AllocSetFree() we don't mark the pfree'd memory with
> NOACCESS, and it also looks like AllocSetReset() sets the keeper
> block's memory to NOACCESS, but that function also clears the
> freelists too, so the freelist chunk is not coming from a recently
> reset context.
>
> I might need to spend a bit more time on this to see if I can figure
> out why this is happening.  On the other hand, maybe we should just
> mark pfree'd memory as NOACCESS as that might find another class of
> issues.


It occurred to me that this hasn't been applied.  Should we add it to
the CF to not lose track of it?

Thanks
Richard

  9. Re: An oversight in ExecInitAgg for grouping sets

    David Rowley <dgrowleyml@gmail.com> — 2023-03-22T08:36:36Z

    On Mon, 20 Mar 2023 at 19:18, Richard Guo <guofenglinux@gmail.com> wrote:
> It occurred to me that this hasn't been applied.  Should we add it to
> the CF to not lose track of it?

I have a git branch with it. That'll work for me personally as a
reminder to come back to it during the v17 cycle.

David

  10. Re: An oversight in ExecInitAgg for grouping sets

    David Rowley <dgrowleyml@gmail.com> — 2023-04-11T13:17:55Z

    On Mon, 9 Jan 2023 at 22:21, David Rowley <dgrowleyml@gmail.com> wrote:
> One extra thing I noticed was that I had to add a new
> VALGRIND_MAKE_MEM_DEFINED in AllocSetAlloc when grabbing an item off
> the freelist. I didn't quite manage to figure out why that's needed as
> when we do AllocSetFree() we don't mark the pfree'd memory with
> NOACCESS, and it also looks like AllocSetReset() sets the keeper
> block's memory to NOACCESS, but that function also clears the
> freelists too, so the freelist chunk is not coming from a recently
> reset context.

It seems I didn't look hard enough for NOACCESS marking. It's in
wipe_mem(). So that explains why the VALGRIND_MAKE_MEM_DEFINED is
required in AllocSetAlloc.

Since this patch really only touches Valgrind macros, I don't really
feel like there's a good reason we can't still do this for v16, but
I'll start another thread to increase visibility to see if anyone else
thinks differently about that.

David