Horton

Thread

Commits

  1. Add INJECTION_POINT_CACHED() to run injection points directly from cache

  2. Support loading of injection points

  1. Injection points: preloading and runtime arguments

    Michael Paquier <michael@paquier.xyz> — 2024-05-20T03:18:43Z

    Hi all,

I have a couple of extra toys for injection points in my bucket that
I'd like to propose for integration in v18, based on some feedback I
have received:
1) Preload an injection point into the backend-level cache without
running it.  This has come up because an injection point run for the
first time needs to be loaded with load_external_function that
internally does some allocations, and this would not work if the
injection point is in a critical section.  Being able to preload an 
injection point requires an extra macro, called
INJECTION_POINT_PRELOAD.  Perhaps "load" makes more sense as keyword,
here.
2) Grab values at runtime from the code path where an injection point
is run and give them to the callback.  The case here is to be able to
do some dynamic manipulation or a stack, reads of some runtime data or
even decide of a different set of actions in a callback based on what
the input has provided.  One case that I've been playing with here is
the dynamic manipulation of pages in specific code paths to stress
internal checks, as one example.  This introduces a 1-argument
version, as multiple args could always be passed down to the callback
within a structure.

The in-core module injection_points is extended to provide a SQL
interface to be able to do the preloading or define a callback with
arguments.  The two changes are split into patches of their own.

These new facilities could be backpatched if there is a need for them
in the future in stable branches, as these are aimed for tests and the
changes do not introduce any ABI breakages with the existing APIs or
the in-core module.

Thoughts and comments are welcome.
--
Michael

  2. Re: Injection points: preloading and runtime arguments

    x4mmm@yandex-team.ru — 2024-05-20T12:01:15Z

    Hi!

> On 20 May 2024, at 08:18, Michael Paquier <michael@paquier.xyz> wrote:

Both features look useful to me.
I've tried to rebase my test of CV sleep during multixact generation[0]. I used it like this:

    INJECTION_POINT_PRELOAD("GetNewMultiXactId-done");
    multi = GetNewMultiXactId(nmembers, &offset); // starts critsection
    INJECTION_POINT("GetNewMultiXactId-done");

And it fails like this:

2024-05-20 16:50:40.430 +05 [21830] 001_multixact.pl LOG:  statement: select test_create_multixact();
TRAP: failed Assert("CritSectionCount == 0 || (context)->allowInCritSection"), File: "mcxt.c", Line: 1185, PID: 21830
0   postgres                            0x0000000101452ed0 ExceptionalCondition + 220
1   postgres                            0x00000001014a6050 MemoryContextAlloc + 208
2   postgres                            0x00000001011c3bf0 dsm_create_descriptor + 72
3   postgres                            0x00000001011c3ef4 dsm_attach + 400
4   postgres                            0x00000001014990d8 dsa_attach + 24
5   postgres                            0x00000001011c716c init_dsm_registry + 240
6   postgres                            0x00000001011c6e60 GetNamedDSMSegment + 456
7   injection_points.dylib              0x0000000101c871f8 injection_init_shmem + 60
8   injection_points.dylib              0x0000000101c86f1c injection_wait + 64
9   postgres                            0x000000010148e228 InjectionPointRunInternal + 376
10  postgres                            0x000000010148e0a4 InjectionPointRun + 32
11  postgres                            0x0000000100cab798 MultiXactIdCreateFromMembers + 344
12  postgres                            0x0000000100cab604 MultiXactIdCreate + 312

Am I doing something wrong? Seems like extension have to know too that it is preloaded.


Best regards, Andrey Borodin.

[0] https://www.postgresql.org/message-id/0925F9A9-4D53-4B27-A87E-3D83A757B0E0%40yandex-team.ru

  3. Re: Injection points: preloading and runtime arguments

    x4mmm@yandex-team.ru — 2024-05-20T12:03:18Z

    
> On 20 May 2024, at 17:01, Andrey M. Borodin <x4mmm@yandex-team.ru> wrote:

Ugh, accidentally send without attaching the patch itself. Sorry for the noise.


Best regards, Andrey Borodin.

  4. Re: Injection points: preloading and runtime arguments

    Michael Paquier <michael@paquier.xyz> — 2024-05-21T01:31:08Z

    On Mon, May 20, 2024 at 05:01:15PM +0500, Andrey M. Borodin wrote:
> Both features look useful to me.
> I've tried to rebase my test of CV sleep during multixact generation[0]. I used it like this:
> 
>     INJECTION_POINT_PRELOAD("GetNewMultiXactId-done");
>     multi = GetNewMultiXactId(nmembers, &offset); // starts critsection
>     INJECTION_POINT("GetNewMultiXactId-done");

Thanks for the feedback.

> And it fails like this:
> 
> 2024-05-20 16:50:40.430 +05 [21830] 001_multixact.pl LOG:  statement: select test_create_multixact();
> TRAP: failed Assert("CritSectionCount == 0 || (context)->allowInCritSection"), File: "mcxt.c", Line: 1185, PID: 21830
> 0   postgres                            0x0000000101452ed0 ExceptionalCondition + 220
> 1   postgres                            0x00000001014a6050 MemoryContextAlloc + 208
> 2   postgres                            0x00000001011c3bf0 dsm_create_descriptor + 72
> 3   postgres                            0x00000001011c3ef4 dsm_attach + 400
> 4   postgres                            0x00000001014990d8 dsa_attach + 24
> 5   postgres                            0x00000001011c716c init_dsm_registry + 240
> 6   postgres                            0x00000001011c6e60 GetNamedDSMSegment + 456
> 7   injection_points.dylib              0x0000000101c871f8 injection_init_shmem + 60
> 8   injection_points.dylib              0x0000000101c86f1c injection_wait + 64
> 9   postgres                            0x000000010148e228 InjectionPointRunInternal + 376
> 10  postgres                            0x000000010148e0a4 InjectionPointRun + 32
> 11  postgres                            0x0000000100cab798 MultiXactIdCreateFromMembers + 344
> 12  postgres                            0x0000000100cab604 MultiXactIdCreate + 312
> 
> Am I doing something wrong? Seems like extension have to know too that it is preloaded.

Your stack is pointing at the shared memory section initialized in the
module injection_points, which is a bit of a chicken-and-egg problem
because you'd want an extra preload to happen before even that, like a
pre-preload.  From what I can see, you have a good point about the
shmem initialized in the module: injection_points_preload() should
call injection_init_shmem() so as this area would not trigger the
assertion.

However, there is a second thing here inherent to your test: shouldn't
the script call injection_points_preload() to make sure that the local
cache behind GetNewMultiXactId-done is fully allocated and prepared
for the moment where injection point will be run?

So I agree that 0002 ought to call injection_init_shmem() when calling
injection_points_preload(), but it also seems to me that the test is
missing the fact that it should heat the backend cache to avoid the
allocations in the critical sections.

Note that I disagree with taking a shortcut in the backend-side
injection point code where we would bypass CritSectionCount or
allowInCritSection.  These states should stay consistent for the sake
of the callbacks registered so as these can rely on the same stack and
conditions as the code where they are called.
--
Michael

  5. Re: Injection points: preloading and runtime arguments

    x4mmm@yandex-team.ru — 2024-05-21T11:29:54Z

    
> On 21 May 2024, at 06:31, Michael Paquier <michael@paquier.xyz> wrote:
> 
> So I agree that 0002 ought to call injection_init_shmem() when calling
> injection_points_preload(), but it also seems to me that the test is
> missing the fact that it should heat the backend cache to avoid the
> allocations in the critical sections.
> 
> Note that I disagree with taking a shortcut in the backend-side
> injection point code where we would bypass CritSectionCount or
> allowInCritSection.  These states should stay consistent for the sake
> of the callbacks registered so as these can rely on the same stack and
> conditions as the code where they are called.

Currently I'm working on the test using this
$creator->query_until(qr/start/, q(
    \echo start
    select injection_points_wakeup('');
    select test_create_multixact();
));

I'm fine if instead of injection_points_wakeup('') I'll have to use select injection_points_preload('point name');.

Thanks!


Best regards, Andrey Borodin.

  6. Re: Injection points: preloading and runtime arguments

    Michael Paquier <michael@paquier.xyz> — 2024-06-04T22:52:32Z

    On Tue, May 21, 2024 at 04:29:54PM +0500, Andrey M. Borodin wrote:
> Currently I'm working on the test using this
> $creator->query_until(qr/start/, q(
>     \echo start
>     select injection_points_wakeup('');
>     select test_create_multixact();
> ));
> 
> I'm fine if instead of injection_points_wakeup('') I'll have to use
> select injection_points_preload('point name');.

Based on our discussion of last week, please find attached the
promised patch set to allow your SLRU tests to work.  I have reversed
the order of the patches, moving the loading part in 0001 and the
addition of the runtime arguments in 0002 as we have a use-case for
the loading, nothing yet for the runtime arguments.

I have also come back to the naming, feeling that "preload" was
overcomplicated.  So I have used the word "load" instead across the
board for 0001.

Note that the SQL function injection_points_load() does now an 
initialization of the shmem area when a process plugs into the module
for the first time, fixing the issue you have mentioned with your SLRU
test.  Hence, you should be able to do a load(), then a wait in the
critical section as there would be no memory allocation done when the
point runs.  Another thing you could do is to define a
INJECTION_POINT_LOAD() in the code path you're stressing outside the 
critical section where the point is run.  This should save from a call
to the SQL function.  This choice is up to the one implementing the
test, both can be useful depending on what one is trying to achieve.
--
Michael

  7. Re: Injection points: preloading and runtime arguments

    x4mmm@yandex-team.ru — 2024-06-06T10:47:47Z

    
> On 5 Jun 2024, at 03:52, Michael Paquier <michael@paquier.xyz> wrote:
> 
> Another thing you could do is to define a
> INJECTION_POINT_LOAD() in the code path you're stressing outside the 
> critical section where the point is run.  This should save from a call
> to the SQL function.  This choice is up to the one implementing the
> test, both can be useful depending on what one is trying to achieve.

Thanks!

Interestingly, previously having INJECTION_POINT_PRELOAD() was not enough.
But now both INJECTION_POINT_LOAD() or injection_points_load() do the trick, so for me any of them is enough.

My test works with current version, but I have one slight problem, I need to call
$node->safe_psql('postgres', q(select injection_points_detach('GetMultiXactIdMembers-CV-sleep')));
Before
$node->safe_psql('postgres', q(select injection_points_wakeup('GetMultiXactIdMembers-CV-sleep')));

Is it OK to detach() before wakeup()? Or, perhaps, can a detach() do a wakeup() automatically?


Best regards, Andrey Borodin.

  8. Re: Injection points: preloading and runtime arguments

    Michael Paquier <michael@paquier.xyz> — 2024-06-06T23:38:46Z

    On Thu, Jun 06, 2024 at 03:47:47PM +0500, Andrey M. Borodin wrote:
> Is it OK to detach() before wakeup()? Or, perhaps, can a detach() do a wakeup() automatically?

It is OK to do a detach before a wakeup.  Noah has been relying on
this behavior in an isolation test for a patch he's worked on.  See
inplace110-successors-v1.patch here:
https://www.postgresql.org/message-id/20240512232923.aa.nmisch@google.com

That's also something we've discussed for 33181b48fd0e, where Noah
wanted to emulate in an automated fashion what one can do with a
debugger and one or more breakpoints.

Not sure that wakeup() involving a automated detach() is the behavior
to hide long-term, actually, as there is also an argument for waking
up a point and *not* detach it to force multiple waits.
--
Michael

  9. Re: Injection points: preloading and runtime arguments

    x4mmm@yandex-team.ru — 2024-06-08T11:52:25Z

    
> On 7 Jun 2024, at 04:38, Michael Paquier <michael@paquier.xyz> wrote:

Thanks Michael! Tests of injection points with injection points are neat :)


Alvaro, here’s the test for multixact CV sleep that I was talking about on PGConf.
It is needed to test [0]. It is based on loaded injection points. This technique is not committed yet, but the patch looks good. When all prerequisites are ready I will post it to corresponding thread and create CF item.

Thanks!


Best regards, Andrey Borodin.

[0] https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=a0e0fb1ba

  10. Re: Injection points: preloading and runtime arguments

    Michael Paquier <michael@paquier.xyz> — 2024-06-10T06:10:33Z

    On Sat, Jun 08, 2024 at 04:52:25PM +0500, Andrey M. Borodin wrote:
> Alvaro, here’s the test for multixact CV sleep that I was talking
> about on PGConf.
> It is needed to test [0]. It is based on loaded injection
> points.

> This technique is not committed yet, but the patch looks good.

OK, cool.  I'll try to get that into the tree once v18 opens up.  I
can see that GetNewMultiXactId() opens a critical section.  I am
slightly surprised that you need both the SQL function
injection_points_load() and the macro INJECTION_POINT_LOAD().
Wouldn't one or the other be sufficient?

The test takes 20ms to run here, which is good enough.

+   INJECTION_POINT_LOAD("GetNewMultiXactId-done");
[...]
+   INJECTION_POINT("GetNewMultiXactId-done");
[...]
+   INJECTION_POINT("GetMultiXactIdMembers-CV-sleep"); 

Be careful about the naming here.  All the points use lower case
characters currently.

+# and another multixact have no offest yet, we must wait until this offset 

s/offest/offset/.

> When all prerequisites are ready I will post it to corresponding
> thread and create CF item.

OK, let's do that.
--
Michael

  11. Re: Injection points: preloading and runtime arguments

    Michael Paquier <michael@paquier.xyz> — 2024-07-05T09:16:18Z

    On Mon, Jun 10, 2024 at 03:10:33PM +0900, Michael Paquier wrote:
> OK, cool.  I'll try to get that into the tree once v18 opens up.

And I've spent more time on this one, and applied it to v18 after some
slight tweaks.  Please feel free to re-post your tests with
multixacts, Andrey.
--
Michael

  12. Re: Injection points: preloading and runtime arguments

    Heikki Linnakangas <hlinnaka@iki.fi> — 2024-07-09T09:08:26Z

    On 05/07/2024 12:16, Michael Paquier wrote:
> On Mon, Jun 10, 2024 at 03:10:33PM +0900, Michael Paquier wrote:
>> OK, cool.  I'll try to get that into the tree once v18 opens up.
> 
> And I've spent more time on this one, and applied it to v18 after some
> slight tweaks.

If you do:

INJECTION_POINT_LOAD(foo);

START_CRIT_SECTION();
INJECTION_POINT(foo);
END_CRIT_SECTION();

And the injection point is attached in between the 
INJECTION_POINT_LOAD() and INJECTION_POINT() calls, you will still get 
an assertion failure. For a testing facility, maybe that's acceptable, 
but it could be fixed pretty easily.

I propose we introduce an INJECTION_POINT_CACHED(name) macro that *only* 
uses the local cache. We could then also add an assertion in 
InjectionPointRun() to check that it's not used in a critical section, 
to enforce correct usage.

-- 
Heikki Linnakangas
Neon (https://neon.tech)

  13. Re: Injection points: preloading and runtime arguments

    Michael Paquier <michael@paquier.xyz> — 2024-07-10T04:16:15Z

    On Tue, Jul 09, 2024 at 12:08:26PM +0300, Heikki Linnakangas wrote:
> And the injection point is attached in between the INJECTION_POINT_LOAD()
> and INJECTION_POINT() calls, you will still get an assertion failure. For a
> testing facility, maybe that's acceptable, but it could be fixed pretty
> easily.
> 
> I propose we introduce an INJECTION_POINT_CACHED(name) macro that *only*
> uses the local cache.

You mean with something that does a injection_point_cache_get()
followed by a callback run if anything is found in the local cache?
Why not.  Based on what you have posted at [1], it looks like this had
better check the contents of the cache's generation with what's in
shmem, as well as destroying InjectionPointCache if there is nothing
else, so there's a possible dependency here depending on how much
maintenance this should do with the cache to keep it consistent.

> We could then also add an assertion in
> InjectionPointRun() to check that it's not used in a critical section, to
> enforce correct usage.

That would be the same as what we do currently with a palloc() coming
from load_external_function() or hash_create(), whichever comes first.
Okay, the stack reported is deeper in this case.

[1]: https://www.postgresql.org/message-id/87c748b3-0786-490f-a17a-51bef63e6c7f@iki.fi
--
Michael

  14. Re: Injection points: preloading and runtime arguments

    Michael Paquier <michael@paquier.xyz> — 2024-07-16T04:09:21Z

    On Wed, Jul 10, 2024 at 01:16:15PM +0900, Michael Paquier wrote:
> You mean with something that does a injection_point_cache_get()
> followed by a callback run if anything is found in the local cache?
> Why not.  Based on what you have posted at [1], it looks like this had
> better check the contents of the cache's generation with what's in
> shmem, as well as destroying InjectionPointCache if there is nothing
> else, so there's a possible dependency here depending on how much
> maintenance this should do with the cache to keep it consistent.

Now that 86db52a5062a is in the tree, this could be done with a
shortcut in InjectionPointCacheRefresh().  What do you think about
something like the attached, with your suggested naming?
--
Michael

  15. Re: Injection points: preloading and runtime arguments

    Heikki Linnakangas <hlinnaka@iki.fi> — 2024-07-16T08:20:57Z

    On 16/07/2024 07:09, Michael Paquier wrote:
> On Wed, Jul 10, 2024 at 01:16:15PM +0900, Michael Paquier wrote:
>> You mean with something that does a injection_point_cache_get()
>> followed by a callback run if anything is found in the local cache?
>> Why not.  Based on what you have posted at [1], it looks like this had
>> better check the contents of the cache's generation with what's in
>> shmem, as well as destroying InjectionPointCache if there is nothing
>> else, so there's a possible dependency here depending on how much
>> maintenance this should do with the cache to keep it consistent.
> 
> Now that 86db52a5062a is in the tree, this could be done with a
> shortcut in InjectionPointCacheRefresh().  What do you think about
> something like the attached, with your suggested naming?

Yes, +1 for something like that.

The "direct" argument to InjectionPointCacheRefresh() feels a bit weird. 
Also weird that it still checks ActiveInjectionPoints->max_inuse, even 
though it otherwise operates on the cached version only. I think you can 
just call injection_point_cache_get() directly from 
InjectionPointCached(), per attached.

I also rephrased the docs section a bit, focusing more on why and how 
you use the LOAD/CACHED pair, and less on the mechanics of how it works.

-- 
Heikki Linnakangas
Neon (https://neon.tech)

  16. Re: Injection points: preloading and runtime arguments

    Michael Paquier <michael@paquier.xyz> — 2024-07-17T02:19:41Z

    On Tue, Jul 16, 2024 at 11:20:57AM +0300, Heikki Linnakangas wrote:
> The "direct" argument to InjectionPointCacheRefresh() feels a bit weird.
> Also weird that it still checks ActiveInjectionPoints->max_inuse, even
> though it otherwise operates on the cached version only. I think you can
> just call injection_point_cache_get() directly from InjectionPointCached(),
> per attached.

My point was just to be more aggressive with the cache correctness
even in this context.  You've also mentioned upthread the point that 
we should worry about a concurrent detach, which is something that
injection_point_cache_get() alone is not able to do as we would not
cross-check the generation with what's in the shared area, so I also
saw a point about being more aggressive with the check here.

It works for me to do as you are proposing at the end, that could
always be changed if there are more arguments in favor of a different
behavior that plays more with the shmem data.

> I also rephrased the docs section a bit, focusing more on why and how you
> use the LOAD/CACHED pair, and less on the mechanics of how it works.

I'm OK with that, as well.  Thanks.
--
Michael

  17. Re: Injection points: preloading and runtime arguments

    Michael Paquier <michael@paquier.xyz> — 2024-07-18T00:55:27Z

    On Wed, Jul 17, 2024 at 11:19:41AM +0900, Michael Paquier wrote:
> It works for me to do as you are proposing at the end, that could
> always be changed if there are more arguments in favor of a different
> behavior that plays more with the shmem data.

I have taken some time this morning and applied that after a second
lookup.  Thanks!

If there is anything else you would like to see adjusted in this area,
please let me know.
--
Michael

  18. Re: Injection points: preloading and runtime arguments

    x4mmm@yandex-team.ru — 2024-08-02T16:03:58Z

    
> On 18 Jul 2024, at 03:55, Michael Paquier <michael@paquier.xyz> wrote:
> 
> If there is anything else you would like to see adjusted in this area,
> please let me know.

I’ve tried to switch my multixact test to new INJECTION_POINT_CACHED… and it does not work for me. Could you please take a look?

2024-08-02 18:52:32.244 MSK [53155] 001_multixact.pl LOG:  statement: select test_create_multixact();
TRAP: failed Assert("CritSectionCount == 0 || (context)->allowInCritSection"), File: "mcxt.c", Line: 1186, PID: 53155
0   postgres                            0x00000001031212f0 ExceptionalCondition + 236
1   postgres                            0x000000010317a01c MemoryContextAlloc + 240
2   postgres                            0x0000000102e66158 dsm_create_descriptor + 80
3   postgres                            0x0000000102e66474 dsm_attach + 416
4   postgres                            0x000000010316c264 dsa_attach + 24
5   postgres                            0x0000000102e69994 init_dsm_registry + 256
6   postgres                            0x0000000102e6965c GetNamedDSMSegment + 492
7   injection_points.dylib              0x000000010388f2cc injection_init_shmem + 68
8   injection_points.dylib              0x000000010388efbc injection_wait + 72
9   postgres                            0x00000001031606bc InjectionPointCached + 72
10  postgres                            0x00000001028ffc70 MultiXactIdCreateFromMembers + 360
11  postgres                            0x00000001028ffac8 MultiXactIdCreate + 344
12  test_slru.dylib                     0x000000010376fa04 test_create_multixact + 52


The test works fine with SQL interface “select injection_points_load('get-new-multixact-id');”.
Thanks!


Best regards, Andrey Borodin.

  19. Re: Injection points: preloading and runtime arguments

    Michael Paquier <michael@paquier.xyz> — 2024-08-03T16:02:22Z

    On Fri, Aug 02, 2024 at 07:03:58PM +0300, Andrey M. Borodin wrote:
> The test works fine with SQL interface “select
> injection_points_load('get-new-multixact-id');”.

Yes, just use a load() here to make sure that the DSM required by the
waits are properly initialized before entering in the critical section
where the wait of the point get-new-multixact-id happens.
--
Michael

  20. Re: Injection points: preloading and runtime arguments

    Michael Paquier <michael@paquier.xyz> — 2024-08-06T07:47:15Z

    On Sun, Aug 04, 2024 at 01:02:22AM +0900, Michael Paquier wrote:
> On Fri, Aug 02, 2024 at 07:03:58PM +0300, Andrey M. Borodin wrote:
> > The test works fine with SQL interface “select
> > injection_points_load('get-new-multixact-id');”.
> 
> Yes, just use a load() here to make sure that the DSM required by the
> waits are properly initialized before entering in the critical section
> where the wait of the point get-new-multixact-id happens.

Hmm.  How about loading injection_points with shared_preload_libraries
now that it has a _PG_init() thanks to 75534436a477 to take care of
the initialization you need here?  We could add two hooks to request
some shmem based on a size and to do the shmem initialization.
--
Michael

  21. Re: Injection points: preloading and runtime arguments

    x4mmm@yandex-team.ru — 2024-08-06T08:40:59Z

    
> On 6 Aug 2024, at 12:47, Michael Paquier <michael@paquier.xyz> wrote:
> 
> Hmm.  How about loading injection_points with shared_preload_libraries
> now that it has a _PG_init() thanks to 75534436a477 to take care of
> the initialization you need here?  We could add two hooks to request
> some shmem based on a size and to do the shmem initialization.

SQL initialisation is fine for test purposes. I just considered that I'd better share that doing the same from C code is non-trivial.

Thanks!


Best regards, Andrey Borodin.