Horton

Thread

Commits

  1. Optimize iteration over PGPROC for fast-path lock searches.

  2. Increase the number of fast-path lock slots

  1. Optimizing FastPathTransferRelationLocks()

    Fujii Masao <masao.fujii@oss.nttdata.com> — 2024-11-12T01:16:37Z

    Hi,

I've identified some opportunities to optimize FastPathTransferRelationLocks(),
which transfers locks with a specific lock tag from per-backend fast-path arrays
to the shared hash table. The attached patch includes these enhancements.

Currently, FastPathTransferRelationLocks() recalculates the fast-path group on
each loop iteration, even though it stays the same. This patch updates
the function to calculate the group once and reuse it, improving efficiency.

The patch also extends the function's logic to skip not only backends from
a different database but also backends with pid=0 (which don’t hold fast-path
locks) and groups with no registered fast-path locks.

Since MyProc->pid is reset to 0 when a backend exits but MyProc->databaseId
remains set, checking only databaseId isn’t enough. Backends with pid=0 also
should be skipped.

Regards,

-- 
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION

  2. Re: Optimizing FastPathTransferRelationLocks()

    Heikki Linnakangas <hlinnaka@iki.fi> — 2024-11-15T22:15:45Z

    On 12/11/2024 03:16, Fujii Masao wrote:
> Hi,
> 
> I've identified some opportunities to optimize 
> FastPathTransferRelationLocks(), which transfers locks with a
> specific lock tag from per-backend fast- path arrays to the shared
> hash table. The attached patch includes these enhancements.
> 
> Currently, FastPathTransferRelationLocks() recalculates the fast-
> path group on each loop iteration, even though it stays the same.
> This patch updates the function to calculate the group once and
> reuse it, improving efficiency.

Makes sense. GetLockConflicts() has similar code, the same optimizations 
would apply there too.

> The patch also extends the function's logic to skip not only
> backends from a different database but also backends with pid=0
> (which don’t hold fast-path locks) and groups with no registered
> fast-path locks.
> 
> Since MyProc->pid is reset to 0 when a backend exits but MyProc-
> >databaseId remains set, checking only databaseId isn’t enough.
> Backends with pid=0 also should be skipped.
Hmm, a PGPROC entry that's not in use would also have 
proc->fpLockBits[group] == 0, so I'm not sure if the check for proc->pid 
== 0 is necessary.

And perhaps we should start clearing databaseid on backend exit.

-- 
Heikki Linnakangas
Neon (https://neon.tech)

  3. Re: Optimizing FastPathTransferRelationLocks()

    Fujii Masao <masao.fujii@oss.nttdata.com> — 2024-11-19T16:43:53Z

    
On 2024/11/16 7:15, Heikki Linnakangas wrote:
> On 12/11/2024 03:16, Fujii Masao wrote:
>> Hi,
>>
>> I've identified some opportunities to optimize FastPathTransferRelationLocks(), which transfers locks with a
>> specific lock tag from per-backend fast- path arrays to the shared
>> hash table. The attached patch includes these enhancements.
>>
>> Currently, FastPathTransferRelationLocks() recalculates the fast-
>> path group on each loop iteration, even though it stays the same.
>> This patch updates the function to calculate the group once and
>> reuse it, improving efficiency.
> 
> Makes sense. GetLockConflicts() has similar code, the same optimizations would apply there too.

Yes, I've updated the patch as suggested.


>> The patch also extends the function's logic to skip not only
>> backends from a different database but also backends with pid=0
>> (which don’t hold fast-path locks) and groups with no registered
>> fast-path locks.
>>
>> Since MyProc->pid is reset to 0 when a backend exits but MyProc-
>> >databaseId remains set, checking only databaseId isn’t enough.
>> Backends with pid=0 also should be skipped.
> Hmm, a PGPROC entry that's not in use would also have proc->fpLockBits[group] == 0, so I'm not sure if the check for proc->pid == 0 is necessary.

You're right. I've removed the check for proc->pid == 0.

Also I added a check for proc->fpLockBits[group] == 0 in GetLockConflicts().

The latest version of the patch is attached.


> And perhaps we should start clearing databaseid on backend exit.

Maybe, but I'm not sure if we really need this..

Regards,

-- 
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION

  4. Re: Optimizing FastPathTransferRelationLocks()

    Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> — 2025-03-11T11:55:56Z

    Hi Fujii-san,

It seems that this was forgotten somehow.

The patch still applies.

Examining c4d5cb71d229095a39fda1121a75ee40e6069a2a, it seems that this patch
could have been part of that commit as well. But may be it wasn't so apparent
that time. I think it's a good improvement.

On Tue, Nov 19, 2024 at 10:14 PM Fujii Masao
<masao.fujii@oss.nttdata.com> wrote:
>
> The latest version of the patch is attached.

@@ -2773,6 +2773,10 @@ FastPathTransferRelationLocks(LockMethod
lockMethodTable, const LOCKTAG *locktag
LWLock *partitionLock = LockHashPartitionLock(hashcode);
Oid relid = locktag->locktag_field2;
uint32 i;
+ uint32 group;
+
+ /* fast-path group the lock belongs to */
+ group = FAST_PATH_REL_GROUP(relid);

We could just combine variable declaration and initialization; similar
to partitionLock.

@@ -2802,16 +2805,16 @@ FastPathTransferRelationLocks(LockMethod
lockMethodTable, const LOCKTAG *locktag
* less clear that our backend is certain to have performed a memory
* fencing operation since the other backend set proc->databaseId. So
* for now, we test it after acquiring the LWLock just to be safe.
+ *
+ * Also skip groups without any registered fast-path locks.
*/
- if (proc->databaseId != locktag->locktag_field1)
+ if (proc->databaseId != locktag->locktag_field1 ||
+ proc->fpLockBits[group] == 0)

I like this. Moving the group computation out of the loop also allows
the computed group to be used for checking existence of lock. That's
double benefit. This test is similar to the test in
GetLockStatusData(), so we already have a precedence.

>
>
> > And perhaps we should start clearing databaseid on backend exit.
>
> Maybe, but I'm not sure if we really need this..

There's a comment
* proc->databaseId is set at backend startup time and never changes
* thereafter, so it might be safe to perform this test before
* acquiring &proc->fpInfoLock.

that seems to assume that databaseid is never cleared, so maybe it's
not safe to do this.

The performance gain from this patch might be tiny and may not be
visible. Still, have you tried to measure the performance improvement?

-- 
Best Wishes,
Ashutosh Bapat

  5. Re: Optimizing FastPathTransferRelationLocks()

    Fujii Masao <masao.fujii@oss.nttdata.com> — 2025-03-11T15:16:34Z

    
On 2025/03/11 20:55, Ashutosh Bapat wrote:
> Hi Fujii-san,
> 
> It seems that this was forgotten somehow.
> 
> The patch still applies.
> 
> Examining c4d5cb71d229095a39fda1121a75ee40e6069a2a, it seems that this patch
> could have been part of that commit as well. But may be it wasn't so apparent
> that time. I think it's a good improvement.

Thanks for reviewing the patch!


> On Tue, Nov 19, 2024 at 10:14 PM Fujii Masao
> <masao.fujii@oss.nttdata.com> wrote:
>>
>> The latest version of the patch is attached.
> 
> @@ -2773,6 +2773,10 @@ FastPathTransferRelationLocks(LockMethod
> lockMethodTable, const LOCKTAG *locktag
> LWLock *partitionLock = LockHashPartitionLock(hashcode);
> Oid relid = locktag->locktag_field2;
> uint32 i;
> + uint32 group;
> +
> + /* fast-path group the lock belongs to */
> + group = FAST_PATH_REL_GROUP(relid);
> 
> We could just combine variable declaration and initialization; similar
> to partitionLock.

I’ve updated the patch as suggested. Updated patch is attached.


> The performance gain from this patch might be tiny and may not be
> visible. Still, have you tried to measure the performance improvement?

I haven’t measured the actual performance gain since the patch optimizes
the logic in a clear and logical way. While we might see some improvement
in artificial scenarios — like with a large max_connections and
all backends slots having their databaseIds set, I’m not sure
how meaningful that would be.

Regards,

-- 
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION

  6. Re: Optimizing FastPathTransferRelationLocks()

    Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> — 2025-03-12T15:32:29Z

    On Tue, Mar 11, 2025 at 8:46 PM Fujii Masao <masao.fujii@oss.nttdata.com> wrote:
>
>
>
> On 2025/03/11 20:55, Ashutosh Bapat wrote:
> > Hi Fujii-san,
> >
> > It seems that this was forgotten somehow.
> >
> > The patch still applies.
> >
> > Examining c4d5cb71d229095a39fda1121a75ee40e6069a2a, it seems that this patch
> > could have been part of that commit as well. But may be it wasn't so apparent
> > that time. I think it's a good improvement.
>
> Thanks for reviewing the patch!
>
>
> > On Tue, Nov 19, 2024 at 10:14 PM Fujii Masao
> > <masao.fujii@oss.nttdata.com> wrote:
> >>
> >> The latest version of the patch is attached.
> >
> > @@ -2773,6 +2773,10 @@ FastPathTransferRelationLocks(LockMethod
> > lockMethodTable, const LOCKTAG *locktag
> > LWLock *partitionLock = LockHashPartitionLock(hashcode);
> > Oid relid = locktag->locktag_field2;
> > uint32 i;
> > + uint32 group;
> > +
> > + /* fast-path group the lock belongs to */
> > + group = FAST_PATH_REL_GROUP(relid);
> >
> > We could just combine variable declaration and initialization; similar
> > to partitionLock.
>
> I’ve updated the patch as suggested. Updated patch is attached.
>

Thanks.

>
> > The performance gain from this patch might be tiny and may not be
> > visible. Still, have you tried to measure the performance improvement?
>
> I haven’t measured the actual performance gain since the patch optimizes
> the logic in a clear and logical way. While we might see some improvement
> in artificial scenarios — like with a large max_connections and
> all backends slots having their databaseIds set, I’m not sure
> how meaningful that would be.

Fair enough. The code is more readable this way. That itself is an improvement.

I stared at c4d5cb71d229095a39fda1121a75ee40e6069a2a to see whether
there's any reason this was left aside at that time. I am convinced
that it was just missed. I think this patch is good to be committed.

-- 
Best Wishes,
Ashutosh Bapat

  7. Re: Optimizing FastPathTransferRelationLocks()

    Fujii Masao <masao.fujii@oss.nttdata.com> — 2025-03-14T13:55:53Z

    
On 2025/03/13 0:32, Ashutosh Bapat wrote:
> Fair enough. The code is more readable this way. That itself is an improvement.
> 
> I stared at c4d5cb71d229095a39fda1121a75ee40e6069a2a to see whether
> there's any reason this was left aside at that time. I am convinced
> that it was just missed. I think this patch is good to be committed.

Thanks for reviewing the patch and confirming! I've pushed the patch.

Regards,

-- 
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION