Horton
Menu

Thread

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Sort guc_parameters.dat alphabetically by name

  1. Adding basic NUMA awareness

    Tomas Vondra <tomas@vondra.me> — 2025-07-01T19:07:00Z

    Hi,

This is a WIP version of a patch series I'm working on, adding some
basic NUMA awareness for a couple parts of our shared memory (shared
buffers, etc.). It's based on Andres' experimental patches he spoke
about at pgconf.eu 2024 [1], and while it's improved and polished in
various ways, it's still experimental.

But there's a recent thread aiming to do something similar [2], so
better to share it now so that we can discuss both approaches. This
patch set is a bit more ambitious, handling NUMA in a way to allow
smarter optimizations later, so I'm posting it in a separate thread.

The series is split into patches addressing different parts of the
shared memory, starting (unsurprisingly) from shared buffers, then
buffer freelists and ProcArray. There's a couple additional parts, but
those are smaller / addressing miscellaneous stuff.

Each patch has a numa_ GUC, intended to enable/disable that part. This
is meant to make development easier, not as a final interface. I'm not
sure how exactly that should look. It's possible some combinations of
GUCs won't work, etc.

Each patch should have a commit message explaining the intent and
implementation, and then also detailed comments explaining various
challenges and open questions.

But let me go over the basics, and discuss some of the design choices
and open questions that need solving.


1) v1-0001-NUMA-interleaving-buffers.patch

This is the main thing when people think about NUMA - making sure the
shared buffers are allocated evenly on all the nodes, not just on a
single node (which can happen easily with warmup). The regular memory
interleaving would address this, but it also has some disadvantages.

Firstly, it's oblivious to the contents of the shared memory segment,
and we may not want to interleave everything. It's also oblivious to
alignment of the items (a buffer can easily end up "split" on multiple
NUMA nodes), or relationship between different parts (e.g. there's a
BufferBlock and a related BufferDescriptor, and those might again end up
on different nodes).

So the patch handles this by explicitly mapping chunks of shared buffers
to different nodes - a bit like interleaving, but in larger chunks.
Ideally each node gets (1/N) of shared buffers, as a contiguous chunk.

It's a bit more complicated, because the patch distributes both the
blocks and descriptors, in the same way. So a buffer and it's descriptor
always end on the same NUMA node. This is one of the reasons why we need
to map larger chunks, because NUMA works on page granularity, and the
descriptors are tiny - many fit on a memory page.

There's a secondary benefit of explicitly assigning buffers to nodes,
using this simple scheme - it allows quickly determining the node ID
given a buffer ID. This is helpful later, when building freelist.

The patch is fairly simple. Most of the complexity is about picking the
chunk size, and aligning the arrays (so that it nicely aligns with
memory pages).

The patch has a GUC "numa_buffers_interleave", with "off" by default.


2) v1-0002-NUMA-localalloc.patch

This simply sets "localalloc" when initializing a backend, so that all
memory allocated later is local, not interleaved. Initially this was
necessary because the patch set the allocation policy to interleaving
before initializing shared memory, and we didn't want to interleave the
private memory. But that's no longer the case - the explicit mapping to
nodes does not have this issue. I'm keeping the patch for convenience,
it allows experimenting with numactl etc.

The patch has a GUC "numa_localalloc", with "off" by default.


3) v1-0003-freelist-Don-t-track-tail-of-a-freelist.patch

Minor optimization. Andres noticed we're tracking the tail of buffer
freelist, without using it. So the patch removes that.


4) v1-0004-NUMA-partition-buffer-freelist.patch

Right now we have a single freelist, and in busy instances that can be
quite contended. What's worse, the freelist may trash between different
CPUs, NUMA nodes, etc. So the idea is to have multiple freelists on
subsets of buffers. The patch implements multiple strategies how the
list can be split (configured using "numa_partition_freelist" GUC), for
experimenting:

* node - One list per NUMA node. This is the most natural option,
because we now know which buffer is on which node, so we can ensure a
list for a node only has buffers from that list.

* cpu - One list per CPU. Pretty simple, each CPU gets it's own list.

* pid - Similar to "cpu", but the processes are mapped to lists based on
PID, not CPU ID.

* none - nothing, sigle freelist

Ultimately, I think we'll want to go with "node", simply because it
aligns with the buffer interleaving. But there are improvements needed.

The main challenge is that with multiple smaller lists, a process can't
really use the whole shared buffers. So a single backed will only use
part of the memory. The more lists there are, the worse this effect is.
This is also why I think we won't use the other partitioning options,
because there's going to be more CPUs than NUMA nodes.

Obviously, this needs solving even with NUMA nodes - we need to allow a
single backend to utilize the whole shared buffers if needed. There
should be a way to "steal" buffers from other freelists (if the
"regular" freelist is empty), but the patch does not implement this.
Shouldn't be hard, I think.

The other missing part is clocksweep - there's still just a single
instance of clocksweep, feeding buffers to all the freelists. But that's
clearly a problem, because the clocksweep returns buffers from all NUMA
nodes. The clocksweep really needs to be partitioned the same way as a
freelists, and each partition will operate on a subset of buffers (from
the right NUMA node).

I do have a separate experimental patch doing something like that, I
need to make it part of this branch.


5) v1-0005-NUMA-interleave-PGPROC-entries.patch

Another area that seems like it might benefit from NUMA is PGPROC, so I
gave it a try. It turned out somewhat challenging. Similarly to buffers
we have two pieces that need to be located in a coordinated way - PGPROC
entries and fast-path arrays. But we can't use the same approach as for
buffers/descriptors, because

(a) Neither of those pieces aligns with memory page size (PGPROC is
~900B, fast-path arrays are variable length).

(b) We could pad PGPROC entries e.g. to 1KB, but that'd still require
rather high max_connections before we use multiple huge pages.

The fast-path arrays are less of a problem, because those tend to be
larger, and are accessed through pointers, so we can just adjust that.

So what I did instead is splitting the whole PGPROC array into one array
per NUMA node, and one array for auxiliary processes and 2PC xacts. So
with 4 NUMA nodes there are 5 separate arrays, for example. Each array
is a multiple of memory pages, so we may waste some of the memory. But
that's simply how NUMA works - page granularity.

This however makes one particular thing harder - in a couple places we
accessed PGPROC entries through PROC_HDR->allProcs, which was pretty
much just one large array. And GetNumberFromPGProc() relied on array
arithmetics to determine procnumber. With the array partitioned, this
can't work the same way.

But there's a simple solution - if we turn allProcs into an array of
*pointers* to PGPROC arrays, there's no issue. All the places need a
pointer anyway. And then we need an explicit procnumber field in PGPROC,
instead of calculating it.

There's a chance this have negative impact on code that accessed PGPROC
very often, but so far I haven't seen such cases. But if you can come up
with such examples, I'd like to see those.

There's another detail - when obtaining a PGPROC entry in InitProcess(),
we try to get an entry from the same NUMA node. And only if that doesn't
work, we grab the first one from the list (there's still just one PGPROC
freelist, I haven't split that - maybe we should?).

This has a GUC "numa_procs_interleave", again "off" by default. It's not
quite correct, though, because the partitioning happens always. It only
affects the PGPROC lookup. (In a way, this may be a bit broken.)


6) v1-0006-NUMA-pin-backends-to-NUMA-nodes.patch

This is an experimental patch, that simply pins the new process to the
NUMA node obtained from the freelist.

Driven by GUC "numa_procs_pin" (default: off).


Summary
-------

So this is what I have at the moment. I've tried to organize the patches
in the order of importance, but that's just my guess. It's entirely
possible there's something I missed, some other order might make more
sense, etc.

There's also the question how this is related to other patches affecting
shared memory - I think the most relevant one is the "shared buffers
online resize" by Ashutosh, simply because it touches the shared memory.

I don't think the splitting would actually make some things simpler, or
maybe more flexible - in particular, it'd allow us to enable huge pages
only for some regions (like shared buffers), and keep the small pages
e.g. for PGPROC. So that'd be good.

But there'd also need to be some logic to "rework" how shared buffers
get mapped to NUMA nodes after resizing. It'd be silly to start with
memory on 4 nodes (25% each), resize shared buffers to 50% and end up
with memory only on 2 of the nodes (because the other 2 nodes were
originally assigned the upper half of shared buffers).

I don't have a clear idea how this would be done, but I guess it'd
require a bit of code invoked sometime after the resize. It'd already
need to rebuild the freelists in some way, I guess.

The other thing I haven't thought about very much is determining on
which CPUs/nodes the instance is allowed to run. I assume we'd start by
simply inherit/determine that at the start through libnuma, not through
some custom PG configuration (which the patch [2] proposed to do).


regards


[1] https://www.youtube.com/watch?v=V75KpACdl6E

[2]
https://www.postgresql.org/message-id/CAKZiRmw6i1W1AwXxa-Asrn8wrVcVH3TO715g_MCoowTS9rkGyw%40mail.gmail.com

-- 
Tomas Vondra

  2. Re: Adding basic NUMA awareness

    Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> — 2025-07-02T11:37:28Z

    On Wed, Jul 2, 2025 at 12:37 AM Tomas Vondra <tomas@vondra.me> wrote:
>
>
> 3) v1-0003-freelist-Don-t-track-tail-of-a-freelist.patch
>
> Minor optimization. Andres noticed we're tracking the tail of buffer
> freelist, without using it. So the patch removes that.
>

The patches for resizing buffers use the lastFreeBuffer to add new
buffers to the end of free list when expanding it. But we could as
well add it at the beginning of the free list.

This patch seems almost independent of the rest of the patches. Do you
need it in the rest of the patches? I understand that those patches
don't need to worry about maintaining lastFreeBuffer after this patch.
Is there any other effect?

If we are going to do this, let's do it earlier so that buffer
resizing patches can be adjusted.

>
> There's also the question how this is related to other patches affecting
> shared memory - I think the most relevant one is the "shared buffers
> online resize" by Ashutosh, simply because it touches the shared memory.

I have added Dmitry to this thread since he has written most of the
shared memory handling code.

>
> I don't think the splitting would actually make some things simpler, or
> maybe more flexible - in particular, it'd allow us to enable huge pages
> only for some regions (like shared buffers), and keep the small pages
> e.g. for PGPROC. So that'd be good.

The resizing patches split the shared buffer related structures into
separate memory segments. I think that itself will help enabling huge
pages for some regions. Would that help in your case?

>
> But there'd also need to be some logic to "rework" how shared buffers
> get mapped to NUMA nodes after resizing. It'd be silly to start with
> memory on 4 nodes (25% each), resize shared buffers to 50% and end up
> with memory only on 2 of the nodes (because the other 2 nodes were
> originally assigned the upper half of shared buffers).
>
> I don't have a clear idea how this would be done, but I guess it'd
> require a bit of code invoked sometime after the resize. It'd already
> need to rebuild the freelists in some way, I guess.

Yes, there's code to build the free list. I think we will need code to
remap the buffers and buffer descriptor.

-- 
Best Wishes,
Ashutosh Bapat

  3. Re: Adding basic NUMA awareness

    Tomas Vondra <tomas@vondra.me> — 2025-07-02T12:36:31Z

    
On 7/2/25 13:37, Ashutosh Bapat wrote:
> On Wed, Jul 2, 2025 at 12:37 AM Tomas Vondra <tomas@vondra.me> wrote:
>>
>>
>> 3) v1-0003-freelist-Don-t-track-tail-of-a-freelist.patch
>>
>> Minor optimization. Andres noticed we're tracking the tail of buffer
>> freelist, without using it. So the patch removes that.
>>
> 
> The patches for resizing buffers use the lastFreeBuffer to add new
> buffers to the end of free list when expanding it. But we could as
> well add it at the beginning of the free list.
> 
> This patch seems almost independent of the rest of the patches. Do you
> need it in the rest of the patches? I understand that those patches
> don't need to worry about maintaining lastFreeBuffer after this patch.
> Is there any other effect?
> 
> If we are going to do this, let's do it earlier so that buffer
> resizing patches can be adjusted.
> 

My patches don't particularly rely on this bit, it would work even with
lastFreeBuffer. I believe Andres simply noticed the current code does
not use lastFreeBuffer, it just maintains is, so he removed that as an
optimization. I don't know how significant is the improvement, but if
it's measurable we could just do that independently of our patches.

>>
>> There's also the question how this is related to other patches affecting
>> shared memory - I think the most relevant one is the "shared buffers
>> online resize" by Ashutosh, simply because it touches the shared memory.
> 
> I have added Dmitry to this thread since he has written most of the
> shared memory handling code.
> 

Thanks.

>>
>> I don't think the splitting would actually make some things simpler, or
>> maybe more flexible - in particular, it'd allow us to enable huge pages
>> only for some regions (like shared buffers), and keep the small pages
>> e.g. for PGPROC. So that'd be good.
> 
> The resizing patches split the shared buffer related structures into
> separate memory segments. I think that itself will help enabling huge
> pages for some regions. Would that help in your case?
> 

Indirectly. My patch can work just fine with a single segment, but being
able to enable huge pages only for some of the segments seems better.

>>
>> But there'd also need to be some logic to "rework" how shared buffers
>> get mapped to NUMA nodes after resizing. It'd be silly to start with
>> memory on 4 nodes (25% each), resize shared buffers to 50% and end up
>> with memory only on 2 of the nodes (because the other 2 nodes were
>> originally assigned the upper half of shared buffers).
>>
>> I don't have a clear idea how this would be done, but I guess it'd
>> require a bit of code invoked sometime after the resize. It'd already
>> need to rebuild the freelists in some way, I guess.
> 
> Yes, there's code to build the free list. I think we will need code to
> remap the buffers and buffer descriptor.
> 

Right. The good thing is that's just "advisory" information, it doesn't
break anything if it's temporarily out of sync. We don't need to "stop"
everything to remap the buffers to other nodes, or anything like that.
Or at least I think so.

It's one thing to "flip" the target mapping (determining which node a
buffer should be on), and actually migrating the buffers. The first part
can be done instantaneously, the second part can happen in the
background over a longer time period.

I'm not sure how you're rebuilding the freelist. Presumably it can
contain buffers that are no longer valid (after shrinking). How is that
handled to not break anything? I think the NUMA variant would do exactly
the same thing, except that there's multiple lists.

regards

-- 
Tomas Vondra

  4. Re: Adding basic NUMA awareness

    Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> — 2025-07-03T14:07:18Z

    On Wed, Jul 2, 2025 at 6:06 PM Tomas Vondra <tomas@vondra.me> wrote:
>
> I'm not sure how you're rebuilding the freelist. Presumably it can
> contain buffers that are no longer valid (after shrinking). How is that
> handled to not break anything? I think the NUMA variant would do exactly
> the same thing, except that there's multiple lists.

Before shrinking the buffers, we walk the free list removing any
buffers that are going to be removed. When expanding, by linking the
new buffers in the order and then adding those to the already existing
free list. 0005 patch in [1] has the code for the same.

[1] https://www.postgresql.org/message-id/my4hukmejato53ef465ev7lk3sqiqvneh7436rz64wmtc7rbfj%40hmuxsf2ngov2

--
Best Wishes,
Ashutosh Bapat

  5. Re: Adding basic NUMA awareness

    Dmitry Dolgov <9erthalion6@gmail.com> — 2025-07-03T14:49:34Z

    > On Wed, Jul 02, 2025 at 05:07:28PM +0530, Ashutosh Bapat wrote:
> > There's also the question how this is related to other patches affecting
> > shared memory - I think the most relevant one is the "shared buffers
> > online resize" by Ashutosh, simply because it touches the shared memory.
>
> I have added Dmitry to this thread since he has written most of the
> shared memory handling code.

Thanks! I like the idea behind this patch series. I haven't read it in
details yet, but I can imagine both patches (interleaving and online
resizing) could benefit from each other. In online resizing we've
introduced a possibility to use multiple shared mappings for different
types of data, maybe it would be convenient to use the same interface to
create separate mappings for different NUMA nodes as well. Using a
separate shared mapping per NUMA node would also make resizing easier,
since it would be more straightforward to fit an increased segment into
NUMA boundaries.

> > I don't think the splitting would actually make some things simpler, or
> > maybe more flexible - in particular, it'd allow us to enable huge pages
> > only for some regions (like shared buffers), and keep the small pages
> > e.g. for PGPROC. So that'd be good.
>
> The resizing patches split the shared buffer related structures into
> separate memory segments. I think that itself will help enabling huge
> pages for some regions. Would that help in your case?

Right, separate segments would allow to mix and match huge pages with
pages of regular size. It's not implemented in the latest version of
online resizing patch, purely to reduce complexity and maintain the same
invariant (everything is either using huge pages or not) -- but we could
do it other way around as well.

  6. Re: Adding basic NUMA awareness

    Jakub Wartak <jakub.wartak@enterprisedb.com> — 2025-07-04T11:05:05Z

    On Tue, Jul 1, 2025 at 9:07 PM Tomas Vondra <tomas@vondra.me> wrote:

Hi!

> 1) v1-0001-NUMA-interleaving-buffers.patch
[..]
> It's a bit more complicated, because the patch distributes both the
> blocks and descriptors, in the same way. So a buffer and it's descriptor
> always end on the same NUMA node. This is one of the reasons why we need
> to map larger chunks, because NUMA works on page granularity, and the
> descriptors are tiny - many fit on a memory page.

Oh, now I get it! OK, let's stick to this one.

> I don't think the splitting would actually make some things simpler, or
> maybe more flexible - in particular, it'd allow us to enable huge pages
> only for some regions (like shared buffers), and keep the small pages
> e.g. for PGPROC. So that'd be good.

You have made assumption that this is good, but small pages (4KB) are
not hugetlb, and are *swappable* (Transparent HP are swappable too,
manually allocated ones as with mmap(MMAP_HUGETLB) are not)[1]. The
most frequent problem I see these days are OOMs, and it makes me
believe that making certain critical parts of shared memory being
swappable just to make pagesize granular is possibly throwing the baby
out with the bathwater. I'm thinking about bad situations like: some
wrong settings of vm.swapiness that people keep (or distros keep?) and
general inability of PG to restrain from allocating more memory in
some cases.

> The other thing I haven't thought about very much is determining on
> which CPUs/nodes the instance is allowed to run. I assume we'd start by
> simply inherit/determine that at the start through libnuma, not through
> some custom PG configuration (which the patch [2] proposed to do).

0. I think that we could do better, some counter arguments to
no-configuration-at-all:

a. as Robert & Bertrand already put it there after review: let's say I
want just to run on NUMA #2 node, so here I would need to override
systemd's script ExecStart= to include that numactl (not elegant?). I
could also use `CPUAffinity=1,3,5,7..` but that's all, and it is even
less friendly. Also it probably requires root to edit/reload systemd,
while having GUC for this like in my proposal makes it more smooth (I
think?)

b. wouldn't it be better if that stayed as drop-in rather than always
on? What if there's a problem, how do you disable those internal
optimizations if they do harm in some cases?  (or let's say I want to
play with MPOL_INTERLEAVE_WEIGHTED?). So at least boolean
numa_buffers_interleave would be nice?

c. What if I want my standby (walreceiver+startup/recovery) to run
with NUMA affinity to get better performance (I'm not going to hack
around systemd script every time, but I could imagine changing
numa=X,Y,Z after restart/before promotion)

d. Now if I would be forced for some reason to do that numactl(1)
voodoo, and use the those above mentioned overrides and PG wouldn't be
having GUC (let's say I would use `numactl
--weighted-interleave=0,1`), then:

> 2) v1-0002-NUMA-localalloc.patch
> This simply sets "localalloc" when initializing a backend, so that all
> memory allocated later is local, not interleaved. Initially this was
> necessary because the patch set the allocation policy to interleaving
> before initializing shared memory, and we didn't want to interleave the
> private memory. But that's no longer the case - the explicit mapping to
> nodes does not have this issue. I'm keeping the patch for convenience,
> it allows experimenting with numactl etc.

.. .is not accurate anymore and we would require to have that in
(still with GUC) ?
Thoughts? I can add that mine part into Your's patches if you want.

Way too quick review and some very fast benchmark probes, I've
concentrated only on v1-0001 and v1-0005 (efficiency of buffermgmt
would be too new topic for me), but let's start:

1. normal pgbench -S (still with just s_b@4GB), done many tries,
consistent benefit for the patch with like +8..10% boost on generic
run:

   numa_buffers_interleave=off  numa_pgproc_interleave=on(due that
always on "if"), s_b just on 1 NUMA node (might happen)
        latency average = 0.373 ms
        latency stddev = 0.237 ms
        initial connection time = 45.899 ms
        tps = 160242.147877 (without initial connection time)

   numa_buffers_interleave=on   numa_pgproc_interleave=on
        latency average = 0.345 ms
        latency stddev = 0.373 ms
        initial connection time = 44.485 ms
        tps = 177564.686094 (without initial connection time)

2. Tested it the same way as I did for mine(problem#2 from Andres's
presentation): 4s32c128t, s_b=4GB (on 128GB), prewarm test (with
seqconcurrscans.pgb as earlier)
   default/numa_buffers_interleave=off
        latency average = 1375.478 ms
        latency stddev = 1141.423 ms
        initial connection time = 46.104 ms
        tps = 45.868075 (without initial connection time)

   numa_buffers_interleave=on
        latency average = 838.128 ms
        latency stddev = 498.787 ms
        initial connection time = 43.437 ms
        tps = 75.413894 (without initial connection time)

    and i've repeated the the same test (identical conditions) with my
patch, got me slightly more juice:
        latency average = 727.717 ms
        latency stddev = 410.767 ms
        initial connection time = 45.119 ms
        tps = 86.844161 (without initial connection time)

    (but mine didn't get that boost from normal pgbench as per #1
pgbench -S -- my numa='all' stays @ 160k TPS just as
numa_buffers_interleave=off), so this idea is clearly better.
    So should I close https://commitfest.postgresql.org/patch/5703/
and you'll open a new one or should I just edit the #5703 and alter it
and add this thread too?

3. Patch is not calling interleave on PQ shmem, do we want to add that
in as some next item like v1-0007? Question is whether OS interleaving
makes sense there ? I believe it does there, please see my thread
(NUMA_pq_cpu_pinning_results.txt), the issue is that PQ workers are
being spawned by postmaster and may end up on different NUMA nodes
randomly, so actually OS-interleaving that memory reduces jitter there
(AKA bandwidth-over-latency). My thinking is that one cannot expect
static/forced CPU-to-just-one-NUMA-node assignment for backend and
it's PQ workers, because it is impossible have always available CPU
power there in that NUMA node, so it might be useful to interleave
that shared mem there too (as separate patch item?)

4 In BufferManagerShmemInit() you call numa_num_configured_nodes()
(also in v1-0005). My worry is should we may put some
known-limitations docs (?) from start and mention that
if the VM is greatly resized and NUMA numa nodes appear, they might
not be used until restart?

5. In v1-0001, pg_numa_interleave_memory()

+                * XXX no return value, to make this fail on error, has to use
+                * numa_set_strict

Yes, my patch has those numa_error() and numa_warn() handlers too in
pg_numa. Feel free to use it for better UX.

+                * XXX Should we still touch the memory first, like
with numa_move_pages,
+                * or is that not necessary?

It's not necessary to touch after numa_tonode_memory() (wrapper around
numa_interleave_memory()), if it is going to be used anyway it will be
correctly placed to best of my knowledge.

6. diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c

Accidental indents (also fails to apply)

7. We miss the pg_numa_* shims, but for sure that's for later and also
avoid those Linux specific #ifdef USE_LIBNUMA and so on?

8. v1-0005 2x + /* if (numa_procs_interleave) */

   Ha! it's a TRAP! I've uncommented it because I wanted to try it out
without it (just by setting GUC off) , but "MyProc->sema" is NULL :

    2025-07-04 12:31:08.103 CEST [28754] LOG:  starting PostgreSQL
19devel on x86_64-linux, compiled by gcc-12.2.0, 64-bit
    [..]
    2025-07-04 12:31:08.109 CEST [28754] LOG:  io worker (PID 28755)
was terminated by signal 11: Segmentation fault
    2025-07-04 12:31:08.109 CEST [28754] LOG:  terminating any other
active server processes
    2025-07-04 12:31:08.114 CEST [28754] LOG:  shutting down because
"restart_after_crash" is off
    2025-07-04 12:31:08.116 CEST [28754] LOG:  database system is shut down

    [New LWP 28755]
    [Thread debugging using libthread_db enabled]
    Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
    Core was generated by `postgres: io worker                     '.
    Program terminated with signal SIGSEGV, Segmentation fault.
    #0  __new_sem_wait_fast (definitive_result=1, sem=sem@entry=0x0)
at ./nptl/sem_waitcommon.c:136
    136     ./nptl/sem_waitcommon.c: No such file or directory.
    (gdb) where
    #0  __new_sem_wait_fast (definitive_result=1, sem=sem@entry=0x0)
at ./nptl/sem_waitcommon.c:136
    #1  __new_sem_trywait (sem=sem@entry=0x0) at ./nptl/sem_wait.c:81
    #2  0x00005561918e0cac in PGSemaphoreReset (sema=0x0) at
../src/backend/port/posix_sema.c:302
    #3  0x0000556191970553 in InitAuxiliaryProcess () at
../src/backend/storage/lmgr/proc.c:992
    #4  0x00005561918e51a2 in AuxiliaryProcessMainCommon () at
../src/backend/postmaster/auxprocess.c:65
    #5  0x0000556191940676 in IoWorkerMain (startup_data=<optimized
out>, startup_data_len=<optimized out>) at
../src/backend/storage/aio/method_worker.c:393
    #6  0x00005561918e8163 in postmaster_child_launch
(child_type=child_type@entry=B_IO_WORKER, child_slot=20086,
startup_data=startup_data@entry=0x0,
        startup_data_len=startup_data_len@entry=0,
client_sock=client_sock@entry=0x0) at
../src/backend/postmaster/launch_backend.c:290
    #7  0x00005561918ea09a in StartChildProcess
(type=type@entry=B_IO_WORKER) at
../src/backend/postmaster/postmaster.c:3973
    #8  0x00005561918ea308 in maybe_adjust_io_workers () at
../src/backend/postmaster/postmaster.c:4404
    [..]
    (gdb) print *MyProc->sem
    Cannot access memory at address 0x0

9. v1-0006: is this just a thought or serious candidate? I can imagine
it can easily blow-up with some backends somehow requesting CPUs only
from one NUMA node, while the second node being idle. Isn't it better
just to leave CPU scheduling, well, to the CPU scheduler? The problem
is that you have tools showing overall CPU usage, even mpstat(1) per
CPU , but no tools for per-NUMA node CPU util%, so it would be hard
for someone to realize that this is happening.

-J.

[1] - https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt

  7. Re: Adding basic NUMA awareness

    Tomas Vondra <tomas@vondra.me> — 2025-07-04T18:12:01Z

    On 7/4/25 13:05, Jakub Wartak wrote:
> On Tue, Jul 1, 2025 at 9:07 PM Tomas Vondra <tomas@vondra.me> wrote:
> 
> Hi!
> 
>> 1) v1-0001-NUMA-interleaving-buffers.patch
> [..]
>> It's a bit more complicated, because the patch distributes both the
>> blocks and descriptors, in the same way. So a buffer and it's descriptor
>> always end on the same NUMA node. This is one of the reasons why we need
>> to map larger chunks, because NUMA works on page granularity, and the
>> descriptors are tiny - many fit on a memory page.
> 
> Oh, now I get it! OK, let's stick to this one.
> 
>> I don't think the splitting would actually make some things simpler, or
>> maybe more flexible - in particular, it'd allow us to enable huge pages
>> only for some regions (like shared buffers), and keep the small pages
>> e.g. for PGPROC. So that'd be good.
> 
> You have made assumption that this is good, but small pages (4KB) are
> not hugetlb, and are *swappable* (Transparent HP are swappable too,
> manually allocated ones as with mmap(MMAP_HUGETLB) are not)[1]. The
> most frequent problem I see these days are OOMs, and it makes me
> believe that making certain critical parts of shared memory being
> swappable just to make pagesize granular is possibly throwing the baby
> out with the bathwater. I'm thinking about bad situations like: some
> wrong settings of vm.swapiness that people keep (or distros keep?) and
> general inability of PG to restrain from allocating more memory in
> some cases.
> 

I haven't observed such issues myself, or maybe I didn't realize it's
happening. Maybe it happens, but it'd be good to see some data showing
that, or a reproducer of some sort. But let's say it's real.

I don't think we should use huge pages merely to ensure something is not
swapped out. The "not swappable" is more of a limitation of huge pages,
not an advantage. You can't just choose to make them swappable.

Wouldn't it be better to keep using 4KB pages, but lock the memory using
mlock/mlockall?


>> The other thing I haven't thought about very much is determining on
>> which CPUs/nodes the instance is allowed to run. I assume we'd start by
>> simply inherit/determine that at the start through libnuma, not through
>> some custom PG configuration (which the patch [2] proposed to do).
> 
> 0. I think that we could do better, some counter arguments to
> no-configuration-at-all:
> 
> a. as Robert & Bertrand already put it there after review: let's say I
> want just to run on NUMA #2 node, so here I would need to override
> systemd's script ExecStart= to include that numactl (not elegant?). I
> could also use `CPUAffinity=1,3,5,7..` but that's all, and it is even
> less friendly. Also it probably requires root to edit/reload systemd,
> while having GUC for this like in my proposal makes it more smooth (I
> think?)
> 
> b. wouldn't it be better if that stayed as drop-in rather than always
> on? What if there's a problem, how do you disable those internal
> optimizations if they do harm in some cases?  (or let's say I want to
> play with MPOL_INTERLEAVE_WEIGHTED?). So at least boolean
> numa_buffers_interleave would be nice?
> 
> c. What if I want my standby (walreceiver+startup/recovery) to run
> with NUMA affinity to get better performance (I'm not going to hack
> around systemd script every time, but I could imagine changing
> numa=X,Y,Z after restart/before promotion)
> 
> d. Now if I would be forced for some reason to do that numactl(1)
> voodoo, and use the those above mentioned overrides and PG wouldn't be
> having GUC (let's say I would use `numactl
> --weighted-interleave=0,1`), then:
> 

I'm not against doing something like this, but I don't plan to do that
in V1. I don't have a clear idea what configurability is actually
needed, so it's likely I'd do the interface wrong.

>> 2) v1-0002-NUMA-localalloc.patch
>> This simply sets "localalloc" when initializing a backend, so that all
>> memory allocated later is local, not interleaved. Initially this was
>> necessary because the patch set the allocation policy to interleaving
>> before initializing shared memory, and we didn't want to interleave the
>> private memory. But that's no longer the case - the explicit mapping to
>> nodes does not have this issue. I'm keeping the patch for convenience,
>> it allows experimenting with numactl etc.
> 
> .. .is not accurate anymore and we would require to have that in
> (still with GUC) ?
> Thoughts? I can add that mine part into Your's patches if you want.
> 

I'm sorry, I don't understand what's the question :-(

> Way too quick review and some very fast benchmark probes, I've
> concentrated only on v1-0001 and v1-0005 (efficiency of buffermgmt
> would be too new topic for me), but let's start:
> 
> 1. normal pgbench -S (still with just s_b@4GB), done many tries,
> consistent benefit for the patch with like +8..10% boost on generic
> run:
> 
>    numa_buffers_interleave=off  numa_pgproc_interleave=on(due that
> always on "if"), s_b just on 1 NUMA node (might happen)
>         latency average = 0.373 ms
>         latency stddev = 0.237 ms
>         initial connection time = 45.899 ms
>         tps = 160242.147877 (without initial connection time)
> 
>    numa_buffers_interleave=on   numa_pgproc_interleave=on
>         latency average = 0.345 ms
>         latency stddev = 0.373 ms
>         initial connection time = 44.485 ms
>         tps = 177564.686094 (without initial connection time)
> 
> 2. Tested it the same way as I did for mine(problem#2 from Andres's
> presentation): 4s32c128t, s_b=4GB (on 128GB), prewarm test (with
> seqconcurrscans.pgb as earlier)
>    default/numa_buffers_interleave=off
>         latency average = 1375.478 ms
>         latency stddev = 1141.423 ms
>         initial connection time = 46.104 ms
>         tps = 45.868075 (without initial connection time)
> 
>    numa_buffers_interleave=on
>         latency average = 838.128 ms
>         latency stddev = 498.787 ms
>         initial connection time = 43.437 ms
>         tps = 75.413894 (without initial connection time)
> 
>     and i've repeated the the same test (identical conditions) with my
> patch, got me slightly more juice:
>         latency average = 727.717 ms
>         latency stddev = 410.767 ms
>         initial connection time = 45.119 ms
>         tps = 86.844161 (without initial connection time)
> 
>     (but mine didn't get that boost from normal pgbench as per #1
> pgbench -S -- my numa='all' stays @ 160k TPS just as
> numa_buffers_interleave=off), so this idea is clearly better.

Good, thanks for the testing. I should have done something like this
when I posted my patches, but I forgot about that (and the email felt
too long anyway).

But this actually brings an interesting question. What exactly should we
expect / demand from these patches? In my mind it'd primarily about
predictability and stability of results.

For example, the results should not depend on how was the database
warmed up - was it done by a single backend or many backends? Was it
restarted, or what? I could probably warmup the system very carefully to
ensure it's balanced. The patches mean I don't need to be that careful.


>     So should I close https://commitfest.postgresql.org/patch/5703/
> and you'll open a new one or should I just edit the #5703 and alter it
> and add this thread too?
> 

Good question. It's probably best to close the original entry as
"withdrawn" and I'll add a new entry. Sounds OK?

> 3. Patch is not calling interleave on PQ shmem, do we want to add that
> in as some next item like v1-0007? Question is whether OS interleaving
> makes sense there ? I believe it does there, please see my thread
> (NUMA_pq_cpu_pinning_results.txt), the issue is that PQ workers are
> being spawned by postmaster and may end up on different NUMA nodes
> randomly, so actually OS-interleaving that memory reduces jitter there
> (AKA bandwidth-over-latency). My thinking is that one cannot expect
> static/forced CPU-to-just-one-NUMA-node assignment for backend and
> it's PQ workers, because it is impossible have always available CPU
> power there in that NUMA node, so it might be useful to interleave
> that shared mem there too (as separate patch item?)
> 

Excellent question. I haven't thought about this at all. I agree it
probably makes sense to interleave this memory, in some way. I don't
know what's the perfect scheme, though.

wild idea: Would it make sense to pin the workers to the same NUMA node
as the leader? And allocate all memory only from that node?

> 4 In BufferManagerShmemInit() you call numa_num_configured_nodes()
> (also in v1-0005). My worry is should we may put some
> known-limitations docs (?) from start and mention that
> if the VM is greatly resized and NUMA numa nodes appear, they might
> not be used until restart?
> 

Yes, this is one thing I need some feedback on. The patches mostly
assume there are no disabled nodes, that the set of allowed nodes does
not change, etc. I think for V1 that's a reasonable limitation.

But let's say we want to relax this a bit. How do we learn about the
change, after a node/CPU gets disabled? For some parts it's not that
difficult (e.g. we can "remap" buffers/descriptors) in the background.
But for other parts that's not practical. E.g. we can't rework how the
PGPROC gets split.

But while discussing this with Andres yesterday, he had an interesting
suggestion - to always use e.g. 8 or 16 partitions, then partition this
by NUMA node. So we'd have 16 partitions, and with 4 nodes the 0-3 would
go to node 0, 4-7 would go to node 1, etc. The advantage is that if a
node gets disabled, we can rebuild just this small "mapping" and not the
16 partitions. And the partitioning may be helpful even without NUMA.

Still have to figure out the details, but seems it might help.

> 5. In v1-0001, pg_numa_interleave_memory()
> 
> +                * XXX no return value, to make this fail on error, has to use
> +                * numa_set_strict
> 
> Yes, my patch has those numa_error() and numa_warn() handlers too in
> pg_numa. Feel free to use it for better UX.
> 
> +                * XXX Should we still touch the memory first, like
> with numa_move_pages,
> +                * or is that not necessary?
> 
> It's not necessary to touch after numa_tonode_memory() (wrapper around
> numa_interleave_memory()), if it is going to be used anyway it will be
> correctly placed to best of my knowledge.
> 
> 6. diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
> 
> Accidental indents (also fails to apply)
> 
> 7. We miss the pg_numa_* shims, but for sure that's for later and also
> avoid those Linux specific #ifdef USE_LIBNUMA and so on?
> 

Right, we need to add those. Or actually, we need to think about how
we'd do this for non-NUMA systems. I wonder if we even want to just
build everything the "old way" (without the partitions, etc.).

But per the earlier comment, the partitioning seems beneficial even on
non-NUMA systems, so maybe the shims are good enough OK.

> 8. v1-0005 2x + /* if (numa_procs_interleave) */
> 
>    Ha! it's a TRAP! I've uncommented it because I wanted to try it out
> without it (just by setting GUC off) , but "MyProc->sema" is NULL :
> 
>     2025-07-04 12:31:08.103 CEST [28754] LOG:  starting PostgreSQL
> 19devel on x86_64-linux, compiled by gcc-12.2.0, 64-bit
>     [..]
>     2025-07-04 12:31:08.109 CEST [28754] LOG:  io worker (PID 28755)
> was terminated by signal 11: Segmentation fault
>     2025-07-04 12:31:08.109 CEST [28754] LOG:  terminating any other
> active server processes
>     2025-07-04 12:31:08.114 CEST [28754] LOG:  shutting down because
> "restart_after_crash" is off
>     2025-07-04 12:31:08.116 CEST [28754] LOG:  database system is shut down
> 
>     [New LWP 28755]
>     [Thread debugging using libthread_db enabled]
>     Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>     Core was generated by `postgres: io worker                     '.
>     Program terminated with signal SIGSEGV, Segmentation fault.
>     #0  __new_sem_wait_fast (definitive_result=1, sem=sem@entry=0x0)
> at ./nptl/sem_waitcommon.c:136
>     136     ./nptl/sem_waitcommon.c: No such file or directory.
>     (gdb) where
>     #0  __new_sem_wait_fast (definitive_result=1, sem=sem@entry=0x0)
> at ./nptl/sem_waitcommon.c:136
>     #1  __new_sem_trywait (sem=sem@entry=0x0) at ./nptl/sem_wait.c:81
>     #2  0x00005561918e0cac in PGSemaphoreReset (sema=0x0) at
> ../src/backend/port/posix_sema.c:302
>     #3  0x0000556191970553 in InitAuxiliaryProcess () at
> ../src/backend/storage/lmgr/proc.c:992
>     #4  0x00005561918e51a2 in AuxiliaryProcessMainCommon () at
> ../src/backend/postmaster/auxprocess.c:65
>     #5  0x0000556191940676 in IoWorkerMain (startup_data=<optimized
> out>, startup_data_len=<optimized out>) at
> ../src/backend/storage/aio/method_worker.c:393
>     #6  0x00005561918e8163 in postmaster_child_launch
> (child_type=child_type@entry=B_IO_WORKER, child_slot=20086,
> startup_data=startup_data@entry=0x0,
>         startup_data_len=startup_data_len@entry=0,
> client_sock=client_sock@entry=0x0) at
> ../src/backend/postmaster/launch_backend.c:290
>     #7  0x00005561918ea09a in StartChildProcess
> (type=type@entry=B_IO_WORKER) at
> ../src/backend/postmaster/postmaster.c:3973
>     #8  0x00005561918ea308 in maybe_adjust_io_workers () at
> ../src/backend/postmaster/postmaster.c:4404
>     [..]
>     (gdb) print *MyProc->sem
>     Cannot access memory at address 0x0
> 

Yeah, good catch. I'll look into that next week.

> 9. v1-0006: is this just a thought or serious candidate? I can imagine
> it can easily blow-up with some backends somehow requesting CPUs only
> from one NUMA node, while the second node being idle. Isn't it better
> just to leave CPU scheduling, well, to the CPU scheduler? The problem
> is that you have tools showing overall CPU usage, even mpstat(1) per
> CPU , but no tools for per-NUMA node CPU util%, so it would be hard
> for someone to realize that this is happening.
> 

Mostly experimental, for benchmarking etc. I agree we may not want to
mess with the task scheduling too much.


Thanks for the feedback!

regards

-- 
Tomas Vondra

  8. Re: Adding basic NUMA awareness - Preliminary feedback and outline for an extensible approach

    Cédric Villemain <cedric.villemain@data-bene.io> — 2025-07-05T07:09:00Z

    Hi Tomas,


I haven't yet had time to fully read all the work and proposals around 
NUMA and related features, but I hope to catch up over the summer.

However, I think it's important to share some thoughts before it's too 
late, as you might find them relevant to the NUMA management code.


> 6) v1-0006-NUMA-pin-backends-to-NUMA-nodes.patch
> 
> This is an experimental patch, that simply pins the new process to the
> NUMA node obtained from the freelist.
> 
> Driven by GUC "numa_procs_pin" (default: off).


In my work on more careful PostgreSQL resource management, I've come to 
the conclusion that we should avoid pushing policy too deeply into the 
PostgreSQL core itself. Therefore, I'm quite skeptical about integrating 
NUMA-specific management directly into core PostgreSQL in such a way.


We are working on a PROFILE and PROFILE MANAGER specification to provide 
PostgreSQL with only the APIs and hooks needed so that extensions can 
manage whatever they want externally.

The basic syntax (not meant to be discussed here, and even the names 
might change) is roughly as follows, just to illustrate the intent:


CREATE PROFILE MANAGER manager_name [IF NOT EXISTS]
   [ HANDLER handler_function | NO HANDLER ]
   [ VALIDATOR validator_function | NO VALIDATOR ]
   [ OPTIONS ( option 'value' [, ... ] ) ]

CREATE PROFILE profile_name
   [IF NOT EXISTS]
   USING profile_manager
     SET key = value [, key = value]...
   [USING profile_manager
     SET key = value [, key = value]...]
   [...];

CREATE PROFILE MAPPING
   [IF NOT EXISTS]
   FOR PROFILE profile_name
   [MATCH [ ALL | ANY ] (
     [ROLE role_name],
     [BACKEND TYPE backend_type],
     [DATABASE database_name],
     [APPLICATION appname]
   )];

## PROFILE RESOLUTION ORDER

1. ALTER ROLE IN DATABASE
2. ALTER ROLE
3. ALTER DATABASE
4. First matching PROFILE MAPPING (global or specific)
5. No profile (fallback)

As currently designed, this approach allows quite a lot of flexibility:

* pg_psi is used to ensure the spec is suitable for a cgroup profile 
manager (moving PIDs as needed; NUMA and cgroups could work well 
together, see e.g. this Linux kernel summary: 
https://blogs.oracle.com/linux/post/numa-balancing )

* Someone else could implement support for Windows or BSD specifics.

* Others might use it to integrate PostgreSQL's own resources (e.g., 
"areas" of shared buffers) into policies.

Hope this perspective is helpful.

Best regards,
-- 
Cédric Villemain +33 6 20 30 22 52
https://www.Data-Bene.io
PostgreSQL Support, Expertise, Training, R&D

  9. Re: Adding basic NUMA awareness - Preliminary feedback and outline for an extensible approach

    Tomas Vondra <tomas@vondra.me> — 2025-07-07T12:01:53Z

    
On 7/5/25 09:09, Cédric Villemain wrote:
> Hi Tomas,
> 
> 
> I haven't yet had time to fully read all the work and proposals around
> NUMA and related features, but I hope to catch up over the summer.
> 
> However, I think it's important to share some thoughts before it's too
> late, as you might find them relevant to the NUMA management code.
> 
> 
>> 6) v1-0006-NUMA-pin-backends-to-NUMA-nodes.patch
>>
>> This is an experimental patch, that simply pins the new process to the
>> NUMA node obtained from the freelist.
>>
>> Driven by GUC "numa_procs_pin" (default: off).
> 
> 
> In my work on more careful PostgreSQL resource management, I've come to
> the conclusion that we should avoid pushing policy too deeply into the
> PostgreSQL core itself. Therefore, I'm quite skeptical about integrating
> NUMA-specific management directly into core PostgreSQL in such a way.
> 
> 
> We are working on a PROFILE and PROFILE MANAGER specification to provide
> PostgreSQL with only the APIs and hooks needed so that extensions can
> manage whatever they want externally.
> 
> The basic syntax (not meant to be discussed here, and even the names
> might change) is roughly as follows, just to illustrate the intent:
> 
> 
> CREATE PROFILE MANAGER manager_name [IF NOT EXISTS]
>   [ HANDLER handler_function | NO HANDLER ]
>   [ VALIDATOR validator_function | NO VALIDATOR ]
>   [ OPTIONS ( option 'value' [, ... ] ) ]
> 
> CREATE PROFILE profile_name
>   [IF NOT EXISTS]
>   USING profile_manager
>     SET key = value [, key = value]...
>   [USING profile_manager
>     SET key = value [, key = value]...]
>   [...];
> 
> CREATE PROFILE MAPPING
>   [IF NOT EXISTS]
>   FOR PROFILE profile_name
>   [MATCH [ ALL | ANY ] (
>     [ROLE role_name],
>     [BACKEND TYPE backend_type],
>     [DATABASE database_name],
>     [APPLICATION appname]
>   )];
> 
> ## PROFILE RESOLUTION ORDER
> 
> 1. ALTER ROLE IN DATABASE
> 2. ALTER ROLE
> 3. ALTER DATABASE
> 4. First matching PROFILE MAPPING (global or specific)
> 5. No profile (fallback)
> 
> As currently designed, this approach allows quite a lot of flexibility:
> 
> * pg_psi is used to ensure the spec is suitable for a cgroup profile
> manager (moving PIDs as needed; NUMA and cgroups could work well
> together, see e.g. this Linux kernel summary: https://blogs.oracle.com/
> linux/post/numa-balancing )
> 
> * Someone else could implement support for Windows or BSD specifics.
> 
> * Others might use it to integrate PostgreSQL's own resources (e.g.,
> "areas" of shared buffers) into policies.
> 
> Hope this perspective is helpful.

Can you explain how you want to manage this by an extension defined at
the SQL level, when most of this stuff has to be done when setting up
shared memory, which is waaaay before we have any access to catalogs?


regards

-- 
Tomas Vondra

  10. Re: Adding basic NUMA awareness

    Jakub Wartak <jakub.wartak@enterprisedb.com> — 2025-07-07T12:31:28Z

    Hi Tomas, some more thoughts after the weekend:

On Fri, Jul 4, 2025 at 8:12 PM Tomas Vondra <tomas@vondra.me> wrote:
>
> On 7/4/25 13:05, Jakub Wartak wrote:
> > On Tue, Jul 1, 2025 at 9:07 PM Tomas Vondra <tomas@vondra.me> wrote:
> >
> > Hi!
> >
> >> 1) v1-0001-NUMA-interleaving-buffers.patch
> > [..]
> >> It's a bit more complicated, because the patch distributes both the
> >> blocks and descriptors, in the same way. So a buffer and it's descriptor
> >> always end on the same NUMA node. This is one of the reasons why we need
> >> to map larger chunks, because NUMA works on page granularity, and the
> >> descriptors are tiny - many fit on a memory page.
> >
> > Oh, now I get it! OK, let's stick to this one.
> >
> >> I don't think the splitting would actually make some things simpler, or
> >> maybe more flexible - in particular, it'd allow us to enable huge pages
> >> only for some regions (like shared buffers), and keep the small pages
> >> e.g. for PGPROC. So that'd be good.
> >
> > You have made assumption that this is good, but small pages (4KB) are
> > not hugetlb, and are *swappable* (Transparent HP are swappable too,
> > manually allocated ones as with mmap(MMAP_HUGETLB) are not)[1]. The
> > most frequent problem I see these days are OOMs, and it makes me
> > believe that making certain critical parts of shared memory being
> > swappable just to make pagesize granular is possibly throwing the baby
> > out with the bathwater. I'm thinking about bad situations like: some
> > wrong settings of vm.swapiness that people keep (or distros keep?) and
> > general inability of PG to restrain from allocating more memory in
> > some cases.
> >
>
> I haven't observed such issues myself, or maybe I didn't realize it's
> happening. Maybe it happens, but it'd be good to see some data showing
> that, or a reproducer of some sort. But let's say it's real.
>
> I don't think we should use huge pages merely to ensure something is not
> swapped out. The "not swappable" is more of a limitation of huge pages,
> not an advantage. You can't just choose to make them swappable.
>
> Wouldn't it be better to keep using 4KB pages, but lock the memory using
> mlock/mlockall?

In my book, not being swappable is a win (it's hard for me to imagine
when it could be beneficial to swap out parts of s_b).

I was trying to think about it and also got those:

Anyway mlock() probably sounds like it, but e.g. Rocky 8.10 by default
has max locked memory (ulimit -l) as low as 64kB due to systemd's
DefaultLimitMEMLOCK, but Debian/Ubuntu have those at higher values.
Wasn't expecting that - those are bizzare low values. I think we would
need something like (10000*900)/1024/1024 or more, but with each
PGPROC on a separate page that would be even way more?

Another thing with 4kB pages: there's this big assumption now made
that once we arrive in InitProcess() we won't ever change NUMA node,
so we stick to the PGPROC from where we started (based on getcpu(2)).
Let's assume CPU scheduler reassigned us to differnt node, but we have
now this 4kB patch ready for PGPROC in theory and this means we would
need to rely on the NUMA autobalancing doing it's job to migrate that
4kB page from node to node (to get better local accesses instead of
remote ones).  The questions in my head are now like that:
- but we have asked intially asked those PGPROC pages to be localized
on certain node (they have policy), so they won't autobalance? We
would need to somewhere call getcpu() again notice the difference and
unlocalize (clear the NUMA/mbind() policy) for the PGPROC page?
- mlocked() as above says stick to physical RAM page (?) , so it won't move?
- after what time kernel's autobalancing would migrate that page since
switching the active CPU<->node? I mean do we execute enough reads on
this page?

BTW: to move this into pragmatic real, what's the most
one-liner/trivial way to exercise/stress PGPROC?

> >> The other thing I haven't thought about very much is determining on
> >> which CPUs/nodes the instance is allowed to run. I assume we'd start by
> >> simply inherit/determine that at the start through libnuma, not through
> >> some custom PG configuration (which the patch [2] proposed to do).
> >
> > 0. I think that we could do better, some counter arguments to
> > no-configuration-at-all:
> >
> > a. as Robert & Bertrand already put it there after review: let's say I
> > want just to run on NUMA #2 node, so here I would need to override
> > systemd's script ExecStart= to include that numactl (not elegant?). I
> > could also use `CPUAffinity=1,3,5,7..` but that's all, and it is even
> > less friendly. Also it probably requires root to edit/reload systemd,
> > while having GUC for this like in my proposal makes it more smooth (I
> > think?)
> >
> > b. wouldn't it be better if that stayed as drop-in rather than always
> > on? What if there's a problem, how do you disable those internal
> > optimizations if they do harm in some cases?  (or let's say I want to
> > play with MPOL_INTERLEAVE_WEIGHTED?). So at least boolean
> > numa_buffers_interleave would be nice?
> >
> > c. What if I want my standby (walreceiver+startup/recovery) to run
> > with NUMA affinity to get better performance (I'm not going to hack
> > around systemd script every time, but I could imagine changing
> > numa=X,Y,Z after restart/before promotion)
> >
> > d. Now if I would be forced for some reason to do that numactl(1)
> > voodoo, and use the those above mentioned overrides and PG wouldn't be
> > having GUC (let's say I would use `numactl
> > --weighted-interleave=0,1`), then:
> >
>
> I'm not against doing something like this, but I don't plan to do that
> in V1. I don't have a clear idea what configurability is actually
> needed, so it's likely I'd do the interface wrong.
>
> >> 2) v1-0002-NUMA-localalloc.patch
> >> This simply sets "localalloc" when initializing a backend, so that all
> >> memory allocated later is local, not interleaved. Initially this was
> >> necessary because the patch set the allocation policy to interleaving
> >> before initializing shared memory, and we didn't want to interleave the
> >> private memory. But that's no longer the case - the explicit mapping to
> >> nodes does not have this issue. I'm keeping the patch for convenience,
> >> it allows experimenting with numactl etc.
> >
> > .. .is not accurate anymore and we would require to have that in
> > (still with GUC) ?
> > Thoughts? I can add that mine part into Your's patches if you want.
> >
>
> I'm sorry, I don't understand what's the question :-(

That patch reference above, it was a chain of thought from step "d".
What I had in mind was that you cannot remove the patch
`v1-0002-NUMA-localalloc.patch` from the scope if forcing people to
use numactl by not having enough configurability on the PG side. That
is: if someone will have to use systemd+numactl
--interleave/--weighted-interleave then, he will also need to have a
way to use numa_localalloc=on (to override the new/user's policy
default, otherwise local mem allocations are also going to be
interleaved, and we are back to square one). Which brings me to a
point why instead of this toggle, should include the configuration
properly inside from start (it's not that hard apparently).

> > Way too quick review and some very fast benchmark probes, I've
> > concentrated only on v1-0001 and v1-0005 (efficiency of buffermgmt
> > would be too new topic for me), but let's start:
> >
> > 1. normal pgbench -S (still with just s_b@4GB), done many tries,
> > consistent benefit for the patch with like +8..10% boost on generic
> > run:
> >
[.. removed numbers]
>
> But this actually brings an interesting question. What exactly should we
> expect / demand from these patches? In my mind it'd primarily about
> predictability and stability of results.
>
> For example, the results should not depend on how was the database
> warmed up - was it done by a single backend or many backends? Was it
> restarted, or what? I could probably warmup the system very carefully to
> ensure it's balanced. The patches mean I don't need to be that careful.

Well, pretty much the same here. I was after minimizing "stddev" (to
have better predictability of results, especially across restarts) and
increasing available bandwidth [which is pretty much related]. Without
our NUMA work, PG can just put that s_b on any random node or spill
randomly from to another (depending on size of allocation request).

> >     So should I close https://commitfest.postgresql.org/patch/5703/
> > and you'll open a new one or should I just edit the #5703 and alter it
> > and add this thread too?
> >
>
> Good question. It's probably best to close the original entry as
> "withdrawn" and I'll add a new entry. Sounds OK?

Sure thing, marked it as `Returned with feedback`, this approach seems
to be much more advanced.

> > 3. Patch is not calling interleave on PQ shmem, do we want to add that
> > in as some next item like v1-0007? Question is whether OS interleaving
> > makes sense there ? I believe it does there, please see my thread
> > (NUMA_pq_cpu_pinning_results.txt), the issue is that PQ workers are
> > being spawned by postmaster and may end up on different NUMA nodes
> > randomly, so actually OS-interleaving that memory reduces jitter there
> > (AKA bandwidth-over-latency). My thinking is that one cannot expect
> > static/forced CPU-to-just-one-NUMA-node assignment for backend and
> > it's PQ workers, because it is impossible have always available CPU
> > power there in that NUMA node, so it might be useful to interleave
> > that shared mem there too (as separate patch item?)
> >
>
> Excellent question. I haven't thought about this at all. I agree it
> probably makes sense to interleave this memory, in some way. I don't
> know what's the perfect scheme, though.
>
> wild idea: Would it make sense to pin the workers to the same NUMA node
> as the leader? And allocate all memory only from that node?

I'm trying to convey exactly the opposite message or at least that it
might depend on configuration. Please see
https://www.postgresql.org/message-id/CAKZiRmxYMPbQ4WiyJWh%3DVuw_Ny%2BhLGH9_9FaacKRJvzZ-smm%2Bw%40mail.gmail.com
(btw it should read there that I don't indent spend a lot of thime on
PQ), but anyway: I think we should NOT pin the PQ workers the same
NODE as you do not know if there's CPU left there (same story as with
v1-0006 here).

I'm just proposing quick OS-based interleaving of PQ shm if using all
nodes, literally:

@@ -334,6 +336,13 @@ dsm_impl_posix(dsm_op op, dsm_handle handle, Size
request_size,
     }
     *mapped_address = address;
     *mapped_size = request_size;
+
+    /* We interleave memory only at creation time. */
+    if (op == DSM_OP_CREATE && numa->setting > NUMA_OFF) {
+        elog(DEBUG1, "interleaving shm mem @ %p size=%zu",
*mapped_address, *mapped_size);
+        pg_numa_interleave_memptr(*mapped_address, *mapped_size, numa->nodes);
+    }
+

Because then if memory is interleaved you have probably less variance
for memory access. But also from that previous thread:

"So if anything:
- latency-wise: it would be best to place leader+all PQ workers close
to s_b, provided s_b fits NUMA shared/huge page memory there and you
won't need more CPU than there's on that NUMA node... (assuming e.g.
hosting 4 DBs on 4-sockets each on it's own, it would be best to pin
everything including shm, but PQ workers too)
- capacity/TPS-wise or s_b > NUMA: just interleave to maximize
bandwidth and get uniform CPU performance out of this"

So wild idea was: maybe PQ shm interleaving should on NUMA
configuration (if intereavling to all nodes, then interleave normally,
but if configuration sets to just 1 NUMA node, it automatically binds
there -- there was '@' support for that in my patch).

> > 4 In BufferManagerShmemInit() you call numa_num_configured_nodes()
> > (also in v1-0005). My worry is should we may put some
> > known-limitations docs (?) from start and mention that
> > if the VM is greatly resized and NUMA numa nodes appear, they might
> > not be used until restart?
> >
>
> Yes, this is one thing I need some feedback on. The patches mostly
> assume there are no disabled nodes, that the set of allowed nodes does
> not change, etc. I think for V1 that's a reasonable limitation.

Sure!

> But let's say we want to relax this a bit. How do we learn about the
> change, after a node/CPU gets disabled? For some parts it's not that
> difficult (e.g. we can "remap" buffers/descriptors) in the background.
> But for other parts that's not practical. E.g. we can't rework how the
> PGPROC gets split.
>
> But while discussing this with Andres yesterday, he had an interesting
> suggestion - to always use e.g. 8 or 16 partitions, then partition this
> by NUMA node. So we'd have 16 partitions, and with 4 nodes the 0-3 would
> go to node 0, 4-7 would go to node 1, etc. The advantage is that if a
> node gets disabled, we can rebuild just this small "mapping" and not the
> 16 partitions. And the partitioning may be helpful even without NUMA.
>
> Still have to figure out the details, but seems it might help.

Right, no idea how the shared_memory remapping patch will work
(how/when the s_b change will be executed), but we could somehow mark
that number of NUMA zones could be rechecked during SIGHUP (?) and
then just simple compare check if old_numa_num_configured_nodes ==
new_numa_num_configured_nodes is true.

Anyway, I think it's way too advanced for now, don't you think? (like
CPU ballooning [s_b itself] is rare, and NUMA ballooning seems to be
super-wild-rare).

As for the rest, forgot to include this too: getcpu() - this really
needs a portable pg_getcpu() wrapper.

-J.

  11. Re: Adding basic NUMA awareness - Preliminary feedback and outline for an extensible approach

    Cédric Villemain <cedric.villemain@data-bene.io> — 2025-07-07T14:51:00Z

    >> * Others might use it to integrate PostgreSQL's own resources (e.g.,
>> "areas" of shared buffers) into policies.
>>
>> Hope this perspective is helpful.
> 
> Can you explain how you want to manage this by an extension defined at
> the SQL level, when most of this stuff has to be done when setting up
> shared memory, which is waaaay before we have any access to catalogs?

I should have said module instead, I didn't follow carefully but at some 
point there were discussion about shared buffers resized "on-line".
Anyway, it was just to give some few examples, maybe this one is to be 
considered later (I'm focused on cgroup/psi, and precisely reassigning 
PIDs as needed).


-- 
Cédric Villemain +33 6 20 30 22 52
https://www.Data-Bene.io
PostgreSQL Support, Expertise, Training, R&D

  12. Re: Adding basic NUMA awareness - Preliminary feedback and outline for an extensible approach

    Tomas Vondra <tomas@vondra.me> — 2025-07-07T22:35:43Z

    On 7/7/25 16:51, Cédric Villemain wrote:
>>> * Others might use it to integrate PostgreSQL's own resources (e.g.,
>>> "areas" of shared buffers) into policies.
>>>
>>> Hope this perspective is helpful.
>>
>> Can you explain how you want to manage this by an extension defined at
>> the SQL level, when most of this stuff has to be done when setting up
>> shared memory, which is waaaay before we have any access to catalogs?
> 
> I should have said module instead, I didn't follow carefully but at some
> point there were discussion about shared buffers resized "on-line".
> Anyway, it was just to give some few examples, maybe this one is to be
> considered later (I'm focused on cgroup/psi, and precisely reassigning
> PIDs as needed).
> 

I don't know. I have a hard time imagining what exactly would the
policies / profiles do exactly to respond to changes in the system
utilization. And why should that interfere with this patch ...

The main thing patch series aims to implement is partitioning different
pieces of shared memory (buffers, freelists, ...) to better work for
NUMA. I don't think there's that many ways to do this, and I doubt it
makes sense to make this easily customizable from external modules of
any kind. I can imagine providing some API allowing to isolate the
instance on selected NUMA nodes, but that's about it.

Yes, there's some relation to the online resizing of shared buffers, in
which case we need to "refresh" some of the information. But AFAICS it's
not very extensive (on top of what already needs to happen after the
resize), and it'd happen within the boundaries of the partitioning
scheme. There's not that much flexibility.

The last bit (pinning backends to a NUMA node) is experimental, and
mostly intended for easier evaluation of the earlier parts (e.g. to
limit the noise when processes get moved to a CPU from a different NUMA
node, and so on).


regards

-- 
Tomas Vondra

  13. Re: Adding basic NUMA awareness - Preliminary feedback and outline for an extensible approach

    Andres Freund <andres@anarazel.de> — 2025-07-07T23:25:35Z

    Hi,

On 2025-07-05 07:09:00 +0000, Cédric Villemain wrote:
> In my work on more careful PostgreSQL resource management, I've come to the
> conclusion that we should avoid pushing policy too deeply into the
> PostgreSQL core itself. Therefore, I'm quite skeptical about integrating
> NUMA-specific management directly into core PostgreSQL in such a way.

I think it's actually the opposite - whenever we pushed stuff like this
outside of core it has hurt postgres substantially. Not having replication in
core was a huge mistake. Not having HA management in core is probably the
biggest current adoption hurdle for postgres.

To deal better with NUMA we need to improve memory placement and various
algorithms, in an interrelated way - that's pretty much impossible to do
outside of core.

Greetings,

Andres Freund

  14. Re: Adding basic NUMA awareness - Preliminary feedback and outline for an extensible approach

    Cédric Villemain <cedric.villemain@data-bene.io> — 2025-07-08T01:47:00Z

    > On 7/7/25 16:51, Cédric Villemain wrote:
>>>> * Others might use it to integrate PostgreSQL's own resources (e.g.,
>>>> "areas" of shared buffers) into policies.
>>>>
>>>> Hope this perspective is helpful.
>>>
>>> Can you explain how you want to manage this by an extension defined at
>>> the SQL level, when most of this stuff has to be done when setting up
>>> shared memory, which is waaaay before we have any access to catalogs?
>>
>> I should have said module instead, I didn't follow carefully but at some
>> point there were discussion about shared buffers resized "on-line".
>> Anyway, it was just to give some few examples, maybe this one is to be
>> considered later (I'm focused on cgroup/psi, and precisely reassigning
>> PIDs as needed).
>>
> 
> I don't know. I have a hard time imagining what exactly would the
> policies / profiles do exactly to respond to changes in the system
> utilization. And why should that interfere with this patch ...
> 
> The main thing patch series aims to implement is partitioning different
> pieces of shared memory (buffers, freelists, ...) to better work for
> NUMA. I don't think there's that many ways to do this, and I doubt it
> makes sense to make this easily customizable from external modules of
> any kind. I can imagine providing some API allowing to isolate the
> instance on selected NUMA nodes, but that's about it.
> 
> Yes, there's some relation to the online resizing of shared buffers, in
> which case we need to "refresh" some of the information. But AFAICS it's
> not very extensive (on top of what already needs to happen after the
> resize), and it'd happen within the boundaries of the partitioning
> scheme. There's not that much flexibility.
> 
> The last bit (pinning backends to a NUMA node) is experimental, and
> mostly intended for easier evaluation of the earlier parts (e.g. to
> limit the noise when processes get moved to a CPU from a different NUMA
> node, and so on).

The backend pinning can be done by replacing your patch on proc.c to 
call an external profile manager doing exactly the same thing maybe ?

Similar to:
pmroutine = GetPmRoutineForInitProcess();
if (pmroutine != NULL &&
	pmroutine->init_process != NULL)
	pmroutine->init_process(MyProc);

...

pmroutine = GetPmRoutineForInitAuxilliary();
if (pmroutine != NULL &&
	pmroutine->init_auxilliary != NULL)
	pmroutine->init_auxilliary(MyProc);

Added on some rare places should cover most if not all the requirement 
around process placement (process_shared_preload_libraries() is called 
earlier in the process creation I believe).

-- 
Cédric Villemain +33 6 20 30 22 52
https://www.Data-Bene.io
PostgreSQL Support, Expertise, Training, R&D

  15. Re: Adding basic NUMA awareness - Preliminary feedback and outline for an extensible approach

    Cédric Villemain <cedric.villemain@data-bene.io> — 2025-07-08T01:55:00Z

    Hi Andres,

> Hi,
> 
> On 2025-07-05 07:09:00 +0000, Cédric Villemain wrote:
>> In my work on more careful PostgreSQL resource management, I've come to the
>> conclusion that we should avoid pushing policy too deeply into the
>> PostgreSQL core itself. Therefore, I'm quite skeptical about integrating
>> NUMA-specific management directly into core PostgreSQL in such a way.
> 
> I think it's actually the opposite - whenever we pushed stuff like this
> outside of core it has hurt postgres substantially. Not having replication in
> core was a huge mistake. Not having HA management in core is probably the
> biggest current adoption hurdle for postgres.
> 
> To deal better with NUMA we need to improve memory placement and various
> algorithms, in an interrelated way - that's pretty much impossible to do
> outside of core.

Except the backend pinning which is easy to achieve, thus my comment on 
the related patch.
I'm not claiming NUMA memory and all should be managed outside of core 
(though I didn't read other patches yet).

-- 
Cédric Villemain +33 6 20 30 22 52
https://www.Data-Bene.io
PostgreSQL Support, Expertise, Training, R&D

  16. Re: Adding basic NUMA awareness - Preliminary feedback and outline for an extensible approach

    Cédric Villemain <cedric.villemain@data-bene.io> — 2025-07-08T02:14:00Z

    




>> On 7/7/25 16:51, Cédric Villemain wrote:
>>>>> * Others might use it to integrate PostgreSQL's own resources (e.g.,
>>>>> "areas" of shared buffers) into policies.
>>>>>
>>>>> Hope this perspective is helpful.
>>>>
>>>> Can you explain how you want to manage this by an extension defined at
>>>> the SQL level, when most of this stuff has to be done when setting up
>>>> shared memory, which is waaaay before we have any access to catalogs?
>>>
>>> I should have said module instead, I didn't follow carefully but at some
>>> point there were discussion about shared buffers resized "on-line".
>>> Anyway, it was just to give some few examples, maybe this one is to be
>>> considered later (I'm focused on cgroup/psi, and precisely reassigning
>>> PIDs as needed).
>>>
>>
>> I don't know. I have a hard time imagining what exactly would the
>> policies / profiles do exactly to respond to changes in the system
>> utilization. And why should that interfere with this patch ...
>>
>> The main thing patch series aims to implement is partitioning different
>> pieces of shared memory (buffers, freelists, ...) to better work for
>> NUMA. I don't think there's that many ways to do this, and I doubt it
>> makes sense to make this easily customizable from external modules of
>> any kind. I can imagine providing some API allowing to isolate the
>> instance on selected NUMA nodes, but that's about it.
>>
>> Yes, there's some relation to the online resizing of shared buffers, in
>> which case we need to "refresh" some of the information. But AFAICS it's
>> not very extensive (on top of what already needs to happen after the
>> resize), and it'd happen within the boundaries of the partitioning
>> scheme. There's not that much flexibility.
>>
>> The last bit (pinning backends to a NUMA node) is experimental, and
>> mostly intended for easier evaluation of the earlier parts (e.g. to
>> limit the noise when processes get moved to a CPU from a different NUMA
>> node, and so on).
> 
> The backend pinning can be done by replacing your patch on proc.c to 
> call an external profile manager doing exactly the same thing maybe ?
> 
> Similar to:
> pmroutine = GetPmRoutineForInitProcess();
> if (pmroutine != NULL &&
>      pmroutine->init_process != NULL)
>      pmroutine->init_process(MyProc);
> 
> ...
> 
> pmroutine = GetPmRoutineForInitAuxilliary();
> if (pmroutine != NULL &&
>      pmroutine->init_auxilliary != NULL)
>      pmroutine->init_auxilliary(MyProc);
> 
> Added on some rare places should cover most if not all the requirement 
> around process placement (process_shared_preload_libraries() is called 
> earlier in the process creation I believe).
> 

After a first read I think this works for patches 002 and 005. For this 
last one, InitProcGlobal() may setup things as you do but then expose 
the choice a bit later, basically in places where you added the if 
condition on the GUC: numa_procs_interleave).


-- 
Cédric Villemain +33 6 20 30 22 52
https://www.Data-Bene.io
PostgreSQL Support, Expertise, Training, R&D

  17. Re: Adding basic NUMA awareness

    Andres Freund <andres@anarazel.de> — 2025-07-08T03:04:48Z

    Hi,

On 2025-07-04 13:05:05 +0200, Jakub Wartak wrote:
> On Tue, Jul 1, 2025 at 9:07 PM Tomas Vondra <tomas@vondra.me> wrote:
> > I don't think the splitting would actually make some things simpler, or
> > maybe more flexible - in particular, it'd allow us to enable huge pages
> > only for some regions (like shared buffers), and keep the small pages
> > e.g. for PGPROC. So that'd be good.
> 
> You have made assumption that this is good, but small pages (4KB) are
> not hugetlb, and are *swappable* (Transparent HP are swappable too,
> manually allocated ones as with mmap(MMAP_HUGETLB) are not)[1]. The
> most frequent problem I see these days are OOMs, and it makes me
> believe that making certain critical parts of shared memory being
> swappable just to make pagesize granular is possibly throwing the baby
> out with the bathwater. I'm thinking about bad situations like: some
> wrong settings of vm.swapiness that people keep (or distros keep?) and
> general inability of PG to restrain from allocating more memory in
> some cases.

The reason it would be advantageous to put something like the procarray onto
smaller pages is that otherwise the entire procarray (unless particularly
large) ends up on a single NUMA node, increasing the latency for backends on
every other numa node and increasing memory traffic on that node.

Greetings,

Andres Freund

  18. Re: Adding basic NUMA awareness

    Tomas Vondra <tomas@vondra.me> — 2025-07-08T12:27:12Z

    On 7/8/25 05:04, Andres Freund wrote:
> Hi,
> 
> On 2025-07-04 13:05:05 +0200, Jakub Wartak wrote:
>> On Tue, Jul 1, 2025 at 9:07 PM Tomas Vondra <tomas@vondra.me> wrote:
>>> I don't think the splitting would actually make some things simpler, or
>>> maybe more flexible - in particular, it'd allow us to enable huge pages
>>> only for some regions (like shared buffers), and keep the small pages
>>> e.g. for PGPROC. So that'd be good.
>>
>> You have made assumption that this is good, but small pages (4KB) are
>> not hugetlb, and are *swappable* (Transparent HP are swappable too,
>> manually allocated ones as with mmap(MMAP_HUGETLB) are not)[1]. The
>> most frequent problem I see these days are OOMs, and it makes me
>> believe that making certain critical parts of shared memory being
>> swappable just to make pagesize granular is possibly throwing the baby
>> out with the bathwater. I'm thinking about bad situations like: some
>> wrong settings of vm.swapiness that people keep (or distros keep?) and
>> general inability of PG to restrain from allocating more memory in
>> some cases.
> 
> The reason it would be advantageous to put something like the procarray onto
> smaller pages is that otherwise the entire procarray (unless particularly
> large) ends up on a single NUMA node, increasing the latency for backends on
> every other numa node and increasing memory traffic on that node.
> 

That's why the patch series splits the procarray into multiple pieces,
so that it can be properly distributed on multiple NUMA nodes even with
huge pages. It requires adjusting a couple places accessing the entries,
but it surprised me how limited the impact was.

If we could selectively use 4KB pages for parts of the shared memory,
maybe this wouldn't be necessary. But it's not too annoying.

The thing I'm not sure about is how much this actually helps with the
traffic between node. Sure, if we pick a PGPROC from the same node, and
the task does not get moved, it'll be local traffic. But if the task
moves, there'll be traffic. I don't have any estimates how often this
happens, e.g. for older tasks.

regards

-- 
Tomas Vondra

  19. Re: Adding basic NUMA awareness - Preliminary feedback and outline for an extensible approach

    Tomas Vondra <tomas@vondra.me> — 2025-07-08T12:34:59Z

    On 7/8/25 03:55, Cédric Villemain wrote:
> Hi Andres,
> 
>> Hi,
>>
>> On 2025-07-05 07:09:00 +0000, Cédric Villemain wrote:
>>> In my work on more careful PostgreSQL resource management, I've come
>>> to the
>>> conclusion that we should avoid pushing policy too deeply into the
>>> PostgreSQL core itself. Therefore, I'm quite skeptical about integrating
>>> NUMA-specific management directly into core PostgreSQL in such a way.
>>
>> I think it's actually the opposite - whenever we pushed stuff like this
>> outside of core it has hurt postgres substantially. Not having
>> replication in
>> core was a huge mistake. Not having HA management in core is probably the
>> biggest current adoption hurdle for postgres.
>>
>> To deal better with NUMA we need to improve memory placement and various
>> algorithms, in an interrelated way - that's pretty much impossible to do
>> outside of core.
> 
> Except the backend pinning which is easy to achieve, thus my comment on
> the related patch.
> I'm not claiming NUMA memory and all should be managed outside of core
> (though I didn't read other patches yet).
> 

But an "optimal backend placement" seems to very much depend on where we
placed the various pieces of shared memory. Which the external module
will have trouble following, I suspect.

I still don't have any idea what exactly would the external module do,
how would it decide where to place the backend. Can you describe some
use case with an example?

Assuming we want to actually pin tasks from within Postgres, what I
think might work is allowing modules to "advise" on where to place the
task. But the decision would still be done by core.


regards

-- 
Tomas Vondra

  20. Re: Adding basic NUMA awareness

    Andres Freund <andres@anarazel.de> — 2025-07-08T12:56:06Z

    Hi,

On 2025-07-08 14:27:12 +0200, Tomas Vondra wrote:
> On 7/8/25 05:04, Andres Freund wrote:
> > On 2025-07-04 13:05:05 +0200, Jakub Wartak wrote:
> > The reason it would be advantageous to put something like the procarray onto
> > smaller pages is that otherwise the entire procarray (unless particularly
> > large) ends up on a single NUMA node, increasing the latency for backends on
> > every other numa node and increasing memory traffic on that node.
> > 
> 
> That's why the patch series splits the procarray into multiple pieces,
> so that it can be properly distributed on multiple NUMA nodes even with
> huge pages. It requires adjusting a couple places accessing the entries,
> but it surprised me how limited the impact was.

Sure, you can do that, but it does mean that iterations over the procarray now
have an added level of indirection...


> The thing I'm not sure about is how much this actually helps with the
> traffic between node. Sure, if we pick a PGPROC from the same node, and
> the task does not get moved, it'll be local traffic. But if the task
> moves, there'll be traffic. I don't have any estimates how often this
> happens, e.g. for older tasks.

I think the most important bit is to not put everything onto one numa node,
otherwise the chance of increased latency for *everyone* due to the increased
memory contention is more likely to hurt.

Greetings,

Andres Freund

  21. Re: Adding basic NUMA awareness - Preliminary feedback and outline for an extensible approach

    Cédric Villemain <cedric.villemain@data-bene.io> — 2025-07-08T16:06:00Z

    




> On 7/8/25 03:55, Cédric Villemain wrote:
>> Hi Andres,
>>
>>> Hi,
>>>
>>> On 2025-07-05 07:09:00 +0000, Cédric Villemain wrote:
>>>> In my work on more careful PostgreSQL resource management, I've come
>>>> to the
>>>> conclusion that we should avoid pushing policy too deeply into the
>>>> PostgreSQL core itself. Therefore, I'm quite skeptical about integrating
>>>> NUMA-specific management directly into core PostgreSQL in such a way.
>>>
>>> I think it's actually the opposite - whenever we pushed stuff like this
>>> outside of core it has hurt postgres substantially. Not having
>>> replication in
>>> core was a huge mistake. Not having HA management in core is probably the
>>> biggest current adoption hurdle for postgres.
>>>
>>> To deal better with NUMA we need to improve memory placement and various
>>> algorithms, in an interrelated way - that's pretty much impossible to do
>>> outside of core.
>>
>> Except the backend pinning which is easy to achieve, thus my comment on
>> the related patch.
>> I'm not claiming NUMA memory and all should be managed outside of core
>> (though I didn't read other patches yet).
>>
> 
> But an "optimal backend placement" seems to very much depend on where we
> placed the various pieces of shared memory. Which the external module
> will have trouble following, I suspect.
> 
> I still don't have any idea what exactly would the external module do,
> how would it decide where to place the backend. Can you describe some
> use case with an example?
> 
> Assuming we want to actually pin tasks from within Postgres, what I
> think might work is allowing modules to "advise" on where to place the
> task. But the decision would still be done by core.

Possibly exactly what you're doing in proc.c when managing allocation of 
process, but not hardcoded in postgresql (patches 02, 05 and 06 are good 
candidates), I didn't get that they require information not available to 
any process executing code from a module.

Parts of your code where you assign/define policy could be in one or 
more relevant routines of a "numa profile manager", like in an 
initProcessRoutine(), and registered in pmroutine struct:

pmroutine = GetPmRoutineForInitProcess();
if (pmroutine != NULL &&
     pmroutine->init_process != NULL)
     pmroutine->init_process(MyProc);

This way it's easier to manage alternative policies, and also to be able 
to adjust when hardware and linux kernel changes.


-- 
Cédric Villemain +33 6 20 30 22 52
https://www.Data-Bene.io
PostgreSQL Support, Expertise, Training, R&D

  22. Re: Adding basic NUMA awareness - Preliminary feedback and outline for an extensible approach

    Tomas Vondra <tomas@vondra.me> — 2025-07-08T21:26:06Z

    
On 7/8/25 18:06, Cédric Villemain wrote:
> 
> 
> 
> 
> 
> 
>> On 7/8/25 03:55, Cédric Villemain wrote:
>>> Hi Andres,
>>>
>>>> Hi,
>>>>
>>>> On 2025-07-05 07:09:00 +0000, Cédric Villemain wrote:
>>>>> In my work on more careful PostgreSQL resource management, I've come
>>>>> to the
>>>>> conclusion that we should avoid pushing policy too deeply into the
>>>>> PostgreSQL core itself. Therefore, I'm quite skeptical about
>>>>> integrating
>>>>> NUMA-specific management directly into core PostgreSQL in such a way.
>>>>
>>>> I think it's actually the opposite - whenever we pushed stuff like this
>>>> outside of core it has hurt postgres substantially. Not having
>>>> replication in
>>>> core was a huge mistake. Not having HA management in core is
>>>> probably the
>>>> biggest current adoption hurdle for postgres.
>>>>
>>>> To deal better with NUMA we need to improve memory placement and
>>>> various
>>>> algorithms, in an interrelated way - that's pretty much impossible
>>>> to do
>>>> outside of core.
>>>
>>> Except the backend pinning which is easy to achieve, thus my comment on
>>> the related patch.
>>> I'm not claiming NUMA memory and all should be managed outside of core
>>> (though I didn't read other patches yet).
>>>
>>
>> But an "optimal backend placement" seems to very much depend on where we
>> placed the various pieces of shared memory. Which the external module
>> will have trouble following, I suspect.
>>
>> I still don't have any idea what exactly would the external module do,
>> how would it decide where to place the backend. Can you describe some
>> use case with an example?
>>
>> Assuming we want to actually pin tasks from within Postgres, what I
>> think might work is allowing modules to "advise" on where to place the
>> task. But the decision would still be done by core.
> 
> Possibly exactly what you're doing in proc.c when managing allocation of
> process, but not hardcoded in postgresql (patches 02, 05 and 06 are good
> candidates), I didn't get that they require information not available to
> any process executing code from a module.
> 

Well, it needs to understand how some other stuff (especially PGPROC
entries) is distributed between nodes. I'm not sure how much of this
internal information we want to expose outside core ...

> Parts of your code where you assign/define policy could be in one or
> more relevant routines of a "numa profile manager", like in an
> initProcessRoutine(), and registered in pmroutine struct:
> 
> pmroutine = GetPmRoutineForInitProcess();
> if (pmroutine != NULL &&
>     pmroutine->init_process != NULL)
>     pmroutine->init_process(MyProc);
> 
> This way it's easier to manage alternative policies, and also to be able
> to adjust when hardware and linux kernel changes.
> 

I'm not against making this extensible, in some way. But I still
struggle to imagine a reasonable alternative policy, where the external
module gets the same information and ends up with a different decision.

So what would the alternate policy look like? What use case would the
module be supporting?


regards

-- 
Tomas Vondra

  23. Re: Adding basic NUMA awareness - Preliminary feedback and outline for an extensible approach

    Cédric Villemain <cedric.villemain@data-bene.io> — 2025-07-09T06:40:00Z

    > On 7/8/25 18:06, Cédric Villemain wrote:
>>
>>
>>
>>
>>
>>
>>> On 7/8/25 03:55, Cédric Villemain wrote:
>>>> Hi Andres,
>>>>
>>>>> Hi,
>>>>>
>>>>> On 2025-07-05 07:09:00 +0000, Cédric Villemain wrote:
>>>>>> In my work on more careful PostgreSQL resource management, I've come
>>>>>> to the
>>>>>> conclusion that we should avoid pushing policy too deeply into the
>>>>>> PostgreSQL core itself. Therefore, I'm quite skeptical about
>>>>>> integrating
>>>>>> NUMA-specific management directly into core PostgreSQL in such a way.
>>>>>
>>>>> I think it's actually the opposite - whenever we pushed stuff like this
>>>>> outside of core it has hurt postgres substantially. Not having
>>>>> replication in
>>>>> core was a huge mistake. Not having HA management in core is
>>>>> probably the
>>>>> biggest current adoption hurdle for postgres.
>>>>>
>>>>> To deal better with NUMA we need to improve memory placement and
>>>>> various
>>>>> algorithms, in an interrelated way - that's pretty much impossible
>>>>> to do
>>>>> outside of core.
>>>>
>>>> Except the backend pinning which is easy to achieve, thus my comment on
>>>> the related patch.
>>>> I'm not claiming NUMA memory and all should be managed outside of core
>>>> (though I didn't read other patches yet).
>>>>
>>>
>>> But an "optimal backend placement" seems to very much depend on where we
>>> placed the various pieces of shared memory. Which the external module
>>> will have trouble following, I suspect.
>>>
>>> I still don't have any idea what exactly would the external module do,
>>> how would it decide where to place the backend. Can you describe some
>>> use case with an example?
>>>
>>> Assuming we want to actually pin tasks from within Postgres, what I
>>> think might work is allowing modules to "advise" on where to place the
>>> task. But the decision would still be done by core.
>>
>> Possibly exactly what you're doing in proc.c when managing allocation of
>> process, but not hardcoded in postgresql (patches 02, 05 and 06 are good
>> candidates), I didn't get that they require information not available to
>> any process executing code from a module.
>>
> 
> Well, it needs to understand how some other stuff (especially PGPROC
> entries) is distributed between nodes. I'm not sure how much of this
> internal information we want to expose outside core ...
> 
>> Parts of your code where you assign/define policy could be in one or
>> more relevant routines of a "numa profile manager", like in an
>> initProcessRoutine(), and registered in pmroutine struct:
>>
>> pmroutine = GetPmRoutineForInitProcess();
>> if (pmroutine != NULL &&
>>      pmroutine->init_process != NULL)
>>      pmroutine->init_process(MyProc);
>>
>> This way it's easier to manage alternative policies, and also to be able
>> to adjust when hardware and linux kernel changes.
>>
> 
> I'm not against making this extensible, in some way. But I still
> struggle to imagine a reasonable alternative policy, where the external
> module gets the same information and ends up with a different decision.
> 
> So what would the alternate policy look like? What use case would the
> module be supporting?


That's the whole point: there are very distinct usages of PostgreSQL in 
the field. And maybe not all of them will require the policy defined by 
PostgreSQL core.

May I ask the reverse: what prevent external modules from taking those 
decisions ? There are already a lot of area where external code can take 
over PostgreSQL processing, like Neon is doing.

There are some very early processing for memory setup that I can see as 
a current blocker, and here I'd refer a more compliant NUMA api as 
proposed by Jakub so it's possible to arrange based on workload, 
hardware configuration or other matters. Reworking to get distinct 
segment and all as you do is great, and combo of both approach probably 
of great interest. There is also this weighted interleave discussed and 
probably much more to come in this area in Linux.

I think some points raised already about possible distinct policies, I 
am precisely claiming that it is hard to come with one good policy with 
limited setup options, thus requirement to keep that flexible enough 
(hooks, api, 100 GUc ?).

There is an EPYC story here also, given the NUMA setup can vary 
depending on BIOS setup, associated NUMA policy must probably take that 
into account (L3 can be either real cache or 4 extra "local" NUMA nodes 
- with highly distinct access cost from a RAM module).
Does that change how PostgreSQL will place memory and process? Is it 
important or of interest ?


-- 
Cédric Villemain +33 6 20 30 22 52
https://www.Data-Bene.io
PostgreSQL Support, Expertise, Training, R&D

  24. Re: Adding basic NUMA awareness - Preliminary feedback and outline for an extensible approach

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2025-07-09T08:09:16Z

    Hi,

On Wed, Jul 09, 2025 at 06:40:00AM +0000, Cédric Villemain wrote:
> > On 7/8/25 18:06, Cédric Villemain wrote:
> > I'm not against making this extensible, in some way. But I still
> > struggle to imagine a reasonable alternative policy, where the external
> > module gets the same information and ends up with a different decision.
> > 
> > So what would the alternate policy look like? What use case would the
> > module be supporting?
> 
> 
> That's the whole point: there are very distinct usages of PostgreSQL in the
> field. And maybe not all of them will require the policy defined by
> PostgreSQL core.
> 
> May I ask the reverse: what prevent external modules from taking those
> decisions ? There are already a lot of area where external code can take
> over PostgreSQL processing, like Neon is doing.
> 
> There are some very early processing for memory setup that I can see as a
> current blocker, and here I'd refer a more compliant NUMA api as proposed by
> Jakub so it's possible to arrange based on workload, hardware configuration
> or other matters. Reworking to get distinct segment and all as you do is
> great, and combo of both approach probably of great interest.

I think that Tomas's approach helps to have more "predictable" performance
expectations, I mean more consistent over time, fewer "surprises".

While your approach (and Jakub's one)) could help to get performance gains
depending on a "known" context (so less generic).

So, probably having both could make sense but I think that they serve different
purposes.

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  25. Re: Adding basic NUMA awareness

    Jakub Wartak <jakub.wartak@enterprisedb.com> — 2025-07-09T10:04:00Z

    On Tue, Jul 8, 2025 at 2:56 PM Andres Freund <andres@anarazel.de> wrote:
>
> Hi,
>
> On 2025-07-08 14:27:12 +0200, Tomas Vondra wrote:
> > On 7/8/25 05:04, Andres Freund wrote:
> > > On 2025-07-04 13:05:05 +0200, Jakub Wartak wrote:
> > > The reason it would be advantageous to put something like the procarray onto
> > > smaller pages is that otherwise the entire procarray (unless particularly
> > > large) ends up on a single NUMA node, increasing the latency for backends on
> > > every other numa node and increasing memory traffic on that node.
> > >

Sure thing, I fully understand the motivation and underlying reason
(without claiming that I understand the exact memory access patterns
that involve procarray/PGPROC/etc and hotspots involved from PG side).
Any single-liner pgbench help for how to really easily stress the
PGPROC or procarray?

> > That's why the patch series splits the procarray into multiple pieces,
> > so that it can be properly distributed on multiple NUMA nodes even with
> > huge pages. It requires adjusting a couple places accessing the entries,
> > but it surprised me how limited the impact was.

Yes, and we are discussing if it is worth getting into smaller pages
for such usecases (e.g. 4kB ones without hugetlb with 2MB hugepages or
what more even more waste 1GB hugetlb if we dont request 2MB for some
small structs: btw, we have ability to select MAP_HUGE_2MB vs
MAP_HUGE_1GB). I'm thinking about two problems:
- 4kB are swappable and mlock() potentially (?) disarms NUMA autobalacning
- using libnuma often leads to MPOL_BIND which disarms NUMA
autobalancing, BUT apparently there are set_mempolicy(2)/mbind(2) and
since 5.12+ kernel they can take additional flag
MPOL_F_NUMA_BALANCING(!), so this looks like it has potential to move
memory anyway (if way too many tasks are relocated, so would be
memory?). It is available only in recent libnuma as
numa_set_membind_balancing(3), but sadly there's no way via libnuma to
do mbind(MPOL_F_NUMA_BALANCING) for a specific addr only? I mean it
would have be something like MPOL_F_NUMA_BALANCING | MPOL_PREFERRED?
(select one node from many for each node while still allowing
balancing?), but in [1][2] (2024) it is stated that "It's not
legitimate (yet) to use MPOL_PREFERRED + MPOL_F_NUMA_BALANCING.", but
maybe stuff has been improved since then.

Something like:
PGPROC/procarray 2MB page for node#1 - mbind(addr1,
MPOL_F_NUMA_BALANCING | MPOL_PREFERRED, [0,1]);
PGPROC/procarray 2MB page for node#2 - mbind(addr2,
MPOL_F_NUMA_BALANCING | MPOL_PREFERRED, [1,0]);

> Sure, you can do that, but it does mean that iterations over the procarray now
> have an added level of indirection...

So the most efficient would be the old-way (no indirections) vs
NUMA-way? Can this be done without #ifdefs at all?

> > The thing I'm not sure about is how much this actually helps with the
> > traffic between node. Sure, if we pick a PGPROC from the same node, and
> > the task does not get moved, it'll be local traffic. But if the task
> > moves, there'll be traffic.

With MPOL_F_NUMA_BALANCING, that should "auto-tune" in the worst case?

> > I don't have any estimates how often this happens, e.g. for older tasks.

We could measure, kernel 6.16+ has per PID numa_task_migrated in
/proc/{PID}/sched , but I assume we would have to throw backends >>
VCPUs at it, to simulate reality and do some "waves" between different
activity periods of certain pools (I can imagine worst case scenario:
a) pgbench "a" open $VCPU connections, all idle, with scripto to sleep
for a while
b) pgbench "b" open some $VCPU new connections to some other DB, all
active from start (tpcbb or readonly)
c) manually ping CPUs using taskset for each PID all from "b" to
specific NUMA node #2 -- just to simulate unfortunate app working on
every 2nd conn
d) pgbench "a" starts working and hits CPU imbalance -- e.g. NUMA node
#1 is idle, #2 is full, CPU scheduler starts puting "a" backends on
CPUs from #1 , and we should notice PIDs being migrated)

> I think the most important bit is to not put everything onto one numa node,
> otherwise the chance of increased latency for *everyone* due to the increased
> memory contention is more likely to hurt.

-J.

p.s. I hope i did write in an understandable way, because I had many
interruptions, so if anything is unclear please let me know.

[1] - https://lkml.org/lkml/2024/7/3/352
[2] - https://lkml.rescloud.iu.edu/2402.2/03227.html

  26. Re: Adding basic NUMA awareness

    Andres Freund <andres@anarazel.de> — 2025-07-09T16:35:13Z

    Hi,

On 2025-07-02 14:36:31 +0200, Tomas Vondra wrote:
> On 7/2/25 13:37, Ashutosh Bapat wrote:
> > On Wed, Jul 2, 2025 at 12:37 AM Tomas Vondra <tomas@vondra.me> wrote:
> >>
> >>
> >> 3) v1-0003-freelist-Don-t-track-tail-of-a-freelist.patch
> >>
> >> Minor optimization. Andres noticed we're tracking the tail of buffer
> >> freelist, without using it. So the patch removes that.
> >>
> >
> > The patches for resizing buffers use the lastFreeBuffer to add new
> > buffers to the end of free list when expanding it. But we could as
> > well add it at the beginning of the free list.

Yea, I don't see any point in adding buffers to the tail instead of to the
front. We probably want more recently used buffers at the front, since they
(and the associated BufferDesc) are more likely to be in a CPU cache.


> > This patch seems almost independent of the rest of the patches. Do you
> > need it in the rest of the patches? I understand that those patches
> > don't need to worry about maintaining lastFreeBuffer after this patch.
> > Is there any other effect?
> >
> > If we are going to do this, let's do it earlier so that buffer
> > resizing patches can be adjusted.
> >
>
> My patches don't particularly rely on this bit, it would work even with
> lastFreeBuffer. I believe Andres simply noticed the current code does
> not use lastFreeBuffer, it just maintains is, so he removed that as an
> optimization.

Optimiziation / simplification.  When building multiple freelists it was
harder to maintain the tail pointer, and since it was never used...

+1 to just applying that part.


> I don't know how significant is the improvement, but if it's measurable we
> could just do that independently of our patches.

I doubt it's really an improvement in any realistic scenario, but it's also
not a regression in any way, since it's never used...


FWIW, I've started to wonder if we shouldn't just get rid of the freelist
entirely. While clocksweep is perhaps minutely slower in a single thread than
the freelist, clock sweep scales *considerably* better [1]. As it's rather
rare to be bottlenecked on clock sweep speed for a single thread (rather then
IO or memory copy overhead), I think it's worth favoring clock sweep.

Also needing to switch between getting buffers from the freelist and the sweep
makes the code more expensive.  I think just having the buffer in the sweep,
with a refcount / usagecount of zero would suffice.

That seems particularly advantageous if we invest energy in making the clock
sweep deal well with NUMA systems, because we don't need have both a NUMA
aware freelist and a NUMA aware clock sweep.

Greetings,

Andres Freund

[1]

A single pg_prewarm of a large relation shows a difference between using the
freelist and not that's around the noise level, whereas 40 parallel
pg_prewarms of seperate relations is over 5x faster when disabling the
freelist.

For the test:
- I modified pg_buffercache_evict_* to put buffers onto the freelist

- Ensured all of shared buffers is allocated by querying
  pg_shmem_allocations_numa, as otherwise the workload is dominated by the
  kernel zeroing out buffers

- used shared_buffers bigger than the data

- data for single threaded is 9.7GB, data for the parallel case is 40
  relations of 610MB each.

- in the single threaded case I pinned postgres to a single core, to make sure
  core-to-core variation doesn't play a role

- single threaded case

  c=1 && psql -Xq -c "select pg_buffercache_evict_all()" -c 'SELECT numa_node, sum(size), count(*) FROM pg_shmem_allocations_numa WHERE size != 0 GROUP BY numa_node;' && pgbench -n -P1 -c$c -j$c -f <(echo "SELECT pg_prewarm('copytest_large');") -t1

  concurrent case:

  c=40 && psql -Xq -c "select pg_buffercache_evict_all()" -c 'SELECT numa_node, sum(size), count(*) FROM pg_shmem_allocations_numa WHERE size != 0 GROUP BY numa_node;' && pgbench -n -P1 -c$c -j$c -f <(echo "SELECT pg_prewarm('copytest_:client_id');") -t1

  27. Re: Adding basic NUMA awareness

    Greg Burd <greg@burd.me> — 2025-07-09T16:55:51Z

    
On Jul 9 2025, at 12:35 pm, Andres Freund <andres@anarazel.de> wrote:

> FWIW, I've started to wonder if we shouldn't just get rid of the freelist
> entirely. While clocksweep is perhaps minutely slower in a single
> thread than
> the freelist, clock sweep scales *considerably* better [1]. As it's rather
> rare to be bottlenecked on clock sweep speed for a single thread
> (rather then
> IO or memory copy overhead), I think it's worth favoring clock sweep.

Hey Andres, thanks for spending time on this.  I've worked before on
freelist implementations (last one in LMDB) and I think you're onto
something.  I think it's an innovative idea and that the speed
difference will either be lost in the noise or potentially entirely
mitigated by avoiding duplicate work.

> Also needing to switch between getting buffers from the freelist and
> the sweep
> makes the code more expensive.  I think just having the buffer in the sweep,
> with a refcount / usagecount of zero would suffice.

If you're not already coding this, I'll jump in. :)
 
> That seems particularly advantageous if we invest energy in making the clock
> sweep deal well with NUMA systems, because we don't need have both a NUMA
> aware freelist and a NUMA aware clock sweep.

100% agree here, very clever approach adapting clock sweep to a NUMA world.

best.

-greg

> 
> Greetings,
> 
> Andres Freund

  28. Re: Adding basic NUMA awareness

    Andres Freund <andres@anarazel.de> — 2025-07-09T17:13:04Z

    Hi,

On 2025-07-09 12:04:00 +0200, Jakub Wartak wrote:
> On Tue, Jul 8, 2025 at 2:56 PM Andres Freund <andres@anarazel.de> wrote:
> > On 2025-07-08 14:27:12 +0200, Tomas Vondra wrote:
> > > On 7/8/25 05:04, Andres Freund wrote:
> > > > On 2025-07-04 13:05:05 +0200, Jakub Wartak wrote:
> > > > The reason it would be advantageous to put something like the procarray onto
> > > > smaller pages is that otherwise the entire procarray (unless particularly
> > > > large) ends up on a single NUMA node, increasing the latency for backends on
> > > > every other numa node and increasing memory traffic on that node.
> > > >
> 
> Sure thing, I fully understand the motivation and underlying reason
> (without claiming that I understand the exact memory access patterns
> that involve procarray/PGPROC/etc and hotspots involved from PG side).
> Any single-liner pgbench help for how to really easily stress the
> PGPROC or procarray?

Unfortunately it's probably going to be slightly more complicated workloads
that show the effect - the very simplest cases don't go iterate through the
procarray itself anymore.


> > > That's why the patch series splits the procarray into multiple pieces,
> > > so that it can be properly distributed on multiple NUMA nodes even with
> > > huge pages. It requires adjusting a couple places accessing the entries,
> > > but it surprised me how limited the impact was.
> 
> Yes, and we are discussing if it is worth getting into smaller pages
> for such usecases (e.g. 4kB ones without hugetlb with 2MB hugepages or
> what more even more waste 1GB hugetlb if we dont request 2MB for some
> small structs: btw, we have ability to select MAP_HUGE_2MB vs
> MAP_HUGE_1GB). I'm thinking about two problems:
> - 4kB are swappable and mlock() potentially (?) disarms NUMA autobalacning

I'm not really bought into this being a problem.  If your system has enough
pressure to swap out the PGPROC array, you're so hosed that this won't make a
difference.


> - using libnuma often leads to MPOL_BIND which disarms NUMA
> autobalancing, BUT apparently there are set_mempolicy(2)/mbind(2) and
> since 5.12+ kernel they can take additional flag
> MPOL_F_NUMA_BALANCING(!), so this looks like it has potential to move
> memory anyway (if way too many tasks are relocated, so would be
> memory?). It is available only in recent libnuma as
> numa_set_membind_balancing(3), but sadly there's no way via libnuma to
> do mbind(MPOL_F_NUMA_BALANCING) for a specific addr only? I mean it
> would have be something like MPOL_F_NUMA_BALANCING | MPOL_PREFERRED?
> (select one node from many for each node while still allowing
> balancing?), but in [1][2] (2024) it is stated that "It's not
> legitimate (yet) to use MPOL_PREFERRED + MPOL_F_NUMA_BALANCING.", but
> maybe stuff has been improved since then.
> 
> Something like:
> PGPROC/procarray 2MB page for node#1 - mbind(addr1,
> MPOL_F_NUMA_BALANCING | MPOL_PREFERRED, [0,1]);
> PGPROC/procarray 2MB page for node#2 - mbind(addr2,
> MPOL_F_NUMA_BALANCING | MPOL_PREFERRED, [1,0]);

I'm rather doubtful that it's a good idea to combine numa awareness with numa
balancing. Numa balancing adds latency and makes it much more expensive for
userspace to act in a numa aware way, since it needs to regularly update its
knowledge about where memory resides.


> > Sure, you can do that, but it does mean that iterations over the procarray now
> > have an added level of indirection...
> 
> So the most efficient would be the old-way (no indirections) vs
> NUMA-way? Can this be done without #ifdefs at all?

If we used 4k pages for the procarray we would just have ~4 procs on one page,
if that range were marked as interleaved, it'd probably suffice.


> > > The thing I'm not sure about is how much this actually helps with the
> > > traffic between node. Sure, if we pick a PGPROC from the same node, and
> > > the task does not get moved, it'll be local traffic. But if the task
> > > moves, there'll be traffic.
> 
> With MPOL_F_NUMA_BALANCING, that should "auto-tune" in the worst case?

I doubt that NUMA balancing is going to help a whole lot here, there are too
many procs on one page for that to be helpful.  One thing that might be worth
doing is to *increase* the size of PGPROC by moving other pieces of data that
are keyed by ProcNumber into PGPROC.

I think the main thing to avoid is the case where all of PGPROC, buffer
mapping table, ... resides on one NUMA node (e.g. because it's the one
postmaster was scheduled on), as the increased memory traffic will lead to
queries on that node being slower than the other node.

Greetings,

Andres Freund

  29. Re: Adding basic NUMA awareness

    Andres Freund <andres@anarazel.de> — 2025-07-09T17:23:06Z

    Hi,

On 2025-07-09 12:55:51 -0400, Greg Burd wrote:
> On Jul 9 2025, at 12:35 pm, Andres Freund <andres@anarazel.de> wrote:
>
> > FWIW, I've started to wonder if we shouldn't just get rid of the freelist
> > entirely. While clocksweep is perhaps minutely slower in a single
> > thread than
> > the freelist, clock sweep scales *considerably* better [1]. As it's rather
> > rare to be bottlenecked on clock sweep speed for a single thread
> > (rather then
> > IO or memory copy overhead), I think it's worth favoring clock sweep.
>
> Hey Andres, thanks for spending time on this.  I've worked before on
> freelist implementations (last one in LMDB) and I think you're onto
> something.  I think it's an innovative idea and that the speed
> difference will either be lost in the noise or potentially entirely
> mitigated by avoiding duplicate work.

Agreed. FWIW, just using clock sweep actually makes things like DROP TABLE
perform better because it doesn't need to maintain the freelist anymore...


> > Also needing to switch between getting buffers from the freelist and
> > the sweep
> > makes the code more expensive.  I think just having the buffer in the sweep,
> > with a refcount / usagecount of zero would suffice.
>
> If you're not already coding this, I'll jump in. :)

My experimental patch is literally a four character addition ;), namely adding
"0 &&" to the relevant code in StrategyGetBuffer().

Obviously a real patch would need to do some more work than that.  Feel free
to take on that project, I am not planning on tackling that in near term.


There's other things around this that could use some attention. It's not hard
to see clock sweep be a bottleneck in concurrent workloads - partially due to
the shared maintenance of the clock hand. A NUMAed clock sweep would address
that. However, we also maintain StrategyControl->numBufferAllocs, which is a
significant contention point and would not necessarily be removed by a
NUMAificiation of the clock sweep.

Greetings,

Andres Freund

  30. Re: Adding basic NUMA awareness - Preliminary feedback and outline for an extensible approach

    Andres Freund <andres@anarazel.de> — 2025-07-09T17:57:36Z

    Hi,

On 2025-07-08 16:06:00 +0000, Cédric Villemain wrote:
> > Assuming we want to actually pin tasks from within Postgres, what I
> > think might work is allowing modules to "advise" on where to place the
> > task. But the decision would still be done by core.
> 
> Possibly exactly what you're doing in proc.c when managing allocation of
> process, but not hardcoded in postgresql (patches 02, 05 and 06 are good
> candidates), I didn't get that they require information not available to any
> process executing code from a module.

> Parts of your code where you assign/define policy could be in one or more
> relevant routines of a "numa profile manager", like in an
> initProcessRoutine(), and registered in pmroutine struct:
> 
> pmroutine = GetPmRoutineForInitProcess();
> if (pmroutine != NULL &&
>     pmroutine->init_process != NULL)
>     pmroutine->init_process(MyProc);
> 
> This way it's easier to manage alternative policies, and also to be able to
> adjust when hardware and linux kernel changes.

I am doubtful this makes sense - as you can see patch 05 needs to change a
fair bit of core code to make this work, there's no way we can delegate much
of that to an extension.

But even if it's doable, I think it's *very* premature to focus on such
extensibility at this point - we need to get the basics into a mergeable
state, if you then want to argue for adding extensibility, we can do that at
this stage.  Trying to design this for extensibility from the get go, where
that extensibility is very unlikely to be used widely, seems rather likely to
just tank this entire project without getting us anything in return.

Greetings,

Andres Freund

  31. Re: Adding basic NUMA awareness

    Andres Freund <andres@anarazel.de> — 2025-07-09T19:42:26Z

    Hi,

Thanks for working on this!  I think it's an area we have long neglected...


On 2025-07-01 21:07:00 +0200, Tomas Vondra wrote:
> Each patch has a numa_ GUC, intended to enable/disable that part. This
> is meant to make development easier, not as a final interface. I'm not
> sure how exactly that should look. It's possible some combinations of
> GUCs won't work, etc.

Wonder if some of it might be worth putting into a multi-valued GUC (like
debug_io_direct).



> 1) v1-0001-NUMA-interleaving-buffers.patch
>
> This is the main thing when people think about NUMA - making sure the
> shared buffers are allocated evenly on all the nodes, not just on a
> single node (which can happen easily with warmup). The regular memory
> interleaving would address this, but it also has some disadvantages.
>
> Firstly, it's oblivious to the contents of the shared memory segment,
> and we may not want to interleave everything. It's also oblivious to
> alignment of the items (a buffer can easily end up "split" on multiple
> NUMA nodes), or relationship between different parts (e.g. there's a
> BufferBlock and a related BufferDescriptor, and those might again end up
> on different nodes).

Two more disadvantages:

With OS interleaving postgres doesn't (not easily at least) know about what
maps to what, which means postgres can't do stuff like numa aware buffer
replacement.

With OS interleaving the interleaving is "too fine grained", with pages being
mapped at each page boundary, making it less likely for things like one
strategy ringbuffer to reside on a single numa node.


I wonder if we should *increase* the size of shared_buffers whenever huge
pages are in use and there's padding space due to the huge page
boundaries. Pretty pointless to waste that memory if we can instead use if for
the buffer pool.  Not that big a deal with 2MB huge pages, but with 1GB huge
pages...



> 4) v1-0004-NUMA-partition-buffer-freelist.patch
>
> Right now we have a single freelist, and in busy instances that can be
> quite contended. What's worse, the freelist may trash between different
> CPUs, NUMA nodes, etc. So the idea is to have multiple freelists on
> subsets of buffers. The patch implements multiple strategies how the
> list can be split (configured using "numa_partition_freelist" GUC), for
> experimenting:
>
> * node - One list per NUMA node. This is the most natural option,
> because we now know which buffer is on which node, so we can ensure a
> list for a node only has buffers from that list.
>
> * cpu - One list per CPU. Pretty simple, each CPU gets it's own list.
>
> * pid - Similar to "cpu", but the processes are mapped to lists based on
> PID, not CPU ID.
>
> * none - nothing, sigle freelist
>
> Ultimately, I think we'll want to go with "node", simply because it
> aligns with the buffer interleaving. But there are improvements needed.

I think we might eventually want something more granular than just "node" -
the freelist (and the clock sweep) can become a contention point even within
one NUMA node.  I'm imagining something like an array of freelists/clocksweep
states, where the current numa node selects a subset of the array and the cpu
is used to choose the entry within that list.

But we can do that later, that should be a fairly simple extension of what
you're doing.



> The other missing part is clocksweep - there's still just a single
> instance of clocksweep, feeding buffers to all the freelists. But that's
> clearly a problem, because the clocksweep returns buffers from all NUMA
> nodes. The clocksweep really needs to be partitioned the same way as a
> freelists, and each partition will operate on a subset of buffers (from
> the right NUMA node).
>
> I do have a separate experimental patch doing something like that, I
> need to make it part of this branch.

I'm really curious about that patch, as I wrote elsewhere in this thread, I
think we should just get rid of the freelist alltogether.  Even if we don't do
so, in a steady state system the clock sweep is commonly much more important
than the freelist...


> 5) v1-0005-NUMA-interleave-PGPROC-entries.patch
>
> Another area that seems like it might benefit from NUMA is PGPROC, so I
> gave it a try. It turned out somewhat challenging. Similarly to buffers
> we have two pieces that need to be located in a coordinated way - PGPROC
> entries and fast-path arrays. But we can't use the same approach as for
> buffers/descriptors, because
>
> (a) Neither of those pieces aligns with memory page size (PGPROC is
> ~900B, fast-path arrays are variable length).

> (b) We could pad PGPROC entries e.g. to 1KB, but that'd still require
> rather high max_connections before we use multiple huge pages.

We should probably pad them regardless? Right now sizeof(PGPROC) happens to be
multiple of 64 (i.e. the most common cache line size), but that hasn't always
been the case, and isn't the case on systems with 128 bit cachelines like
common ARMv8 systems.  And having one cacheline hold one backends fast path
states and another backend's xmin doesn't sound like a recipe for good
performance.

Seems like we should also do some reordering of the contents within PGPROC. We
have e.g. have very frequently changing data (->waitStatus, ->lwWaiting) in
the same caceheline as almost immutable data (->pid, ->pgxactoff,
->databaseId,).


> So what I did instead is splitting the whole PGPROC array into one array
> per NUMA node, and one array for auxiliary processes and 2PC xacts. So
> with 4 NUMA nodes there are 5 separate arrays, for example. Each array
> is a multiple of memory pages, so we may waste some of the memory. But
> that's simply how NUMA works - page granularity.

Theoretically we could use the "padding" memory at the end of each NUMA node's
PGPROC array to for the 2PC entries, for those we presumably don't care for
locality.  Not sure it's worth the complexity though.


For a while I thought I had a better solution: Given that we're going to waste
all the "padding" memory, why not just oversize the PGPROC array so that it
spans the required number of NUMA nodes?

The problem is that that would lead to ProcNumbers to get much larger, and we
do have other arrays that are keyed by ProcNumber. Which probably makes this
not so great an idea.


> This however makes one particular thing harder - in a couple places we
> accessed PGPROC entries through PROC_HDR->allProcs, which was pretty
> much just one large array. And GetNumberFromPGProc() relied on array
> arithmetics to determine procnumber. With the array partitioned, this
> can't work the same way.
>
> But there's a simple solution - if we turn allProcs into an array of
> *pointers* to PGPROC arrays, there's no issue. All the places need a
> pointer anyway. And then we need an explicit procnumber field in PGPROC,
> instead of calculating it.
>
> There's a chance this have negative impact on code that accessed PGPROC
> very often, but so far I haven't seen such cases. But if you can come up
> with such examples, I'd like to see those.

I'd not be surprised if there were overhead, adding a level of indirection to
things like ProcArrayGroupClearXid(), GetVirtualXIDsDelayingChkpt(),
SignalVirtualTransaction() probably won't be free.

BUT: For at least some of these a better answer might be to add additional
"dense" arrays like we have for xids etc, so they don't need to trawl through
PGPROCs.


> There's another detail - when obtaining a PGPROC entry in InitProcess(),
> we try to get an entry from the same NUMA node. And only if that doesn't
> work, we grab the first one from the list (there's still just one PGPROC
> freelist, I haven't split that - maybe we should?).

I guess it might be worth partitioning the freelist, iterating through a few
thousand links just to discover that there's no free proc on the current numa
node, while holding a spinlock, doesn't sound great. Even if it's likely
rarely a huge issue compared to other costs.


> The other thing I haven't thought about very much is determining on
> which CPUs/nodes the instance is allowed to run. I assume we'd start by
> simply inherit/determine that at the start through libnuma, not through
> some custom PG configuration (which the patch [2] proposed to do).

That seems like the right thing to me.


One thing that this patchset afaict doesn't address so far is that there is a
fair bit of other important shared memory that this patch doesn't set up
intelligently e.g. the buffer mapping table itself (but there are loads of
other cases). Because we touch a lot of that memory during startup, most it
will be allocated on whatever NUMA node postmaster was scheduled.  I suspect
that the best we can do for parts of shared memory where we don't have
explicit NUMA awareness is to default to an interleave policy.


> From 9712e50d6d15c18ea2c5fcf457972486b0d4ef53 Mon Sep 17 00:00:00 2001
> From: Tomas Vondra <tomas@vondra.me>
> Date: Tue, 6 May 2025 21:12:21 +0200
> Subject: [PATCH v1 1/6] NUMA: interleaving buffers
>
> Ensure shared buffers are allocated from all NUMA nodes, in a balanced
> way, instead of just using the node where Postgres initially starts, or
> where the kernel decides to migrate the page, etc. With pre-warming
> performed by a single backend, this can easily result in severely
> unbalanced memory distribution (with most from a single NUMA node).
>
> The kernel would eventually move some of the memory to other nodes
> (thanks to zone_reclaim), but that tends to take a long time. So this
> patch improves predictability, reduces the time needed for warmup
> during benchmarking, etc.  It's less dependent on what the CPU
> scheduler does, etc.

FWIW, I don't think zone_reclaim_mode will commonly do that? Even if enabled,
which I don't think it is anymore by default.  At least huge pages can't be
reclaimed by the kernel, but even when not using huge pages, I think the only
scenario where that would happen is if shared_buffers were swapped out.

Numa balancing might eventually "fix" such an imbalance though.


> diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c
> index ed1dc488a42..2ad34624c49 100644
> --- a/src/backend/storage/buffer/buf_init.c
> +++ b/src/backend/storage/buffer/buf_init.c
> @@ -14,9 +14,17 @@
>   */
>  #include "postgres.h"
>
> +#ifdef USE_LIBNUMA
> +#include <numa.h>
> +#include <numaif.h>
> +#endif
> +

I wonder how much of this we should try to put into port/pg_numa.c. Having
direct calls to libnuma code all over the backend will make it rather hard to
add numa awareness for hypothetical platforms not using libnuma compatible
interfaces.


> +/* number of buffers allocated on the same NUMA node */
> +static int64 numa_chunk_buffers = -1;

Given that NBuffers is a 32bit quantity, this probably doesn't need to be
64bit...  Anyway, I'm not going to review on that level going forward, the
patch is probably in too early a state for that.



> @@ -71,18 +92,80 @@ BufferManagerShmemInit(void)
>  				foundDescs,
>  				foundIOCV,
>  				foundBufCkpt;
> +	Size		mem_page_size;
> +	Size		buffer_align;
> +
> +	/*
> +	 * XXX A bit weird. Do we need to worry about postmaster? Could this even
> +	 * run outside postmaster? I don't think so.

It can run in single user mode - but that shouldn't prevent us from using
pg_get_shmem_pagesize().


> +	 * XXX Another issue is we may get different values than when sizing the
> +	 * the memory, because at that point we didn't know if we get huge pages,
> +	 * so we assumed we will. Shouldn't cause crashes, but we might allocate
> +	 * shared memory and then not use some of it (because of the alignment
> +	 * that we don't actually need). Not sure about better way, good for now.
> +	 */

Ugh, not seeing a great way to deal with that either.


> +	 * XXX Maybe with (mem_page_size > PG_IO_ALIGN_SIZE), we don't need to
> +	 * align to mem_page_size? Especially for very large huge pages (e.g. 1GB)
> +	 * that doesn't seem quite worth it. Maybe we should simply align to
> +	 * BLCKSZ, so that buffers don't get split? Still, we might interfere with
> +	 * other stuff stored in shared memory that we want to allocate on a
> +	 * particular NUMA node (e.g. ProcArray).
> +	 *
> +	 * XXX Maybe with "too large" huge pages we should just not do this, or
> +	 * maybe do this only for sufficiently large areas (e.g. shared buffers,
> +	 * but not ProcArray).

I think that's right - there's no point in using 1GB pages for anything other
than shared_buffers, we should allocate shared_buffers separately.




> +/*
> + * Determine the size of memory page.
> + *
> + * XXX This is a bit tricky, because the result depends at which point we call
> + * this. Before the allocation we don't know if we succeed in allocating huge
> + * pages - but we have to size everything for the chance that we will. And then
> + * if the huge pages fail (with 'huge_pages=try'), we'll use the regular memory
> + * pages. But at that point we can't adjust the sizing.
> + *
> + * XXX Maybe with huge_pages=try we should do the sizing twice - first with
> + * huge pages, and if that fails, then without them. But not for this patch.
> + * Up to this point there was no such dependency on huge pages.

Doing it twice sounds somewhat nasty - but perhaps we could just have the
shmem size infrastructure compute two different numbers, one for use with huge
pages and one without?



> +static int64
> +choose_chunk_buffers(int NBuffers, Size mem_page_size, int num_nodes)
> +{
> +	int64		num_items;
> +	int64		max_items;
> +
> +	/* make sure the chunks will align nicely */
> +	Assert(BLCKSZ % sizeof(BufferDescPadded) == 0);
> +	Assert(mem_page_size % sizeof(BufferDescPadded) == 0);
> +	Assert(((BLCKSZ % mem_page_size) == 0) || ((mem_page_size % BLCKSZ) == 0));
> +
> +	/*
> +	 * The minimum number of items to fill a memory page with descriptors and
> +	 * blocks. The NUMA allocates memory in pages, and we need to do that for
> +	 * both buffers and descriptors.
> +	 *
> +	 * In practice the BLCKSZ doesn't really matter, because it's much larger
> +	 * than BufferDescPadded, so the result is determined buffer descriptors.
> +	 * But it's clearer this way.
> +	 */
> +	num_items = Max(mem_page_size / sizeof(BufferDescPadded),
> +					mem_page_size / BLCKSZ);
> +
> +	/*
> +	 * We shouldn't use chunks larger than NBuffers/num_nodes, because with
> +	 * larger chunks the last NUMA node would end up with much less memory (or
> +	 * no memory at all).
> +	 */
> +	max_items = (NBuffers / num_nodes);
> +
> +	/*
> +	 * Did we already exceed the maximum desirable chunk size? That is, will
> +	 * the last node get less than one whole chunk (or no memory at all)?
> +	 */
> +	if (num_items > max_items)
> +		elog(WARNING, "choose_chunk_buffers: chunk items exceeds max (%ld > %ld)",
> +			 num_items, max_items);
> +
> +	/* grow the chunk size until we hit the max limit. */
> +	while (2 * num_items <= max_items)
> +		num_items *= 2;

Something around this logic leads to a fair bit of imbalance - I started postgres with
huge_page_size=1GB, shared_buffers=4GB on a 2 node system and that results in

postgres[4188255][1]=# SELECT * FROM pg_shmem_allocations_numa WHERE name in ('Buffer Blocks', 'Buffer Descriptors');
┌────────────────────┬───────────┬────────────┐
│        name        │ numa_node │    size    │
├────────────────────┼───────────┼────────────┤
│ Buffer Blocks      │         0 │ 5368709120 │
│ Buffer Blocks      │         1 │ 1073741824 │
│ Buffer Descriptors │         0 │ 1073741824 │
│ Buffer Descriptors │         1 │ 1073741824 │
└────────────────────┴───────────┴────────────┘
(4 rows)


With shared_buffers=8GB postgres failed to start, even though 16 1GB huge
pages are available, as 18GB were requested.

After increasing the limit, the top allocations were as follows:
postgres[4189384][1]=# SELECT * FROM pg_shmem_allocations ORDER BY allocated_size DESC LIMIT 5;
┌──────────────────────┬─────────────┬────────────┬────────────────┐
│         name         │     off     │    size    │ allocated_size │
├──────────────────────┼─────────────┼────────────┼────────────────┤
│ Buffer Blocks        │  1192223104 │ 9663676416 │     9663676416 │
│ PGPROC structures    │ 10970279808 │ 3221733342 │     3221733376 │
│ Fast-Path Lock Array │ 14192013184 │ 3221396544 │     3221396608 │
│ Buffer Descriptors   │    51372416 │ 1140850688 │     1140850688 │
│ (null)               │ 17468590976 │  785020032 │      785020032 │
└──────────────────────┴─────────────┴────────────┴────────────────┘

With a fair bit of imbalance:
postgres[4189384][1]=# SELECT * FROM pg_shmem_allocations_numa WHERE name in ('Buffer Blocks', 'Buffer Descriptors');
┌────────────────────┬───────────┬────────────┐
│        name        │ numa_node │    size    │
├────────────────────┼───────────┼────────────┤
│ Buffer Blocks      │         0 │ 8589934592 │
│ Buffer Blocks      │         1 │ 2147483648 │
│ Buffer Descriptors │         0 │          0 │
│ Buffer Descriptors │         1 │ 2147483648 │
└────────────────────┴───────────┴────────────┘
(4 rows)

Note that the buffer descriptors are all on node 1.


> +/*
> + * Calculate the NUMA node for a given buffer.
> + */
> +int
> +BufferGetNode(Buffer buffer)
> +{
> +	/* not NUMA interleaving */
> +	if (numa_chunk_buffers == -1)
> +		return -1;
> +
> +	return (buffer / numa_chunk_buffers) % numa_nodes;
> +}

FWIW, this is likely rather expensive - when not a compile time constant,
divisions and modulo can take a fair number of cycles.



> +/*
> + * pg_numa_interleave_memory
> + *		move memory to different NUMA nodes in larger chunks
> + *
> + * startptr - start of the region (should be aligned to page size)
> + * endptr - end of the region (doesn't need to be aligned)
> + * mem_page_size - size of the memory page size
> + * chunk_size - size of the chunk to move to a single node (should be multiple
> + *              of page size
> + * num_nodes - number of nodes to allocate memory to
> + *
> + * XXX Maybe this should use numa_tonode_memory and numa_police_memory instead?
> + * That might be more efficient than numa_move_pages, as it works on larger
> + * chunks of memory, not individual system pages, I think.
> + *
> + * XXX The "interleave" name is not quite accurate, I guess.
> + */
> +static void
> +pg_numa_interleave_memory(char *startptr, char *endptr,
> +						  Size mem_page_size, Size chunk_size,
> +						  int num_nodes)
> +{

Seems like this should be in pg_numa.c?




> diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
> index 69b6a877dc9..c07de903f76 100644
> --- a/src/bin/pgbench/pgbench.c
> +++ b/src/bin/pgbench/pgbench.c

I assume those changes weren't intentionally part of this patch...


> From 6505848ac8359c8c76dfbffc7150b6601ab07601 Mon Sep 17 00:00:00 2001
> From: Tomas Vondra <tomas@vondra.me>
> Date: Thu, 22 May 2025 18:38:41 +0200
> Subject: [PATCH v1 4/6] NUMA: partition buffer freelist
>
> Instead of a single buffer freelist, partition into multiple smaller
> lists, to reduce lock contention, and to spread the buffers over all
> NUMA nodes more evenly.
>
> There are four strategies, specified by GUC numa_partition_freelist
>
> * none - single long freelist, should work just like now
>
> * node - one freelist per NUMA node, with only buffers from that node
>
> * cpu - one freelist per CPU
>
> * pid - freelist determined by PID (same number of freelists as 'cpu')
>
> When allocating a buffer, it's taken from the correct freelist (e.g.
> same NUMA node).
>
> Note: This is (probably) more important than partitioning ProcArray.

>
> +/*
> + * Represents one freelist partition.
> + */
> +typedef struct BufferStrategyFreelist
> +{
> +	/* Spinlock: protects the values below */
> +	slock_t		freelist_lock;
> +
> +	/*
> +	 * XXX Not sure why this needs to be aligned like this. Need to ask
> +	 * Andres.
> +	 */
> +	int			firstFreeBuffer __attribute__((aligned(64)));	/* Head of list of
> +																 * unused buffers */
> +
> +	/* Number of buffers consumed from this list. */
> +	uint64		consumed;
> +}			BufferStrategyFreelist;

I think this might be a leftover from measuring performance of a *non*
partitioned freelist.  I saw unnecessar contention between

  BufferStrategyControl->{nextVictimBuffer,buffer_strategy_lock,numBufferAllocs}

and was testing what effect the simplest avoidance scheme has.

I don't this should be part of this patchset.



>
>  /*
>   * The shared freelist control information.
> @@ -39,8 +66,6 @@ typedef struct
>  	 */
>  	pg_atomic_uint32 nextVictimBuffer;
>
> -	int			firstFreeBuffer;	/* Head of list of unused buffers */
> -
>  	/*
>  	 * Statistics.  These counters should be wide enough that they can't
>  	 * overflow during a single bgwriter cycle.
> @@ -51,13 +76,27 @@ typedef struct
>  	/*
>  	 * Bgworker process to be notified upon activity or -1 if none. See
>  	 * StrategyNotifyBgWriter.
> +	 *
> +	 * XXX Not sure why this needs to be aligned like this. Need to ask
> +	 * Andres. Also, shouldn't the alignment be specified after, like for
> +	 * "consumed"?
>  	 */
> -	int			bgwprocno;
> +	int			__attribute__((aligned(64))) bgwprocno;
> +
> +	BufferStrategyFreelist freelists[FLEXIBLE_ARRAY_MEMBER];
>  } BufferStrategyControl;

Here the reason was that it's silly to put almost-readonly data (like
bgwprocno) onto the same cacheline as very frequently modified data like
->numBufferAllocs.  That causes unnecessary cache misses in many
StrategyGetBuffer() calls, as another backend's StrategyGetBuffer() will
always have modified ->numBufferAllocs and either ->buffer_strategy_lock or
->nextVictimBuffer.


>
> +static BufferStrategyFreelist *
> +ChooseFreeList(void)
> +{
> +	unsigned	cpu;
> +	unsigned	node;
> +	int			rc;
> +
> +	int			freelist_idx;
> +
> +	/* freelist not partitioned, return the first (and only) freelist */
> +	if (numa_partition_freelist == FREELIST_PARTITION_NONE)
> +		return &StrategyControl->freelists[0];
> +
> +	/*
> +	 * freelist is partitioned, so determine the CPU/NUMA node, and pick a
> +	 * list based on that.
> +	 */
> +	rc = getcpu(&cpu, &node);
> +	if (rc != 0)
> +		elog(ERROR, "getcpu failed: %m");

Probably should put this into somewhere abstracted away...


> +	/*
> +	 * Pick the freelist, based on CPU, NUMA node or process PID. This matches
> +	 * how we built the freelists above.
> +	 *
> +	 * XXX Can we rely on some of the values (especially strategy_nnodes) to
> +	 * be a power-of-2? Then we could replace the modulo with a mask, which is
> +	 * likely more efficient.
> +	 */
> +	switch (numa_partition_freelist)
> +	{
> +		case FREELIST_PARTITION_CPU:
> +			freelist_idx = cpu % strategy_ncpus;

As mentioned earlier, modulo is rather expensive for something executed so
frequently...

> +			break;
> +
> +		case FREELIST_PARTITION_NODE:
> +			freelist_idx = node % strategy_nnodes;
> +			break;

Here we shouldn't need modulo, right?


> +
> +		case FREELIST_PARTITION_PID:
> +			freelist_idx = MyProcPid % strategy_ncpus;
> +			break;
> +
> +		default:
> +			elog(ERROR, "unknown freelist partitioning value");
> +	}
> +
> +	return &StrategyControl->freelists[freelist_idx];
> +}


>  	/* size of lookup hash table ... see comment in StrategyInitialize */
>  	size = add_size(size, BufTableShmemSize(NBuffers + NUM_BUFFER_PARTITIONS));
>
>  	/* size of the shared replacement strategy control block */
> -	size = add_size(size, MAXALIGN(sizeof(BufferStrategyControl)));
> +	size = add_size(size, MAXALIGN(offsetof(BufferStrategyControl, freelists)));
> +
> +	/*
> +	 * Allocate one frelist per CPU. We might use per-node freelists, but the
> +	 * assumption is the number of CPUs is less than number of NUMA nodes.
> +	 *
> +	 * FIXME This assumes the we have more CPUs than NUMA nodes, which seems
> +	 * like a safe assumption. But maybe we should calculate how many elements
> +	 * we actually need, depending on the GUC? Not a huge amount of memory.

FWIW, I don't think that's a safe assumption anymore. With CXL we can get a)
PCIe attached memory and b) remote memory as a separate NUMA nodes, and that
very well could end up as more NUMA nodes than cores.


Ugh, -ETOOLONG. Gotta schedule some other things...


Greetings,

Andres Freund

  32. Re: Adding basic NUMA awareness

    Jakub Wartak <jakub.wartak@enterprisedb.com> — 2025-07-10T10:15:52Z

    On Wed, Jul 9, 2025 at 7:13 PM Andres Freund <andres@anarazel.de> wrote:

> > Yes, and we are discussing if it is worth getting into smaller pages
> > for such usecases (e.g. 4kB ones without hugetlb with 2MB hugepages or
> > what more even more waste 1GB hugetlb if we dont request 2MB for some
> > small structs: btw, we have ability to select MAP_HUGE_2MB vs
> > MAP_HUGE_1GB). I'm thinking about two problems:
> > - 4kB are swappable and mlock() potentially (?) disarms NUMA autobalacning
>
> I'm not really bought into this being a problem.  If your system has enough
> pressure to swap out the PGPROC array, you're so hosed that this won't make a
> difference.

OK I need to bend here, yet still part of me believes that the
situation where we have hugepages (for 'Buffer Blocks') and yet some
smaller more, but way critical structs are more likely to be swapped
out due to pressure of some backend-gone-wild random mallocs() is
unhealthy to me (especially the fact the OS might prefer swapping on
per node rather than global picture)

> I'm rather doubtful that it's a good idea to combine numa awareness with numa
> balancing. Numa balancing adds latency and makes it much more expensive for
> userspace to act in a numa aware way, since it needs to regularly update its
> knowledge about where memory resides.

Well the problem is that backends come here and go to random CPUs
often (migrated++ on very high backend counts and non-uniform
workloads in terms of backend-CPU usage), but the autobalancing
doesn't need to be on or off for everything. It could be autobalancing
for a certain memory region and it is not affecting the app in any way
(well, other than those minor page faulting, literally ).

> If we used 4k pages for the procarray we would just have ~4 procs on one page,
> if that range were marked as interleaved, it'd probably suffice.

OK, this sounds like the best and simplest proposal to me, yet the
patch doesn't do OS-based interleaving for those today. Gonna try that
mlock() sooner or later... ;)

-J.

  33. Re: Adding basic NUMA awareness

    Jakub Wartak <jakub.wartak@enterprisedb.com> — 2025-07-10T10:16:41Z

    On Wed, Jul 9, 2025 at 9:42 PM Andres Freund <andres@anarazel.de> wrote:

> On 2025-07-01 21:07:00 +0200, Tomas Vondra wrote:
> > Each patch has a numa_ GUC, intended to enable/disable that part. This
> > is meant to make development easier, not as a final interface. I'm not
> > sure how exactly that should look. It's possible some combinations of
> > GUCs won't work, etc.
>
> Wonder if some of it might be worth putting into a multi-valued GUC (like
> debug_io_direct).

Long-term or for experimentation? Also please see below as it is related:

[..]

> FWIW, I don't think that's a safe assumption anymore. With CXL we can get a)
> PCIe attached memory and b) remote memory as a separate NUMA nodes, and that
> very well could end up as more NUMA nodes than cores.

In my earlier apparently very way too naive approach, I've tried to
handle this CXL scenario, but I'm afraid this cannot be done without
further configuration, please see review/use cases [1] and [2]

-J.

[1] https://www.postgresql.org/message-id/attachment/178119/v4-0001-Add-capability-to-interleave-shared-memory-across.patch
- just see sgml/GUC and we have numa_parse_nodestring(3)
[2] https://www.postgresql.org/message-id/aAKPMrX1Uq6quKJy%40ip-10-97-1-34.eu-west-3.compute.internal

  34. Re: Adding basic NUMA awareness

    Greg Burd <greg@burd.me> — 2025-07-10T12:13:43Z

    
> On Jul 9, 2025, at 1:23 PM, Andres Freund <andres@anarazel.de> wrote:
> 
> Hi,
> 
> On 2025-07-09 12:55:51 -0400, Greg Burd wrote:
>> On Jul 9 2025, at 12:35 pm, Andres Freund <andres@anarazel.de> wrote:
>> 
>>> FWIW, I've started to wonder if we shouldn't just get rid of the freelist
>>> entirely. While clocksweep is perhaps minutely slower in a single
>>> thread than
>>> the freelist, clock sweep scales *considerably* better [1]. As it's rather
>>> rare to be bottlenecked on clock sweep speed for a single thread
>>> (rather then
>>> IO or memory copy overhead), I think it's worth favoring clock sweep.
>> 
>> Hey Andres, thanks for spending time on this.  I've worked before on
>> freelist implementations (last one in LMDB) and I think you're onto
>> something.  I think it's an innovative idea and that the speed
>> difference will either be lost in the noise or potentially entirely
>> mitigated by avoiding duplicate work.
> 
> Agreed. FWIW, just using clock sweep actually makes things like DROP TABLE
> perform better because it doesn't need to maintain the freelist anymore...
> 
> 
>>> Also needing to switch between getting buffers from the freelist and
>>> the sweep
>>> makes the code more expensive.  I think just having the buffer in the sweep,
>>> with a refcount / usagecount of zero would suffice.
>> 
>> If you're not already coding this, I'll jump in. :)
> 
> My experimental patch is literally a four character addition ;), namely adding
> "0 &&" to the relevant code in StrategyGetBuffer().
> 
> Obviously a real patch would need to do some more work than that.  Feel free
> to take on that project, I am not planning on tackling that in near term.
> 

I started on this last night, making good progress.  Thanks for the inspiration.  I'll create a new thread to track the work and cross-reference when I have something reasonable to show (hopefully later today).

> There's other things around this that could use some attention. It's not hard
> to see clock sweep be a bottleneck in concurrent workloads - partially due to
> the shared maintenance of the clock hand. A NUMAed clock sweep would address
> that.

Working on it.  Other than NUMA-fying clocksweep there is a function have_free_buffer() that might be a tad tricky to re-implement efficiently and/or make NUMA aware.  Or maybe I can remove that too?  It is used in autoprewarm.c and possibly other extensions, but no where else in core.

> However, we also maintain StrategyControl->numBufferAllocs, which is a
> significant contention point and would not necessarily be removed by a
> NUMAificiation of the clock sweep.

Yep, I noted this counter and its potential for contention too.  Fortunately, it seems like it is only used so that "bgwriter can estimate the rate of buffer consumption" which to me opens the door to a less accurate partitioned counter, perhaps something lock-free (no mutex/CAS) that is bucketed then combined when read.

A quick look at bufmgr.c indicates that recent_allocs (which is StrategyControl->numBufferAllocs) is used to track a "moving average" and other voodoo there I've yet to fully grok.  Any thoughts on this approximate count approach?

Also, what are your thoughts on updating the algorithm to CLOCK-Pro [1] while I'm there?  I guess I'd have to try it out, measure it a lot and see if there are any material benefits.  Maybe I'll keep that for a future patch, or at least layer it... back to work!

> Greetings,
> 
> Andres Freund

best.

-greg

[1] https://www.usenix.org/legacy/publications/library/proceedings/usenix05/tech/general/full_papers/jiang/jiang_html/html.html

  35. Re: Adding basic NUMA awareness

    Bertrand Drouvot <bertranddrouvot.pg@gmail.com> — 2025-07-10T14:17:21Z

    Hi,

On Wed, Jul 09, 2025 at 03:42:26PM -0400, Andres Freund wrote:
> Hi,
> 
> Thanks for working on this!

Indeed, thanks!

> On 2025-07-01 21:07:00 +0200, Tomas Vondra wrote:
> > 1) v1-0001-NUMA-interleaving-buffers.patch
> >
> > This is the main thing when people think about NUMA - making sure the
> > shared buffers are allocated evenly on all the nodes, not just on a
> > single node (which can happen easily with warmup). The regular memory
> > interleaving would address this, but it also has some disadvantages.
> >
> > Firstly, it's oblivious to the contents of the shared memory segment,
> > and we may not want to interleave everything. It's also oblivious to
> > alignment of the items (a buffer can easily end up "split" on multiple
> > NUMA nodes), or relationship between different parts (e.g. there's a
> > BufferBlock and a related BufferDescriptor, and those might again end up
> > on different nodes).
> 
> Two more disadvantages:
> 
> With OS interleaving postgres doesn't (not easily at least) know about what
> maps to what, which means postgres can't do stuff like numa aware buffer
> replacement.
> 
> With OS interleaving the interleaving is "too fine grained", with pages being
> mapped at each page boundary, making it less likely for things like one
> strategy ringbuffer to reside on a single numa node.

> > There's a secondary benefit of explicitly assigning buffers to nodes,
> > using this simple scheme - it allows quickly determining the node ID
> > given a buffer ID. This is helpful later, when building freelist.

I do think this is a big advantage as compare to the OS interleaving.

> I wonder if we should *increase* the size of shared_buffers whenever huge
> pages are in use and there's padding space due to the huge page
> boundaries. Pretty pointless to waste that memory if we can instead use if for
> the buffer pool.  Not that big a deal with 2MB huge pages, but with 1GB huge
> pages...

I think that makes sense, except maybe for operations that need to scan
the whole buffer pool (i.e related to BUF_DROP_FULL_SCAN_THRESHOLD)?

> > 5) v1-0005-NUMA-interleave-PGPROC-entries.patch
> >
> > Another area that seems like it might benefit from NUMA is PGPROC, so I
> > gave it a try. It turned out somewhat challenging. Similarly to buffers
> > we have two pieces that need to be located in a coordinated way - PGPROC
> > entries and fast-path arrays. But we can't use the same approach as for
> > buffers/descriptors, because
> >
> > (a) Neither of those pieces aligns with memory page size (PGPROC is
> > ~900B, fast-path arrays are variable length).
> 
> > (b) We could pad PGPROC entries e.g. to 1KB, but that'd still require
> > rather high max_connections before we use multiple huge pages.
> 
> Right now sizeof(PGPROC) happens to be multiple of 64 (i.e. the most common
> cache line size)

Oh right, it's currently 832 bytes and the patch extends that to 840 bytes.

With a bit of reordering:

diff --git a/src/include/storage/proc.h b/src/include/storage/proc.h
index 5cb1632718e..2ed2f94202a 100644
--- a/src/include/storage/proc.h
+++ b/src/include/storage/proc.h
@@ -194,8 +194,6 @@ struct PGPROC
                                                                 * vacuum must not remove tuples deleted by
                                                                 * xid >= xmin ! */

-       int                     procnumber;             /* index in ProcGlobal->allProcs */
-
        int                     pid;                    /* Backend's process ID; 0 if prepared xact */

        int                     pgxactoff;              /* offset into various ProcGlobal->arrays with
@@ -243,6 +241,7 @@ struct PGPROC

        /* Support for condition variables. */
        proclist_node cvWaitLink;       /* position in CV wait list */
+       int                     procnumber;             /* index in ProcGlobal->allProcs */

        /* Info about lock the process is currently waiting for, if any. */
        /* waitLock and waitProcLock are NULL if not currently waiting. */
@@ -268,6 +267,7 @@ struct PGPROC
         */
        XLogRecPtr      waitLSN;                /* waiting for this LSN or higher */
        int                     syncRepState;   /* wait state for sync rep */
+       int                     numa_node;
        dlist_node      syncRepLinks;   /* list link if process is in syncrep queue */

        /*
@@ -321,9 +321,6 @@ struct PGPROC
        PGPROC     *lockGroupLeader;    /* lock group leader, if I'm a member */
        dlist_head      lockGroupMembers;       /* list of members, if I'm a leader */
        dlist_node      lockGroupLink;  /* my member link, if I'm a member */
-
-       /* NUMA node */
-       int                     numa_node;
 };

That could be back to 832 (the order does not make sense logically speaking
though).

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

  36. Re: Adding basic NUMA awareness - Preliminary feedback and outline for an extensible approach

    Tomas Vondra <tomas@vondra.me> — 2025-07-10T15:20:50Z

    
On 7/9/25 08:40, Cédric Villemain wrote:
>> On 7/8/25 18:06, Cédric Villemain wrote:
>>>
>>>
>>>
>>>
>>>
>>>
>>>> On 7/8/25 03:55, Cédric Villemain wrote:
>>>>> Hi Andres,
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> On 2025-07-05 07:09:00 +0000, Cédric Villemain wrote:
>>>>>>> In my work on more careful PostgreSQL resource management, I've come
>>>>>>> to the
>>>>>>> conclusion that we should avoid pushing policy too deeply into the
>>>>>>> PostgreSQL core itself. Therefore, I'm quite skeptical about
>>>>>>> integrating
>>>>>>> NUMA-specific management directly into core PostgreSQL in such a
>>>>>>> way.
>>>>>>
>>>>>> I think it's actually the opposite - whenever we pushed stuff like
>>>>>> this
>>>>>> outside of core it has hurt postgres substantially. Not having
>>>>>> replication in
>>>>>> core was a huge mistake. Not having HA management in core is
>>>>>> probably the
>>>>>> biggest current adoption hurdle for postgres.
>>>>>>
>>>>>> To deal better with NUMA we need to improve memory placement and
>>>>>> various
>>>>>> algorithms, in an interrelated way - that's pretty much impossible
>>>>>> to do
>>>>>> outside of core.
>>>>>
>>>>> Except the backend pinning which is easy to achieve, thus my
>>>>> comment on
>>>>> the related patch.
>>>>> I'm not claiming NUMA memory and all should be managed outside of core
>>>>> (though I didn't read other patches yet).
>>>>>
>>>>
>>>> But an "optimal backend placement" seems to very much depend on
>>>> where we
>>>> placed the various pieces of shared memory. Which the external module
>>>> will have trouble following, I suspect.
>>>>
>>>> I still don't have any idea what exactly would the external module do,
>>>> how would it decide where to place the backend. Can you describe some
>>>> use case with an example?
>>>>
>>>> Assuming we want to actually pin tasks from within Postgres, what I
>>>> think might work is allowing modules to "advise" on where to place the
>>>> task. But the decision would still be done by core.
>>>
>>> Possibly exactly what you're doing in proc.c when managing allocation of
>>> process, but not hardcoded in postgresql (patches 02, 05 and 06 are good
>>> candidates), I didn't get that they require information not available to
>>> any process executing code from a module.
>>>
>>
>> Well, it needs to understand how some other stuff (especially PGPROC
>> entries) is distributed between nodes. I'm not sure how much of this
>> internal information we want to expose outside core ...
>>
>>> Parts of your code where you assign/define policy could be in one or
>>> more relevant routines of a "numa profile manager", like in an
>>> initProcessRoutine(), and registered in pmroutine struct:
>>>
>>> pmroutine = GetPmRoutineForInitProcess();
>>> if (pmroutine != NULL &&
>>>      pmroutine->init_process != NULL)
>>>      pmroutine->init_process(MyProc);
>>>
>>> This way it's easier to manage alternative policies, and also to be able
>>> to adjust when hardware and linux kernel changes.
>>>
>>
>> I'm not against making this extensible, in some way. But I still
>> struggle to imagine a reasonable alternative policy, where the external
>> module gets the same information and ends up with a different decision.
>>
>> So what would the alternate policy look like? What use case would the
>> module be supporting?
> 
> 
> That's the whole point: there are very distinct usages of PostgreSQL in
> the field. And maybe not all of them will require the policy defined by
> PostgreSQL core.
> 
> May I ask the reverse: what prevent external modules from taking those
> decisions ? There are already a lot of area where external code can take
> over PostgreSQL processing, like Neon is doing.
> 

The complexity of making everything extensible in an arbitrary way. To
make it extensible in a useful, we need to have a reasonably clear idea
what aspects need to be extensible, and what's the goal.

> There are some very early processing for memory setup that I can see as
> a current blocker, and here I'd refer a more compliant NUMA api as
> proposed by Jakub so it's possible to arrange based on workload,
> hardware configuration or other matters. Reworking to get distinct
> segment and all as you do is great, and combo of both approach probably
> of great interest. There is also this weighted interleave discussed and
> probably much more to come in this area in Linux.
> 
> I think some points raised already about possible distinct policies, I
> am precisely claiming that it is hard to come with one good policy with
> limited setup options, thus requirement to keep that flexible enough
> (hooks, api, 100 GUc ?).
> 

I'm sorry, I don't want to sound too negative, but "I want arbitrary
extensibility" is not a very useful feedback. I've asked you to give
some examples of policies that'd customize some of the NUMA stuff.

> There is an EPYC story here also, given the NUMA setup can vary
> depending on BIOS setup, associated NUMA policy must probably take that
> into account (L3 can be either real cache or 4 extra "local" NUMA nodes
> - with highly distinct access cost from a RAM module).
> Does that change how PostgreSQL will place memory and process? Is it
> important or of interest ?
> 

So how exactly would the policy handle this? Right now we're entirely
oblivious to L3, or on-CPU caches in general. We don't even consider the
size of L3 when sizing hash tables in a hashjoin etc.


regards

-- 
Tomas Vondra

  37. Re: Adding basic NUMA awareness

    Tomas Vondra <tomas@vondra.me> — 2025-07-10T15:31:45Z

    
On 7/9/25 19:23, Andres Freund wrote:
> Hi,
> 
> On 2025-07-09 12:55:51 -0400, Greg Burd wrote:
>> On Jul 9 2025, at 12:35 pm, Andres Freund <andres@anarazel.de> wrote:
>>
>>> FWIW, I've started to wonder if we shouldn't just get rid of the freelist
>>> entirely. While clocksweep is perhaps minutely slower in a single
>>> thread than
>>> the freelist, clock sweep scales *considerably* better [1]. As it's rather
>>> rare to be bottlenecked on clock sweep speed for a single thread
>>> (rather then
>>> IO or memory copy overhead), I think it's worth favoring clock sweep.
>>
>> Hey Andres, thanks for spending time on this.  I've worked before on
>> freelist implementations (last one in LMDB) and I think you're onto
>> something.  I think it's an innovative idea and that the speed
>> difference will either be lost in the noise or potentially entirely
>> mitigated by avoiding duplicate work.
> 
> Agreed. FWIW, just using clock sweep actually makes things like DROP TABLE
> perform better because it doesn't need to maintain the freelist anymore...
> 
> 
>>> Also needing to switch between getting buffers from the freelist and
>>> the sweep
>>> makes the code more expensive.  I think just having the buffer in the sweep,
>>> with a refcount / usagecount of zero would suffice.
>>
>> If you're not already coding this, I'll jump in. :)
> 
> My experimental patch is literally a four character addition ;), namely adding
> "0 &&" to the relevant code in StrategyGetBuffer().
> 
> Obviously a real patch would need to do some more work than that.  Feel free
> to take on that project, I am not planning on tackling that in near term.
> 
> 
> There's other things around this that could use some attention. It's not hard
> to see clock sweep be a bottleneck in concurrent workloads - partially due to
> the shared maintenance of the clock hand. A NUMAed clock sweep would address
> that. However, we also maintain StrategyControl->numBufferAllocs, which is a
> significant contention point and would not necessarily be removed by a
> NUMAificiation of the clock sweep.
> 

Wouldn't it make sense to partition the numBufferAllocs too, though? I
don't remember if my hacky experimental patch NUMA-partitioning did that
or I just thought about doing that, but why wouldn't that be enough?

Places that need the "total" count would have to sum the counters, but
it seemed to me most of the places would be fine with the "local" count
for that partition. If we also make sure to "sync" the clocksweeps so as
to not work on just a single partition, that might be enough ...

regards

-- 
Tomas Vondra

  38. Re: Adding basic NUMA awareness

    Andres Freund <andres@anarazel.de> — 2025-07-11T16:06:13Z

    Hi,

On 2025-07-10 17:31:45 +0200, Tomas Vondra wrote:
> On 7/9/25 19:23, Andres Freund wrote:
> > There's other things around this that could use some attention. It's not hard
> > to see clock sweep be a bottleneck in concurrent workloads - partially due to
> > the shared maintenance of the clock hand. A NUMAed clock sweep would address
> > that. However, we also maintain StrategyControl->numBufferAllocs, which is a
> > significant contention point and would not necessarily be removed by a
> > NUMAificiation of the clock sweep.
> > 
> 
> Wouldn't it make sense to partition the numBufferAllocs too, though? I
> don't remember if my hacky experimental patch NUMA-partitioning did that
> or I just thought about doing that, but why wouldn't that be enough?

It could be solved together with partitioning, yes - that's what I was trying
to reference with the emphasized bit in "would *not necessarily* be removed by
a NUMAificiation of the clock sweep".

Greetings,

Andres Freund

  39. Re: Adding basic NUMA awareness

    Andres Freund <andres@anarazel.de> — 2025-07-11T16:14:06Z

    Hi,

On 2025-07-10 14:17:21 +0000, Bertrand Drouvot wrote:
> On Wed, Jul 09, 2025 at 03:42:26PM -0400, Andres Freund wrote:
> > I wonder if we should *increase* the size of shared_buffers whenever huge
> > pages are in use and there's padding space due to the huge page
> > boundaries. Pretty pointless to waste that memory if we can instead use if for
> > the buffer pool.  Not that big a deal with 2MB huge pages, but with 1GB huge
> > pages...
> 
> I think that makes sense, except maybe for operations that need to scan
> the whole buffer pool (i.e related to BUF_DROP_FULL_SCAN_THRESHOLD)?

I don't think the increases here are big enough for that to matter, unless
perhaps you're using 1GB huge pages. But if you're concerned about dropping
tables very fast (i.e. you're running schema change heavy regression tests),
you're not going to use 1GB huge pages.



> > > 5) v1-0005-NUMA-interleave-PGPROC-entries.patch
> > >
> > > Another area that seems like it might benefit from NUMA is PGPROC, so I
> > > gave it a try. It turned out somewhat challenging. Similarly to buffers
> > > we have two pieces that need to be located in a coordinated way - PGPROC
> > > entries and fast-path arrays. But we can't use the same approach as for
> > > buffers/descriptors, because
> > >
> > > (a) Neither of those pieces aligns with memory page size (PGPROC is
> > > ~900B, fast-path arrays are variable length).
> > 
> > > (b) We could pad PGPROC entries e.g. to 1KB, but that'd still require
> > > rather high max_connections before we use multiple huge pages.
> > 
> > Right now sizeof(PGPROC) happens to be multiple of 64 (i.e. the most common
> > cache line size)
> 
> Oh right, it's currently 832 bytes and the patch extends that to 840 bytes.

I don't think the patch itself is the problem - it really is just happenstance
that it's a multiple of the line size right now. And it's not on common Armv8
platforms...


> With a bit of reordering:
> 
> That could be back to 832 (the order does not make sense logically speaking
> though).

I don't think shrinking the size in a one-off way just to keep the
"accidental" size-is-multiple-of-64 property is promising. It'll just get
broken again.  I think we should:

a) pad the size of PGPROC to a cache line (or even to a subsequent power of 2,
   to make array access cheaper, right now that involves actual
   multiplications rather than shifts or indexed `lea` instructions).

   That's probably just a pg_attribute_aligned

b) Reorder PGPROC to separate frequently modified from almost-read-only data,
   to increase cache hit ratio.

Greetings,

Andres Freund

  40. Re: Adding basic NUMA awareness

    Greg Burd <greg@burd.me> — 2025-07-11T17:34:26Z

    
> On Jul 10, 2025, at 8:13 AM, Burd, Greg <greg@burd.me> wrote:
> 
> 
>> On Jul 9, 2025, at 1:23 PM, Andres Freund <andres@anarazel.de> wrote:
>> 
>> Hi,
>> 
>> On 2025-07-09 12:55:51 -0400, Greg Burd wrote:
>>> On Jul 9 2025, at 12:35 pm, Andres Freund <andres@anarazel.de> wrote:
>>> 
>>>> FWIW, I've started to wonder if we shouldn't just get rid of the freelist
>>>> entirely. While clocksweep is perhaps minutely slower in a single
>>>> thread than
>>>> the freelist, clock sweep scales *considerably* better [1]. As it's rather
>>>> rare to be bottlenecked on clock sweep speed for a single thread
>>>> (rather then
>>>> IO or memory copy overhead), I think it's worth favoring clock sweep.
>>> 
>>> Hey Andres, thanks for spending time on this.  I've worked before on
>>> freelist implementations (last one in LMDB) and I think you're onto
>>> something.  I think it's an innovative idea and that the speed
>>> difference will either be lost in the noise or potentially entirely
>>> mitigated by avoiding duplicate work.
>> 
>> Agreed. FWIW, just using clock sweep actually makes things like DROP TABLE
>> perform better because it doesn't need to maintain the freelist anymore...
>> 
>> 
>>>> Also needing to switch between getting buffers from the freelist and
>>>> the sweep
>>>> makes the code more expensive.  I think just having the buffer in the sweep,
>>>> with a refcount / usagecount of zero would suffice.
>>> 
>>> If you're not already coding this, I'll jump in. :)
>> 
>> My experimental patch is literally a four character addition ;), namely adding
>> "0 &&" to the relevant code in StrategyGetBuffer().
>> 
>> Obviously a real patch would need to do some more work than that.  Feel free
>> to take on that project, I am not planning on tackling that in near term.
>> 
> 
> I started on this last night, making good progress.  Thanks for the inspiration.  I'll create a new thread to track the work and cross-reference when I have something reasonable to show (hopefully later today).
> 
>> There's other things around this that could use some attention. It's not hard
>> to see clock sweep be a bottleneck in concurrent workloads - partially due to
>> the shared maintenance of the clock hand. A NUMAed clock sweep would address
>> that.
> 
> Working on it.

For archival sake, and to tie up loose ends I'll link from here to a new thread I just started that proposes the removal of the freelist and the buffer_strategy_lock [1].

That patch set doesn't address any NUMA-related tasks directly, but it should remove some pain when working in that direction by removing code that requires partitioning and locking and...

best.

-greg

[1] https://postgr.es/m/E2D6FCDC-BE98-4F95-B45E-699C3E17BA10@burd.me

  41. Re: Adding basic NUMA awareness

    Tomas Vondra <tomas@vondra.me> — 2025-07-17T21:11:16Z

    Hi,

Here's a v2 of the patch series, with a couple changes:

* I simplified the various freelist partitioning by keeping only the
"node" partitioning (so the cpu/pid strategies are gone). Those were
meant for experimenting, but it made the code more complicated so I
ditched it.


* I changed the freelist partitioning scheme a little bit, based on the
discussion in this thread. Instead of having a single "partition" per
NUMA node, there's not a minimum number of partitions (set to 4). So
even if your system is not NUMA, you'll have 4 of them. If you have 2
nodes, you'll still have 4, and each node will get 2. With 3 nodes we
get 6 partitions (we need 2 per node, and we want to keep the number
equal to keep things simple). Once the number of nodes exceeds 4, the
heuristics switches to one partition per node.

I'm aware there's a discussion about maybe simply removing freelists
entirely. If that happens, this becomes mostly irrelevant, of course.

The code should also make sure the freelists "agree" with how the
earlier patch mapped the buffers to NUMA nodes, i.e. the freelist should
only contain buffers from the "correct" NUMA node, etc. I haven't paid
much attention to this - I believe it should work for "nice" values of
shared buffers (when it evenly divides between nodes). But I'm sure it's
possible to confuse that (won't cause crashes, but inefficiency).


* There's now a patch partitioning clocksweep, using the same scheme as
the freelists. I came to the conclusion it doesn't make much sense to
partition these things differently - I can't think of a reason why that
would be advantageous, and it makes it easier to reason about.

The clocksweep partitioning is somewhat harder, because it affects
BgBufferSync() and related code. With the partitioning we now have
multiple "clock hands" for different ranges of buffers, and the clock
sweep needs to consider that. I modified BgBufferSync to simply loop
through the ClockSweep partitions, and do a small cleanup for each.

It does work, as in "it doesn't crash". But this part definitely needs
review to make sure I got the changes to the "pacing" right.


* This new freelist/clocksweep partitioning scheme is however harder to
disable. I now realize the GUC may quite do the trick, and there even is
not a GUC for the clocksweep. I need to think about this, but I'm not
even how feasible it'd be to have two separate GUCs (because of how
these two pieces are intertwined). For now if you want to test without
the partitioning, you need to skip the patch.


I did some quick perf testing on my old xeon machine (2 NUMA nodes), and
the results are encouraging. For a read-only pgbench (2x shared buffers,
within RAM), I saw an increase from 1.1M tps to 1.3M. Not crazy, but not
bad considering the patch is more about consistency than raw throughput.

For a read-write pgbench I however saw some strange drops/increases of
throughput. I suspect this might be due to some thinko in the clocksweep
partitioning, but I'll need to take a closer look.


regards

-- 
Tomas Vondra

  42. Re: Adding basic NUMA awareness

    Tomas Vondra <tomas@vondra.me> — 2025-07-17T21:14:52Z

    On 7/4/25 20:12, Tomas Vondra wrote:
> On 7/4/25 13:05, Jakub Wartak wrote:
>> ...
>>
>> 8. v1-0005 2x + /* if (numa_procs_interleave) */
>>
>>    Ha! it's a TRAP! I've uncommented it because I wanted to try it out
>> without it (just by setting GUC off) , but "MyProc->sema" is NULL :
>>
>>     2025-07-04 12:31:08.103 CEST [28754] LOG:  starting PostgreSQL
>> 19devel on x86_64-linux, compiled by gcc-12.2.0, 64-bit
>>     [..]
>>     2025-07-04 12:31:08.109 CEST [28754] LOG:  io worker (PID 28755)
>> was terminated by signal 11: Segmentation fault
>>     2025-07-04 12:31:08.109 CEST [28754] LOG:  terminating any other
>> active server processes
>>     2025-07-04 12:31:08.114 CEST [28754] LOG:  shutting down because
>> "restart_after_crash" is off
>>     2025-07-04 12:31:08.116 CEST [28754] LOG:  database system is shut down
>>
>>     [New LWP 28755]
>>     [Thread debugging using libthread_db enabled]
>>     Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>     Core was generated by `postgres: io worker                     '.
>>     Program terminated with signal SIGSEGV, Segmentation fault.
>>     #0  __new_sem_wait_fast (definitive_result=1, sem=sem@entry=0x0)
>> at ./nptl/sem_waitcommon.c:136
>>     136     ./nptl/sem_waitcommon.c: No such file or directory.
>>     (gdb) where
>>     #0  __new_sem_wait_fast (definitive_result=1, sem=sem@entry=0x0)
>> at ./nptl/sem_waitcommon.c:136
>>     #1  __new_sem_trywait (sem=sem@entry=0x0) at ./nptl/sem_wait.c:81
>>     #2  0x00005561918e0cac in PGSemaphoreReset (sema=0x0) at
>> ../src/backend/port/posix_sema.c:302
>>     #3  0x0000556191970553 in InitAuxiliaryProcess () at
>> ../src/backend/storage/lmgr/proc.c:992
>>     #4  0x00005561918e51a2 in AuxiliaryProcessMainCommon () at
>> ../src/backend/postmaster/auxprocess.c:65
>>     #5  0x0000556191940676 in IoWorkerMain (startup_data=<optimized
>> out>, startup_data_len=<optimized out>) at
>> ../src/backend/storage/aio/method_worker.c:393
>>     #6  0x00005561918e8163 in postmaster_child_launch
>> (child_type=child_type@entry=B_IO_WORKER, child_slot=20086,
>> startup_data=startup_data@entry=0x0,
>>         startup_data_len=startup_data_len@entry=0,
>> client_sock=client_sock@entry=0x0) at
>> ../src/backend/postmaster/launch_backend.c:290
>>     #7  0x00005561918ea09a in StartChildProcess
>> (type=type@entry=B_IO_WORKER) at
>> ../src/backend/postmaster/postmaster.c:3973
>>     #8  0x00005561918ea308 in maybe_adjust_io_workers () at
>> ../src/backend/postmaster/postmaster.c:4404
>>     [..]
>>     (gdb) print *MyProc->sem
>>     Cannot access memory at address 0x0
>>
> 
> Yeah, good catch. I'll look into that next week.
> 

I've been unable to reproduce this issue, but I'm not sure what settings
you actually used for this instance. Can you give me more details how to
reproduce this?

regards

-- 
Tomas Vondra

  43. Re: Adding basic NUMA awareness

    Andres Freund <andres@anarazel.de> — 2025-07-18T16:46:44Z

    Hi,

On 2025-07-17 23:11:16 +0200, Tomas Vondra wrote:
> Here's a v2 of the patch series, with a couple changes:

Not a deep look at the code, just a quick reply.


> * I changed the freelist partitioning scheme a little bit, based on the
> discussion in this thread. Instead of having a single "partition" per
> NUMA node, there's not a minimum number of partitions (set to 4). So

I assume s/not/now/?


> * There's now a patch partitioning clocksweep, using the same scheme as
> the freelists.

Nice!


> I came to the conclusion it doesn't make much sense to partition these
> things differently - I can't think of a reason why that would be
> advantageous, and it makes it easier to reason about.

Agreed.


> The clocksweep partitioning is somewhat harder, because it affects
> BgBufferSync() and related code. With the partitioning we now have
> multiple "clock hands" for different ranges of buffers, and the clock
> sweep needs to consider that. I modified BgBufferSync to simply loop
> through the ClockSweep partitions, and do a small cleanup for each.

That probably makes sense for now. It might need a bit of a larger adjustment
at some point, but ...



> * This new freelist/clocksweep partitioning scheme is however harder to
> disable. I now realize the GUC may quite do the trick, and there even is
> not a GUC for the clocksweep. I need to think about this, but I'm not
> even how feasible it'd be to have two separate GUCs (because of how
> these two pieces are intertwined). For now if you want to test without
> the partitioning, you need to skip the patch.

I think it's totally fair to enable/disable them at the same time. They're so
closely related, that I don't think it really makes sense to measure them
separately.


> I did some quick perf testing on my old xeon machine (2 NUMA nodes), and
> the results are encouraging. For a read-only pgbench (2x shared buffers,
> within RAM), I saw an increase from 1.1M tps to 1.3M. Not crazy, but not
> bad considering the patch is more about consistency than raw throughput.

Personally I think an 1.18x improvement on a relatively small NUMA machine is
really rather awesome.


> For a read-write pgbench I however saw some strange drops/increases of
> throughput. I suspect this might be due to some thinko in the clocksweep
> partitioning, but I'll need to take a closer look.

Was that with pinning etc enabled or not?



> From c4d51ab87b92f9900e37d42cf74980e87b648a56 Mon Sep 17 00:00:00 2001
> From: Tomas Vondra <tomas@vondra.me>
> Date: Sun, 8 Jun 2025 18:53:12 +0200
> Subject: [PATCH v2 5/7] NUMA: clockweep partitioning
>


> @@ -475,13 +525,17 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
>  	/*
>  	 * Nothing on the freelist, so run the "clock sweep" algorithm
>  	 *
> -	 * XXX Should we also make this NUMA-aware, to only access buffers from
> -	 * the same NUMA node? That'd probably mean we need to make the clock
> -	 * sweep NUMA-aware, perhaps by having multiple clock sweeps, each for a
> -	 * subset of buffers. But that also means each process could "sweep" only
> -	 * a fraction of buffers, even if the other buffers are better candidates
> -	 * for eviction. Would that also mean we'd have multiple bgwriters, one
> -	 * for each node, or would one bgwriter handle all of that?
> +	 * XXX Note that ClockSweepTick() is NUMA-aware, i.e. it only looks at
> +	 * buffers from a single partition, aligned with the NUMA node. That
> +	 * means it only accesses buffers from the same NUMA node.
> +	 *
> +	 * XXX That also means each process "sweeps" only a fraction of buffers,
> +	 * even if the other buffers are better candidates for eviction. Maybe
> +	 * there should be some logic to "steal" buffers from other freelists
> +	 * or other nodes?

I think we *definitely* need "stealing" from other clock sweeps, whenever
there's a meaningful imbalance between the different sweeps.

I don't think we need to be overly precise about it, a small imbalance won't
have that much of an effect. But clearly it doesn't make sense to say that one
backend can only fill buffers in the current partition, that'd lead to massive
performance issues in a lot of workloads.

The hardest thing probably is to make the logic for when to check foreign
clock sweeps cheap enough.

One way would be to do it whenever a sweep wraps around, that'd probably
amortize the cost sufficiently, and I don't think it'd be too imprecise, as
we'd have processed that set of buffers in a row without partitioning as
well. But it'd probably be too coarse when determining for how long to use a
foreign sweep instance. But we probably could address that by rechecking the
balanace more frequently when using a foreign partition.

Another way would be to have bgwriter manage this. Whenever it detects that
one ring is too far ahead, it could set a "avoid this partition" bit, which
would trigger backends that natively use that partition to switch to foreign
partitions that don't currently have that bit set.  I suspect there's a
problem with that approach though, I worry that the amount of time that
bgwriter spends in BgBufferSync() may sometimes be too long, leading to too
much imbalance.

Greetings,

Andres Freund

  44. Re: Adding basic NUMA awareness

    Tomas Vondra <tomas@vondra.me> — 2025-07-18T20:48:00Z

    On 7/18/25 18:46, Andres Freund wrote:
> Hi,
> 
> On 2025-07-17 23:11:16 +0200, Tomas Vondra wrote:
>> Here's a v2 of the patch series, with a couple changes:
> 
> Not a deep look at the code, just a quick reply.
> 
> 
>> * I changed the freelist partitioning scheme a little bit, based on the
>> discussion in this thread. Instead of having a single "partition" per
>> NUMA node, there's not a minimum number of partitions (set to 4). So
> 
> I assume s/not/now/?
> 

Yes.

> 
>> * There's now a patch partitioning clocksweep, using the same scheme as
>> the freelists.
> 
> Nice!
> 
> 
>> I came to the conclusion it doesn't make much sense to partition these
>> things differently - I can't think of a reason why that would be
>> advantageous, and it makes it easier to reason about.
> 
> Agreed.
> 
> 
>> The clocksweep partitioning is somewhat harder, because it affects
>> BgBufferSync() and related code. With the partitioning we now have
>> multiple "clock hands" for different ranges of buffers, and the clock
>> sweep needs to consider that. I modified BgBufferSync to simply loop
>> through the ClockSweep partitions, and do a small cleanup for each.
> 
> That probably makes sense for now. It might need a bit of a larger
adjustment at some point, but ...
> 

I couldn't think of something fundamentally better and not too complex.
I suspect we might want to use multiple bgwriters in the future, and
this scheme seems to be reasonably well suited for that too.

I'm also thinking about having some sort of "unified" partitioning
scheme for all the places partitioning shared buffers. Right now each of
the places does it on it's own, i.e. buff_init, freelist and clocksweep
all have their code splitting NBuffers into partitions. And it should
align. Because what would be the benefit if it didn't? But I guess
having three variants of the same code seems a bit pointless.

I think buff_init should build a common definition of buffer partitions,
and the remaining parts should use that as the source of truth ...

> 
>> * This new freelist/clocksweep partitioning scheme is however harder to
>> disable. I now realize the GUC may quite do the trick, and there even is
>> not a GUC for the clocksweep. I need to think about this, but I'm not
>> even how feasible it'd be to have two separate GUCs (because of how
>> these two pieces are intertwined). For now if you want to test without
>> the partitioning, you need to skip the patch.
> 
> I think it's totally fair to enable/disable them at the same time. They're so
> closely related, that I don't think it really makes sense to measure them
> separately.
> 

Yeah, that's a fair point.

> 
>> I did some quick perf testing on my old xeon machine (2 NUMA nodes), and
>> the results are encouraging. For a read-only pgbench (2x shared buffers,
>> within RAM), I saw an increase from 1.1M tps to 1.3M. Not crazy, but not
>> bad considering the patch is more about consistency than raw throughput.
> 
> Personally I think an 1.18x improvement on a relatively small NUMA machine is
> really rather awesome.
> 

True, but I want to stress out it's just one quick (& simple test). Much
more testing is needed before I can make reliable claims.

> 
>> For a read-write pgbench I however saw some strange drops/increases of
>> throughput. I suspect this might be due to some thinko in the clocksweep
>> partitioning, but I'll need to take a closer look.
> 
> Was that with pinning etc enabled or not?
> 

IIRC it was with everything enabled, except for numa_procs_pin (which
pins backend to NUMA node). I found that to actually harm performance in
some of the tests (even just read-only ones), resulting in uneven usage
of cores and lower throughput.

> 
> 
>> From c4d51ab87b92f9900e37d42cf74980e87b648a56 Mon Sep 17 00:00:00 2001
>> From: Tomas Vondra <tomas@vondra.me>
>> Date: Sun, 8 Jun 2025 18:53:12 +0200
>> Subject: [PATCH v2 5/7] NUMA: clockweep partitioning
>>
> 
> 
>> @@ -475,13 +525,17 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r
>>  	/*
>>  	 * Nothing on the freelist, so run the "clock sweep" algorithm
>>  	 *
>> -	 * XXX Should we also make this NUMA-aware, to only access buffers from
>> -	 * the same NUMA node? That'd probably mean we need to make the clock
>> -	 * sweep NUMA-aware, perhaps by having multiple clock sweeps, each for a
>> -	 * subset of buffers. But that also means each process could "sweep" only
>> -	 * a fraction of buffers, even if the other buffers are better candidates
>> -	 * for eviction. Would that also mean we'd have multiple bgwriters, one
>> -	 * for each node, or would one bgwriter handle all of that?
>> +	 * XXX Note that ClockSweepTick() is NUMA-aware, i.e. it only looks at
>> +	 * buffers from a single partition, aligned with the NUMA node. That
>> +	 * means it only accesses buffers from the same NUMA node.
>> +	 *
>> +	 * XXX That also means each process "sweeps" only a fraction of buffers,
>> +	 * even if the other buffers are better candidates for eviction. Maybe
>> +	 * there should be some logic to "steal" buffers from other freelists
>> +	 * or other nodes?
> 
> I think we *definitely* need "stealing" from other clock sweeps, whenever
> there's a meaningful imbalance between the different sweeps.
> 
> I don't think we need to be overly precise about it, a small imbalance won't
> have that much of an effect. But clearly it doesn't make sense to say that one
> backend can only fill buffers in the current partition, that'd lead to massive
> performance issues in a lot of workloads.
> 

Agreed.

> The hardest thing probably is to make the logic for when to check foreign
> clock sweeps cheap enough.
> 
> One way would be to do it whenever a sweep wraps around, that'd probably
> amortize the cost sufficiently, and I don't think it'd be too imprecise, as
> we'd have processed that set of buffers in a row without partitioning as
> well. But it'd probably be too coarse when determining for how long to use a
> foreign sweep instance. But we probably could address that by rechecking the
> balanace more frequently when using a foreign partition.
> 

What you mean by "it"? What would happen after a sweep wraps around?

> Another way would be to have bgwriter manage this. Whenever it detects that
> one ring is too far ahead, it could set a "avoid this partition" bit, which
> would trigger backends that natively use that partition to switch to foreign
> partitions that don't currently have that bit set.  I suspect there's a
> problem with that approach though, I worry that the amount of time that
> bgwriter spends in BgBufferSync() may sometimes be too long, leading to too
> much imbalance.
> 

I'm afraid having hard "avoid" flags would lead to sudden and unexpected
changes in performance as we enable/disable partitions. I think a good
solution should "smooth it out" somehow, e.g. by not having a true/false
flag, but having some sort of "preference" factor with values between
(0.0, 1.0) which says how much we should use that partition.

I was imagining something like this:

Say we know the number of buffers allocated for each partition (in the
last round), and we (or rather the BgBufferSync) calculate:

    coefficient = 1.0 - (nallocated_partition / nallocated)

and then use that to "correct" which partition to allocate buffers from.
Or maybe just watch how far from the "fair share" we were in the last
interval, and gradually increase/decrease the "partition preference"
which would say how often we need to "steal" from other partitions.

E.g. we find nallocated_partition is 2x the fair share, i.e.

   nallocated_partition / (nallocated / nparts) = 2.0

Then we say 25% of the time look at some other partition, to "cut" the
imbalance in half. And then repeat that in the next cycle, etc.

So a process would look at it's "home partition" by default, but it's
"roll a dice" first and if above the calculated probability it'd pick
some other partition instead (this would need to be done so that it gets
balanced overall).

If the bgwriter interval is too long, maybe the recalculation could be
triggered regularly after any of the clocksweeps wraps around, or after
some number of allocations, or something like that.


regards

-- 
Tomas Vondra

  45. Re: Adding basic NUMA awareness

    Andres Freund <andres@anarazel.de> — 2025-07-18T21:03:59Z

    Hi,

On 2025-07-18 22:48:00 +0200, Tomas Vondra wrote:
> On 7/18/25 18:46, Andres Freund wrote:
> >> For a read-write pgbench I however saw some strange drops/increases of
> >> throughput. I suspect this might be due to some thinko in the clocksweep
> >> partitioning, but I'll need to take a closer look.
> > 
> > Was that with pinning etc enabled or not?
> > 
> 
> IIRC it was with everything enabled, except for numa_procs_pin (which
> pins backend to NUMA node). I found that to actually harm performance in
> some of the tests (even just read-only ones), resulting in uneven usage
> of cores and lower throughput.

FWIW, I really doubt that something like numa_procs_pin is viable outside of
very narrow niches until we have a *lot* more infrastructure in place. Like PG
would need to be threaded, we'd need a separation between thread and
connection and an executor that'd allow us to switch from working on one query
to working on another query.


> > The hardest thing probably is to make the logic for when to check foreign
> > clock sweeps cheap enough.
> > 
> > One way would be to do it whenever a sweep wraps around, that'd probably
> > amortize the cost sufficiently, and I don't think it'd be too imprecise, as
> > we'd have processed that set of buffers in a row without partitioning as
> > well. But it'd probably be too coarse when determining for how long to use a
> > foreign sweep instance. But we probably could address that by rechecking the
> > balanace more frequently when using a foreign partition.
> > 
> 
> What you mean by "it"?

it := Considering switching back from using a "foreign" clock sweep instance
whenever the sweep wraps around.


> What would happen after a sweep wraps around?

The scenario I'm worried about is this:

1) a bunch of backends read buffers on numa node A, using the local clock
   sweep instance

2) due to all of that activity, the clock sweep advances much faster than the
   clock sweep for numa node B

3) the clock sweep on A wraps around, we discover the imbalance, and all the
   backend switch to scanning on numa node B, moving that clock sweep ahead
   much more aggressively

4) clock sweep on B wraps around, there's imbalance the other way round now,
   so they all switch back to A



> > Another way would be to have bgwriter manage this. Whenever it detects that
> > one ring is too far ahead, it could set a "avoid this partition" bit, which
> > would trigger backends that natively use that partition to switch to foreign
> > partitions that don't currently have that bit set.  I suspect there's a
> > problem with that approach though, I worry that the amount of time that
> > bgwriter spends in BgBufferSync() may sometimes be too long, leading to too
> > much imbalance.
> > 
> 
> I'm afraid having hard "avoid" flags would lead to sudden and unexpected
> changes in performance as we enable/disable partitions. I think a good
> solution should "smooth it out" somehow, e.g. by not having a true/false
> flag, but having some sort of "preference" factor with values between
> (0.0, 1.0) which says how much we should use that partition.

Yea, I think that's a fair worry.


> I was imagining something like this:
> 
> Say we know the number of buffers allocated for each partition (in the
> last round), and we (or rather the BgBufferSync) calculate:
> 
>     coefficient = 1.0 - (nallocated_partition / nallocated)
> 
> and then use that to "correct" which partition to allocate buffers from.
> Or maybe just watch how far from the "fair share" we were in the last
> interval, and gradually increase/decrease the "partition preference"
> which would say how often we need to "steal" from other partitions.
> 
> E.g. we find nallocated_partition is 2x the fair share, i.e.
> 
>    nallocated_partition / (nallocated / nparts) = 2.0
> 
> Then we say 25% of the time look at some other partition, to "cut" the
> imbalance in half. And then repeat that in the next cycle, etc.
> 
> So a process would look at it's "home partition" by default, but it's
> "roll a dice" first and if above the calculated probability it'd pick
> some other partition instead (this would need to be done so that it gets
> balanced overall).

That does sound reasonable.


> If the bgwriter interval is too long, maybe the recalculation could be
> triggered regularly after any of the clocksweeps wraps around, or after
> some number of allocations, or something like that.

I'm pretty sure the bgwriter might not be often enough and not predictably
frequently running for that.

Greetings,

Andres Freund

  46. Re: Adding basic NUMA awareness

    Jakub Wartak <jakub.wartak@enterprisedb.com> — 2025-07-25T10:27:11Z

    On Thu, Jul 17, 2025 at 11:15 PM Tomas Vondra <tomas@vondra.me> wrote:
>
> On 7/4/25 20:12, Tomas Vondra wrote:
> > On 7/4/25 13:05, Jakub Wartak wrote:
> >> ...
> >>
> >> 8. v1-0005 2x + /* if (numa_procs_interleave) */
> >>
> >>    Ha! it's a TRAP! I've uncommented it because I wanted to try it out
> >> without it (just by setting GUC off) , but "MyProc->sema" is NULL :
> >>
> >>     2025-07-04 12:31:08.103 CEST [28754] LOG:  starting PostgreSQL
> >> 19devel on x86_64-linux, compiled by gcc-12.2.0, 64-bit
> >>     [..]
> >>     2025-07-04 12:31:08.109 CEST [28754] LOG:  io worker (PID 28755)
> >> was terminated by signal 11: Segmentation fault
> >>     2025-07-04 12:31:08.109 CEST [28754] LOG:  terminating any other
> >> active server processes
> >>     2025-07-04 12:31:08.114 CEST [28754] LOG:  shutting down because
> >> "restart_after_crash" is off
> >>     2025-07-04 12:31:08.116 CEST [28754] LOG:  database system is shut down
> >>
> >>     [New LWP 28755]
> >>     [Thread debugging using libthread_db enabled]
> >>     Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> >>     Core was generated by `postgres: io worker                     '.
> >>     Program terminated with signal SIGSEGV, Segmentation fault.
> >>     #0  __new_sem_wait_fast (definitive_result=1, sem=sem@entry=0x0)
> >> at ./nptl/sem_waitcommon.c:136
> >>     136     ./nptl/sem_waitcommon.c: No such file or directory.
> >>     (gdb) where
> >>     #0  __new_sem_wait_fast (definitive_result=1, sem=sem@entry=0x0)
> >> at ./nptl/sem_waitcommon.c:136
> >>     #1  __new_sem_trywait (sem=sem@entry=0x0) at ./nptl/sem_wait.c:81
> >>     #2  0x00005561918e0cac in PGSemaphoreReset (sema=0x0) at
> >> ../src/backend/port/posix_sema.c:302
> >>     #3  0x0000556191970553 in InitAuxiliaryProcess () at
> >> ../src/backend/storage/lmgr/proc.c:992
> >>     #4  0x00005561918e51a2 in AuxiliaryProcessMainCommon () at
> >> ../src/backend/postmaster/auxprocess.c:65
> >>     #5  0x0000556191940676 in IoWorkerMain (startup_data=<optimized
> >> out>, startup_data_len=<optimized out>) at
> >> ../src/backend/storage/aio/method_worker.c:393
> >>     #6  0x00005561918e8163 in postmaster_child_launch
> >> (child_type=child_type@entry=B_IO_WORKER, child_slot=20086,
> >> startup_data=startup_data@entry=0x0,
> >>         startup_data_len=startup_data_len@entry=0,
> >> client_sock=client_sock@entry=0x0) at
> >> ../src/backend/postmaster/launch_backend.c:290
> >>     #7  0x00005561918ea09a in StartChildProcess
> >> (type=type@entry=B_IO_WORKER) at
> >> ../src/backend/postmaster/postmaster.c:3973
> >>     #8  0x00005561918ea308 in maybe_adjust_io_workers () at
> >> ../src/backend/postmaster/postmaster.c:4404
> >>     [..]
> >>     (gdb) print *MyProc->sem
> >>     Cannot access memory at address 0x0
> >>
> >
> > Yeah, good catch. I'll look into that next week.
> >
>
> I've been unable to reproduce this issue, but I'm not sure what settings
> you actually used for this instance. Can you give me more details how to
> reproduce this?

Better late than never, well feel free to partially ignore me, i've
missed that it is known issue as per FIXME there, but I would just rip
out that commented out `if(numa_proc_interleave)` from
FastPathLockShmemSize() and PGProcShmemSize() unless you want to save
those memory pages of course (in case of no-NUMA). If you do want to
save those pages I think we have problem:

For complete picture, steps:

1. patch -p1 < v2-0001-NUMA-interleaving-buffers.patch
2. patch -p1 < v2-0006-NUMA-interleave-PGPROC-entries.patch

BTW the pgbench accidentinal ident is still there (part of v2-0001 patch))
14 out of 14 hunks FAILED -- saving rejects to file
src/bin/pgbench/pgbench.c.rej

3. As I'm just applying 0001 and 0006, I've got two simple rejects,
but fixed it (due to not applying missing numa_ freelist patches).
That's intentional on my part, because I wanted to play just with
those two.

4. Then I uncomment those two "if (numa_procs_interleave)" related for
optional memory shm initialization - add_size() and so on (that have
XXX comment above that it is causing bootstrap issues)

5. initdb with numa_procs_interleave=on, huge_pages = on (!), start, it is ok

6. restart with numa_procs_interleave=off, which gets me to every bg
worker crashing e.g.:

(gdb) where
#0  __new_sem_wait_fast (definitive_result=1, sem=sem@entry=0x0) at
./nptl/sem_waitcommon.c:136
#1  __new_sem_trywait (sem=sem@entry=0x0) at ./nptl/sem_wait.c:81
#2  0x0000563e2d6e4d5c in PGSemaphoreReset (sema=0x0) at
../src/backend/port/posix_sema.c:302
#3  0x0000563e2d774d93 in InitAuxiliaryProcess () at
../src/backend/storage/lmgr/proc.c:995
#4  0x0000563e2d6e9252 in AuxiliaryProcessMainCommon () at
../src/backend/postmaster/auxprocess.c:65
#5  0x0000563e2d6eb683 in CheckpointerMain (startup_data=<optimized
out>, startup_data_len=<optimized out>) at
../src/backend/postmaster/checkpointer.c:190
#6  0x0000563e2d6ec363 in postmaster_child_launch
(child_type=child_type@entry=B_CHECKPOINTER, child_slot=249,
startup_data=startup_data@entry=0x0,
    startup_data_len=startup_data_len@entry=0,
client_sock=client_sock@entry=0x0) at
../src/backend/postmaster/launch_backend.c:290
#7  0x0000563e2d6ee29a in StartChildProcess
(type=type@entry=B_CHECKPOINTER) at
../src/backend/postmaster/postmaster.c:3973
#8  0x0000563e2d6f17a6 in PostmasterMain (argc=argc@entry=3,
argv=argv@entry=0x563e377cc0e0) at
../src/backend/postmaster/postmaster.c:1386
#9  0x0000563e2d4948fc in main (argc=3, argv=0x563e377cc0e0) at
../src/backend/main/main.c:231

notice sema=0x0, because:
#3  0x000056050928cd93 in InitAuxiliaryProcess () at
../src/backend/storage/lmgr/proc.c:995
995             PGSemaphoreReset(MyProc->sem);
(gdb) print MyProc
$1 = (PGPROC *) 0x7f09a0c013b0
(gdb) print MyProc->sem
$2 = (PGSemaphore) 0x0

or with printfs:

2025-07-25 11:17:23.683 CEST [21772] LOG:  in InitProcGlobal
PGPROC=0x7f9de827b880 requestSize=148770
// after proc && ptr manipulation:
2025-07-25 11:17:23.683 CEST [21772] LOG:  in InitProcGlobal
PGPROC=0x7f9de827bdf0 requestSize=148770 procs=0x7f9de827b880
ptr=0x7f9de827bdf0
[..initialization of aux PGPROCs i=0.., still fromInitProcGlobal(),
each gets proper sem allocated as one would expect:]
[..for i loop:]
2025-07-25 11:17:23.689 CEST [21772] LOG:  i=136 ,
proc=0x7f9de8600000, proc->sem=0x7f9da4e04438
2025-07-25 11:17:23.689 CEST [21772] LOG:  i=137 ,
proc=0x7f9de8600348, proc->sem=0x7f9da4e044b8
2025-07-25 11:17:23.689 CEST [21772] LOG:  i=138 ,
proc=0x7f9de8600690, proc->sem=0x7f9da4e04538
[..but then in the children codepaths, out of the blue in
InitAuxilaryProcess the whole MyProc looks like it would memsetted to
zeros:]
2025-07-25 11:17:23.693 CEST [21784] LOG:  auxiliary process using
MyProc=0x7f9de8600000 auxproc=0x7f9de8600000 proctype=0
MyProcPid=21784 MyProc->sem=(nil)

above got pgproc slot i=136 with addr 0x7f9de8600000 and later that
auxiliary is launched but somehow something NULLified ->sem there
(according to gdb , everything is zero there)

7. Original patch v2-0006 (with commented out 2x if
numa_procs_interleave), behaves OK, so in my case here with 1x NUMA
node that gives add_size(.., 1+1 * 2MB)=4MB

2025-07-25 11:38:54.131 CEST [23939] LOG:  in InitProcGlobal
PGPROC=0x7f25cbe7b880 requestSize=4343074
2025-07-25 11:38:54.132 CEST [23939] LOG:  in InitProcGlobal
PGPROC=0x7f25cbe7bdf0 requestSize=4343074 procs=0x7f25cbe7b880
ptr=0x7f25cbe7bdf0

so something is zeroing out all those MyProc structures apparently on
startup (probably due to some wrong alignment maybe somewhere ?) I was
thinking about trapping via mprotect() this single i=136
0x7f9de8600000 PGPROC to see what is resetting it, but oh well,
mprotect() works only on whole pages...

-J.

  47. Re: Adding basic NUMA awareness

    Tomas Vondra <tomas@vondra.me> — 2025-07-25T10:51:41Z

    
On 7/25/25 12:27, Jakub Wartak wrote:
> On Thu, Jul 17, 2025 at 11:15 PM Tomas Vondra <tomas@vondra.me> wrote:
>>
>> On 7/4/25 20:12, Tomas Vondra wrote:
>>> On 7/4/25 13:05, Jakub Wartak wrote:
>>>> ...
>>>>
>>>> 8. v1-0005 2x + /* if (numa_procs_interleave) */
>>>>
>>>>    Ha! it's a TRAP! I've uncommented it because I wanted to try it out
>>>> without it (just by setting GUC off) , but "MyProc->sema" is NULL :
>>>>
>>>>     2025-07-04 12:31:08.103 CEST [28754] LOG:  starting PostgreSQL
>>>> 19devel on x86_64-linux, compiled by gcc-12.2.0, 64-bit
>>>>     [..]
>>>>     2025-07-04 12:31:08.109 CEST [28754] LOG:  io worker (PID 28755)
>>>> was terminated by signal 11: Segmentation fault
>>>>     2025-07-04 12:31:08.109 CEST [28754] LOG:  terminating any other
>>>> active server processes
>>>>     2025-07-04 12:31:08.114 CEST [28754] LOG:  shutting down because
>>>> "restart_after_crash" is off
>>>>     2025-07-04 12:31:08.116 CEST [28754] LOG:  database system is shut down
>>>>
>>>>     [New LWP 28755]
>>>>     [Thread debugging using libthread_db enabled]
>>>>     Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>>>     Core was generated by `postgres: io worker                     '.
>>>>     Program terminated with signal SIGSEGV, Segmentation fault.
>>>>     #0  __new_sem_wait_fast (definitive_result=1, sem=sem@entry=0x0)
>>>> at ./nptl/sem_waitcommon.c:136
>>>>     136     ./nptl/sem_waitcommon.c: No such file or directory.
>>>>     (gdb) where
>>>>     #0  __new_sem_wait_fast (definitive_result=1, sem=sem@entry=0x0)
>>>> at ./nptl/sem_waitcommon.c:136
>>>>     #1  __new_sem_trywait (sem=sem@entry=0x0) at ./nptl/sem_wait.c:81
>>>>     #2  0x00005561918e0cac in PGSemaphoreReset (sema=0x0) at
>>>> ../src/backend/port/posix_sema.c:302
>>>>     #3  0x0000556191970553 in InitAuxiliaryProcess () at
>>>> ../src/backend/storage/lmgr/proc.c:992
>>>>     #4  0x00005561918e51a2 in AuxiliaryProcessMainCommon () at
>>>> ../src/backend/postmaster/auxprocess.c:65
>>>>     #5  0x0000556191940676 in IoWorkerMain (startup_data=<optimized
>>>> out>, startup_data_len=<optimized out>) at
>>>> ../src/backend/storage/aio/method_worker.c:393
>>>>     #6  0x00005561918e8163 in postmaster_child_launch
>>>> (child_type=child_type@entry=B_IO_WORKER, child_slot=20086,
>>>> startup_data=startup_data@entry=0x0,
>>>>         startup_data_len=startup_data_len@entry=0,
>>>> client_sock=client_sock@entry=0x0) at
>>>> ../src/backend/postmaster/launch_backend.c:290
>>>>     #7  0x00005561918ea09a in StartChildProcess
>>>> (type=type@entry=B_IO_WORKER) at
>>>> ../src/backend/postmaster/postmaster.c:3973
>>>>     #8  0x00005561918ea308 in maybe_adjust_io_workers () at
>>>> ../src/backend/postmaster/postmaster.c:4404
>>>>     [..]
>>>>     (gdb) print *MyProc->sem
>>>>     Cannot access memory at address 0x0
>>>>
>>>
>>> Yeah, good catch. I'll look into that next week.
>>>
>>
>> I've been unable to reproduce this issue, but I'm not sure what settings
>> you actually used for this instance. Can you give me more details how to
>> reproduce this?
> 
> Better late than never, well feel free to partially ignore me, i've
> missed that it is known issue as per FIXME there, but I would just rip
> out that commented out `if(numa_proc_interleave)` from
> FastPathLockShmemSize() and PGProcShmemSize() unless you want to save
> those memory pages of course (in case of no-NUMA). If you do want to
> save those pages I think we have problem:
> 
> For complete picture, steps:
> 
> 1. patch -p1 < v2-0001-NUMA-interleaving-buffers.patch
> 2. patch -p1 < v2-0006-NUMA-interleave-PGPROC-entries.patch
> 
> BTW the pgbench accidentinal ident is still there (part of v2-0001 patch))
> 14 out of 14 hunks FAILED -- saving rejects to file
> src/bin/pgbench/pgbench.c.rej
> 
> 3. As I'm just applying 0001 and 0006, I've got two simple rejects,
> but fixed it (due to not applying missing numa_ freelist patches).
> That's intentional on my part, because I wanted to play just with
> those two.
> 
> 4. Then I uncomment those two "if (numa_procs_interleave)" related for
> optional memory shm initialization - add_size() and so on (that have
> XXX comment above that it is causing bootstrap issues)
> 

Ah, I didn't realize you uncommented these "if" conditions. In that case
the crash is not very surprising, because the actual initialization in
InitProcGlobal ignores the GUCs and just assumes it's enabled. But
without the extra padding that likely messes up something. Or something
allocated later "overwrites" the some of the memory.

I need to clean this up, to actually consider the GUC properly.

FWIW I do have a new patch version that I plan to share in a day or two,
once I get some numbers. It didn't change this particular part, though,
it's more about the buffers/freelists/clocksweep. I'll work on PGPROC
next, I think.


regards

-- 
Tomas Vondra

  48. Re: Adding basic NUMA awareness

    Tomas Vondra <tomas@vondra.me> — 2025-07-28T14:19:07Z

    Hi,

Here's a somewhat cleaned up v3 of this patch series, with various
improvements and a lot of cleanup. Still WIP, but I hope it resolves the
various crashes reported for v2, but it still requires --with-libnuma
(it won't build without it).

I'm aware there's an ongoing discussion about removing the freelists,
and changing the clocksweep in some way. If that happens, the relevant
parts of this series will need some adjustment, of course. I haven't
looked into that yet, I plan to review those patches soon.


main changes in v3
------------------

1) I've introduced "registry" of the buffer partitions (imagine a small
array of structs), serving as a source of truth for places that need
info about the partitions (range of buffers, ...).

With v2 there was no "shared definition" - the shared buffers, freelist
and clocksweep did their own thing. But per the discussion it doesn't
really make much sense for to partition buffers in different ways.

So in v3 the 0001 patch defines the partitions, records them in shared
memory (in a small array), and the later parts just reuse this.

I also added a pg_buffercache_partitions() listing the partitions, with
first/last buffer, etc. The freelist/clocksweep patches add additional
information.


2) The PGPROC part introduces a similar registry, even though there are
no other patches building on this. But it seemed useful to have a clear
place recording this info.

There's also a view pg_buffercache_pgproc. The pg_buffercache location
is a bit bogus - it has nothing to do with buffers, but it was good
enough for now.


3) The PGPROC partitioning is reworked and should fix the crash with the
GUC set to "off".


4) This still doesn't do anything about "balancing" the clocksweep. I
have some ideas how to do that, I'll work on that next.


simple benchmark
----------------

I did a simple benchmark, measuring pgbench throughput with scale still
fitting into RAM, but much larger (~2x) than shared buffers. See the
attached test script, testing builds with more and more of the patches.

I'm attaching results from two different machines (the "usual" 2P xeon
and also a much larger cloud instance with EPYC/Genoa) - both the raw
CSV files, with average tps and percentiles, and PDFs. The PDFs also
have a comparison either to the "preceding" build (right side), or to
master (below the table).

There's results for the three "pgbench pinning" strategies, and that can
have pretty significant impact (colocated generally performs much better
than either "none" or "random").

For the "bigger" machine (wiuth 176 cores) the incremental results look
like this (for pinning=none, i.e. regular pgbench):


      mode   s_b buffers localal no-tail freelist sweep pgproc pinning
  ====================================================================
  prepared  16GB     99%    101%    100%     103%  111%    99%    102%
            32GB     98%    102%     99%     103%  107%   101%    112%
             8GB     97%    102%    100%     102%  101%   101%    106%
  --------------------------------------------------------------------
    simple  16GB    100%    100%     99%     105%  108%    99%    108%
            32GB     98%    101%    100%     103%  100%   101%     97%
             8GB    100%    100%    101%      99%  100%   104%    104%

The way I read this is that the first three patches have about no impact
on throughput. Then freelist partitioning and (especially) clocksweep
partitioning can help quite a bit. pgproc is again close to ~0%, and
PGPROC pinning can help again (but this part is merely experimental).

For the xeon the differences (in either direction) are much smaller, so
I'm not going to post it here. It's in the PDF, though.

I think this looks reasonable. The way I see this patch series is not
about improving peak throughput, but more about reducing imbalance and
making the behavior more consistent.

The results are more a confirmation there's not some sort of massive
overhead, somewhere. But I'll get to this in a minute.

To quantify this kind of improvement, I think we'll need tests that
intentionally cause (or try to) imbalance. If you have ideas for such
tests, let me know.


overhead of partitioning calculation
------------------------------------

Regarding the "overhead", while the results look mostly OK, I think
we'll need to rethink the partitioning scheme - particularly how the
partition size is calculated. The current scheme has to use %, which can
be somewhat expensive.

The 0001 patch calculates a "chunk size", which is the smallest number
of buffers it can "assign" to a NUMA node. This depends on how many
buffer descriptors fit onto a single memory page, and it's either 512KB
(with 4KB pages), or 256MB (with 2MB huge pages). And then each NUMA
node gets multiple chunks, to cover shared_buffers/num_nodes. But this
can be an arbitrary number - it minimizes the imbalance, but it also
forces the use of % and / in the formulas.

AFAIK if we required the partitions to be 2^k multiples of the chunk
size, we could switch to using shifts and masking. Which is supposed to
be much faster. But I haven't measured this, and the cost is that some
of the nodes could get much less memory. Maybe that's fine.


reserving number of huge pages
------------------------------

The other thing I realized is that partitioning buffers with huge pages
is quite tricky, and can easily lead to SIGBUS when accessing the memory
later. The crashes I saw happen like this:

1) figure # of pages needed (using shared_memory_size_in_huge_pages)

   This can be 16828 for shared_buffers=32GB.

2) make sure there's enough huge pages

   echo 16828 > /proc/sys/vm/nr_hugepages

3) start postgres - everything seems to works just fine

4) query pg_buffercache_numa - triggers SIGBUS accessing memory for a
valid buffer (usually ~2GB from the end)

It took me ages to realize what's happening, but it's very simple. The
nr_hugepages is a global limit, but it's also translated into limits for
each NUMA node. So when you write 16828 to it, in a 4-node system each
node gets 1/4 of that. See

  $ numastat -cm

Then we do the mmap(), and everything looks great, because there really
is enough huge pages and the system can allocate memory from any NUMA
node it needs.

And then we come around, and do the numa_tonode_memory(). And that's
where the issues start, because AFAIK this does not check the per-node
limit of huge pages in any way. It just appears to work. And then later,
when we finally touch the buffer, it tries to actually allocate the
memory on the node, and realizes there's not enough huge pages. And
triggers the SIGBUS.

You may ask why the per-node limit is too low. We still need just
shared_memory_size_in_huge_pages, right? And if we were partitioning the
whole memory segment, that'd be true. But we only to that for shared
buffers, and there's a lot of other shared memory - could be 1-2GB or
so, depending on the configuration.

And this gets placed on one of the nodes, and it counts against the
limit on that particular node. And so it doesn't have enough huge pages
to back the partition of shared buffers.

The only way around this I found is by inflating the number of huge
pages, significantly above the shared_memory_size_in_huge_pages value.
Just to make sure the nodes get enough huge pages.

I don't know what to do about this. It's quite annoying. If we only used
huge pages for the partitioned parts, this wouldn't be a problem.

I also realize this can be used to make sure the memory is balanced on
NUMA systems. Because if you set nr_hugepages, the kernel will ensure
the shared memory is distributed on all the nodes.

It won't have the benefits of "coordinating" the buffers and buffer
descriptors, and so on. But it will be balanced.


regards

-- 
Tomas Vondra

  49. Re: Adding basic NUMA awareness

    Jakub Wartak <jakub.wartak@enterprisedb.com> — 2025-07-30T08:29:06Z

    On Mon, Jul 28, 2025 at 4:22 PM Tomas Vondra <tomas@vondra.me> wrote:

Hi Tomas,

just a quick look here:

> 2) The PGPROC part introduces a similar registry, [..]
>
> There's also a view pg_buffercache_pgproc. The pg_buffercache location
> is a bit bogus - it has nothing to do with buffers, but it was good
> enough for now.

If you are looking for better names: pg_shmem_pgproc_numa would sound
like a more natural name.

> 3) The PGPROC partitioning is reworked and should fix the crash with the
> GUC set to "off".

Thanks!

> simple benchmark
> ----------------
[..]
> There's results for the three "pgbench pinning" strategies, and that can
> have pretty significant impact (colocated generally performs much better
> than either "none" or "random").

Hint: real world is that network cards are usually located on some PCI
slot that is assigned to certain node (so traffic is flowing from/to
there), so probably it would make some sense to put pgbench outside
this machine and remove this as "variable" anyway and remove the need
for that pgbench --pin-cpus in script. In optimal conditions: most
optimized layout would be probably to have 2 cards on separate PCI
slots, each for different node and some LACP between those, with
xmit_hash_policy allowing traffic distribution on both of those cards
-- usually there's not just single IP/MAC out there talking to/from
such server, so that would be real-world (or lack of) affinity.

Also classic pgbench workload, seems to be poor fit for testing it out
(at least v3-0001 buffers), there I would propose sticking to just
lots of big (~s_b size) full table seq scans to put stress on shared
memory. Classic pgbench is usually not there enough to put serious
bandwidth on the interconnect by my measurements.

> For the "bigger" machine (wiuth 176 cores) the incremental results look
> like this (for pinning=none, i.e. regular pgbench):
>
>
>       mode   s_b buffers localal no-tail freelist sweep pgproc pinning
>   ====================================================================
>   prepared  16GB     99%    101%    100%     103%  111%    99%    102%
>             32GB     98%    102%     99%     103%  107%   101%    112%
>              8GB     97%    102%    100%     102%  101%   101%    106%
>   --------------------------------------------------------------------
>     simple  16GB    100%    100%     99%     105%  108%    99%    108%
>             32GB     98%    101%    100%     103%  100%   101%     97%
>              8GB    100%    100%    101%      99%  100%   104%    104%
>
> The way I read this is that the first three patches have about no impact
> on throughput. Then freelist partitioning and (especially) clocksweep
> partitioning can help quite a bit. pgproc is again close to ~0%, and
> PGPROC pinning can help again (but this part is merely experimental).

Isn't the "pinning" column representing just numa_procs_pin=on ?
(shouldn't it be tested with numa_procs_interleave = on?)

[..]
> To quantify this kind of improvement, I think we'll need tests that
> intentionally cause (or try to) imbalance. If you have ideas for such
> tests, let me know.

Some ideas:
1. concurrent seq scans hitting s_b-sized table
2. one single giant PX-enabled seq scan with $VCPU workers (stresses
the importance of interleaving dynamic shm for workers)
3. select txid_current() with -M prepared?

> reserving number of huge pages
> ------------------------------
[..]
> It took me ages to realize what's happening, but it's very simple. The
> nr_hugepages is a global limit, but it's also translated into limits for
> each NUMA node. So when you write 16828 to it, in a 4-node system each
> node gets 1/4 of that. See
>
>   $ numastat -cm
>
> Then we do the mmap(), and everything looks great, because there really
> is enough huge pages and the system can allocate memory from any NUMA
> node it needs.

Yup, similiar story as with OOMs just for per-zone/node.

> And then we come around, and do the numa_tonode_memory(). And that's
> where the issues start, because AFAIK this does not check the per-node
> limit of huge pages in any way. It just appears to work. And then later,
> when we finally touch the buffer, it tries to actually allocate the
> memory on the node, and realizes there's not enough huge pages. And
> triggers the SIGBUS.

I think that's why options for strict policy numa allocation exist and
I had the option to use it in my patches (anyway with one big call to
numa_interleave_memory() for everything it was much more simpler and
just not micromanaging things). Good reads are numa(3) but e.g.
mbind(2) underneath will tell you that e.g. `Before Linux 5.7.
MPOL_MF_STRICT was ignored on huge page mappings.` (I was on 6.14.x,
but it could be happening for you too if you start using it). Anyway,
numa_set_strict() is just wrapper around setting this exact flag

Anyway remember that volatile pg_numa_touch_mem_if_required()? - maybe
that should be always called in your patch series to pre-populate
everything during startup, so that others testing will get proper
guaranteed layout, even without issuing any pg_buffercache calls.

> The only way around this I found is by inflating the number of huge
> pages, significantly above the shared_memory_size_in_huge_pages value.
> Just to make sure the nodes get enough huge pages.
>
> I don't know what to do about this. It's quite annoying. If we only used
> huge pages for the partitioned parts, this wouldn't be a problem.

Meh, sacrificing a couple of huge pages (worst-case 1GB ?) just to get
NUMA affinity, seems like a logical trade-off, doesn't it?
But postgres -C shared_memory_size_in_huge_pages  still works OK to
establish the exact count for vm.nr_hugepages, right?

Regards,
-J.

  50. Re: Adding basic NUMA awareness

    Tomas Vondra <tomas@vondra.me> — 2025-07-30T11:10:45Z

    On 7/30/25 10:29, Jakub Wartak wrote:
> On Mon, Jul 28, 2025 at 4:22 PM Tomas Vondra <tomas@vondra.me> wrote:
> 
> Hi Tomas,
> 
> just a quick look here:
> 
>> 2) The PGPROC part introduces a similar registry, [..]
>>
>> There's also a view pg_buffercache_pgproc. The pg_buffercache location
>> is a bit bogus - it has nothing to do with buffers, but it was good
>> enough for now.
> 
> If you are looking for better names: pg_shmem_pgproc_numa would sound
> like a more natural name.
> 
>> 3) The PGPROC partitioning is reworked and should fix the crash with the
>> GUC set to "off".
> 
> Thanks!
> 
>> simple benchmark
>> ----------------
> [..]
>> There's results for the three "pgbench pinning" strategies, and that can
>> have pretty significant impact (colocated generally performs much better
>> than either "none" or "random").
> 
> Hint: real world is that network cards are usually located on some PCI
> slot that is assigned to certain node (so traffic is flowing from/to
> there), so probably it would make some sense to put pgbench outside
> this machine and remove this as "variable" anyway and remove the need
> for that pgbench --pin-cpus in script. In optimal conditions: most
> optimized layout would be probably to have 2 cards on separate PCI
> slots, each for different node and some LACP between those, with
> xmit_hash_policy allowing traffic distribution on both of those cards
> -- usually there's not just single IP/MAC out there talking to/from
> such server, so that would be real-world (or lack of) affinity.
> 

The pgbench pinning certainly reduces some of the noise / overhead you
get when using multiple machines. I use it to "isolate" patches, and
make the effects more visible.

> Also classic pgbench workload, seems to be poor fit for testing it out
> (at least v3-0001 buffers), there I would propose sticking to just
> lots of big (~s_b size) full table seq scans to put stress on shared
> memory. Classic pgbench is usually not there enough to put serious
> bandwidth on the interconnect by my measurements.
> 

Yes, that's possible. The simple pgbench workload is a bit of a "worst
case" for the NUMA patches, in that it's can benefit less from the
improvements, and it's also fairly sensitive to regressions.

I plan to do more tests with other types of workloads, like the one
doing a lot of large sequential scans, etc.

>> For the "bigger" machine (wiuth 176 cores) the incremental results look
>> like this (for pinning=none, i.e. regular pgbench):
>>
>>
>>       mode   s_b buffers localal no-tail freelist sweep pgproc pinning
>>   ====================================================================
>>   prepared  16GB     99%    101%    100%     103%  111%    99%    102%
>>             32GB     98%    102%     99%     103%  107%   101%    112%
>>              8GB     97%    102%    100%     102%  101%   101%    106%
>>   --------------------------------------------------------------------
>>     simple  16GB    100%    100%     99%     105%  108%    99%    108%
>>             32GB     98%    101%    100%     103%  100%   101%     97%
>>              8GB    100%    100%    101%      99%  100%   104%    104%
>>
>> The way I read this is that the first three patches have about no impact
>> on throughput. Then freelist partitioning and (especially) clocksweep
>> partitioning can help quite a bit. pgproc is again close to ~0%, and
>> PGPROC pinning can help again (but this part is merely experimental).
> 
> Isn't the "pinning" column representing just numa_procs_pin=on ?
> (shouldn't it be tested with numa_procs_interleave = on?)
> 

Maybe I don't understand the question, but the last column (pinning)
compares two builds.

1) Build with all the patches up to "pgproc interleaving" (and all of
the GUCs set to "on").

2) Build with all the patches from (1), and "pinning" too (again, all
GUCs set to "on).

Or do I misunderstand the question?

> [..]
>> To quantify this kind of improvement, I think we'll need tests that
>> intentionally cause (or try to) imbalance. If you have ideas for such
>> tests, let me know.
> 
> Some ideas:
> 1. concurrent seq scans hitting s_b-sized table
> 2. one single giant PX-enabled seq scan with $VCPU workers (stresses
> the importance of interleaving dynamic shm for workers)
> 3. select txid_current() with -M prepared?
> 

Thanks. I think we'll try something like (1), but it'll need to be a bit
more elaborate, because scans on tables larger than 1/4 shared buffers
use a small circular buffer.

>> reserving number of huge pages
>> ------------------------------
> [..]
>> It took me ages to realize what's happening, but it's very simple. The
>> nr_hugepages is a global limit, but it's also translated into limits for
>> each NUMA node. So when you write 16828 to it, in a 4-node system each
>> node gets 1/4 of that. See
>>
>>   $ numastat -cm
>>
>> Then we do the mmap(), and everything looks great, because there really
>> is enough huge pages and the system can allocate memory from any NUMA
>> node it needs.
> 
> Yup, similiar story as with OOMs just for per-zone/node.
> 
>> And then we come around, and do the numa_tonode_memory(). And that's
>> where the issues start, because AFAIK this does not check the per-node
>> limit of huge pages in any way. It just appears to work. And then later,
>> when we finally touch the buffer, it tries to actually allocate the
>> memory on the node, and realizes there's not enough huge pages. And
>> triggers the SIGBUS.
> 
> I think that's why options for strict policy numa allocation exist and
> I had the option to use it in my patches (anyway with one big call to
> numa_interleave_memory() for everything it was much more simpler and
> just not micromanaging things). Good reads are numa(3) but e.g.
> mbind(2) underneath will tell you that e.g. `Before Linux 5.7.
> MPOL_MF_STRICT was ignored on huge page mappings.` (I was on 6.14.x,
> but it could be happening for you too if you start using it). Anyway,
> numa_set_strict() is just wrapper around setting this exact flag
> 
> Anyway remember that volatile pg_numa_touch_mem_if_required()? - maybe
> that should be always called in your patch series to pre-populate
> everything during startup, so that others testing will get proper
> guaranteed layout, even without issuing any pg_buffercache calls.
> 

I think I tried using numa_set_strict, but it didn't change the behavior
(i.e. the numa_tonode_memory didn't error out).

>> The only way around this I found is by inflating the number of huge
>> pages, significantly above the shared_memory_size_in_huge_pages value.
>> Just to make sure the nodes get enough huge pages.
>>
>> I don't know what to do about this. It's quite annoying. If we only used
>> huge pages for the partitioned parts, this wouldn't be a problem.
> 
> Meh, sacrificing a couple of huge pages (worst-case 1GB ?) just to get
> NUMA affinity, seems like a logical trade-off, doesn't it?
> But postgres -C shared_memory_size_in_huge_pages  still works OK to
> establish the exact count for vm.nr_hugepages, right?
> 

Well, yes and no. It tells you the exact number of huge pages, but it
does not tell you how much you need to inflate it to account for the
non-shared buffer part that may get allocated on a random node.


regards

-- 
Tomas Vondra

  51. Re: Adding basic NUMA awareness

    Tomas Vondra <tomas@vondra.me> — 2025-08-04T14:24:40Z

    
Hi,

Here's an updated version of the patch series. The main improvement is
the new 0006 patch, adding "adaptive balancing" of allocations. I'll
also share some results from a workload doing a lot of allocations.


adaptive balancing of allocations
---------------------------------

Imagine each backend only allocates buffers from the partition on the
same NUMA node. E.g. you have 4 NUMA nodes (i.e. 4 partitions), and a
backend only allocates buffers from "home" partition (on the same NUMA
node). This is what the earlier patch versions did, and with many
backends that's mostly fine (assuming the backends get spread over all
the NUMA nodes).

But if there's only few backends doing the allocations, this can result
in very inefficient use of shared buffers - a single backend would be
limited to 25% of buffers, even if the rest is unused.

There needs to be some say to "redirect" excess allocations to other
partitions, so that the partitions are utilized about the same. This is
what the 0006 patch aims to do (I kept is separate, but it should
probably get merged into the "clocksweep partitioning" in the end).

The balancing is fairly simple:

(1) It tracks the number of allocations "requested" from each partition.

(2) In regular intervals (by bgwriter) calculate the "fair share" per
partition, and determine what fraction of "requests" to handle from the
partition itself, and how many to redirect to other partitions.

(3) Calculate coefficients to drive this for each partition.


I emphasize (1) talks about "requests", not the actual allocations. Some
of the requests could have been redirected to different partitions, and
be counted as allocations there. We want to balance allocations, but it
relies on the requests.

To give you a simple example - imagine there are 2 partitions with this
number of allocation requests:

  P1: 900,000 requests
  P2: 100,000 requests

This means the "fair share" is 500,000 allocations, so P1 needs to
redirect some requests to P2. And we end with these weights:

  P1: [ 55,  45]
  P2: [  0, 100]

Assuming the workload does not shift in some dramatic way, this should
result in both partitions handling ~500k allocations.

It's not hard to extend this algorithm to more partitions. For more
details see StrategySyncBalance(), which recalculates this.

There are a couple open questions, like:

* The algorithm combines the old/new weights by averaging, to add a bit
of hysteresis. Right now it's a simple average with 0.5 weight, to
dampen sudden changes. I think it works fine (in the long run), but I'm
open to suggestions how to do this better.

* There's probably additional things we should consider when deciding
where to redirect the allocations. For example, we may have multiple
partitions per NUMA node, in which case it's better to redirect to that
node as many allocations as possible. The current patch ignores this.

* The partitions may have slightly different sizes, but the balancing
ignores that for now. This is not very difficult to address.


clocksweep benchmark
--------------------

I ran a simple benchmark focused on allocation-heavy workloads, namely
large concurrent sequential scans. The attached scripts generate a
number of 1GB tables, and then run concurrent sequential scans with
shared buffers set to 60%, 75%, 90% and 110% of the total dataset size.

I did this for master, and with the NUMA patches applied (and the GUCs
set to 'on'). I also increased tried with the of partitions increased to
16 (so a NUMA node got multiple partitions).

There are results from three machines

1) ryzen - small non-NUMA system, mostly to see if there's regressions

2) xeon - older 2-node NUMA system

3) hb176 - big EPYC system with 176 cores / 4 NUMA nodes


The script records detailed TPS stats (e.g. percentiles), I'm attaching
CSV files with complete results, and some PDFs with charts summarizing
that (I'll get to that in a minute).

For the EPYC, the average tps for the three builds looks like this:


    clients  |  master      numa   numa-16    |    numa     numa-16
   ----------|--------------------------------|---------------------
          8  |      20        27        26    |    133%        129%
         16  |      23        39        45    |    170%        193%
         24  |      23        48        58    |    211%        252%
         32  |      21        57        68    |    268%        321%
         40  |      21        56        76    |    265%        363%
         48  |      22        59        82    |    270%        375%
         56  |      22        66        88    |    296%        397%
         64  |      23        62        93    |    277%        411%
         72  |      24        68        95    |    277%        389%
         80  |      24        72        95    |    295%        391%
         88  |      25        71        98    |    283%        392%
         96  |      26        74        97    |    282%        369%
        104  |      26        74        97    |    282%        367%
        112  |      27        77        95    |    287%        355%
        120  |      28        77        92    |    279%        335%
        128  |      27        75        89    |    277%        328%

That's not bad - the clocksweep partitioning increases the throughput
2-3x. Having 16 partitions (instead of 4) helps yet a bit more, to 3-4x.

This is for shared buffers set to 60% of the dataset, which depends on
the number of clients / tables. With 64 clients/tables, there's 64GB of
data, and shared buffers are set to ~39GB.

The results for 75% and 90% follow the same pattern. For 110% there's
much less impact - there are no allocations, so this has to be thanks to
the other NUMA patches.

The charts in the attached PDFs add a bit more detail, with various
percentiles (of per-second throughput). The bands are roughly quartiles:
5-25%, 25-50%, 50-75%, 75-95%. The thick middle line is the median.

There's only charts for 60%, 90% and 110% shared buffers, for fit it on
a single page. There 75% is not very different.

For ryzen there's little difference. Not surprising, it's not a NUMA
system. So this is positive result, as there's no regression.

For xeon the patches help a little bit. Again, not surprising. It's a
fairly old system (~2016), and the differences between NUMA nodes are
not that significant.

For epyc (hb176), the differences are pretty massive.



regards

-- 
Tomas Vondra

  52. Re: Adding basic NUMA awareness

    Tomas Vondra <tomas@vondra.me> — 2025-08-07T09:24:18Z

    Hi!

Here's a slightly improved version of the patch series.

The main improvement is related to rebalancing partitions of different
sizes (which can happen because the sizes have to be a multiple of some
minimal "chunk" determined by memory page size etc.). Part 0009 deals
with that by adjusting the allocations by partition size. It works OK,
but it's also true it matters less as the shared_buffers size increases
(as the relative difference between large/small partition gets smaller).


The other improvements are related to the pg_buffercache_partitions
view, showing the weights and (running) totals of allocations.

I plan to take a break from this patch series for a while, so this would
be a good time to take a look, do a review, run some tests etc. ;-)


One detail about the balancing I forgot to mention in my last message is
how the patch "distributes" allocations to match the balancing weights.
Consider for example the example weights from that message:

  P1: [ 55,  45]
  P2: [  0, 100]

Imagine a backend located on P1 requests allocation of a buffer. The
weights say 55% buffers should be allocated from P1, and 45% should be
redirected to P2. One way to achieve that would be generating a random
number in [1, 100], and if it's [1,55] then P1, otherwise P2.

The patch does a much simpler thing - treat the weight as a "budget",
i.e. number of buffers to allocate before proceeding to the "next"
partition. So it allocates 55 buffers from P1, then 45 buffers from P2,
and then goes back to P1 in a round-robin way. The advantage is it can
do away without a PRNG.


There's two things I'm not entirely sure about:

1) memory model - I'm not quite sure the current code ensures updates to
weights are properly "communicated" to the other processes. That is, if
the bgwriter recalculates the weights, will the other backends see the
new weights right away? Using a stale weights won't cause "failures",
the consequence is just a bit of imbalance. But it shouldn't stay like
that for too long, so maybe it'd be good to add some memory barriers or
something like that.

2) I'm a bit unsure what "NUMA nodes" actually means. The patch mostly
assumes each core / piece of RAM is assigned to a particular NUMA node.
For the buffer partitioning the patch mostly cares about memory, as it
"locates" the buffers on different NUMA nodes. Which works mostly OK
(ignoring the issues with huge pages described in previous message).

But it also cares about the cores (and the node for each core), because
it uses that to pick the right partition for a backend. And here the
situation is less clear, because the CPUs don't need to be assigned to a
particular node, even on a NUMA system. Consider the rpi5 NUMA layout:

$ numactl --hardware
available: 8 nodes (0-7)
node 0 cpus: 0 1 2 3
node 0 size: 992 MB
node 0 free: 274 MB
node 1 cpus: 0 1 2 3
node 1 size: 1019 MB
node 1 free: 327 MB
node 2 cpus: 0 1 2 3
node 2 size: 1019 MB
node 2 free: 321 MB
node 3 cpus: 0 1 2 3
node 3 size: 955 MB
node 3 free: 251 MB
node 4 cpus: 0 1 2 3
node 4 size: 1019 MB
node 4 free: 332 MB
node 5 cpus: 0 1 2 3
node 5 size: 1019 MB
node 5 free: 342 MB
node 6 cpus: 0 1 2 3
node 6 size: 1019 MB
node 6 free: 352 MB
node 7 cpus: 0 1 2 3
node 7 size: 1014 MB
node 7 free: 339 MB
node distances:
node   0   1   2   3   4   5   6   7
  0:  10  10  10  10  10  10  10  10
  1:  10  10  10  10  10  10  10  10
  2:  10  10  10  10  10  10  10  10
  3:  10  10  10  10  10  10  10  10
  4:  10  10  10  10  10  10  10  10
  5:  10  10  10  10  10  10  10  10
  6:  10  10  10  10  10  10  10  10
  7:  10  10  10  10  10  10  10  10

This says there are 8 NUMA nodes, each with ~1GB of RAM. But the 4 cores
are not assigned to particular nodes - each core is mapped to all 8 NUMA
nodes. I'm not sure what to do about this (or how getcpu() or libnuma
handle this). And can the situation be even more complicated?


regards

-- 
Tomas Vondra

  53. Re: Adding basic NUMA awareness

    Tomas Vondra <tomas@vondra.me> — 2025-08-07T09:36:10Z

    On 8/7/25 11:24, Tomas Vondra wrote:
> Hi!
> 
> Here's a slightly improved version of the patch series.
> 

Ah, I made a mistake when generating the patches. The 0001 and 0002
patches are not part of the NUMA stuff, it's just something related to
benchmarking (addressing unrelated bottlenecks etc.). The actual NUMA
patches start with 0003.

Also, 0007, 0008 and 0009 should ultimately be collapsed into a single
patch. It's all about the clocksweep partitioning, I only kept those
separate to make it easier to see the changes and review.


regards

-- 
Tomas Vondra

  54. Re: Adding basic NUMA awareness

    Andres Freund <andres@anarazel.de> — 2025-08-09T00:25:40Z

    Hi,

On 2025-08-07 11:24:18 +0200, Tomas Vondra wrote:
> 2) I'm a bit unsure what "NUMA nodes" actually means. The patch mostly
> assumes each core / piece of RAM is assigned to a particular NUMA node.

There are systems in which some NUMA nodes do *not* contain any CPUs. E.g. if
you attach memory via a CXL/PCIe add-in card, rather than via the CPUs memory
controller. In that case numactl -H (and obviously also the libnuma APIs) will
report that the numa node is not associated with any CPU.

I don't currently have live access to such a system, but this PR piece happens
to have numactl -H output:
https://lenovopress.lenovo.com/lp2184-implementing-cxl-memory-on-linux-on-thinksystem-v4-servers
> numactl -H
> available: 4 nodes (0-3)
> node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143
> node 0 size: 1031904 MB
> node 0 free: 1025554 MB
> node 1 cpus: 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191
> node 1 size: 1032105 MB
> node 1 free: 1024244 MB
> node 2 cpus:
> node 2 size: 262144 MB
> node 2 free: 262143 MB
> node 3 cpus:
> node 3 size: 262144 MB
> node 3 free: 262142 MB
> node distances:
> node   0   1   2   3
>   0:  10  21  14  24
>   1:  21  10  24  14
>   2:  14  24  10  26
>   3:  24  14  26  10

Note that node 2 & 3 don't have associated CPUs (and higher access costs).

I don't think this is common enough to worry about from a performance POV, but
we probably shouldn't crash if we encounter it...


> But it also cares about the cores (and the node for each core), because
> it uses that to pick the right partition for a backend. And here the
> situation is less clear, because the CPUs don't need to be assigned to a
> particular node, even on a NUMA system. Consider the rpi5 NUMA layout:
>
> $ numactl --hardware
> available: 8 nodes (0-7)
> node 0 cpus: 0 1 2 3
> node 0 size: 992 MB
> node 0 free: 274 MB
> node 1 cpus: 0 1 2 3
> node 1 size: 1019 MB
> node 1 free: 327 MB
> ...
> node   0   1   2   3   4   5   6   7
>   0:  10  10  10  10  10  10  10  10
>   1:  10  10  10  10  10  10  10  10
>   2:  10  10  10  10  10  10  10  10
>   3:  10  10  10  10  10  10  10  10
>   4:  10  10  10  10  10  10  10  10
>   5:  10  10  10  10  10  10  10  10
>   6:  10  10  10  10  10  10  10  10
>   7:  10  10  10  10  10  10  10  10
> This says there are 8 NUMA nodes, each with ~1GB of RAM. But the 4 cores
> are not assigned to particular nodes - each core is mapped to all 8 NUMA
> nodes.

FWIW, you can get a different version of this with AMD Epyc too, if "L3 LLC as
NUMA" is enabled.


> I'm not sure what to do about this (or how getcpu() or libnuma handle this).

I don't immediately see any libnuma functions that would care?

I also am somewhat curious about what getcpu() returns for the current node...

Greetings,

Andres Freund

  55. Re: Adding basic NUMA awareness

    Tomas Vondra <tomas@vondra.me> — 2025-08-12T11:04:07Z

    
On 8/9/25 02:25, Andres Freund wrote:
> Hi,
> 
> On 2025-08-07 11:24:18 +0200, Tomas Vondra wrote:
>> 2) I'm a bit unsure what "NUMA nodes" actually means. The patch mostly
>> assumes each core / piece of RAM is assigned to a particular NUMA node.
> 
> There are systems in which some NUMA nodes do *not* contain any CPUs. E.g. if
> you attach memory via a CXL/PCIe add-in card, rather than via the CPUs memory
> controller. In that case numactl -H (and obviously also the libnuma APIs) will
> report that the numa node is not associated with any CPU.
> 
> I don't currently have live access to such a system, but this PR piece happens
> to have numactl -H output:
> https://lenovopress.lenovo.com/lp2184-implementing-cxl-memory-on-linux-on-thinksystem-v4-servers
>> numactl -H
>> available: 4 nodes (0-3)
>> node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143
>> node 0 size: 1031904 MB
>> node 0 free: 1025554 MB
>> node 1 cpus: 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191
>> node 1 size: 1032105 MB
>> node 1 free: 1024244 MB
>> node 2 cpus:
>> node 2 size: 262144 MB
>> node 2 free: 262143 MB
>> node 3 cpus:
>> node 3 size: 262144 MB
>> node 3 free: 262142 MB
>> node distances:
>> node   0   1   2   3
>>   0:  10  21  14  24
>>   1:  21  10  24  14
>>   2:  14  24  10  26
>>   3:  24  14  26  10
> 
> Note that node 2 & 3 don't have associated CPUs (and higher access costs).
> 
> I don't think this is common enough to worry about from a performance POV, but
> we probably shouldn't crash if we encounter it...
> 

Right. I don't think the current patch would crash - I can't test it,
but I don't see why it would crash. In the worst case it'd end up with
partitions that are not ideal. The question is more what would an ideal
partitioning for buffers and PGPROC look like. Any opinions?

For PGPROC, it's simple - it doesn't make sense to allocate partitions
for nodes without CPUs.

For buffers, it probably does not really matter if a node does not have
any CPUs. If a node does not have any CPUs, that does not mean we should
not put any buffers on it. After all, CXL will never have any CPUs (at
least I think that's the case), and not using it for shared buffers
would be a bit strange. Although, it could still be used for page cache.

Maybe it should be "tiered" a bit more? The patch differentiates only
between partitions on "my" NUMA node vs. every other partition. Maybe it
should have more layers?

That'd make the "balancing" harder. But I wanted to make it a smarter
when handling cases with multiple partitions per NUMA node.

> 
>> But it also cares about the cores (and the node for each core), because
>> it uses that to pick the right partition for a backend. And here the
>> situation is less clear, because the CPUs don't need to be assigned to a
>> particular node, even on a NUMA system. Consider the rpi5 NUMA layout:
>>
>> $ numactl --hardware
>> available: 8 nodes (0-7)
>> node 0 cpus: 0 1 2 3
>> node 0 size: 992 MB
>> node 0 free: 274 MB
>> node 1 cpus: 0 1 2 3
>> node 1 size: 1019 MB
>> node 1 free: 327 MB
>> ...
>> node   0   1   2   3   4   5   6   7
>>   0:  10  10  10  10  10  10  10  10
>>   1:  10  10  10  10  10  10  10  10
>>   2:  10  10  10  10  10  10  10  10
>>   3:  10  10  10  10  10  10  10  10
>>   4:  10  10  10  10  10  10  10  10
>>   5:  10  10  10  10  10  10  10  10
>>   6:  10  10  10  10  10  10  10  10
>>   7:  10  10  10  10  10  10  10  10
>> This says there are 8 NUMA nodes, each with ~1GB of RAM. But the 4 cores
>> are not assigned to particular nodes - each core is mapped to all 8 NUMA
>> nodes.
> 
> FWIW, you can get a different version of this with AMD Epyc too, if "L3 LLC as
> NUMA" is enabled.
> 
> 
>> I'm not sure what to do about this (or how getcpu() or libnuma handle this).
> 
> I don't immediately see any libnuma functions that would care?
> 

Not sure what "care" means here. I don't think it's necessarily broken,
it's more about the APIs not making the situation very clear (or
convenient).

How do you determine nodes for a CPU, for example? The closest thing I
see is numa_node_of_cpu(), but that only returns a single node. Or how
would you determine the number of nodes with CPUs (so that we create
PGPROC partitions only for those)? I suppose that requires literally
walking all the nodes.

> I also am somewhat curious about what getcpu() returns for the current node...
> 

It seems it only returns node 0. The cpu changes, but the node does not.

regards

-- 
Tomas Vondra

  56. Re: Adding basic NUMA awareness

    Andres Freund <andres@anarazel.de> — 2025-08-12T14:24:15Z

    Hi,

On 2025-08-12 13:04:07 +0200, Tomas Vondra wrote:
> Right. I don't think the current patch would crash - I can't test it,
> but I don't see why it would crash. In the worst case it'd end up with
> partitions that are not ideal. The question is more what would an ideal
> partitioning for buffers and PGPROC look like. Any opinions?
> 
> For PGPROC, it's simple - it doesn't make sense to allocate partitions
> for nodes without CPUs.
> 
> For buffers, it probably does not really matter if a node does not have
> any CPUs. If a node does not have any CPUs, that does not mean we should
> not put any buffers on it. After all, CXL will never have any CPUs (at
> least I think that's the case), and not using it for shared buffers
> would be a bit strange. Although, it could still be used for page cache.

For CXL memory to be really usable, I think we'd need nontrivial additional
work. CXL memory has considerably higher latency and lower throughput. You'd
*never* want things like BufferDescs or such on such nodes. And even the
buffered data itself, you'd want to make sure that frequently used data,
e.g. inner index pages, never end up on it.

Which leads to:

> Maybe it should be "tiered" a bit more?

Yes, for proper CXL support, we'd need a component that explicitly demotes and
promotes pages from "real" memory to CXL memory and the other way round. The
demotion is relatively easy, you'd probably just do it whenever you'd
otherwise throw out a victim buffer. When to promote back is harder...


> The patch differentiates only between partitions on "my" NUMA node vs. every
> other partition. Maybe it should have more layers?

Given the relative unavailability of CXL memory systems, I think just not
crashing is good enough for now...


> >> I'm not sure what to do about this (or how getcpu() or libnuma handle this).
> > 
> > I don't immediately see any libnuma functions that would care?
> > 
> 
> Not sure what "care" means here. I don't think it's necessarily broken,
> it's more about the APIs not making the situation very clear (or
> convenient).

What I mean is that I was looking through the libnuma functions and didn't see
any that would be affected by having multiple "local" NUMA nodes. But:


> How do you determine nodes for a CPU, for example? The closest thing I
> see is numa_node_of_cpu(), but that only returns a single node. Or how
> would you determine the number of nodes with CPUs (so that we create
> PGPROC partitions only for those)? I suppose that requires literally
> walking all the nodes.

I didn't think of numa_node_of_cpu().

As long as numa_node_of_cpu() returns *something* I think it may be good
enough. Nobody uses an RPi for high-throughput postgres workloads with a lot
of memory. Slightly sub-optimal mappings should really not matter.

I'm kinda wondering if we should deal with such fake numa systems by detecting
them and disabling our numa support.

Greetings,

Andres Freund

  57. Re: Adding basic NUMA awareness

    Tomas Vondra <tomas@vondra.me> — 2025-08-12T15:06:50Z

    
On 8/12/25 16:24, Andres Freund wrote:
> Hi,
> 
> On 2025-08-12 13:04:07 +0200, Tomas Vondra wrote:
>> Right. I don't think the current patch would crash - I can't test it,
>> but I don't see why it would crash. In the worst case it'd end up with
>> partitions that are not ideal. The question is more what would an ideal
>> partitioning for buffers and PGPROC look like. Any opinions?
>>
>> For PGPROC, it's simple - it doesn't make sense to allocate partitions
>> for nodes without CPUs.
>>
>> For buffers, it probably does not really matter if a node does not have
>> any CPUs. If a node does not have any CPUs, that does not mean we should
>> not put any buffers on it. After all, CXL will never have any CPUs (at
>> least I think that's the case), and not using it for shared buffers
>> would be a bit strange. Although, it could still be used for page cache.
> 
> For CXL memory to be really usable, I think we'd need nontrivial additional
> work. CXL memory has considerably higher latency and lower throughput. You'd
> *never* want things like BufferDescs or such on such nodes. And even the
> buffered data itself, you'd want to make sure that frequently used data,
> e.g. inner index pages, never end up on it.
> 

OK, let's keep that out of scope for these patches and assume we're
dealing only with local memory. CXL could still be used by the OS for
page cache, of whatever.

What does that mean for the patch, though. Does it need a way to
configure which nodes to use? I argued to leave this to the OS/numactl,
and we'd just use whatever is made available to Postgres. But maybe
we'll need something within Postgres after all?

FWIW there's work needed a actually inherit NUMA info from the OS. Right
now the patches just use all NUMA nodes, indexed by 0 ... (N-1) etc. I
like the "registry" concept I used for buffer/PGPROC partitions, it made
the patches much simpler. Maybe we should use something like that for
NUMA info too. That is, at startup build a record of the NUMA layout,
and use this as source of truth everywhere (instead of using libnuma
from all those places).

> Which leads to:
> 
>> Maybe it should be "tiered" a bit more?
> 
> Yes, for proper CXL support, we'd need a component that explicitly demotes and
> promotes pages from "real" memory to CXL memory and the other way round. The
> demotion is relatively easy, you'd probably just do it whenever you'd
> otherwise throw out a victim buffer. When to promote back is harder...
> 

Sounds very much like page cache (but that only works for buffered I/O).

> 
>> The patch differentiates only between partitions on "my" NUMA node vs. every
>> other partition. Maybe it should have more layers?
> 
> Given the relative unavailability of CXL memory systems, I think just not
> crashing is good enough for now...
> 

The lowest of bars ;-)

> 
>>>> I'm not sure what to do about this (or how getcpu() or libnuma handle this).
>>>
>>> I don't immediately see any libnuma functions that would care?
>>>
>>
>> Not sure what "care" means here. I don't think it's necessarily broken,
>> it's more about the APIs not making the situation very clear (or
>> convenient).
> 
> What I mean is that I was looking through the libnuma functions and didn't see
> any that would be affected by having multiple "local" NUMA nodes. But:
> 

My question is a bit of a "reverse" to this. That is, how do we even
find (with libnuma) there are multiple local nodes?

> 
>> How do you determine nodes for a CPU, for example? The closest thing I
>> see is numa_node_of_cpu(), but that only returns a single node. Or how
>> would you determine the number of nodes with CPUs (so that we create
>> PGPROC partitions only for those)? I suppose that requires literally
>> walking all the nodes.
> 
> I didn't think of numa_node_of_cpu().
> 

Yeah. I think most of the libnuma API is designed for each CPU belonging
to single NUMA node. I suppose we'd need to use numa_node_to_cpus() to
build this kind of information ourselves.

> As long as numa_node_of_cpu() returns *something* I think it may be good
> enough. Nobody uses an RPi for high-throughput postgres workloads with a lot
> of memory. Slightly sub-optimal mappings should really not matter.
> 

I'm not really concerned about rpi, or the performance on it. I only use
it as an example of system with "weird" NUMA layout.

> I'm kinda wondering if we should deal with such fake numa systems by detecting
> them and disabling our numa support.
> 

That'd be an option too, if we can identify such systems. We could do
that while building the "NUMA registry" I mentioned earlier.


regards

-- 
Tomas Vondra

  58. Re: Adding basic NUMA awareness

    Andres Freund <andres@anarazel.de> — 2025-08-13T15:16:24Z

    Hi,

On 2025-08-07 11:24:18 +0200, Tomas Vondra wrote:
> The patch does a much simpler thing - treat the weight as a "budget",
> i.e. number of buffers to allocate before proceeding to the "next"
> partition. So it allocates 55 buffers from P1, then 45 buffers from P2,
> and then goes back to P1 in a round-robin way. The advantage is it can
> do away without a PRNG.

I think that's a good plan.


A few comments about the clock sweep patch:

- It'd be easier to review if BgBufferSync() weren't basically re-indented
  wholesale. Maybe you could instead move the relevant code to a helper
  function that's called by BgBufferSync() for each clock?

- I think choosing a clock sweep partition in every tick would likely show up
  in workloads that do a lot of buffer replacement, particularly if buffers
  in the workload often have a high usagecount (and thus more ticks are used).
  Given that your balancing approach "sticks" with a partition for a while,
  could we perhaps only choose the partition after exhausting that budget?

- I don't really understand what

> +	/*
> +	 * Buffers that should have been allocated in this partition (but might
> +	 * have been redirected to keep allocations balanced).
> +	 */
> +	pg_atomic_uint32 numRequestedAllocs;
> +

  is intended for.

  Adding yet another atomic increment for every clock sweep tick seems rather
  expensive...


- I wonder if the balancing budgets being relatively low will be good
  enough. It's not too hard to imagine that this frequent "partition choosing"
  will be bad in buffer access heavy workloads. But it's probably the right
  approach until we've measured it being a problem.


- It'd be interesting to do some very simple evaluation like a single
  pg_prewarm() of a relation that's close to the size of shared buffers and
  verify that we don't end up evicting newly read in buffers.  I think your
  approach should work, but verifying that...

  I wonder if we could make some of this into tests somehow. It's pretty easy
  to break this kind of thing and not notice, as everything just continues to
  work, just a tad slower.

Greetings,

Andres Freund

  59. Re: Adding basic NUMA awareness

    Tomas Vondra <tomas@vondra.me> — 2025-08-13T16:36:17Z

    On 8/13/25 17:16, Andres Freund wrote:
> Hi,
> 
> On 2025-08-07 11:24:18 +0200, Tomas Vondra wrote:
>> The patch does a much simpler thing - treat the weight as a "budget",
>> i.e. number of buffers to allocate before proceeding to the "next"
>> partition. So it allocates 55 buffers from P1, then 45 buffers from P2,
>> and then goes back to P1 in a round-robin way. The advantage is it can
>> do away without a PRNG.
> 
> I think that's a good plan.
> 
> 
> A few comments about the clock sweep patch:
> 
> - It'd be easier to review if BgBufferSync() weren't basically re-indented
>   wholesale. Maybe you could instead move the relevant code to a helper
>   function that's called by BgBufferSync() for each clock?
> 

True, I'll rework it like that.

> - I think choosing a clock sweep partition in every tick would likely show up
>   in workloads that do a lot of buffer replacement, particularly if buffers
>   in the workload often have a high usagecount (and thus more ticks are used).
>   Given that your balancing approach "sticks" with a partition for a while,
>   could we perhaps only choose the partition after exhausting that budget?
> 

That should be possible, yes. By "exhausting budget" you mean going
through all the partitions, right?

> - I don't really understand what
> 
>> +	/*
>> +	 * Buffers that should have been allocated in this partition (but might
>> +	 * have been redirected to keep allocations balanced).
>> +	 */
>> +	pg_atomic_uint32 numRequestedAllocs;
>> +
> 
>   is intended for.
> 
>   Adding yet another atomic increment for every clock sweep tick seems rather
>   expensive...
> 

For the balancing (to calculate the budgets), we need to know the number
of allocation requests for each partition, before some of the requests
got redirected to other partitions. We can't use the number of "actual"
allocations. But it seems useful to have both - one to calculate the
budgets, the other to monitor how balanced the result is.

I haven't seen the extra atomic in profiles, even on workloads that do a
lot of buffer allocations (e.g. seqscan with datasets > shared buffers).
But if that happens, I think there are ways to mitigate that.

> 
> - I wonder if the balancing budgets being relatively low will be good
>   enough. It's not too hard to imagine that this frequent "partition choosing"
>   will be bad in buffer access heavy workloads. But it's probably the right
>   approach until we've measured it being a problem.
> 

I don't follow. How would making the budgets higher change any of this?

Anyway, I think choosing the partitions less frequently - e.g. only
after consuming budget for the current partition, or going "full cycle",
would make this a non-issue.

> 
> - It'd be interesting to do some very simple evaluation like a single
>   pg_prewarm() of a relation that's close to the size of shared buffers and
>   verify that we don't end up evicting newly read in buffers.  I think your
>   approach should work, but verifying that...
> 

Will try.

>   I wonder if we could make some of this into tests somehow. It's pretty easy
>   to break this kind of thing and not notice, as everything just continues to
>   work, just a tad slower.
> 

Do you mean a test that'd be a part of make check, or a standalone test?
AFAICS any meaningful test would need to be fairly expensive, so
probably not a good fit for make check.


regards

-- 
Tomas Vondra

  60. Re: Adding basic NUMA awareness

    Tomas Vondra <tomas@vondra.me> — 2025-09-11T08:32:40Z

    Hi,

Here's a fresh version of the NUMA patch series. There's a number of
substantial improvements:

1) Rebase to current master, particularly on top of 2c7894052759 which
removed the freelist. The patch that partitioned the freelist is gone.

2) A bunch of fixes, and it now passes CI workflows on github, and all
other testing I did. Of course, more testing is needed.

3) It builds with/without libnuma support, and so on.

4) The separate GUCs were replaced by a single list GUC, similar to what
we do for debug_io_direct. The GUC is called debug_numa, and accepts
"buffets" and "procs", to partition the two areas.

I'm considering adding a "clock-sweep" option in a future patch. I
didn't do that here, because the buffer partitioning is already enabled
by "buffers", and the clock-sweep just builds on that (partitioning the
same way, pretty much).

5) It also works with EXEC_BACKEND, but this turned out to be a bit more
challenging than expected. The trouble is some of the parameters (e.g.
memory page size) are used both in the "size" and "init" phases, and we
need to be extra careful to not make something "inconsistent".

For example, we may get confused about the memory page size. The "size"
happens before allocation, and at that point we don't know if we succeed
in getting enough huge pages. When "init" happens, we already know that,
so our "memory page size" could be different. We must be careful, e.g.
to not need more memory than we requested.

This is a general problem, but the EXEC_BACKEND makes it a bit trickier.
In regular fork() case we can simply set some global variables in "size"
and use them later in "init", but that doesn't work for EXEC_BACKEND.

The current approach simply does the calculations twice, in a way that
should end with the same results. But I'm not 100% happy with it, and I
suspect it just confirms we should store the results in shmem memory
(which is what I called "NUMA registry" before). That's still a TODO.

6) BufferDescPadded is 64B everywhere. Originally, the padding was
applied only on 64-bit platforms, and on 32-bit systems the struct was
left at 60B. But that is not compatible with buffer partitioning, which
relies on the memory page being a multiple of BufferDescPadded.

Perhaps it could be relaxed (so that a BufferDesc might span two memory
pages), but this seems like the cleanes solution. I don't expect it to
make any measurable difference.

7) I've moved some of the code to BgBufferSyncPartition, to make review
easier (without all the indentation changes).

8) I've realized some of the TAP tests occasionally fail with

    ERROR: no unpinned buffers

and I think I know why. Some of the tests set shared_buffers to a very
low value - like 1MB or even 128kB, and StrategyGetBuffer() may search
only a single partition (but not always). We may run out of unpinned
buffers in that one partition.

This apparently happens more easily on rpi5, due to the weird NUMA
layout (there are 8 nodes with memory, but getcpu() reports node 0 for
all cores).

I suspect the correct fix is to ensure StrategyGetBuffer() scans all
partitions, if there are no unpinned buffers in the current one. On
realistic setups this shouldn't happen very often, I think.

The other issue I just realized is that StrategyGetBuffer() recalculates
the partition index over and over, which seems unnecessary (and possibly
expensive, due to the modulo). And it also does too many loops, because
it used NBuffers instead of the partition size. I'll fix those later.

9) I'm keeping the cloc-sweep balancing patches separate for now. In the
end all the clock-sweep patches should be merged, but it keeps the
changes easier to review this way, I think.

10) There's not many changes in the PGPROC partitioning patch. I merely
fixed issues that broke it on EXEC_BACKEND, and did some smaller tweaks.

11) I'm not including the experimental patches to pin backends to CPUs
(or nodes), and so on. It's clear those are unlikely to go in, so it'd
be just a distraction.

12) What's the right / portable way to determine the current CPU for a
process? The clock-sweep / PGPROC patches need this to pick the right
partition, but it's not clear to me which API is the most portable. In
the end I used sched_getcpu(), and then numa_node_of_cpu(), but maybe
there's a better way.


regards

-- 
Tomas Vondra

  61. Re: Adding basic NUMA awareness

    Tomas Vondra <tomas@vondra.me> — 2025-09-11T15:41:23Z

    On 9/11/25 10:32, Tomas Vondra wrote:
> ...
> 
> For example, we may get confused about the memory page size. The "size"
> happens before allocation, and at that point we don't know if we succeed
> in getting enough huge pages. When "init" happens, we already know that,
> so our "memory page size" could be different. We must be careful, e.g.
> to not need more memory than we requested.

I forgot to mention the other issue with huge pages on NUMA. I already
reported [1] it's trivial to crash with a SIGBUS, because

(1) huge pages get reserved on all NUMA nodes (evenly)

(2) the decision whether to use huge pages is done by mmap(), which only
needs to check if there are enough huge pages in total

(3) numa_tonode_memory is called later, and does not verify if the
target node has enough free pages (I'm not sure it should / can)

(4) we only partition (and locate to NUMA nodes) some of the memory, and
the rest (which is much smaller, but still sizeable) is likely causing
"imbalance" - it gets placed on one (random) node, and it then does not
have enough space for the stuff we explicitly placed there

(5) then at some point we try accessing one of the shared buffers, that
triggers page fault, tries to get a huge page on the NUMA node, realizes
there are no free huge pages, and crashes with SIGBUS

It clearly is not an option to just let it crash, but I still don't have
a great idea how to address it. The only idea I have is to manually
interleave the whole shared memory (when using huge pages), page by
page, so that this imbalance does not happen.

But it's harder than it looks, because we don't necessarily partition
everything evenly. For example, one node can get a smaller chunk of
shared buffers, because we try to partition buffers and buffers
descriptors in a "nice" way. The PGPROC stuff is also not distributed
quite evenly (e.g. aux/2pc entries are not mapped to any node).

A different approach would be to calculate how many per-node huge pages
we'll need (for the stuff we partition explicitly - buffers and PGPROC),
and then the rest of the memory that can get placed on any node. And
require the "maximum" number of pages that can get placed on any node.
But that's annoying wasteful, because every other node will end up with
unusable memory.


regards

[1]
https://www.postgresql.org/message-id/71a46484-053c-4b81-ba32-ddac050a8b5d%40vondra.me

-- 
Tomas Vondra

  62. Re: Adding basic NUMA awareness

    Tomas Vondra <tomas@vondra.me> — 2025-09-18T21:04:45Z

    On 9/11/25 10:32, Tomas Vondra wrote:
> ...
>
> 8) I've realized some of the TAP tests occasionally fail with
> 
>     ERROR: no unpinned buffers
> 
> and I think I know why. Some of the tests set shared_buffers to a very
> low value - like 1MB or even 128kB, and StrategyGetBuffer() may search
> only a single partition (but not always). We may run out of unpinned
> buffers in that one partition.
> 
> This apparently happens more easily on rpi5, due to the weird NUMA
> layout (there are 8 nodes with memory, but getcpu() reports node 0 for
> all cores).
> 
> I suspect the correct fix is to ensure StrategyGetBuffer() scans all
> partitions, if there are no unpinned buffers in the current one. On
> realistic setups this shouldn't happen very often, I think.
> 
> The other issue I just realized is that StrategyGetBuffer() recalculates
> the partition index over and over, which seems unnecessary (and possibly
> expensive, due to the modulo). And it also does too many loops, because
> it used NBuffers instead of the partition size. I'll fix those later.

Here's a version fixing this issue (in the 0006 part). It modifies
StrategyGetBuffer() to walk through all the partitions, in a round-robin
manner. The way it steps to the next partition is a bit ugly, but it
works and I'll think about some better way.

I haven't done anything about the other issue (the one with huge pages
reserved on NUMA nodes, and SIGBUS).

regards

-- 
Tomas Vondra

  63. Re: Adding basic NUMA awareness

    Alexey Makhmutov <a.makhmutov@postgrespro.ru> — 2025-10-12T23:58:38Z

    Hi Tomas,

Thank you very much for working on this problem and the entire line of 
patches prepared! I've tried to play with these patches a little and 
here are some my observations and suggestions.

In the current implementation we try to use all available NUMA nodes on 
the machine, however it's often useful to limit the database only to a 
set of specific nodes, so that other nodes can be used for other 
processes. In my testing I was trying to use one node out of four for 
the client program, so I'd liked to limit the database to the remaining 
nodes. I use a systemd service with AllowedMemoryNodes/AllowedCPUs to 
start the cluster, so the obvious choice for me was to use the 
'numa_get_membind' function instead of 'numa_num_configured_nodes' to 
get the list of usable nodes. However, it is much easier to work with 
logical nodes in the [0; n] range inside the PG code, so I've decided to 
add mapping between 'logical nodes' (0-n in PG) to a set of physical 
nodes actually returned by 'numa_get_membind'. We may need to map number 
in both directions, so two translation tables are allocated and filled 
at the first usage of 'pg_numa' functions. It also seems to be a good 
idea to isolate all 'libnuma' calls inside 'pg_numa.c', so to keep all 
'numa_...' calls in it and this also allows us to hide this mapping in 
static functions. Here is the patch, which I've used to test this idea: 
https://github.com/Lerm/postgres/commit/9ec625c2bf564f5432375ec1d7ad02e4b2559161. 
This idea probably could be extended by adding some view to expose this 
mapping to the user (at least for testing purposes) and allow to 
explicitly override this mapping with a GUC setting. With such GUC 
setting we would be able to control PG memory usage on NUMA nodes 
without the need for systemd resource control or numactl parameters.

Next, I've noticed some problems related to the size alignment for 
'numa_tonode_memory' call in 'pg_numa_move_to_node' function. The 
documentation for the 'numa_tonode_memory' says that 'The  size 
argument  will  be rounded up to a multiple of the system page size'. 
However this does not work well with huge pages as alignment is 
performed for the default kernel page size (i.e. 4K in most cases). If 
addr + size value (rounded to the default page size) does not cover the 
entire huge page, then such invocation seems to be processed incorrectly 
and allocation policy is not applied for next pages access in such 
segment. At least this was the behavior I've observed on Debian 12 / 
6.1.40 kernel (i.e. '/proc/<pid>/numa_maps' shows that the segment 
contains pages from wrong nodes).

There are two location at which we could face such situation in current 
patches. First is related to buffers partitions mapping. With current 
code we basically ensure that combined size of all partitions for a 
single node is aligned to (huge) page size (as size is bound to the 
number of descriptors on one page), but individual partition is not 
explicitly aligned to this size. So, we could get the situation in which 
single page is split between adjacent partitions (e.g. 32GB buffers 
split by 3 nodes). With current code we will try to map each partition 
independently, which will results in unaligned calls to 
'numa_tonode_memory', so resulting mapping will be incorrect. We could 
either try to choose size for individual partition to align it to the 
desired page size or map all the partitions for a single node using a 
single 'pg_numa_move_to_node' invocation. During testing I've used the 
second approach, so here is the change to implement such logic: 
https://github.com/Lerm/postgres/commit/ee8b3603afd6d89e67b755dadc8e4c25ffba88be.

The second location which could expose the same problem is related to 
the mapping of PGPROC arrays in 'pgproc_partition_init': here we need to 
align pointer to the end PGPROC partition. There seems to be also two 
additional problems with PGPROC partitioning: we need to account 
additional padding pages in 'PGProcShmemSize' (using the same logic as 
with fplocks) and we should not call 'MemSet(ptr, 0, ...)' prior to 
partitions mapping call (otherwise it will be mapped to current node). 
Here is a potential change, which tries to address these problems: 
https://github.com/Lerm/postgres/commit/eaf12776f59ff150735d0f187595fc8ce3f0a872.

There are also some potential problems with buffers distribution between 
nodes. I have a feeling that current logic in 
'buffer_partitions_prepare' does not work correctly if number of buffers 
is enough to cover just a single partition per node, but total number of 
nodes is below MIN_BUFFER_PARTITIONS (i.e. 2 or 3). In this case we will 
set 'numa_can_partition' to 'true', but will allocate 2 partitions per 
node (so, 4 or 6 partitions in total), while we can fill just 2 or 3 
partition and leaving remaining partitions empty. This should violate 
the last assert check, as last partition will get zero buffers in this 
case. Another issue is related to the usage of 1GB pages, as minimal 
size for buffers partitions is limited by the minimal number of buffer 
descriptors in a single page. For 2MB pages this gives 2097152 / 64 * 8K 
= 256M as minimal size for partition, but for 1GB page the minimal size 
is equal to 1GB / 64 * 8K  = 128GB. So, if we assume 4 as minimal number 
of partitions, then for 2MB pages we need just 1GB for shared_buffers to 
enable partitioning (which seems a perfectly fine minimal limit for most 
cases), but for 1GB pages we need to allocate at least 512GB to allow 
buffers partitioning. Certainly, 1GB pages are usually used on large 
machines with large number of buffers allocated, but still it may be 
useful to allow configurations with 32GB or 64GB buffer cache to use 
both 1GB pages and buffers partitioning at the same time. However, I 
don't see an easy way to achieve this with the current logic. We either 
need to allow usage of different page sizes here (i.e. 2MB for 
descriptors and 1GB for buffers) or combine both buffers and its 
descriptors in a single object (i.e. 'buffer chunk', which cover enough 
buffers and their descriptors to fit into one or several memory pages), 
effectively replacing both buffers and descriptors arrays with an array 
of such 'chunks'. The latter solution may also help with dynamic buffer 
cache resizing (as we may just add additional 'chunks' in this case) and 
also increase TLB-hits with 1GB page (as both descriptor and its buffer 
will be likely located in the same page). However, both these changes 
seems to be quite large.

I've tried also to run some benchmarks on my server: I've got some 
improvements in 'pgbench/tpcb-like'results - about 8%, but only with 
backends pinning to NUMA node (i.e. adjusting your previous pinning 
patch to 'debug_numa' GUC: 
https://github.com/Lerm/postgres/commit/5942a3e12c7c501aa9febb63972a039e7ce00c20). 
For 'select-only' scenario the gain is more substantial (about 15%), but 
these tests are tricky, as they are more sensitive to other server 
settings and specific functions layout in compiled code, so they need 
more checks.

Thank you again for sharing these patches!

Thanks,
Alexey

  64. Re: Adding basic NUMA awareness

    Tomas Vondra <tomas@vondra.me> — 2025-10-13T11:09:20Z

    On 10/13/25 01:58, Alexey Makhmutov wrote:
> Hi Tomas,
> 
> Thank you very much for working on this problem and the entire line of
> patches prepared! I've tried to play with these patches a little and
> here are some my observations and suggestions.
> 
> In the current implementation we try to use all available NUMA nodes on
> the machine, however it's often useful to limit the database only to a
> set of specific nodes, so that other nodes can be used for other
> processes. In my testing I was trying to use one node out of four for
> the client program, so I'd liked to limit the database to the remaining
> nodes. I use a systemd service with AllowedMemoryNodes/AllowedCPUs to
> start the cluster, so the obvious choice for me was to use the
> 'numa_get_membind' function instead of 'numa_num_configured_nodes' to
> get the list of usable nodes. However, it is much easier to work with
> logical nodes in the [0; n] range inside the PG code, so I've decided to
> add mapping between 'logical nodes' (0-n in PG) to a set of physical
> nodes actually returned by 'numa_get_membind'. We may need to map number
> in both directions, so two translation tables are allocated and filled
> at the first usage of 'pg_numa' functions. It also seems to be a good
> idea to isolate all 'libnuma' calls inside 'pg_numa.c', so to keep all
> 'numa_...' calls in it and this also allows us to hide this mapping in
> static functions. Here is the patch, which I've used to test this idea:
> https://github.com/Lerm/postgres/
> commit/9ec625c2bf564f5432375ec1d7ad02e4b2559161. This idea probably
> could be extended by adding some view to expose this mapping to the user
> (at least for testing purposes) and allow to explicitly override this
> mapping with a GUC setting. With such GUC setting we would be able to
> control PG memory usage on NUMA nodes without the need for systemd
> resource control or numactl parameters.
> 

I've argued to keep this out of scope for v1, to keep it smaller and
simpler. I'm not against adding that feature, though. If someone writes
a patch to support this. I suppose the commit you linked is a step in
that direction.

I agree we should isolate libnuma calls to pg_numa.{c,h}. I wasn't quite
consistent when doing that.

> Next, I've noticed some problems related to the size alignment for
> 'numa_tonode_memory' call in 'pg_numa_move_to_node' function. The
> documentation for the 'numa_tonode_memory' says that 'The  size
> argument  will  be rounded up to a multiple of the system page size'.
> However this does not work well with huge pages as alignment is
> performed for the default kernel page size (i.e. 4K in most cases). If
> addr + size value (rounded to the default page size) does not cover the
> entire huge page, then such invocation seems to be processed incorrectly
> and allocation policy is not applied for next pages access in such
> segment. At least this was the behavior I've observed on Debian 12 /
> 6.1.40 kernel (i.e. '/proc/<pid>/numa_maps' shows that the segment
> contains pages from wrong nodes).
> 

I'm not sure I understand. Are you suggesting there's a bug in the
patch, the kernel, or somewhere else?

There's definitely a possibility of confusion with huge pages, no doubt
about that. The default "system page size" is 4KB, but we need to
process whole huge pages.

But this is exactly why (with hugepages) the code aligns everything to
huge page boundary, and sizes everything as a multiple of huge page. At
least I think so. Maybe I remember wrong?

> There are two location at which we could face such situation in current
> patches. First is related to buffers partitions mapping. With current
> code we basically ensure that combined size of all partitions for a
> single node is aligned to (huge) page size (as size is bound to the
> number of descriptors on one page), but individual partition is not
> explicitly aligned to this size. So, we could get the situation in which
> single page is split between adjacent partitions (e.g. 32GB buffers
> split by 3 nodes). With current code we will try to map each partition
> independently, which will results in unaligned calls to
> 'numa_tonode_memory', so resulting mapping will be incorrect. We could
> either try to choose size for individual partition to align it to the
> desired page size or map all the partitions for a single node using a
> single 'pg_numa_move_to_node' invocation. During testing I've used the
> second approach, so here is the change to implement such logic: https://
> github.com/Lerm/postgres/commit/ee8b3603afd6d89e67b755dadc8e4c25ffba88be.
> 

Can you actually demonstrate this? The code does these two things:

* calculate min_node_buffers so that buffers/descriptors are a multiple
  of page size (either 4K or huge page)

* align buffers and descriptors to memory page

  TYPEALIGN(buffer_align, ...)

I believe this is sufficient to ensure nothing gets split / mapped
incorrectly. Maybe this fails sometimes?

> The second location which could expose the same problem is related to
> the mapping of PGPROC arrays in 'pgproc_partition_init': here we need to
> align pointer to the end PGPROC partition. There seems to be also two
> additional problems with PGPROC partitioning: we need to account
> additional padding pages in 'PGProcShmemSize' (using the same logic as
> with fplocks) and we should not call 'MemSet(ptr, 0, ...)' prior to
> partitions mapping call (otherwise it will be mapped to current node).
> Here is a potential change, which tries to address these problems:
> https://github.com/Lerm/postgres/commit/
> eaf12776f59ff150735d0f187595fc8ce3f0a872.
> 

So you're saying pgproc_partition_init() should not do just this

    ptr = (char *) ptr + num_procs * sizeof(PGPROC);

but align the pointer to numa_page_size too? Sounds reasonable.

Yeah, PGProcShmemSize() should have added huge pages for each partition,
just like FastPathLockShmemSize(). Seems like a bug.

I don't think the memset() is a problem. Yes, it might map it to the
current node, but so what - the numa_tonode_memory() will just move it
to the correct one.

> There are also some potential problems with buffers distribution between
> nodes. I have a feeling that current logic in
> 'buffer_partitions_prepare' does not work correctly if number of buffers
> is enough to cover just a single partition per node, but total number of
> nodes is below MIN_BUFFER_PARTITIONS (i.e. 2 or 3). In this case we will
> set 'numa_can_partition' to 'true', but will allocate 2 partitions per
> node (so, 4 or 6 partitions in total), while we can fill just 2 or 3
> partition and leaving remaining partitions empty. This should violate
> the last assert check, as last partition will get zero buffers in this
> case. Another issue is related to the usage of 1GB pages, as minimal
> size for buffers partitions is limited by the minimal number of buffer
> descriptors in a single page. For 2MB pages this gives 2097152 / 64 * 8K
> = 256M as minimal size for partition, but for 1GB page the minimal size
> is equal to 1GB / 64 * 8K  = 128GB. So, if we assume 4 as minimal number
> of partitions, then for 2MB pages we need just 1GB for shared_buffers to
> enable partitioning (which seems a perfectly fine minimal limit for most
> cases), but for 1GB pages we need to allocate at least 512GB to allow
> buffers partitioning. Certainly, 1GB pages are usually used on large
> machines with large number of buffers allocated, but still it may be
> useful to allow configurations with 32GB or 64GB buffer cache to use
> both 1GB pages and buffers partitioning at the same time. However, I
> don't see an easy way to achieve this with the current logic. We either
> need to allow usage of different page sizes here (i.e. 2MB for
> descriptors and 1GB for buffers) or combine both buffers and its
> descriptors in a single object (i.e. 'buffer chunk', which cover enough
> buffers and their descriptors to fit into one or several memory pages),
> effectively replacing both buffers and descriptors arrays with an array
> of such 'chunks'. The latter solution may also help with dynamic buffer
> cache resizing (as we may just add additional 'chunks' in this case) and
> also increase TLB-hits with 1GB page (as both descriptor and its buffer
> will be likely located in the same page). However, both these changes
> seems to be quite large.
> 

I'll look at handling the case with shared_buffers being too small to
allow partitioning. There well might be a bug. We should simply disable
partitioning in such cases.

As for 1GB huge pages, I don't see a good way to support configurations
with small buffers in these cases. To me it seems acceptable to say that
if you want 1GB huge pages, you should have a lot of memory and shared
buffers large enough.

I'm not against supporting such systems, if we can come up with a good
partitioning scheme. When I tried to come up with a scheme like that, it
always came with a substantial complexity & cost. The main challenge was
that it forced splitting the array of buffer descriptors, similarly to
what the PGPROC partitioning does. And that made buffer access so much
more complex / expensive it seemed not worth it. I was worried about
impact on systems without NUMA partitioning.


> I've tried also to run some benchmarks on my server: I've got some
> improvements in 'pgbench/tpcb-like'results - about 8%, but only with
> backends pinning to NUMA node (i.e. adjusting your previous pinning
> patch to 'debug_numa' GUC: https://github.com/Lerm/postgres/
> commit/5942a3e12c7c501aa9febb63972a039e7ce00c20). For 'select-only'
> scenario the gain is more substantial (about 15%), but these tests are
> tricky, as they are more sensitive to other server settings and specific
> functions layout in compiled code, so they need more checks.
> 

What kind of hardware was that? What/how many cpus, NUMA nodes, how much
memory, what storage?

FWIW the main purpose of these patches was not so much throughput
improvement, but making the behavior more stable / consistent.


regards

-- 
Tomas Vondra

  65. Re: Adding basic NUMA awareness

    Alexey Makhmutov <a.makhmutov@postgrespro.ru> — 2025-10-13T18:34:39Z

    On 10/13/25 14:09, Tomas Vondra wrote:

 > I'm not sure I understand. Are you suggesting there's a bug in the 
patch, the kernel, or somewhere else?

We need to ensure that both addr and (addr + size) are aligned to the 
page size of the target mapping during 'numa_tonode_memory' invocation, 
otherwise it may produce unexpected results.

 > But this is exactly why (with hugepages) the code aligns everything 
to huge page boundary, and sizes everything as a multiple of huge page. 
At least I think so. Maybe I remember wrong?

I assume that there are places in the current patch, which could perform 
such unaligned mapping. See below for samples.

 > Can you actually demonstrate this?

This issue is related to the calculation of partition size for buffer 
descriptors in case we have multiple partitions per node. Currently we 
ensure that each node gets number of buffers, which fits into whole 
memory pages, but if we have several partitions per node, then there is 
no guarantee that partition size will be properly aligned for 
descriptors. We could observe this problem only if we have multiple 
partitions per node and with MIN_BUFFER_PARTITIONS equal to 4, this 
issue can potentially affect only configurations with 2 or 3 nodes.

Two examples here: first, let's assume we want to have shared_buffers 
set to 32GB with 3 NUMA nodes and 2MB pages. The NBuffers will be 
4,194,304,  min_node_buffers will be 32,768 and num_partitions_per_node 
will be 2 (so, 6 partitions in total). NBuffers/min_node_buffers = 128, 
so the nearest multiplier for min_node_buffers which allow us to cover 
all buffers with 3 nodes is 43 (42*3 = 126, 43*3 = 129). The 
num_buffers_per_node is 43*min_node_buffers and it is aligned to page 
size, but we need to split it between two partitions, so each gets 
41.5*min_node_buffers buffers. This still allow us to split buffers 
itself by page boundary, but descriptor partitions will be split just in 
the middle of the page. Here is the log for such configuration:
NUMA: buffers 4194304 partitions 6 num_nodes 3 per_node 2 
buffers_per_node 1409024 (min 32768)
NUMA: buffer 0 node 0 partition 0 buffers 704512 first 0 last 704511
NUMA: buffer 1 node 0 partition 1 buffers 704512 first 704512 last 1409023
NUMA: buffer 2 node 1 partition 0 buffers 704512 first 1409024 last 2113535
NUMA: buffer 3 node 1 partition 1 buffers 704512 first 2113536 last 2818047
NUMA: buffer 4 node 2 partition 0 buffers 688128 first 2818048 last 3506175
NUMA: buffer 5 node 2 partition 1 buffers 688128 first 3506176 last 4194303
NUMA: buffer_partitions_init: 0 => 0 buffers 704512 start 0x7ff7c8c00000 
end 0x7ff920c00000 (size 5771362304)
NUMA: buffer_partitions_init: 0 => 0 descriptors 704512 start 
0x7ff7b8a00000 end 0x7ff7bb500000 (size 45088768)
mbind: Invalid argument
NUMA: buffer_partitions_init: 1 => 0 buffers 704512 start 0x7ff920c00000 
end 0x7ffa78c00000 (size 5771362304)
NUMA: buffer_partitions_init: 1 => 0 descriptors 704512 start 
0x7ff7bb500000 end 0x7ff7be000000 (size 45088768)
mbind: Invalid argument
NUMA: buffer_partitions_init: 2 => 1 buffers 704512 start 0x7ffa78c00000 
end 0x7ffbd0c00000 (size 5771362304)
NUMA: buffer_partitions_init: 2 => 1 descriptors 704512 start 
0x7ff7be000000 end 0x7ff7c0b00000 (size 45088768)
mbind: Invalid argument
NUMA: buffer_partitions_init: 3 => 1 buffers 704512 start 0x7ffbd0c00000 
end 0x7ffd28c00000 (size 5771362304)
NUMA: buffer_partitions_init: 3 => 1 descriptors 704512 start 
0x7ff7c0b00000 end 0x7ff7c3600000 (size 45088768)
mbind: Invalid argument
NUMA: buffer_partitions_init: 4 => 2 buffers 688128 start 0x7ffd28c00000 
end 0x7ffe78c00000 (size 5637144576)
NUMA: buffer_partitions_init: 4 => 2 descriptors 688128 start 
0x7ff7c3600000 end 0x7ff7c6000000 (size 44040192)
NUMA: buffer_partitions_init: 5 => 2 buffers 688128 start 0x7ffe78c00000 
end 0x7fffc8c00000 (size 5637144576)
NUMA: buffer_partitions_init: 5 => 2 descriptors 688128 start 
0x7ff7c6000000 end 0x7ff7c8a00000 (size 44040192)

Another example: 2 nodes and 15872MB shared_buffers. Again, 
NBuffers/min_node_buffers=62, so num_buffers_per_node is 
31*min_node_buffers, which gives each partition 15.5*min_node_buffers. 
Here is the log output:
NUMA: buffers 2031616 partitions 4 num_nodes 2 per_node 2 
buffers_per_node 1015808 (min 32768)
NUMA: buffer 0 node 0 partition 0 buffers 507904 first 0 last 507903
NUMA: buffer 1 node 0 partition 1 buffers 507904 first 507904 last 1015807
NUMA: buffer 2 node 1 partition 0 buffers 507904 first 1015808 last 1523711
NUMA: buffer 3 node 1 partition 1 buffers 507904 first 1523712 last 2031615
NUMA: buffer_partitions_init: 0 => 0 buffers 507904 start 0x7ffbf9c00000 
end 0x7ffcf1c00000 (size 4160749568)
NUMA: buffer_partitions_init: 0 => 0 descriptors 507904 start 
0x7ffbf1e00000 end 0x7ffbf3d00000 (size 32505856)
mbind: Invalid argument
NUMA: buffer_partitions_init: 1 => 0 buffers 507904 start 0x7ffcf1c00000 
end 0x7ffde9c00000 (size 4160749568)
NUMA: buffer_partitions_init: 1 => 0 descriptors 507904 start 
0x7ffbf3d00000 end 0x7ffbf5c00000 (size 32505856)
mbind: Invalid argument
NUMA: buffer_partitions_init: 2 => 1 buffers 507904 start 0x7ffde9c00000 
end 0x7ffee1c00000 (size 4160749568)
NUMA: buffer_partitions_init: 2 => 1 descriptors 507904 start 
0x7ffbf5c00000 end 0x7ffbf7b00000 (size 32505856)
mbind: Invalid argument
NUMA: buffer_partitions_init: 3 => 1 buffers 507904 start 0x7ffee1c00000 
end 0x7fffd9c00000 (size 4160749568)
NUMA: buffer_partitions_init: 3 => 1 descriptors 507904 start 
0x7ffbf7b00000 end 0x7ffbf9a00000 (size 32505856)
mbind: Invalid argument

 > So you're saying pgproc_partition_init() should not do just this
 > ptr = (char *) ptr + num_procs * sizeof(PGPROC);
 > but align the pointer to numa_page_size too? Sounds reasonable.

Yes, that's exactly my point, otherwise we could violate the alignment 
rule for 'numa_tonode_memory'. Here is an extraction from the log for 
system with 2 nodes, 2000 max_connections and 2MB pages:
NUMA: pgproc backends 2056 num_nodes 2 per_node 1028
NUMA: pgproc_init_partition procs 0x7fffe7800000 endptr 0x7fffe78d2d20 
num_procs 1028 node 0
mbind: Invalid argument
NUMA: pgproc_init_partition procs 0x7fffe7a00000 endptr 0x7fffe7ad2d20 
num_procs 1028 node 1
mbind: Invalid argument
NUMA: pgproc_init_partition procs 0x7fffe7c00000 endptr 0x7fffe7c07cb0 
num_procs 38 node -1
mbind: Invalid argument
mbind: Invalid argument

 > I don't think the memset() is a problem. Yes, it might map it to the 
current node, but so what - the numa_tonode_memory() will just move it 
to the correct one.

Well, the 'numa_tonode_memory' call does not move pages to the target 
node. It just sets the policy for mapping, so system will actually try 
to provide page from the correct node once we touch it. However, if the 
page is already faulted, then it won't be affected by this mapping, so 
that's why it works faster compared to 'numa_move_pages'. As stated in 
libnuma documentation:
* numa_tonode_memory() put memory on a specific node. The constraints 
described for numa_interleave_memory() apply here too.
* numa_interleave_memory()  interleaves  size  bytes of memory page by 
page from start on nodes specified in nodemask. <...> This is a lower 
level function to interleave allocated but not yet faulted in memory. 
Not yet faulted in means the memory is allocated using mmap(2) or 
shmat(2), but has not been accessed by  the current process yet. <...> 
If the numa_set_strict() flag is true then the operation will cause a 
numa_error if there were already pages in the mapping that do not follow 
the policy.

I assume, that for the regular page kernel may rebalance memory in the 
future (not immediately), but not for hugepages. So, we really don't 
want to touch the memory area before we call the 'numa_tonode_memory'.

This can be easily tested with the simple program:
#include <stdio.h>
#include <numa.h>
#include <sys/mman.h>
#include <linux/mman.h>

#define MAP_SIZE 2*1024*1024

int main(int argc, char** argv) {
   void* ptr1 = mmap(NULL, MAP_SIZE, PROT_READ | PROT_WRITE, MAP_SHARED 
| MAP_ANONYMOUS | MAP_HUGETLB | MAP_HUGE_2MB, -1, 0);
   void* ptr2 = mmap(NULL, MAP_SIZE, PROT_READ | PROT_WRITE, MAP_SHARED 
| MAP_ANONYMOUS | MAP_HUGETLB | MAP_HUGE_2MB, -1, 0);

   /* Fault first page */
   memset(ptr1, 1, MAP_SIZE);
   /* Move to node 1 */
   numa_tonode_memory(ptr1, MAP_SIZE, 1);
   numa_tonode_memory(ptr2, MAP_SIZE, 1);
   /* Fault second page */
   memset(ptr2, 1, MAP_SIZE);

   /* Wait */
   printf("ptr1=%lx\nptr2=\%lx\nPress Enter to continue...\n",ptr1,ptr2);
   getchar();
   munmap(ptr2, MAP_SIZE);
   munmap(ptr1, MAP_SIZE);
   return 0;
}

Running it on the first node:
# gcc -o test_mem test_mem.c -lnuma
# taskset -c 0 ./test_mem
ptr1=7ffff7a00000
ptr2=7ffff7800000
Press Enter to continue...

 From another terminal:
# grep huge /proc/`pgrep test_mem`/numa_maps
7ffff7800000 bind:1 file=/anon_hugepage\040(deleted) huge dirty=1 N1=1 
kernelpagesize_kB=2048
7ffff7a00000 bind:1 file=/anon_hugepage\040(deleted) huge dirty=1 N0=1 
kernelpagesize_kB=2048

So, while policy (bind:1) is set for both mappings, but only the second 
one (which was not touched before the 'numa_tonode_memory' invocation) 
is actualy located on node 1 rather than 0.

 > What kind of hardware was that? What/how many cpus, NUMA nodes, how 
much memory, what storage?

Of course, that's valid question. I probably should not have commented 
on performance side without providing full data, while I was still 
trying to measure it and it was just preliminary runs. Sorry for that.

Thanks,
Alexey

  66. Re: Adding basic NUMA awareness

    Tomas Vondra <tomas@vondra.me> — 2025-10-15T17:02:38Z

    Hi,

Here's an updated patch series, addressing (some) of the issues. I've
kept the changes in separate patches, to make the changes easier to
review and discuss.

On 10/13/25 20:34, Alexey Makhmutov wrote:
> On 10/13/25 14:09, Tomas Vondra wrote:
> 
>> I'm not sure I understand. Are you suggesting there's a bug in the
> patch, the kernel, or somewhere else?
> 
> We need to ensure that both addr and (addr + size) are aligned to the
> page size of the target mapping during 'numa_tonode_memory' invocation,
> otherwise it may produce unexpected results.
> 

Hmm. The libnuma docs about numa_intereave_memory says:

    ... The size argument will be rounded up to a multiple of the system
    page size. ...

Which I interpreted that it does all the necessary rounding. But if this
ignores huge pages (i.e. "system page size" is 4K, not a HP size), then
aligning the size explicitly is would be needed.

This would be pretty annoying, though. It'd mean we can't rely on any
rounding done by libnuma, at least for code that might use huge pages.

Nevertheless, the updated patches should address both cases ...

>> But this is exactly why (with hugepages) the code aligns everything to
> huge page boundary, and sizes everything as a multiple of huge page. At
> least I think so. Maybe I remember wrong?
> 
> I assume that there are places in the current patch, which could perform
> such unaligned mapping. See below for samples.
> 
>> Can you actually demonstrate this?
> 
> This issue is related to the calculation of partition size for buffer
> descriptors in case we have multiple partitions per node. Currently we
> ensure that each node gets number of buffers, which fits into whole
> memory pages, but if we have several partitions per node, then there is
> no guarantee that partition size will be properly aligned for
> descriptors. We could observe this problem only if we have multiple
> partitions per node and with MIN_BUFFER_PARTITIONS equal to 4, this
> issue can potentially affect only configurations with 2 or 3 nodes.
> 
> Two examples here: first, let's assume we want to have shared_buffers
> set to 32GB with 3 NUMA nodes and 2MB pages. The NBuffers will be
> 4,194,304,  min_node_buffers will be 32,768 and num_partitions_per_node
> will be 2 (so, 6 partitions in total). NBuffers/min_node_buffers = 128,
> so the nearest multiplier for min_node_buffers which allow us to cover
> all buffers with 3 nodes is 43 (42*3 = 126, 43*3 = 129). The
> num_buffers_per_node is 43*min_node_buffers and it is aligned to page
> size, but we need to split it between two partitions, so each gets
> 41.5*min_node_buffers buffers. This still allow us to split buffers
> itself by page boundary, but descriptor partitions will be split just in
> the middle of the page. Here is the log for such configuration:
> NUMA: buffers 4194304 partitions 6 num_nodes 3 per_node 2
> buffers_per_node 1409024 (min 32768)
> ...

I see. I was really puzzled how could a node get chunk of buffers that's
not a multiple of page size, because min_node_buffers was meant to
guarantee that. But now I realize it's not about per-node buffers, it's
about individual partitions.

Initially I thought the right way to fix this is to use min_node_buffers
for each partitions, not for nodes. But that would increase the amount
of memory needed for NUMA partitioning to work. I practice that wouldn't
be an issue, because it'd still be only ~1GB (with 2MB huge pages), and
the relevant systems will have way more.

But then I realized it's we don't need to map the partitions one by one.
We can simply map all partitions for the whole NUMA node at once, and
then we don't have this problem at all.

The attached 0007 patch does this to fix the issue. And I just noticed
this is pretty much exactly how you fixed this in your commit ee8b360.

The last partition may still not have the size aligned, though, because
may not be a multiple of min_node_buffers.

> 
> Another example: 2 nodes and 15872MB shared_buffers. Again, NBuffers/
> min_node_buffers=62, so num_buffers_per_node is 31*min_node_buffers,
> which gives each partition 15.5*min_node_buffers. Here is the log output:
> NUMA: buffers 2031616 partitions 4 num_nodes 2 per_node 2
> buffers_per_node 1015808 (min 32768)
> ...
> mbind: Invalid argument
> NUMA: buffer_partitions_init: 3 => 1 buffers 507904 start 0x7ffee1c00000
> end 0x7fffd9c00000 (size 4160749568)
> NUMA: buffer_partitions_init: 3 => 1 descriptors 507904 start
> 0x7ffbf7b00000 end 0x7ffbf9a00000 (size 32505856)
> mbind: Invalid argument
> 
>> So you're saying pgproc_partition_init() should not do just this
>> ptr = (char *) ptr + num_procs * sizeof(PGPROC);
>> but align the pointer to numa_page_size too? Sounds reasonable.
> 
> Yes, that's exactly my point, otherwise we could violate the alignment
> rule for 'numa_tonode_memory'. Here is an extraction from the log for
> system with 2 nodes, 2000 max_connections and 2MB pages:

Should be fixed by 0010 by explicitly aligning the size like this. It's
a bit more extensive than your eaf1277.

BTW what's the mbind failures about? Is that something we check, at
least in memory

> 
>> I don't think the memset() is a problem. Yes, it might map it to the
> current node, but so what - the numa_tonode_memory() will just move it
> to the correct one.
> 
> Well, the 'numa_tonode_memory' call does not move pages to the target
> node. It just sets the policy for mapping, so system will actually try
> to provide page from the correct node once we touch it. However, if the
> page is already faulted, then it won't be affected by this mapping, so
> that's why it works faster compared to 'numa_move_pages'. As stated in
> libnuma documentation:
> * numa_tonode_memory() put memory on a specific node. The constraints
> described for numa_interleave_memory() apply here too.
> * numa_interleave_memory()  interleaves  size  bytes of memory page by
> page from start on nodes specified in nodemask. <...> This is a lower
> level function to interleave allocated but not yet faulted in memory.
> Not yet faulted in means the memory is allocated using mmap(2) or
> shmat(2), but has not been accessed by  the current process yet. <...>
> If the numa_set_strict() flag is true then the operation will cause a
> numa_error if there were already pages in the mapping that do not follow
> the policy.
> 

Point taken. The 0009 fixes this by moving the MemSet() to after the
partitioning. At that point the policy is already set.

There's a couple more fixes. 0008 improves handling of cases that don't
allow NUMA partitioning (like when shared_buffers are too small). 0011
adds the missing padding to PGProcShmemSize, which you also fixed in one
of your commits.

0012 reduces logging in clock-sweep balancing, which on idle systems was
annoyingly verbose.

I keps 0006 separate for now. It got broken by 5e89985928, and the
conflicts were fairly extensive. Better keep it separate a bit longer.


regards

-- 
Tomas Vondra

  67. Re: Adding basic NUMA awareness

    Tomas Vondra <tomas@vondra.me> — 2025-10-15T17:15:47Z

    
On 10/13/25 13:09, Tomas Vondra wrote:
> On 10/13/25 01:58, Alexey Makhmutov wrote:
>> Hi Tomas,
>>
>> Thank you very much for working on this problem and the entire line of
>> patches prepared! I've tried to play with these patches a little and
>> here are some my observations and suggestions.
>>
>> In the current implementation we try to use all available NUMA nodes on
>> the machine, however it's often useful to limit the database only to a
>> set of specific nodes, so that other nodes can be used for other
>> processes. In my testing I was trying to use one node out of four for
>> the client program, so I'd liked to limit the database to the remaining
>> nodes. I use a systemd service with AllowedMemoryNodes/AllowedCPUs to
>> start the cluster, so the obvious choice for me was to use the
>> 'numa_get_membind' function instead of 'numa_num_configured_nodes' to
>> get the list of usable nodes. However, it is much easier to work with
>> logical nodes in the [0; n] range inside the PG code, so I've decided to
>> add mapping between 'logical nodes' (0-n in PG) to a set of physical
>> nodes actually returned by 'numa_get_membind'. We may need to map number
>> in both directions, so two translation tables are allocated and filled
>> at the first usage of 'pg_numa' functions. It also seems to be a good
>> idea to isolate all 'libnuma' calls inside 'pg_numa.c', so to keep all
>> 'numa_...' calls in it and this also allows us to hide this mapping in
>> static functions. Here is the patch, which I've used to test this idea:
>> https://github.com/Lerm/postgres/
>> commit/9ec625c2bf564f5432375ec1d7ad02e4b2559161. This idea probably
>> could be extended by adding some view to expose this mapping to the user
>> (at least for testing purposes) and allow to explicitly override this
>> mapping with a GUC setting. With such GUC setting we would be able to
>> control PG memory usage on NUMA nodes without the need for systemd
>> resource control or numactl parameters.
>>
> 
> I've argued to keep this out of scope for v1, to keep it smaller and
> simpler. I'm not against adding that feature, though. If someone writes
> a patch to support this. I suppose the commit you linked is a step in
> that direction.
> 
On second thought, I probably spoke too soon ...

What I wanted to keep out of scope for v1 is ability to pick NUMA nodes
from Postgres, e.g. setting a GUC to limit which NUMA nodes to use, etc.

But that's not what you proposed here, clearly. You're saying we should
find which NUMA nodes the process is allowed to run, and use those.
Instead of just using all *configured* nodes. And I agree with that.

I'll take a look at your commit 9ec625c. I'm not sure it's a good idea
to have our internal "logical" node ID, and a mapping to external node
ID values (exposed by the libnuma). I was thinking maybe we should use
just the external IDs, but it's true that'd be tricky when iterating
through nodes, etc. So maybe having such mapping is a good approach.

Another thing I wasn't sure about is checking for memory-only nodes. For
example rpi5 has a NUMA node for each 1GB of memory, and each CPU is
mapped to all those nodes. For buffers this probably does not matter,
but we probably should not use those NUMA nodes for PGPROC partitioning.


regards

-- 
Tomas Vondra

  68. Re: Adding basic NUMA awareness

    Tomas Vondra <tomas@vondra.me> — 2025-10-31T11:57:33Z

    Hi,

here's a significantly reworked version of this patch series.

I had a couple discussions about these patches at pgconf.eu last week,
and one interesting suggestion was that maybe it'd be easier to the
clock-sweep partitioning first, in a NUMA-oblivious way. And then add
the NUMA stuff later.

The logic is that this way we could ignore some of the hard stuff (e.g.
handling huge page reservation), while still reducing clocksweep
contention. Which we speculated might be the main benefit anyway.

The attached patches do this.

0001 - Introduces a simplified version of the "buffer partition
registry" (think array in shmem, storing info about ranges of shared
buffer). The partitions are calculated as simple fraction of shared
buffers. There's no need to align the partitions to memory pages etc.

0002-0005 - Does the clock-sweep partitioning. I chose to keep this
split into smaller increments, to keep the patches easier to review.

0006 - Make the partitioning NUMA-aware. This used to be part of 0001,
but now it's moved on top of the clock-sweep stuff. It ensures the
partitions are properly aligned to memory pages, and all that.

0007 - PGPROC partitioning.

This made the 0001 patch much simpler/smaller - it used to be ~50kB, now
it's 15kB (and most of the complexity is in 0006).

The question however is how this performs, or how much of the benefit
was due to NUMA-awareness and how much was due to just partitioning
clock-sweep. I repeated the benchmark from [1], doing concurrent
sequential scans to put significant pressure on buffer replacements, and
I got this:

    hp   clients |  master  | sweep   sweep-16  |  numa  numa-16
    =============|==========|===================|===============
    off       16 |      24  |    46         46  |    33       40
              32 |      33  |    53         51  |    45       51
              48 |      38  |    51         61  |    46       56
              64 |      41  |    56         75  |    47       65
              80 |      47  |    53         77  |    48       71
              96 |      45  |    54         80  |    47       66
             112 |      45  |    52         83  |    44       65
             128 |      43  |    55         81  |    39       48
    -------------|----------|-------------------|---------------
    on        16 |      26  |    47         47  |    35       42
              32 |      33  |    49         52  |    40       49
              48 |      39  |    52         63  |    43       57
              64 |      42  |    53         72  |    43       66
              80 |      43  |    54         81  |    46       71
              96 |      48  |    58         80  |    49       73
             112 |      51  |    58         78  |    51       76
             128 |      55  |    60         83  |    52       76

"hp" means huge pages, the compared branches are:

- master - current master
- sweep - patches up to 0005, default number of partitions (4)
- sweep-16 - patches up to 0005, 16 partitions
- numa - patches up to 0006, default number of partitions (4)
- numa-16 - patches up to 0006, 16 partitions

Compared to master, the results look like this:

    hp   clients  |  sweep   sweep-16  |  numa   numa-16
    ==============|====================|================
    off       16  |   192%       192%  |  138%      167%
              32  |   161%       155%  |  136%      155%
              48  |   132%       160%  |  121%      145%
              64  |   137%       183%  |  115%      159%
              80  |   113%       164%  |  102%      151%
              96  |   120%       177%  |  104%      146%
             112  |   116%       184%  |   98%      144%
             128  |   128%       186%  |   90%      110%
    --------------|--------------------|----------------
    on        16  |   181%       181%  |  135%      162%
              32  |   148%       158%  |  121%      148%
              48  |   133%       161%  |  110%      144%
              64  |   126%       171%  |  102%      157%
              80  |   126%       188%  |  107%      165%
              96  |   121%       167%  |  102%      152%
             112  |   114%       153%  |  100%      149%
             128  |   109%       151%  |   95%      138%

The attached PDF has more results for runs with somewhat modified
parameters, but the overall it's very similar to these numbers.

I think this confirms most of the benefit really comes from just
partitioning clock-sweep, and it's mostly independent of the NUMA stuff.
In fact, the NUMA partitioning is often slower. Some of this may be due
to inefficiencies in the patch (e.g. division in formula calculating the
partition index, etc.).

So I think this looks quite promising ...

There are a couple unsolved issues, though. While running the tests, I
ran into a bunch of weird issues. I saw two types of failures:


1) Bad address
-----------------------------------------------------------------------
2025-10-30 15:24:21.195 UTC [2038558] LOG:  could not read blocks
114543..114558 in file "base/16384/16588": Bad address
2025-10-30 15:24:21.195 UTC [2038558] STATEMENT:  SELECT * FROM t_41
OFFSET 1000000000

2025-10-30 15:24:21.195 UTC [2038523] LOG:  could not read blocks
119981..119996 in file "base/16384/16869": Bad address
2025-10-30 15:24:21.195 UTC [2038523] CONTEXT:  completing I/O on behalf
of process 2038464
2025-10-30 15:24:21.195 UTC [2038523] STATEMENT:  SELECT * FROM t_96
OFFSET 1000000000

2025-10-30 15:24:21.195 UTC [2038492] LOG:  could not read blocks
118226..118232 in file "base/16384/16478": Bad address
2025-10-30 15:24:21.195 UTC [2038492] STATEMENT:  SELECT * FROM t_19
OFFSET 1000000000

2025-10-30 15:24:21.196 UTC [2038477] LOG:  could not read blocks
120515..120517 in file "base/16384/16945": Bad address
2025-10-30 15:24:21.196 UTC [2038477] CONTEXT:  completing I/O on behalf
of process 2038545
2025-10-30 15:24:21.196 UTC [2038477] STATEMENT:  SELECT * FROM t_111
OFFSET 1000000000
-----------------------------------------------------------------------

2) Operation canceled
-----------------------------------------------------------------------
2025-10-31 10:57:21.742 UTC [2685933] LOG:  could not read blocks
159..174 in file "base/16384/16398": Operation canceled
2025-10-31 10:57:21.742 UTC [2685933] STATEMENT:  SELECT * FROM t_3
OFFSET 1000000000

2025-10-31 10:57:21.742 UTC [2685933] LOG:  could not read blocks
143..158 in file "base/16384/16398": Operation canceled
2025-10-31 10:57:21.742 UTC [2685933] STATEMENT:  SELECT * FROM t_3
OFFSET 1000000000

2025-10-31 10:57:21.781 UTC [2685933] ERROR:  could not read blocks
143..158 in file "base/16384/16398": Operation canceled
2025-10-31 10:57:21.781 UTC [2685933] STATEMENT:  SELECT * FROM t_3
OFFSET 1000000000
-----------------------------------------------------------------------


I'm still not sure what's causing these, and it's happening rarely and
randomly, so it's hard to catch and reproduce. I'd welcome suggestions
what to look for / what might be the issue.

I did run the whole test under valgrind to make sure there's nothing
obviously broken, but that found no issues. Of course, it's much slower
under valgrind, so maybe it just didn't hit the issue.

I suspect the "bad address" might be just a different symptom of the
issues with reserving huge pages I already mentioned [2]. I assume
io_uring might try using huge pages internally, and then it fails
because postgres also reserves huge pages.

I have no idea what "operation canceled" might be about.

I'm not entirely sure if this affect all patches, or just the patches
with NUMA partitioning. Or if this happens with huge pages. I'll do more
runs to test specifically this.

But it does seem to be specific to io_uring - or at least the canceled
issue. I haven't seen it after switching to "worker".



[1]
https://www.postgresql.org/message-id/51e51832-7f47-412a-a1a6-b972101cc8cb%40vondra.me

[2]
https://www.postgresql.org/message-id/1d57d68d-b178-415a-ba11-be0c3714638e%40vondra.me

regards

-- 
Tomas Vondra

  69. Re: Adding basic NUMA awareness

    Jakub Wartak <jakub.wartak@enterprisedb.com> — 2025-11-04T12:10:58Z

    On Fri, Oct 31, 2025 at 12:57 PM Tomas Vondra <tomas@vondra.me> wrote:
>
> Hi,
>
> here's a significantly reworked version of this patch series.
>
> I had a couple discussions about these patches at pgconf.eu last week,[..]

I've just had a quick look at this and oh, my, I've started getting
into this partitioned clocksweep and that's ambitious! Yes, this
sequencing of patches makes it much more understandable. Anyway I've
spotted some things, attempted to fix some and have some basic
questions too (so small baby steps, all of this was on 4s/4 NUMA nodes
with HP on) -- the 000X refers to question/issue/bug in specific
patchset file:

0001: you mention 'debug_numa = buffers' in commitmsg, but there's
nothing there like that? it comes with 0006

0002: dunno, but wouldn't it make some educational/debugging sense to
add a debug function returning clocksweep partition index
(calculate_partition_index) for backend? (so that we know which
partition we are working on right now)

0003: those two "elog(INFO, "rebalance skipped:" should be at DEBUG2+
IMHO (they are way too verbose during runs)

0006a: Needs update - s/patches later in the patch series/patches
earlier in the patch series/

0006b: IMHO longer term, we should hide some complexity of those calls
via src/port numa shims (pg_numa_sched_cpu()?)

0006c: after GUC commit fce7c73fba4e5, apply complains with:
error: patch failed: src/backend/utils/misc/guc_parameters.dat:906
error: src/backend/utils/misc/guc_parameters.dat: patch does not apply

0007a: pg_buffercache_pgproc returns pgproc_ptr and fastpath_ptr in
bigint and not hex? I've wanted to adjust that to TEXTOID, but instead
I've thought it is going to be simpler to use to_hex() -- see 0009
attached.

0007b: pg_buffercache_pgproc -- nitpick, but maybe it would be better
called pg_shm_pgproc?

0007c with check_numa='buffers,procs' throws 'mbind Invalid argument'
during start:

    2025-11-04 10:02:27.055 CET [58464] DEBUG:  NUMA:
pgproc_init_partition procs 0x7f8d30400000 endptr 0x7f8d30800000
num_procs 2523 node 0
    2025-11-04 10:02:27.057 CET [58464] DEBUG:  NUMA:
pgproc_init_partition procs 0x7f8d30800000 endptr 0x7f8d30c00000
num_procs 2523 node 1
    2025-11-04 10:02:27.059 CET [58464] DEBUG:  NUMA:
pgproc_init_partition procs 0x7f8d30c00000 endptr 0x7f8d31000000
num_procs 2523 node 2
    2025-11-04 10:02:27.061 CET [58464] DEBUG:  NUMA:
pgproc_init_partition procs 0x7f8d31000000 endptr 0x7f8d31400000
num_procs 2523 node 3
    2025-11-04 10:02:27.062 CET [58464] DEBUG:  NUMA:
pgproc_init_partition procs 0x7f8d31400000 endptr 0x7f8d31407cb0
num_procs 38 node -1
    mbind: Invalid argument
    mbind: Invalid argument
    mbind: Invalid argument
    mbind: Invalid argument

0007d: so we probably need numa_warn()/numa_error() wrappers (this was
initially part of NUMA observability patches but got removed during
the course of action), I'm attaching 0008. With that you'll get
something a little more up to our standards:
    2025-11-04 10:27:07.140 CET [59696] DEBUG:
fastpath_parititon_init node = 3, ptr = 0x7f4f4d400000, endptr =
0x7f4f4d4b1660
    2025-11-04 10:27:07.140 CET [59696] WARNING:  libnuma: ERROR: mbind

0007e: elog DEBUG says it's pg_proc_init_partition but it's
pgproc_partition_init() actually ;)

0007f: The "mbind: Invalid argument"" issue itself with the below  addition:
    +elog(DEBUG1, "NUMA: fastpath_partition_init ptr %p endptr %p
num_procs %d node %d", ptr, endptr, num_procs, node);
    showed this:
    2025-11-04 11:30:51.089 CET [61841] DEBUG:  NUMA:
fastpath_partition_init ptr 0x7f39eea00000 endptr 0x7f39eeab1660
num_procs 2523 node 0
    2025-11-04 11:30:51.089 CET [61841] WARNING:  libnuma: ERROR: mbind
    2025-11-04 11:30:51.089 CET [61841] DEBUG:  NUMA:
fastpath_partition_init ptr 0x7f39eec00000 endptr 0x7f39eecb1660
num_procs 2523 node 1
    2025-11-04 11:30:51.089 CET [61841] WARNING:  libnuma: ERROR: mbind
    2025-11-04 11:30:51.089 CET [61841] DEBUG:  NUMA:
fastpath_partition_init ptr 0x7f39eee00000 endptr 0x7f39eeeb1660
num_procs 2523 node 2
    2025-11-04 11:30:51.089 CET [61841] WARNING:  libnuma: ERROR: mbind
    [..]

    Meanwhile it's full hugepage size (e.g. 0x7f39eec00000−0x7f39eea00000 = 2MB)
    $ grep --color 7f39ee[ace] /proc/61841/smaps
    7f39ee800000-7f39eea00000 rw-s 87de00000 00:11 122710
      /anon_hugepage (deleted)
    7f39eea00000-7f39eec00000 rw-s 87e000000 00:11 122710
      /anon_hugepage (deleted)
    7f39eec00000-7f39eee00000 rw-s 87e200000 00:11 122710
      /anon_hugepage (deleted)
    7f39eee00000-7f39ef000000 rw-s 87e400000 00:11 122710
      /anon_hugepage (deleted)

    but mbind() was called for just 0x7f39eeab1660−0x7f39eea00000 =
0xB1660 = 726624 bytes, but if adjust blindly endptr in that
fastpath_partition_init() to be "char *endptr = ptr + 2*1024*1024;"
(HP) it doesn't complain anymore and I get success:
    2025-11-04 12:08:30.147 CET [62352] DEBUG:  NUMA:
fastpath_partition_init ptr 0x7f7bf7000000 endptr 0x7f7bf7200000
num_procs 2523 node 0
    2025-11-04 12:08:30.147 CET [62352] DEBUG:  NUMA:
fastpath_partition_init ptr 0x7f7bf7200000 endptr 0x7f7bf7400000
num_procs 2523 node 1
    2025-11-04 12:08:30.147 CET [62352] DEBUG:  NUMA:
fastpath_partition_init ptr 0x7f7bf7400000 endptr 0x7f7bf7600000
num_procs 2523 node 2
    2025-11-04 12:08:30.147 CET [62352] DEBUG:  NUMA:
fastpath_partition_init ptr 0x7f7bf7600000 endptr 0x7f7bf7800000
num_procs 2523 node 3
    2025-11-04 12:08:30.147 CET [62352] DEBUG:  NUMA:
fastpath_partition_init ptr 0x7f7bf7800000 endptr 0x7f7bf7a00000
num_procs 38 node -1
    2025-11-04 12:08:30.239 CET [62352] LOG:  starting PostgreSQL
19devel on x86_64-linux, compiled by gcc-12.2.0, 64-bit

0006d: I've got one SIGBUS during a call to select
pg_buffercache_numa_pages(); and it looks like that memory accessed is
simply not mapped? (bug)

    Program received signal SIGBUS, Bus error.
    pg_buffercache_numa_pages (fcinfo=0x561a97e8e680) at
../contrib/pg_buffercache/pg_buffercache_pages.c:386
    386                                     pg_numa_touch_mem_if_required(ptr);
    (gdb) print ptr
    $1 = 0x7f4ed0200000 <error: Cannot access memory at address 0x7f4ed0200000>
    (gdb) where
    #0  pg_buffercache_numa_pages (fcinfo=0x561a97e8e680) at
../contrib/pg_buffercache/pg_buffercache_pages.c:386
    #1  0x0000561a672a0efe in ExecMakeFunctionResultSet
(fcache=0x561a97e8e5d0, econtext=econtext@entry=0x561a97e8dab8,
argContext=0x561a97ec62a0, isNull=0x561a97e8e578,
isDone=isDone@entry=0x561a97e8e5c0) at
../src/backend/executor/execSRF.c:624
    [..]

    Postmaster had still attached shm (visible via smaps), and if you
compare closely 0x7f4ed0200000 against sorted smaps:

    7f4921400000-7f4b21400000 rw-s 252600000 00:11 151111
      /anon_hugepage (deleted)
    7f4b21400000-7f4d21400000 rw-s 452600000 00:11 151111
      /anon_hugepage (deleted)
    7f4d21400000-7f4f21400000 rw-s 652600000 00:11 151111
      /anon_hugepage (deleted)
    7f4f21400000-7f4f4bc00000 rw-s 852600000 00:11 151111
      /anon_hugepage (deleted)
    7f4f4bc00000-7f4f4c000000 rw-s 87ce00000 00:11 151111
      /anon_hugepage (deleted)

    it's NOT there at all (there's no mmap region starting with
0x"7f4e" ). It looks like because pg_buffercache_numa_pages() is not
aware of this new mmaped() regions and instead does simple loop over
all NBuffers with "for (char *ptr = startptr; ptr < endptr; ptr +=
os_page_size)"?

0006e:
    I'm seeking confirmation, but is this the issue we have discussed
on PgconfEU related to lack of detection of Mems_allowed, right? e.g.
    $ numactl --membind="0,1" --cpunodebind="0,1"
/usr/pgsql19/bin/pg_ctl -D /path start
    still shows 4 NUMA nodes used. Current patches use
numa_num_configured_nodes(), but it says 'This count includes any
nodes that are currently DISABLED'. So I was wondering if I could help
by migrating towards numa_num_task_nodes() / numa_get_mems_allowed()?
It's the same as You wrote earlier to Alexy?

    > But that's not what you proposed here, clearly. You're saying we should
    > find which NUMA nodes the process is allowed to run, and use those.
    > Instead of just using all *configured* nodes. And I agree with that.

    So are you already on it ?

> There are a couple unsolved issues, though. While running the tests, I
> ran into a bunch of weird issues. I saw two types of failures:
> 1) Bad address
> 2) Operation canceled

I did run (with io_uring) a short test(< 10min with -c 128) and didn't
get those. Could you please share specific tips/workload for
reproducing this?

That's all for today, I hope it helps a little.

-J.

  70. Re: Adding basic NUMA awareness

    Tomas Vondra <tomas@vondra.me> — 2025-11-04T21:21:24Z

    
On 11/4/25 13:10, Jakub Wartak wrote:
> On Fri, Oct 31, 2025 at 12:57 PM Tomas Vondra <tomas@vondra.me> wrote:
>>
>> Hi,
>>
>> here's a significantly reworked version of this patch series.
>>
>> I had a couple discussions about these patches at pgconf.eu last week,[..]
> 
> I've just had a quick look at this and oh, my, I've started getting
> into this partitioned clocksweep and that's ambitious! Yes, this
> sequencing of patches makes it much more understandable. Anyway I've
> spotted some things, attempted to fix some and have some basic
> questions too (so small baby steps, all of this was on 4s/4 NUMA nodes
> with HP on) -- the 000X refers to question/issue/bug in specific
> patchset file:
> 
> 0001: you mention 'debug_numa = buffers' in commitmsg, but there's
> nothing there like that? it comes with 0006
> 

Right, I forgot to remove that reference.

> 0002: dunno, but wouldn't it make some educational/debugging sense to
> add a debug function returning clocksweep partition index
> (calculate_partition_index) for backend? (so that we know which
> partition we are working on right now)
> 

Perhaps. I didn't need that, but it might be interesting during
development. I probably would not keep that in the final version.

> 0003: those two "elog(INFO, "rebalance skipped:" should be at DEBUG2+
> IMHO (they are way too verbose during runs)
> 

Agreed.

> 0006a: Needs update - s/patches later in the patch series/patches
> earlier in the patch series/
> 

Agreed.

> 0006b: IMHO longer term, we should hide some complexity of those calls
> via src/port numa shims (pg_numa_sched_cpu()?)
> 

Yeah, there's definitely room for moving more of the code to src/port.

> 0006c: after GUC commit fce7c73fba4e5, apply complains with:
> error: patch failed: src/backend/utils/misc/guc_parameters.dat:906
> error: src/backend/utils/misc/guc_parameters.dat: patch does not apply
> 

Will fix.

> 0007a: pg_buffercache_pgproc returns pgproc_ptr and fastpath_ptr in
> bigint and not hex? I've wanted to adjust that to TEXTOID, but instead
> I've thought it is going to be simpler to use to_hex() -- see 0009
> attached.
> 

I don't know. I added simply because it might be useful for development,
but we probably don't want to expose these pointers at all.

> 0007b: pg_buffercache_pgproc -- nitpick, but maybe it would be better
> called pg_shm_pgproc?
> 

Right. It does not belong to pg_buffercache at all, I just added it
there because I've been messing with that code already.

> 0007c with check_numa='buffers,procs' throws 'mbind Invalid argument'
> during start:
> 
>     2025-11-04 10:02:27.055 CET [58464] DEBUG:  NUMA:
> pgproc_init_partition procs 0x7f8d30400000 endptr 0x7f8d30800000
> num_procs 2523 node 0
>     2025-11-04 10:02:27.057 CET [58464] DEBUG:  NUMA:
> pgproc_init_partition procs 0x7f8d30800000 endptr 0x7f8d30c00000
> num_procs 2523 node 1
>     2025-11-04 10:02:27.059 CET [58464] DEBUG:  NUMA:
> pgproc_init_partition procs 0x7f8d30c00000 endptr 0x7f8d31000000
> num_procs 2523 node 2
>     2025-11-04 10:02:27.061 CET [58464] DEBUG:  NUMA:
> pgproc_init_partition procs 0x7f8d31000000 endptr 0x7f8d31400000
> num_procs 2523 node 3
>     2025-11-04 10:02:27.062 CET [58464] DEBUG:  NUMA:
> pgproc_init_partition procs 0x7f8d31400000 endptr 0x7f8d31407cb0
> num_procs 38 node -1
>     mbind: Invalid argument
>     mbind: Invalid argument
>     mbind: Invalid argument
>     mbind: Invalid argument
> 

I'll take a look, but I don't recall seeing such errors.

> 0007d: so we probably need numa_warn()/numa_error() wrappers (this was
> initially part of NUMA observability patches but got removed during
> the course of action), I'm attaching 0008. With that you'll get
> something a little more up to our standards:
>     2025-11-04 10:27:07.140 CET [59696] DEBUG:
> fastpath_parititon_init node = 3, ptr = 0x7f4f4d400000, endptr =
> 0x7f4f4d4b1660
>     2025-11-04 10:27:07.140 CET [59696] WARNING:  libnuma: ERROR: mbind
> 

Not sure.

> 0007e: elog DEBUG says it's pg_proc_init_partition but it's
> pgproc_partition_init() actually ;)
> 
> 0007f: The "mbind: Invalid argument"" issue itself with the below  addition:
>     +elog(DEBUG1, "NUMA: fastpath_partition_init ptr %p endptr %p
> num_procs %d node %d", ptr, endptr, num_procs, node);
>     showed this:
>     2025-11-04 11:30:51.089 CET [61841] DEBUG:  NUMA:
> fastpath_partition_init ptr 0x7f39eea00000 endptr 0x7f39eeab1660
> num_procs 2523 node 0
>     2025-11-04 11:30:51.089 CET [61841] WARNING:  libnuma: ERROR: mbind
>     2025-11-04 11:30:51.089 CET [61841] DEBUG:  NUMA:
> fastpath_partition_init ptr 0x7f39eec00000 endptr 0x7f39eecb1660
> num_procs 2523 node 1
>     2025-11-04 11:30:51.089 CET [61841] WARNING:  libnuma: ERROR: mbind
>     2025-11-04 11:30:51.089 CET [61841] DEBUG:  NUMA:
> fastpath_partition_init ptr 0x7f39eee00000 endptr 0x7f39eeeb1660
> num_procs 2523 node 2
>     2025-11-04 11:30:51.089 CET [61841] WARNING:  libnuma: ERROR: mbind
>     [..]
> 
>     Meanwhile it's full hugepage size (e.g. 0x7f39eec00000−0x7f39eea00000 = 2MB)
>     $ grep --color 7f39ee[ace] /proc/61841/smaps
>     7f39ee800000-7f39eea00000 rw-s 87de00000 00:11 122710
>       /anon_hugepage (deleted)
>     7f39eea00000-7f39eec00000 rw-s 87e000000 00:11 122710
>       /anon_hugepage (deleted)
>     7f39eec00000-7f39eee00000 rw-s 87e200000 00:11 122710
>       /anon_hugepage (deleted)
>     7f39eee00000-7f39ef000000 rw-s 87e400000 00:11 122710
>       /anon_hugepage (deleted)
> 
>     but mbind() was called for just 0x7f39eeab1660−0x7f39eea00000 =
> 0xB1660 = 726624 bytes, but if adjust blindly endptr in that
> fastpath_partition_init() to be "char *endptr = ptr + 2*1024*1024;"
> (HP) it doesn't complain anymore and I get success:
>     2025-11-04 12:08:30.147 CET [62352] DEBUG:  NUMA:
> fastpath_partition_init ptr 0x7f7bf7000000 endptr 0x7f7bf7200000
> num_procs 2523 node 0
>     2025-11-04 12:08:30.147 CET [62352] DEBUG:  NUMA:
> fastpath_partition_init ptr 0x7f7bf7200000 endptr 0x7f7bf7400000
> num_procs 2523 node 1
>     2025-11-04 12:08:30.147 CET [62352] DEBUG:  NUMA:
> fastpath_partition_init ptr 0x7f7bf7400000 endptr 0x7f7bf7600000
> num_procs 2523 node 2
>     2025-11-04 12:08:30.147 CET [62352] DEBUG:  NUMA:
> fastpath_partition_init ptr 0x7f7bf7600000 endptr 0x7f7bf7800000
> num_procs 2523 node 3
>     2025-11-04 12:08:30.147 CET [62352] DEBUG:  NUMA:
> fastpath_partition_init ptr 0x7f7bf7800000 endptr 0x7f7bf7a00000
> num_procs 38 node -1
>     2025-11-04 12:08:30.239 CET [62352] LOG:  starting PostgreSQL
> 19devel on x86_64-linux, compiled by gcc-12.2.0, 64-bit
> 

Hmm, so it seems like another hugepage-related issue. The mbind manpage
says this about "len":

  EINVAL An invalid value was specified for flags or mode; or addr + len
  was less than addr; or addr is not a multiple of the system page size.

I don't think that requires (addr+len) to be a multiple of page size,
but maybe that is required.

> 0006d: I've got one SIGBUS during a call to select
> pg_buffercache_numa_pages(); and it looks like that memory accessed is
> simply not mapped? (bug)
> 
>     Program received signal SIGBUS, Bus error.
>     pg_buffercache_numa_pages (fcinfo=0x561a97e8e680) at
> ../contrib/pg_buffercache/pg_buffercache_pages.c:386
>     386                                     pg_numa_touch_mem_if_required(ptr);
>     (gdb) print ptr
>     $1 = 0x7f4ed0200000 <error: Cannot access memory at address 0x7f4ed0200000>
>     (gdb) where
>     #0  pg_buffercache_numa_pages (fcinfo=0x561a97e8e680) at
> ../contrib/pg_buffercache/pg_buffercache_pages.c:386
>     #1  0x0000561a672a0efe in ExecMakeFunctionResultSet
> (fcache=0x561a97e8e5d0, econtext=econtext@entry=0x561a97e8dab8,
> argContext=0x561a97ec62a0, isNull=0x561a97e8e578,
> isDone=isDone@entry=0x561a97e8e5c0) at
> ../src/backend/executor/execSRF.c:624
>     [..]
> 
>     Postmaster had still attached shm (visible via smaps), and if you
> compare closely 0x7f4ed0200000 against sorted smaps:
> 
>     7f4921400000-7f4b21400000 rw-s 252600000 00:11 151111
>       /anon_hugepage (deleted)
>     7f4b21400000-7f4d21400000 rw-s 452600000 00:11 151111
>       /anon_hugepage (deleted)
>     7f4d21400000-7f4f21400000 rw-s 652600000 00:11 151111
>       /anon_hugepage (deleted)
>     7f4f21400000-7f4f4bc00000 rw-s 852600000 00:11 151111
>       /anon_hugepage (deleted)
>     7f4f4bc00000-7f4f4c000000 rw-s 87ce00000 00:11 151111
>       /anon_hugepage (deleted)
> 
>     it's NOT there at all (there's no mmap region starting with
> 0x"7f4e" ). It looks like because pg_buffercache_numa_pages() is not
> aware of this new mmaped() regions and instead does simple loop over
> all NBuffers with "for (char *ptr = startptr; ptr < endptr; ptr +=
> os_page_size)"?
> 

I'm confused. How could that mapping be missing? Was this with huge
pages / how many did you reserve on the nodes? Maybe there were not
enough huge pages left on one of the nodes?

I believe I got some SIGBUS in those cases.

> 0006e:
>     I'm seeking confirmation, but is this the issue we have discussed
> on PgconfEU related to lack of detection of Mems_allowed, right? e.g.
>     $ numactl --membind="0,1" --cpunodebind="0,1"
> /usr/pgsql19/bin/pg_ctl -D /path start
>     still shows 4 NUMA nodes used. Current patches use
> numa_num_configured_nodes(), but it says 'This count includes any
> nodes that are currently DISABLED'. So I was wondering if I could help
> by migrating towards numa_num_task_nodes() / numa_get_mems_allowed()?
> It's the same as You wrote earlier to Alexy?
> 

If "mems_allowed" refers to nodes allowing memory allocation, then yes,
this would be one way to get into that issue. Oh, is this what happened
in 0006d?

>     > But that's not what you proposed here, clearly. You're saying we should
>     > find which NUMA nodes the process is allowed to run, and use those.
>     > Instead of just using all *configured* nodes. And I agree with that.
> 
>     So are you already on it ?
> 
>> There are a couple unsolved issues, though. While running the tests, I
>> ran into a bunch of weird issues. I saw two types of failures:
>> 1) Bad address
>> 2) Operation canceled
> 
> I did run (with io_uring) a short test(< 10min with -c 128) and didn't
> get those. Could you please share specific tips/workload for
> reproducing this?
> 

I did get a couple of "operation canceled" failures, but only on fairly
old kernel versions (6.1 which came as default with the VM). I heard
some suggestions this is a bug in older kernels - I don't have any link
to a bug report / fix, though. But I've been unable to reproduce this on
6.17, so maybe it's true.

For me the failures always happened 10 seconds after the start of the
benchmark (and starting the instance), so it's probably sufficient to
keep the runs ~20 seconds (and maybe restart in between?).

But even then it's fairly rare. I've seen ~10 failures for 500 runs.


I haven't seen more "bad address" cases, I have no idea why. I'm still
guessing it's related to huge pages, so maybe I happened to reserve
enough of them.

regards

-- 
Tomas Vondra

  71. Re: Adding basic NUMA awareness

    Jakub Wartak <jakub.wartak@enterprisedb.com> — 2025-11-06T14:02:57Z

    On Tue, Nov 4, 2025 at 10:21 PM Tomas Vondra <tomas@vondra.me> wrote:

Hi Tomas,

> > 0007a: pg_buffercache_pgproc returns pgproc_ptr and fastpath_ptr in
> > bigint and not hex? I've wanted to adjust that to TEXTOID, but instead
> > I've thought it is going to be simpler to use to_hex() -- see 0009
> > attached.
> >
>
> I don't know. I added simply because it might be useful for development,
> but we probably don't want to expose these pointers at all.
>
> > 0007b: pg_buffercache_pgproc -- nitpick, but maybe it would be better
> > called pg_shm_pgproc?
> >
>
> Right. It does not belong to pg_buffercache at all, I just added it
> there because I've been messing with that code already.

Please keep them in for at least for some time (perhaps standalone
patch marked as not intended to be commited would work?). I find the
view extermely useful as it will allow us pinpointing local-vs-remote
NUMA fetches (we need to know the addres).

> > 0007c with check_numa='buffers,procs' throws 'mbind Invalid argument'
> > during start:
> >
> >     2025-11-04 10:02:27.055 CET [58464] DEBUG:  NUMA:
> > pgproc_init_partition procs 0x7f8d30400000 endptr 0x7f8d30800000
> > num_procs 2523 node 0
> >     2025-11-04 10:02:27.057 CET [58464] DEBUG:  NUMA:
> > pgproc_init_partition procs 0x7f8d30800000 endptr 0x7f8d30c00000
> > num_procs 2523 node 1
> >     2025-11-04 10:02:27.059 CET [58464] DEBUG:  NUMA:
> > pgproc_init_partition procs 0x7f8d30c00000 endptr 0x7f8d31000000
> > num_procs 2523 node 2
> >     2025-11-04 10:02:27.061 CET [58464] DEBUG:  NUMA:
> > pgproc_init_partition procs 0x7f8d31000000 endptr 0x7f8d31400000
> > num_procs 2523 node 3
> >     2025-11-04 10:02:27.062 CET [58464] DEBUG:  NUMA:
> > pgproc_init_partition procs 0x7f8d31400000 endptr 0x7f8d31407cb0
> > num_procs 38 node -1
> >     mbind: Invalid argument
> >     mbind: Invalid argument
> >     mbind: Invalid argument
> >     mbind: Invalid argument
> >
>
> I'll take a look, but I don't recall seeing such errors.
>

Alexy also reported this earlier, here
https://www.postgresql.org/message-id/92e23c85-f646-4bab-b5e0-df30d8ddf4bd%40postgrespro.ru
(just use HP, set some high max_connections). I've double checked this
too , numa_tonode_memory() len needs to HP size.

> > 0007d: so we probably need numa_warn()/numa_error() wrappers (this was
> > initially part of NUMA observability patches but got removed during
> > the course of action), I'm attaching 0008. With that you'll get
> > something a little more up to our standards:
> >     2025-11-04 10:27:07.140 CET [59696] DEBUG:
> > fastpath_parititon_init node = 3, ptr = 0x7f4f4d400000, endptr =
> > 0x7f4f4d4b1660
> >     2025-11-04 10:27:07.140 CET [59696] WARNING:  libnuma: ERROR: mbind
> >
>
> Not sure.

Any particular objections? We need to somehow emit them into the logs.

> > 0007f: The "mbind: Invalid argument"" issue itself with the below  addition:
[..]
> >
> >     but mbind() was called for just 0x7f39eeab1660−0x7f39eea00000 =
> > 0xB1660 = 726624 bytes, but if adjust blindly endptr in that
> > fastpath_partition_init() to be "char *endptr = ptr + 2*1024*1024;"
> > (HP) it doesn't complain anymore and I get success:
[..]
>
> Hmm, so it seems like another hugepage-related issue. The mbind manpage
> says this about "len":
>
>   EINVAL An invalid value was specified for flags or mode; or addr + len
>   was less than addr; or addr is not a multiple of the system page size.
>
> I don't think that requires (addr+len) to be a multiple of page size,
> but maybe that is required.

I do think that 'system page size' means above HP page size, but this
time it's just for fastpath_partition_init(), the earlier one seems to
aligned fine (?? -- i havent really checked but there's no error)

> > 0006d: I've got one SIGBUS during a call to select
> > pg_buffercache_numa_pages(); and it looks like that memory accessed is
> > simply not mapped? (bug)
> >
> >     Program received signal SIGBUS, Bus error.
> >     pg_buffercache_numa_pages (fcinfo=0x561a97e8e680) at
> > ../contrib/pg_buffercache/pg_buffercache_pages.c:386
> >     386                                     pg_numa_touch_mem_if_required(ptr);
> >     (gdb) print ptr
> >     $1 = 0x7f4ed0200000 <error: Cannot access memory at address 0x7f4ed0200000>
> >     (gdb) where
> >     #0  pg_buffercache_numa_pages (fcinfo=0x561a97e8e680) at
> > ../contrib/pg_buffercache/pg_buffercache_pages.c:386
> >     #1  0x0000561a672a0efe in ExecMakeFunctionResultSet
> > (fcache=0x561a97e8e5d0, econtext=econtext@entry=0x561a97e8dab8,
> > argContext=0x561a97ec62a0, isNull=0x561a97e8e578,
> > isDone=isDone@entry=0x561a97e8e5c0) at
> > ../src/backend/executor/execSRF.c:624
> >     [..]
> >
> >     Postmaster had still attached shm (visible via smaps), and if you
> > compare closely 0x7f4ed0200000 against sorted smaps:
> >
> >     7f4921400000-7f4b21400000 rw-s 252600000 00:11 151111
> >       /anon_hugepage (deleted)
> >     7f4b21400000-7f4d21400000 rw-s 452600000 00:11 151111
> >       /anon_hugepage (deleted)
> >     7f4d21400000-7f4f21400000 rw-s 652600000 00:11 151111
> >       /anon_hugepage (deleted)
> >     7f4f21400000-7f4f4bc00000 rw-s 852600000 00:11 151111
> >       /anon_hugepage (deleted)
> >     7f4f4bc00000-7f4f4c000000 rw-s 87ce00000 00:11 151111
> >       /anon_hugepage (deleted)
> >
> >     it's NOT there at all (there's no mmap region starting with
> > 0x"7f4e" ). It looks like because pg_buffercache_numa_pages() is not
> > aware of this new mmaped() regions and instead does simple loop over
> > all NBuffers with "for (char *ptr = startptr; ptr < endptr; ptr +=
> > os_page_size)"?
> >
>
> I'm confused. How could that mapping be missing? Was this with huge
> pages / how many did you reserve on the nodes?


OK I made and error and paritally got it correct (it crashes reliably)
and partially mislead You, appologies, let me explain. There were two
questions for me:
a) why we make single mmap() and after numa_tonode_memory() we get
plenty of mappings
b) why we get SIGBUS (I've thought they are not continus, but they are
after triple-checking)

ad a) My testing shows that on HP,as stated initially ("all of this
was on 4s/4 NUMA nodes with HP on"). That's what the codes does, you
get single mmaps() (resulting in single entry in smaps), but afte
noda_tonode_memory() there's many of them. Even on laptop:

System has 1 NUMA nodes (0 to 0).
Attempting to allocate 8.000000 MB of HugeTLB memory...
Successfully allocated HugeTLB memory at 0x755828800000, smaps before:
755828800000-755829000000 rw-s 00000000 00:11 259808
  /anon_hugepage (deleted)
Pinning first part (from 0x755828800000) to NUMA node 0...
smaps after:
755828800000-755828c00000 rw-s 00000000 00:11 259808
  /anon_hugepage (deleted)
755828c00000-755829000000 rw-s 00400000 00:11 259808
  /anon_hugepage (deleted)
Pinning second part (from 0x755828c00000) to NUMA node 0...
smaps after:
755828800000-755828c00000 rw-s 00000000 00:11 259808
  /anon_hugepage (deleted)
755828c00000-755829000000 rw-s 00400000 00:11 259808
  /anon_hugepage (deleted)

It gets even more funny, below I have 8MB HP=on, but just issue 2x
numa_tonode_memory(for len 2MB on 4MB ptr to node0) (two times for
ptr, second time in half of that):

System has 1 NUMA nodes (0 to 0).
Attempting to allocate 8.000000 MB of HugeTLB memory...
Successfully allocated HugeTLB memory at 0x7302dda00000, smaps before:
7302dda00000-7302de200000 rw-s 00000000 00:11 284859
  /anon_hugepage (deleted)
Pinning first part (from 0x7302dda00000) to NUMA node 0...
smaps after:
7302dda00000-7302ddc00000 rw-s 00000000 00:11 284859
  /anon_hugepage (deleted)
7302ddc00000-7302de200000 rw-s 00200000 00:11 284859
  /anon_hugepage (deleted)
Pinning second part (from 0x7302dde00000) to NUMA node 0...
smaps after:
7302dda00000-7302ddc00000 rw-s 00000000 00:11 284859
  /anon_hugepage (deleted)
7302ddc00000-7302dde00000 rw-s 00200000 00:11 284859
  /anon_hugepage (deleted)
7302dde00000-7302de000000 rw-s 00400000 00:11 284859
  /anon_hugepage (deleted)
7302de000000-7302de200000 rw-s 00600000 00:11 284859
  /anon_hugepage (deleted)

Why 4 instead of 1? Because some mappings are now "default" becauswe
their policy was not altered:

$ grep huge /proc/$(pidof testnumammapsplit)/numa_maps
7302dda00000 bind:0 file=/anon_hugepage\040(deleted) huge
7302ddc00000 default file=/anon_hugepage\040(deleted) huge
7302dde00000 bind:0 file=/anon_hugepage\040(deleted) huge
7302de000000 default file=/anon_hugepage\040(deleted) huge

Back to originnal error, they are consecutive regions and earlier problem is

error: 0x7f4ed0200000 <error: Cannot access memory at address 0x7f4ed0200000>
start: 0x7f4921400000
end:   0x7f4f4c000000

so it fits into that range (that was my mistate earlier, using just
grep not checking are they really within that), but...

> Maybe there were not enough huge pages left on one of the nodes?

ad b) right, something like that. I've investigated that SIGBUS there
(it's going to be long):

with shared_buffers=32GB, huge_pages 17715 (+1 from what postgres -C
shared_memory_size_in_huge_pages returns), right after startup, but no
touch:

Program received signal SIGBUS, Bus error.
pg_buffercache_numa_pages (fcinfo=0x5572038790b8) at
../contrib/pg_buffercache/pg_buffercache_pages.c:386
386                                     pg_numa_touch_mem_if_required(ptr);
(gdb) where
#0  pg_buffercache_numa_pages (fcinfo=0x5572038790b8) at
../contrib/pg_buffercache/pg_buffercache_pages.c:386
#1  0x00005571f54ddb7d in ExecMakeTableFunctionResult
(setexpr=0x557203870d40, econtext=0x557203870ba8,
argContext=<optimized out>, expectedDesc=0x557203870f80,
randomAccess=false) at ../src/backend/executor/execSRF.c:234
[..]
(gdb) print ptr
$1 = 0x7f6cf8400000 <error: Cannot access memory at address 0x7f6cf8400000>
(gdb)


then it shows?! no available hugepage on one of the nodes (while gdb
is hanging and preving autorestart):

root@swiatowid:/sys/devices/system/node# grep -r -i HugePages_Free node*/meminfo
node0/meminfo:Node 0 HugePages_Free:    299
node1/meminfo:Node 1 HugePages_Free:    299
node2/meminfo:Node 2 HugePages_Free:    299
node3/meminfo:Node 3 HugePages_Free:      0

but they are equal in terms of size:
node0/meminfo:Node 0 HugePages_Total:  4429
node1/meminfo:Node 1 HugePages_Total:  4429
node2/meminfo:Node 2 HugePages_Total:  4429
node3/meminfo:Node 3 HugePages_Total:  4428

smaps shows that this address (7f6cf8400000) is mapped in this mapping:
7f6b49c00000-7f6d49c00000 rw-s 652600000 00:11 86064
  /anon_hugepage (deleted)

numa_maps for this region shows this is this mapping on node3 (notice
N3 + bind:3 matches lack of memory on Node 3 HugePAges_Free):
7f6b49c00000 bind:3 file=/anon_hugepage\040(deleted) huge dirty=3444
N3=3444 kernelpagesize_kB=2048

the surrounding area of this looks like that:

7f6549c00000 bind:0 file=/anon_hugepage\040(deleted) huge dirty=4096
N0=4096 kernelpagesize_kB=2048
7f6749c00000 bind:1 file=/anon_hugepage\040(deleted) huge dirty=4096
N1=4096 kernelpagesize_kB=2048
7f6949c00000 bind:2 file=/anon_hugepage\040(deleted) huge dirty=4096
N2=4096 kernelpagesize_kB=2048
7f6b49c00000 bind:3 file=/anon_hugepage\040(deleted) huge dirty=3444
N3=3444 kernelpagesize_kB=2048 <-- this is the one
7f6d49c00000 default file=/anon_hugepage\040(deleted) huge dirty=107
mapmax=6 N3=107 kernelpagesize_kB=2048

Notice it's just N3=3444, while the others are much larger. So
something was using that hugepages memory on N3:

# grep kernelpagesize_kB=2048 /proc/1679/numa_maps | grep -Po
N[0-4]=[0-9]+ | sort
N0=2
N0=4096
N1=2
N1=4096
N2=2
N2=4096
N3=1
N3=1
N3=1
N3=1
N3=107
N3=13
N3=3
N3=3444

So per above it's not there (at least not as 2MB HP). But the number
of mappings is wild there! (node where it is failing has plenty of
memory, no hugepage memory left, but it has like 40k+ of small
mappings!)

# grep -Po 'N[0-3]=' /proc/1679/numa_maps | sort | uniq -c
     17 N0=
     10 N1=
      3 N2=
  40434 N3=

most of them are `anon_inode:[io_uring]` (and I had
max_connections=10k). You may ask why in spite of Andres optimization
for reducing number segments for uring, it's not working for me ? Well
I've just noticed way too silent failure to active this (altough I'm
on 6.14.x):
    2025-11-06 13:34:49.128 CET [1658] DEBUG:  can't use combined
memory mapping for io_uring, kernel or liburing too old
and I dont have io_uring_queue_init_mem()/HAVE_LIBURING_QUEUE_INIT_MEM
apparently on liburing-2.3 (Debian's default). See [1] for more info
(fix is not commited yet sadly).

Next try, now with io_method = worker and right before start:

root@swiatowid:/sys/devices/system/node# grep -r -i HugePages_Total
node*/meminfo
node0/meminfo:Node 0 HugePages_Total:  4429
node1/meminfo:Node 1 HugePages_Total:  4429
node2/meminfo:Node 2 HugePages_Total:  4429
node3/meminfo:Node 3 HugePages_Total:  4428
and HugePages_Free were 100% (if postgresql was down). After start
(but without doing anything else):
root@swiatowid:/sys/devices/system/node# grep -r -i HugePages_Free node*/meminfo
node0/meminfo:Node 0 HugePages_Free:   4393
node1/meminfo:Node 1 HugePages_Free:   4395
node2/meminfo:Node 2 HugePages_Free:   4395
node3/meminfo:Node 3 HugePages_Free:   3446

So sadly the picture is the same (something stole my HP on N3 and it's
PostgreSQL on it's own). After some time of investigating that ("who
stole my hugepage across whole OS"), I've just added MAP_POPULATE to
the mix of PG_MMAP_FLAGS and got this after start:

root@swiatowid:/sys/devices/system/node# grep -r -i HugePages_Free node*/meminfo
node0/meminfo:Node 0 HugePages_Free:      0
node1/meminfo:Node 1 HugePages_Free:      0
node2/meminfo:Node 2 HugePages_Free:      0
node3/meminfo:Node 3 HugePages_Free:      1

and then the SELECT to pg_buffercache_numa works fine(!).

Another ways that I have found to eliminate that SIGBUS
a. Would be to throw much more HugePages (so that node does not run to
HugePages_Free), but that's not real option.
b. Then I've reminded myself that I could be running custom kernel
with experimental CONFIG_READ_ONLY_THP_FOR_FS (to reduce iTLB misses
tranparently with specially linked PG; will double check exact stuff
later), so I've thrown never into
/sys/kernel/mm/transparent_hugepage/enabled and defrag too (yes ,
disabled THP) and with that -- drumroll -- that SELECT works. The very
same PG picture after startup (where earlier it would crash), now
after SELECT it looks like that:

root@swiatowid:/sys/devices/system/node# grep -r -i HugePages_Free node*/meminfo
node0/meminfo:Node 0 HugePages_Free:     83
node1/meminfo:Node 1 HugePages_Free:      0
node2/meminfo:Node 2 HugePages_Free:     81
node3/meminfo:Node 3 HugePages_Free:     82

Hope that helps a little. To me it sounds like THP used that memory
somehow and we've also wanted to use. With numa_interleave_ptr() that
wouldn't be a problem because probably it would something else
available, but not here as we indicated exact node.

> > 0006e:
> >     I'm seeking confirmation, but is this the issue we have discussed
> > on PgconfEU related to lack of detection of Mems_allowed, right? e.g.
> >     $ numactl --membind="0,1" --cpunodebind="0,1"
> > /usr/pgsql19/bin/pg_ctl -D /path start
> >     still shows 4 NUMA nodes used. Current patches use
> > numa_num_configured_nodes(), but it says 'This count includes any
> > nodes that are currently DISABLED'. So I was wondering if I could help
> > by migrating towards numa_num_task_nodes() / numa_get_mems_allowed()?
> > It's the same as You wrote earlier to Alexy?
> >
>
> If "mems_allowed" refers to nodes allowing memory allocation, then yes,
> this would be one way to get into that issue. Oh, is this what happened
> in 0006d?

OK, thanks for confirmation. No, 0006d was about normal numactl run,
without --membind.

> I did get a couple of "operation canceled" failures, but only on fairly
> old kernel versions (6.1 which came as default with the VM).

OK, I'll try to see that later too.

btw QQ regarding partitioned clockwise as I had thought: does this
opens a road towards multiple BGwriters? (outside of this
$thread/v1/PoC)

-J.

[1] - https://www.postgresql.org/message-id/CAKZiRmzxj6Lt1w2ffDoUmN533TgyDeYVULEH1PQFLRyBJSFP6w%40mail.gmail.com

  72. Re: Adding basic NUMA awareness

    Tomas Vondra <tomas@vondra.me> — 2025-11-11T11:52:01Z

    Hi,

here's a rebased patch series, fixing most of the smaller issues from
v20251101, and making cfbot happy (hopefully).

On 11/6/25 15:02, Jakub Wartak wrote:
> On Tue, Nov 4, 2025 at 10:21 PM Tomas Vondra <tomas@vondra.me> wrote:
> 
> Hi Tomas,
> 
>>> 0007a: pg_buffercache_pgproc returns pgproc_ptr and fastpath_ptr in
>>> bigint and not hex? I've wanted to adjust that to TEXTOID, but instead
>>> I've thought it is going to be simpler to use to_hex() -- see 0009
>>> attached.
>>>
>>
>> I don't know. I added simply because it might be useful for development,
>> but we probably don't want to expose these pointers at all.
>>
>>> 0007b: pg_buffercache_pgproc -- nitpick, but maybe it would be better
>>> called pg_shm_pgproc?
>>>
>>
>> Right. It does not belong to pg_buffercache at all, I just added it
>> there because I've been messing with that code already.
> 
> Please keep them in for at least for some time (perhaps standalone
> patch marked as not intended to be commited would work?). I find the
> view extermely useful as it will allow us pinpointing local-vs-remote
> NUMA fetches (we need to know the addres).
> 

Are you referring to the _pgproc view specifically, or also to the view
with buffer partitions? I don't intend to remove the view for shared
buffers, that's indeed useful.

>>> 0007c with check_numa='buffers,procs' throws 'mbind Invalid argument'
>>> during start:
>>>
>>>     2025-11-04 10:02:27.055 CET [58464] DEBUG:  NUMA:
>>> pgproc_init_partition procs 0x7f8d30400000 endptr 0x7f8d30800000
>>> num_procs 2523 node 0
>>>     2025-11-04 10:02:27.057 CET [58464] DEBUG:  NUMA:
>>> pgproc_init_partition procs 0x7f8d30800000 endptr 0x7f8d30c00000
>>> num_procs 2523 node 1
>>>     2025-11-04 10:02:27.059 CET [58464] DEBUG:  NUMA:
>>> pgproc_init_partition procs 0x7f8d30c00000 endptr 0x7f8d31000000
>>> num_procs 2523 node 2
>>>     2025-11-04 10:02:27.061 CET [58464] DEBUG:  NUMA:
>>> pgproc_init_partition procs 0x7f8d31000000 endptr 0x7f8d31400000
>>> num_procs 2523 node 3
>>>     2025-11-04 10:02:27.062 CET [58464] DEBUG:  NUMA:
>>> pgproc_init_partition procs 0x7f8d31400000 endptr 0x7f8d31407cb0
>>> num_procs 38 node -1
>>>     mbind: Invalid argument
>>>     mbind: Invalid argument
>>>     mbind: Invalid argument
>>>     mbind: Invalid argument
>>>
>>
>> I'll take a look, but I don't recall seeing such errors.
>>
> 
> Alexy also reported this earlier, here
> https://www.postgresql.org/message-id/92e23c85-f646-4bab-b5e0-df30d8ddf4bd%40postgrespro.ru
> (just use HP, set some high max_connections). I've double checked this
> too , numa_tonode_memory() len needs to HP size.
> 

OK, I'll investigate this.

>>> 0007d: so we probably need numa_warn()/numa_error() wrappers (this was
>>> initially part of NUMA observability patches but got removed during
>>> the course of action), I'm attaching 0008. With that you'll get
>>> something a little more up to our standards:
>>>     2025-11-04 10:27:07.140 CET [59696] DEBUG:
>>> fastpath_parititon_init node = 3, ptr = 0x7f4f4d400000, endptr =
>>> 0x7f4f4d4b1660
>>>     2025-11-04 10:27:07.140 CET [59696] WARNING:  libnuma: ERROR: mbind
>>>
>>
>> Not sure.
> 
> Any particular objections? We need to somehow emit them into the logs.
> 

No idea, I think it'd be better to make sure this failure can't happen,
but maybe it's not possible. I don't understand the mbind failure well
enough.

>>> 0007f: The "mbind: Invalid argument"" issue itself with the below  addition:
> [..]
>>>
>>>     but mbind() was called for just 0x7f39eeab1660−0x7f39eea00000 =
>>> 0xB1660 = 726624 bytes, but if adjust blindly endptr in that
>>> fastpath_partition_init() to be "char *endptr = ptr + 2*1024*1024;"
>>> (HP) it doesn't complain anymore and I get success:
> [..]
>>
>> Hmm, so it seems like another hugepage-related issue. The mbind manpage
>> says this about "len":
>>
>>   EINVAL An invalid value was specified for flags or mode; or addr + len
>>   was less than addr; or addr is not a multiple of the system page size.
>>
>> I don't think that requires (addr+len) to be a multiple of page size,
>> but maybe that is required.
> 
> I do think that 'system page size' means above HP page size, but this
> time it's just for fastpath_partition_init(), the earlier one seems to
> aligned fine (?? -- i havent really checked but there's no error)
> 

Hmmm, ok. Will check. But maybe let's not focus too much on the PGPROC
partitioning, I don't think that's likely to go into 19.

>>> 0006d: I've got one SIGBUS during a call to select
>>> pg_buffercache_numa_pages(); and it looks like that memory accessed is
>>> simply not mapped? (bug)
>>>
>>>     Program received signal SIGBUS, Bus error.
>>>     pg_buffercache_numa_pages (fcinfo=0x561a97e8e680) at
>>> ../contrib/pg_buffercache/pg_buffercache_pages.c:386
>>>     386                                     pg_numa_touch_mem_if_required(ptr);
>>>     (gdb) print ptr
>>>     $1 = 0x7f4ed0200000 <error: Cannot access memory at address 0x7f4ed0200000>
>>>     (gdb) where
>>>     #0  pg_buffercache_numa_pages (fcinfo=0x561a97e8e680) at
>>> ../contrib/pg_buffercache/pg_buffercache_pages.c:386
>>>     #1  0x0000561a672a0efe in ExecMakeFunctionResultSet
>>> (fcache=0x561a97e8e5d0, econtext=econtext@entry=0x561a97e8dab8,
>>> argContext=0x561a97ec62a0, isNull=0x561a97e8e578,
>>> isDone=isDone@entry=0x561a97e8e5c0) at
>>> ../src/backend/executor/execSRF.c:624
>>>     [..]
>>>
>>>     Postmaster had still attached shm (visible via smaps), and if you
>>> compare closely 0x7f4ed0200000 against sorted smaps:
>>>
>>>     7f4921400000-7f4b21400000 rw-s 252600000 00:11 151111
>>>       /anon_hugepage (deleted)
>>>     7f4b21400000-7f4d21400000 rw-s 452600000 00:11 151111
>>>       /anon_hugepage (deleted)
>>>     7f4d21400000-7f4f21400000 rw-s 652600000 00:11 151111
>>>       /anon_hugepage (deleted)
>>>     7f4f21400000-7f4f4bc00000 rw-s 852600000 00:11 151111
>>>       /anon_hugepage (deleted)
>>>     7f4f4bc00000-7f4f4c000000 rw-s 87ce00000 00:11 151111
>>>       /anon_hugepage (deleted)
>>>
>>>     it's NOT there at all (there's no mmap region starting with
>>> 0x"7f4e" ). It looks like because pg_buffercache_numa_pages() is not
>>> aware of this new mmaped() regions and instead does simple loop over
>>> all NBuffers with "for (char *ptr = startptr; ptr < endptr; ptr +=
>>> os_page_size)"?
>>>
>>
>> I'm confused. How could that mapping be missing? Was this with huge
>> pages / how many did you reserve on the nodes?
> 
> 
> OK I made and error and paritally got it correct (it crashes reliably)
> and partially mislead You, appologies, let me explain. There were two
> questions for me:
> a) why we make single mmap() and after numa_tonode_memory() we get
> plenty of mappings
> b) why we get SIGBUS (I've thought they are not continus, but they are
> after triple-checking)
> 
> ad a) My testing shows that on HP,as stated initially ("all of this
> was on 4s/4 NUMA nodes with HP on"). That's what the codes does, you
> get single mmaps() (resulting in single entry in smaps), but afte
> noda_tonode_memory() there's many of them. Even on laptop:
> 
> System has 1 NUMA nodes (0 to 0).
> Attempting to allocate 8.000000 MB of HugeTLB memory...
> Successfully allocated HugeTLB memory at 0x755828800000, smaps before:
> 755828800000-755829000000 rw-s 00000000 00:11 259808
>   /anon_hugepage (deleted)
> Pinning first part (from 0x755828800000) to NUMA node 0...
> smaps after:
> 755828800000-755828c00000 rw-s 00000000 00:11 259808
>   /anon_hugepage (deleted)
> 755828c00000-755829000000 rw-s 00400000 00:11 259808
>   /anon_hugepage (deleted)
> Pinning second part (from 0x755828c00000) to NUMA node 0...
> smaps after:
> 755828800000-755828c00000 rw-s 00000000 00:11 259808
>   /anon_hugepage (deleted)
> 755828c00000-755829000000 rw-s 00400000 00:11 259808
>   /anon_hugepage (deleted)
> 
> It gets even more funny, below I have 8MB HP=on, but just issue 2x
> numa_tonode_memory(for len 2MB on 4MB ptr to node0) (two times for
> ptr, second time in half of that):
> 
> System has 1 NUMA nodes (0 to 0).
> Attempting to allocate 8.000000 MB of HugeTLB memory...
> Successfully allocated HugeTLB memory at 0x7302dda00000, smaps before:
> 7302dda00000-7302de200000 rw-s 00000000 00:11 284859
>   /anon_hugepage (deleted)
> Pinning first part (from 0x7302dda00000) to NUMA node 0...
> smaps after:
> 7302dda00000-7302ddc00000 rw-s 00000000 00:11 284859
>   /anon_hugepage (deleted)
> 7302ddc00000-7302de200000 rw-s 00200000 00:11 284859
>   /anon_hugepage (deleted)
> Pinning second part (from 0x7302dde00000) to NUMA node 0...
> smaps after:
> 7302dda00000-7302ddc00000 rw-s 00000000 00:11 284859
>   /anon_hugepage (deleted)
> 7302ddc00000-7302dde00000 rw-s 00200000 00:11 284859
>   /anon_hugepage (deleted)
> 7302dde00000-7302de000000 rw-s 00400000 00:11 284859
>   /anon_hugepage (deleted)
> 7302de000000-7302de200000 rw-s 00600000 00:11 284859
>   /anon_hugepage (deleted)
> 
> Why 4 instead of 1? Because some mappings are now "default" becauswe
> their policy was not altered:
> 
> $ grep huge /proc/$(pidof testnumammapsplit)/numa_maps
> 7302dda00000 bind:0 file=/anon_hugepage\040(deleted) huge
> 7302ddc00000 default file=/anon_hugepage\040(deleted) huge
> 7302dde00000 bind:0 file=/anon_hugepage\040(deleted) huge
> 7302de000000 default file=/anon_hugepage\040(deleted) huge
> 
> Back to originnal error, they are consecutive regions and earlier problem is
> 
> error: 0x7f4ed0200000 <error: Cannot access memory at address 0x7f4ed0200000>
> start: 0x7f4921400000
> end:   0x7f4f4c000000
> 
> so it fits into that range (that was my mistate earlier, using just
> grep not checking are they really within that), but...
> 
>> Maybe there were not enough huge pages left on one of the nodes?
> 
> ad b) right, something like that. I've investigated that SIGBUS there
> (it's going to be long):
> 
> with shared_buffers=32GB, huge_pages 17715 (+1 from what postgres -C
> shared_memory_size_in_huge_pages returns), right after startup, but no
> touch:
> 
> Program received signal SIGBUS, Bus error.
> pg_buffercache_numa_pages (fcinfo=0x5572038790b8) at
> ../contrib/pg_buffercache/pg_buffercache_pages.c:386
> 386                                     pg_numa_touch_mem_if_required(ptr);
> (gdb) where
> #0  pg_buffercache_numa_pages (fcinfo=0x5572038790b8) at
> ../contrib/pg_buffercache/pg_buffercache_pages.c:386
> #1  0x00005571f54ddb7d in ExecMakeTableFunctionResult
> (setexpr=0x557203870d40, econtext=0x557203870ba8,
> argContext=<optimized out>, expectedDesc=0x557203870f80,
> randomAccess=false) at ../src/backend/executor/execSRF.c:234
> [..]
> (gdb) print ptr
> $1 = 0x7f6cf8400000 <error: Cannot access memory at address 0x7f6cf8400000>
> (gdb)
> 
> 
> then it shows?! no available hugepage on one of the nodes (while gdb
> is hanging and preving autorestart):
> 
> root@swiatowid:/sys/devices/system/node# grep -r -i HugePages_Free node*/meminfo
> node0/meminfo:Node 0 HugePages_Free:    299
> node1/meminfo:Node 1 HugePages_Free:    299
> node2/meminfo:Node 2 HugePages_Free:    299
> node3/meminfo:Node 3 HugePages_Free:      0
> 
> but they are equal in terms of size:
> node0/meminfo:Node 0 HugePages_Total:  4429
> node1/meminfo:Node 1 HugePages_Total:  4429
> node2/meminfo:Node 2 HugePages_Total:  4429
> node3/meminfo:Node 3 HugePages_Total:  4428
> 
> smaps shows that this address (7f6cf8400000) is mapped in this mapping:
> 7f6b49c00000-7f6d49c00000 rw-s 652600000 00:11 86064
>   /anon_hugepage (deleted)
> 
> numa_maps for this region shows this is this mapping on node3 (notice
> N3 + bind:3 matches lack of memory on Node 3 HugePAges_Free):
> 7f6b49c00000 bind:3 file=/anon_hugepage\040(deleted) huge dirty=3444
> N3=3444 kernelpagesize_kB=2048
> 
> the surrounding area of this looks like that:
> 
> 7f6549c00000 bind:0 file=/anon_hugepage\040(deleted) huge dirty=4096
> N0=4096 kernelpagesize_kB=2048
> 7f6749c00000 bind:1 file=/anon_hugepage\040(deleted) huge dirty=4096
> N1=4096 kernelpagesize_kB=2048
> 7f6949c00000 bind:2 file=/anon_hugepage\040(deleted) huge dirty=4096
> N2=4096 kernelpagesize_kB=2048
> 7f6b49c00000 bind:3 file=/anon_hugepage\040(deleted) huge dirty=3444
> N3=3444 kernelpagesize_kB=2048 <-- this is the one
> 7f6d49c00000 default file=/anon_hugepage\040(deleted) huge dirty=107
> mapmax=6 N3=107 kernelpagesize_kB=2048
> 
> Notice it's just N3=3444, while the others are much larger. So
> something was using that hugepages memory on N3:
> 
> # grep kernelpagesize_kB=2048 /proc/1679/numa_maps | grep -Po
> N[0-4]=[0-9]+ | sort
> N0=2
> N0=4096
> N1=2
> N1=4096
> N2=2
> N2=4096
> N3=1
> N3=1
> N3=1
> N3=1
> N3=107
> N3=13
> N3=3
> N3=3444
> 
> So per above it's not there (at least not as 2MB HP). But the number
> of mappings is wild there! (node where it is failing has plenty of
> memory, no hugepage memory left, but it has like 40k+ of small
> mappings!)
> 
> # grep -Po 'N[0-3]=' /proc/1679/numa_maps | sort | uniq -c
>      17 N0=
>      10 N1=
>       3 N2=
>   40434 N3=
> 
> most of them are `anon_inode:[io_uring]` (and I had
> max_connections=10k). You may ask why in spite of Andres optimization
> for reducing number segments for uring, it's not working for me ? Well
> I've just noticed way too silent failure to active this (altough I'm
> on 6.14.x):
>     2025-11-06 13:34:49.128 CET [1658] DEBUG:  can't use combined
> memory mapping for io_uring, kernel or liburing too old
> and I dont have io_uring_queue_init_mem()/HAVE_LIBURING_QUEUE_INIT_MEM
> apparently on liburing-2.3 (Debian's default). See [1] for more info
> (fix is not commited yet sadly).
> 
> Next try, now with io_method = worker and right before start:
> 
> root@swiatowid:/sys/devices/system/node# grep -r -i HugePages_Total
> node*/meminfo
> node0/meminfo:Node 0 HugePages_Total:  4429
> node1/meminfo:Node 1 HugePages_Total:  4429
> node2/meminfo:Node 2 HugePages_Total:  4429
> node3/meminfo:Node 3 HugePages_Total:  4428
> and HugePages_Free were 100% (if postgresql was down). After start
> (but without doing anything else):
> root@swiatowid:/sys/devices/system/node# grep -r -i HugePages_Free node*/meminfo
> node0/meminfo:Node 0 HugePages_Free:   4393
> node1/meminfo:Node 1 HugePages_Free:   4395
> node2/meminfo:Node 2 HugePages_Free:   4395
> node3/meminfo:Node 3 HugePages_Free:   3446
> 
> So sadly the picture is the same (something stole my HP on N3 and it's
> PostgreSQL on it's own). After some time of investigating that ("who
> stole my hugepage across whole OS"), I've just added MAP_POPULATE to
> the mix of PG_MMAP_FLAGS and got this after start:
> 
> root@swiatowid:/sys/devices/system/node# grep -r -i HugePages_Free node*/meminfo
> node0/meminfo:Node 0 HugePages_Free:      0
> node1/meminfo:Node 1 HugePages_Free:      0
> node2/meminfo:Node 2 HugePages_Free:      0
> node3/meminfo:Node 3 HugePages_Free:      1
> 
> and then the SELECT to pg_buffercache_numa works fine(!).
> 
> Another ways that I have found to eliminate that SIGBUS
> a. Would be to throw much more HugePages (so that node does not run to
> HugePages_Free), but that's not real option.
> b. Then I've reminded myself that I could be running custom kernel
> with experimental CONFIG_READ_ONLY_THP_FOR_FS (to reduce iTLB misses
> tranparently with specially linked PG; will double check exact stuff
> later), so I've thrown never into
> /sys/kernel/mm/transparent_hugepage/enabled and defrag too (yes ,
> disabled THP) and with that -- drumroll -- that SELECT works. The very
> same PG picture after startup (where earlier it would crash), now
> after SELECT it looks like that:
> 
> root@swiatowid:/sys/devices/system/node# grep -r -i HugePages_Free node*/meminfo
> node0/meminfo:Node 0 HugePages_Free:     83
> node1/meminfo:Node 1 HugePages_Free:      0
> node2/meminfo:Node 2 HugePages_Free:     81
> node3/meminfo:Node 3 HugePages_Free:     82
> 
> Hope that helps a little. To me it sounds like THP used that memory
> somehow and we've also wanted to use. With numa_interleave_ptr() that
> wouldn't be a problem because probably it would something else
> available, but not here as we indicated exact node.
> 
>>> 0006e:
>>>     I'm seeking confirmation, but is this the issue we have discussed
>>> on PgconfEU related to lack of detection of Mems_allowed, right? e.g.
>>>     $ numactl --membind="0,1" --cpunodebind="0,1"
>>> /usr/pgsql19/bin/pg_ctl -D /path start
>>>     still shows 4 NUMA nodes used. Current patches use
>>> numa_num_configured_nodes(), but it says 'This count includes any
>>> nodes that are currently DISABLED'. So I was wondering if I could help
>>> by migrating towards numa_num_task_nodes() / numa_get_mems_allowed()?
>>> It's the same as You wrote earlier to Alexy?
>>>
>>
>> If "mems_allowed" refers to nodes allowing memory allocation, then yes,
>> this would be one way to get into that issue. Oh, is this what happened
>> in 0006d?
> 
> OK, thanks for confirmation. No, 0006d was about normal numactl run,
> without --membind.
> 

I didn't have time to look into all this info about mappings, io_uring
yet, so no response from me.

>> I did get a couple of "operation canceled" failures, but only on fairly
>> old kernel versions (6.1 which came as default with the VM).
> 
> OK, I'll try to see that later too.
> 
> btw QQ regarding partitioned clockwise as I had thought: does this
> opens a road towards multiple BGwriters? (outside of this
> $thread/v1/PoC)
> 

I don't think the clocksweep partitioning is required for multiple
bgwriters, but it might make it easier.


regards

-- 
Tomas Vondra

  73. Re: Adding basic NUMA awareness

    Jakub Wartak <jakub.wartak@enterprisedb.com> — 2025-11-17T09:23:50Z

    On Tue, Nov 11, 2025 at 12:52 PM Tomas Vondra <tomas@vondra.me> wrote:
>
> Hi,
>
> here's a rebased patch series, fixing most of the smaller issues from
> v20251101, and making cfbot happy (hopefully).

Hi Tomas,

> >>> 0007b: pg_buffercache_pgproc -- nitpick, but maybe it would be better
> >>> called pg_shm_pgproc?
> >>>
> >>
> >> Right. It does not belong to pg_buffercache at all, I just added it
> >> there because I've been messing with that code already.
> >
> > Please keep them in for at least for some time (perhaps standalone
> > patch marked as not intended to be commited would work?). I find the
> > view extermely useful as it will allow us pinpointing local-vs-remote
> > NUMA fetches (we need to know the addres).
> >
>
> Are you referring to the _pgproc view specifically, or also to the view
> with buffer partitions? I don't intend to remove the view for shared
> buffers, that's indeed useful.

Both, even the _pgproc.


> Hmmm, ok. Will check. But maybe let's not focus too much on the PGPROC
> partitioning, I don't think that's likely to go into 19.

Oh ok.

> >>> 0006d: I've got one SIGBUS during a call to select
> >>> pg_buffercache_numa_pages(); and it looks like that memory accessed is
> >>> simply not mapped? (bug)
[..]
> I didn't have time to look into all this info about mappings, io_uring
> yet, so no response from me.
>

Ok, so the proper HP + SIGBUS explanation:

Appologies, earlier I wrote that disabling THP does workaround this,
but I've probably made an error there and used wrong binary back there
(with MAP_POPULATE in PG_MMAP_FLAGS), so please ignore that.

1. Before starting PG, with shared_buffers=32GB, huge_pages=on (2MB
ones), vm.nr_hugepages=17715, 4 NUMA nodes, kernel 6.14.x,
max_connections=10k, wal_buffers=1GB:

node0/hugepages/hugepages-2048kB/free_hugepages:4429
node1/hugepages/hugepages-2048kB/free_hugepages:4429
node2/hugepages/hugepages-2048kB/free_hugepages:4429
node3/hugepages/hugepages-2048kB/free_hugepages:4428

2. Just startup the PG with the older NUMA patchset 20251101. There
will be deficit across NUMA nodes right after startup, mostly one node
NUMA will allocate much more:

node0/hugepages/hugepages-2048kB/free_hugepages:4397
node1/hugepages/hugepages-2048kB/free_hugepages:3453
node2/hugepages/hugepages-2048kB/free_hugepages:4397
node3/hugepages/hugepages-2048kB/free_hugepages:4396

3. Check layout of NUMA maps for postmaster PID

7fc9cb200000 default file=/anon_hugepage\040(deleted) huge dirty=517
mapmax=8 N1=517 kernelpagesize_kB=2048 [!!!]
7fca0d600000 bind:0 file=/anon_hugepage\040(deleted) huge dirty=32
mapmax=2 N0=32 kernelpagesize_kB=2048
7fca11600000 bind:1 file=/anon_hugepage\040(deleted) huge dirty=32
mapmax=2 N1=32 kernelpagesize_kB=2048
7fca15600000 bind:2 file=/anon_hugepage\040(deleted) huge dirty=32
mapmax=2 N2=32 kernelpagesize_kB=2048
7fca19600000 bind:3 file=/anon_hugepage\040(deleted) huge dirty=32
mapmax=2 N3=32 kernelpagesize_kB=2048
7fca1d600000 default file=/anon_hugepage\040(deleted) huge
7fca1d800000 bind:0 file=/anon_hugepage\040(deleted) huge
7fcc1d800000 bind:1 file=/anon_hugepage\040(deleted) huge
7fce1d800000 bind:2 file=/anon_hugepage\040(deleted) huge
7fd01d800000 bind:3 file=/anon_hugepage\040(deleted) huge
7fd21d800000 default file=/anon_hugepage\040(deleted) huge dirty=425
mapmax=8 N1=425 kernelpagesize_kB=2048 [!!!]

So your patch doesn't do anything special for anything other than
Buffer Blocks and PGPROC in the above picture, so the the default
mmap() just keeps on with "default" NUMA policy which takes per above
(517+425) * 2MB = ~1884 MB of really used memory as per N1 entires. PG
does touch those regions on startup, but it doesnt really touch Buffer
Blocks. Anyway, this causes the missing amount of free huge pages on
the N1 (generates pressure on this Node 1).

So as it stands, the patchset is missing some form balancing to use
equal memory across nodes:
- each node to be forced to get certain amount of BufferBlocks/NUMA nodes blocks
- yet we do nothing and leave at the "defaults" the others regions
(e..g $SegHDR (start of shm) .. first Buffers Block), as those are
placed on the current node (due default policy), which in causes turns
this memory overallocation imbalance (so in the example N1 will get
Buffer Blocks + everything else, but that only happens on real access
not during mmap() due to lazy/first touch policy)

Currently, any launch of anything that touches imbalanced NUMA node
memory with deficit (N1 above) - use of pg_shm_allocations,
pg_buffercache - it will cause stress there and end up in SIGBUS.
This looks by design on Linux kernel side: exc:page_fault() ->
do_user_addr_fault() -> do_sigbus() AKA force_sig_fault(). But, if I
hack pg to hack do interleave (or just numactl --interleave=all ... )
to effectivley interleave those 3 "default" regions instead, so I'll
get "interleave" like that:

7fb2dd000000 interleave:0-3 file=/anon_hugepage\040(deleted) huge
dirty=517 mapmax=8 N0=129 N1=132 N2=128 N3=128 kernelpagesize_kB=2048
7fb31f400000 bind:0 file=/anon_hugepage\040(deleted) huge dirty=32
mapmax=2 N0=32 kernelpagesize_kB=2048
7fb323400000 bind:1 file=/anon_hugepage\040(deleted) huge dirty=32
mapmax=2 N1=32 kernelpagesize_kB=2048
7fb327400000 bind:2 file=/anon_hugepage\040(deleted) huge dirty=32
mapmax=2 N2=32 kernelpagesize_kB=2048
7fb32b400000 bind:3 file=/anon_hugepage\040(deleted) huge dirty=32
mapmax=2 N3=32 kernelpagesize_kB=2048
7fb32f400000 interleave:0-3 file=/anon_hugepage\040(deleted) huge
7fb32f600000 bind:0 file=/anon_hugepage\040(deleted) huge
7fb52f600000 bind:1 file=/anon_hugepage\040(deleted) huge
7fb72f600000 bind:2 file=/anon_hugepage\040(deleted) huge
7fb92f600000 bind:3 file=/anon_hugepage\040(deleted) huge
7fbb2f600000 interleave:0-3 file=/anon_hugepage\040(deleted) huge
dirty=425 N0=106 N1=106 N2=105 N3=108 kernelpagesize_kB=2048

then even after fully touching everything (via select to
pg_shm_allocations), it'll run, I'll get much better balance, and wont
have SIGBUS issues:

node0/hugepages/hugepages-2048kB/free_hugepages:23
node1/hugepages/hugepages-2048kB/free_hugepages:23
node2/hugepages/hugepages-2048kB/free_hugepages:23
node3/hugepages/hugepages-2048kB/free_hugepages:22

This somehow demonstrates that enough free memory is out there, it's
just imbalance that causes SIGBUS. I hope this somehow hopefully
answers one of Your's main questions as per in the very first messages
what we should do with remaining shared_buffer members. I would like
to hear your thoughts on this, before I start benchmarking this for
real as I didnt want to bench it yet, as such interleaving could alter
the the test results.

Other things I've noticed:
- smaps Size: && Shared_Hugetlb: reporting are a lie and are showing
really touched memory, not assigned memory
- same goes for procfs's numa_maps, ignore the N[0-3] sizes, it's only
"really used", not assigned
- the best is just to manually calculate size from pointers/address range itself

-J.

  74. Re: Adding basic NUMA awareness

    Tomas Vondra <tomas@vondra.me> — 2025-11-17T17:28:34Z

    
On 11/17/25 10:23, Jakub Wartak wrote:
> On Tue, Nov 11, 2025 at 12:52 PM Tomas Vondra <tomas@vondra.me> wrote:
>>
>> Hi,
>>
>> here's a rebased patch series, fixing most of the smaller issues from
>> v20251101, and making cfbot happy (hopefully).
> 
> Hi Tomas,
> 
>>>>> 0007b: pg_buffercache_pgproc -- nitpick, but maybe it would be better
>>>>> called pg_shm_pgproc?
>>>>>
>>>>
>>>> Right. It does not belong to pg_buffercache at all, I just added it
>>>> there because I've been messing with that code already.
>>>
>>> Please keep them in for at least for some time (perhaps standalone
>>> patch marked as not intended to be commited would work?). I find the
>>> view extermely useful as it will allow us pinpointing local-vs-remote
>>> NUMA fetches (we need to know the addres).
>>>
>>
>> Are you referring to the _pgproc view specifically, or also to the view
>> with buffer partitions? I don't intend to remove the view for shared
>> buffers, that's indeed useful.
> 
> Both, even the _pgproc.
> 
> 
>> Hmmm, ok. Will check. But maybe let's not focus too much on the PGPROC
>> partitioning, I don't think that's likely to go into 19.
> 
> Oh ok.
> 
>>>>> 0006d: I've got one SIGBUS during a call to select
>>>>> pg_buffercache_numa_pages(); and it looks like that memory accessed is
>>>>> simply not mapped? (bug)
> [..]
>> I didn't have time to look into all this info about mappings, io_uring
>> yet, so no response from me.
>>
> 
> Ok, so the proper HP + SIGBUS explanation:
> 
> Appologies, earlier I wrote that disabling THP does workaround this,
> but I've probably made an error there and used wrong binary back there
> (with MAP_POPULATE in PG_MMAP_FLAGS), so please ignore that.
> 
> 1. Before starting PG, with shared_buffers=32GB, huge_pages=on (2MB
> ones), vm.nr_hugepages=17715, 4 NUMA nodes, kernel 6.14.x,
> max_connections=10k, wal_buffers=1GB:
> 
> node0/hugepages/hugepages-2048kB/free_hugepages:4429
> node1/hugepages/hugepages-2048kB/free_hugepages:4429
> node2/hugepages/hugepages-2048kB/free_hugepages:4429
> node3/hugepages/hugepages-2048kB/free_hugepages:4428
> 
> 2. Just startup the PG with the older NUMA patchset 20251101. There
> will be deficit across NUMA nodes right after startup, mostly one node
> NUMA will allocate much more:
> 
> node0/hugepages/hugepages-2048kB/free_hugepages:4397
> node1/hugepages/hugepages-2048kB/free_hugepages:3453
> node2/hugepages/hugepages-2048kB/free_hugepages:4397
> node3/hugepages/hugepages-2048kB/free_hugepages:4396
> 
> 3. Check layout of NUMA maps for postmaster PID
> 
> 7fc9cb200000 default file=/anon_hugepage\040(deleted) huge dirty=517
> mapmax=8 N1=517 kernelpagesize_kB=2048 [!!!]
> 7fca0d600000 bind:0 file=/anon_hugepage\040(deleted) huge dirty=32
> mapmax=2 N0=32 kernelpagesize_kB=2048
> 7fca11600000 bind:1 file=/anon_hugepage\040(deleted) huge dirty=32
> mapmax=2 N1=32 kernelpagesize_kB=2048
> 7fca15600000 bind:2 file=/anon_hugepage\040(deleted) huge dirty=32
> mapmax=2 N2=32 kernelpagesize_kB=2048
> 7fca19600000 bind:3 file=/anon_hugepage\040(deleted) huge dirty=32
> mapmax=2 N3=32 kernelpagesize_kB=2048
> 7fca1d600000 default file=/anon_hugepage\040(deleted) huge
> 7fca1d800000 bind:0 file=/anon_hugepage\040(deleted) huge
> 7fcc1d800000 bind:1 file=/anon_hugepage\040(deleted) huge
> 7fce1d800000 bind:2 file=/anon_hugepage\040(deleted) huge
> 7fd01d800000 bind:3 file=/anon_hugepage\040(deleted) huge
> 7fd21d800000 default file=/anon_hugepage\040(deleted) huge dirty=425
> mapmax=8 N1=425 kernelpagesize_kB=2048 [!!!]
> 
> So your patch doesn't do anything special for anything other than
> Buffer Blocks and PGPROC in the above picture, so the the default
> mmap() just keeps on with "default" NUMA policy which takes per above
> (517+425) * 2MB = ~1884 MB of really used memory as per N1 entires. PG
> does touch those regions on startup, but it doesnt really touch Buffer
> Blocks. Anyway, this causes the missing amount of free huge pages on
> the N1 (generates pressure on this Node 1).
> 
> So as it stands, the patchset is missing some form balancing to use
> equal memory across nodes:
> - each node to be forced to get certain amount of BufferBlocks/NUMA nodes blocks
> - yet we do nothing and leave at the "defaults" the others regions
> (e..g $SegHDR (start of shm) .. first Buffers Block), as those are
> placed on the current node (due default policy), which in causes turns
> this memory overallocation imbalance (so in the example N1 will get
> Buffer Blocks + everything else, but that only happens on real access
> not during mmap() due to lazy/first touch policy)
> 
> Currently, any launch of anything that touches imbalanced NUMA node
> memory with deficit (N1 above) - use of pg_shm_allocations,
> pg_buffercache - it will cause stress there and end up in SIGBUS.
> This looks by design on Linux kernel side: exc:page_fault() ->
> do_user_addr_fault() -> do_sigbus() AKA force_sig_fault(). But, if I
> hack pg to hack do interleave (or just numactl --interleave=all ... )
> to effectivley interleave those 3 "default" regions instead, so I'll
> get "interleave" like that:
> 
> 7fb2dd000000 interleave:0-3 file=/anon_hugepage\040(deleted) huge
> dirty=517 mapmax=8 N0=129 N1=132 N2=128 N3=128 kernelpagesize_kB=2048
> 7fb31f400000 bind:0 file=/anon_hugepage\040(deleted) huge dirty=32
> mapmax=2 N0=32 kernelpagesize_kB=2048
> 7fb323400000 bind:1 file=/anon_hugepage\040(deleted) huge dirty=32
> mapmax=2 N1=32 kernelpagesize_kB=2048
> 7fb327400000 bind:2 file=/anon_hugepage\040(deleted) huge dirty=32
> mapmax=2 N2=32 kernelpagesize_kB=2048
> 7fb32b400000 bind:3 file=/anon_hugepage\040(deleted) huge dirty=32
> mapmax=2 N3=32 kernelpagesize_kB=2048
> 7fb32f400000 interleave:0-3 file=/anon_hugepage\040(deleted) huge
> 7fb32f600000 bind:0 file=/anon_hugepage\040(deleted) huge
> 7fb52f600000 bind:1 file=/anon_hugepage\040(deleted) huge
> 7fb72f600000 bind:2 file=/anon_hugepage\040(deleted) huge
> 7fb92f600000 bind:3 file=/anon_hugepage\040(deleted) huge
> 7fbb2f600000 interleave:0-3 file=/anon_hugepage\040(deleted) huge
> dirty=425 N0=106 N1=106 N2=105 N3=108 kernelpagesize_kB=2048
> 
> then even after fully touching everything (via select to
> pg_shm_allocations), it'll run, I'll get much better balance, and wont
> have SIGBUS issues:
> 
> node0/hugepages/hugepages-2048kB/free_hugepages:23
> node1/hugepages/hugepages-2048kB/free_hugepages:23
> node2/hugepages/hugepages-2048kB/free_hugepages:23
> node3/hugepages/hugepages-2048kB/free_hugepages:22
> 
> This somehow demonstrates that enough free memory is out there, it's
> just imbalance that causes SIGBUS. I hope this somehow hopefully
> answers one of Your's main questions as per in the very first messages
> what we should do with remaining shared_buffer members. I would like
> to hear your thoughts on this, before I start benchmarking this for
> real as I didnt want to bench it yet, as such interleaving could alter
> the the test results.
> 

Thanks for investigating this. If I understand the findings correctly,
it agrees with my imprecise explanation in [1], right? There I said:

> ...
> You may ask why the per-node limit is too low. We still need just
> shared_memory_size_in_huge_pages, right? And if we were partitioning
> the whole memory segment, that'd be true. But we only to that for
> shared buffers, and there's a lot of other shared memory - could be
> 1-2GB or so, depending on the configuration.
>
> And this gets placed on one of the nodes, and it counts against the
> limit on that particular node. And so it doesn't have enough huge
> pages to back the partition of shared buffers.
> ...

Which I think is mostly the same thing you're saying, and you have the
maps to support it.

In any case, I think setting "interleave" as the default policy, and
then overriding it for the areas we partition explicitly (buffers,
pgproc), seems like the right solution. The only other solution would be
balance it ourselves, but how is that different from interleaving?

So I think this makes sense, and you can do --interleave=all for the
benchmark.

[1]
https://www.postgresql.org/message-id/71a46484-053c-4b81-ba32-ddac050a8b5d%40vondra.me

I suppose we may need to adjust shared_memory_size_in_huge_pages,
because the interleave followed by explicit partitioning may still leave
behind a bit of imbalance. It should be only a couple pages, but I
haven't done the math yet.


regards

-- 
Tomas Vondra

  75. Re: Adding basic NUMA awareness

    Tomas Vondra <tomas@vondra.me> — 2025-11-21T18:49:25Z

    Hi,

Here's an updated version of the patch series.

It fixes a bunch of issues in pg_buffercache_pages.c - duplicate attnums
and a incorrect array length.

The main change is in 0006 - it sets the default allocation policy for
shmem to interleaving, before doing the explicit partitioning for shared
buffers. It does it by calling numa_set_membind before the mmap(), and
then numa_interleave_memory() on the allocated shmem. It does this to
allow using MAP_POPULATE - but that's commented out by default.

This does seem to solve the SIGBUS failures for me. I still think there
might be a small chance of hitting that, because of locating an extra
"boundary" page on one of the nodes. But it should be solvable by
reserving a couple more pages.

Jakub, what do you think?


regards

-- 
Tomas Vondra

  76. Re: Adding basic NUMA awareness

    Jakub Wartak <jakub.wartak@enterprisedb.com> — 2025-11-25T14:12:46Z

    Hi Tomas!

[..]
> Which I think is mostly the same thing you're saying, and you have the maps to support it.

Right, the thread is kind of long, you were right back then, well but
at least we've got a solid explanation with data.

> Here's an updated version of the patch series.

Just for double confirmation, I've used those ones (v20251121*) and
they indeed interleaved parts of shm memory.

> It fixes a bunch of issues in pg_buffercache_pages.c - duplicate attnums
> and a incorrect array length.

You'll need to rebase again, pg_buffercache_numa got updated again on
Monday and clashes with 0006.

> The main change is in 0006 - it sets the default allocation policy for
> shmem to interleaving, before doing the explicit partitioning for shared
> buffers. It does it by calling numa_set_membind before the mmap(), and
> then numa_interleave_memory() on the allocated shmem. It does this to
> allow using MAP_POPULATE - but that's commented out by default.
>
> This does seem to solve the SIGBUS failures for me. I still think there
> might be a small chance of hitting that, because of locating an extra
> "boundary" page on one of the nodes. But it should be solvable by
> reserving a couple more pages.

I can confirm, never got any SIGBUS during the later described
benchmarks, so it's much better now.

> Jakub, what do you think?

On one side not using MAP_POPULATE gives instant startup, but on the
other it gives much better predictability latencies especially fresh
after starting up (this might matter to folks who like to benchmark --
us?, but initially I've just used it as a simple hack to touch
memory). I would be wary of using MAP_POPULATE with s_b when it would
be sized in hundreths of GBs, it could take minutes in startup, which
would be terrible if someone would hit SIGSEGV on production and
expect restart_after_crash=true to save him. I mean WAL redo crash
would be terrible, but that would be terrible * 2. Also pretty
long-term with DIO, we'll get much bigger s_b anyway (hopefully), so
it would hurt even more, so I think that would be a bad path(?)

I've benchmarked the thing in two scenarios (readonly pgbench < s_b
size across variations of code and connections and 2nd one with
seqconcurrrentscans) in solid stable conditions: 4s32c64t == 4 NUMA
nodes, 128GB RAM, 31GB shared_buffers dbsize ~29GB, 6.14.x, no idle
CPU states, no turbo boost, and so on, literally great home heater
when there's -3C outside!)

The data is baseline "100%" for master along with HP on/off (so it's
showing diff % from respective HP setting):

scenario I: pgbench -S

                 connections
branch   HP      1       8       64      128     1024
master   off     100.00% 100.00% 100.00% 100.00% 100.00%
master   on      100.00% 100.00% 100.00% 100.00% 100.00%
numa16   off     99.13%  100.46% 99.66%  99.44%  89.60%
numa16   on      101.80% 100.89% 99.36%  99.89%  93.43%
numa4    off     96.82%  100.61% 99.37%  99.92%  94.41%
numa4    on      101.83% 100.61% 99.35%  99.69%  101.48%
pgproc16 off     99.13%  100.84% 99.38%  99.85%  91.15%
pgproc16 on      101.72% 101.40% 99.72%  100.14% 95.20%
pgproc4  off     98.63%  101.44% 100.05% 100.14% 90.97%
pgproc4  on      101.05% 101.46% 99.92%  100.31% 97.60%
sweep16  off     99.53%  101.14% 100.71% 100.75% 101.52%
sweep16  on      97.63%  102.49% 100.42% 100.75% 105.56%
sweep4   off     99.43%  101.59% 100.06% 100.45% 104.63%
sweep4   on      97.69%  101.59% 100.70% 100.69% 104.70%

I would consider everything +/- 3% as noise (technically each branch
was a different compilation/ELF binary, as changing this #define
required to do so to get 4 vs 16; please see attached script). I miss
the explanation why without HP it deteriorates so much with for c=1024
with the patches.

scenario II: pgbench -f seqconcurrscans.pgb; 64 partitions from
pgbench --partitions=64 -i -s 2000 [~29GB] being hammered in modulo
without PQ by:
    \set num (:client_id % 8) + 1
    select sum(octet_length(filler)) from pgbench_accounts_:num;

                 connections
branch   HP      1       8       64      128
master   off     100.00% 100.00% 100.00% 100.00%
master   on      100.00% 100.00% 100.00% 100.00%
numa16   off     115.62% 108.87% 101.08% 111.56%
numa16   on      107.68% 104.90% 102.98% 105.51%
numa4    off     113.55% 111.41% 101.45% 113.10%
numa4    on      107.90% 106.60% 103.68% 106.98%
pgproc16 off     111.70% 108.27% 98.69%  109.36%
pgproc16 on      106.98% 100.69% 101.98% 103.42%
pgproc4  off     112.41% 106.15% 100.03% 112.03%
pgproc4  on      106.73% 105.77% 103.74% 101.13%
sweep16  off     100.63% 100.38% 98.41%  103.46%
sweep16  on      109.03% 99.15%  101.17% 99.19%
sweep4   off     102.04% 101.16% 101.71% 91.86%
sweep4   on      108.33% 101.69% 97.14%  100.92%

The benefit varies with like +3-10% depending on connection count.
Quite frankly I was expecting a little bit more, especially after
re-reading [1]. Maybe you preloaded it there using pg_prewarm? (here
I've randomly warmed it using pgbench). Probably it's something with
my test, I'll take yet another look hopefully soon. The good thing is
that it never crashed and I haven't seen any errors like "Bad address"
probably related to AIO as you saw in [1], perhaps I wasn't using
uring.

0007 (PROCs) still complains with "mbind: Invalid argument" (aligment issue)

-J.

[1] - https://www.postgresql.org/message-id/e4d7e6fc-b5c5-4288-991c-56219db2edd5%40vondra.me

  77. Re: Adding basic NUMA awareness

    Tomas Vondra <tomas@vondra.me> — 2025-11-26T16:19:12Z

    On 11/25/25 15:12, Jakub Wartak wrote:
> Hi Tomas!
> 
> [..]
>> Which I think is mostly the same thing you're saying, and you have the maps to support it.
> 
> Right, the thread is kind of long, you were right back then, well but
> at least we've got a solid explanation with data.
> 
>> Here's an updated version of the patch series.
> 
> Just for double confirmation, I've used those ones (v20251121*) and
> they indeed interleaved parts of shm memory.
> 
>> It fixes a bunch of issues in pg_buffercache_pages.c - duplicate attnums
>> and a incorrect array length.
> 
> You'll need to rebase again, pg_buffercache_numa got updated again on
> Monday and clashes with 0006.
> 

Rebased patch series attached.

>> The main change is in 0006 - it sets the default allocation policy for
>> shmem to interleaving, before doing the explicit partitioning for shared
>> buffers. It does it by calling numa_set_membind before the mmap(), and
>> then numa_interleave_memory() on the allocated shmem. It does this to
>> allow using MAP_POPULATE - but that's commented out by default.
>>
>> This does seem to solve the SIGBUS failures for me. I still think there
>> might be a small chance of hitting that, because of locating an extra
>> "boundary" page on one of the nodes. But it should be solvable by
>> reserving a couple more pages.
> 
> I can confirm, never got any SIGBUS during the later described
> benchmarks, so it's much better now.
> 

Good!

>> Jakub, what do you think?
> 
> On one side not using MAP_POPULATE gives instant startup, but on the
> other it gives much better predictability latencies especially fresh
> after starting up (this might matter to folks who like to benchmark --
> us?, but initially I've just used it as a simple hack to touch
> memory). I would be wary of using MAP_POPULATE with s_b when it would
> be sized in hundreths of GBs, it could take minutes in startup, which
> would be terrible if someone would hit SIGSEGV on production and
> expect restart_after_crash=true to save him. I mean WAL redo crash
> would be terrible, but that would be terrible * 2. Also pretty
> long-term with DIO, we'll get much bigger s_b anyway (hopefully), so
> it would hurt even more, so I think that would be a bad path(?)
> 

I think the MAP_POPULATE should be optional, enabled by GUC.

> I've benchmarked the thing in two scenarios (readonly pgbench < s_b
> size across variations of code and connections and 2nd one with
> seqconcurrrentscans) in solid stable conditions: 4s32c64t == 4 NUMA
> nodes, 128GB RAM, 31GB shared_buffers dbsize ~29GB, 6.14.x, no idle
> CPU states, no turbo boost, and so on, literally great home heater
> when there's -3C outside!)
> 
> The data is baseline "100%" for master along with HP on/off (so it's
> showing diff % from respective HP setting):
> 
> scenario I: pgbench -S
> 
>                  connections
> branch   HP      1       8       64      128     1024
> master   off     100.00% 100.00% 100.00% 100.00% 100.00%
> master   on      100.00% 100.00% 100.00% 100.00% 100.00%
> numa16   off     99.13%  100.46% 99.66%  99.44%  89.60%
> numa16   on      101.80% 100.89% 99.36%  99.89%  93.43%
> numa4    off     96.82%  100.61% 99.37%  99.92%  94.41%
> numa4    on      101.83% 100.61% 99.35%  99.69%  101.48%
> pgproc16 off     99.13%  100.84% 99.38%  99.85%  91.15%
> pgproc16 on      101.72% 101.40% 99.72%  100.14% 95.20%
> pgproc4  off     98.63%  101.44% 100.05% 100.14% 90.97%
> pgproc4  on      101.05% 101.46% 99.92%  100.31% 97.60%
> sweep16  off     99.53%  101.14% 100.71% 100.75% 101.52%
> sweep16  on      97.63%  102.49% 100.42% 100.75% 105.56%
> sweep4   off     99.43%  101.59% 100.06% 100.45% 104.63%
> sweep4   on      97.69%  101.59% 100.70% 100.69% 104.70%
> 
> I would consider everything +/- 3% as noise (technically each branch
> was a different compilation/ELF binary, as changing this #define
> required to do so to get 4 vs 16; please see attached script). I miss
> the explanation why without HP it deteriorates so much with for c=1024
> with the patches.

I wouldn't expect a big difference for "pgbench -S". That workload has
so much other fairly expensive stuff (e.g. initializing index scans
etc.), the cost of buffer replacement is going to be fairly limited.

The regressions for numa/pgproc patches with 1024 clients are annoying,
but how realistic is such scenario? With 32/64 CPUs, having 1024 active
connections is a substantial overload. If we can fix this, great. But I
think such regression may be OK if we get benefits for reasonable setups
(with fewer clients).

I don't know why it's happening, though. I haven't been testing cases
with so many clients (compared to number of CPUs).

> 
> scenario II: pgbench -f seqconcurrscans.pgb; 64 partitions from
> pgbench --partitions=64 -i -s 2000 [~29GB] being hammered in modulo
> without PQ by:
>     \set num (:client_id % 8) + 1
>     select sum(octet_length(filler)) from pgbench_accounts_:num;
> 
>                  connections
> branch   HP      1       8       64      128
> master   off     100.00% 100.00% 100.00% 100.00%
> master   on      100.00% 100.00% 100.00% 100.00%
> numa16   off     115.62% 108.87% 101.08% 111.56%
> numa16   on      107.68% 104.90% 102.98% 105.51%
> numa4    off     113.55% 111.41% 101.45% 113.10%
> numa4    on      107.90% 106.60% 103.68% 106.98%
> pgproc16 off     111.70% 108.27% 98.69%  109.36%
> pgproc16 on      106.98% 100.69% 101.98% 103.42%
> pgproc4  off     112.41% 106.15% 100.03% 112.03%
> pgproc4  on      106.73% 105.77% 103.74% 101.13%
> sweep16  off     100.63% 100.38% 98.41%  103.46%
> sweep16  on      109.03% 99.15%  101.17% 99.19%
> sweep4   off     102.04% 101.16% 101.71% 91.86%
> sweep4   on      108.33% 101.69% 97.14%  100.92%
> 
> The benefit varies with like +3-10% depending on connection count.
> Quite frankly I was expecting a little bit more, especially after
> re-reading [1]. Maybe you preloaded it there using pg_prewarm? (here
> I've randomly warmed it using pgbench). Probably it's something with
> my test, I'll take yet another look hopefully soon. The good thing is
> that it never crashed and I haven't seen any errors like "Bad address"
> probably related to AIO as you saw in [1], perhaps I wasn't using
> uring.
> 

Hmmm. I'd have expected better results for this workload. So I tried
re-running my seqscan benchmark on the 176-core instance, and I got this:

    clients   master   0001   0002   0003   0004   0005   0006   0007
    -----------------------------------------------------------------
     64           44     43     35     40     53     53     46     45
     96           55     54     42     47     57     58     53     53
    128           59     59     46     50     58     58     57     60

    clients   0001   0002   0003   0004   0005   0006   0007
    --------------------------------------------------------
     64        98%    79%    92%   122%   122%   105%   104%
     96        99%    76%    86%   104%   105%    97%    97%
    128        99%    77%    84%    98%    98%    97%   101%

I did the benchmark for individual parts of the patch series. There's a
clear (~20%) speedup for 0005, but 0006 and 0007 make it go away. The
0002/0003 regress it quite a bit. And with 128 clients there's no
improvement at all.

This was with the default number of partitions (i.e. 4). If I increase
the number to 16, I get this:

    clients   master   0001   0002   0003   0004   0005   0006   0007
    -----------------------------------------------------------------
     64           44     43     69     82     87     87     78     79
     96           55     54     65     85     91     91     86     86
    128           59     59     66     77     83     83     82     86

    clients   0001   0002   0003   0004   0005   0006   0007
    --------------------------------------------------------
     64        99%   158%   189%   199%   199%   180%   180%
     96       100%   119%   156%   167%   167%   157%   158%
    128        99%   112%   130%   140%   140%   139%   145%

And with 32 partitions, I get this:

    clients   master   0001   0002   0003   0004   0005   0006   0007
    -----------------------------------------------------------------
     64           44     44     88     91     90     90     84     84
     96           55     54     89     93     93     92     90     91
    128           59     59     85     84     86     85     88     87

    clients   0001   0002   0003   0004   0005   0006   0007
    --------------------------------------------------------
     64       100%   202%   208%   207%   207%   193%   193%
     96       100%   163%   169%   171%   168%   165%   166%
    128        99%   144%   142%   146%   144%   149%   146%

Those are clearly much better results, so I guess the default number of
partitions may be too low.

What bothers me is that this seems like a very narrow benchmark. I mean,
few systems are doing concurrent seqscans putting this much pressure on
buffer replacement. And once the plans start to do other stuff, the
contention on clock sweep seems to go down substantially (as shown by
the read-only pgbench). So the question is - is this really worth it?

> 0007 (PROCs) still complains with "mbind: Invalid argument" (aligment issue)
> 

Should be fixed by the attached patches. The 0006 patch has an issue
with mbind too, but it was visible only when the buffers were not a nice
multiple of memory pages (and multiples of 1GB are fine).

This also moves the memset() until after placing the PGPROC partitions
to different NUMA nodes.

The results above are from v20251121. I'll rerun the tests with the nw
version of the patches. But it can only change the 0006/0007 results, of
course. The 0001-0005 are the same.


regards

-- 
Tomas Vondra

  78. Re: Adding basic NUMA awareness

    Jakub Wartak <jakub.wartak@enterprisedb.com> — 2025-12-02T12:26:33Z

    On Wed, Nov 26, 2025 at 5:19 PM Tomas Vondra <tomas@vondra.me> wrote:

> Rebased patch series attached.

Thanks. BTW still with the old patchset series, One additional thing
that I've found out related to interleave is that in
CreateAnonymousSegment() with the default check_debug='', we still
issue numa_interleave_memory(ptr..). It should be optional (this also
affects earlier calls too). Tiny patch attached.

> I think the MAP_POPULATE should be optional, enabled by GUC.

OK, but you mean it's a new option to debug_numa, right? (not some
separate) so debug_numa='prefault' then?

> > I would consider everything +/- 3% as noise (technically each branch
> > was a different compilation/ELF binary, as changing this #define
> > required to do so to get 4 vs 16; please see attached script). I miss
> > the explanation why without HP it deteriorates so much with for c=1024
> > with the patches.
>
> I wouldn't expect a big difference for "pgbench -S". That workload has
> so much other fairly expensive stuff (e.g. initializing index scans
> etc.), the cost of buffer replacement is going to be fairly limited.

Right. OK, so I've got the seqconcurrentscans comparison done right,
that is when prewarmed and not naturally filled:

@master, 29GB/s mem bandwidth
latency average = 1255.572 ms
latency stddev = 417.162 ms
tps = 50.451925 (without initial connection time)

@v20251121 patchset, 41GB/s (~10GB/s per socket)
latency average = 719.931 ms
latency stddev = 14.874 ms
tps = 88.362091 (without initial connection time)

The main PMC difference seems to be much lower "backend cycles idle"
(51% master and vs 31% for the NUMA debug_numa="buffers,procs", so
less is waiting on memory, thus it gets that speedup and better IPC).

Anyway, the biggest gripe right now (at least to me) is reliable
benchmarking. Below runs are all apples and oranges comparisons (they
measure different stuff although looks the same initially)
- restart and just select pg_shmem_allocations_numa or prewarm puts
everything into 1 NUMA node with check_numa='', because of prefaulting
happening during select-view case
- restart and pgbench -i -s XX (same issue as above) then pgbench -
you get the same, everything on potential one NUMA node (because
pgbench prefaults just on one)
- restart and pgbench -c 64.. with debug_numa='' (off) MIGHT get
random NUMA layout, how's that is supposed to be deterministic? at
least with debug_numa='buffers' you get determinism..
- the shared_buffers size vs size of dataset read, the moment you
start doing something CPU intensive (or like calling syscalls just for
VFS cache), the benefit seems to disappear at least on my hardware

Anyway, depending on the scenario I could get varied results like
34tps .. 88tps here. The debug_numa='buffers,..' gives just assurance
of the proper layout of shared memory is there (one could even argue
that such performance deviations across runs are bug ;)).

> The regressions for numa/pgproc patches with 1024 clients are annoying,
> but how realistic is such scenario? With 32/64 CPUs, having 1024 active
> connections is a substantial overload. If we can fix this, great. But I
> think such regression may be OK if we get benefits for reasonable setups
> (with fewer clients).
>
> I don't know why it's happening, though. I haven't been testing cases
> with so many clients (compared to the number of CPUs).

The only thing in my mind about deterioration of high-connection count
(AKA -c 1024 scenario) with pgprocs, would be related to the question
you raised in 0007 "Note: The scheduler may migrate the process to a
different CPU/node later. Maybe we should consider pinning the process
to the node?"

I think the answer is yes, so to fetch MyProc based on sched_getcpu()
and then maybe with additional numa_flags & new PROCS_PIN_NODE simply
numa_run_on_node(node)? I've tried this:

pgbench -c 1024 -j 64 -P 1 -T 30 -S -M prepared got:

@numa-empty-debug_numa         ~434k TPS, ~12k CPU migrations/second
@numa+buffers+pgproc           ~412k TPS, 7-8k CPU migrations/second
@numa+buffers+pgproc+pinnode   ~434k TPS, still with 7-8k CPU
migrations/second (so same)
but I've verified for the last one, with bpftrace on that
tracepoint:sched:sched_migrate_task did not performed node-to-node
process bounces anymore (it did for pgbench but not for postgres
itself with this numa_run_on_node())

> > scenario II: pgbench -f seqconcurrscans.pgb; 64 partitions from
> > pgbench --partitions=64 -i -s 2000 [~29GB] being hammered in modulo
[..]
>
> Hmmm. I'd have expected better results for this workload. So I tried
> re-running my seqscan benchmark on the 176-core instance, and I got this:

[..]
Thanks!

> I did the benchmark for individual parts of the patch series. There's a
> clear (~20%) speedup for 0005, but 0006 and 0007 make it go away. The
> 0002/0003 regress it quite a bit. And with 128 clients there's no
> improvement at all.
[..]
> Those are clearly much better results, so I guess the default number of
> partitions may be too low.
>
> What bothers me is that this seems like a very narrow benchmark. I mean,
> few systems are doing concurrent seqscans putting this much pressure on
> buffer replacement. And once the plans start to do other stuff, the
> contention on clock sweep seems to go down substantially (as shown by
> the read-only pgbench). So the question is - is this really worth it?

Are you thinking here about whole NUMA patchset or just clocksweep? I
think multiple clocksweep are just not shining because other
bottlenecks hammer the efficiency here. Andres talk about it exactly
here https://youtu.be/V75KpACdl6E?t=1990 (He mentions out of order
execution, I see btrees in reports as top#1). So maybe it's just too
early to see the results of this optimization?

As for classic readonly pgbench -S I still see roughly 1:8 local to
remote (!) DRAM access (1 <-> 3 sockets) even with those patches, so
potentially something could be improved in far future for sure (that
would require some memaddr monitoring for most remote DRAM misses <->
pg inter-shm ptr mapping; think of pg_shmem_allocations_numa with
local/remote counters or maybe just fallback to perf-c2c).

To sum up, IMHO I understand this $thread's NUMA implementation as:
- it's strictly a guard mechanism to get determinism (for most cases)
-- it fixes "imbalance"
- no performance boost for OLTP as such
- for analytics it could be win (in-memory workloads; well PG is not
fully built for this, but it could be one day/or already is with 3rd
party TAMs and extensions), and:
-- we can provide performance jump for seqconcurrentjobs or memory
fitting workloads (patchset does this already). Note: I think PG will
eventually get into such classes in the longer run, we are just ahead
with NUMA, but PG is without proper vectorized executor stuff.
-- we could further enhance PQ here: the leader and PQ workers would
stick to the same NUMA node with some affinity (the earlier thread
measurements for this  [1] -- we could have session GUC to enable this
for planned big PQ whole-NUMA SELECTs; this would be probably done
close to dsm_impl_posix())
- new idea: we could allow exposing tables(spaces) into NUMA nodes or
make it per-user toggle too while we are at it (imagine HTAP-like
workloads: NUMA node #0 for OLTP, node #1 for analytics). Sounds cool
and rather easy and has valid use, but dunno if that would be really
useful?

Way out of scope:
- superlocking btress that Andres mentioned on his presentation

-J.

[1] - https://www.postgresql.org/message-id/attachment/178120/NUMA_pq_cpu_pinning_results.txt

  79. Re: Adding basic NUMA awareness

    Tomas Vondra <tomas@vondra.me> — 2025-12-08T20:02:27Z

    Hi,

I've spent the last couple days considering what to do about this patch
series in this thread. The approach assumes partitioning shared memory
in a NUMA-aware way has enough benefits to justify the extra complexity.
I'm getting somewhat skeptical about that being a good trade off.

We've been able to demonstrate some significant benefits of the patches.
See for example the results from about a month ago [1], showing that in
some cases the throughput almost doubles.

But I'm skeptical about this for two reasons:

* Most of the benefit comes from patches unrelated to NUMA. The initial
patches partition clockweep, in a NUMA oblivious way. In fact, applying
the NUMA patches often *reduces* the throughput. So if we're concerned
about contention on the clocksweep hand, we could apply just these first
patches. That way we wouldn't have to deal with huge pages.

* Furthermore, I'm not quite sure clocksweep really is a bottleneck in
realistic cases. The benchmark used in this thread does many concurrent
sequential scans, on data that exceeds shared buffers / fits into RAM.
Perhaps that happens, but I doubt it's all that common.

I've been unable to demonstrate any benefits on other workloads, even if
there's a lot of buffer misses / reads into shared buffers. As soon as
the query starts doing something else, the clocksweep contention becomes
a non-issue. Consider for example read-only pgbench with database much
larger than shared buffers (but still within RAM). The cost of the index
scans (and other nodes) seems to reduce the pressure on clocksweep.

So I'm skeptical about clocksweep pressure being a serious issue, except
for some very narrow benchmarks (like the concurrent seqscan test). And
even if this happened for some realistic cases, partitioning the buffers
in a NUMA-oblivious way seems to do the trick.


When discussing this stuff off list, it was suggested this might help
with the scenario Andres presented in [3], where the throughput improves
a lot with multiple databases. I've not observed that in practice, and I
don't think these patches really can help with that. That scenario is
about buffer lock contention, not clocksweep contention.

With a single database there's nothing to do - there's one contended
page, located on a single node. There'll be contention, the no matter
which node it ends up on. With multiple databases (or multiple root
pages), it either happens to work by chance (if the buffers happen to be
from different nodes), or it would require figuring out which buffers
are busy, and place them on different nodes. But the patches did not
even try to do anything like that. So it still was a matter of chance.


That does not mean we don't need to worry about NUMA. There's still the
issue of misbalancing the allocation - with memory coming from just one
node, etc. Which is an issue because it "overloads" the memory subsystem
on that particular node. But that can be solved simply by interleaving
the shared memory segment.

That sidesteps most of the complexity - it does not need to figure out
how to partition buffers, does not need to worry about huge pages, etc.
This is not a new idea - it's more or less what Jakub Wartak initially
proposed in [2], before I hijacked the thread into the more complex
approach.

Attached is a tiny patch doing mostly what Jakub did, except that it
does two things. First, it allows interleaving the shared memory on all
relevant NUMA nodes (per numa_get_mems_allowed). Second, it allows
populating all memory by setting MAP_POPULATE in mmap(). There's a new
GUC to enable each of these.


I think we should try this (much simpler) approach first, or something
close to it. Sorry for dragging everyone into a much more complex
approach, which now seems to be a dead end.


regards

[1]
https://www.postgresql.org/message-id/e4d7e6fc-b5c5-4288-991c-56219db2edd5@vondra.me

[2]
https://www.postgresql.org/message-id/CAKZiRmw6i1W1AwXxa-Asrn8wrVcVH3TO715g_MCoowTS9rkGyw@mail.gmail.com

[3] https://youtu.be/V75KpACdl6E?t=2120
-- 
Tomas Vondra

  80. Re: Adding basic NUMA awareness

    Andres Freund <andres@anarazel.de> — 2026-01-10T01:42:36Z

    Hi,

On 2025-12-08 21:02:27 +0100, Tomas Vondra wrote:
> * Most of the benefit comes from patches unrelated to NUMA. The initial
> patches partition clockweep, in a NUMA oblivious way. In fact, applying
> the NUMA patches often *reduces* the throughput. So if we're concerned
> about contention on the clocksweep hand, we could apply just these first
> patches. That way we wouldn't have to deal with huge pages.

> * Furthermore, I'm not quite sure clocksweep really is a bottleneck in
> realistic cases. The benchmark used in this thread does many concurrent
> sequential scans, on data that exceeds shared buffers / fits into RAM.
> Perhaps that happens, but I doubt it's all that common.

I think this misses that this isn't necessarily about peak throughput under
concurrent contention.  Consider this scenario:

1) shared buffers is already allocated from a kernel POV, i.e. pages reside on
   some numa node instead of having to be allocated on the first access

2) one backend does a scan of scan of a relation [largely] not in shared
   buffers

Whether the buffers for the ringbuffer (if the relation is > NBuffers/4) or
for the entire relation (if smaller) is allocated on the same node as the
backend makes a quite substantial difference.  I see about a 25% difference
even on a small-ish numa system.

Partitioned clocksweep makes it vastly more likely that data is on the local
numa node.

If you simulate different locality modes with numactl, I can see pretty
drastic differences for the processing of individual queries, both with
parallel and non-parallel processing.


psql -Xq -c 'SELECT pg_buffercache_evict_all();' -c 'SELECT numa_node, sum(size) FROM pg_shmem_allocations_numa GROUP BY 1;' && perf stat --per-socket  -M memory_bandwidth_read,memory_bandwidth_write -a psql -c 'SELECT sum(abalance) FROM pgbench_accounts;'

membind 0, cpunodebind 1, max_parallel_workers_per_gather=0:
S0        6        341,635,792      UNC_M_CAS_COUNT.WR               #   4276.9 MB/s  memory_bandwidth_write
S0       20      5,116,381,542      duration_time
S0        6        255,977,795      UNC_M_CAS_COUNT.RD               #   3204.6 MB/s  memory_bandwidth_read
S0       20      5,116,391,355      duration_time
S1        6          2,418,579      UNC_M_CAS_COUNT.WR               #     30.3 MB/s  memory_bandwidth_write
S1        6        115,511,123      UNC_M_CAS_COUNT.RD               #   1446.1 MB/s  memory_bandwidth_read

       5.112286670 seconds time elapsed


membind 1, cpunodebind 1, max_parallel_workers_per_gather=0:
S0        6         16,528,154      UNC_M_CAS_COUNT.WR               #    248.1 MB/s  memory_bandwidth_write
S0       20      4,267,078,201      duration_time
S0        6         40,327,670      UNC_M_CAS_COUNT.RD               #    605.4 MB/s  memory_bandwidth_read
S0       20      4,267,088,762      duration_time
S1        6        116,925,559      UNC_M_CAS_COUNT.WR               #   1755.2 MB/s  memory_bandwidth_write
S1        6        244,251,242      UNC_M_CAS_COUNT.RD               #   3666.5 MB/s  memory_bandwidth_read

       4.263442844 seconds time elapsed


interleave 0,1, cpunodebind 1, max_parallel_workers_per_gather=0:

S0        6        196,713,044      UNC_M_CAS_COUNT.WR               #   2757.4 MB/s  memory_bandwidth_write
S0       20      4,569,805,767      duration_time
S0        6        167,497,804      UNC_M_CAS_COUNT.RD               #   2347.9 MB/s  memory_bandwidth_read
S0       20      4,569,816,439      duration_time
S1        6         81,992,696      UNC_M_CAS_COUNT.WR               #   1149.3 MB/s  memory_bandwidth_write
S1        6        192,265,269      UNC_M_CAS_COUNT.RD               #   2695.1 MB/s  memory_bandwidth_read

       4.565722468 seconds time elapsed


membind 0, cpunodebind 1, max_parallel_workers_per_gather=8:
S0        6        336,538,518      UNC_M_CAS_COUNT.WR               #  24130.2 MB/s  memory_bandwidth_write
S0       20        895,976,459      duration_time
S0        6        238,663,716      UNC_M_CAS_COUNT.RD               #  17112.4 MB/s  memory_bandwidth_read
S0       20        895,986,193      duration_time
S1        6          2,594,371      UNC_M_CAS_COUNT.WR               #    186.0 MB/s  memory_bandwidth_write
S1        6        113,981,673      UNC_M_CAS_COUNT.RD               #   8172.6 MB/s  memory_bandwidth_read

       0.892594989 seconds time elapsed


membind 1, cpunodebind 1, max_parallel_workers_per_gather=8:
S0        6          3,492,673      UNC_M_CAS_COUNT.WR               #    322.0 MB/s  memory_bandwidth_write
S0       20        698,175,650      duration_time
S0        6          5,363,152      UNC_M_CAS_COUNT.RD               #    494.4 MB/s  memory_bandwidth_read
S0       20        698,187,522      duration_time
S1        6        117,181,190      UNC_M_CAS_COUNT.WR               #  10802.4 MB/s  memory_bandwidth_write
S1        6        251,059,297      UNC_M_CAS_COUNT.RD               #  23144.0 MB/s  memory_bandwidth_read

       0.694253637 seconds time elapsed


interleave 0,1, cpunodebind 1, max_parallel_workers_per_gather=8:

S0        6        170,352,086      UNC_M_CAS_COUNT.WR               #  13767.3 MB/s  memory_bandwidth_write
S0       20        797,166,139      duration_time
S0        6        121,646,666      UNC_M_CAS_COUNT.RD               #   9831.1 MB/s  memory_bandwidth_read
S0       20        797,175,899      duration_time
S1        6         60,099,863      UNC_M_CAS_COUNT.WR               #   4857.1 MB/s  memory_bandwidth_write
S1        6        182,035,468      UNC_M_CAS_COUNT.RD               #  14711.5 MB/s  memory_bandwidth_read

       0.791915733 seconds time elapsed



You're never going to be quite as good when actually using both NUMA nodes,
but at least simple workloads like the above should be able to get a lot
closer to the good number from above than we currently are.



Maybe the problem is that the patchset doesn't actually quite work right now?
I checked out numa-20251111 and ran a query for a 1GB table in a 40GB s_b
system: there's not much more locality with debug_numa=buffers, than without
(roughly 55% on one node, 45% on the other). Making it not surprising that the
results aren't great.



> I've been unable to demonstrate any benefits on other workloads, even if
> there's a lot of buffer misses / reads into shared buffers. As soon as
> the query starts doing something else, the clocksweep contention becomes
> a non-issue. Consider for example read-only pgbench with database much
> larger than shared buffers (but still within RAM). The cost of the index
> scans (and other nodes) seems to reduce the pressure on clocksweep.
>
> So I'm skeptical about clocksweep pressure being a serious issue, except
> for some very narrow benchmarks (like the concurrent seqscan test). And
> even if this happened for some realistic cases, partitioning the buffers
> in a NUMA-oblivious way seems to do the trick.

I think you're over-indexing on the contention aspect and under-indexing on
the locality benefits.


> When discussing this stuff off list, it was suggested this might help
> with the scenario Andres presented in [3], where the throughput improves
> a lot with multiple databases. I've not observed that in practice, and I
> don't think these patches really can help with that. That scenario is
> about buffer lock contention, not clocksweep contention.

Buffer content and buffer headers being on your local node makes access
faster...


> Attached is a tiny patch doing mostly what Jakub did, except that it
> does two things. First, it allows interleaving the shared memory on all
> relevant NUMA nodes (per numa_get_mems_allowed). Second, it allows
> populating all memory by setting MAP_POPULATE in mmap(). There's a new
> GUC to enable each of these.

> I think we should try this (much simpler) approach first, or something
> close to it. Sorry for dragging everyone into a much more complex
> approach, which now seems to be a dead end.

I'm somewhat doubtful that interleaving is going to be good enough without
some awareness of which buffers to preferrably use. Additionally, without huge
pages, there are significant negative performance effects due to each buffer
being split across two numa nodes.

Greetings,

Andres Freund

  81. Re: Adding basic NUMA awareness

    Tomas Vondra <tomas@vondra.me> — 2026-01-12T23:58:49Z

    On 1/10/26 02:42, Andres Freund wrote:
> Hi,
> 
> On 2025-12-08 21:02:27 +0100, Tomas Vondra wrote:
>> * Most of the benefit comes from patches unrelated to NUMA. The initial
>> patches partition clockweep, in a NUMA oblivious way. In fact, applying
>> the NUMA patches often *reduces* the throughput. So if we're concerned
>> about contention on the clocksweep hand, we could apply just these first
>> patches. That way we wouldn't have to deal with huge pages.
> 
>> * Furthermore, I'm not quite sure clocksweep really is a bottleneck in
>> realistic cases. The benchmark used in this thread does many concurrent
>> sequential scans, on data that exceeds shared buffers / fits into RAM.
>> Perhaps that happens, but I doubt it's all that common.
> 
> I think this misses that this isn't necessarily about peak throughput under
> concurrent contention.  Consider this scenario:
> 
> 1) shared buffers is already allocated from a kernel POV, i.e. pages reside on
>    some numa node instead of having to be allocated on the first access
> 
> 2) one backend does a scan of scan of a relation [largely] not in shared
>    buffers
> 
> Whether the buffers for the ringbuffer (if the relation is > NBuffers/4) or
> for the entire relation (if smaller) is allocated on the same node as the
> backend makes a quite substantial difference.  I see about a 25% difference
> even on a small-ish numa system.
> 
> Partitioned clocksweep makes it vastly more likely that data is on the local
> numa node.
> 
> If you simulate different locality modes with numactl, I can see pretty
> drastic differences for the processing of individual queries, both with
> parallel and non-parallel processing.
> 
> 
> psql -Xq -c 'SELECT pg_buffercache_evict_all();' -c 'SELECT numa_node, sum(size) FROM pg_shmem_allocations_numa GROUP BY 1;' && perf stat --per-socket  -M memory_bandwidth_read,memory_bandwidth_write -a psql -c 'SELECT sum(abalance) FROM pgbench_accounts;'
> 
> membind 0, cpunodebind 1, max_parallel_workers_per_gather=0:
> S0        6        341,635,792      UNC_M_CAS_COUNT.WR               #   4276.9 MB/s  memory_bandwidth_write
> S0       20      5,116,381,542      duration_time
> S0        6        255,977,795      UNC_M_CAS_COUNT.RD               #   3204.6 MB/s  memory_bandwidth_read
> S0       20      5,116,391,355      duration_time
> S1        6          2,418,579      UNC_M_CAS_COUNT.WR               #     30.3 MB/s  memory_bandwidth_write
> S1        6        115,511,123      UNC_M_CAS_COUNT.RD               #   1446.1 MB/s  memory_bandwidth_read
> 
>        5.112286670 seconds time elapsed
> 
> 
> membind 1, cpunodebind 1, max_parallel_workers_per_gather=0:
> S0        6         16,528,154      UNC_M_CAS_COUNT.WR               #    248.1 MB/s  memory_bandwidth_write
> S0       20      4,267,078,201      duration_time
> S0        6         40,327,670      UNC_M_CAS_COUNT.RD               #    605.4 MB/s  memory_bandwidth_read
> S0       20      4,267,088,762      duration_time
> S1        6        116,925,559      UNC_M_CAS_COUNT.WR               #   1755.2 MB/s  memory_bandwidth_write
> S1        6        244,251,242      UNC_M_CAS_COUNT.RD               #   3666.5 MB/s  memory_bandwidth_read
> 
>        4.263442844 seconds time elapsed
> 
> 
> interleave 0,1, cpunodebind 1, max_parallel_workers_per_gather=0:
> 
> S0        6        196,713,044      UNC_M_CAS_COUNT.WR               #   2757.4 MB/s  memory_bandwidth_write
> S0       20      4,569,805,767      duration_time
> S0        6        167,497,804      UNC_M_CAS_COUNT.RD               #   2347.9 MB/s  memory_bandwidth_read
> S0       20      4,569,816,439      duration_time
> S1        6         81,992,696      UNC_M_CAS_COUNT.WR               #   1149.3 MB/s  memory_bandwidth_write
> S1        6        192,265,269      UNC_M_CAS_COUNT.RD               #   2695.1 MB/s  memory_bandwidth_read
> 
>        4.565722468 seconds time elapsed
> 
> 
> membind 0, cpunodebind 1, max_parallel_workers_per_gather=8:
> S0        6        336,538,518      UNC_M_CAS_COUNT.WR               #  24130.2 MB/s  memory_bandwidth_write
> S0       20        895,976,459      duration_time
> S0        6        238,663,716      UNC_M_CAS_COUNT.RD               #  17112.4 MB/s  memory_bandwidth_read
> S0       20        895,986,193      duration_time
> S1        6          2,594,371      UNC_M_CAS_COUNT.WR               #    186.0 MB/s  memory_bandwidth_write
> S1        6        113,981,673      UNC_M_CAS_COUNT.RD               #   8172.6 MB/s  memory_bandwidth_read
> 
>        0.892594989 seconds time elapsed
> 
> 
> membind 1, cpunodebind 1, max_parallel_workers_per_gather=8:
> S0        6          3,492,673      UNC_M_CAS_COUNT.WR               #    322.0 MB/s  memory_bandwidth_write
> S0       20        698,175,650      duration_time
> S0        6          5,363,152      UNC_M_CAS_COUNT.RD               #    494.4 MB/s  memory_bandwidth_read
> S0       20        698,187,522      duration_time
> S1        6        117,181,190      UNC_M_CAS_COUNT.WR               #  10802.4 MB/s  memory_bandwidth_write
> S1        6        251,059,297      UNC_M_CAS_COUNT.RD               #  23144.0 MB/s  memory_bandwidth_read
> 
>        0.694253637 seconds time elapsed
> 
> 
> interleave 0,1, cpunodebind 1, max_parallel_workers_per_gather=8:
> 
> S0        6        170,352,086      UNC_M_CAS_COUNT.WR               #  13767.3 MB/s  memory_bandwidth_write
> S0       20        797,166,139      duration_time
> S0        6        121,646,666      UNC_M_CAS_COUNT.RD               #   9831.1 MB/s  memory_bandwidth_read
> S0       20        797,175,899      duration_time
> S1        6         60,099,863      UNC_M_CAS_COUNT.WR               #   4857.1 MB/s  memory_bandwidth_write
> S1        6        182,035,468      UNC_M_CAS_COUNT.RD               #  14711.5 MB/s  memory_bandwidth_read
> 
>        0.791915733 seconds time elapsed
> 
> 
> 
> You're never going to be quite as good when actually using both NUMA nodes,
> but at least simple workloads like the above should be able to get a lot
> closer to the good number from above than we currently are.
> 

I see no such improvements, unfortunately. Even when I explicitly pin
memory and cpus to different nodes using numactl. Consider a simple
experiment, starting an instance either like this:

numactl --membind=0 --cpunodebind=0 pg_ctl -D /mnt/data/data-numa start

or like this

numactl --membind=0 --cpunodebind=1 pg_ctl -D /mnt/data/data-numa start

on a 2-node NUMA cluster. To the best of my knowledge this means that
either both the memory and all pg processes (including the backend) are
on node 0, of memory is on node 0 and backend is on node 1.

And then I initialized pgbench with scale that is much larger than
shared buffers, but fits into RAM. So cached, but definitely > NB/4. And
then I ran

  select * from pgbench_accounts offset 1000000000;

which does a sequential scan with the circular buffer you mention abobe

I've made all reasonable precautions to stabilize the results, like
enabling huge pages (both for shared memory and binaries), disabling
checksums, ... And I ran that on an Azure instance D96v6 with EPYC 9V74.
This was with scale 10000 (~150GB), shared_buffers=8GB.

And I get this:

worker / 32
-----------

numactl --membind=0 --cpunodebind=0 pg_ctl ...
Time: 26280.437 ms (00:26.280)
Time: 26177.165 ms (00:26.177)
Time: 26182.222 ms (00:26.182)
Time: 26174.421 ms (00:26.174)
Time: 26216.989 ms (00:26.217)

numactl --membind=0 --cpunodebind=1 pg_ctl ...
Time: 26412.878 ms (00:26.413)
Time: 26413.332 ms (00:26.413)
Time: 26202.899 ms (00:26.203)
Time: 26412.627 ms (00:26.413)
Time: 26484.962 ms (00:26.485)

io_uring
--------

numactl --membind=0 --cpunodebind=0 pg_ctl ...
Time: 26286.977 ms (00:26.287)
Time: 26499.830 ms (00:26.500)
Time: 26629.990 ms (00:26.630)
Time: 26443.147 ms (00:26.443)

numactl --membind=0 --cpunodebind=1 pg_ctl ...
Time: 26727.655 ms (00:26.728)
Time: 26787.456 ms (00:26.787)
Time: 26484.260 ms (00:26.484)
Time: 26250.737 ms (00:26.251)
Time: 26208.913 ms (00:26.209)

I don't see any difference. To rule out any virtualization weirdness, I
did the same experiment on my old Xeon machine (also 2-node NUMA), just
with a smaller scale (2000) and shared_buffers=4GB. And that gave me:


xeon scale=2000 nochecksums

worker / 32
-----------

numactl --membind=0 --cpunodebind=0 pg_ctl ...
Time: 5519.728 ms (00:05.520)
Time: 5570.215 ms (00:05.570)
Time: 5568.233 ms (00:05.568)
Time: 5556.465 ms (00:05.556)
Time: 5517.420 ms (00:05.517)

numactl --membind=0 --cpunodebind=1 pg_ctl ...
Time: 5639.281 ms (00:05.639)
Time: 5657.822 ms (00:05.658)
Time: 5653.077 ms (00:05.653)
Time: 5647.780 ms (00:05.648)
Time: 5647.288 ms (00:05.647)

io_uring
--------

numactl --membind=0 --cpunodebind=0 pg_ctl ...
Time: 7517.920 ms (00:07.518)
Time: 7180.628 ms (00:07.181)
Time: 7162.801 ms (00:07.163)
Time: 7164.827 ms (00:07.165)
Time: 7177.757 ms (00:07.178)

numactl --membind=0 --cpunodebind=1 pg_ctl ...
Time: 7622.372 ms (00:07.622)
Time: 7571.923 ms (00:07.572)
Time: 7571.966 ms (00:07.572)
Time: 7568.269 ms (00:07.568)
Time: 7558.195 ms (00:07.558)

If I squint a little bit, there's difference for io_uring. But it's not
even 5%, definitely not 25%.

> 
> 
> Maybe the problem is that the patchset doesn't actually quite work right now?
> I checked out numa-20251111 and ran a query for a 1GB table in a 40GB s_b
> system: there's not much more locality with debug_numa=buffers, than without
> (roughly 55% on one node, 45% on the other). Making it not surprising that the
> results aren't great.
> 

Hard to say, but I'd guess that's because of the clocksweep balancing.
Which ensures that we don't overload a single NUMA node. Imagine an
instance with a single connection - it can't allocate from a single NUMA
node, because that'd mean it'll only ever use 50% of available cache.
Which does not seem great. Maybe there's a better way to address this.

> 
> 
>> I've been unable to demonstrate any benefits on other workloads, even if
>> there's a lot of buffer misses / reads into shared buffers. As soon as
>> the query starts doing something else, the clocksweep contention becomes
>> a non-issue. Consider for example read-only pgbench with database much
>> larger than shared buffers (but still within RAM). The cost of the index
>> scans (and other nodes) seems to reduce the pressure on clocksweep.
>>
>> So I'm skeptical about clocksweep pressure being a serious issue, except
>> for some very narrow benchmarks (like the concurrent seqscan test). And
>> even if this happened for some realistic cases, partitioning the buffers
>> in a NUMA-oblivious way seems to do the trick.
> 
> I think you're over-indexing on the contention aspect and under-indexing on
> the locality benefits.
> 

I've been unable to demonstrate meaningful benefits of locality (like in
the example above), while I've been able to show benefits of reducing
the clocksweep contention. It's entirely possible I'm doing it wrong or
missing something, of course.

> 
>> When discussing this stuff off list, it was suggested this might help
>> with the scenario Andres presented in [3], where the throughput improves
>> a lot with multiple databases. I've not observed that in practice, and I
>> don't think these patches really can help with that. That scenario is
>> about buffer lock contention, not clocksweep contention.
> 
> Buffer content and buffer headers being on your local node makes access
> faster...
> 

That was my expectation too, but I haven't seen meaningful improvements
in any benchmark.

For example in the benchmark I presented earlier, all the memory is on
node 0 (so both headers and buffers). And there does not seem to be any
measurable difference when accessing it from node 0 vs. node 1. So why
would it matter than header may be on node 0 and buffer on node 1?

> 
>> Attached is a tiny patch doing mostly what Jakub did, except that it
>> does two things. First, it allows interleaving the shared memory on all
>> relevant NUMA nodes (per numa_get_mems_allowed). Second, it allows
>> populating all memory by setting MAP_POPULATE in mmap(). There's a new
>> GUC to enable each of these.
> 
>> I think we should try this (much simpler) approach first, or something
>> close to it. Sorry for dragging everyone into a much more complex
>> approach, which now seems to be a dead end.
> 
> I'm somewhat doubtful that interleaving is going to be good enough without
> some awareness of which buffers to preferrably use. Additionally, without huge
> pages, there are significant negative performance effects due to each buffer
> being split across two numa nodes.
> 

I'm rather skeptical this being worth it without huge pages. If you're
trying to get the best performance on a NUMA machine (with is likely big
with a lot of RAM), then huge pages are a huge improvement on their own.

I'd even say this NUMA stuff might/should require huge_pages=on.


-- 
Tomas Vondra

  82. Re: Adding basic NUMA awareness

    Andres Freund <andres@anarazel.de> — 2026-01-13T00:10:00Z

    Hi,

On 2026-01-13 00:58:49 +0100, Tomas Vondra wrote:
> On 1/10/26 02:42, Andres Freund wrote:
> > psql -Xq -c 'SELECT pg_buffercache_evict_all();' -c 'SELECT numa_node, sum(size) FROM pg_shmem_allocations_numa GROUP BY 1;' && perf stat --per-socket  -M memory_bandwidth_read,memory_bandwidth_write -a psql -c 'SELECT sum(abalance) FROM pgbench_accounts;'

> And then I initialized pgbench with scale that is much larger than
> shared buffers, but fits into RAM. So cached, but definitely > NB/4. And
> then I ran
> 
>   select * from pgbench_accounts offset 1000000000;
> 
> which does a sequential scan with the circular buffer you mention abobe

Did you try it with the query I suggested? One plausible reason why you did
not see an effect with your query is that with a huge offset you actually
never deform the tuple, which is an important and rather latency sensitive
path.

Greetings,

Andres Freund

  83. Re: Adding basic NUMA awareness

    Andres Freund <andres@anarazel.de> — 2026-01-13T00:24:34Z

    Hi,

On 2026-01-12 19:10:00 -0500, Andres Freund wrote:
> On 2026-01-13 00:58:49 +0100, Tomas Vondra wrote:
> > On 1/10/26 02:42, Andres Freund wrote:
> > > psql -Xq -c 'SELECT pg_buffercache_evict_all();' -c 'SELECT numa_node, sum(size) FROM pg_shmem_allocations_numa GROUP BY 1;' && perf stat --per-socket  -M memory_bandwidth_read,memory_bandwidth_write -a psql -c 'SELECT sum(abalance) FROM pgbench_accounts;'
>
> > And then I initialized pgbench with scale that is much larger than
> > shared buffers, but fits into RAM. So cached, but definitely > NB/4. And
> > then I ran
> >
> >   select * from pgbench_accounts offset 1000000000;
> >
> > which does a sequential scan with the circular buffer you mention abobe
>
> Did you try it with the query I suggested? One plausible reason why you did
> not see an effect with your query is that with a huge offset you actually
> never deform the tuple, which is an important and rather latency sensitive
> path.

Btw, this doesn't need anywhere close to as much data, it should be visible as
soon as you're >> L3.

To show why
  SELECT * FROM pgbench_accounts OFFSET 100000000
doesn't show an effect but
  SELECT sum(abalance) FROM pgbench_accounts;

does, just look at the difference using the perf command I posted. Here on a
scale 200.


numactl --membind 0 --cpunodebind 0

offset:

S0        6         47,138,135      UNC_M_CAS_COUNT.WR               #   3884.1 MB/s  memory_bandwidth_write
S0       20        780,343,577      duration_time
S0        6         61,685,331      UNC_M_CAS_COUNT.RD               #   5082.8 MB/s  memory_bandwidth_read
S0       20        780,353,818      duration_time
S1        6          1,238,568      UNC_M_CAS_COUNT.WR               #    102.1 MB/s  memory_bandwidth_write
S1        6          1,475,224      UNC_M_CAS_COUNT.RD               #    121.6 MB/s  memory_bandwidth_read

       0.776715450 seconds time elapsed


agg:

S0        6         53,145,706      UNC_M_CAS_COUNT.WR               #   2000.8 MB/s  memory_bandwidth_write
S0       20      1,706,046,493      duration_time
S0        6        111,390,488      UNC_M_CAS_COUNT.RD               #   4193.5 MB/s  memory_bandwidth_read
S0       20      1,706,057,341      duration_time
S1        6          3,968,454      UNC_M_CAS_COUNT.WR               #    149.4 MB/s  memory_bandwidth_write
S1        6          4,026,212      UNC_M_CAS_COUNT.RD               #    151.6 MB/s  memory_bandwidth_read



numactl --membind 0 --cpunodebind 1

offset:

S0        6         91,982,003      UNC_M_CAS_COUNT.WR               #   7036.4 MB/s  memory_bandwidth_write
S0       20        842,785,290      duration_time
S0        6        113,076,316      UNC_M_CAS_COUNT.RD               #   8650.1 MB/s  memory_bandwidth_read
S0       20        842,797,430      duration_time
S1        6          1,545,612      UNC_M_CAS_COUNT.WR               #    118.2 MB/s  memory_bandwidth_write
S1        6          2,354,087      UNC_M_CAS_COUNT.RD               #    180.1 MB/s  memory_bandwidth_read

       0.836623794 seconds time elapsed

agg:

S0        6        133,267,754      UNC_M_CAS_COUNT.WR               #   3980.9 MB/s  memory_bandwidth_write
S0       20      2,146,221,284      duration_time
S0        6        159,951,549      UNC_M_CAS_COUNT.RD               #   4777.9 MB/s  memory_bandwidth_read
S0       20      2,146,233,675      duration_time
S1        6         71,543,708      UNC_M_CAS_COUNT.WR               #   2137.1 MB/s  memory_bandwidth_write
S1        6         49,584,957      UNC_M_CAS_COUNT.RD               #   1481.2 MB/s  memory_bandwidth_read

       2.142535432 seconds time elapsed


Note how much bigger the absolute numbers of reads and writes are for the
aggregate compared to the offset.

Interestingly I do see a performance difference, albeit a smaller one, even
with OFFSET. I see similar numbers on two different 2 socket machines.

Greetings,

Andres Freund

  84. Re: Adding basic NUMA awareness

    Tomas Vondra <tomas@vondra.me> — 2026-01-13T00:51:09Z

    
On 1/13/26 01:10, Andres Freund wrote:
> Hi,
> 
> On 2026-01-13 00:58:49 +0100, Tomas Vondra wrote:
>> On 1/10/26 02:42, Andres Freund wrote:
>>> psql -Xq -c 'SELECT pg_buffercache_evict_all();' -c 'SELECT numa_node, sum(size) FROM pg_shmem_allocations_numa GROUP BY 1;' && perf stat --per-socket  -M memory_bandwidth_read,memory_bandwidth_write -a psql -c 'SELECT sum(abalance) FROM pgbench_accounts;'
> 
>> And then I initialized pgbench with scale that is much larger than
>> shared buffers, but fits into RAM. So cached, but definitely > NB/4. And
>> then I ran
>>
>>   select * from pgbench_accounts offset 1000000000;
>>
>> which does a sequential scan with the circular buffer you mention abobe
> 
> Did you try it with the query I suggested? One plausible reason why you did
> not see an effect with your query is that with a huge offset you actually
> never deform the tuple, which is an important and rather latency sensitive
> path.
> 

I did try with the agg query too, and there's still no difference on
either machine.


I can't do the perf on the Azure VM, because the Ubuntu is image is
borked and does not allow installing the package. But on my xeon I can
do the perf, and that gives me this:

numactl --membind=0 --cpunodebind=0 ~/builds/master-test/bin/pg_ctl
-----------------------------------------------------------------------
S0 1     24,677,226    UNC_M_CAS_COUNT.WR #    79.0 MB/s ... idth_write
S0 1 20,001,829,522 ns duration_time                     ...
S0 1    972,631,426    UNC_M_CAS_COUNT.RD #  3112.2 MB/s ... idth_read
S0 1 20,001,822,807 ns duration_time                     ...
S1 1     15,602,233    UNC_M_CAS_COUNT.WR #    49.9 MB/s ... idth_write
S1 1    712,431,146    UNC_M_CAS_COUNT.RD #  2279.6 MB/s ... idth_read


numactl --membind=0 --cpunodebind=1 ~/builds/master-test/bin/pg_ctl
-----------------------------------------------------------------------
S0 1     47,931,019    UNC_M_CAS_COUNT.WR #    153.4 MB/s ... idth_write
S0 1 20,002,933,380 ns duration_time                      ...
S0 1  1,007,386,994    UNC_M_CAS_COUNT.RD #   3223.2 MB/s ... idth_read
S0 1 20,002,927,341 ns duration_time                      ...
S1 1     10,310,201    UNC_M_CAS_COUNT.WR #     33.0 MB/s ... idth_write
S1 1    714,826,668    UNC_M_CAS_COUNT.RD #   2287.2 MB/s ... idth_read

so there is a little bit of a difference for some stats, but not much.


FWIW this is from

perf stat --per-socket  -M memory_bandwidth_read,memory_bandwidth_write
-a -- sleep 20

while the agg query runs in a loop.


cheers
-- 
Tomas Vondra

  85. Re: Adding basic NUMA awareness

    Andres Freund <andres@anarazel.de> — 2026-01-13T01:08:31Z

    Hi,

On 2026-01-13 01:51:09 +0100, Tomas Vondra wrote:
> On 1/13/26 01:10, Andres Freund wrote:
> > Hi,
> > 
> > On 2026-01-13 00:58:49 +0100, Tomas Vondra wrote:
> >> On 1/10/26 02:42, Andres Freund wrote:
> >>> psql -Xq -c 'SELECT pg_buffercache_evict_all();' -c 'SELECT numa_node, sum(size) FROM pg_shmem_allocations_numa GROUP BY 1;' && perf stat --per-socket  -M memory_bandwidth_read,memory_bandwidth_write -a psql -c 'SELECT sum(abalance) FROM pgbench_accounts;'
> > 
> >> And then I initialized pgbench with scale that is much larger than
> >> shared buffers, but fits into RAM. So cached, but definitely > NB/4. And
> >> then I ran
> >>
> >>   select * from pgbench_accounts offset 1000000000;
> >>
> >> which does a sequential scan with the circular buffer you mention abobe
> > 
> > Did you try it with the query I suggested? One plausible reason why you did
> > not see an effect with your query is that with a huge offset you actually
> > never deform the tuple, which is an important and rather latency sensitive
> > path.
> > 
> 
> I did try with the agg query too, and there's still no difference on
> either machine.

Could you provide numactl --hardware for both?  There may be more than two
numa nodes on a system with 2 sockets, due to one socket being split into two
- in which case the latency between 0,1 might be a lot lower than say 0 and 3.


> I can't do the perf on the Azure VM, because the Ubuntu is image is
> borked and does not allow installing the package. But on my xeon I can
> do the perf, and that gives me this:
> 
> numactl --membind=0 --cpunodebind=0 ~/builds/master-test/bin/pg_ctl
> -----------------------------------------------------------------------
> S0 1     24,677,226    UNC_M_CAS_COUNT.WR #    79.0 MB/s ... idth_write
> S0 1 20,001,829,522 ns duration_time                     ...
> S0 1    972,631,426    UNC_M_CAS_COUNT.RD #  3112.2 MB/s ... idth_read
> S0 1 20,001,822,807 ns duration_time                     ...
> S1 1     15,602,233    UNC_M_CAS_COUNT.WR #    49.9 MB/s ... idth_write
> S1 1    712,431,146    UNC_M_CAS_COUNT.RD #  2279.6 MB/s ... idth_read
> 
> 
> numactl --membind=0 --cpunodebind=1 ~/builds/master-test/bin/pg_ctl
> -----------------------------------------------------------------------
> S0 1     47,931,019    UNC_M_CAS_COUNT.WR #    153.4 MB/s ... idth_write
> S0 1 20,002,933,380 ns duration_time                      ...
> S0 1  1,007,386,994    UNC_M_CAS_COUNT.RD #   3223.2 MB/s ... idth_read
> S0 1 20,002,927,341 ns duration_time                      ...
> S1 1     10,310,201    UNC_M_CAS_COUNT.WR #     33.0 MB/s ... idth_write
> S1 1    714,826,668    UNC_M_CAS_COUNT.RD #   2287.2 MB/s ... idth_read
> 
> so there is a little bit of a difference for some stats, but not much.
> 
> 
> FWIW this is from
> 
> perf stat --per-socket  -M memory_bandwidth_read,memory_bandwidth_write
> -a -- sleep 20
> 
> while the agg query runs in a loop.

FWIW doing one perf stat for each execution is preferrable for comparison,
because otherwise you can hide large differences in total number of memory
accesses if the runtimes for the queries in the two "numa configurations" are
different.

Greetings,

Andres Freund

  86. Re: Adding basic NUMA awareness

    Tomas Vondra <tomas@vondra.me> — 2026-01-13T01:13:40Z

    On 1/13/26 01:24, Andres Freund wrote:
> Hi,
> 
> On 2026-01-12 19:10:00 -0500, Andres Freund wrote:
>> On 2026-01-13 00:58:49 +0100, Tomas Vondra wrote:
>>> On 1/10/26 02:42, Andres Freund wrote:
>>>> psql -Xq -c 'SELECT pg_buffercache_evict_all();' -c 'SELECT numa_node, sum(size) FROM pg_shmem_allocations_numa GROUP BY 1;' && perf stat --per-socket  -M memory_bandwidth_read,memory_bandwidth_write -a psql -c 'SELECT sum(abalance) FROM pgbench_accounts;'
>>
>>> And then I initialized pgbench with scale that is much larger than
>>> shared buffers, but fits into RAM. So cached, but definitely > NB/4. And
>>> then I ran
>>>
>>>   select * from pgbench_accounts offset 1000000000;
>>>
>>> which does a sequential scan with the circular buffer you mention abobe
>>
>> Did you try it with the query I suggested? One plausible reason why you did
>> not see an effect with your query is that with a huge offset you actually
>> never deform the tuple, which is an important and rather latency sensitive
>> path.
> 
> Btw, this doesn't need anywhere close to as much data, it should be visible as
> soon as you're >> L3.
> 
> To show why
>   SELECT * FROM pgbench_accounts OFFSET 100000000
> doesn't show an effect but
>   SELECT sum(abalance) FROM pgbench_accounts;
> 
> does, just look at the difference using the perf command I posted. Here on a
> scale 200.
> 

OK, I tried with smaller scale (and larger shared buffers, to make the
data set smaller than NBuffers/4).

On the azure VM (scale 200, 32GB sb), there's still no difference:

numactl --membind 0 --cpunodebind 0
297.770 ms

numactl --membind 0 --cpunodebind 1
297.924 ms


and on xeon (scale 100, 8GB sb), there's a bit of a difference:

numactl --membind 0 --cpunodebind 0
236.451 ms

numactl --membind 0 --cpunodebind 1
298.418 ms

So roughly 20%. There's also a bigger difference in the perf, about
5944.3 MB/s vs. 5202.3 MB/s.

> 
> Interestingly I do see a performance difference, albeit a smaller one, even
> with OFFSET. I see similar numbers on two different 2 socket machines.
> 

I wonder how significant is the number of sockets. The Azure is a single
socket with 2 NUMA nodes, so maybe the latency differences are not
significant enough to affect this kind of tests.

The xeon is a 2-socket machine, but it's also older (~10y).


regards

-- 
Tomas Vondra

  87. Re: Adding basic NUMA awareness

    Tomas Vondra <tomas@vondra.me> — 2026-01-13T01:25:57Z

    
On 1/13/26 02:08, Andres Freund wrote:
> Hi,
> 
> On 2026-01-13 01:51:09 +0100, Tomas Vondra wrote:
>> On 1/13/26 01:10, Andres Freund wrote:
>>> Hi,
>>>
>>> On 2026-01-13 00:58:49 +0100, Tomas Vondra wrote:
>>>> On 1/10/26 02:42, Andres Freund wrote:
>>>>> psql -Xq -c 'SELECT pg_buffercache_evict_all();' -c 'SELECT numa_node, sum(size) FROM pg_shmem_allocations_numa GROUP BY 1;' && perf stat --per-socket  -M memory_bandwidth_read,memory_bandwidth_write -a psql -c 'SELECT sum(abalance) FROM pgbench_accounts;'
>>>
>>>> And then I initialized pgbench with scale that is much larger than
>>>> shared buffers, but fits into RAM. So cached, but definitely > NB/4. And
>>>> then I ran
>>>>
>>>>   select * from pgbench_accounts offset 1000000000;
>>>>
>>>> which does a sequential scan with the circular buffer you mention abobe
>>>
>>> Did you try it with the query I suggested? One plausible reason why you did
>>> not see an effect with your query is that with a huge offset you actually
>>> never deform the tuple, which is an important and rather latency sensitive
>>> path.
>>>
>>
>> I did try with the agg query too, and there's still no difference on
>> either machine.
> 
> Could you provide numactl --hardware for both?  There may be more than two
> numa nodes on a system with 2 sockets, due to one socket being split into two
> - in which case the latency between 0,1 might be a lot lower than say 0 and 3.
> 

xeon:

available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 44
45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65
node 0 size: 32066 MB
node 0 free: 13081 MB
node 1 cpus: 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41
42 43 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87
node 1 size: 32210 MB
node 1 free: 17764 MB
node distances:
node     0    1
   0:   10   21
   1:   21   10

azure/epyc:

available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47
node 0 size: 193412 MB
node 0 free: 147949 MB
node 1 cpus: 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67
68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91
92 93 94 95
node 1 size: 193513 MB
node 1 free: 151577 MB
node distances:
node   0   1
  0:  10  11
  1:  11  10


> 
>> I can't do the perf on the Azure VM, because the Ubuntu is image is
>> borked and does not allow installing the package. But on my xeon I can
>> do the perf, and that gives me this:
>>
>> numactl --membind=0 --cpunodebind=0 ~/builds/master-test/bin/pg_ctl
>> -----------------------------------------------------------------------
>> S0 1     24,677,226    UNC_M_CAS_COUNT.WR #    79.0 MB/s ... idth_write
>> S0 1 20,001,829,522 ns duration_time                     ...
>> S0 1    972,631,426    UNC_M_CAS_COUNT.RD #  3112.2 MB/s ... idth_read
>> S0 1 20,001,822,807 ns duration_time                     ...
>> S1 1     15,602,233    UNC_M_CAS_COUNT.WR #    49.9 MB/s ... idth_write
>> S1 1    712,431,146    UNC_M_CAS_COUNT.RD #  2279.6 MB/s ... idth_read
>>
>>
>> numactl --membind=0 --cpunodebind=1 ~/builds/master-test/bin/pg_ctl
>> -----------------------------------------------------------------------
>> S0 1     47,931,019    UNC_M_CAS_COUNT.WR #    153.4 MB/s ... idth_write
>> S0 1 20,002,933,380 ns duration_time                      ...
>> S0 1  1,007,386,994    UNC_M_CAS_COUNT.RD #   3223.2 MB/s ... idth_read
>> S0 1 20,002,927,341 ns duration_time                      ...
>> S1 1     10,310,201    UNC_M_CAS_COUNT.WR #     33.0 MB/s ... idth_write
>> S1 1    714,826,668    UNC_M_CAS_COUNT.RD #   2287.2 MB/s ... idth_read
>>
>> so there is a little bit of a difference for some stats, but not much.
>>
>>
>> FWIW this is from
>>
>> perf stat --per-socket  -M memory_bandwidth_read,memory_bandwidth_write
>> -a -- sleep 20
>>
>> while the agg query runs in a loop.
> 
> FWIW doing one perf stat for each execution is preferrable for comparison,
> because otherwise you can hide large differences in total number of memory
> accesses if the runtimes for the queries in the two "numa configurations" are
> different.
> 

Good point, I'll do that next time. But in this case they are not all
that different, I think.

regards

-- 
Tomas Vondra

  88. Re: Adding basic NUMA awareness

    Jakub Wartak <jakub.wartak@enterprisedb.com> — 2026-01-13T13:26:37Z

    On Tue, Jan 13, 2026 at 2:13 AM Tomas Vondra <tomas@vondra.me> wrote:
>
> On 1/13/26 01:24, Andres Freund wrote:
> > Hi,
> >
> > On 2026-01-12 19:10:00 -0500, Andres Freund wrote:
> >> On 2026-01-13 00:58:49 +0100, Tomas Vondra wrote:
> >>> On 1/10/26 02:42, Andres Freund wrote:
> >>>> psql -Xq -c 'SELECT pg_buffercache_evict_all();' -c 'SELECT numa_node, sum(size) FROM pg_shmem_allocations_numa GROUP BY 1;' && perf stat --per-socket  -M memory_bandwidth_read,memory_bandwidth_write -a psql -c 'SELECT sum(abalance) FROM pgbench_accounts;'
> >>
> >>> And then I initialized pgbench with scale that is much larger than
> >>> shared buffers, but fits into RAM. So cached, but definitely > NB/4. And
> >>> then I ran
> >>>
> >>>   select * from pgbench_accounts offset 1000000000;
> >>>
> >>> which does a sequential scan with the circular buffer you mention abobe
> >>
> >> Did you try it with the query I suggested? One plausible reason why you did
> >> not see an effect with your query is that with a huge offset you actually
> >> never deform the tuple, which is an important and rather latency sensitive
> >> path.
> >
> > Btw, this doesn't need anywhere close to as much data, it should be visible as
> > soon as you're >> L3.
> >
> > To show why
> >   SELECT * FROM pgbench_accounts OFFSET 100000000
> > doesn't show an effect but
> >   SELECT sum(abalance) FROM pgbench_accounts;
> >
> > does, just look at the difference using the perf command I posted. Here on a
> > scale 200.
> >
>
> OK, I tried with smaller scale (and larger shared buffers, to make the
> data set smaller than NBuffers/4).
>
> On the azure VM (scale 200, 32GB sb), there's still no difference:
>
[..]
> and on xeon (scale 100, 8GB sb), there's a bit of a difference:
>
[..]
>
> So roughly 20%. There's also a bigger difference in the perf, about
> 5944.3 MB/s vs. 5202.3 MB/s.
>
> >
> > Interestingly I do see a performance difference, albeit a smaller one, even
> > with OFFSET. I see similar numbers on two different 2 socket machines.
> >
>
> I wonder how significant is the number of sockets. The Azure is a single
> socket with 2 NUMA nodes, so maybe the latency differences are not
> significant enough to affect this kind of tests.
>
> The xeon is a 2-socket machine, but it's also older (~10y).

My 0.02$ from intentionally having even more slow hardware with even
more sockets where some effects are even more visible (maybe this
helps the $thread)

There are two ways we could benefit from NUMA multi-socket systems:
a) potentially getting lower latency
b) avoid hitting interconnect max bandwidth limitations

- so it started with offline discussion yesterday that earlier we have
seen some yield from pgbench -S results, however I could not reproduce
the yield from [1] NUMAv2 -- I was trying to replicate that result "I
did some quick perf testing on my old xeon machine (2 NUMA nodes), and
the results are encouraging. For a read-only pgbench (2x shared
buffers, within RAM), I saw an increase from 1.1M tps to 1.3M." [1]. I
was using numa_buffers_interleave=on,numa_localalloc=on,numa_partition_freelist=node,numa_procs_interleave=on,numa_procs_pin=off,
no HPs (due to "Bad address" bug in that patchset), two -i pgbench
scales (2000, 500), with and without checksums, with/without -M
prepared. More or less it's always like that:
  jul17__64__off_pgbench.log:tps = 564153.278183 (without initial
connection time)
  numav2__64__off_pgbench.log:tps = 562068.655263 (without initial
connection time)
 so that probably rules out that we have recently introduced a bug
into the patchset (but that may mean that 1.1->1.3M boost was
something else?)

- so per above and in my opinion, both on master or all patchsets
here, classic OLTP pgbench (-S) is way too CPU computation heavy even
with -M prepared to see NUMA latency effects. Affects *both* NUMAv2
(Aug2025)[1] and Nov 2025 patchset versions. Even getting 0.5M tps on
pgbench scale -i 500 ends up using way less traffic before hitting the
QPI (interconnect) max link bandwidth (just 3.8-4.2 GB/s in my case,
but pgbench -S can consume just max 1.2GB/s assuming proper
interleaving).

- the single seqscan "SELECT sum(abalance) FROM pgbench_accounts;"
problem (or lack of it -- with single session) is that with standard
master, you may end up having data on just a single NUMA node. If that
system is idle and running just 1 instance of this query, I'm getting
like ~750MB/s of memory reads from the socket where the data is
located (again , much below the limit of the interconnect
[3.8-4.2GB/s])

- so yes when I do "synchronize_seqscans=off", and I start throwing
those seqscans from *other* sockets [--cpubindnodes 1,2,3 (while that
relation is hosted on node 0's DRAM)], of course I can see choke point
on the interconnect, (so it can feed data @ ~4.2 GB/s and even more
for short periods of time, but that's it). That's the
problem1(optimization1) realistically we can solve here I think, with
interleaving and that has been shown here multiple times: I can get
simply much more juice in the above scenario, because I have access to
better aggregated bandwidth for seqscans like that and simply put more
CPU cores (from CPUs on nodes 1,2,3) to feed that data from RAM on
node #0. It also helps achieve less deviation in latency in such
conditions. To add more fun Linux kernel likes to put randomly that
shm segments (e.g. if it fits single NUMA free hugepages, it will be
put there OR it uses some spanning across nodes , but not interleaving
and depending on relation you have it here or there). So far, so good
as far interleaving is concerned.

- however the above might be simply not true on single-socket NUMA
systems (EPYC) or just more modern multi-socket (but still same
chassis NUMA systems) - so EPYC again?(~500GB/s wild interconnect)? -
as the interconnect there might be not present (1s, BIOS/UEFI may have
setting to expose group of cores [CCD] as NUMA) OR have the bandwidth
and there's no realistic chokepoint for us there, so it wouldn't make
sense to benchmark using such hardware to spot the effects.

- So the remaining question remains, what about having better
locality/latency of let's say ~100ns on local DRAM vs 3x as much as
the remote [3][4] - how much we can gain?  (BTW this seems to be a
pretty universal rule of thumb -> 3x factor for most hardware, see
[5). If just shm is remote to the CPU for this query I'm getting:
remote: Time: 1925.820 ms
local:  Time: 1796.678 ms
so 1.071x.

and in some different situations if both shm and heap memory are
remote to the CPU for this query I was getting:
local:  Time: 9773.179 ms (00:09.773)
remote: Time: 12154.636 ms (00:12.155) +/- 300ms (+ I can see more
traffic on sockets).

So technically I should be getting this ~7%..22% profit due to lower
latency if I would be fetching just ONLY local memory (but with NUMA
we are not doing it right? we are interleaving - so we hit all sockets
most of the time to fetch data). So think about benchmarking: maybe we
would have to be using multiple pgbench_accounts_N (not just one, but
each per backend/CPU pgbench_accounts_$CPU?), together larger than s_b
to cause natural evictions, as this would then cause the partitioned
clocksweep to make it more likely that data is on the local NUMA node
and keep it hot for some time while avoiding any access to VFS cache
(so stick to reading only s_b)? (and we could measure local vs remote
access ratio too along the way). Also just for research, we should
disable clocksweep balancing. It should be enough to demonstrate that
the patch is working, right? (so 1 backend should be just reading from
1 NUMA node mostly and that should be producing some yield and it
should be having high local access ratio and just build up from there
if necessary rather than enabling all of the v20251126* patches)

XXX but eviction would be followed up by something like pread() from
VFS, and what about if that VFS cache is also on another node?

XXX those BAS strategies are just pain and yet another thing to care
about, couldn't we have some debug GUC to turn them off
(debug_benchmarking=true)?

-J.

[1] - https://www.postgresql.org/message-id/3223cdcd-6d16-4e90-a3a6-b957f762dc5a%40vondra.me

[2] - mlc --bandwidth_matrix: //that's sequential memory reads, but
with -U it is pretty close (90%) to the below ones
Bandwidths are in MB/sec (1 MB/sec = 1,000,000 Bytes/sec)
Using all the threads from each core if Hyper-threading is enabled
Using Read-only traffic type
                Numa node
Numa node            0       1       2       3
       0        22677.4  3784.2  3720.6  4199.1
       1         3855.7 22463.1  4226.8  3732.0
       2         3713.7  4228.3 21816.6  3886.3
       3         4190.7  3692.9  3796.3 22673.0

[3] mlc --latency_matrix:
Measuring idle latencies (in ns)...
                Numa node
Numa node            0       1       2       3
       0          96.7   296.0   300.7   292.5
       1         289.7    96.9   289.7   303.8
       2         309.9   289.3    97.0   291.0
       3         300.0   303.7   296.5    97.1

[4] numactl --hardware lies a little bit (real latency vs below numbers):
node distances:
node   0   1   2   3
  0:  10  20  30  20
  1:  20  10  20  30
  2:  30  20  10  20
  3:  20  30  20  10

[5] https://github.com/nviennot/core-to-core-latency?tab=readme-ov-file#dual-sockets-results

  89. Re: Adding basic NUMA awareness

    Andres Freund <andres@anarazel.de> — 2026-01-13T14:14:17Z

    Hi,

On 2026-01-13 02:13:40 +0100, Tomas Vondra wrote:
> On the azure VM (scale 200, 32GB sb), there's still no difference:

One possibility is that the host is configured with memory interleaving. That
configures the memory so that physical memory addresses interleave between the
different NUMA nodes, instead of really being node local. That can help avoid
bad performance characteristics for NUMA naive applications.

I don't quite know how to figure that out though, particularly from within a
VM :(.  Even something like https://github.com/nviennot/core-to-core-latency
or intel's mlc will not necessarily be helpful, because it depends on which
node the measured cacheline ends up on.

But I'd probably still test it, just to see whether you're observing very
different latencies between the systems.


> > Interestingly I do see a performance difference, albeit a smaller one, even
> > with OFFSET. I see similar numbers on two different 2 socket machines.
> >
>
> I wonder how significant is the number of sockets. The Azure is a single
> socket with 2 NUMA nodes, so maybe the latency differences are not
> significant enough to affect this kind of tests.

Ah, yes, a single socket machine might not show that much of an increase, at
least in simpler cases.  One of my workstations has two sockets, but each
socket has two numa nodes, the latency difference between the same numa node
and the other numa node in the same socket is small, but the difference to the
other socket is ~1.5x.

Using intel's mlc:

Measuring idle latencies for sequential access (in ns)...
		Numa node
Numa node	     0	     1	     2	     3
       0	  98.6	 106.9	 157.6	 167.9
       1	 105.8	  99.4	 158.4	 170.5
       2	 157.2	 167.4	 103.6	 105.6
       3	 158.4	 171.2	 104.5	 104.3

So there's a about a 2-10ns latency difference between 0,1 and 2,3, but about
a 50-60ns diffence across sockets...


> The xeon is a 2-socket machine, but it's also older (~10y).

It's perhaps worth noting that memory access latency has been *in*creasing in
the last generation or two of hardware...

Greetings,

Andres Freund

  90. Re: Adding basic NUMA awareness

    Andres Freund <andres@anarazel.de> — 2026-01-13T14:37:15Z

    Hi,

On 2026-01-13 14:26:37 +0100, Jakub Wartak wrote:
> - so per above and in my opinion, both on master or all patchsets
> here, classic OLTP pgbench (-S) is way too CPU computation heavy even
> with -M prepared to see NUMA latency effects.

I don't think it's that it's too CPU computation heavy, it's that it's very
latency sensitive to a small number of cachelines (ProcArrayLock, buffer
mapping table locks, btree inner pages), which will fundamentally have to
reside on one of the nodes. For pgbench -S to benefit we'd first need to
address at the very least the btree root page contention.

> - the single seqscan "SELECT sum(abalance) FROM pgbench_accounts;"
> problem (or lack of it -- with single session) is that with standard
> master, you may end up having data on just a single NUMA node. If that
> system is idle and running just 1 instance of this query, I'm getting
> like ~750MB/s of memory reads from the socket where the data is
> located (again , much below the limit of the interconnect
> [3.8-4.2GB/s])

The limit of the interconnect should be pretty much irrelevant for a single
query, you're pretty much never going to hit that query.


On my ~8 year old workstation (2x Xeon Gold 5215), with slow-ish RAM:

./mlc --bandwidth_matrix
Intel(R) Memory Latency Checker - v3.11a
Command line parameters: --bandwidth_matrix

Using buffer size of 100.000MiB/thread for reads and an additional 100.000MiB/thread for writes
Measuring Memory Bandwidths between nodes within system
Bandwidths are in MB/sec (1 MB/sec = 1,000,000 Bytes/sec)
Using all the threads from each core if Hyper-threading is enabled
Using Read-only traffic type
		Numa node
Numa node	     0	     1
       0	65173.3	34092.5
       1	33726.3	71244.1


If you're seeing ~3.5GB/s, as your [2] indicates, something either is wrong
with that system, or it's so old that it's useless for benchmarking. That's
worse than single core node-node numbers I've gotten on 10yo hardware.



The reason you're just getting 750MB is presumably because you're *latency*
limited, not because you're bandwidth limited. The problem is that our
deforming code has a, currently, unpredictable memory access at the start that
cannot meaningfully be hidden by out of order execution (because it determines
the address of the first column to actually deform, which cannot be hidden by
speculative execution).


> - however the above might be simply not true on single-socket NUMA
> systems (EPYC)

EPYC supports both single and dual socket systems. And intel has
numa-within-a-socket too...



> or just more modern multi-socket (but still same chassis NUMA systems) - so
> EPYC again?(~500GB/s wild interconnect)?

I don't think EPYC, even in the newer iterations, has anywhere near a 500GB/s
interconnect. But it's really irrelevant, latency is the main factor, not
bandwidth.


> So technically I should be getting this ~7%..22% profit due to lower
> latency if I would be fetching just ONLY local memory (but with NUMA
> we are not doing it right? we are interleaving - so we hit all sockets
> most of the time to fetch data)

We should *not* be interleaving unnecessarily, precisely because of this. We
should use the partitioned clock sweep to default to using local memory as
long as possible.


Greetings,

Andres Freund

  91. Re: Adding basic NUMA awareness

    Tomas Vondra <tomas@vondra.me> — 2026-01-14T23:26:47Z

    On 1/13/26 15:14, Andres Freund wrote:
> Hi,
> 
> On 2026-01-13 02:13:40 +0100, Tomas Vondra wrote:
>> On the azure VM (scale 200, 32GB sb), there's still no difference:
> 
> One possibility is that the host is configured with memory interleaving. That
> configures the memory so that physical memory addresses interleave between the
> different NUMA nodes, instead of really being node local. That can help avoid
> bad performance characteristics for NUMA naive applications.
> 
> I don't quite know how to figure that out though, particularly from within a
> VM :(.  Even something like https://github.com/nviennot/core-to-core-latency
> or intel's mlc will not necessarily be helpful, because it depends on which
> node the measured cacheline ends up on.
> 
> But I'd probably still test it, just to see whether you're observing very
> different latencies between the systems.
> 

I did this on the two Azure instances I've been using for testing (D96
and HB176), and I got this:

D96 (v6):

                Numa node
Numa node            0       1
       0         129.9   129.9
       1         128.3   128.1


HB176 (v4):

                Numa node
Numa node            0       1       2       3
       0         107.3   116.8   207.3   207.0
       1         120.5   110.6   207.5   207.1
       2         207.0   207.2   107.8   116.8
       3         204.4   204.7   117.7   107.9

I guess this confirms that D96 is mostly useless for evaluation of the
NUMA patches. This is a single-socket machine, with one NUMA node per
chiplet (I assume), and there's about no difference in latency.

For HB176 there clearly seems to be a difference of ~90ns between the
sockets, i.e. the latency about doubles in some cases. Each socket has
two chiplets - and there the story is about the same as on D96.

I did this on my old-ish Xeon too, and it's somewhere in between. There
clearly is difference between the sockets, but it's smaller than on
HB176. Which matches with your observation that the latency is really
increasing over time.

I doubt the interleaving mode is enabled. It clearly is not enabled on
the HB176 machine (otherwise we wouldn't see the difference, I think),
and the smaller instance can be explained by having a single socket.

I've attached the complete mlc results, for completeness.


I've also done bigger SQL test with pinning the memory/backends to
different nodes, for a range of scales and the two queries (agg and
offset). I'm attaching results for scale 100 and 10000 from D96 and
HB176 instances.

The numbers are timings per query (avg latency reported by pgbench). I
think this mostly aligns with the mlc results - the D96 shows no
difference, while HB176 shows clear differences when memory/cpu get
pinned to different sockets (but not chiplets in the same socket).

But there are some interesting details too, particularly when it comes
to behavior of the two queries. The "offset" query is affected by
latency even with no parallelism (max_parallel_workers_per_gather=0),
and it shows ~30% hit for cross-socket runs. But for "agg" there's no
difference in that case, and the hit is visible only with 4 or 8
workers. That's interesting.


Anyway, my plan at this point is to revive the old patch (before
changing direction to the simple patch), and see if we can observe a
difference on the "right" hardware. Maybe some of the results with no
improvements were due to this. This workload seems much more realistic.


regards

-- 
Tomas Vondra

  92. Re: Adding basic NUMA awareness

    Andres Freund <andres@anarazel.de> — 2026-01-15T00:01:38Z

    Hi,

On 2026-01-15 00:26:47 +0100, Tomas Vondra wrote:
> D96 (v6):
> 
>                 Numa node
> Numa node            0       1
>        0         129.9   129.9
>        1         128.3   128.1

I wonder if D96 has turned on memory interleaving... These numbers are so
close to each other that they're a tad hard to believe.

> 
> HB176 (v4):
> 
>                 Numa node
> Numa node            0       1       2       3
>        0         107.3   116.8   207.3   207.0
>        1         120.5   110.6   207.5   207.1
>        2         207.0   207.2   107.8   116.8
>        3         204.4   204.7   117.7   107.9
> 
> I guess this confirms that D96 is mostly useless for evaluation of the
> NUMA patches. This is a single-socket machine, with one NUMA node per
> chiplet (I assume), and there's about no difference in latency.


> For HB176 there clearly seems to be a difference of ~90ns between the
> sockets, i.e. the latency about doubles in some cases. Each socket has
> two chiplets - and there the story is about the same as on D96.

It looks to me like within a socket there is a latency difference of about
10ns? Only when going between sockets there's no difference between which of
the remote nodes is accessed - which makes sense to me.

For newer single-node EPYC
https://chipsandcheese.com/p/amds-epyc-9355p-inside-a-32-core
has some numbers for within socket latencies. They also see about 10ns between
inside-socket-local and inside-socket-remote.


> I did this on my old-ish Xeon too, and it's somewhere in between. There
> clearly is difference between the sockets, but it's smaller than on
> HB176. Which matches with your observation that the latency is really
> increasing over time.

FWIW https://chipsandcheese.com/p/a-look-into-intel-xeon-6s-memory has some
numbers in the "NUMA/Chiplet Characteristics" too.

One aspect in it caught my eye:
> Thus accesses to a remote NUMA node are only cached by the remote die’s
> L3. Accessing the L3 on an adjacent die increases latency by about 24
> ns. Crossing two die boundaries adds a similar penalty, increasing latency
> to nearly 80 ns for a L3 hit

Afaict that translates to an L3 hit consistently taking 80ns when accessing
remote memory, that's quite something.


> I doubt the interleaving mode is enabled. It clearly is not enabled on
> the HB176 machine (otherwise we wouldn't see the difference, I think),
> and the smaller instance can be explained by having a single socket.

As you say, there obviously is no interleaving on the HB176. I do wonder about
the D96, but ...

I wonder if the configuration is somehow visible in MSRs...



> The numbers are timings per query (avg latency reported by pgbench). I
> think this mostly aligns with the mlc results - the D96 shows no
> difference, while HB176 shows clear differences when memory/cpu get
> pinned to different sockets (but not chiplets in the same socket).

Yea, that makes sense.


> But there are some interesting details too, particularly when it comes
> to behavior of the two queries. The "offset" query is affected by
> latency even with no parallelism (max_parallel_workers_per_gather=0),
> and it shows ~30% hit for cross-socket runs. But for "agg" there's no
> difference in that case, and the hit is visible only with 4 or 8
> workers. That's interesting.

Huh, that *is* interesting. I guess the hardware prefetchers are good enough
to prefetch of the tuple headers in this case, possibly because the tuples are
small and regular enough that the hardware prefetchers manage to prefetch
everything in time?

E.g. https://docs.amd.com/api/khub/documents/goX~9ubv8i5r60A_Qrp3Rw/content
documents "L1 Stride Prefetcher" as
> The prefetcher uses the L1 cache memory access history of individual
> instructions to fetch additional lines when each access is a constant
> distance from the previous.

Greetings,

Andres Freund

  93. Re: Adding basic NUMA awareness

    Jakub Wartak <jakub.wartak@enterprisedb.com> — 2026-01-21T10:30:03Z

    On Thu, Jan 15, 2026 at 12:26 AM Tomas Vondra <tomas@vondra.me> wrote:
>
[..]
>
> Anyway, my plan at this point is to revive the old patch (before
> changing direction to the simple patch), and see if we can observe a
> difference on the "right" hardware. Maybe some of the results with no
> improvements were due to this. This workload seems much more realistic.
>

I think I have an answer why Your patch is misbehaving, but I might be wrong.
First my 4s16c64t hw numbers from master back from ~21st Nov 2025 (so here
without Your's patch), s_b=8GB, huge_pages, pgbench -i scale 150 (so it flies
under the radar NBuffers/4), which gave me ~1922MB pgbench_accounts, and then
I do the select sum(abalance):

--membind=0 --cpunodebind=0 latency average = 2468.321 ms, stddev = 0.479 ms
S0 @ 825MB/s (uncore_imc/cas_count_read/)

--membind=0 --cpunodebind=1 latency average = 2780.653 ms, stddev = 2.080 ms
S0 @ 729MB/s (uncore_imc/cas_count_read/)

(2 socket hops as old UPI/QPI had max 2 interlinks)
--membind=0 --cpunodebind=2 latency average = 2811.500 ms stddev = 1.958 ms
--membind=0 --cpunodebind=3 latency average = 2777.305 ms stddev = 1.314 ms

So in ideal conditions I should be getting a 13-14% boost if all is
well. However
with the patchset (v20251121, debug_numa='buffers,procs'), it gets somewhat
worse numbers (worse than above than on master).

--membind=0 --cpunodebind=0 (we know that patchset code will
"interleave" anyway)
latency average = 2885.806 ms stddev = 20.349 ms
and we are reading RAM from four sockets, each @ 170-180 MB/s (total ~720MB/s)

The patch spread the 1922MB relation (which would fit into one node) into many:

postgres# select numa_node, count(*) from pg_buffercache_numa where bufferid in
  (select bufferid from pg_buffercache  where relfilenode =
  (select relfilenode from pg_class where relname = 'pgbench_accounts'))
group by 1 order by 2;
 numa_node | count
-----------+-------
         2 | 55404
         3 | 55405
         1 | 55415
         0 | 79678

Also the pg_buffercache_partitions.weights indicates "{24,24,24,24}" (%) split.

Now if I do this command-trick `migratepages(1) <pid> 1-3 0`, it
really disarms our
patchset semi-manual-interleave (numa_maps bind:1-3 to real numa 0) and it
*does* restore performance from 2900ms to 2600ms, therefore it just means it is
non optimal memory placement (by clocksweep balancing).

So the question is why such a table (1922MB with 245902 relpages) would end up
spreading so quickly to the other sockets? I've assumed that hitting 4 sockets
instead of 1 will be too slow and disarm the patchset partitioned clocksweep
balancing like below:
@@ -497,7 +497,9 @@ StrategyGetBuffer(BufferAccessStrategy strategy, [...]
-       sweep = ChooseClockSweep(true);
+       sweep = ChooseClockSweep(false);

and this in next tries gets me perfect (thanks to no balancing) weight there:
postgres=# select partition, numa_node, total_allocs, num_allocs, weights
from pg_buffercache_partitions;
 partition | numa_node | total_allocs | num_allocs |   weights
-----------+-----------+--------------+------------+-------------
         0 |         0 |       246186 |          0 | {100,0,0,0}
         1 |         1 |            0 |          0 | {0,100,0,0}
         2 |         2 |            0 |          0 | {0,0,100,0}
         3 |         3 |            0 |          0 | {0,0,0,100}

which yields (compare to initial measurements of 2468 .. 2780.. 2811ms):
latency average = 2737.440 ms
latency stddev = 5.715 ms
socket 0 reliably outputs 725MB/s constant (other are idle as expected)

I think we may simply need to (re?)think of strategy on how/when to distribute,
because even If I fill just 112 MB of data fresh after startup, weight will
change from 100,0,0,0...

postgres=# create table tmp1 as select id, repeat('A', 1024) t
from generate_series(1, 100000) as id;
SELECT 100000
-- 112 MB
postgres=# \dt+ tmp1
[..]

and then scan it, I'm already having just 62% of local affinity (again
my whole s_b is 8GB, and per-NUMA-node is like 2GB, so we are under
radar of NBuffers/4 too):

postgres=# select partition, numa_node, total_allocs, num_allocs, weights
from pg_buffercache_partitions;
 partition | numa_node | total_allocs | num_allocs |    weights
-----------+-----------+--------------+------------+---------------
         0 |         0 |         2587 |          1 | {62,12,12,12}
         1 |         1 |            0 |          0 | {0,100,0,0}
         2 |         2 |            0 |          0 | {0,0,100,0}
         3 |         3 |            0 |          0 | {0,0,0,100}

-- double-confirmation:
postgres=# select numa_node, count(*), count(*) * 100.0 /
sum(count(*)) OVER() AS pct
from pg_buffercache_numa where bufferid in (
        select bufferid from pg_buffercache where relfilenode =
            (select relfilenode from pg_class where relname = 'tmp1')
) group by 1 order by 2;
 numa_node | count |         pct
-----------+-------+---------------------
         3 |  1680 | 11.7130307467057101
         2 |  1683 | 11.7339468730391132
         1 |  1690 | 11.7827511678170536
         0 |  9290 | 64.7702712124381231

so of course during the scanning in loop, we are stressing all the sockets here
pretty much. It gets even worse from there, as if I use multiple tmp* tables
like that from a single backend (but total size << 1GB), I end up with
"{24,24,24,24}" split (but all of them would fit my node).

Of course all of the above was written with assumption for getting
most of the latency ,
single backend and not having backends from different nodes.

But now If I would hypothetically benchmark pgbench -S on master(from Nov)
against Yourpatchset from back then, with low number of backends I would
be comparing single-node-hugepage allocation (on random node, because
it would fit) vspatchset doing interleaving memory. But if kernel would migrate
all those backends (in case of master) toward the node where most of
s_b is located,
Your patchset simply couldn't win this.

This brings me to a point where I'm suspicious of this clock-sweep balancing
idea (partitioning is fine, it's just the balancing which seems to be kicking
too prematurely). BTW: much earlier You seem to have benchmarked

My thoughts for today are like following how it should work if You want to have
"demonstrate any benefits on other workloads":

1. We should do not pin backends to specific CPU/numa nodes, as the kernel
should be free to move the processes closer to the data more requested (it
knows the state of memory transfers and CPU util% across nodes better than
we do).
- we query for sched_getcpu() and get node
- we stick to PgProc and Buffers from that node and that's all
(we seem to be doing just that in the patchset, great!)

2. We then should try to stick to the local node as much as possible till it's
almost full and maybe only then try to start using remote memory as a
last resort. Or maybe
even try to avoid it at all costs.
- wouldn't eviction (as the first baby step) be more preferable rather
than using
  remote memory?
- we get local affinity boost, so no distribution to the other partitions
  (unless absolutely necessary?)
- ring buffers should protect somehow with one backend filling all RAM memory
  on the node  (well except pg_prewarm? maybe we should adjust it so
it intentionally
  interleaves from start?)
  -- related: NBuffers / 4 magic number drives me nuts, but maybe we should
     tweak to take into account the number of nodes too (NBuffers / nodes / 4?)
     to avoid filling whole node
- we would get more I/O as there would be potentially lower chance to find data
  in memory on that node(?), so we would need somehow to counter this
- maybe that's a little bit sci-fi, or I'm going to be flamed here for it,
  but we could potentially track local vs remote usagecount (Buffer state
  seems to be using 54 bits, so we could add 4 bits more to track "remote"
  access, AKA usagecount_remote? but we seem to not have space to track the
  exact origin/remote node). If we would track local vs remote, we would have
  have some input as if interleaving makes sense or not, wouldn't we?
  (that would somehow be tracked on per relation/"blockset", just food
for thought,
  dunno if we even have infra for that)

-J.

  94. Re: Adding basic NUMA awareness

    Tomas Vondra <tomas@vondra.me> — 2026-06-05T12:52:35Z

    Hi,

Here's an updated version of the NUMA patch series, based on some recent
discussions about this (some at pgconf.dev, but not only that),

The main change is I significantly simplified some of the parts. Whe
patch from 20251126 was ~190K, the new version is maybe 100K, so about
half. Some of that is thanks to dropping the PGPROC partitioning
entirely, but the remaining parts are smaller too. I realize it's not a
great metric, of course.

In this message I'll explain the changes since 20251126. I'm yet to do a
thorough performance evaluation and see if it helps, I'll post that in
the next couple days.


The current patch series has these parts:


v20260605-0001-Add-shmem_populate-and-shmem_interleave-GU.patch
-------------

Somewhat unrelated, I find this useful for benchmarking and as a
baseline (what would happen if we just interleaved the shared segment).


v20260605-0002-Infrastructure-for-partitioning-of-shared-.patch
-------------

Just adds a small registry of partitions (ranges of shared buffers),
stored in shared memory, and pg_buffercache interface to inspect it.
Merely a foundation for the following patches.


v20260605-0003-NUMA-shared-buffers-partitioning.patch
-------------

The interesting part, that places some of the partitions to NUMA nodes.


v20260605-0004-clock-sweep-basic-partitioning.patch
v20260605-0005-clock-sweep-balancing-of-allocations.patch
v20260605-0006-clock-sweep-scan-all-partitions.patch
-------------

Patches that gradually partition clock-sweep. Ultimately, it should
probably be squashed into a single commit (each commit fixes some sort
of issue in the naive partitioning in 0004). But I kept them separate
because it's easier to review / understand what the issue is.


what changed?
-------------

First, I dropped the PGPROC partitioning. We may revisit that in the
future (not sure), but for now it was just a distraction and I see it as
less impactful than shared buffers / clock-sweep.

I also simplified the GUC to use a single on/off parameter (instead of
the debug_io_direct-like approach). We can revisit that, but for now
this seems more convenient.

The most significant change in the remaining parts is simplification of
the shared buffer partitioning. In particular, the partitioning is now
"best-effort" when it maps memory to shared buffers. Let me remind that
NUMA works at memory page granularity - we can't map arbitrary ranges of
memory to a node, it needs to be whole memory pages.

The 20251126 patch went into great lengths to (a) make sure BufferBlocks
and BufferDescriptors start at memory page boundary, are the partitions
are also properly aligned (both for blocks and descriptors). That was a
lot of code, it needs to happen even before we know if huge pages are
used, partitions might have been of (very) different sizes, and so on.

The new patch abandons this "perfect" partitioning, and instead does a
best-effort. It splits the buffers as evenly as possible, i.e. all
partitions have (NBuffers/npartitions) buffers, and then locates as much
memory as possible to a selected NUMA node.

With 4K pages, that's always the whole partition. With huge pages (which
is expected of relevant NUMA systems), there may be a couple buffers at
the beginning/end of a partition. But it's less than one memory page,
per partition, and we expect the systems to have 10s or 100s of GBs, so
in the bigger scheme of things it's negligible (fractions of a percent).

For buffer descriptors the math is a bit worse - descriptors need much
less memory, but even there it should not be more than ~1%.

Seems perfectly fine to me. Or rather, the extra complexity does not
seem worth the possible benefit.

This also allowed dropping a part of the "clock-sweep partitioning"
patches, dealing with cases when the partitions are of different sizes.
With this new best-effort scheme the difference is at most 1 buffer, and
we can just ignore that.


questions
---------

At this point, my main question is whether there's a better way to
partition clock-sweep and/or do the balancing of allocations between
partitions. I believe it does work, but I have a feeling there might be
a more elegant way to do this kind of stuff (like an established
balancing algorithm of some sort).

The other thing I need to verify is how this behaves with
kernel.nr_hugepages. With some earlier versions it was easy to end in a
situation where everything seemed to work, but then much later the
kernel realized it does not have enough huge pages on a particular NUMA
node and crashed with a segfault (or was it sigbus?).

Of course, the other question is performance validation - does it even
help? I plan to repeat the various experiments mentioned in this thread
(by Andres and others) on available NUMA machines. But if someone has an
idea for another benchmark (and/or what metric to measure, not just the
usual duration), let me know.


regards

-- 
Tomas Vondra

  95. Re: Adding basic NUMA awareness

    Jakub Wartak <jakub.wartak@enterprisedb.com> — 2026-06-16T08:16:00Z

    On Fri, Jun 5, 2026 at 2:52 PM Tomas Vondra <tomas@vondra.me> wrote:
>
> Hi,

Hi Tomas, thanks for working on this.

> Here's an updated version of the NUMA patch series, based on some recent
> discussions about this (some at pgconf.dev, but not only that),
[..]

1. 005 says:

+ * XXX We should enforce this in bufmgr.c, when initializing the partitions.
+ */
+#define MAX_BUFFER_PARTITIONS          32

but there isn't direct any check for checking if pg_numa_get_max_node() ->
numa_max_node() is not getting higher than allowed here. In theory this could
happen I think if ClockSweepPartitionIndex() would return
    numa = numa_node_of_cpu()
on some hypothethical very high-end setup (with plenty of sub-NUMA nodes)
and that would cause accesing .balance[] without bounds.

2. If we have in 004 struct ClockSweep with nextVictimBuffer, shouldn't
this be padded/aligned somehow later in BufferStrategyControl which does
    ClockSweep      sweeps[FLEXIBLE_ARRAY_MEMBER];
to avoid contention/false sharing? (comments says it should be but it
doesn't seem so?), maybe the comment should be TODO for now? I have not
quantified any potential benefit

With pahole after some hassle I've got:
 struct ClockSweep {
        slock_t                    clock_sweep_lock;     /*     0     1 */

        /* XXX 3 bytes hole, try to pack */

        int32                      node;                 /*     4     4 */
        int32                      firstBuffer;          /*     8     4 */
        int32                      numBuffers;           /*    12     4 */
        pg_atomic_uint32           nextVictimBuffer;     /*    16     4 */
        uint32                     completePasses;       /*    20     4 */
        pg_atomic_uint32           numBufferAllocs;      /*    24     4 */
        pg_atomic_uint32           numRequestedAllocs;   /*    28     4 */
        pg_atomic_uint64           numTotalAllocs;       /*    32     8 */
        pg_atomic_uint64           numTotalRequestedAllocs; /*    40     8 */
        uint8                      balance[32];          /*    48    32 */

        /* size: 80, cachelines: 2, members: 11 */
        /* sum members: 77, holes: 1, sum holes: 3 */
        /* last cacheline: 16 bytes */
 };
maybe with smaller MAX_BUFFER_PARTITIONS we could pack this into size=64 ?

3. In 004 sched_getcpu() is used and mentioned how to check if it is available

But my $0.02 (maybe not that important), but I've at least saw once where
(on EC2?) some clock_gettime() was very slow and that was because it was not
available in VDSO. It's usually some mix of kernel <-> arch <-> libc (not
always glibc?) compatibility matrix issue. My worry is that StrategyGetBuffer()
-> ChooseClockSweep() -> ClockSweepPartitionIndex() -> sched_getcpu() would be
available, but slow and it would mean real syscall price (and that's not once
there per buffer). I'm also somehow thinking other platforms (FreeBSD comes to
mind, but I haven't checked further). The point is: wouldn't it be cheaper
that to be refreshed from time to time instead otherwise we risk some slow
code on non-x86_64, but I doubt how proliferated is e.g. ARM64 with NUMA..
Or alternative is to have pg_test_numa proggie and this would be measuring
certain things about NUMA including timing of sched_getcpu (just like
pg_test_timing does for time), at least that could explain why somebody's
system/platform is slow.

4. Patch has problem (without fix for #8) that when number of available huge
pages in the OS is greatly higher than shared_memory_size_in_huge_pages it
will use only first NUMA node. This might be a problem when starting mulitple
DBs (they will occupy first available NUMA):

### with s_b=8GB and nr_hugepages=1500 it's OK

# find /sys/devices/system/node/ -name nr_hugepages -exec grep -H . {}
\; | grep 2048 | sort
/sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages:1250
/sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages:1250
/sys/devices/system/node/node2/hugepages/hugepages-2048kB/nr_hugepages:1250
/sys/devices/system/node/node3/hugepages/hugepages-2048kB/nr_hugepages:1250

## note the correct split below for N0/N1..
# grep huge /proc/`pgrep -f /usr/pgsql19/bin/postgres`/numa_maps
7fb1b4400000 default file=/anon_hugepage\040(deleted) huge dirty=4269
mapmax=6 N0=1250 N1=1250 N2=519 N3=1250 kernelpagesize_kB=2048

### still s_b=8GB but nr_hugepages = 19000 (~37GB), it ends all on N0=4269
# find /sys/devices/system/node/ -name nr_hugepages -exec grep -H . {}
\; | grep 2048 | sort
/sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages:4750
/sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages:4750
/sys/devices/system/node/node2/hugepages/hugepages-2048kB/nr_hugepages:4750
/sys/devices/system/node/node3/hugepages/hugepages-2048kB/nr_hugepages:4750
## all on N0...
# grep huge /proc/`pgrep -f /usr/pgsql19/bin/postgres`/numa_maps
7ff3a7a00000 default file=/anon_hugepage\040(deleted) huge dirty=4269
mapmax=6 N0=4269 kernelpagesize_kB=2048

I was even thinking go to lengths and add code for inspecting that /sys on
some later date that the kernel NUMA hugepages are really distributed
on the nodes as they should be (it's easy to end up on just 1 node out of
many; allocating via sysctl -w <higher> and then <lower> allocation is easy
way to force hugepages just to 1 node instead of many :o). I've hit the
problem multiple times, so we should bail out if we want NUMA and the
Buffer Blocks were just put on 1 node (instead of many).

5. In 005 we could mention more clealry what's the difference between
those 3: numRequestedAllocs, numTotalAllocs, numTotalRequestedAllocs
in the defintion to make it easier to read, maybe copy-cat those earlier
descriptions there too as we already have:

+ * The balancing happens in intervals - it adjusts future allocations
+ * based on stats about recent allocations, namely:
+ *
+ * - numBufferAllocs - number of allocations served by a partition
+ *
+ * - numRequestedAllocs - number of allocatios requested in a partition

6. While at it, it would be helpful if we could reset the
pg_buffercache_partitions stats in some way (pg_buffercache_partitions
is very usefull).. or is there way to plug into main pg_reset functions?

7. If I add basic error checking for mbind() then it complains a lot, like
below with annotated strace -ffe mbind to show the point:

[pid  2856] mbind(0x7fd8d0e00000, 2145386496, MPOL_BIND, [], 0,
MPOL_MF_MOVE) = -1 EINVAL (Invalid argument)
WARNING:  mbind(): Invalid argument
WARNING:  buffers descriptors for node 0 not well aligned
[0x7fd8cccf5000,0x7fd8cdcf4fc1] aligned
[0x7fd8cce00000,0x7fd8cdc00000]

[pid  2856] mbind(0x7fd8cce00000, 14680064, MPOL_BIND, [], 0,
MPOL_MF_MOVE) = -1 EINVAL (Invalid argument)
WARNING:  mbind(): Invalid argument
WARNING:  buffers for node 1 not well aligned
[0x7fd950cf5000,0x7fd9d0cf5000] aligned
[0x7fd950e00000,0x7fd9d0c00000]

[pid  2856] mbind(0x7fd950e00000, 2145386496, MPOL_BIND,
0x5589057ded00, 1, MPOL_MF_MOVE) = -1 EINVAL (Invalid argument)
WARNING:  mbind(): Invalid argument
WARNING:  buffers descriptors for node 1 not well aligned
[0x7fd8cdcf5000,0x7fd8cecf4fc1] aligned
[0x7fd8cde00000,0x7fd8cec00000]
[..]

but with pg_numa.c fixed like below (node should be size):
        ret = mbind(startptr, (endptr - startptr),
-                               MPOL_BIND, nodemask->maskp, node, MPOL_MF_MOVE);
+                               MPOL_BIND, nodemask->maskp,
nodemask->size, MPOL_MF_MOVE);

it doesn't report errors anymore and suprisngly hugepages in numa_maps are
altered from:
7fb1b4400000 default file=/anon_hugepage\040(deleted) huge dirty=4269
mapmax=6 N0=1250 N1=1250 N2=519 N3=1250 kernelpagesize_kB=2048

to explicit "binds":
7f8540000000 default file=/anon_hugepage\040(deleted) huge dirty=25
mapmax=6 N0=25 kernelpagesize_kB=2048
7f8543200000 bind:0 file=/anon_hugepage\040(deleted) huge dirty=7
mapmax=3 N0=7 kernelpagesize_kB=2048
7f8544000000 default file=/anon_hugepage\040(deleted) huge dirty=1
mapmax=2 N0=1 kernelpagesize_kB=2048
7f8544200000 bind:1 file=/anon_hugepage\040(deleted) huge dirty=7
mapmax=2 N1=7 kernelpagesize_kB=2048
7f8545000000 default file=/anon_hugepage\040(deleted) huge dirty=1
mapmax=2 N0=1 kernelpagesize_kB=2048
7f8545200000 bind:2 file=/anon_hugepage\040(deleted) huge dirty=7
mapmax=2 N2=7 kernelpagesize_kB=2048
7f8546000000 default file=/anon_hugepage\040(deleted) huge dirty=1
mapmax=2 N0=1 kernelpagesize_kB=2048
7f8546200000 bind:3 file=/anon_hugepage\040(deleted) huge dirty=7
mapmax=2 N3=7 kernelpagesize_kB=2048
7f8547000000 default file=/anon_hugepage\040(deleted) huge dirty=1
mapmax=3 N0=1 kernelpagesize_kB=2048
7f8547200000 bind:0 file=/anon_hugepage\040(deleted) huge dirty=1023
mapmax=2 N0=1023 kernelpagesize_kB=2048
7f85c7000000 default file=/anon_hugepage\040(deleted) huge dirty=1
N0=1 kernelpagesize_kB=2048
7f85c7200000 bind:1 file=/anon_hugepage\040(deleted) huge dirty=1023
N1=1023 kernelpagesize_kB=2048
7f8647000000 default file=/anon_hugepage\040(deleted) huge dirty=1
N1=1 kernelpagesize_kB=2048
7f8647200000 bind:2 file=/anon_hugepage\040(deleted) huge dirty=1023
N2=1023 kernelpagesize_kB=2048
7f86c7000000 default file=/anon_hugepage\040(deleted) huge dirty=1
N3=1 kernelpagesize_kB=2048
7f86c7200000 bind:3 file=/anon_hugepage\040(deleted) huge dirty=1023
N3=1023 kernelpagesize_kB=2048
7f8747000000 default file=/anon_hugepage\040(deleted) huge dirty=117
mapmax=6 N2=117 kernelpagesize_kB=2048

so lots of VMAs were created (it could affect performance in some way, I think
for sure it would affect for worse fork() rates by postmaster for new conns).

To me it looks like there's plenty of "N[0..3]=1" with "default" indicating
single HP page being somehow left/missed in address calculations, but I
haven't pressed this harder.

NOTE: the patch works even without this fix, but I believe if got non-0 we
cannot reliably trust the optimizer memory layout has been deployed (I suspect
it's some kind luck it sharded the shm based on number of hugepages available)


> questions
> ---------
>
> At this point, my main question is whether there's a better way to
> partition clock-sweep and/or do the balancing of allocations between
> partitions. I believe it does work, but I have a feeling there might be
> a more elegant way to do this kind of stuff (like an established
> balancing algorithm of some sort).


8. The crux of this email and stuff I wanted to further discuss, when server
is started with on this 4-NUMA box with
* numactl --cpunodebind=0 pg_ctl start # so that all backends fork()ing will be
  on node#0
* the shm split onto 4 nodes properly
* s_b still just 8GB (with ideal split),
* DB size ~15GB with 8 pgbench partitions (and fully in VFS cache)
* pgbench -c 8 -j 8 postgres -T 20 -P 1 -f seqconcurrscans.pgb with:
  \set num (:client_id % 8) + 1
  select sum(octet_length(filler)) from pgbench_accounts_:num;
* mpstat repors correctly just node#0 used

a. with the patch for GUCs with numa on and defaults two clocksweep settings
   on, I'm getting:

   latency average = 3252.254 ms
   latency stddev = 72.011 ms

b. with debug_clocksweep_balance=off, I'm realiably getting

   latency average = 2688.742 ms
   latency stddev = 61.738 ms

so IMHO clocksweep partitioning is cool, but if we are discussing the current
balancing logic leaves some juice on the table from the most optimized variant
(~1.2x) with ~90ns:270ns (local vs remote latency). In the picture above it
was 8 backends accessing 8x 1.6GB tables (lower than NBuffers / 4).

Dunno if it should be optimized further, certainly we'll get reports from
quick benchmarks run by people that PG 20 could be *slower* because.. well,
they got (sub)optimal layout during startup (all HP on 1 node and some
query hitting just that one query with local affinity and this is visible
to naked eye). I was re-reading thread and Andres also wrote "We should use
the partitioned clock sweep to default to using local memory as long as
possible."

So two further ideas:

I. BufferAccessStrategy: we could derrive affinity from the BAS strategy
itself, couldn't we? If we are using capped ring buffer, we could indicate
that we want it just from local node as priority disregarding weights (?).
Same goes for BAS_VACUUM (why would one it on remote NUMA?). With BAS_BULKREAD
there would be some potential issue with sync-scan-table code though.
With BAS_BULKWRITE e.g. CTAS/CREATE INDEX it makes lot of sense too.
prewarm could be hacked to use some new special BAS_DISTRIBUTE or something
for ideal distribution amongst all NUMA nodes.

II. what if we could track if the relation is just all-local-access?

Another idea is that if we would know that's it's just us working on some
relation (created by us; or it's not being touched remotley) then we could
ask for local-only memory affinity. So something like this:

a. in case of locally-only access rels =>
   ask for local memory first
   if that fails failback to weighted RR (so to to weights)
b. in case other rels => weighted RR (so to to weights) directly

The tracking of the fact that Buffer was accessed just locally or remotley
itself is not hard to imagine (by using some free "bits" in BufferDesc.
"state" where refcount/usagecount itself are stored, well at least 4 bits
for my 4 nodes, but there's plenty of left there), I have some PoC for
that but that's just per-Buffer tracking of "was this Buffer accessed by
remote nodes", but I'm completley lost how to make transition to the
is-the-relation-being-accessed-accross-NUMA-nodes info to drive such
optimization (we would need some shared infra just for tracking such info;
assuming up to 2^31 or 2^32 relations [OID?] and just using at least >=
2..4 bits, that's already huge number: we are talking GBs of shm mem).

BTW: I've been experimenting with this patchset and added couple of things
(see attached), and with I'm able to get optimal just by forcing affinity
too using that earlier bench:

  latency average = 2512.929 ms
  latency stddev = 97.525 ms

and that was with pure 100% affinity to my local node:

  select pg_buffercache_set_partition(0, '{100,0,0,0}');
  debug_clocksweep_balance_recalc=off
  debug_clocksweep_balance=on
  debug_clocksweep_scan_all_partitions=on

  (so it's another proof that code is fine, it's just algorithm that would
  have to adjusted)

For benchmarks with pgbench -S for 100% local affinity vs 100% remote
(I can do that with that pg_buffercache_set_partition() of mine), I'm
getting just +/- 1-2k TPS (42-43k TPS vs 41-42k TPS), so not much, __but__
I've spotted some another bug in from where we are fetching memory from
unoptimal places if we are not on node#0, I'll need to dig into that more
though. Another thing is that pgbench -S runs are much less demanding in
terms of memory bandwidth used (under <1GB/s here vs 6-8GB/s for
seqconcurrscans.sql using the same amout of cores).

> The other thing I need to verify is how this behaves with
> kernel.nr_hugepages. With some earlier versions it was easy to end in a
> situation where everything seemed to work, but then much later the
> kernel realized it does not have enough huge pages on a particular NUMA
> node and crashed with a segfault (or was it sigbus?).

It was SIGBUS and with this patchset I think we are fine: I have never
witnessed this one crashing with SIGBUS.

> Of course, the other question is performance validation - does it even
> help? I plan to repeat the various experiments mentioned in this thread
> (by Andres and others) on available NUMA machines. But if someone has an
> idea for another benchmark (and/or what metric to measure, not just the
> usual duration), let me know.

See above, but I think we would have to fix at at least: mbind() failure,
and those VMAs disconnected regions.

-J.

  96. Re: Adding basic NUMA awareness

    Tomas Vondra <tomas@vondra.me> — 2026-06-16T12:39:45Z

    
On 6/16/26 10:16, Jakub Wartak wrote:
> On Fri, Jun 5, 2026 at 2:52 PM Tomas Vondra <tomas@vondra.me> wrote:
>>
>> Hi,
> 
> Hi Tomas, thanks for working on this.
> 
>> Here's an updated version of the NUMA patch series, based on some recent
>> discussions about this (some at pgconf.dev, but not only that),
> [..]
> 
> 1. 005 says:
> 
> + * XXX We should enforce this in bufmgr.c, when initializing the partitions.
> + */
> +#define MAX_BUFFER_PARTITIONS          32
> 
> but there isn't direct any check for checking if pg_numa_get_max_node() ->
> numa_max_node() is not getting higher than allowed here. In theory this could
> happen I think if ClockSweepPartitionIndex() would return
>     numa = numa_node_of_cpu()
> on some hypothethical very high-end setup (with plenty of sub-NUMA nodes)
> and that would cause accesing .balance[] without bounds.
> 

Yes, this should be capped to the MAX_BUFFER_PARTITIONS.

> 2. If we have in 004 struct ClockSweep with nextVictimBuffer, shouldn't
> this be padded/aligned somehow later in BufferStrategyControl which does
>     ClockSweep      sweeps[FLEXIBLE_ARRAY_MEMBER];
> to avoid contention/false sharing? (comments says it should be but it
> doesn't seem so?), maybe the comment should be TODO for now? I have not
> quantified any potential benefit
> 
> With pahole after some hassle I've got:
>  struct ClockSweep {
>         slock_t                    clock_sweep_lock;     /*     0     1 */
> 
>         /* XXX 3 bytes hole, try to pack */
> 
>         int32                      node;                 /*     4     4 */
>         int32                      firstBuffer;          /*     8     4 */
>         int32                      numBuffers;           /*    12     4 */
>         pg_atomic_uint32           nextVictimBuffer;     /*    16     4 */
>         uint32                     completePasses;       /*    20     4 */
>         pg_atomic_uint32           numBufferAllocs;      /*    24     4 */
>         pg_atomic_uint32           numRequestedAllocs;   /*    28     4 */
>         pg_atomic_uint64           numTotalAllocs;       /*    32     8 */
>         pg_atomic_uint64           numTotalRequestedAllocs; /*    40     8 */
>         uint8                      balance[32];          /*    48    32 */
> 
>         /* size: 80, cachelines: 2, members: 11 */
>         /* sum members: 77, holes: 1, sum holes: 3 */
>         /* last cacheline: 16 bytes */
>  };
> maybe with smaller MAX_BUFFER_PARTITIONS we could pack this into size=64 ?
> 

Possibly. Im not entirely happy with making the ClockSweep struct so
much larger, but I haven't found a better way to store the counters
needed for balancing. The only thing I can think of is storing it
outside the struct, and maybe that's the right thing to do.

But that assumes the current balancing approach is the right one.

> 3. In 004 sched_getcpu() is used and mentioned how to check if it is available
> 
> But my $0.02 (maybe not that important), but I've at least saw once where
> (on EC2?) some clock_gettime() was very slow and that was because it was not
> available in VDSO. It's usually some mix of kernel <-> arch <-> libc (not
> always glibc?) compatibility matrix issue. My worry is that StrategyGetBuffer()
> -> ChooseClockSweep() -> ClockSweepPartitionIndex() -> sched_getcpu() would be
> available, but slow and it would mean real syscall price (and that's not once
> there per buffer). I'm also somehow thinking other platforms (FreeBSD comes to
> mind, but I haven't checked further). The point is: wouldn't it be cheaper
> that to be refreshed from time to time instead otherwise we risk some slow
> code on non-x86_64, but I doubt how proliferated is e.g. ARM64 with NUMA..
> Or alternative is to have pg_test_numa proggie and this would be measuring
> certain things about NUMA including timing of sched_getcpu (just like
> pg_test_timing does for time), at least that could explain why somebody's
> system/platform is slow.
> 

Yes, I think we may need some sort of caching for this / check only
sometimes. I haven't seen it to matter, but that may be luck and on
other systems / platforms it may be worse.

> 4. Patch has problem (without fix for #8) that when number of available huge
> pages in the OS is greatly higher than shared_memory_size_in_huge_pages it
> will use only first NUMA node. This might be a problem when starting mulitple
> DBs (they will occupy first available NUMA):
> 
> ### with s_b=8GB and nr_hugepages=1500 it's OK
> 
> # find /sys/devices/system/node/ -name nr_hugepages -exec grep -H . {}
> \; | grep 2048 | sort
> /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages:1250
> /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages:1250
> /sys/devices/system/node/node2/hugepages/hugepages-2048kB/nr_hugepages:1250
> /sys/devices/system/node/node3/hugepages/hugepages-2048kB/nr_hugepages:1250
> 
> ## note the correct split below for N0/N1..
> # grep huge /proc/`pgrep -f /usr/pgsql19/bin/postgres`/numa_maps
> 7fb1b4400000 default file=/anon_hugepage\040(deleted) huge dirty=4269
> mapmax=6 N0=1250 N1=1250 N2=519 N3=1250 kernelpagesize_kB=2048
> 
> ### still s_b=8GB but nr_hugepages = 19000 (~37GB), it ends all on N0=4269
> # find /sys/devices/system/node/ -name nr_hugepages -exec grep -H . {}
> \; | grep 2048 | sort
> /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages:4750
> /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages:4750
> /sys/devices/system/node/node2/hugepages/hugepages-2048kB/nr_hugepages:4750
> /sys/devices/system/node/node3/hugepages/hugepages-2048kB/nr_hugepages:4750
> ## all on N0...
> # grep huge /proc/`pgrep -f /usr/pgsql19/bin/postgres`/numa_maps
> 7ff3a7a00000 default file=/anon_hugepage\040(deleted) huge dirty=4269
> mapmax=6 N0=4269 kernelpagesize_kB=2048
> 
> I was even thinking go to lengths and add code for inspecting that /sys on
> some later date that the kernel NUMA hugepages are really distributed
> on the nodes as they should be (it's easy to end up on just 1 node out of
> many; allocating via sysctl -w <higher> and then <lower> allocation is easy
> way to force hugepages just to 1 node instead of many :o). I've hit the
> problem multiple times, so we should bail out if we want NUMA and the
> Buffer Blocks were just put on 1 node (instead of many).
> 

How come the pg_numa_bind_to_node() calls don't move the parts to the
correct node?

If something is already using huge pages on the other nodes, then sure,
it will fail. But I think that's OK - it's a best-effort thing. Maybe we
should exit instead in this case?

> 5. In 005 we could mention more clealry what's the difference between
> those 3: numRequestedAllocs, numTotalAllocs, numTotalRequestedAllocs
> in the defintion to make it easier to read, maybe copy-cat those earlier
> descriptions there too as we already have:
> 
> + * The balancing happens in intervals - it adjusts future allocations
> + * based on stats about recent allocations, namely:
> + *
> + * - numBufferAllocs - number of allocations served by a partition
> + *
> + * - numRequestedAllocs - number of allocatios requested in a partition
> 

I agree the explanation for this is not entirely clear.

> 6. While at it, it would be helpful if we could reset the
> pg_buffercache_partitions stats in some way (pg_buffercache_partitions
> is very usefull).. or is there way to plug into main pg_reset functions?
> 

Hmm, I was afraid it'd interfere with the balancing. I'm not sure it
makes sense to reset just some of the fields - it'd make it much harder
to interpret the counters. I'll think abou this.

> 7. If I add basic error checking for mbind() then it complains a lot, like
> below with annotated strace -ffe mbind to show the point:
> 
> [pid  2856] mbind(0x7fd8d0e00000, 2145386496, MPOL_BIND, [], 0,
> MPOL_MF_MOVE) = -1 EINVAL (Invalid argument)
> WARNING:  mbind(): Invalid argument
> WARNING:  buffers descriptors for node 0 not well aligned
> [0x7fd8cccf5000,0x7fd8cdcf4fc1] aligned
> [0x7fd8cce00000,0x7fd8cdc00000]
> 
> [pid  2856] mbind(0x7fd8cce00000, 14680064, MPOL_BIND, [], 0,
> MPOL_MF_MOVE) = -1 EINVAL (Invalid argument)
> WARNING:  mbind(): Invalid argument
> WARNING:  buffers for node 1 not well aligned
> [0x7fd950cf5000,0x7fd9d0cf5000] aligned
> [0x7fd950e00000,0x7fd9d0c00000]
> 
> [pid  2856] mbind(0x7fd950e00000, 2145386496, MPOL_BIND,
> 0x5589057ded00, 1, MPOL_MF_MOVE) = -1 EINVAL (Invalid argument)
> WARNING:  mbind(): Invalid argument
> WARNING:  buffers descriptors for node 1 not well aligned
> [0x7fd8cdcf5000,0x7fd8cecf4fc1] aligned
> [0x7fd8cde00000,0x7fd8cec00000]
> [..]
> 
> but with pg_numa.c fixed like below (node should be size):
>         ret = mbind(startptr, (endptr - startptr),
> -                               MPOL_BIND, nodemask->maskp, node, MPOL_MF_MOVE);
> +                               MPOL_BIND, nodemask->maskp,
> nodemask->size, MPOL_MF_MOVE);
> 

I think this is a silly bug on my side, clearly the nodemask should have
size > 0 even for node 0.

> it doesn't report errors anymore and suprisngly hugepages in numa_maps are
> altered from:
> 7fb1b4400000 default file=/anon_hugepage\040(deleted) huge dirty=4269
> mapmax=6 N0=1250 N1=1250 N2=519 N3=1250 kernelpagesize_kB=2048
> 
> to explicit "binds":
> 7f8540000000 default file=/anon_hugepage\040(deleted) huge dirty=25
> mapmax=6 N0=25 kernelpagesize_kB=2048
> 7f8543200000 bind:0 file=/anon_hugepage\040(deleted) huge dirty=7
> mapmax=3 N0=7 kernelpagesize_kB=2048
> 7f8544000000 default file=/anon_hugepage\040(deleted) huge dirty=1
> mapmax=2 N0=1 kernelpagesize_kB=2048
> 7f8544200000 bind:1 file=/anon_hugepage\040(deleted) huge dirty=7
> mapmax=2 N1=7 kernelpagesize_kB=2048
> 7f8545000000 default file=/anon_hugepage\040(deleted) huge dirty=1
> mapmax=2 N0=1 kernelpagesize_kB=2048
> 7f8545200000 bind:2 file=/anon_hugepage\040(deleted) huge dirty=7
> mapmax=2 N2=7 kernelpagesize_kB=2048
> 7f8546000000 default file=/anon_hugepage\040(deleted) huge dirty=1
> mapmax=2 N0=1 kernelpagesize_kB=2048
> 7f8546200000 bind:3 file=/anon_hugepage\040(deleted) huge dirty=7
> mapmax=2 N3=7 kernelpagesize_kB=2048
> 7f8547000000 default file=/anon_hugepage\040(deleted) huge dirty=1
> mapmax=3 N0=1 kernelpagesize_kB=2048
> 7f8547200000 bind:0 file=/anon_hugepage\040(deleted) huge dirty=1023
> mapmax=2 N0=1023 kernelpagesize_kB=2048
> 7f85c7000000 default file=/anon_hugepage\040(deleted) huge dirty=1
> N0=1 kernelpagesize_kB=2048
> 7f85c7200000 bind:1 file=/anon_hugepage\040(deleted) huge dirty=1023
> N1=1023 kernelpagesize_kB=2048
> 7f8647000000 default file=/anon_hugepage\040(deleted) huge dirty=1
> N1=1 kernelpagesize_kB=2048
> 7f8647200000 bind:2 file=/anon_hugepage\040(deleted) huge dirty=1023
> N2=1023 kernelpagesize_kB=2048
> 7f86c7000000 default file=/anon_hugepage\040(deleted) huge dirty=1
> N3=1 kernelpagesize_kB=2048
> 7f86c7200000 bind:3 file=/anon_hugepage\040(deleted) huge dirty=1023
> N3=1023 kernelpagesize_kB=2048
> 7f8747000000 default file=/anon_hugepage\040(deleted) huge dirty=117
> mapmax=6 N2=117 kernelpagesize_kB=2048
> 
> so lots of VMAs were created (it could affect performance in some way, I think
> for sure it would affect for worse fork() rates by postmaster for new conns).
> 

Isn't this 4 VMAs for buffers ad 4 VMAs for buffer descriptors? Or maybe
it's the parts that could not be mapped due to insufficient alignment?

> To me it looks like there's plenty of "N[0..3]=1" with "default" indicating
> single HP page being somehow left/missed in address calculations, but I
> haven't pressed this harder.
> 
> NOTE: the patch works even without this fix, but I believe if got non-0 we
> cannot reliably trust the optimizer memory layout has been deployed (I suspect
> it's some kind luck it sharded the shm based on number of hugepages available)
> 

I'm not sure I understand what you mean. What non-0?

> 
>> questions
>> ---------
>>
>> At this point, my main question is whether there's a better way to
>> partition clock-sweep and/or do the balancing of allocations between
>> partitions. I believe it does work, but I have a feeling there might be
>> a more elegant way to do this kind of stuff (like an established
>> balancing algorithm of some sort).
> 
> 
> 8. The crux of this email and stuff I wanted to further discuss, when server
> is started with on this 4-NUMA box with
> * numactl --cpunodebind=0 pg_ctl start # so that all backends fork()ing will be
>   on node#0
> * the shm split onto 4 nodes properly
> * s_b still just 8GB (with ideal split),
> * DB size ~15GB with 8 pgbench partitions (and fully in VFS cache)
> * pgbench -c 8 -j 8 postgres -T 20 -P 1 -f seqconcurrscans.pgb with:
>   \set num (:client_id % 8) + 1
>   select sum(octet_length(filler)) from pgbench_accounts_:num;
> * mpstat repors correctly just node#0 used
> 
> a. with the patch for GUCs with numa on and defaults two clocksweep settings
>    on, I'm getting:
> 
>    latency average = 3252.254 ms
>    latency stddev = 72.011 ms
> 
> b. with debug_clocksweep_balance=off, I'm realiably getting
> 
>    latency average = 2688.742 ms
>    latency stddev = 61.738 ms
> 
> so IMHO clocksweep partitioning is cool, but if we are discussing the current
> balancing logic leaves some juice on the table from the most optimized variant
> (~1.2x) with ~90ns:270ns (local vs remote latency). In the picture above it
> was 8 backends accessing 8x 1.6GB tables (lower than NBuffers / 4).
> 

How does this compare to master, i.e. without these NUMA patches?

> Dunno if it should be optimized further, certainly we'll get reports from
> quick benchmarks run by people that PG 20 could be *slower* because.. well,
> they got (sub)optimal layout during startup (all HP on 1 node and some
> query hitting just that one query with local affinity and this is visible
> to naked eye). I was re-reading thread and Andres also wrote "We should use
> the partitioned clock sweep to default to using local memory as long as
> possible."
> 
> So two further ideas:
> 
> I. BufferAccessStrategy: we could derrive affinity from the BAS strategy
> itself, couldn't we? If we are using capped ring buffer, we could indicate
> that we want it just from local node as priority disregarding weights (?).
> Same goes for BAS_VACUUM (why would one it on remote NUMA?). With BAS_BULKREAD
> there would be some potential issue with sync-scan-table code though.
> With BAS_BULKWRITE e.g. CTAS/CREATE INDEX it makes lot of sense too.
> prewarm could be hacked to use some new special BAS_DISTRIBUTE or something
> for ideal distribution amongst all NUMA nodes.
> 

Yes, I think it might make sense to disable balancing in these cases.

> II. what if we could track if the relation is just all-local-access?
> 
> Another idea is that if we would know that's it's just us working on some
> relation (created by us; or it's not being touched remotley) then we could
> ask for local-only memory affinity. So something like this:
> 
> a. in case of locally-only access rels =>
>    ask for local memory first
>    if that fails failback to weighted RR (so to to weights)
> b. in case other rels => weighted RR (so to to weights) directly
> 
> The tracking of the fact that Buffer was accessed just locally or remotley
> itself is not hard to imagine (by using some free "bits" in BufferDesc.
> "state" where refcount/usagecount itself are stored, well at least 4 bits
> for my 4 nodes, but there's plenty of left there), I have some PoC for
> that but that's just per-Buffer tracking of "was this Buffer accessed by
> remote nodes", but I'm completley lost how to make transition to the
> is-the-relation-being-accessed-accross-NUMA-nodes info to drive such
> optimization (we would need some shared infra just for tracking such info;
> assuming up to 2^31 or 2^32 relations [OID?] and just using at least >=
> 2..4 bits, that's already huge number: we are talking GBs of shm mem).
>> BTW: I've been experimenting with this patchset and added couple of things
> (see attached), and with I'm able to get optimal just by forcing affinity
> too using that earlier bench:
> 
>   latency average = 2512.929 ms
>   latency stddev = 97.525 ms
> 
> and that was with pure 100% affinity to my local node:
> 
>   select pg_buffercache_set_partition(0, '{100,0,0,0}');
>   debug_clocksweep_balance_recalc=off
>   debug_clocksweep_balance=on
>   debug_clocksweep_scan_all_partitions=on
> 
>   (so it's another proof that code is fine, it's just algorithm that would
>   have to adjusted)
> 
> For benchmarks with pgbench -S for 100% local affinity vs 100% remote
> (I can do that with that pg_buffercache_set_partition() of mine), I'm
> getting just +/- 1-2k TPS (42-43k TPS vs 41-42k TPS), so not much, __but__
> I've spotted some another bug in from where we are fetching memory from
> unoptimal places if we are not on node#0, I'll need to dig into that more
> though. Another thing is that pgbench -S runs are much less demanding in
> terms of memory bandwidth used (under <1GB/s here vs 6-8GB/s for
> seqconcurrscans.sql using the same amout of cores).
> 

No opinion. I need to look at this closer.

>> The other thing I need to verify is how this behaves with
>> kernel.nr_hugepages. With some earlier versions it was easy to end in a
>> situation where everything seemed to work, but then much later the
>> kernel realized it does not have enough huge pages on a particular NUMA
>> node and crashed with a segfault (or was it sigbus?).
> 
> It was SIGBUS and with this patchset I think we are fine: I have never
> witnessed this one crashing with SIGBUS.
> 

Good. But I wonder if allocating just the precise number of huge pages
(per shared_memory_size_in_huge_pages) can prevent moving the partitions
to the correct node.

>> Of course, the other question is performance validation - does it even
>> help? I plan to repeat the various experiments mentioned in this thread
>> (by Andres and others) on available NUMA machines. But if someone has an
>> idea for another benchmark (and/or what metric to measure, not just the
>> usual duration), let me know.
> 
> See above, but I think we would have to fix at at least: mbind() failure,
> and those VMAs disconnected regions.
> 

Yes, the mbind() failure is a bug.


regards

-- 
Tomas Vondra

  97. Re: Adding basic NUMA awareness

    Jakub Wartak <jakub.wartak@enterprisedb.com> — 2026-06-17T12:13:03Z

    On Tue, Jun 16, 2026 at 2:39 PM Tomas Vondra <tomas@vondra.me> wrote:
>
>
>
> On 6/16/26 10:16, Jakub Wartak wrote:
> > On Fri, Jun 5, 2026 at 2:52 PM Tomas Vondra <tomas@vondra.me> wrote:
> >>
> >> Hi,
> >
> > Hi Tomas, thanks for working on this.
> >
> >> Here's an updated version of the NUMA patch series, based on some recent
> >> discussions about this (some at pgconf.dev, but not only that),
> > [..]
> >
> > 1. 005 says:
> >
> > + * XXX We should enforce this in bufmgr.c, when initializing the partitions.
> > + */
> > +#define MAX_BUFFER_PARTITIONS          32
> >
> > but there isn't direct any check for checking if pg_numa_get_max_node() ->
> > numa_max_node() is not getting higher than allowed here. In theory this could
> > happen I think if ClockSweepPartitionIndex() would return
> >     numa = numa_node_of_cpu()
> > on some hypothethical very high-end setup (with plenty of sub-NUMA nodes)
> > and that would cause accesing .balance[] without bounds.
> >
>
> Yes, this should be capped to the MAX_BUFFER_PARTITIONS.
>
> > 2. If we have in 004 struct ClockSweep with nextVictimBuffer, shouldn't
> > this be padded/aligned somehow later in BufferStrategyControl which does
> >     ClockSweep      sweeps[FLEXIBLE_ARRAY_MEMBER];
> > to avoid contention/false sharing? (comments says it should be but it
> > doesn't seem so?), maybe the comment should be TODO for now? I have not
> > quantified any potential benefit
> >
> > With pahole after some hassle I've got:
> >  struct ClockSweep {
[..]
> >         pg_atomic_uint32           nextVictimBuffer;     /*    16     4 */
[..]
> >         /* size: 80, cachelines: 2, members: 11 */
> >         /* sum members: 77, holes: 1, sum holes: 3 */
> >         /* last cacheline: 16 bytes */
> >  };
> > maybe with smaller MAX_BUFFER_PARTITIONS we could pack this into size=64 ?
> >
>
> Possibly. Im not entirely happy with making the ClockSweep struct so
> much larger, but I haven't found a better way to store the counters
> needed for balancing. The only thing I can think of is storing it
> outside the struct, and maybe that's the right thing to do.
>
> But that assumes the current balancing approach is the right one.

Yeah, I'm just not sure if there is some wasted performance due to
false-sharing in very heavy benchmark scenarios (in theory the
nextVictimBuffer could bounce rather heavily).

> > 3. In 004 sched_getcpu() is used and mentioned how to check if it is available
> >
> > But my $0.02 (maybe not that important), but I've at least saw once where
> > (on EC2?) some clock_gettime() was very slow and that was because it was not
> > available in VDSO. It's usually some mix of kernel <-> arch <-> libc (not
> > always glibc?) compatibility matrix issue. My worry is that StrategyGetBuffer()
> > -> ChooseClockSweep() -> ClockSweepPartitionIndex() -> sched_getcpu() would be
> > available, but slow and it would mean real syscall price (and that's not once
> > there per buffer). I'm also somehow thinking other platforms (FreeBSD comes to
> > mind, but I haven't checked further). The point is: wouldn't it be cheaper
> > that to be refreshed from time to time instead otherwise we risk some slow
> > code on non-x86_64, but I doubt how proliferated is e.g. ARM64 with NUMA..
> > Or alternative is to have pg_test_numa proggie and this would be measuring
> > certain things about NUMA including timing of sched_getcpu (just like
> > pg_test_timing does for time), at least that could explain why somebody's
> > system/platform is slow.
> >
>
> Yes, I think we may need some sort of caching for this / check only
> sometimes. I haven't seen it to matter, but that may be luck and on
> other systems / platforms it may be worse.

Okay, so maybe an action point for us much later would be to try this on 2s+
ARM or some other much more rare setup just to see if we need to do it at all
(perhaps annotate it with TODO, I have short memory ;))

> > 4. Patch has problem (without fix for #8) that when number of available huge
> > pages in the OS is greatly higher than shared_memory_size_in_huge_pages it
> > will use only first NUMA node. This might be a problem when starting mulitple
> > DBs (they will occupy first available NUMA):
> >
> > ### with s_b=8GB and nr_hugepages=1500 it's OK
> >
> > # find /sys/devices/system/node/ -name nr_hugepages -exec grep -H . {}
> > \; | grep 2048 | sort
> > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages:1250
> > /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages:1250
> > /sys/devices/system/node/node2/hugepages/hugepages-2048kB/nr_hugepages:1250
> > /sys/devices/system/node/node3/hugepages/hugepages-2048kB/nr_hugepages:1250
> >
> > ## note the correct split below for N0/N1..
> > # grep huge /proc/`pgrep -f /usr/pgsql19/bin/postgres`/numa_maps
> > 7fb1b4400000 default file=/anon_hugepage\040(deleted) huge dirty=4269
> > mapmax=6 N0=1250 N1=1250 N2=519 N3=1250 kernelpagesize_kB=2048
> >
> > ### still s_b=8GB but nr_hugepages = 19000 (~37GB), it ends all on N0=4269
> > # find /sys/devices/system/node/ -name nr_hugepages -exec grep -H . {}
> > \; | grep 2048 | sort
> > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages:4750
> > /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages:4750
> > /sys/devices/system/node/node2/hugepages/hugepages-2048kB/nr_hugepages:4750
> > /sys/devices/system/node/node3/hugepages/hugepages-2048kB/nr_hugepages:4750
> > ## all on N0...
> > # grep huge /proc/`pgrep -f /usr/pgsql19/bin/postgres`/numa_maps
> > 7ff3a7a00000 default file=/anon_hugepage\040(deleted) huge dirty=4269
> > mapmax=6 N0=4269 kernelpagesize_kB=2048
> >
> > I was even thinking go to lengths and add code for inspecting that /sys on
> > some later date that the kernel NUMA hugepages are really distributed
> > on the nodes as they should be (it's easy to end up on just 1 node out of
> > many; allocating via sysctl -w <higher> and then <lower> allocation is easy
> > way to force hugepages just to 1 node instead of many :o). I've hit the
> > problem multiple times, so we should bail out if we want NUMA and the
> > Buffer Blocks were just put on 1 node (instead of many).
> >
>
> How come the pg_numa_bind_to_node() calls don't move the parts to the
> correct node?

Well, if we were not checking error code mbind() it could behave in
non-deterministic way I think (you ask for MPOL_BIND, maybe it does something,
maybe it doesnt work but we continued anyway).

> If something is already using huge pages on the other nodes, then sure,
> it will fail. But I think that's OK - it's a best-effort thing. Maybe we
> should exit instead in this case?

Yes, I think we should exit with FATAL.

> > 5. In 005 we could mention more clealry what's the difference between
> > those 3: numRequestedAllocs, numTotalAllocs, numTotalRequestedAllocs
> > in the defintion to make it easier to read, maybe copy-cat those earlier
> > descriptions there too as we already have:
> >
> > + * The balancing happens in intervals - it adjusts future allocations
> > + * based on stats about recent allocations, namely:
> > + *
> > + * - numBufferAllocs - number of allocations served by a partition
> > + *
> > + * - numRequestedAllocs - number of allocatios requested in a partition
> >
>
> I agree the explanation for this is not entirely clear.
>
> > 6. While at it, it would be helpful if we could reset the
> > pg_buffercache_partitions stats in some way (pg_buffercache_partitions
> > is very usefull).. or is there way to plug into main pg_reset functions?
> >
>
> Hmm, I was afraid it'd interfere with the balancing. I'm not sure it
> makes sense to reset just some of the fields - it'd make it much harder
> to interpret the counters. I'll think abou this.

Well, small hint: I find the the view very, very usefull in explaining
where/why memory is being allocated from. Maybe if we want to include it
in final version, then it might be worth (later on) to do it, if that
won't be included I'm fine just doing \watch 1 in psql to see what's
happening. Another hint is that maybe it doesnt belong to pg_buffercache
and would make it easier to integrate into pg_stat_reset*() -- dunno, how/
if extension can plug into it.

> > 7. If I add basic error checking for mbind() then it complains a lot, like
> > below with annotated strace -ffe mbind to show the point:
> >[..]
>
> I think this is a silly bug on my side, clearly the nodemask should have
> size > 0 even for node 0.
>
> > it doesn't report errors anymore and suprisngly hugepages in numa_maps are
> > altered from:
> > 7fb1b4400000 default file=/anon_hugepage\040(deleted) huge dirty=4269
> > mapmax=6 N0=1250 N1=1250 N2=519 N3=1250 kernelpagesize_kB=2048
> >
> > to explicit "binds":
> > 7f8540000000 default file=/anon_hugepage\040(deleted) huge dirty=25
> > mapmax=6 N0=25 kernelpagesize_kB=2048
> > 7f8543200000 bind:0 file=/anon_hugepage\040(deleted) huge dirty=7
> > mapmax=3 N0=7 kernelpagesize_kB=2048
> > 7f8544000000 default file=/anon_hugepage\040(deleted) huge dirty=1
> > mapmax=2 N0=1 kernelpagesize_kB=2048
> > 7f8544200000 bind:1 file=/anon_hugepage\040(deleted) huge dirty=7
> > mapmax=2 N1=7 kernelpagesize_kB=2048
> > 7f8545000000 default file=/anon_hugepage\040(deleted) huge dirty=1
> > mapmax=2 N0=1 kernelpagesize_kB=2048
> > 7f8545200000 bind:2 file=/anon_hugepage\040(deleted) huge dirty=7
> > mapmax=2 N2=7 kernelpagesize_kB=2048
> > 7f8546000000 default file=/anon_hugepage\040(deleted) huge dirty=1
> > mapmax=2 N0=1 kernelpagesize_kB=2048
> > 7f8546200000 bind:3 file=/anon_hugepage\040(deleted) huge dirty=7
> > mapmax=2 N3=7 kernelpagesize_kB=2048
> > 7f8547000000 default file=/anon_hugepage\040(deleted) huge dirty=1
> > mapmax=3 N0=1 kernelpagesize_kB=2048
> > 7f8547200000 bind:0 file=/anon_hugepage\040(deleted) huge dirty=1023
> > mapmax=2 N0=1023 kernelpagesize_kB=2048
> > 7f85c7000000 default file=/anon_hugepage\040(deleted) huge dirty=1
> > N0=1 kernelpagesize_kB=2048
> > 7f85c7200000 bind:1 file=/anon_hugepage\040(deleted) huge dirty=1023
> > N1=1023 kernelpagesize_kB=2048
> > 7f8647000000 default file=/anon_hugepage\040(deleted) huge dirty=1
> > N1=1 kernelpagesize_kB=2048
> > 7f8647200000 bind:2 file=/anon_hugepage\040(deleted) huge dirty=1023
> > N2=1023 kernelpagesize_kB=2048
> > 7f86c7000000 default file=/anon_hugepage\040(deleted) huge dirty=1
> > N3=1 kernelpagesize_kB=2048
> > 7f86c7200000 bind:3 file=/anon_hugepage\040(deleted) huge dirty=1023
> > N3=1023 kernelpagesize_kB=2048
> > 7f8747000000 default file=/anon_hugepage\040(deleted) huge dirty=117
> > mapmax=6 N2=117 kernelpagesize_kB=2048
> >
> > so lots of VMAs were created (it could affect performance in some way, I think
> > for sure it would affect for worse fork() rates by postmaster for new conns).
> >
>
> Isn't this 4 VMAs for buffers ad 4 VMAs for buffer descriptors? Or maybe
> it's the parts that could not be mapped due to insufficient alignment?

Yes, like with patchset I'm getting WARNINGs:

WARNING:  buffers for node 0 not well aligned
[0x7efc100f5000,0x7efc900f5000] aligned
[0x7efc10200000,0x7efc90000000]
WARNING:  buffers descriptors for node 0 not well aligned
[0x7efc0c0f4f80,0x7efc0d0f4f41] aligned
[0x7efc0c200000,0x7efc0d000000]
WARNING:  buffers for node 1 not well aligned
[0x7efc900f5000,0x7efd100f5000] aligned
[0x7efc90200000,0x7efd10000000]
WARNING:  buffers descriptors for node 1 not well aligned
[0x7efc0d0f4f80,0x7efc0e0f4f41] aligned
[0x7efc0d200000,0x7efc0e000000]
WARNING:  buffers for node 2 not well aligned
[0x7efd100f5000,0x7efd900f5000] aligned
[0x7efd10200000,0x7efd90000000]
WARNING:  buffers descriptors for node 2 not well aligned
[0x7efc0e0f4f80,0x7efc0f0f4f41] aligned
[0x7efc0e200000,0x7efc0f000000]
WARNING:  buffers for node 3 not well aligned
[0x7efd900f5000,0x7efe100f5000] aligned
[0x7efd90200000,0x7efe10000000]
WARNING:  buffers descriptors for node 3 not well aligned
[0x7efc0f0f4f80,0x7efc100f4f41] aligned
[0x7efc0f200000,0x7efc10000000]
LOG:  starting PostgreSQL 19beta1 on x86_64-linux, compiled by
gcc-12.2.0, 64-bit

relevant numa_maps for postmaster:

7efc0c200000 bind:0 file=/anon_hugepage\040(deleted) huge dirty=7
mapmax=2 N0=7 kernelpagesize_kB=2048
7efc0d000000 default file=/anon_hugepage\040(deleted) huge dirty=1
mapmax=2 N3=1 kernelpagesize_kB=2048
7efc0d200000 bind:1 file=/anon_hugepage\040(deleted) huge dirty=7
mapmax=2 N1=7 kernelpagesize_kB=2048
7efc0e000000 default file=/anon_hugepage\040(deleted) huge dirty=1
mapmax=2 N3=1 kernelpagesize_kB=2048
7efc0e200000 bind:2 file=/anon_hugepage\040(deleted) huge dirty=7
mapmax=2 N2=7 kernelpagesize_kB=2048
7efc0f000000 default file=/anon_hugepage\040(deleted) huge dirty=1
mapmax=2 N3=1 kernelpagesize_kB=2048
7efc0f200000 bind:3 file=/anon_hugepage\040(deleted) huge dirty=7
mapmax=2 N3=7 kernelpagesize_kB=2048
7efc10000000 default file=/anon_hugepage\040(deleted) huge dirty=1
mapmax=3 N3=1 kernelpagesize_kB=2048
7efc10200000 bind:0 file=/anon_hugepage\040(deleted) huge dirty=1023
N0=1023 kernelpagesize_kB=2048
7efc90000000 default file=/anon_hugepage\040(deleted) huge dirty=1
N3=1 kernelpagesize_kB=2048
7efc90200000 bind:1 file=/anon_hugepage\040(deleted) huge dirty=1023
N1=1023 kernelpagesize_kB=2048
7efd10000000 default file=/anon_hugepage\040(deleted) huge dirty=1
N0=1 kernelpagesize_kB=2048
7efd10200000 bind:2 file=/anon_hugepage\040(deleted) huge dirty=1023
N2=1023 kernelpagesize_kB=2048
7efd90000000 default file=/anon_hugepage\040(deleted) huge dirty=1
N2=1 kernelpagesize_kB=2048
7efd90200000 bind:3 file=/anon_hugepage\040(deleted) huge dirty=1023
N3=1023 kernelpagesize_kB=2048
7efe10000000 default file=/anon_hugepage\040(deleted) huge dirty=116
mapmax=6 N1=116 kernelpagesize_kB=2048

so if You take just 7efc10200000..7efc90000000 (from buffers@node0, 1st line)
and zoom in/grep You'll get:

7efc10200000 bind:0 file=/anon_hugepage\040(deleted) huge dirty=1023
N0=1023 kernelpagesize_kB=2048
7efc90000000 default file=/anon_hugepage\040(deleted) huge dirty=1
N3=1 kernelpagesize_kB=2048

so @ 7efc90000000 it's Node3=1 hugepage (so it's wrong by 1x 2048kb offset)

so it contains the tail of node 0 buffers and the head of node - and is
excluded from both mbind() calls. At first I've spotted this in 003/
BufferPartitionsInit() with:
  cstartptr = (char *) &BufferDescriptors[buff_first];
  endptr   = (char *) &BufferDescriptors[buff_last] + 1; // <-- BUG?
with
  endptr   = (char *) &BufferDescriptors[buff_last + 1];

but got the same issue, so I've forced the mbind() to use 2x TYPEALIGN_DOWN
to cover with policy everything and got:

7fc134e00000 default file=/anon_hugepage\040(deleted) huge dirty=24
mapmax=6 N3=24 kernelpagesize_kB=2048
7fc137e00000 bind:0 file=/anon_hugepage\040(deleted) huge dirty=8
mapmax=4 N0=8 kernelpagesize_kB=2048
7fc138e00000 bind:1 file=/anon_hugepage\040(deleted) huge dirty=8
mapmax=2 N1=8 kernelpagesize_kB=2048
7fc139e00000 bind:2 file=/anon_hugepage\040(deleted) huge dirty=8
mapmax=3 N2=8 kernelpagesize_kB=2048
7fc13ae00000 bind:3 file=/anon_hugepage\040(deleted) huge dirty=8
mapmax=2 N3=8 kernelpagesize_kB=2048
7fc13be00000 bind:0 file=/anon_hugepage\040(deleted) huge dirty=1024
N0=1024 kernelpagesize_kB=2048
7fc1bbe00000 bind:1 file=/anon_hugepage\040(deleted) huge dirty=1024
N1=1024 kernelpagesize_kB=2048
7fc23be00000 bind:2 file=/anon_hugepage\040(deleted) huge dirty=1024
mapmax=2 N2=1024 kernelpagesize_kB=2048
7fc2bbe00000 bind:3 file=/anon_hugepage\040(deleted) huge dirty=1024
N3=1024 kernelpagesize_kB=2048
7fc33be00000 default file=/anon_hugepage\040(deleted) huge dirty=116
mapmax=6 N1=116 kernelpagesize_kB=2048

that was with:
@@ -351,7 +351,7 @@ BufferPartitionsInit(void)
                        }

                        /* best effort: align the pointers, so that
the mbind() works */
-                       startptr = (char *) TYPEALIGN(numa_page_size, startptr);
+                       startptr = (char *)
TYPEALIGN_DOWN(numa_page_size, startptr);
                        endptr = (char *)
TYPEALIGN_DOWN(numa_page_size, endptr);

@@ -374,7 +374,7 @@ BufferPartitionsInit(void)
                        }

                        /* best effort: align the pointers, so that
the mbind() works */
-                       startptr = (char *) TYPEALIGN(numa_page_size, startptr);
+                       startptr = (char *)
TYPEALIGN_DOWN(numa_page_size, startptr);
                        endptr = (char *)
TYPEALIGN_DOWN(numa_page_size, endptr);

Looks way better, but it was wild guess (TBH I think I've prefered the
previous patchset version where the input shm addresses were already aligned,
but if You and others say it's easier route then sure)

> > To me it looks like there's plenty of "N[0..3]=1" with "default" indicating
> > single HP page being somehow left/missed in address calculations, but I
> > haven't pressed this harder.
> >
> > NOTE: the patch works even without this fix, but I believe if got non-0 we
> > cannot reliably trust the optimizer memory layout has been deployed (I suspect
> > it's some kind luck it sharded the shm based on number of hugepages available)
> >
>
> I'm not sure I understand what you mean. What non-0?

Non-0 return code from mbind() as the mbind() failed with -1, but it did somehow
alter numa pages probably(?), but without creating specific "isolated" VMA
explicit "bind" policy, probably maybe it did not even work at all..

> >
> >> questions
> >> ---------
> >>
> >> At this point, my main question is whether there's a better way to
> >> partition clock-sweep and/or do the balancing of allocations between
> >> partitions. I believe it does work, but I have a feeling there might be
> >> a more elegant way to do this kind of stuff (like an established
> >> balancing algorithm of some sort).
> >
> >
> > 8. The crux of this email and stuff I wanted to further discuss, when server
> > is started with on this 4-NUMA box with
> > * numactl --cpunodebind=0 pg_ctl start # so that all backends fork()ing will be
> >   on node#0
> > * the shm split onto 4 nodes properly
> > * s_b still just 8GB (with ideal split),
> > * DB size ~15GB with 8 pgbench partitions (and fully in VFS cache)
> > * pgbench -c 8 -j 8 postgres -T 20 -P 1 -f seqconcurrscans.pgb with:
> >   \set num (:client_id % 8) + 1
> >   select sum(octet_length(filler)) from pgbench_accounts_:num;
> > * mpstat repors correctly just node#0 used
> >
> > a. with the patch for GUCs with numa on and defaults two clocksweep settings
> >    on, I'm getting:
> >
> >    latency average = 3252.254 ms
> >    latency stddev = 72.011 ms
> >
> > b. with debug_clocksweep_balance=off, I'm realiably getting
> >
> >    latency average = 2688.742 ms
> >    latency stddev = 61.738 ms
> >
> > so IMHO clocksweep partitioning is cool, but if we are discussing the current
> > balancing logic leaves some juice on the table from the most optimized variant
> > (~1.2x) with ~90ns:270ns (local vs remote latency). In the picture above it
> > was 8 backends accessing 8x 1.6GB tables (lower than NBuffers / 4).
> >
>
> How does this compare to master, i.e. without these NUMA patches?

I had some problems comparing, but with "perfect setup" that includes the
following:
- pgbench/clients on node#0
- backends running on node#0
- hugepages memory on node#0..3, but with with this patchset and those goodies:
  debug_clocksweep_balance=on
  debug_clocksweep_balance_recalc=off
  debug_clocksweep_scan_all_partitions=on
  (so technically node0 backends accessing just Buffers / Buffers Desc from
  node0, technically node0 weights: "{100,0,0,0}")
- with today's new discovery for me that Linux's kernel VFS cache is also
  having also first-touch (!) NUMA policy and really important here (so
  VFS cached data also alters results of testing wildly!), so I had to
  force unloading VFS cache and force-loading it into node#0, I've was
  getting for seqconcurrscans:
  latency average = 2701.705 ms
  latency stddev = 111.608 ms

vs master, huh, but which scenario? the default one without any affinity?
- assuming you get the split shm split like "N0=1059 N1=1299 N2=879 N3=1031"
  but that's appeneded-only (so not interleaved (?)) and you even risk
  having shm placed on just __one__ node (if it is big enough and free
  enough)
- if you go straight to benchmarking it (CPU hits random nodes)
  latency average = 3439.200 ms
  latency stddev = 580.501 ms
- with backends forked() to node#0 (numactl --cpunodebind=0 pg_ctl start)
  latency average = 4937.543 ms
  latency stddev = 573.841 ms
  because of random VFS cache placement (I imagine it as flow of on node0 CPUs
    a. nextVictimBuffer contention
    b. getBuffer() - fetch random shm memory from random NUMA node
    c. pread() - fetch from VFS cache but from *remote* NUMA node
  )
- with backends forked() to node#0 and pinned VFS cached fully on node#0 too
  latency average = 4518.651 ms
  latency stddev = 797.369 ms
  (but this is still Buffers from other nodes)
- same as above above and numactl interleaved shm:
  latency average = 3792.016 ms
  latency stddev = 825.186 ms
- same as above and interleaved shm, but without pining to CPUs on specifc
  node and ensure random VFS cache vs nodes:
  latency average = 2913.813 ms
  latency stddev = 352.552 ms
- but the moment you read anything reads base/ into VFS cache to particular
  node (imagine pg_prewarm or even just tar) assuming it was not there it
  also pins that to that node memory and you'll get:
  latency average = 3594.180 ms
  latency stddev = 851.949 ms

> > Dunno if it should be optimized further, certainly we'll get reports from
> > quick benchmarks run by people that PG 20 could be *slower* because.. well,
> > they got (sub)optimal layout during startup (all HP on 1 node and some
> > query hitting just that one query with local affinity and this is visible
> > to naked eye). I was re-reading thread and Andres also wrote "We should use
> > the partitioned clock sweep to default to using local memory as long as
> > possible."
> >
> > So two further ideas:
> >
> > I. BufferAccessStrategy: we could derrive affinity from the BAS strategy
> > itself, couldn't we? If we are using capped ring buffer, we could indicate
> > that we want it just from local node as priority disregarding weights (?).
> > Same goes for BAS_VACUUM (why would one it on remote NUMA?). With BAS_BULKREAD
> > there would be some potential issue with sync-scan-table code though.
> > With BAS_BULKWRITE e.g. CTAS/CREATE INDEX it makes lot of sense too.
> > prewarm could be hacked to use some new special BAS_DISTRIBUTE or something
> > for ideal distribution amongst all NUMA nodes.
> >
>
> Yes, I think it might make sense to disable balancing in these cases.

OK, I did not code anything of that as

> > II. what if we could track if the relation is just all-local-access?
> >
> > Another idea is that if we would know that's it's just us working on some
> > relation (created by us; or it's not being touched remotley) then we could
> > ask for local-only memory affinity. So something like this:
> >
> > a. in case of locally-only access rels =>
> >    ask for local memory first
> >    if that fails failback to weighted RR (so to to weights)
> > b. in case other rels => weighted RR (so to to weights) directly
> >
> > The tracking of the fact that Buffer was accessed just locally or remotley
> > itself is not hard to imagine (by using some free "bits" in BufferDesc.
> > "state" where refcount/usagecount itself are stored, well at least 4 bits
> > for my 4 nodes, but there's plenty of left there), I have some PoC for
> > that but that's just per-Buffer tracking of "was this Buffer accessed by
> > remote nodes", but I'm completley lost how to make transition to the
> > is-the-relation-being-accessed-accross-NUMA-nodes info to drive such
> > optimization (we would need some shared infra just for tracking such info;
> > assuming up to 2^31 or 2^32 relations [OID?] and just using at least >=
> > 2..4 bits, that's already huge number: we are talking GBs of shm mem).
> >> BTW: I've been experimenting with this patchset and added couple of things
> > (see attached), and with I'm able to get optimal just by forcing affinity
> > too using that earlier bench:
> >
> >   latency average = 2512.929 ms
> >   latency stddev = 97.525 ms
> >
> > and that was with pure 100% affinity to my local node:
> >
> >   select pg_buffercache_set_partition(0, '{100,0,0,0}');
> >   debug_clocksweep_balance_recalc=off
> >   debug_clocksweep_balance=on
> >   debug_clocksweep_scan_all_partitions=on
> >
> >   (so it's another proof that code is fine, it's just algorithm that would
> >   have to adjusted)
> >
> > For benchmarks with pgbench -S for 100% local affinity vs 100% remote
> > (I can do that with that pg_buffercache_set_partition() of mine), I'm
> > getting just +/- 1-2k TPS (42-43k TPS vs 41-42k TPS), so not much, __but__
> > I've spotted some another bug in from where we are fetching memory from
> > unoptimal places if we are not on node#0, I'll need to dig into that more
> > though. Another thing is that pgbench -S runs are much less demanding in
> > terms of memory bandwidth used (under <1GB/s here vs 6-8GB/s for
> > seqconcurrscans.sql using the same amout of cores).
> >
>
> No opinion. I need to look at this closer.

Great !

> >> The other thing I need to verify is how this behaves with
> >> kernel.nr_hugepages. With some earlier versions it was easy to end in a
> >> situation where everything seemed to work, but then much later the
> >> kernel realized it does not have enough huge pages on a particular NUMA
> >> node and crashed with a segfault (or was it sigbus?).
> >
> > It was SIGBUS and with this patchset I think we are fine: I have never
> > witnessed this one crashing with SIGBUS.
> >
>
> Good. But I wonder if allocating just the precise number of huge pages
> (per shared_memory_size_in_huge_pages) can prevent moving the partitions
> to the correct node.
>

Not sure I understand (?) how's that related to SIGBUS?

-J.

  98. Re: Adding basic NUMA awareness

    Tomas Vondra <tomas@vondra.me> — 2026-06-24T20:26:29Z

    Hi,

Here's an updated patch series, with only minor changes to fix the mbind
issues:

1) It uses the correct nodemask size, so that the mbind actually binds
the partition to the right node.

2) It aligns the start/end pointers so that there no pages are left with
the default memory policy. So now there should be only the bind:N
entries, not the single-page "default" ones. This means the last page of
a partition can be mapped to a different node, but that seems fine (in
the end it could have happened with the old approach too).

I've also included Jakub's "goodies" patch with the additional GUCs.
Those seem potentially useful to development.


I have some results from a new round of benchmarks, and it's a bit
disappointing. Or rather, there seem to be some issues that I can't
figure out, causing regressions.

Consider a very simple test, doing a lot of sequential scans to put a
fair amount of pressure on the clocksweep / buffer replacement. There's
a .tgz with the benchmark script attached, but it does about this:

* Initialize a pgbench database with scale 2000 (so ~30GB, about twice
the shared buffers).

* Uses --partitions=100, so that the partitions are small enough not to
trigger the 1/4 threshold (i.e. not use circular buffers).

* Does runs with custom script, forcing sequential scans of the table,
with two queries:

select count(1) from pgbench_accounts;

select * from pgbench_accounts offset 1000000000;

Those are called "count" and "offset" in the results. The script forces
serial sequential scans (no index scans, no parallelism), and does runs
with 1, 8 and 32 clients (this is an old-ish xeon with 44 physical cores
on two sockets, 2 NUMA nodes).

I did runs with "master" and the all the 7 patches, with the NUMA stuff
enabled/disabled since 0003 (which adds it). See the two PDFs with more
complete results, but here's the "count" query for a subset of the
patches (the omitted ones behave similarly to what's shown here).

This chart is for median latency (in milliseconds):

  clients       master     0003      0004    0003/on    0004/on
  -------------------------------------------------------------
        1        12767    12582     14509      12807      15307
        8        14383    14355     14149      14069      16165
       32        14756    15198     14836      14984      17128
       --------------------------------------------------------
        1                  103%      114%       100%       120%
        8                  101%       98%        98%       112%
       32                  102%      101%       102%       116%

The percentages are compared to "master", the columns with "/on" are
with shared_buffers_numa=on.

Clearly, there's no chance with 0003 (which binds shared buffer
partitions to NUMA nodes, even if that's enabled). The differences are
within noise, pretty much, for all client counts.

Then 0004 gets applied, which partitions the clock sweep. And well, that
doesn't go particularly well. There is a bit of a regression even with
numa=off, but it kinda recovers with the following patches. But with
numa=on, there's a consistent ~10% regression (give or take).

I've spent a fair bit of time investigating what's causing this, but so
far I have nothing. I assume it's something silly in the patches
partitioning the clocksweep, or maybe the approach is flawed in some
way. Not sure :-(


regards

-- 
Tomas Vondra

  99. Re: Adding basic NUMA awareness

    Jakub Wartak <jakub.wartak@enterprisedb.com> — 2026-06-25T12:19:36Z

    On Wed, Jun 24, 2026 at 10:29 PM Tomas Vondra <tomas@vondra.me> wrote:

Hi,

> Here's an updated patch series, with only minor changes to fix the mbind
> issues:
[..]
> I've also included Jakub's "goodies" patch with the additional GUCs.
> Those seem potentially useful to development.

Cool!

> I have some results from a new round of benchmarks, and it's a bit
> disappointing. Or rather, there seem to be some issues that I can't
> figure out, causing regressions.
[..]
> This chart is for median latency (in milliseconds):
>
>   clients       master     0003      0004    0003/on    0004/on
>   -------------------------------------------------------------
>         1        12767    12582     14509      12807      15307
>         8        14383    14355     14149      14069      16165
>        32        14756    15198     14836      14984      17128
>        --------------------------------------------------------
>         1                  103%      114%       100%       120%
>         8                  101%       98%        98%       112%
>        32                  102%      101%       102%       116%
>

I haven't tried it yet, however I can spot some things:

No crystal clear idea why, but in the script I can see that you have
io_method = io_uring and are not dropping_caches, so IMHO it is too complex
interaction at this stage.

One hint: such setup is going to be problematic for proving numbers.
On the meeting I've tried to describe that I've been using io_method = sync
instead of 'worker' to get more predicitable results (together with echo 3
> drop_caches), because then it is that backend's CPU/$NODE doing that
pread()/pwrite() -- or any other operating performing the load --
it is going to put that file onto that_specific_$NODE --
so even if you have sequence like:
    pgbench -i
    pg_ctl restart
    pgbench -c XX

then pgbench -i even with shared_buffers_numa=on will spread into many
nodes the Buffers, yet after the restart the VFS cache portion of the data
will still reside on single specific $NODE that wrote it to the filesystem
(due to local-first-tocuh-affinity even for VFS cache), so any further reading:
VFS cache --pread()--> s_b will take the hit of remote interconnect with
some probablity depending on where the new backends are running. Also
with worker it is even worse as we have those memory queue in between. I
think we even can have this:

file in VFS cache @ node0   --because of first touch policy (pgbench -i/prewarm)
io worker @ node1           --hits latency from node0 and node2
shm io worker queue @ node2 --well
client backend @ node 3     --puts into shm io worker on node2

Therefore I'm sticking to 'sync' to ease the pain... but with uring, I suspect
the situation is kind of similiar as we call io_uring_submit(), and we
may endup using io-wq kernel threads, and we have those submission/receive
(memory) queues that are located somewhere (that is on some node) too.

I think, we simply lack affinity for IO/NUMA for all io modes except sync, but
it's too early I suspect and way outside of scope for this $thread. I've
started thinking about it just last week, so... (but hopefully I'll be able
to ship helper fscachenuma.c to show layout of file across VFS caches on nodes
next week I hope)

Maybe some other suggestions:

Q1) Maybe some crosschecks first?
       # balance should be equal between nodes even for baseline
       # linux kernel has tendency to fit shm into one if it fits
       find /sys/devices/system/node*/ -name 'free_hugepages' -exec
grep -H . {} \;

       # check N0 and N1 even for default policy, might also reveal imbalance
       # lots of RAM and too big huge_pages allows fitting whole shm
into just N0
       # see point 4 from [1]
       grep /anon_h /proc/$SOMEREALBACKENDPID/numa_maps

       # then during pgbench -c run maybe those:
       mpstat -N ALL 1
       perf stat -a -e uncore_imc/cas_count_read/,uncore_imc/cas_count_write/ \
          --per-socket -I 1000  # or -M
memory_bandwidth_read,memory_bandwidth_write

    (it might reveal that problem I've described above about io_method:
    even with pgbench -c 1 you might be reading from all sockets/wrong sockets
    instead of the correct one with affinity)

    I like to pin CPUs to just one node for pgbench -c
<NUMBER_OF_CPUS/NUMBER_OF_NODES>
    [to saturate one node only] and start server also with CPU pining
    [or use this debug_numa_node to force] to that one node and cross-check
    what's being read (using perf) and usually I have to disarm clock balancing
    and override weights using pg_buffercache_set_partition() to also force
    weight to stay local only - only then I'm able to outrun master. That's
    how this idea was born that if we are only working on node $N with
some relations
    then let's use only node $N's Buffers. But I have 90us:~280us
local vs remote
    latency, so it's probably way easier for me to see results even without
    disabling CPU-idle-states/turboboost/etc.

Q2) Dunno, but 0007 is not changing anything in runtime and you get huge
    discrepeancy results when going 0006 -> 0007 for clients=1 (see
128% -> 112%).
    Literally, as the same code but different rebuild (ELF image)
would be having
    vastly different layout enough to cause perf issues?

Hopefully next week I'll try to repro those numbers to see if I can
help more.

-J.

[1] - https://www.postgresql.org/message-id/CAKZiRmzo9xnJSgO4b26DTZqPuObcQ-6ncay%2BmOEKs9rzCkegUA%40mail.gmail.com

  100. Re: Adding basic NUMA awareness

    Tomas Vondra <tomas@vondra.me> — 2026-06-25T13:49:00Z

    
On 6/25/26 14:19, Jakub Wartak wrote:
> On Wed, Jun 24, 2026 at 10:29 PM Tomas Vondra <tomas@vondra.me> wrote:
> 
> Hi,
> 
>> Here's an updated patch series, with only minor changes to fix the mbind
>> issues:
> [..]
>> I've also included Jakub's "goodies" patch with the additional GUCs.
>> Those seem potentially useful to development.
> 
> Cool!
> 
>> I have some results from a new round of benchmarks, and it's a bit
>> disappointing. Or rather, there seem to be some issues that I can't
>> figure out, causing regressions.
> [..]
>> This chart is for median latency (in milliseconds):
>>
>>   clients       master     0003      0004    0003/on    0004/on
>>   -------------------------------------------------------------
>>         1        12767    12582     14509      12807      15307
>>         8        14383    14355     14149      14069      16165
>>        32        14756    15198     14836      14984      17128
>>        --------------------------------------------------------
>>         1                  103%      114%       100%       120%
>>         8                  101%       98%        98%       112%
>>        32                  102%      101%       102%       116%
>>
> 
> I haven't tried it yet, however I can spot some things:
> 
> No crystal clear idea why, but in the script I can see that you have
> io_method = io_uring and are not dropping_caches, so IMHO it is too complex
> interaction at this stage.
> 

By caches I assume you mean page cache? The test is meant so simulate a
cached system, copying data between shared buffers and page cache. My
expectation is that once we start hitting I/O, it'll completely hide
most differences due to NUMA.

> One hint: such setup is going to be problematic for proving numbers.
> On the meeting I've tried to describe that I've been using io_method = sync
> instead of 'worker' to get more predicitable results (together with echo 3
>> drop_caches), because then it is that backend's CPU/$NODE doing that
> pread()/pwrite() -- or any other operating performing the load --
> it is going to put that file onto that_specific_$NODE --
> so even if you have sequence like:
>     pgbench -i
>     pg_ctl restart
>     pgbench -c XX
> 

Hmm, I missed that point during the meeting. I wonder if "worker" is
interacting with the NUMA somehow (I mean, does it load it into the
right node?). But I'm using io_uring, and it's not clear to me why sync
would be better for benchmarking?

Ultimately, we need to make sure it works well with io_uring anyway,
right? Even if "sync" happens to be better for benchmarking (or even for
NUMA stuff), we have to make it work with worker/io_uring. Because
that's what practical systems use.

> then pgbench -i even with shared_buffers_numa=on will spread into many
> nodes the Buffers, yet after the restart the VFS cache portion of the data
> will still reside on single specific $NODE that wrote it to the filesystem
> (due to local-first-tocuh-affinity even for VFS cache), so any further reading:
> VFS cache --pread()--> s_b will take the hit of remote interconnect with
> some probablity depending on where the new backends are running. Also
> with worker it is even worse as we have those memory queue in between. I
> think we even can have this:
> 
> file in VFS cache @ node0   --because of first touch policy (pgbench -i/prewarm)
> io worker @ node1           --hits latency from node0 and node2
> shm io worker queue @ node2 --well
> client backend @ node 3     --puts into shm io worker on node2
> 
> Therefore I'm sticking to 'sync' to ease the pain... but with uring, I suspect
> the situation is kind of similiar as we call io_uring_submit(), and we
> may endup using io-wq kernel threads, and we have those submission/receive
> (memory) queues that are located somewhere (that is on some node) too.
> 
> I think, we simply lack affinity for IO/NUMA for all io modes except sync, but
> it's too early I suspect and way outside of scope for this $thread. I've
> started thinking about it just last week, so... (but hopefully I'll be able
> to ship helper fscachenuma.c to show layout of file across VFS caches on nodes
> next week I hope)
> 

Ah, you're suggesting the page cache stuff will be placed on a single
NUMA node? That may be true, it's a good point. And maybe it could skew
the results in a bad way. Still, that would be the case even without the
NUMA partitioning, no?

> Maybe some other suggestions:
> 
> Q1) Maybe some crosschecks first?
>        # balance should be equal between nodes even for baseline
>        # linux kernel has tendency to fit shm into one if it fits
>        find /sys/devices/system/node*/ -name 'free_hugepages' -exec
> grep -H . {} \;
> 
>        # check N0 and N1 even for default policy, might also reveal imbalance
>        # lots of RAM and too big huge_pages allows fitting whole shm
> into just N0
>        # see point 4 from [1]
>        grep /anon_h /proc/$SOMEREALBACKENDPID/numa_maps
> 
>        # then during pgbench -c run maybe those:
>        mpstat -N ALL 1
>        perf stat -a -e uncore_imc/cas_count_read/,uncore_imc/cas_count_write/ \
>           --per-socket -I 1000  # or -M
> memory_bandwidth_read,memory_bandwidth_write
> 
>     (it might reveal that problem I've described above about io_method:
>     even with pgbench -c 1 you might be reading from all sockets/wrong sockets
>     instead of the correct one with affinity)
> 

I'll try, but if you could try running some experiments on your own,
that might be helpful.

>     I like to pin CPUs to just one node for pgbench -c
> <NUMBER_OF_CPUS/NUMBER_OF_NODES>
>     [to saturate one node only] and start server also with CPU pining
>     [or use this debug_numa_node to force] to that one node and cross-check
>     what's being read (using perf) and usually I have to disarm clock balancing
>     and override weights using pg_buffercache_set_partition() to also force
>     weight to stay local only - only then I'm able to outrun master. That's
>     how this idea was born that if we are only working on node $N with
> some relations
>     then let's use only node $N's Buffers. But I have 90us:~280us
> local vs remote
>     latency, so it's probably way easier for me to see results even without
>     disabling CPU-idle-states/turboboost/etc.
> 
> Q2) Dunno, but 0007 is not changing anything in runtime and you get huge
>     discrepeancy results when going 0006 -> 0007 for clients=1 (see
> 128% -> 112%).
>     Literally, as the same code but different rebuild (ELF image)
> would be having
>     vastly different layout enough to cause perf issues?
> 
> Hopefully next week I'll try to repro those numbers to see if I can
> help more.
> 

Thank you! That'd be great.


regards

-- 
Tomas Vondra

  101. Re: Adding basic NUMA awareness

    Jakub Wartak <jakub.wartak@enterprisedb.com> — 2026-06-29T07:42:49Z

    On Thu, Jun 25, 2026 at 3:49 PM Tomas Vondra <tomas@vondra.me> wrote:
>
> >> I have some results from a new round of benchmarks, and it's a bit
> >> disappointing. Or rather, there seem to be some issues that I can't
> >> figure out, causing regressions.
> > [..]
> >> This chart is for median latency (in milliseconds):
> >>
> >>   clients       master     0003      0004    0003/on    0004/on
> >>   -------------------------------------------------------------
> >>         1        12767    12582     14509      12807      15307
> >>         8        14383    14355     14149      14069      16165
> >>        32        14756    15198     14836      14984      17128
> >>        --------------------------------------------------------
> >>         1                  103%      114%       100%       120%
> >>         8                  101%       98%        98%       112%
> >>        32                  102%      101%       102%       116%
> >>
> >
> > I haven't tried it yet, however I can spot some things:
> >
> > No crystal clear idea why, but in the script I can see that you have
> > io_method = io_uring and are not dropping_caches, so IMHO it is too complex
> > interaction at this stage.
> >
>
> By caches I assume you mean page cache? The test is meant so simulate a
> cached system, copying data between shared buffers and page cache. My
> expectation is that once we start hitting I/O, it'll completely hide
> most differences due to NUMA.

No, it wont completley hide it, those differences at least here still matter
(AFAIR right now like +/- 10% here)

> > One hint: such setup is going to be problematic for proving numbers.
> > On the meeting I've tried to describe that I've been using io_method = sync
> > instead of 'worker' to get more predicitable results (together with echo 3
> >> drop_caches), because then it is that backend's CPU/$NODE doing that
> > pread()/pwrite() -- or any other operating performing the load --
> > it is going to put that file onto that_specific_$NODE --
> > so even if you have sequence like:
> >     pgbench -i
> >     pg_ctl restart
> >     pgbench -c XX
> >
>
> Hmm, I missed that point during the meeting. I wonder if "worker" is
> interacting with the NUMA somehow (I mean, does it load it into the
> right node?). But I'm using io_uring, and it's not clear to me why sync
> would be better for benchmarking?
>
> Ultimately, we need to make sure it works well with io_uring anyway,
> right? Even if "sync" happens to be better for benchmarking (or even for
> NUMA stuff), we have to make it work with worker/io_uring. Because
> that's what practical systems use.

Yes, we need to make work with more advanced, but I don't think we are there
yet (we'll need some more patches in orde rto demonstrate it reliably).

> > then pgbench -i even with shared_buffers_numa=on will spread into many
> > nodes the Buffers, yet after the restart the VFS cache portion of the data
> > will still reside on single specific $NODE that wrote it to the filesystem
> > (due to local-first-tocuh-affinity even for VFS cache),
> > [.. blabla , use io_method=sync ]
> >
>
> Ah, you're suggesting the page cache stuff will be placed on a single
> NUMA node? That may be true, it's a good point. And maybe it could skew
> the results in a bad way.

I've just published [0], see for yourself:

This happens especiall after pgbench -i, so:
     pgbench -i # pagecache placement on one NUMA node
     pg_ctl restart
     pgbench -c XX

is day and night different than let's say:
     pgbench -i
     echo 3 > drop_caches
     pg_ctl restart
     pgbench -c XX # pagecache placement happens by many backends
                   # potentially many NUMA nodes

> Still, that would be the case even without the NUMA partitioning, no?

Right, in my experience we should not benchmark against master started
with the default pg_ctl (that's is without numactl --interleave=all) because
it is confusing to reason about it due how the s_b could laid out without
that interleaving. I mean later we can switch to that default, but IMHO not
yet.

> > Maybe some other suggestions:
> >
> > Q1) Maybe some crosschecks first?
> >        # balance should be equal between nodes even for baseline
> >        # linux kernel has tendency to fit shm into one if it fits
> >        find /sys/devices/system/node*/ -name 'free_hugepages' -exec
> > grep -H . {} \;
> >
> >        # check N0 and N1 even for default policy, might also reveal imbalance
> >        # lots of RAM and too big huge_pages allows fitting whole shm
> > into just N0
> >        # see point 4 from [1]
> >        grep /anon_h /proc/$SOMEREALBACKENDPID/numa_maps
> >
> >        # then during pgbench -c run maybe those:
> >        mpstat -N ALL 1
> >        perf stat -a -e uncore_imc/cas_count_read/,uncore_imc/cas_count_write/ \
> >           --per-socket -I 1000  # or -M
> > memory_bandwidth_read,memory_bandwidth_write
> >
> >     (it might reveal that problem I've described above about io_method:
> >     even with pgbench -c 1 you might be reading from all sockets/wrong sockets
> >     instead of the correct one with affinity)
> >
>
> I'll try, but if you could try running some experiments on your own,
> that might be helpful.
[..]
> > Hopefully next week I'll try to repro those numbers to see if I can
> > help more.
> >
>
> Thank you! That'd be great.

Yeah, I'll try my best, we'll see how it goes. Right now I've just dropped
that fscachenuma proggie to aid us in troubleshooting.

-J.

[0] - https://github.com/jakubwartakEDB/fscachenuma

  102. Re: Adding basic NUMA awareness

    Jakub Wartak <jakub.wartak@enterprisedb.com> — 2026-06-30T12:51:18Z

    On Mon, Jun 29, 2026 at 9:42 AM Jakub Wartak
<jakub.wartak@enterprisedb.com> wrote:
>
> On Thu, Jun 25, 2026 at 3:49 PM Tomas Vondra <tomas@vondra.me> wrote:
> >
> > >> I have some results from a new round of benchmarks, and it's a bit
> > >> disappointing. Or rather, there seem to be some issues that I can't
> > >> figure out, causing regressions.
> > > [..]
> > >> This chart is for median latency (in milliseconds):
> > >>
> > >>   clients       master     0003      0004    0003/on    0004/on
> > >>   -------------------------------------------------------------
> > >>         1        12767    12582     14509      12807      15307
> > >>         8        14383    14355     14149      14069      16165
> > >>        32        14756    15198     14836      14984      17128
> > >>        --------------------------------------------------------
> > >>         1                  103%      114%       100%       120%
> > >>         8                  101%       98%        98%       112%
> > >>        32                  102%      101%       102%       116%
> > >>

[..lots of variables..]

> > I'll try, but if you could try running some experiments on your own,
> > that might be helpful.
> [..]
> > > Hopefully next week I'll try to repro those numbers to see if I can
> > > help more.
> > >
> >
> > Thank you! That'd be great.
>
> Yeah, I'll try my best, we'll see how it goes. Right now I've just dropped
> that fscachenuma proggie to aid us in troubleshooting.
>
> -J.
>
> [0] - https://github.com/jakubwartakEDB/fscachenuma

Hi Tomas,

OK, so I've run couple of tests and modified run.sh and also tried to fix
some inefficiencies spotted while testing this. Note the attached
performance matrix is in TPS (so more is better). Raw results/CSV and
scripts are attached too.

* run2 = 2 workloads, partitioned pgbench_accounts
* run3 = just pgbenchS w/o partitioning + warmup
* run4 = semi-like pgbenchS w/o partitioning but 100k rows + warmup

One important modification in those run shell scripts is that they
clean page-cache (drop_cached) as mentioned earlier to avoid false results
where everything would on node#N after pgbench -i ran. Probably I did
not get any regressions you've got, because of this. Or better diff -u
run*.sh scripts.

The "inst-optimized" is just the same patchset (so "inst-patchset") + crude
attempt in 0008 to make further smooth out things and avoid regressions while
I've been working on this. 0008 does couple of things:

a. implements CPU/node caching instead quering it every single buffer. Even
   if on x86_64 that is optimized by vdso/kernel to avoid the real syscall,
   the semi-syscall tax seems to be visible when fetching lots of buffers.
   128 is arbitrary and still kind of low (128*8kB=1MB, and we are doing
   hundreths of MB/s; while rescheduling happened only every couple of
   seconds).

b1. minimize the attempt to use other partittions till some threshold (
   and then it relies on the scan-all-partitions)

b2. avoids selecting idle partitions (defined as avg_allocs/2) - if there
   are low allocations there it is debatable if cache utilization is better
   or sticking to lower latency is better (e.g. in some workloads buffer
   reuse is close to 0, so lower latency is clearly better)

Results are attached, some observations:

0.There were vast differences in how pg_ctl is started (interleaved or not),
  so I've decided in the end to show relative to both situations.

1.In run2/seqconcurrscans I've saturated my interconnect and that's why
  it's giving 129-155% there. I don't have access physiscal hw, but I suspect
  that modern 2socket EPYC5 has like ~614GB/s per socket RAM bandwidth,
  but the max oneway bandwith of the interconnect is around ~220GB/s (
  no way to provie it), so *IF* with hundreths of cores we would be able
  fetch at this rate we could saturte modern hardware too that way (and
  we birefly touched related topic: batched executor, accelerating it
  so fast those effects could be more easily achieveable)

2.run3 has no partiitioning because according to perf and my eyes, it
  spent time not on the buffers itself (thus it was way heavier on CPU
  [partitioning] than on memory...), so that's how run3 was born without
  partitions :D

3.The warmup is critical for run3/pgbenchS, as I've noticed that depending
  on ${luck} if you start the "master" (baseline w/o interleaving) and pgbench
  it right away everything might land on node0 (s_b, pagecache), so "master"
  was basically cheating in benchmarks vs especially Your's patchset where
  it was spreading way too soon. Having drop_caches, additional warump and
  only then proper pgbench kind of reduces that luck-factor. In general I
  think all runs with c=1 seem to have kind of low singal-to-noise ratio. I
  was thinking about pinning to always stick to the same NUMA node from start
  to win against master just for this c=1 scenarios, but "meh".

3b. in short for pgbench -S we can gain like 2-5%

4.run4 was made just to prove that workload fetching more buffers, than
  the standard pgbench -S (1 row?), seems to be the key to prove
  optimizations in 0008 (other than showing good benefits for seqconcurrscans
  of course). So run4 just shows benefit compared to 0001-0007 alone.

Stil on the table:

1. maybe even better balancing is possible (?), but this one is seems enough?
   I'm out of other ideas, well other than the
   "shared-relation-use-by-foreign-node" idea described much earlier (but
   I won't be able to pull that off), so I'm not entering this rabbit hole
   any deeper.

2. Digging into io_method=worker optimizations (answering question: are they
   necessary?) Maybe I'll throw in run5 quite soon, this is going to be
   crucial to answer.

3. Potentially mentioned earlier BAS strategies (forcing just use of local
   partitions for known-to-be-only-local-users: CTAS/VACCUM/etc), but I'm
   afarid that's not for me as I would certainly break/violate some
   invisible to me boundary.

Maybe You could run those run*.sh with master vs inst-patchset/optimized?
(I'm not sure, maybe there's even different factor at play too...)

-J.

  103. Re: Adding basic NUMA awareness

    Jakub Wartak <jakub.wartak@enterprisedb.com> — 2026-07-02T09:24:21Z

    On Tue, Jun 30, 2026 at 2:51 PM Jakub Wartak
<jakub.wartak@enterprisedb.com> wrote:
>
> On Mon, Jun 29, 2026 at 9:42 AM Jakub Wartak
> <jakub.wartak@enterprisedb.com> wrote:
> >
> > On Thu, Jun 25, 2026 at 3:49 PM Tomas Vondra <tomas@vondra.me> wrote:
> > >
> > > >> I have some results from a new round of benchmarks, and it's a bit
> > > >> disappointing. Or rather, there seem to be some issues that I can't
> > > >> figure out, causing regressions.
> > > > [..]
> > > >> This chart is for median latency (in milliseconds):
> > > >>
> > > >>   clients       master     0003      0004    0003/on    0004/on
> > > >>   -------------------------------------------------------------
> > > >>         1        12767    12582     14509      12807      15307
> > > >>         8        14383    14355     14149      14069      16165
> > > >>        32        14756    15198     14836      14984      17128
> > > >>        --------------------------------------------------------
> > > >>         1                  103%      114%       100%       120%
> > > >>         8                  101%       98%        98%       112%
> > > >>        32                  102%      101%       102%       116%
> > > >>
>
> [..lots of variables..]
>
> > > I'll try, but if you could try running some experiments on your own,
> > > that might be helpful.
> > [..]
> > > > Hopefully next week I'll try to repro those numbers to see if I can
> > > > help more.
> > > >
> > >
> > > Thank you! That'd be great.
> >
> > Yeah, I'll try my best, we'll see how it goes. Right now I've just dropped
> > that fscachenuma proggie to aid us in troubleshooting.
> >
> > -J.
> >
> > [0] - https://github.com/jakubwartakEDB/fscachenuma
>
> Hi Tomas,
>
> OK, so I've run couple of tests and modified run.sh and also tried to fix
> some inefficiencies spotted while testing this. Note the attached
> performance matrix is in TPS (so more is better). Raw results/CSV and
> scripts are attached too.
>
> * run2 = 2 workloads, partitioned pgbench_accounts
> * run3 = just pgbenchS w/o partitioning + warmup
> * run4 = semi-like pgbenchS w/o partitioning but 100k rows + warmup
>
[..]
>
> Stil on the table:
>
> 1. maybe even better balancing is possible (?), but this one is seems enough?
>    I'm out of other ideas, well other than the
>    "shared-relation-use-by-foreign-node" idea described much earlier (but
>    I won't be able to pull that off), so I'm not entering this rabbit hole
>    any deeper.

See below, seems like not needed (?)

> 2. Digging into io_method=worker optimizations (answering question: are they
>    necessary?) Maybe I'll throw in run5 quite soon, this is going to be
>    crucial to answer.

OK, I'm attaching are results from mine runs 5 and 6:
- only seqconcurrscans was tested, well because for other workloads io_worker
  method was not getting load for those workers (only seq scans were offloaded)
- checksums were disabled, because IMHO that would be unfair comparision
  (AFAIR there are offloaded)
- those optimizations for 0008 "optimized (numa=on, bal=on)" easily beat
  "patched (numa=on, bal=on)" and seem to be crucial. We get like 1.2x-1.4x
  across every io_method, but only with 0008.
- even when then doing just those logical fully cached reads from fully VFS
  cached case, io_urings shines (I've added raw TPS number to show this,
  compare across tables e.g. io_uring vs sync  13.378/8.993=1.487x for
  io_uring with NUMA, but for master's for io_uring:sync it was just 8.79/7.389
  = 1.189x without NUMA); seems like io_uring is more lightweight to show
  more benefits of remote memory latencies
- there's some more juice to get out of the balancer for 0-reuse workloads
  (but IMHO it's pointless to squeeze more, it's hard already)
- I was probably wrong when expecting that io_worker's worker processes/queues
  should get NUMA affinity. They don't need to be apparently for me to see
  benefits (maybe they could  be and it would even better, but meh).
  So with ruling io_method impact (I speculated earlier that his could be it),
  this means that you were either hitting lack of opimizations needed from
  0008 or were impacted by lack of drop_caches before the runs

> Maybe You could run those run*.sh with master vs inst-patchset/optimized?
> (I'm not sure, maybe there's even different factor at play too...)

This is seems to be crucial now, to double confirm the results / loaded-tested
on your hw with 0008. (but that hardware really needs to have effective latency
difference between at least 2 NUMA nodes -- Intel's mlc is good for this);
maybe also tweak those 125% inside 0008 to some other values, I've got 4 nodes,
so 100/4=25%)

> 3. Potentially mentioned earlier BAS strategies (forcing just use of local
>    partitions for known-to-be-only-local-users: CTAS/VACCUM/etc), but I'm
>    afarid that's not for me as I would certainly break/violate some
>    invisible to me boundary.

And this one is still potentially on the table as nice thing to have.

-J.