Re: Fwd: sensible configuration of max_connections

On 07/02/2020 13:18, Chris Withers wrote:
> On 07/02/2020 12:49, Chris Ellis wrote:
>> What's "too much" for max_connections? What happens when you set it to
>>
>>     high? What factors affect that number?
>>
>>
>> When sizing max_connections you need to trade off how many 
>> connections your application will use at peak vs how much RAM and CPU 
>> you have.
>>
>> Each connection is capable of allocating work_mem and has a stack etc.
>>
>> As such you don't want max_connections to be able to run your system 
>> out of RAM.
> Sure, but that's where I'm trying to find out what's sensible. The box 
> has 196GB memory, most of that in hugepages, 18 core Intel Skylake 
> with HT on giving 36 cores and tonnes of SSD for storage. How would I 
> turn that spec into a sensible number for max_connections? As that 
> number grows, what contention points in postgres will start creaking 
> (shared memory where the IPC happens?)

The max_connections setting  is an upper limit after which postgresql 
will reject connections. You don't really want to hit that limit, rather 
you want to keep the number of concurrent connections to a reasonable 
number (and have max_connections somewhere above that).

Each connection is a postgresql process, so active connections are 
competing for resources and even idle connections take up some RAM. 
Creating a new connection is launching a new process (and doing some 
setup) so it's relatively expensive.

Doing some sort of connection pooling is a good idea, especially for 
web-apps that connect, do a few short queries and disconnect. Django is 
probably doing a passable job at pooling already, so you might want to 
see how many connections it's using under normal load. Adding a 
dedicated pooler in between Django and PostgreSQL would give you more 
flexibility and might be a good idea, but if what Django is already 
doing is reasonable you may not need it.

What's a good number of active connections to aim for? It probably 
depends on whether they tend to be CPU-bound or IO-bound, but I've seen 
the rule of thumb of "around twice the number of CPU cores" tossed 
around, and it's probably a decent place to start, then run it under 
normal load and see how it behaves - cpu usage, RAM, IO, request latency 
and throughput.

Cheers,
   Steve