Re: Estimating HugePages Requirements?

Nathan Bossart <bossartn@amazon.com>

From: "Bossart, Nathan" <bossartn@amazon.com>
To: Michael Paquier <michael@paquier.xyz>
Cc: Justin Pryzby <pryzby@telsasoft.com>, Andres Freund <andres@anarazel.de>, Magnus Hagander <magnus@hagander.net>, Mark Dilger <mark.dilger@enterprisedb.com>, Don Seiler <don@seiler.us>, "pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Date: 2021-09-03T17:36:43Z
Lists: pgsql-hackers

Attachments

On 9/2/21, 6:46 PM, "Michael Paquier" <michael@paquier.xyz> wrote:
> On Thu, Sep 02, 2021 at 04:46:56PM +0000, Bossart, Nathan wrote:
>> Yeah, huge_pages_required might not serve much purpose for Windows.
>> We could always set it to -1 for Windows if it seems like it'll do
>> more harm than good.
>
> I'd be fine with this setup on environments where there is no need for
> it.

I did this in v5.

> One thing that would be incorrect upon more review is that we'd still
> have data_checksums wrong with -C, meaning that the initial read of
> the control file should be moved further up, and getting the control
> file checks done before registering workers would be better.  Keeping
> the lock file at the end would be fine AFAIK, but should we worry
> about the interactions with _PG_init() here?

I think we can avoid so much reordering by moving the -C handling
instead.  That should also fix things like data_checksums.  I split
the reordering part out into its own patch in v5.

You bring up an interesting point about _PG_init().  Presently, you
can safely assume that the data directory is locked during _PG_init(),
so there's no need to worry about breaking something on a running
server.  I don't know how important this is.  Most _PG_init()
functions that I've seen will define some GUCs, request some shared
memory, register some background workers, and/or install some hooks.
Those all seem like safe things to do, but I wouldn't be at all
surprised to hear examples to the contrary.  In any case, it looks
like the current ordering of these two steps has been there for 15+
years.

If this is a concern, one option would be to disallow running "-C
shared_memory_size" on running servers.  That would have to extend to
GUCs like data_checksums and huge_pages_required, too.

> 0001, that refactors the calculation of the shmem size into a
> different routine, is fine as-is, so I'd like to move on and apply
> it.

Sounds good to me.

Nathan

Commits

  1. Silence extra logging when using "postgres -C" on runtime-computed GUCs

  2. doc: Improve postgres command for shared_memory_size_in_huge_pages

  3. Introduce GUC shared_memory_size_in_huge_pages

  4. Support "postgres -C" with runtime-computed GUCs

  5. Make shared_memory_size a preset option

  6. Introduce GUC shared_memory_size

  7. Move the shared memory size calculation to its own function

  8. Add new GUC, max_worker_processes, limiting number of bgworkers.