Horton

Re: Estimating HugePages Requirements?

Michael Paquier <michael@paquier.xyz>

View thread View on postgr.es
From: Michael Paquier <michael@paquier.xyz>
To: "Bossart, Nathan" <bossartn@amazon.com>
Cc: Justin Pryzby <pryzby@telsasoft.com>, Andres Freund <andres@anarazel.de>, Magnus Hagander <magnus@hagander.net>, Mark Dilger <mark.dilger@enterprisedb.com>, Don Seiler <don@seiler.us>, "pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Date: 2021-08-30T07:29:19Z
Lists: pgsql-hackers

Attachments

On Sat, Aug 28, 2021 at 05:36:37AM +0000, Bossart, Nathan wrote:
> Attached is a hacky attempt at adding a shared_memory_size GUC in a
> way that could be used with -C.  This should include the amount of
> shared memory requested by extensions, too.  As long as huge_page_size
> is nonzero, it seems easy enough to provide the number of huge pages
> needed as well.

Yes, the implementation is not good.  The key thing is that by wanting
to support shared_memory_size with the -C switch of postgres, we need
to call process_shared_preload_libraries before output_config_variable. 
This additionally means to call ApplyLauncherRegister() before that so
as all the bgworker slots are not taken first.  Going through
_PG_init() also means that we'd better use ChangeToDataDir()
beforehand.

Attached is a WIP to show how the order of the operations could be
changed, as that's easier to grasp.  Even if we don't do that, having
the GUC and the refactoring of CalculateShmemSize() would still be
useful, as one could just query an existing instance for an estimation
of huge pages for a cloned one.

The GUC shared_memory_size should have GUC_NOT_IN_SAMPLE and
GUC_DISALLOW_IN_FILE, with some documentation, of course.  I added the
flags to the GUC, not the docs.   The code setting up the GUC is not
good either.  It would make sense to just have that in a small wrapper
of ipci.c, perhaps.
--
Michael

Commits

  1. Silence extra logging when using "postgres -C" on runtime-computed GUCs

  2. doc: Improve postgres command for shared_memory_size_in_huge_pages

  3. Introduce GUC shared_memory_size_in_huge_pages

  4. Support "postgres -C" with runtime-computed GUCs

  5. Make shared_memory_size a preset option

  6. Introduce GUC shared_memory_size

  7. Move the shared memory size calculation to its own function

  8. Add new GUC, max_worker_processes, limiting number of bgworkers.