Horton

Thread

  1. Windows SHMMAX (was: Default configuration)

    Merlin Moncure <merlin.moncure@rcsonline.com> — 2003-02-11T18:49:17Z

    Does anyone know whether cygwin has a setting comparable to SHMMAX,
and if so what is its default value?  How about the upcoming native
Windows port --- any issues there?

From a pure win32 point of view, a good approach would be to use the
VirtualAlloc() memory allocation functions and set up a paged memory
allocation system.  From a very top down point of view, this is the
method of choice if portability is not an issue.  An abstraction to use
this technique within pg context is probably complex and requires
writing lots of win32 api code, which is obviously not desirable.

Another way of looking at it is memory mapped files.  This probably most
closely resembles unix shared memory and is the de facto standard way
for interprocess memory block sharing.  Sadly, performance will suffer
because you have to rely on the virtual memory system (think: writing to
files) to do a lot of stupid stuff you don't necessarily want or need.
The OS has to guarantee that the memory can be swapped out to file at
any time and therefore mirrors the pagefile to the allocated memory
blocks.

With the C++/C memory malloc/free api, you are supposed to be able to
get some of the benefits of virtual alloc (in particular, setting a
process memory allocation limit), but personal experience did not bear
this out.  However, this api sits directly over the virtual allocation
system and is the most portable.  The application has to guard against
fragmentation and things like that in this case.  In win32, server
thrashing is public enemy #1 for database servers, mostly due to the
virtual allocation system (which is quite fast when used right, btw).

Merlin

  2. Re: Windows SHMMAX (was: Default configuration)

    Greg Copeland <greg@copelandconsulting.net> — 2003-02-11T19:23:17Z

    On Tue, 2003-02-11 at 12:49, Merlin Moncure wrote:
> Does anyone know whether cygwin has a setting comparable to SHMMAX,
> and if so what is its default value?  How about the upcoming native
> Windows port --- any issues there?
> 
> >From a pure win32 point of view, a good approach would be to use the
> VirtualAlloc() memory allocation functions and set up a paged memory
> allocation system.  From a very top down point of view, this is the
> method of choice if portability is not an issue.  An abstraction to use
> this technique within pg context is probably complex and requires
> writing lots of win32 api code, which is obviously not desirable.
> 
> Another way of looking at it is memory mapped files.  This probably most
> closely resembles unix shared memory and is the de facto standard way
> for interprocess memory block sharing.  Sadly, performance will suffer
> because you have to rely on the virtual memory system (think: writing to
> files) to do a lot of stupid stuff you don't necessarily want or need.
> The OS has to guarantee that the memory can be swapped out to file at
> any time and therefore mirrors the pagefile to the allocated memory
> blocks.
> 
> With the C++/C memory malloc/free api, you are supposed to be able to
> get some of the benefits of virtual alloc (in particular, setting a
> process memory allocation limit), but personal experience did not bear
> this out.  However, this api sits directly over the virtual allocation
> system and is the most portable.  The application has to guard against
> fragmentation and things like that in this case.  In win32, server
> thrashing is public enemy #1 for database servers, mostly due to the
> virtual allocation system (which is quite fast when used right, btw).


IIRC, there is a mechanism which enables it to be directly
supported/mapped via pagefile.  This is the preferred means of memory
mapped files unless you have a specific need which dictates otherwise. 
Meaning, it allows for many supposed optimizations to be used by the OS
as it is suppose to bypass some of the filesystem overhead.


Regards,

-- 
Greg Copeland <greg@copelandconsulting.net>
Copeland Computer Consulting

  3. Re: Windows SHMMAX (was: Default configuration)

    Curt Sampson <cjs@cynic.net> — 2003-02-12T06:34:06Z

    On Tue, 11 Feb 2003, Merlin Moncure wrote:

> Another way of looking at it is memory mapped files.  This probably most
> closely resembles unix shared memory and is the de facto standard way
> for interprocess memory block sharing.  Sadly, performance will suffer
> because you have to rely on the virtual memory system (think: writing to
> files) to do a lot of stupid stuff you don't necessarily want or need.

To the contrary, for the majority of the shared memory usage of
postgres, which is cached file data, the virtual memory system is doing
exactly what you want it to: managing the movement of data between
memory and disk, and caching the more frequently accessed data to reduce
the chances you will actually need to access the disk for it.

For shared memory used only for IPC, typically a VM system treats it no
differently from any other non-shared memory, so if it's doing something
you "don't want or need" (a proposition I quite heartily disagree with),
it's going to be doing that very every piece of memory your application
allocates and uses, shared or not.

> The OS has to guarantee that the memory can be swapped out to file at
> any time and therefore mirrors the pagefile to the allocated memory
> blocks.

The OS does not need to write the pagefile. On modern Unix systems that
are not allowing overcommit, the space will be allocated but never
written unless there's a need to free up some physical memory, and the
pages in question are used infrequently enough that the system decides
that they are good candidates to be paged out. I would imagine that
Windows does the same.

cjs
-- 
Curt Sampson  <cjs@cynic.net>   +81 90 7737 2974   http://www.netbsd.org
    Don't you know, in this new Dark Age, we're all light.  --XTC