Horton

Thread

  1. Re: [QUESTIONS] How to use memory instead of hd?

    Vadim B. Mikheev <vadim@sable.krasnoyarsk.su> — 1998-04-22T05:51:15Z

    Bruce Momjian wrote:
> 
> >
> > Hi All
> >
> > I just upgraded to 128MB RAM, and really don't need ALL of it MOST of the
> > time, and the main reason for it was to help with database speed.
> >
> > But then I got to thinking, is there a way to just semi-permently put a
> > database into RAM? so that when a query is done it goes only to the image
> > of the database in RAM, never even touching the hard drive.  This would be
> > enormasly faster on 50 ~ 100 (or what ever) MB databases, especially on
> > those new boxes with that 6? or 10? nano second RAM.  Especially if one
> > could pick and chose which table(s) to put in RAM
> 
> You can tune your OS to use most of that as buffer cache.  However,
> writes will be flushed to disk by postgresql fync, or the OS syncing
> every so often, but not too bad.  You can also up your postgres shared
> memory buffers, though some OS's have a limit on that.
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                                   ^^^^^^^^^^^^^^^^^^^^
Could we use mmap (instead of shmem) with MAP_ANON flag to get more memory
for shared buffer pool ?
I'm using FreeBSD, man mmap says:

     MAP_ANON    Map anonymous memory not associated with any specific file.
                 The file descriptor used for creating MAP_ANON regions is
                 used only for naming, and may be specified as -1 if no name
                 is associated with the region.

Vadim

  2. Re: [HACKERS] Re: [QUESTIONS] How to use memory instead of hd?

    Bruce Momjian <maillist@candle.pha.pa.us> — 1998-04-22T13:58:12Z

    > 
> Bruce Momjian wrote:
> > 
> > >
> > > Hi All
> > >
> > > I just upgraded to 128MB RAM, and really don't need ALL of it MOST of the
> > > time, and the main reason for it was to help with database speed.
> > >
> > > But then I got to thinking, is there a way to just semi-permently put a
> > > database into RAM? so that when a query is done it goes only to the image
> > > of the database in RAM, never even touching the hard drive.  This would be
> > > enormasly faster on 50 ~ 100 (or what ever) MB databases, especially on
> > > those new boxes with that 6? or 10? nano second RAM.  Especially if one
> > > could pick and chose which table(s) to put in RAM
> > 
> > You can tune your OS to use most of that as buffer cache.  However,
> > writes will be flushed to disk by postgresql fync, or the OS syncing
> > every so often, but not too bad.  You can also up your postgres shared
> > memory buffers, though some OS's have a limit on that.
>   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>                                    ^^^^^^^^^^^^^^^^^^^^
> Could we use mmap (instead of shmem) with MAP_ANON flag to get more memory
> for shared buffer pool ?
> I'm using FreeBSD, man mmap says:
> 
>      MAP_ANON    Map anonymous memory not associated with any specific file.
>                  The file descriptor used for creating MAP_ANON regions is
>                  used only for naming, and may be specified as -1 if no name
>                  is associated with the region.
> 
> Vadim

Yes, we could.  I don't think we do because we an anon-mapped region is
the same as shmem, but if the limit is higher for anon mmap, we could
used it.


-- 
Bruce Momjian                          |  830 Blythe Avenue
maillist@candle.pha.pa.us              |  Drexel Hill, Pennsylvania 19026
  +  If your life is a hard drive,     |  (610) 353-9879(w)
  +  Christ can be your backup.        |  (610) 853-3000(h)

  3. Re: [HACKERS] Re: [QUESTIONS] How to use memory instead of hd?

    Goran Thyni <goran@bildbasen.se> — 1998-04-22T14:37:17Z

    Bruce Momjian wrote:
> 
> >
> > Bruce Momjian wrote:
> > >
> > Could we use mmap (instead of shmem) with MAP_ANON flag to get more memory
> > for shared buffer pool ?
> > I'm using FreeBSD, man mmap says:
> Yes, we could.  I don't think we do because we an anon-mapped region is
> the same as shmem, but if the limit is higher for anon mmap, we could
> used it.

It could be faster too,
and it API is cleaner too (IMHO).
If we resolve the fork/exec -> just fork problem in the server
mmap would make a good improvment in simplicity and probably
speed too.

	regards,
-- 
---------------------------------------------
Göran Thyni, sysadm, JMS Bildbasen, Kiruna

  4. Re: [HACKERS] Re: [QUESTIONS] How to use memory instead of hd?

    Kent S. Gordon <kgor@inetspace.com> — 1998-04-22T16:00:19Z

    >>>>> "maillist" == Bruce Momjian <maillist@candle.pha.pa.us> writes:

    >>  Bruce Momjian wrote: > > > > > Hi All > > > > I just upgraded
    >> to 128MB RAM, and really don't need ALL of it MOST of the > >
    >> time, and the main reason for it was to help with database
    >> speed.  > > > > But then I got to thinking, is there a way to
    >> just semi-permently put a > > database into RAM? so that when a
    >> query is done it goes only to the image > > of the database in
    >> RAM, never even touching the hard drive.  This would be > >
    >> enormasly faster on 50 ~ 100 (or what ever) MB databases,
    >> especially on > > those new boxes with that 6? or 10? nano
    >> second RAM.  Especially if one > > could pick and chose which
    >> table(s) to put in RAM > > You can tune your OS to use most of
    >> that as buffer cache.  However, > writes will be flushed to
    >> disk by postgresql fync, or the OS syncing > every so often,
    >> but not too bad.  You can also up your postgres shared > memory
    >> buffers, though some OS's have a limit on that.
    >> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    >> ^^^^^^^^^^^^^^^^^^^^ Could we use mmap (instead of shmem) with
    >> MAP_ANON flag to get more memory for shared buffer pool ?  I'm
    >> using FreeBSD, man mmap says:
    >> 
    >> MAP_ANON Map anonymous memory not associated with any specific
    >> file.  The file descriptor used for creating MAP_ANON regions
    >> is used only for naming, and may be specified as -1 if no name
    >> is associated with the region.
    >> 
    >> Vadim

    > Yes, we could.  I don't think we do because we an anon-mapped
    > region is the same as shmem, but if the limit is higher for anon
    > mmap, we could used it.
On FreeBSD the limit for mmap is basically the amount of swap vs a
predefined limit when a kernel is built.  This was a project I was
thinking of doing but have not had the time yet.  mmap may also be
faster due to tighter intergration in the kernel.  In general it is
better to avoid all the SYSV IPC calls and use other methods of doing
the needed operations.  Stevens book (either Advanced Unix Programming 
or Unix Network Programming) has a good discussion of the pro/CONS of
the SYSV IPC interfaces.

    > -- Bruce Momjian | 830 Blythe Avenue maillist@candle.pha.pa.us |
    > Drexel Hill, Pennsylvania 19026 + If your life is a hard drive,
    > | (610) 353-9879(w) + Christ can be your backup.  | (610)
    > 853-3000(h)

-- 
Kent S. Gordon
Architect
iNetSpace Co.
voice: (972)851-3494 fax:(972)702-0384 e-mail:kgor@inetspace.com

  5. Re: [HACKERS] Re: [QUESTIONS] How to use memory instead of hd?

    Bruce Momjian <maillist@candle.pha.pa.us> — 1998-04-22T17:18:23Z

    >     > Yes, we could.  I don't think we do because we an anon-mapped
>     > region is the same as shmem, but if the limit is higher for anon
>     > mmap, we could used it.
> On FreeBSD the limit for mmap is basically the amount of swap vs a
> predefined limit when a kernel is built.  This was a project I was
> thinking of doing but have not had the time yet.  mmap may also be
> faster due to tighter intergration in the kernel.  In general it is
> better to avoid all the SYSV IPC calls and use other methods of doing
> the needed operations.  Stevens book (either Advanced Unix Programming 
> or Unix Network Programming) has a good discussion of the pro/CONS of
> the SYSV IPC interfaces.

OK, mmap added to FAQ. 


-- 
Bruce Momjian                          |  830 Blythe Avenue
maillist@candle.pha.pa.us              |  Drexel Hill, Pennsylvania 19026
  +  If your life is a hard drive,     |  (610) 353-9879(w)
  +  Christ can be your backup.        |  (610) 853-3000(h)

  6. Re: [HACKERS] Re: [QUESTIONS] How to use memory instead of hd?

    Bruce Momjian <maillist@candle.pha.pa.us> — 1998-04-22T17:18:42Z

    >     > Yes, we could.  I don't think we do because we an anon-mapped
>     > region is the same as shmem, but if the limit is higher for anon
>     > mmap, we could used it.
> On FreeBSD the limit for mmap is basically the amount of swap vs a
> predefined limit when a kernel is built.  This was a project I was
> thinking of doing but have not had the time yet.  mmap may also be
> faster due to tighter intergration in the kernel.  In general it is
> better to avoid all the SYSV IPC calls and use other methods of doing
> the needed operations.  Stevens book (either Advanced Unix Programming 
> or Unix Network Programming) has a good discussion of the pro/CONS of
> the SYSV IPC interfaces.

I meant, mmap() added to TODO.

-- 
Bruce Momjian                          |  830 Blythe Avenue
maillist@candle.pha.pa.us              |  Drexel Hill, Pennsylvania 19026
  +  If your life is a hard drive,     |  (610) 353-9879(w)
  +  Christ can be your backup.        |  (610) 853-3000(h)

  7. Re: [HACKERS] Re: [QUESTIONS] How to use memory instead of hd?

    Vadim B. Mikheev <vadim@sable.krasnoyarsk.su> — 1998-04-23T10:33:31Z

    Göran Thyni wrote:
> 
> Bruce Momjian wrote:
> >
> > >
> > > Bruce Momjian wrote:
> > > >
> > > Could we use mmap (instead of shmem) with MAP_ANON flag to get more memory
> > > for shared buffer pool ?
> > > I'm using FreeBSD, man mmap says:
> > Yes, we could.  I don't think we do because we an anon-mapped region is
> > the same as shmem, but if the limit is higher for anon mmap, we could
> > used it.
> 
> It could be faster too,
> and it API is cleaner too (IMHO).
> If we resolve the fork/exec -> just fork problem in the server
> mmap would make a good improvment in simplicity and probably
> speed too.

     MAP_INHERIT
                 Permit regions to be inherited across execve(2) system calls.

 - exec isn't problem...

Vadim

  8. Re: [HACKERS] Re: [QUESTIONS] How to use memory instead of hd?

    Goran Thyni <goran@bildbasen.se> — 1998-04-23T16:29:22Z

    Vadim B. Mikheev wrote:
> 
> Göran Thyni wrote:
> >
> > Bruce Momjian wrote:
> > > Yes, we could.  I don't think we do because we an anon-mapped region is
> > > the same as shmem, but if the limit is higher for anon mmap, we could
> > > used it.
> >
> > It could be faster too,
> > and it API is cleaner too (IMHO).
> > If we resolve the fork/exec -> just fork problem in the server
> > mmap would make a good improvment in simplicity and probably
> > speed too.
> 
>      MAP_INHERIT
>                  Permit regions to be inherited across execve(2) system calls.
> 
>  - exec isn't problem...
> 
> Vadim

It is on non-BSD systems (I checked Linux 2.0.33 and DG/UX, no go).
And how do you keep the handles after an exec.

BTW, to get rid of exec to more easily replace shmem with mmap is only
part of it,
to use mmap instead of shmem to easier remove the exec is the other
side. :-)

	best regards,
-- 
---------------------------------------------
Göran Thyni, sysadm, JMS Bildbasen, Kiruna

  9. Re: [QUESTIONS] How to use memory instead of hd?

    Bruce Momjian <maillist@candle.pha.pa.us> — 1998-06-16T02:38:46Z

    > 
> Bruce Momjian wrote:
> > 
> > >
> > > Hi All
> > >
> > > I just upgraded to 128MB RAM, and really don't need ALL of it MOST of the
> > > time, and the main reason for it was to help with database speed.
> > >
> > > But then I got to thinking, is there a way to just semi-permently put a
> > > database into RAM? so that when a query is done it goes only to the image
> > > of the database in RAM, never even touching the hard drive.  This would be
> > > enormasly faster on 50 ~ 100 (or what ever) MB databases, especially on
> > > those new boxes with that 6? or 10? nano second RAM.  Especially if one
> > > could pick and chose which table(s) to put in RAM
> > 
> > You can tune your OS to use most of that as buffer cache.  However,
> > writes will be flushed to disk by postgresql fync, or the OS syncing
> > every so often, but not too bad.  You can also up your postgres shared
> > memory buffers, though some OS's have a limit on that.
>   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>                                    ^^^^^^^^^^^^^^^^^^^^
> Could we use mmap (instead of shmem) with MAP_ANON flag to get more memory
> for shared buffer pool ?
> I'm using FreeBSD, man mmap says:
> 
>      MAP_ANON    Map anonymous memory not associated with any specific file.
>                  The file descriptor used for creating MAP_ANON regions is
>                  used only for naming, and may be specified as -1 if no name
>                  is associated with the region.
> 
> Vadim
> 

I am going through my mailbox and I believe we never found a portable
way to allocate shared memory other than system V shared memory. 
Increasing the amount of buffers beyond a certain amount requires a
kernel change.

Has anyone come up with a better way?

-- 
Bruce Momjian                          |  830 Blythe Avenue
maillist@candle.pha.pa.us              |  Drexel Hill, Pennsylvania 19026
  +  If your life is a hard drive,     |  (610) 353-9879(w)
  +  Christ can be your backup.        |  (610) 853-3000(h)