Thread

1. mmap (was First set of OSDL Shared Mem scalability results, some wierdness ...

   Aaron <aaron.werman@gmail.com> — 2004-10-15T00:25:36Z

   pg to my mind is unique in not trying to avoid OS buffering. Other
   dbmses spend a substantial effort to create a virtual OS (task
   management, I/O drivers, etc.) both in code and support. Choosing mmap
   seems such a limiting an option - it adds OS dependency and limits
   kernel developer options (2G limits, global mlock serializations,
   porting problems, inability to schedule or parallelize I/O, still
   having to coordinate writers and readers).
   
   More to the point, I think it is very hard to effectively coordinate
   multithreaded I/O, and mmap seems used mostly to manage relatively
   simple scenarios. If the I/O options are:
   - OS (which has enormous investment and is stable, but is general
   purpose with overhead)
   - pg (direct I/O would be costly and potentially destabilizing, but
   with big possible performance rewards)
   - mmap (a feature mostly used to reduce buffer copies in less
   concurrent apps such as image processing that has major architectural
   risk including an order of magnitude more semaphores, but can reduce
   some extra block copies)
   mmap doesn't look that promising.
   
   /Aaron
   
   ----- Original Message ----- 
   From: "Kevin Brown" <kevin@sysexperts.com>
   To: <pgsql-performance@postgresql.org>
   Sent: Thursday, October 14, 2004 4:25 PM
   Subject: Re: [PERFORM] First set of OSDL Shared Mem scalability
   results, some wierdness ...
   
   
   > Tom Lane wrote:
   > > Kevin Brown <kevin@sysexperts.com> writes:
   > > > Tom Lane wrote:
   > > >> mmap() is Right Out because it does not afford us sufficient control
   > > >> over when changes to the in-memory data will propagate to disk.
   > > 
   > > > ... that's especially true if we simply cannot
   > > > have the page written to disk in a partially-modified state (something
   > > > I can easily see being an issue for the WAL -- would the same hold
   > > > true of the index/data files?).
   > > 
   > > You're almost there.  Remember the fundamental WAL rule: log entries
   > > must hit disk before the data changes they describe.  That means that we
   > > need not only a way of forcing changes to disk (fsync) but a way of
   > > being sure that changes have *not* gone to disk yet.  In the existing
   > > implementation we get that by just not issuing write() for a given page
   > > until we know that the relevant WAL log entries are fsync'd down to
   > > disk.  (BTW, this is what the LSN field on every page is for: it tells
   > > the buffer manager the latest WAL offset that has to be flushed before
   > > it can safely write the page.)
   > > 
   > > mmap provides msync which is comparable to fsync, but AFAICS it
   > > provides no way to prevent an in-memory change from reaching disk too
   > > soon.  This would mean that WAL entries would have to be written *and
   > > flushed* before we could make the data change at all, which would
   > > convert multiple updates of a single page into a series of write-and-
   > > wait-for-WAL-fsync steps.  Not good.  fsync'ing WAL once per transaction
   > > is bad enough, once per atomic action is intolerable.
   > 
   > Hmm...something just occurred to me about this.
   > 
   > Would a hybrid approach be possible?  That is, use mmap() to handle
   > reads, and use write() to handle writes?
   > 
   > Any code that wishes to write to a page would have to recognize that
   > it's doing so and fetch a copy from the storage manager (or
   > something), which would look to see if the page already exists as a
   > writeable buffer.  If it doesn't, it creates it by allocating the
   > memory and then copying the page from the mmap()ed area to the new
   > buffer, and returning it.  If it does, it just returns a pointer to
   > the buffer.  There would obviously have to be some bookkeeping
   > involved: the storage manager would have to know how to map a mmap()ed
   > page back to a writeable buffer and vice-versa, so that once it
   > decides to write the buffer it can determine which page in the
   > original file the buffer corresponds to (so it can do the appropriate
   > seek()).
   > 
   > In a write-heavy database, you'll end up with a lot of memory copy
   > operations, but with the scheme we currently use you get that anyway
   > (it just happens in kernel code instead of user code), so I don't see
   > that as much of a loss, if any.  Where you win is in a read-heavy
   > database: you end up being able to read directly from the pages in the
   > kernel's page cache and thus save a memory copy from kernel space to
   > user space, not to mention the context switch that happens due to
   > issuing the read().
   > 
   > 
   > Obviously you'd want to mmap() the file read-only in order to prevent
   > the issues you mention regarding an errant backend, and then reopen
   > the file read-write for the purpose of writing to it.  In fact, you
   > could decouple the two: mmap() the file, then close the file -- the
   > mmap()ed region will remain mapped.  Then, as long as the file remains
   > mapped, you need to open the file again only when you want to write to
   > it.
   > 
   > 
   > -- 
   > Kevin Brown       kevin@sysexperts.com
   > 
   > ---------------------------(end of broadcast)---------------------------
   > TIP 8: explain analyze is your friend
   > 
   -- 
   
   Regards,
   /Aaron
   
   

2. Re: mmap (was First set of OSDL Shared Mem scalability results, some wierdness ...

   Kevin Brown <kevin@sysexperts.com> — 2004-10-15T01:08:41Z

   Aaron Werman wrote:
   > pg to my mind is unique in not trying to avoid OS buffering. Other
   > dbmses spend a substantial effort to create a virtual OS (task
   > management, I/O drivers, etc.) both in code and support. Choosing mmap
   > seems such a limiting an option - it adds OS dependency and limits
   > kernel developer options (2G limits, global mlock serializations,
   > porting problems, inability to schedule or parallelize I/O, still
   > having to coordinate writers and readers).
   
   I'm not sure I entirely agree with this.  Whether you access a file
   via mmap() or via read(), the end result is that you still have to
   access it, and since PG has significant chunks of system-dependent
   code that it heavily relies on as it is (e.g., locking mechanisms,
   shared memory), writing the I/O subsystem in a similar way doesn't
   seem to me to be that much of a stretch (especially since PG already
   has the storage manager), though it might involve quite a bit of work.
   
   As for parallelization of I/O, the use of mmap() for reads should
   signficantly improve parallelization -- now instead of issuing read()
   system calls, possibly for the same set of blocks, all the backends
   would essentially be examining the same data directly.  The
   performance improvements as a result of accessing the kernel's cache
   pages directly instead of having it do buffer copies to process-local
   memory should increase as concurrency goes up.  But see below.
   
   > More to the point, I think it is very hard to effectively coordinate
   > multithreaded I/O, and mmap seems used mostly to manage relatively
   > simple scenarios. 
   
   PG already manages and coordinates multithreaded I/O.  The mechanisms
   used to coordinate writes needn't change at all.  But the way reads
   are done relative to writes might have to be rethought, since an
   mmap()ed buffer always reflects what's actually in kernel space at the
   time the buffer is accessed, while a buffer retrieved via read()
   reflects the state of the file at the time of the read().  If it's
   necessary for the state of the buffers to be fixed at examination
   time, then mmap() will be at best a draw, not a win.
   
   > mmap doesn't look that promising.
   
   This ultimately depends on two things: how much time is spent copying
   buffers around in kernel memory, and how much advantage can be gained
   by freeing up the memory used by the backends to store the
   backend-local copies of the disk pages they use (and thus making that
   memory available to the kernel to use for additional disk buffering).
   The gains from the former are likely small.  The gains from the latter
   are probably also small, but harder to estimate.
   
   The use of mmap() is probably one of those optimizations that should
   be done when there's little else left to optimize, because the
   potential gains are possibly (if not probably) relatively small and
   the amount of work involved may be quite large.
   
   
   So I agree -- compared with other, much lower-hanging fruit, mmap()
   doesn't look promising.
   
   
   
   -- 
   Kevin Brown					      kevin@sysexperts.com
   
   

3. Re: mmap (was First set of OSDL Shared Mem scalability results, some wierdness ...

   Sean Chittenden <sean@chittenden.org> — 2004-10-15T20:09:01Z

   >> pg to my mind is unique in not trying to avoid OS buffering. Other
   >> dbmses spend a substantial effort to create a virtual OS (task
   >> management, I/O drivers, etc.) both in code and support. Choosing mmap
   >> seems such a limiting an option - it adds OS dependency and limits
   >> kernel developer options (2G limits, global mlock serializations,
   >> porting problems, inability to schedule or parallelize I/O, still
   >> having to coordinate writers and readers).
   
   2G limits?  That must be a Linux limitation, not a limitation with 
   mmap(2).  On OS-X and FreeBSD it's anywhere from 4GB to ... well, 
   whatever the 64bit limit is (which is bigger than any data file in 
   $PGDATA).  An mlock(2) serialization problem is going to be cheaper 
   than hitting the disk in nearly all cases and should be no worse than a 
   context switch or semaphore (what we use for the current locking 
   scheme), of which PostgreSQL causes plenty of 'em because it's 
   multi-process, not multi-threaded.  Coordination of data isn't 
   necessary if you mmap(2) data as a private block, which takes a 
   snapshot of the page at the time you make the mmap(2) call and gets 
   copied only when the page is written to.  More on that later.
   
   > I'm not sure I entirely agree with this.  Whether you access a file
   > via mmap() or via read(), the end result is that you still have to
   > access it, and since PG has significant chunks of system-dependent
   > code that it heavily relies on as it is (e.g., locking mechanisms,
   > shared memory), writing the I/O subsystem in a similar way doesn't
   > seem to me to be that much of a stretch (especially since PG already
   > has the storage manager), though it might involve quite a bit of work.
   
   Obviously you have to access the file on the hard drive, but you're 
   forgetting an enormous advantage of mmap(2).  With a read(2) system 
   call, the program has to allocate space for the read(2), then it copies 
   data from the kernel into the allocated memory in the userland's newly 
   allocated memory location.  With mmap(2) there is no second copy.
   
   Let's look at what happens with a read(2) call.  To read(2) data you 
   have to have a block of memory to copy data into.  Assume your OS of 
   choice has a good malloc(3) implementation and it only needs to call 
   brk(2) once to extend the process's memory address after the first 
   malloc(3) call.  There's your first system call, which guarantees one 
   context switch.  The second hit, a much larger hit, is the actual 
   read(2) call itself, wherein the kernel has to copy the data twice: 
   once into a kernel buffer, then from the kernel buffer into the 
   userland's memory space.  Yuk.  Webserver's figured this out long ago 
   that read(2) is slow and evil in terms of performance.  Apache uses 
   mmap(2) to send static files at performance levels that don't suck and 
   is actually quite fast (in terms of responsiveness, I'm not talking 
   about Apache's parallelism/concurrency performance levels... which in 
   1.X aren't great).
   
   mmap(2) is a totally different animal in that you don't ever need to 
   make calls to read(2): mmap(2) is used in place of those calls (With 
   #ifdef and a good abstraction, the rest of PostgreSQL wouldn't know it 
   was working with a page of mmap(2)'ed data or need to know that it is). 
     Instead you mmap(2) a file descriptor and the kernel does some heavy 
   lifting/optimized magic in its VM.  The kernel reads the file 
   descriptor and places the data it reads into its buffer (exactly the 
   same as what happens with read(2)), but, instead of copying the data to 
   the userspace, mmap(2) adjusts the process's address space and maps the 
   address of the kernel buffer into the process's address space.  No 
   copying necessary.  The savings here are *huge*!
   
   Depending on the mmap(2) implementation, the VM may not even get a page 
   from disk until its actually needed.  So, lets say you mmap(2) a 16M 
   file.  The address space picks up an extra 16M of bits that the process 
   *can* use, but doesn't necessarily use.  So if a user reads only ten 
   pages out of a 16MB file, only 10 pages (10 * getpagesize()), or 
   usually 40,960K, which is 0.24% the amount of disk access (((4096 * 10) 
   / (16 *1024 * 1024)) * 100).  Did I forget to mention that if the file 
   is already in the kernel's buffers, there's no need for the kernel to 
   access the hard drive?  Another big win for data that's hot/frequently 
   accessed.
   
   There's another large savings if the machine is doing network IO too...
   
   > As for parallelization of I/O, the use of mmap() for reads should
   > signficantly improve parallelization -- now instead of issuing read()
   > system calls, possibly for the same set of blocks, all the backends
   > would essentially be examining the same data directly.  The
   > performance improvements as a result of accessing the kernel's cache
   > pages directly instead of having it do buffer copies to process-local
   > memory should increase as concurrency goes up.  But see below.
   
   That's kinda true... though not quite correct.  The improvement in IO 
   concurrency comes from zero-socket-copy operations from the disk to the 
   network controller.  If a write(2) system call is issued on a page of 
   mmap(2)'ed data (and your operating system supports it, I know FreeBSD 
   does, but don't think Linux does), then the page of data is DMA'ed by 
   the network controller and sent out without the data needing to be 
   copied into the network controller's buffer.  So, instead of the CPU 
   copying data from the OS's buffer to a kernel buffer, the network card 
   grabs the chunk of data in one interrupt because of the DMA (direct 
   memory access).  This is a pretty big deal for web serving, but if 
   you've got a database sending large sets of data over the network, 
   assuming the network isn't the bottle neck, this results in a heafty 
   performance boost (that won't be noticed by most until they're running 
   huge, very busy installations).  This optimization comes for free and 
   without needing to add one line of code to an application once mmap(2) 
   has been added to an application.
   
   >> More to the point, I think it is very hard to effectively coordinate
   >> multithreaded I/O, and mmap seems used mostly to manage relatively
   >> simple scenarios.
   >
   > PG already manages and coordinates multithreaded I/O.  The mechanisms
   > used to coordinate writes needn't change at all.  But the way reads
   > are done relative to writes might have to be rethought, since an
   > mmap()ed buffer always reflects what's actually in kernel space at the
   > time the buffer is accessed, while a buffer retrieved via read()
   > reflects the state of the file at the time of the read().  If it's
   > necessary for the state of the buffers to be fixed at examination
   > time, then mmap() will be at best a draw, not a win.
   
   Here's where things can get interesting from a transaction stand point. 
     Your statement is correct up until you make the assertion that a page 
   needs to be fixed.  If you're doing a read(2) transaction, mmap(2) a 
   region and set the MAP_PRIVATE flag so the ground won't change 
   underneath you.  No copying of this page is done by the kernel unless 
   it gets written to.  If you're doing a write(2) or are directly 
   scribbling on an mmap(2)'ed page[1], you need to grab some kind of an 
   exclusive lock on the page/file (mlock(2) is going to be no more 
   expensive than a semaphore, but probably less expensive).  We already 
   do that with semaphores, however.  So for databases that don't have 
   high contention for the same page/file of data, there are no additional 
   copies made.  When a piece of data is written, a page is duplicated 
   before it gets scribbled on, but the application never knows this 
   happens.  The next time a process mmap(2)'s a region of memory that's 
   been written to, it'll get the updated data without any need to flush a 
   cache or mark pages as dirty: the operating system does all of this for 
   us (and probably faster too).  mmap(2) implementations are, IMHO, more 
   optimized that shared memory implementations (mmap(2) is a VM function, 
   which gets many eyes to look it over and is always being tuned, whereas 
   shared mem is a bastardized subsystem that works, but isn't integral to 
   any performance areas in the kernel so it gets neglected.  Just my 
   observations from the *BSD commit lists.  Linux it may be different).
   
   [1] I forgot to mention earlier, you don't have to write(2) data to a 
   file if it's mmap(2)'ed, you can change the contents of an mmap(2)'ed 
   region, then msync(2) it back to disk (to ensure it gets written out) 
   or let the last munmap(2) call do that for you (which would be just as 
   dangerous as running without fsync... but would result in an additional 
   performance boost).
   
   >> mmap doesn't look that promising.
   >
   > This ultimately depends on two things: how much time is spent copying
   > buffers around in kernel memory, and how much advantage can be gained
   > by freeing up the memory used by the backends to store the
   > backend-local copies of the disk pages they use (and thus making that
   > memory available to the kernel to use for additional disk buffering).
   
   Someone on IRC pointed me to some OSDL benchmarks, which broke down 
   where time is being spent.  Want to know what the most expensive part 
   of PostgreSQL is?  *drum roll*
   
   http://khack.osdl.org/stp/297960/profile/DBT_2_Profile-tick.sort
   
   3967393 total                                      1.7735
   2331284 default_idle                             36426.3125
   825716 do_sigaction                             1290.1813
   133126 __copy_from_user_ll                      1040.0469
     97780 __copy_to_user_ll                        763.9062
     43135 finish_task_switch                       269.5938
     30973 do_anonymous_page                         62.4456
     24175 scsi_request_fn                           22.2197
     23355 __do_softirq                             121.6406
     17039 __wake_up                                133.1172
     16527 __make_request                            10.8730
      9823 try_to_wake_up                            13.6431
      9525 generic_unplug_device                     66.1458
      8799 find_get_page                             78.5625
      7878 scsi_end_request                          30.7734
   
   Copying data to/from userspace and signal handling!!!!  Let's hear it 
   for the need for mmap(2)!!!  *crowd goes wild*
   
   > The gains from the former are likely small.  The gains from the latter
   > are probably also small, but harder to estimate.
   
   I disagree.
   
   > The use of mmap() is probably one of those optimizations that should
   > be done when there's little else left to optimize, because the
   > potential gains are possibly (if not probably) relatively small and
   > the amount of work involved may be quite large.
   
   If system/kernel time is where most of your database spends its time, 
   then mmap(2) is a huge optimization that is very much worth pursuing.  
   It's stable (nearly all webservers use it, notably Apache), widely 
   deployed, POSIX specified (granted not all implementations are 100% 
   consistent, but that's an OS bug and mmap(2) doesn't have to be turned 
   on for those platforms: it's no worse than where we are now), and well 
   optimized by operating system hackers.  I guarantee that your operating 
   system of choice has a faster VM and disk cache than PostgreSQL's 
   userland cache, nevermind using the OSs buffers leads to many 
   performance boosts as the OS can short-circuit common pathways that 
   would require data copying (ex: zero-socket-copy operations and copying 
   data to/from userland).
   
   mmap(2) isn't a panacea or replacement for good software design, but it 
   certainly does make IO operations vastly faster, which is what 
   PostgreSQL does a lot of (hence its need for a userland cache).  
   Remember, back when PostgreSQL had its architecture thunk up, mmap(2) 
   hardly existed in anyone's eyes, nevermind it being widely used or a 
   POSIX function.  It wasn't until Apache started using it that Operating 
   System vendors felt the need to implement it or make it work well.  Now 
   it's integral to nearly all virtual memory implementations and a modern 
   OS can't live without it or have it broken in any way.  It would be 
   largely beneficial to PostgreSQL to heavily utilize mmap(2).
   
   A few places it should be used include:
   
   *) Storage.  It is a good idea to mmap(2) all files instead of 
   read(2)'ing files.  mmap(2) doesn't fetch a page from disk until its 
   actually needed, which is a nifty savings.  Sure it causes a fault in 
   the kernel, but it won't the second time that page is accessed.  
   Changes are necessary to src/backend/storage/file/, possibly 
   src/backend/storage/freespace/ (why is it using fread(3) and not 
   read(2)?), src/backend/storage/large_object/ can remain gimpy since 
   people should use BYTEA instead (IMHO), src/backend/storage/page/ 
   doesn't need changes (I don't think), src/backend/storage/smgr/ 
   shouldn't need any modifications either.
   
   *) ARC.  Why unmmap(2) data if you don't need to?  With ARC, it's 
   possible for the database to coach the operating system in what pages 
   should be persistent.  ARC's a smart algorithm for handling the needs 
   of a database.  Instead of having a cache of pages in userland, 
   PostgreSQL would have a cache of mmap(2)'ed pages.  It's shared between 
   processes, the changes are public to external programs read(2)'ing 
   data, and its quick.  The needs for shared memory by the kernel drops 
   to nearly nothing.  The needs for mmap(2)'able space in the kernel, 
   however, does go up.  Unlike SysV shared mem, this can normally be 
   changed on the fly.  The end result would be, if a page is needed, it 
   checks to see if its in the cache.  If it is, the mmap(2)'ed page is 
   returned.  If it isn't, the page gets read(2)/mmap(2) like it currently 
   is loaded (except in the mmap(2) case where after the data has been 
   loaded, the page gets munmap(2)'ed).  If ARC decides to keep the page, 
   the page doesn't get munmap(2)'ed.  I don't think any changes need to 
   be made though to take advantage of mmap(2) if the changes are made in 
   the places mentioned above in the Storage point.
   
   
   A few other perks:
   
   *) DIRECTIO can be used without much of a cache coherency headache 
   since the cache of data is in the kernel, not userland.
   
   *) NFS.  I'm not suggesting multiple clients use the same data 
   directory via NFS (unless read only), but if there were a single client 
   accessing a data directory over NFS, performance would be much better 
   than it is today because data consistency is handled by the kernel so 
   in flight packets for writes that get dropped or lost won't cause a 
   slow down (mmap(2) behaves differently with NFS pages) or corruption.
   
   *) mmap(2) is conditional on the operating system's abilities, but 
   doesn't require any architectural changes.  It does change the location 
   of the cache, from being in the userland, down in to the kernel.  This 
   is a change for database administrators, but a good one, IMHO.  
   Previously, the operating system would be split 25% kernel, 75% user 
   because PostgreSQL would need the available RAM for its cache.  Now, 
   that can be moved closer to the opposite, 75% kernel, 25% user because 
   most of the memory is mmap(2)'ed pages instead of actual memory in the 
   userland.
   
   *) Pages can be protected via PROT_(EXEC|READ|WRITE).  For backends 
   that aren't making changes to the DDL or system catalogs (permissions, 
   etc.), pages that are loaded from the catalogs could be loaded with the 
   protection PROT_READ, which would prevent changes to the catalogs.  All 
   DDL and permission altering commands (anything that touches the system 
   catalogs) would then load the page with the PROT_WRITE bit set, make 
   their changes, then PROT_READ the page again.  This would provide a 
   first line of defense against buggy programs or exploits.
   
   *) Eliminates the double caching done currently (caching in PostgreSQL 
   and the kernel) by pushing the cache into the kernel... but without 
   PostgreSQL knowing it's working on a page that's in the kernel.
   
   Please ask questions if you have them.
   
   -sc
   
   -- 
   Sean Chittenden
   
   
   

4. Re: mmap (was First set of OSDL Shared Mem scalability results, some wierdness ...

   Mark Wong <markw@osdl.org> — 2004-10-15T20:39:47Z

   On Fri, Oct 15, 2004 at 01:09:01PM -0700, Sean Chittenden wrote:
   [snip]
   > >
   > > This ultimately depends on two things: how much time is spent copying
   > > buffers around in kernel memory, and how much advantage can be gained
   > > by freeing up the memory used by the backends to store the
   > > backend-local copies of the disk pages they use (and thus making that
   > > memory available to the kernel to use for additional disk buffering).
   > 
   > Someone on IRC pointed me to some OSDL benchmarks, which broke down 
   > where time is being spent.  Want to know what the most expensive part 
   > of PostgreSQL is?  *drum roll*
   > 
   > http://khack.osdl.org/stp/297960/profile/DBT_2_Profile-tick.sort
   > 
   > 3967393 total                                      1.7735
   > 2331284 default_idle                             36426.3125
   > 825716 do_sigaction                             1290.1813
   > 133126 __copy_from_user_ll                      1040.0469
   >   97780 __copy_to_user_ll                        763.9062
   >   43135 finish_task_switch                       269.5938
   >   30973 do_anonymous_page                         62.4456
   >   24175 scsi_request_fn                           22.2197
   >   23355 __do_softirq                             121.6406
   >   17039 __wake_up                                133.1172
   >   16527 __make_request                            10.8730
   >    9823 try_to_wake_up                            13.6431
   >    9525 generic_unplug_device                     66.1458
   >    8799 find_get_page                             78.5625
   >    7878 scsi_end_request                          30.7734
   > 
   > Copying data to/from userspace and signal handling!!!!  Let's hear it 
   > for the need for mmap(2)!!!  *crowd goes wild*
   > 
   [snip]
   
   I know where the do_sigaction is coming from in this particular case.
   Manfred Spraul tracked it to a pair of pgsignal calls in libpq.
   Commenting out those two calls out virtually eliminates do_sigaction from
   the kernel profile for this workload.  I've lost track of the discussion
   over the past year, but I heard a rumor that it was finally addressed to
   some degree.  I did understand it touched on a lot of other things, but
   can anyone summarize where that discussion has gone?
   
   Mark
   
   

5. Re: mmap (was First set of OSDL Shared Mem scalability results, some wierdness ...

   Tom Lane <tgl@sss.pgh.pa.us> — 2004-10-15T21:22:26Z

   Sean Chittenden <sean@chittenden.org> writes:
   > Coordination of data isn't 
   > necessary if you mmap(2) data as a private block, which takes a 
   > snapshot of the page at the time you make the mmap(2) call and gets 
   > copied only when the page is written to.  More on that later.
   
   We cannot move to a model where different backends have different
   views of the same page, which seems to me to be inherent in the idea of
   using MAP_PRIVATE for anything.  To take just one example, a backend
   that had mapped one btree index page some time ago could get completely
   confused if that page splits, because it might see the effects of the
   split in nearby index pages but not in the one that was split.  Or it
   could follow an index link to a heap entry that isn't there anymore,
   or miss an entry it should have seen.  MVCC doesn't save you from this
   because btree adjustments happen below the level of transactions.
   
   However the really major difficulty with using mmap is that it breaks
   the scheme we are currently using for WAL, because you don't have any
   way to restrict how soon a change in an mmap'd page will go to disk.
   (No, I don't believe that mlock guarantees this.  It says that the
   page will not be removed from main memory; it does not specify that,
   say, the syncer won't write the contents out anyway.)
   
   > Let's look at what happens with a read(2) call.  To read(2) data you 
   > have to have a block of memory to copy data into.  Assume your OS of 
   > choice has a good malloc(3) implementation and it only needs to call 
   > brk(2) once to extend the process's memory address after the first 
   > malloc(3) call.  There's your first system call, which guarantees one 
   > context switch.
   
   Wrong.  Our reads occur into shared memory allocated at postmaster
   startup, remember?
   
   > mmap(2) is a totally different animal in that you don't ever need to 
   > make calls to read(2): mmap(2) is used in place of those calls (With 
   > #ifdef and a good abstraction, the rest of PostgreSQL wouldn't know it 
   > was working with a page of mmap(2)'ed data or need to know that it is). 
   
   Instead, you have to worry about address space management and keeping a
   consistent view of the data.
   
   > ... If a write(2) system call is issued on a page of 
   > mmap(2)'ed data (and your operating system supports it, I know FreeBSD 
   > does, but don't think Linux does), then the page of data is DMA'ed by 
   > the network controller and sent out without the data needing to be 
   > copied into the network controller's buffer.
   
   Perfectly irrelevant to Postgres, since there is no situation where we'd
   ever write directly from a disk buffer to a socket; in the present
   implementation there are at least two levels of copy needed in between
   (datatype-specific output function and protocol message assembly).  And
   that's not even counting the fact that any data item large enough to
   make the savings interesting would have been sliced, diced, and
   compressed by TOAST.
   
   > ... If you're doing a write(2) or are directly 
   > scribbling on an mmap(2)'ed page[1], you need to grab some kind of an 
   > exclusive lock on the page/file (mlock(2) is going to be no more 
   > expensive than a semaphore, but probably less expensive).
   
   More incorrect information.  The locking involved here is done by
   LWLockAcquire, which is significantly *less* expensive than a kernel
   call in the case where there is no need to block.  (If you have to
   block, any kernel call to do so is probably about as bad as any other.)
   Switching over to mlock would likely make things considerably slower.
   In any case, you didn't actually mean to say mlock did you?  It doesn't
   lock pages against writes by other processes AFAICS.
   
   > shared mem is a bastardized subsystem that works, but isn't integral to 
   > any performance areas in the kernel so it gets neglected.
   
   What performance issues do you think shared memory needs to have fixed?
   We don't issue any shmem kernel calls after the initial shmget, so
   comparing the level of kernel tenseness about shmget to the level of
   tenseness about mmap is simply irrelevant.  Perhaps the reason you don't
   see any traffic about this on the kernel lists is that shared memory
   already works fine and doesn't need any fixing.
   
   > Please ask questions if you have them.
   
   Do you have any arguments that are actually convincing?  What I just
   read was a proposal to essentially throw away not only the entire
   low-level data access model, but the entire low-level locking model,
   and start from scratch.  There is no possible way we could support both
   this approach and the current one, which means that we'd be permanently
   dropping support for all platforms without high-quality mmap
   implementations; and despite your enthusiasm I don't think that that
   category includes every interesting platform.  Furthermore, you didn't
   give any really convincing reasons to think that the enormous effort
   involved would be repaid.  Those oprofile reports Josh just put up
   showed 3% of the CPU time going into userspace/kernelspace copying.
   Even assuming that that number consists entirely of reads and writes of
   shared buffers (and of course no other kernel call ever transfers any
   data across that boundary ;-)), there's no way we are going to buy into
   this sort of project in hopes of a 3% win.
   
   			regards, tom lane
   
   

6. Re: mmap (was First set of OSDL Shared Mem scalability results, some wierdness ...

   Tom Lane <tgl@sss.pgh.pa.us> — 2004-10-15T21:37:50Z

   Mark Wong <markw@osdl.org> writes:
   > I know where the do_sigaction is coming from in this particular case.
   > Manfred Spraul tracked it to a pair of pgsignal calls in libpq.
   > Commenting out those two calls out virtually eliminates do_sigaction from
   > the kernel profile for this workload.
   
   Hmm, I suppose those are the ones associated with suppressing SIGPIPE
   during send().  It looks to me like those should go away in 8.0 if you
   have compiled with ENABLE_THREAD_SAFETY ... exactly how is PG being
   built in the current round of tests?
   
   			regards, tom lane
   
   

7. Re: mmap (was First set of OSDL Shared Mem scalability results, some wierdness ...

   Mark Wong <markw@osdl.org> — 2004-10-15T21:56:30Z

   On Fri, Oct 15, 2004 at 05:37:50PM -0400, Tom Lane wrote:
   > Mark Wong <markw@osdl.org> writes:
   > > I know where the do_sigaction is coming from in this particular case.
   > > Manfred Spraul tracked it to a pair of pgsignal calls in libpq.
   > > Commenting out those two calls out virtually eliminates do_sigaction from
   > > the kernel profile for this workload.
   > 
   > Hmm, I suppose those are the ones associated with suppressing SIGPIPE
   > during send().  It looks to me like those should go away in 8.0 if you
   > have compiled with ENABLE_THREAD_SAFETY ... exactly how is PG being
   > built in the current round of tests?
   > 
   
   Ah, yes.  Ok.  It's not being configured with any options.  That'll be easy to
   rememdy though.  I'll get that change made and we can try again.
   
   Mark
   
   

8. Re: mmap (was First set of OSDL Shared Mem scalability results,

   Bruce Momjian <pgman@candle.pha.pa.us> — 2004-10-16T01:22:03Z

   Tom Lane wrote:
   > Mark Wong <markw@osdl.org> writes:
   > > I know where the do_sigaction is coming from in this particular case.
   > > Manfred Spraul tracked it to a pair of pgsignal calls in libpq.
   > > Commenting out those two calls out virtually eliminates do_sigaction from
   > > the kernel profile for this workload.
   > 
   > Hmm, I suppose those are the ones associated with suppressing SIGPIPE
   > during send().  It looks to me like those should go away in 8.0 if you
   > have compiled with ENABLE_THREAD_SAFETY ... exactly how is PG being
   > built in the current round of tests?
   
   Yes, those calls are gone in 8.0 with --enable-thread-safety and were
   added specifically because of Manfred's reports.
   
   -- 
     Bruce Momjian                        |  http://candle.pha.pa.us
     pgman@candle.pha.pa.us               |  (610) 359-1001
     +  If your life is a hard drive,     |  13 Roberts Road
     +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073
   
   

9. Re: mmap (was First set of OSDL Shared Mem scalability results, some wierdness ...

   Mark Wong <markw@osdl.org> — 2004-10-18T15:28:10Z

   On Fri, Oct 15, 2004 at 09:22:03PM -0400, Bruce Momjian wrote:
   > Tom Lane wrote:
   > > Mark Wong <markw@osdl.org> writes:
   > > > I know where the do_sigaction is coming from in this particular case.
   > > > Manfred Spraul tracked it to a pair of pgsignal calls in libpq.
   > > > Commenting out those two calls out virtually eliminates do_sigaction from
   > > > the kernel profile for this workload.
   > > 
   > > Hmm, I suppose those are the ones associated with suppressing SIGPIPE
   > > during send().  It looks to me like those should go away in 8.0 if you
   > > have compiled with ENABLE_THREAD_SAFETY ... exactly how is PG being
   > > built in the current round of tests?
   > 
   > Yes, those calls are gone in 8.0 with --enable-thread-safety and were
   > added specifically because of Manfred's reports.
   > 
   
   Ok, I had the build commands changed for installing PostgreSQL in STP.
   The do_sigaction call isn't at the top of the profile anymore, here's
   a reference for those who are interested; it should have the same test
   parameters as the one Tom referenced a little earlier:
   	http://khack.osdl.org/stp/298230/
   
   Mark
   
   

10. Re: mmap (was First set of OSDL Shared Mem scalability results, some wierdness ...

    Sean Chittenden <sean@chittenden.org> — 2004-10-21T20:29:34Z

    > However the really major difficulty with using mmap is that it breaks
    > the scheme we are currently using for WAL, because you don't have any
    > way to restrict how soon a change in an mmap'd page will go to disk.
    > (No, I don't believe that mlock guarantees this.  It says that the
    > page will not be removed from main memory; it does not specify that,
    > say, the syncer won't write the contents out anyway.)
    
    I had to think about this for a minute (now nearly a week) and reread 
    the docs on WAL before I groked what could happen here.  You're 
    absolutely right in that WAL needs to be taken into account first.  How 
    does this execution path sound to you?
    
    By default, all mmap(2)'ed pages are MAP_SHARED.  There are no 
    complications with regards to reads.
    
    When a backend wishes to write a page, the following steps are taken:
    
    1) Backend grabs a lock from the lockmgr to write to the page (exactly 
    as it does now)
    
    2) Backend mmap(2)'s a second copy of the page(s) being written to, 
    this time with the MAP_PRIVATE flag set.  Mapping a copy of the page 
    again is wasteful in terms of address space, but does not require any 
    more memory than our current scheme.  The re-mapping of the page with 
    MAP_PRIVATE prevents changes to the data that other backends are 
    viewing.
    
    3) The writing backend, can then scribble on its private copy of the 
    page(s) as it sees fit.
    
    4) Once completed making changes and a transaction is to be committed, 
    the backend WAL logs its changes.
    
    5) Once the WAL logging is complete and it has hit the disk, the 
    backend msync(2)'s its private copy of the pages to disk (ASYNC or 
    SYNC, it doesn't really matter too much to me).
    
    6) Optional(?).  I'm not sure whether or not the backend would need to 
    also issues an msync(2) MS_INVALIDATE, but, I suspect it would not need 
    to on systems with unified buffer caches such as FreeBSD or OS-X.  On 
    HPUX, or other older *NIX'es, it may be necessary.  *shrug*  I could be 
    trying to be overly protective here.
    
    7) Backend munmap(2)'s its private copy of the written on page(s).
    
    8) Backend releases its lock from the lockmgr.
    
    At this point, the remaining backends now are able to see the updated 
    pages of data.
    
    >> Let's look at what happens with a read(2) call.  To read(2) data you
    >> have to have a block of memory to copy data into.  Assume your OS of
    >> choice has a good malloc(3) implementation and it only needs to call
    >> brk(2) once to extend the process's memory address after the first
    >> malloc(3) call.  There's your first system call, which guarantees one
    >> context switch.
    >
    > Wrong.  Our reads occur into shared memory allocated at postmaster
    > startup, remember?
    
    Doh.  Fair enough.  In most programs that involve read(2), a call to 
    alloc(3) needs to be made.
    
    >> mmap(2) is a totally different animal in that you don't ever need to
    >> make calls to read(2): mmap(2) is used in place of those calls (With
    >> #ifdef and a good abstraction, the rest of PostgreSQL wouldn't know it
    >> was working with a page of mmap(2)'ed data or need to know that it 
    >> is).
    >
    > Instead, you have to worry about address space management and keeping a
    > consistent view of the data.
    
    Which is largely handled by mmap() and the VM.
    
    >> ... If a write(2) system call is issued on a page of
    >> mmap(2)'ed data (and your operating system supports it, I know FreeBSD
    >> does, but don't think Linux does), then the page of data is DMA'ed by
    >> the network controller and sent out without the data needing to be
    >> copied into the network controller's buffer.
    >
    > Perfectly irrelevant to Postgres, since there is no situation where 
    > we'd
    > ever write directly from a disk buffer to a socket; in the present
    > implementation there are at least two levels of copy needed in between
    > (datatype-specific output function and protocol message assembly).  And
    > that's not even counting the fact that any data item large enough to
    > make the savings interesting would have been sliced, diced, and
    > compressed by TOAST.
    
    The biggest winners will be columns whos storage type is PLAIN or 
    EXTERNAL.  writev(2) from mmap(2)'ed pages and non-mmap(2)'ed pages 
    would be a nice perk too (not sure if PostgreSQL uses this or not).  
    Since compression isn't happening on most tuples under 1K in size and 
    most tuples in a database are going to be under that, most tuples are 
    going to be uncompressed.  Total pages for the database, however, is 
    likely a different story.  For large tuples that are uncompressed and 
    larger than a page, it is probably beneficial to use sendfile(2) 
    instead of mmap(2) + write(2)'ing the page/file.
    
    If a large tuple is compressed, it'd be interesting to see if it'd be 
    worthwhile to have the data uncompressed onto an anonymously mmap(2)'ed 
    page(s) that way the benefits of zero-socket-copies could be used.
    
    >> shared mem is a bastardized subsystem that works, but isn't integral 
    >> to
    >> any performance areas in the kernel so it gets neglected.
    >
    > What performance issues do you think shared memory needs to have fixed?
    > We don't issue any shmem kernel calls after the initial shmget, so
    > comparing the level of kernel tenseness about shmget to the level of
    > tenseness about mmap is simply irrelevant.  Perhaps the reason you 
    > don't
    > see any traffic about this on the kernel lists is that shared memory
    > already works fine and doesn't need any fixing.
    
    I'm gunna get flamed for this, but I think its improperly used as a 
    second level cache on top of the operating system's cache.  mmap(2) 
    would consolidate all caching into the kernel.
    
    >> Please ask questions if you have them.
    >
    > Do you have any arguments that are actually convincing?
    
    Three things come to mind.
    
    1) A single cache for pages
    2) Ability to give access hints to the kernel regarding future IO
    3) On the fly memory use for a cache.  There would be no need to 
    preallocate slabs of shared memory on startup.
    
    And a more minor point would be:
    
    4) Not having shared pages get lost when the backend dies (mmap(2) uses 
    refcounts and cleans itself up, no need for ipcs/ipcrm/ipcclean).  This 
    isn't too practical in production though, but it sucks doing PostgreSQL 
    development on OS-X because there is no ipcs/ipcrm command.
    
    > What I just read was a proposal to essentially throw away not only the 
    > entire
    > low-level data access model, but the entire low-level locking model,
    > and start from scratch.
    
     From the above list, steps 2, 3, 5, 6, and 7 would be different than 
    our current approach, all of which could be safely handled with some 
    #ifdef's on platforms that don't have mmap(2).
    
    > There is no possible way we could support both
    > this approach and the current one, which means that we'd be permanently
    > dropping support for all platforms without high-quality mmap
    > implementations;
    
    Architecturally, I don't see anything different or incompatibilities 
    that aren't solved with an #ifdef USE_MMAP/#else/#endif.
    
    > Furthermore, you didn't
    > give any really convincing reasons to think that the enormous effort
    > involved would be repaid.
    
    Steven's has a great reimplementaion of cat(1) that uses mmap(1) and 
    benchmarks the two.  I did my own version of that here:
    
    http://people.freebsd.org/~seanc/mmap_test/
    
    When read(2)'ing/write(2)'ing /etc/services 100,000 times without 
    mmap(2), it takes 82 seconds.  With mmap(2), it takes anywhere from 1.1 
    to 18 seconds.  Worst case scenario with mmap(2) yields a speedup by a 
    factor of four.  Best case scenario...  *shrug* something better than 
    4x.  I doubt PostgreSQL would see 4x speedups in the IO department, but 
    I do think it would be vastly greater than the 3% suggested.
    
    > Those oprofile reports Josh just put up
    > showed 3% of the CPU time going into userspace/kernelspace copying.
    > Even assuming that that number consists entirely of reads and writes of
    > shared buffers (and of course no other kernel call ever transfers any
    > data across that boundary ;-)), there's no way we are going to buy into
    > this sort of project in hopes of a 3% win.
    
    Would it be helpful if I created a test program that demonstrated that 
    the execution path for writing mmap(2)'ed pages as outlined above?
    
    -sc
    
    -- 
    Sean Chittenden
    
    
    

11. Re: mmap (was First set of OSDL Shared Mem scalability results, some wierdness ...

    Tom Lane <tgl@sss.pgh.pa.us> — 2004-10-22T04:12:44Z

    Sean Chittenden <sean@chittenden.org> writes:
    > When a backend wishes to write a page, the following steps are taken:
    > ...
    > 2) Backend mmap(2)'s a second copy of the page(s) being written to, 
    > this time with the MAP_PRIVATE flag set.
    > ...
    > 5) Once the WAL logging is complete and it has hit the disk, the 
    > backend msync(2)'s its private copy of the pages to disk (ASYNC or 
    > SYNC, it doesn't really matter too much to me).
    
    My man page for mmap says that changes in a MAP_PRIVATE region are
    private; they do not affect the file at all, msync or no.  So I don't
    think the above actually works.
    
    In any case, this scheme still forces you to flush WAL records to disk
    before making the changed page visible to other backends, so I don't
    see how it improves the situation.  In the existing scheme we only have
    to fsync WAL at (1) transaction commit, (2) when we are forced to write
    a page out from shared buffers because we are short of buffers, or (3)
    checkpoint.  Anything that implies an fsync per atomic action is going
    to be a loser.  It does not matter how great your kernel API is if you
    only get to perform one atomic action per disk rotation :-(
    
    The important point here is that you can't postpone making changes at
    the page level visible to other backends; there's no MVCC at this level.
    Consider for example two backends wanting to insert a new row.  If they
    both MAP_PRIVATE the same page, they'll probably choose the same tuple
    slot on the page to insert into (certainly there is nothing to stop that
    from happening).  Now you have conflicting definitions for the same
    CTID, not to mention probably conflicting uses of the page's physical
    free space; disaster ensues.  So "atomic action" really means "lock
    page, make changes, add WAL record to in-memory WAL buffers, unlock
    page" with the understanding that as soon as you unlock the page the
    changes you've made in it are visible to all other backends.  You
    *can't* afford to put a WAL fsync in this sequence.
    
    You could possibly buy back most of the lossage in this scenario if
    there were some efficient way for a backend to hold the low-level lock
    on a page just until some other backend wanted to modify the page;
    whereupon the previous owner would have to do what's needed to make his
    changes visible before releasing the lock.  Given the right access
    patterns you don't have to fsync very often (though given the wrong
    access patterns you're still in deep trouble).  But we don't have any
    such mechanism and I think the communication costs of one would be
    forbidding.
    
    > [ much snipped ]
    > 4) Not having shared pages get lost when the backend dies (mmap(2) uses 
    > refcounts and cleans itself up, no need for ipcs/ipcrm/ipcclean).
    
    Actually, that is not a bug that's a feature.  One of the things that
    scares me about mmap is that a crashing backend is able to scribble all
    over live disk buffers before it finally SEGV's (think about memcpy gone
    wrong and similar cases).  In our existing scheme there's a pretty good
    chance that we will be able to commit hara-kiri before any of the
    trashed data gets written out.  In an mmap scheme, it's time to dig out
    your backup tapes, because there simply is no distinction between
    transient and permanent data --- the kernel has no way to know that you
    didn't mean it.
    
    In short, I remain entirely unconvinced that mmap is of any interest to us.
    
    			regards, tom lane