Horton
Menu

Thread

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Add missing space in event_source GUC description.

  1. SYSV shared memory vs mmap performance

    Francois Tigeot <ftigeot@wolfpond.org> — 2012-09-13T06:30:03Z

    Hi,

Given the recent decision to switch from SYSV shared memory to mmap and
the concerns which were made with regard to performance on *BSD kernels,
I've run a few Pgbench tests on a spare Xeon box.

I tested PostgreSQL-9.3 from June 28th, as of commits:
- c5b3451a8e72cb7825933d4f4827f467cb38b498 (mmap)
- 5d594b73d988b1ac78c49d8a84deae6bae876d01 (sysv shared memory)

I also used both Scientific Linux-6.2 and DragonFly BSD-3.1; the results
are in the attached PDF document.

To cut a long story short, Linux doesn't show any difference and DragonFly
sees some heavy degradation under load. After a while, it starts swapping
and performance goes to hell.

The only *BSD system tested was DragonFly but I know from previous pgbench
tests FreeBSD and NetBSD follow a similar performance curve

The famous kern.ipc.shm_use_phys sysctl was set to 1, which is the default
setting.

-- 
Francois Tigeot

  2. Re: SYSV shared memory vs mmap performance

    Bruce Momjian <bruce@momjian.us> — 2013-01-25T13:40:20Z

    On Thu, Sep 13, 2012 at 08:30:03AM +0200, Francois Tigeot wrote:
> Hi,
> 
> Given the recent decision to switch from SYSV shared memory to mmap and
> the concerns which were made with regard to performance on *BSD kernels,
> I've run a few Pgbench tests on a spare Xeon box.
> 
> I tested PostgreSQL-9.3 from June 28th, as of commits:
> - c5b3451a8e72cb7825933d4f4827f467cb38b498 (mmap)
> - 5d594b73d988b1ac78c49d8a84deae6bae876d01 (sysv shared memory)
> 
> I also used both Scientific Linux-6.2 and DragonFly BSD-3.1; the results
> are in the attached PDF document.
> 
> To cut a long story short, Linux doesn't show any difference and DragonFly
> sees some heavy degradation under load. After a while, it starts swapping
> and performance goes to hell.
> 
> The only *BSD system tested was DragonFly but I know from previous pgbench
> tests FreeBSD and NetBSD follow a similar performance curve
> 
> The famous kern.ipc.shm_use_phys sysctl was set to 1, which is the default
> setting.

Just a reminder we might have *BSD performance issues with our use of
Posix shared memory in Postgres 9.3.  I am attaching the PDF the user
posted.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + It's impossible for everything to be true. +

  3. Re: SYSV shared memory vs mmap performance

    Robert Haas <robertmhaas@gmail.com> — 2013-01-25T13:47:51Z

    On Fri, Jan 25, 2013 at 8:40 AM, Bruce Momjian <bruce@momjian.us> wrote:
> Just a reminder we might have *BSD performance issues with our use of
> Posix shared memory in Postgres 9.3.  I am attaching the PDF the user
> posted.

This is a good point.  The question which I believe I asked before and
haven't gotten an answer to is whether there's some way to get the
benefit of shm_use_phys with an anonymous mapping.

It seems to me to be slightly insane to impose draconian shared memory
limits out of the box and then complain when people switch to some
other type of shared memory to get around them.  I realize that
Dragonfly may not be doing that (because I think they may have raised
the default shared-memory limits), but I believe some of the more
mainstream BSDs are.

I suppose we could add a GUC for this, but that's not very appealing, either.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  4. Re: SYSV shared memory vs mmap performance

    Bruce Momjian <bruce@momjian.us> — 2013-01-25T13:53:12Z

    On Fri, Jan 25, 2013 at 08:47:51AM -0500, Robert Haas wrote:
> On Fri, Jan 25, 2013 at 8:40 AM, Bruce Momjian <bruce@momjian.us> wrote:
> > Just a reminder we might have *BSD performance issues with our use of
> > Posix shared memory in Postgres 9.3.  I am attaching the PDF the user
> > posted.
> 
> This is a good point.  The question which I believe I asked before and
> haven't gotten an answer to is whether there's some way to get the
> benefit of shm_use_phys with an anonymous mapping.
> 
> It seems to me to be slightly insane to impose draconian shared memory
> limits out of the box and then complain when people switch to some
> other type of shared memory to get around them.  I realize that
> Dragonfly may not be doing that (because I think they may have raised
> the default shared-memory limits), but I believe some of the more
> mainstream BSDs are.
> 
> I suppose we could add a GUC for this, but that's not very appealing, either.

I think we are going to need keep an eye on this during beta.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + It's impossible for everything to be true. +

  5. Re: SYSV shared memory vs mmap performance

    Francois Tigeot <ftigeot@wolfpond.org> — 2013-01-25T14:38:35Z

    Hi,

Robert Haas wrote:
> On Fri, Jan 25, 2013 at 8:40 AM, Bruce Momjian <bruce@momjian.us>
> wrote:
>> Just a reminder we might have *BSD performance issues with our use
>> of Posix shared memory in Postgres 9.3.  I am attaching the PDF the
>> user posted.
>
> This is a good point.  The question which I believe I asked before
> and haven't gotten an answer to is whether there's some way to get
> the benefit of shm_use_phys with an anonymous mapping.

There is. Postgres 9.3+mmap performance on DragonFly is now much better 
than these old benchmark results show.

After the initial disappointing result, I went on a benchmarking/tuning 
binge with our Dear Leader Matt Dillon. Taking advantage of some 
previous cpu topology work from Mihai Carabas, he heavily improved most 
performance shortcomings we found in the DragonFly kernel.

There were a few mail about this changes on the DragonFly mailing-lists 
and Justin Sherill wrote some interesting articles on his blog.

Some links with more details about improvements and final results:
http://www.shiningsilence.com/dbsdlog/2012/09/19/10403.html
http://www.shiningsilence.com/dbsdlog/2012/10/11/10544.html
http://www.dragonflybsd.org/performance/

> It seems to me to be slightly insane to impose draconian shared
> memory limits out of the box and then complain when people switch to
> some other type of shared memory to get around them.  I realize that
> Dragonfly may not be doing that (because I think they may have
> raised the default shared-memory limits), but I believe some of the
> more mainstream BSDs are.

The original SYSV limits looked like something straight from the 1980s; 
we're now autotuning them on DragonFly.
FreeBSD and NetBSD still needed manual tuning last time I had a look.

-- 
Francois Tigeot

  6. Re: SYSV shared memory vs mmap performance

    Robert Haas <robertmhaas@gmail.com> — 2013-01-25T22:55:19Z

    On Fri, Jan 25, 2013 at 9:38 AM, Francois Tigeot <ftigeot@wolfpond.org> wrote:
>> On Fri, Jan 25, 2013 at 8:40 AM, Bruce Momjian <bruce@momjian.us>
>> wrote:
>>>
>>> Just a reminder we might have *BSD performance issues with our use
>>> of Posix shared memory in Postgres 9.3.  I am attaching the PDF the
>>> user posted.
>>
>> This is a good point.  The question which I believe I asked before
>> and haven't gotten an answer to is whether there's some way to get
>> the benefit of shm_use_phys with an anonymous mapping.
>
> There is. Postgres 9.3+mmap performance on DragonFly is now much better than
> these old benchmark results show.
>
> After the initial disappointing result, I went on a benchmarking/tuning
> binge with our Dear Leader Matt Dillon. Taking advantage of some previous
> cpu topology work from Mihai Carabas, he heavily improved most performance
> shortcomings we found in the DragonFly kernel.
>
> There were a few mail about this changes on the DragonFly mailing-lists and
> Justin Sherill wrote some interesting articles on his blog.
>
> Some links with more details about improvements and final results:
> http://www.shiningsilence.com/dbsdlog/2012/09/19/10403.html
> http://www.shiningsilence.com/dbsdlog/2012/10/11/10544.html
> http://www.dragonflybsd.org/performance/

Well, that looks pretty cool.  Is there anything we can sensibly do to
recover the lost performance on FreeBSD and NetBSD?

>> It seems to me to be slightly insane to impose draconian shared
>> memory limits out of the box and then complain when people switch to
>> some other type of shared memory to get around them.  I realize that
>> Dragonfly may not be doing that (because I think they may have
>> raised the default shared-memory limits), but I believe some of the
>> more mainstream BSDs are.
>
> The original SYSV limits looked like something straight from the 1980s;
> we're now autotuning them on DragonFly.

Awesome!

> FreeBSD and NetBSD still needed manual tuning last time I had a look.

Bummer.  :-(

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  7. Re: SYSV shared memory vs mmap performance

    Francois Tigeot <ftigeot@wolfpond.org> — 2013-01-26T12:00:35Z

    On Fri, Jan 25, 2013 at 05:55:19PM -0500, Robert Haas wrote:
> On Fri, Jan 25, 2013 at 9:38 AM, Francois Tigeot <ftigeot@wolfpond.org> wrote:
> >
> > Some links with more details about improvements and final results:
> > http://www.shiningsilence.com/dbsdlog/2012/09/19/10403.html
> > http://www.shiningsilence.com/dbsdlog/2012/10/11/10544.html
> > http://www.dragonflybsd.org/performance/
> 
> Well, that looks pretty cool.  Is there anything we can sensibly do to
> recover the lost performance on FreeBSD and NetBSD?

Apart from reintroducing SYSV shm usage, I don't think the Postgres team
can do much about it. Mmap really needs kernel work to perform well.

Some FreeBSD people are looking into the issue and have started to run
benchmarks of their own to find out what needs to be fixed; I guess they
will adapt in time.

I'm less optimistic on the NetBSD front: even though I reported major
show-stopper bugs (system died under load and was unable to complete
a pgbench run), no one seemed to care.

-- 
Francois Tigeot

  8. Re: SYSV shared memory vs mmap performance

    Robert Haas <robertmhaas@gmail.com> — 2013-01-27T04:30:49Z

    On Sat, Jan 26, 2013 at 7:00 AM, Francois Tigeot <ftigeot@wolfpond.org> wrote:
> On Fri, Jan 25, 2013 at 05:55:19PM -0500, Robert Haas wrote:
>> On Fri, Jan 25, 2013 at 9:38 AM, Francois Tigeot <ftigeot@wolfpond.org> wrote:
>> >
>> > Some links with more details about improvements and final results:
>> > http://www.shiningsilence.com/dbsdlog/2012/09/19/10403.html
>> > http://www.shiningsilence.com/dbsdlog/2012/10/11/10544.html
>> > http://www.dragonflybsd.org/performance/
>>
>> Well, that looks pretty cool.  Is there anything we can sensibly do to
>> recover the lost performance on FreeBSD and NetBSD?
>
> Apart from reintroducing SYSV shm usage, I don't think the Postgres team
> can do much about it. Mmap really needs kernel work to perform well.
>
> Some FreeBSD people are looking into the issue and have started to run
> benchmarks of their own to find out what needs to be fixed; I guess they
> will adapt in time.
>
> I'm less optimistic on the NetBSD front: even though I reported major
> show-stopper bugs (system died under load and was unable to complete
> a pgbench run), no one seemed to care.

Hrm.  I was hoping that you were going to say "yeah, just pass the
MAP_DONT_BE_STUPID flag, and it'll work great".

Frankly, if the NetBSD developers don't think it's important to make
the test case not crash the system, then I'm not sure there's much
point is us caring about whether the test case performs well.  :-(

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  9. Re: SYSV shared memory vs mmap performance

    YAMAMOTO Takashi <yamt@mwd.biglobe.ne.jp> — 2013-01-28T15:27:28Z

    hi,

> I'm less optimistic on the NetBSD front: even though I reported major
> show-stopper bugs (system died under load and was unable to complete
> a pgbench run), no one seemed to care.

can you give me a pointer?

YAMAMOTO Takashi

  10. Re: SYSV shared memory vs mmap performance

    Francois Tigeot <ftigeot@wolfpond.org> — 2013-01-28T18:49:20Z

    Hi,

On Mon, Jan 28, 2013 at 03:27:28PM +0000, YAMAMOTO Takashi wrote:
> 
> can you give me a pointer?

This bug report for a start:
http://gnats.netbsd.org/cgi-bin/query-pr-single.pl?number=46833

This is the only one I've filled; I also remember having irc discussions
with some netbsd developers about the system getting unresponsive under
load.

-- 
Francois Tigeot

  11. Re: SYSV shared memory vs mmap performance

    YAMAMOTO Takashi <yamt@mwd.biglobe.ne.jp> — 2013-01-31T07:39:25Z

    hi,

> Hi,
> 
> On Mon, Jan 28, 2013 at 03:27:28PM +0000, YAMAMOTO Takashi wrote:
>> 
>> can you give me a pointer?
> 
> This bug report for a start:
> http://gnats.netbsd.org/cgi-bin/query-pr-single.pl?number=46833

thanks!

YAMAMOTO Takashi

> 
> This is the only one I've filled; I also remember having irc discussions
> with some netbsd developers about the system getting unresponsive under
> load.
> 
> -- 
> Francois Tigeot
> 
> 
> -- 
> Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers