Horton

Thread

Commits

  1. Make the different Unix-y semaphore implementations ABI-compatible.

  2. Use unnamed POSIX semaphores, if available, on Linux and FreeBSD.

  1. Back-patch use of unnamed POSIX semaphores for Linux?

    Tom Lane <tgl@sss.pgh.pa.us> — 2016-12-07T02:53:06Z

    Just saw another report of what's probably systemd killing off Postgres'
SysV semaphores, as we've discussed previously at, eg,
https://www.postgresql.org/message-id/flat/57828C31.5060409%40gmail.com
Since the systemd people are generally impervious to suggestions that
they might be mistaken, I do not expect this problem to go away.

I think we should give serious consideration to back-patching commit
ecb0d20a9, which changed the default semaphore type to unnamed-POSIX
on Linux.  We've seen no problems in the buildfarm in the two months
that that's been in HEAD.  If we don't do this, we can expect to
continue seeing complaints of this sort until pre-v10 PG releases
fall out of use ... and I don't want to wait that long.

Commit ecb0d20a9 also changed the default for FreeBSD.  I'm not convinced
we should back-patch that part, because (a) unnamed-POSIX semas have
only been there since FreeBSD 9.0, which isn't that long ago, and (b)
the argument for switching is "it'll perform better" not "your server
will fail randomly without this change".

Comments?

			regards, tom lane

  2. Re: Back-patch use of unnamed POSIX semaphores for Linux?

    Robert Haas <robertmhaas@gmail.com> — 2016-12-07T04:43:56Z

    On Tue, Dec 6, 2016 at 9:53 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Just saw another report of what's probably systemd killing off Postgres'
> SysV semaphores, as we've discussed previously at, eg,
> https://www.postgresql.org/message-id/flat/57828C31.5060409%40gmail.com
> Since the systemd people are generally impervious to suggestions that
> they might be mistaken, I do not expect this problem to go away.
>
> I think we should give serious consideration to back-patching commit
> ecb0d20a9, which changed the default semaphore type to unnamed-POSIX
> on Linux.  We've seen no problems in the buildfarm in the two months
> that that's been in HEAD.  If we don't do this, we can expect to
> continue seeing complaints of this sort until pre-v10 PG releases
> fall out of use ... and I don't want to wait that long.
>
> Commit ecb0d20a9 also changed the default for FreeBSD.  I'm not convinced
> we should back-patch that part, because (a) unnamed-POSIX semas have
> only been there since FreeBSD 9.0, which isn't that long ago, and (b)
> the argument for switching is "it'll perform better" not "your server
> will fail randomly without this change".
>
> Comments?

Urk.  That sounds like a scary thing to back-patch.  The fact that the
buildfarm has reported no problems is good as far as it goes, but user
environments can be expected to be considerably more diverse than the
buildfarm.  I wouldn't mind giving users the option to select unnamed
POSIX semas, but I don't think there's any guarantee that that's 100%
certain to work every place where the current implementation works -
and if not, then people will upgrade to the latest minor release and
everything will completely stop working.  Granted, that might not
happen, because maybe unnamed POSIX semas are one of those really
awesome operating system primitives that never has problems on any
system anywhere ever.  But I think it's pretty hard to be certain of
that.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  3. Re: Back-patch use of unnamed POSIX semaphores for Linux?

    Tom Lane <tgl@sss.pgh.pa.us> — 2016-12-07T04:54:43Z

    Robert Haas <robertmhaas@gmail.com> writes:
> On Tue, Dec 6, 2016 at 9:53 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> I think we should give serious consideration to back-patching commit
>> ecb0d20a9, which changed the default semaphore type to unnamed-POSIX
>> on Linux.

> Urk.  That sounds like a scary thing to back-patch.

I don't deny that it's scary, but the alternative seems to be to be
rather badly broken on systemd-using distros for years to come.
That's pretty scary too.

> ... Granted, that might not
> happen, because maybe unnamed POSIX semas are one of those really
> awesome operating system primitives that never has problems on any
> system anywhere ever.  But I think it's pretty hard to be certain of
> that.

You're attacking a straw man.  I didn't propose changing our behavior
anywhere but Linux.  AFAIK, on that platform unnamed POSIX semaphores
are futexes, which have been a stable feature since 2003 according to
https://en.wikipedia.org/wiki/Futex#History.  Anybody who did need
to compile PG for use with a pre-2.6 kernel could override the default,
anyway.

Now, I did think of a problem we'd have to deal with, which is how
to avoid breaking ABI by changing sizeof(PGSemaphoreData).  I think
that's soluble though.

			regards, tom lane

  4. Re: Back-patch use of unnamed POSIX semaphores for Linux?

    Michael Paquier <michael.paquier@gmail.com> — 2016-12-07T04:57:05Z

    On Wed, Dec 7, 2016 at 1:43 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> Urk.  That sounds like a scary thing to back-patch.  The fact that the
> buildfarm has reported no problems is good as far as it goes, but user
> environments can be expected to be considerably more diverse than the
> buildfarm.  I wouldn't mind giving users the option to select unnamed
> POSIX semas, but I don't think there's any guarantee that that's 100%
> certain to work every place where the current implementation works -
> and if not, then people will upgrade to the latest minor release and
> everything will completely stop working.

Potential risks involving minor upgrades are far higher than the risks
involved by systemd, so -1 for a backpatch seen from here.
-- 
Michael

  5. Re: Back-patch use of unnamed POSIX semaphores for Linux?

    Tatsuo Ishii <ishii@sraoss.co.jp> — 2016-12-07T05:13:41Z

    > Potential risks involving minor upgrades are far higher than the risks
> involved by systemd, so -1 for a backpatch seen from here.

As long as we would have a compile time switch to enable/disable the
back-patched feature, it seems it would be acceptable. In the worst
case, the back-patching could bring disasters, but in that case
packagers could turn off the switch and ship updated version of
packages.

Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp

  6. Re: Back-patch use of unnamed POSIX semaphores for Linux?

    Craig Ringer <craig@2ndquadrant.com> — 2016-12-07T06:18:25Z

    On 7 December 2016 at 10:53, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Just saw another report of what's probably systemd killing off Postgres'
> SysV semaphores, as we've discussed previously at, eg,
> https://www.postgresql.org/message-id/flat/57828C31.5060409%40gmail.com
> Since the systemd people are generally impervious to suggestions that
> they might be mistaken, I do not expect this problem to go away.
>
> I think we should give serious consideration to back-patching commit
> ecb0d20a9, which changed the default semaphore type to unnamed-POSIX
> on Linux.  We've seen no problems in the buildfarm in the two months
> that that's been in HEAD.  If we don't do this, we can expect to
> continue seeing complaints of this sort until pre-v10 PG releases
> fall out of use ... and I don't want to wait that long.
>
> Commit ecb0d20a9 also changed the default for FreeBSD.  I'm not convinced
> we should back-patch that part, because (a) unnamed-POSIX semas have
> only been there since FreeBSD 9.0, which isn't that long ago, and (b)
> the argument for switching is "it'll perform better" not "your server
> will fail randomly without this change".

That's a huge change to make for something that doesn't risk data
corruption, and that won't happen when using postgres with distro
packages.

As I understand it, it's only a problem if you're running postgres as
a normal user, not a "system user" which systemd defines at
compile-time as a user < 500 or < 1000 depending on the distro's
default login.conf . So it'll only affect people who're not using
their distro's packages and service mechanism, and are instead running
Pg under some other user, likely started manually with pg_ctl.

I see quite a few people who compile their own Pg rather than using
packages, and some who even fail to use the init system and instead
use custom scripts to launch Pg. But pretty much everything I've seen
uses a 'postgres' system-user. Clearly there are exceptions out there
in the wild, but I don't think it makes sense to backpatch this to
satisfy people who are, IMO, doing it wrong in the first place.

Especially since those people can reconfigure systemd not to do this
with the RemoveIPC and KillUserProcesses directives if they insist on
using a non-system user.

If they defined a systemd service to start postgres they'd be fine...
and isn't it pretty much basic sysadmin 101 to use your init system to
start services?

Don't get me wrong, I think systemd's behaviour is pretty stupid.
Mostly in terms of its magic definition of a "system user", which
shouldn't be something determined by a uid threshold at compile time.
But I don't think we should double down on it by backpatching a big
change that hasn't even seen in-the-wild loads from real world use
yet, just to make it easier on people who're doing things backwards in
the first place.

If it were possible to detect that systemd was about to clobber us and
log something informative, _that_ would be very nice to backpatch. I
don't see how that's possible though.

-- 
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

  7. Re: Back-patch use of unnamed POSIX semaphores for Linux?

    Magnus Hagander <magnus@hagander.net> — 2016-12-07T09:14:42Z

    On Wed, Dec 7, 2016 at 7:18 AM, Craig Ringer <craig@2ndquadrant.com> wrote:

> On 7 December 2016 at 10:53, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> > Just saw another report of what's probably systemd killing off Postgres'
> > SysV semaphores, as we've discussed previously at, eg,
> > https://www.postgresql.org/message-id/flat/57828C31.5060409%40gmail.com
> > Since the systemd people are generally impervious to suggestions that
> > they might be mistaken, I do not expect this problem to go away.
> >
> > I think we should give serious consideration to back-patching commit
> > ecb0d20a9, which changed the default semaphore type to unnamed-POSIX
> > on Linux.  We've seen no problems in the buildfarm in the two months
> > that that's been in HEAD.  If we don't do this, we can expect to
> > continue seeing complaints of this sort until pre-v10 PG releases
> > fall out of use ... and I don't want to wait that long.
> >
> > Commit ecb0d20a9 also changed the default for FreeBSD.  I'm not convinced
> > we should back-patch that part, because (a) unnamed-POSIX semas have
> > only been there since FreeBSD 9.0, which isn't that long ago, and (b)
> > the argument for switching is "it'll perform better" not "your server
> > will fail randomly without this change".
>
> That's a huge change to make for something that doesn't risk data
> corruption, and that won't happen when using postgres with distro
> packages.
>
> As I understand it, it's only a problem if you're running postgres as
> a normal user, not a "system user" which systemd defines at
> compile-time as a user < 500 or < 1000 depending on the distro's
> default login.conf . So it'll only affect people who're not using
> their distro's packages and service mechanism, and are instead running
> Pg under some other user, likely started manually with pg_ctl.
>
> I see quite a few people who compile their own Pg rather than using
> packages, and some who even fail to use the init system and instead
> use custom scripts to launch Pg. But pretty much everything I've seen
> uses a 'postgres' system-user. Clearly there are exceptions out there
> in the wild, but I don't think it makes sense to backpatch this to
> satisfy people who are, IMO, doing it wrong in the first place.
>
> Especially since those people can reconfigure systemd not to do this
> with the RemoveIPC and KillUserProcesses directives if they insist on
> using a non-system user.
>
> If they defined a systemd service to start postgres they'd be fine...
> and isn't it pretty much basic sysadmin 101 to use your init system to
> start services?
>
> Don't get me wrong, I think systemd's behaviour is pretty stupid.
> Mostly in terms of its magic definition of a "system user", which
> shouldn't be something determined by a uid threshold at compile time.
> But I don't think we should double down on it by backpatching a big
> change that hasn't even seen in-the-wild loads from real world use
> yet, just to make it easier on people who're doing things backwards in
> the first place.
>

+1 (or several).

I don't think we should backpatch something that carries risk for people
who do things "the right way" to help those that don't. Even if the
behavior is stupid.



>
> If it were possible to detect that systemd was about to clobber us and
> log something informative, _that_ would be very nice to backpatch. I
> don't see how that's possible though.


Surely there must be some way to ask systemd about it's configuration? And
if we have that, then we could possibly teach pg_ctl about that and have it
throw a big warning?

-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

  8. Re: Back-patch use of unnamed POSIX semaphores for Linux?

    Peter Eisentraut <peter.eisentraut@2ndquadrant.com> — 2016-12-07T13:54:17Z

    On 12/6/16 9:53 PM, Tom Lane wrote:
> I think we should give serious consideration to back-patching commit
> ecb0d20a9, which changed the default semaphore type to unnamed-POSIX
> on Linux.

Even with that change, dynamic shared memory is still vulnerable to be
removed.  So backpatching the semaphore change wouldn't achieve any new
level of safety for users so that we could tell them, "you're good now".

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  9. Re: Back-patch use of unnamed POSIX semaphores for Linux?

    Stephen Frost <sfrost@snowman.net> — 2016-12-07T14:33:57Z

    All,

* Peter Eisentraut (peter.eisentraut@2ndquadrant.com) wrote:
> On 12/6/16 9:53 PM, Tom Lane wrote:
> > I think we should give serious consideration to back-patching commit
> > ecb0d20a9, which changed the default semaphore type to unnamed-POSIX
> > on Linux.
> 
> Even with that change, dynamic shared memory is still vulnerable to be
> removed.  So backpatching the semaphore change wouldn't achieve any new
> level of safety for users so that we could tell them, "you're good now".

I tend to agree with Peter, Magnus, and Craig on this.  If you aren't
running PG as a system user on a system where systemd feels happy to
kill processes and remove shared memory segments and semaphores when you
log out, no amount of back-patching of anything is going to make you
'safe'.  As noted in the thread referenced, users who are trying to
(mistakenly) do this are already having to modify their logind.conf file
to not have PG outright killed when they log out, it's on them to make
sure systemd doesn't do other stupid things when they log out too if
they really want PG to be run as their user account.

Thanks!

Stephen

  10. Re: Back-patch use of unnamed POSIX semaphores for Linux?

    Tom Lane <tgl@sss.pgh.pa.us> — 2016-12-07T14:38:10Z

    Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:
> On 12/6/16 9:53 PM, Tom Lane wrote:
>> I think we should give serious consideration to back-patching commit
>> ecb0d20a9, which changed the default semaphore type to unnamed-POSIX
>> on Linux.

> Even with that change, dynamic shared memory is still vulnerable to be
> removed.

Really?  I thought we concluded that it is safe because it is detectably
attached to running processes.  The trouble with SysV semaphores is that
they lack any such attachment, so systemd is left to guess whether they
are still in use.

			regards, tom lane

  11. Re: Back-patch use of unnamed POSIX semaphores for Linux?

    Robert Haas <robertmhaas@gmail.com> — 2016-12-07T18:53:34Z

    On Tue, Dec 6, 2016 at 11:54 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Robert Haas <robertmhaas@gmail.com> writes:
>> On Tue, Dec 6, 2016 at 9:53 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>> I think we should give serious consideration to back-patching commit
>>> ecb0d20a9, which changed the default semaphore type to unnamed-POSIX
>>> on Linux.
>
>> Urk.  That sounds like a scary thing to back-patch.
>
> I don't deny that it's scary, but the alternative seems to be to be
> rather badly broken on systemd-using distros for years to come.
> That's pretty scary too.

Why can't this be configurable?

>> ... Granted, that might not
>> happen, because maybe unnamed POSIX semas are one of those really
>> awesome operating system primitives that never has problems on any
>> system anywhere ever.  But I think it's pretty hard to be certain of
>> that.
>
> You're attacking a straw man.  I didn't propose changing our behavior
> anywhere but Linux.  AFAIK, on that platform unnamed POSIX semaphores
> are futexes, which have been a stable feature since 2003 according to
> https://en.wikipedia.org/wiki/Futex#History.  Anybody who did need
> to compile PG for use with a pre-2.6 kernel could override the default,
> anyway.

Changing the behavior even just on Linux leaves plenty of room for
failure, even if the feature itself has been stable.  For example,
there are Linux machines where POSIX shared memory doesn't work, even
though POSIX shared memory is in general a supported feature on Linux
and has been for a long time.   So, if we were to change from System V
shared memory to POSIX shared memory in a minor release, anyone in
that situation would break.  It's hard to be sure the same thing
wouldn't happen in this case.  The fact that the feature's stable
doesn't prove that it works on every system in every configuration.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  12. Re: Back-patch use of unnamed POSIX semaphores for Linux?

    Tom Lane <tgl@sss.pgh.pa.us> — 2016-12-07T20:12:44Z

    Robert Haas <robertmhaas@gmail.com> writes:
> On Tue, Dec 6, 2016 at 11:54 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> Robert Haas <robertmhaas@gmail.com> writes:
>>> On Tue, Dec 6, 2016 at 9:53 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>>> I think we should give serious consideration to back-patching commit
>>>> ecb0d20a9, which changed the default semaphore type to unnamed-POSIX
>>>> on Linux.

>>> Urk.  That sounds like a scary thing to back-patch.

>> I don't deny that it's scary, but the alternative seems to be to be
>> rather badly broken on systemd-using distros for years to come.
>> That's pretty scary too.

> Why can't this be configurable?

It already is.  Note that I said "default".

As things stand, it's only a configure-time choice, but I've been
thinking that we might be well advised to make it run-time configurable.
I do not believe that anyone's still using a Linux version wherein
POSIX semas wouldn't work, but I am not convinced that the same is true
for FreeBSD.  And a configure-run-time test is not a pleasant option
because it doesn't work for cross-compiles.  So really, on platforms
where we think POSIX semas might work, it'd be best to try a sem_init()
during postmaster start and then fall back to SysV if it doesn't work.

But this is all kind of moot if Peter is right that systemd will zap
POSIX shmem along with SysV semaphores.  I've been trying to reproduce
the issue on a Fedora 25 installation, and so far I can't get it to
zap anything, so I'm a bit at a loss how to prove things one way or
the other.

			regards, tom lane

  13. Re: Back-patch use of unnamed POSIX semaphores for Linux?

    Robert Haas <robertmhaas@gmail.com> — 2016-12-07T20:22:56Z

    On Wed, Dec 7, 2016 at 3:12 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Robert Haas <robertmhaas@gmail.com> writes:
>> On Tue, Dec 6, 2016 at 11:54 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>> Robert Haas <robertmhaas@gmail.com> writes:
>>>> On Tue, Dec 6, 2016 at 9:53 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>>>> I think we should give serious consideration to back-patching commit
>>>>> ecb0d20a9, which changed the default semaphore type to unnamed-POSIX
>>>>> on Linux.
>
>>>> Urk.  That sounds like a scary thing to back-patch.
>
>>> I don't deny that it's scary, but the alternative seems to be to be
>>> rather badly broken on systemd-using distros for years to come.
>>> That's pretty scary too.
>
>> Why can't this be configurable?
>
> It already is.  Note that I said "default".
>
> As things stand, it's only a configure-time choice, but I've been
> thinking that we might be well advised to make it run-time configurable.

Sure.  A configure-time choice only benefits people who are compiling
from source, which as far as production is concerned is almost nobody.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  14. Re: Back-patch use of unnamed POSIX semaphores for Linux?

    Alex Hunsaker <badalex@gmail.com> — 2016-12-07T21:37:53Z

    On Wed, Dec 7, 2016 at 1:12 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:


> But this is all kind of moot if Peter is right that systemd will zap
> POSIX shmem along with SysV semaphores.  I've been trying to reproduce
> the issue on a Fedora 25 installation, and so far I can't get it to
> zap anything, so I'm a bit at a loss how to prove things one way or
> the other.
>


Don't know precisely about Fedora 25, but I've had success in the past with:
ssh in as the user
start postgres under tmux/screen
logout
do another ssh login/logout cycle

After logon, you should see "/usr/lib/systemd/systemd --user" running for
that
user. After logout out, said proc should exit. If either of those is not
true,
either systemd is not setup to track sessions (probably via pam) or it
thinks
you still have an active logon. Another way to check if systemd thinks the
user
is logged in is if /run/user/$USER exists.

  15. Re: Back-patch use of unnamed POSIX semaphores for Linux?

    Tom Lane <tgl@sss.pgh.pa.us> — 2016-12-07T22:42:52Z

    Alex Hunsaker <badalex@gmail.com> writes:
> On Wed, Dec 7, 2016 at 1:12 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> But this is all kind of moot if Peter is right that systemd will zap
>> POSIX shmem along with SysV semaphores.  I've been trying to reproduce
>> the issue on a Fedora 25 installation, and so far I can't get it to
>> zap anything, so I'm a bit at a loss how to prove things one way or
>> the other.

> After logon, you should see "/usr/lib/systemd/systemd --user" running for
> that user. After logout out, said proc should exit.

Hmm ... after further experimentation, I still can't get this version of
systemd (231) to do anything evil.  It turns out that Fedora ships it with
KillUserProcesses turned off by default, and maybe having that on is a
prerequisite for the other behavior?  But that doesn't make a lot of sense
because we'd never be seeing the reports of databases moaning about lost
semaphores if the processes got killed first.  Anyway, I see nothing bad
happening if KillUserProcesses is off, while if it's on then the database
gets shut down reasonably politely via SIGTERM.

Color me confused ... maybe systemd's behavior has changed?

			regards, tom lane

  16. Re: Back-patch use of unnamed POSIX semaphores for Linux?

    Andres Freund <andres@anarazel.de> — 2016-12-07T22:46:11Z

    Hi,


On 2016-12-06 21:53:06 -0500, Tom Lane wrote:
> Just saw another report of what's probably systemd killing off Postgres'
> SysV semaphores, as we've discussed previously at, eg,
> https://www.postgresql.org/message-id/flat/57828C31.5060409%40gmail.com
> Since the systemd people are generally impervious to suggestions that
> they might be mistaken, I do not expect this problem to go away.

Would doing so actually solve the systemd issue? Doesn't systemd also
remove SYSV shared memory, which we still use a tiny bit of?


> I think we should give serious consideration to back-patching commit
> ecb0d20a9, which changed the default semaphore type to unnamed-POSIX
> on Linux.  We've seen no problems in the buildfarm in the two months
> that that's been in HEAD.  If we don't do this, we can expect to
> continue seeing complaints of this sort until pre-v10 PG releases
> fall out of use ... and I don't want to wait that long.

I'm a bit uncomfortable backpatching this change, before it has seen
production usage. Both the posix and sysv semaphore implementation has
evolved over the years, with changing performance characteristics. I
don't think it's fair to users to swap a proven solution out for
something that hasn't seen a lot of load.


Greetings,

Andres Freund

  17. Re: Back-patch use of unnamed POSIX semaphores for Linux?

    Andres Freund <andres@anarazel.de> — 2016-12-07T22:51:21Z

    On 2016-12-06 23:54:43 -0500, Tom Lane wrote:
> You're attacking a straw man.  I didn't propose changing our behavior
> anywhere but Linux.  AFAIK, on that platform unnamed POSIX semaphores
> are futexes, which have been a stable feature since 2003 according to
> https://en.wikipedia.org/wiki/Futex#History.  Anybody who did need
> to compile PG for use with a pre-2.6 kernel could override the default,
> anyway.

Back then futexes weren't "robust" though (crash handling and such was
unusable). They only started to be reliable in the ~2007-2008 frame
IIRC.  That still should be ok though.

Regards,

Andres

  18. Re: Back-patch use of unnamed POSIX semaphores for Linux?

    Alvaro Herrera <alvherre@2ndquadrant.com> — 2016-12-07T23:27:46Z

    Tom Lane wrote:

> Hmm ... after further experimentation, I still can't get this version of
> systemd (231) to do anything evil.  It turns out that Fedora ships it with
> KillUserProcesses turned off by default, and maybe having that on is a
> prerequisite for the other behavior?  But that doesn't make a lot of sense
> because we'd never be seeing the reports of databases moaning about lost
> semaphores if the processes got killed first.  Anyway, I see nothing bad
> happening if KillUserProcesses is off, while if it's on then the database
> gets shut down reasonably politely via SIGTERM.
> 
> Color me confused ... maybe systemd's behavior has changed?

https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/ZNQW72UP36UAFMX53HPFFQTWTQDZVJ3M/

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  19. Re: Back-patch use of unnamed POSIX semaphores for Linux?

    Tom Lane <tgl@sss.pgh.pa.us> — 2016-12-07T23:49:57Z

    Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> Tom Lane wrote:
>> Color me confused ... maybe systemd's behavior has changed?

> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/ZNQW72UP36UAFMX53HPFFQTWTQDZVJ3M/

I see Lennart hasn't gotten any less convinced that he's always right
since I left Red Hat :-(

But anyway, it's a demonstrable fact that Fedora 25 has KillUserProcesses
off by default, even though it contains systemd-231.  I assume FESCO
brought down the hammer at some point.

This still doesn't address the real question, which is whether RemoveIPC
does anything if KillUserProcesses is off, and whether that behavior
has changed.  I don't see anything about RemoveIPC in that thread.

			regards, tom lane

  20. Re: Back-patch use of unnamed POSIX semaphores for Linux?

    Robert Haas <robertmhaas@gmail.com> — 2016-12-08T00:03:20Z

    On Wed, Dec 7, 2016 at 6:49 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Alvaro Herrera <alvherre@2ndquadrant.com> writes:
>> Tom Lane wrote:
>>> Color me confused ... maybe systemd's behavior has changed?
>
>> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/ZNQW72UP36UAFMX53HPFFQTWTQDZVJ3M/
>
> I see Lennart hasn't gotten any less convinced that he's always right
> since I left Red Hat :-(

This thread does seem to give that impression.  It's nice to know
there are engineers in the world even more arrogant than we are.  :-)

> But anyway, it's a demonstrable fact that Fedora 25 has KillUserProcesses
> off by default, even though it contains systemd-231.  I assume FESCO
> brought down the hammer at some point.

https://pagure.io/fesco/issue/1600 seems to suggest that it's merely
in abeyance.  (See the first two updates and the last one for the
executive summary.)

> This still doesn't address the real question, which is whether RemoveIPC
> does anything if KillUserProcesses is off, and whether that behavior
> has changed.  I don't see anything about RemoveIPC in that thread.

http://www.dsm.fordham.edu/cgi-bin/man-cgi.pl?topic=logind.conf&ampsect=5
suggests that KillUserProcesses and RemoveIPC are separate cleanup
behaviors triggered by the same underlying cause (termination of last
session).

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  21. Re: Back-patch use of unnamed POSIX semaphores for Linux?

    Tom Lane <tgl@sss.pgh.pa.us> — 2016-12-08T00:39:11Z

    Robert Haas <robertmhaas@gmail.com> writes:
> On Wed, Dec 7, 2016 at 6:49 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> This still doesn't address the real question, which is whether RemoveIPC
>> does anything if KillUserProcesses is off, and whether that behavior
>> has changed.  I don't see anything about RemoveIPC in that thread.

> http://www.dsm.fordham.edu/cgi-bin/man-cgi.pl?topic=logind.conf&ampsect=5
> suggests that KillUserProcesses and RemoveIPC are separate cleanup
> behaviors triggered by the same underlying cause (termination of last
> session).

Yeah, I read that man page too, but ...

The test case I was using was to ssh into the box, launch a
postmaster using the old-school "nohup postmaster &" technique, and
log out.  What I saw was that the "/usr/lib/systemd/systemd --user"
process Alex referred to would be launched when the ssh connection
started, and would stick around as long as the postmaster was there,
if KillUserProcesses was off.  (If it was on, something SIGTERM'd
the postmaster as soon as I disconnected.)  So if they really are
independent behaviors, I'd have expected the same something to have
killed the semaphores as soon as I disconnected; but that did NOT
happen.

[ Yes, RemoveIPC is definitely on: I turned it on explicitly in
logind.conf, just in case the comment claiming it's on by default
is a lie. ]

BTW, I also tried this from the console, but the results were confused
by the fact that GNOME seems to launch approximately a metric buttload
of "helper" processes, which don't disappear when I log out.  If that's
the behavior Lennart is trying to get rid of, I can see his point; but
I tend to agree with the other comments in that thread that this should
be fixed in GNOME not by breaking longstanding working assumptions.

When I get a chance, I think I'll try F24 and see if it behaved
differently.  F23 might be interesting too if it's still downloadable.

			regards, tom lane

  22. Re: Back-patch use of unnamed POSIX semaphores for Linux?

    Alex Hunsaker <badalex@gmail.com> — 2016-12-08T02:28:51Z

    On Wed, Dec 7, 2016 at 3:42 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

>
>
> Hmm ... after further experimentation, I still can't get this version of
> systemd (231) to do anything evil.  It turns out that Fedora ships it with
> KillUserProcesses turned off by default, and maybe having that on is a
> prerequisite for the other behavior?  But that doesn't make a lot of sense
> because we'd never be seeing the reports of databases moaning about lost
> semaphores if the processes got killed first.  Anyway, I see nothing bad
> happening if KillUserProcesses is off, while if it's on then the database
> gets shut down reasonably politely via SIGTERM.
>
> Color me confused ... maybe systemd's behavior has changed?
>

Hrm, the following incantation seems to break for me on a fresh Fedora 25
system:
1) As root su to $USER and start postgres.
2) ssh in as $USER and then logout
3) # psql localhost

FATAL: semctl(4980742, 3, SETVAL, 0) failed: Invalid argument
LOG: server process (PID 14569) exited with exit code 1
...

  23. Re: Back-patch use of unnamed POSIX semaphores for Linux?

    Peter Eisentraut <peter.eisentraut@2ndquadrant.com> — 2016-12-08T13:34:24Z

    On 12/7/16 9:38 AM, Tom Lane wrote:
>> Even with that change, dynamic shared memory is still vulnerable to be
>> removed.
> Really?  I thought we concluded that it is safe because it is detectably
> attached to running processes.

The DSM implementation uses POSIX shared memory, which doesn't have an
attachment count like SysV shared memory.

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  24. Re: Back-patch use of unnamed POSIX semaphores for Linux?

    Peter Eisentraut <peter.eisentraut@2ndquadrant.com> — 2016-12-08T13:44:18Z

    On 12/7/16 9:28 PM, Alex Hunsaker wrote:
> Hrm, the following incantation seems to break for me on a fresh Fedora
> 25 system:
> 1) As root su to $USER and start postgres.
> 2) ssh in as $USER and then logout
> 3) # psql localhost
> 
> FATAL: semctl(4980742, 3, SETVAL, 0) failed: Invalid argument
> LOG: server process (PID 14569) exited with exit code 1

Yeah, the way to trigger this is to run the postgres server not in a
"session", then log in interactively as that same user, thus creating a
session, and then logging out from that session, thus completely logging
out that user from all sessions.

(Thus, the way to trigger the KillUserProcesses behavior is quite the
opposite, because that only happens if you have the postgres server
running in a session.)

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  25. Re: Back-patch use of unnamed POSIX semaphores for Linux?

    Tom Lane <tgl@sss.pgh.pa.us> — 2016-12-08T17:40:27Z

    Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:
> On 12/7/16 9:28 PM, Alex Hunsaker wrote:
>> Hrm, the following incantation seems to break for me on a fresh Fedora
>> 25 system:
>> 1) As root su to $USER and start postgres.
>> 2) ssh in as $USER and then logout
>> 3) # psql localhost
>> 
>> FATAL: semctl(4980742, 3, SETVAL, 0) failed: Invalid argument
>> LOG: server process (PID 14569) exited with exit code 1

> Yeah, the way to trigger this is to run the postgres server not in a
> "session", then log in interactively as that same user, thus creating a
> session, and then logging out from that session, thus completely logging
> out that user from all sessions.

> (Thus, the way to trigger the KillUserProcesses behavior is quite the
> opposite, because that only happens if you have the postgres server
> running in a session.)

Ah-hah, thanks for the insight.  I can now reproduce it, and I confirm
that aside from removing the semaphores, our POSIX shmem segment(s)
are removed from /dev/shm.  They presumably still are attached to whatever
processes have them mapped already, but this behavior is going to break
DSM usage in any case.  (The SysV shm segment does survive, presumably
because systemd notices its positive attach count.)

So I now agree that getting out from under SysV semaphores isn't going to
fix our problems with systemd ... at least, not unless we migrate *to*
not away from SysV shared memory for DSM.  Even then, we'd have to be
careful that standard usage patterns keep every DSM segment continually
attached to at least one process.  Dunno how practical that is.  And it
blows chunks in the goal of not being constrained by SHMMAX.

Oh well.

			regards, tom lane

  26. Re: Back-patch use of unnamed POSIX semaphores for Linux?

    Robert Haas <robertmhaas@gmail.com> — 2016-12-08T17:52:26Z

    On Thu, Dec 8, 2016 at 12:40 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Ah-hah, thanks for the insight.  I can now reproduce it, and I confirm
> that aside from removing the semaphores, our POSIX shmem segment(s)
> are removed from /dev/shm.  They presumably still are attached to whatever
> processes have them mapped already, but this behavior is going to break
> DSM usage in any case.  (The SysV shm segment does survive, presumably
> because systemd notices its positive attach count.)

Make sense.  Actually, it would be fairly unlucky for the DSM thing to
cause a failure for parallel query as it exists today, because there's
only about 4ms between when the segment is created and when all
backends attached.  But DSA - especially if we use it for anything
long-lived - will surely break.

> So I now agree that getting out from under SysV semaphores isn't going to
> fix our problems with systemd ... at least, not unless we migrate *to*
> not away from SysV shared memory for DSM.  Even then, we'd have to be
> careful that standard usage patterns keep every DSM segment continually
> attached to at least one process.  Dunno how practical that is.  And it
> blows chunks in the goal of not being constrained by SHMMAX.

dynamic_shared_memory_type = sysv is already supported, but it's not
the default precisely because of SHMMAX.  Keeping every DSM segment
attached to at least one process is normal for parallel query as it
exists today, but tough for any application that "pins" segments.  On
Windows, we do that: pinning a segment actually causes a user backend
to reach inside the postmaster's address space to open a file
descriptor.  Why Microsoft thought that was something a process should
be able to do to another process I couldn't say.  But on Linux where
segments don't go away on last close, and where such frightening APIs
don't exist, there's no guarantee that a segment will always be open
somewhere.

Hey, I have an idea.  Let's switch from processes to threads, and then
shared memory - including the dynamic kind - can be implemented using
malloc().

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  27. Re: Back-patch use of unnamed POSIX semaphores for Linux?

    Tom Lane <tgl@sss.pgh.pa.us> — 2016-12-10T23:42:50Z

    Robert Haas <robertmhaas@gmail.com> writes:
> On Wed, Dec 7, 2016 at 3:12 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> As things stand, it's only a configure-time choice, but I've been
>> thinking that we might be well advised to make it run-time configurable.

> Sure.  A configure-time choice only benefits people who are compiling
> from source, which as far as production is concerned is almost nobody.

Attached is a proposed patch that unifies the ABI for the three Unix-y
types of semaphores.  I figured it wasn't worth trying to unify Windows
as well, since nobody would ever be doing run-time switching between
Windows and Unix code.

The practical impact of this is that the sem_t or semId/semNum data
is removed from the PGPROC array and placed in a separate array elsewhere
in shared memory.  On 64-bit Linux machines, sem_t is 64 bytes (or at
least, it is on my RHEL6 box), so this change undoes the 56-byte addition
that commit ecb0d20a9 caused.  I think this is probably a performance
win, even though it means an extra indirection to get at the sem_t,
because the speed of access to the sem_t really shouldn't matter: we
only touch that when we're putting a process to sleep or waking it up.
Not bloating PGPROC is probably worth something, though.

It would take additional work to get to being able to do run-time
switching between the Unix semaphore APIs, but that work would now be
localized in posix_sema.c and sysv_sema.c and would not affect code
elsewhere.

I think we need to do at least this much for v10, because otherwise
we'll face ABI issues if an extension is compiled against code with
one semaphore API choice and used with code with a different one.
That didn't use to be a problem because there was really no expectation
of anyone using a non-default semaphore API on any platform, but
I fear it will become an issue if we don't do this.

			regards, tom lane

  28. Re: Back-patch use of unnamed POSIX semaphores for Linux?

    Craig Ringer <craig.ringer@2ndquadrant.com> — 2016-12-11T07:04:27Z

    On 11 Dec. 2016 07:44, "Tom Lane" <tgl@sss.pgh.pa.us> wrote:


I think we need to do at least this much for v10, because otherwise
we'll face ABI issues if an extension is compiled against code with
one semaphore API choice and used with code with a different one.


+1, this is a good idea. Your performance comments make sense too.