Horton

Thread

Commits

  1. Make EXEC_BACKEND more convenient on Linux and FreeBSD.

  2. Make EXEC_BACKEND more convenient on macOS.

  1. Worth using personality(ADDR_NO_RANDOMIZE) for EXEC_BACKEND on linux?

    Andres Freund <andres@anarazel.de> — 2021-08-06T03:29:44Z

    Hi,

When testing EXEC_BACKEND on linux I see occasional test failures as long as I
don't disable ASLR. There's a code comment to that effect:

	 * If testing EXEC_BACKEND on Linux, you should run this as root before
	 * starting the postmaster:
	 *
	 * echo 0 >/proc/sys/kernel/randomize_va_space

but I don't like doing that on a system wide basis.

Linux allows disabling ASLR on a per-process basis using
personality(ADDR_NO_RANDOMIZE). There's a wrapper binary to do that as well,
setarch --addr-no-randomize.

I was wondering if we should have postmaster do personality(ADDR_NO_RANDOMIZE)
for EXEC_BACKEND builds? It seems nicer to make it automatically work than
have people remember that they need to call "setarch --addr-no-randomize make check".

Not that it actually matters for EXEC_BACKEND, but theoretically doing
personality(ADDR_NO_RANDOMIZE) in postmaster is a tad more secure than doing
it via setarch, as in the personality() case postmaster's layout itself is
still randomized...


Or perhaps we should just add a comment mentioning setarch.

Greetings,

Andres Freund

  2. Re: Worth using personality(ADDR_NO_RANDOMIZE) for EXEC_BACKEND on linux?

    Andrew Dunstan <andrew@dunslane.net> — 2021-08-06T12:48:25Z

    On 8/5/21 11:29 PM, Andres Freund wrote:
> Hi,
>
> When testing EXEC_BACKEND on linux I see occasional test failures as long as I
> don't disable ASLR. There's a code comment to that effect:
>
> 	 * If testing EXEC_BACKEND on Linux, you should run this as root before
> 	 * starting the postmaster:
> 	 *
> 	 * echo 0 >/proc/sys/kernel/randomize_va_space
>
> but I don't like doing that on a system wide basis.
>
> Linux allows disabling ASLR on a per-process basis using
> personality(ADDR_NO_RANDOMIZE). There's a wrapper binary to do that as well,
> setarch --addr-no-randomize.
>
> I was wondering if we should have postmaster do personality(ADDR_NO_RANDOMIZE)
> for EXEC_BACKEND builds? It seems nicer to make it automatically work than
> have people remember that they need to call "setarch --addr-no-randomize make check".
>
> Not that it actually matters for EXEC_BACKEND, but theoretically doing
> personality(ADDR_NO_RANDOMIZE) in postmaster is a tad more secure than doing
> it via setarch, as in the personality() case postmaster's layout itself is
> still randomized...
>
>
> Or perhaps we should just add a comment mentioning setarch.
>

If we can set it conveniently then that seems worth doing.


(Thinks: do we have non-Windows buildfarm members doing EXEC_BACKEND?)


cheers


andrew


--
Andrew Dunstan
EDB: https://www.enterprisedb.com

  3. Re: Worth using personality(ADDR_NO_RANDOMIZE) for EXEC_BACKEND on linux?

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2021-08-09T17:30:00Z

    On 2021-Aug-06, Andrew Dunstan wrote:

> On 8/5/21 11:29 PM, Andres Freund wrote:

> > I was wondering if we should have postmaster do personality(ADDR_NO_RANDOMIZE)
> > for EXEC_BACKEND builds? It seems nicer to make it automatically work than
> > have people remember that they need to call "setarch --addr-no-randomize make check".

How common is to get a failure?  I know I've run tests under
EXEC_BACKEND and not seen any failures.  Not many runs though.

> > Not that it actually matters for EXEC_BACKEND, but theoretically doing
> > personality(ADDR_NO_RANDOMIZE) in postmaster is a tad more secure than doing
> > it via setarch, as in the personality() case postmaster's layout itself is
> > still randomized...

True.  I think the security aspect is not critically important, since
hopefully nobody should be using such builds for production.


> (Thinks: do we have non-Windows buildfarm members doing EXEC_BACKEND?)

culicidae does that.

-- 
Álvaro Herrera         PostgreSQL Developer  —  https://www.EnterpriseDB.com/
"Pido que me den el Nobel por razones humanitarias" (Nicanor Parra)

  4. Re: Worth using personality(ADDR_NO_RANDOMIZE) for EXEC_BACKEND on linux?

    Robert Haas <robertmhaas@gmail.com> — 2021-08-09T17:43:03Z

    On Mon, Aug 9, 2021 at 1:30 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
> How common is to get a failure?  I know I've run tests under
> EXEC_BACKEND and not seen any failures.  Not many runs though.

On macOS, failures are extremely common. Sometimes I have to run
simple tests many times to get even one success. The proposal on the
table won't help with that problem since it's Linux-specific, but if
there's any way to do something similar on macOS it would be a _huge_
help.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  5. Re: Worth using personality(ADDR_NO_RANDOMIZE) for EXEC_BACKEND on linux?

    Andres Freund <andres@anarazel.de> — 2021-08-09T17:50:28Z

    Hi,

On 2021-08-09 13:43:03 -0400, Robert Haas wrote:
> On Mon, Aug 9, 2021 at 1:30 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
> > How common is to get a failure?  I know I've run tests under
> > EXEC_BACKEND and not seen any failures.  Not many runs though.

I get check-world failures in about 1/2-1/3 of the runs, and a plain check
fails in maybe 1/4 of the cases. It's pretty annoying because it often isn't
trivial to distinguish whether I've broken something or whether it's
randomization related...

The frequency of failures isn't stable over time, making it a bit harder to
know what's going on.


> On macOS, failures are extremely common. Sometimes I have to run
> simple tests many times to get even one success. The proposal on the
> table won't help with that problem since it's Linux-specific, but if
> there's any way to do something similar on macOS it would be a _huge_
> help.

There does seem to be a way of doing so, because I found blog posts talking
about how to get e.g. lldb to *enable* ASLR, which it disables by default...

Greetings,

Andres Freund

  6. Re: Worth using personality(ADDR_NO_RANDOMIZE) for EXEC_BACKEND on linux?

    Tom Lane <tgl@sss.pgh.pa.us> — 2021-08-09T17:54:25Z

    Andres Freund <andres@anarazel.de> writes:
> On 2021-08-09 13:43:03 -0400, Robert Haas wrote:
>> On Mon, Aug 9, 2021 at 1:30 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
> How common is to get a failure?  I know I've run tests under
> EXEC_BACKEND and not seen any failures.  Not many runs though.

> I get check-world failures in about 1/2-1/3 of the runs, and a plain check
> fails in maybe 1/4 of the cases. It's pretty annoying because it often isn't
> trivial to distinguish whether I've broken something or whether it's
> randomization related...

I don't have numbers, but I do know that on Linux EXEC_BACKEND builds fail
often enough to be annoying if you don't disable ASLR.  If we can do
something not-too-invasive about that, it'd be great.

			regards, tom lane

  7. Re: Worth using personality(ADDR_NO_RANDOMIZE) for EXEC_BACKEND on linux?

    Andres Freund <andres@anarazel.de> — 2021-08-09T18:04:07Z

    Hi,

On 2021-08-09 13:54:25 -0400, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> > On 2021-08-09 13:43:03 -0400, Robert Haas wrote:
> >> On Mon, Aug 9, 2021 at 1:30 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
> > How common is to get a failure?  I know I've run tests under
> > EXEC_BACKEND and not seen any failures.  Not many runs though.
> 
> > I get check-world failures in about 1/2-1/3 of the runs, and a plain check
> > fails in maybe 1/4 of the cases. It's pretty annoying because it often isn't
> > trivial to distinguish whether I've broken something or whether it's
> > randomization related...
> 
> I don't have numbers, but I do know that on Linux EXEC_BACKEND builds fail
> often enough to be annoying if you don't disable ASLR.  If we can do
> something not-too-invasive about that, it'd be great.

I now not sure if personality(NO_RANDOMIZE) in postmaster is actually
sufficient. It does seem to drastically reduce the frequency of issues, but
there's still conceivable failures where postmaster's layout would class with
the non-randomized children - obviously personality(NO_RANDOMIZE) cannot
retroactively change the layout of the already running binary.

So maybe we should put it into pg_ctl?

Greetings,

Andres Freund

  8. Re: Worth using personality(ADDR_NO_RANDOMIZE) for EXEC_BACKEND on linux?

    Thomas Munro <thomas.munro@gmail.com> — 2021-08-10T13:19:00Z

    On Tue, Aug 10, 2021 at 5:43 AM Robert Haas <robertmhaas@gmail.com> wrote:
> On Mon, Aug 9, 2021 at 1:30 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
> > How common is to get a failure?  I know I've run tests under
> > EXEC_BACKEND and not seen any failures.  Not many runs though.
>
> On macOS, failures are extremely common. Sometimes I have to run
> simple tests many times to get even one success. The proposal on the
> table won't help with that problem since it's Linux-specific, but if
> there's any way to do something similar on macOS it would be a _huge_
> help.

Yeah, make check always fails for me on macOS 11.  With the attached
experimental hack, it fails only occasionally (1 in 8 runs or so).  I
don't know why.

  9. Re: Worth using personality(ADDR_NO_RANDOMIZE) for EXEC_BACKEND on linux?

    Andres Freund <andres@anarazel.de> — 2021-08-10T14:11:59Z

    Hi,

On Tue, Aug 10, 2021, at 15:19, Thomas Munro wrote:
> On Tue, Aug 10, 2021 at 5:43 AM Robert Haas <robertmhaas@gmail.com> wrote:
> > On Mon, Aug 9, 2021 at 1:30 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
> > > How common is to get a failure?  I know I've run tests under
> > > EXEC_BACKEND and not seen any failures.  Not many runs though.
> >
> > On macOS, failures are extremely common. Sometimes I have to run
> > simple tests many times to get even one success. The proposal on the
> > table won't help with that problem since it's Linux-specific, but if
> > there's any way to do something similar on macOS it would be a _huge_
> > help.
> 
> Yeah, make check always fails for me on macOS 11.  With the attached
> experimental hack, it fails only occasionally (1 in 8 runs or so).  I
> don't know why.

I suspect you'd need to use the hack in pg_ctl to make it reliable. The layout of normally stayed position independent postmaster can be incompatible with the non ASLR spawned child.

Andres

  10. Re: Worth using personality(ADDR_NO_RANDOMIZE) for EXEC_BACKEND on linux?

    Thomas Munro <thomas.munro@gmail.com> — 2021-08-10T19:07:49Z

    On Wed, Aug 11, 2021 at 2:12 AM Andres Freund <andres@anarazel.de> wrote:
> On Tue, Aug 10, 2021, at 15:19, Thomas Munro wrote:
> > Yeah, make check always fails for me on macOS 11.  With the attached
> > experimental hack, it fails only occasionally (1 in 8 runs or so).  I
> > don't know why.
>
> I suspect you'd need to use the hack in pg_ctl to make it reliable. The layout of normally stayed position independent postmaster can be incompatible with the non ASLR spawned child.

Yeah, but the patch already changes both pg_ctl.c and postmaster.c.

  11. Re: Worth using personality(ADDR_NO_RANDOMIZE) for EXEC_BACKEND on linux?

    Thomas Munro <thomas.munro@gmail.com> — 2021-08-10T23:59:02Z

    On Wed, Aug 11, 2021 at 7:07 AM Thomas Munro <thomas.munro@gmail.com> wrote:
> On Wed, Aug 11, 2021 at 2:12 AM Andres Freund <andres@anarazel.de> wrote:
> > On Tue, Aug 10, 2021, at 15:19, Thomas Munro wrote:
> > > Yeah, make check always fails for me on macOS 11.  With the attached
> > > experimental hack, it fails only occasionally (1 in 8 runs or so).  I
> > > don't know why.
> >
> > I suspect you'd need to use the hack in pg_ctl to make it reliable. The layout of normally stayed position independent postmaster can be incompatible with the non ASLR spawned child.
>
> Yeah, but the patch already changes both pg_ctl.c and postmaster.c.

/me stares at vmmap output for a while

Oooh. It's working perfectly (for example if you export
PATH=binarys:$PATH, pg_ctl -D pgdata start, make installcheck), but
pg_regress.c has its own separate fork/exec to launch the temporary
cluster that needs to be similarly hacked.  Unfortunately I have to
give this Macintosh back and go and do some real work on a different
computer now.  That does seem to be a working solution to the problem,
though, and could be polished into proposable form.

I saw claims that you can also link with -Wl,-no_pie or toggle the PIE
bit on your executable and libraries, but that didn't work for me on
11, Intel (no effect) or ARM (linker option gone).

  12. Re: Worth using personality(ADDR_NO_RANDOMIZE) for EXEC_BACKEND on linux?

    Andrew Dunstan <andrew@dunslane.net> — 2021-08-11T12:38:43Z

    On 8/10/21 7:59 PM, Thomas Munro wrote:
> On Wed, Aug 11, 2021 at 7:07 AM Thomas Munro <thomas.munro@gmail.com> wrote:
>> On Wed, Aug 11, 2021 at 2:12 AM Andres Freund <andres@anarazel.de> wrote:
>>> On Tue, Aug 10, 2021, at 15:19, Thomas Munro wrote:
>>>> Yeah, make check always fails for me on macOS 11.  With the attached
>>>> experimental hack, it fails only occasionally (1 in 8 runs or so).  I
>>>> don't know why.
>>> I suspect you'd need to use the hack in pg_ctl to make it reliable. The layout of normally stayed position independent postmaster can be incompatible with the non ASLR spawned child.
>> Yeah, but the patch already changes both pg_ctl.c and postmaster.c.
> /me stares at vmmap output for a while
>
> Oooh. It's working perfectly (for example if you export
> PATH=binarys:$PATH, pg_ctl -D pgdata start, make installcheck), but
> pg_regress.c has its own separate fork/exec to launch the temporary
> cluster that needs to be similarly hacked.  


initdb also runs postgres a bunch of times via system(), similarly to
pg_regress but without the "exec". Does it also need adjusting?


cheers


andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

  13. Re: Worth using personality(ADDR_NO_RANDOMIZE) for EXEC_BACKEND on linux?

    Robert Haas <robertmhaas@gmail.com> — 2021-08-11T13:45:45Z

    On Tue, Aug 10, 2021 at 7:59 PM Thomas Munro <thomas.munro@gmail.com> wrote:
> I saw claims that you can also link with -Wl,-no_pie or toggle the PIE
> bit on your executable and libraries, but that didn't work for me on
> 11, Intel (no effect) or ARM (linker option gone).

I think that worked for me on older macOS releases, and then it
stopped working at some point after some update or reinstall or
something. Unfortunately I don't know any more precisely than that,
but it does seem like we have to find some other approach to work on
modern systems.

Hopefully they don't keep whacking this around...

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  14. Re: Worth using personality(ADDR_NO_RANDOMIZE) for EXEC_BACKEND on linux?

    Tom Lane <tgl@sss.pgh.pa.us> — 2021-08-11T13:55:49Z

    Andrew Dunstan <andrew@dunslane.net> writes:
> initdb also runs postgres a bunch of times via system(), similarly to
> pg_regress but without the "exec". Does it also need adjusting?

That shouldn't be an issue, because we're only running the single
"standalone backend" process, not a cluster.

			regards, tom lane

  15. Re: Worth using personality(ADDR_NO_RANDOMIZE) for EXEC_BACKEND on linux?

    Thomas Munro <thomas.munro@gmail.com> — 2021-08-11T22:24:09Z

    On Thu, Aug 12, 2021 at 1:45 AM Robert Haas <robertmhaas@gmail.com> wrote:
> I think that worked for me on older macOS releases, and then it
> stopped working at some point after some update or reinstall or
> something. Unfortunately I don't know any more precisely than that,
> but it does seem like we have to find some other approach to work on
> modern systems.

I gave up on trying to make that work once I realised that
/usr/lib/dyld doesn't seem to obey the flag, so although other
segments become deterministic and the success rate is fairly high,
there's still a 600kb wrecking ball swinging around.  I wondered what
the "slide" range could be... it appears to be fairly small
(vm_map_get_max_aslr_slide_pages() seems to be the place that's
determined and it's a 16MB or 256MB window, depending on architecture,
if I read that right).  Given that, the death of 32 bit processes
since Catalina, and the typical layout we see, I think just doing
something like (/me rolls dice) export PG_SHMEM_ADDR=0x80000000000 is
a good candidate for something that works on both architectures, being
many TB away from everything else (above everything on ARM, between
heap etc and libs on Intel but with 8TB of space below it and 120TB
above).  That gets the tests passing consistently with unpatched
master, -DEXEC_BACKEND, on both flavours of silicon.

  16. Re: Worth using personality(ADDR_NO_RANDOMIZE) for EXEC_BACKEND on linux?

    Robert Haas <robertmhaas@gmail.com> — 2021-08-12T14:58:02Z

    On Wed, Aug 11, 2021 at 6:24 PM Thomas Munro <thomas.munro@gmail.com> wrote:
> On Thu, Aug 12, 2021 at 1:45 AM Robert Haas <robertmhaas@gmail.com> wrote:
> > I think that worked for me on older macOS releases, and then it
> > stopped working at some point after some update or reinstall or
> > something. Unfortunately I don't know any more precisely than that,
> > but it does seem like we have to find some other approach to work on
> > modern systems.
>
> I gave up on trying to make that work once I realised that
> /usr/lib/dyld doesn't seem to obey the flag, so although other
> segments become deterministic and the success rate is fairly high,
> there's still a 600kb wrecking ball swinging around.  I wondered what
> the "slide" range could be... it appears to be fairly small
> (vm_map_get_max_aslr_slide_pages() seems to be the place that's
> determined and it's a 16MB or 256MB window, depending on architecture,
> if I read that right).  Given that, the death of 32 bit processes
> since Catalina, and the typical layout we see, I think just doing
> something like (/me rolls dice) export PG_SHMEM_ADDR=0x80000000000 is
> a good candidate for something that works on both architectures, being
> many TB away from everything else (above everything on ARM, between
> heap etc and libs on Intel but with 8TB of space below it and 120TB
> above).  That gets the tests passing consistently with unpatched
> master, -DEXEC_BACKEND, on both flavours of silicon.

Ugh, OK. So, is there a way that we can get an "easy button" committed
to the tree?

-- 
Robert Haas
EDB: http://www.enterprisedb.com

  17. Re: Worth using personality(ADDR_NO_RANDOMIZE) for EXEC_BACKEND on linux?

    Tom Lane <tgl@sss.pgh.pa.us> — 2021-08-12T15:13:17Z

    Robert Haas <robertmhaas@gmail.com> writes:
> On Wed, Aug 11, 2021 at 6:24 PM Thomas Munro <thomas.munro@gmail.com> wrote:
>> ... I think just doing
>> something like (/me rolls dice) export PG_SHMEM_ADDR=0x80000000000 is
>> a good candidate for something that works on both architectures, being
>> many TB away from everything else (above everything on ARM, between
>> heap etc and libs on Intel but with 8TB of space below it and 120TB
>> above).  That gets the tests passing consistently with unpatched
>> master, -DEXEC_BACKEND, on both flavours of silicon.

> Ugh, OK. So, is there a way that we can get an "easy button" committed
> to the tree?

I don't see why that approach couldn't be incorporated into pg_ctl,
or the postmaster itself.  Given Andres' point that Linux ASLR
disable probably has to happen in pg_ctl, it seems like doing it
in pg_ctl in all cases is the way to move forward.

			regards, tom lane

  18. Re: Worth using personality(ADDR_NO_RANDOMIZE) for EXEC_BACKEND on linux?

    Thomas Munro <thomas.munro@gmail.com> — 2021-08-12T21:39:36Z

    On Fri, Aug 13, 2021 at 3:13 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Robert Haas <robertmhaas@gmail.com> writes:
> > Ugh, OK. So, is there a way that we can get an "easy button" committed
> > to the tree?
>
> I don't see why that approach couldn't be incorporated into pg_ctl,
> or the postmaster itself.  Given Andres' point that Linux ASLR
> disable probably has to happen in pg_ctl, it seems like doing it
> in pg_ctl in all cases is the way to move forward.

I think doing it in the postmaster is best, since otherwise you have
to put code into pg_regress.c and pg_ctl.c.  Here's a patch like that.

  19. Re: Worth using personality(ADDR_NO_RANDOMIZE) for EXEC_BACKEND on linux?

    Tom Lane <tgl@sss.pgh.pa.us> — 2021-08-12T21:59:38Z

    Thomas Munro <thomas.munro@gmail.com> writes:
> On Fri, Aug 13, 2021 at 3:13 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> I don't see why that approach couldn't be incorporated into pg_ctl,
>> or the postmaster itself.  Given Andres' point that Linux ASLR
>> disable probably has to happen in pg_ctl, it seems like doing it
>> in pg_ctl in all cases is the way to move forward.

> I think doing it in the postmaster is best, since otherwise you have
> to put code into pg_regress.c and pg_ctl.c.  Here's a patch like that.

Hmm, ok.  Small thought: it might be better to put the #if inside
the "else { .... }".  That way it scales easily to allow other
platform-specific defaults if we find anything useful.  As-is,
the obvious extension would end up with multiple else-blocks,
which seems likely to confuse pgindent if nothing else.

			regards, tom lane

  20. Re: Worth using personality(ADDR_NO_RANDOMIZE) for EXEC_BACKEND on linux?

    Thomas Munro <thomas.munro@gmail.com> — 2021-08-13T00:06:19Z

    On Fri, Aug 13, 2021 at 9:59 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Hmm, ok.  Small thought: it might be better to put the #if inside
> the "else { .... }".  That way it scales easily to allow other
> platform-specific defaults if we find anything useful.  As-is,
> the obvious extension would end up with multiple else-blocks,
> which seems likely to confuse pgindent if nothing else.

True.  Thanks.  Pushed to all live branches.

Obviously the address may have to be adjusted if Apple moves stuff
around, or if the initial layout and ASLR slide range turn out to be
less constrained than I inferred by nosing around the source code and
testing on a couple of systems.

  21. Re: Worth using personality(ADDR_NO_RANDOMIZE) for EXEC_BACKEND on linux?

    Tom Lane <tgl@sss.pgh.pa.us> — 2021-08-13T00:29:09Z

    Thomas Munro <thomas.munro@gmail.com> writes:
> Obviously the address may have to be adjusted if Apple moves stuff
> around, or if the initial layout and ASLR slide range turn out to be
> less constrained than I inferred by nosing around the source code and
> testing on a couple of systems.

Sure.  But we're no worse off than before; the workaround of "set
PG_SHMEM_ADDR by hand" is just as applicable as ever.

			regards, tom lane

  22. Re: Worth using personality(ADDR_NO_RANDOMIZE) for EXEC_BACKEND on linux?

    Thomas Munro <thomas.munro@gmail.com> — 2021-11-23T19:27:43Z

    Here's a patch for Linux and also FreeBSD.  The latter OS decided to
turn on ASLR by default recently, causing my workstation to fail like
this quite reliably, which reminded me to follow up with this.  It
disables ASLR in pg_ctl and pg_regress, which is enough for check and
check-world, but doesn't help you if you run the server directly
(unlike the different hack done for macOS).

For whatever random reason the failures are rarer on Linux (could be
my imagination, but I think they might be clustered, I didn't look
into the recipe for the randomness), but even without reproducing a
failure it's clear to see using pmap that this has the right effect.
I didn't bother with a check for the existence of ADDR_NO_RANDOMIZE
because it's since 2.6.12 which is definitely ancient enough.

  23. Re: Worth using personality(ADDR_NO_RANDOMIZE) for EXEC_BACKEND on linux?

    Nathan Bossart <bossartn@amazon.com> — 2022-01-07T20:20:31Z

    On 11/23/21, 11:29 AM, "Thomas Munro" <thomas.munro@gmail.com> wrote:
> Here's a patch for Linux and also FreeBSD.  The latter OS decided to
> turn on ASLR by default recently, causing my workstation to fail like
> this quite reliably, which reminded me to follow up with this.  It
> disables ASLR in pg_ctl and pg_regress, which is enough for check and
> check-world, but doesn't help you if you run the server directly
> (unlike the different hack done for macOS).
>
> For whatever random reason the failures are rarer on Linux (could be
> my imagination, but I think they might be clustered, I didn't look
> into the recipe for the randomness), but even without reproducing a
> failure it's clear to see using pmap that this has the right effect.
> I didn't bother with a check for the existence of ADDR_NO_RANDOMIZE
> because it's since 2.6.12 which is definitely ancient enough.

FWIW I just found this patch very useful for testing some EXEC_BACKEND
stuff on Linux.

Nathan

  24. Re: Worth using personality(ADDR_NO_RANDOMIZE) for EXEC_BACKEND on linux?

    Thomas Munro <thomas.munro@gmail.com> — 2022-01-10T11:29:44Z

    On Sat, Jan 8, 2022 at 9:20 AM Bossart, Nathan <bossartn@amazon.com> wrote:
> FWIW I just found this patch very useful for testing some EXEC_BACKEND
> stuff on Linux.

Thanks for testing.  Tidied and pushed, to master only for now.

  25. Re: Worth using personality(ADDR_NO_RANDOMIZE) for EXEC_BACKEND on linux?

    Michael Paquier <michael@paquier.xyz> — 2023-02-01T01:32:40Z

    On Tue, Jan 11, 2022 at 12:29:44AM +1300, Thomas Munro wrote:
> Thanks for testing.  Tidied and pushed, to master only for now.

I have noticed the following failure for v11~14 on one of my hosts
that compiles with -DEXEC_BACKEND, and Nathan has redirected me here:
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=gokiburi&dt=2023-01-31%2012%3A07%3A32
FATAL:  could not reattach to shared memory (key=1050468, addr=0xffff97eb2000): Invalid argument

Could it be worth back-patching f3e7806?  I don't mind changing this
animal setup by switching the kernel configuration or reducing the
branch scope, but this solution is less invasive because it would not
influence parallel runs.

Thoughts?
--
Michael

  26. Re: Worth using personality(ADDR_NO_RANDOMIZE) for EXEC_BACKEND on linux?

    Tom Lane <tgl@sss.pgh.pa.us> — 2023-02-01T01:37:29Z

    Michael Paquier <michael@paquier.xyz> writes:
> Could it be worth back-patching f3e7806?

That's aged long enough now that it seems like a pretty safe
thing to do.

			regards, tom lane

  27. Re: Worth using personality(ADDR_NO_RANDOMIZE) for EXEC_BACKEND on linux?

    Michael Paquier <michael@paquier.xyz> — 2023-02-01T01:58:46Z

    On Tue, Jan 31, 2023 at 08:37:29PM -0500, Tom Lane wrote:
> That's aged long enough now that it seems like a pretty safe
> thing to do.

Thanks.  I'll wait for a few days before doing something for my
buildfarm stuff, in case somebody thinks this is a bad idea..
--
Michael

  28. Re: Worth using personality(ADDR_NO_RANDOMIZE) for EXEC_BACKEND on linux?

    Thomas Munro <thomas.munro@gmail.com> — 2023-02-01T02:06:16Z

    On Wed, Feb 1, 2023 at 2:58 PM Michael Paquier <michael@paquier.xyz> wrote:
> On Tue, Jan 31, 2023 at 08:37:29PM -0500, Tom Lane wrote:
> > That's aged long enough now that it seems like a pretty safe
> > thing to do.
>
> Thanks.  I'll wait for a few days before doing something for my
> buildfarm stuff, in case somebody thinks this is a bad idea..

+1, go for it.  It shouldn't affect Unix build releases, and on
Windows the function does nothing.

  29. Re: Worth using personality(ADDR_NO_RANDOMIZE) for EXEC_BACKEND on linux?

    Michael Paquier <michael@paquier.xyz> — 2023-02-02T01:06:15Z

    On Wed, Feb 01, 2023 at 03:06:16PM +1300, Thomas Munro wrote:
> +1, go for it.  It shouldn't affect Unix build releases, and on
> Windows the function does nothing.

Thanks for confirming.  I am wondering what these animals may complain
about next, but based on some tests on this buildfarm host with the
same configuration, things are looking OK once this stuff is applied
on 11~14.
--
Michael

  30. Re: Worth using personality(ADDR_NO_RANDOMIZE) for EXEC_BACKEND on linux?

    Michael Paquier <michael@paquier.xyz> — 2023-02-02T02:14:39Z

    On Thu, Feb 02, 2023 at 10:06:15AM +0900, Michael Paquier wrote:
> Thanks for confirming.  I am wondering what these animals may complain
> about next, but based on some tests on this buildfarm host with the
> same configuration, things are looking OK once this stuff is applied
> on 11~14.

Actually, I completely forgot to take into account that there is a
minor release planned for next week:
https://www.postgresql.org/developer/roadmap/

So I'll hold on a bit longer here, until the next versions get their
tags.
--
Michael

  31. Re: Worth using personality(ADDR_NO_RANDOMIZE) for EXEC_BACKEND on linux?

    Michael Paquier <michael@paquier.xyz> — 2023-02-08T04:12:15Z

    On Thu, Feb 02, 2023 at 11:14:39AM +0900, Michael Paquier wrote:
> Actually, I completely forgot to take into account that there is a
> minor release planned for next week:
> https://www.postgresql.org/developer/roadmap/
> 
> So I'll hold on a bit longer here, until the next versions get their
> tags.

So, the backpatch down to 12 is straight-forward, with some conflicts
in ./configure.  ~12 handles also differently its CFLAGS with
pg_config.h.win32.  11 is more annoying because it lacks
HAVE_SYS_PROCCTL_H and it would need a partial backport of f98b847.  I
am not completely sure if this could have more side effects, though,
even if the part of pmsignal.c is left out.  It should not..

At the end, I have just done this stuff down to ~12, 11 does not seem
worth the trouble as the next stable version to go out of support.
I'll reduce gokiburi's script a bit, as a result, until the oldest
version support is v12.
--
Michael

  32. Re: Worth using personality(ADDR_NO_RANDOMIZE) for EXEC_BACKEND on linux?

    Andres Freund <andres@anarazel.de> — 2023-02-08T05:44:47Z

    On 2023-02-08 13:12:15 +0900, Michael Paquier wrote:
> On Thu, Feb 02, 2023 at 11:14:39AM +0900, Michael Paquier wrote:
> > Actually, I completely forgot to take into account that there is a
> > minor release planned for next week:
> > https://www.postgresql.org/developer/roadmap/
> > 
> > So I'll hold on a bit longer here, until the next versions get their
> > tags.
> 
> So, the backpatch down to 12 is straight-forward, with some conflicts
> in ./configure.  ~12 handles also differently its CFLAGS with
> pg_config.h.win32.  11 is more annoying because it lacks
> HAVE_SYS_PROCCTL_H and it would need a partial backport of f98b847.  I
> am not completely sure if this could have more side effects, though,
> even if the part of pmsignal.c is left out.  It should not..
> 
> At the end, I have just done this stuff down to ~12, 11 does not seem
> worth the trouble as the next stable version to go out of support.
> I'll reduce gokiburi's script a bit, as a result, until the oldest
> version support is v12.

That seems reasonable to me.

  33. Re: Worth using personality(ADDR_NO_RANDOMIZE) for EXEC_BACKEND on linux?

    Thomas Munro <thomas.munro@gmail.com> — 2023-02-09T19:15:21Z

    On Wed, Feb 8, 2023 at 5:12 PM Michael Paquier <michael@paquier.xyz> wrote:
> At the end, I have just done this stuff down to ~12, 11 does not seem
> worth the trouble as the next stable version to go out of support.
> I'll reduce gokiburi's script a bit, as a result, until the oldest
> version support is v12.

For the record, according to [1] it's not necessary to use
--reset-author when back-patching.  (Maybe a little confusingly,
because it's not quite clear whether our policies consider the author
field to be meaningful or not.)

[1] https://www.postgresql.org/message-id/flat/1347459696.16215.11.camel%40vanquo.pezone.net

  34. Re: Worth using personality(ADDR_NO_RANDOMIZE) for EXEC_BACKEND on linux?

    Michael Paquier <michael@paquier.xyz> — 2023-02-09T22:56:33Z

    On Fri, Feb 10, 2023 at 08:15:21AM +1300, Thomas Munro wrote:
> For the record, according to [1] it's not necessary to use
> --reset-author when back-patching.  (Maybe a little confusingly,
> because it's not quite clear whether our policies consider the author
> field to be meaningful or not.)
> 
> [1] https://www.postgresql.org/message-id/flat/1347459696.16215.11.camel%40vanquo.pezone.net

Oops, sorry about that.  Using --reset-author is a habit when it comes
to backpatch.  It looks like my mistake when back-patching something
only to stable branches.
--
Michael