Thread

Switching to Homebrew as recommended Mac install?

Jay Levitt <jay.levitt@gmail.com> — 2012-04-01T20:14:14Z

The Mac installation docs currently recommend the EDB one-click installer as 
the first choice. While this does install pgadmin and some other 
refinements, it also is fairly confusing to troubleshoot:

- By default, it installs to /Library/PostgreSQL, which is also (I think) 
where the Apple-supplied Lion install is

- The uninstaller is hidden in /Library/PostgreSQL, which (since Finder 
hides /Library by default) you're likely to go to via Terminal. But the 
uninstaller is a Mac app, so even if you find it you have to know to use 
"open" to run it, because Mac apps are really directories that the Finder 
abstracts away from you.

- The EDB docs are written against 8.4.

- There are, as @lluad points out, no fewer than eight ways to install 
Postgres on a Mac (fink, macports, homebrew, Lion default, build from 
source, EDB, and two other binary installers)

- We have few Mac experts hanging out in #postgresql.

- We just had two folks within an hour, BOTH with conflicting installs of 
Postgres.

So with all respect and thanks to EDB for maintaining those installers, I'd 
like to propose that homebrew become the recommended install method on Mac, 
and I will update the Mac formula to overcome any current objections. The 
nice thing about homebrew is that (a) formulas can contain arbitrary Ruby 
and command-line options, so we can easily deal with things like detecting 
existing installs, handling shared memory, etc. if we want to, and (b) pull 
requests are accepted freely and frequently, so it can always be the 
current, security-patched version.

What do folks think of this idea?  When I mention homebrew in #postgresql, 
there's always an "ick", but I believe that's true of any package manager 
(and possibly any Mac anything, because we're all Rails-loving ORM-using 
SQL-not-understanding fanbois, and I say that with love.)

The current homebrew installer is already 9.1.3, and does a make-world, so 
you get all of contrib built.

POSSIBLE OBJECTIONS/PREREQUISITES

1. homebrew installs everything under /usr/local and makes that 
user-writeable.  Sorry.  It does because most Mac users don't know how to 
edit PATH for GUI apps (it's in a .plist in a hidden directory in your home 
dir), and /usr/local is already in PATH by default.

2. The current formula installs Postgres as the desktop user, not as the 
_postgres role account.

I'm personally of the strong opinion that user-to-user privilege escalation 
attacks are NOT an issue on desktops; all important files are already owned 
by the desktop user. The attack vector is *maybe* root escalation attacks, 
but if you want root, it's so common for installers to ask permission that 
your malware could just ask.  The real attack vector is "I'm in your 
browser", and that has nothing to do with root, permissions, or users at all.

Meanwhile, the EDB installer by default installs both app and data to a 
directory that requires root - so I assume it runs as root too - and 
nobody's complained.

However, if this is a sticking point, I'd have no problem adding a --user 
option that would default to _postgres (underscore-prefixed usernames are 
the Apple standard).

3. The current formula (TCF) spits out instructions telling you how to 
initdb, but they're easy to overlook.  I'm happy to add an option if 
necessary, and might do it anyway.

4. TCF also spits out instructions for adding Postgres to launchctl (Mac's 
version of /etc/init.d or Windows Services), rather than doing it for you, 
but again, I'd happily add the option.  (I'm checking with Homebrew folks to 
see if there's some dictum against that; it's a common pattern to put 
launchctl in the instructions, but IME the usability is poor.)

5. TCF doesn't update your shared memory settings. Again, happy to add that.

6. TCF doesn't look for existing installs. This is especially a problem on 
Lion, since Apple bundles PG 8.4, either client or server IIUC, and although 
/usr/local/bin is in your PATH, it comes *after* /usr/bin (grumble), so 
you'll either have the homebrew server fail to launch (since port 5432 is in 
use), or you'll have an outdated client version.

In IRC, both users had actually installed the EDB version months ago and 
forgotten about it, but over time, Lion users will grow, since all new Macs 
come with only Lion.  There are several ways to address this; my preference 
is to have homebrew warn about existing installs but take care of any magic 
to make them go away, a la 
http://nextmarvel.net/blog/2011/09/brew-install-postgresql-on-os-x-lion/.

7. There's no homebrew formula for pgadmin. I've never built it, and might 
be able to add that, but probably not right away.

8. There might be other popular things that EDB's StackBuilder does.

9. EDB is an important contributor to the PG core community, and maybe the 
link juice/publicity is politically important.  Lemme know.

That's all I can think of... thoughts? Objections? Which do you think are 
prerequisites?

Jay Levitt

Re: Switching to Homebrew as recommended Mac install?

Jay Levitt <jay.levitt@gmail.com> — 2012-04-01T20:19:23Z

Jay Levitt wrote:
> POSSIBLE OBJECTIONS/PREREQUISITES

10. There is no homebrew support for multiple versions, and no current plans 
to add it (though it's on the wishlist). This means homebrew is only useful 
if "I want to install a PostgreSQL thingie" is the common Mac use case. If 
people often need to use specific older versions, to mirror their server 
configs, it's a problem.  It *might* be possible to hack this into our 
formula, but I'm not sure it's either doable or acceptable.

Re: Switching to Homebrew as recommended Mac install?

Tom Lane <tgl@sss.pgh.pa.us> — 2012-04-01T20:38:58Z

Jay Levitt <jay.levitt@gmail.com> writes:
> So with all respect and thanks to EDB for maintaining those installers, I'd 
> like to propose that homebrew become the recommended install method on Mac, 
> and I will update the Mac formula to overcome any current objections.

This proposal doesn't seem to me to have any chance at all of getting
accepted.  While you might not like the EDB installer, at least those
folks are active in the lists and accountable for whatever problems
their code has.  Who in heck is responsible for the "homebrew"
packaging, and do they answer questions in the PG lists?

ISTM a more useful response to the problems you've mentioned is to
suggest to the EDB folk that they'd better install somewhere else
than where the built-in Lion installation is.

			regards, tom lane

Re: Switching to Homebrew as recommended Mac install?

Dave Page <dpage@pgadmin.org> — 2012-04-01T21:10:27Z

Hi

On Sun, Apr 1, 2012 at 4:14 PM, Jay Levitt <jay.levitt@gmail.com> wrote:
> The Mac installation docs currently recommend the EDB one-click installer as
> the first choice. While this does install pgadmin and some other
> refinements, it also is fairly confusing to troubleshoot:
>
> - By default, it installs to /Library/PostgreSQL, which is also (I think)
> where the Apple-supplied Lion install is

No, Apple's version is installed in /usr on mine.

> - The uninstaller is hidden in /Library/PostgreSQL, which (since Finder
> hides /Library by default) you're likely to go to via Terminal. But the
> uninstaller is a Mac app, so even if you find it you have to know to use
> "open" to run it, because Mac apps are really directories that the Finder
> abstracts away from you.

Yes.

> - The EDB docs are written against 8.4.

Only if you install 8.4. If you install 8.3 you get the 8.3 docs, 9.0
the 9.0 docs and so on.

> - There are, as @lluad points out, no fewer than eight ways to install
> Postgres on a Mac (fink, macports, homebrew, Lion default, build from
> source, EDB, and two other binary installers)

That isn't any more of a reason to discount the EDB installer than any other.

> - We have few Mac experts hanging out in #postgresql.

Not sure how this is relevant to the proposal.

> - We just had two folks within an hour, BOTH with conflicting installs of
> Postgres.

Not sure how that is relevant either. You can have conflicting
installation using any of the installation methods, including a
home-built source tree.

> 1. homebrew installs everything under /usr/local and makes that
> user-writeable.  Sorry.  It does because most Mac users don't know how to
> edit PATH for GUI apps (it's in a .plist in a hidden directory in your home
> dir), and /usr/local is already in PATH by default.

Your reasoning doesn't make sense. Why does putting something in the
path require a directory to be world writeable.

In any case, the fact that Homebrew does that to /usr/local should be
enough to make any user run away screaming in terror. If it opens up a
security hole like that, what else does it do to break your system?

> 2. The current formula installs Postgres as the desktop user, not as the
> _postgres role account.

That's not very helpful on shared machines - and whilst it may be fine
for developers etc, it's not the recommended way to setup PostgreSQL
for any kind of production use.

> I'm personally of the strong opinion that user-to-user privilege escalation
> attacks are NOT an issue on desktops; all important files are already owned
> by the desktop user. The attack vector is *maybe* root escalation attacks,
> but if you want root, it's so common for installers to ask permission that
> your malware could just ask.  The real attack vector is "I'm in your
> browser", and that has nothing to do with root, permissions, or users at
> all.

I, and I suspect many others, would disagree that user to user
security is not important.

> Meanwhile, the EDB installer by default installs both app and data to a
> directory that requires root - so I assume it runs as root too - and
> nobody's complained.

No it doesn't. It installs the app to a root owned directory for
security, and the data goes in a postgres owned directory so it can
only be modified by the account the service runs under.

> 4. TCF also spits out instructions for adding Postgres to launchctl (Mac's
> version of /etc/init.d or Windows Services), rather than doing it for you,
> but again, I'd happily add the option.  (I'm checking with Homebrew folks to
> see if there's some dictum against that; it's a common pattern to put
> launchctl in the instructions, but IME the usability is poor.)
>
> 5. TCF doesn't update your shared memory settings. Again, happy to add that.
>
> 6. TCF doesn't look for existing installs. This is especially a problem on
> Lion, since Apple bundles PG 8.4, either client or server IIUC, and although
> /usr/local/bin is in your PATH, it comes *after* /usr/bin (grumble), so
> you'll either have the homebrew server fail to launch (since port 5432 is in
> use), or you'll have an outdated client version.

They seem like a number of reasons not to use Homebrew too (at least
as it is now).

> In IRC, both users had actually installed the EDB version months ago and
> forgotten about it, but over time, Lion users will grow, since all new Macs
> come with only Lion.  There are several ways to address this; my preference
> is to have homebrew warn about existing installs but take care of any magic
> to make them go away, a la
> http://nextmarvel.net/blog/2011/09/brew-install-postgresql-on-os-x-lion/.

So you propose to make it silently disable existing servers? I know
various people whose machines would be broken by that, including mine.
We went to great lengths to allow side by side installations of
different versions, precisely because developers (and occasionally
users) need to use multiple versions to support current and future
versions of their applications, and to experiment with new features.

> 7. There's no homebrew formula for pgadmin. I've never built it, and might
> be able to add that, but probably not right away.
>
> 8. There might be other popular things that EDB's StackBuilder does.

PostGIS, Slony, psqlODBC, pgJDBC, Npgsql, phpPgAdmin...

> 9. EDB is an important contributor to the PG core community, and maybe the
> link juice/publicity is politically important.  Lemme know.

What's right for the community is guiding people towards the easiest
to use and most reliable and useful packages - but we try to make all
alternatives known to the user (assuming they don't do nasty things to
the security of your system etc), so more experienced users can choose
what suits them the best. Currently (in my somewhat biased - see my
sig - but honestly trying to be objective) opinion, I believe the best
option for most users is our installers.

> That's all I can think of... thoughts? Objections? Which do you think are
> prerequisites?

It seems to me that most of your arguments against the installers are
based on incorrect understanding or information, and most of your
arguments for Homebrew actually come across as arguments against!

-- 
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: Switching to Homebrew as recommended Mac install?

Jay Levitt <jay.levitt@gmail.com> — 2012-04-02T04:29:11Z

Dave Page wrote:
 > It seems to me that most of your arguments against the installers are
 > based on incorrect understanding or information, and most of your
 > arguments for Homebrew actually come across as arguments against!

You're right about the former - and as to the latter, they *were* arguments 
against ("potential objections").  I try to pre-argue against my own 
proposals to save everyone time; if I can still prevail, I must have a damn 
good idea :)

At this point I agree with you, but I'm still going to go into detail, 
because I think there are two markets for Postgres, and the database 
community has been so focused around enterprise for so long that you're 
missing opportunities with web startups. I'd love to help bridge the gap, 
having jumped straight from big-iron PL/I to ooh-Ruby-is-shiny. And web 
startups develop on Mac laptops. They just do. So if it helps you to imagine 
me as a 20something "I'm a Mac" hipster, working on some hot Facebook/mobile 
app with funding from Spark Capital, do that. Lord knows it helps me.

 >> - We have few Mac experts hanging out in #postgresql.
 > Not sure how this is relevant to the proposal.

The impetus for the idea was that there seems to be a steady stream of 
novice PG users on Mac who come into #postgresql with installation problems, 
which is bad enough as an out-of-box experience - but worse is that there 
are rarely folks around who can help. (Of course, I'm extrapolating; every 
time *I'm* in IRC and see this, there's someone who can help. But you know 
what I mean.)

And (although my proposal started with documentation) I'm of the firm 
opinion that "there's no such thing as a documentation error"; a user 
problem is a software problem.  Humans will click buttons before they'll 
read, developers are humans, and no amount of RTFM will ever fix that. If we 
can make installers smarter, that's way better than troubleshooting guides, 
IRC, mailing lists, etc. So that's where I was coming from.

I didn't realize that you were actively maintaining the EDB installer (see 
below for the 8.4 doc explanation); obviously, if you can improve that, it's 
the best solution and we should, if anything, recommend it MORE vigorously. 
Still, there's a growing community of developers who expect "brew install" 
to work, and I do want to fix it for them.  The EDB installer will always be 
a one-off experience; most of the other servers you install will be through 
a package manager, and homebrew's popularity (despite its youth) is 
impressive.  Both of my n=2 data points had run across PG a while back, 
installed it with the one-click to try it out, forgotten about it, done 
"brew install postgresql" today, and naturally ran into problems.

 >> - By default, it installs to /Library/PostgreSQL, which is also (I think)
 >> where the Apple-supplied Lion install is
 > No, Apple's version is installed in /usr on mine.

Ah hah. I suppose only the Apple .plist is stored under /Library, then. Let 
me amend that to "this made everyone in IRC, and probably many other 
non-Mac-expert troubleshooters, assume that this is an Apple-installed 
package." It'd be great for this to go somewhere that feels like "Oh, this 
was installed by you"; /Library feels kinda weird for a server, though I can 
understand your reasoning. Maybe even /Library/EnterpriseDB/PostgreSQL to 
make it obvious?

 >> - The uninstaller is hidden in /Library/PostgreSQL, which (since Finder
 >> hides /Library by default) you're likely to go to via Terminal. But the
 >> uninstaller is a Mac app, so even if you find it you have to know to use
 >> "open" to run it, because Mac apps are really directories that the Finder
 >> abstracts away from you.
 > Yes.

How about a one-liner bash script "uninstall-postgresql" that does nothing 
but "open uninstall-postgresql.app"?

 >> - The EDB docs are written against 8.4.
 > Only if you install 8.4. If you install 8.3 you get the 8.3 docs, 9.0
 > the 9.0 docs and so on.

No, I meant on the web:

http://www.enterprisedb.com/resources-community/pginst-guide

That's what made me assume that the installer wasn't maintained (except as 
to repackaging new PG versions, obviously). It's obviously not hard to 
replace "8.3" with "9.1" when you read it, but it still leaves an impression 
akin to "This web site works best with IE7 and above." Allow me to now 
replace most of this thread with "hey, you might wanna update that page."

 >> - There are eight ways to install Postgres on a Mac
 > That isn't any more of a reason to discount the EDB installer than any other.

Nope, just an argument that the recommended installer should handle that nicely.

 >> - We just had two folks within an hour, BOTH with conflicting installs of
 >> Postgres.
 > Not sure how that is relevant either. You can have conflicting
 > installation using any of the installation methods, including a
 > home-built source tree.

Right, but I suspect this is a common problem - not only have I seen it in 
IRC but 3 or 4 times in my 12-person startup, which is kinda amazing given 
that we've been recommending homebrew since day 2 last year (but day 1 was 
macports, and then there was Lion, which is already 30-40% of the Mac 
installed base).

It's worse in the Mac world because it is, very much unlike the enterprise 
database world, a whiplashing, trend-driven ecosystem.  Once there was Fink, 
and there was great rejoicing because now we had a package system at all. 
And then there was MacPorts, and everyone said "that's so much better! Let's 
all switch!" and there was rejoicing and some conflicts. And now the Rails 
community is both (a) collectively realizing that MySQL is not, in fact, 
better than Postgres for any of their use cases, and thus embracing PG as 
their go-to database, and (b) loving homebrew, no doubt because it's written 
in Ruby.  And the Rails community is a Mac community.

So you have a growing user base who may have tried two or three package 
managers over the years and either ignored the warnings to not run both, or 
couldn't find all their packages in one manager, plus people who are using 
Lion and didn't realize it now comes with PG, so they're trying to install 
it, plus the I-forgot-I-tried-this crowd... it'd be nice to handle this case.

 >> 1. homebrew installs everything under /usr/local and makes that
 >> user-writeable.  Sorry.  It does because most Mac users don't know how to
 >> edit PATH for GUI apps (it's in a .plist in a hidden directory in your home
 >> dir), and /usr/local is already in PATH by default.
 >
 > Your reasoning doesn't make sense. Why does putting something in the
 > path require a directory to be world writeable.

Sorry - two separate things plus one misdirected argument:

1. Rubyists in general are sick of sudo'ing on their laptops, because It 
Doesn't Matter (as I'll fail to argue later).  Homebrew puts itself into a 
directory that is user-writable so it does not require sudo for basic 
installation.

2. Because shell's $PATH is hard to change programmatically due to 
shell-config individualism (MacPorts was notorious for failing at this), and 
yet many Mac programmers know nothing of shells at all (so they don't know 
how to edit it manually), Homebrew puts itself into a directory that is 
already in $PATH by default, but which is conveniently nonexistent by default.

Thus, Homebrew chowns /usr/local to (desktop user):admin.

 > In any case, the fact that Homebrew does that to /usr/local should be
 > enough to make any user run away screaming in terror. If it opens up a
 > security hole like that, what else does it do to break your system?

So this is pointless to the discussion now, but if you want to engage 
off-list, I'd frankly love to be reconvinced: Assume that I am on a laptop, 
not a server. There is one physical user, and that user is me. I am always 
logged in as that user. I am also conditioned to enter my password every 
time an installer says it needs me to. I am a developer, and I am writing 
software that will require me to keep my database credentials in cleartext 
in a directory readable by me, and in the non-PCI, non-SOX, non-HIPPA world 
of startups, those are certainly going to be superuser credentials for 
convenience. What, exactly, is the attack vector opened by installing 
postgres or other developer tools under my username? And what is the 
relative risk of being targeted along that vector, and not "You're at 
Starbucks and you're browsing Facebook without SSL", other than spear 
phishing attacks?

 >> 2. The current formula installs Postgres as the desktop user, not as the
 >> _postgres role account.
 >
 > That's not very helpful on shared machines - and whilst it may be fine
 > for developers etc, it's not the recommended way to setup PostgreSQL
 > for any kind of production use.

Oh, of course. Homebrew does not target the three remaining people who run 
production XServes.  It's purely for Mac developer workstations.  At 
startups. Which are MacBooks! :)

 >> Meanwhile, the EDB installer by default installs both app and data to a
 >> directory that requires root - so I assume it runs as root too - and
 >> nobody's complained.
 >
 > No it doesn't. It installs the app to a root owned directory for
 > security, and the data goes in a postgres owned directory so it can
 > only be modified by the account the service runs under.

My bad - false assumption.

 > They seem like a number of reasons not to use Homebrew too (at least
 > as it is now).

Right, but it's about 20 lines of code to fix all that; embrace the wonder 
of Ruby.

 >> In IRC, both users had actually installed the EDB version months ago and
 >> forgotten about it, but over time, Lion users will grow, since new Macs
 >> come with Lion.  There are several ways to address this; my preference
 >> is to warn about existing installs but take care of any magic
 >> to make them go away, a la
 >> http://nextmarvel.net/blog/2011/09/brew-install-postgresql-on-os-x-lion/.
 > So you propose to make it silently disable existing servers?

Oh, God no.  I meant "You have an existing Postgres install. Do you really 
want to replace the binaries? We'll stick them in this directory for 
safekeeping."  I do not think the homebrew use case and the multiple 
side-by-side versions use case overlap at all. (Except me, and I'd cope.)

 >> 8. There might be other popular things that EDB's StackBuilder does.
 > PostGIS, Slony, psqlODBC, pgJDBC, Npgsql, phpPgAdmin...

OK, I'm convinced! (Though for the record, homebrew has formulas for 
postgis, pg_top, pgbouncer, pgpool-ii, and pgtap, so nyah. Formulas are 
simple if building is simple; pgtop.rb attached if you're curious.)

Jay

Re: Switching to Homebrew as recommended Mac install?

Jay Levitt <jay.levitt@gmail.com> — 2012-04-02T04:50:31Z

Tom Lane wrote:
>While you might not like the EDB installer, at least those
> folks are active in the lists and accountable for whatever problems
> their code has.  Who in heck is responsible for the "homebrew"
> packaging, and do they answer questions in the PG lists?

Just for general knowledge... Who's responsible is whoever wants to be; 
homebrew is open source, and with a github-based workflow, it's trivial for 
them to accept pull requests.  On the 1967 formulas (packages) in the repo, 
there have been 1759 contributors.  I was volunteering to be the maintainer 
and liaison if we did this; I'm pretty good at email and IRC.

It's actually pretty clever and elegant - homebrew itself uses git and 
github for formula updates and distribution, and is written in ruby1.8 which 
ships with all Macs. /usr/local is a git repo, "brew update" is mostly "git 
pull", and "brew search" checks for new pull requests if it doesn't find a 
matching formula. The whole thing's all of 1500 lines of code, and you saw 
what formulas look like.

Jay

Re: Switching to Homebrew as recommended Mac install?

David Johnston <polobo@yahoo.com> — 2012-04-02T05:33:56Z

On Apr 1, 2012, at 21:50, Jay Levitt <jay.levitt@gmail.com> wrote:

> Tom Lane wrote:
>> While you might not like the EDB installer, at least those
>> folks are active in the lists and accountable for whatever problems
>> their code has.  Who in heck is responsible for the "homebrew"
>> packaging, and do they answer questions in the PG lists?

> 
> Just for general knowledge... Who's responsible is whoever wants to be; homebrew is open source, and with a github-based workflow, it's trivial for them to accept pull requests.  On the 1967 formulas (packages) in the repo, there have been 1759 contributors.  I was volunteering to be the maintainer and liaison if we did this; I'm pretty good at email and IRC.
> 
> It's actually pretty clever and elegant - homebrew itself uses git and github for formula updates and distribution, and is written in ruby1.8 which ships with all Macs. /usr/local is a git repo, "brew update" is mostly "git pull", and "brew search" checks for new pull requests if it doesn't find a matching formula. The whole thing's all of 1500 lines of code, and you saw what formulas look like.
> 
> Jay
> 
> 

You seem highly approving of "homebrew" and seem willing to develop and support it.  I guess the question to be asked is what requirements you would expect to have to meet before the Mac Downloads section would list your installer routine along with the three already present?  Aside from that unless you are really intent on trying to prove yourself to be the best if you are trying to overcome shortcomings of the existing installers it would still be nice to let them know how you feel things could be improved for the community/user sub-set you belong to.  

As a Windows developer (though production is on Linux) I get where you are coming from with respect to user permissions and the like - what is desirable in a development and in production do differ and so having different installation routines for them makes some sense.  Until your developers go to install on the production server and do not realize that they should be doing something different in order to make the server more secure than their development environment.

From what I follow I think you have really good ideas and sound reasoning.  You do not need permission to contribute to the community in the way you seek so what is it that you are really asking for?  From the sound of things your primary focus is not in supporting the PostgreSQL community via providing services to others or developing new tools.  When brew is replaced by something more popular do you think you will continue to maintain the recipie or is it going to end up stuck showing us how to install version 9.3 or earlier.  I'm beyond my element here but the current installer maintainers are doing so in addition to their other, more regular, contributions.  That said, the contribution, even if it did stall in the future, would still be welcomed and if it is found to be very useful someone would likely pickup the torch as long as it is released under the same terms as PostgreSQL itself.

Just trying to bridge an apparent gap since the original e-mail seems to have come across as too adversarial that the underlying thoughts have been overlooked.  Trying to contribute in my own way with my current resources.

David J.

Re: Switching to Homebrew as recommended Mac install?

David Johnston <polobo@yahoo.com> — 2012-04-02T05:40:06Z

On Apr 1, 2012, at 13:14, Jay Levitt <jay.levitt@gmail.com> wrote:

> The Mac installation docs currently recommend the EDB one-click installer as the first choice. While this does install pgadmin and some other refinements, it also is fairly confusing to troubleshoot:
> 

The items are not numbered and it is impossible to avoid special ordering.  There are three options - and yes EDD is listed at the top of the page - but nothing else implies any kind of order and given that they are dealing with different ways to package if someone prefers MacPorts or Fink the fact they are listed lower shouldn't induce them to pick the unfamiliar one first.

David J.

Re: Switching to Homebrew as recommended Mac install?

Dave Page <dpage@pgadmin.org> — 2012-04-02T09:23:58Z

On Mon, Apr 2, 2012 at 12:29 AM, Jay Levitt <jay.levitt@gmail.com> wrote:
>
> At this point I agree with you, but I'm still going to go into detail,
> because I think there are two markets for Postgres, and the database
> community has been so focused around enterprise for so long that you're
> missing opportunities with web startups. I'd love to help bridge the gap,
> having jumped straight from big-iron PL/I to ooh-Ruby-is-shiny. And web
> startups develop on Mac laptops. They just do. So if it helps you to imagine
> me as a 20something "I'm a Mac" hipster, working on some hot Facebook/mobile
> app with funding from Spark Capital, do that. Lord knows it helps me.

Just as an FYI, a large percentage of the PostgreSQL developers are
Mac users, including myself. They're also the company standard at
EnterpriseDB - so we're not entirely unfamiliar with software
development on them.

>>> - We have few Mac experts hanging out in #postgresql.
>> Not sure how this is relevant to the proposal.
>
> The impetus for the idea was that there seems to be a steady stream of
> novice PG users on Mac who come into #postgresql with installation problems,
> which is bad enough as an out-of-box experience - but worse is that there
> are rarely folks around who can help. (Of course, I'm extrapolating; every
> time *I'm* in IRC and see this, there's someone who can help. But you know
> what I mean.)

If you see someone report a bug with the installers, please have them
report it on the EnterpriseDB forums:
http://forums.enterprisedb.com/forums/show/9.page

> I didn't realize that you were actively maintaining the EDB installer (see
> below for the 8.4 doc explanation); obviously, if you can improve that, it's
> the best solution and we should, if anything, recommend it MORE vigorously.
> Still, there's a growing community of developers who expect "brew install"
> to work, and I do want to fix it for them.  The EDB installer will always be
> a one-off experience; most of the other servers you install will be through
> a package manager, and homebrew's popularity (despite its youth) is
> impressive.

I would disagree with that. Most users I know do not use things like
homebrew (particularly those coming from Windows who have no
familiarity with such package managers at all).

>  Both of my n=2 data points had run across PG a while back,
> installed it with the one-click to try it out, forgotten about it, done
> "brew install postgresql" today, and naturally ran into problems.

As I said, that will happen with any distro. The installers are smart
enough to detect it and avoid trying to reuse the same port. They
won't ever try to touch an existing installation though (except of
their own, which if detected will cause a switch to upgrade mode).

>
>>> - The EDB docs are written against 8.4.
>> Only if you install 8.4. If you install 8.3 you get the 8.3 docs, 9.0
>> the 9.0 docs and so on.
>
> No, I meant on the web:
>
> http://www.enterprisedb.com/resources-community/pginst-guide
>
> That's what made me assume that the installer wasn't maintained (except as
> to repackaging new PG versions, obviously). It's obviously not hard to
> replace "8.3" with "9.1" when you read it, but it still leaves an impression
> akin to "This web site works best with IE7 and above." Allow me to now
> replace most of this thread with "hey, you might wanna update that page."

That hasn't been updated because the installation steps haven't
changed and I'd rather spend time writing software than updating
screenshots. A couple of points of note:

- The introduction says: "This document is based on the 8.4.x
one-click installers but applies equally to later versions."

- The doc also explains where to find the uninstaller.

>>> - There are eight ways to install Postgres on a Mac
>
>> That isn't any more of a reason to discount the EDB installer than any
>> other.
>
> Nope, just an argument that the recommended installer should handle that
> nicely.

It does. It'll detect that the port is in use and suggest a different
one. I don't know of any other of those installation methods that'll
do that.

>
> 1. Rubyists in general are sick of sudo'ing on their laptops, because It
> Doesn't Matter (as I'll fail to argue later).  Homebrew puts itself into a
> directory that is user-writable so it does not require sudo for basic
> installation.

Nice. You just turned me off ever wanting anything related to Ruby on
my Mac either!

> 2. Because shell's $PATH is hard to change programmatically due to
> shell-config individualism (MacPorts was notorious for failing at this), and
> yet many Mac programmers know nothing of shells at all (so they don't know
> how to edit it manually), Homebrew puts itself into a directory that is
> already in $PATH by default, but which is conveniently nonexistent by
> default.

Are you honestly trying to tell me that a developer (using any
language, other than maybe vbscript in Excel) doesn't know about
$PATH?

> Thus, Homebrew chowns /usr/local to (desktop user):admin.
>
>
>> In any case, the fact that Homebrew does that to /usr/local should be
>> enough to make any user run away screaming in terror. If it opens up a
>> security hole like that, what else does it do to break your system?
>
> So this is pointless to the discussion now, but if you want to engage
> off-list, I'd frankly love to be reconvinced: Assume that I am on a laptop,
> not a server. There is one physical user, and that user is me. I am always
> logged in as that user. I am also conditioned to enter my password every
> time an installer says it needs me to. I am a developer, and I am writing
> software that will require me to keep my database credentials in cleartext
> in a directory readable by me, and in the non-PCI, non-SOX, non-HIPPA world
> of startups, those are certainly going to be superuser credentials for
> convenience. What, exactly, is the attack vector opened by installing
> postgres or other developer tools under my username? And what is the
> relative risk of being targeted along that vector, and not "You're at
> Starbucks and you're browsing Facebook without SSL", other than spear
> phishing attacks?

There are hundreds of thousands of pieces of malware for Windows that
relied on the ability to write to "system" directories like this to do
their misdeeds. Anywhere they can write (or modify existing) software
that may get executed at boot time or by an unsuspecting users (or
potentially, root). Microsoft spent millions, probably tens or
hundreds of millions enhancing the security of Windows precisely
because of this type of security issue.

If homebrew intentionally creates a hole like that, then for as long
as I'm one of the PostgreSQL webmasters it will *never* be listed on
our download pages.

>>> 2. The current formula installs Postgres as the desktop user, not as the
>>> _postgres role account.
>>
>> That's not very helpful on shared machines - and whilst it may be fine
>> for developers etc, it's not the recommended way to setup PostgreSQL
>> for any kind of production use.
>
> Oh, of course. Homebrew does not target the three remaining people who run
> production XServes.  It's purely for Mac developer workstations.  At
> startups. Which are MacBooks! :)

"Production" doesn't necessarily mean "server". All those thousands of
Poker Tracker users that run with PostgreSQL on Windows on their home
machines are production users for example.

-- 
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: Switching to Homebrew as recommended Mac install?

Andrew Dunstan <andrew@dunslane.net> — 2012-04-02T11:25:37Z

On 04/02/2012 05:23 AM, Dave Page wrote:
> There are hundreds of thousands of pieces of malware for Windows that
> relied on the ability to write to "system" directories like this to do
> their misdeeds. Anywhere they can write (or modify existing) software
> that may get executed at boot time or by an unsuspecting users (or
> potentially, root). Microsoft spent millions, probably tens or
> hundreds of millions enhancing the security of Windows precisely
> because of this type of security issue.
>
> If homebrew intentionally creates a hole like that, then for as long
> as I'm one of the PostgreSQL webmasters it will *never* be listed on
> our download pages.
>
>

+10

I'm one of the people who doesn't have a horse in this race. (Recently I 
was teaching my daughter about some development stuff and so I got her 
to install Fedora under VirtualBox on her Mac :-) ). But as soon as I 
saw this stuff about the intentional security hole it raised a great big 
red flag with me too.

cheers

andrew

Re: Switching to Homebrew as recommended Mac install?

Jay Levitt <jay.levitt@gmail.com> — 2012-04-02T12:17:07Z

Dave Page wrote:
> On Mon, Apr 2, 2012 at 12:29 AM, Jay Levitt<jay.levitt@gmail.com>  wrote:
> Just as an FYI, a large percentage of the PostgreSQL developers are
> Mac users, including myself. They're also the company standard at
> EnterpriseDB - so we're not entirely unfamiliar with software
> development on them.

Good to know; now I realize that even on Macs there are two worlds. I think 
it's important to understand the new world, because although very little 
infrastructure spans two generations, I believe that PostgreSQL is uniquely 
positioned to do exactly that, just as POSIX has. It's a performant ACID 
relational data store for the old guard, but it's also a web-scale 
cloud-friendly polyglot-extensible preinstalled 
XML/JSON/hstore/tree/graph/schemaless store for the new, and like Java 
HotSpot, it gets smarter and faster every day. It is an operating system for 
your data, it's free, and Oracle hasn't bought it (except yesterday).

This goes back to the "marketing challenge, not technical challenge" threads.

>>>> - We have few Mac experts hanging out in #postgresql.
>>> Not sure how this is relevant to the proposal.
>> The impetus for the idea was that there seems to be a steady stream of
>> novice PG users on Mac who come into #postgresql with installation problems,
> If you see someone report a bug with the installers, please have them
> report it on the EnterpriseDB forums:
> http://forums.enterprisedb.com/forums/show/9.page

It's not a bug with the EDB installer, except insofar as the bug is "the EDB 
installer is not prominent and widely recommended enough and so they used 
homebrew or macports". The EDB installer is what they used once upon a time; 
today they are following a blog post that walked them through homebrew 
(probably as one line of a longer process involving memcached, sphinx, 
nginx, node.js, and seven other things).

>> The EDB installer will always be
>> a one-off experience; most of the other servers you install will be through
>> a package manager, and homebrew's popularity (despite its youth) is
>> impressive.
>
> I would disagree with that. Most users I know do not use things like
> homebrew (particularly those coming from Windows who have no
> familiarity with such package managers at all).

Yep; two worlds.

 >> [docs reference 8.4]
> That hasn't been updated because the installation steps haven't
> changed and I'd rather spend time writing software than updating
> screenshots.

Fair. I wonder if there's an easy way to automate the generation and 
insertion of those screen shots. You could always write an 
AppleScript/Automator/Keyboard Maestro script, but I feel like I've seen a 
better way specifically for tech docs; I'll keep an eye out.

> A couple of points of note:
>
> - The introduction says: "This document is based on the 8.4.x
> one-click installers but applies equally to later versions."
>
> - The doc also explains where to find the uninstaller.

Sure, and if humans read docs, instead of just glancing at them, that'd be 
all you needed. In any case, I could counter myself that nobody reads the 
doc period, so it doesn't matter what version is listed; that's just the 
source of my own misunderstanding about maintenance.

>>>> - There are eight ways to install Postgres on a Mac
>>> That isn't any more of a reason to discount the EDB installer than any
>>> other.
>> Nope, just an argument that the recommended installer should handle that
>> nicely.
>
> It does. It'll detect that the port is in use and suggest a different
> one. I don't know of any other of those installation methods that'll
> do that.

That's terrific, and you're right - it's the only one.

 >>>> 2. The current formula installs Postgres as the desktop user, not as the
 >>>> _postgres role account.
 >>> That's not very helpful on shared machines - and whilst it may be fine
 >>> for developers etc, it's not the recommended way to setup PostgreSQL
 >>> for any kind of production use.
 >> Oh, of course. Homebrew does not target the three remaining people who run
 >> production XServes.  It's purely for Mac developer workstations.  At
 >> startups. Which are MacBooks! :)
 > "Production" doesn't necessarily mean "server". All those thousands of
 > Poker Tracker users that run with PostgreSQL on Windows on their home
 > machines are production users for example.

Excellent point, and one I hadn't thought of.

>> 1. Rubyists in general are sick of sudo'ing on their laptops, because It
>> Doesn't Matter (as I'll fail to argue later).  Homebrew puts itself into a
>> directory that is user-writable so it does not require sudo for basic
>> installation.
>
> Nice. You just turned me off ever wanting anything related to Ruby on
> my Mac either!

Well, for "Rubyists", I should properly substitute "younger developers", aka 
"all developers eventually".

As more toolchains work without sudo (Ruby, Python, JavaScript), using sudo 
actually breaks stuff, because now you've written some files as root and 
you'll later fail to overwrite them as user. Heck, I've had that happen with 
ccache; if you do "sudo make install" without "make", you're compiling and 
caching as root, and three years later you'll recompile as user and get a 
permissions error (or a broken ccache IIRC).

> Are you honestly trying to tell me that a developer (using any
> language, other than maybe vbscript in Excel) doesn't know about
> $PATH?

I'm not only telling you that *a* developer doesn't know; I'm telling you 
that soon, *most* won't.  (Or, if they do, they know it's a magic 
incantation that when something breaks, a blog tells them what to type to 
fix it.) I work with a smart 2005 CS graduate who, when I said "look in 
/usr/lib", would try cd'ing to /Users/Library. He wrote his first shell 
script last week.

I'm not saying it's good. I'm just saying it is - and you're younger than 
me, so you're not going to get to retire before it happens either!

First, go read this, and understand our future userbase, the college class 
of 2015:

http://www.beloit.edu/mindset/2015/

In that vein, here's my take on the average whiz kid of the next five years:

- They have never written C. (It hasn't been taught to undergrads in a 
decade.) They have never used pointers, preprocessors, X11, or spinlocks. 
They have never seen a hex dump or assembly, never heard of a core dump, and 
never run out of disk space, let alone thought to check for it happening.

- When you say "shell", they say "You mean Terminal?" They have not heard of 
autoconf, m4, or POSIX.

- "Here is the whole computer that vi was created on." (They think the 
terminal is a computer: 
http://ignoble-experiment.blogspot.com/2012/03/here-is-why-vim-uses-hjkl-keys-as-arrow.html)

- They will never step inside a data center.

- Caching makes things faster, but "the browser takes care of it for you".

- Compression has always been faster AND smaller. Encryption has never been 
expensive.

- You have always been able to Google any error message.

And the next 10-15:

- Most software has always been open source, on github, and interpreted.

- JavaScript has always been a perfect language because you can write your 
server and your client and your mobile app in it.

- They do not know why Linux "used to be called UNIX".

- Every computer and cell phone has always been continuously connected to 
(and backed up to) the Internet, which has always been fast enough for 
video, and has always magically auto-configured everything. They do not own 
an Ethernet cable and have only heard a modem in old movies.

- Hard drives have never had rotational latency; pixels have always been too 
small to see. Documents have always auto-saved and versioned, cell phones 
have always been multi-core and multitouch, and computers have never needed 
to boot.

- Windows is what their parents used. (Unless Win8 really saves the 
company.)  IBM built the first computer, didn't they? Their cell phones, 
computers, and TVs (the three standard development targets) have always been 
POSIX-based.

- There have never been DBAs; developers have always used ORMs or maybe 
written some inline SQL.

- IPv6 has always wor-- nah, that's crazy talk.

You get the idea. Just as we grew up not having to solder our computers 
(much) or boot them from the front panel, they've always had a whole layer 
of infrastructure that either magically works or that they can fix by 
Googling and typing without understanding the internals.  (GitHub did not 
need their own sysadmins until December.) Let's make PG part of that.

Jay

Re: Switching to Homebrew as recommended Mac install?

Dave Page <dpage@pgadmin.org> — 2012-04-02T14:09:36Z

On Mon, Apr 2, 2012 at 8:17 AM, Jay Levitt <jay.levitt@gmail.com> wrote:
>
> Sure, and if humans read docs, instead of just glancing at them, that'd be
> all you needed. In any case, I could counter myself that nobody reads the
> doc period, so it doesn't matter what version is listed; that's just the
> source of my own misunderstanding about maintenance.

Seems odd that you claim the docs are out of date in one breathe, then
in the next counter with an argument that users never read docs so
they're basically irrelevant.

> Well, for "Rubyists", I should properly substitute "younger developers", aka
> "all developers eventually".
>
> As more toolchains work without sudo (Ruby, Python, JavaScript), using sudo
> actually breaks stuff, because now you've written some files as root and
> you'll later fail to overwrite them as user. Heck, I've had that happen with
> ccache; if you do "sudo make install" without "make", you're compiling and
> caching as root, and three years later you'll recompile as user and get a
> permissions error (or a broken ccache IIRC).

I'm not aware that either Python or JavaScript require that you remove
sensible and long established security measures from your machine.
Come to think of it, neither did the MacPorts installation of Ruby
that I used to have to run a couple of utilities, nor does the version
that Apple ship with OS X.

I would suggest that it's not modern languages that require bypassing
of security measures, but a poorly designed packaging system.

> I'm not only telling you that *a* developer doesn't know; I'm telling you
> that soon, *most* won't.  (Or, if they do, they know it's a magic
> incantation that when something breaks, a blog tells them what to type to
> fix it.) I work with a smart 2005 CS graduate who, when I said "look in
> /usr/lib", would try cd'ing to /Users/Library. He wrote his first shell
> script last week.
>
> I'm not saying it's good. I'm just saying it is - and you're younger than
> me, so you're not going to get to retire before it happens either!

I run multiple teams of engineers at EnterpriseDB, and many of them
are basically fresh out of university (or were when they joined EDB).
Every single one of them knows what the path is and how to change it,
and I can say with confidence that every one of them could explain to
you what's wrong with making /usr/local/ world writeable. They may not
be able to code in a particular language, but they all have the
ability to learn and the basic skills on which to build. If the
students of the future have as little knowledge of computer systems as
you suggest, we might as well throw away our machines now and go live
in caves or huts!

And on that note, I suggest we take this off pgsql-hackers now as it's
drifting (drifted) way off topic for the list.

-- 
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: Switching to Homebrew as recommended Mac install? / apology

Jay Levitt <jay.levitt@gmail.com> — 2012-04-02T16:02:49Z

David Johnston wrote:
 > Just trying to bridge an apparent gap since the original e-mail seems to
 > have come across as too adversarial that the underlying thoughts have
 > been overlooked.  Trying to contribute in my own way with my current
 > resources.

Thanks, but it's my own fault for basing a half-baked "let's rewrite 
everything" idea on a few wrong assumptions without asking some simple 
questions first. (Sorry, David.) And you guys don't know me very well yet, 
so you don't how to interpret my tone, or that I spend the *first* half of 
each day making the exact opposite arguments to all the young turks in the 
hopes we'll all meet in the middle.  I plan to hang around, so by way of 
belated introduction, and you can imagine this in the style of Beetlejuice:

Hi. I wrote AOL's OLTP-style mail server in 1994 and scaled it (with an 
eventual team of 25) from 27 to 4000 TPS on minicomputers as powerful as an 
iPhone. It had multi-site replication, zero-downtime installs (without 
blocking writes), and served 1.5 million simultaneous users. I had to write 
nearly a whole SVM-based OS in the app, because nothing at the time - from 
the message bus to the disk cache to poll() itself - could handle our load, 
and our growth outpaced the hardware manufacturers' ability to build it. I 
did this with no CS background beyond public school (I started as a 6502 
hacker), no search engine, and no access to research papers or computer 
scientists. I have no idea how.

The architecture survived five underlying infrastructure transitions 
(Stratus filesystem, Sybase, Informix, Tandem, and the move from four- to 
eight-byte IDs that PG has successfully staved off) while migrating live. It 
never lost or misrouted a message, and never had a security breach in the 
nine years I led it (or, AFAIK, since), despite the fact that we were a big 
enough target for hackers to respond to our changed defenses within hours. I 
do actually know this stuff, or did.

I spent 10 years taking a break, studying music, successfully sleeping 
through Java; now I'm back in technology, leading development in a 
Rails/web/JS startup, and luxuriating in the ability to actually store data 
in a database this time - because you guys have spent the past 20 years 
studying and improving the hard parts of abstracting performant, reliable, 
scalable data storage.

I have a tendency to see the possible endgame and insist on starting now, 
and if I get too idealistic, ambitious, and "MongoDB is webscale", please 
just drag me into a conversation about lock contention and distributed 2PC 
and I'll forget the whole thing. But I really do think PG can be the 
makes-everything-possible, does-most-things-well data store - really, data 
platform - for the next decade or two, and I want to contribute.

I'm provocative, playful and grandiose, I apologize except not really, and 
it's because in my n=1 experience, the way life works is (a) you decide to 
change the world and then (b) you do.

 > You do not need permission to contribute to the community
 > in the way you seek so what is it that you are really asking for?

Nothing at this point. I was thinking out loud, and at the time was 
temporarily insa^h^h^hconvinced that the homebrew formula should be the 
community standard, and thus that I'd have to bring it up to some level of 
acceptability/review.  I've contributed to the formula in the past, and will 
continue to do so based on the thoughts everyone's shared here. It doesn't 
need to be official to be useful, and as David Page said, it's not gonna be 
listed in the docs no matter what, given the one decision that homebrew 
makes (/usr/local) that I can't override.

> When brew is replaced by something more popular do you
> think you will continue to maintain the recipe or is it going to end
> up stuck showing us how to install version 9.3 or earlier.

Like anything, I'll maintain it until it becomes useless to me or vice 
versa, and someone will pick it up or they won't.  But just to be clear, 
Homebrew's a source-based repo (so there's no cross-compiler issues), 
pulling from the upstream source repository, using only the stock compiler 
toolchain, Intel-only, on a platform where the only hardware manufacturer 
has themselves severely constrained the number of possible configurations. 
For the most part, updating the formula to "package" new versions is a 
matter of changing the following two lines:

   url 'http://ftp.postgresql.org/pub/source/v9.1.3/postgresql-9.1.3.tar.bz2'
   md5 '641e1915f7ebfdc9f138e4c55b6aec0e'

Unless the instructions for "How to build postgres from source" change, 
nothing else in the formula *needs* to.  The current formula is fairly 
simple; aside from user instructions, the code is 75 lines and mostly 
consists of default arguments to ./configure.  (Formula attached for the 
curious.)  Pull requests are freely and quickly accepted after a quick 
review; the homebrew repo is operated more in the "fail early and often" 
spirit than in the mission-critical "do no releases before formal 
verification" style - and again, since it contains nothing more than 
programmatic versions of "install this package from source" docs on a 
constrained platform, it doesn't tend to fail catastrophically. It's for 
developer tools, not deployments.

>  the current installer maintainers are doing so in addition to their
 > other, more regular, contributions

Yes, and I do plan to contribute back to PostgreSQL as well. We have a bunch 
of novel uses that could serve as forcing functions for nice optimizations 
and generalizations, and my lifelong obsession with scaling, performance and 
metering meshes well with some of the core team's future plans. But as a PG 
novice, a SQL novice, and someone who last touched C in 1999 other than a 
few bug fixes, believe me that I am contributing more by *not* trying to 
contribute any code *just* yet.

> Someone would likely pickup the torch

Agreed; the PG formula has had 35 contributors in its 2.5 years, and the 
9.1.3 update was released 41 minutes after the e-mail announcement went out.

The homebrew repo's at https://github.com/mxcl/homebrew, and the license 
(homebrew-wide) is unnamed but looks MIT-compatible. It does not contain the 
PG copyright notices, and I don't know the history of the PG license, so 
perhaps that also prevents some here from contributing (although, again, 
there is no PG code in homebrew).

Jay

Fwd: [HACKERS] Switching to Homebrew as recommended Mac install?

David Johnston <polobo@yahoo.com> — 2012-04-02T16:27:38Z

Moving from hackers

Begin forwarded message:

> From: Jay Levitt <jay.levitt@gmail.com>
> Date: April 2, 2012 5:17:07 MST
> To: Dave Page <dpage@pgadmin.org>
> Cc: PG Hackers <pgsql-hackers@postgresql.org>
> Subject: Re: [HACKERS] Switching to Homebrew as recommended Mac install?
> 
> 
> This goes back to the "marketing challenge, not technical challenge" threads.
> 

Actually it seems the question is whether we suggest trading security for convenience.  If the rest of PostgreSQL is something these whiz kids want or need to use they will learn what is needed to install it.  If they work for a company likely someone will train them.  If they do not they will teach the selves from some form of textbook - whether that be a blog post or official high-quality documentation.

> 
> Sure, and if humans read docs, instead of just glancing at them, that'd be all you needed. In any case, I could counter myself that nobody reads the doc period, so it doesn't matter what version is listed; that's just the source of my own misunderstanding about maintenance.

In a project of this magnitude there is only so much you can learn via trial and error.  Thus original learning requires some (multiple) forms of written text.  Once you have some people with the knowledge you can train others.  The documentation is there and really smart people will realize that they cannot know everything and so they should read documentation.  Its our job to make sure the trough is full.  If the horse dehydrates because it doesn't want to drink I don't lose any sleep - I'll just look for a different horse.  That said I'll at least to make the software indicate when there is an unusual configuration and give the user some guidance on what to look for.  But I'd generally rather strictly notify and let the user make an informed, manual, decision on how to resolve the issue instead of assuming and hoping the cure is no worse than the disease.

> 
>>>>> - There are eight ways to install Postgres on a Mac
>>>> That isn't any more of a reason to discount the EDB installer than any
>>>> other.
>>> Nope, just an argument that the recommended installer should handle that
>>> nicely.
>> 
>> It does. It'll detect that the port is in use and suggest a different
>> one. I don't know of any other of those installation methods that'll
>> do that.
> 
> That's terrific, and you're right - it's the only one.
> 
> 
> 
> In that vein, here's my take on the average whiz kid of the next five years:
> 
> 

Most of those seem irrelevant and none of them are followed by specific actions the community can take to adapt to the new normal.

> You get the idea. Just as we grew up not having to solder our computers (much) or boot them from the front panel, they've always had a whole layer of infrastructure that either magically works or that they can fix by Googling and typing without understanding the internals.  (GitHub did not need their own sysadmins until December.) Let's make PG part of that.

That means nothing without more context.  And while making Postgres "part of that" sounds all good I still haven't and good reasons the status quo is failing nor specific actions that can be taken toward that end.  The few things you mentioned you can do without any permission from the community and getting listed on the website seems to require that you make security a higher priority.  That's just how we roll here and it is highly unlikely that will change.

> 
> 
> Jay
> 
> 

David J.

Fwd: [HACKERS] Switching to Homebrew as recommended Mac install? / apology

David Johnston <polobo@yahoo.com> — 2012-04-02T16:49:41Z

Moving from Hackers as well...

Begin forwarded message:

> From: Jay Levitt <jay.levitt@gmail.com>
> Date: April 2, 2012 9:02:49 MST
> To: David Johnston <polobo@yahoo.com>
> Cc: Tom Lane <tgl@sss.pgh.pa.us>,  PG Hackers <pgsql-hackers@postgresql.org>
> Subject: Re: [HACKERS] Switching to Homebrew as recommended Mac install? / apology
> 
> 
> 
> Nothing at this point. I was thinking out loud, and at the time was temporarily insa^h^h^hconvinced that the homebrew formula should be the community standard, and thus that I'd have to bring it up to some level of acceptability/review.  I've contributed to the formula in the past, and will continue to do so based on the thoughts everyone's shared here. It doesn't need to be official to be useful, and as David Page said, it's not gonna be listed in the docs no matter what, given the one decision that homebrew makes (/usr/local) that I can't override.

I'm curious if David Page et al. would list the recipie if it contained a disclaimer regarding the security flaws inherent in the underlying location choice and suggesting it only be used for developers and not production?

One reason to do so would be so that people who know and would want to use homebrew would be able to see that, yes, one exists but that there are security implications resulting from its use.  The recipie itself should also provide these warnings and explanations but the community download page could be used as an opportunity to try and teach these people more about security.

It makes no sense to pretend that dangerous things do not exist but rather we should teach about the dangers and try to provide alternatives.

David J.

Re: Switching to Homebrew as recommended Mac install?

Greg Stark <stark@mit.edu> — 2012-04-02T17:44:33Z

On Mon, Apr 2, 2012 at 5:29 AM, Jay Levitt <jay.levitt@gmail.com> wrote:
> So this is pointless to the discussion now, but if you want to engage
> off-list, I'd frankly love to be reconvinced:

It may not be an unreasonable thing for an individual user to do to
their own machine. But it's not really Postgres's place to be
re-engineering the whole security model of the user's machine based on
our beliefs of what's important.

-- 
greg

Re: Switching to Homebrew as recommended Mac install?

Robert Haas <robertmhaas@gmail.com> — 2012-04-03T12:22:31Z

On Mon, Apr 2, 2012 at 5:23 AM, Dave Page <dpage@pgadmin.org> wrote:
> If homebrew intentionally creates a hole like that, then for as long
> as I'm one of the PostgreSQL webmasters it will *never* be listed on
> our download pages.

I think that's a bit harsh.  It's not as if the PostgreSQL package
creates the security hole; it's something that the packaging system
does itself, independent of whether or not you try to install
PostgreSQL with it.  So it seems to me that refusing to list it is
making life difficult for people who have already made the decision to
use brew, without any compensating advantage.

That doesn't mean that I approve of brew's approach to this problem,
though.  Even if you think that it's unimportant to keep the desktop
user from usurping root privileges, having some things installed in
/usr/local as root and others as the desktop user (multiple different
desktop users?) seems like a recipe for chaos.  I've made those types
of mistakes, but I got them out of my system in the nineties.  I can't
help but wonder if this isn't just the natural way a packaging system
evolves - you start with something very simple (like what brew is now)
and then gradually you realize that there are some annoyances, so you
file those down by adding some more complexity, and eventually you end
up with a system that's just as complex as the ones that you
originally thought were too complex.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: Switching to Homebrew as recommended Mac install?

Christopher Browne <cbbrowne@gmail.com> — 2012-04-03T15:56:26Z

On Tue, Apr 3, 2012 at 8:22 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> On Mon, Apr 2, 2012 at 5:23 AM, Dave Page <dpage@pgadmin.org> wrote:
>> If homebrew intentionally creates a hole like that, then for as long
>> as I'm one of the PostgreSQL webmasters it will *never* be listed on
>> our download pages.
>
> I think that's a bit harsh.  It's not as if the PostgreSQL package
> creates the security hole; it's something that the packaging system
> does itself, independent of whether or not you try to install
> PostgreSQL with it.  So it seems to me that refusing to list it is
> making life difficult for people who have already made the decision to
> use brew, without any compensating advantage.

+1.

If someone has decided that they want to use homebrew on their Mac,
then they have accepted the whole class of issues of this sort for a
bunch of *other* software.  I don't think Postgres is that much more
special.

And note that if they're using Ruby to access the Postgres instance,
and there are (perceived) security holes with the Ruby toolchain's
installation, it doesn't much matter if the Postgres installation is
'more secure', it's being accessed via a layer that is already Plenty
Holey.

> That doesn't mean that I approve of brew's approach to this problem,
> though.  Even if you think that it's unimportant to keep the desktop
> user from usurping root privileges, having some things installed in
> /usr/local as root and others as the desktop user (multiple different
> desktop users?) seems like a recipe for chaos.  I've made those types
> of mistakes, but I got them out of my system in the nineties.  I can't
> help but wonder if this isn't just the natural way a packaging system
> evolves - you start with something very simple (like what brew is now)
> and then gradually you realize that there are some annoyances, so you
> file those down by adding some more complexity, and eventually you end
> up with a system that's just as complex as the ones that you
> originally thought were too complex.

I suspect that this is Yet Another Case of people using a mostly
"desktop/single user" system building something that's perfectly
convenient for that case.

It's pretty typical for MacOS applications to require "enter your
password; I need to su to root to install this!" in plenty of places
where the UI does not actually tell you what is being done as root.
After enough iterations of "enter your password so my process can do
undisclosed admin stuff," I'm not sure that you've got anything more
secure than you'd have if /usr/local was writable by the desktop user.

I'm not sure you can successfully wrench people in that environment
into a "proper" multi-user setup.  It took Microsoft *years* to get it
to a semblance of working, and people still have a flurry of "oh,
click this to give administrative permissions" activities that are
*begging* to be an exploit.
-- 
When confronted by a difficult problem, solve it by reducing it to the
question, "How would the Lone Ranger handle this?"

Re: Switching to Homebrew as recommended Mac install?

Robert Haas <robertmhaas@gmail.com> — 2012-04-03T16:25:20Z

On Tue, Apr 3, 2012 at 11:56 AM, Christopher Browne <cbbrowne@gmail.com> wrote:
> It's pretty typical for MacOS applications to require "enter your
> password; I need to su to root to install this!" in plenty of places
> where the UI does not actually tell you what is being done as root.
> After enough iterations of "enter your password so my process can do
> undisclosed admin stuff," I'm not sure that you've got anything more
> secure than you'd have if /usr/local was writable by the desktop user.

I think that's somewhat true.  Part of the reason why Windows is so
crufty is because of lousy privilege separation -- MacOS and even
Ubuntu are now busy copying that design, but realistically people do
need to install software, so short of educating users better or
creating a walled garden it's not clear how much better you can do.
However, Windows has another problem, too: there are so many ways that
you can hook into the system and get control that it's basically
impossible to remove spyware and crapware without its cooperation.  I
mean, you can get rid of some of it, but finding all the little fiddly
pieces that have to be ripped out is basically not doable.  You just
reinstall the machine.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: Switching to Homebrew as recommended Mac install?

Jay Levitt <jay.levitt@gmail.com> — 2012-04-03T19:14:44Z

Robert Haas wrote:
> On Mon, Apr 2, 2012 at 5:23 AM, Dave Page<dpage@pgadmin.org>  wrote:
>> If homebrew intentionally creates a hole like that, then for as long
>> as I'm one of the PostgreSQL webmasters it will *never* be listed on
>> our download pages.
>
> I think that's a bit harsh.  It's not as if the PostgreSQL package
> creates the security hole; it's something that the packaging system
> does itself, independent of whether or not you try to install
> PostgreSQL with it.  So it seems to me that refusing to list it is
> making life difficult for people who have already made the decision to
> use brew, without any compensating advantage.

In fairness to Dave, I think he was still reacting to my initial proposal 
that homebrew be the *recommended* install. In that case, people might 
install homebrew specifically because postgresql.org recommended it.

If the consensus is that /usr/local/* user-writable is insecure, I certainly 
wouldn't object to a little note that said "If you're using homebrew, do 
'brew install postgresql', but we don't recommend homebrew for security 
reasons; a little pressure might provide the impetus for homebrew to allow a 
better way.

That said, about 8 months ago Homebrew's defaults changed. It no longer 
requires /usr/local to be directly writable; it will sudo if necessary 
during the initial installation to create its subdirectories.  Those 
directories are mostly user-writable, though:

% ls -l /usr/local
total 8
drwxr-xr-x   37 jay   admin   1.2K Mar 31 16:39 Cellar/
drwxr-xr-x    7 jay   admin   238B Feb 29 10:51 Library/
-rw-r--r--    1 jay   admin   789B Feb 29 10:57 README.md
drwxr-xr-x  499 jay   admin    17K Apr  1 15:29 bin/
drwxr-xr-x    9 jay   admin   306B Mar  7 16:23 etc/
drwxr-xr-x   69 jay   admin   2.3K Mar 16 16:48 include/
drwxr-xr-x  178 jay   admin   5.9K Mar 16 16:48 lib/
drwxr-xr-x    3 root  admin   102B Mar 14 13:20 man/
drwxr-xr-x   20 jay   admin   680B Mar 31 16:40 share/
drwx------    3 jay   admin   102B Feb 29 11:43 var/

At no point was anything in /usr/local *world*-writable, FWIW.

> That doesn't mean that I approve of brew's approach to this problem,
> though.  Even if you think that it's unimportant to keep the desktop
> user from usurping root privileges, having some things installed in
> /usr/local as root and others as the desktop user (multiple different
> desktop users?) seems like a recipe for chaos.

I think the brew designers expect most folks to either not have anything in 
/usr/local from outside homebrew, to not install anything there as root, to 
understand the security consequences, or to use homebrew as root even though 
it's unsupported, and deal with their own bugs.

> I can't help but wonder if this isn't just the natural way a packaging system
> evolves - you start with something very simple (like what brew is now)
> and then gradually you realize that there are some annoyances, so you
> file those down by adding some more complexity, and eventually you end
> up with a system that's just as complex as the ones that you
> originally thought were too complex.

Packaging systems? I thought that's called "all software ever"!

brew's lucky in that the Mac by definition is not a heterogeneous 
environment, and so Mac users don't expect it to be. (Last I checked, it's 
either difficult, impossible or unsupported to boot from a volume other than 
your filesystem root.) By being mostly source-fetch-only, sitting on top of 
git, and not packaging either system-provided features (many of which would 
require root) or repackaging gems/eggs/nodes, they've avoided a lot of the 
hard problems.  But sure, it's only two years old and it will get more 
complex over time.

Jay

Re: Switching to Homebrew as recommended Mac install?

Greg Smith <greg@2ndquadrant.com> — 2012-04-03T22:58:56Z

On 04/01/2012 04:19 PM, Jay Levitt wrote:
>> POSSIBLE OBJECTIONS/PREREQUISITES
>
> 10. There is no homebrew support for multiple versions, and no current
> plans to add it (though it's on the wishlist). This means homebrew is
> only useful if "I want to install a PostgreSQL thingie" is the common
> Mac use case. If people often need to use specific older versions, to
> mirror their server configs, it's a problem. It *might* be possible to
> hack this into our formula, but I'm not sure it's either doable or
> acceptable.

To be a credible alternative to EDB's packages, Homebrew would need to 
match having installers for all of the supported versions at 
http://www.postgresql.org/support/versioning/ , which right now means 
back to 8.3.  While I'm often annoyed at "checkbox compliance" 
exercises, that's unlikely to be a negotiable one.  Your optimism that 
it's "just switch the version number to build another version" will make 
everyone who currently builds packages roll their eyes too.  Wandering 
this road will end up with a formula for each version, because they will 
be different sometimes, and there's going to be packaging bug 
backporting coming out of that.  Just be glad you don't have to worry 
about anything before 8.3 now.

I just started playing in this area recently.  I have a shell utility 
whose purpose in life is to make it easier to install multiple 
PostgreSQL builds from source:  https://github.com/gregs1104/peg

I confirmed recently that it works fine on OS X using brew.  Was even 
toying with renaming the project ('peg' is already squatted on in 
Homebrew) and packaging it up.  There are enough people who hack on 
PostgreSQL on Macs using Homebrew that I think I'd find a few users. 
I'm not sure if someone would be willing to get stuck with maintaining 
that if it got merged into an official package though.

That's wavering off topic though, so back to more officialish 
installers.  You've asked one question and discussion has implied a few 
others.  Let me try to iterate over them with my opinions:

-Will PostgreSQL switch to Homebrew as the recommended Mac install soon?

No.  You've already hit a few nerves suggesting why, as well as 
suggesting some of the issues in your first e-mail.  There's a larger 
problem challenging Homebrew too.  This sort of Mac packaging has never 
escaped having a flavor of the week feel to it, and as a group the 
people working on PostgreSQL are not fans of switching things regularly 
like that.  Don't take that personally:  this is a project that stayed 
on CVS until late 2010, partly because the alternatives didn't seem 
stable enough.  And when we did switch, it was only after finding 
problems that had to be fixed in the tools used.  You just got a first 
round of such griping here.

I do believe Homebrew is the lead dog in the build from source race 
right now, but the history here says the right bet is on none of 
them--that the simplest feasible binary install is the right choice for 
the "recommended" package.  That's what EDB is trying to do, and some of 
the important associated details like "make it easy to use pgAdmin" are 
not really negotiable.

-Should there be better documentation on using Homebrew to satisfy 
people who don't like the assumptions made by the EDB Mac packages?

Yes.  We've had a link from 
http://wiki.postgresql.org/wiki/Detailed_installation_guides to one of 
the early guides for a while. I've been wanting to see a whole page 
hosted directly on the wiki covering this subject; I just created 
https://wiki.postgresql.org/wiki/Homebrew for that purpose.  I'll link 
it into the main install guide section once it's more interesting.

-Should Homebrew add [automatic initdb|service script 
integration|compatibility with using a postgres user]?

Yes to all of these.  If you're going to argue from a perspective like 
"make it easy for Ruby developers", it really should be easy in the end, 
to make for a compelling change from the status quo.

I think you've set your sights beyond their needs.  If the Homebrew 
recipe becomes better, and some rough edges are removed, those are the 
important parts.  I don't think Homebrew shouldn't aim to satisfy 
everyone; it should do a really good job satisfying its intended 
audience.  It is not necessary to make it the "recommended" installer 
for that to happen.  I'd suggest dumping that whole idea as too 
ambitious and focus just how to make things better for the intended 
targets.  Homebrew is never going to be an "enterprise" satisfying 
installer, since most businesses will not tolerate building from source 
themselves unless pressed extremely hard.

-What needs to change in Homebrew before it would be considered more 
seriously by the PostgreSQL community?

You framed your discussion points as a technical argument.  Good 
execution there is a necessary component, but not the only one.  I 
mentioned pgAdmin (and "Formulas are simple if building is simple" will 
surely be of no help there) and multiple version builds already; some 
other sub-topics are important too.

--Security

There's a tricky line between 'secure by default' and 'easy to use' that 
people are always fighting about.  Impossible to make everyone happy at 
the same time.  Right now I think Homebrew is skewed a bit far toward 
the 'easy' side for comfort.  There's surely a better middle ground to 
be found here though, and there are multiple Mac wielding developers who 
know PostgreSQL packaging floating around.  I've been wondering if it's 
worthwhile to consider a pair of formula, differing only in this area: 
a clearly labeled unsecured but easy to use one, and a production server 
one.

--Critical release updates

Another thing that would need to get nailed down is the staff that will 
be maintaining each of the supported PostgreSQL versions, along with 
responding to things like sensitive security updates.  I know where to 
find Dave Page and his team at EDB--I just talked with him last night. 
Homebrew needs some time to build up more direct PostgreSQL community 
connections before the project is going to be trusted similarly.

--Support contacts

We'd need to have a number of people committed to helping with formula 
problems on any version of PostgreSQL, who can schedule commits to the 
master repo.  "Anyone can push an update!" != "someone will push an 
update to the right place".

A lesson from the git conversion here would be to work bottom-up.  The 
whole community doesn't just move to something new via orders from some 
top; it happens by converting one developer at a time until there's a 
critical mass convinced.

--Documentation

Homebrew will have to become more complicated if it's going to try and 
wander down this path.  With complexity and backward compatibility come 
increased needs for documentation.

-- 
Greg Smith   2ndQuadrant US    greg@2ndQuadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com

Re: Switching to Homebrew as recommended Mac install?

Josh Berkus <josh@agliodbs.com> — 2012-04-03T23:48:12Z

On 4/3/12 5:22 AM, Robert Haas wrote:
> On Mon, Apr 2, 2012 at 5:23 AM, Dave Page <dpage@pgadmin.org> wrote:
>> If homebrew intentionally creates a hole like that, then for as long
>> as I'm one of the PostgreSQL webmasters it will *never* be listed on
>> our download pages.

I don't agree. Listed with a warning, sure.  But it should be listed.

Consider that OSX is pretty much purely a desktop platform (if you're
using OSX on the server, security is the least of your problems).  As
such, it doesn't have the same security concerns which server platforms
have.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

Re: Switching to Homebrew as recommended Mac install?

Dave Page <dpage@pgadmin.org> — 2012-04-04T00:19:14Z

On Tue, Apr 3, 2012 at 7:48 PM, Josh Berkus <josh@agliodbs.com> wrote:
> On 4/3/12 5:22 AM, Robert Haas wrote:
>> On Mon, Apr 2, 2012 at 5:23 AM, Dave Page <dpage@pgadmin.org> wrote:
>>> If homebrew intentionally creates a hole like that, then for as long
>>> as I'm one of the PostgreSQL webmasters it will *never* be listed on
>>> our download pages.
>
> I don't agree. Listed with a warning, sure.  But it should be listed.

That's fine - you don't have to agree with me :-).

> Consider that OSX is pretty much purely a desktop platform (if you're
> using OSX on the server, security is the least of your problems).  As
> such, it doesn't have the same security concerns which server platforms
> have.

Can I interest you in a move to Windows XP, pre security shake-up?
It'll be pretty darn snappy on modern hardware!

Having the ability for users to write files to system locations was
one of the major reasons why Windows got into such problems. The only
difference with this situation is that instead of users running with
admin privileges as often (but not always) happened on Windows back
then, we're talking about making parts of the filesystem
world-writeable so it doesn't even matter if the user is running as an
admin for a trojan or some other nasty to attack the system.

That said, Jay has told me he was wrong about the world-writeable
thing anyway. Apparently Homebrew fixed that last year.

-- 
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: Switching to Homebrew as recommended Mac install?

Dave Page <dpage@pgadmin.org> — 2012-04-04T00:23:48Z

On Tue, Apr 3, 2012 at 6:58 PM, Greg Smith <greg@2ndquadrant.com> wrote:
>
> --Documentation
>
> Homebrew will have to become more complicated if it's going to try and
> wander down this path.  With complexity and backward compatibility come
> increased needs for documentation.

One more to add:

--QA

When PostgreSQL updates are prepared, not only do they have to be
built but they also get tested in installation and upgrade mode on all
supported versions of OS X, including some of the server versions.
That obviously takes time and hardware, and is done over a brief
Friday - Sunday period to fit in with the release procedure.

-- 
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: Switching to Homebrew as recommended Mac install?

Greg Stark <stark@mit.edu> — 2012-04-04T03:12:18Z

On Wed, Apr 4, 2012 at 1:19 AM, Dave Page <dpage@pgadmin.org> wrote:
> then, we're talking about making parts of the filesystem
> world-writeable so it doesn't even matter if the user is running as an
> admin for a trojan or some other nasty to attack the system.

The argument is that a trojan or other nasty doesn't *need* to be
admin to attack the system since it can just attack the user's account
since that's where all the interesting data is anyways.

But again, this is all beside the point. It's a judgement for Apple
and Microsoft and individual administrators to make. We can't really
start reconfiguring people's systems to use a different security model
than they expect just because they've installed a database software --
even if we think our security model makes more sense than the one
their used to.

-- 
greg

Re: Switching to Homebrew as recommended Mac install?

Dave Page <dpage@pgadmin.org> — 2012-04-04T08:51:49Z

On Tue, Apr 3, 2012 at 11:12 PM, Greg Stark <stark@mit.edu> wrote:
> On Wed, Apr 4, 2012 at 1:19 AM, Dave Page <dpage@pgadmin.org> wrote:
>> then, we're talking about making parts of the filesystem
>> world-writeable so it doesn't even matter if the user is running as an
>> admin for a trojan or some other nasty to attack the system.
>
> The argument is that a trojan or other nasty doesn't *need* to be
> admin to attack the system since it can just attack the user's account
> since that's where all the interesting data is anyways.

Isn't that what I said?

> But again, this is all beside the point. It's a judgement for Apple
> and Microsoft and individual administrators to make. We can't really
> start reconfiguring people's systems to use a different security model
> than they expect just because they've installed a database software --
> even if we think our security model makes more sense than the one
> their used to.

Exactly - which is why I was objecting to recommending a distribution
of PostgreSQL that came in a packaging system that we were told
changed /usr/local to be world writeable to avoid the use/annoyance of
the standard security measures on the platform.

-- 
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: Switching to Homebrew as recommended Mac install?

Jay Levitt <jay.levitt@gmail.com> — 2012-04-04T20:19:08Z

Dave Page wrote:
> Exactly - which is why I was objecting to recommending a distribution
> of PostgreSQL that came in a packaging system that we were told
> changed /usr/local to be world writeable to avoid the use/annoyance of
> the standard security measures on the platform.

Wellll... that's not exactly what happened:

I originally wrote:
> POSSIBLE OBJECTIONS/PREREQUISITES
>
> 1. homebrew installs everything under /usr/local and makes that user-writeable.

So. :)

Jay