Thread

Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

Justin Clift <justin@postgresql.org> — 2016-04-11T16:39:07Z

Moving over a conversation from the pgsql-advocacy mailing list.  In it
Simon (CC'd) raised the issue of potentially creating a backwards-compatibility
breaking release at some point in the future, to deal with things that
might have no other solution (my wording).

Relevant part of that thread there for reference:

  http://www.postgresql.org/message-id/CANP8+jLtk1NtaJyXc=hAqX=0k+ku4zfavgVBKfs+_sOr9hepNQ@mail.gmail.com

Simon included a short starter list of potentials which might be in
that category:

  * SQL compliant identifiers
  * Remove RULEs
  * Change recovery.conf
  * Change block headers
  * Retire template0, template1
  * Optimise FSM
  * Add heap metapage
  * Alter tuple headers
  et al

This still is better placed on -hackers though, so lets have the
conversation here to figure out if a "backwards compatibility breaking"
release really is needed or not.

Hopefully we can get it all done without giving users a reason to consider
switching. ;)

Regards and best wishes,

Justin Clift

--
"My grandfather once told me that there are two kinds of people: those
who work and those who take the credit. He told me to try to be in the
first group; there was less competition there."
- Indira Gandhi

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

Robbie Harwood <rharwood@redhat.com> — 2016-04-11T19:23:05Z

Justin Clift <justin@postgresql.org> writes:

> Moving over a conversation from the pgsql-advocacy mailing list.  In it
> Simon (CC'd) raised the issue of potentially creating a backwards-compatibility
> breaking release at some point in the future, to deal with things that
> might have no other solution (my wording).
>
> Relevant part of that thread there for reference:
>
>   http://www.postgresql.org/message-id/CANP8+jLtk1NtaJyXc=hAqX=0k+ku4zfavgVBKfs+_sOr9hepNQ@mail.gmail.com
>
> Simon included a short starter list of potentials which might be in
> that category:
>
>   * SQL compliant identifiers
>   * Remove RULEs
>   * Change recovery.conf
>   * Change block headers
>   * Retire template0, template1
>   * Optimise FSM
>   * Add heap metapage
>   * Alter tuple headers
>   et al
>
> This still is better placed on -hackers though, so lets have the
> conversation here to figure out if a "backwards compatibility breaking"
> release really is needed or not.
>
> Hopefully we can get it all done without giving users a reason to consider
> switching. ;)

I'm sure this won't be a popular suggestion, but in the interest of
advocating for more cryptography: if we land GSSAPI auth+encryption, I'd
like the auth-only codepath to go away.

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

Robert Haas <robertmhaas@gmail.com> — 2016-04-11T19:37:32Z

On Mon, Apr 11, 2016 at 3:23 PM, Robbie Harwood <rharwood@redhat.com> wrote:
> I'm sure this won't be a popular suggestion, but in the interest of
> advocating for more cryptography: if we land GSSAPI auth+encryption, I'd
> like the auth-only codepath to go away.

I can't think of a reason that would be a good idea.  Occasionally
good things come from artificially limiting how users can use the
system, but mostly that just annoys people.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

Craig Ringer <craig@2ndquadrant.com> — 2016-04-12T00:52:23Z

On 12 April 2016 at 00:39, Justin Clift <justin@postgresql.org> wrote:

> Moving over a conversation from the pgsql-advocacy mailing list.  In it
> Simon (CC'd) raised the issue of potentially creating a
> backwards-compatibility
> breaking release at some point in the future, to deal with things that
> might have no other solution (my wording).
>
> Relevant part of that thread there for reference:
>
>
> http://www.postgresql.org/message-id/CANP8+jLtk1NtaJyXc=hAqX=0k+ku4zfavgVBKfs+_sOr9hepNQ@mail.gmail.com
>
> Simon included a short starter list of potentials which might be in
> that category:
>
>   * SQL compliant identifiers
>   * Remove RULEs
>   * Change recovery.conf
>   * Change block headers
>   * Retire template0, template1
>   * Optimise FSM
>   * Add heap metapage
>   * Alter tuple headers
>   et al
>

+

* v4 protocol (feature negotiation, lazy blob fetching, etc)
* retire pg_hba.conf and use SQL access management

?

-- 
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

Yury Zhuravlev <u.zhuravlev@postgrespro.ru> — 2016-04-12T13:12:23Z

Justin Clift wrote:
> Simon included a short starter list of potentials which might be in
> that category:
>
>   * SQL compliant identifiers
>   * Remove RULEs
>   * Change recovery.conf
>   * Change block headers
>   * Retire template0, template1
>   * Optimise FSM
>   * Add heap metapage
>   * Alter tuple headers
>   et al

+ CMake build I think.

Now I can build:
* postgres
* bin/* programs
* pl/* languages
* contrib/* (with cmake PGXS analogue)

Can run regression and isolation tests for postgres/pl* and all contrib 
modules. 

There is still a lot of work but I hope everything will turn out. Also it 
would be good to get help.

Thanks.

PS https://github.com/stalkerg/postgres_cmake

-- 
Yury Zhuravlev
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

Justin Clift <justin@postgresql.org> — 2016-04-12T14:08:03Z

On 12 Apr 2016, at 14:12, Yury Zhuravlev <u.zhuravlev@postgrespro.ru> wrote:
> Justin Clift wrote:
>> Simon included a short starter list of potentials which might be in
>> that category:
>> 
>>  * SQL compliant identifiers
>>  * Remove RULEs
>>  * Change recovery.conf
>>  * Change block headers
>>  * Retire template0, template1
>>  * Optimise FSM
>>  * Add heap metapage
>>  * Alter tuple headers
>>  et al
> 
> + CMake build I think.
> 
> Now I can build:
> * postgres
> * bin/* programs
> * pl/* languages
> * contrib/* (with cmake PGXS analogue)
> 
> Can run regression and isolation tests for postgres/pl* and all contrib modules. 
> There is still a lot of work but I hope everything will turn out. Also it would be good to get help.
> 
> Thanks.
> 
> PS https://github.com/stalkerg/postgres_cmake

If/when PostgreSQL can be built and tested with CMake... why would the
resulting code + database files + network protocol (etc) not be compatible
with previous versions? :)

+ Justin

--
"My grandfather once told me that there are two kinds of people: those
who work and those who take the credit. He told me to try to be in the
first group; there was less competition there."
- Indira Gandhi

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

Merlin Moncure <mmoncure@gmail.com> — 2016-04-12T16:23:24Z

On Mon, Apr 11, 2016 at 11:39 AM, Justin Clift <justin@postgresql.org> wrote:
> Moving over a conversation from the pgsql-advocacy mailing list.  In it
> Simon (CC'd) raised the issue of potentially creating a backwards-compatibility
> breaking release at some point in the future, to deal with things that
> might have no other solution (my wording).
>
> Relevant part of that thread there for reference:
>
>   http://www.postgresql.org/message-id/CANP8+jLtk1NtaJyXc=hAqX=0k+ku4zfavgVBKfs+_sOr9hepNQ@mail.gmail.com
>
> Simon included a short starter list of potentials which might be in
> that category:
>
>   * SQL compliant identifiers
>   * Remove RULEs
>   * Change recovery.conf
>   * Change block headers
>   * Retire template0, template1
>   * Optimise FSM
>   * Add heap metapage
>   * Alter tuple headers
>   et al
>
> This still is better placed on -hackers though, so lets have the
> conversation here to figure out if a "backwards compatibility breaking"
> release really is needed or not.

A couple of points here:
*) I don't think having a version number that starts with 10 instead
of 9 magically fixes backwards compatibility problems and I think
that's a dangerous precedent to set unless we're willing to fork
development and support version 9 indefinitely including major release
versions.

*) Compatibility issues at the SQL level have to be taken much more
seriously than other things (like internal layouts or .conf issues).

*) We need to do an honest cost benefit analysis before breaking
things.  Code refactors placed on your users puts an enormous cost
that is often underestimated.  I have some fairly specific examples of
the costs related to the text cast removal for example.  It's not a
pretty picture.

merlin

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

Justin Clift <justin@postgresql.org> — 2016-04-12T16:32:38Z

On 12 Apr 2016, at 17:23, Merlin Moncure <mmoncure@gmail.com> wrote:
> On Mon, Apr 11, 2016 at 11:39 AM, Justin Clift <justin@postgresql.org> wrote:
>> Moving over a conversation from the pgsql-advocacy mailing list.  In it
>> Simon (CC'd) raised the issue of potentially creating a backwards-compatibility
>> breaking release at some point in the future, to deal with things that
>> might have no other solution (my wording).
>> 
>> Relevant part of that thread there for reference:
>> 
>>  http://www.postgresql.org/message-id/CANP8+jLtk1NtaJyXc=hAqX=0k+ku4zfavgVBKfs+_sOr9hepNQ@mail.gmail.com
>> 
>> Simon included a short starter list of potentials which might be in
>> that category:
>> 
>>  * SQL compliant identifiers
>>  * Remove RULEs
>>  * Change recovery.conf
>>  * Change block headers
>>  * Retire template0, template1
>>  * Optimise FSM
>>  * Add heap metapage
>>  * Alter tuple headers
>>  et al
>> 
>> This still is better placed on -hackers though, so lets have the
>> conversation here to figure out if a "backwards compatibility breaking"
>> release really is needed or not.
> 
> A couple of points here:
> *) I don't think having a version number that starts with 10 instead
> of 9 magically fixes backwards compatibility problems and I think
> that's a dangerous precedent to set unless we're willing to fork
> development and support version 9 indefinitely including major release
> versions.
> 
> *) Compatibility issues at the SQL level have to be taken much more
> seriously than other things (like internal layouts or .conf issues).
> 
> *) We need to do an honest cost benefit analysis before breaking
> things.  Code refactors placed on your users puts an enormous cost
> that is often underestimated.  I have some fairly specific examples of
> the costs related to the text cast removal for example.  It's not a
> pretty picture.

Yeah.  Moving the discussion here was more to determine which items
really would need a backwards compatible break.  eg no other approach can
be found.

Seems I started it off badly, as no-one's yet jumped in to discuss the
initial points. :(

+ Justin

--
"My grandfather once told me that there are two kinds of people: those
who work and those who take the credit. He told me to try to be in the
first group; there was less competition there."
- Indira Gandhi

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

Robert Haas <robertmhaas@gmail.com> — 2016-04-12T17:43:41Z

On Tue, Apr 12, 2016 at 12:32 PM, Justin Clift <justin@postgresql.org> wrote:
> Yeah.  Moving the discussion here was more to determine which items
> really would need a backwards compatible break.  eg no other approach can
> be found.
>
> Seems I started it off badly, as no-one's yet jumped in to discuss the
> initial points. :(

I'm going to throw down the gauntlet (again) and say more or less what
I previously said on the pgsql-advocacy thread.  I think that:

1. Large backward compatibility breaks are bad.  Therefore, if any of
these things are absolutely impossible to do without major
compatibility breaks, we shouldn't do them at all.

2. Small backward compatibility breaks are OK, but don't require doing
anything special to the version number.

3. There's no value in aggregating many small backward compatibility
breaks into a single release.  That increases pain for users, rather
than decreasing it, and slows down development, too, because you have
to wait for the special magic release where it's OK to hose users.  We
typically have a few small backward compatibility breaks in each
release, and that's working fine, so I see little reason to change it.

4. To the extent that I can guess what the things on Simon's list
means from what he wrote, and that's a little difficult because his
descriptions were very short, I think that everything on that list is
either (a) a bad idea or (b) something that we can do without any
compatibility break at all.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

Josh Berkus <josh@agliodbs.com> — 2016-04-12T18:25:21Z

On 04/12/2016 10:43 AM, Robert Haas wrote:
> 1. Large backward compatibility breaks are bad.  Therefore, if any of
> these things are absolutely impossible to do without major
> compatibility breaks, we shouldn't do them at all.

+1

> 2. Small backward compatibility breaks are OK, but don't require doing
> anything special to the version number.

+1

> 3. There's no value in aggregating many small backward compatibility
> breaks into a single release.  That increases pain for users, rather
> than decreasing it, and slows down development, too, because you have
> to wait for the special magic release where it's OK to hose users.  We
> typically have a few small backward compatibility breaks in each
> release, and that's working fine, so I see little reason to change it.

+1

> 4. To the extent that I can guess what the things on Simon's list
> means from what he wrote, and that's a little difficult because his
> descriptions were very short, I think that everything on that list is
> either (a) a bad idea or (b) something that we can do without any
> compatibility break at all.

+1

Here's the features I can imagine being worth major backwards
compatibility breaks:

1. Fully pluggable storage with a clean API.

2. Total elimination of VACUUM or XID freezing

3. Fully transparent-to-the user MM replication/clustering or sharding.

4. Perfect partitioning (i.e. transparent to the user, supports keys &
joins, supports expressions on partition key, etc.)

5. Transparent upgrade-in-place (i.e. allowing 10.2 to use 10.1's tables
without pg_upgrade or other modification).

6. Fully pluggable parser/executor with a good API

That's pretty much it.  I can't imagine anything else which would
justify imposing a huge upgrade barrier on users.  And, I'll point out,
that in the above list:

* nobody is currently working on anything in core except #4.

* we don't *know* that any of the above items will require a backwards
compatibility break.

People keep talking about "we might want to break compatibility/file
format one day".  But nobody is working on anything which will and
justifies it.

-- 
--
Josh Berkus
Red Hat OSAS
(any opinions are my own)

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

Andres Freund <andres@anarazel.de> — 2016-04-12T18:27:44Z

On 2016-04-12 11:25:21 -0700, Josh berkus wrote:
> Here's the features I can imagine being worth major backwards
> compatibility breaks:
> 
> 1. Fully pluggable storage with a clean API.
> 
> 2. Total elimination of VACUUM or XID freezing
> 
> 3. Fully transparent-to-the user MM replication/clustering or sharding.
> 
> 4. Perfect partitioning (i.e. transparent to the user, supports keys &
> joins, supports expressions on partition key, etc.)
> 
> 5. Transparent upgrade-in-place (i.e. allowing 10.2 to use 10.1's tables
> without pg_upgrade or other modification).
> 
> 6. Fully pluggable parser/executor with a good API
> 
> That's pretty much it.  I can't imagine anything else which would
> justify imposing a huge upgrade barrier on users.  And, I'll point out,
> that in the above list:
> 
> * nobody is currently working on anything in core except #4.
> 
> * we don't *know* that any of the above items will require a backwards
> compatibility break.

none but 2) seem likely to require a substantial compatibility break.

Greetings,

Andres Freund

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

Robert Haas <robertmhaas@gmail.com> — 2016-04-12T18:44:34Z

On Tue, Apr 12, 2016 at 2:27 PM, Andres Freund <andres@anarazel.de> wrote:
> none but 2) seem likely to require a substantial compatibility break.

And even that doesn't require one, if you keep the only system around
and make the new system optional via some sort of pluggable storage
API.  Which, to me, seems like the only sensible approach from a
development perspective.  If you decide to rip out the entire heapam
and replace it with something new in one fell swoop, you might as well
bother not writing the patch.  It's got the same chance of being
accepted either way.

I really think the time has come that we need an API for the heap the
same way we already have for indexes.  Regardless of exactly how we
choose to implement that, I think a large part of it will end up
looking similar to what we already have for FDWs.  We can either use
the FDW API itself and add whatever additional methods we need for
this purpose, or copy it to a new file, rename everything, and have
two slightly different versions.  AFAICS, the things we need that the
FDW API doesn't currently provide are:

1. The ability to have a local relfilenode associated with the data.
Or, ideally, several, so you have a separate set of files for each
index.

2. The ability to WAL-log changes to that relfilenode (or those
relfilenodes) in a sensible way.  Not sure whether the new generic
XLOG stuff is good enough for a first go-round here or if more is
needed.

3. The ability to intercept DDL commands directed at the table and
handle them in some arbitrary way.  This is really optional; people
could always provide a function-based API until we devise something
better.

4. The ability to build standard PostgreSQL indexes on top of the
data, if the underlying format still has a useful notion of CTIDs.
That is, if the underlying format is basically like our heap format,
but optimized in some way - e.g. append-only table that can't update
or delete with a smaller tuple header and page compression - then it
can reuse our indexing.  If it does something else, like an
index-organized table where rows can move around to different physical
positions, then it has to provide its own indexing facilities.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

Oleg Bartunov <obartunov@gmail.com> — 2016-04-12T20:07:04Z

On Tue, Apr 12, 2016 at 9:25 PM, Josh berkus <josh@agliodbs.com> wrote:

> On 04/12/2016 10:43 AM, Robert Haas wrote:
> > 1. Large backward compatibility breaks are bad.  Therefore, if any of
> > these things are absolutely impossible to do without major
> > compatibility breaks, we shouldn't do them at all.
>
> +1
>
> > 2. Small backward compatibility breaks are OK, but don't require doing
> > anything special to the version number.
>
> +1
>
> > 3. There's no value in aggregating many small backward compatibility
> > breaks into a single release.  That increases pain for users, rather
> > than decreasing it, and slows down development, too, because you have
> > to wait for the special magic release where it's OK to hose users.  We
> > typically have a few small backward compatibility breaks in each
> > release, and that's working fine, so I see little reason to change it.
>
> +1
>
> > 4. To the extent that I can guess what the things on Simon's list
> > means from what he wrote, and that's a little difficult because his
> > descriptions were very short, I think that everything on that list is
> > either (a) a bad idea or (b) something that we can do without any
> > compatibility break at all.
>
> +1
>
> Here's the features I can imagine being worth major backwards
> compatibility breaks:
>
> 1. Fully pluggable storage with a clean API.
>
> 2. Total elimination of VACUUM or XID freezing
>
> 3. Fully transparent-to-the user MM replication/clustering or sharding.
>
> 4. Perfect partitioning (i.e. transparent to the user, supports keys &
> joins, supports expressions on partition key, etc.)
>
> 5. Transparent upgrade-in-place (i.e. allowing 10.2 to use 10.1's tables
> without pg_upgrade or other modification).
>
> 6. Fully pluggable parser/executor with a good API
>
> That's pretty much it.  I can't imagine anything else which would
> justify imposing a huge upgrade barrier on users.  And, I'll point out,
> that in the above list:
>
> * nobody is currently working on anything in core except #4.
>

We are working on #3 (HA multimaster).


>
> * we don't *know* that any of the above items will require a backwards
> compatibility break.
>
> People keep talking about "we might want to break compatibility/file
> format one day".  But nobody is working on anything which will and
> justifies it.
>

Our roadmap http://www.postgresql.org/developer/roadmap/ is the problem. We
don't have clear roadmap and that's why we cannot plan future feature full
release. There are several postgres-centric companies, which have most of
developers, who do all major contributions. All these companies has their
roadmaps, but not the community. I think 9.6 release is inflection point,
where we should combine our roadmaps and release the one for the community.
Than we could plan releases and our customers will see what to expect. I
can't say for other companies, but we have big demand for many features
from russian customers and we have to compete with other databases. Having
community roadmap will helps us to work with customers and plan our
resources.


>
> --
> --
> Josh Berkus
> Red Hat OSAS
> (any opinions are my own)
>
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers
>

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

Josh Berkus <josh@agliodbs.com> — 2016-04-12T20:28:52Z

On 04/12/2016 01:07 PM, Oleg Bartunov wrote:
> 
> Our roadmap http://www.postgresql.org/developer/roadmap/ is the problem.
> We don't have clear roadmap and that's why we cannot plan future feature
> full release. 

As someone who's worked at multiple proprietary software companies,
having a roadmap doesn't magically make code happen.

> There are several postgres-centric companies, which have
> most of developers, who do all major contributions. All these companies
> has their roadmaps, but not the community. I think 9.6 release is
> inflection point, where we should combine our roadmaps and release the
> one for the community. Than we could plan releases and our customers
> will see what to expect. I can't say for other companies, but we have
> big demand for many features from russian customers and we have to
> compete with other databases. Having community roadmap will helps us to
> work with customers and plan our resources.

It would be good to have a place for the companies who do PostgreSQL
feature work would publish their current efforts and timelines, so we at
least have a go-to place for "here's what someone's working on".  But
only if that information is going to be *updated*, something we're very
bad at.  And IMHO, a "roadmap" which is less that 50% accurate is a
waste of time.

There's an easy way for you to kick this off though: have PostgresPro
publish a wiki page or Trello board or github repo or whatever with your
roadmap and invite other full-time PostgreSQL contributors to add their
pieces.

-- 
--
Josh Berkus
Red Hat OSAS
(any opinions are my own)

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

David G. Johnston <david.g.johnston@gmail.com> — 2016-04-12T20:32:04Z

On Tue, Apr 12, 2016 at 1:07 PM, Oleg Bartunov <obartunov@gmail.com> wrote:

>
>
>> * we don't *know* that any of the above items will require a backwards
>> compatibility break.
>>
>> People keep talking about "we might want to break compatibility/file
>> format one day".  But nobody is working on anything which will and
>> justifies it.
>>
>
> Our roadmap http://www.postgresql.org/developer/roadmap/ is the problem.
> We don't have clear roadmap and that's why we cannot plan future feature
> full release. There are several postgres-centric companies, which have most
> of developers, who do all major contributions. All these companies has
> their roadmaps, but not the community. I think 9.6 release is inflection
> point, where we should combine our roadmaps and release the one for the
> community. Than we could plan releases and our customers will see what to
> expect. I can't say for other companies, but we have big demand for many
> features from russian customers and we have to compete with other
> databases. Having community roadmap will helps us to work with customers
> and plan our resources.
>

I've already posited just having our release numbers operate on 5-year
increments 10.0 - 10.4; 11.0 - 11.4, etc on advocacy but was met with
silence.  In any case this comment is just furtherance of the tail wagging
the dog.  I see no fundamental reason to have to plan something momentous
enough, and actively schedule work to meet the plan, in order to justify a
10.0 release.

There is a bunch of hand-waving here, and its an environment I'm not
immersed in, but it seems that creating a roadmap today is tantamount to
waterfall design - useful in moderation but has largely shown to be
undesirable at scale.  Aside from the 1-year release cycle the project is
reasonably agile and well receptive to outside observation, questions, and
contributions.  If you have spare resources you need to keep busy just ask
how you can help.  To be honest the community would likely rather have
those people help review and test everything that is presently in-progress
- future goals on a roadmap are not nearly as important.  And if you have a
demand you think needs to be fulfill put the information out there and get
input.

If you are claiming the balance between community and profit is skewed
undesirably you will need to put forth a more concrete argument in order to
convince me.  For me, having plans reside in the profit-motive parts of the
community and having core simply operate openly seems to strike a solid
balance.

I give a solid +10 to Robert's opinions on the matter and aside from
figuring out if and how to fit first-number versioning dynamics into our
release policies I think the community is doing a sufficient job on the
communication and planning front.  The biggest resource need is quality
control.  I dislike the fact that we are currently in a situation where the
first 3 point releases each year are considered "live betas" based
especially on both 9.3 and 9.5 post-release significant bug counts.

David J.

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

Robert Haas <robertmhaas@gmail.com> — 2016-04-12T20:58:30Z

On Tue, Apr 12, 2016 at 4:07 PM, Oleg Bartunov <obartunov@gmail.com> wrote:
> Our roadmap http://www.postgresql.org/developer/roadmap/ is the problem. We
> don't have clear roadmap and that's why we cannot plan future feature full
> release. There are several postgres-centric companies, which have most of
> developers, who do all major contributions. All these companies has their
> roadmaps, but not the community. I think 9.6 release is inflection point,
> where we should combine our roadmaps and release the one for the community.
> Than we could plan releases and our customers will see what to expect. I
> can't say for other companies, but we have big demand for many features from
> russian customers and we have to compete with other databases. Having
> community roadmap will helps us to work with customers and plan our
> resources.

I don't think it's realistic to plan what is going to go into a
certain release.  We don't know whether we want the patch until we see
the patch and review it and decide whether it's good.  We can't make
the decision, first, that the patch will be in the release, and then,
second, write the patch.  What we *can* do, as Josh says, is discuss
our plans.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

Robert Haas <robertmhaas@gmail.com> — 2016-04-12T21:04:57Z

On Tue, Apr 12, 2016 at 4:32 PM, David G. Johnston
<david.g.johnston@gmail.com> wrote:
> I give a solid +10 to Robert's opinions on the matter and aside from
> figuring out if and how to fit first-number versioning dynamics into our
> release policies I think the community is doing a sufficient job on the
> communication and planning front.  The biggest resource need is quality
> control.  I dislike the fact that we are currently in a situation where the
> first 3 point releases each year are considered "live betas" based
> especially on both 9.3 and 9.5 post-release significant bug counts.

/me blinks.

I find it shocking that you would compare 9.5 to 9.3 that way.  Yeah,
we've had a few bugs in 9.5: in particular, it was disappointing to
have to disable abbreviated keys.  But I'm not sure that I really
believe that affected massive numbers of users in a really negative
way - many locales were just fine, and not every locale that had a
problem with some string data necessarily had a problem with the
strings people were actually storing.  But we haven't eaten anybody's
data, at least not beyond what can be fixed by a REINDEX, unless I
missed something here.

The fact is that this is a fairly hard problem to solve.  Some bugs
are not going to get found before people try the software, and we
can't make them try it while it's in beta.  We can only do our best to
do good code review, but inevitably we will miss some things.

As for your proposal that we blindly consider $(N+1).0 to follow $N.4,
I'm not particularly enthralled with that.  I think it's a good idea
to look for a release that's got some particularly nifty feature(s)
and use that as the time to move the first digit.  And, sure, plan to
have that happen every 4-6 years or so, but adjust based on what
actually gets into which releases.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

David G. Johnston <david.g.johnston@gmail.com> — 2016-04-12T21:35:27Z

On Tue, Apr 12, 2016 at 2:04 PM, Robert Haas <robertmhaas@gmail.com> wrote:

> On Tue, Apr 12, 2016 at 4:32 PM, David G. Johnston
> <david.g.johnston@gmail.com> wrote:
> > I give a solid +10 to Robert's opinions on the matter and aside from
> > figuring out if and how to fit first-number versioning dynamics into our
> > release policies I think the community is doing a sufficient job on the
> > communication and planning front.  The biggest resource need is quality
> > control.  I dislike the fact that we are currently in a situation where
> the
> > first 3 point releases each year are considered "live betas" based
> > especially on both 9.3 and 9.5 post-release significant bug counts.
>
> /me blinks.
>
> I find it shocking that you would compare 9.5 to 9.3 that way.  Yeah,
> we've had a few bugs in 9.5: in particular, it was disappointing to
> have to disable abbreviated keys.  But I'm not sure that I really
> believe that affected massive numbers of users in a really negative
> way - many locales were just fine, and not every locale that had a
> problem with some string data necessarily had a problem with the
> strings people were actually storing.  But we haven't eaten anybody's
> data, at least not beyond what can be fixed by a REINDEX, unless I
> missed something here.
>

I probably over-implied my feelings regarding 9.5 since, yes, abbreviated
keys was largely out of realm of reasonable expectation.


I think I am colored by my involvement in attempting to help research this
one:

9.5.1 "Fix an oversight that caused hash joins to miss joining to some
tuples of the inner relation in rare cases (Tomas Vondra, Tom Lane)"

This one struck me as well for some reason...

9.5.1 "Fix overeager pushdown of HAVING clauses when grouping sets are
used."

The two ROW() comparison fixes for 9.5.2

I'm not exactly someone looking to poke a stick in PostgreSQL's side so
regardless of degree of "validity" of my feelings that fact that I have
them is likely informative.  Or I'm just in a particularly overly-sensitive
mood right now - I wouldn't fully discount the possibility.


> The fact is that this is a fairly hard problem to solve.  Some bugs
> are not going to get found before people try the software, and we
> can't make them try it while it's in beta.  We can only do our best to
> do good code review, but inevitably we will miss some things.
>

Agreed, and to the point of using corporate resources for improvement of
existing work as opposed to roadmap stuff.


> As for your proposal that we blindly consider $(N+1).0 to follow $N.4,
> I'm not particularly enthralled with that.  I think it's a good idea
> to look for a release that's got some particularly nifty feature(s)
> and use that as the time to move the first digit.  And, sure, plan to
> have that happen every 4-6 years or so, but adjust based on what
> actually gets into which releases.
>
>
The main point on that post was we emphasize not only the new stuff in the
just released version but that we re-celebrate everything that has been
accomplished in the previous 4 releases as well.  If the first-digit is
getting such significant attention we might as well play to that fact and
try and remind people who've missed out on prior releases that we are
continually innovating.  That, and the fact that the last N.0 release that
was so highly touted just went out of support.

Otherwise I don't see why we don't just start increment the first-digit
yearly.  Sure, every so often we think we've done enough to warrant an
increase there but the philosophy as a community doesn't actually match our
particular choice - we don't, in advance, place any special importance on
the first-digit releases, as evidenced by this discussion.

David J.

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

Bruce Momjian <bruce@momjian.us> — 2016-04-29T15:30:43Z

On Tue, Apr 12, 2016 at 01:43:41PM -0400, Robert Haas wrote:
> I'm going to throw down the gauntlet (again) and say more or less what
> I previously said on the pgsql-advocacy thread.  I think that:
> 
> 1. Large backward compatibility breaks are bad.  Therefore, if any of
> these things are absolutely impossible to do without major
> compatibility breaks, we shouldn't do them at all.
> 
> 2. Small backward compatibility breaks are OK, but don't require doing
> anything special to the version number.
> 
> 3. There's no value in aggregating many small backward compatibility
> breaks into a single release.  That increases pain for users, rather
> than decreasing it, and slows down development, too, because you have
> to wait for the special magic release where it's OK to hose users.  We
> typically have a few small backward compatibility breaks in each
> release, and that's working fine, so I see little reason to change it.

Well, this is true for SQL-level and admin-level changes, but it does
make sense to group pg_upgrade breaks into a single release.  I think
the plan is for us to have logical replication usable before we make
such a change.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+                     Ancient Roman grave inscription +

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

Bruce Momjian <bruce@momjian.us> — 2016-04-29T15:32:41Z

On Tue, Apr 12, 2016 at 11:25:21AM -0700, Josh Berkus wrote:
> Here's the features I can imagine being worth major backwards
> compatibility breaks:
...
> 5. Transparent upgrade-in-place (i.e. allowing 10.2 to use 10.1's tables
> without pg_upgrade or other modification).

Technically, this is exactly what pg_upgrade does.  I think what you
really mean is for the backend binary to be able to read the system
tables and WAL files of the old clusters --- something I can't see us
implementing anytime soon.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+                     Ancient Roman grave inscription +

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

Joshua D. Drake <jd@commandprompt.com> — 2016-04-29T15:37:57Z

On 04/29/2016 08:32 AM, Bruce Momjian wrote:
> On Tue, Apr 12, 2016 at 11:25:21AM -0700, Josh Berkus wrote:
>> Here's the features I can imagine being worth major backwards
>> compatibility breaks:
> ...
>> 5. Transparent upgrade-in-place (i.e. allowing 10.2 to use 10.1's tables
>> without pg_upgrade or other modification).
>
> Technically, this is exactly what pg_upgrade does.  I think what you
> really mean is for the backend binary to be able to read the system
> tables and WAL files of the old clusters --- something I can't see us
> implementing anytime soon.
>

For the most part, pg_upgrade is good enough. There are exceptions and 
it does need a more thorough test suite but as a whole, it works. As 
nice as being able to install 9.6 right on top of 9.5 and have 9.6 
magically work, it is certainly not a *requirement* anymore.

Sincerely,

JD

-- 
Command Prompt, Inc.                  http://the.postgres.company/
                         +1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Everyone appreciates your honesty, until you are honest with them.

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

Bruce Momjian <bruce@momjian.us> — 2016-04-29T15:44:03Z

On Tue, Apr 12, 2016 at 11:07:04PM +0300, Oleg Bartunov wrote:
> Our roadmap http://www.postgresql.org/developer/roadmap/ is the problem. We
> don't have clear roadmap and that's why we cannot plan future feature full
> release. There are several postgres-centric companies, which have most of
> developers, who do all major contributions. All these companies has their
> roadmaps, but not the community.

I would be concerned if company roadmaps overtly affected the community
roadmap.  In general, I find company roadmaps to be very short-sighted
and quickly changed based on the demands of specific users/customers ---
something we don't want to imitate.

We do want company roadmaps to affect the community roadmap, but in a
healthy, long-term way, and I think, in general, that is happening.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+                     Ancient Roman grave inscription +

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

Bruce Momjian <bruce@momjian.us> — 2016-04-29T15:45:26Z

On Fri, Apr 29, 2016 at 08:37:57AM -0700, Joshua Drake wrote:
> >Technically, this is exactly what pg_upgrade does.  I think what you
> >really mean is for the backend binary to be able to read the system
> >tables and WAL files of the old clusters --- something I can't see us
> >implementing anytime soon.
> >
> 
> For the most part, pg_upgrade is good enough. There are exceptions and it
> does need a more thorough test suite but as a whole, it works. As nice as
> being able to install 9.6 right on top of 9.5 and have 9.6 magically work,
> it is certainly not a *requirement* anymore.

Yes, the trick would be making the new 9.6 features work with the
existing 9.5 system tables that don't know about the 9.6 features.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+                     Ancient Roman grave inscription +

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

Simon Riggs <simon@2ndquadrant.com> — 2016-04-29T16:02:38Z

On 12 April 2016 at 20:25, Josh berkus <josh@agliodbs.com> wrote:


> Here's the features I can imagine being worth major backwards
> compatibility breaks:
>
> 1. Fully pluggable storage with a clean API.
>
> 2. Total elimination of VACUUM or XID freezing
>
> 3. Fully transparent-to-the user MM replication/clustering or sharding.
>
> 4. Perfect partitioning (i.e. transparent to the user, supports keys &
> joins, supports expressions on partition key, etc.)
>
> 5. Transparent upgrade-in-place (i.e. allowing 10.2 to use 10.1's tables
> without pg_upgrade or other modification).
>
> 6. Fully pluggable parser/executor with a good API
>
> That's pretty much it.  I can't imagine anything else which would
> justify imposing a huge upgrade barrier on users.  And, I'll point out,
> that in the above list:
>
> * nobody is currently working on anything in core except #4.
>
> * we don't *know* that any of the above items will require a backwards
> compatibility break.
>
> People keep talking about "we might want to break compatibility/file
> format one day".  But nobody is working on anything which will and
> justifies it.
>

Of your list, I know 2ndQuadrant developers are working on 1, 3, 5.
6 has being discussed recently on list by other hackers.

I'm not really sure what 2 consists of; presumably this means "take the
pain away" rather than removal of MVCC, which is the root cause of those
secondary effects.

I don't think the current focus on manually intensive DDL partitioning is
the right way forwards. I did once; I don't now.

-- 
Simon Riggs                http://www.2ndQuadrant.com/
<http://www.2ndquadrant.com/>
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

Merlin Moncure <mmoncure@gmail.com> — 2016-04-29T16:17:39Z

On Fri, Apr 29, 2016 at 11:02 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
> On 12 April 2016 at 20:25, Josh berkus <josh@agliodbs.com> wrote:
>
>>
>> Here's the features I can imagine being worth major backwards
>> compatibility breaks:
>>
>> 1. Fully pluggable storage with a clean API.
>>
>> 2. Total elimination of VACUUM or XID freezing
>>
>> 3. Fully transparent-to-the user MM replication/clustering or sharding.
>>
>> 4. Perfect partitioning (i.e. transparent to the user, supports keys &
>> joins, supports expressions on partition key, etc.)
>>
>> 5. Transparent upgrade-in-place (i.e. allowing 10.2 to use 10.1's tables
>> without pg_upgrade or other modification).
>>
>> 6. Fully pluggable parser/executor with a good API
>>
>> That's pretty much it.  I can't imagine anything else which would
>> justify imposing a huge upgrade barrier on users.  And, I'll point out,
>> that in the above list:
>>
>> * nobody is currently working on anything in core except #4.
>>
>> * we don't *know* that any of the above items will require a backwards
>> compatibility break.
>>
>> People keep talking about "we might want to break compatibility/file
>> format one day".  But nobody is working on anything which will and
>> justifies it.
>
> Of your list, I know 2ndQuadrant developers are working on 1, 3, 5.
> 6 has being discussed recently on list by other hackers.

#5 (upgrade without pg_upgrade or dump/restore) from my perspective
would be the most useful feature of all time, and would justify the
9.x to 10.x all by itself using the existing standard (I think?) of
major project milestones to advance that number.

7.x ?? (just coming on the scene then)
8.x windows, pitr
9.x replication
10.x easy upgrades

merlin

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

Joshua D. Drake <jd@commandprompt.com> — 2016-04-29T16:40:35Z

On 04/29/2016 08:44 AM, Bruce Momjian wrote:
> On Tue, Apr 12, 2016 at 11:07:04PM +0300, Oleg Bartunov wrote:
>> Our roadmap http://www.postgresql.org/developer/roadmap/ is the problem. We
>> don't have clear roadmap and that's why we cannot plan future feature full
>> release. There are several postgres-centric companies, which have most of
>> developers, who do all major contributions. All these companies has their
>> roadmaps, but not the community.
>
> I would be concerned if company roadmaps overtly affected the community
> roadmap.  In general, I find company roadmaps to be very short-sighted
> and quickly changed based on the demands of specific users/customers ---
> something we don't want to imitate.
>
> We do want company roadmaps to affect the community roadmap, but in a
> healthy, long-term way, and I think, in general, that is happening.
>

The roadmap is not the problem it is the lack of cooperation. Many 
companies are now developing features in a silo and then presenting them 
to the community. Instead we should be working with those companies to 
have them develop transparently so others can be a part of the process.

If the feature is going to be submitted to core anyway (or open source) 
why wouldn't we just do that? Why wouldn't EDB develop directly within 
the Pg infrastructure. Why wouldn't we build teams around the best and 
brightest between EDB, 2Q and Citus?

Egos.

Consider PgLogical, who is working on this outside of 2Q? Where is the 
git repo for it? Where is the bug tracker? Where is the mailing list? 
Oh, its -hackers, except that it isn't, is it?

It used to be that everyone got together and worked together before the 
patch review process. Now it seems like it is a competition between 
companies to see whose ego can get the most inflated via press releases 
because they developed X for Y.

If the companies were to come together and truly recognize that profit 
is the reward not the goal then our community would be much stronger for it.

Sincerely,

JD

-- 
Command Prompt, Inc.                  http://the.postgres.company/
                         +1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Everyone appreciates your honesty, until you are honest with them.

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

Joshua D. Drake <jd@commandprompt.com> — 2016-04-29T17:23:21Z

On 04/29/2016 09:40 AM, Joshua D. Drake wrote:
> On 04/29/2016 08:44 AM, Bruce Momjian wrote:

> Consider PgLogical, who is working on this outside of 2Q? Where is the
> git repo for it? Where is the bug tracker? Where is the mailing list?
> Oh, its -hackers, except that it isn't, is it?
>

FTR: I am not attacking any one entity here. It is just that PgLogical 
is a good example of my point. I certainly recognize and have publicly 
applauded on multiple occasions the good work 2Q is doing. Just as I 
have with EDB, Citus and Crunchy.

Sincerely,

JD

-- 
Command Prompt, Inc.                  http://the.postgres.company/
                         +1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Everyone appreciates your honesty, until you are honest with them.

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

Simon Riggs <simon@2ndquadrant.com> — 2016-04-29T18:36:54Z

On 29 April 2016 at 18:40, Joshua D. Drake <jd@commandprompt.com> wrote:

> On 04/29/2016 08:44 AM, Bruce Momjian wrote:
>
>> On Tue, Apr 12, 2016 at 11:07:04PM +0300, Oleg Bartunov wrote:
>>
>>> Our roadmap http://www.postgresql.org/developer/roadmap/ is the
>>> problem. We
>>> don't have clear roadmap and that's why we cannot plan future feature
>>> full
>>> release. There are several postgres-centric companies, which have most of
>>> developers, who do all major contributions. All these companies has their
>>> roadmaps, but not the community.
>>>
>>
>> I would be concerned if company roadmaps overtly affected the community
>> roadmap.  In general, I find company roadmaps to be very short-sighted
>> and quickly changed based on the demands of specific users/customers ---
>> something we don't want to imitate.
>>
>> We do want company roadmaps to affect the community roadmap, but in a
>> healthy, long-term way, and I think, in general, that is happening.
>>
>>
> The roadmap is not the problem it is the lack of cooperation. Many
> companies are now developing features in a silo and then presenting them to
> the community. Instead we should be working with those companies to have
> them develop transparently so others can be a part of the process.
>
> If the feature is going to be submitted to core anyway (or open source)
> why wouldn't we just do that? Why wouldn't EDB develop directly within the
> Pg infrastructure. Why wouldn't we build teams around the best and
> brightest between EDB, 2Q and Citus?
>
> Egos.
>
> Consider PgLogical, who is working on this outside of 2Q?


Thank you for volunteering to assist. What would you like to work on?

-- 
Simon Riggs                http://www.2ndQuadrant.com/
<http://www.2ndquadrant.com/>
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

Joshua D. Drake <jd@commandprompt.com> — 2016-04-29T18:50:02Z

On 04/29/2016 11:36 AM, Simon Riggs wrote:

>     Egos.
>
>     Consider PgLogical, who is working on this outside of 2Q?
>
>
> Thank you for volunteering to assist. What would you like to work on?

You are very welcome. I have been testing as you know. I would be happy 
to continue that and also was going to look into having a subscriber 
validate if it is connecting to a subscribed node or not which is the 
error I ran into.

I am also interested in creating user docs (versus reference docs).

Sincerely,

JD

-- 
Command Prompt, Inc.                  http://the.postgres.company/
                         +1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Everyone appreciates your honesty, until you are honest with them.

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

Joshua D. Drake <jd@commandprompt.com> — 2016-04-30T17:29:12Z

On 04/29/2016 11:50 AM, Joshua D. Drake wrote:
> On 04/29/2016 11:36 AM, Simon Riggs wrote:
>
>>     Egos.
>>
>>     Consider PgLogical, who is working on this outside of 2Q?
>>
>>
>> Thank you for volunteering to assist. What would you like to work on?
>
> You are very welcome. I have been testing as you know. I would be happy
> to continue that and also was going to look into having a subscriber
> validate if it is connecting to a subscribed node or not which is the
> error I ran into.
>
> I am also interested in creating user docs (versus reference docs).

So what do you think Simon? How about we put pgLogical under community 
infrastructure, use the postgresql redmine to track bugs, feature 
requests and documentation?

I guarantee resources from CMD and I bet we could get others to 
participate as well. Let's turn this into an awesome community driven 
extension.

Sincerely,

JD



-- 
Command Prompt, Inc.                  http://the.postgres.company/
                         +1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Everyone appreciates your honesty, until you are honest with them.

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 - > 10.0

Tomasz Rybak <tomasz.rybak@post.pl> — 2016-04-30T18:25:18Z

I cut many of emails from CC - AFAIR most of you are subscribed to 
pg-hackers.

Dnia 2016-04-30 19:29 Joshua D. Drake napisał(a):

>On 04/29/2016 11:50 AM, Joshua D. Drake wrote:
>> On 04/29/2016 11:36 AM, Simon Riggs wrote:
>>
>>>     Egos.
>>>
>>>     Consider PgLogical, who is working on this outside of 2Q?
>>>
>>>
>>> Thank you for volunteering to assist. What would you like to work on?
>>
>> You are very welcome. I have been testing as you know. I would be happy
>> to continue that and also was going to look into having a subscriber
>> validate if it is connecting to a subscribed node or not which is the
>> error I ran into.
>>
>> I am also interested in creating user docs (versus reference docs).
>
>So what do you think Simon? How about we put pgLogical under community 
>infrastructure, use the postgresql redmine to track bugs, feature 
>requests and documentation?
>
>I guarantee resources from CMD and I bet we could get others to 
>participate as well. Let's turn this into an awesome community driven 
>extension.
>

I reviewed pglogical-output extension in CF 2016-01. It was the only
patch I reviewed - it was quite big and as I was doing it afternoons
and not during work (other responsibilities) it took me more than half
of month. And while initially response was good, after final review there
was no response, no new patch version - also nothing in CF 2016-03.

At the same time I've seen that pglogical got some love in March - but
I'm not sure whether it is usable without *-output plugin. On the one
hand splitting those two makes review easier, or at least manageable.
OTOH one does not make much sense without the other. I can see that,
at least in theory, pglogical-output could be used independently, but
at the same time, its main user will be pglogical proper.

So, in summary - slightly better management and communication regarding
features and patches (not only this one; this is just the patch with
which I tried to do review) would be beneficial.
For now I'm not sure what is going on with pglogical,
and whether my review even mattered.

Best regards
Tomasz Rybak

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

Oleg Bartunov <obartunov@gmail.com> — 2016-04-30T21:07:11Z

On Fri, Apr 29, 2016 at 7:40 PM, Joshua D. Drake <jd@commandprompt.com>
wrote:

> On 04/29/2016 08:44 AM, Bruce Momjian wrote:
>
>> On Tue, Apr 12, 2016 at 11:07:04PM +0300, Oleg Bartunov wrote:
>>
>>> Our roadmap http://www.postgresql.org/developer/roadmap/ is the
>>> problem. We
>>> don't have clear roadmap and that's why we cannot plan future feature
>>> full
>>> release. There are several postgres-centric companies, which have most of
>>> developers, who do all major contributions. All these companies has their
>>> roadmaps, but not the community.
>>>
>>
>> I would be concerned if company roadmaps overtly affected the community
>> roadmap.  In general, I find company roadmaps to be very short-sighted
>> and quickly changed based on the demands of specific users/customers ---
>> something we don't want to imitate.
>>
>> We do want company roadmaps to affect the community roadmap, but in a
>> healthy, long-term way, and I think, in general, that is happening.
>>
>>
> The roadmap is not the problem it is the lack of cooperation. Many
> companies are now developing features in a silo and then presenting them to
> the community. Instead we should be working with those companies to have
> them develop transparently so others can be a part of the process.
>

We are working on our roadmap to have it in form to be presented to the
community. I think we'll publish it somewhere in wiki.


>
> If the feature is going to be submitted to core anyway (or open source)
> why wouldn't we just do that? Why wouldn't EDB develop directly within the
> Pg infrastructure. Why wouldn't we build teams around the best and
> brightest between EDB, 2Q and Citus?
>

This is what I suggested.  Features considered to be open source could be
discussed and developed together.



>
> Egos.
>
> Consider PgLogical, who is working on this outside of 2Q? Where is the git
> repo for it? Where is the bug tracker? Where is the mailing list? Oh, its
> -hackers, except that it isn't, is it?
>
> It used to be that everyone got together and worked together before the
> patch review process. Now it seems like it is a competition between
> companies to see whose ego can get the most inflated via press releases
> because they developed X for Y.
>
>
git log says better than any press releases :)


> If the companies were to come together and truly recognize that profit is
> the reward not the goal then our community would be much stronger for it.


I'd not limited by the companies, individual developes are highly welcome.
I'm afraid there are some.


>
>
> Sincerely,
>
> JD
>
>
> --
> Command Prompt, Inc.                  http://the.postgres.company/
>                         +1-503-667-4564
> PostgreSQL Centered full stack support, consulting and development.
> Everyone appreciates your honesty, until you are honest with them.
>

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

Joshua D. Drake <jd@commandprompt.com> — 2016-05-01T00:46:30Z

<p dir="ltr"><br>
On Apr 30, 2016 2:07 PM, Oleg Bartunov &lt;obartunov@gmail.com&gt; wrote:<br>
&gt;<br>
&gt;<br>
&gt;<br>
&gt; On Fri, Apr 29, 2016 at 7:40 PM, Joshua D. Drake &lt;jd@commandprompt.com&gt; wrote:<br>
&gt;&gt;<br>
&gt; I'd not limited by the companies, individual developes are highly welcome. I'm afraid there are some.<br>
&gt; &#160;</p>
<p dir="ltr">Oh, absolutely. I was just pointing out how a lot of companies are hoarding talent internally for no productive purpose.</p>
<p dir="ltr">Jd</p>
<p dir="ltr">&gt;&gt;<br>
&gt;&gt;<br>
&gt;&gt;<br>
&gt;&gt; Sincerely,<br>
&gt;&gt;<br>
&gt;&gt; JD<br>
&gt;&gt;<br>
&gt;&gt;<br>
&gt;&gt; -- <br>
&gt;&gt; Command Prompt, Inc.&#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; http://the.postgres.company/<br>
&gt;&gt; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; +1-503-667-4564<br>
&gt;&gt; PostgreSQL Centered full stack support, consulting and development.<br>
&gt;&gt; Everyone appreciates your honesty, until you are honest with them.<br>
&gt;<br>
&gt;<br>
</p>

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

Robert Haas <robertmhaas@gmail.com> — 2016-05-13T14:42:50Z

On Sat, Apr 30, 2016 at 8:46 PM, Joshua Drake <jd@commandprompt.com> wrote:
> Oh, absolutely. I was just pointing out how a lot of companies are hoarding
> talent internally for no productive purpose.

Wow, really?

I disagree both with the idea that this is happening and with your
characterization of it.  First, there are lots of people contributing
code to PostgreSQL right now.  To look at the just the last
CommitFest, we've got multiple people from all of Crunchy Data,
2ndQuadrant, EnterpriseDB,  Postgres Pro, and NTT; plus Julien Rouhaud
from Dalibo and Peter Geoghegan at Heroku and Michael Paquier at
VMware, among many others.  I'm not sure anyone at CommandPrompt
submitted a patch, though.  :-)

Second, when people don't contribute as much as you think they should
to the PostgreSQL community, I don't think that necessarily means
their employer is doing something wrong.  Sometimes, it may be the
employee's choice to spend more time on consulting or support or
whatever else they are doing; maybe that's what they like to do.  At
other times, it may be the thing that has to be done to pay the bills,
and I think that's legitimate, too.  People have a right to earning a
living, and companies have to make money to keep paying their
employees.

Let's respect people and companies for what they do contribute, rather
than labeling it as not good enough.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

Joshua D. Drake <jd@commandprompt.com> — 2016-05-13T16:12:23Z

On 05/13/2016 07:42 AM, Robert Haas wrote:
> On Sat, Apr 30, 2016 at 8:46 PM, Joshua Drake <jd@commandprompt.com> wrote:
>> Oh, absolutely. I was just pointing out how a lot of companies are hoarding
>> talent internally for no productive purpose.
>
> Wow, really?
>
> I disagree both with the idea that this is happening and with your
> characterization of it.  First, there are lots of people contributing
> code to PostgreSQL right now.  To look at the just the last
> CommitFest, we've got multiple people from all of Crunchy Data,
> 2ndQuadrant, EnterpriseDB,  Postgres Pro, and NTT; plus Julien Rouhaud
> from Dalibo and Peter Geoghegan at Heroku and Michael Paquier at
> VMware, among many others.

Singular point contribution is not the point of my argument. My point is 
that if three people from EDB and three people from Citus got together 
and worked on a project in full collaboration it would be more 
beneficial to the project.

> I'm not sure anyone at CommandPrompt
> submitted a patch, though.  :-)

This is true. We have been a little busy doing a lot of things for the 
community that -hackers are generally not good at.

>
> Second, when people don't contribute as much as you think they should
> to the PostgreSQL community, I don't think that necessarily means
> their employer is doing something wrong.  Sometimes, it may be the

This is also true.

> employee's choice to spend more time on consulting or support or
> whatever else they are doing; maybe that's what they like to do.  At
> other times, it may be the thing that has to be done to pay the bills,
> and I think that's legitimate, too.  People have a right to earning a
> living, and companies have to make money to keep paying their
> employees.

As someone in this community that is responsible for paying the bills 
including employee salaries, I fully agree with this.

>
> Let's respect people and companies for what they do contribute, rather
> than labeling it as not good enough.
>

There was no disrespect intended. I was trying to push forth an idea 
that multi-company team collaboration is better for the community than 
single company team collaboration. I will stand by that assertion.

Sincerely,

JD


-- 
Command Prompt, Inc.                  http://the.postgres.company/
                         +1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Everyone appreciates your honesty, until you are honest with them.

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

Bruce Momjian <bruce@momjian.us> — 2016-05-13T16:28:12Z

On Fri, May 13, 2016 at 09:12:23AM -0700, Joshua Drake wrote:
> There was no disrespect intended. I was trying to push forth an idea that
> multi-company team collaboration is better for the community than single
> company team collaboration. I will stand by that assertion.

Uh, we are already doing that.  EDB and NTT are working on FDWs and
sharding, PostgresPro and someone else is working on a transaction
manager, and EDB and 2nd Quadrant worked on parallelism.

What is the problem you are trying to solve?

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+                     Ancient Roman grave inscription +

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

Joshua D. Drake <jd@commandprompt.com> — 2016-05-13T16:35:40Z

On 05/13/2016 09:28 AM, Bruce Momjian wrote:
> On Fri, May 13, 2016 at 09:12:23AM -0700, Joshua Drake wrote:
>> There was no disrespect intended. I was trying to push forth an idea that
>> multi-company team collaboration is better for the community than single
>> company team collaboration. I will stand by that assertion.
>
> Uh, we are already doing that.  EDB and NTT are working on FDWs and
> sharding, PostgresPro and someone else is working on a transaction
> manager, and EDB and 2nd Quadrant worked on parallelism.
>
> What is the problem you are trying to solve?

Hey, if I am wrong that's awesome. The impression I have is the general 
workflow is this:

	* Company(1) discusses feature with community
	* Company(1) works on patch/feature for a period of time
	* Company(1) delivers patch to community
	* Standard operation continues (patch review, discussion, etc..)

This is not "bad" but it isn't as productive as something like this 
would be:

	* Company(1) + Company(2) get together and discuss using their 
respective resources to collaboratively develop X (multi-master for 
example).

	* Company(1) + Company(2) discuss feature with community
	* Company(1) + Company(2) work on patch/feature in the open, together
	* Company(1) + Company(2) deliver patch to community
	* Standard operation continues (patch review, discussion, etc...)

The difference being one of coopetition versions competition for the 
betterment of the community. If there are companies that are doing that 
already, that is awesome and I applaud it. I was just trying to further 
drive that idea home.

This on my end all sourced from this event:

https://www.commandprompt.com/blogs/joshua_drake/2016/04/you_are_my_fellow_community_member/

That event is childish and a poor representation of what our community 
stands for: Excellent, Correctness, Inclusion, Collaboration.

Sincerely,

JD



-- 
Command Prompt, Inc.                  http://the.postgres.company/
                         +1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Everyone appreciates your honesty, until you are honest with them.

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

Bruce Momjian <bruce@momjian.us> — 2016-05-13T16:40:23Z

On Fri, May 13, 2016 at 09:35:40AM -0700, Joshua Drake wrote:
> On 05/13/2016 09:28 AM, Bruce Momjian wrote:
> >On Fri, May 13, 2016 at 09:12:23AM -0700, Joshua Drake wrote:
> >>There was no disrespect intended. I was trying to push forth an idea that
> >>multi-company team collaboration is better for the community than single
> >>company team collaboration. I will stand by that assertion.
> >
> >Uh, we are already doing that.  EDB and NTT are working on FDWs and
> >sharding, PostgresPro and someone else is working on a transaction
> >manager, and EDB and 2nd Quadrant worked on parallelism.
> >
> >What is the problem you are trying to solve?
> 
> Hey, if I am wrong that's awesome. The impression I have is the general
> workflow is this:
> 
> 	* Company(1) discusses feature with community
> 	* Company(1) works on patch/feature for a period of time
> 	* Company(1) delivers patch to community
> 	* Standard operation continues (patch review, discussion, etc..)

Yes, there are some cases of that.  I assume it is due to efficiency and
the belief that others aren't interested in helping.  In a way is a
company working on something alone different from a person working on a
patch alone?

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+                     Ancient Roman grave inscription +

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

Joshua D. Drake <jd@commandprompt.com> — 2016-05-13T16:49:00Z

On 05/13/2016 09:40 AM, Bruce Momjian wrote:
> On Fri, May 13, 2016 at 09:35:40AM -0700, Joshua Drake wrote:
>> On 05/13/2016 09:28 AM, Bruce Momjian wrote:
>>> On Fri, May 13, 2016 at 09:12:23AM -0700, Joshua Drake wrote:
>>>> There was no disrespect intended. I was trying to push forth an idea that
>>>> multi-company team collaboration is better for the community than single
>>>> company team collaboration. I will stand by that assertion.
>>>
>>> Uh, we are already doing that.  EDB and NTT are working on FDWs and
>>> sharding, PostgresPro and someone else is working on a transaction
>>> manager, and EDB and 2nd Quadrant worked on parallelism.
>>>
>>> What is the problem you are trying to solve?
>>
>> Hey, if I am wrong that's awesome. The impression I have is the general
>> workflow is this:
>>
>> 	* Company(1) discusses feature with community
>> 	* Company(1) works on patch/feature for a period of time
>> 	* Company(1) delivers patch to community
>> 	* Standard operation continues (patch review, discussion, etc..)
>
> Yes, there are some cases of that.  I assume it is due to efficiency and
> the belief that others aren't interested in helping.  In a way is a
> company working on something alone different from a person working on a
> patch alone?

No but I also think we should discourage that when reasonable as well. 
Obviously some patches just don't need more than one person but when we 
are talking about anything that is taking X time (month or more?) then
we should actively encourage collaboration.

That is all I am really talking about here. A more assertive 
collaboration for the betterment of the community. When I think about 
the size of the brain trust we have as a whole, I imagine the great 
things we could do even better. It isn't magical or overnight but a long 
term goal.

Sincerely,

JD

-- 
Command Prompt, Inc.                  http://the.postgres.company/
                         +1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Everyone appreciates your honesty, until you are honest with them.

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

Robert Haas <robertmhaas@gmail.com> — 2016-05-13T18:48:42Z

On Fri, May 13, 2016 at 12:12 PM, Joshua D. Drake <jd@commandprompt.com> wrote:
> Singular point contribution is not the point of my argument. My point is
> that if three people from EDB and three people from Citus got together and
> worked on a project in full collaboration it would be more beneficial to the
> project.

Well, the scalability work in 9.6 went almost exactly like this,
assuming you count Andres as three people (which is entirely
reasonable) and Dilip, Mithun, Amit, and myself as three people (which
is maybe less reasonable, since I don't really want any of us counted
as less than a whole person).

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

Josh Berkus <josh@agliodbs.com> — 2016-05-13T19:03:14Z

On 05/13/2016 11:48 AM, Robert Haas wrote:
> On Fri, May 13, 2016 at 12:12 PM, Joshua D. Drake <jd@commandprompt.com> wrote:
>> Singular point contribution is not the point of my argument. My point is
>> that if three people from EDB and three people from Citus got together and
>> worked on a project in full collaboration it would be more beneficial to the
>> project.
> 
> Well, the scalability work in 9.6 went almost exactly like this,
> assuming you count Andres as three people (which is entirely
> reasonable) and Dilip, Mithun, Amit, and myself as three people (which
> is maybe less reasonable, since I don't really want any of us counted
> as less than a whole person).

Frankly, PostgreSQL is practically a wonderland of inter-company
collaboration.  Yeah, there's some "does not play nice with others"
which happens, but that's pretty much inevitable.

Plus, it's also useful to have some companies go in different directions
sometimes; the best approach to certain problems isn't always well
defined.  We might have a little more of that than is completely ideal,
but it's rather hard to determine that.

Anyway, all of this is a moot point, because nobody has the power to
tell the various companies what to do.  We're just lucky that everyone
is still committed to writing stuff which adds to PostgreSQL.

-- 
--
Josh Berkus
Red Hat OSAS
(any opinions are my own)

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

Robert Haas <robertmhaas@gmail.com> — 2016-05-13T19:05:26Z

On Fri, May 13, 2016 at 12:35 PM, Joshua D. Drake <jd@commandprompt.com> wrote:
> Hey, if I am wrong that's awesome. The impression I have is the general
> workflow is this:
>
>         * Company(1) discusses feature with community
>         * Company(1) works on patch/feature for a period of time
>         * Company(1) delivers patch to community
>         * Standard operation continues (patch review, discussion, etc..)
>
> This is not "bad" but it isn't as productive as something like this would
> be:
>
>         * Company(1) + Company(2) get together and discuss using their
> respective resources to collaboratively develop X (multi-master for
> example).
>
>         * Company(1) + Company(2) discuss feature with community
>         * Company(1) + Company(2) work on patch/feature in the open,
> together
>         * Company(1) + Company(2) deliver patch to community
>         * Standard operation continues (patch review, discussion, etc...)
>
> The difference being one of coopetition versions competition for the
> betterment of the community. If there are companies that are doing that
> already, that is awesome and I applaud it. I was just trying to further
> drive that idea home.

I think that's already happening.  I'm happy to see more of it.  In
practical terms, though, it's harder to collaborate between companies
because then you need two management teams to be on-board with it, and
there can be other competing priorities.  If either company needs to
pull staff of a project because of some competing priority (say,
fixing a broken customer or addressing an urgent customer need), then
the whole project can stall.  The whole wagon train moves at the pace
of the slowest camel.  It's nice when we can collaborate across
companies and I'm all for it, but sometimes it's faster to for a
single company to just assign a couple of people to a project and tell
them to go do it.

Now, where this gets tricky is when it comes down to whether the
end-product of that effort is something the community wants.  We all
need to be careful not to make our corporate priorities into community
priorities.  Features shouldn't get committed without a consensus that
they are both useful and well-implemented, and prior discussion is a
good way to achieve that.  On the whole, I think we've done reasonably
well in this area.  There is often disagreement but in the end I think
usually end up in a place that is good for PostgreSQL.  Hopefully that
will continue.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

Joshua D. Drake <jd@commandprompt.com> — 2016-05-13T20:03:32Z

On 05/13/2016 12:05 PM, Robert Haas wrote:
> On Fri, May 13, 2016 at 12:35 PM, Joshua D. Drake <jd@commandprompt.com> wrote:
>> Hey, if I am wrong that's awesome. The impression I have is the general
>> workflow is this:

>> The difference being one of coopetition versions competition for the
>> betterment of the community. If there are companies that are doing that
>> already, that is awesome and I applaud it. I was just trying to further
>> drive that idea home.
>
> I think that's already happening.  I'm happy to see more of it.  In
> practical terms, though, it's harder to collaborate between companies
> because then you need two management teams to be on-board with it, and
> there can be other competing priorities.

Yep, that's true.


> If either company needs to
> pull staff of a project because of some competing priority (say,
> fixing a broken customer or addressing an urgent customer need), then
> the whole project can stall.  The whole wagon train moves at the pace
> of the slowest camel.  It's nice when we can collaborate across
> companies and I'm all for it, but sometimes it's faster to for a
> single company to just assign a couple of people to a project and tell
> them to go do it.
>
> Now, where this gets tricky is when it comes down to whether the
> end-product of that effort is something the community wants.  We all
> need to be careful not to make our corporate priorities into community
> priorities.  Features shouldn't get committed without a consensus that
> they are both useful and well-implemented, and prior discussion is a
> good way to achieve that.  On the whole, I think we've done reasonably
> well in this area.  There is often disagreement but in the end I think
> usually end up in a place that is good for PostgreSQL.  Hopefully that
> will continue.
>

+1

Sincerely,

JD

-- 
Command Prompt, Inc.                  http://the.postgres.company/
                         +1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Everyone appreciates your honesty, until you are honest with them.

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

Joshua D. Drake <jd@commandprompt.com> — 2016-05-13T20:04:52Z

On 05/13/2016 12:03 PM, Josh berkus wrote:
> On 05/13/2016 11:48 AM, Robert Haas wrote:
>> On Fri, May 13, 2016 at 12:12 PM, Joshua D. Drake <jd@commandprompt.com> wrote:

> Anyway, all of this is a moot point, because nobody has the power to
> tell the various companies what to do.  We're just lucky that everyone
> is still committed to writing stuff which adds to PostgreSQL.

Lucky? No. We earned it. We earned it through years and years of hard 
work. Should we be thankful? Absolutely. Should we be grateful that we 
have such a powerful and engaged commercial contribution base? 100%.


JD



-- 
Command Prompt, Inc.                  http://the.postgres.company/
                         +1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Everyone appreciates your honesty, until you are honest with them.

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

Merlin Moncure <mmoncure@gmail.com> — 2016-05-13T20:34:22Z

On Fri, May 13, 2016 at 2:05 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> Now, where this gets tricky is when it comes down to whether the
> end-product of that effort is something the community wants.  We all
> need to be careful not to make our corporate priorities into community
> priorities.  Features shouldn't get committed without a consensus that
> they are both useful and well-implemented, and prior discussion is a
> good way to achieve that.  On the whole, I think we've done reasonably
> well in this area.  There is often disagreement but in the end I think
> usually end up in a place that is good for PostgreSQL.  Hopefully that
> will continue.

PostgreSQL is a miracle of good development practices.  Companies can
collaborate or contribute any manner the see fit along with the
community at large.  Demands made on others based on opinion and/of
selfish benefit gets you nowhere fast and that's a good thing.

Open source projects, especially the bigger/better run ones, tend to
migrate towards development processes that are highly informal and
great in terms of the ratio of effort to progress in all things,
including and especially, collaboration. I regularly emulate the way
things are done here as best I can which tends to completely freak out
the more corporate type developers that run most of the bigger shops.
If I'm writing emails around here, it's probably because five minutes
earlier some developer refused to fix a bug because the resolution
wasn't notated properly in some byzantine requirements document.
Solace.

merlin

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

Josh Berkus <josh@agliodbs.com> — 2016-05-13T20:42:07Z

On 05/13/2016 01:04 PM, Joshua D. Drake wrote:
> On 05/13/2016 12:03 PM, Josh berkus wrote:
>> On 05/13/2016 11:48 AM, Robert Haas wrote:
>>> On Fri, May 13, 2016 at 12:12 PM, Joshua D. Drake
>>> <jd@commandprompt.com> wrote:
> 
>> Anyway, all of this is a moot point, because nobody has the power to
>> tell the various companies what to do.  We're just lucky that everyone
>> is still committed to writing stuff which adds to PostgreSQL.
> 
> Lucky? No. We earned it. We earned it through years and years of hard
> work. Should we be thankful? Absolutely. Should we be grateful that we
> have such a powerful and engaged commercial contribution base? 100%.

Lucky.  Sure there was work and personal integrity involved, but like
any success story, there was luck.

But we've also been fortunate in not spawning hostile-but-popular forks
by people who left the project, and that none of the companies who
created hostile forks were very successful with them, and that nobody
has seriously tried using lawyers to control/ruin the project.

And, most importantly, we've been lucky that a lot of competing projects
have self-immolated instead of being successful and brain-draining our
contributors (MySQL, ANTS, MonetDB, etc.)

-- 
--
Josh Berkus
Red Hat OSAS
(any opinions are my own)

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

Justin Clift <justin@postgresql.org> — 2016-05-13T21:12:29Z

On 13 May 2016, at 21:42, Josh berkus <josh@agliodbs.com> wrote:
> On 05/13/2016 01:04 PM, Joshua D. Drake wrote:
>> On 05/13/2016 12:03 PM, Josh berkus wrote:
>>> On 05/13/2016 11:48 AM, Robert Haas wrote:
>>>> On Fri, May 13, 2016 at 12:12 PM, Joshua D. Drake
>>>> <jd@commandprompt.com> wrote:
>> 
>>> Anyway, all of this is a moot point, because nobody has the power to
>>> tell the various companies what to do.  We're just lucky that everyone
>>> is still committed to writing stuff which adds to PostgreSQL.
>> 
>> Lucky? No. We earned it. We earned it through years and years of hard
>> work. Should we be thankful? Absolutely. Should we be grateful that we
>> have such a powerful and engaged commercial contribution base? 100%.
> 
> Lucky.  Sure there was work and personal integrity involved, but like
> any success story, there was luck.
> 
> But we've also been fortunate in not spawning hostile-but-popular forks
> by people who left the project, and that none of the companies who
> created hostile forks were very successful with them, and that nobody
> has seriously tried using lawyers to control/ruin the project.
> 
> And, most importantly, we've been lucky that a lot of competing projects
> have self-immolated instead of being successful and brain-draining our
> contributors (MySQL, ANTS, MonetDB, etc.)

Oracle buying MySQL (via Sun) seems to have helped things along pretty
well too. :)

+ Justin

--
"My grandfather once told me that there are two kinds of people: those
who work and those who take the credit. He told me to try to be in the
first group; there was less competition there."
- Indira Gandhi

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

Joshua D. Drake <jd@commandprompt.com> — 2016-05-13T22:05:14Z

On 05/13/2016 01:42 PM, Josh berkus wrote:
> On 05/13/2016 01:04 PM, Joshua D. Drake wrote:
>> On 05/13/2016 12:03 PM, Josh berkus wrote:
>>> On 05/13/2016 11:48 AM, Robert Haas wrote:
>>>> On Fri, May 13, 2016 at 12:12 PM, Joshua D. Drake
>>>> <jd@commandprompt.com> wrote:
>>
>>> Anyway, all of this is a moot point, because nobody has the power to
>>> tell the various companies what to do.  We're just lucky that everyone
>>> is still committed to writing stuff which adds to PostgreSQL.
>>
>> Lucky? No. We earned it. We earned it through years and years of hard
>> work. Should we be thankful? Absolutely. Should we be grateful that we
>> have such a powerful and engaged commercial contribution base? 100%.
>
> Lucky.  Sure there was work and personal integrity involved, but like
> any success story, there was luck.
>
> But we've also been fortunate in not spawning hostile-but-popular forks
> by people who left the project, and that none of the companies who
> created hostile forks were very successful with them, and that nobody
> has seriously tried using lawyers to control/ruin the project.

I can't get behind you on this. Everything you have said above has to do 
with the hard work and integrity of the people in this project. It isn't 
luck or divine intervention.

>
> And, most importantly, we've been lucky that a lot of competing projects
> have self-immolated instead of being successful and brain-draining our
> contributors (MySQL, ANTS, MonetDB, etc.)
>

Actually there are people that have been drained out, I won't name them 
but it is pretty easy to figure out who they are. The people that are 
left, especially the long timers are here because of their integrity and 
attachment to the project.

This project builds good people, and the good people build a good project.

I am not going to buy into a luck story for something I and many others 
have invested decades of their life into.

JD




-- 
Command Prompt, Inc.                  http://the.postgres.company/
                         +1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Everyone appreciates your honesty, until you are honest with them.

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

David G. Johnston <david.g.johnston@gmail.com> — 2016-05-14T02:28:54Z

On Fri, May 13, 2016 at 6:05 PM, Joshua D. Drake <jd@commandprompt.com>
wrote:

> On 05/13/2016 01:42 PM, Josh berkus wrote:
>
>> On 05/13/2016 01:04 PM, Joshua D. Drake wrote:
>>
>>> On 05/13/2016 12:03 PM, Josh berkus wrote:
>>>
>>>> On 05/13/2016 11:48 AM, Robert Haas wrote:
>>>>
>>>>> On Fri, May 13, 2016 at 12:12 PM, Joshua D. Drake
>>>>> <jd@commandprompt.com> wrote:
>>>>>
>>>>
>>> Anyway, all of this is a moot point, because nobody has the power to
>>>> tell the various companies what to do.  We're just lucky that everyone
>>>> is still committed to writing stuff which adds to PostgreSQL.
>>>>
>>>
>>> Lucky? No. We earned it. We earned it through years and years of hard
>>> work. Should we be thankful? Absolutely. Should we be grateful that we
>>> have such a powerful and engaged commercial contribution base? 100%.
>>>
>>
>> Lucky.  Sure there was work and personal integrity involved, but like
>> any success story, there was luck.
>>
>> But we've also been fortunate in not spawning hostile-but-popular forks
>> by people who left the project, and that none of the companies who
>> created hostile forks were very successful with them, and that nobody
>> has seriously tried using lawyers to control/ruin the project.
>>
>
> I can't get behind you on this. Everything you have said above has to do
> with the hard work and integrity of the people in this project. It isn't
> luck or divine intervention.
>
>
>> And, most importantly, we've been lucky that a lot of competing projects
>> have self-immolated instead of being successful and brain-draining our
>> contributors (MySQL, ANTS, MonetDB, etc.)
>>
>>
> Actually there are people that have been drained out, I won't name them
> but it is pretty easy to figure out who they are. The people that are left,
> especially the long timers are here because of their integrity and
> attachment to the project.
>
> This project builds good people, and the good people build a good project.
>
> I am not going to buy into a luck story for something I and many others
> have invested decades of their life into.


I'm not sure how this ended up a philosophical/religious argument but
given that we are simply lucky that a meteor didn't wipe out our species 20
thousand years ago our present reality is a combination of determination
and luck and trying to look at shorter term slices and assign weights to
how much luck influenced something and how much was determination seems
like something best done over drinks.

In the vein of "praise publicly, sold privately" if anyone here thinks
that someone else should be acting differently I'd suggest sending them a
private message to that effect.  More useful would be to point out on these
lists things you find are done well.

David J.

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

David G. Johnston <david.g.johnston@gmail.com> — 2016-05-14T02:56:33Z

On Fri, May 13, 2016 at 12:12 PM, Joshua D. Drake <jd@commandprompt.com>
wrote:

> On 05/13/2016 07:42 AM, Robert Haas wrote:
>
>> On Sat, Apr 30, 2016 at 8:46 PM, Joshua Drake <jd@commandprompt.com>
>> wrote:
>>
>>> Oh, absolutely. I was just pointing out how a lot of companies are
>>> hoarding
>>> talent internally for no productive purpose.
>>>
>>
>> Wow, really?
>>
>
[...]

>
>
>> Let's respect people and companies for what they do contribute, rather
>> than labeling it as not good enough.
>>
>>
> There was no disrespect intended. I was trying to push forth an idea that
> multi-company team collaboration is better for the community than single
> company team collaboration. I will stand by that assertion.

Yeah, that failed...

I find it much easier to debate the assertion "multi-company team
collaboration is better ..." than debating whether "companies are
hoarding...".

I'll disagree - such a blanket statement is nearly worthless.  Three
one-person companies working together would not automatically produce
better outputs than a single company dedicating 3 people to an endeavor.
But even that retort requires me to cheat because I'm able to draw a
conclusion about the fallacy of generalization without needing to be any
more precise as to my concept of "better".

When it comes to the first statement nothing can be done to avoid the fact
that no frame-of-reference has been established for "productive purpose".

Contrary to my earlier point the nature of our setup makes the whole "scold
privately" aspect difficult - and maybe this approach works better than I
can imagine.  Though the down-thread from the quoted post does seem to
indicate that considerable non-lethal friendly-fire is one of its
byproducts.

David J.

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

Jim Nasby <jim.nasby@bluetreble.com> — 2016-05-15T20:23:52Z

On 4/29/16 10:37 AM, Joshua D. Drake wrote:
>>> 5. Transparent upgrade-in-place (i.e. allowing 10.2 to use 10.1's tables
>>> without pg_upgrade or other modification).
>>
>> Technically, this is exactly what pg_upgrade does.  I think what you
>> really mean is for the backend binary to be able to read the system
>> tables and WAL files of the old clusters --- something I can't see us
>> implementing anytime soon.
>>
>
> For the most part, pg_upgrade is good enough. There are exceptions and
> it does need a more thorough test suite but as a whole, it works. As
> nice as being able to install 9.6 right on top of 9.5 and have 9.6
> magically work, it is certainly not a *requirement* anymore.

My 2 issues with pg_upgrade are:

1) It's prone to bugs, because it operates at the binary level. This is 
similar to how it's MUCH easier to mess up PITR than pg_dump. Perhaps 
there's no way to avoid this.

2) There's no ability at all to revert, other than restore a backup. 
That means if you pull the trigger and discover some major performance 
problem, you have no choice but to deal with it (you can't switch back 
to the old version without losing data).

For many users those issues just don't matter; but in my work with 
financial data it's why I've never actually used it. #2 especially was 
good to have (in our case, via londiste). It also made it a lot easier 
to find performance issues beforehand, by switching reporting replicas 
over to the new version first.

One other consideration is cut-over time. Swapping logical master and 
replica can happen nearly instantly, while pg_upgrade needs some kind of 
outage window.
-- 
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com
855-TREBLE2 (855-873-2532)   mobile: 512-569-9461

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

Bruce Momjian <bruce@momjian.us> — 2016-05-16T07:36:17Z

On Sun, May 15, 2016 at 03:23:52PM -0500, Jim Nasby wrote:
> 2) There's no ability at all to revert, other than restore a backup. That
> means if you pull the trigger and discover some major performance problem,
> you have no choice but to deal with it (you can't switch back to the old
> version without losing data).

In --link mode only

> For many users those issues just don't matter; but in my work with financial
> data it's why I've never actually used it. #2 especially was good to have
> (in our case, via londiste). It also made it a lot easier to find
> performance issues beforehand, by switching reporting replicas over to the
> new version first.
> 
> One other consideration is cut-over time. Swapping logical master and
> replica can happen nearly instantly, while pg_upgrade needs some kind of
> outage window.

Right.  I am thinking of writing some docs about how to avoid downtime
for upgrades of various types.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+                     Ancient Roman grave inscription +

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

Robert Haas <robertmhaas@gmail.com> — 2016-05-20T23:40:53Z

On Mon, May 16, 2016 at 3:36 AM, Bruce Momjian <bruce@momjian.us> wrote:
> On Sun, May 15, 2016 at 03:23:52PM -0500, Jim Nasby wrote:
>> 2) There's no ability at all to revert, other than restore a backup. That
>> means if you pull the trigger and discover some major performance problem,
>> you have no choice but to deal with it (you can't switch back to the old
>> version without losing data).
>
> In --link mode only

No, not really.  Once you let write transactions into the new cluster,
there's no way to get back to the old server version no matter which
option you used.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

Jim Nasby <jim.nasby@bluetreble.com> — 2016-05-24T14:23:27Z

On 5/16/16 2:36 AM, Bruce Momjian wrote:
> Right.  I am thinking of writing some docs about how to avoid downtime
> for upgrades of various types.

If there's some magic sauce to shrink pg_upgrade downtime to near 0 I 
think folks would be very interested in that.

Outside of that scenario, I think what would be far more useful is 
information on how to do seamless master/replica switchovers using tools 
like pgBouncer or pgPool. That ability is useful *all* the time, not 
just when upgrading. It makes it trivial to do OS-level maintenance, and 
if you're using a form of logical replication it also makes it trivial 
to do expensive database maintenance, such as cluster/vacuum 
full/reindex. I've worked with a few clients that had that ability and 
it was a huge stress reducer. As a bonus, an unplanned outage of the 
master becomes far less stressful, because you already know exactly how 
to fail over.
-- 
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com
855-TREBLE2 (855-873-2532)   mobile: 512-569-9461

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

Bruce Momjian <bruce@momjian.us> — 2016-06-21T02:08:03Z

On Fri, May 20, 2016 at 07:40:53PM -0400, Robert Haas wrote:
> On Mon, May 16, 2016 at 3:36 AM, Bruce Momjian <bruce@momjian.us> wrote:
> > On Sun, May 15, 2016 at 03:23:52PM -0500, Jim Nasby wrote:
> >> 2) There's no ability at all to revert, other than restore a backup. That
> >> means if you pull the trigger and discover some major performance problem,
> >> you have no choice but to deal with it (you can't switch back to the old
> >> version without losing data).
> >
> > In --link mode only
> 
> No, not really.  Once you let write transactions into the new cluster,
> there's no way to get back to the old server version no matter which
> option you used.

Yes, there is, and it is documented:

        If you ran <command>pg_upgrade</command> <emphasis>without</>
        <option>--link</> or did not start the new server, the
        old cluster was not modified except that, if linking
        started, a <literal>.old</> suffix was appended to
        <filename>$PGDATA/global/pg_control</>.  To reuse the old
        cluster, possibly remove the <filename>.old</> suffix from
        <filename>$PGDATA/global/pg_control</>; you can then restart the
        old cluster.

What is confusing you?

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+                     Ancient Roman grave inscription +

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

Bruce Momjian <bruce@momjian.us> — 2016-06-21T02:10:02Z

On Tue, May 24, 2016 at 09:23:27AM -0500, Jim Nasby wrote:
> On 5/16/16 2:36 AM, Bruce Momjian wrote:
> >Right.  I am thinking of writing some docs about how to avoid downtime
> >for upgrades of various types.
> 
> If there's some magic sauce to shrink pg_upgrade downtime to near 0 I think
> folks would be very interested in that.
> 
> Outside of that scenario, I think what would be far more useful is
> information on how to do seamless master/replica switchovers using tools
> like pgBouncer or pgPool. That ability is useful *all* the time, not just
> when upgrading. It makes it trivial to do OS-level maintenance, and if
> you're using a form of logical replication it also makes it trivial to do
> expensive database maintenance, such as cluster/vacuum full/reindex. I've
> worked with a few clients that had that ability and it was a huge stress
> reducer. As a bonus, an unplanned outage of the master becomes far less
> stressful, because you already know exactly how to fail over.

pg_upgrade's runtime can't be decreased dramatically --- I wanted to
document other methods like you described.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+                     Ancient Roman grave inscription +

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

Robert Haas <robertmhaas@gmail.com> — 2016-06-21T12:19:55Z

On Mon, Jun 20, 2016 at 10:08 PM, Bruce Momjian <bruce@momjian.us> wrote:
> On Fri, May 20, 2016 at 07:40:53PM -0400, Robert Haas wrote:
>> On Mon, May 16, 2016 at 3:36 AM, Bruce Momjian <bruce@momjian.us> wrote:
>> > On Sun, May 15, 2016 at 03:23:52PM -0500, Jim Nasby wrote:
>> >> 2) There's no ability at all to revert, other than restore a backup. That
>> >> means if you pull the trigger and discover some major performance problem,
>> >> you have no choice but to deal with it (you can't switch back to the old
>> >> version without losing data).
>> >
>> > In --link mode only
>>
>> No, not really.  Once you let write transactions into the new cluster,
>> there's no way to get back to the old server version no matter which
>> option you used.
>
> Yes, there is, and it is documented:
>
>         If you ran <command>pg_upgrade</command> <emphasis>without</>
>         <option>--link</> or did not start the new server, the
>         old cluster was not modified except that, if linking
>         started, a <literal>.old</> suffix was appended to
>         <filename>$PGDATA/global/pg_control</>.  To reuse the old
>         cluster, possibly remove the <filename>.old</> suffix from
>         <filename>$PGDATA/global/pg_control</>; you can then restart the
>         old cluster.
>
> What is confusing you?

I don't think I'm confused.  Sure, you can do that, but the effects of
any writes performed on the new cluster will not be there when you
revert back to the old cluster.  So you will have effectively lost
data, unless you somehow have the ability to re-apply all of those
write transactions somehow.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

Craig Ringer <craig@2ndquadrant.com> — 2016-06-21T12:56:09Z

On 21 June 2016 at 20:19, Robert Haas <robertmhaas@gmail.com> wrote:

> On Mon, Jun 20, 2016 at 10:08 PM, Bruce Momjian <bruce@momjian.us> wrote:
> > On Fri, May 20, 2016 at 07:40:53PM -0400, Robert Haas wrote:
> >> On Mon, May 16, 2016 at 3:36 AM, Bruce Momjian <bruce@momjian.us>
> wrote:
> >> > On Sun, May 15, 2016 at 03:23:52PM -0500, Jim Nasby wrote:
> >> >> 2) There's no ability at all to revert, other than restore a backup.
> That
> >> >> means if you pull the trigger and discover some major performance
> problem,
> >> >> you have no choice but to deal with it (you can't switch back to the
> old
> >> >> version without losing data).
> >> >
> >> > In --link mode only
> >>
> >> No, not really.  Once you let write transactions into the new cluster,
> >> there's no way to get back to the old server version no matter which
> >> option you used.
> >
> > Yes, there is, and it is documented:
> >
> >         If you ran <command>pg_upgrade</command> <emphasis>without</>
> >         <option>--link</> or did not start the new server, the
> >         old cluster was not modified except that, if linking
> >         started, a <literal>.old</> suffix was appended to
> >         <filename>$PGDATA/global/pg_control</>.  To reuse the old
> >         cluster, possibly remove the <filename>.old</> suffix from
> >         <filename>$PGDATA/global/pg_control</>; you can then restart the
> >         old cluster.
> >
> > What is confusing you?
>
> I don't think I'm confused.  Sure, you can do that, but the effects of
> any writes performed on the new cluster will not be there when you
> revert back to the old cluster.  So you will have effectively lost
> data, unless you somehow have the ability to re-apply all of those
> write transactions somehow.
>

Also, if you run *with* --link, IIRC there's no guarantee that the old
version will be happy to see any new infomask bits etc introduced by the
new Pg. I think there will also be issues with oid to relfilenode mappings
in pg_class if the new cluster did any VACUUM FULLs or anything. It seems
likely to be a bit risky to fall back on the old cluster once you've
upgraded with --link . TBH it never even occurred to me that it'd be
possible at all until you mentioned.

I always thought of pg_upgrade as a one-way no-going-back process either
way, really. Either due to a fork in history (without --link) or due to
possibly incompatible datadir changes (with --link).

-- 
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

Bruce Momjian <bruce@momjian.us> — 2016-06-21T15:34:55Z

On Tue, Jun 21, 2016 at 08:19:55AM -0400, Robert Haas wrote:
> On Mon, Jun 20, 2016 at 10:08 PM, Bruce Momjian <bruce@momjian.us> wrote:
> >> No, not really.  Once you let write transactions into the new cluster,
> >> there's no way to get back to the old server version no matter which
> >> option you used.
> >
> > Yes, there is, and it is documented:
> >
> >         If you ran <command>pg_upgrade</command> <emphasis>without</>
> >         <option>--link</> or did not start the new server, the
> >         old cluster was not modified except that, if linking
> >         started, a <literal>.old</> suffix was appended to
> >         <filename>$PGDATA/global/pg_control</>.  To reuse the old
> >         cluster, possibly remove the <filename>.old</> suffix from
> >         <filename>$PGDATA/global/pg_control</>; you can then restart the
> >         old cluster.
> >
> > What is confusing you?
> 
> I don't think I'm confused.  Sure, you can do that, but the effects of
> any writes performed on the new cluster will not be there when you
> revert back to the old cluster.  So you will have effectively lost
> data, unless you somehow have the ability to re-apply all of those
> write transactions somehow.

Yes, that is true.  I assume _revert_ means something really bad
happened and you don't want those writes because they are somehow
corrupt.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+                     Ancient Roman grave inscription +

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

Robert Haas <robertmhaas@gmail.com> — 2016-06-21T16:12:34Z

On Tue, Jun 21, 2016 at 11:34 AM, Bruce Momjian <bruce@momjian.us> wrote:
> On Tue, Jun 21, 2016 at 08:19:55AM -0400, Robert Haas wrote:
>> On Mon, Jun 20, 2016 at 10:08 PM, Bruce Momjian <bruce@momjian.us> wrote:
>> >> No, not really.  Once you let write transactions into the new cluster,
>> >> there's no way to get back to the old server version no matter which
>> >> option you used.
>> >
>> > Yes, there is, and it is documented:
>> >
>> >         If you ran <command>pg_upgrade</command> <emphasis>without</>
>> >         <option>--link</> or did not start the new server, the
>> >         old cluster was not modified except that, if linking
>> >         started, a <literal>.old</> suffix was appended to
>> >         <filename>$PGDATA/global/pg_control</>.  To reuse the old
>> >         cluster, possibly remove the <filename>.old</> suffix from
>> >         <filename>$PGDATA/global/pg_control</>; you can then restart the
>> >         old cluster.
>> >
>> > What is confusing you?
>>
>> I don't think I'm confused.  Sure, you can do that, but the effects of
>> any writes performed on the new cluster will not be there when you
>> revert back to the old cluster.  So you will have effectively lost
>> data, unless you somehow have the ability to re-apply all of those
>> write transactions somehow.
>
> Yes, that is true.  I assume _revert_ means something really bad
> happened and you don't want those writes because they are somehow
> corrupt.

I think that it's pretty likely you could, say, upgrade to a new major
release, discover that it has a performance problem or some other bug
that causes a problem for you, and want to go back to the older
release.  There's not really an easy way to do that, because a pg_dump
taken from the new system might not restore on the older one.  Logical
replication - e.g. Slony - can provide a way, but we don't have
anything in core that can do it.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

Bruce Momjian <bruce@momjian.us> — 2016-06-21T16:15:02Z

On Tue, Jun 21, 2016 at 08:56:09PM +0800, Craig Ringer wrote:
> Also, if you run *with* --link, IIRC there's no guarantee that the old version
> will be happy to see any new infomask bits etc introduced by the new Pg. I

Well, we only write system tables in pg_upgrade in the new cluster, and
those are not hard linked.  As far as I know, we never write to anything
we hard link from the old cluster.

> think there will also be issues with oid to relfilenode mappings in pg_class if
> the new cluster did any VACUUM FULLs or anything. It seems likely to be a bit

pg_upgrade turns off all vacuums.

> risky to fall back on the old cluster once you've upgraded with --link . TBH it
> never even occurred to me that it'd be possible at all until you mentioned.

Well, with --link, you can't start the old cluster, and that is
documented, and pg_control is renamed to prevent accidental start.  I
think it is possible to start the old cluster before the new cluster is
started.

> I always thought of pg_upgrade as a one-way no-going-back process either way,
> really. Either due to a fork in history (without --link) or due to possibly
> incompatible datadir changes (with --link).

Yes.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+                     Ancient Roman grave inscription +

Re: Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

Bruce Momjian <bruce@momjian.us> — 2016-06-21T16:15:34Z

On Tue, Jun 21, 2016 at 12:12:34PM -0400, Robert Haas wrote:
> >> > What is confusing you?
> >>
> >> I don't think I'm confused.  Sure, you can do that, but the effects of
> >> any writes performed on the new cluster will not be there when you
> >> revert back to the old cluster.  So you will have effectively lost
> >> data, unless you somehow have the ability to re-apply all of those
> >> write transactions somehow.
> >
> > Yes, that is true.  I assume _revert_ means something really bad
> > happened and you don't want those writes because they are somehow
> > corrupt.
> 
> I think that it's pretty likely you could, say, upgrade to a new major
> release, discover that it has a performance problem or some other bug
> that causes a problem for you, and want to go back to the older
> release.  There's not really an easy way to do that, because a pg_dump
> taken from the new system might not restore on the older one.  Logical
> replication - e.g. Slony - can provide a way, but we don't have
> anything in core that can do it.

Yes, there is data loss in a rollback to the old cluster, no question.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+                     Ancient Roman grave inscription +