Horton

Thread

Commits

  1. Provide for forward compatibility with future minor protocol versions.

  2. Give nicer error message when connecting to a v10 server requiring SCRAM.

  1. protocol version negotiation (Re: Libpq PGRES_COPY_BOTH - version compatibility)

    Robert Haas <robertmhaas@gmail.com> — 2017-06-28T17:24:03Z

    On Mon, Mar 28, 2011 at 7:07 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> I wrote:
>> Now if we had a track record showing that we could tweak the protocol
>> version without causing problems, it'd be fine with me to do it for this
>> usage.  But we don't, and this particular case doesn't seem like the
>> place to start.
>
> And, btw, a moment's study of the protocol version checking code in
> postmaster.c shows that bumping the minor version number to 3.1 *would*
> break things: a client requesting 3.1 from a current postmaster would
> get a failure.
>
> Maybe we oughta change that logic --- it's not clear to me that there's
> any meaningful difference between major and minor numbers given the
> current postmaster behavior.

I just had a conversation at Postgres Vision that reminded me of the
old thread quoted aboce, and it seems to me that what Tom said there
is a good idea.  We've made quite a number of protocol version changes
over the years -- most recently, SCRAM introduced AuthenticationSASL
-- without ever bumping the protocol version.  In the case of the
change that this thread was originally about, namely the introduction
of CopyBoth, it could be argued that there is no real need to bump the
protocol version, because the client initiates the use of the new
protocol message and, before doing so, can know whether the server
version is new enough to support that protocol message based on the
ParameterStatus message for server_version which is sent after
authentication is complete.  Fair enough, but:

1. If we want to add a new message which is initiated by the server
rather than the client, the server currently has no way of knowing
whether the client is new enough to understand the message, because
the client never identifies itself except by submitting a protocol
version which must always be exactly 3.0 or the connection will be
killed.  SCRAM itself suffers from this problem; if an older libpq
tries to connect to a v10 server, they'll get a lousy error message
unless they've got a minor release new enough to contain
96d0f988b150aa0b52b44b8c1adbc7ef59262a1a or one of its siblings.
That's not the worst problem ever, but at the same time it's not
really a good design to rely on people upgrading the clients in order
to get the correct message about the client being too old.  It would
be much better if the client identified itself to the server.  Then if
the server sees that the client is old, it can give the client an
appropriate error, and if the client is new enough, then it can be
sure that it will be understood.

2. If we want to add a new message is initiated by the client before
authentication is complete, ParameterStatus messages won't have been
sent yet, so the client has no idea what capabilities the server might
be able to support.

3. It's a bit unclean to use the server version as a proxy for the
wire protocol version; there are non-PostgreSQL servers which speak
the PostgreSQL wire protocol, and making them impersonate a specific
version of PostgreSQL in order to clarify which wire protocol features
they support isn't really very nice.  So maybe there's a case for
bumping the minor version each time we actually change the protocol
rather than only in the cases where the client has no other method of
discovering the server version.  For example, you can imagine a
connection proxy like pgpool or pgbouncer wanting to limit the client
to a particular wire protocol version -- the highest that it supports
-- while proxying to a newer server version sitting behind it.  That
doesn't work if the server version is being abused as a sort of
back-door protocol version.

Here's my proposal:

- If the server receives a StartupMessage for v3.x where x > the
version it knows, instead of just slamming the connection shut, it
responds by sending some new message (let's say,
NegotiateProtocolVersion) specifying the highest protocol version it
supports.

- The client may then continue by sending a new StartupMessage with a
version with a version number that is within range.

If we had this, then bumping the minor protocol version wouldn't
require retrying connections, which among other problems generates
scary error messages in the logs; instead, they would succeed, but
with one additional round trip.  That seems like it would represent a
substantial improvement over what we have now, where bumping the
protocol version is such an incredibly painful thing that we never do
it, even when we've actually broken wire protocol compatibility!

Of course, it would take a few releases before server versions that
support this mechanism were widespread, and so clients using a
hypothetical version 3.1 would still have to be prepared to retry the
whole connection, but if we use that as a reason not to make a change
of this sort then it's not clear how things ever get any better than
they are now.  And it's clear that there is pent-up demand for the
ability to make minor additions to the protocol without breaking the
whole thing - see
https://www.postgresql.org/message-id/1318235405.31687.0.camel%40fsopti579.F-Secure.com
and https://wiki.postgresql.org/wiki/Todo#Wire_Protocol_Changes_.2F_v4_Protocol
- and while it's not clear that all of those ideas are good ones, some
of them probably are, and even if by some remote chance all of the
people who suggested them are all dead wrong, there will be more
things suggested in the future.  We shouldn't wait until our back is
totally against the wall before we introduce some mechanism here that
is better than what we have today.  I'm not really keen on trying to
introduce a v4 protocol any time soon, but I really like the idea of
having a reasonably low-pain way to introduce a v3.1 that adds modest
new capabilities.

Thoughts, better ideas?

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  2. Re: protocol version negotiation (Re: Libpq PGRES_COPY_BOTH - version compatibility)

    Tom Lane <tgl@sss.pgh.pa.us> — 2017-06-28T17:47:13Z

    Robert Haas <robertmhaas@gmail.com> writes:
> Here's my proposal:

> - If the server receives a StartupMessage for v3.x where x > the
> version it knows, instead of just slamming the connection shut, it
> responds by sending some new message (let's say,
> NegotiateProtocolVersion) specifying the highest protocol version it
> supports.

> - The client may then continue by sending a new StartupMessage with a
> version with a version number that is within range.

How about a slightly simpler design: we specify that all 3.x protocol
versions shall be mutually compatible so long as each side knows not
to send something that the other side doesn't understand.  The field
in the StartupMessage is understood to be the max protocol version the
client knows how to cope with.  If the client sends something > 3.0,
then the server responds with a ServerProtocolVersion message containing
the max protocol version it understands.  Now both sides know what the
other side can do and should be able to adapt their behavior accordingly.
(If the client doesn't get a ServerProtocolVersion message, it should
assume server protocol 3.0.)

In order to make the world safe for this, we'd have to adjust existing
releases to not complain about 3.x for x > 0, and the sooner we do that
the better chance of clients being able to make use of this within a
reasonable timeframe.

It's possible that we should do something that's not based on just a
linear protocol version number, but instead involves some kind of
bitmask of capabilities, say.  So the ServerProtocolVersion message
maybe needs to be a bit more complicated, and if the client does get
one back, maybe it should forward a ClientProtocolVersion message with
its own bitmask.  But these things could be designed in detail later.

			regards, tom lane

  3. Re: protocol version negotiation (Re: Libpq PGRES_COPY_BOTH - version compatibility)

    Robert Haas <robertmhaas@gmail.com> — 2017-06-28T19:01:29Z

    On Wed, Jun 28, 2017 at 1:47 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Robert Haas <robertmhaas@gmail.com> writes:
>> Here's my proposal:
>
>> - If the server receives a StartupMessage for v3.x where x > the
>> version it knows, instead of just slamming the connection shut, it
>> responds by sending some new message (let's say,
>> NegotiateProtocolVersion) specifying the highest protocol version it
>> supports.
>
>> - The client may then continue by sending a new StartupMessage with a
>> version with a version number that is within range.
>
> How about a slightly simpler design: we specify that all 3.x protocol
> versions shall be mutually compatible so long as each side knows not
> to send something that the other side doesn't understand.  The field
> in the StartupMessage is understood to be the max protocol version the
> client knows how to cope with.  If the client sends something > 3.0,
> then the server responds with a ServerProtocolVersion message containing
> the max protocol version it understands.  Now both sides know what the
> other side can do and should be able to adapt their behavior accordingly.
> (If the client doesn't get a ServerProtocolVersion message, it should
> assume server protocol 3.0.)

One problem with that is that it means that the format of the
StartupMessage itself can never change, which I think is not a good
choice.  If, for example, we want to add a new option to the startup
message (aside from the existing user, database, options, and
replication keywords), we really can't do that today.  It wouldn't be
so bad if unrecognized parameters were just ignored; the client would
know from the ServerProtocolVersion (or ParameterStatus) message that
server had ignored those options and could respond as it saw fit.  But
because they are treated as GUCs the server will error out.   This is
clearly not an entirely theoretical concern considering that
'replication' was added as a keyword here within recent memory.

> In order to make the world safe for this, we'd have to adjust existing
> releases to not complain about 3.x for x > 0, and the sooner we do that
> the better chance of clients being able to make use of this within a
> reasonable timeframe.
>
> It's possible that we should do something that's not based on just a
> linear protocol version number, but instead involves some kind of
> bitmask of capabilities, say.  So the ServerProtocolVersion message
> maybe needs to be a bit more complicated, and if the client does get
> one back, maybe it should forward a ClientProtocolVersion message with
> its own bitmask.  But these things could be designed in detail later.

Right.  So for example we could decide that any parameter names that
are passed in the startup packet that begin with an underscore are not
GUCs but some kind of protocol extension; the server replies with some
message containing a list of the ones which were not understood.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  4. Re: protocol version negotiation (Re: Libpq PGRES_COPY_BOTH - version compatibility)

    Craig Ringer <craig@2ndquadrant.com> — 2017-06-29T01:44:44Z

    On 29 June 2017 at 03:01, Robert Haas <robertmhaas@gmail.com> wrote:

> One problem with that is that it means that the format of the
> StartupMessage itself can never change, which I think is not a good
> choice.

The startup message could be immediately followed by another
supplemental message, though.

- Startup["Protocol 3.2"]
- Extra Info Message

A server that only knows protocol version 3.1 would read the 3.1
message and reply with a NegotiateProtocolVersion or whatever, then
*discard all messages until it sees a new startup message*, so it
wouldn't be upset. Quite like what we do when reading until Sync after
an error.

Existing servers would bounce the connection when they saw the 3.2
field and the client would have to reconnect with 3.0. But we could
possibly even backpatch the nicer behaviour, since it's simple and
will only ever come into effect when sent a higher protocol version
msg that no existing client generates.

 If we ever landed up wanting to greatly revise the startup message
down the track, we could land up sending a stub startup message with
protocol 3.4 or whatever that just tells aware servers "real startup
message follows". Bit of a waste, but not that bad given how expensive
connections are. A dummy startup message is quite small.

I don't have a strong opinion here, just raising the possibility that
not being able to vary the startup message format may not be the end
of the world.

>  If, for example, we want to add a new option to the startup
> message (aside from the existing user, database, options, and
> replication keywords), we really can't do that today.

Right. An obvious example would be to put a starttls-like request in
it, asking the server to initiate TLS negotiation if supported.

Or an option asking the server to determine the DB to connect to based
on some property of the user.

Similarly, I've repeatedly wanted ways to specify client support for /
requests for optional protocol messages, like sending the xid and
commit record lsn of a commit along with the commit message.

Both those could be done in immediate follow-up messages, though. I
can't personally think of much right away that wouldn't work pretty
well in a follow-on message. There's a small one-off protocol size
overhead with doing it that way, but no extra latency, so who cares?

> It wouldn't be
> so bad if unrecognized parameters were just ignored; the client would
> know from the ServerProtocolVersion (or ParameterStatus) message that
> server had ignored those options and could respond as it saw fit.

Yeah. In retrospect it's a pity the key/value pairs don't come with
some kind of optional/required flag so the client can say "error if
you don't understand this, it's important" or "I don't care if you
don't understand this, and I'll notice when you fail to GUC_REPORT it
back to me". Or some convention like underscore-prefixing for
optional.

>> It's possible that we should do something that's not based on just a
>> linear protocol version number, but instead involves some kind of
>> bitmask of capabilities, say.  So the ServerProtocolVersion message
>> maybe needs to be a bit more complicated, and if the client does get
>> one back, maybe it should forward a ClientProtocolVersion message with
>> its own bitmask.  But these things could be designed in detail later.
>
> Right.  So for example we could decide that any parameter names that
> are passed in the startup packet that begin with an underscore are not
> GUCs but some kind of protocol extension; the server replies with some
> message containing a list of the ones which were not understood.

There's a *lot* of value to that idea when you consider proxies like
PgPool-II and PgBouncer that may struggle to satisfy all capabilities
of some protocol 3.3 but want to offer some key parts of it.

There's a price in terms of complexity but probably not a big one.
Some places that would be "client protocol version >= 3.3" would
instead be "client has_capability(x)". Possibly cleaner to read even
if we _did_ use a linear protocol version, frankly.

Also, importantly in my opinion, this would let clients turn things
on/off easily. I badly want this myself, as I really want the server
to be able to reply to each COMMIT with an extended CommandComplete
message or extra-info message after it, containing the commit record
LSN. Similarly, I really want to be able to send the xid of an xact on
the wire when one is assigned to an xact. Many clients won't need,
want or understand these things and whether they're protocol 3.0 or
3.9 they can just not ask for them.

Capabilities will make startup messages bigger. Personally I don't
care much about that, as on modern networks it's all about latency not
message size. We'd use abbreviated capability names I expect. If the
list gets too big we could always roll up capabilities that have
become universally adopted into the next protocol version bump so it's
assumed that a client announcing proto 3.3 supports feature x and
doesn't need to list it.



-- 
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

  5. Re: protocol version negotiation (Re: Libpq PGRES_COPY_BOTH - version compatibility)

    Craig Ringer <craig@2ndquadrant.com> — 2017-06-29T01:55:17Z

    On 29 June 2017 at 09:44, Craig Ringer <craig@2ndquadrant.com> wrote:

> I
> can't personally think of much right away that wouldn't work pretty
> well in a follow-on message.

Actually, I take that back, there's one thing that's bugged me for a
while that wouldn't work well this way: determining the correct text
encoding with which to interpret the database name, user name, and any
string GUC values. Right now we take a wild stab in the dark and use
the server encoding.

e.g. in a terminal with LC_ALL=en_US.UTF-8 I run

postgres=# CREATE ROLE "café";
CREATE ROLE
postgres=# CREATE DATABASE "café" with owner "café";
CREATE DATABASE
postgres=# \du ca*
            List of roles
 Role name |  Attributes  | Member of
-----------+--------------+-----------
 café      | Cannot login | {}

postgres=# \l ca*
                            List of databases
 Name | Owner | Encoding |   Collate   |    Ctype    | Access privileges
------+-------+----------+-------------+-------------+-------------------
 café | café  | UTF8     | en_AU.UTF-8 | en_AU.UTF-8 |
(1 row)


then in a terminal with LC_ALL=en_US.ISO-8859-1 I run:

[craig@ayaki-localdomain log]$ psql -U "café"
psql: FATAL:  role "café" does not exist
[craig@ayaki-localdomain log]$ psql
psql (9.6.3, server 9.5.7)
Type "help" for help.

craig=> \du ca*
            List of roles
 Role name |  Attributes  | Member of
-----------+--------------+-----------
 café      | Cannot login | {}

craig=> \l ca*
                            List of databases
 Name | Owner | Encoding |   Collate   |    Ctype    | Access privileges
------+-------+----------+-------------+-------------+-------------------
 café | café  | UTF8     | en_AU.UTF-8 | en_AU.UTF-8 |
(1 row)

craig=> \c café
FATAL:  database "café" does not exist
Previous connection kept


Um, say what?

This happens because psql does server=>client encoding conversion once
connected, but there's no way to convert the startup message with the
username.

That IMO is an argument to allow startup message format change. (It'd
also then let the server reply with correctly-encoded pre-auth
messages for errors, something we currently fail to do).

-- 
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

  6. Re: protocol version negotiation (Re: Libpq PGRES_COPY_BOTH - version compatibility)

    Tom Lane <tgl@sss.pgh.pa.us> — 2017-06-29T02:27:45Z

    Craig Ringer <craig@2ndquadrant.com> writes:
> On 29 June 2017 at 03:01, Robert Haas <robertmhaas@gmail.com> wrote:
>> It wouldn't be
>> so bad if unrecognized parameters were just ignored; the client would
>> know from the ServerProtocolVersion (or ParameterStatus) message that
>> server had ignored those options and could respond as it saw fit.

> Yeah. In retrospect it's a pity the key/value pairs don't come with
> some kind of optional/required flag so the client can say "error if
> you don't understand this, it's important" or "I don't care if you
> don't understand this, and I'll notice when you fail to GUC_REPORT it
> back to me". Or some convention like underscore-prefixing for
> optional.

Yeah.  Back in the day I helped design the PNG image format, and one
of the better ideas in it was to make a distinction between critical and
noncritical chunks within a PNG file; that was exactly the idea you're
getting at here.  I agree with the suggestion to drive this off a
parameter-name-based convention.  Having leading underscore indicate
a noncritical parameter sounds fine.

I don't really like any of the ideas that have been mentioned that would
introduce extra network round trips into session startup.  It's expensive
enough already.  The only thing we seem to be really hurting on is the
ability for the client to add extensions to the original StartupMessage,
and this seems like it can fix that.

			regards, tom lane

  7. Re: protocol version negotiation (Re: Libpq PGRES_COPY_BOTH - version compatibility)

    Craig Ringer <craig@2ndquadrant.com> — 2017-06-29T04:23:00Z

    On 29 June 2017 at 10:27, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Craig Ringer <craig@2ndquadrant.com> writes:
>> On 29 June 2017 at 03:01, Robert Haas <robertmhaas@gmail.com> wrote:
>>> It wouldn't be
>>> so bad if unrecognized parameters were just ignored; the client would
>>> know from the ServerProtocolVersion (or ParameterStatus) message that
>>> server had ignored those options and could respond as it saw fit.
>
>> Yeah. In retrospect it's a pity the key/value pairs don't come with
>> some kind of optional/required flag so the client can say "error if
>> you don't understand this, it's important" or "I don't care if you
>> don't understand this, and I'll notice when you fail to GUC_REPORT it
>> back to me". Or some convention like underscore-prefixing for
>> optional.
>
> Yeah.  Back in the day I helped design the PNG image format, and one
> of the better ideas in it was to make a distinction between critical and
> noncritical chunks within a PNG file; that was exactly the idea you're
> getting at here.  I agree with the suggestion to drive this off a
> parameter-name-based convention.  Having leading underscore indicate
> a noncritical parameter sounds fine.
>
> I don't really like any of the ideas that have been mentioned that would
> introduce extra network round trips into session startup.  It's expensive
> enough already.  The only thing we seem to be really hurting on is the
> ability for the client to add extensions to the original StartupMessage,
> and this seems like it can fix that.

It does. But I don't see anywhere that extra round trips have been discussed.

-- 
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

  8. Re: protocol version negotiation (Re: Libpq PGRES_COPY_BOTH - version compatibility)

    Craig Ringer <craig@2ndquadrant.com> — 2017-06-29T04:24:44Z

    On 29 June 2017 at 12:23, Craig Ringer <craig@2ndquadrant.com> wrote:

> It does. But I don't see anywhere that extra round trips have been discussed.

Ah, right, they're implied by having the server respond with some
downversion message and ignore input until the client sends a new
startup message. That'll only happen when a too-new client connects to
an older server, but that's probably not an especially unusual case.

-- 
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

  9. Re: protocol version negotiation (Re: Libpq PGRES_COPY_BOTH - version compatibility)

    Robert Haas <robertmhaas@gmail.com> — 2017-06-29T12:18:17Z

    On Wed, Jun 28, 2017 at 10:27 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Yeah.  Back in the day I helped design the PNG image format, and one
> of the better ideas in it was to make a distinction between critical and
> noncritical chunks within a PNG file; that was exactly the idea you're
> getting at here.  I agree with the suggestion to drive this off a
> parameter-name-based convention.  Having leading underscore indicate
> a noncritical parameter sounds fine.

I'm not sure if non-critical is exactly the right terminology.  What
we want to do is distinguish between things that are intended as
protocol-level options vs. things that are intended as GUCs.  Of
course, there's nothing that actually prevents a a GUC from starting
with an underscore:

rhaas=# set _foo._bar = 1;
SET

Maybe we should instead pick a GUC namespace and reserve it for the
use of protocol level options; e.g. pg_protocol.<anything> becomes
invalid as the name of a GUC, but can be included in a startup packet
(or do we need to pick something shorter, like _pg, to keep the
message short?).  Servers ignore anything that they don't understand.
So:

1. The client sends a StartupMessage 3.x for version 3.x.  We could
bump the version explicitly, or perhaps we should just coin a version
of libpq for every server release; e.g. whatever PostgreSQL 11 ships
is version 3.11, etc.  It includes any protocol options that don't
exist today as pg_protocol.<whatever> in the startup packet.

2. If the client version is anything other than 3.0, the server
responds with a ServerProtocolVersion indicating the highest version
it supports, and ignores any pg_protocol.<whatever> options not known
to it as being either third-party extensions or something from a
future version.  If the initial response to the startup message is
anything other than a ServerProtocolVersion message, the client should
assume it's talking to a 3.0 server.  (To make this work, we would
back-patch a change into existing releases to allow any 3.x protocol
version and ignore any pg_protocol.<whatever> options that were
specified.)

If either the client or the server is unhappy about the age of the
other, then it can disconnect; e.g. if the server is configured to
require the use of whizzbang-2 security, and the client protocol
version indicates that at most whizzbang-1.9 is available, then the
server can close the connection with a suitable complaint; conversely,
if the connection string had require_whizzbang=2, and the server is
too old to support that, then the client can decide to bail out when
it receives the ServerProtocolVersion message.

Still just thinking out loud here.  Thoughts?

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  10. Re: protocol version negotiation (Re: Libpq PGRES_COPY_BOTH - version compatibility)

    Satyanarayana Narlapuram <satyanarayana.narlapuram@microsoft.com> — 2017-06-29T23:29:43Z

    -----Original Message-----
From: pgsql-hackers-owner@postgresql.org [mailto:pgsql-hackers-owner@postgresql.org] On Behalf Of Robert Haas
Sent: Thursday, June 29, 2017 5:18 AM
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Craig Ringer <craig@2ndquadrant.com>; Peter Eisentraut <peter_e@gmx.net>; Magnus Hagander <magnus@hagander.net>; PostgreSQL-development <pgsql-hackers@postgresql.org>
Subject: Re: protocol version negotiation (Re: [HACKERS] Libpq PGRES_COPY_BOTH - version compatibility)

> 1. The client sends a StartupMessage 3.x for version 3.x.  We could bump the version explicitly, or perhaps we should just coin a version of libpq for every server release; e.g. whatever PostgreSQL 11 ships is version 3.11, etc.  It includes any protocol options that don't exist today as > pg_protocol.<whatever> in the startup packet.

+1 on this. Happy to read this conversation. I am hopeful that this provides us a path to include parameters needed for Azure database for PostgreSQL service (host name, and connection id in the startupmessage). For someone wondering what they are, please see the threads below.

https://www.postgresql.org/message-id/DM2PR03MB416343FC02D6E977FEF2EB191C00%40DM2PR03MB416.namprd03.prod.outlook.com
https://www.postgresql.org/message-id/DM2PR03MB4168F3C796B2965FDC4CF9991C00%40DM2PR03MB416.namprd03.prod.outlook.com

2. If the client version is anything other than 3.0, the server responds with a ServerProtocolVersion indicating the highest version it supports, and ignores any pg_protocol.<whatever> options not known to it as being either third-party extensions or something from a future version.  If the initial response to the startup message is anything other than a ServerProtocolVersion message, the client should assume it's talking to a 3.0 server.  (To make this work, we would back-patch a change into existing releases to allow any 3.x protocol version and ignore any pg_protocol.<whatever> options that were specified.)

> If the client version is anything other than 3.0, the server responds with a ServerProtocolVersion indicating the highest version it supports, and ignores any pg_protocol.<whatever> options not known to it as being either third-party extensions or something from a future version.  If > the initial response to the startup message is anything other than a ServerProtocolVersion message, the client should assume it's talking to a 3.0 server.  (To make this work, we would back-patch a change into existing releases to allow any 3.x protocol version and ignore any             > pg_protocol.<whatever> options that were specified.)

We can avoid one round trip if the server accepts the startupmessage as is (including understanding all the parameters supplied by the client), and in the cases where server couldn’t accept the startupmessage / require negotiation, it should send ServerProtocolVersion message that contains both MIN and MAX versions it can support. Providing Min version helps server enforce the client Min protocol version, and provides a path to deprecate older versions. Thoughts?


> If either the client or the server is unhappy about the age of the other, then it can disconnect; e.g. if the server is configured to require the use of whizzbang-2 security, and the client protocol version indicates that at most whizzbang-1.9 is available, then the server can close the       > connection with a suitable complaint; conversely, if the connection string had require_whizzbang=2, and the server is too old to support that, then the client can decide to bail out when it receives the ServerProtocolVersion message.

Does the proposal also include the client can negotiate the protocol version on the same connection rather than going through connection setup process again? The state machine may not sound simple with this proposal but helps bringing down total time taken for the login.
Client / server can disconnect any time they think the negotiation failed.


Thanks,
Satya

  11. Re: protocol version negotiation (Re: Libpq PGRES_COPY_BOTH - version compatibility)

    Craig Ringer <craig@2ndquadrant.com> — 2017-06-30T02:49:30Z

    On 29 June 2017 at 20:18, Robert Haas <robertmhaas@gmail.com> wrote:

> I'm not sure if non-critical is exactly the right terminology.  What
> we want to do is distinguish between things that are intended as
> protocol-level options vs. things that are intended as GUCs.

We probably also need to be able to differentiate between "this
protocol option would be nice if you support it, please tell me in
your parameter status message if you managed to honour it" vs "this
connection request is invalid if you cannot honour this protocol
option, do not attempt to proceed".

If the option changes the meaning of the request entirely in some way
we probably don't want a server that doesn't understand carrying on
anyway.

But I think it's probably sufficient if the server's reply to the
client includes the options it was able to recognise and honour, so
the client can immediately nope out if it doesn't like the server's
lack of support for some option.

> Maybe we should instead pick a GUC namespace and reserve it for the
> use of protocol level options; e.g. pg_protocol.<anything> becomes
> invalid as the name of a GUC, but can be included in a startup packet
> (or do we need to pick something shorter, like _pg, to keep the
> message short?).  Servers ignore anything that they don't understand.
> So:

Yes, but something shorter probably, Tom's already expressed concerns
elsewhere about bandwidth use from reporting server_version_num and I
can't imagine numerous pg_protocol_blahblah entries being popular with
many people at all.

_pg_... seems sufficient IMO, we don't use that in GUCs to date, and
nobody's likely to.

Or to be consistent with protocol and libpq stuff, _pq_... ?

> 1. The client sends a StartupMessage 3.x for version 3.x.  We could
> bump the version explicitly, or perhaps we should just coin a version
> of libpq for every server release; e.g. whatever PostgreSQL 11 ships
> is version 3.11, etc.  It includes any protocol options that don't
> exist today as pg_protocol.<whatever> in the startup packet.

Not a fan of bumping for the sake of it, it puts more work on 3rd
party drivers for what seems like no real gain.

Bump when we change something.

> 2. If the client version is anything other than 3.0, the server
> responds with a ServerProtocolVersion indicating the highest version
> it supports

Highest and lowest, I think Satyanarayana Narlapuram is right there.
Also which startup parameter protocol options were recognised and
honoured, so the client can immediately notice and bail if the server
didn't recognise something it requires. (think behaviour akin to
sslmode=require for _pg_starttls).

> and ignores any pg_protocol.<whatever> options not known
> to it as being either third-party extensions or something from a
> future version.

I think ignoring is fine if the server lists the ones it recognises in
its reply, per above.

> If the initial response to the startup message is
> anything other than a ServerProtocolVersion message, the client should
> assume it's talking to a 3.0 server.  (To make this work, we would
> back-patch a change into existing releases to allow any 3.x protocol
> version and ignore any pg_protocol.<whatever> options that were
> specified.)

Such a backpatch makes sense, and is pretty safe.

-- 
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

  12. Re: protocol version negotiation (Re: Libpq PGRES_COPY_BOTH - version compatibility)

    Robert Haas <robertmhaas@gmail.com> — 2017-07-02T22:44:38Z

    On Thu, Jun 29, 2017 at 7:29 PM, Satyanarayana Narlapuram
<Satyanarayana.Narlapuram@microsoft.com> wrote:
> -----Original Message-----

The formatting of this message differs from the style normally used on
this mailing list, and is hard to read.

> 2. If the client version is anything other than 3.0, the server responds with a ServerProtocolVersion indicating the highest version it supports, and ignores any pg_protocol.<whatever> options not known to it as being either third-party extensions or something from a future version.  If the initial response to the startup message is anything other than a ServerProtocolVersion message, the client should assume it's talking to a 3.0 server.  (To make this work, we would back-patch a change into existing releases to allow any 3.x protocol version and ignore any pg_protocol.<whatever> options that were specified.)
>
> We can avoid one round trip if the server accepts the startupmessage as is (including understanding all the parameters supplied by the client), and in the cases where server couldn’t accept the startupmessage / require negotiation, it should send ServerProtocolVersion message that contains both MIN and MAX versions it can support. Providing Min version helps server enforce the client Min protocol version, and provides a path to deprecate older versions. Thoughts?

With this latest proposal, there are no extra round-trips anyway.  I
don't much see the point of having the server advertise a minimum
supported version.  The idea of new minor protocol versions is to add
*optional* features, so there shouldn't be an issue with the client
being too old to talk to the server altogether.  Of course, the server
might be configured to reject the client unless some particular new
feature is in use, but that's best handled by a message about the
specific problem at hand rather than a generic complaint.

> Does the proposal also include the client can negotiate the protocol version on the same connection rather than going through connection setup process again? The state machine may not sound simple with this proposal but helps bringing down total time taken for the login.

Nothing in that proposal involved an extra connection setup process;
if that's not clear, you might want to reread it.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company