Thread

Commits

Fix distinctness check for queries with grouping sets
- f00484c170f5 19 (unreleased) cited
Provide for forward compatibility with future minor protocol versions.
- ae65f6066dc3 11.0 cited

Proposal to allow setting cursor options on Portals

Dave Cramer <davecramer@gmail.com> — 2025-12-07T14:37:34Z

Greetings,

My main driver here is to allow the creation of Holdable portals at the
protocol level for drivers. Currently the only way to create a holdable
cursor is at the SQL level.

DECLARE liahona CURSOR WITH HOLD FOR SELECT * FROM films;


The JDBC driver has an option in the API to have result sets survive
commits see
https://docs.oracle.com/javase/8/docs/api/java/sql/Connection.html#createStatement-int-int-int-

Doing this at the protocol level is the correct way to do this as modifying
the SQL to create a cursor is very cumbersome and we already have existing
code to create a portal. Adding the ability to specify options

Looking for feedback.

Dave Cramer

Re: Proposal to allow setting cursor options on Portals

Sami Imseih <samimseih@gmail.com> — 2025-12-08T19:43:10Z

Hi,

I did not look into this patch in detail yet, but I am +1 for being
able to create cursors at the protocol level.

I think this should be allowed for regular cursors as well. One
big use-case I see is allowing postgres_fdw to create and fetch
from cursors at the protocol level rather than SQL (DECLARE
CURSOR, FETCH, etc.)

--
Sami Imseih
Amazon Web Services (AWS)

Re: Proposal to allow setting cursor options on Portals

Jelte Fennema-Nio <postgres@jeltef.nl> — 2025-12-08T21:42:56Z

On Sun, 7 Dec 2025 at 15:38, Dave Cramer <davecramer@gmail.com> wrote:
> My main driver here is to allow the creation of Holdable portals at the protocol level for drivers.

Overall seems like a sensible feature to want. A somewhat random
collection of thoughts:

1. We still have fairly limited experience with protocol options, so
afaik not everyone agrees what we should use a version bump for vs a
protocol extension.
2. I think I like the idea of optional fields that a client can add to
the existing messages. That way "implementing" the new protocol
version is a no-op for clients.
3. I think we should mark optional fields more clearly in the docs
somehow. e.g. Make the docs say <term>Optional Int32</term> and
explain what Optional means in the "Message Data Types" section.
4. I think the server should be strict that it only receives this
optional field for the expected protocol version.
5. Do we really need to add the CURSOR_BINARY flag? Seems confusing
with our other way of indicating binary support, i.e. what does it
mean to say text as the format code but then specify CURSOR_BINARY.
6. What is the benefit of PQsendQueryPreparedWithCursorOptions? I
understand the use case for PQsendBindWithCursorOptions, but not for
PQsendQueryPreparedWithCursorOptions.
7. The server should check that no unknown flags are passed
8. Docs need to be added for the new libpq function(s)

I have one question about your intended usage: I expect you intend to
make using this opt-in for the users of pgjdbc? (i.e. by using some
flag/different method to use this HOLD behaviour)

Re: Proposal to allow setting cursor options on Portals

Dave Cramer <davecramer@gmail.com> — 2025-12-08T22:07:50Z

On Mon, Dec 8, 2025 at 4:43 PM Jelte Fennema-Nio <postgres@jeltef.nl> wrote:

> On Sun, 7 Dec 2025 at 15:38, Dave Cramer <davecramer@gmail.com> wrote:
> > My main driver here is to allow the creation of Holdable portals at the
> protocol level for drivers.
>
> Overall seems like a sensible feature to want. A somewhat random
> collection of thoughts:
>
> 1. We still have fairly limited experience with protocol options, so
> afaik not everyone agrees what we should use a version bump for vs a
> protocol extension.
> 2. I think I like the idea of optional fields that a client can add to
> the existing messages. That way "implementing" the new protocol
> version is a no-op for clients.
> 3. I think we should mark optional fields more clearly in the docs
> somehow. e.g. Make the docs say <term>Optional Int32</term> and
> explain what Optional means in the "Message Data Types" section.
> 4. I think the server should be strict that it only receives this
> optional field for the expected protocol version.
> 5. Do we really need to add the CURSOR_BINARY flag? Seems confusing
> with our other way of indicating binary support, i.e. what does it
> mean to say text as the format code but then specify CURSOR_BINARY.
> 6. What is the benefit of PQsendQueryPreparedWithCursorOptions? I
> understand the use case for PQsendBindWithCursorOptions, but not for
> PQsendQueryPreparedWithCursorOptions.
> 7. The server should check that no unknown flags are passed
> 8. Docs need to be added for the new libpq function(s)
>
> I have one question about your intended usage: I expect you intend to
> make using this opt-in for the users of pgjdbc? (i.e. by using some
> flag/different method to use this HOLD behaviour)


Thx for the comments. Yes JDBC has a holdable resultset as a standard part
of the API

Dave

>
>

Re: Proposal to allow setting cursor options on Portals

Jelte Fennema-Nio <postgres@jeltef.nl> — 2025-12-09T09:08:49Z

On Mon, 8 Dec 2025 at 23:08, Dave Cramer <davecramer@gmail.com> wrote:
> Thx for the comments.

One more comment: It would be good to enable tracing[1][2] for your
test, especially because I think you still need to update the tracing
logic in libpq for your new packet type.

[1]: https://github.com/postgres/postgres/blob/f00484c170f56199c3eeacc82bd72f8c1e3baf6b/src/test/modules/libpq_pipeline/README#L29-L34
[2]: https://github.com/postgres/postgres/blob/f00484c170f56199c3eeacc82bd72f8c1e3baf6b/src/test/modules/libpq_pipeline/t/001_libpq_pipeline.pl#L39-L42

Re: Proposal to allow setting cursor options on Portals

Jacob Champion <jacob.champion@enterprisedb.com> — 2025-12-10T17:41:44Z

On Mon, Dec 8, 2025 at 1:43 PM Jelte Fennema-Nio <postgres@jeltef.nl> wrote:
> 1. We still have fairly limited experience with protocol options, so
> afaik not everyone agrees what we should use a version bump for vs a
> protocol extension.

I think it'd be helpful for proposals to describe why a minor version
bump was chosen over a protocol extension parameter (or vice versa),
so that we can begin to develop some consensus.

To me, the conversation on the wire for this feature seems perfect for
an extension parameter: "Hello server, do you support this optional
thing in this one message type? If not, let me know." Especially since
the optional thing is itself an extensible bitmap! With the
minor-version strategy, if we added new bits in 3.6, clients who just
wanted those new bits would then have to implement support for every
feature in versions 3.4, 3.5, and 3.6 just to improve that one use
case, and that incentive mismatch leads to more ossification IMO.

= Soapbox Follows =

I've talked about it face-to-face with people, but to go on the public
record: I don't think this is a wise use of a minor version upgrade
strategy. I prefer protocol architectures that introduce separate
extensions first, then periodically bundle the critical and
highly-used extensions into a new minor version once they're sure that
_everyone_ should support those things.

I know that 3.2 didn't do that. My view of 3.2 is that it was a big
compromise to get some things unstuck, so overall I'm glad we have it
-- but now that we have it, I'd rather that 3.next be more
intentional. Plus I think it's unwise to introduce a 3.3 before we're
confident that 3.2 can be widely deployed, and I'm trying to put
effort into the latter for 19, so that I'm not just sitting here
gatekeeping.

IETF has a bunch of related case studies [1,2,3] that might be useful
reading, even if we decide that their experience differs heavily from
ours.

--Jacob

[1] https://www.rfc-editor.org/rfc/rfc5218
[2] https://www.rfc-editor.org/rfc/rfc8170
[3] https://www.rfc-editor.org/rfc/rfc9170

Re: Proposal to allow setting cursor options on Portals

Jelte Fennema-Nio <postgres@jeltef.nl> — 2025-12-10T23:01:15Z

On Wed, 10 Dec 2025 at 18:41, Jacob Champion
<jacob.champion@enterprisedb.com> wrote:
> I think it'd be helpful for proposals to describe why a minor version
> bump was chosen over a protocol extension parameter (or vice versa),
> so that we can begin to develop some consensus.

Agreed.

> With the
> minor-version strategy, if we added new bits in 3.6, clients who just
> wanted those new bits would then have to implement support for every
> feature in versions 3.4, 3.5, and 3.6 just to improve that one use
> case, and that incentive mismatch leads to more ossification IMO.

I think in this optional bitmap field case, there's no work for the
client to "implement" it. It can simply request 3.3, but not send the
bitmap field. Similarly for my proposed GoAway message, a client can
simply ignore that message completely when it receives it.

If we keep the features that are bundled with a protocol version bump
of the kind where a client, either has to do nothing to implement it,
or at worst has to ignore the contents of a new message/field. Then
implementing support becomes so trivial for clients that I don't think
it'd be a hurdle for client authors to implement support for 3.3, 3.4,
3.5 and if they only wanted a feature from the 3.6 protocol.^1 I'll
call these things "no-op implementations" from now on.

> I've talked about it face-to-face with people, but to go on the public
> record: I don't think this is a wise use of a minor version upgrade
> strategy. I prefer protocol architectures that introduce separate
> extensions first, then periodically bundle the critical and
> highly-used extensions into a new minor version once they're sure that
> _everyone_ should support those things.

I think we disagree on this. I think the downside of using protocol
extensions for everything is that we then end up with N*N different
combinations of features in the wild that servers and clients need to
deal with. We have to start to define what happens when features
interact, but either of them is not enabled. With incrementing
versions you don't have that problem, which results in simpler logic
in the spec, servers and clients.

Finally, because we don't have any protocol extensions yet. All
clients still need to build infrastructure for them, including libpq.
So I'd argue that if we make such "no-op implementation" features use
protocol extensions, then it'd be more work for everyone.

> I know that 3.2 didn't do that. My view of 3.2 is that it was a big
> compromise to get some things unstuck, so overall I'm glad we have it
> -- but now that we have it, I'd rather that 3.next be more
> intentional.

> Plus I think it's unwise to introduce a 3.3 before we're
> confident that 3.2 can be widely deployed, and I'm trying to put
> effort into the latter for 19, so that I'm not just sitting here
> gatekeeping.

I'm not sure what you mean with this. People use libpq18 and PG18, and
I've heard no complaints about protocol problems. So I think it was a
success. Do you mean widely deployed by default? Why exactly does that
matter for 3.3? Anything that stands default deployment in the way for
3.2, will continue to stand default deployment in the way for 3.3.

Personally, if we flip the default in e.g. 5 years from now. I'd much
rather have it be flipped to a very nice 3.6 protocol, than still only
having the single new feature that was added in 3.2.

> IETF has a bunch of related case studies [1,2,3] that might be useful
> reading, even if we decide that their experience differs heavily from
> ours.

I gave them a skim and they seem like a good read (which I'll do
later). But I'm not sure part of them you thought was actionable for
the discussion about version bumps vs protocol extensions. (I did see
useful stuff for the grease thread, but that seems better to discuss
there)

^1: You and I only talked about clients above, but obviously there's
also proxies and other servers that implement the protocol to
consider. If a feature that is "no-op implementation" on the client is
a complicated implementation on the proxy/server then maybe a protocol
extension is indeed the better choice. I think for GoAway it's trivial
to "no-op implement" too on the proxy/server. For this cursor option
proposal it's less clear cut imo. Proxies can probably simply forward
the message to the server, although maybe PgBouncer would want to
throw an error when a client uses a hold cursor (but it also doesn't
do that for SQL level hold cursors, so that seems like an optional
enhancement). Other servers might not even support hold cursors, but
then they could simply throw a clear error (like pgbouncer would do).
If throwing an error is an acceptable server implementation, then I
think a "no-op implementation" is again trivial.

Re: Proposal to allow setting cursor options on Portals

Jacob Champion <jacob.champion@enterprisedb.com> — 2025-12-11T19:21:04Z

[I considered splitting this off into a new thread, but I think Dave
has to wait for it to be resolved before much can happen with the
patch. Sorry Dave.]

On Wed, Dec 10, 2025 at 3:01 PM Jelte Fennema-Nio <postgres@jeltef.nl> wrote:
> If we keep the features that are bundled with a protocol version bump
> of the kind where a client, either has to do nothing to implement it,
> or at worst has to ignore the contents of a new message/field. Then
> implementing support becomes so trivial for clients that I don't think
> it'd be a hurdle for client authors to implement support for 3.3, 3.4,
> 3.5 and if they only wanted a feature from the 3.6 protocol.^1 I'll
> call these things "no-op implementations" from now on.

It's too late for that, isn't it? 3.2's only feature doesn't work that
way (and couldn't have been designed that way, as far as I can tell).
So I don't have any confidence that all future features will fall in
line with this new rule.

NegotiateProtocolVersion is the only in-band tool we have to ratchet
the protocol forward. Why go through all this pain of getting NPV
packets working, only to immediately limit its power to the most
trivial cases?

> I think we disagree on this. I think the downside of using protocol
> extensions for everything is that we then end up with N*N different
> combinations of features in the wild that servers and clients need to
> deal with. We have to start to define what happens when features
> interact, but either of them is not enabled.

In the worst case? Yes. (That worst case doesn't really bother me.
Many other protocols regularly navigate extension combinations.)

But! The two extension proposals in flight at the moment -- GoAway and
cursor options -- are completely orthogonal, no? Both to each other,
and to the functionality in 3.2. There are no combinatorics yet. So it
seems strange to optimize for combinatorics out of the gate, by
burning through a client-mandatory minor version every year.

> With incrementing
> versions you don't have that problem,

You still have N*M. Implementers have to test each feature of their
3.10 client against server versions 3.0-9, rather than testing against
a single server that turns individual extension support on and off. I
prefer the latter (but maybe that's just because it's what I'm used
to). Middleboxes increase the matrix further, as you point out below.

Paradoxically, if all N features happen to be orthogonal, the testing
burden for the extension strategy collapses to... N.
Minor-version-per-year is worse for that case.

> which results in simpler logic
> in the spec, servers and clients.

I don't want to dissuade a proof of concept for this, because simpler
logic everywhere sounds amazing. But it sounds like magical thinking
to me. A bit like telling Christoph that the dpkg dependency graph is
too complicated, so it should be a straight line instead -- if that
worked, presumably everyone would have done it that way, right?
Convince me that you're not just ignoring necessary complexity in an
attempt to stamp out unnecessary complexity.

An example of an established network protocol that follows this same
strategy would be helpful. How do their clients deal with the
minor-version treadmill?

> Finally, because we don't have any protocol extensions yet. All
> clients still need to build infrastructure for them, including libpq.

For clients still on 3.0 (the vast majority of them), they'd have to
add infrastructure for sliding minor version ranges, too.

> So I'd argue that if we make such "no-op implementation" features use
> protocol extensions, then it'd be more work for everyone.

Why advertise a protocol extension if you plan to ignore it? Don't
advertise it. Do nothing. That's even less work than retrofitting
packet parsers to correctly ignore a byte range when minorversion > X.

> > Plus I think it's unwise to introduce a 3.3 before we're
> > confident that 3.2 can be widely deployed, and I'm trying to put
> > effort into the latter for 19, so that I'm not just sitting here
> > gatekeeping.
>
> I'm not sure what you mean with this. People use libpq18 and PG18, and
> I've heard no complaints about protocol problems. So I think it was a
> success. Do you mean widely deployed by default?

Yes. Or even just "deployed". GitHub shows zero hits outside of the
Postgres fork graph.

Google's results show that an organization called "cardo" tried
max_protocol_version=latest. They had to revert it. :( Time for
grease.

> Why exactly does that
> matter for 3.3? Anything that stands default deployment in the way for
> 3.2, will continue to stand default deployment in the way for 3.3.

Exactly. Don't you want to make sure that clients in the ecosystem are
able to use this _before_ we rev the version again, and again? We
don't ever get these numbers back.

Like, I'm arguing as hard as I can against the very existence of the
treadmill. But if I'm outvoted on that, *please* don't start the
treadmill before other people can climb on -- otherwise, they won't be
able to give us any feedback at all!

> Personally, if we flip the default in e.g. 5 years from now. I'd much
> rather have it be flipped to a very nice 3.6 protocol, than still only
> having the single new feature that was added in 3.2.

Those are not the only two choices. I'd rather we get a bunch of nice
features without any flipping at all, if that's possible. It looks
possible to me.

> > IETF has a bunch of related case studies [1,2,3] that might be useful
> > reading, even if we decide that their experience differs heavily from
> > ours.
>
> I gave them a skim and they seem like a good read (which I'll do
> later). But I'm not sure part of them you thought was actionable for
> the discussion about version bumps vs protocol extensions. (I did see
> useful stuff for the grease thread, but that seems better to discuss
> there)

For this conversation, I'm focused on RFC 8170. Specifically the
concepts of incremental transitions and incentive alignment
(cost/benefit to individual community members).

I view minor-version-per-year as violating both of those principles.
It instead focuses on the ease of the people who are most plugged into
this mailing list, and who have the most power to change things on a
whim.

> ^1: You and I only talked about clients above, but obviously there's
> also proxies and other servers that implement the protocol to
> consider. If a feature that is "no-op implementation" on the client is
> a complicated implementation on the proxy/server then maybe a protocol
> extension is indeed the better choice. I think for GoAway it's trivial
> to "no-op implement" too on the proxy/server. For this cursor option
> proposal it's less clear cut imo. Proxies can probably simply forward
> the message to the server, although maybe PgBouncer would want to
> throw an error when a client uses a hold cursor (but it also doesn't
> do that for SQL level hold cursors, so that seems like an optional
> enhancement).

I think proposals should attempt to answer those questions as a
prerequisite to commit, personally. Or at least, we should be moving
in that direction, if that's too harsh on the first authors who are
trying to get things moving inside the protocol.

More generally, it bothers me that we still don't have a clear mental
model of middlebox extensibility. We're just retreading the
discussions from [1] instead of starting from where we stopped, and
that's exhausting for me.

(As a reminder: 3.2 broke my testing rig, which relied on implicit
assumptions around minor-version extensibility for middleboxes. I
didn't speak up until very late, because it was just a testing rig,
and I could change it. I should have spoken up immediately, because
IIRC, pgpool then broke as well.)

> Other servers might not even support hold cursors, but
> then they could simply throw a clear error (like pgbouncer would do).
> If throwing an error is an acceptable server implementation, then I
> think a "no-op implementation" is again trivial.

A server is always free to decide at the _application_ layer that it
will error out for a particular packet that it can parse at the
_network_ layer. But it seems a lot more user-friendly to just decline
the protocol bit, if it's directly tied to an application-level
feature that isn't implemented. I think we should encourage that when
possible; otherwise we've traded protocol fragmentation for
application fragmentation.

--Jacob

[1] https://postgr.es/m/CAGECzQR5PMud4q8Atyz0gOoJ1xNH33g7g-MLXFML1_Vrhbzs6Q%40mail.gmail.com

Re: Proposal to allow setting cursor options on Portals

Jelte Fennema-Nio <postgres@jeltef.nl> — 2025-12-11T23:32:24Z

Let me start with this: I agree with you that both HOLD and GoAway
would work well as protocol extensions. And if that's what is needed
to get stuff to continue moving in the protocol space, then fine
that's what I'll do... But I have some reasons to prefer a protocol
version bump at least for GoAway

The primary reason I have is that protocol extensions are currently
enormously underspecified. Especially in regards to what their values
can be and also in regards to their own versioning and compatibility.
e.g. if we add _pq_.goaway=true. And later we want to add a field to
it, how do we do that? I can see a few options:
1. _pq_.goaway=v2,on (first acceptable value is used, this would need
immediate and constant grease)
2. _pq_.goaway=true _pq_.goaway_v2=true (have clients specify both a
new and old one and specify how the server should behave it gets both
of these)

I feel like this requires significant discussion, design and
implementation work. I tried to do that in patch 7 and 8 here[2], but
those parts of the patchset got very little review and/or feedback. I
think a big reason was that the proposal went in a too complex
direction, by trying to handle too many of the possible usecases for
the protocol extensions. I think as long as we limit the discussion to
protocol extensions that don't need to be changed on an active
connection using something like SetProtocolParameter and those
protocol extensions only have an "on/off" style value, then I think we
can make some incremental progress here. This would apply to both
GoAway (middleware should just not forward GoAway messages to clients)
and Hold (server does not send anything different for this feature).

Still this seems like quite a bit more work than "simply" including
"no-op implementation" features in a protocol bump. Especially because
I think the benefit of protocol parameters for these features is
negligible, or even negative because of the secondary reason:

The secondary reason is that I'd really like clients to actually
support the longer cancel token feature in 3.2. It's not that hard to
implement for client authors, but I don't think many users care about
it (because the primary beneficiary are server implementers, but those
only benefit if there's enough clients that implement it, so chicken
and egg). By giving people some extra goodies in 3.3 my hope is that
clients will actually implement it. So basically I agree that protocol
versions do require some additional work on client author side, but I
(selfishly) think that would be a good thing in this case. Because it
resolves this chicken and egg problem. To take advice from RFC 8170,
I'd like to align incentives better, by having protocol 3.3 contain
features that are beneficial for both client and server authors.

[2]: https://www.postgresql.org/message-id/CAGECzQRbAGqJnnJJxTdKewTsNOovUt4bsx3NFfofz3m2j-t7tA%40mail.gmail.com

-- detailed response below (things I did not respond to I agree with) --

On Thu, 11 Dec 2025 at 20:21, Jacob Champion
<jacob.champion@enterprisedb.com> wrote:
> NegotiateProtocolVersion is the only in-band tool we have to ratchet
> the protocol forward. Why go through all this pain of getting NPV
> packets working, only to immediately limit its power to the most
> trivial cases?

I think it's a fairly easy test to uphold. To be clear, I'm not saying
we should indefinitely limit that power. Eventually we'd probably want
to add things that are more difficult to implement for clients
(possibly after evaluating them as a protocol extension), but that
discussion can be punted to when we get there imo.

> So it
> seems strange to optimize for combinatorics out of the gate, by
> burning through a client-mandatory minor version every year.

To me 2 protocol extensions a year is strictly more complexity added
than 1 minor version a year. i.e. IF the changes are "no-op
implementable", why not group them together in a single identifier.

> You still have N*M. Implementers have to test each feature of their
> 3.10 client against server versions 3.0-9, rather than testing against
> a single server that turns individual extension support on and off.

I don't understand this argument. If you can have a single server
version that turns protocol extensions on and off, then why couldn't
you have a single server version that can turn different protocol
versions on and off.

> An example of an established network protocol that follows this same
> strategy would be helpful. How do their clients deal with the
> minor-version treadmill?

I agree that it would be helpful, but I'm not sure there's such a
network protocol. All protocols I know have infrequent version bumps,
which then often results in ossification. So frequent version bumps
seem like a good way to avoid that from happening. Using protocol
extension for everything might mean we ossify the protocol version
(again).

> > Finally, because we don't have any protocol extensions yet. All
> > clients still need to build infrastructure for them, including libpq.
>
> For clients still on 3.0 (the vast majority of them), they'd have to
> add infrastructure for sliding minor version ranges, too.

Yes, but adding infrastructure for both protocol versions (which we
already have now) and protocol extensions is even more work. libpq
still has no support for protocol parameters.

> Yes. Or even just "deployed". GitHub shows zero hits outside of the
> Postgres fork graph.

Yeah, that's sad, but unsurprising. Almost no-one cares about security
and that's the only end-user feature of 3.2.

> Google's results show that an organization called "cardo" tried
> max_protocol_version=latest. They had to revert it. :( Time for
> grease.

While I totally agree that we need grease, this case actually involved
people that did not update their PgBouncer version to a new enough
version that supports NegotiateProtocolVersion. [3]

[3]: https://www.cardogis.com/AenderungenIwan7#oktober-2025

> > Why exactly does that
> > matter for 3.3? Anything that stands default deployment in the way for
> > 3.2, will continue to stand default deployment in the way for 3.3.
>
> Exactly. Don't you want to make sure that clients in the ecosystem are
> able to use this _before_ we rev the version again, and again? We
> don't ever get these numbers back.

To me not every protocol version needs to be implemented by every
client. If 3.2 is never used by anyone in the wild, then half of the
world immediately switches to 3.3, and then the other half implements
3.4, then I'll be extremely happy.

> I'd rather we get a bunch of nice
> features without any flipping at all, if that's possible. It looks
> possible to me.

Me too, but I don't understand how that would work. Sending protocol
extensions is just as much of a breaking change for this ungreased
middleware as a protocol version bump. So having libpq request
_pq_.bindhold=true by default would also need some flip.

> I think proposals should attempt to answer those questions as a
> prerequisite to commit, personally. Or at least, we should be moving
> in that direction, if that's too harsh on the first authors who are
> trying to get things moving inside the protocol.

Agreed, but I don't think that has to come from the author
necessarily. I'm happy to provide that input on proposals and explain
if and why it would be hard for something like pgbouncer or other
servers.

> More generally, it bothers me that we still don't have a clear mental
> model of middlebox extensibility. We're just retreading the
> discussions from [1] instead of starting from where we stopped, and
> that's exhausting for me.

I'm still of the opinion that the requirements for [1] are good enough
for middleboxes to handle extensibility. I think those requirements
could be extended to allow GoAway too, by adding possibility 3 with
"The new message sent by the server can be dropped completely by the
middleware to imitate the lower protocol version". Remembering and
re-reading the thread and this email, it's unclear to me what your
thoughts on this are.

> (As a reminder: 3.2 broke my testing rig, which relied on implicit
> assumptions around minor-version extensibility for middleboxes. I
> didn't speak up until very late, because it was just a testing rig,
> and I could change it. I should have spoken up immediately, because
> IIRC, pgpool then broke as well.)

I'm not sure what exactly you're talking about here. You mean libpq
complaining about not receiving a BackendKeyData? If so, I agree that
wasn't a great situation. But I don't think it was related to the
current protocol being under specified, more than the new feature.

> A server is always free to decide at the _application_ layer that it
> will error out for a particular packet that it can parse at the
> _network_ layer. But it seems a lot more user-friendly to just decline
> the protocol bit, if it's directly tied to an application-level
> feature that isn't implemented. I think we should encourage that when
> possible; otherwise we've traded protocol fragmentation for
> application fragmentation.

I agree in principle, but does it really matter in practice in the
case of Hold in practice? If the network layer does not support it,
then really all that the user's application can do is throw an error.
Whether that error is thrown by the database/middleware or by the
client doesn't matter much in the end I think. The main reason where
it would matter is if the client could fall back to something else,
but in the case of HOLD that something else would probably be send
HOLD with SQL. And any server that would throw an error for protocol
based HOLD probably (should) also throw one for application level
HOLD.

> [1] https://postgr.es/m/CAGECzQR5PMud4q8Atyz0gOoJ1xNH33g7g-MLXFML1_Vrhbzs6Q%40mail.gmail.com

Re: Proposal to allow setting cursor options on Portals

Dave Cramer <davecramer@gmail.com> — 2025-12-14T12:15:22Z

On Wed, 10 Dec 2025 at 12:41, Jacob Champion <
jacob.champion@enterprisedb.com> wrote:

> On Mon, Dec 8, 2025 at 1:43 PM Jelte Fennema-Nio <postgres@jeltef.nl>
> wrote:
> > 1. We still have fairly limited experience with protocol options, so
> > afaik not everyone agrees what we should use a version bump for vs a
> > protocol extension.
>
> I think it'd be helpful for proposals to describe why a minor version
> bump was chosen over a protocol extension parameter (or vice versa),
> so that we can begin to develop some consensus.
>

 The reasons I chose a protocol bump include:
1/ I actually think this was an oversight from the original spec. I am not
adding any new features to the server, only implementing existing options
on a portal/cursor that should have been in the original protocol
2/ I'm hoping and expect that there will be other additions to the protocol
for 3.3 such as returning the LSN after commit, binary return values per
session

>
> To me, the conversation on the wire for this feature seems perfect for
> an extension parameter: "Hello server, do you support this optional
> thing in this one message type? If not, let me know." Especially since
> the optional thing is itself an extensible bitmap! With the
> minor-version strategy, if we added new bits in 3.6, clients who just
> wanted those new bits would then have to implement support for every
> feature in versions 3.4, 3.5, and 3.6 just to improve that one use
> case, and that incentive mismatch leads to more ossification IMO.
>
> = Soapbox Follows =
>
> I've talked about it face-to-face with people, but to go on the public
> record: I don't think this is a wise use of a minor version upgrade
> strategy. I prefer protocol architectures that introduce separate
> extensions first, then periodically bundle the critical and
> highly-used extensions into a new minor version once they're sure that
> _everyone_ should support those things.
>
> I know that 3.2 didn't do that. My view of 3.2 is that it was a big
> compromise to get some things unstuck, so overall I'm glad we have it
> -- but now that we have it, I'd rather that 3.next be more
> intentional. Plus I think it's unwise to introduce a 3.3 before we're
> confident that 3.2 can be widely deployed, and I'm trying to put
> effort into the latter for 19, so that I'm not just sitting here
> gatekeeping.
>
> pgjdbc already supports 3.2. Unfortunately we have no idea how many people
actually use it.


> IETF has a bunch of related case studies [1,2,3] that might be useful
> reading, even if we decide that their experience differs heavily from
> ours.
>

I read the articles which sadly gloss over protocol negotiation issues.

Dave

>
>
>

Re: Proposal to allow setting cursor options on Portals

Dave Cramer <davecramer@gmail.com> — 2025-12-14T12:31:05Z

On Thu, 11 Dec 2025 at 14:21, Jacob Champion <
jacob.champion@enterprisedb.com> wrote:

> [I considered splitting this off into a new thread, but I think Dave
> has to wait for it to be resolved before much can happen with the
> patch. Sorry Dave.]
>

No worries, I expected discussion.

>
> On Wed, Dec 10, 2025 at 3:01 PM Jelte Fennema-Nio <postgres@jeltef.nl>
> wrote:
> > If we keep the features that are bundled with a protocol version bump
> > of the kind where a client, either has to do nothing to implement it,
> > or at worst has to ignore the contents of a new message/field. Then
> > implementing support becomes so trivial for clients that I don't think
> > it'd be a hurdle for client authors to implement support for 3.3, 3.4,
> > 3.5 and if they only wanted a feature from the 3.6 protocol.^1 I'll
> > call these things "no-op implementations" from now on.
>
> It's too late for that, isn't it? 3.2's only feature doesn't work that
> way (and couldn't have been designed that way, as far as I can tell).
> So I don't have any confidence that all future features will fall in
> line with this new rule.
>
> NegotiateProtocolVersion is the only in-band tool we have to ratchet
> the protocol forward. Why go through all this pain of getting NPV
> packets working, only to immediately limit its power to the most
> trivial cases?
>
> > I think we disagree on this. I think the downside of using protocol
> > extensions for everything is that we then end up with N*N different
> > combinations of features in the wild that servers and clients need to
> > deal with. We have to start to define what happens when features
> > interact, but either of them is not enabled.
>
> In the worst case? Yes. (That worst case doesn't really bother me.
> Many other protocols regularly navigate extension combinations.)
>
> But! The two extension proposals in flight at the moment -- GoAway and
> cursor options -- are completely orthogonal, no? Both to each other,
> and to the functionality in 3.2. There are no combinatorics yet. So it
> seems strange to optimize for combinatorics out of the gate, by
> burning through a client-mandatory minor version every year.
>
> > With incrementing
> > versions you don't have that problem,
>
> You still have N*M. Implementers have to test each feature of their
> 3.10 client against server versions 3.0-9, rather than testing against
> a single server that turns individual extension support on and off. I
> prefer the latter (but maybe that's just because it's what I'm used
> to). Middleboxes increase the matrix further, as you point out below.
>

As a client author we test against multiple options all the time.  I don't
think this should be an argument against changing the protocol, otherwise
we will never change it.

>
> Paradoxically, if all N features happen to be orthogonal, the testing
> burden for the extension strategy collapses to... N.
> Minor-version-per-year is worse for that case.
>

I had never contemplated features being dependent on one another, only
additive and orthogonal.

>
> > which results in simpler logic
> > in the spec, servers and clients.
>
> I don't want to dissuade a proof of concept for this, because simpler
> logic everywhere sounds amazing. But it sounds like magical thinking
> to me. A bit like telling Christoph that the dpkg dependency graph is
> too complicated, so it should be a straight line instead -- if that
> worked, presumably everyone would have done it that way, right?
> Convince me that you're not just ignoring necessary complexity in an
> attempt to stamp out unnecessary complexity.
>
> An example of an established network protocol that follows this same
> strategy would be helpful. How do their clients deal with the
> minor-version treadmill?
>
> > Finally, because we don't have any protocol extensions yet. All
> > clients still need to build infrastructure for them, including libpq.
>
> For clients still on 3.0 (the vast majority of them), they'd have to
> add infrastructure for sliding minor version ranges, too.
>
> > So I'd argue that if we make such "no-op implementation" features use
> > protocol extensions, then it'd be more work for everyone.
>
> Why advertise a protocol extension if you plan to ignore it? Don't
> advertise it. Do nothing. That's even less work than retrofitting
> packet parsers to correctly ignore a byte range when minorversion > X.
>
> > > Plus I think it's unwise to introduce a 3.3 before we're
> > > confident that 3.2 can be widely deployed, and I'm trying to put
> > > effort into the latter for 19, so that I'm not just sitting here
> > > gatekeeping.
> >
> > I'm not sure what you mean with this. People use libpq18 and PG18, and
> > I've heard no complaints about protocol problems. So I think it was a
> > success. Do you mean widely deployed by default?
>
> Yes. Or even just "deployed". GitHub shows zero hits outside of the
> Postgres fork graph.
>

As mentioned pgjdbc supports 3.2. It was trivial to implement.

>
> Google's results show that an organization called "cardo" tried
> max_protocol_version=latest. They had to revert it. :( Time for
> grease.
>
> > Why exactly does that
> > matter for 3.3? Anything that stands default deployment in the way for
> > 3.2, will continue to stand default deployment in the way for 3.3.
>
> Exactly. Don't you want to make sure that clients in the ecosystem are
> able to use this _before_ we rev the version again, and again? We
> don't ever get these numbers back.
>

Well there are 97 of them. 1 per year is a long time.

>
> Like, I'm arguing as hard as I can against the very existence of the
> treadmill. But if I'm outvoted on that, *please* don't start the
> treadmill before other people can climb on -- otherwise, they won't be
> able to give us any feedback at all!
>
> > Personally, if we flip the default in e.g. 5 years from now. I'd much
> > rather have it be flipped to a very nice 3.6 protocol, than still only
> > having the single new feature that was added in 3.2.
>
> Those are not the only two choices. I'd rather we get a bunch of nice
> features without any flipping at all, if that's possible. It looks
> possible to me.
>
> > > IETF has a bunch of related case studies [1,2,3] that might be useful
> > > reading, even if we decide that their experience differs heavily from
> > > ours.
> >
> > I gave them a skim and they seem like a good read (which I'll do
> > later). But I'm not sure part of them you thought was actionable for
> > the discussion about version bumps vs protocol extensions. (I did see
> > useful stuff for the grease thread, but that seems better to discuss
> > there)
>
> For this conversation, I'm focused on RFC 8170. Specifically the
> concepts of incremental transitions and incentive alignment
> (cost/benefit to individual community members).
>
> I view minor-version-per-year as violating both of those principles.
> It instead focuses on the ease of the people who are most plugged into
> this mailing list, and who have the most power to change things on a
> whim.
>
> > ^1: You and I only talked about clients above, but obviously there's
> > also proxies and other servers that implement the protocol to
> > consider. If a feature that is "no-op implementation" on the client is
> > a complicated implementation on the proxy/server then maybe a protocol
> > extension is indeed the better choice. I think for GoAway it's trivial
> > to "no-op implement" too on the proxy/server. For this cursor option
> > proposal it's less clear cut imo. Proxies can probably simply forward
> > the message to the server, although maybe PgBouncer would want to
> > throw an error when a client uses a hold cursor (but it also doesn't
> > do that for SQL level hold cursors, so that seems like an optional
> > enhancement).
>

FWIW, HOLDABLE cursors are not the only option this enables. It enables all
of the other cursor options.

>
> I think proposals should attempt to answer those questions as a
> prerequisite to commit, personally. Or at least, we should be moving
> in that direction, if that's too harsh on the first authors who are
> trying to get things moving inside the protocol.
>
> More generally, it bothers me that we still don't have a clear mental
> model of middlebox extensibility. We're just retreading the
> discussions from [1] instead of starting from where we stopped, and
> that's exhausting for me.
>
> (As a reminder: 3.2 broke my testing rig, which relied on implicit
> assumptions around minor-version extensibility for middleboxes. I
> didn't speak up until very late, because it was just a testing rig,
> and I could change it. I should have spoken up immediately, because
> IIRC, pgpool then broke as well.)
>
> > Other servers might not even support hold cursors, but
> > then they could simply throw a clear error (like pgbouncer would do).
> > If throwing an error is an acceptable server implementation, then I
> > think a "no-op implementation" is again trivial.
>
> A server is always free to decide at the _application_ layer that it
> will error out for a particular packet that it can parse at the
> _network_ layer. But it seems a lot more user-friendly to just decline
> the protocol bit, if it's directly tied to an application-level
> feature that isn't implemented. I think we should encourage that when
> possible; otherwise we've traded protocol fragmentation for
> application fragmentation.
>

Are we concerned with servers that are not compatible with Postgres ?
As far as protocol fragmentation goes, I see this more as evolution to a
more complete usable implementation. I do see that we will have to be
careful with interdependent protocol options.

Dave

Re: Proposal to allow setting cursor options on Portals

Jelte Fennema-Nio <postgres@jeltef.nl> — 2025-12-14T13:41:53Z

On Sun, 14 Dec 2025 at 13:31, Dave Cramer <davecramer@gmail.com> wrote:
>> Exactly. Don't you want to make sure that clients in the ecosystem are
>> able to use this _before_ we rev the version again, and again? We
>> don't ever get these numbers back.
>
> Well there are 97 of them. 1 per year is a long time.

I don't think Jacob was concerned about the actual numbers running
out, but in case he was: it's actually 9997 versions that we still
have (9996 after we'd commit the grease proposal[1]).

[1]: https://commitfest.postgresql.org/patch/6157/

> FWIW, HOLDABLE cursors are not the only option this enables. It enables all of the other cursor options.

As mentioned upthread, I'm not sure BINARY makes sense. For any other
options, the protocol docs should specify which ones are allowed and
what their bits are. Looking at the DECLARE docs[2].
1. I think supporting ASENSITVE/INSENSITIVE/SENSITIVE bits is
unnecessary, since postgres cursors are always INSENSITIVE.
2. For SCROLL vs NO SCROLL, it would be nice if we could get rid of
the intermediate mode where if neither SCROLL or NO SCROLL is
specified, it's still SCROLL sometimes. I'm not sure backwards
compatibility would allow that, i.e. can you currently sometimes do a
BACKWARD scan on a portal created with Bind. I guess we could make it
so that if you specify the portal flags, then you have to be explicit
abuot specifying SCROLL or NO SCROLL
3. All the flags with no SQL variant probably shouldn't be
configurable through the protocol too (e.g. CURSOR_OPT_FAST_PLAN)

[2]: https://www.postgresql.org/docs/18/sql-declare.html

> Are we concerned with servers that are not compatible with Postgres ?

I think there's enough re-implementations of the postgres protocol by
other databases that it would be a shame if we didn't even try to
consider them, but I wouldn't consider it critical to get it right.
Since they can always throw application errors for features they don't
support, just like they do now for SQL that they don't support. They
can always contribute changes to clients to make using unsupported
features opt-in in the rare case where they are not.

Re: Proposal to allow setting cursor options on Portals

Dave Cramer <davecramer@gmail.com> — 2025-12-14T13:48:57Z

On Sun, 14 Dec 2025 at 08:42, Jelte Fennema-Nio <postgres@jeltef.nl> wrote:

> On Sun, 14 Dec 2025 at 13:31, Dave Cramer <davecramer@gmail.com> wrote:
> >> Exactly. Don't you want to make sure that clients in the ecosystem are
> >> able to use this _before_ we rev the version again, and again? We
> >> don't ever get these numbers back.
> >
> > Well there are 97 of them. 1 per year is a long time.
>
> I don't think Jacob was concerned about the actual numbers running
> out, but in case he was: it's actually 9997 versions that we still
> have (9996 after we'd commit the grease proposal[1]).
>
> [1]: https://commitfest.postgresql.org/patch/6157/
>
> > FWIW, HOLDABLE cursors are not the only option this enables. It enables
> all of the other cursor options.
>
> As mentioned upthread, I'm not sure BINARY makes sense. For any other
> options, the protocol docs should specify which ones are allowed and
> what their bits are. Looking at the DECLARE docs[2].
>

Here I was thinking that binary was the one that did make sense. The pgjdbc
driver would like the results back in binary, I believe others would as
well.



> 1. I think supporting ASENSITVE/INSENSITIVE/SENSITIVE bits is
> unnecessary, since postgres cursors are always INSENSITIVE.
> 2. For SCROLL vs NO SCROLL, it would be nice if we could get rid of
> the intermediate mode where if neither SCROLL or NO SCROLL is
> specified, it's still SCROLL sometimes. I'm not sure backwards
> compatibility would allow that, i.e. can you currently sometimes do a
> BACKWARD scan on a portal created with Bind. I guess we could make it
> so that if you specify the portal flags, then you have to be explicit
> abuot specifying SCROLL or NO SCROLL
> 3. All the flags with no SQL variant probably shouldn't be
> configurable through the protocol too (e.g. CURSOR_OPT_FAST_PLAN)
>
> [2]: https://www.postgresql.org/docs/18/sql-declare.html
>
> > Are we concerned with servers that are not compatible with Postgres ?
>
> I think there's enough re-implementations of the postgres protocol by
> other databases that it would be a shame if we didn't even try to
> consider them, but I wouldn't consider it critical to get it right.
> Since they can always throw application errors for features they don't
> support, just like they do now for SQL that they don't support. They
> can always contribute changes to clients to make using unsupported
> features opt-in in the rare case where they are not.
>

Fair, but from my POV, we are only concerned with Postgres. I would say
it's up to the other implementations to deal with incompatibilities.

Dave

Re: Proposal to allow setting cursor options on Portals

Jelte Fennema-Nio <postgres@jeltef.nl> — 2025-12-14T14:04:33Z

On Sun, 14 Dec 2025 at 14:49, Dave Cramer <davecramer@gmail.com> wrote:
> Here I was thinking that binary was the one that did make sense. The pgjdbc driver would like the results back in binary, I believe others would as well.

I agree drivers would like binary results back, but it's unclear to me
how CURSOR_OPT_BINARY is different from setting the result column
format codes to an array of a single 1? That should also change all
columns to be binary right?

> Fair, but from my POV, we are only concerned with Postgres. I would say it's up to the other implementations to deal with incompatibilities.

I get what you mean, but I feel like we should at least be concerned
with popular ecosystem tools like, pgbouncer and pgpool. But then it
quickly becomes an exercise in where we draw the line, what about
postgres forks like Yugabyte? Or things very similar like cockroachdb.
Both of those are distributed, and probably don't use our LSNs. So as
a concrete example, if we add LSNs to the protocol, it would be nice
to work with their version too if it's not too much effort. e.g. by
specifing a length for the commit id in the protocol instead of
forcing it at the protocol level to always be a 64bit integer.

Re: Proposal to allow setting cursor options on Portals

Dave Cramer <davecramer@gmail.com> — 2025-12-15T11:32:18Z

On Sun, 14 Dec 2025 at 09:04, Jelte Fennema-Nio <postgres@jeltef.nl> wrote:

> On Sun, 14 Dec 2025 at 14:49, Dave Cramer <davecramer@gmail.com> wrote:
> > Here I was thinking that binary was the one that did make sense. The
> pgjdbc driver would like the results back in binary, I believe others would
> as well.
>
> I agree drivers would like binary results back, but it's unclear to me
> how CURSOR_OPT_BINARY is different from setting the result column
> format codes to an array of a single 1? That should also change all
> columns to be binary right?
>

Fair point.

>
> > Fair, but from my POV, we are only concerned with Postgres. I would say
> it's up to the other implementations to deal with incompatibilities.
>
> I get what you mean, but I feel like we should at least be concerned
> with popular ecosystem tools like, pgbouncer and pgpool. But then it
> quickly becomes an exercise in where we draw the line, what about
> postgres forks like Yugabyte? Or things very similar like cockroachdb.
> Both of those are distributed, and probably don't use our LSNs. So as
> a concrete example, if we add LSNs to the protocol, it would be nice
> to work with their version too if it's not too much effort. e.g. by
> specifing a length for the commit id in the protocol instead of
> forcing it at the protocol level to always be a 64bit integer.
>

It would make sense to be forward looking here in the event that Postgres
ever has wider LSN's agreed.

Dave

Re: Proposal to allow setting cursor options on Portals

Dave Cramer <davecramer@gmail.com> — 2026-01-06T12:14:06Z

On Mon, 15 Dec 2025 at 06:32, Dave Cramer <davecramer@gmail.com> wrote:

>
>
> On Sun, 14 Dec 2025 at 09:04, Jelte Fennema-Nio <postgres@jeltef.nl>
> wrote:
>
>> On Sun, 14 Dec 2025 at 14:49, Dave Cramer <davecramer@gmail.com> wrote:
>> > Here I was thinking that binary was the one that did make sense. The
>> pgjdbc driver would like the results back in binary, I believe others would
>> as well.
>>
>> I agree drivers would like binary results back, but it's unclear to me
>> how CURSOR_OPT_BINARY is different from setting the result column
>> format codes to an array of a single 1? That should also change all
>> columns to be binary right?
>>
>
> Fair point.
>
>>
>> > Fair, but from my POV, we are only concerned with Postgres. I would say
>> it's up to the other implementations to deal with incompatibilities.
>>
>> I get what you mean, but I feel like we should at least be concerned
>> with popular ecosystem tools like, pgbouncer and pgpool. But then it
>> quickly becomes an exercise in where we draw the line, what about
>> postgres forks like Yugabyte? Or things very similar like cockroachdb.
>> Both of those are distributed, and probably don't use our LSNs. So as
>> a concrete example, if we add LSNs to the protocol, it would be nice
>> to work with their version too if it's not too much effort. e.g. by
>> specifing a length for the commit id in the protocol instead of
>> forcing it at the protocol level to always be a 64bit integer.
>>
>
> It would make sense to be forward looking here in the event that Postgres
> ever has wider LSN's agreed.
>

Any progress on this ?

Dave

>

Re: Proposal to allow setting cursor options on Portals

Jacob Champion <jacob.champion@enterprisedb.com> — 2026-01-06T16:17:04Z

On Tue, Jan 6, 2026 at 4:14 AM Dave Cramer <davecramer@gmail.com> wrote:
> Any progress on this ?

Well, I'd hoped that you and Jelte would maybe hash out your
differences in opinion a bit before I jumped back in. You think
extensions are orthogonal -- seemingly negating the primary advantage
cited for regular minor bumps? -- but Jelte is optimizing for
interrelated features.

--Jacob

Re: Proposal to allow setting cursor options on Portals

Jelte Fennema-Nio <postgres@jeltef.nl> — 2026-01-07T14:48:10Z

On Tue, 6 Jan 2026 at 17:17, Jacob Champion
<jacob.champion@enterprisedb.com> wrote:
> Well, I'd hoped that you and Jelte would maybe hash out your
> differences in opinion a bit before I jumped back in. You think
> extensions are orthogonal -- seemingly negating the primary advantage
> cited for regular minor bumps? -- but Jelte is optimizing for
> interrelated features.

I had a quick discord chat with Dave. And we don't disagree much with
each other: We both would like to use a version bump for these kinds
of very simple to implement features.

For an important part because we hope to do multiple of such small
changes in a single PG release, so the protocol can actually move
forward at a decent speed. Having a single version is only 1 option,
while having N protocol extensions a year gives at least N different
configurations (if they're all orthogonal, and at worst N*N).

In your first email you (Jacob) wrote this:
> I prefer protocol architectures that introduce separate
> extensions first, then periodically bundle the critical and
> highly-used extensions into a new minor version once they're sure that
> _everyone_ should support those things.

Dave and I both agree that if we create a protocol extension for every
tiny feature and then in 3 years include some of them in a protocol
bump, that's just a lot more complexity that every client author will
have to deal with in the long run.

Re: Proposal to allow setting cursor options on Portals

Jacob Champion <jacob.champion@enterprisedb.com> — 2026-01-08T00:39:33Z

On Wed, Jan 7, 2026 at 6:48 AM Jelte Fennema-Nio <postgres@jeltef.nl> wrote:
> I had a quick discord chat with Dave. And we don't disagree much with
> each other: We both would like to use a version bump for these kinds
> of very simple to implement features.

I asked because I'm worried that the strongest technical argument for
this strategy is "it's simpler (for us)", outweighing all
consideration of downstream consequences. I'm not really on board with
that.

Dave seems not to be particularly worried about our compatibility with
third parties. You seem to be hoping to _force_ clients to update,
even if they disagree with you that they need the new features. I
think I'm on record as saying these are both bad starting points when
making changes to a widely implemented protocol. (If not, now I am.)
That combination will burn hard-earned trust and goodwill.

> Having a single version is only 1 option,

Seems like clients must support 3.0 up to 3.N in practice, and test
all of those. If you want a feature in 3.6 and the server says it only
supports 3.4, you're speaking 3.4 now. That's still N options.

You're saying "well hopefully clients don't actually have to support
all of them," but I don't think you gave a reason why that would be
okay for a production implementation. Is there an unstated assumption
here, that we'll eventually drop support for 3.0 at some point
relatively soon? (And then 3.2, and then...) If so, I'd prefer to
focus the conversation on that assumption. Because that seems like a
complete nonstarter to me, personally.

--Jacob

Re: Proposal to allow setting cursor options on Portals

Tom Lane <tgl@sss.pgh.pa.us> — 2026-01-08T02:51:04Z

Jacob Champion <jacob.champion@enterprisedb.com> writes:
> You're saying "well hopefully clients don't actually have to support
> all of them," but I don't think you gave a reason why that would be
> okay for a production implementation. Is there an unstated assumption
> here, that we'll eventually drop support for 3.0 at some point
> relatively soon? (And then 3.2, and then...) If so, I'd prefer to
> focus the conversation on that assumption. Because that seems like a
> complete nonstarter to me, personally.

After we introduced protocol version 3.0, it took us 18 years to drop
support for version 2 (2003..2021).  We introduced 3.2 in 2025, so on
the same schedule we won't drop 3.0 support before 2043.  If anyone's
thinking that it can happen significantly faster this time around,
I'm here to disabuse you of that notion.  Considering how much larger
the Postgres ecosystem is now than it was in 2003, it will probably
take *longer* for everything to be sufficiently on-board for another
hard break.

I'm pretty bemused by this entire discussion.  We have a perfectly
good design for handling new protocol features without any hard
protocol break, so I don't understand why people are insisting on
doing things incompatibly when they could be doing them compatibly.
I quote from Robert's commit ae65f6066 (the same one that invented
NegotiateProtocolVersion):

    In addition, if the startup packet includes name/value pairs where
    the name starts with "_pq_.", assume that those are protocol options,
    not GUCs.  Include those we don't support (i.e. all of them, at
    present) in the NegotiateProtocolVersion message so that the client
    knows they were not understood.  This makes it possible for the
    client to request previously-unsupported features without bumping
    the protocol version at all; the client can tell from the server's
    response whether the option was understood.

I think that the right way forward is that the protocol version
stays at 3.2 for several decades more, and we implement requests for
individual protocol-level features through the "_pq_." mechanism.
I would not have any use for 3.2 as such at all, except that
asking for that is needed to ensure that the server knows about
NegotiateProtocolVersion.

			regards, tom lane

Re: Proposal to allow setting cursor options on Portals

Jelte Fennema-Nio <postgres@jeltef.nl> — 2026-01-08T07:38:28Z

On Thu, 8 Jan 2026 at 01:39, Jacob Champion
<jacob.champion@enterprisedb.com> wrote:
> > Having a single version is only 1 option,
>
> Seems like clients must support 3.0 up to 3.N in practice, and test
> all of those. If you want a feature in 3.6 and the server says it only
> supports 3.4, you're speaking 3.4 now. That's still N options.

I meant 1 per yearly set of protocol features. So if there's M years
with protocol features and on average N features per each of those
years the amount of different options to consider over time are:
For only version bumps: M
For completely orthogonal protocol extensions: M*N
For completely non-orthogonal options: (M*N)^2

> You're saying "well hopefully clients don't actually have to support
> all of them," but I don't think you gave a reason why that would be
> okay for a production implementation.

Clients don't have to care about every postgres version that ever
existed, or every possible proxy in existence. In 5 years, a client
author might just say: well I only care about my client being able to
connect to supported postgres servers, so if a server downgrades to
protocol 3.0 I close the connection.

This is a decision for the client author to make and for the client
author only. And they don't need to care about any of the other
clients in existence at all. Only the servers their client should be
able to connect to.

> Is there an unstated assumption
> here, that we'll eventually drop support for 3.0 at some point
> relatively soon? (And then 3.2, and then...) If so, I'd prefer to
> focus the conversation on that assumption. Because that seems like a
> complete nonstarter to me, personally.

No, that's not what I meant. Because postgres has to care about the
clients it wants to support. And that's all of them. There will
probably be clients supporting only 3.0 for a very long time. So the
postgres server will need to be supporting 3.0 for a very long time.

Re: Proposal to allow setting cursor options on Portals

Jelte Fennema-Nio <postgres@jeltef.nl> — 2026-01-08T07:44:16Z

On Thu, 8 Jan 2026 at 03:51, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> I'm pretty bemused by this entire discussion.  We have a perfectly
> good design for handling new protocol features without any hard
> protocol break, so I don't understand why people are insisting on
> doing things incompatibly when they could be doing them compatibly.

The whole point of minor protocol versions is that they don't do a
hard protocol break. 3.2 will not be a hard protocol break, 3.3 won't
be and neither will 3.4

> I quote from Robert's commit ae65f6066 (the same one that invented
> NegotiateProtocolVersion):

The section before is just as worth quoting, because it makes clear
that minor protocol versions are not a hard protocol break.

    Previously, any attempt to request a 3.x protocol version other than
    3.0 would lead to a hard connection failure, which made the minor
    protocol version really no different from the major protocol version
    and precluded gentle protocol version breaks.  Instead, when the
    client requests a 3.x protocol version where x is greater than 0, send
    the new NegotiateProtocolVersion message to convey that we support
    only 3.0.  This makes it possible to introduce new minor protocol
    versions without requiring a connection retry when the server is
    older.

> I think that the right way forward is that the protocol version
> stays at 3.2 for several decades more, and we implement requests for
> individual protocol-level features through the "_pq_." mechanism.

Is the misunderstanding of 3.3 being a hard protocol break the only
reason you think that? Or do you have some more reasons?

Re: Proposal to allow setting cursor options on Portals

Jelte Fennema-Nio <postgres@jeltef.nl> — 2026-01-08T09:45:54Z

On Thu, 8 Jan 2026 at 01:39, Jacob Champion
<jacob.champion@enterprisedb.com> wrote:
> Dave seems not to be particularly worried about our compatibility with
> third parties. You seem to be hoping to _force_ clients to update,
> even if they disagree with you that they need the new features. I
> think I'm on record as saying these are both bad starting points when
> making changes to a widely implemented protocol. (If not, now I am.)
> That combination will burn hard-earned trust and goodwill.

tl;dr I give up, let's do protocol extensions for everything. I've
updated my GoAway patch do so[1].

I don't think I can convince you that slightly more forceful push
forward that I'm suggesting is worth the gained simplicity (both for
us, users and client authors). And I'm starting to get pretty sick of
discussing the same points over and over again, without making any
progress. So instead of continuing to do so, I'll just agree to
disagree with you.

If in 5 years, when we have 15 protocol extensions with completely
distinct support across clients and proxies instead, and no-one knows
what features they can rely on in practice. While we could have had 5
new protocol versions. I'll just think (and probably tell you) "I told
you so". But you might just be right, and that won't happen, or even
if it does it will somehow be trivial to compare all the compatibility
matrices.

[1]: https://www.postgresql.org/message-id/flat/DDPQ1RV5FE9U.I2WW34NGRD8Z@jeltef.nl

Re: Proposal to allow setting cursor options on Portals

Robert Haas <robertmhaas@gmail.com> — 2026-01-08T19:01:38Z

On Thu, Jan 8, 2026 at 4:46 AM Jelte Fennema-Nio <postgres@jeltef.nl> wrote:
> tl;dr I give up, let's do protocol extensions for everything. I've
> updated my GoAway patch do so[1].
>
> I don't think I can convince you that slightly more forceful push
> forward that I'm suggesting is worth the gained simplicity (both for
> us, users and client authors). And I'm starting to get pretty sick of
> discussing the same points over and over again, without making any
> progress. So instead of continuing to do so, I'll just agree to
> disagree with you.

That sounds like the right approach to me. Note that I have also
previously expressed my disagreement with the idea of bumping the
protocol version regularly. I'm not entirely comfortable with the idea
of using protocol extensions for everything, because I really imagined
that they would be used for larger features that made a cluster of
related changes rather than solitary changes, and that there wouldn't
be many of them. If we have lots of them and use them for solitary
changes like GoAway, we will eventually end up with a very long list
of protocol extensions. I agree with you that this is not great. On
the other hand, I also do not think the alternative of bumping the
protocol version every year is viable. And even if I did, working in
this community means that you sometimes have to defer to a consensus
with which you do not personally agree in the interest of getting
stuff done. I find that when 2 or 3 people all disagree with the same
decision that I've made, it is usually a sign that I am wrong,
regardless of how sure I am that I am not wrong.

> If in 5 years, when we have 15 protocol extensions with completely
> distinct support across clients and proxies instead, and no-one knows
> what features they can rely on in practice. While we could have had 5
> new protocol versions. I'll just think (and probably tell you) "I told
> you so". But you might just be right, and that won't happen, or even
> if it does it will somehow be trivial to compare all the compatibility
> matrices.

IMHO, one thing upon which this greatly depends is the degree to which
all the changes are interdependent. If we end up with an extension for
a GoAway message, an extension for column encryption, and an extension
for trace IDs, I don't see why the compatibility matrix should be a
huge issue. Like, yes, some 3rd-party implementations will support all
of those and some will support only some of them, but that's sort of
the point. Our own code should work with any combination -- I hope --
because the code should live in pretty separate places. If we end up
with extensions for column encryption, column use-of-binary format,
and column encoding, well then it's probably going to be quite a mess
to keep our stuff working with all combinations. In that case, we need
to either bump the version and mandate that you have to support all of
them, or have extremely good testing frameworks, or maybe decide not
to add all those features.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

Re: Proposal to allow setting cursor options on Portals

Tom Lane <tgl@sss.pgh.pa.us> — 2026-01-08T19:22:50Z

Robert Haas <robertmhaas@gmail.com> writes:
> That sounds like the right approach to me. Note that I have also
> previously expressed my disagreement with the idea of bumping the
> protocol version regularly. I'm not entirely comfortable with the idea
> of using protocol extensions for everything, because I really imagined
> that they would be used for larger features that made a cluster of
> related changes rather than solitary changes, and that there wouldn't
> be many of them.

I kind of doubt that there will ever be many of them, but if we start
to feel like there's a lot, we could invent abbreviations: single
feature names that clients can ask for that are defined to represent
a particular set of older features.  But I'd argue that those sets
should be groups of related functions, not "whatever random stuff
exists as of Postgres 27".  I think it'll be highly useful for clients
to declare which features they want, rather than leave people
wondering exactly which features this client intends to support.

			regards, tom lane

Re: Proposal to allow setting cursor options on Portals

Dave Cramer <davecramer@gmail.com> — 2026-01-09T18:20:34Z

On Thu, 8 Jan 2026 at 14:22, Tom Lane <tgl@sss.pgh.pa.us> wrote:

> Robert Haas <robertmhaas@gmail.com> writes:
> > That sounds like the right approach to me. Note that I have also
> > previously expressed my disagreement with the idea of bumping the
> > protocol version regularly. I'm not entirely comfortable with the idea
> > of using protocol extensions for everything, because I really imagined
> > that they would be used for larger features that made a cluster of
> > related changes rather than solitary changes, and that there wouldn't
> > be many of them.
>
> I kind of doubt that there will ever be many of them, but if we start
> to feel like there's a lot, we could invent abbreviations: single
> feature names that clients can ask for that are defined to represent
> a particular set of older features.  But I'd argue that those sets
> should be groups of related functions, not "whatever random stuff
> exists as of Postgres 27".  I think it'll be highly useful for clients
> to declare which features they want, rather than leave people
> wondering exactly which features this client intends to support.
>
>                         regards, tom lane
>

For the particular case of adding the ability to create holdable cursors at
the protocol level which is what my patch is concerned with I don't think
it's even necessary to bump the protocol or create a protocol extension.
The change in the message is backward compatible and clients only need to
know that after version 18 they can use the message to create a holdable
cursor. I would argue that this patch is just rectifying an oversight in
the original protocol.

Dave

Re: Proposal to allow setting cursor options on Portals

Robert Haas <robertmhaas@gmail.com> — 2026-01-14T19:24:09Z

On Fri, Jan 9, 2026 at 1:20 PM Dave Cramer <davecramer@gmail.com> wrote:
> For the particular case of adding the ability to create holdable cursors at the protocol level which is what my patch is concerned with I don't think it's even necessary to bump the protocol or create a protocol extension.
> The change in the message is backward compatible and clients only need to know that after version 18 they can use the message to create a holdable cursor. I would argue that this patch is just rectifying an oversight in the original protocol.

I'm not sure what I think about the patch itself, but I find myself
somewhat in agreement with this logic. If the server is supposed to
start sending something different to the client, or the client must
send something different to the server, that's clearly got to be
negotiated. But I wonder whether we should just consider slipping
things like this into the protocol without bumping the version at all.
That makes me a bit nervous because it kind of makes a mockery of the
idea of a version number, but it's also not entirely without
precedent. For example, COPY BOTH mode didn't use to exist, and now it
does, and the protocol version number didn't change in the process.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

Re: Proposal to allow setting cursor options on Portals

Jelte Fennema-Nio <postgres@jeltef.nl> — 2026-01-14T23:12:03Z

On Wed, 14 Jan 2026 at 20:24, Robert Haas <robertmhaas@gmail.com> wrote:
> But I wonder whether we should just consider slipping
> things like this into the protocol without bumping the version at all.

I feel like I've said this many times already, but I really do not
understand why there's such a hesitation on bumping the minor protocol
version. Bumping the minor protocol version has zero downsides to me.
IMO we could bump it every PG release even if we don't make any
changes to the protocol. Nothing would break. The only thing that
breaks things is requesting anything other than 3.0, because most
proxies haven't implemented NegotiateProtocolVersion. But hopefully
that will be a concern of the past once Jacob merges the grease
stuff[1]. After that it doesn't matter what minor version is
requested, if it's 3.2, 3.3, or 3.456. Requesting any of those will
not cause breakage.

> That makes me a bit nervous because it kind of makes a mockery of the
> idea of a version number, but it's also not entirely without
> precedent. For example, COPY BOTH mode didn't use to exist, and now it
> does, and the protocol version number didn't change in the process.

I think CopyBoth is very special, because it's only used in logical
replication. In a sense it's practically a protocol extension, because
it is only sent by the server if the client explicitly requested it
through a keyword in the StartupMessage (replication=database).

As a maintainer of a proxy, I'd be pretty annoyed if clients start
sending fields or messages that I don't expect, without *something* in
the StartupMessage advertising that that will happen. This specific
patch would be fine for pgbouncer, because it only parses the first
few fields of the Bind message. But e.g pgcat seems to parse the whole
Bind message[2]

[1]: https://www.postgresql.org/message-id/flat/DDPR5BPWH1RJ.1LWAK6QAURVAY@jeltef.nl
[2]: https://github.com/postgresml/pgcat/blob/5b038813eb14f181434ab7b5509e74d9b1fe123b/src/messages.rs#L1019-L1064

P.S. This *really* is my final on-list message on the topic of bumping
protocol version numbers . Off-list I'm happy to explain to people and
maybe I'll reconsider re-opening the discussion in a few years. But
the current discussion is only going in circles, and I'd rather spend
my energy on something that is useful.

Re: Proposal to allow setting cursor options on Portals

Tom Lane <tgl@sss.pgh.pa.us> — 2026-01-14T23:30:14Z

Jelte Fennema-Nio <postgres@jeltef.nl> writes:
> I feel like I've said this many times already, but I really do not
> understand why there's such a hesitation on bumping the minor protocol
> version. Bumping the minor protocol version has zero downsides to me.

I think you have that backwards.  The right way to think about it
is that bumping the minor version has zero upside.  What we actually
want is for the client and server to agree on what specific optional
features they will use, and we have a design that allows doing that
in a fine-grained, extensible way.  We don't need to change the
protocol version number ever again, as long as we use protocol
options correctly.

Having said that, I share Robert's distaste for "silent" protocol
bumps that change the behavior without any negotiation.

			regards, tom lane

Re: Proposal to allow setting cursor options on Portals

Jacob Champion <jacob.champion@enterprisedb.com> — 2026-01-15T00:42:00Z

On Wed, Jan 14, 2026 at 3:12 PM Jelte Fennema-Nio <postgres@jeltef.nl> wrote:
> As a maintainer of a proxy, I'd be pretty annoyed if clients start
> sending fields or messages that I don't expect, without *something* in
> the StartupMessage advertising that that will happen. This specific
> patch would be fine for pgbouncer, because it only parses the first
> few fields of the Bind message. But e.g pgcat seems to parse the whole
> Bind message[2]

Yeah... I'm pretty skeptical of the phrase "backwards compatible" in
the way you've used it, Dave. I think that assumes either that
everyone's packet parsers are incredibly lenient, or else that `ERROR:
invalid message format` should be considered a compatible response.

(This ties into my upthread concern that we don't have a good shared
mental model for middlebox compatibility in the protocol.)

--Jacob

Re: Proposal to allow setting cursor options on Portals

Dave Cramer <davecramer@gmail.com> — 2026-01-15T10:33:11Z

Dave Cramer


On Wed, 14 Jan 2026 at 18:30, Tom Lane <tgl@sss.pgh.pa.us> wrote:

> Jelte Fennema-Nio <postgres@jeltef.nl> writes:
> > I feel like I've said this many times already, but I really do not
> > understand why there's such a hesitation on bumping the minor protocol
> > version. Bumping the minor protocol version has zero downsides to me.
>
> I think you have that backwards.  The right way to think about it
> is that bumping the minor version has zero upside.  What we actually
> want is for the client and server to agree on what specific optional
> features they will use, and we have a design that allows doing that
> in a fine-grained, extensible way.  We don't need to change the
> protocol version number ever again, as long as we use protocol
> options correctly.
>

I would argue in the case of "cursor with hold" this should have been in
the original protocol.
This is not an added feature this just enables an existing feature in the
server. This is not unlike widening the cancel key.
Something like encryption would be a feature that I could see using the
extension mechanism

>
> Having said that, I share Robert's distaste for "silent" protocol
> bumps that change the behavior without any negotiation.
>
My understanding reading his message he was in favour of it

As for proxies or "middleboxes" I will concede that not advertising that we
are going to change that message is a non-starter

Dave

Re: Proposal to allow setting cursor options on Portals

Robert Haas <robertmhaas@gmail.com> — 2026-01-15T19:00:00Z

On Thu, Jan 15, 2026 at 5:33 AM Dave Cramer <davecramer@gmail.com> wrote:
> I would argue in the case of "cursor with hold" this should have been in the original protocol.
> This is not an added feature this just enables an existing feature in the server. This is not unlike widening the cancel key.
> Something like encryption would be a feature that I could see using the extension mechanism

I agree that encryption is a better fit for the extension mechanism.
But I don't really agree that this isn't a new feature. Exposing
existing functionality in new ways counts as a new feature in my book.

>> Having said that, I share Robert's distaste for "silent" protocol
>> bumps that change the behavior without any negotiation.
>
> My understanding reading his message he was in favour of it

Well, my feelings are mixed on that point, honestly. If the server
needs to know whether the client supports something, you pretty much
have to have a protocol negotiation of some kind, whether that's a
version bump or an extension. Otherwise, the server simply has no way
to find out what the client supports. In the opposite direction, it's
more fuzzy: the client can see the server version number, and so can
draw some conclusions on that basis. We have used this for protocol
changes in the past. However, something like changing the cancel key
or adding cursor with hold affects a lot more clients than adding
CopyBoth that is only triggered by a very specific command in a
special mode. And it's not that theoretically appealing to conflate
the *protocol* version number with the *server* version number.
Sometimes things that are not theoretically appealing are nonetheless
pragmatically appealing. But it is not really clear to me whether this
is one of those cases, and it sounds like Tom and Jacob think it
isn't. I could probably be persuaded either way on the best thing to
do here in a vacuum, but I'm strongly disinclined to argue against two
committers who have a firm position on the topic already.

I think what I like least about this proposal is the feeling that
we're about to embark on a long slippery slope of changing the
protocol a little bit in every new PG version. The cancel key thing is
a small change, look here's another. If we just keep doing that, we'll
end up with either a lot of minor version bumps or a lot of
extensions. I don't foresee a good outcome either way. This is a
widely used, widely adopted protocol. The idea that we can just start
tweaking it a little bit every year and have nothing bad happened
seems wild, regardless of how we do the tweaking.

> As for proxies or "middleboxes" I will concede that not advertising that we are going to change that message is a non-starter

Yeah.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

Re: Proposal to allow setting cursor options on Portals

Dave Cramer <davecramer@gmail.com> — 2026-01-15T19:11:22Z

On Thu, 15 Jan 2026 at 14:00, Robert Haas <robertmhaas@gmail.com> wrote:

> On Thu, Jan 15, 2026 at 5:33 AM Dave Cramer <davecramer@gmail.com> wrote:
> > I would argue in the case of "cursor with hold" this should have been in
> the original protocol.
> > This is not an added feature this just enables an existing feature in
> the server. This is not unlike widening the cancel key.
> > Something like encryption would be a feature that I could see using the
> extension mechanism
>
> I agree that encryption is a better fit for the extension mechanism.
> But I don't really agree that this isn't a new feature. Exposing
> existing functionality in new ways counts as a new feature in my book.
>
> >> Having said that, I share Robert's distaste for "silent" protocol
> >> bumps that change the behavior without any negotiation.
> >
> > My understanding reading his message he was in favour of it
>
> Well, my feelings are mixed on that point, honestly. If the server
> needs to know whether the client supports something, you pretty much
> have to have a protocol negotiation of some kind, whether that's a
> version bump or an extension. Otherwise, the server simply has no way
> to find out what the client supports. In the opposite direction, it's
> more fuzzy: the client can see the server version number, and so can
> draw some conclusions on that basis. We have used this for protocol
> changes in the past. However, something like changing the cancel key
> or adding cursor with hold affects a lot more clients than adding
> CopyBoth that is only triggered by a very specific command in a
> special mode. And it's not that theoretically appealing to conflate
> the *protocol* version number with the *server* version number.
> Sometimes things that are not theoretically appealing are nonetheless
> pragmatically appealing. But it is not really clear to me whether this
> is one of those cases, and it sounds like Tom and Jacob think it
> isn't. I could probably be persuaded either way on the best thing to
> do here in a vacuum, but I'm strongly disinclined to argue against two
> committers who have a firm position on the topic already.
>
> I think what I like least about this proposal is the feeling that
> we're about to embark on a long slippery slope of changing the
> protocol a little bit in every new PG version. The cancel key thing is
> a small change, look here's another. If we just keep doing that, we'll
> end up with either a lot of minor version bumps or a lot of
> extensions. I don't foresee a good outcome either way. This is a
> widely used, widely adopted protocol. The idea that we can just start
> tweaking it a little bit every year and have nothing bad happened
> seems wild, regardless of how we do the tweaking.


This leaves us with an all or nothing solution, which practically means we
do nothing, since we have to wait until we have a sufficient backlog of
changes or features to change the protocol. I see that as untenable, unless
you are now advocating for using extensions for everything ?

Dave

Re: Proposal to allow setting cursor options on Portals

Hannu Krosing <hannuk@google.com> — 2026-01-15T21:06:23Z

First, let me say that I very much support getting this into the wire protocol.

As for ways to extend the protocol, I have been myself working on
another patch + extension where one can return extra info in
ReadyForQuery message

The first things to add are
* CommitLSN so we can make use of ability to WAIT FOR LSN on replica
and two connection-pooling helpers
* a flag telling that there are HOLD CURSORS
* a flag telling that there are temp tables

This extra feedback is enabled by setting a flag, so no flag --
nothing to confuse the client.
The extras themselves are uniform (length, tag, data) so client can
ignore any tag they do not recognize.

On Thu, Jan 15, 2026 at 8:11 PM Dave Cramer <davecramer@gmail.com> wrote:
>
>
> On Thu, 15 Jan 2026 at 14:00, Robert Haas <robertmhaas@gmail.com> wrote:
>>
...
>> I think what I like least about this proposal is the feeling that
>> we're about to embark on a long slippery slope of changing the
>> protocol a little bit in every new PG version. The cancel key thing is
>> a small change, look here's another. If we just keep doing that, we'll
>> end up with either a lot of minor version bumps or a lot of
>> extensions. I don't foresee a good outcome either way. This is a
>> widely used, widely adopted protocol. The idea that we can just start
>> tweaking it a little bit every year and have nothing bad happened
>> seems wild, regardless of how we do the tweaking.

I think "tweaking ait  little bit" and only whhere needed is exactly
the right approach, if it can be cleanly isolated.

My approach to protocol extension modulation *is* the ability to
enable different parts of the protocol individually.

For example the protocol extension to allow cursor/portal flags could
be implemented this way

Client has to set a flag to PROTOCOL_PORTAL_OPTIONS=true to tell the
server that new protocol messages are coming
- If flag setting fails, client will not send the new protocol messages
- If flag setting succeeds, then it is ok to send the new messages
corresponding to the flag.

This way the extra packets are disconnected from protocol version and
can be enabled/disabled individually and per connection

And it also lets one cleanly backport the change as needed without
affecting anything else.

> This leaves us with an all or nothing solution, which practically means we do nothing, since we have to wait until we have a sufficient backlog of
> changes or features to change the protocol. I see that as untenable, unless you are now advocating for using extensions for everything ?
>
> Dave

Re: Proposal to allow setting cursor options on Portals

Dave Cramer <davecramer@gmail.com> — 2026-03-04T14:26:05Z

On Thu, 15 Jan 2026 at 16:06, Hannu Krosing <hannuk@google.com> wrote:

> First, let me say that I very much support getting this into the wire
> protocol.
>
> As for ways to extend the protocol, I have been myself working on
> another patch + extension where one can return extra info in
> ReadyForQuery message
>
> The first things to add are
> * CommitLSN so we can make use of ability to WAIT FOR LSN on replica
> and two connection-pooling helpers
> * a flag telling that there are HOLD CURSORS
> * a flag telling that there are temp tables
>
> This extra feedback is enabled by setting a flag, so no flag --
> nothing to confuse the client.
> The extras themselves are uniform (length, tag, data) so client can
> ignore any tag they do not recognize.
>
> On Thu, Jan 15, 2026 at 8:11 PM Dave Cramer <davecramer@gmail.com> wrote:
> >
> >
> > On Thu, 15 Jan 2026 at 14:00, Robert Haas <robertmhaas@gmail.com> wrote:
> >>
> ...
> >> I think what I like least about this proposal is the feeling that
> >> we're about to embark on a long slippery slope of changing the
> >> protocol a little bit in every new PG version. The cancel key thing is
> >> a small change, look here's another. If we just keep doing that, we'll
> >> end up with either a lot of minor version bumps or a lot of
> >> extensions. I don't foresee a good outcome either way. This is a
> >> widely used, widely adopted protocol. The idea that we can just start
> >> tweaking it a little bit every year and have nothing bad happened
> >> seems wild, regardless of how we do the tweaking.
>
> I think "tweaking ait  little bit" and only whhere needed is exactly
> the right approach, if it can be cleanly isolated.
>
> My approach to protocol extension modulation *is* the ability to
> enable different parts of the protocol individually.
>
> For example the protocol extension to allow cursor/portal flags could
> be implemented this way
>
> Client has to set a flag to PROTOCOL_PORTAL_OPTIONS=true to tell the
> server that new protocol messages are coming
> - If flag setting fails, client will not send the new protocol messages
> - If flag setting succeeds, then it is ok to send the new messages
> corresponding to the flag.
>
> This way the extra packets are disconnected from protocol version and
> can be enabled/disabled individually and per connection
>
> And it also lets one cleanly backport the change as needed without
> affecting anything else.
>
> > This leaves us with an all or nothing solution, which practically means
> we do nothing, since we have to wait until we have a sufficient backlog of
> > changes or features to change the protocol. I see that as untenable,
> unless you are now advocating for using extensions for everything ?
> >
> > Dave
>

I have modified the patch to use protocol options instead of protocol
version

See new version attached

Re: Proposal to allow setting cursor options on Portals

Dave Cramer <davecramer@gmail.com> — 2026-03-04T15:35:48Z

On Wed, 4 Mar 2026 at 09:26, Dave Cramer <davecramer@gmail.com> wrote:

>
>
> On Thu, 15 Jan 2026 at 16:06, Hannu Krosing <hannuk@google.com> wrote:
>
>> First, let me say that I very much support getting this into the wire
>> protocol.
>>
>> As for ways to extend the protocol, I have been myself working on
>> another patch + extension where one can return extra info in
>> ReadyForQuery message
>>
>> The first things to add are
>> * CommitLSN so we can make use of ability to WAIT FOR LSN on replica
>> and two connection-pooling helpers
>> * a flag telling that there are HOLD CURSORS
>> * a flag telling that there are temp tables
>>
>> This extra feedback is enabled by setting a flag, so no flag --
>> nothing to confuse the client.
>> The extras themselves are uniform (length, tag, data) so client can
>> ignore any tag they do not recognize.
>>
>> On Thu, Jan 15, 2026 at 8:11 PM Dave Cramer <davecramer@gmail.com> wrote:
>> >
>> >
>> > On Thu, 15 Jan 2026 at 14:00, Robert Haas <robertmhaas@gmail.com>
>> wrote:
>> >>
>> ...
>> >> I think what I like least about this proposal is the feeling that
>> >> we're about to embark on a long slippery slope of changing the
>> >> protocol a little bit in every new PG version. The cancel key thing is
>> >> a small change, look here's another. If we just keep doing that, we'll
>> >> end up with either a lot of minor version bumps or a lot of
>> >> extensions. I don't foresee a good outcome either way. This is a
>> >> widely used, widely adopted protocol. The idea that we can just start
>> >> tweaking it a little bit every year and have nothing bad happened
>> >> seems wild, regardless of how we do the tweaking.
>>
>> I think "tweaking ait  little bit" and only whhere needed is exactly
>> the right approach, if it can be cleanly isolated.
>>
>> My approach to protocol extension modulation *is* the ability to
>> enable different parts of the protocol individually.
>>
>> For example the protocol extension to allow cursor/portal flags could
>> be implemented this way
>>
>> Client has to set a flag to PROTOCOL_PORTAL_OPTIONS=true to tell the
>> server that new protocol messages are coming
>> - If flag setting fails, client will not send the new protocol messages
>> - If flag setting succeeds, then it is ok to send the new messages
>> corresponding to the flag.
>>
>> This way the extra packets are disconnected from protocol version and
>> can be enabled/disabled individually and per connection
>>
>> And it also lets one cleanly backport the change as needed without
>> affecting anything else.
>>
>> > This leaves us with an all or nothing solution, which practically means
>> we do nothing, since we have to wait until we have a sufficient backlog of
>> > changes or features to change the protocol. I see that as untenable,
>> unless you are now advocating for using extensions for everything ?
>> >
>> > Dave
>>
>
> I have modified the patch to use protocol options instead of protocol
> version
>

resending with a different patch name as I think the commitfest app won't
pick it up with the same name


>
> See new version attached
>

Re: Proposal to allow setting cursor options on Portals

Patrick Reinhart <patrick.reinhart@gmail.com> — 2026-03-09T18:43:07Z

Hi Dave,

The Company I working is in the progress to migrate a lot of databases 
from Oracle to PostreSQL lately and have the need currently to hold 
cursors over commits at the moment.

This does lead to nasty Java out-of-memory for big tables as in this 
case the specified fetch size is been ignored.

There was also some thread around this issue here if I'm not mistaken:

https://www.postgresql.org/message-id/CAMT0RQSpqcvjHy3tam%2BnuKhoDH7XWBVy2mOp8J%3Df0dH30Rukew%40mail.gmail.com

I could also offer some type of I-the-field test as soon there is a way 
to setup the needed environment locally...

Best regards

Patrick

Am 07.12.25 um 15:37 schrieb Dave Cramer:
> Greetings,
>
> My main driver here is to allow the creation of Holdable portals at 
> the protocol level for drivers. Currently the only way to create a 
> holdable cursor is at the SQL level.
>
> DECLARE liahona CURSOR WITH HOLD FOR SELECT * FROM films;
>
> The JDBC driver has an option in the API to have result sets survive 
> commits see 
> https://docs.oracle.com/javase/8/docs/api/java/sql/Connection.html#createStatement-int-int-int-
>
> Doing this at the protocol level is the correct way to do this as 
> modifying the SQL to create a cursor is very cumbersome and we already 
> have existing code to create a portal. Adding the ability to specify 
> options
>
> Looking for feedback.
>
> Dave Cramer

Re: Proposal to allow setting cursor options on Portals

Dave Cramer <davecramer@gmail.com> — 2026-03-17T14:41:41Z

Patch rebased, now ready for review


Dave Cramer


On Wed, 4 Mar 2026 at 10:35, Dave Cramer <davecramer@gmail.com> wrote:

>
>
> On Wed, 4 Mar 2026 at 09:26, Dave Cramer <davecramer@gmail.com> wrote:
>
>>
>>
>> On Thu, 15 Jan 2026 at 16:06, Hannu Krosing <hannuk@google.com> wrote:
>>
>>> First, let me say that I very much support getting this into the wire
>>> protocol.
>>>
>>> As for ways to extend the protocol, I have been myself working on
>>> another patch + extension where one can return extra info in
>>> ReadyForQuery message
>>>
>>> The first things to add are
>>> * CommitLSN so we can make use of ability to WAIT FOR LSN on replica
>>> and two connection-pooling helpers
>>> * a flag telling that there are HOLD CURSORS
>>> * a flag telling that there are temp tables
>>>
>>> This extra feedback is enabled by setting a flag, so no flag --
>>> nothing to confuse the client.
>>> The extras themselves are uniform (length, tag, data) so client can
>>> ignore any tag they do not recognize.
>>>
>>> On Thu, Jan 15, 2026 at 8:11 PM Dave Cramer <davecramer@gmail.com>
>>> wrote:
>>> >
>>> >
>>> > On Thu, 15 Jan 2026 at 14:00, Robert Haas <robertmhaas@gmail.com>
>>> wrote:
>>> >>
>>> ...
>>> >> I think what I like least about this proposal is the feeling that
>>> >> we're about to embark on a long slippery slope of changing the
>>> >> protocol a little bit in every new PG version. The cancel key thing is
>>> >> a small change, look here's another. If we just keep doing that, we'll
>>> >> end up with either a lot of minor version bumps or a lot of
>>> >> extensions. I don't foresee a good outcome either way. This is a
>>> >> widely used, widely adopted protocol. The idea that we can just start
>>> >> tweaking it a little bit every year and have nothing bad happened
>>> >> seems wild, regardless of how we do the tweaking.
>>>
>>> I think "tweaking ait  little bit" and only whhere needed is exactly
>>> the right approach, if it can be cleanly isolated.
>>>
>>> My approach to protocol extension modulation *is* the ability to
>>> enable different parts of the protocol individually.
>>>
>>> For example the protocol extension to allow cursor/portal flags could
>>> be implemented this way
>>>
>>> Client has to set a flag to PROTOCOL_PORTAL_OPTIONS=true to tell the
>>> server that new protocol messages are coming
>>> - If flag setting fails, client will not send the new protocol messages
>>> - If flag setting succeeds, then it is ok to send the new messages
>>> corresponding to the flag.
>>>
>>> This way the extra packets are disconnected from protocol version and
>>> can be enabled/disabled individually and per connection
>>>
>>> And it also lets one cleanly backport the change as needed without
>>> affecting anything else.
>>>
>>> > This leaves us with an all or nothing solution, which practically
>>> means we do nothing, since we have to wait until we have a sufficient
>>> backlog of
>>> > changes or features to change the protocol. I see that as untenable,
>>> unless you are now advocating for using extensions for everything ?
>>> >
>>> > Dave
>>>
>>
>> I have modified the patch to use protocol options instead of protocol
>> version
>>
>
> resending with a different patch name as I think the commitfest app won't
> pick it up with the same name
>
>
>>
>> See new version attached
>>
>

Re: Proposal to allow setting cursor options on Portals

Dave Cramer <davecramer@gmail.com> — 2026-03-18T18:00:25Z

On Tue, 17 Mar 2026 at 10:41, Dave Cramer <davecramer@gmail.com> wrote:

> Patch rebased, now ready for review
>
>
> Dave Cramer
>
>
> On Wed, 4 Mar 2026 at 10:35, Dave Cramer <davecramer@gmail.com> wrote:
>
>>
>>
>> On Wed, 4 Mar 2026 at 09:26, Dave Cramer <davecramer@gmail.com> wrote:
>>
>>>
>>>
>>> On Thu, 15 Jan 2026 at 16:06, Hannu Krosing <hannuk@google.com> wrote:
>>>
>>>> First, let me say that I very much support getting this into the wire
>>>> protocol.
>>>>
>>>> As for ways to extend the protocol, I have been myself working on
>>>> another patch + extension where one can return extra info in
>>>> ReadyForQuery message
>>>>
>>>> The first things to add are
>>>> * CommitLSN so we can make use of ability to WAIT FOR LSN on replica
>>>> and two connection-pooling helpers
>>>> * a flag telling that there are HOLD CURSORS
>>>> * a flag telling that there are temp tables
>>>>
>>>> This extra feedback is enabled by setting a flag, so no flag --
>>>> nothing to confuse the client.
>>>> The extras themselves are uniform (length, tag, data) so client can
>>>> ignore any tag they do not recognize.
>>>>
>>>> On Thu, Jan 15, 2026 at 8:11 PM Dave Cramer <davecramer@gmail.com>
>>>> wrote:
>>>> >
>>>> >
>>>> > On Thu, 15 Jan 2026 at 14:00, Robert Haas <robertmhaas@gmail.com>
>>>> wrote:
>>>> >>
>>>> ...
>>>> >> I think what I like least about this proposal is the feeling that
>>>> >> we're about to embark on a long slippery slope of changing the
>>>> >> protocol a little bit in every new PG version. The cancel key thing
>>>> is
>>>> >> a small change, look here's another. If we just keep doing that,
>>>> we'll
>>>> >> end up with either a lot of minor version bumps or a lot of
>>>> >> extensions. I don't foresee a good outcome either way. This is a
>>>> >> widely used, widely adopted protocol. The idea that we can just start
>>>> >> tweaking it a little bit every year and have nothing bad happened
>>>> >> seems wild, regardless of how we do the tweaking.
>>>>
>>>> I think "tweaking ait  little bit" and only whhere needed is exactly
>>>> the right approach, if it can be cleanly isolated.
>>>>
>>>> My approach to protocol extension modulation *is* the ability to
>>>> enable different parts of the protocol individually.
>>>>
>>>> For example the protocol extension to allow cursor/portal flags could
>>>> be implemented this way
>>>>
>>>> Client has to set a flag to PROTOCOL_PORTAL_OPTIONS=true to tell the
>>>> server that new protocol messages are coming
>>>> - If flag setting fails, client will not send the new protocol messages
>>>> - If flag setting succeeds, then it is ok to send the new messages
>>>> corresponding to the flag.
>>>>
>>>> This way the extra packets are disconnected from protocol version and
>>>> can be enabled/disabled individually and per connection
>>>>
>>>> And it also lets one cleanly backport the change as needed without
>>>> affecting anything else.
>>>>
>>>> > This leaves us with an all or nothing solution, which practically
>>>> means we do nothing, since we have to wait until we have a sufficient
>>>> backlog of
>>>> > changes or features to change the protocol. I see that as untenable,
>>>> unless you are now advocating for using extensions for everything ?
>>>> >
>>>> > Dave
>>>>
>>>
>>> I have modified the patch to use protocol options instead of protocol
>>> version
>>>
>>
>> resending with a different patch name as I think the commitfest app won't
>> pick it up with the same name
>>
>
Apparently that patch does not apply. Checked this one against master

Dave

>

Re: Proposal to allow setting cursor options on Portals

Sami Imseih <samimseih@gmail.com> — 2026-03-19T00:52:22Z

Hi,

> Apparently that patch does not apply. Checked this one against master

I spent some time today reviewing the discussion and testing the patch. I have
some comments:

# 1. Remove protocol 3.3 version bump, since we will be using the protocol
extension mechanism. Right?

+ * Test holdable cursors using protocol 3.3 cursor options in Bind message.
+ */
+static void
+test_holdable_cursor(PGconn *conn)
+{
+       PGresult   *res;
+
+       fprintf(stderr, "holdable cursor... ");
+
+       /* Verify protocol 3.3 */
+       if (PQfullProtocolVersion(conn) < 30003)
+               pg_fatal("protocol 3.3 required, got %d",
PQfullProtocolVersion(conn));
+


+       if (strcmp(value, "3.3") == 0)
+       {
+               *result = PG_PROTOCOL(3, 3);
+               return true;
+       }


# 2. Handle case where _pq_.holdable_portal is rejected in
pqGetNegotiateProtocolVersion3

I was testing on a patched client and unpatched server with a
connection string of:
"host=localhost dbname=postgres holdable_portal=1"

I received this error:

```
failed: received invalid protocol negotiation message: server reported
an unsupported parameter that was not requested
("_pq_.holdable_portal")
```

The patch only checks _pq_.test_protocol_negotiation (the grease test
parameter), but we also need to deal
with the rejected extension parameters, rather than fall through to the error.

@@ -1544,6 +1547,16 @@ pqGetNegotiateProtocolVersion3(PGconn *conn)
                        strcmp(conn->workBuffer.data,
"_pq_.test_protocol_negotiation") == 0)
                {
                        found_test_protocol_negotiation = true;
+                       continue;
+               }
+
+               /*
+                * Handle rejected protocol extensions we requested. Disable the
+                * corresponding feature so the client doesn't try to use it.
+                */
+               if (strcmp(conn->workBuffer.data, "_pq_.holdable_portal") == 0)
+               {
+                       conn->holdable_portal_enabled = false;
                }
                else
                {

The above is the change I tested with.
This will be the pattern for other protocol extensions that will be
added in the future as well.

Also, should there also be an API such `PQholdablePortalEnabled` which
takes a connection and
returns if conn->holdable_portal_enabled is enabled? This lets the
client detect at
runtime whether the extension was enabled, or disabled because the
server rejected it.

# 3. Remove PQsendQueryPreparedWithCursorOptions

This was also asked earlier [1], but I don't see an answer. Why do we need this?
Wouldn't PQsendBindWithCursorOption() be sufficient?

# 4. PQsendBindWithCursorOptions

a. Looking at other PQSend patterns, it looking like
PQsendBindWithCursorOptions/PQsendQueryPreparedWithCursorOptions are
missing
`pqAppendCmdQueueEntry(conn, entry)` before returning.

b. The last message type sent in `PQsendBindWithCursorOptions` is a
describe, so the query class should be PGQUERY_DESCRIBE, right?

c. Require a named portal (not just for the HOLD case). An unnamed
portal will not be executed in this API, so
subsequent calls to this API with an unnamed portal will just
overwrite the last one.

# 5. Replace hex with constants defined in libpq-fe.h which mirror parsenodes.h

for example 0x0020 → PQ_CURSOR_OPT_HOLD, etc.

Although I am not sure how these could remain in sync with core.

# 6. In the test_holdable_cursor, can we use PQsendClosePortal() instead of
"CLOSE"? Not for this patch, but it will be good to also have an API
for FETCH.

[1] [https://www.postgresql.org/message-id/CADK3HH%2B9V58vJzCkgvMwk2fyaCtYwr-Dv5em7rXzgUiVrnpuFA%40mail.gmail.com]

--
Sami Imseih
Amazon Web Services (AWS)

Re: Proposal to allow setting cursor options on Portals

Sami Imseih <samimseih@gmail.com> — 2026-03-24T01:59:11Z

Hi,

One thing that just occurred to me is why must we limit
the new protocol extension to holdable cursors?

PQsendBindWithCursorOptions() will work with any cursor option,
Maybe it should be called "_pq_.named_portal" or perhaps better "_pq_.cursor"?

--
Sami Imseih
Amazon Web Services (AWS)

Re: Proposal to allow setting cursor options on Portals

Dave Cramer <davecramer@gmail.com> — 2026-03-24T10:01:05Z

On Mon, 23 Mar 2026 at 21:59, Sami Imseih <samimseih@gmail.com> wrote:

> Hi,
>
> One thing that just occurred to me is why must we limit
> the new protocol extension to holdable cursors?
>
> PQsendBindWithCursorOptions() will work with any cursor option,
> Maybe it should be called "_pq_.named_portal" or perhaps better
> "_pq_.cursor"?
>

Well there are currently ways to name a portal with V3 so named portal
doesn't make sense.

_pq_.cursor would be fine.

Dave

Re: Proposal to allow setting cursor options on Portals

Sami Imseih <samimseih@gmail.com> — 2026-03-24T20:54:41Z

> Well there are currently ways to name a portal with V3 so named portal doesn't make sense.
>
> _pq_.cursor would be fine.

I spent some time today on this, and implemented this as a new protocol
extension called _pq_.protocol_cursor, and the libpq option will also have
the same name. Maybe someone else has a better name, but I like this
as it differentiates from a sql-level cursor. Maybe someone has a better
suggestion.

All the cursor options can be passed, though CURSOR_OPT_BINARY is
irrelevant in the extended query protocol as noted here [1]. Binary output is
controlled by the result format codes on the FETCH instead. So,
CURSOR_OPT_BINARY can be passed as a cursor option, but will be
silently ignored.

I added a new test module libpq_protocol_cursor that expands the
tests to cover all options, except for cursor sensitivity. Also, there
are tests for multiple flags, enabling/disabling the extension and
unnamed portal being used ( which should not be allowed ).

A few other comments to earlier points:

> Also, should there also be an API such `PQholdablePortalEnabled` which
> takes a connection and

I did not implement this and instead rely on
if (cursorOptions != 0 && !conn->protocol_cursor_enabled) to reject the
usage of the API. So, if the API is called with cursor options but
protocol_cursor is not enabled, we reject the attempt. So, I am
not sure this Enabled() API has any value after thinking about it a bit more.

> # 5. Replace hex with constants defined in libpq-fe.h which mirror parsenodes.h
> for example 0x0020 → PQ_CURSOR_OPT_HOLD, etc.

Also, did not implement this since as mentioned earlier, I am not sure
how to keep these in sync with core. The docs do mention that
parsenodes.h is to be referenced for the bitmasks that are defined
as CURSOR_OPT_*. That may be good enough.

See v3 attached.


[1] https://www.postgresql.org/docs/18/protocol-flow.html#PROTOCOL-FLOW-EXT-QUERY

Re: Proposal to allow setting cursor options on Portals

Jelte Fennema-Nio <postgres@jeltef.nl> — 2026-03-25T07:27:05Z

On Tue, 24 Mar 2026 at 21:54, Sami Imseih <samimseih@gmail.com> wrote:
> All the cursor options can be passed, though CURSOR_OPT_BINARY is
> irrelevant in the extended query protocol as noted here [1]. Binary output is
> controlled by the result format codes on the FETCH instead. So,
> CURSOR_OPT_BINARY can be passed as a cursor option, but will be
> silently ignored.

This is what this patch originally did in one of the earlier versions.
And if I understand correctly it was changed after this feedback from
me:

On Sun, 14 Dec 2025 at 14:41, Jelte Fennema-Nio <postgres@jeltef.nl> wrote:
> As mentioned upthread, I'm not sure BINARY makes sense. For any other
> options, the protocol docs should specify which ones are allowed and
> what their bits are. Looking at the DECLARE docs[2].
> 1. I think supporting ASENSITVE/INSENSITIVE/SENSITIVE bits is
> unnecessary, since postgres cursors are always INSENSITIVE.
> 2. For SCROLL vs NO SCROLL, it would be nice if we could get rid of
> the intermediate mode where if neither SCROLL or NO SCROLL is
> specified, it's still SCROLL sometimes. I'm not sure backwards
> compatibility would allow that, i.e. can you currently sometimes do a
> BACKWARD scan on a portal created with Bind. I guess we could make it
> so that if you specify the portal flags, then you have to be explicit
> abuot specifying SCROLL or NO SCROLL
> 3. All the flags with no SQL variant probably shouldn't be
> configurable through the protocol too (e.g. CURSOR_OPT_FAST_PLAN)

Re: Proposal to allow setting cursor options on Portals

Jelte Fennema-Nio <postgres@jeltef.nl> — 2026-03-25T07:46:35Z

On Tue, 24 Mar 2026 at 11:01, Dave Cramer <davecramer@gmail.com> wrote:
> _pq_.cursor would be fine.

I think that makes sense as a name for the option. I think adding flag
support for SCROLL and NO SCROLL would make sense in that case.

Some notes on the patch (but I didn't look look at the client side
libpq code in detail):

For the protocol definition I'd like a few changes:
1. I'd like the new field in the bind message that you add to be
described as an extension bitmap, not specifically for cursor options,
so that future extensions could add bits too it too.
2. Related to that, I think the used bits should not align with the
internal bits. Having the only valid flag bit be 0x0020 is kinda
weird. Let's just make that 0x0001. We could update the internal ones
to match if desired, but I think it's fine for the protocol bits to
differ from the bits in the postgres server.

Docs still mention CURSOR_OPT_BINARY, but support for that has been
removed from the code afaict (which I think is indeed what should
happen)

There's a bunch of protocol version 3.3 code still around, which
should be removed now that the protocol option is added.

PQsendBindWithCursorOptions and PQsendQueryPreparedWithCursorOptions
should error out if conn->holdable_portal_enabled is false. Right now
it silently skips the cursor options if the connection does not
support the protocol extension.

There should be a libpq function to inspect whether the connection
supports cursor options, so some kind of graceful fallback logic can
be implemented by the application when it's not supported.

libpq docs are missing

Re: Proposal to allow setting cursor options on Portals

Dave Cramer <davecramer@gmail.com> — 2026-03-25T14:34:42Z

On Wed, 25 Mar 2026 at 03:46, Jelte Fennema-Nio <postgres@jeltef.nl> wrote:

> On Tue, 24 Mar 2026 at 11:01, Dave Cramer <davecramer@gmail.com> wrote:
> > _pq_.cursor would be fine.
>
> I think that makes sense as a name for the option. I think adding flag
> support for SCROLL and NO SCROLL would make sense in that case.
>
> Some notes on the patch (but I didn't look look at the client side
> libpq code in detail):
>
> For the protocol definition I'd like a few changes:
> 1. I'd like the new field in the bind message that you add to be
> described as an extension bitmap, not specifically for cursor options,
> so that future extensions could add bits too it too.
> 2. Related to that, I think the used bits should not align with the
> internal bits. Having the only valid flag bit be 0x0020 is kinda
> weird. Let's just make that 0x0001. We could update the internal ones
> to match if desired, but I think it's fine for the protocol bits to
> differ from the bits in the postgres server.
>
> Docs still mention CURSOR_OPT_BINARY, but support for that has been
> removed from the code afaict (which I think is indeed what should
> happen)
>
> There's a bunch of protocol version 3.3 code still around, which
> should be removed now that the protocol option is added.
>
> PQsendBindWithCursorOptions and PQsendQueryPreparedWithCursorOptions
> should error out if conn->holdable_portal_enabled is false. Right now
> it silently skips the cursor options if the connection does not
> support the protocol extension.
>
> There should be a libpq function to inspect whether the connection
> supports cursor options, so some kind of graceful fallback logic can
> be implemented by the application when it's not supported.
>
> libpq docs are missing
>

Attached is v4 of the patch
Co-Authored by Sami Imseih

Adds docs and test module

Dave

Re: Proposal to allow setting cursor options on Portals

Jelte Fennema-Nio <postgres@jeltef.nl> — 2026-04-06T09:37:01Z

On Wed, 25 Mar 2026 at 15:34, Dave Cramer <davecramer@gmail.com> wrote:
> Attached is v4 of the patch
> Co-Authored by Sami Imseih
>
> Adds docs and test module

It's looking much more finished

> The portal can
>       later be operated on with cursor commands such as FETCH, MOVE,
>       and CLOSE.

This made me realize that, adding the Bind side of cursors is only
half of the equation. The "Execute" message should also gain new
behaviour to support all the same functionality as FETCH and MOVE. I
think we can do that fairly easily by adding similar flags to Execute.
I think we'd need three flags:
1. MOVE
2. BACKWARD
3. ABSOLUTE

I do realize the scope creep of this, but it feels that without
addressing Execute we have a half-finished feature. That could be
fine, but then I don't think we should call the option
_pq_.protocol_cursor. Because that sounds like it solves the whole
half-baked protocol-level cursor implemention that we currently have.
Maybe _pq_.cursor_bind instead.

I think the "protocol_" part in _pq_.protocol_cursor is duplicative.
The _pq_ part already indicates that it's a protocol option, so I'd
leave that out.

>         <symbol>PQ_BIND_CURSOR_SCROLL</symbol> (scroll),
>        <symbol>PQ_BIND_CURSOR_NO_SCROLL</symbol> (no scroll), and
>        <symbol>PQ_BIND_CURSOR_HOLD</symbol> (hold).
>        These are defined in <filename>libpq-fe.h</filename>.

and

>        <para>
>         Bitmap set by protocol extensions.
>        </para>

I think the Message Formats page should list the actual flag values
that are valid. The protocol docs should not require you to look at
the postgres source code.

        /*
         * Only override the default cursorOptions when the client has
         * explicitly set flags.  A value of 0 means no cursor options were
         * requested, so keep the CreatePortal defaults.
         */

What is the difference between setting cursorOptions = 0 and the
CreatePortal defaults?

>     {"protocol_cursor", NULL, "0", NULL,
>        "Protocol-Cursor", "", 1,
>    offsetof(struct pg_conn, protocol_cursor)},

In my GoAway patchset I linked enabling the protocol extension to the
user requesting protocol v3.2 (or higher).

>         /* Reject any bits we don't recognize */
>         if (bind_ext_flags & ~0x0007)

Let's use PQ_BIND_CURSOR_VALID_FLAGS here too instead of this magic number.

> src/test/modules/libpq_protocol_cursor/libpq_protocol_cursor.c

I think it'd be better to put these tests in the libpq_pipeline test
file. Then we can keep all the libpq tests together so they can share
the helper logic.