Horton

Thread

  1. [PATH] Idle in transaction cancellation

    Andres Freund <andres@anarazel.de> — 2010-10-17T17:58:43Z

    Hi all.

Here is a proposed patch which enables cancellation of $subject. The 
problematic point about doing so is that the client is not expecting any 
messages from the server when its in an idle state during an transaction and 
that simply suppressing that message is not enough as ready for query messages 
would get sent out at unexpected places.

Thus the first patch adds the possibility to add flags to ereport()s error level 
to suppress notifying the client.
It also switches the earlier "COMERROR" log level over to this flag 
(LOG_NO_CLIENT) with a typedef to support the old method.

The second patch sets up a variable "silent_error_while_idle" when its 
cancelling an idle txn which suppresses sending out messages at wrong places 
and resets its  after having read a command in the simple protocol and after 
having read a 'sync' message in the extended protocol.

Currently it does *not* report any special error message to the client if it 
starts sending commands in an (unbekownst to it) failed transaction, but just 
the normal "25P02: current transaction is aborted..." message.

It shouldn't be hard to add that and I will propose a patch if people would 
like it (I personally am not very interested, but I can see people validly 
wanting it), but I would like to have some feedback on the patch first.

Greetings,

Andres

  2. Re: [PATH] Idle in transaction cancellation

    Kevin Grittner <kevin.grittner@wicourts.gov> — 2010-10-19T14:18:29Z

    Andres Freund <andres@anarazel.de> wrote:
 
> Here is a proposed patch which enables cancellation of $subject.
 
Cool.  Some enhancements we'd like to do to Serializable Snapshot
Isolation (SSI), should the base patch make it in, would require
this capability.
 
> Currently it does *not* report any special error message to the
> client if it 
> 
> starts sending commands in an (unbekownst to it) failed
> transaction, but just the normal "25P02: current transaction is
> aborted..." message.
> 
> It shouldn't be hard to add that and I will propose a patch if
> people would like it (I personally am not very interested, but I
> can see people validly wanting it)
 
For SSI purposes, it would be highly desirable to be able to set the
SQLSTATE and message generated when the canceled transaction
terminates.
 
> but I would like to have some feedback on the patch first.
 
I'll take a look when I can, but it may be a couple weeks from now.
 
-Kevin

  3. Re: [PATH] Idle in transaction cancellation

    Robert Haas <robertmhaas@gmail.com> — 2010-10-19T14:27:17Z

    On Tue, Oct 19, 2010 at 10:18 AM, Kevin Grittner
<Kevin.Grittner@wicourts.gov> wrote:
> Andres Freund <andres@anarazel.de> wrote:
>
>> Here is a proposed patch which enables cancellation of $subject.
>
> I'll take a look when I can, but it may be a couple weeks from now.

How about adding it to
https://commitfest.postgresql.org/action/commitfest_view/open ?

Seems like a good feature, at least from 20,000 feet.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  4. Re: [PATCH] V3: Idle in transaction cancellation

    Andres Freund <andres@anarazel.de> — 2010-10-30T08:49:21Z

    Hi,

On Tuesday 19 October 2010 16:18:29 Kevin Grittner wrote:
> Andres Freund <andres@anarazel.de> wrote:
> > Here is a proposed patch which enables cancellation of $subject.
> 
> Cool.  Some enhancements we'd like to do to Serializable Snapshot
> Isolation (SSI), should the base patch make it in, would require
> this capability.
> 
> > Currently it does *not* report any special error message to the
> > client if it
> > 
> > starts sending commands in an (unbekownst to it) failed
> > transaction, but just the normal "25P02: current transaction is
> > aborted..." message.
> > 
> > It shouldn't be hard to add that and I will propose a patch if
> > people would like it (I personally am not very interested, but I
> > can see people validly wanting it)
> 
> For SSI purposes, it would be highly desirable to be able to set the
> SQLSTATE and message generated when the canceled transaction
> terminates.
Ok, I implemented that capability, but the patch feels somewhat wrong to me, 
so its a separate patch on top the others:

* it intermingles logic between elog.c and postgres.c far too much - requiring 
exporting variables which should not get exported. And it would still need 
more to allow some sensible Assert()s. I.e. Assert(DoingCommandRead) would 
need to be available in elog.c to avoid somebody using the re-throwing 
capability out of place....

* You wont see an error if the next command after the IDLE in transaction is a 
COMMIT/ROLLBACK. I don’t see any sensible way around that.

* the copied edata lives in TopMemoryContext and gets only reset in the next 
reported error, after the re-raised error was reported.

That’s how it looks like to raise one such error right now:

error = ERROR
if (DoingCommandRead)
{
	silent_error_while_idle = true;
	error |= LOG_NO_CLIENT|LOG_RE_THROW_AFTER_SYNC;
}

...

ereport(error,
        (errcode(ERRCODE_QUERY_CANCELED),
         errmsg("canceling statement due to user request")));



One could mingle together  LOG_NO_CLIENT|LOG_RE_THROW_AFTER_SYNC into a macro 
and set silent_error_while_idle in elog.c, but I don’t see many callsites 
coming up, so I think its better to be explicit.

Ill set this up for the next commitfest, I don't think I can do much more 
without further input.

Andres

  5. Re: [PATCH] V3: Idle in transaction cancellation

    Kevin Grittner <kevin.grittner@wicourts.gov> — 2010-11-02T17:33:15Z

    Andres Freund <andres@anarazel.de> wrote:
 
> * You wont see an error if the next command after the IDLE in
> transaction is a COMMIT/ROLLBACK. I don*t see any sensible way
> around that.
 
Well, on a ROLLBACK I'm not sure it's a problem.  On a COMMIT,
couldn't you call a function to check for it in CommitTransaction
and PrepareTransaction?
 
-Kevin

  6. Re: [PATCH] V3: Idle in transaction cancellation

    Andres Freund <andres@anarazel.de> — 2010-11-02T21:36:19Z

    On Tuesday 02 November 2010 18:33:15 Kevin Grittner wrote:
> Andres Freund <andres@anarazel.de> wrote:
> > * You wont see an error if the next command after the IDLE in
> > transaction is a COMMIT/ROLLBACK. I don*t see any sensible way
> > around that.
> Well, on a ROLLBACK I'm not sure it's a problem.  On a COMMIT,
> couldn't you call a function to check for it in CommitTransaction
> and PrepareTransaction?
Sure, throwing an error somewhere wouldnt be that hard. But at the moment a 
COMMIT is always successfull (and just reporting a ROLLBACK in a failed txn). 
Doesn't seem to be something to changed lightly.

Does anybody have any idea why COMMIT is allowed there? Seems pretty strange 
to me.

Andres

  7. Re: [PATCH] V3: Idle in transaction cancellation

    Alvaro Herrera <alvherre@commandprompt.com> — 2010-11-02T21:52:17Z

    Excerpts from Andres Freund's message of mar nov 02 18:36:19 -0300 2010:
> On Tuesday 02 November 2010 18:33:15 Kevin Grittner wrote:
> > Andres Freund <andres@anarazel.de> wrote:
> > > * You wont see an error if the next command after the IDLE in
> > > transaction is a COMMIT/ROLLBACK. I don*t see any sensible way
> > > around that.
> > Well, on a ROLLBACK I'm not sure it's a problem.  On a COMMIT,
> > couldn't you call a function to check for it in CommitTransaction
> > and PrepareTransaction?
> Sure, throwing an error somewhere wouldnt be that hard. But at the moment a 
> COMMIT is always successfull (and just reporting a ROLLBACK in a failed txn). 
> Doesn't seem to be something to changed lightly.

If the user calls COMMIT, it calls EndTransactionBlock, which ends up
calling AbortTransaction.  Can't you do it at that point?  (Says he who
hasn't looked at the patch at all)

-- 
Álvaro Herrera <alvherre@commandprompt.com>
The PostgreSQL Company - Command Prompt, Inc.
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

  8. Re: [PATCH] V3: Idle in transaction cancellation

    Kevin Grittner <kevin.grittner@wicourts.gov> — 2010-11-02T21:59:15Z

    Andres Freund <andres@anarazel.de> wrote:
> On Tuesday 02 November 2010 18:33:15 Kevin Grittner wrote:
 
>>  Well, on a ROLLBACK I'm not sure it's a problem.  On a COMMIT,
>> couldn't you call a function to check for it in CommitTransaction
>> and PrepareTransaction?
> Sure, throwing an error somewhere wouldnt be that hard. But at the
> moment a COMMIT is always successfull (and just reporting a
> ROLLBACK in a failed txn). 
> Doesn't seem to be something to changed lightly.
 
Well, I'm looking at using this with the Serializable Snapshot
Isolation (SSI) patch, which can throw an error from a COMMIT, if
the commit completes the conditions necessary for an anomaly to
occur (i.e., the committing transaction is on the rw-conflict *out*
side of a pivot, and it is the first in the set to commit). If we
succeed in enhancing SSI to use lists of rw-conflicted transactions
rather than its current techniques, we might be able to (and find it
desirable to) always commit in that circumstance and roll back some
*other* transaction which is part of the problem.  Of course, that
other transaction might be idle at the time, and the next thing *it*
tries to execute *might* be a COMMIT.
 
So if the SSI patch goes in, there will always be some chance that a
COMMIT can fail, but doing it through the mechanism your patch
provides could improve performance, because we could guarantee that
nobody ever has a serialization failure without first successfully
committing a transaction which wrote to one or more tables.  If a
commit fails due to SSI, it is highly desirable that the error use
the serialization failure SQLSTATE, so that an application framework
can know that it is reasonable to retry the transaction.
 
> Does anybody have any idea why COMMIT is allowed there? Seems
> pretty strange to me.
 
So that the "failed transaction" state can be cleared.  The
transaction as a whole has failed, but you don't want the connection
to become useless.
 
-Kevin

  9. Re: [PATCH] V3: Idle in transaction cancellation

    Andres Freund <andres@anarazel.de> — 2010-11-02T22:11:56Z

    On Tuesday 02 November 2010 22:59:15 Kevin Grittner wrote:
> > Does anybody have any idea why COMMIT is allowed there? Seems
> > pretty strange to me.
> 
>  
> So that the "failed transaction" state can be cleared.  The
> transaction as a whole has failed, but you don't want the connection
> to become useless.
Well, allowing ROLLBACK is enought there, isnt it?

  10. Re: [PATCH] V3: Idle in transaction cancellation

    Tom Lane <tgl@sss.pgh.pa.us> — 2010-11-02T22:52:40Z

    Andres Freund <andres@anarazel.de> writes:
> On Tuesday 02 November 2010 22:59:15 Kevin Grittner wrote:
>>> Does anybody have any idea why COMMIT is allowed there? Seems
>>> pretty strange to me.

>> So that the "failed transaction" state can be cleared.  The
>> transaction as a whole has failed, but you don't want the connection
>> to become useless.

> Well, allowing ROLLBACK is enought there, isnt it?

The client has no reason to think the transaction has failed, so what
it's going to send is COMMIT, not ROLLBACK.  From the point of view of
the client, this should look like its COMMIT failed (or in general its
next command failed); not like there was some magic action-at-a-distance
on the state of its transaction.

Now you could argue that if you send COMMIT, and that fails, you should
have to send ROLLBACK to get to an idle state ... but that's not how
this ever worked before, and I don't think it's what the SQL standard
expects either.  After a COMMIT, you're out of the transaction either
way.

			regards, tom lane

  11. Re: [PATCH] V3: Idle in transaction cancellation

    Kevin Grittner <kevin.grittner@wicourts.gov> — 2010-11-29T22:38:48Z

    Andres Freund <andres@anarazel.de> wrote:
 
> Ok, I implemented that capability
 
I applied all three patches with minor offsets, and it builds, but
several regression tests fail.  I backed out the patches in reverse
order and confirmed that while the regression tests pass  without
any of these patches, they fail with just the first, the first and
the second, or all three patches.
 
If you're not seeing the same thing there, I'll be happy to provide
the details.
 
-Kevin

  12. Re: [PATCH] V3: Idle in transaction cancellation

    Kevin Grittner <kevin.grittner@wicourts.gov> — 2010-11-29T23:42:57Z

    I wrote:
 
> I applied all three patches with minor offsets, and it builds, but
> several regression tests fail.
 
Sorry, after sending that I realized I hadn't done a make distclean.
After that it passes.  Please ignore the previous post.
 
-Kevin

  13. Re: V3: Idle in transaction cancellation

    Kevin Grittner <kevin.grittner@wicourts.gov> — 2010-12-01T23:48:53Z

    Andres Freund <andres@anarazel.de> wrote:
> On Tuesday 19 October 2010 16:18:29 Kevin Grittner wrote:
 
>> For SSI purposes, it would be highly desirable to be able to set
>> the SQLSTATE and message generated when the canceled transaction
>> terminates.
 
> Ok, I implemented that capability, but the patch feels somewhat
> wrong to me, 
> 
> so its a separate patch on top the others:
 
The patch is in context diff format, applies with minor offsets,
compiles and passes regression tests.
 
I have to admit that after reading the patch, I think I previously
misunderstood the scope of it.  Am I correct in reading that the
main thrust of this is to improve error handling on standbys?  Is
there any provision for one backend to cause a *different* backend
which is idle in a transaction to terminate cleanly when it attempts
to process its next statement?  (That is what I was hoping to find,
for use in the SSI patch.)
 
Anyway, if the third patch file is only there because of my request,
I think it might be best to focus on the first two as a solution for
the standby issues this was originally meant to address, and then to
look at an API for the usage I have in mind after that is settled.
 
-Kevin

  14. Re: V3: Idle in transaction cancellation

    Andres Freund <andres@anarazel.de> — 2010-12-02T14:57:18Z

    On Thursday 02 December 2010 00:48:53 Kevin Grittner wrote:
> Andres Freund <andres@anarazel.de> wrote:
> > On Tuesday 19 October 2010 16:18:29 Kevin Grittner wrote:
> >> For SSI purposes, it would be highly desirable to be able to set
> >> the SQLSTATE and message generated when the canceled transaction
> >> terminates.
> > 
> > Ok, I implemented that capability, but the patch feels somewhat
> > wrong to me,
> 
> > so its a separate patch on top the others:
> The patch is in context diff format, applies with minor offsets,
> compiles and passes regression tests.
> 
> I have to admit that after reading the patch, I think I previously
> misunderstood the scope of it.  Am I correct in reading that the
> main thrust of this is to improve error handling on standbys?  Is
> there any provision for one backend to cause a *different* backend
> which is idle in a transaction to terminate cleanly when it attempts
> to process its next statement?  (That is what I was hoping to find,
> for use in the SSI patch.)
Do you wan't to terminate it immediately or on next statement?

You might want to check out SendProcSignal() et al.

> Anyway, if the third patch file is only there because of my request,
> I think it might be best to focus on the first two as a solution for
> the standby issues this was originally meant to address, and then to
> look at an API for the usage I have in mind after that is settled.
Besides that I dont like the implementation very much, I think its generally a 
good idea...

  15. Re: V3: Idle in transaction cancellation

    Kevin Grittner <kevin.grittner@wicourts.gov> — 2010-12-02T18:35:35Z

    Andres Freund <andres@anarazel.de> wrote:
 
> Do you wan't to terminate it immediately or on next statement?
 
I want to have one backend terminate the transaction on another
backend as soon as practicable.  If a query is active, it would be
best if it was canceled.  It appears that if it is "idle in
transaction" there is a need to wait for the next request.  It would
be a big plus for the backend requesting the cancellation to be able
to specify the SQLSTATE, message, etc., used by the other backend on
failure.
 
> You might want to check out SendProcSignal() et al.
 
Will take a look.
 
> Besides that I dont like the implementation very much, I think its
> generally a good idea...
 
OK.  While browsing around, I'll try to think of an alternative
approach, but this is new territory for me -- I've been learning
about areas in the code at need so far....
 
-Kevin

  16. Re: V3: Idle in transaction cancellation

    Kevin Grittner <kevin.grittner@wicourts.gov> — 2010-12-02T20:54:54Z

    Andres Freund <andres@anarazel.de> wrote:
> On Thursday 02 December 2010 00:48:53 Kevin Grittner wrote:

>> Is there any provision for one backend to cause a *different*
>> backend which is idle in a transaction to terminate cleanly when
>> it attempts to process its next statement?
 
> You might want to check out SendProcSignal() et al.
 
Yeah, that was the missing link for me.  Thanks!
 
>> Anyway, if the third patch file is only there because of my
>> request, I think it might be best to focus on the first two as a
>> solution for the standby issues this was originally meant to
>> address, and then to look at an API for the usage I have in mind
>> after that is settled.
 
> Besides that I dont like the implementation very much, I think its
> generally a good idea...
 
Is it sane to leave the implementation of this for the specific
areas which need it (like SSI), or do you think a generalized API
for it is needed?
 
I'll look at it more closely tonight, but at first scan it appears
that just reserving one flag for PROCSIG_SERIALIZATION_FAILURE (or
PROCSIG_SSI_CANCELLATION?) would allow me to code up the desired
behavior in a function called from procsignal_sigusr1_handler.  I
can arrange for passing any needed detail through the SSI-controlled
structures somehow.  Would that allow you to skip the parts you
didn't like?
 
It looks as though this is something which could easily be split off
as a separate patch within the SSI effort.
 
-Kevin

  17. Re: [PATCH] V3: Idle in transaction cancellation

    Alvaro Herrera <alvherre@commandprompt.com> — 2010-12-02T21:21:37Z

    Excerpts from Andres Freund's message of sáb oct 30 05:49:21 -0300 2010:

> Ill set this up for the next commitfest, I don't think I can do much more 
> without further input.

Are you reserving about 20 bits for levels, and 12 for flags?  Given the
relatively scarce growth of levels, I think we could live with about 6
or 7 bits for level, rest for flags.

-- 
Álvaro Herrera <alvherre@commandprompt.com>
The PostgreSQL Company - Command Prompt, Inc.
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

  18. Re: [PATCH] V3: Idle in transaction cancellation

    Andres Freund <andres@anarazel.de> — 2010-12-02T21:36:51Z

    On Thursday 02 December 2010 22:21:37 Alvaro Herrera wrote:
> Excerpts from Andres Freund's message of sáb oct 30 05:49:21 -0300 2010:
> > Ill set this up for the next commitfest, I don't think I can do much
> > more  without further input.
> 
> Are you reserving about 20 bits for levels, and 12 for flags?  Given the
> relatively scarce growth of levels, I think we could live with about 6
> or 7 bits for level, rest for flags.
The number I picked was absolutely arbitrary I admit. Neither did I think it 
would be likely to see more levels, nor did I forsee many flags, so I just 
chose some number I liked in that certain moment ;-)

Andres

  19. Re: [PATCH] V3: Idle in transaction cancellation

    Robert Haas <robertmhaas@gmail.com> — 2010-12-15T01:20:31Z

    On Sat, Oct 30, 2010 at 4:49 AM, Andres Freund <andres@anarazel.de> wrote:
>> > Here is a proposed patch which enables cancellation of $subject.

Disclaimer: This isn't my area of expertise, so take the below with a
grain or seven of salt.

It sort of looks to me like the LOG_NO_CLIENT error flag and the
silent_error_while_idle flag are trying to cooperate to get the effect
of throwing an error without actually throwing an error.  I'm
wondering if it would be at all sensible to do that more directly by
making ProcessInterrupts() call AbortCurrentTransaction() in this
case.

Assuming that works at all, it would presumably mean that the client
would thereafter get something like this:

current transaction is aborted, commands ignored until end of transaction block

...which might be thought unhelpful.  But that could be fixed either
by modifying errdetail_abort() or perhaps even by abstracting the
"current transaction is aborted, commands..." message into a function
that could produce an entirely different message if on either the
first or all calls within a given transaction.

I'm not sure if this would work, or if it's better.  I'm just throwing
it out there, because the current approach looks a little grotty to
me.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  20. Re: [PATCH] V3: Idle in transaction cancellation

    Andres Freund <andres@anarazel.de> — 2010-12-15T12:13:30Z

    On Wednesday 15 December 2010 02:20:31 Robert Haas wrote:
> On Sat, Oct 30, 2010 at 4:49 AM, Andres Freund <andres@anarazel.de> wrote:
> >> > Here is a proposed patch which enables cancellation of $subject.
> 
> Disclaimer: This isn't my area of expertise, so take the below with a
> grain or seven of salt.
I don't know whos area of expertise it is except maybe, surprise, surprise, 
Toms.

> It sort of looks to me like the LOG_NO_CLIENT error flag and the
> silent_error_while_idle flag are trying to cooperate to get the effect
> of throwing an error without actually throwing an error.  I'm
> wondering if it would be at all sensible to do that more directly by
> making ProcessInterrupts() call AbortCurrentTransaction() in this
> case.
Hm. I think you want the normal server-side error logging continuing to work.

Its not really throwing an error without throwing one - its throwing one 
without confusing the heck out of the client because the protocol is not ready 
for that. I don't think introducing an "half-error" state is a good idea 
because one day the protocol maybe ready to actually transport an error while 
idle in txn (I would like to get there).

> I'm not sure if this would work, or if it's better.  I'm just throwing
> it out there, because the current approach looks a little grotty to
> me.
I with you on the grotty aspect... On the other hand the whole code is not 
exactly nice...

Andres

  21. Re: [PATCH] V3: Idle in transaction cancellation

    Robert Haas <robertmhaas@gmail.com> — 2010-12-15T12:33:30Z

    On Wed, Dec 15, 2010 at 7:13 AM, Andres Freund <andres@anarazel.de> wrote:
>> It sort of looks to me like the LOG_NO_CLIENT error flag and the
>> silent_error_while_idle flag are trying to cooperate to get the effect
>> of throwing an error without actually throwing an error.  I'm
>> wondering if it would be at all sensible to do that more directly by
>> making ProcessInterrupts() call AbortCurrentTransaction() in this
>> case.
> Hm. I think you want the normal server-side error logging continuing to work.

I was thinking we could get around that by doing elog(LOG), but I
guess that doesn't quite work either since we don't know what
client_min_messages is.  Hrm...

>> I'm not sure if this would work, or if it's better.  I'm just throwing
>> it out there, because the current approach looks a little grotty to
>> me.
> I with you on the grotty aspect... On the other hand the whole code is not
> exactly nice...

Yeah.  I'll try to find some time to think about this some more.  It
would sure be nice if we could find a solution that's a bit
conceptually cleaner, even if it basically works the same way as what
you've done here.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  22. Re: [PATCH] V3: Idle in transaction cancellation

    Andres Freund <andres@anarazel.de> — 2010-12-15T12:47:08Z

    On Wednesday 15 December 2010 13:33:30 Robert Haas wrote:
> On Wed, Dec 15, 2010 at 7:13 AM, Andres Freund <andres@anarazel.de> wrote:
> >> It sort of looks to me like the LOG_NO_CLIENT error flag and the
> >> silent_error_while_idle flag are trying to cooperate to get the effect
> >> of throwing an error without actually throwing an error.  I'm
> >> wondering if it would be at all sensible to do that more directly by
> >> making ProcessInterrupts() call AbortCurrentTransaction() in this
> >> case.
> > 
> > Hm. I think you want the normal server-side error logging continuing to
> > work.
> 
> I was thinking we could get around that by doing elog(LOG), but I
> guess that doesn't quite work either since we don't know what
> client_min_messages is.  Hrm...
I thought about doing that first. Btw, LOG_NO_CLIENT is just a more abstracted 
way of what COMERROR did before...

> >> I'm not sure if this would work, or if it's better.  I'm just throwing
> >> it out there, because the current approach looks a little grotty to
> >> me.
> > 
> > I with you on the grotty aspect... On the other hand the whole code is
> > not exactly nice...
> 
> Yeah.  I'll try to find some time to think about this some more.  It
> would sure be nice if we could find a solution that's a bit
> conceptually cleaner, even if it basically works the same way as what
> you've done here.
I would like that as well. I am not sure you can achieve that in a reasonable 
amount of work. At least I couldn't.

Andres

  23. Re: [PATCH] V3: Idle in transaction cancellation

    Robert Haas <robertmhaas@gmail.com> — 2010-12-15T14:40:20Z

    On Wed, Dec 15, 2010 at 7:47 AM, Andres Freund <andres@anarazel.de> wrote:
> I thought about doing that first. Btw, LOG_NO_CLIENT is just a more abstracted
> way of what COMERROR did before...

Hmm, but it must not be quite the same, because that didn't require
the silent_error_while_idle flag.

>> Yeah.  I'll try to find some time to think about this some more.  It
>> would sure be nice if we could find a solution that's a bit
>> conceptually cleaner, even if it basically works the same way as what
>> you've done here.
> I would like that as well. I am not sure you can achieve that in a reasonable
> amount of work. At least I couldn't.

Is there a way that errstart() and/or errfinish() can know enough
about the state of the communication with the frontend to decide
whether to suppress edata->output_to_client?  In other words, instead
of explicitly passing in a flag that says whether to inform the
client, it would be better for the error-reporting machinery to
intrinsically know whether it's right to send_message_to_frontend().
Otherwise, an error thrown from an unexpected location might not have
the flag set correctly.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  24. Re: [PATCH] V3: Idle in transaction cancellation

    Andres Freund <andres@anarazel.de> — 2010-12-15T15:02:14Z

    On Wednesday 15 December 2010 15:40:20 Robert Haas wrote:
> On Wed, Dec 15, 2010 at 7:47 AM, Andres Freund <andres@anarazel.de> wrote:
> > I thought about doing that first. Btw, LOG_NO_CLIENT is just a more
> > abstracted way of what COMERROR did before...
> 
> Hmm, but it must not be quite the same, because that didn't require
> the silent_error_while_idle flag.
True. Thats a separate thing.

> >> Yeah.  I'll try to find some time to think about this some more.  It
> >> would sure be nice if we could find a solution that's a bit
> >> conceptually cleaner, even if it basically works the same way as what
> >> you've done here.
> > 
> > I would like that as well. I am not sure you can achieve that in a
> > reasonable amount of work. At least I couldn't.
> Is there a way that errstart() and/or errfinish() can know enough
> about the state of the communication with the frontend to decide
> whether to suppress edata->output_to_client?  In other words, instead
> of explicitly passing in a flag that says whether to inform the
> client, it would be better for the error-reporting machinery to
> intrinsically know whether it's right to send_message_to_frontend().
> Otherwise, an error thrown from an unexpected location might not have
> the flag set correctly.
Currently there are no other locations where we errors could get thrown at 
that point but I see where youre going.

You could use "DoingCommandRead" to solve that specific use-case, but the 
COMERROR ones I don't see as being replaced that easily.
We could introduce something like

NoLogToClientBegin();
NoLogToClientEnd();
int NoLogToClientCntr = 0;

but that sounds like overdoing it for me.

Andres

  25. Re: [PATCH] V3: Idle in transaction cancellation

    Robert Haas <robertmhaas@gmail.com> — 2010-12-15T19:12:45Z

    On Wed, Dec 15, 2010 at 10:02 AM, Andres Freund <andres@anarazel.de> wrote:
>> Is there a way that errstart() and/or errfinish() can know enough
>> about the state of the communication with the frontend to decide
>> whether to suppress edata->output_to_client?  In other words, instead
>> of explicitly passing in a flag that says whether to inform the
>> client, it would be better for the error-reporting machinery to
>> intrinsically know whether it's right to send_message_to_frontend().
>> Otherwise, an error thrown from an unexpected location might not have
>> the flag set correctly.
>
> You could use "DoingCommandRead" to solve that specific use-case, but the
> COMERROR ones I don't see as being replaced that easily.

Well, again, I'm not an expert on this, but why would we need to unify
the two mechanisms?  Asynchronous rollbacks (what we're trying to do
here) and protocol violations (which is what COMMERROR looks to be
used for) are really sort of different.  I'm not really sure we need
to handle them in the same way.  Let's think about a recovery conflict
where ProcessInterrupts() has been called.  Right now, if that
situation occurs and we are not DoingCommandRead, then we just throw
an error.  That's either safe, or an already-existing bug.  So the
question is what to do if we ARE DoingCommandRead.  Right now, we
throw a fatal error.  There's no comment explaining why, but I'm
guessing that the reason is the same problem we're trying to fix here:
the protocol state gets confused - but if we throw a FATAL then the
client goes away and we don't have to worry about it any more.  Our
goal here, as I understand it, is to handle that case without a FATAL.

So let's see... if we're DoingCommandRead at that point, and
whereToSendOutput == DestRemote then we set whereToSendOutput =
DestNone before throwing the error, and restore it just after we reset
DoingCommandRead?  <stabs blindly at target>

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  26. Re: [PATCH] V3: Idle in transaction cancellation

    Greg Smith <greg@2ndquadrant.com> — 2010-12-16T12:32:46Z

    Andres Freund wrote:
> On Thursday 02 December 2010 22:21:37 Alvaro Herrera wrote:
>   
>> Excerpts from Andres Freund's message of sáb oct 30 05:49:21 -0300 2010:
>>     
>>> Ill set this up for the next commitfest, I don't think I can do much
>>> more  without further input.
>>>       
>> Are you reserving about 20 bits for levels, and 12 for flags?  Given the
>> relatively scarce growth of levels, I think we could live with about 6
>> or 7 bits for level, rest for flags.
>>     
> The number I picked was absolutely arbitrary I admit. Neither did I think it 
> would be likely to see more levels, nor did I forsee many flags, so I just 
> chose some number I liked in that certain moment ;-)
>   

This bit of detail seems to have died down without being resolved; 
bumping it to add a reminder about that.

I count four issues of various sizes left with this patch right now:

1) This levels bit
2) Can the approach used be simplified or the code made cleaner?
3) What is the interaction with Hot Standby error handling?
4) The usual code formatting nitpicking, Kevin mentioned braces being an 
issue

Robert is already thinking about (2); I'll keep an eye out for someone 
who can test (3) now that it's been identified as a concern; and the 
other two are pretty small details once those are crossed.  I don't see 
this as being ready to commit just yet though, so I don't see this going 
anywhere other than returned for now; will mark it as such.  Hopefully 
this will gather enough additional review to continue moving forward now 
that the main issues are identified.

-- 
Greg Smith   2ndQuadrant US    greg@2ndQuadrant.com   Baltimore, MD
PostgreSQL Training, Services and Support        www.2ndQuadrant.us
"PostgreSQL 9.0 High Performance": http://www.2ndQuadrant.com/books

  27. Re: [PATCH] V3: Idle in transaction cancellation

    Andres Freund <andres@anarazel.de> — 2010-12-16T13:49:45Z

    Hi Greg,

On Thursday 16 December 2010 13:32:46 Greg Smith wrote:
> Andres Freund wrote:
> > On Thursday 02 December 2010 22:21:37 Alvaro Herrera wrote:
> >> Excerpts from Andres Freund's message of sáb oct 30 05:49:21 -0300 2010:
> >>> Ill set this up for the next commitfest, I don't think I can do much
> >>> more  without further input.
> >> 
> >> Are you reserving about 20 bits for levels, and 12 for flags?  Given the
> >> relatively scarce growth of levels, I think we could live with about 6
> >> or 7 bits for level, rest for flags.
> > 
> > The number I picked was absolutely arbitrary I admit. Neither did I think
> > it would be likely to see more levels, nor did I forsee many flags, so I
> > just chose some number I liked in that certain moment ;-)
> This bit of detail seems to have died down without being resolved;
> bumping it to add a reminder about that.
> I count four issues of various sizes left with this patch right now:
> 
> 1) This levels bit
If we go with that approach I think the amount of bits reserved for both is 
big enough to never be reached in reality. Also there is no backward-compat 
issue in changing as were not ABI stable between major releases anyway...

> 2) Can the approach used be simplified or the code made cleaner?
> 3) What is the interaction with Hot Standby error handling?
It works for me - not to say that independent testing wouldn't be good though.

> 4) The usual code formatting nitpicking, Kevin mentioned braces being an
> issue
Will redo if the other issues are cleared.

Andres

  28. Re: [PATCH] V3: Idle in transaction cancellation

    Andres Freund <andres@anarazel.de> — 2010-12-16T14:12:38Z

    On Wednesday 15 December 2010 20:12:45 Robert Haas wrote:
> On Wed, Dec 15, 2010 at 10:02 AM, Andres Freund <andres@anarazel.de> wrote:
> >> Is there a way that errstart() and/or errfinish() can know enough
> >> about the state of the communication with the frontend to decide
> >> whether to suppress edata->output_to_client?  In other words, instead
> >> of explicitly passing in a flag that says whether to inform the
> >> client, it would be better for the error-reporting machinery to
> >> intrinsically know whether it's right to send_message_to_frontend().
> >> Otherwise, an error thrown from an unexpected location might not have
> >> the flag set correctly.
> > 
> > You could use "DoingCommandRead" to solve that specific use-case, but the
> > COMERROR ones I don't see as being replaced that easily.
> 
> Well, again, I'm not an expert on this, but why would we need to unify
> the two mechanisms?  Asynchronous rollbacks (what we're trying to do
> here) and protocol violations (which is what COMMERROR looks to be
> used for) are really sort of different.
I think its not only protocol violations. Data-Leakage during auth are also 
handled with it I think.

> I'm not really sure we need to handle them in the same way.  Let's think
> about a recovery conflict where ProcessInterrupts() has been called.  Right
> now, if that situation occurs and we are not DoingCommandRead, then we just
> throw an error.  That's either safe, or an already-existing bug.
That should be safe.

> So the question is what to do if we ARE DoingCommandRead.  Right now, we
> throw a fatal error.  There's no comment explaining why, but I'm
> guessing that the reason is the same problem we're trying to fix here:
> the protocol state gets confused - but if we throw a FATAL then the
> client goes away and we don't have to worry about it any more.  Our
> goal here, as I understand it, is to handle that case without a FATAL.
Yes, thats it.

> So let's see... if we're DoingCommandRead at that point, and
> whereToSendOutput == DestRemote then we set whereToSendOutput =
> DestNone before throwing the error, and restore it just after we reset
> DoingCommandRead?  <stabs blindly at target>
That won't be enough unfortunately. A transaction-aborted error will get re-
raised before the client is ready to re-accept it. Thats what the 
silent_error_while_idle flag is needed for.
Also I think such a simple implementation would cause problems with single 
user mode. Now that could get fixed by saving and resetting the old mode - that 
might have exception handling problems in turn (COMERRORS during an ssl 
connection for example) because it will leave the section without having reset 
whereToSendOutput. Sprinkling PG_TRY everywhere hardly seems simpler...

Andres

  29. Re: [PATCH] V3: Idle in transaction cancellation

    Tom Lane <tgl@sss.pgh.pa.us> — 2010-12-16T16:58:24Z

    Greg Smith <greg@2ndquadrant.com> writes:
> I count four issues of various sizes left with this patch right now:

> 1) This levels bit
> 2) Can the approach used be simplified or the code made cleaner?
> 3) What is the interaction with Hot Standby error handling?
> 4) The usual code formatting nitpicking, Kevin mentioned braces being an 
> issue

You forgot (5) it doesn't work and (6) it's impossibly ugly :-(.

The reason it doesn't work is you can *not* throw a longjmp while in
DoingCommandRead state.  This isn't just a matter of not breaking the
protocol at our level; it risks leaving openssl in a confused state,
either violating the ssl protocol or simply with internal state trashed
because it got interrupted in the middle of changing it.

It's possible we could refactor things so we abort the open transaction
while inside the interrupt handler, then queue up an error to be
reported whenever we next get a command (as envisioned in part 0003),
then just return control back to the input stack.  But things could get
messy if we get another error to be reported while trying to abort.

In any case I really do not care for flag bits in the elog level field.
That's horrid, and I don't think it's even well matched to the problem.
What we need is for elog.c to understand that we're in a *state* that
forbids transmission to the client; it's not a property of specific
error messages, much less a property that the code reporting the error
should be required to be aware of.

I think a more realistic approach to the problem might be to expose the
DoingCommandRead flag to elog.c, and have it take responsibility for
queuing messages that can't be sent to the client right away.  Plus the
above-mentioned refactoring of where transaction abort happens.

			regards, tom lane

  30. Re: [PATCH] V3: Idle in transaction cancellation

    Robert Haas <robertmhaas@gmail.com> — 2010-12-16T17:12:30Z

    On Thu, Dec 16, 2010 at 11:58 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Greg Smith <greg@2ndquadrant.com> writes:
>> I count four issues of various sizes left with this patch right now:
>
>> 1) This levels bit
>> 2) Can the approach used be simplified or the code made cleaner?
>> 3) What is the interaction with Hot Standby error handling?
>> 4) The usual code formatting nitpicking, Kevin mentioned braces being an
>> issue
>
> You forgot (5) it doesn't work and (6) it's impossibly ugly :-(.
>
> The reason it doesn't work is you can *not* throw a longjmp while in
> DoingCommandRead state.  This isn't just a matter of not breaking the
> protocol at our level; it risks leaving openssl in a confused state,
> either violating the ssl protocol or simply with internal state trashed
> because it got interrupted in the middle of changing it.

Ah!  An excellent point.  Thanks for weighing in on this.

> It's possible we could refactor things so we abort the open transaction
> while inside the interrupt handler, then queue up an error to be
> reported whenever we next get a command (as envisioned in part 0003),
> then just return control back to the input stack.  But things could get
> messy if we get another error to be reported while trying to abort.
>
> In any case I really do not care for flag bits in the elog level field.
> That's horrid, and I don't think it's even well matched to the problem.
> What we need is for elog.c to understand that we're in a *state* that
> forbids transmission to the client; it's not a property of specific
> error messages, much less a property that the code reporting the error
> should be required to be aware of.
>
> I think a more realistic approach to the problem might be to expose the
> DoingCommandRead flag to elog.c, and have it take responsibility for
> queuing messages that can't be sent to the client right away.  Plus the
> above-mentioned refactoring of where transaction abort happens.

This is along the lines of what Andres and I have already been groping
towards upthread.  But the question of what to do if another error is
encountered while trying to abort the transaction is one that I also
thought about, and I don't see an easy solution.  I suppose we could
upgrade such an error to FATAL, given that right now we throw an
*unconditional* FATAL here when DoingCommandRead.   That's not
super-appealing, but it might be the most practical solution.

Another thing I don't quite understand is - at what point does the
protocol allow us to emit an error?  Suppose that the transaction gets
cancelled due to a conflict with recovery while we're
DoingCommandRead, and then the user now sends us "SELCT 2+2".  Are we
going to send them back both errors now, or just one of them?  Which
one?

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  31. Re: [PATCH] V3: Idle in transaction cancellation

    Tom Lane <tgl@sss.pgh.pa.us> — 2010-12-16T17:46:43Z

    Robert Haas <robertmhaas@gmail.com> writes:
> On Thu, Dec 16, 2010 at 11:58 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> It's possible we could refactor things so we abort the open transaction
>> while inside the interrupt handler, then queue up an error to be
>> reported whenever we next get a command (as envisioned in part 0003),
>> then just return control back to the input stack. But things could get
>> messy if we get another error to be reported while trying to abort.

> This is along the lines of what Andres and I have already been groping
> towards upthread.  But the question of what to do if another error is
> encountered while trying to abort the transaction is one that I also
> thought about, and I don't see an easy solution.

Yeah, it's a bit messy, because you really can't send multiple ERROR
messages to the client when it next sends a query: the protocol says
there'll be at most one.  We could either discard all but the first
(or last?) of the queued error events, or add code to elog.c that
somehow merges them into one error event, perhaps by adding info
to the DETAIL field.  I'm handwaving there --- I think probably the
first cut should just discard errors after the first, and see how
well that works in practice.

> Another thing I don't quite understand is - at what point does the
> protocol allow us to emit an error?

Basically, you can send an error in response to a query.  In the current
FATAL cases, we're cheating a bit by sending something while idle ---
what will typically happen at the client end is that it will not notice
the input until it sends a query, and then it will see the
already-pending input, which it will think is a (remarkably fast)
response to its query.  Or possibly it will bomb out on send() failure
and not ever consume the input at all.  But if we want to keep the
connection going, we can't cheat like that.

> Suppose that the transaction gets
> cancelled due to a conflict with recovery while we're
> DoingCommandRead, and then the user now sends us "SELCT 2+2".  Are we
> going to send them back both errors now, or just one of them?  Which
> one?

You can only send one, and in that situation you probably want the
cancellation to be reported.

FWIW, I'm not too worried about preserving the existing
recovery-conflict behavior, as I think the odds are at least ten to one
that that code is broken when you look closely enough.  I do like the
idea that this patch would provide a better-thought-out framework for
handling the conflict case.

			regards, tom lane

  32. Re: [PATCH] V3: Idle in transaction cancellation

    Robert Haas <robertmhaas@gmail.com> — 2010-12-16T18:03:14Z

    On Thu, Dec 16, 2010 at 12:46 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> I'm handwaving there --- I think probably the
> first cut should just discard errors after the first, and see how
> well that works in practice.

Seems reasonable.

>> Another thing I don't quite understand is - at what point does the
>> protocol allow us to emit an error?
>
> Basically, you can send an error in response to a query.

What about some other message that's not a query?

>> Suppose that the transaction gets
>> cancelled due to a conflict with recovery while we're
>> DoingCommandRead, and then the user now sends us "SELCT 2+2".  Are we
>> going to send them back both errors now, or just one of them?  Which
>> one?
>
> You can only send one, and in that situation you probably want the
> cancellation to be reported.

What about an elog or ereport with severity < ERROR?  Surely there
must at least be provision for multiple non-error messages per
transaction.

> FWIW, I'm not too worried about preserving the existing
> recovery-conflict behavior, as I think the odds are at least ten to one
> that that code is broken when you look closely enough.  I do like the
> idea that this patch would provide a better-thought-out framework for
> handling the conflict case.

We already have pg_terminate_backend() and pg_cancel_backend().  Are
you imagining a general mechanism like pg_rollback_backend()?

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  33. Re: [PATCH] V3: Idle in transaction cancellation

    Tom Lane <tgl@sss.pgh.pa.us> — 2010-12-16T18:24:50Z

    Robert Haas <robertmhaas@gmail.com> writes:
> On Thu, Dec 16, 2010 at 12:46 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>> Another thing I don't quite understand is - at what point does the
>>> protocol allow us to emit an error?

>> Basically, you can send an error in response to a query.

> What about some other message that's not a query?

There aren't any (I'm using a loose definition of "query" here --- any
client request counts).

>> You can only send one, and in that situation you probably want the
>> cancellation to be reported.

> What about an elog or ereport with severity < ERROR?  Surely there
> must at least be provision for multiple non-error messages per
> transaction.

You can send NOTICEs freely, but downgrading an error to a notice is
probably not a great solution --- keep in mind that some clients just
discard those altogether.

>> FWIW, I'm not too worried about preserving the existing
>> recovery-conflict behavior, as I think the odds are at least ten to one
>> that that code is broken when you look closely enough. I do like the
>> idea that this patch would provide a better-thought-out framework for
>> handling the conflict case.

> We already have pg_terminate_backend() and pg_cancel_backend().  Are
> you imagining a general mechanism like pg_rollback_backend()?

No, not really, I'm just concerned about the fact that it's trying to
send a message while in DoingCommandRead state.  FE/BE protocol
considerations aside, that's likely to break if using SSL, because who
knows where we've interrupted openssl.  In fairness, the various
pre-existing FATAL-interrupt cases have that problem already, but I was
willing to live with it for things that don't happen during normal
operation.

			regards, tom lane

  34. Re: [PATCH] V3: Idle in transaction cancellation

    Robert Haas <robertmhaas@gmail.com> — 2010-12-16T20:10:31Z

    On Thu, Dec 16, 2010 at 1:24 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Robert Haas <robertmhaas@gmail.com> writes:
>> On Thu, Dec 16, 2010 at 12:46 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>>> Another thing I don't quite understand is - at what point does the
>>>> protocol allow us to emit an error?
>
>>> Basically, you can send an error in response to a query.
>
>> What about some other message that's not a query?
>
> There aren't any (I'm using a loose definition of "query" here --- any
> client request counts).

OK.

>>> You can only send one, and in that situation you probably want the
>>> cancellation to be reported.
>
>> What about an elog or ereport with severity < ERROR?  Surely there
>> must at least be provision for multiple non-error messages per
>> transaction.
>
> You can send NOTICEs freely, but downgrading an error to a notice is
> probably not a great solution --- keep in mind that some clients just
> discard those altogether.

Yeah, I wasn't proposing that, just trying to understand the rules.

>>> FWIW, I'm not too worried about preserving the existing
>>> recovery-conflict behavior, as I think the odds are at least ten to one
>>> that that code is broken when you look closely enough.  I do like the
>>> idea that this patch would provide a better-thought-out framework for
>>> handling the conflict case.
>
>> We already have pg_terminate_backend() and pg_cancel_backend().  Are
>> you imagining a general mechanism like pg_rollback_backend()?
>
> No, not really, I'm just concerned about the fact that it's trying to
> send a message while in DoingCommandRead state.  FE/BE protocol
> considerations aside, that's likely to break if using SSL, because who
> knows where we've interrupted openssl.  In fairness, the various
> pre-existing FATAL-interrupt cases have that problem already, but I was
> willing to live with it for things that don't happen during normal
> operation.

Hmm.  It's seeming to me that what we want to do is something like this:

1. If an error is thrown while DoingCommandRead, it gets upgraded to
FATAL.  I don't think we have much choice about this because, per your
previous comments, we can't longjmp() here without risking protocol
breakage, and we certainly can't return from an elog(ERROR) or
ereport(ERROR).

2. If a recovery conflict arrives while DoingCommandRead(), we
AbortCurrentTransaction().  If this runs into unexpected trouble,
it'll turn into a FATAL per #1.  If it completes successfully, then
we'll set a flag indicating that upon emerging from DoingCommandRead
state, we need to signal the recovery conflict to the client.

3. When we clear DoingCommandRead, we'll check whether the flag is set
and if so ereport(ERROR).

Step #2 seems like the dangerous part, but I'm not immediately sure
what hazards may be lurking there.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  35. Re: [PATCH] V3: Idle in transaction cancellation

    Tom Lane <tgl@sss.pgh.pa.us> — 2010-12-16T20:18:58Z

    Robert Haas <robertmhaas@gmail.com> writes:
> Hmm.  It's seeming to me that what we want to do is something like this:

> 1. If an error is thrown while DoingCommandRead, it gets upgraded to
> FATAL.  I don't think we have much choice about this because, per your
> previous comments, we can't longjmp() here without risking protocol
> breakage, and we certainly can't return from an elog(ERROR) or
> ereport(ERROR).

Um, if that's the ground rules then we have no advance over the current
situation.

I guess you misunderstood what I said.  What I meant was that we cannot
longjmp *out to the outer level*, ie we cannot take control away from
the input stack.  We could however have a TRY block inside the interrupt
handler that catches and handles (queues) any errors occurring during
transaction abort.  As long as we eventually return control to openssl
I think it should work.  (Hm, but I wonder whether there are any hard
timing constraints in the ssl protocol ... although hopefully xact abort
won't ever take long enough that that's a real problem.)

			regards, tom lane

  36. Re: [PATCH] V3: Idle in transaction cancellation

    Robert Haas <robertmhaas@gmail.com> — 2010-12-16T20:32:17Z

    On Thu, Dec 16, 2010 at 3:18 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Robert Haas <robertmhaas@gmail.com> writes:
>> Hmm.  It's seeming to me that what we want to do is something like this:
>
>> 1. If an error is thrown while DoingCommandRead, it gets upgraded to
>> FATAL.  I don't think we have much choice about this because, per your
>> previous comments, we can't longjmp() here without risking protocol
>> breakage, and we certainly can't return from an elog(ERROR) or
>> ereport(ERROR).
>
> Um, if that's the ground rules then we have no advance over the current
> situation.
>
> I guess you misunderstood what I said.  What I meant was that we cannot
> longjmp *out to the outer level*, ie we cannot take control away from
> the input stack.  We could however have a TRY block inside the interrupt
> handler that catches and handles (queues) any errors occurring during
> transaction abort.  As long as we eventually return control to openssl
> I think it should work.

Is there any real advantage to that?  How often do we hit an error
trying to abort a transaction?  And how will we report the error
anyway?  I thought the next thing we'd report would be the recovery
conflict, not any bizarre can't-abort-the-transaction scenario.

> (Hm, but I wonder whether there are any hard
> timing constraints in the ssl protocol ... although hopefully xact abort
> won't ever take long enough that that's a real problem.)

That would be incredibly broken.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  37. Re: [PATCH] V3: Idle in transaction cancellation

    Tom Lane <tgl@sss.pgh.pa.us> — 2010-12-16T20:41:10Z

    Robert Haas <robertmhaas@gmail.com> writes:
> On Thu, Dec 16, 2010 at 3:18 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> I guess you misunderstood what I said. What I meant was that we cannot
>> longjmp *out to the outer level*, ie we cannot take control away from
>> the input stack. We could however have a TRY block inside the interrupt
>> handler that catches and handles (queues) any errors occurring during
>> transaction abort. As long as we eventually return control to openssl
>> I think it should work.

> Is there any real advantage to that?

Not crashing when something funny happens seems like a real advantage to
me.  (And an unexpected elog(FATAL) will look like a crash to most
users, even if you want to try to define it as not a crash.)

> How often do we hit an error
> trying to abort a transaction?  And how will we report the error
> anyway?

Queue it up and report it at the next opportunity, as per upthread.

> I thought the next thing we'd report would be the recovery
> conflict, not any bizarre can't-abort-the-transaction scenario.

Well, if we discard it because we're too lazy to implement error message
merging, that's OK.  Presumably it'll still get into the postmaster log.

>> (Hm, but I wonder whether there are any hard
>> timing constraints in the ssl protocol ... although hopefully xact abort
>> won't ever take long enough that that's a real problem.)

> That would be incredibly broken.

Think "authentication timeout".  I wouldn't be a bit surprised if the
remote end would drop the connection if certain events didn't come back
reasonably promptly.  There might even be security reasons for that,
ie, somebody could brute-force a key if you give them long enough.
(But this is all speculation; I don't actually know SSL innards.)

			regards, tom lane

  38. Re: [PATCH] V3: Idle in transaction cancellation

    Robert Haas <robertmhaas@gmail.com> — 2010-12-16T20:49:38Z

    On Thu, Dec 16, 2010 at 3:41 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> I thought the next thing we'd report would be the recovery
>> conflict, not any bizarre can't-abort-the-transaction scenario.
>
> Well, if we discard it because we're too lazy to implement error message
> merging, that's OK.  Presumably it'll still get into the postmaster log.

OK, that's reasonable.

>>> (Hm, but I wonder whether there are any hard
>>> timing constraints in the ssl protocol ... although hopefully xact abort
>>> won't ever take long enough that that's a real problem.)
>
>> That would be incredibly broken.
>
> Think "authentication timeout".  I wouldn't be a bit surprised if the
> remote end would drop the connection if certain events didn't come back
> reasonably promptly.  There might even be security reasons for that,
> ie, somebody could brute-force a key if you give them long enough.
> (But this is all speculation; I don't actually know SSL innards.)

I would be really surprised if aborting a transaction takes long
enough to mess up SSL.  I mean, there could be a network delay at any
time, too.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  39. Re: [PATCH] V3: Idle in transaction cancellation

    Tom Lane <tgl@sss.pgh.pa.us> — 2010-12-16T20:54:51Z

    Robert Haas <robertmhaas@gmail.com> writes:
> On Thu, Dec 16, 2010 at 3:41 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> (But this is all speculation; I don't actually know SSL innards.)

> I would be really surprised if aborting a transaction takes long
> enough to mess up SSL.  I mean, there could be a network delay at any
> time, too.

Yeah, that's what I think too.  I was just wondering if there could be
any situation where xact abort takes minutes.

			regards, tom lane

  40. Re: [PATCH] V3: Idle in transaction cancellation

    Alvaro Herrera <alvherre@commandprompt.com> — 2010-12-16T21:30:02Z

    Excerpts from Tom Lane's message of jue dic 16 17:54:51 -0300 2010:
> Robert Haas <robertmhaas@gmail.com> writes:
> > On Thu, Dec 16, 2010 at 3:41 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> >> (But this is all speculation; I don't actually know SSL innards.)
> 
> > I would be really surprised if aborting a transaction takes long
> > enough to mess up SSL.  I mean, there could be a network delay at any
> > time, too.
> 
> Yeah, that's what I think too.  I was just wondering if there could be
> any situation where xact abort takes minutes.

Maybe if it's transaction commit there could be a long queue of pending
deferred triggers, but on abort, those things are discarded.

-- 
Álvaro Herrera <alvherre@commandprompt.com>
The PostgreSQL Company - Command Prompt, Inc.
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

  41. Re: [PATCH] V3: Idle in transaction cancellation

    Andres Freund <andres@anarazel.de> — 2010-12-16T22:55:36Z

    On Thursday 16 December 2010 21:41:10 Tom Lane wrote:
> Robert Haas <robertmhaas@gmail.com> writes:
> > On Thu, Dec 16, 2010 at 3:18 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> >> I guess you misunderstood what I said. �What I meant was that we cannot
> >> longjmp *out to the outer level*, ie we cannot take control away from
> >> the input stack. �We could however have a TRY block inside the interrupt
> >> handler that catches and handles (queues) any errors occurring during
> >> transaction abort. �As long as we eventually return control to openssl
> >> I think it should work.
> > 
> > Is there any real advantage to that?
> 
> Not crashing when something funny happens seems like a real advantage to
> me.  (And an unexpected elog(FATAL) will look like a crash to most
> users, even if you want to try to define it as not a crash.)
> 
> > How often do we hit an error
> > trying to abort a transaction?  And how will we report the error
> > anyway?
> 
> Queue it up and report it at the next opportunity, as per upthread.
> 
> > I thought the next thing we'd report would be the recovery
> > conflict, not any bizarre can't-abort-the-transaction scenario.
> 
> Well, if we discard it because we're too lazy to implement error message
> merging, that's OK.  Presumably it'll still get into the postmaster log.
> 
> >> (Hm, but I wonder whether there are any hard
> >> timing constraints in the ssl protocol ... although hopefully xact abort
> >> won't ever take long enough that that's a real problem.)
> > 
> > That would be incredibly broken.
> 
> Think "authentication timeout".  I wouldn't be a bit surprised if the
> remote end would drop the connection if certain events didn't come back
> reasonably promptly.  There might even be security reasons for that,
> ie, somebody could brute-force a key if you give them long enough.
> (But this is all speculation; I don't actually know SSL innards.)
I will try to read the thread and make a proposal for a more carefull 
implementation - just not today... I think the results would be interesting...

Thanks for the input,

Andres

  42. Re: [PATCH] V3: Idle in transaction cancellation

    Kevin Grittner <kevin.grittner@wicourts.gov> — 2010-12-23T22:04:43Z

    Andres Freund <andres@anarazel.de> wrote:
 
> I will try to read the thread and make a proposal for a more
> carefull implementation - just not today... I think the results
> would be interesting...
 
FWIW, the SSI patch that Dan and I are working on can't have a
guarantee that it is immediately safe to retry a transaction which
rolls back with a serialization failure (without potentially failing
again on exactly the same set of transactions) unless there is a
capability such as this "Idle in transaction cancellation" patch
would provide.  Safe immediate retry would be a nice guarantee for
SSI to provide.
 
That being the case, we may not be able to generate the final form
of the SSI patch until a patch for this issue is applied.  Obviously
I know that nobody is in a position to make any promises on this,
but I just thought I'd let people know that this issue could
possibly be on the critical path to a timely release -- at least if
that release will include SSI with the safe retry guarantee.  (At
least when I'm planning for a release, I like to know such
things....)
 
-Kevin

  43. Re: [PATCH] V3: Idle in transaction cancellation

    Alvaro Herrera <alvherre@commandprompt.com> — 2011-01-03T14:38:56Z

    Is anybody working on this patch?


-- 
Álvaro Herrera <alvherre@commandprompt.com>
The PostgreSQL Company - Command Prompt, Inc.
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

  44. Re: [PATCH] V3: Idle in transaction cancellation

    Kevin Grittner <kevin.grittner@wicourts.gov> — 2011-01-03T14:47:31Z

    Alvaro Herrera <alvherre@commandprompt.com> wrote:
 
> Is anybody working on this patch?
 
I'm not, but I sure hope someone is -- we could *really* use this in
the SSI patch.  With something providing the equivalent
functionality to Andres's previous patch, and about one day's work
in the SSI patch, SSI could guarantee that an immediate retry of a
transaction rolled back with a serialization failure would not fail
again on conflicts with the same transaction(s).  This would be a
very nice guarantee to be able to make.
 
-Kevin

  45. Re: [PATCH] V3: Idle in transaction cancellation

    Andres Freund <andres@anarazel.de> — 2011-01-03T15:03:58Z

    On Monday, January 03, 2011 03:38:56 PM Alvaro Herrera wrote:
> Is anybody working on this patch?
I am. Wont be finished in the next two days though (breakin last night...)

Andres


PS: Alvarro: commandprompt.com doesn't resolv anymore, so I can't send you the 
email directly...

  46. Re: [PATCH] V3: Idle in transaction cancellation

    Alvaro Herrera <alvherre@commandprompt.com> — 2011-01-04T03:25:22Z

    Excerpts from Andres Freund's message of lun ene 03 12:03:58 -0300 2011:
> On Monday, January 03, 2011 03:38:56 PM Alvaro Herrera wrote:
> > Is anybody working on this patch?
> I am. Wont be finished in the next two days though (breakin last night...)

Okay ... I will be moving to a new house this week anyway :-P

> PS: Alvarro: commandprompt.com doesn't resolv anymore, so I can't send you the 
> email directly...

You gotta be kidding --- hmm, oops!  Let me find the flamethrower ...

(I guess it's a good thing that my suscription to the list uses a
different email address)

-- 
Álvaro Herrera <alvherre@commandprompt.com>
The PostgreSQL Company - Command Prompt, Inc.
PostgreSQL Replication, Consulting, Custom Development, 24x7 support