Horton

Thread

  1. Coping with backend crash in libpq

    Tom Lane <tgl@sss.pgh.pa.us> — 1998-07-28T17:23:35Z

    I've just noticed that libpq doesn't cope very gracefully if the backend
exits when not in the middle of a query (ie, because the postmaster told
it to quit after some other BE crashed).  The behavior in psql, for
example, is that the next time you issue a query, psql just exits
without printing anything at all.  This is Not Friendly, especially
considering that the BE sent a nice little notice message before it quit.

The main problem is that if the next thing you do is to send a new query,
send() sees that the connection has been closed and generates a SIGPIPE
signal.  By default that terminates the frontend process.

We could cure this by having libpq disable SIGPIPE, but we would have
to disable it before each send() and re-enable afterwards to avoid
affecting the behavior of the rest of the frontend application.
Two additional kernel calls per query sounds like a lot of overhead.
(We do actually do this when trying to close the connection, but not
during normal queries.)

Perhaps a better answer is to have PQsendQuery check for fresh input
from the backend before trying to send the query.  This would have two
side effects:
  1. If a NOTICE message has arrived, we could print it.
  2. If EOF is detected, we will reset the connection state to
     CONNECTION_BAD, which PQsendQuery can use to avoid trying to send.

The minimum cost to do this is one kernel call (a select(), which
unfortunately is probably a fairly expensive call) in the normal
case where no new input has arrived.  Another objection is that it's
not 100% bulletproof --- if the backend closes the connection in the
window between select() and send() then you can still get SIGPIPE'd.
The odds of this seem pretty small however.

I'm inclined to go with answer #2, because it seems to have less
of a performance impact, and it will ensure that the backend's polite
"The Postmaster has informed me that some other backend died abnormally
and possibly corrupted shared memory." message gets displayed.  With
approach #1 we'd still have to go through some pushups to get the
notice to come out.

Does anyone have an objection, or a better idea?

			regards, tom lane

  2. Re: [INTERFACES] Coping with backend crash in libpq

    Karl Denninger <karl@mcs.net> — 1998-07-28T17:44:59Z

    On Tue, Jul 28, 1998 at 01:23:35PM -0400, Tom Lane wrote:
> I've just noticed that libpq doesn't cope very gracefully if the backend
> exits when not in the middle of a query (ie, because the postmaster told
> it to quit after some other BE crashed).  The behavior in psql, for
> example, is that the next time you issue a query, psql just exits
> without printing anything at all.  This is Not Friendly, especially
> considering that the BE sent a nice little notice message before it quit.
> 
> The main problem is that if the next thing you do is to send a new query,
> send() sees that the connection has been closed and generates a SIGPIPE
> signal.  By default that terminates the frontend process.
> 
> We could cure this by having libpq disable SIGPIPE, but we would have
> to disable it before each send() and re-enable afterwards to avoid
> affecting the behavior of the rest of the frontend application.
> Two additional kernel calls per query sounds like a lot of overhead.
> (We do actually do this when trying to close the connection, but not
> during normal queries.)
> 
> Perhaps a better answer is to have PQsendQuery check for fresh input
> from the backend before trying to send the query.  This would have two
> side effects:
>   1. If a NOTICE message has arrived, we could print it.
>   2. If EOF is detected, we will reset the connection state to
>      CONNECTION_BAD, which PQsendQuery can use to avoid trying to send.
> 
> The minimum cost to do this is one kernel call (a select(), which
> unfortunately is probably a fairly expensive call) in the normal
> case where no new input has arrived.  Another objection is that it's
> not 100% bulletproof --- if the backend closes the connection in the
> window between select() and send() then you can still get SIGPIPE'd.
> The odds of this seem pretty small however.
> 
> I'm inclined to go with answer #2, because it seems to have less
> of a performance impact, and it will ensure that the backend's polite
> "The Postmaster has informed me that some other backend died abnormally
> and possibly corrupted shared memory." message gets displayed.  With
> approach #1 we'd still have to go through some pushups to get the
> notice to come out.
> 
> Does anyone have an objection, or a better idea?
> 
> 			regards, tom lane
> 

Not really.

I've noticed this kind of problem where the backend will fault in some way, 
and after it does so, the library gets "confused".

We have a couple of processes here that are NEVER supposed to exit.  They
open a connection for each transaction, and close it at the end.  If
something happens to the backend where it dies abnormally, these processes
will sometimes get into an odd state in the libpq library where all new
connection attempts fail immediately.

I've yet to find a foolproof coding way around this particular problem.

--
-- 
Karl Denninger (karl@MCS.Net)| MCSNet - Serving Chicagoland and Wisconsin
http://www.mcs.net/          | T1's from $600 monthly / All Lines K56Flex/DOV
			     | NEW! Corporate ISDN Prices dropped by up to 50%!
Voice: [+1 312 803-MCS1 x219]| EXCLUSIVE NEW FEATURE ON ALL PERSONAL ACCOUNTS
Fax:   [+1 312 803-4929]     | *SPAMBLOCK* Technology now included at no cost

  3. Re: [HACKERS] Coping with backend crash in libpq

    Bruce Momjian <maillist@candle.pha.pa.us> — 1998-07-29T04:59:32Z

    > I've just noticed that libpq doesn't cope very gracefully if the backend
> exits when not in the middle of a query (ie, because the postmaster told
> it to quit after some other BE crashed).  The behavior in psql, for
> example, is that the next time you issue a query, psql just exits
> without printing anything at all.  This is Not Friendly, especially
> considering that the BE sent a nice little notice message before it quit.

I say, install the signal handler for SIGPIPE on connection startup, but
when you install it, it returns the previous defined action.  If we find
there was a previous defined action, we can re-install theirs, and let
it handle the sigpipe.  If an application later defines it's own
sigpipe, over-riding ours, then they get no error message.

However, I see psql setting the SIGPIPE handler all over the place, so I
don't think that will work there.  How about SIGURG?  Oops, not portable
for unix domain sockets.  Can we send a signal to the process, telling
it the backend has exited.  We have that information now, so why not use
it.  Define a signal handler for SIGURG or SIGUSR1, and have that print
out a message.  If the app redefines that, it will get confused when we
send the signal from the postmaster.  Oops, we can't send signals to the
client because they may be owned by other users.

I am stumped.  Let me think about it.



-- 
Bruce Momjian                          |  830 Blythe Avenue
maillist@candle.pha.pa.us              |  Drexel Hill, Pennsylvania 19026
  +  If your life is a hard drive,     |  (610) 353-9879(w)
  +  Christ can be your backup.        |  (610) 853-3000(h)

  4. Re: [HACKERS] Coping with backend crash in libpq

    David Gould <dg@illustra.com> — 1998-07-29T05:16:02Z

    > > I've just noticed that libpq doesn't cope very gracefully if the backend
> > exits when not in the middle of a query (ie, because the postmaster told
> > it to quit after some other BE crashed).  The behavior in psql, for
> > example, is that the next time you issue a query, psql just exits
> > without printing anything at all.  This is Not Friendly, especially
> > considering that the BE sent a nice little notice message before it quit.
> 
> I say, install the signal handler for SIGPIPE on connection startup, but
> when you install it, it returns the previous defined action.  If we find
> there was a previous defined action, we can re-install theirs, and let
> it handle the sigpipe.  If an application later defines it's own
> sigpipe, over-riding ours, then they get no error message.
> 
> However, I see psql setting the SIGPIPE handler all over the place, so I
> don't think that will work there.  How about SIGURG?  Oops, not portable
> for unix domain sockets.  Can we send a signal to the process, telling
> it the backend has exited.  We have that information now, so why not use
> it.  Define a signal handler for SIGURG or SIGUSR1, and have that print
> out a message.  If the app redefines that, it will get confused when we
> send the signal from the postmaster.  Oops, we can't send signals to the
> client because they may be owned by other users.
> 
> I am stumped.  Let me think about it.

Hmmm, perhaps fix psql so that it uses SIGPIPE more sensibly. SIGPIPE really
is the right signal to catch here. 

-dg

David Gould            dg@illustra.com           510.628.3783 or 510.305.9468 
Informix Software  (No, really)         300 Lakeside Drive  Oakland, CA 94612
 - If simplicity worked, the world would be overrun with insects. -