Re: max_standby_delay considered harmful

On Wed, 2010-05-12 at 21:10 +0200, Stefan Kaltenbrunner wrote:

> > There is no evidence to link this behaviour with HS, as yet, and you
> > should be considering the possibility the problem lies elsewhere,
> > especially since it could be code you committed that is at fault.
> 
> Well I'm not sure why people seem to have that hard a time reproducing 
> that issue - it seems that I can provoke it really trivially(in this 
> case no loops, no pgbench, no tricks). A few minutes ago I logged into 
> my test standby (which is idle except for the odd connect to template1 
> caused by nagios - the master is idle as well and has been for days):

Thanks, good report.

> so it restarted two times successfully - however if one looks at the 
> third time one can see that it received the smart shutdown request 
> BEFORE it reached a consistent recovery state - yet it continued to 
> enable HS and reenabled SR as well.
> 
> The database is now sitting there doing nothing and it more or less 
> broken because you cannot connect to it in the current state:
> 
> ~$ psql
> psql: FATAL:  the database system is shutting down
> 
> the startup process has the following backtrace:
> 
> (gdb) bt
> #0  0x00007fbe24cb2c83 in select () from /lib/libc.so.6
> #1  0x00000000006e811a in pg_usleep ()
> #2  0x000000000048c333 in XLogPageRead ()
> #3  0x000000000048c967 in ReadRecord ()
> #4  0x0000000000493ab6 in StartupXLOG ()
> #5  0x0000000000495a88 in StartupProcessMain ()
> #6  0x00000000004ab25e in AuxiliaryProcessMain ()
> #7  0x00000000005d4a7d in StartChildProcess ()
> #8  0x00000000005d70c2 in PostmasterMain ()
> #9  0x000000000057d898 in main ()

Well, its waiting for new info from primary. Nothing to do with locking,
but that's not an indication that its an SR issue though either. ;-)

I'll put some waits into that part of the code and see if I can induce
the failure. Maybe its just a simple lack of a CHECK_FOR_INTERRUPTS().

-- 
 Simon Riggs           www.2ndQuadrant.com