clarifying a few error messages

Thomas F.O'Connell <tfo@monsterlabs.com>

From: Thomas O'Connell <tfo@monsterlabs.com>
To: pgsql-general@postgresql.org
Date: 2003-01-09T21:10:43Z
Lists: pgsql-general
i'm hoping someone might be able to help me understand some of what 
might been going on in the environment external to postgres based on 
some error messages i just got in the logs of one of my pg installations.

in a period of duress (i.e., the box itself was rebooting and postgres 
was dying) on one of my servers, i saw the following a few times:

ERROR:  deadlock detected

shortly afterward, there was this:

DEBUG:  database system was interrupted at <timestamp>

what can interrupt the database? i've seen it get terminated by signal 9 
when the box is failing, but is there any way to know what might have 
interrupted it? it seems to have died altogether, as it then goes 
through the checkpoint record, etc.

after it starts up, this:

DEBUG:  database system is ready
DEBUG:  server process (pid 882) was terminated by signal 11

does this mean postgres itself segfaulted or that it received an 
external SIGSEGV from a critical system process (e.g., the kernel)?

also, is there any significance to the fact that those two statements 
occurred one right after the other?

a little later, after another recovery, i see these:

DEBUG:  all server processes terminated; reinitializing shared memory 
and semaphores
DEBUG:  database system was interrupted at <timestamp>

finally, it seems to stabilize for a bit. then, a little later, a whole 
spew of garbage characters in the log immediately preceding another:

DEBUG:  database system was interrupted at <timestamp>

how would garbage data end up in the log? does that indicate anything 
about the manner in which postgres was interrupted?

after yet another recovery, i see the following:

FATAL 1:  The database system is starting up
FATAL 2:  open of $PGDATA/pg_clog/0419 failed: No such file or directory
FATAL 2:  open of $PGDATA/pg_clog/0419 failed: No such file or directory
FATAL 2:  open of $PGDATA/pg_clog/0419 failed: No such file or directory
FATAL 2:  open of $PGDATA/pg_clog/0419 failed: No such file or directory
FATAL 2:  open of $PGDATA/pg_clog/0419 failed: No such file or directory
DEBUG:  server process (pid 945) exited with exit code 2

here, what is exit code 2? does that just mean that postgres found a 
significant problem with clog files?

then, there was one more of the shared memory/interruption pairs, a 
final recovery, and smooth sailing again.

i guess i'd like to be able to determine if this is a system resources 
issue, and if so, which system resources. is this sequence something 
that can be prevented in the future via postgresql.conf? more memory?

is there any way of knowing what actually brought down postgres from 
these messages?

this installation is on a Linux box running kernel 2.4.18 with 1GB RAM.

i can provide postgresql.conf settings upon request.

thanks!

-tfo