Horton

Re: Hot Backup with rsync fails at pg_clog if under load

Chris Redekop <chris@replicon.com>

View thread
From: Chris Redekop <chris@replicon.com>
To: Simon Riggs <simon@2ndquadrant.com>
Cc: Florian Pflug <fgp@phlo.org>, Daniel Farina <daniel@heroku.com>, PostgreSQL-development <pgsql-hackers@postgresql.org>
Date: 2011-10-26T14:51:30Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Derive oldestActiveXid at correct time for Hot Standby.

  2. Start Hot Standby faster when initial snapshot is incomplete.

  3. Fix timing of Startup CLOG and MultiXact during Hot Standby

Attachments

  • 000A (application/octet-stream) 
>
> The way forwards here is for someone to show the clog file that causes
> the error and find out why the call to read() fails.


Well let me hook ya up :)  Here's the log, controldata, and erroring clog
attached


2011-10-26 08:44:15.419 MDT [1544]: [1-1] LOG:  database system was
interrupted; last known up at 2011-10-26 08:43:54 MDT
2011-10-26 08:44:15.419 MDT [1544]: [2-1] LOG:  creating missing WAL
directory "pg_xlog/archive_status"
2011-10-26 08:44:15.421 MDT [1544]: [3-1] LOG:  entering standby mode
DEBUG:  received replication command: IDENTIFY_SYSTEM
DEBUG:  received replication command: START_REPLICATION 6/CE000000
2011-10-26 08:44:15.425 MDT [1545]: [1-1] LOG:  streaming replication
successfully connected to primary
2011-10-26 08:44:16.246 MDT [1547]: [1-1] FATAL:  the database system is
starting up
2011-10-26 08:44:17.501 MDT [1544]: [4-1] DEBUG:  checkpoint record is at
6/D0BC36A0
2011-10-26 08:44:17.501 MDT [1544]: [5-1] DEBUG:  redo record is at
6/CE0010E0; shutdown FALSE
2011-10-26 08:44:17.501 MDT [1544]: [6-1] DEBUG:  next transaction ID:
0/10846655; next OID: 2826628
2011-10-26 08:44:17.501 MDT [1544]: [7-1] DEBUG:  next MultiXactId: 1741;
next MultiXactOffset: 3498
2011-10-26 08:44:17.501 MDT [1544]: [8-1] DEBUG:  oldest unfrozen
transaction ID: 1669, in database 1
2011-10-26 08:44:17.501 MDT [1544]: [9-1] DEBUG:  transaction ID wrap limit
is 2147485316, limited by database with OID 1
2011-10-26 08:44:17.742 MDT [1544]: [10-1] DEBUG:  resetting unlogged
relations: cleanup 1 init 0
2011-10-26 08:44:17.751 MDT [1544]: [11-1] DEBUG:  initializing for hot
standby
2011-10-26 08:44:17.751 MDT [1544]: [12-1] FATAL:  could not access status
of transaction 10846655
2011-10-26 08:44:17.751 MDT [1544]: [13-1] DETAIL:  Could not read from file
"pg_clog/000A" at offset 90112: Success.
2011-10-26 08:44:17.751 MDT [1537]: [2-1] LOG:  startup process (PID 1544)
exited with exit code 1
2011-10-26 08:44:17.751 MDT [1537]: [3-1] LOG:  aborting startup due to
startup process failure
2011-10-26 08:44:17.843 MDT [1543]: [1-1] DEBUG:  logger shutting down



pg_control version number:            903
Catalog version number:               201105231
Database system identifier:           5667259861501982685
Database cluster state:               in production
pg_control last modified:             Wed 26 Oct 2011 08:43:54 AM MDT
Latest checkpoint location:           6/D0BC36A0
Prior checkpoint location:            6/877B9950
Latest checkpoint's REDO location:    6/CE0010E0
Latest checkpoint's TimeLineID:       1
Latest checkpoint's NextXID:          0/10846655
Latest checkpoint's NextOID:          2826628
Latest checkpoint's NextMultiXactId:  1741
Latest checkpoint's NextMultiOffset:  3498
Latest checkpoint's oldestXID:        1669
Latest checkpoint's oldestXID's DB:   1
Latest checkpoint's oldestActiveXID:  10845042
Time of latest checkpoint:            Wed 26 Oct 2011 08:43:44 AM MDT
Minimum recovery ending location:     0/0
Backup start location:                0/0
Current wal_level setting:            hot_standby
Current max_connections setting:      600
Current max_prepared_xacts setting:   0
Current max_locks_per_xact setting:   256
Maximum data alignment:               8
Database block size:                  8192
Blocks per segment of large relation: 131072
WAL block size:                       8192
Bytes per WAL segment:                16777216
Maximum length of identifiers:        64
Maximum columns in an index:          32
Maximum size of a TOAST chunk:        1996
Date/time type storage:               64-bit integers
Float4 argument passing:              by value
Float8 argument passing:              by value