Thread
Commits
-
Keep timeline history files restored from archive in pg_xlog.
- 60df192aea0e 9.3.0 cited
-
More subtle issues with cascading replication over timeline switches
Heikki Linnakangas <hlinnakangas@vmware.com> — 2013-01-18T11:57:54Z
When a standby starts up, and catches up with the master through the archive, it copies the target timeline's history file from the archive to pg_xlog. That's enough for that standby's purposes, but if there is a cascading standby or pg_receivexlog connected to it, it will request the timeline history files of *all* timelines between the starting point and current target. For example, create a master, and take a base backup from it. Use the base backup to initialize two standby servers. Now perform failover first to the first standby, and once the second standby has caught up, fail over again, to the second standby. (Or as a shortcut, forget about the standbys, and just create a recovery.conf file in the master with restore_command='/bin/false' and restart it. That causes a timeline switch. Repeat twice) Now use the base backup to initialize a new standby server (you can kill and delete the old ones), using the WAL archive. Set up a second, cascading, standby server that connects to the first standby using streaming replication, not WAL archive. This cascading standby will fail to cross the timeline switch, because it doesn't find all the history files in the standby: C 2013-01-18 13:38:46.047 EET 7695 FATAL: could not receive timeline history file from the primary server: ERROR: could not open file "pg_xlog/00000002.history": No such file or directory Indeed, looking at the pg_xlog, it's not there (I did a couple of extra timeline switches: ~/pgsql.master$ ls -l data-master/pg_xlog/ total 131084 -rw------- 1 heikki heikki 16777216 Jan 18 13:38 000000010000000000000001 -rw------- 1 heikki heikki 16777216 Jan 18 13:38 000000010000000000000002 -rw------- 1 heikki heikki 16777216 Jan 18 13:38 000000010000000000000003 -rw------- 1 heikki heikki 41 Jan 18 13:38 00000002.history -rw------- 1 heikki heikki 16777216 Jan 18 13:38 000000020000000000000003 -rw------- 1 heikki heikki 16777216 Jan 18 13:38 000000020000000000000004 -rw------- 1 heikki heikki 16777216 Jan 18 13:38 000000020000000000000005 -rw------- 1 heikki heikki 83 Jan 18 13:38 00000003.history -rw------- 1 heikki heikki 16777216 Jan 18 13:38 000000030000000000000005 -rw------- 1 heikki heikki 16777216 Jan 18 13:38 000000030000000000000006 drwx------ 2 heikki heikki 4096 Jan 18 13:38 archive_status ~/pgsql.master$ ls -l data-standbyB/pg_xlog/ total 81928 -rw------- 1 heikki heikki 16777216 Jan 18 13:38 000000010000000000000001 -rw------- 1 heikki heikki 16777216 Jan 18 13:38 000000010000000000000002 -rw------- 1 heikki heikki 16777216 Jan 18 13:38 000000020000000000000003 -rw------- 1 heikki heikki 16777216 Jan 18 13:38 000000020000000000000004 -rw------- 1 heikki heikki 83 Jan 18 13:38 00000003.history -rw------- 1 heikki heikki 16777216 Jan 18 13:38 000000030000000000000005 drwx------ 2 heikki heikki 4096 Jan 18 13:38 archive_status This can be thought of as another variant of the same issue that was fixed by commit 60df192aea0e6458f20301546e11f7673c102101. When standby B scans for the latest timeline, it finds it to be 3, and it reads the timeline history file for 3. After that patch, it also saves it in pg_xlog. It doesn't save the timeline history file for timeline 2, because that's included in the history of timeline 3. However, when standby C connects, it will try to fetch all the history files that it doesn't have, including 00000002.history, which throws the error. A related problem is that at the segment containing the timeline switch, standby has only restored from archive the WAL file of the new timeline, not the old one. For example above, the timeline switch 1 -> 2 happened while inserting to segment 000000010000000000000003, and a copy of that partial segment was created with the timeline's ID as 000000020000000000000003. The standby only has the segment from the new timeline in pg_xlog, which is enough for that standby's purposes, but will cause an error when the cascading standby tries to stream it: C 2013-01-18 13:46:12.334 EET 8579 FATAL: error reading result of streaming command: ERROR: requested WAL segment 000000010000000000000003 has already been removed A straightforward fix would be for the standby to restore those files that the cascading standby needs from the WAL archive, even if they're not strictly required for that standby itself. But actually, isn't it a bad idea that we store the partial segment, 000000010000000000000003 in this case, in the WAL archive? There's no way to tell that it's partial, and it can clash with a complete segment if more WAL is generated on that timeline. I just argued that pg_receivexlog should not do that, and hence keep the .partial suffix in the same situation, in http://www.postgresql.org/message-id/50F56245.8050802@vmware.com. This needs some more thought. I'll try to come up with something next week, but if anyone has any ideas.. - Heikki
-
Re: More subtle issues with cascading replication over timeline switches
Amit Kapila <amit.kapila@huawei.com> — 2013-01-19T12:26:47Z
On Friday, January 18, 2013 5:27 PM Heikki Linnakangas wrote: > Indeed, looking at the pg_xlog, it's not there (I did a couple of extra > timeline switches: > ~/pgsql.master$ ls -l data-master/pg_xlog/ > total 131084 > -rw------- 1 heikki heikki 16777216 Jan 18 13:38 000000010000000000000001 > -rw------- 1 heikki heikki 16777216 Jan 18 13:38 000000010000000000000002 > -rw------- 1 heikki heikki 16777216 Jan 18 13:38 000000010000000000000003 > -rw------- 1 heikki heikki 41 Jan 18 13:38 00000002.history > -rw------- 1 heikki heikki 16777216 Jan 18 13:38 000000020000000000000003 > -rw------- 1 heikki heikki 16777216 Jan 18 13:38 000000020000000000000004 > -rw------- 1 heikki heikki 16777216 Jan 18 13:38 000000020000000000000005 > -rw------- 1 heikki heikki 83 Jan 18 13:38 00000003.history > -rw------- 1 heikki heikki 16777216 Jan 18 13:38 000000030000000000000005 > -rw------- 1 heikki heikki 16777216 Jan 18 13:38 000000030000000000000006 > drwx------ 2 heikki heikki 4096 Jan 18 13:38 archive_status > ~/pgsql.master$ ls -l data-standbyB/pg_xlog/ > total 81928 > -rw------- 1 heikki heikki 16777216 Jan 18 13:38 000000010000000000000001 > -rw------- 1 heikki heikki 16777216 Jan 18 13:38 000000010000000000000002 > -rw------- 1 heikki heikki 16777216 Jan 18 13:38 000000020000000000000003 > -rw------- 1 heikki heikki 16777216 Jan 18 13:38 000000020000000000000004 > -rw------- 1 heikki heikki 83 Jan 18 13:38 00000003.history > -rw------- 1 heikki heikki 16777216 Jan 18 13:38 000000030000000000000005 > drwx------ 2 heikki heikki 4096 Jan 18 13:38 archive_status > This can be thought of as another variant of the same issue that was > fixed by commit 60df192aea0e6458f20301546e11f7673c102101. When standby B > scans for the latest timeline, it finds it to be 3, and it reads the > timeline history file for 3. After that patch, it also saves it in > pg_xlog. It doesn't save the timeline history file for timeline 2, > because that's included in the history of timeline 3. However, when > standby C connects, it will try to fetch all the history files that it > doesn't have, including 00000002.history, which throws the error. Is the file 00000002.history really required by standby C for any useful purpose? Can we think of change in current design such that when standby C connects, even if some old history file (like 00000002.history) is not present, it ignores the same and continue. With Regards, Amit Kapila.
-
Re: More subtle issues with cascading replication over timeline switches
Heikki Linnakangas <hlinnakangas@vmware.com> — 2013-01-21T10:51:56Z
On 19.01.2013 14:26, Amit kapila wrote: > On Friday, January 18, 2013 5:27 PM Heikki Linnakangas wrote: > > >> Indeed, looking at the pg_xlog, it's not there (I did a couple of extra >> timeline switches: > >> ~/pgsql.master$ ls -l data-master/pg_xlog/ >> total 131084 >> -rw------- 1 heikki heikki 16777216 Jan 18 13:38 000000010000000000000001 >> -rw------- 1 heikki heikki 16777216 Jan 18 13:38 000000010000000000000002 >> -rw------- 1 heikki heikki 16777216 Jan 18 13:38 000000010000000000000003 >> -rw------- 1 heikki heikki 41 Jan 18 13:38 00000002.history >> -rw------- 1 heikki heikki 16777216 Jan 18 13:38 000000020000000000000003 >> -rw------- 1 heikki heikki 16777216 Jan 18 13:38 000000020000000000000004 >> -rw------- 1 heikki heikki 16777216 Jan 18 13:38 000000020000000000000005 >> -rw------- 1 heikki heikki 83 Jan 18 13:38 00000003.history >> -rw------- 1 heikki heikki 16777216 Jan 18 13:38 000000030000000000000005 >> -rw------- 1 heikki heikki 16777216 Jan 18 13:38 000000030000000000000006 >> drwx------ 2 heikki heikki 4096 Jan 18 13:38 archive_status >> ~/pgsql.master$ ls -l data-standbyB/pg_xlog/ >> total 81928 >> -rw------- 1 heikki heikki 16777216 Jan 18 13:38 000000010000000000000001 >> -rw------- 1 heikki heikki 16777216 Jan 18 13:38 000000010000000000000002 >> -rw------- 1 heikki heikki 16777216 Jan 18 13:38 000000020000000000000003 >> -rw------- 1 heikki heikki 16777216 Jan 18 13:38 000000020000000000000004 >> -rw------- 1 heikki heikki 83 Jan 18 13:38 00000003.history >> -rw------- 1 heikki heikki 16777216 Jan 18 13:38 000000030000000000000005 >> drwx------ 2 heikki heikki 4096 Jan 18 13:38 archive_status > >> This can be thought of as another variant of the same issue that was >> fixed by commit 60df192aea0e6458f20301546e11f7673c102101. When standby B >> scans for the latest timeline, it finds it to be 3, and it reads the >> timeline history file for 3. After that patch, it also saves it in >> pg_xlog. It doesn't save the timeline history file for timeline 2, >> because that's included in the history of timeline 3. However, when >> standby C connects, it will try to fetch all the history files that it >> doesn't have, including 00000002.history, which throws the error. > > Is the file 00000002.history really required by standby C for any useful purpose? No, not really. > Can we think of change in current design such that when standby C connects, even if some old history file (like 00000002.history) > is not present, it ignores the same and continue. That would be possible too, with some rejiggering of the code. At the moment, however, the code to find the latest timeline works by checking the existence of timeline history files in order. So it first checks for 00000002.history, then 00000003.history, then 00000004.history and so on, until it gets a file-not-found. That logic doesn't work if there are gaps in the sequence. So I'm inclined to just make sure the history files are always copied. I think it's good to have them around anyway, for debugging purposes. - Heikki
-
Re: More subtle issues with cascading replication over timeline switches
Heikki Linnakangas <hlinnakangas@vmware.com> — 2013-01-21T14:16:49Z
On 18.01.2013 13:57, Heikki Linnakangas wrote: > When a standby starts up, and catches up with the master through the > archive, it copies the target timeline's history file from the archive > to pg_xlog. That's enough for that standby's purposes, but if there is a > cascading standby or pg_receivexlog connected to it, it will request the > timeline history files of *all* timelines between the starting point and > current target. > > For example, create a master, and take a base backup from it. Use the > base backup to initialize two standby servers. Now perform failover > first to the first standby, and once the second standby has caught up, > fail over again, to the second standby. (Or as a shortcut, forget about > the standbys, and just create a recovery.conf file in the master with > restore_command='/bin/false' and restart it. That causes a timeline > switch. Repeat twice) > > Now use the base backup to initialize a new standby server (you can kill > and delete the old ones), using the WAL archive. Set up a second, > cascading, standby server that connects to the first standby using > streaming replication, not WAL archive. This cascading standby will fail > to cross the timeline switch, because it doesn't find all the history > files in the standby: > > C 2013-01-18 13:38:46.047 EET 7695 FATAL: could not receive timeline > history file from the primary server: ERROR: could not open file > "pg_xlog/00000002.history": No such file or directory > > Indeed, looking at the pg_xlog, it's not there (I did a couple of extra > timeline switches: > > ~/pgsql.master$ ls -l data-master/pg_xlog/ > total 131084 > -rw------- 1 heikki heikki 16777216 Jan 18 13:38 000000010000000000000001 > -rw------- 1 heikki heikki 16777216 Jan 18 13:38 000000010000000000000002 > -rw------- 1 heikki heikki 16777216 Jan 18 13:38 000000010000000000000003 > -rw------- 1 heikki heikki 41 Jan 18 13:38 00000002.history > -rw------- 1 heikki heikki 16777216 Jan 18 13:38 000000020000000000000003 > -rw------- 1 heikki heikki 16777216 Jan 18 13:38 000000020000000000000004 > -rw------- 1 heikki heikki 16777216 Jan 18 13:38 000000020000000000000005 > -rw------- 1 heikki heikki 83 Jan 18 13:38 00000003.history > -rw------- 1 heikki heikki 16777216 Jan 18 13:38 000000030000000000000005 > -rw------- 1 heikki heikki 16777216 Jan 18 13:38 000000030000000000000006 > drwx------ 2 heikki heikki 4096 Jan 18 13:38 archive_status > ~/pgsql.master$ ls -l data-standbyB/pg_xlog/ > total 81928 > -rw------- 1 heikki heikki 16777216 Jan 18 13:38 000000010000000000000001 > -rw------- 1 heikki heikki 16777216 Jan 18 13:38 000000010000000000000002 > -rw------- 1 heikki heikki 16777216 Jan 18 13:38 000000020000000000000003 > -rw------- 1 heikki heikki 16777216 Jan 18 13:38 000000020000000000000004 > -rw------- 1 heikki heikki 83 Jan 18 13:38 00000003.history > -rw------- 1 heikki heikki 16777216 Jan 18 13:38 000000030000000000000005 > drwx------ 2 heikki heikki 4096 Jan 18 13:38 archive_status > > This can be thought of as another variant of the same issue that was > fixed by commit 60df192aea0e6458f20301546e11f7673c102101. When standby B > scans for the latest timeline, it finds it to be 3, and it reads the > timeline history file for 3. After that patch, it also saves it in > pg_xlog. It doesn't save the timeline history file for timeline 2, > because that's included in the history of timeline 3. However, when > standby C connects, it will try to fetch all the history files that it > doesn't have, including 00000002.history, which throws the error. > > A related problem is that at the segment containing the timeline switch, > standby has only restored from archive the WAL file of the new timeline, > not the old one. For example above, the timeline switch 1 -> 2 happened > while inserting to segment 000000010000000000000003, and a copy of that > partial segment was created with the timeline's ID as > 000000020000000000000003. The standby only has the segment from the new > timeline in pg_xlog, which is enough for that standby's purposes, but > will cause an error when the cascading standby tries to stream it: > > C 2013-01-18 13:46:12.334 EET 8579 FATAL: error reading result of > streaming command: ERROR: requested WAL segment 000000010000000000000003 > has already been removed > > A straightforward fix would be for the standby to restore those files > that the cascading standby needs from the WAL archive, even if they're > not strictly required for that standby itself. But actually, isn't it a > bad idea that we store the partial segment, 000000010000000000000003 in > this case, in the WAL archive? There's no way to tell that it's partial, > and it can clash with a complete segment if more WAL is generated on > that timeline. I just argued that pg_receivexlog should not do that, and > hence keep the .partial suffix in the same situation, in > http://www.postgresql.org/message-id/50F56245.8050802@vmware.com. I came up with the attached. I decided to use different approaches for the timeline history files and the WAL segments. For timeline history files, try to copy any missing timeline history files from archive into pg_xlog. For example, if recovery begins at timeline 5, and target timeline is 7, recovery will try to restore 00000006.history even though it doesn't itself need it. A cascading standby or pg_receivexlog will then find it there. For WAL files, I changed walsender so that when it reads WAL belonging to a segment that also contains a timeline switch to the next timeline, ie. the last segment on the old timeline, walsender opens the file with the new timeline's ID instead. This comment explains it better: /*------- * When reading from a historic timeline, and there is a timeline * switch within the this segment, read from the WAL segment * belonging to the new timeline. * * For example, imagine that this server is currently on timeline * 5, and we're streaming timeline 4. The switch from timeline 4 * to 5 happened at 0/13002088. In pg_xlog, we have these files: * * ... * 000000040000000000000012 * 000000040000000000000013 * 000000050000000000000013 * 000000050000000000000014 * ... * * In this situation, when requested to send the WAL from * segment 0x13, on timeline 4, we read the WAL from file * 000000050000000000000013. Archive recovery prefers files from * newer timelines, so if the segment was restored from the * archive on this server, the file belonging to the old timeline, * 000000040000000000000013, might not exist. Their contents are * equal up to the switchpoint, because at a timeline switch, the * used portion of the old segment is copied to the new file. *------- */ That seems a bit weird at first, but it matches how archive recovery behaves when it restores the file with the newer timeline's ID, and reads the WAL belonging to the old timeline from that. - Heikki