v2-0001-Document-the-backup-manifest-file-format.patch

From 96ed7014cc4c3e2e16ffacfe88869e640c66ff99 Mon Sep 17 00:00:00 2001
From: Robert Haas <rhaas@postgresql.org>
Date: Mon, 13 Apr 2020 14:08:01 -0400
Subject: [PATCH v2] Document the backup manifest file format.

Patch by me, at the request of Andres Freund. Reviewed by
Justin Pryzby.

Discussion: http://postgr.es/m/20200327203225.hcm6ag4grwsiruea@alap3.anarazel.de
---
 doc/src/sgml/backup-manifest.sgml | 208 ++++++++++++++++++++++++++++++
 doc/src/sgml/filelist.sgml        |   1 +
 doc/src/sgml/postgres.sgml        |   1 +
 3 files changed, 210 insertions(+)
 create mode 100644 doc/src/sgml/backup-manifest.sgml

diff --git a/doc/src/sgml/backup-manifest.sgml b/doc/src/sgml/backup-manifest.sgml
new file mode 100644
index 0000000000..574e015b44
--- /dev/null
+++ b/doc/src/sgml/backup-manifest.sgml
@@ -0,0 +1,208 @@
+<!-- doc/src/sgml/backupmanifest.sgml -->
+
+<chapter id="backup-manifest-format">
+ <title>Backup Manifest Format</title>
+
+  <para>
+   The backup manifest generated by <xref linkend="app-pgbasebackup" /> is
+   primarily intended to permit the backup to be verified using
+   <xref linkend="app-pgverifybackup" />. However, it is
+   also possible for other tools to read the backup manifest file and use
+   the information contained therein for their own purposes. To that end,
+   this chapter describes the format of the backup manifest file.
+  </para>
+
+  <para>
+   A backup manifest is a JSON document encoded as UTF-8. (Although in
+   general JSON documents are required to be Unicode, PostgreSQL permits
+   the <type>json</type> and <type>jsonb</type> data types to be used with any
+   supported server encoding. There is no similar exception for backup
+   manifests.) The JSON document is always an object; the keys that are present
+   in this object are described in the next section.
+  </para>
+
+ <sect1 id="backup-manifest-toplevel">
+  <title>Backup Manifest Toplevel Object</title>
+
+  <para>
+   The backup manifest JSON document contains the following keys.
+  </para>
+
+  <variablelist>
+   <varlistentry>
+    <term><literal>PostgreSQL-Backup-Manifest-Version</literal></term>
+    <listitem>
+     <para>
+      The associated value is always the integer 1.
+     </para>
+    </listitem>
+   </varlistentry>
+
+   <varlistentry>
+    <term><literal>Files</literal></term>
+    <listitem>
+     <para>
+      The associated value is always a list of objects, each describing one
+      file that is present in the backup. No entries are present in this
+      list for the WAL files that are needed in order to use the backup,
+      or for the backup manifest itself.  The structure of each object in the
+      list is described in <xref linkend="backup-manifest-files" />.
+     </para>
+    </listitem>
+   </varlistentry>
+
+   <varlistentry>
+    <term><literal>WAL-Ranges</literal></term>
+    <listitem>
+     <para>
+      The associated value is always a list of objects, each describing a
+      range of WAL records that must be readable from a particular timeline
+      in order to make use of the backup.  The structure of these objects is
+      further described in <xref linkend="backup-manifest-wal-ranges" />.
+     </para>
+    </listitem>
+   </varlistentry>
+
+   <varlistentry>
+    <term><literal>Manifest-Checksum</literal></term>
+    <listitem>
+     <para>
+      This key is always present on the last line of the backup manifest file.
+      The associated value is a SHA256 checksum of all the preceding lines.
+     </para>
+    </listitem>
+   </varlistentry>
+  </variablelist>
+ </sect1>
+
+ <sect1 id="backup-manifest-files">
+  <title>Backup Manifest File Object</title>
+
+  <para>
+   The object which describes a single file contains either a
+   <literal>Path</literal> key or an <literal>Encoded-Path</literal> key.
+   Normally, the <literal>Path</literal> key will be present. The
+   associated string value is the path of the file relative to the root
+   of the backup directory. Files located in a user-defined tablespace
+   will have paths whose first two components are pg_tblspc and the OID
+   of the tablespace. If the path is not a string that is legal in UTF-8,
+   or if the user requests that encoded paths be used for all files, then
+   the <literal>Encoded-Path</literal> key will be present instead.  This
+   stores the same data, but it is encoded as a string of hexadecimal
+   digits. Each pair of hexadecimal digits in the string represents a
+   single octet.
+  </para>
+
+  <para>
+   The following two keys are always present:
+  </para>
+
+  <variablelist>
+   <varlistentry>
+    <term><literal>Size</literal></term>
+    <listitem>
+     <para>
+      The expected size of this file, as an integer.
+     </para>
+    </listitem>
+   </varlistentry>
+
+   <varlistentry>
+    <term><literal>Last-Modified</literal></term>
+    <listitem>
+     <para>
+      The last modification time of the file as reported by the server at
+      the time of the backup. Unlike the other fields stored in the backup,
+      this field is not used by <xref linkend="app-pgverifybackup" />.
+      It is included only for informational purposes.
+     </para>
+    </listitem>
+   </varlistentry>
+  </variablelist>
+
+  <para>
+   If the backup was taken with file checksums enabled, the following
+   keys will be present:
+  </para>
+
+  <variablelist>
+   <varlistentry>
+    <term><literal>Checksum-Algorithm</literal></term>
+    <listitem>
+     <para>
+      The checksum algorithm used to compute a checksum for this file.
+      Currently, this will be the same for every file in the backup
+      manifest, but this may change in future releases. At present, the
+      supported checksum algorithms are <literal>CRC32C</literal>,
+      <literal>SHA224</literal>,
+      <literal>SHA256</literal>,
+      <literal>SHA384</literal>, and
+      <literal>SHA512</literal>.
+     </para>
+    </listitem>
+   </varlistentry>
+
+   <varlistentry>
+    <term><literal>Checksum</literal></term>
+    <listitem>
+     <para>
+      The checksum computed for this file, stored as a series of
+      hexadecimal characters, two for each byte of the checksum.
+     </para>
+    </listitem>
+   </varlistentry>
+  </variablelist>
+ </sect1>
+
+ <sect1 id="backup-manifest-wal-ranges">
+  <title>Backup Manifest WAL Range Object</title>
+
+  <para>
+   The object which describes a WAL range always has three keys:
+  </para>
+
+  <variablelist>
+   <varlistentry>
+    <term><literal>Timeline</literal></term>
+    <listitem>
+     <para>
+      The timeline from which this range of WAL records will be required in
+      order to make use of this backup. The value is an integer.
+     </para>
+    </listitem>
+   </varlistentry>
+
+   <varlistentry>
+    <term><literal>Start-LSN</literal></term>
+    <listitem>
+     <para>
+      The LSN at which replay must begin on the indicated timeline in order to
+      make use of this backup.  The LSN is stored in the format normally used
+      by <productname>PostgreSQL</productname>; that is, it is a string
+      consisting of two strings of hexademical characters, each with a length
+      of between 1 and 8, separated by a slash.
+     </para>
+    </listitem>
+   </varlistentry>
+
+   <varlistentry>
+    <term><literal>End-LSN</literal></term>
+    <listitem>
+     <para>
+      The earliest LSN at which replay may end when making using of this
+      backup. This is stored in the same format as
+      <literal>Start-LSN</literal>.
+     </para>
+    </listitem>
+   </varlistentry>
+  </variablelist>
+
+  <para>
+   Ordinarily, there will be only a single WAL range. However, if a backup is
+   taken from a standby which switches timelines during the backup due to an
+   upstream promotion, it is possible for multiple ranges to be present, each
+   with a different timeline. There will never be multiple WAL ranges present
+   for the same timeline.
+  </para>
+ </sect1>
+</chapter>
diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml
index cf21ef857e..68179f71cd 100644
--- a/doc/src/sgml/filelist.sgml
+++ b/doc/src/sgml/filelist.sgml
@@ -102,6 +102,7 @@
 <!ENTITY storage    SYSTEM "storage.sgml">
 <!ENTITY tablesample-method SYSTEM "tablesample-method.sgml">
 <!ENTITY generic-wal SYSTEM "generic-wal.sgml">
+<!ENTITY backup-manifest SYSTEM "backup-manifest.sgml">
 
 <!-- contrib information -->
 <!ENTITY contrib         SYSTEM "contrib.sgml">
diff --git a/doc/src/sgml/postgres.sgml b/doc/src/sgml/postgres.sgml
index f4a3c7e5e7..c41ce9499b 100644
--- a/doc/src/sgml/postgres.sgml
+++ b/doc/src/sgml/postgres.sgml
@@ -268,6 +268,7 @@ break is not needed in a wider output rendering.
   &storage;
   &bki;
   &planstats;
+  &backup-manifest;
 
  </part>
 
-- 
2.17.2 (Apple Git-113)