Re: Teach pg_receivewal to use lz4 compression

Jian Guo <gjian@vmware.com>

From: Jian Guo <gjian@vmware.com>
To: pgsql-hackers@lists.postgresql.org
Cc: Georgios Kokolatos <gkokolatos@protonmail.com>
Date: 2021-09-11T05:02:42Z
Lists: pgsql-hackers
@@ -250,14 +302,18 @@ FindStreamingStart(uint32 *tli)
 		/*
 		 * Check that the segment has the right size, if it's supposed to be
 		 * completed.  For non-compressed segments just check the on-disk size
-		 * and see if it matches a completed segment. For compressed segments,
-		 * look at the last 4 bytes of the compressed file, which is where the
-		 * uncompressed size is located for gz files with a size lower than
-		 * 4GB, and then compare it to the size of a completed segment. The 4
-		 * last bytes correspond to the ISIZE member according to
+		 * and see if it matches a completed segment. For zlib compressed
+		 * segments, look at the last 4 bytes of the compressed file, which is
+		 * where the uncompressed size is located for gz files with a size lower
+		 * than 4GB, and then compare it to the size of a completed segment.
+		 * The 4 last bytes correspond to the ISIZE member according to
 		 * http://www.zlib.org/rfc-gzip.html.
+		 *
+		 * For lz4 compressed segments read the header using the exposed API and
+		 * compare the uncompressed file size, stored in
+		 * LZ4F_frameInfo_t{.contentSize}, to that of a completed segment.
 		 */
-		if (!ispartial && !iscompress)
+		if (!ispartial && wal_compression_method == COMPRESSION_NONE)
 		{
 			struct stat statbuf;
 			char		fullpath[MAXPGPATH * 2];
@@ -276,7 +332,7 @@ FindStreamingStart(uint32 *tli)
 				continue;
 			}
 		}
-		else if (!ispartial && iscompress)
+		else if (!ispartial && wal_compression_method == COMPRESSION_ZLIB)
 		{
 			int			fd;
 			char		buf[4];
@@ -322,6 +378,70 @@ FindStreamingStart(uint32 *tli)
 				continue;
 			}
 		}
+		else if (!ispartial && compression_method == COMPRESSION_LZ4)
+		{
+#ifdef HAVE_LIBLZ4
+			int			fd;
+			int			r;
+			size_t		consumed_len = LZ4F_HEADER_SIZE_MAX;
+			char	    buf[LZ4F_HEADER_SIZE_MAX];
+			char		fullpath[MAXPGPATH * 2];
+			LZ4F_frameInfo_t frame_info = { 0 };
+			LZ4F_decompressionContext_t ctx = NULL;
+
+			snprintf(fullpath, sizeof(fullpath), "%s/%s", basedir, dirent->d_name);
+
+			fd = open(fullpath, O_RDONLY | PG_BINARY, 0);

Should close the fd before exit or abort.

Commits

  1. Fix header inclusion order in pg_receivewal.c

  2. Remove useless LZ4 system call on failure when writing file header

  3. Add support for LZ4 compression in pg_receivewal

  4. Rework compression options of pg_receivewal

  5. Clarify some errors in pg_receivewal when closing WAL segments