[WIP] Pipelined Recovery

Hi,

Based on a suggestion by my colleague Ants Aasma, I worked on this
idea of adding parallelism to the WAL recovery process.

The crux of this idea is to decode the WAL using parallel workers. Now
the replay process can get the records from the shared memory queue
directly. This way, we can decrease some CPU load on the recovery process.

Implementing this idea yielded an improvement of around 20% in the
recovery times, but results may differ based on workloads. I have
attached some benchmarks for different workloads.

Following are some recovery tests with the default configs. Here p1
shows pipeline enabled. (db size) is the backup database size on
which the recovery happens. You can see more detail related to the
benchmarks in the attached file `recoveries-benchmark-v01`.

                           elapsed (p0)       elapsed (p1)   % perf    db
size

inserts.sql            272s 10ms       197s 570ms    27.37%     480 MB
updates.sql         177s 420ms      117s 80ms      34.01%     480 MB
hot-updates.sql   36s 940ms       29s 240ms       20.84%     480 MB
nonhot.sql           36s 570ms       28s 980ms       20.75%     480 MB
simple-update     20s 160ms       11s 580ms       42.56%     4913 MB
tpcb-like              20s 590ms       13s 640ms       33.75%     4913 MB

Similar approach was also suggested by Matthias van de Meent earlier in a
separate thread [1]. Right now I am using one bgw for decoding and filling
up the shared message queue, and the redo apply loop simply receives the
decoded record
from the queue. After the redo is finished, the consumer (startup
process) can request a shutdown from the producer (pipeline bgw)
before exiting recovery.

This idea can be coupled with another idea of pinning the buffers in
parallel before the recovery process needs them. This will try to
parallelize most of the work being done in
`XLogReadBufferForRedoExtended`. The Redo can simply receive
the already pinned buffers from a queue, but for implementing
this, we still need some R&D on that, as IPC and pinning/unpinning of
buffers across two processes can be tricky.

If someone wants to reproduce the benchmark, they can do so using
these scripts [2].

Looking forward to your reviews, comments, etc.

[1]:
https://www.postgresql.org/message-id/CAEze2Wh6C_QfxLii%2B%2BeZue5%3DKvbVXKkHyZW8PLmtLgyjmFzwCQ%40mail.gmail.com
[2]: https://github.com/imranzaheer612/pg-recovery-testing

--
Regards,
Imran Zaheer
CYBERTEC PostgreSQL International GmbH