Re: .ready and .done files considered harmful

+	 * This .ready file is created out of order, notify archiver to perform
+	 * a full directory scan to archive corresponding WAL file.
+	 */
+	StatusFilePath(archiveStatusPath, xlog, ".ready");
+	if (stat(archiveStatusPath, &stat_buf) == 0)
+		PgArchEnableDirScan();

We may want to call PgArchWakeup() after setting the flag.

+	 * Perform a full directory scan to identify the next log segment. There
+	 * may be one of the following scenarios which may require us to perform a
+	 * full directory scan.
...
+	 * - The next anticipated log segment is not available.

I wonder if we really need to perform a directory scan in this case.
Unless there are other cases where the .ready files are created out of
order, I think this just causes an unnecessary directory scan every
time the archiver catches up.

+	 * Flag to enable/disable directory scan. If this flag is set then it
+	 * forces archiver to perform a full directory scan to get the next log
+	 * segment.
+	 */
+	pg_atomic_flag dirScan;

I personally don't think it's necessary to use an atomic here.  A
spinlock or LWLock would probably work just fine, as contention seems
unlikely.  If we use a lock, we also don't have to worry about memory
barriers.

Nathan