Horton

Re: .ready and .done files considered harmful

Nathan Bossart <bossartn@amazon.com>

View thread View on postgr.es
From: "Bossart, Nathan" <bossartn@amazon.com>
To: Dipesh Pandit <dipesh.pandit@gmail.com>, Kyotaro Horiguchi <horikyota.ntt@gmail.com>
Cc: Robert Haas <robertmhaas@gmail.com>, Jeevan Ladhe <jeevan.ladhe@enterprisedb.com>, Stephen Frost <sfrost@snowman.net>, "Andres Freund" <andres@anarazel.de>, Hannu Krosing <hannuk@google.com>, "pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Date: 2021-09-09T22:24:30Z
Lists: pgsql-hackers 
On 9/8/21, 10:49 AM, "Dipesh Pandit" <dipesh.pandit@gmail.com> wrote:
> Updated log level to DEBUG3 and rebased the patch. PFA patch.

Thanks for the new patch.

+ * by checking the availability of next WAL file. "xlogState" specifies the
+ * segment number and timeline ID corresponding to the next WAL file.

"xlogState" probably needs to be updated here.

As noted before [0], I think we need to force a directory scan at the
beginning of pgarch_MainLoop() and when pgarch_ArchiverCopyLoop()
returns before we exit the "while" loop.  Else, there's probably a
risk that we skip archiving a file until the next directory scan.  IMO
forcing a directory scan at the beginning of pgarch_ArchiverCopyLoop()
is a simpler way to do roughly the same thing.  I'm skeptical that
persisting the next-anticipated state between calls to
pgarch_ArchiverCopyLoop() is worth the complexity.

Nathan

[0] https://www.postgresql.org/message-id/AC78607B-9DA6-41F4-B253-840D3DD964BF%40amazon.com

Commits

  1. Reduce overhead of renaming archive status files.

  2. Improve performance of pgarch_readyXlog() with many status files.

  3. Prioritize history files when archiving