Re: .ready and .done files considered harmful

Nathan Bossart <bossartn@amazon.com>

From: "Bossart, Nathan" <bossartn@amazon.com>
To: David Steele <david@pgmasters.net>, Robert Haas <robertmhaas@gmail.com>
Cc: Dipesh Pandit <dipesh.pandit@gmail.com>, Kyotaro Horiguchi <horikyota.ntt@gmail.com>, Jeevan Ladhe <jeevan.ladhe@enterprisedb.com>, Stephen Frost <sfrost@snowman.net>, Andres Freund <andres@anarazel.de>, Hannu Krosing <hannuk@google.com>, "pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Date: 2021-09-27T18:03:33Z
Lists: pgsql-hackers

Attachments

On 9/24/21, 9:29 AM, "Robert Haas" <robertmhaas@gmail.com> wrote:
> So what I am inclined to do is commit
> v1-0001-Improve-performance-of-pgarch_readyXlog-with-many.patch.
> However, v6-0001-Do-fewer-directory-scans-of-archive_status.patch has
> perhaps evolved a bit more than the other one, so I thought I should
> first ask whether any of those changes have influenced your thinking
> about the batching approach and whether you want to make any updates
> to that patch first. I don't really see that this is needed, but I
> might be missing something.

Besides sprucing up the comments a bit, I don't think there is
anything that needs to be changed.  The only other thing I considered
was getting rid of the arch_files array.  Instead, I would swap the
comparator function the heap uses with a reverse one, rebuild the
heap, and then have pgarch_readyXlog() return files via
binaryheap_remove_first().  However, this seemed like a bit more
complexity than necessary.

Attached is a new version of the patch with some expanded comments.

Nathan

Commits

  1. Reduce overhead of renaming archive status files.

  2. Improve performance of pgarch_readyXlog() with many status files.

  3. Prioritize history files when archiving