Horton

Re: .ready and .done files considered harmful

Robert Haas <robertmhaas@gmail.com>

View thread View on postgr.es
From: Robert Haas <robertmhaas@gmail.com>
To: Alvaro Herrera <alvherre@alvh.no-ip.org>
Cc: "Bossart, Nathan" <bossartn@amazon.com>, Dipesh Pandit <dipesh.pandit@gmail.com>, Kyotaro Horiguchi <horikyota.ntt@gmail.com>, Jeevan Ladhe <jeevan.ladhe@enterprisedb.com>, Stephen Frost <sfrost@snowman.net>, Andres Freund <andres@anarazel.de>, Hannu Krosing <hannuk@google.com>, "pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Date: 2021-09-21T17:24:44Z
Lists: pgsql-hackers 
On Mon, Sep 20, 2021 at 4:42 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
> I was going to say that perhaps we can avoid repeated scans by having a
> bitmap of future files that were found by a scan; so if we need to do
> one scan, we keep track of the presence of the next (say) 64 files in
> our timeline, and then we only have to do another scan when we need to
> archive a file that wasn't present the last time we scanned.

There are two different proposed patches on this thread. One of them
works exactly that way, and the other one tries to optimize by
assuming that if we just optimized WAL file N, we likely will next
want to archive WAL file N+1. It's been hard to decide which way is
better.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

Commits

  1. Reduce overhead of renaming archive status files.

  2. Improve performance of pgarch_readyXlog() with many status files.

  3. Prioritize history files when archiving