Horton
Menu

Re: cleanup patches for incremental backup

Nathan Bossart <nathandbossart@gmail.com>

View thread
From: Nathan Bossart <nathandbossart@gmail.com>
To: Robert Haas <robertmhaas@gmail.com>
Cc: Matthias van de Meent <boekewurm+postgres@gmail.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2024-01-24T18:05:15Z
Lists: pgsql-hackers 
On Wed, Jan 24, 2024 at 12:46:16PM -0500, Robert Haas wrote:
> The "examining summary" line is generated based on the output of
> pg_available_wal_summaries(). The way that works is that the server
> calls readdir(), disassembles the filename into a TLI and two LSNs,
> and returns the result. Then, a fraction of a second later, the test
> script reassembles those components into a filename and finds the file
> missing. If the logic to translate between filenames and TLIs & LSNs
> were incorrect, the test would fail consistently. So the only
> explanation that seems to fit the facts is the file disappearing out
> from under us. But that really shouldn't happen. We do have code to
> remove such files in MaybeRemoveOldWalSummaries(), but it's only
> supposed to be nuking files more than 10 days old.
> 
> So I don't really have a theory here as to what could be happening. :-(

There might be an overflow risk in the cutoff time calculation, but I doubt
that's the root cause of these failures:

	/*
	 * Files should only be removed if the last modification time precedes the
	 * cutoff time we compute here.
	 */
	cutoff_time = time(NULL) - 60 * wal_summary_keep_time;

Otherwise, I think we'll probably need to add some additional logging to
figure out what is happening...

-- 
Nathan Bossart
Amazon Web Services: https://aws.amazon.com

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Avoid overflow in MaybeRemoveOldWalSummaries().

  2. Revert "Temporary patch to help debug pg_walsummary test failures."

  3. Revise pg_walsummary's 002_blocks test to avoid spurious failures.

  4. Temporary patch to help debug pg_walsummary test failures.

  5. More documentation updates for incremental backup.

  6. Fix typos.

  7. Fix typo.

  8. Try to fix pg_walsummary buildfarm failures.

  9. Repair various defects in dc212340058b4e7ecfc5a7a81ec50e7a207bf288.

  10. Add new pg_walsummary tool.

  11. Add new function pg_get_wal_summarizer_state().

  12. Document WAL summarization information functions.