Horton
Menu

Re: Adding CI to our tree

Andres Freund <andres@anarazel.de>

View thread
From: Andres Freund <andres@anarazel.de>
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Justin Pryzby <pryzby@telsasoft.com>, Robert Haas <robertmhaas@gmail.com>, Andrew Dunstan <andrew@dunslane.net>, pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>, Melanie Plageman <melanieplageman@gmail.com>, Peter Eisentraut <peter.eisentraut@enterprisedb.com>, Daniel Gustafsson <daniel@yesql.se>
Date: 2022-01-19T04:16:46Z
Lists: pgsql-hackers 
Hi,

On 2022-01-18 21:50:07 -0500, Tom Lane wrote:
> I just found one thing making check-world slower than it ought to be:
> src/test/recovery/t/008_fsm_truncation.pl does
> 
> $node_primary->append_conf(
> 	'postgresql.conf', qq{
> fsync = on
> wal_log_hints = on
> max_prepared_transactions = 5
> autovacuum = off
> });
> 
> There is no reason for this script to be overriding Cluster.pm's
> fsync = off setting.
> 
> This appears to go back to 917dc7d23 of 2016, so I think it just
> predates our recognition that we should disable fsync in routine
> tests.

Yea, I noticed this too. I was wondering if there's a conceivable reason to
actually want fsyncs, but I couldn't come up with one.

On systems where IO isn't overloaded, the main problem with this test are
elsewhere: It multiple times waits for VACUUMs that are blocked truncating the
table. Which these days takes 5 seconds. Thus the test takes quite a while.

To me VACUUM_TRUNCATE_LOCK_TIMEOUT = 5s seems awfully long. On a system with a
lot of tables that's much more than vacuum will take. So this can easily lead
to using up all autovacuum workers...



> This actually causes parallel check-world to fail altogether on florican's
> host, because the initial fsync of the recovered primary takes more than 3
> minutes when there's conflicting I/O traffic, causing pg_ctl to time out.

Ugh.

I noticed a few other sources of "unnecessary" fsyncs.  The most frequent
being the durable_rename() of backup_manifest in pg_basebackup.c. Manifests are
surprisingly large, 135k for a freshly initdb'd cluster.


There's an fsync in walmethods.c:tar_close() that sounds intentional, but I
don't really understand what the comment:

	/* Always fsync on close, so the padding gets fsynced */
	if (tar_sync(f) < 0)

Greetings,

Andres Freund

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. ci: enable zstd where available.

  2. ci: compile with -Og where applicable.

  3. ci: include hints how to install OS packages.

  4. ci: fix copy-paste mistake in 16eb8231d1b.

  5. ci: macos: align sysinfo_script to other tasks.

  6. ci: Only use one artifact instruction for logs.

  7. ci: s/CCACHE_SIZE/CCACHE_MAXSIZE/.

  8. pg_basebackup: Skip a few more fsyncs if --no-sync is specified.

  9. TAP tests: check for postmaster.pid anyway when "pg_ctl start" fails.

  10. Don't enable fsync in src/test/recovery/t/008_fsm_truncation.pl.

  11. ci: windows: run initdb with --no-sync.

  12. ci: windows: enable build summary to make it easier to spot warnings / errors.

  13. ci: Add continuous integration for github repositories via cirrus-ci.

  14. Fix TestLib::slurp_file() with offset on windows.