Horton

Re: Refactoring the checkpointer's fsync request queue

Andres Freund <andres@anarazel.de>

View thread View on postgr.es
From: Andres Freund <andres@anarazel.de>
To: Kevin Grittner <kgrittn@gmail.com>
Cc: Thomas Munro <thomas.munro@enterprisedb.com>, Dmitry Dolgov <9erthalion6@gmail.com>, Robert Haas <robertmhaas@gmail.com>, PostgreSQL-development <pgsql-hackers@postgresql.org>, Shawn Debnath <sdn@amazon.com>
Date: 2019-01-22T21:01:52Z
Lists: pgsql-hackers 
Hi,

On 2019-01-22 14:53:11 -0600, Kevin Grittner wrote:
> On Tue, Jan 22, 2019 at 2:38 PM Andres Freund <andres@anarazel.de> wrote:
> 
> > close() doesn't trigger an fsync() in general
> 
> What sort of a performance hit was observed when testing the addition
> of fsync or fdatasync before any PostgreSQL close() of a writable
> file, or have we not yet checked that?

I briefly played with it, and it was so atrocious (as in, less than
something like 0.2x the throughput) that I didn't continue far down that
path.  Two ways I IIRC (and it's really just memory) I tried were:

a) Short lived connections that do a bunch of writes to files each. That
   turns each disconnect into an fsync of most files.
b) Workload with > max_files_per_process files (IIRC I just used a bunch
   of larger tables with a few indexes each) in a read/write workload
   that's a bit larger than shared buffers. That lead to most file
   closes being integrity writes, with obvious downsides.

Greetings,

Andres Freund

Commits

  1. Fix bugs in mdsyncfiletag().

  2. Refactor the fsync queue for wider use.

  3. Don't forget about failed fsync() requests.

  4. PANIC on fsync() failure.

  5. Move LockClauseStrength, LockWaitPolicy into new file nodes/lockoptions.h.

  6. Add new file for checkpointer.c

  7. Split work of bgwriter between 2 processes: bgwriter and checkpointer.

  8. Install infrastructure for shared-memory free space map. Doesn't actually