Horton

Re: Refactoring the checkpointer's fsync request queue

Shawn Debnath <sdn@amazon.com>

View thread View on postgr.es
From: Shawn Debnath <sdn@amazon.com>
To: Robert Haas <robertmhaas@gmail.com>
Cc: Andres Freund <andres@anarazel.de>, Thomas Munro <thomas.munro@gmail.com>, Thomas Munro <thomas.munro@enterprisedb.com>, Dmitry Dolgov <9erthalion6@gmail.com>, PostgreSQL-development <pgsql-hackers@postgresql.org>
Date: 2019-03-04T16:55:06Z
Lists: pgsql-hackers 
On Fri, Mar 01, 2019 at 04:27:47PM -0500, Robert Haas wrote:
> On Fri, Mar 1, 2019 at 3:35 PM Shawn Debnath <sdn@amazon.com> wrote:
> > On Fri, Mar 01, 2019 at 03:03:19PM -0500, Robert Haas wrote:
> > > On Fri, Mar 1, 2019 at 2:36 PM Shawn Debnath <sdn@amazon.com> wrote:
> > > > I disagree, at least with combining and retaining enums. Encoding all
> > > > the possible request types with the current, planned and future SMGRs
> > > > would cause a sheer explosion in the number of enum  values.
> > >
> > > How big of an explosion would it be?
> >
> > 4 enum values x # of smgrs; currently md, soon undo and slru so 12 in
> > total. Any future smgr addition will expand this further.
> 
> I thought the idea was that each smgr might have a different set of
> requests.  If they're all going to have the same set of requests then
> I agree with you.

Yeah, in this particular case and at this layer, the operations are 
consistent across all storage managers, in that, they want to queue a 
new sync request for a specific file, forget an already queued request, 
forget a hierarchy of requests, or unlink a specific file.

The fun is at the smgr layer which was discussed in a sub-thread in the 
"Drop type smgr" thread started by Thomas. I started on a patch and will 
be sending it out after the refactor patch is revised.


-- 
Shawn Debnath
Amazon Web Services (AWS)

Commits

  1. Fix bugs in mdsyncfiletag().

  2. Refactor the fsync queue for wider use.

  3. Don't forget about failed fsync() requests.

  4. PANIC on fsync() failure.

  5. Move LockClauseStrength, LockWaitPolicy into new file nodes/lockoptions.h.

  6. Add new file for checkpointer.c

  7. Split work of bgwriter between 2 processes: bgwriter and checkpointer.

  8. Install infrastructure for shared-memory free space map. Doesn't actually