Re: Postgres, fsync, and OSs (specifically linux)

On 29 April 2018 at 00:15, Andres Freund <andres@anarazel.de> wrote:
> Hi,
>
> On 2018-04-28 08:25:53 -0700, Simon Riggs wrote:
>> > - Use direct IO. Due to architectural performance issues in PG and the
>> >   fact that it'd not be applicable for all installations I don't think
>> >   this is a reasonable fix for the issue presented here. Although it's
>> >   independently something we should work on.  It might be worthwhile to
>> >   provide a configuration that allows to force DIO to be enabled for WAL
>> >   even if replication is turned on.
>>
>> "Use DirectIO" is roughly same suggestion as "don't trust Linux filesystems".
>
> I want to emphasize that this is NOT a linux only issue. It's a problem
> across a number of operating systems, including linux.
>
>
>> It would be a major admission of defeat for us to take that as our
>> main route to a solution.
>
> Well, I think we were wrong to not engineer towards DIO. There's just
> too many issues with buffered IO to not have a supported path for
> DIO. But given that it's unrealistic to do so without major work, and
> wouldn't be applicable for all installations (shared_buffer size becomes
> critical), I don't think it matters that much for the issue discussed
> here.


20/20 hindsight, really. Not much to be done now.

Even with the work you and others have done on shared_buffers
scalability, there's likely still improvement needed there if it
becomes more important to evict buffers into per-device queues, etc,
too.

Personally I'd rather not have to write half the kernel's job because
the kernel doesn't feel like doing it :( . I'd kind of hoped to go in
the other direction if anything, with some kind of pseudo-write op
that let us swap a dirty shared_buffers entry from our shared_buffers
into the OS dirty buffer cache (on Linux at least) and let it handle
writeback, so we reduce double-buffering. Ha! So much for that!

-- 
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services