Re: Postgres, fsync, and OSs (specifically linux)

On 28 April 2018 at 08:25, Simon Riggs <simon@2ndquadrant.com> wrote:
> On 27 April 2018 at 15:28, Andres Freund <andres@anarazel.de> wrote:
>
>> - Add a pre-checkpoint hook that checks for filesystem errors *after*
>>   fsyncing all the files, but *before* logging the checkpoint completion
>>   record. Operating systems, filesystems, etc. all log the error format
>>   differently, but for larger installations it'd not be too hard to
>>   write code that checks their specific configuration.
>>
>>   While I'm a bit concerned adding user-code before a checkpoint, if
>>   we'd do it as a shell command it seems pretty reasonable. And useful
>>   even without concern for the fsync issue itself. Checking for IO
>>   errors could e.g. also include checking for read errors - it'd not be
>>   unreasonable to not want to complete a checkpoint if there'd been any
>>   media errors.
>
> It seems clear that we need to evaluate our compatibility not just
> with an OS, as we do now, but with an OS/filesystem.
>
> Although people have suggested some approaches, I'm more interested in
> discovering how we can be certain we got it right.
>
> And the end result seems to be that PostgreSQL will be forced, in the
> short term, to declare certain combinations of OS/filesystem
> unsupported, with clear warning sent out to users.
>
> Adding a pre-checkpoint hook encourages people to fix this themselves
> without reporting issues, so I initially oppose this until we have a
> clearer argument as to why we need it. The answer is not to make this
> issue more obscure, but to make it more public.

Thinking some more, I think I understand, but please explain if not.

We need behavior that varies according to OS and filesystem, which
varies per tablespace.

We could have that variable behavior using

* a hook

* a set of GUC parameters that can be set at tablespace level

* a separate config file for each tablespace

My preference would be to avoid a hook.

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services