Re: initdb and fsync

On Tuesday, March 13, 2012 04:49:40 AM Jeff Davis wrote:
> On Sun, 2012-02-05 at 17:56 -0500, Noah Misch wrote:
> > I meant primarily to illustrate the need to be comprehensive, not comment
> > on which executable should fsync a particular file.  Bootstrap-mode
> > backends do not sync anything during an initdb run on my system.  With
> > your patch, we'll fsync a small handful of files and leave nearly
> > everything else vulnerable.
> 
> Thank you for pointing that out. With that in mind, I have a new version
> of the patch which just recursively fsync's the whole directory
> (attached).
> 
> I also introduced a new option --nosync (-N) to disable this behavior.
> 
> The bad news is that it introduces a lot more time to initdb -- it goes
> from about 1s to about 10s on my machine. I tried fsync'ing the whole
> directory twice just to make sure that the second was a no-op, and
> indeed it didn't make much difference (still about 10s). 
I suggest you try making it two loops:

for recursively everything in dir:
   posix_fadvise(fd, POSIX_FADV_DONTNEED);

for recursively everything in dir:
   fsync(fd);

In my experience that gives way much better performance due to the fact that 
it does not force its own metadata/journal commit/transaction for every file 
but can be batched. copydir() does the same since some releases...

Obviously its not that nice to use _DONTNEED but I havent found something that 
works equally well. You could try sync_file_range(fd, 0, 0, 
SYNC_FILE_RANGE_WRITE) in the first loop but my experience with that hasn't 
been that good.

Andres