Horton

Re: WAL logging problem in 9.4.3?

Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>

View thread View on postgr.es
From: Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
To: michael.paquier@gmail.com
Cc: hlinnaka@iki.fi, alvherre@2ndquadrant.com, simon@2ndquadrant.com, tgl@sss.pgh.pa.us, andres@anarazel.de, masao.fujii@gmail.com, kleptog@svana.org, pgsql-hackers@postgresql.org
Date: 2016-03-16T02:01:58Z
Lists: pgsql-hackers 
Thank you for the comment.

I understand that this is not an issue in a hurry so don't bother
to reply.

At Tue, 15 Mar 2016 18:21:34 +0100, Michael Paquier <michael.paquier@gmail.com> wrote in <CAB7nPqSVm-X1-w9i=U=DCyMxDxzfNT-41pqTSvh0DUmUgi8BQg@mail.gmail.com>
> On Fri, Mar 11, 2016 at 9:32 AM, Kyotaro HORIGUCHI
> <horiguchi.kyotaro@lab.ntt.co.jp> wrote:
> > At Fri, 19 Feb 2016 22:27:00 +0900, Michael Paquier <michael.paquier@gmail.com> wrote in <CAB7nPqSGFKUAFqPe5t30jeEA+V9yFMM4yJGa3SnkgY1RHzn7Dg@mail.gmail.com>
> >> > Worth noting that this patch does not address the problem with index
> >> > relations when a TRUNCATE is used in the same transaction as its
> >
> > Focusing this issue, what we should do is somehow building empty
> > index just after a index truncation. The attached patch does the
> > following things to fix this.
> >
> > - make index_build use ambuildempty when the relation on which
> >   the index will be built is apparently empty. That is, when the
> >   relation has no block.
> > - add one parameter "persistent" to ambuildempty(). It behaves as
> >   before if the parameter is false. If not, it creates an empty
> >   index on MAIN_FORK and emits logs even if wal_level is minimal.
> 
> Hm. It seems to me that this patch is just a bandaid for the real
> problem which is that we should not TRUNCATE the underlying index
> relations when the TRUNCATE optimization is running.

The eventual problem is a 0-length index relation left just after
a relation truncation. We assume that an index with an empty
relation after a recovery is not valid. However just skipping
TRUNCATE of the index relation won't resolve it since it in turn
leaves an index with garbage entries. Am I missing something?

Since the index relation should be "validly emptied" in-place in
any way in the case of TRUNCATE optimization, I tried that by
TRUNCATE + ambuildempty, which can be redo'ed properly, too. A
repeated TRUNCATEs issues eventully-useless logs but it would be
inevitable since we cannot fortell of any succeeding TRUNCATEs.

(TRUNCATE+)COPY+INSERT seems another kind of problem, which would
be fixed by Heikki's patch.

> In short I would
> let the empty routines in AM code paths alone, and just continue using
> them for the generation of INIT_FORKNUM with unlogged relations. Your
> patch is not something backpatchable anyway I think.

It seems to be un-backpatchable if the change of the manner to
call ambuildempty inhibits this.

> > The new parameter 'persistent' would be better be forknum because
> > it actually represents the persistency of the index to be
> > created. But I'm out of time now..
> 
> I actually have some users running with wal_level to minimal, even if
> I don't think they use this optimization, we had better fix even index
> relations at the same time as table relations.. I'll try to get some
> time once the patch review storm goes down a little, except if someone
> beats me to it first.

Ok, I understand that this is not an issue in a hurry. I'll go to
another patch that needs review.

regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

Commits

  1. Add perl2host call missing from a new test file.

  2. Skip WAL for new relfilenodes, under wal_level=minimal.

  3. Revert "Skip WAL for new relfilenodes, under wal_level=minimal."

  4. Back-patch log_newpage_range().

  5. During heap rebuild, lock any TOAST index until end of transaction.

  6. In log_newpage_range(), heed forkNum and page_std arguments.

  7. Back-patch src/test/recovery and PostgresNode from 9.6 to 9.5.

  8. Reduce pg_ctl's reaction time when waiting for postmaster start/stop.

  9. Accelerate end-of-transaction dropping of relations

  10. Redesign the planner's handling of index-descent cost estimation.

  11. Make TRUNCATE do truncate-in-place when processing a relation that was created