Horton

Re: [HACKERS] WAL logging problem in 9.4.3?

Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>

View thread View on postgr.es
From: Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
To: noah@leadboat.com
Cc: 9erthalion6@gmail.com, andrew.dunstan@2ndquadrant.com, hlinnaka@iki.fi, robertmhaas@gmail.com, michael@paquier.xyz, pgsql-hackers@postgresql.org
Date: 2019-03-26T07:35:07Z
Lists: pgsql-hackers

Attachments

Hello. I revised the patch I think addressing all your comments.

Differences from v7 patch are:

v9-0001:

 - Renamed the script from 016_ to 017_.

 - Added some additional tests.


v9-0002:
 - Fixed _bt_blwritepage().
   It is re-modified by v9-0007.


v9-0003: New patch.
 - Refactors out xlog sutff from heap_insert/delete.
   (log_heap_insert(), log_heap_udpate())


v9-0004: (v7-0003, v8-0004)
 - Renamed some struct names and member names.
   (PendingRelSync -> RelWalRequirement
     .sync_above -> skip_wal_min_blk, .truncated_to -> wal_log_min_blk)

 - Rename the addtional members in RelationData to rd_*.

 - Explicitly initialize the additional members only in
   load_relcache_init_file().

 - Added new interface functions that accept block number and
   SMgrRelation.
   (BlockNeedsWAL(), RecordPendingSync())

 - Support subtransaction, (or invalidation).
   (RelWalRequirement.create_sxid, invalidate_sxid,
    RelationInvalidateWALRequirements(), smgrDoPendingSyncs())

 - Support forks.
   (RelWalRequirement.forks, smgrDoPendingSyncs(), RecordPendingSync())

 - Removd elog(LOG)s and a leftover comment.

v9-0005: (v7-0004, v8-0005)

 - Fixed heap_update().
   (heap_update())

v9-0006: New patch.

 - Modifies CLUSTER to skip WAL logging.

v9-0007: New patch.

 - Modifies ALTER TABLE SET TABLESPACE to skip WAL logging.

v9-0008: New patch.

 - Modifies btbuild to skip WAL logging.

 - Modifies btinsertonpg to skip WAL logging after truncation.

 - Overrites on v9-0002's change.


ALL:

 - Rebased.

 - Fixed typos and mistakes in comments.
   

> At Wed, 20 Mar 2019 17:17:54 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> wrote in <20190320.171754.171896368.horiguchi.kyotaro@lab.ntt.co.jp>
> > > We still use heap_sync() in CLUSTER.  Can we migrate CLUSTER to the newer
> > > heap_register_sync()?  Patch v7 makes some commands use the new way (COPY,
> > > CREATE TABLE AS, REFRESH MATERIALIZED VIEW, ALTER TABLE) and leaves other
> > > commands using the old way (CREATE INDEX USING btree, ALTER TABLE SET
> > > TABLESPACE, CLUSTER).  It would make the system simpler to understand if we
> > > eliminated the old way.  If that creates more problems than it solves, please
> > > at least write down a coding rule to explain why certain commands shouldn't
> > > use the old way.
> > 
> > Perhaps doable for TABLESPACE and CLUSTER. I'm not sure about
> > CREATE INDEX. I'll consider them.
> 
> I added the CLUSTER case in the new patchset.  For the SET
> TABLESPACE case, it works on SMGR layer and manipulates fork
> files explicitly but this stuff is Relation based and doesn't
> distinguish forks. We can modify this stuff to work on smgr and
> make it fork-aware but I don't think it is worth doing.
> 
> CREATE INDEX is not changed in this version. I continue to
> consider it.

I managed to simplify the change. Please look at v9-0008.

regards.

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

Commits

  1. Add perl2host call missing from a new test file.

  2. Skip WAL for new relfilenodes, under wal_level=minimal.

  3. Revert "Skip WAL for new relfilenodes, under wal_level=minimal."

  4. Back-patch log_newpage_range().

  5. During heap rebuild, lock any TOAST index until end of transaction.

  6. In log_newpage_range(), heed forkNum and page_std arguments.

  7. Back-patch src/test/recovery and PostgresNode from 9.6 to 9.5.

  8. Reduce pg_ctl's reaction time when waiting for postmaster start/stop.

  9. Accelerate end-of-transaction dropping of relations

  10. Redesign the planner's handling of index-descent cost estimation.

  11. Make TRUNCATE do truncate-in-place when processing a relation that was created