Horton

Re: refactoring relation extension and BufferAlloc(), faster COPY

Nasby, Jim <nasbyj@amazon.com>

View thread View on postgr.es
From: Jim Nasby <nasbyj@amazon.com>
To: Andres Freund <andres@anarazel.de>, <pgsql-hackers@postgresql.org>, "Thomas Munro" <thomas.munro@gmail.com>, Melanie Plageman <melanieplageman@gmail.com>
Cc: Yura Sokolov <y.sokolov@postgrespro.ru>, Robert Haas <robertmhaas@gmail.com>
Date: 2023-02-21T21:00:15Z
Lists: pgsql-hackers 
On 10/28/22 9:54 PM, Andres Freund wrote:
> b) I found that is quite beneficial to bulk-extend the relation with
>     smgrextend() even without concurrency. The reason for that is the primarily
>     the aforementioned dirty buffers that our current extension method causes.
>
>     One bit that stumped me for quite a while is to know how much to extend the
>     relation by. RelationGetBufferForTuple() drives the decision whether / how
>     much to bulk extend purely on the contention on the extension lock, which
>     obviously does not work for non-concurrent workloads.
>
>     After quite a while I figured out that we actually have good information on
>     how much to extend by, at least for COPY /
>     heap_multi_insert(). heap_multi_insert() can compute how much space is
>     needed to store all tuples, and pass that on to
>     RelationGetBufferForTuple().
>
>     For that to be accurate we need to recompute that number whenever we use an
>     already partially filled page. That's not great, but doesn't appear to be a
>     measurable overhead.
Some food for thought: I think it's also completely fine to extend any 
relation over a certain size by multiple blocks, regardless of 
concurrency. E.g. 10 extra blocks on an 80MB relation is 0.1%. I don't 
have a good feel for what algorithm would make sense here; maybe 
something along the lines of extend = max(relpages / 2048, 128); if 
extend < 8 extend = 1; (presumably extending by just a couple extra 
pages doesn't help much without concurrency).

Commits

  1. lwlock: Fix quadratic behavior with very long wait lists

  2. Support RBM_ZERO_AND_CLEANUP_LOCK in ExtendBufferedRelTo(), add tests

  3. Use ExtendBufferedRelTo() in XLogReadBufferExtended()

  4. hio: Use ExtendBufferedRelBy() to extend tables more efficiently

  5. heapam: Pass number of required pages to RelationGetBufferForTuple()

  6. Convert many uses of ReadBuffer[Extended](P_NEW) to ExtendBufferedRel()

  7. Use ExtendBufferedRelTo() in {vm,fsm}_extend()

  8. bufmgr: Introduce infrastructure for faster relation extension

  9. bufmgr: Support multiple in-progress IOs by using resowner

  10. bufmgr: Acquire and clean victim buffer separately

  11. bufmgr: Add Pin/UnpinLocalBuffer()

  12. bufmgr: Add some more error checking [infrastructure] around pinning

  13. Add smgrzeroextend(), FileZero(), FileFallocate()

  14. Don't initialize page in {vm,fsm}_extend(), not needed

  15. bufmgr: Remove buffer-write-dirty tracepoints

  16. hio: Release extension lock before initializing page / pinning VM

  17. bufmgr: Fix undefined behaviour with, unrealistically, large temp_buffers

  18. Fix memory leak and inefficiency in CREATE DATABASE ... STRATEGY WAL_LOG

  19. pgstat: Track more detailed relation IO statistics

  20. Use unnamed POSIX semaphores on Cygwin.

  21. Include RelFileLocator fields individually in BufferTag.