Horton

Re: refactoring relation extension and BufferAlloc(), faster COPY

Heikki Linnakangas <hlinnaka@iki.fi>

View thread View on postgr.es
From: Heikki Linnakangas <hlinnaka@iki.fi>
To: Andres Freund <andres@anarazel.de>
Cc: vignesh C <vignesh21@gmail.com>, pgsql-hackers@postgresql.org, Thomas Munro <thomas.munro@gmail.com>, Melanie Plageman <melanieplageman@gmail.com>, Yura Sokolov <y.sokolov@postgrespro.ru>, Robert Haas <robertmhaas@gmail.com>
Date: 2023-02-22T09:18:57Z
Lists: pgsql-hackers 
On 21/02/2023 21:22, Andres Freund wrote:
> On 2023-02-21 18:18:02 +0200, Heikki Linnakangas wrote:
>> Is it ever possible to call this without a relcache entry? WAL redo
>> functions do that with ReadBuffer, but they only extend a relation
>> implicitly, by replay a record for a particular block.
> 
> I think we should use it for crash recovery as well, but the patch doesn't
> yet. We have some gnarly code there, see the loop using P_NEW in
> XLogReadBufferExtended(). Extending the file one-by-one is a lot more
> expensive than doing it in bulk.

Hmm, XLogReadBufferExtended() could use smgrzeroextend() to fill the 
gap, and then call ExtendRelationBuffered for the target page. Or the 
new ExtendRelationBufferedTo() function that you mentioned.

In the common case that you load a lot of data to a relation extending 
it, and then crash, the WAL replay would still extend the relation one 
page at a time, which is inefficient. Changing that would need bigger 
changes, to WAL-log the relation extension as a separate WAL record, for 
example. I don't think we need to solve that right now, it can be 
addressed separately later.

- Heikki

Commits

  1. lwlock: Fix quadratic behavior with very long wait lists

  2. Support RBM_ZERO_AND_CLEANUP_LOCK in ExtendBufferedRelTo(), add tests

  3. Use ExtendBufferedRelTo() in XLogReadBufferExtended()

  4. hio: Use ExtendBufferedRelBy() to extend tables more efficiently

  5. heapam: Pass number of required pages to RelationGetBufferForTuple()

  6. Convert many uses of ReadBuffer[Extended](P_NEW) to ExtendBufferedRel()

  7. Use ExtendBufferedRelTo() in {vm,fsm}_extend()

  8. bufmgr: Introduce infrastructure for faster relation extension

  9. bufmgr: Support multiple in-progress IOs by using resowner

  10. bufmgr: Acquire and clean victim buffer separately

  11. bufmgr: Add Pin/UnpinLocalBuffer()

  12. bufmgr: Add some more error checking [infrastructure] around pinning

  13. Add smgrzeroextend(), FileZero(), FileFallocate()

  14. Don't initialize page in {vm,fsm}_extend(), not needed

  15. bufmgr: Remove buffer-write-dirty tracepoints

  16. hio: Release extension lock before initializing page / pinning VM

  17. bufmgr: Fix undefined behaviour with, unrealistically, large temp_buffers

  18. Fix memory leak and inefficiency in CREATE DATABASE ... STRATEGY WAL_LOG

  19. pgstat: Track more detailed relation IO statistics

  20. Use unnamed POSIX semaphores on Cygwin.

  21. Include RelFileLocator fields individually in BufferTag.