Horton
Menu

Re: Adding REPACK [concurrently]

Antonin Houska <ah@cybertec.at>

View thread
From: Antonin Houska <ah@cybertec.at>
To: Andres Freund <andres@anarazel.de>
Cc: Mihail Nikalayeu <mihailnikalayeu@gmail.com>, Amit Kapila <amit.kapila16@gmail.com>, Alvaro Herrera <alvherre@alvh.no-ip.org>, Srinath Reddy Sadipiralla <srinath2133@gmail.com>, Matthias van de Meent <boekewurm+postgres@gmail.com>, Pg Hackers <pgsql-hackers@lists.postgresql.org>, Robert Treat <rob@xzilla.net>
Date: 2026-04-15T14:50:11Z
Lists: pgsql-hackers 
Andres Freund <andres@anarazel.de> wrote:

> On 2026-04-14 15:37:56 +0200, Antonin Houska wrote:
> > Andres Freund <andres@anarazel.de> wrote:
> > 
> > > On 2026-04-12 15:31:20 +0200, Mihail Nikalayeu wrote:
> > > > Instead of cancelling the backend entered the deadlock detector - it
> > > > cancel some another (nearest hard edge) until it is possible to get the
> > > > lock (either by
> > > > reordering or directly).
> > > 
> > > I don't think that's as good.  The problem is that that way you're only
> > > detecting the deadlocks once they have materialized (i.e. once repack actually
> > > does the lock upgrade), rather than cancelling when we know that the problem
> > > starts.
> > 
> > This is my hack that tries to do that.
> 
> I still think this needs to be in the deadlock detector.  The lock cycle just
> needs to be a bit more complicated for a hack in JoinWaitQueue not to work.
> There's no guarantee that the wait that triggers the deadlock is actually on
> the relation being repacked.

ok, I see.

I thought of a "hypothetical graph", which would include the to-be-granted
lock, but the major issue is that it will not work correctly without the
locking the LMGR's LW locks we do in CheckDeadLock():

    for (i = 0; i < NUM_LOCK_PARTITIONS; i++)
        LWLockAcquire(LockHashPartitionLockByIndex(i), LW_EXCLUSIVE);

And obviously, doing this each time we want to insert a lock into the queue
would be bad for performance. It's even mentioned in the storage/lmgr/README
that the current approach is optimistic, so I think that major rework would be
needed if we wanted to entirely avoid waiting that leads to deadlock.

The approach proposed by Mihail [1] seems the least problematic to me, and
something like that occurred to me when I thought about the problem the first
time. However, when we wake up the other processes in order to run the
deadlock detection, they should do that immediately. I've got no good idea
about implementation at the moment, since latch can be set for unrelated
reasons. (Besides that, I have some more questions about this patch, which I
can post separately.)


[1] https://www.postgresql.org/message-id/CADzfLwURKVNQ%2B%2BDpi7bjoGfj-8pchDQEVex3eWBx0NCYn6TbDQ%40mail.gmail.com

-- 
Antonin Houska
Web: https://www.cybertec-postgresql.com

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Allow old WAL recycling during REPACK CONCURRENTLY

  2. Advance restart_lsn more eagerly in LogicalConfirmReceivedLocation

  3. Remove unnecessary signal handler change

  4. Improve REPACK (CONCURRENTLY) error messages some more

  5. Revert "Allow logical replication snapshots to be database-specific"

  6. Move REPACK (CONCURRENTLY) test out of stock regression tests

  7. REPACK: do not require REPLICATION or LOGIN

  8. Add missing initialization

  9. Simplify declaration of memcpy target

  10. Reserve replication slots specifically for REPACK

  11. doc: Add an example of REPACK (CONCURRENTLY)

  12. Allow logical replication snapshots to be database-specific

  13. Avoid different-size pointer-to-integer cast

  14. Fix valgrind failure

  15. Add CONCURRENTLY option to REPACK

  16. Rename cluster.c to repack.c (and corresponding .h)

  17. Allow index_create to suppress index_build progress reporting

  18. Make index_concurrently_create_copy more general

  19. Document the 'command' column of pg_stat_progress_repack

  20. Introduce the REPACK command

  21. Toggle logical decoding dynamically based on logical slot presence.

  22. Split vacuumdb to create vacuuming.c/h

  23. Remove ReorderBufferTupleBuf structure.

  24. Revert changes to CONCURRENTLY that "sped up" Xmin advance

  25. VACUUM: ignore indexing operations with CONCURRENTLY