Horton
Menu

Re: Adding REPACK [concurrently]

Alvaro Herrera <alvherre@alvh.no-ip.org>

View thread
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
To: Antonin Houska <ah@cybertec.at>
Cc: Mihail Nikalayeu <mihailnikalayeu@gmail.com>, Srinath Reddy Sadipiralla <srinath2133@gmail.com>, Matthias van de Meent <boekewurm+postgres@gmail.com>, Pg Hackers <pgsql-hackers@lists.postgresql.org>, Robert Treat <rob@xzilla.net>
Date: 2026-03-27T17:01:09Z
Lists: pgsql-hackers

Attachments

On 2026-Mar-27, Antonin Houska wrote:

> I tried to write a patch that allows progress tracking of two commands at the
> same time (a "main command" and a "subcommand"), but regression tests revealed
> that contrib/file_fdw is broken in a way that I could not fix easily: during
> execution of a join, two COPY FROM commands are executed at the same time and
> they overwrite the status of each other. Unlike the concept of a sub-command,
> we cannot assume here that the command that started the reporting as the
> second will stop as the first. Thus in pgstat_progress_end_command() we cannot
> figure out which node is trying to stop the reporting.

Yeah, I think we've discussed this kind of thing in the context of the
index creation by CLUSTER before, but no ideas have emerged on what the
best implementation is.

> It needs more work, I can get back to it after PG 19 feature freeze.

Yeah, clearly it's not getting in pg19.  I'm not terribly upset about
that though; it would be nice to have, but it's not a dealbreaker.

Anyway, I've been changing how all the new code for REPACK is
distributed; here's patch series v45, and the breakdown and my
assessment is:

0001 is looking good.  I don't commit it just yet because there's no
point in doing so until 0003 is also committable.

0002 continues to baffle me.  I would much rather do without it.

0003 is the addition of CONCURRENTLY and all it needs.  Regarding
structure, I think it's good to have the code that runs in a separate
worker have its own file, which I named commands/repack_worker.c.
Luckily it's all well contained.  It shares data structures and whatnot
with the other parts of REPACK via include/commands/repack_internal.h.

Patch "Use BulkInsertState when copying data to the new heap." is now
0004 and I put it right after 0003, and I will probably be squashing
them into a single one.

0005 is new -- I renamed cluster.c to repack.c, as previously mentioned.

Then 0006 is "Fix a few problems in index build progress reporting",
which I'm likely to also squash in 0003 for v46.

0007 is "Error out any process that would block at REPACK" which is the
complete patch to change the deadlock detector to raise an error at any
process that competes with REPACK.  This one I think I would commit
separately from 0003 -- if nothing else, because it introduces novel
behavior that could be contested, so I want to be able to easily revert
it without endangering the rest of CONCURRENTLY.

Lastly, 0008 is "Teach snapshot builder to skip transactions running
REPACK (CONCURRENTLY)" which I see as the least mature of the pack.  I
would really like to be able to squash it with 0003, but I'm not yet
trusting it enough.

-- 
Álvaro Herrera         PostgreSQL Developer  —  https://www.EnterpriseDB.com/
"No nos atrevemos a muchas cosas porque son difíciles,
pero son difíciles porque no nos atrevemos a hacerlas" (Séneca)

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Allow old WAL recycling during REPACK CONCURRENTLY

  2. Advance restart_lsn more eagerly in LogicalConfirmReceivedLocation

  3. Remove unnecessary signal handler change

  4. Improve REPACK (CONCURRENTLY) error messages some more

  5. Revert "Allow logical replication snapshots to be database-specific"

  6. Move REPACK (CONCURRENTLY) test out of stock regression tests

  7. REPACK: do not require REPLICATION or LOGIN

  8. Add missing initialization

  9. Simplify declaration of memcpy target

  10. Reserve replication slots specifically for REPACK

  11. doc: Add an example of REPACK (CONCURRENTLY)

  12. Allow logical replication snapshots to be database-specific

  13. Avoid different-size pointer-to-integer cast

  14. Fix valgrind failure

  15. Add CONCURRENTLY option to REPACK

  16. Rename cluster.c to repack.c (and corresponding .h)

  17. Allow index_create to suppress index_build progress reporting

  18. Make index_concurrently_create_copy more general

  19. Document the 'command' column of pg_stat_progress_repack

  20. Introduce the REPACK command

  21. Toggle logical decoding dynamically based on logical slot presence.

  22. Split vacuumdb to create vacuuming.c/h

  23. Remove ReorderBufferTupleBuf structure.

  24. Revert changes to CONCURRENTLY that "sped up" Xmin advance

  25. VACUUM: ignore indexing operations with CONCURRENTLY