Horton
Menu

Re: Adding REPACK [concurrently]

Andres Freund <andres@anarazel.de>

View thread
From: Andres Freund <andres@anarazel.de>
To: Antonin Houska <ah@cybertec.at>
Cc: Alvaro Herrera <alvherre@alvh.no-ip.org>, Amit Kapila <amit.kapila16@gmail.com>, Mihail Nikalayeu <mihailnikalayeu@gmail.com>, Srinath Reddy Sadipiralla <srinath2133@gmail.com>, Matthias van de Meent <boekewurm+postgres@gmail.com>, Pg Hackers <pgsql-hackers@lists.postgresql.org>, Robert Treat <rob@xzilla.net>
Date: 2026-04-07T15:48:42Z
Lists: pgsql-hackers 
Hi,

On 2026-04-07 17:38:24 +0200, Antonin Houska wrote:
> Andres Freund <andres@anarazel.de> wrote:
> 
> > On 2026-04-07 14:33:50 +0200, Alvaro Herrera wrote:
> > > On 2026-Apr-07, Amit Kapila wrote:
> > > 
> > > > I have a question based on 0001's commit message: "This patch adds a
> > > > new option to logical replication output plugin, to declare that it
> > > > does not use shared catalogs (i.e. catalogs that can be changed by
> > > > transactions running in other databases in the cluster).". In which
> > > > cases, currently plugin needs to access multi-database transactions or
> > > > transactions that need to access shared catalogs and on what basis a
> > > > plugin can decide that the changes it requires won't need any such
> > > > access.
> > > 
> > > I don't think any plugin needs "multi-database" access as such, but
> > > needing access to shared catalogs is likely normal.  Repack knows it
> > > won't access any shared catalogs, so it can set the flag at ease.
> > > 
> > > There's a cross-check added in the commit that tests for access to
> > > shared catalogs if the flag is set to false.  I guess you could set it
> > > to false and see what breaks :-)
> > 
> > I think this has a quite high chance of indirect breakages. You just need some
> > cache invalidation processing / building accessing shared catalogs to violate
> > the rule, and whether that happens very heavily depends on what cache entries
> > are present and whether something registers relcache callbacks or such.
> > 
> > This can be triggered by an output function during logical decoding or such,
> > so you don't really have control over it.
> 
> The REPACK plugin only deforms tuples and writes them to a file, so I think
> that things like this should not happen.

You don't need to do it yourself.  It just requires a shared_preload_library
extension to register a relcache invalidation callback that accesses shared
catalog.

It's only kind of an accident that we don't have a case today that accesses
shared catcaches during a relcache build in core (I'm not even sure there's
nothing). You'd just need somebody to add e.g. relcache caching for
publications for that to change. Or look up information about a reloption in
pg_parameter_acl. Or lookup tablespace configuration.


> However, I admit that an option that allows the plugin developer to declare
> "I don't need shared catalogs" may be considered deceptive.

At the very least it would need to be a runtime check rather than just an
assert. This would much more likely to be hit in production because otherwise
it's probably hard to hit the case where shared invalidations happen in the
wrong moment.  And the consequences are corrupted caches, which could cause
all kinds of havoc.


But I think this may need more infrastructure / deeper analysis than what we
can do right now.

Greetings,

Andres Freund

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Allow old WAL recycling during REPACK CONCURRENTLY

  2. Advance restart_lsn more eagerly in LogicalConfirmReceivedLocation

  3. Remove unnecessary signal handler change

  4. Improve REPACK (CONCURRENTLY) error messages some more

  5. Revert "Allow logical replication snapshots to be database-specific"

  6. Move REPACK (CONCURRENTLY) test out of stock regression tests

  7. REPACK: do not require REPLICATION or LOGIN

  8. Add missing initialization

  9. Simplify declaration of memcpy target

  10. Reserve replication slots specifically for REPACK

  11. doc: Add an example of REPACK (CONCURRENTLY)

  12. Allow logical replication snapshots to be database-specific

  13. Avoid different-size pointer-to-integer cast

  14. Fix valgrind failure

  15. Add CONCURRENTLY option to REPACK

  16. Rename cluster.c to repack.c (and corresponding .h)

  17. Allow index_create to suppress index_build progress reporting

  18. Make index_concurrently_create_copy more general

  19. Document the 'command' column of pg_stat_progress_repack

  20. Introduce the REPACK command

  21. Toggle logical decoding dynamically based on logical slot presence.

  22. Split vacuumdb to create vacuuming.c/h

  23. Remove ReorderBufferTupleBuf structure.

  24. Revert changes to CONCURRENTLY that "sped up" Xmin advance

  25. VACUUM: ignore indexing operations with CONCURRENTLY