Horton
Menu

Thread

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Skip empty transactions for logical replication.

  1. Missing empty transaction optimization in pgoutput plugin

    Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> — 2025-12-08T06:48:55Z

    Hi All,
test_decoding output plugin skips sending stream_start and stream_end
if no change from a segment of a streamed transaction is sent
downstream. If no change has been sent downstream across all the
streamed segments it doesn't send commit or abort for that streamed
transaction. But I don't see similar changes in stream related
callbacks in pgoutput output plugin.

While the changes discussed in thread [1] were committed, those didn't
include this optimization for test_decoding plugin as well. The
optimization was discussed in [2] separately as a bug fix. Maybe
because it wasn't considered as an optimization that time, we may not
have considered it for pgoutput plugin which doesn't have explicit
skip-empty-xacts option. I think the optimization would be useful in
setups where multiple publications and long transactions are common.
Any reason not to have it.

[1] https://www.postgresql.org/message-id/flat/688b0b7f-2f6c-d827-c27b-216a8e3ea700%402ndquadrant.com
[2] https://www.postgresql.org/message-id/flat/CAA4eK1%2BOqgFNZkf7%3DETe_y5ntjgDk3T0wcdkd4Sot_u1hySGfw%40mail.gmail.com
-- 
Best Wishes,
Ashutosh Bapat

  2. Re: Missing empty transaction optimization in pgoutput plugin

    Amit Kapila <amit.kapila16@gmail.com> — 2025-12-08T12:10:53Z

    On Mon, Dec 8, 2025 at 12:19 PM Ashutosh Bapat
<ashutosh.bapat.oss@gmail.com> wrote:
>
> test_decoding output plugin skips sending stream_start and stream_end
> if no change from a segment of a streamed transaction is sent
> downstream. If no change has been sent downstream across all the
> streamed segments it doesn't send commit or abort for that streamed
> transaction. But I don't see similar changes in stream related
> callbacks in pgoutput output plugin.
>
> While the changes discussed in thread [1] were committed, those didn't
> include this optimization for test_decoding plugin as well. The
> optimization was discussed in [2] separately as a bug fix. Maybe
> because it wasn't considered as an optimization that time, we may not
> have considered it for pgoutput plugin which doesn't have explicit
> skip-empty-xacts option. I think the optimization would be useful in
> setups where multiple publications and long transactions are common.
> Any reason not to have it.
>

I don't remember any reason why we can't have the same optimization
for streamed transactions. IIRC, we implemented this optimization
because we got reports for extra messages for short empty-transactions
but if you see any practical cases where it could be useful for
streamed transactions as well then feel free to propose a patch.

-- 
With Regards,
Amit Kapila.

  3. RE: Missing empty transaction optimization in pgoutput plugin

    Zhijie Hou (Fujitsu) <houzj.fnst@fujitsu.com> — 2025-12-10T03:47:28Z

    On Monday, December 8, 2025 2:49 PM Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> wrote:
> 
> Hi All,
> test_decoding output plugin skips sending stream_start and stream_end if no
> change from a segment of a streamed transaction is sent downstream. If no
> change has been sent downstream across all the streamed segments it
> doesn't send commit or abort for that streamed transaction. But I don't see
> similar changes in stream related callbacks in pgoutput output plugin.
> 
> While the changes discussed in thread [1] were committed, those didn't
> include this optimization for test_decoding plugin as well. The optimization
> was discussed in [2] separately as a bug fix. Maybe because it wasn't
> considered as an optimization that time, we may not have considered it for
> pgoutput plugin which doesn't have explicit skip-empty-xacts option. I think
> the optimization would be useful in setups where multiple publications and
> long transactions are common.
> Any reason not to have it.

One possible reason we did not implement the optimization to avoid sending empty stream
blocks along with commit d5a9d86 is the handling of prepared transactions, for
which STREAM PREPARE is sent. The logical replication worker currently relies on
receiving at least one STREAM START/STREAM STOP prior to this; otherwise, it
might encounter errors when applying STREAM PREPARE. Although the optimization
is feasible, it requires some modifications to ensure the replication worker works
correctly, such as sending an empty stream block before issuing STREAM PREPARE.

One might think that we could avoid sending STREAM PREPARE for empty txn as
well, but that could be tricky, see the comment atop of (typedef struct
PGOutputTxnData) for details.

> 
> [1] https://www.postgresql.org/message-id/flat/688b0b7f-2f6c-d827-c27b-
> 216a8e3ea700%402ndquadrant.com
> [2] https://www.postgresql.org/message-
> id/flat/CAA4eK1%2BOqgFNZkf7%3DETe_y5ntjgDk3T0wcdkd4Sot_u1hySGfw%
> 40mail.gmail.com

Best Regards,
Hou zj

  4. Re: Missing empty transaction optimization in pgoutput plugin

    Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> — 2025-12-10T05:55:49Z

    On Wed, Dec 10, 2025 at 9:17 AM Zhijie Hou (Fujitsu)
<houzj.fnst@fujitsu.com> wrote:
>
> On Monday, December 8, 2025 2:49 PM Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> wrote:
> >
> > Hi All,
> > test_decoding output plugin skips sending stream_start and stream_end if no
> > change from a segment of a streamed transaction is sent downstream. If no
> > change has been sent downstream across all the streamed segments it
> > doesn't send commit or abort for that streamed transaction. But I don't see
> > similar changes in stream related callbacks in pgoutput output plugin.
> >
> > While the changes discussed in thread [1] were committed, those didn't
> > include this optimization for test_decoding plugin as well. The optimization
> > was discussed in [2] separately as a bug fix. Maybe because it wasn't
> > considered as an optimization that time, we may not have considered it for
> > pgoutput plugin which doesn't have explicit skip-empty-xacts option. I think
> > the optimization would be useful in setups where multiple publications and
> > long transactions are common.
> > Any reason not to have it.
>
> One possible reason we did not implement the optimization to avoid sending empty stream
> blocks along with commit d5a9d86 is the handling of prepared transactions, for
> which STREAM PREPARE is sent. The logical replication worker currently relies on
> receiving at least one STREAM START/STREAM STOP prior to this; otherwise, it
> might encounter errors when applying STREAM PREPARE. Although the optimization
> is feasible, it requires some modifications to ensure the replication worker works
> correctly, such as sending an empty stream block before issuing STREAM PREPARE.
>
> One might think that we could avoid sending STREAM PREPARE for empty txn as
> well, but that could be tricky, see the comment atop of (typedef struct
> PGOutputTxnData) for details.
>
> >
> > [1] https://www.postgresql.org/message-id/flat/688b0b7f-2f6c-d827-c27b-
> > 216a8e3ea700%402ndquadrant.com
> > [2] https://www.postgresql.org/message-
> > id/flat/CAA4eK1%2BOqgFNZkf7%3DETe_y5ntjgDk3T0wcdkd4Sot_u1hySGfw%
> > 40mail.gmail.com
>

Thanks a lot for the details. I see it's much larger change than just
a few lines like test_decoding.c.

-- 
Best Wishes,
Ashutosh Bapat