Horton

Re: Logical replication timeout problem

Fabrice Chapuis <fabrice636861@gmail.com>

View thread View on postgr.es
From: Fabrice Chapuis <fabrice636861@gmail.com>
To: "wangw.fnst@fujitsu.com" <wangw.fnst@fujitsu.com>
Cc: Amit Kapila <amit.kapila16@gmail.com>, Simon Riggs <simon.riggs@enterprisedb.com>, Petr Jelinek <petr.jelinek@enterprisedb.com>, "tanghy.fnst@fujitsu.com" <tanghy.fnst@fujitsu.com>, PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Date: 2022-02-09T09:41:12Z
Lists: pgsql-hackers 
Thanks for your patch, it works well in my test lab.
I added the definition *extern in wal_sender_timeout;* in the
*output_plugin.h* file for compilation works.
I tested the patch for version 10 which is currently in production on our
systems.
The functions below are only in master branch:
pgoutput_prepare_txn functions,
pgoutput_commit_prepared_txn,
pgoutput_rollback_prepared_txn,
pgoutput_stream_commit,
pgoutput_stream_prepare_txn

Will the patch be proposed retroactively to version 13-12-11-10.

Best regards,

Fabrice

On Tue, Feb 8, 2022 at 3:59 AM wangw.fnst@fujitsu.com <
wangw.fnst@fujitsu.com> wrote:

> On Wed, Jan 26, 2022 at 11:37 AM I wrote:
> > On Sat, Jan 22, 2022 at 7:12 PM Amit Kapila <amit.kapila16@gmail.com>
> wrote:
> > > Now, one idea to solve this problem could be that whenever we skip
> > > sending any change we do try to update the plugin progress via
> > > OutputPluginUpdateProgress(for walsender, it will invoke
> > > WalSndUpdateProgress), and there it tries to process replies and send
> > > keep_alive if necessary as we do when we send some data via
> > > OutputPluginWrite(for walsender, it will invoke WalSndWriteData). I
> > > don't know whether it is a good idea to invoke such a mechanism for
> > > every change we skip to send or we should do it after we skip sending
> > > some threshold of continuous changes. I think later would be
> > > preferred. Also, we might want to introduce a new parameter
> > > send_keep_alive to this API so that there is flexibility to invoke
> > > this mechanism as we don't need to invoke it while we are actually
> > > sending data and before that, we just update the progress via this
> > > API.
> > ......
> > Based on above, I think the second idea that sending some threshold of
> > continuous changes might be better, I will do some research about this
> > approach.
> Based on the second idea, I wrote a new patch(see attachment).
>
> Regards,
> Wang wei
>

Commits

  1. Fix the logical replication timeout during large DDLs.

  2. Fix the logical replication timeout during large transactions.

  3. Rethink the delay-checkpoint-end mechanism in the back-branches.

  4. Revert "Logical decoding of sequences"

  5. Skip empty transactions for logical replication.

  6. Allow specifying column lists for logical replication

  7. Add decoding of sequences to built-in replication

  8. Fix ABI break introduced by commit 4daa140a2f.

  9. Lag tracking for logical replication