Horton

Thread

Commits

  1. Fix doc oversight for custom WAL resource managers.

  2. Custom WAL Resource Managers.

  1. Logical replication support for generic wal record

    Natarajan R <nataraj3098@gmail.com> — 2022-08-22T06:29:05Z

    Hi All,

I am writing a postgres extension which writes only generic wal record, but
this wal is not recognized by logical replication decoder. I have a basic
understanding of how logical replication(COPY command for initial sync, wal
replica for final sync) works, can you please tell us a way to support this?


Thanks,
Natarajan.R

  2. Re: Logical replication support for generic wal record

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2022-08-22T06:45:45Z

    On Mon, Aug 22, 2022 at 11:59 AM Natarajan R <nataraj3098@gmail.com> wrote:
>
> Hi All,
>
> I am writing a postgres extension which writes only generic wal record, but this wal is not recognized by logical replication decoder. I have a basic understanding of how logical replication(COPY command for initial sync, wal replica for final sync) works, can you please tell us a way to support this?

"Generic" resource manager doesn't have a decoding API, see [1], which
means that the generic WAL records will not get decoded.

Can you be more specific about the use-case? Why use only "Generic"
type WAL records? Why not use "LogicalMessage" type WAL records if you
want your WAL records to be decoded?

[1] https://github.com/postgres/postgres/blob/master/src/include/access/rmgrlist.h#L48

-- 
Bharath Rupireddy
RDS Open Source Databases: https://aws.amazon.com/rds/postgresql/

  3. Re: Logical replication support for generic wal record

    Amit Kapila <amit.kapila16@gmail.com> — 2022-08-22T08:31:50Z

    On Mon, Aug 22, 2022 at 11:59 AM Natarajan R <nataraj3098@gmail.com> wrote:
>
> Hi All,
>
> I am writing a postgres extension which writes only generic wal record, but this wal is not recognized by logical replication decoder. I have a basic understanding of how logical replication(COPY command for initial sync, wal replica for final sync) works, can you please tell us a way to support this?
>

Did you try with a custom WAL resource manager [1][2]?

[1] - https://www.postgresql.org/docs/devel/custom-rmgr.html
[2] - https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=5c279a6d350205cc98f91fb8e1d3e4442a6b25d1

-- 
With Regards,
Amit Kapila.

  4. Re: Logical replication support for generic wal record

    Natarajan R <nataraj3098@gmail.com> — 2022-08-24T11:42:28Z

    On Mon, 22 Aug 2022 at 12:16, Bharath Rupireddy <
bharath.rupireddyforpostgres@gmail.com> wrote:

> On Mon, Aug 22, 2022 at 11:59 AM Natarajan R <nataraj3098@gmail.com>
> wrote:
> >
> > Hi All,
> >
> > I am writing a postgres extension which writes only generic wal record,
> but this wal is not recognized by logical replication decoder. I have a
> basic understanding of how logical replication(COPY command for initial
> sync, wal replica for final sync) works, can you please tell us a way to
> support this?
>
> "Generic" resource manager doesn't have a decoding API, see [1], which
> means that the generic WAL records will not get decoded.
>
> Can you be more specific about the use-case? Why use only "Generic"
> type WAL records? Why not use "LogicalMessage" type WAL records if you
> want your WAL records to be decoded?
>
> I am writing an extension which implements postgres table access method
interface[1] with master-slave architecture, with the help of doc[1] i
decided to go with generic_wal to achieve crash_safety and also for
streaming replication. It seems like generic_wal couldn't help with logical
replication..
But, I don't have knowledge on "LogicalMessage" Resource Manager, need to
explore about it.


[1] https://www.postgresql.org/docs/current/tableam.html

  5. Re: Logical replication support for generic wal record

    Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> — 2022-08-24T12:30:16Z

    On Wed, Aug 24, 2022 at 5:12 PM Natarajan R <nataraj3098@gmail.com> wrote:
>
>
> On Mon, 22 Aug 2022 at 12:16, Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> wrote:
>>
>> On Mon, Aug 22, 2022 at 11:59 AM Natarajan R <nataraj3098@gmail.com> wrote:
>> >
>> > Hi All,
>> >
>> > I am writing a postgres extension which writes only generic wal record, but this wal is not recognized by logical replication decoder. I have a basic understanding of how logical replication(COPY command for initial sync, wal replica for final sync) works, can you please tell us a way to support this?
>>
>> "Generic" resource manager doesn't have a decoding API, see [1], which
>> means that the generic WAL records will not get decoded.
>>
>> Can you be more specific about the use-case? Why use only "Generic"
>> type WAL records? Why not use "LogicalMessage" type WAL records if you
>> want your WAL records to be decoded?
>>
> I am writing an extension which implements postgres table access method interface[1] with master-slave architecture, with the help of doc[1] i decided to go with generic_wal to achieve crash_safety and also for streaming replication. It seems like generic_wal couldn't help with logical replication..
> But, I don't have knowledge on "LogicalMessage" Resource Manager, need to explore about it.
>
>
> [1] https://www.postgresql.org/docs/current/tableam.html

I think the 'Custom WAL Resource Managers' feature would serve the
exact same purpose (as also pointed out by Amit upthread), you may
want to explore that feature [1]. Here's a sample extension using that
feature [2], for a different purpose though, but helps to understand
the usage of custom WAL rmgrs.

I notice that the docs [3] don't mention the feature in the right
place, IMO, it can be improved to refer to custom-rmgr.sgml page,
cc-ing Jeff Davis for his thoughts. This would help developers quickly
try the feature out and saves time.

[1] https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=5c279a6d350205cc98f91fb8e1d3e4442a6b25d1
[2] https://github.com/BRupireddy/pg_synthesize_wal
[2] https://www.postgresql.org/docs/devel/tableam.html
"For crash safety, an AM can use postgres' WAL, or a custom
implementation. If WAL is chosen, either Generic WAL Records can be
used, or a new type of WAL records can be implemented. Generic WAL
Records are easy, but imply higher WAL volume. Implementation of a new
type of WAL record currently requires modifications to core code
(specifically, src/include/access/rmgrlist.h)."

--
Bharath Rupireddy
RDS Open Source Databases: https://aws.amazon.com/rds/postgresql/

  6. Re: Logical replication support for generic wal record

    Natarajan R <nataraj3098@gmail.com> — 2022-08-24T13:00:29Z

    Thanks, I'll check it out.

On Wed, 24 Aug 2022 at 18:00, Bharath Rupireddy <
bharath.rupireddyforpostgres@gmail.com> wrote:

> On Wed, Aug 24, 2022 at 5:12 PM Natarajan R <nataraj3098@gmail.com> wrote:
> >
> >
> > On Mon, 22 Aug 2022 at 12:16, Bharath Rupireddy <
> bharath.rupireddyforpostgres@gmail.com> wrote:
> >>
> >> On Mon, Aug 22, 2022 at 11:59 AM Natarajan R <nataraj3098@gmail.com>
> wrote:
> >> >
> >> > Hi All,
> >> >
> >> > I am writing a postgres extension which writes only generic wal
> record, but this wal is not recognized by logical replication decoder. I
> have a basic understanding of how logical replication(COPY command for
> initial sync, wal replica for final sync) works, can you please tell us a
> way to support this?
> >>
> >> "Generic" resource manager doesn't have a decoding API, see [1], which
> >> means that the generic WAL records will not get decoded.
> >>
> >> Can you be more specific about the use-case? Why use only "Generic"
> >> type WAL records? Why not use "LogicalMessage" type WAL records if you
> >> want your WAL records to be decoded?
> >>
> > I am writing an extension which implements postgres table access method
> interface[1] with master-slave architecture, with the help of doc[1] i
> decided to go with generic_wal to achieve crash_safety and also for
> streaming replication. It seems like generic_wal couldn't help with logical
> replication..
> > But, I don't have knowledge on "LogicalMessage" Resource Manager, need
> to explore about it.
> >
> >
> > [1] https://www.postgresql.org/docs/current/tableam.html
>
> I think the 'Custom WAL Resource Managers' feature would serve the
> exact same purpose (as also pointed out by Amit upthread), you may
> want to explore that feature [1]. Here's a sample extension using that
> feature [2], for a different purpose though, but helps to understand
> the usage of custom WAL rmgrs.
>
> I notice that the docs [3] don't mention the feature in the right
> place, IMO, it can be improved to refer to custom-rmgr.sgml page,
> cc-ing Jeff Davis for his thoughts. This would help developers quickly
> try the feature out and saves time.
>
> [1]
> https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=5c279a6d350205cc98f91fb8e1d3e4442a6b25d1
> [2] https://github.com/BRupireddy/pg_synthesize_wal
> [2] https://www.postgresql.org/docs/devel/tableam.html
> "For crash safety, an AM can use postgres' WAL, or a custom
> implementation. If WAL is chosen, either Generic WAL Records can be
> used, or a new type of WAL records can be implemented. Generic WAL
> Records are easy, but imply higher WAL volume. Implementation of a new
> type of WAL record currently requires modifications to core code
> (specifically, src/include/access/rmgrlist.h)."
>
> --
> Bharath Rupireddy
> RDS Open Source Databases: https://aws.amazon.com/rds/postgresql/
>