Horton

Thread

Commits

  1. doc: Improve pgoutput documentation.

  2. doc: Improve documentation discoverability for pgoutput.

  3. Use PG_MODULE_MAGIC_EXT in our installable shared libraries.

  1. Make pgoutput documentation easier to find

    Fujii Masao <masao.fujii@gmail.com> — 2025-08-03T01:32:17Z

    Hi,

The current documentation for pgoutput is buried in the logical streaming
replication protocol section (in protocol.sgml), and there's no index entry
for it. This makes it hard to discover and access, for example, when trying
to look up the options it supports.

I've often struggled to locate this information myself, so I'd like to
propose moving the pgoutput documentation to the logical decoding section
and adding an index entry. The attached patch does that. I think this change
will make it much easier for users to find the relevant details.

Thoughts?

Regards,

-- 
Fujii Masao

  2. Re: Make pgoutput documentation easier to find

    Peter Eisentraut <peter@eisentraut.org> — 2025-08-06T11:36:21Z

    On 03.08.25 03:32, Fujii Masao wrote:
> The current documentation for pgoutput is buried in the logical streaming
> replication protocol section (in protocol.sgml), and there's no index entry
> for it. This makes it hard to discover and access, for example, when trying
> to look up the options it supports.
> 
> I've often struggled to locate this information myself, so I'd like to
> propose moving the pgoutput documentation to the logical decoding section
> and adding an index entry. The attached patch does that. I think this change
> will make it much easier for users to find the relevant details.

This would move the documentation of pgoutput from "Internals" to 
"Server Programming".  So it's a question of whether this is something 
we want to advertise that people can use directly.  In the past, 
pgoutput was an implementation detail of logical replication.  But I 
gather people are using it for other things now?  Still not clear what 
kind of guarantees we want to give about its interfaces, for example.

  3. Re: Make pgoutput documentation easier to find

    Fujii Masao <masao.fujii@gmail.com> — 2025-08-06T13:48:05Z

    On Wed, Aug 6, 2025 at 8:36 PM Peter Eisentraut <peter@eisentraut.org> wrote:
>
> On 03.08.25 03:32, Fujii Masao wrote:
> > The current documentation for pgoutput is buried in the logical streaming
> > replication protocol section (in protocol.sgml), and there's no index entry
> > for it. This makes it hard to discover and access, for example, when trying
> > to look up the options it supports.
> >
> > I've often struggled to locate this information myself, so I'd like to
> > propose moving the pgoutput documentation to the logical decoding section
> > and adding an index entry. The attached patch does that. I think this change
> > will make it much easier for users to find the relevant details.
>
> This would move the documentation of pgoutput from "Internals" to
> "Server Programming".  So it's a question of whether this is something
> we want to advertise that people can use directly.  In the past,
> pgoutput was an implementation detail of logical replication.  But I
> gather people are using it for other things now?

I've heard that Debezium users, a tool for change data capture, can use
pgoutput as the logical decoding plugin. I also know users, including
some of my colleagues, who use pgoutput with pg_recvlogical to capture
messages inserted via pg_logical_emit_message().

So I think making the pgoutput documentation easier to find would be
helpful for those users.

Regards,

-- 
Fujii Masao

  4. Re: Make pgoutput documentation easier to find

    Euler Taveira <euler@eulerto.com> — 2025-08-07T13:35:23Z

    On Wed, Aug 6, 2025, at 10:48 AM, Fujii Masao wrote:
> On Wed, Aug 6, 2025 at 8:36 PM Peter Eisentraut <peter@eisentraut.org> wrote:
>>
>> On 03.08.25 03:32, Fujii Masao wrote:
>> > The current documentation for pgoutput is buried in the logical streaming
>> > replication protocol section (in protocol.sgml), and there's no index entry
>> > for it. This makes it hard to discover and access, for example, when trying
>> > to look up the options it supports.
>> >
>> > I've often struggled to locate this information myself, so I'd like to
>> > propose moving the pgoutput documentation to the logical decoding section
>> > and adding an index entry. The attached patch does that. I think this change
>> > will make it much easier for users to find the relevant details.
>>
>> This would move the documentation of pgoutput from "Internals" to
>> "Server Programming".  So it's a question of whether this is something
>> we want to advertise that people can use directly.  In the past,
>> pgoutput was an implementation detail of logical replication.  But I
>> gather people are using it for other things now?
>
> I've heard that Debezium users, a tool for change data capture, can use
> pgoutput as the logical decoding plugin. I also know users, including
> some of my colleagues, who use pgoutput with pg_recvlogical to capture
> messages inserted via pg_logical_emit_message().
>

pgoutput is our defacto output plugin and I think we consider it a mature piece
of code. If you are worried about compatibility, version 18 added the
pg_get_loaded_modules() function to find the loadable module version -- see
commit 55527368bd07.

There might be cases where you need to debug logical replication (conflicts?)
and you need to pass the exact output plugin options via SQL
(pg_logical_slot_*_changes) to obtain the changes.

If we keep it in the "Internals" chapter, it seems it is something to
PostgreSQL developers (as the chapter description says); it is not. Advanced
users can make good use of this. The new links will help users to find the
moved information if you are searching it into the "Internals" section.

Since you are proposing this change, I'm wondering if we shouldn't move
"Writing Logical Decoding Output Plugins" to "Internals" chapter. It contains
information to PostgreSQL developers. It is a bit off-topic for this thread but
we can also move part of the "Background Worker Processes", part of the
"Archive Modules" and part of the "OAuth Validator Modules" section (2
subsections that describe the API) into "Internals" chapter.


-- 
Euler Taveira
EDB   https://www.enterprisedb.com/

  5. Re: Make pgoutput documentation easier to find

    Fujii Masao <masao.fujii@gmail.com> — 2025-08-08T15:26:02Z

    On Thu, Aug 7, 2025 at 10:35 PM Euler Taveira <euler@eulerto.com> wrote:
> Since you are proposing this change, I'm wondering if we shouldn't move
> "Writing Logical Decoding Output Plugins" to "Internals" chapter. It contains
> information to PostgreSQL developers.

Maybe, but I don’t have a strong opinion. If we decide to do it, it would
be better to make the change in a separate patch.

Regards,

-- 
Fujii Masao

  6. Re: Make pgoutput documentation easier to find

    Chao Li <li.evan.chao@gmail.com> — 2025-08-14T04:04:40Z

    I applied the batch and built the html docs. I tested that all hyperlinks worked for me.

There are a few nit comments about wording:

diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index 593f784b69d..4516650983d 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml

+    <xref linkend="test-decoding"/>.  You can also write custom output plugins

nit: "write" -> "develop"

+         Protocol version. Currently versions <literal>1</literal>, <literal>2</literal>,

"Protocol version." -> "Specifies the protocol version."

+         Comma-separated list of publication names for which to subscribe
+         (receive changes). The individual publication names are treated

A comma-separated list of publication names to subscribe to. The individual publication names are treated

+         Boolean option to use binary transfer mode.  Binary mode is faster

nit: "use" -> "enable", as other places all use "enable"

diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index e56eac8fd0f..9fe48c5acad 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml

+   The <literal>START_REPLICATION</literal> command allows us to
+   pass options to the logical decoding output plugin associated with
+   the replication slot specified in the command.
+   See <xref linkend="logicaldecoding-pgoutput-options"/> for options
+   that are accepted by the standard (<literal>pgoutput</literal>) plugin.

nit: The START_REPLICATION command can pass options to the logical decoding output plugin associated with the specified replication slot. See Section 47.6.1.1 for the options supported by the standard (pgoutput) plugin.

The new status of this patch is: Waiting on Author

  7. RE: Make pgoutput documentation easier to find

    Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com> — 2025-08-14T12:12:44Z

    Dear Fujii-san,

> The current documentation for pgoutput is buried in the logical streaming
> replication protocol section (in protocol.sgml), and there's no index entry
> for it. This makes it hard to discover and access, for example, when trying
> to look up the options it supports.
>
> I've often struggled to locate this information myself, so I'd like to
> propose moving the pgoutput documentation to the logical decoding section
> and adding an index entry. The attached patch does that. I think this change
> will make it much easier for users to find the relevant details.

Personally considered pgoutput is not the user-facing and extension developers
can dig by themselves, but this is not a huge complaint.
DBA may check their settings via pg_recvlogical, which is a use-case.

I did agree to add new chapter for plugin developers, but it could be done at
different thread.

Since pgoutput can only generate the binary output, it pg_logical_slot_get_changes()
cannot be used with this plugin. I felt this should be also documented.

Another point: test_decoding does not have descriptions for their options neither.
Should we also add as well?

Best regards,
Hayato Kuroda
FUJITSU LIMITED

  8. Re: Make pgoutput documentation easier to find

    Fujii Masao <masao.fujii@gmail.com> — 2025-08-14T16:05:48Z

    On Thu, Aug 14, 2025 at 1:05 PM Chao Li <li.evan.chao@gmail.com> wrote:
>
> I applied the batch and built the html docs. I tested that all hyperlinks worked for me.

Thanks for the review and testing!


> +    <xref linkend="test-decoding"/>.  You can also write custom output plugins
>
> nit: "write" -> "develop"

OK, updated. I've attached the updated version of the patch (0001).


> +         Protocol version. Currently versions <literal>1</literal>, <literal>2</literal>,
>
> "Protocol version." -> "Specifies the protocol version."
>
> +         Comma-separated list of publication names for which to subscribe
> +         (receive changes). The individual publication names are treated
>
> A comma-separated list of publication names to subscribe to. The individual publication names are treated
>
> +         Boolean option to use binary transfer mode.  Binary mode is faster
>
> nit: "use" -> "enable", as other places all use "enable"

The 0001 patch mainly moves the pgoutput docs to a more visible location.
On the other hand, it looks like you're also suggesting further improvements
to the docs. I agree this is worth doing, but I think it should be a separate
patch. So I've created 0002, which updates the docs based on your comments
and also improves several other descriptions.


> +   The <literal>START_REPLICATION</literal> command allows us to
> +   pass options to the logical decoding output plugin associated with
> +   the replication slot specified in the command.
> +   See <xref linkend="logicaldecoding-pgoutput-options"/> for options
> +   that are accepted by the standard (<literal>pgoutput</literal>) plugin.
>
> nit: The START_REPLICATION command can pass options to the logical decoding output plugin associated with the specified replication slot. See Section 47.6.1.1 for the options supported by the standard (pgoutput) plugin.

OK, updated (0001 patch).

Regards,

-- 
Fujii Masao

  9. Re: Make pgoutput documentation easier to find

    Fujii Masao <masao.fujii@gmail.com> — 2025-08-14T16:10:58Z

    On Thu, Aug 14, 2025 at 9:12 PM Hayato Kuroda (Fujitsu)
<kuroda.hayato@fujitsu.com> wrote:
> Since pgoutput can only generate the binary output, it pg_logical_slot_get_changes()
> cannot be used with this plugin. I felt this should be also documented.

OK, I added this note in the 0002 patch attached earlier.


> Another point: test_decoding does not have descriptions for their options neither.
> Should we also add as well?

Yes, I think it's worth doing this (but it's better to do as a separate patch).
TBH this was already on my TODO list.

Regards,

-- 
Fujii Masao

  10. Re: Make pgoutput documentation easier to find

    Chao Li <li.evan.chao@gmail.com> — 2025-08-15T01:44:29Z

    The following review has been posted through the commitfest application:
make installcheck-world:  not tested
Implements feature:       not tested
Spec compliant:           not tested
Documentation:            tested, passed

Thanks for addressing my comments. I downloaded the new patch files and tested again. Now this patch looks good to me.

BTW, in the pushed commit, when you add me as a reviewer, would you please use my company email address: "Chao Li <lic@highgo.com>". Thanks a lot.

The new status of this patch is: Ready for Committer

  11. RE: Make pgoutput documentation easier to find

    Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com> — 2025-08-15T04:46:17Z

    Dear Fujii-san,

Thanks for updating the patch. Your patches look good to me.

Best regards,
Hayato Kuroda
FUJITSU LIMITED

  12. Re: Make pgoutput documentation easier to find

    Fujii Masao <masao.fujii@gmail.com> — 2025-08-18T02:55:02Z

    On Fri, Aug 15, 2025 at 1:46 PM Hayato Kuroda (Fujitsu)
<kuroda.hayato@fujitsu.com> wrote:
>
> Dear Fujii-san,
>
> Thanks for updating the patch. Your patches look good to me.

Thanks for the review! Unless there are objections, I'll commit these
two patches.

Regards,

-- 
Fujii Masao

  13. Re: Make pgoutput documentation easier to find

    Fujii Masao <masao.fujii@gmail.com> — 2025-08-19T09:57:47Z

    On Mon, Aug 18, 2025 at 11:55 AM Fujii Masao <masao.fujii@gmail.com> wrote:
>
> On Fri, Aug 15, 2025 at 1:46 PM Hayato Kuroda (Fujitsu)
> <kuroda.hayato@fujitsu.com> wrote:
> >
> > Dear Fujii-san,
> >
> > Thanks for updating the patch. Your patches look good to me.
>
> Thanks for the review! Unless there are objections, I'll commit these
> two patches.

I've pushed the patches. Thanks!

Regards,

-- 
Fujii Masao