Re: Binary support for pgoutput plugin

On Sat, Jun 08, 2019 at 08:40:43PM -0400, Dave Cramer wrote:
>On Sat, 8 Jun 2019 at 20:09, Andres Freund <andres@anarazel.de> wrote:
>
>> Hi,
>>
>> On 2019-06-08 19:41:34 -0400, Dave Cramer wrote:
>> > So the reason we are discussing using pgoutput plugin is because it is
>> part
>> > of core and guaranteed to be in cloud providers solutions.
>>
>> IMO people needing this should then band together and write one that's
>> suitable, rather than trying to coerce test_decoding and now pgoutput
>> into something they're not made for.
>>
>
>At the moment it would look a lot like pgoutput. For now I'm fine with no
>changes to pgoutput other than binary
>Once we have some real use cases we can look at writing a new one. I would
>imagine we would actually start with
>pgoutput and just add to it.
>

I understand the desire to make this work for managed cloud environments,
we support quite a few customers who would benefit from it. But pgoutput
is meant specifically for built-in replication, and adding complexity that
is useless for that use case does not seem like a good tradeoff.

From this POV the binary mode is fine, because it'd benefit pgoutput, but
the various other stuff mentioned here (e.g. nullability) is not.

And if we implement a new plugin for use by out-of-core stuff, I guess
we'd probably done it in an extension. But even having it in contrib would
not make it automatically installed on managed systems, because AFAIK the
various providers only allow whitelisted extensions. At which point
there's there's little difference compared to external extensions.

I think the best party to implement such extension is whoever implements
such replication system (say Debezium), because they are the ones who know
which format / behavior would work for them. And they can also show the
benefit to their users, who can then push the cloud providers to install
the extension. Of course, that'll take a long time (but it's unclear how
long), and until then they'll have to provide some fallback.

This is a bit of a chicken-egg problem, with three parties - our project,
projects building on logical replication and cloud providers. And no
matter how you slice it, the party implementing it has only limited (if
any) control over what the other parties allow.

regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services