Binary support for pgoutput plugin

This should have gone to hackers as well

---------- Forwarded message ---------
From: Dave Cramer <davecramer@gmail.com>
Date: Sat, Jun 8, 2019, 6:41 PM
Subject: Re: Binary support for pgoutput plugin
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>




On Sat, Jun 8, 2019, 6:27 PM Tomas Vondra, <tomas.vondra@2ndquadrant.com>
wrote:

> On Fri, Jun 07, 2019 at 06:01:12PM -0700, Andres Freund wrote:
> >Hi,
> >
> >On 2019-06-07 20:52:38 -0400, Chapman Flack wrote:
> >> It seems they had ended up designing a whole 'nother "protocol level"
> >> involving queries wrapping their results as JSON and an app layer that
> >> unwraps again, after trying a simpler first approach that was foiled by
> the
> >> inability to see into arrays and anonymous record types in the
> 'describe'
> >> response.
> >
> >I suspect quite a few people would have to have left the projectbefore
> >this would happen.
> >
> >
> >> I thought, in a new protocol rev, why not let the driver send additional
> >> 'describe' messages after the first one, to drill into structure of
> >> individual columns mentioned in the first response, before sending the
> >> 'execute' message?
> >>
> >> If it doesn't want the further detail, it doesn't have to ask.
> >>
> >> > And then we suddenly need tracking for all these, so we don't always
> >> > send out that information when we previously already did
> >>
> >> If it's up to the client driver, it can track what it needs or already
> has.
> >
> >> I haven't looked too deeply into the replication protocol ... it happens
> >> under a kind of copy-both, right?, so maybe there's a way for the
> receiver
> >> to send some inquiries back, but maybe in a windowed, full-duplex way
> where
> >> it might have to buffer some incoming messages before getting the
> response
> >> to an inquiry message it sent.
> >
> >That'd be a *lot* of additional complexity, and pretty much prohibitive
> >from a performance POV. We'd have to not continue decoding on the server
> >side *all* the time to give the client a chance to inquire additional
> >information.
> >
>
> I kinda agree with this, and I think it's an argument why replication
> solutions that need such additional metadata (e.g. because they have no
> database to query) should not rely on pgoutput but should invent their own
> decoding plugin. Which is why it's a plugin.
>

So the reason we are discussing using pgoutput plugin is because it is part
of core and guaranteed to be in cloud providers solutions. I'm trying to
find a balance here of using what we have as opposed to burdening core to
take on additional code to take care of. Not sending the metadata is not a
deal breaker but i can see some value in it.


Dave

>
>