Horton

Re: Allow logical replication to copy tables in binary format

Melih Mutlu <m.melihmutlu@gmail.com>

View thread
From: Melih Mutlu <m.melihmutlu@gmail.com>
To: vignesh C <vignesh21@gmail.com>, Amit Kapila <amit.kapila16@gmail.com>, Peter Smith <smithpb2250@gmail.com>, "Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com>
Cc: "shiy.fnst@fujitsu.com" <shiy.fnst@fujitsu.com>, Euler Taveira <euler@eulerto.com>, Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>, Dilip Kumar <dilipbalaut@gmail.com>, "osumi.takamichi@fujitsu.com" <osumi.takamichi@fujitsu.com>, pgsql-hackers <pgsql-hackers@postgresql.org>
Date: 2023-03-20T10:58:59Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Allow logical replication to copy tables in binary format.

  2. Common function for percent placeholder replacement

  3. Doc: add XML ID attributes to <sectN> and <varlistentry> tags.

  4. Re-order disable_on_error in tab-complete.

  5. Make subscription tests pass with log_error_verbosity=verbose

  6. Weaken type-OID-matching checks in array_recv and record_recv.

  7. Allow logical replication to transfer data in binary format.

Attachments

Hi,

Please see the attached patch.

vignesh C <vignesh21@gmail.com>, 18 Mar 2023 Cmt, 12:03 tarihinde şunu
yazdı:

> On Fri, 17 Mar 2023 at 17:55, Melih Mutlu <m.melihmutlu@gmail.com> wrote:
> 1) Currently we refer the link to the beginning of create subscription
> page, this can be changed to refer to binary option contents in create
> subscription:
>

Done.


> 2) Running pgperltidy shows the test script 014_binary.pl could be
> slightly improved as in the attachment.
>

Couldn't apply the patch you attached but I ran pgperltidy myself and I
guess the result should be similar.


Peter Smith <smithpb2250@gmail.com>, 18 Mar 2023 Cmt, 12:41 tarihinde şunu
yazdı:

> Commit message
>
> 1.
> Binary copy is supported for v16 or later.
>

Done as you suggested.

Amit Kapila <amit.kapila16@gmail.com>, 18 Mar 2023 Cmt, 13:11 tarihinde
şunu yazdı:

> On Sat, Mar 18, 2023 at 3:11 PM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > SUGGESTION
> > If the publisher is v16 or later, then any initial table
> > synchronization will use the same format as specified by the
> > subscription binary mode. If the publisher is before v16, then any
> > initial table synchronization will use text format regardless of the
> > subscription binary mode.
> >
>
> I agree that the previous comment should be updated but I would prefer
> something along the lines: "Prior to v16, initial table
> synchronization will use text format even if the binary option is
> enabled for a subscription."
>

Done.

Amit Kapila <amit.kapila16@gmail.com>, 20 Mar 2023 Pzt, 05:13 tarihinde
şunu yazdı:

> On Mon, Mar 20, 2023 at 3:37 AM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > There are a couple of TAP tests where the copy binary is expected to
> > fail. And when it fails, you do binary=false (changing the format back
> > to 'text') so the test is then expected to be able to proceed.
> >
> > I don't know if this happens in practice, but IIUC in theory, if the
> > timing is extremely bad, the tablesync could relaunch in binary mode
> > multiple times (any fail multiple times?) before your binary=false
> > change takes effect.
> >
> > So, I was wondering if it could help to use the subscription
> > 'disable_on_error=true' parameter for those cases so that the
> > tablesync won't needlessly attempt to relaunch until you have set
> > binary=false and then re-enabled the subscription.
> >
>
> +1. That would make tests more reliable.
>

Done.

Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>, 20 Mar 2023 Pzt, 07:13
tarihinde şunu yazdı:

> Dear Melih,
>
> Thank you for updating the patch.
> I checked your added description about initial data sync and I think it's
> OK.
>
> Few minor comments:
>

Fixed both comments.

Thanks,
-- 
Melih Mutlu
Microsoft