RE: Allow logical replication to copy tables in binary format

Yu Shi (Fujitsu) <shiy.fnst@fujitsu.com>

From: "shiy.fnst@fujitsu.com" <shiy.fnst@fujitsu.com>

To: Melih Mutlu <m.melihmutlu@gmail.com>, "Takamichi Osumi (Fujitsu)" <osumi.takamichi@fujitsu.com>

Cc: Euler Taveira <euler@eulerto.com>, Amit Kapila <amit.kapila16@gmail.com>, pgsql-hackers <pgsql-hackers@postgresql.org>

Date: 2023-01-11T08:56:28Z

Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

Allow logical replication to copy tables in binary format.
- ecb696527c01 16.0 landed
Common function for percent placeholder replacement
- c96de2ce1782 16.0 cited
Doc: add XML ID attributes to <sectN> and <varlistentry> tags.
- 78ee60ed84bb 16.0 cited
Re-order disable_on_error in tab-complete.
- d547f7cf5efc 16.0 cited
Make subscription tests pass with log_error_verbosity=verbose
- 19408aae7fa2 15.0 cited
Weaken type-OID-matching checks in array_recv and record_recv.
- 670c0a1d474b 14.0 cited
Allow logical replication to transfer data in binary format.
- 9de77b545313 14.0 cited

On Mon, Nov 14, 2022 8:08 PM Melih Mutlu <m.melihmutlu@gmail.com> wrote:
>
> Attached patch with updated version of this patch.

Thanks for your patch.

I tried to do a performance test for this patch, the result looks good to me.
(The steps are similar to what Melih shared [1].)

The time to synchronize about 1GB data in binary (the average of the middle
eight was taken):
HEAD: 16.854 s
Patched: 6.805 s

Besides, here are some comments.

1.
+# Binary enabled subscription should fail
+$node_subscriber_binary->wait_for_log("ERROR:  insufficient data left in message");

Should it be changed to "ERROR: ( [A-Z0-9]+:)? ", like other subscription tests.

2.
+# Binary disabled subscription should succeed
+$node_publisher->wait_for_catchup('tap_sub');

If we want to wait for table synchronization to finish, should we call
wait_for_subscription_sync()?

3.
I also think it might be better to support copy binary only for publishers of
v16 or later. Do you plan to implement it in the patch?

[1] https://www.postgresql.org/message-id/CAGPVpCQEKDVKQPf6OFQ-9WiRYB1YRejm--YJTuwgzuvj1LEJ2A%40mail.gmail.com

Regards,
Shi yu