Horton

RE: Allow logical replication to copy tables in binary format

Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>

View thread
From: "Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com>
To: 'Melih Mutlu' <m.melihmutlu@gmail.com>
Cc: "shiy.fnst@fujitsu.com" <shiy.fnst@fujitsu.com>, "Takamichi Osumi (Fujitsu)" <osumi.takamichi@fujitsu.com>, Euler Taveira <euler@eulerto.com>, pgsql-hackers <pgsql-hackers@postgresql.org>, Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>, Amit Kapila <amit.kapila16@gmail.com>, Dilip Kumar <dilipbalaut@gmail.com>
Date: 2023-03-08T05:24:55Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Allow logical replication to copy tables in binary format.

  2. Common function for percent placeholder replacement

  3. Doc: add XML ID attributes to <sectN> and <varlistentry> tags.

  4. Re-order disable_on_error in tab-complete.

  5. Make subscription tests pass with log_error_verbosity=verbose

  6. Weaken type-OID-matching checks in array_recv and record_recv.

  7. Allow logical replication to transfer data in binary format.

Dear Melih,

>> I think we should add description to doc that it is more likely happen to fail
>> the initial copy user should enable binary format after synchronization if
>> tables have original datatype.
>
> I tried to explain when binary copy can cause failures in the doc. What exactly
> do you think is missing?

I assumed here that "copy_format" and "binary" were combined into one option.
Currently the binary option has descriptions :

```
Even when this option is enabled, only data types having binary send and receive functions will be transferred in binary
```

But this is not suitable for initial data sync, as we knew. I meant to say that
it must be updated if options are combined.

Note that following is not necessary for PG16, just an improvement for newer version.

Is it possible to automatically switch the binary option from 'true' to 'false'
when data transfer fails? As we found that while synchronizing the initial data
with binary format may lead another error, and it can be solved if the options
is changed. When DBAs check a log after synchronization and find the output like
"binary option was changed and worker will restart..." or something, they can turn
"binary" on again.

Best Regards,
Hayato Kuroda
FUJITSU LIMITED