Horton

Re: Allow logical replication to copy tables in binary format

Melih Mutlu <m.melihmutlu@gmail.com>

View thread
From: Melih Mutlu <m.melihmutlu@gmail.com>
To: "Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com>
Cc: "shiy.fnst@fujitsu.com" <shiy.fnst@fujitsu.com>, "Takamichi Osumi (Fujitsu)" <osumi.takamichi@fujitsu.com>, Euler Taveira <euler@eulerto.com>, pgsql-hackers <pgsql-hackers@postgresql.org>, Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>, Amit Kapila <amit.kapila16@gmail.com>, Dilip Kumar <dilipbalaut@gmail.com>
Date: 2023-03-01T18:09:44Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Allow logical replication to copy tables in binary format.

  2. Common function for percent placeholder replacement

  3. Doc: add XML ID attributes to <sectN> and <varlistentry> tags.

  4. Re-order disable_on_error in tab-complete.

  5. Make subscription tests pass with log_error_verbosity=verbose

  6. Weaken type-OID-matching checks in array_recv and record_recv.

  7. Allow logical replication to transfer data in binary format.

Hi,

Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>, 1 Mar 2023 Çar, 18:40
tarihinde şunu yazdı:

> Dear Melih,
>
> If we do not have to treat the case Shi pointed out[1] as code-level, I
> agreed to
> same option binary because it is simpler.


How is this an issue if we let the binary option do binary copy and not an
issue if we have a separate copy_binary option?
You can easily have the similar errors when you set copy_binary=true if a
type is missing binary send/receive functions.
And also, as Amit mentioned, the same issue can easily be avoided if
binary=false until the initial sync is done. It can be set to true later.


> I read the use-cases addressed by Bharath[2]
> and I cannot find advantage for case 1 and 3 expect the case that binary
> functions
> are not implemented.
>

Note that case 3 is already how it works on HEAD. Its advantages, as you
already mentioned, is when some types are missing the binary functions.
I think that's why case 3 should be allowed even if a new option is added
or not.

Previously I said that the combination of "copy_format = binary" and
> "copy_data = false"
> seemed strange[3], but this approach could solve it and other related ones
> automatically.
>

I think it is quite similar to the case where binary=true and
enabled=false. In that case, the format is set but the subscription does
not replicate anything. And this is allowed.
copy_binary=true and copy_data=false combination also sets the copy format
but does not copy anything. Even if any table will not be copied at that
moment, tables which might be added later might need to be copied (by ALTER
SUBSCRIPTION). And setting the copy format beforehand can be useful in such
cases.


> I think we should add description to doc that it is more likely happen to
> fail
> the initial copy user should enable binary format after synchronization if
> tables have original datatype.
>

I tried to explain when binary copy can cause failures in the doc. What
exactly do you think is missing?

Best,
-- 
Melih Mutlu
Microsoft