Horton

RE: Allow logical replication to copy tables in binary format

Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>

View thread
From: "Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com>
To: 'Jelte Fennema' <postgres@jeltef.nl>
Cc: Melih Mutlu <m.melihmutlu@gmail.com>, "Takamichi Osumi (Fujitsu)" <osumi.takamichi@fujitsu.com>, "shiy.fnst@fujitsu.com" <shiy.fnst@fujitsu.com>, Euler Taveira <euler@eulerto.com>, Amit Kapila <amit.kapila16@gmail.com>, pgsql-hackers <pgsql-hackers@postgresql.org>, Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Date: 2023-02-23T04:40:01Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Allow logical replication to copy tables in binary format.

  2. Common function for percent placeholder replacement

  3. Doc: add XML ID attributes to <sectN> and <varlistentry> tags.

  4. Re-order disable_on_error in tab-complete.

  5. Make subscription tests pass with log_error_verbosity=verbose

  6. Weaken type-OID-matching checks in array_recv and record_recv.

  7. Allow logical replication to transfer data in binary format.

Dear Jelte,

> I don't think it's necessary to check versions. Yes, there are
> situations where binary will fail across major versions. But in many
> cases it does not. To me it seems the responsibility of the operator
> to evaluate this risk. And if the operator chooses wrong and uses
> binary copy across incompatible versions, then it will still fail hard
> in that case during the copy phase (so still a very early error). So I
> don't see a reason to check pre-emptively, afaict it will only
> disallow some valid usecases and introduce more code.
> 
> Furthermore no major version check is done for "binary = true" either
> (afaik). The only additional failure scenario that copy_format=binary
> introduces is when one of the types does not implement a send function
> on the source. With binary=true, this would continue to work, but with
> copy_format=binary this stops working. All other failure scenarios
> that binary encoding of types introduces apply to both binary=true and
> copy_format=binary (the only difference being in which phase of the
> replication these failures happen, the apply or the copy phase).

I thought that current specification was lack of consideration, but you meant to
say that it is intentional one to keep the availability, right? 
Indeed my suggestion seems to be too pessimistic, but I want to listen to other
opinions more...

> > I'm not sure the combination of "copy_format = binary" and "copy_data = false"
> > should be accepted or not. How do you think?
> 
> It seems quite useless indeed to specify the format of a copy that won't happen.

I understood that the conbination of "copy_format = binary" and "copy_data = false"
should be rejected in parse_subscription_options() and AlterSubscription(). Is it right?
I'm expecting that is done in next version.

Best Regards,
Hayato Kuroda
FUJITSU LIMITED