Horton

Thread

Commits

  1. Fix WITH CHECK OPTION on views referencing postgres_fdw tables.

  2. Allow insert and update tuple routing and COPY for foreign tables.

  3. When WCOs are present, disable direct foreign table modification.

  1. Another oddity in handling of WCO constraints in postgres_fdw

    Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> — 2017-09-28T12:15:29Z

    Hi,

Commit 7086be6e3627c1ad797e32ebbdd232905b5f577f addressed mishandling of 
WCO in direct foreign table modification by disabling it when we have 
WCO, but I noticed another oddity in postgres_fdw:

postgres=# create table base_tbl (a int, b int);
postgres=# create function row_before_insupd_trigfunc() returns trigger 
as $$begin new.a := new.a + 10; return new; end$$ language plpgsql;
postgres=# create trigger row_before_insupd_trigger before insert or 
update on base_tbl for each row execute procedure 
row_before_insupd_trigfunc();
postgres=# create server loopback foreign data wrapper postgres_fdw 
options (dbname 'postgres');
postgres=# create user mapping for CURRENT_USER server loopback;
postgres=# create foreign table foreign_tbl (a int, b int) server 
loopback options (table_name 'base_tbl');
postgres=# create view rw_view as select * from foreign_tbl where a < b 
with check option;

So, this should fail, but

postgres=# insert into rw_view values (0, 5);
INSERT 0 1

The reason for that is: this is processed using postgres_fdw's 
non-direct foreign table modification (ie. ForeignModify), but unlike 
the RETURNING or local after trigger case, the ForeignModify doesn't 
take care that remote triggers might change the data in that case, so 
the WCO is evaluated using the data supplied, not the data actually 
inserted, which I think is wrong.  (I should have noticed that as well 
while working on the fix, though.)  So, I'd propose to fix that by 
modifying postgresPlanForeignModify so that it handles WCO the same way 
as for the RETURNING case.  Attached is a patch for that.  I'll add the 
patch to the next commitfest.

Best regards,
Etsuro Fujita

  2. Re: Another oddity in handling of WCO constraints in postgres_fdw

    Ashutosh Bapat <ashutosh.bapat@enterprisedb.com> — 2017-10-03T09:16:23Z

    On Thu, Sep 28, 2017 at 5:45 PM, Etsuro Fujita
<fujita.etsuro@lab.ntt.co.jp> wrote:
> Hi,
>
> Commit 7086be6e3627c1ad797e32ebbdd232905b5f577f addressed mishandling of WCO
> in direct foreign table modification by disabling it when we have WCO, but I
> noticed another oddity in postgres_fdw:
>
> postgres=# create table base_tbl (a int, b int);
> postgres=# create function row_before_insupd_trigfunc() returns trigger as
> $$begin new.a := new.a + 10; return new; end$$ language plpgsql;
> postgres=# create trigger row_before_insupd_trigger before insert or update
> on base_tbl for each row execute procedure row_before_insupd_trigfunc();
> postgres=# create server loopback foreign data wrapper postgres_fdw options
> (dbname 'postgres');
> postgres=# create user mapping for CURRENT_USER server loopback;
> postgres=# create foreign table foreign_tbl (a int, b int) server loopback
> options (table_name 'base_tbl');
> postgres=# create view rw_view as select * from foreign_tbl where a < b with
> check option;
>
> So, this should fail, but
>
> postgres=# insert into rw_view values (0, 5);
> INSERT 0 1
>
> The reason for that is: this is processed using postgres_fdw's non-direct
> foreign table modification (ie. ForeignModify), but unlike the RETURNING or
> local after trigger case, the ForeignModify doesn't take care that remote
> triggers might change the data in that case, so the WCO is evaluated using
> the data supplied, not the data actually inserted, which I think is wrong.
> (I should have noticed that as well while working on the fix, though.)  So,
> I'd propose to fix that by modifying postgresPlanForeignModify so that it
> handles WCO the same way as for the RETURNING case.  Attached is a patch for
> that.  I'll add the patch to the next commitfest.
>

Enforcing WCO constraints imposed by the local server on the row/DML
being passed to the foreign server is fine, but trying to impose them
on the row being inserted/updated at the foreign server looks odd. May
be we should just leave this case as it is. I am comparing this case
with the way we handle constraints on a foreign table.

-- 
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

  3. Re: Another oddity in handling of WCO constraints in postgres_fdw

    Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> — 2017-10-04T10:15:26Z

    On 2017/10/03 18:16, Ashutosh Bapat wrote:
> Enforcing WCO constraints imposed by the local server on the row/DML
> being passed to the foreign server is fine, but trying to impose them
> on the row being inserted/updated at the foreign server looks odd. May
> be we should just leave this case as it is. I am comparing this case
> with the way we handle constraints on a foreign table.

Hmm, I think that would be okay in the case where WCO constraints match 
constraints on the foreign table, but I'm not sure that would be okay 
even in the case where WCO constraints don't match?  Consider:

create table bt (a int check (a % 2 = 0));
create foreign table ft (a int check (a % 2 = 0)) server loopback 
options (table_name 'bt');
create view rw_view_2 as select * from ft where a % 2 = 0 with check option;

In that case the WCO constraint matches the constraint on the foreign 
table, so there would be no need to ensure the WCO constraint locally 
(to make the explanation simple, we assume here that we don't have 
triggers on the remote end).  BUT: for another auto-updatable view 
defined using the same foreign table like this:

create view rw_view_4 as select * from ft where a % 4 = 0 with check option;

how is the WCO constraint (ie, a % 4 = 0) ensured remotely, which is 
different from the constraint on the foreign table (ie, a % 2 = 0)? 
Maybe I'm missing something, though.

Best regards,
Etsuro Fujita

  4. Re: Another oddity in handling of WCO constraints in postgres_fdw

    Ashutosh Bapat <ashutosh.bapat@enterprisedb.com> — 2017-10-04T10:40:12Z

    On Wed, Oct 4, 2017 at 3:45 PM, Etsuro Fujita
<fujita.etsuro@lab.ntt.co.jp> wrote:
> On 2017/10/03 18:16, Ashutosh Bapat wrote:
>>
>> Enforcing WCO constraints imposed by the local server on the row/DML
>> being passed to the foreign server is fine, but trying to impose them
>> on the row being inserted/updated at the foreign server looks odd. May
>> be we should just leave this case as it is. I am comparing this case
>> with the way we handle constraints on a foreign table.
>
>
> Hmm, I think that would be okay in the case where WCO constraints match
> constraints on the foreign table, but I'm not sure that would be okay even
> in the case where WCO constraints don't match?  Consider:
>
> create table bt (a int check (a % 2 = 0));
> create foreign table ft (a int check (a % 2 = 0)) server loopback options
> (table_name 'bt');
> create view rw_view_2 as select * from ft where a % 2 = 0 with check option;
>
> In that case the WCO constraint matches the constraint on the foreign table,
> so there would be no need to ensure the WCO constraint locally (to make the
> explanation simple, we assume here that we don't have triggers on the remote
> end).  BUT: for another auto-updatable view defined using the same foreign
> table like this:
>
> create view rw_view_4 as select * from ft where a % 4 = 0 with check option;
>
> how is the WCO constraint (ie, a % 4 = 0) ensured remotely, which is
> different from the constraint on the foreign table (ie, a % 2 = 0)? Maybe
> I'm missing something, though.

Just like the local constraints on a foreign table are not ensured on
remote table (unless user takes steps to make that sure), WCO defined
locally need not be (and probably can not be) ensured remotely. We can
check whether a row being sent from the local server to the foreign
server obeys WCO, but what foreign server does to that row is beyond
local server's scope.

-- 
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

  5. Re: Another oddity in handling of WCO constraints in postgres_fdw

    Robert Haas <robertmhaas@gmail.com> — 2017-10-04T12:02:28Z

    On Wed, Oct 4, 2017 at 6:40 AM, Ashutosh Bapat
<ashutosh.bapat@enterprisedb.com> wrote:
> Just like the local constraints on a foreign table are not ensured on
> remote table (unless user takes steps to make that sure), WCO defined
> locally need not be (and probably can not be) ensured remotely. We can
> check whether a row being sent from the local server to the foreign
> server obeys WCO, but what foreign server does to that row is beyond
> local server's scope.

But I think right now we're not checking the row being sent from the
local server, either.  The WCO that is being ignored isn't a
constraint on the foreign table; it's a constraint on a view which
happens to reference the foreign table.  It seems quite odd for the
"assume constraints are valid" property of the foreign table to
propagate back up into the view that references it.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  6. Re: Another oddity in handling of WCO constraints in postgres_fdw

    Ashutosh Bapat <ashutosh.bapat@enterprisedb.com> — 2017-10-04T12:28:16Z

    On Wed, Oct 4, 2017 at 5:32 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> On Wed, Oct 4, 2017 at 6:40 AM, Ashutosh Bapat
> <ashutosh.bapat@enterprisedb.com> wrote:
>> Just like the local constraints on a foreign table are not ensured on
>> remote table (unless user takes steps to make that sure), WCO defined
>> locally need not be (and probably can not be) ensured remotely. We can
>> check whether a row being sent from the local server to the foreign
>> server obeys WCO, but what foreign server does to that row is beyond
>> local server's scope.
>
> But I think right now we're not checking the row being sent from the
> local server, either.

Didn't 7086be6e3627c1ad797e32ebbdd232905b5f577f fix that?

> The WCO that is being ignored isn't a
> constraint on the foreign table; it's a constraint on a view which
> happens to reference the foreign table.  It seems quite odd for the
> "assume constraints are valid" property of the foreign table to
> propagate back up into the view that references it.
>

The view with WCO is local but the modification which violates WCO is
being made on remote server by a trigger on remote table. Trying to
control that doesn't seem to be a good idea, just like we can't
control what rows get inserted on the foreign server when they violate
local constraints. I am using local constraints as an example of
precedence where we ignore what's happening on remote side and enforce
whatever we could enforce locally. Local server should make sure that
any rows sent from local server to the remote server do not violate
any local WCO. But once it's handed over to the foreign server, we
shouldn't worry about what happens there. That behaviour is ensured by
the above commit, isn't it?  I am not suggesting that we use local
constraints to enforce WCO or something like that.

-- 
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

  7. Re: Another oddity in handling of WCO constraints in postgres_fdw

    Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> — 2017-10-05T09:08:50Z

    On 2017/10/04 21:28, Ashutosh Bapat wrote:
> On Wed, Oct 4, 2017 at 5:32 PM, Robert Haas <robertmhaas@gmail.com> wrote:
>> On Wed, Oct 4, 2017 at 6:40 AM, Ashutosh Bapat
>> <ashutosh.bapat@enterprisedb.com> wrote:
>>> We can
>>> check whether a row being sent from the local server to the foreign
>>> server obeys WCO, but what foreign server does to that row is beyond
>>> local server's scope.
>>
>> But I think right now we're not checking the row being sent from the
>> local server, either.

We don't check the row *before* sending it to the remote server, but 
check the row returned by ExecForeignInsert/ExecForeignUpdate, which is 
allowed to have been changed by the remote server.  In postgres_fdw, we 
currently return the data actually inserted/updated if RETURNING/AFTER 
TRIGGER present, but not if WCO only presents.  So, for the postgres_fdw 
foreign table, WCO is enforced on the data that was actually 
inserted/updated if RETURNING/AFTER TRIGGER present and on the original 
data core supplied if WCO only presents, which is inconsistent behavior.

> Didn't 7086be6e3627c1ad797e32ebbdd232905b5f577f fix that?

No.  The commit addressed another issue.

>> The WCO that is being ignored isn't a
>> constraint on the foreign table; it's a constraint on a view which
>> happens to reference the foreign table.  It seems quite odd for the
>> "assume constraints are valid" property of the foreign table to
>> propagate back up into the view that references it.

Agreed.

> The view with WCO is local but the modification which violates WCO is
> being made on remote server by a trigger on remote table. Trying to
> control that doesn't seem to be a good idea, just like we can't
> control what rows get inserted on the foreign server when they violate
> local constraints. I am using local constraints as an example of
> precedence where we ignore what's happening on remote side and enforce
> whatever we could enforce locally. Local server should make sure that
> any rows sent from local server to the remote server do not violate
> any local WCO.

Seems odd (and too restrictive) to me too.

Best regards,
Etsuro Fujita

  8. Re: Another oddity in handling of WCO constraints in postgres_fdw

    Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> — 2017-10-05T11:06:31Z

    At Thu, 5 Oct 2017 18:08:50 +0900, Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> wrote in <60e94494-4e5d-afed-e482-b9ad1986bbf6@lab.ntt.co.jp>
> On 2017/10/04 21:28, Ashutosh Bapat wrote:
> > On Wed, Oct 4, 2017 at 5:32 PM, Robert Haas <robertmhaas@gmail.com>
> > wrote:
> >> On Wed, Oct 4, 2017 at 6:40 AM, Ashutosh Bapat
> >> <ashutosh.bapat@enterprisedb.com> wrote:
> >>> We can
> >>> check whether a row being sent from the local server to the foreign
> >>> server obeys WCO, but what foreign server does to that row is beyond
> >>> local server's scope.
> >>
> >> But I think right now we're not checking the row being sent from the
> >> local server, either.
> 
> We don't check the row *before* sending it to the remote server, but
> check the row returned by ExecForeignInsert/ExecForeignUpdate, which
> is allowed to have been changed by the remote server.  In
> postgres_fdw, we currently return the data actually inserted/updated
> if RETURNING/AFTER TRIGGER present, but not if WCO only presents.  So,
> for the postgres_fdw foreign table, WCO is enforced on the data that
> was actually inserted/updated if RETURNING/AFTER TRIGGER present and
> on the original data core supplied if WCO only presents, which is
> inconsistent behavior.
> 
> > Didn't 7086be6e3627c1ad797e32ebbdd232905b5f577f fix that?
> 
> No.  The commit addressed another issue.
> 
> >> The WCO that is being ignored isn't a
> >> constraint on the foreign table; it's a constraint on a view which
> >> happens to reference the foreign table.  It seems quite odd for the
> >> "assume constraints are valid" property of the foreign table to
> >> propagate back up into the view that references it.
> 
> Agreed.
> 
> > The view with WCO is local but the modification which violates WCO is
> > being made on remote server by a trigger on remote table. Trying to
> > control that doesn't seem to be a good idea, just like we can't
> > control what rows get inserted on the foreign server when they violate
> > local constraints. I am using local constraints as an example of
> > precedence where we ignore what's happening on remote side and enforce
> > whatever we could enforce locally. Local server should make sure that
> > any rows sent from local server to the remote server do not violate
> > any local WCO.
> 
> Seems odd (and too restrictive) to me too.

Since WCO ensures finally inserted values, we can't do other than
acturally requesting for the values. So just merging WCO columns
to RETURNING in deparsed query is ok. But can't we concatenate
returningList and withCheckOptionList at more higher level?
Specifically, just passing calculated used_attr to
deparse(Insert|Update)Sql instead of returningList and
withCheckOptionList separately.  Deparsed queries anyway forget
the origin of requested columns.

regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

  9. Re: Another oddity in handling of WCO constraints in postgres_fdw

    Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> — 2017-10-06T06:48:13Z

    On 2017/10/05 20:06, Kyotaro HORIGUCHI wrote:
> Since WCO ensures finally inserted values, we can't do other than
> acturally requesting for the values.

I think so too.

> So just merging WCO columns
> to RETURNING in deparsed query is ok. But can't we concatenate
> returningList and withCheckOptionList at more higher level?
> Specifically, just passing calculated used_attr to
> deparse(Insert|Update)Sql instead of returningList and
> withCheckOptionList separately.  Deparsed queries anyway forget
> the origin of requested columns.

We could do that, but I think that would need a bit more code to 
postgresPlanForeignModify including changes to the deparseDeleteSql API 
in addition to the deparse(Insert|Update)Sql APIs.  I prefer making high 
level functions simple, so I'd vote for just passing withCheckOptionList 
separately to deparse(Insert|Update)Sql, as proposed in the patch.

Best regards,
Etsuro Fujita

  10. Re: Another oddity in handling of WCO constraints in postgres_fdw

    Robert Haas <robertmhaas@gmail.com> — 2017-11-01T02:16:53Z

    On Wed, Oct 4, 2017 at 5:58 PM, Ashutosh Bapat
<ashutosh.bapat@enterprisedb.com> wrote:
> The view with WCO is local but the modification which violates WCO is
> being made on remote server by a trigger on remote table. Trying to
> control that doesn't seem to be a good idea, just like we can't
> control what rows get inserted on the foreign server when they violate
> local constraints.

I think that's a fair point.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  11. Re: Another oddity in handling of WCO constraints in postgres_fdw

    Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> — 2017-11-10T11:36:01Z

    (2017/11/01 11:16), Robert Haas wrote:
> On Wed, Oct 4, 2017 at 5:58 PM, Ashutosh Bapat
> <ashutosh.bapat@enterprisedb.com>  wrote:
>> The view with WCO is local but the modification which violates WCO is
>> being made on remote server by a trigger on remote table. Trying to
>> control that doesn't seem to be a good idea, just like we can't
>> control what rows get inserted on the foreign server when they violate
>> local constraints.
>
> I think that's a fair point.

For local constraints on foreign tables, it's the user's responsibility 
to ensure that those constraints matches the remote side, so we don't 
need to ensure those constraints locally.  But I'm not sure if the same 
thing applies to WCOs on views defined on foreign tables, because in 
some case it's not possible to impose constraints on the remote side 
that match those WCOs, as I explained before.

Best regards,
Etsuro Fujita

  12. Re: [HACKERS] Another oddity in handling of WCO constraints in postgres_fdw

    Michael Paquier <michael.paquier@gmail.com> — 2017-11-29T05:05:44Z

    On Fri, Nov 10, 2017 at 8:36 PM, Etsuro Fujita
<fujita.etsuro@lab.ntt.co.jp> wrote:
> For local constraints on foreign tables, it's the user's responsibility to
> ensure that those constraints matches the remote side, so we don't need to
> ensure those constraints locally.  But I'm not sure if the same thing
> applies to WCOs on views defined on foreign tables, because in some case
> it's not possible to impose constraints on the remote side that match those
> WCOs, as I explained before.

Moved to CF 2018-01.
-- 
Michael

  13. Re: [HACKERS] Another oddity in handling of WCO constraints in postgres_fdw

    Stephen Frost <sfrost@snowman.net> — 2018-01-17T13:00:21Z

    Greetings Etsuro, Robert, all,

* Etsuro Fujita (fujita.etsuro@lab.ntt.co.jp) wrote:
> (2017/11/01 11:16), Robert Haas wrote:
> >On Wed, Oct 4, 2017 at 5:58 PM, Ashutosh Bapat
> ><ashutosh.bapat@enterprisedb.com>  wrote:
> >>The view with WCO is local but the modification which violates WCO is
> >>being made on remote server by a trigger on remote table. Trying to
> >>control that doesn't seem to be a good idea, just like we can't
> >>control what rows get inserted on the foreign server when they violate
> >>local constraints.
> >
> >I think that's a fair point.
> 
> For local constraints on foreign tables, it's the user's responsibility to
> ensure that those constraints matches the remote side, so we don't need to
> ensure those constraints locally.  But I'm not sure if the same thing
> applies to WCOs on views defined on foreign tables, because in some case
> it's not possible to impose constraints on the remote side that match those
> WCOs, as I explained before.

Reviewing this thread, I tend to agree with Etsuro and I'm not sure I
see where there's a good argument for having a foreign table under a
view behave differently than a local table under a view for WCO (which
is an option of the view- not about the table underneath it or if it's
local or remote).  I've not done a detailed review of the patch but it
seems pretty reasonable and pretty small.

Thanks!

Stephen

  14. Re: [HACKERS] Another oddity in handling of WCO constraints in postgres_fdw

    Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> — 2018-01-18T07:16:35Z

    (2018/01/17 22:00), Stephen Frost wrote:
> Reviewing this thread, I tend to agree with Etsuro and I'm not sure I
> see where there's a good argument for having a foreign table under a
> view behave differently than a local table under a view for WCO (which
> is an option of the view- not about the table underneath it or if it's
> local or remote).  I've not done a detailed review of the patch but it
> seems pretty reasonable and pretty small.

Thanks for the comments!

I noticed the patch doesn't apply.  Attached is a rebased patch.

Best regards,
Etsuro Fujita

  15. Re: [HACKERS] Another oddity in handling of WCO constraints in postgres_fdw

    Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> — 2018-02-28T08:22:42Z

    (2018/01/18 16:16), Etsuro Fujita wrote:
> Attached is a rebased patch.

I rebased the patch over HEAD and revised comments/docs a little bit. 
Please find attached a new version of the patch.

Best regards,
Etsuro Fujita

  16. Re: [HACKERS] Another oddity in handling of WCO constraints in postgres_fdw

    Artur Zakirov <a.zakirov@postgrespro.ru> — 2018-03-03T09:51:10Z

    Hello,

On Wed, Feb 28, 2018 at 05:22:42PM +0900, Etsuro Fujita wrote:
> I rebased the patch over HEAD and revised comments/docs a little bit. Please
> find attached a new version of the patch.

I've reviewed the patch.

The code is good, clear and it is pretty small. There are documentation
fixes and additional regression tests.

Unfortunately the patch is outdated and it needs rebasing. Outdated
files are regression tests files.

After rebasing regression tests they pass.

-- 
Arthur Zakirov
Postgres Professional: http://www.postgrespro.com
Russian Postgres Company

  17. Re: [HACKERS] Another oddity in handling of WCO constraints in postgres_fdw

    Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> — 2018-03-05T12:44:37Z

    Hi Arthur,

(2018/03/03 18:51), Arthur Zakirov wrote:
> On Wed, Feb 28, 2018 at 05:22:42PM +0900, Etsuro Fujita wrote:
>> I rebased the patch over HEAD and revised comments/docs a little bit. Please
>> find attached a new version of the patch.
>
> I've reviewed the patch.
>
> The code is good, clear and it is pretty small. There are documentation
> fixes and additional regression tests.
>
> Unfortunately the patch is outdated and it needs rebasing. Outdated
> files are regression tests files.
>
> After rebasing regression tests they pass.

I rebased the patch over HEAD.  Please find attached an updated patch.

Thank you for the review!

Best regards,
Etsuro Fujita

  18. Re: [HACKERS] Another oddity in handling of WCO constraints in postgres_fdw

    Artur Zakirov <a.zakirov@postgrespro.ru> — 2018-03-05T16:57:09Z

    On Mon, Mar 05, 2018 at 09:44:37PM +0900, Etsuro Fujita wrote:
> I rebased the patch over HEAD.  Please find attached an updated patch.

Thank you!

IMHO, it is worth to add more explaining comment into
deparseReturningList, why it is necessary to merge WCO attributes to
RETURNING clause. You already noted it in the thread. I think it could
confuse someone who not very familiar how RETURNING is related with WITH
CHECK OPTION.

-- 
Arthur Zakirov
Postgres Professional: http://www.postgrespro.com
Russian Postgres Company

  19. Re: [HACKERS] Another oddity in handling of WCO constraints in postgres_fdw

    Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> — 2018-03-06T11:09:50Z

    (2018/03/06 1:57), Arthur Zakirov wrote:
> IMHO, it is worth to add more explaining comment into
> deparseReturningList, why it is necessary to merge WCO attributes to
> RETURNING clause. You already noted it in the thread. I think it could
> confuse someone who not very familiar how RETURNING is related with WITH
> CHECK OPTION.

Agreed.  I added a comment to that function.  I think that that comment 
in combination with changes to the FDW docs in the patch would help FDW 
authors understand why that is needed.  Please find attached an updated 
version of the patch.

Thanks for the comments!

Best regards,
Etsuro Fujita

  20. Re: [HACKERS] Another oddity in handling of WCO constraints in postgres_fdw

    Artur Zakirov <a.zakirov@postgrespro.ru> — 2018-03-06T14:00:25Z

    On Tue, Mar 06, 2018 at 08:09:50PM +0900, Etsuro Fujita wrote:
> Agreed.  I added a comment to that function.  I think that that comment in
> combination with changes to the FDW docs in the patch would help FDW authors
> understand why that is needed.  Please find attached an updated version of
> the patch.

Thank you.

All tests pass, the documentation builds. There was the suggestion [1]
of different approach. But the patch fix the issue in much more simple
way.

Marked as "Ready for Commiter".


1 - https://www.postgresql.org/message-id/20171005.200631.134118679.horiguchi.kyotaro%40lab.ntt.co.jp

-- 
Arthur Zakirov
Postgres Professional: http://www.postgrespro.com
Russian Postgres Company

  21. Re: [HACKERS] Another oddity in handling of WCO constraints in postgres_fdw

    Stephen Frost <sfrost@snowman.net> — 2018-03-07T03:25:39Z

    Greetings Robert, Ashutosh, Arthur, Etsuro, all,

* Arthur Zakirov (a.zakirov@postgrespro.ru) wrote:
> On Tue, Mar 06, 2018 at 08:09:50PM +0900, Etsuro Fujita wrote:
> > Agreed.  I added a comment to that function.  I think that that comment in
> > combination with changes to the FDW docs in the patch would help FDW authors
> > understand why that is needed.  Please find attached an updated version of
> > the patch.
> 
> Thank you.
> 
> All tests pass, the documentation builds. There was the suggestion [1]
> of different approach. But the patch fix the issue in much more simple
> way.
> 
> Marked as "Ready for Commiter".
>
> 1 - https://www.postgresql.org/message-id/20171005.200631.134118679.horiguchi.kyotaro%40lab.ntt.co.jp

Thanks,  I've looked through this patch and thread again and continue to
feel that this is both a good and sensible improvment and that the patch
is in pretty good shape.

The remaining question is if the subsequent discussion has swayed the
opinion of Robert and Ashutosh.  If we can get agreement that these
semantics are acceptable and an improvement over the status quo then I'm
happy to try and drive this patch to commit.

Robert, Ashutosh?

Thanks!

Stephen

  22. Re: [HACKERS] Another oddity in handling of WCO constraints in postgres_fdw

    Ashutosh Bapat <ashutosh.bapat@enterprisedb.com> — 2018-03-08T05:24:09Z

    On Wed, Mar 7, 2018 at 8:55 AM, Stephen Frost <sfrost@snowman.net> wrote:
> Greetings Robert, Ashutosh, Arthur, Etsuro, all,
>
> * Arthur Zakirov (a.zakirov@postgrespro.ru) wrote:
>> On Tue, Mar 06, 2018 at 08:09:50PM +0900, Etsuro Fujita wrote:
>> > Agreed.  I added a comment to that function.  I think that that comment in
>> > combination with changes to the FDW docs in the patch would help FDW authors
>> > understand why that is needed.  Please find attached an updated version of
>> > the patch.
>>
>> Thank you.
>>
>> All tests pass, the documentation builds. There was the suggestion [1]
>> of different approach. But the patch fix the issue in much more simple
>> way.
>>
>> Marked as "Ready for Commiter".
>>
>> 1 - https://www.postgresql.org/message-id/20171005.200631.134118679.horiguchi.kyotaro%40lab.ntt.co.jp
>
> Thanks,  I've looked through this patch and thread again and continue to
> feel that this is both a good and sensible improvment and that the patch
> is in pretty good shape.
>
> The remaining question is if the subsequent discussion has swayed the
> opinion of Robert and Ashutosh.  If we can get agreement that these
> semantics are acceptable and an improvement over the status quo then I'm
> happy to try and drive this patch to commit.
>
> Robert, Ashutosh?

If there is a local constraint on the foreign table, we don't check
it. So, a row that was inserted through this foreign table may not
show up when selected from the foreign table. Apply same logic to the
WCO on a view on the foreign table, it should be fine if a row
inserted through the view doesn't show up in the view. Somebody who
created the view, knew that it's a foreign table underneath.

Stephen said [1] that the view is local and irrespective of what's
underneath it, it should obey WCO. Which seems to be a fair point when
considered alone, but with the above context, it doesn't look any
fair.

Etsuro said [2] that WCO constraints can not be implemented on foreign
server and normal check constraints can be, and for that he provides
an example in [3]. But I think that example is going the wrong
direction. For local constraints to be enforced, we use remote
constraints. For local WCO we need to use remote WCO. That means we
create many foreign tables pointing to same local table on the foreign
server through many views, but it's not impossible.


[1] https://www.postgresql.org/message-id/20180117130021.GC2416%40tamriel.snowman.net
[2] https://www.postgresql.org/message-id/5A058F21.2040201%40lab.ntt.co.jp
[3] https://www.postgresql.org/message-id/a3955a1d-ad07-5b0a-7618-b6ef5ff0e1c5%40lab.ntt.co.jp

-- 
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

  23. Re: [HACKERS] Another oddity in handling of WCO constraints in postgres_fdw

    Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> — 2018-03-09T11:55:46Z

    Hi Ashutosh,

(2018/03/08 14:24), Ashutosh Bapat wrote:
> Etsuro said [2] that WCO constraints can not be implemented on foreign
> server and normal check constraints can be, and for that he provides
> an example in [3]. But I think that example is going the wrong
> direction.

More precisely, what I'm saying there is: for WCO constraints created by 
an auto-updatable view over a foreign table, we cannot always implement 
constraints on the remote side that match with those WCO constraints.

> For local constraints to be enforced, we use remote
> constraints. For local WCO we need to use remote WCO. That means we
> create many foreign tables pointing to same local table on the foreign
> server through many views, but it's not impossible.

Maybe I don't understand this correctly, but I guess that it would be 
the user's responsibility to not create foreign tables in such a way.

Best regards,
Etsuro Fujita

  24. Re: [HACKERS] Another oddity in handling of WCO constraints in postgres_fdw

    Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> — 2018-03-12T07:55:03Z

    (2018/03/09 20:55), Etsuro Fujita wrote:
> (2018/03/08 14:24), Ashutosh Bapat wrote:
>> For local constraints to be enforced, we use remote
>> constraints. For local WCO we need to use remote WCO. That means we
>> create many foreign tables pointing to same local table on the foreign
>> server through many views, but it's not impossible.
>
> Maybe I don't understand this correctly, but I guess that it would be
> the user's responsibility to not create foreign tables in such a way.

I think I misunderstood your words.  Sorry for that.  I think what you 
proposed would be a solution for this issue, but I'm not sure that's a 
good one because that wouldn't work for the data sources that don't 
support views with WCO options.

Best regards,
Etsuro Fujita

  25. Re: [HACKERS] Another oddity in handling of WCO constraints in postgres_fdw

    Ashutosh Bapat <ashutosh.bapat@enterprisedb.com> — 2018-03-13T04:11:15Z

    On Mon, Mar 12, 2018 at 1:25 PM, Etsuro Fujita
<fujita.etsuro@lab.ntt.co.jp> wrote:
> (2018/03/09 20:55), Etsuro Fujita wrote:
>>
>> (2018/03/08 14:24), Ashutosh Bapat wrote:
>>>
>>> For local constraints to be enforced, we use remote
>>> constraints. For local WCO we need to use remote WCO. That means we
>>> create many foreign tables pointing to same local table on the foreign
>>> server through many views, but it's not impossible.
>>
>>
>> Maybe I don't understand this correctly, but I guess that it would be
>> the user's responsibility to not create foreign tables in such a way.
>
>
> I think I misunderstood your words.  Sorry for that.  I think what you
> proposed would be a solution for this issue, but I'm not sure that's a good
> one because that wouldn't work for the data sources that don't support views
> with WCO options.

Our solution for the constraints doesn't work with the data sources
(like flat files) which don't support constraints. So, that argument
doesn't help.

-- 
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

  26. Re: [HACKERS] Another oddity in handling of WCO constraints in postgres_fdw

    Stephen Frost <sfrost@snowman.net> — 2018-03-13T15:04:09Z

    Greetings,

* Ashutosh Bapat (ashutosh.bapat@enterprisedb.com) wrote:
> On Mon, Mar 12, 2018 at 1:25 PM, Etsuro Fujita
> <fujita.etsuro@lab.ntt.co.jp> wrote:
> > (2018/03/09 20:55), Etsuro Fujita wrote:
> >> (2018/03/08 14:24), Ashutosh Bapat wrote:
> >>> For local constraints to be enforced, we use remote
> >>> constraints. For local WCO we need to use remote WCO. That means we
> >>> create many foreign tables pointing to same local table on the foreign
> >>> server through many views, but it's not impossible.
> >>
> >> Maybe I don't understand this correctly, but I guess that it would be
> >> the user's responsibility to not create foreign tables in such a way.
> >
> > I think I misunderstood your words.  Sorry for that.  I think what you
> > proposed would be a solution for this issue, but I'm not sure that's a good
> > one because that wouldn't work for the data sources that don't support views
> > with WCO options.
> 
> Our solution for the constraints doesn't work with the data sources
> (like flat files) which don't support constraints. So, that argument
> doesn't help.

It would really help to have some examples of exactly what is being
proposed here wrt solutions.

WCO is defined at a view level, so I'm not following the notion that
we're going to depend on something remote to enforce the WCO when the
remote object is just a regular table that you can't define a WCO on top
of.  That's not the case when we're talking about foreign tables vs.
local tables, so it's not the same.  I certainly don't think we should
require a remote view to exist to perform the WCO check.  If the remote
WCO is a view itself then I would expect it to operate in the same
manner as WCO on local views does- you can have them defined as being
cascaded or not.

In other words, there is no case where we have a "foreign view."  Views
are always local.  A "foreign table" could actually be a view, in which
case everything we treat it as a table in the local database, but WCO
doesn't come up in that case at all- there's no way to define WCO on a
table, foreign or not.  If WCO is defined on the view on the remote
server, then it should operate properly and not require anything from the
local side.

Thanks!

Stephen

  27. Re: [HACKERS] Another oddity in handling of WCO constraints in postgres_fdw

    Ashutosh Bapat <ashutosh.bapat@enterprisedb.com> — 2018-03-16T11:01:45Z

    On Tue, Mar 13, 2018 at 8:34 PM, Stephen Frost <sfrost@snowman.net> wrote:
>
> It would really help to have some examples of exactly what is being
> proposed here wrt solutions.
>
> WCO is defined at a view level, so I'm not following the notion that
> we're going to depend on something remote to enforce the WCO when the
> remote object is just a regular table that you can't define a WCO on top
> of.  That's not the case when we're talking about foreign tables vs.
> local tables, so it's not the same.  I certainly don't think we should
> require a remote view to exist to perform the WCO check.  If the remote
> WCO is a view itself then I would expect it to operate in the same
> manner as WCO on local views does- you can have them defined as being
> cascaded or not.
>
> In other words, there is no case where we have a "foreign view."  Views
> are always local.  A "foreign table" could actually be a view, in which
> case everything we treat it as a table in the local database, but WCO
> doesn't come up in that case at all- there's no way to define WCO on a
> table, foreign or not.  If WCO is defined on the view on the remote
> server, then it should operate properly and not require anything from the
> local side.

I agree with this analysis. I have no objection about the patch anymore.

-- 
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company

  28. Re: [HACKERS] Another oddity in handling of WCO constraints in postgres_fdw

    Jeff Davis <pgsql@j-davis.com> — 2018-07-09T00:00:18Z

    On Tue, 2018-03-06 at 20:09 +0900, Etsuro Fujita wrote:
> Agreed.  I added a comment to that function.  I think that that
> comment 
> in combination with changes to the FDW docs in the patch would help
> FDW 
> authors understand why that is needed.  Please find attached an
> updated 
> version of the patch.
> 
> Thanks for the comments!

Committed.

I made some small modifications and added a test for the case where the
foreign table is a partition of a local table, which follows a
different code path after commit 3d956d95.

Thank you!

Regards,
	Jeff Davis

  29. Re: [HACKERS] Another oddity in handling of WCO constraints in postgres_fdw

    Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> — 2018-07-09T11:05:25Z

    (2018/07/09 9:00), Jeff Davis wrote:
> Committed.
>
> I made some small modifications and added a test for the case where the
> foreign table is a partition of a local table, which follows a
> different code path after commit 3d956d95.

Great!  Thanks for revising and committing, Jeff.  Thanks for reviewing, 
Arthur and Stephen.  Thanks for commenting on this, Ashutosh and Robert.

Best regards,
Etsuro Fujita