Horton

Thread

Commits

  1. Add support for MERGE SQL command

  2. Add API of sorts for transition table handling in trigger.c

  3. Revert MERGE patch

  4. Fix several bugs related to ON CONFLICT's EXCLUDED pseudo relation.

  1. MERGE SQL Statement for PG11

    Simon Riggs <simon@2ndquadrant.com> — 2017-10-27T08:55:05Z

    I'm working on re-submitting MERGE for PG11

Earlier thoughts on how this could/could not be done were sometimes
imprecise or inaccurate, so I have gone through the command per
SQL:2011 spec and produced a definitive spec in the form of an SGML
ref page. This is what I intend to deliver for PG11.

MERGE will use the same mechanisms as INSERT ON CONFLICT, so
concurrent behavior does not require further infrastructure changes,
just detailed work on the statement itself.

I'm building up the code from scratch based upon the spec, rather than
trying to thwack earlier attempts into shape. This looks more likely
to yield a commitable patch.

Major spanners or objections, please throw them in now cos I don't see any.

Questions?

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  2. Re: MERGE SQL Statement for PG11

    Szymon Guz <mabewlun@gmail.com> — 2017-10-27T09:33:45Z

    Hey,
It looks quite nice. Personally I'd like to also have the returning
statement, and have the number of deleted and inserted rows as separate
numbers in the output message.

regards
Szymon Lipiński

pt., 27.10.2017, 10:56 użytkownik Simon Riggs <simon@2ndquadrant.com>
napisał:

> I'm working on re-submitting MERGE for PG11
>
> Earlier thoughts on how this could/could not be done were sometimes
> imprecise or inaccurate, so I have gone through the command per
> SQL:2011 spec and produced a definitive spec in the form of an SGML
> ref page. This is what I intend to deliver for PG11.
>
> MERGE will use the same mechanisms as INSERT ON CONFLICT, so
> concurrent behavior does not require further infrastructure changes,
> just detailed work on the statement itself.
>
> I'm building up the code from scratch based upon the spec, rather than
> trying to thwack earlier attempts into shape. This looks more likely
> to yield a commitable patch.
>
> Major spanners or objections, please throw them in now cos I don't see any.
>
> Questions?
>
> --
> Simon Riggs                http://www.2ndQuadrant.com/
> PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers
>

  3. Re: MERGE SQL Statement for PG11

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2017-10-27T12:45:55Z

    Simon Riggs wrote:

> Earlier thoughts on how this could/could not be done were sometimes
> imprecise or inaccurate, so I have gone through the command per
> SQL:2011 spec and produced a definitive spec in the form of an SGML
> ref page. This is what I intend to deliver for PG11.

Nice work.  I didn't verify the SQL spec, just read your HTML page;
some very minor comments based on that:

* use "and" not "where" as initial words in "when_clause" and
  "merge_update" clause definitions

* missing word here: "the DELETE privilege on the if you specify"

* I think the word "match." is leftover from some editing in the phrase
  " that specifies which rows in the data_source match rows in the
  target_table_name. match."  In the same paragraph, it is not clear
  whether all columns must be matched or it can be a partial match.

* In the when_clause note, it is not clear whether you can have multiple
  WHEN MATCHED and WHEN NOT MATCHED clauses.  Obviously you can have one
  of each, but I think your doc says it is possible to have more than one of
  each, with different conditions (WHEN MATCHED AND foo THEN bar WHEN
  MATCHED AND baz THEN qux).  No example shows more than one.

  On the same point: Is there short-circuiting of such conditions, i.e.
  will the execution will stop looking for further WHEN matches if some
  rule matches, or will it rather check all rules and raise an error if
  more than one WHEN rules match each given row?

* Your last example uses ELSE but that appears nowhere in the synopsys.

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  4. Re: MERGE SQL Statement for PG11

    Robert Haas <robertmhaas@gmail.com> — 2017-10-27T13:24:12Z

    On Fri, Oct 27, 2017 at 10:55 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
> Questions?

I think one of the reasons why Peter Geoghegan decided to pursue
INSERT .. ON CONFLICT UPDATE was that, because it is non-standard SQL
syntax, he felt free to mandate a non-standard SQL requirement, namely
the presence of a unique index on the arbiter columns.  If MERGE's
join clause happens to involve equality conditions on precisely the
same set of columns as some unique index on the target table, then I
think you can reuse the INSERT .. ON CONFLICT UPDATE infrastructure
and I suspect there won't be too many problems.  However, if it
doesn't, then what?  You could decree that such cases will fail, but
that might not meet your use case for developing the feature.  Or you
could try to soldier on without the INSERT .. ON CONFLICT UPDATE
machinery, but that means, I think, that sometimes you will get
serialization anomalies - at least, I think, you will sometimes obtain
results that couldn't have been obtained under any serial order of
execution, and maybe it would end up being possible to fail with
serialization errors or unique index violations.

In the past, there have been objections to implementations of MERGE
which would give rise to such serialization anomalies, but I'm not
sure we should feel bound by those discussions.  One thing that's
different is that the common and actually-useful case can now be made
to work in a fairly satisfying way using INSERT .. ON CONFLICT UPDATE;
if less useful cases are vulnerable to some weirdness, maybe it's OK
to just document the problems.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  5. Re: MERGE SQL Statement for PG11

    Simon Riggs <simon@2ndquadrant.com> — 2017-10-27T14:41:29Z

    On 27 October 2017 at 15:24, Robert Haas <robertmhaas@gmail.com> wrote:
> On Fri, Oct 27, 2017 at 10:55 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
>> Questions?
>
> I think one of the reasons why Peter Geoghegan decided to pursue
> INSERT .. ON CONFLICT UPDATE was that, because it is non-standard SQL
> syntax, he felt free to mandate a non-standard SQL requirement, namely
> the presence of a unique index on the arbiter columns.  If MERGE's
> join clause happens to involve equality conditions on precisely the
> same set of columns as some unique index on the target table, then I
> think you can reuse the INSERT .. ON CONFLICT UPDATE infrastructure
> and I suspect there won't be too many problems.

Agreed

> However, if it
> doesn't, then what?  You could decree that such cases will fail, but
> that might not meet your use case for developing the feature.  Or you
> could try to soldier on without the INSERT .. ON CONFLICT UPDATE
> machinery, but that means, I think, that sometimes you will get
> serialization anomalies - at least, I think, you will sometimes obtain
> results that couldn't have been obtained under any serial order of
> execution, and maybe it would end up being possible to fail with
> serialization errors or unique index violations.
>
> In the past, there have been objections to implementations of MERGE
> which would give rise to such serialization anomalies, but I'm not
> sure we should feel bound by those discussions.  One thing that's
> different is that the common and actually-useful case can now be made
> to work in a fairly satisfying way using INSERT .. ON CONFLICT UPDATE;
> if less useful cases are vulnerable to some weirdness, maybe it's OK
> to just document the problems.

Good points.

I didn't say it but my intention was to just throw an ERROR if no
single unique index can be identified.

It could be possible to still run MERGE in that situaton but we would
need to take a full table lock at ShareRowExclusive. It's quite likely
that such statements would throw duplicate update errors, so I
wouldn't be aiming to do anything with that for PG11.

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  6. Re: MERGE SQL Statement for PG11

    Peter Geoghegan <pg@bowt.ie> — 2017-10-27T15:24:38Z

    On Fri, Oct 27, 2017 at 7:41 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
> Good points.
>
> I didn't say it but my intention was to just throw an ERROR if no
> single unique index can be identified.

You'd also throw an error when there was no "upsert compatible" join
quals, I take it?

I don't see the point in that. That's just mapping one syntax on to another.

> It could be possible to still run MERGE in that situaton but we would
> need to take a full table lock at ShareRowExclusive. It's quite likely
> that such statements would throw duplicate update errors, so I
> wouldn't be aiming to do anything with that for PG11.

I would avoid mixing up ON CONFLICT DO UPDATE and MERGE. The
"anomalies" you describe in MERGE are not really anomalies IMV.
They're simply how the feature is supposed to operate, and how it's
possible to make MERGE use alternative join algorithms based only on
the underlying cost. You might use a merge join for a bulk load
use-case, for example.

I think an SQL MERGE feature would be compelling, but I don't think
that it should take much from ON CONFLICT. As I've said many times
[1], they're really two different features (Teradata had features
similar to each, for example). I suggest that you come up with
something that has the semantics that the standard requires, and
therefore makes none of the ON CONFLICT guarantees about an outcome
under concurrency (INSERT or UPDATE). Those guarantees are basically
incompatible with how MERGE needs to work.

In case it matters, I think that the idea of varying relation
heavyweight lock strength based on subtle semantics within a DML
statement is a bad one. Frankly, I think that that's going to be a
nonstarter.

[1] https://www.postgresql.org/message-id/CAM3SWZRP0c3g6+aJ=YYDGYAcTZg0xA8-1_FCVo5Xm7hrEL34kw@mail.gmail.com
-- 
Peter Geoghegan

  7. Re: MERGE SQL Statement for PG11

    Peter Geoghegan <pg@bowt.ie> — 2017-10-27T15:35:05Z

    On Fri, Oct 27, 2017 at 6:24 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> I think one of the reasons why Peter Geoghegan decided to pursue
> INSERT .. ON CONFLICT UPDATE was that, because it is non-standard SQL
> syntax, he felt free to mandate a non-standard SQL requirement, namely
> the presence of a unique index on the arbiter columns.

That's true, but what I was really insistent on, more than anything
else, was that the user would get a practical guarantee about an
insert-or-update outcome under concurrency. There could be no
"unprincipled deadlocks", nor could there be spurious unique
violations.

This is the kind of thing that the SQL standard doesn't really concern
itself with, and yet it's of significant practical importance to
users. Both Oracle and SQL Server allow these things that I
specifically set out to avoid. I think that that's mostly a good
thing, though; they do a really bad job of explaining what's what, and
don't provide for a very real need ("upsert") in some other way, but
their MERGE semantics do make sense to me.

-- 
Peter Geoghegan

  8. Re: MERGE SQL Statement for PG11

    Robert Haas <robertmhaas@gmail.com> — 2017-10-27T16:00:05Z

    On Fri, Oct 27, 2017 at 4:41 PM, Simon Riggs <simon@2ndquadrant.com> wrote:
> I didn't say it but my intention was to just throw an ERROR if no
> single unique index can be identified.
>
> It could be possible to still run MERGE in that situaton but we would
> need to take a full table lock at ShareRowExclusive. It's quite likely
> that such statements would throw duplicate update errors, so I
> wouldn't be aiming to do anything with that for PG11.

Like Peter, I think taking such a strong lock for a DML statement
doesn't sound like a very desirable way forward.  It means, for
example, that you can only have one MERGE in progress on a table at
the same time, which is quite limiting.  It could easily be the case
that you have multiple MERGE statements running at once but they touch
disjoint groups of rows and therefore everything works.  I think the
code should be able to cope with concurrent changes, if nothing else
by throwing an ERROR, and then if the user wants to ensure that
doesn't happen by taking ShareRowExclusiveLock they can do that via an
explicit LOCK TABLE statement -- or else they can prevent concurrency
by any other means they see fit.

Other problems with taking ShareRowExclusiveLock include (1) probable
lock upgrade hazards and (2) do you really want MERGE to kick
autovacuum off of your giant table?  Probably not.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  9. Re: MERGE SQL Statement for PG11

    Serge Rielau <serge@rielau.com> — 2017-10-27T21:13:27Z

    Simon,

Nice writeup.

While the standard may not require a unique index for the ON clause I have
never seen a MERGE statement that did not have this property.  So IMHO this
is a reasonable restrictions.

In fact I have only ever seen two flavors of usage:
* Single row source (most often simply a VALUES clause) in OLTP
  In that case there was lots of concurrency
* Massive source which affects a significant portion of the target table in
DW.
 In this case there were no concurrent MERGEs

I believe support for returning rows at a later stage would prove to be very
powerful, especially in combination with chaining MERGE statements in CTEs.
To do that would require language extensions to pass the coloring of the
source row through, especially for rows that fell into "do nothing".




--
Sent from: http://www.postgresql-archive.org/PostgreSQL-hackers-f1928748.html

  10. Re: MERGE SQL Statement for PG11

    Peter Geoghegan <pg@bowt.ie> — 2017-10-27T21:42:17Z

    On Fri, Oct 27, 2017 at 2:13 PM, srielau <serge@rielau.com> wrote:
> While the standard may not require a unique index for the ON clause I have
> never seen a MERGE statement that did not have this property.  So IMHO this
> is a reasonable restrictions.

The Oracle docs on MERGE say nothing about unique indexes or
constraints. They don't even mention them in passing. They do say
"This statement is a convenient way to combine multiple operations. It
lets you avoid multiple INSERT, UPDATE, and DELETE DML statements."

SQL Server's MERGE docs do mention unique indexes, but only in
passing, saying something about unique violations, and that unique
violations *cannot* be suppressed in MERGE, even though that's
possible with other DML statements (with something called
IGNORE_DUP_KEY).

What other systems *do* have this restriction? I've never seen one that did.

-- 
Peter Geoghegan

  11. Re: MERGE SQL Statement for PG11

    Serge Rielau <serge@rielau.com> — 2017-10-27T22:00:47Z

    via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=9.8.79&pv=10.12.6&source=email_footer_2]
On Fri, Oct 27, 2017 at 2:42 PM, Peter Geoghegan <pg@bowt.ie> wrote:
On Fri, Oct 27, 2017 at 2:13 PM, srielau <serge@rielau.com> wrote:
> While the standard may not require a unique index for the ON clause I have
> never seen a MERGE statement that did not have this property. So IMHO this
> is a reasonable restrictions.

The Oracle docs on MERGE say nothing about unique indexes or
constraints. They don't even mention them in passing. They do say
"This statement is a convenient way to combine multiple operations. It
lets you avoid multiple INSERT, UPDATE, and DELETE DML statements."

SQL Server's MERGE docs do mention unique indexes, but only in
passing, saying something about unique violations, and that unique
violations *cannot* be suppressed in MERGE, even though that's
possible with other DML statements (with something called
IGNORE_DUP_KEY).

What other systems *do* have this restriction? I've never seen one that did. Not clear what you are leading up to here. When I did MERGE in DB2 there was also no limitation: " Each row in the target can only be operated on once. A row in the target can only be identified as MATCHED with one row in the result table of the table-reference” What there was however was a significant amount of code I had to write and test to enforce the above second sentence. IIRC it involved, in the absence of a proof that the join could not expand, adding a row_number() over() AS rn over the target leg of the join and then a row_number() over(partition by rn) > 1 THEN RAISE_ERROR() to catch violators. Maybe in PG there is a trivial way to detect an expanding join and block it at runtime.
So the whole point I’m trying to make is that I haven’t seen the need for the extra work I had to do once the feature appeared in the wild.
Cheers Serge

  12. Re: MERGE SQL Statement for PG11

    Michael Paquier <michael.paquier@gmail.com> — 2017-10-27T22:31:44Z

    On Fri, Oct 27, 2017 at 9:00 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> On Fri, Oct 27, 2017 at 4:41 PM, Simon Riggs <simon@2ndquadrant.com> wrote:
>> I didn't say it but my intention was to just throw an ERROR if no
>> single unique index can be identified.
>>
>> It could be possible to still run MERGE in that situaton but we would
>> need to take a full table lock at ShareRowExclusive. It's quite likely
>> that such statements would throw duplicate update errors, so I
>> wouldn't be aiming to do anything with that for PG11.
>
> Like Peter, I think taking such a strong lock for a DML statement
> doesn't sound like a very desirable way forward.  It means, for
> example, that you can only have one MERGE in progress on a table at
> the same time, which is quite limiting.  It could easily be the case
> that you have multiple MERGE statements running at once but they touch
> disjoint groups of rows and therefore everything works.  I think the
> code should be able to cope with concurrent changes, if nothing else
> by throwing an ERROR, and then if the user wants to ensure that
> doesn't happen by taking ShareRowExclusiveLock they can do that via an
> explicit LOCK TABLE statement -- or else they can prevent concurrency
> by any other means they see fit.

+1, I would suspect users to run this query in parallel of the same
table for multiple data sets.

Peter has taken some time to explain me a bit his arguments today, and
I agree that it does not sound much appealing to have constraint
limitations for MERGE. Particularly using the existing ON CONFLICT
structure gets the feeling of having twice a grammar for what's
basically the same feature, with pretty much the same restrictions.

By the way, this page sums up nicely the situation about many
implementations of UPSERT taken for all systems:
https://en.wikipedia.org/wiki/Merge_(SQL)
-- 
Michael

  13. Re: MERGE SQL Statement for PG11

    Nico Williams <nico@cryptonector.com> — 2017-10-27T23:54:30Z

    On Fri, Oct 27, 2017 at 02:13:27PM -0700, srielau wrote:
> While the standard may not require a unique index for the ON clause I have
> never seen a MERGE statement that did not have this property.  So IMHO this
> is a reasonable restrictions.

I don't understand how one could have a conflict upon which to turn
INSERT into UPDATE without having a UNIQUE constraint violated...

The only question is whether one should have control over -or have to
specify- which constraint violations lead to UPDATE vs. which ones lead
to failure vs. which ones lead to doing nothing.

The row to update is the one that the to-be-inserted row conflicted with
-- there can only have been one if the constraint violated was a PRIMARY
KEY constraint, or if there is a PRIMARY KEY at all, but if there's no
PRIMARY KEY, then there can have been more conflicting rows because of
NULL columns in the to-be-inserted row.  If the to-be-inserted row
conflicts with multiple rows, then just fail, or don't allow MERGE on
tables that have no PK (as you know, many think it makes no sense to not
have a PK on a table in SQL).

In the common case one does not care about which UNIQUE constraint is
violated because there's only one that could have been violated, or
because if the UPDATE should itself cause some other UNIQUE constraint
to be violated, then the whole statement should fail.

PG's UPSERT is fantastic -- it allows very fine-grained control, but it
isn't as pithy as it could be when the author doesn't care to specify
all that detail.

Also, something like SQLite3's INSERT OR REPLACE is very convenient:
pithy, INSERT syntax, upsert-like semantics[*].

I'd like to have this in PG:

  INSERT INTO ..
  ON CONFLICT DO UPDATE;  -- I.e., update all columns of the existing
                          -- row to match the ones from the row that
                          -- would have been inserted had there not been
                          -- a conflict.
                          --
                          -- If an INSERTed row conflicts and then the
                          -- UPDATE it devolves to also conflicts, then
                          -- fail.

and

  INSERT INTO ..
  ON CONFLICT DO UPDATE   -- I.e., update all columns of the existing
                          -- row to match the ones from the row that
                          -- would have been inserted had there not been
                          -- a conflict.
                          --
  ON CONFLICT DO NOTHING; -- If an INSERTed row conflicts and then the
                          -- UPDATE it devolves to also conflicts, then
                          -- DO NOTHING.


[*] SQLite3's INSERT OR REPLACE is NOT an insert-or-update, but an
    insert-or-delete-and-insert, and any deletions that occur in the
    process do fire triggers.  INSERT OR UPDATE would be much more
    useful.


Nico
--

  14. Re: MERGE SQL Statement for PG11

    Simon Riggs <simon@2ndquadrant.com> — 2017-10-28T10:10:50Z

    On 28 October 2017 at 00:31, Michael Paquier <michael.paquier@gmail.com> wrote:
> On Fri, Oct 27, 2017 at 9:00 AM, Robert Haas <robertmhaas@gmail.com> wrote:
>> On Fri, Oct 27, 2017 at 4:41 PM, Simon Riggs <simon@2ndquadrant.com> wrote:
>>> I didn't say it but my intention was to just throw an ERROR if no
>>> single unique index can be identified.
>>>
>>> It could be possible to still run MERGE in that situaton but we would
>>> need to take a full table lock at ShareRowExclusive. It's quite likely
>>> that such statements would throw duplicate update errors, so I
>>> wouldn't be aiming to do anything with that for PG11.
>>
>> Like Peter, I think taking such a strong lock for a DML statement
>> doesn't sound like a very desirable way forward.  It means, for
>> example, that you can only have one MERGE in progress on a table at
>> the same time, which is quite limiting.  It could easily be the case
>> that you have multiple MERGE statements running at once but they touch
>> disjoint groups of rows and therefore everything works.  I think the
>> code should be able to cope with concurrent changes, if nothing else
>> by throwing an ERROR, and then if the user wants to ensure that
>> doesn't happen by taking ShareRowExclusiveLock they can do that via an
>> explicit LOCK TABLE statement -- or else they can prevent concurrency
>> by any other means they see fit.
>
> +1, I would suspect users to run this query in parallel of the same
> table for multiple data sets.
>
> Peter has taken some time to explain me a bit his arguments today, and
> I agree that it does not sound much appealing to have constraint
> limitations for MERGE. Particularly using the existing ON CONFLICT
> structure gets the feeling of having twice a grammar for what's
> basically the same feature, with pretty much the same restrictions.
>
> By the way, this page sums up nicely the situation about many
> implementations of UPSERT taken for all systems:
> https://en.wikipedia.org/wiki/Merge_(SQL)

That Wikipedia article is badly out of date and regrettably does NOT
sum up the current situation nicely any more since MERGE has changed
in definition in SQL:2011 since its introduction in SQL:2003.

I'm proposing a MERGE statement for PG11 that
i) takes a RowExclusiveLock on rows, so can be run concurrently
ii) uses the ON CONFLICT infrastructure to do that
and so requires a unique constraint.

The above is useful behaviour that will be of great benefit to
PostgreSQL users. There are no anomalies remaining.

SQL:2011 specifically states "The extent to which an
SQL-implementation may disallow independent changes that are not
significant is implementation-defined”, so in my reading the above
behaviour would make us fully spec compliant. Thank you to Peter for
providing the infrastructure on which this is now possible for PG11.

Serge puts this very nicely by identifying two different use cases for MERGE.

Now, I accept that you might also want a MERGE statement that
continues to work even if there is no unique constraint, but it would
need to have different properties to the above. I do not in any way
argue against adding that. I also agree that adding RETURNING at a
later stage would be fine as well. I am proposing that those and any
other additional properties people come up with can be added in later
releases once we have the main functionality in core in PG11.

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  15. Re: MERGE SQL Statement for PG11

    Peter Geoghegan <pg@bowt.ie> — 2017-10-28T18:39:33Z

    On Sat, Oct 28, 2017 at 3:10 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
> SQL:2011 specifically states "The extent to which an
> SQL-implementation may disallow independent changes that are not
> significant is implementation-defined”, so in my reading the above
> behaviour would make us fully spec compliant. Thank you to Peter for
> providing the infrastructure on which this is now possible for PG11.
>
> Serge puts this very nicely by identifying two different use cases for MERGE.

MERGE benefits from having a join that is more or less implemented in
the same way as any other join. It can be a merge join, hash join, or
nestloop join. ON CONFLICT doesn't work using a join.

Should I to take it that you won't be supporting any of these
alternative join algorithms? If not, then you'll have something that
really isn't comparable to MERGE as implemented in Oracle, SQL Server,
or DB2. They *all* do this.

Would the user be able to omit WHEN NOT MATCHED/INSERT, as is the case
with every existing MERGE implementation? If so, what actually happens
under the hood when WHEN NOT MATCHED is omitted? For example, would
you actually use a regular "UPDATE FROM" style join, as opposed to the
ON CONFLICT infrastructure? And, if that is so, can you justify the
semantic difference for rows that are updated in each scenario
(omitted vs. not omitted) in READ COMMITTED mode? Note that this could
be the difference between updating a row when *no* version is visible
to our MVCC snapshot, as opposed to doing the EPQ stuff and updating
the latest row version if possible. That's a huge, surprising
difference. On top of all this, you risk live-lock if INSERT isn't a
possible outcome (this is also why ON CONFLICT can never accept a
predicate on its INSERT portion -- again, quite unlike MERGE).

Why not just follow what other systems do? It's actually easier to go
that way, and you get a better outcome. ON CONFLICT involves what you
could call a sleight of hand, and I fear that you don't appreciate
just how specialized the internal infrastructure is.

> Now, I accept that you might also want a MERGE statement that
> continues to work even if there is no unique constraint, but it would
> need to have different properties to the above. I do not in any way
> argue against adding that.

Maybe you *should* be arguing against it, though, and arguing against
ever supporting anything but equijoins, because these things will
*become* impossible if you go down that road. By starting with the ON
CONFLICT infrastructure, while framing no-unique-index-support as work
for some unspecified future release, you're leaving it up to someone
else to resolve the problems. Someone else must square the circle of
mixing ON CONFLICT semantics with fully generalized MERGE semantics.
But who?

-- 
Peter Geoghegan

  16. Re: MERGE SQL Statement for PG11

    Simon Riggs <simon@2ndquadrant.com> — 2017-10-28T19:49:12Z

    On 28 October 2017 at 20:39, Peter Geoghegan <pg@bowt.ie> wrote:
> On Sat, Oct 28, 2017 at 3:10 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
>> SQL:2011 specifically states "The extent to which an
>> SQL-implementation may disallow independent changes that are not
>> significant is implementation-defined”, so in my reading the above
>> behaviour would make us fully spec compliant. Thank you to Peter for
>> providing the infrastructure on which this is now possible for PG11.
>>
>> Serge puts this very nicely by identifying two different use cases for MERGE.
>
> MERGE benefits from having a join that is more or less implemented in
> the same way as any other join. It can be a merge join, hash join, or
> nestloop join. ON CONFLICT doesn't work using a join.
>
> Should I to take it that you won't be supporting any of these
> alternative join algorithms? If not, then you'll have something that
> really isn't comparable to MERGE as implemented in Oracle, SQL Server,
> or DB2. They *all* do this.
>
> Would the user be able to omit WHEN NOT MATCHED/INSERT, as is the case
> with every existing MERGE implementation? If so, what actually happens
> under the hood when WHEN NOT MATCHED is omitted? For example, would
> you actually use a regular "UPDATE FROM" style join, as opposed to the
> ON CONFLICT infrastructure? And, if that is so, can you justify the
> semantic difference for rows that are updated in each scenario
> (omitted vs. not omitted) in READ COMMITTED mode? Note that this could
> be the difference between updating a row when *no* version is visible
> to our MVCC snapshot, as opposed to doing the EPQ stuff and updating
> the latest row version if possible. That's a huge, surprising
> difference. On top of all this, you risk live-lock if INSERT isn't a
> possible outcome (this is also why ON CONFLICT can never accept a
> predicate on its INSERT portion -- again, quite unlike MERGE).
>
> Why not just follow what other systems do? It's actually easier to go
> that way, and you get a better outcome. ON CONFLICT involves what you
> could call a sleight of hand, and I fear that you don't appreciate
> just how specialized the internal infrastructure is.
>
>> Now, I accept that you might also want a MERGE statement that
>> continues to work even if there is no unique constraint, but it would
>> need to have different properties to the above. I do not in any way
>> argue against adding that.
>
> Maybe you *should* be arguing against it, though, and arguing against
> ever supporting anything but equijoins, because these things will
> *become* impossible if you go down that road. By starting with the ON
> CONFLICT infrastructure, while framing no-unique-index-support as work
> for some unspecified future release, you're leaving it up to someone
> else to resolve the problems. Someone else must square the circle of
> mixing ON CONFLICT semantics with fully generalized MERGE semantics.
> But who?

Nothing I am proposing blocks later work.

Everything you say makes it clear that a fully generalized solution is
going to be many years in the making, assuming we agree.

"The extent to which an SQL-implementation may disallow independent
changes that are not significant is implementation-defined”.

So we get to choose. I recommend that we choose something practical.
We're approaching the 10 year anniversary of my first serious attempt
to do MERGE. I say that its time to move forwards with useful
solutions, rather than wait another 10 years for the perfect one, even
assuming it exists.

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  17. Re: MERGE SQL Statement for PG11

    Peter Geoghegan <pg@bowt.ie> — 2017-10-28T19:50:15Z

    On Fri, Oct 27, 2017 at 3:00 PM, Serge Rielau <serge@rielau.com> wrote:
>> What other systems *do* have this restriction? I've never seen one that did.
>
> Not clear what you are leading up to here.
> When I did MERGE in DB2 there was also no limitation:
> "Each row in the target can only be operated on once. A row in the target can only be identified as MATCHED with one row in the result table of the table-reference”
> What there was however was a significant amount of code I had to write and test to enforce the above second sentence.

Then it seems that we were talking about two different things all along.

> Maybe in PG there is a trivial way to detect an expanding join and block it at runtime.

There is for ON CONFLICT. See the cardinality violation logic within
ExecOnConflictUpdate(). (There are esoteric cases where this error can
be raised due to a wCTE that does an insert "from afar", which is
theoretically undesirable but not actually a problem.)

The MERGE implementation that I have in mind would probably do almost
the same thing, and make the "HeapTupleSelfUpdated" case within
ExecUpdate() raise an error when the caller happened to be a MERGE,
rather than following the historic UPDATE behavior. (The behavior is
to silently suppress a second or subsequent UPDATE attempt from the
same command, a behavior that Simon's mock MERGE documentation
references.)

> So the whole point I’m trying to make is that I haven’t seen the need for the extra work I had to do once the feature appeared in the wild.

That seems pretty reasonable to me.

My whole point is that I think it's a mistake to do things like lock
rows ahead of evaluating any UPDATE predicate, in the style of ON
CONFLICT, in order to replicate the ON CONFLICT guarantees [1].

I'm arguing for implementation simplicity, too. Trying to implement
MERGE in a way that extends ON CONFLICT seems like a big mistake to
me, because ON CONFLICT updates rows on the basis of a would-be
duplicate violation, along with all the baggage that that carries.
This is actually enormously different to an equi-join that is fed by a
scan using an MVCC snapshot. The main difference is that there
actually is no MVCC snapshot in play in most cases [2]. If *no* row
with the PK value of 5 is visible to our MVCC snapshot, but an xact
committed having inserted such a row, that still counts as a CONFLICT
with READ COMMITTED.

[1] https://wiki.postgresql.org/wiki/UPSERT#Goals_for_implementation
[2] https://www.postgresql.org/docs/devel/static/transaction-iso.html#xact-read-committed
-- 
Peter Geoghegan

  18. Re: MERGE SQL Statement for PG11

    Peter Geoghegan <pg@bowt.ie> — 2017-10-28T20:04:29Z

    On Sat, Oct 28, 2017 at 12:49 PM, Simon Riggs <simon@2ndquadrant.com> wrote:
> Nothing I am proposing blocks later work.

Actually, many things will block future work if you go down that road.
You didn't respond to the specific points I raised, but that doesn't
mean that they're not real.

> Everything you say makes it clear that a fully generalized solution is
> going to be many years in the making, assuming we agree.

I think that it's formally impossible as long as you preserve the ON
CONFLICT guarantees, unless you somehow define the problems out of
existence. Those are guarantees which no other MERGE implementation
has ever made, and which the SQL standard says nothing about. And for
good reasons.

> "The extent to which an SQL-implementation may disallow independent
> changes that are not significant is implementation-defined”.
>
> So we get to choose. I recommend that we choose something practical.
> We're approaching the 10 year anniversary of my first serious attempt
> to do MERGE. I say that its time to move forwards with useful
> solutions, rather than wait another 10 years for the perfect one, even
> assuming it exists.

As far as I'm concerned, you're the one arguing for an unobtainable
solution over a good one, not me. I *don't* think you should solve the
problems that I raise -- you should instead implement MERGE without
any of the ON CONFLICT guarantees, just like everyone else has.
Building MERGE on top of the ON CONFLICT guarantees, and ultimately
arriving at something that is comparable to other implementations over
many releases might be okay if anyone had the slightest idea of what
that would look like. You haven't even _described the semantics_,
which you could do by addressing the specific points that I raised.

-- 
Peter Geoghegan

  19. Re: MERGE SQL Statement for PG11

    Simon Riggs <simon@2ndquadrant.com> — 2017-10-29T11:48:50Z

    On 28 October 2017 at 22:04, Peter Geoghegan <pg@bowt.ie> wrote:
> On Sat, Oct 28, 2017 at 12:49 PM, Simon Riggs <simon@2ndquadrant.com> wrote:
>> Nothing I am proposing blocks later work.
>
> Actually, many things will block future work if you go down that road.
> You didn't respond to the specific points I raised, but that doesn't
> mean that they're not real.
>
>> Everything you say makes it clear that a fully generalized solution is
>> going to be many years in the making, assuming we agree.
>
> I think that it's formally impossible as long as you preserve the ON
> CONFLICT guarantees, unless you somehow define the problems out of
> existence. Those are guarantees which no other MERGE implementation
> has ever made, and which the SQL standard says nothing about. And for
> good reasons.
>
>> "The extent to which an SQL-implementation may disallow independent
>> changes that are not significant is implementation-defined”.
>>
>> So we get to choose. I recommend that we choose something practical.
>> We're approaching the 10 year anniversary of my first serious attempt
>> to do MERGE. I say that its time to move forwards with useful
>> solutions, rather than wait another 10 years for the perfect one, even
>> assuming it exists.
>
> As far as I'm concerned, you're the one arguing for an unobtainable
> solution over a good one, not me. I *don't* think you should solve the
> problems that I raise -- you should instead implement MERGE without
> any of the ON CONFLICT guarantees, just like everyone else has.
> Building MERGE on top of the ON CONFLICT guarantees, and ultimately
> arriving at something that is comparable to other implementations over
> many releases might be okay if anyone had the slightest idea of what
> that would look like. You haven't even _described the semantics_,
> which you could do by addressing the specific points that I raised.

I have no objection to you writing a better version than me and if my
work inspires you to complete that in a reasonable timescale then we
all win. I'm also very happy to work together on writing the version
described - you have already done much work in this area.

I don't see any major problems in points you've raised so far, though
obviously there is much detail.

All of this needs to be written and then committed, so I'll get on and
write my proposal. We can then see whether that is an 80% solution or
something less. There are more obvious barriers to completion at this
point, like time and getting on with it.

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  20. Re: MERGE SQL Statement for PG11

    Peter Geoghegan <pg@bowt.ie> — 2017-10-29T20:25:25Z

    On Sun, Oct 29, 2017 at 4:48 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
> I have no objection to you writing a better version than me and if my
> work inspires you to complete that in a reasonable timescale then we
> all win.

My whole point is that the way that you seem determined to go on this
is a dead end. I don't think that *anyone* can go improve on what you
come up with if that's based heavily on ON CONFLICT, for the simple
reason that the final user visible design is totally unclear. There is
an easy way to make me shut up - come up with a design for MERGE that
more or less builds on how UPDATE FROM works, rather than building
MERGE on ON CONFLICT. (You might base things like RLS handling on ON
CONFLICT, but in the main MERGE should be like an UPDATE FROM with an
outer join, that can do INSERTs and DELETEs, too.)

The original effort to add MERGE didn't do anything upsert-like, which
Heikki (the GSOC mentor of the project) was perfectly comfortable
with. I'm too lazy to go search the archives right now, but it's
there. Heikki cites the SQL standard.

This is what MERGE *actually is*, which you can clearly see from the
Oracle/SQL Server/DB2 docs. It says this in the first paragraph of
their MERGE documentation. It's crystal clear from their docs -- "This
statement is a convenient way to combine multiple operations. It lets
you avoid multiple INSERT, UPDATE, and DELETE DML statements."

> I'm also very happy to work together on writing the version
> described - you have already done much work in this area.

You seem to want to preserve the ON CONFLICT guarantees at great cost.
But you haven't even defended that based on a high level goal, or a
use case, or something that makes sense to users (describing how it is
possible is another matter). You haven't even tried to convince me.

> I don't see any major problems in points you've raised so far, though
> obviously there is much detail.

Did you even read them? They are not mere details. They're fundamental
to the semantics of the feature (if you base it on ON CONFLICT). It's
not actually important that you understand them all; the important
message is that generalizing ON CONFLICT has all kinds of terrible
problems.

> All of this needs to be written and then committed, so I'll get on and
> write my proposal. We can then see whether that is an 80% solution or
> something less. There are more obvious barriers to completion at this
> point, like time and getting on with it.

Getting on with *what*, exactly?

In general, I have nothing against an 80% solution, or even a 50%
solution, provided there is a plausible path to a 100% solution. I
don't think that you have such a path, but only because you're tacitly
inserting requirements that no other MERGE implementation has to live
with, that I doubt any implementation *could* live with. Again, I'm
not the one making this complicated, or adding requirements that will
be difficult for you to get in to your v1 -- you're the one doing
that.

The semantics that I suggest (the SQL standard's semantics) will
require less code, and will be far simpler. Right now, I simply don't
understand why you're insisting on using ON CONFLICT without even
saying why. I can only surmise that you think that doing so will
simplify the implementation, but I can guarantee you that it won't.

-- 
Peter Geoghegan

  21. Re: MERGE SQL Statement for PG11

    Robert Haas <robertmhaas@gmail.com> — 2017-10-30T08:44:40Z

    On Sun, Oct 29, 2017 at 1:19 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
> Nothing I am proposing blocks later work.

That's not really true.  Nobody's going to be happy if MERGE has one
behavior in one set of cases and an astonishingly different behavior
in another set of cases.  If you adopt a behavior for certain cases
that can't be extended to other cases, then you're blocking a
general-purpose MERGE.

And, indeed, it seems that you're proposing an implementation that
adds no new functionality, just syntax compatibility.  Do we really
want or need two syntaxes  for the same thing in core?  I kinda think
Peter might have the right idea here.  Under his proposal, we'd be
getting something that is, in a way, new.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  22. Re: MERGE SQL Statement for PG11

    Simon Riggs <simon@2ndquadrant.com> — 2017-10-30T13:07:48Z

    On 29 October 2017 at 21:25, Peter Geoghegan <pg@bowt.ie> wrote:

> The semantics that I suggest (the SQL standard's semantics) will
> require less code, and will be far simpler. Right now, I simply don't
> understand why you're insisting on using ON CONFLICT without even
> saying why. I can only surmise that you think that doing so will
> simplify the implementation, but I can guarantee you that it won't.

If you see problems in my proposal, please show the specific MERGE SQL
statements that you think will give problems and explain how and what
the failures will be.

We can then use those test cases to drive developments. If we end up
with code for multiple approaches we will be able to evaluate the
differences between proposals using the test cases produced.

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  23. Re: MERGE SQL Statement for PG11

    Simon Riggs <simon@2ndquadrant.com> — 2017-10-30T13:21:20Z

    On 30 October 2017 at 09:44, Robert Haas <robertmhaas@gmail.com> wrote:
> On Sun, Oct 29, 2017 at 1:19 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
>> Nothing I am proposing blocks later work.
>
> That's not really true.  Nobody's going to be happy if MERGE has one
> behavior in one set of cases and an astonishingly different behavior
> in another set of cases.  If you adopt a behavior for certain cases
> that can't be extended to other cases, then you're blocking a
> general-purpose MERGE.

If a general purpose solution exists, please explain what it is. The
problem is that nobody has ever done so, so its not like we are
discussing the difference between bad solution X and good solution Y,
we are comparing reasonable solution X with non-existent solution Y.

> And, indeed, it seems that you're proposing an implementation that
> adds no new functionality, just syntax compatibility.  Do we really
> want or need two syntaxes  for the same thing in core?  I kinda think
> Peter might have the right idea here.  Under his proposal, we'd be
> getting something that is, in a way, new.

Partitioning looked like "just new syntax", but it has been a useful
new feature.

MERGE provides new capabilities that we do not have and is much more
powerful than INSERT/UPDATE, in a syntax that follow what other
databases use today. Just like partitioning.

Will what I propose do everything in the first release? No, just like
partitioning.

If other developers are able to do things in phases, then I claim that
right also.

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  24. Re: MERGE SQL Statement for PG11

    Peter Geoghegan <pg@bowt.ie> — 2017-10-30T17:59:43Z

    On Mon, Oct 30, 2017 at 6:21 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
>> That's not really true.  Nobody's going to be happy if MERGE has one
>> behavior in one set of cases and an astonishingly different behavior
>> in another set of cases.  If you adopt a behavior for certain cases
>> that can't be extended to other cases, then you're blocking a
>> general-purpose MERGE.
>
> If a general purpose solution exists, please explain what it is.

For the umpteenth time, a general purpose solution is one that more or
less works like an UPDATE FROM, with an outer join, whose ModifyTable
node is capable of insert, update, or delete (and accepts quals for
MATCHED and NOT matched cases, etc). You could still get duplicate
violations due to concurrent activity in READ COMMITTED mode, but not
at higher isolation levels thanks to Thomas Munro's work there. In
this world, ON CONFLICT and MERGE are fairly distinct things.

What's wrong with that? You haven't actually told us why you don't like that.

> The problem is that nobody has ever done so, so its not like we are
> discussing the difference between bad solution X and good solution Y,
> we are comparing reasonable solution X with non-existent solution Y.

Nobody knows what your proposal would be like when time came to remove
the restrictions that you suggest could be removed later. You're the
one with the information here. We need to know what those semantics
would be up-front, since you're effectively committing us down that
path.

You keep making vague appeals to pragmatism, but, in all sincerity, I
don't understand where you're coming from at all. I strongly believe
that generalizing from ON CONFLICT doesn't even make the
implementation easier in the short term. ISTM that you're making this
difficult for yourself for reasons that are known only to you.

>> And, indeed, it seems that you're proposing an implementation that
>> adds no new functionality, just syntax compatibility.  Do we really
>> want or need two syntaxes  for the same thing in core?  I kinda think
>> Peter might have the right idea here.  Under his proposal, we'd be
>> getting something that is, in a way, new.
>
> Partitioning looked like "just new syntax", but it has been a useful
> new feature.

False equivalency. Nobody, including you, ever argued that that work
risked painting us into a corner. (IIRC you said something like the
progress was too small to justify putting into a single release.)

> MERGE provides new capabilities that we do not have and is much more
> powerful than INSERT/UPDATE, in a syntax that follow what other
> databases use today. Just like partitioning.

But you haven't told us *how* it is more powerful. Again, the
semantics of a MERGE that is a generalization of ON CONFLICT are not
at all obvious, and seem like they might be very surprising and risky.
There is no question that it's your job to (at a minimum) define those
semantics ahead of time, since you're going to commit us to them in
the long term if you continue down this path. It is most emphatically
*not* just a "small matter of programming".

-- 
Peter Geoghegan

  25. Re: MERGE SQL Statement for PG11

    Simon Riggs <simon@2ndquadrant.com> — 2017-10-30T18:07:18Z

    On 30 October 2017 at 18:59, Peter Geoghegan <pg@bowt.ie> wrote:

> It is most emphatically *not* just a "small matter of programming".

Please explain in detail the MERGE SQL statements that you think will
be problematic and why.

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  26. Re: MERGE SQL Statement for PG11

    Peter Geoghegan <pg@bowt.ie> — 2017-10-30T18:17:47Z

    On Mon, Oct 30, 2017 at 11:07 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
> Please explain in detail the MERGE SQL statements that you think will
> be problematic and why.

Your proposal is totally incomplete, so I can only surmise its
behavior in certain cases, to make a guess at what the problems might
be (see my remarks on EPQ, live lock, etc). This makes it impossible
to do what you ask right now.

Besides, you haven't answered the question from my last e-mail
("What's wrong with that [set of MERGE semantics]?"), so why should I
go to further trouble? You're just not constructively engaging with me
at this point. We're going around in circles.

-- 
Peter Geoghegan

  27. Re: MERGE SQL Statement for PG11

    Nico Williams <nico@cryptonector.com> — 2017-10-30T18:36:50Z

    On Mon, Oct 30, 2017 at 10:59:43AM -0700, Peter Geoghegan wrote:
> On Mon, Oct 30, 2017 at 6:21 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
> > If a general purpose solution exists, please explain what it is.
> 
> For the umpteenth time, a general purpose solution is one that more or
> less works like an UPDATE FROM, with an outer join, whose ModifyTable
> node is capable of insert, update, or delete (and accepts quals for
> MATCHED and NOT matched cases, etc). You could still get duplicate
> violations due to concurrent activity in READ COMMITTED mode, but not
> at higher isolation levels thanks to Thomas Munro's work there. In
> this world, ON CONFLICT and MERGE are fairly distinct things.

FWIW, and as an outsider, having looked at MERGE docs from other
RDBMSes, I have to agree that the PG UPSERT (ON CONFLICT .. DO) and
MERGE are rather different beasts.

In particular, I suspect all UPSERT statements can be mapped onto
equivalent MERGE statements, but not all MERGE statements can be mapped
onto UPSERTs.

The reason is that UPSERT depends on UNIQUE constraints, whereas MERGE
uses a generic join condition that need not even refer to any INDEXes,
let alone UNIQUE ones.

Perhaps PG's UPSERT can be generalized to create a temporary UNIQUE
constraint on the  specified in the conflict_target portion of the
statement, increasing the number of MERGE statements that could be
mapped onto UPSERT.  But even then, that would still be a UNIQUE
constraint, whereas MERGE does not even imply such a thing.

Now, a subset of MERGE (those using equijoins in the ON condition) can
be mapped onto UPSERT provided either a suitable UNIQUE index exists (or
that PG notionally creates a temporary UNIQUE constraint for the purpose
of evaluating the UPSERT).  This approach would NOT preclude a more
complete subsequent implementation of MERGE.  But I wonder if that's
worthwhile given that a proper and complete implementation of MERGE is
probably very desirable.

On a tangentially related note, I've long wanted to have an RDBMS-
independent SQL parser for the purpose of implementing external query-
rewriting (and external optimizers), syntax highlighting, and so on.
Having an external / plug-in method for rewriting unsupported SQL as a
way of bridging functionality gaps (like lack of MERGE support) would be
very nice.  PG does have a way to expose its AST...  It might be a good
idea to start by implementing unsupported SQL features in such a way
that they parse and can produce an AST along with a syntax/unsupported
error -- then one might rewrite the parsed AST, generate appropriate
SQL, and execute that.

Nico
--

  28. Re: MERGE SQL Statement for PG11

    Stephen Frost <sfrost@snowman.net> — 2017-10-30T18:55:11Z

    * Robert Haas (robertmhaas@gmail.com) wrote:
> On Sun, Oct 29, 2017 at 1:19 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
> > Nothing I am proposing blocks later work.
> 
> That's not really true.  Nobody's going to be happy if MERGE has one
> behavior in one set of cases and an astonishingly different behavior
> in another set of cases.  If you adopt a behavior for certain cases
> that can't be extended to other cases, then you're blocking a
> general-purpose MERGE.
> 
> And, indeed, it seems that you're proposing an implementation that
> adds no new functionality, just syntax compatibility.  Do we really
> want or need two syntaxes  for the same thing in core?  I kinda think
> Peter might have the right idea here.  Under his proposal, we'd be
> getting something that is, in a way, new.

+1.

I don't think MERGE should be radically different from other database
systems and just syntax sugar over a capability we have.  The
downthread comparison to partitioning isn't accurate either.

There's a reason that we have INSERT .. ON CONFLICT and not MERGE and
it's because they aren't the same thing, as Peter's already explained,
both now and when he and I had exactly this same discussion years ago
when he was working on implementing INSERT .. ON CONFLICT.  Time changes
many things, but I don't think anything's changed in this from the prior
discussions about it.

Thanks!

Stephen

  29. Re: MERGE SQL Statement for PG11

    Simon Riggs <simon@2ndquadrant.com> — 2017-10-31T09:25:04Z

    On 30 October 2017 at 19:17, Peter Geoghegan <pg@bowt.ie> wrote:
> On Mon, Oct 30, 2017 at 11:07 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
>> Please explain in detail the MERGE SQL statements that you think will
>> be problematic and why.
>
> Your proposal is totally incomplete, so I can only surmise its
> behavior in certain cases, to make a guess at what the problems might
> be (see my remarks on EPQ, live lock, etc). This makes it impossible
> to do what you ask right now.

Impossible, huh. Henry Ford was right.

If there are challenges ahead, its reasonable to ask for test cases
for that now especially if you think you know what they already are.
Imagine we go forwards 2 months - if you dislike my patch when it
exists you will submit a test case showing the fault. Why not save us
all the trouble and describe that now? Test Driven Development.

> Besides, you haven't answered the question from my last e-mail
> ("What's wrong with that [set of MERGE semantics]?"), so why should I
> go to further trouble? You're just not constructively engaging with me
> at this point.

It's difficult to discuss anything with someone that refuses to
believe that there are acceptable ways around things. I believe there
are.

If you can calm down the rhetoric we can work together, but if you
continue to grandstand it makes it more difficult.


> We're going around in circles.

Not really. You've said some things and I'm waiting for further
details about the problems you've raised.

You've said its possible another way. Show that assertion is actually
true. We're all listening, me especially, for the technical details.

I've got a fair amount of work to do to get the rest of the patch in
shape, so take your time and make it a complete explanation.

My only goal is the MERGE feature in PG11. For me this is a
collaborative engineering challenge not a debate and definitely not an
argument. If you describe your plan of how to do this, I may be able
to follow it and include that design. If you don't, then it will be
difficult for me to include your thoughts. If you or others wish to
write something as well, I have already said that is OK too.

If anybody's goal is to block development or to wait for perfection to
exist at some unstated time in the future, than I disagree with those
thoughts.

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  30. Re: MERGE SQL Statement for PG11

    Geoff Winkless <pgsqladmin@geoff.dj> — 2017-10-31T11:06:27Z

    Can I add my 2c worth, as someone without a horse in the race, as it
were, in the hope that telling me how I've got this wrong might
clarify the argument a bit (or at least you can all start shouting at
me rather than each other :) )

The point of merge is to allow you to choose to either INSERT or
UPDATE (or indeed DELETE) records based on existing state, yes? That
state is whatever the state of the system at the start of the
transaction?

If I understand correctly, the only time when this would be
problematic is if you try to insert a record into a table which would
not allow that INSERT because another transaction has performed an
INSERT by the time the COMMIT happens, and where that new record would
have changed the state of the MERGE clause, yes?

Isn't the only reason this would fail if there is a unique constraint
on that table?

Yes, you could end up INSERTing values from the merge when another
transaction has INSERTed another, but (again, unless I've
misunderstood) there's nothing in the spec that says that shouldn't
happen; meanwhile for those tables that do require unique values you
can use the UPSERT mechanism, no?

Geoff

  31. Re: MERGE SQL Statement for PG11

    Simon Riggs <simon@2ndquadrant.com> — 2017-10-31T11:44:53Z

    On 30 October 2017 at 19:55, Stephen Frost <sfrost@snowman.net> wrote:
> * Robert Haas (robertmhaas@gmail.com) wrote:
>> On Sun, Oct 29, 2017 at 1:19 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
>> > Nothing I am proposing blocks later work.
>>
>> That's not really true.  Nobody's going to be happy if MERGE has one
>> behavior in one set of cases and an astonishingly different behavior
>> in another set of cases.  If you adopt a behavior for certain cases
>> that can't be extended to other cases, then you're blocking a
>> general-purpose MERGE.
>>
>> And, indeed, it seems that you're proposing an implementation that
>> adds no new functionality, just syntax compatibility.  Do we really
>> want or need two syntaxes  for the same thing in core?  I kinda think
>> Peter might have the right idea here.  Under his proposal, we'd be
>> getting something that is, in a way, new.
>
> +1.
>
> I don't think MERGE should be radically different from other database
> systems and just syntax sugar over a capability we have.

I've proposed a SQL Standard compliant implementation that would do
much more than be new syntax over what we already have.

So these two claims aren't accurate: "radical difference" and "syntax
sugar over a capability we have".

> Time changes
> many things, but I don't think anything's changed in this from the prior
> discussions about it.

My proposal is new, that is what has changed.

At this stage, general opinions can be misleading.

Hi ho, hi ho.

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  32. Re: MERGE SQL Statement for PG11

    Stephen Frost <sfrost@snowman.net> — 2017-10-31T11:56:19Z

    Simon,

* Simon Riggs (simon@2ndquadrant.com) wrote:
> On 30 October 2017 at 19:55, Stephen Frost <sfrost@snowman.net> wrote:
> > I don't think MERGE should be radically different from other database
> > systems and just syntax sugar over a capability we have.
> 
> I've proposed a SQL Standard compliant implementation that would do
> much more than be new syntax over what we already have.
> 
> So these two claims aren't accurate: "radical difference" and "syntax
> sugar over a capability we have".

Based on the discussion so far, those are the conclusions I've come to.
Saying they're isn't accurate without providing anything further isn't
likely to be helpful.

> > Time changes
> > many things, but I don't think anything's changed in this from the prior
> > discussions about it.
> 
> My proposal is new, that is what has changed.

I'm happy to admit that I've missed something in the discussion, but
from what I read the difference appeared to be primairly that you're
proposing it, and doing so a couple years later.

> At this stage, general opinions can be misleading.

I'd certainly love to see a MERGE capability that meets the standard and
works in much the same way from a user's perspective as the other RDBMS'
which already implement it.  From prior discussions with Peter on
exactly that subject, I'm also convinced that having that would be a
largely independent piece of work from the INSERT .. ON CONFLICT
capability that we have (just as MERGE is distinct from similar UPSERT
capabilities in other RDBMSs).

The goal here is really just to avoid time wasted developing MERGE based
on top of the INSERT .. ON CONFLICT system only to have it be rejected
later because multiple other committers and major contributors have said
that they don't agree with that approach.  If, given all of this
discussion, you don't feel that's a high risk with your approach then by
all means continue on with what you're thinking and we can review the
patch once it's posted.

Thanks!

Stephen

  33. Re: MERGE SQL Statement for PG11

    Simon Riggs <simon@2ndquadrant.com> — 2017-10-31T12:50:59Z

    On 31 October 2017 at 12:56, Stephen Frost <sfrost@snowman.net> wrote:
> Simon,
>
> * Simon Riggs (simon@2ndquadrant.com) wrote:
>> On 30 October 2017 at 19:55, Stephen Frost <sfrost@snowman.net> wrote:
>> > I don't think MERGE should be radically different from other database
>> > systems and just syntax sugar over a capability we have.
>>
>> I've proposed a SQL Standard compliant implementation that would do
>> much more than be new syntax over what we already have.
>>
>> So these two claims aren't accurate: "radical difference" and "syntax
>> sugar over a capability we have".
>
> Based on the discussion so far, those are the conclusions I've come to.
> Saying they're isn't accurate without providing anything further isn't
> likely to be helpful.

I'm trying to be clear, accurate and non-confrontational.

I appreciate those are your conclusions, which is why I need to be
clear that the proposal has been misunderstood on those stated points.

What else would you like me to add to be helpful? I will try...

I've spent weeks looking at other implementations and combing the
SQLStandard with a fine tooth comb to see what is possible and what is
not. My proposal shows a new way forwards and yet follows the standard
in horrible detail. It has taken me nearly 2 months to write the doc
page submitted here. It is not simply new syntax stretched over
existing capability. The full syntax of MERGE is quite powerful and
will take some months to make work, even with my proposal.

I'm not sure what else to say, but I am happy to answer specific
technical questions that have full context to allow a rational
discussion.

>> > Time changes
>> > many things, but I don't think anything's changed in this from the prior
>> > discussions about it.
>>
>> My proposal is new, that is what has changed.
>
> I'm happy to admit that I've missed something in the discussion, but
> from what I read the difference appeared to be primairly that you're
> proposing it, and doing so a couple years later.
>
>> At this stage, general opinions can be misleading.
>
> I'd certainly love to see a MERGE capability that meets the standard and
> works in much the same way from a user's perspective as the other RDBMS'
> which already implement it.

The standard explains how it should work, with the proviso that the
standard allows us to define concurrent behavior. Serge has explained
that he sees that my proposal covers the main use cases. I don't yet
see how to cover all, but I'm looking to do the most important use
cases first and other things later.

> From prior discussions with Peter on
> exactly that subject, I'm also convinced that having that would be a
> largely independent piece of work from the INSERT .. ON CONFLICT
> capability that we have (just as MERGE is distinct from similar UPSERT
> capabilities in other RDBMSs).
>
> The goal here is really just to avoid time wasted developing MERGE based
> on top of the INSERT .. ON CONFLICT system only to have it be rejected
> later because multiple other committers and major contributors have said
> that they don't agree with that approach.  If, given all of this
> discussion, you don't feel that's a high risk with your approach then by
> all means continue on with what you're thinking and we can review the
> patch once it's posted.

It is certainly a risk and I don't want to waste time either. I am
happy to consider any complete proposal for how to proceed, including
details and/or code, but I will be prioritising things to ensure that
we have a candidate feature for PG11 from at least one author, rather
than a research project for PG12+.

The key point here is concurrency. I have said that there is an
intermediate step that is useful and achievable, but does not cover
every case. Serge has also stated this - who it should be noted is the
only person in this discussion that has already written the MERGE
statement in SQL, for DB2, so I give more weight to his technical
opinion than I do to others, at this time.

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  34. Re: MERGE SQL Statement for PG11

    Peter Geoghegan <pg@bowt.ie> — 2017-10-31T17:55:48Z

    On Tue, Oct 31, 2017 at 2:25 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
> If there are challenges ahead, its reasonable to ask for test cases
> for that now especially if you think you know what they already are.
> Imagine we go forwards 2 months - if you dislike my patch when it
> exists you will submit a test case showing the fault. Why not save us
> all the trouble and describe that now? Test Driven Development.

I already have, on several occasions now. But if you're absolutely
insistent on my constructing the test case in terms of a real SQL
statement, then that's what I'll do.

Consider this MERGE statement, from your mock documentation:

MERGE INTO wines w
USING wine_stock_changes s
ON s.winename = w.winename
WHEN NOT MATCHED AND s.stock_delta > 0 THEN
  INSERT VALUES(s.winename, s.stock_delta)
WHEN MATCHED AND w.stock + s.stock_delta > 0 THEN
  UPDATE SET stock = w.stock + s.stock_delta
ELSE
  DELETE;

Suppose we remove the WHEN NOT MATCHED case, leaving us with:

MERGE INTO wines w
USING wine_stock_changes s
ON s.winename = w.winename
WHEN MATCHED AND w.stock + s.stock_delta > 0 THEN
  UPDATE SET stock = w.stock + s.stock_delta
ELSE
  DELETE;

We now have a MERGE that will not INSERT, but will continue to UPDATE
and DELETE. (It's implied that your syntax cannot do this at all,
because you propose use the ON CONFLICT infrastructure, but I think we
have to imagine a world in which that restriction either never existed
or has subsequently been lifted.)

The problem here is: Iff the first statement uses ON CONFLICT
infrastructure, doesn't the absence of WHEN NOT MATCHED imply
different semantics for the remaining updates and deletes in the
second version of the query? You've removed what seems like a neat
adjunct to the MERGE, but it actually changes everything else too when
using READ COMMITTED. Isn't that pretty surprising? If you're not
clear on what I mean, see my previous remarks on EPQ, live lock, and
what a CONFLICT could be in READ COMMITTED mode. Concurrent activity
at READ COMMITTED mode can be expected to significantly alter the
outcome here.

Why not just always use the behavior that the second query requires,
which is very much like an UPDATE FROM with an outer join that can
sometimes do deletes (and inserts)? We should always use an MVCC
snapshot, and never play ON CONFLICT style games with visibility/dirty
snapshots.

> It's difficult to discuss anything with someone that refuses to
> believe that there are acceptable ways around things. I believe there
> are.

Isn't that blind faith? Again, it seems like you haven't really tried
to convince me.

> If you can calm down the rhetoric we can work together, but if you
> continue to grandstand it makes it more difficult.

I'm trying to break the deadlock, by getting you to actually consider
what I'm saying. I don't enjoy confrontation. Currently, it seems like
you're just ignoring my objections, which you actually could fairly
easily work through. That is rather frustrating.

> You've said its possible another way. Show that assertion is actually
> true. We're all listening, me especially, for the technical details.

My proposal, if you want to call it that, has the merit of actually
being how MERGE works in every other system. Both Robert and Stephen
seem to be almost sold on what I suggest, so I think that I've
probably already explained my position quite well.

-- 
Peter Geoghegan

  35. Mapping MERGE onto CTEs (Re: MERGE SQL Statement for PG11)

    Nico Williams <nico@cryptonector.com> — 2017-11-01T16:56:20Z

    Is it possible to map MERGE onto a query with CTEs that does the the
various DMLs, with all but the last RETURNING?  Here's a sketch:

WITH matched_rows AS (
        SELECT FROM <target> t WHERE <condition>
     ),
     updated_rows AS (
        UPDATE <target> t
        SET ...
        WHERE ... AND t in (SELECT j FROM matched_rows j)
        RETURNING t
     ),
     inserted_rows AS (
        INSERT INTO <target> t
        SELECT ...
        WHERE ... AND t NOT IN (SELECT j FROM matched_rows j)
        RETURNING t
     ),
DELETE FROM <target> t
WHERE ...;

Now, one issue is that in PG CTEs are basically like temp tables, and
also like optimizer barriers, so this construction is not online, and if
matched_rows is very large, that would be a problem.

As an aside, I'd like to be able to control which CTEs are view-like and
which are table-like.  In SQLite3, for example, they are all view-like,
and the optimizer will act accordingly, whereas in PG they are all
table-like, and thus optimizer barriers.

Nico
--

  36. Re: Mapping MERGE onto CTEs (Re: MERGE SQL Statement for PG11)

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2017-11-01T17:14:03Z

    Nico Williams wrote:

> As an aside, I'd like to be able to control which CTEs are view-like and
> which are table-like.  In SQLite3, for example, they are all view-like,
> and the optimizer will act accordingly, whereas in PG they are all
> table-like, and thus optimizer barriers.

There was a short and easy to grasp (OK, maybe not) discussion on the
topic of CTEs acting differently.  I think the consensus is that for
CTEs that are read-only and do not use functions that aren't immutable,
they may be considered for inlining.
https://www.postgresql.org/message-id/5351711493487900@web53g.yandex.ru

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  37. Re: MERGE SQL Statement for PG11

    Simon Riggs <simon@2ndquadrant.com> — 2017-11-01T17:19:36Z

    On 31 October 2017 at 18:55, Peter Geoghegan <pg@bowt.ie> wrote:
> On Tue, Oct 31, 2017 at 2:25 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
>> If there are challenges ahead, its reasonable to ask for test cases
>> for that now especially if you think you know what they already are.
>> Imagine we go forwards 2 months - if you dislike my patch when it
>> exists you will submit a test case showing the fault. Why not save us
>> all the trouble and describe that now? Test Driven Development.
>
> I already have, on several occasions now. But if you're absolutely
> insistent on my constructing the test case in terms of a real SQL
> statement, then that's what I'll do.
>
> Consider this MERGE statement, from your mock documentation:
>
> MERGE INTO wines w
> USING wine_stock_changes s
> ON s.winename = w.winename
> WHEN NOT MATCHED AND s.stock_delta > 0 THEN
>   INSERT VALUES(s.winename, s.stock_delta)
> WHEN MATCHED AND w.stock + s.stock_delta > 0 THEN
>   UPDATE SET stock = w.stock + s.stock_delta
> ELSE
>   DELETE;
>
> Suppose we remove the WHEN NOT MATCHED case, leaving us with:
>
> MERGE INTO wines w
> USING wine_stock_changes s
> ON s.winename = w.winename
> WHEN MATCHED AND w.stock + s.stock_delta > 0 THEN
>   UPDATE SET stock = w.stock + s.stock_delta
> ELSE
>   DELETE;
>
> We now have a MERGE that will not INSERT, but will continue to UPDATE
> and DELETE.

Agreed

> (It's implied that your syntax cannot do this at all,
> because you propose use the ON CONFLICT infrastructure, but I think we
> have to imagine a world in which that restriction either never existed
> or has subsequently been lifted.)
>
> The problem here is: Iff the first statement uses ON CONFLICT
> infrastructure, doesn't the absence of WHEN NOT MATCHED imply
> different semantics for the remaining updates and deletes in the
> second version of the query?

Not according to the SQL Standard, no. I have no plans for such
differences to exist.

Spec says: If we hit HeapTupleSelfUpdated then we throw an ERROR.

> You've removed what seems like a neat
> adjunct to the MERGE, but it actually changes everything else too when
> using READ COMMITTED. Isn't that pretty surprising?

I think you're presuming things I haven't said and don't mean, so
we're both surprised.

> If you're not
> clear on what I mean, see my previous remarks on EPQ, live lock, and
> what a CONFLICT could be in READ COMMITTED mode. Concurrent activity
> at READ COMMITTED mode can be expected to significantly alter the
> outcome here.

And I still have questions about what exactly you mean, but at least
this post is going in the right direction and I'm encouraged. Thank
you,

I think we need some way of expressing the problems clearly.

> That is rather frustrating.

Guess so.

>> You've said its possible another way. Show that assertion is actually
>> true. We're all listening, me especially, for the technical details.
>
> My proposal, if you want to call it that, has the merit of actually
> being how MERGE works in every other system. Both Robert and Stephen
> seem to be almost sold on what I suggest, so I think that I've
> probably already explained my position quite well.

The only info I have is

"a general purpose solution is one that more or
less works like an UPDATE FROM, with an outer join, whose ModifyTable
node is capable of insert, update, or delete (and accepts quals for
MATCHED and NOT matched cases, etc). You could still get duplicate
violations due to concurrent activity in READ COMMITTED mode".

Surely the whole point of this is to avoid duplicate violations due to
concurrent activity?

I'm not seeing how either design sketch rigorously avoids live locks,
but those are fairly unlikely and easy to detect and abort.

Thank you for a constructive email, we are on the way to somewhere good.

I have more to add, but wanted to get back to you soonish.

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  38. Re: MERGE SQL Statement for PG11

    Peter Geoghegan <pg@bowt.ie> — 2017-11-01T18:20:32Z

    On Wed, Nov 1, 2017 at 10:19 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
>> The problem here is: Iff the first statement uses ON CONFLICT
>> infrastructure, doesn't the absence of WHEN NOT MATCHED imply
>> different semantics for the remaining updates and deletes in the
>> second version of the query?
>
> Not according to the SQL Standard, no. I have no plans for such
> differences to exist.
>
> Spec says: If we hit HeapTupleSelfUpdated then we throw an ERROR.

Your documentation said that the MERGE was driven by a speculative
insertion (BTW, I don't think that this internal implementation detail
should be referenced in user-facing docs). I inferred that that could
not always be true, since there won't always be an INSERT/WHEN NOT
MATCHED case, assuming that you allow that at all (I now gather that you
will).

>> You've removed what seems like a neat
>> adjunct to the MERGE, but it actually changes everything else too when
>> using READ COMMITTED. Isn't that pretty surprising?
>
> I think you're presuming things I haven't said and don't mean, so
> we're both surprised.

You're right -- I'm surmising what I think might be true, because I
don't have the information available to know one way or the other. As
far as this issue with using speculative insertions in one context but
not in another goes, I still don't really know where you stand. I can
still only surmise that you must want both implementations, and will use
one or the other as circumstances dictate (to avoid dup violations in
the style of ON CONFLICT where that's possible).

This seems true because you now say that it will be possible to omit
WHEN NOT MATCHED, and yet there is no such thing as a speculative
insertion without the insertion. You haven't said that that conclusion
is true yourself, but it's the only conclusion that I can draw based on
what you have said.

> I think we need some way of expressing the problems clearly.

It's certainly hard to talk about these problems. I know this from
experience.

> "a general purpose solution is one that more or
> less works like an UPDATE FROM, with an outer join, whose ModifyTable
> node is capable of insert, update, or delete (and accepts quals for
> MATCHED and NOT matched cases, etc). You could still get duplicate
> violations due to concurrent activity in READ COMMITTED mode".
>
> Surely the whole point of this is to avoid duplicate violations due to
> concurrent activity?

Now we're getting somewhere.

I *don't* think that that's the whole point of MERGE. No other MERGE
implementation does that, or claims to do that. The SQL standard says
nothing about this. Heikki found this to be acceptable when working on
the GSoC MERGE implementation that went nowhere.

My position is that we ought to let MERGE be MERGE, and let ON CONFLICT
be ON CONFLICT.

In Postgres, you can avoid duplicate violations with MERGE by using a
higher isolation level (these days, those are turned into a
serialization error at higher isolation levels when no duplicate is
visible to the xact's snapshot).  MERGE isn't and shouldn't be special
when it comes to concurrency.

> I'm not seeing how either design sketch rigorously avoids live locks,
> but those are fairly unlikely and easy to detect and abort.

My MERGE semantics (which really are not mine at all) avoid live
lock/lock starvation by simply never retrying anything without making
forward progress. MERGE doesn't take any special interest in
concurrency, just like any other DML statement that isn't INSERT with ON
CONFLICT.

ON CONFLICT would have live locks if it didn't always have the choice of
inserting [1]. In many ways, the syntax of INSERT ON CONFLICT DO UPDATE
is restricted in exactly the way it needs to be in order to function
correctly. It wasn't an accident that it didn't end up being UPDATE ...
ON NOUPDATE DO INSERT, or something like that, which Robert proposed at
one point.

ON CONFLICT plays by its own rules to a certain extent, because that's
what you need in order to get the desired guarantees in READ COMMITTED
mode [2]. This is the main reason why it was as painful a project as it
was. Further generalizing that seems fraught with difficulties. It seems
logically impossible to generalize it in a way where you don't end up
with two behaviors masquerading as one.

[1] https://wiki.postgresql.org/wiki/UPSERT#Theoretical_lock_starvation_hazards
[2] https://wiki.postgresql.org/wiki/UPSERT#Goals_for_implementation
--
Peter Geoghegan

  39. Re: MERGE SQL Statement for PG11

    Craig Ringer <craig@2ndquadrant.com> — 2017-11-02T02:58:33Z

    On 1 November 2017 at 01:55, Peter Geoghegan <pg@bowt.ie> wrote:

> The problem here is: Iff the first statement uses ON CONFLICT
> infrastructure, doesn't the absence of WHEN NOT MATCHED imply
> different semantics for the remaining updates and deletes in the
> second version of the query? You've removed what seems like a neat
> adjunct to the MERGE, but it actually changes everything else too when
> using READ COMMITTED.

Would these concerns be alleviated by adding some kind of Pg-specific
decoration that constrained concurrency-safe MERGEs?

So your first statement would be

 MERGE CONCURRENTLY ...

and when you removed the WHEN NOT MATCHED clause it'd ERROR because
that's no longer able to be done with the same concurrency-safe
semantics?

I don't know if this would be helpful TBH, or if it would negate
Simon's compatibility goals. Just another idea.

-- 
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

  40. Re: Mapping MERGE onto CTEs (Re: MERGE SQL Statement for PG11)

    Craig Ringer <craig@2ndquadrant.com> — 2017-11-02T08:06:50Z

    On 2 November 2017 at 01:14, Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
> Nico Williams wrote:
>
>> As an aside, I'd like to be able to control which CTEs are view-like and
>> which are table-like.  In SQLite3, for example, they are all view-like,
>> and the optimizer will act accordingly, whereas in PG they are all
>> table-like, and thus optimizer barriers.
>
> There was a short and easy to grasp (OK, maybe not) discussion on the
> topic of CTEs acting differently.  I think the consensus is that for
> CTEs that are read-only and do not use functions that aren't immutable,
> they may be considered for inlining.
> https://www.postgresql.org/message-id/5351711493487900@web53g.yandex.ru

Yep. All theoretical though, I don't think anyone (myself included)
stumped up a patch.


-- 
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

  41. Re: MERGE SQL Statement for PG11

    Robert Haas <robertmhaas@gmail.com> — 2017-11-02T16:51:26Z

    On Thu, Nov 2, 2017 at 8:28 AM, Craig Ringer <craig@2ndquadrant.com> wrote:
> On 1 November 2017 at 01:55, Peter Geoghegan <pg@bowt.ie> wrote:
>> The problem here is: Iff the first statement uses ON CONFLICT
>> infrastructure, doesn't the absence of WHEN NOT MATCHED imply
>> different semantics for the remaining updates and deletes in the
>> second version of the query? You've removed what seems like a neat
>> adjunct to the MERGE, but it actually changes everything else too when
>> using READ COMMITTED.
>
> Would these concerns be alleviated by adding some kind of Pg-specific
> decoration that constrained concurrency-safe MERGEs?
>
> So your first statement would be
>
>  MERGE CONCURRENTLY ...
>
> and when you removed the WHEN NOT MATCHED clause it'd ERROR because
> that's no longer able to be done with the same concurrency-safe
> semantics?
>
> I don't know if this would be helpful TBH, or if it would negate
> Simon's compatibility goals. Just another idea.

Yes, that fixes the problem.  Of course, it also turns MERGE
CONCURRENTLY into syntactic sugar for INSERT ON CONFLICT UPDATE, which
brings one back to the question of exactly what we're trying to
achieve here.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  42. Re: MERGE SQL Statement for PG11

    Robert Haas <robertmhaas@gmail.com> — 2017-11-02T17:06:40Z

    On Tue, Oct 31, 2017 at 5:14 PM, Simon Riggs <simon@2ndquadrant.com> wrote:
> I've proposed a SQL Standard compliant implementation that would do
> much more than be new syntax over what we already have.
>
> So these two claims aren't accurate: "radical difference" and "syntax
> sugar over a capability we have".

I think those claims are pretty accurate.

The design of INSERT .. ON CONFLICT UPDATE and the supporting
machinery only work if there is a unique index.  It provides radically
different behavior than what you get with any other DML statement,
with completely novel concurrency behavior, and there is no reasonable
way to emulate that behavior in cases where no relevant unique index
exists.  Therefore, if MERGE eventually uses INSERT .. ON CONFLICT
UPDATE when a relevant unique index exists and does something else,
such as your proposal of taking a strong lock, or Peter's proposal of
doing this in a concurrency-oblivious manner, in other cases, then
those two cases will behave very differently.

And if, in the meantime, MERGE can only handle the cases where there
is a unique index, then it can only handle the cases INSERT .. ON
CONFLICT UPDATE can cover, which makes it, as far as I can see,
syntactic sugar over what we already have.  Maybe it's not entirely -
you might be planning to make some minor functional enhancements - but
it's not clear what those are, and I feel like whatever it is could be
done with less work and more elegance by just extending the INSERT ..
ON CONFLICT UPDATE syntax.

And it does seem to be your intention to only handle the cases which
the INSERT .. ON CONFLICT UPDATE infrastructure can cover, because
upthread you wrote this: "I didn't say it but my intention was to just
throw an ERROR if no single unique index can be identified."  I don't
think anybody's putting words into your mouth here.  We're just
reading what you wrote.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  43. Re: MERGE SQL Statement for PG11

    Peter Geoghegan <pg@bowt.ie> — 2017-11-02T18:14:01Z

    Robert Haas <robertmhaas@gmail.com> wrote:
>And if, in the meantime, MERGE can only handle the cases where there
>is a unique index, then it can only handle the cases INSERT .. ON
>CONFLICT UPDATE can cover, which makes it, as far as I can see,
>syntactic sugar over what we already have.  Maybe it's not entirely -
>you might be planning to make some minor functional enhancements - but
>it's not clear what those are, and I feel like whatever it is could be
>done with less work and more elegance by just extending the INSERT ..
>ON CONFLICT UPDATE syntax.

+1

Marko Tiikkaja's INSERT ... ON CONFLICT SELECT patch, which is in the
current CF [1], moves things in this direction. I myself have
occasionally wondered if it was worth adding an alternative DO DELETE
conflict_action. This could appear alongside DO UPDATE, and be applied
using MERGE-style conditions.

All of these things seem like small adjuncts to ON CONFLICT because
they're all just an alternative way of modifying or projecting the tuple
that is locked by ON CONFLICT. Everything new would have to happen after
the novel ON CONFLICT handling has already completed.

The only reason that I haven't pursued this is because it doesn't seem
that compelling. I mention it now because It's worth acknowledging that
ON CONFLICT could be pushed a bit further in this direction. Of course,
this still falls far short of making ON CONFLICT entirely like MERGE.

[1] https://commitfest.postgresql.org/15/1241/
-- 
Peter Geoghegan

  44. Re: MERGE SQL Statement for PG11

    Simon Riggs <simon@2ndquadrant.com> — 2017-11-02T18:49:18Z

    On 1 November 2017 at 18:20, Peter Geoghegan <pg@bowt.ie> wrote:

> In Postgres, you can avoid duplicate violations with MERGE by using a
> higher isolation level (these days, those are turned into a
> serialization error at higher isolation levels when no duplicate is
> visible to the xact's snapshot).

So if I understand you correctly, in your view MERGE should just fail
with an ERROR if it runs concurrently with other DML?

i.e. if a race condition between the query and an INSERT runs
concurrently with another INSERT

We have no interest in making that work?

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  45. Re: MERGE SQL Statement for PG11

    Nico Williams <nico@cryptonector.com> — 2017-11-02T19:16:13Z

    On Thu, Nov 02, 2017 at 06:49:18PM +0000, Simon Riggs wrote:
> On 1 November 2017 at 18:20, Peter Geoghegan <pg@bowt.ie> wrote:
> > In Postgres, you can avoid duplicate violations with MERGE by using a
> > higher isolation level (these days, those are turned into a
> > serialization error at higher isolation levels when no duplicate is
> > visible to the xact's snapshot).
> 
> So if I understand you correctly, in your view MERGE should just fail
> with an ERROR if it runs concurrently with other DML?
> 
> i.e. if a race condition between the query and an INSERT runs
> concurrently with another INSERT
> 
> We have no interest in making that work?

If you map MERGE to a DML with RETURNING-DML CTEs as I suggested before,
how would that interact with concurrent DMLs?  The INSERT DML of the
mapped statement could produce conflicts that abort the whole MERGE,
correct?

If you want to ignore conflicts arising from concurrency you could
always add an ON CONFLICT DO NOTHING to the INSERT DML in the mapping I
proposed earlier.  Thus a MERGE CONCURRENTLY could just do that.

Is there any reason not to map MERGE as I proposed?

Such an implementation of MERGE wouldn't be online because CTEs are
always implemented sequentially currently.  That's probably reason
enough to eventually produce a native implementation of MERGE, ... or to
revamp the CTE machinery to allow such a mapping to be online.

Nico
--

  46. Re: MERGE SQL Statement for PG11

    Peter Geoghegan <pg@bowt.ie> — 2017-11-02T19:16:36Z

    Simon Riggs <simon@2ndquadrant.com> wrote:
>So if I understand you correctly, in your view MERGE should just fail
>with an ERROR if it runs concurrently with other DML?

That's certainly my opinion on the matter. It seems like that might be
the consensus, too.

Obviously there are things that you as a user can do about this on your
own, like opt to use a higher isolation level, or manually LOCK TABLE.
For some use cases, including bulk loading for OLAP, users might just
know that there isn't going to be concurrent activity because it's not
an OLTP system.

If this still seems odd to you, then consider that exactly the same
situation exists with UPDATE. A user could want their UPDATE to affect a
row where no row version is actually visible to their MVCC snapshot,
because they have an idea about reliably updating the latest row. UPDATE
doesn't work like that, of course. Is this unacceptable because the user
expects that it should work that way?

Bear in mind that ON CONFLICT DO UPDATE *can* actually update a row when
there is no version of it visible to the snapshot. It can also update a
row where there is a concurrent DELETE + INSERT, and the tuples with the
relevant unique index values end up not even being part of the same
update chain in each case (MVCC-snapshot-visible vs. latest). IOW, you
may end up updating a completely different logical row to the row with
the conflicting value that is visible to your MVCC snapshot!

>i.e. if a race condition between the query and an INSERT runs
>concurrently with another INSERT
>
>We have no interest in making that work?

Without meaning to sound glib: we already did make it work for a
special, restricted case that is important enough to justify introducing
a couple of kludges -- ON CONFLICT DO UPDATE/upsert.

I do agree that what I propose for MERGE will probably cause confusion;
just look into Oracle's MERGE implementation for examples of this.  We
ought to go out of our way to make it clear that MERGE doesn't provide
these guarantees.

--
Peter Geoghegan

  47. Re: MERGE SQL Statement for PG11

    Nico Williams <nico@cryptonector.com> — 2017-11-02T19:39:33Z

    If nothing else, anyone needing MERGE can port their MERGE statements to
a DML with DML-containing CTEs...

The generic mapping would be something like this, I think:

WITH
    rows AS (SELECT <target> FROM <target> WHERE <condition>)
  , updated AS (
    UPDATE <target>
    SET ...
    WHERE <key> IN (SELECT <key> FROM rows) /* matched */
    RETURNING <target>
  )
  , inserted AS (
    INSERT INTO <target>
    SELECT ...
    WHERE <key> NOT IN (SELECT <key> FROM rows) /* not matched */
    RETURNING <target>
  )
DELETE FROM <target>
WHERE (...) AND
  <key> NOT IN (SELECT <key> FROM updated UNION
                SELECT <key> FROM inserted);

Nico
--

  48. Re: MERGE SQL Statement for PG11

    Peter Geoghegan <pg@bowt.ie> — 2017-11-02T19:51:45Z

    Nico Williams <nico@cryptonector.com> wrote:
>If you want to ignore conflicts arising from concurrency you could
>always add an ON CONFLICT DO NOTHING to the INSERT DML in the mapping I
>proposed earlier.  Thus a MERGE CONCURRENTLY could just do that.
>
>Is there any reason not to map MERGE as I proposed?

Performance, for one. MERGE generally has a join that can be optimized
like an UPDATE FROM join.

I haven't studied this question in any detail, but FWIW I think that
using CTEs for merging is morally equivalent to a traditional MERGE
implementation. It may actually be possible to map from CTEs to a MERGE
statement, but I don't think that that's a good approach to implementing
MERGE.

Most of the implementation time will probably be spent doing things like
making sure MERGE behaves appropriately with triggers, RLS, updatable
views, and so on. That will take quite a while, but isn't particularly
technically challenging IMV.

--
Peter Geoghegan

  49. Re: MERGE SQL Statement for PG11

    Nico Williams <nico@cryptonector.com> — 2017-11-02T20:32:24Z

    On Thu, Nov 02, 2017 at 12:51:45PM -0700, Peter Geoghegan wrote:
> Nico Williams <nico@cryptonector.com> wrote:
> >If you want to ignore conflicts arising from concurrency you could
> >always add an ON CONFLICT DO NOTHING to the INSERT DML in the mapping I
> >proposed earlier.  Thus a MERGE CONCURRENTLY could just do that.
> >
> >Is there any reason not to map MERGE as I proposed?
> 
> Performance, for one. MERGE generally has a join that can be optimized
> like an UPDATE FROM join.

Ah, right, I think my mapping was pessimal.  How about this mapping
instead then:

WITH
    updated AS (
      UPDATE <target>
      SET ...
      WHERE <condition>
      RETURNING <target>
  )
  , inserted AS (
      INSERT INTO <target>
      SELECT ...
      WHERE <key> NOT IN (SELECT <key> FROM updated) AND ..
      /*
       * Add ON CONFLICT DO NOTHING here to avoid conflicts in the face
       * of concurrency.
       */
      RETURNING <target>
  )
DELETE FROM <target>
WHERE <key> NOT IN (SELECT <key> FROM updated) AND
      <key> NOT IN (SELECT <key> FROM inserted) AND ...;

?

If a MERGE has no delete clause, then the mapping would be:

WITH
    updated AS (
      UPDATE <target>
      SET ...
      WHERE <condition>
      RETURNING <target>
  )
INSERT INTO <target>
SELECT ...
WHERE <key> NOT IN (SELECT <key> FROM updated) AND ..
/*
 * Add ON CONFLICT DO NOTHING here to avoid conflicts in the face
 * of concurrency.
 */
;

> I haven't studied this question in any detail, but FWIW I think that
> using CTEs for merging is morally equivalent to a traditional MERGE
> implementation. [...]

I agree.  So why not do that initially?  Optimize later.

Such a MERGE mapping could be implemented entirely within
src/backend/parser/gram.y ...

Talk about cheap to implement, review, and maintain!

Also, this would be notionally very simple.

Any optimizations to CTE query/DML execution would be generic and
applicable to MERGE and other things besides.  If mapping MERGE to
CTE-using DMLs motivates such optimizations, all the better.

>          [...]. It may actually be possible to map from CTEs to a MERGE
> statement, but I don't think that that's a good approach to implementing
> MERGE.

Surely not every DML with CTEs can map to MERGE.  Maybe I misunderstood
your comment?

> Most of the implementation time will probably be spent doing things like
> making sure MERGE behaves appropriately with triggers, RLS, updatable
> views, and so on. That will take quite a while, but isn't particularly
> technically challenging IMV.

Note that mapping to a DML with CTEs as above gets triggers, RLS, and
updateable views right from the get-go, because DMLs with CTEs, and DMLs
as CTEs, surely do as well.

Nico
--

  50. Re: MERGE SQL Statement for PG11

    Simon Riggs <simon@2ndquadrant.com> — 2017-11-02T20:36:00Z

    On 2 November 2017 at 19:16, Peter Geoghegan <pg@bowt.ie> wrote:
> Simon Riggs <simon@2ndquadrant.com> wrote:
>>
>> So if I understand you correctly, in your view MERGE should just fail
>> with an ERROR if it runs concurrently with other DML?
>
>
> That's certainly my opinion on the matter. It seems like that might be
> the consensus, too.

Given that I only just found out what you've been talking about, I
don't believe that anybody else did either.

I think people imagined you had worked out how to make MERGE run
concurrently, I certainly did, but in fact you're just saying you
don't believe it ever should.

That is strange since the SQL Standard specifically allows the
implementation to decide upon concurrent behaviour.


> Without meaning to sound glib: we already did make it work for a
> special, restricted case that is important enough to justify introducing
> a couple of kludges -- ON CONFLICT DO UPDATE/upsert.
>
> I do agree that what I propose for MERGE will probably cause confusion;
> just look into Oracle's MERGE implementation for examples of this.  We
> ought to go out of our way to make it clear that MERGE doesn't provide
> these guarantees.

So in your view we should make no attempt to avoid concurrent errors,
even when we have the capability to do so (in some cases) and doing so
would be perfectly compliant with the SQLStandard.

Yes, that certainly will make an easier patch for MERGE.

Or are you arguing against allowing any patch for MERGE?

Now we have more clarity, who else agrees with this?

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  51. Re: MERGE SQL Statement for PG11

    Peter Geoghegan <pg@bowt.ie> — 2017-11-02T21:05:19Z

    Simon Riggs <simon@2ndquadrant.com> wrote:
>I think people imagined you had worked out how to make MERGE run
>concurrently, I certainly did, but in fact you're just saying you
>don't believe it ever should.

I'm certain that they didn't think that at all. But I'll let them speak
for themselves.

>That is strange since the SQL Standard specifically allows the
>implementation to decide upon concurrent behaviour.

And yet nobody else decided to do what you propose with this apparent
leeway.  (See the UPSERT wiki page for many references that confirm
this.)

>So in your view we should make no attempt to avoid concurrent errors,
>even when we have the capability to do so (in some cases) and doing so
>would be perfectly compliant with the SQLStandard.

Yes. That's what I believe. I believe this because I can't see a way to
do this that isn't a mess, and because ON CONFLICT DO UPDATE exists and
works well for the cases where we can do better in READ COMMITTED mode.

Did you know that Oracle doesn't have an EPQ style mechanism at all?
Instead, it rolls back the entire statement and retries it from scratch.
That is not really any further from or closer to the standard than the
EPQ stuff, because the standard doesn't say anything about what should
happen as far as READ COMMITTED conflict handling goes.

My point here is that all of the stuff I'm talking about is only
relevant in READ COMMITTED mode, in areas where the standard never
provides us with guidance. If you can rely on SSI, then there is no
difference between what you propose and what I propose anyway, except
that what I propose is more general and will have better performance,
especially for batch MERGEs. If READ COMMITTED didn't exist,
implementing ON CONFLICT would have been more or less free of
controversy.

>Yes, that certainly will make an easier patch for MERGE.

Indeed, it will.

--
Peter Geoghegan

  52. Re: MERGE SQL Statement for PG11

    Nico Williams <nico@cryptonector.com> — 2017-11-02T22:00:35Z

    On Thu, Nov 02, 2017 at 02:05:19PM -0700, Peter Geoghegan wrote:
> Simon Riggs <simon@2ndquadrant.com> wrote:
> >So in your view we should make no attempt to avoid concurrent errors,
> >even when we have the capability to do so (in some cases) and doing so
> >would be perfectly compliant with the SQLStandard.
> 
> Yes. That's what I believe. I believe this because I can't see a way to
> do this that isn't a mess, and because ON CONFLICT DO UPDATE exists and
> works well for the cases where we can do better in READ COMMITTED mode.

A MERGE mapped to a DML like this:

  WITH
      updated AS (
        UPDATE <target>
        SET ...
        WHERE <condition>
        RETURNING <target>
    )
    , inserted AS (
        INSERT INTO <target>
        SELECT ...
        WHERE <key> NOT IN (SELECT <key> FROM updated) AND ..
        ON CONFLICT DO NOTHING -- see below!
        RETURNING <target>
    )
  DELETE FROM <target>
  WHERE <key> NOT IN (SELECT <key> FROM updated) AND
        <key> NOT IN (SELECT <key> FROM inserted) AND ...;

can handle concurrency via ON CONFLICT DO NOTHING in the INSERT CTE.

Now, one could write a MERGE that produces conflicts even without
concurrency, so adding ON CONFLICT DO NOTHING by default as above...
seems not-quite-correct.  But presumably one wouldn't write MERGE
statements that produce conflicts in the absence of concurrency, so this
seems close enough to me.

Another thing is that MERGE itself could get an ON CONFLICT clause for
the INSERT portion of the MERGE, allowing one to ignore some conflicts
and not others, though there would be no need for DO UPDATE, only DO
NOTHING for conflict resolution :)  This seems better.

I do believe this mapping is correct, and could be implemented entirely
in src/backend/parser/gram.y!  Am I wrong about this?

Nico
--

  53. Re: MERGE SQL Statement for PG11

    Peter Geoghegan <pg@bowt.ie> — 2017-11-02T22:25:48Z

    Nico Williams <nico@cryptonector.com> wrote:
>A MERGE mapped to a DML like this:
>
>  WITH
>      updated AS (
>        UPDATE <target>
>        SET ...
>        WHERE <condition>
>        RETURNING <target>
>    )
>    , inserted AS (
>        INSERT INTO <target>
>        SELECT ...
>        WHERE <key> NOT IN (SELECT <key> FROM updated) AND ..
>        ON CONFLICT DO NOTHING -- see below!
>        RETURNING <target>
>    )
>  DELETE FROM <target>
>  WHERE <key> NOT IN (SELECT <key> FROM updated) AND
>        <key> NOT IN (SELECT <key> FROM inserted) AND ...;
>

This is a bad idea. An implementation like this is not at all
maintainable.

>can handle concurrency via ON CONFLICT DO NOTHING in the INSERT CTE.

That's not handling concurrency -- it's silently ignoring an error. Who
is to say that the conflict that IGNORE ignored is associated with a row
visible to the MVCC snapshot of the statement? IOW, why should the DELETE
affect any row?

There are probably a great many reasons why you need a ModifyTable
executor node that keeps around state, and explicitly indicates that a
MERGE is a MERGE. For example, we'll probably want statement level
triggers to execute in a fixed order, regardless of the MERGE, RLS will
probably require explicitly knowledge of MERGE semantics, and so on.

FWIW, your example doesn't actually have a source (just a target), so it
isn't actually like MERGE.

-- 
Peter Geoghegan

  54. Re: MERGE SQL Statement for PG11

    Nico Williams <nico@cryptonector.com> — 2017-11-02T22:59:25Z

    On Thu, Nov 02, 2017 at 03:25:48PM -0700, Peter Geoghegan wrote:
> Nico Williams <nico@cryptonector.com> wrote:
> >A MERGE mapped to a DML like this:
> 
> This is a bad idea. An implementation like this is not at all
> maintainable.

Assuming the DELETE issue can be addressed, why would this not be
maintainable?

> >can handle concurrency via ON CONFLICT DO NOTHING in the INSERT CTE.
> 
> That's not handling concurrency -- it's silently ignoring an error. Who
> is to say that the conflict that IGNORE ignored is associated with a row
> visible to the MVCC snapshot of the statement? IOW, why should the DELETE
> affect any row?

Ah, yes, we'd have to make sure the DELETE does not delete rows that
could not be inserted.  There's... no way to find out what those would
have been -- RETURNING won't mention them, though it'd be a nice
addition to UPSERT to have a way to do that, and it'd make this mapping
feasible.

> There are probably a great many reasons why you need a ModifyTable
> executor node that keeps around state, and explicitly indicates that a
> MERGE is a MERGE. For example, we'll probably want statement level
> triggers to execute in a fixed order, regardless of the MERGE, RLS will
> probably require explicitly knowledge of MERGE semantics, and so on.

Wouldn't those fire anyways in a statement like the one I mentioned?

> FWIW, your example doesn't actually have a source (just a target), so it
> isn't actually like MERGE.

That can be added.  I was trying to keep it pithy.

Nico
--

  55. Re: MERGE SQL Statement for PG11

    Simon Riggs <simon@2ndquadrant.com> — 2017-11-03T07:35:07Z

    On 2 November 2017 at 17:06, Robert Haas <robertmhaas@gmail.com> wrote:
> On Tue, Oct 31, 2017 at 5:14 PM, Simon Riggs <simon@2ndquadrant.com> wrote:
>> I've proposed a SQL Standard compliant implementation that would do
>> much more than be new syntax over what we already have.
>>
>> So these two claims aren't accurate: "radical difference" and "syntax
>> sugar over a capability we have".
>
> I think those claims are pretty accurate.

No, because there is misunderstanding on some technical points.

They key point is at the end - what do we do next?

> The design of INSERT .. ON CONFLICT UPDATE and the supporting
> machinery only work if there is a unique index.  It provides radically
> different behavior than what you get with any other DML statement,
> with completely novel concurrency behavior, and there is no reasonable
> way to emulate that behavior in cases where no relevant unique index
> exists.

Agreed

> Therefore, if MERGE eventually uses INSERT .. ON CONFLICT
> UPDATE when a relevant unique index exists and does something else,
> such as your proposal of taking a strong lock, or Peter's proposal of
> doing this in a concurrency-oblivious manner, in other cases, then
> those two cases will behave very differently.

The *only* behavioural difference I have proposed would be the *lack*
of an ERROR in (some) concurrent cases.

We have a way to avoid some concurrency errors during MERGE, while
still maintaining exact SQL Standard behaviour. Peter has pointed out
that we could not catch errors in certain cases; that does nothing to
the cases where it will work well. It would be fallacious to imagine
it is an all or nothing situation.

> And if, in the meantime, MERGE can only handle the cases where there
> is a unique index,

I haven't proposed that MERGE would be forever limited to cases with a
unique index, only that MERGE-for-unique-indexes would be the version
in PG11, for good reason.

> then it can only handle the cases INSERT .. ON
> CONFLICT UPDATE can cover, which makes it, as far as I can see,
> syntactic sugar over what we already have.

SQL:2011 is greatly enhanced and this is no longer true; it was in
earlier versions but is not now.

MERGE allows you do DELETE, as well as conditional UPDATE and INSERT.
It is very clearly much more than UPSERT.

> Maybe it's not entirely -
> you might be planning to make some minor functional enhancements - but
> it's not clear what those are, and I feel like whatever it is could be
> done with less work and more elegance by just extending the INSERT ..
> ON CONFLICT UPDATE syntax.

The differences are clearly apparent in the Standard and in my submitted docs.

But I am interested in submitting a SQL Standard compliant feature.

> And it does seem to be your intention to only handle the cases which
> the INSERT .. ON CONFLICT UPDATE infrastructure can cover, because
> upthread you wrote this: "I didn't say it but my intention was to just
> throw an ERROR if no single unique index can be identified."  I don't
> think anybody's putting words into your mouth here.  We're just
> reading what you wrote.

Happy to have written it because that covers the common use case I was
hoping to deliver in PG11. Incremental development.

My proposal was to implement the common case, while avoiding
concurrency errors. Peter proposes that I cover both the common case
and more general cases, without avoiding concurrency errors.


All I have at the moment is that a few people disagree, but that
doesn't help determine the next action.

We seem to have a few options for PG11

1. Do nothing, we reject MERGE

2. Implement MERGE for unique index situations only, attempting to
avoid errors (Simon OP)

3. Implement MERGE, but without attempting to avoid concurrent ERRORs (Peter)

4. Implement MERGE, while attempting to avoid concurrent ERRORs in
cases where that is possible.

Stephen, Robert, please say which option you now believe we should pick.

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  56. Re: MERGE SQL Statement for PG11

    Simon Riggs <simon@2ndquadrant.com> — 2017-11-03T07:44:16Z

    On 2 November 2017 at 22:59, Nico Williams <nico@cryptonector.com> wrote:
> On Thu, Nov 02, 2017 at 03:25:48PM -0700, Peter Geoghegan wrote:
>> Nico Williams <nico@cryptonector.com> wrote:
>> >A MERGE mapped to a DML like this:
>>
>> This is a bad idea. An implementation like this is not at all
>> maintainable.
>
> Assuming the DELETE issue can be addressed, why would this not be
> maintainable?

It would only take one change to make this approach infeasible and
when that happened we would need to revert to the full-executor
version.

One difference that comes to mind is that MERGE doesn't behave the
same way as an UPDATE-join, according to SQL:2011 in that it must
throw an error if duplicate changes are requested. That would be hard
to emulate using a parser only version.

I would call it impressively clever but likely fragile, in this case,
though I encourage more ideas like that in the future.

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  57. Re: MERGE SQL Statement for PG11

    Thomas Kellerer <spam_eater@gmx.net> — 2017-11-03T07:46:20Z

    PMFJI

> We seem to have a few options for PG11
> 
> 1. Do nothing, we reject MERGE
> 
> 2. Implement MERGE for unique index situations only, attempting to
> avoid errors (Simon OP)
> 
> 3. Implement MERGE, but without attempting to avoid concurrent ERRORs
> (Peter)
> 
> 4. Implement MERGE, while attempting to avoid concurrent ERRORs in
> cases where that is possible.

From an end-users point of view I would prefer 3 (or 4 if that won't prevent
this from going into 11) 

INSERT ... ON CONFLICT is great, but there are situations where the
restrictions can get in the way and it would be nice to have an alternative
- albeit with some (documented) drawbacks. As far as I know Oracle also
doesn't guarantee that MERGE is safe for concurrent use - you can still wind
up with a unique key violation. 





--
Sent from: http://www.postgresql-archive.org/PostgreSQL-hackers-f1928748.html

  58. Re: MERGE SQL Statement for PG11

    Robert Haas <robertmhaas@gmail.com> — 2017-11-03T08:26:37Z

    On Fri, Nov 3, 2017 at 1:05 PM, Simon Riggs <simon@2ndquadrant.com> wrote:
>> Therefore, if MERGE eventually uses INSERT .. ON CONFLICT
>> UPDATE when a relevant unique index exists and does something else,
>> such as your proposal of taking a strong lock, or Peter's proposal of
>> doing this in a concurrency-oblivious manner, in other cases, then
>> those two cases will behave very differently.
>
> The *only* behavioural difference I have proposed would be the *lack*
> of an ERROR in (some) concurrent cases.

I think that's a big difference.  Error vs. non-error is a big deal by
itself; also, the non-error case involves departing from MVCC
semantics just as INSERT .. ON CONFLICT UPDATE does.

> All I have at the moment is that a few people disagree, but that
> doesn't help determine the next action.
>
> We seem to have a few options for PG11
>
> 1. Do nothing, we reject MERGE
>
> 2. Implement MERGE for unique index situations only, attempting to
> avoid errors (Simon OP)
>
> 3. Implement MERGE, but without attempting to avoid concurrent ERRORs (Peter)
>
> 4. Implement MERGE, while attempting to avoid concurrent ERRORs in
> cases where that is possible.
>
> Stephen, Robert, please say which option you now believe we should pick.

I think Peter has made a good case for #3, so I lean toward that
option.  I think #4 is too much of a non-obvious behavior difference
between the cases where we can avoid those errors and the cases where
we can't, and I don't see where #2 can go in the future other than #4.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  59. Re: MERGE SQL Statement for PG11

    Simon Riggs <simon@2ndquadrant.com> — 2017-11-03T09:00:29Z

    On 3 November 2017 at 07:46, Thomas Kellerer <spam_eater@gmx.net> wrote:
> PMFJI
>
>> We seem to have a few options for PG11
>>
>> 1. Do nothing, we reject MERGE
>>
>> 2. Implement MERGE for unique index situations only, attempting to
>> avoid errors (Simon OP)
>>
>> 3. Implement MERGE, but without attempting to avoid concurrent ERRORs
>> (Peter)
>>
>> 4. Implement MERGE, while attempting to avoid concurrent ERRORs in
>> cases where that is possible.
>
> From an end-users point of view I would prefer 3 (or 4 if that won't prevent
> this from going into 11)

Sounds reasonable approach.

> INSERT ... ON CONFLICT is great, but there are situations where the
> restrictions can get in the way and it would be nice to have an alternative
> - albeit with some (documented) drawbacks. As far as I know Oracle also
> doesn't guarantee that MERGE is safe for concurrent use - you can still wind
> up with a unique key violation.

Yes, Oracle allows some unique key violations. It's clear that we
would need to allow some also. So we clearly can't infer whether they
avoid some errors just because they allow some.

My approach will be to reduce the errors in the best way, not to try
to copy errors Oracle makes, if any. But that error avoidance can
easily be a later add-on if we prefer it that way.

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  60. Re: MERGE SQL Statement for PG11

    Stephen Frost <sfrost@snowman.net> — 2017-11-03T11:07:25Z

    * Robert Haas (robertmhaas@gmail.com) wrote:
> On Fri, Nov 3, 2017 at 1:05 PM, Simon Riggs <simon@2ndquadrant.com> wrote:
> > We seem to have a few options for PG11
> >
> > 1. Do nothing, we reject MERGE
> >
> > 2. Implement MERGE for unique index situations only, attempting to
> > avoid errors (Simon OP)
> >
> > 3. Implement MERGE, but without attempting to avoid concurrent ERRORs (Peter)
> >
> > 4. Implement MERGE, while attempting to avoid concurrent ERRORs in
> > cases where that is possible.
> >
> > Stephen, Robert, please say which option you now believe we should pick.
> 
> I think Peter has made a good case for #3, so I lean toward that
> option.  I think #4 is too much of a non-obvious behavior difference
> between the cases where we can avoid those errors and the cases where
> we can't, and I don't see where #2 can go in the future other than #4.

Agreed.

Thanks!

Stephen

  61. Re: MERGE SQL Statement for PG11

    Simon Riggs <simon@2ndquadrant.com> — 2017-11-03T14:16:03Z

    On 3 November 2017 at 08:26, Robert Haas <robertmhaas@gmail.com> wrote:
> On Fri, Nov 3, 2017 at 1:05 PM, Simon Riggs <simon@2ndquadrant.com> wrote:
>>> Therefore, if MERGE eventually uses INSERT .. ON CONFLICT
>>> UPDATE when a relevant unique index exists and does something else,
>>> such as your proposal of taking a strong lock, or Peter's proposal of
>>> doing this in a concurrency-oblivious manner, in other cases, then
>>> those two cases will behave very differently.
>>
>> The *only* behavioural difference I have proposed would be the *lack*
>> of an ERROR in (some) concurrent cases.
>
> I think that's a big difference.  Error vs. non-error is a big deal by
> itself;

Are you saying avoiding an ERROR is a bad or good thing?

> also, the non-error case involves departing from MVCC
> semantics just as INSERT .. ON CONFLICT UPDATE does.

Meaning what exactly? What situation occurs that a user would be concerned with?

Please describe exactly what you mean so we get it clear.

The concurrent behaviour for MERGE is allowed to be
implementation-specific, so we can define it any way we want.

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  62. Re: MERGE SQL Statement for PG11

    Peter Geoghegan <pg@bowt.ie> — 2017-11-03T16:35:39Z

    Simon Riggs <simon@2ndquadrant.com> wrote:
>>> The *only* behavioural difference I have proposed would be the *lack*
>>> of an ERROR in (some) concurrent cases.
>>
>> I think that's a big difference.  Error vs. non-error is a big deal by
>> itself;
>
>Are you saying avoiding an ERROR is a bad or good thing?

Are you really asking Robert to repeat what has already been said about
a dozen different ways?

That's *not* the only difference. You need to see a couple of steps
ahead to see further differences, as the real dilemma comes when you
have to reconcile having provided the UPSERT-guarantees with cases that
that doesn't map on to (which can happen in a number of different ways).

I don't understand why you'll talk about just about anything but that.
This is a high-level concern about the overarching design. Do you really
not understand the concern at this point?

>> also, the non-error case involves departing from MVCC
>> semantics just as INSERT .. ON CONFLICT UPDATE does.
>
>Meaning what exactly? What situation occurs that a user would be concerned with?
>
>Please describe exactly what you mean so we get it clear.
>
>The concurrent behaviour for MERGE is allowed to be
>implementation-specific, so we can define it any way we want.

Agreed -- we can. It isn't controversial at all to say that the SQL
standard has nothing to say on this question. The problem is that the
semantics you argue for are ill-defined, and seem to create more
problems than they solve. Why keep bringing up the SQL standard?

-- 
Peter Geoghegan

  63. Re: MERGE SQL Statement for PG11

    Simon Riggs <simon@2ndquadrant.com> — 2017-11-06T17:35:13Z

    On 3 November 2017 at 16:35, Peter Geoghegan <pg@bowt.ie> wrote:
> Simon Riggs <simon@2ndquadrant.com> wrote:
>>>>
>>>> The *only* behavioural difference I have proposed would be the *lack*
>>>> of an ERROR in (some) concurrent cases.
>>>
>>>
>>> I think that's a big difference.  Error vs. non-error is a big deal by
>>> itself;
>>
>>
>> Are you saying avoiding an ERROR is a bad or good thing?
>
>
> Are you really asking Robert to repeat what has already been said about
> a dozen different ways?

I'm asking for clarity of explanation rather than assertions.


> That's *not* the only difference. You need to see a couple of steps
> ahead to see further differences, as the real dilemma comes when you
> have to reconcile having provided the UPSERT-guarantees with cases that
> that doesn't map on to (which can happen in a number of different ways).
>
> I don't understand why you'll talk about just about anything but that.
> This is a high-level concern about the overarching design. Do you really
> not understand the concern at this point?

You're either referring to what is in the docs, which is INSERT ... ON
CONFLICT violates MVCC in a particular way, or something as yet
unstated. If it is the former, then I still don't see the problem (see
later). If it is the latter, I need more. So either way I need more.

> Robert Haas said
>In the past, there have been objections to implementations of MERGE
>which would give rise to such serialization anomalies, but I'm not
>sure we should feel bound by those discussions.  One thing that's
>different is that the common and actually-useful case can now be made
>to work in a fairly satisfying way using INSERT .. ON CONFLICT UPDATE;
>if less useful cases are vulnerable to some weirdness, maybe it's OK
>to just document the problems.

I agreed with that, and still do.


We need a clear, explicit description of this situation so I will
attempt that in detail here.


The basic concurrency problem we have is this

APPROACH1
1. Join to produce results based upon snapshot at start of query
2. Apply results for INSERT, UPDATE or DELETE

Given there is a time delay between 1 and 2 there is a race condition
so that if another user concurrently inserts the same value into a
unique index then an INSERT will fail with a uniqueness violation.

Such failures are of great concern in practice because the time
between 1 and 2 could be very long for large statements, or for
smaller statements we might have sufficiently high concurrency to
allow us to see regular failures.

APPROACH2 (modified from my original proposal slightly)
1. Join...
2. Apply results for UPDATE, if present not visible via the snapshot
taken at 1, do EPQ to ensure we locate current live tuple
3. If still not visible, do speculative insertion if we have a unique
index available, otherwise ERROR. If spec insertion fails, go to 2

The loop created above can live-lock, meaning that an infinite loop
could be created.

In practice, such live-locks are rare and we could detect them by
falling out of the loop after a few tries. Approach2's purpose is to
alleviate errors in Approach1, so falling out of the loop merely takes
us back to the error we would have got if we didn't try, so Approach2
has considerable benefit over Approach1. This only applies if we do an
INSERT, so if there is a WHEN NOT MATCHED ... AND clause with
probability W, that makes the INSERT rare then we simply have the
probablility of error in Approach2 approach the probability of error
in Approach1 as the W drops to zero, but with W high we may avoid many
errors. Approach2 never generates more errors than Approach1.

I read that step 3 in Approach2 is some kind of problem in MVCC
semantics. My understanding is that SQL Standard allows us to define
what the semantics of the statement are in relation to concurrency, so
any semantic issue can be handled by defining it to work the way we
want. The semantics are:
a) when a unique index is available we avoid errors by using semantics
of INSERT .. ON CONFLICT UPDATE.
b) when a unique index is not available we use other semantics.
To me this is the same as INSERTs failing in the presence of unique
indexes, but not failing when no index is present. The presence of a
unique constraint alters the semantics of the query.
We can choose Approach2 - as Robert says "[we should not] feel bound
by those [earlier] discussions"

Please explain what is wrong with the above without merely asserting
there is a problem.

As you point out, whichever we choose, we will be bound by those
semantics. So if we take Approach1, as has been indicated currently,
what is the written explanation for that, so we can show that to the
people who ask in the future about our decisions?

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  64. Re: MERGE SQL Statement for PG11

    Peter Geoghegan <pg@bowt.ie> — 2017-11-06T18:35:37Z

    Simon Riggs <simon@2ndquadrant.com> wrote:
>APPROACH1
>1. Join to produce results based upon snapshot at start of query
>2. Apply results for INSERT, UPDATE or DELETE

>Such failures are of great concern in practice because the time
>between 1 and 2 could be very long for large statements, or for
>smaller statements we might have sufficiently high concurrency to
>allow us to see regular failures.

I'm not sure that they're a *great* concern in a world with something
that targets UPSERT use cases, which is a situation that does not exist
in DBMSs with MERGE (with the notable exception of Teradata). But it's
clearly a concern that users may expect to avoid duplicate violations in
READ COMMITTED, since this caused confusion among users of other
database systems with MERGE.

>APPROACH2 (modified from my original proposal slightly)

This write-up actually begins to confront the issues that I've raised.
I'm glad to see this.

>1. Join...
>2. Apply results for UPDATE, if present not visible via the snapshot
>taken at 1, do EPQ to ensure we locate current live tuple
>3. If still not visible, do speculative insertion if we have a unique
>index available, otherwise ERROR. If spec insertion fails, go to 2
>
>The loop created above can live-lock, meaning that an infinite loop
>could be created.

The loop is *guaranteed* to live-lock once you "goto 2". So you might as
well just throw an error at that point, which is the behavior that I've
been arguing for all along!

If this isn't guaranteed to live-lock at "goto 2", then it's not clear
why. The outcome of step 2 is clearly going to be identical if you don't
acquire a new MVCC snapshot, but you don't address that.

You might have meant "apply an equivalent ON CONFLICT DO UPDATE", or
something like that, despite the fact that the use of ON CONFLICT DO
NOTHING was clearly implied by the "goto 2". I also see problems with
that, but I'll wait for you to clarify what you meant before going into
what they are.

>In practice, such live-locks are rare and we could detect them by
>falling out of the loop after a few tries. Approach2's purpose is to
>alleviate errors in Approach1, so falling out of the loop merely takes
>us back to the error we would have got if we didn't try, so Approach2
>has considerable benefit over Approach1.

I don't hate the idea of retrying a fixed number of times for things
like this, but I don't like it either. I'm going to assume that it's
fine for now.

>I read that step 3 in Approach2 is some kind of problem in MVCC
>semantics. My understanding is that SQL Standard allows us to define
>what the semantics of the statement are in relation to concurrency, so
>any semantic issue can be handled by defining it to work the way we
>want.

My only concern is that our choices here should be good ones, based on
practical considerations. We both more or less agree on how this should
be assessed, I think; we just reach different conclusions.

>As you point out, whichever we choose, we will be bound by those
>semantics. So if we take Approach1, as has been indicated currently,
>what is the written explanation for that, so we can show that to the
>people who ask in the future about our decisions?

Well, Approach1 is what other systems implement. I think that it would
be important to point out that MERGE with Approach1 isn't special, but
ON CONFLICT DO UPDATE is special. We'd also say that higher isolation
levels will not have duplicate violations.

--
Peter Geoghegan

  65. Re: MERGE SQL Statement for PG11

    Simon Riggs <simon@2ndquadrant.com> — 2017-11-06T20:23:20Z

    On 6 November 2017 at 18:35, Peter Geoghegan <pg@bowt.ie> wrote:

>> APPROACH2 (modified from my original proposal slightly)
>
>
> This write-up actually begins to confront the issues that I've raised.
> I'm glad to see this.
>
>> 1. Join...
>> 2. Apply results for UPDATE, if present not visible via the snapshot
>> taken at 1, do EPQ to ensure we locate current live tuple
>> 3. If still not visible, do speculative insertion if we have a unique
>> index available, otherwise ERROR. If spec insertion fails, go to 2
>>
>> The loop created above can live-lock, meaning that an infinite loop
>> could be created.
>
>
> The loop is *guaranteed* to live-lock once you "goto 2". So you might as
> well just throw an error at that point, which is the behavior that I've
> been arguing for all along!
>
> If this isn't guaranteed to live-lock at "goto 2", then it's not clear
> why. The outcome of step 2 is clearly going to be identical if you don't
> acquire a new MVCC snapshot, but you don't address that.
>
> You might have meant "apply an equivalent ON CONFLICT DO UPDATE", or
> something like that, despite the fact that the use of ON CONFLICT DO
> NOTHING was clearly implied by the "goto 2". I also see problems with
> that, but I'll wait for you to clarify what you meant before going into
> what they are.

In step 3 we discover that an entry exists in the index for a committed row.

Since we have a unique index we use it to locate the row we know
exists and UPDATE that.

We don't use a new MVCC snapshot, we do what EPQ does. EPQ is already
violating MVCC for UPDATEs, so why does it matter if we do it for
INSERTs also?


Where hides the problem?

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  66. Re: MERGE SQL Statement for PG11

    Peter Geoghegan <pg@bowt.ie> — 2017-11-06T21:50:03Z

    Simon Riggs <simon@2ndquadrant.com> wrote:
>In step 3 we discover that an entry exists in the index for a committed row.
>
>Since we have a unique index we use it to locate the row we know
>exists and UPDATE that.
>
>We don't use a new MVCC snapshot, we do what EPQ does. EPQ is already
>violating MVCC for UPDATEs, so why does it matter if we do it for
>INSERTs also?

Before I go on to say why I think that this approach is problematic, I
want to point out a few things that I think we actually agree on:

* EPQ is fairly arbitrary as a behavior for READ COMMITTED UPDATE
  conflict handling. It has more to do with how VACUUM works than about
  some platonic ideal that everyone agrees on.

* We can imagine other alternatives, such as the behavior in Oracle
  (statement level rollback + optimistic retry).

* Those alternatives are probably better in some ways but worse in other
  ways.

* EPQ violates snapshot consistency, even though that's not inherently
  necessary to avoid "READ COMMITTED serialization errors".

* ON CONFLICT also violates snapshot consistency, in rather a different
  way. (Whether or not this is necessary is more debatable.)

I actually think that other MVCC systems don't actually copy Oracle here,
either, and for similar pragmatic reasons. It's a mixed bag.

>Where hides the problem?

The problem is violating MVCC is something that can be done in different
ways, and by meaningful degrees:

* EPQ semantics are believed to be fine because we don't get complaints
  about it. I think that that's because it's specialized to UPDATEs and
  UPDATE-like operations, where we walk an UPDATE chain specifically,
  and only use a dirty snapshot for the chain's newer tuples.

* ON CONFLICT doesn't care about UPDATE chains. Unlike EPQ, it makes no
  distinction between a concurrent UPDATE, and a concurrent DELETE + fresh
  INSERT. It's specialized to CONFLICTs.

This might seem abstract, but it has real, practical implications.
Certain contradictions exist when you start with MVCC semantics, then
fall back to EPQ semantics, then finally fall back to ON CONFLICT
semantics.

Questions about mixing these two things:

* What do we do if someone concurrently UPDATEs in a way that makes the
  qual not pass during EPQ traversal? Should we INSERT when that
  happens?

* If so, what about the case when the MERGE join qual/unique index
  values didn't change (just some other attributes that do not pass the
  additional WHEN MATCHED qual)?

* What about when there was a concurrent DELETE -- should we INSERT then?

ON CONFLICT goes from a CONFLICT, and then applies its own qual. That's
hugely different to doing it the other way around: starting from your
own MVCC snapshot qual, and going to a CONFLICT. This is because
evaluating the DO UPDATE's WHERE clause is just one little extra step
after the one and only latest row for that value has been locked.  You
could theoretically go this way with 2PL, I think, because that's a bit
like locking every row that the predicate touches, but of course that
isn't at all practical.

I should stop trying to make a watertight case against this, even though
I still think that's possible. For now, instead, I'll just say that this
is *extremely* complicated, and still has unresolved questions about
semantics.

-- 
Peter Geoghegan

  67. Re: MERGE SQL Statement for PG11

    Geoff Winkless <pgsqladmin@geoff.dj> — 2017-11-07T16:07:42Z

    On 6 November 2017 at 17:35, Simon Riggs <simon@2ndquadrant.com> wrote:
> I read that step 3 in Approach2 is some kind of problem in MVCC
> semantics. My understanding is that SQL Standard allows us to define
> what the semantics of the statement are in relation to concurrency, so
> any semantic issue can be handled by defining it to work the way we
> want. The semantics are:
> a) when a unique index is available we avoid errors by using semantics
> of INSERT .. ON CONFLICT UPDATE.
> b) when a unique index is not available we use other semantics.

I'm obviously being obtuse.

If a unique index is not available, then surely there won't _be_ a
failure? The INSERT (or indeed UPDATE) that results in two similar
records will simply happen, and you will end up with two records the
same. That's OK, based on the semantics of MERGE, no? At the
transaction-start INSERT was the correct thing to do.

Geoff

  68. Re: MERGE SQL Statement for PG11

    Nico Williams <nico@cryptonector.com> — 2017-11-07T23:29:22Z

    On Thu, Nov 02, 2017 at 03:25:48PM -0700, Peter Geoghegan wrote:
> Nico Williams <nico@cryptonector.com> wrote:
> >A MERGE mapped to a DML like this:

I needed to spend more time reading MERGE docs from other RDBMSes.

The best MERGE so far is MS SQL Server's, which looks like:

  MERGE INTO <target> <target_alias>
  USING <source> <source_alias>
  ON (<join condition>)
  
  -- optional:
  WHEN MATCHED THEN UPDATE SET ...

  -- optional:
  WHEN NOT MATCHED [ BY TARGET ] THEN INSERT ...

  -- optional:
  WHEN NOT MATCHED BY SOURCE THEN DELETE

  -- optional:
  OUTPUT ...
  
  ;

(The other MERGEs are harder to use because they lack a WHEN NOT MATCHED
BY SOURCE THEN DELETE, instead having a DELETE clause on the UPDATE,
which is then difficult to use.)

This is *trivial* to map to a CTE, and, in fact, I and my colleagues
have resorted to hand-coded CTEs like this precisely because PG lacks
MERGE (though we ourselves didn't know about MERGE -- it's new to us).

If <source> is a query, then we start with a CTE for that, else if it's
a view or table, then we don't setup a CTE for it.  Each of the UPDATE,
INSERT, and/or DELETE can be it's own CTE.  If there's an OUTPUT clause,
that can be a final SELECT that queries from the CTEs that ran the DMLs
with RETURNING.  If there's no OUTPUT then none of the DMLs need to have
RETURNING, and one of them will be the main statement, rather than a
CTE.

The pattern is:

  WITH
    -- IFF <source> is a query:
    <source_alias> AS (<source>),

    -- IFF there's a WHEN MATCHED THEN UPDATE
    updates AS (
      UPDATE <target> AS <target_alias> SET ...
      FROM <source>
      WHERE <join_condition>
      -- IFF there's an OUTPUT clause, then:
      RETURNING 'update' as "@action", ...
    ),

    inserts AS (
      INSERT INTO <target> (<column_list>)
      SELECT ...
      FROM <source>
      LEFT JOIN <target> ON <join_condition>
      WHERE <target> IS NOT DISTINCT FROM NULL
      -- IFF there's a CONCURRENTLY clause:
      ON CONFLICT DO NOTHING
      -- IFF there's an OUTPUT clause, then:
      RETURNING 'insert' as "@action", ...
    ),

    deletes AS (
      DELETE FROM <target>
      WHERE NOT EXISTS (SELECT * FROM <source> WHERE <join_condition>)
      -- IFF there's an OUTPUT clause, then:
      RETURNING 'delete' as "@action", ...
    ),

  -- IFF there's an OUTPUT clause
  SELECT * FROM updates
  UNION
  SELECT * FROM inserts
  UNION
  SELECT * FROM deletes;

If there's not an output clause then one of the DMLs has to be the main
statement:

  WITH ...
  DELETE ...; -- or UPDATE, or INSERT

Note that if the source is a view or table and there's no OUTPUT clause,
then it's one DML with up to (but not referring to) two CTEs, and in all
cases the CTEs do not refer to each other.  This means that the executor
can parallelize all of the DMLs.

If the source is a query, then that could be made a temp view to avoid
having to run the query first.  The CTE executor needs to learn to
sometimes do this anyways, so this is good.

The <deletes> CTE can be equivalently written without a NOT EXISTS:

    to_be_deleted AS (
      SELECT <target>
      FROM <target>
      LEFT JOIN <source> ON (<join_condition>)
      WHERE <source> IS NOT DISTINCT FROM NULL
    ),
    deletes AS (
      DELETE FROM <target>
      USING to_be_deleted tbd
      WHERE <target> = <tbd>
    )

if that were to run faster (probably not, since PG today would first run
the to_be_deleted CTE, then the deletes CTE).  I mention only because
it's nice to see the symmetry of LEFT JOINs for the two WHEN NOT MATCHED
cases.

(Here <source> is the alias for it if one was given.)

***

This mapping triggers triggers as one would expect (at least FOR EACH
ROW; I expect the DMLs in CTEs should also trigger FOR EACH STATEMENT
triggers, and if they don't I consider that a bug).

> This is a bad idea. An implementation like this is not at all
> maintainable.

I beg to differ.  First of all, not having to add an executor for MERGE
is a win: much, much less code to maintain.  The code to map MERGE to
CTEs can easily be contained entirely in src/backend/parser/gram.y,
which is a maintainability win: any changes to how CTEs are compiled
will fail to compile if they break the MERGE mapping to CTEs.

> >can handle concurrency via ON CONFLICT DO NOTHING in the INSERT CTE.
> 
> That's not handling concurrency -- it's silently ignoring an error. Who
> is to say that the conflict that IGNORE ignored is associated with a row
> visible to the MVCC snapshot of the statement? IOW, why should the DELETE
> affect any row?

That was me misunderstanding MERGE.  The DELETE is independent of the
INSERT -- if an INSERT does nothing because of an ON CONFLICT DO
NOTHING clause, then that won't cause that row to be deleted -- the
inserts and deletes CTEs are independent in the latest mapping (see
above).

I believe adding ON CONFLICT DO NOTHING to the INSERT in this mapping is
all that's needed to support concurrency.

> There are probably a great many reasons why you need a ModifyTable
> executor node that keeps around state, and explicitly indicates that a
> MERGE is a MERGE. For example, we'll probably want statement level
> triggers to execute in a fixed order, regardless of the MERGE, RLS will
> probably require explicitly knowledge of MERGE semantics, and so on.

Let's take those examples one at a time:

 - Is there a reason to believe that MERGE could not parallelize the
   DMLs it implies?

   If they can be parallelized, then we should not define the order in
   which the corresponding triggers fire.

   Surely we want to leave that possibility (parallelization) open,
   rather than exclude it.

   The user should not depend on the order in which the FOR EACH
   STATEMENT and FOR EACH ROW triggers will fire.  They can always check
   at the end of the transaction with DEFERRED triggers (see also my
   patch for ALWAYS DEFERRED constraints and triggers).

   AFTER <op> FOR EACH STATEMENT triggers will only run after all the
   corresponding DMLs in the mapping have completed, but their relative
   orders should still not be defined.

 - I don't see how RLS isn't entirely orthogonal.

   RLS would (does) apply as normal to all of the DMLs in the mapping.

   If that was not the case, then there'd be a serious bug in PG right
   now!  Using a CTE must *not* disable RLS.

   FOR UPDATE RLS policies are broken, however, since they don't get to
   see the OLD and NEW values.  But that's orthogonal here.

> FWIW, your example doesn't actually have a source (just a target), so it
> isn't actually like MERGE.

That was my mistake -- as I say above, I had to spend more time with the
various RDBMSes' MERGE docs.

Nico
--

  69. Re: MERGE SQL Statement for PG11

    Peter Geoghegan <pg@bowt.ie> — 2017-11-07T23:31:22Z

    On Tue, Nov 7, 2017 at 3:29 PM, Nico Williams <nico@cryptonector.com> wrote:
> On Thu, Nov 02, 2017 at 03:25:48PM -0700, Peter Geoghegan wrote:
>> Nico Williams <nico@cryptonector.com> wrote:
>> >A MERGE mapped to a DML like this:
>
> I needed to spend more time reading MERGE docs from other RDBMSes.

Please don't hijack this thread. It's about the basic question of
semantics, and is already hard enough for others to follow as-is.

-- 
Peter Geoghegan

  70. Re: MERGE SQL Statement for PG11

    Nico Williams <nico@cryptonector.com> — 2017-11-08T00:45:52Z

    On Tue, Nov 07, 2017 at 03:31:22PM -0800, Peter Geoghegan wrote:
> On Tue, Nov 7, 2017 at 3:29 PM, Nico Williams <nico@cryptonector.com> wrote:
> > On Thu, Nov 02, 2017 at 03:25:48PM -0700, Peter Geoghegan wrote:
> >> Nico Williams <nico@cryptonector.com> wrote:
> >> >A MERGE mapped to a DML like this:
> >
> > I needed to spend more time reading MERGE docs from other RDBMSes.
> 
> Please don't hijack this thread. It's about the basic question of
> semantics, and is already hard enough for others to follow as-is.

I'm absolutely not.  If you'd like a pithy summary devoid of detail, it
is this:

  I'm making the argument that using ON CONFLICT to implement MERGE
  cannot produce a complete implementation [you seem to agree], but
  there is at least one light-weight way to implement MERGE with
  _existing_ machinery in PG: CTEs.

  It's perfectly fine to implement an executor for MERGE, but I think
  that's a bit silly and I explain why.

Further, I explored your question regarding order of events, which you
(and I) think is a very important semantics question.  You thought order
of execution / trigger firing should be defined, whereas I think it
should not because MERGE explicitly says, at least MSFT's!

MSFT's MERGE says:

| For every insert, update, or delete action specified in the MERGE
| statement, SQL Server fires any corresponding AFTER triggers defined
| on the target table, but does not guarantee on which action to fire
| triggers first or last. Triggers defined for the same action honor the
| order you specify.

Impliedly (though not stated explicitly), the actual updates, inserts,
and deletes, can happen in any order as well as the triggers firing in
any order.

As usual, in the world of programming language design, leaving order of
execution undefined as much as possible increases the level of available
opportunities to parallelize.  Presumably MSFT is leaving the door open
to parallizing MERGE, if they haven't already.

Impliedly, CTEs that have no dependencies on each other are also ripe
for parallelization.  This is important too!  For one of my goals is: to
improve CTE performance.  If implementing MERGE as a mapping to CTEs
leads to improvements in CTEs, so much the better.  But also this *is* a
simple implementation of MERGE, and simplicity seems like a good thing.

Nico
--

  71. Re: MERGE SQL Statement for PG11

    Nico Williams <nico@cryptonector.com> — 2017-11-08T05:49:47Z

    Ah, there is one reason not to use a mapping to CTEs to implement MERGE:
it might be faster to use a single query that is a FULL OUTER JOIN of the
source and target to drive the update/insert/delete operations.

  72. Re: [HACKERS] MERGE SQL Statement for PG11

    Simon Riggs <simon@2ndquadrant.com> — 2017-11-14T19:02:54Z

    On 6 November 2017 at 16:50, Peter Geoghegan <pg@bowt.ie> wrote:

>> Where hides the problem?
>
>
> The problem is violating MVCC is something that can be done in different
> ways, and by meaningful degrees:
>
> * EPQ semantics are believed to be fine because we don't get complaints
>  about it. I think that that's because it's specialized to UPDATEs and
>  UPDATE-like operations, where we walk an UPDATE chain specifically,
>  and only use a dirty snapshot for the chain's newer tuples.
>
> * ON CONFLICT doesn't care about UPDATE chains. Unlike EPQ, it makes no
>  distinction between a concurrent UPDATE, and a concurrent DELETE + fresh
>  INSERT. It's specialized to CONFLICTs.
>
> This might seem abstract, but it has real, practical implications.
> Certain contradictions exist when you start with MVCC semantics, then
> fall back to EPQ semantics, then finally fall back to ON CONFLICT
> semantics.
>
> Questions about mixing these two things:
>
> * What do we do if someone concurrently UPDATEs in a way that makes the
>  qual not pass during EPQ traversal? Should we INSERT when that
>  happens?
>
> * If so, what about the case when the MERGE join qual/unique index
>  values didn't change (just some other attributes that do not pass the
>  additional WHEN MATCHED qual)?
>
> * What about when there was a concurrent DELETE -- should we INSERT then?
>
> ON CONFLICT goes from a CONFLICT, and then applies its own qual. That's
> hugely different to doing it the other way around: starting from your
> own MVCC snapshot qual, and going to a CONFLICT. This is because
> evaluating the DO UPDATE's WHERE clause is just one little extra step
> after the one and only latest row for that value has been locked.  You
> could theoretically go this way with 2PL, I think, because that's a bit
> like locking every row that the predicate touches, but of course that
> isn't at all practical.
>
> I should stop trying to make a watertight case against this, even though
> I still think that's possible. For now, instead, I'll just say that this
> is *extremely* complicated, and still has unresolved questions about
> semantics.

That's a good place to leave this for now - we're OK to make progress
with the main feature, and we have some questions to be addressed once
we have a cake to decorate.

Thanks for your input.

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  73. Re: [HACKERS] MERGE SQL Statement for PG11

    Peter Geoghegan <pg@bowt.ie> — 2017-11-14T19:37:58Z

    On Tue, Nov 14, 2017 at 11:02 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
> That's a good place to leave this for now - we're OK to make progress
> with the main feature, and we have some questions to be addressed once
> we have a cake to decorate.
>
> Thanks for your input.

Thanks for listening. I regret that it became heated, but I'm glad
that we now understand each other's perspective. I'm also glad that
you're pushing ahead with MERGE as a project, because MERGE is
certainly a compelling feature.

-- 
Peter Geoghegan

  74. Re: [HACKERS] MERGE SQL Statement for PG11

    Simon Riggs <simon@2ndquadrant.com> — 2017-12-30T11:01:31Z

    On 27 October 2017 at 13:45, Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
> Simon Riggs wrote:
>
>> Earlier thoughts on how this could/could not be done were sometimes
>> imprecise or inaccurate, so I have gone through the command per
>> SQL:2011 spec and produced a definitive spec in the form of an SGML
>> ref page. This is what I intend to deliver for PG11.
>
> Nice work.  I didn't verify the SQL spec, just read your HTML page;
> some very minor comments based on that:
>
> * use "and" not "where" as initial words in "when_clause" and
>   "merge_update" clause definitions
>
> * missing word here: "the DELETE privilege on the if you specify"
>
> * I think the word "match." is leftover from some editing in the phrase
>   " that specifies which rows in the data_source match rows in the
>   target_table_name. match."  In the same paragraph, it is not clear
>   whether all columns must be matched or it can be a partial match.

Thanks for these review points, I have included them.

> * In the when_clause note, it is not clear whether you can have multiple
>   WHEN MATCHED and WHEN NOT MATCHED clauses.  Obviously you can have one
>   of each, but I think your doc says it is possible to have more than one of
>   each, with different conditions (WHEN MATCHED AND foo THEN bar WHEN
>   MATCHED AND baz THEN qux).  No example shows more than one.
>
>   On the same point: Is there short-circuiting of such conditions, i.e.
>   will the execution will stop looking for further WHEN matches if some
>   rule matches, or will it rather check all rules and raise an error if
>   more than one WHEN rules match each given row?

Docs rewritten to better explain.

> * Your last example uses ELSE but that appears nowhere in the synopsys.

This last has been removed.

New version of HTML docs attached for easy reading.


Attached: MERGE patch is now MOSTLY complete, but still WIP.

Patch works sufficiently well to take data from source and use it
correctly against target, for the DELETE operation and INSERT DEFAULT
VALUES. Patch also includes PL/pgSQL changes.

Patch has full set of docs and tests, but does not yet pass all tests.
UPDATE and INSERT are not yet working because I've chosen to leave
targetist handling until last, which is still WIP. The patch doesn't
crash, but does not yet work for those subcommands - though I haven't
prevented it from executing those subcommands.

Patch uses mechanism as agreed previously with Peter G et al. on this thread.

SUMMARY
Works
* EXPLAIN
* DELETE actions
* DO NOTHING actions
* PL/pgSQL
* Triggers for row and statement

Not yet working
* Execute UPDATE actions
* Execute INSERT actions
* EvalPlanQual
* No isolation tests yet
* RLS
* Partitioning

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  75. Re: [HACKERS] MERGE SQL Statement for PG11

    Robert Haas <robertmhaas@gmail.com> — 2018-01-04T17:29:58Z

    On Sat, Dec 30, 2017 at 6:01 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
> Patch uses mechanism as agreed previously with Peter G et al. on this thread.

I'm not sure that an agreement was reached, or what the substance of
that agreement was.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  76. Re: [HACKERS] MERGE SQL Statement for PG11

    Simon Riggs <simon@2ndquadrant.com> — 2018-01-04T17:38:16Z

    On 4 January 2018 at 17:29, Robert Haas <robertmhaas@gmail.com> wrote:
> On Sat, Dec 30, 2017 at 6:01 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
>> Patch uses mechanism as agreed previously with Peter G et al. on this thread.
>
> I'm not sure that an agreement was reached, or what the substance of
> that agreement was.

I refer to this... and confirm I have implemented option 3

On 3 November 2017 at 11:07, Stephen Frost <sfrost@snowman.net> wrote:
> * Robert Haas (robertmhaas@gmail.com) wrote:
>> On Fri, Nov 3, 2017 at 1:05 PM, Simon Riggs <simon@2ndquadrant.com> wrote:
>> > We seem to have a few options for PG11
>> >
>> > 1. Do nothing, we reject MERGE
>> >
>> > 2. Implement MERGE for unique index situations only, attempting to
>> > avoid errors (Simon OP)
>> >
>> > 3. Implement MERGE, but without attempting to avoid concurrent ERRORs (Peter)
>> >
>> > 4. Implement MERGE, while attempting to avoid concurrent ERRORs in
>> > cases where that is possible.
>> >
>> > Stephen, Robert, please say which option you now believe we should pick.
>>
>> I think Peter has made a good case for #3, so I lean toward that
>> option.  I think #4 is too much of a non-obvious behavior difference
>> between the cases where we can avoid those errors and the cases where
>> we can't, and I don't see where #2 can go in the future other than #4.
>
> Agreed.

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  77. Re: [HACKERS] MERGE SQL Statement for PG11

    Robert Haas <robertmhaas@gmail.com> — 2018-01-04T17:40:34Z

    On Thu, Jan 4, 2018 at 12:38 PM, Simon Riggs <simon@2ndquadrant.com> wrote:
> On 4 January 2018 at 17:29, Robert Haas <robertmhaas@gmail.com> wrote:
>> On Sat, Dec 30, 2017 at 6:01 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
>>> Patch uses mechanism as agreed previously with Peter G et al. on this thread.
>>
>> I'm not sure that an agreement was reached, or what the substance of
>> that agreement was.
>
> I refer to this... and confirm I have implemented option 3

Thanks.  Sorry, I had forgotten about that.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  78. Re: [HACKERS] MERGE SQL Statement for PG11

    Simon Riggs <simon@2ndquadrant.com> — 2018-01-18T17:19:44Z

    On 30 December 2017 at 11:01, Simon Riggs <simon@2ndquadrant.com> wrote:

> Attached: MERGE patch is now MOSTLY complete, but still WIP.

New v10a attached, with additional dev work by Pavan and some review
from Andrew

> Patch works sufficiently well to take data from source and use it
> correctly against target, for the DELETE operation and INSERT DEFAULT
> VALUES. Patch also includes PL/pgSQL changes.
>
> Patch has full set of docs and tests, but does not yet pass all tests.

Now passes all tests, including throwing new type of semantic error
discovered during dev.

> Patch uses mechanism as agreed previously with Peter G et al. on this thread.

LATEST SUMMARY
Works
* EXPLAIN
* INSERT actions (thanks Pavan)
* UPDATE actions (thanks Pavan)
* DELETE actions
* DO NOTHING actions
* PL/pgSQL
* Triggers for row and statement
* SQL Standard error requirements

Not yet working
* AND conditions (currently WIP, expected soon)
* No isolation tests yet, so EvalPlanQual untested
* RLS
* Partitioning

Based on this successful progress I imagine I'll be looking to commit
this by the end of the CF, allowing us 2 further months to bugfix.

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  79. Re: [HACKERS] MERGE SQL Statement for PG11

    Simon Riggs <simon@2ndquadrant.com> — 2018-01-23T13:51:25Z

    On 18 January 2018 at 17:19, Simon Riggs <simon@2ndquadrant.com> wrote:
> On 30 December 2017 at 11:01, Simon Riggs <simon@2ndquadrant.com> wrote:
>
>> Attached: MERGE patch is now MOSTLY complete, but still WIP.

v11 attached

> LATEST SUMMARY
> Works
> * EXPLAIN
> * INSERT actions (thanks Pavan)
> * UPDATE actions (thanks Pavan)
> * DELETE actions
> * AND conditions (thanks Pavan)
> * Isolation tests and EvalPlanQual
> * DO NOTHING actions
> * PL/pgSQL
> * Triggers for row and statement
> * SQL Standard error requirements
>
> Not yet working
> * Partitioning
> * RLS
>
> Based on this successful progress I imagine I'll be looking to commit
> this by the end of the CF, allowing us 2 further months to bugfix.

This is complete and pretty clean now. 1200 lines of code, plus docs and tests.

I'm expecting to commit this and then come back for the Partitioning &
RLS later, but will wait a few days for comments and other reviews.

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  80. Re: [HACKERS] MERGE SQL Statement for PG11

    Peter Geoghegan <pg@bowt.ie> — 2018-01-24T01:35:03Z

    On Tue, Jan 23, 2018 at 5:51 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
>> Not yet working
>> * Partitioning
>> * RLS
>>
>> Based on this successful progress I imagine I'll be looking to commit
>> this by the end of the CF, allowing us 2 further months to bugfix.
>
> This is complete and pretty clean now. 1200 lines of code, plus docs and tests.

That timeline seems aggressive to me. Also, the patch appears to have
bitrot. Please rebase, and post a new version.

Some feedback based on a short read-through of your v11:

* What's the postgres_fdw status?

* Relatedly, when/where does applying the junkfilter happen, if not here?:

> @@ -1767,9 +1833,9 @@ ExecModifyTable(PlanState *pstate)
>             }
>
>             /*
> -            * apply the junkfilter if needed.
> +            * apply the junkfilter if needed - we do this later for CMD_MERGE
>              */
> -           if (operation != CMD_DELETE)
> +           if (operation == CMD_UPDATE || operation == CMD_INSERT)
>                 slot = ExecFilterJunk(junkfilter, slot);

* Isn't "consider INSERT ... ON CONFLICT DO UPDATE" obsolete in these
doc changes?:

> -F312   MERGE statement         NO  consider INSERT ... ON CONFLICT DO UPDATE
> -F313   Enhanced MERGE statement            NO
> -F314   MERGE statement with DELETE branch          NO
> +F312   MERGE statement         YES consider INSERT ... ON CONFLICT DO UPDATE
> +F313   Enhanced MERGE statement            YES
> +F314   MERGE statement with DELETE branch          YES

* What's the deal with transition tables? Your docs say this:

> +   <varlistentry>
> +    <term><replaceable class="parameter">source_table_name</replaceable></term>
> +    <listitem>
> +     <para>
> +      The name (optionally schema-qualified) of the source table, view or
> +      transition table.
> +     </para>
> +    </listitem>
> +   </varlistentry>

But the code says this:

> +           /*
> +            * XXX if we support transition tables this would need to move earlier
> +            * before ExecSetupTransitionCaptureState()
> +            */
> +           switch (action->commandType)

* Can UPDATEs themselves accept an UPDATE-style WHERE clause, in
addition to the WHEN quals, and the main ON join quals?

I don't see any examples of this in your regression tests.

* You added a new MERGE command tag. Shouldn't there be changes to
libpq's fe-exec.c, to go along with that?

* I noticed this:

> + else if (node->operation == CMD_MERGE)
> + {
> +     /*
> +     * XXX Add more detailed instrumentation for MERGE changes
> +     * when running EXPLAIN ANALYZE?
> +     */
> + }

I think that doing better here is necessary.

* I'm not a fan of stored rules, but I still think that this needs to
be discussed:

> -       product_queries = fireRules(parsetree,
> +       /*
> +        * First rule of MERGE club is we don't talk about rules
> +        */
> +       if (event == CMD_MERGE)
> +           product_queries = NIL;
> +       else
> +           product_queries = fireRules(parsetree,
>                                     result_relation,
>                                     event,
>                                     locks,

* This seems totally unnecessary at first blush:

> +                           /*
> +                            * Lock tuple for update.
> +                            *
> +                            * XXX Is this really needed? I put this in
> +                            * just to get hold of the existing tuple.
> +                            * But if we do need, then we probably
> +                            * should be looking at the return value of
> +                            * heap_lock_tuple() and take appropriate
> +                            * action.
> +                            */
> +                           tuple.t_self = *tupleid;
> +                           test = heap_lock_tuple(relation, &tuple, estate->es_output_cid,
> +                                   lockmode, LockWaitBlock, false, &buffer,
> +                                   &hufd);

* I don't know what the deal is with EPQ and MERGE in general, because
just after the above heap_lock_tuple(), you do this:

> +                       /*
> +                        * Test condition, if any
> +                        *
> +                        * In the absence of a condition we perform the action
> +                        * unconditionally (no need to check separately since
> +                        * ExecQual() will return true if there are no
> +                        * conditions to evaluate).
> +                        */
> +                       if (!ExecQual(action->whenqual, econtext))
> +                       {
> +                           if (BufferIsValid(buffer))
> +                               ReleaseBuffer(buffer);
> +                           continue;
> +                       }

Maybe *this* is why you call heap_lock_tuple(), actually, but that's
not clear, and in any case I don't see any point in it if you don't
check heap_lock_tuple()'s return value, and do some other extra thing
on the basis of that return value. No existing heap_lock_tuple()
ignores its return value, and ignoring the return value simply can't
make sense.

Don't WHEN quals need to participate in EPQ reevaluation, in order to
preserve the behavior that we see with UPDATE and DELETE? Why, or why
not?

I suppose that the way you evaluate WHEN quals separately makes a
certain amount of sense, but I think that you need to get things
straight with WHEN quals and READ COMMITTED conflict handling at a
high level (the semantics), which may or may not mean that WHEN quals
participate in EPQ evaluation. If you're going to introduce a special
case, I think you need to note it under the EvalPlanQual section of
the executor README.

This much I'm sure of: you should reevaluate WHEN quals if the UPDATE
chain is walked in READ COMMITTED mode, one way or another. It might
end up happening in your new heap_lock_tuple() retry loop, or you
might do something differently with EPQ, but something should happen
(I haven't got an opinion on the implementation details just yet,
though).

* Your isolation test should be commented. I'd expect you to talk
about what is different about MERGE as far as concurrency goes, if
anything. I note that you don't use additional WHEN quals in your
isolation test at all (just simple WHEN NOT MATCHED + WHEN MATCHED),
which was the first thing I looked for there. Look at
insert-conflict-do-update-3.spec for an example of roughly the kind of
commentary I had hoped to see in your regression test.

> I'm expecting to commit this and then come back for the Partitioning &
> RLS later, but will wait a few days for comments and other reviews.

I don't think that it's okay to defer RLS or partitioning support till
after an initial commit. While it's probably true that MERGE can just
follow ON CONFLICT's example when it comes to column-level privileges,
this won't be true with RLS.

-- 
Peter Geoghegan

  81. Re: [HACKERS] MERGE SQL Statement for PG11

    Simon Riggs <simon@2ndquadrant.com> — 2018-01-24T04:12:16Z

    On 24 January 2018 at 01:35, Peter Geoghegan <pg@bowt.ie> wrote:
> On Tue, Jan 23, 2018 at 5:51 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
>>> Not yet working
>>> * Partitioning
>>> * RLS
>>>
>>> Based on this successful progress I imagine I'll be looking to commit
>>> this by the end of the CF, allowing us 2 further months to bugfix.
>>
>> This is complete and pretty clean now. 1200 lines of code, plus docs and tests.
>
> That timeline seems aggressive to me.

Thank you for the review.

> Also, the patch appears to have
> bitrot. Please rebase, and post a new version.

Will do, though I'm sure that's only minor since we rebased only a few days ago.

> Some feedback based on a short read-through of your v11:
>
> * What's the postgres_fdw status?

MERGE currently works with normal relations and materialized views only.

> * Relatedly, when/where does applying the junkfilter happen, if not here?:

A few lines later in the same file. Junkfilters are still used.

> * Isn't "consider INSERT ... ON CONFLICT DO UPDATE" obsolete in these
> doc changes?:
>
>> -F312   MERGE statement         NO  consider INSERT ... ON CONFLICT DO UPDATE
>> -F313   Enhanced MERGE statement            NO
>> -F314   MERGE statement with DELETE branch          NO
>> +F312   MERGE statement         YES consider INSERT ... ON CONFLICT DO UPDATE
>> +F313   Enhanced MERGE statement            YES
>> +F314   MERGE statement with DELETE branch          YES

I think that it is still useful to highlight the existence of a
non-standard feature which people might not be otherwise unaware.

If you wish me to remove a reference to your work, I will bow to your wish.

> * What's the deal with transition tables? Your docs say this:
>
>> +   <varlistentry>
>> +    <term><replaceable class="parameter">source_table_name</replaceable></term>
>> +    <listitem>
>> +     <para>
>> +      The name (optionally schema-qualified) of the source table, view or
>> +      transition table.
>> +     </para>
>> +    </listitem>
>> +   </varlistentry>
>
> But the code says this:
>
>> +           /*
>> +            * XXX if we support transition tables this would need to move earlier
>> +            * before ExecSetupTransitionCaptureState()
>> +            */
>> +           switch (action->commandType)

Changes made by MERGE can theoretically be visible in a transition
table. When I tried to implement that there were problems caused by
the way INSERT ON CONFLICT has been implemented, so it will take a
fair amount of study and lifting to make that work. For now, they are
not supported and generate an error message. That behaviour was not
documented, but I've done that now.

SQL Std says "Transition tables are not generally updatable (and
therefore not simply updatable) and their columns are not updatable."

So MERGE cannot be a target of a transition table, but it can be the
source of one. So the docs you cite are correct, as is the comment.

> * Can UPDATEs themselves accept an UPDATE-style WHERE clause, in
> addition to the WHEN quals, and the main ON join quals?
>
> I don't see any examples of this in your regression tests.

No, they can't. Oracle allows it but it isn't SQL Standard.

> * You added a new MERGE command tag. Shouldn't there be changes to
> libpq's fe-exec.c, to go along with that?

Nice catch. One liner patch and docs updated in repo for next patch.

> * I noticed this:
>
>> + else if (node->operation == CMD_MERGE)
>> + {
>> +     /*
>> +     * XXX Add more detailed instrumentation for MERGE changes
>> +     * when running EXPLAIN ANALYZE?
>> +     */
>> + }
>
> I think that doing better here is necessary.

Please define better. It may be possible.

> * I'm not a fan of stored rules, but I still think that this needs to
> be discussed:
>
>> -       product_queries = fireRules(parsetree,
>> +       /*
>> +        * First rule of MERGE club is we don't talk about rules
>> +        */
>> +       if (event == CMD_MERGE)
>> +           product_queries = NIL;
>> +       else
>> +           product_queries = fireRules(parsetree,
>>                                     result_relation,
>>                                     event,
>>                                     locks,

It has been discussed, in my recent proposal, mentioned in the doc
page for clarity.

It has also been discussed previously in discussions around MERGE.
It's not clear how they would work, but we already have an example of
a statement type that doesn't support rules.

> * This seems totally unnecessary at first blush:
>
>> +                           /*
>> +                            * Lock tuple for update.
>> +                            *
>> +                            * XXX Is this really needed? I put this in
>> +                            * just to get hold of the existing tuple.
>> +                            * But if we do need, then we probably
>> +                            * should be looking at the return value of
>> +                            * heap_lock_tuple() and take appropriate
>> +                            * action.
>> +                            */
>> +                           tuple.t_self = *tupleid;
>> +                           test = heap_lock_tuple(relation, &tuple, estate->es_output_cid,
>> +                                   lockmode, LockWaitBlock, false, &buffer,
>> +                                   &hufd);
>
> * I don't know what the deal is with EPQ and MERGE in general, because
> just after the above heap_lock_tuple(), you do this:
>
>> +                       /*
>> +                        * Test condition, if any
>> +                        *
>> +                        * In the absence of a condition we perform the action
>> +                        * unconditionally (no need to check separately since
>> +                        * ExecQual() will return true if there are no
>> +                        * conditions to evaluate).
>> +                        */
>> +                       if (!ExecQual(action->whenqual, econtext))
>> +                       {
>> +                           if (BufferIsValid(buffer))
>> +                               ReleaseBuffer(buffer);
>> +                           continue;
>> +                       }
>
> Maybe *this* is why you call heap_lock_tuple(), actually, but that's
> not clear, and in any case I don't see any point in it if you don't
> check heap_lock_tuple()'s return value, and do some other extra thing
> on the basis of that return value. No existing heap_lock_tuple()
> ignores its return value, and ignoring the return value simply can't
> make sense.

Agreed, I don't think it is necessary.

> Don't WHEN quals need to participate in EPQ reevaluation, in order to
> preserve the behavior that we see with UPDATE and DELETE? Why, or why
> not?

WHEN qual evaluation occurs to establish which action to take. The
Standard is clear that this happens prior to the action.

> I suppose that the way you evaluate WHEN quals separately makes a
> certain amount of sense, but I think that you need to get things
> straight with WHEN quals and READ COMMITTED conflict handling at a
> high level (the semantics), which may or may not mean that WHEN quals
> participate in EPQ evaluation. If you're going to introduce a special
> case, I think you need to note it under the EvalPlanQual section of
> the executor README.

I wasn't going to introduce a special case.

> This much I'm sure of: you should reevaluate WHEN quals if the UPDATE
> chain is walked in READ COMMITTED mode, one way or another. It might
> end up happening in your new heap_lock_tuple() retry loop, or you
> might do something differently with EPQ, but something should happen
> (I haven't got an opinion on the implementation details just yet,
> though).

An interesting point, thank you for raising it.

WHEN quals, if they exist, may have dependencies on either the source
or target. If there is a dependency on the target there might be an
issue.

If the WHEN has a condition AND the WHEN qual fails a re-check after
we do EPQ, then I think we should just throw an error. If we
re-evaluate everything, I'm sure we'll get into some weird cases that
make MATCHED/NOT MATCHED change and that is a certain error case for
MERGE.

We might do better than that after some thought, but that seems like a
rabbit hole we should avoid in the first release. As we agreed
earlier, we can later extend MERGE to produce less errors in certain
concurrency cases.

> * Your isolation test should be commented. I'd expect you to talk
> about what is different about MERGE as far as concurrency goes, if
> anything. I note that you don't use additional WHEN quals in your
> isolation test at all (just simple WHEN NOT MATCHED + WHEN MATCHED),
> which was the first thing I looked for there. Look at
> insert-conflict-do-update-3.spec for an example of roughly the kind of
> commentary I had hoped to see in your regression test.

I will be happy to add one that exercises some new code resulting from
the above.

>> I'm expecting to commit this and then come back for the Partitioning &
>> RLS later, but will wait a few days for comments and other reviews.
>
> I don't think that it's okay to defer RLS or partitioning support till
> after an initial commit. While it's probably true that MERGE can just
> follow ON CONFLICT's example when it comes to column-level privileges,
> this won't be true with RLS.

I think it is OK to do things in major pieces, as has been done with
many other commands. We have more time in this release to do that,
though we want to find and fix any issues in basic functionality like
concurrency ahead of trying to add fancy stuff and hitting problems
with it.

I've already made two changes your review has raised, thanks. Will
re-post soon. Thanks.

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  82. Re: [HACKERS] MERGE SQL Statement for PG11

    Simon Riggs <simon@2ndquadrant.com> — 2018-01-26T11:25:59Z

    On 24 January 2018 at 04:12, Simon Riggs <simon@2ndquadrant.com> wrote:
> On 24 January 2018 at 01:35, Peter Geoghegan <pg@bowt.ie> wrote:
>>
> Please rebase, and post a new version.
>
> Will do, though I'm sure that's only minor since we rebased only a few days ago.

New v12 with various minor corrections and rebased.

Main new aspect here is greatly expanded isolation tests. Please read
and suggest new tests.

We've used those to uncover a few unhandled cases in the concurrency
of very comple MERGE statements, so we will repost again on Mon/Tues
with a new version covering all the new tests and any comments made
here. Nothing to worry about, just some changed logic.

I will post again later today with written details of the concurrency
rules we're working to now. I've left most of the isolation test
expected output as "TO BE DECIDED", so that we can agree our way
forwards.

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  83. Re: [HACKERS] MERGE SQL Statement for PG11

    Simon Riggs <simon@2ndquadrant.com> — 2018-01-29T14:11:47Z

    On 26 January 2018 at 11:25, Simon Riggs <simon@2ndquadrant.com> wrote:
> On 24 January 2018 at 04:12, Simon Riggs <simon@2ndquadrant.com> wrote:
>> On 24 January 2018 at 01:35, Peter Geoghegan <pg@bowt.ie> wrote:
>>>
>> Please rebase, and post a new version.
>>
>> Will do, though I'm sure that's only minor since we rebased only a few days ago.
>
> New v12 with various minor corrections and rebased.
>
> Main new aspect here is greatly expanded isolation tests. Please read
> and suggest new tests.
>
> We've used those to uncover a few unhandled cases in the concurrency
> of very comple MERGE statements, so we will repost again on Mon/Tues
> with a new version covering all the new tests and any comments made
> here. Nothing to worry about, just some changed logic.
>
> I will post again later today with written details of the concurrency
> rules we're working to now. I've left most of the isolation test
> expected output as "TO BE DECIDED", so that we can agree our way
> forwards.

New patch attached that correctly handles all known concurrency cases,
with expected test output.

The concurrency rules are very simple:
If a MATCHED row is concurrently updated/deleted
1. We run EvalPlanQual
2. If the updated row is gone EPQ returns NULL slot or EPQ returns a
row with NULL values, then
{
   if NOT MATCHED action exists, then raise ERROR
   else continue to next row
}
else
  re-check all MATCHED AND conditions and execute the first action
whose WHEN Condition evaluates to TRUE


This means MERGE will work just fine for "normal" UPDATEs, but it will
often fail (deterministically) in concurrent tests with mixed
insert/deletes or UPDATEs that touch the PK, as requested.

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  84. Re: [HACKERS] MERGE SQL Statement for PG11

    Pavel Stehule <pavel.stehule@gmail.com> — 2018-01-29T14:19:55Z

    Hi

2018-01-29 15:11 GMT+01:00 Simon Riggs <simon@2ndquadrant.com>:

> On 26 January 2018 at 11:25, Simon Riggs <simon@2ndquadrant.com> wrote:
> > On 24 January 2018 at 04:12, Simon Riggs <simon@2ndquadrant.com> wrote:
> >> On 24 January 2018 at 01:35, Peter Geoghegan <pg@bowt.ie> wrote:
> >>>
> >> Please rebase, and post a new version.
> >>
> >> Will do, though I'm sure that's only minor since we rebased only a few
> days ago.
> >
> > New v12 with various minor corrections and rebased.
> >
> > Main new aspect here is greatly expanded isolation tests. Please read
> > and suggest new tests.
> >
> > We've used those to uncover a few unhandled cases in the concurrency
> > of very comple MERGE statements, so we will repost again on Mon/Tues
> > with a new version covering all the new tests and any comments made
> > here. Nothing to worry about, just some changed logic.
> >
> > I will post again later today with written details of the concurrency
> > rules we're working to now. I've left most of the isolation test
> > expected output as "TO BE DECIDED", so that we can agree our way
> > forwards.
>
> New patch attached that correctly handles all known concurrency cases,
> with expected test output.
>
> The concurrency rules are very simple:
> If a MATCHED row is concurrently updated/deleted
> 1. We run EvalPlanQual
> 2. If the updated row is gone EPQ returns NULL slot or EPQ returns a
> row with NULL values, then
> {
>    if NOT MATCHED action exists, then raise ERROR
>    else continue to next row
> }
> else
>   re-check all MATCHED AND conditions and execute the first action
> whose WHEN Condition evaluates to TRUE
>
>
> This means MERGE will work just fine for "normal" UPDATEs, but it will
> often fail (deterministically) in concurrent tests with mixed
> insert/deletes or UPDATEs that touch the PK, as requested.
>

can be nice to have part about differences between MERGE and INSERT ON
CONFLICT DO

Regards

Pavel


> --
> Simon Riggs                http://www.2ndQuadrant.com/
> PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
>

  85. Re: [HACKERS] MERGE SQL Statement for PG11

    Simon Riggs <simon@2ndquadrant.com> — 2018-01-29T14:40:28Z

    On 29 January 2018 at 14:19, Pavel Stehule <pavel.stehule@gmail.com> wrote:

>> The concurrency rules are very simple:
>> If a MATCHED row is concurrently updated/deleted
>> 1. We run EvalPlanQual
>> 2. If the updated row is gone EPQ returns NULL slot or EPQ returns a
>> row with NULL values, then
>> {
>>    if NOT MATCHED action exists, then raise ERROR
>>    else continue to next row
>> }
>> else
>>   re-check all MATCHED AND conditions and execute the first action
>> whose WHEN Condition evaluates to TRUE
>>
>>
>> This means MERGE will work just fine for "normal" UPDATEs, but it will
>> often fail (deterministically) in concurrent tests with mixed
>> insert/deletes or UPDATEs that touch the PK, as requested.
>
>
> can be nice to have part about differences between MERGE and INSERT ON
> CONFLICT DO

We've agreed not to attempt to make it do anything like INSERT ON
CONFLICT, so we don't need to discuss that here anymore.

MERGE can be semantically equivalent to an UPDATE join or a DELETE
join, and in those cases, MERGE behaves the same. It handles much more
complex cases also.

MERGE as submitted here follows all MVCC rules similar to an UPDATE
join. If it hits a problem with concurent activity it throws
   ERROR:  could not serialize access due to concurrent update
to make sure there is no ambiguity (as described directly above).

As we discussed earlier, removing some of those ERRORs and making it
do something useful instead may be possible later.

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  86. Re: [HACKERS] MERGE SQL Statement for PG11

    Pavel Stehule <pavel.stehule@gmail.com> — 2018-01-29T14:55:06Z

    2018-01-29 15:40 GMT+01:00 Simon Riggs <simon@2ndquadrant.com>:

> On 29 January 2018 at 14:19, Pavel Stehule <pavel.stehule@gmail.com>
> wrote:
>
> >> The concurrency rules are very simple:
> >> If a MATCHED row is concurrently updated/deleted
> >> 1. We run EvalPlanQual
> >> 2. If the updated row is gone EPQ returns NULL slot or EPQ returns a
> >> row with NULL values, then
> >> {
> >>    if NOT MATCHED action exists, then raise ERROR
> >>    else continue to next row
> >> }
> >> else
> >>   re-check all MATCHED AND conditions and execute the first action
> >> whose WHEN Condition evaluates to TRUE
> >>
> >>
> >> This means MERGE will work just fine for "normal" UPDATEs, but it will
> >> often fail (deterministically) in concurrent tests with mixed
> >> insert/deletes or UPDATEs that touch the PK, as requested.
> >
> >
> > can be nice to have part about differences between MERGE and INSERT ON
> > CONFLICT DO
>
> We've agreed not to attempt to make it do anything like INSERT ON
> CONFLICT, so we don't need to discuss that here anymore.
>

My note was not against MERGE or INSERT ON CONFLICT. If I understand to
this topic, I agree so these commands should be implemented separately. But
if we use two commands with some intersection, there can be nice to have
documentation about recommended use cases. Probably it will be very often
question.

Regards

Pavel


>
> MERGE can be semantically equivalent to an UPDATE join or a DELETE
> join, and in those cases, MERGE behaves the same. It handles much more
> complex cases also.
>
> MERGE as submitted here follows all MVCC rules similar to an UPDATE
> join. If it hits a problem with concurent activity it throws
>    ERROR:  could not serialize access due to concurrent update
> to make sure there is no ambiguity (as described directly above).
>
> As we discussed earlier, removing some of those ERRORs and making it
> do something useful instead may be possible later.
>
> --
> Simon Riggs                http://www.2ndQuadrant.com/
> PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
>

  87. Re: [HACKERS] MERGE SQL Statement for PG11

    Robert Haas <robertmhaas@gmail.com> — 2018-01-29T15:07:38Z

    On Tue, Jan 23, 2018 at 8:51 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
> This is complete and pretty clean now. 1200 lines of code, plus docs and tests.
>
> I'm expecting to commit this and then come back for the Partitioning &
> RLS later, but will wait a few days for comments and other reviews.

I agree with Peter: that's unacceptable.  You're proposing to commit a
patch that is not only has had only a very limited amount of review
yet but by your own admission is not even complete.  Partitioning and
RLS shouldn't be afterthoughts; they should be in the original patch.
Moreover, the patch should have had meaningful review from people not
involved in writing it, and that is a process that generally takes a
few months or at least several weeks, not a few days.

An argument could be made that this patch is already too late for PG
11, because it's a major feature that was not submitted in relatively
complete form before the beginning of the penultimate CommitFest.  I'm
not going to make that argument, because I believe this patch is
probably sufficiently low-risk that it can be committed between now
and feature freeze without great risk of destabilizing the release.
But committing it without some in-depth review is not the way to get
there.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  88. Re: [HACKERS] MERGE SQL Statement for PG11

    Simon Riggs <simon@2ndquadrant.com> — 2018-01-29T15:12:23Z

    On 29 January 2018 at 14:55, Pavel Stehule <pavel.stehule@gmail.com> wrote:

> My note was not against MERGE or INSERT ON CONFLICT. If I understand to this
> topic, I agree so these commands should be implemented separately. But if we
> use two commands with some intersection, there can be nice to have
> documentation about recommended use cases. Probably it will be very often
> question.

That is more qualitative assessment of each, which I think I will defer on.

This patch is about implementing the SQL Standard compliant MERGE
command which is widely used in other databases and by various tools.

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  89. Re: [HACKERS] MERGE SQL Statement for PG11

    Simon Riggs <simon@2ndquadrant.com> — 2018-01-29T15:32:52Z

    On 29 January 2018 at 15:07, Robert Haas <robertmhaas@gmail.com> wrote:

> Moreover, the patch should have had meaningful review from people not
> involved in writing it, and that is a process that generally takes a
> few months or at least several weeks, not a few days.

The code is about 1200 lines and has extensive docs, comments and tests.

There are no contentious infrastructure changes, so the debate around
concurrency is probably the main one. So it looks to me like
meaningful review has taken place, though I know Andrew and Pavan have
also looked at it in detail.

But having said that, I'm not rushing to commit and further detailed
review is welcome, hence the CF status.

> An argument could be made that this patch is already too late for PG
> 11, because it's a major feature that was not submitted in relatively
> complete form before the beginning of the penultimate CommitFest.  I'm
> not going to make that argument, because I believe this patch is
> probably sufficiently low-risk that it can be committed between now
> and feature freeze without great risk of destabilizing the release.
> But committing it without some in-depth review is not the way to get
> there.

The patch was substantially complete at that time (was v9d). Later
work has changed isolated areas.

I agree that this is low-risk. If I suggest committing it sooner
rather than later it is because that is more likely to throw up bugs
that will increase the eventual quality.

Overall, I'm following the style of development process you have
yourself used a number of times now. Waiting for mega-patches to be
complete is not as useful as phased development.

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  90. Re: [HACKERS] MERGE SQL Statement for PG11

    Bruce Momjian <bruce@momjian.us> — 2018-01-29T15:44:57Z

    On Mon, Jan 29, 2018 at 03:12:23PM +0000, Simon Riggs wrote:
> On 29 January 2018 at 14:55, Pavel Stehule <pavel.stehule@gmail.com> wrote:
> 
> > My note was not against MERGE or INSERT ON CONFLICT. If I understand to this
> > topic, I agree so these commands should be implemented separately. But if we
> > use two commands with some intersection, there can be nice to have
> > documentation about recommended use cases. Probably it will be very often
> > question.
> 
> That is more qualitative assessment of each, which I think I will defer on.
> 
> This patch is about implementing the SQL Standard compliant MERGE
> command which is widely used in other databases and by various tools.

Uh, if we know we are going to get question on this, the patch had
better have an explanation of when to use it.  Pushing the problem to
later doesn't seem helpful.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

  91. Re: [HACKERS] MERGE SQL Statement for PG11

    Tom Lane <tgl@sss.pgh.pa.us> — 2018-01-29T16:06:12Z

    Simon Riggs <simon@2ndquadrant.com> writes:
> On 29 January 2018 at 15:07, Robert Haas <robertmhaas@gmail.com> wrote:
>> An argument could be made that this patch is already too late for PG
>> 11, because it's a major feature that was not submitted in relatively
>> complete form before the beginning of the penultimate CommitFest.

> Overall, I'm following the style of development process you have
> yourself used a number of times now. Waiting for mega-patches to be
> complete is not as useful as phased development.

An important part of that style is starting at an appropriate time in the
release cycle.  As things stand, you are proposing to commit an unfinished
feature to v11, and then we have to see if the missing parts show up on
time (ie before 1 March) and with adequate quality.  Otherwise we'll be
having a debate on whether to revert the feature or not ... and if it
comes to that, my vote will be for reverting.

I'd be much happier about committing this with some essential parts
missing if it were done at the start of a devel cycle rather than near
the end.

			regards, tom lane

  92. Re: [HACKERS] MERGE SQL Statement for PG11

    Simon Riggs <simon@2ndquadrant.com> — 2018-01-29T16:13:38Z

    On 29 January 2018 at 15:44, Bruce Momjian <bruce@momjian.us> wrote:
> On Mon, Jan 29, 2018 at 03:12:23PM +0000, Simon Riggs wrote:
>> On 29 January 2018 at 14:55, Pavel Stehule <pavel.stehule@gmail.com> wrote:
>>
>> > My note was not against MERGE or INSERT ON CONFLICT. If I understand to this
>> > topic, I agree so these commands should be implemented separately. But if we
>> > use two commands with some intersection, there can be nice to have
>> > documentation about recommended use cases. Probably it will be very often
>> > question.
>>
>> That is more qualitative assessment of each, which I think I will defer on.
>>
>> This patch is about implementing the SQL Standard compliant MERGE
>> command which is widely used in other databases and by various tools.
>
> Uh, if we know we are going to get question on this, the patch had
> better have an explanation of when to use it.  Pushing the problem to
> later doesn't seem helpful.

What problem are you referring to? MERGE is not being implemented as
some kind of rival to existing functionality, it does things we cannot
yet do.

Info below is for interest only, it is unrelated to this patch:

INSERT ON CONFLICT UPDATE does only INSERT and UPDATE and has various
restrictions. It violates MVCC when it needed to allow it to succeed
more frequently in updating a concurrently inserted row. It is not SQL
Standard.

MERGE allows you to make INSERTs, UPDATEs and DELETEs against a single
target table using complex conditionals. It follows the SQLStandard;
many developers from other databases, much existing code and many
tools know it.
e.g.

MERGE INTO target t
USING source s
ON t.tid = s.sid
WHEN MATCHED AND balance > delta THEN
  UPDATE SET balance = balance - delta
WHEN MATCHED
 DELETE;
WHEN NOT MATCHED THEN
  INSERT (balance, tid) VALUES (balance + delta, sid)

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  93. Re: [HACKERS] MERGE SQL Statement for PG11

    Chapman Flack <chap@anastigmatix.net> — 2018-01-29T16:23:24Z

    On 01/29/2018 11:13 AM, Simon Riggs wrote:
> On 29 January 2018 at 15:44, Bruce Momjian <bruce@momjian.us> wrote:
>> Uh, if we know we are going to get question on this, the patch had
>> better have an explanation of when to use it.  Pushing the problem to
>> later doesn't seem helpful.
> 
> What problem are you referring to?
> 
> INSERT ON CONFLICT UPDATE does ...
> 
> MERGE allows you to ...
In my reading of Pavel and Bruce, the only 'problem' being suggested
is that the patch hasn't added a bit of documentation somewhere that
lays out the relationship between these two things, more or less as
you just did.

-Chap

  94. Re: [HACKERS] MERGE SQL Statement for PG11

    Simon Riggs <simon@2ndquadrant.com> — 2018-01-29T16:34:48Z

    On 29 January 2018 at 16:06, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Simon Riggs <simon@2ndquadrant.com> writes:
>> On 29 January 2018 at 15:07, Robert Haas <robertmhaas@gmail.com> wrote:
>>> An argument could be made that this patch is already too late for PG
>>> 11, because it's a major feature that was not submitted in relatively
>>> complete form before the beginning of the penultimate CommitFest.
>
>> Overall, I'm following the style of development process you have
>> yourself used a number of times now. Waiting for mega-patches to be
>> complete is not as useful as phased development.
>
> An important part of that style is starting at an appropriate time in the
> release cycle.  As things stand, you are proposing to commit an unfinished
> feature to v11, and then we have to see if the missing parts show up on
> time (ie before 1 March) and with adequate quality.  Otherwise we'll be
> having a debate on whether to revert the feature or not ... and if it
> comes to that, my vote will be for reverting.
>
> I'd be much happier about committing this with some essential parts
> missing if it were done at the start of a devel cycle rather than near
> the end.

I agree with all of the above.

In terms of timing of commits, I have marked the patch Ready For
Committer. To me that signifies that it is ready for review by a
Committer prior to commit.

In case of doubt, I would not even suggest committing this if it had
any concurrency issues. That would be clearly unacceptable.

The only discussion would be about the word "unfinished". I'm not
clear why this patch, which has current caveats all clearly indicated
in the docs, differs substantially from other projects that have
committed their work ahead of having everything everybody wants, such
as replication, materialized views, parallel query, partitioning,
logical decoding etc.. All of those features had caveats in the first
release in which they were included and many of them were committed
prior to the last CF. We are working now to remove those caveats. Why
is this different? It shouldn't be. If unfinished means it has caveats
that is different to unfinished meaning crappy, risky, contentious
etc..


Anyway, reviews welcome, but few people know anything about
targetlists and column handling.

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  95. Re: [HACKERS] MERGE SQL Statement for PG11

    Bruce Momjian <bruce@momjian.us> — 2018-01-29T16:44:22Z

    On Mon, Jan 29, 2018 at 04:34:48PM +0000, Simon Riggs wrote:
> I agree with all of the above.
> 
> In terms of timing of commits, I have marked the patch Ready For
> Committer. To me that signifies that it is ready for review by a
> Committer prior to commit.
> 
> In case of doubt, I would not even suggest committing this if it had
> any concurrency issues. That would be clearly unacceptable.
> 
> The only discussion would be about the word "unfinished". I'm not
> clear why this patch, which has current caveats all clearly indicated
> in the docs, differs substantially from other projects that have
> committed their work ahead of having everything everybody wants, such
> as replication, materialized views, parallel query, partitioning,
> logical decoding etc.. All of those features had caveats in the first
> release in which they were included and many of them were committed
> prior to the last CF. We are working now to remove those caveats. Why
> is this different? It shouldn't be. If unfinished means it has caveats
> that is different to unfinished meaning crappy, risky, contentious
> etc..

I think the question is how does it handle cases it doesn't support? 
Does it give wrong answers?  Does it give a helpful error message?  Can
you summarize that?

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

  96. Re: [HACKERS] MERGE SQL Statement for PG11

    Tom Lane <tgl@sss.pgh.pa.us> — 2018-01-29T16:50:02Z

    Bruce Momjian <bruce@momjian.us> writes:
> On Mon, Jan 29, 2018 at 04:34:48PM +0000, Simon Riggs wrote:
>> ... If unfinished means it has caveats
>> that is different to unfinished meaning crappy, risky, contentious
>> etc..

> I think the question is how does it handle cases it doesn't support? 
> Does it give wrong answers?  Does it give a helpful error message?  Can
> you summarize that?

What I was reacting to was the comments just upthread that it doesn't
yet handle partitions or RLS.  Those things don't seem optional to me.
Maybe they're small additions, but if so why aren't they done already?

Also, as far as phased development goes: Simon's drawing analogies
to things like parallel query, which we all understood had to be
done over multiple dev cycles because they were too big to finish
in one cycle.  I don't think MERGE qualifies: there seems no good
reason why it can't be done, full stop, in the first release where
it appears.

			regards, tom lane

  97. Re: [HACKERS] MERGE SQL Statement for PG11

    Simon Riggs <simon@2ndquadrant.com> — 2018-01-29T16:51:07Z

    On 29 January 2018 at 16:44, Bruce Momjian <bruce@momjian.us> wrote:

> I think the question is how does it handle cases it doesn't support?
> Does it give wrong answers?  Does it give a helpful error message?  Can
> you summarize that?

I'm happy to report that it gives correct answers to every known MERGE
test, except

* where it hits a concurrency issue and throws SQLCODE =
ERRCODE_T_R_SERIALIZATION_FAILURE and the standard text for that

* where it hits an unsupported feature and throws SQLCODE =
ERRCODE_FEATURE_NOT_SUPPORTED, with appropriate text

but of course Robert is correct and everything benefits from further review.

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  98. Re: [HACKERS] MERGE SQL Statement for PG11

    Simon Riggs <simon@2ndquadrant.com> — 2018-01-29T17:03:08Z

    On 29 January 2018 at 16:50, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Bruce Momjian <bruce@momjian.us> writes:
>> On Mon, Jan 29, 2018 at 04:34:48PM +0000, Simon Riggs wrote:
>>> ... If unfinished means it has caveats
>>> that is different to unfinished meaning crappy, risky, contentious
>>> etc..
>
>> I think the question is how does it handle cases it doesn't support?
>> Does it give wrong answers?  Does it give a helpful error message?  Can
>> you summarize that?
>
> What I was reacting to was the comments just upthread that it doesn't
> yet handle partitions or RLS.  Those things don't seem optional to me.
> Maybe they're small additions, but if so why aren't they done already?

Phasing and risk.

Partitioning doesn't look too bad, so that looks comfortable for PG11,
assuming it doesn't hit some unhandled complexity.

Including RLS in the first commit/release turns this into a high risk
patch. Few people use it, but if they do, they don't want me putting a
hole in their battleship (literally) should we discover some weird
unhandled logic in a complex new command.

My recommendation would be to support that later for those that use
it. For those that don't, it doesn't matter so can also be done later.

> Also, as far as phased development goes: Simon's drawing analogies
> to things like parallel query, which we all understood had to be
> done over multiple dev cycles because they were too big to finish
> in one cycle.  I don't think MERGE qualifies: there seems no good
> reason why it can't be done, full stop, in the first release where
> it appears.

That remains the plan, barring delays.

If you want to include RLS, then I would appreciate an early review.

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  99. Re: [HACKERS] MERGE SQL Statement for PG11

    Simon Riggs <simon@2ndquadrant.com> — 2018-01-29T17:08:33Z

    On 29 January 2018 at 16:23, Chapman Flack <chap@anastigmatix.net> wrote:
> On 01/29/2018 11:13 AM, Simon Riggs wrote:
>> On 29 January 2018 at 15:44, Bruce Momjian <bruce@momjian.us> wrote:
>>> Uh, if we know we are going to get question on this, the patch had
>>> better have an explanation of when to use it.  Pushing the problem to
>>> later doesn't seem helpful.
>>
>> What problem are you referring to?
>>
>> INSERT ON CONFLICT UPDATE does ...
>>
>> MERGE allows you to ...
> In my reading of Pavel and Bruce, the only 'problem' being suggested
> is that the patch hasn't added a bit of documentation somewhere that
> lays out the relationship between these two things, more or less as
> you just did.

I am happy to write docs as requested.

There are currently no docs saying when INSERT ON CONFLICT UPDATE
should be used other than the ref page for that command. There is no
mention of it in the "Data Manipulation" section of the docs.

I've included docs for MERGE so it is mentioned in concurrency and
reference sections, so it follows the same model.

Where would people like me to put these docs?

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  100. Re: [HACKERS] MERGE SQL Statement for PG11

    Pavel Stehule <pavel.stehule@gmail.com> — 2018-01-29T17:18:08Z

    2018-01-29 18:08 GMT+01:00 Simon Riggs <simon@2ndquadrant.com>:

> On 29 January 2018 at 16:23, Chapman Flack <chap@anastigmatix.net> wrote:
> > On 01/29/2018 11:13 AM, Simon Riggs wrote:
> >> On 29 January 2018 at 15:44, Bruce Momjian <bruce@momjian.us> wrote:
> >>> Uh, if we know we are going to get question on this, the patch had
> >>> better have an explanation of when to use it.  Pushing the problem to
> >>> later doesn't seem helpful.
> >>
> >> What problem are you referring to?
> >>
> >> INSERT ON CONFLICT UPDATE does ...
> >>
> >> MERGE allows you to ...
> > In my reading of Pavel and Bruce, the only 'problem' being suggested
> > is that the patch hasn't added a bit of documentation somewhere that
> > lays out the relationship between these two things, more or less as
> > you just did.
>
> I am happy to write docs as requested.
>
> There are currently no docs saying when INSERT ON CONFLICT UPDATE
> should be used other than the ref page for that command. There is no
> mention of it in the "Data Manipulation" section of the docs.
>
> I've included docs for MERGE so it is mentioned in concurrency and
> reference sections, so it follows the same model.
>
> Where would people like me to put these docs?
>

Depends on size - small note can be placed in MERGE docs and link from
INSERT ON CONFLICT DO.

Regards

Pavel



>
> --
> Simon Riggs                http://www.2ndQuadrant.com/
> PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
>
>

  101. Re: [HACKERS] MERGE SQL Statement for PG11

    Peter Geoghegan <pg@bowt.ie> — 2018-01-29T17:23:58Z

    On Mon, Jan 29, 2018 at 8:44 AM, Bruce Momjian <bruce@momjian.us> wrote:
> On Mon, Jan 29, 2018 at 04:34:48PM +0000, Simon Riggs wrote:
>> The only discussion would be about the word "unfinished". I'm not
>> clear why this patch, which has current caveats all clearly indicated
>> in the docs, differs substantially from other projects that have
>> committed their work ahead of having everything everybody wants, such
>> as replication, materialized views, parallel query, partitioning,
>> logical decoding etc.. All of those features had caveats in the first
>> release in which they were included and many of them were committed
>> prior to the last CF. We are working now to remove those caveats. Why
>> is this different? It shouldn't be. If unfinished means it has caveats
>> that is different to unfinished meaning crappy, risky, contentious
>> etc..
>
> I think the question is how does it handle cases it doesn't support?
> Does it give wrong answers?  Does it give a helpful error message?  Can
> you summarize that?

+1

ON CONFLICT had support for logical decoding, updatable views, and RLS
in the first commit. ON CONFLICT was committed in a form that worked
seamlessly with any other feature in the system you can name. Making
ON CONFLICT play nice with all adjacent features was a great deal of
work, and I definitely needed Andres' help for the logical decoding
part, but we got it done. (Logical decoding for MERGE should be quite
a lot easier, though.)

I'm willing to talk about why MERGE is different to ON CONFLICT, and
why it may not need to tick all of the same boxes. DML statements are
supposed to be highly composable things, though. That's the premise
you should start from IMV.

-- 
Peter Geoghegan

  102. Re: [HACKERS] MERGE SQL Statement for PG11

    Peter Geoghegan <pg@bowt.ie> — 2018-01-29T17:35:15Z

    On Mon, Jan 29, 2018 at 8:51 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
> On 29 January 2018 at 16:44, Bruce Momjian <bruce@momjian.us> wrote:
>
>> I think the question is how does it handle cases it doesn't support?
>> Does it give wrong answers?  Does it give a helpful error message?  Can
>> you summarize that?
>
> I'm happy to report that it gives correct answers to every known MERGE
> test, except
>
> * where it hits a concurrency issue and throws SQLCODE =
> ERRCODE_T_R_SERIALIZATION_FAILURE and the standard text for that
>
> * where it hits an unsupported feature and throws SQLCODE =
> ERRCODE_FEATURE_NOT_SUPPORTED, with appropriate text

What specific features does it not work with already? A list would be helpful.

-- 
Peter Geoghegan

  103. Re: [HACKERS] MERGE SQL Statement for PG11

    Simon Riggs <simon@2ndquadrant.com> — 2018-01-29T18:14:39Z

    On 29 January 2018 at 17:35, Peter Geoghegan <pg@bowt.ie> wrote:
> On Mon, Jan 29, 2018 at 8:51 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
>> On 29 January 2018 at 16:44, Bruce Momjian <bruce@momjian.us> wrote:
>>
>>> I think the question is how does it handle cases it doesn't support?
>>> Does it give wrong answers?  Does it give a helpful error message?  Can
>>> you summarize that?
>>
>> I'm happy to report that it gives correct answers to every known MERGE
>> test, except
>>
>> * where it hits a concurrency issue and throws SQLCODE =
>> ERRCODE_T_R_SERIALIZATION_FAILURE and the standard text for that
>>
>> * where it hits an unsupported feature and throws SQLCODE =
>> ERRCODE_FEATURE_NOT_SUPPORTED, with appropriate text
>
> What specific features does it not work with already? A list would be helpful.

Yes, I added that to the docs as a result of your review comments.

I also mentioned them here last week in your review in answer to your
specific questions.


The current list of features that return ERRCODE_FEATURE_NOT_SUPPORTED is
* Tables with Row Security enabled
* Partitioning & Inheritance
* Foreign Tables

Rules are ignored, as they are with COPY.

If people have concerns or find problems following review, I will be
happy to update this list and/or fix issues, as normal.

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  104. Re: [HACKERS] MERGE SQL Statement for PG11

    Peter Geoghegan <pg@bowt.ie> — 2018-01-29T20:41:52Z

    On Mon, Jan 29, 2018 at 6:11 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
> New patch attached that correctly handles all known concurrency cases,
> with expected test output.

This revision, v13, seems much improved in this area.

> This means MERGE will work just fine for "normal" UPDATEs, but it will
> often fail (deterministically) in concurrent tests with mixed
> insert/deletes or UPDATEs that touch the PK, as requested.

* While looking at how you're handling concurrency/EPQ now, I noticed this code:

> +                       /*
> +                        * Test condition, if any
> +                        *
> +                        * In the absence of a condition we perform the action
> +                        * unconditionally (no need to check separately since
> +                        * ExecQual() will return true if there are no
> +                        * conditions to evaluate).
> +                        */
> +                       if (!ExecQual(action->whenqual, econtext))
> +                       {
> +                           if (BufferIsValid(buffer))
> +                               ReleaseBuffer(buffer);
> +                           continue;
> +                       }

(As well as its interactions with ExecUpdate() + ExecDelete(), which
are significant.)

The way that routines like ExecUpdate() interact with MERGE for WHEN
qual + EPQ handling seems kind of convoluted. I hope for something
cleaner in the next revision.

* This also stood out (not sure if you were referring/alluding to this
in the quoted text):

> +                               /*
> +                                * If EvalPlanQual did not return a tuple, it means we
> +                                * have seen a concurrent delete, or a concurrent update
> +                                * where the row has moved to another partition.
> +                                *
> +                                * UPDATE ignores this case and continues.
> +                                *
> +                                * If MERGE has a WHEN NOT MATCHED clause we know that the
> +                                * user would like to INSERT something in this case, yet
> +                                * we can't see the delete with our snapshot, so take the
> +                                * safe choice and throw an ERROR. If the user didn't care
> +                                * about WHEN NOT MATCHED INSERT then neither do we.
> +                                *
> +                                * XXX We might consider setting matched = false and loop
> +                                * back to lmerge though we'd need to do something like
> +                                * EvalPlanQual, but not quite.
> +                                */
> +                               else if (epqstate->epqresult == EPQ_TUPLE_IS_NULL &&
> +                                        node->mt_merge_subcommands & ACL_INSERT)
> +                               {
> +                                   /*
> +                                    * We need to throw a retryable ERROR because of the
> +                                    * concurrent update which we can't handle.
> +                                    */
> +                                   ereport(ERROR,
> +                                           (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
> +                                            errmsg("could not serialize access due to concurrent update")));
> +                               }

I don't think that ERRCODE_T_R_SERIALIZATION_FAILURE is ever okay in
READ COMMITTED mode. Anyway, I wonder why we shouldn't just go ahead
an do the WHEN NOT MATCHED INSERT on the basis of information which is
not visible to our snapshot. We choose to not UPDATE/DELETE on the
basis of information from the future already, within EPQ. My first
feeling is that this is a distinction without a difference, and you
should actually go and INSERT at this point, though I reserve the
right to change my mind about that.

Yeah, EPQ semantics are icky, but would actually inserting here really
be any worse than what we do?

Other things I noticed (not related to concurrency) following a fairly
quick pass:

* Basic tab completion support would be nice.

* The SQL standard explicitly imposes this limitation:

postgres=# explain merge into cities a using cities b on a.city =
b.city when matched and (select 1=1) then update set country =
b.country;
ERROR:  0A000: cannot use subquery in WHEN AND condition
LINE 1: ...sing cities b on a.city = b.city when matched and (select 1=...
                                                             ^
LOCATION:  transformSubLink, parse_expr.c:1865

Why do you, though? Do we really need it? I'm just curious about your
thoughts on it. To be clear, I'm not asserting that you're wrong.

* ISTM that this patch should have inserted/updated/deleted rows, in
roughly the same style as ON CONFLICT's EXPLAIN ANALYZE output. I
mentioned this already, and you seemed unclear on what I meant.
Hopefully my remarks here are clearer.

* Subselect handling is buggy:

postgres=# merge into cities a using cities b on a.city = b.city when
matched and a.city = 'Straffan' then update set country = (select
'Some country');
ERROR:  XX000: unrecognized node type: 114
LOCATION:  ExecInitExprRec, execExpr.c:2114

* Why no CTE support? SQL Server has this.

* The INSERT ... SELECT syntax doesn't work:

postgres=# merge into array_test a using (select '{1,2,3}'::int[] aaa)
b on a.aaa = b.aaa when matched then update set aaa = default when not
matched then insert (aaa) select '{1,2,3}';
ERROR:  42601: syntax error at or near "select"
LINE 1: ... aaa = default when not matched then insert (aaa) select '{1...
                                                             ^
LOCATION:  scanner_yyerror, scan.l:1092

But docs imply otherwise -- "source_query -- A query (SELECT statement
or VALUES statement) that supplies the rows to be merged into the
target_table_name". Either the docs are wrong, or the code is wrong.
Hopefully you can just fix the code.

* Rules are not going to be supported, on the grounds that the
behavior is unclear, which I suppose is fine. But what about
ruleutils.c support?

That seems like entirely another matter to me. What about EXPLAIN,
etc? Deparse support seems to be considered generic infrastructure,
that doesn't necessarily have much to do with the user-visible rules
feature.

* This restriction seems arbitrary and unjustified:

postgres=# merge into testoids a using (select 1 "key", 'foo' "data")
b on a.key = b.key when matched and a.oid = 5 then update set data =
b.data when not matched then insert (key, data) values (1, 'foo');
ERROR:  42P10: system column "oid" reference in WHEN AND condition is invalid
LINE 1: ...'foo' "data") b on a.key = b.key when matched and a.oid = 5 ...
                                                             ^
LOCATION:  scanRTEForColumn, parse_relation.c:738

* Wholerow vars are broken:

postgres=# merge into testoids a using (select 1 "key", 'foo' "data")
b on a.key = b.key when matched then update set data = b.*::text when
not matched then insert (key, data) values (1, 'foo');
ERROR:  XX000: variable not found in subplan target lists
LOCATION:  fix_join_expr_mutator, setrefs.c:2351

That's all I have for now.

-- 
Peter Geoghegan

  105. Re: [HACKERS] MERGE SQL Statement for PG11

    Simon Riggs <simon@2ndquadrant.com> — 2018-01-29T21:34:45Z

    On 29 January 2018 at 20:41, Peter Geoghegan <pg@bowt.ie> wrote:
> On Mon, Jan 29, 2018 at 6:11 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
>> New patch attached that correctly handles all known concurrency cases,
>> with expected test output.
>
> This revision, v13, seems much improved in this area.
>
>> This means MERGE will work just fine for "normal" UPDATEs, but it will
>> often fail (deterministically) in concurrent tests with mixed
>> insert/deletes or UPDATEs that touch the PK, as requested.
>
> * While looking at how you're handling concurrency/EPQ now, I noticed this code:
>
>> +                       /*
>> +                        * Test condition, if any
>> +                        *
>> +                        * In the absence of a condition we perform the action
>> +                        * unconditionally (no need to check separately since
>> +                        * ExecQual() will return true if there are no
>> +                        * conditions to evaluate).
>> +                        */
>> +                       if (!ExecQual(action->whenqual, econtext))
>> +                       {
>> +                           if (BufferIsValid(buffer))
>> +                               ReleaseBuffer(buffer);
>> +                           continue;
>> +                       }
>
> (As well as its interactions with ExecUpdate() + ExecDelete(), which
> are significant.)
>
> The way that routines like ExecUpdate() interact with MERGE for WHEN
> qual + EPQ handling seems kind of convoluted. I hope for something
> cleaner in the next revision.

Cleaner?

> * This also stood out (not sure if you were referring/alluding to this
> in the quoted text):
>
>> +                               /*
>> +                                * If EvalPlanQual did not return a tuple, it means we
>> +                                * have seen a concurrent delete, or a concurrent update
>> +                                * where the row has moved to another partition.
>> +                                *
>> +                                * UPDATE ignores this case and continues.
>> +                                *
>> +                                * If MERGE has a WHEN NOT MATCHED clause we know that the
>> +                                * user would like to INSERT something in this case, yet
>> +                                * we can't see the delete with our snapshot, so take the
>> +                                * safe choice and throw an ERROR. If the user didn't care
>> +                                * about WHEN NOT MATCHED INSERT then neither do we.
>> +                                *
>> +                                * XXX We might consider setting matched = false and loop
>> +                                * back to lmerge though we'd need to do something like
>> +                                * EvalPlanQual, but not quite.
>> +                                */
>> +                               else if (epqstate->epqresult == EPQ_TUPLE_IS_NULL &&
>> +                                        node->mt_merge_subcommands & ACL_INSERT)
>> +                               {
>> +                                   /*
>> +                                    * We need to throw a retryable ERROR because of the
>> +                                    * concurrent update which we can't handle.
>> +                                    */
>> +                                   ereport(ERROR,
>> +                                           (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
>> +                                            errmsg("could not serialize access due to concurrent update")));
>> +                               }
>
> I don't think that ERRCODE_T_R_SERIALIZATION_FAILURE is ever okay in
> READ COMMITTED mode.

We use that code already in Hot Standby in READ COMMITTED mode.

What code should it be? It needs to be a retryable errcode.

> Anyway, I wonder why we shouldn't just go ahead
> an do the WHEN NOT MATCHED INSERT on the basis of information which is
> not visible to our snapshot. We choose to not UPDATE/DELETE on the
> basis of information from the future already, within EPQ. My first
> feeling is that this is a distinction without a difference, and you
> should actually go and INSERT at this point, though I reserve the
> right to change my mind about that.
>
> Yeah, EPQ semantics are icky, but would actually inserting here really
> be any worse than what we do?

I argued that was possible and desirable, but you argued it was not
and got everybody else to agree with you. I'm surprised to see you
change your mind on that.

At your request I have specifically written the logic to avoid that.
I'm happy with that, for now, since what we have is correct, even if
we can do better later.

We can have fun with looping, live locks and weird errors another day.
SQL Standard says this area is implementation defined.

> Other things I noticed (not related to concurrency) following a fairly
> quick pass:
>
> * Basic tab completion support would be nice.

OK, but as most other patches do, that can be done later.

> * The SQL standard explicitly imposes this limitation:
>
> postgres=# explain merge into cities a using cities b on a.city =
> b.city when matched and (select 1=1) then update set country =
> b.country;
> ERROR:  0A000: cannot use subquery in WHEN AND condition
> LINE 1: ...sing cities b on a.city = b.city when matched and (select 1=...
>                                                              ^
> LOCATION:  transformSubLink, parse_expr.c:1865
>
> Why do you, though? Do we really need it? I'm just curious about your
> thoughts on it. To be clear, I'm not asserting that you're wrong.

Which limitation? Not allowing sub-selects? They are not supported, as
the message says.

> * ISTM that this patch should have inserted/updated/deleted rows, in
> roughly the same style as ON CONFLICT's EXPLAIN ANALYZE output. I
> mentioned this already, and you seemed unclear on what I meant.
> Hopefully my remarks here are clearer.

Yes, thanks. Can do.

> * Subselect handling is buggy:
>
> postgres=# merge into cities a using cities b on a.city = b.city when
> matched and a.city = 'Straffan' then update set country = (select
> 'Some country');
> ERROR:  XX000: unrecognized node type: 114
> LOCATION:  ExecInitExprRec, execExpr.c:2114

Not buggy, subselects are not supported in WHEN AND clauses because
they are not part of the planned query, nor can they be if we want to
handle the WHEN clause logic per spec.

> * Why no CTE support? SQL Server has this.

The SQL Standard doesn't require CTEs or RETURNING syntax, but they
could in time be supported.

> * The INSERT ... SELECT syntax doesn't work:
>
> postgres=# merge into array_test a using (select '{1,2,3}'::int[] aaa)
> b on a.aaa = b.aaa when matched then update set aaa = default when not
> matched then insert (aaa) select '{1,2,3}';
> ERROR:  42601: syntax error at or near "select"
> LINE 1: ... aaa = default when not matched then insert (aaa) select '{1...
>                                                              ^
> LOCATION:  scanner_yyerror, scan.l:1092
>
> But docs imply otherwise -- "source_query -- A query (SELECT statement
> or VALUES statement) that supplies the rows to be merged into the
> target_table_name". Either the docs are wrong, or the code is wrong.
> Hopefully you can just fix the code.

Neither. The docs show that is unsupported. The source query is the
USING phrase, there is no INSERT SELECT.

> * Rules are not going to be supported, on the grounds that the
> behavior is unclear, which I suppose is fine. But what about
> ruleutils.c support?
>
> That seems like entirely another matter to me. What about EXPLAIN,
> etc? Deparse support seems to be considered generic infrastructure,
> that doesn't necessarily have much to do with the user-visible rules
> feature.

The MERGE query is a normal query, so that should all just work.

EXPLAIN is specifically regression tested.

> * This restriction seems arbitrary and unjustified:
>
> postgres=# merge into testoids a using (select 1 "key", 'foo' "data")
> b on a.key = b.key when matched and a.oid = 5 then update set data =
> b.data when not matched then insert (key, data) values (1, 'foo');
> ERROR:  42P10: system column "oid" reference in WHEN AND condition is invalid
> LINE 1: ...'foo' "data") b on a.key = b.key when matched and a.oid = 5 ...
>                                                              ^
> LOCATION:  scanRTEForColumn, parse_relation.c:738

I followed the comments of how we handle CHECK constraints.

It's hard to think of a real world example that would use that; your
example seems strange. Why would you want to use Oids in the WHEN AND
clause? In the ON or DML clauses, sure, but not there.

This is a non-standard feature, so we can decide whether to support
that or not. Allowing them is easy, just not very meaningful. Not sure
if it has consequences.

> * Wholerow vars are broken:
>
> postgres=# merge into testoids a using (select 1 "key", 'foo' "data")
> b on a.key = b.key when matched then update set data = b.*::text when
> not matched then insert (key, data) values (1, 'foo');
> ERROR:  XX000: variable not found in subplan target lists
> LOCATION:  fix_join_expr_mutator, setrefs.c:2351
>
> That's all I have for now.

Good catch; that one is a valid error. I hadn't tried to either
support them or block them.

Maybe its in the SQL Standard, not sure. Support for whole row vars
probably isn't a priority though.

Thanks for your comments.

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  106. Re: [HACKERS] MERGE SQL Statement for PG11

    Peter Geoghegan <pg@bowt.ie> — 2018-01-29T22:31:21Z

    On Mon, Jan 29, 2018 at 1:34 PM, Simon Riggs <simon@2ndquadrant.com> wrote:
>> The way that routines like ExecUpdate() interact with MERGE for WHEN
>> qual + EPQ handling seems kind of convoluted. I hope for something
>> cleaner in the next revision.
>
> Cleaner?

Yeah, cleaner. The fact that when quals kind of participate in EPQ
evaluation without that being in execMain.c seems like it could be a
lot cleaner. I don't have more specifics than that right now.

>> I don't think that ERRCODE_T_R_SERIALIZATION_FAILURE is ever okay in
>> READ COMMITTED mode.
>
> We use that code already in Hot Standby in READ COMMITTED mode.
>
> What code should it be? It needs to be a retryable errcode.

I don't think that there should be any error, so I can't say.

>> Anyway, I wonder why we shouldn't just go ahead
>> an do the WHEN NOT MATCHED INSERT on the basis of information which is
>> not visible to our snapshot. We choose to not UPDATE/DELETE on the
>> basis of information from the future already, within EPQ. My first
>> feeling is that this is a distinction without a difference, and you
>> should actually go and INSERT at this point, though I reserve the
>> right to change my mind about that.
>>
>> Yeah, EPQ semantics are icky, but would actually inserting here really
>> be any worse than what we do?
>
> I argued that was possible and desirable, but you argued it was not
> and got everybody else to agree with you. I'm surprised to see you
> change your mind on that.

You're mistaken. Nothing I've said here is inconsistent with my
previous remarks on how we deal with concurrency.

> At your request I have specifically written the logic to avoid that.
> I'm happy with that, for now, since what we have is correct, even if
> we can do better later.
>
> We can have fun with looping, live locks and weird errors another day.
> SQL Standard says this area is implementation defined.

Who said anything about a loop? Where is the loop here?

I will rephrase what I said, to make it top-down rather than
bottom-up, which may make my intent clearer:

According to your documentation, "MERGE provides a single SQL
statement that can conditionally INSERT, UPDATE or DELETE rows, a task
that would otherwise require multiple procedural language statements".
But you're introducing a behavior/error that would not occur in
equivalent procedural client code consisting of an UPDATE followed by
a (conditionally executed) INSERT statement when run in READ COMMITTED
mode. You actually get exactly the concurrency issue that you cite as
unacceptable in justifying your serialization error with such
procedural code (when the UPDATE didn't affect any rows, only
following EPQ walking the UPDATE chain from the snapshot-visible
tuple, making the client code decide to do an INSERT on the basis of
information "from the future").

I think that it isn't this patch's job to make READ COMMITTED mode any
safer than it is in that existing scenario. A scenario that doesn't
involve ON CONFLICT at all.

>> * The SQL standard explicitly imposes this limitation:
>>
>> postgres=# explain merge into cities a using cities b on a.city =
>> b.city when matched and (select 1=1) then update set country =
>> b.country;
>> ERROR:  0A000: cannot use subquery in WHEN AND condition
>> LINE 1: ...sing cities b on a.city = b.city when matched and (select 1=...
>>                                                              ^
>> LOCATION:  transformSubLink, parse_expr.c:1865
>>
>> Why do you, though? Do we really need it? I'm just curious about your
>> thoughts on it. To be clear, I'm not asserting that you're wrong.
>
> Which limitation? Not allowing sub-selects? They are not supported, as
> the message says.

I'm simply asking you to explain why you think that it would be
problematic or even impossible to support it. The question is asked
without any agenda. I'm verifying my own understanding, as much as
anything else. I've acknowledged that the standard has something to
say on this that supports your position, which has real weight.

>> * Subselect handling is buggy:
>>
>> postgres=# merge into cities a using cities b on a.city = b.city when
>> matched and a.city = 'Straffan' then update set country = (select
>> 'Some country');
>> ERROR:  XX000: unrecognized node type: 114
>> LOCATION:  ExecInitExprRec, execExpr.c:2114
>
> Not buggy, subselects are not supported in WHEN AND clauses because
> they are not part of the planned query, nor can they be if we want to
> handle the WHEN clause logic per spec.

I'm not asking about WHEN AND here (that was my last question). I'm
asking about a subselect that appears in the targetlist.

(In any case, "unrecognized node type: 114" seems buggy to me in any context.)

>> * Why no CTE support? SQL Server has this.
>
> The SQL Standard doesn't require CTEs or RETURNING syntax, but they
> could in time be supported.

No time like the present. Is there some reason why it would be
difficult with our implementation of CTEs? I can't think why it would
be.

>> But docs imply otherwise -- "source_query -- A query (SELECT statement
>> or VALUES statement) that supplies the rows to be merged into the
>> target_table_name". Either the docs are wrong, or the code is wrong.
>> Hopefully you can just fix the code.
>
> Neither. The docs show that is unsupported. The source query is the
> USING phrase, there is no INSERT SELECT.

My mistake.

>> * Rules are not going to be supported, on the grounds that the
>> behavior is unclear, which I suppose is fine. But what about
>> ruleutils.c support?
>>
>> That seems like entirely another matter to me. What about EXPLAIN,
>> etc? Deparse support seems to be considered generic infrastructure,
>> that doesn't necessarily have much to do with the user-visible rules
>> feature.
>
> The MERGE query is a normal query, so that should all just work.

Look at get_query_def(), for example. That clearly isn't going to work
with MERGE.

> EXPLAIN is specifically regression tested.

You do very little with EXPLAIN right now, though. More importantly, I
think that this is considered a necessary piece of functionality, even
if no core code uses it. There are definitely third-party extensions
that use ruleutils in a fairly broad way.

> I followed the comments of how we handle CHECK constraints.
>
> It's hard to think of a real world example that would use that; your
> example seems strange. Why would you want to use Oids in the WHEN AND
> clause? In the ON or DML clauses, sure, but not there.
>
> This is a non-standard feature, so we can decide whether to support
> that or not. Allowing them is easy, just not very meaningful. Not sure
> if it has consequences.

It's easier to just support them. That way we don't have to think about it.

>> * Wholerow vars are broken:
>>
>> postgres=# merge into testoids a using (select 1 "key", 'foo' "data")
>> b on a.key = b.key when matched then update set data = b.*::text when
>> not matched then insert (key, data) values (1, 'foo');
>> ERROR:  XX000: variable not found in subplan target lists
>> LOCATION:  fix_join_expr_mutator, setrefs.c:2351
>>
>> That's all I have for now.
>
> Good catch; that one is a valid error. I hadn't tried to either
> support them or block them.
>
> Maybe its in the SQL Standard, not sure. Support for whole row vars
> probably isn't a priority though.

I think that this needs to work, on general principle. Again, just
fixing it is much easier than arguing about it.

-- 
Peter Geoghegan

  107. Re: [HACKERS] MERGE SQL Statement for PG11

    Michael Paquier <michael.paquier@gmail.com> — 2018-01-30T00:15:13Z

    On Mon, Jan 29, 2018 at 04:34:48PM +0000, Simon Riggs wrote:
> In terms of timing of commits, I have marked the patch Ready For
> Committer. To me that signifies that it is ready for review by a
> Committer prior to commit.

My understanding of this meaning is different than yours.  It should not
be the author's role to mark his own patch as ready for committer, but
the role of one or more people who have reviewed in-depth the proposed
patch and feature concepts.  If you can get a committer-level individual
to review your patch, then good for you.  But review basics need to
happen first.  And based on my rough lookup of this thread this has not
happened yet.  Other people on this thread are pointing out that as
well.
--
Michael

  108. Re: [HACKERS] MERGE SQL Statement for PG11

    Simon Riggs <simon@2ndquadrant.com> — 2018-01-30T09:16:41Z

    On 29 January 2018 at 17:18, Pavel Stehule <pavel.stehule@gmail.com> wrote:
>
>
> 2018-01-29 18:08 GMT+01:00 Simon Riggs <simon@2ndquadrant.com>:
>>
>> On 29 January 2018 at 16:23, Chapman Flack <chap@anastigmatix.net> wrote:
>> > On 01/29/2018 11:13 AM, Simon Riggs wrote:
>> >> On 29 January 2018 at 15:44, Bruce Momjian <bruce@momjian.us> wrote:
>> >>> Uh, if we know we are going to get question on this, the patch had
>> >>> better have an explanation of when to use it.  Pushing the problem to
>> >>> later doesn't seem helpful.
>> >>
>> >> What problem are you referring to?
>> >>
>> >> INSERT ON CONFLICT UPDATE does ...
>> >>
>> >> MERGE allows you to ...
>> > In my reading of Pavel and Bruce, the only 'problem' being suggested
>> > is that the patch hasn't added a bit of documentation somewhere that
>> > lays out the relationship between these two things, more or less as
>> > you just did.
>>
>> I am happy to write docs as requested.
>>
>> There are currently no docs saying when INSERT ON CONFLICT UPDATE
>> should be used other than the ref page for that command. There is no
>> mention of it in the "Data Manipulation" section of the docs.
>>
>> I've included docs for MERGE so it is mentioned in concurrency and
>> reference sections, so it follows the same model.
>>
>> Where would people like me to put these docs?
>
>
> Depends on size - small note can be placed in MERGE docs and link from
> INSERT ON CONFLICT DO.

I've put in cross-referencing comments in those two places.

v14 attached, with minor additions as requested or notes

Changes
* Add: X-ref docs
* Add: New self-referencing test case
* Add: EXPLAIN ANALYZE in the same style as INSERT .. ON CONFLICT
* Add: Allow Oids to be used in WHEN AND conditions
* Add: Prevent WHEN AND clause from writing data to db, per SQL spec

I'll set up a wiki page to track open items.

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  109. Re: [HACKERS] MERGE SQL Statement for PG11

    Simon Riggs <simon@2ndquadrant.com> — 2018-01-30T09:45:49Z

    On 29 January 2018 at 22:31, Peter Geoghegan <pg@bowt.ie> wrote:

> I don't think that there should be any error, so I can't say.

>> I argued that was possible and desirable, but you argued it was not
>> and got everybody else to agree with you. I'm surprised to see you
>> change your mind on that.
>
> You're mistaken. Nothing I've said here is inconsistent with my
> previous remarks on how we deal with concurrency.

Please see here

https://www.postgresql.org/message-id/20171102191636.GA27644%40marmot

On 2 November 2017 at 19:16, Peter Geoghegan <pg@bowt.ie> wrote:
> Simon Riggs <simon@2ndquadrant.com> wrote:
>>
>> So if I understand you correctly, in your view MERGE should just fail
>> with an ERROR if it runs concurrently with other DML?
>
>
> That's certainly my opinion on the matter. It seems like that might be
> the consensus, too.


You've changed your position, which is good, thanks. No problem at all.

The proposal you make here had already been discussed in detail by
Pavan and myself. My understanding of that discussion was that he
thinks it might be possible, but I had said we must stick to the
earlier agreement on how to proceed. I am willing to try to produce
fewer concurrent errors, since that was an important point from
earlier work.

My only issue now is to make sure this does not cause confusion and
ask if that changes the views of others.



> According to your documentation, "MERGE provides a single SQL
> statement that can conditionally INSERT, UPDATE or DELETE rows, a task
> that would otherwise require multiple procedural language statements".
> But you're introducing a behavior/error that would not occur in
> equivalent procedural client code consisting of an UPDATE followed by
> a (conditionally executed) INSERT statement when run in READ COMMITTED
> mode. You actually get exactly the concurrency issue that you cite as
> unacceptable in justifying your serialization error with such
> procedural code (when the UPDATE didn't affect any rows, only
> following EPQ walking the UPDATE chain from the snapshot-visible
> tuple, making the client code decide to do an INSERT on the basis of
> information "from the future").

"You're introducing a behavior/error"... No, I advocate nothing in the
patch, I am merely following the agreement made here:

https://www.postgresql.org/message-id/CA%2BTgmoYOyX4nyu9mbMdYTLzT9X-1RptxaTKSQfbSdpVGXgeAJQ%40mail.gmail.com

Robert, Stephen, may we attempt to implement option 4 as Peter now suggests?

> 4. Implement MERGE, while attempting to avoid concurrent ERRORs in
> cases where that is possible.

I will discuss in more detail at the Brussels Dev meeting and see if
we can achieve consensus on how to proceed.



v14 posted with changes requested by multiple people. Patch status:
Needs Review.

Current summary: 0 wrong answers; 2 ERRORs raised need better
handling; some open requests for change/enhancement.

I will open a wiki page to track open items by the end of the week.

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  110. Re: [HACKERS] MERGE SQL Statement for PG11

    Robert Haas <robertmhaas@gmail.com> — 2018-01-30T15:59:32Z

    On Mon, Jan 29, 2018 at 10:32 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
> The code is about 1200 lines and has extensive docs, comments and tests.
>
> There are no contentious infrastructure changes, so the debate around
> concurrency is probably the main one. So it looks to me like
> meaningful review has taken place, though I know Andrew and Pavan have
> also looked at it in detail.

Only design-level review, not detailed review of the code.  To be
clear, I think the design-level review was quite productive and I'm
glad it happened, but it's not a substitute for someone going over the
code in detail to look for problems.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  111. Re: [HACKERS] MERGE SQL Statement for PG11

    Robert Haas <robertmhaas@gmail.com> — 2018-01-30T16:01:14Z

    On Mon, Jan 29, 2018 at 7:15 PM, Michael Paquier
<michael.paquier@gmail.com> wrote:
> On Mon, Jan 29, 2018 at 04:34:48PM +0000, Simon Riggs wrote:
>> In terms of timing of commits, I have marked the patch Ready For
>> Committer. To me that signifies that it is ready for review by a
>> Committer prior to commit.
>
> My understanding of this meaning is different than yours.  It should not
> be the author's role to mark his own patch as ready for committer, but
> the role of one or more people who have reviewed in-depth the proposed
> patch and feature concepts.  If you can get a committer-level individual
> to review your patch, then good for you.  But review basics need to
> happen first.  And based on my rough lookup of this thread this has not
> happened yet.  Other people on this thread are pointing out that as
> well.

+1 to all of that.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  112. Re: [HACKERS] MERGE SQL Statement for PG11

    Robert Haas <robertmhaas@gmail.com> — 2018-01-30T16:15:36Z

    On Mon, Jan 29, 2018 at 12:03 PM, Simon Riggs <simon@2ndquadrant.com> wrote:
> Partitioning doesn't look too bad, so that looks comfortable for PG11,
> assuming it doesn't hit some unhandled complexity.
>
> Including RLS in the first commit/release turns this into a high risk
> patch. Few people use it, but if they do, they don't want me putting a
> hole in their battleship (literally) should we discover some weird
> unhandled logic in a complex new command.
>
> My recommendation would be to support that later for those that use
> it. For those that don't, it doesn't matter so can also be done later.

-1.  Every other feature we've added recently, including partitioning,
has had to decide what to do about RLS before the initial commit, and
this feature shouldn't be exempt.  In general, newer features need to
work with older features unless there is some extremely good
architectural reason why that is unreasonably difficult.  If that is
the case here, I don't see that you've made an argument for it.  The
proper way to avoid having you put a hole in their battleship is good
code, proper code review, and good testing, not leaving that case off
to one side.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  113. Re: [HACKERS] MERGE SQL Statement for PG11

    Robert Haas <robertmhaas@gmail.com> — 2018-01-30T16:27:26Z

    On Tue, Jan 30, 2018 at 4:45 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
> "You're introducing a behavior/error"... No, I advocate nothing in the
> patch, I am merely following the agreement made here:
>
> https://www.postgresql.org/message-id/CA%2BTgmoYOyX4nyu9mbMdYTLzT9X-1RptxaTKSQfbSdpVGXgeAJQ%40mail.gmail.com
>
> Robert, Stephen, may we attempt to implement option 4 as Peter now suggests?

Simon, I don't think you should represent Peter as having changed his
position when he has already said that he didn't.  To be honest, the
discussion up to this point suggests to me that Peter has a
significantly better grip of the issues than you do, yet your posts
convey, at least to me, that you're quite sure he's wrong about a lot
of stuff, including whether or not his current statements are
compatible with his previous statements.  I think that the appropriate
course of action is for you and he to spend a few more emails trying
to actually get on the same page here.

As far as I am able to understand, the substantive issue here is what
to do when we match an invisible tuple rather than a visible tuple.
The patch currently throws a serialization error on the basis that you
(Simon) thought that's what was previously agreed.  Peter is arguing
that we don't normally issue a serialization error at READ COMMITTED
(which I think is true) and proposed that we instead try to INSERT.  I
don't necessarily think that's different from consensus to implement
option #3 from https://www.postgresql.org/message-id/CA%2BTgmoYOyX4nyu9mbMdYTLzT9X-1RptxaTKSQfbSdpVGXgeAJQ%40mail.gmail.com
because that point #3 says that we're not going to try to AVOID errors
under concurrency, not that we're going to create NEW errors.  In
other words, I understand Peter, then and now, to be saying that MERGE
should behave just as if invisible tuples didn't exist at all; if that
leads some other part of the system to throw an ERROR, then that's
what happens.  Presumably, in a case like this, that would be a common
outcome, because the merge would be performed on the basis of a unique
key and so inserting would trigger a duplicate key violation.  But
maybe not, because I don't think MERGE requires there to be a unique
key on that column, so maybe the insert would just work, or maybe the
conflicting transaction would abort just in time to let it work
anyway.

It is possible that I am confused, here, among other reasons because I
haven't read the code, but if I'm not confused then Peter's analysis
is correct.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  114. Re: [HACKERS] MERGE SQL Statement for PG11

    Simon Riggs <simon@2ndquadrant.com> — 2018-01-30T16:56:42Z

    On 30 January 2018 at 16:27, Robert Haas <robertmhaas@gmail.com> wrote:

> As far as I am able to understand, the substantive issue here is what
> to do when we match an invisible tuple rather than a visible tuple.
> The patch currently throws a serialization error on the basis that you
> (Simon) thought that's what was previously agreed.

Correct. We discussed this and agreed point (3) below

> 3. Implement MERGE, but without attempting to avoid concurrent ERRORs (Peter)
>
> 4. Implement MERGE, while attempting to avoid concurrent ERRORs in
> cases where that is possible.

Acting in good faith, in respect of all of your wishes, I implemented
exactly that and not what I had personally argued in favour of.

> Peter is arguing
> that we don't normally issue a serialization error at READ COMMITTED
> (which I think is true) and proposed that we instead try to INSERT.

Which IMHO is case 4 since it would avoid a concurrent ERROR. This
meets exactly my original implementation goals as clearly stated on
this thread, so of course I agree with him and have already said I am
happy to change the code, though I am still wary of the dangers he
noted upthread.

If you now agree with doing that and are happy that there are no
dangers, then I'm happy we now have consensus again and we can
continue implementing MERGE for PG11.

This is a good outcome, thanks, our users will be happy.

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  115. Re: [HACKERS] MERGE SQL Statement for PG11

    Robert Haas <robertmhaas@gmail.com> — 2018-01-30T19:02:06Z

    On Tue, Jan 30, 2018 at 11:56 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
> Which IMHO is case 4 since it would avoid a concurrent ERROR. This
> meets exactly my original implementation goals as clearly stated on
> this thread, so of course I agree with him and have already said I am
> happy to change the code, though I am still wary of the dangers he
> noted upthread.
>
> If you now agree with doing that and are happy that there are no
> dangers, then I'm happy we now have consensus again and we can
> continue implementing MERGE for PG11.

I can't certify that there are no dangers because I haven't studied it
in that much detail, and I still don't think this is the same thing as
#4 for the reasons I already stated.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  116. Re: [HACKERS] MERGE SQL Statement for PG11

    Peter Geoghegan <pg@bowt.ie> — 2018-01-30T19:28:13Z

    On Tue, Jan 30, 2018 at 8:27 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> As far as I am able to understand, the substantive issue here is what
> to do when we match an invisible tuple rather than a visible tuple.
> The patch currently throws a serialization error on the basis that you
> (Simon) thought that's what was previously agreed.  Peter is arguing
> that we don't normally issue a serialization error at READ COMMITTED
> (which I think is true) and proposed that we instead try to INSERT.  I
> don't necessarily think that's different from consensus to implement
> option #3 from https://www.postgresql.org/message-id/CA%2BTgmoYOyX4nyu9mbMdYTLzT9X-1RptxaTKSQfbSdpVGXgeAJQ%40mail.gmail.com
> because that point #3 says that we're not going to try to AVOID errors
> under concurrency, not that we're going to create NEW errors.

> In other words, I understand Peter, then and now, to be saying that MERGE
> should behave just as if invisible tuples didn't exist at all; if that
> leads some other part of the system to throw an ERROR, then that's
> what happens.

Yes, I am still saying that.

What's at issue here specifically is the exact behavior of
EvalPlanQual() in the context of having *multiple* sets of WHEN quals
that need to be evaluated one at a time (in addition to conventional
EPQ join quals). This is a specific, narrow question about the exact
steps that are taken by EPQ when we have to switch between WHEN
MATCHED and WHEN NOT MATCHED cases *as we walk the UPDATE chain*.

Right now, I suspect that we will require some minor variation of
EPQ's logic to account for new risks. The really interesting question
is what happens when we walk the UPDATE chain, while reevaluating EPQ
quals alongside WHEN quals, and then determine that no UPDATE/DELETE
should happen for the first WHEN case -- what then? I suspect that we
may not want to start from scratch (from the MVCC-visible tuple) as we
reach the second or subsequent WHEN case, but that's a very tentative
view, and I definitely want to hear more opinions it. (Simon wants to
just throw a serialization error here instead, even in READ COMMITTED
mode, which I see as a cop-out.)

Note in particular that this EPQ question has nothing to do with
seeing tuples that are not either visible to our MVCC snapshot, or
visible to EPQ through an UPDATE chain (which starts from the MVCC
visible tuple). The idea that I have done some kind of about-face on
how concurrency should work is just plain wrong. It is not a helpful
way of framing things. What I am talking about here is very
complicated, but also really narrow.

> Presumably, in a case like this, that would be a common
> outcome, because the merge would be performed on the basis of a unique
> key and so inserting would trigger a duplicate key violation.  But
> maybe not, because I don't think MERGE requires there to be a unique
> key on that column, so maybe the insert would just work, or maybe the
> conflicting transaction would abort just in time to let it work
> anyway.

I think that going on to INSERT having decided against an UPDATE only
having done an EPQ walk (rather than throwing a serialization error)
is very likely to result in the INSERT succeeding, actually. But there
is no guarantee that you won't get a duplicate violation, because
there is nothing to stop a concurrent *INSERT* with the same PK value.
(That's something that's *always* true, regardless of whether or not
somebody needs to do EPQ.)

-- 
Peter Geoghegan

  117. Re: [HACKERS] MERGE SQL Statement for PG11

    Peter Geoghegan <pg@bowt.ie> — 2018-01-30T21:47:44Z

    On Tue, Jan 30, 2018 at 8:56 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
>> Peter is arguing
>> that we don't normally issue a serialization error at READ COMMITTED
>> (which I think is true) and proposed that we instead try to INSERT.
>
> Which IMHO is case 4 since it would avoid a concurrent ERROR.

It would avoid the error that you added in v13 of your patch, a type
of error that I never proposed or even considered.

> This meets exactly my original implementation goals as clearly stated on
> this thread

I'm glad that you see it that way. I didn't and don't, though I
objected to your characterization mostly because it muddied some
already rather muddy waters. I'm happy to let it go now, though.

> If you now agree with doing that and are happy that there are no
> dangers, then I'm happy we now have consensus again and we can
> continue implementing MERGE for PG11.

My outline from earlier today about EPQ handling, and how it needs to
deal with multiple independent sets of WHEN ... AND quals is, as I
said, very tentative. There isn't even consensus in my own mind about
this, much less between everyone that has taken an interest in this
project.

I'm glad that we all seem to agree that serialization failures as a
way of dealing with concurrency issues in READ COMMITTED mode are a
bad idea. Unfortunately, I still think that we have a lot of work
ahead of us when it comes to agreeing to the right semantics with READ
COMMITTED conflict handling with multiple WHEN ... AND quals.

I see that your v14 still has the serialization error, even though
it's now clear that nobody wants to go that way. So...where do we go
from here? (For the avoidance of doubt, this is *not* a rhetorical
question.)

-- 
Peter Geoghegan

  118. Re: [HACKERS] MERGE SQL Statement for PG11

    Peter Geoghegan <pg@bowt.ie> — 2018-01-31T00:02:51Z

    On Tue, Jan 30, 2018 at 8:27 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> As far as I am able to understand, the substantive issue here is what
> to do when we match an invisible tuple rather than a visible tuple.
> The patch currently throws a serialization error on the basis that you
> (Simon) thought that's what was previously agreed.  Peter is arguing
> that we don't normally issue a serialization error at READ COMMITTED
> (which I think is true) and proposed that we instead try to INSERT.  I
> don't necessarily think that's different from consensus to implement
> option #3 from https://www.postgresql.org/message-id/CA%2BTgmoYOyX4nyu9mbMdYTLzT9X-1RptxaTKSQfbSdpVGXgeAJQ%40mail.gmail.com
> because that point #3 says that we're not going to try to AVOID errors
> under concurrency, not that we're going to create NEW errors.

I should have mentioned earlier that you have this exactly right: I do
not want to make any special effort to avoid duplicate violation
errors. I also don't want to create any novel new kind of error (e.g.,
READ COMMITTED serialization errors).

That having been said, I think that Simon was correct to propose a
novel solution. It just seemed like READ COMMITTED serialization
errors were the wrong novel solution, because that takes the easy way
out. ISTM that the right thing is to adapt EvalPlanQual() (or READ
COMMITTED conflict handling more generally) to the new complication
that is multiple "WHEN ... AND" quals (that need to be evaluated one
at a time, a bona fide new requirement). In short, his novel solution
seemed much too novel.

As I've pointed out already, we will define MERGE to users as
something that "provides a single SQL statement that can conditionally
INSERT, UPDATE or DELETE rows, a task that would otherwise require
multiple procedural language statements". I believe that MERGE's
charter should be to live up to that definition in the least
surprising way possible, up to and including preserving the
maybe-surprising aspects of how multiple procedural language
statements can behave when the system does READ COMMITTED conflict
handling. That's my opinion in a nutshell.

-- 
Peter Geoghegan

  119. Re: [HACKERS] MERGE SQL Statement for PG11

    Robert Haas <robertmhaas@gmail.com> — 2018-01-31T15:17:37Z

    On Tue, Jan 30, 2018 at 2:28 PM, Peter Geoghegan <pg@bowt.ie> wrote:
> What's at issue here specifically is the exact behavior of
> EvalPlanQual() in the context of having *multiple* sets of WHEN quals
> that need to be evaluated one at a time (in addition to conventional
> EPQ join quals). This is a specific, narrow question about the exact
> steps that are taken by EPQ when we have to switch between WHEN
> MATCHED and WHEN NOT MATCHED cases *as we walk the UPDATE chain*.
>
> Right now, I suspect that we will require some minor variation of
> EPQ's logic to account for new risks. The really interesting question
> is what happens when we walk the UPDATE chain, while reevaluating EPQ
> quals alongside WHEN quals, and then determine that no UPDATE/DELETE
> should happen for the first WHEN case -- what then? I suspect that we
> may not want to start from scratch (from the MVCC-visible tuple) as we
> reach the second or subsequent WHEN case, but that's a very tentative
> view, and I definitely want to hear more opinions it. (Simon wants to
> just throw a serialization error here instead, even in READ COMMITTED
> mode, which I see as a cop-out.)

I don't fully grok merge but suppose you have:

WHEN MATCHED AND a = 0 THEN UPDATE ...
WHEN MATCHED AND a = 1 THEN UPDATE ...
WHEN NOT MATCHED THEN INSERT ...

Suppose you match a tuple with a = 0 but, upon trying to update it,
find that it's been updated to a = 1.  It seems like there are a few
possible behaviors:

1. Throw an error!  I guess this is what the patch does now.

2. Do absolutely nothing.  I think this is what would happen with an
ordinary UPDATE; the tuple fails the EPQ recheck and so is not
updated, but that doesn't trigger anything else.

3. Fall through to the NOT MATCHED clause and try that instead.
Allows MERGE to work as UPSERT in some simple cases, I think.

4. Continue walking the chain of WHEN MATCHED items in order and test
them against the new tuple.  This is actually pretty weird because a
0->1 update will fall through to the second UPDATE rule, but a 1->0
update will fall through to the NOT MATCHED clause.

5. Retry from the top of the chain with the updated tuple.  Could
theoretically livelock - not sure how much of a risk that is in
practice.

Maybe there are more options?

My initial reaction is to wonder what's wrong with #2.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  120. Re: [HACKERS] MERGE SQL Statement for PG11

    Peter Geoghegan <pg@bowt.ie> — 2018-01-31T18:07:11Z

    On Wed, Jan 31, 2018 at 7:17 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> I don't fully grok merge but suppose you have:
>
> WHEN MATCHED AND a = 0 THEN UPDATE ...
> WHEN MATCHED AND a = 1 THEN UPDATE ...
> WHEN NOT MATCHED THEN INSERT ...
>
> Suppose you match a tuple with a = 0 but, upon trying to update it,
> find that it's been updated to a = 1.  It seems like there are a few
> possible behaviors:
>
> 1. Throw an error!  I guess this is what the patch does now.

Right.

> 2. Do absolutely nothing.  I think this is what would happen with an
> ordinary UPDATE; the tuple fails the EPQ recheck and so is not
> updated, but that doesn't trigger anything else.

I think #2 is fine if you're talking about join quals. Which, of
course, you're not. These WHEN quals really do feel like
tuple-at-a-time procedural code, more than set-orientated quals (if
that wasn't true, we'd have to allow cardinality violations, which we
at least try to avoid). Simon said something like "the SQL standard
requires that WHEN quals be evaluated first" at one point, which makes
sense to me.

> 3. Fall through to the NOT MATCHED clause and try that instead.
> Allows MERGE to work as UPSERT in some simple cases, I think.

I probably wouldn't put it that way myself, FWIW.

> 4. Continue walking the chain of WHEN MATCHED items in order and test
> them against the new tuple.  This is actually pretty weird because a
> 0->1 update will fall through to the second UPDATE rule, but a 1->0
> update will fall through to the NOT MATCHED clause.

You have two WHEN MATCHED cases here, which is actually quite a
complicated example. If you didn't, then IIUC there would be no
distinction between #3 and #4.

Whether or not this "pretty weird" behavior would be weirder than
equivalent procedural code consisting of multiple (conditionally
executed) DML statements is subjective. If you imagine that the
equivalent procedural code is comprised of two UPDATE statements and
an INSERT (one statement for every WHEN case), then it's not weird, I
think, or at least is no weirder. If you imagine that it's only one
UPDATE (plus an INSERT), then is does indeed seem weirder.

I'm inclined to think of it as two UPDATE statements (that can only
affect zero or one tuples) rather than one (the statements are inside
a loop that processes many input rows, equivalent to the left side of
MERGE's join). After all, it seems very likely that one of the two
WHEN MATCHED items would actually end up containing a DELETE in real
world queries, and not another UPDATE. That's why I lean towards #4
ever so slightly right now.

> 5. Retry from the top of the chain with the updated tuple.  Could
> theoretically livelock - not sure how much of a risk that is in
> practice.

I'd say the livelock risk is non-zero, but it might still be worth it.
Isn't this like rolling back and repeating the statement in most
real-world cases?

Apparently READ COMMITTED conflict handling in a system that's similar
to Postgres occurs through a statement-level rollback. A little bird
told me that it can repeat again and again, and that there is a little
known mechanism for that to eventually error out after a fixed number
of retries. It might be desirable to emulate that in a rudimentary way
-- by implementing #5. This doesn't seem all that appealing to me
right now, though.

> Maybe there are more options?

Probably.

Minor point on semantics: There is clearly a two phase nature to WHEN
quals, which is the actual structure that Simon chose. Technically,
what you described wouldn't ever require EPQ recheck -- it might
require WHEN recheck. I think we should start being careful about
which we're talking about going forward. Hopefully Simon's MERGE wiki
page can establish a standard lexicon to talk about this stuff without
everyone becoming even more confused. That seems like it would be a
big help.

-- 
Peter Geoghegan

  121. Re: [HACKERS] MERGE SQL Statement for PG11

    Simon Riggs <simon@2ndquadrant.com> — 2018-02-01T11:07:49Z

    On 31 January 2018 at 15:17, Robert Haas <robertmhaas@gmail.com> wrote:
> On Tue, Jan 30, 2018 at 2:28 PM, Peter Geoghegan <pg@bowt.ie> wrote:
>> What's at issue here specifically is the exact behavior of
>> EvalPlanQual() in the context of having *multiple* sets of WHEN quals
>> that need to be evaluated one at a time (in addition to conventional
>> EPQ join quals). This is a specific, narrow question about the exact
>> steps that are taken by EPQ when we have to switch between WHEN
>> MATCHED and WHEN NOT MATCHED cases *as we walk the UPDATE chain*.
>>
>> Right now, I suspect that we will require some minor variation of
>> EPQ's logic to account for new risks. The really interesting question
>> is what happens when we walk the UPDATE chain, while reevaluating EPQ
>> quals alongside WHEN quals, and then determine that no UPDATE/DELETE
>> should happen for the first WHEN case -- what then? I suspect that we
>> may not want to start from scratch (from the MVCC-visible tuple) as we
>> reach the second or subsequent WHEN case, but that's a very tentative
>> view, and I definitely want to hear more opinions it. (Simon wants to
>> just throw a serialization error here instead, even in READ COMMITTED
>> mode, which I see as a cop-out.)
>
> I don't fully grok merge but suppose you have:
>
> WHEN MATCHED AND a = 0 THEN UPDATE ...
> WHEN MATCHED AND a = 1 THEN UPDATE ...
> WHEN NOT MATCHED THEN INSERT ...
>
> Suppose you match a tuple with a = 0 but, upon trying to update it,
> find that it's been updated to a = 1.  It seems like there are a few
> possible behaviors:
>
> 1. Throw an error!  I guess this is what the patch does now.
>
> 2. Do absolutely nothing.  I think this is what would happen with an
> ordinary UPDATE; the tuple fails the EPQ recheck and so is not
> updated, but that doesn't trigger anything else.
>
> 3. Fall through to the NOT MATCHED clause and try that instead.
> Allows MERGE to work as UPSERT in some simple cases, I think.
>
> 4. Continue walking the chain of WHEN MATCHED items in order and test
> them against the new tuple.  This is actually pretty weird because a
> 0->1 update will fall through to the second UPDATE rule, but a 1->0
> update will fall through to the NOT MATCHED clause.
>
> 5. Retry from the top of the chain with the updated tuple.  Could
> theoretically livelock - not sure how much of a risk that is in
> practice.
>
> Maybe there are more options?
>
> My initial reaction is to wonder what's wrong with #2.

No, we don't throw an ERROR in that case, because it is a simple
variation of existing EvalPlanQual behavior.

#2 is possible, yes, and is how we had it coded in v11.

#4 was how I first assumed it had to work, but it gives the wrong
answer in some cases and right answer in others, depending upon order
of WHEN clauses. That was ruled out as inconsistent.

#5 is what the patch does now. There are tests covering that behavior
in specs/merge-match-recheck.spec


There are more complex cases to consider.

If a concurrent DELETE hits, then we can try #3, i.e. changing MATCHED
to NOT MATCHED. That currently throws an error, as requested. It looks
to be possible, but it would require some variation of EvalPlanQual.
My prototype of that doesn't yet work, so I can't yet confirm whether
it is even possible. If it is, I will submit as an option for PG11.

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  122. Re: [HACKERS] MERGE SQL Statement for PG11

    Simon Riggs <simon@2ndquadrant.com> — 2018-02-01T12:45:09Z

    On 30 January 2018 at 21:47, Peter Geoghegan <pg@bowt.ie> wrote:

> I'm glad that we all seem to agree that serialization failures as a
> way of dealing with concurrency issues in READ COMMITTED mode are a
> bad idea.

ERRORs are undesirable, yet safe and correct. Doing better is as yet
unclear if it can be done correctly in all cases, or whether a
practical subset exists.

> Unfortunately, I still think that we have a lot of work
> ahead of us when it comes to agreeing to the right semantics with READ
> COMMITTED conflict handling with multiple WHEN ... AND quals.

OK

> I see that your v14 still has the serialization error, even though
> it's now clear that nobody wants to go that way. So...where do we go
> from here? (For the avoidance of doubt, this is *not* a rhetorical
> question.)

This way forward is new. We're trying it, but it may not be possible.
I haven't made it work yet.

If we can find a way to do this, we will. If you want to propose some
code, please do.


I think it would be very helpful if we could discuss everything with
direct relevance to v14, so this becomes a patch review, not just a
debate.
i.e. which isolation test would we like to change from ERROR to
success? or which new test would you like to add?

Thanks

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  123. Re: [HACKERS] MERGE SQL Statement for PG11

    Simon Riggs <simon@2ndquadrant.com> — 2018-02-01T15:21:10Z

    On 1 February 2018 at 12:45, Simon Riggs <simon@2ndquadrant.com> wrote:

> I think it would be very helpful if we could discuss everything with
> direct relevance to v14, so this becomes a patch review, not just a
> debate.
> i.e. which isolation test would we like to change from ERROR to
> success? or which new test would you like to add?
>
> Thanks

In my understanding, we are discussing changing the potential outcome
from concurrent operations on the small subset of test results noted
in the attached patch-on-patch. (This will break the patch tester)

If you can confirm these are the ones we are discussing and say what
you think the output should be, that will help us be very specific .

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  124. Re: [HACKERS] MERGE SQL Statement for PG11

    Peter Geoghegan <pg@bowt.ie> — 2018-02-01T19:39:23Z

    On Thu, Feb 1, 2018 at 4:45 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
> I think it would be very helpful if we could discuss everything with
> direct relevance to v14, so this becomes a patch review, not just a
> debate.

I wish I could give you a clear answer on the way forward with total
confidence, or even moderate confidence, but right now I can't.
Hopefully I'll be able to convince myself that I've understood all the
nuances of this EPQ + WHEN ... AND qual business shortly, perhaps in
the next couple of days, at which point I'll have a position that I'm
willing to defend. I don't even have that much right now.

On a more positive note, I do agree with you that this only affects a
small subset of real-world queries (even if we assume READ COMMITTED
conflicts are common). To put it another way, I would be able to give
you a simple opinion right now if WHEN ... AND quals were prohibited.
Not that I'm proposing actually prohibiting them. (BTW, don't you
think it's interesting that Oracle doesn't allow them, but instead
requires WHERE clauses, even on an INSERT?)

Leaving concurrency aside for a moment, I want to talk about bugs. I
read through this:

https://wiki.postgresql.org/wiki/SQL_MERGE_Patch_Status#Bugs

This suggestion you make, which is that the only problem with wholerow
vars is that there is a weird error message (and they're actually
unsupported) is not helping your cause. While I think that users will
expect that to work, that isn't really the main point. I tried using
wholerow vars as a smoke test for setrefs.c handling, as well as the
handling of targetlists within the rewriter. The fact that wholerow
vars don't work is likely indicative of your general approach in these
areas needing to be thought through in more detail. That's the really
important thing. Wholerow vars shouldn't be a special case to the
underlying implementation, and if you found a way to make them work
that was a special case that would seem questionable to me for similar
reasons. Sometimes wholerow vars actually *do* work with your patch --
the problem only happens with data_source whole-row vars, and never
target_table_name wholerow vars:

postgres=# merge into testoids a using (select i "key", 'foo' "data"
from generate_series(0,3) i) b on a.key = b.key when matched then
update set data = a.*::text when not matched then insert (key, data)
values (b.key, 'foo');
MERGE 4
postgres=# merge into testoids a using (select i "key", 'foo' "data"
from generate_series(0,3) i) b on a.key = b.key when matched then
update set data = b.*::text when not matched then insert (key, data)
values (b.key, 'foo');
ERROR:  variable not found in subplan target lists

There is also the matter of subselects in the update targetlist, which
you similarly claim is only a problem of having the wrong error
message. The idea that those are unsupported for any principled reason
doesn't have any justification (unlike WHEN ... AND quals, and their
restrictions, which I agree are necessary). It clearly works with
Oracle, for example:

http://sqlfiddle.com/#!4/2d5405/10

You're reusing set_clause_list in the grammar, so I don't see why it
shouldn't work within MERGE in just the same way as it works in
UPDATE. While I think that there is a legitimate need for restrictions
on some merge_when_clause cases, such as the VALUES() of merge_insert,
this isn't an example of that. Again, this suggests to me a need for
more work within the optimizer.

Finally, I noticed a problem with your new EXPLAIN ANALYZE instrumentation:

Is it 4 rows inserted, or 0 inserted?

postgres=# merge into testoids a using (select i "key", 'foo' "data"
from generate_series(0,3) i) b on a.key = b.key when matched and 1=0
then update set data = b.data when not matched then insert (key, data)
values (b.key, 'foo');
MERGE 0
postgres=# explain analyze merge into testoids a using (select i
"key", 'foo' "data" from generate_series(0,3) i) b on a.key = b.key
when matched and 1=0 then update set data = b.data when not matched
then insert (key, data) values (b.key, 'foo');
                                                           QUERY PLAN
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 Merge on testoids a  (cost=38.58..61.19 rows=1000 width=42) (actual
time=0.043..0.049 rows=4 loops=1)
   Tuples Inserted: 4
   Tuples Updated: 0
   Tuples Deleted: 0
   ->  Hash Left Join  (cost=38.58..61.19 rows=1000 width=42) (actual
time=0.039..0.043 rows=4 loops=1)
         Hash Cond: (i.i = a.key)
         ->  Function Scan on generate_series i  (cost=0.00..10.00
rows=1000 width=4) (actual time=0.009..0.010 rows=4 loops=1)
         ->  Hash  (cost=22.70..22.70 rows=1270 width=10) (actual
time=0.021..0.021 rows=4 loops=1)
               Buckets: 2048  Batches: 1  Memory Usage: 17kB
               ->  Seq Scan on testoids a  (cost=0.00..22.70 rows=1270
width=10) (actual time=0.014..0.016 rows=4 loops=1)
 Planning time: 0.202 ms
 Execution time: 0.109 ms
(12 rows)

(It should be 0 rows inserted, not 4.)

-- 
Peter Geoghegan

  125. Re: [HACKERS] MERGE SQL Statement for PG11

    Peter Geoghegan <pg@bowt.ie> — 2018-02-02T01:59:33Z

    On Thu, Feb 1, 2018 at 11:39 AM, Peter Geoghegan <pg@bowt.ie> wrote:
> There is also the matter of subselects in the update targetlist, which
> you similarly claim is only a problem of having the wrong error
> message. The idea that those are unsupported for any principled reason
> doesn't have any justification (unlike WHEN ... AND quals, and their
> restrictions, which I agree are necessary). It clearly works with
> Oracle, for example:
>
> http://sqlfiddle.com/#!4/2d5405/10
>
> You're reusing set_clause_list in the grammar, so I don't see why it
> shouldn't work within MERGE in just the same way as it works in
> UPDATE.

Actually, I now wonder if there is a good reason for restrictions
(e.g. no subselects) on WHEN ... AND quals, too. See this SQL fiddle
from SQL Server:

http://sqlfiddle.com/#!18/8acef/27

I started looking at SQL Server's MERGE to verify that it also does
not impose any restrictions on subselects in a MERGE UPDATE's
targetlist, just like Oracle. Unsurprisingly, it does not. More
surprisingly, I noticed that it also doesn't seem to impose
restrictions on what can appear in WHEN ... AND quals. Most
surprisingly of all, even the main join ON condition itself can have
subselects (though that's probably a bad idea).

What this boils down to is that I don't think that this part of your
design is committable (from your recent v14):

>> +    * As top-level statements INSERT, UPDATE and DELETE have a Query,
>> +    * whereas with MERGE the individual actions do not require
>> +    * separate planning, only different handling in the executor.
>> +    * See nodeModifyTable handling of commandType CMD_MERGE.
>> +    *
>> +    * A sub-query can include the Target, but otherwise the sub-query
>> +    * cannot reference the outermost Target table at all.
>> +    */
>> +   qry->querySource = QSRC_PARSER;
>> +   joinexpr = makeNode(JoinExpr);
>> +   joinexpr->isNatural = false;
>> +   joinexpr->alias = NULL;
>> +   joinexpr->usingClause = NIL;
>> +   joinexpr->quals = stmt->join_condition;

I am willing to continue to engage with you on the concurrency issues
for the time being, since that is the most pressing issue for the
patch. We can get to this stuff later. Note that I consider cleaning
up the Query and plan representations to be prerequisite to commit.

-- 
Peter Geoghegan

  126. Re: [HACKERS] MERGE SQL Statement for PG11

    Andreas Seltenreich <seltenreich@gmx.de> — 2018-02-03T19:57:24Z

    Hi,

as promised in Brussels, I taught sqlsmith about MERGE and did some
testing with merge.v14.patch applied on master at 9aef173163.

So far, it found a couple of failing assertions and a suspicous error
message.  Details and Testcases against the regression database below.

regards,
Andreas

-- TRAP: FailedAssertion("!(!((((const Node*)(node))->type) == T_SubLink))", File: "clauses.c", Line: 440)
MERGE INTO public.brin_test as target_0
USING pg_catalog.pg_database as ref_0
  left join pg_catalog.pg_user_mapping as sample_0 tablesample system (2.3)
  on (pg_catalog.mul_d_interval(
        cast(pg_catalog.pi() as float8),
        cast(case when sample_0.umoptions is not NULL then (select write_lag from pg_catalog.pg_stat_replication limit 1 offset 2)
             else (select write_lag from pg_catalog.pg_stat_replication limit 1 offset 2)
             end
           as "interval")) = (select intervalcol from public.brintest limit 1 offset 2)
        )
ON target_0.a = ref_0.encoding
WHEN NOT MATCHED AND cast(null as "timestamp") < cast(null as date)
   THEN INSERT VALUES ( 62, 6)
WHEN NOT MATCHED
  AND false
   THEN DO NOTHING;

-- TRAP: FailedAssertion("!(!((((const Node*)(node))->type) == T_SubLink))", File: "prepunion.c", Line: 2246)
MERGE INTO public.onek2 as target_0
USING public.prt1 as ref_0
  inner join public.tenk1 as ref_1
  on ((select t from public.btree_tall_tbl limit 1 offset 63)
         is not NULL)
ON target_0.stringu1 = ref_1.stringu1
WHEN NOT MATCHED THEN DO NOTHING;

-- TRAP: FailedAssertion("!(!((((const Node*)(node))->type) == T_Query))", File: "var.c", Line: 248)
MERGE INTO public.clstr_tst_inh as target_0
USING pg_catalog.pg_statio_sys_tables as ref_0
  left join public.rule_and_refint_t3 as ref_1
  on (((ref_0.heap_blks_hit is not NULL)
        or (((select f1 from public.path_tbl limit 1 offset 5)
               >= (select thepath from public.shighway limit 1 offset 33)
              )
          or (cast(null as tsvector) <> cast(null as tsvector))))
      and (ref_0.toast_blks_read is not NULL))
ON target_0.d = ref_1.data
WHEN NOT MATCHED
  AND cast(null as int2) = pg_catalog.lastval()
   THEN DO NOTHING;

-- ERROR:  unrecognized node type: 114
MERGE INTO public.found_test_tbl as target_0
USING public.itest7a as ref_0
ON target_0.a = ref_0.a
WHEN NOT MATCHED
   THEN INSERT VALUES ((select a from public.rtest_t3 limit 1 offset 6));

  127. Re: [HACKERS] MERGE SQL Statement for PG11

    Simon Riggs <simon@2ndquadrant.com> — 2018-02-03T22:15:12Z

    On 2 February 2018 at 01:59, Peter Geoghegan <pg@bowt.ie> wrote:
> On Thu, Feb 1, 2018 at 11:39 AM, Peter Geoghegan <pg@bowt.ie> wrote:
>> There is also the matter of subselects in the update targetlist, which
>> you similarly claim is only a problem of having the wrong error
>> message. The idea that those are unsupported for any principled reason
>> doesn't have any justification (unlike WHEN ... AND quals, and their
>> restrictions, which I agree are necessary). It clearly works with
>> Oracle, for example:
>>
>> http://sqlfiddle.com/#!4/2d5405/10
>>
>> You're reusing set_clause_list in the grammar, so I don't see why it
>> shouldn't work within MERGE in just the same way as it works in
>> UPDATE.
>
> Actually, I now wonder if there is a good reason for restrictions
> (e.g. no subselects) on WHEN ... AND quals, too. See this SQL fiddle
> from SQL Server:
>
> http://sqlfiddle.com/#!18/8acef/27
>
> I started looking at SQL Server's MERGE to verify that it also does
> not impose any restrictions on subselects in a MERGE UPDATE's
> targetlist, just like Oracle. Unsurprisingly, it does not. More
> surprisingly, I noticed that it also doesn't seem to impose
> restrictions on what can appear in WHEN ... AND quals.

You earlier agreed that subselects were not part of the Standard.

> Most
> surprisingly of all, even the main join ON condition itself can have
> subselects (though that's probably a bad idea).

That should be supported, though I can't think of why you'd want that either.

> What this boils down to is that I don't think that this part of your
> design is committable (from your recent v14):

So your opinion is that because v14 patch doesn't include a feature
extension that is in Oracle and SQLServer that we cannot commit this
patch.

There are quite a few minor additional things in that category and the
syntax of those two differ, so its clearly impossible to match both
exactly.

That seems like poor reasoning on why we should block the patch.

If you would like to say how you think the design should look, it
might be possible to change it for this release. Changing it in the
future would not be blocked by commiting without that.

>>> +    * As top-level statements INSERT, UPDATE and DELETE have a Query,
>>> +    * whereas with MERGE the individual actions do not require
>>> +    * separate planning, only different handling in the executor.
>>> +    * See nodeModifyTable handling of commandType CMD_MERGE.
>>> +    *
>>> +    * A sub-query can include the Target, but otherwise the sub-query
>>> +    * cannot reference the outermost Target table at all.
>>> +    */
>>> +   qry->querySource = QSRC_PARSER;
>>> +   joinexpr = makeNode(JoinExpr);
>>> +   joinexpr->isNatural = false;
>>> +   joinexpr->alias = NULL;
>>> +   joinexpr->usingClause = NIL;
>>> +   joinexpr->quals = stmt->join_condition;
>
> I am willing to continue to engage with you on the concurrency issues
> for the time being, since that is the most pressing issue for the
> patch. We can get to this stuff later.

There are no concurrency issues. The patch gives the correct answer in
all cases, or an error to avoid problems. We've agreed that it is
desirable we remove some of those if possible, though they are there
as a result of our earlier discussions.

> Note that I consider cleaning
> up the Query and plan representations to be prerequisite to commit.

You'll need to be more specific. Later patches do move some things.

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  128. Re: [HACKERS] MERGE SQL Statement for PG11

    Simon Riggs <simon@2ndquadrant.com> — 2018-02-03T22:18:03Z

    On 3 February 2018 at 19:57, Andreas Seltenreich <seltenreich@gmx.de> wrote:

> as promised in Brussels, I taught sqlsmith about MERGE and did some
> testing with merge.v14.patch applied on master at 9aef173163.
>
> So far, it found a couple of failing assertions and a suspicous error
> message.  Details and Testcases against the regression database below.

Brilliant work, thank you.

It will likely take some time to work through these and the current
work items but will fix.

Do you have the DDL so we can recreate these easily?

Thanks

> regards,
> Andreas
>
> -- TRAP: FailedAssertion("!(!((((const Node*)(node))->type) == T_SubLink))", File: "clauses.c", Line: 440)
> MERGE INTO public.brin_test as target_0
> USING pg_catalog.pg_database as ref_0
>   left join pg_catalog.pg_user_mapping as sample_0 tablesample system (2.3)
>   on (pg_catalog.mul_d_interval(
>         cast(pg_catalog.pi() as float8),
>         cast(case when sample_0.umoptions is not NULL then (select write_lag from pg_catalog.pg_stat_replication limit 1 offset 2)
>              else (select write_lag from pg_catalog.pg_stat_replication limit 1 offset 2)
>              end
>            as "interval")) = (select intervalcol from public.brintest limit 1 offset 2)
>         )
> ON target_0.a = ref_0.encoding
> WHEN NOT MATCHED AND cast(null as "timestamp") < cast(null as date)
>    THEN INSERT VALUES ( 62, 6)
> WHEN NOT MATCHED
>   AND false
>    THEN DO NOTHING;
>
> -- TRAP: FailedAssertion("!(!((((const Node*)(node))->type) == T_SubLink))", File: "prepunion.c", Line: 2246)
> MERGE INTO public.onek2 as target_0
> USING public.prt1 as ref_0
>   inner join public.tenk1 as ref_1
>   on ((select t from public.btree_tall_tbl limit 1 offset 63)
>          is not NULL)
> ON target_0.stringu1 = ref_1.stringu1
> WHEN NOT MATCHED THEN DO NOTHING;
>
> -- TRAP: FailedAssertion("!(!((((const Node*)(node))->type) == T_Query))", File: "var.c", Line: 248)
> MERGE INTO public.clstr_tst_inh as target_0
> USING pg_catalog.pg_statio_sys_tables as ref_0
>   left join public.rule_and_refint_t3 as ref_1
>   on (((ref_0.heap_blks_hit is not NULL)
>         or (((select f1 from public.path_tbl limit 1 offset 5)
>                >= (select thepath from public.shighway limit 1 offset 33)
>               )
>           or (cast(null as tsvector) <> cast(null as tsvector))))
>       and (ref_0.toast_blks_read is not NULL))
> ON target_0.d = ref_1.data
> WHEN NOT MATCHED
>   AND cast(null as int2) = pg_catalog.lastval()
>    THEN DO NOTHING;
>
> -- ERROR:  unrecognized node type: 114
> MERGE INTO public.found_test_tbl as target_0
> USING public.itest7a as ref_0
> ON target_0.a = ref_0.a
> WHEN NOT MATCHED
>    THEN INSERT VALUES ((select a from public.rtest_t3 limit 1 offset 6));



-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  129. Re: [HACKERS] MERGE SQL Statement for PG11

    Simon Riggs <simon@2ndquadrant.com> — 2018-02-03T22:22:34Z

    On 1 February 2018 at 19:39, Peter Geoghegan <pg@bowt.ie> wrote:

> Finally, I noticed a problem with your new EXPLAIN ANALYZE instrumentation:
>
> Is it 4 rows inserted, or 0 inserted?
>
> postgres=# merge into testoids a using (select i "key", 'foo' "data"
> from generate_series(0,3) i) b on a.key = b.key when matched and 1=0
> then update set data = b.data when not matched then insert (key, data)
> values (b.key, 'foo');
> MERGE 0

Got it. I'm reporting the number of rows processed instead of the
number of rows inserted. My test happened to have those values set
equal.

Minor bug, thanks for spotting.

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  130. Re: [HACKERS] MERGE SQL Statement for PG11

    Peter Geoghegan <pg@bowt.ie> — 2018-02-03T23:17:28Z

    On Sat, Feb 3, 2018 at 2:15 PM, Simon Riggs <simon@2ndquadrant.com> wrote:
>> I started looking at SQL Server's MERGE to verify that it also does
>> not impose any restrictions on subselects in a MERGE UPDATE's
>> targetlist, just like Oracle. Unsurprisingly, it does not. More
>> surprisingly, I noticed that it also doesn't seem to impose
>> restrictions on what can appear in WHEN ... AND quals.
>
> You earlier agreed that subselects were not part of the Standard.

You know that I didn't say that, Simon.

>> What this boils down to is that I don't think that this part of your
>> design is committable (from your recent v14):
>
> So your opinion is that because v14 patch doesn't include a feature
> extension that is in Oracle and SQLServer that we cannot commit this
> patch.
>
> There are quite a few minor additional things in that category and the
> syntax of those two differ, so its clearly impossible to match both
> exactly.
>
> That seems like poor reasoning on why we should block the patch.

It certainly is. Good thing I never said anything of the sort.

There are 3 specific issues on query structure, that together paint a
picture about what you're not doing in the optimizer:

1. Whether or not subselects in the UPDATE targetlist are supported.

2. Whether or not subselects in the WHEN ... AND quals support subselects.

3. Whether or not subselects are possible within the main ON () join.

I gave a lukewarm endorsement of not supporting #3, was unsure with
#2, and was very clear on #1 as soon as I saw the restriction: UPDATE
targetlist in a MERGE are *not* special, and so must support
subselects, just like ON CONFLICT DO UPDATE, for example.

> If you would like to say how you think the design should look, it
> might be possible to change it for this release. Changing it in the
> future would not be blocked by commiting without that.

I can tell you right now that you need to support subselects in the
targetlist. They are *not* an extension to the standard, AFAICT. And
even if they were, every other system supports them, and there is
absolutely no logical reason to not support them other than the fact
that doing so requires significant changes to the data structures in
the parser, planner, and executor. Reworking that will probably turn
out to be necessary for other reasons that I haven't thought of.

I think that restrictions like this are largely an accident of how
your patch evolved. It would be a lot easier to work with you if you
acknowledged that.

>> I am willing to continue to engage with you on the concurrency issues
>> for the time being, since that is the most pressing issue for the
>> patch. We can get to this stuff later.
>
> There are no concurrency issues. The patch gives the correct answer in
> all cases, or an error to avoid problems. We've agreed that it is
> desirable we remove some of those if possible, though they are there
> as a result of our earlier discussions.

You seem to presume to be in charge of the parameters of this
discussion. I don't see it that way. I think that READ COMMITTED
conflict handling semantics are by far the biggest issue for the
patch, and that we should prioritize reaching agreement there. This
needs to be worked out through community consensus, since it concerns
fundamental semantics much more than implementation details. (In
contrast, the optimizer issues I mentioned are fairly heavy on
relatively uncontentious implementation questions.)

The problem with how you've represented MERGE in the parser,
optimizer, and executor is not that it's "half-baked crap", as you
suggested others might think at the FOSDEM developer meeting [1]. I
wouldn't say that at all. What I'd say is that it's *unfinished*. It's
definitely sufficient to prototype different approaches to
concurrency, as well as to determine how triggers should work, and
many other such things. That's a good start.

I am willing to mostly put aside the other issues for the time being,
to get the difficult questions on concurrency out of the way first.
But if you don't make some broad concessions on the secondary issues
pretty quickly, then I will have to conclude that our positions are
irreconcilable. I will have nothing further to contribute to the
discussion.

[1] https://wiki.postgresql.org/wiki/FOSDEM/PGDay_2018_Developer_Meeting#Minutes
-- 
Peter Geoghegan

  131. Re: [HACKERS] MERGE SQL Statement for PG11

    Amit Kapila <amit.kapila16@gmail.com> — 2018-02-04T06:32:09Z

    On Wed, Jan 31, 2018 at 11:37 PM, Peter Geoghegan <pg@bowt.ie> wrote:
> On Wed, Jan 31, 2018 at 7:17 AM, Robert Haas <robertmhaas@gmail.com> wrote:
>> I don't fully grok merge but suppose you have:
>>
>> WHEN MATCHED AND a = 0 THEN UPDATE ...
>> WHEN MATCHED AND a = 1 THEN UPDATE ...
>> WHEN NOT MATCHED THEN INSERT ...
>>
>> Suppose you match a tuple with a = 0 but, upon trying to update it,
>> find that it's been updated to a = 1.  It seems like there are a few
>> possible behaviors:
>>
>> 1. Throw an error!  I guess this is what the patch does now.
>
> Right.
>
>> 2. Do absolutely nothing.  I think this is what would happen with an
>> ordinary UPDATE; the tuple fails the EPQ recheck and so is not
>> updated, but that doesn't trigger anything else.
>
> I think #2 is fine if you're talking about join quals. Which, of
> course, you're not. These WHEN quals really do feel like
> tuple-at-a-time procedural code, more than set-orientated quals (if
> that wasn't true, we'd have to allow cardinality violations, which we
> at least try to avoid). Simon said something like "the SQL standard
> requires that WHEN quals be evaluated first" at one point, which makes
> sense to me.
>

It is not clear to me what is exactly your concern if we try to follow
#2?  To me, #2 seems like a natural choice.


-- 
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

  132. Re: [HACKERS] MERGE SQL Statement for PG11

    Simon Riggs <simon@2ndquadrant.com> — 2018-02-04T08:41:57Z

    On 4 February 2018 at 06:32, Amit Kapila <amit.kapila16@gmail.com> wrote:
> On Wed, Jan 31, 2018 at 11:37 PM, Peter Geoghegan <pg@bowt.ie> wrote:
>> On Wed, Jan 31, 2018 at 7:17 AM, Robert Haas <robertmhaas@gmail.com> wrote:
>>> I don't fully grok merge but suppose you have:
>>>
>>> WHEN MATCHED AND a = 0 THEN UPDATE ...
>>> WHEN MATCHED AND a = 1 THEN UPDATE ...
>>> WHEN NOT MATCHED THEN INSERT ...
>>>
>>> Suppose you match a tuple with a = 0 but, upon trying to update it,
>>> find that it's been updated to a = 1.  It seems like there are a few
>>> possible behaviors:
>>>
>>> 1. Throw an error!  I guess this is what the patch does now.
>>
>> Right.
>>
>>> 2. Do absolutely nothing.  I think this is what would happen with an
>>> ordinary UPDATE; the tuple fails the EPQ recheck and so is not
>>> updated, but that doesn't trigger anything else.
>>
>> I think #2 is fine if you're talking about join quals. Which, of
>> course, you're not. These WHEN quals really do feel like
>> tuple-at-a-time procedural code, more than set-orientated quals (if
>> that wasn't true, we'd have to allow cardinality violations, which we
>> at least try to avoid). Simon said something like "the SQL standard
>> requires that WHEN quals be evaluated first" at one point, which makes
>> sense to me.
>>
>
> It is not clear to me what is exactly your concern if we try to follow
> #2?  To me, #2 seems like a natural choice.

At first, but it gives an anomaly so is not a good choice. The patch
does behavior #5, it rechecks the conditions with the latest row.

Otherwise
WHEN MATCHED AND a=0 THEN UPDATE SET b=0
WHEN MATCHED AND a=1 THEN UPDATE SET b=1
would result in (a=1, b=0) in case of concurrent updates, which the
user clearly doesn't want.

The evaluation of the WHEN qual must occur prior to the update, which
will still be true in #5.

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  133. Re: [HACKERS] MERGE SQL Statement for PG11

    Simon Riggs <simon@2ndquadrant.com> — 2018-02-04T08:42:01Z

    On 3 February 2018 at 23:17, Peter Geoghegan <pg@bowt.ie> wrote:
> On Sat, Feb 3, 2018 at 2:15 PM, Simon Riggs <simon@2ndquadrant.com> wrote:
>>> I started looking at SQL Server's MERGE to verify that it also does
>>> not impose any restrictions on subselects in a MERGE UPDATE's
>>> targetlist, just like Oracle. Unsurprisingly, it does not. More
>>> surprisingly, I noticed that it also doesn't seem to impose
>>> restrictions on what can appear in WHEN ... AND quals.
>>
>> You earlier agreed that subselects were not part of the Standard.
>
> You know that I didn't say that, Simon.

I'm happy to quote your words.

"I've acknowledged that the standard has something to
say on this that supports your position, which has real weight."

https://www.postgresql.org/message-id/CAH2-WzkAjSN1H-ym-sSDh%2B6EJWmEhyHdDStzXDB%2BFxt1hcKEgg%40mail.gmail.com

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  134. Re: [HACKERS] MERGE SQL Statement for PG11

    Simon Riggs <simon@2ndquadrant.com> — 2018-02-04T10:15:50Z

    On 3 February 2018 at 23:17, Peter Geoghegan <pg@bowt.ie> wrote:

> There are 3 specific issues on query structure, that together paint a
> picture about what you're not doing in the optimizer:
>
> 1. Whether or not subselects in the UPDATE targetlist are supported.
>
> 2. Whether or not subselects in the WHEN ... AND quals support subselects.
>
> 3. Whether or not subselects are possible within the main ON () join.
>
> I gave a lukewarm endorsement of not supporting #3, was unsure with
> #2, and was very clear on #1 as soon as I saw the restriction: UPDATE
> targetlist in a MERGE are *not* special, and so must support
> subselects, just like ON CONFLICT DO UPDATE, for example.

All three of the above give errors in the current patch, as we already
discussed for (1) and (2).

I've added these to the tests so we can track support for them explicitly.

The current patch runs one query then executes the quals
post-execution as we do for check constraints and RLS. Changes would
be required to support subselects.

Changes to support sub-selects don't invalidate what is there now in
the current patch with regard to query representation or optimization.
So support of those extra features can be added later if we choose.
Frankly, whatever we do now, I'm sure we will discover cases that need
further optimization, just as we have done with RLS and Partitioning,
so the query representation was likely to change over time anyway.

Whatever we decide for concurrent behavior will affect how we support
them. We can't move forwards on them until we have that nailed down.

I could give a longer technical commentary but I will be unavailable
now for some time, so unable to give further discussion.

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  135. Re: [HACKERS] MERGE SQL Statement for PG11

    Andreas Seltenreich <seltenreich@gmx.de> — 2018-02-04T12:15:44Z

    Simon Riggs writes:

> It will likely take some time to work through these and the current
> work items but will fix.
>
> Do you have the DDL so we can recreate these easily?

Attached are testcases that trigger the assertions when run against an
empty database instead of the one left behind by make installcheck.  The
"unrecognized node type" one appears to be a known issue according to
the wiki, so I didn't bother doing the work for this one as well.

regards,
Andreas

  136. Re: [HACKERS] MERGE SQL Statement for PG11

    Peter Geoghegan <pg@bowt.ie> — 2018-02-04T21:47:28Z

    On Sun, Feb 4, 2018 at 12:42 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
> I'm happy to quote your words.
>
> "I've acknowledged that the standard has something to
> say on this that supports your position, which has real weight."
>
> https://www.postgresql.org/message-id/CAH2-WzkAjSN1H-ym-sSDh%2B6EJWmEhyHdDStzXDB%2BFxt1hcKEgg%40mail.gmail.com

Immediately afterwards, in that same e-mail, I go on to say: "I'm not
asking about WHEN AND here (that was my last question) [Simon quoted
my last response to said question]. I'm asking about a subselect that
appears in the targetlist."

Even if you are right to take "I've acknowledged that the standard has
something to say on this that supports your position" as a blanket
endorsement of disallowing subselects in all 3 places, I still don't
see why this is worth even talking about. That would mean that I said
something on January 29th that I subsequently withdrew on February
1st.

-- 
Peter Geoghegan

  137. Re: [HACKERS] MERGE SQL Statement for PG11

    Robert Haas <robertmhaas@gmail.com> — 2018-02-06T03:49:29Z

    On Sun, Feb 4, 2018 at 3:41 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
>> It is not clear to me what is exactly your concern if we try to follow
>> #2?  To me, #2 seems like a natural choice.
>
> At first, but it gives an anomaly so is not a good choice. The patch
> does behavior #5, it rechecks the conditions with the latest row.
>
> Otherwise
> WHEN MATCHED AND a=0 THEN UPDATE SET b=0
> WHEN MATCHED AND a=1 THEN UPDATE SET b=1
> would result in (a=1, b=0) in case of concurrent updates, which the
> user clearly doesn't want.

I am unable to understand this.  What are you presuming the tuple was
originally?

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  138. Re: [HACKERS] MERGE SQL Statement for PG11

    Robert Haas <robertmhaas@gmail.com> — 2018-02-06T03:56:39Z

    On Sun, Feb 4, 2018 at 5:15 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
> Changes to support sub-selects don't invalidate what is there now in
> the current patch with regard to query representation or optimization.
> So support of those extra features can be added later if we choose.

I don't think you get to make a unilateral decision to exclude
features that work everywhere else from the scope of this patch.  If
there is agreement that those features can be left out of scope, then
that is one thing, but so far all the commentary about the things that
you've chosen to exclude has been negative.  Nor have you really given
any reason why they should be exempt.  You've pointed out that
parallel query doesn't handle everything (which is certainly true, but
does not mean that any feature from now and the end of time is allowed
to exclude from scope whatever seems inconvenient regardless of
contrary community consensus) and you've pointed out here and
elsewhere that somebody could go add the features you omitted later
(which is also true, but misses the general point that we want
committed patches to be reasonably complete already, not have big gaps
that someone will have to fix later).

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  139. Re: [HACKERS] MERGE SQL Statement for PG11

    Peter Geoghegan <pg@bowt.ie> — 2018-02-06T04:20:39Z

    On Mon, Feb 5, 2018 at 7:56 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> I don't think you get to make a unilateral decision to exclude
> features that work everywhere else from the scope of this patch.  If
> there is agreement that those features can be left out of scope, then
> that is one thing, but so far all the commentary about the things that
> you've chosen to exclude has been negative.  Nor have you really given
> any reason why they should be exempt.  You've pointed out that
> parallel query doesn't handle everything (which is certainly true, but
> does not mean that any feature from now and the end of time is allowed
> to exclude from scope whatever seems inconvenient regardless of
> contrary community consensus) and you've pointed out here and
> elsewhere that somebody could go add the features you omitted later
> (which is also true, but misses the general point that we want
> committed patches to be reasonably complete already, not have big gaps
> that someone will have to fix later).

For me, the concern is not really the omission of support for certain
features as such. The concern is that those omissions hint that there
is a problem with the design itself, particularly in the optimizer.
Allowing subselects in the UPDATE part of a MERGE do not seem like
they could be written as a neat adjunct to what Simon already came up
with. If that was possible, Simon probably already would have done it.

-- 
Peter Geoghegan

  140. Re: [HACKERS] MERGE SQL Statement for PG11

    Pavan Deolasee <pavan.deolasee@gmail.com> — 2018-02-06T06:06:36Z

    On Tue, Feb 6, 2018 at 9:50 AM, Peter Geoghegan <pg@bowt.ie> wrote:

> On Mon, Feb 5, 2018 at 7:56 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> > I don't think you get to make a unilateral decision to exclude
> > features that work everywhere else from the scope of this patch.  If
> > there is agreement that those features can be left out of scope, then
> > that is one thing, but so far all the commentary about the things that
> > you've chosen to exclude has been negative.  Nor have you really given
> > any reason why they should be exempt.  You've pointed out that
> > parallel query doesn't handle everything (which is certainly true, but
> > does not mean that any feature from now and the end of time is allowed
> > to exclude from scope whatever seems inconvenient regardless of
> > contrary community consensus) and you've pointed out here and
> > elsewhere that somebody could go add the features you omitted later
> > (which is also true, but misses the general point that we want
> > committed patches to be reasonably complete already, not have big gaps
> > that someone will have to fix later).
>
> For me, the concern is not really the omission of support for certain
> features as such. The concern is that those omissions hint that there
> is a problem with the design itself, particularly in the optimizer.
> Allowing subselects in the UPDATE part of a MERGE do not seem like
> they could be written as a neat adjunct to what Simon already came up
> with. If that was possible, Simon probably already would have done it.
>
>
As someone who's helping Simon with that part of the code, I must say that
omission of sub-selects in the UPDATE targetlist and WHEN quals is not
because of some known design problems.  So while it may be true that we've
a design problem, it's also quite likely that we are missing some
planner/optimiser trick and once we add those missing pieces, it will start
working. Same is the case with RLS.

Partitioned table is something I am actively working on. I must say that
the very fact that INSERT and UPDATE/DELETE take completely different paths
in partitioned/inherited table, makes MERGE quite difficult because it has
to carry out both the operations and hence require all the required
machinery. If I understand correctly, INSERT ON CONFLICT must have faced
similar problems and hence DO UPDATE does not work with partitioned table.
I am not sure if that choice was made when INSERT ON CONFLICT was
implemented or when partitioned table support was added. But the challenges
look similar.

I first tried to treat MERGE similar to UPDATE/DELETE case and ensure that
the INSERTs go through the root partition. That mostly works, but the RIGHT
OUTER join between the child tables and the source relation ends up
emitting duplicate rows, if the partitioned table is the resultRelation and
when it gets expanded in inheritance_planner(). That's a blocker. So what I
am trying now is to push the join between the Append relation and the
source relation below the ModifyTable node, so that we get the final join
result. We can then look up the tableoid in the row returned from the join,
find the corresponding result relation and then carry out MERGE actions.
Note that unlike regular ExecModifyTable(), here we must execute just one
subplan as that will return all the required tuples.

Does anyone see a potential blocker with this approach, except that it may
not be the most elegant way? I think EvalPlanQual might need some treatment
because when the plan is re-executed, it will expect to the find the
updated tuple in the slot of the underlying query's RTE and not in the
resultRelation's RTE, which does not participate in the join at all.
Anything else I could be missing out completely?

Thanks,
Pavan


-- 
 Pavan Deolasee                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

  141. Re: [HACKERS] MERGE SQL Statement for PG11

    Stephen Frost <sfrost@snowman.net> — 2018-02-06T10:07:45Z

    Greetings,

* Peter Geoghegan (pg@bowt.ie) wrote:
> On Mon, Feb 5, 2018 at 7:56 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> > I don't think you get to make a unilateral decision to exclude
> > features that work everywhere else from the scope of this patch.  If
> > there is agreement that those features can be left out of scope, then
> > that is one thing, but so far all the commentary about the things that
> > you've chosen to exclude has been negative.  Nor have you really given
> > any reason why they should be exempt.  You've pointed out that
> > parallel query doesn't handle everything (which is certainly true, but
> > does not mean that any feature from now and the end of time is allowed
> > to exclude from scope whatever seems inconvenient regardless of
> > contrary community consensus) and you've pointed out here and
> > elsewhere that somebody could go add the features you omitted later
> > (which is also true, but misses the general point that we want
> > committed patches to be reasonably complete already, not have big gaps
> > that someone will have to fix later).
> 
> For me, the concern is not really the omission of support for certain
> features as such. The concern is that those omissions hint that there
> is a problem with the design itself, particularly in the optimizer.
> Allowing subselects in the UPDATE part of a MERGE do not seem like
> they could be written as a neat adjunct to what Simon already came up
> with. If that was possible, Simon probably already would have done it.

I tend to agree with Robert here that we should be trying to include
relatively complete features which work with the rest of the system
whenever possible.  To the extent that partitioning wasn't entirely
complete, I think the subsequent work on it (partituclarly in this
release cycle) clearly demonstrates that it's a huge project which
needed multiple releases, but that's an exception to the general rule.

That was also what seemed to be the consensus coming out of the FOSDEM
Developer meeting (notes here:
https://wiki.postgresql.org/wiki/FOSDEM/PGDay_2018_Developer_Meeting).
There was also discussion and apparent consensus that performance issues
or sub-par plans are reasonable for an initial feature (as happened with
leakproof views and subsequently RLS until the rework which improved it
greatly in later releases).

Coming out of that, my understanding is that Simon is planning to have a
patch which implements RLS and partitioning (though the query plans for
partitioning may be sub-par and not ideal) as part of MERGE and I've
agreed to review at least the RLS bits (though my intention is to at
least go through the rest of the patch as well, though likely in less
detail).  Of course, I encourage others to review it as well.

Thanks!

Stephen

  142. Re: [HACKERS] MERGE SQL Statement for PG11

    Amit Kapila <amit.kapila16@gmail.com> — 2018-02-06T12:51:02Z

    On Tue, Feb 6, 2018 at 9:19 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> On Sun, Feb 4, 2018 at 3:41 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
>>> It is not clear to me what is exactly your concern if we try to follow
>>> #2?  To me, #2 seems like a natural choice.
>>
>> At first, but it gives an anomaly so is not a good choice. The patch
>> does behavior #5, it rechecks the conditions with the latest row.
>>
>> Otherwise
>> WHEN MATCHED AND a=0 THEN UPDATE SET b=0
>> WHEN MATCHED AND a=1 THEN UPDATE SET b=1
>> would result in (a=1, b=0) in case of concurrent updates, which the
>> user clearly doesn't want.
>
> I am unable to understand this.
>

Neither do I.  There is nothing in above statement which changes 'a'.

>  What are you presuming the tuple was
> originally?
>

I have tried to think of one example which can result in what Simon is
saying.  Consider original tuple has a = 0 and b = 1

Session -1
WHEN MATCHED AND a=0 THEN UPDATE SET b=0
WHEN MATCHED AND a=1 THEN UPDATE SET b=1

Session-2
WHEN MATCHED AND b=0 THEN UPDATE SET a=0
WHEN MATCHED AND b=1 THEN UPDATE SET a=1

Now assume both the session got the tuple to Update, Session-1 locks
to Update b = 0 and Session-2 will wait for Session-1 to complete.
After Session-1 commits, Session-2 will wake up and performs
EvalPlanQual because it will find the tuple as Updated.  Now, I think
EvalPlanQual mechanism will succeed if we don't match WHEN clauses as
part of EvalPlanQual mechanism and it will update a = 1.  So, now we
will have a=1, b=0.  I think if this is going to happen with
approach-2 (#2), then one can argue that Session-2's update shouldn't
have succeeded.

-- 
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

  143. Re: [HACKERS] MERGE SQL Statement for PG11

    Tomas Vondra <tomas.vondra@2ndquadrant.com> — 2018-02-06T14:40:53Z

    On 02/06/2018 05:20 AM, Peter Geoghegan wrote:
> On Mon, Feb 5, 2018 at 7:56 PM, Robert Haas <robertmhaas@gmail.com> wrote:
>> I don't think you get to make a unilateral decision to exclude 
>> features that work everywhere else from the scope of this patch.
>> If there is agreement that those features can be left out of scope,
>> then that is one thing, but so far all the commentary about the
>> things that you've chosen to exclude has been negative. Nor have
>> you really given any reason why they should be exempt. You've
>> pointed out that parallel query doesn't handle everything (which is
>> certainly true, but does not mean that any feature from now and the
>> end of time is allowed to exclude from scope whatever seems
>> inconvenient regardless of contrary community consensus) and you've
>> pointed out here and elsewhere that somebody could go add the
>> features you omitted later (which is also true, but misses the
>> general point that we want committed patches to be reasonably
>> complete already, not have big gaps that someone will have to fix
>> later).
> 
> For me, the concern is not really the omission of support for
> certain features as such. The concern is that those omissions hint
> that there is a problem with the design itself, particularly in the
> optimizer. Allowing subselects in the UPDATE part of a MERGE do not
> seem like they could be written as a neat adjunct to what Simon
> already came up with. If that was possible, Simon probably already
> would have done it.
> 

I agree with both of these statements wholeheartedly.

As I mentioned at the developer meeting last week, whenever someone asks
me for reasons why I find PostgreSQL better than MySQL, I usually point
them to "MySQL Restrictions and Limitations" document. Which is pretty
much a matrix of features that do not work together in some way. It
would be somewhat unfortunate to get on that route too ...

So I think the general principle should be to make the features as
complete as possible. For the more complex features that may not be
quite achievable, though, forcing us to add the pieces over multiple
releases. Which is what happened with partitioning, for example.

I think we can do that for MERGE too, assuming we actually understand

(1) why each of the pieces is missing

(2) what would it take to make it work

I'm not sure we have those answers for MERGE, though.

I haven't done anything on the MERGE patch so far, and I've been unable
to convince myself what the reasoning is for the limitations. Perhaps
there are explanations somewhere far back in the thread, but considering
how much both the implementation and our understanding of it changed
over time, I'm not sure how relevant those past arguments are.

Take for example the "sub-selects are not supported" limitation. It's
not clear to me why this does not work. I see claims that SQL standard
says something about it (but not what exactly), that it supposedly
depends on how we handle concurrency, and perhaps other reasons.
Furthermore, there are multiple places where the sub-select might be,
and I'm not sure which of those arguments applies to which case.

Without answering (2) I think it's perfectly understandable Peter is
concerned we may run into design issues later, when we try to address
some of the limitations.


I plan to go through the patch and this thread over the couple of days,
and summarize what the current status is (or my understanding of it).
That is (a) what are the missing pieces, (b) why are they missing, (c)
how we plan to address them in the future and (d) opinions on these
issues expressed by others on this thread.


kind regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  144. Re: [HACKERS] MERGE SQL Statement for PG11

    Robert Haas <robertmhaas@gmail.com> — 2018-02-06T15:12:04Z

    On Tue, Feb 6, 2018 at 9:40 AM, Tomas Vondra
<tomas.vondra@2ndquadrant.com> wrote:
> I think we can do that for MERGE too, assuming we actually understand
>
> (1) why each of the pieces is missing
>
> (2) what would it take to make it work

Right, I completely agree with that.  For example, if we find that a
certain thing can't be supported without implementing global indexes
on partitions, then I have no problem saying "OK, that's not going to
be supported", because global indexes are a huge project unto
themselves.  That's the same reason ON CONFLICT .. UPDATE isn't
supported on partitioned tables, and it's just as reasonable for this
patch as it is for that one.  What we don't want is things that this
patch doesn't support because it would take a couple of days of work
and the people who wrote the patch were busy and didn't have a couple
of extra days.  We routinely expect patch authors to plug gaps caused
by oversight or lack of round tuits, and this patch should be held to
the same standard.

In short, we don't have a hard and fast rule that every feature must
work with every other feature, but when it doesn't, the burden is on
the patch author to justify why that omission is reasonable.  I know I
expended a lot of ink explaining why parallel query couldn't
reasonably be made to support writes.

> I plan to go through the patch and this thread over the couple of days,
> and summarize what the current status is (or my understanding of it).
> That is (a) what are the missing pieces, (b) why are they missing, (c)
> how we plan to address them in the future and (d) opinions on these
> issues expressed by others on this thread.

Thank you.  That sounds fantastic.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  145. Re: [HACKERS] MERGE SQL Statement for PG11

    Pavan Deolasee <pavan.deolasee@gmail.com> — 2018-02-07T09:24:45Z

    On Tue, Feb 6, 2018 at 8:10 PM, Tomas Vondra <tomas.vondra@2ndquadrant.com>
wrote:

>
>
> Take for example the "sub-selects are not supported" limitation. It's
> not clear to me why this does not work. I see claims that SQL standard
> says something about it (but not what exactly), that it supposedly
> depends on how we handle concurrency, and perhaps other reasons.
> Furthermore, there are multiple places where the sub-select might be,
> and I'm not sure which of those arguments applies to which case.
>
> Without answering (2) I think it's perfectly understandable Peter is
> concerned we may run into design issues later, when we try to address
> some of the limitations.
>
>
> I plan to go through the patch and this thread over the couple of days,
> and summarize what the current status is (or my understanding of it).
> That is (a) what are the missing pieces, (b) why are they missing, (c)
> how we plan to address them in the future and (d) opinions on these
> issues expressed by others on this thread.
>
>
Thanks Tomas. That will certainly help a lot.

Here is v15 of the patch. It now fully supports partitioned tables. As I
explained upthread, supporting partitioned table turned out much trickier
than what I initially thought because of complete different code paths that
INSERT and UPDATE/DELETE take in case of inheritance. Since MERGE need both
the facilities, I'd to pretty much merge both the machineries. But the end
result seems okay. I am sure we can improve this further, but whatever I
have tested so far (which may not be necessarily thorough) seems to work
fine.

Since the way RIGHT OUTER join is now pushed below the ModifyTable node,
I'd to make appropriate changes to EvalPlanQual calling locations so that
we pass in the RT index of the table used in the join, and not of the
resultRelInfo, when we are handling MERGE.

Initially I was a bit surprised that EvalPlanQual silently ignores the case
when partition key is updated and a row is moved from one partition to
another. But then I realised that this is the behaviour of the partitioned
tables and not MERGE itself.

The revised version also supports subqueries in SET targetlist as well as
WHEN AND conditions. As expected, this needed a minor tweak in
query/expression mutators. So once I worked on that for partitioned tables,
subqueries started working with very minimal adjustments elsewhere. Other
things such as whole-var references are still broken. A few new tests are
also added.

Next I am going to look at RLS. While I've no prior experience with RLS, I
am expecting it to be less cumbersome in comparison to partitioning. I am
out of action till Monday and may not be able to respond in time. But any
reviews and comments are appreciated as always.

Thanks,
Pavan

-- 
 Pavan Deolasee                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

  146. Re: [HACKERS] MERGE SQL Statement for PG11

    Peter Geoghegan <pg@bowt.ie> — 2018-02-07T17:22:02Z

    On Wed, Feb 7, 2018 at 1:24 AM, Pavan Deolasee <pavan.deolasee@gmail.com> wrote:
> Here is v15 of the patch.

Cool.

> Initially I was a bit surprised that EvalPlanQual silently ignores the case
> when partition key is updated and a row is moved from one partition to
> another. But then I realised that this is the behaviour of the partitioned
> tables and not MERGE itself.

Apparently there is a pending patch to do better there - the commit
message of 2f178441 refers to this.

> The revised version also supports subqueries in SET targetlist as well as
> WHEN AND conditions. As expected, this needed a minor tweak in
> query/expression mutators. So once I worked on that for partitioned tables,
> subqueries started working with very minimal adjustments elsewhere. Other
> things such as whole-var references are still broken. A few new tests are
> also added.

Great!

Wholerow references are expected to be a bit trickier. See commit
ad227837 for some hints on how you could fix this.

> Next I am going to look at RLS. While I've no prior experience with RLS, I
> am expecting it to be less cumbersome in comparison to partitioning. I am
> out of action till Monday and may not be able to respond in time. But any
> reviews and comments are appreciated as always.

I don't think that RLS support will be particularly challenging. It
might take a while.

If your rapid progress here is any indication, most open items are not
likely to be particularly challenging. Once again, I suggest that a
good area for us to focus on is the semantics of READ COMMITTED
conflict handling. Maybe you'd prefer to just blast through the
simpler open items, which is fine, but do bear in mind that the EPQ
and EPQ-adjacent stuff is probably going to be the thing that makes or
breaks this patch for v11.

-- 
Peter Geoghegan

  147. Re: [HACKERS] MERGE SQL Statement for PG11

    Pavan Deolasee <pavan.deolasee@gmail.com> — 2018-02-08T03:51:42Z

    On Wed, Feb 7, 2018 at 10:52 PM, Peter Geoghegan <pg@bowt.ie> wrote:

>
>
> Apparently there is a pending patch to do better there - the commit
> message of 2f178441 refers to this.
>
>
Thanks. Will look at it.


> > The revised version also supports subqueries in SET targetlist as well as
> > WHEN AND conditions. As expected, this needed a minor tweak in
> > query/expression mutators. So once I worked on that for partitioned
> tables,
> > subqueries started working with very minimal adjustments elsewhere. Other
> > things such as whole-var references are still broken. A few new tests are
> > also added.
>
> Great!
>
> Wholerow references are expected to be a bit trickier. See commit
> ad227837 for some hints on how you could fix this.
>

Thanks again.


>
> > Next I am going to look at RLS. While I've no prior experience with RLS,
> I
> > am expecting it to be less cumbersome in comparison to partitioning. I am
> > out of action till Monday and may not be able to respond in time. But any
> > reviews and comments are appreciated as always.
>
> I don't think that RLS support will be particularly challenging. It
> might take a while.
>

Ok. I would start from writing a test case to check what works and what
doesn't with the current patch and work from there. My understanding of RLS
is limited right now, but from what I've seen in the code (while hacking
other stuff), my guess is it will require us evaluating a set of quals and
then deciding on the action.


>
> If your rapid progress here is any indication, most open items are not
> likely to be particularly challenging. Once again, I suggest that a
> good area for us to focus on is the semantics of READ COMMITTED
> conflict handling.


I understand getting EPQ semantics right is very important. Can you please
(once again) summarise your thoughts on what you think is the *most*
appropriate behaviour? I can then think how much efforts might be involved
in that. If the efforts are disproportionately high, we can discuss if
settling for some not-so-nice semantics, like we apparently did for
partition key updates.

I am sorry, I know you and Simon have probably done that a few times
already and I should rather study those proposals first. So it's okay if
you don't want to repeat those; I will work on them next week once I am
back from holidays.

Maybe you'd prefer to just blast through the
> simpler open items, which is fine, but do bear in mind that the EPQ
> and EPQ-adjacent stuff is probably going to be the thing that makes or
> breaks this patch for v11.
>

TBH I did not consider partitioning any less complex and it was indeed very
complex, requiring at least 3 reworks by me. And from what I understood, it
would have been a blocker too. So is subquery handling and RLS. That's why
I focused on addressing those items while you and Simon were still debating
EPQ semantics.

Thanks,
Pavan

-- 
 Pavan Deolasee                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

  148. Re: [HACKERS] MERGE SQL Statement for PG11

    Peter Geoghegan <pg@bowt.ie> — 2018-02-09T01:23:09Z

    On Wed, Feb 7, 2018 at 7:51 PM, Pavan Deolasee <pavan.deolasee@gmail.com> wrote:
> I understand getting EPQ semantics right is very important. Can you please
> (once again) summarise your thoughts on what you think is the *most*
> appropriate behaviour? I can then think how much efforts might be involved
> in that. If the efforts are disproportionately high, we can discuss if
> settling for some not-so-nice semantics, like we apparently did for
> partition key updates.

I personally believe that the existing EPQ semantics are already
not-so-nice. They're what we know, though, and we haven't actually had
any real world complaints, AFAIK.

My concern is mostly just that MERGE manages to behave in a way that
actually "provides a single SQL statement that can conditionally
INSERT, UPDATE or DELETE rows, a task that would otherwise require
multiple procedural language statements", as the docs put it. As long
as MERGE manages to do something as close to that high level
description as possible in READ COMMITTED mode (with our current
semantics for multiple statements in RC taken as the baseline), then
I'll probably be happy.

Some novel new behavior -- "EPQ with a twist"-- is clearly necessary.
I feel a bit uneasy about it because anything that anybody suggests is
likely to be at least a bit arbitrary (EPQ itself is kind of
arbitrary). We only get to make a decision on how "EPQ with a twist"
will work once, and that should be a decision that is made following
careful deliberation. Ambiguity is much more likely to kill a patch
than a specific technical defect, at least in my experience. Somebody
can usually just fix a technical defect.

> I am sorry, I know you and Simon have probably done that a few times already
> and I should rather study those proposals first. So it's okay if you don't
> want to repeat those; I will work on them next week once I am back from
> holidays.

Unfortunately, I didn't get very far with Simon on this. I didn't
really start talking about this until recently, though, so it's not
like you missed much. The first time I asked Simon about this was
January 23rd, and I first proposed something about 10 days ago.
Something very tentative.

(I did ask some questions about EPQ, and even WHEN ... AND quals much
earlier, but that was in the specific context of a debate about
MERGE's use of ON CONFLICT's speculative insertion mechanism. I
consider this to be a totally different discussion, that ended before
Simon even posted his V1 patch, and isn't worth spending your time on
now.)

> TBH I did not consider partitioning any less complex and it was indeed very
> complex, requiring at least 3 reworks by me. And from what I understood, it
> would have been a blocker too. So is subquery handling and RLS. That's why I
> focused on addressing those items while you and Simon were still debating
> EPQ semantics.

Sorry if I came across as dismissive of that effort. That was
certainly not my intention. I am pleasantly surprised that you've
managed to move a number of things forward rather quickly.

I'll rephrase: while it would probably have been a blocker in theory
(I didn't actually weigh in on that), I doubted that it would actually
end up doing so in practice (and it now looks like I was right to
doubt that, since you got it done). It was a theoretical blocker, as
opposed to an open item that could drag on indefinitely despite
everyone's best efforts. Obviously details matter, and obviously there
are a lot of details to get right outside of RC semantics, but it
seems wise to focus on the big risk that is EPQ/RC conflict handling.

The only other thing that comes close to that risk is the risk that
we'll get stuck on RLS. Though even the RLS discussion may actually
end up being blocked on this crucial question of EPQ/RC conflict
handling. Did you know that the RLS docs [1] have a specific
discussion of the implications of EPQ for users of RLS, and that it
mentions doing things like using SELECT ... FOR SHARE to work around
the problem? It has a whole example of a scenario that users actually
kind of need to know about, at least in theory. RC conflict handling
semantics could bleed into a number of other things.

I'll need to think some more about RC conflict handling (deciding what
"EPQ with a twist" actually means), since I haven't focused on MERGE
recently. Bear with me.

[1] https://www.postgresql.org/docs/current/static/ddl-rowsecurity.html
-- 
Peter Geoghegan

  149. Re: [HACKERS] MERGE SQL Statement for PG11

    Robert Haas <robertmhaas@gmail.com> — 2018-02-09T14:36:25Z

    On Thu, Feb 8, 2018 at 8:23 PM, Peter Geoghegan <pg@bowt.ie> wrote:
> Some novel new behavior -- "EPQ with a twist"-- is clearly necessary.
> I feel a bit uneasy about it because anything that anybody suggests is
> likely to be at least a bit arbitrary (EPQ itself is kind of
> arbitrary). We only get to make a decision on how "EPQ with a twist"
> will work once, and that should be a decision that is made following
> careful deliberation. Ambiguity is much more likely to kill a patch
> than a specific technical defect, at least in my experience. Somebody
> can usually just fix a technical defect.

Here's my $0.02: I think that new concurrency errors thrown by the
merge code itself deserve strict scrutiny and can survive only if they
have a compelling justification, but if the merge code happens to
choose an action that leads to a concurrency error of its own, I think
that deserves only mild scrutiny.

On that basis, of the options I listed in
http://postgr.es/m/CA+TgmoZDL-caukHkWet7sr7sqr0-e2T91+DEvhqeN5sfqsMjqw@mail.gmail.com
I like #1 least.

I also dislike #4 from that list for the reasons stated there.  For
example, if you say WHEN MATCHED AND x.some_boolean and then WHEN
MATCHED, you expect that every tuple that hits the latter clause will
have that Boolean as false or null, but #4 makes that not true.

I think the best options are #2 and #5 -- #2 because it's simple, and
#5 because it's (maybe) more intuitive, albeit at the risk of
livelock.  But I'm willing to convinced of some other option; like
you, I'm willing to be open-minded about this.  But, as you say, we
need a coherent, well-considered justification for the selected
option, not just "well, this is what we implemented so you should like
it".  The specification should define the implementation, not the
reverse.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  150. Re: [HACKERS] MERGE SQL Statement for PG11

    Tomas Vondra <tomas.vondra@2ndquadrant.com> — 2018-02-10T01:49:01Z

    Hi,

On 02/07/2018 10:24 AM, Pavan Deolasee wrote:
> 
> ...
>
> Here is v15 of the patch.
>

I've been looking at this version of the patch, mostly to educate myself
before attempting to write the "status summary".

One bit that I don't quite understand is GetXactWALBytes(). It pretty
much just returns XactLastRecEnd and is used in ExecMerge like this:

       int64 startWAL = GetXactWALBytes();
       bool	qual = ExecQual(action->whenqual, econtext);

       /*
        * SQL Standard says that WHEN AND conditions must not
        * write to the database, so check we haven't written
        * any WAL during the test. Very sensible that is, since
        * we can end up evaluating some tests multiple times if
        * we have concurrent activity and complex WHEN clauses.
        *
        * XXX If we had some clear form of functional labelling
        * we could use that, if we trusted it.
        */
       if (startWAL < GetXactWALBytes())
           ereport(ERROR,
                   (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
                    errmsg("cannot write to database ...")));

I think this actually fails to enforce the rule, because some writes may
not produce WAL (think of unlogged tables). I also suspect it may be
incorrect "in the opposite direction" because a query may not do any
changes and yet it may produce WAL (e.g. due to wal_hint_bits=true).

So we may need to think of a different way to enforce this ...

regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  151. Re: [HACKERS] MERGE SQL Statement for PG11

    Pavan Deolasee <pavan.deolasee@gmail.com> — 2018-02-12T07:09:28Z

    On Fri, Feb 9, 2018 at 6:53 AM, Peter Geoghegan <pg@bowt.ie> wrote:

> On Wed, Feb 7, 2018 at 7:51 PM, Pavan Deolasee <pavan.deolasee@gmail.com>
> wrote:
> > I understand getting EPQ semantics right is very important. Can you
> please
> > (once again) summarise your thoughts on what you think is the *most*
> > appropriate behaviour? I can then think how much efforts might be
> involved
> > in that. If the efforts are disproportionately high, we can discuss if
> > settling for some not-so-nice semantics, like we apparently did for
> > partition key updates.
>
> I personally believe that the existing EPQ semantics are already
> not-so-nice. They're what we know, though, and we haven't actually had
> any real world complaints, AFAIK.
>

I agree.


>
> My concern is mostly just that MERGE manages to behave in a way that
> actually "provides a single SQL statement that can conditionally
> INSERT, UPDATE or DELETE rows, a task that would otherwise require
> multiple procedural language statements", as the docs put it. As long
> as MERGE manages to do something as close to that high level
> description as possible in READ COMMITTED mode (with our current
> semantics for multiple statements in RC taken as the baseline), then
> I'll probably be happy.
>

IMO it will be quite hard, if not impossible, to guarantee the same
semantics to a single statement MERGE and multi statement
UPDATE/DELETE/INSERT in RC mode. For example, the multi statement model
will execute each statement with a new MVCC snapshot and hence the rows
visible to individual statement may vary. Whereas in MERGE, everything runs
with a single snapshot. There could be other such subtle differences.


>
> Some novel new behavior -- "EPQ with a twist"-- is clearly necessary.
> I feel a bit uneasy about it because anything that anybody suggests is
> likely to be at least a bit arbitrary (EPQ itself is kind of
> arbitrary). We only get to make a decision on how "EPQ with a twist"
> will work once, and that should be a decision that is made following
> careful deliberation. Ambiguity is much more likely to kill a patch
> than a specific technical defect, at least in my experience. Somebody
> can usually just fix a technical defect.
>

While I agree, I think we need to make these decisions in a time bound
fashion. If there is too much ambiguity, then it's not a bad idea to settle
for throwing appropriate errors instead of providing semantically wrong
answers, even in some remote corner case.


>
>
>
> > TBH I did not consider partitioning any less complex and it was indeed
> very
> > complex, requiring at least 3 reworks by me. And from what I understood,
> it
> > would have been a blocker too. So is subquery handling and RLS. That's
> why I
> > focused on addressing those items while you and Simon were still debating
> > EPQ semantics.
>
> Sorry if I came across as dismissive of that effort. That was
> certainly not my intention. I am pleasantly surprised that you've
> managed to move a number of things forward rather quickly.
>
> I'll rephrase: while it would probably have been a blocker in theory
> (I didn't actually weigh in on that), I doubted that it would actually
> end up doing so in practice (and it now looks like I was right to
> doubt that, since you got it done). It was a theoretical blocker, as
> opposed to an open item that could drag on indefinitely despite
> everyone's best efforts. Obviously details matter, and obviously there
> are a lot of details to get right outside of RC semantics, but it
> seems wise to focus on the big risk that is EPQ/RC conflict handling.
>

Ok. I am now back from holidays and I will too start thinking about this.
I've also requested a colleague to help us with comparing it against
Oracle's behaviour. N That's not a gold standard for us, but knowing how
other major databases handle RC conflicts, is not a bad idea.

I see the following important areas and as long as we have a consistent and
coherent handling of these cases, we should not have difficulty agreeing on
a outcome.

1. Concurrent UPDATE does not affect MATCHED  case. The WHEN conditions may
or may not be affected.
2. Concurrently UPDATEd tuple fails the join qual and the current source
tuple no longer matches with the updated target tuple that the EPQ is set
for. It matches no other target tuple either. So a MATCHED case is turned
into a NOT MATCHED case.
3. Concurrently UPDATEd tuple fails the join qual and the current source
tuple no longer matches with the updated target tuple that the EPQ is set
for. But it matches some other target tuple. So it's still a MATCHED case,
but with different target tuple(s).
4. Concurrent UPDATE/INSERT creates a matching target tuple for a source
tuple, thus turning a NOT MATCHED case to a MATCHED case.
5. Concurrent DELETE turns a MATCHED case into NOT MATCHED case

Any other case that I am missing? Assuming all cases are covered, what
should we do in each of these cases, so that there is no or very little
ambiguity and the outcome seems consistent (at the very least as far as
MERGE goes and hopefully with regular UPDATE/DELETE handling)

I think #1 is pretty straight forward. We should start from the top,
re-evaluate the WHEN conditions again and execute the first matching action.

For #2, it seems natural that we skip the MATCHED actions and execute the
NOT MATCHED action for the current source tuple. But the trouble is how to
differentiate between #2 and #3. I'm not sure if we can really distinguish
between these cases i.e. given the current source tuple, does another
matching target tuple exists? If another matching target tuple exists, we
must not invoke the NOT MATCHED action. Otherwise we might end up executing
NOT MATCHED as well as MATCHED action for the same source tuple, which
seems weird.

#4 looks similar to INSERT ON CONFLICT and one may argue that we should
detect concurrent inserts/updates and execute the MATCHED action. But I
don't know if that can be done in a reasonable way. It will probably
require us to have a primary key on the table to detect those conflicts. I
think we should just let the operation fail, like a regular INSERT.

It seems natural that #5 should skip the MATCHED action and instead execute
the first satisfying NOT MATCHED action. But it's outcome may depend on how
we handle #2 and #3 so that they are all consistent. If we allow #2 and #3
to error out, whenever there is ambiguity, we should do the same for #5.

Thanks,
Pavan

-- 
 Pavan Deolasee                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

  152. Re: [HACKERS] MERGE SQL Statement for PG11

    Pavan Deolasee <pavan.deolasee@gmail.com> — 2018-02-12T08:18:40Z

    On Sat, Feb 10, 2018 at 7:19 AM, Tomas Vondra <tomas.vondra@2ndquadrant.com>
wrote:

> Hi,
>
> On 02/07/2018 10:24 AM, Pavan Deolasee wrote:
> >
>
>        if (startWAL < GetXactWALBytes())
>            ereport(ERROR,
>                    (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
>                     errmsg("cannot write to database ...")));
>
> I think this actually fails to enforce the rule, because some writes may
> not produce WAL (think of unlogged tables). I also suspect it may be
> incorrect "in the opposite direction" because a query may not do any
> changes and yet it may produce WAL (e.g. due to wal_hint_bits=true).
>
> So we may need to think of a different way to enforce this ...


Yes, this does look problematic.

Thanks,
Pavan

-- 
 Pavan Deolasee                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

  153. Re: [HACKERS] MERGE SQL Statement for PG11

    Pavan Deolasee <pavan.deolasee@gmail.com> — 2018-02-12T08:26:46Z

    On Fri, Feb 9, 2018 at 8:06 PM, Robert Haas <robertmhaas@gmail.com> wrote:

>
>
> On that basis, of the options I listed in
> http://postgr.es/m/CA+TgmoZDL-caukHkWet7sr7sqr0-e2T91+
> DEvhqeN5sfqsMjqw@mail.gmail.com
> I like #1 least.
>
> I also dislike #4 from that list for the reasons stated there.  For
> example, if you say WHEN MATCHED AND x.some_boolean and then WHEN
> MATCHED, you expect that every tuple that hits the latter clause will
> have that Boolean as false or null, but #4 makes that not true.
>
> I think the best options are #2 and #5 -- #2 because it's simple, and
> #5 because it's (maybe) more intuitive, albeit at the risk of
> livelock.


As you said, #5 seems the best and that's what the patch does. But ISTM
that the options you listed are not really the concerning points. As the
patch stands today, we evaluate WHEN AND conditions separately, outside the
EPQ. The problem arises when the join qual returns a different result with
the updated tuple. I listed down those cases in my earlier email in the
day. To me (and I assume to Peter and Simon too), those are the more
interesting cases.

Thanks,
Pavan

-- 
 Pavan Deolasee                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

  154. Re: [HACKERS] MERGE SQL Statement for PG11

    Pavan Deolasee <pavan.deolasee@gmail.com> — 2018-02-14T05:56:55Z

    Hi Stephen,

On Tue, Feb 6, 2018 at 3:37 PM, Stephen Frost <sfrost@snowman.net> wrote:

>
>
> Coming out of that, my understanding is that Simon is planning to have a
> patch which implements RLS and partitioning (though the query plans for
> partitioning may be sub-par and not ideal) as part of MERGE and I've
> agreed to review at least the RLS bits (though my intention is to at
> least go through the rest of the patch as well, though likely in less
> detail).  Of course, I encourage others to review it as well.
>
>
Thanks for volunteering to review the RLS bits. I am planning to send a
revised version soon. As I work through it, I am faced with some semantic
questions again. Would appreciate if you or anyone have answers to those.

While executing MERGE, for existing tuples in the target table, we may end
up doing an UPDATE or DELETE, depending on the WHEN MATCHED AND conditions.
So it seems unlikely that we would be able to push down USING security
quals down to the scan. For example, if the target row is set for deletion,
it seems wrong to exclude the row from the join based on UPDATE policy's
USING quals. So I am thinking that we should apply the respective USING
quals *after* the decision to either update, delete or do nothing for the
given target tuple is made.

The question I have is, if the USING qual evaluates to false or NULL,
should we silently ignore the tuple (like regular UPDATE does) or throw an
error (like INSERT ON CONFLICT DO UPDATE)? ISTM that we might have decided
to throw an error in case of INSERT ON CONFLICT to avoid any confusion
where the tuple is neither inserted nor updated. Similar situation may
arise with MERGE because for a source row, we may neither do UPDATE
(because of RLS) nor INSERT because a matching tuple already exists. But
someone may argue that we should stay closer to regular UPDATE/DELETE.
Apart from that, are there any security angles that we need to be mindful
of and would those impact the choice?

SELECT policies will be applied to the target table during the scan and
rows which do not pass SELECT quals will not be processed at all. If there
are NOT MATCHED actions, we might end up inserting duplicate rows in that
case or throw errors, but I don't see anything wrong with that. Similar
things would have happened if someone tried to insert rows into the table
using regular INSERT.

Similarly, INSERT policies will be applied when MERGE attempts to INSERT a
row into the table and error will be thrown if the row does not satisfy
INSERT policies.

Thanks,
Pavan

-- 
 Pavan Deolasee                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

  155. Re: [HACKERS] MERGE SQL Statement for PG11

    Stephen Frost <sfrost@snowman.net> — 2018-02-14T16:29:28Z

    Greetings Pavan,

* Pavan Deolasee (pavan.deolasee@gmail.com) wrote:
> On Tue, Feb 6, 2018 at 3:37 PM, Stephen Frost <sfrost@snowman.net> wrote:
> > Coming out of that, my understanding is that Simon is planning to have a
> > patch which implements RLS and partitioning (though the query plans for
> > partitioning may be sub-par and not ideal) as part of MERGE and I've
> > agreed to review at least the RLS bits (though my intention is to at
> > least go through the rest of the patch as well, though likely in less
> > detail).  Of course, I encourage others to review it as well.
>
> Thanks for volunteering to review the RLS bits. I am planning to send a
> revised version soon. As I work through it, I am faced with some semantic
> questions again. Would appreciate if you or anyone have answers to those.

Thanks for working on this.

> While executing MERGE, for existing tuples in the target table, we may end
> up doing an UPDATE or DELETE, depending on the WHEN MATCHED AND conditions.
> So it seems unlikely that we would be able to push down USING security
> quals down to the scan. For example, if the target row is set for deletion,
> it seems wrong to exclude the row from the join based on UPDATE policy's
> USING quals. So I am thinking that we should apply the respective USING
> quals *after* the decision to either update, delete or do nothing for the
> given target tuple is made.
> 
> The question I have is, if the USING qual evaluates to false or NULL,
> should we silently ignore the tuple (like regular UPDATE does) or throw an
> error (like INSERT ON CONFLICT DO UPDATE)? ISTM that we might have decided
> to throw an error in case of INSERT ON CONFLICT to avoid any confusion
> where the tuple is neither inserted nor updated. Similar situation may
> arise with MERGE because for a source row, we may neither do UPDATE
> (because of RLS) nor INSERT because a matching tuple already exists. But
> someone may argue that we should stay closer to regular UPDATE/DELETE.

The reasoning behind the INSERT ON CONFLICT DO UPDATE approach when it
comes to RLS is that we specifically didn't want to end up "losing"
data- an INSERT which doesn't actually INSERT a row (or take the UPDATE
action if the row already exists) ends up throwing that data away even
though clearly the user expected us to do something with it, which is
why we throw an error in that case instead.

For my part, at least, it seems like MERGE would likewise possibly be
throwing away data that the user is expecting to be incorporated if no
action is taken due to RLS and that then argues that we should be
throwing an error in such a case, similar to the INSERT ON CONFLICT DO
UPDATE case.

> Apart from that, are there any security angles that we need to be mindful
> of and would those impact the choice?

Regarding the above, no security issues come to mind with either
approach.  The security concerns are primairly not allowing rows to be
directly seen which the user does not have access to, and not allowing
the action to result in rows being added, modified, or removed in a way
which would violate the policies defined.

> SELECT policies will be applied to the target table during the scan and
> rows which do not pass SELECT quals will not be processed at all. If there
> are NOT MATCHED actions, we might end up inserting duplicate rows in that
> case or throw errors, but I don't see anything wrong with that. Similar
> things would have happened if someone tried to insert rows into the table
> using regular INSERT.

I agree, this seems like the right approach.

> Similarly, INSERT policies will be applied when MERGE attempts to INSERT a
> row into the table and error will be thrown if the row does not satisfy
> INSERT policies.

That sounds correct to me.

Thanks!

Stephen

  156. Re: [HACKERS] MERGE SQL Statement for PG11

    Pavan Deolasee <pavan.deolasee@gmail.com> — 2018-02-15T11:17:26Z

    Hi Stephen,

On Wed, Feb 14, 2018 at 9:59 PM, Stephen Frost <sfrost@snowman.net> wrote:

>
>
> The reasoning behind the INSERT ON CONFLICT DO UPDATE approach when it
> comes to RLS is that we specifically didn't want to end up "losing"
> data- an INSERT which doesn't actually INSERT a row (or take the UPDATE
> action if the row already exists) ends up throwing that data away even
> though clearly the user expected us to do something with it, which is
> why we throw an error in that case instead.
>
> For my part, at least, it seems like MERGE would likewise possibly be
> throwing away data that the user is expecting to be incorporated if no
> action is taken due to RLS and that then argues that we should be
> throwing an error in such a case, similar to the INSERT ON CONFLICT DO
> UPDATE case.
>

Thanks for confirming the approach. That matches my own thinking too.

Here is an updated v16a patch. This now supports RLS based on what we
agreed above. I've added a few test cases to confirm that RLS works
correctly with MERGE. We can more later if needed.

This version now also adds support for OVERRIDING clause in the INSERT
statement. The whole var referencing issue pointed out by Peter is also
fixed by this version. So to summarise the following things are now working:

- Partitioning
- Subqueries in WHEN AND quals and UPDATE targetlists
- Row level security (documentation changes are pending)
- OVERRIDING clause
- Various bugs reported by Peter are fixed (I haven't double checked if all
issues are addressed or not, but we should be fairly close)
- Issues so far reported by Andreas Seltenreich as part of sqlsmith testing
are fixed

I haven't yet addressed Tomas's review comment on using WAL position for
write detection. I'm waiting for Simon to come back from holidays before
doing anything there. One idea is to use some variant
of contain_mutable_functions() and throw error during query planning, but
not do anything during execution.

Other than that, I think we are getting to a point where the patch is
mostly feature complete. We still need to decide the concurrent execution
behaviour, but hopefully code changes there will be limited (unless we try
to do something very invasive, which I hope we don't).

I'm not entirely happy with the way we're going about resolving column
references in INSERT/UPDATE targetlist and subsequently what we are doing
in setrefs.c. I'll continue to look for improvements there and also hoping
to get suggestions from this list.

Thanks,
Pavan

-- 
 Pavan Deolasee                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

  157. Re: [HACKERS] MERGE SQL Statement for PG11

    Pavan Deolasee <pavan.deolasee@gmail.com> — 2018-02-15T11:19:46Z

    Hi Andreas,

On Sun, Feb 4, 2018 at 5:45 PM, Andreas Seltenreich <seltenreich@gmx.de>
wrote:

>
>
> Attached are testcases that trigger the assertions when run against an
> empty database instead of the one left behind by make installcheck.  The
> "unrecognized node type" one appears to be a known issue according to
> the wiki, so I didn't bother doing the work for this one as well.
>

Thanks for doing those tests. I've just sent v16a version of the patch and
I think it fixes the issues reported so far. Can you please recheck? Please
let me know if there are other issues detected by sqlsmith or otherwise.

Thanks,
Pavan

-- 
 Pavan Deolasee                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

  158. Re: [HACKERS] MERGE SQL Statement for PG11

    Peter Geoghegan <pg@bowt.ie> — 2018-02-16T01:07:08Z

    On Sun, Feb 11, 2018 at 11:09 PM, Pavan Deolasee
<pavan.deolasee@gmail.com> wrote:
> On Fri, Feb 9, 2018 at 6:53 AM, Peter Geoghegan <pg@bowt.ie> wrote:
>> My concern is mostly just that MERGE manages to behave in a way that
>> actually "provides a single SQL statement that can conditionally
>> INSERT, UPDATE or DELETE rows, a task that would otherwise require
>> multiple procedural language statements", as the docs put it. As long
>> as MERGE manages to do something as close to that high level
>> description as possible in READ COMMITTED mode (with our current
>> semantics for multiple statements in RC taken as the baseline), then
>> I'll probably be happy.
>
>
> IMO it will be quite hard, if not impossible, to guarantee the same
> semantics to a single statement MERGE and multi statement
> UPDATE/DELETE/INSERT in RC mode. For example, the multi statement model will
> execute each statement with a new MVCC snapshot and hence the rows visible
> to individual statement may vary. Whereas in MERGE, everything runs with a
> single snapshot. There could be other such subtle differences.

I didn't mean this literally. For simple cases, an EPQ walk of the
update chain is kind of like acquiring a new snapshot.

ISTM that a MERGE isn't really a thing that replaces 2 or 3 other DML
statements, at least in most cases. It's more like a replacement for
procedural code with an outer join, with an INSERT, UPDATE or DELETE
that affects zero or one rows inside the procedural loop that
processes matching/non-matching rows. The equivalent procedural code
could ultimately perform *thousands* of snapshot acquisitions for
thousands of RC DML statements. MERGE is sometimes explained in terms
of "here is the kind of procedural code that you don't have to write
anymore, thanks to MERGE" -- that's what the code looks like.

I attach a rough example of this, that uses plpgsql.

>> Some novel new behavior -- "EPQ with a twist"-- is clearly necessary.
>> I feel a bit uneasy about it because anything that anybody suggests is
>> likely to be at least a bit arbitrary (EPQ itself is kind of
>> arbitrary). We only get to make a decision on how "EPQ with a twist"
>> will work once, and that should be a decision that is made following
>> careful deliberation. Ambiguity is much more likely to kill a patch
>> than a specific technical defect, at least in my experience. Somebody
>> can usually just fix a technical defect.
>
>
> While I agree, I think we need to make these decisions in a time bound
> fashion. If there is too much ambiguity, then it's not a bad idea to settle
> for throwing appropriate errors instead of providing semantically wrong
> answers, even in some remote corner case.

Everything is still on the table, I think.

> Ok. I am now back from holidays and I will too start thinking about this.
> I've also requested a colleague to help us with comparing it against
> Oracle's behaviour. N That's not a gold standard for us, but knowing how
> other major databases handle RC conflicts, is not a bad idea.

The fact that Oracle doesn't allow WHEN MATCHED ... AND quals did seem
like it might be significant to me.

I think that any theoretical justification for one behavior over
another will be hard for anyone to come up with. As I said before,
this is not an area where something like the SQL standard provides us
with a platonic ideal. The best behavior is likely to be one that
lives up to the high level description of MERGE, is as close as
possible to existing behaviors, and is not otherwise surprising.

> I see the following important areas and as long as we have a consistent and
> coherent handling of these cases, we should not have difficulty agreeing on
> a outcome.
>
> 1. Concurrent UPDATE does not affect MATCHED  case. The WHEN conditions may
> or may not be affected.
> 2. Concurrently UPDATEd tuple fails the join qual and the current source
> tuple no longer matches with the updated target tuple that the EPQ is set
> for. It matches no other target tuple either. So a MATCHED case is turned
> into a NOT MATCHED case.
> 3. Concurrently UPDATEd tuple fails the join qual and the current source
> tuple no longer matches with the updated target tuple that the EPQ is set
> for. But it matches some other target tuple. So it's still a MATCHED case,
> but with different target tuple(s).
> 4. Concurrent UPDATE/INSERT creates a matching target tuple for a source
> tuple, thus turning a NOT MATCHED case to a MATCHED case.
> 5. Concurrent DELETE turns a MATCHED case into NOT MATCHED case
>
> Any other case that I am missing? Assuming all cases are covered, what
> should we do in each of these cases, so that there is no or very little
> ambiguity and the outcome seems consistent (at the very least as far as
> MERGE goes and hopefully with regular UPDATE/DELETE handling)

Uhhh...I still need to spend more time on this. Sorry.

> It seems natural that #5 should skip the MATCHED action and instead execute
> the first satisfying NOT MATCHED action. But it's outcome may depend on how
> we handle #2 and #3 so that they are all consistent. If we allow #2 and #3
> to error out, whenever there is ambiguity, we should do the same for #5.

I definitely agree on behaving consistently in the way you describe here.

-- 
Peter Geoghegan

  159. Re: [HACKERS] MERGE SQL Statement for PG11

    Andreas Seltenreich <seltenreich@gmx.de> — 2018-02-18T11:18:43Z

    Pavan Deolasee writes:

> Thanks for doing those tests. I've just sent v16a version of the patch and
> I think it fixes the issues reported so far. Can you please recheck? Please
> let me know if there are other issues detected by sqlsmith or otherwise.

I re-did the testing with merge_v16a applied to master at 7923118c16
with ad7dbee368a reverted because of conflicts.  I can confirm that the
previous testcases don't fail anymore, but sqlsmith readily triggers the
following assertion:

TRAP: FailedAssertion("!(mergeTargetRelation > 0)", File: "planner.c",
Line: 1496)

Testcase attached.

regards,
Andreas

  160. Re: [HACKERS] MERGE SQL Statement for PG11

    Pavan Deolasee <pavan.deolasee@gmail.com> — 2018-02-27T17:53:06Z

    Hi Andreas,

Sorry for the late response; I was busy at PGConf India.

On Sun, Feb 18, 2018 at 4:48 PM, Andreas Seltenreich <seltenreich@gmx.de>
wrote:

> Pavan Deolasee writes:
>
> > Thanks for doing those tests. I've just sent v16a version of the patch
> and
> > I think it fixes the issues reported so far. Can you please recheck?
> Please
> > let me know if there are other issues detected by sqlsmith or otherwise.
>
> I re-did the testing with merge_v16a applied to master at 7923118c16
> with ad7dbee368a reverted because of conflicts.  I can confirm that the
> previous testcases don't fail anymore,


Thanks for confirming.


> but sqlsmith readily triggers the
> following assertion:
>
> TRAP: FailedAssertion("!(mergeTargetRelation > 0)", File: "planner.c",
> Line: 1496)
>
>
Ah. Looks like the support and test cases for sub-partitioning were
missing. I've attached v17a of the patch which fixes this and some other
issues I noticed as part of your testing. To provide more details, while we
allow more complex un-aliased joins for the source relation, referencing
columns from the join was broken. I've now fixed that.

I've added relevant test cases to the regression.  Also added some tests to
check function scans.

The patch is rebased on the current master. Please let me know how the
sqlsmith tests go with this new version.

Thanks,
Pavan

-- 
 Pavan Deolasee                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

  161. Re: [HACKERS] MERGE SQL Statement for PG11

    Pavan Deolasee <pavan.deolasee@gmail.com> — 2018-02-27T18:19:11Z

    On Fri, Feb 16, 2018 at 6:37 AM, Peter Geoghegan <pg@bowt.ie> wrote:

>
> ISTM that a MERGE isn't really a thing that replaces 2 or 3 other DML
> statements, at least in most cases. It's more like a replacement for
> procedural code with an outer join, with an INSERT, UPDATE or DELETE
> that affects zero or one rows inside the procedural loop that
> processes matching/non-matching rows. The equivalent procedural code
> could ultimately perform *thousands* of snapshot acquisitions for
> thousands of RC DML statements. MERGE is sometimes explained in terms
> of "here is the kind of procedural code that you don't have to write
> anymore, thanks to MERGE" -- that's what the code looks like.
>
> I attach a rough example of this, that uses plpgsql.
>

Thanks for writing the sample code. I understand you probably don't mean to
suggest that we need to mimic the behaviour of the plpgsql code and the
semantics offered by MERGE would most likely be different than what the
plpgsql sample does. Because there are several problems with the plpgsql
code:

- It would never turn a MATCHED case into a NOT MATCHED case because of
concurrent UPDATE/DELETE
- The WHERE clauses attached to the UPDATE/DELETE statement should be using
the quals attached to the WHEN clauses to ensure they are evaluated on the
new version of the row, if needed.


>
> >> Some novel new behavior -- "EPQ with a twist"-- is clearly necessary.
> >> I feel a bit uneasy about it because anything that anybody suggests is
> >> likely to be at least a bit arbitrary (EPQ itself is kind of
> >> arbitrary). We only get to make a decision on how "EPQ with a twist"
> >> will work once, and that should be a decision that is made following
> >> careful deliberation. Ambiguity is much more likely to kill a patch
> >> than a specific technical defect, at least in my experience. Somebody
> >> can usually just fix a technical defect.
>

TBH that's one reason why I like Simon's proposed behaviour of throwing
errors in case of corner cases. I am not suggesting that's what we do at
the end, but it's definitely worth considering.


> >
> >
> > While I agree, I think we need to make these decisions in a time bound
> > fashion. If there is too much ambiguity, then it's not a bad idea to
> settle
> > for throwing appropriate errors instead of providing semantically wrong
> > answers, even in some remote corner case.
>
> Everything is still on the table, I think.
>

Ok.


>
> > Ok. I am now back from holidays and I will too start thinking about this.
> > I've also requested a colleague to help us with comparing it against
> > Oracle's behaviour. N That's not a gold standard for us, but knowing how
> > other major databases handle RC conflicts, is not a bad idea.
>
> The fact that Oracle doesn't allow WHEN MATCHED ... AND quals did seem
> like it might be significant to me.
>
>
Here are some observations from Rahila's analysis so far. I must say,
Oracle's handling seems quite inconsistent, especially the conditions under
which it sometimes re-evaluates the join and sometimes don't.

- Oracle does not support multiple WHEN MATCHED clauses. So the question of
re-checking all WHEN clauses does not arise.

- Only one UPDATE and one DELETE clause is supported. The DELETE must be
used in conjunction with UPDATE.

- The DELETE clause is invoked iff the UPDATE clause is invoked. It works
on the updated rows. Since the row is already updated (and locked) by the
MERGE, DELETE action never blocks on a concurrent update/delete

- MERGE does not allow updating the column used in the JOIN's ON qual

- In case of concurrent UPDATE, the join is re-evaluated iff the concurrent
  UPDATE updates (modifies?) the same column that MERGE is updating OR a
column
  that MERGE is referencing in the WHERE clause is updated by the
concurrent update. IOW if the
  MERGE and concurrent UPDATE is operating on different columns, join is NOT
  re-evaluated, thus possibly invoking WHEN MATCHED action on a row which no
  longer matches the join condition.

- In case of concurrent DELETE, the join is re-evaluated and the action may
change from MATCHED to NOT MATCHED

I am curiously surprised by it's behaviour of re-evaluating join only when
certain columns are updated. It looks to me irrespective of what we choose,
our implementation would be much superior to what Oracle offers.

BTW I've sent v17a of the patch, which is very close to being complete from
my perspective (except some documentation fixes/improvements). The only
thing pending is the decision to accept or change the currently implemented
concurrency semantics.

Thanks,
Pavan

-- 
 Pavan Deolasee                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

  162. Re: [HACKERS] MERGE SQL Statement for PG11

    Peter Geoghegan <pg@bowt.ie> — 2018-02-27T22:07:28Z

    On Tue, Feb 27, 2018 at 10:19 AM, Pavan Deolasee
<pavan.deolasee@gmail.com> wrote:
>> I attach a rough example of this, that uses plpgsql.
>
> Thanks for writing the sample code. I understand you probably don't mean to
> suggest that we need to mimic the behaviour of the plpgsql code and the
> semantics offered by MERGE would most likely be different than what the
> plpgsql sample does. Because there are several problems with the plpgsql
> code:
>
> - It would never turn a MATCHED case into a NOT MATCHED case because of
> concurrent UPDATE/DELETE
> - The WHERE clauses attached to the UPDATE/DELETE statement should be using
> the quals attached to the WHEN clauses to ensure they are evaluated on the
> new version of the row, if needed.

It's definitely possible to poke holes in my plpgsql example, most (or
all) of which are not fixable within the confines of what plpgsql can
do. I still think that it's really useful to frame the discussion with
examples of the kind of procedural code MERGE replaces, though,
because:

* That's the explicit purpose of MERGE, according to the SQL standard.
Everyone is very clear that the join outputs rows that are separately
inserted, updated, or deleted (additional "WHEN ... AND" quals are
evaluated separately). We should be clear on that, too. We're quite
specifically replacing procedural code that follows a general pattern.
We might even give an example of such procedural code in the docs, as
SQL Server does.

* It shows that in some ways, the INSERT/UPDATE/DELETE parts are
separate "zero or one row" statements. They could do their own
snapshot acquisitions within RC mode, for example. Also, you don't get
the ON CONFLICT behavior with excluded being affected by BEFORE ROW
triggers within UPDATE expression evaluation for ON CONFLICT DO
UPDATE.

* My example is buggy, but seemingly only in a way that is just about
unavoidable -- all the bugs are in RC mode with concurrent writes.
Poking holes in what I came up with is actually useful, and may be
less confusing than discussing the same issues some other way.

There are very few users in the world that would understand these
issues. Moreover, if all affected users that have this kind of code in
the wild were to somehow magically develop a strong understanding of
this stuff overnight, even then I'm not sure that much would change.
They still use RC mode for all the usual reasons, and they mostly
won't have any way of fixing concurrency issues that is actually
acceptable to them. In short, I'm not sure that I can fix the problems
with my plpgsql code, so what chance do they have of fixing theirs?

>> >> Some novel new behavior -- "EPQ with a twist"-- is clearly necessary.
>> >> I feel a bit uneasy about it because anything that anybody suggests is
>> >> likely to be at least a bit arbitrary (EPQ itself is kind of
>> >> arbitrary). We only get to make a decision on how "EPQ with a twist"
>> >> will work once, and that should be a decision that is made following
>> >> careful deliberation. Ambiguity is much more likely to kill a patch
>> >> than a specific technical defect, at least in my experience. Somebody
>> >> can usually just fix a technical defect.
>
>
> TBH that's one reason why I like Simon's proposed behaviour of throwing
> errors in case of corner cases. I am not suggesting that's what we do at the
> end, but it's definitely worth considering.

I now feel like Simon's suggestion of throwing an error in corner
cases isn't so bad. It still seems like we could do better, but the
more I think about it, the less that seems like a cop-out. My reasons
are:

* As I already said, my plpgsql example, while buggy, might actually
be the safest way to get the intended behavior in RC mode today. I can
definitely imagine a way of dealing with concurrency that is both
safer and less prone to throwing weird errors, but the fact remains
that my example is the state of the art here, in a way.

* Simon has already introduced something that looks like "EPQ with a
twist" to me -- the steps that happen before he even raises this
error. IOW, he does something extra that is EPQ-like. He likely does a
lot better than my plpgsql example manages to, I think.

* I suspect that the kind of users that really like the ON CONFLICT DO
UPDATE's simplicity (in terms of what it guarantees them) are unlikely
to care about MERGE at all. The kind of user that cares about MERGE is
likely to have at least heard of the isolation levels.

* I do see an important difference between making likely-unexpected
errors in RC mode very unlikely, and making them *impossible*. This
patch is not ON CONFLICT DO UPDATE, though, and that strong guarantee
simply isn't on the table.

* We can all agree that *not* raising an error in the specific way
Simon proposes is possible, somehow or other. We also all agree that
avoiding the broader category of RC errors can only be taken so far
(e.g. in any event duplicate violations errors are entirely possible,
in RC mode, when a MERGE inserts a row). So this is a question of what
exact middle ground to take. Neither of the two extremes (throwing an
error on the first sign of a RC conflict, and magically preventing
concurrency anomalies) are actually on the table.

I will not block this patch because it merely makes throwing
likely-unexpected errors in RC mode "very unlikely", rather than "very
very unlikely". Not least because I have a hard time imagining a
single user caring about the difference that still exists with Simon's
less ambitious (though not entirely unambitious) version of "EPQ with
a twist".

> Here are some observations from Rahila's analysis so far. I must say,
> Oracle's handling seems quite inconsistent, especially the conditions under
> which it sometimes re-evaluates the join and sometimes don't.

> I am curiously surprised by it's behaviour of re-evaluating join only when
> certain columns are updated. It looks to me irrespective of what we choose,
> our implementation would be much superior to what Oracle offers.

I'm not that surprised that it's generally kind of arbitrary, though
the fact that you can update something while the join qual no longer
passes does seem particularly poor.

BTW, one thing that I remember very clearly from my research on MERGE
years ago is this: all of the major implementations were in some way
or other quite buggy. At least in RC mode, MERGE kind of promises
something that it really can't quite deliver. Implementations are
seemingly left to pick and choose how to paper over those cracks.

The description of what Oracle allows here does make me feel better
about our direction. Having the least-worst semantics on RC conflict
handling certainly seems good enough to me. Especially because we have
a "true UPSERT" already, unlike both Oracle and SQL Server.

> BTW I've sent v17a of the patch, which is very close to being complete from
> my perspective (except some documentation fixes/improvements). The only
> thing pending is the decision to accept or change the currently implemented
> concurrency semantics.

I need to go look at that. I'll try to take a firmer position on this.
I know that I've been saying that for a quite a while now, but my
failure to take a firmer position for so long is not because I didn't
try. It's because there is no really good answer.

-- 
Peter Geoghegan

  163. Re: [HACKERS] MERGE SQL Statement for PG11

    Robert Haas <robertmhaas@gmail.com> — 2018-02-28T16:53:56Z

    On Tue, Feb 27, 2018 at 5:07 PM, Peter Geoghegan <pg@bowt.ie> wrote:
> I now feel like Simon's suggestion of throwing an error in corner
> cases isn't so bad. It still seems like we could do better, but the
> more I think about it, the less that seems like a cop-out. My reasons
> are:

I still think we really ought to try not to add a new class of error.

> * We can all agree that *not* raising an error in the specific way
> Simon proposes is possible, somehow or other. We also all agree that
> avoiding the broader category of RC errors can only be taken so far
> (e.g. in any event duplicate violations errors are entirely possible,
> in RC mode, when a MERGE inserts a row). So this is a question of what
> exact middle ground to take. Neither of the two extremes (throwing an
> error on the first sign of a RC conflict, and magically preventing
> concurrency anomalies) are actually on the table.

Just because there's no certainty about which behavior is best doesn't
mean that none of them are better than throwing an error.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  164. Re: [HACKERS] MERGE SQL Statement for PG11

    Peter Geoghegan <pg@bowt.ie> — 2018-02-28T21:00:13Z

    On Wed, Feb 28, 2018 at 8:53 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> On Tue, Feb 27, 2018 at 5:07 PM, Peter Geoghegan <pg@bowt.ie> wrote:
>> I now feel like Simon's suggestion of throwing an error in corner
>> cases isn't so bad. It still seems like we could do better, but the
>> more I think about it, the less that seems like a cop-out. My reasons
>> are:
>
> I still think we really ought to try not to add a new class of error.

I do too. However, it's only fair to acknowledge that the lack of any
strong counterproposal after a month or so tells us something.

>> * We can all agree that *not* raising an error in the specific way
>> Simon proposes is possible, somehow or other. We also all agree that
>> avoiding the broader category of RC errors can only be taken so far
>> (e.g. in any event duplicate violations errors are entirely possible,
>> in RC mode, when a MERGE inserts a row). So this is a question of what
>> exact middle ground to take. Neither of the two extremes (throwing an
>> error on the first sign of a RC conflict, and magically preventing
>> concurrency anomalies) are actually on the table.
>
> Just because there's no certainty about which behavior is best doesn't
> mean that none of them are better than throwing an error.

Bear in mind that the patch doesn't throw an error at the first sign
of trouble. It throws an error after doing an EPQ style walk, which
individually verifies the extra "WHEN ... AND" quals for every tuple
in the update chain (ExecUpdate()/ExecDelete() return after first EPQ
tuple check for MERGE, so that an ExecMerge() caller can do that extra
part with special EPQ slot). The controversial serialization error
only comes when the original basic assessment of MATCHED needs to
change, without regard to extra "WHEN ... AND" quals (and only when
the MERGE statement was written in such a way that that matters -- it
must have a WHEN NOT MATCHED clause, too).

AFAICT, the patch will only throw an error when the join quals no
longer pass -- not when the extra "WHEN ... AND" quals no longer pass.
That would be far worse, IMV. ExecMerge() will retry until it reaches
the end of the update chain, possibly changing its mind about which
particular WHEN case applies repeatedly, going back and forth between
mergeActions (WHEN cases) as the update chain in walked. The patch can
do a lot to roll with conflicts before it gives up. Conflicts are
generally caused by concurrent DELETEs, so you could say that this
offers a kind of symmetry with concurrent INSERTs causing duplicate
violation errors. (Concurrent UPDATEs that change the join qual values
could provoke the error too, but presumably we're joining on the PK in
most cases, so that seems much less likely than a simple DELETE.)

Bear in mind that both the SQL Server and Oracle implementations have
many significant restrictions/caveats around doing something cute with
the main join quals. It's not surprising that we'd have something like
that, too. According to Rahila, Oracle doesn't allow MERGE to update
the columns in the join qual at all, and only allows a single WHEN
MATCHED case (a DELETE can only come after an UPDATE in Oracle, oddly
enough). SQL Server's docs have a warning that states: "Do not attempt
to improve query performance by filtering out rows in the target table
in the ON clause, such as by specifying AND NOT target_table.column_x
= value. Doing so may return unexpected and incorrect results.". What
does that even mean!?

I am slightly concerned that the patch may still have livelock hazards
in its approach to RC conflict handling. I haven't studied that aspect
in enough detail, though. Since we always make forward progress by
following an update chain, any problem here is probably fixable.

-- 
Peter Geoghegan

  165. Re: [HACKERS] MERGE SQL Statement for PG11

    Peter Geoghegan <pg@bowt.ie> — 2018-02-28T22:20:31Z

    On Fri, Feb 9, 2018 at 6:36 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> Here's my $0.02: I think that new concurrency errors thrown by the
> merge code itself deserve strict scrutiny and can survive only if they
> have a compelling justification, but if the merge code happens to
> choose an action that leads to a concurrency error of its own, I think
> that deserves only mild scrutiny.
>
> On that basis, of the options I listed in
> http://postgr.es/m/CA+TgmoZDL-caukHkWet7sr7sqr0-e2T91+DEvhqeN5sfqsMjqw@mail.gmail.com
> I like #1 least.
>
> I also dislike #4 from that list for the reasons stated there.  For
> example, if you say WHEN MATCHED AND x.some_boolean and then WHEN
> MATCHED, you expect that every tuple that hits the latter clause will
> have that Boolean as false or null, but #4 makes that not true.
>
> I think the best options are #2 and #5 -- #2 because it's simple, and
> #5 because it's (maybe) more intuitive, albeit at the risk of
> livelock.  But I'm willing to convinced of some other option; like
> you, I'm willing to be open-minded about this.  But, as you say, we
> need a coherent, well-considered justification for the selected
> option, not just "well, this is what we implemented so you should like
> it".  The specification should define the implementation, not the
> reverse.

At first I hated option #2. I'm warming to #2 a lot now, though,
because I've come to understand the patch's approach a little better.
(Pavan and Simon should still verify that I got things right in my
mail from earlier today, though.)

It now seems like the patch throws a RC serialization error more or
less only due to concurrent deletions (rarely, it will actually be a
concurrent update that changed the join qual values of our MERGE).
We're already not throwing the error (we just move on to the next
input row from the join) when we happen to not have a WHEN NOT MATCHED
case. But why even make that distinction? Why not just go ahead and
WHEN NOT MATCHED ... INSERT when the MERGE happens to have such a
clause? The INSERT will succeed, barring any concurrent conflicting
insertion by a third session -- a hazard that has nothing to do with
RC conflict handling in particular.

Just inserting without throwing a RC serialization error is almost
equivalent to a new MVCC snapshot being acquired due to a RC conflict,
so all of this now looks okay to me. Especially because every other
MERGE implementation seems to have serious issues with doing anything
too fancy with the MERGE target table's quals within the main ON join.
I think that SQL Server actually assumes that you're using the
target's PK in a simple equi-join. All the examples look like that,
and this assumption is heavily suggested by the "Do not attempt to
improve query performance by filtering out rows in the target table in
the ON clause" weasel words from their docs, that I referenced in my
mail from earlier today.

I can get my head around all of this now only because I've come to
understand that once we've decided that a given input from the main
join is NOT MATCHED, we stick to that determination. We don't bounce
around between MATCHED and NOT MATCHED cases during an EPQ update
chain walk. We can bounce around between multiple alternative MATCHED
merge actions/WHEN cases, but that seems okay because it's still part
of essentially the same EPQ update chain walk -- no obvious livelock
hazard. It seems fairly clean to restart everything ("goto lmerge")
for each and every tuple in the update chain.

Maybe we should actually formalize that you're only supposed to do a
PK or unique index equi-join within the main ON join, though -- you
can do something fancy with the source table, but not the target
table.

-- 
Peter Geoghegan

  166. Re: [HACKERS] MERGE SQL Statement for PG11

    Pavan Deolasee <pavan.deolasee@gmail.com> — 2018-03-01T12:33:11Z

    On Thu, Mar 1, 2018 at 3:50 AM, Peter Geoghegan <pg@bowt.ie> wrote:

> On Fri, Feb 9, 2018 at 6:36 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> > Here's my $0.02: I think that new concurrency errors thrown by the
> > merge code itself deserve strict scrutiny and can survive only if they
> > have a compelling justification, but if the merge code happens to
> > choose an action that leads to a concurrency error of its own, I think
> > that deserves only mild scrutiny.
> >
> > On that basis, of the options I listed in
> > http://postgr.es/m/CA+TgmoZDL-caukHkWet7sr7sqr0-e2T91+
> DEvhqeN5sfqsMjqw@mail.gmail.com
> > I like #1 least.
> >
> > I also dislike #4 from that list for the reasons stated there.  For
> > example, if you say WHEN MATCHED AND x.some_boolean and then WHEN
> > MATCHED, you expect that every tuple that hits the latter clause will
> > have that Boolean as false or null, but #4 makes that not true.
> >
> > I think the best options are #2 and #5 -- #2 because it's simple, and
> > #5 because it's (maybe) more intuitive, albeit at the risk of
> > livelock.  But I'm willing to convinced of some other option; like
> > you, I'm willing to be open-minded about this.  But, as you say, we
> > need a coherent, well-considered justification for the selected
> > option, not just "well, this is what we implemented so you should like
> > it".  The specification should define the implementation, not the
> > reverse.
>
> At first I hated option #2. I'm warming to #2 a lot now, though,
> because I've come to understand the patch's approach a little better.
> (Pavan and Simon should still verify that I got things right in my
> mail from earlier today, though.)
>
> It now seems like the patch throws a RC serialization error more or
> less only due to concurrent deletions (rarely, it will actually be a
> concurrent update that changed the join qual values of our MERGE).
> We're already not throwing the error (we just move on to the next
> input row from the join) when we happen to not have a WHEN NOT MATCHED
> case. But why even make that distinction? Why not just go ahead and
> WHEN NOT MATCHED ... INSERT when the MERGE happens to have such a
> clause? The INSERT will succeed, barring any concurrent conflicting
> insertion by a third session -- a hazard that has nothing to do with
> RC conflict handling in particular.
>
> Just inserting without throwing a RC serialization error is almost
> equivalent to a new MVCC snapshot being acquired due to a RC conflict,
> so all of this now looks okay to me. Especially because every other
> MERGE implementation seems to have serious issues with doing anything
> too fancy with the MERGE target table's quals within the main ON join.
> I think that SQL Server actually assumes that you're using the
> target's PK in a simple equi-join. All the examples look like that,
> and this assumption is heavily suggested by the "Do not attempt to
> improve query performance by filtering out rows in the target table in
> the ON clause" weasel words from their docs, that I referenced in my
> mail from earlier today.
>
>
I think you've fairly accurately described what the patch does today. I
take your point that we can very well just execute the WHEN NOT MATCHED
action if the join condition fails for the updated tuple. There is one case
we ought to think about though and that might explain why executing the
WHEN NOT MATCHED action may not be best choice. Or for that matter even
skipping the target when no NOT MATCHED action exists, as the patch does
today.

What if the updated tuple fails the join qual with respect to the current
tuple from the source relation but it now matches some other tuple from the
source relation? I described this case in one of the earlier emails too. In
this case, we might end up doing an INSERT (if we decide to execute WHEN
NOT MATCHED action), even though a MATCH exists. If there is no WHEN NOT
MATCHED action, the current patch will just skip the updated tuple even
though a match exists, albeit it's not the current source tuple.

Oracle behaves differently and it actually finds a new matching tuple from
the source relation and executes the WHEN MATCHED action, using that source
tuple. But I am seriously doubtful that we want to go down that path and
whether it's even feasible. Our regular UPDATE .. FROM does not do that
either. Given that, it seems better to just throw an error (even when no
NOT MATCHED action exists) and explain to the users that MERGE will work as
long as concurrent updates don't modify the columns used in the join
condition. Concurrent deletes should be fine and we may actually even
invoke WHEN NOT MATCHED action in that case.

Thanks,
Pavan

-- 
 Pavan Deolasee                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

  167. Re: [HACKERS] MERGE SQL Statement for PG11

    Peter Geoghegan <pg@bowt.ie> — 2018-03-01T19:06:09Z

    On Thu, Mar 1, 2018 at 4:33 AM, Pavan Deolasee <pavan.deolasee@gmail.com> wrote:
> What if the updated tuple fails the join qual with respect to the current
> tuple from the source relation but it now matches some other tuple from the
> source relation? I described this case in one of the earlier emails too. In
> this case, we might end up doing an INSERT (if we decide to execute WHEN NOT
> MATCHED action), even though a MATCH exists. If there is no WHEN NOT MATCHED
> action, the current patch will just skip the updated tuple even though a
> match exists, albeit it's not the current source tuple.

If it does happen, then that will typically be because someone else
concurrently updated a row, changing the primary key attributes, or
some unique index attributes that our MERGE joins on, which I think is
pretty rare. I'm assuming that the user does an idiomatic MERGE, like
every example I can find shows, where the join quals on the target
table are simple equality predicates on the primary key attribute(s).

I think it's fine to simply let the insertion fail with a duplicate
error. Is this any different to a concurrent INSERT that produces that
same outcome?

If the MERGE isn't idiomatic in the way I describe, then the INSERT
may actually succeed, which also seems fine.

> Oracle behaves differently and it actually finds a new matching tuple from
> the source relation and executes the WHEN MATCHED action, using that source
> tuple. But I am seriously doubtful that we want to go down that path and
> whether it's even feasible.

I think that that's just a consequence of Oracle using statement level
rollback to do RC conflict handling. It's the same with an UPDATE ...
FROM, or any other type of UPDATE.

> Our regular UPDATE .. FROM does not do that
> either. Given that, it seems better to just throw an error (even when no NOT
> MATCHED action exists) and explain to the users that MERGE will work as long
> as concurrent updates don't modify the columns used in the join condition.
> Concurrent deletes should be fine and we may actually even invoke WHEN NOT
> MATCHED action in that case.

Again, I have to ask: is such an UPDATE actually meaningfully
different from a concurrent DELETE + INSERT? If so, why is a special
error better than a dup violation, or maybe even having the INSERT
(and whole MERGE statement) succeed?

-- 
Peter Geoghegan

  168. Re: [HACKERS] MERGE SQL Statement for PG11

    Tomas Vondra <tomas.vondra@2ndquadrant.com> — 2018-03-01T22:22:46Z

    On 02/06/2018 03:40 PM, Tomas Vondra wrote:
> 
> 
> I plan to go through the patch and this thread over the couple of
> days, and summarize what the current status is (or my understanding
> of it). That is (a) what are the missing pieces, (b) why are they
> missing, (c) how we plan to address them in the future and (d)
> opinions on these issues expressed by others on this thread.
> 

So, I've promised a summary of the patch status about three weeks ago.
I've been postponing that as Pavan was moving the patch forward pretty
quickly, and now there's not much to summarize ... which is a good thing
I guess.

If my understanding is correct, the MERGE now supports both partitioning
and RLS, and subselects should also work on various places (which is
somewhat consistent with my impression claims about SQL standard
prohibiting them were not entirely accurate).

Which leaves us with figuring out the right concurrency behavior, and
that discussion seems to be in progress.

So at this point I'm not aware of any other missing features in the
patch, and a more detailed summary is not really needed.


regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  169. Re: [HACKERS] MERGE SQL Statement for PG11

    Pavan Deolasee <pavan.deolasee@gmail.com> — 2018-03-05T11:02:42Z

    On Fri, Mar 2, 2018 at 12:36 AM, Peter Geoghegan <pg@bowt.ie> wrote:

>
> > Our regular UPDATE .. FROM does not do that
> > either. Given that, it seems better to just throw an error (even when no
> NOT
> > MATCHED action exists) and explain to the users that MERGE will work as
> long
> > as concurrent updates don't modify the columns used in the join
> condition.
> > Concurrent deletes should be fine and we may actually even invoke WHEN
> NOT
> > MATCHED action in that case.
>
> Again, I have to ask: is such an UPDATE actually meaningfully
> different from a concurrent DELETE + INSERT? If so, why is a special
> error better than a dup violation, or maybe even having the INSERT
> (and whole MERGE statement) succeed?
>
>
Ok, I agree. I have updated the patch to remove the serialization error. If
MATCHED changes to NOT MATCHED because of concurrent update/delete, we now
simply retry from the top and execute the first NOT MATCHED action, if WHEN
AND qual passes on the updated version. Of course, if the MERGE does not
contain any NOT MATCHED action then we simply ignore the target row and
move to the next row. Since a NOT MATCHED case can never turn into a
MATCHED case, there is no risk of a live lock.

I've updated the documentation and the test cases to reflect this change.

Thanks,
Pavan

-- 
 Pavan Deolasee                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

  170. Re: [HACKERS] MERGE SQL Statement for PG11

    Peter Geoghegan <pg@bowt.ie> — 2018-03-06T04:25:27Z

    On Mon, Mar 5, 2018 at 3:02 AM, Pavan Deolasee <pavan.deolasee@gmail.com> wrote:
>> Again, I have to ask: is such an UPDATE actually meaningfully
>> different from a concurrent DELETE + INSERT? If so, why is a special
>> error better than a dup violation, or maybe even having the INSERT
>> (and whole MERGE statement) succeed?
>>
>
> Ok, I agree. I have updated the patch to remove the serialization error.

Cool. This is really shaping up. Thank you for working so hard to
address my concerns.

> If MATCHED changes to NOT MATCHED because of concurrent update/delete, we now
> simply retry from the top and execute the first NOT MATCHED action, if WHEN
> AND qual passes on the updated version. Of course, if the MERGE does not
> contain any NOT MATCHED action then we simply ignore the target row and move
> to the next row. Since a NOT MATCHED case can never turn into a MATCHED
> case, there is no risk of a live lock.

Makes sense. We either follow an UPDATE chain, which must always make
forward progress, or do NOT MATCHED handling/insert a row, which only
happens when NOT MATCHED handling has been abandoned, and so similarly
must make forward progress. I think that this needs to be documented
in a central place, though. I'd like to see arguments for why the
design is livelock safe, as well as an explanation for how EPQ
handling works, what the goals of how it works are, and so on. I would
perhaps summarize it by saying something like the following within
ExecMerge():

"""
ExecMerge() EPQ handling repeatedly rechecks all WHEN MATCHED actions
for each new candidate tuple as an update chain is followed, either
until a tuple is actually updated/deleted, or until we decide that the
new join input row actually requires WHEN NOT MATCHED handling after
all. Rechecking join quals for a single candidate tuple is performed
by ExecUpdate() and ExecDelete(), which can specially instruct us to
advance to the next tuple in the update chain so that we can recheck
our own WHEN ... AND quals (when the join quals no longer pass due to
a concurrent UPDATE), or to switch to the WHEN NOT MATCHED case (when
join quals no longer pass due to a concurrent DELETE).

EPQ only ever installs a new tuple for the target relation (never the
source), so changing from one WHEN MATCHED action to another during
READ COMMITTED conflict handling must be due to a concurrent UPDATE
that changes WHEN ... AND qual referenced attribute(s). If it was due
to a concurrent DELETE, or, equivalently, some concurrent UPDATE that
affects the target's primary key attribute(s) (or whatever the outer
join's target qual attributes are), then we switch to WHEN NOT MATCHED
handling, which will never switch back to WHEN MATCHED. ExecMerge()
avoids livelocks by always either walking the UPDATE chain, which
makes forward progress, or by finally switching to WHEN NOT MATCHED
processing.

"""

ExecUpdate()/ExecDelete() are not "driving" EPQ here, which is new --
they're doing one small part of it (the join quals for one tuple) on
behalf of ExecMerge(). I don't expect you to just take my wording
without editing or moving parts around a bit; I think that you'll get
the general idea from what I've written, though.

Actually, my wording may need to be changed to reflect a new code
structure for EPQ. The current control flow makes the reader think
about UPDATE quals, as well as DELETE quals. Instead, the reader
should think about WHEN ... AND quals, as well as MERGE's outer JOIN
quals (the join quals are the same, regardless of whether an UPDATE or
DELETE happens to be involved). The control flow between ExecMerge(),
ExecUpdate(), and ExecDelete() is just confusing.

Sure, ExecUpdate() and ExecDelete() *do* require a little special
handling for MERGE, but ExecMerge() should itself call EvalPlanQual()
for the join quals, rather than getting ExecUpdate()/ExecDelete() to
do it. All that ExecUpdate()/ExecDelete() need to tell ExecMerge() is
"you need to do your HeapTupleUpdated stuff". This not only clarifies
the new EPQ design -- it also lets you do that special case rti EPQ
stuff in the right place (BTW, I'm not a huge fan of that detail in
general, but haven't thought about it enough to say more). The new EPQ
code within ExecUpdate() and ExecDelete() is already identical, so why
not do it that way?

I especially like this organization because ExecOnConflictUpdate()
already calls ExecUpdate() without expecting it to do EPQ handling at
all, which is kind of similar to what I suggest -- in both cases, the
caller "takes care of all of the details of the scan". And, by
returning a HTSU_Result value to ExecMerge() from
ExecUpdate()/ExecDelete(), you could also do the new cardinality
violation stuff from within ExecMerge() in exactly one place -- no
need to invent new error_on_SelfUpdate arguments.

I would also think about organizing ExecMerge() handling so that
CMD_INSERT handling became WHEN NOT MATCHED handling. It's
unnecessarily confusing to have that included in the same switch
statement as CMD_UPDATE and CMD_DELETE, since that organization fails
to convey that once we reach WHEN NOT MATCHED, there is no going back.
Actually, I suppose the new EPQ organization described in the last few
paragraphs already implies that you'd do this CMD_INSERT stuff, since
the high level goal is breaking ExecMerge() into WHEN MATCHED and WHEN
NOT MATCHED sections (doing "commonality and variability analysis" for
CMD_UPDATE and CMD_DELETE only serves that high level goal).

Other feedback:

* Is a new ExecMaterializeSlot() call needed for the EPQ slot? Again,
doing your own EvalPlanQual() within ExecMerge() would perhaps have
avoided this.

* Do we really need UpdateStmt raw parser node pointers in structs
that appear in execnodes.h? What's that all about?

* Tests for transition table behavior (mixed INSERTs, UPDATEs, and
DELETEs) within triggers.sql seems like a good idea.

* Is this comment really accurate?:

> +               /*
> +                * If EvalPlanQual did not return a tuple, it means we
> +                * have seen a concurrent delete, or a concurrent update
> +                * where the row has moved to another partition.
> +                *
> +                * Set matched = false and loop back if there exists a NOT
> +                * MATCHED action. Otherwise, we have nothing to do for this
> +                * tuple and we can continue to the next tuple.
> +                */

Won't we loop back when a concurrent update occurs that makes the new
candidate tuple not satisfy the join qual anymore? What does this have
to do with partitioning?

* Why does MergeJoin get referenced here?:

> + * Also, since the internal MergeJoin node can hide the source and target
> + * relations, we must explicitly make the respective relation as visible so
> + * that columns can be referenced unqualified from these relations.

That's a struct that has nothing to do with SQL MERGE in particular
(no more than hash join does, for example).

* This patch could use a pg_indent.

* We already heard about this code from Tomas, who found it unnecessary:

> +           /*
> +            * SQL Standard says that WHEN AND conditions must not
> +            * write to the database, so check we haven't written
> +            * any WAL during the test. Very sensible that is, since
> +            * we can end up evaluating some tests multiple times if
> +            * we have concurrent activity and complex WHEN clauses.
> +            *
> +            * XXX If we had some clear form of functional labelling
> +            * we could use that, if we trusted it.
> +            */
> +           if (startWAL < GetXactWALBytes())
> +               ereport(ERROR,
> +                       (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
> +                        errmsg("cannot write to database within WHEN AND condition")));

This needs to go. Apart from the fact that GetXactWALBytes() is buggy
(it returns int64 for the unsigned type XLogRecPtr), the whole idea
just seems unnecessary. I don't see why this is any different to using
a volatile function in a regular UPDATE.

* I still think we probably need to add something to ruleutils.c, so
that MERGE Query structs can be deparsed -- see get_query_def().

Yeah, we've decided we're not going to support user-visible rules, but
I continue to think that deparsing support is necessary on general
principle, and for the benefit of extensions like Citus that use
deparsing in a fairly broad way. At the very least, if we're not going
to support deparsing, there needs to be a better reason than "we're
not supporting user-visible rules".

-- 
Peter Geoghegan

  171. Re: [HACKERS] MERGE SQL Statement for PG11

    Pavan Deolasee <pavan.deolasee@gmail.com> — 2018-03-08T12:54:25Z

    On Tue, Mar 6, 2018 at 9:55 AM, Peter Geoghegan <pg@bowt.ie> wrote:

>
>
> Sure, ExecUpdate() and ExecDelete() *do* require a little special
> handling for MERGE, but ExecMerge() should itself call EvalPlanQual()
> for the join quals, rather than getting ExecUpdate()/ExecDelete() to
> do it. All that ExecUpdate()/ExecDelete() need to tell ExecMerge() is
> "you need to do your HeapTupleUpdated stuff". This not only clarifies
> the new EPQ design -- it also lets you do that special case rti EPQ
> stuff in the right place (BTW, I'm not a huge fan of that detail in
> general, but haven't thought about it enough to say more). The new EPQ
> code within ExecUpdate() and ExecDelete() is already identical, so why
> not do it that way?
>
> I especially like this organization because ExecOnConflictUpdate()
> already calls ExecUpdate() without expecting it to do EPQ handling at
> all, which is kind of similar to what I suggest -- in both cases, the
> caller "takes care of all of the details of the scan". And, by
> returning a HTSU_Result value to ExecMerge() from
> ExecUpdate()/ExecDelete(), you could also do the new cardinality
> violation stuff from within ExecMerge() in exactly one place -- no
> need to invent new error_on_SelfUpdate arguments.
>
> I would also think about organizing ExecMerge() handling so that
> CMD_INSERT handling became WHEN NOT MATCHED handling. It's
> unnecessarily confusing to have that included in the same switch
> statement as CMD_UPDATE and CMD_DELETE, since that organization fails
> to convey that once we reach WHEN NOT MATCHED, there is no going back.
> Actually, I suppose the new EPQ organization described in the last few
> paragraphs already implies that you'd do this CMD_INSERT stuff, since
> the high level goal is breaking ExecMerge() into WHEN MATCHED and WHEN
> NOT MATCHED sections (doing "commonality and variability analysis" for
> CMD_UPDATE and CMD_DELETE only serves that high level goal).
>
>
Thanks for the feedback. I've greatly refactored the code based on your
comments and I too like the resultant code. What we have now have
essentially is: two functions:

ExecMergeMatched() - deals with matched rows. In case of concurrent
update/delete, it also runs EvalPlanQual and checks if the updated row
still meets the join quals. If so, it will restart and recheck if some
other WHEN MATCHED action should be executed. If the target row is deleted
or if the join quals fail, it simply returns and let the next function
handle it

ExecMargeNotMatched() -  deals with not-matched rows. Essentially it only
needs to execute INSERT if a WHEN NOT MATCHED action exists and the
additional quals pass.

Per your suggestion, ExecUpdate() and ExecDelete() only returns enough
information for ExecMergeMatched() to run EvalPlanQual or take any other
action, as needed. So changes to these functions are now minimal. Also, the
EPQ handling for UPDATE/DELETE is now common, which is nice.

I've also added a bunch of comments, but it might still not be enough. Feel
free to suggest improvements there.



> Other feedback:
>
> * Is a new ExecMaterializeSlot() call needed for the EPQ slot? Again,
> doing your own EvalPlanQual() within ExecMerge() would perhaps have
> avoided this.
>

Fixed.


>
> * Do we really need UpdateStmt raw parser node pointers in structs
> that appear in execnodes.h? What's that all about?
>

No, it was left-over from the earlier code. Removed.


>
> * Tests for transition table behavior (mixed INSERTs, UPDATEs, and
> DELETEs) within triggers.sql seems like a good idea.
>

Ok, I will add. But not done in this version.


>
> * Is this comment really accurate?:
>
> > +               /*
> > +                * If EvalPlanQual did not return a tuple, it means we
> > +                * have seen a concurrent delete, or a concurrent update
> > +                * where the row has moved to another partition.
> > +                *
> > +                * Set matched = false and loop back if there exists a
> NOT
> > +                * MATCHED action. Otherwise, we have nothing to do for
> this
> > +                * tuple and we can continue to the next tuple.
> > +                */
>
> Won't we loop back when a concurrent update occurs that makes the new
> candidate tuple not satisfy the join qual anymore? What does this have
> to do with partitioning?
>

Yeah, it was wrong. Fixed as part of larger reorganisation anyways.


>
> * Why does MergeJoin get referenced here?:
>
> > + * Also, since the internal MergeJoin node can hide the source and
> target
> > + * relations, we must explicitly make the respective relation as
> visible so
> > + * that columns can be referenced unqualified from these relations.
>

I meant to say underlying JOIN for MERGE. Fixed by simply saying Join.


>
> * This patch could use a pg_indent.
>
>
Done.


> * We already heard about this code from Tomas, who found it unnecessary:
>
> > +           /*
> > +            * SQL Standard says that WHEN AND conditions must not
> > +            * write to the database, so check we haven't written
> > +            * any WAL during the test. Very sensible that is, since
> > +            * we can end up evaluating some tests multiple times if
> > +            * we have concurrent activity and complex WHEN clauses.
> > +            *
> > +            * XXX If we had some clear form of functional labelling
> > +            * we could use that, if we trusted it.
> > +            */
> > +           if (startWAL < GetXactWALBytes())
> > +               ereport(ERROR,
> > +                       (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
> > +                        errmsg("cannot write to database within WHEN
> AND condition")));
>
> This needs to go. Apart from the fact that GetXactWALBytes() is buggy
> (it returns int64 for the unsigned type XLogRecPtr), the whole idea
> just seems unnecessary. I don't see why this is any different to using
> a volatile function in a regular UPDATE.
>

I removed this code since it was wrong. We might want to add some basic
checks for existence of volatile functions in the WHEN or SET clauses. But
I agree, it's no different than regular UPDATEs. So may be not a big deal.


>
> * I still think we probably need to add something to ruleutils.c, so
> that MERGE Query structs can be deparsed -- see get_query_def().
>

Ok. I will look at it. Not done in this version though.

Rebased on the current master too.

Thanks,
Pavan

-- 
 Pavan Deolasee                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

  172. Re: [HACKERS] MERGE SQL Statement for PG11

    Robert Haas <robertmhaas@gmail.com> — 2018-03-08T14:52:19Z

    On Thu, Mar 8, 2018 at 7:54 AM, Pavan Deolasee <pavan.deolasee@gmail.com> wrote:
>> > +           /*
>> > +            * SQL Standard says that WHEN AND conditions must not
>> > +            * write to the database, so check we haven't written
>> > +            * any WAL during the test. Very sensible that is, since
>> > +            * we can end up evaluating some tests multiple times if
>> > +            * we have concurrent activity and complex WHEN clauses.
>> > +            *
>> > +            * XXX If we had some clear form of functional labelling
>> > +            * we could use that, if we trusted it.
>> > +            */
>> > +           if (startWAL < GetXactWALBytes())
>> > +               ereport(ERROR,
>> > +                       (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
>> > +                        errmsg("cannot write to database within WHEN
>> > AND condition")));
>>
>> This needs to go. Apart from the fact that GetXactWALBytes() is buggy
>> (it returns int64 for the unsigned type XLogRecPtr), the whole idea
>> just seems unnecessary. I don't see why this is any different to using
>> a volatile function in a regular UPDATE.
>
> I removed this code since it was wrong. We might want to add some basic
> checks for existence of volatile functions in the WHEN or SET clauses. But I
> agree, it's no different than regular UPDATEs. So may be not a big deal.

I just caught up on this thread.  I'm definitely glad to see that code
go because, wow, that is all kinds of wrong.  I don't see a real need
to add any kind of replacement check, either.  Prohibiting volatile
functions here doesn't seem likely to accomplish anything useful.  It
seems like the most we'd want to do is mention this the documentation
somehow, and I'm not even sure we really need to do that much.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  173. Re: [HACKERS] MERGE SQL Statement for PG11

    Peter Geoghegan <pg@bowt.ie> — 2018-03-08T22:46:13Z

    On Thu, Mar 8, 2018 at 6:52 AM, Robert Haas <robertmhaas@gmail.com> wrote:
>> I removed this code since it was wrong. We might want to add some basic
>> checks for existence of volatile functions in the WHEN or SET clauses. But I
>> agree, it's no different than regular UPDATEs. So may be not a big deal.
>
> I just caught up on this thread.  I'm definitely glad to see that code
> go because, wow, that is all kinds of wrong.  I don't see a real need
> to add any kind of replacement check, either.  Prohibiting volatile
> functions here doesn't seem likely to accomplish anything useful.  It
> seems like the most we'd want to do is mention this the documentation
> somehow, and I'm not even sure we really need to do that much.

Thanks in large part to Pavan's excellent work, the situation in
nodeModifyTable.c is much clearer than it was a few weeks ago. It's
now obvious that MERGE is very similar to UPDATE ... FROM, which
doesn't have any restrictions on volatile functions.

I don't see any sense in prohibiting volatile functions in either
case, because it should be obvious to users that that's just asking
for trouble. I can believe that someone would make that mistake, just
about, but they'd have to be writing their DML statement on
auto-pilot.

-- 
Peter Geoghegan

  174. Re: [HACKERS] MERGE SQL Statement for PG11

    Tomas Vondra <tomas.vondra@2ndquadrant.com> — 2018-03-08T23:29:35Z

    
On 03/08/2018 11:46 PM, Peter Geoghegan wrote:
> On Thu, Mar 8, 2018 at 6:52 AM, Robert Haas <robertmhaas@gmail.com> wrote:
>>> I removed this code since it was wrong. We might want to add some basic
>>> checks for existence of volatile functions in the WHEN or SET clauses. But I
>>> agree, it's no different than regular UPDATEs. So may be not a big deal.
>>
>> I just caught up on this thread.  I'm definitely glad to see that code
>> go because, wow, that is all kinds of wrong.  I don't see a real need
>> to add any kind of replacement check, either.  Prohibiting volatile
>> functions here doesn't seem likely to accomplish anything useful.  It
>> seems like the most we'd want to do is mention this the documentation
>> somehow, and I'm not even sure we really need to do that much.
> 
> Thanks in large part to Pavan's excellent work, the situation in
> nodeModifyTable.c is much clearer than it was a few weeks ago. It's
> now obvious that MERGE is very similar to UPDATE ... FROM, which
> doesn't have any restrictions on volatile functions.
> 

Yeah, I agree Pavan did an excellent work on moving this patch forward.

> I don't see any sense in prohibiting volatile functions in either
> case, because it should be obvious to users that that's just asking
> for trouble. I can believe that someone would make that mistake, just
> about, but they'd have to be writing their DML statement on
> auto-pilot.
> 

The reason why the patch tried to prevent that is because the SQL
standard says this (p. 1176 of SQL 2016):

15) The <search condition> immediately contained in a <merge statement>,
the <search condition> immediately contained in a <merge when matched
clause>, and the <search condition> immediately contained in a <merge
when not matched clause> shall not generally contain a <routine
invocation> whose subject routine is an SQL-invoked routine that
possibly modifies SQL-data.

I'm not quite sure what is required to be compliant with this rule. For
example what does "immediately contained" or "shall not generally
contain" mean? Does that mean user are expected not to do that because
it's obviously silly, or do we need to implement some protection?

That being said the volatility check seems reasonable to me (and i would
not expect it to be a huge amount of code).

regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  175. Re: [HACKERS] MERGE SQL Statement for PG11

    Peter Geoghegan <pg@bowt.ie> — 2018-03-09T00:43:38Z

    On Thu, Mar 8, 2018 at 3:29 PM, Tomas Vondra
<tomas.vondra@2ndquadrant.com> wrote:
> The reason why the patch tried to prevent that is because the SQL
> standard says this (p. 1176 of SQL 2016):
>
> 15) The <search condition> immediately contained in a <merge statement>,
> the <search condition> immediately contained in a <merge when matched
> clause>, and the <search condition> immediately contained in a <merge
> when not matched clause> shall not generally contain a <routine
> invocation> whose subject routine is an SQL-invoked routine that
> possibly modifies SQL-data.
>
> I'm not quite sure what is required to be compliant with this rule. For
> example what does "immediately contained" or "shall not generally
> contain" mean? Does that mean user are expected not to do that because
> it's obviously silly, or do we need to implement some protection?

My impression is that this means that you shouldn't treat this as a
particularly likely case, or try to facilitate it. The <search
condition> blurb is about intent, rather than actual restrictions
implementations must enforce, AFAICT. Though the UPDATE precedent is
what really matters here -- not the SQL standard. The SQL standard
doesn't say anything to make me doubt that that's the right precedent
to want to follow.

Close by, under "General Rules", rule #4 is: "The extent to which an
SQL-implementation may disallow independent changes that are not
significant is implementation-defined". This same sentence appears in
quite a few different places, including in the description of UPDATE.
ISTM that the SQL standard actually enforces that volatile qual
weirdness (and what to do about it) is a general
INSERT/UPDATE/DELETE/MERGE issue.

> That being said the volatility check seems reasonable to me (and i would
> not expect it to be a huge amount of code).

If we're going to do this, we'd have to do the same with UPDATE, IMV.
And, well, we're not gonna do that.

-- 
Peter Geoghegan

  176. Re: [HACKERS] MERGE SQL Statement for PG11

    Peter Geoghegan <pg@bowt.ie> — 2018-03-09T02:59:30Z

    On Thu, Mar 8, 2018 at 4:54 AM, Pavan Deolasee <pavan.deolasee@gmail.com> wrote:
> Thanks for the feedback. I've greatly refactored the code based on your
> comments and I too like the resultant code. What we have now have
> essentially is: two functions:

I really like how ExecMerge() now mostly just consists of this code
(plus a lot of explanatory comments):

> +   if (!matched ||
> +       !ExecMergeMatched(mtstate, estate, slot, junkfilter, tupleid))
> +       ExecMergeNotMatched(mtstate, estate, slot);

This is far easier to follow. In general, I'm now a lot less worried
about what was previously the #1 concern -- READ COMMITTED conflict
handling (EvalPlanQual() stuff). My #1 concern has become RLS, and
perhaps only because I haven't studied it in enough detail. It seems
like getting this patch committable is now a matter of verifying
details. Of course, there are a lot of details to verify.  :-)

> I've also added a bunch of comments, but it might still not be enough. Feel
> free to suggest improvements there.

I think that the ExecMerge() comments are very good, although they
could perhaps use some light copy-editing. Also, I noticed some
specific issues with comments in ExecMerge() and friends, as well as a
few code issues. Those are:

* MERGE code needs to do cardinality violations like ON CONFLICT.
Specifically, you need the TransactionIdIsCurrentTransactionId()
check, and a second error fallback "attempted to lock invisible tuple"
error (though this should say "attempted to update or delete invisible
tuple" instead). The extra check is redundant, but better safe than
sorry. I like defensive errors like this because it makes the code
easier to read -- I don't get distracted by their absence.

* The last item on cardinality violations also implies this new merge
comment should be changed:

> +                   /*
> +                    * The target tuple was already updated or deleted by the
> +                    * current command, or by a later command in the current
> +                    * transaction.
> +                    */

It should be changed because it can't have been a later command in
this xact. We handled that case when we called ExecUpdate or
ExecDelete() (plus we now have the extra defensive "can't happen"
elog() error).

* This related comment shouldn't even talk about update/historic
behavior, now that it isn't in ExecUpdate() -- just MERGE:

> +                   /*
> +                    * The former case is possible in a join UPDATE where
> +                    * multiple tuples join to the same target tuple. This is
> +                    * pretty questionable, but Postgres has always allowed
> +                    * it: we just execute the first update action and ignore
> +                    * additional update attempts.  SQLStandard disallows this
> +                    * for MERGE, so allow the caller to select how to handle
> +                    * this.
> +                    */

* This wording should be tweaked:

> +                    * If the current tuple is that last tuple in the update
> +                    * chain, then we know that the tuple was concurrently
> +                    * deleted. We just switch to NOT MATCHED case and let the
> +                    * caller retry the NOT MATCHED actions.

This should say something like "caller can move on to NOT MATCHED
actions". They can never go back from there, of course, which I want
us to be clear on.

* This check of whether whenqual is set is unnecessary, and doesn't
match MATCHED code, or the accompanying comments:

> +       /*
> +        * Test condition, if any
> +        *
> +        * In the absence of a condition we perform the action unconditionally
> +        * (no need to check separately since ExecQual() will return true if
> +        * there are no conditions to evaluate).
> +        */
> +       if (action->whenqual && !ExecQual(action->whenqual, econtext))
> +           continue;

* I think that this ExecMerge() assertion is not helpful, since you go
on to dereference the pointer in all cases anyway:

> +   Assert(junkfilter != NULL);

* Executor README changes, particularly about projecting twice, really
should be ExecMerge() comments. Maybe just get rid of these?

* Why are we using CMD_NOTHING at all? That constant has something to
do with user-visible rules, and there is no need to reuse it (make a
new CMD_* if you have to). More importantly, why do we even have the
corresponding DO NOTHING stuff in the grammar? Why would users want
that?

For quite a while, I thought that patch must have been support for ON
CONFLICT DO NOTHING within MERGE INSERTs (the docs don't even say what
DO NOTHING is). But that's not what it is at all. It seems like this
is a way of having an action that terminates early, so you don't have
to go on to evaluate other action quals. I can't see much point,
though. More importantly, supporting this necessitates code like the
following RLS code within ExecMergeMatched():

> +       if ((action->commandType == CMD_UPDATE ||
> +            action->commandType == CMD_DELETE) &&
> +           resultRelInfo->ri_WithCheckOptions)
> +       {
> +           ExecWithCheckOptions(action->commandType == CMD_UPDATE ?
> +                                WCO_RLS_MERGE_UPDATE_CHECK : WCO_RLS_MERGE_DELETE_CHECK,
> +                                resultRelInfo,
> +                                mtstate->mt_merge_existing[ud_target],
> +                                mtstate->ps.state);
> +       }

I wonder if the CMD_NOTHING case makes this code prone to information
leak attacks. Even if it that isn't the case, ISTM that several code
blocks within ExecMergeMatched() could do without the repeated
"action->commandType" tests. Have I missed something that makes
supporting DO NOTHING seem more compelling?

* The docs say that we use a LEFT OUTER JOIN for MERGE, while the
implementation uses a RIGHT OUTER JOIN because it's convenient for
parse analysis finding the target. This difference is a bit confusing.
Why say that we use any kind of join at all, though?

The SQL standard doesn't say outer join at all. Neither do the MERGE
docs of the other database systems that I checked. Taking a position
on this seems to add nothing at all. Let's make the MERGE docs refer
to it as a join, without further elaboration.

* I don't find this comment from analyze.c very helpful:

> +        * We don't have a separate plan for each action, so the when
> +        * condition must be executed as a per-row check, making it very
> +        * similar to a CHECK constraint and so we adopt the same semantics
> +        * for that.

Why explain it that way at all? There are two rels, unlike a check constraint.

* The first time I read this comment, it made me laugh:

> +       /*
> +        * First rule of MERGE club is we don't talk about rules
> +        */

The joke has become significantly less funny since then, though. I'd
just say that MERGE doesn't support rules, as it's unclear how that
could work.

* This comment seems redundant, since pstate is always allocated with palloc0():

+ * Note: we assume that the pstate's p_rtable, p_joinlist, and p_namespace
+ * lists were initialized to NIL when the pstate was created.

make_parsestate() knows about this. This is widespread, normal
practice during parse analysis.

* Is this actually needed at all?:

> +       /* In MERGE when and condition, no system column is allowed */
> +       if (pstate->p_expr_kind == EXPR_KIND_MERGE_WHEN_AND &&
> +           attnum < InvalidAttrNumber &&
> +           !(attnum == TableOidAttributeNumber || attnum == ObjectIdAttributeNumber))
> +           ereport(ERROR,
> +                   (errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
> +                    errmsg("system column \"%s\" reference in WHEN AND condition is invalid",
> +                           colname),
> +                    parser_errposition(pstate, location)));

We're talking about the scantuple here. It's not like excluded.*.

* Are these checks really needed?:

> +void
> +rewriteTargetListMerge(Query *parsetree, Relation target_relation)
> +{
> +   Var        *var = NULL;
> +   const char *attrname;
> +   TargetEntry *tle;
> +
> +   if (target_relation->rd_rel->relkind == RELKIND_RELATION ||
> +       target_relation->rd_rel->relkind == RELKIND_MATVIEW ||
> +       target_relation->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)

* I think that you should remove this debug include:

> index 3a02307bd9..ed4e857477 100644
> --- a/src/backend/parser/parse_clause.c
> +++ b/src/backend/parser/parse_clause.c
> @@ -31,6 +31,7 @@
>  #include "commands/defrem.h"
>  #include "nodes/makefuncs.h"
>  #include "nodes/nodeFuncs.h"
> +#include "nodes/print.h"
>  #include "optimizer/tlist.h"
>  #include "optimizer/var.h"
>  #include "parser/analyze.h"

It may be worth considering custom dprintf code within your .gdbinit
to do stuff like this automatically, without code changes that may end
up in the patch you post to the list. Here is one that I sometimes use
for tuplesort.c:

define print_tuplesort_counts
  dprintf tuplesort_sort_memtuples, "memtupsize: %d, memtupcount:
%d\n", state->memtupsize, state->memtupcount
end

This is way easier than adding custom printf style debug code, since
it doesn't require that you rebuild. It would be safe to add a similar
dprintf that called pprint() or similar when some function is reached.

* find_mergetarget_parents() and find_mergetarget_for_rel() could both
use at least a one-liner header comment.

* This analyze.c comment, which is in transformMergeStmt(), seems
pretty questionable:

> +   /*
> +    * Simplify the MERGE query as much as possible
> +    *
> +    * These seem like things that could go into Optimizer, but they are
> +    * semantic simplications rather than optimizations, per se.
> +    *
> +    * If there are no INSERT actions we won't be using the non-matching
> +    * candidate rows for anything, so no need for an outer join. We do still
> +    * need an inner join for UPDATE and DELETE actions.

This is talking about an ad-hoc form of join strength reduction. Yeah,
it's a semantic simplification, but that's generally true of join
strength reduction, which mostly exists because of bad ORMs that
sprinkle OUTER on top of queries indifferently. MERGE is hardly an
ideal candidate for join strength reduction, and I would just lose
this code entirely for Postgres v11.

* This sounds really brittle, and so doesn't make sense even as an aspiration:

> +    * XXX if we were really keen we could look through the actionList and
> +    * pull out common conditions, if there were no terminal clauses and put
> +    * them into the main query as an early row filter but that seems like an
> +    * atypical case and so checking for it would be likely to just be wasted
> +    * effort.
> +    */

Again, I suggest removing everything on join strength reduction.

* You should say *when* this happens (what later point):

> +    *
> +    * Track the RTE index of the target table used in the join query. This is
> +    * later used to add required junk attributes to the targetlist.
> +    */

* This seems unnecessary, as we don't say anything like it for ON
CONFLICT DO UPDATE:

> +    * As top-level statements INSERT, UPDATE and DELETE have a Query, whereas
> +    * with MERGE the individual actions do not require separate planning,
> +    * only different handling in the executor. See nodeModifyTable handling
> +    * of commandType CMD_MERGE.

* What does this mean?:

> +    * A sub-query can include the Target, but otherwise the sub-query cannot
> +    * reference the outermost Target table at all.

* I don't see the point in this:

> +        * XXX Perhaps we require Parallel Safety since that is a superset of
> +        * the restriction and enforcing that makes it easier to consider
> +        * running MERGE plans in parallel in future.

* This references the now-removed executor WAL check thing, so you'll
need to remove it too:

> +        * SQL Standard says we should not allow anything that possibly
> +        * modifies SQL-data. We enforce that with an executor check that we
> +        * have not written any WAL.

* This should be reworded:

> +        * Note that we don't add this to the MERGE Query's quals because
> +        * that's not the logic MERGE uses.
> +        */
> +       action->qual = transformWhereClause(pstate, action->condition,
> +                                           EXPR_KIND_MERGE_WHEN_AND, "WHEN");

Perhaps say "WHEN ... AND quals are evaluated separately from the
MERGE statement's join quals" instead. Or just lose it altogether.

* This comment seems inaccurate:

> +                       /*
> +                        * Process INSERT ... VALUES with a single VALUES
> +                        * sublist.  We treat this case separately for
> +                        * efficiency.  The sublist is just computed directly
> +                        * as the Query's targetlist, with no VALUES RTE.  So
> +                        * it works just like a SELECT without any FROM.
> +                        */

Wouldn't it be more accurate to say that this it totally different to
what transformInsertStmt() does for a SELECT without any FROM? Also,
do you think that that's good?

>> * Tests for transition table behavior (mixed INSERTs, UPDATEs, and
>> DELETEs) within triggers.sql seems like a good idea.
>
> Ok, I will add. But not done in this version.

Note that it's implied in at least one place that we don't support
transition tables at all:

> +               /*
> +                * XXX if we support transition tables this would need to move
> +                * earlier before ExecSetupTransitionCaptureState()
> +                */
> +               switch (action->commandType)
> +               {

You'll want to get to this as part of that transition table effort.

Any plan to fix this/support identity columns? I see that you don't
support them here:

> +                   /*
> +                    * Handle INSERT much like in transformInsertStmt
> +                    *
> +                    * XXX currently ignore stmt->override, if present
> +                    */

I think that this is a blocker, unfortunately.

>> * I still think we probably need to add something to ruleutils.c, so
>> that MERGE Query structs can be deparsed -- see get_query_def().
>
> Ok. I will look at it. Not done in this version though.

I also wonder what it would take to support referencing CTEs. Other
implementations do have this. Why can't we?

I do accept that RETURNING support and user-defined rule support are
not desirable, because it isn't at all clear what that means, and
there is no precedent for those from other database systems. CTEs and
identity columns are not in that same category, though.

Phew! Thanks for your patience and perseverance. I do have more
feedback on the docs lined up, but that isn't so important right now.

-- 
Peter Geoghegan

  177. Re: [HACKERS] MERGE SQL Statement for PG11

    Pavan Deolasee <pavan.deolasee@gmail.com> — 2018-03-09T14:55:35Z

    On 9 March 2018 at 08:29, Peter Geoghegan <pg@bowt.ie> wrote:

> On Thu, Mar 8, 2018 at 4:54 AM, Pavan Deolasee <pavan.deolasee@gmail.com>
> wrote:
> > Thanks for the feedback. I've greatly refactored the code based on your
> > comments and I too like the resultant code. What we have now have
> > essentially is: two functions:
>
> I really like how ExecMerge() now mostly just consists of this code
> (plus a lot of explanatory comments):
>

Thanks!


>  My #1 concern has become RLS, and
> perhaps only because I haven't studied it in enough detail.


Sure. I've done what I thought is the right thing to do, but please check.
Stephen also wanted to review RLS related code; I don't know if he had
chance to do so.


> It seems
> like getting this patch committable is now a matter of verifying
> details. Of course, there are a lot of details to verify.  :-)
>

Let's work through them together and hopefully we shall get there.


>
>
>
> * MERGE code needs to do cardinality violations like ON CONFLICT.
> Specifically, you need the TransactionIdIsCurrentTransactionId()
> check, and a second error fallback "attempted to lock invisible tuple"
> error (though this should say "attempted to update or delete invisible
> tuple" instead). The extra check is redundant, but better safe than
> sorry. I like defensive errors like this because it makes the code
> easier to read -- I don't get distracted by their absence.
>

Ok. Fixed. I also added the missing case for HeapTupleInvisible, though I
don't see how we can reach there. Since ExecUpdate()/ExecDelete() always
wait for any concurrent transaction to finish, I don't see how we can
reach HeapTupleBeingUpdated. So I skipped that test (or may be we should
just add an error for completeness).


>
> * The last item on cardinality violations also implies this new merge
> comment should be changed:
>
> > +                   /*
> > +                    * The target tuple was already updated or deleted
> by the
> > +                    * current command, or by a later command in the
> current
> > +                    * transaction.
> > +                    */
>
> It should be changed because it can't have been a later command in
> this xact. We handled that case when we called ExecUpdate or
> ExecDelete() (plus we now have the extra defensive "can't happen"
> elog() error).
>

Removed.


>
> * This related comment shouldn't even talk about update/historic
> behavior, now that it isn't in ExecUpdate() -- just MERGE:
>
> > +                   /*
> > +                    * The former case is possible in a join UPDATE where
> > +                    * multiple tuples join to the same target tuple.
> This is
> > +                    * pretty questionable, but Postgres has always
> allowed
> > +                    * it: we just execute the first update action and
> ignore
> > +                    * additional update attempts.  SQLStandard
> disallows this
> > +                    * for MERGE, so allow the caller to select how to
> handle
> > +                    * this.
> > +                    */
>

Removed.


>
> * This wording should be tweaked:
>
> > +                    * If the current tuple is that last tuple in the
> update
> > +                    * chain, then we know that the tuple was
> concurrently
> > +                    * deleted. We just switch to NOT MATCHED case and
> let the
> > +                    * caller retry the NOT MATCHED actions.
>
> This should say something like "caller can move on to NOT MATCHED
> actions". They can never go back from there, of course, which I want
> us to be clear on.
>

Fixed, but please check if the new wording is OK.


>
> * This check of whether whenqual is set is unnecessary, and doesn't
> match MATCHED code, or the accompanying comments:
>
> > +       /*
> > +        * Test condition, if any
> > +        *
> > +        * In the absence of a condition we perform the action
> unconditionally
> > +        * (no need to check separately since ExecQual() will return
> true if
> > +        * there are no conditions to evaluate).
> > +        */
> > +       if (action->whenqual && !ExecQual(action->whenqual, econtext))
> > +           continue;
>

Fixed.


>
> * I think that this ExecMerge() assertion is not helpful, since you go
> on to dereference the pointer in all cases anyway:
>
> > +   Assert(junkfilter != NULL);
>

Removed.


>
> * Executor README changes, particularly about projecting twice, really
> should be ExecMerge() comments. Maybe just get rid of these?
>

Fixed, with addition to TABLEOID column that we now fetch along with CTID
column.


>
> * Why are we using CMD_NOTHING at all? That constant has something to
> do with user-visible rules, and there is no need to reuse it (make a
> new CMD_* if you have to). More importantly, why do we even have the
> corresponding DO NOTHING stuff in the grammar? Why would users want
> that?
>
> For quite a while, I thought that patch must have been support for ON
> CONFLICT DO NOTHING within MERGE INSERTs (the docs don't even say what
> DO NOTHING is). But that's not what it is at all. It seems like this
> is a way of having an action that terminates early, so you don't have
> to go on to evaluate other action quals.


Hmm. I was under the impression that the SQL Standards support DO NOTHING.
But now that I read the standards, I can't find any mention of DO NOTHING.
It might still be useful for users to skip all (matched/not-matched/both)
actions for some specific conditions. For example,

WHEN MATCHED AND balance > min_balance THEN
   DO NOTHING
WHEN MATCHED AND custcat = 'priority' THEN
   DO NOTHING
....

But I agree that if we remove DO NOTHING, we can simplify the code. Or we
can probably re-arrange the code to check DO NOTHING early and exit the
loop. That simplifies it a lot, as in attached. In passing, I also realised
that the target tuple may also be needed for DO NOTHING actions since they
may have quals attached to them. Also, the tuple should really be fetched
just once, not once per action. Those changes are done.


>
> * The docs say that we use a LEFT OUTER JOIN for MERGE, while the
> implementation uses a RIGHT OUTER JOIN because it's convenient for
> parse analysis finding the target. This difference is a bit confusing.
> Why say that we use any kind of join at all, though?
>

Hmm, right. In fact, we revert to INNER JOIN when there are no NOT MATCHED
actions, so probably we should just not mention any kind of joins,
definitely not in the user documentation.


>
> The SQL standard doesn't say outer join at all. Neither do the MERGE
> docs of the other database systems that I checked. Taking a position
> on this seems to add nothing at all. Let's make the MERGE docs refer
> to it as a join, without further elaboration.
>
> * I don't find this comment from analyze.c very helpful:
>
> > +        * We don't have a separate plan for each action, so the when
> > +        * condition must be executed as a per-row check, making it very
> > +        * similar to a CHECK constraint and so we adopt the same
> semantics
> > +        * for that.
>
> Why explain it that way at all? There are two rels, unlike a check
> constraint.
>

Agreed. I think it was left-over from the time when sub-selects were not
allowed and we were using EXPR_KIND_CHECK_CONSTRAINT to run those checks.
Now we have a separate expression kind and we do support sub-selects.


>
> * The first time I read this comment, it made me laugh:
>
> > +       /*
> > +        * First rule of MERGE club is we don't talk about rules
> > +        */
>
> The joke has become significantly less funny since then, though. I'd
> just say that MERGE doesn't support rules, as it's unclear how that
> could work.
>

Changed that way.


>
> * This comment seems redundant, since pstate is always allocated with
> palloc0():
>
> + * Note: we assume that the pstate's p_rtable, p_joinlist, and p_namespace
> + * lists were initialized to NIL when the pstate was created.
>
> make_parsestate() knows about this. This is widespread, normal
> practice during parse analysis.
>

Ok. Removed.


>
> * Is this actually needed at all?:
>
> > +       /* In MERGE when and condition, no system column is allowed */
> > +       if (pstate->p_expr_kind == EXPR_KIND_MERGE_WHEN_AND &&
> > +           attnum < InvalidAttrNumber &&
> > +           !(attnum == TableOidAttributeNumber || attnum ==
> ObjectIdAttributeNumber))
> > +           ereport(ERROR,
> > +                   (errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
> > +                    errmsg("system column \"%s\" reference in WHEN AND
> condition is invalid",
> > +                           colname),
> > +                    parser_errposition(pstate, location)));
>
> We're talking about the scantuple here. It's not like excluded.*.
>

We might be able to support them, but do we really care?


>
> * Are these checks really needed?:
>
> > +void
> > +rewriteTargetListMerge(Query *parsetree, Relation target_relation)
> > +{
> > +   Var        *var = NULL;
> > +   const char *attrname;
> > +   TargetEntry *tle;
> > +
> > +   if (target_relation->rd_rel->relkind == RELKIND_RELATION ||
> > +       target_relation->rd_rel->relkind == RELKIND_MATVIEW ||
> > +       target_relation->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
>
>
You're right. Those checks are done at the beginning and we should probably
just turn this into an Assert, just like ExecMerge(). Changed that way.


> * I think that you should remove this debug include:
>
> > index 3a02307bd9..ed4e857477 100644
> > --- a/src/backend/parser/parse_clause.c
> > +++ b/src/backend/parser/parse_clause.c
> > @@ -31,6 +31,7 @@
> >  #include "commands/defrem.h"
> >  #include "nodes/makefuncs.h"
> >  #include "nodes/nodeFuncs.h"
> > +#include "nodes/print.h"
> >  #include "optimizer/tlist.h"
> >  #include "optimizer/var.h"
> >  #include "parser/analyze.h"
>

Done.


>
> It may be worth considering custom dprintf code within your .gdbinit
> to do stuff like this automatically, without code changes that may end
> up in the patch you post to the list. Here is one that I sometimes use
> for tuplesort.c:
>
> define print_tuplesort_counts
>   dprintf tuplesort_sort_memtuples, "memtupsize: %d, memtupcount:
> %d\n", state->memtupsize, state->memtupcount
> end
>
> This is way easier than adding custom printf style debug code, since
> it doesn't require that you rebuild. It would be safe to add a similar
> dprintf that called pprint() or similar when some function is reached.
>

Ok. Thanks for the hint.


>
> * find_mergetarget_parents() and find_mergetarget_for_rel() could both
> use at least a one-liner header comment.
>

Done. It was indeed confusing, so comments should help.


>
> * This analyze.c comment, which is in transformMergeStmt(), seems
> pretty questionable:
>
> > +   /*
> > +    * Simplify the MERGE query as much as possible
> > +    *
> > +    * These seem like things that could go into Optimizer, but they are
> > +    * semantic simplications rather than optimizations, per se.
> > +    *
> > +    * If there are no INSERT actions we won't be using the non-matching
> > +    * candidate rows for anything, so no need for an outer join. We do
> still
> > +    * need an inner join for UPDATE and DELETE actions.
>
> This is talking about an ad-hoc form of join strength reduction. Yeah,
> it's a semantic simplification, but that's generally true of join
> strength reduction, which mostly exists because of bad ORMs that
> sprinkle OUTER on top of queries indifferently. MERGE is hardly an
> ideal candidate for join strength reduction, and I would just lose
> this code entirely for Postgres v11.
>

I am not sure if there could be additional query optimisation possibilities
when a RIGHT OUTER JOIN is changed to INNER JOIN. Apart from existing
optimisations, I am also thinking about some of the work that David and
Amit are doing for partition pruning and I wonder if the choice of join
might have a non-trivial effect. Having said that, I am yet to explore
those things. But when we definitely know that a different kind of JOIN is
all we need (because there are no NOT MATCHED actions), why not use that?
Or do you see a problem there?


>
> * This sounds really brittle, and so doesn't make sense even as an
> aspiration:
>
> > +    * XXX if we were really keen we could look through the actionList
> and
> > +    * pull out common conditions, if there were no terminal clauses and
> put
> > +    * them into the main query as an early row filter but that seems
> like an
> > +    * atypical case and so checking for it would be likely to just be
> wasted
> > +    * effort.
> > +    */
>

We might actually want to do this, if not for v11 then later. I am not sure
if we should keep it in the final commit though.
 .

>
> * You should say *when* this happens (what later point):
>
> > +    *
> > +    * Track the RTE index of the target table used in the join query.
> This is
> > +    * later used to add required junk attributes to the targetlist.
> > +    */
>

Done.


>
> * This seems unnecessary, as we don't say anything like it for ON
> CONFLICT DO UPDATE:
>
> > +    * As top-level statements INSERT, UPDATE and DELETE have a Query,
> whereas
> > +    * with MERGE the individual actions do not require separate
> planning,
> > +    * only different handling in the executor. See nodeModifyTable
> handling
> > +    * of commandType CMD_MERGE.
>
>
I don't see anything wrong with keeping it. May be it needs rewording?


> * What does this mean?:
>
> > +    * A sub-query can include the Target, but otherwise the sub-query
> cannot
> > +    * reference the outermost Target table at all.
>

Don't know :-) I think what Simon probably meant to say that you can't
reference the Target table in the sub-query used in the source relation.
But you can add the Target as a separate RTE


>
> * I don't see the point in this:
>
> > +        * XXX Perhaps we require Parallel Safety since that is a
> superset of
> > +        * the restriction and enforcing that makes it easier to consider
> > +        * running MERGE plans in parallel in future.
>
>
Yeah, removed.


> * This references the now-removed executor WAL check thing, so you'll
> need to remove it too:
>
> > +        * SQL Standard says we should not allow anything that possibly
> > +        * modifies SQL-data. We enforce that with an executor check
> that we
> > +        * have not written any WAL.
>
>
Yes, removed too.


> * This should be reworded:
>
> > +        * Note that we don't add this to the MERGE Query's quals because
> > +        * that's not the logic MERGE uses.
> > +        */
> > +       action->qual = transformWhereClause(pstate, action->condition,
> > +                                           EXPR_KIND_MERGE_WHEN_AND,
> "WHEN");
>
> Perhaps say "WHEN ... AND quals are evaluated separately from the
> MERGE statement's join quals" instead. Or just lose it altogether.
>
>
Reworded slightly.


> * This comment seems inaccurate:
>
> > +                       /*
> > +                        * Process INSERT ... VALUES with a single VALUES
> > +                        * sublist.  We treat this case separately for
> > +                        * efficiency.  The sublist is just computed
> directly
> > +                        * as the Query's targetlist, with no VALUES
> RTE.  So
> > +                        * it works just like a SELECT without any FROM.
> > +                        */
>
> Wouldn't it be more accurate to say that this it totally different to
> what transformInsertStmt() does for a SELECT without any FROM? Also,
> do you think that that's good?
>

Actually it's pretty much same as what transformInsertStmt(). Even the
comments are verbatim copied from there. I don't see anything wrong with
the comment or the code. Can you please explain what problem you see?


>
> >> * Tests for transition table behavior (mixed INSERTs, UPDATEs, and
> >> DELETEs) within triggers.sql seems like a good idea.
> >
> > Ok, I will add. But not done in this version.
>
> Note that it's implied in at least one place that we don't support
> transition tables at all:
>
> > +               /*
> > +                * XXX if we support transition tables this would need
> to move
> > +                * earlier before ExecSetupTransitionCaptureState()
> > +                */
> > +               switch (action->commandType)
> > +               {
>
> You'll want to get to this as part of that transition table effort.
>

I actually didn't even know what transition tables are until today. But
today I studied them and the new version now supports transition tables
with MERGE. We might consider it to WIP given the amount of time I've spent
coding that, though I am fairly happy with the result so far. The comment
above turned out to be bogus.

I decided to create new tuplestores and mark them as new_update,
old_update, new_insert and old_delete. This is necessary because MERGE can
run all three kinds of commands i.e. UPDATE, DELETE and INSERT, and we
would like to track their transition tables separately.

(Hmm.. I just noticed though INSERT ON CONFLICT does not do this and still
able to track transition tables for INSERT and UPDATE correctly. So may be
what I did wasn't necessary after all, though it's also likely that we can
get IOC to use this new mechanism)


>
> Any plan to fix this/support identity columns? I see that you don't
> support them here:
>
> > +                   /*
> > +                    * Handle INSERT much like in transformInsertStmt
> > +                    *
> > +                    * XXX currently ignore stmt->override, if present
> > +                    */
>
>
I have already added support for OVERRIDING. The comment needed adjustments
which I have done now.


> I think that this is a blocker, unfortunately.
>

You mean OVERRIDING or ruleutils?


>
> >> * I still think we probably need to add something to ruleutils.c, so
> >> that MERGE Query structs can be deparsed -- see get_query_def().
> >
> > Ok. I will look at it. Not done in this version though.
>
> I also wonder what it would take to support referencing CTEs. Other
> implementations do have this. Why can't we?
>

Hmm. I will look at them. But TBH I would like to postpone them to v12 if
they turn out to be tricky. We have already covered a very large ground in
the last few weeks, but we're approaching the feature freeze date.


>
> Phew! Thanks for your patience and perseverance. I do have more
> feedback on the docs lined up, but that isn't so important right now.
>

Thanks! Those were really useful review comments. In passing, I made some
updates to the doc, but I really should make a complete pass over the patch.

Thanks,
Pavan

-- 
Pavan Deolasee                   http://www.2ndQuadrant.com/
<http://www.2ndquadrant.com/>
PostgreSQL Development, 24x7 Support, Training & Services

  178. Re: [HACKERS] MERGE SQL Statement for PG11

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2018-03-09T15:17:57Z

    Please don't forget to remove the xlog.c and miscadmin.h hunks from the
patch.

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  179. Re: [HACKERS] MERGE SQL Statement for PG11

    Peter Geoghegan <pg@bowt.ie> — 2018-03-11T03:57:08Z

    On Fri, Mar 9, 2018 at 6:55 AM, Pavan Deolasee <pavan.deolasee@gmail.com> wrote:
>> * Is this actually needed at all?:
>>
>> > +       /* In MERGE when and condition, no system column is allowed */
>> > +       if (pstate->p_expr_kind == EXPR_KIND_MERGE_WHEN_AND &&
>> > +           attnum < InvalidAttrNumber &&
>> > +           !(attnum == TableOidAttributeNumber || attnum ==
>> > ObjectIdAttributeNumber))
>> > +           ereport(ERROR,
>> > +                   (errcode(ERRCODE_INVALID_COLUMN_REFERENCE),
>> > +                    errmsg("system column \"%s\" reference in WHEN AND
>> > condition is invalid",
>> > +                           colname),
>> > +                    parser_errposition(pstate, location)));
>>
>> We're talking about the scantuple here. It's not like excluded.*.

I often care about things like system columns not because of the
user-visible functionality, but because it reassures me that the
design is robust.

>> I think that this is a blocker, unfortunately.
>
> You mean OVERRIDING or ruleutils?

I meant OVERRIDING, but ruleutils seems like something we need, too.

>> >> * I still think we probably need to add something to ruleutils.c, so
>> >> that MERGE Query structs can be deparsed -- see get_query_def().
>> >
>> > Ok. I will look at it. Not done in this version though.
>>
>> I also wonder what it would take to support referencing CTEs. Other
>> implementations do have this. Why can't we?
>
>
> Hmm. I will look at them. But TBH I would like to postpone them to v12 if
> they turn out to be tricky. We have already covered a very large ground in
> the last few weeks, but we're approaching the feature freeze date.

I quickly implemented CTE support myself (not wCTE support, since
MERGE doesn't use RETURNING), and it wasn't tricky. It seems to work
when I mechanically duplicate the approach taken with other types of
DML statement in the parser. I have written a few tests, and so far it
holds up.

I also undertook something quite a bit harder: Changing the
representation of the range table from parse analysis on. As I
mentioned in passing at one point, I'm concerned about the fact that
there are two RTEs for the target relation in all cases. You
introduced a separate Query.resultRelation-style RTI index,
Query.mergeTarget_relation, and we see stuff like this through every
stage of query processing, from parse analysis right through to
execution. One problem with the existing approach is that it leaves
many cases where EXPLAIN MERGE shows the target relation alias "t" as
"t_1" for some planner nodes, though not for others. More importantly,
I suspect that having two distinct RTEs has introduced special cases
that are not really needed, and will be more bug-prone (in fact, there
are bugs already, which I'll get to at the end). I think that it's
fair to say that what happens in the patch with EvalPlanQual()'s RTI
argument is ugly, especially because we also have a separate
resultRelInfo that we *don't* use. We should aspire to have a MERGE
implementation that isn't terribly different to UPDATE or DELETE,
especially within the executor.

I wrote a very rough patch that arranged for the MERGE rtable to have
only a single relation RTE. My approach was to teach
transformFromClauseItem() and friends to recognize that they shouldn't
create a new RTE for the MERGE target RangeVar. I actually added some
hard coding to getRTEForSpecialRelationTypes() for this, which is ugly
as sin, but the general approach is likely salvageable (we could
invent a special type of RangeVar to do this, perhaps). The point here
is that everything just works if there isn't a separate RTE for the
join's leftarg and the setTargetTable() target, with exactly one
exception: partitioning becomes *thoroughly* broken. Every single
partitioning test fails with "ERROR: no relation entry for relid 1",
and occasionally some other "can't happen" error. This looks like it
would be hard to fix -- at least, I'd find it hard to fix.

This is an extract from header comments for inheritance_planner()
(master branch):

 * We have to handle this case differently from cases where a source relation
 * is an inheritance set. Source inheritance is expanded at the bottom of the
 * plan tree (see allpaths.c), but target inheritance has to be expanded at
 * the top.  The reason is that for UPDATE, each target relation needs a
 * different targetlist matching its own column set.  Fortunately,
 * the UPDATE/DELETE target can never be the nullable side of an outer join,
 * so it's OK to generate the plan this way.

Of course, that isn't true of MERGE: The MERGE target *can* be the
nullable side of an outer join. That's probably a big complication for
using one target RTE. Your approach to implementing partitioning [1]
seems to benefit from having two different RTEs, in a way -- you
sidestep the restriction. As you put it, "Since MERGE need both the
facilities [both INSERT and UPDATE facilities], I'd to pretty much
merge both the machineries". As I understand it, you "merge" these
machineries by having find_mergetarget_for_rel() work backwards. You
are mapping from one relation tree to another relation tree in an
ad-hoc fashion -- both trees for the same underlying RTE. (You
compensate for this later, in the executor, with the special
EvalPlanQual() RTI stuff I mentioned already.)

I have some broad concerns here. I would especially like to hear from
hackers that know more about partitioning/inheritance than I do on
these concerns. They are:

* Is it okay to take this approach with partitioning?

I worry about things like ExecAuxRowMark handling. We avoid calling
EvalPlanQualSetPlan() within ExecModifyTable() for CMD_MERGE, so I'm
pretty sure that that's already broken as-is. Separately, can't see
why it's okay that CMD_MERGE doesn't have mt_transition_capture
initialization occur per-child, so that's probably another bug of the
same general variety. To be clear, this is the specific part of the
patch that avoids going through child plans as described:

> @@ -1927,7 +2590,14 @@ ExecModifyTable(PlanState *pstate)
>         {
>             /* advance to next subplan if any */
>             node->mt_whichplan++;
> -           if (node->mt_whichplan < node->mt_nplans)
> +
> +           /*
> +            * If we are executing MERGE, we only need to execute the first
> +            * subplan since it's guranteed to return all the required tuples.
> +            * In fact, running remaining subplans would be a problem since we
> +            * will end up fetching the same tuples N times.
> +            */
> +           if (node->mt_whichplan < node->mt_nplans && (operation != CMD_MERGE))
>             {
>                 resultRelInfo++;
>                 subplanstate = node->mt_plans[node->mt_whichplan];

* Is there a way to make what I describe (having a single target RTE)
work with partitioning, without any big new special cases, especially
in the executor?

* Any thoughts on this multiple-RTEs-for-target-rel business more generally?

[1] https://postgr.es/m/CABOikdPjjG+JcdNeegrL7=BtPdJ6yEv--V4hU8KzJTTwX1SNmw@mail.gmail.com
-- 
Peter Geoghegan

  180. Re: [HACKERS] MERGE SQL Statement for PG11

    Pavan Deolasee <pavan.deolasee@gmail.com> — 2018-03-11T05:22:02Z

    On Sun, Mar 11, 2018 at 9:27 AM, Peter Geoghegan <pg@bowt.ie> wrote:

>
> >>
> >> We're talking about the scantuple here. It's not like excluded.*.
>
> I often care about things like system columns not because of the
> user-visible functionality, but because it reassures me that the
> design is robust.
>
>
Ok. I will look at it. I think it shouldn't be too difficult and the
original restriction was mostly a fallout of expecting CHECK constraint
style expressions there.


> >> I think that this is a blocker, unfortunately.
> >
> > You mean OVERRIDING or ruleutils?
>
> I meant OVERRIDING, but ruleutils seems like something we need, too.
>

Ok. OVERRIDING is done. I think we can support ruleutils easily too. I
don't know how to test that though.


>
> >> >> * I still think we probably need to add something to ruleutils.c, so
> >> >> that MERGE Query structs can be deparsed -- see get_query_def().
> >> >
> >> > Ok. I will look at it. Not done in this version though.
> >>
> >> I also wonder what it would take to support referencing CTEs. Other
> >> implementations do have this. Why can't we?
> >
> >
> > Hmm. I will look at them. But TBH I would like to postpone them to v12 if
> > they turn out to be tricky. We have already covered a very large ground
> in
> > the last few weeks, but we're approaching the feature freeze date.
>
> I quickly implemented CTE support myself (not wCTE support, since
> MERGE doesn't use RETURNING), and it wasn't tricky. It seems to work
> when I mechanically duplicate the approach taken with other types of
> DML statement in the parser. I have written a few tests, and so far it
> holds up.
>

Ok, thanks. I started doing something similar, but great if you have
already implemented. I will focus on other things for now.


>
> I also undertook something quite a bit harder: Changing the
> representation of the range table from parse analysis on. As I
> mentioned in passing at one point, I'm concerned about the fact that
> there are two RTEs for the target relation in all cases. You
> introduced a separate Query.resultRelation-style RTI index,
> Query.mergeTarget_relation, and we see stuff like this through every
> stage of query processing, from parse analysis right through to
> execution. One problem with the existing approach is that it leaves
> many cases where EXPLAIN MERGE shows the target relation alias "t" as
> "t_1" for some planner nodes, though not for others. More importantly,
> I suspect that having two distinct RTEs has introduced special cases
> that are not really needed, and will be more bug-prone (in fact, there
> are bugs already, which I'll get to at the end). I think that it's
> fair to say that what happens in the patch with EvalPlanQual()'s RTI
> argument is ugly, especially because we also have a separate
> resultRelInfo that we *don't* use. We should aspire to have a MERGE
> implementation that isn't terribly different to UPDATE or DELETE,
> especially within the executor.
>

I thought for a while about this and even tried multiple approaches before
settling for what we have today. The biggest challenge is that
inheritance/partition tables take completely different paths for INSERTs
and UPDATE/DELETE. The RIGHT OUTER JOIN makes it kinda difficult because
the regular UPDATE/DELETE code path ends up throwing duplicates when the
source table is joined with individual partitions. IIRC that's the sole
reason why I'd to settle on pushing the JOIN underneath, give it SELECT
like treatment and then handle UPDATE/DELETE in the executor.


>
> I wrote a very rough patch that arranged for the MERGE rtable to have
> only a single relation RTE. My approach was to teach
> transformFromClauseItem() and friends to recognize that they shouldn't
> create a new RTE for the MERGE target RangeVar. I actually added some
> hard coding to getRTEForSpecialRelationTypes() for this, which is ugly
> as sin, but the general approach is likely salvageable (we could
> invent a special type of RangeVar to do this, perhaps). The point here
> is that everything just works if there isn't a separate RTE for the
> join's leftarg and the setTargetTable() target, with exactly one
> exception: partitioning becomes *thoroughly* broken. Every single
> partitioning test fails with "ERROR: no relation entry for relid 1",
> and occasionally some other "can't happen" error. This looks like it
> would be hard to fix -- at least, I'd find it hard to fix.
>
>
Ok. If you've something which is workable, then great. But AFAICS this is
what the original patch was doing until we came to support partitioning.
Even with partitioning I could get everything to work, without duplicating
the RTE, except the duplicate rows issue. I don't know how to solve that
without doing what I've done or completely rewriting UPDATE/DELETE handling
for inheritance/partition table. If you or others have better ideas, they
are most welcome.


> This is an extract from header comments for inheritance_planner()
> (master branch):
>
>  * We have to handle this case differently from cases where a source
> relation
>  * is an inheritance set. Source inheritance is expanded at the bottom of
> the
>  * plan tree (see allpaths.c), but target inheritance has to be expanded at
>  * the top.  The reason is that for UPDATE, each target relation needs a
>  * different targetlist matching its own column set.  Fortunately,
>  * the UPDATE/DELETE target can never be the nullable side of an outer
> join,
>  * so it's OK to generate the plan this way.
>
> Of course, that isn't true of MERGE: The MERGE target *can* be the
> nullable side of an outer join. That's probably a big complication for
> using one target RTE. Your approach to implementing partitioning [1]
> seems to benefit from having two different RTEs, in a way -- you
> sidestep the restriction.


Right. The entire purpose of having two different RTEs is to work around
this problem. I explained this approach here [1]. I didn't receive any
objections then, but that's mostly because nobody read it carefully. As I
said, if we have an alternate feasible and better mechanism, let's go for
it as long as efforts are justifiable.

Thanks,
Pavan

[1]
https://www.postgresql.org/message-id/CABOikdM%2Bc1vB_%2B3tYEjO%3DJ6U2uNHzKU_b%3DU72tadD5-9xQcbHA%40mail.gmail.com

-- 
 Pavan Deolasee                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

  181. Re: [HACKERS] MERGE SQL Statement for PG11

    Peter Geoghegan <pg@bowt.ie> — 2018-03-11T05:48:37Z

    On Sat, Mar 10, 2018 at 9:22 PM, Pavan Deolasee
<pavan.deolasee@gmail.com> wrote:
> Ok. I will look at it. I think it shouldn't be too difficult and the
> original restriction was mostly a fallout of expecting CHECK constraint
> style expressions there.

Good, thanks.

> Ok. OVERRIDING is done. I think we can support ruleutils easily too. I don't
> know how to test that though.

Glad to hear it.

> I thought for a while about this and even tried multiple approaches before
> settling for what we have today. The biggest challenge is that
> inheritance/partition tables take completely different paths for INSERTs and
> UPDATE/DELETE. The RIGHT OUTER JOIN makes it kinda difficult because the
> regular UPDATE/DELETE code path ends up throwing duplicates when the source
> table is joined with individual partitions. IIRC that's the sole reason why
> I'd to settle on pushing the JOIN underneath, give it SELECT like treatment
> and then handle UPDATE/DELETE in the executor.

It sounds like we should try to thoroughly understand why these
duplicates arose. Did you actually call EvalPlanQualSetPlan() for all
subplans at the time?

> Ok. If you've something which is workable, then great. But AFAICS this is
> what the original patch was doing until we came to support partitioning.
> Even with partitioning I could get everything to work, without duplicating
> the RTE, except the duplicate rows issue. I don't know how to solve that
> without doing what I've done or completely rewriting UPDATE/DELETE handling
> for inheritance/partition table. If you or others have better ideas, they
> are most welcome.

I don't claim that what I wrote was workable with partitioning. But
I'm not getting how we can get away with not calling
EvalPlanQualSetPlan() for child plans, or something like it, as things
are.

> Right. The entire purpose of having two different RTEs is to work around
> this problem. I explained this approach here [1]. I didn't receive any
> objections then, but that's mostly because nobody read it carefully. As I
> said, if we have an alternate feasible and better mechanism, let's go for it
> as long as efforts are justifiable.

FWIW, you're right that I didn't give that aspect much thought until
quite recently. I'm no expert on partitioning.

As you know, there is an ON CONFLICT DO UPDATE + partitioning patch in
the works from Alvaro. In your explanation about that approach that
you cited, you wondered what the trouble might have been with ON
CONFLICT + partitioning, and supposed that the issues were similar
there. Are they? Has that turned up much?

-- 
Peter Geoghegan

  182. Re: [HACKERS] MERGE SQL Statement for PG11

    Pavan Deolasee <pavan.deolasee@gmail.com> — 2018-03-12T12:13:43Z

    On Sun, Mar 11, 2018 at 11:18 AM, Peter Geoghegan <pg@bowt.ie> wrote:

>
> It sounds like we should try to thoroughly understand why these
> duplicates arose. Did you actually call EvalPlanQualSetPlan() for all
> subplans at the time?
>
>
The reason for duplicates or even wrong answers is quite simple. The way
UPDATE/DELETE currently works for partition table is that we expand the
inheritance tree for the parent result relation and then create a subplan
for each partition separately. This works fine, even when there exists a
FROM/USING clause in the UPDATE/DELETE statement because the final result
does not change irrespective of whether you first do a UNION ALL between
all partitions and then find the candidate rows or whether you find
candidate rows from individual partitions separately.

In case of MERGE though, since we are performing a RIGHT OUTER JOIN between
the result relation and the source relation, we may conclude that a
matching target  row does not exist for a source row, whereas it actually
exists but in some other partition. For example,

CREATE TABLE target (key int, val text) PARTITION BY LIST ( key);
CREATE TABLE part1 PARTITION OF target FOR VALUES IN (1, 2, 3);
CREATE TABLE part2 PARTITION OF target FOR VALUES IN (4, 5, 6);
CREATE TABLE source (skey integer);
INSERT INTO source VALUES (1), (4), (7);
INSERT INTO part1 VALUES (1, 'v1'), (2, 'v2'), (3, 'v3');
INSERT INTO part2 VALUES (4, 'v4'), (5, 'v5'), (6, 'v6');

postgres=# SELECT * FROM target RIGHT OUTER JOIN source ON key = skey;
 key | val | skey
-----+-----+------
   1 | v1  |    1
   4 | v4  |    4
     |     |    7
(3 rows)

This gives the right answer. But if we join individual partitions and then
do a UNION ALL,

postgres=# SELECT * FROM part1 RIGHT OUTER JOIN source ON key = skey UNION
ALL SELECT * FROM part2 RIGHT OUTER JOIN source ON key = skey;
 key | val | skey
-----+-----+------
   1 | v1  |    1
     |     |    4
     |     |    7
     |     |    1
   4 | v4  |    4
     |     |    7
(6 rows)

This is what nodeModifyTable does and hence we end up getting duplicates or
even incorrectly declared NOT MATCHED rows, where as they are matched in a
different partition.

I don't think not calling EvalPlanQualSetPlan() on all subplans is a
problem because we really never execute those subplans. In fact. we should
fix that so that those subplans are not even initialised.


> As you know, there is an ON CONFLICT DO UPDATE + partitioning patch in
> the works from Alvaro. In your explanation about that approach that
> you cited, you wondered what the trouble might have been with ON
> CONFLICT + partitioning, and supposed that the issues were similar
> there. Are they? Has that turned up much?
>
>
Well, I initially thought that ON CONFLICT DO UPDATE on partition table may
have the same challenges, but that's probably not the case. For INSERT ON
CONFLICT it's still just an INSERT path, with some special handling for
UPDATEs. Currently, for partition or inherited table, UPDATEs/DELETEs go
via inheritance_planner() thus expanding inheritance for the result
relation where as INSERTs go via simple grouping_planner().

For MERGE, we do all three DMLs. That doesn't mean we could not
re-implement MERGE on the lines of INSERTs, but that would most likely mean
complete re-writing of the UPDATEs/DELETEs for partition/inheritance
tables. The challenges would just be the same in both cases.

Thanks,
Pavan

-- 
 Pavan Deolasee                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

  183. Re: [HACKERS] MERGE SQL Statement for PG11

    Stephen Frost <sfrost@snowman.net> — 2018-03-16T01:58:59Z

    Greetings Pavan, all,

* Pavan Deolasee (pavan.deolasee@gmail.com) wrote:
> On 9 March 2018 at 08:29, Peter Geoghegan <pg@bowt.ie> wrote:
> >  My #1 concern has become RLS, and
> > perhaps only because I haven't studied it in enough detail.
> 
> Sure. I've done what I thought is the right thing to do, but please check.
> Stephen also wanted to review RLS related code; I don't know if he had
> chance to do so.

I've started looking through the code from an RLS perspective and, at
least on my initial review, it looks alright.

A couple test that aren't included that I think should be in the
regression suire are where both tables have RLS policies and
where the RLS tables have actual SELECT policies beyond just 'true'.

I certainly see SELECT policies which limits the records that
individuals are allowed to see very frequently and it's an
important case to ensure works correctly.  I did a few tests here myself
and they behaved as I expected, and reading through the code it looks
reasonable, but they should be simple to write tests which run very
quickly.

I'm a bit on the fence about it, but as MERGE is added in quite a few
places which previously mentioned UPDATE and DELETE throughout the
system, I wonder if we shouldn't do better than this:

=*# create policy p1 on t1 for merge using ((c1 % 2) = 0);
ERROR:  syntax error at or near "merge"
LINE 1: create policy p1 on t1 for merge using ((c1 % 2) = 0);

Specifically, perhaps we should change that to pick up on MERGE being
asked for there and return an error saying that policies for the MERGE
command aren't supported with a HINT that MERGE respects
INSERT/UPDATE/DELETE policies and that users should use those instead.

Also, in nodeModifyTable.c, there's a comment:

 * The WITH CHECK quals are applied in ExecUpdate() and hence we need
 * not do anything special to handle them.

Which I believe is actually getting at the fact that ExecUpdate() will
run ExecWithCheckOptions(WCO_RLS_UPDATE_CHECK ...) and therefore in
ExecMergeMatched() we don't need to check WCO_RLS_UPDATE_CHECK, but we
do still need to check WCO_RLS_MERGE_UPDATE_CHECK (and that's what the
code does).  One thing I wonder about there though is if we really need
to segregate those..?  What's actually making WCO_RLS_MERGE_UPDATE_CHECK
different from WCO_RLS_UPDATE_CHECK?  I get that the DELETE case is
different, because in a regular DELETE we'll never even see the row, but
for MERGE, we will see the row (assuming it passes SELECT policies, of
course) and then will check if it matches and that's when we realize
that we've been asked to run a DELETE, so we have the special-case of
WCO_RLS_MERGE_DELETE_CHECK, but that's not true of UPDATE, so I think
this might be adding a bit of unnecessary complication by introducing
WCO_RLS_MERGE_UPDATE_CHECK.

Thanks!

Stephen

  184. Re: [HACKERS] MERGE SQL Statement for PG11

    Pavan Deolasee <pavan.deolasee@gmail.com> — 2018-03-16T05:39:07Z

    Hi Stephen,


On Fri, Mar 16, 2018 at 7:28 AM, Stephen Frost <sfrost@snowman.net> wrote:

> Greetings Pavan, all,
>
> * Pavan Deolasee (pavan.deolasee@gmail.com) wrote:
> > On 9 March 2018 at 08:29, Peter Geoghegan <pg@bowt.ie> wrote:
> > >  My #1 concern has become RLS, and
> > > perhaps only because I haven't studied it in enough detail.
> >
> > Sure. I've done what I thought is the right thing to do, but please
> check.
> > Stephen also wanted to review RLS related code; I don't know if he had
> > chance to do so.
>
> I've started looking through the code from an RLS perspective and, at
> least on my initial review, it looks alright.
>


Thanks for taking time out to review the patch. It certainly helps a lot.


>
> A couple test that aren't included that I think should be in the
> regression suire are where both tables have RLS policies and
> where the RLS tables have actual SELECT policies beyond just 'true'.
>
> I certainly see SELECT policies which limits the records that
> individuals are allowed to see very frequently and it's an
> important case to ensure works correctly.  I did a few tests here myself
> and they behaved as I expected, and reading through the code it looks
> reasonable, but they should be simple to write tests which run very
> quickly.
>

Ok. I will add those tests.


>
> I'm a bit on the fence about it, but as MERGE is added in quite a few
> places which previously mentioned UPDATE and DELETE throughout the
> system, I wonder if we shouldn't do better than this:
>
> =*# create policy p1 on t1 for merge using ((c1 % 2) = 0);
> ERROR:  syntax error at or near "merge"
> LINE 1: create policy p1 on t1 for merge using ((c1 % 2) = 0);
>
> Specifically, perhaps we should change that to pick up on MERGE being
> asked for there and return an error saying that policies for the MERGE
> command aren't supported with a HINT that MERGE respects
> INSERT/UPDATE/DELETE policies and that users should use those instead.
>

Hmm. I am not sure if that would be a good idea just for RLS. We might then
also want to change several other places in the grammar to accept
INSERT/UPDATE/DELETE keyword and handle that during parse analysis. We can
certainly do that, but I am not sure if it adds a lot of value and
certainly adds a lot more code. We should surely document these things, if
we are not already.


>
> Also, in nodeModifyTable.c, there's a comment:
>
>  * The WITH CHECK quals are applied in ExecUpdate() and hence we need
>  * not do anything special to handle them.
>
> Which I believe is actually getting at the fact that ExecUpdate() will
> run ExecWithCheckOptions(WCO_RLS_UPDATE_CHECK ...) and therefore in
> ExecMergeMatched() we don't need to check WCO_RLS_UPDATE_CHECK, but we
> do still need to check WCO_RLS_MERGE_UPDATE_CHECK (and that's what the
> code does).


Right.


>   One thing I wonder about there though is if we really need
> to segregate those..?  What's actually making WCO_RLS_MERGE_UPDATE_CHECK
> different from WCO_RLS_UPDATE_CHECK?  I get that the DELETE case is
> different, because in a regular DELETE we'll never even see the row, but
> for MERGE, we will see the row (assuming it passes SELECT policies, of
> course) and then will check if it matches and that's when we realize
> that we've been asked to run a DELETE, so we have the special-case of
> WCO_RLS_MERGE_DELETE_CHECK, but that's not true of UPDATE, so I think
> this might be adding a bit of unnecessary complication by introducing
> WCO_RLS_MERGE_UPDATE_CHECK.
>

I've modelled this code on the lines of ON CONFLICT DO NOTHING. And quite
similar to that, I believe we need separate handling for MERGE as well
because we don't (and can't) push down the USING security quals of
UPDATE/DELETE to the scan. So we need to separately check that the target
row actually passes the USING quals. WCO_RLS_MERGE_UPDATE_CHECK and
WCO_RLS_MERGE_DELETE_CHECK
are used for that purpose.

One point to deliberate though is whether it's a good idea to throw an
error when the USING quals fail or should we silently ignore such rows.
Regular UPDATE/DELETE does the latter and ON CONFLICT DO UPDATE does the
former. I chose to throw an error because otherwise it may get confusing to
the user since a row would neither be updated (meaning, it will be seen as
a case of NOT MATCHED), but nor be inserted. I hope no problem there.

Thanks,
Pavan


-- 
 Pavan Deolasee                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

  185. Re: [HACKERS] MERGE SQL Statement for PG11

    Pavan Deolasee <pavan.deolasee@gmail.com> — 2018-03-18T06:01:58Z

    On Mon, Mar 12, 2018 at 5:43 PM, Pavan Deolasee <pavan.deolasee@gmail.com>
wrote:

>
>
> On Sun, Mar 11, 2018 at 11:18 AM, Peter Geoghegan <pg@bowt.ie> wrote:
>
>>
>>
>> As you know, there is an ON CONFLICT DO UPDATE + partitioning patch in
>> the works from Alvaro. In your explanation about that approach that
>> you cited, you wondered what the trouble might have been with ON
>> CONFLICT + partitioning, and supposed that the issues were similar
>> there. Are they? Has that turned up much?
>>
>>
> Well, I initially thought that ON CONFLICT DO UPDATE on partition table
> may have the same challenges, but that's probably not the case. For INSERT
> ON CONFLICT it's still just an INSERT path, with some special handling for
> UPDATEs. Currently, for partition or inherited table, UPDATEs/DELETEs go
> via inheritance_planner() thus expanding inheritance for the result
> relation where as INSERTs go via simple grouping_planner().
>
> For MERGE, we do all three DMLs. That doesn't mean we could not
> re-implement MERGE on the lines of INSERTs, but that would most likely mean
> complete re-writing of the UPDATEs/DELETEs for partition/inheritance
> tables. The challenges would just be the same in both cases.
>
>
Having thought more about this in the last couple of days, I am actually
inclined to try out rewrite the UPDATE handling of MERGE on the lines of
what ON CONFLICT DO UPDATE patch is doing. This might help us to completely
eliminate invoking inheritance_planner() for partition table and that will
be a huge win for tables with several hundred partitions. The code might
also look much cleaner that way. I am gonna give it a try for next couple
of days and see if its doable.

Thanks,
Pavan

  186. Re: [HACKERS] MERGE SQL Statement for PG11

    Pavan Deolasee <pavan.deolasee@gmail.com> — 2018-03-20T14:08:00Z

    On Sun, Mar 18, 2018 at 11:31 AM, Pavan Deolasee <pavan.deolasee@gmail.com>
wrote:

>
>
> On Mon, Mar 12, 2018 at 5:43 PM, Pavan Deolasee <pavan.deolasee@gmail.com>
> wrote:
>
>>
>>
>> On Sun, Mar 11, 2018 at 11:18 AM, Peter Geoghegan <pg@bowt.ie> wrote:
>>
>>>
>>>
>>> As you know, there is an ON CONFLICT DO UPDATE + partitioning patch in
>>> the works from Alvaro. In your explanation about that approach that
>>> you cited, you wondered what the trouble might have been with ON
>>> CONFLICT + partitioning, and supposed that the issues were similar
>>> there. Are they? Has that turned up much?
>>>
>>>
>> Well, I initially thought that ON CONFLICT DO UPDATE on partition table
>> may have the same challenges, but that's probably not the case. For INSERT
>> ON CONFLICT it's still just an INSERT path, with some special handling for
>> UPDATEs. Currently, for partition or inherited table, UPDATEs/DELETEs go
>> via inheritance_planner() thus expanding inheritance for the result
>> relation where as INSERTs go via simple grouping_planner().
>>
>> For MERGE, we do all three DMLs. That doesn't mean we could not
>> re-implement MERGE on the lines of INSERTs, but that would most likely mean
>> complete re-writing of the UPDATEs/DELETEs for partition/inheritance
>> tables. The challenges would just be the same in both cases.
>>
>>
> Having thought more about this in the last couple of days, I am actually
> inclined to try out rewrite the UPDATE handling of MERGE on the lines of
> what ON CONFLICT DO UPDATE patch is doing. This might help us to completely
> eliminate invoking inheritance_planner() for partition table and that will
> be a huge win for tables with several hundred partitions. The code might
> also look much cleaner that way. I am gonna give it a try for next couple
> of days and see if its doable.
>
>
So here is a version that completely avoids taking the
inheritance_planner() route and rather handles the MERGE UPDATE/DELETE
during execution, by translating tuples between child and root partition
and vice versa. To achieve this we no longer expand partition inheritance
for result relation, just like INSERT.

Apart from making the code look better, this also gives a very nice boost
to performance. In fact, v20 performed very poorly with 10 or more
partitions.  Whereas this version even beats regular UPDATE when there are
10 or more partitions. This is because we are able to avoid calling
grouping_planner() repeatedly for all the partitions, by not expanding
partition tree for the result relation.

This patch is based on the on-going work on ON CONFLICT DO UPDATE for
partitioned table. I've attached the base patch that I am using from that
work and I will re-base MERGE patch once the other patch set takes a final
shape.

Apart from this major change, there are several other changes:

- moved some of the new code to two new source files,
src/backend/executor/nodeMerge.c and src/backend/parser/parse_merge.c. This
necessitated exporting ExecUpdate/ExecInsert/ExecDelete functions from
nodeModifyTable.c, but I think it's still a better change.
- changed the way slots were created per action-state. Instead, now we have
a single slot per result relation and we only change slot-descriptor while
working with specific action. We of course need to track tuple descriptor
per action though
- moved some merge-specific state, such as list of matched and not-matched
actions to result-relation. In case of partitioned table, if the partition
child'd schema differs from the root, then we setup new merge-state for the
partition and transform various members of the action-state to reflect the
child table's schema

Regarding few other things:
- I wrote code for deparsing MERGE, but don't know how to test that
- I could support the system column references to the target relation by
simply removing the blocking error, but it currently does not work for
source relation. I am not sure if we even want to support that. But we
should definitely throw a more user friendly error than what the patch does
today.


Thanks,
Pavan

-- 
 Pavan Deolasee                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

  187. Re: [HACKERS] MERGE SQL Statement for PG11

    Peter Geoghegan <pg@bowt.ie> — 2018-03-20T20:47:40Z

    On Thu, Mar 8, 2018 at 6:59 PM, Peter Geoghegan <pg@bowt.ie> wrote:
> Phew! Thanks for your patience and perseverance. I do have more
> feedback on the docs lined up, but that isn't so important right now.

I want to get this feedback on the docs to you now. This is a little
raw, as it's taken from notes made 2 weeks ago. I haven't checked if
these still points haven't been addressed already, so apologies if
you've already done something about some number of them.

* Why is terminating semi-colon on its own line in MERGE sql examples
from the docs?

* ON CONFLICT references in MERGE docs should be a "Tip" box, IMV.

* Docs don't actually say what DO NOTHING does.

* The docs say "A condition cannot contain subqueries, set returning
functions, nor can it contain window or aggregate functions".  Thought
it can now?

* The docs say "INSERT actions cannot contain sub-selects".  Didn't that change?

* The docs on merge_insert grammar element say "Do not include the
table name", which seems like it was already covered.  It's obvious.
Suggest removing it.

* The docs say "The number of rows inserted, updated and deleted can
be seen in the output of EXPLAIN ANALYZE". This seems unnecessary. We
don't see that for ON CONFLICT.

* The docs say "it should be noted that no error is raised if that
occurs". But should it? I don't think that this addresses a perception
that users are likely to have.

*  "See MERGE" hyperlink within MERGE docs themselves seem odd.
Suggest changing this.

* Trigger behavior doesn't belong in main MERGE doc page, IMV.  Just
like with ON CONFLICT, it should be documented where triggers are
already documented. The specifics of MERGE can be discussed there.

* It might make sense to point out in the docs that join_condition
should not filter the target table too much. Like SQL server docs say,
don't put things in the join that filter the target that actually
belong in the WHEN .. AND quals. In a way, this should be obvious,
because it's an outer join. But I don't think it is, and ISTM that the
sensible thing to do is to warn against it.

* We never actually get around to saying that MERGE is good with bulk
loading, ETL, and so on. I think that we should remark on that in
passing.

* I think that the mvcc.sgml changes can go. Perhaps a passing
reference to MERGE can be left behind, that makes it clear that it's
really rather like UPDATE FROM and so on. The fact that it's like
UPDATE FROM now seems crystal clear.

* insert.sgml need not mention MERGE IMV.

I also have some additional minor feedback on the code itself that
I'll send now, which is a bit more recent (based on the version posted
on 2018-03-08):

* Most of the field comments here should be improved:

> /* ----------------
>  *   MergeActionState information
>  * ----------------
>  */
> typedef struct MergeActionState
> {
>     NodeTag     type;
>     bool        matched;        /* MATCHED or NOT MATCHED */
>     ExprState  *whenqual;       /* WHEN quals */
>     CmdType     commandType;    /* type of action */
>     TupleTableSlot *slot;       /* instead of ResultRelInfo */
>     ProjectionInfo *proj;       /* instead of ResultRelInfo */
> } MergeActionState;

* I wouldn't talk about a precedent like this:

> >  /*
> >   * Process BEFORE EACH STATEMENT triggers
> > + *
> > + * The precedent set by ON CONFLICT is that we fire INSERT then UPDATE.
> > + * MERGE follows the same logic, firing INSERT, then UPDATE, then DELETE.
> >   */

The comments should read as one voice. Ideally, it will look like ON
CONFLICT could have just as easily been added after MERGE, unless
there is a very compelling reason to mention a precedent. I mean, is
there really any reason to think that the precedent set by ON CONFLICT
actually made any difference? Is the suggestion here that there is a
better behavior, that we cannot go with for historical reasons?

* This ExecModifyTable() code seems a bit odd:

>
>         if (operation == CMD_MERGE)
>         {
>             ExecMerge(node, estate, slot, junkfilter, resultRelInfo);
>             continue;
>         }
>
>         tupleid = NULL;
>         oldtuple = NULL;
>         if (junkfilter != NULL)
>         {
>             /*
>              * extract the 'ctid' or 'wholerow' junk attribute.
>              */
>             if (operation == CMD_UPDATE ||
>                 operation == CMD_DELETE ||
>                 operation == CMD_MERGE)
>             {

Why is there a test for CMD_MERGE if control flow cannot even reach
there? What's going on here?

* No need to say "not planned" here, I think:

> +typedef struct MergeAction
> +{
> +   NodeTag     type;
> +   bool        matched;        /* MATCHED or NOT MATCHED */
> +   Node       *condition;      /* conditional expr (raw parser) */
> +   Node       *qual;           /* conditional expr (transformWhereClause) */
> +   CmdType     commandType;    /* type of action */
> +   Node       *stmt;           /* T_UpdateStmt etc - not planned */
> +   List       *targetList;     /* the target list (of ResTarget) */
> +} MergeAction;

* Do tables with rules reject MERGE commands sensibly? We should have
a test for that.

* Logical decoding tests (test_decoding regression tests) seem like a
good idea. This is much less important than with ON CONFLICT, since
you don't have the significant complication of speculative insertions,
but it seems like something to add on general principle.

-- 
Peter Geoghegan

  188. Re: [HACKERS] MERGE SQL Statement for PG11

    Pavan Deolasee <pavan.deolasee@gmail.com> — 2018-03-21T12:23:47Z

    Hi Peter,

Thanks for the additional comments.

On Wed, Mar 21, 2018 at 2:17 AM, Peter Geoghegan <pg@bowt.ie> wrote:

>
>
> * Why is terminating semi-colon on its own line in MERGE sql examples
> from the docs?
>

I checked a few other places and didn't find another example which puts
semi-colon on its own new line. So fixed per your suggestion.


>
> * ON CONFLICT references in MERGE docs should be a "Tip" box, IMV.
>

Changed.


>
> * Docs don't actually say what DO NOTHING does.
>

Added a para about that. In passing, I noticed that even though the grammar
and the docs agree that DO NOTHING is only supported for WHEN NOT MATCHED,
ExecMergeMatched() was unnecessarily checking for DO NOTHING. So fixed the
code.


>
> * The docs say "A condition cannot contain subqueries, set returning
> functions, nor can it contain window or aggregate functions".  Thought
> it can now?
>

Yes, it now supports sub-queries. What about set-returning, aggregates etc?
I assume they are not supported in other places such as WHERE conditions
and JOIN quals. So they will continue to remain blocked even in WHEN
conditions. Do you think it's worth mentioning or we should not mention
anything at all?


>
> * The docs say "INSERT actions cannot contain sub-selects".  Didn't that
> change?
>

No, it did not. We only support VALUES clause with INSERT action.


>
> * The docs on merge_insert grammar element say "Do not include the
> table name", which seems like it was already covered.  It's obvious.
> Suggest removing it.
>

Ok, fixed.


>
> * The docs say "The number of rows inserted, updated and deleted can
> be seen in the output of EXPLAIN ANALYZE". This seems unnecessary. We
> don't see that for ON CONFLICT.
>

You mean the doc changes are unnecessary or the EXPLAIN ANALYZE output is
unnecessary? I assume the doc changes, but let me know if that's wrong
assumption.


>
> * The docs say "it should be noted that no error is raised if that
> occurs". But should it? I don't think that this addresses a perception
> that users are likely to have.
>

Again, do you mean we should raise error or just that the docs should not
mention anything about it? I don't think raising an error because the
candidate row did not meet any specified action is a good idea. May be some
day we add another option to store such rows in a separate temporary table.


>
> *  "See MERGE" hyperlink within MERGE docs themselves seem odd.
> Suggest changing this.
>

Removed.


>
> * Trigger behavior doesn't belong in main MERGE doc page, IMV.  Just
> like with ON CONFLICT, it should be documented where triggers are
> already documented. The specifics of MERGE can be discussed there.
>
>
Ok. I added couple of paras to trigger.sgml, but left merge.sgml untouched.
Suggestions for better wording are welcome.


> * It might make sense to point out in the docs that join_condition
> should not filter the target table too much. Like SQL server docs say,
> don't put things in the join that filter the target that actually
> belong in the WHEN .. AND quals. In a way, this should be obvious,
> because it's an outer join. But I don't think it is, and ISTM that the
> sensible thing to do is to warn against it.
>
>
Hmm, ok. Not sure how exactly to put that in words without confusing users.
Do you want to suggest something?


> * We never actually get around to saying that MERGE is good with bulk
> loading, ETL, and so on. I think that we should remark on that in
> passing.
>

Suggestion?


>
> * I think that the mvcc.sgml changes can go. Perhaps a passing
> reference to MERGE can be left behind, that makes it clear that it's
> really rather like UPDATE FROM and so on. The fact that it's like
> UPDATE FROM now seems crystal clear.
>
>
It seems useful to me. Should we move it to merge.sgml instead?


> * insert.sgml need not mention MERGE IMV.
>
>
Hmm. I am not sure. It seems worth leaving a reference there since MERGE
provides a new way to handle INSERTs.


>
>
> * Most of the field comments here should be improved:
>
> > /* ----------------
> >  *   MergeActionState information
> >  * ----------------
> >  */
> > typedef struct MergeActionState
> > {
> >     NodeTag     type;
> >     bool        matched;        /* MATCHED or NOT MATCHED */
> >     ExprState  *whenqual;       /* WHEN quals */
> >     CmdType     commandType;    /* type of action */
> >     TupleTableSlot *slot;       /* instead of ResultRelInfo */
> >     ProjectionInfo *proj;       /* instead of ResultRelInfo */
> > } MergeActionState;
>
>
Done.


> * I wouldn't talk about a precedent like this:
>
> > >  /*
> > >   * Process BEFORE EACH STATEMENT triggers
> > > + *
> > > + * The precedent set by ON CONFLICT is that we fire INSERT then
> UPDATE.
> > > + * MERGE follows the same logic, firing INSERT, then UPDATE, then
> DELETE.
> > >   */
>
> The comments should read as one voice. Ideally, it will look like ON
> CONFLICT could have just as easily been added after MERGE, unless
> there is a very compelling reason to mention a precedent. I mean, is
> there really any reason to think that the precedent set by ON CONFLICT
> actually made any difference? Is the suggestion here that there is a
> better behavior, that we cannot go with for historical reasons?
>
>
I agree. I removed those comments.



> * This ExecModifyTable() code seems a bit odd:
>
> >
> >         if (operation == CMD_MERGE)
> >         {
> >             ExecMerge(node, estate, slot, junkfilter, resultRelInfo);
> >             continue;
> >         }
> >
> >         tupleid = NULL;
> >         oldtuple = NULL;
> >         if (junkfilter != NULL)
> >         {
> >             /*
> >              * extract the 'ctid' or 'wholerow' junk attribute.
> >              */
> >             if (operation == CMD_UPDATE ||
> >                 operation == CMD_DELETE ||
> >                 operation == CMD_MERGE)
> >             {
>
> Why is there a test for CMD_MERGE if control flow cannot even reach
> there? What's going on here?
>

Just leftover bits from the previous code. Removed.


>
> * No need to say "not planned" here, I think:
>
> > +typedef struct MergeAction
> > +{
> > +   NodeTag     type;
> > +   bool        matched;        /* MATCHED or NOT MATCHED */
> > +   Node       *condition;      /* conditional expr (raw parser) */
> > +   Node       *qual;           /* conditional expr
> (transformWhereClause) */
> > +   CmdType     commandType;    /* type of action */
> > +   Node       *stmt;           /* T_UpdateStmt etc - not planned */
> > +   List       *targetList;     /* the target list (of ResTarget) */
> > +} MergeAction;
>
>
Fixed and also improved other comments for the struct.



> * Do tables with rules reject MERGE commands sensibly? We should have
> a test for that.
>

That check was indeed missing. Added the check and the test.


>
> * Logical decoding tests (test_decoding regression tests) seem like a
> good idea. This is much less important than with ON CONFLICT, since
> you don't have the significant complication of speculative insertions,
> but it seems like something to add on general principle.
>

Good point, added a test.

Thanks,
Pavan

-- 
 Pavan Deolasee                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

  189. Re: [HACKERS] MERGE SQL Statement for PG11

    Peter Geoghegan <pg@bowt.ie> — 2018-03-21T19:45:24Z

    On Wed, Mar 21, 2018 at 5:23 AM, Pavan Deolasee
<pavan.deolasee@gmail.com> wrote:
>> * The docs say "A condition cannot contain subqueries, set returning
>> functions, nor can it contain window or aggregate functions".  Thought
>> it can now?
>
>
> Yes, it now supports sub-queries. What about set-returning, aggregates etc?
> I assume they are not supported in other places such as WHERE conditions and
> JOIN quals. So they will continue to remain blocked even in WHEN conditions.
> Do you think it's worth mentioning or we should not mention anything at all?

I would not mention anything at all. It's just the same as other DML statements.

>> * The docs say "INSERT actions cannot contain sub-selects".  Didn't that
>> change?
>
>
> No, it did not. We only support VALUES clause with INSERT action.

But can't you have a subselect in the VALUES()? Support for subselects
seems like a totally distinct thing to the restriction that only a
(single row) VALUES() is allowed in INSERT actions.

> You mean the doc changes are unnecessary or the EXPLAIN ANALYZE output is
> unnecessary? I assume the doc changes, but let me know if that's wrong
> assumption.

I meant the doc changes.

> Again, do you mean we should raise error or just that the docs should not
> mention anything about it? I don't think raising an error because the
> candidate row did not meet any specified action is a good idea. May be some
> day we add another option to store such rows in a separate temporary table.

I agree that it's not a good idea to raise an error because the
candidate row did not meet any specified action. My point is that you
don't even need to mention that we don't do that in the docs. It's not
like there is any reason to expect that we should -- there is no
precedent.

I think that that's in the docs because this behavior was contemplated
during MERGE's development. But that behavior was ultimately rejected.
In large part because there was no precedent.

>> * It might make sense to point out in the docs that join_condition
>> should not filter the target table too much. Like SQL server docs say,
>> don't put things in the join that filter the target that actually
>> belong in the WHEN .. AND quals. In a way, this should be obvious,
>> because it's an outer join. But I don't think it is, and ISTM that the
>> sensible thing to do is to warn against it.
>>
>
> Hmm, ok. Not sure how exactly to put that in words without confusing users.
> Do you want to suggest something?

Perhaps a Warning box should say:

Only columns from "target_table_name" that attempt to match
"data_source" rows should appear in "join_condition".
"join_condition" subexpressions that only reference
"target_table_name" columns can only affect which action is taken,
often in surprising ways.

>> * We never actually get around to saying that MERGE is good with bulk
>> loading, ETL, and so on. I think that we should remark on that in
>> passing.
>
>
> Suggestion?

How about adding this sentence after "MERGE ... a task that would
otherwise require multiple procedural language statements":

MERGE can synchronize two tables by modifying one table based on
differences between it and some other table.

The point here is that we're primarily talking about two whole tables.
That deserves such prominent placement, as that suggests where users
might really find MERGE useful, but without being too prescriptive.

Also, instead of saying "There are a variety of differences and
restrictions between the two statement types [MERGE and INSERT ... ON
CONFLICT DO UPDATE] and they are not interchangeable", you could
instead be specific, and say:

MERGE is well suited to synchronizing two tables using multiple
complex conditions. Using INSERT with ON CONFLICT DO UPDATE works well
when requirements are simpler. Only ON CONFLICT provides an atomic
INSERT or UPDATE outcome in READ COMMITTED mode.

BTW, the docs should be clear on the fact that "INSERT ... ON
CONFLICT" isn't a statement. INSERT is. ON CONFLICT is a clause.

>> * I think that the mvcc.sgml changes can go. Perhaps a passing
>> reference to MERGE can be left behind, that makes it clear that it's
>> really rather like UPDATE FROM and so on. The fact that it's like
>> UPDATE FROM now seems crystal clear.
>>
>
> It seems useful to me. Should we move it to merge.sgml instead?

The mvcc.sgml changes read like a status report on the patch's
behavior with concurrency. Obviously that general tone is not
appropriate for a committed patch. Also, what it describes doesn't
seem to have much to do with MVCC rules per say. The only thing that
seems to warrant discussion in mvcc.sgml is how MERGE really *isn't* a
special case. ISTM that you only really need to mention how the
decision to use one particular WHEN action can change repeatedly -
every time you walk the UPDATE chain, you start that part from the
beginning.

The "you might get a duplicate violation" bit can definitely live in
merge.sgml, right at the point that ON CONFLICT is mentioned (the Tip
box). I don't think that you need too much on this.

-- 
Peter Geoghegan

  190. Re: [HACKERS] MERGE SQL Statement for PG11

    Simon Riggs <simon@2ndquadrant.com> — 2018-03-22T07:59:41Z

    On 21 March 2018 at 19:45, Peter Geoghegan <pg@bowt.ie> wrote:

>>> * We never actually get around to saying that MERGE is good with bulk
>>> loading, ETL, and so on. I think that we should remark on that in
>>> passing.
>>
>>
>> Suggestion?
>
> How about adding this sentence after "MERGE ... a task that would
> otherwise require multiple procedural language statements":
>
> MERGE can synchronize two tables by modifying one table based on
> differences between it and some other table.
>
> The point here is that we're primarily talking about two whole tables.
> That deserves such prominent placement, as that suggests where users
> might really find MERGE useful, but without being too prescriptive.

The information I have is that many people are expecting MERGE to work
for OLTP since that is how it is used in other databases, not solely
as an ETL command.

So we're not primarily talking about two whole tables.

> Also, instead of saying "There are a variety of differences and
> restrictions between the two statement types [MERGE and INSERT ... ON
> CONFLICT DO UPDATE] and they are not interchangeable", you could
> instead be specific, and say:
>
> MERGE is well suited to synchronizing two tables using multiple
> complex conditions. Using INSERT with ON CONFLICT DO UPDATE works well
> when requirements are simpler. Only ON CONFLICT provides an atomic
> INSERT or UPDATE outcome in READ COMMITTED mode.
>
> BTW, the docs should be clear on the fact that "INSERT ... ON
> CONFLICT" isn't a statement. INSERT is. ON CONFLICT is a clause.

I think it would be better if you wrote a separate additional doc
patch to explain all of this, perhaps in Performance Tips section or
otherwise.

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  191. Re: [HACKERS] MERGE SQL Statement for PG11

    Simon Riggs <simon@2ndquadrant.com> — 2018-03-22T08:06:41Z

    On 21 March 2018 at 12:23, Pavan Deolasee <pavan.deolasee@gmail.com> wrote:

> Fixed

Looks like that this completes all outstanding items with the MERGE code.

I'm now proposing that I move to commit this, following my own final
review, on Tues 27 Mar in 5 days time, giving time for cleanup of
related issues.

If there are any items you believe are still open, please say so now
or mention any other objections you have.

Thanks for all of your detailed comments,

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  192. Re: [HACKERS] MERGE SQL Statement for PG11

    Pavan Deolasee <pavan.deolasee@gmail.com> — 2018-03-22T12:24:07Z

    On Thu, Mar 22, 2018 at 1:15 AM, Peter Geoghegan <pg@bowt.ie> wrote:

>
> >
> >
> > No, it did not. We only support VALUES clause with INSERT action.
>
> But can't you have a subselect in the VALUES()? Support for subselects
> seems like a totally distinct thing to the restriction that only a
> (single row) VALUES() is allowed in INSERT actions.
>
>
Ah, right. That works even today.

postgres=# CREATE TABLE target (a int, b text);
CREATE TABLE

postgres=# MERGE INTO target USING (SELECT 1) s ON false WHEN NOT MATCHED
THEN INSERT VALUES ((SELECT count(*) FROM pg_class), (SELECT relname FROM
pg_class LIMIT 1));
MERGE 1

postgres=# SELECT * FROM target;
  a  |       b
-----+----------------
 755 | pgbench_source
(1 row)


Thanks,
Pavan


-- 
 Pavan Deolasee                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

  193. Re: [HACKERS] MERGE SQL Statement for PG11

    Pavan Deolasee <pavan.deolasee@gmail.com> — 2018-03-22T18:42:04Z

    On Thu, Mar 22, 2018 at 1:36 PM, Simon Riggs <simon@2ndquadrant.com> wrote:

> On 21 March 2018 at 12:23, Pavan Deolasee <pavan.deolasee@gmail.com>
> wrote:
>
> > Fixed
>
> Looks like that this completes all outstanding items with the MERGE code.
>
>
A slightly improved version attached. Apart from doc cleanup based on
earlier feedback, fixed one assertion failure based on Rahila's report.
This was happening when target relation is referenced in the source
subquery. Fixed that and added a test case to test that situation.

Rebased on current master.

Thanks,
Pavan

-- 
 Pavan Deolasee                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

  194. Re: [HACKERS] MERGE SQL Statement for PG11

    Peter Geoghegan <pg@bowt.ie> — 2018-03-22T23:26:19Z

    On Thu, Mar 22, 2018 at 12:59 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
>> The point here is that we're primarily talking about two whole tables.
>> That deserves such prominent placement, as that suggests where users
>> might really find MERGE useful, but without being too prescriptive.
>
> The information I have is that many people are expecting MERGE to work
> for OLTP since that is how it is used in other databases, not solely
> as an ETL command.

I'm sure that that's true, which is why I said "...without being too
prescriptive".

> So we're not primarily talking about two whole tables.

Sure, but that's where MERGE is going to be compelling. Especially for
Postgres, which already has ON CONFLICT DO UPDATE.

>> Also, instead of saying "There are a variety of differences and
>> restrictions between the two statement types [MERGE and INSERT ... ON
>> CONFLICT DO UPDATE] and they are not interchangeable", you could
>> instead be specific, and say:
>>
>> MERGE is well suited to synchronizing two tables using multiple
>> complex conditions. Using INSERT with ON CONFLICT DO UPDATE works well
>> when requirements are simpler. Only ON CONFLICT provides an atomic
>> INSERT or UPDATE outcome in READ COMMITTED mode.
>>
>> BTW, the docs should be clear on the fact that "INSERT ... ON
>> CONFLICT" isn't a statement. INSERT is. ON CONFLICT is a clause.
>
> I think it would be better if you wrote a separate additional doc
> patch to explain all of this, perhaps in Performance Tips section or
> otherwise.

I don't think that it has much to do with performance.

-- 
Peter Geoghegan

  195. Re: [HACKERS] MERGE SQL Statement for PG11

    Peter Geoghegan <pg@bowt.ie> — 2018-03-22T23:45:01Z

    On Thu, Mar 22, 2018 at 1:06 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
> I'm now proposing that I move to commit this, following my own final
> review, on Tues 27 Mar in 5 days time, giving time for cleanup of
> related issues.
>
> If there are any items you believe are still open, please say so now
> or mention any other objections you have.

I would like to see clarity on the use of multiple RTEs for the target
table by MERGE. See my remarks to Pavan just now. Also, I think that
the related issue of how partitioning was implemented needs to get a
lot more review (partitioning is the whole reason for using multiple
RTEs, last I checked). It would be easy enough to fix the multiple
RTEs issue if partitioning wasn't a factor. I didn't manage to do much
review of partitioning at all. I don't really understand how the patch
implements partitioning. Input from a subject matter expert might help
matters quite a lot.

Pavan hasn't added support for referencing CTEs, which other database
systems with MERGE have. I think that it ought to be quite doable. It
didn't take me long to get it working myself, but there wasn't follow
through on that (I could have posted the patch, which looked exactly
as you'd expect it to look). I think that we should add support for
CTEs now, as I see no reason for the omission.

In general, I still think that this patch could do with more review,
but I'm running out of time. If you want to commit it, I will not
explicitly try to block it, but I do have misgivings about your
timeframe.

-- 
Peter Geoghegan

  196. Re: [HACKERS] MERGE SQL Statement for PG11

    Alvaro Herrera <alvherre@alvh.no-ip.org> — 2018-03-23T01:02:06Z

    Peter Geoghegan wrote:

> Pavan hasn't added support for referencing CTEs, which other database
> systems with MERGE have. I think that it ought to be quite doable. It
> didn't take me long to get it working myself, but there wasn't follow
> through on that (I could have posted the patch, which looked exactly
> as you'd expect it to look). I think that we should add support for
> CTEs now, as I see no reason for the omission.

Incremental development is a good thing.  Trying to do everything in a
single commit is great when time is infinite or even merely very long,
but if you run out of it, which I'm sure is common, leaving some things
out that can be reasonable implemented in a separate patch is perfectly
acceptable.

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  197. Re: [HACKERS] MERGE SQL Statement for PG11

    Peter Geoghegan <pg@bowt.ie> — 2018-03-23T01:15:54Z

    On Thu, Mar 22, 2018 at 6:02 PM, Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
> Incremental development is a good thing.  Trying to do everything in a
> single commit is great when time is infinite or even merely very long,
> but if you run out of it, which I'm sure is common, leaving some things
> out that can be reasonable implemented in a separate patch is perfectly
> acceptable.

We're talking about something that took me less than an hour to get
working. AFAICT, it's just a matter of tweaking the grammar, and
adding a bit of transformWithClause() boilerplate to the start of
transformMergeStmt().

As I've pointed out on this thread already, I'm often concerned about
supporting functionality like this because it increases my overall
confidence in the design. If it was genuinely hard to add WITH clause
support, then that would probably tell us something about the overall
design that likely creates problems elsewhere. It's easy to say that
it isn't worth holding the patch up for WITH clause support, because
that's true, but it's also beside the point.

-- 
Peter Geoghegan

  198. Re: [HACKERS] MERGE SQL Statement for PG11

    Amit Langote <langote_amit_f8@lab.ntt.co.jp> — 2018-03-23T04:30:47Z

    On 2018/03/23 3:42, Pavan Deolasee wrote:
> A slightly improved version attached. Apart from doc cleanup based on
> earlier feedback, fixed one assertion failure based on Rahila's report.
> This was happening when target relation is referenced in the source
> subquery. Fixed that and added a test case to test that situation.
> 
> Rebased on current master.

I tried these patches (applied 0002 on top of 0001).  When applying 0002,
I got some apply errors:

The next patch would create the file
src/test/isolation/expected/merge-delete.out,
which already exists!  Assume -R? [n]

I managed to apply it by ignoring the errors, but couldn't get make check
to pass; attached regressions.diffs if you want to take a look.

Btw, is 0001 redundant with the latest patch on ON CONFLICT DO UPDATE
thread?  Can I apply just 0002 on top of that patch?  So, I tried that --
that is, skipped your 0001 and instead applied ON CONFLICT DO UPDATE
patch, and then applied your 0002.  I had to fix a couple of places to get
MERGE working correctly for partitioned tables; attached find a delta
patch for the fixes I made, which were needed because I skipped 0001 in
favor of the ON CONFLICT DO UPDATE patch.  But the regression test failure
I mentioned above didn't go away, so it seems to have nothing to do with
partitioning.

Thanks,
Amit

  199. Re: [HACKERS] MERGE SQL Statement for PG11

    Pavan Deolasee <pavan.deolasee@gmail.com> — 2018-03-23T04:57:30Z

    On Fri, Mar 23, 2018 at 10:00 AM, Amit Langote <
Langote_Amit_f8@lab.ntt.co.jp> wrote:

> On 2018/03/23 3:42, Pavan Deolasee wrote:
> > A slightly improved version attached. Apart from doc cleanup based on
> > earlier feedback, fixed one assertion failure based on Rahila's report.
> > This was happening when target relation is referenced in the source
> > subquery. Fixed that and added a test case to test that situation.
> >
> > Rebased on current master.
>
> I tried these patches (applied 0002 on top of 0001).  When applying 0002,
> I got some apply errors:
>
> The next patch would create the file
> src/test/isolation/expected/merge-delete.out,
> which already exists!  Assume -R? [n]
>
> I managed to apply it by ignoring the errors, but couldn't get make check
> to pass; attached regressions.diffs if you want to take a look.
>


Thanks. Are you sure you're using a clean repo? I suspect you'd a previous
version of the patch applied and hence the apply errors now. I also suspect
that you may have made a mistake while resolving the conflicts while
applying the patch (since a file at the same path existed). The failures
also seem related to past version of the patch.

I just checked with a freshly checked out repo and the patches apply
correctly on the current master and regression passes too.
http://commitfest.cputube.org/ also reported success overnight.



>
> Btw, is 0001 redundant with the latest patch on ON CONFLICT DO UPDATE
> thread?  Can I apply just 0002 on top of that patch?  So, I tried that --
> that is, skipped your 0001 and instead applied ON CONFLICT DO UPDATE
> patch, and then applied your 0002.


Yes.  I should probably rebase my patch on your v9 or just include the
relevant changes in the MERGE patch itself to avoid any dependency right
now. Will check.

Thanks,
Pavan

-- 
 Pavan Deolasee                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

  200. Re: [HACKERS] MERGE SQL Statement for PG11

    Amit Langote <langote_amit_f8@lab.ntt.co.jp> — 2018-03-23T07:27:01Z

    On 2018/03/23 13:57, Pavan Deolasee wrote:
> On Fri, Mar 23, 2018 at 10:00 AM, Amit Langote wrote:
>> I managed to apply it by ignoring the errors, but couldn't get make check
>> to pass; attached regressions.diffs if you want to take a look.
>
> Thanks. Are you sure you're using a clean repo? I suspect you'd a previous
> version of the patch applied and hence the apply errors now. I also suspect
> that you may have made a mistake while resolving the conflicts while
> applying the patch (since a file at the same path existed). The failures
> also seem related to past version of the patch.
> 
> I just checked with a freshly checked out repo and the patches apply
> correctly on the current master and regression passes too.
> http://commitfest.cputube.org/ also reported success overnight.

You're right, I seem to have messed something up.  Sorry about the noise.

Also, it seems that the delta patch I sent in the last email didn't
contain all the changes I had to make.  It didn't contain, for example,
replacing adjust_and_expand_inherited_tlist() with
adjust_partition_tlist().  I guess you'll know when you rebase anyway.

Sorry that this is me coming a bit late to this thread, but I noticed a
few things in patch that I thought I should comment on.

1. White space errors

$ git diff master --check
src/backend/executor/execPartition.c:737: trailing whitespace.
+        /*
src/backend/executor/nodeMerge.c:90: indent with spaces.
+        PartitionTupleRouting *proute = mtstate->mt_partition_tuple_routing;
src/backend/executor/nodeMerge.c:116: trailing whitespace.
+
src/backend/executor/nodeMerge.c:565: new blank line at EOF.

2. Sorry if this has been discussed before, but is it OK to use AclMode
like this:

+
+    AclMode     mt_merge_subcommands;   /* Flags show which cmd types are
+                                         * present */

3. I think the comment above this should be updated to explain why the
map_index is being set to the leaf index in case of MERGE instead of the
subplan index as is done in case of plain UPDATE:

-            map_index = resultRelInfo - mtstate->resultRelInfo;
-            Assert(map_index >= 0 && map_index < mtstate->mt_nplans);
-            tupconv_map = tupconv_map_for_subplan(mtstate, map_index);
+            if (mtstate->operation == CMD_MERGE)
+            {
+                map_index = resultRelInfo->ri_PartitionLeafIndex;
+                Assert(mtstate->rootResultRelInfo == NULL);
+                tupconv_map = TupConvMapForLeaf(proute,
+                                mtstate->resultRelInfo,
+                                map_index);
+            }
+            else
+            {

4. Do you  think it would be possible at a later late to change this junk
attribute to contain something other than "tableoid"?

+                        if (operation == CMD_MERGE)
+                        {
+                            j->jf_otherJunkAttNo =
ExecFindJunkAttribute(j, "tableoid");
+                            if
(!AttributeNumberIsValid(j->jf_otherJunkAttNo))
+                                elog(ERROR, "could not find junk tableoid
column");
+
+                        }

Currently, it seems ExecMergeMatched will take this OID and look up the
partition (its ResultRelInfo) by calling ExecFindPartitionByOid which in
turn looks it up in the PartitionTupleRouting struct.  I'm imagining it
might be possible to instead return an integer that specifies "a partition
number".  Of course, nothing like that exists currently, but just curious
if we're going to be "stuck" with this junk attribute always containing
"tableoid".  Or maybe putting a "partition number" into the junk attribute
is not doable to begin with.

5. In ExecInitModifyTable, in the if (node->mergeActionList) block:

+
+        /* initialize slot for the existing tuple */
+        mtstate->mt_existing = ExecInitExtraTupleSlot(estate, NULL);

Maybe a good idea to Assert that mt_existing is NULL before.  Also, why
not set the slot's descriptor right away if tuple routing is not going to
be used.  I did it that way in the ON CONFLICT DO UPDATE patch.

6. I see that there is a slot called mergeSlot that becomes part of
ResultRelInfo of the table (partition) via ri_MergeState.  That means we
might end up creating as many slots as there are partitions (* number of
actions?).  Can't we have just one, say, mt_mergeproj in ModifyTableState
similar to mt_conflproj and just reset its descriptor before use.  I guess
reset will have happen before carrying out an action applied to a given
partition.  When I tried that (see attached delta), nothing got broken.

Thanks,
Amit

  201. Re: [HACKERS] MERGE SQL Statement for PG11

    Pavan Deolasee <pavan.deolasee@gmail.com> — 2018-03-23T11:07:52Z

    On Fri, Mar 23, 2018 at 12:57 PM, Amit Langote <
Langote_Amit_f8@lab.ntt.co.jp> wrote:

>
> Also, it seems that the delta patch I sent in the last email didn't
> contain all the changes I had to make.  It didn't contain, for example,
> replacing adjust_and_expand_inherited_tlist() with
> adjust_partition_tlist().  I guess you'll know when you rebase anyway.
>

Yes, I am planning to fix that once the ON CONFLICT patch is
ready/committed.


>
> 1. White space errors
>
> $ git diff master --check
>
>
Fixed.


>
> 2. Sorry if this has been discussed before, but is it OK to use AclMode
> like this:
>
> +
> +    AclMode     mt_merge_subcommands;   /* Flags show which cmd types are
> +                                         * present */
>

Hmm. I think you're right. Defined required flags in nodeModifyTable.c and
using those now.


>
> 3. I think the comment above this should be updated to explain why the
> map_index is being set to the leaf index in case of MERGE instead of the
> subplan index as is done in case of plain UPDATE:
>
> -            map_index = resultRelInfo - mtstate->resultRelInfo;
> -            Assert(map_index >= 0 && map_index < mtstate->mt_nplans);
> -            tupconv_map = tupconv_map_for_subplan(mtstate, map_index);
> +            if (mtstate->operation == CMD_MERGE)
> +            {
> +                map_index = resultRelInfo->ri_PartitionLeafIndex;
> +                Assert(mtstate->rootResultRelInfo == NULL);
> +                tupconv_map = TupConvMapForLeaf(proute,
> +                                mtstate->resultRelInfo,
> +                                map_index);
> +            }
> +            else
> +            {
>

Done. I wonder though if we should just always set ri_PartitionLeafIndex
even for regular UPDATE and always use that to retrieve the map.


>
> 4. Do you  think it would be possible at a later late to change this junk
> attribute to contain something other than "tableoid"?
>
> +                        if (operation == CMD_MERGE)
> +                        {
> +                            j->jf_otherJunkAttNo =
> ExecFindJunkAttribute(j, "tableoid");
> +                            if
> (!AttributeNumberIsValid(j->jf_otherJunkAttNo))
> +                                elog(ERROR, "could not find junk tableoid
> column");
> +
> +                        }
>
> Currently, it seems ExecMergeMatched will take this OID and look up the
> partition (its ResultRelInfo) by calling ExecFindPartitionByOid which in
> turn looks it up in the PartitionTupleRouting struct.  I'm imagining it
> might be possible to instead return an integer that specifies "a partition
> number".  Of course, nothing like that exists currently, but just curious
> if we're going to be "stuck" with this junk attribute always containing
> "tableoid".  Or maybe putting a "partition number" into the junk attribute
> is not doable to begin with.
>

I am not sure. Wouldn't adding a new junk column require a whole new
machinery? It might be worth adding it someday to reduce the cost
associated with the lookups. But I don't want to include the change in this
already largish patch.


>
> 5. In ExecInitModifyTable, in the if (node->mergeActionList) block:
>
> +
> +        /* initialize slot for the existing tuple */
> +        mtstate->mt_existing = ExecInitExtraTupleSlot(estate, NULL);
>
> Maybe a good idea to Assert that mt_existing is NULL before.  Also, why
> not set the slot's descriptor right away if tuple routing is not going to
> be used.  I did it that way in the ON CONFLICT DO UPDATE patch.
>

Yeah, I plan to update this code once the other patch gets in. This change
was mostly borrowed from your/Alvaro's patch, but I don't think it will be
part of the MERGE patch if the ON CONFLICT DO UPDATE patch gets in ahead of
this.


>
> 6. I see that there is a slot called mergeSlot that becomes part of
> ResultRelInfo of the table (partition) via ri_MergeState.  That means we
> might end up creating as many slots as there are partitions (* number of
> actions?).  Can't we have just one, say, mt_mergeproj in ModifyTableState
> similar to mt_conflproj and just reset its descriptor before use.  I guess
> reset will have happen before carrying out an action applied to a given
> partition.  When I tried that (see attached delta), nothing got broken.
>
>
Thanks! It was on my TODO list. So thanks for taking care of it. I've
included your patch in the main patch. I imagine we can similarly set the
tuple descriptor for this slot during initialisation if target table is a
non-partitioned table. But I shall take care of that along with
mt_existing. In fact, I wonder if we should combine mt_confproj and
mt_mergeproj and just have one slot. They are mutually exclusive in their
use, but have lot in common.

As someone who understands partitioning best, do you have any other
comments/concerns regarding partitioning related code in the patch? I would
appreciate if you can give it a look and also run any tests that you may
have handy.

Thanks,
Pavan

-- 
 Pavan Deolasee                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

  202. Re: [HACKERS] MERGE SQL Statement for PG11

    Pavan Deolasee <pavan.deolasee@gmail.com> — 2018-03-23T11:26:44Z

    On Fri, Mar 23, 2018 at 6:45 AM, Peter Geoghegan <pg@bowt.ie> wrote:

> On Thu, Mar 22, 2018 at 6:02 PM, Alvaro Herrera <alvherre@alvh.no-ip.org>
> wrote:
> > Incremental development is a good thing.  Trying to do everything in a
> > single commit is great when time is infinite or even merely very long,
> > but if you run out of it, which I'm sure is common, leaving some things
> > out that can be reasonable implemented in a separate patch is perfectly
> > acceptable.
>
> We're talking about something that took me less than an hour to get
> working. AFAICT, it's just a matter of tweaking the grammar, and
> adding a bit of transformWithClause() boilerplate to the start of
> transformMergeStmt().
>>
>>
>> I quickly implemented CTE support myself (not wCTE support, since
>> MERGE doesn't use RETURNING), and it wasn't tricky. It seems to work
>> when I mechanically duplicate the approach taken with other types of
>> DML statement in the parser. I have written a few tests, and so far it
>> holds up.
>>
>
> Ok, thanks. I started doing something similar, but great if you have
> already implemented. I will focus on other things for now.
>
>

I am sorry. I was under the impression that you're actually writing this
piece of code and hence did not pay much attention till now. I should have
confirmed with you instead of assuming. I think it's a bit too late now,
but I will give it a fair try tomorrow. I don't want to spend too much time
on it though given how close we are to the deadline. As Alvaro said, we can
always revisit this for pg12.


> As I've pointed out on this thread already, I'm often concerned about
> supporting functionality like this because it increases my overall
> confidence in the design. If it was genuinely hard to add WITH clause
> support, then that would probably tell us something about the overall
> design that likely creates problems elsewhere. It's easy to say that
> it isn't worth holding the patch up for WITH clause support, because
> that's true, but it's also beside the point.
>

Understood.

Thanks,
Pavan

-- 
 Pavan Deolasee                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

  203. Re: [HACKERS] MERGE SQL Statement for PG11

    Amit Langote <langote_amit_f8@lab.ntt.co.jp> — 2018-03-23T12:33:57Z

    On 2018/03/23 20:07, Pavan Deolasee wrote:
> On Fri, Mar 23, 2018 at 12:57 PM, Amit Langote wrote:
>> Also, it seems that the delta patch I sent in the last email didn't
>> contain all the changes I had to make.  It didn't contain, for example,
>> replacing adjust_and_expand_inherited_tlist() with
>> adjust_partition_tlist().  I guess you'll know when you rebase anyway.
>>
> 
> Yes, I am planning to fix that once the ON CONFLICT patch is
> ready/committed.

OK, thanks.

>> 3. I think the comment above this should be updated to explain why the
>> map_index is being set to the leaf index in case of MERGE instead of the
>> subplan index as is done in case of plain UPDATE:
>>
>> -            map_index = resultRelInfo - mtstate->resultRelInfo;
>> -            Assert(map_index >= 0 && map_index < mtstate->mt_nplans);
>> -            tupconv_map = tupconv_map_for_subplan(mtstate, map_index);
>> +            if (mtstate->operation == CMD_MERGE)
>> +            {
>> +                map_index = resultRelInfo->ri_PartitionLeafIndex;
>> +                Assert(mtstate->rootResultRelInfo == NULL);
>> +                tupconv_map = TupConvMapForLeaf(proute,
>> +                                mtstate->resultRelInfo,
>> +                                map_index);
>> +            }
>> +            else
>> +            {
>>
> 
> Done. I wonder though if we should just always set ri_PartitionLeafIndex
> even for regular UPDATE and always use that to retrieve the map.

In the regular UPDATE case, we'd be looking at a resultRelInfo that's from
the ModifyTableState's per-subplan result rel array and the
TupleConversionMap array perhaps accepts subscripts that are in range 0 to
mtstate->nplans - 1 (not 0 to nparts - 1), so using ri_PartitionLeafIndex
there would be incorrect, I think.

>> 4. Do you  think it would be possible at a later late to change this junk
>> attribute to contain something other than "tableoid"?
>>
>> +                        if (operation == CMD_MERGE)
>> +                        {
>> +                            j->jf_otherJunkAttNo =
>> ExecFindJunkAttribute(j, "tableoid");
>> +                            if
>> (!AttributeNumberIsValid(j->jf_otherJunkAttNo))
>> +                                elog(ERROR, "could not find junk tableoid
>> column");
>> +
>> +                        }
>>
>> Currently, it seems ExecMergeMatched will take this OID and look up the
>> partition (its ResultRelInfo) by calling ExecFindPartitionByOid which in
>> turn looks it up in the PartitionTupleRouting struct.  I'm imagining it
>> might be possible to instead return an integer that specifies "a partition
>> number".  Of course, nothing like that exists currently, but just curious
>> if we're going to be "stuck" with this junk attribute always containing
>> "tableoid".  Or maybe putting a "partition number" into the junk attribute
>> is not doable to begin with.
>>
> 
> I am not sure. Wouldn't adding a new junk column require a whole new
> machinery? It might be worth adding it someday to reduce the cost
> associated with the lookups. But I don't want to include the change in this
> already largish patch.

No I wasn't suggesting that we do that in this patch.  Also, I wasn't
saying that we store the "partition index" in a *new* junk column, but
*instead of* tableoid as this patch does.  My question was whether we can
replace tableoid that we are going to store with this patch in the MERGE's
source plan's targetlist with something else in the future.

>> 5. In ExecInitModifyTable, in the if (node->mergeActionList) block:
>>
>> +
>> +        /* initialize slot for the existing tuple */
>> +        mtstate->mt_existing = ExecInitExtraTupleSlot(estate, NULL);
>>
>> Maybe a good idea to Assert that mt_existing is NULL before.  Also, why
>> not set the slot's descriptor right away if tuple routing is not going to
>> be used.  I did it that way in the ON CONFLICT DO UPDATE patch.
>>
> 
> Yeah, I plan to update this code once the other patch gets in. This change
> was mostly borrowed from your/Alvaro's patch, but I don't think it will be
> part of the MERGE patch if the ON CONFLICT DO UPDATE patch gets in ahead of
> this.

OK.

>> 6. I see that there is a slot called mergeSlot that becomes part of
>> ResultRelInfo of the table (partition) via ri_MergeState.  That means we
>> might end up creating as many slots as there are partitions (* number of
>> actions?).  Can't we have just one, say, mt_mergeproj in ModifyTableState
>> similar to mt_conflproj and just reset its descriptor before use.  I guess
>> reset will have happen before carrying out an action applied to a given
>> partition.  When I tried that (see attached delta), nothing got broken.
>>
>>
> Thanks! It was on my TODO list. So thanks for taking care of it. I've
> included your patch in the main patch. I imagine we can similarly set the
> tuple descriptor for this slot during initialisation if target table is a
> non-partitioned table. But I shall take care of that along with
> mt_existing. In fact, I wonder if we should combine mt_confproj and
> mt_mergeproj and just have one slot. They are mutually exclusive in their
> use, but have lot in common.

OK.

> As someone who understands partitioning best, do you have any other
> comments/concerns regarding partitioning related code in the patch? I would
> appreciate if you can give it a look and also run any tests that you may
> have handy.

Actually, I don't yet understand the full scope of what MERGE is supposed
to do.  I guess if it gives same answers for a partitioned table as it
does for regular tables for different MERGE commands, that's enough to say
that MERGE supports partitioning.  But maybe there are corner cases where
MERGE doesn't work same with partitioning because of some underlying
implementation details (mostly executor, afaics).  I will try to test more
early next week.

Thanks,
Amit

  204. Re: [HACKERS] MERGE SQL Statement for PG11

    Simon Riggs <simon@2ndquadrant.com> — 2018-03-23T13:55:14Z

    On 23 March 2018 at 11:26, Pavan Deolasee <pavan.deolasee@gmail.com> wrote:
>
>
> On Fri, Mar 23, 2018 at 6:45 AM, Peter Geoghegan <pg@bowt.ie> wrote:
>>
>> On Thu, Mar 22, 2018 at 6:02 PM, Alvaro Herrera <alvherre@alvh.no-ip.org>
>> wrote:
>> > Incremental development is a good thing.  Trying to do everything in a
>> > single commit is great when time is infinite or even merely very long,
>> > but if you run out of it, which I'm sure is common, leaving some things
>> > out that can be reasonable implemented in a separate patch is perfectly
>> > acceptable.
>>
>> We're talking about something that took me less than an hour to get
>> working. AFAICT, it's just a matter of tweaking the grammar, and
>> adding a bit of transformWithClause() boilerplate to the start of
>> transformMergeStmt().
>>>
>>>
>>> I quickly implemented CTE support myself (not wCTE support, since
>>> MERGE doesn't use RETURNING), and it wasn't tricky. It seems to work
>>> when I mechanically duplicate the approach taken with other types of
>>> DML statement in the parser. I have written a few tests, and so far it
>>> holds up.

Peter, if you have the code and you consider it important that this
subfeature is in PostgreSQL, why not post the code so we can commit
it?

Why would we repeat what has already been done?

-- 
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  205. Re: [HACKERS] MERGE SQL Statement for PG11

    Peter Geoghegan <pg@bowt.ie> — 2018-03-23T20:06:42Z

    On Fri, Mar 23, 2018 at 6:55 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
> Peter, if you have the code and you consider it important that this
> subfeature is in PostgreSQL, why not post the code so we can commit
> it?

Fair enough. Attached patch shows what I'm on about. This should be
applied on top of 0001_merge_v23e_onconflict_work.patch +
0002_merge_v23e_main.patch. I'm not expecting an authorship credit for
posting this patch.

One thing that the test output shows that is interesting is that there
is never a "SubPlan 1" or "InitPlan 1" in EXPLAIN output -- it seems
to always start at "SubPlan 2". This probably has nothing to do with
CTEs in particular. I didn't notice this before now, although there
were no existing tests of EXPLAIN in the patch that show subplans or
initplans.

Is this somehow related to the issue of using two RTEs for the target
relation? That's certainly why we always see unaliased target table
"m" with the alias "m_1" in EXPLAIN output, so I would not be
surprised if it caused another EXPLAIN issue.

-- 
Peter Geoghegan

  206. Re: [HACKERS] MERGE SQL Statement for PG11

    Pavan Deolasee <pavan.deolasee@gmail.com> — 2018-03-24T06:52:20Z

    On Sat, Mar 24, 2018 at 1:36 AM, Peter Geoghegan <pg@bowt.ie> wrote:

>
> Fair enough. Attached patch shows what I'm on about. This should be
> applied on top of 0001_merge_v23e_onconflict_work.patch +
> 0002_merge_v23e_main.patch. I'm not expecting an authorship credit for
> posting this patch.
>

Thanks for the patch. I will study and integrate this into the main patch.


>
> One thing that the test output shows that is interesting is that there
> is never a "SubPlan 1" or "InitPlan 1" in EXPLAIN output -- it seems
> to always start at "SubPlan 2". This probably has nothing to do with
> CTEs in particular. I didn't notice this before now, although there
> were no existing tests of EXPLAIN in the patch that show subplans or
> initplans.
>

This query e.g. correctly starts at InitPlan 1

postgres=# EXPLAIN MERGE INTO m USING (SELECT 1 a, 'val' b) s ON m.k = s.a
WHEN NOT MATCHED THEN INSERT VALUES ((select count(*) from pg_class), s.b);
                               QUERY PLAN
-------------------------------------------------------------------------
 Merge on m  (cost=16.30..43.83 rows=6 width=106)
   InitPlan 1 (returns $0)
     ->  Aggregate  (cost=16.26..16.27 rows=1 width=8)
           ->  Seq Scan on pg_class  (cost=0.00..15.41 rows=341 width=0)
   ->  Hash Right Join  (cost=0.03..27.55 rows=6 width=106)
         Hash Cond: (m_1.k = s.a)
         ->  Seq Scan on m m_1  (cost=0.00..22.70 rows=1270 width=14)
         ->  Hash  (cost=0.02..0.02 rows=1 width=96)
               ->  Subquery Scan on s  (cost=0.00..0.02 rows=1 width=96)
                     ->  Result  (cost=0.00..0.01 rows=1 width=36)
(10 rows)



>
> Is this somehow related to the issue of using two RTEs for the target
> relation? That's certainly why we always see unaliased target table
> "m" with the alias "m_1" in EXPLAIN output, so I would not be
> surprised if it caused another EXPLAIN issue.
>

I don't think it's related to using two RTEs. The following EXPLAIN for a
regular UPDATE query also shows a SubPlan starting at 2. I think it's just
to do with how planner assigns the plan_id.

postgres=# EXPLAIN WITH cte_basic AS (SELECT 1 a, 'cte_basic val' b) UPDATE
m SET v = (SELECT b || ' merge update' FROM cte_basic WHERE cte_basic.a =
m.k LIMIT 1) ;
                                  QUERY PLAN

------------------------------------------------------------------------------
 Update on m  (cost=0.01..54.46 rows=1270 width=42)
   CTE cte_basic
     ->  Result  (cost=0.00..0.01 rows=1 width=36)
   ->  Seq Scan on m  (cost=0.00..54.45 rows=1270 width=42)
         SubPlan 2
           ->  Limit  (cost=0.00..0.02 rows=1 width=32)
                 ->  CTE Scan on cte_basic  (cost=0.00..0.02 rows=1
width=32)
                       Filter: (a = m.k)
(8 rows)

A quick gdb tracing shows that the CTE itself is assigned plan_id 1 and the
SubPlan then gets plan_id 2. I can investigate further, but given that we
see a similar behaviour with regular UPDATE, I don't think it's worth.

Thanks,
Pavan

-- 
 Pavan Deolasee                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

  207. Re: [HACKERS] MERGE SQL Statement for PG11

    Peter Geoghegan <pg@bowt.ie> — 2018-03-24T17:19:33Z

    On Fri, Mar 23, 2018 at 11:52 PM, Pavan Deolasee
<pavan.deolasee@gmail.com> wrote:
> A quick gdb tracing shows that the CTE itself is assigned plan_id 1 and the
> SubPlan then gets plan_id 2. I can investigate further, but given that we
> see a similar behaviour with regular UPDATE, I don't think it's worth.

Clearly I jumped the gun on this one. I agree that this is fine.

-- 
Peter Geoghegan

  208. Re: [HACKERS] MERGE SQL Statement for PG11

    Andreas Seltenreich <seltenreich@gmx.de> — 2018-04-07T19:11:48Z

    Hi,

sqlsmith triggered an assertion with the following MERGE statement
against the regression database.  Testing was done with master at
039eb6e92f.  Backtrace below.

regards,
Andreas

MERGE INTO public.pagg_tab_ml_p3 as target_0
USING public.hash_i4_heap as ref_0
ON target_0.b = ref_0.seqno
WHEN MATCHED AND ((select bitcol from public.brintest limit 1 offset 92)
	 > cast(null as "bit"))
    and (false)
   THEN UPDATE  set
    b = target_0.b,
    a = target_0.b
WHEN NOT MATCHED
  AND cast(null as text) ~ cast(nullif(case when cast(null as float8) <= cast(null as float8) then cast(null as text) else cast(null as text) end
	,
      cast(null as text)) as text)
THEN DO NOTHING;

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x00007f25474cf42a in __GI_abort () at abort.c:89
#2  0x0000556c14b75bb3 in ExceptionalCondition (
    conditionName=conditionName@entry=0x556c14d09bf8 "!(list != ((List *) ((void *)0)))", 
    errorType=errorType@entry=0x556c14bc4dbd "FailedAssertion", fileName=fileName@entry=0x556c14d3022c "list.c", 
    lineNumber=lineNumber@entry=390) at assert.c:54
#3  0x0000556c1495d580 in list_nth_cell (list=<optimized out>, n=<optimized out>) at list.c:390
#4  0x0000556c1495d5d6 in list_nth (list=list@entry=0x0, n=<optimized out>) at list.c:413
#5  0x0000556c14911fa5 in adjust_partition_tlist (tlist=0x0, map=<optimized out>, map=<optimized out>) at execPartition.c:1266
#6  0x0000556c14913049 in ExecInitPartitionInfo (mtstate=mtstate@entry=0x556c16c163e8, resultRelInfo=<optimized out>, 
    proute=proute@entry=0x556c16c29988, estate=estate@entry=0x556c16c15bf8, partidx=0) at execPartition.c:683
#7  0x0000556c1490ff80 in ExecMergeMatched (junkfilter=0x556c16c15bf8, tupleid=0x7ffe8088a10a, slot=0x556c16c22e20, 
    estate=0x556c16c15bf8, mtstate=0x556c16c163e8) at execMerge.c:205
#8  ExecMerge (mtstate=mtstate@entry=0x556c16c163e8, estate=estate@entry=0x556c16c15bf8, slot=slot@entry=0x556c16c22e20, 
    junkfilter=junkfilter@entry=0x556c16c2b730, resultRelInfo=resultRelInfo@entry=0x556c16c15e48) at execMerge.c:127
#9  0x0000556c14933614 in ExecModifyTable (pstate=0x556c16c163e8) at nodeModifyTable.c:2179
#10 0x0000556c1490c0ca in ExecProcNode (node=0x556c16c163e8) at ../../../src/include/executor/executor.h:239
#11 ExecutePlan (execute_once=<optimized out>, dest=0x556c16c111b8, direction=<optimized out>, numberTuples=0, 
    sendTuples=<optimized out>, operation=CMD_MERGE, use_parallel_mode=<optimized out>, planstate=0x556c16c163e8, 
    estate=0x556c16c15bf8) at execMain.c:1729
#12 standard_ExecutorRun (queryDesc=0x556c16c1bce8, direction=<optimized out>, count=0, execute_once=<optimized out>)
    at execMain.c:364
#13 0x0000556c14a6ba52 in ProcessQuery (plan=<optimized out>, 
    sourceText=0x556c16b2ac08 "...", params=0x0, 
    queryEnv=0x0, dest=0x556c16c111b8, completionTag=0x7ffe8088a500 "") at pquery.c:161
#14 0x0000556c14a6bceb in PortalRunMulti (portal=portal@entry=0x556c16b96468, isTopLevel=isTopLevel@entry=true, 
    setHoldSnapshot=setHoldSnapshot@entry=false, dest=dest@entry=0x556c16c111b8, altdest=altdest@entry=0x556c16c111b8, 
    completionTag=completionTag@entry=0x7ffe8088a500 "") at pquery.c:1291
#15 0x0000556c14a6c979 in PortalRun (portal=portal@entry=0x556c16b96468, count=count@entry=9223372036854775807, 
    isTopLevel=isTopLevel@entry=true, run_once=run_once@entry=true, dest=dest@entry=0x556c16c111b8, 
    altdest=altdest@entry=0x556c16c111b8, completionTag=0x7ffe8088a500 "") at pquery.c:804
#16 0x0000556c14a6859b in exec_simple_query (
    query_string=0x556c16b2ac08 "MERGE INTO public.pagg_tab_ml_p3 as target_0\nUSING public.hash_i4_heap as ref_0\nON target_0.b = ref_0.seqno\nWHEN MATCHED AND ((select bitcol from public.brintest limit 1 offset 92)\n > cast(null as \"bi"...) at postgres.c:1121
#17 0x0000556c14a6a341 in PostgresMain (argc=<optimized out>, argv=argv@entry=0x556c16b56ad8, dbname=<optimized out>, 
    username=<optimized out>) at postgres.c:4149
#18 0x0000556c1474eac4 in BackendRun (port=0x556c16b4c030) at postmaster.c:4409
#19 BackendStartup (port=0x556c16b4c030) at postmaster.c:4081
#20 ServerLoop () at postmaster.c:1754
#21 0x0000556c149ec017 in PostmasterMain (argc=3, argv=0x556c16b257d0) at postmaster.c:1362
#22 0x0000556c1475006d in main (argc=3, argv=0x556c16b257d0) at main.c:228

  209. Re: MERGE SQL Statement for PG11

    André Brière <brierand@outlook.com> — 2018-06-25T16:54:35Z

    The status of this is quite unclear to me:
- There are two email threads and the most recent email is in the original one (https://www.postgresql.org/message-id/flat/CANP8+jKitBSrB7oTgT9CY2i1ObfOt36z0XMraQc+Xrz8QB0nXA@mail.gmail.com/); I think the status should be set to "Waiting for Author" but it was reset to "Needs review" by Pavan on 06/19/2018 (based on the second email thread?)
- The patch was reverted in https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=08ea7a2291db21a618d19d612c8060cda68f1892